In the meantime, here are some supporting videos and slides:

• PDF slides: https://drive.google.com/file/d/1D9OO1shPkpnrVKuJcZmGrw-AGg-tv0KK/view?usp=drive_link
• PPTX slides: https://docs.google.com/presentation/d/17ZH-qTsZlWqPjqv2rrOZHxDOVnFXE3nW/edit?usp=drive_link&ouid=110353084444571868371&rtpof=true&sd=true
• music slide: https://drive.google.com/file/d/16jw30i53bTL7VuaH0RNTq2qrOe65HNMd/view?usp=drive_link
• architectural time: https://drive.google.com/file/d/11Rk3rXdWWYGSJ1FfQt7tvoSr8hCkjuvz/view?usp=drive_link
• Code used in slides: https://github.com/dbernheisel/elixir_conf_2024_dancing_with_data

When we recognize music, movies, old photos, or video games of the past, it sometimes has a powerful effect: nostalgia. It’s a pattern of the past that we remember fondly.

When we write code, we also recall patterns of the past to consider as tools to help solve today’s problem.

I’d like to have some fun exploring the connection between software development and nostalgia.

What is Nostalgia

The word “nostalgia” started as a medical condition. Doctors invented the word with the Greek roots ‘nostos’, meaning returning home, and ‘algos’ meaning pain. It was invented in 1688, when Swiss soldiers would miss home so much, that they would mentally break down, and no longer be able to ‘be a soldier’; they longed for who they used to be.

The condition of ‘nostalgia’ evolved into much less than a medical condition, and more as a temporary emotion. The word is no longer associated with the negative effects to the person; in fact quite the opposite: it now means “fondly remembering the past”.

Inducing nostalgia can boost psychological well-being, increase feelings of self-esteem, social belonging, and encourage psychological growth. Nostalgia can be a restorative way of coping with negative stress.

Marketing knows about nostalgia; it’s a powerful selling tool. There are 11 Star Wars films; 10 Batman movies; 16 Final Fantasy games. Remasters of remasters, remakes on top of remakes. Remixes of older songs; you get it. Generally, folks want to be in that mental space.

Where Nostalgia Comes From

This is called the Lifespan Retrieval Curve.

• 0-5yrs Childhood Amnesia
• 10-30yrs Reminisence Bump (this is our nostalgic period)
• 30+ Period of Recency

Our autobiographical memory – that is, our psychological history of ourselves – on average shows that most of our memories are formed between the ages of 10-30 years old. Younger than 10 is considered childhood amnesia; greater than that is considered too recent. For the life of me, I can’t remember what being 6 was like. Likewise, I couldn’t tell you what I did 2 years ago that affected my life profoundly. It’s unconsciously regarded as too recent to be formative of “who I am.”

Our young adulthood is where we develop our self-identity; it’s when we discover who we are and want to be. We’ll come back around to that bit. For me, the time period is the 1990s and 2000s.

Back in the 1990s, there were several things that I did:

• Listened to cassettes I recorded from the stereo on my fuzzy earphones.
• Stopped at every Magic Eye book in Books-a-million, Borders, Hastings, or Barnes & Noble to see if I could see “it”.
• Spent 30 minutes or more just trying to decide what movie to rent; maybe it took so long because I couldn’t accept that the movie I wanted to see didn’t have the VHS behind it.
• Got really excited when the phone rang.
• Watched Bewitched and Dick Van Dyke on Nick-at-night.
• Replayed in my head the “The Log Song” from Ren & Stimpy. It was an earworm that never left. To this day I still hum it.
• I cleaned up poop from my Tamagotchi, jumped on goombas, and a little latter double-barrel shotgunned demons in Doom.

Let’s stop at that last point: video games.

Theory of Fun

Super Mario. The hero plumber that would grow up to be the most iconic and recognizable video game character of all time. Made by a company formed in 1889: Nintendo.

Video games are a source of nostalgia for many people. They’re fun. They’re a way to escape the stress of the day. Many video games, such as Final Fantasy, are epic stories that we can get lost within; just like books, but these are interactive! Others, like classic Mario, have very little story and just focus on addictive game mechanics.

Video games are developed to be fun; otherwise why would we play them? Most gamers aren’t playing for pure punishment (looking at you, Mega Man 9). Turns out, there’s a lot of social science that goes into designing games to get and keep players!

The book “Theory of Fun” outlines some basics of what it takes to create a good game. In it, the author Raph Koster includes a quote by Chris Crawford saying:

fun is the emotional response to learning

And later Raph Koster:

Fun in games arises out of mastery. It arises out of comprehension. It is the act of solving puzzles that makes games fun. With games, learning is the drug.

The idea was, games are systems built to help us learn patterns, and fun is a neurochemical reward to encourage us to keep trying.

These games are built to teach us patterns, and then delightfully introduce variations on them, which keeps us learning. We see new situations as the game progresses and have new ways to apply the learned patterns to them; or perhaps combine patterns.

Let’s look at an example.

In the original Super Mario Bros, we know that Mario moves right to advance to the goal; we can tell because he’s on the left side of the screen– you can’t go left; you must go right.

The game introduces a pattern by establishing if you touch the Goombas, you die. Noted– don’t touch bad guys.

Let’s try pushing one of the buttons on the gamepad. Turns out Mario can jump! Let’s avoid those bad guys. Noted– I can move right and jump

There are blocks now. Looks like if I jump into them, I might get something. If I jump on top, it acts as a platform.

It introduces a pattern by establishing if you touch the mushroom, you grow. Noted– there are power-ups that make me stronger

You experiment with two elements: Getting a mushroom, and then touching the enemy. Turns out that’s a mistake and now you shrink back down. Noted– can’t kill bad guys by only being big, and when I’m big and get hurt, I become small.

You experiment with two more elements: jump on the enemy’s heads!. Noted– bad guys can be jumped on.

Now we’re at a pit. Probably not safe to go in there, but maybe the screen will move down and reveal a new area. Let’s try. Noted– can’t go down pits.

These are universal patterns now in games. We know instinctively that we just don’t touch enemies, and we should jump over pits.

Patterns are Fun

Earlier we talked about nostalgia. Remember, nostalgia is fondly remembering the past. It’s remembering a pattern that may be long gone from our current life.

Like nostalgia, video games also has psychological benefits. “Theory of Fun” includes a study from 2011 released by East Carolina University, that found that casual gamers who played in 30 minute periods showed an 87% improvement in cognitive response time, reduced depression symptoms by 57%, and 215% increase in executive functioning. That’s incredible!

I’m starting to get the idea that nostalgia and video games are good news for our brains and well-being. This isn’t limited to video games either; although I’m focusing on them; these taught patterns also exist in music and movies.

Thinking about pop music, almost every song on the radio has a recognizable pattern, whether you’ve picked it out or not:

• Intro
• Verse 1
• Chorus
• Verse 2
• Chorus
• Bridge
• Chorus
• Outro

If we analyzed the musical notes, there are patterns of notes that musicians know sound good together called scales. Notes played that are not on the given scale will create a dissonant sound; an out-of-tune guitar, for example.

Likewise, movies and books typically follow a pattern:

• Introduce characters and setting
• Rising action or inciting incident.
• Climax and Falling
• Denouement (lessons learned)

Consider Tic-Tac-Toe; it’s definitely a game, but the pattern is easily perceived and there is no continuing variation. Mastery of Tie-Tac-Toe is quick.

Consider Korean dramas; usually a love story between a jerk guy and a shy girl. A series of events leads to the shy girl turning away from jerk guy, almost falling in love with another fella but then jerk guy gets hit by a car BAM now he has amnesia. This turns jerk guy into nice guy. Shy girl now likes nice guy. They get together, and then OH NOEES nice guy remembers stuff and now he’s a mostly-jerk-but-now-somehow-ok-guy. Seriously, every k-drama is this in a nutshell. Predictable.

Alas, there is a point where patterns become boring and no longer fun.

Delight tends to wear thin very quickly. Real fun comes from challenges that are always at the margin of our ability. When the balance is really perfect, people often zone out.

There has to be variance in the patterns to stay interesting. Games do this by having multiple stages or areas, power-ups, faster cars, etc. Music does this with different instruments, tempos, singers, etc. Movies does this new characters, settings, twists, etc.

Micro-lifecycles

We’ve explored nostalgia and patterns in popular media such as video games, movies, and music. What does this have to do with software development?

Generally, software development consists of applying code patterns to problems. Software developers, or really any artisans of their craft, notice and remember patterns. They remember which patterns work well and which don’t; what problems those solution-patterns are good fits for and terrible fits for. Software developers are players in the game of software.

People are amazing pattern-matchers. They look for patterns in everything; sometimes even finding patterns where there wasn’t supposed to be, like a cloud formation. In software, patterns that emerge are often collected into groups. At the macro-level, these patterns are called architectures, such as MVC (Model-View-Controller) common in backend frameworks, or MVVM (Model-View-ViewModel) common in frontend frameworks, or REST (Representational state transfer) common in web communication.

At the micro-level, it looks like spoilers a game. This micro-lifecycle is how every game works. Let’s look at the event loop of a game from the perspective of a gamer:

+----------------------------------------------------------+
|                                                          |
|              All forms of feedback:                      |
|     Art, animation, sound, music, movement, story.       |
|                                                          |
+-------+-------------------------------------------^------+
        |                                           |
        |                 A GAME ATOM               |
        | updates                                   |
        |                                      +----+-----------+
        |                                      |                |
   +----v--------+    +---------------+        |  🔁  place     |
   |             |    |               | input  |                |
   |   problem   +---->  preparation  +------->+ +------------+ |
   |             |    |               |        | |    core    | |
   +-------------+    +---------------+        | |  mechanic  | |
                                               | +------------+ |
                                               |                |
                                               +----------------+

Let’s change some of these words around and apply it to software development.

+----------------------------------------------------------+
|                                                          |
|              All forms of feedback:                      |
|      code review, performance, test suite, correctness,  |
|               those dang customers                       |
+-------+-------------------------------------------^------+
        |                                           |
        |                 A SOFTWARE UNIT           |
        | updates                                   |
        |                                      +----+-----------+
        |                                      |                |
   +----v--------+    +---------------+        |  🔁  test      |
   |             |    |               | write  |                |
   |   problem   +---->  preparation  +------->+ +------------+ |
   |             |    |               | code   | |   logic    | |
   +-------------+    +---------------+        | |            | |
                                               | +------------+ |
                                               |                |
                                               +----------------+

This looks like test-driven development!

• We’re presented a problem
• We prepare for a solution
• Have a test (place)
• Run the logic (core mechanic)
• And adjust according to the feedback we experience.

Software developers and gamers experience the same micro-lifecycles! But let’s call it what it is: it’s a rapid pattern.

Software development revolves around languages, architectures, and frameworks. These are all codified patterns with dissertations, white-papers, systems, and books written about them. Coders leverage these patterns to solve their problems faster.

Ruby on Rails is considered a really productive and fast framework for the Ruby language. It introduced the MVC architecture to a lot of young developers. Many developers of folks find Rails enjoyable; you have to follow the patterns that they set out to keep progress smooth. PHP Laravel is similar. There are dozens of frameworks that are aimed to be productive.

It wasn’t always like this though. Let’s talk about JavaScript.

Back in the early 2000s, the web looked entirely different. JavaScript, believe it or not, wasn’t as widely used as it is today. It was a mess; different browsers implemented different features, and sometimes the same feature but with different interfaces to it. There weren’t many patterns. It wasn’t as popular because it wasn’t as enjoyable to work within. Think of it as the wild wild west; only those willing to leave the comforts of what they knew and explore the unknown would be able to move the language and its ecosystem forward and help define it.

2006 came around and so did a little library called jQuery, which codified how to do basic actions in JavaScript. It standardized, unofficially, how to access elements on the page, how to handle events, how to animate. It was an amazing contribution to the JavaScript community. It’s still used today! jQuery inspired browser makers and some of those behaviors were codified into the language properly, and some of those APIs are integrated into the browsers now and work natively without the library.

Several years later in 2010, Google released a framework called Angular (known as AngularJS now). This was a full framework that applied the MVC and/or MVVM architecture to JavaScript.

Years later in 2013, another library called React was released. Together with other libraries like React Router, Redux and more, it essentially formed a new framework.

Shortly after in 2014, Vue was released, picking up some loved patterns from both Angular and React.

These islands of patterns are where developers congregated. Like little nation-states, they liked their leaders, disliked their leaders, stagnated, experienced rapid growth, poached employees, etc.

Together, the patterns that these libraries and frameworks introduced helped make frontend development much more enjoyable. JavaScript today is one of most prolific languages out there, no doubt in part because patterns emerged.

I think these patterns determine how much fun we’re going to have programming.

Elixir

Let’s talk about a software language called Elixir.

Elixir is a relatively new (born ~2011) functional language.

Elixir is a dynamic, functional language designed for building scalable and maintainable applications.

Elixir leverages the Erlang VM, known for running low-latency, distributed and fault-tolerant systems, while also being successfully used in web development, embedded software, data ingestion, and multimedia processing domains.

Elixir is beloved by developers, particularly seasoned developers, and I want to explore why. In fact, Elixir is the entire motivation for me to produce this content.

The Vault team [at Heroku] doing Elixir is only three engineers. Most of their apps are used internally, so they are generally not worried about performance. They continue using Elixir because they feel productive and happy with it. They have also found it is an easier language to maintain compared to their previous experiences.

– “PaaS with Elixir at Heroku”

What makes Elixir so enjoyable? As we have been exploring, fun is the emotional response to learning, and since people are amazing pattern-matchers, we constantly look for patterns. Perhaps Elixir has good patterns.

Let’s look at some common Elixir programming patterns. These patterns can be found in many other language. Maybe these patterns are why it’s enjoyable to develop in Elixir?

Pipelines

When we think about what an application accomplishes, we often frame it in our minds as a ruleset and transformation of data. That’s where we start, at least.

• Data comes in
• Data is transformed
• Data goes out

In Elixir, you can accomplish this with the pipeline operator:

"My   String"
|> String.trim()
|> String.replace("My", "Your")
|> String.upcase()
#=> "YOUR STRING"

The pipeline starts with a string "My String" and passes it into the first position of the next function String.trim/1. The result of that function is then passed into the first position of the next function String.replace(..., "My", "Your") and so on until the end when we have the result "YOUR STRING".

Elixir developers reading this are probably already bored. We’ve seen this pattern a million times. But stick with me!

It doesn’t have to be all about transformation either. If at any point, we wanted to see what the data was between any of those functions, we could also throw in an IO.inspect() between the functions.

"My   String"
|> String.trim()
|> IO.inspect(label: "the string is")
|> String.replace("My", "Your")
|> String.upcase()

#=> the string is: "My String"
#=> "YOUR STRING"

Move the IO.inspect down a little bit…

"My   String"
|> String.trim()
|> String.replace("My", "Your")
|> IO.inspect(label: "the string is")
|> String.upcase()

#=> the string is: "Your String"
#=> "YOUR STRING"

It’s a simple and effective way to see the state of your string as it’s passed through the pipeline.

The pipeline pattern is everywhere in Elixir, and it makes it clear how the data is being operated upon. We can see instantly that it’s being trimmed, we’re replacing a word, and upcasing the string. There’s no need to name interstitial stages of the string while it’s being operated upon. Opposed to this:

trimmed_string = String.trim("My String")
replaced_string = String.replace(trimmed_string, "My", "Your")
upcased_string = String.upcase(replaced_string)

This is a simple example, but your eyes have to scan to the right to see the beginning of the data transformation, and then search for where that variable goes in the next function.

Pipelines are everywhere with Elixir.

The pipeline operator |> is not in every language, nor does it need to be in order to write your code towards data transformation pipelines, but because the operator exists, it steers developers to work in such a pattern.

Another form of a data pipeline is using the with macro. You may have noticed in our simple example about that a string is passed in and out of all the functions; there wasn’t a great way to check the output if it was successful; with helps with that. Let’s look at that in our next pattern:

Monads

Ok, before I lose you, we’re not going to explore what monads are; it doesn’t matter. But, their effect is powerful and I want to explore a pattern that they enable. If you don’t know what a monad is, that’s totally fine. Here’s a simple definition: A monad is a unit of data and metadata (for the developer) about the data.

Not to say that Elixir has complete support for academic and mathematical monads like Haskell, but the Elixir community’s code style leans towards using monads, perhaps without even knowing it.

The best and smallest example I can think of for monads is the Result Monad:

case this_might_work() do
  {:ok, success_data} ->
    happy_path(success_data)

  {:error, data_about_error} ->
    unhappy_path(data_about_error)
end

In Elixir, we’re pattern-matching on the result of this_might_work(), and that function returns a result monad. The monad in this case is a 2-item tuple. The tuple begins with either :ok or :error which is metadata about the accompanied data. {METADATA, DATA}

If the returned tuple’s first element is :ok, then bind the data to success_data and move on. Likewise, if it’s :error, then bind the unsuccessful data to data_about_error and move on.

This is a pervasive pattern in Elixir. I bet that 100% of Elixir codebases out there have this pattern in it. It’s extremely effective in how to route the data in the pipeline.

Here’s another one:

with {:ok, burger} <- McDonalds.order("burger"),
     {:ok, fries} <- McDonalds.order("fries"),
     {:ok, milkshake} <- McDonalds.order("milkshake") do
  {:ok, [burger, fries, milkshake]}
else
  {:error, error_message} ->
    Emotions.frustrate()
    Memory.remember(error_message)
    {:error, "did not complete order"}
end

Look! We’re combining two patterns now: a pipeline and result monads. Here, we are chaining several result monads together to form a pipeline with the help of the with macro. Inside the with statement:

• order a burger. If you get a burger, then continue
• order some fries. If you get fries, then continue
• order a milkshake. If you get a milkshake, then you’re done! Grab your food and go!

If they don’t have one of the above items, then the pipeline will stop and return the first encountered error. SPOILERS the ice cream machine was broken, so we’re going to get an error when ordering a milkshake.

The result monad is extremely effective in crafting how to route data in pipelines.

Happy Path

The previous example includes another pattern: seeing the happy path. I want to show you a couple of implementations of a process; one in Ruby and another in Elixir.

def order(params)
  user = User.find(params["user_id"])       #| Happy path
  return unless user                        #| Unhappy path

  item = Warehouse.find(params)             #| Happy path
  return unless item                        #| Unhappy path

  money = Billing.charge_user(user, item)   #| Happy path
  return unless money                       #| Unhappy path

  schedule_delivery(user, item)
end

In this implementation, I can see that it needs a user, an in-stock item, and money charged before we schedule a delivery, but visually, I see a mixture of unhappy and happy paths. It’s a little difficult to quickly pick out what the desired outcomes are supposed to be. Again, this is a simple example; real-world code is harder than this in many cases.

Let’s try this in Elixir using the patterns we’ve identified earlier:

def order(params) do
  with {:ok, user} <- User.find(params),               #|\
       {:ok, item} <- Warehouse.find(params),          #| \ Happy Path
       {:ok, _money} <- Billing.charge(user, item) do  #| /
    schedule_delivery(user, item)                      #|/
  else                                                 #|\
    {:error, _} -> nil                                 #|_\ Unhappy Path
  end
end

Here I can see that the happy path is consolidated into one area. I know exactly what is expected for the process to complete successfully. I also know where the error cases are handled.

This code focuses on the solution first, which is important because for many many developers, the first goal is to make it work, then make it fast, then make it beautiful. With the solution consolidated at the top of the function, any readers know what makes it work and where to handle error cases.

Elixir isn’t perfect in this in all cases; it’s possible in any language to obfuscate a function’s happy path, but again the tool is here which helps visually organize the happy path.

Focusing on the solution is important; developers tend to think of what can go wrong and develop for those first. This leads to “gold-plating” your code before it often sees any real usage. More importantly, this way of thinking infects other areas of your psyche. It leads to anxiety!

Totality

Total functions are functions that give you a valid return value for every combination of valid arguments. Total functions are really about communication. In Elixir, this is accomplished with a mixture of pattern matching and typespecs.

Elixir is NOT a statically-typed language, so it’s missing some tools here to ensure complete totality, but it does have tooling to help with communicating totality. Elixir can define types, and there’s a tool called Dialyzer that can integrate with your editor of choice that can provide hints to the developer. ElixirLS is one of those tools.

The pattern of totality in Elixir, in a sense, is simply good documentation. Developers annotate their code with docs and types to communicate to other developers what the expected input and output should be, and state guarantees on how it should work.

A basic example: String.upcase expects an input string and outputs a string. It will always succeed if you give it that input, and will always return to you a string. It does not lie to you; it’s total.

Here’s another example:

defmodule HeroAction do
  @type hero :: WonderWoman.t() | Batman.t()
  @type baddie :: GenericBaddie.t() | BigBoss.t()
  @type slogan :: String.t()

  @spec fight_criminal(hero(), baddie()) :: {:ok, slogan()}
  def fight_criminal(%WonderWoman{} = hero, %GenericBaddie{} = baddie) do
    %FightScene{
      protagonist: hero,
      antagonist: baddie
    }
    |> beat_up(hero.bracelets_of_submission)
    |> interrogate(hero.lasso_of_truth)
    |> hand_to_police()

    {:ok, "I will fight for those who cannot fight for themselves"}
  end

  def fight_criminal(%Batman{} = hero, %GenericBaddie{} = baddie) do
    %FightScene{
      protagonist: hero,
      antagonist: baddie
    }
    |> beat_up(hero.tools)
    |> interrogate(hero.intimindation)
    |> hand_to_police()

    {:ok, "I am vengence"}
  end

  # ...
end

HeroAction.fight_criminal(batman, criminal)

The guarantee of this code is that you’re able to hand it anything that matches the type of hero and baddie, and you will always get an {:ok, slogan} in return. There are no other possible way this could fail; it’s a total function. The heroes always win and they take every opportunity to say their slogan.

Additionally, if you’re using an ElixirLS-enabled editor, you can get some hints while developing:

Communicating with each other is an important part of developing software. Rarely is software developed by one and only one person, and even if developed solo, you have your future and past selves to communicate with. Documenting such typespecs assist you with creating total functions that let you compose functions safely. That leads us to our final pattern we’ll explore today.

Composition

Here’s the clencher, all the patterns we see above are repeated everywhere. When you have a big problem to solve with code, you approach it the same exact way as you would a small problem: with functions. You don’t have to shift paradigms in how to solve a problem depending on its scale.

Object-oriented programming has objects in the large, and methods in the small.

Functional programming has functions in the large, and functions in the small.

– Scott Wlaschin: Functional Programming Design Patterns

This is comforting, because when we already have a business problem we’re trying to solve, we don’t need a language problem or tooling problem clouding our judgment. Our brain can focus on one thing at a time and “zone out” on solving the business problem with our familiar and patterned toolset.

In gaming, it’s important to not frustrate the player with too much difficulty too soon. You have to give the player tools to solve the presented problems first. Once they’ve mastered the tools, then you can make them apply their newfound knowledge to the new problems. Likewise, we want Elixir (our tool) to not be one of the problems as we solve the main problem (the business). Otherwise they will give up and tell all their friends “this game sux.”

Understood

These patterns are pervasive in all Elixir codebases I’ve seen, and because those patterns are common, developers have a good way to communicate with each other which is the ultimate goal.

Nostalgia has a positive effect on the person. These patterns of our past remind us of who we used to be or some aspect of how we identify ourselves. Most importantly, nostalgia increases our sense of belonging and encourages growth.

The Theory of Fun explains that fun is the emotional response to learning; the reward for continuing to learn. What people learn are patterns in a variety of settings: school, childhood, adulthood, that one time at the bar where I talked too much, competitions, and (like we explored) video games. We often have fun when seeing patterns reoccur.

Software development is chock-full of patterns. Developers apply patterns constantly to problems they encounter when trying to build something. I focused on Elixir here, but this is true for many languages and frameworks; some maybe more true than others. Elixir has excellent patterns, and developers that share the same patterns understand each other, which is all we really want in the end anyway – to be understood.

I’ll leave you with a quote:

To [find good patterns] we must rely on feelings more than intellect.

To work our way towards a shared language once again, we must first learn how to discover patterns which are deep, and capable of generating life.

The specific patterns, out of which a building or a town is made, may be alive or dead. To the extent they are alive, they let our inner forces loose, and, set us free; but when they are dead they keep us locked in inner conflict.

– Christopher Alexandar (author of A Pattern Language)

Want to learn more about this topic? Check out these inspiring talks:

• Garrett Smith: The Timeless Way of Building Erlang Apps
• Cameron Price: Micropatterns

Special thanks to Quinn Wilton for proofreading!

What I’m giving up:

1. Offline access
2. Progressive Web App capabilities.
3. Free hosting (bummer.)
4. Pre-compiled syntax highlighting. Gatsby has a great VSCode-powered highlighter that runs during compilation.
5. A mature asset-processing pipeline to optimize images, especially for responsiveness.
6. Easier scaling if traffic gets out of hand. It’s fundamentally easier to scale static content using a CDN service. But, given Elixir’s web request performance, I’m not too worried about this yet, and maybe I shouldn’t be for this little ol’ blog.

What I’m gaining:

1. A tool-chain that I understand thoroughly and can contribute to; I actually know Elixir and Phoenix.
2. The ability to show off LiveView-enabled components (stay tuned!)
3. The ability to send UI updates from the backend to the frontend efficiently. If you didn’t notice, and if there is more than 1 current reader on this article, look underneath the title of this article. You should see “x current readers”. That’s enabled by LiveView and reacts immediately. If there aren’t any other readers, then open a couple of tabs on this page, and watch the number climb.

What I’m not losing:

1. A fast load time.
2. A reactive local development tool-chain. For example, when editing the post, I can save the file and my dev server shows the changes almost immediately.

The “giving up” list doesn’t bother me too much, at least not on my blog. If you’re managing a huge static site with lots of pages; perhaps look elsewhere at the moment. The biggest reason is that Elixir lacks an asset-pipeline to optimize images. It wouldn’t be impossible to have with a LiveView-powered blog with optimized images, but as far as I know, you’ll have to roll-your-own.

I was initially worried that moving a static site to an Elixir-powered web application would have too many trade-offs, but I was wrong! Many static sites today don’t offer pre-compiled syntax highlighting, but I was using one, so it’s a loss for me. Losing the PWA and cached offline access don’t bother me.

The “gaining” list is worth it to me, as an Elixir developer, because it unlocks some potential for what I want to do. I’m considering collecting LiveView-powered components, and how else would I be able to show them off without a LiveView site? So, the trade-off is totally worth it to me.

The “not losing” list was surprising. Let’s talk about it; I’ll show you some informal benchmarks.

Let’s look at the dev server

Elixir and Phoenix offer live-reloading when assets change (ie, JavaScript and CSS), and code-reloading when Elixir code changes. To make this reloading work in the context of a blog written with Markdown, I used NimblePublisher. NimblePublisher is a small system that sets up compiling Markdown files into HTML at compile-time. Create a /blog folder, put your markdown in there, and configure Elixir to watch it for changes.

# config/dev.exs
config :bern, BernWeb.Endpoint,
  live_reload: [
    patterns: [
      ~r"priv/static/.*(js|css|png|jpeg|jpg|gif|svg)$",
      ~r"lib/bern_web/(live|views)/.*(ex)$",
      ~r"lib/bern_web/templates/.*(eex)$",
      ~r"posts/*/.*(md)$" # <-- right here!
    ]
  ]

Now, a big plus for Elixir is the boot-up time. Granted if you haven’t changed files that requires the entire Elixir project to re-compile, booting up the dev server is very quick. How quick? 36 times quicker. But before you run off to social media with this stat, just remember that it’s not very scientific, nor is it always fair. It’s not apples-to-apples.

Let’s say that I have inspiration one day and I want to write a blog post. I go to my blog folder, start the dev server, and start writing. I haven’t changed anything; I just want to write a post.

Here’s the Elixir dev server running iex -S mix phx.server:

Now the Gatsby dev server running yarn run gatsby develop:

If I’m reading this correctly, Gatsby took about 36 seconds to boot. I hadn’t changed anything in the project! In comparison, Elixir and Phoenix booted in 1 second, and that includes Webpack; granted, nothing changed in the project and some of this is thanks to HardSourceWebpackPlugin; but even without that plugin, Elixir is still up and running and serving requests and whenever Webpack finishes it will live-reload with those compiled CSS and JS resources.

Again, I’ll repeat, this is not apples-to-apples. For example, I’m giving up compile-time syntax highlighting for non-Elixir code blocks. Gatsby is likely recompiling everything without needing to, whereas Elixir is a compiled language that only recompiles what it needs to.

Syntax Highlighting

NimblePublisher also handles syntax-highlighting at compile-time, but unfortunately only when the language is Elixir or Erlang. Since I write about other languages, including Ruby, JavaScript, Bash, Vim, and more, I need more syntax highlighting options. NimblePublisher is using Makeup for highlighting; so perhaps I can contribute by making more Makeup lexers.

To cover the gap in the meantime, I’m going to syntax-highlight at runtime; in other words, you visiting this page ran some JavaScript to highlight syntax for me! Since this blog is now powered with LiveView, I handled it in a hook.

Let’s see how it works

// Configure Webpack to make another bundle for vendored code like Highlight.js
// assets/webpack.config.js

module.exports = (_env, options) => {
  return {
    // ...
    entry: {
      app: ["./js/app.js", "./css/app.css"],
      vendor: ["./js/vendor.js"] // <-- this part
    }
  }
}

// ==================================================
// assets/js/vendor.js
import "./highlighter";


// ==================================================
// assets/js/highlighter.js
import hljs from 'highlight.js/lib/core';
import javascript from 'highlight.js/lib/languages/javascript';
import shell from 'highlight.js/lib/languages/shell';
import bash from 'highlight.js/lib/languages/bash';
import erb from 'highlight.js/lib/languages/erb';
import ruby from 'highlight.js/lib/languages/ruby';
import vim from 'highlight.js/lib/languages/vim';
import yaml from 'highlight.js/lib/languages/yaml';
import json from 'highlight.js/lib/languages/json';
import diff from 'highlight.js/lib/languages/diff';
import xml from 'highlight.js/lib/languages/xml';

// Yeah, this isn't very sexy, but I'm trying to keep the bundle small
// by only opting into languages I use in blog posts.
hljs.registerLanguage('javascript', javascript);
hljs.registerLanguage('shell', shell);
hljs.registerLanguage('bash', bash);
hljs.registerLanguage('eex', erb);
hljs.registerLanguage('ruby', ruby);
hljs.registerLanguage('vim', vim);
hljs.registerLanguage('yaml', yaml);
hljs.registerLanguage('json', json);
hljs.registerLanguage('diff', diff);
hljs.registerLanguage('html', xml);

window.highlightAll = function(where = document) {
  where.querySelectorAll('pre code').forEach((block) => {
    const lang = block.getAttribute("class")
    // Since Makeup handles Elixir code at compile-time, I don't need
    // highlight.js to care about this language.
    if (lang !== "makeup elixir") {
      const { value: value } = hljs.highlight(lang, block.innerText);
      block.innerHTML = value;
    }
  });
}

window.highlightAll()  // this covers on page load

// ==================================================
// assets/js/hooks.js
let hooks = {}

hooks.Highlight = {
  mounted() {
    window.highlightAll(this.el) // this covers LiveView patches
  }
}

export default hooks

// ==================================================
// assets/js/app.js
import hooks from "./hooks";

// the normal phoenix LiveView initialization, but passing in the hooks:
window.liveSocket = new LiveSocket("/live", Socket, {
  hooks,
});

window.liveSocket.connect();

<!-- lib/bern_web/templates/layout/root.html.leex -->
<script defer phx-track-static type="text/javascript"
  src="<%= Routes.static_path(@conn, "/js/vendor.js") %>"></script>


<!-- lib/bern_web/live/blog_show.html.leex -->
<div id="post-content-<%= @post.id %>" phx-hook="Highlight" phx-update="ignore">
  <%= raw(@post.body) %>
</div>

Cool. That’s not terrible, but still I would much-prefer ALL syntax highlighting happen at compile-time. It’s not like this stuff changes in runtime so there’s no reason for every visitor’s browser to do syntax highlighting for me. Kudos to the Gatsby tool-chain for solving this problem via gatsby-remark-vscode.

What about performance?

I’m glad you asked.

There’s no practical difference in my opinion, which is an upset to common perception of JavaScript-powered static sites being much slimmer and faster.

Let’s look.

Here’s the Blog Show page powered by Gatsby:

Here’s the Blog Show page powered by Elixir Phoenix LiveView:

REMEMBER the LiveView-powered blog now requires browser-based syntax highlighing, so there’s an extra 79.5kb of JavaScript that needs to download and parse and run. Also notice that there’s an extra font on the Gatsby site since it needed it on this page.

All-in-all this is very comparable judging by the DomContentLoaded and Load times, but notice that Elixir and Phoenix are barely beating Gatsby in this scenario.

Let’s look at the Blog Index page powered by Gatsby:

Here’s the Blog Index page powered by Elixir Phoenix LiveView:

Elixir is slower in this case for DOMContentLoaded and Load times by tens of milliseconds. Hopefully imperceptible to most visitors. Also, that’s not bad considering this is running on a $5/mo server in the US East Coast, opposed to CDN-cached static content at the edge by Cloudflare and GitHub Pages. Also considering that LiveView-powered index page is literally loading every single post it their entirety, but not rendering the bodies of the posts; so there is room for optimization here.

Lastly, just for the giggles, here’s the Lighthouse scores:

Gatsby:

Elixir Phoenix LiveView:

The SEO hit on the Phoenix app is because of a canonical URL not being correct. Once the LiveView blog replaces the Gatsby blog, it’ll be 100 again. Also, the Accessibility score took a hit because of different styling where the contrast isn’t high enough in one spot. Really, the only difference is Gatsby’s score of 95 on performance vs LiveView’s score of 99 on performance, and the lack of PWA on LiveView. I don’t really think these scores mean anything between the two sites; but take it as you will.

Now, of course, if you check the score now, you’ll see it’s different because I’ve since added some web font preloading which kills the score. This only happens on the first load though, thankfully.

Migrating to Phoenix LiveView from Gatsby

Here are the big changes. You’ll find several more details on your own if you consider migrating to NimblePublisher depending on your setup.

1. Update all the front-matter to a new format.
2. Rename all your markdown files. In my case, the structure was ./blog/blog-id/index.md with images beside the post, and it moved to ./posts/DATE-blog-id.md with all images moving to ./assets/static/images.
3. Change all references to images in Markdown posts. Bummer point here is that it does not use the digested version of images referenced in your Markdown posts. This would require NimblePublisher or Earmark (the markdown parser) to be aware of resources like images and relative URLs, and replace them with Phoenix-generated paths. Currently it is not, and not sure how it could be since it’s happening at compile-time, and asset digests occur after compilation.
4. Convert any React components into Phoenix templates.
5. Make navbar LiveView-aware.

For example, the front-matter went from this:

---
title: "Phoenix LiveView and Views"
tags: ["elixir", "phoenix"]
date: 2020-06-29
excerpt: |
  Everytime I build a LiveView application, I learn something new and find a new
  pattern, and some concept finally _clicks_. Today, that concept that cemented
  in my mind is how Phoenix and Phoenix LiveView renders templates.

  I want to show you a couple different View-rendering strategies. This should
  help you decide which strategy to use.
---

To this:

%{
  title: "Phoenix LiveView and Views",
  tags: ["elixir", "phoenix"],
  description: """
  Every time I build a LiveView application, I learn something new and find a
  new pattern, and some concept finally _clicks_. Today, that concept that
  cemented in my mind is how Phoenix and Phoenix LiveView renders templates.

  I want to show you a couple different View-rendering strategies. This should
  help you decide which strategy to use.
  """
}
---

The date moved out of the front-matter and relies only on the filename now on disk. It was a little tedious, but since my blog only has a handful of articles, it wasn’t worth it to script something.

For the most part, the HTML structure of the page is the same as it was on Gatsby. I literally copy-pasted the HTML from my Gatsby React components into the Phoenix templates and modified them to not rely on React.

Getting the navbar links to know what page they were on was another challenge. I’m using AlpineJS on this blog, so I leverage some event dispatching in AlpineJS when clicking on navbar links, which is caught and assigns the active/inactive classes on the links. It’s optimistic UI and shadows the LiveView click events, but it should be fine.

For example:

<!-- lib/bern_web/templates/layout/nav.html.eex -->
<!-- loading the route from the conn will take care of the initial page load -->
<!-- further LiveView-powered navigation needs some help from AlpineJS -->

<% [route | _] = @conn.path_info %>
<nav @navigate="open = false; currentRoute = $event.detail" x-data="{currentRoute: '<%= route %>', open: false}">
  <%= live_redirect "Blog", to: Routes.blog_path(@conn, :index),
    class: "all my classes",
    "@click": "$dispatch('navigate', 'blog')",
    ":class": "{
      'all my active classes': currentRoute === 'blog',
      'all my inactive classes': currentRoute !== 'blog'
    }" %>
  <!-- ... -->
</nav>

Conclusion

I enjoyed the process. I’m making the right decision for myself to evolve a static-site blog into a web application that also serves static content. If you’re only using it for static content, then it’s probably not the right choice to move to LiveView if all you care about is efficiency.

It turns out that argument for JavaScript-powered static site being much better than server-generated content is simply not true. Elixir-powered static sites could also use some help in the areas of syntax highlighting and asset pipelines; so perhaps that’s where I can turn some of my attention to next!

NimblePublisher isn’t the only option out there for Phoenix! You could have a full-fledged Markdown engine in Phoenix that renders the markdown in runtime and interpolates dynamic content during the request. This would solve the issue of asset versioning.

If I got anything wrong or missed something obvious (totally likely) then please tell me! Hit me up on twitter @bernheisel

Oh yeah, it’s open source

Check out the source. Lastly, this is hosted on Linode.

1. Elixir Regex Tester
2. A request logger, much like Phoenix Live Dashboard‘s
3. An internal webmail server, for me to receive and send email through SendGrid. I hope to open-source this soon when it’s ready.
4. Another private work-related project.

Everytime I build one, I learn something new and find a new pattern, and some concept finally clicks. Today, that concept that cemented in my mind is how Phoenix and Phoenix LiveView renders templates.

I want to show you a couple different View-rendering strategies. This should help you decide which strategy to use.

All of these strategies work, this is purely about opinionated code organizing, but who doesn’t love reading opinions? Plus we’ll learn how the views are rendered.

This is written while using Phoenix LiveView 0.13.3.

TL;DR

Glossary of examples:

1. MyLive = The LiveView module
2. MyView = The standard Phoenix View module, not a LiveView.
3. my_live.html.leex = The template rendered by MyLive or MyView

If you have a simple LiveView, then you can implement render(assigns) and inline your html with the ~L sigil. No my_live.html.leex file needed.

If you have a LiveView with lots of HTML, then you should use the standard LiveView placement, and put your my_live.ex and my_live.html.leex next to each other under lib/my_app_web/live. You don’t need to define render/1 because the default will work. Omit it.

If you have a LiveView with lots of HTML helper functions that you want to separate from business logic in the LiveView:

1. Add your own standard Phoenix view MyView (or a better name).
2. Move your my_live.html.leex file to the standard Phoenix locations (ie, lib/my_app_web/templates/my).
3. Implement your own render(assigns) that calls MyAppWeb.MyView.render("my_live.html", assigns). Phoenix LiveView will still work; just remember to keep the html file named with an .html.leex extension so the LiveView rendering engine kicks in.

Remember that you can create shared Views. Alternatively, if your helpers are used across multiple views and are generic, you can create a plain module that encapsulates your HTML helpers. I usually call mine ComponentView and use it inside any of my templates, for example: Component.primary_button("My Link", to: "yadayada").

If you want to use a regular View, but co-locate the template to the LiveView module, as in you don’t want to go back to the vanilla Phoenix file structure but still need a separate MyView for your HTML helpers, you can specify the root folder and path to look in when creating your MyView by supplying an option: use Phoenix.View, root: "lib/my_app_web/live", path: "". This is explained in the Phoenix.View docs. This can be wrapped up into a convenience macro though. Read on for more info.

This totally ignores LiveComponent as an option. If your LiveView can be broken up into interactive components, then breaking out into a LiveComponent is a good option to look into and works just like a LiveView. For the purpose of this post and exploring how rendering works, we’re going to treat LiveComponents the same as a LiveView.

ToC

• Phoenix Controller/View/Template
• Phoenix LiveView with a template Part 1
• Pluggy Controllers
• Phoenix LiveView with a template Part 2
• Phoenix LiveView with an inline template
• Phoenix LiveView with an external template

Default Phoenix Controller/View/Template

First, to remember where we came from, I want to show a standard Phoenix Controller/View/Template pattern. There are several modules involved that the Plug.Conn travels through in order to turn into a response for the end-user.

1. Incoming request via :cowboy
2. Endpoint
3. Router
4. Controller
5. View
6. Template (not a module)
7. Outgoing response via :cowboy

### Router - lib/my_app_web/router.ex
defmodule MyAppWeb.Router do
  use MyAppWeb, :router

  # ...snip...
  scope "/", MyAppWeb do
    pipe_through :browser

    get "/", PageController, :home
  end
  # ...snip...
end


### Controller - lib/my_app_web/controllers/page_controller.ex
defmodule MyAppWeb.PageController do
  use MyAppWeb, :controller  #<-- injects some logic to handle receiving
  # the conn and passing the conn on to cowboy

  def home(conn, _params) do
    render(conn, "home.html")
  end
end


### View - lib/my_app_web/views/page_view.ex
defmodule MyAppWeb.PageView do
  use MyAppWeb, :view  #<-- injects some logic to handle evaluating
  # the embedded elixir in your templates
end

<!-- Template - lib/my_app_web/templates/page/home.html.eex
We're going to ignore the layout stuff for now. Just know that it's
also evaluated and this template is a part of it -->
<p>Yo! You're rendering the home page</p>

In my mind, the template is the end of the show, though that’s not technically correct; the real end of the line is the controller. The controller is using the view module to evaluate the HTML and puts the result into the Plug.Conn’s resp_body. The controller terminates the flow and the once-a-request-and-now-a-response Plug.Conn is returned to the to the underlying web server, which delivers the payload to the end-user through the HTTP connection.

Default Phoenix LiveView without render/1

We’re here to learn about LiveView though, so let’s see an example of a LiveView without a render/1 function.

### lib/my_app_web/live/my_live.ex
defmodule MyAppWeb.MyLive do
  use MyAppWeb, :live_view

  @impl Phoenix.LiveView
  def mount(_params, _session, socket) do
    # do stuff

    {:ok, socket}
  end
end

<!-- lib/my_app_web/live/my_live.html.leex -->
<p>Yo! I'm rendered by a LiveView</p>

Ok, without a controller, how does a given Phoenix LiveView handle the request? Here’s a secret: a LiveView is also an ordinary controller.

Now… we may not use it like an ordinary Phoenix controller, but the request is firstly handled like an ordinary web request; one with a Plug.Conn and a full HTML response back to the user. The LiveView spices are garnished after the HTML is delivered to the user and a new websocket is initiated to the server to the page updates to the page.

As said in the LiveView docs:

A LiveView begins as a regular HTTP request and HTML response, and then upgrades to a stateful view on client connect, guaranteeing a regular HTML page even if JavaScript is disabled. Any time a stateful view changes or updates its socket assigns, it is automatically re-rendered and the updates are pushed to the client.

Prove it!

– you

ok ok.. I’ll prove it. To prove that it’s a regular controller, we’ll need to look at some of Phoenix LiveView’s source code. Let’s look at the code that makes live("/my-route", MyLive) work in the router.

defmacro live(path, live_view, action \\ nil, opts \\ []) do
  quote bind_quoted: binding() do
    {action, router_options} =
      Phoenix.LiveView.Router.__live__(__MODULE__, live_view, action, opts)

    # vvvvv THIS PART
    Phoenix.Router.get(path, Phoenix.LiveView.Plug, action, router_options)
    # ^^^^^ THIS PART
  end
end

You see it?! live() is calling this function:

Phoenix.Router.get("/my-route", Phoenix.LiveView.Plug, _action, _options)

You may recognize this as:

get "/my-route", MyController, :show

in your own router. We’re going to ignore the action and options for this post, but the important part is that the live() macro is adding a GET route and calls the Phoenix.LiveView.Plug

But, that plug isn’t a controller…

– you

Ah, but it is! A Phoenix Controller, even the ones you make, are indeed all just plugs underneath. All Phoenix controllers are plugs.

Pluggy Controllers

When your controllers call use MyAppWeb, :controller, it’s injecting code into your controller at compile-time. Let’s explore how that works.

First at step 0 we need to understand that when Elixir code calls use MyUsingModule it’s actually calling MyUsingModule.__using__(opts) at compile-time, and that resulting code is put into the module that called it. Knowing that, let’s follow the use trail.

Starting at the top in our own code:

######################################
### Inside MyAppWeb.PageController ###
######################################

defmodule MyAppWeb.PageController do
  use MyAppWeb, :controller

  def home(conn, _) do
    render(conn, "home.html")
  end
end

#######################
### Inside MyAppWeb ###
#######################

defmacro __using__(which) when is_atom(which) do
  apply(__MODULE__, which, [])
end

def controller do
  quote do
    # vvv let's look in here vvv
    use Phoenix.Controller, namespace: MyAppWeb
    # ^^^ let's look in here ^^^
  end
end


#################################
### Inside Phoenix.Controller ###
#################################

defmacro __using__(opts) do
  quote bind_quoted: [opts: opts] do
    import Phoenix.Controller

    # vvv let's look in here vvv
    use Phoenix.Controller.Pipeline, opts
    # ^^^ let's look in here ^^^

    if Keyword.get(opts, :put_default_views, true) do
      plug :put_new_layout, {Phoenix.Controller.__layout__(__MODULE__, opts), :app}
      plug :put_new_view, Phoenix.Controller.__view__(__MODULE__)
    end
  end
end


##########################################
### Inside Phoenix.Controller.Pipeline ###
##########################################

defmacro __using__(opts) do
  quote bind_quoted: [opts: opts] do

    @behaviour Plug
    ## AHA! HERE'S YOUR CONTROLLER PLUG BEHAVIOUR

    require Phoenix.Endpoint
    import Phoenix.Controller.Pipeline

    Module.register_attribute(__MODULE__, :plugs, accumulate: true)
    @before_compile Phoenix.Controller.Pipeline
    @phoenix_log_level Keyword.get(opts, :log, :debug)
    @phoenix_fallback :unregistered

    @doc false
    def init(opts), do: opts

    @doc false
    def call(conn, action) when is_atom(action) do
      conn
      |> merge_private(
        phoenix_controller: __MODULE__,
        phoenix_action: action
      )
      # fun fact, this function below was introduced
      # ~6 years ago in Phoenix 0.5.0 and utilizes unhygienic functions.
      # (as in you're in deep macro-land and your normal rules don't apply)
      |> phoenix_controller_pipeline(action)
    end

    @doc false
    def action(%Plug.Conn{private: %{phoenix_action: action}} = conn, _options) do
      apply(__MODULE__, action, [conn, conn.params])
    end

    defoverridable init: 1, call: 2, action: 2
  end
end

Wow! Wild. All this means our slim controllers actually have a lot more code in it than it appears, and that’s ok because it makes working in Phoenix much more convenient.

All plugs must implement call/2 which accepts a conn and returns a conn. In our case, we’re looking for a conn that has some rendered HTML.

Back to Default Phoenix LiveView without render/1

Now that we know that LiveViews are a GET request using a standard controller/plug underneath, let’s look at the Phoenix.LiveView.Plug. We’re still looking for how a LiveView gets to the template.

LiveView has a similar __using__ code-path. Let’s look at LiveView’s plug:

def call(%{private: %{phoenix_live_view: {view, opts}}} = conn, _) do
  opts = maybe_dispatch_session(conn, opts)

  # ...snip... there's a lot of code here we're going to skip
  conn
  |> Phoenix.Controller.put_layout(false)
  |> put_root_layout_from_router(opts)
  # this actually is piped into `Controller.live_render(view, opts)`
  # but I'm going to cut/paste what that ends up doing
  |> LiveView.Static.render(conn, view, opts)
  # ... more snipping...
  |> to_rendered_content_tag(socket, tag, view, attrs)
  # ... more snipping...
  |> view.render()  #<-- here here here!
end

Cool; this isn’t anything new so far. This is just confirming that Phoenix LiveView starts off as a regular HTTP request with a full HTML response. How does it render?

We see that it’s calling view.render() where view is our own LiveView, but we didn’t define render/1 yet! Where’s it coming from?

When we called use MyAppWeb, :live_view it kicked off a series of __using__, which includes use Phoenix.LiveView. Inside Phoenix.LiveView it included a @before_compile Phoenix.LiveView.Renderer hook. Let’s check that out.

render? = Module.defines?(env.module, {:render, 1})
root = Path.dirname(env.file)
filename = template_filename(env)
templates = Phoenix.Template.find_all(root, filename)

case {render?, templates} do
  {false, [template]} ->
    ext = template |> Path.extname() |> String.trim_leading(".") |> String.to_atom()
    engine = Map.fetch!(Phoenix.Template.engines(), ext)
    ast = engine.compile(template, filename)

    quote do
      @file unquote(template)
      @external_resource unquote(template)
      def render(var!(assigns)) do
        unquote(ast)
      end
    end
  # ... other clauses
end

Finally! This is where the default render/1 function comes from. Before our LiveView compiles, it checks to see if a render/1 is defined, and if not, it will drop one in for us. The default location for LiveView templates is right next to the LiveView file itself. We see this from the root = Path.dirname(env.file).

Phoenix LiveView with inline render/1

Another option is to implement render/1 ourselves. The docs make this pretty clear how to do that.

### lib/my_app_web/live/my_live.ex
defmodule MyAppWeb.MyLive do
  use MyAppWeb, :live_view

  @impl Phoenix.LiveView
  def mount(_params, _session, socket) do
    # do stuff

    {:ok, socket}
  end

  @impl Phoenix.LiveView
  def render(assigns) do
    ~L"""
    <p>Yo! I'm rendered from a <%= my_helper("vanilla") %> view</p>
    """
  end

  def my_helper("vanilla"), do: "whoops no this is actually live"
end

This feels the most similar to frontend frameworks such as Vue with single-file-components (SFCs), or React.

This is a great option in case your LiveView doesn’t have a lot of HTML. Perhaps you’re implementing a small widget. At some point, however, it becomes a little crowded if you have a lot of business logic handling changes in the LiveView as well as hundreds of lines of HTML and functions to conditionally render some HTML or apply CSS classes; so you might consider separating the HTML out into its own file.

Phoenix LiveView with external render/1

### lib/my_app_web/live/my_live.ex
defmodule MyAppWeb.MyLive do
  use MyAppWeb, :live_view

  @impl Phoenix.LiveView
  def mount(_params, _session, socket) do
    # do stuff

    {:ok, socket}
  end

  @impl Phoenix.LiveView
  def render(assigns) do
    MyAppWeb.MyView.render("my_live.html", assigns)
  end
end


### lib/my_app_web/views/my_view.ex
defmodule MyAppWeb.MyView do
  use MyAppWeb, :view

  def my_helper("vanilla"), do: "whoops no this is actually live"
end

<!-- lib/my_app_web/templates/my/my_live.html.leex -->
<p>Yo! I'm rendered from a <%= my_helper("vanilla") %> view</p>
<!-- this renders "whoops no this is actually live" instead of "vanilla" -->

If I have a lot of HTML helpers, then I tend to prefer separating that into a View module. It’s a little tedious to setup and separate the files, and then jump between them when developing, but it’s clear where functions should go.

This bugged me though, I have HTML floating in ./templates and sometimes in ./live and sometimes inline. Can we consolidate?

Sure we can! Phoenix.View provides an option to look for templates in a different folder. Let’s try it out. We need to supply root and path with use Phoenix.View:

### lib/my_app_web/views/my_view.ex
defmodule MyAppWeb.MyView do
  use Phoenix.View,
    root: "lib/my_app_web/live",
    path: "",
    namespace: MyAppWeb
  # and all the other imports that come with `use MyAppWeb, :view`

  def my_helper("vanilla"), do: "whoops no this is actually live"
end

<!-- lib/my_app_web/live/my_live.html.leex -->
<p>Yo! I'm rendered from a <%= my_helper("vanilla") %> view</p>
<!-- this renders "whoops no this is actually live" instead of "vanilla" -->

It’s exactly the same, except where the HTML is on disk and that we can’t use our use MyAppWeb, :view as-is anymore without some further adjustment. To prove the concept though, copy out all the additional imports you find for views in my_app_web.ex and place it here for now. If it works out, then you can add another clause in my_app_web.ex to handle these kinds of views. Maybe something like this.

### lib/my_app_web.ex

def view_for_live do
  quote do
    use Phoenix.View,
      root: "lib/my_app_web/live",
      path: "",
      namespace: MyAppWeb

    # Import convenience functions from controllers
    import Phoenix.Controller, only: [get_flash: 1, get_flash: 2, view_module: 1]

    # Use all HTML functionality (forms, tags, etc)
    use Phoenix.HTML

    import MyAppWeb.ErrorHelpers
    import MyAppWeb.Gettext
    import Phoenix.LiveView.Helpers
    import MyAppWeb.LiveHelpers
    alias MyAppWeb.Router.Helpers, as: Routes
  end
end

# and then use this instead for your LiveView-centric Views
### lib/my_app_web/views/my_view.ex
defmodule MyAppWeb.MyView do
  use MyAppWeb, :view_for_live

  def my_helper("vanilla"), do: "whoops no this is actually live"
end

What about LiveComponents?

LiveComponents are totally ignored in this article. They’re another great option for organizing interactive partials from your LiveViews. Their rendering strategy is very similar to LiveViews though, and most of this applies to them as well.

Hope these tips help you out! If you have any more tips, tweet at me @bernheisel

Thank you zporter for helping me with the post by proof-reading!

I would really love it if you tweeted about the new podcast and shared with your friends who know or are discovering Elixir. We’re wanting it to be a good and succinct source to catch up on Elixir news and happenings, with some interviews with other Elixir folks!

Check it out at https://podcast.thinkingelixir.com or your favorite source for podcasts (Google, Apple, etc.).

In the first episode, we’re introducing ourselves and getting to know each other.

One of the benefits I love about LiveView is that it enables me to consolidate some of common front-end logic into the backend, where the source of truth belongs. A great example is a form, especially long-running or multi-step forms.

Let me show you what I mean.

This is accomplished without any AJAX calls, no SPAs, and no page reloads.

I coded this form twice. Let me share with you my journey and some techniques I used to help organize code.

The Ugly Way (First pass)

I coded it all with a single LiveView.

It become quite ugly.

I was still trying to figure out what I wanted on the form and still learning LiveView generally. Eventually, this LiveView became an ugly 1000+-line horror show that managed state in multiple places.

It was a single <form> that handled all the fields for the database-backed record, and each step was hidden until you hit “next”, so every change in the form sends the entire form values.

The EEX was something like this:

<div class="container">
  <%= f = form_for(@changeset, phx_validate: :validate, phx_save: :save %>

  <div class="<%= unless @progress.name == "who", do: "hidden" %>">
    <%# my Who-related form inputs %>
  </div>

  <div class="<%= unless @progress.name == "what", do: "hidden" %>">
    <%# my What-related form inputs %>
  </div>

  <div class="<%= unless @progress.name == "when", do: "hidden" %>">
    <%# my When-related form inputs %>
  </div>

  </form>
</div>

I found that this approach has several drawbacks:

• When the user hits “enter”, the form will try to submit. If you’re on the first step, you probably don’t want that to submitted yet until they’re on the last step. You can override this with some JavaScript, but this non-standard behavior made things more complicated than it should be. I’ll need the JavaScript to know which step is last, and track which step it’s currently on. Ugh… I did this and it wasn’t great. I wanted to delete myself.
• When the user is on a different step, you still need to manage all the “state” of other steps. This is a lot of “weight” to worry about and ensure doesn’t change.
• As soon as the user interacts with the form on the first step, validations will occur for the entire form, even for those inputs on hidden steps. This means errors will already be populated before the user even interacted with them.
• Testing the big form was difficult. The tools were great– I just bad-developered and didn’t break it down well.

Generally, I found it harder to “reason about”, especially when I have computed fields and help text based on user input.

For example, I need to persist two DateTimes with timezones, but I don’t want to present that to the user as datetime_selects and have them select a timezone from a drop-down.

Instead I want a date picker, and then separately collect the times and merge it with the user’s detected timezone (this will later be improved to allow them to select a timezone and prefer a user’s set timezone while registering). Something like this:

<%= date_select f, :date %>
<%= time_input f, :start_time %>
<%= time_input f, :end_time %>
Your duration is <%= @duration %>

so in my params, I would receive something like this:

def handle_event("validate", %{"myform" => params}, socket) do
  IO.inspect params, label: "PARAMS"
  {:noreply, socket}
end

#=> PARAMS: %{
#  "date" => %{"year" => 2020, "month" => 1, "day" => 1},
#  "start_time" => "08:00",
# " end_time" => "10:00"
# }

There’s a complicated mechanism in the time pickers that made it harder. I needed to detect what changed:

1. Was it the end_time? Then let’s extend the duration as well and accept the new end_time.
2. Was it the start_time? Then let’s back the end_time up to the same duration away from the start_time.
3. At some point, if we accept user input for duration, then we we’d want to extend the end_time with the new duration.

Now I have some fields, I need to compute them into my event struct somehow. This is how it needs to end up:

record.start_at_tz #=> "America/New_York"
record.start_at_wall #=> ~N[2020-01-01T08:00:00]
record.start_at_utc #=> ~U[2020-01-01T13:00:00Z]

record.end_at_tz #=> "America/New_York"
record.end_at_wall #=> ~N[2020-01-01T10:00:00]
record.end_at_utc #=> ~U[2020-01-01T15:00:00Z]

record.duration #=> 7200 # seconds which is 2 hours

This is going to be a lot of work!

Let’s not have the giant form all be in one template, or even partials; let’s split the form up into components. These components will let me manage these computed fields easier, as well as solve some other UX issues mentioned above.

Let’s break it down:

• Manage form progress in the parent LiveView.
• Split the multi-step form into LiveComponents. At least one for each visible step.
• Send input supplied client-side via phx-hook.
• Handle input changes from the users from the component
• Handle stepped-form submission
• Handle final form submission.

Managing the form progress

I managed the form step progress by defining a %Step{} and then writing the order out in the liveview as a module attribute.

defmodule MyAppWeb.EventLive.Step do
  @moduledoc "Describe a step in the multi-step form and where it can go."
  defstruct [:name, :prev, :next]
end

# in the liveview

defmodule MyAppWeb.EventLive.New do
  # ...snip...

  @steps [
    %Step{name: "who", prev: nil, next: "what"},
    %Step{name: "what", prev: "who", next: "when"},
    %Step{name: "when", prev: "what", next: "where"},
    %Step{name: "where", prev: "when", next: nil},
  ]

  def mount(_params, session, socket) do
    socket = authenticate(socket, session, [:with_organizations, :with_profile])
    first_step = List.first(@steps)
    event = %Event{}
    params = %{creator_id: socket.assigns.current_user.id}

    {:ok,
      socket
      |> assign(:event, event)
      |> assign(:params, params)
      |> assign(:changeset, Event.changeset(event, params))
      |> assign(:progress, first_step)}
  end

  # ...snip...
end

When the underlying live components are finished, they’ll send a message to the parent LiveView which will re-assign :progress; the conditionals in the template will apply/remove the “hidden” class for the next appropriate step, or previous step. You’ll see that as you read on.

Let’s chop up the form.

Extract to LiveComponents

All this ugly-but-necessary logic should live in “form objects”. In Ecto-land these can be managed with embedded schemas. These form objects are responsible for the state of their own fields, and compute their own values without affecting other steps’ values. The domain becomes much clearer.

When the form is submitted, it will trigger the “save” event from the LiveComponent. The LiveComponent can then pass the completed params up to its parent LiveView if the changeset is valid. The parent LiveView can track these params separately, sitting on it until final save, persisted as a draft, or whatever you need.

This has some benefits:

• form submission (hitting enter) no longer needs to override default behavior.
• isolates testing to it’s own form and LiveComponent.
• your form’s “domain” has clearer boundaries.
• user interaction and form validation makes more sense; only the visible form is “tainted” when the user changes it (opposed to it being tainted before the user even sees it).

The multi-step form now looks like this:

<div class="container">
  <div class="<%= unless @progress.name == "who", do: "hidden" %>">
    <%= live_component @socket, WhoComponent,
      id: "who",
      event: @event,
      current_user: @current_user
    %>
  </div>

  <div class="<%= unless @progress.name == "what", do: "hidden" %>">
    <%= live_component @socket, WhatComponent, id: "what", event: @event %>
  </div>

  <div class="<%= unless @progress.name == "when", do: "hidden" %>">
    <%= live_component @socket, WhenComponent,
      id: "when",
      event: @event,
      current_user: @current_user
    %>
  </div>

  <div class="<%= unless @progress.name == "where", do: "hidden" %>">
    <%= live_component @socket, WhereComponent,
      id: "where",
      submit_text: t("Create"),
      event: @event
    %>
  </div>
</div>

Let’s focus on the WhenComponent.

Here’s the big idea:

• Inside of the WhenComponent, we need our embedded_schema to represent and store the fields we care about on the step.

• When loading/updating the component itself, we’re going to initialize the changeset with the fields from the record.

• When handling validation events, we’re going to throw the params into the changeset and assign the new changeset back.

• The computed values will be updated from the changeset and/or pulled out of the changeset and assigned into the socket.

• When handling the save event, we’re going to ensure the changeset is valid, and if so, tell the parent LiveView that we’re good to proceed. We’ll send the struct up to the parent LiveView. This struct will contain the computed fields so it should be easier for the parent to stitch these steps’ params together into the final changeset that’s actually persisted.

Again, the flow should look like this:

1. On mounting, take the Event and pluck the relevant fields out of it to create a WhenComponent form backed by an embedded_schema.
2. When the user is on the step, take the changes as they come and let the user iterate on the form until it’s valid.
3. When the changeset is valid and the user tries to submit it, pass the final struct up to the parent LiveView. The parent LiveView can then switch to the next step.

Here is the component code:

defmodule MyAppWeb.EventLive.WhenComponent do
  use MyAppWeb, :live_component
  alias Ecto.Changeset

  @primary_key false
  embedded_schema do
    field :timezone, :string
    field :start_at_date, :date
    field :start_at_time, :time
    field :start_at, :utc_datetime  # Not on the form. This is computed

    field :end_at_date, :date
    field :end_at_time, :time
    field :end_at, :utc_datetime  # Not on the form. This is computed

    field :duration, :integer  # Not on the form. This is computed and displayed
  end

  @impl Phoenix.LiveComponent
  def update(assigns, socket) do
    whenevent = from_event(assigns.event, assigns.current_user.profile.timezone)
    params = %{}
    changeset = changeset(whenevent, params)

    {:ok,
      socket
      |> assign(assigns)
      |> assign(:when, whenevent)
      |> assign(:when_changeset, changeset)
      |> assign_computed(changeset)
    }
  end

  @impl Phoenix.LiveComponent
  def handle_event("validate", %{"when_component" => params}, socket) do
    adjusted_params = adjust_time_params(params, socket.assigns.when_changeset)
    changeset = changeset(socket.assigns.when, adjusted_params)

    {:noreply,
      socket
      |> assign(:when_changeset, changeset)
      |> assign_computed(changeset)
    }
  end

  @impl Phoenix.LiveComponent
  def handle_event("save", %{"when_component" => params}, socket) do
    socket.assigns.when
    |> changeset(params)
    |> Changeset.apply_action(:insert)
    |> case do
      {:ok, record} ->
        send(self(), {:proceed, record})
        {:noreply, socket}
      {:error, changeset} ->
        {:noreply,
          socket
          |> assign(:when_changeset, changeset)
          |> assign_computed(changeset)
        }
    end
  end

  @impl Phoenix.LiveComponent
  def handle_event("timezone", detected_timezone, socket) do
    params = Map.put(socket.assigns.when_changeset.params, "timezone", detected_timezone)
    changeset = changeset(socket.assigns.when_changeset, params)
    {:noreply,
      socket
      |> assign(:when_changeset, changeset)
      |> assign_computed(changeset)
    }
  end

  @fields ~w[timezone start_at_date start_at_time end_at_date end_at_time duration]a
  def changeset(whenevent, params) do
    whenevent
    |> Changeset.cast(params, @fields)
    |> put_stitched_datetime(:start_at)
    |> put_stitched_datetime(:end_at)
    |> ensure_duration()
  end

  defp put_stitched_datetime(changeset, field) do
    timezone = Changeset.get_field(changeset, :timezone)
    date = Changeset.get_field(changeset, :"#{field}_date")
    time = Changeset.get_field(changeset, :"#{field}_time", ~T[00:00:00])
    {:ok, ndt} = NaiveDateTime.new(date, time)

    Changeset.put_change(changeset, field, DateTime.from_naive!(ndt, timezone))
  end

  def from_event(event, profile_timezone) do
    %__MODULE__{timezone: event.start_at_tz || profile_timezone || "Etc/UTC"}
    |> put_start_at_date(to_date(event.start_at))
    |> put_start_at_time(to_time(event.start_at))
    |> put_end_at_date(to_date(event.end_at))
    |> put_end_at_time(to_time(event.end_at))
    |> put_end_at(event.end_at)
    |> put_start_at(event.start_at)
    |> put_duration(calc_duration(event.start_at, event.end_at))
  end

  defp ensure_duration(changeset) do
    if Changeset.get_field(changeset, :duration) do
      changeset
    else
      start_at = Changeset.get_field(changeset, :start_at)
      end_at = Changeset.get_field(changeset, :end_at)
      Changeset.put_change(changeset, :duration, calc_duration(start_at, end_at))
    end
  end

  defp assign_computed(socket, changeset) do
    # I need the start_at for the DatePicker component that I render from this
    # component. I render the timezone and duration on the form. Lastly I
    # compute and render some autocomplete suggestions from the values in the
    # changeset.
    socket
    |> assign(:timezone, Changeset.get_field(changeset, :timezone))
    |> assign(:start_at, Changeset.get_field(changeset, :start_at))
    |> assign(:duration, Changeset.get_field(changeset, :duration))
    |> assign(:start_time_autocomplete, start_autocompletes(changeset))
    |> assign(:end_time_autocomplete, end_autocompletes(changeset))
  end

  # I'll leave some of these helper functions out, but they're essentially
  # providing nil-safety, applying defaults, and calculating from other fields
end

Getting the user’s timezone with phx-hook

We can estimate what the user’s timezone is by asking the browser. NOTE I don’t recommend you use this as your only source of user timezone. Use this as an example for how to get JavaScript-sourced input

Let’s get the timezone. We’ll need some JavaScript.

window.userTimezone = Intl.DateTimeFormat().resolvedOptions().timeZone

// initialize the Phoenix LiveView socket, and pass this in as a hook:

hooks.UserTimeZone = {
  mounted() {
    const phoenix = this;
    const target = this.el.dataset.phoenixTarget;
    const els = phoenix.el.querySelectorAll("input")
    for (let el of els) { el.value = window.userTimezone; }
    phoenix.pushEventTo(target, "timezone", window.userTimezone)
  }
}

<%# Timezone in the WhenComponent form %>
<div phx-hook="UserTimeZone" data-phoenix-target="#<%= @id %>" id="user-time-zone">
  <div phx-update="ignore">
    <%= hidden_input f, :timezone %>
  </div>
</div>

When the page is rendered, I’ll get a hidden_input populated with the detected timezone. This will be included in further form changes and params sent to the LiveView process. Remember to wrap it with a phx-update="ignore" so the JavaScript-mutated value isn’t overwritten by LiveView.

You’ll notice that I’m also using pushEventTo after mounting. This is needed because the user may not have interacted with the form yet to trigger a change, so until then, I won’t have user’s timezone! I want it pushed immediately so I can update the form’s changeset. Also, pushEventTo is used instead of pushEvent because this is a LiveComponent, so I want the event pushed to the LiveComponent and not the parent LiveView. I pass the target in via a data attribute so I don’t confuse it with Phoenix’s own phx-target.

When handling the event, we’ll merge the timezone with the existing params of the changeset, and then re-apply the changeset and re-compute fields.

@impl Phoenix.LiveComponent
def handle_event("timezone", detected_timezone, socket) do
  params = Map.put(socket.assigns.when_changeset.params, "timezone", detected_timezone)
  changeset = changeset(socket.assigns.when_changeset, params)
  {:noreply,
    socket
    |> assign(:when_changeset, changeset)
    |> assign_computed(changeset)
  }
end

Handling sub-form change events

Handling form change events doesn’t change with this embedded_schema and component-ized approach. It’s standard Phoenix and Ecto changeset forms, so it’s not very interesting to look at. But remember that you’ll need to use phx-target to send changes to the LiveComponent, otherwise they may bubble up to your parent LiveView.

In my case, I also need to adjust the parameters that come in, so we’ll look at that! I need to check to see what field is changing and apply new parameters based on what is changing.

@impl Phoenix.LiveComponent
def handle_event("validate", %{"when_component" => params}, socket) do
  adjusted_params = adjust_time_params(params, socket.assigns.when_changeset)
  changeset = changeset(socket.assigns.when, adjusted_params)

  {:noreply,
    socket
    |> assign(:when_changeset, changeset)
    |> assign_computed(changeset)
  }
end

defp assign_computed(socket, changeset) do
  # I need the start_at for the DatePicker component that I render from this
  # component. I render the timezone and duration on the form. Lastly I
  # compute and render some autocomplete suggestions from the values in the
  # changeset
  socket
  |> assign(:timezone, Changeset.get_field(changeset, :timezone))
  |> assign(:start_at, Changeset.get_field(changeset, :start_at))
  |> assign(:duration, Changeset.get_field(changeset, :duration))
  |> assign(:start_time_autocomplete, start_autocompletes(changeset))
  |> assign(:end_time_autocomplete, end_autocompletes(changeset))
end

defp adjust_time_params(params, changeset) do
  start_at_time = params_to_time(params["start_at_time"])
  end_at_time = params_to_time(params["end_at_time"])

  cond do
    end_at_time != Changeset.get_field(changeset, :end_at_time) ->
      params_from_new_end_time(start_at_time, end_at_time, params)

    start_at_time != Changeset.get_field(changeset, :start_at_time) ->
      duration = Changeset.get_field(changeset, :duration)
      params_from_new_start_time(start_at_time, duration, params)

    true ->
      params
  end
end

Remember, we’re in a LiveComponent so we want to target the changes to itself and not the parent LiveView. This is accomplished with phx-target on the form.

<%= f = form_for @when_changeset, "#",
  phx_change: :validate,
  phx_target: "##{@id}",
  phx_submit: :save,
  id: @id %>

  <%# ... timezone input mentioned above ... %>

  <%= date_select f, :date %>
  <%= time_input f, :start_time %>
  <%= time_input f, :end_time %>
  Your duration is <%= @duration %>

</form>

Handling the sub-form submission

When the user tries to submit the form, either by hitting “enter” or clicking on the submit button, I need to validate the form once again, and if it’s good tell the parent LiveView that it’s ok to proceed and supply all the helpfully-computed values.

This time we’ll check if the changeset is valid with Ecto.Changeset.apply_action/2. Based on that result, we’ll let the LiveComponent send a message to itself. Actually, a LiveComponent doesn’t run in its own process, instead it’s running inside the parent LiveView’s process. So self() is actually the LiveView and not the LiveComponent. This is how we can send the parent LiveView the result!

You can read more about LiveComponent and sources of truth in the docs.

@impl Phoenix.LiveComponent
def handle_event("save", %{"when_component" => params}, socket) do
  socket.assigns.when
  |> changeset(params)
  |> Changeset.apply_action(:insert)
  |> case do
    {:ok, record} ->
      send(self(), {:proceed, record})
      {:noreply, socket}
    {:error, changeset} ->
      {:noreply,
        socket
        |> assign(:when_changeset, changeset)
        |> assign_computed(changeset)
      }
  end
end

# and caught in the parent LiveView:

@impl Phoenix.LiveView
def handle_info({:proceed, %MyAppWeb.EventLive.WhenComponent{} = form}, socket) do
  {:ok, start_at_wall} = NaiveDateTime.new(form.start_at_date, form.start_at_time)
  {:ok, end_at_wall} = NaiveDateTime.new(form.end_at_date, form.end_at_time)

  params = %{
    start_at_utc: form.start_at,
    start_at_wall: start_at_wall,
    start_at_tz: form.timezone,
    end_at_utc: form.end_at,
    end_at_wall: end_at_wall,
    end_at_tz: form.timezone
  }

  {:noreply,
    socket
    |> assign(:params, Map.merge(socket.assigns.params, params))
    |> assign_step(:next)
  }
end

Handling the overall form submission

You’ll notice that I have a function assign_step above. Let’s go to the parent LiveView and figure out how to change steps, except on the last step we want to persist. We’ll look for the steps in the @steps module attribute, assign it, and that should swap-out the form for the next one!

If there isn’t a next step, then that must mean that we’re finished, so we should try to save.

defmodule MyAppWeb.EventLive.Step do
  @moduledoc "Describe a step in the multi-step form and where it can go."
  defstruct [:name, :prev, :next]
end

defmodule MyAppWeb.EventLive.New do
  use MyAppWeb, :live_view
  alias Ecto.Changeset
  # ...snip...

  defp assign_step(socket, step) do
    if new_step = Enum.find(@steps, & &1.name == Map.get(socket.assigns.progress, step)) do
      assign(socket, :progress, new_step)
    else
      save(socket)
    end
  end

  def save(socket) do
    # remember we've merged all params together, so this should be the complete
    # picture.
    case Schedule.create_event(socket.assigns.params) do
      {:ok, event} ->
          socket
          |> put_flash(:info, t("Event Created"))
          |> push_redirect(to: Routes.live_path(socket, EventLive.Show, event.id))

      {:error, %Changeset{} = changeset} ->
         socket
         |> assign(:changeset, changeset)
         |> put_flash(:error, "There is an issue with what you filled in")
    end
  end

  @impl Phoenix.LiveView
  def handle_event("prev-step", _content, socket) do
    {:noreply, assign_step(socket, :prev)}
  end
end

<%= MyAppWeb.Components.secondary_button(t("Back"), phx_click: "prev-step") %>
<%= MyAppWeb.Components.primary_button(t("Next"), phx_disable_with: "...", submit: true) %>

Back buttons are easy too. Add phx-click="prev-step" and handle the event in the same way, except using :prev. Make sure there’s no back button on the first step! (otherwise you’ll mistakenly try to save).

Conclusion

I hope this helps you out in your endeavors to tackle long and complicated forms. Tweet me @bernheisel if you have suggestions or enjoyed this post!

Update

You might want to persist a draft record in-between steps. This is a great idea! If you do this, then you can leverage LiveView’s handle_params to navigate to the appropriate step depending on the draft’s progress.

Also, some of my code examples aren’t very good for managing existing resources. Keep that in mind when developing your own multi-step form. This was written for the context of creating a new event, and not editing an existing event. Subscribe to my RSS feed to check for a new post that revisits this problem.

https://topenddevs.com/podcasts/elixir-mix/episodes/emx-091-managing-change-with-ecto-with-david-bernheisel

Coming from ActiveRecord, Ecto and Changesets were a wonderful alternative! They cover my other blog post where I shared some tips and tricks for working with Changesets. We also cover Ecto.Multi, how to compose Changesets, using “embedded” schemas, and much more!

I made it open-source as well.

I’ve been trying to get in the habit of taking more notes and writing notes down before I forget about them. I also use the Chrome browser. The way I look for information in Chrome is by opening a new tab and start typing away for the question. What if the new tab had some information on it that I could stow away?

There’s plenty of existing extensions out there and I tried about 5 or 6 of them but they didn’t suite my taste or needs. If it didn’t sync between Chrome devices, then my notes would be scattered across machines – no go. Often times, the markdown syntax it provided was vanilla Markdown, but I’m used to newer features and extensions offered by GitHub Flavored Markdown, such as task lists, tables, and autolinking. If it didn’t support GFM – no go.

I’m working on a project using Elixir, Vue, and Tailwind CSS, and I wanted to practice using those frameworks so I can understand them more. It’s all about the repetition for learning.

Since I know nothing about Chrome Extension development, I found an existing extension that was pretty close to what I wanted. I forked it and started customizing it for my own wants. Fiddling around with the code made me understand more of what makes it Chrome-specific, which isn’t too much, and the rest was just plain-old-javascript. This led me down the path of using Vue and TailwindCSS, and packaging it with Webpack– just like I do with my Elixir project.

• Chrome needs a manifest.json to specify the icon, description, and permissions it needs to operate within Chrome.
• In the manifest, you specify the entry point HTML. In my case, I needed the new tab page.
• There are some Chrome APIs you can use. The only one that I care about is chrome.storage.sync. This handles storing small data related to your extension and handles offline and online capabilities. If you’re offline, then it’s ok and will store the data in local storage until you’re online again, which then it will sync through the net and update your other devices.

Maybe one day I can make it more like vimwiki which offers multi-page linking, but for now it’s fine.

Since asdf removed their .tool-versions file I needed to adjust the steps below. They renamed .tool-versions to .release-tool-versions, so we need to account for that in our vimscript.

Also, if you’re running Linux, then you might need to ensure you have GCC version 9 installed. The Erlang release that ElixirLS is using needs 9, and I found that my system had 10 installed.

Kassio’s Post was inspirational, and I adapted from his setup. My setup is a little different from his:

• I use asdf to manage my installed Elixir and Erlang versions. The ElixirLS project has a tested Elixir version it was developed with; and I’d like to use that same version. I also don’t want to have to worry about not having the same installed versions as them.
• I use vim-plug. It has a neat feature where you can clone any repository whether or not it’s built for vim or not. In this case, I’m going to use it to grab a copy of ElixirLS, and have it run a post-update hook. ElixirLS doesn’t have any vim code that gets loaded, so it’s benign.
• I wanted to let the compilation happen asynchronously. I don’t want compilation to lock up the UI.

The Proof

Here’s ElixirLS in action inside vim with coc.nvim:

Here’s me manually calling to update ElixirLS. I have a terminal on the right that is watching the filesystem so we can see it’s actually doing something:

Here’s me using vim-plug to update ElixirLS. I have a terminal on the right that is watching the filesystem so we can see it’s actually doing something:

The Vimscript

Here’s the vim setup:

call plug#begin('~/.config/nvim/plugged')
  Plug 'elixir-lsp/elixir-ls', { 'do': { -> g:ElixirLS.compile() } }
  Plug 'neoclide/coc.nvim', {'branch': 'release'}
call plug#end()

let g:coc_global_extensions = ['coc-elixir', 'coc-diagnostic']

let g:ElixirLS = {}
let ElixirLS.path = stdpath('config').'/plugged/elixir-ls'
let ElixirLS.lsp = ElixirLS.path.'/release/language_server.sh'
let ElixirLS.cmd = join([
        \ 'cp .release-tool-versions .tool-versions &&',
        \ 'asdf install &&',
        \ 'mix do',
        \ 'local.hex --force --if-missing,',
        \ 'local.rebar --force,',
        \ 'deps.get,',
        \ 'compile,',
        \ 'elixir_ls.release &&',
        \ 'rm .tool-versions'
        \ ], ' ')

function ElixirLS.on_stdout(_job_id, data, _event)
  let self.output[-1] .= a:data[0]
  call extend(self.output, a:data[1:])
endfunction

let ElixirLS.on_stderr = function(ElixirLS.on_stdout)

function ElixirLS.on_exit(_job_id, exitcode, _event)
  if a:exitcode[0] == 0
    echom '>>> ElixirLS compiled'
  else
    echoerr join(self.output, ' ')
    echoerr '>>> ElixirLS compilation failed'
  endif
endfunction

function ElixirLS.compile()
  let me = copy(g:ElixirLS)
  let me.output = ['']
  echom '>>> compiling ElixirLS'
  let me.id = jobstart('cd ' . me.path . ' && git pull && ' . me.cmd, me)
endfunction

" If you want to wait on the compilation only when running :PlugUpdate
" then have the post-update hook use this function instead:

" function ElixirLS.compile_sync()
"   echom '>>> compiling ElixirLS'
"   silent call system(g:ElixirLS.cmd)
"   echom '>>> ElixirLS compiled'
" endfunction


" Then, update the Elixir language server
call coc#config('elixir', {
  \ 'command': g:ElixirLS.lsp,
  \ 'filetypes': ['elixir', 'eelixir']
  \})
call coc#config('elixir.pathToElixirLS', g:ElixirLS.lsp)

And this is in my :CocConfig (~/.config/nvim/coc-settings.json):

{
  "codeLens.enable": true,
  "diagnostic-languageserver.filetypes": {
    "elixir": ["mix_credo", "mix_credo_compile"],
    "eelixir": ["mix_credo", "mix_credo_compile"]
  }
}

Include coc.nvim, ElixirLS in plug

Starting from the top:

call plug#begin('~/.config/nvim/plugged')
  Plug 'elixir-lsp/elixir-ls', { 'do': { -> g:ElixirLS.compile() } }
  Plug 'neoclide/coc.nvim', {'branch': 'release'}
call plug#end()

let g:coc_global_extensions = ['coc-elixir', 'coc-diagnostic']

I’m using vim-plug to grab some plugins. All we care about for now is coc.nvim and elixir-ls. I’m providing options for elixir-ls to perform an after-update action. In this case, a lambda which immediately evaluates { 'do': { -> g:ElixirLS.compile() } }

To learn more about the vim lambda, check out :h expr-lambda. We’re going to look at the ElixirLS.compile() function later.

The coc.nvim setup is straight from their readme. I’m also adding some extensions that coc.nvim will install on its own after startup. In this case I want coc-elixir and coc-diagnostic.

coc-elixir provides coc.nvim the settings to know how to work with Elixir projects and the language server. It also will build ElixirLS on its own, but we’re going to circumvent that in a moment.

coc-diagnostic is a generic bridge for many non-Language-Server tools like shellcheck and credo. In this case, I’m adding it for credo. I don’t need coc-diagnostic to provide a formatter, since the main Elixir language server will provide that already.

Define your vim ElixirLS dictionary

Next we’re going to create a dictionary with a couple of functions. This dict is going to manage several things for us:

• The path to the language server executable and directory.
• The commands to run when needing to compile
• Job hook functions so Neovim can run this task asynchronously

Check out :h dictionary-function in vim for more info on how to be a bit more object-oriented in your vim scripts. If you go down this rabbit hole, I really encourage you to look into :h lua as well which is better-suited for serious vim programming.

let g:ElixirLS = {}
let ElixirLS.path = stdpath('config').'/plugged/elixir-ls'
let ElixirLS.lsp = ElixirLS.path.'/release/language_server.sh'
let ElixirLS.cmd = join([
        \ 'cp .release-tool-versions .tool-versions &&',
        \ 'asdf install &&',
        \ 'mix do',
        \ 'local.hex --force --if-missing,',
        \ 'local.rebar --force,',
        \ 'deps.get,',
        \ 'compile,',
        \ 'elixir_ls.release &&',
        \ 'rm .tool-versions'
        \ ], ' ')

So far it’s pretty standard stuff. We initialize an empty global dictionary first, then start stuffing some values in there. We’re using the function stdpath so we avoid hard-coding any paths.

The join([...], ' ') is only a way to organize the commands in a visual way. It’s not necessary; you can totally just concat some strings together. The end result of this join is:

$ cp .release-tool-versions .tool-versions && \
    asdf install && \
    mix do local.hex --force --if-missing, local.rebar --force, deps.get, compile, elixir_ls.release && \
    rm .tool-versions

Since I’m using asdf and so are the ElixirLS developers I want to make sure I’m using the ElixirLS developers’ tools so I know for sure I won’t run into trouble while developing; I want my ElixirLS to be stable since it’s such an important tool for me.

We’re going to leverage mix do so we’re not starting Elixir fresh for each command. This should speed some things up.

Run it in the background

function ElixirLS.on_stdout(_job_id, data, _event)
  let self.output[-1] .= a:data[0]
  call extend(self.output, a:data[1:])
endfunction

let ElixirLS.on_stderr = function(ElixirLS.on_stdout)

function ElixirLS.on_exit(_job_id, exitcode, _event)
  if a:exitcode[0] == 0
    echom '>>> ElixirLS compiled'
  else
    echoerr join(self.output, ' ')
    echoerr '>>> ElixirLS compilation failed'
  endif
endfunction

function ElixirLS.compile()
  let me = copy(g:ElixirLS)
  let me.output = ['']
  echom '>>> compiling ElixirLS'
  let me.id = jobstart('cd ' . me.path . ' && git pull && ' . me.cmd, me)
endfunction

These functions are adding keys to the ElixirLS dictionary. If I echo out the dictionary, you’ll see a normal dictionary with some funcrefs.

:echo ElixirLS
{
  'cmd': 'asdf install && mix do local.hex --force --if-missing, local.rebar --force, deps.get, compile, elixir_ls.release',
  'path': '/home/me/.config/nvim/plugged/elixir-ls',
  'on_exit': function('2'),
  'on_stdout': function('1'),
  'lsp': '/home/me/.config/nvim/plugged/elixir-ls/release/language_server.sh',
  'on_stderr': function('1', {...@0}),
  'compile': function('3')
}

One of the great things about Neovim (and Vim8+) is that it really pushed asynchronous work forward. Neovim introduced some functions to manage background jobs. The one we end up using is jobstart({cmd}[, {opts}]) (check out :h jobstart). Heads up this is for Neovim; Vim8 has a different API for asynchronous work. It’s still jobstart but the options are different, so be sure to check out :h job-options.

function ElixirLS.compile()
  let me = copy(g:ElixirLS)
  let me.output = ['']
  echom '>>> compiling ElixirLS'
  let me.id = jobstart('cd ' . me.path . ' && git pull && ' . me.cmd, me)
endfunction

First we’re going to make a copy of the dictionary since this can be asynchronous; we’ll call it me. Then we’ll initialize a new key output so we can store all the background job’s output into it. Lastly, we’ll start the job. The first argument (if a string) will shell out and execute the command you fed it.

Here’s the complete command that ends up being sent:

$ cd {the-path} && \
    git pull && \
    cp .release-tool-versions .tool-versions && \
    asdf install && \
    mix do local.hex --force --if-missing, local.rebar --force, deps.get, compile, elixir_ls.release && \
    rm .tool-versions

If you’re only using this via vim-plug, then vim-plug will take care of the cd {the-path} && git pull on its own, so we don’t need to include that. Totally skip it and only include me.cmd. In my case, I wanted to be able to run :call ElixirLS.compile() myself as well which will need to perform those tasks. It doesn’t hurt to keep those commands but they’re redundant.

The last argument me is a dictionary that contains the keys that point to functions that will accept a certain signature; the three functions it cares about are:

• on_stdout(job_id, data, event)
• on_stderr(job_id, data, event)
• on_exit(job_id, exitcode, _event)

The values that are passed into these functions are a bit odd, but remember it’s focused on a stream of data, and not all the data at once. This means that the data you get will be an array of values from the background job’s output (either stdout or stderr).

Let’s look at one of the functions that receives the hook:

function ElixirLS.on_stdout(_job_id, data, _event)
  let self.output[-1] .= a:data[0]
  call extend(self.output, a:data[1:])
endfunction

self refers to the copy of the ElixirLS dictionary that started this job. (check out :h self). Before we started the job, we initialized the dictionary to have an output key that had a list with one empty string ['']. We’re going to use this list and append all the incoming output into it. At the very end, self.output have something like vim•['hey', 'hi\nthere', "I'm d", 'one now']. Since the data isn’t necessarily split at newlines, we’re going to combine the last stored output’s with the first incoming element, and then add the rest of the incoming data to the stored output.

vim•let self.output[-1] .= a:data[0]. Take the last stored element and concat the first incoming data’s element, and then assign it back to vim•self.output[-1]. Then add the two lists together. extend() will mutate the first element.

Since we want to treat stderr and stdout as the same kind of output, we’re going to have the on_stderr callback forward the call to the on_stdout function. This avoids duplicating the code.

Finally, let’s look at the on_exit callback:

function ElixirLS.on_exit(_job_id, exitcode, _event)
  if a:exitcode[0] == 0
    echom '>>> ElixirLS compiled'
  else
    echoerr join(self.output, ' ')
    echoerr '>>> ElixirLS compilation failed'
  endif
endfunction

The exitcode is passed into this function, but it’s still the funky buffer-like array but it should always just be the one element with the exit code. If it’s 0, then it exited ok without error so let’s echo a message indicating we’re done.

Otherwise, let’s echo out the entire output as an error that I can find with :messages and investigate what went wrong.

All this means now is that we can run :PlugUpdate and ElixirLS will now update itself, ensuring it’s running on the best version of Elixir for itself, everything’s updated, downloaded, and recompiled without issue. I can also run ElixirLS.compile() at any time if I suspect I need to update ElixirLS.

With coc.nvim I can also check :CocInfo to see if the language servers are running ok.

Use the fruits of the labor

call coc#config('elixir', {
  \ 'command': g:ElixirLS.lsp,
  \ 'filetypes': ['elixir', 'eelixir']
  \})
call coc#config('elixir.pathToElixirLS', g:ElixirLS.lsp)

Almost done!!

We have a somewhat dynamic path for the newly-compiled ElixirLS. On my Mac, the path could be /Users/me/.config/..., but on my Linux computer it would be /home/me/.config/.... CocConfig is a JSON file that can’t evaluate any environment variables, so I need to resort to calling it from within vim. This really works out though.

The first coc#config is telling coc.nvim in general that there is an available language server for the elixir and eexlixir filetypes. Lastly, we’re going to tell coc-elixir to use our own compiled ElixirLS so it doesn’t need to go off on its own and try to manage the installation and compilation of ElixirLS.

Have any vim and Elixir tips of your own? TWEEEEEEEET at me @bernheisel

• Extract Boilerplate
• DB-generated UUIDs
• Interpolate Your Docs
• Compose Changesets

Extract Boilerplate

I use UUIDs for all my IDs, specifically v4.

Long ago I managed some legacy codebases that chose (defaulted) integer-based IDs, and two times now I’ve had to migrate them to tables with a BigSerial as the ID because we reached our limit of IDs for the table. It’s easy to forget.

I’d be fine with BigSerial, but since I started using Ecto, I found myself using UUIDs instead, and now it’s become a habit.

What about showing UUIDs in URLs in basic CRUD endpoints?

Yea, it’s really ugly to show UUIDs in the browser URL bar, especially for nested routes. But, should you show internal database IDs to users? I don’t think so, so when using UUIDs I am constantly reminded to design towards generating unique slugs for UX. If you see a UUID in the browser URL, I feel like I should replace it with a human-readable unique slug instead. IDs are for machines, slugs are for humans. URLs are for humans too, though I understand some may not agree with that.

Let’s default to UUID in Ecto

There’s some configuration for Ecto to default to UUIDs:

1. Configure Generators
2. Configure Migrations
3. Configure Schema

Then lastly, we’ll wrap that up and make it easier by pulling it into a macro

Configure Generators

If you’re using Phoenix and use its generators, you might care about this section. When you run mix phx.gen.schema ARGS, Phoenix and Ecto will throw in some boilerplate into your schemas and migrations. Even if you don’t, it’s harmless to configure it just in case things change later, or other developers on your project choose to use the generators.

  #./config/config.exs
  config :my_app,
    ecto_repos: [MyApp.Repo],
    generators: [binary_id: true]

  # This will tell your Schema that the primary_key is a binary UUID:

  #./lib/my_app/random_schema.ex
  @primary_key {:id, :binary_id, autogenerate: true}
  @foreign_key_type :binary_id
  schema "my_table" do
    # ...
  end

Configure Migrations

When creating tables through a migration, you’ll need to specify that it should not create an ID column that generates it’s own IDs (more on this later). Instead, we’ll supply our own primary key column called id.

defmodule MyApp.Repo.Migrations.CreateUsers do
  @moduledoc "Creating Users in the database"
  use Ecto.Migration

  def change do
    create table(:users, primary_key: false) do
      add :id, :binary_id, primary_key: true

      timestamps()
    end
  end
end

In Ecto 3.x, you can also reconfigure default ID column settings with :migration_primary_key so you wouldn’t even have to add :id yourself, but unfortunately it does not play well with Ecto macros that I will end up using since it’s via Mix config.

Configure Schema

If you’re using the generators above, then they should insert these options in for you, but if not, you’ll need to make sure they’re present.

defmodule MyApp.RandomSchema do
  use Ecto.Schema
  import Ecto.Changeset

  @primary_key {:id, :binary_id, autogenerate: true}
  @foreign_key_type :binary_id

  schema "users" do
    field :name, :string

    timestamps()
  end
end

That’s it! It should work. Ecto’s schema macro will use the module attributes to configure how it should treat the primary key. The documentation has more information if you want to read more.

Pull Into Macro

That’s a lot of boilerplate for each schema. Let’s make it easier. Notice at the top of the schema definition? use Ecto.Schema.

This injects some code into the module. We can do that too! Let’s create our own schema file that injects our boilerplate.

defmodule MyApp.Schema do
  @moduledoc "Ecto Schema Helpers"

  defmacro __using__(_) do
    quote do
      use Ecto.Schema
      import Ecto.Changeset

      @primary_key {:id, :binary_id, autogenerate: true}
      @foreign_key_type :binary_id
    end
  end
end

defmodule MyApp.RandomSchema do
-   use Ecto.Schema
-   import Ecto.Changeset
-
-   @primary_key {:id, :binary_id, autogenerate: true}
-   @foreign_key_type :binary_id
+   use MyApp.Schema

  schema "users" do
    field :name, :string

    timestamps()
  end
end

Now for each of your schemas, use this new module and you don’t have to remember the boilerplate anymore.

DB-Generated UUIDs

Maybe you noticed above that we have autogenerate: true above. This is telling Ecto to generate those ID UUIDs instead of the database. That bothered me; I feel like that’s a database responsibility, not an app responsibility.

Let’s move that into the database. I’m using PostgreSQL, but I’m sure there are similar tools for other databases.

1. Enable pgcrypto
2. Update our Boilerplate

Enable pgcrypto

Postgres unfortunately cannot generate UUIDs simply out of the box, but it does ship with functions that you can enable. You can get away with creating a function that’ll generate UUIDs, but I prefer some battle-tested code that ships with postgres contrib.

One of those extensions is pgcrypto which supplies a function gen_random_uuid().

Let’s create a migration to have Postgres enable the extension.

$ mix ecto.gen.migration add_pgcrypto

defmodule MyApp.Repo.Migrations.AddPgcrypto do
  @moduledoc "Add PgCrypto so we can have Postgres generate it's own IDs"
  use Ecto.Migration

  def change do
    execute(
      "CREATE EXTENSION IF NOT EXISTS \"pgcrypto\"",
      "DROP EXTENSION IF EXISTS \"pgcrypto\""
    )
  end
end

This sometimes requires special permissions on the database user. If this doesn’t work for you, then you might need an additional procedure to enable the extension for you. However, this should work for your local database for development.

Use pgcrypto

Now that the database can generate UUIDs, let’s use it our ID columns!

First we’ll tell ID column to default its value. Looking at our earlier migration, let’s modify it.

defmodule MyApp.Repo.Migrations.CreateUsers do
  @moduledoc "Creating Users in the database"
  use Ecto.Migration

  def change do
    create table(:users, primary_key: false) do
-      add :id, :binary_id, primary_key: true
+      add :id, :binary_id, primary_key: true, default: fragment("gen_random_uuid()")

      timestamps()
    end
  end
end

Now it’ll generate it’s own ID. If you stopped here, you’ll notice that when you insert records with Ecto, none of your returned structs will have an ID! What happened?!

We need to tell Ecto to read the ID back into the struct after writing. Thankfully, that’s easy. Earlier we pulled some boilerplate into a MyApp.Schema; let’s modify it:

defmodule MyApp.Schema do
  @moduledoc "Ecto Schema Helpers"

  defmacro __using__(_) do
    quote do
      use Ecto.Schema
      import Ecto.Changeset

-      @primary_key {:id, :binary_id, autogenerate: true}
+      @primary_key {:id, :binary_id, read_after_writes: true}
      @foreign_key_type :binary_id
    end
  end
end

Now Ecto will pull the database-generated UUID back. Yes! Now we have IDs generated in the database again.

Avoiding Collisions

Ok, so it’s works, but what happens in the low low chance that it generated a duplicate UUID? It will fail to insert since the primary key is unique. In this case, you’ll need to handle retrying with application code to retry once. The probability of it failing a second time is galaxy-scale low. Probably not worth handling in new code for smallish tables, and worth revisiting if you have very large-scale tables in the millions or billions.

Have Elixir Write Docs For You

One tedious task I find myself doing is trying to remember what fields I need, don’t need, and which ones have a default value if not supplied when interacting with a changeset function. I’ll constantly flip back and forth between my forms, contexts, and schemas.

In VIM, I’m using coc.nvim to enable Language Server integration. You can do this easily in VSCode as well. One of the great features of this is that you can lookup the documentation on a function with a keypress or hover.

See the screenshot and how it’s listing out the required, optional, and default fields? Let’s make that happen.

defmodule MyApp.RandomSchema do
  @moduledoc "The Random Schema."
  use MyApp.Schema

  @defaults %{
    type: "topical",
    is_invite_only: false,
    is_screening: false
  }

  schema "organizations" do
    field :name, :string
    field :type, :string, default: @defaults[:type]

    field :audience_description, :string
    field :is_invite_only, :boolean, default: @defaults[:is_invite_only]
    field :is_screening, :boolean, default: @defaults[:is_screening]

    timestamps()
  end

  @optional_fields ~w[is_invite_only is_screening screen_questions]a
  @required_fields ~w[name type audience_description]a

  @doc """
  Required fields: #{inspect @required_fields}
  Optional fields: #{inspect @optional_fields}
  Defaults: #{inspect @defaults}
  """
  @spec changeset(%__MODULE__{} | Ecto.Changeset.t(), map()) ::
    Ecto.Changeset.t()
  def changeset(struct \\ %__MODULE__{}, attrs) do
    struct
    |> cast(attrs, @optional_fields ++ @required_fields)
    |> validate_required(@required_fields)
  end
end

It’s not hard at all! It’s a feature I often forget exists with ExDocs and Elixir: you can interpolate compiled values into documentation. In this case, I’m defining my @optional_fields, @required_fields, and @defaults, and then interpolating them into the changeset docs.

Easy peasy!

Compose Changesets

Did you know that changesets can chain together?

For example:

Let’s say you have a “main” changeset that performs the basic validations. These validations should occur every single time an insert or update occurs for this schema. This happens for both admins and users.

Let’s also say that you have another changeset that runs additional validations if the User is updating the record versus the Admin updating the same record. You don’t have to duplicate that code!

# In MySchema
@spec changeset(%__MODULE__{} | Ecto.Changeset.t(), map()) ::
  Ecto.Changeset.t()
def changeset(struct_or_changeset \\ %__MODULE__{}, attrs) do
  struct_or_changeset
  |> cast(attrs, @optional_fields ++ @required_fields)
  |> validate_required(@required_fields)
end

@spec additional_restrictions_changeset(%__MODULE__{} | Ecto.Changeset.t(), map()) ::
  Ecto.Changeset.t()
def additional_restrictions_changeset(struct_or_changeset \\ %__MODULE__{}, attrs) do
  struct_or_changeset
  |> changeset(attrs)
  |> more_validations()
end

@spec user_changeset(%__MODULE__{} | Ecto.Changeset.t(), map()) ::
  Ecto.Changeset.t()
def user_changeset(changeset, attrs) do
  changeset
  |> cast(attrs, [:some_more_fields])
  |> even_more_validations()
end

## ... In some other app code

def create(params) do
  MySchema.changeset(%MySchema{}, params)
end

def restricted_create(params) do
  MySchema.additional_restrictions_changeset(%MySchema{}, params)
end

def user_create(params) do
  %MySchema{}
  |> MySchema.changeset(params)
  |> MySchema.user_changeset(params)
end

This way, you can use additional_restrictions_changeset/2 by itself and get all the same logic within changeset/2. Or alternatively, compose them together from the outside like in user_create/1

A common mistake that prevents changesets from being composable is that we’ll write our function signatures to require the struct as the first param:

# don't do this:
@spec changeset(%__MODULE__{}, map()) :: Ecto.Changeset.t()
def changeset(%__MODULE__{} = struct, attrs) do
  struct
  |> cast(attrs, @optional_fields ++ @required_fields)
  |> validate_required(@required_fields)
end

This makes it more restrictive and keeps it from being composable.

https://topenddevs.com/podcasts/elixir-mix/episodes/emx-068-contributing-to-the-elixir-community-with-david-bernheisel-cory-schmitt

Cory and I talk about our adventures in the Elixir community, Meetup.com, and how we’ve contributed. The big contribution in here is the DateTimeParser that we built.

If you develop in Ruby or Elixir or write Markdown, you might find this helpful!

Let’s start with some basics:

Environment

• neovim - right now I’m using 0.4.0 so I can use floating windows.
• kitty Terminal Emulator
• built-in terminal in neovim
• ArchLinux - My distro of choice. Shouldn’t matter for this article.
• i3 - My window manager. Shouldn’t matter for this article.
• Plug - for managing neovim plugins.
• coc.nvim for languageserver integration.
• dotfiles - My dotfiles if you want the complete picture

There are three ways that I can split my workspace: (1) via my window manager i3, (2) via my terminal emulator kitty, (3) via neovim with splits/buffers. Generally I adhere to this practice:

1. Split with i3 if it’s an application, especially a GUI app. This gives me the ability to move the window to another desktop if I want.
2. Split with built-in neovim terminal for tests.
3. Don’t split with kitty ever. It’d be too confusing for me to have 3 sets of keyboard shortcuts to keep track of for switching windows/panes/splits. Maybe one day I’ll replace the built-in neovim terminal with a kitty split or an i3 split.

Testing

I use vim-test and it’s pretty incredible. I picked this workflow up while working at thoughtbot from some good friends and the thoughtbot dotfiles, and it changed the way I code. The whole TDD workflow is great despite that I still have trouble actually writing tests first - I tend to spike, iterate, iterate, THEN write tests, then open a PR. Yea– this probably means I’m not a 10x developer 😛

I also use neoterm to help open up terminal splits. When I’m at home, I have an ultrawide that I use so splitting windows vertically is the way to go; but when I’m mobile with my laptop then I typically split horizontally. I want tests to be visible either way, so I need this to be flexible.

The vim-test neoterm strategy defaults to sending tests to the last-used neoterm buffer; I can have more terminal buffers, but the first one I open is what vim-test will use going forward.

Here’s how I configure vim-test with neoterm. (my dotfiles for reference)

" ~/.config/nvim/init.vim

" Test
let g:test#strategy = "neoterm"
let g:neoterm_shell = '$SHELL -l' " use the login shell
let g:neoterm_default_mod = 'vert'
let g:neoterm_autoscroll = 1      " autoscroll to the bottom when entering insert mode
let g:neoterm_size = 80
let g:neoterm_fixedsize = 1       " fixed size. The autosizing was wonky for me
let g:neoterm_keep_term_open = 0  " when buffer closes, exit the terminal too.
let test#ruby#rspec#options = { 'suite': '--profile 5' }

" Create some commands that makes the splits easy

function! OpenTermV(...)
  let g:neoterm_size = 80
  let l:cmd = a:1 == '' ? 'pwd' : a:1
  execute 'vert T '.l:cmd
endfunction

function! OpenTermH(...)
  let g:neoterm_size = 10
  let l:cmd = a:1 == '' ? 'pwd' : a:1
  execute 'belowright T '.l:cmd
endfunction

command! -nargs=? VT call OpenTermV(<q-args>)
command! -nargs=? HT call OpenTermH(<q-args>)

" Use the project's test suite script if it exists

function! RunTestSuite()
  Tclear
  if filereadable('bin/test_suite')
    T echo 'bin/test_suite'
    T bin/test_suite
  elseif filereadable("bin/test")
    T echo 'bin/test'
    T bin/test
  else
    TestSuite
  endif
endfunction

nmap <silent> <leader>t :call TestNearest<CR>
nmap <silent> <leader>T :call TestFile<CR>
nmap <silent> <leader>a :call RunTestSuite()<CR>
nmap <silent> <leader>l :call TestLast<CR>

I’ve found it conventional to have a bin/test_suite or bin/test script in the project that takes care of a lot of details like environment exports, cleanup, or making sure the test environment’s database is setup as well as running all the tests. Even if the test suite isn’t complicated, it’s still helpful for new developers on the project.

If that script is present and when I want to run all tests, I should execute that file; otherwise use the default vim-test suite command. For non-suite tests, I use the default vim-test commands.

<space>a triggers all tests. If a neoterm split isn’t open, then it’ll automatically open one with the default settings– in my case, a vertical split at 80 columns wide. If a neoterm split is already open, then it’ll send the test to that split. In situations where my vertical space is lacking, I prep by opening up a split, and then hit my test shortcut. :HT to open the terminal up.

If I’m testing a method or function, then <space>t to send the nearest test to it. If I’m trying to make a test pass, I’ll modify the code and then <space>l to run the last test. If I’m refactoring a class or module, I’ll run all the Tests for it. I haven’t found myself using vim-test’s TestVisit. If you have some examples on where that command helps, I’d love to hear it!

Transformations

This is a great start! But eventually there might be a pesky app where I need to opt-into an environment variable, but only when I’m running a small number of tests. vim-test lets me define my own transformations to the commands. I can check for a certain file and string in it to determine what project I’m in. If I’m in that project, then change the command where I can.

" ~/.config/nvim/after/ftplugin/ruby.vim

function! MyAppRspec(cmd) abort
  " If I'm in the pesky app and
  " not running the entire test suite indicated by the --profile flag
  " Add the SKIP_FIXTURES env var.
  call system("cat README.md | grep 'MyApp'")
  if match(a:cmd, '--profile') == -1 && v:shell_error == 0
    return substitute(a:cmd, 'bundle exec', 'SKIP_FIXTURES=true bundle exec', '')
  else
    return a:cmd
  endif
endfunction

let g:test#custom_transformations = {
      \ 'myapp_ruby': function('MyAppRspec')
      \ }
let g:test#transformation = 'myapp_ruby'

On the Elixir side, umbrella apps can be a little tricky. vim-test will send the path of the test to mix test {file}, but mix will run that command for each of the apps in the umbrella. That’s probably not what we want to do since that test exists for only one for apps. Again, we can solve it with a transformation.

" ~/.config/nvim/after/ftplugin/elixir.vim

function! ElixirUmbrellaTransform(cmd) abort
  " if in an umbrella project indicated by the existence of an ./apps folder
  " limit the mix command to the app to which the test belongs
  if match(a:cmd, 'apps/') != -1
    " capture the app from the file path, and send it to the --app flag instead
    return substitute(a:cmd, 'mix test apps/\([^/]*\)/', 'mix cmd --app \1 mix test --color ', '')
  else
    return a:cmd
  end
endfunction

let g:test#custom_transformations = {
       \ 'elixir_umbrella': function('ElixirUmbrellaTransform')
       \ }
let g:test#transformation = 'elixir_umbrella'

That’s it for tests!

Hope you picked up something nifty. If you have any tips for me, send them my way @bernheisel

When I came across a JSON API service that was returning binary, I was a bit puzzled; “what is this binary? I’m supposed to be getting text back…” and, it’s not consistent either: sometimes I get text on the same exact request a minute later. Baffling.

On top of that, I was using ExVCR in some tests which serializes the request/response chain into JSON. ExVCR takes binary responses, encodes it into Erlang Term format, then Base64 encodes that, and then puts that in the JSON file it writes; and does all that in reverse when it’s replaying the “cassette” when the tests run.

I’ve seen text before on this endpoint, and now I’m getting binary sometimes; and wait-a-second, don’t HTTP clients decompress responses? I’m pretty sure I didn’t worry about this with Ruby’s Net::HTTP or HTTParty.

Turns out, in the Elixir ecosystem, HTTPoison along with other common HTTP clients like HTTPotion, the new Mint, Gun, and probably others don’t do this automatically.

Let’s back up

HTTP requests and responses have some headers that tell the client/server what format of content we’re looking for. The ones we care about here is Accept-Encoding and Content-Encoding. There’s another one that’s related called Content-Type, but that’s not exactly about compression, but we’ll get to this one later.

Accept-Encoding is what the client will use to say “YO SERVER! I need some of this resource, and I can handle it compressed with brotli“

Content-Encoding is what the server will respond with, as in “Oh hay Client! Nice to see you; here’s your content as requested. I even compressed it in brotli”

What REALLY happens (in my experience), is that Accept-Encoding is ignored, and the server’s gonna give whatever it wants to you. To complicate it more, there are layers between the client and server that may compress data and modify headers (or not). So, the server might have sent plaintext and provided a Content-Encoding of identity or not a Content-Encoding at all (both of these mean there is no compression.), but a load balancer, router, CDN, whatever, might have compressed the body of data on the way back from the server to the client.

So what’s the client to do? It has to guess. This is probably why some clients don’t automatically decompress data for you.

Here are some of the options for Content-Encoding:

As an interesting sidenote, Phoenix supports compression into Brotli, but otherwise there’s not yet built-in support for decompressing Brotli in Erlang/Elixir. There’s also no built-in support for LZW, but that’s ok because it’s not as good or popular as the other formats. The only built-ins for Erlang and Elixir are gzip and deflate so that’s what I’ll support on this first iteration.

HTTPoison

In Elixir, the most popular HTTP client is HTTPoison according to hex.pm. Actually, let me clarify: HTTPoison itself doesn’t do any HTTP requests itself; what I mean is it’s a wrapper for the Erlang HTTP client called hackney which actually does the HTTP requests, and HTTPoison wraps around that to make the API a bit friendlier for Elixir.

Let me re-word that for my use-case: I’m making a wrapper for a wrapper.

I’m not the first to notice that it doesn’t decompress responses automatically. There’s been an issue open since 2015 for them to auto-decompress, but the author has decided, (me paraphrasing), “I’m not going to do it, but hackney is so we’ll get to benefit from it soon enough!”, and that issue has been open for a since Jan 2015. We’re still waiting. It’s June 2019. 4.5 years.

Ok, cool, but I need to handle this now, and it doesn’t seem like there’s movement in the popular library of choice.

It’s a little unfair

It’s unfair for me to suggest these libraries should absolutely support decompression out of the box, because these clients are really powerful. They also support streaming which complicates decompression. But, for simple JSON request/responses and for most APIs, we’re not streaming.

Major props for these libraries making my life easier; my issue is that I didn’t know about these concepts before diving in, and through an issue I learned about HTTP decompression.

One more problem: Character Encoding

I am also working with an API service that responds with characters encoded in ISO-8859-1 sometimes; not in UTF-8. In Elixir, strings are UTF-8 so I need to make sure I can convert those characters to something readable for my logs, and ultimately the clients. This character encoding is indicated in the HTTP header Content-Type, paired with the format of content, like JSON or XML. It’s going to look something like text/plain;charset=utf-8 or application/json;charset=ISO-8859-1.

Let’s do it

Let’s stick with HTTPoison out of pure laziness. If you’re implementing from scratch, I’d recommend you to look at Mint first because it has no dependencies and has a better philosophy with OTP, which is a good thing. Tesla is also a good HTTP client to consider.

Let me re-word that again: I’m going to write a wrapper (MyApp.HTTPClient) for a wrapper (HTTPoison) of hackney for a wrapper (my layer that covers the 3rd party API) of a 3rd party service. Exciting.

Also please know that my project also includes Phoenix and Plug, so you might see some helpers in the tests and implementation. If you’re not using Phoenix or Plug, it should be pretty easy to replace these functions with your own.

My interface

It’s going to be exactly like HTTPoison’s. Creative, I know. But, this way I can replace any usage of HTTPoison.get or HTTPoison.post with my own HttpClient.get or HttpClient.post. Easy peasy.

I’m also going to give room for dependency-injection so I can test this easily. And, maybe one day I won’t want to use HTTPoison anymore, so this new interface might also help transition my app to another HTTP client without disrupting too much.

Let’s write tests first:

defmodule MyApp.HttpClientTest do
  use MyApp.DataCase, async: true
  import ExUnit.CaptureLog
  alias MyApp.HTTPClient
  require HTTPoison

  # I used Erlang's :zlib.gzip("Hello")
  @gzipped_response <<31, 139, 8, 0, 0, 0, 0, 0, 0, 3, 243, 72, 205, 201, 201, 7, 0, 130, 137, 209, 247, 5, 0, 0, 0>>
  # I used Erlang's :zlib.zip("Hello")
  @zipped_response <<243, 72, 205, 201, 201, 7, 0>>
  @decoded "Hello"

  describe "get" do
    test "decompresses a gzipped body" do
      requester = fn _url, _headers, _options ->
        {:ok, %HTTPoison.Response{
          body: @gzipped_response,
          headers: [{"Content-Encoding", "gzip"}]
        }}
      end

      assert {:ok, %{body: @decoded}} = HTTPClient.get(nil, [], requester: requester)
    end

    test "decompresses a gzipped body with x-gzip header" do
      requester = fn _url, _headers, _options ->
        {:ok, %HTTPoison.Response{
          body: @gzipped_response,
          headers: [{"Content-Encoding", "x-gzip"}]
        }}
      end

      assert {:ok, %{body: @decoded}} = HTTPClient.get(nil, [], requester: requester)
    end

    test "does not attempt to decompress a plain body with gzip header" do
      body = "Hallo"
      requester = fn _url, _headers, _options ->
        {:ok, %HTTPoison.Response{
          body: body,
          headers: [{"Content-Encoding", "gzip"}]
        }}
      end

      assert {:ok, %{body: ^body}} = HTTPClient.get(nil, [], requester: requester)
    end

    test "decompresses a zipped body" do
      requester = fn _url, _headers, _options ->
        {:ok, %HTTPoison.Response{
          body: @zipped_response,
          headers: [{"Content-Encoding", "deflate"}]
        }}
      end

      assert {:ok, %{body: @decoded}} = HTTPClient.get(nil, [], requester: requester)
    end

    test "emits log when encountering unsupported compression" do
      requester = fn _url, _headers, _options ->
        {:ok, %HTTPoison.Response{
          body: "Hallo",
          headers: [{"Content-Encoding", "br"}]
        }}
      end

      assert capture_log(fn ->
        assert {:ok, _} = HTTPClient.get(nil, [], requester: requester)
      end) =~ "No support for decompression of body using 'br' algorithm"
    end

    test "emits log when failing to decompress" do
      requester = fn _url, _headers, _options ->
        {:ok, %HTTPoison.Response{
          body: <<1, 2, 3>>,
          headers: [{"Content-Encoding", "deflate"}]
        }}
      end

      assert capture_log(fn ->
        assert {:ok, _} = HTTPClient.get(nil, [], requester: requester)
      end) =~ "Failed to decompress response"
    end

    test "re-encodes a latin1 body to UTF-8" do
      latin1 = <<163, 233, 100, 117, 102, 102>>
      utf8 = "£éduff"
      requester = fn _url, _headers, _options ->
        {:ok, %HTTPoison.Response{
          body: latin1,
          headers: [{"Content-Type", "text/plain;charset=ISO-8859-1"}]
        }}
      end

      assert {:ok, %{body: ^utf8}} = HTTPClient.get(nil, [], requester: requester)
    end

    test "does not re-encode utf8 bodies" do
      utf8 = "£éduff"
      requester = fn _url, _headers, _options ->
        {:ok, %HTTPoison.Response{
          body: utf8,
          headers: [{"Content-Type", "text/plain;charset=utf-8"}]
        }}
      end

      assert {:ok, %{body: ^utf8}} = HTTPClient.get(nil, [], requester: requester)
    end

    test "emits log when encountering unknown encoding" do
      requester = fn _url, _headers, _options ->
        {:ok, %HTTPoison.Response{
          body: "Hallo",
          headers: [{"Content-Type", "text/plain;charset=duurf"}]
        }}
      end

      assert capture_log(fn ->
        assert {:ok, _} = HTTPClient.get(nil, [], requester: requester)
      end) =~ "Need to implement re-encoding support for: duurf"
    end

    test "emits log when failing to reencode" do
      body = <<163, 233, 100, 117, 102, 102, 833::3>>
      requester = fn _url, _headers, _options ->
        {:ok, %HTTPoison.Response{
          body: body,
          headers: [{"Content-Type", "text/plain;charset=ISO-8859-1"}]
        }}
      end

      assert capture_log(fn ->
        assert {:ok, _} = HTTPClient.get(nil, [], requester: requester)
      end) =~ "Failed to re-encode response"
    end

    test "does not re-encode un-specified bodies" do
      body = "£éduff" <> <<163, 233, 100, 117, 102, 102>>
      requester = fn _url, _headers, _options ->
        {:ok, %HTTPoison.Response{
          body: body,
          headers: []
        }}
      end

      assert {:ok, %{body: ^body}} = HTTPClient.get(nil, [], requester: requester)
    end
  end

  # copy/paste above, but adjust it for the post/4 function. Or, if you
  # want to be creative, make a macro to generate these tests for you.
end

Cool. That’s a lot of tests.

Let’s make those tests pass:

defmodule MyApp.HTTPClient do
  @moduledoc """
  A wrapper around HTTPoison that takes care of post-processing depending on the response, namely:
    1) decompress the response if gzipped
    2) re-encode the body into UTF-8 if ISO-8859-1
  """
  @default_getter &HTTPoison.get/3
  @default_poster &HTTPoison.post/4
  @default_options [timeout: 300_000, recv_timeout: 60_000]

  require Logger

  @doc """
  GET a URL with headers. Supported options:
    requester: fn(url, headers, request_options)
  """
  @spec get(String.t(), HTTPoison.headers(), list()) ::
    {:ok, HTTPoison.Response.t()} | {:error, HTTPoison.Error.t()}
  def get(url, headers, request_options \\ []) do
    opts = request_options ++ @default_options
    {get, opts} = Keyword.pop(opts, :requester, @default_getter)

    url
    |> get.(headers, opts)
    |> process_response
  end

  @doc """
  POST a URL with a body, headers. Supported options:
    requester: fn(url, body, headers, request_options)
  """
  @spec post(String.t(), HTTPoison.body(), HTTPoison.headers(), list()) ::
    {:ok, HTTPoison.Response.t()} | {:error, HTTPoison.Error.t()}
  def post(url, body, headers, request_options \\ []) do
    opts = request_options ++ @default_options
    {post, opts} = Keyword.pop(opts, :requester, @default_poster)

    url
    |> post.(body, headers, opts)
    |> process_response
  end

  defp process_response(response) do
    response
    |> decompress_response
    |> reencode_response_to_utf8
  end

  defp decompress_response({:error, response}), do: {:error, response}
  defp decompress_response({status, %{headers: headers, body: body} = response}) do
    try do
      decompressed_body =
        headers
        |> find_header("content-encoding")
        |> decompress_body(body)

      {status, %{response | body: decompressed_body}}

    rescue
      _ ->
        Logger.error("Failed to decompress response: #{inspect response}")
        {status, response}
    end
  end

  defp reencode_response_to_utf8({:error, response}), do: {:error, response}
  defp reencode_response_to_utf8({status, %{headers: headers, body: body} = response}) do
    try do
      reencoded_body =
        headers
        |> find_header("content-type")
        |> parse_charset()
        |> reencode_body(body)

      {status, %{response | body: reencoded_body}}

    rescue
      _ ->
        Logger.error("Failed to re-encode response: #{inspect response}")
        {status, response}
    end
  end

  defp find_header(headers, header_name) do
    Enum.find_value(
      headers,
      fn {name, value} ->
        name =~ ~r/#{header_name}/i && String.downcase(value)
      end
    )
  end

  # gzip's magic header is 0x1F 0x8B, with the 3rd byte specifying the compression method, 0x08
  # meaning "deflate". More info https://en.wikipedia.org/wiki/Gzip
  defp decompress_body(nil, body), do: body
  defp decompress_body("identity", body), do: body
  defp decompress_body("gzip", <<31, 139, 8, _::binary>> = body), do: :zlib.gunzip(body)
  defp decompress_body("gzip", body), do: body
  defp decompress_body("x-gzip", <<31, 139, 8, _::binary>> = body), do: :zlib.gunzip(body)
  defp decompress_body("x-gzip", body), do: body
  defp decompress_body("deflate", body), do: :zlib.unzip(body)
  defp decompress_body(other, body) do
    Logger.error("No support for decompression of body using '#{other}' algorithm.")
    body
  end

  defp parse_charset(nil), do: nil
  defp parse_charset(content_type) do
    case Plug.Conn.Utils.content_type(content_type) do
      {:ok, _, _, %{"charset" => charset}} ->
        cond do
          charset =~ ~r/utf-?8/ -> :utf8
          charset =~ ~r/iso-?8859-?1/ -> :latin1
          true -> charset
        end
      _ ->
        nil
    end
  end

  # When the header isn't sent, the RFC spec says we should assume ISO-8859-1, but the default is
  # actually different per format, eg, XML should be assumed UTF-8. We're going to not re-encode
  # if it's not sent and assume UTF-8. This should be safe for most cases.
  defp reencode_body(nil, body), do: body
  defp reencode_body(:utf8, body), do: body
  defp reencode_body(:latin1, body) do
    case :unicode.characters_to_binary(body, :latin1, :utf8) do
      {:error, binary, rest} ->
        Logger.error("Failed to re-encode text. BODY: #{inspect binary} REST: #{inspect rest}")
        body

      {:incomplete, reencoded_text, rest} ->
        Logger.warn("Failed to re-encode entire text. Dropping characters: #{inspect rest}")
        reencoded_text

      reencoded_text ->
        reencoded_text
    end
  end
  defp reencode_body(other, body) do
    Logger.error("Need to implement re-encoding support for: #{other}")
    body
  end
end