Deploying Elixir/Phoenix projects to production

Setting up a bare Ubuntu 18 machine to deplaoy an Elixir/Phoenix service supervised by systemd behind nginx to handle TLS (SSL) encryption.

systemctl shows it’s running nicely

Just scroll down to the bottom if you’re just after the script without any explanation.

Preface

So, we’ve all got to grind through this and I’s like to share my way how I am setting up a simple Elixir/Phoenix service on a bare Ubuntu machine.

I do not recommend this for bigger installations, though: Just use Docker and Kubernetes if possible. This guide shall be a howto for small projects and doesn’t touch CI/CD and all the pipeline work that this usually entails.

I assume you have a git repo somewhere and you’d like to simply deploy the master branch. I also assume that there’s no database involved — this I will cover in a next episode: It’s pretty starightforward but I’d like to concentrate on the Elixir part in this article.

The plan

• basic system configuration (auto-updates)
• install erlang and nodejs from vendor’s repositories via apt — so we have one tool apt to handle all updates
• get the source from a git repo
• fetch dependencies and compile your Elixir source and possibly JS via webpack (or whatever your JS build tool is)
• install nginx as reverse proxy to handle all TLS and to show a nice “down” page should our Elixir service be down
• create an Erlang release and install this as a service

A plain bash script

At the bottom I added a script that I simply run on install. It’s usually part of my project repository and and also serves as cloud-init script.

What it does

1. It installs all necessary runtimes (Erlang, Elixir, NodeJS as I am assuming Phoenix and webpack)
2. It enables auto-updates and upgrades for most system packages and lets the system send you emails about that.
3. Checks out a git repository, assuming this is an Elixir mix-controlled project, gets dependencies, builds it and compiles a release. See → Releases
4. Registers and starts your application as a systemd service for auto-restarts on failure and nice logs via journalctl.
5. Sets up nginx to act as a reverse proxy, handling all those TLS details for you. Also shows a plain “down for maintenance” page if your service goes down which is bit nicer than the infamous BAD GATEWAY.
6. Retrieves a TLS certificate from letsencrypt for you and auto-renews it if due (every 3 months).

Tadaa! Your application should now be available at https://$DOMAIN

Prerequisites

The IP of your server must be resolvable as $DOMAIN in the DNS. So be sure to set the A record of your domain (e.g. foobar.gutschilla.com) to your IPv4 address and if you’re extra cool set the AAAA record to your IPv6 address.

Change/set the variables at the beginning of the script:

1. EMAIL → mail address that update notification will be sent to
2. MAIL_HOST, MAIL_USER, MAIL_PASS →email address from which update notifications will be sent from
3. DOMAIN →the domain name your project shall be available from (e.g. foobar.remoteceros.com)
4. PROJECT → a name that will be used as a common handle within the system, e.g. the project directory, the service name etc. Something like super-api or drinking-web that may be a file name is great. Don’t use something likely to clash with exiting services like “nginx”.
5. GIT_URL → something to feed to “git clone”. Ideally this is a https checkout that includes a user and a password. For projects hosted at Gitlab I create read-only deployment tokens (in a project under Settings → Repository) which are exactly these: a username and a password.

Your project should support releases with distillery. This is optional. See → Releases for a howto and a workaround if you don’t want it.

Caveats

• For simplicity’s sake credentials are not taken from environment variables or some credential service.
• If the application crashes on startup you won’t necessarily get notified. Be sure to run systemctl status $PROJECT.service or journalctl -u $PROJECT.service to check if all is green.
• The application runs as userwww-data just as nginx does. If you want more isolation, create another user and chown the release’s var directory to it.
• Even if there are auto-updates, your system might still need some maintenance every now and then as new Erlang/Elixir packages won’t get automatically installed (and they shouldn’t to stay compatible).
• You are still responsible for security updates of your application’s libraries, both Elixir and Javascript ones.

On Erlang releases

They basically pack the Erlang runtime along with your compiled project as beam-files. This makes releases handy when you want to build them on your machine and the deploy the release as-is on some server. This is pretty cool as you don’t need Erlang and all this on your target. There are tools like edeliver that help automate this process.

In this example we’re compiling the release on your target machine. It’s considerably easy to do and gives you all the wonders of a remote shell into your project without caring too much.

Adding distillery is quite easy (for smaller projects). The distillery documentation shows how. It’s basically adding {:distillery, "~> 2.0"} to your deps and running mix release init checking-in the resulting ./rel directory into your repo. Make sure to read the docs for more info. But usually that’s just it: One extra dependency and one init command.

Even simpler: without releases

OK, you don’t care about how nice this all would be? Simply omit the release step “mix release” and in your systemd unit file

• remove the mix release line 245
• remove following chown www-data… line 248
• remove the ExecStop line 280
• uncomment the whole line 268 EcecStart … iex -S mix phx.server and remove or comment-out the existing ExecStart … foregorundcommand line 269

Going further from here

This machine does too much.

This is OK for small projects but in real™ projects you might want to use nginx as a load-balancing proxy. This way you can simply spawn new machines if you want to update (just reinstall) or scale (just add more instances) your application.

For this to work smoothly it’s best to have a cloud provider and some internal networking. Also you can spin up more machines for databases, KV-stores and whatnot.

But if you’re going that far you’re probably better off using Docker, Kubernetes and some fancy CI/CD system with build pipelines for development, staging and production stages on top of that.

That’s out of scope for today.

And finally, the script:

I am not a very proficient bash developer so while this works it might be scary to the experienced bash hacker out there. Oh and if you have more elegant solutions for example to get a random reboot time than resorting to perl in line 6 — I love to hear about it.

Todo

• provide project example ready to check out and install

Changes

• Edit 1: (2019–09–07): Thanks to https://gist.github.com/thenrio for suggesting tee instead of (cat …) > file. Also, finally found out how to do inline code.