letsencrypt, haproxy, and auto-renewal

If you have not heard about letsencrypt it is an amazing, and free, certificate authority. It proves free (as in beer) ssl certificates for anyone who can prove they own the domain. There is a little helper utility that can you can use to help you get a cert called certbot. The concept is pretty simple, a quick breakdown looks like this:

  1. Client says to Server “I want a cert for x.y.z domain”
  2. Server says verify you own this domain by serving file “/.well-known/acme-challenge/1234567890abcdef” from x.y.z domain
  3. Client setups the file as requested
  4. Server verifies file exists
  5. Server issues certificate for x.y.z domain

I have glossed over the massive amount of research and security involved in doing all of this, but that is the general concept. Of note, the certificate is only valid for 3 months and cannot be a wildcard cert.

Now lets talk about the issue. Once you receive your certificate you then use it on your webserver or application you are consuming the ports letsencrypt is expecting to use, namely 80 and 443. How do you renew the certificate without stopping the services? Enter Haproxy. If you happen to be loadbalancing through haproxy, you are in luck! You can host your site _and_ still do proper renewals with no downtime. The way it works is quite simple, haproxy can check certain things about the request and trigger conditions based on that. In this case we will be testing for if the URI begins with “/.well-known/acme-challenge”. If it does, we know to forward that to our certbot client. Here is how the haproxy config looks.

frontend ssl_redirector
    bind ssl crt /etc/haproxy/ssl/
    http-request del-header X-Forwarded-Proto
    http-request set-header X-Forwarded-Proto https if { ssl_fc }

    # Check if this is a letsencrypt request based on URI
    acl letsencrypt-request path_beg -i /.well-known/acme-challenge/
    # Send to letsencrypt-backend if it is a letsencrypt-request
    use_backend letsencrypt_backend if letsencrypt-request

    default_backend website_backend

frontend http_redirect
    # Redirect to HTTPS if this is not a letsencrypt-request
    redirect scheme https code 301 if !letsencrypt-request

    # Check if this is a letsencrypt request based on URI
    acl letsencrypt-request path_beg -i /.well-known/acme-challenge/
    # Send to letsencrypt-backend if it is a letsencrypt-request
    use_backend letsencrypt_backend if letsencrypt-request

backend letsencrypt_backend
    server letsencrypt

backend website_backend
    server server01
    server server02

Lets go through each section. The first section, ‘ssl_redirector’, listens on public ip and port 443. It has all certs it can server in /etc/haproxy/ssl/. It sets the X-Forward-Proto to https (some applications may require this). The next step is the meat of the issue we are solving. The acl checks if the path begins with “/.well-known/acme-challenge/” and if it does it sends it to the “backend letsencrypt_backend” section. All of this is the same for the ‘http_redirect’ section. If it doesn’t detect anything letsencrypt related, it forwards the request to one of the ‘website_backend’ servers like normal.

So that’s it. Haproxy will now detect and forward letsencrypt requests to a server located at “”. Now its time to setup that server.

I wrote a little bash script to do a cert renewal using a docker container I created. The docker container is samyaple/certbot. It is based on the github repo SamYaple/certbot. It builds automatically in DockerHub when I push changes to the github repo, which is pretty sweet. That’s a subject for another time, though. The script I use for autorenewing is here. I’ve left comments throughout the script to explain why certain code gets run.

# cert_renewal.sh

set -o errexit


# This should only run when fetching a new cert
function http_failback {
    docker run --rm -v /etc/letsencrypt:/etc/letsencrypt -p samyaple/certbot:v0.8.1 --standalone --standalone-supported-challenges http-01 --http-01-port 49494 -d ${FQDN}

function fetch_certs {
    # If SNI fails, fail back to http authorization
    docker run --rm -v /etc/letsencrypt:/etc/letsencrypt -p samyaple/certbot:v0.8.1 --standalone --standalone-supported-challenges tls-sni-01 --tls-sni-01-port 49494 -d ${FQDN} || http_failback

function install_certs {
    if [[ -e "/etc/letsencrypt/live/${FQDN}/fullchain.pem" ]]; then
        cat /etc/letsencrypt/live/${FQDN}/{fullchain.pem,privkey.pem} > /etc/haproxy/ssl/${FQDN}.pem


systemctl reload haproxy

You execute this script with the paramater of your domain name and you are golden. It should create/renew your ssl cert and then reload haproxy. This can be made into a cronjob or simply run every 2 months to ensure renewal before the cert expires.

`./cert_renewal.sh test.example.com` will produce a test.example.com.pem file that haproxy will be able to use. Magic!