Using Let's Encrypt with internal web servers (without DNS challenge)

2020-07-05

TL;DR

Use internet facing domain on an internal network, I normally use subdomains for this.
Domain must have a DNS A record pointing to a public facing web server so Let's Encrypt can find it for the HTTP-01 challenge. This can be served as an empty site or just as a 404 response.
Remote VPS uses certbot to renew SSL certificates as normal.
Use a script like renew-letsencrypt-certificates.sh to copy the SSL certs from the remote machine to our local private machine. Run this as a cron job.

Details

If you have a web site on an internal network that is not accesible by a public URL, then the most popular HTTP-01 challenge for Let's Encrypt is not going to work. Let's Encrypt needs to access http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN> which it won't be able to do if your internal or private server is not internet facing.

The alternative is a DNS challenge, which requires a DNS provider with an API interface. It can also be a slow process since you may need to wait for the TTL for your domain.

We'll assume your internal network's web server is not accessible from the internet and that you're running your own DNS server pointing a A record (or CNAME) of mydomain.com to an internet facing server.

Here's an example of how we can get around this and use HTTP-01 challenge.

You'll need your domain name with a web server accessible online, which could be serving a 404 response, or just an empty page. Do this separate to your private server.

Configure certbot to auto renew your SSL certificates as you normally would. I have a separate article about how to use certbot.

Set up a script renew-letsencrypt-certificates.sh on your private server to run automatically. The script will:

Connect to your remote host via SSH and obtains a tarball of your remote SSL certs. You'l need to make sure you have the correct SSH keys configured so that the SSH commands can run without user interaction. It assumes your certs are located in /etc/letsencrypt/live/$DOMAIN
The tarball is copied to the private server using scp and extracted to /etc/letsencrypt/live/$DOMAIN
Removes remote tarball
Check whether the certs are different (i.e renewed) using sha256sum. If they're different, restart or reload the web server.

We also determine using the script, what the remote IP address is of the domain, by quering for the DNS A record using the domain's SOA DNS server. We do this using:

# Get SOA DNS server
DNS=$(dig soa $DOMAIN | grep -v ^\; | grep SOA | sed -r 's/^(.*)SOA\t//' | cut -d " " -f 1 | sed -r 's/\.$//')

# Get remote public facing web server IP address
IP_REMOTE=$(dig @$DNS +short $DOMAIN A)

Here is a listing of the script overall, however you should use the most updated version on GitHub Gist.

#!/bin/bash
#
# renew-letsencrypt-certificates.sh DOMAIN [EMAIL]
#
# Copy Let's Encrypt SSL certs from a remote public facing web server to local filesystem
# Look for changes, if any change, restarts the web service
# Useful for using Let's Encrypt with local internal servers, with custom DNS.
# Working "mail" command needed for email alerts
#

if [ -z $1 ]; then
    echo "Syntax error, use:"
    echo "  renew-letsencrypt-certificates.sh DOMAIN [EMAIL]"
    exit 1
fi

# SSH options to remote VPS, e.g different port
SSH_OPTS="-p 2222"
SCP_OPTS="-P 2222"

DOMAIN=$1

# send email message here when a renewal occurs, or on error
EMAIL=$2

# Get SOA DNS server
DNS=$(dig soa $DOMAIN | grep -v ^\; | grep SOA | sed -r 's/^(.*)SOA\t//' | cut -d " " -f 1 | sed -r 's/\.$//')

# Get remote public facing web server IP address
IP_REMOTE=$(dig @$DNS +short $DOMAIN A)
if [[ -z $IP_REMOTE ]]; then
    echo "Can not determine remote IP for $DOMAIN"
    exit 1
fi

# .pem certificates will be saved here. You must have write permissions here
# and your Apache or nginx .conf files should refer to this path
# Actual cert files will be in a subdir e.g $CERT_PATH/$DOMAIN/*.pem
CERT_PATH=/etc/letsencrypt/live
CERT_PATH_DOMAIN="$CERT_PATH/$DOMAIN"

# e.g httpd, nginx, refer to restart_www() for methods of restart or reload
WEB_SERVER=nginx

DIR=$(dirname $0)

CERT_TARBALL_REMOTE=/root/certs_${DOMAIN}.tar.gz
CERT_TARBALL_LOCAL=/root/certs_${DOMAIN}.tar.gz

# this flag is used by this script, leave this alone!
certs_updated=0

restart_www () {
    if [[ "$WEB_SERVER" == "nginx" ]]; then
        nginx -s reload
    else
        systemctl restart $WEB_SERVER
    fi
}

# return 0 if renewed, 1 if not renewed for whatever reason
refresh_certificates () {

    # create tarball on public server of current letsencrypt certs for desired domain
    echo "Create \"$CERT_TARBALL_REMOTE\" on remote..."
    
    ssh $SSH_OPTS root@$IP_REMOTE "rm -f \"$CERT_TARBALL_REMOTE\"; tar -zchf \"$CERT_TARBALL_REMOTE\" -C \"$CERT_PATH\" $DOMAIN"

    # copy tarball from public to internal server, remove existing file if it iexsts
    [[ -f "$CERT_TARBALL_LOCAL" ]] && rm -f "$CERT_TARBALL_LOCAL"
    echo "Copy remote \"$CERT_TARBALL_REMOTE\" to local \"$CERT_TARBALL_LOCAL\"..."
    scp $SCP_OPTS root@$IP_REMOTE:"$CERT_TARBALL_REMOTE" "$CERT_TARBALL_LOCAL"

    # create cert path if we need to
    [[ -d "$CERT_PATH_DOMAIN" ]] || mkdir -p "$CERT_PATH_DOMAIN"

    # check local tarball is persent
    if [[ ! -f "$CERT_TARBALL_LOCAL" ]]; then
        echo "Error: Local \"$CERT_TARBALL_LOCAL\" is missing. The copy or write operation above has failed. Quitting"
        exit 1
    fi

    echo "Extracting certificates to local \"$CERT_PATH_DOMAIN\"..."
    tar -zxvf "$CERT_TARBALL_REMOTE" -C "$CERT_PATH_DOMAIN" --strip-components=1 > /dev/null

    echo "Delete remote \"$CERT_TARBALL_REMOTE\"..."
    ssh $SSH_OPTS root@$IP_REMOTE "rm -f '$CERT_TARBALL_REMOTE'"

    # Restart web server only if certs changed since last run of this script
    # Although it's probably fine if we just restarted it each time anyway
    # Create a CHECKSUMS file in $CERT_PATH_DOMAIN, verify against this in the future to detect changes
    pushd . &> /dev/null
    cd "$CERT_PATH_DOMAIN"
    if [[ -f CHECKSUMS ]]; then
        echo -n "Checking whether certificates have changed... "
        sha256sum --status -c CHECKSUMS
        if [[ $? -ne 0 ]]; then
            # checksum is different, certificate has changed, restart web server and reclaculate checksums
            echo "Change found. Certificates updated. Restarting or reloading $WEB_SERVER ..."
            restart_www
            sha256sum *.pem > CHECKSUMS
            return 0
        else
            # No update performed, certficates the same as previous
            echo "Certificates not changed."
            return 1
        fi
    else
        # no checksum performed yet
        echo "Certificates updated. Restarting or reloading $WEB_SERVER ..."
        restart_www
        sha256sum *.pem > CHECKSUMS
        return 0
    fi

    popd > /dev/null

}

#
# Script starts here
#

# check cert stats by connecting to local web server
cert_stats () {
    echo | openssl s_client -showcerts -connect $1:443 2> /dev/null | openssl x509 -noout -dates
}

# we need to copy those certs to this machine at regular machine
refresh_certificates

if [[ $? -eq 0 ]]; then
    # updated, both cron and interactive mode

    # restart web server
    nginx -s reload

    echo "SSL cert updated. New certificate validity dates:"

    # get SSL cert stats
    if [[ ! -z $EMAIL ]]; then
        cert_stats $DOMAIN | mail -s "SSL cert updated" $EMAIL
    fi
    cert_stats $DOMAIN
elif [[ -t 0 ]]; then
    # not updated - interactive mode
    echo "SSL cert does not need updating. Current certificate validity dates:"
    cert_stats $DOMAIN
fi

exit 0