Dehydrated

Use dehydrated on FreeBSD to automatically renew SSL certificates using DNS-01 verification method with bind on a remote host. I will show here an example for the domain fechner.net using a wildcard certificate one and two levels deep (*.fechner.net, *.idefix.fechner.net).

Installation

Install it:

pkg install dehydrated

To enable automatic renewal:

echo weekly_dehydrated_enable=\"YES\" >> /etc/periodic.conf

Configuration

Bind

We do now the configuration on the external server bind is running.

For this we will use for each domain an extra key and an isolated zone file, to archive this, we delegate the acme related parts to an extra zone file.

Let’s create the key. I use as domain fechner.net, so replace the value with your domain name:

tsig-keygen -a sha512 acme_fechner.net >> /usr/local/etc/namedb/keys.conf
chown bind:bind /usr/local/etc/namedb/keys.conf
chmod 640 /usr/local/etc/namedb/keys.conf

Make sure the keys.conf is loaded in /usr/local/etc/namedb/named.conf:

/usr/local/etc/namedb/named.conf 
...
include "/usr/local/etc/namedb/keys.conf";

Now we create a new zone file for the acme related zone updates. I store my zone files in /usr/local/etc/namedb/master/fechner.net/, so create there a new file _acme-challenge.fechner.net:

/usr/local/etc/namedb/master/fechner.net/_acme-challenge.fechner.net 
$TTL 2m ; default TTL for zone
@       IN      SOA     ns.fechner.net. hostmaster.fechner.net. (
                        1 ; serial number
                        2m ; refresh
                        2m ; retry
                        2m ; expire
                        2m ; minimum
                        )
@       IN      NS      ns.fechner.net.
If you want to add a wildcard one level deeper, e.g. *.idefix.fechner.net, create also a file _acme-challenge.idefix.fechner.net.

Now we load this new zone. My master zones are defined in /usr/local/etc/namedb/named.zones.master:

/usr/local/etc/namedb/named.zones.master 
zone "_acme-challenge.fechner.net" {
        type master;
        file "/usr/local/etc/namedb/master/fechner.net/_acme-challenge.fechner.net";
        masterfile-format text;
        allow-update { key acme_fechner.net; };
};

Make sure permissions are correct:

chown bind:bind /usr/local/etc/namedb/keys.conf
chmod 640 /usr/local/etc/namedb/keys.conf
chown bind:bind /usr/local/etc/namedb/master/fechner.net/_acme-challenge*fechner.net
chmod 644 /usr/local/etc/namedb/master/fechner.net/_acme-challenge*fechner.net

Restart bind and verify the zone is correctly loaded:

service named restart
dig _acme-challenge.fechner.net.

You should see something like this (the SOA record):

; <<>> DiG 9.18.24 <<>> _acme-challenge.fechner.net.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1852
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: dc08ffe2f65683be0100000065d99f05e8c78dde55084b9d (good)
;; QUESTION SECTION:
;_acme-challenge.fechner.net.   IN      A

;; AUTHORITY SECTION:
_acme-challenge.fechner.net. 120 IN     SOA     ns.fechner.net. hostmaster.fechner.net. 1 120 120 120 120

;; Query time: 11 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Sat Feb 24 08:47:17 CET 2024
;; MSG SIZE  rcvd: 134

Now we add a delegation in the domain fechner.net by adding the following line:

/usr/local/etc/namedb/master/fechner.net 
;-- DNS delegation for acme validation
_acme-challenge     IN      NS      fechner.net.
; if you use a two level subdomain like *.idefix.fechner.net
; _acme-challenge.idefix     IN      NS      fechner.net.

Make sure you reload the zone file and it is valid.

Dehydrated

Now we can start to finalize the configuration for dehydrated.

Edit /usr/local/etc/dehydrated/config.

For testing we set the CA to letsencrypt-test make sure you use letsencrypt if everything is working as expected!

/usr/local/etc/dehydrated/config 
CA="letsencrypt-test"
CHALLENGETYPE="dns-01"
CONTACT_EMAIL="_your-email_"
HOOK="/usr/local/etc/dehydrated/hook.sh"
OCSP_FETCH="yes"

Now edit /usr/local/etc/dehydrated/domains.txt (here fechner.net is not part of the certificate, maybe you want to for your certificate!):

/usr/local/etc/dehydrated/domains.txt 
*.fechner.net *.idefix.fechner.net > star_fechner_net_rsa
*.fechner.net *.idefix.fechner.net > star_fechner_net_ecdsa

Now we configure the keys with (if you do not want to have a RSA key, you can skip this and remove the rsa line in domains.txt):

mkdir -p /usr/local/etc/dehydrated/certs/star_fechner_net_rsa
echo KEY_ALGO=\"rsa\" > /usr/local/etc/dehydrated/certs/star_fechner_net_rsa/config
chmod 700 /usr/local/etc/dehydrated/certs/star_fechner_net_rsa

Now we must store the acme_fechner.net key file we create with the tsig-keygen command on the bind server:

mkdir -p /usr/local/etc/dehydrated/tsig_keys

Make sure you paste the content from tsig genarete key from /usr/local/etc/namedb/keys.conf to the file /usr/local/etc/dehydrated/tsig_keys/fechner.net.key Secure it with:

chown root:wheel /usr/local/etc/dehydrated/tsig_keys/fechner.net.key
chmod 600 /usr/local/etc/dehydrated/tsig_keys/fechner.net.key
# if you use *.idefix.fechner.net
# ln -s /usr/local/etc/dehydrated/tsig_keys/fechner.net.key /usr/local/etc/dehydrated/tsig_keys/idefix.fechner.net.key

Now edit /usr/local/etc/dehydrated/hook.sh To create the required DNS entries:

/usr/local/etc/dehydrated/hook.sh 
#!/usr/local/bin/bash

DNSSERVER="fechner.net"
declare -A alg2ext=( ["rsaEncryption"]="rsa" ["id-ecPublicKey"]="ecdsa" )

deploy_challenge() {
  local DOMAIN="${1}" TOKEN_FILENAME="${2}" TOKEN_VALUE="${3}"
  local NSUPDATE="nsupdate -k /usr/local/etc/dehydrated/tsig_keys/${DOMAIN}.key"

  # This hook is called once for every domain that needs to be
  # validated, including any alternative names you may have listed.
  #
  # Parameters:
  # - DOMAIN
  #   The domain name (CN or subject alternative name) being
  #   validated.
  # - TOKEN_FILENAME
  #   The name of the file containing the token to be served for HTTP
  #   validation. Should be served by your web server as
  #   /.well-known/acme-challenge/${TOKEN_FILENAME}.
  # - TOKEN_VALUE
  #   The token value that needs to be served for validation. For DNS
  #   validation, this is what you want to put in the _acme-challenge
  #   TXT record. For HTTP validation it is the value that is expected
  #   be found in the $TOKEN_FILENAME file.

  printf 'server %s\nupdate add _acme-challenge.%s 300 IN TXT "%s"\nsend\n' "${DNSSERVER}" "${DOMAIN}" "${TOKEN_VALUE}" | ${NSUPDATE}
}

The remove the required DNS entries again (edit /usr/local/etc/dehydrated/hook.sh):

/usr/local/etc/dehydrated/hook.sh 
clean_challenge() {
  local DOMAIN="${1}" TOKEN_FILENAME="${2}" TOKEN_VALUE="${3}"
  local NSUPDATE="nsupdate -k /usr/local/etc/dehydrated/tsig_keys/${DOMAIN}.key"

  # This hook is called after attempting to validate each domain,
  # whether or not validation was successful. Here you can delete
  # files or DNS records that are no longer needed.
  #
  # The parameters are the same as for deploy_challenge.

  printf 'server %s\nupdate delete _acme-challenge.%s TXT "%s"\nsend\n' "${DNSSERVER}" "${DOMAIN}" "${TOKEN_VALUE}" | ${NSUPDATE}
}

To automatically copy the created certificates to the destination your services are expecting it (edit /usr/local/etc/dehydrated/hook.sh):

/usr/local/etc/dehydrated/hook.sh 
deploy_cert() {
  local DOMAIN="${1}" KEYFILE="${2}" CERTFILE="${3}" FULLCHAINFILE="${4}" CHAINFILE="${5}" TIMESTAMP="${6}"
  local SRC=$(dirname ${KEYFILE})
  local DST=/usr/local/etc/haproxy/certs
  local ALG=$(openssl x509 -in ${SRC}/cert.pem -noout -text | awk -F':' '/Public Key Algorithm/ {print $2}' | tr -d ' ')
  local EXT=${alg2ext[${ALG}]}

  # This hook is called once for each certificate that has been
  # produced. Here you might, for instance, copy your new certificates
  # to service-specific locations and reload the service.
  #
  # Parameters:
  # - DOMAIN
  #   The primary domain name, i.e. the certificate common
  #   name (CN).
  # - KEYFILE
  #   The path of the file containing the private key.
  # - CERTFILE
  #   The path of the file containing the signed certificate.
  # - FULLCHAINFILE
  #   The path of the file containing the full certificate chain.
  # - CHAINFILE
  #   The path of the file containing the intermediate certificate(s).
  # - TIMESTAMP
  #   Timestamp when the specified certificate was created.

  # dovecot
  service dovecot restart
  #postfix
  service postfix restart
  # haproxy
  ln -sf ${FULLCHAINFILE} ${DST}/${DOMAIN}.${EXT}
  ln -sf ${KEYFILE} ${DST}/${DOMAIN}.${EXT}.key
  service haproxy restart
}

For the OCSP information to be deployed to haproxy:

/usr/local/etc/dehydrated/hook.sh 
deploy_ocsp() {
  local DOMAIN="${1}" OCSPFILE="${2}" TIMESTAMP="${3}"
  local SRC=$(dirname ${OCSPFILE})
  local DST=/usr/local/etc/haproxy/certs
  local ALG=$(openssl x509 -in ${SRC}/cert.pem -noout -text | awk -F':' '/Public Key Algorithm/ {print $2}' | tr -d ' ')
  local EXT=${alg2ext[${ALG}]}

  # This hook is called once for each updated ocsp stapling file that has
  # been produced. Here you might, for instance, copy your new ocsp stapling
  # files to service-specific locations and reload the service.
  #
  # Parameters:
  # - DOMAIN
  #   The primary domain name, i.e. the certificate common
  #   name (CN).
  # - OCSPFILE
  #   The path of the ocsp stapling file
  # - TIMESTAMP
  #   Timestamp when the specified ocsp stapling file was created.

  ln -sf ${OCSPFILE} ${DST}/${DOMAIN}.${EXT}.ocsp
  service haproxy restart
}

To get errors by email if something fails:

/usr/local/etc/dehydrated/hook.sh 
invalid_challenge() {
  local DOMAIN="${1}" RESPONSE="${2}"

  # This hook is called if the challenge response has failed, so domain
  # owners can be aware and act accordingly.
  #
  # Parameters:
  # - DOMAIN
  #   The primary domain name, i.e. the certificate common
  #   name (CN).
  # - RESPONSE
  #   The response that the verification server returned

  printf "Subject: Validation of ${DOMAIN} failed!\n\nOh noez!" | sendmail root
}

request_failure() {
  local STATUSCODE="${1}" REASON="${2}" REQTYPE="${3}" HEADERS="${4}"

  # This hook is called when an HTTP request fails (e.g., when the ACME
  # server is busy, returns an error, etc). It will be called upon any
  # response code that does not start with '2'. Useful to alert admins
  # about problems with requests.
  #
  # Parameters:
  # - STATUSCODE
  #   The HTML status code that originated the error.
  # - REASON
  #   The specified reason for the error.
  # - REQTYPE
  #   The kind of request that was made (GET, POST...)
  # - HEADERS
  #   HTTP headers returned by the CA

  printf "Subject: HTTP request failed failed!\n\nA http request failed with status ${STATUSCODE}!" | sendmail root
}

The rest of the file you can leave untouched.

Make the hook executable:

chmod +x /usr/local/etc/dehydrated/hook.sh

Haproxy

Configuration part for haproxy:

/usr/local/etc/haproxy.conf 
...
        frontend www-https
                bind 0.0.0.0:443 ssl crt /usr/local/etc/haproxy/certs/ alpn h2,http/1.1
                bind :::443 ssl crt /usr/local/etc/haproxy/certs/ alpn h2,http/1.1
...

Postfix

Configuration part for postfix:

/usr/local/etc/postfix/main.cf 
smtpd_tls_chain_files =
        /usr/local/etc/dehydrated/certs/star_fechner_net_ecdsa/privkey.pem
        /usr/local/etc/dehydrated/certs/star_fechner_net_ecdsa/fullchain.pem
        /usr/local/etc/dehydrated/certs/star_fechner_net_rsa/privkey.pem
        /usr/local/etc/dehydrated/certs/star_fechner_net_rsa/fullchain.pem

Dovecot

Configuration part for dovecot:

/usr/local/etc/dovecot/local.conf 
ssl_cert = </usr/local/etc/dehydrated/certs/star_fechner_net_ecdsa/fullchain.pem
ssl_key = </usr/local/etc/dehydrated/certs/star_fechner_net_ecdsa/privkey.pem
ssl_alt_cert = </usr/local/etc/dehydrated/certs/star_fechner_net_rsa/fullchain.pem
ssl_alt_key  = </usr/local/etc/dehydrated/certs/star_fechner_net_rsa/privkey.pem

Test

We make the tests against to test environment of letsencrypt, make sure you have CA="letsencrypt-test".

At first register at the CA and accept their terms:

dehydrated --register --accept-terms

To test it (make sure you use the test CA):

dehydrated -c --force --force-validation

Go Live

If everything succeeded you must switch to the letsencrypt production environment.

Change in config:

/usr/local/etc/dehydrated/config 
CA="letsencrypt"

Remove the test certificates:

rm -R /usr/local/etc/dehydrated/certs

Now get the certificates with:

dehydrated --register --accept-terms
dehydrated -c

You maybe want to monitor now your certificates and the OCSP information that the refresh is working as expected.

Acme.sh

Installation:

pkg install acme.sh

Configuration is in:

/var/db/acme/.acme.sh/account.conf

Certificates are stored in

/var/db/acme/certs/

As the certificates are only accessible by user acme, we need to do an additional step to make the certificates available to dovecot/postfix/haproxy.

We do not modify any daemon but we let acme.sh write into a common/shared directory each website is using, so doing anything with acme.sh does not have any impact on any service from your server

As next we configure log rotation:

cp /usr/local/share/examples/acme.sh/acme.sh.conf /usr/local/etc/newsyslog.conf.d/

Make sure you uncomment the line in /usr/local/etc/newsyslog.conf.d/acme.sh.conf:

/var/log/acme.sh.log  acme:acme       640  90    *    @T00   BC

Next is to configure cron to automatically renew your certificates. For this we edit /etc/crontab

# Renew certificates created by acme.sh
MAILTO="idefix"
7       2       *       *       *       acme    /usr/local/sbin/acme.sh --cron --home /var/db/acme/.acme.sh > /dev/null

We need to create the logfile:

touch /var/log/acme.sh.log
chown acme /var/log/acme.sh.log

Allow acme to write the challenge files:

mkdir -p /usr/local/www/letsencrypt/.well-known/
chgrp acme /usr/local/www/letsencrypt/.well-known/
chmod g+w /usr/local/www/letsencrypt/.well-known/

Setup configuration of acme.sh:

echo ACCOUNT_EMAIL=\"name@yourdomain.tld\" >> account.conf

Hook the own custom deploy scripts from: https://gitlab.fechner.net/mfechner/letsencrypt_hooks Make sure you create a config file and now symlink the hook:

cd /var/db/acme/.acme.sh/deploy
ln -s /usr/home/idefix/letsencrypt/create-haproxy-ssl-restart-all_acme.sh

Now we can create our first test certificate (run this as root):

su -l acme -c "cd /var/db/acme && acme.sh --issue --test -k ec-256 -w /usr/local/www/letsencrypt -d beta.fechner.net -d vmail2.fechner.net -d smtp2.fechner.net --deploy-hook create-haproxy-ssl-restart-all_acme"
su -l acme -c "cd /var/db/acme && acme.sh --issue --test -k 2048 -w /usr/local/www/letsencrypt -d beta.fechner.net -d vmail2.fechner.net -d smtp2.fechner.net --deploy-hook create-haproxy-ssl-restart-all_acme"

If everything is fine, you can get the real certificates with:

su -l acme -c "cd /var/db/acme && acme.sh --issue -k ec-256 -w /usr/local/www/letsencrypt -d beta.fechner.net -d vmail2.fechner.net -d smtp2.fechner.net --deploy-hook create-haproxy-ssl-restart-all_acme --server letsencrypt --force"
su -l acme -c "cd /var/db/acme && acme.sh --issue -k 2048 -w /usr/local/www/letsencrypt -d beta.fechner.net -d vmail2.fechner.net -d smtp2.fechner.net --deploy-hook create-haproxy-ssl-restart-all_acme --server letsencrypt --force"

Now you should find an RSA and a ECDSA certificate in:

/var/db/acme/certs

As we will renew certificates of many domains, but tools like dovecot/postfix/haproxy need a directory or a single file we need to prepare these files and copy them with correct permissions to destination folders.

Add a new subdomain

You have already a certificate for vmail2.fechner.net and would like now to add more hosts to it.

Go to folder:

cd /var/db/acme/certs/vmail2.fechner.net_ecc

And add a new line to vmail2.fechner.net.conf or just attach a new subdomain seperated by comma:

Le_Alt='oldhost.fechner.net,newhost.fechner.net'

Tell acme to renew the certificate (I have problem make a forced renewal for ec-256 cert, I had to recreate it):

I have problem make a forced renewal for ec-256 cert, I had to recreate it

su -l acme -c "cd /var/db/acme && acme.sh --renew --force -k ec-256 -d vmail2.fechner.net"
su -l acme -c "cd /var/db/acme && acme.sh --renew --force -k 2048 -d vmail2.fechner.net"

Letsencrypt

We would like to use letsencrypt to get signed certificates for all our domains.

Approach with websites offline

I did this all from a virtual machine, as I do not want to let the client running with root permissions on my real server.

Everything was executed from an ubuntu machine running in a virtual machine. Create two shell scripts to get the certificate request simply created for several ALT entries:

create-crt-for-idefix.fechner.net.sh 
openssl req -new -sha256 -key domain.key -subj "/" -reqexts SAN -config <(cat /etc/ssl/openssl.cnf <(printf "[SAN]\nsubjectAltName=DNS:abook.fechner.net,DNS:amp.fechner.net,DNS:atlantis.fechner.net,DNS:caldav.fechner.net,DNS:carddav.fechner.net,DNS:git.fechner.net,DNS:gogs.fechner.net,DNS:idefix.fechner.net,DNS:idisk.fechner.net,DNS:imap.fechner.net,DNS:jenkins.fechner.net,DNS:knx.fechner.net,DNS:mail.fechner.net,DNS:moviesync.fechner.net,DNS:owncloud.fechner.net,DNS:pkg.fechner.net,DNS:safe.fechner.net,DNS:smtp.fechner.net,DNS:video.fechner.net,DNS:webcal.fechner.net,DNS:webmail.fechner.net,DNS:wiki.idefix.fechner.net,DNS:vmail.fechner.net,DNS:zpush.fechner.net")) > domain.csr
create-crt-for-fechner.net.sh 
openssl req -new -sha256 -key domain.key -subj "/" -reqexts SAN -config <(cat /etc/ssl/openssl.cnf <(printf "[SAN]\nsubjectAltName=DNS:fechner.net,DNS:www.fechner.net,DNS:wirkstoffreich.de,DNS:www.wirkstoffreich.de,DNS:vmail.lostinspace.de,DNS:lostinspace.de,DNS:admin.lostinspace.de,DNS:stats.wirkstoffreich.de,DNS:stats.fechner.net")) > domain.csr

To sign the certificates I did the following:

git clone https://github.com/diafygi/letsencrypt-nosudo.git
cd letsencrypt-nosudo/
openssl genrsa 4096 > user.key
openssl rsa -in user.key -pubout > user.pub
openssl genrsa 4096 > domain.key

python sign_csr.py --public-key user.pub domain.csr > signed.crt

Execute on the second terminal the commands the client asks you in the same directory.

You have to start a small python based webserver on the domain for each domain to verify you are the owner. Do this as the script is requesting it.

Now we install the certificate and key on our server. Copy the file domain.key and signed.crt to you server and execute the following:

cd /etc/mail/certs
wget https://letsencrypt.org/certs/lets-encrypt-x1-cross-signed.pem
cat signed.crt lets-encrypt-x1-cross-signed.pem > chained.pem

Edit you apache config to have:

/usr/local/etc/apache24/ssl/ssl-template.conf 
SSLCertificateChainFile /etc/mail/certs/chained.pem
SSLCertificateFile /etc/mail/certs/signed.crt
SSLCertificateKeyFile /etc/mail/certs/domain.key

Approach to authenticate domains while websites are online

We want to use the existing webserver to not make websites offline while authenticate the domains.

/usr/local/etc/apache24/ssl/letsencrypt.conf 
Alias /.well-known/acme-challenge /usr/local/www/letsencrypt/.well-known/acme-challenge
<Directory /usr/local/www/letsencrypt>
        Require all granted
</Directory>
ProxyPass /.well-known/acme-challenge !
Make sure you include this config file before you define other ProxyPass definitions.

Create the directory:

mkdir -p /usr/local/www/letsencrypt

Install the client:

pkg install security/py-letsencrypt

Create a script:

create-csr-idefix.fechner.net.sh 
#OPTIONS="--webroot --webroot-path=/usr/local/www/letsencrypt/ --renew-by-default --agree-tos"
OPTIONS="--webroot --webroot-path=/usr/local/www/letsencrypt/ --renew-by-default --agree-tos --server https://acme-staging.api.letsencrypt.org/directory"
sudo letsencrypt certonly ${OPTIONS} --email spam@fechner.net -d webmail.fechner.net -d idefix.fechner.net -d wiki.idefix.fechner.net -d pkg.fechner.net -d owncloud.fechner.net -d knx.fechner.net -d jenkins.fechner.net -d gogs.fechner.net -d git.fechner.net -d drupal8.fechner.net -d drupal7.fechner.net -d atlantis.fechner.net -d amp.fechner.net -d admin.fechner.net -d abook.fechner.net

Remove the –server directive from the OPTIONS after you have verified the run is successfull.
As letsencrypt has currently a heavy rate limit I recommend to request all sub domains with one certificate. This is not good for security but protects you from the problem that you cannot renew your certificate anymore and this is very bad if you use HSTS.

/usr/local/etc/apache24/ssl/ssl-template.conf 
SSLEngine on
<IfModule http2_module>
    Protocols h2 http/1.1
</IfModule>

SSLCertificateFile /usr/local/etc/letsencrypt/live/${SSLCertDomain}/fullchain.pem
SSLCertificateKeyFile /usr/local/etc/letsencrypt/live/${SSLCertDomain}/privkey.pem

<Files ~ "\.(cgi|shtml|phtml|php3?)$">
    SSLOptions +StdEnvVars
</Files>
<Directory "/usr/local/www/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>

SetEnvIf User-Agent ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0
/usr/local/etc/apache24/Includes/mydomain.de.conf 
Define SSLCertDomain mydomain.de
Include etc/apache24/ssl/letsencrypt.conf
Include etc/apache24/ssl/ssl-template.conf
Make sure you define the SSLCertDomain for the master domain you requested the certificate (it is normally the first domain you run the letsencrypt script).