In October 2023, Jabber.ru, “the largest Russian XMPP messaging service”, discovered that both Hetzner and Linode had been targeting them with Machine-In-The-Middle (MITM) attacks for up to 6 months. MITM attacks are when an unauthorised third party intercepts traffic intended for someone else. At the point of interception, the attacker can inspect and even modify that traffic. TLS was created to mitigate this; all communication between the two parties is encrypted, so the third party sees nothing but gibberish (ciphertext).

TLS is great, but it’s actually not enough when the attacker owns your network, as in Jabber.ru’s situation. Jabber.ru rented servers from Hetzner and Linode, who altered their network’s routing setup to obtain TLS certificates for Jabber.ru’s domains and successfully carry out a MITM. When connecting to an XMPP server, most clients are only configured to look for a valid certificate. A valid certificate matches the service’s domain name, is not expired, and is authorised by a known and trusted Certificate Authority (CA). If the client sees a certificate that’s signed by an unknown CA or whose expiry has passed or the domain in the cert doesn’t match the service domain or any combination of those, it’s considered invalid; the client should terminate the connection before transmitting sensitive data, such as the user’s password.

Because Hetzner and Linode controlled Jabber.ru’s network, they were able to meet all of those conditions. XMPP clients would just accept the rogue (but valid!) certificates and continue along as normal, unaware that they were actually connecting to a rogue server that forwarded their traffic (possibly with modifications) to the proper server.

A fairly straightforward mitigation involves DNS-based Authentication of Named Entities, or DANE. This is just a standard way to securely communicate to clients what certificate keys they should expect when connecting. When clients initiate a connection to the XMPP server, they receive a TLS certificate that includes a public key. If the server admin has implemented DANE, the client can verify that the public key they received matches what the server administrator said they should receive. If they don’t match, the client should terminate the connection before transmitting sensitive data.

Please note that while this post continually refers to DANE as it relates to XMPP, it could just as easily refer to any system that uses TLS, such as SMTP, Matrix, Mattermost, Rocket Chat, and more. The servers don’t need to do anything with DANE, just the clients connecting to the servers.

Additionally note that this doesn’t mitigate cases where a provider has access to the server’s filesystem. If it’s a VPS, the provider could just snapshot the virtual disk and pick out the certificate files (as well as any other files they find interesting). If it’s a baremetal server, they’d have a harder time interacting with the filesystem without notifying the owner of their presence, but they could definitely still do it. Physical access is equivalent to root access.

DANE requires the XMPP server’s authoritative nameserver, TLD, and domain registrar to all support DNSSEC. If those prerequisites are met, implementing DANE involves hashing the public keys of the current certificates and publishing them to DNS as TLSA records. The following commands extract the public key from a local PEM-encoded x509 ECC certificate, re-encode it to DER, then hash it and print the hash. If your key is RSA, replace ec with rsa.

$ openssl x509 -in xmppserver.example.pem -inform pem -pubkey -noout \
  2>/dev/null | openssl ec -pubin -outform der 2>/dev/null | sha256sum \
  | awk '{print $1}

9ff8a6d7aab386dfbd8272022d04f82204d1093332e6fc33d1c55ee21e0aedd0

The long sequence of letters and numbers is the hash of the key and what gets published to DNS. The following commands initiate a connection to retrieve the certificates, extract and convert the public key, then hash it and print the hash.

$ echo | openssl s_client -showcerts -servername xmppserver.example \
  -connect 198.51.100.7:5270 2>/dev/null | openssl x509 -inform pem \
  -pubkey -noout 2>/dev/null | openssl ec -pubin -outform der \
  2>/dev/null | sha256sum | awk '{print $1}'

9ff8a6d7aab386dfbd8272022d04f82204d1093332e6fc33d1c55ee21e0aedd0

When it comes to rotating certificates, admins have two options. The first and easiest is to reuse the key-pair from the original certificate. Certbot allows you to do this with the –reuse-key option. Caddy has an equivalent option. The other route is rotating both the certificates and the key-pair. This should be done well before the certificates expire. After obtaining the new certificates and keys, do not immediately swap them into production! Hash the new keys, then publish them as new TLSA records. Wait at least two TTLs, then swap the new certificates in and replace the old ones. Wait at least two more TTLs, then remove the TLSA records corresponding to the old key-pair.

Waiting in between steps is necessary to reduce false positives and mitigate race conditions. Say the TTL is two hours and a client connects half an hour before the administrator starts rotating things. They obtain the new keys, hash them, publish the hashes, then swap the certificates and reload the XMPP server. Say the client reconnects five minutes after the administrator finishes. It’ll receive the new certificate file, but not pick up on the new record because administrator has said, through the two-hour TTL, that resolvers should only request DNS records once every two hours. For the next 1h25m, until the cache expires and their resolver re-requests all the TLSA records, the client will refuse to connect to the server and might even warn the user that the server and their account are compromised. Waiting two more TTLs before removing the old record is necessary to handle the very unlikely event where the client connects and receives the old certificate file right before the admin removes the old record. If they check DNS for that old, now-missing record after receiving the old certificate, the client should refuse the connection.

danectl is a tool that uses Certbot to create and manage certificates and key-pairs, it generates BIND9 DNS records you can copy/paste into your DNS management tool, and even verifies that you’ve published the records correctly. It also works with SSHFP records to implement DANE with SSH servers, OPENPGPKEY records for GPG keys, and SMIMEA records for S/MIME certificates.

Some clients are currently unaware of DANE, so it can be helpful to monitor TLS setups through an external service. Later in 2023, we created and announced a tool to fill this gap, CertWatch. You provide your XMPP server’s domain and it performs the same checks a client would over Tor, to prevent easy detection by an attacker.