In October 2023, Jabber.ru, “the largest Russian XMPP messaging service”, discovered that both Hetzner and Linode had been targeting them with Machine-In-The-Middle (MITM) attacks for up to 6 months. MITM attacks are when an unauthorised third party intercepts traffic intended for someone else. At the point of interception, the attacker can inspect and even modify that traffic. TLS was created to mitigate this; all communication between the two parties is encrypted, so the third party sees nothing but gibberish (ciphertext).

TLS is great, but it’s actually not enough when the attacker owns your network, as in Jabber.ru’s situation. Jabber.ru rented servers from Hetzner and Linode, who altered their network’s routing setup to obtain TLS certificates for Jabber.ru’s domains and successfully carry out a MITM. When connecting to an XMPP server, most clients are only configured to look for a valid certificate. A valid certificate matches the service’s domain name, is not expired, and is authorised by a known and trusted Certificate Authority (CA). If the client sees a certificate that’s signed by an unknown CA or whose expiry has passed or the domain in the cert doesn’t match the service domain or any combination of those, it’s considered invalid; the client should terminate the connection before transmitting sensitive data, such as the user’s password.

Because Hetzner and Linode controlled Jabber.ru’s network, they were able to meet all of those conditions. XMPP clients would just accept the rogue (but valid!) certificates and continue along as normal, unaware that they were actually connecting to a rogue server that forwarded their traffic (possibly with modifications) to the proper server.

A fairly straightforward mitigation involves DNS-based Authentication of Named Entities, or DANE. This is just a standard way to securely communicate to clients what certificate keys they should expect when connecting. When clients initiate a connection to the XMPP server, they receive a TLS certificate that includes a public key. If the server admin has implemented DANE, the client can verify that the public key they received matches what the server administrator said they should receive. If they don’t match, the client should terminate the connection before transmitting sensitive data.

Please note that while this post continually refers to DANE as it relates to XMPP, it could just as easily refer to any system that uses TLS, such as SMTP, Matrix, Mattermost, Rocket Chat, and more. The servers don’t need to do anything with DANE, just the clients connecting to the servers.

Additionally note that this doesn’t mitigate cases where a provider has access to the server’s filesystem. If it’s a VPS, the provider could just snapshot the virtual disk and pick out the certificate files (as well as any other files they find interesting). If it’s a baremetal server, they’d have a harder time interacting with the filesystem without notifying the owner of their presence, but they could definitely still do it. Physical access is equivalent to root access.

DANE requires the XMPP server’s authoritative nameserver, TLD, and domain registrar to all support DNSSEC. If those prerequisites are met, implementing DANE involves hashing the public keys of the current certificates and publishing them to DNS as TLSA records. The following commands extract the public key from a local PEM-encoded x509 ECC certificate, re-encode it to DER, then hash it and print the hash. If your key is RSA, replace ec with rsa.

$ openssl x509 -in xmppserver.example.pem -inform pem -pubkey -noout \
  2>/dev/null | openssl ec -pubin -outform der 2>/dev/null | sha256sum \
  | awk '{print $1}

9ff8a6d7aab386dfbd8272022d04f82204d1093332e6fc33d1c55ee21e0aedd0

The long sequence of letters and numbers is the hash of the key and what gets published to DNS. The following commands initiate a connection to retrieve the certificates, extract and convert the public key, then hash it and print the hash.

$ echo | openssl s_client -showcerts -servername xmppserver.example \
  -connect 198.51.100.7:5270 2>/dev/null | openssl x509 -inform pem \
  -pubkey -noout 2>/dev/null | openssl ec -pubin -outform der \
  2>/dev/null | sha256sum | awk '{print $1}'

9ff8a6d7aab386dfbd8272022d04f82204d1093332e6fc33d1c55ee21e0aedd0

When it comes to rotating certificates, admins have two options. The first and easiest is to reuse the key-pair from the original certificate. Certbot allows you to do this with the –reuse-key option. Caddy has an equivalent option. The other route is rotating both the certificates and the key-pair. This should be done well before the certificates expire. After obtaining the new certificates and keys, do not immediately swap them into production! Hash the new keys, then publish them as new TLSA records. Wait at least two TTLs, then swap the new certificates in and replace the old ones. Wait at least two more TTLs, then remove the TLSA records corresponding to the old key-pair.

Waiting in between steps is necessary to reduce false positives and mitigate race conditions. Say the TTL is two hours and a client connects half an hour before the administrator starts rotating things. They obtain the new keys, hash them, publish the hashes, then swap the certificates and reload the XMPP server. Say the client reconnects five minutes after the administrator finishes. It’ll receive the new certificate file, but not pick up on the new record because administrator has said, through the two-hour TTL, that resolvers should only request DNS records once every two hours. For the next 1h25m, until the cache expires and their resolver re-requests all the TLSA records, the client will refuse to connect to the server and might even warn the user that the server and their account are compromised. Waiting two more TTLs before removing the old record is necessary to handle the very unlikely event where the client connects and receives the old certificate file right before the admin removes the old record. If they check DNS for that old, now-missing record after receiving the old certificate, the client should refuse the connection.

danectl is a tool that uses Certbot to create and manage certificates and key-pairs, it generates BIND9 DNS records you can copy/paste into your DNS management tool, and even verifies that you’ve published the records correctly. It also works with SSHFP records to implement DANE with SSH servers, OPENPGPKEY records for GPG keys, and SMIMEA records for S/MIME certificates.

Some clients are currently unaware of DANE, so it can be helpful to monitor TLS setups through an external service. Later in 2023, we created and announced a tool to fill this gap, CertWatch. You provide your XMPP server’s domain and it performs the same checks a client would over Tor, to prevent easy detection by an attacker.

As you may have already seen, on October 21st, it was reported that a long-running, successful MITM (Machine-In-The-Middle) attack against jabber.ru had been detected. The nature of this attack was not specific to the XMPP protocol in any way, but it was of special interest to us as members of the XMPP community. This kind of attack relies on being able to present a TLS certificate which anyone trying to connect will accept as valid. In this case, it was done by getting a valid certificate from Let’s Encrypt.

When it comes to mitigation strategies for client-to-server connections, luckily there is already an excellent option called channel binding. Most XMPP clients and servers already have some amount of support for this technique, and in the wake of this attack, most are scrambling to make sure their implementations are complete. Many service providers have also added CAA DNS records which can prevent the very specific way this attack was executed from succeeding.

We’ve been hard at work on a different tool that can also help with defense-in-depth for this kind of situation. Ultimately, a MITM will use a different public key from the one the server uses, even if it is wrapped in a signed certificate declared as valid by a trustworthy authority (like Let’s Encrypt). If we know what key is seen when trying to connect, and we know what key the server administrator expects us to see, we can detect an ongoing MITM of this variety even when the certificate presented is valid. The tool we have developed is in early testing now. We call it CertWatch.

The premise is simple. The server administrator knows exactly what public/private keypair they are using (or can easily find out) and publishes this in DNSSEC-signed DNS records for our tool to find. The tool then periodically polls the XMPP server over Tor to see what certificate is presented. If the key in the certificate matches the key in the DNS zone, we know the session is not MITM’d (some caveats below). CertWatch checks the current setup of any domain entered, and if not yet declaring any keys, it displays setup instructions. It will either tell you to enable DNSSEC or it will tell you which DNS records to add. Note that these records are additive, so it is safe to add multiple sets when serving multiple domains from one host through SRV records. Once everything looks good, running a domain through CertWatch will display a success message and instructions for getting notified of any issues. It will then poll the domain periodically, and if any key mismatches are found, those subscribing to notifications will receive an alert.

Some tools change your key on every certificate renewal, which means you would have to update your zone setup every time your certificates renew. Other tools allow you to reuse existing keys and save some hassle, such as certbot with the --reuse-key option.

Caveats

If we did our polls from our main server IPs, it would be easy for any attacker to detect our probes and selectively disable the MITM attack for us, making themselves invisible. Probing over Tor gives CertWatch a different IP for every request and a traffic profile almost certainly consistent with the sort that many MITM attackers are going to want to inspect. This is not perfect, however, and it may be possible to fingerprint our probes in other ways to selectively MITM some traffic and ignore others. Just because our tool’s sessions were not MITM’d does not prove that no sessions are.

Anyone with physical access to the server may also scrape the actual certificates and keys off the disk, or use similar techniques in order to execute a MITM with exactly the same key the server operator expects and would use. The particular mitigation technique CertWatch helps administrators implement is ineffective against this. Rotating the key occasionally may help, but it really depends on the sophistication of the attacker and how much access they have.

Check it Out

So head over to CertWatch, enter your service domain, and let us know what you think.

Some apps and services (but not JMP!) require an SMS verification code in order to create a new account.  (Note that this is different from using SMS for authentication; which is a bad idea since SMS can be easily intercepted, are not encrypted in transit, and are vulnerable to simple swap scams, etc.; but has different incentives and issues.)  Why do they do this, and how can it affect you as a user?

Tarpit

In the fight against service abuse and SPAM, there are no sure-fire one-size-fits-all solutions.  Often preventing abusive accounts and spammers entirely is not possible, so targets turn to other strategies, such as tarpits.  This is anything that slows down the abusive activity, thus resulting in less of it.  This is the best way to think about most account-creation verification measures.  Receiving an SMS to a unique phone number is something that is not hard for most customers creating an account.  Even a customer who does not wish to give out their phone number or does not have a phone number can (in many countries, with enough money) get a new cell phone and cell phone number fairly quickly and use that to create the account.

If a customer is expected to be able to pass this check easily, and an abuser is indistiguishable from a customer, then how can any SMS verification possibly help prevent abuse?  Well, if the abuser needs to create only one account, it cannot.  However, in many cases an abuser is trying to create tens of thousands of accounts.  Now imagine trying to buy ten thousand new cell phones at your local store every day.  It is not going to be easy.

“VoIP Numbers”

Now, JMP can easily get ten thousand new SMS-enabled numbers in a day.  So can almost any other carrier or reseller.  If there is no physical device that needs to be handed over (such as with VoIP, eSIM, and similar services), the natural tarpit is gone and all that is left is the prices and policies of the provider.  JMP has many times received requests to help with getting “10,000 numbers, only need them for one day”.  Of course, we do not serve such customers.  JMP is not here to facilitate abuse, but to help create a gateway to the phone network for human beings whose contacts are still only found there.  That doesn’t mean there are no resellers who will work with such a customer, however.

So now the targets are in a pickle if they want to keep using this strategy.  If the abuser can get ten thousand SMS-enabled numbers a day, and if it doesn’t cost too much, then it won’t work as a tarpit at all!  So many of them have chosen a sort of scorched-earth policy.  They buy and create heuristics to guess if a phone number was “too easy” to get, blocking entire resellers, entire carriers, entire countries.  These rules change daily, are different for every target, and can be quite unpredictable.  This may help when it comes to foiling the abusers, but is bad if you are a customer who just wants to create an account.  Some targets, especially “big” ones, have made the decision to lose some customers (or make their lives much more difficult) in order to slow the abusers down.

De-anonymization

Many apps and services also make money by selling your viewing time to advertisers (e.g. ads interspersed in a social media feed, as pre-/mid-roll in a video, etc.) based on your demographics and behaviour.  To do this, they need to know who you are and what your habits are so they can target the ads you see for the advertisers’ benefit.  As a result, they have an incentive to associate your activity with just one identity, and to make it difficult for you to separate your behaviour in ways that reduce their ability to get a complete picture of who you are.  Some companies might choose to use SMS verification as one of the ways they try to ensure a given person can’t get more than one account, or for associating the account (via the provided phone number) with information they can acquire from other sources, such as where you are at any given time.

Can I make a new account with JMP numbers?

The honest answer is, we cannot say.  While JMP would never work with abusers, and has pricing and incentives set up to cater to long-term users rather than those looking for something “disposable”, communicating that to every app and service out there is a big job.  Many of our customers try to help us with this job by contacting the services they are also customers of; after all, a company is more likely to listen to their own customers than a cold-call from some other company.  The Soprani.ca project has a wiki page where users keep track of what has worked for them, and what hasn’t, so everyone can remain informed of the current state (since a service may work today, but not tomorrow, then work again next week, it is important to track success over time).

Many customers use JMP as their only phone number, often ported in from their previous carrier and already associated with many online accounts.  This often works very well, but everyone’s needs are different.  Especially those creating new personas which start with a JMP number find that creating new accounts at some services for the persona can be frustrating to impossible.  It is an active area of work for us and all other small, easy-access phone network resellers.