This is almost certainly related to some sort of Russian cybercrime investigation. If you ever read Krebs or peruse some of the seedier Russian forums, xmpp.ru will sound familiar to you, because it is. Not imputing anything to the operator, that's just the nature of operating an anonymous service.

https://ddanchev.blogspot.com/2021/03/exposing-currently-act...

https://ddanchev.blogspot.com/2019/07/profiling-currently-ac...

https://blog.talosintelligence.com/picking-apart-remcos/

https://flashpoint.io/wp-content/uploads/Plea-Agreement-USA-...

Really interesting writeup though. I guess it's a practical example of why everyone should get a CT monitoring service!

I agree with most of the mitigation suggestions.

For high risk targets, consider layering an additional auth mechanism that doesn't rely on trusted CAs: Tor onion services, SSH, or Wireguard.

> All issued SSL/TLS certificates are subject to certificate transparency.

+1. crt.sh has RSS feeds for this.

> Limit validation methods and set exact account identifiers

Using CAA is a good idea in general, but would it help in this case? The attacker would just request the exact cert configuration that is permitted by CAA. Maybe this helps if you can strengthen one validation method?

> Monitor SSL/TLS certificate changes on all your services using external service

+1. High-risk targets should be aware of what certs are valid at any time, and be checking for those.

> Monitor MAC address of default gateway for changes

A more sophisticated attack could preserve the MAC address.

> "Channel binding" is a feature in XMPP which can detect a MiTM even if the interceptor present a valid certificate.

TIL.

Perfectly pulling off an actual MitM attack and then forgetting to renew the certificate is certainly a very German thing :-)

Archive [1]

Their theories are interesting and would explain how they obtained the certs if true. Perhaps the take-away here is to have multiple probes on multiple providers checking all of ones TLS fingerprints and alerting on unknown fingerprints and then checking the certificate transparency log or lack thereof.

    openssl s_client -servername news.ycombinator.com -connect news.ycombinator.com:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin
    SHA1 Fingerprint=7E:49:BA:40:86:87:B3:39:66:93:94:9E:9C:45:71:85:3C:8D:95:16

I was a bit curious about the MAC address since it isn't locally administered which indicates to me that someone likely deliberately chose this range of addresses.

https://www.google.com/search?q=%2290%253Ade%253A01%22+linod...

Googling for '"90:de:01" linode' and '"90:de:00" linode' indicates that addresses from these blocks have been assigned to other linode VMs recently. I sadly don't have a linode VM of my own to compare with right now but it would seem like the traffic has been routed to another VM on linode infrastructure

This is baseless speculation, but I'm assuming Jabber is being targeted as it's famously used on darknet markets for drug trades (or other illicit activity). Goes to show that you should never just trust "it's encrypted, bro". You need to PGP your messages at the very least. Is PGP crackable by quantum computers? Will there be hardening against those kinds of attacks in the future? Since, if the messages have been hoovered up in encrypted form, it's just a matter of time until they get decrypted. And this appears to be done for just about all web traffic they can get their hands on... see https://en.wikipedia.org/wiki/Utah_Data_Center

>End-to-end encrypted communications, such as OMEMO, OTR or PGP, are protected from the interception only if both parties have validated the encryption keys. The users are asked to check their accounts for new unauthorized OMEMO and PGP keys in their PEP storage, and change passwords.

The attacker would need to set up a separate MiTM for the particular E2EE scheme used. Some of the XMPP clients I have encountered will not let you use a particular cryptographic identity unless you have explicitly claimed to have verified it.

Still a good reminder. If you have not seen and dealt with a ridiculously long number (or equivalent) you have not achieved end to end.

There's one thing I can't understand in this story: if that's lawful interception, why Hetzner and Linode bothered to set up MitM interception with different LE certificate and key, rather than extract the TLS private key directly from the RAM and/or storage device of the VPS? Even if this is a physically dedicated server, they can extract the private key from the RAM by dumping the RAM contents after unscheduled reboot. Extraction of the private key isn't visible in CT logs, much more stealthier, practically undetectable.

A user reported this on Reddit 3 days ago: https://www.reddit.com/r/hetzner/comments/17ankoh/does_hetzn...

I wonder if there is a legal way to demand Hetzner/Linode comments on this situation. Likely, the entity behind the interception is some government agency or police.

This really wants me to re-evaluate all the times I have gracefuly TOFU'd while SSHing into a machine or even logging onto a website.

I've always thought a solution like LetsEncrypt would be an ideal backdoor for LEAs wanting to circumvent SSL/TLS.

The same thing applies to VPN services. Why bother trying to develop tools etc that can decrypt traffic when you can just have people send their traffic to you?

Wouldn't be surprised if this happened in Russia; but not in Europe. Also, Let's Encrypt security is a joke - they issued fake certificate without serious checking. Using unencrypted HTTP for confirmation is a vulnerability - doesn't this mean that large national ISPs can easily issue ceritificates for any site hosted within a country?

Could this be a blue pill attack? A vulnerability in the xmpp server exploited to inject a rootkit, which then hides itself inside the kernel?

Or creates network/pid namespaces and puts you in them, while leaving the mitm server in the original one?

If so, the mitm could be on the same host, and wouldn't need the cooperation of the hosting provider.

I'm not sure how to check for either of these without restarting (which the admin does not seem to want to do, as it is a live service).

https://en.wikipedia.org/wiki/Blue_Pill_(software)

>The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023

Why is it even possible to issue more than 1 certificate on the same domain via Let’s Encrypt? Shouldn't the previous certificate be revoked when a new one is issued?

Are there any services that monitor the certificate transparency logs and send an email when a new certificate is issued for your domain? That would alert you to this kind of MITM

As an end user, how can I monitor if I’m being MITM’d? If everything I do is proxied through an attacker’s network, nothing I do online could be trusted to properly test if I were being proxied, right? I know apps can do cert pinning, but as an end user how can I validate that when connect to anything that it’s the real one and not a request from being middled?

@dang: .org.ru should probably be treated as TLD

Next time someone in Matrix discussion will ask "why we need Matrix when there is jabber/xmpp" show them this:

> All jabber.ru and xmpp.ru communications between these dates should be assumed compromised. Given the nature of the interception, the attacker have been able to execute any action as if it is executed from the authorized account, without knowing the account password. This means that the attacker could download account's roster, lifetime unencrypted server-side message history, send new messages or alter them in real time.

> The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023

> We tend to assume this is lawful interception Hetzner and Linode were forced to setup based on German police request.

> Another possible, although much more unlikely scenario is an intrusion on the internal networks of both Hetzner and Linode targeting specifically jabber.ru — much harder to believe but not entirely impossible.

And what if the attacker tricked somehow the letsencrypt challenges?

Or this is supposed to be impossible?

Using DANE or so (RRs CAA/TLSA) can help in this case (when provider route traffic to specific port), but does not solve the problem completely. If DNS record will be compromised, then hacker can setup "correct" TLSA/CAA record to trust his fake certificate. As result, problem can be solved, if we will have reliable DNS subsystem, where is impossible to compromise DNS provider or infrastructure.

I think, emerDNS can be used in such critical application: https://emercoin.com/en/documentation/blockchain-services/em...

And this is why, kids, you need to enforce certificate pinning on your critical infrastructure!

We are using self developed application layer security. It requires specialized client code and is terminated at a gateway that proxies requests to our normal applications.

I really, really hate the idea of this kind of eavesdropping.

Interesting reddit thread ending here: https://www.reddit.com/r/hetzner/comments/17ccp3i/hetzner_do...

let's encrypt is a public non-profit, who would definitely assist in finding out how domain control verification (DCV) happened - somehow that obvious line of pursuit was missing from the investigation. The mystery of the MAC 90:de:01:49:76:ae is interesting though, but CIA-custom-Made NIC might be a bit superfluous as an idea. Linode (acquired by Akamai) might use something in-house designed and OEM-manufactured, so I wouldn't treat it as strong evidence. Talk to Let's Encrypt - that will likely unveil what happened.

Interestingly, search "90:de:01" in this page: https://www.cyberciti.biz/faq/howto-linux-configuring-defaul...

^^ looks like another victim VM of the alleged mysterious interceptor :-)

What a nefarious move by Hetzner and Linode. How to trust them after this?

I’m wondering if mTLS (aka zero-trust) is used, would take prevent this kind of MITM attack? My understanding is that it should since the cert is self-signed to be used from both ends.

A reminder e2ee is a must.

so is jabber.ru still secure now? what needs to be done on the user side to make it so? can one reissue/request new server certificate from the server and not from let's encrypt? thanks

The way that certbot works with let's encrypt the only surprise is that this does not occur more often. We have monitoring on several certs used for TLS. We should probably add an alert if an A-record changes as well.

I assume the root cause is DNS tricks.

well i must admit monitoring certificate issuance with LetsEncrypt is quite boring even if you HAVE alerts set up (I do)

so… not surprised. Still cool. What a time to be alive

So, if I rent server in those services, it all potentially can be compromised and there is no way to make sure things are secure?..

http://archive.today/0R7Or

This is a deal breaker for me. Which is a shame because Hetzner has such great pricing. Oh well.

> We tend to assume this is lawful interception Hetzner and Linode were forced to setup based on German police request.

This seems to be somewhat jumping the gun I think.

Given the certs and the target assuming it's an lawful interception seems reasonable.

But there is nothing there which requires Hetzner or Lindoe complying or knowing about this.

Given the nature of the attacks you can do the interception on the carrier level, and carriers being forced to comply with lawful wiretapping is pretty much anywhere in the world pretty much standard and many laws are based around that approach. Much less so around approaches involving data centers.

Getting a valid cert for a domain you don't own is getting close to trivial. All of the mitigation strategies are defeated a number of ways, have been for years. I've commented on incidents like these for years on HN. I'm not some internet big wig, and any blog I write isn't going to trend, so this never gets traction. But anyone who understands the tech can find a half dozen ways to get a cert. If you really need to trust your connection, don't rely on internet PKI / public certs. It was secure enough for general purpose in the aughts, it's not anymore.