This is probably pretty common. When I first deployed virtualization at a previous job ~5 years ago, it took almost two years before I realized all images used the same host keys.
Can someone versed in cryptography comment on whether this allows passive eavesdropping? Active MITM sure, but I thought session keys would be unique.
SSH uses ephemeral diffie hellman which gives you forward secrecy, meaning that recorded sessions can't be decrypted later if the server private key is compromised. So sessions sniffed in the past can't be decrypted using the private key that was shared among VM's. So provided that all server owners immediately change their keys and no one was man-in-the-middled before Hetzner announced this, everyone should stay secure. I very much doubt anyone was MITM'd because it would be difficult and the prize for MITM'ing a Hetzner server isn't that motivating.
"Impersonification" of the server, you mean (inservication if that's a word). It would allow an attacker to operate a machine under their control that poses as mine.
But then, for password based logins, that will actually be enough to impersonify the user as well (the attacker will create an ssh connection to the real server with the given password). But I wonder how things work with key based logins.
I've ordered a server from Hetzner recently, I guess I'll just ditch it (and perhaps order a new one). BTW I haven't gotten any email notification from Hetzner about this yet.
My point was that the issue didn't just allow a MITM to capture traffic, but also take over the machine(?). So cancelling the server and installing a new one will be the safer bet (I'm considering moving to a different company at the same time, as the IP block my old server is in seems to be the reason I can't deliver mail to Google, Hotmail and Yahoo (my own IP is all green in blacklists, but my neighbours' IPs aren't)). YMMV.
I suppose it depends on your paranoia level and whether or not you were using keybased auth or passwords. If you were using keybased auth then simply regenerating the host key should be enough.
Thanks for the info about your mail. I'm using SPF and DKIM too, I don't see how DMARC would help deliverability so I haven't tried that yet. I've verified my domains (including the reverse name(s) of my server's IPs) with postmaster.google.com to no avail (and I'm not sending enough mail to get the actual tools enabled). Interesting that you think you could solve it that way, perhaps you're sending more mail, or perhaps M$/Google/AOL are interchanging info and it only helps when going through all of their tools, or perhaps you've sent enough mail that what actually helped was people marking mail as ham, or perhaps you're in a block that's tainted less. My strategy remains to try another IP block. I might also try disabling IPv6 (although IIRC it delivers to Google over IPv4?).
Given that key based logins require signing a per-session token to verify the owner of the key is trying to connect (otherwise they would be useless since someone could just perform a replay attack) MITM'ing a key-authenticated session would allow you to perform attacks with that session, but it would not allow you to re-authenticate against that server later.
Would active MITM be possible with key login? I would think it should be impossible.
If the symmetric session key is chosen by the server and encrypted with the client's public key, the attacker won't know it and cannot mess with the session.
The only thing the attacker can do is completely impersonate the server. But the attacker cannot communicate with the actual server at all.
Not that I'm saying anyone wouldn't try it but how would you even sweep it under the rug? Either you quietly regenerate host keys, in which case all your clients will get a host key mismatch when they try logging in, or you fix it in new images and leave everyone exposed and someone will notice the host key being the same at some point.
It's a pretty noisy flaw and one that's common on virtualized platforms deployed from an image.
1) just don't say anything and hope nobody notices.
2) fix the underlying problem and then regenerate all keys and post some nonsense saying they had to do so for some "not their fault bullshit reason".
3) remove support for the key/cipher type affected with some bs reason and force all customers to use something else.
And some creative persons can probably come up with more...
Good on them that they acknowledged the problem. Warned customers and provide instructions on how to fix things.
Makes me want to consider them if I ever need a service like theirs in the future.
2) and 3) are not really viable/realistic with the kind of service Hetzner offers -> if they just went in and modified customer systems, possibly breaking stuff in the process, they'd be in even more trouble.
> Their business is being a trustworthy, high quality service.
Really. Here I thought their business is renting out servers made from desktop parts for insanely low amounts of money. I have no problems with this strategy, I enjoy my 42EUR a month 2x3TB HDD + 2x120GB SSD, 16GB i7 2600 server, thanks much. It runs hobby sites and such. I know what I bought and what I can expect.
no.
you've forgotten something.
you need to pay at least the flexi pack for ever server, too.
So your math is
Switch + 12.95/month (per server) + 15 /month (per server) and now you outpriced nearly every cloud.
Ah ok. I've only ever rented servers from them that already had a second NIC present (so I just needed to pay the flexi-pack charge to activate it). Didn't realise they also sold servers with single NICs.
Most of them are. Only if you use the "bigger" ones which aren't cheaper than the cloud providers anyway so. If you care for pricing it's cheaper to either have a colocation or a cloud or a own datacenter. renting servers is mostly not cheap if you need more than a single server.
I mean most people's need definitely suites inside the cloud since they don't need to have the full capacity 24 hours.
There are some other people who need the full capacity every time, but if you run a globally distributed services you mostly run into other problems than datacenter costs.
In my case, I have three servers (each with ECC 32GB, Xenon quad core, 2x3TB disks and unlimited bandwidth), a 5 port internal switch, two public IPv4 addresses per server, a public failover IP, 300GB of network backup space and two additional rack slots reserved in the same rack for future expansion.
The total monthly cost for this (exc. VAT) is €180. I haven't been able to find any co-location or cloud providers that can come anywhere close to that price for a similar resource setup.
They do offer cheap and cheerful desktop grade gear but their PX and R ranges are server grade (Xeon chips, ECC memory etc). They offer PowerEdge boxes too.
I've been a customer for a while but needed some spare, cheap server grade boxes this month. Decided to give their server bidding system a try and picked up two 32GB quad-core Xeon boxes (with 2x3TB SATA disks, Intel NICs and unlimited bandwidth) for €39/month each. So far, no complaints.
If anyone works for a hosting provider, please write a script that checks all of your customers' SSH ports for identical fingerprints. If you have any collisions, you have a problem.
Except that performing network scans, even on your own network, is illegal in many countries.
Even in countries where this is legal, providers cannot take the risk of running a scan that could possibly crash a customer application, or they could be sued for that.
Which T&Cs do not have a blanket clause to indemnify the hosting company in such situations? If I accidentally reboot the virtualisation host, can my company be sued for making the application "crash"?
I can imagine a scenario where a user that logs into a machine and tries to read a very sensitive file that they shouldn't have access to (should only be read by sshd and a specific automation user) trips an alarm via hooks in SELinux or something similar that causes the host to shut down immediately and security admins notified. It should just outright deny it I'd argue instead of shutting down, but I've seen really weird security policies before that make little practical sense and wouldn't be too surprised if someone created an IDS enforcement policy similar to this.
> I can imagine a scenario where a user that logs into a machine and tries to read a very sensitive file...
That's several steps beyond connecting to port 22 and slurping up the host fingerprint. At that point, SSH has prompted for credentials, accepted them, and set up a shell or SFTP session for you.
Hell, if the system lets you log in using SSH and read files, that really sounds like the system has granted you permission to access those files. (I'm sure a lawyer could argue with me, but I think we're all aware of how (willfully?) ignorant members of the Judicial branch can be of the nuances of computer things.)
An interesting argument. With telnet it's easy to argue that it wasn't supposed to allow access, it was just a default and you should have known better. With ssh you have to go pretty far out of your way to allow someone to log in without having a key or password. Sounds like something that would imply authorization.
> With ssh you have to go pretty far out of your way to allow someone to log in without having a key or password.
Yeah. It's certainly not the default config. From what I can tell, you can either
* Activate PAM support and have a PAM module that just authorizes users, regardless of credentials.
* Enable PermitEmptyPasswords and have an account on the system that has no password.
So, I guess if you SSH to a machine that lets you log in with no password, or any username/password combo, you might want to consider that maybe your use of the machine is not necessarily authorized. :p
My hypothetical in the previous comment presumed that you presented a username/key or a username/password combo that you knew to the system and it granted you access... but I guess the situation can get a little more murky, if you're accessing a system managed by a pathologically lazy sysadmin. ;)
Eh, if you're in a position to know a legitimate account like that, you should already know whether you're authorized. In that case "it let me in so surely it was okay" is not a very strong argument.
True unless the server is not on the public internet. In that case, pulling the plug if port 22 is accessed would be a reasonably defensible idea (you are being attacked for sure, and it's hard to attack a machine that is off) ...albeit paranoid in the extreme.
Affected customers have received an email about the issue. I got mine about 2 hours ago.
It would be great not to publish an article like that until some time later, so that everyone gets a chance to fix the problem on their machines before it goes fully public - especially since it's the holiday season and some people don't check email or news that often.
Some additional details and their mail to customers here: http://blogs.intevation.de/thomas/hetzner-duplicate-ed25519-...
"I must say, I’m impressed. Especially at this time of the year I would have expected a slower reaction or a less detailed announcement."
Yet another reason not to use images provided by your hosting company.
In anycase, if you are relying on hosting company images (complete with their root backdoor for "performance monitoring control panel!"), it's not like you care very much about your servers' security in the first place...
> In order to be able to intervene on your dedicated server without your root password, the automatic installation of ssh key is done. Only authorized employees of OVH will use it. It is not a gap in security, contrary, thanks to this OVH has root rights to your server and may identify the problems with your server. When you request an intervention, we need to have access to ssh.
Sounds to me like what is needed for a managed root-server. Actually it's kinda cool because you can enjoy timely upgrades, security fixes and response to ddos attacks while at the same time you also have the flexibility to install needed software without going through a change request workflow...
A SSH private key is just some bytes. Once you have access to it, you have access to it forever in the future. RSA doesn't care if you are an "authorized employee".
A static SSH key for a single human is already risky over a long time period, but for a key that is shared between multiple humans... wow.
It's not as though it's impossible to make a secure image. Mistakes can and do happen, though. Better to own up to it and fix it than to hide the problem.
I reported a similar issue on Chunkhost (https://chunkhost.com/) back in 2011... They never responded to my email about remediation, but hopefully the issue was fixed.
I do hope this page is not the only source for the announcement, since it's a wiki page on a non-HTTPS MediaWiki instance that runs an abysmally old and unsupported (= unsafe) version of the software.
Can someone versed in cryptography comment on whether this allows passive eavesdropping? Active MITM sure, but I thought session keys would be unique.