This type of attack seems to be becoming more and more common - social engineering the server provider to gain root access. Good work by the localbitcoins guys in making everything so secure that even with root access the intruder (probably) gained nothing.
> The attacker gained a root access to the server for ~40 minutes
and
> All data on the website server is encrypted. Manual actions are needed to make this data readable, so the attacker could not gain access to the data even when having a server console access.
Unless the "encrypted data" is simply being stored on that server, and not used, this seems like a hopeful statement to make. If a server process is accessing it, then it must have the key, at the very least, in memory (which root can access, though finding the key might be somewhat of a feat). Even if the truly valuable stuff was on another server, could the hacker not masquerade as the webserver itself, and make requests that appear to have originated from the webserver?
Granted, this assumes a lot about the ability of the attacker. I'm mostly saying it's possible, not that it's probable; the latter is harder to determine. Good see them rebuilding on fresh hardware: better to start from scratch than to try and "uncompromise" a system. Shame on the provider though; I'm curious to know who: I feel like such situations deserve a bit of name-and-shame, as they represent a very severe failure on the part of the provider. And the provider should detail very clearly what steps will be taken such that this will not happen again.
> Unless the "encrypted data" is simply being stored on that server, and not used, this seems like a hopeful statement to make. If a server process is accessing it, then it must have the key, at the very least, in memory (which root can access, though finding the key might be somewhat of a feat)
The attacker got access via a reboot, so something as simple as passphrase-protected private keys would probably be sufficient here - "manual actions" could well refer to "enter the passphrase after startup". Or using something like kerberos, where credentials have to be fetched/validated via a separate server.
I suspect they have a base OS installation and then have a post-boot encrypted partition which requires manual passphrase entry over ssh or console to unlock, containing all the relevant data. That's a pretty normal way to set up remotely-adminned fairly secure servers.
When it's "cold" (offline, or running but in system-low mode), you have no real protection of the OS, but you essentially distrust the machine whenever it gets rebooted without your consent. If someone can get root on the server while it's in "high mode" (normal operation, after keys are entered), any protections are irrelevant anyway.
The risk is if the attacker can cause enough reboots that you stop caring about reboots so much, and just blindly enter the passphrase; the attacker then puts up something which steals the passphrase, and then uses the passphrase on an already-imaged drive, winning the game. Or if the attacker can use one of a large number of vulnerabilities to get access to your data when it's running in system-high mode; either an app-level vulnerability or either local access + local root, or remote root.
Server technology sucks for anything needing great security but adminned strictly remotely.
> I suspect they have a base OS installation and then have a post-boot encrypted partition which requires manual passphrase entry over ssh or console to unlock, containing all the relevant data. That's a pretty normal way to set up remotely-adminned fairly secure servers.
Debian and Ubuntu both support this pretty painlessly.
1) Do the base install with full disk encryption.
2) Put an ssh public key into /root/.ssh/authorized_keys
3) Install dropbear.
4) Force a initramfs rebuild if it doesn't happen automatically.
You'll get an initramfs that has dropbear ssh embedded which you can log into and unlock the disk.
Right, but there's no real proof the dropbear/sh/etc. you talk to is untampered after reboot; it could easily be logging, if someone shut your system down and replaced the boot drive. ssh vs. serial console at least requires they extract some (unencrypted) key from the drive, though.
The DNS resolves to 81.17.24.30 which apparently is owned by Private Layer Inc and is located in Switzerland. Private Layer's homepage (https://www.privatelayer.com/) shows a selection of dediboxes, so I'm guessing that's the provider.
Good use of full disk encryption (or encryption of everything but the base bootable OS) and separation of systems (bitcoind on a separate server.) That's the way to do things! Good on them.
It seems that if a bitcoin site is to use a 3rd party hoster, they would be wise to at least require all email correspondence be cryptographically signed; better still , why not require encrypted email exchanges.
Is there a hosting provider which supports this level of service?
I'm not referring to just private mail services, but rather if you can implore customer service to remotely administer you servers (even just rebooting) via email they should only act on crypto signed correspondence. Seems this would reduce the success of email social engineering as an attack vector. It might not make sense in all cases but if I am storing monetary value for others it would be a nice extra level to promote.
i strongly suggest that sites concerned with their security use (1) full-on colocation services and (2) encrypt the partition they store user data on or encrypt the entire disk.
if you have proper disk encryption, it is non-trivial to game the remote physical access to the machine, e.g. attacker convinces someone to use 'remote hands' to reboot server and then gets console redirected pre-boot. if you have disk encryption in this scenario, whether it is on the user data partition or the whole disk, you will surely be notified of the unscheduled reboot and can investigate it.
it is always best to host your own machines (_not_ VPSes) and be able to provide some level of compartmentalization to your hosting setup.
Careful, disk encryption usually doesn't cover the entire disk. So an attacker can place an evil initramfs in the /boot partition that stores away your disk encryption passphrase, for example.
This is even easier if the operating system partition is plaintext and only the data partition is encrypted. Then it's trivial to modify any binary, library, or startup scripts!
Encryption just means an attacker cannot get at the data right away. But once the admin brings up the system again (not knowing something has been tampered with) it's pretty easy to get access.
Of course, there are availability concerns with that, which you can mitigate with onsite redundancy (dual power supply machines fed separately, etc.), and multi-site replication.
The big issue is how to get keys back into the system. I suggest having some onsite tamper-resistant component and some typed in remotely component.
The annoying thing is there is no viable low-end solution for this yet. It makes sense once you're dropping $100-150k of equipment per site, but for 2 servers per site, it's a huge pain and lots of overhead.
i always advocate for FDE, but that often has issues with remote serial console. the threat model of running without disk encryption is far worse for most bitcoin-related sites than the complexity associated with redundancy. if they get hacked, they are likely going to eat downtime anyhow.
as far as low-end solutions are concerned, a usb serial console adapter plus a few machines runs about USD 5K. set machines to redirect console to serial and have an OOB machine for unlocking downed servers.
i'd be interested to hear what kind of solutions there are for onsite tamper-resistant components.
