Lessons to be learned from the Linode Bitcoin incident
Firstly let me say that I like Linode a lot. They had a promotion running a little while ago which got me going with my first virtual private server (VPS), and I only moved off to somewhere from lowendbox after the promotion because my needs are small (and I wanted to match my spend accordingly)[1]. This tweet probably sums things up perfectly:
I first heard about the incident via Hacker News, where somebody had posted a blog post from one of the victims. The comments on both sites make for some interesting reading about security and liability. Shortly later Linode posted a statement. The true details of what went down, and whether it was an inside job as some speculate, or ‘hackers’ remains to be seen. Clearly whoever perpetrated the attack knew exactly what they were after and went straight for it – what law enforcement would normally call a ‘professional job’.
So… what can we learn from this? Here are some of my initial thoughts:
Physical access === game over
You have to be able to trust your service provider with your data. If that data has a cash equivalent of thousands of dollars then you have to be able to trust them a lot. There’s a special sort of service provider that we normally use for this – one that’s heavily regulated and where the customer (normally) gets reimbursed if mistakes are made – we call these banks. Of course regular banks haven’t got into servicing novel cryptocurrencies like bitcoin. Ian Grigg does a pretty good job of explaining why (and indeed why Bitcoin will slip into a sewer of criminality where this incident is but one example).
Bottom line – if you can’t trust the people that have physical access, then don’t do it.
Admin access === game over
If a service provider provides out of band management tools that provide the equivalent of physical access then you also need to trust whoever has access to that too. whilst it’s not clear yet who perpetrated the attack, it is pretty clear that it was done by subverting the management tools. This is particularly true when the management tool has direct control over security functionality, such as the ability to reset the root password.
In this case mileage may vary. Some VPS admin tools provide the ability to reset passwords, whilst others don’t.
Bottom line: Management tools might be convenient for service providers and their users, but can present a massive security back door for any measures taken on the machine itself.
Passwords === game over
If I look in the logs of any of my VPSs then I see a constant flood of password guessing attacks. This is why I either turn off passwords altogether with passwd -l account or disable password login in the SSH daemon.
It seems that at least some of the victims had chosen long, hard to guess (or brute force) passwords, which can raise the bar on how long an attack takes. Few people are that disciplined though. Passwords (on their own[2]) are evil, and should be avoided at all costs.
Of course SSH keys aren’t a panacea. Private keys need to be looked after very carefully.
Bottom line: Disable passwords, use SSH keys, look after the private key.
Conclusion
It seems to have become best practice in the IaaS business to build machines that only work with SSH keys, and where the management console (and the API under it) don’t have security features than can be used to subvert anything done to secure the machine[3]. These two steps go a long way towards ensuring that security in a VM/VPS has a sound foundation.
There is nothing that can be done about physical access (at least until homomorphic encryption becomes a reality) – so if you can’t trust your service provider (or at least get contractual recompense for any incident) then think again.
[1] Apart from some experimentation I pretty much never actually do much with any VPS that I run. They’re just used as end points for SSH and OpenVPN tunnels for when I want to swerve around some web filters (or keep my traffic from the prying eyes of those running WiFi).
[2] When I was running a VPS on Linode I took the precaution of adding two factor authentication (2FA) using Google Authenticator.
[3] Most management functions have serious implications for integrity and availability, but if they can’t hurt confidentiality then that’s a good start.
Filed under: security | 4 Comments
Tags: admin, Bitcoin, console, iaas, Linode, management, password, security, SSH, VM, VPS
Subscribe
Recent Comments
Tim Coote on Ross Anderson RIP iain on Skiing in Espace Killy (Val… Chris Swan on Getting more from a British Ga… Murray Cowell on Getting more from a British Ga… JL on AI MacGuffin 
Pinboard.in bookmarks- Open Source Security (OpenSSF) and OpenJS Foundations Issue Alert for Social Engineering Takeovers of Open Source Projects
- AI isn't useless. But is it worth it?
- Pluralistic: The true post-cyberpunk hero is a noir forensic accountant
- Subaru battery drain due to missing 3G network
- Paying maintainers: the HOWTO
- SARS-CoV-2 airborne infection probability estimated by using indoor carbon dioxide | Environmental Science and Pollution Research
- An unbiased evaluation of environment management and packaging tools
- Computer says no: British judge refuses to cancel divorce resulting from computer mistake
- Philosophy Museum
- Royal Navy offer must improve to attract and retain staff – you can't motivate an empty chair
Chris Swan's Weblog 
Hey Chris,
Great post. A couple of questions:
Given that some employees of all service providers have physical access to their kit, doesn’t that rule out using SPs for anything that is more valuable than the compensation terms? What has to be put in place to allow valuable data / services to be hosted in a 3rd party SP.
What do you think about generated passwords (from something like 1Password)? I suppose they typically aren’t as long as an SSH key, and therefore not as secure. A long enough random pwd should be ok against a brute force attack however, don’t you think?
K.
The answer to the first question is obvious, but uncomfortable – such things can only be made to work if there’s a contract with appropriate reparations for when things go wrong. This isn’t the case with the boilerplate for most ‘public’ services, which goes some way to explaining their price. Insurance from a 3rd party provides another option, and gets into some interesting questions around how an insurer would price the risk of a service provider (and assess the suitability of their control environment).
Just about the only way to make passwords reasonably secure is to generate them, and once generated it’s preferable to use an automated tool to help manage them. Such tools not only provide entropy, but also enforce discipline, so they’re a good thing. The attack surface for passwords is somewhat greater than SSH keys, but if we assume a worst case of compromised end points then I’d argue there isn’t much difference.
PS The security of SSH keys isn’t purely a function of their length. It’s entirely possible to get generated passwords that are longer than keys. The main difference is that the private key is used to sign a challenge, and hence never directly revealed to the outside world.
Some other comments over at Hacker News
You know I remember back in 2002 when I had a physical theft at my property and after three successful for me rounds of negotiation with the loss adjusters they agreed to the cGBP1k cost to retrieve the data destroyed on the hard drives that were left behind. Their argument for agreeing to pay out was no-one had added to contracts an explicit statement limiting liability for electronic data.
Roll forward a few years and most re/insurance contracts now exclude the area. Hosting companies by extension have followed a similar path, by that 1and1, MSFT’s office365, ectera. It should be no surprise that compensation will not be forthcoming, unless the percived reputational damage is viewed as limiting business growth.
As an aside, I would say if you ignore half the security requirements in your development (cost/time to build reasons) and you save costs on your support model (everyone in hosting) its not that surprising the results. I do remember amusingly telling Freeserve tech team back in the day I had uploaded a terminal emulator to my hosting (to see if it worked of course :) ), and had access to all their user account sites. They left that hole for months, I moved hosting.
A thought: Let’s assume the break glass/ privileged access model will fail (SSH or X in place) – how to still protect at a data level?
Perhaps: We can use a rsync type model to have chunks of a physical file moved between systems – following that if we only can take like a jigsaw, 33% and put on one hosting company; and same on two other companies (perhaps one a NAS), then any intrusion at one site gains nothing, unless they can compromise all three locations. And then they need to unencrypt the physical file assuming they know how to reorder the chunks.
fin