Webfaction
A few years ago, Webfaction had been offering
something somewhat unique on the market. Hosting by developers, for developers,
for less than 10 USD a month. For that price I could get SSH access to a shared
machine, deploy almost whatever I wanted and get it served by nginx, fast. This
was unique: at that time you either had to pay for pricier dedicated machines to
get SSH access, or downgrade to simple already-configured bricks that had very
little flexibility.
I particularly enjoyed the idea that I could push-over-SSH my content via my
version control system, and get it deployed via a hook.
Adopting Webfaction was a pretty great experience. Ticket support a few years
ago was responsive and helpful. A much better experience than what I had hoped
for.
First they came for Let’s Encrypt
And then Let’s Encrypt happened. Free SSL certificates, I thought, great!
You see, I’m not great at it yet, but I do care about security, and in that
great idea. I jumped in, and moved this modest website to https. respect an
opportunity to participate and help spread SSL further seemed like a
This is where trouble started. Webfaction has no first-party support for SSL
certificates. You’re not root, they do not offer direct access to nginx
configuration files. Which means that the only way to deploy a certificate is to
bother a human via a ticket. And since Let’s Encrypt certificates expire every
responsive enough, and I had a script automating renewal and ticket email. But
90 days, it means bothering a human every ~2 months. It’s fine, support is for a
coder, it does give you a bad conscience.
After a while (I suppose because support started receiving way too many
tickets), they implemented a way via their webadmin dashboard, to upload your
own certificates. So instead of my script bothering a human every 2 months, I
had to upload, by hand, via a web form, new certificates every 2 months. Not
really an improvement.
Webfaction always claimed that they had support coming for full Let’s Encrypt
automation, but a year after the CA launch, nothing had landed.
Then they came for gzip
BREACH happened.
tl;dr: this is a compression side-channel attack. For HTTPS websites that
(a) do use compression, (b) include query data in the response, and (c) serve
some kind of secret, bad guys might be able to guess what the secret is, by
repeatedly issuing requests, with various payloads, and observing how the
compressed (encrypted) response changes.
It’s a serious class of attack. Any website serving secrets should care and
implement counter-measures. But if you’re serious about security (and you should
be, if you’re serving secrets over the web, right?), I do hope that way before
the publication of this class of attacks, you were already implementing some, if
not, all, of the following countermeasures:
- Try really hard to avoid responses that do include query data, or responses
that depend deterministically on user input. Because any website
allowing such thing probably has way too many XSS vulnerabilities
lurking. Hello
?username=alert(1);.
- Defend against XSRF issues, using CSRF tokens. For instance, on login pages,
it’s common for servers to return a single-use random CSRF token/nonce
that the client must submit along their request, to prevent
repeatability.
- Randomize response length and content.
- Rate-limiting requests.
All of those basic security measures prevent repeatability, and randomize
responses, which, de facto, prevent exploiting BREACH class of attacks.
Unfortunately, Webfaction had a very different response to BREACH, possibly
because they are not security experts. Or more likely because customers that
were at-least-as-ignorant-of-security asked for that specific change: they
disabled gzip compression for all their SSL websites.
That’s quite a pity. Sure, it’s a safe, large hammer to use which appears to
immediately hide away all problems. But websites written with poor security
practices are still vulnerable websites, very likely prone to XSS or CSRF
attacks. Disabling gzip as a way to save them from BREACH is a poor decision,
not improving web security as a whole.
Webfaction is working on nginx config fragment support, to allow users to e.g.
enable compression per-website. But as of writing, there’s no ETA for that
feature, and their support denied manual overrides to configs. No way to
re-enable gzip compression on your website.
Time to speak out?
Do I really care that much about running a command and submitting a form every 2
months to renew a certificate? Do I even need compression that bad on this
static website? No, and no. Definitely not that critical.
But I reached a clear, fundamental disagreement with the way this host is
running their services. I care enough about SSL to require first-party,
automated support for certificate changes. And the way to secure websites, is
to, duh actually fix the root cause, not disable the perceived cause of a
vulnerability. Sending more bytes over the wire by disabling compression does
not protect websites from basic XSS/CSRF issues.
And why won’t you just let me customize my nginx configuration, urgh.
Digital Ocean
I moved out. Looking around, Digital Ocean
seemed like a great option. 5 USD a month, and in a few seconds I could create
an instance running a clean OS. SH access, root. I get to run whatever I want.
Switch gzip on, and off, and back on, as much as I want.
In a couple of hours, on a lazy Sunday, I migrated my data from Webfaction to
Digital Ocean. Configuring nginx and auto-renewal for Let’s Encrypt was easier
than Hello World in Haskell, thanks to their community-maintained guides that
even sysadmin dummies like me can follow.
Looking back, it’s even somewhat strange. How did I survive not having full
access to the machine before?
Hello Digital Ocean, and thanks: I, for one, embrace change.