My heart bleeds for you (security-wise, anyway)

What is Heartbleed, and why should I care?

If you've paid any attention to tech news over the last few days, you may have heard of a serious vulnerability called Heartbleed. In a nutshell, this is a vulnerability found in OpenSSL. What's OpenSSL? It's the program used by many web servers to provide HTTPS access via Transport Layer Security (TLS, which we used to call SSL). In other words, when you open a browser and buy something on Amazon, or log into Google Apps, you're connecting to a web server that uses TLS.

Uh-oh, I use Amazon/Gmail/Facebook, do I have to worry?

The answer is, it depends! This doesn't affect your computer directly. But, it affects the servers that you connect to. It's possible that you could connect to a vulnerable server and enter your password, followed by an attacker connecting to that server and being able to read your password from the memory of the server! This appears to be pretty unlikely, but within the realm of possibility. If you've used these services lately, you might want to change your password as a prophylactic measure. Google has said that this isn't necessary for Gmail and Google Apps users, but it can't hurt - or can it? If a server was vulnerable, and that vulnerability led to another compromise, it's theoretically possible that the new exploit could capture your password even if the original one didn't! If you're feeling a bit woozy at this point, you're not alone.

Uh-oh, I run a web server myself! Do I have to worry?

Maybe. If your web server uses HTTPS, and relies on OpenSSL for this, and you're using OpenSSL 1.0.1a through 1.0.1f, you should definitely worry! These are the vulnerable versions of OpenSSL. If you're using one of those versions, you should upgrade to 1.0.1g if you're using precompiled binaries, or recompile OpenSSL. If you're recompiling it, you can either use the latest source or you can recompile your existing source with the OPENSSL_NO_HEARTBEATS flag.

Also, in theory, it's possible that an attacker was able to read some data on your server before you fixed it. This could be anything stored in memory: user passwords, database credentials, even the private key used by your TLS certificates! There isn't really a fix for that other than to change ... everything. Have fun!

What web servers rely on OpenSSL?

Pretty much everything except Microsoft Internet Information Server (IIS), as far as I can tell. Apache on Unix and Windows use OpenSSL. You can either upgrade to a newer version of Apache, or you can upgrade your OpenSSL libraries. It's pretty easy to do this on Windows, at least. You can download OpenSSL for Windows separately from Apache, and copy the DLLs and openssl.exe into the appropriate location within your Apache directory. You can identify the version number for the files in Windows Explorer by right-clicking and viewing Properties.

One of the two OpenSSL DLLs on Windows, showing the version number

The two DLLs on Windows are libeay32.dll and libssl32.dll - even apparently on 64-bit Windows.

You can test your web servers. There are a few online testing tools for Heartbleed. I've been using this one. There are also downloadable test tools suitable for use within private networks, as well, including the source for the one linked in the previous sentence, and this one. There's even a "bulk" testing tool, masscan, which can be used. This can actually be used to retrieve data from memory on a vulnerable server.

Wait, you mean only Microsoft IIS is immune to this?

Yes. Don't be smug, though. Just savor the moment quietly.

I'm running an SMTP/IMAP/SSL VPN/FTP/stunnel server. Do I have to worry?

Unfortunately, yes. Lots and lots of servers use OpenSSL for TLS, and TLS is used with lots of network protocols. Fortunately, most versions of OpenSSL aren't vulnerable - again, only 1.0.1a through 1.0.1f.

If you're using Adobe Connect and have it installed on your own network, you may be using stunnel with it to handle TLS. If so, you may need to replace the OpenSSL libraries there as well. This only applies to locally-installed Connect servers, not those hosted by Adobe or other vendors.

I'm using OpenSSL in a client to connect to another server. Do I need to do anything?

As far as I can tell, you are under no direct threat unless you connect to a server that is attacking you. But you can and should upgrade your copy of OpenSSL to prevent even that possibility. If you're using the free, open-source OpenVPN client, you can upgrade to the latest version and read about how the vulnerability affects OpenVPN. Similar upgrades are also available for other OpenSSL VPN clients like Viscosity and Tunnelblick.

[UPDATE - I've received additional information about client vulnerabilities from a security professional acquaintance - I am not a security professional, just a very scared system administrator - and it appears that clients can, in fact, be attacked via MITM - a compromised server is not necessary. So, at a minimum, update your client OpenSSL libraries too.]

I'm using a firewall/antispam/search appliance. Am I in trouble?

Many appliances that use OpenSSL are actually still using older versions like 1.0.0. Check with your vendor, though! For example, I work a lot with the Google Search Appliance, which can use TLS for serving secure results, access to the admin console, access to Version Manager, and feed submission. So, after reading about this vulnerability and then changing my underwear, I asked my Google support rep about it. I was relieved to learn that no supported versions have this vulnerability on ports that are sometimes exposed to untrusted networks - secure serve and admin console. That said, there is a vulnerability on another port that should never be exposed to an untrusted network, and Google will be releasing a patch shortly.

I'm using a cloud environment like Amazon AWS or Google Cloud Platform. What about me?

If you're running Amazon Linux AMIs 2013.03 or later, they have vulnerable versions. It's easy to upgrade them, though, using a simple yum update command and restarting affected services. The same is true for Google Compute Engine users. There's a vulnerability for Google Cloud SQL users, but you can block access to affected ports from untrusted networks - and you should be doing that already anyway.

My server was vulnerable. I've fixed it. But how do I know whether the vulnerability was exploited before I fixed it?

This is a good question! Unfortunately, I don't have an answer. I haven't found any way to identify this yet at the web server level.

Are there any lessons from all this?

While this is a very serious incident, I think these kinds of things are simply unavoidable. Systems are very complex and diverse, but they often share a lot of the same core components. But the same lessons we've learned in the past still apply here: defense in depth, regular password changes, proper network access controls, and so on, can mitigate the risk of these problems in many cases. Good luck, and be careful out there!

[Note: cross-posted on the Fig Leaf Software blog]