Loading ...
ssh-vulnkey -a
Yesterday, 13 May 2008, was a really bad day for the Debian project, probably one of the worst days in the history of Debian. Luciano Bello discovered that the random number generator in Debian’s openssl package is predictable. This is caused by an incorrect Debian-specific change to the openssl package (CVE-2008-0166). As a result, cryptographic key material may be guessable.
Systems which are running any of the following releases are affected :
- Debian 4.0 (etch):
http://www.debian.org/security/2008/dsa-1571 - Ubuntu 7.04 (Feisty), 7.10 (Gutsy), 8.04 LTS (Hardy):
http://www.ubuntu.com/usn/usn-612-1 - and other debian etch based distros.
What does this mean? It means that a Debian specific patch that was trying to fix vulnerability added another one in the openssl package, that doesn’t exist in the mainstream openssl code. Now this has caused some ‘interesting’ discussions about who’s responsibility should be on patching software.
Once something like this happens it is normal that developers will say that they should be the ones to approve and fix bugs. But on the other hand as experience has shown they are in general not able to do them fast enough, so it understandable and laudable that package maintainers and security groups of the linux distributions fix these themselves sometimes in a matter of hours. Still they should submit their patches upstream and see them included in the software helping like that the wider community of peoples using that software.
Anyway, regardless of what and how this happened it puzzled me that such a flaw was not observed for such a long time… 2 years. Unbelievable.
Enough with the talking, and let’s get back to work. How do we fix this? Unfortunately it is not enough to just upgrade openssl to the fixed version as we also have to regenerate any key material which has been generated by OpenSSL versions starting with 0.9.8c-1 on Debian systems. Affected keys include SSH keys, OpenVPN keys, DNSSEC keys, and key material for use in X.509 certificates and session keys used in SSL/TLS connections. Keys generated with GnuPG or GNUTLS are not affected, though.
Probably the most common keys that sysadmins will be working to regenerate are ssh keys (host, and user keys also). An upgrade of openssh is available from debian also: ssh_4.3p2-9etch1 that has a dependency on the openssl update, but will also install a new package openssh-blacklist that is intended to help peoples solve this issue. During the upgrade of the openssh package this will show a warning like the one bellow and will regenerate the HOST ssh keys of the system automatically (/etc/ssh ssh_host_dsa_key and ssh_host_rsa_key) and save the original ones as *.broken:
Vulnerable host keys will be regenerated
Some of the OpenSSH server host keys on this system were generated with a version of OpenSSL that had a broken random number generator. As a result, these host keys are from a well-known set, are subject to brute-force attacks, and must be regenerated.
Users of this system should be informed of this change, as they will be prompted about the host key change the next time they log in. Use 'ssh-keygen -l -f HOST_KEY_FILE' after the upgrade has changed to print the fingerprints of the new host keys.
The affected host keys are:
/etc/ssh/ssh_host_rsa_key /etc/ssh/ssh_host_dsa_key
User keys may also be affected by this problem. The 'ssh-vulnkey' command may be used as a partial test for this. See /usr/share/doc/openssh-server/README.compromised-keys.gz for more details.
Once this will be completed you need to keep in mind:
- to update OpenSSH known_hosts files: as after any key regeneration, without updating known_hosts file ssh will show a warning or an error like:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.- check and regenerate all OpenSSH user keys: this can be quite a hard thing to do depending on the size of the user base, keys locations, etc. The debian team packed a tool called ssh-vulnkey to help identify and locate the keys that need to be regenerated on the system.
To check all your keys in the standard locations (~/.ssh/id_rsa, ~/.ssh/id_dsa, or ~/.ssh/identity) for all your system run:
sudo ssh-vulnkey -a
To check a key in a non-standard location:
ssh-vulnkey /path/to/key
Regenerate any affected user keys (ssh-keygen) and update authorized_keys files (if necessary).
Warning: once you install this openssh update it will refuse to authorize any keys that are weak:
May 14 16:03:56 srv sshd[4231]: Public key xxx blacklisted (see ssh-vulnkey(1))
so you will probably want to fix the keys asap. Good luck and hopefully this post has helped you getting started and having this issue fixed on your systems.
Tags: debian-etch, openssl, ssh
Share This
15th May 2008, 07:09
Just a short note: it’s not that the Debian patch was trying to fix vulnerability in openssl. It was trying to remove the use of uninitialized memory. Which could have been easilly accomplished with -DPURIFY.
Please refer to Ben article you have linked to above, as well as http://www.links.org/?p=328.
15th May 2008, 10:17
* openssh-blacklist was not installed automatically on the update of ssh
* ssh-vulnkey is not found on my system.
This is on etch. No idea what to do on lenny.
# dpkg -l *ssh* | grep ^i
ii openssh-blacklist 0.1.1 list of blacklisted OpenSSH RSA and DSA keys
ii openssh-client 4.3p2-9 Secure shell client, an rlogin/rsh/rcp repla
ii openssh-server 4.3p2-9 Secure shell server, an rshd replacement
ii ssh 4.3p2-9etch1 Secure shell client and server (transitional
15th May 2008, 11:15
e.r.: your openssh-client/server packages are not upgraded to the version 4.3p2-9etch1 (as ssh for ex.); this means they are probably kept back. Try to force the upgrade with:
apt-get install openssh-client openssh-serverAbout lenny, this will receive a separate upgrade for this issue. Patched ssh version for lenny is 4.7p1-9.
16th May 2008, 02:32
[...] ssh-vulnkey -a | MDLog:/sysadmin Yesterday, 13 May 2008, was a really bad day for the Debian project, probably one of the worst days in the history of Debian. (tags: debian linux software security) - Posted in del.icio.us by del.icio.us trackback [...]
17th May 2008, 10:53
Hi Marius, many thanks for the very useful post. I updated openssl and thought it was enough, but luckily I am subscribed to your blog and noticed this post
One question please: I maintain some self-signed ssl certificates (using TinyCA if that matters). It is enough to regenerate the certificates, or do I need to re-generate the root CA cert as well?
18th May 2008, 20:45
Tony: I am happy you found my post useful. Regarding your question, if I were you I would regenerate the root CA also. Debian team is updating very quickly some pages where you can find infos targeted to individual programs:
http://www.debian.org/security/key-rollover/
http://wiki.debian.org/SSLkeys
21st May 2008, 07:50
thanks Marius, I did as you recommended and instead I feel much better now after all ssh/ssl keys and certificates were updated, though it was a tough job
21st May 2008, 08:42
Tony: I hear you. It was a tough week for me also ;-).