Loading ...
How to restore a hacked Linux server
Every sysadmin will try its best to secure the system/s he is managing. Hopefully you never had to restore your own system from a compromise and you will not have to do this in the future. Working on several projects to restore a compromised Linux system for various clients, I have developed a set of rules that others might find useful in similar situations. The type of hacks encountered can be very variate and you might see very different ones than the one I will present, or I have seen live, but even so, this rules might be used as a starting point to develop your own recovery plan.
In most cases if you have a system compromise at root level, you will hear that you have to fully reinstall the system and start fresh because it will be very hard to remove all the hidden files the attacker has placed on the system. This is completely true and if you can afford to do this then you should do it. Still even in this case the compromised system contains valuable information that can be used to understand the attack and prevent it in the future.
Here is a short overview of the steps that I will present:
- Don’t panic. Keep your calm and develop a plan of actions
- Disconnect the system from the network
- Discover the method used to compromise the system
- Stop all the attacker scripts and remove his files
- Restore not affected services
- Fix the problem that caused the compromise
- Restore the affected services
- Monitor the system
Don’t panic. Keep your calm and develop a plan of actions
Ok. You have just found out that you have to restore a hacked system. My first suggestion is to remain calm. Don’t rush and do something you will regret later. Why? Of course you will have to take action as soon as possible, but you can assume that the compromise is probably active for some time and if you act in second 1 or minute 10 this will probably not make much of a difference. If you have experience with such situations and have a proper plan in your mind, go for it, and don’t waste any time. If not, just relax, take 5 minutes to think about what you should do and how to solve this problem.
Example of bad actions during this time: you will rush and kill all the running scripts the attacker has launched and then have a timeout when you will think what to do next… In this time the attacker might see you have discovered him (for example from his irc bot, etc.) and might become upset and clean up the system for you…
Of course you should not go on with your planned trip and take care of this when you get back, I am just saying to use 5-10 minutes to think on this and develop a short action plan. There should be no timeout in your actions and you should always know what is the next step.
Disconnect the system from the network
This might not always be possible. But if you have physical access to the system or even if you are remotely on a system in a datacenter that provides a way to connect from a console (either a regular remote console, or a KVM, or a DRAC card in Dell servers, etc.), then this should be the next step. Connect to the remote console and bring down the network interface.
If you don’t have a remote console, here are some other ideas: you might be able to rent a KVM for a limited time from your datacenter, or you might have to write some iptables rules to block any kind of access besides your own IP.
After this your system will appear down to everyone, including the attacker that will see that the system is completely down.
Discover the method used to compromise the system
This step in my opinion is the most important one and you should not proceed any further until you have a proper answer to the question: “How did the attacker get in?” This step will be probably the most time consuming as the type of attacks can be quite variate. Still if you don’t find out how the attacker got in, then you will risk that you place the system online and he will be able to compromise the system in a matter of minutes. And this time he might not be so nice and you will not have anything to restore… So even if there is not a general method for this, here are some ideas to get you started:
Depending from what tools you have already configured, you need to identify the files uploaded on the system:
- if you have a system like tripwire configured use it to find out what files where added/changed.
- if you don’t have any such system installed, you might have to use the find command to search for the files newer than x days that were added to the system (also changed files in the respective interval).
Who owns the uploaded files?
- finding out the owner of the files will probably show you what application was used to get in. For example files uploaded as the web user will indicate that the web service was used to get in.
Investigate the uploaded files.
- the files that were uploaded on the system might provide valuable information about the attack. For example the attacker might use the same exploit to attack other systems from your compromised server. This can quickly show you what exploit he is using.
Get as much information from the running scripts launched by the attacker.
- as you have seen I have not recommended to stop yet the running scripts that the attacker might have launched. Why? Because they contain invaluable information to identify the attack. Use lsof on them (lsof -p PID) to see useful details. Where are they located? what user owns them? You might find the source of the attack from this information.
Use Rootkit detection tools.
- you might want to run some rootkit detections tools like rkhunter or chkrootkit to quickly identify common attacks.
Investigate system logs.
- with the information gathered by now you might reduce the size of the log information you have to investigate.
Hopefully you have found the source of the attack by now. Again this is very dependent of the type of attack. The most common one you might see these days is an exploit using a vulnerable web application and an attacker that will launch various scripts (irc bots, scanning tools, attacks on other systems, etc.). Still you might see something different like an attacker not launching any script, loading kernel modules to hide its tracks, to make it more difficult to identify or even see the compromise.
Stop all the attacker scripts and remove his files
Once you have identified the cause of the attack you can safely kill all the running scripts launched by the attacker and remove all his files (save them in a different location for further investigation). At this point we no longer need the scripts running as we got the information we could find from them. The system is still unavailable at this point and no service is available to the world.
Don’t forget to clean-up also the locations that the attacker might have entered to start his scripts on system reboot. Look in init scripts, rc.local, cron tabs for this.
Restore not affected services
Once we know the service used for the attack we can stop it and restore the network connection and all the unaffected services the system might provide. For example if the web server was used to get in, we can stop it, and restore other services like mail, dns, to minimize the downtime of the system.
Fix the problem that caused the compromise
Before starting the service used to compromise the system you should fix the problem so this will not happen again once the service is open to the public… Depending on the problem this might involve: patching a vulnerable daemon, upgrading a vulnerable web application (or temporary disable it), writing some special rulers to block it (for ex. mod_security rules might help in case of no patch available for a web application), etc.
Restore the affected services
Once you have fixed the problem you can restart the service used to get inside the system.
Monitor the system
Now is the time to monitor closely the system and see if the fix you have implemented is working. Most certainly the attacker will try to get in again as he will see he has ‘lost’ the system. If you notice any problem stop the service at once and reiterate again to the step to fix the problem (stop the service, fix it, restore it).
Conclusion
These steps are obviously not usable all the time because of the variety of attacks you might encounter, but they can be used as a baseline to develop your own plan of actions. Again in such situations, keep your calm, don’t rush, and work to restore the system based on a clear set of steps.
If you had similar experiences please feel free to share your own tips to help others that might find themselves in such a situation.
Share This
17th July 2006, 16:35
Way too much work - plus it’s not even close to being a guaranteed fix.
Scrub the system, reimage the system, restore the data from the last backup (you do rsync your data right?).
Then do the forensics to determine what wasn’t patched (you probably already know, you were just to lazy to fix it when the security alert was published) to allow what sort of exploit, fix it, then put it back online.
17th July 2006, 16:52
vonskippy,
Imagine that you have been hired to restore (or bring online a compromised system)… There are no backups… what can you do? blame the client and tell him that the only solution is to reimage the system? and try to get his data back online from his own local copies, etc.
“Then do the forensics to determine what wasn’t patched”… how? if you have just reimaged the system? You are seeing this as a local system where you can just add another HDD and start from backups and investigate the old one… Well most of the times this is not the case and the system is remote in a datacenter that will only help you by reinstalling on the same HDD over the existing data.
“you probably already know, you were just to lazy to fix it when the security alert was published”… this is not always true, imagine yourself working on such a project where you know that the web server was used to get in, but have to find out the particular application.. and there are 500 domains hosted on the server… you get the idea? Those are all user domains and the actual client has no idea on the particular uses of each domain or what applications are installed and used on the domains…
But hey, I respect your opinion and as I said in the beginning of my post there will be many peoples that will say this…
17th July 2006, 17:50
vonskippy… a quick and probably an amateur’s opinion !
Marius… a more mature and professional opinion !!!
I ‘ll stick with the second one…
18th July 2006, 06:13
How to restore a hacked Linux server…
Very helpful Yes, very helpful indeed. When dealing with a hacked Linux server, having a clear set of guidelines to help you along really does make things go a lot smoother I imagine…….
18th July 2006, 08:13
The problem is, if you not reinstall the system, you can never trust the system again. Check packetstorm or securityfocus, there you can find a lot of malware (kernelmodules, rootkits and other funny stuff). Even if you have tripwire on it, you never know. Espacially if the tripwire database is on a rw medium on the server. Every kiddy know how to use tripwire.
Without tripwire, just forget it. Make a image of the system, reinstall it and restore the latest safe backup. (to setup a random server with some standard services shouldn’t be a real problem for experienced sysadmin)
Another question is, how can you guarantee the customer that non of the 500 domains are back doored?
But I can be wrong and the incident handler is a experienced kernel hacker, reverse engineer, Apache committer, postfix hacker and tcp/ip guru. Some skills in asm could also be very helpful. Than, it’s no problem.
18th July 2006, 09:25
Sometimes the server uptime is really important, so usually I detach from LAN the compromised server then:
- boot with a livecd (like knoppix or FCCU GNU/Linux Forensic Bootable CD - http://www.lnx4n6.be/ )
- connect an USB/Firewire external disk
- make a dd of any partition (dd is a UNIX command useful for cloning disks/partitions) on the external drive
- reinstall the compromised server with a fresh and updated distro
- re-add services and configure them (maybe copying the config files form the dd images with the static data)
After these steps, you can have a clean&safe server online and you can begin to “waste” time with any forensic activity playing with the previous created dd images. IMHO this is the faster approach.
18th July 2006, 09:28
My humble contribution :
I’ve once had a computer compromized and found in interesting thing : chattr
My attacker saved the original files to a difficult do see folder ( a . space named folder) in /dev (no devfs by then).
His binary replacements where chattr’ed immutable (+i) so that I could not remove nor rename them even being root. I’ve spent some times figuring out how this tick was possible.
Since then, I systematically flag immutable all /lib /bin /sbin /usr/lib /usr/bin /usr/sbin directories, subdirectories and files of my servers. Some other critical files (some /etc ones) may also be flagged. It’s not the perfect protection, but it helps.
18th July 2006, 12:33
lagnis, I’ve got the same experience
1. Get the list of the immutable files
2. Remove immutable attribute (chattr)
3. Looked for all packages containing affected files from list - dpkg -S /sbin/xyz
4. Reinstalled those packages
Moreover I used “netstat -tunp” to detect hacker’s processes. Compromited computer was company’s NAT firewall with samba and maybe some unneccesary services. After everything had been restored, I configured the firewall, removed all unneccessary services and configured all internal services to listen on internal eth
18th July 2006, 13:09
[...] Dieser Artikel beschreibt, wie man eine gehackte Linux-Box wieder “säubert”. [...]
18th July 2006, 14:57
In your fine article you missed very important detail — log your actions. This is first thing you should do, i.e. run “script” and then investigate situation. ps/netstat/etc. results might be really handy later.
18th July 2006, 15:08
Definitely try to catch them in the act if you can afford the luxury of having a hacked system. Try to keep them from getting into anything else without tipping them off. Gather as much information as you can so you at least know what went wrong, if not who did it. Maybe quietly plant some fake warez/pr0n with trojans/virii or some fake business data they can waste their time with and/or try to sell and get caught that way.
After that, rebuilding is definitely the only way to go. Take off and nuke ‘em from orbit: it’s the only way to be sure.
Of course what you really want to do is prevent it from happening in the first place. Use virtual machines or zones with file sharing or zone rights to give your web server read-only access to content while your content management server (deep behind the firewall) has write access. That way they still may be able to put up a fake web site but you can just rebuild without having to restore your content. SMTP server is trickier - you don’t want people spoofing your e-mail or reading incoming/outgoing. Best bet is just use a secure MTA and a well-configured firewall, and use crypto on any important e-mail, so people you correspond with know that a cleartext e-mail is fake and the hackers can’t read anything important.
Use a DMZ - anything accessible from the outside should not be able to reach anything inside the firewall. Use OpenBSD for your firewall, NOT Linux and definitely not a Linksys or Netgear or other non-OpenBSD-based appliance. There really isn’t any excuse not to learn how to build a good OpenBSD based firewall.
If you can’t do these things, you shouldn’t be running your own servers. Just use GMail for Domains or FastMail, and some decent web hosting service like TextDrive or Pair Networks or whatever.
18th July 2006, 16:14
You forgot the missing step: Never trust the system again.
* 100% of everything on the system must be assumed to be compromised.
* While “cleaning it up”, assume the system is automatically healing itself.
* While “updating it”, assume the system is transparently undoing your fixes.
* Heck, when remotely connected.. assume 100% of everything you see is a little virtual machine that has nothing to do with the real system.
* While “monitoring it” assume that it’s spoofing all reports.
This article is a hack fix, which makes all sorts of leaps and assumptions.
The only way to determine what got hacked is to compare the system to backups. If you have backups, just back up and then blow the entire setup away and restore it from backups.
Get everything working from backups, update to (hopefully) plug holes and then gently restore data — ensuring that the data won’t assist in re-infection.
Then you can monitor and do remote write-only logging.
18th July 2006, 16:45
One more thing: change the root password.
18th July 2006, 18:31
I agree that the only way to get a system clean is to start over.
Like Sy Ali says, asume everything you see or do is virtual.
I also had a hacked system once - who hasn’t? (like marius said: “you probably already know, you were just to lazy to fix it when the security alert was publishedâ€â€¦) I needed webserver @ home for testing, and didn’t care much for updates, since i was in a hurry. one day later system was brute forced and a rootkit was installed, bugger. I just dd ‘ed everything to a spare disk - to learn from it - and did a reinstall. quick and “dirty”, but verry efficient. Only then you can be sure it’s clean.
Oh, and I did updates before putting the system online.
18th July 2006, 20:22
Restoring a Hacked Linux Server…
This is a great post at MDLog. Marius goes into great detail on how you should respond when you know a linux server has been compromised.
It’s an interesting read if you’ve ever had the opportunity to be the administrator of a linux server…
18th July 2006, 21:50
Stop scripts and remove files?!? That rootkit kernel module in your RAM? Hum. You mean those scripts you can still see, right? Can’t think of a reason why I would waste my time on such an insecure scratching on the outside of my system.
Paolo is totally right. Backup the compromised disks, do a clean reinstall of the software and restore data files from regular backups, in worst case from the backed up compromised filesystem images (not executables!). Make sure that your system is running latest updates. That’s, after all, the *only* way to make it almost possible to negate the possibility that there’s something left.
18th July 2006, 22:24
[…] Hopefully you never had to restore your own system from a compromise and you will not have to do this in the future. Working on several projects to restore a compromised Linux […]
19th July 2006, 02:16
[...]Paolo is totally right. Backup the compromised disks, do a clean reinstall of the software and restore data files from regular backups, in worst case from the backed up compromised filesystem images (not executables!). Make sure that your system is running latest updates. That’s, after all, the *only* way to make it almost possible to negate the possibility that there’s something left.[...]
Right, So How well do you think this will work if I image on a raid daily and oh I don’t know the system has been say comprimised for more then 48 hours. Doh! There went your all so grand Idea of restoring a backup and cleaning up the system. While I an agree a system resotre is a good Idea, Backups are not always an option and never will be. Im sorry but some of you guys really need to think outside the box.
19th July 2006, 03:55
[...] Read Full Story « | [...]
19th July 2006, 07:56
I would like to thank to all the peoples that have commented so far on this post.
Basically I would just like to outline some things already in my original post:
1. I never said that fully restoring a system is not the good idea. Actually I said that if you can afford to do this then you should reinstall the system and use the hacked one to identify the vulnerability used to protect the new system. If you don’t do this then you get in a cycle where the attacker will break in again and you start all over.
2. The type of attacks you might encounter are very variate and this is why there will never be a general global solution for such situations. This is why I have suggested taking this article as an example and developing your own rules.
And for the peoples that think that having a system uptodate with the latest security patches and feel secure… think again… we are living in the era of web applications attacks and even if you have your base system updated regularly you might have a surprise… You can have your system compromised even with an innocent installation of mambo for example (or phpmyadmin, or whatever) that you have been tested in an obscure folder and forgot about it…
This is basically learned from experiences (bad experiences normally) and this is why sharing each other experiences will hopefully help others.
19th July 2006, 16:02
nice read
19th July 2006, 16:20
[...] http://www.ducea.com/2006/07/17/how-to-restore-a-hacked-linux-server/ [...]
19th July 2006, 19:55
I agree with marvin. Use “script”.
IMO there is no substitute for a complete re-install. Feel free to shutdown and reboot off uncompromized media, like a knoppix cd, to backup/recover the latest data if necessary, but there’s no trusting a system that’s been rooted. It needs to be erased and re-installed. If you can’t get just your data from backups so you can re-install from scratch, then your backups are no good. If your backups are no good, boot from uncompromised media and start making yourself a good backup.
19th July 2006, 22:02
“I say we take off and nuke the entire site from orbit. It’s the only way to be sure.”
Ripley said it best in Aliens.
Also, you have to change passwords and cycle all security certificates. Phil mentioned changing the root password but you must also consider that the hacker could have captured any or all of your password stores and certificates and could be happily running offline attacks against them in the comfort of his own home. All passwords must be changed everywhere. Anything that was on that system should be considered compromised.
Easiest is to pull the drives, put in new ones. Then you don’t have to wait for dd to run. Mount the old drives read-only on another system so you get get all your config and data files that you can assure are not corrupted.
19th July 2006, 23:10
# Discover the method used to compromise the system
# Stop all the attacker scripts and remove his files
hmmm… not exactly sure how you do this when you’ve been rootkit’d
As Dom said, pull the drives, save the data and start over…. its the only way to know for sure.
The steps listed above may be fine for a non critical or home system; however, for any enterprise/critical system, you need to be sure.
20th July 2006, 03:28
whslddidtw…
mgsmnhze qrtewcfpu cjunttsfoc…
20th July 2006, 05:31
[...] read more | digg story [...]
20th July 2006, 07:56
Hi,
I’d personally always take the wipe and reinstall approach, and if necessary reuse data files from the compromised system (e.g. database data files etc). Knowing when the system was compromised can be hard - after all you can’t really trust any data or log files on the system once it has been ‘rooted’ and the high number of recent kernel level root exploits surely can’t help.
However, the article is useful, and I’m not sure I would have thought of doing an lsof before running “halt -f”.
Thanks
20th July 2006, 08:45
image, wipe, re-install.
This is the ONLY alternative, this article is another example of the piecemeal approach to technical writing that has overtaken the web. The only thing it illustrates is the authors lack of real experience in these scenarios.
Has Marius honestly never heard of LKM rootkits?
20th July 2006, 09:21
Weird there is not a single word about IDS (-;
20th July 2006, 11:40
[...] Da, tudi najboljÅ¡im se zgodi, da poÄepnejo pod silo vdora. Ne bom se spuÅ¡Äal v detajle kateri OS je najbolj varen, ampak vas bom kar lepo napotil do Älanka, kjer so izpostavljena dejsta, kaj nam je storiti, ko ugotovimo da je bil naÅ¡ Linux servah “shekan”. Sicer so v Älanku res samo smernice, a vendar par uporabnih namigov… Prijetno branje [...]
20th July 2006, 12:04
If there is no backup and you have to keep the hacked system around to recover the data, why not build a new machine and migrate the data over?
20th July 2006, 12:15
[...] Posted in Uncategorized, Coding, On the Intertron by will on July 20, 2006. This article - How to restore a hacked Linux server is being dugg quite a lot. [...]
20th July 2006, 14:53
[...] Link: How to restore hacked Linux server [...]
20th July 2006, 17:46
[...] Ali vam je že kdo namestil na kateri od strežnikov ti. RootKit. Navodila kaj naredit v tem primeru si lahko preberete v Älanku. S vsebino se jaz osebno ne strinjam, saj lahko veÄino najpopularnejših RootKitov odstranite roÄno, seveda za kaj takega rabite samo dovolj Äasa in dovolj znanja. [...]
20th July 2006, 22:51
[...] Gasp…Yes it CAN happen. Here’s what to do if it does…read more | digg story [...]
21st July 2006, 02:01
[...] Gasp…Yes it CAN happen. Here’s what to do if it does…read more | digg story [...]
22nd July 2006, 23:46
[...] How to restore a hacked Linux server | MDLog:/sysadmin (tags: hack linux security) [...]
25th July 2006, 07:28
[...] How to restore a hacked Linux server | MDLog:/sysadmin Tags: admin linux security [...]
26th July 2006, 08:17
Marius, you wrote:
“…you might have to write some iptables rules to block any kind of access besides your own IP.
After this your system will appear down to everyone, including the attacker that will see that the system is completely down.”
Could you please provide an example of such an iptables script. AFAIK, neither DROP nor REJECT targets make a system look as if it is down. They make protected ports look either “stealthed” or “closed”.
29th July 2006, 06:20
[...] How to restore a hacked Linux server | MDLog:/sysadminEvery sysadmin will try its best to secure the system/s he is managing. Hopefully you never had to restore your own system from a compromise and you will not have to do this in the future. [...]
3rd August 2006, 14:22
[...] Marius Ducea has a great article on How to restore a hacked Linux Server. He provides a very practical baseline on how you should develop your own plan of action to restore a hacked Linux Server. These are the steps he recommends: [...]
10th August 2006, 02:47
[...] Hopefully you never had to restore your own system from a compromise and you will not have to do this in the future. Working on several projects to restore a compromised Linux system for various clients, I have developed a set of rules that others might find useful in similar situations. The type of hacks encountered can be very variate and you might see very different ones than the one I will present, or I have seen live, but even so, this rules might be used as a starting point to develop your own recovery plan. [...]
1st September 2006, 12:57
[...] Gasp…Yes it CAN happen. Here’s what to do if it does…read more | digg story [...]
8th September 2006, 02:30
One issue seems to be when or if to do an image, reinstall, restore from backups. I’d say this depends on the version of the OS /apps and time.
If it is the latest version, then an image, reinstall, restore from backup may not fix things as you may have just reinstalled the hole again. So even if everything apears to be clean they could get in again. Also i’ve seen attackers exploits make it into the backups so they get restored anyway which looks very stupid.
If the OS and apps are not up to date then an image and reinstall straight away would be better as software can be upgraded to the latest version which gives the intruder more pain in finding an exploit to get back in. User data restored from backups will also have to be checked to see if it runs with the upgraded OS and apps. Which may help to identify if the exploit was in the user’s data rather than the OS/apps.
If time is an issue, then an image, reinstall, restore from backup is preferable however the ‘clean’ system may not last very long. In the long term its always better to find out the answer to ‘how did they get in?’
I like the ‘baseline’ idea, very helpfull
5th July 2007, 19:10
[...] Copié depuis le site: http://www.ducea.com/2006/07/17/how-to-restore-a-hacked-linux-server/ [...]
11th July 2007, 15:47
[...] BLOG 1 [...]
18th November 2007, 09:24
[...] to more secure ones. It may not be a complete security solution, here’s a few guidelines of how to restore a hacked system. No Comments, Comment or [...]
15th April 2008, 00:21
[...] find themselves in such a situation. thanks to this wonderful and straight to the point steps from Marius Ducea addthis_url = [...]