The Hacked Server Hangover

Every Server Admin managing a shared hosting environment knows what the hacked server hangover is like:
– it is painful
– it is frustrating, because it happens despite the strict security you already put in place, already having an antivirus in place, strict firewall rules, upgraded services…
and
– it almost always starts with the same – boring – story
“once upon a time one of your users installed WordPress and forgot about it”, leaving it in a corner of his space and letting it get to a point where it’s several versions older than the current releases -> full of security holes.
I could talk for much longer about these CMSes, Not only WordPress, but basically the security holes are more often originating from the third party templates, plugins, or custom developments rather than the WordPress or other CMSes themselves. These old unmaintained installations make the ideal ground for all the exploit scanners that intruders/hackers are using to get access to your servers. Most of the time a malware has been injected after a series of XSS (Cross-site scripting) attacks or SQL injection exploits in the user’s folder.
The pain is propagating as far as it can to suck out all the resources it can take from your server, in some extreme cases you could also lose access to your own server… Generally, the first symptoms will come with an excess of emails in the queue – thousaaaaaaands – of garbage emails delivering the best offers to the whole world, from your shared server IP -> disaster, in this case where the server is used as a spamming resource, you will also find out that the infected user’s websites have been defaced and are now used to relay the spam sent and host temporary phishing pages to try to steal people’s sensitive data (generally banking details).
This also has a few exciting side effects like exposing your server to high CPU load, leading to such poor performance that you almost cannot SSH-in anymore.

It’s bad times and the fun has just started but action has to be taken as quickly as possible because you are in a very bad spiral of events; next steps are your IP being reported to the DNSBL (DNS Black Listing) -> server blacklisted -> Ip reputation downgraded – break down.

All you need is a hug, a fast, efficient painkiller and a good way of preventing it from ever happening again.

You can ask around for the hug, if you are not spending too much time on your servers I’m sure you’ll find a friend – easy.

I can provide painkillers :

Linux Malware Detect (LMD) is a malware scanner that detects and removes malware on your Linux server. It says it all. It is particulary great for webservers in shared hosting environment and this is LMD’s main focus.

The threat landscape in shared hosted environments is different from the scenario typically addressed by the standard Anti Virus products. These detect primarily OS level trojans, rootkits and traditional file-infecting viruses but are quite poor at detecting the increasing variety of malware at the user account level which serves as an attack platform.

Linux Malet uses MD5 file hashes and HEX pattern matches as signatures.
These signatures are built from malware data that was delivered by IPSes, community data, ClamAV and user submissions.

I have implemented LMD on every Linux Webserver that I install, even my own laptop:

let’s take it onboard.

Please note that you need a linux server, the root access and a bit of paranoia – also in case you are wondering, LMD is already watching all our On-Rev servers.

1) Download the latest LMD package.

# cd /tmp
# wget http://www.rfxn.com/downloads/maldetect-current.tar.gz

2) Install it

# tar xfz maldetect-current.tar.gz
# cd maldetect-*
# ./install.sh

3) configure it

(every param in the config file is commented)

# nano /usr/local/maldetect/conf.maldet

here's a sample file

# [ EMAIL ALERTS ]
##
# The default email alert toggle
# [0 = disabled, 1 = enabled]
email_alert=1

# The subject line for email alerts
email_subj="maldet alert from $(hostname)"

# The destination addresses for email alerts
# [ values are comma (,) spaced ]
email_addr="you@domain.com"

# Ignore e-mail alerts for reports in which all hits have been cleaned.
# This is ideal on very busy servers where cleaned hits can drown out
# other more actionable reports.
email_ignore_clean=0

##
# [ QUARANTINE OPTIONS ]
##
# The default quarantine action for malware hits
# [0 = alert only, 1 = move to quarantine & alert]
quar_hits=1

# Try to clean string based malware injections
# [NOTE: quar_hits=1 required]
# [0 = disabled, 1 = clean]
quar_clean=1

# The default suspend action for users wih hits
# Cpanel suspend or set shell /bin/false on non-Cpanel
# [NOTE: quar_hits=1 required]
# [0 = disabled, 1 = suspend account]
quar_susp=0
# minimum userid that can be suspended
quar_susp_minuid=500

4) Manual Scans and Usage

You may just wanna scan the “suspect” user, so that’d be the command you’d want to run.

# maldet --scan-all /home/suspect/

The following command is good to know too, If you previously ran a scan but forgot to turn on the quarantine option, you still can quarantine or clean all previous malware scan results using their SCANID :

# maldet --quarantine SCANID

OR

# maldet --clean SCANID

5) Daily Scans

By default LMD script is located here /etc/cron.daily/maldet and it performs a daily scans, update of signatures, quarantine etc, and sends a daily report of malware scan to your specified emails. The apache folders are already listed but if you need to add additional paths to be scanned, then you will need to add them here:

# nano /etc/cron.daily/maldet

Voila, after having cured a few servers and saved -a lot- of time using LMD I felt I had to tell you about it, hoping it will be as useful to some of you as it has been to me.

Finally I would like to add a little something about WordPress, I like wordpress and the only reason I have mentioned it in our catastrophe scenario above is because this is by far the most popular CMS and so you get a higher probability to find out it has been the gateway of some hack, just because of it’s popular. This does not mean that WordPress is less secure than other CMSes like Joomla or Drupal. As long as you keep your WordPress up to date and respect a few things you’re ok, I could eventually detail these rules in a future blog post.