Crack Attack

Posted January 30, 2005

At around 4:00AM this morning Prime was attacked.

We traced the source of the attack to a bug in awstats. We use(d) awstats to display usage statistics for our customers.

The exploit was used to install this root kit. Because of the firewall we have in place, nothing too bad has resulted. We are treating the machine as tainted, and will be replacing it over the coming days.

I expect that a web crawler first searched for domains that had awstats installed on them (by checking for http://domain.com/cgi-bin/awstats.pl). Once a target machine was found, the command was passed to awstats to download and install the root kit. It also appears that the perl file (awstats.pl) was removed from the system, which has prevented anyone else getting into the machine the same way. How thoughtful...

The NetFlow data that Jono collects from his Cisco router may tell us where the attack came from, but only if it was outside of WAIX. He uses this for billing purposes, and WAIX traffic is free. My bet is that it was a hacked machine in China or Russia that was being used as a zombie to attack others.

I'm glad that it wasn't a personal attack, because we could have had much more dangerous repercussions (lost client data, locked out of our own machine, stolen passwords).

If you have awstats on your machines, you'd better make sure that it's v6.3 or later!

Edit: 30/01/2005 - 4:02PM
Things we learnt:

  • The attack happened at 4:03AM, but as chance would have it, Jono's syslog server ran out of diskspace around that time, so the NetFlow data was lost! Argh.

  • Remote syslogging is a good idea. syslog-ng is what I need to investigate.
  • chkrootkit is great. It checks for root kits, as the name would suggest, and helped us uncover a few files that we didn't know were messed with.
  • Having a backup copy of your system and data is essential. Whilst we haven't bare-metal restored, we could get known good copies of /bin/ls and the like.
  • chattr and lsattr are good for finding more out about a file's attributes.

I'm glad the lessons haven't been too expensive; it's meant that we've needed to fast-track some of the plans we had for moving services off to other machines, and getting some equipment out at Steve's.

3 Comments so far

  1. Hale on January 31st, 2005

    Sorry to hear it guys. It’s a bitch when it happens.

    UpMyStreet does remote syslogging - there is an ancient Ultra 10 running Solaris 7 doing all the logs, and thats all it does. You guys could probably stream updates home, or to a spare machine in the office.

    If you are worried about awstats, try Webalizer, its a little old but it does most of what awstats does.

  2. Aaron on January 31st, 2005

    As per usual you only really kick security into gear when it happens to you :P

    Glad to hear that nothing major happened.

    Are you able to setup VPN connection for stats viewing (or any outside access)? Or have stats running on a seperate server to DB? Another level of protection.

    Aaron

  3. mlambie on January 31st, 2005

    I’m glad it wasn’t really an oversite on our part, if you know what I mean… it would have been much worse if it was a personal attack, or through software we’d built.

    We can/will still use awstats; it’ll be secured differently behind a customer section in future, rather than relying on awstats to secure itself (which it did, through a .htaccess file).