If we haven't written it, then it cannot be trusted. It really is that simple.

Today Adam send me a message asking if I knew why Apache wasn't running on Wheeljack, our backup and monitoring server. I checked it out, found a few unusual processed and Googled.

This is what I got.

It appears as though the hacker exploited Cacti, our graphing software. We use it to monitor network links and load, the other servers and anything that answers SNMP. It was installed as a Debian package from the Ubuntu Universe repository, so security patches are not maintained by the Ubuntu staff.

In the two and half years we've been running The Frontier Group we've had two hacks, and both have been through web applications that we've not written. The solution I think is to protect everything with a .htaccess file that won't let unauthorized people use these systems.

Thankfully the server doesn't do anything mission-critical yet (secondary mail and DNS, but they can/did both suffer a rebuild). It's a lesson learned, and hasn't caused anywhere near as much pain as Prime's hack did.