Or the story of a hack and the recovery of a compromised system. Namely server “Detroit” went down today at 4:06am throughout MAC poisoning, which was discovered by an IP Switch and therefore rejected(?). Sad, but, another journey of terror of support terror began…
Day 1
[9:00am] The server cannot be reached!
Are you sure? Damn, but I guess it’s their fault (once again). If you already had used 1&1 or former Schlund you only have two choices to convince them that they have an issue with their network and it’s not your fault.
- Their service team is separated from their tech team. So coordination is not as always as it should be. First choice is always to get the service contact overloaded by tech speak so he has to make a query to the tech team. Then they’re suddenly very cooperative.
- Second option is to tellthem it’s about a shop that makes a hel-load of orders and is going to be far under expected sales if they don’t fix the error.
Well, i catched up the syslog via hardware console and saw that the system is unable to bring up the network adapter (eth0), but booted normaly. And still i wasn’T able to connect via SSH. I phoned the support several times and about 3h later i was reported that the server poisoned the network with an invalid MAC address since 4:06. I wasn’t sure if they’re right – but probably it was a reason for the network adapter beeing rejected by their IP Switch.
[12:00] Tracing the intruder
Well, they told me our system is now seen as compromised and i will be only able to boot the rescue system for now. I tried this via their admin section on their website, but the option was not available. Therefore they system bootet the normal system isntead, which could also be seen over hardware console. Another phone call was made. After 1h i had the option and i finally was able to boot the system. It took another 2h until their automated system gave me finally a shell to the a rescue system. But i wasn’t able to login with the given password from the administration site. I tried the original password for the server and it suddenly gave me access. Ok, that’s really secure when the hacker already has decrypted the passwd from the old system, where i switched off pure passwd-auth (only RSA-keys).
Well, now i was finaly able to mount the drives and search for traces of the intruder. I didn’t find much in the logs, or something that really spoke for MAC poisoning, but i did find some w00tw00t.at log entries, an IP address from an AOL network and some local exploits in “/tmp/ /../”. Probably the addrsses were also spoofed and our little server was going to become a spambot or a proxy.
[17:00] Making a backup of the files
One thing at 1&1 is really nice: not having a good working solution for backup or instant recovery. Other hosters provide you with drive images from the last 3 days or automated backup systems or instant fallback image servers.
1&1 says this is not possible for root servers – hell knows why – and provide you instead with a restrictive FTP server with the same size as your server HD for two-times the money of other offers. Okay, that’s probably ok for diff-tarballs but it’s not a real deal when you want a mirror or you have to backup large amounts or big files. And it’s pain in the ass when you cannot stream to the FTP server, but have to pack a tarball first and then transfer it. That’s fine when the HD is almost full, und you only have the ftp, screen and vim command in the shell. Right, No Midnight Commander or tools to do instant batch processing of the files. I wouldn’t call it rescue system but minimal system instead.
Well, it was an 120GB partition with about 52GB free. I deleted some stuff, made some tarballs for the /var/logs, /etc /home and then transferred it to FTP. I was very happy to have enough free drive space to do that. And i did the backup to two times, one on another FTP Server – only to make sure they don’t reinitialize and kill the data an the FTP that’s in one contract/package with the Backup Server. Well that was about 50GB of data compressed to about 27GB.
[19:00] Go home and wait till tomorrow
The server was still transferring and since it was „only“ our development server which got compromised i could walk home without having to fear a work day till the full moon. The server still transferred some files…
Day 2
[9:00] Backup is made, please reinitialize the server!
The first thing i’ve done in morning – even before i got my coffee – was to call they support hotline again. The server is ready, please re-initailize it!
[11:00] I still can login to the rescue!
And again another phone call was made since no once actualy did something. Again, the same story had to be told (platinum service witrh different employees!), the tech team had to be queried once again (please hold the line for about 10min!), and finally i was asked which system i wanted – after telling them my contract and customer number for 10 times. I told them „Debian Etch Stable“, or „Debian 4.0“ or the „latest Debian available“.
“Thanks. You’re system will now be reinitialized with your desired system. It will be available in some hours, because they have to do it manually this time…” – *click* – “Manually?!”
[16:00] Manualy means 5h for installing Linux Debian and bringing up network access
I had meanwhile a closer look on the logs which i downloaded yesterday, ,did answer the phone two times and wondered if “hopefully online again tomorrow evening” was the wrong sentence in the mails i sent toyour developers yesterday evening.
Meanwhile the server could be reached for short time, there was a shell to login (but i had no passwords), and then again the server was off the line, and then it showed “ready” with bold text “SUSE Linux 9.3 (PLESK 7.3)” on their recovery admin site.
Still I cannot ping or connect to the system. And i wonder how long it takes to reintialize the server once again with Debian…
[17:15] Compromised system released again
Ok, although it’s against their policy they did release the compromised system(!). And it still runs. Yet it was not blocked from the IP switch and no rootkit was found Still i wonder what data the guy leeched, or if he was even able to download something and how he got into the system. I guess he used the server as proxy (
Anyway, vnstat reports about tx 3,971 MB, rx 2,517 MB, total 6,489 MB, wheras the server only does about 500MB total per day. I forgot to check that and unless vnstat only does 24h i have no way to revert that data.
And some are going crazy here about mailing about 20 new passwords, whereas i have to set about 200 new passwords and copy and unpack about 40GB of data.
Resumé of the hack
Still the biggest problems with 1&1 are bureaucrazy and strict information policy as well as an overloaded network that has really amazing latency.
Anyway, it seems the intruder got into the server over a PHP upload exploit, but could not much do inside /tmp. Still, he could mainipulte the network adapter (probably through a local buffer overflow?).
He began scanning for security holes at 1:36am and was blocked by the intrusion detection system about 4:06am.
I guess all relevant data can be restored and nothing was lost. But what was once compromised stays compromised. I looked for file changes in the application files online but didn’t find anything – hopefully we’re safe again.
But Platinum service at 1&1 only means you get a ticket number which enables you to ask how far your query has progressed. Ah, and well the service employees are not the dumbest in the firm. That’s all mystery behind it! Forget 1&1! They suck. And the decision for 1&1 was not my choice…
