SKID LEADER Wrote:What a day this is turning out to be.

Quote:Back in February 2011 we were shelled. Yes that's right. We were shelled.

Quote:I became aware of the shells weeks ago while reviewing server files. It was a very alarming moment of course. Site was shut down for "maintenance" and I started reviewing logs and other forensics were done. Ultimately I found out who uploaded the shells but their English was terrible and they didn't really provide me much info.

All evidence points to a XSS vulnerability in modcp that is now patched.

However the real question remained. Did they grab the database? I wasn't sure. They weren't saying. It did not seem appropriate to alarm the member base without knowing more information. We have too many members to just change all passwords.

Today a member is boasting about having access to the DB. So someone or some group may be releasing it public. After reviewing data given I've been convinced it's valid and it's from the time period I know we were shelled.

If you'd like to change your password or email now is the time to do it. While I'm firmly convinced accounts are safe that doesn't mean you're not exposed somewhere else.

Quote:I arrived home at 10:30 PM and server was offline.

I contacted datacenter because server wasn't responsive. After 2 hours they get server back online.

During that time I decided to restart mysql server since it's been running about a month. When server comes back MySQL is hosed. I had not noticed it didn't successfully reboot.

MySQL if sucked. The INNODB ibdata file appears corrupted and so far I've not been able to figure out how to recover. I'm going to assume I've not yet mastered something and there is a hole in my knowledge.

After 2-3 hours of crap trying to recover the database I gave up. Database rollback has taken place and we're forced to use a DB backup from about 20 hours ago.

All posts, PMs, and users from the past 24 hours are basically gone. Be aware orders are done off site so anyone buying l33t/ub3r will get upgraded without problem. Unless you only signed up yesterday and immediately bought l33t. But I will see that no uid exists and contact you via email.

This totally sucks and it's stressful. Toying with the my.cnf we may not have the most optimal settings either. So DB could be slow and produce errors until I can take more time to fine tune it.

Quote:It's probably that the past 16 hours of hell was because of faulty server RAM. Logs showed numerous errors related to memory which could have caused the problems with the server going offline.

Thanks to the datacenter for quickly swapping out the RAM once I pointed out the log error messages.

This still doesn't make me feel any better about the rollback. These two incidents were in many ways separate even though one problem led to the other.

Had all this downtime occurred without the rollback I'd be feeling awesome right now that we're back online. But as it is I'm simply grateful we are online.

So let's get back to our routines and move on. I still have a day of work ahead of me. Praying these problems are past us now.

Quote:2. Post topics in their correct forum category. Read each category and it's description before you begin making threads.

(11-05-2011, 07:49 PM)Mark Wrote: Dammit.

Quote:2. Post topics in their correct forum category. Read each category and it's description before you begin making threads.

I blame the mod for noticing that it was in the wrong place and not moving it! *phew* that was a close call.