Wednesday, November 25, 2015

Server Died

My server (snowcloude.com, what the link on the right there points to) died about an hour ago.
Right now, it looks like a total loss, but we'll have to see.

I have a RAID5 in it, which we decided to go to a while ago, because they're more robust and we wanted to protect the data on the system. Well, checking it today, and one of the drives had died and  another one was throwing errors. So out to the store to buy two new drives.
We replaced the one that had died and let it rebuild, which it did just fine, and I guess it removed the bad sectors on the one throwing errors, because that one said it was now OK.

So, once that is all done we powered down the system, to put in a new backup drive (even with the raid we'd been doing backups, but the drive we had was no longer up to the task, so not everything was getting backed up). And apparently that killed the system. The raid is fried, apparently the raid card (very expensive raid card) decided to crap itself, and we've lost everything on the system. Some of it was backed up, but a lot of it was not.

I don't know when I'll get the system back up, we may be able to throw something temporary up for a while, but the server is 5 years old, so it may just be time to trash it and build a new one. I think if we do a raid again, we'll do a software raid, because then at least you're not spending a fortune on a fancy hardware raid controller, that only lasts a few years. Unlike the motherboard, which lasted a good long time.

EDIT: We were able to recover the system late last night. Still no idea why it crapped out on us. I immediately did a system wide backup to the new backup drive (now that we have one large enough). Not sure what the long term plan for that server is going to be right now though. Probably will have to start thinking about replacing it.

No comments:

Post a Comment

Please try to keep the comments clean and civil.