Topic: Server Administration Lessons Learned

[Trampas]

I am not a server admin, but I do admin a e-smith box for my company. I noticed the new 5.6 and was thinking about upgrading. As I was thinking about it I came up with a list of things I wish I had done to make my my life a bit easier. I thought it might be nice to create a thread of such lessons learned to help thouse like me in the future.

Here is my short list:
1) Backup! Backup! Backup!
I use a hardware RAID 0 with hotswap harddrives, http://www.synetic.net/Synetic-Products/ARAID99-1000.htm. This product with multiple harddrives is much cheaper than a tape backup and if a drive fails your server does not go down, I have had two IBM drive failures and zero down time!  Also you can replace an old hard drive with a larger one, then repartition to add disk space. The bad part is to get data from a backup you have to boot from the backup or see #3 in the list. Rember a backup system needs to: Provide recovery from hardware failure, provide recovery from fire or theft, and provide recovery from "rm *.*", or hacks.
Think about if your server was hacked last week and you noticed it this week, can you recover to a pre hacked version?

2) Get a note book and write down everything you do to your server with the date and time. Also place CD copies of software scripts, etc that are installed with the notebook. When you have to upgrade the server and have to remeber what you did to get your dynamic DNS working before you will be thankful for the notebook.

3) If you are using your server in a production environment, get another machine that is excatly like your server to use as a test bed and hardware backup if the main server quits. The idea is that hardware is much cheaper that having network down for a day. This especially true if you are using the RAID 0 back up method, in which case it is hard to reinstall and restore from backup. Remeber hardware will fail, the only question is when.

Does others have anything to add to help first time network admins? I wonder if e-smith could make a page for us first time admins with some nice lessons and such?
Trampas

[Kelvin]

Hi Trampas,

I just though I'd throw in my 2 bits worth.

I think you meant RAID 1. RAID 0 from memory is striping and cannot have a disk removed without TOTAL loss of data.

Also, a word of caution about using hotswap RAID-1 systems for backup (especially for SME versions prior to 5.6). If you removed a drive while in operation to be kept as a backup, remember that :-
a) SME treats the drive as not been shut down correctly and chances are will not boot up straight away as it may contain orphaned inodes, etc. which needs to be corrected (if it is correctable at all) by using fsck.
b) If you use applications that run off MySQL, remember that the databases are normally open all the time and thus you run the risk of leaving the databases in an unknown state when you just pull out a drive during the day.

The only safe way I know of treating the hotswap RAID-1 systems as a backup device is to shutdown the server before removing the backup disk. Most of these systems would probably require you to power on the server without the second hdd in place and then only insert the next backup hdd after the unit has initialised and realised there is a drive missing (or run the very real risk of totally screwing up your hdds !).

Kelvin

[Trampas]

Yea you are correct it is a RAID 1 system, my goof....

I have tried booting the system from one of the backup drives and it is just like you have improperly shut down and requires the fsck. I could not figure out a way around this problem using the RAID 1 setup and figured it would be no worse than losing power. How does the server backup up to workstation allow for open files?
Maybe I should run add a script to back-up the server to a drive on the server, thus this file will always be closed?

That is another good lesson to learn, test your backups regularly I worked for a multi-million dollar company that lost a hard drive on a server and lost months of work because no one ever thought to test their back ups.

Thanks
Trampas

[Kelvin]

Hi Trampas,

Personally, I don't recommend clients to use this method for backups. As a standby in case of total failure / theft / fire /etc. of the server or drives, yes but as a backup no. It is actually quite troublesome to just "restore" one file from backup using this method and if you are talking about non-technical end users, you also run the risk that they might miss a step in swapping drives and render one or all drives in the backups and the live hdd totally useless. But some prefer it, so to each their own....

When you backup to desktop or backup to tape, a script runs which exports the databases out to a dump file. It is this dump file which is backed up, not the running databases themselves.

There is no way around the problem of the server detecting that the hdd is not properly shutdown if you just pulled one out without powering down first.


Kelvin