servers

Last night we upped the RAM count to 6GB (from 2GB).  I'm pretty sure that was completely unnecessary, but it is still sweet.  There are some weird things you notice when you get that much RAM: the pagefile is reported by Task Manager to be over 12GB, and the hibernate file takes up over 6GB.  Hopefully it doesn't actually think it's going to be using that hiberfile anytime soon--it would be tough to check my e-mail.  :)

It's also now running a Radeon HD 3850 512MB graphics card, but the purpose of that is to test how good of an output we can get on a TV through component outputs, and it will soon be replaced with a lower end Radeon HD 3450 that is passively cooled.  Then our poor little server will have yet another job: HTPC.

Well I wouldn't say we are experts, but we definitely did have a near disaster.

It started 3 days ago when I installed an NVIDIA driver and program to monitor the RAID 5 running on our server's M2N-E motherboard.  It reported to me that the array was in a degraded state, but did not allow for repairing it from within Windows, requiring some BIOS-level maintenance.  So I planned to go investigate the situation the next day.

When I got to the server and hooked up a monitor and keyboard the video output was garbled.  We could tell it was going into BIOS but it was unreadable.  That's what we get for using a 10 year old PCI video card I guess.  So I planned to come back in a day or two with a different video card.

Of course it then decided to drop another drive from the array.  This took down our website because it was stored on the array.  By website, I mean Default Website, which has the misfortune of being linked to Outlook Web Access for Exchange 2007.  Which means the web.config no longer existed.  OWA would no longer run, and reportedly the only way to fix that is to do a complete re-install of IIS and Exchange 2007.

Needless to say, we really wanted to get that RAID going again.  We rebooted between the RAID BIOS and Windows many times trying to rebuild it, but it would never show up in Windows.  After 10 fruitless repetitions of this we about ready to call it a loss and get ready to wipe the array and start over.

But then I thought of how the monitoring driver and utility I installed had been a relatively new version, and maybe it had incompatibilities with the older BIOS.  Updating the BIOS was worth a shot.

We downloaded the 1305 BIOS from the Asus website, which is still as horrible as it has ever been.  It wouldn't even load from the server, so we had to use another computer, network the downloaded files onto the server, and from there put them on the floppy.

After flashing the board from within BIOS we were greeted with a lockup immediately after the splash screen.  I spent 1/2 an hour unplugging things one-by-one trying to root out the problem, but it didn't help.  I tried unplugging all the cables and plugging them all back in (it sounds weird, but I have seen it fix many problems).  Visions of RMAing the board and going server-less for 2 weeks were running through my head, and I wasn't happy.  Finally I tried removing the battery for a minute, which somehow un-froze the BIOS.

From there it was a quick boot into Windows and then some big smiles as we saw that the array was back, and rebuilding.  No big re-installs this time.

But we weren't without issues.  Immediate attempts to copy files off of the drives were met by network timeouts--it seemed someone had forgotten to plug the network cable back in. :)

And this morning I saw that all the e-mails I received overnight reported being delivered on January 1st, 2008.  The clock needed to be set.  Also, OWA was still not running.  Some Google-ing of the error messages revealed that a few stopped services were probably the cause.  Starting them did get OWA running.  But outbound e-mails were not sending.  I figured this was due to the time issue still, and all the Exchange processes would need to be restarted, so I just rebooted the server.

20 minutes later the server was still not responding (I was at a remote location).  I quickly realized that it was probably halted at the BIOS screen because no keyboard was connected and the default of the new BIOS would be to halt on all errors.  Consider that lesson learned.

And now the happy ending.  Everything is running as it used to, no data was lost, e-mail is working, and I don't have to (get to?) spend my weekend re-doing a server!

Well here we go; I'm going to detail the unpleasant experience of setting up our new server as best as I can remember it.

Problem 1:  Floppy disk with drivers needed for RAID functionality.  We actually had a floppy drive, and even a computer to connect it to, but no floppy disks could be found.  So we drove a couple miles to someone's house and found one floppy--and old Intel motherboard driver disk.  We fired up the ancient computer there, put the disk in, put the CD-ROM from Asus in the optical drive...and got stuck.  It wouldn't read the disk.  Closer examination revealed that it was actually a DVD disk, which the 5+ year old computer couldn't read.  We took the floppy home and made the disk there.

Problem 2:  Getting the computer to boot correctly.  Having not dealt with a floppy drive for several years, we were both unfamiliar with the cause of those cryptic "failure to find boot disk" messages, which were very vexing.  We initially blamed it on the RAID and how that fit into the boot order.

Problem 3:  Not having disk 2.  Server 2003 64-bit comes on two CDs.  We had 2 MSDN-iso burned disks, one labeled disk 1 and one labeled disk 2.  The second one was most certainly not disk 2.  Off to MSDN to download...and in the meantime we went ahead and installed updates and Service Pack 2.

Problem 4:  We ran disk 2, only to get a warning that Service Pack 2 had already been installed.  We proceeded on anyways.  Around this time we started getting random lockups.  Then a message popped up detailing that the RAID had entered a degraded state.  After messing around in the RAID software for a couple minutes, we decided that one of the drives was bad, and that we would have to reinstall on a RAID composed of the three remaining disks.

Problem 5:  Windows installed again, everything updated, RAID fails again.  So this time I backed up an image of everything we had setup to another computer, re-installed Windows on one of the SATA HDDs (not in a RAID) and restored from the backup.  This seemed to work alright, until we started to have a LOT of problems installing Exchange Server 2007.

So we re-installed again (fourth time if you're counting).  By now I figured that something was up and these disks weren't actually failing.  But we were also sick of the RAID idea so just installed Windows on a spare IDE HDD we had laying around.  In the meantime, we figured out that the disks probably hadn't been given adequate time to rebuild (although I'm still not sure why a new RAID with empty disks needs to be built).

This is the install we are currently running on, and it's working quite well.  After the RAID was given time to build (I went into the BIOS RAID control panel and told it to rebuild) it has been running fine.  We had quite a lot of trouble again with Exchange Server 2007, but that is another story altogether...

Yep, we finally did it.  After inching along for two years on 1.0 GHz servers (one or two or three as we saw fit at the time) we have finally invested in some big iron.  $800 got us the following:

• 2x 1GB DDR2 667MHz RAM
• 4x 500GB Seagate 7200.10 HDDs
• 400-watt Rosewill PSU
• 4U Rackmount Case
• 2.2GHz AMD Athlon 64 X2 ProcessorAsus M2N-E Motherboard

I managed to get the HDDs for $100 through some sort of error on Tigerdirect's website I think.  They had ones with 16MB cache listed there for $120 (and the same price was at Newegg), but 8MB ones for $100.  I didn't really care that much about that so I ordered them.  Then I read online that those drives aren't available with 16MB cache so I clicked on the link in the shipping confirmation e-mail they had given me and that lead me to a non-existant page.  Then 16MB cache drives showed up at the door, which made me happy.  But the HDDs still made up 1/2 of the total price.

Buying the RAM second-hand from my boss in Medora also helped stay under budget as I got it for $60.  The power supply was pretty cheap ($35) and I have some doubts about it (it's not very heavy) but it had lots of good reviews on Newegg.

I got that motherboard because it's an Asus and it has 6 SATA ports with RAID 5 support.  So the HDDs are in a RAID 5 with an end total capacity in Windows of 1.36TB.  Unfortunately due to some initial issues installing Windows I ended up not putting the OS in the 60GB partition on the RAID that I had planned.  I threw in an old 160GB drive for that, and we are now using the RAID solely for file storage.

I also threw in one out-of-budget expenditure: an 8-port gigabit switch which I picked up for ~$60.

Tomorrow I will detail all the headaches that we had installing everything.