[LUNI] Need help determining what's causing a server to be slow

[Branko Kotur]

I've recently put together 2 new servers to replace older existing ones.  
However, whenever these new servers do any kind of disk intensive access, 
they seem to fall flat on their faces to the point where they're at times 
unresponsive (for example, untarring a 1GB tar file).  The only thing I can 
think of is that something isn't configured right as they should basically 
blow away the older servers that I'm using.  Even doing a simple operation 
like ls in a director that only had 10 files takes forever.

Whenever I've noticed the unresponsiveness, the load tends to be around 4 - 8 
with the IO Wait (as reported by top) is anywhere from 80% - 100% for 1 or 
both CPU's.  I'm wondering if the bottleneck is the RAID card or the driver 
for the card.  I'm also wondering if the PCI-X slot (that the RAID card is 
plugged into) may be the bottleneck.

Some info on these servers:

Dual core Opteron 180, 4GB RAM, 3ware 9550SX RAID card, CentOS 4.4 64bit, WD 2 
Caviar RE SATA 3.0G/s drive in RAID 1, Supermicro H8SSL-i motherboard.

CentOS 4.4 comes with the 3ware drivers as part of the install CD so I didn't 
think I needed to install them separately.  However, the actual file is much 
smaller then the one provided by 3ware (53K as opposed to the 313K version 
offered by 3ware).  Is the included one just a basic driver?  3ware seems to 
offer open source drivers, so I don't see why the included one would be any 
different then the official one.  

When I do an hdparm -tT /dev/sda, the results look pretty good (at least 
compared to the IDE drives I've been using).  However, this is my first time 
using SATA drives, so I don't have anything to compare with.  If anyone has 
any SATA drives, can you let me know if these results are good or bad?

/dev/sda:
Timing cached reads: 3772 MB in 2.00 seconds = 1884.40 MB/sec
Timing buffered disk reads: 190 MB in 3.00 seconds = 63.26 MB/sec

I've used 3wares' 3DM2 utility to set the RAID card to performance mode, but 
that doesn't seem to have helped much, if any.  I'm also wondering if the 
PCI-X slot is the bottleneck.  I'm pretty sure it can't do the full 3.0Gb/s, 
but I would think it can come close.

Any thoughts or ideas that I can look into?  Right now, my 3 year old servers 
with IDE drives and software RAID are out performing the new servers when 
under load or doing heavy disk access.  I know they can do better, but I just 
can't seem to find out what the problem is.