Build Your Own RAID 5 Array

Posted by JohnP 08/20/2007 at 16:13

October 2010 Update

This article has been getting hit a bunch, so I thought an update might be helpful. The array built below is still working nicely. I’m still using it as a NAS, NFS, CIFS Server, Samba Server, DLNA server and for protected storage to the local machine. The local machine is now a Core i5-750 with lots of RAM that also runs a few virtual machines. The machine also runs compute processes since it has plenty of excess CPU and excess RAM available.

The hardware RAID card was removed when the motherboard was replaced. The new CPU/MB has 6 SATA connectors internally, so those are being used. No complaints. For more information on Migrating Software RAID.

The Infiniband cable just works and has never had any issue.

The machine has an Intel PRO/1000 NIC ($25 @ newegg) to be certain of reasonable GigE network performance and possible future compatibility with VMware ESX/ESXi.

I’ve tuned both the disk cache and network buffers available to improve performance for large file transfers.

I’m considering adding AoE, ATA over Ethernet, and iSCSI servers to this machine so others on the network can access the storage too. AoE seems like a more efficient solution available for block storage access.

The RAID5 setup hasn’t had any issues in years. No dropped disks due to vibration or any other issues … besides being nearly full most of the time. I guess that’s the way of all storage.

Original Article Follows:

Ok, after months of research and finally finding all the parts at "reasonable prices", the array has been ordered, delivered, built, tested on multiple OSes and brought into production use. I’m mostly happy and will outline what I’d do differently this time.

Parts:

  • Addonics Array w/ Infiniband port – $139
    http://addonics.com/products/raid_system/mst4.asp
    http://addonics.com/products/raid_system/images/amst_front_large.gif
  • Addonics Infiniband Cable – $80
    http://www.shopaddonics.com/mmSHOPADDONICS/Images/aaib4c150.gif
  • Addonics Infiniband 2 quad SATA bridge (inside computer) – $50
    http://www.shopaddonics.com/mmSHOPADDONICS/Images/ad4saml-pci.gif
  • 4x Seagate 320GB perpendicular drives (running quiet and cool) – $400
  • Promise TX4310 4xSATA2 Hardware RAID PCI card (RAID5 capable) – $115
    http://www.promise.com/product/product_detail_eng.asp?segment=RAID%205%20HBAs&product_id=165

Step 1. Put the 4 drives into the array and connect all the power and SATA2 connections.
Step 2. Connect the Infiniband cable to the array.

Step 3. Install the Infiniband-2-SATA in the computer; leave room for the SATA cables that will waste 2 card slots – yes, this design sucks. I connected the SATA cables before installing this inside the computer.

Step 4. Install the Promise card in the computer, connect the SATA cables from the converter to the RAID card. Test the drives with Spinrite.

Step 5. Fight with the Promise drivers (not included b.c. they are complete idiots), you’ll need to update the firmware, windows drivers and web management software. Promise doesn’t include any of this software on the CD that comes with the card. Worse, they use a proprietary extension to the ZIP format that only a current WinZip supports. After all the zipping is done, they saved 4K from a self-extracting EXE. Stupid, stupid. Long story short – HW RAID works under WinXP, not Ubuntu Edgy. Promise support answers email with – that’s an old card and we don’t support that kernel.

Step 6. Curse … a lot.

Step 7. After spending 3+ weeks fighting to get HW RAID working under Linux, give up and use software RAID – ‘mdadm’.

Step 8. Good news – under current Linux kernels, the Promise controllers are recognized as JBOD controllers, so software RAID can be used.

Step 9. Partition each drive with an "autoRAID filesystem", use mdadm to create a RAID5 array (chunk size 128K), mkfs -t jfs /dev/md0 (or whereever the mdadm tells you the device is located), use mdadm to pull the array info needed for the /etc/mdadm/mdadm.conf file, update the /etc/fstab with the /dev/md0 device and mount point, create the mount point, mount /dev/md0 /raid5, and start using your array with some test data.

Step 10. reboot your box and verify that it comes up and the array is automatically assembled and mounted. If not, start reading the man pages.

What I would do differently:

  • Don’t buy Promise RAID card (I really wanted an PCI-X1 card anyway)
  • Pick a hw RAID w/ XOR engine that is supported by the Linux kernel (no proprietary drivers without complete source).
  • Don’t use Infiniband cables. Don’t get me wrong, the cable is great and solid connection. My problem is the $80 for it and the $50 for the converter into SATA2 connector I needed. I’d try the eSATA-MP cable next time and find a RAID card that can accept it. There are very few Infiniband RAID cards (I gave up looking). The converter design really bugs me. They should have "end-mounted" the sata connectors, not side-mounted them.
  • Don’t bother with LVM2 unless you really need it. I learned this lesson a few years ago and I won’t make that mistake again.
  • Use either XFS or JFS file systems. These are the highest performance general purpose file systems that are included with standard kernels. Don’t use a file system that isn’t part of your normal kernel – another lesson I learned a few years ago. I really would like to use ZFS, but it isn’t included in any kernels to my knowledge. I’ve been a fan of JFS since 1996 running AIX. I’m still a fan, it has never let me down. Some dated file system benchmarks. Some newer RAID benchmarks that show hw raid is often faster. However, here’s an article from a Linux kernel developer as to why software RAID should almost always be used for internally connected disks.
  • Run Spinrite on all drives before making them part of any RAID config. Best to find any issue before getting everything up and RAIDing.
  • Double check all your connections – a loose SATA connection in the array caused me to waste about a week assuming the Promise controller was bad hardware, not bad software.

A few links:
http://addonics.com/
http://www.newegg.com/
http://www.pricewatch.com/
http://www.linuxjournal.com/article/8874
http://www.linuxquestions.org/questions/showthread.php?t=387769
http://www.howtoforge.org/archive/2007/1/14?s=988d6534035fb8aea0a7793bb3318eb7&
http://www.linuxdevcenter.com/pub/a/linux/2002/12/05/RAID.html