RAID

Here's some very old info copied from my old site about RAID. It still may be useful, though.

Linux Software RAID

Online Reshaping

According to my research, as of Linux 2.6.17, it is possible to reshape (read: grow) a RAID-5 array onto new disks while the array is active. This facility was added for RAID-6 in 2.6.21. The kernel must be built with the option CONFIG_MD_RAID5_RESHAPE enabled. Reshaping RAID-5 also requires an updated mdadm. I wouldn't recommend any less than mdadm 2.4.1 based on the changelog. RAID-6 requires mdadm 2.6.1.

It should be noted that it has been possible for some time to grow a RAID-5 or RAID-6 onto larger disks, just not more disks. This is a fairly trivial thing to implement. But in ordinary practice, the only way for the disks suddenly to become larger is to replace all the disks, one by one, with larger ones, letting the array become degraded and resync each time. If downtime is possible, all disks could be copied to larger disks in a safer manner.

Trial

I tried this functionality to test is basic operation. I did so not on many physical disks, but with LVM logical volumes on one physical disk. My test setup was as follows:

    • Ubuntu Feisty

      • Linux 2.6.20-16-generic

      • mdadm 2.5.6

    • 1 GB LVM volumes in a one-physical-volume non-RAID volume group

My process was as follows:

    1. Initial setup (3-disk RAID-5)

      1. Create three 1 GB volumes

      2. Create one RAID-5 array from these three volumes

      3. Format with XFS and mount

    2. Test reading while reshaping onto one more disk (4-disk RAID-5)

      1. Generate a 2 GB file from /dev/urandom into the new filesystem

      2. Calculate the md5sum of this file

      3. Create another 1 GB volume

      4. Begin calculating the md5sum of that file again

      5. While calculating, reshape the array to include the new volume

      6. Compare md5sums

    1. Test writing while reshaping onto two more disks (6-disk RAID-5)

      1. Grow XFS

      2. Generate a 1 GB file from /dev/urandom in a temporary place (not in the RAID)

      3. Calculate the md5sum of this file

      4. Begin copying this file to the RAID

      5. While copying, reshape the array to include the two new volumes

      6. Once copy is finished, calculate the md5sum of the new file

      7. Compare md5sums

    1. Final sanity checks

      1. Recalculate md5sum on all files on the RAID and compare.

      2. Run xfs_check

All these steps went smoothly, and all md5sums were correct. The basic functionality seems to be there. See below for results of a test on a larger array with actual independent disks.

First real usage case

I decided to grow my MythTV box's RAID 5 with this method. The setup was:

    • Debian Etch

      • Linux 2.6.18-5-686

      • mdadm 2.5.6

    • 931 GB XFS on LVM on 6-disk (200 GB) RAID 5

I added a 7th 200 GB disk, and followed similar procedures as above. I was able to record and watch programming simultaneously while the reshaping commenced. I was able to use pvresize to alert LVM to the bigger RAID, and then lvextend and xfs_growfs in the usual way. I now show 1.1 TB of usable storage. The only downtime I encountered was to add the extra drive (PATA, no hotswap); all services kept running throughout the actual growth of the array, LVM, and filesystem.

Mean Time to Data Loss

As drives get bigger, their guarantees on how many bits can be read before an unrecoverable read error occurs do not improve. Therefore arrays with large disks are very much at risk of hitting an unrecoverable read error in the process of rebuilding a failed disk, just when you can't afford to have another failure. It doesn't take too many 1 TB disks with typical MTBF ratings before your array is essentially guaranteed to fall over during a rebuild!

This blog post from Sun explains things and gives formulas to calculate your risk.