On growing mdadm RAID5

There is a right way and there are wrong ways to add drives to a RAID5 array with mdadm. Annoyingly, I chose one of the wrong ways last week when I went to increase the hard drive space in thevoid. Thankfully, my mistake has proven more bothersome and time consuming than harmful.

Starting out last week, thevoid had 4x 500GB drives in an mdadm RAID5 configuration: /dev/hde1, /dev/hdg1, /dev/sda1, /dev/sdb1. This had me sitting on 1.5TB of space with one drive worth of failability, which was proving insufficient. In order to resolve my space issues, I ordered 3x 500GB drives and proceeded to install them when the arrived: /dev/hdb, /dev/hdc, /dev/hdd.

In my haste to have more space, I proceeded to add them to the array and grow it:

mdadm --manage /dev/md1 --add /dev/hdb
mdadm --manage /dev/md1 --add /dev/hdc
mdadm --manage /dev/md1 --add /dev/hdd
mdadm --grow /dev/md1 --raid-devices=7

This process then got to work and took about 3-4 days to complete. I assume the time was on account of most of the drives being IDE drives, many sharing channels and there being an awful lot of space. Once the growing process was complete, I merely needed to resize the ext3 partition on the array:

resize2fs /dev/md1

This then took an hour or so and I was sitting on 3TB of total space. The only part of the process during which my drive was inaccessible was when I had to turn off the computer to physically install my IDE drives.

Everything is perfect now, right? Wrong. Guess who forgot that you should partition drives before using them? That's right, me. I really wanted to put a linux raid autodetect partition on each of the drives before adding them to the array. Not that it really does much harm to add the drives straight to the array but it's poor form and it might pose problems that I am not aware of in the future so, clearly, it's a thing that should be fixed. Thankfully, being RAID5, my array is able to lose drives and still be fine, thus allowing the solution of failing, removing, partitioning and adding each of the new drives back into the array:

mdadm --manage /dev/md1 --fail /dev/hdb
mdadm --manage /dev/md1 --remove /dev/hdb

partion /dev/hdb and then:

mdadm --manage /dev/md1 --add /dev/hdb1

now wait until the array has rebuilt itself:

cat /proc/mdstat

and repeat for the other drives. Overall, it's not hard but it's annoying and it does take about 1/2 to 2/3 of a day per drive to fix. Thankfully, though, now the endeavor is done and my array works correctly with no lost data and minimal down time. Hooray for software RAID and having 3TB in a single place.

Oh, and you don't want to forget to update /etc/mdadm.conf after every step of the process of you could have some potential problems.

Comments

Ah!

I wish I had read this a couple of days ago. I just finished growing my RAID5 (2x1TB) with another drive. Have been impatient about the long process but finally, today, it finished the rebuild.

Before I was going to resize the RAID I thought I'd do a fdisk -l and then I noticed that also I forgot to add the Linux Raid Autodetect flag on the partition :(

I followed your guide here and hope I (my data) will be OK, I'm sure it will.

I reinstalled the system the other day and really do appreciate the autodetect of RAIDs so I would't want it any other way. It's probably possible to set the RAID up also without the flag but since I'm no linux guru I have no idea how to do it - do you know?

Too bad there are several RAID HOWTOs out there that neglect to mention that you should set the Linux Raid Autodetect flag. The GROW HOWTO I followed did not and I didn't really pay attention to the fact that it should be partitioned before hand, even though I knew I should and intended to but forgot in my impatience of getting the larger disk space.

Anywho, thanks for your mini-guide :)

Glad that I could help.

FYI, if you forget the Linux RAID Autodetect partition type, any unmarked drives will not be added to your array on reboot. I missed the flag on one of the drives in my current RAID6 setup and every time that I rebooted, I had to manually re-add the missing drive and rebuild the array. HOWTOs are, indeed, somewhat sparse and I had rather a tough time tracking down the problem. My rebuilds are quick, RAID6 has extra redundancy, and Linux reboots are rare, though, so it took me a while to get fed up enough to actually fix the problem.