Raid 5 rebuild and Raid 6 reshape question
This is a potentially dumb question...
I had a 5 disk Raid5 with 3TB disks. One disk threw some errors (but still seemed to work), and i decided to replace it. I actually bought two drives, thinking I'd keep one as a cold spare, but when i got to replacing the drive (after days of load testing to see if I'd gotten a dud), I figured maybe I'd use the spare as a hot spare, or make a Raid6 array. In the end I decided on a Raid6, as it is the recommended way to go these days for large disks.
When I did the add for the first disk, it started rebuilding the raid5, and said it'd only take like 5 hours, then I added the second disk and it skyrocketed to about 116 hours (7000s or so). I presume this is expected, due to having to re-stripe all of the data, and re-calculate the parity at the same time.
How much faster would this have finished if I had just waited for the first drive to finish rebuilding and done the reshape as a separate step? I decided to just let this reshape continue, its about 30% the way through, and this is actually the backup array for my main NAS, so it isn't actually a problem if it dies mid-reshape. I'm just curious about the alternatives to how I did it. I know i could have just copied the old disk to one of the new ones, and skipped the entire rebuild on one disk, but i didn't feel like it ;D
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 2 lines · Thomas Fjellstrom
I had a 5 disk Raid5 with 3TB disks. One disk threw some errors (but still seemed to work), and i decided to replace it. I actually bought two drives,
If you have a not-ancient kernel, you should have issued the replace command. You should have added one new drive as spare, then (with a new mdadm) issued the --replace command. This would have started copying the drive you wanted replaced onto the spare. Then after all this was done, you could have added another spare and told it to re-shape to raid6. That's at least what I would have done.
Signature · 1 line
Mikael Abrahamsson email: [email protected]
Quoted 12 lines · Mikael Abrahamsson
Quoted 3 lines · Thomas Fjellstrom
I had a 5 disk Raid5 with 3TB disks. One disk threw some errors (but still seemed to work), and i decided to replace it. I actually bought two drives,
If you have a not-ancient kernel, you should have issued the replace command. You should have added one new drive as spare, then (with a new mdadm) issued the --replace command. This would have started copying the drive you wanted replaced onto the spare. Then after all this was done, you could have added another spare and told it to re-shape to raid6. That's at least what I would have done.
Quoted 2 lines
http://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-ye
t-failed-disk-in-a-linux-raid5-array
Yes, I could have. The machine's drive bays were full at the time, it wasnt till I had inserted the new drives that I realized i could have left the old one in and done a copy or replace. But then the machine was booting without it and probably not a great time to put the old disk back in (though an assemble force could have brought it back?). It's almost 60% done, so it's not a huge problem. I was just curious as to how long an actual raid5->raid6 reshape would normally be or this setup. I expect it normally can take a while as it has to rewrite a lot of data (11TiB worth).
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 1 line · Thomas Fjellstrom
problem. I was just curious as to how long an actual
raid5->raid6reshape would normally be or this setup. I expect it normally can take a while as it has to rewrite a lot of data (11TiB worth).
In my experience it takes 5-15 times longer than a normal resync takes, to reshape from raid5->raid6. So for your 3TB disk size case, I'd say expect 4-8 days or so.
Signature · 1 line
Mikael Abrahamsson email: [email protected]
Quoted 8 lines · Mikael Abrahamsson
Quoted 3 lines · Thomas Fjellstrom
problem. I was just curious as to how long an actual
raid5->raid6reshape would normally be or this setup. I expect it normally can take a while as it has to rewrite a lot of data (11TiB worth).In my experience it takes 5-15 times longer than a normal resync takes, to reshape from
raid5->raid6. So for your 3TB disk size case, I'd say expect 4-8 days or so.
Ah. Good to know. mdstat was saying about 4-5 days. I kindof thought that was about normal given the amount of work involved to restripe all drives and calculate parity.
Thanks :)
Signature · 2 lines
Thomas Fjellstrom [email protected]