Cancel reshape?
I've just started reshaping a RAID-5 to a RAID-6. I had meant to increase the number of drives too, so avoiding using the backup file all the way through. Unfortunately I forgot to specify that, so it's going rather slowly. I said:
mdadm /dev/md1 --grow --level=6 --backup-file=/some/where/else
where md1 had 3 active and 2 spare devices. It's now reshaping itself to RAID-6 with 4 active and 1 spare.
What I meant was
mdadm /dev/md1 --grow --level=6 --raid-disks=5 --backup-file=...
Can I cancel reshape that's currently running, and using the backup file that's actually on another (RAID-1) array on three of the same drives, and running at ~7MB/s, and try again?
Cheers,
John.
Quoted 10 lines · On Wed, 12 Jan 2011 04:19:30 +0000 John Robinson…
I've just started reshaping a RAID-5 to a RAID-6. I had meant to increase the number of drives too, so avoiding using the backup file all the way through. Unfortunately I forgot to specify that, so it's going rather slowly. I said:
mdadm
/dev/md1--grow --level=6 --backup-file=/some/where/elsewhere md1 had 3 active and 2 spare devices. It's now reshaping itself to RAID-6 with 4 active and 1 spare.
Quoted 8 lines
What I meant was
mdadm
/dev/md1--grow --level=6 --raid-disks=5 --backup-file=...Can I cancel reshape that's currently running, and using the backup file that's actually on another (RAID-1) array on three of the same drives, and running at ~7MB/s, and try again?
Sorry, but no. This isn't supported yet.
It would require modification to the kernel to be able to run a reshape in reverse.
NeilBrown
On 12/01/2011 04:19, John Robinson wrote:
[...]
Quoted 3 lines
Can I cancel reshape that's currently running, and using the backup file that's actually on another (RAID-1) array on three of the same drives, and running at ~7MB/s, and try again?
Blast. I found this patch to add a revert-reshape option from archives of this list - http://www.spinics.net/lists/raid/msg30961.html - but it's not included in mdadm 3.1.4 and I'm running from a rescue CD so I can't rebuild mdadm with it. I think I might just have to wait a while.
Perhaps I can stop the reshape and restart it with the backup file elsewhere. Hmm, I've an SSD I haven't installed yet... Maybe time to give it a wee workout :-)
Cheers,
John.
Quoted 11 lines · On Wed, 12 Jan 2011 04:49:35 +0000 John Robinson…
On 12/01/2011 04:19, John Robinson wrote:
[...]Quoted 3 lines
Can I cancel reshape that's currently running, and using the backup file that's actually on another (RAID-1) array on three of the same drives, and running at ~7MB/s, and try again?
Blast. I found this patch to add a revert-reshape option from archives of this list - http://www.spinics.net/lists/raid/msg30961.html - but it's not included in mdadm 3.1.4 and I'm running from a rescue CD so I can't rebuild mdadm with it. I think I might just have to wait a while.
That patch only works to revert a reshape that is increasing the number of data-devices in the array.
Quoted 3 lines
Perhaps I can stop the reshape and restart it with the backup file elsewhere. Hmm, I've an SSD I haven't installed yet... Maybe time to give it a wee workout :-)
That should work.
mdadm --stop ...
copy file
mdadm --assemble ...... --backup-file=/new/place
NeilBrown
Quoted 8 lines
Cheers,
John.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Speed up reshape? (was Re: Cancel reshape?)
Quoted 2 lines · NeilBrown
On Wed, 12 Jan 2011 04:49:35 +0000 John Robinson
<[email protected]> wrote:
[...]
Quoted 9 lines
Quoted 3 lines
Perhaps I can stop the reshape and restart it with the backup file elsewhere. Hmm, I've an SSD I haven't installed yet... Maybe time to give it a wee workout :-)
That should work.
mdadm --stop ... copy file mdadm --assemble ...... --backup-file=/new/place
Yup. Not much of an improvement, though; the reshape is now running at ~10MB/s instead of ~6.5MB/s, so ~27 hours to go instead of ~40 hours.
iostat tells me the original 3 discs are doing ~120 transactions per second, reading ~20MB/s and writing ~10MB/s, while the one new disc is doing ~60 tps, reading nothing (obviously) and writing ~10MB/s and the backup (SSD) is doing ~45 tps and writing ~20MB/s. ~120tps sounds about right for 120 seeks, 60 reads and 60 writes, and would explain the speed, but otherwise this looks odd to me - why are we reading ~60MB/s in total and writing ~40MB/s? I'd have expected to be reading less than we're writing. And why are we apparently reading and writing in blocks of under 512KB per device, when the backup file is 32MB? Is there something I can tweak to run the reshape in bigger chunks? sync_speed_min is already 200000...
Currently running SystemRescueCD 2.0.0, 32-bit kernel 2.6.35 and mdadm 3.1.4. CPU usage is 2% system, 4% iowait.
Cheers,
John.
Re: Speed up reshape? (was Re: Cancel reshape?)
Quoted 17 lines · On Wed, 12 Jan 2011 05:53:40 +0000 John Robinson…
Quoted 2 lines · NeilBrown
On Wed, 12 Jan 2011 04:49:35 +0000 John Robinson
<[email protected]> wrote:[...]
Quoted 9 lines
Quoted 3 lines
Perhaps I can stop the reshape and restart it with the backup file elsewhere. Hmm, I've an SSD I haven't installed yet... Maybe time to give it a wee workout :-)
That should work.
mdadm --stop ... copy file mdadm --assemble ...... --backup-file=/new/placeYup. Not much of an improvement, though; the reshape is now running at ~10MB/s instead of ~6.5MB/s, so ~27 hours to go instead of ~40 hours.
Quoted 11 lines
iostat tells me the original 3 discs are doing ~120 transactions per second, reading ~20MB/s and writing ~10MB/s, while the one new disc is doing ~60 tps, reading nothing (obviously) and writing ~10MB/s and the backup (SSD) is doing ~45 tps and writing ~20MB/s. ~120tps sounds about right for 120 seeks, 60 reads and 60 writes, and would explain the speed, but otherwise this looks odd to me - why are we reading ~60MB/s in total and writing ~40MB/s? I'd have expected to be reading less than we're writing. And why are we apparently reading and writing in blocks of under 512KB per device, when the backup file is 32MB? Is there something I can tweak to run the reshape in bigger chunks?
sync_speed_minis already 200000...
1/ everything is read twice, once to back it up, once to relocate it.
That is unfortunate, but awkward to avoid.
2/ try increasing the stripe_cache_size - it might help.
NeilBrown
Quoted 7 lines
Currently running SystemRescueCD 2.0.0, 32-bit kernel 2.6.35 and mdadm 3.1.4. CPU usage is 2% system, 4% iowait.
Cheers,
John.
Re: Speed up reshape? (was Re: Cancel reshape?)
Quoted 1 line · NeilBrown
On Wed, 12 Jan 2011 05:53:40 +0000 John Robinson
[...]
Quoted 2 lines
1/ everything is read twice, once to back it up, once to relocate it.
That is unfortunate, but awkward to avoid.
OK... but why only one stripe at a time? I eventually worked out that the disc transactions (shown by iostat) were of 256KB at a time which is the chunk size of the array, and the ~120 transactions per second would relate to the spin speed of the discs, 7200rpm being 120 revolutions per second. Even at the slow end of the discs, there's at least 512KB of data per revolution, so slurping up at least a couple of revolutions' worth before rewriting it might save waiting for quite so many disc revolutions to finish.
I may test this by doing another in-place reshape, changing the chunk size to 512KB, which I expect will run at twice the speed of the reshape on 256KB chunks.
Also, would there be any way to allow the second read to come from cache rather than from the media surface again? Is this a side-effect of using O_DIRECT or something? Could O_DIRECT be used only for writing?
Quoted 1 line
2/ try increasing the
stripe_cache_size- it might help.
mdadm had already set it to 1071, I upped it to 8192, then 32768, but it didn't make any difference at all, and stripe_cache_active showed 8 or even 0 when I checked. I imagine this is because the in-place reshape needs to wait for writes to be sync'ed all the time (or written with O_DIRECT or whatever) so there's little or no scope for cacheing stripes outside the reshape process itself.
I'm now reshaping again, from 4 to 5 discs. That's whizzing along at a fairly healthy 35MB/s, which according to iostat breaks down as about 50MB/s read from 4 drives and 35MB/s written to 5 drives and is as much as I could reasonably expect from these discs. stripe_cache_active shows ~25000, CPU usage is 12% system, 14% iowait. I expect the speed to slow down a bit towards the end of the reshape, where the slowest of the drives can only manage 60MB/s streaming and the seeks between reading and writing are longer (further across the disc surface).
Anyway, quite apart from all my whingeing about the speed of the in-place reshape, it's all working perfectly which is after all the most important thing, so many thanks for all your hard work!
Cheers,
John.
Re: Speed up reshape? (was Re: Cancel reshape?)
On 13/01/2011 13:44, John Robinson wrote:
[...]
Quoted 3 lines
I may test this by doing another in-place reshape, changing the chunk size to 512KB, which I expect will run at twice the speed of the reshape on 256KB chunks.
Well, I got that wrong. Currently doing ~9MB/s, though the backup file is writing faster (~27MB/s). Another 30 hours reshaping. Oh well it should be worth it in the end.
Cheers,
John.