Skip to contentSkip to conversation

Assume-clean for md grow

DISCUSSION Chris Webb 2 participants 5 messages Feb 25, 2009 – Feb 26, 2009 MD RAID

lore
Share & tools
lore

Chris Webb

author
#0 lore raw

I use md arrays made up of slots synthesized through device mapper from physical storage distributed across a cluster of machines. When these slots are created, they are guaranteed to be zero-initialised, so I can safely do mdadm --create --assume-clean to avoid an initial resync.

When I grow the backing slots, the new storage space is also zero-initialised, so I'd like to be able to do the equivalent of mdadm --grow --size=max --assume-clean. However, --assume-clean isn't supported for grow operations.

Is there some way I can tell the kernel driver (perhaps through sysfs) that the resync is clean/already complete to avoid unnecessary heavy IO on every grow? I tried things like

echo idle >/sys/block/mdX/md/sync_action

without success.

Cheers,

Chris.

#1 lore raw
Quoted 18 lines · Chris Webb

I use md arrays made up of slots synthesized through device mapper from physical storage distributed across a cluster of machines. When these slots
are created, they are guaranteed to be zero-initialised, so I can safely do
mdadm --create --assume-clean to avoid an initial resync.

When I grow the backing slots, the new storage space is also zero-initialised, so I'd like to be able to do the equivalent of mdadm --grow --size=max --assume-clean. However, --assume-clean isn't supported for grow operations.

Is there some way I can tell the kernel driver (perhaps through sysfs) that
the resync is clean/already complete to avoid unnecessary heavy IO on every
grow? I tried things like

Quoted 3 lines

echo idle >/sys/block/mdX/md/sync_action

without success.

No, there isn't any way do get --grow to --assume-clean.
And I cannot immediately think of a neat way to implement it. I'll have a think about it and let you know if I come up with anything.

If you want to hack your own kernel, you could just remove the

		mddev->recovery_cp = mddev->dev_sectors;

from raid1_resuze (Assuming it is raid1 arrays that you are growing). Then the resync-after-grow would never happen.

Or if you want to be slightly more subtle, remove the

	if (mddev->pers)
		return -EBUSY;

from resync_start_store. Then before a grow that you want to be --assume-clean, write into /sys/block/mdXX/md/resync_start the number of sectors in the final raid1 array.

That last might be the way I end up doing it, but I'm not sure yet.

NeilBrown

author
#2 lore raw
Quoted 6 lines · NeilBrown

Or if you want to be slightly more subtle, remove the

	if (mddev->pers)
		return -EBUSY;

from resync_start_store. Then before a grow that you want to be --assume-clean, write into /sys/block/mdXX/md/resync_start the number of sectors in the final raid1 array.

Hi Neil. Thanks for this suggestion: it looks fine for what we're looking to achieve. Interestingly, this will mean writing a smaller value that shown by a read from this file for an in-sync array, but will still work?

I'll build a patched kernel and give it a go a little later on.

Best wishes,

Chris.

author
#3 lore raw
Quoted 9 lines · Chris Webb
Quoted 6 lines · NeilBrown

Or if you want to be slightly more subtle, remove the

	if (mddev->pers)
		return -EBUSY;

from resync_start_store. Then before a grow that you want to be --assume-clean, write into /sys/block/mdXX/md/resync_start the number of sectors in the final raid1 array.

Quoted 3 lines

Hi Neil. Thanks for this suggestion: it looks fine for what we're looking to achieve. Interestingly, this will mean writing a smaller value that shown by a read from this file for an in-sync array, but will still work?

For instance,

3# cat /sys/block/md127/{size,md/resync_start}
2097136
18446744073709551615

I just tried growing the slots up from 1G to 3G, then

  echo $((2097136 + 2*1024*1024*2)) >/sys/block/md127/md/resync_start
  mdadm --grow 3145720

but this gives me

md127 : active raid1 dm-3[1] dm-2[0]

        3145720 blocks super 1.1 [2/2] [UU]
          resync=PENDING

in /proc/mdstat, which is presumably not right?

Does this sysfs file hold the size of the array or the size of the components, for implementing the non-RAID1 case?

Also, if I don't know the size of the final array[1], is it safe to write a value much larger than the size of the array in here, or will that cause future grows to be clean when this isn't necessarily intended?

[1] One of the things which has been most awkward with using md as part of a automated storage system has been going from component size to available array size and back again, given that bitmap reservation depends on the original size of the array not the current size of the array. (In the end, we've cheated and always written everything in terms of change in component size vs change in array size, and been generous with the amount of space we allocate to components on initial device create. It does feel like I'm coding far too much knowledge of the internal choices mdadm makes into my management layer, though!)

Cheers,

Chris.

author
#4 lore raw
Quoted 5 lines · Chris Webb

I just tried growing the slots up from 1G to 3G, then

  echo $((2097136 + 2*1024*1024*2)) >/sys/block/md127/md/resync_start
  mdadm --grow 3145720
Quoted 7 lines

but this gives me

md127 : active raid1 dm-3[1] dm-2[0]

        3145720 blocks super 1.1 [2/2] [UU]
          resync=PENDING

in /proc/mdstat, which is presumably not right?

Rewriting the original value of 18446744073709551615 back to /sys/block/md127/md/resync_start removes this, although it looks like any value lower than this gives a PENDING state. Presumably this means that:-

  echo 18446744073709551614 >/sys/block/md127/md/resync_start
  mdadm --grow --size=... /dev/md127
  echo 18446744073709551615 >/sys/block/md127/md/resync_start

is a generic recipe for growing an array 'assume-clean' without needing to know anything more about the eventual size of the array other than the parameter passed to --size= (max or component size).