RFC: use TRIM data from filesystems to speed up array rebuild?
Subsystems (2)
Hi,
I'm not really a filesystem guy so this may be a really dumb question.
We currently have an issue where we have a ~1TB RAID1 array that is mostly given over to LVM. If we swap one of the disks it will rebuild everything, even though we may only be using a small fraction of the space.
This got me thinking. Has anyone given thought to using the TRIM information from filesystems to allow the RAID code to maintain a bitmask of used disk blocks and only sync the ones that are actually used?
Presumably this bitmask would itself need to be stored on the disk.
Thanks,
Chris
Signature · 6 lines
Chris Friesen Software Designer 3500 Carling Avenue Ottawa, Ontario K2H 8E9 www.genband.com
Quoted 5 lines · Chris Friesen
Hi,
I'm not really a filesystem guy so this may be a really dumb question.
Quoted 9 lines
We currently have an issue where we have a ~1TB RAID1 array that is mostly given over to LVM. If we swap one of the disks it will rebuild everything, even though we may only be using a small fraction of the space.
This got me thinking. Has anyone given thought to using the TRIM information from filesystems to allow the RAID code to maintain a bitmask of used disk blocks and only sync the ones that are actually used?
Presumably this bitmask would itself need to be stored on the disk.
Thanks,
Chris
Device mapper has a "thin" target now that tracks blocks that are allocated or free (and works with discard).
That might be a basis for doing an focused RAID rebuild,
Ric
Quoted 16 lines · On Tue, 04 Sep 2012 15:11:26 -0400 Ric Wheeler <[email protected]> wrote:
Quoted 15 lines · Chris Friesen
Hi,
I'm not really a filesystem guy so this may be a really dumb question.
We currently have an issue where we have a ~1TB RAID1 array that is mostly given over to LVM. If we swap one of the disks it will rebuild everything, even though we may only be using a small fraction of the space.
This got me thinking. Has anyone given thought to using the TRIM information from filesystems to allow the RAID code to maintain a bitmask of used disk blocks and only sync the ones that are actually used?
Presumably this bitmask would itself need to be stored on the disk.
Quoted 8 lines
Quoted 3 lines
Thanks,
ChrisDevice mapper has a "thin" target now that tracks blocks that are allocated or free (and works with discard).
That might be a basis for doing an focused RAID rebuild,
I wonder how....
Maybe the block-later interface could grow something equivalent to "SEEK_HOLE" and friends so that the upper level can find "holes" and "allocated space" in the underlying device.
I wonder if it is time to discard the 'block device' abstraction and just use files every .... but I seriously doubt it.
NeilBrown
Quoted 24 lines · NeilBrown
Quoted 15 lines · On Tue, 04 Sep 2012 15:11:26 -0400 Ric Wheeler <[email protected]> wrote:
Quoted 14 lines · Chris Friesen
Hi,
I'm not really a filesystem guy so this may be a really dumb question.
We currently have an issue where we have a ~1TB RAID1 array that is mostly given over to LVM. If we swap one of the disks it will rebuild everything, even though we may only be using a small fraction of the space.
This got me thinking. Has anyone given thought to using the TRIM information from filesystems to allow the RAID code to maintain a bitmask of used disk blocks and only sync the ones that are actually used?
Presumably this bitmask would itself need to be stored on the disk.
Quoted 7 lines
Quoted 3 lines
Thanks,
ChrisDevice mapper has a "thin" target now that tracks blocks that are allocated or free (and works with discard).
That might be a basis for doing an focused RAID rebuild,
Quoted 8 lines
I wonder how....
Maybe the block-later interface could grow something equivalent to "SEEK_HOLE" and friends so that the upper level can find "holes" and "allocated space" in the underlying device.
I wonder if it is time to discard the 'block device' abstraction and just use files every .... but I seriously doubt it.NeilBrown
I don't think that we have to go to that extreme, but I think it would be very useful to see if the device mapper people have ideas in how the thin target might be used in combination with MD :)
ric
Quoted 24 lines · NeilBrown
Quoted 15 lines · On Tue, 04 Sep 2012 15:11:26 -0400 Ric Wheeler<[email protected]> wrote:
Quoted 14 lines · Chris Friesen
Hi,
I'm not really a filesystem guy so this may be a really dumb question.
We currently have an issue where we have a ~1TB RAID1 array that is mostly given over to LVM. If we swap one of the disks it will rebuild everything, even though we may only be using a small fraction of the space.
This got me thinking. Has anyone given thought to using the TRIM information from filesystems to allow the RAID code to maintain a bitmask of used disk blocks and only sync the ones that are actually used?
Presumably this bitmask would itself need to be stored on the disk.
Quoted 7 lines
Quoted 3 lines
Thanks,
ChrisDevice mapper has a "thin" target now that tracks blocks that are allocated or free (and works with discard).
That might be a basis for doing an focused RAID rebuild,
Quoted 8 lines
I wonder how....
Maybe the block-later interface could grow something equivalent to "SEEK_HOLE" and friends so that the upper level can find "holes" and "allocated space" in the underlying device.
I wonder if it is time to discard the 'block device' abstraction and just use files every .... but I seriously doubt it.NeilBrown
Hi,
I've got a brief question about this feature that seems extremely promising:
You mentioned on your blog:
"A 'write' to a non-in-sync region should cause that region to be resynced. Writing zeros would in some sense be ideal, but to do that we would have to block the write, which would be unfortunate."
So, if we had a write on a "non-in-sync" region (let's imagine the bitmap allows for 1M granularity), we would compute the parity of every stripe that this write "touches" and update it? Is the solution zeroing the area used to save time reading and writing the data on the stripe to compute the parity, as well as any other stripes that are referenced by this "non-in-sync" region, even if the write wouldn't affect them, allowing us to then flip that entire region to "clean"?
Would this open the door to some "thin provisioned" MD RAID, where one could grow the underlying devices (in the case of a RAID built ontop of say LVM devices), and marking the new "space" as "non-in-sync" without disrupting (slowing) operations on the array with a sync?
In any case, seems like a great feature.
Regards,
Ben.
Quoted 51 lines · Benjamin ESTRABAUD
Quoted 38 lines · NeilBrown
Quoted 19 lines · On Tue, 04 Sep 2012 15:11:26 -0400 Ric Wheeler<[email protected]> wrote:
Quoted 18 lines · Chris Friesen
Hi,
I'm not really a filesystem guy so this may be a really dumb question.
We currently have an issue where we have a ~1TB RAID1 array that is mostly
given over to LVM. If we swap one of the disks it will rebuild everything,
even though we may only be using a small fraction of the space.This got me thinking. Has anyone given thought to using the TRIM information
from filesystems to allow the RAID code to maintain a bitmask of used disk
blocks and only sync the ones that are actually used?Presumably this bitmask would itself need to be stored on the disk.
Quoted 8 lines
Quoted 3 lines
Thanks,
ChrisDevice mapper has a "thin" target now that tracks blocks that are allocated or
free (and works with discard).That might be a basis for doing an focused RAID rebuild,
I wonder how....
Maybe the block-later interface could grow something equivalent to "SEEK_HOLE" and friends so that the upper level can find "holes" and "allocated space" in the underlying device.
I wonder if it is time to discard the 'block device' abstraction and just use
files every .... but I seriously doubt it.Quoted 1 line
NeilBrown
Hi,
I've got a brief question about this feature that seems extremely promising:
You mentioned on your blog:
"A 'write' to a non-in-sync region should cause that region to be resynced. Writing zeros would in some sense be ideal, but to do that we would have to block the write, which would be unfortunate."
Quoted 7 lines
So, if we had a write on a "non-in-sync" region (let's imagine the bitmap allows for 1M granularity), we would compute the parity of every stripe that this write "touches" and update it? Is the solution zeroing the area used to save time reading and writing the data on the stripe to compute the parity, as well as any other stripes that are referenced by this "non-in-sync" region, even if the write wouldn't affect them, allowing us to then flip that entire region to "clean"?
That would, I think, be correct. All zeros are the easiest to calculate - the parities (raid5 and raid6) are all zeros too. It is also the ideal pattern to write to SSDs - many SSDs these days implement transparent compression, and you don't get more compressible than zeros!
Quoted 6 lines
Would this open the door to some "thin provisioned" MD RAID, where one could grow the underlying devices (in the case of a RAID built ontop of say LVM devices), and marking the new "space" as "non-in-sync" without disrupting (slowing) operations on the array with a sync?
Yes, that would work. More importantly (because it would affect more people), it means that the creation of a md raid array on top of disks or partitions will immediately be "in sync", and there would be no need for a long and effectively useless re-sync process at creation.
Quoted 1 line
In any case, seems like a great feature.
Yes indeed.
Quoted 9 lines
Regards,
Ben.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 76 lines · David Brown
Quoted 51 lines · Benjamin ESTRABAUD
Quoted 38 lines · NeilBrown
Quoted 19 lines · On Tue, 04 Sep 2012 15:11:26 -0400 Ric Wheeler<[email protected]> wrote:
Quoted 18 lines · Chris Friesen
Hi,
I'm not really a filesystem guy so this may be a really dumb question.
We currently have an issue where we have a ~1TB RAID1 array that is mostly
given over to LVM. If we swap one of the disks it will rebuild everything,
even though we may only be using a small fraction of the space.This got me thinking. Has anyone given thought to using the TRIM information
from filesystems to allow the RAID code to maintain a bitmask of used disk
blocks and only sync the ones that are actually used?Presumably this bitmask would itself need to be stored on the disk.
Quoted 8 lines
Quoted 3 lines
Thanks,
ChrisDevice mapper has a "thin" target now that tracks blocks that are allocated or
free (and works with discard).That might be a basis for doing an focused RAID rebuild,
I wonder how....
Maybe the block-later interface could grow something equivalent to "SEEK_HOLE" and friends so that the upper level can find "holes" and "allocated space" in the underlying device.
I wonder if it is time to discard the 'block device' abstraction and just use
files every .... but I seriously doubt it.Quoted 1 line
NeilBrown
Hi,
I've got a brief question about this feature that seems extremely promising:
You mentioned on your blog:
"A 'write' to a non-in-sync region should cause that region to be resynced. Writing zeros would in some sense be ideal, but to do that we would have to block the write, which would be unfortunate."
Quoted 7 lines
So, if we had a write on a "non-in-sync" region (let's imagine the bitmap allows for 1M granularity), we would compute the parity of every stripe that this write "touches" and update it? Is the solution zeroing the area used to save time reading and writing the data on the stripe to compute the parity, as well as any other stripes that are referenced by this "non-in-sync" region, even if the write wouldn't affect them, allowing us to then flip that entire region to "clean"?
That would, I think, be correct. All zeros are the easiest to calculate - the parities (raid5 and raid6) are all zeros too. It is also the ideal pattern to write to SSDs - many SSDs these days implement transparent compression, and you don't get more compressible than zeros!
Quoted 6 lines
Would this open the door to some "thin provisioned" MD RAID, where one could grow the underlying devices (in the case of a RAID built ontop of say LVM devices), and marking the new "space" as "non-in-sync" without disrupting (slowing) operations on the array with a sync?
Yes, that would work. More importantly (because it would affect more people), it means that the creation of a md raid array on top of disks or partitions will immediately be "in sync", and there would be no need for a long and effectively useless re-sync process at creation.
Quoted 1 line
In any case, seems like a great feature.
Yes indeed.
Quoted 1 line
Quoted 14 lines
Quoted 8 lines
Regards,
Ben.Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlSignature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Thank you very much for your reply!
Regards,
Ben.
Quoted 9 lines · On Tue, 04 Sep 2012 12:06:26 -0600 Chris Friesen <[email protected]>…
Hi,
I'm not really a filesystem guy so this may be a really dumb question.
We currently have an issue where we have a ~1TB RAID1 array that is mostly given over to LVM. If we swap one of the disks it will rebuild everything, even though we may only be using a small fraction of the space.
Quoted 10 lines
This got me thinking. Has anyone given thought to using the TRIM information from filesystems to allow the RAID code to maintain a bitmask of used disk blocks and only sync the ones that are actually used?
Presumably this bitmask would itself need to be stored on the disk.
Thanks,
Chris
Something like this?
http://neil.brown.name/blog/20110216044002#5
NeilBrown
Quoted 10 lines · NeilBrown
Quoted 8 lines · On Tue, 04 Sep 2012 12:06:26 -0600 Chris Friesen<[email protected]>…
Hi,
I'm not really a filesystem guy so this may be a really dumb question.
We currently have an issue where we have a ~1TB RAID1 array that is mostly given over to LVM. If we swap one of the disks it will rebuild everything, even though we may only be using a small fraction of the space.
Quoted 7 lines
Quoted 5 lines
This got me thinking. Has anyone given thought to using the TRIM information from filesystems to allow the RAID code to maintain a bitmask of used disk blocks and only sync the ones that are actually used?
Presumably this bitmask would itself need to be stored on the disk.
Something like this?
http://neil.brown.name/blog/20110216044002#5
Something like that would indeed cover the use-case that triggered this.
Chris