thin provisioned LUN support

After talking to some vendors, one issue that came up is that the arrays all have a different size that is used internally to track the SCSI equivalent of TRIM commands (POKE/unmap).

What they would like is for us to coalesce these commands into aligned multiples of these chunks. If not, the target device will most likely ignore the bits at the beginning and end (and all small requests).

I have been thinking about whether or not we can (and should) do anything more than our current best effort to send down large chunks (note that the "chunk" size can range from reasonable sizes like 8KB or so up to close to 1MB!).

One suggestion is that a modified defrag sweep could be used periodically to update the device (a proposal I am not keen on).

Thoughts?

The way to do this properly would be to run a chequerboard of partials, but this would effectively have trim region tracking done in the block layer ... is this worth it?

By the way, the latest (from 2 days ago) version of the Thin Provisioning proposal is here:

http://www.t10.org/ftp/t10/document.08/08-149r4.pdf

I skimmed it but don't see any update implying that trim might be ineffective if we align wrongly ... where is this?

Quoted 7 lines · James Bottomley

The way to do this properly would be to run a chequerboard of partials, but this would effectively have trim region tracking done in the block layer ... is this worth it?

By the way, the latest (from 2 days ago) version of the Thin Provisioning proposal is here:

Quoted 4 lines

http://www.t10.org/ftp/t10/document.08/08-149r4.pdf

I skimmed it but don't see any update implying that trim might be ineffective if we align wrongly ... where is this?

I think we should be content to declare such devices 'broken'.

They have to keep track of individual sectors _anyway_, and dropping information for small discard requests is just careless.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Ric" == Ric Wheeler <[email protected]> writes:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

"Ric" == Ric Wheeler <[email protected]> writes:

Ric> Big arrays have an internal "track" size that is much larger than Ric> 512 bytes (Symm for example is 64k. Everything smaller than that Ric> is a read-modify-write. Effectively, they have few to no bits Ric> left to track this other bit of state at the smal level of Ric> granularity.

My point still stands that this is an implementation problem in the array firmware. It's not really our problem to solve. Especially since drive vendors and mid-range storage vendors are getting it right.

If EMC wants to provide thin provisioning on the Symmetrix they'll have to overcome the inherent limitations in their own internal architecture.

What's next? Requiring us to exclusively read and write in multiples of block sizes that are artifacts of internal array implementation details?

For thin provisioning to really work on the Symm, then, we're effectively requiring the filesystem to use 64k filesystem blocks. Lame!

I don't have a problem with honoring some bits akin to the block device characteristics VPD in trying to do a best-effort scheduling of the I/O. But effectively disabling thin provisioning for blocks smaller than that is simply broken.

I think we should be content to declare such devices 'broken'.

They have to keep track of individual sectors _anyway_, and dropping information for small discard requests is just careless.

Quoted 9 lines · James Bottomley

The way to do this properly would be to run a chequerboard of partials, but this would effectively have trim region tracking done in the block layer ... is this worth it?

By the way, the latest (from 2 days ago) version of the Thin Provisioning proposal is here:

http://www.t10.org/ftp/t10/document.08/08-149r4.pdf

Quoted 2 lines

I skimmed it but don't see any update implying that trim might be ineffective if we align wrongly ... where is this?

I think we should be content to declare such devices 'broken'.

They have to keep track of individual sectors _anyway_, and dropping information for small discard requests is just careless.

Quoted 9 lines · David Woodhouse

Quoted 7 lines · James Bottomley

The way to do this properly would be to run a chequerboard of partials, but this would effectively have trim region tracking done in the block layer ... is this worth it?

By the way, the latest (from 2 days ago) version of the Thin Provisioning proposal is here:

Quoted 9 lines

Quoted 5 lines

http://www.t10.org/ftp/t10/document.08/08-149r4.pdf

I skimmed it but don't see any update implying that trim might be ineffective if we align wrongly ... where is this?

I think we should be content to declare such devices 'broken'.

They have to keep track of individual sectors _anyway_, and dropping information for small discard requests is just careless.

I agree, seems pretty pointless. Lets let evolution take care of this issue. I have to say I'm surprised that it really IS an issue to begin with, are array firmwares really that silly?

It's not that it would be hard to support (and it would eliminate the need to do discard merging in the block layer), but it seems like one of those things that will be of little use in even in the near future. Discard merging should be useful, I have no problem merging something like that.

I think that discard merging would be helpful (especially for devices with more reasonable sized unmap chunks).

Quoted 26 lines · Jens Axboe

Quoted 11 lines · David Woodhouse

Quoted 9 lines · James Bottomley

The way to do this properly would be to run a chequerboard of partials, but this would effectively have trim region tracking done in the block layer ... is this worth it?

By the way, the latest (from 2 days ago) version of the Thin Provisioning proposal is here:

http://www.t10.org/ftp/t10/document.08/08-149r4.pdf

Quoted 8 lines

Quoted 3 lines

I skimmed it but don't see any update implying that trim might be ineffective if we align wrongly ... where is this?

I think we should be content to declare such devices 'broken'.

They have to keep track of individual sectors _anyway_, and dropping information for small discard requests is just careless.

I agree, seems pretty pointless. Lets let evolution take care of this issue. I have to say I'm surprised that it really IS an issue to begin with, are array firmwares really that silly?

Quoted 7 lines

It's not that it would be hard to support (and it would eliminate the need to do discard merging in the block layer), but it seems like one of those things that will be of little use in even in the near future. Discard merging should be useful, I have no problem merging something like that.

I think that discard merging would be helpful (especially for devices with more reasonable sized unmap chunks).

One more consideration that I should have mentioned is that we can also make our file system allocation policies "thin provisioned LUN" friendly.

Basically, we need to try to re-allocate blocks instead of letting the allocations happily progress across the entire block range. This might be the inverse of an SSD friendly allocation policy, but would seem to be fairly trivial to implement :-)

It's the opposite of a _flash_ friendly policy. But SSDs are not naive flash implementations -- if you overwrite a block, it'll just write elsewhere and update its internal mapping of LBAs to sectors. I

One more consideration that I should have mentioned is that we can also make our file system allocation policies "thin provisioned LUN" friendly.

Basically, we need to try to re-allocate blocks instead of letting the allocations happily progress across the entire block range. This might be the inverse of an SSD friendly allocation policy, but would seem to be fairly trivial to implement :-)

Quoted 4 lines · Ric Wheeler

One more consideration that I should have mentioned is that we can also make our file system allocation policies "thin provisioned LUN" friendly.

Basically, we need to try to re-allocate blocks instead of letting the allocations happily progress across the entire block range. This might be the inverse of an SSD friendly allocation policy, but would seem to be fairly trivial to implement :-)

I would think that most non log-structured filesystems do this by default.

The one thing we might need for SSD-friendly allocation policies is to tell the allocators to not try so hard to make sure allocations are contiguous, but there are other reasons why you want contiguous extents anyway (such as reducing the size of your extent tree and reducing the number of block allocation data structures that need to be updated). And, I think to some extent SSD's do care to some level about contiguous extents, from the point of view of reducing scatter gather operations if nothing else, right?

- Ted

The type of allocation that would help most is something that tries to keep the lower block ranges "hot" for allocation, second best policy would simply keep the allocated blocks in each block group hot and re-allocate them.

Quoted 1 line · Ric Wheeler

The type of allocation that would help most is something that tries to keep the lower block ranges "hot" for allocation, second best policy would simply keep the allocated blocks in each block group hot and re-allocate them.

This block reuse policy ignores the issue of wear leveling... as in most design things, trading one problem for another.

Quoted 6 lines · jim owens

Quoted 1 line · Ric Wheeler

The type of allocation that would help most is something that tries to keep the lower block ranges "hot" for allocation, second best policy would simply keep the allocated blocks in each block group hot and re-allocate them.

This block reuse policy ignores the issue of wear leveling... as in most design things, trading one problem for another.

For SSDs we're being told not to worry our pretty little heads about wear levelling. That gets done for us, with varying degrees of competence, within the black box. All we can do to improve that is pray... and maybe sacrifice the occasional goat.

I'm talking DISK wear not SSD. The array vendors who are causing this problem are doing petabyte san devices, not SSDs.

Rewriting the same sectors causes more bad block remaps until the drive eventually runs out of remap space.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Ted" == Theodore Tso <[email protected]> writes:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

"Ted" == Theodore Tso <[email protected]> writes:

Ted> How much of a disk wear factor is there with modern disk drives? Ted> The heads aren't touching the disk, and we have plenty of sectors Ted> which are constantly getting rewritten with traditional Ted> filesystems, with no ill effects as far as I know.

Modern disk firmware maintains a list of write hot spots and will regularly rewrite adjacent sectors to prevent bleed.

Quoted 6 lines · jim owens

I'm talking DISK wear not SSD. The array vendors who are causing this problem are doing petabyte san devices, not SSDs.

Rewriting the same sectors causes more bad block remaps until the drive eventually runs out of remap space.

How much of a disk wear factor is there with modern disk drives? The heads aren't touching the disk, and we have plenty of sectors which are constantly getting rewritten with traditional filesystems, with no ill effects as far as I know. For example, FAT filesystems, the superblock, block allocation bitmaps all are constantly getting rewritten today, and I haven't heard of disk manufacturers complaining that this is a horrible thing.

Quoted 7 lines · jim owens

Quoted 4 lines · Ric Wheeler

The type of allocation that would help most is something that tries to keep the lower block ranges "hot" for allocation, second best policy would simply keep the allocated blocks in each block group hot and re-allocate them.

Quoted 2 lines

This block reuse policy ignores the issue of wear leveling... as in most design things, trading one problem for another.

For SSDs we're being told not to worry our pretty little heads about wear levelling. That gets done for us, with varying degrees of competence, within the black box. All we can do to improve that is pray... and maybe sacrifice the occasional goat.

If we're being shown native flash with no intervening disk interface then, yes, we need to do wear levelling (although I suspect this will really only occur in the embedded space).

Quoted 4 lines · Ric Wheeler

The type of allocation that would help most is something that tries to keep the lower block ranges "hot" for allocation, second best policy would simply keep the allocated blocks in each block group hot and re-allocate them.

This block reuse policy ignores the issue of wear leveling... as in most design things, trading one problem for another.

The discussion here has been around Intel-style SSD's, which apparently have a log-structured filesystem in the device, such that wear leveling is done automatically, and in fact it is *better* for these devices if we reuse the same block since then the SSD automatically knows that contents at the old location is logically "gone". (I don't believe, or at least don't see, why there would be any benefit of reusing block ranges versus explicitly using a TRIM command to tell the SSD that the old block was no longer being used; it should have the same effect as far as the SSD is concerned.)

The one thing which I am somewhat concerned about is whether all SSD's will be doing things the Intel way, or whether other SSD's might not

We have become confused and combined 2 independent issues...

UNMAP thin storage array provisioning is not SSD TRIM.

Quoted 3 lines · jim owens

We have become confused and combined 2 independent issues...

UNMAP thin storage array provisioning is not SSD TRIM.

Actually, currently they are.

The discussion here has been around Intel-style SSD's, which apparently have a log-structured filesystem in the device, such that wear leveling is done automatically, and in fact it is *better* for these devices if we reuse the same block since then the SSD automatically knows that contents at the old location is logically "gone". (I don't believe, or at least don't see, why there would be any benefit of reusing block ranges versus explicitly using a TRIM command to tell the SSD that the old block was no longer being used; it should have the same effect as far as the SSD is concerned.)

The one thing which I am somewhat concerned about is whether all SSD's will be doing things the Intel way, or whether other SSD's might not

Quoted 1 line · Ric Wheeler

The type of allocation that would help most is something that tries to keep the lower block ranges "hot" for allocation, second best policy would simply keep the allocated blocks in each block group hot and re-allocate them.

This block reuse policy ignores the issue of wear leveling... as in most design things, trading one problem for another.

jim

Quoted 4 lines · Ric Wheeler

One more consideration that I should have mentioned is that we can also make our file system allocation policies "thin provisioned LUN" friendly.

Basically, we need to try to re-allocate blocks instead of letting the allocations happily progress across the entire block range. This might be the inverse of an SSD friendly allocation policy, but would seem to be fairly trivial to implement :-)

I would think that most non log-structured filesystems do this by default.

The one thing we might need for SSD-friendly allocation policies is to tell the allocators to not try so hard to make sure allocations are contiguous, but there are other reasons why you want contiguous extents anyway (such as reducing the size of your extent tree and reducing the number of block allocation data structures that need to be updated). And, I think to some extent SSD's do care to some level about contiguous extents, from the point of view of reducing scatter gather operations if nothing else, right?

- Ted

The one thing we might need for SSD-friendly allocation policies is to tell the allocators to not try so hard to make sure allocations are contiguous, but there are other reasons why you want contiguous extents anyway (such as reducing the size of your extent tree and reducing the number of block allocation data structures that need to be updated). And, I think to some extent SSD's do care to some level about contiguous extents, from the point of view of reducing scatter gather operations if nothing else, right?

The one thing we might need for SSD-friendly allocation policies is to tell the allocators to not try so hard to make sure allocations are contiguous, but there are other reasons why you want contiguous extents anyway (such as reducing the size of your extent tree and reducing the number of block allocation data structures that need to be updated). And, I think to some extent SSD's do care to some level about contiguous extents, from the point of view of reducing scatter gather operations if nothing else, right?

Quoted 26 lines · Jens Axboe

Quoted 11 lines · David Woodhouse

Quoted 9 lines · James Bottomley

The way to do this properly would be to run a chequerboard of partials, but this would effectively have trim region tracking done in the block layer ... is this worth it?

By the way, the latest (from 2 days ago) version of the Thin Provisioning proposal is here:

http://www.t10.org/ftp/t10/document.08/08-149r4.pdf

Quoted 8 lines

Quoted 3 lines

I skimmed it but don't see any update implying that trim might be ineffective if we align wrongly ... where is this?

I think we should be content to declare such devices 'broken'.

They have to keep track of individual sectors _anyway_, and dropping information for small discard requests is just careless.

I agree, seems pretty pointless. Lets let evolution take care of this issue. I have to say I'm surprised that it really IS an issue to begin with, are array firmwares really that silly?

Quoted 7 lines

It's not that it would be hard to support (and it would eliminate the need to do discard merging in the block layer), but it seems like one of those things that will be of little use in even in the near future. Discard merging should be useful, I have no problem merging something like that.

I think that discard merging would be helpful (especially for devices with more reasonable sized unmap chunks).

Quoted 3 lines · Ric Wheeler

Jens Axboe wrote:
I think that discard merging would be helpful (especially for devices with more reasonable sized unmap chunks).

One of the ways the unmap command is set up is with a disjoint scatterlist, so we can send a large number of unmaps together. Whether they're merged or not really doesn't matter.

The probable way a discard system would work if we wanted to endure the complexity would be to have the discard system in the underlying device driver (or possibly just above it in block, but different devices like SCSI or ATA have different discard characteristics). It would just accumulate block discard requests as ranges (and it would have to poke holes in the ranges as it sees read/write requests) which it flushes periodically.

Quoted 9 lines · James Bottomley

Quoted 3 lines · Ric Wheeler

Jens Axboe wrote:
I think that discard merging would be helpful (especially for devices with more reasonable sized unmap chunks).

One of the ways the unmap command is set up is with a disjoint scatterlist, so we can send a large number of unmaps together. Whether they're merged or not really doesn't matter.

Quoted 7 lines

The probable way a discard system would work if we wanted to endure the complexity would be to have the discard system in the underlying device driver (or possibly just above it in block, but different devices like SCSI or ATA have different discard characteristics). It would just accumulate block discard requests as ranges (and it would have to poke holes in the ranges as it sees read/write requests) which it flushes periodically.

It appears to me that discard requests are only being considered here at a block and device level, and nobody is thinking about the system level effects of such aggregation of discard requests.

What happens on a system crash? We lose all the pending discard requests, never to be sent again?

If so, how do we tell the device
that certain ranges have actually been discarded after the crash? Are you expecting them to get replayed by a filesystem during recovery? What if it was a userspace discard from something like mkfs that was lost? How does this interact with sync or other such user level filesystems synchronisation primitives? Does sync_blockdev() flush out pending discard requests? Should fsync?

And if the filesystem has to wait for discard requests to complete to guarantee that they are done or can be recovered and replayed after a crash, most filesystems are going to need modification. e.g. XFS would need to prevent the tail of the log moving forward until the discard request associated with a given extent free transaction has been completed. That means we need to be able to specifically flush queued discard requests and we'd need I/O completions to run when they are done to do the filesytem level cleanup work....

Let's keep the OS level interactions simple - if the array vendors want to keep long queues of requests around before acting on them to aggregate them, then that is an optimisation for them to implement. They already do this with small data writes to NVRAM, so I don't see how this should be treated any differently...

Quoted 18 lines · Dave Chinner

Quoted 9 lines · James Bottomley

Quoted 3 lines · Ric Wheeler

Jens Axboe wrote:
I think that discard merging would be helpful (especially for devices with more reasonable sized unmap chunks).

One of the ways the unmap command is set up is with a disjoint scatterlist, so we can send a large number of unmaps together. Whether they're merged or not really doesn't matter.

Quoted 7 lines

The probable way a discard system would work if we wanted to endure the complexity would be to have the discard system in the underlying device driver (or possibly just above it in block, but different devices like SCSI or ATA have different discard characteristics). It would just accumulate block discard requests as ranges (and it would have to poke holes in the ranges as it sees read/write requests) which it flushes periodically.

Quoted 6 lines

It appears to me that discard requests are only being considered here at a block and device level, and nobody is thinking about the system level effects of such aggregation of discard requests.

What happens on a system crash? We lose all the pending discard requests, never to be sent again?

Yes ... since this is for thin provisioning. Discard is best guess ... it doesn't affect integrity if we lose one and from the point of view of the array, 99% transmitted is far better than we do today. All that happens for a lost discard is that the array keeps a block that the filesystem isn't currently using. However, the chances are that it will get reused, so it shares a good probability of getting discarded again.

Quoted 7 lines

If so, how do we tell the device
that certain ranges have actually been discarded after the crash? Are you expecting them to get replayed by a filesystem during recovery? What if it was a userspace discard from something like mkfs that was lost? How does this interact with sync or other such user level filesystems synchronisation primitives? Does sync_blockdev() flush out pending discard requests? Should fsync?

No .. the syncs are all integrity based. Discard is simple opportunity based.

Quoted 8 lines

And if the filesystem has to wait for discard requests to complete to guarantee that they are done or can be recovered and replayed after a crash, most filesystems are going to need modification. e.g. XFS would need to prevent the tail of the log moving forward until the discard request associated with a given extent free transaction has been completed. That means we need to be able to specifically flush queued discard requests and we'd need I/O completions to run when they are done to do the filesytem level cleanup work....

OK, I really don't follow the logic here. Discards have no effect on data integrity ... unless you're confusing them with secure deletion?

A
discard merely tells the array that it doesn't need to back this block with an actual storage location anymore (until the next write for that region comes down).

The ordering worry can be coped with in the same way we do barriers ... it's even safer for discards because if we know the block is going to be rewritten, we simply discard the discard.

Quoted 5 lines

Let's keep the OS level interactions simple - if the array vendors want to keep long queues of requests around before acting on them to aggregate them, then that is an optimisation for them to implement. They already do this with small data writes to NVRAM, so I don't see how this should be treated any differently...

Well, that's Chris' argument, and it has merit. I'm coming from the point of view that discards are actually a fundamentally different entity from anything else we process.

From a filesystem perspective, they are no different to any other

Quoted 6 lines

Yes ... since this is for thin provisioning. Discard is best guess ... it doesn't affect integrity if we lose one and from the point of view of the array, 99% transmitted is far better than we do today. All that happens for a lost discard is that the array keeps a block that the filesystem isn't currently using. However, the chances are that it will get reused, so it shares a good probability of getting discarded again.

Ok. Given that a single extent free in XFS could span up to 2^37 bytes, is it considered acceptible to lose the discard request that this issued from this transaction? I don't think it is....

Quoted 10 lines

Quoted 7 lines

If so, how do we tell the device
that certain ranges have actually been discarded after the crash? Are you expecting them to get replayed by a filesystem during recovery? What if it was a userspace discard from something like mkfs that was lost? How does this interact with sync or other such user level filesystems synchronisation primitives? Does sync_blockdev() flush out pending discard requests? Should fsync?

No .. the syncs are all integrity based. Discard is simple opportunity based.

Given that discard requests modify the stable storage associated with the filesystem, then shouldn't an integrity synchronisation issue and complete all pending requests to the underlying storage device?

If not, how do we guarantee them to all be flushed on remount-ro or unmount-before-hot-unplug type of events?

Quoted 11 lines

Quoted 8 lines

And if the filesystem has to wait for discard requests to complete to guarantee that they are done or can be recovered and replayed after a crash, most filesystems are going to need modification. e.g. XFS would need to prevent the tail of the log moving forward until the discard request associated with a given extent free transaction has been completed. That means we need to be able to specifically flush queued discard requests and we'd need I/O completions to run when they are done to do the filesytem level cleanup work....

OK, I really don't follow the logic here. Discards have no effect on data integrity ... unless you're confusing them with secure deletion?

Not at all. I'm considering what is needed to allow the filesystem's discard requests to be replayed during recovery. i.e. what is needed to allow a filesystem to handle discard requests for thin provisioning robustly.

If discard requests are not guaranteed to be issued to the storage on a crash, then it is up to the filesystem to ensure that it happens during recovery. That requires discard requests to behave just like all other types of I/O and definitely requires a mechanism to flush and wait for all discard requests to complete....

Quoted 4 lines

A
discard merely tells the array that it doesn't need to back this block with an actual storage location anymore (until the next write for that region comes down).

Right. But really, it's the filesystem that is saying this, not the block layer, so if the filesytem wants to be robust, then block layer can't queue these forever - they have to be issued in a timely fashion so the filesystem can keep track of which discards have completed or not....

Quoted 3 lines

The ordering worry can be coped with in the same way we do barriers ... it's even safer for discards because if we know the block is going to be rewritten, we simply discard the discard.

Ordering is determined by the filesystem - barriers are just a mechanism the filesystem uses to guarantee I/O ordering. If the filesystem is tracking discard completion status, then it won't be issuing I/O over the top of that region as the free transaction won't be complete until the discard is done....

Quoted 6 lines

Quoted 5 lines

Let's keep the OS level interactions simple - if the array vendors want to keep long queues of requests around before acting on them to aggregate them, then that is an optimisation for them to implement. They already do this with small data writes to NVRAM, so I don't see how this should be treated any differently...

Quoted 5 lines

Well, that's Chris' argument, and it has merit. I'm coming from the point of view that discards are actually a fundamentally different entity from anything else we process.

From a filesystem perspective, they are no different to any other

metadata I/O. They need to be tracked to allow robust crash recovery semantics to be implemented in the filesystem.

Quoted 7 lines

Quoted 2 lines

I skimmed it but don't see any update implying that trim might be ineffective if we align wrongly ... where is this?

I think we should be content to declare such devices 'broken'.

They have to keep track of individual sectors _anyway_, and dropping information for small discard requests is just careless.

I agree, seems pretty pointless. Lets let evolution take care of this issue. I have to say I'm surprised that it really IS an issue to begin with, are array firmwares really that silly?

It's not that it would be hard to support (and it would eliminate the need to do discard merging in the block layer), but it seems like one of those things that will be of little use in even in the near future. Discard merging should be useful, I have no problem merging something like that.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Chris" == Chris Mason <[email protected]> writes:

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Chris" == Chris Mason <[email protected]> writes:

Chris> Hmmm, it's surprising to me that arrays who tell us please use Chris> the noop elevator suddenly want us to merge discard requests. Chris> The array really needs to be able to deal with this internally.

Let's also not forget that we're talking about merging discard requests for the purpose making internal array housekeeping efficient. That involves merging discards up to the internal array block sizes which may be on the order of 512/768/1024 KB.

If we were talking about merging discards up to a 4/8/16 KB boundary that might be something we'd have a chance to do within a reasonable amount of time (bigger than normal read/write I/O but not hours).

But keeping discard state around for long enough to attempt to aggregate 768KB (and 768KB-aligned) chunks is icky.

Quoted 11 lines · Martin K. Petersen

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Chris" == Chris Mason <[email protected]> writes:

Chris> Hmmm, it's surprising to me that arrays who tell us please use Chris> the noop elevator suddenly want us to merge discard requests. Chris> The array really needs to be able to deal with this internally.

Let's also not forget that we're talking about merging discard requests for the purpose making internal array housekeeping efficient. That involves merging discards up to the internal array block sizes which may be on the order of 512/768/1024 KB.

Quoted 6 lines

If we were talking about merging discards up to a 4/8/16 KB boundary that might be something we'd have a chance to do within a reasonable amount of time (bigger than normal read/write I/O but not hours).

But keeping discard state around for long enough to attempt to aggregate 768KB (and 768KB-aligned) chunks is icky.

Icky but possible. It's the same rb tree affair we use to keep vma lists (with the same characteristics). The point is that technically we can do this pretty easily ... all the way down to not losing any potential discards that the array would ignore. However, procedurally it would certainly be sending the wrong message to the array vendors (the message being "sure the OS will sanitise any crap you care to dump").

On the other hand, if we have to do it for flash and MMC anyway ...

Quoted 27 lines · James Bottomley

Quoted 11 lines · Martin K. Petersen

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Chris" == Chris Mason <[email protected]> writes:

Chris> Hmmm, it's surprising to me that arrays who tell us please use Chris> the noop elevator suddenly want us to merge discard requests. Chris> The array really needs to be able to deal with this internally.

Let's also not forget that we're talking about merging discard requests for the purpose making internal array housekeeping efficient. That involves merging discards up to the internal array block sizes which may be on the order of 512/768/1024 KB.

Quoted 6 lines

If we were talking about merging discards up to a 4/8/16 KB boundary that might be something we'd have a chance to do within a reasonable amount of time (bigger than normal read/write I/O but not hours).

But keeping discard state around for long enough to attempt to aggregate 768KB (and 768KB-aligned) chunks is icky.

Icky but possible. It's the same rb tree affair we use to keep vma lists (with the same characteristics). The point is that technically we can do this pretty easily ... all the way down to not losing any potential discards that the array would ignore. However, procedurally it would certainly be sending the wrong message to the array vendors (the message being "sure the OS will sanitise any crap you care to dump").

Quoted 1 line

On the other hand, if we have to do it for flash and MMC anyway ...

It doesn't seem like a good idea to maintain a ton of code that gets exercised so rarely, especially wrt filesystem crashes.

Just testing it would be a fairly large challenge, spread out across N filesystems. I think we need to keep discard as simple as we possibly can.

Quoted 30 lines · Chris Mason

Quoted 28 lines · James Bottomley

Quoted 11 lines · Martin K. Petersen

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

"Chris" == Chris Mason <[email protected]> writes:

Chris> Hmmm, it's surprising to me that arrays who tell us please use Chris> the noop elevator suddenly want us to merge discard requests. Chris> The array really needs to be able to deal with this internally.

Let's also not forget that we're talking about merging discard requests for the purpose making internal array housekeeping efficient. That involves merging discards up to the internal array block sizes which may be on the order of 512/768/1024 KB.

Quoted 7 lines

If we were talking about merging discards up to a 4/8/16 KB boundary that might be something we'd have a chance to do within a reasonable amount of time (bigger than normal read/write I/O but not hours).

But keeping discard state around for long enough to attempt to aggregate 768KB (and 768KB-aligned) chunks is icky.

Icky but possible. It's the same rb tree affair we use to keep vma lists (with the same characteristics). The point is that technically we can do this pretty easily ... all the way down to not losing any potential discards that the array would ignore. However, procedurally it would certainly be sending the wrong message to the array vendors (the message being "sure the OS will sanitise any crap you care to dump").

Quoted 5 lines

Quoted 2 lines

On the other hand, if we have to do it for flash and MMC anyway ...

It doesn't seem like a good idea to maintain a ton of code that gets exercised so rarely, especially wrt filesystem crashes.

Heh, am I the only person here who deletes files on a regular basis (principally to get my disk down from 99%)?

Quoted 4 lines

Just testing it would be a fairly large challenge, spread out across N filesystems. I think we need to keep discard as simple as we possibly can.

I don't disagree with that ... I'm not saying we *should* merely that we *could*.

James

Quoted 7 lines · James Bottomley

Quoted 4 lines

Just testing it would be a fairly large challenge, spread out across N filesystems. I think we need to keep discard as simple as we possibly can.

I don't disagree with that ... I'm not saying we *should* merely that we *could*.

I agree that simple and robust are key, but we will need to try and do reasonable coalescing of the requests.

Depending on how vendors implement those unmap commands, sending down a sequence of commands might cause a performance issue if done at too fine a granularity. Easiest way to handle that is to make sure that we have a way of disabling the unmap/discard support (mount option?).

Depending on how vendors implement those unmap commands, sending down a sequence of commands might cause a performance issue if done at too fine a granularity. Easiest way to handle that is to make sure that we have a way of disabling the unmap/discard support (mount option?).

On Fri, 2008-11-07 at 11:11 -0500, Chris Mason wrote:

Quoted 3 lines

It doesn't seem like a good idea to maintain a ton of code that gets exercised so rarely, especially wrt filesystem crashes.

Heh, am I the only person here who deletes files on a regular basis (principally to get my disk down from 99%)?

Quoted 2 lines · James Bottomley

On Fri, 2008-11-07 at 11:11 -0500, Chris Mason wrote:

[ complex trim management code ]

Quoted 6 lines

Quoted 3 lines

It doesn't seem like a good idea to maintain a ton of code that gets exercised so rarely, especially wrt filesystem crashes.

Heh, am I the only person here who deletes files on a regular basis (principally to get my disk down from 99%)?

Using it is easy, but the failure case is the storage forgets about a block the FS cares about, so actually testing the code means we have to test all the blocks allocated to the FS to make sure they have the correct values. Given remapping and everything else we've talked about, any block in the FS could become corrupt due to a trim bug on block X.

So, trim bugs will look like a dozen other bugs, and they may not get reported for months after the actual bug was triggered (all things I know you already know ;)

I think the bar for keeping it simple is much higher here than in other parts of the storage stack.

-chris

I don't think that trim bugs should be that common - we just have to be very careful never to send down a trim for any uncommitted block.

Simple is always good, but I still think that the coalescing (even basic coalescing) will be a critical performance feature.

Quoted 3 lines · Ric Wheeler

I don't think that trim bugs should be that common - we just have to be very careful never to send down a trim for any uncommitted block.

The trim code probably deserves a very aggressive unit test to make sure it works correctly, but yeah, we should be able to control any trim bugs.

Quoted 2 lines

Simple is always good, but I still think that the coalescing (even basic coalescing) will be a critical performance feature.

Will we be able to query the device and find out its TRIM/UNMAP alignment requirements? There is also a balanace between performance (at least if the concern is sending too many separate TRIM commands) and giving the SSD more flexibility in its wear-leveling allocation decisions by sending TRIM commands sooner rather than later.

- Ted

Quoted 6 lines

Will we be able to query the device and find out its TRIM/UNMAP alignment requirements? There is also a balanace between performance (at least if the concern is sending too many separate TRIM commands) and giving the SSD more flexibility in its wear-leveling allocation decisions by sending TRIM commands sooner rather than later.

T10 is still working on the proposal for how to display unmap related information for SCSI, so we don't even have a consistent way to find this out today for this population.

Not sure what is possible for the ATA devices,

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Ted" == Theodore Tso <[email protected]> writes:

The current UNMAP proposal in SCSI doesn't have requirements either.

Array vendors, suddenly realizing all the work they have to do to support this, are now talking about imposing additional constraints (orthogonal to the UNMAP command set) because of limitations in their existing firmware architectures.

Let's be just a *little* bit fair here. Suppose we wanted to implement thin-provisioned disks using devicemapper and LVM; consider that LVM uses a default PE size of 4M for some very good reasons. Asking filesystems to be a little smarter about allocation policies so that we allocate in existing 4M chunks before going onto the next, and asking the block layer to pool trim requests to 4M chunks is not totally unreasonable.

Array vendors use chunk sizes > than typical filesystem chunk sizes for the same reason that LVM does. So to say that this is due to purely a "broken firmware architecture" is a little unfair.

Quoted 8 lines · Theodore Tso

Let's be just a *little* bit fair here. Suppose we wanted to implement thin-provisioned disks using devicemapper and LVM; consider that LVM uses a default PE size of 4M for some very good reasons. Asking filesystems to be a little smarter about allocation policies so that we allocate in existing 4M chunks before going onto the next, and asking the block layer to pool trim requests to 4M chunks is not totally unreasonable.

Quoted 4 lines

Array vendors use chunk sizes > than typical filesystem chunk sizes for the same reason that LVM does. So to say that this is due to purely a "broken firmware architecture" is a little unfair.

I think we would have a full-throated discussion about whether the right thing to do was to put the tracking in the block layer or in LVM. Rather similar to what we're doing now, in fact.

Quoted 10 lines · Matthew Wilcox

Quoted 8 lines · Theodore Tso

Let's be just a *little* bit fair here. Suppose we wanted to implement thin-provisioned disks using devicemapper and LVM; consider that LVM uses a default PE size of 4M for some very good reasons. Asking filesystems to be a little smarter about allocation policies so that we allocate in existing 4M chunks before going onto the next, and asking the block layer to pool trim requests to 4M chunks is not totally unreasonable.

Quoted 9 lines

Quoted 4 lines

Array vendors use chunk sizes > than typical filesystem chunk sizes for the same reason that LVM does. So to say that this is due to purely a "broken firmware architecture" is a little unfair.

I think we would have a full-throated discussion about whether the right thing to do was to put the tracking in the block layer or in LVM. Rather similar to what we're doing now, in fact.

You definitely could imagine having a device mapper target that could track the discards commands and subsequent writes which would invalidate the previous discards.

Actually, it would be kind of nice to move all of this away from the file systems entirely.

Quoted 18 lines · Ric Wheeler

Quoted 16 lines · Matthew Wilcox

Quoted 13 lines · Theodore Tso

Let's be just a *little* bit fair here. Suppose we wanted to implement thin-provisioned disks using devicemapper and LVM; consider that LVM uses a default PE size of 4M for some very good reasons. Asking filesystems to be a little smarter about allocation policies so that we allocate in existing 4M chunks before going onto the next, and asking the block layer to pool trim requests to 4M chunks is not totally unreasonable.

Array vendors use chunk sizes > than typical filesystem chunk sizes for the same reason that LVM does. So to say that this is due to purely a "broken firmware architecture" is a little unfair.

Quoted 9 lines

Quoted 5 lines

I think we would have a full-throated discussion about whether the right thing to do was to put the tracking in the block layer or in LVM. Rather similar to what we're doing now, in fact.

You definitely could imagine having a device mapper target that could track the discards commands and subsequent writes which would invalidate the previous discards.

Actually, it would be kind of nice to move all of this away from the file systems entirely.

* Fast
* Crash safe
* Bounded ram usage
* Accurately deliver the trims

Pick any three ;) If we're dealing with large files, I can see it working well. For files that are likely to be smaller than the physical extent size, you end up with either extra state bits on disk (and keeping them in sync) or a log structured lvm.

I do agree that an offline tool to account for bytes used would be able to make up for this, and from a thin provisioning point of view, we might be better off if we don't accurately deliver all the trims all the time.

People just use the space again soon anyway, I'd have to guess the filesystems end up in a steady state outside of special events.

In another email Ted mentions that it makes sense for the FS allocator to notice we've just freed the last block in an aligned region of size X, and I'd agree with that.

The trim command we send down when we free the block could just contain the entire range that is free (and easy for the FS to determine) every time.

-chris

* Fast
* Crash safe
* Bounded ram usage
* Accurately deliver the trims

Pick any three ;)

I think we would have a full-throated discussion about whether the right thing to do was to put the tracking in the block layer or in LVM. Rather similar to what we're doing now, in fact.

Let's be just a *little* bit fair here. Suppose we wanted to implement thin-provisioned disks using devicemapper and LVM; consider that LVM uses a default PE size of 4M for some very good reasons. Asking filesystems to be a little smarter about allocation policies so that we allocate in existing 4M chunks before going onto the next, and asking the block layer to pool trim requests to 4M chunks is not totally unreasonable.

Array vendors use chunk sizes > than typical filesystem chunk sizes for the same reason that LVM does. So to say that this is due to purely a "broken firmware architecture" is a little unfair.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Ted" == Theodore Tso <[email protected]> writes:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

"Ted" == Theodore Tso <[email protected]> writes:

Ted> I thought ATA didn't have any TRIM alignment requirements, and Ted> it's T10 that wants to add it to the SCSI side?

The current UNMAP proposal in SCSI doesn't have requirements either.

Array vendors, suddenly realizing all the work they have to do to support this, are now talking about imposing additional constraints (orthogonal to the UNMAP command set) because of limitations in their existing firmware architectures.

It obviously much easier for the array vendors to export a Somebody Elses Problem VPD page containing a constant than it is to fix inherent limitations in their internal architecture.

We're trying to point out that that's an unacceptable cop out for something that's clearly their problem to deal with.

My concern is that if we start doing the array people's homework at the OS level they won't be inclined to fix their broken firmware design. Ever.

Quoted 4 lines

Ted> I thought ATA didn't have any TRIM alignment requirements, and Ted> it's T10 that wants to add it to the SCSI side?

The current UNMAP proposal in SCSI doesn't have requirements either.

I think that is being actively debated & due to go out in the next update. Not sure that it will matter much since vendors will not have implemented it yet in their boxes (at least, not yet)....

Quoted 2 lines

Quoted 2 lines

Will we be able to query the device and find out its TRIM/UNMAP alignment requirements? There is also a balance between

Quoted 1 line

Quoted 1 line

(at least if the concern is sending too many separate TRIM

Quoted 4 lines

Quoted 3 lines

and giving the SSD more flexibility in its wear-leveling allocation decisions by sending TRIM commands sooner rather than later.

T10 is still working on the proposal for how to display unmap

Quoted 3 lines

information for SCSI, so we don't even have a consistent way to find

this out today for this population.

Yeah, I know, the rhetorical question was mostly addressed at David Black. :-)

Quoted 3 lines · Ric Wheeler

I don't think that trim bugs should be that common - we just have to be very careful never to send down a trim for any uncommitted block.

The trim code probably deserves a very aggressive unit test to make sure it works correctly, but yeah, we should be able to control any trim bugs.

Quoted 2 lines

Simple is always good, but I still think that the coalescing (even basic coalescing) will be a critical performance feature.

Will we be able to query the device and find out its TRIM/UNMAP alignment requirements? There is also a balanace between performance (at least if the concern is sending too many separate TRIM commands) and giving the SSD more flexibility in its wear-leveling allocation decisions by sending TRIM commands sooner rather than later.

- Ted

Quoted 2 lines · Ric Wheeler

I don't think that trim bugs should be that common - we just have to be very careful never to send down a trim for any uncommitted block.

The trim code probably deserves a very aggressive unit test to make sure it works correctly, but yeah, we should be able to control any trim bugs.

Quoted 1 line

Simple is always good, but I still think that the coalescing (even basic coalescing) will be a critical performance feature.

Will we be able to query the device and find out its TRIM/UNMAP alignment requirements? There is also a balanace between performance (at least if the concern is sending too many separate TRIM commands) and giving the SSD more flexibility in its wear-leveling allocation decisions by sending TRIM commands sooner rather than later.

To do that you kind of have to build a filesystem into the block layer to persistently store "mapped/unmapped blocks in chunk" and then do the "unmap-this-chunk" when a region is all unmapped.

250 MB per 1TiB 512b sector disk for a simple 1-bit-per-sector state. And that assumes you don't replicate it for safety. That is what the array vendors are trying to avoid by pushing it off to the OS.

Whoever supports thin provisioning better get their unmapping correct because those big customers will be looking for who to blame if they don't get all the features.

Quoted 8 lines · Theodore Tso

Quoted 2 lines · Ric Wheeler

I don't think that trim bugs should be that common - we just have to be very careful never to send down a trim for any uncommitted block.

The trim code probably deserves a very aggressive unit test to make sure it works correctly, but yeah, we should be able to control any trim bugs.

Quoted 7 lines

Quoted 1 line

Simple is always good, but I still think that the coalescing (even basic coalescing) will be a critical performance feature.

Will we be able to query the device and find out its TRIM/UNMAP alignment requirements? There is also a balanace between performance (at least if the concern is sending too many separate TRIM commands) and giving the SSD more flexibility in its wear-leveling allocation decisions by sending TRIM commands sooner rather than later.

This is all good if the design is bounded by the requirements of trim for flash devices. Because AFAIK the use of trim for flash ssd is a performance optimization. The ssd won't loose functionality if the trim is less than the chunk size. It may run slower and wear out faster, but that is all.

If I understand correctly, with thin provisioning, unmapping less than the chunk will not release that chunk for other use. So you have lost the thin provision feature of the array.

The concern (Chris I think) and I have is that doing a design to handle thin provision arrays *when chunk > fs_block_size* that guarantees you will *always* release on chunk boundaries is a lot more complicated.

To do that you kind of have to build a filesystem into the block layer to persistently store "mapped/unmapped blocks in chunk" and then do the "unmap-this-chunk" when a region is all unmapped.

250 MB per 1TiB 512b sector disk for a simple 1-bit-per-sector state. And that assumes you don't replicate it for safety. That is what the array vendors are trying to avoid by pushing it off to the OS.

Whoever supports thin provisioning better get their unmapping correct because those big customers will be looking for who to blame if they don't get all the features.

jim

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Chris" == Chris Mason <[email protected]> writes:

Chris> Hmmm, it's surprising to me that arrays who tell us please use Chris> the noop elevator suddenly want us to merge discard requests. Chris> The array really needs to be able to deal with this internally.

Let's also not forget that we're talking about merging discard requests for the purpose making internal array housekeeping efficient. That involves merging discards up to the internal array block sizes which may be on the order of 512/768/1024 KB.

If we were talking about merging discards up to a 4/8/16 KB boundary that might be something we'd have a chance to do within a reasonable amount of time (bigger than normal read/write I/O but not hours).

But keeping discard state around for long enough to attempt to aggregate 768KB (and 768KB-aligned) chunks is icky.

It also ignores the fact that as the filesystem ages they will have fewer and fewer aligned free chunks as the free space fragments. Over time, arrays using large allocation chunks are simply going to be full of wasted space as filesystem allocation patterns degrade if the array vendors ignore this problem.

And no matter what us filesystem developers do, there is always going to be degradation in allocation patterns as the filesystems fill up and age. While we can try to improve aging behaviour, it doesn't solve the problem for array vendors - they need to be smarter about their allocation and block mapping....

In another email Ted mentions that it makes sense for the FS allocator to notice we've just freed the last block in an aligned region of size X, and I'd agree with that.

The trim command we send down when we free the block could just

the entire range that is free (and easy for the FS to determine) every time.

Wow, this discussion can generate a lot of traffic .... I think Dave Chinner's recent message is as good a place as any to start:

Quoted 5 lines

It also ignores the fact that as the filesystem ages they will have fewer and fewer aligned free chunks as the free space fragments. Over time, arrays using large allocation chunks are simply going to be full of wasted space as filesystem allocation patterns degrade if the array vendors ignore this problem.

My take on this is that I agree with Ric's comment:

Quoted 5 lines

In another email Ted mentions that it makes sense for the FS allocator to notice we've just freed the last block in an aligned region of size X, and I'd agree with that.

The trim command we send down when we free the block could just

contain

Quoted 2 lines

the entire range that is free (and easy for the FS to determine) every time.

In other words, the filesystem ought to do a small amount of work to send down the largest (reasonable) range that it knows is free - this seems likely to be more effective than relying on the elevators to make this happen.

There will be a chunk size value available in a VPD page that can be used to determine minimum size/alignment. For openers, I see essentially
no point in a 512-byte UNMAP, even though it's allowed by the standard - I suspect most arrays (and many SSDs) will ignore it, and ignoring it is definitely within the spirit of the proposed T10 standard (hint:
I'm one of the people directly working on that proposal).

OTOH, it
may not be possible to frequently assemble large chunks for arrays that use them, and I agree with Dave Chinner's remarks on free space fragmentation (quoted above) - there's no "free lunch" there.

Beyond this, I think there may be an underlying assumption that the array and filesystem ought to be in sync (or close to it) on based on overhead and diminishing marginal returns of trying to get ever- closer sync.

Elsewhere, mention has been made of having the filesystem's free list behavior be LIFO-like, rather than constantly allocating new blocks from previously-unused space. IMHO, that's a good idea for thin provisioning.

Now, if the workload running on the filesystem causes the capacity used to stay within a range, there will be a set of relatively "hot" blocks on the free list that are being frequently freed and reallocated. It's a performance win not to UNMAP those blocks (saves work in both the kernel and on the array), and hence to have the filesystem and array views of what's in use not line up.

Despite the derogatory comment about defrag, it's making a comeback. I already know of two thin-provisioning-specific defrag utilities from other storage vendors (neither is for Linux, AFAIK). While defrag is not a wonderful solution, it does free up space in large contiguous ranges, and most of what it frees will be "cold".

Quoted 7 lines · On Sun, Nov 09, 2008 at 10:40:24PM -0500, [email protected] wrote:

There will be a chunk size value available in a VPD page that can be used to determine minimum size/alignment. For openers, I see essentially
no point in a 512-byte UNMAP, even though it's allowed by the standard - I suspect most arrays (and many SSDs) will ignore it, and ignoring it is definitely within the spirit of the proposed T10 standard (hint:
I'm one of the people directly working on that proposal).

I think this is the crux of the issue. IMO, it's not much of a standard when the spirit of the standard is to allow everyone to implement different, non-deterministic behaviour....

Unmapping can and should be made reliable so that we don't have to waste effort trying to fix up mismatches that shouldn't have occurred in the first place...

Storage devices are complex enough that they _already_ exhibit behaviour which is fairly much non-deterministic in a number of ways. Especially if we're talking about SSDs or large arrays, rather than just disks.

Quoted 3 lines · David Woodhouse

Storage devices are complex enough that they _already_ exhibit behaviour which is fairly much non-deterministic in a number of ways. Especially if we're talking about SSDs or large arrays, rather than just disks.

If anything, SSDs are more deterministic than rotating storage. Variable numbers of sectors per track, unpredictable sector remapping, track re-reads due to errors during reads ... SSDs seem like a real improvement.

Quoted 2 lines · On Sun, Nov 09, 2008 at 10:40:24PM -0500, [email protected] wrote:

Quoted 1 line

There will be a chunk size value available in a VPD page that can

Quoted 1 line

Quoted 1 line

used to determine minimum size/alignment. For openers, I see

Quoted 1 line

Quoted 1 line

no point in a 512-byte UNMAP, even though it's allowed by the

Quoted 2 lines

Quoted 2 lines

I suspect most arrays (and many SSDs) will ignore it, and ignoring it is definitely within the spirit of the proposed T10 standard

Quoted 3 lines

Quoted 1 line

I'm one of the people directly working on that proposal).

I think this is the crux of the issue. IMO, it's not much of a

Quoted 2 lines

when the spirit of the standard is to allow everyone to implement different, non-deterministic behaviour....

I disagree. The discard request is a _hint_ from the upper layers, and the storage device can act on that hint as it sees fit. There's

wrong with that; it doesn't make it "not much of a standard".

So I think it's perfectly acceptable for the operating system to treat discard requests as a hint, with best-effort semantics. And any device which _really_ cares will need to make sure for _itself_ that it

those hints reliably.

Quoted 2 lines · Dave Chinner

Quoted 1 line

I think this is the crux of the issue. IMO, it's not much of a

standard

Quoted 3 lines

Quoted 2 lines

when the spirit of the standard is to allow everyone to implement different, non-deterministic behaviour....

Quoted 2 lines

I disagree. The discard request is a _hint_ from the upper layers, and the storage device can act on that hint as it sees fit. There's

nothing

Quoted 1 line

wrong with that; it doesn't make it "not much of a standard".

Bingo! That is exactly the spirit and thinking behind the UNMAP proposal.

Quoted 16 lines · On Mon, Nov 10, 2008 at 12:05:57PM -0500, [email protected] wrote:

Quoted 4 lines · Dave Chinner

Quoted 3 lines

I think this is the crux of the issue. IMO, it's not much of a

standard

Quoted 6 lines

Quoted 3 lines

when the spirit of the standard is to allow everyone to implement different, non-deterministic behaviour....

I disagree. The discard request is a _hint_ from the upper layers, and the storage device can act on that hint as it sees fit. There's

nothing

Quoted 2 lines

wrong with that; it doesn't make it "not much of a standard".

Quoted 3 lines

Bingo! That is exactly the spirit and thinking behind the UNMAP proposal.

While that may be, it's hardly the spirit that Ric (at least) has been promoting with dire warnings about how 'Enterprise class' customers will react if Linux does the wrong thing for EMC arrays with discard/trim/unmap.

Quoted 9 lines · Dave Chinner

Quoted 7 lines · On Sun, Nov 09, 2008 at 10:40:24PM -0500, [email protected] wrote:

There will be a chunk size value available in a VPD page that can be used to determine minimum size/alignment. For openers, I see essentially
no point in a 512-byte UNMAP, even though it's allowed by the standard - I suspect most arrays (and many SSDs) will ignore it, and ignoring it is definitely within the spirit of the proposed T10 standard (hint:
I'm one of the people directly working on that proposal).

Quoted 3 lines

I think this is the crux of the issue. IMO, it's not much of a standard when the spirit of the standard is to allow everyone to implement different, non-deterministic behaviour....

I disagree. The discard request is a _hint_ from the upper layers, and the storage device can act on that hint as it sees fit. There's nothing wrong with that; it doesn't make it "not much of a standard".

Storage devices are complex enough that they _already_ exhibit behaviour which is fairly much non-deterministic in a number of ways. Especially if we're talking about SSDs or large arrays, rather than just disks. A standard needs to be clear about what _is_ guaranteed, and what is _not_ guaranteed. If it is explicit that the storage device is permitted to ignore the discard hint, and some storage devices do so under some circumstances, then that is just fine.

Quoted 3 lines

Unmapping can and should be made reliable so that we don't have to waste effort trying to fix up mismatches that shouldn't have occurred in the first place...

Perhaps so. But remember, this can only really be considered a correctness issue on thin-provisioned arrays -- because they may run out of space sooner than they should. But that kind of failure mode is something that is explicitly accepted by those designing and using such thin-provisioned arrays. It's not as if we're introducing any _new_ kind of problem.

So I think it's perfectly acceptable for the operating system to treat discard requests as a hint, with best-effort semantics. And any device which _really_ cares will need to make sure for _itself_ that it handles those hints reliably.

Treating it as a reliable command (i.e. it succeeds or returns an error) means that we can implement filesystems that can do unmapping in such a way that when the array reports that it is out of space we *know* that there is no free space that can be unmapped. i.e. no need for a "defrag" tool.

So how do you propose that a storage architect who is trying to design a reliable thin provisioning storage stack finds out which devices actually do reliable unmapping? Vendors are simply going to say they support the unmap command, which currently means anything from "ignore completely" to "always do the right thing".

Dave,
The thin provisioning chunk size (coming) in the VPD page is a possible place to start.

Quoted 4 lines · On Mon, 10 Nov 2008 20:23:17 -0500, [email protected] wrote:

Dave,
The thin provisioning chunk size (coming) in the VPD page is a possible place to start.

VPD page for which device? Consider a filesystem that is striped across devices from multiple arrays or even multiple vendors. How is the filesystem supposed to "align" an unmap command when the underlying disks all have different alignments?

Thing is being pitched to answer a very specific customer use case - shared storage (mid to high end almost exclusively) with several different users and applications....

Quoted 1 line · Ric Wheeler

Thing is being pitched to answer a very specific customer use case - shared storage (mid to high end almost exclusively) with several different users and applications....

And by "different users" these customers almost always mean different operating systems. They are combining storage into a central location for easier management.

So "exact unmapped tracking by the filesystem" is impossible and not part of the requirement. Doesn't mean we can't make our filesystems better, but forget about a perfect ability to known just how much space we really have once we do an unmap.

We can't tell how much of our unmapped space the device has given away to someone else and we cannot prevent the device from failing a write to an unmapped block if all the space is gone. It is just an IO error, and possible fs-is-offline if that block we failed was metadata!

It is up to the customer to manage their storage so it never reaches the unable-to-write state.

jim

Quoted 4 lines · jim owens

And by "different users" these customers almost always mean different operating systems. They are combining storage into a central location for easier management.

When you have one specific LUN exported from an array, it is owned by one OS. You can definitely have different LUN's used by different OS's, but that seems to be irrelevant to our challenges here, right?

Quoted 7 lines · Ric Wheeler

Quoted 4 lines · jim owens

And by "different users" these customers almost always mean different operating systems. They are combining storage into a central location for easier management.

When you have one specific LUN exported from an array, it is owned by one OS. You can definitely have different LUN's used by different OS's, but that seems to be irrelevant to our challenges here, right?

But the total thin storage pool is shared by multiple luns and thus maybe multiple not-able-to-cooperate hosts.

I was only pointing this out because earlier threads seemed to be "linux filesystems to be exact across multiple hosts" (which is really a cluster design) and even if we did that for linux it would not solve the customer need.

I just wanted to make it clear why trying to do a complicated change to linux for exactness is pointless because the customer requirement is for more than linux attached to the thin pool.

So the relevance is our design boundary.

jim

I think that the concern is that the exact implementation is actually already coded and relatively easy for us to do (i.e., send down unmap commands at natural file system level units after a truncate/delete).

The irony is that the hard part is to try to approach that level of exactness with the other techniques (coalescing unmaps, defrag, etc) :-)

Quoted 3 lines · Ric Wheeler

Thing is being pitched to answer a very specific customer use case - shared storage (mid to high end almost exclusively) with several different users and applications....

It is up to the customer to manage their storage so it never reaches the unable-to-write state.

Quoted 2 lines · jim owens

Quoted 1 line · Ric Wheeler

Thing is being pitched to answer a very specific customer use case - shared storage (mid to high end almost exclusively) with several different users and applications....

...

Quoted 2 lines

It is up to the customer to manage their storage so it never reaches the unable-to-write state.

Sure, but putting the entire management burden of obtaining and running defrag tools in every one of their large set of OS's is the wrong approach.

We can and should be designing new functionality for the data center in such a manner that does not require large scale manual intervention to maintain the systems. Your customers won't thank you for solving the thin provisioning management problem by requiring them to do extra hand-holding....

Quoted 2 lines · jim owens

Quoted 1 line · Ric Wheeler

Thing is being pitched to answer a very specific customer use case - shared storage (mid to high end almost exclusively) with several different users and applications....

...

Quoted 2 lines

It is up to the customer to manage their storage so it never reaches the unable-to-write state.

Sure, but putting the entire management burden of obtaining and running defrag tools in every one of their large set of OS's is the wrong approach.

We can and should be designing new functionality for the data center in such a manner that does not require large scale manual intervention to maintain the systems. Your customers won't thank you for solving the thin provisioning management problem by requiring them to do extra hand-holding....

Dave,

Quoted 5 lines

Treating it as a reliable command (i.e. it succeeds or returns an error) means that we can implement filesystems that can do unmapping in such a way that when the array reports that it is out of space we *know* that there is no free space that can be unmapped. i.e. no need for a "defrag" tool.

What if the filesystem block size and the array thin provisioning chunk size don't match? It's still "defrag" time ...

After talking to some vendors, one issue that came up is that the arrays all have a different size that is used internally to track the SCSI equivalent of TRIM commands (POKE/unmap).

What they would like is for us to coalesce these commands into aligned multiples of these chunks. If not, the target device will most likely ignore the bits at the beginning and end (and all small requests).

I have been thinking about whether or not we can (and should) do anything more than our current best effort to send down large chunks (note that the "chunk" size can range from reasonable sizes like 8KB or so up to close to 1MB!).

One suggestion is that a modified defrag sweep could be used periodically to update the device (a proposal I am not keen on).

By the way, the latest (from 2 days ago) version of the Thin Provisioning proposal is here:

http://www.t10.org/ftp/t10/document.08/08-149r4.pdf

And the vendors need to provide the device trim chunk size in a standard way (like scsi geometry) to the filesystem.

Quoted 4 lines · James Bottomley

By the way, the latest (from 2 days ago) version of the Thin Provisioning proposal is here:

http://www.t10.org/ftp/t10/document.08/08-149r4.pdf

If I understand the spec [ not likely ;) ] ...

jim owens wrote:

Quoted 2 lines

And the vendors need to provide the device trim chunk size in a standard way (like scsi geometry) to the filesystem.

It may be that the READ CAPACITY (16) provides the trim chunk size via the "logical blocks per physical block exponent".

But since this is just a T10 spec, I would want that interpretation verified by the array vendors.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

"Ric" == Ric Wheeler <[email protected]> writes:

Ric> After talking to some vendors, one issue that came up is that the Ric> arrays all have a different size that is used internally to track Ric> the SCSI equivalent of TRIM commands (POKE/unmap).

I haven't had time to completely digest the latest (Nov. 4th) UNMAP proposal yet. However, I don't recall seeing any notion of blocks bigger than the logical block length. And the command clearly takes (a list of) <start LBA, number of blocks>.

Ric> What they would like is for us to coalesce these commands into Ric> aligned multiples of these chunks.

Ric> If not, the target device will most likely ignore the bits at the Ric> beginning and end (and all small requests).

That really just sounds like a broken firmware implementation on their end. If they can not UNMAP a single logical block then they are clearly not within the spirit of the proposed standard. Their thin provisioning is going to suck and customers will hopefully complain/buy a competing product.

Just not sure that we have any way to handle this better than what is in already (best effort, maybe a post/defrag like user util that can be run offline to clean up?)

After talking to some vendors, one issue that came up is that the arrays all have a different size that is used internally to track the SCSI equivalent of TRIM commands (POKE/unmap).

What they would like is for us to coalesce these commands into aligned multiples of these chunks. If not, the target device will most likely ignore the bits at the beginning and end (and all small requests).

I have been thinking about whether or not we can (and should) do anything more than our current best effort to send down large chunks (note that the "chunk" size can range from reasonable sizes like 8KB or so up to close to 1MB!).

One suggestion is that a modified defrag sweep could be used periodically to update the device (a proposal I am not keen on).

Quoted 4 lines · Ric Wheeler

After talking to some vendors, one issue that came up is that the arrays all have a different size that is used internally to track the SCSI equivalent of TRIM commands (POKE/unmap).

What they would like is for us to coalesce these commands into aligned multiples of these chunks. If not, the target device will most likely ignore the bits at the beginning and end (and all small requests).

There's lots of questions that need to be answered here. e.g:

Where are these free spaces going to be aggregated before dispatch?

What happens if they are re-allocated and re-written by the filesystem before they've been dispatched?

How is the chunk size going to be passed to the aggregation layer?

What about passing itto the filesystem so it can align all it's allocations in a manner that simplifies the dispatch problem?

What happens if a crash occurs before the aggregated free space is dispatched?

Are there coherency problems with filesystem recovery after a crash?

Quoted 3 lines

I have been thinking about whether or not we can (and should) do anything more than our current best effort to send down large chunks (note that the "chunk" size can range from reasonable sizes like 8KB or so up to close to 1MB!).

Any aggregation is only as good as the original allocation the filesystem did. Look as the mess ext3 extracting untarring a kernel tarball creates - blocks are written to all over the place. You'd need to fix that to have any hope of behaviour nicely for a RAID that has a sub-optimal thin provisioning algorithm.

The problem is not with the filesystem, the block layer or the OS. If they array vendors have optimised themselves into a corner, then they shoul dbe fixing their problem, not asking the rest of the world to expend large amounts of effort to work around the shortcomings of their products.....

Quoted 4 lines

One suggestion is that a modified defrag sweep could be used periodically to update the device (a proposal I am not keen on).

No thanks. That needs an implementation per filesystem, and it will need to be done with the filesystem on line which means it will still need substantial help from the kernel.

Cheers,

Dave.

Quoted 20 lines · Dave Chinner

Quoted 8 lines · Ric Wheeler

After talking to some vendors, one issue that came up is that the arrays all have a different size that is used internally to track the SCSI equivalent of TRIM commands (POKE/unmap).

What they would like is for us to coalesce these commands into aligned multiples of these chunks. If not, the target device will most likely ignore the bits at the beginning and end (and all small requests).

There's lots of questions that need to be answered here. e.g:

Where are these free spaces going to be aggregated before dispatch?

What happens if they are re-allocated and re-written by the filesystem before they've been dispatched?

How is the chunk size going to be passed to the aggregation layer?

Quoted 8 lines

What about passing itto the filesystem so it can align all it's allocations in a manner that simplifies the dispatch problem?

What happens if a crash occurs before the aggregated free space is dispatched?

Are there coherency problems with filesystem recovery after a crash?

The good thing about these "unmap" commands (SCSI speak this week for TRIM) is that we can drop them if we have to without data integrity concerns.

The only thing that you cannot do is to send down an unmap for a block still in use (including ones that have not been committed in a transaction).

In SCSI, they plan to zero those blocks so that you will always read a block of zeros back if you try to read an unmapped sector.

I have no idea how we can pass the aggregation size up from the block layer since it is not currently exported in a uniform way from SCSI. Even if it is, we have struggled to get RAID stripe alignment handled so far.

Quoted 12 lines

Quoted 5 lines

I have been thinking about whether or not we can (and should) do anything more than our current best effort to send down large chunks (note that the "chunk" size can range from reasonable sizes like 8KB or so up to close to 1MB!).

Any aggregation is only as good as the original allocation the filesystem did. Look as the mess ext3 extracting untarring a kernel tarball creates - blocks are written to all over the place. You'd need to fix that to have any hope of behaviour nicely for a RAID that has a sub-optimal thin provisioning algorithm.

Quoted 6 lines

The problem is not with the filesystem, the block layer or the OS. If they array vendors have optimised themselves into a corner, then they shoul dbe fixing their problem, not asking the rest of the world to expend large amounts of effort to work around the shortcomings of their products.....

I agree - I think that eventually vendors will end up having to cache the requests internally. The problem is with the customers who will be getting the first generation of gear and have had their expectations set already....

Quoted 9 lines

Quoted 4 lines

One suggestion is that a modified defrag sweep could be used periodically to update the device (a proposal I am not keen on).

No thanks. That needs an implementation per filesystem, and it will need to be done with the filesystem on line which means it will still need substantial help from the kernel.

Quoted 4 lines

Cheers,

Dave.

It does seem to be a mess - especially since people have already gone to the trouble to put the hooks in to inform the storage in a consistent and timely way :-)

Quoted 30 lines · Ric Wheeler

Quoted 17 lines · Dave Chinner

Quoted 5 lines · Ric Wheeler

After talking to some vendors, one issue that came up is that the arrays all have a different size that is used internally to track the SCSI equivalent of TRIM commands (POKE/unmap).

What they would like is for us to coalesce these commands into aligned multiples of these chunks. If not, the target device will most likely ignore the bits at the beginning and end (and all small requests).

There's lots of questions that need to be answered here. e.g:

Where are these free spaces going to be aggregated before dispatch?

What happens if they are re-allocated and re-written by the filesystem before they've been dispatched?

How is the chunk size going to be passed to the aggregation layer?

Quoted 9 lines

What about passing itto the filesystem so it can align all it's allocations in a manner that simplifies the dispatch problem?

What happens if a crash occurs before the aggregated free space is dispatched?

Are there coherency problems with filesystem recovery after a crash?

The good thing about these "unmap" commands (SCSI speak this week for TRIM) is that we can drop them if we have to without data integrity concerns.

Quoted 4 lines

The only thing that you cannot do is to send down an unmap for a block still in use (including ones that have not been committed in a transaction).

In SCSI, they plan to zero those blocks so that you will always read a block of zeros back if you try to read an unmapped sector.

Actually, they left this up to the array in the latest spec. If the TPRZ bit is set in the Block Device Characteristics VPD then, yes, it will return zeros. If not, the return is undefined.

Quoted 2 lines

I have no idea how we can pass the aggregation size up from the block layer since it is not currently exported in a uniform way from SCSI. Even if it is, we have struggled to get RAID stripe alignment handled so far.

Well, this is identical to the erase block size (and array stripe size) problems we've been discussing. I thought we'd more or less agreed on the generic attributes model.

Quoted 35 lines

Quoted 10 lines

Quoted 4 lines

I have been thinking about whether or not we can (and should) do anything more than our current best effort to send down large chunks (note that the "chunk" size can range from reasonable sizes like 8KB or so up to close to 1MB!).

Any aggregation is only as good as the original allocation the filesystem did. Look as the mess ext3 extracting untarring a kernel tarball creates - blocks are written to all over the place. You'd need to fix that to have any hope of behaviour nicely for a RAID that has a sub-optimal thin provisioning algorithm.

Quoted 7 lines

The problem is not with the filesystem, the block layer or the OS. If they array vendors have optimised themselves into a corner, then they shoul dbe fixing their problem, not asking the rest of the world to expend large amounts of effort to work around the shortcomings of their products.....

I agree - I think that eventually vendors will end up having to cache the requests internally. The problem is with the customers who will be getting the first generation of gear and have had their expectations set already....

Quoted 15 lines

Quoted 6 lines

One suggestion is that a modified defrag sweep could be used periodically to update the device (a proposal I am not keen on).

No thanks. That needs an implementation per filesystem, and it will need to be done with the filesystem on line which means it will still need substantial help from the kernel.

Cheers,

Dave.

Quoted 2 lines

It does seem to be a mess - especially since people have already gone to the trouble to put the hooks in to inform the storage in a consistent and timely way :-)

I'm sure we can iterate to a conclusion ... even if it's that we won't actually do anything other than send down properly formed unmap commands and if the array chooses to ignore them, that's its lookout.

James

Quoted 7 lines · James Bottomley

Quoted 5 lines · Ric Wheeler

I have no idea how we can pass the aggregation size up from the block layer since it is not currently exported in a uniform way from SCSI. Even if it is, we have struggled to get RAID stripe alignment handled so far.

Quoted 4 lines

Well, this is identical to the erase block size (and array stripe size) problems we've been discussing. I thought we'd more or less agreed on the generic attributes model.

We could do it, but need them to put it in a standard place first. Today, it is exposed only in device specific ways.

Quoted 4 lines · Ric Wheeler

After talking to some vendors, one issue that came up is that the arrays all have a different size that is used internally to track the SCSI equivalent of TRIM commands (POKE/unmap).

What they would like is for us to coalesce these commands into aligned multiples of these chunks. If not, the target device will most likely ignore the bits at the beginning and end (and all small requests).

There's lots of questions that need to be answered here. e.g:

Where are these free spaces going to be aggregated before dispatch?

What happens if they are re-allocated and re-written by the filesystem before they've been dispatched?

How is the chunk size going to be passed to the aggregation layer?

What about passing itto the filesystem so it can align all it's allocations in a manner that simplifies the dispatch problem?

What happens if a crash occurs before the aggregated free space is dispatched?

Are there coherency problems with filesystem recovery after a crash?

Quoted 3 lines

I have been thinking about whether or not we can (and should) do anything more than our current best effort to send down large chunks (note that the "chunk" size can range from reasonable sizes like 8KB or so up to close to 1MB!).

Any aggregation is only as good as the original allocation the filesystem did. Look as the mess ext3 extracting untarring a kernel tarball creates - blocks are written to all over the place. You'd need to fix that to have any hope of behaviour nicely for a RAID that has a sub-optimal thin provisioning algorithm.

The problem is not with the filesystem, the block layer or the OS. If they array vendors have optimised themselves into a corner, then they shoul dbe fixing their problem, not asking the rest of the world to expend large amounts of effort to work around the shortcomings of their products.....

Quoted 4 lines

One suggestion is that a modified defrag sweep could be used periodically to update the device (a proposal I am not keen on).

No thanks. That needs an implementation per filesystem, and it will need to be done with the filesystem on line which means it will still need substantial help from the kernel.

Cheers,

Dave.

By the way, the latest (from 2 days ago) version of the Thin Provisioning proposal is here:

http://www.t10.org/ftp/t10/document.08/08-149r4.pdf

I skimmed it but don't see any update implying that trim might be ineffective if we align wrongly ... where is this?

Quoted 4 lines

I haven't had time to completely digest the latest (Nov. 4th) UNMAP proposal yet. However, I don't recall seeing any notion of blocks bigger than the logical block length. And the command clearly takes (a list of) <start LBA, number of blocks>.

There is a proposal to expose this internal device size in a standard way, but it has not been finalized.

Quoted 2 lines

And the vendors need to provide the device trim chunk size in a standard way (like scsi geometry) to the filesystem.

It may be that the READ CAPACITY (16) provides the trim chunk size via the "logical blocks per physical block exponent".

Quoted 1 line

In SCSI, they plan to zero those blocks so that you will always read

Quoted 1 line

block of zeros back if you try to read an unmapped sector.

Actually, they left this up to the array in the latest spec. If the TPRZ bit is set in the Block Device Characteristics VPD then, yes, it will return zeros. If not, the return is undefined.

-----Original Message-----
From: Ric Wheeler [mailto:[email protected]]
Sent: Thursday, November 06, 2008 9:43 AM
To: David Woodhouse; James Bottomley; [email protected]; [email protected]
Cc: Black, David; Martin K. Petersen; Tom Coughlan; Matthew Wilcox; Jens Axboe
Subject: thin provisioned LUN support

After talking to some vendors, one issue that came up is that the arrays
all have a different size that is used internally to track the SCSI equivalent of TRIM commands (POKE/unmap).

What they would like is for us to coalesce these commands into aligned
multiples of these chunks. If not, the target device will most likely ignore the bits at the beginning and end (and all small requests).

I have been thinking about whether or not we can (and should) do anything more than our current best effort to send down large chunks (note that the "chunk" size can range from reasonable sizes like 8KB or
so up to close to 1MB!).

One suggestion is that a modified defrag sweep could be used periodically to update the device (a proposal I am not keen on).

Thoughts?

Ric

After talking to some vendors, one issue that came up is that the arrays all have a different size that is used internally to track the SCSI equivalent of TRIM commands (POKE/unmap).

What they would like is for us to coalesce these commands into aligned multiples of these chunks. If not, the target device will most likely ignore the bits at the beginning and end (and all small requests).

I have been thinking about whether or not we can (and should) do anything more than our current best effort to send down large chunks (note that the "chunk" size can range from reasonable sizes like 8KB or so up to close to 1MB!).

I have been thinking about whether or not we can (and should) do anything more than our current best effort to send down large chunks (note that the "chunk" size can range from reasonable sizes like 8KB or so up to close to 1MB!).

Quoted 4 lines · Ric Wheeler

I have been thinking about whether or not we can (and should) do anything more than our current best effort to send down large chunks (note that the "chunk" size can range from reasonable sizes like 8KB or so up to close to 1MB!).

One of the proposals in this thread (that has got buried somewhere) was to expand any discard request sent down from the filesystem to encompass all the adjacent free space. I've checked with our SSD people and they're fine with this idea.

dwmw2 says "it isn't actually that hard in FAT" and then interjects some personal opinion about this solution ;-)

Is it hard in XFS? btrfs? ext2? Does anyone have a problem with this as a solution?

One of the proposals in this thread (that has got buried somewhere) was to expand any discard request sent down from the filesystem to encompass all the adjacent free space. I've checked with our SSD people and they're fine with this idea.

dwmw2 says "it isn't actually that hard in FAT" and then interjects some personal opinion about this solution ;-)

Is it hard in XFS? btrfs? ext2? Does anyone have a problem with this as a solution?

Quoted 5 lines · Matthew Wilcox

One of the proposals in this thread (that has got buried somewhere) was to expand any discard request sent down from the filesystem to encompass all the adjacent free space. I've checked with our SSD people and they're fine with this idea.

Quoted 5 lines

dwmw2 says "it isn't actually that hard in FAT" and then interjects some personal opinion about this solution ;-)

Is it hard in XFS? btrfs? ext2? Does anyone have a problem with this as a solution?

I suspect how hard it is depends somewhat on exactly what you mean by "all the adjacent free space" and what is expected of the file system.

Why do most people assume that sending unmap/trim commands for every deletet extent ASAP is a good idea?

So I think not actually doing these on every alloc/free is a good idea. Instead the filesystem would free bits when big enough regions happen,

I agree with you. Thus my earlier assertion:

- trim/unmap for SSD garbage collection has a different goal than trim/unmap for thin provisioning.

Quoted 2 lines · we want to delay them as you

So I think not actually doing these on every alloc/free is a good idea. Instead the filesystem would free bits when big enough regions happen,

to be filesystem friendly.

But this won't change the block layer. This is a per-filesystem coding issue to decide when to send the discard.

jim
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html