Cloud storage optimizations

Emulated block devices offered by cloud VM’s can provide functionality to guest kernels and applications that traditionally have not been available to users of consumer-grade HDD and SSD’s. For example, today it’s possible to create a block device in Google’s Persistent

Disk with a 16k physical sector size, which promises that aligned 16k writes will be atomically. With NVMe, it is possible for a storage device to promise this without requiring read-modify-write updates for sub-16k writes. All that is necessary are some changes in the block layer so that the kernel does not inadvertently tear a write request when splitting a bio because it is too large (perhaps because it got merged with some other request, and then it gets split at an inconvenient boundary).

There are also more interesting, advanced optimizations that might be possible. For example, Jens had observed the passing hints that journaling writes (either from file systems or databases) could be potentially useful. Unfortunately most common storage devices have not supported write hints, and support for write hints were ripped out last year. That can be easily reversed, but there are some other interesting related subjects that are very much suited for LSF/MM.

For example, most cloud storage devices are doing read-ahead to try to anticipate read requests from the VM. This can interfere with the read-ahead being done by the guest kernel. So being able to tell cloud storage device whether a particular read request is stemming from a read-ahead or not. At the moment, as Matthew Wilcox has pointed out, we currently use the read-ahead code path for synchronous buffered reads. So plumbing this information so it can passed through multiple levels of the mm, fs, and block layers will probably be needed.

Quoted 9 lines

For example, most cloud storage devices are doing read-ahead to try to anticipate read requests from the VM. This can interfere with the read-ahead being done by the guest kernel. So being able to tell cloud storage device whether a particular read request is stemming from a read-ahead or not. At the moment, as Matthew Wilcox has pointed out, we currently use the read-ahead code path for synchronous buffered reads. So plumbing this information so it can passed through multiple levels of the mm, fs, and block layers will probably be needed.

It seems that is also useful as well, yet if my understanding is correct, it's somewhat unclear for me if we could do more and have a better form compared with the current REQ_RAHEAD (currently REQ_RAHEAD use cases and impacts are quite limited.)

Quoted 10 lines · Gao Xiang

Quoted 9 lines

For example, most cloud storage devices are doing read-ahead to try to anticipate read requests from the VM. This can interfere with the read-ahead being done by the guest kernel. So being able to tell cloud storage device whether a particular read request is stemming from a read-ahead or not. At the moment, as Matthew Wilcox has pointed out, we currently use the read-ahead code path for synchronous buffered reads. So plumbing this information so it can passed through multiple levels of the mm, fs, and block layers will probably be needed.

Quoted 4 lines

It seems that is also useful as well, yet if my understanding is correct, it's somewhat unclear for me if we could do more and have a better form compared with the current REQ_RAHEAD (currently REQ_RAHEAD use cases and impacts are quite limited.)

I'm pretty sure the Linux readahead algorithms could do with some serious tuning (as opposed to the hacks the Android device vendors are doing). Outside my current level of enthusiasm / knowledge, alas. And it's hard because while we no longer care about performance on floppies, we do care about performance from CompactFlash to 8GB/s NVMe drives. I had one person recently complain that 200Gbps ethernet was too slow for their storage, so there's a faster usecase to care about.

For example, most cloud storage devices are doing read-ahead to try to anticipate read requests from the VM. This can interfere with the read-ahead being done by the guest kernel. So being able to tell cloud storage device whether a particular read request is stemming from a read-ahead or not. At the moment, as Matthew Wilcox has pointed out, we currently use the read-ahead code path for synchronous buffered reads. So plumbing this information so it can passed through multiple levels of the mm, fs, and block layers will probably be needed.

Quoted 9 lines · Theodore Ts'o

For example, most cloud storage devices are doing read-ahead to try to anticipate read requests from the VM. This can interfere with the read-ahead being done by the guest kernel. So being able to tell cloud storage device whether a particular read request is stemming from a read-ahead or not. At the moment, as Matthew Wilcox has pointed out, we currently use the read-ahead code path for synchronous buffered reads. So plumbing this information so it can passed through multiple levels of the mm, fs, and block layers will probably be needed.

This shouldn't be _too_ painful. For example, the NVMe driver already does the right thing:

         if (req->cmd_flags & (REQ_FAILFAST_DEV | REQ_RAHEAD))
                 control |= NVME_RW_LR;

         if (req->cmd_flags & REQ_RAHEAD)
                 dsmgmt |= NVME_RW_DSM_FREQ_PREFETCH;

(LR is Limited Retry; FREQ_PREFETCH is "Speculative read. The command is part of a prefetch operation")

The only problem is that the readahead code doesn't tell the filesystem whether the request is sync or async. This should be a simple matter of adding a new 'bool async' to the readahead_control and then setting REQ_RAHEAD based on that, rather than on whether the request came in through readahead() or read_folio() (eg see mpage_readahead()).

Another thing to fix is that SCSI doesn't do anything with the REQ_RAHEAD flag, so I presume T10 has some work to do (maybe they could borrow the Access Frequency field from NVMe, since that was what the drive vendors told us they wanted; maybe they changed their minds since).

Quoted 5 lines

The only problem is that the readahead code doesn't tell the filesystem whether the request is sync or async. This should be a simple matter of adding a new 'bool async' to the readahead_control and then setting REQ_RAHEAD based on that, rather than on whether the request came in through readahead() or read_folio() (eg see mpage_readahead()).

Great! In addition to that, just (somewhat) off topic, if we have a "bool async" now, I think it will immediately have some users (such as EROFS), since we'd like to do post-processing (such as decompression) immediately in the same context with sync readahead (due to missing pages) and leave it to another kworker for async readahead (I think it's almost same for decryption and verification).

So "bool async" is quite useful on my side if it could be possible passed to fs side. I'd like to raise my hands to have it.

Quoted 13 lines · Gao Xiang

Quoted 5 lines

The only problem is that the readahead code doesn't tell the filesystem whether the request is sync or async. This should be a simple matter of adding a new 'bool async' to the readahead_control and then setting REQ_RAHEAD based on that, rather than on whether the request came in through readahead() or read_folio() (eg see mpage_readahead()).

Great! In addition to that, just (somewhat) off topic, if we have a "bool async" now, I think it will immediately have some users (such as EROFS), since we'd like to do post-processing (such as decompression) immediately in the same context with sync readahead (due to missing pages) and leave it to another kworker for async readahead (I think it's almost same for decryption and verification).

Quoted 2 lines

So "bool async" is quite useful on my side if it could be possible passed to fs side. I'd like to raise my hands to have it.

That's a really interesting use-case; thanks for bringing it up.

Ideally, we'd have the waiting task do the decompression/decryption/verification for proper accounting of CPU. Unfortunately, if the folio isn't uptodate, the task doesn't even hold a reference to the folio while it waits, so there's no way to wake the task and let it know that it has work to do. At least not at the moment ... let me think about that a bit (and if you see a way to do it, feel free to propose it)

Quoted 19 lines · Matthew Wilcox

Quoted 13 lines · Gao Xiang

Quoted 5 lines

The only problem is that the readahead code doesn't tell the filesystem whether the request is sync or async.  This should be a simple matter of adding a new 'bool async' to the readahead_control and then setting REQ_RAHEAD based on that, rather than on whether the request came in through readahead() or read_folio() (eg see mpage_readahead()).

Great!  In addition to that, just (somewhat) off topic, if we have a "bool async" now, I think it will immediately have some users (such as EROFS), since we'd like to do post-processing (such as decompression) immediately in the same context with sync readahead (due to missing pages) and leave it to another kworker for async readahead (I think it's almost same for decryption and verification).

Quoted 2 lines

So "bool async" is quite useful on my side if it could be possible passed to fs side.  I'd like to raise my hands to have it.

That's a really interesting use-case; thanks for bringing it up.

Quoted 7 lines

Ideally, we'd have the waiting task do the decompression/decryption/verification for proper accounting of CPU. Unfortunately, if the folio isn't uptodate, the task doesn't even hold a reference to the folio while it waits, so there's no way to wake the task and let it know that it has work to do.  At least not at the moment ... let me think about that a bit (and if you see a way to do it, feel free to propose it)

Honestly, I'd like to take the folio lock until all post-processing is done and make it uptodate and unlock so that only we need is to pass locked-folios requests to kworkers for async way or sync handling in the original context.

If we unlocked these folios in advance without uptodate, which means we have to lock it again (which could have more lock contention) and need to have a way to trace I/Oed but not post-processed stuff in addition to no I/Oed stuff.

Thanks,
Gao Xiang

Quoted 19 lines · Matthew Wilcox

Quoted 13 lines · Gao Xiang

Quoted 5 lines

The only problem is that the readahead code doesn't tell the filesystem whether the request is sync or async. This should be a simple matter of adding a new 'bool async' to the readahead_control and then setting REQ_RAHEAD based on that, rather than on whether the request came in through readahead() or read_folio() (eg see mpage_readahead()).

Great! In addition to that, just (somewhat) off topic, if we have a "bool async" now, I think it will immediately have some users (such as EROFS), since we'd like to do post-processing (such as decompression) immediately in the same context with sync readahead (due to missing pages) and leave it to another kworker for async readahead (I think it's almost same for decryption and verification).

Quoted 2 lines

So "bool async" is quite useful on my side if it could be possible passed to fs side. I'd like to raise my hands to have it.

That's a really interesting use-case; thanks for bringing it up.

Quoted 7 lines

Ideally, we'd have the waiting task do the decompression/decryption/verification for proper accounting of CPU. Unfortunately, if the folio isn't uptodate, the task doesn't even hold a reference to the folio while it waits, so there's no way to wake the task and let it know that it has work to do. At least not at the moment ... let me think about that a bit (and if you see a way to do it, feel free to propose it)

Honestly, I'd like to take the folio lock until all post-processing is done and make it uptodate and unlock so that only we need is to pass locked-folios requests to kworkers for async way or sync handling in the original context.

If we unlocked these folios in advance without uptodate, which means we have to lock it again (which could have more lock contention) and need to have a way to trace I/Oed but not post-processed stuff in addition to no I/Oed stuff.

Quoted 33 lines · Gao Xiang

Quoted 19 lines · Matthew Wilcox

Quoted 13 lines · Gao Xiang

Quoted 5 lines

The only problem is that the readahead code doesn't tell the filesystem whether the request is sync or async. This should be a simple matter of adding a new 'bool async' to the readahead_control and then setting REQ_RAHEAD based on that, rather than on whether the request came in through readahead() or read_folio() (eg see mpage_readahead()).

Great! In addition to that, just (somewhat) off topic, if we have a "bool async" now, I think it will immediately have some users (such as EROFS), since we'd like to do post-processing (such as decompression) immediately in the same context with sync readahead (due to missing pages) and leave it to another kworker for async readahead (I think it's almost same for decryption and verification).

Quoted 2 lines

So "bool async" is quite useful on my side if it could be possible passed to fs side. I'd like to raise my hands to have it.

That's a really interesting use-case; thanks for bringing it up.

Quoted 7 lines

Ideally, we'd have the waiting task do the decompression/decryption/verification for proper accounting of CPU. Unfortunately, if the folio isn't uptodate, the task doesn't even hold a reference to the folio while it waits, so there's no way to wake the task and let it know that it has work to do. At least not at the moment ... let me think about that a bit (and if you see a way to do it, feel free to propose it)

Honestly, I'd like to take the folio lock until all post-processing is done and make it uptodate and unlock so that only we need is to pass locked-folios requests to kworkers for async way or sync handling in the original context.

Quoted 4 lines

If we unlocked these folios in advance without uptodate, which means we have to lock it again (which could have more lock contention) and need to have a way to trace I/Oed but not post-processed stuff in addition to no I/Oed stuff.

Right, look at how it's handled right now ...

sys_read() ends up in filemap_get_pages() which (assuming no folio in cache) calls page_cache_sync_readahead(). That creates locked, !uptodate folios and asks the filesystem to fill them. Unless that completes incredibly quickly, filemap_get_pages() ends up in filemap_update_page() which calls folio_put_wait_locked().

If the filesystem BIO completion routine could identify if there was a task waiting and select one of them, it could wake up the waiter and pass it a description of what work it needed to do (with the folio still locked), rather than do the postprocessing itself and unlock the folio

But that all seems _very_ hard to do with 100% reliability. Note the comment in folio_wait_bit_common() which points out that the waiters bit may be set even when there are no waiters. The wake_up code doesn't seem to support this kind of thing (all waiters are non-exclusive, but only wake up one of them).

Quoted 47 lines · Matthew Wilcox

Quoted 33 lines · Gao Xiang

Quoted 19 lines · Matthew Wilcox

Quoted 13 lines · Gao Xiang

Quoted 5 lines

The only problem is that the readahead code doesn't tell the filesystem whether the request is sync or async.  This should be a simple matter of adding a new 'bool async' to the readahead_control and then setting REQ_RAHEAD based on that, rather than on whether the request came in through readahead() or read_folio() (eg see mpage_readahead()).

Great!  In addition to that, just (somewhat) off topic, if we have a "bool async" now, I think it will immediately have some users (such as EROFS), since we'd like to do post-processing (such as decompression) immediately in the same context with sync readahead (due to missing pages) and leave it to another kworker for async readahead (I think it's almost same for decryption and verification).

Quoted 2 lines

So "bool async" is quite useful on my side if it could be possible passed to fs side.  I'd like to raise my hands to have it.

That's a really interesting use-case; thanks for bringing it up.

Quoted 7 lines

Ideally, we'd have the waiting task do the decompression/decryption/verification for proper accounting of CPU. Unfortunately, if the folio isn't uptodate, the task doesn't even hold a reference to the folio while it waits, so there's no way to wake the task and let it know that it has work to do.  At least not at the moment ... let me think about that a bit (and if you see a way to do it, feel free to propose it)

Honestly, I'd like to take the folio lock until all post-processing is done and make it uptodate and unlock so that only we need is to pass locked-folios requests to kworkers for async way or sync handling in the original context.

Quoted 4 lines

If we unlocked these folios in advance without uptodate, which means we have to lock it again (which could have more lock contention) and need to have a way to trace I/Oed but not post-processed stuff in addition to no I/Oed stuff.

Right, look at how it's handled right now ...

sys_read() ends up in filemap_get_pages() which (assuming no folio in cache) calls page_cache_sync_readahead().  That creates locked, !uptodate folios and asks the filesystem to fill them.  Unless that completes incredibly quickly, filemap_get_pages() ends up in filemap_update_page() which calls folio_put_wait_locked().

Quoted 4 lines

If the filesystem BIO completion routine could identify if there was a task waiting and select one of them, it could wake up the waiter and pass it a description of what work it needed to do (with the folio still locked), rather than do the postprocessing itself and unlock the folio

Currently EROFS sync decompression is waiting in .readahead() with locked page cache folios, one "completion" together than BIO descriptor (bi_private) in the original context, so that the filesystem BIO completion just needs to complete the completion and wakeup the original context (due to missing pages, so the original context will need the page data immediately as well) to go on .readhead() and unlock folios.

Does this way have some flew? Or I'm missing something?

Thanks,
Gao Xiang

Quoted 6 lines

But that all seems _very_ hard to do with 100% reliability.  Note the comment in folio_wait_bit_common() which points out that the waiters bit may be set even when there are no waiters.  The wake_up code doesn't seem to support this kind of thing (all waiters are non-exclusive, but only wake up one of them).

Emulated block devices offered by cloud VM’s can provide functionality to guest kernels and applications that traditionally have not been available to users of consumer-grade HDD and SSD’s. For example, today it’s possible to create a block device in Google’s Persistent Disk with a 16k physical sector size, which promises that aligned 16k writes will be atomically. With NVMe, it is possible for a storage device to promise this without requiring read-modify-write updates for sub-16k writes. All that is necessary are some changes in the block layer so that the kernel does not inadvertently tear a write request when splitting a bio because it is too large (perhaps because it got merged with some other request, and then it gets split at an inconvenient boundary).

There are also more interesting, advanced optimizations that might be possible. For example, Jens had observed the passing hints that journaling writes (either from file systems or databases) could be potentially useful. Unfortunately most common storage devices have not supported write hints, and support for write hints were ripped out last year. That can be easily reversed, but there are some other interesting related subjects that are very much suited for LSF/MM.

For example, most cloud storage devices are doing read-ahead to try to anticipate read requests from the VM. This can interfere with the read-ahead being done by the guest kernel. So being able to tell cloud storage device whether a particular read request is stemming from a read-ahead or not. At the moment, as Matthew Wilcox has pointed out, we currently use the read-ahead code path for synchronous buffered reads. So plumbing this information so it can passed through multiple levels of the mm, fs, and block layers will probably be needed.

Emulated block devices offered by cloud VM’s can provide functionality to guest kernels and applications that traditionally have not been available to users of consumer-grade HDD and SSD’s. For example, today it’s possible to create a block device in Google’s Persistent

Disk with a 16k physical sector size, which promises that aligned 16k writes will be atomically. With NVMe, it is possible for a storage device to promise this without requiring read-modify-write updates for sub-16k writes. All that is necessary are some changes in the block layer so that the kernel does not inadvertently tear a write request when splitting a bio because it is too large (perhaps because it got merged with some other request, and then it gets split at an inconvenient boundary).

There are also more interesting, advanced optimizations that might be possible. For example, Jens had observed the passing hints that journaling writes (either from file systems or databases) could be potentially useful. Unfortunately most common storage devices have not supported write hints, and support for write hints were ripped out last year. That can be easily reversed, but there are some other interesting related subjects that are very much suited for LSF/MM.

For example, most cloud storage devices are doing read-ahead to try to anticipate read requests from the VM. This can interfere with the read-ahead being done by the guest kernel. So being able to tell cloud storage device whether a particular read request is stemming from a read-ahead or not. At the moment, as Matthew Wilcox has pointed out, we currently use the read-ahead code path for synchronous buffered reads. So plumbing this information so it can passed through multiple levels of the mm, fs, and block layers will probably be needed.

T10 PI gives the kernel a means to associate its own checksums (and a goofy u16 tag) with LBAs on disk. There haven't been that many actual SCSI devices that implement it,

Unfortunately most common storage devices have
not supported write hints, and support for write hints were ripped out last year.

Work is ongoing in T10 to add write hint support to SBC. We plan to propose to restore write hint support after there is agreement in T10 about the approach. See also "Constrained SBC-5 Streams"
(http://www.t10.org/cgi-bin/ac.pl?t=d&f=23-024r0.pdf). This proposal has been uploaded yesterday.

Emulated block devices offered by cloud VM’s can provide functionality to guest kernels and applications that traditionally have not been available to users of consumer-grade HDD and SSD’s. For example, today it’s possible to create a block device in Google’s Persistent Disk with a 16k physical sector size, which promises that aligned 16k writes will be atomically. With NVMe, it is possible for a storage device to promise this without requiring read-modify-write updates for sub-16k writes.

All that is necessary are some changes in the block
layer so that the kernel does not inadvertently tear a write request when splitting a bio because it is too large (perhaps because it got merged with some other request, and then it gets split at an inconvenient boundary).

That said, I was hoping you were going to suggest supporting 16k logical block sizes. Not a problem on some arch's, but still problematic when PAGE_SIZE is 4k. :)

Quoted 3 lines · Keith Busch

That said, I was hoping you were going to suggest supporting 16k logical block sizes. Not a problem on some arch's, but still problematic when PAGE_SIZE is 4k. :)

I was hoping Luis was going to propose a session on LBA size > PAGE_SIZE. Funnily, while the pressure is coming from the storage vendors, I don't think there's any work to be done in the storage layers. It's purely a FS+MM problem.

Quoted 3 lines · Keith Busch

That said, I was hoping you were going to suggest supporting 16k logical block sizes. Not a problem on some arch's, but still problematic when PAGE_SIZE is 4k. :)

I was hoping Luis was going to propose a session on LBA size > PAGE_SIZE. Funnily, while the pressure is coming from the storage vendors, I don't think there's any work to be done in the storage layers.  It's purely a FS+MM problem.

Quoted 9 lines · Matthew Wilcox

Quoted 3 lines · Keith Busch

That said, I was hoping you were going to suggest supporting 16k logical block sizes. Not a problem on some arch's, but still problematic when PAGE_SIZE is 4k. :)

I was hoping Luis was going to propose a session on LBA size > PAGE_SIZE. Funnily, while the pressure is coming from the storage vendors, I don't think there's any work to be done in the storage layers.  It's purely a FS+MM problem.

Heh, I can do the fools rush in bit, especially if what we're interested in the minimum it would take to support this ...

The FS problem could be solved simply by saying FS block size must equal device block size, then it becomes purely a MM issue.

The MM
issue could be solved by adding a page order attribute to struct address_space and insisting that pagecache/filemap functions in mm/filemap.c all have to operate on objects that are an integer multiple of the address space order. The base allocator is filemap_alloc_folio, which already has an apparently always zero order parameter (hmmm...) and it always seems to be called from sites that have the address_space, so it could simply be modified to always operate at the address_space order.

Quoted 14 lines · James Bottomley

Quoted 9 lines · Matthew Wilcox

Quoted 3 lines · Keith Busch

That said, I was hoping you were going to suggest supporting 16k logical block sizes. Not a problem on some arch's, but still problematic when PAGE_SIZE is 4k. :)

I was hoping Luis was going to propose a session on LBA size > PAGE_SIZE. Funnily, while the pressure is coming from the storage vendors, I don't think there's any work to be done in the storage layers.  It's purely a FS+MM problem.

Heh, I can do the fools rush in bit, especially if what we're interested in the minimum it would take to support this ...

Quoted 2 lines

The FS problem could be solved simply by saying FS block size must equal device block size, then it becomes purely a MM issue.

Spoken like somebody who's never converted a filesystem to supporting large folios.  There are a number of issues:

1. The obvious; use of PAGE_SIZE and/or PAGE_SHIFT

2. Use of kmap-family to access, eg directories.  You can't kmap an entire folio, only one page at a time.  And if a dentry is split across a page boundary ...

3. buffer_heads do not currently support large folios.  Working on it.

Probably a few other things I forget.  But look through the recent patches to AFS, CIFS, NFS, XFS, iomap that do folio conversions. A lot of it is pretty mechanical, but some of it takes hard thought. And if you have ideas about how to handle ext2 directories, I'm all ears.

Quoted 10 lines

The MM issue could be solved by adding a page order attribute to struct address_space and insisting that pagecache/filemap functions in mm/filemap.c all have to operate on objects that are an integer multiple of the address space order.  The base allocator is filemap_alloc_folio, which already has an apparently always zero order parameter (hmmm...) and it always seems to be called from sites that
have the address_space, so it could simply be modified to always operate at the address_space order.

Oh, I have a patch for that.  That's the easy part.  The hard part is plugging your ears to the screams of the MM people who are convinced that fragmentation will make it impossible to mount your filesystem.

Quoted 15 lines · Matthew Wilcox

Quoted 14 lines · James Bottomley

Quoted 9 lines · Matthew Wilcox

Quoted 3 lines · Keith Busch

That said, I was hoping you were going to suggest supporting 16k logical block sizes. Not a problem on some arch's, but still problematic when PAGE_SIZE is 4k. :)

I was hoping Luis was going to propose a session on LBA size > PAGE_SIZE. Funnily, while the pressure is coming from the storage vendors, I don't think there's any work to be done in the storage layers.  It's purely a FS+MM problem.

Heh, I can do the fools rush in bit, especially if what we're interested in the minimum it would take to support this ...

Quoted 7 lines

Quoted 2 lines

The FS problem could be solved simply by saying FS block size must equal device block size, then it becomes purely a MM issue.

Spoken like somebody who's never converted a filesystem to supporting large folios.  There are a number of issues:

1. The obvious; use of PAGE_SIZE and/or PAGE_SHIFT

Well, yes, a filesystem has to be aware it's using a block size larger than page size.

Quoted 3 lines

2. Use of kmap-family to access, eg directories.  You can't kmap an entire folio, only one page at a time.  And if a dentry is split across a page boundary ...

Is kmap relevant? It's only used for reading user pages in the kernel and I can't see why a filesystem would use it unless it wants to pack inodes into pages that also contain user data, which is an optimization not a fundamental issue (although I grant that as the blocksize grows it becomes more useful) so it doesn't have to be part of the minimum viable prototype.

Quoted 2 lines

3. buffer_heads do not currently support large folios.  Working on it.

Yes, I always forget filesystems still use the buffer cache. But fundamentally the buffer_head structure can cope with buffers that span pages so most of the logic changes would be around grow_dev_page(). It seems somewhat messy but not too hard.

Quoted 5 lines

Probably a few other things I forget.  But look through the recent patches to AFS, CIFS, NFS, XFS, iomap that do folio conversions. A lot of it is pretty mechanical, but some of it takes hard thought. And if you have ideas about how to handle ext2 directories, I'm all ears.

OK, so I can see you were waiting for someone to touch a nerve, but if I can go back to the stated goal, I never really thought *every* filesystem would be suitable for block size > page size, so simply getting a few of the modern ones working would be good enough for the minimum viable prototype.

I fully understand that eventually we'll need to get a single large buffer to span discontiguous pages ... I noted that in the bit you cut, but I don't see why the prototype shouldn't start with contiguous pages.

I'm getting more and more
comfortable with the idea that "Linux doesn't support block sizes > PAGE_SIZE on 32-bit machines" is an acceptable answer.

XFS already works with arbitrary-order folios.

The only needed piece is
specifying to the VFS that there's a minimum order for this particular inode, and having the VFS honour that everywhere.

Quoted 3 lines · Matthew Wilcox

I'm getting more and more
comfortable with the idea that "Linux doesn't support block sizes > PAGE_SIZE on 32-bit machines" is an acceptable answer.

First of all filesystems would need to add support for a larger block sizes > PAGE_SIZE, and that takes effort. It is also a support question too.

I think garnering consensus from filesystem developers we don't want to support block sizes > PAGE_SIZE on 32-bit systems would be a good thing to review at LSFMM or even on this list. I hightly doubt anyone is interested in that support.

Quoted 1 line

XFS already works with arbitrary-order folios.

But block sizes > PAGE_SIZE is work which is still not merged. It *can* be with time. That would allow one to muck with larger block sizes than 4k on x86-64 for instance. Without this, you can't play ball.

Quoted 3 lines

The only needed piece is
specifying to the VFS that there's a minimum order for this particular inode, and having the VFS honour that everywhere.

Other than the above too, don't we still also need to figure out what fs APIs would incur larger order folios? And then what about corner cases with the page cache?

I was hoping some of these nooks and crannies could be explored with tmpfs.

Quoted 7 lines · Luis Chamberlain

Quoted 1 line · Matthew Wilcox

XFS already works with arbitrary-order folios.

But block sizes > PAGE_SIZE is work which is still not merged. It *can* be with time. That would allow one to muck with larger block sizes than 4k on x86-64 for instance. Without this, you can't play ball.

Do you mean that XFS is checking that fs block size <= PAGE_SIZE and that check needs to be dropped? If so, I don't see where that happens.

Quoted 9 lines · Matthew Wilcox

Quoted 7 lines · Luis Chamberlain

Quoted 1 line · Matthew Wilcox

XFS already works with arbitrary-order folios.

But block sizes > PAGE_SIZE is work which is still not merged. It *can* be with time. That would allow one to muck with larger block sizes than 4k on x86-64 for instance. Without this, you can't play ball.

Quoted 2 lines

Do you mean that XFS is checking that fs block size <= PAGE_SIZE and that check needs to be dropped? If so, I don't see where that happens.

None of that. Back in 2018 Chinner had prototyped XFS support with larger block size > PAGE_SIZE:

https://lwn.net/ml/linux-fsdevel/[email protected]/

I just did a quick attempt to rebased it and most of the left over work is actually on IOMAP for writeback and zero / writes requiring a new zero-around functionality. All bugs on the rebase are my own, only compile tested so far, and not happy with some of the changes I had to make so likely could use tons more love:

https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git/log/?h=20230307-larger-bs-then-ps-xfs

But it should give you an idea of what type of things filesystems need to do.

Quoted 3 lines · Matthew Wilcox

I'm getting more and more
comfortable with the idea that "Linux doesn't support block sizes > PAGE_SIZE on 32-bit machines" is an acceptable answer.

First of all filesystems would need to add support for a larger block sizes > PAGE_SIZE, and that takes effort. It is also a support question too.

I think garnering consensus from filesystem developers we don't want to support block sizes > PAGE_SIZE on 32-bit systems would be a good thing to review at LSFMM or even on this list. I hightly doubt anyone is interested in that support.

Quoted 1 line

XFS already works with arbitrary-order folios.

But block sizes > PAGE_SIZE is work which is still not merged. It *can* be with time. That would allow one to muck with larger block sizes than 4k on x86-64 for instance. Without this, you can't play ball.

Quoted 3 lines

The only needed piece is
specifying to the VFS that there's a minimum order for this particular inode, and having the VFS honour that everywhere.

Other than the above too, don't we still also need to figure out what fs APIs would incur larger order folios? And then what about corner cases with the page cache?

I was hoping some of these nooks and crannies could be explored with tmpfs.

Quoted 4 lines

I fully understand that eventually we'll need to get a single large buffer to span discontiguous pages ... I noted that in the bit you cut, but I don't see why the prototype shouldn't start with contiguous pages.

I disagree that this is a desirable goal.  To solve the scalability issues we have in the VFS, we need to manage memory in larger chunks than PAGE_SIZE.  That makes the concerns expressed in previous years moot.

The hard part is plugging your ears to the screams of the MM people who are convinced that fragmentation will make it impossible to mount your filesystem.

Quoted 3 lines · Keith Busch

That said, I was hoping you were going to suggest supporting 16k logical block sizes. Not a problem on some arch's, but still problematic when PAGE_SIZE is 4k. :)

I was hoping Luis was going to propose a session on LBA size > PAGE_SIZE. Funnily, while the pressure is coming from the storage vendors, I don't think there's any work to be done in the storage layers. It's purely a FS+MM problem.

You'd hope most of it is left to FS + MM, but I'm not yet sure that's quite it yet. Initial experimentation shows just enabling > PAGE_SIZE physical & logical block NVMe devices gets brought down to 512 bytes. That seems odd to say the least. Would changing this be an issue now?

Quoted 5 lines · Luis Chamberlain

You'd hope most of it is left to FS + MM, but I'm not yet sure that's quite it yet. Initial experimentation shows just enabling > PAGE_SIZE physical & logical block NVMe devices gets brought down to 512 bytes. That seems odd to say the least. Would changing this be an issue now?

I think you're talking about removing this part:

---

drivers/nvme/host/core.c +0 -2

index c2730b116dc68..2c528f56c2973 100644
@@ -1828,17 +1828,7 @@ static void nvme_update_disk_info(struct gendisk *disk,
 	unsigned short bs = 1 << ns->lba_shift;
 	u32 atomic_bs, phys_bs, io_opt = 0;
 
-	/*
-	 * The block layer can't support LBA sizes larger than the page size

Quoted 18 lines · Keith Busch

Quoted 5 lines · Luis Chamberlain

You'd hope most of it is left to FS + MM, but I'm not yet sure that's quite it yet. Initial experimentation shows just enabling > PAGE_SIZE physical & logical block NVMe devices gets brought down to 512 bytes. That seems odd to say the least. Would changing this be an issue now?

I think you're talking about removing this part:

---

drivers/nvme/host/core.c +0 -0

index c2730b116dc68..2c528f56c2973 100644
@@ -1828,17 +1828,7 @@ static void nvme_update_disk_info(struct gendisk *disk,
 	unsigned short bs = 1 << ns->lba_shift;
 	u32 atomic_bs, phys_bs, io_opt = 0;
 

drivers/nvme/host/core.c static void nvme_update_disk_info(struct gendisk *disk, · line 1832

Quoted 12 lines

drivers/nvme/host/core.c +0 -10

-	/*
-	 * The block layer can't support LBA sizes larger than the page size
-	 * yet, so catch this early and don't allow block I/O.
-	 */
-	if (ns->lba_shift > PAGE_SHIFT) {
-		capacity = 0;
-		bs = (1 << 9);
-	}
-
 	blk_integrity_unregister(disk);
-
 	atomic_bs = phys_bs = bs;

Yes, clearly it says *yet* so that begs the question what would be required?

Also, going down to 512 seems a bit dramatic, so why not just match the PAGE_SIZE so 4k? Would such a comprmise for now break some stuff?

Quoted 19 lines · Luis Chamberlain

Quoted 18 lines · Keith Busch

Quoted 5 lines · Luis Chamberlain

You'd hope most of it is left to FS + MM, but I'm not yet sure that's quite it yet. Initial experimentation shows just enabling > PAGE_SIZE physical & logical block NVMe devices gets brought down to 512 bytes. That seems odd to say the least. Would changing this be an issue now?

I think you're talking about removing this part:

---

drivers/nvme/host/core.c +0 -0

index c2730b116dc68..2c528f56c2973 100644
@@ -1828,17 +1828,7 @@ static void nvme_update_disk_info(struct gendisk *disk,
 	unsigned short bs = 1 << ns->lba_shift;
 	u32 atomic_bs, phys_bs, io_opt = 0;
 

Quoted 15 lines

drivers/nvme/host/core.c static void nvme_update_disk_info(struct gendisk *disk, · line 1832

Quoted 12 lines

drivers/nvme/host/core.c +0 -10

-	/*
-	 * The block layer can't support LBA sizes larger than the page size
-	 * yet, so catch this early and don't allow block I/O.
-	 */
-	if (ns->lba_shift > PAGE_SHIFT) {
-		capacity = 0;
-		bs = (1 << 9);
-	}
-
 	blk_integrity_unregister(disk);
-
 	atomic_bs = phys_bs = bs;

Yes, clearly it says *yet* so that begs the question what would be required?

Oh, gotcha. I'll work on a list of places it currently crashes.

Quoted 2 lines

Also, going down to 512 seems a bit dramatic, so why not just match the PAGE_SIZE so 4k? Would such a comprmise for now break some stuff?

The capacity set to zero ensures it can't be used through the block stack, so the logical block size limit is unused.

512 is just a default value. We only
bring up the handle so you can administrate it with passthrough commands.

Quoted 2 lines

Yes, clearly it says *yet* so that begs the question what would be required?

Oh, gotcha. I'll work on a list of places it currently crashes.

Hi Keith,

Quoted 5 lines · Keith Busch

Quoted 2 lines

Yes, clearly it says *yet* so that begs the question what would be required?

Oh, gotcha. I'll work on a list of places it currently crashes.

I started looking into this to see why it crashes when we increase the LBA size of a block device greater than the page size. These are my primary findings:

- Block device aops (address_space_operations) are all based on buffer head, which limits us to work on only PAGE_SIZE chunks.

For a 8k LBA size, the stack trace you posted ultimately fails inside alloc_page_buffers as the size will be > PAGE_SIZE.

struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size, bool retry)

I'm gathering there is generic interest in this topic though.

This also got me thinking about ways to try to replicate larger IO virtual devices a bit better too.

Quoted 9 lines · Matthew Wilcox

Quoted 3 lines · Keith Busch

That said, I was hoping you were going to suggest supporting 16k logical block sizes. Not a problem on some arch's, but still problematic when PAGE_SIZE is 4k. :)

I was hoping Luis was going to propose a session on LBA size > PAGE_SIZE. Funnily, while the pressure is coming from the storage vendors, I don't think there's any work to be done in the storage layers. It's purely a FS+MM problem.

You'd hope most of it is left to FS + MM, but I'm not yet sure that's quite it yet. Initial experimentation shows just enabling > PAGE_SIZE physical & logical block NVMe devices gets brought down to 512 bytes. That seems odd to say the least. Would changing this be an issue now?

I'm gathering there is generic interest in this topic though. So one thing we *could* do is perhaps review lay-of-the-land of interest and break down what we all think are things likely could be done / needed. At the very least we can come out together knowing the unknowns together.

I started to think about some of these things a while ago and with the help of Willy I tried to break down some of the items I gathered from him into community OKRs (super informal itemization of goals and sub tasks which would complete such goals) and started trying to take a stab at them with our team, but obviously I think it would be great if we all just divide & and conquer here. So maybe reviewing these and extending them as a community would be good:

https://kernelnewbies.org/KernelProjects/large-block-size

I'm recently interested in tmpfs so will be taking a stab at higher order page size support there to see what blows up.

The other stuff like general IOMAP conversion is pretty well known, and we already I think have a proposed session on that. But there is also even smaller fish to fry, like *just* doing a baseline with some filesystems with 4 KiB block size seems in order.

Hearing filesystem developer's thoughts on support for larger block size in light of lower order PAGE_SIZE would be good, given one of the odd situations some distributions / teams find themselves in is trying to support larger block sizes but with difficult access to higher PAGE_SIZE systems. Are there ways to simplify this / help us in general? Without it's a bit hard to muck around with some of this in terms of support long term. This also got me thinking about ways to try to replicate larger IO virtual devices a bit better too. While paying a cloud provider to test this is one nice option, it'd be great if I can just do this in house with some hacks too. For virtio-blk-pci at least, for instance, I wondered whether using just the host page cache suffices, or would a 4K page cache on the host modify say a 16 k emualated io controller results significantly? How do we most effectively virtualize 16k controllers in-house?

To help with experimenting with large io and NVMe / virtio-blk-pci I recented added support to intantiate tons of large IO devices to kdevops [0], with it it should be easy to reproduce odd issues we may come up with. For instnace it should be possible to subsequently extend the kdevops fstests or blktests automation support with just a few Kconfig files to use some of these largio devices to see what blows up.

Quoted 29 lines · Luis Chamberlain

Quoted 9 lines · Matthew Wilcox

Quoted 3 lines · Keith Busch

That said, I was hoping you were going to suggest supporting 16k logical block sizes. Not a problem on some arch's, but still problematic when PAGE_SIZE is 4k. :)

I was hoping Luis was going to propose a session on LBA size > PAGE_SIZE. Funnily, while the pressure is coming from the storage vendors, I don't think there's any work to be done in the storage layers. It's purely a FS+MM problem.

You'd hope most of it is left to FS + MM, but I'm not yet sure that's quite it yet. Initial experimentation shows just enabling > PAGE_SIZE physical & logical block NVMe devices gets brought down to 512 bytes. That seems odd to say the least. Would changing this be an issue now?

I'm gathering there is generic interest in this topic though. So one thing we *could* do is perhaps review lay-of-the-land of interest and break down what we all think are things likely could be done / needed. At the very least we can come out together knowing the unknowns together.

I started to think about some of these things a while ago and with the help of Willy I tried to break down some of the items I gathered from him into community OKRs (super informal itemization of goals and sub tasks which would complete such goals) and started trying to take a stab at them with our team, but obviously I think it would be great if we all just divide & and conquer here. So maybe reviewing these and extending them as a community would be good:

Quoted 5 lines

https://protect2.fireeye.com/v1/url?k=bd8b143b-dcf6fc7c-bd8a9f74-74fe485fff30-e62d6b1f7e2b2236&q=1&e=64cdf12c-742d-4d0b-9870-bfc5c26dba21&u=https%3A%2F%2Fkernelnewbies.org%2FKernelProjects%2Flarge-block-size

I'm recently interested in tmpfs so will be taking a stab at higher order page size support there to see what blows up.

Cool.

Quoted 20 lines

The other stuff like general IOMAP conversion is pretty well known, and we already I think have a proposed session on that. But there is also even smaller fish to fry, like *just* doing a baseline with some filesystems with 4 KiB block size seems in order.

Hearing filesystem developer's thoughts on support for larger block size in light of lower order PAGE_SIZE would be good, given one of the odd situations some distributions / teams find themselves in is trying to support larger block sizes but with difficult access to higher PAGE_SIZE systems. Are there ways to simplify this / help us in general? Without it's a bit hard to muck around with some of this in terms of support long term. This also got me thinking about ways to try to replicate larger IO virtual devices a bit better too. While paying a cloud provider to test this is one nice option, it'd be great if I can just do this in house with some hacks too. For virtio-blk-pci at least, for instance, I wondered whether using just the host page cache suffices, or would a 4K page cache on the host modify say a 16 k emualated io controller results significantly? How do we most effectively virtualize 16k controllers in-house?

Quoted 7 lines

To help with experimenting with large io and NVMe / virtio-blk-pci I recented added support to intantiate tons of large IO devices to kdevops [0], with it it should be easy to reproduce odd issues we may come up with. For instnace it should be possible to subsequently extend the kdevops fstests or blktests automation support with just a few Kconfig files to use some of these largio devices to see what blows up.

We could implement a (virtual) zoned device, and expose each zone as a block. That gives us the required large block characteristics, and with
a bit of luck we might be able to dial up to really large block sizes like the 256M sizes on current SMR drives.

ublk might be a good starting point.

We could implement a (virtual) zoned device, and expose each zone as a block. That gives us the required large block characteristics, and with a bit of luck we might be able to dial up to really large block sizes like the 256M sizes on current SMR drives.
ublk might be a good starting point.

Quoted 5 lines · Hannes Reinecke

We could implement a (virtual) zoned device, and expose each zone as a block. That gives us the required large block characteristics, and with a bit of luck we might be able to dial up to really large block sizes like the 256M sizes on current SMR drives.
ublk might be a good starting point.

Ummmm. Is supporting 256MB block sizes really a desired goal? I suggest that is far past the knee of the curve; if we can only write 256MB chunks as a single entity, we're looking more at a filesystem redesign than we are at making filesystems and the MM support 256MB size blocks.

The current work is all going towards tracking memory in larger chunks, so writing back, eg, 64kB chunks of the file. But if 256MB is where we're going, we need to be thinking more like a RAID device and accumulating writes into a log that we can then blast out in a single giant write.

Quoted 7 lines · Matthew Wilcox

Quoted 5 lines · Hannes Reinecke

We could implement a (virtual) zoned device, and expose each zone as a block. That gives us the required large block characteristics, and with a bit of luck we might be able to dial up to really large block sizes like the 256M sizes on current SMR drives.
ublk might be a good starting point.

Quoted 5 lines

Ummmm. Is supporting 256MB block sizes really a desired goal? I suggest that is far past the knee of the curve; if we can only write 256MB chunks as a single entity, we're looking more at a filesystem redesign than we are at making filesystems and the MM support 256MB size blocks.

Naa, not really. It _would_ be cool as we could get rid of all the cludges which have nowadays re sequential writes.
And, remember, 256M is just a number someone thought to be a good compromise. If we end up with a lower number (16M?) we might be able to convince the powers that be to change their zone size. Heck, with 16M block size there wouldn't be a _need_ for zones in the first place.

But yeah, 256M is excessive. Initially I would shoot for something like 2M.

Quoted 21 lines · Hannes Reinecke

Quoted 7 lines · Matthew Wilcox

Quoted 5 lines · Hannes Reinecke

We could implement a (virtual) zoned device, and expose each zone as a block. That gives us the required large block characteristics, and with a bit of luck we might be able to dial up to really large block sizes like the 256M sizes on current SMR drives.
ublk might be a good starting point.

Quoted 5 lines

Ummmm. Is supporting 256MB block sizes really a desired goal? I suggest that is far past the knee of the curve; if we can only write 256MB chunks as a single entity, we're looking more at a filesystem redesign than we are at making filesystems and the MM support 256MB size blocks.

Naa, not really. It _would_ be cool as we could get rid of all the cludges which have nowadays re sequential writes.
And, remember, 256M is just a number someone thought to be a good compromise. If we end up with a lower number (16M?) we might be able to convince the powers that be to change their zone size. Heck, with 16M block size there wouldn't be a _need_ for zones in the first place.

Quoted 2 lines

But yeah, 256M is excessive. Initially I would shoot for something like 2M.

I think we're talking about different things (probably different storage vendors want different things, or even different people at the same storage vendor want different things).

Luis and I are talking about larger LBA sizes. That is, the minimum read/write size from the block device is 16kB or 64kB or whatever. In this scenario, the minimum amount of space occupied by a file goes up from 512 bytes or 4kB to 64kB. That's doable, even if somewhat suboptimal.

Your concern seems to be more around shingled devices (or their equivalent in SSD terms) where there are large zones which are append-only, but you can still random-read 512 byte LBAs. I think there are different solutions to these problems, and people are working on both of these problems.

But if storage vendors are really pushing for 256MB LBAs, then that's going to need a third kind of solution, and I'm not aware of anyone working on that.

Quoted 21 lines · Hannes Reinecke

Quoted 7 lines · Matthew Wilcox

Quoted 5 lines · Hannes Reinecke

We could implement a (virtual) zoned device, and expose each zone as a block. That gives us the required large block characteristics, and with a bit of luck we might be able to dial up to really large block sizes like the 256M sizes on current SMR drives.
ublk might be a good starting point.

Quoted 5 lines

Ummmm. Is supporting 256MB block sizes really a desired goal? I suggest that is far past the knee of the curve; if we can only write 256MB chunks as a single entity, we're looking more at a filesystem redesign than we are at making filesystems and the MM support 256MB size blocks.

Naa, not really. It _would_ be cool as we could get rid of all the cludges which have nowadays re sequential writes.
And, remember, 256M is just a number someone thought to be a good compromise. If we end up with a lower number (16M?) we might be able to convince the powers that be to change their zone size. Heck, with 16M block size there wouldn't be a _need_ for zones in the first place.

Quoted 2 lines

But yeah, 256M is excessive. Initially I would shoot for something like 2M.

I think we're talking about different things (probably different storage vendors want different things, or even different people at the same storage vendor want different things).

Luis and I are talking about larger LBA sizes. That is, the minimum read/write size from the block device is 16kB or 64kB or whatever. In this scenario, the minimum amount of space occupied by a file goes up from 512 bytes or 4kB to 64kB. That's doable, even if somewhat suboptimal.

Your concern seems to be more around shingled devices (or their equivalent in SSD terms) where there are large zones which are append-only, but you can still random-read 512 byte LBAs. I think there are different solutions to these problems, and people are working on both of these problems.

But if storage vendors are really pushing for 256MB LBAs, then that's going to need a third kind of solution, and I'm not aware of anyone working on that.

Quoted 21 lines · Hannes Reinecke

Quoted 7 lines · Matthew Wilcox

Quoted 5 lines · Hannes Reinecke

We could implement a (virtual) zoned device, and expose each zone as a block. That gives us the required large block characteristics, and with a bit of luck we might be able to dial up to really large block sizes like the 256M sizes on current SMR drives.
ublk might be a good starting point.

Quoted 5 lines

Ummmm. Is supporting 256MB block sizes really a desired goal? I suggest that is far past the knee of the curve; if we can only write 256MB chunks as a single entity, we're looking more at a filesystem redesign than we are at making filesystems and the MM support 256MB size blocks.

Naa, not really. It _would_ be cool as we could get rid of all the cludges which have nowadays re sequential writes.
And, remember, 256M is just a number someone thought to be a good compromise. If we end up with a lower number (16M?) we might be able to convince the powers that be to change their zone size. Heck, with 16M block size there wouldn't be a _need_ for zones in the first place.

Quoted 2 lines

But yeah, 256M is excessive. Initially I would shoot for something like 2M.

I think we're talking about different things (probably different storage vendors want different things, or even different people at the same storage vendor want different things).

Luis and I are talking about larger LBA sizes. That is, the minimum read/write size from the block device is 16kB or 64kB or whatever. In this scenario, the minimum amount of space occupied by a file goes up from 512 bytes or 4kB to 64kB. That's doable, even if somewhat suboptimal.

Your concern seems to be more around shingled devices (or their equivalent in SSD terms) where there are large zones which are append-only, but you can still random-read 512 byte LBAs. I think there are different solutions to these problems, and people are working on both of these problems.

But if storage vendors are really pushing for 256MB LBAs, then that's going to need a third kind of solution, and I'm not aware of anyone working on that.

Quoted 10 lines · Matthew Wilcox

I think we're talking about different things (probably different storage vendors want different things, or even different people at the same storage vendor want different things).

Luis and I are talking about larger LBA sizes. That is, the minimum read/write size from the block device is 16kB or 64kB or whatever. In this scenario, the minimum amount of space occupied by a file goes up from 512 bytes or 4kB to 64kB. That's doable, even if somewhat suboptimal.

And so do I. One can view zones as really large LBAs.

Indeed it might be suboptimal from the OS point of view. But from the device point of view it won't.
And, in fact, with devices becoming faster and faster the question is whether sticking with relatively small sectors won't become a limiting factor eventually.

Quoted 6 lines

Your concern seems to be more around shingled devices (or their equivalent in SSD terms) where there are large zones which are append-only, but you can still random-read 512 byte LBAs. I think there are different solutions to these problems, and people are working on both of these problems.

My point being that zones are just there because the I/O stack can only deal with sectors up to 4k. If the I/O stack would be capable of dealing with larger LBAs one could identify a zone with an LBA, and the entire issue of append-only and sequential writes would be moot.
Even the entire concept of zones becomes irrelevant as the OS would trivially only write entire zones.

Quoted 25 lines · Hannes Reinecke

Quoted 10 lines · Matthew Wilcox

I think we're talking about different things (probably different storage vendors want different things, or even different people at the same storage vendor want different things).

Luis and I are talking about larger LBA sizes. That is, the minimum read/write size from the block device is 16kB or 64kB or whatever. In this scenario, the minimum amount of space occupied by a file goes up from 512 bytes or 4kB to 64kB. That's doable, even if somewhat suboptimal.

And so do I. One can view zones as really large LBAs.

Indeed it might be suboptimal from the OS point of view. But from the device point of view it won't.
And, in fact, with devices becoming faster and faster the question is whether sticking with relatively small sectors won't become a limiting factor eventually.

Quoted 6 lines

Your concern seems to be more around shingled devices (or their equivalent in SSD terms) where there are large zones which are append-only, but you can still random-read 512 byte LBAs. I think there are different solutions to these problems, and people are working on both of these problems.

Quoted 6 lines

My point being that zones are just there because the I/O stack can only deal with sectors up to 4k. If the I/O stack would be capable of dealing with larger LBAs one could identify a zone with an LBA, and the entire issue of append-only and sequential writes would be moot.
Even the entire concept of zones becomes irrelevant as the OS would trivially only write entire zones.

All current filesystems that I'm aware of require their fs block size to be >= LBA size. That is, you can't take a 512-byte blocksize ext2 filesystem and put it on a 4kB LBA storage device.

That means that files can only grow/shrink in 256MB increments. I don't think that amount of wasted space is going to be acceptable. So if we're serious about going down this path, we need to tell filesystem people to start working out how to support fs block size < LBA size.

That's a big ask, so let's be sure storage vendors actually want this. Both supporting zoned devices & suporting 16k/64k block sizes are easier asks.

All current filesystems that I'm aware of require their fs block size to be >= LBA size. That is, you can't take a 512-byte blocksize ext2 filesystem and put it on a 4kB LBA storage device.

That means that files can only grow/shrink in 256MB increments. I don't think that amount of wasted space is going to be acceptable. So if we're serious about going down this path, we need to tell filesystem people to start working out how to support fs block size < LBA size.

That's a big ask, so let's be sure storage vendors actually want this. Both supporting zoned devices & suporting 16k/64k block sizes are easier asks.

What HDD vendors want is to be able to have 32k or even 64k *physical* sector sizes. This allows for much more efficient erasure codes, so it will increase their byte capacity now that it's no longer easier to get capacity boosts by squeezing the tracks closer and closer, and their have been various engineering tradeoffs with SMR, HAMR, and MAMR. HDD vendors have been asking for this at LSF/MM, and in other venues for ***years***.

Quoted 7 lines · Theodore Ts'o

What HDD vendors want is to be able to have 32k or even 64k *physical* sector sizes.  This allows for much more efficient erasure codes, so it will increase their byte capacity now that it's no longer easier to get capacity boosts by squeezing the tracks closer and closer, and their have been various engineering tradeoffs with SMR, HAMR, and MAMR.  HDD vendors have been asking for this at LSF/MM, and in othervenues for ***years***.

I've been reminded by a friend who works on the drive side that a motivation for the SSD vendors is (essentially) the size of sector_t. Once the drive needs to support more than 2/4 billion sectors, they need to move to a 64-bit sector size, so the amount of memory consumed by the FTL doubles, the CPU data cache becomes half as effective, etc. That significantly increases the BOM for the drive, and so they have to charge more.  With a 512-byte LBA, that's 2TB; with a 4096-byte LBA, it's at 16TB and with a 64k LBA, they can keep using 32-bit LBA numbers all the way up to 256TB.

Quoted 9 lines · Matthew Wilcox

Quoted 7 lines · Theodore Ts'o

What HDD vendors want is to be able to have 32k or even 64k *physical* sector sizes.  This allows for much more efficient erasure codes, so it will increase their byte capacity now that it's no longer easier to get capacity boosts by squeezing the tracks closer and closer, and their have been various engineering tradeoffs with SMR, HAMR, and MAMR.  HDD vendors have been asking for this at LSF/MM, and in othervenues for ***years***.

Quoted 9 lines

I've been reminded by a friend who works on the drive side that a motivation for the SSD vendors is (essentially) the size of sector_t. Once the drive needs to support more than 2/4 billion sectors, they need to move to a 64-bit sector size, so the amount of memory consumed by the FTL doubles, the CPU data cache becomes half as effective, etc. That significantly increases the BOM for the drive, and so they have to charge more.  With a 512-byte LBA, that's 2TB; with a 4096-byte LBA, it's at 16TB and with a 64k LBA, they can keep using 32-bit LBA numbers all the way up to 256TB.

I thought the FTL operated on physical sectors and the logical to physical was done as a RMW through the FTL? In which case sector_t shouldn't matter to the SSD vendors for FTL management because they can keep the logical sector size while increasing the physical one. Obviously if physical size goes above the FS block size, the drives will behave suboptimally with RMWs, which is why 4k physical is the max currently.

Quoted 22 lines · James Bottomley

Quoted 10 lines · Matthew Wilcox

Quoted 8 lines · Theodore Ts'o

What HDD vendors want is to be able to have 32k or even 64k *physical* sector sizes.  This allows for much more efficient erasure codes, so it will increase their byte capacity now that it's no longer easier to get capacity boosts by squeezing the tracks closer and closer, and their have been various engineering tradeoffs with SMR, HAMR, and MAMR.  HDD vendors have been asking for this at LSF/MM, and in othervenues for ***years***.

Quoted 10 lines

I've been reminded by a friend who works on the drive side that a motivation for the SSD vendors is (essentially) the size of sector_t. Once the drive needs to support more than 2/4 billion sectors, they need to move to a 64-bit sector size, so the amount of memory consumed by the FTL doubles, the CPU data cache becomes half as effective, etc. That significantly increases the BOM for the drive, and so they have to charge more.  With a 512-byte LBA, that's 2TB; with a 4096-byte LBA, it's at 16TB and with a 64k LBA, they can keep using 32-bit LBA numbers all the way up to 256TB.

Quoted 8 lines

I thought the FTL operated on physical sectors and the logical to physical was done as a RMW through the FTL?  In which case sector_t shouldn't matter to the SSD vendors for FTL management because they can keep the logical sector size while increasing the physical one. Obviously if physical size goes above the FS block size, the drives will behave suboptimally with RMWs, which is why 4k physical is the max currently.

FTL designs are complex. We have ways to maintain sector sizes under 64 bits, but this is a common industry problem.

The media itself does not normally oeprate at 4K. Page siges can be 16K, 32K, etc.

Increasing the block size would allow for better host/device
cooperation. As Ted mentions, this has been a requirement for HDD and SSD vendor for years. It seems to us that the time is right now and that we have mechanisms in Linux to do the plumbing. Folios is ovbiously a big part of this.

Quoted 5 lines · Javier González

FTL designs are complex. We have ways to maintain sector sizes under 64 bits, but this is a common industry problem.

The media itself does not normally oeprate at 4K. Page siges can be 16K, 32K, etc.

Right, and we've always said if we knew what this size was we could make better block write decisions. However, today if you look what most NVMe devices are reporting, it's a bit sub-optimal:

jejb@lingrow:/sys/block/nvme1n1/queue> cat logical_block_size 512
jejb@lingrow:/sys/block/nvme1n1/queue> cat physical_block_size 512
jejb@lingrow:/sys/block/nvme1n1/queue> cat optimal_io_size 0

If we do get Linux to support large block sizes, are we actually going to get better information out of the devices?

Quoted 5 lines

Increasing the block size would allow for better host/device
cooperation. As Ted mentions, this has been a requirement for HDD and SSD vendor for years. It seems to us that the time is right now and that we have mechanisms in Linux to do the plumbing. Folios is ovbiously a big part of this.

Well a decade ago we did a lot of work to support 4k sector devices. Ultimately the industry went with 512 logical/4k physical devices because of problems with non-Linux proprietary OSs but you could still use 4k today if you wanted (I've actually still got a working 4k SCSI drive), so why is no NVMe device doing that?

Well a decade ago we did a lot of work to support 4k sector devices. Ultimately the industry went with 512 logical/4k physical devices because of problems with non-Linux proprietary OSs but you could still use 4k today if you wanted (I've actually still got a working 4k SCSI drive), so why is no NVMe device doing that?

This is not to say I think larger block sizes is in any way a bad idea ... I just think that given the history, it will be driven by application needs rather than what the manufacturers tell us.

Quoted 3 lines

This is not to say I think larger block sizes is in any way a bad idea ... I just think that given the history, it will be driven by application needs rather than what the manufacturers tell us.

I think it would be beneficial for Linux to support filesystem blocks larger than the page size. Based on experience outlined above, I am not convinced larger logical block sizes will get much traction. But that doesn't prevent devices from advertising larger physical/minimum/optimal I/O sizes and for us to handle those more gracefully than we currently do.

Quoted 10 lines · Martin K. Petersen

Quoted 3 lines

This is not to say I think larger block sizes is in any way a bad idea ... I just think that given the history, it will be driven by application needs rather than what the manufacturers tell us.

I think it would be beneficial for Linux to support filesystem blocks larger than the page size. Based on experience outlined above, I am not convinced larger logical block sizes will get much traction. But that doesn't prevent devices from advertising larger physical/minimum/optimal I/O sizes and for us to handle those more gracefully than we currently do.

Right, I was wondering if we could try to persuade the Manufacturers to advertise a more meaningful optimal I/O size ...

A big future question is of course how / when to use these for filesystems. Should there be, for instance a 'mkfs --optimal-bs' or something which may look whatever hints the media uses ? Or do we just leaves the magic incantations to the admins?

-----Original Message-----
From: Martin K. Petersen [mailto:[email protected]]
Sent: Thursday, March 9, 2023 2:28 PM
To: Luis Chamberlain <[email protected]>
Cc: James Bottomley <[email protected]>; Dan Helmick <[email protected]>; Martin K. Petersen <[email protected]>; Javier González <[email protected]>; Matthew Wilcox <[email protected]>; Theodore Ts'o <[email protected]>; Hannes Reinecke <[email protected]>; Keith Busch <[email protected]>; Pankaj Raghav <[email protected]>; Daniel Gomez <[email protected]>; [email protected]; [email protected]; [email protected]; linux- [email protected]
Subject: Re: [LSF/MM/BPF TOPIC] Cloud storage optimizations

Luis,

Quoted 4 lines

A big future question is of course how / when to use these for filesystems. Should there be, for instance a 'mkfs --optimal-bs' or something which may look whatever hints the media uses ? Or do we just leaves the magic incantations to the admins?

mkfs already considers the reported queue limits (for the filesystems most people use, anyway).

The problem is mainly that the devices don't report them. At least not very often in the NVMe space. For SCSI devices, reporting these parameters is quite common.

--
Martin K. Petersen Oracle Linux Engineering

Quoted 32 lines · Javier González

Quoted 22 lines · James Bottomley

Quoted 10 lines · Matthew Wilcox

Quoted 8 lines · Theodore Ts'o

What HDD vendors want is to be able to have 32k or even 64k *physical* sector sizes.  This allows for much more efficient erasure codes, so it will increase their byte capacity now that it's no longer easier to get capacity boosts by squeezing the tracks closer and closer, and their have been various engineering tradeoffs with SMR, HAMR, and MAMR.  HDD vendors have been asking for this at LSF/MM, and in othervenues for ***years***.

Quoted 10 lines

I've been reminded by a friend who works on the drive side that a motivation for the SSD vendors is (essentially) the size of sector_t. Once the drive needs to support more than 2/4 billion sectors, they need to move to a 64-bit sector size, so the amount of memory consumed by the FTL doubles, the CPU data cache becomes half as effective, etc. That significantly increases the BOM for the drive, and so they have to charge more.  With a 512-byte LBA, that's 2TB; with a 4096-byte LBA, it's at 16TB and with a 64k LBA, they can keep using 32-bit LBA numbers all the way up to 256TB.

Quoted 8 lines

I thought the FTL operated on physical sectors and the logical to physical was done as a RMW through the FTL?  In which case sector_t shouldn't matter to the SSD vendors for FTL management because they can keep the logical sector size while increasing the physical one. Obviously if physical size goes above the FS block size, the drives will behave suboptimally with RMWs, which is why 4k physical is the max currently.

Quoted 5 lines

FTL designs are complex. We have ways to maintain sector sizes under 64 bits, but this is a common industry problem.

The media itself does not normally oeprate at 4K. Page siges can be 16K, 32K, etc.

Right, and we've always said if we knew what this size was we could make better block write decisions. However, today if you look what most NVMe devices are reporting, it's a bit sub-optimal:

jejb@lingrow:/sys/block/nvme1n1/queue> cat logical_block_size 512
jejb@lingrow:/sys/block/nvme1n1/queue> cat physical_block_size 512
jejb@lingrow:/sys/block/nvme1n1/queue> cat optimal_io_size 0

If we do get Linux to support large block sizes, are we actually going to get better information out of the devices?

Quoted 6 lines

Increasing the block size would allow for better host/device
cooperation. As Ted mentions, this has been a requirement for HDD and SSD vendor for years. It seems to us that the time is right now and that we have mechanisms in Linux to do the plumbing. Folios is ovbiously a big part of this.

Well a decade ago we did a lot of work to support 4k sector devices. Ultimately the industry went with 512 logical/4k physical devices because of problems with non-Linux proprietary OSs but you could still use 4k today if you wanted (I've actually still got a working 4k SCSI drive), so why is no NVMe device doing that?

This is not to say I think larger block sizes is in any way a bad idea ... I just think that given the history, it will be driven by application needs rather than what the manufacturers tell us.

One can view zones as really large LBAs.

Indeed it might be suboptimal from the OS point of view. But from the device point of view it won't.
And, in fact, with devices becoming faster and faster the question is whether sticking with relatively small sectors won't become a limiting factor eventually.

My point being that zones are just there because the I/O stack can only deal with sectors up to 4k. If the I/O stack would be capable of dealing with larger LBAs one could identify a zone with an LBA, and the entire issue of append-only and sequential writes would be moot.
Even the entire concept of zones becomes irrelevant as the OS would trivially only write entire zones.

What I was saying is that 256M is not set in stone. It's just a compromise vendors used. Even if in the course of development we arrive at a lower number of max LBA we can handle (say, 2MB) I am pretty sure vendors will be quite interested in that.

My point being that zones are just there because the I/O stack can only deal with sectors up to 4k. If the I/O stack would be capable of dealing with larger LBAs one could identify a zone with an LBA, and the entire issue of append-only and sequential writes would be moot. Even the entire concept of zones becomes irrelevant as the OS would trivially only write entire zones.

With NVMe, it is possible for a storage device to promise this without requiring read-modify-write updates for sub-16k writes. All that is necessary are some changes in the block layer so that the kernel does not inadvertently tear a write request when splitting a bio because it is too large (perhaps because it got merged with some other request, and then it gets split at an inconvenient boundary).

There are also more interesting, advanced optimizations that might be possible. For example, Jens had observed the passing hints that journaling writes (either from file systems or databases) could be potentially useful.

Unfortunately most common storage devices have not supported write hints, and support for write hints were ripped out last year. That can be easily reversed, but there are some other interesting related subjects that are very much suited for LSF/MM.

For example, most cloud storage devices are doing read-ahead to try to anticipate read requests from the VM. This can interfere with the read-ahead being done by the guest kernel. So being able to tell cloud storage device whether a particular read request is stemming from a read-ahead or not.

Quoted 5 lines

For example, most cloud storage devices are doing read-ahead to try to anticipate read requests from the VM. This can interfere with the read-ahead being done by the guest kernel. So being able to tell cloud storage device whether a particular read request is stemming from a read-ahead or not.

Indeed. In our experience the hints that work best are the ones which convey to the storage device why the I/O is being performed.

Hi Ted!

Quoted 6 lines

With NVMe, it is possible for a storage device to promise this without requiring read-modify-write updates for sub-16k writes. All that is necessary are some changes in the block layer so that the kernel does not inadvertently tear a write request when splitting a bio because it is too large (perhaps because it got merged with some other request, and then it gets split at an inconvenient boundary).

We have been working on support for atomic writes and it is not a simple as it sounds. Atomic operations in SCSI and NVMe have semantic differences which are challenging to reconcile. On top of that, both the SCSI and NVMe specs are buggy in the atomics department. We are working to get things fixed in both standards and aim to discuss our implementation at LSF/MM.

Hinting didn't see widespread adoption because we in Linux, as well as the various interested databases, preferred hints to be per-I/O properties. Whereas $OTHER_OS insisted that hints should be statically assigned to LBA ranges on media. This left vendors having to choose between two very different approaches and consequently they chose not to support any of them.