Enable use of Solid State Hybrid Drives

From: Dan Williams <[email protected]>

Steal one unused bit from the priority class and two bits from the priority data, to implement a 3 bit cache-advice field. Similar to the page cache advice from fadvise() these hints are meant to be consumed by hybrid drives. Solid State Hyrbid-Drives, as defined by the SATA-IO Specification, implement up to a 4-bit cache priority that can be specified along with a FPDMA command.

Quoted 8 lines · Jason B. Akers

From: Dan Williams <[email protected]>

Steal one unused bit from the priority class and two bits from the priority data, to implement a 3 bit cache-advice field. Similar to the page cache advice from fadvise() these hints are meant to be consumed by hybrid drives. Solid State Hyrbid-Drives, as defined by the SATA-IO Specification, implement up to a 4-bit cache priority that can be specified along with a FPDMA command.

ionice is about setting an I/O scheduling class for a *process*. So, unless I've missed something, this does not seem like the right interface for passing I/O hints.

You mention fadvise hints, which sounds like a good fit (and madvise would be equally interesting), but I don't see where you've wired them up in this patch set. Did I miss it?

The following series enables the use of Solid State hybrid drives
ATA standard 3.2 defines the hybrid information feature, which provides a means for the host driver to provide hints to the SSHDs to guide what to place on the SSD/NAND portion and what to place on the magnetic media.

This implementation allows user space applications to provide the cache hints to the kernel using the existing ionice syscall.

An application can pass a priority number coding up bits 11, 12, and 15 of the ionice command to form a 3 bit field that encodes the following priorities:
OPRIO_ADV_NONE,
IOPRIO_ADV_EVICT, /* actively discard cached data */
IOPRIO_ADV_DONTNEED, /* caching this data has little value */
IOPRIO_ADV_NORMAL, /* best-effort cache priority (default) */
IOPRIO_ADV_RESERVED1, /* reserved for future use */
IOPRIO_ADV_RESERVED2,
IOPRIO_ADV_RESERVED3,
IOPRIO_ADV_WILLNEED, /* high temporal locality */

For example the following commands from the user space will make dd IOs to be generated with a hint of IOPRIO_ADV_DONTNEED assuming the SSHD is /dev/sdc.

ionice -c2 -n4096 dd if=/dev/zero of=/dev/sdc bs=1M count=1024 ionice -c2 -n4096 dd if=/dev/sdc of=/dev/null bs=1M count=1024

Quoted 15 lines · Jason B. Akers

The following series enables the use of Solid State hybrid drives
ATA standard 3.2 defines the hybrid information feature, which provides a means for the host driver to provide hints to the SSHDs to guide what to place on the SSD/NAND portion and what to place on the magnetic media.

This implementation allows user space applications to provide the cache hints to the kernel using the existing ionice syscall.

An application can pass a priority number coding up bits 11, 12, and 15 of the ionice command to form a 3 bit field that encodes the following priorities:
OPRIO_ADV_NONE,
IOPRIO_ADV_EVICT, /* actively discard cached data */
IOPRIO_ADV_DONTNEED, /* caching this data has little value */
IOPRIO_ADV_NORMAL, /* best-effort cache priority (default) */
IOPRIO_ADV_RESERVED1, /* reserved for future use */
IOPRIO_ADV_RESERVED2,
IOPRIO_ADV_RESERVED3,
IOPRIO_ADV_WILLNEED, /* high temporal locality */

Quoted 4 lines

For example the following commands from the user space will make dd IOs to be generated with a hint of IOPRIO_ADV_DONTNEED assuming the SSHD is /dev/sdc.

ionice -c2 -n4096 dd if=/dev/zero of=/dev/sdc bs=1M count=1024 ionice -c2 -n4096 dd if=/dev/sdc of=/dev/null bs=1M count=1024

This looks to be the wrong way to implement per-IO priority information.

How does a filesystem make use of this to make sure it's metadata ends up with IOPRIO_ADV_WILLNEED to store frequently accessed metadata in flash. Conversely, journal writes need to be issued with IOPRIO_ADV_DONTNEED so they don't unneceessarily consume flash space as they are never-read IOs...

Quoted 21 lines · Dave Chinner

Quoted 15 lines · Jason B. Akers

The following series enables the use of Solid State hybrid drives
ATA standard 3.2 defines the hybrid information feature, which provides a means for the host driver to provide hints to the SSHDs to guide what to place on the SSD/NAND portion and what to place on the magnetic media.

This implementation allows user space applications to provide the cache hints to the kernel using the existing ionice syscall.

An application can pass a priority number coding up bits 11, 12, and 15 of the ionice command to form a 3 bit field that encodes the following priorities:
OPRIO_ADV_NONE,
IOPRIO_ADV_EVICT, /* actively discard cached data */
IOPRIO_ADV_DONTNEED, /* caching this data has little value */
IOPRIO_ADV_NORMAL, /* best-effort cache priority (default) */
IOPRIO_ADV_RESERVED1, /* reserved for future use */
IOPRIO_ADV_RESERVED2,
IOPRIO_ADV_RESERVED3,
IOPRIO_ADV_WILLNEED, /* high temporal locality */

Quoted 4 lines

For example the following commands from the user space will make dd IOs to be generated with a hint of IOPRIO_ADV_DONTNEED assuming the SSHD is /dev/sdc.

ionice -c2 -n4096 dd if=/dev/zero of=/dev/sdc bs=1M count=1024 ionice -c2 -n4096 dd if=/dev/sdc of=/dev/null bs=1M count=1024

Quoted 8 lines

This looks to be the wrong way to implement per-IO priority information.

How does a filesystem make use of this to make sure it's metadata ends up with IOPRIO_ADV_WILLNEED to store frequently accessed metadata in flash. Conversely, journal writes need to be issued with IOPRIO_ADV_DONTNEED so they don't unneceessarily consume flash space as they are never-read IOs...

Not disagreeing that loading more into the io priority fields is a bit... icky. I see why it's done, though, it requires the least amount of plumbing.

As for the fs accessing this, the io nice fields are readily exposed through the ->bi_rw setting. So while the above example uses ionice to set a task io priority (that a bio will then inherit), nothing prevents you from passing it in directly from the kernel.

Quoted 31 lines · Jens Axboe

Quoted 29 lines · Dave Chinner

Quoted 15 lines · Jason B. Akers

The following series enables the use of Solid State hybrid drives
ATA standard 3.2 defines the hybrid information feature, which provides a means for the host driver to provide hints to the SSHDs to guide what to place on the SSD/NAND portion and what to place on the magnetic media.

This implementation allows user space applications to provide the cache hints to the kernel using the existing ionice syscall.

An application can pass a priority number coding up bits 11, 12, and 15 of the ionice command to form a 3 bit field that encodes the following priorities:
OPRIO_ADV_NONE,
IOPRIO_ADV_EVICT, /* actively discard cached data */
IOPRIO_ADV_DONTNEED, /* caching this data has little value */
IOPRIO_ADV_NORMAL, /* best-effort cache priority (default) */
IOPRIO_ADV_RESERVED1, /* reserved for future use */
IOPRIO_ADV_RESERVED2,
IOPRIO_ADV_RESERVED3,
IOPRIO_ADV_WILLNEED, /* high temporal locality */

Quoted 4 lines

For example the following commands from the user space will make dd IOs to be generated with a hint of IOPRIO_ADV_DONTNEED assuming the SSHD is /dev/sdc.

ionice -c2 -n4096 dd if=/dev/zero of=/dev/sdc bs=1M count=1024 ionice -c2 -n4096 dd if=/dev/sdc of=/dev/null bs=1M count=1024

This looks to be the wrong way to implement per-IO priority information.

How does a filesystem make use of this to make sure it's metadata ends up with IOPRIO_ADV_WILLNEED to store frequently accessed metadata in flash. Conversely, journal writes need to be issued with IOPRIO_ADV_DONTNEED so they don't unneceessarily consume flash space as they are never-read IOs...

Quoted 3 lines

Not disagreeing that loading more into the io priority fields is a bit... icky. I see why it's done, though, it requires the least amount of plumbing.

Yeah, but we don't do things the easy way just because it's easy. We do things the right way. ;)

Quoted 4 lines

As for the fs accessing this, the io nice fields are readily exposed through the ->bi_rw setting. So while the above example uses ionice to set a task io priority (that a bio will then inherit), nothing prevents you from passing it in directly from the kernel.

Right, but now the filesystem needs to provide that on a per-inode basis, not from the task structure as the task that is submitting the bio is not necesarily the task doing the read/write syscall.

e.g. the write case above doesn't actually inherit the task priority at the bio level at all because the IO is being dispatched by a background flusher thread, not the ioniced task calling write(2).

IMO using ionice is a nice hack, but utimately it looks mostly useless from a user and application perspective as cache residency is a property of the data being read/written, not the task doing the IO. e.g. a database will want it's indexes in flash and bulk data in non-cached storage.

IOWs, to make effective use of this the task will need different cache hints for each different type of data needs to do IO on, and so overloading IO priorities just seems the wrong direction to be starting from.

Quoted 10 lines · Dave Chinner

Quoted 4 lines · Jens Axboe

As for the fs accessing this, the io nice fields are readily exposed through the ->bi_rw setting. So while the above example uses ionice to set a task io priority (that a bio will then inherit), nothing prevents you from passing it in directly from the kernel.

Right, but now the filesystem needs to provide that on a per-inode basis, not from the task structure as the task that is submitting the bio is not necesarily the task doing the read/write syscall.

Quoted 3 lines

e.g. the write case above doesn't actually inherit the task priority at the bio level at all because the IO is being dispatched by a background flusher thread, not the ioniced task calling write(2).

When the ioniced task calling write(2) inserts the page into the page cache then the current priority is recorded in the struct page. The

background flusher likely runs at a lower / neutral caching priority and the priority carried in the page will be the effective caching priority applied to the bio.

Quoted 4 lines

from a user and application perspective as cache residency is a property of the data being read/written, not the task doing the IO. e.g. a database will want it's indexes in flash and bulk data in non-cached storage.

Right, if those are doing direct-i/o then have a separate thread-id for those write(2) calls.

Otherwise if they are dirtying page cache
the struct page carries the hint.

Quoted 4 lines

IOWs, to make effective use of this the task will need different cache hints for each different type of data needs to do IO on, and so overloading IO priorities just seems the wrong direction to be starting from.

There's also the fadvise() enabling that could be bolted on top of this capability. But, before that step, is a thread-id per-caching context too much to ask?

Quoted 15 lines · Dan Williams

Quoted 13 lines · Dave Chinner

Quoted 4 lines · Jens Axboe

As for the fs accessing this, the io nice fields are readily exposed through the ->bi_rw setting. So while the above example uses ionice to set a task io priority (that a bio will then inherit), nothing prevents you from passing it in directly from the kernel.

Right, but now the filesystem needs to provide that on a per-inode basis, not from the task structure as the task that is submitting the bio is not necesarily the task doing the read/write syscall.

e.g. the write case above doesn't actually inherit the task priority at the bio level at all because the IO is being dispatched by a background flusher thread, not the ioniced task calling write(2).

Quoted 2 lines

When the ioniced task calling write(2) inserts the page into the page cache then the current priority is recorded in the struct page. The

It does? Can you point me to where the page cache code does this, because I've clearly missed something important go by in the past few months...

Quoted 15 lines · Dan Williams

Quoted 13 lines · Dave Chinner

Quoted 4 lines · Jens Axboe

As for the fs accessing this, the io nice fields are readily exposed through the ->bi_rw setting. So while the above example uses ionice to set a task io priority (that a bio will then inherit), nothing prevents you from passing it in directly from the kernel.

Right, but now the filesystem needs to provide that on a per-inode basis, not from the task structure as the task that is submitting the bio is not necesarily the task doing the read/write syscall.

e.g. the write case above doesn't actually inherit the task priority at the bio level at all because the IO is being dispatched by a background flusher thread, not the ioniced task calling write(2).

Quoted 2 lines

When the ioniced task calling write(2) inserts the page into the page cache then the current priority is recorded in the struct page. The

It does? Can you point me to where the page cache code does this, because I've clearly missed something important go by in the past few months...

Quoted 3 lines

background flusher likely runs at a lower / neutral caching priority and the priority carried in the page will be the effective caching priority applied to the bio.

How? The writepage code that adds the pages to the bio doesn't look at priorities at all. If we're supposed to be doing this, then it isn't being done in XFS when we are building bios, and nobody has told me we need to do it...

Hmmm - ok, so what happens if an IO is made up of pages from different tasks with different priorities? what then? ;)

Quoted 5 lines

Quoted 4 lines

from a user and application perspective as cache residency is a property of the data being read/written, not the task doing the IO. e.g. a database will want it's indexes in flash and bulk data in non-cached storage.

Quoted 2 lines

Right, if those are doing direct-i/o then have a separate thread-id for those write(2) calls.

Which, again, is not how applications are designed or implemented. If the current transaction needs to read/write index blocks, it does it directly, not have to wait for some other dispatch thread to do it for it....

Quoted 8 lines

Otherwise if they are dirtying page cache
the struct page carries the hint.

Quoted 4 lines

IOWs, to make effective use of this the task will need different cache hints for each different type of data needs to do IO on, and so overloading IO priorities just seems the wrong direction to be starting from.

Quoted 3 lines

There's also the fadvise() enabling that could be bolted on top of this capability. But, before that step, is a thread-id per-caching context too much to ask?

If we do it that way, we are stuck with it forever. So let's get our ducks in line first before pulling the trigger...

Quoted 16 lines · Dave Chinner

Quoted 15 lines · Dan Williams

Quoted 13 lines · Dave Chinner

Quoted 4 lines · Jens Axboe

As for the fs accessing this, the io nice fields are readily exposed through the ->bi_rw setting. So while the above example uses ionice to set a task io priority (that a bio will then inherit), nothing prevents you from passing it in directly from the kernel.

Right, but now the filesystem needs to provide that on a per-inode basis, not from the task structure as the task that is submitting the bio is not necesarily the task doing the read/write syscall.

e.g. the write case above doesn't actually inherit the task priority at the bio level at all because the IO is being dispatched by a background flusher thread, not the ioniced task calling write(2).

Quoted 6 lines

Quoted 2 lines

When the ioniced task calling write(2) inserts the page into the page cache then the current priority is recorded in the struct page. The

It does? Can you point me to where the page cache code does this, because I've clearly missed something important go by in the past few months...

Sorry, should have been more clear that this patch set added that capability in patch-4. The idea is to claim some unused extended page flags to stash priority bits. Yes, the PageSetAdvice() helper needs to be fixed up to do the flags update atomically, and yes this precludes hinting on 32-bit platforms. I also think that bio_add_page() is the better place to read the per-page priority into the bio. We felt ok deferring these items until after the initial RFC.

Quoted 9 lines

Quoted 8 lines

Quoted 4 lines

IOWs, to make effective use of this the task will need different cache hints for each different type of data needs to do IO on, and so overloading IO priorities just seems the wrong direction to be starting from.

There's also the fadvise() enabling that could be bolted on top of this capability. But, before that step, is a thread-id per-caching context too much to ask?

Quoted 2 lines

If we do it that way, we are stuck with it forever. So let's get our ducks in line first before pulling the trigger...

Are you objecting to ionice as the interface or per-pid based hinting in general?

[Been distrcted with other issues, so just getting back to this.]

Quoted 24 lines · Dan Williams

Quoted 16 lines · Dave Chinner

Quoted 15 lines · Dan Williams

Quoted 13 lines · Dave Chinner

Quoted 4 lines · Jens Axboe

As for the fs accessing this, the io nice fields are readily exposed through the ->bi_rw setting. So while the above example uses ionice to set a task io priority (that a bio will then inherit), nothing prevents you from passing it in directly from the kernel.

Right, but now the filesystem needs to provide that on a per-inode basis, not from the task structure as the task that is submitting the bio is not necesarily the task doing the read/write syscall.

e.g. the write case above doesn't actually inherit the task priority at the bio level at all because the IO is being dispatched by a background flusher thread, not the ioniced task calling write(2).

Quoted 6 lines

Quoted 2 lines

When the ioniced task calling write(2) inserts the page into the page cache then the current priority is recorded in the struct page. The

It does? Can you point me to where the page cache code does this, because I've clearly missed something important go by in the past few months...

Quoted 8 lines

Sorry, should have been more clear that this patch set added that capability in patch-4. The idea is to claim some unused extended page flags to stash priority bits. Yes, the PageSetAdvice() helper needs to be fixed up to do the flags update atomically, and yes this precludes hinting on 32-bit platforms. I also think that bio_add_page() is the better place to read the per-page priority into the bio. We felt ok deferring these items until after the initial RFC.

I think that using page flags for this is a 'orrible idea. Yeah, it's a neat hack that you can use for proff of concept demonstrations, but my biggest concern is that it isn't a scalable channel for carrying IO priority information through the page cache. e.g. it can't carry existing ionice priority scheduling information, it can't carry blkcg IO control information, etc.

So, really, I think that this buffered write IO priority issue is bigger than this patch series, and we need to solve it properly rather than hack ugly special cases into core infrastructure that are an evolutionary dead-end....

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

Quoted 4 lines

IOWs, to make effective use of this the task will need different cache hints for each different type of data needs to do IO on, and so overloading IO priorities just seems the wrong direction to be starting from.

There's also the fadvise() enabling that could be bolted on top of this capability. But, before that step, is a thread-id per-caching context too much to ask?

Quoted 5 lines

Quoted 2 lines

If we do it that way, we are stuck with it forever. So let's get our ducks in line first before pulling the trigger...

Are you objecting to ionice as the interface or per-pid based hinting in general?

Neither. It's the implementation I don't like.

Quoted 31 lines · Jens Axboe

Quoted 29 lines · Dave Chinner

Quoted 15 lines · Jason B. Akers

The following series enables the use of Solid State hybrid drives
ATA standard 3.2 defines the hybrid information feature, which provides a means for the host driver to provide hints to the SSHDs to guide what to place on the SSD/NAND portion and what to place on the magnetic media.

This implementation allows user space applications to provide the cache hints to the kernel using the existing ionice syscall.

An application can pass a priority number coding up bits 11, 12, and 15 of the ionice command to form a 3 bit field that encodes the following priorities:
OPRIO_ADV_NONE,
IOPRIO_ADV_EVICT, /* actively discard cached data */
IOPRIO_ADV_DONTNEED, /* caching this data has little value */
IOPRIO_ADV_NORMAL, /* best-effort cache priority (default) */
IOPRIO_ADV_RESERVED1, /* reserved for future use */
IOPRIO_ADV_RESERVED2,
IOPRIO_ADV_RESERVED3,
IOPRIO_ADV_WILLNEED, /* high temporal locality */

Quoted 4 lines

For example the following commands from the user space will make dd IOs to be generated with a hint of IOPRIO_ADV_DONTNEED assuming the SSHD is /dev/sdc.

ionice -c2 -n4096 dd if=/dev/zero of=/dev/sdc bs=1M count=1024 ionice -c2 -n4096 dd if=/dev/sdc of=/dev/null bs=1M count=1024

This looks to be the wrong way to implement per-IO priority information.

How does a filesystem make use of this to make sure it's metadata ends up with IOPRIO_ADV_WILLNEED to store frequently accessed metadata in flash. Conversely, journal writes need to be issued with IOPRIO_ADV_DONTNEED so they don't unneceessarily consume flash space as they are never-read IOs...

Quoted 3 lines

Not disagreeing that loading more into the io priority fields is a bit... icky. I see why it's done, though, it requires the least amount of plumbing.

Yeah, but we don't do things the easy way just because it's easy. We do things the right way. ;)

Quoted 4 lines

As for the fs accessing this, the io nice fields are readily exposed through the ->bi_rw setting. So while the above example uses ionice to set a task io priority (that a bio will then inherit), nothing prevents you from passing it in directly from the kernel.

Right, but now the filesystem needs to provide that on a per-inode basis, not from the task structure as the task that is submitting the bio is not necesarily the task doing the read/write syscall.

e.g. the write case above doesn't actually inherit the task priority at the bio level at all because the IO is being dispatched by a background flusher thread, not the ioniced task calling write(2).

IMO using ionice is a nice hack, but utimately it looks mostly useless from a user and application perspective as cache residency is a property of the data being read/written, not the task doing the IO. e.g. a database will want it's indexes in flash and bulk data in non-cached storage.

IOWs, to make effective use of this the task will need different cache hints for each different type of data needs to do IO on, and so overloading IO priorities just seems the wrong direction to be starting from.

Quoted 15 lines · Jason B. Akers

The following series enables the use of Solid State hybrid drives
ATA standard 3.2 defines the hybrid information feature, which provides a means for the host driver to provide hints to the SSHDs to guide what to place on the SSD/NAND portion and what to place on the magnetic media.

This implementation allows user space applications to provide the cache hints to the kernel using the existing ionice syscall.

An application can pass a priority number coding up bits 11, 12, and 15 of the ionice command to form a 3 bit field that encodes the following priorities:
OPRIO_ADV_NONE,
IOPRIO_ADV_EVICT, /* actively discard cached data */
IOPRIO_ADV_DONTNEED, /* caching this data has little value */
IOPRIO_ADV_NORMAL, /* best-effort cache priority (default) */
IOPRIO_ADV_RESERVED1, /* reserved for future use */
IOPRIO_ADV_RESERVED2,
IOPRIO_ADV_RESERVED3,
IOPRIO_ADV_WILLNEED, /* high temporal locality */

Quoted 4 lines

For example the following commands from the user space will make dd IOs to be generated with a hint of IOPRIO_ADV_DONTNEED assuming the SSHD is /dev/sdc.

ionice -c2 -n4096 dd if=/dev/zero of=/dev/sdc bs=1M count=1024 ionice -c2 -n4096 dd if=/dev/sdc of=/dev/null bs=1M count=1024

This looks to be the wrong way to implement per-IO priority information.

How does a filesystem make use of this to make sure it's metadata ends up with IOPRIO_ADV_WILLNEED to store frequently accessed metadata in flash. Conversely, journal writes need to be issued with IOPRIO_ADV_DONTNEED so they don't unneceessarily consume flash space as they are never-read IOs...

Quoted 15 lines · Jason B. Akers

The following series enables the use of Solid State hybrid drives
ATA standard 3.2 defines the hybrid information feature, which provides a means for the host driver to provide hints to the SSHDs to guide what to place on the SSD/NAND portion and what to place on the magnetic media.

This implementation allows user space applications to provide the cache hints to the kernel using the existing ionice syscall.

An application can pass a priority number coding up bits 11, 12, and 15 of the ionice command to form a 3 bit field that encodes the following priorities:
OPRIO_ADV_NONE,
IOPRIO_ADV_EVICT, /* actively discard cached data */
IOPRIO_ADV_DONTNEED, /* caching this data has little value */
IOPRIO_ADV_NORMAL, /* best-effort cache priority (default) */
IOPRIO_ADV_RESERVED1, /* reserved for future use */
IOPRIO_ADV_RESERVED2,
IOPRIO_ADV_RESERVED3,
IOPRIO_ADV_WILLNEED, /* high temporal locality */

Quoted 4 lines

For example the following commands from the user space will make dd IOs to be generated with a hint of IOPRIO_ADV_DONTNEED assuming the SSHD is /dev/sdc.

ionice -c2 -n4096 dd if=/dev/zero of=/dev/sdc bs=1M count=1024 ionice -c2 -n4096 dd if=/dev/sdc of=/dev/null bs=1M count=1024

This looks to be the wrong way to implement per-IO priority information.

How does a filesystem make use of this to make sure it's metadata ends up with IOPRIO_ADV_WILLNEED to store frequently accessed metadata in flash. Conversely, journal writes need to be issued with IOPRIO_ADV_DONTNEED so they don't unneceessarily consume flash space as they are never-read IOs...

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Jason" == Jason B Akers <[email protected]> writes:

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Jason" == Jason B Akers <[email protected]> writes:

Jason> The following series enables the use of Solid State hybrid drives Jason> ATA standard 3.2 defines the hybrid information feature, which Jason> provides a means for the host driver to provide hints to the Jason> SSHDs to guide what to place on the SSD/NAND portion and what to Jason> place on the magnetic media.

I have been ripping my hair out in this department for a while.

A colleague and I presented our findings at SNIA SDC a few weeks ago. I'm trying to find out if there's an embargo on the slides or if I can post them.

First of all I completely agree with Dave's comments about hooking into fadvise()/madvise().

My challenge with hints has been trying to bridge all the various existing approaches with the new stuff that's coming down the pipe in T10/T13 (LBMD hints) and NFS v4.2 ditto. That turned into a huge mapping table as well as a few amendments to what's currently being worked on in the standards bodies.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Jens" == Jens Axboe <[email protected]> writes:

The next step was trying to map these hints into what was available in xadvise(), NFS 4.2 and the recent T10/T13 efforts. That wasn't trivial and there really isn't a 1:1 mapping that works. So I went to T10 and tried to nudge things in the same direction as NFS 4.2. Mainly because that's closer to what we already have in xadvise().

Jens> I think we've needed a proper API for passing in appropriate hints Jens> on a per-io basis for a LONG time.

Yup.

Jens> That is the big challenge. We've tried (and failed) in the past to Jens> define a set of hints that make sense. It'd be a shame to add Jens> something that's specific to a given transport/technology.

Absolutely!

I understand the desire to have per-io / per-inode xadvise()-style hints, but I don't see why not also include a per-pid capability?

Per-pid was not "icky" for flashcache [1]. It let's you flag processes that should not pollute the cache, as well "cache warming" processes pre-loading sub-ranges of files that is awkward to do with a per-inode hint. Per-pid also allows hinting on behalf of other otherwise cache-unaware processes.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Jens" == Jens Axboe <[email protected]> writes:

Jens> The problem with xadvise() is that it handles only one part of Jens> this - it handles the case of tying some sort of IO related Jens> priority information to an inode. It does not handle the case of Jens> different parts of the file, at least not without adding specific Jens> extra tracking for this on the kernel side.

Are there actually people asking for sub-file granularity? I didn't get any requests for that in the survey I did this summer.

I talked to several application people about what they really needed and wanted. That turned into a huge twisted mess of a table with ponies of various sizes.

I condensed all those needs and desires into something like this:

+-----------------+------------+----------+------------+
| I/O Class       | Command    | Desired  | Predicted  |
|                 | Completion | Future   | Future     |
|                 | Urgency    | Access   | Access     |
|                 |            | Latency  | Frequency  |
+-----------------+------------+----------+------------+
| Transaction     | High       | Low      | High       |
+-----------------+------------+----------+------------+
| Metadata        | High       | Low      | Normal     |
+-----------------+------------+----------+------------+
| Paging          | High       | Normal   | Normal     |
+-----------------+------------+----------+------------+
| Streaming       | High       | Normal   | Low        |
+-----------------+------------+----------+------------+
| Data            | Normal     | Normal   | Normal     |
+-----------------+------------+----------+------------+
| Background      | Low        | Normal*  | Low        |
+-----------------+------------+----------+------------+

Command completion urgency is really just the existing I/O priority. Desired future access latency affects data placement in a tiered device. Predicted future access frequency is essentially a caching hint.

The names and I/O classes themselves are not really important. It's just a reduced version of all the things people asked for. Essentially:
Relative priority, data placement and caching.

I had also asked why people wanted to specify any hints. And that boiled down to the I/O classes in the left column above. People wanted stuff on a low latency storage tier because it was a transactional or metadata type of I/O. Or to isolate production I/O from any side effects of a background scrub or backup run.

Incidentally, the classes data, transaction and background covered almost all the use cases that people had asked for. The metadata class mostly came about from good results with REQ_META tagging in a previous prototype. A few vendors wanted to be able to identify swap to prevent platter spin-ups. Streaming was requested by a couple of video folks.

The notion of telling the storage *why* you're doing I/O instead of telling it how to manage its cache and where to put stuff is closely aligned with our internal experiences with I/O hints over the last decade. But it's a bit of a departure from where things are going in the standards bodies. In any case I thought it was interesting that pretty much every use case that people came up with could be adequately described by a handful of I/O classes.

Quoted 7 lines · Martin K. Petersen

The notion of telling the storage *why* you're doing I/O instead of telling it how to manage its cache and where to put stuff is closely aligned with our internal experiences with I/O hints over the last decade. But it's a bit of a departure from where things are going in the standards bodies. In any case I thought it was interesting that pretty much every use case that people came up with could be adequately described by a handful of I/O classes.

Definitely agree on this, it's about notifying storage on what type of IO this is, or why we are doing it. I'm just still worried that this will then end up being unusable by applications, since they can't rely on anything. Say one vendor treats WONTNEED in a much colder fashion than others, the user/application will then complain about the access latencies for the next IO to that location. "Yes it's cold, but I didn't expect it to be THAT cold" and then come to the conclusion that they can't feasibly use these hints as they don't do exactly what they want.

It'd be nice if we could augment this with a query interface of some sort, that could give the application some idea of what happens for each of the passed in hints. That would improve the situation from a "lets set this hint and hope it does what we think it does" to a more predictable and robust environment.