block: Fix IO priority mess
cover block: Fix IO priority mess
Hello,
recently I've been looking into 10% regression reported by our performance measurement infrastructure in reaim benchmark that was bisected down to 5a9d041ba2f6 ("block: move io_context creation into where it's needed"). This didn't really make much sense and it took me a while to understand this but the culprit is actually in even older commit e70344c05995 ("block: fix default IO priority handling") and 5a9d041ba2f6 just made the breakage visible. Essentially the problem was that after these commits some IO was queued with IO priority class IOPRIO_CLASS_BE while other IO was queued with IOPRIO_CLASS_NONE and as a result they could not be merged together resulting in performance regression. I think what commit e70344c05995 ("block: fix default IO priority handling") did is actually broken not only because of this performance regression but because of other reasons as well (see changelog of patch 3/3 for details) so this patch set aims at fixing it.
Honza
PATCH 1/3 block: Fix handling of tasks without ioprio in ioprio_get(2)
ioprio_get(2) can be asked to return the best IO priority from several tasks (IOPRIO_WHO_PGRP, IOPRIO_WHO_USER). Currently the call treats tasks without set IO priority as having priority IOPRIO_CLASS_BE/IOPRIO_BE_NORM however this does not really reflect the IO priority the task will get (which depends on task's nice value) and with the following fix it will not even match returned IO priority for a single task. So fix IO priority comparison to treat unset IO priority as the lowest possible one. This way we will return IOPRIO_CLASS_NONE priority only if none of the considered tasks has explicitely set IO priority, otherwise we return the highest set IO priority. This changes userspace visible behavior but hopefully the results are clearer and nothing breaks.
--- block/ioprio.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
block/ioprio.c +2 -3
@@ -157,10 +157,9 @@ static int get_task_ioprio(struct task_struct *p)int ioprio_best (unsigned short aprio,unsigned short bprio) {if (!ioprio_valid (aprio)) - aprio =IOPRIO_DEFAULT ; +return bprio;if (!ioprio_valid (bprio)) - bprio =IOPRIO_DEFAULT ; - +return aprio;return min (aprio, bprio); }
Signature · 1 line
2.35.3
Re: [PATCH 1/3] block: Fix handling of tasks without ioprio in ioprio_get(2)
Quoted 23 lines · Jan Kara
ioprio_get(2)can be asked to return the best IO priority from several tasks (IOPRIO_WHO_PGRP,IOPRIO_WHO_USER). Currently the call treats tasks without set IO priority as having priority IOPRIO_CLASS_BE/IOPRIO_BE_NORM however this does not really reflect the IO priority the task will get (which depends on task's nice value) and with the following fix it will not even match returned IO priority for a single task. So fix IO priority comparison to treat unset IO priority as the lowest possible one. This way we will returnIOPRIO_CLASS_NONEpriority only if none of the considered tasks has explicitely set IO priority, otherwise we return the highest set IO priority. This changes userspace visible behavior but hopefully the results are clearer and nothing breaks.Signed-off-by Jan Kara <[email protected]>--- block/ioprio.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)block/ioprio.c +0
-0
block/ioprio.c static int get_task_ioprio(struct task_struct *p) · line 160
Quoted 5 lines
block/ioprio.c +1
-1@@ -157,10 +157,9 @@ static int get_task_ioprio(struct task_struct *p)int ioprio_best (unsigned short aprio,unsigned short bprio) {if (!ioprio_valid (aprio)) - aprio =IOPRIO_DEFAULT ; +return bprio;
bprio may not be valid...
Quoted 7 lines
block/ioprio.c +1
-2if (!ioprio_valid (bprio)) - bprio =IOPRIO_DEFAULT ; - +return aprio;return min (aprio, bprio); }
Signature · 2 lines
Damien Le Moal Western Digital Research
Re: [PATCH 1/3] block: Fix handling of tasks without ioprio in ioprio_get(2)
Quoted 19 lines · Damien Le Moal
Quoted 18 lines · Jan Kara
ioprio_get(2)can be asked to return the best IO priority from several tasks (IOPRIO_WHO_PGRP,IOPRIO_WHO_USER). Currently the call treats tasks without set IO priority as having priority IOPRIO_CLASS_BE/IOPRIO_BE_NORM however this does not really reflect the IO priority the task will get (which depends on task's nice value) and with the following fix it will not even match returned IO priority for a single task. So fix IO priority comparison to treat unset IO priority as the lowest possible one. This way we will returnIOPRIO_CLASS_NONEpriority only if none of the considered tasks has explicitely set IO priority, otherwise we return the highest set IO priority. This changes userspace visible behavior but hopefully the results are clearer and nothing breaks.Signed-off-by Jan Kara <[email protected]>--- block/ioprio.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
Quoted 12 lines
block/ioprio.c static int get_task_ioprio(struct task_struct *p) · line 160
Quoted 10 lines
block/ioprio.c +1
-1@@ -157,10 +157,9 @@ static int get_task_ioprio(struct task_struct *p)int ioprio_best (unsigned short aprio,unsigned short bprio) {if (!ioprio_valid (aprio)) - aprio =IOPRIO_DEFAULT ; +return bprio;bprio may not be valid...
Yes, bprio may be from IOPRIO_CLASS_NONE as well and IMHO that is a desirable return in that case. ioprio_valid() is IMHO a bit of a misnomer because IOPRIO_CLASS_NONE is a valid class and can be even set by userspace. It actually means, set IO priority based on task's CPU priority. But lets first settle on the desired meaning of ioprio in the discussion over patch 3/3. How we should behave in this case is a detail we can sort out after the general meaning is clear.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 1/3] block: Fix handling of tasks without ioprio in ioprio_get(2)
Quoted 33 lines · Jan Kara
Quoted 19 lines · Damien Le Moal
Quoted 18 lines · Jan Kara
ioprio_get(2)can be asked to return the best IO priority from several tasks (IOPRIO_WHO_PGRP,IOPRIO_WHO_USER). Currently the call treats tasks without set IO priority as having priority IOPRIO_CLASS_BE/IOPRIO_BE_NORM however this does not really reflect the IO priority the task will get (which depends on task's nice value) and with the following fix it will not even match returned IO priority for a single task. So fix IO priority comparison to treat unset IO priority as the lowest possible one. This way we will returnIOPRIO_CLASS_NONEpriority only if none of the considered tasks has explicitely set IO priority, otherwise we return the highest set IO priority. This changes userspace visible behavior but hopefully the results are clearer and nothing breaks.Signed-off-by Jan Kara <[email protected]>--- block/ioprio.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)Quoted 12 lines
block/ioprio.c static int get_task_ioprio(struct task_struct *p) · line 160
Quoted 10 lines
block/ioprio.c +1
-1@@ -157,10 +157,9 @@ static int get_task_ioprio(struct task_struct *p)int ioprio_best (unsigned short aprio,unsigned short bprio) {if (!ioprio_valid (aprio)) - aprio =IOPRIO_DEFAULT ; +return bprio;bprio may not be valid...
Quoted 7 lines
Yes, bprio may be from
IOPRIO_CLASS_NONEas well and IMHO that is a desirable return in that case.ioprio_valid()is IMHO a bit of a misnomer becauseIOPRIO_CLASS_NONEis a valid class and can be even set by userspace. It actually means, set IO priority based on task's CPU priority. But lets first settle on the desired meaning of ioprio in the discussion over patch 3/3. How we should behave in this case is a detail we can sort out after the general meaning is clear.
Sounds all good to me.
Quoted 2 lines
Honza
Signature · 2 lines
Damien Le Moal Western Digital Research
PATCH 2/3 block: Make ioprio_best() static
In reply to Jan KaraNobody outside of block/ioprio.c uses it.
--- block/ioprio.c | 2 +- include/linux/ioprio.h | 5 ----- 2 files changed, 1 insertion(+), 6 deletions(-)
block/ioprio.c +1 -1
@@ -154,7 +154,7 @@ static int get_task_ioprio(struct task_struct *p)return ret; } -int ioprio_best (unsigned short aprio,unsigned short bprio) +static int ioprio_best (unsigned short aprio,unsigned short bprio) {if (!ioprio_valid (aprio))return bprio;
include/linux/ioprio.h +0 -5
@@ -59,11 +59,6 @@ static inline int get_current_ioprio(void)return IOPRIO_DEFAULT ; } -/* -* For inheritance, return the highest of the two given priorities -*/ -extern int ioprio_best (unsigned short aprio,unsigned short bprio); -extern int set_task_ioprio (struct task_struct *task,int ioprio);#ifdef CONFIG_BLOCK
Signature · 1 line
2.35.3
PATCH 3/3 block: fix default IO priority handling again
In reply to Jan KaraCommit e70344c05995 ("block: fix default IO priority handling") introduced an inconsistency in get_current_ioprio() that tasks without IO context return IOPRIO_DEFAULT priority while tasks with freshly allocated IO context will return 0 (IOPRIO_CLASS_NONE/0) IO priority. Tasks without IO context used to be rare before 5a9d041ba2f6 ("block:
move io_context creation into where it's needed") but after this commit they became common because now only BFQ IO scheduler setups task's IO context. Similar inconsistency is there for get_task_ioprio() so this inconsistency is now exposed to userspace and userspace will see different IO priority for tasks operating on devices with BFQ compared to devices without BFQ. Furthemore the changes done by commit e70344c05995 change the behavior when no IO priority is set for BFQ IO scheduler which is also documented in ioprio_set(2) manpage - namely that tasks without set IO priority will use IO priority based on their nice value.
So make sure we default to IOPRIO_CLASS_NONE as used to be the case before commit e70344c05995. Also cleanup alloc_io_context() to explicitely set this IO priority for the allocated IO context.
--- block/blk-ioc.c | 2 ++ include/linux/ioprio.h | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)
block/blk-ioc.c +2 -0
@@ -247,6 +247,8 @@ static struct io_context *alloc_io_context(gfp_t gfp_flags, int node)INIT_HLIST_HEAD (&ioc->icq_list);INIT_WORK (&ioc->release_work, ioc_release_fn);#endif + ioc->ioprio =IOPRIO_DEFAULT ; +return ioc; }
include/linux/ioprio.h +1 -1
@@ -11,7 +11,7 @@/* * Default IO priority. */ -#define IOPRIO_DEFAULT IOPRIO_PRIO_VALUE(IOPRIO_CLASS_BE, IOPRIO_BE_NORM) +#define IOPRIO_DEFAULT IOPRIO_PRIO_VALUE(IOPRIO_CLASS_NONE, 0)/* * Check that a priority value has a valid class.
Signature · 1 line
2.35.3
Re: [PATCH 3/3] block: fix default IO priority handling again
Quoted 45 lines · Jan Kara
Commit
e70344c05995("block: fix default IO priority handling") introduced an inconsistency inget_current_ioprio()that tasks without IO context returnIOPRIO_DEFAULTpriority while tasks with freshly allocated IO context will return 0 (IOPRIO_CLASS_NONE/0) IO priority. Tasks without IO context used to be rare before5a9d041ba2f6("block:
moveio_contextcreation into where it's needed") but after this commit they became common because now only BFQ IO scheduler setups task's IO context. Similar inconsistency is there forget_task_ioprio()so this inconsistency is now exposed to userspace and userspace will see different IO priority for tasks operating on devices with BFQ compared to devices without BFQ. Furthemore the changes done by commite70344c05995change the behavior when no IO priority is set for BFQ IO scheduler which is also documented inioprio_set(2)manpage - namely that tasks without set IO priority will use IO priority based on their nice value.So make sure we default to
IOPRIO_CLASS_NONEas used to be the case before commite70344c05995. Also cleanupalloc_io_context()to explicitely set this IO priority for the allocated IO context.Fixes e70344c05995 ("block: fix default IO priority handling")Signed-off-by Jan Kara <[email protected]>--- block/blk-ioc.c | 2 ++ include/linux/ioprio.h | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)block/blk-ioc.c +2
-0@@ -247,6 +247,8 @@ static struct io_context *alloc_io_context(gfp_t gfp_flags, int node)INIT_HLIST_HEAD (&ioc->icq_list);INIT_WORK (&ioc->release_work, ioc_release_fn);#endif + ioc->ioprio =IOPRIO_DEFAULT ; +return ioc; }include/linux/ioprio.h +0
-0
include/linux/ioprio.h line 14
Quoted 5 lines
include/linux/ioprio.h +1
-1@@ -11,7 +11,7 @@/* * Default IO priority. */ -#define IOPRIO_DEFAULT IOPRIO_PRIO_VALUE(IOPRIO_CLASS_BE, IOPRIO_BE_NORM) +#define IOPRIO_DEFAULT IOPRIO_PRIO_VALUE(IOPRIO_CLASS_NONE, 0)
"man ioprio_set" says:
IOPRIO_CLASS_BE (2)
This is the best-effort scheduling class, which is the default for any process that hasn't set a specific I/O priority.
Which is why patch e70344c05995 introduced IOPRIO_DEFAULT definition using the BE class, to have the kernel in sync with the manual.
The different ioprio leading to no BIO merging is definitely a problem but this patch is not really fixing anything in my opinion. It simply gets back to the previous "all 0s" ioprio initialization, which do not show the places where we actually have missing ioprio initialization to IOPRIO_DEFAULT.
Isn't it simply that IOPRIO_DEFAULT should be set as the default for any bio being allocated (in bio_alloc ?) before it is setup and inherits the user io priority ? Otherwise, the bio io prio is indeed IOPRIO_CLASS_NONE/0 and changing IOPRIO_DEFAULT to that value removes the differences you observed.
Quoted 3 lines
include/linux/ioprio.h +0
-0/* * Check that a priority value has a valid class.
Signature · 2 lines
Damien Le Moal Western Digital Research
Re: [PATCH 3/3] block: fix default IO priority handling again
Quoted 54 lines · Damien Le Moal
Quoted 45 lines · Jan Kara
Commit
e70344c05995("block: fix default IO priority handling") introduced an inconsistency inget_current_ioprio()that tasks without IO context returnIOPRIO_DEFAULTpriority while tasks with freshly allocated IO context will return 0 (IOPRIO_CLASS_NONE/0) IO priority. Tasks without IO context used to be rare before5a9d041ba2f6("block:
moveio_contextcreation into where it's needed") but after this commit they became common because now only BFQ IO scheduler setups task's IO context. Similar inconsistency is there forget_task_ioprio()so this inconsistency is now exposed to userspace and userspace will see different IO priority for tasks operating on devices with BFQ compared to devices without BFQ. Furthemore the changes done by commite70344c05995change the behavior when no IO priority is set for BFQ IO scheduler which is also documented inioprio_set(2)manpage - namely that tasks without set IO priority will use IO priority based on their nice value.So make sure we default to
IOPRIO_CLASS_NONEas used to be the case before commite70344c05995. Also cleanupalloc_io_context()to explicitely set this IO priority for the allocated IO context.Fixes e70344c05995 ("block: fix default IO priority handling")Signed-off-by Jan Kara <[email protected]>--- block/blk-ioc.c | 2 ++ include/linux/ioprio.h | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)block/blk-ioc.c +2
-0@@ -247,6 +247,8 @@ static struct io_context *alloc_io_context(gfp_t gfp_flags, int node)INIT_HLIST_HEAD (&ioc->icq_list);INIT_WORK (&ioc->release_work, ioc_release_fn);#endif + ioc->ioprio =IOPRIO_DEFAULT ; +return ioc; }include/linux/ioprio.h +0
-0include/linux/ioprio.h line 14
Quoted 5 lines
include/linux/ioprio.h +1
-1@@ -11,7 +11,7 @@/* * Default IO priority. */ -#define IOPRIO_DEFAULT IOPRIO_PRIO_VALUE(IOPRIO_CLASS_BE, IOPRIO_BE_NORM) +#define IOPRIO_DEFAULT IOPRIO_PRIO_VALUE(IOPRIO_CLASS_NONE, 0)"man
ioprio_set" says:
Quoted 6 lines
IOPRIO_CLASS_BE(2)
This is the best-effort scheduling class, which is the default for any process that hasn't set a specific I/O priority.Which is why patch
e70344c05995introducedIOPRIO_DEFAULTdefinition using the BE class, to have the kernel in sync with the manual.
Yes, but it also has:
If no I/O scheduler has been set for a thread, then by default the I/O priority will follow the CPU nice value (setpriority(2)). In Linux kernels before version 2.6.24, once an I/O priority had been set using ioprio_set(), there was no way to reset the I/O scheduling behavior to the default. Since Linux 2.6.24, specifying ioprio as 0 can be used to reset to the default I/O scheduling behavior.
So apparently even the manpage itself is inconsistent ;) (and I will neglect the mistake that the text says "no I/O scheduler" instead of "no I/O priority"). And there is actually code in BFQ (and there used to be code in CFQ) that does:
switch (ioprio_class) {
...
case IOPRIO_CLASS_NONE:
/*
* No prio set, inherit CPU scheduling settings.
*/
bfqq->new_ioprio = task_nice_ioprio(tsk);
bfqq->new_ioprio_class = task_nice_ioclass(tsk);
break;So IOPRIO_CLASS_NONE indeed has a meaning and it used to be the default one until Christoph's 5a9d041ba2f6 in most cases. Your change e70344c05995 happening before that actually didn't change much practically because IO contexts were initialized with 0 priority anyway and that was what get_current_ioprio() was returning.
Quoted 5 lines
The different ioprio leading to no BIO merging is definitely a problem but this patch is not really fixing anything in my opinion. It simply gets back to the previous "all 0s" ioprio initialization, which do not show the places where we actually have missing ioprio initialization to
IOPRIO_DEFAULT.
So I agree we should settle on how to treat IOs with unset IO priority. The behavior to use task's CPU priority when IO priority is unset is there for a *long* time and so I think we should preserve that. The question is where in the stack should the switch from "unset" value to "effective ioprio" happen. Switching in IO context is IMO too early since userspace needs to be able to differentiate "unset" from "set to IOPRIO_CLASS_BE,IOPRIO_BE_NORM". But we could have a helper like current_effective_ioprio() that will do the magic with mangling unset IO priority based on task's CPU priority.
The fact is that bio->bi_ioprio gets set to anything only for direct IO in iomap_dio_rw(). The rest of the IO has priority unset (BFQ fetches the priority from task's IO context and ignores priority on bios BTW). And the only place where req->ioprio (inherited from bio->bi_ioprio) gets used is in a few drivers to set HIGHPRI flag for IOPRIO_CLASS_RT IO and then the relatively new code in mq-deadline.
So it all is very inconsistent mess :-|
Quoted 4 lines
Isn't it simply that
IOPRIO_DEFAULTshould be set as the default for any bio being allocated (inbio_alloc?) before it is setup and inherits the user io priority ? Otherwise, the bio io prio is indeed IOPRIO_CLASS_NONE/0 and changingIOPRIO_DEFAULTto that value removes the differences you observed.
Yes, I think that would make sence although as I explain above this is somewhat independent to what the default IO priority behavior should be.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 3/3] block: fix default IO priority handling again
Quoted 71 lines · Jan Kara
Quoted 54 lines · Damien Le Moal
Quoted 45 lines · Jan Kara
Commit
e70344c05995("block: fix default IO priority handling") introduced an inconsistency inget_current_ioprio()that tasks without IO context returnIOPRIO_DEFAULTpriority while tasks with freshly allocated IO context will return 0 (IOPRIO_CLASS_NONE/0) IO priority. Tasks without IO context used to be rare before5a9d041ba2f6("block:
moveio_contextcreation into where it's needed") but after this commit they became common because now only BFQ IO scheduler setups task's IO context. Similar inconsistency is there forget_task_ioprio()so this inconsistency is now exposed to userspace and userspace will see different IO priority for tasks operating on devices with BFQ compared to devices without BFQ. Furthemore the changes done by commite70344c05995change the behavior when no IO priority is set for BFQ IO scheduler which is also documented inioprio_set(2)manpage - namely that tasks without set IO priority will use IO priority based on their nice value.So make sure we default to
IOPRIO_CLASS_NONEas used to be the case before commite70344c05995. Also cleanupalloc_io_context()to explicitely set this IO priority for the allocated IO context.Fixes e70344c05995 ("block: fix default IO priority handling")Signed-off-by Jan Kara <[email protected]>--- block/blk-ioc.c | 2 ++ include/linux/ioprio.h | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)block/blk-ioc.c +2
-0@@ -247,6 +247,8 @@ static struct io_context *alloc_io_context(gfp_t gfp_flags, int node)INIT_HLIST_HEAD (&ioc->icq_list);INIT_WORK (&ioc->release_work, ioc_release_fn);#endif + ioc->ioprio =IOPRIO_DEFAULT ; +return ioc; }include/linux/ioprio.h +0
-0include/linux/ioprio.h line 14
Quoted 5 lines
include/linux/ioprio.h +1
-1@@ -11,7 +11,7 @@/* * Default IO priority. */ -#define IOPRIO_DEFAULT IOPRIO_PRIO_VALUE(IOPRIO_CLASS_BE, IOPRIO_BE_NORM) +#define IOPRIO_DEFAULT IOPRIO_PRIO_VALUE(IOPRIO_CLASS_NONE, 0)"man
ioprio_set" says:Quoted 6 lines
IOPRIO_CLASS_BE(2)
This is the best-effort scheduling class, which is the default for any process that hasn't set a specific I/O priority.Which is why patch
e70344c05995introducedIOPRIO_DEFAULTdefinition using the BE class, to have the kernel in sync with the manual.Yes, but it also has:
If no I/O scheduler has been set for a thread, then by default the I/O priority will follow the CPU nice value (setpriority(2)). In Linux kernels before version 2.6.24, once an I/O priority had been set using
ioprio_set(), there was no way to reset the I/O scheduling behavior to the default. Since Linux 2.6.24, specifying ioprio as 0 can be used to reset to the default I/O scheduling behavior.
Quoted 4 lines
So apparently even the manpage itself is inconsistent ;) (and I will neglect the mistake that the text says "no I/O scheduler" instead of "no I/O priority"). And there is actually code in BFQ (and there used to be code in CFQ) that does:
Arg, indeed, it is a mess. We need to fix this manpage too.
Quoted 10 lines
switch (ioprio_class) {
...
case IOPRIO_CLASS_NONE:
/*
* No prio set, inherit CPU scheduling settings.
*/
bfqq->new_ioprio = task_nice_ioprio(tsk);
bfqq->new_ioprio_class = task_nice_ioclass(tsk);
break;Quoted 5 lines
So
IOPRIO_CLASS_NONEindeed has a meaning and it used to be the default one until Christoph's5a9d041ba2f6in most cases. Your changee70344c05995happening before that actually didn't change much practically because IO contexts were initialized with 0 priority anyway and that was whatget_current_ioprio()was returning.
5a9d041ba2f6 was from Jens, but agreed, io priority initialization never has been correct.
Quoted 7 lines
Quoted 6 lines
The different ioprio leading to no BIO merging is definitely a problem but this patch is not really fixing anything in my opinion. It simply gets back to the previous "all 0s" ioprio initialization, which do not show the places where we actually have missing ioprio initialization to
IOPRIO_DEFAULT.
Quoted 9 lines
So I agree we should settle on how to treat IOs with unset IO priority. The behavior to use task's CPU priority when IO priority is unset is there for a *long* time and so I think we should preserve that. The question is where in the stack should the switch from "unset" value to "effective ioprio" happen. Switching in IO context is IMO too early since userspace needs to be able to differentiate "unset" from "set to IOPRIO_CLASS_BE,IOPRIO_BE_NORM". But we could have a helper like
current_effective_ioprio()that will do the magic with mangling unset IO priority based on task's CPU priority.
I agree that the task's CPU priority is a more sensible default. However, I do not understand your point about the IO context being too early to set the effective priority. If we do not do that, getting the issuer CPU priority will not be easily possible, right ?
Quoted 7 lines
The fact is that
bio->bi_iopriogets set to anything only for direct IO iniomap_dio_rw(). The rest of the IO has priority unset (BFQ fetches the priority from task's IO context and ignores priority on bios BTW). And the only place wherereq->ioprio(inherited frombio->bi_ioprio) gets used is in a few drivers to set HIGHPRI flag forIOPRIO_CLASS_RTIO and then the relatively new code in mq-deadline.
Yes, only IOPRIO_CLASS_RT has an effect at the hardware level right now. Everything else is only for the IO scheduler to play with. I am preparing a patch series to support scsi/ata command duration limits though. That adds a new IOPRIO_CLASS_DL and that one will also have an effect on the hardware.
Note that if a process/thread use ioprio_set(), we should also honor the ioprio set for buffered reads, at least those page IOs that are issued directly from the IO issuer context (which may include readahead... hmmm).
Quoted 2 lines
So it all is very inconsistent mess :-|
Indeed it is.
Quoted 8 lines
Quoted 5 lines
Isn't it simply that
IOPRIO_DEFAULTshould be set as the default for any bio being allocated (inbio_alloc?) before it is setup and inherits the user io priority ? Otherwise, the bio io prio is indeed IOPRIO_CLASS_NONE/0 and changingIOPRIO_DEFAULTto that value removes the differences you observed.Yes, I think that would make sence although as I explain above this is somewhat independent to what the default IO priority behavior should be.
I am OK with the use of the task CPU priority/ionice value as the default when no other ioprio is set for a bio using the user aio_reqprio or ioprio_set(). If this relies on task_nice_ioclass() as it is today (I see no reason to change that), then the default class for regular tasks remains IOPRIO_CLASS_BE as is defined by IOPRIO_DEFAULT.
But to avoid the performance regression you observed, we really need to be 100% sure that all bios have their ->bi_ioprio field correctly initialized. Something like:
void bio_set_effective_ioprio(struct *bio)
{
switch (IOPRIO_PRIO_CLASS(bio->bi_ioprio)) {
case IOPRIO_CLASS_RT:
case IOPRIO_CLASS_BE:
case IOPRIO_CLASS_IDLE:
/*
* the bio ioprio was already set from an aio kiocb ioprio
* (aio->aio_reqprio) or from the issuer context ioprio if that
* context used ioprio_set().
*/;
return;
case IOPRIO_CLASS_NONE:
default:
/* Use the current task CPU priority */
bio->ioprio =
IOPRIO_PRIO_VALUE(task_nice_ioclass(current),
task_nice_ioprio(current));
return;
}
}being called before a bio is inserted in a scheduler or bypass inserted in the dispatch queues should result in all BIOs having an ioprio that is set to something other than IOPRIO_CLASS_NONE. And the obvious place may be simply at the beginning of submit_bio(), before submit_bio_noacct() is called.
I am tempted to argue that block device drivers should never see any req with an ioprio set to IOPRIO_CLASS_NONE, which means that no bio should ever enter the block stack with that ioprio either. With the above solution, bios from DM targets submitted with submit_bio_noacct() could still have IOPRIO_CLASS_NONE... So would submit_bio_noacct() be the better place to call the effective ioprio helper ?
Signature · 2 lines
Damien Le Moal Western Digital Research
Re: [PATCH 3/3] block: fix default IO priority handling again
Quoted 18 lines · Damien Le Moal
Quoted 7 lines · Jan Kara
Quoted 5 lines · Damien Le Moal
The different ioprio leading to no BIO merging is definitely a problem but this patch is not really fixing anything in my opinion. It simply gets back to the previous "all 0s" ioprio initialization, which do not show the places where we actually have missing ioprio initialization to
IOPRIO_DEFAULT.Quoted 9 lines
So I agree we should settle on how to treat IOs with unset IO priority. The behavior to use task's CPU priority when IO priority is unset is there for a *long* time and so I think we should preserve that. The question is where in the stack should the switch from "unset" value to "effective ioprio" happen. Switching in IO context is IMO too early since userspace needs to be able to differentiate "unset" from "set to IOPRIO_CLASS_BE,IOPRIO_BE_NORM". But we could have a helper like
current_effective_ioprio()that will do the magic with mangling unset IO priority based on task's CPU priority.
Quoted 4 lines
I agree that the task's CPU priority is a more sensible default. However, I do not understand your point about the IO context being too early to set the effective priority. If we do not do that, getting the issuer CPU priority will not be easily possible, right ?
I just meant that in the IO context we need to keep the information whether IO priority is set to a particular value, or whether it is set to 0 meaning "inherit from CPU priority". So we cannot just store the effective IO priority immediately in the IO context instead of 0.
Quoted 7 lines
Quoted 6 lines
The fact is that
bio->bi_iopriogets set to anything only for direct IO iniomap_dio_rw(). The rest of the IO has priority unset (BFQ fetches the priority from task's IO context and ignores priority on bios BTW). And the only place wherereq->ioprio(inherited frombio->bi_ioprio) gets used is in a few drivers to set HIGHPRI flag forIOPRIO_CLASS_RTIO and then the relatively new code in mq-deadline.
Quoted 8 lines
Yes, only
IOPRIO_CLASS_RThas an effect at the hardware level right now. Everything else is only for the IO scheduler to play with. I am preparing a patch series to support scsi/ata command duration limits though. That adds a newIOPRIO_CLASS_DLand that one will also have an effect on the hardware.Note that if a process/thread use
ioprio_set(), we should also honor the ioprio set for buffered reads, at least those page IOs that are issued directly from the IO issuer context (which may include readahead... hmmm).
Yes, that would make sense as well.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Isn't it simply that
IOPRIO_DEFAULTshould be set as the default for any bio being allocated (inbio_alloc?) before it is setup and inherits the user io priority ? Otherwise, the bio io prio is indeed IOPRIO_CLASS_NONE/0 and changingIOPRIO_DEFAULTto that value removes the differences you observed.
Quoted 7 lines
Quoted 2 lines
Yes, I think that would make sence although as I explain above this is somewhat independent to what the default IO priority behavior should be.
I am OK with the use of the task CPU priority/ionice value as the default when no other ioprio is set for a bio using the user
aio_reqprioorioprio_set(). If this relies ontask_nice_ioclass()as it is today (I see no reason to change that), then the default class for regular tasks remainsIOPRIO_CLASS_BEas is defined byIOPRIO_DEFAULT.
Yes, good.
Quoted 31 lines
But to avoid the performance regression you observed, we really need to be 100% sure that all bios have their ->bi_ioprio field correctly initialized. Something like:
void
bio_set_effective_ioprio(struct *bio){ switch (IOPRIO_PRIO_CLASS(bio->bi_ioprio)) { case IOPRIO_CLASS_RT: case IOPRIO_CLASS_BE: case IOPRIO_CLASS_IDLE: /* * the bio ioprio was already set from an aio kiocb ioprio * (aio->aio_reqprio) or from the issuer context ioprio if that * context used ioprio_set(). */; return; case IOPRIO_CLASS_NONE: default: /* Use the current task CPU priority */ bio->ioprio = IOPRIO_PRIO_VALUE(task_nice_ioclass(current), task_nice_ioprio(current)); return; } }being called before a bio is inserted in a scheduler or bypass inserted in the dispatch queues should result in all BIOs having an ioprio that is set to something other than
IOPRIO_CLASS_NONE. And the obvious place may be simply at the beginning ofsubmit_bio(), beforesubmit_bio_noacct()is called.
Quoted 6 lines
I am tempted to argue that block device drivers should never see any req with an ioprio set to
IOPRIO_CLASS_NONE, which means that no bio should ever enter the block stack with that ioprio either. With the above solution, bios from DM targets submitted withsubmit_bio_noacct()could still haveIOPRIO_CLASS_NONE... So wouldsubmit_bio_noacct()be the better place to call the effective ioprio helper ?
Yes, I also think it would be the cleanest if we made sure bio->ioprio is always set to some value other than IOPRIO_CLASS_NONE. I'll see how we can make that happen in the least painful way :). Thanks for your input!
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 3/3] block: fix default IO priority handling again
Quoted 39 lines · Jan Kara
Quoted 31 lines · Damien Le Moal
But to avoid the performance regression you observed, we really need to be 100% sure that all bios have their ->bi_ioprio field correctly initialized. Something like:
void
bio_set_effective_ioprio(struct *bio){ switch (IOPRIO_PRIO_CLASS(bio->bi_ioprio)) { case IOPRIO_CLASS_RT: case IOPRIO_CLASS_BE: case IOPRIO_CLASS_IDLE: /* * the bio ioprio was already set from an aio kiocb ioprio * (aio->aio_reqprio) or from the issuer context ioprio if that * context used ioprio_set(). */; return; case IOPRIO_CLASS_NONE: default: /* Use the current task CPU priority */ bio->ioprio = IOPRIO_PRIO_VALUE(task_nice_ioclass(current), task_nice_ioprio(current)); return; } }being called before a bio is inserted in a scheduler or bypass inserted in the dispatch queues should result in all BIOs having an ioprio that is set to something other than
IOPRIO_CLASS_NONE. And the obvious place may be simply at the beginning ofsubmit_bio(), beforesubmit_bio_noacct()is called.Quoted 6 lines
I am tempted to argue that block device drivers should never see any req with an ioprio set to
IOPRIO_CLASS_NONE, which means that no bio should ever enter the block stack with that ioprio either. With the above solution, bios from DM targets submitted withsubmit_bio_noacct()could still haveIOPRIO_CLASS_NONE... So wouldsubmit_bio_noacct()be the better place to call the effective ioprio helper ?
Quoted 3 lines
Yes, I also think it would be the cleanest if we made sure
bio->iopriois always set to some value other thanIOPRIO_CLASS_NONE. I'll see how we can make that happen in the least painful way :). Thanks for your input!
When looking into this I've hit a snag: ioprio rq_qos policy relies on the fact that bio->bi_ioprio remains at 0 (unless explicitely set to some other value by userspace) until we call rq_qos_track() in blk_mq_submit_bio(). BTW this happens after we have attempted to merge the bio to existing requests so ioprio rq_qos policy is going to show strange behavior wrt merging - most of the bios will not be able to merge to existing queued requests due to ioprio mismatch.
I'd say .track hook gets called too late to properly set bio->bi_ioprio. Shouldn't we set the io priority much earlier - I'd be tempted to use bio_associate_blkg_from_css() for this... What do people think?
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 3/3] block: fix default IO priority handling again
Quoted 52 lines · Jan Kara
Quoted 39 lines · Jan Kara
Quoted 31 lines · Damien Le Moal
But to avoid the performance regression you observed, we really need to be 100% sure that all bios have their ->bi_ioprio field correctly initialized. Something like:
void
bio_set_effective_ioprio(struct *bio){ switch (IOPRIO_PRIO_CLASS(bio->bi_ioprio)) { case IOPRIO_CLASS_RT: case IOPRIO_CLASS_BE: case IOPRIO_CLASS_IDLE: /* * the bio ioprio was already set from an aio kiocb ioprio * (aio->aio_reqprio) or from the issuer context ioprio if that * context used ioprio_set(). */; return; case IOPRIO_CLASS_NONE: default: /* Use the current task CPU priority */ bio->ioprio = IOPRIO_PRIO_VALUE(task_nice_ioclass(current), task_nice_ioprio(current)); return; } }being called before a bio is inserted in a scheduler or bypass inserted in the dispatch queues should result in all BIOs having an ioprio that is set to something other than
IOPRIO_CLASS_NONE. And the obvious place may be simply at the beginning ofsubmit_bio(), beforesubmit_bio_noacct()is called.Quoted 6 lines
I am tempted to argue that block device drivers should never see any req with an ioprio set to
IOPRIO_CLASS_NONE, which means that no bio should ever enter the block stack with that ioprio either. With the above solution, bios from DM targets submitted withsubmit_bio_noacct()could still haveIOPRIO_CLASS_NONE... So wouldsubmit_bio_noacct()be the better place to call the effective ioprio helper ?Quoted 3 lines
Yes, I also think it would be the cleanest if we made sure
bio->iopriois always set to some value other thanIOPRIO_CLASS_NONE. I'll see how we can make that happen in the least painful way :). Thanks for your input!When looking into this I've hit a snag: ioprio
rq_qospolicy relies on the fact thatbio->bi_ioprioremains at 0 (unless explicitely set to some other value by userspace) until we callrq_qos_track()inblk_mq_submit_bio(). BTW this happens after we have attempted to merge the bio to existing requests so iopriorq_qospolicy is going to show strange behavior wrt merging - most of the bios will not be able to merge to existing queued requests due to ioprio mismatch.
Quoted 3 lines
I'd say .track hook gets called too late to properly set
bio->bi_ioprio. Shouldn't we set the io priority much earlier - I'd be tempted to usebio_associate_blkg_from_css()for this... What do people think?
Hello Jan,
bio_associate_blkg_from_css() is just an empty stub if CONFIG_BLK_CGROUP is not set.
Having the effective ioprio set should correctly shouldn't depend on if CONFIG_BLK_CGROUP is set or not, no?
The function name bio_associate_blkg_from_css() (css - cgroup_subsys_state) also seems to imply that it should only perform cgroup related things, no?
AFAICT, both bfq and mq-deadline can currently prioritize requests without CONFIG_BLK_CGROUP enabled.
Kind regards,
Niklas
Re: [PATCH 3/3] block: fix default IO priority handling again
Quoted 62 lines · Niklas Cassel
Quoted 52 lines · Jan Kara
Quoted 39 lines · Jan Kara
Quoted 31 lines · Damien Le Moal
But to avoid the performance regression you observed, we really need to be 100% sure that all bios have their ->bi_ioprio field correctly initialized. Something like:
void
bio_set_effective_ioprio(struct *bio){ switch (IOPRIO_PRIO_CLASS(bio->bi_ioprio)) { case IOPRIO_CLASS_RT: case IOPRIO_CLASS_BE: case IOPRIO_CLASS_IDLE: /* * the bio ioprio was already set from an aio kiocb ioprio * (aio->aio_reqprio) or from the issuer context ioprio if that * context used ioprio_set(). */; return; case IOPRIO_CLASS_NONE: default: /* Use the current task CPU priority */ bio->ioprio = IOPRIO_PRIO_VALUE(task_nice_ioclass(current), task_nice_ioprio(current)); return; } }being called before a bio is inserted in a scheduler or bypass inserted in the dispatch queues should result in all BIOs having an ioprio that is set to something other than
IOPRIO_CLASS_NONE. And the obvious place may be simply at the beginning ofsubmit_bio(), beforesubmit_bio_noacct()is called.Quoted 6 lines
I am tempted to argue that block device drivers should never see any req with an ioprio set to
IOPRIO_CLASS_NONE, which means that no bio should ever enter the block stack with that ioprio either. With the above solution, bios from DM targets submitted withsubmit_bio_noacct()could still haveIOPRIO_CLASS_NONE... So wouldsubmit_bio_noacct()be the better place to call the effective ioprio helper ?Quoted 3 lines
Yes, I also think it would be the cleanest if we made sure
bio->iopriois always set to some value other thanIOPRIO_CLASS_NONE. I'll see how we can make that happen in the least painful way :). Thanks for your input!When looking into this I've hit a snag: ioprio
rq_qospolicy relies on the fact thatbio->bi_ioprioremains at 0 (unless explicitely set to some other value by userspace) until we callrq_qos_track()inblk_mq_submit_bio(). BTW this happens after we have attempted to merge the bio to existing requests so iopriorq_qospolicy is going to show strange behavior wrt merging - most of the bios will not be able to merge to existing queued requests due to ioprio mismatch.Quoted 3 lines
I'd say .track hook gets called too late to properly set
bio->bi_ioprio. Shouldn't we set the io priority much earlier - I'd be tempted to usebio_associate_blkg_from_css()for this... What do people think?Hello Jan,
bio_associate_blkg_from_css()is just an empty stub ifCONFIG_BLK_CGROUPis not set.
Quoted 8 lines
Having the effective ioprio set should correctly shouldn't depend on if
CONFIG_BLK_CGROUPis set or not, no?The function name
bio_associate_blkg_from_css()(css -cgroup_subsys_state) also seems to imply that it should only perform cgroup related things, no?AFAICT, both bfq and mq-deadline can currently prioritize requests without
CONFIG_BLK_CGROUPenabled.
Correct on all points. However the ioprio rq_qos policy very much depends on the cgroup support. So at least the update of bio->bi_ioprio based on that policy would make sense in bio_associate_blkg_from_css(). OTOH thinking about it now, we would have a problem that if bio blkcg association changes, we don't have enough information to update the bi_ioprio accordingly (since we don't know whether the resulting ioprio is set by userspace or by ioprio rq_qos policy). So probably we need make ioprio rq_qos policy to set the priority later (likely at bio submission time when bio blkcg association is stable) but we need to do it earlier than after we try to merge the bio...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR