block: Fix IO priority mess
Subsystems (3)
v5 cover block: Fix IO priority mess
Hello,
This is the fifth revision of my patches fixing IO priority handling in the block layer. Damien has reviewed all the patches and tested them as well so I think patches are ready to be merged.
Changes since v4:
* Added Reviewed-by and Tested-by tags
* Fixed prototype of stub function for !CONFIG_BLOCK
Changes since v3:
* Added Reviewed-by tags from Damien
* Fixed build failure without CONFIG_BLOCK
* Separated refactoring of get_current_ioprio() into a separate patch
Changes since v2:
* added some comments to better explain things
* changed handling of ioprio_get(2)
* a few small tweaks based on Damien's feedback
Original cover letter:
Recently, I've been looking into 10% regression reported by our performance measurement infrastructure in reaim benchmark that was bisected down to 5a9d041ba2f6 ("block: move io_context creation into where it's needed"). This didn't really make much sense and it took me a while to understand this but the culprit is actually in even older commit e70344c05995 ("block: fix default IO priority handling") and 5a9d041ba2f6 just made the breakage visible. Essentially the problem was that after these commits some IO was queued with IO priority class IOPRIO_CLASS_BE while other IO was queued with IOPRIO_CLASS_NONE and as a result they could not be merged together resulting in performance regression. I think what commit e70344c05995 ("block: fix default IO priority handling") did is actually broken not only because of this performance regression but because of other reasons as well (see changelog of patch 3/8 for details). Besides this problem, there are multiple other inconsistencies in the IO priority handling throughout the block stack we have identified when discussing this with Damien Le Moal. So this patch set aims at fixing all these various issues.
Note that there are a few choices I've made I'm not 100% sold on. In particular the conversion of blk-ioprio from rqos is somewhat disputable since it now incurs a cost similar to blk-throttle in the bio submission fast path (need to load bio->bi_blkg->pd[ioprio_policy.plid]). If people think the removed boilerplate code is not worth the cost, I can certainly go via the "additional rqos hook" path.
Honza
Previous versions:
PATCH v5 1/9 block: fix default IO priority handling again
Commit e70344c05995 ("block: fix default IO priority handling") introduced an inconsistency in get_current_ioprio() that tasks without IO context return IOPRIO_DEFAULT priority while tasks with freshly allocated IO context will return 0 (IOPRIO_CLASS_NONE/0) IO priority. Tasks without IO context used to be rare before 5a9d041ba2f6 ("block:
move io_context creation into where it's needed") but after this commit they became common because now only BFQ IO scheduler setups task's IO context. Similar inconsistency is there for get_task_ioprio() so this inconsistency is now exposed to userspace and userspace will see different IO priority for tasks operating on devices with BFQ compared to devices without BFQ. Furthemore the changes done by commit e70344c05995 change the behavior when no IO priority is set for BFQ IO scheduler which is also documented in ioprio_set(2) manpage:
"If no I/O scheduler has been set for a thread, then by default the I/O priority will follow the CPU nice value (setpriority(2)). In Linux kernels before version 2.6.24, once an I/O priority had been set using ioprio_set(), there was no way to reset the I/O scheduling behavior to the default. Since Linux 2.6.24, specifying ioprio as 0 can be used to reset to the default I/O scheduling behavior."
So make sure we default to IOPRIO_CLASS_NONE as used to be the case before commit e70344c05995. Also cleanup alloc_io_context() to explicitely set this IO priority for the allocated IO context to avoid future surprises. Note that we tweak ioprio_best() to maintain ioprio_get(2) behavior and make this commit easily backportable.
--- block/blk-ioc.c | 2 ++ block/ioprio.c | 4 ++-- include/linux/ioprio.h | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-)
block/blk-ioc.c +2 -0
@@ -247,6 +247,8 @@ static struct io_context *alloc_io_context(gfp_t gfp_flags, int node)INIT_HLIST_HEAD (&ioc->icq_list);INIT_WORK (&ioc->release_work, ioc_release_fn);#endif + ioc->ioprio =IOPRIO_DEFAULT ; +return ioc; }
block/ioprio.c +2 -2
@@ -157,9 +157,9 @@ static int get_task_ioprio(struct task_struct *p)int ioprio_best (unsigned short aprio,unsigned short bprio) {if (!ioprio_valid (aprio)) - aprio =IOPRIO_DEFAULT ; + aprio =IOPRIO_PRIO_VALUE (IOPRIO_CLASS_BE ,IOPRIO_BE_NORM );if (!ioprio_valid (bprio)) - bprio =IOPRIO_DEFAULT ; + bprio =IOPRIO_PRIO_VALUE (IOPRIO_CLASS_BE ,IOPRIO_BE_NORM );return min (aprio, bprio); }
include/linux/ioprio.h +1 -1
@@ -11,7 +11,7 @@/* * Default IO priority. */ -#define IOPRIO_DEFAULT IOPRIO_PRIO_VALUE(IOPRIO_CLASS_BE, IOPRIO_BE_NORM) +#define IOPRIO_DEFAULT IOPRIO_PRIO_VALUE(IOPRIO_CLASS_NONE, 0)/* * Check that a priority value has a valid class.
Signature · 1 line
2.35.3
Re: [PATCH 1/9] block: fix default IO priority handling again
Looks good:
Re: [PATCH 1/9] block: fix default IO priority handling again
In reply to Jan KaraQuoted 14 lines · Jan Kara
Commit
e70344c05995("block: fix default IO priority handling") introduced an inconsistency inget_current_ioprio()that tasks without IO context returnIOPRIO_DEFAULTpriority while tasks with freshly allocated IO context will return 0 (IOPRIO_CLASS_NONE/0) IO priority. Tasks without IO context used to be rare before5a9d041ba2f6("block:
moveio_contextcreation into where it's needed") but after this commit they became common because now only BFQ IO scheduler setups task's IO context. Similar inconsistency is there forget_task_ioprio()so this inconsistency is now exposed to userspace and userspace will see different IO priority for tasks operating on devices with BFQ compared to devices without BFQ. Furthemore the changes done by commite70344c05995change the behavior when no IO priority is set for BFQ IO scheduler which is also documented inioprio_set(2)manpage:
Quoted 1 line
[...]
Applied, thanks!
[1/9] block: fix default IO priority handling again
commit: f0f5a5e24fa5412f187f429232334ad6832d1a66
[2/9] block: Return effective IO priority from get_current_ioprio()
commit: 93fd10125cd702d86f1c4005349b54eeb3c02af3
[3/9] block: Generalize get_current_ioprio() for any task
commit: 86f80bd5f639921c59afb113fa3ebb3ccb46be84
[4/9] block: Make ioprio_best() static
commit: c85fb98c51a66ff7346f2e12f6c20fb4f60de812
[5/9] block: Fix handling of tasks without ioprio in ioprio_get(2)
commit: caf2c269be20c536009ccd815a4e493d0c6c6634
[6/9] blk-ioprio: Remove unneeded field
commit: d2adb01a5bcbe36bc05fbb383028da755f7a919b
[7/9] blk-ioprio: Convert from rqos policy to direct call
commit: 8f3d8d7f56aba4c6171e48b107b9167255044653
[8/9] block: Initialize bio priority earlier
commit: 92c3dfe1cfce7dd7cf6cd32b78b05885d824656e
[9/9] block: Always initialize bio IO priority on submit
commit: 71ad7aabb8968164b1963fff7216b225fdd80f84Best regards,
Signature · 1 line
Jens Axboe
PATCH v5 2/9 block: Return effective IO priority from get_current_ioprio()
In reply to Jan Karaget_current_ioprio() is used to initialize IO priority of various requests. As such it should be returning the effective IO priority of the task (i.e., reflecting the fact that unset IO priority should get set based on task's CPU priority) so that the conversion is concentrated in one place.
--- include/linux/ioprio.h | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
include/linux/ioprio.h +9 -2
@@ -53,10 +53,17 @@ static inline int task_nice_ioclass(struct task_struct *task)static inline int get_current_ioprio (void ) {struct io_context *ioc = current->io_context; +int prio;if (ioc) -return ioc->ioprio; -return IOPRIO_DEFAULT ; + prio = ioc->ioprio; +else + prio =IOPRIO_DEFAULT ; + +if (IOPRIO_PRIO_CLASS (prio) ==IOPRIO_CLASS_NONE ) + prio =IOPRIO_PRIO_VALUE (task_nice_ioclass (current), +task_nice_ioprio (current)); +return prio; }/*
Signature · 1 line
2.35.3
Re: [PATCH 2/9] block: Return effective IO priority from get_current_ioprio()
Quoted 5 lines · Jan Kara
get_current_ioprio()is used to initialize IO priority of various requests. As such it should be returning the effective IO priority of the task (i.e., reflecting the fact that unset IO priority should get set based on task's CPU priority) so that the conversion is concentrated in one place.
Looks good:
PATCH v5 3/9 block: Generalize get_current_ioprio() for any task
In reply to Jan Karaget_current_ioprio() operates only on current task. We will need the same functionality for other tasks as well. Generalize get_current_ioprio() for that and also move the bulk out of the header file because it is large enough.
--- block/ioprio.c | 26 ++++++++++++++++++++++++++ include/linux/ioprio.h | 26 ++++++++++---------------- 2 files changed, 36 insertions(+), 16 deletions(-)
block/ioprio.c +26 -0
@@ -138,6 +138,32 @@ SYSCALL_DEFINE3(ioprio_set, int, which, int, who, int, ioprio)return ret; } +/* +* If the task has set an I/O priority, use that. Otherwise, return +* the default I/O priority. +* +* Expected to be called for current task or with task_lock() held to keep +* io_context stable. +*/ +int __get_task_ioprio (struct task_struct *p) +{ +struct io_context *ioc = p->io_context; +int prio; + +if (p != current) +lockdep_assert_held (&p->alloc_lock); +if (ioc) + prio = ioc->ioprio; +else + prio =IOPRIO_DEFAULT ; + +if (IOPRIO_PRIO_CLASS (prio) ==IOPRIO_CLASS_NONE ) + prio =IOPRIO_PRIO_VALUE (task_nice_ioclass (p), +task_nice_ioprio (p)); +return prio; +} +EXPORT_SYMBOL_GPL (__get_task_ioprio); +static int get_task_ioprio (struct task_struct *p) {int ret;
include/linux/ioprio.h +10 -16
@@ -46,24 +46,18 @@ static inline int task_nice_ioclass(struct task_struct *task)return IOPRIO_CLASS_BE ; } -/* -* If the calling process has set an I/O priority, use that. Otherwise, return -* the default I/O priority. -*/ -static inline int get_current_ioprio (void ) +#ifdef CONFIG_BLOCK +int __get_task_ioprio (struct task_struct *p); +#else +static inline int __get_task_ioprio (struct task_struct *p) { -struct io_context *ioc = current->io_context; -int prio; - -if (ioc) - prio = ioc->ioprio; -else - prio =IOPRIO_DEFAULT ; +return IOPRIO_DEFAULT ; +} +#endif /* CONFIG_BLOCK */ -if (IOPRIO_PRIO_CLASS (prio) ==IOPRIO_CLASS_NONE ) - prio =IOPRIO_PRIO_VALUE (task_nice_ioclass (current), -task_nice_ioprio (current)); -return prio; +static inline int get_current_ioprio (void ) +{ +return __get_task_ioprio (current); }/*
Signature · 1 line
2.35.3
Re: [PATCH 3/9] block: Generalize get_current_ioprio() for any task
Quoted 4 lines · Jan Kara
get_current_ioprio()operates only on current task. We will need the same functionality for other tasks as well. Generalizeget_current_ioprio()for that and also move the bulk out of the header file because it is large enough.
We don't really need the general version exported. But if you prefer an inline wrapper over a tail call I guess this is fine, too:
PATCH v5 4/9 block: Make ioprio_best() static
In reply to Jan KaraNobody outside of block/ioprio.c uses it.
--- block/ioprio.c | 2 +- include/linux/ioprio.h | 5 ----- 2 files changed, 1 insertion(+), 6 deletions(-)
block/ioprio.c +1 -1
@@ -180,7 +180,7 @@ static int get_task_ioprio(struct task_struct *p)return ret; } -int ioprio_best (unsigned short aprio,unsigned short bprio) +static int ioprio_best (unsigned short aprio,unsigned short bprio) {if (!ioprio_valid (aprio)) aprio =IOPRIO_PRIO_VALUE (IOPRIO_CLASS_BE ,IOPRIO_BE_NORM );
include/linux/ioprio.h +0 -5
@@ -60,11 +60,6 @@ static inline int get_current_ioprio(void)return __get_task_ioprio (current); } -/* -* For inheritance, return the highest of the two given priorities -*/ -extern int ioprio_best (unsigned short aprio,unsigned short bprio); -extern int set_task_ioprio (struct task_struct *task,int ioprio);#ifdef CONFIG_BLOCK
Signature · 1 line
2.35.3
Re: [PATCH 4/9] block: Make ioprio_best() static
Looks good:
PATCH v5 5/9 block: Fix handling of tasks without ioprio in ioprio_get(2)
In reply to Jan Karaioprio_get(2) can be asked to return the best IO priority from several tasks (IOPRIO_WHO_PGRP, IOPRIO_WHO_USER). Currently the call treats tasks without set IO priority as having priority IOPRIO_CLASS_BE/IOPRIO_BE_NORM however this does not really reflect the IO priority the task will get (which depends on task's nice value).
Fix the code to use the real IO priority task's IO will use. We have to modify code for ioprio_get(IOPRIO_WHO_PROCESS) to keep returning IOPRIO_CLASS_NONE priority for tasks without set IO priority as a special case to maintain userspace visible API.
--- block/ioprio.c | 30 +++++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-)
block/ioprio.c +23 -7
@@ -171,10 +171,31 @@ static int get_task_ioprio(struct task_struct *p) ret =security_task_getioprio (p);if (ret) goto out; - ret =IOPRIO_DEFAULT ; +task_lock (p); + ret =__get_task_ioprio (p); +task_unlock (p); +out: +return ret; +} + +/* +* Return raw IO priority value as set by userspace. We use this for +* ioprio_get(pid, IOPRIO_WHO_PROCESS) so that we keep historical behavior and +* also so that userspace can distinguish unset IO priority (which just gets +* overriden based on task's nice value) from IO priority set to some value. +*/ +static int get_task_raw_ioprio (struct task_struct *p) +{ +int ret; + + ret =security_task_getioprio (p); +if (ret) + goto out;task_lock (p);if (p->io_context) ret = p->io_context->ioprio; +else + ret =IOPRIO_DEFAULT ;task_unlock (p); out:return ret;
@@ -182,11 +203,6 @@ static int get_task_ioprio(struct task_struct *p)static int ioprio_best (unsigned short aprio,unsigned short bprio) { -if (!ioprio_valid (aprio)) - aprio =IOPRIO_PRIO_VALUE (IOPRIO_CLASS_BE ,IOPRIO_BE_NORM ); -if (!ioprio_valid (bprio)) - bprio =IOPRIO_PRIO_VALUE (IOPRIO_CLASS_BE ,IOPRIO_BE_NORM ); -return min (aprio, bprio); }
@@ -207,7 +223,7 @@ SYSCALL_DEFINE2(ioprio_get, int, which, int, who)else p =find_task_by_vpid (who);if (p) - ret =get_task_ioprio (p); + ret =get_task_raw_ioprio (p);break ;case IOPRIO_WHO_PGRP :if (!who)
Signature · 1 line
2.35.3
PATCH v5 6/9 blk-ioprio: Remove unneeded field
In reply to Jan Karablkcg->ioprio_set field is not really useful except for avoiding possibly more expensive checks inside blkcg_ioprio_track(). The check for blkcg->prio_policy being equal to POLICY_NO_CHANGE does the same service so just remove the ioprio_set field and replace the check.
--- block/blk-ioprio.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-)
block/blk-ioprio.c +3 -6
@@ -62,7 +62,6 @@ struct ioprio_blkg {struct ioprio_blkcg {struct blkcg_policy_data cpd;enum prio_policy prio_policy; -bool prio_set; };static inline struct ioprio_blkg *pd_to_ioprio (struct blkg_policy_data *pd)
@@ -113,7 +112,6 @@ static ssize_t ioprio_set_prio_policy(struct kernfs_open_file *of, char *buf,if (ret <0 )return ret; blkcg->prio_policy = ret; - blkcg->prio_set = true;return nbytes; }
@@ -193,16 +191,15 @@ static void blkcg_ioprio_track(struct rq_qos *rqos, struct request *rq,struct ioprio_blkcg *blkcg =ioprio_blkcg_from_bio (bio);u16 prio; -if (!blkcg->prio_set) +if (blkcg->prio_policy ==POLICY_NO_CHANGE )return ;/* * Except for IOPRIO_CLASS_NONE, higher I/O priority numbers * correspond to a lower priority. Hence, the max_t() below selects * the lower priority of bi_ioprio and the cgroup I/O priority class. -* If the cgroup policy has been set to POLICY_NO_CHANGE == 0, the -* bio I/O priority is not modified. If the bio I/O priority equals -* IOPRIO_CLASS_NONE, the cgroup I/O priority is assigned to the bio. +* If the bio I/O priority equals IOPRIO_CLASS_NONE, the cgroup I/O +* priority is assigned to the bio. */ prio =max_t (u16, bio->bi_ioprio,IOPRIO_PRIO_VALUE (blkcg->prio_policy,0 ));
Signature · 1 line
2.35.3
Re: [PATCH 6/9] blk-ioprio: Remove unneeded field
Looks god:
PATCH v5 7/9 blk-ioprio: Convert from rqos policy to direct call
In reply to Jan KaraConvert blk-ioprio handling from a rqos policy to a direct call from blk_mq_submit_bio(). Firstly, blk-ioprio is not much of a rqos policy anyway, it just needs a hook in bio submission path to set the bio's IO priority. Secondly, the rqos .track hook gets actually called too late for blk-ioprio purposes and introducing a special rqos hook just for blk-ioprio looks even weirder.
--- block/blk-cgroup.c | 1 + block/blk-ioprio.c | 50 +++++----------------------------------------- block/blk-ioprio.h | 9 +++++++++ block/blk-mq.c | 8 ++++++++ 4 files changed, 23 insertions(+), 45 deletions(-)
block/blk-cgroup.c +1 -0
@@ -1299,6 +1299,7 @@ int blkcg_init_queue(struct request_queue *q) ret =blk_iolatency_init (q);if (ret) {blk_throtl_exit (q); +blk_ioprio_exit (q); goto err_destroy_all; }
block/blk-ioprio.c +5 -45
71 lines not on this page · raw message
block/blk-ioprio.h +9 -0
@@ -6,14 +6,23 @@#include <linux/kconfig.h> struct request_queue ; +struct bio ;#ifdef CONFIG_BLK_CGROUP_IOPRIO int blk_ioprio_init (struct request_queue *q); +void blk_ioprio_exit (struct request_queue *q); +void blkcg_set_ioprio (struct bio *bio);#else static inline int blk_ioprio_init (struct request_queue *q) {return 0 ; } +static inline void blk_ioprio_exit (struct request_queue *q) +{ +} +static inline void blkcg_set_ioprio (struct bio *bio) +{ +}#endif #endif /* _BLK_IOPRIO_H_ */
block/blk-mq.c +8 -0
@@ -42,6 +42,7 @@#include "blk-stat.h" #include "blk-mq-sched.h" #include "blk-rq-qos.h" +#include "blk-ioprio.h" static DEFINE_PER_CPU(struct llist_head, blk_cpu_done);
@@ -2790,6 +2791,11 @@ static inline struct request *blk_mq_get_cached_request(struct request_queue *q,return rq; } +static void bio_set_ioprio (struct bio *bio) +{ +blkcg_set_ioprio (bio); +} +/** * blk_mq_submit_bio - Create and send a request to block device. * @bio: Bio pointer.
@@ -2830,6 +2836,8 @@ void blk_mq_submit_bio(struct bio *bio)trace_block_getrq (bio); +bio_set_ioprio (bio); +rq_qos_track (q, rq, bio);blk_mq_bio_to_request (rq, bio, nr_segs);
Signature · 1 line
2.35.3
Re: [PATCH 7/9] blk-ioprio: Convert from rqos policy to direct call
Nice! The rqos calling conventions always seem like a bit of obsfucation anyway..
PATCH v5 8/9 block: Initialize bio priority earlier
In reply to Jan KaraBio's IO priority needs to be initialized before we try to merge the bio with other bios. Otherwise we could merge bios which would otherwise receive different IO priorities leading to possible QoS issues.
--- block/blk-mq.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
block/blk-mq.c +2 -2
@@ -2825,6 +2825,8 @@ void blk_mq_submit_bio(struct bio *bio)if (!bio_integrity_prep (bio))return ; +bio_set_ioprio (bio); + rq =blk_mq_get_cached_request (q, plug, &bio, nr_segs);if (!rq) {if (!bio)
@@ -2836,8 +2838,6 @@ void blk_mq_submit_bio(struct bio *bio)trace_block_getrq (bio); -bio_set_ioprio (bio); -rq_qos_track (q, rq, bio);blk_mq_bio_to_request (rq, bio, nr_segs);
Signature · 1 line
2.35.3
Re: [PATCH 8/9] block: Initialize bio priority earlier
Quoted 3 lines · Jan Kara
Bio's IO priority needs to be initialized before we try to merge the bio with other bios. Otherwise we could merge bios which would otherwise receive different IO priorities leading to possible QoS issues.
Looks good:
PATCH v5 9/9 block: Always initialize bio IO priority on submit
In reply to Jan KaraCurrently, IO priority set in task's IO context is not reflected in the bio->bi_ioprio for most IO (only io_uring and direct IO set it). This results in odd results where process is submitting some bios with one priority and other bios with a different (unset) priority and due to differing priorities bios cannot be merged. Make sure bio->bi_ioprio is always set on bio submission.
--- block/blk-mq.c | 3 +++ 1 file changed, 3 insertions(+)
block/blk-mq.c +3 -0
@@ -2793,6 +2793,9 @@ static inline struct request *blk_mq_get_cached_request(struct request_queue *q,static void bio_set_ioprio (struct bio *bio) { +/* Nobody set ioprio so far? Initialize it based on task's nice value */ +if (IOPRIO_PRIO_CLASS (bio->bi_ioprio) ==IOPRIO_CLASS_NONE ) + bio->bi_ioprio =get_current_ioprio ();blkcg_set_ioprio (bio); }
Signature · 1 line
2.35.3
Re: [PATCH 9/9] block: Always initialize bio IO priority on submit
Quoted 6 lines · Jan Kara
Currently, IO priority set in task's IO context is not reflected in the
bio->bi_iopriofor most IO (onlyio_uringand direct IO set it). This results in odd results where process is submitting some bios with one priority and other bios with a different (unset) priority and due to differing priorities bios cannot be merged. Make surebio->bi_iopriois always set on bio submission.
Looks good: