block: invalidate cached plug timestamp on context switch
Subsystems (5)
cover block: invalidate cached plug timestamp on context switch
The details for this are in patch 2. The main reason for this series is to invalidate the cached timestamp on context switch. This was done in sched_update_worker() only before which was resulting in blk-iocost reading stale timestamps and throttling based on wrong information.
Patch 1 is a prerequisite to create the invariant that PF_BLOCK_TS set implies current->plug != NULL.
v2 -> v3:
https://lore.kernel.org/all/[email protected]/
- Add patch 1 to clear PF_BLOCK_TS in copy_process() so the invariant survives fork.
- Drop the if (plug) NULL check inside blk_plug_invalidate_ts(), relying on the invariant established by patch 1. (Peter Zijlstra)
v1 -> v2:
https://lore.kernel.org/all/[email protected]/
- Move the PF_BLOCK_TS check into blk_plug_invalidate_ts() and upgrade it to __always_inline (Peter Zijlstra).
- Drop the tsk parameter; the helper only ever operates on current.
Usama Arif (2):
kernel/fork: clear PF_BLOCK_TS in copy_process()
block: invalidate cached plug timestamp after task switch
include/linux/blkdev.h | 16 ++++++---------- kernel/fork.c | 1 + kernel/sched/core.c | 12 ++++++++---- 3 files changed, 15 insertions(+), 14 deletions(-)
Signature · 1 line
2.53.0-Meta
PATCH 1/2 kernel/fork: clear PF_BLOCK_TS in copy_process()
PF_BLOCK_TS is only set in blk_time_get_ns() when current->plug is non-NULL, and blk_finish_plug() clears it via __blk_flush_plug() before NULLing the plug pointer. copy_process() breaks the invariant by inheriting PF_BLOCK_TS from the parent while resetting the child's plug to NULL.
Clear PF_BLOCK_TS alongside that assignment so callers can rely on "PF_BLOCK_TS set implies current->plug != NULL" and dereference current->plug unguarded.
--- kernel/fork.c | 1 + 1 file changed, 1 insertion(+)
kernel/fork.c +1 -0
@@ -2338,6 +2338,7 @@ __latent_entropy struct task_struct *copy_process(#ifdef CONFIG_BLOCK p->plug =NULL ; + p->flags &= ~PF_BLOCK_TS ;#endif futex_init_task (p);
Signature · 1 line
2.53.0-Meta
PATCH 2/2 block: invalidate cached plug timestamp after task switch
In reply to Usama Arifblk_time_get_ns() caches ktime_get_ns() in current->plug->cur_ktime and marks the task with PF_BLOCK_TS. That cache is only valid while the task keeps running; if the task is switched out, wall-clock time advances and the cached value must not be reused when the task runs again.
The existing invalidation covers explicit plug flushes through __blk_flush_plug(), and the schedule() / rtmutex paths through sched_update_worker(). It does not cover in-kernel preemption paths such as preempt_schedule(), preempt_schedule_notrace(), and preempt_schedule_irq(), which enter __schedule(SM_PREEMPT) directly and return without calling sched_update_worker().
As a result, a task preempted while holding a plug with PF_BLOCK_TS set can reuse a stale plug->cur_ktime after it is scheduled back in. blk-iocost then consumes that stale timestamp through ioc_now(), producing stale vnow values for throttle decisions, and through ioc_rqos_done(), inflating on-queue time and feeding false missed-QoS samples into vrate adjustment.
Move the schedule-side invalidation to finish_task_switch(), which runs for the scheduled-in task after every actual context switch regardless of which schedule entry point was used. Keep __blk_flush_plug() as the explicit flush/finish-plug invalidation path, and remove only the PF_BLOCK_TS handling from sched_update_worker().
--- include/linux/blkdev.h | 16 ++++++---------- kernel/sched/core.c | 12 ++++++++---- 2 files changed, 14 insertions(+), 14 deletions(-)
include/linux/blkdev.h +6 -10
@@ -1216,16 +1216,12 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async)__blk_flush_plug (plug, async); } -/* -* tsk == current here -*/ -static inline void blk_plug_invalidate_ts (struct task_struct *tsk) +static __always_inline void blk_plug_invalidate_ts(void) { -struct blk_plug *plug = tsk->plug; - -if (plug) - plug->cur_ktime =0 ; - current->flags &= ~PF_BLOCK_TS ; +if (unlikely (current->flags &PF_BLOCK_TS )) { + current->plug->cur_ktime =0 ; + current->flags &= ~PF_BLOCK_TS ; + } }int blkdev_issue_flush (struct block_device *bdev);
@@ -1251,7 +1247,7 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async) { } -static inline void blk_plug_invalidate_ts (struct task_struct *tsk) +static inline void blk_plug_invalidate_ts (void ) { }
kernel/sched/core.c +8 -4
@@ -5368,6 +5368,12 @@ static struct rq *finish_task_switch(struct task_struct *prev)*/ kmap_local_sched_in (); +/* +* Any cached block-layer timestamp (plug->cur_ktime) is stale now, +* invalidate it. +*/ +blk_plug_invalidate_ts (); +fire_sched_in_preempt_notifiers (current);/* * When switching through a kernel thread, the loop in
@@ -7290,12 +7296,10 @@ static inline void sched_submit_work(struct task_struct *tsk)static void sched_update_worker (struct task_struct *tsk) { -if (tsk->flags & (PF_WQ_WORKER |PF_IO_WORKER |PF_BLOCK_TS )) { -if (tsk->flags &PF_BLOCK_TS ) -blk_plug_invalidate_ts (tsk); +if (tsk->flags & (PF_WQ_WORKER |PF_IO_WORKER )) {if (tsk->flags &PF_WQ_WORKER )wq_worker_running (tsk); -else if (tsk->flags &PF_IO_WORKER ) +else io_wq_worker_running (tsk); } }
Signature · 1 line
2.53.0-Meta
Quoted 10 lines · Usama Arif
The details for this are in patch 2. The main reason for this series is to invalidate the cached timestamp on context switch. This was done in
sched_update_worker()only before which was resulting in blk-iocost reading stale timestamps and throttling based on wrong information.Patch 1 is a prerequisite to create the invariant that
PF_BLOCK_TSset impliescurrent->plug!= NULL.[...]
Applied, thanks!
[1/2] kernel/fork: clear PF_BLOCK_TS in copy_process()
commit: fd38b75c4b43295b10d69772a46d1c74dbd6fc81
[2/2] block: invalidate cached plug timestamp after task switch
commit: fad156c2af227f42ca796cbb20ddc354a6dd9932Best regards,
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 11 lines · Jens Axboe
Quoted 9 lines · Usama Arif
The details for this are in patch 2. The main reason for this series is to invalidate the cached timestamp on context switch. This was done in
sched_update_worker()only before which was resulting in blk-iocost reading stale timestamps and throttling based on wrong information.Patch 1 is a prerequisite to create the invariant that
PF_BLOCK_TSset impliescurrent->plug!= NULL.
Quoted 8 lines
Quoted 1 line
[...]
Applied, thanks!
[1/2] kernel/fork: clear
PF_BLOCK_TSincopy_process()commit: fd38b75c4b43295b10d69772a46d1c74dbd6fc81 [2/2] block: invalidate cached plug timestamp after task switch commit: fad156c2af227f42ca796cbb20ddc354a6dd9932
Note: I tentatively queued this on up as a) it looks good to me (and thanks Usama for fixing this!), and b) about to head OOO for a week or so. If Peter or any of the sched people disagree, let me know and we can deal with it. If not, then I plan on sending this in with the usual follow-up merge window fixes next week.
Signature · 1 line
Jens Axboe
Peter Zijlstra
maintainerQuoted 21 lines · Jens Axboe
Quoted 11 lines · Jens Axboe
Quoted 9 lines · Usama Arif
The details for this are in patch 2. The main reason for this series is to invalidate the cached timestamp on context switch. This was done in
sched_update_worker()only before which was resulting in blk-iocost reading stale timestamps and throttling based on wrong information.Patch 1 is a prerequisite to create the invariant that
PF_BLOCK_TSset impliescurrent->plug!= NULL.Quoted 8 lines
Quoted 1 line
[...]
Applied, thanks!
[1/2] kernel/fork: clear
PF_BLOCK_TSincopy_process()commit: fd38b75c4b43295b10d69772a46d1c74dbd6fc81 [2/2] block: invalidate cached plug timestamp after task switch commit: fad156c2af227f42ca796cbb20ddc354a6dd9932
Quoted 5 lines
Note: I tentatively queued this on up as a) it looks good to me (and thanks Usama for fixing this!), and b) about to head OOO for a week or so. If Peter or any of the sched people disagree, let me know and we can deal with it. If not, then I plan on sending this in with the usual follow-up merge window fixes next week.
FWIW, looks good to me.
Jens Axboe
maintainerQuoted 22 lines · Peter Zijlstra
Quoted 21 lines · Jens Axboe
Quoted 19 lines · Jens Axboe
Quoted 10 lines · Usama Arif
The details for this are in patch 2. The main reason for this series is to invalidate the cached timestamp on context switch. This was done in
sched_update_worker()only before which was resulting in blk-iocost reading stale timestamps and throttling based on wrong information.Patch 1 is a prerequisite to create the invariant that
PF_BLOCK_TSset impliescurrent->plug!= NULL.[...]
Applied, thanks!
[1/2] kernel/fork: clear
PF_BLOCK_TSincopy_process()commit: fd38b75c4b43295b10d69772a46d1c74dbd6fc81 [2/2] block: invalidate cached plug timestamp after task switch commit: fad156c2af227f42ca796cbb20ddc354a6dd9932
Quoted 7 lines
Quoted 5 lines
Note: I tentatively queued this on up as a) it looks good to me (and thanks Usama for fixing this!), and b) about to head OOO for a week or so. If Peter or any of the sched people disagree, let me know and we can deal with it. If not, then I plan on sending this in with the usual follow-up merge window fixes next week.
FWIW, looks good to me.
Great, thanks Peter!
Signature · 1 line
Jens Axboe