block: invalidate cached plug timestamp after task switch
Subsystems (5)
PATCH v2 1/1 block: invalidate cached plug timestamp after task switch
blk_time_get_ns() caches ktime_get_ns() in current->plug->cur_ktime and marks the task with PF_BLOCK_TS. That cache is only valid while the task keeps running; if the task is switched out, wall-clock time advances and the cached value must not be reused when the task runs again.
The existing invalidation covers explicit plug flushes through __blk_flush_plug(), and the schedule() / rtmutex paths through sched_update_worker(). It does not cover in-kernel preemption paths such as preempt_schedule(), preempt_schedule_notrace(), and preempt_schedule_irq(), which enter __schedule(SM_PREEMPT) directly and return without calling sched_update_worker().
As a result, a task preempted while holding a plug with PF_BLOCK_TS set can reuse a stale plug->cur_ktime after it is scheduled back in. blk-iocost then consumes that stale timestamp through ioc_now(), producing stale vnow values for throttle decisions, and through ioc_rqos_done(), inflating on-queue time and feeding false missed-QoS samples into vrate adjustment.
Move the schedule-side invalidation to finish_task_switch(), which runs for the scheduled-in task after every actual context switch regardless of which schedule entry point was used. Keep __blk_flush_plug() as the explicit flush/finish-plug invalidation path, and remove only the PF_BLOCK_TS handling from sched_update_worker().
---
v1 -> v2: https://lore.kernel.org/all/[email protected]/
- Make the function just blk_plug_invalidate_ts(), move the check for PF_BLOCK_TS flag into blk_plug_invalidate_ts and make it __always_inline (Peter Zijlstra).
--- include/linux/blkdev.h | 17 ++++++++--------- kernel/sched/core.c | 12 ++++++++---- 2 files changed, 16 insertions(+), 13 deletions(-)
include/linux/blkdev.h +8 -9
@@ -1216,16 +1216,15 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async)__blk_flush_plug (plug, async); } -/* -* tsk == current here -*/ -static inline void blk_plug_invalidate_ts (struct task_struct *tsk) +static __always_inline void blk_plug_invalidate_ts(void) { -struct blk_plug *plug = tsk->plug; +if (unlikely (current->flags &PF_BLOCK_TS )) { +struct blk_plug *plug = current->plug; -if (plug) - plug->cur_ktime =0 ; - current->flags &= ~PF_BLOCK_TS ; +if (plug) + plug->cur_ktime =0 ; + current->flags &= ~PF_BLOCK_TS ; + } }int blkdev_issue_flush (struct block_device *bdev);
@@ -1251,7 +1250,7 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async) { } -static inline void blk_plug_invalidate_ts (struct task_struct *tsk) +static inline void blk_plug_invalidate_ts (void ) { }
kernel/sched/core.c +8 -4
@@ -5368,6 +5368,12 @@ static struct rq *finish_task_switch(struct task_struct *prev)*/ kmap_local_sched_in (); +/* +* Any cached block-layer timestamp (plug->cur_ktime) is stale now, +* invalidate it. +*/ +blk_plug_invalidate_ts (); +fire_sched_in_preempt_notifiers (current);/* * When switching through a kernel thread, the loop in
@@ -7290,12 +7296,10 @@ static inline void sched_submit_work(struct task_struct *tsk)static void sched_update_worker (struct task_struct *tsk) { -if (tsk->flags & (PF_WQ_WORKER |PF_IO_WORKER |PF_BLOCK_TS )) { -if (tsk->flags &PF_BLOCK_TS ) -blk_plug_invalidate_ts (tsk); +if (tsk->flags & (PF_WQ_WORKER |PF_IO_WORKER )) {if (tsk->flags &PF_WQ_WORKER )wq_worker_running (tsk); -else if (tsk->flags &PF_IO_WORKER ) +else io_wq_worker_running (tsk); } }
Signature · 1 line
2.53.0-Meta
Peter Zijlstra
maintainerinclude/linux/blkdev.h static inline void blk_flush_plug(struct blk_plug *plug, bool async) · line 1228
Quoted 10 lines · Usama Arif
include/linux/blkdev.h +7
-0+static __always_inline void blk_plug_invalidate_ts(void) { +if (unlikely (current->flags &PF_BLOCK_TS )) { +struct blk_plug *plug = current->plug; +if (plug) + plug->cur_ktime =0 ; + current->flags &= ~PF_BLOCK_TS ; + } }
If you can guarantee PF_BLOCK_TS is only ever set when current->plug, this can be reduced further.
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Usama Arif
(quoted diff) +3
-0+static __always_inline void blk_plug_invalidate_ts(void) { + if (unlikely(current->flags & PF_BLOCK_TS)) { + struct blk_plug *plug = current->plug;
Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +4
-0+ if (plug) + plug->cur_ktime = 0; + current->flags &= ~PF_BLOCK_TS; + } }If you can guarantee
PF_BLOCK_TSis only ever set whencurrent->plug, this can be reduced further.
Thanks for the reviews!
The invariant holds at set time (the only set in blk_time_get_ns() is gated by if (!plug)) and through the only legitimate plug clear in blk_finish_plug() (which goes through __blk_flush_plug() that clears PF_BLOCK_TS first).
However, copy_process() sets p->plug = NULL for the child but doesn't strip PF_BLOCK_TS from the inherited flags.
I think the if(plug) is a good defensive check, but can also do the below if you prefer?
include/linux/blkdev.h +1 -4
@@ -1219,10 +1219,7 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async) static __always_inline void blk_plug_invalidate_ts(void) {if (unlikely (current->flags &PF_BLOCK_TS )) { -struct blk_plug *plug = current->plug; - -if (plug) - plug->cur_ktime =0 ; + current->plug->cur_ktime =0 ; current->flags &= ~PF_BLOCK_TS ; } }
kernel/fork.c +2 -1
@@ -2167,7 +2167,8 @@ __latent_entropy struct task_struct *copy_process( goto bad_fork_cleanup_count;delayacct_tsk_init (p);/* Must remain after dup_task_struct() */ - p->flags &= ~(PF_SUPERPRIV |PF_WQ_WORKER |PF_IDLE |PF_NO_SETAFFINITY ); + p->flags &= ~(PF_SUPERPRIV |PF_WQ_WORKER |PF_IDLE |PF_NO_SETAFFINITY | +PF_BLOCK_TS ); p->flags |=PF_FORKNOEXEC ;INIT_LIST_HEAD (&p->children);INIT_LIST_HEAD (&p->sibling);
Peter Zijlstra
maintainerQuoted 26 lines · Usama Arif
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Usama Arif
(quoted diff) +3
-0+static __always_inline void blk_plug_invalidate_ts(void) { + if (unlikely(current->flags & PF_BLOCK_TS)) { + struct blk_plug *plug = current->plug;Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +4
-0+ if (plug) + plug->cur_ktime = 0; + current->flags &= ~PF_BLOCK_TS; + } }If you can guarantee
PF_BLOCK_TSis only ever set whencurrent->plug, this can be reduced further.Thanks for the reviews!
The invariant holds at set time (the only set in
blk_time_get_ns()is gated by if (!plug)) and through the only legitimate plug clear inblk_finish_plug()(which goes through__blk_flush_plug()that clearsPF_BLOCK_TSfirst).
Quoted 5 lines
However,
copy_process()setsp->plug= NULL for the child but doesn't stripPF_BLOCK_TSfrom the inherited flags.I think the if(plug) is a good defensive check, but can also do the below if you prefer?
I think that's worth the extra few lines.
Quoted 27 lines · Peter Zijlstra
Quoted 26 lines · Usama Arif
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Usama Arif
(quoted diff) +3
-0+static __always_inline void blk_plug_invalidate_ts(void) { + if (unlikely(current->flags & PF_BLOCK_TS)) { + struct blk_plug *plug = current->plug;Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +4
-0+ if (plug) + plug->cur_ktime = 0; + current->flags &= ~PF_BLOCK_TS; + } }If you can guarantee
PF_BLOCK_TSis only ever set whencurrent->plug, this can be reduced further.Thanks for the reviews!
The invariant holds at set time (the only set in
blk_time_get_ns()is gated by if (!plug)) and through the only legitimate plug clear inblk_finish_plug()(which goes through__blk_flush_plug()that clearsPF_BLOCK_TSfirst).
Quoted 7 lines
Quoted 5 lines
However,
copy_process()setsp->plug= NULL for the child but doesn't stripPF_BLOCK_TSfrom the inherited flags.I think the if(plug) is a good defensive check, but can also do the below if you prefer?
I think that's worth the extra few lines.
ah sorry didnt understand you completely, by extra lines do you mean keep the if(plug) or get rid of it and clear the flag in fork?
I though more about it and I think its much nicer to get rid of the if(plug) and clear the flag in fork.