Skip to contentSkip to conversation

block: invalidate cached plug timestamp after task switch

PATCH In review Usama Arif 2 participants 5 messages Jun 12

Subsystems (5)
lore
Share & tools
lore

Usama Arif

author
#0 lore raw

PATCH v2 1/1 block: invalidate cached plug timestamp after task switch

blk_time_get_ns() caches ktime_get_ns() in current->plug->cur_ktime and marks the task with PF_BLOCK_TS. That cache is only valid while the task keeps running; if the task is switched out, wall-clock time advances and the cached value must not be reused when the task runs again.

The existing invalidation covers explicit plug flushes through __blk_flush_plug(), and the schedule() / rtmutex paths through sched_update_worker(). It does not cover in-kernel preemption paths such as preempt_schedule(), preempt_schedule_notrace(), and preempt_schedule_irq(), which enter __schedule(SM_PREEMPT) directly and return without calling sched_update_worker().

As a result, a task preempted while holding a plug with PF_BLOCK_TS set can reuse a stale plug->cur_ktime after it is scheduled back in. blk-iocost then consumes that stale timestamp through ioc_now(), producing stale vnow values for throttle decisions, and through ioc_rqos_done(), inflating on-queue time and feeding false missed-QoS samples into vrate adjustment.

Move the schedule-side invalidation to finish_task_switch(), which runs for the scheduled-in task after every actual context switch regardless of which schedule entry point was used. Keep __blk_flush_plug() as the explicit flush/finish-plug invalidation path, and remove only the PF_BLOCK_TS handling from sched_update_worker().

Fixes 06b23f92af87 ("block: update cached timestamp post schedule/preemption")

---
v1 -> v2: https://lore.kernel.org/all/[email protected]/
- Make the function just blk_plug_invalidate_ts(), move the check for PF_BLOCK_TS flag into blk_plug_invalidate_ts and make it __always_inline (Peter Zijlstra).

---
 include/linux/blkdev.h | 17 ++++++++---------
 kernel/sched/core.c    | 12 ++++++++----
 2 files changed, 16 insertions(+), 13 deletions(-)
include/linux/blkdev.h +8 -9
index 57e84d59a642..1c1fd31ce187 100644
@@ -1216,16 +1216,15 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async)
 		__blk_flush_plug(plug, async);
 }
 
-/*
- * tsk == current here
- */
-static inline void blk_plug_invalidate_ts(struct task_struct *tsk)
+static __always_inline void blk_plug_invalidate_ts(void)
 {
-	struct blk_plug *plug = tsk->plug;
+	if (unlikely(current->flags & PF_BLOCK_TS)) {
+		struct blk_plug *plug = current->plug;
 
-	if (plug)
-		plug->cur_ktime = 0;
-	current->flags &= ~PF_BLOCK_TS;
+		if (plug)
+			plug->cur_ktime = 0;
+		current->flags &= ~PF_BLOCK_TS;
+	}
 }
 
 int blkdev_issue_flush(struct block_device *bdev);
@@ -1251,7 +1250,7 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async)
 {
 }
 
-static inline void blk_plug_invalidate_ts(struct task_struct *tsk)
+static inline void blk_plug_invalidate_ts(void)
 {
 }
 
kernel/sched/core.c +8 -4
index 8b791e9e9f67..e97e98c33be5 100644
@@ -5368,6 +5368,12 @@ static struct rq *finish_task_switch(struct task_struct *prev)
 	 */
 	kmap_local_sched_in();
 
+	/*
+	 * Any cached block-layer timestamp (plug->cur_ktime) is stale now,
+	 * invalidate it.
+	 */
+	blk_plug_invalidate_ts();
+
 	fire_sched_in_preempt_notifiers(current);
 	/*
 	 * When switching through a kernel thread, the loop in
@@ -7290,12 +7296,10 @@ static inline void sched_submit_work(struct task_struct *tsk)
 
 static void sched_update_worker(struct task_struct *tsk)
 {
-	if (tsk->flags & (PF_WQ_WORKER | PF_IO_WORKER | PF_BLOCK_TS)) {
-		if (tsk->flags & PF_BLOCK_TS)
-			blk_plug_invalidate_ts(tsk);
+	if (tsk->flags & (PF_WQ_WORKER | PF_IO_WORKER)) {
 		if (tsk->flags & PF_WQ_WORKER)
 			wq_worker_running(tsk);
-		else if (tsk->flags & PF_IO_WORKER)
+		else
 			io_wq_worker_running(tsk);
 	}
 }
Signature · 1 line
2.53.0-Meta
maintainer
#1 lore raw

include/linux/blkdev.h static inline void blk_flush_plug(struct blk_plug *plug, bool async) · line 1228

Quoted 10 lines · Usama Arif
include/linux/blkdev.h +7 -0
+static __always_inline void blk_plug_invalidate_ts(void)
 {
+	if (unlikely(current->flags & PF_BLOCK_TS)) {
+		struct blk_plug *plug = current->plug;
 
+		if (plug)
+			plug->cur_ktime = 0;
+		current->flags &= ~PF_BLOCK_TS;
+	}
 }

If you can guarantee PF_BLOCK_TS is only ever set when current->plug, this can be reduced further.

author
#2 lore raw
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Usama Arif
(quoted diff) +3 -0
+static __always_inline void blk_plug_invalidate_ts(void)
 {
+	if (unlikely(current->flags & PF_BLOCK_TS)) {
+		struct blk_plug *plug = current->plug;
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +4 -0
+		if (plug)
+			plug->cur_ktime = 0;
+		current->flags &= ~PF_BLOCK_TS;
+	}
 }

If you can guarantee PF_BLOCK_TS is only ever set when current->plug, this can be reduced further.

Thanks for the reviews!

The invariant holds at set time (the only set in blk_time_get_ns() is gated by if (!plug)) and through the only legitimate plug clear in blk_finish_plug() (which goes through __blk_flush_plug() that clears PF_BLOCK_TS first).

However, copy_process() sets p->plug = NULL for the child but doesn't strip PF_BLOCK_TS from the inherited flags.

I think the if(plug) is a good defensive check, but can also do the below if you prefer?

include/linux/blkdev.h +1 -4
index 1c1fd31ce187..c285a4d9837d 100644
@@ -1219,10 +1219,7 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async)
 static __always_inline void blk_plug_invalidate_ts(void)
 {
        if (unlikely(current->flags & PF_BLOCK_TS)) {
-               struct blk_plug *plug = current->plug;
-
-               if (plug)
-                       plug->cur_ktime = 0;
+               current->plug->cur_ktime = 0;
                current->flags &= ~PF_BLOCK_TS;
        }
 }
kernel/fork.c +2 -1
index 892a95214c54..9a062149e0d8 100644
@@ -2167,7 +2167,8 @@ __latent_entropy struct task_struct *copy_process(
                goto bad_fork_cleanup_count;
 
        delayacct_tsk_init(p);  /* Must remain after dup_task_struct() */
-       p->flags &= ~(PF_SUPERPRIV | PF_WQ_WORKER | PF_IDLE | PF_NO_SETAFFINITY);
+       p->flags &= ~(PF_SUPERPRIV | PF_WQ_WORKER | PF_IDLE | PF_NO_SETAFFINITY |
+                     PF_BLOCK_TS);
        p->flags |= PF_FORKNOEXEC;
        INIT_LIST_HEAD(&p->children);
        INIT_LIST_HEAD(&p->sibling);
maintainer
#3 lore raw
Quoted 26 lines · Usama Arif
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Usama Arif
(quoted diff) +3 -0
+static __always_inline void blk_plug_invalidate_ts(void)
 {
+	if (unlikely(current->flags & PF_BLOCK_TS)) {
+		struct blk_plug *plug = current->plug;
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +4 -0
+		if (plug)
+			plug->cur_ktime = 0;
+		current->flags &= ~PF_BLOCK_TS;
+	}
 }

If you can guarantee PF_BLOCK_TS is only ever set when current->plug, this can be reduced further.

Thanks for the reviews!

The invariant holds at set time (the only set in blk_time_get_ns() is gated by if (!plug)) and through the only legitimate plug clear in blk_finish_plug() (which goes through __blk_flush_plug() that clears PF_BLOCK_TS first).

Quoted 5 lines

However, copy_process() sets p->plug = NULL for the child but doesn't strip PF_BLOCK_TS from the inherited flags.

I think the if(plug) is a good defensive check, but can also do the below if you prefer?

I think that's worth the extra few lines.

author
#4 lore raw
Quoted 27 lines · Peter Zijlstra
Quoted 26 lines · Usama Arif
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Usama Arif
(quoted diff) +3 -0
+static __always_inline void blk_plug_invalidate_ts(void)
 {
+	if (unlikely(current->flags & PF_BLOCK_TS)) {
+		struct blk_plug *plug = current->plug;
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +4 -0
+		if (plug)
+			plug->cur_ktime = 0;
+		current->flags &= ~PF_BLOCK_TS;
+	}
 }

If you can guarantee PF_BLOCK_TS is only ever set when current->plug, this can be reduced further.

Thanks for the reviews!

The invariant holds at set time (the only set in blk_time_get_ns() is gated by if (!plug)) and through the only legitimate plug clear in blk_finish_plug() (which goes through __blk_flush_plug() that clears PF_BLOCK_TS first).

Quoted 7 lines
Quoted 5 lines

However, copy_process() sets p->plug = NULL for the child but doesn't strip PF_BLOCK_TS from the inherited flags.

I think the if(plug) is a good defensive check, but can also do the below if you prefer?

I think that's worth the extra few lines.

ah sorry didnt understand you completely, by extra lines do you mean keep the if(plug) or get rid of it and clear the flag in fork?

I though more about it and I think its much nicer to get rid of the if(plug) and clear the flag in fork.