Skip to contentSkip to conversation

block: invalidate cached plug timestamp on context switch

PATCH Applied Usama Arif 3 participants 7 messages Jun 16

Subsystems (5)
lore
Share & tools
lore

Usama Arif

author
#0 lore raw

cover block: invalidate cached plug timestamp on context switch

The details for this are in patch 2. The main reason for this series is to invalidate the cached timestamp on context switch. This was done in sched_update_worker() only before which was resulting in blk-iocost reading stale timestamps and throttling based on wrong information.

Patch 1 is a prerequisite to create the invariant that PF_BLOCK_TS set implies current->plug != NULL.

v2 -> v3:
https://lore.kernel.org/all/[email protected]/
- Add patch 1 to clear PF_BLOCK_TS in copy_process() so the invariant survives fork.
- Drop the if (plug) NULL check inside blk_plug_invalidate_ts(), relying on the invariant established by patch 1. (Peter Zijlstra)

v1 -> v2:
https://lore.kernel.org/all/[email protected]/
- Move the PF_BLOCK_TS check into blk_plug_invalidate_ts() and upgrade it to __always_inline (Peter Zijlstra).
- Drop the tsk parameter; the helper only ever operates on current.

Usama Arif (2):
kernel/fork: clear PF_BLOCK_TS in copy_process()
block: invalidate cached plug timestamp after task switch

 include/linux/blkdev.h | 16 ++++++----------
 kernel/fork.c          |  1 +
 kernel/sched/core.c    | 12 ++++++++----
 3 files changed, 15 insertions(+), 14 deletions(-)
Signature · 1 line
2.53.0-Meta
author
#1 lore raw

PATCH 1/2 kernel/fork: clear PF_BLOCK_TS in copy_process()

PF_BLOCK_TS is only set in blk_time_get_ns() when current->plug is non-NULL, and blk_finish_plug() clears it via __blk_flush_plug() before NULLing the plug pointer. copy_process() breaks the invariant by inheriting PF_BLOCK_TS from the parent while resetting the child's plug to NULL.

Clear PF_BLOCK_TS alongside that assignment so callers can rely on "PF_BLOCK_TS set implies current->plug != NULL" and dereference current->plug unguarded.

Fixes 06b23f92af87 ("block: update cached timestamp post schedule/preemption")
---
 kernel/fork.c | 1 +
 1 file changed, 1 insertion(+)
kernel/fork.c +1 -0
index 892a95214c54..13e38e89a1f3 100644
@@ -2338,6 +2338,7 @@ __latent_entropy struct task_struct *copy_process(
 
 #ifdef CONFIG_BLOCK
 	p->plug = NULL;
+	p->flags &= ~PF_BLOCK_TS;
 #endif
 	futex_init_task(p);
 
Signature · 1 line
2.53.0-Meta
author
#2 lore raw

PATCH 2/2 block: invalidate cached plug timestamp after task switch

In reply to Usama Arif

blk_time_get_ns() caches ktime_get_ns() in current->plug->cur_ktime and marks the task with PF_BLOCK_TS. That cache is only valid while the task keeps running; if the task is switched out, wall-clock time advances and the cached value must not be reused when the task runs again.

The existing invalidation covers explicit plug flushes through __blk_flush_plug(), and the schedule() / rtmutex paths through sched_update_worker(). It does not cover in-kernel preemption paths such as preempt_schedule(), preempt_schedule_notrace(), and preempt_schedule_irq(), which enter __schedule(SM_PREEMPT) directly and return without calling sched_update_worker().

As a result, a task preempted while holding a plug with PF_BLOCK_TS set can reuse a stale plug->cur_ktime after it is scheduled back in. blk-iocost then consumes that stale timestamp through ioc_now(), producing stale vnow values for throttle decisions, and through ioc_rqos_done(), inflating on-queue time and feeding false missed-QoS samples into vrate adjustment.

Move the schedule-side invalidation to finish_task_switch(), which runs for the scheduled-in task after every actual context switch regardless of which schedule entry point was used. Keep __blk_flush_plug() as the explicit flush/finish-plug invalidation path, and remove only the PF_BLOCK_TS handling from sched_update_worker().

Fixes 06b23f92af87 ("block: update cached timestamp post schedule/preemption")
---
 include/linux/blkdev.h | 16 ++++++----------
 kernel/sched/core.c    | 12 ++++++++----
 2 files changed, 14 insertions(+), 14 deletions(-)
include/linux/blkdev.h +6 -10
index 57e84d59a642..c285a4d9837d 100644
@@ -1216,16 +1216,12 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async)
 		__blk_flush_plug(plug, async);
 }
 
-/*
- * tsk == current here
- */
-static inline void blk_plug_invalidate_ts(struct task_struct *tsk)
+static __always_inline void blk_plug_invalidate_ts(void)
 {
-	struct blk_plug *plug = tsk->plug;
-
-	if (plug)
-		plug->cur_ktime = 0;
-	current->flags &= ~PF_BLOCK_TS;
+	if (unlikely(current->flags & PF_BLOCK_TS)) {
+		current->plug->cur_ktime = 0;
+		current->flags &= ~PF_BLOCK_TS;
+	}
 }
 
 int blkdev_issue_flush(struct block_device *bdev);
@@ -1251,7 +1247,7 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async)
 {
 }
 
-static inline void blk_plug_invalidate_ts(struct task_struct *tsk)
+static inline void blk_plug_invalidate_ts(void)
 {
 }
 
kernel/sched/core.c +8 -4
index 8b791e9e9f67..e97e98c33be5 100644
@@ -5368,6 +5368,12 @@ static struct rq *finish_task_switch(struct task_struct *prev)
 	 */
 	kmap_local_sched_in();
 
+	/*
+	 * Any cached block-layer timestamp (plug->cur_ktime) is stale now,
+	 * invalidate it.
+	 */
+	blk_plug_invalidate_ts();
+
 	fire_sched_in_preempt_notifiers(current);
 	/*
 	 * When switching through a kernel thread, the loop in
@@ -7290,12 +7296,10 @@ static inline void sched_submit_work(struct task_struct *tsk)
 
 static void sched_update_worker(struct task_struct *tsk)
 {
-	if (tsk->flags & (PF_WQ_WORKER | PF_IO_WORKER | PF_BLOCK_TS)) {
-		if (tsk->flags & PF_BLOCK_TS)
-			blk_plug_invalidate_ts(tsk);
+	if (tsk->flags & (PF_WQ_WORKER | PF_IO_WORKER)) {
 		if (tsk->flags & PF_WQ_WORKER)
 			wq_worker_running(tsk);
-		else if (tsk->flags & PF_IO_WORKER)
+		else
 			io_wq_worker_running(tsk);
 	}
 }
Signature · 1 line
2.53.0-Meta
maintainer Applied
#3 lore raw
In reply to Usama Arif
Quoted 10 lines · Usama Arif

The details for this are in patch 2. The main reason for this series is to invalidate the cached timestamp on context switch. This was done in sched_update_worker() only before which was resulting in blk-iocost reading stale timestamps and throttling based on wrong information.

Patch 1 is a prerequisite to create the invariant that PF_BLOCK_TS set implies current->plug != NULL.

[...]

Applied, thanks!

[1/2] kernel/fork: clear PF_BLOCK_TS in copy_process()

      commit: fd38b75c4b43295b10d69772a46d1c74dbd6fc81
[2/2] block: invalidate cached plug timestamp after task switch
      commit: fad156c2af227f42ca796cbb20ddc354a6dd9932

Best regards,

Signature · 1 line
Jens Axboe
maintainer
#4 lore raw
Quoted 11 lines · Jens Axboe
Quoted 9 lines · Usama Arif

The details for this are in patch 2. The main reason for this series is to invalidate the cached timestamp on context switch. This was done in sched_update_worker() only before which was resulting in blk-iocost reading stale timestamps and throttling based on wrong information.

Patch 1 is a prerequisite to create the invariant that PF_BLOCK_TS set implies current->plug != NULL.

Quoted 8 lines
Quoted 1 line

[...]

Applied, thanks!

[1/2] kernel/fork: clear PF_BLOCK_TS in copy_process()

      commit: fd38b75c4b43295b10d69772a46d1c74dbd6fc81
[2/2] block: invalidate cached plug timestamp after task switch
      commit: fad156c2af227f42ca796cbb20ddc354a6dd9932

Note: I tentatively queued this on up as a) it looks good to me (and thanks Usama for fixing this!), and b) about to head OOO for a week or so. If Peter or any of the sched people disagree, let me know and we can deal with it. If not, then I plan on sending this in with the usual follow-up merge window fixes next week.

Signature · 1 line
Jens Axboe
maintainer
#5 lore raw
Quoted 21 lines · Jens Axboe
Quoted 11 lines · Jens Axboe
Quoted 9 lines · Usama Arif

The details for this are in patch 2. The main reason for this series is to invalidate the cached timestamp on context switch. This was done in sched_update_worker() only before which was resulting in blk-iocost reading stale timestamps and throttling based on wrong information.

Patch 1 is a prerequisite to create the invariant that PF_BLOCK_TS set implies current->plug != NULL.

Quoted 8 lines
Quoted 1 line

[...]

Applied, thanks!

[1/2] kernel/fork: clear PF_BLOCK_TS in copy_process()

      commit: fd38b75c4b43295b10d69772a46d1c74dbd6fc81
[2/2] block: invalidate cached plug timestamp after task switch
      commit: fad156c2af227f42ca796cbb20ddc354a6dd9932
Quoted 5 lines

Note: I tentatively queued this on up as a) it looks good to me (and thanks Usama for fixing this!), and b) about to head OOO for a week or so. If Peter or any of the sched people disagree, let me know and we can deal with it. If not, then I plan on sending this in with the usual follow-up merge window fixes next week.

FWIW, looks good to me.

maintainer
#6 lore raw
Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Jens Axboe
Quoted 19 lines · Jens Axboe
Quoted 10 lines · Usama Arif

The details for this are in patch 2. The main reason for this series is to invalidate the cached timestamp on context switch. This was done in sched_update_worker() only before which was resulting in blk-iocost reading stale timestamps and throttling based on wrong information.

Patch 1 is a prerequisite to create the invariant that PF_BLOCK_TS set implies current->plug != NULL.

[...]

Applied, thanks!

[1/2] kernel/fork: clear PF_BLOCK_TS in copy_process()

      commit: fd38b75c4b43295b10d69772a46d1c74dbd6fc81
[2/2] block: invalidate cached plug timestamp after task switch
      commit: fad156c2af227f42ca796cbb20ddc354a6dd9932
Quoted 7 lines
Quoted 5 lines

Note: I tentatively queued this on up as a) it looks good to me (and thanks Usama for fixing this!), and b) about to head OOO for a week or so. If Peter or any of the sched people disagree, let me know and we can deal with it. If not, then I plan on sending this in with the usual follow-up merge window fixes next week.

FWIW, looks good to me.

Great, thanks Peter!

Signature · 1 line
Jens Axboe