Skip to contentSkip to conversation

sched_ext: Introduce rq lock tracking

PATCH Superseded Andrea Righi 2 participants 9 messages Apr 19, 2025 – Apr 20, 2025

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Andrea Righi

author reviewer
#0 lore raw

cover sched_ext: Introduce rq lock tracking

Add rq lock tracking to sched_ext ops callbacks to enable scx_bpf_*() kfuncs to detect whether a specific rq is currently locked and safely operate with it.

If no rq is locked, the target rq lock can be acquired. If a different rq is already locked, the operation can either fail (triggering an ops error), deferred using IRQ work, or managed accordingly to avoid deadlocks.

This patchset is also available in the following git branch:

git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.git scx-rq-lock-tracking

diffstat follows:

Andrea Righi (2):
sched_ext: Track currently locked rq
sched_ext: Fix missing rq lock in scx_bpf_cpuperf_set()

 include/linux/sched/ext.h |   1 +
 kernel/sched/ext.c        | 153 ++++++++++++++++++++++++++++++----------------
 kernel/sched/ext_idle.c   |   2 +-
 3 files changed, 101 insertions(+), 55 deletions(-)
author reviewer
#1 lore raw

PATCH 1/2 sched_ext: Track currently locked rq

Some kfuncs provided by sched_ext may need to operate on a struct rq, but they can be invoked from various contexts, specifically, different scx callbacks.

While some of these callbacks are invoked with a particular rq already locked, others are not. This makes it impossible for a kfunc to reliably determine whether it's safe to access a given rq, triggering potential bugs or unsafe behaviors, see for example [1].

To address this, track the currently locked rq whenever a sched_ext callback is invoked via SCX_CALL_OP*().

This allows kfuncs that need to operate on an arbitrary rq to retrieve the currently locked one and apply the appropriate action as needed.

[1] https://lore.kernel.org/lkml/[email protected]/

---
 include/linux/sched/ext.h |   1 +
 kernel/sched/ext.c        | 126 +++++++++++++++++++++++---------------
 kernel/sched/ext_idle.c   |   2 +-
 3 files changed, 78 insertions(+), 51 deletions(-)
include/linux/sched/ext.h +1 -0
index f7545430a5482..9de57f2a284fd 100644
@@ -149,6 +149,7 @@ struct sched_ext_entity {
 	s32			selected_cpu;
 	u32			kf_mask;	/* see scx_kf_mask above */
 	struct task_struct	*kf_tasks[2];	/* see SCX_CALL_OP_TASK() */
+	struct rq		*locked_rq;	/* currently locked rq */
 	atomic_long_t		ops_state;
 
 	struct list_head	runnable_node;	/* rq->scx.runnable_list */
kernel/sched/ext.c +76 -50

407 lines not on this page · raw message

kernel/sched/ext_idle.c +1 -1
index 023ae6df5e8ca..9213989e72b4e 100644
@@ -745,7 +745,7 @@ void __scx_update_idle(struct rq *rq, bool idle, bool do_notify)
 	 * managed by put_prev_task_idle()/set_next_task_idle().
 	 */
 	if (SCX_HAS_OP(update_idle) && do_notify && !scx_rq_bypassing(rq))
-		SCX_CALL_OP(SCX_KF_REST, update_idle, cpu_of(rq), idle);
+		SCX_CALL_OP(SCX_KF_REST, rq, update_idle, cpu_of(rq), idle);
 
 	/*
 	 * Update the idle masks:
Signature · 1 line
2.49.0
reviewer
#2 lore raw

Re: [PATCH 1/2] sched_ext: Track currently locked rq

Hello, Andrea.

include/linux/sched/ext.h struct sched_ext_entity { · line 152

Quoted 5 lines · Andrea Righi
include/linux/sched/ext.h +1 -0
@@ -149,6 +149,7 @@ struct sched_ext_entity {
 	s32			selected_cpu;
 	u32			kf_mask;	/* see scx_kf_mask above */
 	struct task_struct	*kf_tasks[2];	/* see SCX_CALL_OP_TASK() */
+	struct rq		*locked_rq;	/* currently locked rq */

Can this be a percpu variable? While rq is locked, current can't switch out anyway and that way we don't have to increase the size of task. Note that kf_tasks[] are different in that some ops may, at least theoretically, sleep.

kernel/sched/ext.c static void scx_kf_disallow(u32 mask) · line 1129

Quoted 11 lines
kernel/sched/ext.c +11 -0
+static inline void update_locked_rq(struct rq *rq)
+{
+	/*
+	 * Check whether @rq is actually locked. This can help expose bugs
+	 * or incorrect assumptions about the context in which a kfunc or
+	 * callback is executed.
+	 */
+	if (rq)
+		lockdep_assert_rq_held(rq);
+	current->scx.locked_rq = rq;
+	barrier();

As these conditions are program-order checks on the local CPU, I don't think any barrier is necessary.

Thanks.

Signature · 1 line
tejun
author reviewer
#3 lore raw

Re: [PATCH 1/2] sched_ext: Track currently locked rq

Quoted 13 lines · Tejun Heo

Hello, Andrea.

include/linux/sched/ext.h struct sched_ext_entity { · line 152

Quoted 5 lines · Andrea Righi
include/linux/sched/ext.h +1 -0
@@ -149,6 +149,7 @@ struct sched_ext_entity {
 	s32			selected_cpu;
 	u32			kf_mask;	/* see scx_kf_mask above */
 	struct task_struct	*kf_tasks[2];	/* see SCX_CALL_OP_TASK() */
+	struct rq		*locked_rq;	/* currently locked rq */

Can this be a percpu variable? While rq is locked, current can't switch out anyway and that way we don't have to increase the size of task. Note that kf_tasks[] are different in that some ops may, at least theoretically, sleep.

Yeah, I was debating between using a percpu variable or storing it in current. I went with current just to stay consistent with kf_tasks.

But you're right about not to increasing the size of the task, and as you pointed out, we can’t switch if the rq is locked, so a percpu variable should work. I’ll update that in v2.

Quoted 15 lines

kernel/sched/ext.c static void scx_kf_disallow(u32 mask) · line 1129

Quoted 12 lines
kernel/sched/ext.c +11 -0
+static inline void update_locked_rq(struct rq *rq)
+{
+	/*
+	 * Check whether @rq is actually locked. This can help expose bugs
+	 * or incorrect assumptions about the context in which a kfunc or
+	 * callback is executed.
+	 */
+	if (rq)
+		lockdep_assert_rq_held(rq);
+	current->scx.locked_rq = rq;
+	barrier();

As these conditions are program-order checks on the local CPU, I don't think any barrier is necessary.

Right, these are local CPU access only, I'll drop the barrier.

Thanks,
-Andrea

author reviewer
#4 lore raw

Re: [PATCH 1/2] sched_ext: Track currently locked rq

Quoted 15 lines · Andrea Righi
Quoted 13 lines · Tejun Heo

Hello, Andrea.

include/linux/sched/ext.h struct sched_ext_entity { · line 152

Quoted 5 lines · Andrea Righi
include/linux/sched/ext.h +1 -0
@@ -149,6 +149,7 @@ struct sched_ext_entity {
 	s32			selected_cpu;
 	u32			kf_mask;	/* see scx_kf_mask above */
 	struct task_struct	*kf_tasks[2];	/* see SCX_CALL_OP_TASK() */
+	struct rq		*locked_rq;	/* currently locked rq */

Can this be a percpu variable? While rq is locked, current can't switch out anyway and that way we don't have to increase the size of task. Note that kf_tasks[] are different in that some ops may, at least theoretically, sleep.

Quoted 6 lines

Yeah, I was debating between using a percpu variable or storing it in current. I went with current just to stay consistent with kf_tasks.

But you're right about not to increasing the size of the task, and as you pointed out, we can’t switch if the rq is locked, so a percpu variable should work. I’ll update that in v2.

Hm... actually thinking more about this, a problem with the percpu variable is that, if no rq is locked, we could move to a different CPU and end up reading the wrong rq_locked via scx_locked_rq(). I don't think we want to preempt_disable/enable all the callbacks just to fix this... Maybe storing in current is a safer choice?

Thanks,
-Andrea

author reviewer
#5 lore raw

Re: [PATCH 1/2] sched_ext: Track currently locked rq

Quoted 23 lines · Andrea Righi
Quoted 15 lines · Andrea Righi
Quoted 13 lines · Tejun Heo

Hello, Andrea.

include/linux/sched/ext.h struct sched_ext_entity { · line 152

Quoted 5 lines · Andrea Righi
include/linux/sched/ext.h +1 -0
@@ -149,6 +149,7 @@ struct sched_ext_entity {
 	s32			selected_cpu;
 	u32			kf_mask;	/* see scx_kf_mask above */
 	struct task_struct	*kf_tasks[2];	/* see SCX_CALL_OP_TASK() */
+	struct rq		*locked_rq;	/* currently locked rq */

Can this be a percpu variable? While rq is locked, current can't switch out anyway and that way we don't have to increase the size of task. Note that kf_tasks[] are different in that some ops may, at least theoretically, sleep.

Quoted 6 lines

Yeah, I was debating between using a percpu variable or storing it in current. I went with current just to stay consistent with kf_tasks.

But you're right about not to increasing the size of the task, and as you pointed out, we can’t switch if the rq is locked, so a percpu variable should work. I’ll update that in v2.

Quoted 5 lines

Hm... actually thinking more about this, a problem with the percpu variable is that, if no rq is locked, we could move to a different CPU and end up reading the wrong rq_locked via scx_locked_rq(). I don't think we want to preempt_disable/enable all the callbacks just to fix this... Maybe storing in current is a safer choice?

And if we don't want to increase the size of sched_ext_entity, we could store the cpu of the currently locked rq, right before "disallow", like:

struct sched_ext_entity {

	struct scx_dispatch_q *    dsq;                  /*     0     8 */
	struct scx_dsq_list_node   dsq_list;             /*     8    24 */
	struct rb_node             dsq_priq __attribute__((__aligned__(8))); /*    32    24 */
	u32                        dsq_seq;              /*    56     4 */
	u32                        dsq_flags;            /*    60     4 */

/* --- cacheline 1 boundary (64 bytes) --- */
u32 flags; /* 64 4 */
u32 weight; /* 68 4 */
s32 sticky_cpu; /* 72 4 */
s32 holding_cpu; /* 76 4 */
s32 selected_cpu; /* 80 4 */
u32 kf_mask; /* 84 4 */
struct task_struct * kf_tasks[2]; /* 88 16 */
atomic_long_t ops_state; /* 104 8 */
struct list_head runnable_node; /* 112 16 */
/* --- cacheline 2 boundary (128 bytes) --- */
long unsigned int runnable_at; /* 128 8 */
u64 core_sched_at; /* 136 8 */
u64 ddsp_dsq_id; /* 144 8 */
u64 ddsp_enq_flags; /* 152 8 */
u64 slice; /* 160 8 */
u64 dsq_vtime; /* 168 8 */
int locked_cpu; /* 176 4 */
bool disallow; /* 180 1 */

/* XXX 3 bytes hole, try to pack */

struct cgroup * cgrp_moving_from; /* 184 8 */
/* --- cacheline 3 boundary (192 bytes) --- */
struct list_head tasks_node; /* 192 16 */

/* size: 208, cachelines: 4, members: 24 */ /* sum members: 205, holes: 1, sum holes: 3 */ /* forced alignments: 1 */
/* last cacheline: 16 bytes */
} __attribute__((__aligned__(8)));

(before the hole was 7 bytes)

Then use cpu_rq()/cpu_of() to resolve that to/from the corresponding rq.

-Andrea

reviewer
#6 lore raw

Re: [PATCH 1/2] sched_ext: Track currently locked rq

In reply to Andrea Righi

Hello,

Quoted 5 lines · Andrea Righi

Hm... actually thinking more about this, a problem with the percpu variable is that, if no rq is locked, we could move to a different CPU and end up reading the wrong rq_locked via scx_locked_rq(). I don't think we want to preempt_disable/enable all the callbacks just to fix this... Maybe storing in current is a safer choice?

Hmm... I have a hard time imagining a timeline of events which would lead to the wrong conclusion. The percpu variable is set only while an rq is locked and cleared before the rq lock is released and thus can only be read as non-NULL while the rq is locked by that CPU. Maybe I'm missing something. Can you illustrate a timeline of events which would lead to the wrong conclusion?

Thanks.

Signature · 1 line
tejun
author reviewer
#7 lore raw

Re: [PATCH 1/2] sched_ext: Track currently locked rq

Quoted 9 lines · Tejun Heo

Hello,

Quoted 5 lines · Andrea Righi

Hm... actually thinking more about this, a problem with the percpu variable is that, if no rq is locked, we could move to a different CPU and end up reading the wrong rq_locked via scx_locked_rq(). I don't think we want to preempt_disable/enable all the callbacks just to fix this... Maybe storing in current is a safer choice?

Quoted 6 lines

Hmm... I have a hard time imagining a timeline of events which would lead to the wrong conclusion. The percpu variable is set only while an rq is locked and cleared before the rq lock is released and thus can only be read as non-NULL while the rq is locked by that CPU. Maybe I'm missing something. Can you illustrate a timeline of events which would lead to the wrong conclusion?

Oh ok, I was only thinking of setting the percpu variable when we call SCX_CALL_OP*(), but if we clear it after the scx op returns, then it should work. If no rq is locked and we bounce to a different CPU, we'd still read NULL, so it should be always correct.

Alright, I'll send a v2 with this logic and the percpu variable.

Thanks,
-Andrea

author reviewer
#8 lore raw

PATCH 2/2 sched_ext: Fix missing rq lock in scx_bpf_cpuperf_set()

In reply to Andrea Righi

scx_bpf_cpuperf_set() can be used to set a performance target level on any CPU. However, it doesn't correctly acquire the corresponding rq lock, which may lead to unsafe behavior and trigger the following warning, due to the lockdep_assert_rq_held() check:

[   51.713737] WARNING: CPU: 3 PID: 3899 at kernel/sched/sched.h:1512 scx_bpf_cpuperf_set+0x1a0/0x1e0
...
[   51.713836] Call trace:
[   51.713837]  scx_bpf_cpuperf_set+0x1a0/0x1e0 (P)
[   51.713839]  bpf_prog_62d35beb9301601f_bpfland_init+0x168/0x440
[   51.713841]  bpf__sched_ext_ops_init+0x54/0x8c
[   51.713843]  scx_ops_enable.constprop.0+0x2c0/0x10f0
[   51.713845]  bpf_scx_reg+0x18/0x30
[   51.713847]  bpf_struct_ops_link_create+0x154/0x1b0
[   51.713849]  __sys_bpf+0x1934/0x22a0

Fix by properly acquiring the rq lock when possible or raising an error if we try to operate on a CPU that is not the one currently locked.

Fixes d86adb4fc0655 ("sched_ext: Add cpuperf support")
---
 kernel/sched/ext.c | 27 +++++++++++++++++++++++----
 1 file changed, 23 insertions(+), 4 deletions(-)
kernel/sched/ext.c +23 -4
index a14a5c3bc38ac..eddd84731c019 100644
@@ -7078,13 +7078,32 @@ __bpf_kfunc void scx_bpf_cpuperf_set(s32 cpu, u32 perf)
 	}
 
 	if (ops_cpu_valid(cpu, NULL)) {
-		struct rq *rq = cpu_rq(cpu);
+		struct rq *rq = cpu_rq(cpu), *locked_rq = scx_locked_rq();
+		struct rq_flags rf;
+
+		/*
+		 * When called with an rq lock held, restrict the operation
+		 * to the corresponding CPU to prevent ABBA deadlocks.
+		 */
+		if (locked_rq && rq != locked_rq) {
+			scx_error("Invalid target CPU %d", cpu);
+			return;
+		}
+
+		/*
+		 * If no rq lock is held, allow to operate on any CPU by
+		 * acquiring the corresponding rq lock.
+		 */
+		if (!locked_rq) {
+			rq_lock_irqsave(rq, &rf);
+			update_rq_clock(rq);
+		}
 
 		rq->scx.cpuperf_target = perf;
+		cpufreq_update_util(rq, 0);
 
-		rcu_read_lock_sched_notrace();
-		cpufreq_update_util(cpu_rq(cpu), 0);
-		rcu_read_unlock_sched_notrace();
+		if (!locked_rq)
+			rq_unlock_irqrestore(rq, &rf);
 	}
 }
 
Signature · 1 line
2.49.0