Skip to contentSkip to conversation

sched_ext: pass the initial cpu.idle state in scx_cgroup_init_args

PATCH Superseded Tao Cui 3 participants 5 messages Aug 24 – Aug 25

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Tao Cui

author
#0 lore raw

PATCH v2 1/1 sched_ext: pass the initial cpu.idle state in scx_cgroup_init_args

From: Tao Cui <[email protected]>

scx_cgroup_init_args carries the initial weight and bandwidth control parameters of a cgroup to ops.cgroup_init(), but not its cpu.idle state. A cgroup that was already configured idle before the scheduler was loaded (or before it was onlined under it) is presented as non-idle, and the BPF scheduler only learns about it if cpu.idle is written again later.

Add the idle state to scx_cgroup_init_args and fill it in all four places that build the args: scx_tg_online() for cgroups onlined under the scheduler, scx_cgroup_init() for cgroups that already exist when the scheduler is loaded, and the sub-scheduler handover paths scx_cgroup_claim_subtree() and scx_cgroup_return_subtree().

Verified in a VM with a probe scheduler printing the init args: a cgroup configured cpu.idle=1 before loading shows idle=1 in ops.cgroup_init(), the default shows 0, and later cpu.idle writes still come through ops.cgroup_set_idle(). The sub-scheduler paths are compile-tested only.

---
v1 -> v2: Also fill .idle in the sub-scheduler handover paths in sub.c, scx_cgroup_claim_subtree() and scx_cgroup_return_subtree(), missed in v1 and pointed out by Andrea Righi and the sashiko AI review bot.

v1: https://lore.kernel.org/r/[email protected]

 kernel/sched/ext/ext.c      | 4 +++-
 kernel/sched/ext/internal.h | 3 +++
 kernel/sched/ext/sub.c      | 2 ++
 3 files changed, 8 insertions(+), 1 deletion(-)
kernel/sched/ext/ext.c +3 -1
index b646711a45fe..a7218314dcfb 100644
@@ -4764,7 +4764,8 @@ int scx_tg_online(struct task_group *tg)
 				{ .weight = tg->scx.weight,
 				  .bw_period_us = tg->scx.bw_period_us,
 				  .bw_quota_us = tg->scx.bw_quota_us,
-				  .bw_burst_us = tg->scx.bw_burst_us };
+				  .bw_burst_us = tg->scx.bw_burst_us,
+				  .idle = tg->scx.idle };
 
 			ret = SCX_CALL_OP_RET(sch, cgroup_init,
 					      NULL, tg->css.cgroup, &args);
@@ -5185,6 +5186,7 @@ static int scx_cgroup_init(struct scx_sched *sch)
 				.bw_period_us = tg->scx.bw_period_us,
 				.bw_quota_us = tg->scx.bw_quota_us,
 				.bw_burst_us = tg->scx.bw_burst_us,
+				.idle = tg->scx.idle,
 			};
 
 			ret = SCX_CALL_OP_RET(sch, cgroup_init, NULL, css->cgroup, &args);
kernel/sched/ext/internal.h +3 -0
index 53e136a47924..aa149a9c29f7 100644
@@ -259,6 +259,9 @@ struct scx_cgroup_init_args {
 	u64			bw_period_us;
 	u64			bw_quota_us;
 	u64			bw_burst_us;
+
+	/* whether the cgroup is configured idle via cpu.idle */
+	bool			idle;
 };
 
 enum scx_cpu_preempt_reason {
kernel/sched/ext/sub.c +2 -0
index 0554448835bd..9701bd1d8ea0 100644
@@ -1361,6 +1361,7 @@ static s32 scx_cgroup_claim_subtree(struct scx_sched *sch)
 			.bw_period_us = tg->scx.bw_period_us,
 			.bw_quota_us = tg->scx.bw_quota_us,
 			.bw_burst_us = tg->scx.bw_burst_us,
+			.idle = tg->scx.idle,
 		};
 
 		if (tg->scx.sched != parent ||
@@ -1464,6 +1465,7 @@ static void scx_cgroup_return_subtree(struct scx_sched *sch)
 			.bw_period_us = tg->scx.bw_period_us,
 			.bw_quota_us = tg->scx.bw_quota_us,
 			.bw_burst_us = tg->scx.bw_burst_us,
+			.idle = tg->scx.idle,
 		};
 
 		/* the first pass must have transferred everything */
Signature · 1 line
2.43.0
reviewer Reviewed-by on v2 1/1
#1 lore raw

Hi Tao,

Quoted 15 lines · Tao Cui

From: Tao Cui <[email protected]>

scx_cgroup_init_args carries the initial weight and bandwidth control parameters of a cgroup to ops.cgroup_init(), but not its cpu.idle state. A cgroup that was already configured idle before the scheduler was loaded (or before it was onlined under it) is presented as non-idle, and the BPF scheduler only learns about it if cpu.idle is written again later.

Add the idle state to scx_cgroup_init_args and fill it in all four places that build the args: scx_tg_online() for cgroups onlined under the scheduler, scx_cgroup_init() for cgroups that already exist when the scheduler is loaded, and the sub-scheduler handover paths scx_cgroup_claim_subtree() and scx_cgroup_return_subtree().

Quoted 7 lines

Verified in a VM with a probe scheduler printing the init args: a cgroup configured cpu.idle=1 before loading shows idle=1 in ops.cgroup_init(), the default shows 0, and later cpu.idle writes still come through ops.cgroup_set_idle(). The sub-scheduler paths are compile-tested only.

We should probably add:

Fixes 347ed2d566da ("sched/ext: Implement cgroup_set_idle() callback"

With that:

Thanks,
-Andrea

Quoted 72 lines

---
v1 -> v2: Also fill .idle in the sub-scheduler handover paths in sub.c, scx_cgroup_claim_subtree() and scx_cgroup_return_subtree(), missed in v1 and pointed out by Andrea Righi and the sashiko AI review bot.

v1: https://lore.kernel.org/r/[email protected]

 kernel/sched/ext/ext.c      | 4 +++-
 kernel/sched/ext/internal.h | 3 +++
 kernel/sched/ext/sub.c      | 2 ++
 3 files changed, 8 insertions(+), 1 deletion(-)
kernel/sched/ext/ext.c +3 -1
index b646711a45fe..a7218314dcfb 100644
@@ -4764,7 +4764,8 @@ int scx_tg_online(struct task_group *tg)
 				{ .weight = tg->scx.weight,
 				  .bw_period_us = tg->scx.bw_period_us,
 				  .bw_quota_us = tg->scx.bw_quota_us,
-				  .bw_burst_us = tg->scx.bw_burst_us };
+				  .bw_burst_us = tg->scx.bw_burst_us,
+				  .idle = tg->scx.idle };
 
 			ret = SCX_CALL_OP_RET(sch, cgroup_init,
 					      NULL, tg->css.cgroup, &args);
@@ -5185,6 +5186,7 @@ static int scx_cgroup_init(struct scx_sched *sch)
 				.bw_period_us = tg->scx.bw_period_us,
 				.bw_quota_us = tg->scx.bw_quota_us,
 				.bw_burst_us = tg->scx.bw_burst_us,
+				.idle = tg->scx.idle,
 			};
 
 			ret = SCX_CALL_OP_RET(sch, cgroup_init, NULL, css->cgroup, &args);
kernel/sched/ext/internal.h +3 -0
index 53e136a47924..aa149a9c29f7 100644
@@ -259,6 +259,9 @@ struct scx_cgroup_init_args {
 	u64			bw_period_us;
 	u64			bw_quota_us;
 	u64			bw_burst_us;
+
+	/* whether the cgroup is configured idle via cpu.idle */
+	bool			idle;
 };
 
 enum scx_cpu_preempt_reason {
kernel/sched/ext/sub.c +2 -0
index 0554448835bd..9701bd1d8ea0 100644
@@ -1361,6 +1361,7 @@ static s32 scx_cgroup_claim_subtree(struct scx_sched *sch)
 			.bw_period_us = tg->scx.bw_period_us,
 			.bw_quota_us = tg->scx.bw_quota_us,
 			.bw_burst_us = tg->scx.bw_burst_us,
+			.idle = tg->scx.idle,
 		};
 
 		if (tg->scx.sched != parent ||
@@ -1464,6 +1465,7 @@ static void scx_cgroup_return_subtree(struct scx_sched *sch)
 			.bw_period_us = tg->scx.bw_period_us,
 			.bw_quota_us = tg->scx.bw_quota_us,
 			.bw_burst_us = tg->scx.bw_burst_us,
+			.idle = tg->scx.idle,
 		};
 
 		/* the first pass must have transferred everything */
Signature · 1 line
2.43.0
reviewer
#3 lore raw
In reply to Tao Cui

Hello, Tao.

kernel/sched/ext/internal.h struct scx_cgroup_init_args { · line 264

Quoted 3 lines · Tao Cui
kernel/sched/ext/internal.h +3 -0
+
+	/* whether the cgroup is configured idle via cpu.idle */
+	bool			idle;

In sched_ext, a bare "idle" reads as CPU idle state (ops.update_idle(), idle cpumasks, scx_bpf_pick_idle_cpu()). cpu.idle is the cgroup analog of the SCHED_IDLE policy, so please name the field sched_idle instead. Renaming tg->scx.idle to match would be nice too.

Thanks.

Signature · 1 line
tejun
author
#2 lore raw
In reply to Andrea Righi

Hello, Andrea,

Quoted 26 lines · Andrea Righi

Hi Tao,

Quoted 15 lines · Tao Cui

From: Tao Cui <[email protected]>

scx_cgroup_init_args carries the initial weight and bandwidth control parameters of a cgroup to ops.cgroup_init(), but not its cpu.idle state. A cgroup that was already configured idle before the scheduler was loaded (or before it was onlined under it) is presented as non-idle, and the BPF scheduler only learns about it if cpu.idle is written again later.

Add the idle state to scx_cgroup_init_args and fill it in all four places that build the args: scx_tg_online() for cgroups onlined under the scheduler, scx_cgroup_init() for cgroups that already exist when the scheduler is loaded, and the sub-scheduler handover paths scx_cgroup_claim_subtree() and scx_cgroup_return_subtree().

Quoted 7 lines

Verified in a VM with a probe scheduler printing the init args: a cgroup configured cpu.idle=1 before loading shows idle=1 in ops.cgroup_init(), the default shows 0, and later cpu.idle writes still come through ops.cgroup_set_idle(). The sub-scheduler paths are compile-tested only.

Quoted 8 lines

We should probably add:

Fixes 347ed2d566da ("sched/ext: Implement cgroup_set_idle() callback"

With that:

Thank you for the review, and for catching the sub.c handover paths earlier. v3 adds the Fixes: tag you suggested and carries your Reviewed-by. The field is also renamed to sched_idle per Tejun's review; the rename is mechanical, so I kept the tag, but I'm happy to wait for a re-review if you prefer.

Thanks,
Tao

Quoted 75 lines

Thanks,
-Andrea

Quoted 72 lines

---
v1 -> v2: Also fill .idle in the sub-scheduler handover paths in sub.c, scx_cgroup_claim_subtree() and scx_cgroup_return_subtree(), missed in v1 and pointed out by Andrea Righi and the sashiko AI review bot.

v1: https://lore.kernel.org/r/[email protected]

 kernel/sched/ext/ext.c      | 4 +++-
 kernel/sched/ext/internal.h | 3 +++
 kernel/sched/ext/sub.c      | 2 ++
 3 files changed, 8 insertions(+), 1 deletion(-)
kernel/sched/ext/ext.c +3 -1
index b646711a45fe..a7218314dcfb 100644
@@ -4764,7 +4764,8 @@ int scx_tg_online(struct task_group *tg)
 				{ .weight = tg->scx.weight,
 				  .bw_period_us = tg->scx.bw_period_us,
 				  .bw_quota_us = tg->scx.bw_quota_us,
-				  .bw_burst_us = tg->scx.bw_burst_us };
+				  .bw_burst_us = tg->scx.bw_burst_us,
+				  .idle = tg->scx.idle };
 
 			ret = SCX_CALL_OP_RET(sch, cgroup_init,
 					      NULL, tg->css.cgroup, &args);
@@ -5185,6 +5186,7 @@ static int scx_cgroup_init(struct scx_sched *sch)
 				.bw_period_us = tg->scx.bw_period_us,
 				.bw_quota_us = tg->scx.bw_quota_us,
 				.bw_burst_us = tg->scx.bw_burst_us,
+				.idle = tg->scx.idle,
 			};
 
 			ret = SCX_CALL_OP_RET(sch, cgroup_init, NULL, css->cgroup, &args);
kernel/sched/ext/internal.h +3 -0
index 53e136a47924..aa149a9c29f7 100644
@@ -259,6 +259,9 @@ struct scx_cgroup_init_args {
 	u64			bw_period_us;
 	u64			bw_quota_us;
 	u64			bw_burst_us;
+
+	/* whether the cgroup is configured idle via cpu.idle */
+	bool			idle;
 };
 
 enum scx_cpu_preempt_reason {
kernel/sched/ext/sub.c +2 -0
index 0554448835bd..9701bd1d8ea0 100644
@@ -1361,6 +1361,7 @@ static s32 scx_cgroup_claim_subtree(struct scx_sched *sch)
 			.bw_period_us = tg->scx.bw_period_us,
 			.bw_quota_us = tg->scx.bw_quota_us,
 			.bw_burst_us = tg->scx.bw_burst_us,
+			.idle = tg->scx.idle,
 		};
 
 		if (tg->scx.sched != parent ||
@@ -1464,6 +1465,7 @@ static void scx_cgroup_return_subtree(struct scx_sched *sch)
 			.bw_period_us = tg->scx.bw_period_us,
 			.bw_quota_us = tg->scx.bw_quota_us,
 			.bw_burst_us = tg->scx.bw_burst_us,
+			.idle = tg->scx.idle,
 		};
 
 		/* the first pass must have transferred everything */
Signature · 1 line
2.43.0
author
#4 lore raw
In reply to Tejun Heo

Hello, Tejun,

Quoted 12 lines · Tejun Heo

Hello, Tao.

kernel/sched/ext/internal.h struct scx_cgroup_init_args { · line 264

Quoted 3 lines · Tao Cui
kernel/sched/ext/internal.h +3 -0
+
+	/* whether the cgroup is configured idle via cpu.idle */
+	bool			idle;

In sched_ext, a bare "idle" reads as CPU idle state (ops.update_idle(), idle cpumasks, scx_bpf_pick_idle_cpu()). cpu.idle is the cgroup analog of the SCHED_IDLE policy, so please name the field sched_idle instead. Renaming tg->scx.idle to match would be nice too.

Thanks for the review. Both are renamed in v3: the new scx_cgroup_init_args field, and tg->scx.idle -> tg->scx.sched_idle as a separate second patch in the series, so 1/2 stays minimal for stable backport.

Thanks,
Tao

Quoted 2 lines

Thanks.