Skip to contentSkip to conversation

sched_ext: pass the initial cpu.idle state in scx_cgroup_init_args

PATCH Superseded Tao Cui 3 participants 4 messages Aug 24

Subsystems (3)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Tao Cui

author
#0 lore raw

PATCH 1/1 sched_ext: pass the initial cpu.idle state in scx_cgroup_init_args

From: Tao Cui <[email protected]>

scx_cgroup_init_args carries the initial weight and bandwidth control parameters of a cgroup to ops.cgroup_init(), but not its cpu.idle state. A cgroup that was already configured idle before the scheduler was loaded (or before it was onlined under it) is presented as non-idle, and the BPF scheduler only learns about it if cpu.idle is written again later.

Add the idle state to scx_cgroup_init_args and fill it in both places that build the args -- scx_tg_online() for cgroups created under the scheduler and scx_cgroup_init() for cgroups that already exist when the scheduler is loaded.

Verified in a VM with a probe scheduler printing the init args: a cgroup configured cpu.idle=1 before loading shows idle=1 in ops.cgroup_init(), the default shows 0, and later cpu.idle writes still come through ops.cgroup_set_idle().

---
 kernel/sched/ext/ext.c      | 4 +++-
 kernel/sched/ext/internal.h | 3 +++
 2 files changed, 6 insertions(+), 1 deletion(-)
kernel/sched/ext/ext.c +3 -1
index b646711a45fe..a7218314dcfb 100644
@@ -4764,7 +4764,8 @@ int scx_tg_online(struct task_group *tg)
 				{ .weight = tg->scx.weight,
 				  .bw_period_us = tg->scx.bw_period_us,
 				  .bw_quota_us = tg->scx.bw_quota_us,
-				  .bw_burst_us = tg->scx.bw_burst_us };
+				  .bw_burst_us = tg->scx.bw_burst_us,
+				  .idle = tg->scx.idle };
 
 			ret = SCX_CALL_OP_RET(sch, cgroup_init,
 					      NULL, tg->css.cgroup, &args);
@@ -5185,6 +5186,7 @@ static int scx_cgroup_init(struct scx_sched *sch)
 				.bw_period_us = tg->scx.bw_period_us,
 				.bw_quota_us = tg->scx.bw_quota_us,
 				.bw_burst_us = tg->scx.bw_burst_us,
+				.idle = tg->scx.idle,
 			};
 
 			ret = SCX_CALL_OP_RET(sch, cgroup_init, NULL, css->cgroup, &args);
kernel/sched/ext/internal.h +3 -0
index 53e136a47924..aa149a9c29f7 100644
@@ -259,6 +259,9 @@ struct scx_cgroup_init_args {
 	u64			bw_period_us;
 	u64			bw_quota_us;
 	u64			bw_burst_us;
+
+	/* whether the cgroup is configured idle via cpu.idle */
+	bool			idle;
 };
 
 enum scx_cpu_preempt_reason {
Signature · 1 line
2.43.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The patch fails to initialize the new `idle` field in `struct scx_cgroup_init_args` in the…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The patch fails to initialize the new idle field in struct scx_cgroup_init_args in the sub-scheduler paths, causing it to implicitly default to false and lose the correct idle state. --

--- Patch [1]: [PATCH] sched_ext: pass the initial cpu.idle state in scx_cgroup_init_args --- commit 6b122f2bce26c31e18660129955114867f5a36b0
Author: Tao Cui <[email protected]>

sched_ext: pass the initial cpu.idle state in scx_cgroup_init_args

This commit adds the idle state to scx_cgroup_init_args and initializes it in scx_tg_online() and scx_cgroup_init(). This ensures that cgroups already configured as idle before the scheduler is loaded are correctly presented as idle to the BPF scheduler.

kernel/sched/ext/internal.h struct scx_cgroup_init_args { · line 265

Quoted 12 lines
kernel/sched/ext/internal.h +3 -0
index 53e136a47924d..aa149a9c29f7b 100644
@@ -259,6 +259,9 @@ struct scx_cgroup_init_args {
 	u64			bw_period_us;
 	u64			bw_quota_us;
 	u64			bw_burst_us;
+
+	/* whether the cgroup is configured idle via cpu.idle */
+	bool			idle;
 };

[Severity: Medium]
When adding this new field, did we miss updating the struct initialization in the sub-scheduler paths?

Looking at kernel/sched/ext/sub.c, both scx_cgroup_claim_subtree() and scx_cgroup_return_subtree() initialize the struct but omit the new field:

kernel/sched/ext/sub.c:scx_cgroup_claim_subtree() {
    ...
		struct scx_cgroup_init_args args = {
			.weight = tg->scx.weight,
			.bw_period_us = tg->scx.bw_period_us,
			.bw_quota_us = tg->scx.bw_quota_us,
			.bw_burst_us = tg->scx.bw_burst_us,
		};
    ...
}

The exact same pattern also exists in scx_cgroup_return_subtree().

Without explicitly setting the idle field in these paths, wouldn't it implicitly default to false, causing cgroups to incorrectly lose their idle state when a sub-scheduler is enabled or disabled?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
reviewer
#2 lore raw
In reply to Tao Cui

Hi Tao,

Quoted 52 lines · Tao Cui

From: Tao Cui <[email protected]>

scx_cgroup_init_args carries the initial weight and bandwidth control parameters of a cgroup to ops.cgroup_init(), but not its cpu.idle state. A cgroup that was already configured idle before the scheduler was loaded (or before it was onlined under it) is presented as non-idle, and the BPF scheduler only learns about it if cpu.idle is written again later.

Add the idle state to scx_cgroup_init_args and fill it in both places that build the args -- scx_tg_online() for cgroups created under the scheduler and scx_cgroup_init() for cgroups that already exist when the scheduler is loaded.

Verified in a VM with a probe scheduler printing the init args: a cgroup configured cpu.idle=1 before loading shows idle=1 in ops.cgroup_init(), the default shows 0, and later cpu.idle writes still come through ops.cgroup_set_idle().

---
 kernel/sched/ext/ext.c      | 4 +++-
 kernel/sched/ext/internal.h | 3 +++
 2 files changed, 6 insertions(+), 1 deletion(-)
kernel/sched/ext/ext.c +3 -1
index b646711a45fe..a7218314dcfb 100644
@@ -4764,7 +4764,8 @@ int scx_tg_online(struct task_group *tg)
 				{ .weight = tg->scx.weight,
 				  .bw_period_us = tg->scx.bw_period_us,
 				  .bw_quota_us = tg->scx.bw_quota_us,
-				  .bw_burst_us = tg->scx.bw_burst_us };
+				  .bw_burst_us = tg->scx.bw_burst_us,
+				  .idle = tg->scx.idle };
 
 			ret = SCX_CALL_OP_RET(sch, cgroup_init,
 					      NULL, tg->css.cgroup, &args);
@@ -5185,6 +5186,7 @@ static int scx_cgroup_init(struct scx_sched *sch)
 				.bw_period_us = tg->scx.bw_period_us,
 				.bw_quota_us = tg->scx.bw_quota_us,
 				.bw_burst_us = tg->scx.bw_burst_us,
+				.idle = tg->scx.idle,
 			};
 
 			ret = SCX_CALL_OP_RET(sch, cgroup_init, NULL, css->cgroup, &args);
kernel/sched/ext/internal.h +0 -0
index 53e136a47924..aa149a9c29f7 100644

kernel/sched/ext/internal.h struct scx_cgroup_init_args { · line 265

Quoted 7 lines
kernel/sched/ext/internal.h +3 -0
@@ -259,6 +259,9 @@ struct scx_cgroup_init_args {
 	u64			bw_period_us;
 	u64			bw_quota_us;
 	u64			bw_burst_us;
+
+	/* whether the cgroup is configured idle via cpu.idle */
+	bool			idle;
 };

It looks like there are two more places in kernel/sched/ext/sub.c, scx_cgroup_claim_subtree() and scx_cgroup_return_subtree(), they should also initialize idle. Otherwise the field defaults to false and an idle cgroup would be presented as non-idle when it's handed to a sub-scheduler.

Thanks,
-Andrea

author
#3 lore raw

Hello, Andrea

Quoted 59 lines · Andrea Righi

Hi Tao,

Quoted 56 lines · Tao Cui

From: Tao Cui <[email protected]>

scx_cgroup_init_args carries the initial weight and bandwidth control parameters of a cgroup to ops.cgroup_init(), but not its cpu.idle state. A cgroup that was already configured idle before the scheduler was loaded (or before it was onlined under it) is presented as non-idle, and the BPF scheduler only learns about it if cpu.idle is written again later.

Add the idle state to scx_cgroup_init_args and fill it in both places that build the args -- scx_tg_online() for cgroups created under the scheduler and scx_cgroup_init() for cgroups that already exist when the scheduler is loaded.

Verified in a VM with a probe scheduler printing the init args: a cgroup configured cpu.idle=1 before loading shows idle=1 in ops.cgroup_init(), the default shows 0, and later cpu.idle writes still come through ops.cgroup_set_idle().

---
 kernel/sched/ext/ext.c      | 4 +++-
 kernel/sched/ext/internal.h | 3 +++
 2 files changed, 6 insertions(+), 1 deletion(-)
kernel/sched/ext/ext.c +3 -1
index b646711a45fe..a7218314dcfb 100644
@@ -4764,7 +4764,8 @@ int scx_tg_online(struct task_group *tg)
 				{ .weight = tg->scx.weight,
 				  .bw_period_us = tg->scx.bw_period_us,
 				  .bw_quota_us = tg->scx.bw_quota_us,
-				  .bw_burst_us = tg->scx.bw_burst_us };
+				  .bw_burst_us = tg->scx.bw_burst_us,
+				  .idle = tg->scx.idle };
 
 			ret = SCX_CALL_OP_RET(sch, cgroup_init,
 					      NULL, tg->css.cgroup, &args);
@@ -5185,6 +5186,7 @@ static int scx_cgroup_init(struct scx_sched *sch)
 				.bw_period_us = tg->scx.bw_period_us,
 				.bw_quota_us = tg->scx.bw_quota_us,
 				.bw_burst_us = tg->scx.bw_burst_us,
+				.idle = tg->scx.idle,
 			};
 
 			ret = SCX_CALL_OP_RET(sch, cgroup_init, NULL, css->cgroup, &args);
kernel/sched/ext/internal.h +1 -0
index 53e136a47924..aa149a9c29f7 100644
@@ -259,6 +259,9 @@ struct scx_cgroup_init_args {
 	u64			bw_period_us;
 	u64			bw_quota_us;
 	u64			bw_burst_us;
+
Quoted 9 lines

kernel/sched/ext/internal.h struct scx_cgroup_init_args { · line 265

Quoted 3 lines
kernel/sched/ext/internal.h +2 -0
+	/* whether the cgroup is configured idle via cpu.idle */
+	bool			idle;
 };

It looks like there are two more places in kernel/sched/ext/sub.c, scx_cgroup_claim_subtree() and scx_cgroup_return_subtree(), they should also initialize idle. Otherwise the field defaults to false and an idle cgroup would be presented as non-idle when it's handed to a sub-scheduler.

Right, I missed those two. v2 fills .idle in all four places that build scx_cgroup_init_args, including both sub-scheduler handover paths in sub.c. The main paths are verified in a VM; the sub.c ones are compile-tested only.

Thanks,
Tao> Thanks,

Quoted 1 line

-Andrea