Skip to contentSkip to conversation

sched_ext: Work around pahole 1.32 dropping scx_bpf_task_set_lazy_resched() from BTF

PATCH Applied Tejun Heo 4 participants 7 messages Sep 29 – Sep 30

Subsystems (3)
lore
Share & tools
lore

Tejun Heo

author reviewer
#0 lore raw

PATCH 1/1 sched_ext: Work around pahole 1.32 dropping scx_bpf_task_set_lazy_resched() from BTF

With clang and pahole 1.32, x86-64 vmlinux BTF lacks scx_bpf_task_set_lazy_resched() and sched_ext fails to initialize:

sched_ext: Failed to register kfunc sets (-22)

clang describes the lazy argument only after the prologue moves it into a callee-saved register, which pahole 1.32 rejects as not being the argument register. Released pahole is affected whatever the eventual fix. Keep the argument on the stack with barrier_data() so that its location names no register. OPTIMIZER_HIDE_VAR() would fix x86-64 but break arm64.

Fixes f8e5a4e3f3be ("sched_ext: Add lazy preemption support")
---
 kernel/sched/ext/ext.c |    5 +++++
 1 file changed, 5 insertions(+)
kernel/sched/ext/ext.c +5 -0
@@ -10136,6 +10136,11 @@ __bpf_kfunc bool scx_bpf_task_set_lazy_r
 {
 	struct scx_sched *sch;
 
+	/*
+	 * clang's register locations for @lazy make pahole 1.32 drop this kfunc
+	 * from BTF. Keep @lazy in memory so its location names no register.
+	 */
+	barrier_data(&lazy);
 	guard(rcu)();
 	sch = scx_prog_sched(aux);
 	if (unlikely(!sch || !scx_task_on_sched(sch, p)))
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] sched: build failure due to misapplied patch in scx_bpf_task_set_dsq_vtime() --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] sched: build failure due to misapplied patch in scx_bpf_task_set_dsq_vtime() --

commit 16f6b2129331609ed5db39cd3ea7260f54a1df1f
Author: Tejun Heo <[email protected]>

sched_ext: Work around pahole 1.32 dropping scx_bpf_task_set_lazy_resched() from BTF

This commit works around an issue with pahole 1.32 by forcing the lazy argument onto the stack using barrier_data() so its location doesn't name a register.

kernel/sched/ext/ext.c __bpf_kfunc bool scx_bpf_task_set_dsq_vtime(struct task_struct *p, u64 vtime, · line 10126

Quoted 13 lines
kernel/sched/ext/ext.c +5 -0
index 6b037d27c99d3..feaa784d7a0dc 100644
@@ -10119,6 +10119,11 @@ __bpf_kfunc bool scx_bpf_task_set_dsq_vtime(struct task_struct *p, u64 vtime,
 {
 	struct scx_sched *sch;
 
+	/*
+	 * clang's register locations for @lazy make pahole 1.32 drop this kfunc
+	 * from BTF. Keep @lazy in memory so its location names no register.
+	 */
+	barrier_data(&lazy);

[Severity: Low]
Will this cause a build failure? It looks like the lazy variable is not declared in scx_bpf_task_set_dsq_vtime().

Could it be that this patch was mistakenly applied to scx_bpf_task_set_dsq_vtime() instead of scx_bpf_task_set_lazy_resched()? They appear to share identical context lines, meaning the patch might have matched guard(rcu)(); followed by sch = scx_prog_sched(aux); in the wrong function.

Quoted 4 lines
kernel/sched/ext/ext.c +0 -0
 	guard(rcu)();
 	sch = scx_prog_sched(aux);
 	if (unlikely(!sch || !scx_task_on_sched(sch, p)))

return false;

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
reviewer
#2 lore raw
In reply to Tejun Heo

Hi Tejun,

Quoted 21 lines · Tejun Heo

With clang and pahole 1.32, x86-64 vmlinux BTF lacks scx_bpf_task_set_lazy_resched() and sched_ext fails to initialize:

sched_ext: Failed to register kfunc sets (-22)

clang describes the lazy argument only after the prologue moves it into a callee-saved register, which pahole 1.32 rejects as not being the argument register. Released pahole is affected whatever the eventual fix. Keep the argument on the stack with barrier_data() so that its location names no register. OPTIMIZER_HIDE_VAR() would fix x86-64 but break arm64.

Fixes f8e5a4e3f3be ("sched_ext: Add lazy preemption support")
---
 kernel/sched/ext/ext.c |    5 +++++
 1 file changed, 5 insertions(+)

kernel/sched/ext/ext.c __bpf_kfunc bool scx_bpf_task_set_lazy_r · line 10143

Quoted 8 lines
kernel/sched/ext/ext.c +5 -0
@@ -10136,6 +10136,11 @@ __bpf_kfunc bool scx_bpf_task_set_lazy_r
 {
 	struct scx_sched *sch;
 
+	/*
+	 * clang's register locations for @lazy make pahole 1.32 drop this kfunc
+	 * from BTF. Keep @lazy in memory so its location names no register.
+	 */
+	barrier_data(&lazy);

First of all thanks for figuring this out, it was really annoying, I've been holding pahole to 1.31 because of this.

I also looked a bit and it seems to be triggered specifically by the bool type. This is with clang 22.1.8 on x86_64 without barrier_data and only changing the type:

  bool  movl %esi, %ebp    -> dropped by pahole
  u32   movl %esi, %ebp    -> ok
  u64   movq %rsi, %r15    -> ok

With u32 the generated code is identical to the bool case, but clang keeps the RSI entry range, so pahole is happy.

Should we just change lazy to u32 (or use a u64 flags) instead of using barrier_data()?

Thanks,
-Andrea

Quoted 3 lines
kernel/sched/ext/ext.c +0 -0
 	guard(rcu)();
 	sch = scx_prog_sched(aux);
 	if (unlikely(!sch || !scx_task_on_sched(sch, p)))
author reviewer
#3 lore raw
Quoted 11 lines · Andrea Righi

First of all thanks for figuring this out, it was really annoying, I've been holding pahole to 1.31 because of this.

I also looked a bit and it seems to be triggered specifically by the bool type. This is with clang 22.1.8 on x86_64 without barrier_data and only changing the type:

  bool  movl %esi, %ebp    -> dropped by pahole
  u32   movl %esi, %ebp    -> ok
  u64   movq %rsi, %r15    -> ok
Quoted 5 lines

With u32 the generated code is identical to the bool case, but clang keeps the RSI entry range, so pahole is happy.

Should we just change lazy to u32 (or use a u64 flags) instead of using barrier_data()?

I mean, bool is the right and I'm hoping clang can be fixed and the workaround seems to work, so keep it as-is unless there are other reasons to change it (e.g. this becomes a wider interface taking u64 flags or sth like that)?

Thanks.

Signature · 1 line
tejun
reviewer Reviewed-by
#4 lore raw
Quoted 15 lines · Tejun Heo
Quoted 14 lines · Andrea Righi

First of all thanks for figuring this out, it was really annoying, I've been holding pahole to 1.31 because of this.

I also looked a bit and it seems to be triggered specifically by the bool type. This is with clang 22.1.8 on x86_64 without barrier_data and only changing the type:

  bool  movl %esi, %ebp    -> dropped by pahole
  u32   movl %esi, %ebp    -> ok
  u64   movq %rsi, %r15    -> ok

With u32 the generated code is identical to the bool case, but clang keeps the RSI entry range, so pahole is happy.

Quoted 7 lines
Quoted 2 lines

Should we just change lazy to u32 (or use a u64 flags) instead of using barrier_data()?

I mean, bool is the right and I'm hoping clang can be fixed and the workaround seems to work, so keep it as-is unless there are other reasons to change it (e.g. this becomes a wider interface taking u64 flags or sth like that)?

Yeah, at the moment I don't see any valid reason to change lazy to a u64 flags. And it's probably better to introduce a workaround in the code rather than a workaround in the ABI, so let's keep barrier_data().

Thanks,
-Andrea

#6 lore raw
In reply to Tejun Heo
Does the Fixes tag reference a commit that exists in this branch's history? The tag points to f8e5a4e3f3be, but if that commit is not an ancestor of this commit, the Fixes tag would be referencing a…
Quoted 7 lines

sched_ext: Work around pahole 1.32 dropping scx_bpf_task_set_lazy_resched() from BTF

With clang and pahole 1.32, x86-64 vmlinux BTF lacks scx_bpf_task_set_lazy_resched() and sched_ext fails to initialize:

sched_ext: Failed to register kfunc sets (-22)

Quoted 7 lines

clang describes the lazy argument only after the prologue moves it into a callee-saved register, which pahole 1.32 rejects as not being the argument register. Released pahole is affected whatever the eventual fix. Keep the argument on the stack with barrier_data() so that its location names no register. OPTIMIZER_HIDE_VAR() would fix x86-64 but break arm64.

Fixes f8e5a4e3f3be ("sched_ext: Add lazy preemption support")

Does the Fixes tag reference a commit that exists in this branch's history? The tag points to f8e5a4e3f3be, but if that commit is not an ancestor of this commit, the Fixes tag would be referencing a commit from a different tree or branch.

Can you verify that f8e5a4e3f3be is reachable from this commit's history, or clarify which commit in this branch originally introduced the lazy preemption support that needs the workaround?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/36651688234