sched/rt: mitigate root_domain cache line contention
cover sched/rt: mitigate root_domain cache line contention
From: Deng Pan <[email protected]>
When running multi-instance FFmpeg workload in cloud environment, cache line contention is severe during the access to root_domain data structures, which significantly degrades performance.
The SUT is a 2-socket machine with 240 physical cores and 480 logical CPUs. 60 FFmpeg instances are launched, each pinned to 4 physical cores (8 logical CPUs) for transcoding tasks. Sub-threads use RT priority 99 with FIFO scheduling. FPS is used as score.
Profiling shows the kernel consumes ~20% of CPU cycles, which is excessive in this scenario. The overhead primarily comes from RT task scheduling functions like cpupri_set, cpupri_find_fitness, dequeue_pushable_task, enqueue_pushable_task, pull_rt_task, __find_first_and_bit, and __bitmap_and. This is due to read/write contention on root_domain cache lines.
The perf c2c report, sorted by contention severity, reveals:
root_domain cache line 3:
- cpupri->pri_to_cpu[0].count is heavily loaded/stored, since counts[0] is more frequently updated than others along with a rt task enqueues an empty runq or dequeues from a non-overloaded runq.
- rto_mask is heavily loaded
- rto_loop_next and rto_loop_start are frequently stored
- rto_push_work and rto_lock are lightly accessed
- cycles per load: ~10K to 59K.
root_domain cache line 1:
- rto_count is frequently loaded/stored
- overloaded is heavily loaded
- cycles per load: ~2.8K to 44K
cpumask (bitmap) cache line of cpupri_vec->mask:
- bits are loaded during cpupri_find
- bits are stored during cpupri_set
- cycles per load: ~2.2K to 8.7K
The end cache line of cpupri:
- cpupri_vec->count and mask contends. The transcoding threads use rt pri 99, so that the contention occurs in the end.
- cycles per load: ~1.5K to 10.5K
According to above, we propose 4 patches to mitigate the contention. Patch 1: Reorganize cpupri_vec, separate count, mask fields,
reducing contention on root_domain cache line 3 and cpupri's last cache line.
Patch 2: Restructure root_domain structure to minimize contention of
root_domain cache line 1 and 3 by reordering fields.
Patch 3: Split root_domain->rto_count to per-NUMA-node counters,
reducing the contention on root_domain cache line 1.
Patch 4: Split cpupri_vec->cpumask to per-NUMA-node bitmaps, reducing
load/store contention on the cpumask bitmap cache line.
Evaluation:
Performance improvements (FPS, relative to baseline):
- Patch 1: +11.0%
- Patch 2: +5.0% - Patch 3: +4.0% - Patch 4: +3.8%
Kernel CPU cycle usage reduction:
- Patch 1: 20.0% -> 11.0%
- Patch 2: 20.0% -> 17.7%
- Patch 3: 20.0% -> 18.6%
- Patch 4: 20.0% -> 18.7%
Cycles per load reduction (by perf c2c report):
- Patch 1:
- `root_domain` cache line 3: 10K–59K -> 0.5K–8K - `cpupri` last cache line: 1.5K–10.5K -> eliminated - Patch 2: - `root_domain` cache line 1: 2.8K–44K -> 2.1K–2.7K - `root_domain` cache line 3: 10K–59K -> eliminated
- Patch 3:
- root_domain cache line 1: 2.8K–44K -> eliminated
- Patch 4:
- cpupri_vec->mask cache line: 2.2K–8.7K -> 0.5K–2.2K
Comments are appreciated.
Pan Deng (4):
sched/rt: Optimize cpupri_vec layout to mitigate cache line contention sched/rt: Restructure root_domain to reduce cacheline contention sched/rt: Split root_domain->rto_count to per-NUMA-node counters sched/rt: Split cpupri_vec->cpumask to per NUMA node to reduce
contention
kernel/sched/cpupri.c | 200 ++++++++++++++++++++++++++++++++++++---- kernel/sched/cpupri.h | 6 +- kernel/sched/rt.c | 65 ++++++++++++- kernel/sched/sched.h | 61 ++++++------ kernel/sched/topology.c | 7 ++ 5 files changed, 291 insertions(+), 48 deletions(-)
Signature · 1 line
2.43.5
PATCH 1/4 sched/rt: Optimize cpupri_vec layout to mitigate cache line contention
When running a multi-instance FFmpeg workload on an HCC system, significant cache line contention is observed around cpupri_vec->count and mask in struct root_domain.
The SUT is a 2-socket machine with 240 physical cores and 480 logical CPUs. 60 FFmpeg instances are launched, each pinned to 4 physical cores (8 logical CPUs) for transcoding tasks. Sub-threads use RT priority 99 with FIFO scheduling. FPS is used as score.
perf c2c tool reveals:root_domain cache line 3:
- cpupri->pri_to_cpu[0].count (offset 0x38) is heavily loaded/stored and contends with other fields, since counts[0] is more frequently updated than others along with a rt task enqueues an empty runq or dequeues from a non-overloaded runq.
- cycles per load: ~10K to 59K
cpupri's last cache line:
- cpupri_vec->count and mask contends. The transcoding threads use rt pri 99, so that the contention occurs in the end.
- cycles per load: ~1.5K to 10.5K
This change mitigates cpupri_vec->count, mask related contentions by separating each count and mask into different cache lines.
As a result:
- FPS improves by ~11%
- Kernel cycles% drops from ~20% to ~11%
- count and mask related cache line contention is mitigated, perf c2c shows root_domain cache line 3 cycles per load drops from ~10K-59K to ~0.5K-8K, cpupri's last cache line no longer appears in the report.
Note: The side effect of this change is that struct cpupri size is increased from 26 cache lines to 203 cache lines.
An alternative approach could be separating counts and masks into 2 vectors in cpupri_vec (counts[] and masks[]), and add two paddings:
1. Between counts[0] and counts[1], since counts[0] is more frequently updated than others.
2. Between the two vectors, since counts[] is read-write access while masks[] is read access when it stores pointers.
The alternative approach introduces the complexity of 31+/21- LoC changes, it achieves almost the same performance, at the same time, struct cpupri size is reduced from 26 cache lines to 21 cache lines.
Appendix:
1. Current layout of contended data structures:
struct root_domain {
atomic_t refcount; /* 0 4 */
atomic_t rto_count; /* 4 4 */ struct callback_head rcu __attribute__((__aligned__(8)));/*8 16 */
cpumask_var_t span; /* 24 8 */
cpumask_var_t online; /* 32 8 */
bool overloaded; /* 40 1 */
bool overutilized; /* 41 1 */
/* XXX 6 bytes hole, try to pack */
cpumask_var_t dlo_mask; /* 48 8 */
atomic_t dlo_count; /* 56 4 */
/* XXX 4 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
struct dl_bw dl_bw; /* 64 24 */
struct cpudl cpudl; /* 88 24 */
u64 visit_gen; /* 112 8 */
struct irq_work rto_push_work; /* 120 32 */
/* --- cacheline 2 boundary (128 bytes) was 24 bytes ago --- */
raw_spinlock_t rto_lock; /* 152 4 */
int rto_loop; /* 156 4 */
int rto_cpu; /* 160 4 */
atomic_t rto_loop_next; /* 164 4 */
atomic_t rto_loop_start; /* 168 4 */
/* XXX 4 bytes hole, try to pack */
cpumask_var_t rto_mask; /* 176 8 */
struct cpupri cpupri; /* 184 1624 */
/* --- cacheline 28 boundary (1792 bytes) was 16 bytes ago --- */
struct perf_domain * pd; /* 1808 8 */
/* size: 1816, cachelines: 29, members: 21 */
/* sum members: 1802, holes: 3, sum holes: 14 */
/* forced alignments: 1 */
/* last cacheline: 24 bytes */
} __attribute__((__aligned__(8)));
struct cpupri {
struct cpupri_vec pri_to_cpu[101]; /* 0 1616 */
/* --- cacheline 25 boundary (1600 bytes) was 16 bytes ago --- */
int * cpu_to_pri; /* 1616 8 */
/* size: 1624, cachelines: 26, members: 2 */
/* last cacheline: 24 bytes */
};struct cpupri_vec {
atomic_t count; /* 0 4 */
/* XXX 4 bytes hole, try to pack */
cpumask_var_t mask; /* 8 8 */
/* size: 16, cachelines: 1, members: 2 */
/* sum members: 12, holes: 1, sum holes: 4 */
/* last cacheline: 16 bytes */
};2. Perf c2c report of root_domain cache line 3:
------- ------- ------ ------ ------ ------ ------------------------ Rmt Lcl Store Data Load Total Symbol Hitm% Hitm% L1 Hit% offset cycles records ------- ------- ------ ------ ------ ------ ------------------------ 353 44 62 0xff14d42c400e3880 ------- ------- ------ ------ ------ ------ ------------------------ 0.00% 2.27% 0.00% 0x0 21683 6 __flush_smp_call_function_ 0.00% 2.27% 0.00% 0x0 22294 5 __flush_smp_call_function_ 0.28% 0.00% 0.00% 0x0 0 2 irq_work_queue_on 0.28% 0.00% 0.00% 0x0 27824 4 irq_work_single 0.00% 0.00% 1.61% 0x0 28151 6 irq_work_queue_on 0.57% 0.00% 0.00% 0x18 21822 8 native_queued_spin_lock_sl 0.28% 2.27% 0.00% 0x18 16101 10 native_queued_spin_lock_sl 0.57% 0.00% 0.00% 0x18 33199 5 native_queued_spin_lock_sl 0.00% 0.00% 1.61% 0x18 10908 32 _raw_spin_lock 0.00% 0.00% 1.61% 0x18 59770 2 _raw_spin_lock 0.00% 0.00% 1.61% 0x18 0 1 _raw_spin_unlock 1.42% 0.00% 0.00% 0x20 12918 20 pull_rt_task 0.85% 0.00% 25.81% 0x24 31123 199 pull_rt_task 0.85% 0.00% 3.23% 0x24 38218 24 pull_rt_task 0.57% 4.55% 19.35% 0x28 30558 207 pull_rt_task 0.28% 0.00% 0.00% 0x28 55504 10 pull_rt_task 18.70% 18.18% 0.00% 0x30 26438 291 dequeue_pushable_task 17.28% 22.73% 0.00% 0x30 29347 281 enqueue_pushable_task 1.70% 2.27% 0.00% 0x30 12819 31 enqueue_pushable_task 0.28% 0.00% 0.00% 0x30 17726 18 dequeue_pushable_task 34.56% 29.55% 0.00% 0x38 25509 527 cpupri_find_fitness 13.88% 11.36% 24.19% 0x38 30654 342 cpupri_set 3.12% 2.27% 0.00% 0x38 18093 39 cpupri_set 1.70% 0.00% 0.00% 0x38 37661 52 cpupri_find_fitness 1.42% 2.27% 19.35% 0x38 31110 211 cpupri_set 1.42% 0.00% 1.61% 0x38 45035 31 cpupri_set
3. Perf c2c report of cpupri's last cache line
------- ------- ------ ------ ------ ------ ------------------------ Rmt Lcl Store Data Load Total Symbol Hitm% Hitm% L1 Hit% offset cycles records ------- ------- ------ ------ ------ ------ ------------------------ 149 43 41 0xff14d42c400e3ec0 ------- ------- ------ ------ ------ ------ ------------------------ 8.72% 11.63% 0.00% 0x8 2001 165 cpupri_find_fitness 1.34% 2.33% 0.00% 0x18 1456 151 cpupri_find_fitness 8.72% 9.30% 58.54% 0x28 1744 263 cpupri_set 2.01% 4.65% 41.46% 0x28 1958 301 cpupri_set 1.34% 0.00% 0.00% 0x28 10580 6 cpupri_set 69.80% 67.44% 0.00% 0x30 1754 347 cpupri_set 8.05% 4.65% 0.00% 0x30 2144 256 cpupri_set
--- kernel/sched/cpupri.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/cpupri.h +1 -1
@@ -9,7 +9,7 @@struct cpupri_vec {atomic_t count; -cpumask_var_t mask; +cpumask_var_t mask____cacheline_aligned ; };struct cpupri {
Signature · 1 line
2.43.5
PATCH 2/4 sched/rt: Restructure root_domain to reduce cacheline contention
In reply to Pan DengWhen running a multi-instance FFmpeg workload on HCC system, significant contention is observed in root_domain cacheline 1 and 3.
The SUT is a 2-socket machine with 240 physical cores and 480 logical CPUs. 60 FFmpeg instances are launched, each pinned to 4 physical cores (8 logical CPUs) for transcoding tasks. Sub-threads use RT priority 99 with FIFO scheduling. FPS is used as score.
perf c2c tool reveals (sorted by contention severity):root_domain cache line 3:
- cpupri->pri_to_cpu[0].count (offset 0x38) is heavily loaded/stored, since counts[0] is more frequently updated than others along with a rt task enqueues an empty runq or dequeues from a non-overloaded runq.
- rto_mask (0x30) is heavily loaded
- rto_loop_next (0x24) and rto_loop_start (0x28) are frequently stored
- rto_push_work (0x0) and rto_lock (0x18) are lightly accessed
- cycles per load: ~10K to 59K
root_domain cache line 1:
- rto_count (0x4) is frequently loaded/stored
- overloaded (0x28) is heavily loaded
- cycles per load: ~2.8K to 44K:
This change adjusts the layout of root_domain to isolate these contended fields across separate cache lines:
1. rto_count remains in the 1st cache line; overloaded and overutilized are moved to the last cache line
2. rto_push_work is placed in the 2nd cache line
3. rto_loop_start, rto_loop_next, and rto_lock remain in the 3rd cache line; rto_mask is moved near pd in the penultimate cache line
4. cpupri starts at the 4th cache line to prevent pri_to_cpu[0].count contending with fields in cache line 3.
With this change:
- FPS improves by ~5%
- Kernel cycles% drops from ~20% to ~17.7%
- root_domain cache line 3 no longer appears in perf-c2c report
- cycles per load of root_domain cache line 1 is reduced to from ~2.8K-44K to ~2.1K-2.7K
According to the nature of the change, to my understanding, it doesn`t introduce any negative impact in other scenario.
Note: This change increases the size of root_domain from 29 to 31 cache lines, it's considered acceptable since root_domain is a single global object.
Appendix:
1. Current layout of contended data structures:
struct root_domain {
atomic_t refcount; /* 0 4 */
atomic_t rto_count; /* 4 4 */ struct callback_head rcu __attribute__((__aligned__(8)));/*8 16 */
cpumask_var_t span; /* 24 8 */
cpumask_var_t online; /* 32 8 */
bool overloaded; /* 40 1 */
bool overutilized; /* 41 1 */
/* XXX 6 bytes hole, try to pack */
cpumask_var_t dlo_mask; /* 48 8 */
atomic_t dlo_count; /* 56 4 */
/* XXX 4 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
struct dl_bw dl_bw; /* 64 24 */
struct cpudl cpudl; /* 88 24 */
u64 visit_gen; /* 112 8 */
struct irq_work rto_push_work; /* 120 32 */
/* --- cacheline 2 boundary (128 bytes) was 24 bytes ago --- */
raw_spinlock_t rto_lock; /* 152 4 */
int rto_loop; /* 156 4 */
int rto_cpu; /* 160 4 */
atomic_t rto_loop_next; /* 164 4 */
atomic_t rto_loop_start; /* 168 4 */
/* XXX 4 bytes hole, try to pack */
cpumask_var_t rto_mask; /* 176 8 */
struct cpupri cpupri; /* 184 1624 */
/* --- cacheline 28 boundary (1792 bytes) was 16 bytes ago --- */
struct perf_domain * pd; /* 1808 8 */
/* size: 1816, cachelines: 29, members: 21 */
/* sum members: 1802, holes: 3, sum holes: 14 */
/* forced alignments: 1 */
/* last cacheline: 24 bytes */
} __attribute__((__aligned__(8)));
struct cpupri {
struct cpupri_vec pri_to_cpu[101]; /* 0 1616 */
/* --- cacheline 25 boundary (1600 bytes) was 16 bytes ago --- */
int * cpu_to_pri; /* 1616 8 */
/* size: 1624, cachelines: 26, members: 2 */
/* last cacheline: 24 bytes */
};struct cpupri_vec {
atomic_t count; /* 0 4 */
/* XXX 4 bytes hole, try to pack */
cpumask_var_t mask; /* 8 8 */
/* size: 16, cachelines: 1, members: 2 */
/* sum members: 12, holes: 1, sum holes: 4 */
/* last cacheline: 16 bytes */
};2. Perf c2c report of root_domain cache line 3:
------- ------- ------ ------ ------ ------ ------------------------ Rmt Lcl Store Data Load Total Symbol Hitm% Hitm% L1 Hit% offset cycles records ------- ------- ------ ------ ------ ------ ------------------------ 353 44 62 0xff14d42c400e3880 ------- ------- ------ ------ ------ ------ ------------------------ 0.00% 2.27% 0.00% 0x0 21683 6 __flush_smp_call_function_ 0.00% 2.27% 0.00% 0x0 22294 5 __flush_smp_call_function_ 0.28% 0.00% 0.00% 0x0 0 2 irq_work_queue_on 0.28% 0.00% 0.00% 0x0 27824 4 irq_work_single 0.00% 0.00% 1.61% 0x0 28151 6 irq_work_queue_on 0.57% 0.00% 0.00% 0x18 21822 8 native_queued_spin_lock_sl 0.28% 2.27% 0.00% 0x18 16101 10 native_queued_spin_lock_sl 0.57% 0.00% 0.00% 0x18 33199 5 native_queued_spin_lock_sl 0.00% 0.00% 1.61% 0x18 10908 32 _raw_spin_lock 0.00% 0.00% 1.61% 0x18 59770 2 _raw_spin_lock 0.00% 0.00% 1.61% 0x18 0 1 _raw_spin_unlock 1.42% 0.00% 0.00% 0x20 12918 20 pull_rt_task 0.85% 0.00% 25.81% 0x24 31123 199 pull_rt_task 0.85% 0.00% 3.23% 0x24 38218 24 pull_rt_task 0.57% 4.55% 19.35% 0x28 30558 207 pull_rt_task 0.28% 0.00% 0.00% 0x28 55504 10 pull_rt_task 18.70% 18.18% 0.00% 0x30 26438 291 dequeue_pushable_task 17.28% 22.73% 0.00% 0x30 29347 281 enqueue_pushable_task 1.70% 2.27% 0.00% 0x30 12819 31 enqueue_pushable_task 0.28% 0.00% 0.00% 0x30 17726 18 dequeue_pushable_task 34.56% 29.55% 0.00% 0x38 25509 527 cpupri_find_fitness 13.88% 11.36% 24.19% 0x38 30654 342 cpupri_set 3.12% 2.27% 0.00% 0x38 18093 39 cpupri_set 1.70% 0.00% 0.00% 0x38 37661 52 cpupri_find_fitness 1.42% 2.27% 19.35% 0x38 31110 211 cpupri_set 1.42% 0.00% 1.61% 0x38 45035 31 cpupri_set
3. Perf c2c report of root_domain cache line 1:
------- ------- ------ ------ ------ ------ ------------------------ Rmt Lcl Store Data Load Total Symbol Hitm% Hitm% L1 Hit% offset cycles records ------- ------- ------ ------ ------ ------ ------------------------ 231 43 48 0xff14d42c400e3800 ------- ------- ------ ------ ------ ------ ------------------------ 22.51% 18.60% 0.00% 0x4 5041 247 pull_rt_task 5.63% 2.33% 45.83% 0x4 6995 315 dequeue_pushable_task 3.90% 4.65% 54.17% 0x4 6587 370 enqueue_pushable_task 0.43% 0.00% 0.00% 0x4 17111 4 enqueue_pushable_task 0.43% 0.00% 0.00% 0x4 44062 4 dequeue_pushable_task 32.03% 27.91% 0.00% 0x28 6393 285 enqueue_task_rt 16.45% 27.91% 0.00% 0x28 5534 139 sched_balance_newidle 14.72% 18.60% 0.00% 0x28 5287 110 dequeue_task_rt 3.46% 0.00% 0.00% 0x28 2820 25 enqueue_task_fair 0.43% 0.00% 0.00% 0x28 220 3 enqueue_task_stop
--- kernel/sched/sched.h | 52 +++++++++++++++++++++++--------------------- 1 file changed, 27 insertions(+), 25 deletions(-)
kernel/sched/sched.h +27 -25
@@ -968,24 +968,29 @@ struct root_domain {cpumask_var_t span;cpumask_var_t online; +atomic_t dlo_count; +struct dl_bw dl_bw; +struct cpudl cpudl; + +#ifdef HAVE_RT_PUSH_IPI /* -* Indicate pullable load on at least one CPU, e.g: -* - More than one runnable task -* - Running task is misfit +* For IPI pull requests, loop across the rto_mask. */ -bool overloaded; - -/* Indicate one or more CPUs over-utilized (tipping point) */ -bool overutilized; +struct irq_work rto_push_work; +raw_spinlock_t rto_lock; +/* These are only updated and read within rto_lock */ +int rto_loop; +int rto_cpu; +/* These atomics are updated outside of a lock */ +atomic_t rto_loop_next; +atomic_t rto_loop_start; +#endif /* * The bit corresponding to a CPU gets set here if such CPU has more * than one runnable -deadline task (as it is below for RT tasks). */ cpumask_var_t dlo_mask; -atomic_t dlo_count; -struct dl_bw dl_bw; -struct cpudl cpudl;/* * Indicate whether a root_domain's dl_bw has been checked or
@@ -995,32 +1000,29 @@ struct root_domain {* that u64 is 'big enough'. So that shouldn't be a concern. */ u64 visit_cookie; +struct cpupri cpupri____cacheline_aligned ; -#ifdef HAVE_RT_PUSH_IPI /* -* For IPI pull requests, loop across the rto_mask. +* NULL-terminated list of performance domains intersecting with the +* CPUs of the rd. Protected by RCU. */ -struct irq_work rto_push_work; -raw_spinlock_t rto_lock; -/* These are only updated and read within rto_lock */ -int rto_loop; -int rto_cpu; -/* These atomics are updated outside of a lock */ -atomic_t rto_loop_next; -atomic_t rto_loop_start; -#endif +struct perf_domain __rcu *pd____cacheline_aligned ; +/* * The "RT overload" flag: it gets set if a CPU has more than * one runnable RT task. */ cpumask_var_t rto_mask; -struct cpupri cpupri;/* -* NULL-terminated list of performance domains intersecting with the -* CPUs of the rd. Protected by RCU. +* Indicate pullable load on at least one CPU, e.g: +* - More than one runnable task +* - Running task is misfit */ -struct perf_domain __rcu *pd; +bool overloaded____cacheline_aligned ; + +/* Indicate one or more CPUs over-utilized (tipping point) */ +bool overutilized; };extern void init_defrootdomain (void );
Signature · 1 line
2.43.5
PATCH 3/4 sched/rt: Split root_domain->rto_count to per-NUMA-node counters
In reply to Pan DengWhen running a multi-instance FFmpeg workload on HCC system, significant contention is observed on root_domain rto_count and overloaded fields.
The SUT is a 2-socket machine with 240 physical cores and 480 logical CPUs. 60 FFmpeg instances are launched, each pinned to 4 physical cores (8 logical CPUs) for transcoding tasks. Sub-threads use RT priority 99 with FIFO scheduling. FPS is used as score.
perf c2c tool reveals:root_domain cache line 1:
- rto_count (0x4) is frequently loaded/stored
- overloaded (0x28) is heavily loaded
- cycles per load: ~2.8K to 44K:
A separate patch rearranges root_domain to place overloaded on a different cache line, but this alone is insufficient to resolve the contention on rto_count. As a complementary, this patch splits rto_count into per-numa-node counters to reduce the contention.
With this change:
- FPS improves by ~4%
- Kernel cycles% drops from ~20% to ~18.6%
- The cache line no longer appears in perf-c2c report
Appendix:
1. Perf c2c report of root_domain cache line 1:
------- ------- ------ ------ ------ ------ ------------------------ Rmt Lcl Store Data Load Total Symbol Hitm% Hitm% L1 Hit% offset cycles records ------- ------- ------ ------ ------ ------ ------------------------ 231 43 48 0xff14d42c400e3800 ------- ------- ------ ------ ------ ------ ------------------------ 22.51% 18.60% 0.00% 0x4 5041 247 pull_rt_task 5.63% 2.33% 45.83% 0x4 6995 315 dequeue_pushable_task 3.90% 4.65% 54.17% 0x4 6587 370 enqueue_pushable_task 0.43% 0.00% 0.00% 0x4 17111 4 enqueue_pushable_task 0.43% 0.00% 0.00% 0x4 44062 4 dequeue_pushable_task 32.03% 27.91% 0.00% 0x28 6393 285 enqueue_task_rt 16.45% 27.91% 0.00% 0x28 5534 139 sched_balance_newidle 14.72% 18.60% 0.00% 0x28 5287 110 dequeue_task_rt 3.46% 0.00% 0.00% 0x28 2820 25 enqueue_task_fair 0.43% 0.00% 0.00% 0x28 220 3 enqueue_task_stop
--- kernel/sched/rt.c | 65 +++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 9 +++++- kernel/sched/topology.c | 7 +++++ 3 files changed, 77 insertions(+), 4 deletions(-)
kernel/sched/rt.c +62 -3
@@ -337,9 +337,58 @@ static inline bool need_pull_rt_task(struct rq *rq, struct task_struct *prev)return rq->online && rq->rt.highest_prio.curr > prev->prio; } +int rto_counts_init (atomic_tp **rto_counts) +{ +int i; +atomic_tp *counts =kzalloc (nr_node_ids *sizeof (atomic_tp),GFP_KERNEL ); + +if (!counts) +return -ENOMEM ; + +for (i =0 ; i < nr_node_ids; i++) { + counts[i] =kzalloc_node (sizeof (atomic_t),GFP_KERNEL , i); + +if (!counts[i]) + goto cleanup; + } + + *rto_counts = counts; +return 0 ; + +cleanup: +while (i--) +kfree (counts[i]); + +kfree (counts); +return -ENOMEM ; +} + +void rto_counts_cleanup (atomic_tp *rto_counts) +{ +for (int i =0 ; i < nr_node_ids; i++) +kfree (rto_counts[i]); + +kfree (rto_counts); +} +static inline int rt_overloaded (struct rq *rq) { -return atomic_read (&rq->rd->rto_count); +int count =0 ; +int cur_node, nid; + + cur_node =numa_node_id (); + +for (int i =0 ; i < nr_node_ids; i++) { + nid = (cur_node + i) % nr_node_ids; + count +=atomic_read (rq->rd->rto_counts[nid]); + +// The caller only checks if it is 0 +// or 1, so that return once > 1 +if (count >1 ) +return count; + } + +return count; }static inline void rt_set_overload (struct rq *rq)
@@ -358,7 +407,7 @@ static inline void rt_set_overload(struct rq *rq)* Matched by the barrier in pull_rt_task(). */ smp_wmb (); -atomic_inc (&rq->rd->rto_count); +atomic_inc (rq->rd->rto_counts[cpu_to_node (rq->cpu)]); }static inline void rt_clear_overload (struct rq *rq)
@@ -367,7 +416,7 @@ static inline void rt_clear_overload(struct rq *rq)return ;/* the order here really doesn't matter */ -atomic_dec (&rq->rd->rto_count); +atomic_dec (rq->rd->rto_counts[cpu_to_node (rq->cpu)]);cpumask_clear_cpu (rq->cpu, rq->rd->rto_mask); }
@@ -443,6 +492,16 @@ static inline void dequeue_pushable_task(struct rq *rq, struct task_struct *p)static inline void rt_queue_push_tasks (struct rq *rq) { } + +int rto_counts_init (atomic_tp **rto_counts) +{ +return 0 ; +} + +void rto_counts_cleanup (atomic_tp *rto_counts) +{ +} +#endif /* CONFIG_SMP */ static void enqueue_top_rt_rq (struct rt_rq *rt_rq);
kernel/sched/sched.h +8 -1
@@ -953,6 +953,8 @@ struct perf_domain {struct rcu_head rcu; }; +typedef atomic_t *atomic_tp ; +/* * We add the notion of a root-domain which will be used to define per-domain * variables. Each exclusive cpuset essentially defines an island domain by
@@ -963,12 +965,15 @@ struct perf_domain {*/ struct root_domain {atomic_t refcount; -atomic_t rto_count;struct rcu_head rcu;cpumask_var_t span;cpumask_var_t online;atomic_t dlo_count; + +/* rto_count per node */ +atomic_tp *rto_counts; +struct dl_bw dl_bw;struct cpudl cpudl;
@@ -1030,6 +1035,8 @@ extern int sched_init_domains(const struct cpumask *cpu_map);extern void rq_attach_root (struct rq *rq,struct root_domain *rd);extern void sched_get_rd (struct root_domain *rd);extern void sched_put_rd (struct root_domain *rd); +extern int rto_counts_init (atomic_tp **rto_counts); +extern void rto_counts_cleanup (atomic_tp *rto_counts);static inline int get_rd_overloaded (struct root_domain *rd) {
kernel/sched/topology.c +7 -0
@@ -457,6 +457,7 @@ static void free_rootdomain(struct rcu_head *rcu) { struct root_domain *rd = container_of(rcu, struct root_domain, rcu); +rto_counts_cleanup (rd->rto_counts);cpupri_cleanup (&rd->cpupri);cpudl_cleanup (&rd->cpudl);free_cpumask_var (rd->dlo_mask);
@@ -549,8 +550,14 @@ static int init_rootdomain(struct root_domain *rd)if (cpupri_init (&rd->cpupri) !=0 ) goto free_cpudl; + +if (rto_counts_init (&rd->rto_counts) !=0 ) + goto free_cpupri; +return 0 ; +free_cpupri: +cpupri_cleanup (&rd->cpupri); free_cpudl:cpudl_cleanup (&rd->cpudl); free_rto_mask:
Signature · 1 line
2.43.5
PATCH 4/4 sched/rt: Split cpupri_vec->cpumask to per NUMA node to reduce contention
In reply to Pan DengWhen running a multi-instance FFmpeg workload on HCC system, significant contention is observed on bitmap of cpupri_vec->cpumask.
The SUT is a 2-socket machine with 240 physical cores and 480 logical CPUs. 60 FFmpeg instances are launched, each pinned to 4 physical cores (8 logical CPUs) for transcoding tasks. Sub-threads use RT priority 99 with FIFO scheduling. FPS is used as score.
perf c2c tool reveals:
cpumask (bitmap) cache line of cpupri_vec->mask:
- bits are loaded during cpupri_find
- bits are stored during cpupri_set
- cycles per load: ~2.2K to 8.7K
This change splits cpupri_vec->cpumask into per-NUMA-node data to mitigate false sharing.
As a result:
- FPS improves by ~3.8%
- Kernel cycles% drops from ~20% to ~18.7%
- Cache line contention is mitigated, perf-c2c shows cycles per load drops from ~2.2K-8.7K to ~0.5K-2.2K
Note: CONFIG_CPUMASK_OFFSTACK=n remains unchanged.
Appendix:
1. Perf c2c report of cpupri_vec->mask bitmap cache line:
------- ------- ------ ------ ------ ------ ------------------------ Rmt Lcl Store Data Load Total Symbol Hitm% Hitm% L1 Hit% offset cycles records ------- ------- ------ ------ ------ ------ ------------------------ 155 39 39 0xff14d52c4682d800 ------- ------- ------ ------ ------ ------ ------------------------ 43.23% 43.59% 0.00% 0x0 3489 415 _find_first_and_bit 3.23% 5.13% 0.00% 0x0 3478 107 __bitmap_and 3.23% 0.00% 0.00% 0x0 2712 33 _find_first_and_bit 1.94% 0.00% 7.69% 0x0 5992 33 cpupri_set 0.00% 0.00% 5.13% 0x0 3733 19 cpupri_set 12.90% 12.82% 0.00% 0x8 3452 297 _find_first_and_bit 1.29% 2.56% 0.00% 0x8 3007 117 __bitmap_and 0.00% 5.13% 0.00% 0x8 3041 20 _find_first_and_bit 0.00% 2.56% 2.56% 0x8 2374 22 cpupri_set 0.00% 0.00% 7.69% 0x8 4194 38 cpupri_set 8.39% 2.56% 0.00% 0x10 3336 264 _find_first_and_bit 3.23% 0.00% 0.00% 0x10 3023 46 _find_first_and_bit 2.58% 0.00% 0.00% 0x10 3040 130 __bitmap_and 1.29% 0.00% 12.82% 0x10 4075 34 cpupri_set 0.00% 0.00% 2.56% 0x10 2197 19 cpupri_set 0.00% 2.56% 7.69% 0x18 4085 27 cpupri_set 0.00% 2.56% 0.00% 0x18 3128 220 _find_first_and_bit 0.00% 0.00% 5.13% 0x18 3028 20 cpupri_set 2.58% 2.56% 0.00% 0x20 3089 198 _find_first_and_bit 1.29% 0.00% 5.13% 0x20 5114 29 cpupri_set 0.65% 2.56% 0.00% 0x20 3224 96 __bitmap_and 0.65% 0.00% 7.69% 0x20 4392 31 cpupri_set 2.58% 0.00% 0.00% 0x28 3327 214 _find_first_and_bit 0.65% 2.56% 5.13% 0x28 5252 31 cpupri_set 0.65% 0.00% 7.69% 0x28 8755 25 cpupri_set 0.65% 0.00% 0.00% 0x28 4414 14 _find_first_and_bit 1.29% 2.56% 0.00% 0x30 3139 171 _find_first_and_bit 0.65% 0.00% 7.69% 0x30 2185 18 cpupri_set 0.65% 0.00% 0.00% 0x30 3404 108 __bitmap_and 0.00% 0.00% 2.56% 0x30 5542 21 cpupri_set 3.23% 5.13% 0.00% 0x38 3493 190 _find_first_and_bit 3.23% 2.56% 0.00% 0x38 3171 108 __bitmap_and 0.00% 2.56% 7.69% 0x38 3285 14 cpupri_set 0.00% 0.00% 5.13% 0x38 4035 27 cpupri_set
--- kernel/sched/cpupri.c | 200 ++++++++++++++++++++++++++++++++++++++---- kernel/sched/cpupri.h | 4 + 2 files changed, 186 insertions(+), 18 deletions(-)
kernel/sched/cpupri.c +182 -18
@@ -64,6 +64,143 @@ static int convert_prio(int prio)return cpupri; } +#ifdef CONFIG_CPUMASK_OFFSTACK +static inline int alloc_vec_masks (struct cpupri_vec *vec) +{ +int i; + +for (i =0 ; i < nr_node_ids; i++) { +if (!zalloc_cpumask_var_node (&vec->masks[i],GFP_KERNEL , i)) + goto cleanup; + +// Clear masks of cur node, set others +bitmap_complement (cpumask_bits (vec->masks[i]), +cpumask_bits (cpumask_of_node (i)), small_cpumask_bits); + } +return 0 ; + +cleanup: +while (i--) +free_cpumask_var (vec->masks[i]); +return -ENOMEM ; +} + +static inline void free_vec_masks (struct cpupri_vec *vec) +{ +for (int i =0 ; i < nr_node_ids; i++) +free_cpumask_var (vec->masks[i]); +} + +static inline int setup_vec_mask_var_ts (struct cpupri *cp) +{ +int i; + +for (i =0 ; i <CPUPRI_NR_PRIORITIES ; i++) { +struct cpupri_vec *vec = &cp->pri_to_cpu[i]; + + vec->masks =kcalloc (nr_node_ids,sizeof (cpumask_var_t),GFP_KERNEL ); +if (!vec->masks) + goto cleanup; + } +return 0 ; + +cleanup: +/* Free any already allocated masks */ +while (i--) { +kfree (cp->pri_to_cpu[i].masks); + cp->pri_to_cpu[i].masks =NULL ; + } + +return -ENOMEM ; +} + +static inline void free_vec_mask_var_ts (struct cpupri *cp) +{ +for (int i =0 ; i <CPUPRI_NR_PRIORITIES ; i++) { +kfree (cp->pri_to_cpu[i].masks); + cp->pri_to_cpu[i].masks =NULL ; + } +} + +static inline int +available_cpu_in_nodes (struct task_struct *p,struct cpupri_vec *vec) +{ +int cur_node =numa_node_id (); + +for (int i =0 ; i < nr_node_ids; i++) { +int nid = (cur_node + i) % nr_node_ids; + +if (cpumask_first_and_and (&p->cpus_mask, vec->masks[nid], +cpumask_of_node (nid)) < nr_cpu_ids) +return 1 ; + } + +return 0 ; +} + +#define available_cpu_in_vec available_cpu_in_nodes + +#else /* !CONFIG_CPUMASK_OFFSTACK */ + +static inline int alloc_vec_masks (struct cpupri_vec *vec) +{ +if (!zalloc_cpumask_var (&vec->mask,GFP_KERNEL )) +return -ENOMEM ; + +return 0 ; +} + +static inline void free_vec_masks (struct cpupri_vec *vec) +{ +free_cpumask_var (vec->mask); +} + +static inline int setup_vec_mask_var_ts (struct cpupri *cp) +{ +return 0 ; +} + +static inline void free_vec_mask_var_ts (struct cpupri *cp) +{ +} + +static inline int +available_cpu_in_vec (struct task_struct *p,struct cpupri_vec *vec) +{ +if (cpumask_any_and (&p->cpus_mask, vec->mask) >= nr_cpu_ids) +return 0 ; + +return 1 ; +} +#endif + +static inline int alloc_all_masks (struct cpupri *cp) +{ +int i; + +for (i =0 ; i <CPUPRI_NR_PRIORITIES ; i++) { +if (alloc_vec_masks (&cp->pri_to_cpu[i])) + goto cleanup; + } + +return 0 ; + +cleanup: +while (i--) +free_vec_masks (&cp->pri_to_cpu[i]); + +return -ENOMEM ; +} + +static inline void setup_vec_counts (struct cpupri *cp) +{ +for (int i =0 ; i <CPUPRI_NR_PRIORITIES ; i++) { +struct cpupri_vec *vec = &cp->pri_to_cpu[i]; + +atomic_set (&vec->count,0 ); + } +} +static inline int __cpupri_find (struct cpupri *cp,struct task_struct *p,struct cpumask *lowest_mask,int idx) {
@@ -96,11 +233,24 @@ static inline int __cpupri_find(struct cpupri *cp, struct task_struct *p,if (skip)return 0 ; -if (cpumask_any_and (&p->cpus_mask, vec->mask) >= nr_cpu_ids) +if (!available_cpu_in_vec (p, vec))return 0 ; +#ifdef CONFIG_CPUMASK_OFFSTACK +struct cpumask *cpupri_mask = lowest_mask; + +// available && lowest_mask +if (lowest_mask) { +cpumask_copy (cpupri_mask, vec->masks[0 ]); +for (int nid =1 ; nid < nr_node_ids; nid++) +cpumask_and (cpupri_mask, cpupri_mask, vec->masks[nid]); + } +#else +struct cpumask *cpupri_mask = vec->mask; +#endif +if (lowest_mask) { -cpumask_and (lowest_mask, &p->cpus_mask, vec->mask); +cpumask_and (lowest_mask, &p->cpus_mask, cpupri_mask);cpumask_and (lowest_mask, lowest_mask, cpu_active_mask);/*
@@ -229,7 +379,11 @@ void cpupri_set(struct cpupri *cp, int cpu, int newpri)if (likely (newpri !=CPUPRI_INVALID )) {struct cpupri_vec *vec = &cp->pri_to_cpu[newpri]; +#ifdef CONFIG_CPUMASK_OFFSTACK +cpumask_set_cpu (cpu, vec->masks[cpu_to_node (cpu)]); +#else cpumask_set_cpu (cpu, vec->mask); +#endif /* * When adding a new vector, we update the mask first, * do a write memory barrier, and then update the count, to
@@ -263,7 +417,11 @@ void cpupri_set(struct cpupri *cp, int cpu, int newpri)*/ atomic_dec (&(vec)->count);smp_mb__after_atomic (); +#ifdef CONFIG_CPUMASK_OFFSTACK +cpumask_clear_cpu (cpu, vec->masks[cpu_to_node (cpu)]); +#else cpumask_clear_cpu (cpu, vec->mask); +#endif } *currpri = newpri;
@@ -279,26 +437,31 @@ int cpupri_init(struct cpupri *cp) {int i; -for (i =0 ; i <CPUPRI_NR_PRIORITIES ; i++) { -struct cpupri_vec *vec = &cp->pri_to_cpu[i]; - -atomic_set (&vec->count,0 ); -if (!zalloc_cpumask_var (&vec->mask,GFP_KERNEL )) - goto cleanup; - } - +/* Allocate the cpu_to_pri array */ cp->cpu_to_pri =kcalloc (nr_cpu_ids,sizeof (int ),GFP_KERNEL );if (!cp->cpu_to_pri) - goto cleanup; +return -ENOMEM ; +/* Initialize all CPUs to invalid priority */ for_each_possible_cpu (i) cp->cpu_to_pri[i] =CPUPRI_INVALID ; +/* Setup priority vectors */ +setup_vec_counts (cp); +if (setup_vec_mask_var_ts (cp)) + goto fail_setup_vectors; + +/* Allocate masks for each priority vector */ +if (alloc_all_masks (cp)) + goto fail_alloc_masks; +return 0 ; -cleanup: -for (i--; i >=0 ; i--) -free_cpumask_var (cp->pri_to_cpu[i].mask); +fail_alloc_masks: +free_vec_mask_var_ts (cp); + +fail_setup_vectors: +kfree (cp->cpu_to_pri);return -ENOMEM ; }
@@ -308,9 +471,10 @@ int cpupri_init(struct cpupri *cp)*/ void cpupri_cleanup (struct cpupri *cp) { -int i; -kfree (cp->cpu_to_pri); -for (i =0 ; i <CPUPRI_NR_PRIORITIES ; i++) -free_cpumask_var (cp->pri_to_cpu[i].mask); + +for (int i =0 ; i <CPUPRI_NR_PRIORITIES ; i++) +free_vec_masks (&cp->pri_to_cpu[i]); + +free_vec_mask_var_ts (cp); }
kernel/sched/cpupri.h +4 -0
@@ -9,7 +9,11 @@struct cpupri_vec {atomic_t count; +#ifdef CONFIG_CPUMASK_OFFSTACK +cpumask_var_t *masks____cacheline_aligned ; +#else cpumask_var_t mask____cacheline_aligned ; +#endif };struct cpupri {
Signature · 1 line
2.43.5
kernel test robot
build botRe: [PATCH 3/4] sched/rt: Split root_domain->rto_count to per-NUMA-node counters
In reply to Pan Dengkernel test robot noticed the following build errors:
Hi Pan,
kernel test robot noticed the following build errors:
[auto build test ERROR on v6.16-rc5]
[also build test ERROR on linus/master]
[cannot apply to tip/sched/core peterz-queue/sched/core tip/master tip/auto-latest next-20250704] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Pan-Deng/sched-rt-Optimize-cpupri_vec-layout-to-mitigate-cache-line-contention/20250707-131831 base: v6.16-rc5 patch link: https://lore.kernel.org/r/2c1e1dbacaddd881f3cca340ece1f9268029b620.1751852370.git.pan.deng%40intel.com patch subject: [PATCH 3/4] sched/rt: Split root_domain->rto_count to per-NUMA-node counters config: arm-randconfig-002-20250707 (https://download.01.org/0day-ci/archive/20250707/[email protected]/config) compiler: arm-linux-gnueabi-gcc (GCC) 11.5.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250707/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from kernel/sched/build_policy.c:52:
>> kernel/sched/rt.c:496:21: error: unknown type name 'atomic_tp'; did you mean 'atomic_t'?
496 | int rto_counts_init(atomic_tp **rto_counts)
| ^~~~~~~~~
| atomic_t
kernel/sched/rt.c:501:25: error: unknown type name 'atomic_tp'; did you mean 'atomic_t'?
501 | void rto_counts_cleanup(atomic_tp *rto_counts)
| ^~~~~~~~~
| atomic_tvim +496 kernel/sched/rt.c
495
> 496 int rto_counts_init(atomic_tp **rto_counts)
497 {
498 return 0;
499 }
500
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH 3/4] sched/rt: Split root_domain->rto_count to per-NUMA-node counters
In reply to Pan Dengkernel test robot noticed the following build warnings:
Hi Pan,
kernel test robot noticed the following build warnings:
[auto build test WARNING on v6.16-rc5]
[also build test WARNING on linus/master]
[cannot apply to tip/sched/core peterz-queue/sched/core tip/master tip/auto-latest next-20250704] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Pan-Deng/sched-rt-Optimize-cpupri_vec-layout-to-mitigate-cache-line-contention/20250707-131831 base: v6.16-rc5 patch link: https://lore.kernel.org/r/2c1e1dbacaddd881f3cca340ece1f9268029b620.1751852370.git.pan.deng%40intel.com patch subject: [PATCH 3/4] sched/rt: Split root_domain->rto_count to per-NUMA-node counters config: arm-allnoconfig (https://download.01.org/0day-ci/archive/20250707/[email protected]/config) compiler: clang version 21.0.0git (https://github.com/llvm/llvm-project 01c97b4953e87ae455bd4c41e3de3f0f0f29c61c) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250707/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
In file included from kernel/sched/build_policy.c:52:
kernel/sched/rt.c:496:21: error: unknown type name 'atomic_tp'; did you mean 'atomic_t'?
496 | int rto_counts_init(atomic_tp **rto_counts)
| ^~~~~~~~~
| atomic_t
include/linux/types.h:183:3: note: 'atomic_t' declared here
183 | } atomic_t;
| ^
In file included from kernel/sched/build_policy.c:52:
>> kernel/sched/rt.c:496:5: warning: no previous prototype for function 'rto_counts_init' [-Wmissing-prototypes]
496 | int rto_counts_init(atomic_tp **rto_counts)
| ^
kernel/sched/rt.c:496:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
496 | int rto_counts_init(atomic_tp **rto_counts)
| ^
| static
kernel/sched/rt.c:501:25: error: unknown type name 'atomic_tp'; did you mean 'atomic_t'?
501 | void rto_counts_cleanup(atomic_tp *rto_counts)
| ^~~~~~~~~
| atomic_t
include/linux/types.h:183:3: note: 'atomic_t' declared here
183 | } atomic_t;
| ^
In file included from kernel/sched/build_policy.c:52:
>> kernel/sched/rt.c:501:6: warning: no previous prototype for function 'rto_counts_cleanup' [-Wmissing-prototypes]
501 | void rto_counts_cleanup(atomic_tp *rto_counts)
| ^
kernel/sched/rt.c:501:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
501 | void rto_counts_cleanup(atomic_tp *rto_counts)
| ^
| static 2 warnings and 2 errors generated.
vim +/rto_counts_init +496 kernel/sched/rt.c
495
> 496 int rto_counts_init(atomic_tp **rto_counts)
497 {
498 return 0;
499 }
500
> 501 void rto_counts_cleanup(atomic_tp *rto_counts) 502 {
503 }
504
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
RE: [PATCH 3/4] sched/rt: Split root_domain->rto_count to per-NUMA-node counters
The issue arises from redundant functions when CONFIG_SMP is disabled, it will be addressed along with other feedback in the next version.
Best Regards
Pan
Quoted 103 lines
-----Original Message-----
From: lkp <[email protected]>
Sent: Monday, July 7, 2025 2:53 PM
To: Deng, Pan <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; linux- [email protected]; Li, Tianyou <[email protected]>; [email protected]; Chen, Yu C <[email protected]>; Deng, Pan <[email protected]>
Subject: Re: [PATCH 3/4] sched/rt: Splitroot_domain->rto_countto per-NUMA- node countersHi Pan,
kernel test robot noticed the following build warnings:
[auto build test WARNING on v6.16-rc5]
[also build test WARNING on linus/master] [cannot apply to tip/sched/core peterz-queue/sched/core tip/master tip/auto-latest next-20250704] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]url: https://github.com/intel-lab-lkp/linux/commits/Pan-Deng/sched-rt-
Optimize-cpupri_vec-layout-to-mitigate-cache-line-contention/20250707- 131831
base: v6.16-rc5
patch link:
https://lore.kernel.org/r/2c1e1dbacaddd881f3cca340ece1f9268029b620.1751
852370.git.pan.deng%40intel.com
patch subject: [PATCH 3/4] sched/rt: Splitroot_domain->rto_countto per- NUMA-node counters
config: arm-allnoconfig (https://download.01.org/0day-
ci/archive/20250707/[email protected]/config)
compiler: clang version 21.0.0git (https://github.com/llvm/llvm-project01c97b4953e87ae455bd4c41e3de3f0f0f29c61c)
reproduce (this is a W=1 build): (https://download.01.org/0day-
ci/archive/20250707/[email protected]/reproduce)If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]> | Closes: | https://lore.kernel.org/oe-kbuild-all/202507071418.sFa0bilv-lkp@intel. | com/All warnings (new ones prefixed by >>):
In file included from
kernel/sched/build_policy.c:52:
kernel/sched/rt.c:496:21: error: unknown type name 'atomic_tp'; did you
mean 'atomic_t'?496 | int rto_counts_init(atomic_tp **rto_counts) | ^~~~~~~~~ | atomic_t include/linux/types.h:183:3: note: 'atomic_t' declared here 183 | } atomic_t; | ^In file included from
kernel/sched/build_policy.c:52:Quoted 2 lines
Quoted 2 lines
kernel/sched/rt.c:496:5: warning: no previous prototype for function 'rto_counts_init' [-Wmissing-prototypes]496 | int
rto_counts_init(atomic_tp **rto_counts)
| ^
kernel/sched/rt.c:496:1: note: declare 'static' if the function is not intended to
be used outside of this translation unit496 | int rto_counts_init(atomic_tp **rto_counts) | ^ | static kernel/sched/rt.c:501:25: error: unknown type name 'atomic_tp'; did you mean 'atomic_t'? 501 | void rto_counts_cleanup(atomic_tp *rto_counts) | ^~~~~~~~~ | atomic_t include/linux/types.h:183:3: note: 'atomic_t' declared here 183 | } atomic_t; | ^In file included from
kernel/sched/build_policy.c:52:Quoted 2 lines
Quoted 2 lines
kernel/sched/rt.c:501:6: warning: no previous prototype for function 'rto_counts_cleanup' [-Wmissing-prototypes]501 | void
rto_counts_cleanup(atomic_tp *rto_counts)
| ^
kernel/sched/rt.c:501:1: note: declare 'static' if the function is not intended to
be used outside of this translation unit501 | void rto_counts_cleanup(atomic_tp *rto_counts) | ^ | static2 warnings and 2 errors generated.
vim +/rto_counts_init +496
kernel/sched/rt.c495Quoted 1 line
496 int
rto_counts_init(atomic_tp **rto_counts)497 { 498 return 0; 499 } 500Quoted 1 line
501 void
rto_counts_cleanup(atomic_tp *rto_counts)502 { 503 } 504--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH 3/4] sched/rt: Split root_domain->rto_count to per-NUMA-node counters
In reply to Pan Dengkernel test robot noticed the following build warnings:
Hi Pan,
kernel test robot noticed the following build warnings:
[auto build test WARNING on v6.16-rc5]
[also build test WARNING on linus/master]
[cannot apply to tip/sched/core peterz-queue/sched/core tip/master tip/auto-latest next-20250704] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Pan-Deng/sched-rt-Optimize-cpupri_vec-layout-to-mitigate-cache-line-contention/20250707-131831 base: v6.16-rc5 patch link: https://lore.kernel.org/r/2c1e1dbacaddd881f3cca340ece1f9268029b620.1751852370.git.pan.deng%40intel.com patch subject: [PATCH 3/4] sched/rt: Split root_domain->rto_count to per-NUMA-node counters config: loongarch-randconfig-r112-20250708 (https://download.01.org/0day-ci/archive/20250708/[email protected]/config) compiler: loongarch64-linux-gcc (GCC) 15.1.0 reproduce: (https://download.01.org/0day-ci/archive/20250708/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
sparse warnings: (new ones prefixed by >>)
kernel/sched/rt.c:1679:45: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/rt.c:1679:45: sparse: expected struct task_struct *p
kernel/sched/rt.c:1679:45: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/rt.c:1722:39: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected struct task_struct *donor @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/rt.c:1722:39: sparse: expected struct task_struct *donor
kernel/sched/rt.c:1722:39: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/rt.c:1742:64: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *tsk @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/rt.c:1742:64: sparse: expected struct task_struct *tsk
kernel/sched/rt.c:1742:64: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/rt.c:2084:40: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *task @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/rt.c:2084:40: sparse: expected struct task_struct *task
kernel/sched/rt.c:2084:40: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/rt.c:2107:13: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/rt.c:2107:13: sparse: struct task_struct *
kernel/sched/rt.c:2107:13: sparse: struct task_struct [noderef] __rcu *
kernel/sched/rt.c:2453:54: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *tsk @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/rt.c:2453:54: sparse: expected struct task_struct *tsk
kernel/sched/rt.c:2453:54: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/rt.c:2455:40: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/rt.c:2455:40: sparse: expected struct task_struct *p
kernel/sched/rt.c:2455:40: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/rt.c:2455:62: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/rt.c:2455:62: sparse: expected struct task_struct *p
kernel/sched/rt.c:2455:62: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/build_policy.c: note: in included file:
kernel/sched/deadline.c:2717:23: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/deadline.c:2717:23: sparse: expected struct task_struct *p
kernel/sched/deadline.c:2717:23: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/deadline.c:2727:13: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/deadline.c:2727:13: sparse: struct task_struct *
kernel/sched/deadline.c:2727:13: sparse: struct task_struct [noderef] __rcu *
kernel/sched/deadline.c:2833:25: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/deadline.c:2833:25: sparse: struct task_struct *
kernel/sched/deadline.c:2833:25: sparse: struct task_struct [noderef] __rcu *
kernel/sched/deadline.c:2357:42: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct sched_dl_entity const *b @@ got struct sched_dl_entity [noderef] __rcu * @@
kernel/sched/deadline.c:2357:42: sparse: expected struct sched_dl_entity const *b
kernel/sched/deadline.c:2357:42: sparse: got struct sched_dl_entity [noderef] __rcu *
kernel/sched/deadline.c:2368:38: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *tsk @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/deadline.c:2368:38: sparse: expected struct task_struct *tsk
kernel/sched/deadline.c:2368:38: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/deadline.c:1262:39: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/deadline.c:1262:39: sparse: expected struct task_struct *p
kernel/sched/deadline.c:1262:39: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/deadline.c:1262:85: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct sched_dl_entity const *b @@ got struct sched_dl_entity [noderef] __rcu * @@
kernel/sched/deadline.c:1262:85: sparse: expected struct sched_dl_entity const *b
kernel/sched/deadline.c:1262:85: sparse: got struct sched_dl_entity [noderef] __rcu *
kernel/sched/deadline.c:1362:23: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/deadline.c:1362:23: sparse: expected struct task_struct *p
kernel/sched/deadline.c:1362:23: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/deadline.c:1671:31: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/deadline.c:1671:31: sparse: expected struct task_struct *p
kernel/sched/deadline.c:1671:31: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/deadline.c:1671:70: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct sched_dl_entity const *b @@ got struct sched_dl_entity [noderef] __rcu * @@
kernel/sched/deadline.c:1671:70: sparse: expected struct sched_dl_entity const *b
kernel/sched/deadline.c:1671:70: sparse: got struct sched_dl_entity [noderef] __rcu *
kernel/sched/deadline.c:1760:39: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected struct task_struct *donor @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/deadline.c:1760:39: sparse: expected struct task_struct *donor
kernel/sched/deadline.c:1760:39: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/deadline.c:2578:9: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct sched_domain *[assigned] sd @@ got struct sched_domain [noderef] __rcu *parent @@
kernel/sched/deadline.c:2578:9: sparse: expected struct sched_domain *[assigned] sd
kernel/sched/deadline.c:2578:9: sparse: got struct sched_domain [noderef] __rcu *parent
kernel/sched/deadline.c:2242:14: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct task_struct *curr @@ got struct task_struct [noderef] __rcu * @@
kernel/sched/deadline.c:2242:14: sparse: expected struct task_struct *curr
kernel/sched/deadline.c:2242:14: sparse: got struct task_struct [noderef] __rcu *
kernel/sched/deadline.c:2243:15: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct task_struct *donor @@ got struct task_struct [noderef] __rcu * @@
kernel/sched/deadline.c:2243:15: sparse: expected struct task_struct *donor
kernel/sched/deadline.c:2243:15: sparse: got struct task_struct [noderef] __rcu *
kernel/sched/deadline.c:2318:43: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/deadline.c:2318:43: sparse: expected struct task_struct *p
kernel/sched/deadline.c:2318:43: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/deadline.c:2878:38: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *tsk @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/deadline.c:2878:38: sparse: expected struct task_struct *tsk
kernel/sched/deadline.c:2878:38: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/deadline.c:2880:23: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/deadline.c:2880:23: sparse: expected struct task_struct *p
kernel/sched/deadline.c:2880:23: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/deadline.c:2882:44: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct sched_dl_entity const *b @@ got struct sched_dl_entity [noderef] __rcu * @@
kernel/sched/deadline.c:2882:44: sparse: expected struct sched_dl_entity const *b
kernel/sched/deadline.c:2882:44: sparse: got struct sched_dl_entity [noderef] __rcu *
kernel/sched/deadline.c:3071:23: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/deadline.c:3071:23: sparse: struct task_struct [noderef] __rcu *
kernel/sched/deadline.c:3071:23: sparse: struct task_struct *
kernel/sched/deadline.c:3120:32: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/build_policy.c: note: in included file:
kernel/sched/syscalls.c:206:22: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/syscalls.c:206:22: sparse: struct task_struct [noderef] __rcu *
kernel/sched/syscalls.c:206:22: sparse: struct task_struct *
kernel/sched/build_policy.c: note: in included file:
kernel/sched/sched.h:2276:25: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2276:25: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2276:25: sparse: struct task_struct *
kernel/sched/sched.h:2276:25: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2276:25: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2276:25: sparse: struct task_struct *
kernel/sched/sched.h:2287:26: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2287:26: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2287:26: sparse: struct task_struct *
kernel/sched/build_policy.c: note: in included file:
kernel/sched/rt.c:2413:45: sparse: sparse: dereference of noderef expression kernel/sched/build_policy.c: note: in included file:
>> kernel/sched/sched.h:2627:35: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/build_policy.c: note: in included file:
kernel/sched/rt.c:2456:32: sparse: sparse: dereference of noderef expression
kernel/sched/rt.c:2457:32: sparse: sparse: dereference of noderef expression
kernel/sched/build_policy.c: note: in included file:
kernel/sched/sched.h:2276:25: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2276:25: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2276:25: sparse: struct task_struct *
kernel/sched/sched.h:2287:26: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2287:26: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2287:26: sparse: struct task_struct *
kernel/sched/sched.h:2276:25: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2276:25: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2276:25: sparse: struct task_struct *
kernel/sched/sched.h:2276:25: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2276:25: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2276:25: sparse: struct task_struct *
kernel/sched/sched.h:2287:26: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2287:26: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2287:26: sparse: struct task_struct *
kernel/sched/sched.h:2287:26: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2287:26: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2287:26: sparse: struct task_struct *
kernel/sched/sched.h:2476:9: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2476:9: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2476:9: sparse: struct task_struct *
kernel/sched/sched.h:2287:26: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2287:26: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2287:26: sparse: struct task_struct *
kernel/sched/sched.h:2476:9: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2476:9: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2476:9: sparse: struct task_struct *
kernel/sched/build_policy.c: note: in included file:
kernel/sched/syscalls.c:1296:6: sparse: sparse: context imbalance in 'sched_getaffinity' - different lock contexts for basic block kernel/sched/build_policy.c: note: in included file:
kernel/sched/rt.c:1767:15: sparse: sparse: dereference of noderef expression
vim +2627 kernel/sched/sched.h
04746ed80bcf31 Ingo Molnar 2024-04-07 2624
a7c81556ec4d34 Peter Zijlstra 2020-09-28 2625 static inline struct task_struct *get_push_task(struct rq *rq)
a7c81556ec4d34 Peter Zijlstra 2020-09-28 2626 {
af0c8b2bf67b25 Peter Zijlstra 2024-10-09 @2627 struct task_struct *p = rq->donor;
a7c81556ec4d34 Peter Zijlstra 2020-09-28 2628
5cb9eaa3d274f7 Peter Zijlstra 2020-11-17 2629 lockdep_assert_rq_held(rq);
a7c81556ec4d34 Peter Zijlstra 2020-09-28 2630
a7c81556ec4d34 Peter Zijlstra 2020-09-28 2631 if (rq->push_busy)
a7c81556ec4d34 Peter Zijlstra 2020-09-28 2632 return NULL;
a7c81556ec4d34 Peter Zijlstra 2020-09-28 2633
a7c81556ec4d34 Peter Zijlstra 2020-09-28 2634 if (p->nr_cpus_allowed == 1)
a7c81556ec4d34 Peter Zijlstra 2020-09-28 2635 return NULL;
a7c81556ec4d34 Peter Zijlstra 2020-09-28 2636
e681dcbaa4b284 Sebastian Andrzej Siewior 2021-08-26 2637 if (p->migration_disabled)
e681dcbaa4b284 Sebastian Andrzej Siewior 2021-08-26 2638 return NULL;
e681dcbaa4b284 Sebastian Andrzej Siewior 2021-08-26 2639
a7c81556ec4d34 Peter Zijlstra 2020-09-28 2640 rq->push_busy = true;
a7c81556ec4d34 Peter Zijlstra 2020-09-28 2641 return get_task_struct(p);
a7c81556ec4d34 Peter Zijlstra 2020-09-28 2642 }
a7c81556ec4d34 Peter Zijlstra 2020-09-28 2643 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
RE: [PATCH 3/4] sched/rt: Split root_domain->rto_count to per-NUMA-node counters
Quoted 219 lines
-----Original Message-----
From: lkp <[email protected]>
Sent: Tuesday, July 8, 2025 1:34 PM
To: Deng, Pan <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Li, Tianyou <[email protected]>; [email protected]; Chen, Yu C <[email protected]>; Deng, Pan <[email protected]>
Subject: Re: [PATCH 3/4] sched/rt: Splitroot_domain->rto_countto per-NUMA- node countersHi Pan,
kernel test robot noticed the following build warnings:
[auto build test WARNING on v6.16-rc5]
[also build test WARNING on linus/master] [cannot apply to tip/sched/core peterz-queue/sched/core tip/master tip/auto-latest next-20250704] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]url: https://github.com/intel-lab-lkp/linux/commits/Pan-Deng/sched-rt-
Optimize-cpupri_vec-layout-to-mitigate-cache-line-contention/20250707-131831
base: v6.16-rc5
patch link:
https://lore.kernel.org/r/2c1e1dbacaddd881f3cca340ece1f9268029b620.175185
2370.git.pan.deng%40intel.com
patch subject: [PATCH 3/4] sched/rt: Splitroot_domain->rto_countto per-NUMA- node counters
config: loongarch-randconfig-r112-20250708 (https://download.01.org/0day-
ci/archive/20250708/[email protected]/config)
compiler: loongarch64-linux-gcc (GCC) 15.1.0
reproduce: (https://download.01.org/0day-
ci/archive/20250708/[email protected]/reproduce)If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]> | Closes: | https://lore.kernel.org/oe-kbuild-all/202507081317.4IdE2euZ-lkp@intel. | com/sparse warnings: (new ones prefixed by >>)
kernel/sched/rt.c:1679:45: sparse: sparse: incorrect type in argument 2
(different address spaces) @@ expected structtask_struct*p @@ got structtask_struct[noderef]__rcu*donor @@
kernel/sched/rt.c:1679:45: sparse: expected structtask_struct*p
kernel/sched/rt.c:1679:45: sparse: got structtask_struct[noderef]__rcu
*donor
kernel/sched/rt.c:1722:39: sparse: sparse: incorrect type in initializer (different
address spaces) @@ expected structtask_struct*donor @@ got structtask_struct[noderef]__rcu*donor @@
kernel/sched/rt.c:1722:39: sparse: expected structtask_struct*donor
kernel/sched/rt.c:1722:39: sparse: got structtask_struct[noderef]__rcu
*donor
kernel/sched/rt.c:1742:64: sparse: sparse: incorrect type in argument 1
(different address spaces) @@ expected structtask_struct*tsk @@ got
structtask_struct[noderef]__rcu*curr @@
kernel/sched/rt.c:1742:64: sparse: expected structtask_struct*tsk
kernel/sched/rt.c:1742:64: sparse: got structtask_struct[noderef]__rcu
*curr
kernel/sched/rt.c:2084:40: sparse: sparse: incorrect type in argument 1
(different address spaces) @@ expected structtask_struct*task @@ got
structtask_struct[noderef]__rcu*curr @@
kernel/sched/rt.c:2084:40: sparse: expected structtask_struct*task
kernel/sched/rt.c:2084:40: sparse: got structtask_struct[noderef]__rcu
*curr
kernel/sched/rt.c:2107:13: sparse: sparse: incompatible types in comparison
expression (different address spaces):kernel/sched/rt.c:2107:13: sparse: struct task_struct * kernel/sched/rt.c:2107:13: sparse: struct task_struct [noderef] __rcu * kernel/sched/rt.c:2453:54: sparse: sparse: incorrect type in argument 1(different address spaces) @@ expected struct
task_struct*tsk @@ got
structtask_struct[noderef]__rcu*curr @@
kernel/sched/rt.c:2453:54: sparse: expected structtask_struct*tsk
kernel/sched/rt.c:2453:54: sparse: got structtask_struct[noderef]__rcu
*curr
kernel/sched/rt.c:2455:40: sparse: sparse: incorrect type in argument 1
(different address spaces) @@ expected structtask_struct*p @@ got structtask_struct[noderef]__rcu*donor @@
kernel/sched/rt.c:2455:40: sparse: expected structtask_struct*p
kernel/sched/rt.c:2455:40: sparse: got structtask_struct[noderef]__rcu
*donor
kernel/sched/rt.c:2455:62: sparse: sparse: incorrect type in argument 1
(different address spaces) @@ expected structtask_struct*p @@ got structtask_struct[noderef]__rcu*donor @@
kernel/sched/rt.c:2455:62: sparse: expected structtask_struct*p
kernel/sched/rt.c:2455:62: sparse: got structtask_struct[noderef]__rcu
*donor
kernel/sched/build_policy.c: note: in included file:
kernel/sched/deadline.c:2717:23: sparse: sparse: incorrect type in argument 1
(different address spaces) @@ expected structtask_struct*p @@ got structtask_struct[noderef]__rcu*donor @@
kernel/sched/deadline.c:2717:23: sparse: expected structtask_struct*p
kernel/sched/deadline.c:2717:23: sparse: got structtask_struct[noderef]__rcu*donor
kernel/sched/deadline.c:2727:13: sparse: sparse: incompatible types in
comparison expression (different address spaces):
kernel/sched/deadline.c:2727:13: sparse: structtask_struct*
kernel/sched/deadline.c:2727:13: sparse: structtask_struct[noderef]__rcu*
kernel/sched/deadline.c:2833:25: sparse: sparse: incompatible types in
comparison expression (different address spaces):
kernel/sched/deadline.c:2833:25: sparse: structtask_struct*
kernel/sched/deadline.c:2833:25: sparse: structtask_struct[noderef]__rcu*
kernel/sched/deadline.c:2357:42: sparse: sparse: incorrect type in argument 2
(different address spaces) @@ expected structsched_dl_entityconst *b @@
got structsched_dl_entity[noderef]__rcu* @@
kernel/sched/deadline.c:2357:42: sparse: expected structsched_dl_entity
const *b
kernel/sched/deadline.c:2357:42: sparse: got structsched_dl_entity
[noderef]__rcu*
kernel/sched/deadline.c:2368:38: sparse: sparse: incorrect type in argument 1
(different address spaces) @@ expected structtask_struct*tsk @@ got
structtask_struct[noderef]__rcu*curr @@
kernel/sched/deadline.c:2368:38: sparse: expected structtask_struct*tsk
kernel/sched/deadline.c:2368:38: sparse: got structtask_struct[noderef]__rcu*curr
kernel/sched/deadline.c:1262:39: sparse: sparse: incorrect type in argument 1
(different address spaces) @@ expected structtask_struct*p @@ got structtask_struct[noderef]__rcu*curr @@
kernel/sched/deadline.c:1262:39: sparse: expected structtask_struct*p
kernel/sched/deadline.c:1262:39: sparse: got structtask_struct[noderef]__rcu*curr
kernel/sched/deadline.c:1262:85: sparse: sparse: incorrect type in argument 2
(different address spaces) @@ expected structsched_dl_entityconst *b @@
got structsched_dl_entity[noderef]__rcu* @@
kernel/sched/deadline.c:1262:85: sparse: expected structsched_dl_entity
const *b
kernel/sched/deadline.c:1262:85: sparse: got structsched_dl_entity
[noderef]__rcu*
kernel/sched/deadline.c:1362:23: sparse: sparse: incorrect type in argument 1
(different address spaces) @@ expected structtask_struct*p @@ got structtask_struct[noderef]__rcu*donor @@
kernel/sched/deadline.c:1362:23: sparse: expected structtask_struct*p
kernel/sched/deadline.c:1362:23: sparse: got structtask_struct[noderef]__rcu*donor
kernel/sched/deadline.c:1671:31: sparse: sparse: incorrect type in argument 1
(different address spaces) @@ expected structtask_struct*p @@ got structtask_struct[noderef]__rcu*curr @@
kernel/sched/deadline.c:1671:31: sparse: expected structtask_struct*p
kernel/sched/deadline.c:1671:31: sparse: got structtask_struct[noderef]__rcu*curr
kernel/sched/deadline.c:1671:70: sparse: sparse: incorrect type in argument 2
(different address spaces) @@ expected structsched_dl_entityconst *b @@
got structsched_dl_entity[noderef]__rcu* @@
kernel/sched/deadline.c:1671:70: sparse: expected structsched_dl_entity
const *b
kernel/sched/deadline.c:1671:70: sparse: got structsched_dl_entity
[noderef]__rcu*
kernel/sched/deadline.c:1760:39: sparse: sparse: incorrect type in initializer
(different address spaces) @@ expected structtask_struct*donor @@ got
structtask_struct[noderef]__rcu*donor @@
kernel/sched/deadline.c:1760:39: sparse: expected structtask_struct*donor
kernel/sched/deadline.c:1760:39: sparse: got structtask_struct[noderef]__rcu*donor
kernel/sched/deadline.c:2578:9: sparse: sparse: incorrect type in assignment
(different address spaces) @@ expected structsched_domain*[assigned] sd
@@ got structsched_domain[noderef]__rcu*parent @@
kernel/sched/deadline.c:2578:9: sparse: expected structsched_domain
*[assigned] sd
kernel/sched/deadline.c:2578:9: sparse: got structsched_domain[noderef]__rcu*parent
kernel/sched/deadline.c:2242:14: sparse: sparse: incorrect type in assignment
(different address spaces) @@ expected structtask_struct*curr @@ got
structtask_struct[noderef]__rcu* @@
kernel/sched/deadline.c:2242:14: sparse: expected structtask_struct*curr
kernel/sched/deadline.c:2242:14: sparse: got structtask_struct[noderef]__rcu*
kernel/sched/deadline.c:2243:15: sparse: sparse: incorrect type in assignment
(different address spaces) @@ expected structtask_struct*donor @@ got
structtask_struct[noderef]__rcu* @@
kernel/sched/deadline.c:2243:15: sparse: expected structtask_struct*donor
kernel/sched/deadline.c:2243:15: sparse: got structtask_struct[noderef]__rcu*
kernel/sched/deadline.c:2318:43: sparse: sparse: incorrect type in argument 2
(different address spaces) @@ expected structtask_struct*p @@ got structtask_struct[noderef]__rcu*donor @@
kernel/sched/deadline.c:2318:43: sparse: expected structtask_struct*p
kernel/sched/deadline.c:2318:43: sparse: got structtask_struct[noderef]__rcu*donor
kernel/sched/deadline.c:2878:38: sparse: sparse: incorrect type in argument 1
(different address spaces) @@ expected structtask_struct*tsk @@ got
structtask_struct[noderef]__rcu*curr @@
kernel/sched/deadline.c:2878:38: sparse: expected structtask_struct*tsk
kernel/sched/deadline.c:2878:38: sparse: got structtask_struct[noderef]__rcu*curr
kernel/sched/deadline.c:2880:23: sparse: sparse: incorrect type in argument 1
(different address spaces) @@ expected structtask_struct*p @@ got structtask_struct[noderef]__rcu*donor @@
kernel/sched/deadline.c:2880:23: sparse: expected structtask_struct*p
kernel/sched/deadline.c:2880:23: sparse: got structtask_struct[noderef]__rcu*donor
kernel/sched/deadline.c:2882:44: sparse: sparse: incorrect type in argument 2
(different address spaces) @@ expected structsched_dl_entityconst *b @@
got structsched_dl_entity[noderef]__rcu* @@
kernel/sched/deadline.c:2882:44: sparse: expected structsched_dl_entity
const *b
kernel/sched/deadline.c:2882:44: sparse: got structsched_dl_entity
[noderef]__rcu*
kernel/sched/deadline.c:3071:23: sparse: sparse: incompatible types in
comparison expression (different address spaces):
kernel/sched/deadline.c:3071:23: sparse: structtask_struct[noderef]__rcu*
kernel/sched/deadline.c:3071:23: sparse: structtask_struct*
kernel/sched/deadline.c:3120:32: sparse: sparse: incorrect type in argument 1
(different address spaces) @@ expected structtask_struct*p @@ got structtask_struct[noderef]__rcu*curr @@
kernel/sched/build_policy.c: note: in included file:
kernel/sched/syscalls.c:206:22: sparse: sparse: incompatible types in
comparison expression (different address spaces):
kernel/sched/syscalls.c:206:22: sparse: structtask_struct[noderef]__rcu*
kernel/sched/syscalls.c:206:22: sparse: structtask_struct*
kernel/sched/build_policy.c: note: in included file:
kernel/sched/sched.h:2276:25: sparse: sparse: incompatible types in
comparison expression (different address spaces):
kernel/sched/sched.h:2276:25: sparse: structtask_struct[noderef]__rcu*
kernel/sched/sched.h:2276:25: sparse: structtask_struct*
kernel/sched/sched.h:2276:25: sparse: sparse: incompatible types in
comparison expression (different address spaces):
kernel/sched/sched.h:2276:25: sparse: structtask_struct[noderef]__rcu*
Quoted 9 lines
kernel/sched/sched.h:2276:25: sparse: structtask_struct*
kernel/sched/sched.h:2287:26: sparse: sparse: incompatible types in
comparison expression (different address spaces):
kernel/sched/sched.h:2287:26: sparse: structtask_struct[noderef]__rcu*
kernel/sched/sched.h:2287:26: sparse: structtask_struct*
kernel/sched/build_policy.c: note: in included file:
kernel/sched/rt.c:2413:45: sparse: sparse: dereference of noderef expressionkernel/sched/build_policy.c: note: in included file:Quoted 1 line
Quoted 1 line
kernel/sched/sched.h:2627:35: sparse: sparse: incorrect type in initializer
This warning is not about the change we made, @lkp, could you please check it?
Quoted 81 lines
(different address spaces) @@ expected struct
task_struct*p @@ got structtask_struct[noderef]__rcu*donor @@
kernel/sched/build_policy.c: note: in included file:
kernel/sched/rt.c:2456:32: sparse: sparse: dereference of noderef expressionkernel/sched/rt.c:2457:32: sparse: sparse: dereference of noderef expressionkernel/sched/build_policy.c: note: in included file:
kernel/sched/sched.h:2276:25: sparse: sparse: incompatible types in
comparison expression (different address spaces):
kernel/sched/sched.h:2276:25: sparse: structtask_struct[noderef]__rcu*
kernel/sched/sched.h:2276:25: sparse: structtask_struct*
kernel/sched/sched.h:2287:26: sparse: sparse: incompatible types in
comparison expression (different address spaces):
kernel/sched/sched.h:2287:26: sparse: structtask_struct[noderef]__rcu*
kernel/sched/sched.h:2287:26: sparse: structtask_struct*
kernel/sched/sched.h:2276:25: sparse: sparse: incompatible types in
comparison expression (different address spaces):
kernel/sched/sched.h:2276:25: sparse: structtask_struct[noderef]__rcu*
kernel/sched/sched.h:2276:25: sparse: structtask_struct*
kernel/sched/sched.h:2276:25: sparse: sparse: incompatible types in
comparison expression (different address spaces):
kernel/sched/sched.h:2276:25: sparse: structtask_struct[noderef]__rcu*
kernel/sched/sched.h:2276:25: sparse: structtask_struct*
kernel/sched/sched.h:2287:26: sparse: sparse: incompatible types in
comparison expression (different address spaces):
kernel/sched/sched.h:2287:26: sparse: structtask_struct[noderef]__rcu*
kernel/sched/sched.h:2287:26: sparse: structtask_struct*
kernel/sched/sched.h:2287:26: sparse: sparse: incompatible types in
comparison expression (different address spaces):
kernel/sched/sched.h:2287:26: sparse: structtask_struct[noderef]__rcu*
kernel/sched/sched.h:2287:26: sparse: structtask_struct*
kernel/sched/sched.h:2476:9: sparse: sparse: incompatible types in comparison
expression (different address spaces):
kernel/sched/sched.h:2476:9: sparse: structtask_struct[noderef]__rcu*
kernel/sched/sched.h:2476:9: sparse: structtask_struct*
kernel/sched/sched.h:2287:26: sparse: sparse: incompatible types in
comparison expression (different address spaces):
kernel/sched/sched.h:2287:26: sparse: structtask_struct[noderef]__rcu*
kernel/sched/sched.h:2287:26: sparse: structtask_struct*
kernel/sched/sched.h:2476:9: sparse: sparse: incompatible types in comparison
expression (different address spaces):
kernel/sched/sched.h:2476:9: sparse: structtask_struct[noderef]__rcu*
kernel/sched/sched.h:2476:9: sparse: structtask_struct*
kernel/sched/build_policy.c: note: in included file:
kernel/sched/syscalls.c:1296:6: sparse: sparse: context imbalance in
'sched_getaffinity' - different lock contexts for basic block
kernel/sched/build_policy.c: note: in included file:
kernel/sched/rt.c:1767:15: sparse: sparse: dereference of noderef expressionvim +2627
kernel/sched/sched.h04746ed80bcf31 Ingo Molnar 2024-04-07 2624 a7c81556ec4d34 Peter Zijlstra 2020-09-28 2625 static inline struct task_struct *get_push_task(struct rq *rq) a7c81556ec4d34 Peter Zijlstra 2020-09-28 2626 { af0c8b2bf67b25 Peter Zijlstra 2024-10-09 @2627 struct task_struct *p = rq->donor; a7c81556ec4d34 Peter Zijlstra 2020-09-28 2628 5cb9eaa3d274f7 Peter Zijlstra 2020-11-17 2629 lockdep_assert_rq_held(rq); a7c81556ec4d34 Peter Zijlstra 2020-09-28 2630 a7c81556ec4d34 Peter Zijlstra 2020-09-28 2631 if (rq->push_busy) a7c81556ec4d34 Peter Zijlstra 2020-09-28 2632 return NULL; a7c81556ec4d34 Peter Zijlstra 2020-09-28 2633 a7c81556ec4d34 Peter Zijlstra 2020-09-28 2634 if (p->nr_cpus_allowed == 1) a7c81556ec4d34 Peter Zijlstra 2020-09-28 2635 return NULL; a7c81556ec4d34 Peter Zijlstra 2020-09-28 2636 e681dcbaa4b284 Sebastian Andrzej Siewior 2021-08-26 2637 if (p-Quoted 1 line
migration_disabled)
e681dcbaa4b284Sebastian Andrzej Siewior 2021-08-26 2638 return
NULL;e681dcbaa4b284 Sebastian Andrzej Siewior 2021-08-26 2639 a7c81556ec4d34 Peter Zijlstra 2020-09-28 2640 rq->push_busy = true; a7c81556ec4d34 Peter Zijlstra 2020-09-28 2641 return get_task_struct(p); a7c81556ec4d34 Peter Zijlstra 2020-09-28 2642 } a7c81556ec4d34 Peter Zijlstra 2020-09-28 2643--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
RE: [PATCH 3/4] sched/rt: Split root_domain->rto_count to per-NUMA-node counters
Quoted 14 lines
Quoted 14 lines
comparison expression (different address spaces):
kernel/sched/sched.h:2276:25: sparse: structtask_struct[noderef]__rcu*
kernel/sched/sched.h:2276:25: sparse: structtask_struct*
kernel/sched/sched.h:2276:25: sparse: sparse: incompatible types in
comparison expression (different address spaces):
kernel/sched/sched.h:2276:25: sparse: structtask_struct[noderef]__rcu*
kernel/sched/sched.h:2276:25: sparse: structtask_struct*
kernel/sched/sched.h:2287:26: sparse: sparse: incompatible types in
comparison expression (different address spaces):
kernel/sched/sched.h:2287:26: sparse: structtask_struct[noderef]__rcu*
kernel/sched/sched.h:2287:26: sparse: structtask_struct*
kernel/sched/build_policy.c: note: in included file:
kernel/sched/rt.c:2413:45: sparse: sparse: dereference of noderef expressionkernel/sched/build_policy.c: note: in included file:
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
kernel/sched/sched.h:2627:35: sparse: sparse: incorrect type in initializerThis warning is not about the change we made, @lkp, could you please check it?
Sorry for this false report, it should not be related to your changes. We will follow up to figure out what is wrong during the bisections. Sorry for wasting your time.
Re: [PATCH 4/4] sched/rt: Split cpupri_vec->cpumask to per NUMA node to reduce contention
In reply to Pan DengQuoted 14 lines · Pan Deng
When running a multi-instance FFmpeg workload on HCC system, significant contention is observed on bitmap of
cpupri_vec->cpumask.The SUT is a 2-socket machine with 240 physical cores and 480 logical CPUs. 60 FFmpeg instances are launched, each pinned to 4 physical cores (8 logical CPUs) for transcoding tasks. Sub-threads use RT priority 99 with FIFO scheduling. FPS is used as score.
perf c2c tool reveals:
cpumask (bitmap) cache line ofcpupri_vec->mask:
- bits are loaded duringcpupri_find
- bits are stored duringcpupri_set
- cycles per load: ~2.2K to 8.7K
Quoted 9 lines
This change splits
cpupri_vec->cpumaskinto per-NUMA-node data to mitigate false sharing.As a result:
- FPS improves by ~3.8%
- Kernel cycles% drops from ~20% to ~18.7%
- Cache line contention is mitigated, perf-c2c shows cycles per load drops from ~2.2K-8.7K to ~0.5K-2.2K
This brings noticeable improvement for RT workload, and it would be even more convincing if we can have try on normal task workload, at least not bring regression(schbench/hackbenc, etc).
thanks,
Chenyu
Quoted 2 lines
Note:
CONFIG_CPUMASK_OFFSTACK=nremains unchanged.
RE: [PATCH 4/4] sched/rt: Split cpupri_vec->cpumask to per NUMA node to reduce contention
Quoted 36 lines
-----Original Message-----
From: Chen, Yu C <[email protected]>
Sent: Monday, July 21, 2025 7:24 PM
To: Deng, Pan <[email protected]>
Cc: [email protected]; Li, Tianyou <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 4/4] sched/rt: Splitcpupri_vec->cpumaskto per NUMA node to reduce contentionQuoted 15 lines · Pan Deng
When running a multi-instance FFmpeg workload on HCC system, significant contention is observed on bitmap of
cpupri_vec->cpumask.The SUT is a 2-socket machine with 240 physical cores and 480 logical CPUs. 60 FFmpeg instances are launched, each pinned to 4 physical cores
(8 logical CPUs) for transcoding tasks. Sub-threads use RT priority 99 with FIFO scheduling. FPS is used as score.perf c2c tool reveals:
cpumask (bitmap) cache line ofcpupri_vec->mask:
- bits are loaded duringcpupri_find
- bits are stored duringcpupri_set
- cycles per load: ~2.2K to 8.7KQuoted 9 lines
This change splits
cpupri_vec->cpumaskinto per-NUMA-node data to mitigate false sharing.As a result:
- FPS improves by ~3.8%
- Kernel cycles% drops from ~20% to ~18.7%
- Cache line contention is mitigated, perf-c2c shows cycles per load drops from ~2.2K-8.7K to ~0.5K-2.2K
Quoted 4 lines
This brings noticeable improvement for RT workload, and it would be even more convincing if we can have try on normal task workload, at least not bring
regression(schbench/hackbenc, etc).
Thanks Yu, hackbench and schbench data will be provided later.
Quoted 6 lines
thanks,
ChenyuQuoted 2 lines
Note:
CONFIG_CPUMASK_OFFSTACK=nremains unchanged.
RE: [PATCH 4/4] sched/rt: Split cpupri_vec->cpumask to per NUMA node to reduce contention
Quoted 45 lines
-----Original Message-----
From: Deng, Pan
Sent: Tuesday, July 22, 2025 10:47 PM
To: Chen, Yu C <[email protected]>
Cc: [email protected]; Li, Tianyou <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 4/4] sched/rt: Splitcpupri_vec->cpumaskto per NUMA node to reduce contentionQuoted 35 lines
-----Original Message-----
From: Chen, Yu C <[email protected]>
Sent: Monday, July 21, 2025 7:24 PM
To: Deng, Pan <[email protected]>
Cc: [email protected]; Li, Tianyou <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 4/4] sched/rt: Splitcpupri_vec->cpumaskto per NUMA node to reduce contentionQuoted 24 lines · Pan Deng
When running a multi-instance FFmpeg workload on HCC system, significant contention is observed on bitmap of
cpupri_vec->cpumask.The SUT is a 2-socket machine with 240 physical cores and 480 logical CPUs. 60 FFmpeg instances are launched, each pinned to 4 physical cores
(8 logical CPUs) for transcoding tasks. Sub-threads use RT priority 99 with FIFO scheduling. FPS is used as score.perf c2c tool reveals:
cpumask (bitmap) cache line ofcpupri_vec->mask:
- bits are loaded duringcpupri_find
- bits are stored duringcpupri_set
- cycles per load: ~2.2K to 8.7KThis change splits
cpupri_vec->cpumaskinto per-NUMA-node data to mitigate false sharing.As a result:
- FPS improves by ~3.8%
- Kernel cycles% drops from ~20% to ~18.7%
- Cache line contention is mitigated, perf-c2c shows cycles per load drops from ~2.2K-8.7K to ~0.5K-2.2K
Quoted 9 lines
Quoted 2 lines
This brings noticeable improvement for RT workload, and it would be even more convincing if we can have try on normal task workload, at least not
bring
Quoted 2 lines
regression(schbench/hackbenc, etc).Thanks Yu, hackbench and schbench data will be provided later.
TLDR;
====
Hackbench, both old and new version schbench were evaluted on SUT with 2-socket/6 NUMA nodes/240 physical cores/480 logical CPUs. No regressions were detected for patch 1-4. In addition, symbol-level analysis from perf record -a profiling data indicates that the changes introduced in patch 1-4 are unlikely to cause regressions in hackbench or schbench.
Details
=======
Hackbench
=========
The workload is ran by test framework
https://github.com/yu-chen-surf/schedtests, procedures:
1. Reboot system to run a workload.
2. Run 5 iterations of 1st configuration with 30s cool down period.
3. Run 5 iterations of 2nd configuration..
...
The test results are as follows: regressions exceeding -10% are marked with ** at the end of the line. However, when re-running the tests using the test framework or a vanilla workload, the regressions could not be reproduced.
Notes: 15/30/45 are fds# as well as process/thread pairs in 1 group. Patch 1
case load baseline(std%) patch1%( std%)
process-pipe-15 1-groups 1.00 ( 14.03) -8.81 ( 6.53) process-pipe-15 2-groups 1.00 ( 3.46) +1.82 ( 2.59) process-pipe-15 4-groups 1.00 ( 6.20) +8.60 ( 5.59) process-pipe-15 8-groups 1.00 ( 2.41) -0.21 ( 3.22) process-pipe-30 1-groups 1.00 ( 2.51) +2.24 ( 3.12) process-pipe-30 2-groups 1.00 ( 3.86) -0.58 ( 2.46) process-pipe-30 4-groups 1.00 ( 2.19) -1.81 ( 1.05) process-pipe-30 8-groups 1.00 ( 1.69) +0.52 ( 3.01) process-pipe-45 1-groups 1.00 ( 1.63) +1.63 ( 1.23) process-pipe-45 2-groups 1.00 ( 0.79) +0.08 ( 1.82) process-pipe-45 4-groups 1.00 ( 1.62) -0.06 ( 0.63) process-pipe-45 8-groups 1.00 ( 1.66) -4.12 ( 3.27) process-sockets-15 1-groups 1.00 ( 3.57) +2.36 ( 5.15) process-sockets-15 2-groups 1.00 ( 3.59) -1.33 ( 6.86) process-sockets-15 4-groups 1.00 ( 7.10) +5.44 ( 6.97) process-sockets-15 8-groups 1.00 ( 2.63) -3.05 ( 1.94) process-sockets-30 1-groups 1.00 ( 3.73) -2.69 ( 4.89) process-sockets-30 2-groups 1.00 ( 3.90) -4.25 ( 3.94) process-sockets-30 4-groups 1.00 ( 1.03) -1.58 ( 1.51) process-sockets-30 8-groups 1.00 ( 0.48) +1.09 ( 0.68) process-sockets-45 1-groups 1.00 ( 0.62) -2.25 ( 0.57) process-sockets-45 2-groups 1.00 ( 2.56) -0.61 ( 0.63) process-sockets-45 4-groups 1.00 ( 0.57) -0.51 ( 0.79) process-sockets-45 8-groups 1.00 ( 0.18) -5.23 ( 2.18) threads-pipe-15 1-groups 1.00 ( 5.30) -1.47 ( 5.38) threads-pipe-15 2-groups 1.00 ( 7.97) -1.31 ( 8.61) threads-pipe-15 4-groups 1.00 ( 4.94) -3.31 ( 5.48) threads-pipe-15 8-groups 1.00 ( 1.69) +7.28 ( 5.54) threads-pipe-30 1-groups 1.00 ( 5.12) -1.58 ( 4.82) threads-pipe-30 2-groups 1.00 ( 1.63) +3.29 ( 1.72) threads-pipe-30 4-groups 1.00 ( 3.41) +3.05 ( 3.22) threads-pipe-30 8-groups 1.00 ( 2.85) +1.58 ( 4.05) threads-pipe-45 1-groups 1.00 ( 5.13) -0.78 ( 6.78) threads-pipe-45 2-groups 1.00 ( 1.92) -2.87 ( 1.27) threads-pipe-45 4-groups 1.00 ( 2.41) -4.37 ( 1.23) threads-pipe-45 8-groups 1.00 ( 1.81) +1.85 ( 1.54) threads-sockets-15 1-groups 1.00 ( 4.72) -0.73 ( 2.75) threads-sockets-15 2-groups 1.00 ( 3.05) +3.09 ( 3.39) threads-sockets-15 4-groups 1.00 ( 5.92) +0.87 ( 2.25) threads-sockets-15 8-groups 1.00 ( 3.75) -7.24 ( 3.34) threads-sockets-30 1-groups 1.00 ( 5.96) -6.27 ( 3.35) threads-sockets-30 2-groups 1.00 ( 1.68) -1.78 ( 3.60) threads-sockets-30 4-groups 1.00 ( 5.02) -0.95 ( 3.60) threads-sockets-30 8-groups 1.00 ( 0.41) -3.09 ( 2.03) threads-sockets-45 1-groups 1.00 ( 2.55) -1.32 ( 1.37) threads-sockets-45 2-groups 1.00 ( 3.53) -0.46 ( 3.99) threads-sockets-45 4-groups 1.00 ( 0.51) +0.67 ( 0.74) threads-sockets-45 8-groups 1.00 ( 3.01) -16.85 ( 2.13) **
Patch 2
case load baseline(std%) patch2%( std%)
process-pipe-15 1-groups 1.00 ( 14.03) -3.32 ( 11.34) process-pipe-15 2-groups 1.00 ( 3.46) +2.19 ( 7.27) process-pipe-15 4-groups 1.00 ( 6.20) +2.01 ( 2.83) process-pipe-15 8-groups 1.00 ( 2.41) +1.65 ( 4.39) process-pipe-30 1-groups 1.00 ( 2.51) -0.88 ( 3.26) process-pipe-30 2-groups 1.00 ( 3.86) +2.25 ( 3.21) process-pipe-30 4-groups 1.00 ( 2.19) +0.20 ( 1.72) process-pipe-30 8-groups 1.00 ( 1.69) +0.85 ( 0.61) process-pipe-45 1-groups 1.00 ( 1.63) +3.10 ( 4.01) process-pipe-45 2-groups 1.00 ( 0.79) -1.00 ( 0.69) process-pipe-45 4-groups 1.00 ( 1.62) +0.07 ( 0.63) process-pipe-45 8-groups 1.00 ( 1.66) +0.20 ( 1.47) process-sockets-15 1-groups 1.00 ( 3.57) -5.44 ( 3.45) process-sockets-15 2-groups 1.00 ( 3.59) +1.00 ( 4.35) process-sockets-15 4-groups 1.00 ( 7.10) +0.46 ( 4.45) process-sockets-15 8-groups 1.00 ( 2.63) -1.48 ( 4.56) process-sockets-30 1-groups 1.00 ( 3.73) -0.17 ( 3.57) process-sockets-30 2-groups 1.00 ( 3.90) +3.83 ( 7.54) process-sockets-30 4-groups 1.00 ( 1.03) -2.35 ( 6.11) process-sockets-30 8-groups 1.00 ( 0.48) -0.43 ( 0.79) process-sockets-45 1-groups 1.00 ( 0.62) -2.24 ( 1.63) process-sockets-45 2-groups 1.00 ( 2.56) -1.41 ( 3.17) process-sockets-45 4-groups 1.00 ( 0.57) -0.29 ( 0.33) process-sockets-45 8-groups 1.00 ( 0.18) -6.05 ( 1.55) threads-pipe-15 1-groups 1.00 ( 5.30) -5.83 ( 7.96) threads-pipe-15 2-groups 1.00 ( 7.97) -3.74 ( 4.22) threads-pipe-15 4-groups 1.00 ( 4.94) -2.23 ( 5.75) threads-pipe-15 8-groups 1.00 ( 1.69) +0.21 ( 3.08) threads-pipe-30 1-groups 1.00 ( 5.12) -5.73 ( 4.97) threads-pipe-30 2-groups 1.00 ( 1.63) -1.76 ( 4.49) threads-pipe-30 4-groups 1.00 ( 3.41) -0.99 ( 2.50) threads-pipe-30 8-groups 1.00 ( 2.85) +0.71 ( 1.04) threads-pipe-45 1-groups 1.00 ( 5.13) +0.08 ( 5.72) threads-pipe-45 2-groups 1.00 ( 1.92) -1.78 ( 1.30) threads-pipe-45 4-groups 1.00 ( 2.41) -3.79 ( 0.81) threads-pipe-45 8-groups 1.00 ( 1.81) -3.62 ( 1.41) threads-sockets-15 1-groups 1.00 ( 4.72) +2.52 ( 2.66) threads-sockets-15 2-groups 1.00 ( 3.05) -7.59 ( 1.80) threads-sockets-15 4-groups 1.00 ( 5.92) +1.59 ( 7.12) threads-sockets-15 8-groups 1.00 ( 3.75) -0.34 ( 3.62) threads-sockets-30 1-groups 1.00 ( 5.96) -2.45 ( 4.89) threads-sockets-30 2-groups 1.00 ( 1.68) -0.61 ( 4.80) threads-sockets-30 4-groups 1.00 ( 5.02) -2.15 ( 8.62) threads-sockets-30 8-groups 1.00 ( 0.41) -17.32 ( 0.88) ** threads-sockets-45 1-groups 1.00 ( 2.55) -3.24 ( 3.37) threads-sockets-45 2-groups 1.00 ( 3.53) -1.38 ( 2.40) threads-sockets-45 4-groups 1.00 ( 0.51) -0.17 ( 0.85) threads-sockets-45 8-groups 1.00 ( 3.01) -14.59 ( 5.48) **
Patch 3
case load baseline(std%) patch3%( std%)
process-pipe-15 1-groups 1.00 ( 14.03) -10.18 ( 3.39) ** process-pipe-15 2-groups 1.00 ( 3.46) +5.18 ( 3.12) process-pipe-15 4-groups 1.00 ( 6.20) +8.63 ( 5.72) process-pipe-15 8-groups 1.00 ( 2.41) +5.37 ( 2.24) process-pipe-30 1-groups 1.00 ( 2.51) +5.53 ( 3.55) process-pipe-30 2-groups 1.00 ( 3.86) +5.70 ( 4.27) process-pipe-30 4-groups 1.00 ( 2.19) +3.95 ( 3.34) process-pipe-30 8-groups 1.00 ( 1.69) -3.38 ( 1.51) process-pipe-45 1-groups 1.00 ( 1.63) +5.19 ( 2.51) process-pipe-45 2-groups 1.00 ( 0.79) -0.63 ( 2.06) process-pipe-45 4-groups 1.00 ( 1.62) -5.83 ( 2.22) process-pipe-45 8-groups 1.00 ( 1.66) -6.13 ( 2.34) process-sockets-15 1-groups 1.00 ( 3.57) -1.51 ( 4.21) process-sockets-15 2-groups 1.00 ( 3.59) -1.30 ( 7.50) process-sockets-15 4-groups 1.00 ( 7.10) -1.80 ( 5.58) process-sockets-15 8-groups 1.00 ( 2.63) -1.68 ( 3.40) process-sockets-30 1-groups 1.00 ( 3.73) -7.74 ( 1.58) process-sockets-30 2-groups 1.00 ( 3.90) -1.98 ( 5.48) process-sockets-30 4-groups 1.00 ( 1.03) -0.33 ( 3.47) process-sockets-30 8-groups 1.00 ( 0.48) -0.40 ( 0.84) process-sockets-45 1-groups 1.00 ( 0.62) -0.21 ( 0.54) process-sockets-45 2-groups 1.00 ( 2.56) -1.97 ( 2.48) process-sockets-45 4-groups 1.00 ( 0.57) -0.61 ( 0.83) process-sockets-45 8-groups 1.00 ( 0.18) -5.09 ( 1.85) threads-pipe-15 1-groups 1.00 ( 5.30) +3.62 ( 11.04) threads-pipe-15 2-groups 1.00 ( 7.97) +8.08 ( 4.63) threads-pipe-15 4-groups 1.00 ( 4.94) +6.46 ( 5.27) threads-pipe-15 8-groups 1.00 ( 1.69) +2.68 ( 3.23) threads-pipe-30 1-groups 1.00 ( 5.12) +3.60 ( 7.09) threads-pipe-30 2-groups 1.00 ( 1.63) -0.80 ( 4.43) threads-pipe-30 4-groups 1.00 ( 3.41) +2.37 ( 2.16) threads-pipe-30 8-groups 1.00 ( 2.85) +4.17 ( 1.41) threads-pipe-45 1-groups 1.00 ( 5.13) +7.41 ( 4.48) threads-pipe-45 2-groups 1.00 ( 1.92) -1.40 ( 2.69) threads-pipe-45 4-groups 1.00 ( 2.41) -1.25 ( 2.15) threads-pipe-45 8-groups 1.00 ( 1.81) +1.62 ( 0.73) threads-sockets-15 1-groups 1.00 ( 4.72) +10.11 ( 7.95) threads-sockets-15 2-groups 1.00 ( 3.05) -8.41 ( 5.93) threads-sockets-15 4-groups 1.00 ( 5.92) -10.89 ( 4.29) ** threads-sockets-15 8-groups 1.00 ( 3.75) -7.66 ( 3.33) threads-sockets-30 1-groups 1.00 ( 5.96) -5.18 ( 2.77) threads-sockets-30 2-groups 1.00 ( 1.68) -4.91 ( 3.89) threads-sockets-30 4-groups 1.00 ( 5.02) -6.32 ( 4.19) threads-sockets-30 8-groups 1.00 ( 0.41) -11.73 ( 0.96) ** threads-sockets-45 1-groups 1.00 ( 2.55) -3.16 ( 1.97) threads-sockets-45 2-groups 1.00 ( 3.53) -0.21 ( 4.33) threads-sockets-45 4-groups 1.00 ( 0.51) -0.75 ( 2.07) threads-sockets-45 8-groups 1.00 ( 3.01) -20.52 ( 1.44) **
Patch 4
case load baseline(std%) patch4%( std%)
process-pipe-15 1-groups 1.00 ( 14.03) -2.68 ( 9.64) process-pipe-15 2-groups 1.00 ( 3.46) +1.82 ( 7.55) process-pipe-15 4-groups 1.00 ( 6.20) +3.67 ( 8.17) process-pipe-15 8-groups 1.00 ( 2.41) +1.87 ( 0.92) process-pipe-30 1-groups 1.00 ( 2.51) -3.34 ( 3.96) process-pipe-30 2-groups 1.00 ( 3.86) -0.33 ( 3.53) process-pipe-30 4-groups 1.00 ( 2.19) -3.22 ( 1.31) process-pipe-30 8-groups 1.00 ( 1.69) -1.95 ( 1.07) process-pipe-45 1-groups 1.00 ( 1.63) +0.63 ( 2.86) process-pipe-45 2-groups 1.00 ( 0.79) -1.27 ( 1.39) process-pipe-45 4-groups 1.00 ( 1.62) -2.04 ( 1.87) process-pipe-45 8-groups 1.00 ( 1.66) -1.45 ( 3.20) process-sockets-15 1-groups 1.00 ( 3.57) -9.16 ( 5.33) process-sockets-15 2-groups 1.00 ( 3.59) -1.83 ( 5.36) process-sockets-15 4-groups 1.00 ( 7.10) +7.55 ( 6.34) process-sockets-15 8-groups 1.00 ( 2.63) -2.98 ( 5.95) process-sockets-30 1-groups 1.00 ( 3.73) +3.50 ( 4.92) process-sockets-30 2-groups 1.00 ( 3.90) +1.80 ( 5.68) process-sockets-30 4-groups 1.00 ( 1.03) -1.23 ( 4.79) process-sockets-30 8-groups 1.00 ( 0.48) -0.15 ( 0.33) process-sockets-45 1-groups 1.00 ( 0.62) -0.70 ( 1.12) process-sockets-45 2-groups 1.00 ( 2.56) +0.64 ( 0.86) process-sockets-45 4-groups 1.00 ( 0.57) +0.09 ( 0.53) process-sockets-45 8-groups 1.00 ( 0.18) -7.31 ( 2.11) threads-pipe-15 1-groups 1.00 ( 5.30) +4.94 ( 9.52) threads-pipe-15 2-groups 1.00 ( 7.97) -4.28 ( 2.30) threads-pipe-15 4-groups 1.00 ( 4.94) -1.83 ( 4.24) threads-pipe-15 8-groups 1.00 ( 1.69) -2.35 ( 1.50) threads-pipe-30 1-groups 1.00 ( 5.12) +2.06 ( 5.00) threads-pipe-30 2-groups 1.00 ( 1.63) +0.93 ( 4.53) threads-pipe-30 4-groups 1.00 ( 3.41) -2.85 ( 3.20) threads-pipe-30 8-groups 1.00 ( 2.85) -2.20 ( 2.68) threads-pipe-45 1-groups 1.00 ( 5.13) -0.97 ( 4.70) threads-pipe-45 2-groups 1.00 ( 1.92) -2.11 ( 1.21) threads-pipe-45 4-groups 1.00 ( 2.41) -2.69 ( 1.33) threads-pipe-45 8-groups 1.00 ( 1.81) -2.41 ( 1.14) threads-sockets-15 1-groups 1.00 ( 4.72) +0.82 ( 4.21) threads-sockets-15 2-groups 1.00 ( 3.05) -1.28 ( 2.48) threads-sockets-15 4-groups 1.00 ( 5.92) -1.75 ( 7.25) threads-sockets-15 8-groups 1.00 ( 3.75) -2.54 ( 3.49) threads-sockets-30 1-groups 1.00 ( 5.96) -0.46 ( 5.30) threads-sockets-30 2-groups 1.00 ( 1.68) -0.45 ( 1.75) threads-sockets-30 4-groups 1.00 ( 5.02) -1.48 ( 6.51) threads-sockets-30 8-groups 1.00 ( 0.41) -13.09 ( 1.61) ** threads-sockets-45 1-groups 1.00 ( 2.55) -1.68 ( 0.66) threads-sockets-45 2-groups 1.00 ( 3.53) +0.21 ( 2.23) threads-sockets-45 4-groups 1.00 ( 0.51) -1.27 ( 1.43) threads-sockets-45 8-groups 1.00 ( 3.01) -3.41 ( 0.43)
Additionally, profiling data was collected using perf record -a for this workload. Firstly, the cycles distribution are almost the same among baseline and patch1-4. Secondly, the patch1-4 relevant symbols identified were set_rd_overloaded/set_rd_overutilized, which is potentially invoked (actually inlined) by update_sd_lb_stats. The update_sd_lb_stats itself takes ~2.6% cycles in the baseline configuration threads-sockets-45fd, 8 groups, while no regressions were observed in patches 1-4 about this function. So I think the patches won't cause regressions to hackbench.
Schbench(old, 91ea787)
======================
The workload is ran by the same metholody as hackbench, with runtime 100s.
Test result is as following, the regression over -5% is marked with ** at the end of the line, while, when re-run the test with either test framework or vanilla workload, the regression cannot re-produced.
case load baseline(std%) opt1%( std%) normal 1-mthreads-8-workers 1.00 ( 1.44) -5.60 ( 2.96) ** normal 1-mthreads-2-workers 1.00 ( 2.79) -2.65 ( 5.48) normal 1-mthreads-1-workers 1.00 ( 1.27) -1.60 ( 1.03) normal 1-mthreads-31-workers 1.00 ( 1.30) -0.87 ( 2.34) normal 1-mthreads-16-workers 1.00 ( 1.74) -2.23 ( 1.15) normal 1-mthreads-4-workers 1.00 ( 3.35) -1.92 ( 1.62) normal 2-mthreads-8-workers 1.00 ( 2.17) -2.09 ( 1.38) normal 2-mthreads-31-workers 1.00 ( 1.83) +1.93 ( 1.84) normal 2-mthreads-16-workers 1.00 ( 2.06) +0.36 ( 2.38) normal 2-mthreads-1-workers 1.00 ( 3.86) +0.50 ( 2.46) normal 2-mthreads-2-workers 1.00 ( 1.76) -6.91 ( 2.55) normal 2-mthreads-4-workers 1.00 ( 1.59) -5.58 ( 5.99) normal 4-mthreads-8-workers 1.00 ( 0.85) +0.59 ( 0.54) normal 4-mthreads-31-workers 1.00 ( 15.31) +15.04 ( 12.71) normal 4-mthreads-16-workers 1.00 ( 0.99) -2.62 ( 2.15) normal 4-mthreads-4-workers 1.00 ( 1.42) -2.72 ( 1.70) normal 4-mthreads-1-workers 1.00 ( 1.43) -2.84 ( 1.73) normal 4-mthreads-2-workers 1.00 ( 1.78) -4.28 ( 2.08) normal 8-mthreads-16-workers 1.00 ( 10.04) +7.06 ( 0.73) normal 8-mthreads-31-workers 1.00 ( 1.94) -1.66 ( 2.28) normal 8-mthreads-2-workers 1.00 ( 2.51) -0.30 ( 1.53) normal 8-mthreads-8-workers 1.00 ( 1.56) -1.83 ( 1.39) normal 8-mthreads-1-workers 1.00 ( 4.08) +0.45 ( 1.45) normal 8-mthreads-4-workers 1.00 ( 1.84) +2.85 ( 1.07)
case load baseline(std%) opt2%( std%) normal 1-mthreads-8-workers 1.00 ( 1.44) -1.48 ( 3.79) normal 1-mthreads-2-workers 1.00 ( 2.79) +3.32 ( 0.90) normal 1-mthreads-1-workers 1.00 ( 1.27) +1.98 ( 1.02) normal 1-mthreads-31-workers 1.00 ( 1.30) +5.84 ( 3.01) normal 1-mthreads-16-workers 1.00 ( 1.74) +5.90 ( 0.68) normal 1-mthreads-4-workers 1.00 ( 3.35) +1.82 ( 1.65) normal 2-mthreads-8-workers 1.00 ( 2.17) +2.80 ( 2.04) normal 2-mthreads-31-workers 1.00 ( 1.83) -0.07 ( 1.09) normal 2-mthreads-16-workers 1.00 ( 2.06) +2.45 ( 2.55) normal 2-mthreads-1-workers 1.00 ( 3.86) +2.41 ( 2.92) normal 2-mthreads-2-workers 1.00 ( 1.76) -1.29 ( 2.03) normal 2-mthreads-4-workers 1.00 ( 1.59) +0.44 ( 1.15) normal 4-mthreads-8-workers 1.00 ( 0.85) -0.81 ( 3.03) normal 4-mthreads-31-workers 1.00 ( 15.31) +2.06 ( 15.97) normal 4-mthreads-16-workers 1.00 ( 0.99) -1.46 ( 2.29) normal 4-mthreads-4-workers 1.00 ( 1.42) -0.15 ( 3.37) normal 4-mthreads-1-workers 1.00 ( 1.43) +0.97 ( 1.95) normal 4-mthreads-2-workers 1.00 ( 1.78) -0.38 ( 2.53) normal 8-mthreads-16-workers 1.00 ( 10.04) +5.80 ( 1.72) normal 8-mthreads-31-workers 1.00 ( 1.94) -0.76 ( 2.33) normal 8-mthreads-2-workers 1.00 ( 2.51) +2.47 ( 2.17) normal 8-mthreads-8-workers 1.00 ( 1.56) -0.66 ( 1.47) normal 8-mthreads-1-workers 1.00 ( 4.08) +2.71 ( 2.78) normal 8-mthreads-4-workers 1.00 ( 1.84) +2.35 ( 4.88)
case load baseline(std%) opt3%( std%)
normal 1-mthreads-8-workers 1.00 ( 1.44) -6.90 ( 3.85) ** normal 1-mthreads-2-workers 1.00 ( 2.79) +3.23 ( 3.09) normal 1-mthreads-1-workers 1.00 ( 1.27) -1.04 ( 2.22) normal 1-mthreads-31-workers 1.00 ( 1.30) +2.16 ( 1.64) normal 1-mthreads-16-workers 1.00 ( 1.74) -0.72 ( 5.70) normal 1-mthreads-4-workers 1.00 ( 3.35) -1.92 ( 4.31) normal 2-mthreads-8-workers 1.00 ( 2.17) +0.82 ( 1.90) normal 2-mthreads-31-workers 1.00 ( 1.83) +2.08 ( 1.16) normal 2-mthreads-16-workers 1.00 ( 2.06) +4.04 ( 2.42) normal 2-mthreads-1-workers 1.00 ( 3.86) +2.57 ( 3.44) normal 2-mthreads-2-workers 1.00 ( 1.76) -0.12 ( 1.29) normal 2-mthreads-4-workers 1.00 ( 1.59) -2.04 ( 2.83) normal 4-mthreads-8-workers 1.00 ( 0.85) +0.22 ( 1.65) normal 4-mthreads-31-workers 1.00 ( 15.31) +15.09 ( 9.83) normal 4-mthreads-16-workers 1.00 ( 0.99) +1.46 ( 1.88) normal 4-mthreads-4-workers 1.00 ( 1.42) +2.34 ( 1.57) normal 4-mthreads-1-workers 1.00 ( 1.43) -0.77 ( 2.45) normal 4-mthreads-2-workers 1.00 ( 1.78) -1.16 ( 1.85) normal 8-mthreads-16-workers 1.00 ( 10.04) +7.39 ( 1.65) normal 8-mthreads-31-workers 1.00 ( 1.94) -0.81 ( 2.14) normal 8-mthreads-2-workers 1.00 ( 2.51) -1.93 ( 2.00) normal 8-mthreads-8-workers 1.00 ( 1.56) +1.17 ( 1.40) normal 8-mthreads-1-workers 1.00 ( 4.08) +1.63 ( 0.51) normal 8-mthreads-4-workers 1.00 ( 1.84) +4.77 ( 2.36)
case load baseline(std%) opt4%( std%)
normal 1-mthreads-8-workers 1.00 ( 1.44) -0.27 ( 3.05) normal 1-mthreads-2-workers 1.00 ( 2.79) -0.31 ( 1.19) normal 1-mthreads-1-workers 1.00 ( 1.27) +1.62 ( 1.77) normal 1-mthreads-31-workers 1.00 ( 1.30) +1.30 ( 3.34) normal 1-mthreads-16-workers 1.00 ( 1.74) +0.07 ( 3.38) normal 1-mthreads-4-workers 1.00 ( 3.35) +1.08 ( 2.48) normal 2-mthreads-8-workers 1.00 ( 2.17) +0.04 ( 3.87) normal 2-mthreads-31-workers 1.00 ( 1.83) +1.29 ( 1.44) normal 2-mthreads-16-workers 1.00 ( 2.06) +0.94 ( 2.96) normal 2-mthreads-1-workers 1.00 ( 3.86) +2.85 ( 2.12) normal 2-mthreads-2-workers 1.00 ( 1.76) -0.30 ( 2.37) normal 2-mthreads-4-workers 1.00 ( 1.59) +2.22 ( 1.51) normal 4-mthreads-8-workers 1.00 ( 0.85) +2.20 ( 3.06) normal 4-mthreads-31-workers 1.00 ( 15.31) +15.65 ( 12.68) normal 4-mthreads-16-workers 1.00 ( 0.99) -1.96 ( 3.30) normal 4-mthreads-4-workers 1.00 ( 1.42) -1.19 ( 3.42) normal 4-mthreads-1-workers 1.00 ( 1.43) +2.26 ( 2.45) normal 4-mthreads-2-workers 1.00 ( 1.78) -1.36 ( 2.75) normal 8-mthreads-16-workers 1.00 ( 10.04) -0.33 ( 11.13) normal 8-mthreads-31-workers 1.00 ( 1.94) -1.14 ( 2.01) normal 8-mthreads-2-workers 1.00 ( 2.51) +2.32 ( 2.26) normal 8-mthreads-8-workers 1.00 ( 1.56) -0.44 ( 1.54) normal 8-mthreads-1-workers 1.00 ( 4.08) +2.17 ( 2.10) normal 8-mthreads-4-workers 1.00 ( 1.84) +3.42 ( 2.34)
Again, per perf record data, the cycles distribution are almost the same among baseline and patch1-4. The symbols related to patches 1-4 are set_rd_overloaded/set_rd_overutilized that is inlined in update_sd_lb_stats, which accounts for ~0.47% (self) cycles in baseline in 1 message thread and 8 workers configuration, and no regressions were observed in patches 1-4 about this function. So I think the patches won't cause regressions to schbench(old).
Schbench(new, 48aed1d)
======================
The workload was executed using the test framework available at
https://github.com/gormanm/mmtests. Each configuration was run for 5 iterations, with a runtime of 100 seconds per iteration. No significant regressions were observed, as detailed below:
Notes:
1. message threads# are always 6, the same to numa node#
2. 1/2/4/8/16/32/64/79 are worker# per message thread
baseline patch1
Amean request-99.0th-qrtle-1 1.00 0.00%
Amean rps-50.0th-qrtle-1 1.00 0.06%
Amean wakeup-99.0th-qrtle-1 1.00 0.26%
Amean request-99.0th-qrtle-2 1.00 0.23%
Amean rps-50.0th-qrtle-2 1.00 0.00%
Amean wakeup-99.0th-qrtle-2 1.00 1.09%
Amean request-99.0th-qrtle-4 1.00 -1.32%
Amean rps-50.0th-qrtle-4 1.00 0.11%
Amean wakeup-99.0th-qrtle-4 1.00 -0.41%
Amean request-99.0th-qrtle-8 1.00 -0.08%
Amean rps-50.0th-qrtle-8 1.00 -0.17%
Amean wakeup-99.0th-qrtle-8 1.00 0.37% Amean request-99.0th-qrtle-16 1.00 0.23%
Amean rps-50.0th-qrtle-16 1.00 -0.06%
Amean wakeup-99.0th-qrtle-16 1.00 1.03%
Amean request-99.0th-qrtle-32 1.00 0.27%
Amean rps-50.0th-qrtle-32 1.00 0.06%
Amean wakeup-99.0th-qrtle-32 1.00 -0.37%
Amean request-99.0th-qrtle-64 1.00 0.57%
Amean rps-50.0th-qrtle-64 1.00 -0.28%
Amean wakeup-99.0th-qrtle-64 1.00 -3.00%
Amean request-99.0th-qrtle-79 1.00 0.21%
Amean rps-50.0th-qrtle-79 1.00 -0.23%
Amean wakeup-99.0th-qrtle-79 1.00 2.00%
baseline patch2
Amean request-99.0th-qrtle-1 1.00 -0.46%
Amean rps-50.0th-qrtle-1 1.00 0.11%
Amean wakeup-99.0th-qrtle-1 1.00 -2.01%
Amean request-99.0th-qrtle-2 1.00 -0.08%
Amean rps-50.0th-qrtle-2 1.00 0.00%
Amean wakeup-99.0th-qrtle-2 1.00 -1.42%
Amean request-99.0th-qrtle-4 1.00 -1.16%
Amean rps-50.0th-qrtle-4 1.00 0.11%
Amean wakeup-99.0th-qrtle-4 1.00 -1.30%
Amean request-99.0th-qrtle-8 1.00 -0.08%
Amean rps-50.0th-qrtle-8 1.00 -0.40%
Amean wakeup-99.0th-qrtle-8 1.00 1.25% Amean request-99.0th-qrtle-16 1.00 0.46%
Amean rps-50.0th-qrtle-16 1.00 -0.06%
Amean wakeup-99.0th-qrtle-16 1.00 2.52%
Amean request-99.0th-qrtle-32 1.00 14.83%
Amean rps-50.0th-qrtle-32 1.00 0.75%
Amean wakeup-99.0th-qrtle-32 1.00 3.03%
Amean request-99.0th-qrtle-64 1.00 -0.44%
Amean rps-50.0th-qrtle-64 1.00 0.28%
Amean wakeup-99.0th-qrtle-64 1.00 -3.50%
Amean request-99.0th-qrtle-79 1.00 -0.09%
Amean rps-50.0th-qrtle-79 1.00 0.08%
Amean wakeup-99.0th-qrtle-79 1.00 -1.20%
baseline patch3
Amean request-99.0th-qrtle-1 1.00 0.31%
Amean rps-50.0th-qrtle-1 1.00 -0.17%
Amean wakeup-99.0th-qrtle-1 1.00 0.44%
Amean request-99.0th-qrtle-2 1.00 -0.61%
Amean rps-50.0th-qrtle-2 1.00 -0.29%
Amean wakeup-99.0th-qrtle-2 1.00 1.93%
Amean request-99.0th-qrtle-4 1.00 -1.62%
Amean rps-50.0th-qrtle-4 1.00 -0.17%
Amean wakeup-99.0th-qrtle-4 1.00 0.00%
Amean request-99.0th-qrtle-8 1.00 0.00%
Amean rps-50.0th-qrtle-8 1.00 -0.40%
Amean wakeup-99.0th-qrtle-8 1.00 -0.29%
Amean request-99.0th-qrtle-16 1.00 0.53%
Amean rps-50.0th-qrtle-16 1.00 -0.17%
Amean wakeup-99.0th-qrtle-16 1.00 -1.03%
Amean request-99.0th-qrtle-32 1.00 0.09%
Amean rps-50.0th-qrtle-32 1.00 -0.17%
Amean wakeup-99.0th-qrtle-32 1.00 2.41%
Amean request-99.0th-qrtle-64 1.00 0.26%
Amean rps-50.0th-qrtle-64 1.00 -0.16%
Amean wakeup-99.0th-qrtle-64 1.00 -2.00%
Amean request-99.0th-qrtle-79 1.00 0.26%
Amean rps-50.0th-qrtle-79 1.00 -0.46%
Amean wakeup-99.0th-qrtle-79 1.00 1.20%
baseline patch4
Amean request-99.0th-qrtle-1 1.00 -0.15%
Amean rps-50.0th-qrtle-1 1.00 -0.06%
Amean wakeup-99.0th-qrtle-1 1.00 -2.88%
Amean request-99.0th-qrtle-2 1.00 -0.31%
Amean rps-50.0th-qrtle-2 1.00 -0.29%
Amean wakeup-99.0th-qrtle-2 1.00 -0.59%
Amean request-99.0th-qrtle-4 1.00 -0.23%
Amean rps-50.0th-qrtle-4 1.00 -0.11%
Amean wakeup-99.0th-qrtle-4 1.00 -0.41%
Amean request-99.0th-qrtle-8 1.00 -0.08%
Amean rps-50.0th-qrtle-8 1.00 -0.52%
Amean wakeup-99.0th-qrtle-8 1.00 1.91% Amean request-99.0th-qrtle-16 1.00 0.76%
Amean rps-50.0th-qrtle-16 1.00 0.06%
Amean wakeup-99.0th-qrtle-16 1.00 1.03%
Amean request-99.0th-qrtle-32 1.00 8.36%
Amean rps-50.0th-qrtle-32 1.00 0.00%
Amean wakeup-99.0th-qrtle-32 1.00 -1.05%
Amean request-99.0th-qrtle-64 1.00 0.13%
Amean rps-50.0th-qrtle-64 1.00 0.00%
Amean wakeup-99.0th-qrtle-64 1.00 -4.00%
Amean request-99.0th-qrtle-79 1.00 -0.39%
Amean rps-50.0th-qrtle-79 1.00 0.14%
Amean wakeup-99.0th-qrtle-79 1.00 -0.40%
Re: [PATCH 1/4] sched/rt: Optimize cpupri_vec layout to mitigate cache line contention
In reply to Pan DengQuoted 4 lines · Pan Deng
When running a multi-instance FFmpeg workload on an HCC system, significant cache line contention is observed around
cpupri_vec->countandmaskin structroot_domain.
[it seems that my last reply did not make it to the lkml][snip]
kernel/sched/cpupri.h line 12
Quoted 10 lines
kernel/sched/cpupri.h +1
-1@@ -9,7 +9,7 @@struct cpupri_vec {atomic_t count; -cpumask_var_t mask; +cpumask_var_t mask____cacheline_aligned ;
Just curious, since this is to avoid cache contention among CPUs, is it better to use ____cacheline_aligned_in_smp, so the single CPU system is not impacted.
thanks,
Chenyu> };
Quoted 2 lines
kernel/sched/cpupri.h +0
-0struct cpupri {
RE: [PATCH 1/4] sched/rt: Optimize cpupri_vec layout to mitigate cache line contention
Thanks Yu, will update the patch.
Best Regards
Pan
Quoted 38 lines
-----Original Message-----
From: Chen, Yu C <[email protected]>
Sent: Monday, September 1, 2025 1:10 PM
To: Deng, Pan <[email protected]>
Cc: [email protected]; Li, Tianyou <[email protected]>; [email protected]; [email protected]; [email protected]; Chen Yu <[email protected]>
Subject: Re: [PATCH 1/4] sched/rt: Optimizecpupri_veclayout to mitigate cache line contentionQuoted 1 line · Pan Deng
When running a multi-instance FFmpeg workload on an HCC system,
significant
Quoted 3 lines
cache line contention is observed around
cpupri_vec->countandmaskin structroot_domain.[it seems that my last reply did not make it to the lkml][snip]
kernel/sched/cpupri.h line 12
Quoted 10 lines
kernel/sched/cpupri.h +1
-1@@ -9,7 +9,7 @@struct cpupri_vec {atomic_t count; -cpumask_var_t mask; +cpumask_var_t mask____cacheline_aligned ;Just curious, since this is to avoid cache contention among CPUs, is it better to use
____cacheline_aligned_in_smp, so the single CPU system is not impacted.thanks,
Chenyu> };Quoted 2 lines
kernel/sched/cpupri.h +0
-0struct cpupri {