Cache aware load-balancing
Subsystems (5)
v4 cover Cache aware load-balancing
From: Tim Chen <[email protected]>
This is the forth revision of the cache aware scheduling patches, based on the original patch proposed by Peter[1].
The main change in v4 is to address the performance regressions reported in v3, which are caused by over-aggregation of tasks in a single LLC, but they don't actually share data. Such aggregation could cause regression on platforms with smaller LLC size. It can also occur when running workloads with a large memory footprint (e.g., stream) or when workloads involve too many threads (e.g., hackbench).
Patches 1 to 20 are almost identical to those in v3; the key fixes are included in patches 21 to 28. The approach involves tracking a process's resident pages and comparing this to the LLC cache size.
To that effect, /sys/kernel/debug/sched/sched_cache_ignore_rss is added where
0 turns off cache aware scheduling entirely
1 turns off cache aware scheduling when resident memory of process
exceed LLC size (default)
100 RSS will not be taken into account during cache aware scheduling N translates to turn off cache aware scheduling when RSS is greater than (N-1) * 256 * LLC size
So for folks who want to make cache aware scheduling to be aggressive and they know their process threads share lots of data, they could set it to 100.
Similarly, the number of active threads within each process is monitored and compared to the number of cores (excluding SMT) in each LLC.
The second update is to limit the scan of a process's occupancy for new preferred LLC candidate to preferred NUMA node from NUMA balancing, node of current preferred LLC and node of current CPU. It is very likely that these nodes overlap and reduces the scanning cost. We also check to see if there are no NUMA node with more than one LLC, cache aware scheduling is turned off and we will strictly let NUMA balancing consolidate the tasks of a process.
v4 of cache aware scheduling relies only on load balancing to do task aggregation as doing that during wake up is expensive. By default, LLC task aggregation during wake-up is disabled. Conversely, cache-aware load balancing is enabled by default. For easier comparison, two scheduler features are introduced:SCHED_CACHE_WAKE and SCHED_CACHE_LB,
which control cache-aware wake up and cache-aware load balancing, respectively. We plan to obsolete SCHED_CACHE_WAKE in future.
The load balancing and migration policy are now implemented in a single location within the function _get_migrate_hint(). Debugfs knobs are also introduced to fine-tune the cache aware load balancing. Please refer to patch 7 for detail.
Test results:
Tested on a 2 socket Intel Sapphire Rapids with 30 cores per socket. The DRAM interleaving is enabled in the BIOS so it essential has one NUMA node with two last level caches. AMD Milan is also tested. There are 4 Nodes and 32 CPUs per node. Each node has 4 CCX(shared LLC) and each CCX has 8 CPUs.
[TL;DR]
Sappire Rapids:
hackbench shows significant improvement when there is 1 group with different number of fd pairs(threads) within this process. schbench shows overall wakeup latency improvement. Other micro-workloads did not show much difference.
Milan:
Regression in v3 has been mitigated a lot. For hackbench and schbench, we still saw some regressions, and it can be further tuned via debugfs to inhibit the cache aware scheduling when it is overloaded.
[Sapphire Rapids details]
[hackbench]
Hackbench show overall improvement when there is only 1 group, with different number of fd(pairs). This is the expected behavior because this test scenario would benefit from cache aware load balance most. Other number of groups shows not much difference(using default fd = 20).
baseline sched_cache
groups hackbench hackbench
Min 1 35.2340 ( 0.00%) 27.2560 ( 22.64%) Min 3 36.0700 ( 0.00%) 38.1750 ( -5.84%) Min 5 36.4790 ( 0.00%) 39.6540 ( -8.70%) Min 7 51.2000 ( 0.00%) 52.9050 ( -3.33%) Min 12 63.1010 ( 0.00%) 64.2940 ( -1.89%) Min 16 75.0700 ( 0.00%) 77.1050 ( -2.71%) Amean 1 37.3802 ( 0.00%) 28.0908 * 24.85%* Amean 3 40.3703 ( 0.00%) 40.1250 ( 0.61%) Amean 5 42.1782 ( 0.00%) 41.4368 ( 1.76%) Amean 7 51.4848 ( 0.00%) 53.3075 * -3.54%* Amean 12 63.6612 ( 0.00%) 65.4685 ( -2.84%) Amean 16 75.3333 ( 0.00%) 78.5802 * -4.31%*
[stream]
No much difference is observed.
512MB:
baseline_512 sched_cache_512
baseline_512M sched_cache_512M
GB/sec copy-2 34.34 ( 0.00%) 35.14 ( 2.32%)
GB/sec scale-2 23.80 ( 0.00%) 23.91 ( 0.44%)
GB/sec add-2 28.89 ( 0.00%) 28.92 ( 0.12%)
GB/sec triad-2 28.62 ( 0.00%) 28.67 ( 0.17%)2GB:
baseline_2 sched_cache_2
baseline_2G sched_cache_2G
GB/sec copy-2 35.68 ( 0.00%) 36.03 ( 0.97%) GB/sec scale-2 23.85 ( 0.00%) 23.83 ( -0.07%) GB/sec add-2 28.47 ( 0.00%) 28.42 ( -0.18%) GB/sec triad-2 28.38 ( 0.00%) 28.34 ( -0.15%)
[netperf]
No much difference is observed.
baseline sched_cache
nr_pairs netperf netperf
Hmean 1 1343.86 ( 0.00%) 1344.58 ( 0.05%) BHmean-99 1 1346.54 ( 0.00%) 1345.66 ( -0.07%) Hmean 2 1322.98 ( 0.00%) 1324.21 ( 0.09%) BHmean-99 2 1324.14 ( 0.00%) 1325.78 ( 0.12%) Hmean 4 1327.85 ( 0.00%) 1330.82 ( 0.22%) BHmean-99 4 1328.92 ( 0.00%) 1331.68 ( 0.21%) Hmean 8 1313.72 ( 0.00%) 1314.35 ( 0.05%) BHmean-99 8 1316.47 ( 0.00%) 1315.65 ( -0.06%) Hmean 16 1288.49 ( 0.00%) 1294.18 ( 0.44%) BHmean-99 16 1290.63 ( 0.00%) 1295.03 ( 0.34%) Hmean 32 1254.83 ( 0.00%) 1260.84 ( 0.48%) BHmean-99 32 1258.50 ( 0.00%) 1264.73 ( 0.49%) Hmean 64 1128.96 ( 0.00%) 1127.00 ( -0.17%) BHmean-99 64 1131.06 ( 0.00%) 1127.83 ( -0.29%) Hmean 128 799.18 ( 0.00%) 796.50 ( -0.33%) BHmean-99 128 802.29 ( 0.00%) 799.37 ( -0.36%) Hmean 256 400.59 ( 0.00%) 399.65 ( -0.24%) BHmean-99 256 402.76 ( 0.00%) 401.76 ( -0.25%)
[schbench]
Overall 99.0th wakeup latency improvement is observed.
Metric Base (mean std) Compare (mean std) Change
-------------------------------------------------------------------------------------
schbench thread = 1
Wakeup Latencies 99.0th 15.80(2.39) 14.60(1.52) +7.59% Request Latencies 99.0th 3924.00(24.66) 3935.20(23.73) -0.29% RPS 50.0th 267.40(1.34) 266.40(1.34) -0.37% Average RPS 268.53(1.68) 267.57(1.46) -0.36%
schbench thread = 2
Wakeup Latencies 99.0th 17.20(0.45) 15.80(0.45) +8.14% Request Latencies 99.0th 3900.00(11.31) 3914.40(31.19) -0.37% RPS 50.0th 537.00(0.00) 536.20(1.10) -0.15% Average RPS 537.80(0.25) 535.08(1.93) -0.51%
schbench thread = 4
Wakeup Latencies 99.0th 12.80(0.84) 12.40(0.55) +3.13% Request Latencies 99.0th 3864.80(36.04) 3839.20(36.49) +0.66% RPS 50.0th 1076.40(2.19) 1077.20(1.79) +0.07% Average RPS 1074.00(2.43) 1073.44(2.24) -0.05%
schbench thread = 8
Wakeup Latencies 99.0th 13.60(1.67) 9.80(0.84) +27.94% Request Latencies 99.0th 3746.40(22.20) 3794.40(32.69) -1.28% RPS 50.0th 2159.20(4.38) 2159.20(4.38) 0.00% Average RPS 2155.74(2.77) 2154.16(1.86) -0.07%
schbench thread = 16
Wakeup Latencies 99.0th 12.60(1.52) 9.00(0.71) +28.57% Request Latencies 99.0th 3775.20(26.29) 3783.20(15.59) -0.21% RPS 50.0th 4318.40(8.76) 4312.00(0.00) -0.15% Average RPS 4311.62(5.94) 4308.49(3.69) -0.07%
schbench thread = 32
Wakeup Latencies 99.0th 12.20(0.45) 8.80(0.84) +27.87% Request Latencies 99.0th 3781.60(27.36) 3812.00(23.32) -0.80% RPS 50.0th 8630.40(14.31) 8630.40(14.31) 0.00% Average RPS 8621.32(3.34) 8614.98(8.24) -0.07%
schbench thread = 64
Wakeup Latencies 99.0th 12.60(0.55) 10.20(1.10) +19.05% Request Latencies 99.0th 3778.40(19.10) 5498.40(2130.50) -45.52% RPS 50.0th 17248.00(0.00) 16774.40(560.98) -2.75% Average RPS 17249.44(6.83) 16727.70(544.81) -3.02%
schbench thread = 128
Wakeup Latencies 99.0th 14.00(0.00) 14.40(0.55) -2.86% Request Latencies 99.0th 7860.80(7.16) 8123.20(20.86) -3.34% RPS 50.0th 32147.20(145.94) 29587.20(2081.43) -7.96% Average RPS 31546.92(444.17) 28827.07(1842.18) -8.62%
schbench thread = 239
Wakeup Latencies 99.0th 41.60(2.07) 39.20(2.05) +5.77% Request Latencies 99.0th 8056.00(0.00) 8361.60(26.77) -3.79% RPS 50.0th 31020.80(28.62) 30137.60(57.24) -2.85% Average RPS 31052.18(32.53) 30158.53(47.73) -2.88%
[Milan details]
[hackbench]
Regressions are observed when many hackbench instances are running simultaneously. Considering that the per-process nr_running statistic is tracked in v4, reintroducing the per-LLC nr_running statistic (as in v1/v2) might mitigate the regression (and could be implemented in v5).
baseline sched_cache
hackbench hackbench
Min 1 50.8910 ( 0.00%) 51.3470 ( -0.90%) Min 3 51.1380 ( 0.00%) 52.1710 ( -2.02%) Min 5 75.5740 ( 0.00%) 79.9060 ( -5.73%) Min 7 106.0020 ( 0.00%) 111.6930 ( -5.37%) Min 12 177.3530 ( 0.00%) 195.9230 ( -10.47%) Min 16 238.8580 ( 0.00%) 268.5540 ( -12.43%) Amean 1 52.0164 ( 0.00%) 52.0198 ( -0.01%) Amean 3 51.6346 ( 0.00%) 52.9120 ( -2.47%) Amean 5 76.9840 ( 0.00%) 81.1256 ( -5.38%) Amean 7 107.1258 ( 0.00%) 112.5528 * -5.07%* Amean 12 178.5502 ( 0.00%) 197.8780 * -10.82%* Amean 16 240.5200 ( 0.00%) 270.3840 * -12.42%*
[stream]
No huge regression is observed, which was seen in v3.
512MB:
baseline_512 sched_cache_512
baseline_512M sched_cache_512M
GB/sec copy-16 442.53 ( 0.00%) 421.83 ( -4.68%)
GB/sec scale-16 387.74 ( 0.00%) 377.00 ( -2.77%)
GB/sec add-16 413.29 ( 0.00%) 410.84 ( -0.59%)
GB/sec triad-16 411.34 ( 0.00%) 407.73 ( -0.88%)2GB:
baseline_2 sched_cache_2
baseline_2G sched_cache_2G
GB/sec copy-16 309.30 ( 0.00%) 315.93 ( 2.15%) GB/sec scale-16 208.16 ( 0.00%) 209.99 ( 0.88%) GB/sec add-16 229.94 ( 0.00%) 231.51 ( 0.68%) GB/sec triad-16 240.40 ( 0.00%) 242.02 ( 0.67%)
[netperf]
No much difference is observed.
baseline sched_cache
nr_pairs netperf netperf
Hmean 1 919.00 ( 0.00%) 923.82 ( 0.52%) BHmean-99 1 920.38 ( 0.00%) 925.34 ( 0.54%) Hmean 2 917.23 ( 0.00%) 916.01 ( -0.13%) BHmean-99 2 918.99 ( 0.00%) 918.08 ( -0.10%) Hmean 4 908.65 ( 0.00%) 908.47 ( -0.02%) BHmean-99 4 910.04 ( 0.00%) 909.56 ( -0.05%) Hmean 8 911.31 ( 0.00%) 909.19 ( -0.23%) BHmean-99 8 911.97 ( 0.00%) 910.23 ( -0.19%) Hmean 16 905.37 ( 0.00%) 901.83 ( -0.39%) BHmean-99 16 905.92 ( 0.00%) 902.43 ( -0.39%) Hmean 32 865.56 ( 0.00%) 860.52 ( -0.58%) BHmean-99 32 866.40 ( 0.00%) 863.23 ( -0.37%) Hmean 64 734.50 ( 0.00%) 729.11 ( -0.73%) BHmean-99 64 737.50 ( 0.00%) 732.97 ( -0.61%) Hmean 128 735.01 ( 0.00%) 728.00 ( -0.95%) BHmean-99 128 738.62 ( 0.00%) 731.48 ( -0.97%) Hmean 256 335.52 ( 0.00%) 335.57 ( 0.01%) BHmean-99 256 338.19 ( 0.00%) 337.99 ( -0.06%)
[schbench]
No huge regression is observed(consider the std), which was seen in v3.
Metric Base (mean std) Compare (mean std) Change
-------------------------------------------------------------------------------------
schbench thread = 1
Wakeup Latencies 99.0th 14.80(1.10) 15.00(2.74) -1.35% Request Latencies 99.0th 5310.40(18.24) 5342.40(56.11) -0.60% RPS 50.0th 384.20(3.27) 386.00(5.43) +0.47% Average RPS 385.94(1.98) 387.37(4.74) +0.37%
schbench thread = 2
Wakeup Latencies 99.0th 15.20(0.45) 16.80(1.30) -10.53% Request Latencies 99.0th 5291.20(7.16) 5336.00(27.71) -0.85% RPS 50.0th 778.60(2.61) 772.20(7.01) -0.82% Average RPS 778.54(2.42) 773.14(6.19) -0.69%
schbench thread = 4
Wakeup Latencies 99.0th 16.60(1.52) 17.60(0.55) -6.02% Request Latencies 99.0th 5233.60(8.76) 7432.00(2946.91) -42.01% RPS 50.0th 1578.80(1.79) 1570.00(9.38) -0.56% Average RPS 1576.94(1.88) 1544.12(44.90) -2.08%
schbench thread = 8
Wakeup Latencies 99.0th 18.40(0.89) 19.40(0.55) -5.43% Request Latencies 99.0th 5236.80(7.16) 5236.80(7.16) 0.00% RPS 50.0th 3152.80(4.38) 3148.00(0.00) -0.15% Average RPS 3150.89(2.71) 3145.30(1.21) -0.18%
schbench thread = 16
Wakeup Latencies 99.0th 16.80(1.64) 18.40(0.55) -9.52% Request Latencies 99.0th 5224.00(0.00) 5195.20(47.19) +0.55% RPS 50.0th 6312.00(0.00) 6296.00(0.00) -0.25% Average RPS 6308.84(3.79) 6293.32(2.61) -0.25%
schbench thread = 32
Wakeup Latencies 99.0th 18.80(2.17) 21.00(0.71) -11.70% Request Latencies 99.0th 9716.80(2458.04) 10806.40(35.05) -11.21% RPS 50.0th 12457.60(79.68) 12483.20(62.38) +0.21% Average RPS 12300.14(125.37) 12269.54(66.16) -0.25%
schbench thread = 63
Wakeup Latencies 99.0th 27.20(0.45) 29.00(0.71) -6.62% Request Latencies 99.0th 11337.60(14.31) 11401.60(14.31) -0.56% RPS 50.0th 11632.00(0.00) 11600.00(0.00) -0.28% Average RPS 11628.59(1.85) 11597.09(2.17) -0.27%
-------------------------------------------------------------------------------------
This patch set is applied on vanilla v6.16 kernel.
Thanks to Yangyu Chen, K Prateek Nayak, Shrikanth Hegde, Libo Chen, Jianyong Wu, and Madadi Vineeth Reddy for their testing and valuable suggestions on v3. Since the code has been modified, Yangyu's Tested-by tag has not been included in v4.
Comments and tests are much appreciated.
Patch 1: Peter's original patch. Patch 2-5: Various fixes and tuning of the original v1 patch. Patch 6-12: Infrastructure and helper functions for load balancing to be cache aware. Patch 13-18: Add logic to load balancing for preferred LLC aggregation. Patch 19: Add process LLC aggregation in load balancing sched feature. Patch 20: Add Process LLC aggregation in wake up sched feature (turn off by default). Patch 21-28: Fix the performance regressions reported in v3, by checking
the process's working set and number of active threads, finding the busiest CPU in the preferred NUMA node.
v1:
https://lore.kernel.org/lkml/[email protected]/
v2:
https://lore.kernel.org/lkml/[email protected]/
v3:
https://lore.kernel.org/all/[email protected]/
Chen Yu (11):
sched: Several fixes for cache aware scheduling
sched: Avoid task migration within its preferred LLC
sched: Save the per LLC utilization for better cache aware scheduling
sched: Introduce a static key to enable cache aware only for multi
LLCs
sched: Turn EPOCH_PERIOD and EPOCH_OLD into tunnable debugfs
sched: Scan a task's preferred node for preferred LLC
sched: Record average number of runninhg tasks per process
sched: Skip cache aware scheduling if the process has many active
threads
sched: Do not enable cache aware scheduling for process with large RSS
sched: Allow the user space to tune the scale factor for RSS
comparison
sched: Add ftrace to track cache aware load balance and hottest CPU
changes
K Prateek Nayak (1):
sched: Avoid calculating the cpumask if the system is overloaded
Peter Zijlstra (1):
sched: Cache aware load-balancing
Tim Chen (15):
sched: Add hysteresis to switch a task's preferred LLC
sched: Add helper function to decide whether to allow cache aware
scheduling
sched: Set up LLC indexing
sched: Introduce task preferred LLC field
sched: Calculate the number of tasks that have LLC preference on a
runqueue
sched: Introduce per runqueue task LLC preference counter
sched: Calculate the total number of preferred LLC tasks during load
balance
sched: Tag the sched group as llc_balance if it has tasks prefer other
LLC
sched: Introduce update_llc_busiest() to deal with groups having
preferred LLC tasks
sched: Introduce a new migration_type to track the preferred LLC load
balance
sched: Consider LLC locality for active balance
sched: Consider LLC preference when picking tasks from busiest queue
sched: Do not migrate task if it is moving out of its preferred LLC
sched: Introduce SCHED_CACHE_LB to control cache aware load balance
sched: Introduce SCHED_CACHE_WAKE to control LLC aggregation on wake
up
include/linux/mm_types.h | 45 ++ include/linux/sched.h | 8 + include/linux/sched/topology.h | 3 + include/trace/events/sched.h | 93 +++ init/Kconfig | 8 + init/init_task.c | 3 + kernel/fork.c | 5 + kernel/sched/core.c | 25 +- kernel/sched/debug.c | 86 +++ kernel/sched/fair.c | 1022 +++++++++++++++++++++++++++++++- kernel/sched/features.h | 3 + kernel/sched/sched.h | 27 + kernel/sched/topology.c | 51 +- 13 files changed, 1348 insertions(+), 31 deletions(-)
Signature · 1 line
2.25.1
RFC PATCH v4 1/28 sched: Cache aware load-balancing
From: Peter Zijlstra <[email protected]>
Hi all,
One of the many things on the eternal todo list has been finishing the below hackery.
It is an attempt at modelling cache affinity -- and while the patch really only targets LLC, it could very well be extended to also apply to clusters (L2). Specifically any case of multiple cache domains inside a node.
Anyway, I wrote this about a year ago, and I mentioned this at the recent OSPM conf where Gautham and Prateek expressed interest in playing with this code.
So here goes, very rough and largely unproven code ahead :-)
It applies to current tip/master, but I know it will fail the __percpu validation that sits in -next, although that shouldn't be terribly hard to fix up.
As is, it only computes a CPU inside the LLC that has the highest recent runtime, this CPU is then used in the wake-up path to steer towards this LLC and in task_hot() to limit migrations away from it.
More elaborate things could be done, notably there is an XXX in there somewhere about finding the best LLC inside a NODE (interaction with NUMA_BALANCING).
--- include/linux/mm_types.h | 44 ++++++ include/linux/sched.h | 4 + init/Kconfig | 4 + kernel/fork.c | 5 + kernel/sched/core.c | 13 +- kernel/sched/fair.c | 330 +++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 8 + 7 files changed, 388 insertions(+), 20 deletions(-)
include/linux/mm_types.h +44 -0
@@ -928,6 +928,12 @@ struct mm_cid { };#endif +struct mm_sched { +u64 runtime; +unsigned long epoch; +unsigned long occ; +}; +struct kioctx_table ;struct iommu_mm_data ;struct mm_struct {
@@ -1018,6 +1024,17 @@ struct mm_struct {*/ raw_spinlock_t cpus_allowed_lock;#endif +#ifdef CONFIG_SCHED_CACHE +/* +* Track per-cpu-per-process occupancy as a proxy for cache residency. +* See account_mm_sched() and ... +*/ +struct mm_sched __percpu *pcpu_sched; +raw_spinlock_t mm_sched_lock; +unsigned long mm_sched_epoch; +int mm_sched_cpu; +#endif +#ifdef CONFIG_MMU atomic_long_t pgtables_bytes;/* size of all page tables */ #endif
@@ -1432,6 +1449,33 @@ static inline unsigned int mm_cid_size(void)static inline void mm_set_cpus_allowed (struct mm_struct *mm,const struct cpumask *cpumask) { }#endif /* CONFIG_SCHED_MM_CID */ +#ifdef CONFIG_SCHED_CACHE +extern void mm_init_sched (struct mm_struct *mm,struct mm_sched *pcpu_sched); + +static inline int mm_alloc_sched_noprof (struct mm_struct *mm) +{ + struct mm_sched *pcpu_sched = alloc_percpu_noprof(struct mm_sched); +if (!pcpu_sched) +return -ENOMEM ; + +mm_init_sched (mm, pcpu_sched); +return 0 ; +} + +#define mm_alloc_sched (...) alloc_hooks(mm_alloc_sched_noprof(__VA_ARGS__)) + +static inline void mm_destroy_sched (struct mm_struct *mm) +{ +free_percpu (mm->pcpu_sched); + mm->pcpu_sched =NULL ; +} +#else /* !CONFIG_SCHED_CACHE */ + +static inline int mm_alloc_sched (struct mm_struct *mm) {return 0 ; } +static inline void mm_destroy_sched (struct mm_struct *mm) { } + +#endif /* CONFIG_SCHED_CACHE */ +struct mmu_gather ;extern void tlb_gather_mmu (struct mmu_gather *tlb,struct mm_struct *mm);extern void tlb_gather_mmu_fullmm (struct mmu_gather *tlb,struct mm_struct *mm);
include/linux/sched.h +4 -0
@@ -1403,6 +1403,10 @@ struct task_struct {unsigned long numa_pages_migrated;#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SCHED_CACHE +struct callback_head cache_work; +#endif +#ifdef CONFIG_RSEQ struct rseq __user *rseq;u32 rseq_len;
init/Kconfig +4 -0
@@ -947,6 +947,10 @@ config NUMA_BALANCING This system will be inactive on UMA systems. +config SCHED_CACHE + bool "Cache aware scheduler" + default y + config NUMA_BALANCING_DEFAULT_ENABLED bool "Automatically enable NUMA aware memory/task placement" default y
kernel/fork.c +5 -0
@@ -1073,6 +1073,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,if (mm_alloc_cid (mm, p)) goto fail_cid; +if (mm_alloc_sched (mm)) + goto fail_sched; +if (percpu_counter_init_many (mm->rss_stat,0 ,GFP_KERNEL_ACCOUNT ,NR_MM_COUNTERS )) goto fail_pcpu;
@@ -1082,6 +1085,8 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,return mm; fail_pcpu: +mm_destroy_sched (mm); +fail_sched:mm_destroy_cid (mm); fail_cid:destroy_context (mm);
kernel/sched/core.c +10 -3
@@ -4539,6 +4539,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->migration_pending =NULL ;#endif init_sched_mm_cid (p); +init_sched_mm (p); }DEFINE_STATIC_KEY_FALSE (sched_numa_balancing);
@@ -8508,6 +8509,7 @@ static struct kmem_cache *task_group_cache __ro_after_init;void __init sched_init (void ) { +unsigned long now = jiffies;unsigned long ptr =0 ;int i;
@@ -8582,7 +8584,7 @@ void __init sched_init(void)raw_spin_lock_init (&rq->__lock); rq->nr_running =0 ; rq->calc_load_active =0 ; - rq->calc_load_update = jiffies +LOAD_FREQ ; + rq->calc_load_update = now +LOAD_FREQ ;init_cfs_rq (&rq->cfs);init_rt_rq (&rq->rt);init_dl_rq (&rq->dl);
@@ -8626,7 +8628,7 @@ void __init sched_init(void) rq->cpu_capacity =SCHED_CAPACITY_SCALE ; rq->balance_callback = &balance_push_callback; rq->active_balance =0 ; - rq->next_balance = jiffies; + rq->next_balance = now; rq->push_cpu =0 ; rq->cpu = i; rq->online =0 ;
@@ -8638,7 +8640,7 @@ void __init sched_init(void)rq_attach_root (rq, &def_root_domain);#ifdef CONFIG_NO_HZ_COMMON - rq->last_blocked_load_update_tick = jiffies; + rq->last_blocked_load_update_tick = now;atomic_set (&rq->nohz_flags,0 );INIT_CSD (&rq->nohz_csd, nohz_csd_func, rq);
@@ -8663,6 +8665,11 @@ void __init sched_init(void) rq->core_cookie =0UL ;#endif +#ifdef CONFIG_SCHED_CACHE +raw_spin_lock_init (&rq->cpu_epoch_lock); + rq->cpu_epoch_next = now; +#endif +zalloc_cpumask_var_node (&rq->scratch_mask,GFP_KERNEL ,cpu_to_node (i)); }
kernel/sched/fair.c +313 -17
450 lines not on this page · raw message
kernel/sched/sched.h +8 -0
@@ -1173,6 +1173,12 @@ struct rq {u64 clock_pelt_idle_copy;u64 clock_idle_copy;#endif +#ifdef CONFIG_SCHED_CACHE +raw_spinlock_t cpu_epoch_lock; +u64 cpu_runtime; +unsigned long cpu_epoch; +unsigned long cpu_epoch_next; +#endif atomic_t nr_iowait;
@@ -3885,6 +3891,8 @@ static inline void task_tick_mm_cid(struct rq *rq, struct task_struct *curr) { }static inline void init_sched_mm_cid (struct task_struct *t) { }#endif /* !CONFIG_SCHED_MM_CID */ +extern void init_sched_mm (struct task_struct *p); +extern u64 avg_vruntime (struct cfs_rq *cfs_rq);extern int entity_eligible (struct cfs_rq *cfs_rq,struct sched_entity *se);#ifdef CONFIG_SMP
Signature · 1 line
2.25.1
kernel test robot
build botRe: [RFC PATCH v4 01/28] sched: Cache aware load-balancing
kernel test robot noticed "Oops:general_protection_fault,probably_for_non-canonical_address#:#[##]SMP_KASAN" on:
Hello,
kernel test robot noticed "Oops:general_protection_fault,probably_for_non-canonical_address#:#[##]SMP_KASAN" on:
commit: 2ae6ef14daceb295598846a328dd65fde6b16c99 ("[RFC PATCH v4 01/28] sched: Cache aware load-balancing")
url: https://github.com/intel-lab-lkp/linux/commits/Chen-Yu/sched-Cache-aware-load-balancing/20250809-131716
base: v6.16
patch link: https://lore.kernel.org/all/9157186cf9e3fd541f62c637579ff736b3704c51.1754712565.git.tim.c.chen@linux.intel.com/
patch subject: [RFC PATCH v4 01/28] sched: Cache aware load-balancing
in testcase: boot
config: x86_64-randconfig-103-20250810
compiler: gcc-12
test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
(please refer to attached dmesg/kmsg for entire log/backtrace)
+----------------------------------------------------------------------------------+-------+------------+ | | v6.16 | 2ae6ef14da | +----------------------------------------------------------------------------------+-------+------------+ | Oops:general_protection_fault,probably_for_non-canonical_address#:#[##]SMP_KASAN | 0 | 5 | | KASAN:null-ptr-deref_in_range[#-#] | 0 | 5 | | RIP:_find_next_bit | 0 | 5 | | Kernel_panic-not_syncing:Fatal_exception | 0 | 5 | +----------------------------------------------------------------------------------+-------+------------+
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]
[ 18.108061][ T1] usbcore: registered new interface driver i2c-diolan-u2c [ 18.111512][ T36] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 [ 18.113593][ T1] Driver for 1-wire Dallas network protocol. [ 18.119966][ T1] f71882fg: Not a Fintek device [ 18.120517][ T1] f71882fg: Not a Fintek device [ 18.142462][ T63] Oops: general protection fault, probably for non-canonical address 0xdffffc0000000012: 0000 [#1] SMP KASAN [ 18.144289][ T63] KASAN: null-ptr-deref in range [0x0000000000000090-0x0000000000000097] [ 18.145594][ T63] CPU: 0 UID: 0 PID: 63 Comm: modprobe Not tainted 6.16.0-00001-g2ae6ef14dace #1 PREEMPT(none) [ 18.146207][ T63] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 18.146207][ T63] RIP: 0010:_find_next_bit (kbuild/src/consumer/lib/find_bit.c:156 (discriminator 2)) [ 18.146207][ T63] Code: 48 89 d3 48 83 c8 ff 83 e2 3f 49 89 fe 48 c1 eb 06 c4 62 e9 f7 e8 b8 ff ff 37 00 4c 8d 3c df 48 c1 e0 2a 4c 89 fa 48 c1 ea 03 <80> 3c 02 00 74 08 4c 89 ff e8 cc 76 52 ff 49 8b 07 4c 21 e8 49 bd All code ======== 0: 48 89 d3 mov %rdx,%rbx 3: 48 83 c8 ff or $0xffffffffffffffff,%rax 7: 83 e2 3f and $0x3f,%edx a: 49 89 fe mov %rdi,%r14 d: 48 c1 eb 06 shr $0x6,%rbx 11: c4 62 e9 f7 e8 shlx %rdx,%rax,%r13 16: b8 ff ff 37 00 mov $0x37ffff,%eax 1b: 4c 8d 3c df lea (%rdi,%rbx,8),%r15 1f: 48 c1 e0 2a shl $0x2a,%rax 23: 4c 89 fa mov %r15,%rdx 26: 48 c1 ea 03 shr $0x3,%rdx 2a:* 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1) <-- trapping instruction 2e: 74 08 je 0x38 30: 4c 89 ff mov %r15,%rdi 33: e8 cc 76 52 ff call 0xffffffffff527704 38: 49 8b 07 mov (%r15),%rax 3b: 4c 21 e8 and %r13,%rax 3e: 49 rex.WB 3f: bd .byte 0xbd
Code starting with the faulting instruction
===========================================
0: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1) 4: 74 08 je 0xe 6: 4c 89 ff mov %r15,%rdi 9: e8 cc 76 52 ff call 0xffffffffff5276da e: 49 8b 07 mov (%r15),%rax 11: 4c 21 e8 and %r13,%rax 14: 49 rex.WB 15: bd .byte 0xbd [ 18.146207][ T63] RSP: 0000:ffffc900001f7d28 EFLAGS: 00010206 [ 18.146207][ T63] RAX: dffffc0000000000 RBX: 0000000000000000 RCX: fffffbfff0bdd81a [ 18.146207][ T63] RDX: 0000000000000012 RSI: 0000000000000002 RDI: 0000000000000090 [ 18.146207][ T63] RBP: ffffc900001f7d50 R08: 0000000000000000 R09: 0000000000000003 [ 18.146207][ T63] R10: ffffffff8145362f R11: ffffed103099ffd5 R12: 0000000000000002 [ 18.146207][ T63] R13: ffffffffffffffff R14: 0000000000000090 R15: 0000000000000090 [ 18.146207][ T63] FS: 0000000000000000(0000) GS:ffff888428798000(0000) knlGS:0000000000000000 [ 18.146207][ T63] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 [ 18.146207][ T63] CR2: 00000000f7fc18c8 CR3: 0000000185368000 CR4: 00000000000406b0 [ 18.146207][ T63] Call Trace: [ 18.146207][ T63] <TASK> [ 18.146207][ T63] task_cache_work (kbuild/src/consumer/include/linux/find.h:71 (discriminator 2) kbuild/src/consumer/kernel/sched/fair.c:1325 (discriminator 2)) [ 18.146207][ T63] ? _raw_spin_unlock_irq (kbuild/src/consumer/arch/x86/include/asm/paravirt.h:671 kbuild/src/consumer/include/linux/spinlock_api_smp.h:159 kbuild/src/consumer/kernel/locking/spinlock.c:202) [ 18.146207][ T63] ? __pfx_task_cache_work (kbuild/src/consumer/kernel/sched/fair.c:1299) [ 18.146207][ T63] ? __this_cpu_preempt_check (kbuild/src/consumer/lib/smp_processor_id.c:67)
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20250811/[email protected]
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [RFC PATCH v4 01/28] sched: Cache aware load-balancing
Hi,
Quoted 3 lines · kernel test robot
[ 18.146207][ T63] RIP: 0010:_find_next_bit (kbuild/src/consumer/lib/find_bit.c:156 (discriminator 2)) [ 18.146207][ T63] task_cache_work (kbuild/src/consumer/include/linux/find.h:71 (discriminator 2)
kbuild/src/consumer/kernel/sched/fair.c:1325(discriminator 2))
This should have been fixed in the second patch:
02/28 ("sched: Several fixes for cache aware scheduling")that the LLC sched domain could be NULL and we should skip the NULL LLC sched domain.
It might be more applicable to test with the whole patch set. As this is a RFC version and we will create a formal one with the fixes integrated into the 01/28 patch, after the issues have been addressed.
thanks,
Chenyu
RFC PATCH v4 2/28 sched: Several fixes for cache aware scheduling
In reply to Chen Yu1. Fix compile error on percpu allocation.
2. Enqueue to the target CPU rather than the current CPU.
3. NULL LLC sched domain check(Libo Chen).
4. Introduce sched feature SCHED_CACHE to control cache aware scheduling
5. Fix unsigned occupancy initialization to -1.
6. If there is only 1 thread in the process, no need to enable cache awareness
7. Add __maybe_unused to __migrate_degrades_locality() to avoid compile warnings.
8. Do not enable gcov coverage for task_cache_work() and fraction_mm_sched() to avoid softlockup by gcov.
9. Make CONFIG_SCHED_CACHE depending on CONFIG_SMP to
avoid compile error on non-SMP system like microblaze architecture.
10. Do not enable account cache aware statistics in account_mm_sched() for non-normal tasks, as it could
be invoked by RT tasks.(Shrikanth Hegde)
11. Place cpu_epoch related fields in a dedicated cache line to avoid interfering with clock_idle* fields.
(Shrikanth Hegde)
--- include/linux/mm_types.h | 4 ++-- init/Kconfig | 4 ++++ kernel/sched/fair.c | 41 +++++++++++++++++++++++++++------------- kernel/sched/features.h | 1 + kernel/sched/sched.h | 2 +- 5 files changed, 36 insertions(+), 16 deletions(-)
include/linux/mm_types.h +2 -2
@@ -1450,11 +1450,11 @@ static inline void mm_set_cpus_allowed(struct mm_struct *mm, const struct cpumas#endif /* CONFIG_SCHED_MM_CID */ #ifdef CONFIG_SCHED_CACHE -extern void mm_init_sched (struct mm_struct *mm,struct mm_sched *pcpu_sched); +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *pcpu_sched);static inline int mm_alloc_sched_noprof (struct mm_struct *mm) { - struct mm_sched *pcpu_sched = alloc_percpu_noprof(struct mm_sched); + struct mm_sched __percpu *pcpu_sched = alloc_percpu_noprof(struct mm_sched);if (!pcpu_sched)return -ENOMEM ;
init/Kconfig +4 -0
@@ -950,6 +950,10 @@ config NUMA_BALANCING config SCHED_CACHE bool "Cache aware scheduler" default y + depends on SMP + help + If set, the scheduler will try to aggregate tasks in the same process to + a single LLC if possible. config NUMA_BALANCING_DEFAULT_ENABLED bool "Automatically enable NUMA aware memory/task placement"
kernel/sched/fair.c +28 -13
122 lines not on this page · raw message
kernel/sched/features.h +1 -0
@@ -87,6 +87,7 @@ SCHED_FEAT(TTWU_QUEUE, true)*/ SCHED_FEAT(SIS_UTIL, true) +SCHED_FEAT (SCHED_CACHE , true)/* * Issue a WARN when we do multiple update_rq_clock() calls * in a single rq->lock section. Default disabled because the
kernel/sched/sched.h +1 -1
@@ -1174,7 +1174,7 @@ struct rq {u64 clock_idle_copy;#endif #ifdef CONFIG_SCHED_CACHE -raw_spinlock_t cpu_epoch_lock; +raw_spinlock_t cpu_epoch_lock____cacheline_aligned ;u64 cpu_runtime;unsigned long cpu_epoch;unsigned long cpu_epoch_next;
Signature · 1 line
2.25.1
RFC PATCH v4 3/28 sched: Avoid task migration within its preferred LLC
In reply to Chen YuIt was found that when running schbench, there is a significant amount of in-LLC task migrations, even if the wakee is woken up on its preferred LLC. This leads to core-to-core latency and impairs performance.
Inhibit task migration if the wakee is already in its preferred LLC.
--- kernel/sched/fair.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/sched/fair.c +2 -0
@@ -8835,6 +8835,8 @@ static int select_cache_cpu(struct task_struct *p, int prev_cpu)if (cpu <0 )return prev_cpu; +if (cpus_share_cache (cpu, prev_cpu)) +return prev_cpu;if (static_branch_likely (&sched_numa_balancing) &&__migrate_degrades_locality (p, prev_cpu, cpu, false) >0 ) {
Signature · 1 line
2.25.1
RFC PATCH v4 4/28 sched: Avoid calculating the cpumask if the system is overloaded
In reply to Chen YuFrom: K Prateek Nayak <[email protected]>
If SIS_UTIL terminates the search for idle CPUs, the result of cpumask_and() becomes irrelevant. Given that select_idle_cpu() may now be invoked twice per wake-up within select_idle_sibling() due to cache-aware wake-ups, this overhead can be observed in benchmarks such as hackbench.
To conserve additional cycles-particularly in scenarios where the LLC is frequently targeted and the search aborts because the LLC is busy - calculate the cpumask only when the system is not overloaded.
--- kernel/sched/fair.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/fair.c +2 -2
@@ -7940,8 +7940,6 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, boolint i, cpu, idle_cpu =-1 , nr =INT_MAX ;struct sched_domain_shared *sd_share; -cpumask_and (cpus,sched_domain_span (sd), p->cpus_ptr); -if (sched_feat (SIS_UTIL )) { sd_share =rcu_dereference (per_cpu (sd_llc_shared, target));if (sd_share) {
@@ -7953,6 +7951,8 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool } } +cpumask_and (cpus,sched_domain_span (sd), p->cpus_ptr); +if (static_branch_unlikely (&sched_cluster_active)) {struct sched_group *sg = sd->groups;
Signature · 1 line
2.25.1
RFC PATCH v4 5/28 sched: Add hysteresis to switch a task's preferred LLC
In reply to Chen YuFrom: Tim Chen <[email protected]>
Switching a process's preferred LLC generates lots of task migrations across LLCs. To avoid frequent switches of home LLC, implement the following policy:
1. Require a 2x occ change threshold to switch preferred LLC
2. Don't discard preferred LLC for a task
--- kernel/sched/fair.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)
kernel/sched/fair.c +16 -8
@@ -1175,6 +1175,14 @@ static s64 update_curr_se(struct rq *rq, struct sched_entity *curr)#define EPOCH_PERIOD (HZ/100)/* 10 ms */ #define EPOCH_OLD 5/* 50 ms */ +static int llc_id (int cpu) +{ +if (cpu <0 ) +return -1 ; + +return per_cpu (sd_llc_id, cpu); +} +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched) {unsigned long epoch;
@@ -1307,6 +1315,7 @@ static void __no_profile task_cache_work(struct callback_head *work)struct task_struct *p = current;struct mm_struct *mm = p->mm;unsigned long m_a_occ =0 ; +unsigned long last_m_a_occ =0 ;int cpu, m_a_cpu =-1 ;cpumask_var_t cpus;
@@ -1345,11 +1354,13 @@ static void __no_profile task_cache_work(struct callback_head *work) per_cpu(sd_llc_id, i), occ, m_occ, m_cpu, nr); } - a_occ /= nr; +// a_occ /= nr; if (a_occ > m_a_occ) { m_a_occ = a_occ; m_a_cpu = m_cpu; } +if (llc_id (cpu) ==llc_id (mm->mm_sched_cpu)) + last_m_a_occ = a_occ;trace_printk ("(%d) a_occ: %ld m_a_occ: %ld\n" ,per_cpu (sd_llc_id, cpu), a_occ, m_a_occ);
@@ -1363,13 +1374,10 @@ static void __no_profile task_cache_work(struct callback_head *work) } } -/* -* If the max average cache occupancy is 'small' we don't care. -*/ -if (m_a_occ < (NICE_0_LOAD >>EPOCH_OLD )) - m_a_cpu =-1 ; - - mm->mm_sched_cpu = m_a_cpu; +if (m_a_occ > (2 * last_m_a_occ)) { +/* avoid the bouncing of mm_sched_cpu */ + mm->mm_sched_cpu = m_a_cpu; + }free_cpumask_var (cpus); }
Signature · 1 line
2.25.1
Peter Zijlstra
maintainerRe: [RFC PATCH v4 05/28] sched: Add hysteresis to switch a task's preferred LLC
Quoted 42 lines · Chen Yu
From: Tim Chen <[email protected]>
Switching a process's preferred LLC generates lots of task migrations across LLCs. To avoid frequent switches of home LLC, implement the following policy:
1. Require a 2x occ change threshold to switch preferred LLC
2. Don't discard preferred LLC for a taskSigned-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)kernel/sched/fair.c +9
-0@@ -1175,6 +1175,14 @@ static s64 update_curr_se(struct rq *rq, struct sched_entity *curr)#define EPOCH_PERIOD (HZ/100)/* 10 ms */ #define EPOCH_OLD 5/* 50 ms */ +static int llc_id (int cpu) +{ +if (cpu <0 ) +return -1 ; + +return per_cpu (sd_llc_id, cpu); +} +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched) {unsigned long epoch;@@ -1307,6 +1315,7 @@ static void __no_profile task_cache_work(struct callback_head *work)struct task_struct *p = current;struct mm_struct *mm = p->mm;unsigned long m_a_occ =0 ; +unsigned long last_m_a_occ =0 ;int cpu, m_a_cpu =-1 ;cpumask_var_t cpus;
kernel/sched/fair.c static void __no_profile task_cache_work(struct callback_head *work) · line 1357
Quoted 5 lines
kernel/sched/fair.c +1
-1@@ -1345,11 +1354,13 @@ static void __no_profile task_cache_work(struct callback_head *work) per_cpu(sd_llc_id, i), occ, m_occ, m_cpu, nr); } - a_occ /= nr; +// a_occ /= nr;
This seems broken.
kernel/sched/fair.c static void __no_profile task_cache_work(struct callback_head *work) · line 1363
Quoted 6 lines
kernel/sched/fair.c +2
-0if (a_occ > m_a_occ) { m_a_occ = a_occ; m_a_cpu = m_cpu; } +if (llc_id (cpu) ==llc_id (mm->mm_sched_cpu)) + last_m_a_occ = a_occ;
Not 'last', 'curr' perhaps?
Quoted 8 lines
kernel/sched/fair.c +0
-1trace_printk ("(%d) a_occ: %ld m_a_occ: %ld\n" ,per_cpu (sd_llc_id, cpu), a_occ, m_a_occ);@@ -1363,13 +1374,10 @@ static void __no_profile task_cache_work(struct callback_head *work) } } -/*
kernel/sched/fair.c static void __no_profile task_cache_work(struct callback_head *work) · line 1380
Quoted 10 lines
kernel/sched/fair.c +4
-6-* If the max average cache occupancy is 'small' we don't care. -*/ -if (m_a_occ < (NICE_0_LOAD >>EPOCH_OLD )) - m_a_cpu =-1 ; - - mm->mm_sched_cpu = m_a_cpu; +if (m_a_occ > (2 * last_m_a_occ)) { +/* avoid the bouncing of mm_sched_cpu */ + mm->mm_sched_cpu = m_a_cpu; + }
The whole double thing doesn't seem right either. That means that anything over .5 will never be able to change, even when confronted with say a .8.
Also, while this is a threshold, this is not in fact hysteresis.
Re: [RFC PATCH v4 05/28] sched: Add hysteresis to switch a task's preferred LLC
Quoted 42 lines · Peter Zijlstra
Quoted 41 lines · Chen Yu
From: Tim Chen <[email protected]>
Switching a process's preferred LLC generates lots of task migrations across LLCs. To avoid frequent switches of home LLC, implement the following policy:
1. Require a 2x occ change threshold to switch preferred LLC
2. Don't discard preferred LLC for a taskSigned-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)kernel/sched/fair.c +9
-0@@ -1175,6 +1175,14 @@ static s64 update_curr_se(struct rq *rq, struct sched_entity *curr)#define EPOCH_PERIOD (HZ/100)/* 10 ms */ #define EPOCH_OLD 5/* 50 ms */ +static int llc_id (int cpu) +{ +if (cpu <0 ) +return -1 ; + +return per_cpu (sd_llc_id, cpu); +} +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched) {unsigned long epoch;@@ -1307,6 +1315,7 @@ static void __no_profile task_cache_work(struct callback_head *work)struct task_struct *p = current;struct mm_struct *mm = p->mm;unsigned long m_a_occ =0 ; +unsigned long last_m_a_occ =0 ;int cpu, m_a_cpu =-1 ;cpumask_var_t cpus;
Quoted 9 lines
kernel/sched/fair.c static void __no_profile task_cache_work(struct callback_head *work) · line 1357
Quoted 6 lines
kernel/sched/fair.c +1
-1@@ -1345,11 +1354,13 @@ static void __no_profile task_cache_work(struct callback_head *work) per_cpu(sd_llc_id, i), occ, m_occ, m_cpu, nr); } - a_occ /= nr; +// a_occ /= nr; This seems broken.
The original thought was that not dividing the nr might help in an asymmetric LLC scenario.
If there are 2 threads in the 4-CPU LLC and 3 threads in the 8-CPU LLC, it might be better to choose the one with 3 threads. But it wouldn't be the case if division is done by the nr.
And the NUMA balancing fault statistic behaves similarly:
it compares the total number of faults per node rather than the average number of faults per CPU, IIUC.
Quoted 9 lines
kernel/sched/fair.c static void __no_profile task_cache_work(struct callback_head *work) · line 1363
Quoted 6 lines
kernel/sched/fair.c +2
-0if (a_occ > m_a_occ) { m_a_occ = a_occ; m_a_cpu = m_cpu; } +if (llc_id (cpu) ==llc_id (mm->mm_sched_cpu)) + last_m_a_occ = a_occ;Not 'last', 'curr' perhaps?
OK.
Quoted 24 lines
Quoted 8 lines
kernel/sched/fair.c +0
-1trace_printk ("(%d) a_occ: %ld m_a_occ: %ld\n" ,per_cpu (sd_llc_id, cpu), a_occ, m_a_occ);@@ -1363,13 +1374,10 @@ static void __no_profile task_cache_work(struct callback_head *work) } } -/* kernel/sched/fair.c static void __no_profile task_cache_work(struct callback_head *work) · line 1380
Quoted 10 lines
kernel/sched/fair.c +4
-6-* If the max average cache occupancy is 'small' we don't care. -*/ -if (m_a_occ < (NICE_0_LOAD >>EPOCH_OLD )) - m_a_cpu =-1 ; - - mm->mm_sched_cpu = m_a_cpu; +if (m_a_occ > (2 * last_m_a_occ)) { +/* avoid the bouncing of mm_sched_cpu */ + mm->mm_sched_cpu = m_a_cpu; + }The whole double thing doesn't seem right either. That means that anything over .5 will never be able to change, even when confronted with say a .8.
Quoted 2 lines
Also, while this is a threshold, this is not in fact hysteresis.
I agree 2X might be too aggressive. The threshold here was intended to avoid switching to a new mm_sched_cpu too quickly, but it is not intended to address the back-and-forth bouncing issue that hysteresis would resolve. Instead, the task migration bouncing issue is solved in patch 7/28 via a hysteresis.
Maybe the comment above:
/* avoid the bouncing of mm_sched_cpu */
should be changed to:
/* avoid switching to mm_sched_cpu too fast */
I'll investigate an imbalance_pct value that is less aggressive than 2X.
thanks,
Chenyu
RFC PATCH v4 6/28 sched: Save the per LLC utilization for better cache aware scheduling
In reply to Chen YuWhen a system gets busy and a process's preferred LLC
is saturated by too many threads within this process, there are significant in-LLC task migrations within its preferred LLC. This leads to migration latency and degrades performance. Ideally, task aggregation should be inhibited if the task's preferred LLC is overloaded. This implies that a metric is needed to indicate whether the LLC is busy.
Store the per-LLC utilization calculated via periodic load balancing. These statistics will be used in subsequent patches to determine whether tasks should be aggregated to their preferred LLC.
--- include/linux/sched/topology.h | 3 ++ kernel/sched/fair.c | 53 ++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+)
include/linux/sched/topology.h +3 -0
@@ -78,6 +78,9 @@ struct sched_domain_shared {atomic_t nr_busy_cpus;int has_idle_cores;int nr_idle_scan; +#ifdef CONFIG_SCHED_CACHE +unsigned long util_avg; +#endif };struct sched_domain {
kernel/sched/fair.c +53 -0
71 lines not on this page · raw message
Signature · 1 line
2.25.1
Peter Zijlstra
maintainerRe: [RFC PATCH v4 06/28] sched: Save the per LLC utilization for better cache aware scheduling
Quoted 10 lines · Chen Yu
kernel/sched/fair.c +10
-0+#ifdef CONFIG_SCHED_CACHE +/* +* Save this sched group's statistic for later use: +* The task wakeup and load balance can make better +* decision based on these statistics. +*/ +static void update_sg_if_llc (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +/* Find the sched domain that spans this group. */
kernel/sched/fair.c sched_reduced_capacity(struct rq *rq, struct sched_domain *sd) · line 10708
Quoted 10 lines
kernel/sched/fair.c +10
-0+struct sched_domain *sd = env->sd->child; +struct sched_domain_shared *sd_share; + +if (!sched_feat (SCHED_CACHE ) || env->idle ==CPU_NEWLY_IDLE ) +return ; + +/* only care the sched domain that spans 1 LLC */ +if (!sd || !(sd->flags &SD_SHARE_LLC ) || + !sd->parent || (sd->parent->flags &SD_SHARE_LLC )) +return ;
Did you want to write:
if (sd != per_cpu(sd_llc)) return;
Or something?
kernel/sched/fair.c sched_reduced_capacity(struct rq *rq, struct sched_domain *sd) · line 10716
Quoted 7 lines
kernel/sched/fair.c +7
-0+ sd_share =rcu_dereference (per_cpu (sd_llc_shared, +cpumask_first (sched_group_span (group)))); +if (!sd_share) +return ; + +if (likely (READ_ONCE (sd_share->util_avg) != sgs->group_util)) +WRITE_ONCE (sd_share->util_avg, sgs->group_util);
If you expect it to be different, does that whole load and compare still matter?
Quoted 1 line
+}
Re: [RFC PATCH v4 06/28] sched: Save the per LLC utilization for better cache aware scheduling
Quoted 25 lines · Peter Zijlstra
Quoted 10 lines · Chen Yu
kernel/sched/fair.c +10
-0+#ifdef CONFIG_SCHED_CACHE +/* +* Save this sched group's statistic for later use: +* The task wakeup and load balance can make better +* decision based on these statistics. +*/ +static void update_sg_if_llc (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +/* Find the sched domain that spans this group. */ kernel/sched/fair.c sched_reduced_capacity(struct rq *rq, struct sched_domain *sd) · line 10708
Quoted 10 lines
kernel/sched/fair.c +10
-0+struct sched_domain *sd = env->sd->child; +struct sched_domain_shared *sd_share; + +if (!sched_feat (SCHED_CACHE ) || env->idle ==CPU_NEWLY_IDLE ) +return ; + +/* only care the sched domain that spans 1 LLC */ +if (!sd || !(sd->flags &SD_SHARE_LLC ) || + !sd->parent || (sd->parent->flags &SD_SHARE_LLC )) +return ;Did you want to write:
Quoted 5 lines
if (sd != per_cpu(sd_llc)) return;Or something?
Ah right, will do it.
Quoted 11 lines
kernel/sched/fair.c sched_reduced_capacity(struct rq *rq, struct sched_domain *sd) · line 10716
Quoted 7 lines
kernel/sched/fair.c +7
-0+ sd_share =rcu_dereference (per_cpu (sd_llc_shared, +cpumask_first (sched_group_span (group)))); +if (!sd_share) +return ; + +if (likely (READ_ONCE (sd_share->util_avg) != sgs->group_util)) +WRITE_ONCE (sd_share->util_avg, sgs->group_util);If you expect it to be different, does that whole load and compare still matter?
OK, will write to it directly.
thanks,
Chenyu>> +}
RFC PATCH v4 7/28 sched: Add helper function to decide whether to allow cache aware scheduling
In reply to Chen YuFrom: Tim Chen <[email protected]>
Cache-aware scheduling is designed to aggregate threads into their preferred LLC, either via the task wake up path or the load balancing path. One side effect is that when the preferred LLC is saturated, more threads will continue to be stacked on it, degrading the workload's latency. A strategy is needed to prevent this aggregation from going too far such that the preferred LLC is too overloaded.
Introduce helper function _get_migrate_hint() to implement the LLC migration policy:
1) A task is aggregated to its preferred LLC if both source/dest LLC are not too busy (<50% utilization, tunable), or the preferred LLC will not be too out of balanced from the non preferred LLC (>20% utilization, tunable, close to imbalance_pct of the LLC domain).
2) Allow a task to be moved from the preferred LLC to the non-preferred one if the non-preferred LLC will not be too out of balanced from the preferred prompting an aggregation task migration later. We are still experimenting with the aggregation and migration policy. Some other possibilities are policy based on LLC's load or average number of tasks running. Those could be tried out by tweaking _get_migrate_hint().
The function _get_migrate_hint() returns migration suggestions for the upper-level functions.
Aggregation will tend to make utilization on the preferred LLC to be more than the non-preferred one. Parameter "sysctl_llc_aggr_imb" is the imbalance allowed. If it is set to 0, as long as the preferred LLC is not utilized more than the source LLC, we could still aggregate towards the preferred LLC and a preference could still be there.
--- kernel/sched/debug.c | 4 ++ kernel/sched/fair.c | 110 ++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 5 ++ 3 files changed, 118 insertions(+), 1 deletion(-)
kernel/sched/debug.c +4 -0
@@ -532,6 +532,10 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("hot_threshold_ms" ,0644 , numa, &sysctl_numa_balancing_hot_threshold);#endif +#ifdef CONFIG_SCHED_CACHE +debugfs_create_u32 ("llc_aggr_cap" ,0644 , debugfs_sched, &sysctl_llc_aggr_cap); +debugfs_create_u32 ("llc_aggr_imb" ,0644 , debugfs_sched, &sysctl_llc_aggr_imb); +#endif debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);debugfs_fair_server_init ();
kernel/sched/fair.c +109 -1
122 lines not on this page · raw message
kernel/sched/sched.h +5 -0
@@ -2844,6 +2844,11 @@ extern unsigned int sysctl_numa_balancing_scan_period_max;extern unsigned int sysctl_numa_balancing_scan_size;extern unsigned int sysctl_numa_balancing_hot_threshold; +#ifdef CONFIG_SCHED_CACHE +extern unsigned int sysctl_llc_aggr_cap; +extern unsigned int sysctl_llc_aggr_imb; +#endif +#ifdef CONFIG_SCHED_HRTICK /*
Signature · 1 line
2.25.1
Peter Zijlstra
maintainerRe: [RFC PATCH v4 07/28] sched: Add helper function to decide whether to allow cache aware scheduling
Quoted 8 lines · Chen Yu
From: Tim Chen <[email protected]>
Cache-aware scheduling is designed to aggregate threads into their preferred LLC, either via the task wake up path or the load balancing path. One side effect is that when the preferred LLC is saturated, more threads will continue to be stacked on it, degrading the workload's latency. A strategy is needed to prevent this aggregation from going too far such that the preferred LLC is too overloaded.
So one of the ideas was to extend the preferred llc number to a mask. Update the preferred mask with (nr_threads / llc_size) bits, indicating the that many top llc as sorted by occupancy.
Re: [RFC PATCH v4 07/28] sched: Add helper function to decide whether to allow cache aware scheduling
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Chen Yu
From: Tim Chen <[email protected]>
Cache-aware scheduling is designed to aggregate threads into their preferred LLC, either via the task wake up path or the load balancing path. One side effect is that when the preferred LLC is saturated, more threads will continue to be stacked on it, degrading the workload's latency. A strategy is needed to prevent this aggregation from going too far such that the preferred LLC is too overloaded.
Quoted 5 lines
So one of the ideas was to extend the preferred llc number to a mask. Update the preferred mask with (
nr_threads/llc_size) bits, indicating the that many top llc as sorted by occupancy.
Having more than one preferred LLC helps prevent aggregation from going too far on a single preferred LLC.
One question would be: if one LLC cannot hold all the threads of a process, does a second preferred LLC help in this use case? Currently, this patch gives up task aggregation and falls back to legacy load balancing if the preferred LLC is overloaded. If we place threads across two preferred LLCs, these threads might encounter cross-LLC latency anyway - so we may as well let legacy load balancing spread them out IMO.
Another issue that Patch 7 tries to address is avoiding task bouncing between preferred LLCs and non-preferred LLCs. If we introduce a preferred LLC priority list, logic to prevent task bouncing between different preferred LLCs might be needed in load balancing, which could become complicated. Currently, we mainly implement cache-aware scheduling in load balancing rather than during task wakeup, because the wakeup path conflicts with the load balance path and causes task migration bouncing.
thanks,
Chenyu
Peter Zijlstra
maintainerRe: [RFC PATCH v4 07/28] sched: Add helper function to decide whether to allow cache aware scheduling
Quoted 20 lines · Yu C
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Chen Yu
From: Tim Chen <[email protected]>
Cache-aware scheduling is designed to aggregate threads into their preferred LLC, either via the task wake up path or the load balancing path. One side effect is that when the preferred LLC is saturated, more threads will continue to be stacked on it, degrading the workload's latency. A strategy is needed to prevent this aggregation from going too far such that the preferred LLC is too overloaded.
Quoted 5 lines
So one of the ideas was to extend the preferred llc number to a mask. Update the preferred mask with (
nr_threads/llc_size) bits, indicating the that many top llc as sorted by occupancy.Having more than one preferred LLC helps prevent aggregation from going too far on a single preferred LLC.
Quoted 7 lines
One question would be: if one LLC cannot hold all the threads of a process, does a second preferred LLC help in this use case? Currently, this patch gives up task aggregation and falls back to legacy load balancing if the preferred LLC is overloaded. If we place threads across two preferred LLCs, these threads might encounter cross-LLC latency anyway - so we may as well let
legacy load balancing spread them out IMO.
Well, being stuck on 2 LLCs instead of being spread across 10 still seems like a win, no?
Remember, our friends at AMD have *MANY* LLCs.
Quoted 5 lines
Another issue that Patch 7 tries to address is avoiding task bouncing between preferred LLCs and non-preferred LLCs. If we introduce a preferred LLC priority list, logic to prevent task bouncing between different preferred LLCs might be needed in load balancing, which could become complicated.
It doesn't really become more difficult to tell preferred LLC from non-preferred LLC with a asm. So why should things get more complicatd?
Anyway, it was just one of the 'random' ideas I had kicking about. Reality always ruins things, *shrug* :-)
Re: [RFC PATCH v4 07/28] sched: Add helper function to decide whether to allow cache aware scheduling
Quoted 29 lines · Peter Zijlstra
Quoted 20 lines · Yu C
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Chen Yu
From: Tim Chen <[email protected]>
Cache-aware scheduling is designed to aggregate threads into their preferred LLC, either via the task wake up path or the load balancing path. One side effect is that when the preferred LLC is saturated, more threads will continue to be stacked on it, degrading the workload's latency. A strategy is needed to prevent this aggregation from going too far such that the preferred LLC is too overloaded.
Quoted 5 lines
So one of the ideas was to extend the preferred llc number to a mask. Update the preferred mask with (
nr_threads/llc_size) bits, indicating the that many top llc as sorted by occupancy.Having more than one preferred LLC helps prevent aggregation from going too far on a single preferred LLC.
Quoted 7 lines
One question would be: if one LLC cannot hold all the threads of a process, does a second preferred LLC help in this use case? Currently, this patch gives up task aggregation and falls back to legacy load balancing if the preferred LLC is overloaded. If we place threads across two preferred LLCs, these threads might encounter cross-LLC latency anyway - so we may as well let
legacy load balancing spread them out IMO.
Quoted 5 lines
Well, being stuck on 2 LLCs instead of being spread across 10 still seems like a win, no?
Remember, our friends at AMD have *MANY* LLCs.
I see, this makes sense.
Quoted 9 lines
Quoted 5 lines
Another issue that Patch 7 tries to address is avoiding task bouncing between preferred LLCs and non-preferred LLCs. If we introduce a preferred LLC priority list, logic to prevent task bouncing between different preferred LLCs might be needed in load balancing, which could become complicated.
It doesn't really become more difficult to tell preferred LLC from non-preferred LLC with a asm. So why should things get more complicatd?
Besides distinguishing between preferred LLCs and non-preferred LLCs, we might also want to distinguish between the ith preferred LLC and the jth preferred LLC—this would help avoid task migration bouncing between them by using hysteresis.
Quoted 3 lines
Anyway, it was just one of the 'random' ideas I had kicking about. Reality always ruins things, *shrug* :-)
Yes, multiple preferred LLCs is a promising direction if the data shows good results. We are planning to clean up the RFC patch and send a refreshed version to summarize the current status. Meanwhile, we will evaluate the multi-preferred LLC approach internally. Thanks for providing this idea.
thanks,
Chenyu
Re: [RFC PATCH v4 07/28] sched: Add helper function to decide whether to allow cache aware scheduling
In reply to Peter ZijlstraQuoted 34 lines · Peter Zijlstra
Quoted 20 lines · Yu C
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Chen Yu
From: Tim Chen <[email protected]>
Cache-aware scheduling is designed to aggregate threads into their preferred LLC, either via the task wake up path or the load balancing path. One side effect is that when the preferred LLC is saturated, more threads will continue to be stacked on it, degrading the workload's latency. A strategy is needed to prevent this aggregation from going too far such that the preferred LLC is too overloaded.
Quoted 5 lines
So one of the ideas was to extend the preferred llc number to a mask. Update the preferred mask with (
nr_threads/llc_size) bits, indicating the that many top llc as sorted by occupancy.Having more than one preferred LLC helps prevent aggregation from going too far on a single preferred LLC.
Quoted 7 lines
One question would be: if one LLC cannot hold all the threads of a process, does a second preferred LLC help in this use case? Currently, this patch gives up task aggregation and falls back to legacy load balancing if the preferred LLC is overloaded. If we place threads across two preferred LLCs, these threads might encounter cross-LLC latency anyway - so we may as well let
legacy load balancing spread them out IMO.Well, being stuck on 2 LLCs instead of being spread across 10 still seems like a win, no?
Remember, our friends at AMD have *MANY* LLCs.
Quoted 9 lines
Quoted 5 lines
Another issue that Patch 7 tries to address is avoiding task bouncing between preferred LLCs and non-preferred LLCs. If we introduce a preferred LLC priority list, logic to prevent task bouncing between different preferred LLCs might be needed in load balancing, which could become complicated.
It doesn't really become more difficult to tell preferred LLC from non-preferred LLC with a asm. So why should things get more complicatd?
For secondary and maybe tertiary LLCs to work well, the ordering of the occupancy between the LLCs have to be relatively stable. Or else we could have many tasks migration between the LLCs when the ordering change. Frequent task migrations could be worse for performance.
From previous experiments, we saw that the occupancy could have some fairly big fluctuations. That's the reason we set the preferred LLC threshold to be high (2x). We want to be sure before jerking tasks around to a new LLC.
With the secondary, tertiary LLCs, LLC ordering would change more frequently than having just a single preferred LLC. The secondary and tertiary LLCs have fewer tasks/mm and occupancy could fluctuate more.
One concern is this could lead to extra task migrations that could negate any cache consolidation benefits gained.
Tim
Quoted 3 lines
Anyway, it was just one of the 'random' ideas I had kicking about. Reality always ruins things, *shrug* :-)
RFC PATCH v4 8/28 sched: Set up LLC indexing
In reply to Chen YuFrom: Tim Chen <[email protected]>
Prepare for indexing arrays that track in each run queue: the number of tasks preferring current LLC and each of the other LLC.
The reason to introduce LLC index is because the per LLC-scope data is needed to do cache aware load balancing. However, the native lld_id is usually the first CPU of that LLC domain, which is not continuous, which might waste the space if the per LLC-scope data is stored in an array (in current implementation).
In the future, this LLC index could be removed after the native llc_id is used as the key to search into xarray based array.
--- include/linux/sched.h | 3 +++ kernel/sched/fair.c | 12 ++++++++++++ kernel/sched/sched.h | 2 ++ kernel/sched/topology.c | 29 +++++++++++++++++++++++++++++ 4 files changed, 46 insertions(+)
include/linux/sched.h +3 -0
@@ -809,6 +809,9 @@ struct kmap_ctrl {#endif }; +/* XXX need fix to not use magic number */ +#define MAX_LLC 64 +struct task_struct {#ifdef CONFIG_THREAD_INFO_IN_TASK /*
kernel/sched/fair.c +12 -0
@@ -1183,6 +1183,18 @@ static int llc_id(int cpu)return per_cpu (sd_llc_id, cpu); } +/* +* continuous index. +* TBD: replace by xarray with key llc_id() +*/ +static inline int llc_idx (int cpu) +{ +if (cpu <0 ) +return -1 ; + +return per_cpu (sd_llc_idx, cpu); +} +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched) {unsigned long epoch;
kernel/sched/sched.h +2 -0
@@ -2056,6 +2056,7 @@ static inline struct sched_domain *lowest_flag_domain(int cpu, int flag) DECLARE_PER_CPU(struct sched_domain __rcu *, sd_llc);DECLARE_PER_CPU (int, sd_llc_size);DECLARE_PER_CPU (int, sd_llc_id); +DECLARE_PER_CPU (int, sd_llc_idx);DECLARE_PER_CPU (int, sd_share_id); DECLARE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_numa);
@@ -2064,6 +2065,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);extern struct static_key_false sched_asym_cpucapacity;extern struct static_key_false sched_cluster_active; +extern int max_llcs; static __always_inline bool sched_asym_cpucap_active(void) {
kernel/sched/topology.c +29 -0
@@ -657,6 +657,7 @@ static void destroy_sched_domains(struct sched_domain *sd) DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc);DEFINE_PER_CPU (int, sd_llc_size);DEFINE_PER_CPU (int, sd_llc_id); +DEFINE_PER_CPU (int, sd_llc_idx);DEFINE_PER_CPU (int, sd_share_id); DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa);
@@ -666,6 +667,25 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);DEFINE_STATIC_KEY_FALSE (sched_asym_cpucapacity);DEFINE_STATIC_KEY_FALSE (sched_cluster_active); +int max_llcs =-1 ; + +static void update_llc_idx (int cpu) +{ +#ifdef CONFIG_SCHED_CACHE +int idx =-1 , llc_id =-1 ; + + llc_id =per_cpu (sd_llc_id, cpu); + idx =per_cpu (sd_llc_idx, llc_id); + +if (idx <0 ) { + idx = max_llcs++; +BUG_ON (idx >MAX_LLC ); +per_cpu (sd_llc_idx, llc_id) = idx; + } +per_cpu (sd_llc_idx, cpu) = idx; +#endif +} +static void update_top_cache_domain (int cpu) {struct sched_domain_shared *sds =NULL ;
@@ -684,6 +704,7 @@ static void update_top_cache_domain(int cpu)per_cpu (sd_llc_size, cpu) = size;per_cpu (sd_llc_id, cpu) = id;rcu_assign_pointer (per_cpu (sd_llc_shared, cpu), sds); +update_llc_idx (cpu); sd =lowest_flag_domain (cpu,SD_CLUSTER );if (sd)
@@ -2456,6 +2477,14 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attbool has_asym = false;bool has_cluster = false; +#ifdef CONFIG_SCHED_CACHE +if (max_llcs <0 ) { +for_each_possible_cpu (i) +per_cpu (sd_llc_idx, i) =-1 ; + max_llcs =0 ; + } +#endif +if (WARN_ON (cpumask_empty (cpu_map))) goto error;
Signature · 1 line
2.25.1
Re: [RFC PATCH v4 08/28] sched: Set up LLC indexing
Hi Chen Yu,
I tested the patch set on AmpereOne CPU with 192 cores. With certain firmware setting, each core has its own L1/L2 cache. But *no* cores share LLC (L3). So *no* schedule domain has flag 'SD_SHARE_LLC'.
With this topology:per_cpu(sd_llc_id, cpu) is actually the cpu id (0-191).
And kernel bug will be triggered at:
'BUG_ON(idx > MAX_LLC)'
Please see details bellow.
The bug will disappear if setting 'MAX_LLC' to 192. But I think we might disable CAS(cache aware scheduling) if no domain has 'SD_SHARE_LLC'.
Quoted 28 lines · Chen Yu
From: Tim Chen <[email protected]>
Prepare for indexing arrays that track in each run queue: the number of tasks preferring current LLC and each of the other LLC.
The reason to introduce LLC index is because the per LLC-scope data is needed to do cache aware load balancing. However, the native
lld_idis usually the first CPU of that LLC domain, which is not continuous, which might waste the space if the per LLC-scope data is stored in an array (in current implementation).In the future, this LLC index could be removed after the native
llc_idis used as the key to search into xarray based array.Signed-off-by Tim Chen <[email protected]>--- include/linux/sched.h | 3 +++ kernel/sched/fair.c | 12 ++++++++++++ kernel/sched/sched.h | 2 ++ kernel/sched/topology.c | 29 +++++++++++++++++++++++++++++ 4 files changed, 46 insertions(+)include/linux/sched.h +0
-0
include/linux/sched.h struct kmap_ctrl { · line 813
Quoted 5 lines
include/linux/sched.h +2
-0@@ -809,6 +809,9 @@ struct kmap_ctrl {#endif }; +/* XXX need fix to not use magic number */ +#define MAX_LLC 64
The bug will disappear if setting 'MAX_LLC' to 192.
Quoted 66 lines
include/linux/sched.h +1
-0+struct task_struct {#ifdef CONFIG_THREAD_INFO_IN_TASK /* kernel/sched/fair.c +12
-0@@ -1183,6 +1183,18 @@ static int llc_id(int cpu)return per_cpu (sd_llc_id, cpu); } +/* +* continuous index. +* TBD: replace by xarray with key llc_id() +*/ +static inline int llc_idx (int cpu) +{ +if (cpu <0 ) +return -1 ; + +return per_cpu (sd_llc_idx, cpu); +} +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched) {unsigned long epoch;kernel/sched/sched.h +2
-0@@ -2056,6 +2056,7 @@ static inline struct sched_domain *lowest_flag_domain(int cpu, int flag) DECLARE_PER_CPU(struct sched_domain __rcu *, sd_llc);DECLARE_PER_CPU (int, sd_llc_size);DECLARE_PER_CPU (int, sd_llc_id); +DECLARE_PER_CPU (int, sd_llc_idx);DECLARE_PER_CPU (int, sd_share_id); DECLARE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_numa);@@ -2064,6 +2065,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);extern struct static_key_false sched_asym_cpucapacity;extern struct static_key_false sched_cluster_active; +extern int max_llcs; static __always_inline bool sched_asym_cpucap_active(void) {kernel/sched/topology.c +4
-0@@ -657,6 +657,7 @@ static void destroy_sched_domains(struct sched_domain *sd) DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc);DEFINE_PER_CPU (int, sd_llc_size);DEFINE_PER_CPU (int, sd_llc_id); +DEFINE_PER_CPU (int, sd_llc_idx);DEFINE_PER_CPU (int, sd_share_id); DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa);@@ -666,6 +667,25 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);DEFINE_STATIC_KEY_FALSE (sched_asym_cpucapacity);DEFINE_STATIC_KEY_FALSE (sched_cluster_active); +int max_llcs =-1 ; + +static void update_llc_idx (int cpu)
kernel/sched/topology.c DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); · line 682
Quoted 10 lines
kernel/sched/topology.c +10
-0+{ +#ifdef CONFIG_SCHED_CACHE +int idx =-1 , llc_id =-1 ; + + llc_id =per_cpu (sd_llc_id, cpu); + idx =per_cpu (sd_llc_idx, llc_id); + +if (idx <0 ) { + idx = max_llcs++; +BUG_ON (idx >MAX_LLC );
[ 2.793048] ------------[ cut here ]------------ [ 2.797737] kernel BUG at kernel/sched/topology.c:682! [ 2.802957] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP [snip] [ 2.917450] update_top_cache_domain+0x248/0x260 (P) [ 2.922493] cpu_attach_domain+0x170/0x250 [ 2.926653] build_sched_domains+0x640/0x7b0 [ 2.930989] sched_init_domains+0xbc/0xd8 [ 2.935062] sched_init_smp+0x48/0xd0 [ 2.938778] kernel_init_freeable+0xf4/0x148 [ 2.943115] kernel_init+0x2c/0x150 [ 2.946658] ret_from_fork+0x10/0x20 [ 2.950288] Code: d2800001 17ffffc7 d2800000 17ffffdf (d4210000) [ 2.956481] ---[ end trace 0000000000000000 ]--- [ 2.961170] Kernel panic - not syncing: Oops - BUG: Fatal exception [ 2.967540] SMP: stopping secondary CPUs [ 2.971529] ---[ end Kernel panic - not syncing: Oops - BUG: Fatal exception ]---
Quoted 22 lines
kernel/sched/topology.c +8
-0+ per_cpu(sd_llc_idx, llc_id) = idx; + } +per_cpu (sd_llc_idx, cpu) = idx; +#endif +} +static void update_top_cache_domain (int cpu) {struct sched_domain_shared *sds =NULL ;@@ -684,6 +704,7 @@ static void update_top_cache_domain(int cpu)per_cpu (sd_llc_size, cpu) = size;per_cpu (sd_llc_id, cpu) = id;rcu_assign_pointer (per_cpu (sd_llc_shared, cpu), sds); +update_llc_idx (cpu); sd =lowest_flag_domain (cpu,SD_CLUSTER );if (sd)@@ -2456,6 +2477,14 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attbool has_asym = false;bool has_cluster = false; +#ifdef CONFIG_SCHED_CACHE
kernel/sched/topology.c build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att · line 2490
Quoted 10 lines
kernel/sched/topology.c +7
-0+if (max_llcs <0 ) { +for_each_possible_cpu (i) +per_cpu (sd_llc_idx, i) =-1 ; + max_llcs =0 ; + } +#endif +if (WARN_ON (cpumask_empty (cpu_map))) goto error;
A draft patch like bellow can fix the kernel BUG:
1) Do not call update_llc_idx() if domain has no SD_SHARE_LLC
2) Disable CAS if domain has no SD_SHARE_LLC
kernel/sched/topology.c +8 -2
@@ -704,7 +704,8 @@ static void update_top_cache_domain(int cpu)per_cpu (sd_llc_size, cpu) = size;per_cpu (sd_llc_id, cpu) = id;rcu_assign_pointer (per_cpu (sd_llc_shared, cpu), sds); -update_llc_idx (cpu); +if (sd) +update_llc_idx (cpu); sd =lowest_flag_domain (cpu,SD_CLUSTER );if (sd)
@@ -2476,6 +2477,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attint i, ret = -ENOMEM ;bool has_asym = false;bool has_cluster = false; +bool has_llc = false;bool llc_has_parent_sd = false;unsigned int multi_llcs_node =1 ;
@@ -2621,6 +2623,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attif (lowest_flag_domain (i,SD_CLUSTER )) has_cluster = true; + +if (highest_flag_domain (i,SD_SHARE_LLC )) + has_llc = true; }rcu_read_unlock ();
@@ -2631,7 +2636,8 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attstatic_branch_inc_cpuslocked (&sched_cluster_active);#ifdef CONFIG_SCHED_CACHE -if (llc_has_parent_sd && multi_llcs_node && !sched_asym_cpucap_active ()) +if (has_llc && llc_has_parent_sd && multi_llcs_node && + !sched_asym_cpucap_active ())static_branch_inc_cpuslocked (&sched_cache_present);#endif
Thanks,
-adam
Re: [RFC PATCH v4 08/28] sched: Set up LLC indexing
Hi Adam,
Quoted 7 lines · Adam Li
Hi Chen Yu,
I tested the patch set on AmpereOne CPU with 192 cores. With certain firmware setting, each core has its own L1/L2 cache. But *no* cores share LLC (L3). So *no* schedule domain has flag '
SD_SHARE_LLC'.
Good catch! And many thanks for your detailed testing and analysis.
Is this issue triggered with CONFIG_SCHED_CLUSTER disabled?
Quoted 6 lines
With this topology:
per_cpu(sd_llc_id, cpu)is actually the cpu id (0-191).And kernel bug will be triggered at:
'BUG_ON(idx > MAX_LLC)'
Yes, the sd_llc_idx thing is a bit tricky - we want to use it to index into the static array struct sg_lb_stat.nr_pref_llc, and we have to limit its range. A better approach would be to dynamically allocate the buffer, so we could get rid of the 'idx > MAX_LLC' check, but that might complicate the code.
Quoted 6 lines
Please see details bellow.
The bug will disappear if setting '
MAX_LLC' to 192. But I think we might disable CAS(cache aware scheduling) if no domain has 'SD_SHARE_LLC'.
I agree with you. Simply disabling cache-aware scheduling if there is no SD_SHARE_LLC would be simpler.
Quoted 10 lines · Chen Yu
A draft patch like bellow can fix the kernel BUG:
1) Do not callupdate_llc_idx()if domain has noSD_SHARE_LLC
2) Disable CAS if domain has noSD_SHARE_LLCkernel/sched/topology.c +0
-0
kernel/sched/topology.c static void update_top_cache_domain(int cpu) · line 709
Quoted 7 lines
kernel/sched/topology.c +2
-1@@ -704,7 +704,8 @@ static void update_top_cache_domain(int cpu)per_cpu (sd_llc_size, cpu) = size;per_cpu (sd_llc_id, cpu) = id;rcu_assign_pointer (per_cpu (sd_llc_shared, cpu), sds); -update_llc_idx (cpu); +if (sd) +update_llc_idx (cpu);
OK, that make sense.
Quoted 21 lines
kernel/sched/topology.c +4
-0sd =lowest_flag_domain (cpu,SD_CLUSTER );if (sd)@@ -2476,6 +2477,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attint i, ret = -ENOMEM ;bool has_asym = false;bool has_cluster = false; +bool has_llc = false;bool llc_has_parent_sd = false;unsigned int multi_llcs_node =1 ;@@ -2621,6 +2623,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attif (lowest_flag_domain (i,SD_CLUSTER )) has_cluster = true; + +if (highest_flag_domain (i,SD_SHARE_LLC )) + has_llc = true; }rcu_read_unlock ();
kernel/sched/topology.c build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att · line 2639
Quoted 5 lines
kernel/sched/topology.c +1
-1@@ -2631,7 +2636,8 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attstatic_branch_inc_cpuslocked (&sched_cluster_active);#ifdef CONFIG_SCHED_CACHE -if (llc_has_parent_sd && multi_llcs_node && !sched_asym_cpucap_active ()) + if (has_llc && llc_has_parent_sd && multi_llcs_node &&
multi_llcs_node will be false if there is no SD_SHARE_LLC domain on the platform, so I suppose we don’t have to introduce has_llc?multi_llcs is set to true iff there are more than 1 SD_SHARE_LLC domains under its SD_SHARE_LLC parent domain.
thanks,
Chenyu
Quoted 7 lines
kernel/sched/topology.c +1
-0+ !sched_asym_cpucap_active())static_branch_inc_cpuslocked (&sched_cache_present);#endif Thanks,
-adam
Re: [RFC PATCH v4 08/28] sched: Set up LLC indexing
Quoted 11 lines · Yu C
Hi Adam,
Quoted 7 lines · Adam Li
Hi Chen Yu,
I tested the patch set on AmpereOne CPU with 192 cores. With certain firmware setting, each core has its own L1/L2 cache. But *no* cores share LLC (L3). So *no* schedule domain has flag '
SD_SHARE_LLC'.
Quoted 5 lines
Good catch! And many thanks for your detailed testing and analysis.
Is this issue triggered with
CONFIG_SCHED_CLUSTERdisabled?
Yes. With CONFIG_SCHED_CLUSTER enabled this issue will
not be triggered. The maximum sd_llc_idx will be less than MAX_LLC(64) since we have 24 (192/8) cluster domains.
Quoted 66 lines
Quoted 6 lines
With this topology:
per_cpu(sd_llc_id, cpu)is actually the cpu id (0-191).And kernel bug will be triggered at:
'BUG_ON(idx > MAX_LLC)'Yes, the
sd_llc_idxthing is a bit tricky - we want to use it to index into the static array struct sg_lb_stat.nr_pref_llc, and we have to limit its range. A better approach would be to dynamically allocate the buffer, so we could get rid of the 'idx >MAX_LLC' check, but that might complicate the code.Quoted 6 lines
Please see details bellow.
The bug will disappear if setting '
MAX_LLC' to 192. But I think we might disable CAS(cache aware scheduling) if no domain has 'SD_SHARE_LLC'.I agree with you. Simply disabling cache-aware scheduling if there is no
SD_SHARE_LLCwould be simpler.Quoted 10 lines · Chen Yu
A draft patch like bellow can fix the kernel BUG:
1) Do not callupdate_llc_idx()if domain has noSD_SHARE_LLC
2) Disable CAS if domain has noSD_SHARE_LLCkernel/sched/topology.c +0
-0kernel/sched/topology.c static void update_top_cache_domain(int cpu) · line 709
Quoted 7 lines
kernel/sched/topology.c +2
-1@@ -704,7 +704,8 @@ static void update_top_cache_domain(int cpu) per_cpu(sd_llc_size, cpu) = size; per_cpu(sd_llc_id, cpu) = id; rcu_assign_pointer(per_cpu(sd_llc_shared, cpu), sds); - update_llc_idx(cpu); + if (sd) + update_llc_idx(cpu);OK, that make sense.
Quoted 23 lines
kernel/sched/topology.c +4
-0sd = lowest_flag_domain(cpu, SD_CLUSTER); if (sd)@@ -2476,6 +2477,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att int i, ret = -ENOMEM; bool has_asym = false; bool has_cluster = false; + bool has_llc = false; bool llc_has_parent_sd = false; unsigned int multi_llcs_node = 1;@@ -2621,6 +2623,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att if (lowest_flag_domain(i, SD_CLUSTER)) has_cluster = true; + + if (highest_flag_domain(i, SD_SHARE_LLC)) + has_llc = true; } rcu_read_unlock();@@ -2631,7 +2636,8 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att static_branch_inc_cpuslocked(&sched_cluster_active);
Quoted 9 lines
kernel/sched/topology.c build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att · line 2639
Quoted 3 lines
kernel/sched/topology.c +1
-1#ifdef CONFIG_SCHED_CACHE - if (llc_has_parent_sd && multi_llcs_node && !sched_asym_cpucap_active()) + if (has_llc && llc_has_parent_sd && multi_llcs_node &&
multi_llcs_nodewill be false if there is noSD_SHARE_LLCdomain on the platform, so I suppose we don’t have to introducehas_llc?multi_llcsis set to true iff there are more than 1SD_SHARE_LLCdomains under itsSD_SHARE_LLCparent domain.
If there is *no* SD_SHARE_LLC domain, my test shows 'multi_llcs_node' is still 1 (true).
Looks it is because the default value of 'multi_llcs_node' is 1.
build_sched_domains():
unsigned int multi_llcs_node = 1;
And this condition is always false since we have no SD_SHARE_LLC domain, therefore 'multi_llcs_node' will not be changed:
if (!(sd->flags & SD_SHARE_LLC) && child &&
(child->flags & SD_SHARE_LLC))Thanks,
-adam
Re: [RFC PATCH v4 08/28] sched: Set up LLC indexing
Quoted 4 lines · Adam Li
Quoted 3 lines · Yu C
Hi Adam,
On 9/26/2025 2:14 PM, Adam Li wrote:
[snip]
Quoted 14 lines
Quoted 9 lines
Quoted 3 lines
#ifdef
CONFIG_SCHED_CACHE
- if (llc_has_parent_sd&&multi_llcs_node&& !sched_asym_cpucap_active()) + if (has_llc&&llc_has_parent_sd&&multi_llcs_node&&
multi_llcs_nodewill be false if there is noSD_SHARE_LLCdomain on the platform, so I suppose we don’t have to introducehas_llc?multi_llcsis set to true iff there are more than 1SD_SHARE_LLCdomains under itsSD_SHARE_LLCparent domain.If there is *no*
SD_SHARE_LLCdomain, my test shows 'multi_llcs_node' is still 1 (true).Looks it is because the default value of '
multi_llcs_node' is 1.
Quoted 9 lines
build_sched_domains():unsigned int multi_llcs_node = 1;And this condition is always false since we have no
SD_SHARE_LLCdomain, therefore 'multi_llcs_node' will not be changed:if (!(sd->flags & SD_SHARE_LLC) && child && (child->flags & SD_SHARE_LLC))
I see. I was looking at the v5 internal version where multi_llcs was initialized to 0. Let me double check and refine the logic. Thanks!
Chenyu
Quoted 2 lines
Thanks,
-adam
RFC PATCH v4 9/28 sched: Introduce task preferred LLC field
In reply to Chen YuFrom: Tim Chen <[email protected]>
With cache aware scheduling enabled, each process is assigned a preferred LLC id, which will be used to quickly identify the LLC domain this thread prefers to run. This is similar to numa_preferred_nid for NUMA balance.
--- include/linux/sched.h | 1 + init/init_task.c | 3 +++ kernel/sched/fair.c | 7 +++++++ 3 files changed, 11 insertions(+)
include/linux/sched.h +1 -0
@@ -1408,6 +1408,7 @@ struct task_struct {#ifdef CONFIG_SCHED_CACHE struct callback_head cache_work; +int preferred_llc;#endif #ifdef CONFIG_RSEQ
init/init_task.c +3 -0
@@ -188,6 +188,9 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = { .numa_group = NULL, .numa_faults = NULL,#endif +#ifdef CONFIG_SCHED_CACHE + .preferred_llc = -1, +#endif #if defined(CONFIG_KASAN_GENERIC ) || defined(CONFIG_KASAN_SW_TAGS ) .kasan_depth = 1,#endif
kernel/sched/fair.c +7 -0
@@ -1267,6 +1267,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)struct mm_struct *mm = p->mm;struct mm_sched *pcpu_sched;unsigned long epoch; +int mm_sched_llc =-1 ;if (!sched_feat (SCHED_CACHE ))return ;
@@ -1298,6 +1299,12 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) mm->mm_sched_cpu =-1 ; pcpu_sched->occ =0 ; } + +if (mm->mm_sched_cpu !=-1 ) + mm_sched_llc =per_cpu (sd_llc_id, mm->mm_sched_cpu); + +if (p->preferred_llc != mm_sched_llc) + p->preferred_llc = mm_sched_llc; }static void task_tick_cache (struct rq *rq,struct task_struct *p)
Signature · 1 line
2.25.1
RFC PATCH v4 10/28 sched: Calculate the number of tasks that have LLC preference on a runqueue
In reply to Chen YuFrom: Tim Chen <[email protected]>
Track for each run queue, the number of tasks that have a LLC preference and how many of those tasks are running in its preferred LLC. This is similar to nr_numa_running and nr_preferred_running for NUMA balance, and will be used by the cache-aware load balancing in subsequent patches.
--- kernel/sched/core.c | 12 +++++++++++ kernel/sched/fair.c | 51 +++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 7 ++++++ 3 files changed, 69 insertions(+), 1 deletion(-)
kernel/sched/core.c +12 -0
@@ -499,6 +499,18 @@ void __trace_set_current_state(int state_value) }EXPORT_SYMBOL (__trace_set_current_state); +#ifdef CONFIG_SMP +int task_llc (const struct task_struct *p) +{ +return per_cpu (sd_llc_id,task_cpu (p)); +} +#else +int task_llc (const struct task_struct *p) +{ +return 0 ; +} +#endif +/* * Serialization rules: *
kernel/sched/fair.c +50 -1
92 lines not on this page · raw message
kernel/sched/sched.h +7 -0
@@ -1106,6 +1106,10 @@ struct rq {unsigned int nr_preferred_running;unsigned int numa_migrate_on;#endif +#ifdef CONFIG_SCHED_CACHE +unsigned int nr_pref_llc_running; +unsigned int nr_llc_running; +#endif #ifdef CONFIG_NO_HZ_COMMON #ifdef CONFIG_SMP unsigned long last_blocked_load_update_tick;
@@ -1967,6 +1971,9 @@ init_numa_balancing(unsigned long clone_flags, struct task_struct *p)#endif /* !CONFIG_NUMA_BALANCING */ +void reset_llc_stats (struct rq *rq); +int task_llc (const struct task_struct *p); +#ifdef CONFIG_SMP static inline void
Signature · 1 line
2.25.1
RFC PATCH v4 11/28 sched: Introduce per runqueue task LLC preference counter
In reply to Chen YuFrom: Tim Chen <[email protected]>
Each runqueue is assigned a static array, where each element indicates the number of tasks preferring a particular LLC mapped to the array index.
For example, rq->nr_pref_llc[3] = 2 signifies that there are 2 tasks on this runqueue which prefer to run within LLC3 (indexed from 0 to MAX_LLC across the entire system). With this information, the load balancer can make better decisions to select the busiest runqueue and migrate tasks to their preferred LLC domains.
Note: The static array could be converted to an xarray in the future.
--- kernel/sched/fair.c | 42 ++++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 1 + 2 files changed, 41 insertions(+), 2 deletions(-)
kernel/sched/fair.c +40 -2
@@ -1195,22 +1195,51 @@ static inline int llc_idx(int cpu)return per_cpu (sd_llc_idx, cpu); } +static inline int pref_llc_idx (struct task_struct *p) +{ +return llc_idx (p->preferred_llc); +} +static void account_llc_enqueue (struct rq *rq,struct task_struct *p) { +int pref_llc; +if (!sched_feat (SCHED_CACHE ))return ; rq->nr_llc_running += (p->preferred_llc !=-1 ); rq->nr_pref_llc_running += (p->preferred_llc ==task_llc (p)); + +if (p->preferred_llc <0 ) +return ; + + pref_llc =pref_llc_idx (p); +if (pref_llc <0 ) +return ; + + ++rq->nr_pref_llc[pref_llc]; }static void account_llc_dequeue (struct rq *rq,struct task_struct *p) { +int pref_llc; +if (!sched_feat (SCHED_CACHE ))return ; rq->nr_llc_running -= (p->preferred_llc !=-1 ); rq->nr_pref_llc_running -= (p->preferred_llc ==task_llc (p)); + +if (p->preferred_llc <0 ) +return ; + + pref_llc =pref_llc_idx (p); +if (pref_llc <0 ) +return ; + +/* avoid negative counter */ +if (rq->nr_pref_llc[pref_llc] >0 ) + --rq->nr_pref_llc[pref_llc]; }void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched)
@@ -1279,6 +1308,8 @@ static unsigned long __no_profile fraction_mm_sched(struct rq *rq, struct mm_schreturn div64_u64 (NICE_0_LOAD * pcpu_sched->runtime, rq->cpu_runtime +1 ); } +static unsigned int task_running_on_cpu (int cpu,struct task_struct *p); +static inline void account_mm_sched (struct rq *rq,struct task_struct *p,s64 delta_exec) {
@@ -1321,7 +1352,9 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)if (mm->mm_sched_cpu !=-1 ) mm_sched_llc =per_cpu (sd_llc_id, mm->mm_sched_cpu); -if (p->preferred_llc != mm_sched_llc) { +/* task not on rq accounted later in account_entity_enqueue() */ +if (task_running_on_cpu (rq->cpu, p) && + p->preferred_llc != mm_sched_llc) {account_llc_dequeue (rq, p); p->preferred_llc = mm_sched_llc;account_llc_enqueue (rq, p);
@@ -1431,11 +1464,16 @@ void init_sched_mm(struct task_struct *p)void reset_llc_stats (struct rq *rq) { +int i; +if (!sched_feat (SCHED_CACHE ))return ; -if (rq->nr_llc_running) +if (rq->nr_llc_running) { +for (i =0 ; i <MAX_LLC ; ++i) + rq->nr_pref_llc[i] =0 ; rq->nr_llc_running =0 ; + } rq->nr_pref_llc_running =0 ; }
kernel/sched/sched.h +1 -0
@@ -1109,6 +1109,7 @@ struct rq {#ifdef CONFIG_SCHED_CACHE unsigned int nr_pref_llc_running;unsigned int nr_llc_running; +unsigned int nr_pref_llc[MAX_LLC ];#endif #ifdef CONFIG_NO_HZ_COMMON #ifdef CONFIG_SMP
Signature · 1 line
2.25.1
RFC PATCH v4 12/28 sched: Calculate the total number of preferred LLC tasks during load balance
In reply to Chen YuFrom: Tim Chen <[email protected]>
During load balancing between LLCs, gather the number of tasks on each runqueue of a source LLC.
For example, consider a system with 4 sched groups LLC0, LLC1, ..., LLC3. We are balancing towards LLC3 and LLC0 has 3 tasks preferring LLC3, LLC1 has 2 tasks preferring LLC3 and LLC2 has 1 task preferring LLC3. LLC0 with most tasks preferring LLC3 will be chosen as the busiest LLC to pick the tasks from.
The number of tasks preferring the destination LLC are gathered from each run queue for a source LLC.
For example, consider the sched_group LLC0 with two CPUs, CPU0 and CPU1. On CPU0, 2 tasks prefer to run on LLC3, and on CPU1, one task prefers LLC3. The total number of tasks preferring LLC3 in LLC0 is 2 + 1 = 3.
These statistics enable the load balancer to select tasks from a sched_group that best aligns tasks with their preferred LLCs.
--- kernel/sched/fair.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
kernel/sched/fair.c +11 -0
@@ -10496,6 +10496,9 @@ struct sg_lb_stats {unsigned int nr_numa_running;unsigned int nr_preferred_running;#endif +#ifdef CONFIG_SCHED_CACHE +unsigned int nr_pref_llc[MAX_LLC ]; +#endif };/*
@@ -10974,6 +10977,14 @@ static inline void update_sg_lb_stats(struct lb_env *env,if (cpu_overutilized (i)) *sg_overutilized =1 ; +#ifdef CONFIG_SCHED_CACHE +if (sched_feat (SCHED_CACHE )) { +int j; + +for (j =0 ; j < max_llcs; ++j) + sgs->nr_pref_llc[j] += rq->nr_pref_llc[j]; + } +#endif /* * No need to call idle_cpu() if nr_running is not 0 */
Signature · 1 line
2.25.1
RFC PATCH v4 13/28 sched: Tag the sched group as llc_balance if it has tasks prefer other LLC
In reply to Chen YuFrom: Tim Chen <[email protected]>
During load balancing between LLCs, check whether there are tasks preferring the destination LLC. If so, balance those tasks to the destination LLC first.
Tag the sched_group that has tasks preferring to run on other LLCs (non-local) with the group_llc_balance flag. This way, the load balancer will later attempt to pull/push these tasks to their preferred LLCs.
--- kernel/sched/fair.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
kernel/sched/fair.c +43 -0
@@ -10491,6 +10491,7 @@ struct sg_lb_stats {enum group_type group_type;unsigned int group_asym_packing;/* Tasks should be moved to preferred CPU */ unsigned int group_smt_balance;/* Task on busy SMT be moved */ +unsigned int group_llc_balance;/* Tasks should be moved to preferred LLC */ unsigned long group_misfit_task_load;/* A CPU has a task too big for its capacity */ #ifdef CONFIG_NUMA_BALANCING unsigned int nr_numa_running;
@@ -10855,6 +10856,43 @@ static inline bool smt_balance(struct lb_env *env, struct sg_lb_stats *sgs,return false; } +/* +* Do LLC balance on sched group that contains LLC, and have tasks preferring +* to run on LLC in idle dst_cpu. +*/ +#ifdef CONFIG_SCHED_CACHE +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +struct sched_domain *child = env->sd->child; +int llc; + +if (!sched_feat (SCHED_CACHE )) +return false; + +if (env->sd->flags &SD_SHARE_LLC ) +return false; + +/* only care about task migration among LLCs */ +if (child && !(child->flags &SD_SHARE_LLC )) +return false; + + llc =llc_idx (env->dst_cpu); +if (sgs->nr_pref_llc[llc] >0 && +_get_migrate_hint (env->src_cpu, env->dst_cpu, +0 , true) == mig_allow) +return true; + +return false; +} +#else +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +return false; +} +#endif + static inline long sibling_imbalance(struct lb_env *env, struct sd_lb_stats *sds, struct sg_lb_stats *busiest,
@@ -11037,6 +11075,11 @@ static inline void update_sg_lb_stats(struct lb_env *env, sgs->group_type =group_classify (env->sd->imbalance_pct, group, sgs);update_sg_if_llc (env, sgs, group); + +/* Check for tasks in this group can be moved to their preferred LLC */ +if (!local_group &&llc_balance (env, sgs, group)) + sgs->group_llc_balance =1 ; +/* Computing avg_load makes sense only when group is overloaded */ if (sgs->group_type == group_overloaded) sgs->avg_load = (sgs->group_load * SCHED_CAPACITY_SCALE) /
Signature · 1 line
2.25.1
RFC PATCH v4 14/28 sched: Introduce update_llc_busiest() to deal with groups having preferred LLC tasks
In reply to Chen YuFrom: Tim Chen <[email protected]>
The load balancer attempts to identify the busiest sched_group with the highest load and migrates some tasks to a less busy sched_group to distribute the load across different CPUs.
When cache-aware scheduling is enabled, the busiest sched_group is defined as the one with the highest number of tasks preferring to run on the destination LLC. If the busiest group has llc_balance tag, the cache aware load balance will be launched.
Introduce the helper function update_llc_busiest() to identify such sched group with most tasks preferring the destination LLC.
--- kernel/sched/fair.c | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-)
kernel/sched/fair.c +35 -1
@@ -10885,12 +10885,36 @@ static inline bool llc_balance(struct lb_env *env, struct sg_lb_stats *sgs,return false; } + +static bool update_llc_busiest (struct lb_env *env, +struct sg_lb_stats *busiest, +struct sg_lb_stats *sgs) +{ +int idx; + +/* Only the candidate with llc_balance need to be taken care of */ +if (!sgs->group_llc_balance) +return false; + +/* +* There are more tasks that want to run on dst_cpu's LLC. +*/ + idx =llc_idx (env->dst_cpu); +return sgs->nr_pref_llc[idx] > busiest->nr_pref_llc[idx]; +}#else static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs,struct sched_group *group) {return false; } + +static bool update_llc_busiest (struct lb_env *env, +struct sg_lb_stats *busiest, +struct sg_lb_stats *sgs) +{ +return false; +}#endif static inline long sibling_imbalance(struct lb_env *env,
@@ -11122,6 +11146,14 @@ static bool update_sd_pick_busiest(struct lb_env *env, sds->local_stat.group_type != group_has_spare))return false; +/* deal with prefer LLC load balance, if failed, fall into normal load balance */ +if (update_llc_busiest (env, busiest, sgs)) +return true; + +/* if there is already a busy group, skip the normal load balance */ +if (busiest->group_llc_balance) +return false; +if (sgs->group_type > busiest->group_type)return true;
@@ -12029,9 +12061,11 @@ static struct sched_group *sched_balance_find_src_group(struct lb_env *env)/* * Try to move all excess tasks to a sibling domain of the busiest * group's child domain. +* Also do so if we can move some tasks that prefer the local LLC. */ if (sds.prefer_sibling && local->group_type == group_has_spare && -sibling_imbalance (env, &sds, busiest, local) >1 ) + (busiest->group_llc_balance || +sibling_imbalance (env, &sds, busiest, local) >1 )) goto force_balance;if (busiest->group_type != group_overloaded) {
Signature · 1 line
2.25.1
RFC PATCH v4 15/28 sched: Introduce a new migration_type to track the preferred LLC load balance
In reply to Chen YuFrom: Tim Chen <[email protected]>
Introduce a new migration type named migrate_llc_task to facilitate cache-aware load balancing.
After the busiest sched_group is identified as the one that needs migration due to having most tasks preferring destination LLC, tag the migration type as the newly introduced migrate_llc_task. During load balancing, each runqueue within the busiest preferred-LLC sched_group is checked, and the runqueue with the highest number of tasks preferring to run on the destination CPU is chosen as the busiest runqueue.
--- kernel/sched/fair.c | 37 ++++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-)
kernel/sched/fair.c +36 -1
@@ -9746,7 +9746,8 @@ enum migration_type { migrate_load =0 , migrate_util, migrate_task, - migrate_misfit + migrate_misfit, + migrate_llc_task };#define LBF_ALL_PINNED 0x01
@@ -10180,6 +10181,15 @@ static int detach_tasks(struct lb_env *env) env->imbalance -= util;break ; +case migrate_llc_task: +/* +* Since can_migrate_task() succeed, when we reach here, it means that p +* can be migrated even if dst_cpu is not p's preferred_llc, because there +* are no idle cores for p to do in-llc load balance. +*/ + env->imbalance--; +break ; +case migrate_task: env->imbalance--;break ;
@@ -11817,6 +11827,15 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *sreturn ; } +#ifdef CONFIG_SCHED_CACHE +if (busiest->group_llc_balance) { +/* Move a task that prefer local LLC */ + env->migration_type = migrate_llc_task; + env->imbalance =1 ; +return ; + } +#endif +if (busiest->group_type == group_imbalanced) {/* * In the group_imb case we cannot rely on group-wide averages
@@ -12125,6 +12144,10 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,struct rq *busiest =NULL , *rq;unsigned long busiest_util =0 , busiest_load =0 , busiest_capacity =1 ;unsigned int busiest_nr =0 ; +#ifdef CONFIG_SCHED_CACHE +unsigned int busiest_pref_llc =0 ; +int dst_llc; +#endif int i;for_each_cpu_and (i,sched_group_span (group), env->cpus) {
@@ -12233,6 +12256,16 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env, }break ; +case migrate_llc_task: +#ifdef CONFIG_SCHED_CACHE + dst_llc =llc_idx (env->dst_cpu); +if (!cpus_share_cache (env->dst_cpu, rq->cpu) && + busiest_pref_llc < rq->nr_pref_llc[dst_llc]) { + busiest_pref_llc = rq->nr_pref_llc[dst_llc]; + busiest = rq; + } +#endif +break ;case migrate_task:if (busiest_nr < nr_running) { busiest_nr = nr_running;
@@ -12415,6 +12448,8 @@ static void update_lb_imbalance_stat(struct lb_env *env, struct sched_domain *sdcase migrate_misfit:__schedstat_add (sd->lb_imbalance_misfit[idle], env->imbalance);break ; +case migrate_llc_task: +break ; } }
Signature · 1 line
2.25.1
RFC PATCH v4 16/28 sched: Consider LLC locality for active balance
In reply to Chen YuFrom: Tim Chen <[email protected]>
If busiest run queue has only one task, active balance is enlisted to actually move the task. However, before moving the task, we should consider whether we are moving the task from its preferred LLC.
Don't move the single running task in a run queue to another LLC, if we are moving it from its desired LLC, or moving it will cause too much imbalance between the LLCs.
--- kernel/sched/fair.c | 51 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 48 insertions(+), 3 deletions(-)
kernel/sched/fair.c +48 -3
@@ -12332,10 +12332,43 @@ imbalanced_active_balance(struct lb_env *env)return 0 ; } +#ifdef CONFIG_SCHED_CACHE +static inline bool +break_llc_locality (struct lb_env *env) +{ +if (!sched_feat (SCHED_CACHE )) +return 0 ; + +if (cpus_share_cache (env->src_cpu, env->dst_cpu)) +return 0 ; +/* +* All tasks want to stay put. Move only if LLC is +* heavily loaded or don't pull a task from its +* preferred CPU if it is the only one running. +*/ +if (env->src_rq->nr_pref_llc_running == env->src_rq->cfs.h_nr_runnable && + (env->src_rq->nr_running <=1 || +_get_migrate_hint (env->src_cpu, env->dst_cpu, +0 , false) == mig_forbid)) +return 1 ; + +return 0 ; +} +#else +static inline bool +break_llc_locality (struct lb_env *env) +{ +return 0 ; +} +#endif +static int need_active_balance (struct lb_env *env) {struct sched_domain *sd = env->sd; +if (break_llc_locality (env)) +return 0 ; +if (asym_active_balance (env))return 1 ;
@@ -12355,7 +12388,8 @@ static int need_active_balance(struct lb_env *env)return 1 ; } -if (env->migration_type == migrate_misfit) +if (env->migration_type == migrate_misfit || + env->migration_type == migrate_llc_task)return 1 ;return 0 ;
@@ -12800,9 +12834,20 @@ static int active_load_balance_cpu_stop(void *data) goto out_unlock;/* Is there any task to move? */ -if (busiest_rq->nr_running <=1 ) - goto out_unlock; +if (busiest_rq->nr_running <=1 ) { +#ifdef CONFIG_SCHED_CACHE +int llc =llc_idx (target_cpu); +if (!sched_feat (SCHED_CACHE )) + goto out_unlock; + +if (llc <0 ) + goto out_unlock; +/* don't migrate if task does not prefer target */ + if (busiest_rq->nr_pref_llc[llc] < 1) +#endif + goto out_unlock; + }/* * This condition is "impossible", if it occurs * we need to fix it. Originally reported by
Signature · 1 line
2.25.1
RFC PATCH v4 17/28 sched: Consider LLC preference when picking tasks from busiest queue
In reply to Chen YuFrom: Tim Chen <[email protected]>
When picking tasks from busiest queue for load balance, we currently do not consider LLC preference.
Order the task in the busiest queue such that we picked the tasks in the following order:
1. tasks that prefer dst cpu's LLC
2. tasks that have no preference in LLC
3. tasks that prefer LLC other than the ones they are on
4. tasks that prefer the LLC that they are currently on
This will allow tasks better chances to wind up in its preferred LLC.
--- kernel/sched/fair.c | 66 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 65 insertions(+), 1 deletion(-)
kernel/sched/fair.c +65 -1
@@ -10093,6 +10093,68 @@ static struct task_struct *detach_one_task(struct lb_env *env)return NULL ; } +#ifdef CONFIG_SCHED_CACHE +/* +* Prepare lists to detach tasks in the following order: +* 1. tasks that prefer dst cpu's LLC +* 2. tasks that have no preference in LLC +* 3. tasks that prefer LLC other than the ones they are on +* 4. tasks that prefer the LLC that they are currently on. +*/ +static struct list_head +*order_tasks_by_llc (struct lb_env *env,struct list_head *tasks) +{ +struct task_struct *p; +LIST_HEAD (pref_old_llc); +LIST_HEAD (pref_new_llc); +LIST_HEAD (no_pref_llc); +LIST_HEAD (pref_other_llc); + +if (!sched_feat (SCHED_CACHE )) +return tasks; + +if (cpus_share_cache (env->dst_cpu, env->src_cpu)) +return tasks; + +while (!list_empty (tasks)) { + p = list_last_entry(tasks, struct task_struct, se.group_node); + +if (p->preferred_llc ==llc_id (env->dst_cpu)) { +list_move (&p->se.group_node, &pref_new_llc); +continue ; + } + +if (p->preferred_llc ==llc_id (env->src_cpu)) { +list_move (&p->se.group_node, &pref_old_llc); +continue ; + } + +if (p->preferred_llc ==-1 ) { +list_move (&p->se.group_node, &no_pref_llc); +continue ; + } + +list_move (&p->se.group_node, &pref_other_llc); + } + +/* +* We detach tasks from list tail in detach tasks. Put tasks +* to be chosen first at end of list. +*/ +list_splice (&pref_new_llc, tasks); +list_splice (&no_pref_llc, tasks); +list_splice (&pref_other_llc, tasks); +list_splice (&pref_old_llc, tasks); +return tasks; +} +#else +static inline struct list_head +*order_tasks_by_llc (struct lb_env *env,struct list_head *tasks) +{ +return tasks; +} +#endif +/* * detach_tasks() -- tries to detach up to imbalance load/util/tasks from * busiest_rq, as part of a balancing operation within domain "sd".
@@ -10101,7 +10163,7 @@ static struct task_struct *detach_one_task(struct lb_env *env)*/ static int detach_tasks (struct lb_env *env) { -struct list_head *tasks = &env->src_rq->cfs_tasks; +struct list_head *tasks;unsigned long util, load;struct task_struct *p;int detached =0 ;
@@ -10120,6 +10182,8 @@ static int detach_tasks(struct lb_env *env)if (env->imbalance <=0 )return 0 ; + tasks =order_tasks_by_llc (env, &env->src_rq->cfs_tasks); +while (!list_empty (tasks)) {/* * We don't want to steal all, otherwise we may be treated likewise,
Signature · 1 line
2.25.1
RFC PATCH v4 18/28 sched: Do not migrate task if it is moving out of its preferred LLC
In reply to Chen YuFrom: Tim Chen <[email protected]>
In the final step of task migration during load balancing, can_migrate_task() is used to determine whether a task can be moved to the destination. If the task has an LLC preference, consider this preference when moving it out of its preferred LLC. With this check in place, there is no need to retain the task's cache-hot CPU check in task_hot(); remove it accordingly.
Besides, add more checks in detach_tasks() to avoid choosing tasks that prefer their current LLC.
--- kernel/sched/fair.c | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-)
kernel/sched/fair.c +17 -11
@@ -9822,17 +9822,6 @@ static int task_hot(struct task_struct *p, struct lb_env *env)if (sysctl_sched_migration_cost ==0 )return 0 ; -#ifdef CONFIG_SCHED_CACHE -if (sched_feat (SCHED_CACHE ) && p->mm && p->mm->pcpu_sched) { -/* -* XXX things like Skylake have non-inclusive L3 and might not -* like this L3 centric view. What to do about L2 stickyness ? -*/ -return per_cpu_ptr (p->mm->pcpu_sched, env->src_cpu)->occ > -per_cpu_ptr (p->mm->pcpu_sched, env->dst_cpu)->occ; - } -#endif - delta =rq_clock_task (env->src_rq) - p->se.exec_start;return delta < (s64 )sysctl_sched_migration_cost;
@@ -10029,6 +10018,12 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)if (env->flags &LBF_ACTIVE_LB )return 1 ; +#ifdef CONFIG_SCHED_CACHE +if (sched_feat (SCHED_CACHE ) && +get_migrate_hint (env->src_cpu, env->dst_cpu, p) == mig_forbid) +return 0 ; +#endif + degrades =migrate_degrades_locality (p, env);if (!degrades) hot =task_hot (p, env);
@@ -10289,6 +10284,17 @@ static int detach_tasks(struct lb_env *env)if (env->imbalance <=0 )break ; +#ifdef CONFIG_SCHED_CACHE +/* +* Don't detach more tasks if remaining tasks want to stay: +* The tasks have already been sorted by order_tasks_by_llc(), +* they are tasks that prefer the current LLC. +*/ +if (sched_feat (SCHED_CACHE ) && p->preferred_llc !=-1 && +llc_id (env->src_cpu) == p->preferred_llc) +break ; +#endif +continue ; next:if (p->sched_task_hot)
Signature · 1 line
2.25.1
RFC PATCH v4 19/28 sched: Introduce SCHED_CACHE_LB to control cache aware load balance
In reply to Chen YuFrom: Tim Chen <[email protected]>
Introduce the SCHED_CACHE_LB sched feature to enable or disable cache aware load balance in the schduler.
--- kernel/sched/fair.c | 18 ++++++++++-------- kernel/sched/features.h | 1 + 2 files changed, 11 insertions(+), 8 deletions(-)
kernel/sched/fair.c +10 -8
@@ -10019,7 +10019,7 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)return 1 ;#ifdef CONFIG_SCHED_CACHE -if (sched_feat (SCHED_CACHE ) && +if (sched_feat (SCHED_CACHE ) &&sched_feat (SCHED_CACHE_LB ) &&get_migrate_hint (env->src_cpu, env->dst_cpu, p) == mig_forbid)return 0 ;#endif
@@ -10105,7 +10105,7 @@ static struct list_headLIST_HEAD (no_pref_llc);LIST_HEAD (pref_other_llc); -if (!sched_feat (SCHED_CACHE )) +if (!sched_feat (SCHED_CACHE ) || !sched_feat (SCHED_CACHE_LB ))return tasks;if (cpus_share_cache (env->dst_cpu, env->src_cpu))
@@ -10290,7 +10290,8 @@ static int detach_tasks(struct lb_env *env)* The tasks have already been sorted by order_tasks_by_llc(), * they are tasks that prefer the current LLC. */ -if (sched_feat (SCHED_CACHE ) && p->preferred_llc !=-1 && +if (sched_feat (SCHED_CACHE ) &&sched_feat (SCHED_CACHE_LB ) && + p->preferred_llc !=-1 &&llc_id (env->src_cpu) == p->preferred_llc)break ;#endif
@@ -10947,7 +10948,7 @@ static inline bool llc_balance(struct lb_env *env, struct sg_lb_stats *sgs,struct sched_domain *child = env->sd->child;int llc; -if (!sched_feat (SCHED_CACHE )) +if (!sched_feat (SCHED_CACHE ) || !sched_feat (SCHED_CACHE_LB ))return false;if (env->sd->flags &SD_SHARE_LLC )
@@ -11058,7 +11059,8 @@ static void update_sg_if_llc(struct lb_env *env, struct sg_lb_stats *sgs,struct sched_domain *sd = env->sd->child;struct sched_domain_shared *sd_share; -if (!sched_feat (SCHED_CACHE ) || env->idle ==CPU_NEWLY_IDLE ) +if (!sched_feat (SCHED_CACHE ) || env->idle ==CPU_NEWLY_IDLE || + !sched_feat (SCHED_CACHE_LB ))return ;/* only care the sched domain that spans 1 LLC */
@@ -11120,7 +11122,7 @@ static inline void update_sg_lb_stats(struct lb_env *env, *sg_overutilized =1 ;#ifdef CONFIG_SCHED_CACHE -if (sched_feat (SCHED_CACHE )) { +if (sched_feat (SCHED_CACHE ) &&sched_feat (SCHED_CACHE_LB )) {int j;for (j =0 ; j < max_llcs; ++j)
@@ -12406,7 +12408,7 @@ imbalanced_active_balance(struct lb_env *env)static inline bool break_llc_locality (struct lb_env *env) { -if (!sched_feat (SCHED_CACHE )) +if (!sched_feat (SCHED_CACHE ) || !sched_feat (SCHED_CACHE_LB ))return 0 ;if (cpus_share_cache (env->src_cpu, env->dst_cpu))
@@ -12908,7 +12910,7 @@ static int active_load_balance_cpu_stop(void *data)#ifdef CONFIG_SCHED_CACHE int llc =llc_idx (target_cpu); -if (!sched_feat (SCHED_CACHE )) +if (!sched_feat (SCHED_CACHE ) || !sched_feat (SCHED_CACHE_LB )) goto out_unlock;if (llc <0 )
kernel/sched/features.h +1 -0
@@ -88,6 +88,7 @@ SCHED_FEAT(TTWU_QUEUE, true) SCHED_FEAT(SIS_UTIL, true) SCHED_FEAT(SCHED_CACHE, true) +SCHED_FEAT (SCHED_CACHE_LB , true)/* * Issue a WARN when we do multiple update_rq_clock() calls * in a single rq->lock section. Default disabled because the
Signature · 1 line
2.25.1
RFC PATCH v4 20/28 sched: Introduce SCHED_CACHE_WAKE to control LLC aggregation on wake up
In reply to Chen YuFrom: Tim Chen <[email protected]>
Introduce SCHED_CACHE_WAKE feature to enable or disable cache-aware wake up. Disable this feature by default because cache-aware wakeup is overly aggressive in stacking wakees of the same process on the same LLC, if they are frequently woken up.
The wake ups can be much more frequent than load balances, adding much overhead when load balance alone for LLC aggregation is sufficient.
--- kernel/sched/fair.c | 6 +++++- kernel/sched/features.h | 1 + 2 files changed, 6 insertions(+), 1 deletion(-)
kernel/sched/fair.c +5 -1
@@ -9063,7 +9063,7 @@ static int select_cache_cpu(struct task_struct *p, int prev_cpu)struct mm_struct *mm = p->mm;int cpu; -if (!sched_feat (SCHED_CACHE )) +if (!sched_feat (SCHED_CACHE ) || !sched_feat (SCHED_CACHE_WAKE ))return prev_cpu;if (!mm || p->nr_cpus_allowed ==1 )
@@ -9076,6 +9076,10 @@ static int select_cache_cpu(struct task_struct *p, int prev_cpu)if (cpus_share_cache (cpu, prev_cpu))return prev_cpu; +if (_get_migrate_hint (prev_cpu, cpu, +task_util (p), true) == mig_forbid) +return prev_cpu; +if (static_branch_likely (&sched_numa_balancing) &&__migrate_degrades_locality (p, prev_cpu, cpu, false) >0 ) {/*
kernel/sched/features.h +1 -0
@@ -89,6 +89,7 @@ SCHED_FEAT(SIS_UTIL, true) SCHED_FEAT(SCHED_CACHE, true) SCHED_FEAT(SCHED_CACHE_LB, true) +SCHED_FEAT (SCHED_CACHE_WAKE , false)/* * Issue a WARN when we do multiple update_rq_clock() calls * in a single rq->lock section. Default disabled because the
Signature · 1 line
2.25.1
RFC PATCH v4 21/28 sched: Introduce a static key to enable cache aware only for multi LLCs
In reply to Chen YuIf there are more than one LLCs in the node, the cache aware scheduling is enabled. Otherwise, the cache aware scheduling is disabled.
The definition of multiple LLCs in a node is that every node in the system should have more than one LLC. For example, if node0, node1, and node2 each have 4 LLCs, while node3 has 1 LLC (possibly due to CPU hotplug), cache-aware scheduling should be disabled.
--- kernel/sched/fair.c | 31 ++++++++++++++++++++++--------- kernel/sched/sched.h | 1 + kernel/sched/topology.c | 22 ++++++++++++++++++++-- 3 files changed, 43 insertions(+), 11 deletions(-)
kernel/sched/fair.c +22 -9
@@ -1175,6 +1175,8 @@ static s64 update_curr_se(struct rq *rq, struct sched_entity *curr)#define EPOCH_PERIOD (HZ/100)/* 10 ms */ #define EPOCH_OLD 5/* 50 ms */ +DEFINE_STATIC_KEY_FALSE (sched_cache_present); +static int llc_id (int cpu) {if (cpu <0 )
@@ -1318,7 +1320,8 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)unsigned long epoch;int mm_sched_llc =-1 ; -if (!sched_feat (SCHED_CACHE )) +if (!sched_feat (SCHED_CACHE ) || + !static_branch_likely (&sched_cache_present))return ;if (p->sched_class != &fair_sched_class)
@@ -1366,7 +1369,8 @@ static void task_tick_cache(struct rq *rq, struct task_struct *p)struct callback_head *work = &p->cache_work;struct mm_struct *mm = p->mm; -if (!sched_feat (SCHED_CACHE )) +if (!sched_feat (SCHED_CACHE ) || + !static_branch_likely (&sched_cache_present))return ;if (!mm || !mm->pcpu_sched)
@@ -9063,7 +9067,8 @@ static int select_cache_cpu(struct task_struct *p, int prev_cpu)struct mm_struct *mm = p->mm;int cpu; -if (!sched_feat (SCHED_CACHE ) || !sched_feat (SCHED_CACHE_WAKE )) +if (!sched_feat (SCHED_CACHE ) || !sched_feat (SCHED_CACHE_WAKE ) || + !static_branch_likely (&sched_cache_present))return prev_cpu;if (!mm || p->nr_cpus_allowed ==1 )
@@ -10024,6 +10029,7 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)#ifdef CONFIG_SCHED_CACHE if (sched_feat (SCHED_CACHE ) &&sched_feat (SCHED_CACHE_LB ) && +static_branch_likely (&sched_cache_present) &&get_migrate_hint (env->src_cpu, env->dst_cpu, p) == mig_forbid)return 0 ;#endif
@@ -10109,7 +10115,8 @@ static struct list_headLIST_HEAD (no_pref_llc);LIST_HEAD (pref_other_llc); -if (!sched_feat (SCHED_CACHE ) || !sched_feat (SCHED_CACHE_LB )) +if (!sched_feat (SCHED_CACHE ) || !sched_feat (SCHED_CACHE_LB ) || + !static_branch_likely (&sched_cache_present))return tasks;if (cpus_share_cache (env->dst_cpu, env->src_cpu))
@@ -10295,6 +10302,7 @@ static int detach_tasks(struct lb_env *env)* they are tasks that prefer the current LLC. */ if (sched_feat (SCHED_CACHE ) &&sched_feat (SCHED_CACHE_LB ) && +static_branch_likely (&sched_cache_present) && p->preferred_llc !=-1 &&llc_id (env->src_cpu) == p->preferred_llc)break ;
@@ -10952,7 +10960,8 @@ static inline bool llc_balance(struct lb_env *env, struct sg_lb_stats *sgs,struct sched_domain *child = env->sd->child;int llc; -if (!sched_feat (SCHED_CACHE ) || !sched_feat (SCHED_CACHE_LB )) +if (!sched_feat (SCHED_CACHE ) || !sched_feat (SCHED_CACHE_LB ) || + !static_branch_likely (&sched_cache_present))return false;if (env->sd->flags &SD_SHARE_LLC )
@@ -11064,7 +11073,8 @@ static void update_sg_if_llc(struct lb_env *env, struct sg_lb_stats *sgs,struct sched_domain_shared *sd_share;if (!sched_feat (SCHED_CACHE ) || env->idle ==CPU_NEWLY_IDLE || - !sched_feat (SCHED_CACHE_LB )) + !sched_feat (SCHED_CACHE_LB ) || + !static_branch_likely (&sched_cache_present))return ;/* only care the sched domain that spans 1 LLC */
@@ -11126,7 +11136,8 @@ static inline void update_sg_lb_stats(struct lb_env *env, *sg_overutilized =1 ;#ifdef CONFIG_SCHED_CACHE -if (sched_feat (SCHED_CACHE ) &&sched_feat (SCHED_CACHE_LB )) { +if (sched_feat (SCHED_CACHE ) &&sched_feat (SCHED_CACHE_LB ) && +static_branch_likely (&sched_cache_present)) {int j;for (j =0 ; j < max_llcs; ++j)
@@ -12412,7 +12423,8 @@ imbalanced_active_balance(struct lb_env *env)static inline bool break_llc_locality (struct lb_env *env) { -if (!sched_feat (SCHED_CACHE ) || !sched_feat (SCHED_CACHE_LB )) +if (!sched_feat (SCHED_CACHE ) || !sched_feat (SCHED_CACHE_LB ) || + !static_branch_likely (&sched_cache_present))return 0 ;if (cpus_share_cache (env->src_cpu, env->dst_cpu))
@@ -12914,7 +12926,8 @@ static int active_load_balance_cpu_stop(void *data)#ifdef CONFIG_SCHED_CACHE int llc =llc_idx (target_cpu); -if (!sched_feat (SCHED_CACHE ) || !sched_feat (SCHED_CACHE_LB )) +if (!sched_feat (SCHED_CACHE ) || !sched_feat (SCHED_CACHE_LB ) || + !static_branch_likely (&sched_cache_present)) goto out_unlock;if (llc <0 )
kernel/sched/sched.h +1 -0
@@ -2857,6 +2857,7 @@ extern unsigned int sysctl_numa_balancing_hot_threshold;#ifdef CONFIG_SCHED_CACHE extern unsigned int sysctl_llc_aggr_cap;extern unsigned int sysctl_llc_aggr_imb; +extern struct static_key_false sched_cache_present;#endif #ifdef CONFIG_SCHED_HRTICK
kernel/sched/topology.c +20 -2
@@ -2476,6 +2476,8 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attint i, ret = -ENOMEM ;bool has_asym = false;bool has_cluster = false; +bool llc_has_parent_sd = false; +unsigned int multi_llcs_node =1 ;#ifdef CONFIG_SCHED_CACHE if (max_llcs <0 ) {
@@ -2545,6 +2547,8 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attstruct sched_domain __rcu *top_p;unsigned int nr_llcs; +if (!llc_has_parent_sd) + llc_has_parent_sd = true;/* * For a single LLC per node, allow an * imbalance up to 12.5% of the node. This is
@@ -2566,10 +2570,19 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att* between LLCs and memory channels. */ nr_llcs = sd->span_weight / child->span_weight; -if (nr_llcs ==1 ) +/* +* iff all nodes have multiple LLCs, the +* multi_llcs_node will be set to 1. If +* there is at least 1 node having 1 single +* LLC, the multi_llcs_node remains 0. +*/ +if (nr_llcs ==1 ) { imb = sd->span_weight >>3 ; -else + multi_llcs_node =0 ; + }else { imb = nr_llcs; + multi_llcs_node &=1 ; + } imb =max (1U , imb); sd->imb_numa_nr = imb;
@@ -2617,6 +2630,11 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attif (has_cluster)static_branch_inc_cpuslocked (&sched_cluster_active); +#ifdef CONFIG_SCHED_CACHE +if (llc_has_parent_sd && multi_llcs_node && !sched_asym_cpucap_active ()) +static_branch_inc_cpuslocked (&sched_cache_present); +#endif +if (rq && sched_debug_verbose)pr_info ("root domain span: %*pbl\n" ,cpumask_pr_args (cpu_map));
Signature · 1 line
2.25.1
RFC PATCH v4 22/28 sched: Turn EPOCH_PERIOD and EPOCH_OLD into tunnable debugfs
In reply to Chen YuConvert EPOCH_PERIOD and EPOCH_OLD into tunable debugfs entries. Users can adjust the decay rate as needed. By default, occupancy decays by half every 10 ms.
--- kernel/sched/debug.c | 2 ++ kernel/sched/fair.c | 9 ++++++--- kernel/sched/sched.h | 2 ++ 3 files changed, 10 insertions(+), 3 deletions(-)
kernel/sched/debug.c +2 -0
@@ -535,6 +535,8 @@ static __init int sched_init_debug(void)#ifdef CONFIG_SCHED_CACHE debugfs_create_u32 ("llc_aggr_cap" ,0644 , debugfs_sched, &sysctl_llc_aggr_cap);debugfs_create_u32 ("llc_aggr_imb" ,0644 , debugfs_sched, &sysctl_llc_aggr_imb); +debugfs_create_u32 ("llc_period" ,0644 , debugfs_sched, &sysctl_llc_period); +debugfs_create_u32 ("llc_old" ,0644 , debugfs_sched, &sysctl_llc_old);#endif debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);
kernel/sched/fair.c +6 -3
@@ -1175,6 +1175,9 @@ static s64 update_curr_se(struct rq *rq, struct sched_entity *curr)#define EPOCH_PERIOD (HZ/100)/* 10 ms */ #define EPOCH_OLD 5/* 50 ms */ +__read_mostly unsigned int sysctl_llc_period =EPOCH_PERIOD ; +__read_mostly unsigned int sysctl_llc_old =EPOCH_OLD ; +DEFINE_STATIC_KEY_FALSE (sched_cache_present);static int llc_id (int cpu)
@@ -1283,9 +1286,9 @@ static inline void __update_mm_sched(struct rq *rq, struct mm_sched *pcpu_sched)long delta = now - rq->cpu_epoch_next;if (delta >0 ) { - n = (delta +EPOCH_PERIOD -1 ) /EPOCH_PERIOD ; + n = (delta + sysctl_llc_period -1 ) / sysctl_llc_period; rq->cpu_epoch += n; - rq->cpu_epoch_next += n *EPOCH_PERIOD ; + rq->cpu_epoch_next += n * sysctl_llc_period;__shr_u64 (&rq->cpu_runtime, n); }
@@ -1346,7 +1349,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)* has only 1 thread, invalidate * it's preferred state. */ -if (epoch -READ_ONCE (mm->mm_sched_epoch) >EPOCH_OLD || +if (epoch -READ_ONCE (mm->mm_sched_epoch) > sysctl_llc_old ||get_nr_threads (p) <=1 ) { mm->mm_sched_cpu =-1 ; pcpu_sched->occ =0 ;
kernel/sched/sched.h +2 -0
@@ -2858,6 +2858,8 @@ extern unsigned int sysctl_numa_balancing_hot_threshold;extern unsigned int sysctl_llc_aggr_cap;extern unsigned int sysctl_llc_aggr_imb;extern struct static_key_false sched_cache_present; +extern unsigned int sysctl_llc_period; +extern unsigned int sysctl_llc_old;#endif #ifdef CONFIG_SCHED_HRTICK
Signature · 1 line
2.25.1
RFC PATCH v4 23/28 sched: Scan a task's preferred node for preferred LLC
In reply to Chen YuWhen sched_cache is enabled, fully scanning all online CPUs to find the hottest one is very costly. As a first step, limit the scan to only the CPUs within the task's preferred node. If the node containing the task's preferred LLC is not in the CPU scan mask, add it. Additionally, if the node where the current task is running is not in the scan mask, add it too.
--- kernel/sched/fair.c | 36 +++++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-)
kernel/sched/fair.c +33 -3
@@ -1390,13 +1390,36 @@ static void task_tick_cache(struct rq *rq, struct task_struct *p) } } +static void get_scan_cpumasks (cpumask_var_t cpus,int cache_cpu, +int pref_nid,int curr_cpu) +{ +#ifdef CONFIG_NUMA_BALANCING +/* first honor the task's preferred node */ +if (pref_nid !=NUMA_NO_NODE ) +cpumask_or (cpus, cpus,cpumask_of_node (pref_nid)); +#endif + +/* secondly honor the task's cache CPU if it is not included */ +if (cache_cpu !=-1 && !cpumask_test_cpu (cache_cpu, cpus)) +cpumask_or (cpus, cpus, +cpumask_of_node (cpu_to_node (cache_cpu))); + +/* +* Thirdly honor the task's current running node +* as the last resort. +*/ +if (!cpumask_test_cpu (curr_cpu, cpus)) +cpumask_or (cpus, cpus,cpumask_of_node (cpu_to_node (curr_cpu))); +} + static void __no_profile task_cache_work(struct callback_head *work) {struct task_struct *p = current;struct mm_struct *mm = p->mm;unsigned long m_a_occ =0 ;unsigned long last_m_a_occ =0 ; -int cpu, m_a_cpu =-1 ; +int cpu, m_a_cpu =-1 , cache_cpu, + pref_nid =NUMA_NO_NODE , curr_cpu =smp_processor_id ();cpumask_var_t cpus;WARN_ON_ONCE (work != &p->cache_work);
@@ -1406,11 +1429,18 @@ static void __no_profile task_cache_work(struct callback_head *work)if (p->flags &PF_EXITING )return ; -if (!alloc_cpumask_var (&cpus,GFP_KERNEL )) +if (!zalloc_cpumask_var (&cpus,GFP_KERNEL ))return ; + cache_cpu = mm->mm_sched_cpu; +#ifdef CONFIG_NUMA_BALANCING +if (static_branch_likely (&sched_numa_balancing)) + pref_nid = p->numa_preferred_nid; +#endif +scoped_guard (cpus_read_lock) { -cpumask_copy (cpus, cpu_online_mask); +get_scan_cpumasks (cpus, cache_cpu, + pref_nid, curr_cpu);for_each_cpu (cpu, cpus) {/* XXX sched_cluster_active */
Signature · 1 line
2.25.1
kernel test robot
build botRe: [RFC PATCH v4 23/28] sched: Scan a task's preferred node for preferred LLC
kernel test robot noticed "BUG:using_smp_processor_id()in_preemptible" on:
Hello,
kernel test robot noticed "BUG:using_smp_processor_id()in_preemptible" on:
commit: ff797757d658ecda432e4c33c0cfc2b704af2b06 ("[RFC PATCH v4 23/28] sched: Scan a task's preferred node for preferred LLC")
url: https://github.com/intel-lab-lkp/linux/commits/Chen-Yu/sched-Cache-aware-load-balancing/20250809-131716
base: v6.16
patch link: https://lore.kernel.org/all/178bf43d7cbc9b2c9aea408dd56b87391067df37.1754712565.git.tim.c.chen@linux.intel.com/
patch subject: [RFC PATCH v4 23/28] sched: Scan a task's preferred node for preferred LLC
in testcase: boot
config: x86_64-randconfig-103-20250810
compiler: gcc-12
test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
(please refer to attached dmesg/kmsg for entire log/backtrace)
+------------------------------------------------------+------------+------------+ | | 552a2db758 | ff797757d6 | +------------------------------------------------------+------------+------------+ | BUG:using_smp_processor_id()in_preemptible | 0 | 11 | +------------------------------------------------------+------------+------------+
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]
[ 19.177186][ T64] BUG: using smp_processor_id() in preemptible [00000000] code: modprobe/64 [ 19.179622][ T64] caller is debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.181347][ T64] CPU: 0 UID: 0 PID: 64 Comm: modprobe Not tainted 6.16.0-00023-gff797757d658 #1 PREEMPT(none) [ 19.181372][ T64] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 19.181385][ T64] Call Trace: [ 19.181391][ T64] <TASK> [ 19.181401][ T64] dump_stack_lvl (lib/dump_stack.c:123) [ 19.181440][ T64] dump_stack (lib/dump_stack.c:130) [ 19.181465][ T64] check_preemption_disabled (arch/x86/include/asm/preempt.h:85 lib/smp_processor_id.c:53) [ 19.181494][ T64] debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.181514][ T64] task_cache_work (kernel/sched/fair.c:1422) [ 19.181531][ T64] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.181569][ T64] ? lock_is_held (include/linux/lockdep.h:249) [ 19.181594][ T64] ? _raw_spin_unlock_irq (arch/x86/include/asm/paravirt.h:671 include/linux/spinlock_api_smp.h:159 kernel/locking/spinlock.c:202) [ 19.181626][ T64] ? __pfx_task_cache_work (kernel/sched/fair.c:1416) [ 19.181642][ T64] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.181663][ T64] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4475) [ 19.181697][ T64] task_work_run (kernel/task_work.c:228 (discriminator 1)) [ 19.181724][ T64] ? __pfx_task_work_run (kernel/task_work.c:195) [ 19.181747][ T64] ? preempt_latency_start (include/linux/ftrace.h:1091 kernel/sched/core.c:5866) [ 19.181782][ T64] resume_user_mode_work (arch/x86/include/asm/current.h:25 include/linux/resume_user_mode.h:53) [ 19.181817][ T64] irqentry_exit_to_user_mode (kernel/entry/common.c:124 include/linux/entry-common.h:330 kernel/entry/common.c:184) [ 19.181843][ T64] irqentry_exit (kernel/entry/common.c:287) [ 19.181862][ T64] exc_page_fault (arch/x86/mm/fault.c:1536) [ 19.181898][ T64] asm_exc_page_fault (arch/x86/include/asm/idtentry.h:623) [ 19.181916][ T64] RIP: 0023:0xf7ee9067 [ 19.181933][ T64] Code: fe ff ff 8b 00 85 c0 0f 84 96 00 00 00 8b 8d 60 fe ff ff 8b 51 30 85 d2 74 03 01 42 04 8b 8d 60 fe ff ff 8b 51 2c 85 d2 74 03 <01> 42 04 8b bd 60 fe ff ff 8b 57 34 85 d2 74 03 01 42 04 8b 8d 60 All code ======== 0: fe (bad) 1: ff (bad) 2: ff 8b 00 85 c0 0f decl 0xfc08500(%rbx) 8: 84 96 00 00 00 8b test %dl,-0x75000000(%rsi) e: 8d 60 fe lea -0x2(%rax),%esp 11: ff (bad) 12: ff 8b 51 30 85 d2 decl -0x2d7acfaf(%rbx) 18: 74 03 je 0x1d 1a: 01 42 04 add %eax,0x4(%rdx) 1d: 8b 8d 60 fe ff ff mov -0x1a0(%rbp),%ecx 23: 8b 51 2c mov 0x2c(%rcx),%edx 26: 85 d2 test %edx,%edx 28: 74 03 je 0x2d 2a:* 01 42 04 add %eax,0x4(%rdx) <-- trapping instruction 2d: 8b bd 60 fe ff ff mov -0x1a0(%rbp),%edi 33: 8b 57 34 mov 0x34(%rdi),%edx 36: 85 d2 test %edx,%edx 38: 74 03 je 0x3d 3a: 01 42 04 add %eax,0x4(%rdx) 3d: 8b .byte 0x8b 3e: 8d .byte 0x8d 3f: 60 (bad)
Code starting with the faulting instruction
===========================================
0: 01 42 04 add %eax,0x4(%rdx) 3: 8b bd 60 fe ff ff mov -0x1a0(%rbp),%edi 9: 8b 57 34 mov 0x34(%rdi),%edx c: 85 d2 test %edx,%edx e: 74 03 je 0x13 10: 01 42 04 add %eax,0x4(%rdx) 13: 8b .byte 0x8b 14: 8d .byte 0x8d 15: 60 (bad) [ 19.181949][ T64] RSP: 002b:00000000fffd59c0 EFLAGS: 00010206 [ 19.181966][ T64] RAX: 00000000565ad000 RBX: 0000000000000001 RCX: 00000000f7f11980 [ 19.181978][ T64] RDX: 00000000565dbd44 RSI: 00000000effffef5 RDI: 000000006fffffff [ 19.181990][ T64] RBP: 00000000fffd5bb8 R08: 0000000000000000 R09: 0000000000000000 [ 19.182001][ T64] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 19.182013][ T64] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 19.182063][ T64] </TASK> [ 19.230078][ T64] BUG: using smp_processor_id() in preemptible [00000000] code: modprobe/64 [ 19.230948][ T64] caller is debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.231549][ T64] CPU: 0 UID: 0 PID: 64 Comm: modprobe Not tainted 6.16.0-00023-gff797757d658 #1 PREEMPT(none) [ 19.231557][ T64] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 19.231561][ T64] Call Trace: [ 19.231564][ T64] <TASK> [ 19.231567][ T64] dump_stack_lvl (lib/dump_stack.c:123) [ 19.231580][ T64] dump_stack (lib/dump_stack.c:130) [ 19.231588][ T64] check_preemption_disabled (arch/x86/include/asm/preempt.h:85 lib/smp_processor_id.c:53) [ 19.231598][ T64] debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.231604][ T64] task_cache_work (kernel/sched/fair.c:1422) [ 19.231615][ T64] ? find_held_lock (kernel/locking/lockdep.c:5353) [ 19.231627][ T64] ? _raw_spin_unlock_irq (arch/x86/include/asm/paravirt.h:671 include/linux/spinlock_api_smp.h:159 kernel/locking/spinlock.c:202) [ 19.231637][ T64] ? __pfx_task_cache_work (kernel/sched/fair.c:1416) [ 19.231643][ T64] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.231649][ T64] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4475) [ 19.231659][ T64] task_work_run (kernel/task_work.c:228 (discriminator 1)) [ 19.231668][ T64] ? __pfx_task_work_run (kernel/task_work.c:195) [ 19.231680][ T64] resume_user_mode_work (arch/x86/include/asm/current.h:25 include/linux/resume_user_mode.h:53) [ 19.231691][ T64] irqentry_exit_to_user_mode (kernel/entry/common.c:124 include/linux/entry-common.h:330 kernel/entry/common.c:184) [ 19.231699][ T64] irqentry_exit (kernel/entry/common.c:287) [ 19.231705][ T64] exc_page_fault (arch/x86/mm/fault.c:1536) [ 19.231717][ T64] asm_exc_page_fault (arch/x86/include/asm/idtentry.h:623) [ 19.231723][ T64] RIP: 0023:0xf7f03797 [ 19.231729][ T64] Code: 24 0c e3 2c 89 d7 83 e2 03 74 11 7a 04 aa 49 74 1f aa 49 74 1b 83 f2 01 75 02 aa 49 89 ca c1 e9 02 83 e2 03 69 c0 01 01 01 01 <f3> ab 89 d1 f3 aa 8b 44 24 08 5f c3 66 90 66 90 66 90 66 90 66 90 All code ======== 0: 24 0c and $0xc,%al 2: e3 2c jrcxz 0x30 4: 89 d7 mov %edx,%edi 6: 83 e2 03 and $0x3,%edx 9: 74 11 je 0x1c b: 7a 04 jp 0x11 d: aa stos %al,%es:(%rdi) e: 49 74 1f rex.WB je 0x30 11: aa stos %al,%es:(%rdi) 12: 49 74 1b rex.WB je 0x30 15: 83 f2 01 xor $0x1,%edx 18: 75 02 jne 0x1c 1a: aa stos %al,%es:(%rdi) 1b: 49 89 ca mov %rcx,%r10 1e: c1 e9 02 shr $0x2,%ecx 21: 83 e2 03 and $0x3,%edx 24: 69 c0 01 01 01 01 imul $0x1010101,%eax,%eax 2a:* f3 ab rep stos %eax,%es:(%rdi) <-- trapping instruction 2c: 89 d1 mov %edx,%ecx 2e: f3 aa rep stos %al,%es:(%rdi) 30: 8b 44 24 08 mov 0x8(%rsp),%eax 34: 5f pop %rdi 35: c3 ret 36: 66 90 xchg %ax,%ax 38: 66 90 xchg %ax,%ax 3a: 66 90 xchg %ax,%ax 3c: 66 90 xchg %ax,%ax 3e: 66 90 xchg %ax,%ax
Code starting with the faulting instruction
===========================================
0: f3 ab rep stos %eax,%es:(%rdi) 2: 89 d1 mov %edx,%ecx 4: f3 aa rep stos %al,%es:(%rdi) 6: 8b 44 24 08 mov 0x8(%rsp),%eax a: 5f pop %rdi b: c3 ret c: 66 90 xchg %ax,%ax e: 66 90 xchg %ax,%ax 10: 66 90 xchg %ax,%ax 12: 66 90 xchg %ax,%ax 14: 66 90 xchg %ax,%ax [ 19.231735][ T64] RSP: 002b:00000000fffd4f58 EFLAGS: 00010206 [ 19.231741][ T64] RAX: 0000000000000000 RBX: 00000000f7f11000 RCX: 0000000000000001 [ 19.231745][ T64] RDX: 0000000000000000 RSI: 00000000f7edf148 RDI: 00000000f7edf144 [ 19.231749][ T64] RBP: 00000000fffd5138 R08: 0000000000000000 R09: 0000000000000000 [ 19.231753][ T64] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 19.231756][ T64] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 19.231771][ T64] </TASK> [ 19.251371][ T64] BUG: using smp_processor_id() in preemptible [00000000] code: modprobe/64 [ 19.252249][ T64] caller is debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.252877][ T64] CPU: 0 UID: 0 PID: 64 Comm: modprobe Not tainted 6.16.0-00023-gff797757d658 #1 PREEMPT(none) [ 19.252886][ T64] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 19.252890][ T64] Call Trace: [ 19.252892][ T64] <TASK> [ 19.252895][ T64] dump_stack_lvl (lib/dump_stack.c:123) [ 19.252907][ T64] dump_stack (lib/dump_stack.c:130) [ 19.252916][ T64] check_preemption_disabled (arch/x86/include/asm/preempt.h:85 lib/smp_processor_id.c:53) [ 19.252925][ T64] debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.252931][ T64] task_cache_work (kernel/sched/fair.c:1422) [ 19.252941][ T64] ? find_held_lock (kernel/locking/lockdep.c:5353) [ 19.252953][ T64] ? _raw_spin_unlock_irq (arch/x86/include/asm/paravirt.h:671 include/linux/spinlock_api_smp.h:159 kernel/locking/spinlock.c:202) [ 19.252963][ T64] ? __pfx_task_cache_work (kernel/sched/fair.c:1416) [ 19.252969][ T64] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.252976][ T64] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4475) [ 19.252985][ T64] task_work_run (kernel/task_work.c:228 (discriminator 1)) [ 19.252994][ T64] ? __pfx_task_work_run (kernel/task_work.c:195) [ 19.253006][ T64] resume_user_mode_work (arch/x86/include/asm/current.h:25 include/linux/resume_user_mode.h:53) [ 19.253018][ T64] exit_to_user_mode_loop (kernel/entry/common.c:124) [ 19.253025][ T64] do_int80_emulation (include/linux/entry-common.h:332 include/linux/entry-common.h:414 include/linux/entry-common.h:449 arch/x86/entry/syscall_32.c:175) [ 19.253035][ T64] ? find_held_lock (kernel/locking/lockdep.c:5353) [ 19.253044][ T64] ? preempt_latency_start (include/linux/ftrace.h:1091 kernel/sched/core.c:5866) [ 19.253055][ T64] ? __up_write (arch/x86/include/asm/preempt.h:104 kernel/locking/rwsem.c:1375) [ 19.253061][ T64] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.253070][ T64] ? validate_chain (kernel/locking/lockdep.c:3826 kernel/locking/lockdep.c:3879) [ 19.253081][ T64] ? __lock_acquire (kernel/locking/lockdep.c:5240) [ 19.253093][ T64] ? find_held_lock (kernel/locking/lockdep.c:5353) [ 19.253106][ T64] ? validate_chain (kernel/locking/lockdep.c:3826 kernel/locking/lockdep.c:3879) [ 19.253117][ T64] ? __lock_acquire (kernel/locking/lockdep.c:5240) [ 19.253128][ T64] ? find_held_lock (kernel/locking/lockdep.c:5353) [ 19.253138][ T64] ? vtime_user_exit (kernel/sched/cputime.c:737) [ 19.253149][ T64] ? vtime_user_exit (kernel/sched/cputime.c:737) [ 19.253159][ T64] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.253170][ T64] ? debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.253177][ T64] ? rcu_is_watching (include/linux/context_tracking.h:128 kernel/rcu/tree.c:745) [ 19.253185][ T64] ? trace_irq_disable (include/trace/events/preemptirq.h:36 (discriminator 21)) [ 19.253195][ T64] ? trace_hardirqs_off_finish (kernel/trace/trace_preemptirq.c:98) [ 19.253203][ T64] ? do_int80_emulation (arch/x86/include/asm/jump_label.h:36 arch/x86/entry/syscall_32.c:148) [ 19.253212][ T64] asm_int80_emulation (arch/x86/include/asm/idtentry.h:626) [ 19.253218][ T64] RIP: 0023:0xf7f02cb0 [ 19.253223][ T64] Code: 4c 24 34 89 44 24 0c 8b 44 24 44 8b 54 24 38 8b 74 24 3c 8b 7c 24 40 a9 ff 0f 00 00 75 1c c1 e8 0c 89 c5 b8 c0 00 00 00 cd 80 <3d> 00 f0 ff ff 77 21 83 c4 1c 5b 5e 5f 5d c3 90 83 c4 1c b8 ea ff All code ======== 0: 4c 24 34 rex.WR and $0x34,%al 3: 89 44 24 0c mov %eax,0xc(%rsp) 7: 8b 44 24 44 mov 0x44(%rsp),%eax b: 8b 54 24 38 mov 0x38(%rsp),%edx f: 8b 74 24 3c mov 0x3c(%rsp),%esi 13: 8b 7c 24 40 mov 0x40(%rsp),%edi 17: a9 ff 0f 00 00 test $0xfff,%eax 1c: 75 1c jne 0x3a 1e: c1 e8 0c shr $0xc,%eax 21: 89 c5 mov %eax,%ebp 23: b8 c0 00 00 00 mov $0xc0,%eax 28: cd 80 int $0x80 2a:* 3d 00 f0 ff ff cmp $0xfffff000,%eax <-- trapping instruction 2f: 77 21 ja 0x52 31: 83 c4 1c add $0x1c,%esp 34: 5b pop %rbx 35: 5e pop %rsi 36: 5f pop %rdi 37: 5d pop %rbp 38: c3 ret 39: 90 nop 3a: 83 c4 1c add $0x1c,%esp 3d: b8 .byte 0xb8 3e: ea (bad) 3f: ff .byte 0xff
Code starting with the faulting instruction
===========================================
0: 3d 00 f0 ff ff cmp $0xfffff000,%eax 5: 77 21 ja 0x28 7: 83 c4 1c add $0x1c,%esp a: 5b pop %rbx b: 5e pop %rsi c: 5f pop %rdi d: 5d pop %rbp e: c3 ret f: 90 nop 10: 83 c4 1c add $0x1c,%esp 13: b8 .byte 0xb8 14: ea (bad) 15: ff .byte 0xff [ 19.253229][ T64] RSP: 002b:00000000fffd4f00 EFLAGS: 00000202 ORIG_RAX: 00000000000000c0 [ 19.253236][ T64] RAX: 00000000f7db9000 RBX: 00000000f7db9000 RCX: 00000000000da000 [ 19.253240][ T64] RDX: 0000000000000001 RSI: 0000000000000812 RDI: 0000000000000000 [ 19.253244][ T64] RBP: 00000000000001ce R08: 0000000000000000 R09: 0000000000000000 [ 19.253248][ T64] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 19.253251][ T64] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 19.253266][ T64] </TASK> [ 19.256786][ T64] BUG: using smp_processor_id() in preemptible [00000000] code: modprobe/64 [ 19.304780][ T64] caller is debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.305760][ T64] CPU: 1 UID: 0 PID: 64 Comm: modprobe Not tainted 6.16.0-00023-gff797757d658 #1 PREEMPT(none) [ 19.305774][ T64] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 19.305782][ T64] Call Trace: [ 19.305787][ T64] <TASK> [ 19.305795][ T64] dump_stack_lvl (lib/dump_stack.c:123) [ 19.305817][ T64] dump_stack (lib/dump_stack.c:130) [ 19.305832][ T64] check_preemption_disabled (arch/x86/include/asm/preempt.h:85 lib/smp_processor_id.c:53) [ 19.305849][ T64] debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.305862][ T64] task_cache_work (kernel/sched/fair.c:1422) [ 19.305881][ T64] ? find_held_lock (kernel/locking/lockdep.c:5353) [ 19.305903][ T64] ? _raw_spin_unlock_irq (arch/x86/include/asm/paravirt.h:671 include/linux/spinlock_api_smp.h:159 kernel/locking/spinlock.c:202) [ 19.305921][ T64] ? __pfx_task_cache_work (kernel/sched/fair.c:1416) [ 19.305931][ T64] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.305944][ T64] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4475) [ 19.305962][ T64] task_work_run (kernel/task_work.c:228 (discriminator 1)) [ 19.305980][ T64] ? __pfx_task_work_run (kernel/task_work.c:195) [ 19.306002][ T64] resume_user_mode_work (arch/x86/include/asm/current.h:25 include/linux/resume_user_mode.h:53) [ 19.306023][ T64] irqentry_exit_to_user_mode (kernel/entry/common.c:124 include/linux/entry-common.h:330 kernel/entry/common.c:184) [ 19.306039][ T64] irqentry_exit (kernel/entry/common.c:287) [ 19.306050][ T64] exc_page_fault (arch/x86/mm/fault.c:1536) [ 19.306072][ T64] asm_exc_page_fault (arch/x86/include/asm/idtentry.h:623) [ 19.306083][ T64] RIP: 0023:0xf7db80cb [ 19.306094][ T64] Code: 01 00 00 00 83 c4 08 5b c3 66 90 66 90 66 90 55 53 56 57 31 d2 9c 58 89 c1 35 00 00 20 00 50 9d 9c 58 31 c1 31 c0 8b 74 24 14 <89> 46 08 0f ba e1 15 0f 83 6b 01 00 00 0f a2 89 c7 31 c0 81 fb 47 All code ======== 0: 01 00 add %eax,(%rax) 2: 00 00 add %al,(%rax) 4: 83 c4 08 add $0x8,%esp 7: 5b pop %rbx 8: c3 ret 9: 66 90 xchg %ax,%ax b: 66 90 xchg %ax,%ax d: 66 90 xchg %ax,%ax f: 55 push %rbp 10: 53 push %rbx 11: 56 push %rsi 12: 57 push %rdi 13: 31 d2 xor %edx,%edx 15: 9c pushf 16: 58 pop %rax 17: 89 c1 mov %eax,%ecx 19: 35 00 00 20 00 xor $0x200000,%eax 1e: 50 push %rax 1f: 9d popf 20: 9c pushf 21: 58 pop %rax 22: 31 c1 xor %eax,%ecx 24: 31 c0 xor %eax,%eax 26: 8b 74 24 14 mov 0x14(%rsp),%esi 2a:* 89 46 08 mov %eax,0x8(%rsi) <-- trapping instruction 2d: 0f ba e1 15 bt $0x15,%ecx 31: 0f 83 6b 01 00 00 jae 0x1a2 37: 0f a2 cpuid 39: 89 c7 mov %eax,%edi 3b: 31 c0 xor %eax,%eax 3d: 81 .byte 0x81 3e: fb sti 3f: 47 rex.RXB
Code starting with the faulting instruction
===========================================
0: 89 46 08 mov %eax,0x8(%rsi) 3: 0f ba e1 15 bt $0x15,%ecx 7: 0f 83 6b 01 00 00 jae 0x178 d: 0f a2 cpuid f: 89 c7 mov %eax,%edi 11: 31 c0 xor %eax,%eax 13: 81 .byte 0x81 14: fb sti 15: 47 rex.RXB [ 19.306104][ T64] RSP: 002b:00000000fffd5bd8 EFLAGS: 00210246 [ 19.306115][ T64] RAX: 0000000000000000 RBX: 00000000f7eac684 RCX: 0000000000200000 [ 19.306123][ T64] RDX: 0000000000000000 RSI: 00000000f7eb3ad0 RDI: 00000000f7eb3ad0 [ 19.306131][ T64] RBP: 00000000f7eac684 R08: 0000000000000000 R09: 0000000000000000 [ 19.306138][ T64] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 19.306145][ T64] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 19.306175][ T64] </TASK> [ 19.328981][ T64] BUG: using smp_processor_id() in preemptible [00000000] code: modprobe/64 [ 19.329865][ T64] caller is debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.330477][ T64] CPU: 1 UID: 0 PID: 64 Comm: modprobe Not tainted 6.16.0-00023-gff797757d658 #1 PREEMPT(none) [ 19.330486][ T64] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 19.330490][ T64] Call Trace: [ 19.330492][ T64] <TASK> [ 19.330495][ T64] dump_stack_lvl (lib/dump_stack.c:123) [ 19.330508][ T64] dump_stack (lib/dump_stack.c:130) [ 19.330516][ T64] check_preemption_disabled (arch/x86/include/asm/preempt.h:85 lib/smp_processor_id.c:53) [ 19.330525][ T64] debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.330531][ T64] task_cache_work (kernel/sched/fair.c:1422) [ 19.330541][ T64] ? find_held_lock (kernel/locking/lockdep.c:5353) [ 19.330553][ T64] ? _raw_spin_unlock_irq (arch/x86/include/asm/paravirt.h:671 include/linux/spinlock_api_smp.h:159 kernel/locking/spinlock.c:202) [ 19.330564][ T64] ? __pfx_task_cache_work (kernel/sched/fair.c:1416) [ 19.330569][ T64] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.330576][ T64] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4475) [ 19.330586][ T64] task_work_run (kernel/task_work.c:228 (discriminator 1)) [ 19.330595][ T64] ? __pfx_task_work_run (kernel/task_work.c:195) [ 19.330606][ T64] resume_user_mode_work (arch/x86/include/asm/current.h:25 include/linux/resume_user_mode.h:53) [ 19.330618][ T64] irqentry_exit_to_user_mode (kernel/entry/common.c:124 include/linux/entry-common.h:330 kernel/entry/common.c:184) [ 19.330626][ T64] irqentry_exit (kernel/entry/common.c:287) [ 19.330632][ T64] exc_page_fault (arch/x86/mm/fault.c:1536) [ 19.330644][ T64] asm_exc_page_fault (arch/x86/include/asm/idtentry.h:623) [ 19.330650][ T64] RIP: 0023:0xf7a9f4c0 [ 19.330656][ T64] Code: 57 8d 1c 06 e8 21 0d 00 00 83 c4 0c c6 04 03 00 50 57 53 89 eb e8 70 15 00 00 83 c4 1c 89 f0 5b 5e 5f 5d c3 66 90 66 90 66 90 <57> 8b 4c 24 08 66 0f 6e 4c 24 0c 66 0f ef d2 89 cf 66 0f 60 c9 66 All code ======== 0: 57 push %rdi 1: 8d 1c 06 lea (%rsi,%rax,1),%ebx 4: e8 21 0d 00 00 call 0xd2a 9: 83 c4 0c add $0xc,%esp c: c6 04 03 00 movb $0x0,(%rbx,%rax,1) 10: 50 push %rax 11: 57 push %rdi 12: 53 push %rbx 13: 89 eb mov %ebp,%ebx 15: e8 70 15 00 00 call 0x158a 1a: 83 c4 1c add $0x1c,%esp 1d: 89 f0 mov %esi,%eax 1f: 5b pop %rbx 20: 5e pop %rsi 21: 5f pop %rdi 22: 5d pop %rbp 23: c3 ret 24: 66 90 xchg %ax,%ax 26: 66 90 xchg %ax,%ax 28: 66 90 xchg %ax,%ax 2a:* 57 push %rdi <-- trapping instruction 2b: 8b 4c 24 08 mov 0x8(%rsp),%ecx 2f: 66 0f 6e 4c 24 0c movd 0xc(%rsp),%xmm1 35: 66 0f ef d2 pxor %xmm2,%xmm2 39: 89 cf mov %ecx,%edi 3b: 66 0f 60 c9 punpcklbw %xmm1,%xmm1 3f: 66 data16
Code starting with the faulting instruction
===========================================
0: 57 push %rdi 1: 8b 4c 24 08 mov 0x8(%rsp),%ecx 5: 66 0f 6e 4c 24 0c movd 0xc(%rsp),%xmm1 b: 66 0f ef d2 pxor %xmm2,%xmm2 f: 89 cf mov %ecx,%edi 11: 66 0f 60 c9 punpcklbw %xmm1,%xmm1 15: 66 data16 [ 19.330662][ T64] RSP: 002b:00000000fffd38cc EFLAGS: 00210246 [ 19.330667][ T64] RAX: 0000000000000000 RBX: 00000000565dbdd4 RCX: 00000000fffd49a8 [ 19.330671][ T64] RDX: 0000000000000000 RSI: 00000000fffd394c RDI: 000000005799e1f0 [ 19.330675][ T64] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 19.330679][ T64] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 19.330682][ T64] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 19.330697][ T64] </TASK> [ 19.332542][ T64] modprobe (64) used greatest stack depth: 25120 bytes left [ 19.344814][ T65] BUG: using smp_processor_id() in preemptible [00000000] code: modprobe/65 [ 19.360113][ T65] caller is debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.361051][ T65] CPU: 0 UID: 0 PID: 65 Comm: modprobe Not tainted 6.16.0-00023-gff797757d658 #1 PREEMPT(none) [ 19.361065][ T65] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 19.361071][ T65] Call Trace: [ 19.361074][ T65] <TASK> [ 19.361079][ T65] dump_stack_lvl (lib/dump_stack.c:123) [ 19.361099][ T65] dump_stack (lib/dump_stack.c:130) [ 19.361111][ T65] check_preemption_disabled (arch/x86/include/asm/preempt.h:85 lib/smp_processor_id.c:53) [ 19.361125][ T65] debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.361136][ T65] task_cache_work (kernel/sched/fair.c:1422) [ 19.361144][ T65] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.361162][ T65] ? lock_is_held (include/linux/lockdep.h:249) [ 19.361175][ T65] ? _raw_spin_unlock_irq (arch/x86/include/asm/paravirt.h:671 include/linux/spinlock_api_smp.h:159 kernel/locking/spinlock.c:202) [ 19.361191][ T65] ? __pfx_task_cache_work (kernel/sched/fair.c:1416) [ 19.361199][ T65] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.361210][ T65] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4475) [ 19.361225][ T65] task_work_run (kernel/task_work.c:228 (discriminator 1)) [ 19.361238][ T65] ? __pfx_task_work_run (kernel/task_work.c:195) [ 19.361256][ T65] resume_user_mode_work (arch/x86/include/asm/current.h:25 include/linux/resume_user_mode.h:53) [ 19.361273][ T65] irqentry_exit_to_user_mode (kernel/entry/common.c:124 include/linux/entry-common.h:330 kernel/entry/common.c:184) [ 19.361286][ T65] irqentry_exit (kernel/entry/common.c:287) [ 19.361295][ T65] exc_page_fault (arch/x86/mm/fault.c:1536) [ 19.361313][ T65] asm_exc_page_fault (arch/x86/include/asm/idtentry.h:623) [ 19.361322][ T65] RIP: 0023:0xf7f7b148 [ 19.361330][ T65] Code: bb 41 ff ff 6f 29 d3 89 04 99 e9 f1 fe ff ff 8b b5 74 ff ff ff 85 f6 74 60 8b 47 30 85 c0 74 03 01 70 04 8b 47 2c 85 c0 74 03 <01> 70 04 8b 47 34 85 c0 74 03 01 70 04 8b 47 38 85 c0 74 03 01 70 All code ======== 0: bb 41 ff ff 6f mov $0x6fffff41,%ebx 5: 29 d3 sub %edx,%ebx 7: 89 04 99 mov %eax,(%rcx,%rbx,4) a: e9 f1 fe ff ff jmp 0xffffffffffffff00 f: 8b b5 74 ff ff ff mov -0x8c(%rbp),%esi 15: 85 f6 test %esi,%esi 17: 74 60 je 0x79 19: 8b 47 30 mov 0x30(%rdi),%eax 1c: 85 c0 test %eax,%eax 1e: 74 03 je 0x23 20: 01 70 04 add %esi,0x4(%rax) 23: 8b 47 2c mov 0x2c(%rdi),%eax 26: 85 c0 test %eax,%eax 28: 74 03 je 0x2d 2a:* 01 70 04 add %esi,0x4(%rax) <-- trapping instruction 2d: 8b 47 34 mov 0x34(%rdi),%eax 30: 85 c0 test %eax,%eax 32: 74 03 je 0x37 34: 01 70 04 add %esi,0x4(%rax) 37: 8b 47 38 mov 0x38(%rdi),%eax 3a: 85 c0 test %eax,%eax 3c: 74 03 je 0x41 3e: 01 .byte 0x1 3f: 70 .byte 0x70
Code starting with the faulting instruction
===========================================
0: 01 70 04 add %esi,0x4(%rax) 3: 8b 47 34 mov 0x34(%rdi),%eax 6: 85 c0 test %eax,%eax 8: 74 03 je 0xd a: 01 70 04 add %esi,0x4(%rax) d: 8b 47 38 mov 0x38(%rdi),%eax 10: 85 c0 test %eax,%eax 12: 74 03 je 0x17 14: 01 .byte 0x1 15: 70 .byte 0x70 [ 19.361339][ T65] RSP: 002b:00000000ffdeddc0 EFLAGS: 00010286 [ 19.361348][ T65] RAX: 00000000f7f395e4 RBX: 0000000000000028 RCX: 00000000f7f6d310 [ 19.361355][ T65] RDX: 0000000000000000 RSI: 00000000f7c78000 RDI: 00000000f7f6d2f0 [ 19.361361][ T65] RBP: 00000000ffdedea8 R08: 0000000000000000 R09: 0000000000000000 [ 19.361367][ T65] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 19.361372][ T65] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 19.361395][ T65] </TASK> [ 19.405251][ T65] BUG: using smp_processor_id() in preemptible [00000000] code: modprobe/65 [ 19.406850][ T65] caller is debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.407947][ T65] CPU: 0 UID: 0 PID: 65 Comm: modprobe Not tainted 6.16.0-00023-gff797757d658 #1 PREEMPT(none) [ 19.407964][ T65] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 19.407972][ T65] Call Trace: [ 19.407977][ T65] <TASK> [ 19.407984][ T65] dump_stack_lvl (lib/dump_stack.c:123) [ 19.408010][ T65] dump_stack (lib/dump_stack.c:130) [ 19.408027][ T65] check_preemption_disabled (arch/x86/include/asm/preempt.h:85 lib/smp_processor_id.c:53) [ 19.408048][ T65] debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.408062][ T65] task_cache_work (kernel/sched/fair.c:1422) [ 19.408083][ T65] ? find_held_lock (kernel/locking/lockdep.c:5353) [ 19.408108][ T65] ? _raw_spin_unlock_irq (arch/x86/include/asm/paravirt.h:671 include/linux/spinlock_api_smp.h:159 kernel/locking/spinlock.c:202) [ 19.408130][ T65] ? __pfx_task_cache_work (kernel/sched/fair.c:1416) [ 19.408141][ T65] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.408155][ T65] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4475) [ 19.408182][ T65] task_work_run (kernel/task_work.c:228 (discriminator 1)) [ 19.408198][ T65] ? __pfx_task_work_run (kernel/task_work.c:195) [ 19.408211][ T65] resume_user_mode_work (arch/x86/include/asm/current.h:25 include/linux/resume_user_mode.h:53) [ 19.408225][ T65] irqentry_exit_to_user_mode (kernel/entry/common.c:124 include/linux/entry-common.h:330 kernel/entry/common.c:184) [ 19.408235][ T65] irqentry_exit (kernel/entry/common.c:287) [ 19.408242][ T65] exc_page_fault (arch/x86/mm/fault.c:1536) [ 19.408256][ T65] asm_exc_page_fault (arch/x86/include/asm/idtentry.h:623) [ 19.408264][ T65] RIP: 0023:0xf7b9cca0 [ 19.408271][ T65] Code: 04 00 00 00 c7 44 24 2c 03 00 00 00 e9 3a fe ff ff e8 a4 1e 00 00 66 90 66 90 c3 66 90 66 90 66 90 66 90 66 90 66 90 66 90 90 <e8> 14 01 03 00 05 5b 73 0d 00 53 8b 88 f4 fe ff ff 8d 90 e0 38 f6 All code ======== 0: 04 00 add $0x0,%al 2: 00 00 add %al,(%rax) 4: c7 44 24 2c 03 00 00 movl $0x3,0x2c(%rsp) b: 00 c: e9 3a fe ff ff jmp 0xfffffffffffffe4b 11: e8 a4 1e 00 00 call 0x1eba 16: 66 90 xchg %ax,%ax 18: 66 90 xchg %ax,%ax 1a: c3 ret 1b: 66 90 xchg %ax,%ax 1d: 66 90 xchg %ax,%ax 1f: 66 90 xchg %ax,%ax 21: 66 90 xchg %ax,%ax 23: 66 90 xchg %ax,%ax 25: 66 90 xchg %ax,%ax 27: 66 90 xchg %ax,%ax 29: 90 nop 2a:* e8 14 01 03 00 call 0x30143 <-- trapping instruction 2f: 05 5b 73 0d 00 add $0xd735b,%eax 34: 53 push %rbx 35: 8b 88 f4 fe ff ff mov -0x10c(%rax),%ecx 3b: 8d .byte 0x8d 3c: 90 nop 3d: e0 38 loopne 0x77 3f: f6 .byte 0xf6
Code starting with the faulting instruction
===========================================
0: e8 14 01 03 00 call 0x30119 5: 05 5b 73 0d 00 add $0xd735b,%eax a: 53 push %rbx b: 8b 88 f4 fe ff ff mov -0x10c(%rax),%ecx 11: 8d .byte 0x8d 12: 90 nop 13: e0 38 loopne 0x4d 15: f6 .byte 0xf6 [ 19.408277][ T65] RSP: 002b:00000000ffdee69c EFLAGS: 00010202 [ 19.408285][ T65] RAX: 0000000000000000 RBX: 0000000000000007 RCX: 00000000f7f6d5d0 [ 19.408289][ T65] RDX: 0000000000000015 RSI: 00000000f7c81904 RDI: 00000000f7b9cca0 [ 19.408294][ T65] RBP: 00000000ffdee738 R08: 0000000000000000 R09: 0000000000000000 [ 19.408299][ T65] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 19.408303][ T65] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 19.408320][ T65] </TASK> [ 19.430721][ T65] BUG: using smp_processor_id() in preemptible [00000000] code: modprobe/65 [ 19.431596][ T65] caller is debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.432219][ T65] CPU: 0 UID: 0 PID: 65 Comm: modprobe Not tainted 6.16.0-00023-gff797757d658 #1 PREEMPT(none) [ 19.432227][ T65] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 19.432232][ T65] Call Trace: [ 19.432234][ T65] <TASK> [ 19.432237][ T65] dump_stack_lvl (lib/dump_stack.c:123) [ 19.432250][ T65] dump_stack (lib/dump_stack.c:130) [ 19.432258][ T65] check_preemption_disabled (arch/x86/include/asm/preempt.h:85 lib/smp_processor_id.c:53) [ 19.432267][ T65] debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.432274][ T65] task_cache_work (kernel/sched/fair.c:1422) [ 19.432283][ T65] ? find_held_lock (kernel/locking/lockdep.c:5353) [ 19.432295][ T65] ? _raw_spin_unlock_irq (arch/x86/include/asm/paravirt.h:671 include/linux/spinlock_api_smp.h:159 kernel/locking/spinlock.c:202) [ 19.432306][ T65] ? __pfx_task_cache_work (kernel/sched/fair.c:1416) [ 19.432311][ T65] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.432318][ T65] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4475) [ 19.432327][ T65] task_work_run (kernel/task_work.c:228 (discriminator 1)) [ 19.432336][ T65] ? __pfx_task_work_run (kernel/task_work.c:195) [ 19.432348][ T65] resume_user_mode_work (arch/x86/include/asm/current.h:25 include/linux/resume_user_mode.h:53) [ 19.432359][ T65] exit_to_user_mode_loop (kernel/entry/common.c:124) [ 19.432367][ T65] do_int80_emulation (include/linux/entry-common.h:332 include/linux/entry-common.h:414 include/linux/entry-common.h:449 arch/x86/entry/syscall_32.c:175) [ 19.432375][ T65] ? rcu_read_unlock (include/linux/rcupdate.h:874 (discriminator 9)) [ 19.432381][ T65] ? do_read_fault (mm/memory.c:5600) [ 19.432394][ T65] ? do_pte_missing (mm/memory.c:4252) [ 19.432401][ T65] ? handle_pte_fault (mm/memory.c:6069) [ 19.432412][ T65] ? handle_pte_fault (mm/memory.c:6026) [ 19.432421][ T65] ? __pfx_handle_pte_fault (mm/memory.c:6026) [ 19.432430][ T65] ? validate_chain (kernel/locking/lockdep.c:3826 kernel/locking/lockdep.c:3879) [ 19.432439][ T65] ? validate_chain (kernel/locking/lockdep.c:3826 kernel/locking/lockdep.c:3879) [ 19.432449][ T65] ? __lock_acquire (kernel/locking/lockdep.c:5240) [ 19.432461][ T65] ? find_held_lock (kernel/locking/lockdep.c:5353) [ 19.432471][ T65] ? vtime_user_exit (kernel/sched/cputime.c:737) [ 19.432482][ T65] ? vtime_user_exit (kernel/sched/cputime.c:737) [ 19.432492][ T65] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.432504][ T65] ? debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.432510][ T65] ? rcu_is_watching (include/linux/context_tracking.h:128 kernel/rcu/tree.c:745) [ 19.432519][ T65] ? trace_irq_disable (include/trace/events/preemptirq.h:36 (discriminator 21)) [ 19.432529][ T65] ? trace_hardirqs_off_finish (kernel/trace/trace_preemptirq.c:98) [ 19.432537][ T65] ? do_int80_emulation (arch/x86/include/asm/jump_label.h:36 arch/x86/entry/syscall_32.c:148) [ 19.432546][ T65] asm_int80_emulation (arch/x86/include/asm/idtentry.h:626) [ 19.432552][ T65] RIP: 0023:0xf7f8fd24 [ 19.432558][ T65] Code: b8 5b 00 00 00 cd 80 89 d3 3d 01 f0 ff ff 0f 83 b2 ef ff ff c3 90 53 8b 54 24 10 8b 4c 24 0c 8b 5c 24 08 b8 7d 00 00 00 cd 80 <5b> 3d 01 f0 ff ff 0f 83 90 ef ff ff c3 66 90 66 90 66 90 66 90 66 All code ======== 0: b8 5b 00 00 00 mov $0x5b,%eax 5: cd 80 int $0x80 7: 89 d3 mov %edx,%ebx 9: 3d 01 f0 ff ff cmp $0xfffff001,%eax e: 0f 83 b2 ef ff ff jae 0xffffffffffffefc6 14: c3 ret 15: 90 nop 16: 53 push %rbx 17: 8b 54 24 10 mov 0x10(%rsp),%edx 1b: 8b 4c 24 0c mov 0xc(%rsp),%ecx 1f: 8b 5c 24 08 mov 0x8(%rsp),%ebx 23: b8 7d 00 00 00 mov $0x7d,%eax 28: cd 80 int $0x80 2a:* 5b pop %rbx <-- trapping instruction 2b: 3d 01 f0 ff ff cmp $0xfffff001,%eax 30: 0f 83 90 ef ff ff jae 0xffffffffffffefc6 36: c3 ret 37: 66 90 xchg %ax,%ax 39: 66 90 xchg %ax,%ax 3b: 66 90 xchg %ax,%ax 3d: 66 90 xchg %ax,%ax 3f: 66 data16
Code starting with the faulting instruction
===========================================
0: 5b pop %rbx 1: 3d 01 f0 ff ff cmp $0xfffff001,%eax 6: 0f 83 90 ef ff ff jae 0xffffffffffffef9c c: c3 ret d: 66 90 xchg %ax,%ax f: 66 90 xchg %ax,%ax 11: 66 90 xchg %ax,%ax 13: 66 90 xchg %ax,%ax 15: 66 data16 [ 19.432564][ T65] RSP: 002b:00000000ffdee668 EFLAGS: 00000206 ORIG_RAX: 000000000000007d [ 19.432571][ T65] RAX: 0000000000000000 RBX: 00000000f7f21000 RCX: 000000000001b000 [ 19.432575][ T65] RDX: 0000000000000001 RSI: 00000000f7f9e000 RDI: 00000000f7f6d2f0 [ 19.432579][ T65] RBP: 00000000ffdee738 R08: 0000000000000000 R09: 0000000000000000 [ 19.432583][ T65] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 19.432586][ T65] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 19.432601][ T65] </TASK> [ 19.477057][ T66] BUG: using smp_processor_id() in preemptible [00000000] code: modprobe/66 [ 19.481747][ T66] caller is debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.484904][ T66] CPU: 1 UID: 0 PID: 66 Comm: modprobe Not tainted 6.16.0-00023-gff797757d658 #1 PREEMPT(none) [ 19.484945][ T66] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 19.484963][ T66] Call Trace: [ 19.484975][ T66] <TASK> [ 19.484991][ T66] dump_stack_lvl (lib/dump_stack.c:123) [ 19.485052][ T66] dump_stack (lib/dump_stack.c:130) [ 19.485092][ T66] check_preemption_disabled (arch/x86/include/asm/preempt.h:85 lib/smp_processor_id.c:53) [ 19.485139][ T66] debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.485176][ T66] task_cache_work (kernel/sched/fair.c:1422) [ 19.485202][ T66] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.485264][ T66] ? lock_is_held (include/linux/lockdep.h:249) [ 19.485307][ T66] ? _raw_spin_unlock_irq (arch/x86/include/asm/paravirt.h:671 include/linux/spinlock_api_smp.h:159 kernel/locking/spinlock.c:202) [ 19.485357][ T66] ? __pfx_task_cache_work (kernel/sched/fair.c:1416) [ 19.485384][ T66] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.485418][ T66] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4475) [ 19.485468][ T66] task_work_run (kernel/task_work.c:228 (discriminator 1)) [ 19.485513][ T66] ? __pfx_task_work_run (kernel/task_work.c:195) [ 19.485550][ T66] ? preempt_latency_start (include/linux/ftrace.h:1091 kernel/sched/core.c:5866) [ 19.485607][ T66] resume_user_mode_work (arch/x86/include/asm/current.h:25 include/linux/resume_user_mode.h:53) [ 19.485660][ T66] exit_to_user_mode_loop (kernel/entry/common.c:124) [ 19.485700][ T66] do_int80_emulation (include/linux/entry-common.h:332 include/linux/entry-common.h:414 include/linux/entry-common.h:449 arch/x86/entry/syscall_32.c:175) [ 19.485758][ T66] ? __lock_acquire (kernel/locking/lockdep.c:5240) [ 19.485821][ T66] ? find_held_lock (kernel/locking/lockdep.c:5353) [ 19.485871][ T66] ? vtime_user_exit (kernel/sched/cputime.c:737) [ 19.485924][ T66] ? vtime_user_exit (kernel/sched/cputime.c:737) [ 19.485976][ T66] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.486035][ T66] ? debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.486070][ T66] ? rcu_is_watching (include/linux/context_tracking.h:128 kernel/rcu/tree.c:745) [ 19.486111][ T66] ? trace_irq_disable (include/trace/events/preemptirq.h:36 (discriminator 21)) [ 19.486159][ T66] ? trace_hardirqs_off_finish (kernel/trace/trace_preemptirq.c:98) [ 19.486198][ T66] ? do_int80_emulation (arch/x86/include/asm/jump_label.h:36 arch/x86/entry/syscall_32.c:148) [ 19.486244][ T66] asm_int80_emulation (arch/x86/include/asm/idtentry.h:626) [ 19.486274][ T66] RIP: 0023:0xf7f82cb0 [ 19.486300][ T66] Code: 4c 24 34 89 44 24 0c 8b 44 24 44 8b 54 24 38 8b 74 24 3c 8b 7c 24 40 a9 ff 0f 00 00 75 1c c1 e8 0c 89 c5 b8 c0 00 00 00 cd 80 <3d> 00 f0 ff ff 77 21 83 c4 1c 5b 5e 5f 5d c3 90 83 c4 1c b8 ea ff All code ======== 0: 4c 24 34 rex.WR and $0x34,%al 3: 89 44 24 0c mov %eax,0xc(%rsp) 7: 8b 44 24 44 mov 0x44(%rsp),%eax b: 8b 54 24 38 mov 0x38(%rsp),%edx f: 8b 74 24 3c mov 0x3c(%rsp),%esi 13: 8b 7c 24 40 mov 0x40(%rsp),%edi 17: a9 ff 0f 00 00 test $0xfff,%eax 1c: 75 1c jne 0x3a 1e: c1 e8 0c shr $0xc,%eax 21: 89 c5 mov %eax,%ebp 23: b8 c0 00 00 00 mov $0xc0,%eax 28: cd 80 int $0x80 2a:* 3d 00 f0 ff ff cmp $0xfffff000,%eax <-- trapping instruction 2f: 77 21 ja 0x52 31: 83 c4 1c add $0x1c,%esp 34: 5b pop %rbx 35: 5e pop %rsi 36: 5f pop %rdi 37: 5d pop %rbp 38: c3 ret 39: 90 nop 3a: 83 c4 1c add $0x1c,%esp 3d: b8 .byte 0xb8 3e: ea (bad) 3f: ff .byte 0xff
Code starting with the faulting instruction
===========================================
0: 3d 00 f0 ff ff cmp $0xfffff000,%eax 5: 77 21 ja 0x28 7: 83 c4 1c add $0x1c,%esp a: 5b pop %rbx b: 5e pop %rsi c: 5f pop %rdi d: 5d pop %rbp e: c3 ret f: 90 nop 10: 83 c4 1c add $0x1c,%esp 13: b8 .byte 0xb8 14: ea (bad) 15: ff .byte 0xff [ 19.486327][ T66] RSP: 002b:00000000ff907830 EFLAGS: 00000246 ORIG_RAX: 00000000000000c0 [ 19.486359][ T66] RAX: 00000000f7f34000 RBX: 0000000000000000 RCX: 000000000002b148 [ 19.486380][ T66] RDX: 0000000000000001 RSI: 0000000000000802 RDI: 0000000000000000 [ 19.486399][ T66] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 19.486418][ T66] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 19.486437][ T66] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 19.486519][ T66] </TASK> [ 19.544246][ T66] BUG: using smp_processor_id() in preemptible [00000000] code: modprobe/66 [ 19.545273][ T66] caller is debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.545971][ T66] CPU: 1 UID: 0 PID: 66 Comm: modprobe Not tainted 6.16.0-00023-gff797757d658 #1 PREEMPT(none) [ 19.545981][ T66] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 19.545985][ T66] Call Trace: [ 19.545988][ T66] <TASK> [ 19.545992][ T66] dump_stack_lvl (lib/dump_stack.c:123) [ 19.546007][ T66] dump_stack (lib/dump_stack.c:130) [ 19.546016][ T66] check_preemption_disabled (arch/x86/include/asm/preempt.h:85 lib/smp_processor_id.c:53) [ 19.546027][ T66] debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.546035][ T66] task_cache_work (kernel/sched/fair.c:1422) [ 19.546046][ T66] ? find_held_lock (kernel/locking/lockdep.c:5353) [ 19.546060][ T66] ? _raw_spin_unlock_irq (arch/x86/include/asm/paravirt.h:671 include/linux/spinlock_api_smp.h:159 kernel/locking/spinlock.c:202) [ 19.546072][ T66] ? __pfx_task_cache_work (kernel/sched/fair.c:1416) [ 19.546078][ T66] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) [ 19.546086][ T66] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4475) [ 19.546098][ T66] task_work_run (kernel/task_work.c:228 (discriminator 1)) [ 19.546108][ T66] ? __pfx_task_work_run (kernel/task_work.c:195) [ 19.546122][ T66] resume_user_mode_work (arch/x86/include/asm/current.h:25 include/linux/resume_user_mode.h:53) [ 19.546135][ T66] exit_to_user_mode_loop (kernel/entry/common.c:124) [ 19.546144][ T66] do_int80_emulation (include/linux/entry-common.h:332 include/linux/entry-common.h:414 include/linux/entry-common.h:449 arch/x86/entry/syscall_32.c:175) [ 19.546155][ T66] ? debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.546163][ T66] ? rcu_is_watching (include/linux/context_tracking.h:128 kernel/rcu/tree.c:745) [ 19.546173][ T66] ? trace_irq_disable (include/trace/events/preemptirq.h:36 (discriminator 21)) [ 19.546184][ T66] ? trace_hardirqs_off_finish (kernel/trace/trace_preemptirq.c:98) [ 19.546193][ T66] ? do_int80_emulation (arch/x86/include/asm/jump_label.h:36 arch/x86/entry/syscall_32.c:148) [ 19.546203][ T66] asm_int80_emulation (arch/x86/include/asm/idtentry.h:626) [ 19.546211][ T66] RIP: 0023:0xf7f82cb0 [ 19.546218][ T66] Code: 4c 24 34 89 44 24 0c 8b 44 24 44 8b 54 24 38 8b 74 24 3c 8b 7c 24 40 a9 ff 0f 00 00 75 1c c1 e8 0c 89 c5 b8 c0 00 00 00 cd 80 <3d> 00 f0 ff ff 77 21 83 c4 1c 5b 5e 5f 5d c3 90 83 c4 1c b8 ea ff All code ======== 0: 4c 24 34 rex.WR and $0x34,%al 3: 89 44 24 0c mov %eax,0xc(%rsp) 7: 8b 44 24 44 mov 0x44(%rsp),%eax b: 8b 54 24 38 mov 0x38(%rsp),%edx f: 8b 74 24 3c mov 0x3c(%rsp),%esi 13: 8b 7c 24 40 mov 0x40(%rsp),%edi 17: a9 ff 0f 00 00 test $0xfff,%eax 1c: 75 1c jne 0x3a 1e: c1 e8 0c shr $0xc,%eax 21: 89 c5 mov %eax,%ebp 23: b8 c0 00 00 00 mov $0xc0,%eax 28: cd 80 int $0x80 2a:* 3d 00 f0 ff ff cmp $0xfffff000,%eax <-- trapping instruction 2f: 77 21 ja 0x52 31: 83 c4 1c add $0x1c,%esp 34: 5b pop %rbx 35: 5e pop %rsi 36: 5f pop %rdi 37: 5d pop %rbp 38: c3 ret 39: 90 nop 3a: 83 c4 1c add $0x1c,%esp 3d: b8 .byte 0xb8 3e: ea (bad) 3f: ff .byte 0xff
Code starting with the faulting instruction
===========================================
0: 3d 00 f0 ff ff cmp $0xfffff000,%eax 5: 77 21 ja 0x28 7: 83 c4 1c add $0x1c,%esp a: 5b pop %rbx b: 5e pop %rsi c: 5f pop %rdi d: 5d pop %rbp e: c3 ret f: 90 nop 10: 83 c4 1c add $0x1c,%esp 13: b8 .byte 0xb8 14: ea (bad) 15: ff .byte 0xff [ 19.546225][ T66] RSP: 002b:00000000ff907810 EFLAGS: 00000246 ORIG_RAX: 00000000000000c0 [ 19.546233][ T66] RAX: 00000000f7c6b000 RBX: 0000000000000000 RCX: 00000000002c8ae0 [ 19.546238][ T66] RDX: 0000000000000001 RSI: 0000000000000802 RDI: 0000000000000000 [ 19.546243][ T66] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 19.546247][ T66] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 19.546252][ T66] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 19.546269][ T66] </TASK>
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20250812/[email protected]
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [RFC PATCH v4 23/28] sched: Scan a task's preferred node for preferred LLC
Quoted 13 lines · kernel test robot
[ 19.177186][ T64] BUG: using smp_processor_id() in preemptible [00000000] code: modprobe/64 [ 19.179622][ T64] caller is debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.181347][ T64] CPU: 0 UID: 0 PID: 64 Comm: modprobe Not tainted 6.16.0-00023-gff797757d658 #1 PREEMPT(none) [ 19.181372][ T64] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 19.181385][ T64] Call Trace: [ 19.181391][ T64] <TASK> [ 19.181401][ T64] dump_stack_lvl (lib/dump_stack.c:123) [ 19.181440][ T64] dump_stack (lib/dump_stack.c:130) [ 19.181465][ T64] check_preemption_disabled (arch/x86/include/asm/preempt.h:85 lib/smp_processor_id.c:53) [ 19.181494][ T64] debug_smp_processor_id (lib/smp_processor_id.c:61) [ 19.181514][ T64] task_cache_work (kernel/sched/fair.c:1422)
Thanks for the report, will use get_cpu() in next version.
thanks,
Chenyu
RFC PATCH v4 24/28 sched: Record average number of runninhg tasks per process
In reply to Chen YuPerformance regression was found when running hackbench with many threads per process(the fd number is high). To avoid this regression, process having a large number of active threads should be excluded from cache aware scheduling.
With sched_cache enabled, record the number of active threads within the process. This calculation occurs in the periodic task_cache_work():
when iterating over the CPUs, check the currently running task on that CPU; if the running task belongs to the same process as the task that launches task_cache_work(), increment the active thread count by 1.
If the number exceeds the number of CPUs in the preferred LLC, sched_cache is prevented from aggregating too many threads in one LLC domain.
--- include/linux/mm_types.h | 1 + kernel/sched/fair.c | 14 ++++++++++++-- 2 files changed, 13 insertions(+), 2 deletions(-)
include/linux/mm_types.h +1 -0
@@ -1033,6 +1033,7 @@ struct mm_struct {raw_spinlock_t mm_sched_lock;unsigned long mm_sched_epoch;int mm_sched_cpu; +u64 nr_running_avg;#endif #ifdef CONFIG_MMU
kernel/sched/fair.c +12 -2
@@ -1414,12 +1414,13 @@ static void get_scan_cpumasks(cpumask_var_t cpus, int cache_cpu, static void __no_profile task_cache_work(struct callback_head *work) { -struct task_struct *p = current; +struct task_struct *p = current, *cur;struct mm_struct *mm = p->mm;unsigned long m_a_occ =0 ;unsigned long last_m_a_occ =0 ;int cpu, m_a_cpu =-1 , cache_cpu, - pref_nid =NUMA_NO_NODE , curr_cpu =smp_processor_id (); + pref_nid =NUMA_NO_NODE , curr_cpu =smp_processor_id (), + nr_running =0 ;cpumask_var_t cpus;WARN_ON_ONCE (work != &p->cache_work);
@@ -1460,6 +1461,14 @@ static void __no_profile task_cache_work(struct callback_head *work) m_cpu = i; } nr++; + +rcu_read_lock (); + cur =rcu_dereference (cpu_rq (i)->curr); +if (cur && !(cur->flags & (PF_EXITING |PF_KTHREAD )) && + cur->mm == mm) + nr_running++; +rcu_read_unlock (); +trace_printk ("(%d) occ: %ld m_occ: %ld m_cpu: %d nr: %d\n" ,per_cpu (sd_llc_id, i), occ, m_occ, m_cpu, nr); }
@@ -1489,6 +1498,7 @@ static void __no_profile task_cache_work(struct callback_head *work) mm->mm_sched_cpu = m_a_cpu; } +update_avg (&mm->nr_running_avg, nr_running);free_cpumask_var (cpus); }
Signature · 1 line
2.25.1
RFC PATCH v4 25/28 sched: Skip cache aware scheduling if the process has many active threads
In reply to Chen YuIf the number of active threads within the process exceeds the number of Cores(divided by SMTs number) in the LLC, do not enable cache-aware scheduling. This is because there is a risk of cache contention within the preferred LLC when too many threads are present.
--- kernel/sched/fair.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)
kernel/sched/fair.c +23 -1
@@ -1205,6 +1205,18 @@ static inline int pref_llc_idx(struct task_struct *p)return llc_idx (p->preferred_llc); } +static bool exceed_llc_nr (struct mm_struct *mm,int cpu) +{ +int smt_nr =1 ; + +#ifdef CONFIG_SCHED_SMT +if (sched_smt_active ()) + smt_nr =cpumask_weight (cpu_smt_mask (cpu)); +#endif + +return ((mm->nr_running_avg * smt_nr) >per_cpu (sd_llc_size, cpu)); +} +static void account_llc_enqueue (struct rq *rq,struct task_struct *p) {int pref_llc;
@@ -1350,7 +1362,8 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)* it's preferred state. */ if (epoch -READ_ONCE (mm->mm_sched_epoch) > sysctl_llc_old || -get_nr_threads (p) <=1 ) { +get_nr_threads (p) <=1 || +exceed_llc_nr (mm,cpu_of (rq))) { mm->mm_sched_cpu =-1 ; pcpu_sched->occ =0 ; }
@@ -1430,6 +1443,11 @@ static void __no_profile task_cache_work(struct callback_head *work)if (p->flags &PF_EXITING )return ; +if (get_nr_threads (p) <=1 ) { + mm->mm_sched_cpu =-1 ; +return ; + } +if (!zalloc_cpumask_var (&cpus,GFP_KERNEL ))return ;
@@ -9095,6 +9113,10 @@ static __maybe_unused enum llc_mig_hint get_migrate_hint(int src_cpu, int dst_cpif (cpu <0 )return mig_allow; +/* skip cache aware load balance for single/too many threads */ +if (get_nr_threads (p) <=1 ||exceed_llc_nr (mm, dst_cpu)) +return mig_allow; +if (cpus_share_cache (dst_cpu, cpu))return _get_migrate_hint (src_cpu, dst_cpu,task_util (p), true);
Signature · 1 line
2.25.1
Re: [RFC PATCH v4 25/28] sched: Skip cache aware scheduling if the process has many active threads
Re: [RFC PATCH v4 25/28] sched: Skip cache aware scheduling if the process has many active threads
In reply to Chen YuSeveral different test cases with mysql and sysbench shows that this patch causes about 10% performance regressions on my computer with 256 cores. Perf-top shows exceed_llc_nr is high. Could you help to address this problems ?
Re: [RFC PATCH v4 25/28] sched: Skip cache aware scheduling if the process has many active threads
Quoted 3 lines · Tingyin Duan
Several different test cases with mysql and sysbench shows that this patch causes about 10% performance regressions on my computer with 256 cores. Perf-top shows
exceed_llc_nris high. Could you help to address this problems?
Thanks for bringing this to public for discussion. As we synced offline, the performance regression is likely to be caused by the cache contention introduced by the [25/28] patch:
The 1st issue:Multiple threads within the same process try to read the mm_struct->nr_running_avg while the task_cache_work() modifies the mm_struct->mm_sched_cpu from time to time. Since the mm_sched_cpu and nr_running_avg are in the same cacheline, this update turns the cacheline into "Modified" and the read triggers the costly "HITM" event. We should move nr_running_avg and mm_sched_cpu to different cachelines.
The 2nd issue:
If nr_running_avg remains consistently above the threshold in your test case, exceed_llc_nr() will always return true. This might cause the frequent write of -1 to mm->mm_sched_cpu, even if mm->mm_sched_cpu is already -1. This causes another cache contention that threads on other CPUs trying to read the mm->mm_sched_cpu. We should update the mm_struct's mm_sched_cpu field only if the value has been changed.
That is to say, the following patch should fix the regression:
include/linux/mm_types.h +5 -1
@@ -1032,7 +1032,11 @@ struct mm_struct {raw_spinlock_t mm_sched_lock;unsigned long mm_sched_epoch;int mm_sched_cpu; -u64 nr_running_avg; +/* +* mm_sched_cpu and nr_running_avg are put into seperate +* cachelines to avoid cache contention. +*/ +u64 nr_running_avg____cacheline_aligned_in_smp ;#endif #ifdef CONFIG_MMU
kernel/sched/fair.c +2 -1
@@ -1428,7 +1428,8 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) get_nr_threads(p) <= 1 || exceed_llc_nr(mm, cpu_of(rq)) || exceed_llc_capacity(mm, cpu_of(rq))) { - mm->mm_sched_cpu =-1 ; +if (mm->mm_sched_cpu !=-1 ) + mm->mm_sched_cpu =-1 ; pcpu_sched->occ =0 ; }
Signature · 16 lines
2.25.1 With above change, the regression I mentioned in the cover letter, when running multiple instances of hackbench on AMD Milan has disappeared. And max latency improvement of sysbench+MariaDB are observed on Milan: transactions per sec.: -0.72% min latency: -0.00% avg latency: -0.00% max latency: +78.90% 95th percentile: -0.00% events avg: -0.72% events stddev: +50.72% thanks, Chenyu
Re: [RFC PATCH v4 25/28] sched: Skip cache aware scheduling if the process has many active threads
After applying this patch, the performance regression has been significantly improved. Thank you!
RFC PATCH v4 26/28 sched: Do not enable cache aware scheduling for process with large RSS
In reply to Chen YuIt has been reported that when running memory-intensive workloads such as stream, sched_cache may saturate the memory bandwidth on the preferred LLC.
To prevent this from happening, evaluate the process's memory footprint by checking the size of RSS (anonymous pages and shared pages) and comparing it to the size of the LLC. If the former is larger, skip cache-aware scheduling. This is because if tasks do not actually share data, aggregating tasks with large RSS will likely result in cache contention and performance depredation.
However, in theory, RSS is not the same as memory footprint. This is just an estimated approach to prevent over-aggregation. The default behavior is to strictly compare the size of RSS with the size of the LLC. The next patch will introduce a user-provided hint to customize this comparison.
--- kernel/sched/fair.c | 47 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 44 insertions(+), 3 deletions(-)
kernel/sched/fair.c +44 -3
@@ -1205,6 +1205,34 @@ static inline int pref_llc_idx(struct task_struct *p)return llc_idx (p->preferred_llc); } +static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) +{ +struct cpu_cacheinfo *this_cpu_ci; +struct cacheinfo *l3_leaf; +unsigned long rss; +unsigned int llc; + +/* +* get_cpu_cacheinfo_level() can not be used +* because it requires the cpu_hotplug_lock +* to be held. Use get_cpu_cacheinfo() +* directly because the 'cpu' can not be +* offlined at the moment. +*/ + this_cpu_ci =get_cpu_cacheinfo (cpu); +if (!this_cpu_ci->info_list || + this_cpu_ci->num_leaves <3 ) +return true; + + l3_leaf = this_cpu_ci->info_list +3 ; + llc = l3_leaf->size; + + rss =get_mm_counter (mm,MM_ANONPAGES ) + +get_mm_counter (mm,MM_SHMEMPAGES ); + +return (llc <= (rss *PAGE_SIZE )); +} +static bool exceed_llc_nr (struct mm_struct *mm,int cpu) {int smt_nr =1 ;
@@ -1363,7 +1391,8 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)*/ if (epoch -READ_ONCE (mm->mm_sched_epoch) > sysctl_llc_old ||get_nr_threads (p) <=1 || -exceed_llc_nr (mm,cpu_of (rq))) { +exceed_llc_nr (mm,cpu_of (rq)) || +exceed_llc_capacity (mm,cpu_of (rq))) { mm->mm_sched_cpu =-1 ; pcpu_sched->occ =0 ; }
@@ -1448,6 +1477,14 @@ static void __no_profile task_cache_work(struct callback_head *work)return ; } +/* +* Do not check exceed_llc_nr() because +* the active number of threads needs to +* been updated anyway. +*/ +if (exceed_llc_capacity (mm, curr_cpu)) +return ; +if (!zalloc_cpumask_var (&cpus,GFP_KERNEL ))return ;
@@ -9113,8 +9150,12 @@ static __maybe_unused enum llc_mig_hint get_migrate_hint(int src_cpu, int dst_cpif (cpu <0 )return mig_allow; -/* skip cache aware load balance for single/too many threads */ -if (get_nr_threads (p) <=1 ||exceed_llc_nr (mm, dst_cpu)) +/* +* skip cache aware load balance for single/too many threads +* and large footprint. +*/ +if (get_nr_threads (p) <=1 ||exceed_llc_nr (mm, dst_cpu) || +exceed_llc_capacity (mm, dst_cpu))return mig_allow;if (cpus_share_cache (dst_cpu, cpu))
Signature · 1 line
2.25.1
Re: [RFC PATCH v4 26/28] sched: Do not enable cache aware scheduling for process with large RSS
Hi Chen Yu,
Thanks for your work.
I tested the patch set on AmpereOne CPU with 192 cores.
With CONFIG_SCHED_CLUSTER enabled, and with certain firmware setting, every eight cores will be grouped into a 'cluster' schedule domain with 'SD_SHARE_LLC' flag.
However, these eight cores do *no* share L3 cache in this setup.
In exceed_llc_capacity() of this patch, we have 'llc = l3_leaf->size', 'llc' will be zero if there is *no* L3 cache.
So exceed_llc_capacity() will be true and 'Cache Aware Scheduling' will not work. Please see details bellow.
I read in patch 01/28 "sched: Cache aware load-balancing" [1], Peter mentioned:
"It is an attempt at modelling cache affinity -- and while the patch really only targets LLC, it could very well be extended to also apply to clusters (L2). Specifically any case of multiple cache domains inside a node".
Do you have any idea how we can apply the cache aware load-balancing to clusters? The cores in the cluster may share L2 or LLC tags.
Quoted 46 lines · Chen Yu
It has been reported that when running memory-intensive workloads such as stream,
sched_cachemay saturate the memory bandwidth on the preferred LLC.To prevent this from happening, evaluate the process's memory footprint by checking the size of RSS (anonymous pages and shared pages) and comparing it to the size of the LLC. If the former is larger, skip cache-aware scheduling. This is because if tasks do not actually share data, aggregating tasks with large RSS will likely result in cache contention and performance depredation.
However, in theory, RSS is not the same as memory footprint. This is just an estimated approach to prevent over-aggregation. The default behavior is to strictly compare the size of RSS with the size of the LLC. The next patch will introduce a user-provided hint to customize this comparison.
Reported-by K Prateek Nayak <[email protected]>Co-developed-by Tim Chen <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 47 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 44 insertions(+), 3 deletions(-)kernel/sched/fair.c +12
-0@@ -1205,6 +1205,34 @@ static inline int pref_llc_idx(struct task_struct *p)return llc_idx (p->preferred_llc); } +static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) +{ +struct cpu_cacheinfo *this_cpu_ci; +struct cacheinfo *l3_leaf; +unsigned long rss; +unsigned int llc; + +/* +* get_cpu_cacheinfo_level() can not be used +* because it requires the cpu_hotplug_lock +* to be held. Use get_cpu_cacheinfo() +* directly because the 'cpu' can not be
kernel/sched/fair.c static inline int pref_llc_idx(struct task_struct *p) · line 1229
Quoted 10 lines
kernel/sched/fair.c +10
-0+* offlined at the moment. +*/ + this_cpu_ci =get_cpu_cacheinfo (cpu); +if (!this_cpu_ci->info_list || + this_cpu_ci->num_leaves <3 ) +return true; + + l3_leaf = this_cpu_ci->info_list +3 ; + llc = l3_leaf->size; +
For some arm64 CPU topology, cores can be grouped into 'cluster'. Cores in a cluster may not share L3 cache. 'l3_leaf->size' will be 0.
It looks we assume LLC is L3 cache?
Can we skip exceed_llc_capacity() check if no L3?
Like this draft patch:
kernel/sched/fair.c +2 -0
@@ -1227,6 +1227,8 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu) l3_leaf = this_cpu_ci->info_list +3 ; llc = l3_leaf->size; +if (!llc) +return false; rss =get_mm_counter (mm,MM_ANONPAGES ) +get_mm_counter (mm,MM_SHMEMPAGES );
kernel/sched/fair.c static inline int pref_llc_idx(struct task_struct *p) · line 1233
Quoted 4 lines
kernel/sched/fair.c +4
-0+ rss =get_mm_counter (mm,MM_ANONPAGES ) + +get_mm_counter (mm,MM_SHMEMPAGES ); + +return (llc <= (rss *PAGE_SIZE ));
If 'llc' is 0, exceed_llc_capacity() will always return true.
Quoted 35 lines
kernel/sched/fair.c +12
-2+} +static bool exceed_llc_nr (struct mm_struct *mm,int cpu) {int smt_nr =1 ;@@ -1363,7 +1391,8 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)*/ if (epoch -READ_ONCE (mm->mm_sched_epoch) > sysctl_llc_old ||get_nr_threads (p) <=1 || -exceed_llc_nr (mm,cpu_of (rq))) { +exceed_llc_nr (mm,cpu_of (rq)) || +exceed_llc_capacity (mm,cpu_of (rq))) { mm->mm_sched_cpu =-1 ; pcpu_sched->occ =0 ; }@@ -1448,6 +1477,14 @@ static void __no_profile task_cache_work(struct callback_head *work)return ; } +/* +* Do not check exceed_llc_nr() because +* the active number of threads needs to +* been updated anyway. +*/ +if (exceed_llc_capacity (mm, curr_cpu)) +return ; +if (!zalloc_cpumask_var (&cpus,GFP_KERNEL ))return ;@@ -9113,8 +9150,12 @@ static __maybe_unused enum llc_mig_hint get_migrate_hint(int src_cpu, int dst_cpif (cpu <0 )return mig_allow; -/* skip cache aware load balance for single/too many threads */
kernel/sched/fair.c static __maybe_unused enum llc_mig_hint get_migrate_hint(int src_cpu, int dst_cp · line 9161
Quoted 10 lines
kernel/sched/fair.c +6
-1-if (get_nr_threads (p) <=1 ||exceed_llc_nr (mm, dst_cpu)) +/* +* skip cache aware load balance for single/too many threads +* and large footprint. +*/ +if (get_nr_threads (p) <=1 ||exceed_llc_nr (mm, dst_cpu) || +exceed_llc_capacity (mm, dst_cpu))return mig_allow;if (cpus_share_cache (dst_cpu, cpu))
Thanks,-adam
Re: [RFC PATCH v4 26/28] sched: Do not enable cache aware scheduling for process with large RSS
Quoted 22 lines · Adam Li
Hi Chen Yu,
Thanks for your work.
I tested the patch set on AmpereOne CPU with 192 cores.With
CONFIG_SCHED_CLUSTERenabled, and with certain firmware setting, every eight cores will be grouped into a 'cluster' schedule domain with 'SD_SHARE_LLC' flag.
However, these eight cores do *no* share L3 cache in this setup.In
exceed_llc_capacity()of this patch, we have 'llc =l3_leaf->size', 'llc' will be zero if there is *no* L3 cache.
Soexceed_llc_capacity()will be true and 'Cache Aware Scheduling' will not work. Please see details bellow.I read in patch 01/28 "sched: Cache aware load-balancing" [1], Peter mentioned:
"It is an attempt at modelling cache affinity -- and while the patch really only targets LLC, it could very well be extended to also apply to clusters (L2). Specifically any case of multiple cache domains inside a node".
Quoted 2 lines
Do you have any idea how we can apply the cache aware load-balancing to clusters? The cores in the cluster may share L2 or LLC tags.
My understanding is that if there is no L3 cache, then the L2 becomes the LLC. We don’t need to modify the code specific to L2-aware scheduling because the L2 is now the last-level cache (LLC). However, as you observed, there are some cases that need to be taken care of. For example, Patch 8 needs to be fixed so that it does not always retrieve the cache size of L3.
On the other hand, if the system has both an L2 cluster and an L3, the code might need to be changed if we want to perform L2 cache aggregation rather than L3 cache aggregation.
Quoted 12 lines
Review
Quoted 4 lines · Chen Yu
(quoted diff) +4
-0+ + l3_leaf = this_cpu_ci->info_list + 3; + llc = l3_leaf->size; +For some arm64 CPU topology, cores can be grouped into 'cluster'. Cores in a cluster may not share L3 cache. '
l3_leaf->size' will be 0.
Quoted 1 line
It looks we assume LLC is L3 cache?
Right, but LLC should not always be L3, need a fix here.
Quoted 2 lines
Can we skip
exceed_llc_capacity()check if no L3?
I thought we should return the size of L2 instead, no?
thanks,
Chenyu> Like this draft patch:
Quoted 9 lines
kernel/sched/fair.c +2
-0@@ -1227,6 +1227,8 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu) l3_leaf = this_cpu_ci->info_list +3 ; llc = l3_leaf->size; +if (!llc) +return false;
RFC PATCH v4 27/28 sched: Allow the user space to tune the scale factor for RSS comparison
In reply to Chen Yusched_cache compares the process's resident pages with the size of the LLC to determine whether task aggregation on the preferred LLC might cause cache contention. If the former is larger than the latter, skip cache-aware task aggregation. However, some workloads with large resident pages have a small memory footprint; such workloads could benefit from cache-aware scheduling. The kernel lacks a efficient mechanism to track the task's memory footprint (yes, we have resctrl, but it is for user-space query, and not process scope), so it is up to userspace to pass this hint to the kernel.
Introduce /sys/kernel/debug/sched/sched_cache_ignore_rss to control the extent to which users ignore the RSS restriction. This value ranges from 0 to 100. A value of 0 means that the user disables the cache aware scheduling. 1 means if a process's RSS is larger than the LLC size, cache-aware scheduling will be skipped. 100 means cache aware scheduling is alwasy enabled regardless of RSS size. N (between 1 and 100) means turn off cache aware scheduling when RSS is greater than (N-1) * 256 * LLC size
For example, suppose the L3 size is 32MB. If the sysctl_sched_cache_ignore_rss is 1: When the RSS is larger than 32MB, the process is regarded as exceeding the LLC capacity. If the sysctl_sched_cache_ignore_rss is 99: When the RSS is larger than 784GB, the process is regarded as exceeding the LLC capacity(please refer to the code):
784GB = (1 + (99 - 1) * 256) * 32MB
Additionally, the number of SMTs is also considered for sysctl_sched_cache_aggr_cap; if there are many SMTs in the core, sysctl_llc_aggr_cap will be reduced. This inhibits task aggregation from cache-aware scheduling on systems with a high number of SMTs, like Power 10 and Power 11.
--- kernel/sched/debug.c | 82 +++++++++++++++++++++++++++++++++++++++++++- kernel/sched/fair.c | 10 ++++-- kernel/sched/sched.h | 3 +- 3 files changed, 90 insertions(+), 5 deletions(-)
kernel/sched/debug.c +81 -1
@@ -166,6 +166,83 @@ static const struct file_operations sched_feat_fops = { .release = single_release, }; +#ifdef CONFIG_SCHED_CACHE +#define SCHED_CACHE_CREATE_CONTROL (name, val) \ +static int sysctl_sched_cache_##name = val; \ +static ssize_t sched_cache_write_##name(struct file *filp, \ + const char __user *ubuf, \ + size_t cnt, loff_t *ppos) \ +{ \ + char buf[16]; \ + unsigned int percent; \ + if (cnt > 15) \ + cnt = 15; \ + if (copy_from_user(&buf, ubuf, cnt)) \ + return -EFAULT; \ + buf[cnt] = '\0'; \ + if (kstrtouint(buf, 10, &percent)) \ + return -EINVAL; \ + if (percent > 100) \ + return -EINVAL; \ + sysctl_sched_cache_##name = percent; \ + *ppos += cnt; \ + return cnt; \ +} \ +static int sched_cache_show_##name(struct seq_file *m, void *v) \ +{ \ + seq_printf(m, "%d\n", sysctl_sched_cache_##name); \ + return 0; \ +} \ +static int sched_cache_open_##name(struct inode *inode, \ + struct file *filp) \ +{ \ + return single_open(filp, sched_cache_show_##name, NULL); \ +} \ +static const struct file_operations sched_cache_fops_##name = { \ + .open = sched_cache_open_##name, \ + .write = sched_cache_write_##name, \ + .read = seq_read, \ + .llseek = seq_lseek, \ + .release = single_release, \ +} + +SCHED_CACHE_CREATE_CONTROL (ignore_rss,1 ); +int get_sched_cache_rss_scale (void ) +{ +if (!sysctl_sched_cache_ignore_rss) +return 0 ; + +if (sysctl_sched_cache_ignore_rss >=100 ) +return INT_MAX ; +/* +* Suppose the L3 size is 32MB. If the +* sysctl_sched_cache_ignore_rss is 1: +* When the RSS is larger than 32MB, +* the process is regarded as exceeding +* the LLC capacity. If the +* sysctl_sched_cache_ignore_rss is 99: +* When the RSS is larger than 784GB, +* the process is regarded as exceeding +* the LLC capacity: +* 784GB = (1 + (99 - 1) * 256) * 32MB +*/ +return (1 + (sysctl_sched_cache_ignore_rss -1 ) *256 ); +} + +SCHED_CACHE_CREATE_CONTROL (aggr_cap,50 ); +int get_sched_cache_cap_scale (void ) +{ +int smt_nr =1 ; + +#ifdef CONFIG_SCHED_SMT +if (sched_smt_active ()) + smt_nr = +cpumask_weight (cpu_smt_mask (raw_smp_processor_id ())); +#endif +return (sysctl_sched_cache_aggr_cap / smt_nr); +} +#endif /* SCHED_CACHE */ +#ifdef CONFIG_SMP static ssize_t sched_scaling_write(struct file *filp, const char __user *ubuf,
@@ -533,10 +610,13 @@ static __init int sched_init_debug(void)#endif #ifdef CONFIG_SCHED_CACHE -debugfs_create_u32 ("llc_aggr_cap" ,0644 , debugfs_sched, &sysctl_llc_aggr_cap);debugfs_create_u32 ("llc_aggr_imb" ,0644 , debugfs_sched, &sysctl_llc_aggr_imb);debugfs_create_u32 ("llc_period" ,0644 , debugfs_sched, &sysctl_llc_period);debugfs_create_u32 ("llc_old" ,0644 , debugfs_sched, &sysctl_llc_old); +debugfs_create_file ("llc_aggr_cap" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_aggr_cap); +debugfs_create_file ("llc_ignore_rss" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_ignore_rss);#endif debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);
kernel/sched/fair.c +7 -3
@@ -1211,6 +1211,7 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)struct cacheinfo *l3_leaf;unsigned long rss;unsigned int llc; +int scale;/* * get_cpu_cacheinfo_level() can not be used
@@ -1230,7 +1231,11 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu) rss =get_mm_counter (mm,MM_ANONPAGES ) +get_mm_counter (mm,MM_SHMEMPAGES ); -return (llc <= (rss *PAGE_SIZE )); + scale =get_sched_cache_rss_scale (); +if (scale ==INT_MAX ) +return false; + +return ((llc * scale) <= (rss *PAGE_SIZE )); }static bool exceed_llc_nr (struct mm_struct *mm,int cpu)
@@ -9037,7 +9042,6 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)static long __migrate_degrades_locality (struct task_struct *p,int src_cpu,int dst_cpu,bool idle); -__read_mostly unsigned int sysctl_llc_aggr_cap =50 ;__read_mostly unsigned int sysctl_llc_aggr_imb =20 ;/*
@@ -9049,7 +9053,7 @@ __read_mostly unsigned int sysctl_llc_aggr_imb = 20;* (default: ~50%) */ #define fits_llc_capacity (util, max) \ - ((util) * 100 < (max) * sysctl_llc_aggr_cap) + ((util) * 100 < (max) * get_sched_cache_cap_scale())/* * The margin used when comparing utilization.
kernel/sched/sched.h +2 -1
@@ -2855,11 +2855,12 @@ extern unsigned int sysctl_numa_balancing_scan_size;extern unsigned int sysctl_numa_balancing_hot_threshold;#ifdef CONFIG_SCHED_CACHE -extern unsigned int sysctl_llc_aggr_cap;extern unsigned int sysctl_llc_aggr_imb;extern struct static_key_false sched_cache_present;extern unsigned int sysctl_llc_period;extern unsigned int sysctl_llc_old; +int get_sched_cache_rss_scale (void ); +int get_sched_cache_cap_scale (void );#endif #ifdef CONFIG_SCHED_HRTICK
Signature · 1 line
2.25.1
RFC PATCH v4 28/28 sched: Add ftrace to track cache aware load balance and hottest CPU changes
In reply to Chen YuIntroduce 3 trace events:
1.
The average time spent scanning CPUs and calculating occupancy in each sample period. This event can be used to track the overhead of cache-aware scheduling.
2.
The footprint when switching to a new mm_sched_cpu (a cache-hot CPU). This event can be used to track whether there is any abnormal bouncing of mm_sched_cpu.
3.
The footprint of load balancing when migrating a task between CPUs. This event can be used to track whether cache-aware load balancing behaves as expected.
All these events can be used with bpftrace to gain a basic understanding of whether cache-aware scheduling is effective.
--- include/trace/events/sched.h | 93 ++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 25 ++++++++-- 2 files changed, 113 insertions(+), 5 deletions(-)
include/trace/events/sched.h +93 -0
@@ -10,6 +10,99 @@#include <linux/tracepoint.h> #include <linux/binfmts.h> +TRACE_EVENT (sched_scan_cost, + + TP_PROTO(struct task_struct *t, u64 cost, int nr, + u64 old_running, u64 new_running), + +TP_ARGS (t, cost, nr, old_running, new_running), + +TP_STRUCT__entry ( + __array( char, comm, TASK_COMM_LEN ) + __field( pid_t, pid ) + __field( u64, cost ) + __field( int, nr ) + __field( u64, old_running ) +__field ( u64, new_running ) + ), + +TP_fast_assign ( + memcpy(__entry->comm, t->comm, TASK_COMM_LEN); + __entry->pid = t->pid; + __entry->cost = cost; + __entry->nr = nr; + __entry->old_running = old_running; + __entry->new_running = new_running; + ), + +TP_printk ("comm=%s pid=%d cost=%llu nr=%d old_r=%lld new_r=%lld" , + __entry->comm, __entry->pid, + __entry->cost, __entry->nr, + __entry->old_running, __entry->new_running) +); + +TRACE_EVENT (sched_cache_work, + + TP_PROTO(struct task_struct *t, int pref_cpu, int pref_llc, + int new_cpu, int new_llc), + +TP_ARGS (t, pref_cpu, pref_llc, new_cpu, new_llc), + +TP_STRUCT__entry ( + __array( char, comm, TASK_COMM_LEN ) + __field( pid_t, pid ) + __field( int, pref_cpu ) + __field( int, pref_llc ) + __field( int, new_cpu ) +__field ( int, new_llc ) + ), + +TP_fast_assign ( + memcpy(__entry->comm, t->comm, TASK_COMM_LEN); + __entry->pid = t->pid; + __entry->pref_cpu = pref_cpu; + __entry->pref_llc = pref_llc; + __entry->new_cpu = new_cpu; + __entry->new_llc = new_llc; + ), + +TP_printk ("comm=%s pid=%d pref_cpu=%d pref_llc=%d attach_cpu=%d attach_llc=%d" , + __entry->comm, __entry->pid, + __entry->pref_cpu, __entry->pref_llc, + __entry->new_cpu, __entry->new_llc) +); + +TRACE_EVENT (sched_attach_task, + + TP_PROTO(struct task_struct *t, int pref_cpu, int pref_llc, + int attach_cpu, int attach_llc), + +TP_ARGS (t, pref_cpu, pref_llc, attach_cpu, attach_llc), + +TP_STRUCT__entry ( + __array( char, comm, TASK_COMM_LEN ) + __field( pid_t, pid ) + __field( int, pref_cpu ) + __field( int, pref_llc ) + __field( int, attach_cpu ) +__field ( int, attach_llc ) + ), + +TP_fast_assign ( + memcpy(__entry->comm, t->comm, TASK_COMM_LEN); + __entry->pid = t->pid; + __entry->pref_cpu = pref_cpu; + __entry->pref_llc = pref_llc; + __entry->attach_cpu = attach_cpu; + __entry->attach_llc = attach_llc; + ), + +TP_printk ("comm=%s pid=%d pref_cpu=%d pref_llc=%d attach_cpu=%d attach_llc=%d" , + __entry->comm, __entry->pid, + __entry->pref_cpu, __entry->pref_llc, + __entry->attach_cpu, __entry->attach_llc) +); +/* * Tracepoint for calling kthread_stop, performed to end a kthread: */
kernel/sched/fair.c +20 -5
@@ -1467,8 +1467,9 @@ static void __no_profile task_cache_work(struct callback_head *work)unsigned long last_m_a_occ =0 ;int cpu, m_a_cpu =-1 , cache_cpu, pref_nid =NUMA_NO_NODE , curr_cpu =smp_processor_id (), - nr_running =0 ; + nr_running =0 , nr_scan =0 ;cpumask_var_t cpus; +u64 t0, scan_cost =0 ;WARN_ON_ONCE (work != &p->cache_work);
@@ -1499,6 +1500,7 @@ static void __no_profile task_cache_work(struct callback_head *work) pref_nid = p->numa_preferred_nid;#endif + t0 =sched_clock_cpu (curr_cpu);scoped_guard (cpus_read_lock) {get_scan_cpumasks (cpus, cache_cpu, pref_nid, curr_cpu);
@@ -1521,6 +1523,7 @@ static void __no_profile task_cache_work(struct callback_head *work) m_cpu = i; } nr++; + nr_scan++;rcu_read_lock (); cur =rcu_dereference (cpu_rq (i)->curr);
@@ -1529,8 +1532,8 @@ static void __no_profile task_cache_work(struct callback_head *work) nr_running++;rcu_read_unlock (); -trace_printk ("(%d) occ: %ld m_occ: %ld m_cpu: %d nr: %d\n" , -per_cpu (sd_llc_id, i), occ, m_occ, m_cpu, nr); +//trace_printk("(%d) occ: %ld m_occ: %ld m_cpu: %d nr: %d\n", +// per_cpu(sd_llc_id, i), occ, m_occ, m_cpu, nr); }// a_occ /= nr;
@@ -1541,8 +1544,8 @@ static void __no_profile task_cache_work(struct callback_head *work)if (llc_id (cpu) ==llc_id (mm->mm_sched_cpu)) last_m_a_occ = a_occ; -trace_printk ("(%d) a_occ: %ld m_a_occ: %ld\n" , -per_cpu (sd_llc_id, cpu), a_occ, m_a_occ); +//trace_printk("(%d) a_occ: %ld m_a_occ: %ld\n", +// per_cpu(sd_llc_id, cpu), a_occ, m_a_occ); for_each_cpu (i,sched_domain_span (sd)) {/* XXX threshold ? */
@@ -1553,12 +1556,17 @@ static void __no_profile task_cache_work(struct callback_head *work) } } + scan_cost =sched_clock_cpu (curr_cpu) - t0; +if (m_a_occ > (2 * last_m_a_occ)) {/* avoid the bouncing of mm_sched_cpu */ +trace_sched_cache_work (p, mm->mm_sched_cpu,llc_id (mm->mm_sched_cpu), + m_a_cpu,llc_id (m_a_cpu)); mm->mm_sched_cpu = m_a_cpu; }update_avg (&mm->nr_running_avg, nr_running); +trace_sched_scan_cost (p, scan_cost, nr_scan, mm->nr_running_avg, nr_running);free_cpumask_var (cpus); }
@@ -10443,6 +10451,13 @@ static void attach_task(struct rq *rq, struct task_struct *p) {lockdep_assert_rq_held (rq); +#ifdef CONFIG_SCHED_CACHE +if (p->mm) +trace_sched_attach_task (p, + p->mm->mm_sched_cpu, + p->mm->mm_sched_cpu !=-1 ?llc_id (p->mm->mm_sched_cpu) :-1 , +cpu_of (rq),llc_id (cpu_of (rq))); +#endif WARN_ON_ONCE (task_rq (p) != rq);activate_task (rq, p,ENQUEUE_NOCLOCK );wakeup_preempt (rq, p,0 );
Signature · 1 line
2.25.1