sched/cache: Refresh LLC capacity across CPU hotplug
Subsystems (2)
PATCH 1/1 sched/cache: Refresh LLC capacity across CPU hotplug
The scheduler scales LLC capacity by the fraction of cache-sharing CPUs covered by a domain:
llc_bytes = cache_size * span_weight / shared_weight
During CPU teardown, sched_cpu_deactivate() rebuilds scheduler domains before cacheinfo_cpu_pre_down() removes the CPU from shared_cpu_map. The new domains therefore use the old sharing weight. The later call to sched_update_llc_bytes() looks up the departing CPU's sd_llc, which has already been detached, and returns without correcting the surviving CPUs.
On a Ryzen 5 7535U with twelve logical CPUs sharing a 16 MiB LLC, offlining one SMT sibling left the remaining CPUs with:
llc_bytes = floor(16777216 * 11 / 12) = 15379114 bytes
The correct capacity is still 16777216 bytes. On systems with active cache-aware scheduling, an underestimated capacity can cause exceed_llc_capacity() to reject aggregation for a process whose footprint would fit. Unchanged cpuset partitions sharing the physical cache can also retain stale capacity when a CPU comes online in another partition.
Pass the cache-sharing mask already retained by cacheinfo to the scheduler update. Refresh every surviving CPU using its own LLC domain so that each partition receives the correct share. This also preserves the correction needed as cache-sharing maps grow during boot.
Keep the existing CPU-hotplug and scheduler-domain synchronization. The update remains on the hotplug path; no steady-state scheduling operation or persistent allocation is added.
---
The issue was identified by tracing the scheduler/cacheinfo teardown ordering, then checking live llc_bytes values using the running kernel's BTF layout and /proc/kcore.
Validation:
- Reproduced the stale value on 7.2.3-arch1-3 on the Ryzen system above. The patched kernel retained 16777216 bytes on every surviving CPU.
- Ten SMT-thread and ten whole-core hotplug cycles passed on the patched kernel, including capacity checks after each removal and restoration. The existing limited CPU-hotplug selftest also passed.
- Source-level state fixtures: five failures in eight scenarios before the fix, eight passes after it. These cover partition changes, unequal spans, sparse CPU IDs and boot-time map growth, but not concurrency.
- Full x86-64 baseline and patched bzImage/modules builds passed with matched configs apart from LOCALVERSION. Focused ARM64, x86 without CONFIG_SCHED_CACHE, and x86 UP builds also passed.
This host has only one LLC, so the live checks establish the accounting correction, not an aggregation speedup. No controlled same-version performance comparison or multi-LLC hardware result is claimed.
drivers/base/cacheinfo.c | 11 ++++++----- include/linux/sched/topology.h | 4 ++-- kernel/sched/topology.c | 27 ++++++++++++--------------- 3 files changed, 20 insertions(+), 22 deletions(-)
drivers/base/cacheinfo.c +6 -5
@@ -1040,9 +1040,10 @@ static int cacheinfo_cpu_online(unsigned int cpu) rc =cache_add_dev (cpu);if (rc) goto err; -if (cpu_map_shared_cache (true, cpu, &cpu_map)) +if (cpu_map_shared_cache (true, cpu, &cpu_map)) {update_per_cpu_data_slice_size (true, cpu, cpu_map); -sched_update_llc_bytes (cpu); +sched_update_llc_bytes (cpu_map); + }return 0 ; err:free_cache_attributes (cpu);
@@ -1059,10 +1060,10 @@ static int cacheinfo_cpu_pre_down(unsigned int cpu)cpu_cache_sysfs_exit (cpu);free_cache_attributes (cpu); -if (nr_shared >1 ) +if (nr_shared >1 ) {update_per_cpu_data_slice_size (false, cpu, cpu_map); - -sched_update_llc_bytes (cpu); +sched_update_llc_bytes (cpu_map); + }return 0 ; }
include/linux/sched/topology.h +2 -2
@@ -281,9 +281,9 @@ static inline int task_node(const struct task_struct *p) }#ifdef CONFIG_SCHED_CACHE -extern void sched_update_llc_bytes (unsigned int cpu); +extern void sched_update_llc_bytes (const struct cpumask *cpus);#else -static inline void sched_update_llc_bytes (unsigned int cpu) { } +static inline void sched_update_llc_bytes (const struct cpumask *cpus) { }#endif #endif /* _LINUX_SCHED_TOPOLOGY_H */
kernel/sched/topology.c +12 -15
@@ -985,8 +985,8 @@ void sched_cache_active_set(void) }/* -* Update the bottom sched_domain's llc_bytes for @cpu and all its -* LLC siblings. Called from cacheinfo_cpu_online() or +* Update the bottom sched_domain's llc_bytes for @cpus sharing a physical +* LLC. Called from cacheinfo_cpu_online() or * cacheinfo_cpu_pre_down() with cpu hotplug lock held. * * Note: get_effective_llc_bytes() returns 0 on PowerPC.
@@ -996,32 +996,29 @@ void sched_cache_active_set(void)* and does not populates the per-CPU struct cpu_cacheinfo array * that get_cpu_cacheinfo_llc() reads. */ -void sched_update_llc_bytes (unsigned int cpu) +void sched_update_llc_bytes (const struct cpumask *cpus) {struct sched_domain *sd, *sdp;unsigned int i;sched_domains_mutex_lock (); - sdp =rcu_dereference_sched_domain (per_cpu (sd_llc, cpu)); -if (!sdp) - goto unlock; -/* -* ci->shared_cpu_map is built incrementally as CPUs come -* online, so the first CPU in an LLC initially sees -* hw_weight == 1 and computes an inflated llc_bytes in -* get_effective_llc_bytes(). Re-evaluating every LLC -* sibling on each online event corrects this once the full -* shared_cpu_map is known. +* The departing CPU's domains have already been detached when +* cacheinfo removes it. Use the surviving cache siblings instead. +* They may belong to different cpuset partitions, so use each CPU's +* own LLC domain to scale its share of the physical cache. */ -for_each_cpu (i,sched_domain_span (sdp)) { +for_each_cpu (i, cpus) { + sdp =rcu_dereference_sched_domain (per_cpu (sd_llc, i)); +if (!sdp) +continue ; + sd =rcu_dereference_sched_domain (cpu_rq (i)->sd);if (sd) sd->llc_bytes =get_effective_llc_bytes (i, sdp); } -unlock:sched_domains_mutex_unlock (); }
base-commit: 50d05c7c76c96b90462f24debacca971d2e86713
Hi Davi,
Quoted 14 lines · Davi Chaves Azevedo
The scheduler scales LLC capacity by the fraction of cache-sharing CPUs covered by a domain:
llc_bytes=cache_size*span_weight/shared_weightDuring CPU teardown,
sched_cpu_deactivate()rebuilds scheduler domains beforecacheinfo_cpu_pre_down()removes the CPU fromshared_cpu_map. The new domains therefore use the old sharing weight. The later call tosched_update_llc_bytes()looks up the departing CPU'ssd_llc, which has already been detached, and returns without correcting the surviving CPUs.On a Ryzen 5 7535U with twelve logical CPUs sharing a 16 MiB LLC, offlining one SMT sibling left the remaining CPUs with:
Quoted 8 lines
llc_bytes = floor(16777216 * 11 / 12) = 15379114 bytesThe correct capacity is still 16777216 bytes. On systems with active cache-aware scheduling, an underestimated capacity can cause
exceed_llc_capacity()to reject aggregation for a process whose footprint would fit. Unchanged cpuset partitions sharing the physical cache can also retain stale capacity when a CPU comes online in another partition.
Ah right, thanks very much for catching this. The current sched_update_llc_bytes() only considers the bootup sequence for build_sched_domains() and cacheinfo_cpu_online(), but misses the runtime CPU hotplug scenario, where it incorrectly ignores all the surviving CPUs due to an offline CPU.
Quoted 113 lines
Pass the cache-sharing mask already retained by cacheinfo to the scheduler update. Refresh every surviving CPU using its own LLC domain so that each partition receives the correct share. This also preserves the correction needed as cache-sharing maps grow during boot.
Keep the existing CPU-hotplug and scheduler-domain synchronization. The update remains on the hotplug path; no steady-state scheduling operation or persistent allocation is added.
Fixes 7030513a0877 ("sched/cache: Calculate the LLC size and store it in sched_domain")Signed-off-by Davi Chaves Azevedo <[email protected]>---
The issue was identified by tracing the scheduler/cacheinfo teardown ordering, then checking livellc_bytesvalues using the running kernel's BTF layout and/proc/kcore.Validation:
- Reproduced the stale value on 7.2.3-arch1-3 on the Ryzen system above. The patched kernel retained 16777216 bytes on every surviving CPU.
- Ten SMT-thread and ten whole-core hotplug cycles passed on the patched kernel, including capacity checks after each removal and restoration. The existing limited CPU-hotplug selftest also passed.
- Source-level state fixtures: five failures in eight scenarios before the fix, eight passes after it. These cover partition changes, unequal spans, sparse CPU IDs and boot-time map growth, but not concurrency.
- Full x86-64 baseline and patched bzImage/modules builds passed with matched configs apart from LOCALVERSION. Focused ARM64, x86 withoutCONFIG_SCHED_CACHE, and x86 UP builds also passed.This host has only one LLC, so the live checks establish the accounting correction, not an aggregation speedup. No controlled same-version performance comparison or multi-LLC hardware result is claimed.
drivers/base/cacheinfo.c | 11 ++++++----- include/linux/sched/topology.h | 4 ++-- kernel/sched/topology.c | 27 ++++++++++++--------------- 3 files changed, 20 insertions(+), 22 deletions(-)drivers/base/cacheinfo.c +6
-5@@ -1040,9 +1040,10 @@ static int cacheinfo_cpu_online(unsigned int cpu) rc =cache_add_dev (cpu);if (rc) goto err; -if (cpu_map_shared_cache (true, cpu, &cpu_map)) +if (cpu_map_shared_cache (true, cpu, &cpu_map)) {update_per_cpu_data_slice_size (true, cpu, cpu_map); -sched_update_llc_bytes (cpu); +sched_update_llc_bytes (cpu_map); + }return 0 ; err:free_cache_attributes (cpu);@@ -1059,10 +1060,10 @@ static int cacheinfo_cpu_pre_down(unsigned int cpu)cpu_cache_sysfs_exit (cpu);free_cache_attributes (cpu); -if (nr_shared >1 ) +if (nr_shared >1 ) {update_per_cpu_data_slice_size (false, cpu, cpu_map); - -sched_update_llc_bytes (cpu); +sched_update_llc_bytes (cpu_map); + }return 0 ; }include/linux/sched/topology.h +2
-2@@ -281,9 +281,9 @@ static inline int task_node(const struct task_struct *p) }#ifdef CONFIG_SCHED_CACHE -extern void sched_update_llc_bytes (unsigned int cpu); +extern void sched_update_llc_bytes (const struct cpumask *cpus);#else -static inline void sched_update_llc_bytes (unsigned int cpu) { } +static inline void sched_update_llc_bytes (const struct cpumask *cpus) { }#endif #endif /* _LINUX_SCHED_TOPOLOGY_H */ kernel/sched/topology.c +3
-4@@ -985,8 +985,8 @@ void sched_cache_active_set(void) }/* -* Update the bottom sched_domain's llc_bytes for @cpu and all its -* LLC siblings. Called from cacheinfo_cpu_online() or +* Update the bottom sched_domain's llc_bytes for @cpus sharing a physical +* LLC. Called from cacheinfo_cpu_online() or * cacheinfo_cpu_pre_down() with cpu hotplug lock held. * * Note: get_effective_llc_bytes() returns 0 on PowerPC. @@ -996,32 +996,29 @@ void sched_cache_active_set(void)* and does not populates the per-CPU struct cpu_cacheinfo array * that get_cpu_cacheinfo_llc() reads. */ -void sched_update_llc_bytes (unsigned int cpu) +void sched_update_llc_bytes (const struct cpumask *cpus) {struct sched_domain *sd, *sdp;unsigned int i;sched_domains_mutex_lock (); - sdp =rcu_dereference_sched_domain (per_cpu (sd_llc, cpu));
kernel/sched/topology.c void sched_cache_active_set(void) · line 1016
Quoted 10 lines
kernel/sched/topology.c +0
-9-if (!sdp) - goto unlock; -/* -* ci->shared_cpu_map is built incrementally as CPUs come -* online, so the first CPU in an LLC initially sees -* hw_weight == 1 and computes an inflated llc_bytes in -* get_effective_llc_bytes(). Re-evaluating every LLC -* sibling on each online event corrects this once the full -* shared_cpu_map is known.
Maybe the above comments can be kept, because they describe the bootup scenario, with your offline case/domain partition added — just in case in the future the reader might wonder why we iterate every online CPU again and again during cacheinfo_cpu_online()/offline().
Quoted 20 lines
kernel/sched/topology.c +9
-2+* The departing CPU's domains have already been detached when +* cacheinfo removes it. Use the surviving cache siblings instead. +* They may belong to different cpuset partitions, so use each CPU's +* own LLC domain to scale its share of the physical cache. */ -for_each_cpu (i,sched_domain_span (sdp)) { +for_each_cpu (i, cpus) { + sdp =rcu_dereference_sched_domain (per_cpu (sd_llc, i)); +if (!sdp) +continue ; + sd =rcu_dereference_sched_domain (cpu_rq (i)->sd);if (sd) sd->llc_bytes =get_effective_llc_bytes (i, sdp); } -unlock:sched_domains_mutex_unlock (); }
Quoted 1 line
base-commit:
50d05c7c76c96b90462f24debacca971d2e86713
I have reproduced this issue on an AMD Ryzen 8945HX machine, which has 2 LLCs and 8 cores/LLC. It is also reproduced on a Xeon platform with 4 LLCs per node. With the patch applied, sd->llc_bytes is back to normal. This patch looks reasonable to me.
thanks,
Chenyu
Hi Chenyu,
Agreed. I've restored the original boot-time shared_cpu_map explanation and kept the CPU-offline and cpuset-partition rationale alongside it.
Thanks for the review and for testing the fix on both the Ryzen 8945HX and Xeon systems. I've included your Reviewed-by tag and noted those results in v2. There are no functional changes from v1.
Thanks,
Davi
PATCH v2 1/1 sched/cache: Refresh LLC capacity across CPU hotplug
In reply to Davi Chaves AzevedoThe scheduler scales LLC capacity by the fraction of cache-sharing CPUs covered by a domain:
llc_bytes = cache_size * span_weight / shared_weight
During CPU teardown, sched_cpu_deactivate() rebuilds scheduler domains before cacheinfo_cpu_pre_down() removes the CPU from shared_cpu_map. The new domains therefore use the old sharing weight. The later call to sched_update_llc_bytes() looks up the departing CPU's sd_llc, which has already been detached, and returns without correcting the surviving CPUs.
On a Ryzen 5 7535U with twelve logical CPUs sharing a 16 MiB LLC, offlining one SMT sibling left the remaining CPUs with:
llc_bytes = floor(16777216 * 11 / 12) = 15379114 bytes
The correct capacity is still 16777216 bytes. On systems with active cache-aware scheduling, an underestimated capacity can cause exceed_llc_capacity() to reject aggregation for a process whose footprint would fit. Unchanged cpuset partitions sharing the physical cache can also retain stale capacity when a CPU comes online in another partition.
Pass the cache-sharing mask already retained by cacheinfo to the scheduler update. Refresh every surviving CPU using its own LLC domain so that each partition receives the correct share. This also preserves the correction needed as cache-sharing maps grow during boot.
Keep the existing CPU-hotplug and scheduler-domain synchronization. The update remains on the hotplug path; no steady-state scheduling operation or persistent allocation is added.
---
Changes in v2:
- Restore the original boot-time shared_cpu_map explanation, as Chen Yu suggested, alongside the CPU-offline and cpuset-partition rationale. No functional changes from v1.
- Add Chen Yu's Reviewed-by tag and document his multi-LLC testing.
v1:
https://lore.kernel.org/r/[email protected]
Review:
https://lore.kernel.org/r/[email protected]
The issue was identified by tracing the scheduler/cacheinfo teardown ordering, then checking live llc_bytes values using the running kernel's BTF layout and /proc/kcore.
Local validation performed for v1 (no functional changes in v2):
- Reproduced the stale value on 7.2.3-arch1-3 on the Ryzen system above. The patched kernel retained 16777216 bytes on every surviving CPU.
- Ten SMT-thread and ten whole-core hotplug cycles passed on the patched kernel, including capacity checks after each removal and restoration. The existing limited CPU-hotplug selftest also passed.
- Source-level state fixtures: five failures in eight scenarios before the fix, eight passes after it. These cover partition changes, unequal spans, sparse CPU IDs and boot-time map growth, but not concurrency.
- Full x86-64 baseline and patched bzImage/modules builds passed with matched configs apart from LOCALVERSION. Focused ARM64, x86 without CONFIG_SCHED_CACHE, and x86 UP builds also passed.
Thanks to Chen Yu <[email protected]> for the additional verification and review. He reproduced the issue and confirmed that v1 restored the expected sd->llc_bytes on:
- AMD Ryzen 8945HX: two LLCs, eight cores per LLC.
- Xeon: four LLCs per node.
The local tests used a single-LLC host; Chen Yu reported the multi-LLC results above. These checks validate LLC accounting.
Builds and hotplug tests were not rerun for this comment revision.
drivers/base/cacheinfo.c | 11 ++++++----- include/linux/sched/topology.h | 4 ++-- kernel/sched/topology.c | 22 +++++++++++++--------- 3 files changed, 21 insertions(+), 16 deletions(-)
drivers/base/cacheinfo.c +6 -5
@@ -1040,9 +1040,10 @@ static int cacheinfo_cpu_online(unsigned int cpu) rc =cache_add_dev (cpu);if (rc) goto err; -if (cpu_map_shared_cache (true, cpu, &cpu_map)) +if (cpu_map_shared_cache (true, cpu, &cpu_map)) {update_per_cpu_data_slice_size (true, cpu, cpu_map); -sched_update_llc_bytes (cpu); +sched_update_llc_bytes (cpu_map); + }return 0 ; err:free_cache_attributes (cpu);
@@ -1059,10 +1060,10 @@ static int cacheinfo_cpu_pre_down(unsigned int cpu)cpu_cache_sysfs_exit (cpu);free_cache_attributes (cpu); -if (nr_shared >1 ) +if (nr_shared >1 ) {update_per_cpu_data_slice_size (false, cpu, cpu_map); - -sched_update_llc_bytes (cpu); +sched_update_llc_bytes (cpu_map); + }return 0 ; }
include/linux/sched/topology.h +2 -2
@@ -281,9 +281,9 @@ static inline int task_node(const struct task_struct *p) }#ifdef CONFIG_SCHED_CACHE -extern void sched_update_llc_bytes (unsigned int cpu); +extern void sched_update_llc_bytes (const struct cpumask *cpus);#else -static inline void sched_update_llc_bytes (unsigned int cpu) { } +static inline void sched_update_llc_bytes (const struct cpumask *cpus) { }#endif #endif /* _LINUX_SCHED_TOPOLOGY_H */
kernel/sched/topology.c +13 -9
@@ -985,8 +985,8 @@ void sched_cache_active_set(void) }/* -* Update the bottom sched_domain's llc_bytes for @cpu and all its -* LLC siblings. Called from cacheinfo_cpu_online() or +* Update the bottom sched_domain's llc_bytes for @cpus sharing a physical +* LLC. Called from cacheinfo_cpu_online() or * cacheinfo_cpu_pre_down() with cpu hotplug lock held. * * Note: get_effective_llc_bytes() returns 0 on PowerPC.
@@ -996,17 +996,13 @@ void sched_cache_active_set(void)* and does not populates the per-CPU struct cpu_cacheinfo array * that get_cpu_cacheinfo_llc() reads. */ -void sched_update_llc_bytes (unsigned int cpu) +void sched_update_llc_bytes (const struct cpumask *cpus) {struct sched_domain *sd, *sdp;unsigned int i;sched_domains_mutex_lock (); - sdp =rcu_dereference_sched_domain (per_cpu (sd_llc, cpu)); -if (!sdp) - goto unlock; -/* * ci->shared_cpu_map is built incrementally as CPUs come * online, so the first CPU in an LLC initially sees
@@ -1014,14 +1010,22 @@ void sched_update_llc_bytes(unsigned int cpu)* get_effective_llc_bytes(). Re-evaluating every LLC * sibling on each online event corrects this once the full * shared_cpu_map is known. +* +* The departing CPU's domains have already been detached when +* cacheinfo removes it. Use the surviving cache siblings instead. +* They may belong to different cpuset partitions, so use each CPU's +* own LLC domain to scale its share of the physical cache. */ -for_each_cpu (i,sched_domain_span (sdp)) { +for_each_cpu (i, cpus) { + sdp =rcu_dereference_sched_domain (per_cpu (sd_llc, i)); +if (!sdp) +continue ; + sd =rcu_dereference_sched_domain (cpu_rq (i)->sd);if (sd) sd->llc_bytes =get_effective_llc_bytes (i, sdp); } -unlock:sched_domains_mutex_unlock (); }
base-commit: 50d05c7c76c96b90462f24debacca971d2e86713
Quoted 22 lines · Davi Chaves Azevedo
The scheduler scales LLC capacity by the fraction of cache-sharing CPUs covered by a domain:
llc_bytes=cache_size*span_weight/shared_weightDuring CPU teardown,
sched_cpu_deactivate()rebuilds scheduler domains beforecacheinfo_cpu_pre_down()removes the CPU fromshared_cpu_map. The new domains therefore use the old sharing weight. The later call tosched_update_llc_bytes()looks up the departing CPU'ssd_llc, which has already been detached, and returns without correcting the surviving CPUs.On a Ryzen 5 7535U with twelve logical CPUs sharing a 16 MiB LLC, offlining one SMT sibling left the remaining CPUs with:
llc_bytes=floor(16777216 * 11 / 12)= 15379114 bytesThe correct capacity is still 16777216 bytes. On systems with active cache-aware scheduling, an underestimated capacity can cause
exceed_llc_capacity()to reject aggregation for a process whose footprint would fit. Unchanged cpuset partitions sharing the physical cache can also retain stale capacity when a CPU comes online in another partition.
Quoted 9 lines
Pass the cache-sharing mask already retained by cacheinfo to the scheduler update. Refresh every surviving CPU using its own LLC domain so that each partition receives the correct share. This also preserves the correction needed as cache-sharing maps grow during boot.
Keep the existing CPU-hotplug and scheduler-domain synchronization. The update remains on the hotplug path; no steady-state scheduling operation or persistent allocation is added.
Thanks. The patch looks good to me.
Tim
Quoted 154 lines
Fixes 7030513a0877 ("sched/cache: Calculate the LLC size and store it in sched_domain")Signed-off-by Davi Chaves Azevedo <[email protected]>Reviewed-by Chen Yu <[email protected]>---
Changes in v2:
- Restore the original boot-timeshared_cpu_mapexplanation, as Chen Yu suggested, alongside the CPU-offline and cpuset-partition rationale. No functional changes from v1.
- Add Chen Yu's Reviewed-by tag and document his multi-LLC testing.v1:
https://lore.kernel.org/r/[email protected]
Review:
https://lore.kernel.org/r/[email protected]The issue was identified by tracing the scheduler/cacheinfo teardown ordering, then checking live
llc_bytesvalues using the running kernel's BTF layout and/proc/kcore.Local validation performed for v1 (no functional changes in v2):
- Reproduced the stale value on 7.2.3-arch1-3 on the Ryzen system above. The patched kernel retained 16777216 bytes on every surviving CPU.
- Ten SMT-thread and ten whole-core hotplug cycles passed on the patched kernel, including capacity checks after each removal and restoration. The existing limited CPU-hotplug selftest also passed.
- Source-level state fixtures: five failures in eight scenarios before the fix, eight passes after it. These cover partition changes, unequal spans, sparse CPU IDs and boot-time map growth, but not concurrency.
- Full x86-64 baseline and patched bzImage/modules builds passed with matched configs apart from LOCALVERSION. Focused ARM64, x86 withoutCONFIG_SCHED_CACHE, and x86 UP builds also passed.Thanks to Chen Yu <[email protected]> for the additional verification and review. He reproduced the issue and confirmed that v1 restored the expected
sd->llc_byteson:
- AMD Ryzen 8945HX: two LLCs, eight cores per LLC.
- Xeon: four LLCs per node.The local tests used a single-LLC host; Chen Yu reported the multi-LLC results above. These checks validate LLC accounting.
Builds and hotplug tests were not rerun for this comment revision.drivers/base/cacheinfo.c | 11 ++++++----- include/linux/sched/topology.h | 4 ++-- kernel/sched/topology.c | 22 +++++++++++++--------- 3 files changed, 21 insertions(+), 16 deletions(-)drivers/base/cacheinfo.c +6
-5@@ -1040,9 +1040,10 @@ static int cacheinfo_cpu_online(unsigned int cpu) rc =cache_add_dev (cpu);if (rc) goto err; -if (cpu_map_shared_cache (true, cpu, &cpu_map)) +if (cpu_map_shared_cache (true, cpu, &cpu_map)) {update_per_cpu_data_slice_size (true, cpu, cpu_map); -sched_update_llc_bytes (cpu); +sched_update_llc_bytes (cpu_map); + }return 0 ; err:free_cache_attributes (cpu);@@ -1059,10 +1060,10 @@ static int cacheinfo_cpu_pre_down(unsigned int cpu)cpu_cache_sysfs_exit (cpu);free_cache_attributes (cpu); -if (nr_shared >1 ) +if (nr_shared >1 ) {update_per_cpu_data_slice_size (false, cpu, cpu_map); - -sched_update_llc_bytes (cpu); +sched_update_llc_bytes (cpu_map); + }return 0 ; }include/linux/sched/topology.h +2
-2@@ -281,9 +281,9 @@ static inline int task_node(const struct task_struct *p) }#ifdef CONFIG_SCHED_CACHE -extern void sched_update_llc_bytes (unsigned int cpu); +extern void sched_update_llc_bytes (const struct cpumask *cpus);#else -static inline void sched_update_llc_bytes (unsigned int cpu) { } +static inline void sched_update_llc_bytes (const struct cpumask *cpus) { }#endif #endif /* _LINUX_SCHED_TOPOLOGY_H */ kernel/sched/topology.c +13
-9@@ -985,8 +985,8 @@ void sched_cache_active_set(void) }/* -* Update the bottom sched_domain's llc_bytes for @cpu and all its -* LLC siblings. Called from cacheinfo_cpu_online() or +* Update the bottom sched_domain's llc_bytes for @cpus sharing a physical +* LLC. Called from cacheinfo_cpu_online() or * cacheinfo_cpu_pre_down() with cpu hotplug lock held. * * Note: get_effective_llc_bytes() returns 0 on PowerPC. @@ -996,17 +996,13 @@ void sched_cache_active_set(void)* and does not populates the per-CPU struct cpu_cacheinfo array * that get_cpu_cacheinfo_llc() reads. */ -void sched_update_llc_bytes (unsigned int cpu) +void sched_update_llc_bytes (const struct cpumask *cpus) {struct sched_domain *sd, *sdp;unsigned int i;sched_domains_mutex_lock (); - sdp =rcu_dereference_sched_domain (per_cpu (sd_llc, cpu)); -if (!sdp) - goto unlock; -/* * ci->shared_cpu_map is built incrementally as CPUs come * online, so the first CPU in an LLC initially sees @@ -1014,14 +1010,22 @@ void sched_update_llc_bytes(unsigned int cpu)* get_effective_llc_bytes(). Re-evaluating every LLC * sibling on each online event corrects this once the full * shared_cpu_map is known. +* +* The departing CPU's domains have already been detached when +* cacheinfo removes it. Use the surviving cache siblings instead. +* They may belong to different cpuset partitions, so use each CPU's +* own LLC domain to scale its share of the physical cache. */ -for_each_cpu (i,sched_domain_span (sdp)) { +for_each_cpu (i, cpus) { + sdp =rcu_dereference_sched_domain (per_cpu (sd_llc, i)); +if (!sdp) +continue ; + sd =rcu_dereference_sched_domain (cpu_rq (i)->sd);if (sd) sd->llc_bytes =get_effective_llc_bytes (i, sdp); } -unlock:sched_domains_mutex_unlock (); }base-commit:
50d05c7c76c96b90462f24debacca971d2e86713
Hello Davi,
Quoted 27 lines · Davi Chaves Azevedo
The scheduler scales LLC capacity by the fraction of cache-sharing CPUs covered by a domain:
llc_bytes=cache_size*span_weight/shared_weightDuring CPU teardown,
sched_cpu_deactivate()rebuilds scheduler domains beforecacheinfo_cpu_pre_down()removes the CPU fromshared_cpu_map. The new domains therefore use the old sharing weight. The later call tosched_update_llc_bytes()looks up the departing CPU'ssd_llc, which has already been detached, and returns without correcting the surviving CPUs.On a Ryzen 5 7535U with twelve logical CPUs sharing a 16 MiB LLC, offlining one SMT sibling left the remaining CPUs with:
llc_bytes=floor(16777216 * 11 / 12)= 15379114 bytesThe correct capacity is still 16777216 bytes. On systems with active cache-aware scheduling, an underestimated capacity can cause
exceed_llc_capacity()to reject aggregation for a process whose footprint would fit. Unchanged cpuset partitions sharing the physical cache can also retain stale capacity when a CPU comes online in another partition.Pass the cache-sharing mask already retained by cacheinfo to the scheduler update. Refresh every surviving CPU using its own LLC domain so that each partition receives the correct share. This also preserves the correction needed as cache-sharing maps grow during boot.
Quoted 7 lines
Keep the existing CPU-hotplug and scheduler-domain synchronization. The update remains on the hotplug path; no steady-state scheduling operation or persistent allocation is added.
Fixes 7030513a0877 ("sched/cache: Calculate the LLC size and store it in sched_domain")Signed-off-by Davi Chaves Azevedo <[email protected]>Reviewed-by Chen Yu <[email protected]>
Feel free to include:
Signature · 2 lines
Thanks and Regards, Prateek
Hi Prateek,
Thanks for the review and testing. I've added both tags to my local patch, with no code changes.
Tim, may I include your Reviewed-by? Chenyu, may I include your Tested-by for the Ryzen and Xeon checks you reported too?
Thanks,
Davi
Quoted 7 lines · Davi Chaves Azevedo
Hi Prateek,
Thanks for the review and testing. I've added both tags to my local patch, with no code changes.
Tim, may I include your Reviewed-by?
Yes, you can add my Reviewed-by.
Tim
Quoted 5 lines
Chenyu, may I include your Tested-by
for the Ryzen and Xeon checks you reported too?Thanks,
Davi
Chen, Yu C
Tested-byQuoted 6 lines · Davi Chaves Azevedo
Hi Prateek,
Thanks for the review and testing. I've added both tags to my local patch,
Quoted 8 lines
with no code changes.
Tim, may I include your Reviewed-by? Chenyu, may I include your Tested-by
for the Ryzen and Xeon checks you reported too?
Sure,
thanks,
Chenyu
Quoted 5 lines
Thanks,
Davi
PATCH v3 1/1 sched/cache: Refresh LLC capacity across CPU hotplug
In reply to Davi Chaves AzevedoThe scheduler scales LLC capacity by the fraction of cache-sharing CPUs covered by a domain:
llc_bytes = cache_size * span_weight / shared_weight
During CPU teardown, sched_cpu_deactivate() rebuilds scheduler domains before cacheinfo_cpu_pre_down() removes the CPU from shared_cpu_map. The new domains therefore use the old sharing weight. The later call to sched_update_llc_bytes() looks up the departing CPU's sd_llc, which has already been detached, and returns without correcting the surviving CPUs.
On a Ryzen 5 7535U with twelve logical CPUs sharing a 16 MiB LLC, offlining one SMT sibling left the remaining CPUs with:
llc_bytes = floor(16777216 * 11 / 12) = 15379114 bytes
The correct capacity is still 16777216 bytes. On systems with active cache-aware scheduling, an underestimated capacity can cause exceed_llc_capacity() to reject aggregation for a process whose footprint would fit. Unchanged cpuset partitions sharing the physical cache can also retain stale capacity when a CPU comes online in another partition.
Pass the cache-sharing mask already retained by cacheinfo to the scheduler update. Refresh every surviving CPU using its own LLC domain so that each partition receives the correct share. This also preserves the correction needed as cache-sharing maps grow during boot.
Keep the existing CPU-hotplug and scheduler-domain synchronization. The update remains on the hotplug path; no steady-state scheduling operation or persistent allocation is added.
---
Changes in v3:
- Add K Prateek Nayak's Reviewed-by and Tested-by tags, Tim Chen's Reviewed-by, and Chen Yu's Tested-by. No functional changes from v2.
Changes in v2:
- Restore the original boot-time shared_cpu_map explanation, as Chen Yu suggested, alongside the CPU-offline and cpuset-partition rationale. No functional changes from v1.
- Add Chen Yu's Reviewed-by tag and document his multi-LLC testing.
v1:
https://lore.kernel.org/r/[email protected]
v2:
https://lore.kernel.org/r/[email protected]
Review:
https://lore.kernel.org/r/[email protected]
The issue was identified by tracing the scheduler/cacheinfo teardown ordering, then checking live llc_bytes values using the running kernel's BTF layout and /proc/kcore.
Local validation performed for v1 (no functional changes in v2 or v3):
- Reproduced the stale value on 7.2.3-arch1-3 on the Ryzen system above. The patched kernel retained 16777216 bytes on every surviving CPU.
- Ten SMT-thread and ten whole-core hotplug cycles passed on the patched kernel, including capacity checks after each removal and restoration. The existing limited CPU-hotplug selftest also passed.
- Source-level state fixtures: five failures in eight scenarios before the fix, eight passes after it. These cover partition changes, unequal spans, sparse CPU IDs and boot-time map growth, but not concurrency.
- Full x86-64 baseline and patched bzImage/modules builds passed with matched configs apart from LOCALVERSION. Focused ARM64, x86 without CONFIG_SCHED_CACHE, and x86 UP builds also passed.
Thanks to Chen Yu <[email protected]> for the additional verification and review. He reproduced the issue and confirmed that v1 restored the expected sd->llc_bytes on:
- AMD Ryzen 8945HX: two LLCs, eight cores per LLC.
- Xeon: four LLCs per node.
Thanks to K Prateek Nayak <[email protected]> for reviewing and testing, and to Tim Chen <[email protected]> for the review.
The local tests used a single-LLC host; Chen Yu reported the multi-LLC results above. These checks validate LLC accounting.
Builds and hotplug tests were not rerun for this tag-only revision.
drivers/base/cacheinfo.c | 11 ++++++----- include/linux/sched/topology.h | 4 ++-- kernel/sched/topology.c | 22 +++++++++++++--------- 3 files changed, 21 insertions(+), 16 deletions(-)
drivers/base/cacheinfo.c +6 -5
@@ -1040,9 +1040,10 @@ static int cacheinfo_cpu_online(unsigned int cpu) rc =cache_add_dev (cpu);if (rc) goto err; -if (cpu_map_shared_cache (true, cpu, &cpu_map)) +if (cpu_map_shared_cache (true, cpu, &cpu_map)) {update_per_cpu_data_slice_size (true, cpu, cpu_map); -sched_update_llc_bytes (cpu); +sched_update_llc_bytes (cpu_map); + }return 0 ; err:free_cache_attributes (cpu);
@@ -1059,10 +1060,10 @@ static int cacheinfo_cpu_pre_down(unsigned int cpu)cpu_cache_sysfs_exit (cpu);free_cache_attributes (cpu); -if (nr_shared >1 ) +if (nr_shared >1 ) {update_per_cpu_data_slice_size (false, cpu, cpu_map); - -sched_update_llc_bytes (cpu); +sched_update_llc_bytes (cpu_map); + }return 0 ; }
include/linux/sched/topology.h +2 -2
@@ -281,9 +281,9 @@ static inline int task_node(const struct task_struct *p) }#ifdef CONFIG_SCHED_CACHE -extern void sched_update_llc_bytes (unsigned int cpu); +extern void sched_update_llc_bytes (const struct cpumask *cpus);#else -static inline void sched_update_llc_bytes (unsigned int cpu) { } +static inline void sched_update_llc_bytes (const struct cpumask *cpus) { }#endif #endif /* _LINUX_SCHED_TOPOLOGY_H */
kernel/sched/topology.c +13 -9
@@ -985,8 +985,8 @@ void sched_cache_active_set(void) }/* -* Update the bottom sched_domain's llc_bytes for @cpu and all its -* LLC siblings. Called from cacheinfo_cpu_online() or +* Update the bottom sched_domain's llc_bytes for @cpus sharing a physical +* LLC. Called from cacheinfo_cpu_online() or * cacheinfo_cpu_pre_down() with cpu hotplug lock held. * * Note: get_effective_llc_bytes() returns 0 on PowerPC.
@@ -996,17 +996,13 @@ void sched_cache_active_set(void)* and does not populates the per-CPU struct cpu_cacheinfo array * that get_cpu_cacheinfo_llc() reads. */ -void sched_update_llc_bytes (unsigned int cpu) +void sched_update_llc_bytes (const struct cpumask *cpus) {struct sched_domain *sd, *sdp;unsigned int i;sched_domains_mutex_lock (); - sdp =rcu_dereference_sched_domain (per_cpu (sd_llc, cpu)); -if (!sdp) - goto unlock; -/* * ci->shared_cpu_map is built incrementally as CPUs come * online, so the first CPU in an LLC initially sees
@@ -1014,14 +1010,22 @@ void sched_update_llc_bytes(unsigned int cpu)* get_effective_llc_bytes(). Re-evaluating every LLC * sibling on each online event corrects this once the full * shared_cpu_map is known. +* +* The departing CPU's domains have already been detached when +* cacheinfo removes it. Use the surviving cache siblings instead. +* They may belong to different cpuset partitions, so use each CPU's +* own LLC domain to scale its share of the physical cache. */ -for_each_cpu (i,sched_domain_span (sdp)) { +for_each_cpu (i, cpus) { + sdp =rcu_dereference_sched_domain (per_cpu (sd_llc, i)); +if (!sdp) +continue ; + sd =rcu_dereference_sched_domain (cpu_rq (i)->sd);if (sd) sd->llc_bytes =get_effective_llc_bytes (i, sdp); } -unlock:sched_domains_mutex_unlock (); }
base-commit: 50d05c7c76c96b90462f24debacca971d2e86713