Harden preferred CPUs management
Subsystems (7)
cover Harden preferred CPUs management
On top of tip sched/core.
This series hardens preferred CPU mask management against CPU hotplug and prevents competing governors from overriding each other’s selections.
This is partially a resend of [1], excluding the write access, and partially the new material.
The write access to the preferred cpumask is a subject of the ongoing discussion. I'll submit it as a follow up shortly.
[1] https://lore.kernel.org/all/[email protected]/
Yury Norov (7):
cpumask: add reset_preferred_cpumask() helper
sched: restore preferred CPUs when hotplug removes the last one
virt: steal_governor: remove the minimum elapsed-time check
virt: steal_governor: serialize sampling against CPU hotplug
virt: steal_governor: warn on unexpected preferred mask states
virt: steal_governor: restart sampling immediately after CPU hotplug
cpu: make preferred CPU policy writers mutually exclusive
Documentation/scheduler/sched-paravirt.rst | 2 +- drivers/virt/steal_governor.c | 82 +++++++++++++++------- include/linux/cpumask.h | 30 ++++++++ kernel/sched/core.c | 8 +++ lib/cpumask.c | 67 ++++++++++++++++++ 5 files changed, 164 insertions(+), 25 deletions(-)
Signature · 1 line
2.53.0
PATCH 1/7 cpumask: add reset_preferred_cpumask() helper
This series adds more flexibility to control the 'preferred' status of CPUs. As a preparation, move the steal governor's preferred mask reset into cpumask.h to make it globally available.
While there, rename it to reset_preferred_cpumask().
--- drivers/virt/steal_governor.c | 16 +++++----------- include/linux/cpumask.h | 13 +++++++++++++ 2 files changed, 18 insertions(+), 11 deletions(-)
drivers/virt/steal_governor.c +5 -11
@@ -50,15 +50,6 @@ static struct steal_governor sg_ctx = { .low_threshold = 200, /* 2% */ }; -static void restore_preferred_to_active (void ) -{ -int cpu; - -guard (cpus_read_lock)(); -for_each_cpu (cpu, cpu_active_mask) -set_cpu_preferred (cpu, true); -} -static int param_set_interval_ms (const char *val,const struct kernel_param *kp) {unsigned int interval;
@@ -252,7 +243,9 @@ static void steal_governor_loop(struct work_struct *work)* module restores to active if CPU hotplug breaks those assumptions. */ if (!preferred_cpus_valid ()) { -restore_preferred_to_active (); +guard (cpus_read_lock)(); + +reset_preferred_cpumask ();return ; }
@@ -288,7 +281,8 @@ static int __init steal_governor_init(void) static void __exit steal_governor_exit(void) {disable_delayed_work_sync (&sg_ctx.work); -restore_preferred_to_active (); +guard (cpus_read_lock)(); +reset_preferred_cpumask ();pr_info ("disabled\n" ); }
include/linux/cpumask.h +13 -0
@@ -1195,6 +1195,19 @@ void init_cpu_possible(const struct cpumask *src);#define set_cpu_preferred (cpu, preferred) do { } while (0)#endif +/** +* reset_preferred_cpumask - reset preferred CPUs to the active mask. +* +* Callers must serialize against CPU hotplug and other preferred cpumask +* writers. +*/ +static inline void reset_preferred_cpumask (void ) +{ +#ifdef CONFIG_PREFERRED_CPU +cpumask_copy (&__cpu_preferred_mask, cpu_active_mask); +#endif +} +void set_cpu_online (unsigned int cpu,bool online);void set_cpu_possible (unsigned int cpu,bool possible);
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH 1/7] cpumask: add reset_preferred_cpumask() helper
Sashiko has reviewed this patch and found no issues.
Quoted 7 lines
This series adds more flexibility to control the 'preferred' status of CPUs. As a preparation, move the steal governor's preferred mask reset into
cpumask.hto make it globally available.While there, rename it to
reset_preferred_cpumask().Signed-off-by Yury Norov <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
PATCH 2/7 sched: restore preferred CPUs when hotplug removes the last one
In reply to Yury NorovOfflining the last preferred CPU leaves cpu_preferred_mask empty. Normal fair load balancing stops, potentially leaving CPUs idle while having queued tasks.
Currently, steal governor is the only module that may modify the mask, so it takes care of it by resetting the mask to mirror the active cpumask.
The design allows alternative preferred cpumask managers; relying on loadable modules to maintain consistency of the core data structure is not worthwhile in general.
Restore all remaining active CPUs as preferred in sched_cpu_deactivate() path, without relying on the steal governor to recover.
--- kernel/sched/core.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/sched/core.c +8 -0
@@ -8723,6 +8723,14 @@ int sched_cpu_deactivate(unsigned int cpu)set_cpu_active (cpu, false); +#ifdef CONFIG_PREFERRED_CPU +if (cpumask_empty (cpu_preferred_mask)) { +pr_notice ("sched: last preferred CPU%u offlined; make all active CPUs preferred\n" , + cpu); +reset_preferred_cpumask (); + } +#endif +/* * From this point forward, this CPU will refuse to run any task that * is not: migrate_disable() or KTHREAD_IS_PER_CPU, and will actively
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH 2/7] sched: restore preferred CPUs when hotplug removes the last one
Sashiko has reviewed this patch and found no issues.
Quoted 7 lines
Offlining the last preferred CPU leaves
cpu_preferred_maskempty. Normal fair load balancing stops, potentially leaving CPUs idle while having queued tasks.Currently, steal governor is the only module that may modify the mask, so it takes care of it by resetting the mask to mirror the active cpumask.
Quoted 8 lines
The design allows alternative preferred cpumask managers; relying on loadable modules to maintain consistency of the core data structure is not worthwhile in general.
Restore all remaining active CPUs as preferred in
sched_cpu_deactivate()path, without relying on the steal governor to recover.Signed-off-by Yury Norov <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Shrikanth Hegde
maintainerRe: [PATCH 2/7] sched: restore preferred CPUs when hotplug removes the last one
In reply to Yury NorovHi Yury.
Quoted 24 lines · Yury Norov
Offlining the last preferred CPU leaves
cpu_preferred_maskempty. Normal fair load balancing stops, potentially leaving CPUs idle while having queued tasks.Currently, steal governor is the only module that may modify the mask, so it takes care of it by resetting the mask to mirror the active cpumask.
The design allows alternative preferred cpumask managers; relying on loadable modules to maintain consistency of the core data structure is not worthwhile in general.
Restore all remaining active CPUs as preferred in
sched_cpu_deactivate()path, without relying on the steal governor to recover.Signed-off-by Yury Norov <[email protected]>--- kernel/sched/core.c | 8 ++++++++ 1 file changed, 8 insertions(+)kernel/sched/core.c +0
-0
kernel/sched/core.c int sched_cpu_deactivate(unsigned int cpu) · line 8729
Quoted 7 lines
kernel/sched/core.c +4
-0@@ -8723,6 +8723,14 @@ int sched_cpu_deactivate(unsigned int cpu)set_cpu_active (cpu, false); +#ifdef CONFIG_PREFERRED_CPU +if (cpumask_empty (cpu_preferred_mask)) { +pr_notice ("sched: last preferred CPU%u offlined; make all active CPUs preferred\n" , + cpu);
pr_warn instead?
Quoted 7 lines
kernel/sched/core.c +4
-0+ reset_preferred_cpumask(); + } +#endif +/* * From this point forward, this CPU will refuse to run any task that * is not: migrate_disable() or KTHREAD_IS_PER_CPU, and will actively
PATCH 3/7 virt: steal_governor: remove the minimum elapsed-time check
In reply to Yury NorovThe governor queues its worker with a sampling interval of at least 100ms, so its one-millisecond elapsed-time check is redundant. The existing denominator clamp already prevents division by zero when calculating the steal ratio.
This series adds a hook on hotplug events. It may trigger this error path pointlessly. Just drop it.
--- drivers/virt/steal_governor.c | 6 ------ 1 file changed, 6 deletions(-)
drivers/virt/steal_governor.c +0 -6
@@ -216,11 +216,6 @@ static void steal_governor_loop(struct work_struct *work) now =ktime_get (); delta_ns =ktime_to_ns (ktime_sub (now, sg_ctx.time)); -if (unlikely (delta_ns <NSEC_PER_MSEC )) { -pr_err_ratelimited ("work scheduled too soon delta_ns: %llu\n" , delta_ns); - goto requeue_work; - } - curr_steal =get_system_steal_time (); delta_steal = curr_steal > sg_ctx.steal ? curr_steal - sg_ctx.steal :0 ; sg_ctx.steal = curr_steal;
@@ -249,7 +244,6 @@ static void steal_governor_loop(struct work_struct *work)return ; } -requeue_work:schedule_delayed_work (&sg_ctx.work, sg_ctx.delay); }
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH 3/7] virt: steal_governor: remove the minimum elapsed-time check
Sashiko has reviewed this patch and found no issues.
Quoted 9 lines
The governor queues its worker with a sampling interval of at least 100ms, so its one-millisecond elapsed-time check is redundant. The existing denominator clamp already prevents division by zero when calculating the steal ratio.
This series adds a hook on hotplug events. It may trigger this error path pointlessly. Just drop it.
Signed-off-by Yury Norov <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
PATCH 4/7 virt: steal_governor: serialize sampling against CPU hotplug
In reply to Yury NorovSteal governor is not safe against a concurrent CPU hotplug operation because it can change active and preferred cpumasks during the governor sampling.
To address the race, the governor would take the cpus_write_lock(). But that may be impractical for the loadable module.
Instead, implement the following locking policy:
- if the code takes cpus_write_lock(), it's enough to write-access the preferred cpumask;
- if the code takes cpus_read_lock(), it must also take cpu_preferred_mutex in order to modify the preferred mask.
The core scheduler and hotplug paths already modify the preferred cpumask under the cpus_write_lock(), so no change in core logic is required.
This provides a common update protocol for multiple policy drivers without requiring writers to take the broader CPU hotplug write lock.
--- drivers/virt/steal_governor.c | 17 +++++++++++++---- include/linux/cpumask.h | 11 ++++++++++- lib/cpumask.c | 5 +++++ 3 files changed, 28 insertions(+), 5 deletions(-)
drivers/virt/steal_governor.c +13 -4
@@ -23,6 +23,7 @@#include <linux/ktime.h> #include <linux/math64.h> #include <linux/module.h> +#include <linux/mutex.h> #include <linux/sched/isolation.h> #include <linux/topology.h> #include <linux/types.h>
@@ -139,7 +140,8 @@ static void decrease_preferred_cpus(void)int target_cpu = nr_cpu_ids;int cpu; -guard (cpus_read_lock)(); +lockdep_assert_cpus_held (); +lockdep_assert_held (&cpu_preferred_mutex); cpu =cpumask_first_and (housekeeping_cpumask (HK_TYPE_KERNEL_NOISE ), cpu_preferred_mask);if (cpu >= nr_cpu_ids)
@@ -179,7 +181,8 @@ static void increase_preferred_cpus(void) {int first_cpu, cpu; -guard (cpus_read_lock)(); +lockdep_assert_cpus_held (); +lockdep_assert_held (&cpu_preferred_mutex); first_cpu =cpumask_first_andnot (cpu_active_mask, cpu_preferred_mask);/* All CPUs are preferred. Nothing to increase further */
@@ -193,6 +196,9 @@ static void increase_preferred_cpus(void)static bool preferred_cpus_valid (void ) { +lockdep_assert_cpus_held (); +lockdep_assert_held (&cpu_preferred_mutex); +if (cpumask_empty (cpu_preferred_mask)) {pr_err ("empty preferred mask. stopping\n" );return false;
@@ -213,6 +219,9 @@ static void steal_governor_loop(struct work_struct *work)u64 curr_steal, delta_steal, delta_ns, steal_ratio;ktime_t now; +guard (cpus_read_lock)(); +guard (mutex)(&cpu_preferred_mutex); + now =ktime_get (); delta_ns =ktime_to_ns (ktime_sub (now, sg_ctx.time));
@@ -238,8 +247,6 @@ static void steal_governor_loop(struct work_struct *work)* module restores to active if CPU hotplug breaks those assumptions. */ if (!preferred_cpus_valid ()) { -guard (cpus_read_lock)(); -reset_preferred_cpumask ();return ; }
@@ -276,6 +283,8 @@ static void __exit steal_governor_exit(void) {disable_delayed_work_sync (&sg_ctx.work);guard (cpus_read_lock)(); +guard (mutex)(&cpu_preferred_mutex); +reset_preferred_cpumask ();pr_info ("disabled\n" ); }
include/linux/cpumask.h +10 -1
@@ -124,6 +124,14 @@ extern struct cpumask __cpu_dying_mask;#ifdef CONFIG_PREFERRED_CPU extern struct cpumask __cpu_preferred_mask; + +/* +* Drivers must take cpus_read_lock(), then cpu_preferred_mutex and +* hold both across mask validation and updates. If cpus_write_lock() +* is held, it guarantees the exclusive access to the mask, thus +* holding cpu_preferred_mutex isn't necessary. +*/ +extern struct mutex cpu_preferred_mutex;#else #define __cpu_preferred_mask __cpu_active_mask#endif
@@ -1199,7 +1207,8 @@ void init_cpu_possible(const struct cpumask *src);* reset_preferred_cpumask - reset preferred CPUs to the active mask. * * Callers must serialize against CPU hotplug and other preferred cpumask -* writers. +* writers. Requires cpus_read_lock() and cpu_preferred_mutex; or +* cpus_write_lock(). */ static inline void reset_preferred_cpumask (void ) {
lib/cpumask.c +5 -0
@@ -166,3 +166,8 @@ unsigned int cpumask_any_distribute(const struct cpumask *srcp)return next; }EXPORT_SYMBOL (cpumask_any_distribute); + +#ifdef CONFIG_PREFERRED_CPU +DEFINE_MUTEX (cpu_preferred_mutex); +EXPORT_SYMBOL_GPL (cpu_preferred_mutex); +#endif
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH 4/7] virt: steal_governor: serialize sampling against CPU hotplug
Sashiko has reviewed this patch and found no issues.
Quoted 14 lines
Steal governor is not safe against a concurrent CPU hotplug operation because it can change active and preferred cpumasks during the governor sampling.
To address the race, the governor would take the
cpus_write_lock(). But that may be impractical for the loadable module.Instead, implement the following locking policy:
- if the code takes
cpus_write_lock(), it's enough to write-access the preferred cpumask;
- if the code takescpus_read_lock(), it must also takecpu_preferred_mutexin order to modify the preferred mask.
Quoted 8 lines
The core scheduler and hotplug paths already modify the preferred cpumask under the
cpus_write_lock(), so no change in core logic is required.This provides a common update protocol for multiple policy drivers without requiring writers to take the broader CPU hotplug write lock.
Fixes 27d47ebce4d6 ("virt/steal_governor: Implement steal_governor policy loop")Signed-off-by Yury Norov <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Shrikanth Hegde
maintainerRe: [PATCH 4/7] virt: steal_governor: serialize sampling against CPU hotplug
In reply to Yury NorovQuoted 11 lines · Yury Norov
Steal governor is not safe against a concurrent CPU hotplug operation because it can change active and preferred cpumasks during the governor sampling.
To address the race, the governor would take the
cpus_write_lock(). But that may be impractical for the loadable module.Instead, implement the following locking policy:
- if the code takes
cpus_write_lock(), it's enough to write-accessthe preferred cpumask;
Quoted 9 lines
- if the code takes
cpus_read_lock(), it must also takecpu_preferred_mutexin order to modify the preferred mask.The core scheduler and hotplug paths already modify the preferred cpumask under the
cpus_write_lock(), so no change in core logic is required.This provides a common update protocol for multiple policy drivers without requiring writers to take the broader CPU hotplug write lock.
This is also necessary if there are multiple writers outside of core hotplug No? So should it go with the write interface series?
Quoted 120 lines
Fixes 27d47ebce4d6 ("virt/steal_governor: Implement steal_governor policy loop")Signed-off-by Yury Norov <[email protected]>--- drivers/virt/steal_governor.c | 17 +++++++++++++---- include/linux/cpumask.h | 11 ++++++++++- lib/cpumask.c | 5 +++++ 3 files changed, 28 insertions(+), 5 deletions(-)drivers/virt/steal_governor.c +13
-4@@ -23,6 +23,7 @@#include <linux/ktime.h> #include <linux/math64.h> #include <linux/module.h> +#include <linux/mutex.h> #include <linux/sched/isolation.h> #include <linux/topology.h> #include <linux/types.h> @@ -139,7 +140,8 @@ static void decrease_preferred_cpus(void)int target_cpu = nr_cpu_ids;int cpu; -guard (cpus_read_lock)(); +lockdep_assert_cpus_held (); +lockdep_assert_held (&cpu_preferred_mutex); cpu =cpumask_first_and (housekeeping_cpumask (HK_TYPE_KERNEL_NOISE ), cpu_preferred_mask);if (cpu >= nr_cpu_ids)@@ -179,7 +181,8 @@ static void increase_preferred_cpus(void) {int first_cpu, cpu; -guard (cpus_read_lock)(); +lockdep_assert_cpus_held (); +lockdep_assert_held (&cpu_preferred_mutex); first_cpu =cpumask_first_andnot (cpu_active_mask, cpu_preferred_mask);/* All CPUs are preferred. Nothing to increase further */ @@ -193,6 +196,9 @@ static void increase_preferred_cpus(void)static bool preferred_cpus_valid (void ) { +lockdep_assert_cpus_held (); +lockdep_assert_held (&cpu_preferred_mutex); +if (cpumask_empty (cpu_preferred_mask)) {pr_err ("empty preferred mask. stopping\n" );return false;@@ -213,6 +219,9 @@ static void steal_governor_loop(struct work_struct *work)u64 curr_steal, delta_steal, delta_ns, steal_ratio;ktime_t now; +guard (cpus_read_lock)(); +guard (mutex)(&cpu_preferred_mutex); + now =ktime_get (); delta_ns =ktime_to_ns (ktime_sub (now, sg_ctx.time));@@ -238,8 +247,6 @@ static void steal_governor_loop(struct work_struct *work)* module restores to active if CPU hotplug breaks those assumptions. */ if (!preferred_cpus_valid ()) { -guard (cpus_read_lock)(); -reset_preferred_cpumask ();return ; }@@ -276,6 +283,8 @@ static void __exit steal_governor_exit(void) {disable_delayed_work_sync (&sg_ctx.work);guard (cpus_read_lock)(); +guard (mutex)(&cpu_preferred_mutex); +reset_preferred_cpumask ();pr_info ("disabled\n" ); }include/linux/cpumask.h +10
-1@@ -124,6 +124,14 @@ extern struct cpumask __cpu_dying_mask;#ifdef CONFIG_PREFERRED_CPU extern struct cpumask __cpu_preferred_mask; + +/* +* Drivers must take cpus_read_lock(), then cpu_preferred_mutex and +* hold both across mask validation and updates. If cpus_write_lock() +* is held, it guarantees the exclusive access to the mask, thus +* holding cpu_preferred_mutex isn't necessary. +*/ +extern struct mutex cpu_preferred_mutex;#else #define __cpu_preferred_mask __cpu_active_mask#endif @@ -1199,7 +1207,8 @@ void init_cpu_possible(const struct cpumask *src);* reset_preferred_cpumask - reset preferred CPUs to the active mask. * * Callers must serialize against CPU hotplug and other preferred cpumask -* writers. +* writers. Requires cpus_read_lock() and cpu_preferred_mutex; or +* cpus_write_lock(). */ static inline void reset_preferred_cpumask (void ) {lib/cpumask.c +5
-0@@ -166,3 +166,8 @@ unsigned int cpumask_any_distribute(const struct cpumask *srcp)return next; }EXPORT_SYMBOL (cpumask_any_distribute); + +#ifdef CONFIG_PREFERRED_CPU +DEFINE_MUTEX (cpu_preferred_mutex); +EXPORT_SYMBOL_GPL (cpu_preferred_mutex); +#endif
PATCH 5/7 virt: steal_governor: warn on unexpected preferred mask states
In reply to Yury NorovNow that preferred cpumask access is properly serialized, and CPU hotplug restores all remaining active CPUs as preferred when the last preferred CPU gets offlined, the corresponding logic in the driver is redundant.
Instead of removing it completely, add WARN_ON()s in the corresponding paths to harden the preferred CPUs mask management and highlight that this condition is not expected during normal operation.
--- drivers/virt/steal_governor.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
drivers/virt/steal_governor.c +6 -3
@@ -142,6 +142,11 @@ static void decrease_preferred_cpus(void)lockdep_assert_cpus_held ();lockdep_assert_held (&cpu_preferred_mutex); + +/* We can't end up with no preferred CPUs under normal conditions. */ +if (WARN_ON (cpumask_empty (cpu_preferred_mask))) +return ; + cpu =cpumask_first_and (housekeeping_cpumask (HK_TYPE_KERNEL_NOISE ), cpu_preferred_mask);if (cpu >= nr_cpu_ids)
@@ -150,8 +155,6 @@ static void decrease_preferred_cpus(void)/* Always leave first housekeeping core as preferred. */ first_hk_core =topology_sibling_cpumask (cpu); cpu =cpumask_last (cpu_preferred_mask); -if (cpu >= nr_cpu_ids) -return ;/* Find the last CPU which doesn't belong to that first hk_core. */ if (!cpumask_test_cpu (cpu, first_hk_core)) {
@@ -246,7 +249,7 @@ static void steal_governor_loop(struct work_struct *work)* else: steal ratio is within bounds. Still do design checks so that * module restores to active if CPU hotplug breaks those assumptions. */ -if (!preferred_cpus_valid ()) { +if (WARN_ON (!preferred_cpus_valid ())) {reset_preferred_cpumask ();return ; }
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH 5/7] virt: steal_governor: warn on unexpected preferred mask states
Sashiko has reviewed this patch and found no issues.
Quoted 10 lines
Now that preferred cpumask access is properly serialized, and CPU hotplug restores all remaining active CPUs as preferred when the last preferred CPU gets offlined, the corresponding logic in the driver is redundant.
Instead of removing it completely, add WARN_ON()s in the corresponding paths to harden the preferred CPUs mask management and highlight that this condition is not expected during normal operation.
Signed-off-by Yury Norov <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
PATCH 6/7 virt: steal_governor: restart sampling immediately after CPU hotplug
In reply to Yury NorovCPU hotplug between samples can distort the calculated steal ratio because the governor divides accumulated steal time by the current active CPU count, even when that count changed during the sampling interval.
For example, assume a constant 3% steal rate on each CPU, with four active CPUs for 900ms followed by two active CPUs for 100ms. The accumulated steal time is:
4 * 900ms * 3% + 2 * 100ms * 3% = 108ms + 6ms = 114ms
The elapsed time is 1000ms. Using the final count of two active CPUs, the governor calculates:
114ms / (2 * 1000ms) = 5.7%
This exceeds the default 5% high threshold, even though the actual steal rate stayed at 3%. Using the CPU counts over the whole interval gives:
114ms / (4 * 900ms + 2 * 100ms) = 3%
Checking num_online_cpus() doesn't account for cases where CPUs go online and offline such that the total number of online CPUs doesn't change. So, implement the sampling restart on hotplug event.
--- drivers/virt/steal_governor.c | 37 ++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-)
drivers/virt/steal_governor.c +34 -3
@@ -14,6 +14,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt#include <linux/cleanup.h> +#include <linux/cpuhotplug.h> #include <linux/cpuhplock.h> #include <linux/cpumask.h> #include <linux/init.h>
@@ -42,6 +43,8 @@ struct steal_governor {unsigned long delay;unsigned int high_threshold;unsigned int low_threshold; +int hp_state; +bool sample_invalidated;struct delayed_work work; };
@@ -217,6 +220,13 @@ static bool preferred_cpus_valid(void)return true; } +static int hotplug_callback (unsigned int cpu) +{ + sg_ctx.sample_invalidated = true; +mod_delayed_work (system_wq, &sg_ctx.work,0 ); +return 0 ; +} +static void steal_governor_loop (struct work_struct *work) {u64 curr_steal, delta_steal, delta_ns, steal_ratio;
@@ -233,6 +243,11 @@ static void steal_governor_loop(struct work_struct *work) sg_ctx.steal = curr_steal; sg_ctx.time = now; +if (sg_ctx.sample_invalidated) { + sg_ctx.sample_invalidated = false; + goto requeue_work; + } +/* * steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus()) * To avoid possible overflow, divide the denominator early.
@@ -254,11 +269,14 @@ static void steal_governor_loop(struct work_struct *work)return ; } -schedule_delayed_work (&sg_ctx.work, sg_ctx.delay); +requeue_work: +mod_delayed_work (system_wq, &sg_ctx.work, sg_ctx.delay); }static int __init steal_governor_init (void ) { +int ret; +#ifdef CONFIG_XEN if (xen_initial_domain ()) {pr_err ("Cannot load in Xen Dom0 (Host OS). Driver is for guests only.\n" );
@@ -273,8 +291,20 @@ static int __init steal_governor_init(void) }INIT_DELAYED_WORK (&sg_ctx.work, steal_governor_loop); - sg_ctx.steal =get_system_steal_time (); - sg_ctx.time =ktime_get (); +/* Lock to prevent unobserved hotplug event during setup. */ +scoped_guard (cpus_read_lock) { + sg_ctx.steal =get_system_steal_time (); + sg_ctx.time =ktime_get (); + sg_ctx.sample_invalidated = false; + ret =cpuhp_setup_state_nocalls_cpuslocked (CPUHP_AP_ONLINE_DYN , +"virt/steal_governor:online" , + hotplug_callback, + hotplug_callback); + } +if (ret <0 ) +return ret; + sg_ctx.hp_state = ret; +schedule_delayed_work (&sg_ctx.work, sg_ctx.delay);pr_info ("enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n" ,jiffies_to_msecs (sg_ctx.delay), sg_ctx.high_threshold, sg_ctx.low_threshold);
@@ -285,6 +315,7 @@ static int __init steal_governor_init(void) static void __exit steal_governor_exit(void) {disable_delayed_work_sync (&sg_ctx.work); +cpuhp_remove_state_nocalls (sg_ctx.hp_state);guard (cpus_read_lock)();guard (mutex)(&cpu_preferred_mutex);
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH 6/7] virt: steal_governor: restart sampling immediately after CPU hotplug
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] virt: steal_governor: deprecated system_wq use and workqueue mismatch --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] virt: steal_governor: deprecated system_wq use and workqueue mismatch
--
commit 71c885eda2b0a72a307fd87b15575e6e5a4178ed
Author: Yury Norov <[email protected]>
virt: steal_governor: restart sampling immediately after CPU hotplug
This commit registers a CPU hotplug callback to invalidate the current steal-time sample and immediately requeue the governor work when a CPU goes online or offline.
Quoted 4 lines
drivers/virt/steal_governor.c +0
-0
[ ... ]
Quoted 27 lines
drivers/virt/steal_governor.c +12
-0@@ -217,6 +220,13 @@ static bool preferred_cpus_valid(void)return true; } +static int hotplug_callback (unsigned int cpu) +{ + sg_ctx.sample_invalidated = true; +mod_delayed_work (system_wq, &sg_ctx.work,0 ); +return 0 ; +} +static void steal_governor_loop (struct work_struct *work) {u64 curr_steal, delta_steal, delta_ns, steal_ratio;@@ -233,6 +243,11 @@ static void steal_governor_loop(struct work_struct *work) sg_ctx.steal = curr_steal; sg_ctx.time = now; +if (sg_ctx.sample_invalidated) { + sg_ctx.sample_invalidated = false; + goto requeue_work; + } +/* * steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus()) * To avoid possible overflow, divide the denominator early.
drivers/virt/steal_governor.c static void steal_governor_loop(struct work_struct *work) · line 274
Quoted 7 lines
drivers/virt/steal_governor.c +2
-1@@ -254,11 +269,14 @@ static void steal_governor_loop(struct work_struct *work)return ; } -schedule_delayed_work (&sg_ctx.work, sg_ctx.delay); +requeue_work: +mod_delayed_work (system_wq, &sg_ctx.work, sg_ctx.delay); }
[Severity: Medium]
Should both mod_delayed_work() calls use system_percpu_wq instead of system_wq?
system_wq is allocated with __WQ_DEPRECATED, so queueing sg_ctx.work onto it triggers a pr_warn_once() deprecation warning in __queue_work().
In addition, steal_governor_init() queues sg_ctx.work onto system_percpu_wq via schedule_delayed_work(), which is a separate workqueue_struct from system_wq. Because __queue_work() only enforces cross-CPU non-reentrancy when worker->current_pwq->wq == wq, a hotplug event during the first execution of steal_governor_loop() can queue sg_ctx.work on a second CPU while it is still running on the first:
CPU0 (system_percpu_wq) CPU1 (hotplug)steal_governor_loop()
guard(cpus_read_lock)()
(blocks on cpus_write_lock)
hotplug_callback()
sg_ctx.sample_invalidated = true mod_delayed_work(system_wq, ..., 0)
(wq != current_pwq->wq, queues on CPU1)
cpus_write_unlock()
sg_ctx.sample_invalidated = false
...
steal_governor_loop()
guard(cpus_read_lock)() (sample_invalidated is false, delta_ns ~ 0) increase_preferred_cpus() <-- bogus sample
When the second worker runs right after the first finishes, sample_invalidated has already been cleared and delta_ns is near zero, yielding steal_ratio == 0 and calling increase_preferred_cpus() prematurely.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Re: [PATCH 6/7] virt: steal_governor: restart sampling immediately after CPU hotplug
In reply to Yury NorovQuoted 16 lines
static void
steal_governor_loop(struct work_struct *work){drivers/virt/steal_governor.c +5
-0u64 curr_steal, delta_steal, delta_ns, steal_ratio;@@ -233,6 +243,11 @@ static void steal_governor_loop(struct work_struct *work) sg_ctx.steal = curr_steal; sg_ctx.time = now; +if (sg_ctx.sample_invalidated) { + sg_ctx.sample_invalidated = false; + goto requeue_work; + } +/* * steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus()) * To avoid possible overflow, divide the denominator early.
drivers/virt/steal_governor.c static void steal_governor_loop(struct work_struct *work) · line 273
Quoted 6 lines
drivers/virt/steal_governor.c +2
-1@@ -254,11 +269,14 @@ static void steal_governor_loop(struct work_struct *work)return ; } -schedule_delayed_work (&sg_ctx.work, sg_ctx.delay); +requeue_work: +mod_delayed_work (system_wq, &sg_ctx.work, sg_ctx.delay);
Sashiko has correctly pointed it should be system_percpu_wq.
Quoted 1 line
(quoted diff) +0
-0}
Shrikanth Hegde
maintainerRe: [PATCH 6/7] virt: steal_governor: restart sampling immediately after CPU hotplug
In reply to Yury NorovHi Yury.
Quoted 9 lines · Yury Norov
CPU hotplug between samples can distort the calculated steal ratio because the governor divides accumulated steal time by the current active CPU count, even when that count changed during the sampling interval.
For example, assume a constant 3% steal rate on each CPU, with four active CPUs for 900ms followed by two active CPUs for 100ms. The accumulated steal time is:
4 * 900ms * 3% + 2 * 100ms * 3% = 108ms + 6ms = 114ms
This example is not realistic. If you had 3% steal with 4 CPUs, and if they become 2, your steal time will definitely increase. It will likely double. So governor likely calculates 6%.
But yes, you can have realistic examples, where governor may calculate the values which goes slightly above or below. But IMHO those examples are very limited. Even in the above examples, the steal governor was supposed to trigger high threshold since the steal time with 2 CPUs is 6%. So if not now, in the next delta it would have triggered it anyways.
Some examples are really corner cases. Maybe like
60 CPUs 3% steal, and then 40 CPUs 5.1% steal = 4.55%
(Below threshold, but should have been high). One interval missed. But if you see large duration was in low threshold, so it is agreeable to miss it.
Quoted 6 lines
The elapsed time is 1000ms. Using the final count of two active CPUs, the governor calculates:
114ms / (2 * 1000ms) = 5.7%
Quoted 9 lines
This exceeds the default 5% high threshold, even though the actual steal rate stayed at 3%. Using the CPU counts over the whole interval gives:
114ms / (4 * 900ms + 2 * 100ms) = 3%Checking
num_online_cpus()doesn't account for cases where CPUs go online and offline such that the total number of online CPUs doesn't change. So, implement the sampling restart on hotplug event.
I am not sure if we really need to stress about hotplug cases. IIUC, when a user does hotplug, systems are usually idle. (Atleast that is what we have observed in real life use cases). And hotplug is really user driven, so user knows system is going through some transitions and will be stable soon.
Quoted 100 lines
Fixes 27d47ebce4d6 ("virt/steal_governor: Implement steal_governor policy loop")Signed-off-by Yury Norov <[email protected]>--- drivers/virt/steal_governor.c | 37 ++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-)drivers/virt/steal_governor.c +34
-3@@ -14,6 +14,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt#include <linux/cleanup.h> +#include <linux/cpuhotplug.h> #include <linux/cpuhplock.h> #include <linux/cpumask.h> #include <linux/init.h> @@ -42,6 +43,8 @@ struct steal_governor {unsigned long delay;unsigned int high_threshold;unsigned int low_threshold; +int hp_state; +bool sample_invalidated;struct delayed_work work; };@@ -217,6 +220,13 @@ static bool preferred_cpus_valid(void)return true; } +static int hotplug_callback (unsigned int cpu) +{ + sg_ctx.sample_invalidated = true; +mod_delayed_work (system_wq, &sg_ctx.work,0 ); +return 0 ; +} +static void steal_governor_loop (struct work_struct *work) {u64 curr_steal, delta_steal, delta_ns, steal_ratio;@@ -233,6 +243,11 @@ static void steal_governor_loop(struct work_struct *work) sg_ctx.steal = curr_steal; sg_ctx.time = now; +if (sg_ctx.sample_invalidated) { + sg_ctx.sample_invalidated = false; + goto requeue_work; + } +/* * steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus()) * To avoid possible overflow, divide the denominator early. @@ -254,11 +269,14 @@ static void steal_governor_loop(struct work_struct *work)return ; } -schedule_delayed_work (&sg_ctx.work, sg_ctx.delay); +requeue_work: +mod_delayed_work (system_wq, &sg_ctx.work, sg_ctx.delay); }static int __init steal_governor_init (void ) { +int ret; +#ifdef CONFIG_XEN if (xen_initial_domain ()) {pr_err ("Cannot load in Xen Dom0 (Host OS). Driver is for guests only.\n" );@@ -273,8 +291,20 @@ static int __init steal_governor_init(void) }INIT_DELAYED_WORK (&sg_ctx.work, steal_governor_loop); - sg_ctx.steal =get_system_steal_time (); - sg_ctx.time =ktime_get (); +/* Lock to prevent unobserved hotplug event during setup. */ +scoped_guard (cpus_read_lock) { + sg_ctx.steal =get_system_steal_time (); + sg_ctx.time =ktime_get (); + sg_ctx.sample_invalidated = false; + ret =cpuhp_setup_state_nocalls_cpuslocked (CPUHP_AP_ONLINE_DYN , +"virt/steal_governor:online" , + hotplug_callback, + hotplug_callback); + } +if (ret <0 ) +return ret; + sg_ctx.hp_state = ret; +schedule_delayed_work (&sg_ctx.work, sg_ctx.delay);pr_info ("enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n" ,jiffies_to_msecs (sg_ctx.delay), sg_ctx.high_threshold, sg_ctx.low_threshold);@@ -285,6 +315,7 @@ static int __init steal_governor_init(void) static void __exit steal_governor_exit(void) {disable_delayed_work_sync (&sg_ctx.work); +cpuhp_remove_state_nocalls (sg_ctx.hp_state);guard (cpus_read_lock)();guard (mutex)(&cpu_preferred_mutex);
PATCH 7/7 cpu: make preferred CPU policy writers mutually exclusive
In reply to Yury NorovSerializing preferred mask updates still lets competing policy drivers repeatedly override each other's selections.
Disallow multiple policy drivers running at the same time with the exclusive ownership rule.
--- Documentation/scheduler/sched-paravirt.rst | 2 +- drivers/virt/steal_governor.c | 13 +++-- include/linux/cpumask.h | 8 +++ lib/cpumask.c | 62 ++++++++++++++++++++++ 4 files changed, 79 insertions(+), 6 deletions(-)
Documentation/scheduler/sched-paravirt.rst +1 -1
@@ -48,7 +48,7 @@ Notes: makes decisions should enable it. For example, steal_governor driver (``CONFIG_STEAL_GOVERNOR``). On enabling the driver, CPU preferred state can change based on steal time. Without the driver, preferred CPUs is - same as active CPUs. + same as active CPUs. Only one governor may be enabled at a time. 2. This feature works for the FAIR class only.
drivers/virt/steal_governor.c +8 -5
@@ -291,6 +291,10 @@ static int __init steal_governor_init(void) }INIT_DELAYED_WORK (&sg_ctx.work, steal_governor_loop); + ret =claim_preferred_cpumask (&sg_ctx); +if (ret) +return ret; +/* Lock to prevent unobserved hotplug event during setup. */ scoped_guard (cpus_read_lock) { sg_ctx.steal =get_system_steal_time ();
@@ -301,8 +305,10 @@ static int __init steal_governor_init(void) hotplug_callback, hotplug_callback); } -if (ret <0 ) +if (ret <0 ) { +release_preferred_cpumask (&sg_ctx);return ret; + } sg_ctx.hp_state = ret;schedule_delayed_work (&sg_ctx.work, sg_ctx.delay);
@@ -316,10 +322,7 @@ static void __exit steal_governor_exit(void) {disable_delayed_work_sync (&sg_ctx.work);cpuhp_remove_state_nocalls (sg_ctx.hp_state); -guard (cpus_read_lock)(); -guard (mutex)(&cpu_preferred_mutex); - -reset_preferred_cpumask (); +release_preferred_cpumask (&sg_ctx);pr_info ("disabled\n" ); }
include/linux/cpumask.h +8 -0
@@ -132,6 +132,14 @@ extern struct cpumask __cpu_preferred_mask;* holding cpu_preferred_mutex isn't necessary. */ extern struct mutex cpu_preferred_mutex; + +/* +* Drivers must claim exclusive ownership of cpu_preferred_mask +* for the whole period of operation. +*/ +int claim_preferred_cpumask (const void *owner); +void release_preferred_cpumask (const void *owner); +bool preferred_cpumask_owned_by (const void *owner);#else #define __cpu_preferred_mask __cpu_active_mask#endif
lib/cpumask.c +62 -0
74 lines not on this page · raw message
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH 7/7] cpu: make preferred CPU policy writers mutually exclusive
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] cpumask: preferred_cpumask_owned_by(NULL) returns true when unclaimed --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] cpumask: preferred_cpumask_owned_by(NULL) returns true when unclaimed
--
commit 62477968d02f2aaa9afd9c7a8010ab357c98eb83
Author: Yury Norov <[email protected]>
cpu: make preferred CPU policy writers mutually exclusive
This commit adds helpers to claim, release, and check exclusive ownership of the preferred CPU mask, and updates steal_governor to use them.
Quoted 4 lines
lib/cpumask.c +0
-0
[ ... ]
Quoted 49 lines
lib/cpumask.c +45
-0@@ -170,4 +172,64 @@ EXPORT_SYMBOL(cpumask_any_distribute);#ifdef CONFIG_PREFERRED_CPU DEFINE_MUTEX (cpu_preferred_mutex);EXPORT_SYMBOL_GPL (cpu_preferred_mutex); + +static const void *preferred_cpumask_owner; + +/** +* preferred_cpumask_owned_by - test preferred CPU policy ownership +* @owner: policy identity +* +* Return: true if @owner holds the policy, false otherwise. +*/ +bool preferred_cpumask_owned_by (const void *owner) +{ +lockdep_assert_held (&cpu_preferred_mutex); + +return preferred_cpumask_owner == owner; +} +EXPORT_SYMBOL_GPL (preferred_cpumask_owned_by); + +/** +* claim_preferred_cpumask - acquire exclusive preferred CPU policy ownership +* @owner: non-NULL identity that remains valid until ownership is released +* +* Call before starting policy work, normally on module init. +* +* Return: 0 on success, -EBUSY if claimed, or -EINVAL for a NULL identity. +*/ +int claim_preferred_cpumask (const void *owner) +{ +if (!owner) +return -EINVAL ; + +guard (cpus_read_lock)(); +guard (mutex)(&cpu_preferred_mutex); + +if (preferred_cpumask_owner) +return -EBUSY ; + + preferred_cpumask_owner = owner; +reset_preferred_cpumask (); +return 0 ; +} +EXPORT_SYMBOL_GPL (claim_preferred_cpumask); + +/** +* release_preferred_cpumask - release exclusive preferred CPU policy ownership +* @owner: identity used to claim ownership
lib/cpumask.c EXPORT_SYMBOL(cpumask_any_distribute); · line 229
Quoted 10 lines
lib/cpumask.c +10
-0+* +* Call after stopping policy work, normally on module exit. +*/ +void release_preferred_cpumask (const void *owner) +{ +guard (cpus_read_lock)(); +guard (mutex)(&cpu_preferred_mutex); + +if (WARN_ON_ONCE (!owner || !preferred_cpumask_owned_by (owner))) +return ;
[Severity: Low]
Does preferred_cpumask_owned_by(NULL) return true when no policy driver owns the preferred cpumask? Checking owner && preferred_cpumask_owner == owner in preferred_cpumask_owned_by() would also avoid needing the separate !owner check in release_preferred_cpumask().
Also, the kernel-doc comments for preferred_cpumask_owned_by(), claim_preferred_cpumask(), and release_preferred_cpumask() do not mention their locking expectations (holding cpu_preferred_mutex for preferred_cpumask_owned_by(), and acquiring cpus_read_lock() and cpu_preferred_mutex internally in the claim/release helpers).
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Shrikanth Hegde
maintainerRe: [PATCH 7/7] cpu: make preferred CPU policy writers mutually exclusive
In reply to Yury NorovHi Yury.
Quoted 6 lines · Yury Norov
Serializing preferred mask updates still lets competing policy drivers repeatedly override each other's selections.
Disallow multiple policy drivers running at the same time with the exclusive ownership rule.
This patch should go with the write interface patch series IMHO. There are no new writers/governors with this series.
Quoted 160 lines
Suggested-by Peter Zijlstra (Intel) <[email protected]>Signed-off-by Yury Norov <[email protected]>--- Documentation/scheduler/sched-paravirt.rst | 2 +- drivers/virt/steal_governor.c | 13 +++-- include/linux/cpumask.h | 8 +++ lib/cpumask.c | 62 ++++++++++++++++++++++ 4 files changed, 79 insertions(+), 6 deletions(-)Documentation/scheduler/sched-paravirt.rst +1
-1@@ -48,7 +48,7 @@ Notes: makes decisions should enable it. For example, steal_governor driver (``CONFIG_STEAL_GOVERNOR``). On enabling the driver, CPU preferred state can change based on steal time. Without the driver, preferred CPUs is - same as active CPUs. + same as active CPUs. Only one governor may be enabled at a time. 2. This feature works for the FAIR class only.drivers/virt/steal_governor.c +8
-5@@ -291,6 +291,10 @@ static int __init steal_governor_init(void) }INIT_DELAYED_WORK (&sg_ctx.work, steal_governor_loop); + ret =claim_preferred_cpumask (&sg_ctx); +if (ret) +return ret; +/* Lock to prevent unobserved hotplug event during setup. */ scoped_guard (cpus_read_lock) { sg_ctx.steal =get_system_steal_time ();@@ -301,8 +305,10 @@ static int __init steal_governor_init(void) hotplug_callback, hotplug_callback); } -if (ret <0 ) +if (ret <0 ) { +release_preferred_cpumask (&sg_ctx);return ret; + } sg_ctx.hp_state = ret;schedule_delayed_work (&sg_ctx.work, sg_ctx.delay);@@ -316,10 +322,7 @@ static void __exit steal_governor_exit(void) {disable_delayed_work_sync (&sg_ctx.work);cpuhp_remove_state_nocalls (sg_ctx.hp_state); -guard (cpus_read_lock)(); -guard (mutex)(&cpu_preferred_mutex); - -reset_preferred_cpumask (); +release_preferred_cpumask (&sg_ctx);pr_info ("disabled\n" ); }include/linux/cpumask.h +8
-0@@ -132,6 +132,14 @@ extern struct cpumask __cpu_preferred_mask;* holding cpu_preferred_mutex isn't necessary. */ extern struct mutex cpu_preferred_mutex; + +/* +* Drivers must claim exclusive ownership of cpu_preferred_mask +* for the whole period of operation. +*/ +int claim_preferred_cpumask (const void *owner); +void release_preferred_cpumask (const void *owner); +bool preferred_cpumask_owned_by (const void *owner);#else #define __cpu_preferred_mask __cpu_active_mask#endif lib/cpumask.c +62
-0@@ -3,8 +3,10 @@#include <linux/kernel.h> #include <linux/bitops.h> #include <linux/cpumask.h> +#include <linux/cpuhplock.h> #include <linux/export.h> #include <linux/memblock.h> +#include <linux/mutex.h> #include <linux/numa.h> /* These are not inline because of header tangles. */ @@ -170,4 +172,64 @@ EXPORT_SYMBOL(cpumask_any_distribute);#ifdef CONFIG_PREFERRED_CPU DEFINE_MUTEX (cpu_preferred_mutex);EXPORT_SYMBOL_GPL (cpu_preferred_mutex); + +static const void *preferred_cpumask_owner; + +/** +* preferred_cpumask_owned_by - test preferred CPU policy ownership +* @owner: policy identity +* +* Return: true if @owner holds the policy, false otherwise. +*/ +bool preferred_cpumask_owned_by (const void *owner) +{ +lockdep_assert_held (&cpu_preferred_mutex); + +return preferred_cpumask_owner == owner; +} +EXPORT_SYMBOL_GPL (preferred_cpumask_owned_by); + +/** +* claim_preferred_cpumask - acquire exclusive preferred CPU policy ownership +* @owner: non-NULL identity that remains valid until ownership is released +* +* Call before starting policy work, normally on module init. +* +* Return: 0 on success, -EBUSY if claimed, or -EINVAL for a NULL identity. +*/ +int claim_preferred_cpumask (const void *owner) +{ +if (!owner) +return -EINVAL ; + +guard (cpus_read_lock)(); +guard (mutex)(&cpu_preferred_mutex); + +if (preferred_cpumask_owner) +return -EBUSY ; + + preferred_cpumask_owner = owner; +reset_preferred_cpumask (); +return 0 ; +} +EXPORT_SYMBOL_GPL (claim_preferred_cpumask); + +/** +* release_preferred_cpumask - release exclusive preferred CPU policy ownership +* @owner: identity used to claim ownership +* +* Call after stopping policy work, normally on module exit. +*/ +void release_preferred_cpumask (const void *owner) +{ +guard (cpus_read_lock)(); +guard (mutex)(&cpu_preferred_mutex); + +if (WARN_ON_ONCE (!owner || !preferred_cpumask_owned_by (owner))) +return ; + +reset_preferred_cpumask (); + preferred_cpumask_owner =NULL ; +} +EXPORT_SYMBOL_GPL (release_preferred_cpumask);#endif