Skip to content

cgroup/cpuset: Enable runtime update of nohz_full and managed_irq CPUs

Waiman Long cgroups linux-doc linux-hwmon linux-hyperv linux-kselftest lkml netdev rcu sashiko 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest23 patches 39 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/23sched/isolation: Add HK_TYPE_KERNEL_NOISE_BOOT & HK_TYPE_MANAGED_IRQ_BOOTinclude/linux/sched/isolation.h kernel/sched/isolation.c+23 -90 replies
2/23sched/isolation: Enhance housekeeping_update() to support updating more than one HK cpumaskinclude/linux/sched/isolation.h kernel/cgroup/cpuset.c kernel/sched/isolation.c+71 -321 reply
3/23tick/nohz: Make nohz_full parameter optionalDocumentation/admin-guide/kernel-parameters.txt include/linux/context_tracking.h kernel/context_tracking.c +3 more+40 -123 replies
4/23tick/nohz: Allow runtime changes in full dynticks CPUsinclude/linux/context_tracking.h include/linux/tick.h kernel/context_tracking.c +2 more+48 -103 replies
5/23tick: Pass timer tick job to an online HK CPU in tick_cpu_dying()kernel/time/tick-common.c+12 -42 replies
6/23rcu/nocbs: Allow runtime changes in RCU NOCBS cpumaskinclude/linux/rcupdate.h kernel/rcu/tree_nocb.h kernel/sched/isolation.c+27 -11 reply
7/23watchdog: Sync up with runtime change of isolated CPUsinclude/linux/nmi.h kernel/sched/isolation.c kernel/watchdog.c+27 -01 reply
8/23arm64: topology: Use RCU to protect access to HK_TYPE_TICK cpumaskarch/arm64/kernel/topology.c+14 -32 replies
9/23workqueue: Use RCU to protect access of HK_TYPE_TIMER cpumaskkernel/workqueue.c+4 -20 replies
10/23cpu: Use RCU to protect access of HK_TYPE_TIMER cpumaskkernel/cpu.c+2 -02 replies
11/23hrtimer: Use RCU to protect access of HK_TYPE_TIMER cpumaskkernel/time/hrtimer.c+3 -11 reply
12/23net: Use boot time housekeeping cpumask settings for nownet/core/net-sysfs.c+1 -10 replies
13/23sched/core: Use RCU to protect access of HK_TYPE_KERNEL_NOISE cpumaskkernel/sched/core.c+2 -21 reply
14/23hwmon/coretemp: Use RCU to protect access of HK_TYPE_MISC cpumaskdrivers/hwmon/coretemp.c+4 -20 replies
15/23Drivers: hv: Use RCU to protect access of HK_TYPE_MANAGED_IRQ cpumaskdrivers/hv/channel_mgmt.c drivers/hv/vmbus_drv.c+14 -81 reply
16/23genirq/cpuhotplug: Use RCU to protect access of HK_TYPE_MANAGED_IRQ cpumaskkernel/irq/cpuhotplug.c kernel/irq/manage.c+2 -02 replies
17/23sched/isolation: Extend housekeeping_dereference_check() to cover changes in nohz_full or manged_irqs cpumaskskernel/sched/isolation.c+2 -11 reply
18/23cpu/hotplug: Add a new cpuhp_offline_cb() APIinclude/linux/cpuhplock.h kernel/cpu.c+79 -03 replies
19/23cgroup/cpuset: Improve check for calling housekeeping_update()kernel/cgroup/cpuset.c+20 -162 replies
20/23cgroup/cpuset: Enable runtime update of HK_TYPE_{KERNEL_NOISE,MANAGED_IRQ} cpumaskskernel/cgroup/cpuset.c+182 -170 replies
21/23cgroup/cpuset: Limit the side effect of using CPU hotplug on isolated partitionkernel/cgroup/cpuset.c+18 -00 replies
22/23cgroup/cpuset: Prevent offline_disabled CPUs from being used in isolated partitionkernel/cgroup/cpuset-internal.h kernel/cgroup/cpuset.c+88 -20 replies
23/23cgroup/cpuset: Documentation and kselftest updatesDocumentation/admin-guide/cgroup-v2.rst tools/testing/selftests/cgroup/test_cpuset_prs.sh+92 -130 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH-next 00/23] cgroup/cpuset: Enable runtime update of nohz_full and managed_irq CPUs

The "isolcpus=domain" CPU list implied by the HK_TYPE_DOMAIN housekeeping cpumask is being dynamically updated whenever the set of cpuset isolated partitions change. This patch series extends the isolated partition code to make dynamic changes to the "nohz_full" and "isolcpus=managed_irq" CPU lists. These CPU lists correspoond to equivalent changes in the HK_TYPE_KERNEL_NOISE and HK_TYPE_MANAGED_IRQ housekeeping cpumasks.

To facilitate the changing of these CPU lists, the CPU hotplug code which is doing a lot of heavy lifting is now being used. For changing the "nohz_full" and HK_TYPE_KERNEL_NOISE cpumasks, the affected CPUs are torn down into offline mode first. Then the housekeeping cpumask and the corresponding cpumasks in the RCU, tick and watchdog subsystems are modified. After that, the affected CPUs are brought online again.

It is slightly different with the "managed_irq" HK_TYPE_MANAGED_IRQ cpumask, the cpumask is updated first and the affected CPUs are torn down and brought up to move the managed interrupts away from the newly isolated CPUs.

Using CPU hotplug does have its drawback when multiple isolated partitions are being managed in a system. The CPU offline process uses the stop_machine mechanism to stop all the CPUs except the one being torn down. This will cause latency spike for isolated CPUs in other isolated partitions. It is not a problem if only one isolated partition is needed. This is an issue we need to address in the near future.

Patches 1-7 enables runtime updates to the nohz_full/managed_irq cpumasks in sched/isolation, tick, RCU and watchdog subsystems.

Patches 8-17 modifies various subsystems that need access to the HK_TYPE_MANAGED_IRQ cpumask and HK_TYPE_KERNEL_NOISE cpumask and its aliases. Like the runtime modifiable HK_TYPE_DOMAIN cpumask, RCU is used to protect access to cpumasks to avoid potential UAF problem. Patch 17 updates the housekeeping_dereference_check() to print a WARNING when lockdep is enabled if those housekeeping cpumasks are used without the proper lock protection.

Patch 18 introduces a new cpuhp_offline_cb() API that enables the shutting down of a given list of CPUs, run a callback function and then brought those CPUs up again while disallowing any concurrent CPU hotplug activity.

Patches 19-23 updates the cpuset code, selftest and documentation to allow change in isolated partition configuration to be reflected in the housekeeping and other cpumasks dynamically.

As there is a slight overhead in enabling dynamic update to the nohz_full cpumask, this new nohz_full and managed_irq runtime update feature has to be explicitly opted in by adding a nohz_full kernel command line parameter with or without a CPU list to indicate a desire to use this feature. It is also because a number of subsystems have explicit check of nohz_full at boot time to adjust their behavior which may not be easy to modify after boot.

Waiman Long (23):
sched/isolation: Add HK_TYPE_KERNEL_NOISE_BOOT &
HK_TYPE_MANAGED_IRQ_BOOT
sched/isolation: Enhance housekeeping_update() to support updating
more than one HK cpumask
tick/nohz: Make nohz_full parameter optional
tick/nohz: Allow runtime changes in full dynticks CPUs
tick: Pass timer tick job to an online HK CPU in tick_cpu_dying() rcu/nocbs: Allow runtime changes in RCU NOCBS cpumask
watchdog: Sync up with runtime change of isolated CPUs
arm64: topology: Use RCU to protect access to HK_TYPE_TICK cpumask
workqueue: Use RCU to protect access of HK_TYPE_TIMER cpumask
cpu: Use RCU to protect access of HK_TYPE_TIMER cpumask
hrtimer: Use RCU to protect access of HK_TYPE_TIMER cpumask
net: Use boot time housekeeping cpumask settings for now sched/core: Use RCU to protect access of HK_TYPE_KERNEL_NOISE cpumask hwmon/coretemp: Use RCU to protect access of HK_TYPE_MISC cpumask
Drivers: hv: Use RCU to protect access of HK_TYPE_MANAGED_IRQ cpumask genirq/cpuhotplug: Use RCU to protect access of HK_TYPE_MANAGED_IRQ
cpumask
sched/isolation: Extend housekeeping_dereference_check() to cover
changes in nohz_full or manged_irqs cpumasks
cpu/hotplug: Add a new cpuhp_offline_cb() API cgroup/cpuset: Improve check for calling housekeeping_update() cgroup/cpuset: Enable runtime update of
HK_TYPE_{KERNEL_NOISE,MANAGED_IRQ} cpumasks
cgroup/cpuset: Limit the side effect of using CPU hotplug on isolated
partition
cgroup/cpuset: Prevent offline_disabled CPUs from being used in
isolated partition
cgroup/cpuset: Documentation and kselftest updates

 Documentation/admin-guide/cgroup-v2.rst       |  35 ++-
 .../admin-guide/kernel-parameters.txt         |  15 +-
 arch/arm64/kernel/topology.c                  |  17 +-
 drivers/hv/channel_mgmt.c                     |  15 +-
 drivers/hv/vmbus_drv.c                        |   7 +-
 drivers/hwmon/coretemp.c                      |   6 +-
 include/linux/context_tracking.h              |   8 +-
 include/linux/cpuhplock.h                     |   9 +
 include/linux/nmi.h                           |   2 +
 include/linux/rcupdate.h                      |   2 +
 include/linux/sched/isolation.h               |  18 +-
 include/linux/tick.h                          |   2 +
 kernel/cgroup/cpuset-internal.h               |   1 +
 kernel/cgroup/cpuset.c                        | 292 +++++++++++++++++-
 kernel/context_tracking.c                     |  19 +-
 kernel/cpu.c                                  |  72 +++++
 kernel/irq/cpuhotplug.c                       |   1 +
 kernel/irq/manage.c                           |   1 +
 kernel/rcu/tree_nocb.h                        |  24 +-
 kernel/sched/core.c                           |   4 +-
 kernel/sched/isolation.c                      | 135 +++++---
 kernel/time/hrtimer.c                         |   4 +-
 kernel/time/tick-common.c                     |  16 +-
 kernel/time/tick-sched.c                      |  48 ++-
 kernel/watchdog.c                             |  24 ++
 kernel/workqueue.c                            |   6 +-
 net/core/net-sysfs.c                          |   2 +-
 .../selftests/cgroup/test_cpuset_prs.sh       |  70 ++++-
 28 files changed, 747 insertions(+), 108 deletions(-)
Signature · 1 line
2.53.0