Skip to contentSkip to conversation

tick/nohz: Add per-CPU control of non-idle tick suppression

RFC PATCH In review Peng Wang 2 participants 11 messages Oct 17 – Oct 30

Subsystems (6)
Tracing Driver core cgroup ABI/API +2 more subsystems
lore
Share & tools
lore

Peng Wang

author
#0 lore raw

v2 cover tick/nohz: Add per-CPU control of non-idle tick suppression

This series adds per-CPU control of non-idle tick suppression within the boot-time nohz_full set. The policy can be changed without taking CPUs offline.

Some cloud hosts pin vCPU threads one-to-one to host CPUs. At full allocation, housekeeping CPUs also provide vCPU capacity and always retain their non-idle tick. Operators want comparable performance for otherwise equivalent vCPUs, regardless of CPU placement. Retaining the non-idle tick on nohz_full CPUs as well avoids one placement-dependent difference in host tick interference.

With spare capacity, selected nohz_full CPUs may stop the tick to reduce host scheduler-tick interference with vCPUs. Allocation changes as VMs are created or destroyed while other VMs remain running. A vCPU thread may be the only runnable task in either state, so existing tick dependencies do not express this policy.

The per-CPU nohz_tick_allowed boolean defaults to 1 (allowed). Writing 0 prevents non-idle tick suppression; writing 1 allows it, subject to existing dependencies. Readback reports permission, not tick state. This leaves idle tick suppression, nohz_full membership, context tracking, RCU offloading and housekeeping configuration unchanged.

This series depends on the separately submitted nohz kick fix [2]. Testing used that fix together with this series in x86_64 KVM guests, with NO_HZ_FULL enabled and disabled.

Changes since v1:
- Following Greg's sysfs feedback [1], replace the global CPU list with a per-CPU boolean, removing the list-size and custom parsing issues.
- Rename nohz_full_allowed to nohz_tick_allowed, as suggested by Waiman.
- Explain the vCPU allocation use case and clarify permission semantics.

v1: https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/2026101743-monotype-demeaning-f97b@gregkh/
[2] "tick/nohz: Requeue kicks flushed outside hardirq context"

    Message-ID: <43668cf88d976084e2865ec07dea82b64ad402ba.1792240863.git.peng_wang@linux.alibaba.com>

Peng Wang (2):
tick/nohz: Add a policy dependency for full dynticks
cpu: Add per-CPU control of non-idle tick suppression

 .../ABI/testing/sysfs-devices-system-cpu      | 27 ++++++++++
 drivers/base/cpu.c                            | 53 +++++++++++++++++++
 include/linux/tick.h                          |  8 ++-
 include/trace/events/timer.h                  |  3 +-
 kernel/time/tick-sched.c                      | 42 +++++++++++++++
 5 files changed, 130 insertions(+), 3 deletions(-)

base-commit: 70dd4ee72eb2d5793c62e71c14e3aba78e5a4e56
prerequisite-patch-id: 6036ece815f0ab92a0413d433675d5b240b7a8c1

Signature · 1 line
2.47.0
author
#1 lore raw

RFC PATCH v2 1/2 tick/nohz: Add a policy dependency for full dynticks

Cloud hosts may pin vCPU threads one-to-one to host CPUs. At full allocation this includes the housekeeping CPUs, which always retain their non-idle tick. Operators want comparable performance for otherwise equivalent vCPUs. Retaining the non-idle tick on nohz_full CPUs as well avoids one placement-dependent difference in host tick interference. With spare capacity, selected CPUs may stop the tick to reduce host scheduler-tick interference with vCPUs.

Allocation changes as VMs are created or destroyed while other VMs remain running. A vCPU thread may be the only runnable task in either state, so existing tick dependencies do not express this policy.

Add a per-CPU POLICY dependency within the boot-time nohz_full set. Permission defaults to enabled. The policy can be changed without taking CPUs offline. Revoking permission uses the existing asynchronous kick; granting it leaves all other dependencies effective. Preserve POLICY across CPU hotplug and report it in tick_stop tracing.

Idle tick suppression, nohz_full membership, context tracking, RCU offloading and housekeeping configuration are unchanged.

---
 include/linux/tick.h         |  8 +++++--
 include/trace/events/timer.h |  3 ++-
 kernel/time/tick-sched.c     | 42 ++++++++++++++++++++++++++++++++++++
 3 files changed, 50 insertions(+), 3 deletions(-)
include/linux/tick.h +6 -2
index b121c5d53e308..8b250f245ff31 100644
@@ -111,9 +111,10 @@ enum tick_dep_bits {
 	TICK_DEP_BIT_SCHED		= 2,
 	TICK_DEP_BIT_CLOCK_UNSTABLE	= 3,
 	TICK_DEP_BIT_RCU		= 4,
-	TICK_DEP_BIT_RCU_EXP		= 5
+	TICK_DEP_BIT_RCU_EXP		= 5,
+	TICK_DEP_BIT_POLICY		= 6
 };
-#define TICK_DEP_BIT_MAX TICK_DEP_BIT_RCU_EXP
+#define TICK_DEP_BIT_MAX TICK_DEP_BIT_POLICY
 
 #define TICK_DEP_MASK_NONE		0
 #define TICK_DEP_MASK_POSIX_TIMER	(1 << TICK_DEP_BIT_POSIX_TIMER)
@@ -122,6 +123,7 @@ enum tick_dep_bits {
 #define TICK_DEP_MASK_CLOCK_UNSTABLE	(1 << TICK_DEP_BIT_CLOCK_UNSTABLE)
 #define TICK_DEP_MASK_RCU		(1 << TICK_DEP_BIT_RCU)
 #define TICK_DEP_MASK_RCU_EXP		(1 << TICK_DEP_BIT_RCU_EXP)
+#define TICK_DEP_MASK_POLICY		(1 << TICK_DEP_BIT_POLICY)
 
 #ifdef CONFIG_NO_HZ_COMMON
 extern bool tick_nohz_enabled;
@@ -205,6 +207,8 @@ extern void tick_nohz_dep_set_signal(struct task_struct *tsk,
 extern void tick_nohz_dep_clear_signal(struct signal_struct *signal,
 				       enum tick_dep_bits bit);
 extern bool tick_nohz_cpu_hotpluggable(unsigned int cpu);
+extern bool tick_nohz_full_get_allowed(unsigned int cpu);
+extern int tick_nohz_full_set_allowed(unsigned int cpu, bool allowed);
 
 /*
  * The below are tick_nohz_[set,clear]_dep() wrappers that optimize off-cases
include/trace/events/timer.h +2 -1
index 3aa0608c63611..dff9579cd90eb 100644
@@ -435,7 +435,8 @@ TRACE_EVENT(itimer_expire,
 		tick_dep_name(SCHED)			\
 		tick_dep_name(CLOCK_UNSTABLE)		\
 		tick_dep_name(RCU)			\
-		tick_dep_name_end(RCU_EXP)
+		tick_dep_name(RCU_EXP)			\
+		tick_dep_name_end(POLICY)
 
 #undef tick_dep_name
 #undef tick_dep_mask_name
kernel/time/tick-sched.c +42 -0

67 lines not on this page · raw message

Signature · 1 line
2.47.0
#2 lore raw

Re: [RFC PATCH v2 1/2] tick/nohz: Add a policy dependency for full dynticks

Quoted 11 lines · Peng Wang

Cloud hosts may pin vCPU threads one-to-one to host CPUs. At full allocation this includes the housekeeping CPUs, which always retain their non-idle tick. Operators want comparable performance for otherwise equivalent vCPUs. Retaining the non-idle tick on nohz_full CPUs as well avoids one placement-dependent difference in host tick interference. With spare capacity, selected CPUs may stop the tick to reduce host scheduler-tick interference with vCPUs.

Allocation changes as VMs are created or destroyed while other VMs remain running. A vCPU thread may be the only runnable task in either state, so existing tick dependencies do not express this policy.

I'm not sure I understand correctly what you are trying to solve. Are you saying that you want to keep the tick running when there is more than one vCPU per CPU? Because the scheduler already takes care of that.

Quoted 9 lines

Add a per-CPU POLICY dependency within the boot-time nohz_full set. Permission defaults to enabled. The policy can be changed without taking CPUs offline. Revoking permission uses the existing asynchronous kick; granting it leaves all other dependencies effective. Preserve POLICY across CPU hotplug and report it in tick_stop tracing.

Idle tick suppression, nohz_full membership, context tracking, RCU offloading and housekeeping configuration are unchanged.

This still keeps the overhead of nohz_full even if the tick stays on, right? (RCU nocb, kernel <-> user overhead, etc...). Isn't it a problem for you?

Thanks.

Quoted 136 lines
---
 include/linux/tick.h         |  8 +++++--
 include/trace/events/timer.h |  3 ++-
 kernel/time/tick-sched.c     | 42 ++++++++++++++++++++++++++++++++++++
 3 files changed, 50 insertions(+), 3 deletions(-)
include/linux/tick.h +6 -2
index b121c5d53e308..8b250f245ff31 100644
@@ -111,9 +111,10 @@ enum tick_dep_bits {
 	TICK_DEP_BIT_SCHED		= 2,
 	TICK_DEP_BIT_CLOCK_UNSTABLE	= 3,
 	TICK_DEP_BIT_RCU		= 4,
-	TICK_DEP_BIT_RCU_EXP		= 5
+	TICK_DEP_BIT_RCU_EXP		= 5,
+	TICK_DEP_BIT_POLICY		= 6
 };
-#define TICK_DEP_BIT_MAX TICK_DEP_BIT_RCU_EXP
+#define TICK_DEP_BIT_MAX TICK_DEP_BIT_POLICY
 
 #define TICK_DEP_MASK_NONE		0
 #define TICK_DEP_MASK_POSIX_TIMER	(1 << TICK_DEP_BIT_POSIX_TIMER)
@@ -122,6 +123,7 @@ enum tick_dep_bits {
 #define TICK_DEP_MASK_CLOCK_UNSTABLE	(1 << TICK_DEP_BIT_CLOCK_UNSTABLE)
 #define TICK_DEP_MASK_RCU		(1 << TICK_DEP_BIT_RCU)
 #define TICK_DEP_MASK_RCU_EXP		(1 << TICK_DEP_BIT_RCU_EXP)
+#define TICK_DEP_MASK_POLICY		(1 << TICK_DEP_BIT_POLICY)
 
 #ifdef CONFIG_NO_HZ_COMMON
 extern bool tick_nohz_enabled;
@@ -205,6 +207,8 @@ extern void tick_nohz_dep_set_signal(struct task_struct *tsk,
 extern void tick_nohz_dep_clear_signal(struct signal_struct *signal,
 				       enum tick_dep_bits bit);
 extern bool tick_nohz_cpu_hotpluggable(unsigned int cpu);
+extern bool tick_nohz_full_get_allowed(unsigned int cpu);
+extern int tick_nohz_full_set_allowed(unsigned int cpu, bool allowed);
 
 /*
  * The below are tick_nohz_[set,clear]_dep() wrappers that optimize off-cases
include/trace/events/timer.h +2 -1
index 3aa0608c63611..dff9579cd90eb 100644
@@ -435,7 +435,8 @@ TRACE_EVENT(itimer_expire,
 		tick_dep_name(SCHED)			\
 		tick_dep_name(CLOCK_UNSTABLE)		\
 		tick_dep_name(RCU)			\
-		tick_dep_name_end(RCU_EXP)
+		tick_dep_name(RCU_EXP)			\
+		tick_dep_name_end(POLICY)
 
 #undef tick_dep_name
 #undef tick_dep_mask_name
kernel/time/tick-sched.c +42 -0
index 87e2a7a875265..0d6fcc3736bae 100644
@@ -375,6 +375,11 @@ static bool check_tick_dependency(atomic_t *dep)
 		return true;
 	}
 
+	if (val & TICK_DEP_MASK_POLICY) {
+		trace_tick_stop(0, TICK_DEP_MASK_POLICY);
+		return true;
+	}
+
 	return false;
 }
 
@@ -561,6 +566,39 @@ void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit)
 }
 EXPORT_SYMBOL_GPL(tick_nohz_dep_clear_cpu);
 
+/* Read policy, not whether a CPU currently has its tick stopped. */
+bool tick_nohz_full_get_allowed(unsigned int cpu)
+{
+	struct tick_sched *ts = per_cpu_ptr(&tick_cpu_sched, cpu);
+
+	/* CPU teardown resets and restores this bit under stop_machine(). */
+	return !(atomic_read(&ts->tick_dep_mask) & TICK_DEP_MASK_POLICY);
+}
+
+int tick_nohz_full_set_allowed(unsigned int cpu, bool allowed)
+{
+	struct tick_sched *ts = per_cpu_ptr(&tick_cpu_sched, cpu);
+
+	if (!tick_nohz_full_cpu(cpu))
+		return -EINVAL;
+
+	/* CPU device removal may wait for an active sysfs callback. */
+	if (!cpus_read_trylock())
+		return -EBUSY;
+
+	if (allowed) {
+		tick_nohz_dep_clear_cpu(cpu, TICK_DEP_BIT_POLICY);
+	} else if (cpu_online(cpu)) {
+		/* The existing kick requests an asynchronous restart. */
+		tick_nohz_dep_set_cpu(cpu, TICK_DEP_BIT_POLICY);
+	} else {
+		/* Record policy without queuing irq_work on an offline CPU. */
+		atomic_or(TICK_DEP_MASK_POLICY, &ts->tick_dep_mask);
+	}
+	cpus_read_unlock();
+	return 0;
+}
+
 /*
  * Set a per-task tick dependency. RCU needs this. Also posix CPU timers
  * in order to elapse per task timers.
@@ -1533,6 +1571,7 @@ void tick_sched_timer_dying(int cpu)
 {
 	struct tick_sched *ts = &per_cpu(tick_cpu_sched, cpu);
 	unsigned long idle_calls, idle_sleeps;
+	int policy;
 
 	/* This must happen before hrtimers are migrated! */
 	if (tick_sched_flag_test(ts, TS_FLAG_HIGHRES))
@@ -1540,7 +1579,10 @@ void tick_sched_timer_dying(int cpu)
 
 	idle_calls = ts->idle_calls;
 	idle_sleeps = ts->idle_sleeps;
+	/* Preserve only the policy dependency across CPU offline/online. */
+	policy = atomic_read(&ts->tick_dep_mask) & TICK_DEP_MASK_POLICY;
 	memset(ts, 0, sizeof(*ts));
+	atomic_set(&ts->tick_dep_mask, policy);
 	ts->idle_calls = idle_calls;
 	ts->idle_sleeps = idle_sleeps;
 }
Signature · 1 line
2.47.0
Signature · 2 lines
Frederic Weisbecker
SUSE Labs
author
#3 lore raw

Re: [RFC PATCH v2 1/2] tick/nohz: Add a policy dependency for full dynticks

Quoted 9 lines · Frederic Weisbecker

Le Sat, Oct 17, 2026 at 09:13:41PM +0800, Peng Wang a 殂rit :

Quoted 8 lines

Cloud hosts may pin vCPU threads one-to-one to host CPUs. At full allocation this includes the housekeeping CPUs, which always retain their non-idle tick. Operators want comparable performance for otherwise equivalent vCPUs. Retaining the non-idle tick on nohz_full CPUs as well avoids one placement-dependent difference in host tick interference. With spare capacity, selected CPUs may stop the tick to reduce host scheduler-tick interference with vCPUs.

Quoted 8 lines
Quoted 3 lines

Allocation changes as VMs are created or destroyed while other VMs remain running. A vCPU thread may be the only runnable task in either state, so existing tick dependencies do not express this policy.

I'm not sure I understand correctly what you are trying to solve. Are you saying that you want to keep the tick running when there is more than one vCPU per CPU? Because the scheduler already takes care of that.

Hi Frederic,

No. We pin vCPU threads one-to-one to host CPUs, without oversubscription. By "full allocation" I meant that all allocatable host CPUs are in use, including CPUs outside the nohz_full set.

For example, on an eight-CPU host booted with isolcpus=nohz,1-7, CPU0 is outside the nohz_full set:

- With eight pinned vCPUs, CPU0 retains its non-idle tick while CPUs1-7 may stop theirs. Retaining the tick on CPUs1-7 too avoids this placement-dependent difference in tick interference between otherwise equivalent vCPUs.

- With seven or fewer pinned vCPUs, we can leave CPU0 without a vCPU and keep them on CPUs1-7. We can then allow tick suppression on CPUs1-7, subject to the existing dependencies. This gives these vCPUs a common tick policy while reducing host tick interference.

If the vCPU on CPU0 is released while the other seven remain running, we allow CPUs1-7 to suppress their ticks. If a new vCPU is later assigned to CPU0, we retain the ticks on CPUs1-7 again to avoid the difference in tick interference.

Throughout these transitions, CPU1 can still have its original pinned vCPU thread as its only runnable task. Its runnable task count does not change merely because CPU0 gains or loses a vCPU, so the scheduler's existing tick dependency cannot express this allocation policy.

Quoted 8 lines
Quoted 8 lines

Add a per-CPU POLICY dependency within the boot-time nohz_full set. Permission defaults to enabled. The policy can be changed without taking CPUs offline. Revoking permission uses the existing asynchronous kick; granting it leaves all other dependencies effective. Preserve POLICY across CPU hotplug and report it in tick_stop tracing.

Quoted 5 lines
Quoted 2 lines

Idle tick suppression, nohz_full membership, context tracking, RCU offloading and housekeeping configuration are unchanged.

This still keeps the overhead of nohz_full even if the tick stays on, right? (RCU nocb, kernel <-> user overhead, etc...). Isn't it a problem for you?

Yes. Our setup uses only the nohz flag of isolcpus, without additional housekeeping affinity tuning. Context tracking and RCU callback offloading remain enabled when the tick is kept running.

We have observed a performance benefit from this nohz_full setup with our database workload running on pinned vCPUs. We retain the boot-time nohz_full configuration so that tick suppression remains available as vCPU allocation changes, even though its other costs remain when the tick is kept running.

This control changes only whether non-idle ticks may be suppressed; it does not change the nohz_full CPU set or housekeeping CPU masks.

Thanks,
Peng

Quoted 143 lines

Thanks.

Quoted 136 lines
---
 include/linux/tick.h         |  8 +++++--
 include/trace/events/timer.h |  3 ++-
 kernel/time/tick-sched.c     | 42 ++++++++++++++++++++++++++++++++++++
 3 files changed, 50 insertions(+), 3 deletions(-)
include/linux/tick.h +6 -2
index b121c5d53e308..8b250f245ff31 100644
@@ -111,9 +111,10 @@ enum tick_dep_bits {
 	TICK_DEP_BIT_SCHED		= 2,
 	TICK_DEP_BIT_CLOCK_UNSTABLE	= 3,
 	TICK_DEP_BIT_RCU		= 4,
-	TICK_DEP_BIT_RCU_EXP		= 5
+	TICK_DEP_BIT_RCU_EXP		= 5,
+	TICK_DEP_BIT_POLICY		= 6
 };
-#define TICK_DEP_BIT_MAX TICK_DEP_BIT_RCU_EXP
+#define TICK_DEP_BIT_MAX TICK_DEP_BIT_POLICY
 
 #define TICK_DEP_MASK_NONE		0
 #define TICK_DEP_MASK_POSIX_TIMER	(1 << TICK_DEP_BIT_POSIX_TIMER)
@@ -122,6 +123,7 @@ enum tick_dep_bits {
 #define TICK_DEP_MASK_CLOCK_UNSTABLE	(1 << TICK_DEP_BIT_CLOCK_UNSTABLE)
 #define TICK_DEP_MASK_RCU		(1 << TICK_DEP_BIT_RCU)
 #define TICK_DEP_MASK_RCU_EXP		(1 << TICK_DEP_BIT_RCU_EXP)
+#define TICK_DEP_MASK_POLICY		(1 << TICK_DEP_BIT_POLICY)
 
 #ifdef CONFIG_NO_HZ_COMMON
 extern bool tick_nohz_enabled;
@@ -205,6 +207,8 @@ extern void tick_nohz_dep_set_signal(struct task_struct *tsk,
 extern void tick_nohz_dep_clear_signal(struct signal_struct *signal,
 				       enum tick_dep_bits bit);
 extern bool tick_nohz_cpu_hotpluggable(unsigned int cpu);
+extern bool tick_nohz_full_get_allowed(unsigned int cpu);
+extern int tick_nohz_full_set_allowed(unsigned int cpu, bool allowed);
 
 /*
  * The below are tick_nohz_[set,clear]_dep() wrappers that optimize off-cases
include/trace/events/timer.h +2 -1
index 3aa0608c63611..dff9579cd90eb 100644
@@ -435,7 +435,8 @@ TRACE_EVENT(itimer_expire,
 		tick_dep_name(SCHED)			\
 		tick_dep_name(CLOCK_UNSTABLE)		\
 		tick_dep_name(RCU)			\
-		tick_dep_name_end(RCU_EXP)
+		tick_dep_name(RCU_EXP)			\
+		tick_dep_name_end(POLICY)
 
 #undef tick_dep_name
 #undef tick_dep_mask_name
kernel/time/tick-sched.c +42 -0
index 87e2a7a875265..0d6fcc3736bae 100644
@@ -375,6 +375,11 @@ static bool check_tick_dependency(atomic_t *dep)
 		return true;
 	}
 
+	if (val & TICK_DEP_MASK_POLICY) {
+		trace_tick_stop(0, TICK_DEP_MASK_POLICY);
+		return true;
+	}
+
 	return false;
 }
 
@@ -561,6 +566,39 @@ void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit)
 }
 EXPORT_SYMBOL_GPL(tick_nohz_dep_clear_cpu);
 
+/* Read policy, not whether a CPU currently has its tick stopped. */
+bool tick_nohz_full_get_allowed(unsigned int cpu)
+{
+	struct tick_sched *ts = per_cpu_ptr(&tick_cpu_sched, cpu);
+
+	/* CPU teardown resets and restores this bit under stop_machine(). */
+	return !(atomic_read(&ts->tick_dep_mask) & TICK_DEP_MASK_POLICY);
+}
+
+int tick_nohz_full_set_allowed(unsigned int cpu, bool allowed)
+{
+	struct tick_sched *ts = per_cpu_ptr(&tick_cpu_sched, cpu);
+
+	if (!tick_nohz_full_cpu(cpu))
+		return -EINVAL;
+
+	/* CPU device removal may wait for an active sysfs callback. */
+	if (!cpus_read_trylock())
+		return -EBUSY;
+
+	if (allowed) {
+		tick_nohz_dep_clear_cpu(cpu, TICK_DEP_BIT_POLICY);
+	} else if (cpu_online(cpu)) {
+		/* The existing kick requests an asynchronous restart. */
+		tick_nohz_dep_set_cpu(cpu, TICK_DEP_BIT_POLICY);
+	} else {
+		/* Record policy without queuing irq_work on an offline CPU. */
+		atomic_or(TICK_DEP_MASK_POLICY, &ts->tick_dep_mask);
+	}
+	cpus_read_unlock();
+	return 0;
+}
+
 /*
  * Set a per-task tick dependency. RCU needs this. Also posix CPU timers
  * in order to elapse per task timers.
@@ -1533,6 +1571,7 @@ void tick_sched_timer_dying(int cpu)
 {
 	struct tick_sched *ts = &per_cpu(tick_cpu_sched, cpu);
 	unsigned long idle_calls, idle_sleeps;
+	int policy;
 
 	/* This must happen before hrtimers are migrated! */
 	if (tick_sched_flag_test(ts, TS_FLAG_HIGHRES))
@@ -1540,7 +1579,10 @@ void tick_sched_timer_dying(int cpu)
 
 	idle_calls = ts->idle_calls;
 	idle_sleeps = ts->idle_sleeps;
+	/* Preserve only the policy dependency across CPU offline/online. */
+	policy = atomic_read(&ts->tick_dep_mask) & TICK_DEP_MASK_POLICY;
 	memset(ts, 0, sizeof(*ts));
+	atomic_set(&ts->tick_dep_mask, policy);
 	ts->idle_calls = idle_calls;
 	ts->idle_sleeps = idle_sleeps;
 }
Signature · 1 line
2.47.0
Signature · 2 lines
Frederic Weisbecker
SUSE Labs
#4 lore raw

Re: [RFC PATCH v2 1/2] tick/nohz: Add a policy dependency for full dynticks

Quoted 44 lines · Peng Wang
Quoted 9 lines · Frederic Weisbecker

Le Sat, Oct 17, 2026 at 09:13:41PM +0800, Peng Wang a 殂rit :

Quoted 8 lines

Cloud hosts may pin vCPU threads one-to-one to host CPUs. At full allocation this includes the housekeeping CPUs, which always retain their non-idle tick. Operators want comparable performance for otherwise equivalent vCPUs. Retaining the non-idle tick on nohz_full CPUs as well avoids one placement-dependent difference in host tick interference. With spare capacity, selected CPUs may stop the tick to reduce host scheduler-tick interference with vCPUs.

Quoted 8 lines
Quoted 3 lines

Allocation changes as VMs are created or destroyed while other VMs remain running. A vCPU thread may be the only runnable task in either state, so existing tick dependencies do not express this policy.

I'm not sure I understand correctly what you are trying to solve. Are you saying that you want to keep the tick running when there is more than one vCPU per CPU? Because the scheduler already takes care of that.

Hi Frederic,

No. We pin vCPU threads one-to-one to host CPUs, without oversubscription. By "full allocation" I meant that all allocatable host CPUs are in use, including CPUs outside the nohz_full set.

For example, on an eight-CPU host booted with isolcpus=nohz,1-7, CPU0 is outside the nohz_full set:

- With eight pinned vCPUs, CPU0 retains its non-idle tick while CPUs1-7 may stop theirs. Retaining the tick on CPUs1-7 too avoids this placement-dependent difference in tick interference between otherwise equivalent vCPUs.

- With seven or fewer pinned vCPUs, we can leave CPU0 without a vCPU and keep them on CPUs1-7. We can then allow tick suppression on CPUs1-7, subject to the existing dependencies. This gives these vCPUs a common tick policy while reducing host tick interference.

If the vCPU on CPU0 is released while the other seven remain running, we allow CPUs1-7 to suppress their ticks. If a new vCPU is later assigned to CPU0, we retain the ticks on CPUs1-7 again to avoid the difference in tick interference.

Quoted 5 lines

Throughout these transitions, CPU1 can still have its original pinned vCPU thread as its only runnable task. Its runnable task count does not change merely because CPU0 gains or loses a vCPU, so the scheduler's existing tick dependency cannot express this allocation policy.

Ok now I understand what you want to do, I just don't understand why. Why it matters to turn off nohz_full on all CPUs when vCPUs run on housekeeping?

Quoted 12 lines
Quoted 11 lines
Quoted 8 lines

Add a per-CPU POLICY dependency within the boot-time nohz_full set. Permission defaults to enabled. The policy can be changed without taking CPUs offline. Revoking permission uses the existing asynchronous kick; granting it leaves all other dependencies effective. Preserve POLICY across CPU hotplug and report it in tick_stop tracing.

Idle tick suppression, nohz_full membership, context tracking, RCU offloading and housekeeping configuration are unchanged.

This still keeps the overhead of nohz_full even if the tick stays on, right? (RCU nocb, kernel <-> user overhead, etc...). Isn't it a problem for you?

Quoted 7 lines

Yes. Our setup uses only the nohz flag of isolcpus, without additional housekeeping affinity tuning. Context tracking and RCU callback offloading remain enabled when the tick is kept running.

We have observed a performance benefit from this nohz_full setup with our database workload running on pinned vCPUs.

Do you mean that your database runs better when hosts CPUs are in nohz_full? But how does this new interface help? Ie: why is it worse to have all CPUs in nohz_full except the housekeeping rather than shutting down nohz_full altogether?

Or to put it another way: why is it better to have either:

- all vCPUs on nohz_full host CPUs
or
- all vCPUs on ticking CPUs

And why does it hurt to have a single vCPU running on a ticking CPU.

Thanks.

Signature · 2 lines
Frederic Weisbecker
SUSE Labs
author
#5 lore raw

Re: [RFC PATCH v2 1/2] tick/nohz: Add a policy dependency for full dynticks

Quoted 45 lines · Frederic Weisbecker
Quoted 44 lines · Peng Wang
Quoted 9 lines · Frederic Weisbecker

Le Sat, Oct 17, 2026 at 09:13:41PM +0800, Peng Wang a 殂rit :

Quoted 8 lines

Cloud hosts may pin vCPU threads one-to-one to host CPUs. At full allocation this includes the housekeeping CPUs, which always retain their non-idle tick. Operators want comparable performance for otherwise equivalent vCPUs. Retaining the non-idle tick on nohz_full CPUs as well avoids one placement-dependent difference in host tick interference. With spare capacity, selected CPUs may stop the tick to reduce host scheduler-tick interference with vCPUs.

Quoted 8 lines
Quoted 3 lines

Allocation changes as VMs are created or destroyed while other VMs remain running. A vCPU thread may be the only runnable task in either state, so existing tick dependencies do not express this policy.

I'm not sure I understand correctly what you are trying to solve. Are you saying that you want to keep the tick running when there is more than one vCPU per CPU? Because the scheduler already takes care of that.

Hi Frederic,

No. We pin vCPU threads one-to-one to host CPUs, without oversubscription. By "full allocation" I meant that all allocatable host CPUs are in use, including CPUs outside the nohz_full set.

For example, on an eight-CPU host booted with isolcpus=nohz,1-7, CPU0 is outside the nohz_full set:

- With eight pinned vCPUs, CPU0 retains its non-idle tick while CPUs1-7 may stop theirs. Retaining the tick on CPUs1-7 too avoids this placement-dependent difference in tick interference between otherwise equivalent vCPUs.

- With seven or fewer pinned vCPUs, we can leave CPU0 without a vCPU and keep them on CPUs1-7. We can then allow tick suppression on CPUs1-7, subject to the existing dependencies. This gives these vCPUs a common tick policy while reducing host tick interference.

If the vCPU on CPU0 is released while the other seven remain running, we allow CPUs1-7 to suppress their ticks. If a new vCPU is later assigned to CPU0, we retain the ticks on CPUs1-7 again to avoid the difference in tick interference.

Quoted 29 lines
Quoted 5 lines

Throughout these transitions, CPU1 can still have its original pinned vCPU thread as its only runnable task. Its runnable task count does not change merely because CPU0 gains or loses a vCPU, so the scheduler's existing tick dependency cannot express this allocation policy.

Ok now I understand what you want to do, I just don't understand why. Why it matters to turn off nohz_full on all CPUs when vCPUs run on housekeeping?

Quoted 19 lines
Quoted 11 lines
Quoted 8 lines

Add a per-CPU POLICY dependency within the boot-time nohz_full set. Permission defaults to enabled. The policy can be changed without taking CPUs offline. Revoking permission uses the existing asynchronous kick; granting it leaves all other dependencies effective. Preserve POLICY across CPU hotplug and report it in tick_stop tracing.

Idle tick suppression, nohz_full membership, context tracking, RCU offloading and housekeeping configuration are unchanged.

This still keeps the overhead of nohz_full even if the tick stays on, right? (RCU nocb, kernel <-> user overhead, etc...). Isn't it a problem for you?

Yes. Our setup uses only the nohz flag of isolcpus, without additional housekeeping affinity tuning. Context tracking and RCU callback offloading remain enabled when the tick is kept running.

We have observed a performance benefit from this nohz_full setup with our database workload running on pinned vCPUs.

Hi Frederic,

Quoted 1 line

Do you mean that your database runs better when hosts CPUs are in nohz_full?

Yes. In our database tests, suppressing the host's non-idle periodic tick improved performance, with the rest of the host configuration unchanged.

Quoted 1 line

But how does this new interface help? Ie: why is it worse to have all CPUs in

The tick policy itself can therefore affect guest performance. A large host runs VMs for many customers with different workloads. Customers create and release VMs independently, so the same host CPU can be reassigned to a different workload while other VMs remain running.

We want to choose whether to retain the non-idle tick or allow its suppression for the workload currently assigned to each CPU, and revisit that choice when the CPU is reassigned. Per-CPU control lets us do this without rebooting or taking CPUs offline.

Quoted 11 lines

nohz_full except the housekeeping rather than shutting down nohz_full altogether?

Or to put it another way: why is it better to have either:

- all vCPUs on nohz_full host CPUs
or
- all vCPUs on ticking CPUs

And why does it hurt to have a single vCPU running on a ticking CPU.

If a VM's vCPUs span housekeeping and nohz_full CPUs, some may benefit from tick suppression while others cannot. For workloads sensitive to tick interference, this can cause performance differences within the VM. A common tick policy is one deployment option in that case.

In summary, the runtime control would let us:

1. Select or withdraw tick suppression as CPUs are assigned to different workloads.
2. Apply a common non-idle tick policy across a VM's vCPUs when preferred.

Thanks,
Peng

Quoted 5 lines

Thanks.

Signature · 2 lines
Frederic Weisbecker
SUSE Labs
#6 lore raw

Re: [RFC PATCH v2 1/2] tick/nohz: Add a policy dependency for full dynticks

Quoted 7 lines · Peng Wang

Hi Frederic,

Quoted 1 line

Do you mean that your database runs better when hosts CPUs are in nohz_full?

Yes. In our database tests, suppressing the host's non-idle periodic tick improved performance, with the rest of the host configuration unchanged.

Out of curiosity, for such database tests, only the host tick is off, not the guest tick, right?

Quoted 8 lines
Quoted 1 line

But how does this new interface help? Ie: why is it worse to have all CPUs in

The tick policy itself can therefore affect guest performance. A large host runs VMs for many customers with different workloads. Customers create and release VMs independently, so the same host CPU can be reassigned to a different workload while other VMs remain running.

Quoted 4 lines

We want to choose whether to retain the non-idle tick or allow its suppression for the workload currently assigned to each CPU, and revisit that choice when the CPU is reassigned. Per-CPU control lets us do this without rebooting or taking CPUs offline.

Ok in that case, do you know about the current work in progress to make nohz_full opt-able through cpusets? It implies taking the target CPUs offline shortly but the policy is to be setup before VM is launched, right?

Thanks.

Signature · 2 lines
Frederic Weisbecker
SUSE Labs
author
#7 lore raw

Re: [RFC PATCH v2 1/2] tick/nohz: Add a policy dependency for full dynticks

Quoted 11 lines · Frederic Weisbecker

Le Thu, Oct 29, 2026 at 02:41:28PM +0800, Peng Wang a 殂rit :

Quoted 7 lines

Hi Frederic,

Quoted 1 line

Do you mean that your database runs better when hosts CPUs are in nohz_full?

Yes. In our database tests, suppressing the host's non-idle periodic tick improved performance, with the rest of the host configuration unchanged.

Out of curiosity, for such database tests, only the host tick is off, not the guest tick, right?

Yes. Only the host's non-idle tick policy changed. The guest did not use nohz_full, and its tick configuration was unchanged.

Quoted 14 lines
Quoted 9 lines
Quoted 2 lines

But how does this new interface help? Ie: why is it worse to have all CPUs in

The tick policy itself can therefore affect guest performance. A large host runs VMs for many customers with different workloads. Customers create and release VMs independently, so the same host CPU can be reassigned to a different workload while other VMs remain running.

Quoted 4 lines

We want to choose whether to retain the non-idle tick or allow its suppression for the workload currently assigned to each CPU, and revisit that choice when the CPU is reassigned. Per-CPU control lets us do this without rebooting or taking CPUs offline.

Quoted 4 lines

Ok in that case, do you know about the current work in progress to make nohz_full opt-able through cpusets? It implies taking the target CPUs offline shortly but the policy is to be setup before VM is launched, right?

Not always. Waiman mentioned the cpuset work earlier; it could cover selecting the policy before a new VM starts.

Some customers express a preference for host tick suppression through the management interface when changing workloads within a running VM. The host operator decides whether to allow suppression on the CPUs backing that VM.

We want to support revisiting that decision while keeping those CPUs online and the VM running, with nohz_full membership unchanged. Could such an operation fit into the cpuset work?

Thanks,
Peng

Quoted 6 lines

Thanks.

Signature · 2 lines
Frederic Weisbecker
SUSE Labs
#8 lore raw

Re: [RFC PATCH v2 1/2] tick/nohz: Add a policy dependency for full dynticks

Quoted 5 lines · Peng Wang
Quoted 4 lines

Ok in that case, do you know about the current work in progress to make nohz_full opt-able through cpusets? It implies taking the target CPUs offline shortly but the policy is to be setup before VM is launched, right?

Quoted 7 lines

Not always. Waiman mentioned the cpuset work earlier; it could cover selecting the policy before a new VM starts.

Some customers express a preference for host tick suppression through the management interface when changing workloads within a running VM. The host operator decides whether to allow suppression on the CPUs backing that VM.

I think people should know ahead if they will run latency critical workloads or common workloads in their VMs. Because those are completely different.

Quoted 4 lines

We want to support revisiting that decision while keeping those CPUs online and the VM running, with nohz_full membership unchanged. Could such an operation fit into the cpuset work?

If that is really necessary then I guess we can have a file to control that in cpuset once the nohz_full interface is ready. Though I'm a bit skeptical due to the overhead of user <-> kernel transitions and housekeeping.

Thanks.

Quoted 10 lines

Thanks,
Peng

Quoted 6 lines

Thanks.

Signature · 2 lines
Frederic Weisbecker
SUSE Labs
Signature · 2 lines
Frederic Weisbecker
SUSE Labs
author
#9 lore raw

Re: [RFC PATCH v2 1/2] tick/nohz: Add a policy dependency for full dynticks

Quoted 18 lines · Frederic Weisbecker

Le Fri, Oct 30, 2026 at 12:02:53AM +0800, Peng Wang a 殂rit :

Quoted 5 lines
Quoted 4 lines

Ok in that case, do you know about the current work in progress to make nohz_full opt-able through cpusets? It implies taking the target CPUs offline shortly but the policy is to be setup before VM is launched, right?

Quoted 7 lines

Not always. Waiman mentioned the cpuset work earlier; it could cover selecting the policy before a new VM starts.

Some customers express a preference for host tick suppression through the management interface when changing workloads within a running VM. The host operator decides whether to allow suppression on the CPUs backing that VM.

I think people should know ahead if they will run latency critical workloads or common workloads in their VMs. Because those are completely different.

Quoted 1 line
Quoted 7 lines
Quoted 3 lines

We want to support revisiting that decision while keeping those CPUs online and the VM running, with nohz_full membership unchanged. Could such an operation fit into the cpuset work?

If that is really necessary then I guess we can have a file to control that in cpuset once the nohz_full interface is ready. Though I'm a bit skeptical due to the overhead of user <-> kernel transitions and housekeeping.

Hi Frederic,

Thanks for clarifying.

For some of our database workloads, we have observed a net benefit from nohz_full despite the transition and housekeeping overheads. We understand that those overheads remain when the tick is kept running.

We'll follow the cpuset work and see whether this control would fit.

Thanks,
Peng

Quoted 17 lines

Thanks.

Quoted 10 lines

Thanks,
Peng

Quoted 6 lines

Thanks.

Signature · 2 lines
Frederic Weisbecker
SUSE Labs
Signature · 2 lines
Frederic Weisbecker
SUSE Labs
author
#10 lore raw

RFC PATCH v2 2/2 cpu: Add per-CPU control of non-idle tick suppression

In reply to Peng Wang

Expose cpuN/nohz_tick_allowed on CPUs in the boot-time nohz_full set. The boolean defaults to 1: allow non-idle tick suppression, subject to existing dependencies. Writing 0 prevents suppression through POLICY. Readback reports permission, not the current tick state.

Each write affects only that CPU. Registered offline CPUs remain configurable, and policy persists across offline/online transitions. Retry writes when CPU hotplug is in progress so CPU device removal can drain active sysfs callbacks without a lock dependency cycle.

---
 .../ABI/testing/sysfs-devices-system-cpu      | 27 ++++++++++
 drivers/base/cpu.c                            | 53 +++++++++++++++++++
 2 files changed, 80 insertions(+)
Documentation/ABI/testing/sysfs-devices-system-cpu +27 -0
index 82d10d556cc89..4f6ac6b41e2b6 100644
@@ -767,6 +767,33 @@ Description:	Preferred MTE tag checking mode
 
 		See also: Documentation/arch/arm64/memory-tagging-extension.rst
 
+What:		/sys/devices/system/cpu/cpuX/nohz_tick_allowed
+Date:		October 2026
+Contact:	Linux kernel mailing list <[email protected]>
+Description:
+		(RW) Boolean allowing this CPU to stop its non-idle scheduler
+		tick. Present only for registered CPUs in the boot-time
+		nohz_full set, with CONFIG_NO_HZ_FULL.
+
+		The default is 1 (allowed), subject to existing tick
+		dependencies. Writing 0 prevents non-idle tick suppression;
+		writing 1 allows it without forcing the tick to stop. Each
+		write affects one CPU. Readback reports permission, not the
+		current tick state.
+
+		Revoking permission requests an asynchronous restart of a
+		stopped non-idle tick on an online CPU. The write does not wait
+		for restart. Idle tick suppression is unchanged; the policy
+		applies when the CPU leaves idle.
+
+		Registered offline CPUs may be configured. Policy is retained
+		per logical CPU across offline/online and CPU device removal
+		and re-addition. CPUs without a registered cpuN device cannot
+		be configured through this interface.
+
+		This does not change nohz_full membership, context tracking,
+		RCU offloading or housekeeping configuration.
+
 What:		/sys/devices/system/cpu/nohz_full
 Date:		Apr 2015
 Contact:	Linux kernel mailing list <[email protected]>
drivers/base/cpu.c +53 -0

72 lines not on this page · raw message

Signature · 1 line
2.47.0