Skip to contentSkip to conversation

Nohz and isolation fixes v2

GIT PULL New Frederic Weisbecker 2 participants 7 messages Dec 14, 2017 – Dec 18, 2017 Documentation

lore
Share & tools
lore

Frederic Weisbecker

author
#0 lore raw

Ingo,

Please pull the nohz/fixes-v2 branch that can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/fixes-v2

HEAD: 5ecdec22346b99967e8b4e8d78a5b4d23bf216f9

Thanks,
Frederic
---

Frederic Weisbecker (2):
sched/isolation: Enable CONFIG_CPU_ISOLATION=y by default sched/isolation: Document boot parameters dependency on CONFIG_CPU_ISOLATION

Paul E. McKenney (1):
sched/isolation: Make NO_HZ_FULL select CPU_ISOLATION

 Documentation/admin-guide/kernel-parameters.rst | 1 +
 Documentation/admin-guide/kernel-parameters.txt | 4 ++--
 init/Kconfig                                    | 6 +++++-
 kernel/time/Kconfig                             | 1 +
 4 files changed, 9 insertions(+), 3 deletions(-)
author
#1 lore raw

PATCH 1/3 sched/isolation: Make NO_HZ_FULL select CPU_ISOLATION

From: "Paul E. McKenney" <[email protected]>

CONFIG_NO_HZ_FULL doesn't make sense without CONFIG_CPU_ISOLATION. In fact enabling the first without the second is a regression as nohz_full= boot parameter gets silently ignored.

Besides this unnatural combination hangs RCU gp kthread when running rcutorture for reasons that are not yet fully understood:

	rcu_preempt kthread starved for 9974 jiffies! g4294967208
	+c4294967207 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=0
	rcu_preempt     I 7464     8      2 0x80000000
	Call Trace:
		__schedule+0x493/0x620
		schedule+0x24/0x40
		schedule_timeout+0x330/0x3b0
		? preempt_count_sub+0xea/0x140
		? collect_expired_timers+0xb0/0xb0
		rcu_gp_kthread+0x6bf/0xef0

This commit therefore makes NO_HZ_FULL select CPU_ISOLATION, which prevents all these bad behaviours.

Fixes 5c4991e24c69 ("sched/isolation: Split out new CONFIG_CPU_ISOLATION=y config from CONFIG_NO_HZ_FULL")
Reported-by kernel test robot <[email protected]>
Signed-off-by Paul E. McKenney <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Chris Metcalf <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Luiz Capitulino <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc John Stultz <[email protected]>
Signed-off-by Frederic Weisbecker <[email protected]>
---
 kernel/time/Kconfig | 1 +
 1 file changed, 1 insertion(+)
kernel/time/Kconfig +1 -0
index e776fc8..f6b5f19 100644
@@ -95,6 +95,7 @@ config NO_HZ_FULL
 	select RCU_NOCB_CPU
 	select VIRT_CPU_ACCOUNTING_GEN
 	select IRQ_WORK
+	select CPU_ISOLATION
 	help
 	 Adaptively try to shutdown the tick whenever possible, even when
 	 the CPU is running tasks. Typically this requires running a single
Signature · 1 line
2.7.4
merge bot
#2 lore raw

[tip:sched/urgent] sched/isolation: Make CONFIG_NO_HZ_FULL select CONFIG_CPU_ISOLATION

sched/isolation: Make CONFIG_NO_HZ_FULL select CONFIG_CPU_ISOLATION
Commit-ID:  bf29cb238dc0656e6564b6a94bb82e11d2129437
Gitweb:     https://git.kernel.org/tip/bf29cb238dc0656e6564b6a94bb82e11d2129437
Author:     Paul E. McKenney <[email protected]>
AuthorDate: Thu, 14 Dec 2017 19:18:25 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Mon, 18 Dec 2017 13:46:42 +0100

sched/isolation: Make CONFIG_NO_HZ_FULL select CONFIG_CPU_ISOLATION

CONFIG_NO_HZ_FULL doesn't make sense without CONFIG_CPU_ISOLATION. In fact enabling the first without the second is a regression as nohz_full= boot parameter gets silently ignored.

Besides this unnatural combination hangs RCU gp kthread when running rcutorture for reasons that are not yet fully understood:

	rcu_preempt kthread starved for 9974 jiffies! g4294967208
	+c4294967207 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=0
	rcu_preempt     I 7464     8      2 0x80000000
	Call Trace:
		__schedule+0x493/0x620
		schedule+0x24/0x40
		schedule_timeout+0x330/0x3b0
		? preempt_count_sub+0xea/0x140
		? collect_expired_timers+0xb0/0xb0
		rcu_gp_kthread+0x6bf/0xef0

This commit therefore makes NO_HZ_FULL select CPU_ISOLATION, which prevents all these bad behaviours.

Reported-by kernel test robot <[email protected]>
Signed-off-by Paul E. McKenney <[email protected]>
Signed-off-by Frederic Weisbecker <[email protected]>
Cc Chris Metcalf <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc John Stultz <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Luiz Capitulino <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Wanpeng Li <[email protected]>
Fixes 5c4991e24c69 ("sched/isolation: Split out new CONFIG_CPU_ISOLATION=y config from CONFIG_NO_HZ_FULL")
Link http://lkml.kernel.org/r/[email protected] open here
Signed-off-by Ingo Molnar <[email protected]>
---
 kernel/time/Kconfig | 1 +
 1 file changed, 1 insertion(+)
kernel/time/Kconfig +1 -0
index e776fc8..f6b5f19 100644
@@ -95,6 +95,7 @@ config NO_HZ_FULL
 	select RCU_NOCB_CPU
 	select VIRT_CPU_ACCOUNTING_GEN
 	select IRQ_WORK
+	select CPU_ISOLATION
 	help
 	 Adaptively try to shutdown the tick whenever possible, even when
 	 the CPU is running tasks. Typically this requires running a single
author
#3 lore raw

PATCH 2/3 sched/isolation: Enable CONFIG_CPU_ISOLATION=y by default

In reply to Frederic Weisbecker

The "isolcpus=" boot parameter support was always built-in before we moved the related code under CONFIG_CPU_ISOLATION. Having it disabled by default is very confusing for people accustomed to use this parameter.

So enable it by dafault to keep the previous behaviour but keep it optable for those who want to tinify their kernels.

Signed-off-by Frederic Weisbecker <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Chris Metcalf <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Luiz Capitulino <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc John Stultz <[email protected]>
---
 init/Kconfig | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
init/Kconfig +5 -1
index 2934249..690a381 100644
@@ -461,10 +461,14 @@ endmenu # "CPU/Task time and stats accounting"
 
 config CPU_ISOLATION
 	bool "CPU isolation"
+	default y
 	help
 	  Make sure that CPUs running critical tasks are not disturbed by
 	  any source of "noise" such as unbound workqueues, timers, kthreads...
-	  Unbound jobs get offloaded to housekeeping CPUs.
+	  Unbound jobs get offloaded to housekeeping CPUs. This is driven by
+	  the "isolcpus=" boot parameter.
+
+	  Say Y if unsure.
 
 source "kernel/rcu/Kconfig"
Signature · 1 line
2.7.4
merge bot
#4 lore raw

[tip:sched/urgent] sched/isolation: Enable CONFIG_CPU_ISOLATION=y by default

sched/isolation: Enable CONFIG_CPU_ISOLATION=y by default
Commit-ID:  2c43838c99d9d23f17eb2bdadafcb2879cca6995
Gitweb:     https://git.kernel.org/tip/2c43838c99d9d23f17eb2bdadafcb2879cca6995
Author:     Frederic Weisbecker <[email protected]>
AuthorDate: Thu, 14 Dec 2017 19:18:26 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Mon, 18 Dec 2017 13:46:42 +0100

sched/isolation: Enable CONFIG_CPU_ISOLATION=y by default

The "isolcpus=" boot parameter support was always built-in before we moved the related code under CONFIG_CPU_ISOLATION. Having it disabled by default is very confusing for people accustomed to use this parameter.

So enable it by dafault to keep the previous behaviour but keep it optable for those who want to tinify their kernels.

Signed-off-by Frederic Weisbecker <[email protected]>
Cc Chris Metcalf <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc John Stultz <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Luiz Capitulino <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc kernel test robot <[email protected]>
Link http://lkml.kernel.org/r/[email protected] open here
Signed-off-by Ingo Molnar <[email protected]>
---
 init/Kconfig | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
init/Kconfig +5 -1
index 2934249..690a381 100644
@@ -461,10 +461,14 @@ endmenu # "CPU/Task time and stats accounting"
 
 config CPU_ISOLATION
 	bool "CPU isolation"
+	default y
 	help
 	  Make sure that CPUs running critical tasks are not disturbed by
 	  any source of "noise" such as unbound workqueues, timers, kthreads...
-	  Unbound jobs get offloaded to housekeeping CPUs.
+	  Unbound jobs get offloaded to housekeeping CPUs. This is driven by
+	  the "isolcpus=" boot parameter.
+
+	  Say Y if unsure.
 
 source "kernel/rcu/Kconfig"
author
#5 lore raw

PATCH 3/3 sched/isolation: Document boot parameters dependency on CONFIG_CPU_ISOLATION

In reply to Frederic Weisbecker

The "isolcpus=" and "nohz_full=" boot parameters depend on CPU Isolation support. Let's document that.

Signed-off-by Frederic Weisbecker <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Chris Metcalf <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Luiz Capitulino <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc John Stultz <[email protected]>
---
 Documentation/admin-guide/kernel-parameters.rst | 1 +
 Documentation/admin-guide/kernel-parameters.txt | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
Documentation/admin-guide/kernel-parameters.rst +1 -0
index b2598cc..7242cbd 100644
@@ -109,6 +109,7 @@ parameter is applicable::
 	IPV6	IPv6 support is enabled.
 	ISAPNP	ISA PnP code is enabled.
 	ISDN	Appropriate ISDN support is enabled.
+	ISOL	CPU Isolation is enabled.
 	JOY	Appropriate joystick support is enabled.
 	KGDB	Kernel debugger support is enabled.
 	KVM	Kernel Virtual Machine support is enabled.
Documentation/admin-guide/kernel-parameters.txt +2 -2
index 6571fbf..1683107 100644
@@ -1737,7 +1737,7 @@
 	isapnp=		[ISAPNP]
 			Format: <RDP>,<reset>,<pci_scan>,<verbosity>
 
-	isolcpus=	[KNL,SMP] Isolate a given set of CPUs from disturbance.
+	isolcpus=	[KNL,SMP,ISOL] Isolate a given set of CPUs from disturbance.
 			[Deprecated - use cpusets instead]
 			Format: [flag-list,]<cpu-list>
 

@@ -2662,7 +2662,7 @@
 			Valid arguments: on, off
 			Default: on
 
-	nohz_full=	[KNL,BOOT]
+	nohz_full=	[KNL,BOOT,SMP,ISOL]
 			The argument is a cpu list, as described above.
 			In kernels built with CONFIG_NO_HZ_FULL=y, set
 			the specified list of CPUs whose tick will be stopped
Signature · 1 line
2.7.4
merge bot
#6 lore raw

[tip:sched/urgent] sched/isolation: Document boot parameters dependency on CONFIG_CPU_ISOLATION=y

sched/isolation: Document boot parameters dependency on CONFIG_CPU_ISOLATION=y
Commit-ID:  d94d105329e4a8a874853b5bd854b6587c41adda
Gitweb:     https://git.kernel.org/tip/d94d105329e4a8a874853b5bd854b6587c41adda
Author:     Frederic Weisbecker <[email protected]>
AuthorDate: Thu, 14 Dec 2017 19:18:27 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Mon, 18 Dec 2017 13:46:42 +0100

sched/isolation: Document boot parameters dependency on CONFIG_CPU_ISOLATION=y

The "isolcpus=" and "nohz_full=" boot parameters depend on CPU Isolation support. Let's document that.

Signed-off-by Frederic Weisbecker <[email protected]>
Cc Chris Metcalf <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc John Stultz <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Luiz Capitulino <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc kernel test robot <[email protected]>
Link http://lkml.kernel.org/r/[email protected] open here
Signed-off-by Ingo Molnar <[email protected]>
---
 Documentation/admin-guide/kernel-parameters.rst | 1 +
 Documentation/admin-guide/kernel-parameters.txt | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
Documentation/admin-guide/kernel-parameters.rst +1 -0
index b2598cc..7242cbd 100644
@@ -109,6 +109,7 @@ parameter is applicable::
 	IPV6	IPv6 support is enabled.
 	ISAPNP	ISA PnP code is enabled.
 	ISDN	Appropriate ISDN support is enabled.
+	ISOL	CPU Isolation is enabled.
 	JOY	Appropriate joystick support is enabled.
 	KGDB	Kernel debugger support is enabled.
 	KVM	Kernel Virtual Machine support is enabled.
Documentation/admin-guide/kernel-parameters.txt +2 -2
index 6571fbf..1683107 100644
@@ -1737,7 +1737,7 @@
 	isapnp=		[ISAPNP]
 			Format: <RDP>,<reset>,<pci_scan>,<verbosity>
 
-	isolcpus=	[KNL,SMP] Isolate a given set of CPUs from disturbance.
+	isolcpus=	[KNL,SMP,ISOL] Isolate a given set of CPUs from disturbance.
 			[Deprecated - use cpusets instead]
 			Format: [flag-list,]<cpu-list>
 

@@ -2662,7 +2662,7 @@
 			Valid arguments: on, off
 			Default: on
 
-	nohz_full=	[KNL,BOOT]
+	nohz_full=	[KNL,BOOT,SMP,ISOL]
 			The argument is a cpu list, as described above.
 			In kernels built with CONFIG_NO_HZ_FULL=y, set
 			the specified list of CPUs whose tick will be stopped