Skip to contentSkip to conversation

ARM: Kconfig: allow full nohz CPU accounting

tip:timers/core In review tip-bot for Kevin Hilman 6 participants 18 messages Oct 3, 2013 – Nov 9, 2013 Arm (32-bit)

lore
Share & tools
lore

tip-bot for Kevin Hilman

author merge bot
#0 lore raw
Commit-ID:  31c1fc8187158cb80ccd57c19e024c55af901797
Gitweb:     http://git.kernel.org/tip/31c1fc8187158cb80ccd57c19e024c55af901797
Author:     Kevin Hilman <[email protected]>
AuthorDate: Mon, 16 Sep 2013 15:28:22 -0700
Committer:  Frederic Weisbecker <[email protected]>
CommitDate: Mon, 30 Sep 2013 15:37:05 +0200

ARM: Kconfig: allow full nohz CPU accounting

With the 64-bit requirement removed from VIRT_CPU_ACCOUNTING_GEN, allow ARM platforms to enable it. Since VIRT_CPU_ACCOUNTING_GEN is a dependency for full NO_HZ, this allows ARM platforms to enable full NO_HZ as well.

Cc Ingo Molnar <[email protected]>
Cc Russell King <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Arm Linux <[email protected]>
---
 arch/arm/Kconfig | 1 +
 1 file changed, 1 insertion(+)
arch/arm/Kconfig +1 -0
index 1ad6fb6..323baf0 100644
@@ -54,6 +54,7 @@ config ARM
 	select HAVE_REGS_AND_STACK_ACCESS_API
 	select HAVE_SYSCALL_TRACEPOINTS
 	select HAVE_UID16
+	select HAVE_VIRT_CPU_ACCOUNTING_GEN
 	select IRQ_FORCED_THREADING
 	select KTIME_SCALAR
 	select MODULES_USE_ELF_REL
#1 lore raw
Quoted 7 lines · tip-bot for Kevin Hilman
Commit-ID:  31c1fc8187158cb80ccd57c19e024c55af901797
Gitweb:     http://git.kernel.org/tip/31c1fc8187158cb80ccd57c19e024c55af901797
Author:     Kevin Hilman <[email protected]>
AuthorDate: Mon, 16 Sep 2013 15:28:22 -0700
Committer:  Frederic Weisbecker <[email protected]>
CommitDate: Mon, 30 Sep 2013 15:37:05 +0200
Quoted 6 lines

ARM: Kconfig: allow full nohz CPU accounting

With the 64-bit requirement removed from VIRT_CPU_ACCOUNTING_GEN, allow ARM platforms to enable it. Since VIRT_CPU_ACCOUNTING_GEN is a dependency for full NO_HZ, this allows ARM platforms to enable full NO_HZ as well.

I realize that arm doesn't implement irq work raise. It hooks on the timer to execute pending irq work. You'll need to implement arch_irq_work_raise() and implement some sort of self IPI to trigger irq work. The full dynticks subsystem relies on that to work correctly. It often use that to re-evaluate and possibly restart the tick after scheduler, posix timer updates, etc...

That notwithstanding that if you have no tick, tick based irq work can't work.

Thanks.

Quoted 23 lines
Cc Ingo Molnar <[email protected]>
Cc Russell King <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Arm Linux <[email protected]>
---
 arch/arm/Kconfig | 1 +
 1 file changed, 1 insertion(+)
arch/arm/Kconfig +1 -0
index 1ad6fb6..323baf0 100644
@@ -54,6 +54,7 @@ config ARM
 	select HAVE_REGS_AND_STACK_ACCESS_API
 	select HAVE_SYSCALL_TRACEPOINTS
 	select HAVE_UID16
+	select HAVE_VIRT_CPU_ACCOUNTING_GEN
 	select IRQ_FORCED_THREADING
 	select KTIME_SCALAR
 	select MODULES_USE_ELF_REL
#2 lore raw
Quoted 15 lines · Frederic Weisbecker
Quoted 7 lines · tip-bot for Kevin Hilman
Commit-ID:  31c1fc8187158cb80ccd57c19e024c55af901797
Gitweb:     http://git.kernel.org/tip/31c1fc8187158cb80ccd57c19e024c55af901797
Author:     Kevin Hilman <[email protected]>
AuthorDate: Mon, 16 Sep 2013 15:28:22 -0700
Committer:  Frederic Weisbecker <[email protected]>
CommitDate: Mon, 30 Sep 2013 15:37:05 +0200
Quoted 6 lines

ARM: Kconfig: allow full nohz CPU accounting

With the 64-bit requirement removed from VIRT_CPU_ACCOUNTING_GEN, allow ARM platforms to enable it. Since VIRT_CPU_ACCOUNTING_GEN is a dependency for full NO_HZ, this allows ARM platforms to enable full NO_HZ as well.

Quoted 9 lines

I realize that arm doesn't implement irq work raise. It hooks on the timer to execute pending irq work. You'll need to implement arch_irq_work_raise() and implement some sort of self IPI to trigger irq work. The full dynticks subsystem relies on that to work correctly. It often use that to re-evaluate and possibly restart the tick after scheduler, posix timer updates, etc...

That notwithstanding that if you have no tick, tick based irq work can't work.

Something like this? The only problem is that this latches on to the SMP support for IPIs. If CONFIG_SMP is disabled we won't get arch_irq_work_raise() but perhaps that isn't so big of a deal? Or we should think about exposing IPI support to UP systems on ARM.

----8<-----

 arch/arm/include/asm/hardirq.h |  2 +-
 arch/arm/kernel/smp.c          | 18 ++++++++++++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)
arch/arm/include/asm/hardirq.h +1 -1
index 2740c2a2..3d7351c 100644
@@ -5,7 +5,7 @@
 #include <linux/threads.h>
 #include <asm/irq.h>
 
-#define NR_IPI	6
+#define NR_IPI	7
 
 typedef struct {
 	unsigned int __softirq_pending;
arch/arm/kernel/smp.c +18 -0
index 72024ea..bf9a0d6d 100644
@@ -25,6 +25,7 @@
 #include <linux/clockchips.h>
 #include <linux/completion.h>
 #include <linux/cpufreq.h>
+#include <linux/irq_work.h>
 
 #include <linux/atomic.h>
 #include <asm/smp.h>
@@ -66,6 +67,7 @@ enum ipi_msg_type {
 	IPI_CALL_FUNC,
 	IPI_CALL_FUNC_SINGLE,
 	IPI_CPU_STOP,
+	IPI_IRQ_WORK,
 };
 
 static DECLARE_COMPLETION(cpu_running);
@@ -448,6 +450,13 @@ void arch_send_call_function_single_ipi(int cpu)
 	smp_cross_call(cpumask_of(cpu), IPI_CALL_FUNC_SINGLE);
 }
 
+#ifdef CONFIG_IRQ_WORK
+void arch_irq_work_raise(void)
+{
+	smp_cross_call(cpumask_of(smp_processor_id()), IPI_IRQ_WORK);
+}
+#endif
+
 static const char *ipi_types[NR_IPI] = {
 #define S(x,s)	[x] = s
 	S(IPI_WAKEUP, "CPU wakeup interrupts"),
@@ -456,6 +465,7 @@ static const char *ipi_types[NR_IPI] = {
 	S(IPI_CALL_FUNC, "Function call interrupts"),
 	S(IPI_CALL_FUNC_SINGLE, "Single function call interrupts"),
 	S(IPI_CPU_STOP, "CPU stop interrupts"),
+	S(IPI_IRQ_WORK, "IRQ work interrupts"),
 };
 
 void show_ipi_list(struct seq_file *p, int prec)
@@ -565,6 +575,14 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
 		irq_exit();
 		break;
 
+#ifdef CONFIG_IRQ_WORK
+	case IPI_IRQ_WORK:
+		irq_enter();
+		irq_work_run();
+		irq_exit();
+		break;
+#endif
+
 	default:
 		printk(KERN_CRIT "CPU%u: Unknown IPI message 0x%x\n",
 		       cpu, ipinr);
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation
#3 lore raw

2013/10/18 Stephen Boyd <[email protected]>:

Quoted 96 lines · Frederic Weisbecker
Quoted 15 lines · tip-bot for Kevin Hilman
Quoted 7 lines
Commit-ID:  31c1fc8187158cb80ccd57c19e024c55af901797
Gitweb:     http://git.kernel.org/tip/31c1fc8187158cb80ccd57c19e024c55af901797
Author:     Kevin Hilman <[email protected]>
AuthorDate: Mon, 16 Sep 2013 15:28:22 -0700
Committer:  Frederic Weisbecker <[email protected]>
CommitDate: Mon, 30 Sep 2013 15:37:05 +0200
Quoted 6 lines

ARM: Kconfig: allow full nohz CPU accounting

With the 64-bit requirement removed from VIRT_CPU_ACCOUNTING_GEN, allow ARM platforms to enable it. Since VIRT_CPU_ACCOUNTING_GEN is a dependency for full NO_HZ, this allows ARM platforms to enable full NO_HZ as well.

Quoted 9 lines

I realize that arm doesn't implement irq work raise. It hooks on the timer to execute pending irq work. You'll need to implement arch_irq_work_raise() and implement some sort of self IPI to trigger irq work. The full dynticks subsystem relies on that to work correctly. It often use that to re-evaluate and possibly restart the tick after scheduler, posix timer updates, etc...

That notwithstanding that if you have no tick, tick based irq work can't work.

Something like this? The only problem is that this latches on to the SMP support for IPIs. If CONFIG_SMP is disabled we won't get arch_irq_work_raise() but perhaps that isn't so big of a deal? Or we should think about exposing IPI support to UP systems on ARM.

----8<-----

 arch/arm/include/asm/hardirq.h |  2 +-
 arch/arm/kernel/smp.c          | 18 ++++++++++++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)
arch/arm/include/asm/hardirq.h +1 -1
index 2740c2a2..3d7351c 100644
@@ -5,7 +5,7 @@
 #include <linux/threads.h>
 #include <asm/irq.h>
 
-#define NR_IPI 6
+#define NR_IPI 7
 
 typedef struct {
        unsigned int __softirq_pending;
arch/arm/kernel/smp.c +11 -0
index 72024ea..bf9a0d6d 100644
@@ -25,6 +25,7 @@
 #include <linux/clockchips.h>
 #include <linux/completion.h>
 #include <linux/cpufreq.h>
+#include <linux/irq_work.h>
 
 #include <linux/atomic.h>
 #include <asm/smp.h>
@@ -66,6 +67,7 @@ enum ipi_msg_type {
        IPI_CALL_FUNC,
        IPI_CALL_FUNC_SINGLE,
        IPI_CPU_STOP,
+       IPI_IRQ_WORK,
 };
 
 static DECLARE_COMPLETION(cpu_running);
@@ -448,6 +450,13 @@ void arch_send_call_function_single_ipi(int cpu)
        smp_cross_call(cpumask_of(cpu), IPI_CALL_FUNC_SINGLE);
 }
 
+#ifdef CONFIG_IRQ_WORK
+void arch_irq_work_raise(void)
+{
+       smp_cross_call(cpumask_of(smp_processor_id()), IPI_IRQ_WORK);
+}
+#endif
+
 static const char *ipi_types[NR_IPI] = {
 #define S(x,s) [x] = s
        S(IPI_WAKEUP, "CPU wakeup interrupts"),
@@ -456,6 +465,7 @@ static const char *ipi_types[NR_IPI] = {
        S(IPI_CALL_FUNC, "Function call interrupts"),
        S(IPI_CALL_FUNC_SINGLE, "Single function call interrupts"),
        S(IPI_CPU_STOP, "CPU stop interrupts"),
+       S(IPI_IRQ_WORK, "IRQ work interrupts"),
 };
 
 void show_ipi_list(struct seq_file *p, int prec)
@@ -565,6 +575,14 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
                irq_exit();
                break;
 
+#ifdef CONFIG_IRQ_WORK

arch/arm/kernel/smp.c void handle_IPI(int ipinr, struct pt_regs *regs) · line 588

Quoted 10 lines
arch/arm/kernel/smp.c +7 -0
+       case IPI_IRQ_WORK:
+               irq_enter();
+               irq_work_run();
+               irq_exit();
+               break;
+#endif
+
        default:
                printk(KERN_CRIT "CPU%u: Unknown IPI message 0x%x\n",
                       cpu, ipinr);

Yeah that looks good! Indeed it's no big deal if it only support SMP for now since full dynticks only works on SMP. It might work on UP one day but we are not yet rushing on that :)

#4 lore raw
In reply to Stephen Boyd

Hi Stephen,

Quoted 22 lines · Stephen Boyd
Quoted 21 lines · Frederic Weisbecker
Quoted 7 lines · tip-bot for Kevin Hilman
Commit-ID:  31c1fc8187158cb80ccd57c19e024c55af901797
Gitweb:     http://git.kernel.org/tip/31c1fc8187158cb80ccd57c19e024c55af901797
Author:     Kevin Hilman <[email protected]>
AuthorDate: Mon, 16 Sep 2013 15:28:22 -0700
Committer:  Frederic Weisbecker <[email protected]>
CommitDate: Mon, 30 Sep 2013 15:37:05 +0200
Quoted 6 lines

ARM: Kconfig: allow full nohz CPU accounting

With the 64-bit requirement removed from VIRT_CPU_ACCOUNTING_GEN, allow ARM platforms to enable it. Since VIRT_CPU_ACCOUNTING_GEN is a dependency for full NO_HZ, this allows ARM platforms to enable full NO_HZ as well.

I realize that arm doesn't implement irq work raise. It hooks on the timer to execute pending irq work. You'll need to implement arch_irq_work_raise() and implement some sort of self IPI to trigger irq work. The full dynticks subsystem relies on that to work correctly. It often use that to re-evaluate and possibly restart the tick after scheduler, posix timer updates, etc...

Quoted 8 lines
Quoted 3 lines

That notwithstanding that if you have no tick, tick based irq work can't work.

Something like this? The only problem is that this latches on to the SMP support for IPIs. If CONFIG_SMP is disabled we won't get arch_irq_work_raise() but perhaps that isn't so big of a deal? Or we should think about exposing IPI support to UP systems on ARM.

Care to submit a proper patch for this to LAKML? or if you prefer I can
do it (keeping your authorship.) We need this for proper full NOHZ
support on ARM. Thanks.

Kevin

#5 lore raw
Quoted 27 lines · Kevin Hilman

Hi Stephen,

Quoted 24 lines · Stephen Boyd
Quoted 23 lines · Frederic Weisbecker
Quoted 7 lines · tip-bot for Kevin Hilman
Commit-ID:  31c1fc8187158cb80ccd57c19e024c55af901797
Gitweb:     http://git.kernel.org/tip/31c1fc8187158cb80ccd57c19e024c55af901797
Author:     Kevin Hilman <[email protected]>
AuthorDate: Mon, 16 Sep 2013 15:28:22 -0700
Committer:  Frederic Weisbecker <[email protected]>
CommitDate: Mon, 30 Sep 2013 15:37:05 +0200
Quoted 6 lines

ARM: Kconfig: allow full nohz CPU accounting

With the 64-bit requirement removed from VIRT_CPU_ACCOUNTING_GEN, allow ARM platforms to enable it. Since VIRT_CPU_ACCOUNTING_GEN is a dependency for full NO_HZ, this allows ARM platforms to enable full NO_HZ as well.

I realize that arm doesn't implement irq work raise. It hooks on the timer to execute pending irq work. You'll need to implement arch_irq_work_raise() and implement some sort of self IPI to trigger irq work. The full dynticks subsystem relies on that to work correctly. It often use that to re-evaluate and possibly restart the tick after scheduler, posix timer updates, etc...

That notwithstanding that if you have no tick, tick based irq work can't work.

Quoted 7 lines
Quoted 4 lines

Something like this? The only problem is that this latches on to the SMP support for IPIs. If CONFIG_SMP is disabled we won't get arch_irq_work_raise() but perhaps that isn't so big of a deal? Or we should think about exposing IPI support to UP systems on ARM.

Care to submit a proper patch for this to LAKML? or if you prefer I can
do it (keeping your authorship.) We need this for proper full NOHZ
support on ARM. Thanks.

Sure no problem. Let me write up some sort of commit text and I'll send it off again.

Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation
#6 lore raw

PATCH 1/1 ARM: Support arch_irq_work_raise() via self IPIs

In reply to Kevin Hilman

This will allow the scheduler tick to be restarted if we're in full NOHZ mode.

Cc Kevin Hilman <[email protected]>
Cc Frederic Weisbecker <[email protected]>
---
 arch/arm/include/asm/hardirq.h |  2 +-
 arch/arm/kernel/smp.c          | 18 ++++++++++++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)
arch/arm/include/asm/hardirq.h +1 -1
index 2740c2a2..3d7351c 100644
@@ -5,7 +5,7 @@
 #include <linux/threads.h>
 #include <asm/irq.h>
 
-#define NR_IPI	6
+#define NR_IPI	7
 
 typedef struct {
 	unsigned int __softirq_pending;
arch/arm/kernel/smp.c +18 -0
index 72024ea..bf9a0d6d 100644
@@ -25,6 +25,7 @@
 #include <linux/clockchips.h>
 #include <linux/completion.h>
 #include <linux/cpufreq.h>
+#include <linux/irq_work.h>
 
 #include <linux/atomic.h>
 #include <asm/smp.h>
@@ -66,6 +67,7 @@ enum ipi_msg_type {
 	IPI_CALL_FUNC,
 	IPI_CALL_FUNC_SINGLE,
 	IPI_CPU_STOP,
+	IPI_IRQ_WORK,
 };
 
 static DECLARE_COMPLETION(cpu_running);
@@ -448,6 +450,13 @@ void arch_send_call_function_single_ipi(int cpu)
 	smp_cross_call(cpumask_of(cpu), IPI_CALL_FUNC_SINGLE);
 }
 
+#ifdef CONFIG_IRQ_WORK
+void arch_irq_work_raise(void)
+{
+	smp_cross_call(cpumask_of(smp_processor_id()), IPI_IRQ_WORK);
+}
+#endif
+
 static const char *ipi_types[NR_IPI] = {
 #define S(x,s)	[x] = s
 	S(IPI_WAKEUP, "CPU wakeup interrupts"),
@@ -456,6 +465,7 @@ static const char *ipi_types[NR_IPI] = {
 	S(IPI_CALL_FUNC, "Function call interrupts"),
 	S(IPI_CALL_FUNC_SINGLE, "Single function call interrupts"),
 	S(IPI_CPU_STOP, "CPU stop interrupts"),
+	S(IPI_IRQ_WORK, "IRQ work interrupts"),
 };
 
 void show_ipi_list(struct seq_file *p, int prec)
@@ -565,6 +575,14 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
 		irq_exit();
 		break;
 
+#ifdef CONFIG_IRQ_WORK
+	case IPI_IRQ_WORK:
+		irq_enter();
+		irq_work_run();
+		irq_exit();
+		break;
+#endif
+
 	default:
 		printk(KERN_CRIT "CPU%u: Unknown IPI message 0x%x\n",
 		       cpu, ipinr);
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
hosted by The Linux Foundation
Reviewed-by on v1 1/1
#7 lore raw

Re: [PATCH] ARM: Support arch_irq_work_raise() via self IPIs

Quoted 6 lines · Stephen Boyd

This will allow the scheduler tick to be restarted if we're in full NOHZ mode.

Cc Kevin Hilman <[email protected]>
Cc Frederic Weisbecker <[email protected]>

Minor nit, but I'd prefer a more verbose changelog (I forget things quickly and like to rely on changelogs for my memory.) Probably worth adding something like: "By default, irq_work is tied to the tick processing (update_process_times()) but in full NOHZ mode, no tick means no IRQ work. In order for IRQ work to be done in full NOHZ mode, a self-IPI is used to process IRQ work."

Other than the changelog nit, patch looks good, feel free to add

If Russell is OK with this, it can go to his patch system.

Kevin

Reviewed-by Kevin Hilman
#8 lore raw

Re: [PATCH] ARM: Support arch_irq_work_raise() via self IPIs

Quoted 16 lines · Kevin Hilman
Quoted 6 lines · Stephen Boyd

This will allow the scheduler tick to be restarted if we're in full NOHZ mode.

Cc Kevin Hilman <[email protected]>
Cc Frederic Weisbecker <[email protected]>

Minor nit, but I'd prefer a more verbose changelog (I forget things quickly and like to rely on changelogs for my memory.) Probably worth adding something like: "By default, irq_work is tied to the tick processing (update_process_times()) but in full NOHZ mode, no tick means no IRQ work. In order for IRQ work to be done in full NOHZ mode, a self-IPI is used to process IRQ work."

Quoted 6 lines

Other than the changelog nit, patch looks good, feel free to add

If Russell is OK with this, it can go to his patch system.

Fair enough. This is what I came up with. I'll send it off to the patch tracker in about 12 hours if nobody else has anymore comments.

----8<-----
ARM: Support arch_irq_work_raise() via self IPIs

By default, IRQ work is run from the tick interrupt (see irq_work_run() in update_process_times()). When we're in full NOHZ mode, restarting the tick requires the use of IRQ work and if the only place we run IRQ work is in the tick interrupt we have an unbreakable cycle. Implement arch_irq_work_raise() via self IPIs to break this cycle and get the tick started again. Note that we implement this via IPIs which are only available on SMP builds. This shouldn't be a problem because full NOHZ is only supported on SMP builds anyway.

Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation
#9 lore raw

Re: [PATCH] ARM: Support arch_irq_work_raise() via self IPIs

Quoted 31 lines · Stephen Boyd
Quoted 16 lines · Kevin Hilman
Quoted 6 lines · Stephen Boyd

This will allow the scheduler tick to be restarted if we're in full NOHZ mode.

Cc Kevin Hilman <[email protected]>
Cc Frederic Weisbecker <[email protected]>

Minor nit, but I'd prefer a more verbose changelog (I forget things quickly and like to rely on changelogs for my memory.) Probably worth adding something like: "By default, irq_work is tied to the tick processing (update_process_times()) but in full NOHZ mode, no tick means no IRQ work. In order for IRQ work to be done in full NOHZ mode, a self-IPI is used to process IRQ work."

Quoted 6 lines

Other than the changelog nit, patch looks good, feel free to add

If Russell is OK with this, it can go to his patch system.

Fair enough. This is what I came up with. I'll send it off to the patch tracker in about 12 hours if nobody else has anymore comments.

----8<-----
ARM: Support arch_irq_work_raise() via self IPIs

Quoted 9 lines

By default, IRQ work is run from the tick interrupt (see irq_work_run() in update_process_times()). When we're in full NOHZ mode, restarting the tick requires the use of IRQ work and if the only place we run IRQ work is in the tick interrupt we have an unbreakable cycle. Implement arch_irq_work_raise() via self IPIs to break this cycle and get the tick started again. Note that we implement this via IPIs which are only available on SMP builds. This shouldn't be a problem because full NOHZ is only supported on SMP builds anyway.

Nice description!

Thanks!

Quoted 7 lines
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation
#10 lore raw

Re: [PATCH] ARM: Support arch_irq_work_raise() via self IPIs

In reply to Stephen Boyd
Quoted 19 lines · Stephen Boyd
Quoted 18 lines · Kevin Hilman
Quoted 6 lines · Stephen Boyd

This will allow the scheduler tick to be restarted if we're in full NOHZ mode.

Cc Kevin Hilman <[email protected]>
Cc Frederic Weisbecker <[email protected]>

Minor nit, but I'd prefer a more verbose changelog (I forget things quickly and like to rely on changelogs for my memory.) Probably worth adding something like: "By default, irq_work is tied to the tick processing (update_process_times()) but in full NOHZ mode, no tick means no IRQ work. In order for IRQ work to be done in full NOHZ mode, a self-IPI is used to process IRQ work."

Other than the changelog nit, patch looks good, feel free to add

Quoted 8 lines
Quoted 4 lines

If Russell is OK with this, it can go to his patch system.

Fair enough. This is what I came up with. I'll send it off to the patch tracker in about 12 hours if nobody else has anymore comments.

Nice description, thanks for the respin.

Kevin

#11 lore raw

Re: [PATCH] ARM: Support arch_irq_work_raise() via self IPIs

In reply to Stephen Boyd
Quoted 41 lines · Stephen Boyd
Quoted 16 lines · Kevin Hilman
Quoted 6 lines · Stephen Boyd

This will allow the scheduler tick to be restarted if we're in full NOHZ mode.

Cc Kevin Hilman <[email protected]>
Cc Frederic Weisbecker <[email protected]>

Minor nit, but I'd prefer a more verbose changelog (I forget things quickly and like to rely on changelogs for my memory.) Probably worth adding something like: "By default, irq_work is tied to the tick processing (update_process_times()) but in full NOHZ mode, no tick means no IRQ work. In order for IRQ work to be done in full NOHZ mode, a self-IPI is used to process IRQ work."

Quoted 6 lines

Other than the changelog nit, patch looks good, feel free to add

If Russell is OK with this, it can go to his patch system.

Fair enough. This is what I came up with. I'll send it off to the patch tracker in about 12 hours if nobody else has anymore comments.

----8<-----
ARM: Support arch_irq_work_raise() via self IPIs

By default, IRQ work is run from the tick interrupt (see irq_work_run() in update_process_times()). When we're in full NOHZ mode, restarting the tick requires the use of IRQ work and if the only place we run IRQ work is in the tick interrupt we have an unbreakable cycle. Implement arch_irq_work_raise() via self IPIs to break this cycle and get the tick started again. Note that we implement this via IPIs which are only available on SMP builds. This shouldn't be a problem because full NOHZ is only supported on SMP builds anyway.

Quoted 3 lines

Hm, so I think this just landed in -next, which seems... late for 3.13. Anyway, it causes boot failures on Cubieboards (Allwinner A10), I just bisected it down to this patch (fails to boot multi_v7_defconfig, no console output at all).

Unfortunately I'm not getting any debug_ll output from the board at the moment, so I can't actually get to a panic stack or other error info. :( I hope I can get back to you with one later today.

-Olof

#12 lore raw

Re: [PATCH] ARM: Support arch_irq_work_raise() via self IPIs

Quoted 47 lines · Olof Johansson
Quoted 41 lines · Stephen Boyd
Quoted 16 lines · Kevin Hilman
Quoted 6 lines · Stephen Boyd

This will allow the scheduler tick to be restarted if we're in full NOHZ mode.

Cc Kevin Hilman <[email protected]>
Cc Frederic Weisbecker <[email protected]>

Minor nit, but I'd prefer a more verbose changelog (I forget things quickly and like to rely on changelogs for my memory.) Probably worth adding something like: "By default, irq_work is tied to the tick processing (update_process_times()) but in full NOHZ mode, no tick means no IRQ work. In order for IRQ work to be done in full NOHZ mode, a self-IPI is used to process IRQ work."

Quoted 6 lines

Other than the changelog nit, patch looks good, feel free to add

If Russell is OK with this, it can go to his patch system.

Fair enough. This is what I came up with. I'll send it off to the patch tracker in about 12 hours if nobody else has anymore comments.

----8<-----
ARM: Support arch_irq_work_raise() via self IPIs

By default, IRQ work is run from the tick interrupt (see irq_work_run() in update_process_times()). When we're in full NOHZ mode, restarting the tick requires the use of IRQ work and if the only place we run IRQ work is in the tick interrupt we have an unbreakable cycle. Implement arch_irq_work_raise() via self IPIs to break this cycle and get the tick started again. Note that we implement this via IPIs which are only available on SMP builds. This shouldn't be a problem because full NOHZ is only supported on SMP builds anyway.

Quoted 3 lines
Quoted 8 lines

Hm, so I think this just landed in -next, which seems... late for 3.13. Anyway, it causes boot failures on Cubieboards (Allwinner A10), I just bisected it down to this patch (fails to boot multi_v7_defconfig, no console output at all).

Unfortunately I'm not getting any debug_ll output from the board at the moment, so I can't actually get to a panic stack or other error info. :( I hope I can get back to you with one later today.

Turns out that I hit this on the BeagleBone as well, and I got a trace out of it. Seems like this might be broken on all non-SMP platforms built with CONFIG_SMP=y?

[    0.206322] Unable to handle kernel NULL pointer dereference at
virtual address 00000000
[    0.214752] pgd = c0004000
[    0.217628] [00000000] *pgd=00000000
[    0.221421] Internal error: Oops: 80000005 [#1] SMP ARM
[    0.226890] Modules linked in:
[    0.230136] CPU: 0 PID: 1 Comm: swapper/0 Not tainted
3.12.0-rc6-00018-g8d45144-dirty #16
[    0.238646] task: de05b440 ti: de05c000 task.ti: de05c000
[    0.244294] PC is at 0x0
[    0.247010] LR is at arch_irq_work_raise+0x3c/0x48
[    0.252034] pc : [<00000000>]    lr : [<c0019590>]    psr: 60000193
[    0.252034] sp : de05dd60  ip : 00000001  fp : 00000000
[    0.264029] r10: c085e2f0  r9 : de05c000  r8 : c07be0a4
[    0.269496] r7 : de05c000  r6 : de05c000  r5 : c07c5778  r4 : c0824554
[    0.276305] r3 : 00000000  r2 : 00000000  r1 : 00000006  r0 : c0529a58
[    0.283116] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM
Segment kernel
[    0.290818] Control: 10c5387d  Table: 80004019  DAC: 00000017
[    0.296821] Process swapper/0 (pid: 1, stack limit = 0xde05c248)
[    0.303094] Stack: (0xde05dd60 to 0xde05e000)
[    0.307677] dd60: c07b9dbc c00cb2dc 00000001 c08242c0 c08242c0
60000113 c07be0a8 c00b0590
[    0.316191] dd80: de05c000 c085e2f0 c08242c0 c08242c0 c1414c28
c00b07cc de05b440 c1414c28
[    0.324705] dda0: c08242c0 c00b0af8 c0862bb0 c0862db0 c1414cd8
de05c028 c0824840 de05ddb8
[    0.333218] ddc0: 00000000 00000009 00000001 00000024 c07be0a8
c07be0a4 de05c000 c085e2f0
[    0.341731] dde0: 00000000 c004a4b0 00000010 de00d2dc 00000054
00000100 00000024 00000000
[    0.350244] de00: de05c028 0000000a ffff8ae7 00200040 00000016
de05c000 60000193 de05c000
[    0.358756] de20: 00000054 00000000 00000000 00000000 00000000
c004a704 00000000 de05c008
[    0.367269] de40: c07ba254 c004aa1c c07c5778 c0014b70 fa200000
00000054 de05de80 c0861244
[    0.375782] de60: 00000000 c0008634 de05b440 c051c778 20000113
ffffffff de05deb4 c051d0a4
[    0.384295] de80: 00000001 00000001 00000000 de05b440 c082afac
de057ac0 de057ac0 de0443c0
[    0.392807] dea0: 00000000 00000000 00000000 00000000 c082afbc
de05dec8 c009f2a0 c051c778
[    0.401321] dec0: 20000113 ffffffff 00000000 c016edb0 00000000
000002b0 de057ac0 de057ac0
[    0.409833] dee0: 00000000 c016ee40 c0875e50 de05df2e de057ac0
00000000 00000013 00000000
[    0.418345] df00: 00000000 c016f054 de043600 de0443c0 c008eb38
de004ec0 c0875e50 c008eb44
[    0.426858] df20: 00000012 00000000 00000000 3931f0f8 00000000
00000000 00000014 c0822e84
[    0.435370] df40: 00000000 c008ed2c 00000000 00000000 00000000
c07b7490 c07b7490 c075ab3c
[    0.443882] df60: 00000000 c00701ac 00000002 00000000 c0070160
dffadb73 7bf8edb4 00000000
[    0.452394] df80: c051092c 00000000 00000000 00000000 00000000
00000000 00000000 c0510934
[    0.460907] dfa0: de05aa40 00000000 c051092c c0013ce8 00000000
00000000 00000000 00000000
[    0.469419] dfc0: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[    0.477930] dfe0: 00000000 00000000 00000000 00000000 00000013
00000000 07efffe5 4dfac6f5
[    0.486468] [<c0019590>] (arch_irq_work_raise+0x3c/0x48) from
[<c00cb2dc>] (irq_work_queue+0xe4/0xf8)
[    0.496073] [<c00cb2dc>] (irq_work_queue+0xe4/0xf8) from
[<c00b0590>] (rcu_accelerate_cbs+0x1d4/0x1d8)
[    0.505756] [<c00b0590>] (rcu_accelerate_cbs+0x1d4/0x1d8) from
[<c00b07cc>] (rcu_start_gp+0x34/0x48)
[    0.515259] [<c00b07cc>] (rcu_start_gp+0x34/0x48) from [<c00b0af8>]
(rcu_process_callbacks+0x318/0x608)
[    0.525040] [<c00b0af8>] (rcu_process_callbacks+0x318/0x608) from
[<c004a4b0>] (__do_softirq+0x114/0x2a0)
[    0.534989] [<c004a4b0>] (__do_softirq+0x114/0x2a0) from
[<c004a704>] (do_softirq+0x6c/0x74)
[    0.543776] [<c004a704>] (do_softirq+0x6c/0x74) from [<c004aa1c>]
(irq_exit+0xac/0x100)
[    0.552126] [<c004aa1c>] (irq_exit+0xac/0x100) from [<c0014b70>]
(handle_IRQ+0x54/0xb4)
[    0.560466] [<c0014b70>] (handle_IRQ+0x54/0xb4) from [<c0008634>]
(omap3_intc_handle_irq+0x60/0x74)
[    0.569885] [<c0008634>] (omap3_intc_handle_irq+0x60/0x74) from
[<c051d0a4>] (__irq_svc+0x44/0x5c)
[    0.579196] Exception stack(0xde05de80 to 0xde05dec8)
[    0.584491] de80: 00000001 00000001 00000000 de05b440 c082afac
de057ac0 de057ac0 de0443c0
[    0.593004] dea0: 00000000 00000000 00000000 00000000 c082afbc
de05dec8 c009f2a0 c051c778
[    0.601510] dec0: 20000113 ffffffff
[    0.605202] [<c051d0a4>] (__irq_svc+0x44/0x5c) from [<c051c778>]
(_raw_spin_unlock_irq+0x28/0x2c)
[    0.614447] [<c051c778>] (_raw_spin_unlock_irq+0x28/0x2c) from
[<c016edb0>] (proc_alloc_inum+0x30/0xa8)
[    0.624220] [<c016edb0>] (proc_alloc_inum+0x30/0xa8) from
[<c016ee40>] (proc_register+0x18/0x130)
[    0.633454] [<c016ee40>] (proc_register+0x18/0x130) from
[<c016f054>] (proc_mkdir_data+0x44/0x6c)
[    0.642702] [<c016f054>] (proc_mkdir_data+0x44/0x6c) from
[<c008eb44>] (register_irq_proc+0x6c/0x128)
[    0.652292] [<c008eb44>] (register_irq_proc+0x6c/0x128) from
[<c008ed2c>] (init_irq_proc+0x74/0xb0)
[    0.661706] [<c008ed2c>] (init_irq_proc+0x74/0xb0) from
[<c075ab3c>] (kernel_init_freeable+0x84/0x1c8)
[    0.671390] [<c075ab3c>] (kernel_init_freeable+0x84/0x1c8) from
[<c0510934>] (kernel_init+0x8/0x150)
[    0.680892] [<c0510934>] (kernel_init+0x8/0x150) from [<c0013ce8>]
(ret_from_fork+0x14/0x2c)
[    0.689674] Code: bad PC value
[    0.692984] ---[ end trace 1b75b31a2719ed1c ]---
#13 lore raw

Re: [PATCH] ARM: Support arch_irq_work_raise() via self IPIs

Quoted 50 lines · Olof Johansson
Quoted 49 lines · Olof Johansson
Quoted 39 lines · Stephen Boyd
Quoted 15 lines · Kevin Hilman
Quoted 6 lines · Stephen Boyd

This will allow the scheduler tick to be restarted if we're in full NOHZ mode.

Cc Kevin Hilman <[email protected]>
Cc Frederic Weisbecker <[email protected]>

Minor nit, but I'd prefer a more verbose changelog (I forget things quickly and like to rely on changelogs for my memory.) Probably worth adding something like: "By default, irq_work is tied to the tick processing (update_process_times()) but in full NOHZ mode, no tick means no IRQ work. In order for IRQ work to be done in full NOHZ mode, a self-IPI is used to process IRQ work."

Quoted 6 lines

Other than the changelog nit, patch looks good, feel free to add

If Russell is OK with this, it can go to his patch system.

Fair enough. This is what I came up with. I'll send it off to the patch tracker in about 12 hours if nobody else has anymore comments.

----8<-----
ARM: Support arch_irq_work_raise() via self IPIs

By default, IRQ work is run from the tick interrupt (see irq_work_run() in update_process_times()). When we're in full NOHZ mode, restarting the tick requires the use of IRQ work and if the only place we run IRQ work is in the tick interrupt we have an unbreakable cycle. Implement arch_irq_work_raise() via self IPIs to break this cycle and get the tick started again. Note that we implement this via IPIs which are only available on SMP builds. This shouldn't be a problem because full NOHZ is only supported on SMP builds anyway.

Quoted 3 lines

Hm, so I think this just landed in -next, which seems... late for 3.13. Anyway, it causes boot failures on Cubieboards (Allwinner A10), I just bisected it down to this patch (fails to boot multi_v7_defconfig, no console output at all).

Quoted 6 lines
Quoted 3 lines

Unfortunately I'm not getting any debug_ll output from the board at the moment, so I can't actually get to a panic stack or other error info. :( I hope I can get back to you with one later today.

Turns out that I hit this on the BeagleBone as well, and I got a trace out of it. Seems like this might be broken on all non-SMP platforms built with CONFIG_SMP=y?

Ah yes. We don't have IPI support on UP kernels but SMP_ON_UP exposes arch_irq_work_raise(). How about this? We should just skip this function if we're running on UP.

---8<----

arch/arm/kernel/smp.c +2 -1
index bf9a0d6d..e115cbb 100644
@@ -453,7 +453,8 @@ void arch_send_call_function_single_ipi(int cpu)
 #ifdef CONFIG_IRQ_WORK
 void arch_irq_work_raise(void)
 {
-	smp_cross_call(cpumask_of(smp_processor_id()), IPI_IRQ_WORK);
+	if (is_smp())
+		smp_cross_call(cpumask_of(smp_processor_id()), IPI_IRQ_WORK);
 }
 #endif
 
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation
Tested-by
#14 lore raw

Re: [PATCH] ARM: Support arch_irq_work_raise() via self IPIs

Quoted 54 lines · Stephen Boyd
Quoted 53 lines · Olof Johansson
Quoted 52 lines · Olof Johansson
Quoted 39 lines · Stephen Boyd
Quoted 15 lines · Kevin Hilman
Quoted 6 lines · Stephen Boyd

This will allow the scheduler tick to be restarted if we're in full NOHZ mode.

Cc Kevin Hilman <[email protected]>
Cc Frederic Weisbecker <[email protected]>

Minor nit, but I'd prefer a more verbose changelog (I forget things quickly and like to rely on changelogs for my memory.) Probably worth adding something like: "By default, irq_work is tied to the tick processing (update_process_times()) but in full NOHZ mode, no tick means no IRQ work. In order for IRQ work to be done in full NOHZ mode, a self-IPI is used to process IRQ work."

Quoted 6 lines

Other than the changelog nit, patch looks good, feel free to add

If Russell is OK with this, it can go to his patch system.

Fair enough. This is what I came up with. I'll send it off to the patch tracker in about 12 hours if nobody else has anymore comments.

----8<-----
ARM: Support arch_irq_work_raise() via self IPIs

By default, IRQ work is run from the tick interrupt (see irq_work_run() in update_process_times()). When we're in full NOHZ mode, restarting the tick requires the use of IRQ work and if the only place we run IRQ work is in the tick interrupt we have an unbreakable cycle. Implement arch_irq_work_raise() via self IPIs to break this cycle and get the tick started again. Note that we implement this via IPIs which are only available on SMP builds. This shouldn't be a problem because full NOHZ is only supported on SMP builds anyway.

Quoted 3 lines

Hm, so I think this just landed in -next, which seems... late for 3.13. Anyway, it causes boot failures on Cubieboards (Allwinner A10), I just bisected it down to this patch (fails to boot multi_v7_defconfig, no console output at all).

Unfortunately I'm not getting any debug_ll output from the board at the moment, so I can't actually get to a panic stack or other error info. :( I hope I can get back to you with one later today.

Quoted 7 lines
Quoted 3 lines

Turns out that I hit this on the BeagleBone as well, and I got a trace out of it. Seems like this might be broken on all non-SMP platforms built with CONFIG_SMP=y?

Ah yes. We don't have IPI support on UP kernels but SMP_ON_UP exposes arch_irq_work_raise(). How about this? We should just skip this function if we're running on UP.

Yep.

Send it to Russell's tracker, plz.

-Olof

#15 lore raw

Re: [PATCH] ARM: Support arch_irq_work_raise() via self IPIs

Quoted 11 lines · Olof Johansson
Quoted 4 lines · Stephen Boyd

Ah yes. We don't have IPI support on UP kernels but SMP_ON_UP exposes arch_irq_work_raise(). How about this? We should just skip this function if we're running on UP.

Yep.

Send it to Russell's tracker, plz.

Thanks. 7887/1.

Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation
#16 lore raw

Re: [PATCH] ARM: Support arch_irq_work_raise() via self IPIs

Quoted 6 lines · Stephen Boyd
Quoted 5 lines · Olof Johansson
Quoted 4 lines · Stephen Boyd

Ah yes. We don't have IPI support on UP kernels but SMP_ON_UP exposes arch_irq_work_raise(). How about this? We should just skip this function if we're running on UP.

Quoted 8 lines
Quoted 6 lines

Yep.

Send it to Russell's tracker, plz.

Thanks. 7887/1.

It's probably going to miss Stephen's pull from my tree this evening (if he even does one) so... that gives me two options: either drop the original commit or apply this.

I think I'll apply this and put it in place just after the original commit which caused the breakage.

Thanks.

#17 lore raw

Re: [PATCH] ARM: Support arch_irq_work_raise() via self IPIs

Quoted 11 lines · Russell King - ARM Linux
Quoted 10 lines · Stephen Boyd
Quoted 9 lines · Olof Johansson
Quoted 4 lines · Stephen Boyd

Ah yes. We don't have IPI support on UP kernels but SMP_ON_UP exposes arch_irq_work_raise(). How about this? We should just skip this function if we're running on UP.

Yep.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Send it to Russell's tracker, plz.

Thanks. 7887/1.

It's probably going to miss Stephen's pull from my tree this evening (if he even does one) so... that gives me two options: either drop the original commit or apply this.

It's Saturday for Stephen, so he won't do another one until Sunday night our time.

Quoted 2 lines

I think I'll apply this and put it in place just after the original commit which caused the breakage.

Thanks, sounds good to me.

-Olof