Skip to contentSkip to conversation

Problem: scaling of /proc/stat on large systems

PATCH In review Jack Steiner 4 participants 13 messages Sep 29, 2010 – Oct 12, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I'm looking for suggestions on how to fix a scaling problem with access to /proc/stat.

On a large x86_64 system (4096p, 256 nodes, 5530 IRQs), access to /proc/stat takes too long - more than 12 sec:

	# time cat /proc/stat >/dev/null
	real	12.630s
	user	 0.000s
	sys	12.629s

This affects top, ps (some variants), w, glibc (sysconf) and much more.

One of the items reported in /proc/stat is a total count of interrupts that have been received. This calculation requires summation of the interrupts received on each cpu (kstat_irqs_cpu()).

The data is kept in per-cpu arrays linked to each irq_desc. On a 4096p/5530IRQ system summing this data requires accessing ~90MB.

Deleting the summation of the kstat_irqs_cpu data eliminates the high access time but is an API breakage that I assume is unacceptible.

Another possibility would be using delayed work (similar to vmstat_update) that periodically sums the data into a single array. The disadvantage in this approach is that there would be a delay between receipt of an interrupt & it's count appearing /proc/stat. Is this an issue for anyone? Another disadvantage is that it adds to the overall "noise" introduced by kernel threads.

Is there a better approach to take?

--- jack

#1 lore raw
Quoted 14 lines · On Wed, 29 Sep 2010 07:22:06 -0500 Jack Steiner <[email protected]> wrote:

I'm looking for suggestions on how to fix a scaling problem with access to /proc/stat.

On a large x86_64 system (4096p, 256 nodes, 5530 IRQs), access to /proc/stat takes too long - more than 12 sec:

	# time cat /proc/stat >/dev/null
	real	12.630s
	user	 0.000s
	sys	12.629s

This affects top, ps (some variants), w, glibc (sysconf) and much more.

Quoted 7 lines

One of the items reported in /proc/stat is a total count of interrupts that have been received. This calculation requires summation of the interrupts received on each cpu (kstat_irqs_cpu()).

The data is kept in per-cpu arrays linked to each irq_desc. On a 4096p/5530IRQ system summing this data requires accessing ~90MB.

Wow.

Quoted 4 lines

Deleting the summation of the kstat_irqs_cpu data eliminates the high access time but is an API breakage that I assume is unacceptible.

Quoted 9 lines

Another possibility would be using delayed work (similar to vmstat_update) that periodically sums the data into a single array. The disadvantage in this approach is that there would be a delay between receipt of an interrupt & it's count appearing /proc/stat. Is this an issue for anyone? Another disadvantage is that it adds to the overall "noise" introduced by kernel threads.

Is there a better approach to take?

Hmm, this ?
==
From: KAMEZAWA Hiroyuki <[email protected]>

/proc/stat shows the total number of all interrupts to each cpu. But when the number of IRQs are very large, it take very long time and 'cat /proc/stat' takes more than 10 secs. This is because sum of all irq events are counted when /proc/stat is read. This patch adds "sum of all irq" counter percpu and reduce read costs.

---
 fs/proc/stat.c              |    4 +---
 include/linux/kernel_stat.h |   14 ++++++++++++--
 2 files changed, 13 insertions(+), 5 deletions(-)
fs/proc/stat.c +1 -3
Index: mmotm-0922/fs/proc/stat.c
===================================================================
@@ -52,9 +52,7 @@ static int show_stat(struct seq_file *p,
 		guest = cputime64_add(guest, kstat_cpu(i).cpustat.guest);
 		guest_nice = cputime64_add(guest_nice,
 			kstat_cpu(i).cpustat.guest_nice);
-		for_each_irq_nr(j) {
-			sum += kstat_irqs_cpu(j, i);
-		}
+		sum = kstat_cpu_irqs_sum(i);
 		sum += arch_irq_stat_cpu(i);
 
 		for (j = 0; j < NR_SOFTIRQS; j++) {
include/linux/kernel_stat.h +12 -2
Index: mmotm-0922/include/linux/kernel_stat.h
===================================================================
@@ -33,6 +33,7 @@ struct kernel_stat {
 #ifndef CONFIG_GENERIC_HARDIRQS
        unsigned int irqs[NR_IRQS];
 #endif
+	unsigned long irqs_sum;
 	unsigned int softirqs[NR_SOFTIRQS];
 };
 
@@ -54,6 +55,7 @@ static inline void kstat_incr_irqs_this_
 					    struct irq_desc *desc)
 {
 	kstat_this_cpu.irqs[irq]++;
+	kstat_this_cpu.irqs_sum++;
 }
 
 static inline unsigned int kstat_irqs_cpu(unsigned int irq, int cpu)
@@ -65,8 +67,9 @@ static inline unsigned int kstat_irqs_cp
 extern unsigned int kstat_irqs_cpu(unsigned int irq, int cpu);
 #define kstat_irqs_this_cpu(DESC) \
 	((DESC)->kstat_irqs[smp_processor_id()])
-#define kstat_incr_irqs_this_cpu(irqno, DESC) \
-	((DESC)->kstat_irqs[smp_processor_id()]++)
+#define kstat_incr_irqs_this_cpu(irqno, DESC) do {\
+	((DESC)->kstat_irqs[smp_processor_id()]++);\
+	kstat_this_cpu.irqs_sum++;} while (0)
 
 #endif
 
@@ -94,6 +97,13 @@ static inline unsigned int kstat_irqs(un
 	return sum;
 }
 
+/*
+ * Number of interrupts per cpu, since bootup
+ */
+static inline unsigned long kstat_cpu_irqs_sum(unsigned int cpu)
+{
+	return kstat_cpu(cpu).irqs_sum;
+}
 
 /*
  * Lock/unlock the current runqueue - to extract task statistics:
#2 lore raw
Quoted 2 lines · KAMEZAWA Hiroyuki

On Wed, 29 Sep 2010 07:22:06 -0500
Jack Steiner <[email protected]> wrote:

I was able to run on the 4096p system over the weekend. The patch is a definite improvement & partially fixes the problem:

A "cat /proc/stat >/dev/null" improved:

        OLD:    real    12.627s
        NEW:    real     2.459

A large part of the remaining overhead is in the second summation
of irq information:

    static int show_stat(struct seq_file *p, void *v)
        ...
        /* sum again ? it could be updated? */
        for_each_irq_nr(j) {
                per_irq_sum = 0;
                for_each_possible_cpu(i)
                        per_irq_sum += kstat_irqs_cpu(j, i);

                seq_printf(p, " %u", per_irq_sum);
        }

Can this be fixed using the same approach as in the current patch?

--- jack

Quoted 116 lines
Quoted 15 lines

I'm looking for suggestions on how to fix a scaling problem with access to /proc/stat.

On a large x86_64 system (4096p, 256 nodes, 5530 IRQs), access to /proc/stat takes too long - more than 12 sec:

	# time cat /proc/stat >/dev/null
	real	12.630s
	user	 0.000s
	sys	12.629s

This affects top, ps (some variants), w, glibc (sysconf) and much more.

Quoted 7 lines

One of the items reported in /proc/stat is a total count of interrupts that have been received. This calculation requires summation of the interrupts received on each cpu (kstat_irqs_cpu()).

The data is kept in per-cpu arrays linked to each irq_desc. On a 4096p/5530IRQ system summing this data requires accessing ~90MB.

Wow.

Quoted 4 lines

Deleting the summation of the kstat_irqs_cpu data eliminates the high access time but is an API breakage that I assume is unacceptible.

Quoted 9 lines

Another possibility would be using delayed work (similar to vmstat_update) that periodically sums the data into a single array. The disadvantage in this approach is that there would be a delay between receipt of an interrupt & it's count appearing /proc/stat. Is this an issue for anyone? Another disadvantage is that it adds to the overall "noise" introduced by kernel threads.

Is there a better approach to take?

Hmm, this ?
==
From: KAMEZAWA Hiroyuki <[email protected]>

/proc/stat shows the total number of all interrupts to each cpu. But when the number of IRQs are very large, it take very long time and 'cat /proc/stat' takes more than 10 secs. This is because sum of all irq events are counted when /proc/stat is read. This patch adds "sum of all irq" counter percpu and reduce read costs.

---
 fs/proc/stat.c              |    4 +---
 include/linux/kernel_stat.h |   14 ++++++++++++--
 2 files changed, 13 insertions(+), 5 deletions(-)

Index: mmotm-0922/fs/proc/stat.c
===================================================================

fs/proc/stat.c +1 -3
@@ -52,9 +52,7 @@ static int show_stat(struct seq_file *p,
 		guest = cputime64_add(guest, kstat_cpu(i).cpustat.guest);
 		guest_nice = cputime64_add(guest_nice,
 			kstat_cpu(i).cpustat.guest_nice);
-		for_each_irq_nr(j) {
-			sum += kstat_irqs_cpu(j, i);
-		}
+		sum = kstat_cpu_irqs_sum(i);
 		sum += arch_irq_stat_cpu(i);
 
 		for (j = 0; j < NR_SOFTIRQS; j++) {

Index: mmotm-0922/include/linux/kernel_stat.h
===================================================================

include/linux/kernel_stat.h +12 -2
@@ -33,6 +33,7 @@ struct kernel_stat {
 #ifndef CONFIG_GENERIC_HARDIRQS
        unsigned int irqs[NR_IRQS];
 #endif
+	unsigned long irqs_sum;
 	unsigned int softirqs[NR_SOFTIRQS];
 };
 
@@ -54,6 +55,7 @@ static inline void kstat_incr_irqs_this_
 					    struct irq_desc *desc)
 {
 	kstat_this_cpu.irqs[irq]++;
+	kstat_this_cpu.irqs_sum++;
 }
 
 static inline unsigned int kstat_irqs_cpu(unsigned int irq, int cpu)
@@ -65,8 +67,9 @@ static inline unsigned int kstat_irqs_cp
 extern unsigned int kstat_irqs_cpu(unsigned int irq, int cpu);
 #define kstat_irqs_this_cpu(DESC) \
 	((DESC)->kstat_irqs[smp_processor_id()])
-#define kstat_incr_irqs_this_cpu(irqno, DESC) \
-	((DESC)->kstat_irqs[smp_processor_id()]++)
+#define kstat_incr_irqs_this_cpu(irqno, DESC) do {\
+	((DESC)->kstat_irqs[smp_processor_id()]++);\
+	kstat_this_cpu.irqs_sum++;} while (0)
 
 #endif
 
@@ -94,6 +97,13 @@ static inline unsigned int kstat_irqs(un
 	return sum;
 }
 
+/*
+ * Number of interrupts per cpu, since bootup
+ */
+static inline unsigned long kstat_cpu_irqs_sum(unsigned int cpu)
+{
+	return kstat_cpu(cpu).irqs_sum;
+}
 
 /*
  * Lock/unlock the current runqueue - to extract task statistics:
#3 lore raw
Quoted 5 lines · On Mon, 4 Oct 2010 09:34:15 -0500 Jack Steiner <[email protected]> wrote:
Quoted 2 lines · KAMEZAWA Hiroyuki

On Wed, 29 Sep 2010 07:22:06 -0500
Jack Steiner <[email protected]> wrote:

Quoted 9 lines

I was able to run on the 4096p system over the weekend. The patch is a definite improvement & partially fixes the problem:

A "cat /proc/stat >/dev/null" improved:

        OLD:    real    12.627s
        NEW:    real     2.459

Thank you.

Quoted 15 lines

A large part of the remaining overhead is in the second summation
of irq information:

    static int show_stat(struct seq_file *p, void *v)
        ...
        /* sum again ? it could be updated? */
        for_each_irq_nr(j) {
                per_irq_sum = 0;
                for_each_possible_cpu(i)
                        per_irq_sum += kstat_irqs_cpu(j, i);

                seq_printf(p, " %u", per_irq_sum);
        }
Quoted 3 lines

Can this be fixed using the same approach as in the current patch?

I guess this requres different approarch as per-cpu counter + threshould. like vmstat[] or lib/percpu_counter.
Maybe people don't like to access shared counter in IRQ.

But, this seems to call radixtree-lookup for the # of possible cpus. I guess impleimenting a call to calculate a sum of irqs in a radix-tree lookup will reduce overhead. If it's not enough, we'll have to make the counter not-precise. I'll write an another patch.

Thanks,
-Kame

#4 lore raw
Quoted 9 lines · On Tue, 5 Oct 2010 10:36:50 +0900 KAMEZAWA Hiroyuki…

I guess this requres different approarch as per-cpu counter + threshould. like vmstat[] or lib/percpu_counter.
Maybe people don't like to access shared counter in IRQ.

But, this seems to call radixtree-lookup for the # of possible cpus. I guess impleimenting a call to calculate a sum of irqs in a radix-tree lookup will reduce overhead. If it's not enough, we'll have to make the counter not-precise. I'll write an another patch.

How about this ? This is an add-on patch.
==
From: KAMEZAWA Hiroyuki <[email protected]>

In /proc/stat, the number of per-IRQ event is shown by making a sum each irq's events on all cpus. But we can make use of kstat_irqs().

kstat_irqs() make a sum of IRQ events per cpu, if !CONFIG_GENERIC_HARDIRQ, it's not a big cost. (Both of the number of cpus and irqs are small.)

If a system is very big, it does

	for_each_irq()
		for_each_cpu()
			- look up a radix tree
			- read desc->irq_stat[cpu]

This seems not efficient. This patch adds kstat_irqs() for CONFIG_GENRIC_HARDIRQ and change the calculation as

	for_each_irq()
		look up radix tree
		for_each_cpu()
			- read desc->irq_stat[cpu]

and reduces cost.

---
 fs/proc/stat.c              |    9 ++-------
 include/linux/kernel_stat.h |    5 +++++
 kernel/irq/handle.c         |   16 ++++++++++++++++
 3 files changed, 23 insertions(+), 7 deletions(-)
fs/proc/stat.c +2 -7
Index: mmotm-0928/fs/proc/stat.c
===================================================================
@@ -108,13 +108,8 @@ static int show_stat(struct seq_file *p,
 	seq_printf(p, "intr %llu", (unsigned long long)sum);
 
 	/* sum again ? it could be updated? */
-	for_each_irq_nr(j) {
-		per_irq_sum = 0;
-		for_each_possible_cpu(i)
-			per_irq_sum += kstat_irqs_cpu(j, i);
-
-		seq_printf(p, " %u", per_irq_sum);
-	}
+	for_each_irq_nr(j)
+		seq_printf(p, " %u", kstat_irqs(j));
 
 	seq_printf(p,
 		"\nctxt %llu\n"
include/linux/kernel_stat.h +5 -0
Index: mmotm-0928/include/linux/kernel_stat.h
===================================================================
@@ -62,6 +62,7 @@ static inline unsigned int kstat_irqs_cp
 {
        return kstat_cpu(cpu).irqs[irq];
 }
+
 #else
 #include <linux/irq.h>
 extern unsigned int kstat_irqs_cpu(unsigned int irq, int cpu);
@@ -86,6 +87,7 @@ static inline unsigned int kstat_softirq
 /*
  * Number of interrupts per specific IRQ source, since bootup
  */
+#ifndef CONFIG_GENERIC_HARDIRQS
 static inline unsigned int kstat_irqs(unsigned int irq)
 {
 	unsigned int sum = 0;
@@ -96,6 +98,9 @@ static inline unsigned int kstat_irqs(un
 
 	return sum;
 }
+#else
+extern unsigned int unsigned int kstat_irqs(unsigned int irq);
+#endif
 
 /*
  * Number of interrupts per cpu, since bootup
kernel/irq/handle.c +16 -0
Index: mmotm-0928/kernel/irq/handle.c
===================================================================
@@ -553,3 +553,19 @@ unsigned int kstat_irqs_cpu(unsigned int
 }
 EXPORT_SYMBOL(kstat_irqs_cpu);
 
+#ifdef CONFIG_GENERIC_HARDIRQS
+unsigned int kstat_irqs(unsigned int irq)
+{
+	struct irq_desc *desc = irq_to_desc(irq);
+	int cpu;
+	int sum = 0;
+
+	if (!desc)
+		return 0;
+
+	for_each_possible_cpu(cpu)
+		sum += desc->kstat_irqs[cpu];
+	return sum;
+}
+EXPORT_SYMBOL(kstat_irqs);
+#endif
author Acked-by
#5 lore raw
Quoted 7 lines · KAMEZAWA Hiroyuki
Quoted 4 lines · On Tue, 5 Oct 2010 10:36:50 +0900 KAMEZAWA Hiroyuki…

I guess this requres different approarch as per-cpu counter + threshould. like vmstat[] or lib/percpu_counter.
Maybe people don't like to access shared counter in IRQ.

Quoted 7 lines
Quoted 5 lines

But, this seems to call radixtree-lookup for the # of possible cpus. I guess impleimenting a call to calculate a sum of irqs in a radix-tree lookup will reduce overhead. If it's not enough, we'll have to make the counter not-precise. I'll write an another patch.

How about this ? This is an add-on patch.

Nice!!

The combination of the 2 patches solves the problem. The timings are (4096p, 256 nodes, 4592 irqs):

	# time cat /proc/stat > /dev/null
	Baseline:		12.627 sec
	Patch1  :		 2.459 sec
	Patch 1 + Patch 2:	  .561 sec

Thanks!!
--- jack

Quoted 106 lines

==
From: KAMEZAWA Hiroyuki <[email protected]>

In /proc/stat, the number of per-IRQ event is shown by making a sum each irq's events on all cpus. But we can make use of kstat_irqs().

kstat_irqs() make a sum of IRQ events per cpu, if !CONFIG_GENERIC_HARDIRQ, it's not a big cost. (Both of the number of cpus and irqs are small.)

If a system is very big, it does

	for_each_irq()
		for_each_cpu()
			- look up a radix tree
			- read desc->irq_stat[cpu]

This seems not efficient. This patch adds kstat_irqs() for CONFIG_GENRIC_HARDIRQ and change the calculation as

	for_each_irq()
		look up radix tree
		for_each_cpu()
			- read desc->irq_stat[cpu]

and reduces cost.

---
 fs/proc/stat.c              |    9 ++-------
 include/linux/kernel_stat.h |    5 +++++
 kernel/irq/handle.c         |   16 ++++++++++++++++
 3 files changed, 23 insertions(+), 7 deletions(-)

Index: mmotm-0928/fs/proc/stat.c
===================================================================

fs/proc/stat.c +2 -7
@@ -108,13 +108,8 @@ static int show_stat(struct seq_file *p,
 	seq_printf(p, "intr %llu", (unsigned long long)sum);
 
 	/* sum again ? it could be updated? */
-	for_each_irq_nr(j) {
-		per_irq_sum = 0;
-		for_each_possible_cpu(i)
-			per_irq_sum += kstat_irqs_cpu(j, i);
-
-		seq_printf(p, " %u", per_irq_sum);
-	}
+	for_each_irq_nr(j)
+		seq_printf(p, " %u", kstat_irqs(j));
 
 	seq_printf(p,
 		"\nctxt %llu\n"

Index: mmotm-0928/include/linux/kernel_stat.h
===================================================================

include/linux/kernel_stat.h +5 -0
@@ -62,6 +62,7 @@ static inline unsigned int kstat_irqs_cp
 {
        return kstat_cpu(cpu).irqs[irq];
 }
+
 #else
 #include <linux/irq.h>
 extern unsigned int kstat_irqs_cpu(unsigned int irq, int cpu);
@@ -86,6 +87,7 @@ static inline unsigned int kstat_softirq
 /*
  * Number of interrupts per specific IRQ source, since bootup
  */
+#ifndef CONFIG_GENERIC_HARDIRQS
 static inline unsigned int kstat_irqs(unsigned int irq)
 {
 	unsigned int sum = 0;
@@ -96,6 +98,9 @@ static inline unsigned int kstat_irqs(un
 
 	return sum;
 }
+#else
+extern unsigned int unsigned int kstat_irqs(unsigned int irq);
+#endif
 
 /*
  * Number of interrupts per cpu, since bootup

Index: mmotm-0928/kernel/irq/handle.c
===================================================================

kernel/irq/handle.c +16 -0
@@ -553,3 +553,19 @@ unsigned int kstat_irqs_cpu(unsigned int
 }
 EXPORT_SYMBOL(kstat_irqs_cpu);
 
+#ifdef CONFIG_GENERIC_HARDIRQS
+unsigned int kstat_irqs(unsigned int irq)
+{
+	struct irq_desc *desc = irq_to_desc(irq);
+	int cpu;
+	int sum = 0;
+
+	if (!desc)
+		return 0;
+
+	for_each_possible_cpu(cpu)
+		sum += desc->kstat_irqs[cpu];
+	return sum;
+}
+EXPORT_SYMBOL(kstat_irqs);
+#endif
#6 lore raw
Quoted 21 lines · On Fri, 8 Oct 2010 11:35:57 -0500 Jack Steiner <[email protected]> wrote:
Quoted 7 lines · KAMEZAWA Hiroyuki
Quoted 4 lines · On Tue, 5 Oct 2010 10:36:50 +0900 KAMEZAWA Hiroyuki…

I guess this requres different approarch as per-cpu counter + threshould. like vmstat[] or lib/percpu_counter.
Maybe people don't like to access shared counter in IRQ.

Quoted 7 lines
Quoted 5 lines

But, this seems to call radixtree-lookup for the # of possible cpus. I guess impleimenting a call to calculate a sum of irqs in a radix-tree lookup will reduce overhead. If it's not enough, we'll have to make the counter not-precise. I'll write an another patch.

How about this ? This is an add-on patch.

Nice!!

The combination of the 2 patches solves the problem. The timings are (4096p, 256 nodes, 4592 irqs):

Quoted 9 lines
	# time cat /proc/stat > /dev/null
	Baseline:		12.627 sec
	Patch1  :		 2.459 sec
	Patch 1 + Patch 2:	  .561 sec

Thank you for testing. I'll post again if necessary.

-Kame

#7 lore raw
Quoted 15 lines · On Tue, 12 Oct 2010 09:09:07 +0900 KAMEZAWA Hiroyuki…
Quoted 15 lines
Quoted 2 lines

How about this ? This is an add-on patch.

Nice!!

The combination of the 2 patches solves the problem. The timings are (4096p, 256 nodes, 4592 irqs):

	# time cat /proc/stat > /dev/null
	Baseline:		12.627 sec
	Patch1  :		 2.459 sec
	Patch 1 + Patch 2:	  .561 sec
Quoted 4 lines
Quoted 2 lines

Thank you for testing. I'll post again if necessary.

Yes please. This has been going on for two weeks so memories need refreshing. Also we have no patch title and no changelog which includes the testing results.

I could stitch all that together of course, but it's best that it all be done afresh, I think.

#8 lore raw
Quoted 20 lines · On Mon, 11 Oct 2010 17:22:26 -0700 Andrew Morton <[email protected]>…
Quoted 18 lines · On Tue, 12 Oct 2010 09:09:07 +0900 KAMEZAWA Hiroyuki…
Quoted 17 lines
Quoted 2 lines

How about this ? This is an add-on patch.

Nice!!

The combination of the 2 patches solves the problem. The timings are (4096p, 256 nodes, 4592 irqs):

	# time cat /proc/stat > /dev/null
	Baseline:		12.627 sec
	Patch1  :		 2.459 sec
	Patch 1 + Patch 2:	  .561 sec
Quoted 9 lines
Quoted 1 line

Thank you for testing. I'll post again if necessary.

Yes please. This has been going on for two weeks so memories need refreshing. Also we have no patch title and no changelog which includes the testing results.

I could stitch all that together of course, but it's best that it all be done afresh, I think.

ok, will do today.

-Kame

#9 lore raw

PATCH 1/2 fix slowness of /proc/stat per-cpu IRQ sum calculation on large system by a new counter

In reply to Andrew Morton

Jack Steiner reported slowness of /proc/stat on a large system. This patch set tries to improve it.

Quoted 8 lines

The combination of the 2 patches solves the problem. The timings are (4096p, 256 nodes, 4592 irqs):

	# time cat /proc/stat > /dev/null
	Baseline:		12.627 sec
	Patch1  :		 2.459 sec
	Patch 1 + Patch 2:	  .561 sec

please review.

==
From: KAMEZAWA Hiroyuki <[email protected]>

Problem: 'cat /proc/stat' is too slow on verrry bis system.

/proc/stat shows the total number of all interrupts to each cpu. But when the number of IRQs are very large, it takes very long time and 'cat /proc/stat' takes more than 10 secs. This is because sum of all irq events are counted when /proc/stat is read. This patch adds "sum of all irq" counter percpu and update it at events.

The cost of reading /proc/stat is important because it's used by major applications as 'top', 'ps', 'w', etc....

A test on a host (4096cpu, 256 nodes, 4592 irqs) shows

%time cat /proc/stat > /dev/null
Before Patch: 12.627 sec
After Patch: 2.459 sec

---
 fs/proc/stat.c              |    4 +---
 include/linux/kernel_stat.h |   14 ++++++++++++--
 2 files changed, 13 insertions(+), 5 deletions(-)
fs/proc/stat.c +1 -3
Index: linux-2.6.36-rc7/fs/proc/stat.c
===================================================================
@@ -52,9 +52,7 @@ static int show_stat(struct seq_file *p,
 		guest = cputime64_add(guest, kstat_cpu(i).cpustat.guest);
 		guest_nice = cputime64_add(guest_nice,
 			kstat_cpu(i).cpustat.guest_nice);
-		for_each_irq_nr(j) {
-			sum += kstat_irqs_cpu(j, i);
-		}
+		sum = kstat_cpu_irqs_sum(i);
 		sum += arch_irq_stat_cpu(i);
 
 		for (j = 0; j < NR_SOFTIRQS; j++) {
include/linux/kernel_stat.h +12 -2
Index: linux-2.6.36-rc7/include/linux/kernel_stat.h
===================================================================
@@ -33,6 +33,7 @@ struct kernel_stat {
 #ifndef CONFIG_GENERIC_HARDIRQS
        unsigned int irqs[NR_IRQS];
 #endif
+	unsigned long irqs_sum;
 	unsigned int softirqs[NR_SOFTIRQS];
 };
 
@@ -54,6 +55,7 @@ static inline void kstat_incr_irqs_this_
 					    struct irq_desc *desc)
 {
 	kstat_this_cpu.irqs[irq]++;
+	kstat_this_cpu.irqs_sum++;
 }
 
 static inline unsigned int kstat_irqs_cpu(unsigned int irq, int cpu)
@@ -65,8 +67,9 @@ static inline unsigned int kstat_irqs_cp
 extern unsigned int kstat_irqs_cpu(unsigned int irq, int cpu);
 #define kstat_irqs_this_cpu(DESC) \
 	((DESC)->kstat_irqs[smp_processor_id()])
-#define kstat_incr_irqs_this_cpu(irqno, DESC) \
-	((DESC)->kstat_irqs[smp_processor_id()]++)
+#define kstat_incr_irqs_this_cpu(irqno, DESC) do {\
+	((DESC)->kstat_irqs[smp_processor_id()]++);\
+	kstat_this_cpu.irqs_sum++; } while (0)
 
 #endif
 
@@ -94,6 +97,13 @@ static inline unsigned int kstat_irqs(un
 	return sum;
 }
 
+/*
+ * Number of interrupts per cpu, since bootup
+ */
+static inline unsigned int kstat_cpu_irqs_sum(unsigned int cpu)
+{
+	return kstat_cpu(cpu).irqs_sum;
+}
 
 /*
  * Lock/unlock the current runqueue - to extract task statistics:
#10 lore raw

PATCH 2/2 improve footprint of kstat_irqs() for large system's /proc/stat

From: KAMEZAWA Hiroyuki <[email protected]>

Problem: 'cat /proc/stat' takes too long in verrrry big system.

In /proc/stat, the number of per-IRQ event is shown by making a sum each irq's events on all cpus. We can make use of kstat_irqs(). This patch replaces it, at first.

Here, performance of kstat_irqs() is a problem.
If !CONFIG_GENERIC_HARDIRQ, it's not very slow, but if CONFIG_GENERIC_HARDIRQ, it's slow because the logic is not efficient.

If CONFIG_GENERIC_HARDIRQ, it does

	for_each_irq()
		for_each_cpu()
			- look up a radix tree
			- read desc->irq_stat[cpu]

This seems not efficient. This patch adds kstat_irqs() for CONFIG_GENRIC_HARDIRQ and change the calculation as

	for_each_irq()
		look up radix tree
		for_each_cpu()
			- read desc->irq_stat[cpu]

This reduces cost of scanning.

A test on (4096cpusp, 256 nodes, 4592 irqs) host (by Jack Steiner)

%time cat /proc/stat > /dev/null

Before Patch: 2.459 sec
After Patch : .561 sec

---
 fs/proc/stat.c              |    9 ++-------
 include/linux/kernel_stat.h |    4 ++++
 kernel/irq/handle.c         |   16 ++++++++++++++++
 3 files changed, 22 insertions(+), 7 deletions(-)
fs/proc/stat.c +2 -7
Index: linux-2.6.36-rc7/fs/proc/stat.c
===================================================================
@@ -108,13 +108,8 @@ static int show_stat(struct seq_file *p,
 	seq_printf(p, "intr %llu", (unsigned long long)sum);
 
 	/* sum again ? it could be updated? */
-	for_each_irq_nr(j) {
-		per_irq_sum = 0;
-		for_each_possible_cpu(i)
-			per_irq_sum += kstat_irqs_cpu(j, i);
-
-		seq_printf(p, " %u", per_irq_sum);
-	}
+	for_each_irq_nr(j)
+		seq_printf(p, " %u", kstat_irqs(j));
 
 	seq_printf(p,
 		"\nctxt %llu\n"
include/linux/kernel_stat.h +4 -0
Index: linux-2.6.36-rc7/include/linux/kernel_stat.h
===================================================================
@@ -86,6 +86,7 @@ static inline unsigned int kstat_softirq
 /*
  * Number of interrupts per specific IRQ source, since bootup
  */
+#ifndef CONFIG_GENERIC_HARDIRQS
 static inline unsigned int kstat_irqs(unsigned int irq)
 {
 	unsigned int sum = 0;
@@ -96,6 +97,9 @@ static inline unsigned int kstat_irqs(un
 
 	return sum;
 }
+#else
+extern unsigned int kstat_irqs(unsigned int irq);
+#endif
 
 /*
  * Number of interrupts per cpu, since bootup
kernel/irq/handle.c +16 -0
Index: linux-2.6.36-rc7/kernel/irq/handle.c
===================================================================
@@ -554,3 +554,19 @@ unsigned int kstat_irqs_cpu(unsigned int
 }
 EXPORT_SYMBOL(kstat_irqs_cpu);
 
+#ifdef CONFIG_GENERIC_HARDIRQS
+unsigned int kstat_irqs(unsigned int irq)
+{
+	struct irq_desc *desc = irq_to_desc(irq);
+	int cpu;
+	int sum = 0;
+
+	if (!desc)
+		return 0;
+
+	for_each_possible_cpu(cpu)
+		sum += desc->kstat_irqs[cpu];
+	return sum;
+}
+EXPORT_SYMBOL(kstat_irqs);
+#endif
#11 lore raw

Re: [PATCH 1/2] fix slowness of /proc/stat per-cpu IRQ sum calculation on large system by a new counter

In reply to KAMEZAWA Hiroyuki
Quoted 47 lines · KAMEZAWA Hiroyuki

Jack Steiner reported slowness of /proc/stat on a large system. This patch set tries to improve it.

Quoted 8 lines

The combination of the 2 patches solves the problem. The timings are (4096p, 256 nodes, 4592 irqs):

	# time cat /proc/stat > /dev/null
	Baseline:		12.627 sec
	Patch1  :		 2.459 sec
	Patch 1 + Patch 2:	  .561 sec

please review.

==
From: KAMEZAWA Hiroyuki <[email protected]>

Problem: 'cat /proc/stat' is too slow on verrry bis system.

/proc/stat shows the total number of all interrupts to each cpu. But when the number of IRQs are very large, it takes very long time and 'cat /proc/stat' takes more than 10 secs. This is because sum of all irq events are counted when /proc/stat is read. This patch adds "sum of all irq" counter percpu and update it at events.

The cost of reading /proc/stat is important because it's used by major applications as 'top', 'ps', 'w', etc....

A test on a host (4096cpu, 256 nodes, 4592 irqs) shows

%time cat /proc/stat > /dev/null
Before Patch: 12.627 sec
After Patch: 2.459 sec

---
 fs/proc/stat.c              |    4 +---
 include/linux/kernel_stat.h |   14 ++++++++++++--
 2 files changed, 13 insertions(+), 5 deletions(-)

Index: linux-2.6.36-rc7/fs/proc/stat.c
===================================================================

fs/proc/stat.c static int show_stat(struct seq_file *p, · line 55

Quoted 7 lines
fs/proc/stat.c +1 -3
@@ -52,9 +52,7 @@ static int show_stat(struct seq_file *p,
 		guest = cputime64_add(guest, kstat_cpu(i).cpustat.guest);
 		guest_nice = cputime64_add(guest_nice,
 			kstat_cpu(i).cpustat.guest_nice);
-		for_each_irq_nr(j) {
-			sum += kstat_irqs_cpu(j, i);
-		}
+		sum = kstat_cpu_irqs_sum(i);

should be
+ sum += kstat_cpu_irqs_sum(i);

Quoted 49 lines
fs/proc/stat.c +0 -0
 		sum += arch_irq_stat_cpu(i);
 
 		for (j = 0; j < NR_SOFTIRQS; j++) {

Index: linux-2.6.36-rc7/include/linux/kernel_stat.h
===================================================================

include/linux/kernel_stat.h +12 -2
@@ -33,6 +33,7 @@ struct kernel_stat {
 #ifndef CONFIG_GENERIC_HARDIRQS
        unsigned int irqs[NR_IRQS];
 #endif
+	unsigned long irqs_sum;
 	unsigned int softirqs[NR_SOFTIRQS];
 };
 
@@ -54,6 +55,7 @@ static inline void kstat_incr_irqs_this_
 					    struct irq_desc *desc)
 {
 	kstat_this_cpu.irqs[irq]++;
+	kstat_this_cpu.irqs_sum++;
 }
 
 static inline unsigned int kstat_irqs_cpu(unsigned int irq, int cpu)
@@ -65,8 +67,9 @@ static inline unsigned int kstat_irqs_cp
 extern unsigned int kstat_irqs_cpu(unsigned int irq, int cpu);
 #define kstat_irqs_this_cpu(DESC) \
 	((DESC)->kstat_irqs[smp_processor_id()])
-#define kstat_incr_irqs_this_cpu(irqno, DESC) \
-	((DESC)->kstat_irqs[smp_processor_id()]++)
+#define kstat_incr_irqs_this_cpu(irqno, DESC) do {\
+	((DESC)->kstat_irqs[smp_processor_id()]++);\
+	kstat_this_cpu.irqs_sum++; } while (0)
 
 #endif
 
@@ -94,6 +97,13 @@ static inline unsigned int kstat_irqs(un
 	return sum;
 }
 
+/*
+ * Number of interrupts per cpu, since bootup
+ */
+static inline unsigned int kstat_cpu_irqs_sum(unsigned int cpu)
+{
+	return kstat_cpu(cpu).irqs_sum;
+}
 
 /*
  * Lock/unlock the current runqueue - to extract task statistics:
Tested-by Jack Steiner Acked-by Jack Steiner
#12 lore raw

Re: [PATCH 1/2] fix slowness of /proc/stat per-cpu IRQ sum calculation on large system by a new counter

On Mon, 11 Oct 2010 20:05:59 -0700
Yinghai Lu <[email protected]> wrote:

			kstat_cpu(i).cpustat.guest_nice);

Review

Quoted 8 lines

Review

Quoted 4 lines
(quoted diff) +1 -3
-		for_each_irq_nr(j) {
-			sum += kstat_irqs_cpu(j, i);
-		}
+		sum = kstat_cpu_irqs_sum(i);

should be

(quoted diff) +1 -0
+		sum += kstat_cpu_irqs_sum(i);

Ouch...thanks.
==
From: KAMEZAWA Hiroyuki <[email protected]>

Problem: 'cat /proc/stat' is too slow on verrry bis system.

/proc/stat shows the total number of all interrupts to each cpu. But when the number of IRQs are very large, it takes very long time and 'cat /proc/stat' takes more than 10 secs. This is because sum of all irq events are counted when /proc/stat is read. This patch adds "sum of all irq" counter percpu and update it at events.

The cost of reading /proc/stat is important because it's used by major applications as 'top', 'ps', 'w', etc....

A test on a host (4096cpu, 256 nodes, 4592 irqs) shows

%time cat /proc/stat > /dev/null
Before Patch: 12.627 sec
After Patch: 2.459 sec

Changelog v1->v2:
- fixed wrong update of a value "sum".

---
 fs/proc/stat.c              |    4 +---
 include/linux/kernel_stat.h |   14 ++++++++++++--
 2 files changed, 13 insertions(+), 5 deletions(-)
fs/proc/stat.c +1 -3
Index: linux-2.6.36-rc7/fs/proc/stat.c
===================================================================
@@ -52,9 +52,7 @@ static int show_stat(struct seq_file *p,
 		guest = cputime64_add(guest, kstat_cpu(i).cpustat.guest);
 		guest_nice = cputime64_add(guest_nice,
 			kstat_cpu(i).cpustat.guest_nice);
-		for_each_irq_nr(j) {
-			sum += kstat_irqs_cpu(j, i);
-		}
+		sum += kstat_cpu_irqs_sum(i);
 		sum += arch_irq_stat_cpu(i);
 
 		for (j = 0; j < NR_SOFTIRQS; j++) {
include/linux/kernel_stat.h +12 -2
Index: linux-2.6.36-rc7/include/linux/kernel_stat.h
===================================================================
@@ -33,6 +33,7 @@ struct kernel_stat {
 #ifndef CONFIG_GENERIC_HARDIRQS
        unsigned int irqs[NR_IRQS];
 #endif
+	unsigned long irqs_sum;
 	unsigned int softirqs[NR_SOFTIRQS];
 };
 
@@ -54,6 +55,7 @@ static inline void kstat_incr_irqs_this_
 					    struct irq_desc *desc)
 {
 	kstat_this_cpu.irqs[irq]++;
+	kstat_this_cpu.irqs_sum++;
 }
 
 static inline unsigned int kstat_irqs_cpu(unsigned int irq, int cpu)
@@ -65,8 +67,9 @@ static inline unsigned int kstat_irqs_cp
 extern unsigned int kstat_irqs_cpu(unsigned int irq, int cpu);
 #define kstat_irqs_this_cpu(DESC) \
 	((DESC)->kstat_irqs[smp_processor_id()])
-#define kstat_incr_irqs_this_cpu(irqno, DESC) \
-	((DESC)->kstat_irqs[smp_processor_id()]++)
+#define kstat_incr_irqs_this_cpu(irqno, DESC) do {\
+	((DESC)->kstat_irqs[smp_processor_id()]++);\
+	kstat_this_cpu.irqs_sum++; } while (0)
 
 #endif
 
@@ -94,6 +97,13 @@ static inline unsigned int kstat_irqs(un
 	return sum;
 }
 
+/*
+ * Number of interrupts per cpu, since bootup
+ */
+static inline unsigned int kstat_cpu_irqs_sum(unsigned int cpu)
+{
+	return kstat_cpu(cpu).irqs_sum;
+}
 
 /*
  * Lock/unlock the current runqueue - to extract task statistics: