Problem: scaling of /proc/stat on large systems
Subsystems (2)
I'm looking for suggestions on how to fix a scaling problem with access to /proc/stat.
On a large x86_64 system (4096p, 256 nodes, 5530 IRQs), access to /proc/stat takes too long - more than 12 sec:
# time cat /proc/stat >/dev/null real 12.630s user 0.000s sys 12.629s
This affects top, ps (some variants), w, glibc (sysconf) and much more.
One of the items reported in /proc/stat is a total count of interrupts that have been received. This calculation requires summation of the interrupts received on each cpu (kstat_irqs_cpu()).
The data is kept in per-cpu arrays linked to each irq_desc. On a 4096p/5530IRQ system summing this data requires accessing ~90MB.
Deleting the summation of the kstat_irqs_cpu data eliminates the high access time but is an API breakage that I assume is unacceptible.
Another possibility would be using delayed work (similar to vmstat_update) that periodically sums the data into a single array. The disadvantage in this approach is that there would be a delay between receipt of an interrupt & it's count appearing /proc/stat. Is this an issue for anyone? Another disadvantage is that it adds to the overall "noise" introduced by kernel threads.
Is there a better approach to take?
--- jack
Quoted 14 lines · On Wed, 29 Sep 2010 07:22:06 -0500 Jack Steiner <[email protected]> wrote:
I'm looking for suggestions on how to fix a scaling problem with access to
/proc/stat.On a large
x86_64system (4096p, 256 nodes, 5530 IRQs), access to/proc/stattakes too long - more than 12 sec:# time cat /proc/stat >/dev/null real 12.630s user 0.000s sys 12.629sThis affects top, ps (some variants), w, glibc (sysconf) and much more.
Quoted 7 lines
One of the items reported in
/proc/statis a total count of interrupts that have been received. This calculation requires summation of the interrupts received on each cpu (kstat_irqs_cpu()).The data is kept in per-cpu arrays linked to each
irq_desc. On a 4096p/5530IRQ system summing this data requires accessing ~90MB.
Wow.
Quoted 4 lines
Deleting the summation of the
kstat_irqs_cpudata eliminates the high access time but is an API breakage that I assume is unacceptible.
Quoted 9 lines
Another possibility would be using delayed work (similar to
vmstat_update) that periodically sums the data into a single array. The disadvantage in this approach is that there would be a delay between receipt of an interrupt & it's count appearing/proc/stat. Is this an issue for anyone? Another disadvantage is that it adds to the overall "noise" introduced by kernel threads.Is there a better approach to take?
Hmm, this ?
==
From: KAMEZAWA Hiroyuki <[email protected]>
/proc/stat shows the total number of all interrupts to each cpu. But when the number of IRQs are very large, it take very long time and 'cat /proc/stat' takes more than 10 secs. This is because sum of all irq events are counted when /proc/stat is read. This patch adds "sum of all irq" counter percpu and reduce read costs.
--- fs/proc/stat.c | 4 +--- include/linux/kernel_stat.h | 14 ++++++++++++-- 2 files changed, 13 insertions(+), 5 deletions(-)
fs/proc/stat.c +1 -3
@@ -52,9 +52,7 @@ static int show_stat(struct seq_file *p, guest =cputime64_add (guest,kstat_cpu (i).cpustat.guest); guest_nice =cputime64_add (guest_nice,kstat_cpu (i).cpustat.guest_nice); -for_each_irq_nr (j) { - sum +=kstat_irqs_cpu (j, i); - } + sum =kstat_cpu_irqs_sum (i); sum +=arch_irq_stat_cpu (i);for (j =0 ; j <NR_SOFTIRQS ; j++) {
include/linux/kernel_stat.h +12 -2
@@ -33,6 +33,7 @@ struct kernel_stat {#ifndef CONFIG_GENERIC_HARDIRQS unsigned int irqs[NR_IRQS ];#endif +unsigned long irqs_sum;unsigned int softirqs[NR_SOFTIRQS ]; };
@@ -54,6 +55,7 @@ static inline void kstat_incr_irqs_this_ struct irq_desc *desc) { kstat_this_cpu.irqs[irq]++; + kstat_this_cpu.irqs_sum++; }static inline unsigned int kstat_irqs_cpu (unsigned int irq,int cpu)
@@ -65,8 +67,9 @@ static inline unsigned int kstat_irqs_cpextern unsigned int kstat_irqs_cpu (unsigned int irq,int cpu);#define kstat_irqs_this_cpu (DESC ) \ ((DESC)->kstat_irqs[smp_processor_id()]) -#define kstat_incr_irqs_this_cpu (irqno,DESC ) \ - ((DESC)->kstat_irqs[smp_processor_id()]++) +#define kstat_incr_irqs_this_cpu (irqno,DESC ) do {\ + ((DESC)->kstat_irqs[smp_processor_id()]++);\ + kstat_this_cpu.irqs_sum++;} while (0)#endif
@@ -94,6 +97,13 @@ static inline unsigned int kstat_irqs(unreturn sum; } +/* +* Number of interrupts per cpu, since bootup +*/ +static inline unsigned long kstat_cpu_irqs_sum (unsigned int cpu) +{ +return kstat_cpu (cpu).irqs_sum; +}/* * Lock/unlock the current runqueue - to extract task statistics:
Quoted 2 lines · KAMEZAWA Hiroyuki
On Wed, 29 Sep 2010 07:22:06 -0500
Jack Steiner <[email protected]> wrote:
I was able to run on the 4096p system over the weekend. The patch is a definite improvement & partially fixes the problem:
A "cat /proc/stat >/dev/null" improved:
OLD: real 12.627s
NEW: real 2.459A large part of the remaining overhead is in the second summation
of irq information:
static int show_stat(struct seq_file *p, void *v)
...
/* sum again ? it could be updated? */
for_each_irq_nr(j) {
per_irq_sum = 0;
for_each_possible_cpu(i)
per_irq_sum += kstat_irqs_cpu(j, i);
seq_printf(p, " %u", per_irq_sum);
}Can this be fixed using the same approach as in the current patch?
--- jack
Quoted 116 lines
Quoted 15 lines
I'm looking for suggestions on how to fix a scaling problem with access to
/proc/stat.On a large
x86_64system (4096p, 256 nodes, 5530 IRQs), access to/proc/stattakes too long - more than 12 sec:# time cat /proc/stat >/dev/null real 12.630s user 0.000s sys 12.629sThis affects top, ps (some variants), w, glibc (sysconf) and much more.
Quoted 7 lines
One of the items reported in
/proc/statis a total count of interrupts that have been received. This calculation requires summation of the interrupts received on each cpu (kstat_irqs_cpu()).The data is kept in per-cpu arrays linked to each
irq_desc. On a 4096p/5530IRQ system summing this data requires accessing ~90MB.Wow.
Quoted 4 lines
Deleting the summation of the
kstat_irqs_cpudata eliminates the high access time but is an API breakage that I assume is unacceptible.Quoted 9 lines
Another possibility would be using delayed work (similar to
vmstat_update) that periodically sums the data into a single array. The disadvantage in this approach is that there would be a delay between receipt of an interrupt & it's count appearing/proc/stat. Is this an issue for anyone? Another disadvantage is that it adds to the overall "noise" introduced by kernel threads.Is there a better approach to take?
Hmm, this ?
==
From: KAMEZAWA Hiroyuki <[email protected]>
/proc/statshows the total number of all interrupts to each cpu. But when the number of IRQs are very large, it take very long time and 'cat/proc/stat' takes more than 10 secs. This is because sum of all irq events are counted when/proc/statis read. This patch adds "sum of all irq" counter percpu and reduce read costs.Signed-off-by KAMEZAWA Hiroyuki <[email protected]>--- fs/proc/stat.c | 4 +--- include/linux/kernel_stat.h | 14 ++++++++++++-- 2 files changed, 13 insertions(+), 5 deletions(-)Index:
mmotm-0922/fs/proc/stat.c
===================================================================fs/proc/stat.c +1
-3@@ -52,9 +52,7 @@ static int show_stat(struct seq_file *p, guest =cputime64_add (guest,kstat_cpu (i).cpustat.guest); guest_nice =cputime64_add (guest_nice,kstat_cpu (i).cpustat.guest_nice); -for_each_irq_nr (j) { - sum +=kstat_irqs_cpu (j, i); - } + sum =kstat_cpu_irqs_sum (i); sum +=arch_irq_stat_cpu (i);for (j =0 ; j <NR_SOFTIRQS ; j++) {Index:
mmotm-0922/include/linux/kernel_stat.h
===================================================================include/linux/kernel_stat.h +12
-2@@ -33,6 +33,7 @@ struct kernel_stat {#ifndef CONFIG_GENERIC_HARDIRQS unsigned int irqs[NR_IRQS ];#endif +unsigned long irqs_sum;unsigned int softirqs[NR_SOFTIRQS ]; };@@ -54,6 +55,7 @@ static inline void kstat_incr_irqs_this_ struct irq_desc *desc) { kstat_this_cpu.irqs[irq]++; + kstat_this_cpu.irqs_sum++; }static inline unsigned int kstat_irqs_cpu (unsigned int irq,int cpu)@@ -65,8 +67,9 @@ static inline unsigned int kstat_irqs_cpextern unsigned int kstat_irqs_cpu (unsigned int irq,int cpu);#define kstat_irqs_this_cpu (DESC ) \ ((DESC)->kstat_irqs[smp_processor_id()]) -#define kstat_incr_irqs_this_cpu (irqno,DESC ) \ - ((DESC)->kstat_irqs[smp_processor_id()]++) +#define kstat_incr_irqs_this_cpu (irqno,DESC ) do {\ + ((DESC)->kstat_irqs[smp_processor_id()]++);\ + kstat_this_cpu.irqs_sum++;} while (0)#endif @@ -94,6 +97,13 @@ static inline unsigned int kstat_irqs(unreturn sum; } +/* +* Number of interrupts per cpu, since bootup +*/ +static inline unsigned long kstat_cpu_irqs_sum (unsigned int cpu) +{ +return kstat_cpu (cpu).irqs_sum; +}/* * Lock/unlock the current runqueue - to extract task statistics:
Quoted 5 lines · On Mon, 4 Oct 2010 09:34:15 -0500 Jack Steiner <[email protected]> wrote:
Quoted 2 lines · KAMEZAWA Hiroyuki
On Wed, 29 Sep 2010 07:22:06 -0500
Jack Steiner <[email protected]> wrote:
Quoted 9 lines
I was able to run on the 4096p system over the weekend. The patch is a definite improvement & partially fixes the problem:
A "cat
/proc/stat>/dev/null" improved:OLD: real 12.627s NEW: real 2.459
Thank you.
Quoted 15 lines
A large part of the remaining overhead is in the second summation
of irq information:static int show_stat(struct seq_file *p, void *v) ... /* sum again ? it could be updated? */ for_each_irq_nr(j) { per_irq_sum = 0; for_each_possible_cpu(i) per_irq_sum += kstat_irqs_cpu(j, i); seq_printf(p, " %u", per_irq_sum); }
Quoted 3 lines
Can this be fixed using the same approach as in the current patch?
I guess this requres different approarch as per-cpu counter + threshould. like vmstat[] or lib/percpu_counter.
Maybe people don't like to access shared counter in IRQ.
But, this seems to call radixtree-lookup for the # of possible cpus. I guess impleimenting a call to calculate a sum of irqs in a radix-tree lookup will reduce overhead. If it's not enough, we'll have to make the counter not-precise. I'll write an another patch.
Thanks,
-Kame
Quoted 9 lines · On Tue, 5 Oct 2010 10:36:50 +0900 KAMEZAWA Hiroyuki…
I guess this requres different approarch as per-cpu counter + threshould. like vmstat[] or
lib/percpu_counter.
Maybe people don't like to access shared counter in IRQ.But, this seems to call radixtree-lookup for the # of possible cpus. I guess impleimenting a call to calculate a sum of irqs in a radix-tree lookup will reduce overhead. If it's not enough, we'll have to make the counter not-precise. I'll write an another patch.
How about this ? This is an add-on patch.
==
From: KAMEZAWA Hiroyuki <[email protected]>
In /proc/stat, the number of per-IRQ event is shown by making a sum each irq's events on all cpus. But we can make use of kstat_irqs().
kstat_irqs() make a sum of IRQ events per cpu, if !CONFIG_GENERIC_HARDIRQ, it's not a big cost. (Both of the number of cpus and irqs are small.)
If a system is very big, it does
for_each_irq() for_each_cpu() - look up a radix tree - read desc->irq_stat[cpu]
This seems not efficient. This patch adds kstat_irqs() for CONFIG_GENRIC_HARDIRQ and change the calculation as
for_each_irq() look up radix tree for_each_cpu() - read desc->irq_stat[cpu]
and reduces cost.
--- fs/proc/stat.c | 9 ++------- include/linux/kernel_stat.h | 5 +++++ kernel/irq/handle.c | 16 ++++++++++++++++ 3 files changed, 23 insertions(+), 7 deletions(-)
fs/proc/stat.c +2 -7
@@ -108,13 +108,8 @@ static int show_stat(struct seq_file *p,seq_printf (p,"intr %llu" , (unsigned long long )sum);/* sum again ? it could be updated? */ -for_each_irq_nr (j) { - per_irq_sum =0 ; -for_each_possible_cpu (i) - per_irq_sum +=kstat_irqs_cpu (j, i); - -seq_printf (p," %u" , per_irq_sum); - } +for_each_irq_nr (j) +seq_printf (p," %u" ,kstat_irqs (j));seq_printf (p, "\nctxt %llu\n"
include/linux/kernel_stat.h +5 -0
@@ -62,6 +62,7 @@ static inline unsigned int kstat_irqs_cp {return kstat_cpu (cpu).irqs[irq]; } +#else #include <linux/irq.h> extern unsigned int kstat_irqs_cpu (unsigned int irq,int cpu);
@@ -86,6 +87,7 @@ static inline unsigned int kstat_softirq/* * Number of interrupts per specific IRQ source, since bootup */ +#ifndef CONFIG_GENERIC_HARDIRQS static inline unsigned int kstat_irqs (unsigned int irq) {unsigned int sum =0 ;
@@ -96,6 +98,9 @@ static inline unsigned int kstat_irqs(unreturn sum; } +#else +extern unsigned int unsigned int kstat_irqs(unsigned int irq); +#endif /* * Number of interrupts per cpu, since bootup
kernel/irq/handle.c +16 -0
@@ -553,3 +553,19 @@ unsigned int kstat_irqs_cpu(unsigned int }EXPORT_SYMBOL (kstat_irqs_cpu); +#ifdef CONFIG_GENERIC_HARDIRQS +unsigned int kstat_irqs (unsigned int irq) +{ +struct irq_desc *desc =irq_to_desc (irq); +int cpu; +int sum =0 ; + +if (!desc) +return 0 ; + +for_each_possible_cpu (cpu) + sum += desc->kstat_irqs[cpu]; +return sum; +} +EXPORT_SYMBOL (kstat_irqs); +#endif
Jack Steiner
Acked-byQuoted 7 lines · KAMEZAWA Hiroyuki
Quoted 4 lines · On Tue, 5 Oct 2010 10:36:50 +0900 KAMEZAWA Hiroyuki…
I guess this requres different approarch as per-cpu counter + threshould. like vmstat[] or
lib/percpu_counter.
Maybe people don't like to access shared counter in IRQ.
Quoted 7 lines
Quoted 5 lines
But, this seems to call radixtree-lookup for the # of possible cpus. I guess impleimenting a call to calculate a sum of irqs in a radix-tree lookup will reduce overhead. If it's not enough, we'll have to make the counter not-precise. I'll write an another patch.
How about this ? This is an add-on patch.
Nice!!
The combination of the 2 patches solves the problem. The timings are (4096p, 256 nodes, 4592 irqs):
# time cat /proc/stat > /dev/null
Baseline: 12.627 sec Patch1 : 2.459 sec Patch 1 + Patch 2: .561 sec
Thanks!!
--- jack
Quoted 106 lines
==
From: KAMEZAWA Hiroyuki <[email protected]>In
/proc/stat, the number of per-IRQ event is shown by making a sum each irq's events on all cpus. But we can make use ofkstat_irqs().
kstat_irqs()make a sum of IRQ events per cpu, if !CONFIG_GENERIC_HARDIRQ, it's not a big cost. (Both of the number of cpus and irqs are small.)If a system is very big, it does
for_each_irq() for_each_cpu() - look up a radix tree - read desc->irq_stat[cpu]This seems not efficient. This patch adds
kstat_irqs()forCONFIG_GENRIC_HARDIRQand change the calculation asfor_each_irq() look up radix tree for_each_cpu() - read desc->irq_stat[cpu]and reduces cost.
Signged-off-by KAMEZAWA Hiroyuki <[email protected]>--- fs/proc/stat.c | 9 ++------- include/linux/kernel_stat.h | 5 +++++ kernel/irq/handle.c | 16 ++++++++++++++++ 3 files changed, 23 insertions(+), 7 deletions(-)Index:
mmotm-0928/fs/proc/stat.c
===================================================================fs/proc/stat.c +2
-7@@ -108,13 +108,8 @@ static int show_stat(struct seq_file *p,seq_printf (p,"intr %llu" , (unsigned long long )sum);/* sum again ? it could be updated? */ -for_each_irq_nr (j) { - per_irq_sum =0 ; -for_each_possible_cpu (i) - per_irq_sum +=kstat_irqs_cpu (j, i); - -seq_printf (p," %u" , per_irq_sum); - } +for_each_irq_nr (j) +seq_printf (p," %u" ,kstat_irqs (j));seq_printf (p, "\nctxt %llu\n"Index:
mmotm-0928/include/linux/kernel_stat.h
===================================================================include/linux/kernel_stat.h +5
-0@@ -62,6 +62,7 @@ static inline unsigned int kstat_irqs_cp {return kstat_cpu (cpu).irqs[irq]; } +#else #include <linux/irq.h> extern unsigned int kstat_irqs_cpu (unsigned int irq,int cpu);@@ -86,6 +87,7 @@ static inline unsigned int kstat_softirq/* * Number of interrupts per specific IRQ source, since bootup */ +#ifndef CONFIG_GENERIC_HARDIRQS static inline unsigned int kstat_irqs (unsigned int irq) {unsigned int sum =0 ;@@ -96,6 +98,9 @@ static inline unsigned int kstat_irqs(unreturn sum; } +#else +extern unsigned int unsigned int kstat_irqs(unsigned int irq); +#endif /* * Number of interrupts per cpu, since bootup Index:
mmotm-0928/kernel/irq/handle.c
===================================================================kernel/irq/handle.c +16
-0@@ -553,3 +553,19 @@ unsigned int kstat_irqs_cpu(unsigned int }EXPORT_SYMBOL (kstat_irqs_cpu); +#ifdef CONFIG_GENERIC_HARDIRQS +unsigned int kstat_irqs (unsigned int irq) +{ +struct irq_desc *desc =irq_to_desc (irq); +int cpu; +int sum =0 ; + +if (!desc) +return 0 ; + +for_each_possible_cpu (cpu) + sum += desc->kstat_irqs[cpu]; +return sum; +} +EXPORT_SYMBOL (kstat_irqs); +#endif
Quoted 21 lines · On Fri, 8 Oct 2010 11:35:57 -0500 Jack Steiner <[email protected]> wrote:
Quoted 7 lines · KAMEZAWA Hiroyuki
Quoted 4 lines · On Tue, 5 Oct 2010 10:36:50 +0900 KAMEZAWA Hiroyuki…
I guess this requres different approarch as per-cpu counter + threshould. like vmstat[] or
lib/percpu_counter.
Maybe people don't like to access shared counter in IRQ.Quoted 7 lines
Quoted 5 lines
But, this seems to call radixtree-lookup for the # of possible cpus. I guess impleimenting a call to calculate a sum of irqs in a radix-tree lookup will reduce overhead. If it's not enough, we'll have to make the counter not-precise. I'll write an another patch.
How about this ? This is an add-on patch.
Nice!!
The combination of the 2 patches solves the problem. The timings are (4096p, 256 nodes, 4592 irqs):
Quoted 9 lines
# time cat /proc/stat > /dev/nullBaseline: 12.627 sec Patch1 : 2.459 sec Patch 1 + Patch 2: .561 secAcked-by Jack Steiner <[email protected]>
Thank you for testing. I'll post again if necessary.
-Kame
Quoted 15 lines · On Tue, 12 Oct 2010 09:09:07 +0900 KAMEZAWA Hiroyuki…
Quoted 15 lines
Quoted 2 lines
How about this ? This is an add-on patch.
Nice!!
The combination of the 2 patches solves the problem. The timings are (4096p, 256 nodes, 4592 irqs):
# time cat /proc/stat > /dev/nullBaseline: 12.627 sec Patch1 : 2.459 sec Patch 1 + Patch 2: .561 sec
Quoted 4 lines
Quoted 2 lines
Acked-by Jack Steiner <[email protected]>Thank you for testing. I'll post again if necessary.
Yes please. This has been going on for two weeks so memories need refreshing. Also we have no patch title and no changelog which includes the testing results.
I could stitch all that together of course, but it's best that it all be done afresh, I think.
Quoted 20 lines · On Mon, 11 Oct 2010 17:22:26 -0700 Andrew Morton <[email protected]>…
Quoted 18 lines · On Tue, 12 Oct 2010 09:09:07 +0900 KAMEZAWA Hiroyuki…
Quoted 17 lines
Quoted 2 lines
How about this ? This is an add-on patch.
Nice!!
The combination of the 2 patches solves the problem. The timings are (4096p, 256 nodes, 4592 irqs):
# time cat /proc/stat > /dev/nullBaseline: 12.627 sec Patch1 : 2.459 sec Patch 1 + Patch 2: .561 secAcked-by Jack Steiner <[email protected]>
Quoted 9 lines
Quoted 1 line
Thank you for testing. I'll post again if necessary.
Yes please. This has been going on for two weeks so memories need refreshing. Also we have no patch title and no changelog which includes the testing results.
I could stitch all that together of course, but it's best that it all be done afresh, I think.
ok, will do today.
-Kame
PATCH 1/2 fix slowness of /proc/stat per-cpu IRQ sum calculation on large system by a new counter
In reply to Andrew MortonJack Steiner reported slowness of /proc/stat on a large system. This patch set tries to improve it.
Quoted 8 lines
The combination of the 2 patches solves the problem. The timings are (4096p, 256 nodes, 4592 irqs):
# time cat /proc/stat > /dev/nullBaseline: 12.627 sec Patch1 : 2.459 sec Patch 1 + Patch 2: .561 sec
please review.
==
From: KAMEZAWA Hiroyuki <[email protected]>
Problem: 'cat /proc/stat' is too slow on verrry bis system.
/proc/stat shows the total number of all interrupts to each cpu. But when the number of IRQs are very large, it takes very long time and 'cat /proc/stat' takes more than 10 secs. This is because sum of all irq events are counted when /proc/stat is read. This patch adds "sum of all irq" counter percpu and update it at events.
The cost of reading /proc/stat is important because it's used by major applications as 'top', 'ps', 'w', etc....
A test on a host (4096cpu, 256 nodes, 4592 irqs) shows
%time cat /proc/stat > /dev/null
Before Patch: 12.627 sec
After Patch: 2.459 sec
--- fs/proc/stat.c | 4 +--- include/linux/kernel_stat.h | 14 ++++++++++++-- 2 files changed, 13 insertions(+), 5 deletions(-)
fs/proc/stat.c +1 -3
@@ -52,9 +52,7 @@ static int show_stat(struct seq_file *p, guest =cputime64_add (guest,kstat_cpu (i).cpustat.guest); guest_nice =cputime64_add (guest_nice,kstat_cpu (i).cpustat.guest_nice); -for_each_irq_nr (j) { - sum +=kstat_irqs_cpu (j, i); - } + sum =kstat_cpu_irqs_sum (i); sum +=arch_irq_stat_cpu (i);for (j =0 ; j <NR_SOFTIRQS ; j++) {
include/linux/kernel_stat.h +12 -2
@@ -33,6 +33,7 @@ struct kernel_stat {#ifndef CONFIG_GENERIC_HARDIRQS unsigned int irqs[NR_IRQS ];#endif +unsigned long irqs_sum;unsigned int softirqs[NR_SOFTIRQS ]; };
@@ -54,6 +55,7 @@ static inline void kstat_incr_irqs_this_ struct irq_desc *desc) { kstat_this_cpu.irqs[irq]++; + kstat_this_cpu.irqs_sum++; }static inline unsigned int kstat_irqs_cpu (unsigned int irq,int cpu)
@@ -65,8 +67,9 @@ static inline unsigned int kstat_irqs_cpextern unsigned int kstat_irqs_cpu (unsigned int irq,int cpu);#define kstat_irqs_this_cpu (DESC ) \ ((DESC)->kstat_irqs[smp_processor_id()]) -#define kstat_incr_irqs_this_cpu (irqno,DESC ) \ - ((DESC)->kstat_irqs[smp_processor_id()]++) +#define kstat_incr_irqs_this_cpu (irqno,DESC ) do {\ + ((DESC)->kstat_irqs[smp_processor_id()]++);\ + kstat_this_cpu.irqs_sum++; } while (0)#endif
@@ -94,6 +97,13 @@ static inline unsigned int kstat_irqs(unreturn sum; } +/* +* Number of interrupts per cpu, since bootup +*/ +static inline unsigned int kstat_cpu_irqs_sum (unsigned int cpu) +{ +return kstat_cpu (cpu).irqs_sum; +}/* * Lock/unlock the current runqueue - to extract task statistics:
PATCH 2/2 improve footprint of kstat_irqs() for large system's /proc/stat
From: KAMEZAWA Hiroyuki <[email protected]>
Problem: 'cat /proc/stat' takes too long in verrrry big system.
In /proc/stat, the number of per-IRQ event is shown by making a sum each irq's events on all cpus. We can make use of kstat_irqs(). This patch replaces it, at first.
Here, performance of kstat_irqs() is a problem.
If !CONFIG_GENERIC_HARDIRQ, it's not very slow, but if CONFIG_GENERIC_HARDIRQ, it's slow because the logic is not efficient.
If CONFIG_GENERIC_HARDIRQ, it does
for_each_irq() for_each_cpu() - look up a radix tree - read desc->irq_stat[cpu]
This seems not efficient. This patch adds kstat_irqs() for CONFIG_GENRIC_HARDIRQ and change the calculation as
for_each_irq() look up radix tree for_each_cpu() - read desc->irq_stat[cpu]
This reduces cost of scanning.
A test on (4096cpusp, 256 nodes, 4592 irqs) host (by Jack Steiner)
%time cat /proc/stat > /dev/null
Before Patch: 2.459 sec
After Patch : .561 sec
--- fs/proc/stat.c | 9 ++------- include/linux/kernel_stat.h | 4 ++++ kernel/irq/handle.c | 16 ++++++++++++++++ 3 files changed, 22 insertions(+), 7 deletions(-)
fs/proc/stat.c +2 -7
@@ -108,13 +108,8 @@ static int show_stat(struct seq_file *p,seq_printf (p,"intr %llu" , (unsigned long long )sum);/* sum again ? it could be updated? */ -for_each_irq_nr (j) { - per_irq_sum =0 ; -for_each_possible_cpu (i) - per_irq_sum +=kstat_irqs_cpu (j, i); - -seq_printf (p," %u" , per_irq_sum); - } +for_each_irq_nr (j) +seq_printf (p," %u" ,kstat_irqs (j));seq_printf (p, "\nctxt %llu\n"
include/linux/kernel_stat.h +4 -0
@@ -86,6 +86,7 @@ static inline unsigned int kstat_softirq/* * Number of interrupts per specific IRQ source, since bootup */ +#ifndef CONFIG_GENERIC_HARDIRQS static inline unsigned int kstat_irqs (unsigned int irq) {unsigned int sum =0 ;
@@ -96,6 +97,9 @@ static inline unsigned int kstat_irqs(un return sum; } +#else +extern unsigned int kstat_irqs (unsigned int irq); +#endif /* * Number of interrupts per cpu, since bootup
kernel/irq/handle.c +16 -0
@@ -554,3 +554,19 @@ unsigned int kstat_irqs_cpu(unsigned int }EXPORT_SYMBOL (kstat_irqs_cpu); +#ifdef CONFIG_GENERIC_HARDIRQS +unsigned int kstat_irqs (unsigned int irq) +{ +struct irq_desc *desc =irq_to_desc (irq); +int cpu; +int sum =0 ; + +if (!desc) +return 0 ; + +for_each_possible_cpu (cpu) + sum += desc->kstat_irqs[cpu]; +return sum; +} +EXPORT_SYMBOL (kstat_irqs); +#endif
Re: [PATCH 1/2] fix slowness of /proc/stat per-cpu IRQ sum calculation on large system by a new counter
In reply to KAMEZAWA HiroyukiQuoted 47 lines · KAMEZAWA Hiroyuki
Jack Steiner reported slowness of
/proc/staton a large system. This patch set tries to improve it.Quoted 8 lines
The combination of the 2 patches solves the problem. The timings are (4096p, 256 nodes, 4592 irqs):
# time cat /proc/stat > /dev/nullBaseline: 12.627 sec Patch1 : 2.459 sec Patch 1 + Patch 2: .561 secplease review.
==
From: KAMEZAWA Hiroyuki <[email protected]>Problem: 'cat
/proc/stat' is too slow on verrry bis system.
/proc/statshows the total number of all interrupts to each cpu. But when the number of IRQs are very large, it takes very long time and 'cat/proc/stat' takes more than 10 secs. This is because sum of all irq events are counted when/proc/statis read. This patch adds "sum of all irq" counter percpu and update it at events.The cost of reading
/proc/statis important because it's used by major applications as 'top', 'ps', 'w', etc....A test on a host (4096cpu, 256 nodes, 4592 irqs) shows
%time cat
/proc/stat>/dev/null
Before Patch: 12.627 sec
After Patch: 2.459 secTested-by Jack Steiner <[email protected]>Acked-by Jack Steiner <[email protected]>Signed-off-by KAMEZAWA Hiroyuki <[email protected]>--- fs/proc/stat.c | 4 +--- include/linux/kernel_stat.h | 14 ++++++++++++-- 2 files changed, 13 insertions(+), 5 deletions(-)Index:
linux-2.6.36-rc7/fs/proc/stat.c
===================================================================
fs/proc/stat.c static int show_stat(struct seq_file *p, · line 55
Quoted 7 lines
fs/proc/stat.c +1
-3@@ -52,9 +52,7 @@ static int show_stat(struct seq_file *p, guest =cputime64_add (guest,kstat_cpu (i).cpustat.guest); guest_nice =cputime64_add (guest_nice,kstat_cpu (i).cpustat.guest_nice); -for_each_irq_nr (j) { - sum +=kstat_irqs_cpu (j, i); - } + sum =kstat_cpu_irqs_sum (i);
should be
+ sum += kstat_cpu_irqs_sum(i);
Quoted 49 lines
fs/proc/stat.c +0
-0sum +=arch_irq_stat_cpu (i);for (j =0 ; j <NR_SOFTIRQS ; j++) {Index:
linux-2.6.36-rc7/include/linux/kernel_stat.h
===================================================================include/linux/kernel_stat.h +12
-2@@ -33,6 +33,7 @@ struct kernel_stat {#ifndef CONFIG_GENERIC_HARDIRQS unsigned int irqs[NR_IRQS ];#endif +unsigned long irqs_sum;unsigned int softirqs[NR_SOFTIRQS ]; };@@ -54,6 +55,7 @@ static inline void kstat_incr_irqs_this_ struct irq_desc *desc) { kstat_this_cpu.irqs[irq]++; + kstat_this_cpu.irqs_sum++; }static inline unsigned int kstat_irqs_cpu (unsigned int irq,int cpu)@@ -65,8 +67,9 @@ static inline unsigned int kstat_irqs_cpextern unsigned int kstat_irqs_cpu (unsigned int irq,int cpu);#define kstat_irqs_this_cpu (DESC ) \ ((DESC)->kstat_irqs[smp_processor_id()]) -#define kstat_incr_irqs_this_cpu (irqno,DESC ) \ - ((DESC)->kstat_irqs[smp_processor_id()]++) +#define kstat_incr_irqs_this_cpu (irqno,DESC ) do {\ + ((DESC)->kstat_irqs[smp_processor_id()]++);\ + kstat_this_cpu.irqs_sum++; } while (0)#endif @@ -94,6 +97,13 @@ static inline unsigned int kstat_irqs(unreturn sum; } +/* +* Number of interrupts per cpu, since bootup +*/ +static inline unsigned int kstat_cpu_irqs_sum (unsigned int cpu) +{ +return kstat_cpu (cpu).irqs_sum; +}/* * Lock/unlock the current runqueue - to extract task statistics:
Re: [PATCH 1/2] fix slowness of /proc/stat per-cpu IRQ sum calculation on large system by a new counter
On Mon, 11 Oct 2010 20:05:59 -0700
Yinghai Lu <[email protected]> wrote:
kstat_cpu(i).cpustat.guest_nice);
Review
Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +1
-3- for_each_irq_nr(j) { - sum += kstat_irqs_cpu(j, i); - } + sum = kstat_cpu_irqs_sum(i);should be
(quoted diff) +1
-0+ sum += kstat_cpu_irqs_sum(i);
Ouch...thanks.
==
From: KAMEZAWA Hiroyuki <[email protected]>
Problem: 'cat /proc/stat' is too slow on verrry bis system.
/proc/stat shows the total number of all interrupts to each cpu. But when the number of IRQs are very large, it takes very long time and 'cat /proc/stat' takes more than 10 secs. This is because sum of all irq events are counted when /proc/stat is read. This patch adds "sum of all irq" counter percpu and update it at events.
The cost of reading /proc/stat is important because it's used by major applications as 'top', 'ps', 'w', etc....
A test on a host (4096cpu, 256 nodes, 4592 irqs) shows
%time cat /proc/stat > /dev/null
Before Patch: 12.627 sec
After Patch: 2.459 sec
Changelog v1->v2:
- fixed wrong update of a value "sum".
--- fs/proc/stat.c | 4 +--- include/linux/kernel_stat.h | 14 ++++++++++++-- 2 files changed, 13 insertions(+), 5 deletions(-)
fs/proc/stat.c +1 -3
@@ -52,9 +52,7 @@ static int show_stat(struct seq_file *p, guest =cputime64_add (guest,kstat_cpu (i).cpustat.guest); guest_nice =cputime64_add (guest_nice,kstat_cpu (i).cpustat.guest_nice); -for_each_irq_nr (j) { - sum +=kstat_irqs_cpu (j, i); - } + sum +=kstat_cpu_irqs_sum (i); sum +=arch_irq_stat_cpu (i);for (j =0 ; j <NR_SOFTIRQS ; j++) {
include/linux/kernel_stat.h +12 -2
@@ -33,6 +33,7 @@ struct kernel_stat {#ifndef CONFIG_GENERIC_HARDIRQS unsigned int irqs[NR_IRQS ];#endif +unsigned long irqs_sum;unsigned int softirqs[NR_SOFTIRQS ]; };
@@ -54,6 +55,7 @@ static inline void kstat_incr_irqs_this_ struct irq_desc *desc) { kstat_this_cpu.irqs[irq]++; + kstat_this_cpu.irqs_sum++; }static inline unsigned int kstat_irqs_cpu (unsigned int irq,int cpu)
@@ -65,8 +67,9 @@ static inline unsigned int kstat_irqs_cpextern unsigned int kstat_irqs_cpu (unsigned int irq,int cpu);#define kstat_irqs_this_cpu (DESC ) \ ((DESC)->kstat_irqs[smp_processor_id()]) -#define kstat_incr_irqs_this_cpu (irqno,DESC ) \ - ((DESC)->kstat_irqs[smp_processor_id()]++) +#define kstat_incr_irqs_this_cpu (irqno,DESC ) do {\ + ((DESC)->kstat_irqs[smp_processor_id()]++);\ + kstat_this_cpu.irqs_sum++; } while (0)#endif
@@ -94,6 +97,13 @@ static inline unsigned int kstat_irqs(unreturn sum; } +/* +* Number of interrupts per cpu, since bootup +*/ +static inline unsigned int kstat_cpu_irqs_sum (unsigned int cpu) +{ +return kstat_cpu (cpu).irqs_sum; +}/* * Lock/unlock the current runqueue - to extract task statistics: