Measuring tools - top and interrupts
Running 2.6.17, it seems that top is reporting 100% idle with a network load of about 75K pps (bridged) , which seems unlikely. Is it possible that system load accounting is turned off by some tunning knob?
Is there something that shows the current interrupts/second in LINUX (such as systat in 'BSD)?
DT
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Quoted 9 lines · On Thu, 22 Jun 2006 08:26:21 -0700 (PDT) Danial Thom wrote:
Running 2.6.17, it seems that top is reporting 100% idle with a network load of about 75K pps (bridged) , which seems unlikely. Is it possible that system load accounting is turned off by some tunning knob?
Is there something that shows the current interrupts/second in LINUX (such as systat in 'BSD)?
You can use/modify http://www.xenotime.net/linux/scripts/sysalive.pl
for interrupts/second (it already does that).
---
~Randy
Quoted 6 lines · Randy.Dunlap
Quoted 3 lines
Is there something that shows the current
interrupts/second in LINUX (such as systat in
'BSD)?You can use/modify http://www.xenotime.net/linux/scripts/sysalive.pl
for interrupts/second (it already does that).
vmstat shoes a "in" column, and mpstat shows multiple interrupt related numbers.
Gruss
Bernd
Quoted 5 lines · Danial Thom
Running 2.6.17, it seems that top is reporting 100% idle with a network load of about 75K pps (bridged) , which seems unlikely. Is it possible that system load accounting is turned off by some tunning knob?
75K packets/s isn't too hard for modern NICs, especially when using NAPI.
Quoted 3 lines
Is there something that shows the current interrupts/second in LINUX (such as systat in 'BSD)?
"vmstat 1� has the number of interrupts in the "in" column.
Erik
Signature · 2 lines
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 -- | Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
Quoted 7 lines · --- Erik Mouw
Quoted 1 line · Danial Thom
Running 2.6.17, it seems that top is
reporting
Quoted 1 line
100% idle with a network load of about 75K
pps
Quoted 1 line
(bridged) , which seems unlikely. Is it
Quoted 8 lines
possible
Quoted 1 line
that system load accounting is turned off by
some
Quoted 1 line
tunning knob?
75K packets/s isn't too hard for modern NICs,
especially when using
NAPI.
Well thats just a ridiculous answer, so why bother?
You polling guys just crack me up. There isn't much less work to be done with polling. The only reason you THINK its less work is because the measuring tools don't work properly. You still have to process the same number of packets when you poll, and you have polls instead of interrupts. Since you can control the # of interrupts with most cards, there is zero advantage to polling, and more negatives.
And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.
DT
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Quoted 8 lines · Danial Thom
Quoted 3 lines · --- Erik Mouw
75K packets/s isn't too hard for modern NICs,
especially when using
NAPI.Well thats just a ridiculous answer, so why bother?
Quoted 9 lines
You polling guys just crack me up. There isn't much less work to be done with polling. The only reason you THINK its less work is because the measuring tools don't work properly. You still have to process the same number of packets when you poll, and you have polls instead of interrupts. Since you can control the # of interrupts with most cards, there is zero advantage to polling, and more negatives.
There certainly is less work to be done with polling. Less IRQs means less expensive context switches, which means a lower system load. See Documentation/NAPI_HOWTO.txt for information and a link to the Linux NAPI paper.
Quoted 5 lines
And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.
Network traffic is usually IO bound, not CPU bound. The load figures top shows tell something about the amount of work the CPU has to do, not about how busy your PCI bus (or whatever bus the NIC lives on) is.
IIRC the networking layer in 2.6 differs quite a lot from 2.4, so the load average figures can be quite misleading.
Erik
Signature · 2 lines
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 -- | Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
Quoted 51 lines · --- Erik Mouw
Quoted 1 line · Danial Thom
--- Erik Mouw <[email protected]>
Quoted 1 line · wrote:
Quoted 1 line
75K packets/s isn't too hard for modern
NICs,
Quoted 7 lines
Quoted 2 lines
especially when using
NAPI.Well thats just a ridiculous answer, so why bother?
You polling guys just crack me up. There
isn't
Quoted 1 line
much less work to be done with polling. The
only
Quoted 2 lines
reason you THINK its less work is because the measuring tools don't work properly. You
still
Quoted 1 line
have to process the same number of packets
when
Quoted 4 lines
you poll, and you have polls instead of interrupts. Since you can control the # of interrupts with most cards, there is zero advantage to polling, and more negatives.
There certainly is less work to be done with polling. Less IRQs means
less expensive context switches, which means a lower system load. SeeDocumentation/NAPI_HOWTO.txtfor information and a link to the Linux
NAPI paper.Quoted 1 line
And 75K pps may not be "much", but its still
at
Quoted 1 line
least 10% of what the system can handle, so
it
Quoted 1 line
should measure around a 10% load. 2.4
measures
Quoted 1 line
about 12% load. So the only conclusion is
that
Quoted 1 line
load accounting is broken in 2.6.
Network traffic is usually IO bound, not CPU bound. The load figures
top shows tell something about the amount of work the CPU has to do,
not about how busy your PCI bus (or whatever bus the NIC lives on) is.
Quoted 3 lines
IIRC the networking layer in 2.6 differs quite a lot from 2.4, so the
load average figures can be quite misleading.
I'm sorry, but you're being an idiot if you think that 16K interrupts per second and forwarding 75K pps generate no cpu load. Its just that simple. It also means that you've never profiled a kernel because you don't understand where the loads are generated. You've probably been on too many lists with too many people who have no idea what they're talking about.
The NAPI paper is complete rubbish because its based on a principle which no longer exists: that is the assumption that you'll get an interrupt for every packet. Not only is that hardly ever the case, modern controllers (such as Intel's GigE controllers) can precisely mitigate the interrupts in hardware. You *could* tune this on the fly, but there is really no reason to, because at low interrupt loads cpu overhead is not a concern and lower latencies are preferred. You'll never get an interrupt for consecutive packets, and since packets usually arrive in bursts, the mitigation model works even better than if they arrived randomly.
With simple tuning the controller is more efficient with hardware interrupts. None of the arguments in the NAPI paper hold true when using a modern GigE controller.
Of course you folks could never really know that, because the tools to measure things don't work, and you all seem to be using systems that are so kludged up that you don't even know what you're measuring.
DT
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Quoted 8 lines · Danial Thom
I'm sorry, but you're being an idiot if you think that 16K interrupts per second and forwarding 75K pps generate no cpu load. Its just that simple. It also means that you've never profiled a kernel because you don't understand where the loads are generated. You've probably been on too many lists with too many people who have no idea what they're talking about.
(what horrid manners)
Hm. You may be right about the load average calculation being broken.
Below is a 100 second profile sample of my 3GHz P4 handling 15K interrupts per second while receiving a flood ping. My interpretation is that tools should be showing ~10% cpu load rather than zero. Am I'm misinterpreting it?
97574 total 0.0258
89549 default_idle 1017.6023
1734 ioread16 36.8936
1138 ioread8 24.7391
974 rhine_start_tx 1.3994
534 __do_softirq 3.8417
331 handle_IRQ_event 3.2772
223 rhine_interrupt 0.0739
222 memset 7.9286
194 nf_iterate 1.5520
140 local_bh_enable 1.0769
99 __kmalloc 1.0532
92 net_rx_action 0.2000
85 kfree 0.9884
82 skb_release_data 0.6406
77 csum_partial_copy_generic 0.3105
73 ip_push_pending_frames 0.0681
71 __alloc_skb 0.2898
69 kmem_cache_free 1.3529
66 kmem_cache_alloc 1.3750
62 csum_partial 0.2153
61 rt_hash_code 0.4959
61 ip_append_data 0.0253
60 netif_receive_skb 0.0516
58 ip_rcv 0.0471
58 ip_local_deliver 0.0854
58 eth_type_trans 0.2489
55 ip_output 0.0957
52 icmp_reply 0.1187Quoted 79 lines · --- Mike Galbraith
Quoted 1 line · Danial Thom
I'm sorry, but you're being an idiot if you
think
Quoted 1 line
that 16K interrupts per second and forwarding
75K
Quoted 1 line
pps generate no cpu load. Its just that
simple.
Quoted 1 line
It also means that you've never profiled a
kernel
Quoted 1 line
because you don't understand where the loads
are
Quoted 1 line
generated. You've probably been on too many
lists
Quoted 2 lines
with too many people who have no idea what
they're talking about.(what horrid manners)
Hm. You may be right about the load average
calculation being broken.Below is a 100 second profile sample of my 3GHz
P4 handling 15K
interrupts per second while receiving a flood
ping. My interpretation
is that tools should be showing ~10% cpu load
rather than zero. Am I'm
misinterpreting it?97574 total
0.0258
89549default_idle
1017.6023
1734 ioread16
36.8936
1138 ioread8
24.7391
974rhine_start_tx
1.3994
534__do_softirq
3.8417
331handle_IRQ_event
3.2772
223rhine_interrupt
0.0739
222 memset
7.9286
194nf_iterate
1.5520
140local_bh_enable
1.076999 __kmalloc 1.0532 92 net_rx_action 0.2000 85 kfree 0.9884 82 skb_release_data 0.6406 77 csum_partial_copy_generic 0.3105 73 ip_push_pending_frames 0.0681 71 __alloc_skb 0.2898 69 kmem_cache_free 1.3529 66 kmem_cache_alloc 1.3750 62 csum_partial 0.2153 61 rt_hash_code 0.4959 61 ip_append_data 0.0253 60 netif_receive_skb 0.0516 58 ip_rcv
Quoted 10 lines
0.0471
58ip_local_deliver
0.0854
58eth_type_trans
0.2489
55ip_output
0.0957
52icmp_reply
0.1187
Thats a pretty crappy controller you have in with that shiny P4...
I'm not sure that they want the tools to work. They'll just call you a troll and go on developing unnecessary things like NAPI because they're still using controllers designed by DEC (remember them?) back in the stone ages.
Yet I regularly encounter people using cheap NICs with expensive cpus on network-intensive applications. But you'd think one or two people would have a clue.
DT
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Quoted 3 lines · Danial Thom
Thats a pretty crappy controller you have in with that shiny P4...
Shrug. It's a grocery store desktop box, and I'm quite happy with it, but that has roughly zip to do with the price of tea in China.
Quoted 10 lines
I'm not sure that they want the tools to work. They'll just call you a troll and go on developing unnecessary things like NAPI because they're still using controllers designed by DEC (remember them?) back in the stone ages.
Yet I regularly encounter people using cheap NICs with expensive cpus on network-intensive applications. But you'd think one or two people would have a clue.
As does that.
-Mike
Quoted 4 lines · Danial Thom
Quoted 4 lines
Quoted 2 lines · Danial Thom
Well thats just a ridiculous answer, so why
bother?
[...]
Quoted 1 line
I'm sorry, but you're being an idiot if you think
So much for good manners. I accept such replies only from a few people on this list, and you are not one of them.
*plonk*
Erik
Signature · 2 lines
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 -- | Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
Quoted 5 lines · Danial Thom
Of course you folks could never really know that, because the tools to measure things don't work, and you all seem to be using systems that are so kludged up that you don't even know what you're measuring.
Next time attach a patch fixing, or at least bother to investigate the problem which you're so rudely flaming about. Maybe Linux is broken. Nobody said it wasn't. But you're closer to (and clearly care more about) the problem than 99% of the people reading your email.
So why don't you help fix it, instead of laying into Erik and the netdev people (who incidentally aren't CC'ed)?
Signature · 5 lines
Cheers, Alistair. Third year Computer Science undergraduate. 1F2 55 South Clerk Street, Edinburgh, UK.
--- Alistair John Strachan
Quoted 20 lines · <[email protected]> wrote:
Quoted 1 line · Danial Thom
Of course you folks could never really know
that,
Quoted 1 line
because the tools to measure things don't
work,
Quoted 1 line
and you all seem to be using systems that are
so
Quoted 1 line
kludged up that you don't even know what
you're
Quoted 1 line
measuring.
Next time attach a patch fixing, or at least
bother to investigate the problem
which you're so rudely flaming about. Maybe
Linux is broken. Nobody said it
wasn't. But you're closer to (and clearly care
more about) the problem than
99% of the people reading your email.
Quoted 3 lines
So why don't you help fix it, instead of laying
into Erik and the netdev
people (who incidentally aren't CC'ed)?
When you report a problem and some clown pipes in with "its not a problem, linux is just so fast it doesn't use any cpu!" you really don't have the foundation for coming up with a solution or helping to find one.
DT
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Hi Erik,
We have fixed this problem in the kernel 2.4.19 which has been installed in few selected computers of the "Middleware grid" Sydney University.
Also load average has been calculated for each login user.
Thanks
Sena Seneviratne
Computer Engineering Lab
Sydney University
Quoted 48 lines · you
Quoted 8 lines · Danial Thom
Quoted 3 lines · --- Erik Mouw
75K packets/s isn't too hard for modern NICs,
especially when using
NAPI.Well thats just a ridiculous answer, so why bother?
Quoted 9 lines
You polling guys just crack me up. There isn't much less work to be done with polling. The only reason you THINK its less work is because the measuring tools don't work properly. You still have to process the same number of packets when you poll, and you have polls instead of interrupts. Since you can control the # of interrupts with most cards, there is zero advantage to polling, and more negatives.
There certainly is less work to be done with polling. Less IRQs means less expensive context switches, which means a lower system load. See
Documentation/NAPI_HOWTO.txtfor information and a link to the Linux NAPI paper.Quoted 5 lines
And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.
Network traffic is usually IO bound, not CPU bound. The load figures top shows tell something about the amount of work the CPU has to do, not about how busy your PCI bus (or whatever bus the NIC lives on) is.
IIRC the networking layer in 2.6 differs quite a lot from 2.4, so the load average figures can be quite misleading.
Erik
--
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 -- | Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 11 lines
Network traffic is usually IO bound, not CPU
bound. The load figures
top shows tell something about the amount of
work the CPU has to do,
not about how busy your PCI bus (or whatever
bus the NIC lives on) is.IIRC the networking layer in 2.6 differs quite
a lot from 2.4, so the
load average figures can be quite misleading.
For the record, *most* of the work are I/O calls (ie reading and writing registers), which are not in the "background". I/O calls become more and more expensive as the bus becomes saturated as it takes longer to get the bus to do the operation.
DT
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Danial Thom <[email protected]> :
[bull removed]
You have been told twice in this thread to use vmstat.
Read again and shut up until you can provide a sensible bug report, thank you.
Signature · 1 line
Ueimor
Quoted 9 lines · --- Francois Romieu
Danial Thom <[email protected]> :
[bull removed]You have been told twice in this thread to use
vmstat.Read again and shut up until you can provide a
sensible bug report,
thank you.
vmstat gives the same #s as top, and it doesn't break down the interrupts, so its not the answer to the question "is there something similar to systat". Plus its clear that the guy who gave the answer doesn't know what he's talking about, since he's actually trying to explain away the problem as being normal.
Here's a sensible bug report. Bridging 75K pps. vmstat output:
0 0 0 425424 9984 40496 0 0 0
0 16238 12 0 0 100 0
0 0 0 425424 9984 40496 0 0 0
0 16227 16 0 0 100 0
0 0 0 425424 9984 40496 0 0 0
0 16244 9 0 0 100 0Over 16,000 interrupts per second and 100% idle.
netstat:
eth2 1500 0 5 0 0 468886
0 0 0 BMRU
eth3 1500 473201 0 5680 0 5
0 0 0 BMRU
the system is perpetually 100% idle but its dropping packets due to excessive backlog.
DT
Quoted 13 lines
Signature · 11 lines
-- Ueimor - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Danial Thom <[email protected]> :
[...]
Quoted 4 lines
systat". Plus its clear that the guy who gave the answer doesn't know what he's talking about, since he's actually trying to explain away the problem as being normal.
"75 kpps means 10% of the max load" was quite fun too.
[...]
Quoted 2 lines
the system is perpetually 100% idle but its dropping packets due to excessive backlog.
No difference when you renice ksoftirqd to a strongly negative value ?
Signature · 1 line
Ueimor
Quoted 12 lines · --- Francois Romieu
Danial Thom <[email protected]> :
[...]Quoted 1 line
systat". Plus its clear that the guy who gave
the
Quoted 2 lines
answer doesn't know what he's talking about,
since he's actually trying to explain awaythe
Quoted 1 line
problem as being normal.
"75 kpps means 10% of the max load" was quite
fun too.
What's *fun* about it? 2.4.24 shows a 12-13%
load.
Quoted 7 lines
[...]
Quoted 2 lines
the system is perpetually 100% idle but its
dropping packets due to excessive backlog.No difference when you renice ksoftirqd to a
strongly
negative value ?
No, not really. Running the compiler causes
drops. re-nicing eliminates drops with no
userland activity.
DT
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Danial Thom <[email protected]> :
[...]
Quoted 2 lines
What's *fun* about it? 2.4.24 shows a 12-13%
load.
The assumption of linearity.
[...]
Quoted 7 lines
Quoted 3 lines
No difference when you renice ksoftirqd to a
strongly
negative value ?No, not really. Running the compiler causes
drops. re-nicing eliminates drops with no
userland activity.
It may be interesting to know if the same drops happen when you generate a disk-only load, ksoftirqd still at the minimal value.
Signature · 1 line
Ueimor
Quoted 5 lines · Danial Thom
And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.
Are you by chance using procps < 3.1.12? The kernel reports absolute values for cpu usage, the conversion to percentage is done by top/vmstat itself. And those old versions don't know about the new fields that 2.6 kernels have in /proc/stat, thus they simply ignore the si and hi values, producing quite misleading results...
Björn
PS: procps 3.1.12 was released in 2003, so if DEC was stone age and my assumption about your tools holds, then your tools are like... medieval :)
Björn,
Yes you have to change the procps to read the new /proc/stat
For example, after separating the load and disk IO measurements, also separating for per user at the kernel level, I had to introduce new code into procps/ to reflect those changes
Otherwise those tools (top, uptime etc) would not know the new formats etc.
Thanks
Sena
Sydney University
Quoted 22 lines · you
Quoted 5 lines · Danial Thom
And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.
Are you by chance using procps < 3.1.12? The kernel reports absolute values for cpu usage, the conversion to percentage is done by top/vmstat itself. And those old versions don't know about the new fields that 2.6 kernels have in
/proc/stat, thus they simply ignore the si and hi values, producing quite misleading results...Björn
PS: procps 3.1.12 was released in 2003, so if DEC was stone age and my assumption about your tools holds, then your tools are like... medieval :) -
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 5 lines · Danial Thom
And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.
For UP, yes. SMP kernel accounts irq processing time properly.
-Mike
Quoted 9 lines · Mike Galbraith
Quoted 5 lines · Danial Thom
And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.
For UP, yes. SMP kernel accounts irq processing time properly.
For my little box, the below cures it.
arch/i386/kernel/apic.c renamed from arch/i386/kernel/apic.c.org +0 -2
@@ -1175,9 +1175,7 @@ EXPORT_SYMBOL(switch_ipi_to_APIC_timer);inline void smp_local_timer_interrupt (struct pt_regs * regs) {profile_tick (CPU_PROFILING , regs); -#ifdef CONFIG_SMP update_process_times (user_mode_vm (regs)); -#endif /* * We take the 'long' return path, and there every subsystem
Quoted 10 lines · Mike Galbraith
Quoted 9 lines · Mike Galbraith
Quoted 5 lines · Danial Thom
And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.
For UP, yes. SMP kernel accounts irq processing time properly.
Do you actually see 100% idle? On both, UP and SMP, I see non-zero hi/si values using "top". With IO-APIC enabled, I see only non-zero si values for my tg3 NICs, and non-zero hi and si values for the nVidia NIC. With IO-APIC disabled, I also see a non-zero hi value for the tg3 on UP, guess that's normal... But I never see 100% idle while flooding the box with pings.
Quoted 5 lines
For my little box, the below cures it.
arch/i386/kernel/apic.c EXPORT_SYMBOL(switch_ipi_to_APIC_timer); · line 1181
Quoted 9 lines
arch/i386/kernel/apic.c renamed from arch/i386/kernel/apic.c.org +0
-2@@ -1175,9 +1175,7 @@ EXPORT_SYMBOL(switch_ipi_to_APIC_timer);inline void smp_local_timer_interrupt (struct pt_regs * regs) {profile_tick (CPU_PROFILING , regs); -#ifdef CONFIG_SMP update_process_times (user_mode_vm (regs)); -#endif /* * We take the 'long' return path, and there every subsystem
The non-SMP call to update_process_times() is in do_timer_interrupt_hook(), so I guess the above is not the Right Thing to do. I don't even see how you could reach smp_local_timer_interrupt() without going through do_timer_interrupt_hook() first.
Björn
Quoted 10 lines · Björn Steinbrink
Quoted 9 lines · Mike Galbraith
Quoted 8 lines · Mike Galbraith
Quoted 5 lines · Danial Thom
And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.
Quoted 3 lines
Quoted 1 line
Quoted 1 line
For UP, yes. SMP kernel accounts irq processing time properly.
Do you actually see 100% idle?
Yes.
Quoted 6 lines
On both, UP and SMP, I see non-zero hi/si
values using "top". With IO-APIC enabled, I see only non-zero si values for my tg3 NICs, and non-zero hi and si values for the nVidia NIC. With IO-APIC disabled, I also see a non-zero hi value for the tg3 on UP, guess that's normal... But I never see 100% idle while flooding the box with pings.
I have APIC and IO-APIC enabled.
Quoted 12 lines
Quoted 12 lines
For my little box, the below cures it.
arch/i386/kernel/apic.c renamed from arch/i386/kernel/apic.c.org +0
-2@@ -1175,9 +1175,7 @@ EXPORT_SYMBOL(switch_ipi_to_APIC_timer);inline void smp_local_timer_interrupt (struct pt_regs * regs) {profile_tick (CPU_PROFILING , regs); -#ifdef CONFIG_SMP update_process_times (user_mode_vm (regs)); -#endif
Quoted 5 lines
arch/i386/kernel/apic.c EXPORT_SYMBOL(switch_ipi_to_APIC_timer); · line 1181
Quoted 2 lines
arch/i386/kernel/apic.c renamed from arch/i386/kernel/apic.c.org +0
-0/* * We take the 'long' return path, and there every subsystem The non-SMP call to
update_process_times()is indo_timer_interrupt_hook(), so I guess the above is not the Right Thing to do.
Ah, there it is. That's what I was looking for. I figured that doing what I did had to be wrong, but tried it for grins anyway... was pretty surprised when it worked (kinda).
-Mike
Quoted 8 lines · Mike Galbraith
Quoted 3 lines · Björn Steinbrink
The non-SMP call to
update_process_times()is indo_timer_interrupt_hook(), so I guess the above is not the Right Thing to do.Ah, there it is. That's what I was looking for. I figured that doing what I did had to be wrong, but tried it for grins anyway... was pretty surprised when it worked (kinda).
Calling update_process_times() in do_timer_interrupt_hook() flat does not work here. Calling it in smp_local_timer_interrupt() works fine.
Oh joy.
-Mike
Quoted 6 lines · Mike Galbraith
Quoted 5 lines · Mike Galbraith
Quoted 3 lines · Björn Steinbrink
The non-SMP call to
update_process_times()is indo_timer_interrupt_hook(), so I guess the above is not the Right Thing to do.
Quoted 8 lines
Quoted 3 lines
Ah, there it is. That's what I was looking for. I figured that doing what I did had to be wrong, but tried it for grins anyway... was pretty surprised when it worked (kinda).
Calling
update_process_times()indo_timer_interrupt_hook()flat does not work here. Calling it insmp_local_timer_interrupt()works fine.Oh joy.
I can reproduce it now, seems to require CONFIG_4KSTACKS to fail. Can you confirm that?
Björn
Quoted 11 lines · Björn Steinbrink
Quoted 10 lines · Mike Galbraith
Quoted 8 lines · Mike Galbraith
Quoted 3 lines · Björn Steinbrink
The non-SMP call to
update_process_times()is indo_timer_interrupt_hook(), so I guess the above is not the Right Thing to do.Ah, there it is. That's what I was looking for. I figured that doing what I did had to be wrong, but tried it for grins anyway... was pretty surprised when it worked (kinda).
Quoted 7 lines
Quoted 4 lines
Calling
update_process_times()indo_timer_interrupt_hook()flat does not work here. Calling it insmp_local_timer_interrupt()works fine.Oh joy.
I can reproduce it now, seems to require
CONFIG_4KSTACKSto fail. Can you confirm that?
What a coincidence. After trying a different compiler, and slogging through a bunch of assembler trying to figure out how the heck this can happen, I was just booting an 8k stack kernel (as a wild-ass guess;).
let's see. Yeah, confirmed.
-Mike
Quoted 17 lines · Mike Galbraith
Quoted 16 lines · Björn Steinbrink
Quoted 14 lines · Mike Galbraith
Quoted 8 lines · Mike Galbraith
Quoted 3 lines · Björn Steinbrink
The non-SMP call to
update_process_times()is indo_timer_interrupt_hook(), so I guess the above is not the Right Thing to do.Ah, there it is. That's what I was looking for. I figured that doing what I did had to be wrong, but tried it for grins anyway... was pretty surprised when it worked (kinda).
Calling
update_process_times()indo_timer_interrupt_hook()flat does not work here. Calling it insmp_local_timer_interrupt()works fine.Oh joy.
Quoted 8 lines
Quoted 2 lines
I can reproduce it now, seems to require
CONFIG_4KSTACKSto fail. Can you confirm that?What a coincidence. After trying a different compiler, and slogging through a bunch of assembler trying to figure out how the heck this can happen, I was just booting an 8k stack kernel (as a wild-ass guess;).
let's see. Yeah, confirmed.
OK, it also depends on IO APIC being enabled and active, ie. noapic on the kernel command line will fix it as well as disabling CONFIG_X86_IO_APIC. That doesn't help me at all to understand why it happens though.
The only difference with IO APIC disabled seems to be that the irq doesn't get ACKed before update_process_times() gets called. And your "fix" makes it being called outside of the xtime_lock spinlock and with a slightly different stack usage AFAICT.
But none of these should make a difference, right?
Björn
Quoted 8 lines · Björn Steinbrink
Quoted 2 lines · Mike Galbraith
let's see. Yeah, confirmed.
OK, it also depends on IO APIC being enabled and active, ie. noapic on the kernel command line will fix it as well as disabling
CONFIG_X86_IO_APIC. That doesn't help me at all to understand why it happens though.
Ditto.
Quoted 4 lines
The only difference with IO APIC disabled seems to be that the irq doesn't get ACKed before
update_process_times()gets called. And your "fix" makes it being called outside of thextime_lockspinlock and with a slightly different stack usage AFAICT.
(it's still under the xtime lock)
Quoted 1 line
But none of these should make a difference, right?
Not that I can see, but then it's pretty dark down here. Anybody got a flashlight I can borrow? ;-)
-Mike
Quoted 10 lines · Mike Galbraith
Quoted 8 lines · Björn Steinbrink
Quoted 2 lines · Mike Galbraith
let's see. Yeah, confirmed.
OK, it also depends on IO APIC being enabled and active, ie. noapic on the kernel command line will fix it as well as disabling
CONFIG_X86_IO_APIC. That doesn't help me at all to understand why it happens though.
Quoted 8 lines
Ditto.
Quoted 4 lines
The only difference with IO APIC disabled seems to be that the irq doesn't get ACKed before
update_process_times()gets called. And your "fix" makes it being called outside of thextime_lockspinlock and with a slightly different stack usage AFAICT.(it's still under the xtime lock)
No, with IO-APIC enabled, it's using the local APIC timer, thus using_apic_timer is 1 and the path right after unlocking in timer_interrupt() is taken towards update_process_times().
Quoted 4 lines
Quoted 1 line
But none of these should make a difference, right?
Not that I can see, but then it's pretty dark down here. Anybody got a flashlight I can borrow? ;-)
Guess I found one, not sure if it works correctly though ;)
Using 4K stacks, we have one stack for hard irqs and one for soft irqs, both having their own threadinfo and thus preemptcount. Thus the call to softirq_count() in account_system_time() will always return 0 when called in hard irq context. Additionally preemptcount is always set to HARDIRQ_OFFSET in hard irq context, sohardirq_count() - hardirq_offset is 0 all the time as well.
But that doesn't fit the fact that I at least get hard irq accounting when booting with noapic. And it also doesn't explain why your fix works, fixing both, soft and hard irq accounting. Am I missing some code path that calls smp_local_timer_interrupt? There's smp_apic_timer_interrupt(), but that seems to be unused on i386.
Björn
PATCH 1/1 i386: Fix softirq accounting with 4K stacks
Quoted 34 lines · Björn Steinbrink
Quoted 6 lines · Mike Galbraith
Quoted 4 lines · Björn Steinbrink
OK, it also depends on IO APIC being enabled and active, ie. noapic on the kernel command line will fix it as well as disabling
CONFIG_X86_IO_APIC. That doesn't help me at all to understand why it happens though.Quoted 8 lines
Ditto.
Quoted 4 lines
The only difference with IO APIC disabled seems to be that the irq doesn't get ACKed before
update_process_times()gets called. And your "fix" makes it being called outside of thextime_lockspinlock and with a slightly different stack usage AFAICT.(it's still under the xtime lock)
No, with IO-APIC enabled, it's using the local APIC timer, thus
using_apic_timeris 1 and the path right after unlocking intimer_interrupt()is taken towardsupdate_process_times().Quoted 4 lines
Quoted 1 line
But none of these should make a difference, right?
Not that I can see, but then it's pretty dark down here. Anybody got a flashlight I can borrow? ;-)
Guess I found one, not sure if it works correctly though ;)
Using 4K stacks, we have one stack for hard irqs and one for soft irqs, both having their own threadinfo and thus preemptcount. Thus the call to
softirq_count()inaccount_system_time()will always return 0 when called in hard irq context. Additionally preemptcount is always set toHARDIRQ_OFFSETin hard irq context, sohardirq_count()-hardirq_offsetis 0 all the time as well.
Quoted 5 lines
But that doesn't fit the fact that I at least get hard irq accounting when booting with noapic. And it also doesn't explain why your fix works, fixing both, soft and hard irq accounting. Am I missing some code path that calls
smp_local_timer_interrupt? There'ssmp_apic_timer_interrupt(), but that seems to be unused on i386.
OK, the hardirq_count() thing is not HARDIRQ_OFFSET all the time. When we interrupt a hard irq, we are already using the hard irq stack and thus irq_enter() in do_IRQ() works as it should. The zero value for hi is there with IO-APIC and 8K stacks as well, so that's either right or an other bug.
Still no idea why your "fix" works, but the following patch also fixes the problem and is at least a little more like the RightThing.
---
Copy the softirq bits in preempt_count from the current context into the hardirq context when using 4K stacks to make the softirq_count macro work correctly and thereby fix softirq cpu time accounting.
arch/i386/kernel/irq.c +8 -0
@@ -95,6 +95,14 @@ fastcall unsigned int do_IRQ(struct pt_r irqctx->tinfo.task = curctx->tinfo.task; irqctx->tinfo.previous_esp = current_stack_pointer; +/* +* Copy the softirq bits in preempt_count so that the +* softirq checks work in the hardirq context. +*/ + irqctx->tinfo.preempt_count = + irqctx->tinfo.preempt_count & ~SOFTIRQ_MASK | + curctx->tinfo.preempt_count &SOFTIRQ_MASK ; + asmvolatile (" xchgl %%ebx,%%esp \n" " call __do_IRQ \n"
Re: [PATCH] i386: Fix softirq accounting with 4K stacks
Quoted 14 lines · Björn Steinbrink
Copy the softirq bits in
preempt_countfrom the current context into the hardirq context when using 4K stacks to make thesoftirq_countmacro work correctly and thereby fix softirq cpu time accounting.Signed-off-by Björn Steinbrink <[email protected]>diff -Nurp
a/arch/i386/kernel/irq.cb/arch/i386/kernel/irq.carch/i386/kernel/irq.c +1
-0@@ -95,6 +95,14 @@ fastcall unsigned int do_IRQ(struct pt_r irqctx->tinfo.task = curctx->tinfo.task; irqctx->tinfo.previous_esp = current_stack_pointer; +/*
arch/i386/kernel/irq.c fastcall unsigned int do_IRQ(struct pt_r · line 108
Quoted 10 lines
arch/i386/kernel/irq.c +7
-0+* Copy the softirq bits in preempt_count so that the +* softirq checks work in the hardirq context. +*/ + irqctx->tinfo.preempt_count = + irqctx->tinfo.preempt_count & ~SOFTIRQ_MASK | + curctx->tinfo.preempt_count &SOFTIRQ_MASK ; + asmvolatile (" xchgl %%ebx,%%esp \n" " call __do_IRQ \n"
Hi,
at first I got nervous about the asymmetry of this (eg why only do this copying only on entry, and not a copy back on exit)... but then again these bits shouldn't change so your patch is ok as is... it's regrettable that we need to add this for the softirq accounting; part of me wishes we would just count irq-vs-user time and be done with it ;)
Re: [PATCH] i386: Fix softirq accounting with 4K stacks
In reply to Björn SteinbrinkQuoted 2 lines · Björn Steinbrink
Still no idea why your "fix" works, but the following patch also fixes the problem and is at least a little more like the RightThing.
Yeah. I don't know about you, but I fully intend to blatantly ignore that 'why' ;-)
-Mike
Re: [PATCH] i386: Fix softirq accounting with 4K stacks
Quoted 7 lines · Mike Galbraith
Quoted 2 lines · Björn Steinbrink
Still no idea why your "fix" works, but the following patch also fixes the problem and is at least a little more like the RightThing.
Yeah. I don't know about you, but I fully intend to blatantly ignore that 'why' ;-)
the why is relatively easy ;)
since the "is a softirq executing" bit is on the stack, and each context (user, soft and hard irq) has their own stack, it's not automatic that the hardirq stack gets the softirq-executing flag... which your patch fixes.
NMI's and apic irqs generally don't go via the normal irq path and thus don't do a stack switch... so they don't lose the bit (for accounting purposes)
Re: [PATCH] i386: Fix softirq accounting with 4K stacks
Quoted 9 lines · Arjan van de Ven
Quoted 7 lines · Mike Galbraith
Quoted 2 lines · Björn Steinbrink
Still no idea why your "fix" works, but the following patch also fixes the problem and is at least a little more like the RightThing.
Yeah. I don't know about you, but I fully intend to blatantly ignore that 'why' ;-)
Quoted 6 lines
the why is relatively easy ;)
since the "is a softirq executing" bit is on the stack, and each context (user, soft and hard irq) has their own stack, it's not automatic that the hardirq stack gets the softirq-executing flag... which your patch fixes.
That's mine, not Mike's. Mike's patch removed the #ifdef CONFIG_SMP around update_process_times() in smp_local_timer_interrupt().
Quoted 3 lines
NMI's and apic irqs generally don't go via the normal irq path and thus don't do a stack switch... so they don't lose the bit (for accounting purposes)
Hm, doesn't that mean that mean that hardirq accounting is still broken? APIC irq comes in, increases hardirq count, then the timer irq fires, switches the stack and looses the hardirq count that was incremented by the APIC irq.
I just booted with both patches applied, mine and Mike's, and that actually makes a difference in hardirq cpu time accounting. With my patch only, hi is 0 in top while the box gets a ping flood. With both patches, I get about 1% hi. Mike's patch causes update_process_times() to be called twice on UP, but that alone shouldn't change the percentages, right?
OTOH top shows "hi" as zero with 8K stacks as well unless Mike's patch is applied, so the results with Mike's patch are bogus (if so, why?) or hardirq accounting is broken in general.
Btw, which path do apic irqs go? I stumbled across the nmi stuff, but didn't see anything special for the apic irqs.
Björn
Re: [PATCH] i386: Fix softirq accounting with 4K stacks
Quoted 6 lines · Björn Steinbrink
I just booted with both patches applied, mine and Mike's, and that actually makes a difference in hardirq cpu time accounting. With my patch only, hi is 0 in top while the box gets a ping flood. With both patches, I get about 1% hi. Mike's patch causes
update_process_times()to be called twice on UP, but that alone shouldn't change the percentages, right?
Yes, you definitely need to comment out the other call if you test the SMP path on UP+IO-APIC.
Quoted 3 lines
OTOH top shows "hi" as zero with 8K stacks as well unless Mike's patch is applied, so the results with Mike's patch are bogus (if so, why?) or hardirq accounting is broken in general.
Something is certainly still b0rken. I still get three different answers to the question "what is my cpu usage" depending on configuration. With stock UP kernel with no IO-APIC, interrupt load is all hi. With your patch and IO-APIC, it's all si. SMP shows a mix of both.
I like the result of using the SMP path if you have an IO-APIC best, though I haven't verified them against a profile for accuracy. Taking a peek at the profile confirms that it is indeed mixed, so anything showing the load as being either hi or si has to be wrong.
-Mike
Re: [PATCH] i386: Fix softirq accounting with 4K stacks
Quoted 5 lines · Mike Galbraith
Something is certainly still b0rken. I still get three different answers to the question "what is my cpu usage" depending on configuration. With stock UP kernel with no IO-APIC, interrupt load is all hi. With your patch and IO-APIC, it's all si. SMP shows a mix of both.
I didn't say that quite right.
The third case for my box is XT-PIC and your patch. Previously, top showed the 10% interrupt load of a flood ping as all hi. Now it's both hi and si when using XT-PIC, but it's no longer the 10% that agreed with the profile, it's 10% hi, but with an added ~7% si.
-Mike
Re: [PATCH] i386: Fix softirq accounting with 4K stacks
Quoted 8 lines · Mike Galbraith
Quoted 5 lines · Mike Galbraith
Something is certainly still b0rken. I still get three different answers to the question "what is my cpu usage" depending on configuration. With stock UP kernel with no IO-APIC, interrupt load is all hi. With your patch and IO-APIC, it's all si. SMP shows a mix of both.
Quoted 6 lines
I didn't say that quite right.
The third case for my box is XT-PIC and your patch. Previously, top showed the 10% interrupt load of a flood ping as all hi. Now it's both hi and si when using XT-PIC, but it's no longer the 10% that agreed with the profile, it's 10% hi, but with an added ~7% si.
I see the following here, when a box is being ping flooded:
UP/SMP stack-size PIC-type hi si
-----------------------------------
UP 8K IO-APIC 0 16 UP 8K XT-PIC 7 7 UP 4K IO-APIC 0 16 UP 4K XT-PIC 7 7
SMP 8K IO-APIC 5 4 (forcedeth) SMP 8K IO-APIC 0 11 (tg3)
The UP system is a Thinkpad which only has a tg3 driven NIC. The SMP system is x86_64, so there's no 4K stacks to test with, maybe I'll fetch a i386 live CD to do some more valid tests on SMP.
The UP tests seem to show the IO-APIC hardirq are completely deferred to be handled in a softirq, as the sum of "hi" and "si" with XT-PIC is about equal to the "si" value with IO-APIC (the numbers are guessed averages of the observed values, so the 2% difference is not too be taken too serious).
The results for the forcedeth driven NIC do not agree though, and your results differ from what I see as well, right? So I'm kinda lost again.
Björn
Re: [PATCH] i386: Fix softirq accounting with 4K stacks
Quoted 2 lines · Björn Steinbrink
The results for the forcedeth driven NIC do not agree though, and your results differ from what I see as well, right? So I'm kinda lost again.
I'm not. Not any more. After much fruitless rummaging, I did some more profiling to verify the numbers. It really does take that much more cpu when booted with noapic, nearly 50%! I've been chasing swamp gas.
Oh well. I got a warm fuzzy wrt tools cpu usage numbers out of it.
Stick a fork in this bug, it's done ;-)
-Mike
Quoted 70 lines · --- Björn Steinbrink
Quoted 1 line · Mike Galbraith
On Sat, 2006-06-24 at 21:25 +0200, Björn
Quoted 1 line · Steinbrink
Quoted 1 line
On 2006.06.24 18:23:12 +0200, Mike
Quoted 4 lines · Galbraith
Quoted 4 lines
Quoted 2 lines
let's see. Yeah, confirmed.
OK, it also depends on IO APIC being
enabled and active, ie. noapic on
Quoted 1 line
Quoted 1 line
the kernel command line will fix it as well
as disabling
Quoted 1 line
Quoted 1 line
CONFIG_X86_IO_APIC. That doesn't help me atall to understand why it
Quoted 5 lines
Quoted 1 line
happens though.
Ditto.
Quoted 1 line
The only difference with IO APIC disabled
seems to be that the irq
Quoted 1 line
Quoted 1 line
doesn't get ACKed before
update_process_times()gets called.Quoted 1 line
Quoted 1 line
And your "fix" makes it being called
outside of the
xtime_lockspinlockQuoted 1 line
Quoted 1 line
and with a slightly different stack usage
AFAICT.
Quoted 2 lines
(it's still under the xtime lock)
No, with IO-APIC enabled, it's using the local
APIC timer, thususing_apic_timeris 1 and the path right after
unlocking intimer_interrupt()is taken towardsupdate_process_times().Quoted 1 line
Quoted 1 line
But none of these should make a difference,
right?
Quoted 2 lines
Not that I can see, but then it's pretty dark
down here. Anybody got a
Quoted 1 line
flashlight I can borrow? ;-)
Guess I found one, not sure if it works
correctly though ;)Using 4K stacks, we have one stack for hard
irqs and one for soft irqs,
both having their own threadinfo and thus
preemptcount. Thus the call tosoftirq_count()inaccount_system_time()will
always return 0 when
called in hard irq context. Additionally
preemptcount is always set toHARDIRQ_OFFSETin hard irq context, sohardirq_count()-hardirq_offsetis 0 all the
time as well.But that doesn't fit the fact that I at least
get hard irq accounting
when booting with noapic. And it also doesn't
explain why your fix
works, fixing both, soft and hard irq
accounting. Am I missing some
code path that callssmp_local_timer_interrupt?
There'ssmp_apic_timer_interrupt(), but that seems to
be unused on i386.
Quoted 1 line
Björn
I think the one thing we can surmise from this thread is that you can't rely on kernel usage statistics to be accurate, as its likely that there are many, many cases that don't work properly. It was always wrong in 2.4 as well.
DT
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Quoted 5 lines · Danial Thom
I think the one thing we can surmise from this thread is that you can't rely on kernel usage statistics to be accurate, as its likely that there are many, many cases that don't work properly. It was always wrong in 2.4 as well.
Once identified, problems tend to get fixed. This one will probably be history soon. You know the old saying though... "There are lies, there are _damn_ lies, and then there are _statistics_".
Quoted 10 lines · --- Mike Galbraith
Quoted 1 line · Danial Thom
I think the one thing we can surmise from
this
Quoted 4 lines
thread is that you can't rely on kernel usage statistics to be accurate, as its likely that there are many, many cases that don't work properly. It was always wrong in 2.4 as well.
Quoted 6 lines
Once identified, problems tend to get fixed. This one will probably be
history soon. You know the old saying though... "There are lies, there
are _damn_ lies, and then there are _statistics_".
The usefulness of statistics is a function of the breadth of understand of the person interpreting them.
But I don't think that applies here, because we are not questioning the conclusions from the statistics, only the accuracy of the gathering. Its a completely different thing to say, for example, that population growth is due to too many babies when it might be immigration or a reduction in the death rate, than it is to say that the numbers gathered are simply wrong.
The truth is that you guys can't be bothered unless the numbers are so far off that its an embarrasment. Until someone who cares about it (and knows how to do it) puts their nose to the grindstone and makes sure that everything is being done correctly, they'll be largely useless.
DT
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Quoted 33 lines · Danial Thom
Quoted 10 lines · --- Mike Galbraith
Quoted 1 line · Danial Thom
I think the one thing we can surmise from
this
Quoted 4 lines
thread is that you can't rely on kernel usage statistics to be accurate, as its likely that there are many, many cases that don't work properly. It was always wrong in 2.4 as well.
Quoted 6 lines
Once identified, problems tend to get fixed. This one will probably be
history soon. You know the old saying though... "There are lies, there
are _damn_ lies, and then there are _statistics_".The usefulness of statistics is a function of the breadth of understand of the person interpreting them.
But I don't think that applies here, because we are not questioning the conclusions from the statistics, only the accuracy of the gathering. Its a completely different thing to say, for example, that population growth is due to too many babies when it might be immigration or a reduction in the death rate, than it is to say that the numbers gathered are simply wrong.
Quoted 3 lines
The truth is that you guys can't be bothered unless the numbers are so far off that its an embarrasment.
Hey, let's play hide and go fuck yourself. Tag, you're it.
*plonk*
-Mike