Skip to contentSkip to conversation

Measuring tools - top and interrupts

PATCH In review Danial Thom 10 participants 45 messages Jun 22, 2006 – Jun 30, 2006

lore
Share & tools
lore
#0 lore raw

Running 2.6.17, it seems that top is reporting 100% idle with a network load of about 75K pps (bridged) , which seems unlikely. Is it possible that system load accounting is turned off by some tunning knob?

Is there something that shows the current interrupts/second in LINUX (such as systat in 'BSD)?

DT

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#1 lore raw
Quoted 9 lines · On Thu, 22 Jun 2006 08:26:21 -0700 (PDT) Danial Thom wrote:

Running 2.6.17, it seems that top is reporting 100% idle with a network load of about 75K pps (bridged) , which seems unlikely. Is it possible that system load accounting is turned off by some tunning knob?

Is there something that shows the current interrupts/second in LINUX (such as systat in 'BSD)?

You can use/modify http://www.xenotime.net/linux/scripts/sysalive.pl
for interrupts/second (it already does that).

---
~Randy

#2 lore raw
Quoted 6 lines · Randy.Dunlap
Quoted 3 lines

Is there something that shows the current
interrupts/second in LINUX (such as systat in
'BSD)?

You can use/modify http://www.xenotime.net/linux/scripts/sysalive.pl
for interrupts/second (it already does that).

vmstat shoes a "in" column, and mpstat shows multiple interrupt related numbers.

Gruss
Bernd

#3 lore raw
In reply to Danial Thom
Quoted 5 lines · Danial Thom

Running 2.6.17, it seems that top is reporting 100% idle with a network load of about 75K pps (bridged) , which seems unlikely. Is it possible that system load accounting is turned off by some tunning knob?

75K packets/s isn't too hard for modern NICs, especially when using NAPI.

Quoted 3 lines

Is there something that shows the current interrupts/second in LINUX (such as systat in 'BSD)?

"vmstat 1� has the number of interrupts in the "in" column.

Erik

Signature · 2 lines
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
#4 lore raw
Quoted 7 lines · --- Erik Mouw
Quoted 1 line · Danial Thom

Running 2.6.17, it seems that top is

reporting

Quoted 1 line

100% idle with a network load of about 75K

pps

Quoted 1 line

(bridged) , which seems unlikely. Is it

Quoted 8 lines

possible

Quoted 1 line

that system load accounting is turned off by

some

Quoted 1 line

tunning knob?

75K packets/s isn't too hard for modern NICs,
especially when using
NAPI.

Well thats just a ridiculous answer, so why bother?

You polling guys just crack me up. There isn't much less work to be done with polling. The only reason you THINK its less work is because the measuring tools don't work properly. You still have to process the same number of packets when you poll, and you have polls instead of interrupts. Since you can control the # of interrupts with most cards, there is zero advantage to polling, and more negatives.

And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#5 lore raw
Quoted 8 lines · Danial Thom
Quoted 3 lines · --- Erik Mouw

75K packets/s isn't too hard for modern NICs,
especially when using
NAPI.

Well thats just a ridiculous answer, so why bother?

Quoted 9 lines

You polling guys just crack me up. There isn't much less work to be done with polling. The only reason you THINK its less work is because the measuring tools don't work properly. You still have to process the same number of packets when you poll, and you have polls instead of interrupts. Since you can control the # of interrupts with most cards, there is zero advantage to polling, and more negatives.

There certainly is less work to be done with polling. Less IRQs means less expensive context switches, which means a lower system load. See Documentation/NAPI_HOWTO.txt for information and a link to the Linux NAPI paper.

Quoted 5 lines

And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.

Network traffic is usually IO bound, not CPU bound. The load figures top shows tell something about the amount of work the CPU has to do, not about how busy your PCI bus (or whatever bus the NIC lives on) is.

IIRC the networking layer in 2.6 differs quite a lot from 2.4, so the load average figures can be quite misleading.

Erik

Signature · 2 lines
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
#6 lore raw
Quoted 51 lines · --- Erik Mouw
Quoted 1 line · Danial Thom

--- Erik Mouw <[email protected]>

Quoted 1 line · wrote:
Quoted 1 line

75K packets/s isn't too hard for modern

NICs,

Quoted 7 lines
Quoted 2 lines

especially when using
NAPI.

Well thats just a ridiculous answer, so why bother?

You polling guys just crack me up. There

isn't

Quoted 1 line

much less work to be done with polling. The

only

Quoted 2 lines

reason you THINK its less work is because the measuring tools don't work properly. You

still

Quoted 1 line

have to process the same number of packets

when

Quoted 4 lines

you poll, and you have polls instead of interrupts. Since you can control the # of interrupts with most cards, there is zero advantage to polling, and more negatives.

There certainly is less work to be done with polling. Less IRQs means
less expensive context switches, which means a lower system load. See Documentation/NAPI_HOWTO.txt for information and a link to the Linux
NAPI paper.

Quoted 1 line

And 75K pps may not be "much", but its still

at

Quoted 1 line

least 10% of what the system can handle, so

it

Quoted 1 line

should measure around a 10% load. 2.4

measures

Quoted 1 line

about 12% load. So the only conclusion is

that

Quoted 1 line

load accounting is broken in 2.6.

Network traffic is usually IO bound, not CPU bound. The load figures
top shows tell something about the amount of work the CPU has to do,
not about how busy your PCI bus (or whatever bus the NIC lives on) is.

Quoted 3 lines

IIRC the networking layer in 2.6 differs quite a lot from 2.4, so the
load average figures can be quite misleading.

I'm sorry, but you're being an idiot if you think that 16K interrupts per second and forwarding 75K pps generate no cpu load. Its just that simple. It also means that you've never profiled a kernel because you don't understand where the loads are generated. You've probably been on too many lists with too many people who have no idea what they're talking about.

The NAPI paper is complete rubbish because its based on a principle which no longer exists: that is the assumption that you'll get an interrupt for every packet. Not only is that hardly ever the case, modern controllers (such as Intel's GigE controllers) can precisely mitigate the interrupts in hardware. You *could* tune this on the fly, but there is really no reason to, because at low interrupt loads cpu overhead is not a concern and lower latencies are preferred. You'll never get an interrupt for consecutive packets, and since packets usually arrive in bursts, the mitigation model works even better than if they arrived randomly.

With simple tuning the controller is more efficient with hardware interrupts. None of the arguments in the NAPI paper hold true when using a modern GigE controller.

Of course you folks could never really know that, because the tools to measure things don't work, and you all seem to be using systems that are so kludged up that you don't even know what you're measuring.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#7 lore raw
Quoted 8 lines · Danial Thom

I'm sorry, but you're being an idiot if you think that 16K interrupts per second and forwarding 75K pps generate no cpu load. Its just that simple. It also means that you've never profiled a kernel because you don't understand where the loads are generated. You've probably been on too many lists with too many people who have no idea what they're talking about.

(what horrid manners)

Hm. You may be right about the load average calculation being broken.

Below is a 100 second profile sample of my 3GHz P4 handling 15K interrupts per second while receiving a flood ping. My interpretation is that tools should be showing ~10% cpu load rather than zero. Am I'm misinterpreting it?

97574 total 0.0258
89549 default_idle 1017.6023
1734 ioread16 36.8936
1138 ioread8 24.7391

   974 rhine_start_tx                             1.3994
   534 __do_softirq                               3.8417
   331 handle_IRQ_event                           3.2772
   223 rhine_interrupt                            0.0739
   222 memset                                     7.9286
   194 nf_iterate                                 1.5520
   140 local_bh_enable                            1.0769
    99 __kmalloc                                  1.0532
    92 net_rx_action                              0.2000
    85 kfree                                      0.9884
    82 skb_release_data                           0.6406
    77 csum_partial_copy_generic                  0.3105
    73 ip_push_pending_frames                     0.0681
    71 __alloc_skb                                0.2898
    69 kmem_cache_free                            1.3529
    66 kmem_cache_alloc                           1.3750
    62 csum_partial                               0.2153
    61 rt_hash_code                               0.4959
    61 ip_append_data                             0.0253
    60 netif_receive_skb                          0.0516
    58 ip_rcv                                     0.0471
    58 ip_local_deliver                           0.0854
    58 eth_type_trans                             0.2489
    55 ip_output                                  0.0957
    52 icmp_reply                                 0.1187
#8 lore raw
Quoted 79 lines · --- Mike Galbraith
Quoted 1 line · Danial Thom

I'm sorry, but you're being an idiot if you

think

Quoted 1 line

that 16K interrupts per second and forwarding

75K

Quoted 1 line

pps generate no cpu load. Its just that

simple.

Quoted 1 line

It also means that you've never profiled a

kernel

Quoted 1 line

because you don't understand where the loads

are

Quoted 1 line

generated. You've probably been on too many

lists

Quoted 2 lines

with too many people who have no idea what
they're talking about.

(what horrid manners)

Hm. You may be right about the load average
calculation being broken.

Below is a 100 second profile sample of my 3GHz
P4 handling 15K
interrupts per second while receiving a flood
ping. My interpretation
is that tools should be showing ~10% cpu load
rather than zero. Am I'm
misinterpreting it?

97574 total
0.0258
89549 default_idle
1017.6023
1734 ioread16
36.8936
1138 ioread8
24.7391
974 rhine_start_tx
1.3994
534 __do_softirq
3.8417
331 handle_IRQ_event
3.2772
223 rhine_interrupt
0.0739
222 memset
7.9286
194 nf_iterate
1.5520
140 local_bh_enable
1.0769

    99 __kmalloc                               
  1.0532
    92 net_rx_action                           
  0.2000
    85 kfree                                   
  0.9884
    82 skb_release_data                        
  0.6406
    77 csum_partial_copy_generic               
  0.3105
    73 ip_push_pending_frames                  
  0.0681
    71 __alloc_skb                             
  0.2898
    69 kmem_cache_free                         
  1.3529
    66 kmem_cache_alloc                        
  1.3750
    62 csum_partial                            
  0.2153
    61 rt_hash_code                            
  0.4959
    61 ip_append_data                          
  0.0253
    60 netif_receive_skb                       
  0.0516
    58 ip_rcv                                  
Quoted 10 lines

0.0471
58 ip_local_deliver
0.0854
58 eth_type_trans
0.2489
55 ip_output
0.0957
52 icmp_reply
0.1187

Thats a pretty crappy controller you have in with that shiny P4...

I'm not sure that they want the tools to work. They'll just call you a troll and go on developing unnecessary things like NAPI because they're still using controllers designed by DEC (remember them?) back in the stone ages.

Yet I regularly encounter people using cheap NICs with expensive cpus on network-intensive applications. But you'd think one or two people would have a clue.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#9 lore raw
Quoted 3 lines · Danial Thom

Thats a pretty crappy controller you have in with that shiny P4...

Shrug. It's a grocery store desktop box, and I'm quite happy with it, but that has roughly zip to do with the price of tea in China.

Quoted 10 lines

I'm not sure that they want the tools to work. They'll just call you a troll and go on developing unnecessary things like NAPI because they're still using controllers designed by DEC (remember them?) back in the stone ages.

Yet I regularly encounter people using cheap NICs with expensive cpus on network-intensive applications. But you'd think one or two people would have a clue.

As does that.

-Mike

#10 lore raw
In reply to Danial Thom
Quoted 4 lines · Danial Thom
Quoted 4 lines
Quoted 2 lines · Danial Thom

Well thats just a ridiculous answer, so why
bother?

[...]

Quoted 1 line

I'm sorry, but you're being an idiot if you think

So much for good manners. I accept such replies only from a few people on this list, and you are not one of them.

*plonk*

Erik

Signature · 2 lines
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
#11 lore raw
In reply to Danial Thom
Quoted 5 lines · Danial Thom

Of course you folks could never really know that, because the tools to measure things don't work, and you all seem to be using systems that are so kludged up that you don't even know what you're measuring.

Next time attach a patch fixing, or at least bother to investigate the problem which you're so rudely flaming about. Maybe Linux is broken. Nobody said it wasn't. But you're closer to (and clearly care more about) the problem than 99% of the people reading your email.

So why don't you help fix it, instead of laying into Erik and the netdev people (who incidentally aren't CC'ed)?

Signature · 5 lines
Cheers,
Alistair.

Third year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.
#12 lore raw

--- Alistair John Strachan

Quoted 20 lines · <[email protected]> wrote:
Quoted 1 line · Danial Thom

Of course you folks could never really know

that,

Quoted 1 line

because the tools to measure things don't

work,

Quoted 1 line

and you all seem to be using systems that are

so

Quoted 1 line

kludged up that you don't even know what

you're

Quoted 1 line

measuring.

Next time attach a patch fixing, or at least
bother to investigate the problem
which you're so rudely flaming about. Maybe
Linux is broken. Nobody said it
wasn't. But you're closer to (and clearly care
more about) the problem than
99% of the people reading your email.

Quoted 3 lines

So why don't you help fix it, instead of laying
into Erik and the netdev
people (who incidentally aren't CC'ed)?

When you report a problem and some clown pipes in with "its not a problem, linux is just so fast it doesn't use any cpu!" you really don't have the foundation for coming up with a solution or helping to find one.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#13 lore raw
In reply to Erik Mouw

Hi Erik,

We have fixed this problem in the kernel 2.4.19 which has been installed in few selected computers of the "Middleware grid" Sydney University.

Also load average has been calculated for each login user.

Thanks
Sena Seneviratne
Computer Engineering Lab
Sydney University

Quoted 48 lines · you
Quoted 8 lines · Danial Thom
Quoted 3 lines · --- Erik Mouw

75K packets/s isn't too hard for modern NICs,
especially when using
NAPI.

Well thats just a ridiculous answer, so why bother?

Quoted 9 lines

You polling guys just crack me up. There isn't much less work to be done with polling. The only reason you THINK its less work is because the measuring tools don't work properly. You still have to process the same number of packets when you poll, and you have polls instead of interrupts. Since you can control the # of interrupts with most cards, there is zero advantage to polling, and more negatives.

There certainly is less work to be done with polling. Less IRQs means less expensive context switches, which means a lower system load. See Documentation/NAPI_HOWTO.txt for information and a link to the Linux NAPI paper.

Quoted 5 lines

And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.

Network traffic is usually IO bound, not CPU bound. The load figures top shows tell something about the amount of work the CPU has to do, not about how busy your PCI bus (or whatever bus the NIC lives on) is.

IIRC the networking layer in 2.6 differs quite a lot from 2.4, so the load average figures can be quite misleading.

Erik

--

+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#14 lore raw
In reply to Erik Mouw
Quoted 11 lines

Network traffic is usually IO bound, not CPU
bound. The load figures
top shows tell something about the amount of
work the CPU has to do,
not about how busy your PCI bus (or whatever
bus the NIC lives on) is.

IIRC the networking layer in 2.6 differs quite
a lot from 2.4, so the
load average figures can be quite misleading.

For the record, *most* of the work are I/O calls (ie reading and writing registers), which are not in the "background". I/O calls become more and more expensive as the bus becomes saturated as it takes longer to get the bus to do the operation.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#16 lore raw
Quoted 9 lines · --- Francois Romieu

Danial Thom <[email protected]> :
[bull removed]

You have been told twice in this thread to use
vmstat.

Read again and shut up until you can provide a
sensible bug report,
thank you.

vmstat gives the same #s as top, and it doesn't break down the interrupts, so its not the answer to the question "is there something similar to systat". Plus its clear that the guy who gave the answer doesn't know what he's talking about, since he's actually trying to explain away the problem as being normal.

Here's a sensible bug report. Bridging 75K pps. vmstat output:

 0  0      0 425424   9984  40496    0    0     0
    0 16238    12  0  0 100  0
 0  0      0 425424   9984  40496    0    0     0
    0 16227    16  0  0 100  0
 0  0      0 425424   9984  40496    0    0     0
    0 16244     9  0  0 100  0

Over 16,000 interrupts per second and 100% idle.

netstat:

eth2 1500 0 5 0 0 468886
0 0 0 BMRU
eth3 1500 473201 0 5680 0 5
0 0 0 BMRU

the system is perpetually 100% idle but its dropping packets due to excessive backlog.

DT

Quoted 13 lines
Signature · 11 lines
-- 
Ueimor
-
To unsubscribe from this list: send the line
"unsubscribe linux-kernel" in
the body of a message to
[email protected]
More majordomo info at 
http://vger.kernel.org/majordomo-info.html
Please read the FAQ at 
http://www.tux.org/lkml/

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#17 lore raw

Danial Thom <[email protected]> :
[...]

Quoted 4 lines

systat". Plus its clear that the guy who gave the answer doesn't know what he's talking about, since he's actually trying to explain away the problem as being normal.

"75 kpps means 10% of the max load" was quite fun too.

[...]

Quoted 2 lines

the system is perpetually 100% idle but its dropping packets due to excessive backlog.

No difference when you renice ksoftirqd to a strongly negative value ?

Signature · 1 line
Ueimor
#18 lore raw
Quoted 12 lines · --- Francois Romieu

Danial Thom <[email protected]> :
[...]

Quoted 1 line

systat". Plus its clear that the guy who gave

the

Quoted 2 lines

answer doesn't know what he's talking about,
since he's actually trying to explain away

the

Quoted 1 line

problem as being normal.

"75 kpps means 10% of the max load" was quite
fun too.

What's *fun* about it? 2.4.24 shows a 12-13%
load.

Quoted 7 lines

[...]

Quoted 2 lines

the system is perpetually 100% idle but its
dropping packets due to excessive backlog.

No difference when you renice ksoftirqd to a
strongly
negative value ?

No, not really. Running the compiler causes
drops. re-nicing eliminates drops with no
userland activity.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#19 lore raw

Danial Thom <[email protected]> :
[...]

Quoted 2 lines

What's *fun* about it? 2.4.24 shows a 12-13%
load.

The assumption of linearity.

[...]

Quoted 7 lines
Quoted 3 lines

No difference when you renice ksoftirqd to a
strongly
negative value ?

No, not really. Running the compiler causes
drops. re-nicing eliminates drops with no
userland activity.

It may be interesting to know if the same drops happen when you generate a disk-only load, ksoftirqd still at the minimal value.

Signature · 1 line
Ueimor
#20 lore raw
In reply to Danial Thom
Quoted 5 lines · Danial Thom

And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.

Are you by chance using procps < 3.1.12? The kernel reports absolute values for cpu usage, the conversion to percentage is done by top/vmstat itself. And those old versions don't know about the new fields that 2.6 kernels have in /proc/stat, thus they simply ignore the si and hi values, producing quite misleading results...

Björn

PS: procps 3.1.12 was released in 2003, so if DEC was stone age and my assumption about your tools holds, then your tools are like... medieval :)

#21 lore raw

Björn,

Yes you have to change the procps to read the new /proc/stat

For example, after separating the load and disk IO measurements, also separating for per user at the kernel level, I had to introduce new code into procps/ to reflect those changes

Otherwise those tools (top, uptime etc) would not know the new formats etc.

Thanks
Sena
Sydney University

Quoted 22 lines · you
Quoted 5 lines · Danial Thom

And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.

Are you by chance using procps < 3.1.12? The kernel reports absolute values for cpu usage, the conversion to percentage is done by top/vmstat itself. And those old versions don't know about the new fields that 2.6 kernels have in /proc/stat, thus they simply ignore the si and hi values, producing quite misleading results...

Björn

PS: procps 3.1.12 was released in 2003, so if DEC was stone age and my assumption about your tools holds, then your tools are like... medieval :) -
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#22 lore raw
In reply to Danial Thom
Quoted 5 lines · Danial Thom

And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.

For UP, yes. SMP kernel accounts irq processing time properly.

-Mike

#23 lore raw
Quoted 9 lines · Mike Galbraith
Quoted 5 lines · Danial Thom

And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.

For UP, yes. SMP kernel accounts irq processing time properly.

For my little box, the below cures it.

arch/i386/kernel/apic.c renamed from arch/i386/kernel/apic.c.org +0 -2
@@ -1175,9 +1175,7 @@ EXPORT_SYMBOL(switch_ipi_to_APIC_timer);
 inline void smp_local_timer_interrupt(struct pt_regs * regs)
 {
 	profile_tick(CPU_PROFILING, regs);
-#ifdef CONFIG_SMP
 	update_process_times(user_mode_vm(regs));
-#endif
 
 	/*
 	 * We take the 'long' return path, and there every subsystem
#24 lore raw
Quoted 10 lines · Mike Galbraith
Quoted 9 lines · Mike Galbraith
Quoted 5 lines · Danial Thom

And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.

For UP, yes. SMP kernel accounts irq processing time properly.

Do you actually see 100% idle? On both, UP and SMP, I see non-zero hi/si values using "top". With IO-APIC enabled, I see only non-zero si values for my tg3 NICs, and non-zero hi and si values for the nVidia NIC. With IO-APIC disabled, I also see a non-zero hi value for the tg3 on UP, guess that's normal... But I never see 100% idle while flooding the box with pings.

Quoted 5 lines

For my little box, the below cures it.

arch/i386/kernel/apic.c EXPORT_SYMBOL(switch_ipi_to_APIC_timer); · line 1181

Quoted 9 lines
arch/i386/kernel/apic.c renamed from arch/i386/kernel/apic.c.org +0 -2
@@ -1175,9 +1175,7 @@ EXPORT_SYMBOL(switch_ipi_to_APIC_timer);
 inline void smp_local_timer_interrupt(struct pt_regs * regs)
 {
 	profile_tick(CPU_PROFILING, regs);
-#ifdef CONFIG_SMP
 	update_process_times(user_mode_vm(regs));
-#endif
 
 	/*
 	 * We take the 'long' return path, and there every subsystem

The non-SMP call to update_process_times() is in do_timer_interrupt_hook(), so I guess the above is not the Right Thing to do. I don't even see how you could reach smp_local_timer_interrupt() without going through do_timer_interrupt_hook() first.

Björn

#25 lore raw
Quoted 10 lines · Björn Steinbrink
Quoted 9 lines · Mike Galbraith
Quoted 8 lines · Mike Galbraith
Quoted 5 lines · Danial Thom

And 75K pps may not be "much", but its still at least 10% of what the system can handle, so it should measure around a 10% load. 2.4 measures about 12% load. So the only conclusion is that load accounting is broken in 2.6.

Quoted 3 lines
Quoted 1 line
Quoted 1 line

For UP, yes. SMP kernel accounts irq processing time properly.

Do you actually see 100% idle?

Yes.

Quoted 6 lines

On both, UP and SMP, I see non-zero hi/si
values using "top". With IO-APIC enabled, I see only non-zero si values for my tg3 NICs, and non-zero hi and si values for the nVidia NIC. With IO-APIC disabled, I also see a non-zero hi value for the tg3 on UP, guess that's normal... But I never see 100% idle while flooding the box with pings.

I have APIC and IO-APIC enabled.

Quoted 12 lines
Quoted 12 lines

For my little box, the below cures it.

arch/i386/kernel/apic.c renamed from arch/i386/kernel/apic.c.org +0 -2
@@ -1175,9 +1175,7 @@ EXPORT_SYMBOL(switch_ipi_to_APIC_timer);
 inline void smp_local_timer_interrupt(struct pt_regs * regs)
 {
 	profile_tick(CPU_PROFILING, regs);
-#ifdef CONFIG_SMP
 	update_process_times(user_mode_vm(regs));
-#endif
 
Quoted 5 lines

arch/i386/kernel/apic.c EXPORT_SYMBOL(switch_ipi_to_APIC_timer); · line 1181

Quoted 2 lines
arch/i386/kernel/apic.c renamed from arch/i386/kernel/apic.c.org +0 -0
 	/*
 	 * We take the 'long' return path, and there every subsystem

The non-SMP call to update_process_times() is in do_timer_interrupt_hook(), so I guess the above is not the Right Thing to do.

Ah, there it is. That's what I was looking for. I figured that doing what I did had to be wrong, but tried it for grins anyway... was pretty surprised when it worked (kinda).

-Mike

#26 lore raw
Quoted 8 lines · Mike Galbraith
Quoted 3 lines · Björn Steinbrink

The non-SMP call to update_process_times() is in do_timer_interrupt_hook(), so I guess the above is not the Right Thing to do.

Ah, there it is. That's what I was looking for. I figured that doing what I did had to be wrong, but tried it for grins anyway... was pretty surprised when it worked (kinda).

Calling update_process_times() in do_timer_interrupt_hook() flat does not work here. Calling it in smp_local_timer_interrupt() works fine.

Oh joy.

-Mike

#27 lore raw
Quoted 6 lines · Mike Galbraith
Quoted 5 lines · Mike Galbraith
Quoted 3 lines · Björn Steinbrink

The non-SMP call to update_process_times() is in do_timer_interrupt_hook(), so I guess the above is not the Right Thing to do.

Quoted 8 lines
Quoted 3 lines

Ah, there it is. That's what I was looking for. I figured that doing what I did had to be wrong, but tried it for grins anyway... was pretty surprised when it worked (kinda).

Calling update_process_times() in do_timer_interrupt_hook() flat does not work here. Calling it in smp_local_timer_interrupt() works fine.

Oh joy.

I can reproduce it now, seems to require CONFIG_4KSTACKS to fail. Can you confirm that?

Björn

#28 lore raw
Quoted 11 lines · Björn Steinbrink
Quoted 10 lines · Mike Galbraith
Quoted 8 lines · Mike Galbraith
Quoted 3 lines · Björn Steinbrink

The non-SMP call to update_process_times() is in do_timer_interrupt_hook(), so I guess the above is not the Right Thing to do.

Ah, there it is. That's what I was looking for. I figured that doing what I did had to be wrong, but tried it for grins anyway... was pretty surprised when it worked (kinda).

Quoted 7 lines
Quoted 4 lines

Calling update_process_times() in do_timer_interrupt_hook() flat does not work here. Calling it in smp_local_timer_interrupt() works fine.

Oh joy.

I can reproduce it now, seems to require CONFIG_4KSTACKS to fail. Can you confirm that?

What a coincidence. After trying a different compiler, and slogging through a bunch of assembler trying to figure out how the heck this can happen, I was just booting an 8k stack kernel (as a wild-ass guess;).

let's see. Yeah, confirmed.

-Mike

#29 lore raw
Quoted 17 lines · Mike Galbraith
Quoted 16 lines · Björn Steinbrink
Quoted 14 lines · Mike Galbraith
Quoted 8 lines · Mike Galbraith
Quoted 3 lines · Björn Steinbrink

The non-SMP call to update_process_times() is in do_timer_interrupt_hook(), so I guess the above is not the Right Thing to do.

Ah, there it is. That's what I was looking for. I figured that doing what I did had to be wrong, but tried it for grins anyway... was pretty surprised when it worked (kinda).

Calling update_process_times() in do_timer_interrupt_hook() flat does not work here. Calling it in smp_local_timer_interrupt() works fine.

Oh joy.

Quoted 8 lines
Quoted 2 lines

I can reproduce it now, seems to require CONFIG_4KSTACKS to fail. Can you confirm that?

What a coincidence. After trying a different compiler, and slogging through a bunch of assembler trying to figure out how the heck this can happen, I was just booting an 8k stack kernel (as a wild-ass guess;).

let's see. Yeah, confirmed.

OK, it also depends on IO APIC being enabled and active, ie. noapic on the kernel command line will fix it as well as disabling CONFIG_X86_IO_APIC. That doesn't help me at all to understand why it happens though.
The only difference with IO APIC disabled seems to be that the irq doesn't get ACKed before update_process_times() gets called. And your "fix" makes it being called outside of the xtime_lock spinlock and with a slightly different stack usage AFAICT.
But none of these should make a difference, right?

Björn

#30 lore raw
Quoted 8 lines · Björn Steinbrink
Quoted 2 lines · Mike Galbraith

let's see. Yeah, confirmed.

OK, it also depends on IO APIC being enabled and active, ie. noapic on the kernel command line will fix it as well as disabling CONFIG_X86_IO_APIC. That doesn't help me at all to understand why it happens though.

Ditto.

Quoted 4 lines

The only difference with IO APIC disabled seems to be that the irq doesn't get ACKed before update_process_times() gets called. And your "fix" makes it being called outside of the xtime_lock spinlock and with a slightly different stack usage AFAICT.

(it's still under the xtime lock)

Quoted 1 line

But none of these should make a difference, right?

Not that I can see, but then it's pretty dark down here. Anybody got a flashlight I can borrow? ;-)

-Mike

#31 lore raw
Quoted 10 lines · Mike Galbraith
Quoted 8 lines · Björn Steinbrink
Quoted 2 lines · Mike Galbraith

let's see. Yeah, confirmed.

OK, it also depends on IO APIC being enabled and active, ie. noapic on the kernel command line will fix it as well as disabling CONFIG_X86_IO_APIC. That doesn't help me at all to understand why it happens though.

Quoted 8 lines

Ditto.

Quoted 4 lines

The only difference with IO APIC disabled seems to be that the irq doesn't get ACKed before update_process_times() gets called. And your "fix" makes it being called outside of the xtime_lock spinlock and with a slightly different stack usage AFAICT.

(it's still under the xtime lock)

No, with IO-APIC enabled, it's using the local APIC timer, thus using_apic_timer is 1 and the path right after unlocking in timer_interrupt() is taken towards update_process_times().

Quoted 4 lines
Quoted 1 line

But none of these should make a difference, right?

Not that I can see, but then it's pretty dark down here. Anybody got a flashlight I can borrow? ;-)

Guess I found one, not sure if it works correctly though ;)

Using 4K stacks, we have one stack for hard irqs and one for soft irqs, both having their own threadinfo and thus preemptcount. Thus the call to softirq_count() in account_system_time() will always return 0 when called in hard irq context. Additionally preemptcount is always set to HARDIRQ_OFFSET in hard irq context, so
hardirq_count() - hardirq_offset is 0 all the time as well.

But that doesn't fit the fact that I at least get hard irq accounting when booting with noapic. And it also doesn't explain why your fix works, fixing both, soft and hard irq accounting. Am I missing some code path that calls smp_local_timer_interrupt? There's smp_apic_timer_interrupt(), but that seems to be unused on i386.

Björn

#32 lore raw

PATCH 1/1 i386: Fix softirq accounting with 4K stacks

Quoted 34 lines · Björn Steinbrink
Quoted 6 lines · Mike Galbraith
Quoted 4 lines · Björn Steinbrink

OK, it also depends on IO APIC being enabled and active, ie. noapic on the kernel command line will fix it as well as disabling CONFIG_X86_IO_APIC. That doesn't help me at all to understand why it happens though.

Quoted 8 lines

Ditto.

Quoted 4 lines

The only difference with IO APIC disabled seems to be that the irq doesn't get ACKed before update_process_times() gets called. And your "fix" makes it being called outside of the xtime_lock spinlock and with a slightly different stack usage AFAICT.

(it's still under the xtime lock)

No, with IO-APIC enabled, it's using the local APIC timer, thus using_apic_timer is 1 and the path right after unlocking in timer_interrupt() is taken towards update_process_times().

Quoted 4 lines
Quoted 1 line

But none of these should make a difference, right?

Not that I can see, but then it's pretty dark down here. Anybody got a flashlight I can borrow? ;-)

Guess I found one, not sure if it works correctly though ;)

Using 4K stacks, we have one stack for hard irqs and one for soft irqs, both having their own threadinfo and thus preemptcount. Thus the call to softirq_count() in account_system_time() will always return 0 when called in hard irq context. Additionally preemptcount is always set to HARDIRQ_OFFSET in hard irq context, so
hardirq_count() - hardirq_offset is 0 all the time as well.

Quoted 5 lines

But that doesn't fit the fact that I at least get hard irq accounting when booting with noapic. And it also doesn't explain why your fix works, fixing both, soft and hard irq accounting. Am I missing some code path that calls smp_local_timer_interrupt? There's smp_apic_timer_interrupt(), but that seems to be unused on i386.

OK, the hardirq_count() thing is not HARDIRQ_OFFSET all the time. When we interrupt a hard irq, we are already using the hard irq stack and thus irq_enter() in do_IRQ() works as it should. The zero value for hi is there with IO-APIC and 8K stacks as well, so that's either right or an other bug.

Still no idea why your "fix" works, but the following patch also fixes the problem and is at least a little more like the RightThing.

---

Copy the softirq bits in preempt_count from the current context into the hardirq context when using 4K stacks to make the softirq_count macro work correctly and thereby fix softirq cpu time accounting.

arch/i386/kernel/irq.c +8 -0
diff -Nurp a/arch/i386/kernel/irq.c b/arch/i386/kernel/irq.c
@@ -95,6 +95,14 @@ fastcall unsigned int do_IRQ(struct pt_r
 		irqctx->tinfo.task = curctx->tinfo.task;
 		irqctx->tinfo.previous_esp = current_stack_pointer;
 
+		/*
+		 * Copy the softirq bits in preempt_count so that the
+		 * softirq checks work in the hardirq context.
+		 */
+		irqctx->tinfo.preempt_count =
+			irqctx->tinfo.preempt_count & ~SOFTIRQ_MASK |
+			curctx->tinfo.preempt_count & SOFTIRQ_MASK;
+
 		asm volatile(
 			"       xchgl   %%ebx,%%esp      \n"
 			"       call    __do_IRQ         \n"
Acked-by on v1 1/1
#33 lore raw

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

Quoted 14 lines · Björn Steinbrink

Copy the softirq bits in preempt_count from the current context into the hardirq context when using 4K stacks to make the softirq_count macro work correctly and thereby fix softirq cpu time accounting.

diff -Nurp a/arch/i386/kernel/irq.c b/arch/i386/kernel/irq.c

arch/i386/kernel/irq.c +1 -0
@@ -95,6 +95,14 @@ fastcall unsigned int do_IRQ(struct pt_r
 		irqctx->tinfo.task = curctx->tinfo.task;
 		irqctx->tinfo.previous_esp = current_stack_pointer;
 
+		/*

arch/i386/kernel/irq.c fastcall unsigned int do_IRQ(struct pt_r · line 108

Quoted 10 lines
arch/i386/kernel/irq.c +7 -0
+		 * Copy the softirq bits in preempt_count so that the
+		 * softirq checks work in the hardirq context.
+		 */
+		irqctx->tinfo.preempt_count =
+			irqctx->tinfo.preempt_count & ~SOFTIRQ_MASK |
+			curctx->tinfo.preempt_count & SOFTIRQ_MASK;
+
 		asm volatile(
 			"       xchgl   %%ebx,%%esp      \n"
 			"       call    __do_IRQ         \n"

Hi,

at first I got nervous about the asymmetry of this (eg why only do this copying only on entry, and not a copy back on exit)... but then again these bits shouldn't change so your patch is ok as is... it's regrettable that we need to add this for the softirq accounting; part of me wishes we would just count irq-vs-user time and be done with it ;)

#34 lore raw

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

In reply to Björn Steinbrink
Quoted 2 lines · Björn Steinbrink

Still no idea why your "fix" works, but the following patch also fixes the problem and is at least a little more like the RightThing.

Yeah. I don't know about you, but I fully intend to blatantly ignore that 'why' ;-)

-Mike

#35 lore raw

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

Quoted 7 lines · Mike Galbraith
Quoted 2 lines · Björn Steinbrink

Still no idea why your "fix" works, but the following patch also fixes the problem and is at least a little more like the RightThing.

Yeah. I don't know about you, but I fully intend to blatantly ignore that 'why' ;-)

the why is relatively easy ;)

since the "is a softirq executing" bit is on the stack, and each context (user, soft and hard irq) has their own stack, it's not automatic that the hardirq stack gets the softirq-executing flag... which your patch fixes.

NMI's and apic irqs generally don't go via the normal irq path and thus don't do a stack switch... so they don't lose the bit (for accounting purposes)

#36 lore raw

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

Quoted 9 lines · Arjan van de Ven
Quoted 7 lines · Mike Galbraith
Quoted 2 lines · Björn Steinbrink

Still no idea why your "fix" works, but the following patch also fixes the problem and is at least a little more like the RightThing.

Yeah. I don't know about you, but I fully intend to blatantly ignore that 'why' ;-)

Quoted 6 lines

the why is relatively easy ;)

since the "is a softirq executing" bit is on the stack, and each context (user, soft and hard irq) has their own stack, it's not automatic that the hardirq stack gets the softirq-executing flag... which your patch fixes.

That's mine, not Mike's. Mike's patch removed the #ifdef CONFIG_SMP around update_process_times() in smp_local_timer_interrupt().

Quoted 3 lines

NMI's and apic irqs generally don't go via the normal irq path and thus don't do a stack switch... so they don't lose the bit (for accounting purposes)

Hm, doesn't that mean that mean that hardirq accounting is still broken? APIC irq comes in, increases hardirq count, then the timer irq fires, switches the stack and looses the hardirq count that was incremented by the APIC irq.

I just booted with both patches applied, mine and Mike's, and that actually makes a difference in hardirq cpu time accounting. With my patch only, hi is 0 in top while the box gets a ping flood. With both patches, I get about 1% hi. Mike's patch causes update_process_times() to be called twice on UP, but that alone shouldn't change the percentages, right?
OTOH top shows "hi" as zero with 8K stacks as well unless Mike's patch is applied, so the results with Mike's patch are bogus (if so, why?) or hardirq accounting is broken in general.

Btw, which path do apic irqs go? I stumbled across the nmi stuff, but didn't see anything special for the apic irqs.

Björn

#37 lore raw

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

Quoted 6 lines · Björn Steinbrink

I just booted with both patches applied, mine and Mike's, and that actually makes a difference in hardirq cpu time accounting. With my patch only, hi is 0 in top while the box gets a ping flood. With both patches, I get about 1% hi. Mike's patch causes update_process_times() to be called twice on UP, but that alone shouldn't change the percentages, right?

Yes, you definitely need to comment out the other call if you test the SMP path on UP+IO-APIC.

Quoted 3 lines

OTOH top shows "hi" as zero with 8K stacks as well unless Mike's patch is applied, so the results with Mike's patch are bogus (if so, why?) or hardirq accounting is broken in general.

Something is certainly still b0rken. I still get three different answers to the question "what is my cpu usage" depending on configuration. With stock UP kernel with no IO-APIC, interrupt load is all hi. With your patch and IO-APIC, it's all si. SMP shows a mix of both.

I like the result of using the SMP path if you have an IO-APIC best, though I haven't verified them against a profile for accuracy. Taking a peek at the profile confirms that it is indeed mixed, so anything showing the load as being either hi or si has to be wrong.

-Mike

#38 lore raw

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

Quoted 5 lines · Mike Galbraith

Something is certainly still b0rken. I still get three different answers to the question "what is my cpu usage" depending on configuration. With stock UP kernel with no IO-APIC, interrupt load is all hi. With your patch and IO-APIC, it's all si. SMP shows a mix of both.

I didn't say that quite right.

The third case for my box is XT-PIC and your patch. Previously, top showed the 10% interrupt load of a flood ping as all hi. Now it's both hi and si when using XT-PIC, but it's no longer the 10% that agreed with the profile, it's 10% hi, but with an added ~7% si.

-Mike

#39 lore raw

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

Quoted 8 lines · Mike Galbraith
Quoted 5 lines · Mike Galbraith

Something is certainly still b0rken. I still get three different answers to the question "what is my cpu usage" depending on configuration. With stock UP kernel with no IO-APIC, interrupt load is all hi. With your patch and IO-APIC, it's all si. SMP shows a mix of both.

Quoted 6 lines

I didn't say that quite right.

The third case for my box is XT-PIC and your patch. Previously, top showed the 10% interrupt load of a flood ping as all hi. Now it's both hi and si when using XT-PIC, but it's no longer the 10% that agreed with the profile, it's 10% hi, but with an added ~7% si.

I see the following here, when a box is being ping flooded:

UP/SMP stack-size PIC-type hi si
-----------------------------------

UP      8K          IO-APIC  0  16
UP      8K          XT-PIC   7  7
UP      4K          IO-APIC  0  16
UP      4K          XT-PIC   7  7
SMP     8K          IO-APIC  5  4 (forcedeth)
SMP     8K          IO-APIC  0  11 (tg3)

The UP system is a Thinkpad which only has a tg3 driven NIC. The SMP system is x86_64, so there's no 4K stacks to test with, maybe I'll fetch a i386 live CD to do some more valid tests on SMP.

The UP tests seem to show the IO-APIC hardirq are completely deferred to be handled in a softirq, as the sum of "hi" and "si" with XT-PIC is about equal to the "si" value with IO-APIC (the numbers are guessed averages of the observed values, so the 2% difference is not too be taken too serious).
The results for the forcedeth driven NIC do not agree though, and your results differ from what I see as well, right? So I'm kinda lost again.

Björn

#40 lore raw

Re: [PATCH] i386: Fix softirq accounting with 4K stacks

Quoted 2 lines · Björn Steinbrink

The results for the forcedeth driven NIC do not agree though, and your results differ from what I see as well, right? So I'm kinda lost again.

I'm not. Not any more. After much fruitless rummaging, I did some more profiling to verify the numbers. It really does take that much more cpu when booted with noapic, nearly 50%! I've been chasing swamp gas.

Oh well. I got a warm fuzzy wrt tools cpu usage numbers out of it.

Stick a fork in this bug, it's done ;-)

-Mike

#41 lore raw
In reply to Björn Steinbrink
Quoted 70 lines · --- Björn Steinbrink
Quoted 1 line · Mike Galbraith

On Sat, 2006-06-24 at 21:25 +0200, Björn

Quoted 1 line · Steinbrink
Quoted 1 line

On 2006.06.24 18:23:12 +0200, Mike

Quoted 4 lines · Galbraith
Quoted 4 lines
Quoted 2 lines

let's see. Yeah, confirmed.

OK, it also depends on IO APIC being

enabled and active, ie. noapic on

Quoted 1 line
Quoted 1 line

the kernel command line will fix it as well

as disabling

Quoted 1 line
Quoted 1 line

CONFIG_X86_IO_APIC. That doesn't help me at

all to understand why it

Quoted 5 lines
Quoted 1 line

happens though.

Ditto.

Quoted 1 line

The only difference with IO APIC disabled

seems to be that the irq

Quoted 1 line
Quoted 1 line

doesn't get ACKed before

update_process_times() gets called.

Quoted 1 line
Quoted 1 line

And your "fix" makes it being called

outside of the xtime_lock spinlock

Quoted 1 line
Quoted 1 line

and with a slightly different stack usage

AFAICT.

Quoted 2 lines

(it's still under the xtime lock)

No, with IO-APIC enabled, it's using the local
APIC timer, thus
using_apic_timer is 1 and the path right after
unlocking in
timer_interrupt() is taken towards
update_process_times().

Quoted 1 line
Quoted 1 line

But none of these should make a difference,

right?

Quoted 2 lines

Not that I can see, but then it's pretty dark

down here. Anybody got a

Quoted 1 line

flashlight I can borrow? ;-)

Guess I found one, not sure if it works
correctly though ;)

Using 4K stacks, we have one stack for hard
irqs and one for soft irqs,
both having their own threadinfo and thus
preemptcount. Thus the call to
softirq_count() in account_system_time() will
always return 0 when
called in hard irq context. Additionally
preemptcount is always set to
HARDIRQ_OFFSET in hard irq context, so
hardirq_count() - hardirq_offset is 0 all the
time as well.

But that doesn't fit the fact that I at least
get hard irq accounting
when booting with noapic. And it also doesn't
explain why your fix
works, fixing both, soft and hard irq
accounting. Am I missing some
code path that calls smp_local_timer_interrupt?
There's
smp_apic_timer_interrupt(), but that seems to
be unused on i386.

Quoted 1 line

Björn

I think the one thing we can surmise from this thread is that you can't rely on kernel usage statistics to be accurate, as its likely that there are many, many cases that don't work properly. It was always wrong in 2.4 as well.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#42 lore raw
Quoted 5 lines · Danial Thom

I think the one thing we can surmise from this thread is that you can't rely on kernel usage statistics to be accurate, as its likely that there are many, many cases that don't work properly. It was always wrong in 2.4 as well.

Once identified, problems tend to get fixed. This one will probably be history soon. You know the old saying though... "There are lies, there are _damn_ lies, and then there are _statistics_".

#43 lore raw
Quoted 10 lines · --- Mike Galbraith
Quoted 1 line · Danial Thom

I think the one thing we can surmise from

this

Quoted 4 lines

thread is that you can't rely on kernel usage statistics to be accurate, as its likely that there are many, many cases that don't work properly. It was always wrong in 2.4 as well.

Quoted 6 lines

Once identified, problems tend to get fixed. This one will probably be
history soon. You know the old saying though... "There are lies, there
are _damn_ lies, and then there are _statistics_".

The usefulness of statistics is a function of the breadth of understand of the person interpreting them.

But I don't think that applies here, because we are not questioning the conclusions from the statistics, only the accuracy of the gathering. Its a completely different thing to say, for example, that population growth is due to too many babies when it might be immigration or a reduction in the death rate, than it is to say that the numbers gathered are simply wrong.

The truth is that you guys can't be bothered unless the numbers are so far off that its an embarrasment. Until someone who cares about it (and knows how to do it) puts their nose to the grindstone and makes sure that everything is being done correctly, they'll be largely useless.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#44 lore raw
Quoted 33 lines · Danial Thom
Quoted 10 lines · --- Mike Galbraith
Quoted 1 line · Danial Thom

I think the one thing we can surmise from

this

Quoted 4 lines

thread is that you can't rely on kernel usage statistics to be accurate, as its likely that there are many, many cases that don't work properly. It was always wrong in 2.4 as well.

Quoted 6 lines

Once identified, problems tend to get fixed. This one will probably be
history soon. You know the old saying though... "There are lies, there
are _damn_ lies, and then there are _statistics_".

The usefulness of statistics is a function of the breadth of understand of the person interpreting them.

But I don't think that applies here, because we are not questioning the conclusions from the statistics, only the accuracy of the gathering. Its a completely different thing to say, for example, that population growth is due to too many babies when it might be immigration or a reduction in the death rate, than it is to say that the numbers gathered are simply wrong.

Quoted 3 lines

The truth is that you guys can't be bothered unless the numbers are so far off that its an embarrasment.

Hey, let's play hide and go fuck yourself. Tag, you're it.

*plonk*

-Mike