genirq, proc: Speedup /proc/stat interrupt statistics

Waiman reported that on large systems with a large amount of interrupts the readout of /proc/stat takes a long time to sum up the interrupt statistics. In principle this is not a problem. but for unknown reasons some enterprise quality software reads /proc/stat with a high frequency.

The reason for this is that interrupt statistics are accounted per cpu. So the /proc/stat logic has to sum up the interrupt stats for each interrupt.

This can be largely avoided for interrupts which are not marked as 'PER_CPU' interrupts by simply adding a per interrupt summation counter which is incremented along with the per interrupt per cpu counter.

The PER_CPU interrupts need to avoid that and use only per cpu accounting because they share the interrupt number and the interrupt descriptor and concurrent updates would conflict or require unwanted synchronization.

...

Quoted 18 lines · On Fri, 08 Feb 2019 14:48:03 +0100 Thomas Gleixner <[email protected]> wrote:

Waiman reported that on large systems with a large amount of interrupts the readout of /proc/stat takes a long time to sum up the interrupt statistics. In principle this is not a problem. but for unknown reasons some enterprise quality software reads /proc/stat with a high frequency.

The reason for this is that interrupt statistics are accounted per cpu. So the /proc/stat logic has to sum up the interrupt stats for each interrupt.

This can be largely avoided for interrupts which are not marked as 'PER_CPU' interrupts by simply adding a per interrupt summation counter which is incremented along with the per interrupt per cpu counter.

The PER_CPU interrupts need to avoid that and use only per cpu accounting because they share the interrupt number and the interrupt descriptor and concurrent updates would conflict or require unwanted synchronization.

...

include/linux/irqdesc.h struct irq_desc { · line 68

Quoted 7 lines

include/linux/irqdesc.h +1 -0

@@ -65,6 +65,7 @@ struct irq_desc {
 	unsigned int		core_internal_state__do_not_mess_with_it;
 	unsigned int		depth;		/* nested irq disables */
 	unsigned int		wake_depth;	/* nested wake enables */
+	unsigned int		tot_count;

Confused. Isn't this going to quickly overflow?

Quoted 20 lines · Andrew Morton

Quoted 18 lines · On Fri, 08 Feb 2019 14:48:03 +0100 Thomas Gleixner <[email protected]> wrote:

Waiman reported that on large systems with a large amount of interrupts the readout of /proc/stat takes a long time to sum up the interrupt statistics. In principle this is not a problem. but for unknown reasons some enterprise quality software reads /proc/stat with a high frequency.

The reason for this is that interrupt statistics are accounted per cpu. So the /proc/stat logic has to sum up the interrupt stats for each interrupt.

This can be largely avoided for interrupts which are not marked as 'PER_CPU' interrupts by simply adding a per interrupt summation counter which is incremented along with the per interrupt per cpu counter.

The PER_CPU interrupts need to avoid that and use only per cpu accounting because they share the interrupt number and the interrupt descriptor and concurrent updates would conflict or require unwanted synchronization.

...

Quoted 10 lines

include/linux/irqdesc.h struct irq_desc { · line 68

Quoted 7 lines

include/linux/irqdesc.h +1 -0

@@ -65,6 +65,7 @@ struct irq_desc {
 	unsigned int		core_internal_state__do_not_mess_with_it;
 	unsigned int		depth;		/* nested irq disables */
 	unsigned int		wake_depth;	/* nested wake enables */
+	unsigned int		tot_count;

Confused. Isn't this going to quickly overflow?

All the current irq count computations for each individual irqs are using unsigned int type. Only the sum of all the irqs is u64. Yes, it is possible for an individual irq count to exceed 32 bits given sufficient uptime.  My PC has an uptime of 36 days and the highest irq count value is 79,227,699. Given the current rate, the overflow will happen after about 5 years. A larger server system may have an overflow in much shorter period. So maybe we should consider changing all the irq counts to unsigned long then.

It sounds like it. A 10khz interrupt will overflow in 4 days...

Sent: 09 February 2019 03:41

Quoted 1 line · Andrew Morton

It sounds like it. A 10khz interrupt will overflow in 4 days...

If you've got a 10kHz interrupt, you have a bigger problem. Anything happening 10,000 times a second is going to need interrupt mitigation to perform acceptably.

More importantly, userspace can (and must) cope with wrapping. This isn't anything new from Thomas' patch. As long as userspace is polling more often than once a day, it's going to see a wrapped value before it wraps again, so it won't miss 4 billion interrupts.

Waiman reported that on large systems with a large amount of interrupts the readout of /proc/stat takes a long time to sum up the interrupt statistics. In principle this is not a problem. but for unknown reasons some enterprise quality software reads /proc/stat with a high frequency.

The reason for this is that interrupt statistics are accounted per cpu. So the /proc/stat logic has to sum up the interrupt stats for each interrupt.

The following series addresses this by making the interrupt statitics code in the core generate the sum directly and by making the loop in the /proc/stat read function smarter.

V1 -> V2: Address review feedback: undo struct layout changes, make
variables unsigned and add test results to the changelog.

Thanks,

tglx

8<----------------

 fs/proc/stat.c          |   29 ++++++++++++++++++++++++++---
 include/linux/irqdesc.h |    1 +
 kernel/irq/chip.c       |   12 ++++++++++--
 kernel/irq/internals.h  |    8 +++++++-
 kernel/irq/irqdesc.c    |    7 ++++++-
 5 files changed, 50 insertions(+), 7 deletions(-)

Waiman reported that on large systems with a large amount of interrupts the readout of /proc/stat takes a long time to sum up the interrupt statistics. In principle this is not a problem. but for unknown reasons some enterprise quality software reads /proc/stat with a high frequency.

The reason for this is that interrupt statistics are accounted per cpu. So the /proc/stat logic has to sum up the interrupt stats for each interrupt.

The following series addresses this by making the interrupt statitics code in the core generate the sum directly and by making the loop in the /proc/stat read function smarter.

V1 -> V2: Address review feedback: undo struct layout changes, make
variables unsigned and add test results to the changelog.

Thanks,

tglx

8<----------------

fs/proc/stat.c          |   29 ++++++++++++++++++++++++++---
include/linux/irqdesc.h |    1 +
kernel/irq/chip.c       |   12 ++++++++++--
kernel/irq/internals.h  |    8 +++++++-
kernel/irq/irqdesc.c    |    7 ++++++-
5 files changed, 50 insertions(+), 7 deletions(-)

Waiman reported that on large systems with a large amount of interrupts the readout of /proc/stat takes a long time to sum up the interrupt statistics. In principle this is not a problem. but for unknown reasons some enterprise quality software reads /proc/stat with a high frequency.

The reason for this is that interrupt statistics are accounted per cpu. So the /proc/stat logic has to sum up the interrupt stats for each interrupt.

The following series addresses this by making the interrupt statitics code in the core generate the sum directly and by making the loop in the /proc/stat read function smarter.

V1 -> V2: Address review feedback: undo struct layout changes, make
variables unsigned and add test results to the changelog.

Thanks,

tglx

8<----------------

 fs/proc/stat.c          |   29 ++++++++++++++++++++++++++---
 include/linux/irqdesc.h |    1 +
 kernel/irq/chip.c       |   12 ++++++++++--
 kernel/irq/internals.h  |    8 +++++++-
 kernel/irq/irqdesc.c    |    7 ++++++-
 5 files changed, 50 insertions(+), 7 deletions(-)