Skip to content

*** Detect interrupt storm in softlockup ***

Bitao Hu linux-mips linux-parisc linuxppc-dev lkml 14 versions – Applied

Status from this message: The following commit has been merged into the irq/core branch of tip:

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v13 Latest5 patches 0 repliesApplied
  2. v124 patches 6 repliesSuperseded
  3. v114 patches 9 repliesSuperseded
Earlier versions · 11
VersionPostedPatchesRepliesStatus
v104 patches7 repliesSuperseded
v93 patches3 repliesSuperseded
v82 patches2 repliesSuperseded
v72 patches3 repliesSuperseded
v62 patches12 repliesSuperseded
v53 patches8 repliesSuperseded
v43 patches2 repliesSuperseded
v3 posting 2/22 patches7 repliesSuperseded
v3 posting 1/22 patches1 replySuperseded
v22 patches4 repliesSuperseded
v13 patches11 repliesSuperseded

Changes in v13 from the cover letter

- Update patch #1 based on the latest kernel code.

- From Thomas, split patch #1 into two. The new patch #1 converts kstat_irqs into a struct with just the count in it; the new patch #2 introduces a snapshot mechanism for interrupt statistics. Due to the code being split, I removed the Reviewed-by tags from LiuSong and Douglas in patch #1 and patch #2.
Please review it again, and all comments are welcome.

- Revised the comment for using printk() instead of pr_crit() to make the reasoning clearer.

Patches in v13 Applied

#Subject and filesSizeRepliesApplied
1/5genirq: Convert kstat_irqs to a structarch/mips/dec/setup.c arch/parisc/kernel/smp.c arch/powerpc/kvm/book3s_hv_rm_xics.c +5 more+23 -170 replies
2/5genirq: Provide a snapshot mechanism for interrupt statisticsinclude/linux/irqdesc.h include/linux/kernel_stat.h kernel/irq/Kconfig +1 more+41 -00 replies
3/5genirq: Avoid summation loops for /proc/interruptskernel/irq/internals.h kernel/irq/irqdesc.c kernel/irq/proc.c+15 -90 replies
4/5watchdog/softlockup: low-overhead detection of interrupt stormkernel/watchdog.c lib/Kconfig.debug+112 -10 replies
5/5watchdog/softlockup: report the most frequent interruptskernel/watchdog.c+112 -40 replies

Reviews trailers per patch

3 of 5 patches have a Reviewed-by or Acked-by.

PatchThomas GleixnerLiu SongDouglas Anderson
1/5Originally-by
2/5Originally-by
3/5Originally-by R-b R-b
4/5R-b R-b
5/5R-b R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v13 thread

Cover letter [PATCHv13 0/5] *** Detect interrupt storm in softlockup ***

Hi, guys.
I have implemented a low-overhead method for detecting interrupt storm in softlockup. Please review it, all comments are welcome.

Changes from v12 to v13:

- Update patch #1 based on the latest kernel code.

- From Thomas, split patch #1 into two. The new patch #1 converts kstat_irqs into a struct with just the count in it; the new patch #2 introduces a snapshot mechanism for interrupt statistics. Due to the code being split, I removed the Reviewed-by tags from LiuSong and Douglas in patch #1 and patch #2.
Please review it again, and all comments are welcome.

- Revised the comment for using printk() instead of pr_crit() to make the reasoning clearer.

Changes from v11 to v12:

- From Douglas and Thomas, add a new kconfig knob save memory when the softlock detector code is not enabled.

- Adjust the order of the patches; patch #1 and patch #2 are related to genirq, while patch #3 and patch #4 are related to watchdog/softlockup, making the dependency relationships clearer.

- Add the 'Reviewed-by' tag of Douglas.

Changes from v10 to v11:

- Only patch #2 and patch #3 have been changed.

- Add comments to explain each field of 'struct irqstat' in patch #2.

- Split the inner summation logic out of kstat_irqs() and encapsulate it into kstat_irqs_desc() in patch #3.

- Adopt Thomas's change log for patch #3.

- Add the 'Reviewed-by' tag of Liu Song.

Changes from v9 to v10:

- The two patches related to 'watchdog/softlockup' remain unchanged.

- The majority of the work related to 'genirq' is contributed by Thomas, indicated by adding 'Originally-by' tag. And I'd like to express my gratitude for Thomas's contributions and guidance here.

- Adopt Thomas's change log for the snapshot mechanism for interrupt statistics.

- Split unrelated change in patch #2 into a separate patch #3.

Changes from v8 to v9:

- Patch #1 remains unchanged.

- From Thomas Gleixner, split patch #2 into two patches. Interrupt infrastructure first and then the actual usage site in the watchdog code.

Changes from v7 to v8:

- From Thomas Gleixner, implement statistics within the interrupt core code and provide sensible interfaces for the watchdog code.

- Patch #1 remains unchanged. Patch #2 has significant changes based on Thomas's suggestions, which is why I have removed Liu Song and Douglas's Reviewed-by from patch #2. Please review it again, and all comments are welcome.

Changes from v6 to v7:

- Remove "READ_ONCE" in "start_counting_irqs"

- Replace the hard-coded 5 with "NUM_SAMPLE_PERIODS" macro in "set_sample_period".

- Add empty lines to help with reading the code.

- Remove the branch that processes IRQs where "counts_diff = 0".

- Add the Reviewed-by of Liu Song and Douglas.

Changes from v5 to v6:

- Use "./scripts/checkpatch.pl --strict" to get a few extra style nits and fix them.

- Squash patch #3 into patch #1, and wrapp the help text to 80 columns.

- Sort existing headers alphabetically in watchdog.c

- Drop "softlockup_hardirq_cpus", just read "hardirq_counts" and see if it's non-NULL.

- Store "nr_irqs" in a local variable.

- Simplify the calculation of "cpu_diff".

Changes from v4 to v5:

- Rearranging variable placement to make code look neater.

Changes from v3 to v4:

- Renaming some variable and function names to make the code logic more readable.

- Change the code location to avoid predeclaring.

- Just swap rather than a double loop in tabulate_irq_count.

- Since nr_irqs has the potential to grow at runtime, bounds-check logic has been implemented.

- Add SOFTLOCKUP_DETECTOR_INTR_STORM Kconfig knob.

Changes from v2 to v3:

- From Liu Song, using enum instead of macro for cpu_stats, shortening the name 'idx_to_stat' to 'stats', adding 'get_16bit_precesion' instead of using right shift operations, and using 'struct irq_counts'.

- From kernel robot test, using '__this_cpu_read' and '__this_cpu_write' instead of accessing to an per-cpu array directly, in order to avoid this warning.
'sparse: incorrect type in initializer (different modifiers)'

Changes from v1 to v2:

- From Douglas, optimize the memory of cpustats. With the maximum number of CPUs, that's now this.
2 * 8192 * 4 + 1 * 8192 * 5 * 4 + 1 * 8192 = 237,568 bytes.

- From Liu Song, refactor the code format and add necessary comments.

- From Douglas, use interrupt counts instead of interrupt time to determine the cause of softlockup.

- Remove the cmdline parameter added in PATCHv1.

Bitao Hu (5):
genirq: Convert kstat_irqs to a struct
genirq: Provide a snapshot mechanism for interrupt statistics
genirq: Avoid summation loops for /proc/interrupts watchdog/softlockup: low-overhead detection of interrupt storm watchdog/softlockup: report the most frequent interrupts

 arch/mips/dec/setup.c                |   2 +-
 arch/parisc/kernel/smp.c             |   2 +-
 arch/powerpc/kvm/book3s_hv_rm_xics.c |   2 +-
 include/linux/irqdesc.h              |  16 +-
 include/linux/kernel_stat.h          |   8 +
 kernel/irq/Kconfig                   |   4 +
 kernel/irq/internals.h               |   4 +-
 kernel/irq/irqdesc.c                 |  50 +++++--
 kernel/irq/proc.c                    |   9 +-
 kernel/watchdog.c                    | 215 ++++++++++++++++++++++++++-
 lib/Kconfig.debug                    |  14 ++
 scripts/gdb/linux/interrupts.py      |   6 +-
 12 files changed, 302 insertions(+), 30 deletions(-)
Signature · 1 line
2.37.1 (Apple Git-137.1)