Global Software Interrupt Moderation (GSIM)
Subsystems (6)
v5 cover Global Software Interrupt Moderation (GSIM)
Global Software Interrupt Moderation (GSIM) addresses a limitation of platforms, from many vendors, whose I/O performance drops significantly when the total rate of MSI-X interrupts is too high (e.g 1..3M intr/s depending on the platform).
Conventional interrupt moderation, typically implemented in hardware by NICs or storage devices, operates separately on each source (e.g. a completion queue). Large servers can have hundreds of sources, and without knowledge of global activity, keeping the total rate bounded would require moderation delays of 100-200us. Per-source adaptive moderation would reach those delays with as little as 10K intr/s per source. These values (high delay triggering very early) are unacceptable for RPC or transactional workloads.
To address this problem, GSIM measures efficiently the total and per-CPU interrupt rates, so that individual moderation delays can be dynamically adjusted based on current global and local load. This way, delays are normally 0 or very small except during actual local/global overload.
As an additional benefit, GSIM also monitors the percentage of time spent by each CPU in hardirq, and can use moderation to reserve some time for other, lower priority, tasks.
Configuration is easy and robust. System administrators specify the maximum targets (moderation delay; interrupt rate; percentage of time spent in hardirq), and which interrupt sources should be moderated. Independent per-CPU control loops adjust actual delays to try and keep metrics within the targets.
The system is adaptive. Moderation affects only latency and only in high load scenarios. Throughput and CPU efficiencly generally benefits significantly. Targets don't need to match precisely the platform limits, and one can make conservative and robust choices. Values like delay_us=100, target_intr_rate=1000000, hardirq_percent=70 are a very good starting point.
GSIM does not rely on any special hardware feature.
Global parameters can be modified at runtime with
echo ${VALUE} | sudo tee /proc/irq/sw_moderation/${NAME}/proc/irq/sw_moderation/stats exports statistics when enabled.
and moderation on individual interrupts can be turned on/off at runtime with
echo 1 | sudo tee /proc/irq/NN/allow_moderation # use 0 to disable
EXAMPLE:
# global configuration: 50us, target max 1M intr/s, and 70% in intr
echo 50 | sudo tee /proc/irq/sw_moderation/delay_us
echo 1000000 | sudo tee /proc/irq/sw_moderation/target_intr_rate
echo 70 | sudo tee /proc/irq/sw_moderation/hardirq_percent # allow moderation on all interrupts that support it
# remember to periodically check and set the flag for dynamically
# created interrupts since the default is 0
echo 1 | sudo tee /proc/irq/*/allow_moderation # check the status
grep -r . /proc/irq/*/*/../allow_moderation
# look at statistics
less /proc/irq/sw_moderation/stats
PERFORMANCE BENEFITS:
Below are some experimental results under high load comparing conventional moderation with GSIM:
- 100Gbps NIC, 32 queues: rx goes from 50 Gbps to 92.8 Gbps (line rate).
- 200Gbps NIC, 10 VMs (total 160 queues): rx goes from 30 Gbps to 190 Gbps (line rate).
- 12 SSD, 96 queues: 4K random read goes from 6M to 20.5M IOPS (device max).
In all cases, with adaptive moderatrion, latency up to p95 is unaffected at low/moderate load, even if compared with no moderation at all.
Changes in v5:
- refactored the commits based on previous feedback
- various cleanups
- conditionally reverted parent IRQ mask/unmask, which would completely defeat the mechanism GSIM is based on.
Changes in v4:
- added irqdesc and irqdata flags as suggested by maintainer
- parameters are only configured via independent procfs entries. No module parameters anymore.
- merged control and interrupt functions back into a single header/C files
- applied various annotations (lockdep, data_race())
- formatting and various renaming as suggested by maintainer.
- added performance measurements with adaptive moderation.
- removed the mechanism to conditionally enable moderation at interrupt creation. This can be done in userspace and suitable udev extensions will be handled separately.
Changes in v3:
- clearly documented architecture in kernel/irq/irq_moderation.c including how to handle enable/disable/mask, interrupt migration, hotplug and suspend.
- split implementation in 4 files irq_moderation.[ch] and irq_moderation_hook.[ch] for better separation of control plane and "dataplane" (functions ran on each interrupt)
- limited scope to handle_edge_irq() and handle_fasteoi_irq() which have been tested on actual hardware.
- tested on Intel (also with intremap=posted_msi), AMD, ARM, with NIC, nvme, vfio
Changes in v2:
- many style fixes (mostly on comments) based on reviewers' comments on v1
- removed background from Documentation/core-api/irq/irq-moderation.rst
- split procfs handlers
- moved internal details to kernel/irq/irq_moderation.h
- use cpu hotplug for per-CPU setup, removed unnecessary arch-specific changes
- select suitable irqs based on !irqd_is_level_type(irqd) && irqd_is_single_target(irqd)
- use a static_key to enable/disable the feature
Luigi Rizzo (7):
genirq: Add flags for software interrupt moderation.
genirq: Add GSIM infrastructure
genirq: Implement core GSIM moderation logic
genirq: Integrate GSIM into interrupt flow
genirq: Add GSIM user space configuration (procfs)
genirq: Adaptive Global Software Interrupt Moderation (GSIM). PCI/MSI: re-enable conditional parent mask/unmask with sw moderation
drivers/irqchip/irq-msi-lib.c | 8 + drivers/pci/msi/irqdomain.c | 20 + include/linux/irq.h | 11 +- include/linux/irqdesc.h | 12 + kernel/irq/Kconfig | 11 + kernel/irq/Makefile | 1 + kernel/irq/chip.c | 14 + kernel/irq/debugfs.c | 3 + kernel/irq/internals.h | 20 + kernel/irq/irq_moderation.c | 959 ++++++++++++++++++++++++++++++++++ kernel/irq/irq_moderation.h | 156 ++++++ kernel/irq/irqdesc.c | 1 + kernel/irq/manage.c | 10 + kernel/irq/proc.c | 2 + kernel/irq/settings.h | 17 + 15 files changed, 1244 insertions(+), 1 deletion(-) create mode 100644 kernel/irq/irq_moderation.c create mode 100644 kernel/irq/irq_moderation.h
Signature · 1 line
2.55.0.737.g08866a6d13-goog
PATCH v5 1/7 genirq: Add flags for software interrupt moderation.
Add two flags to support software interrupt moderation:
- IRQ_MODERATABLE is an irqdesc flag indicating that an interrupt supports moderation. This is a capability that can be set for eligible sources.
- IRQD_MODERATED is an internal irqdata flag indicating that the interrupt is currently being moderated. This is a state flag.
--- include/linux/irq.h | 11 ++++++++++- kernel/irq/debugfs.c | 3 +++ kernel/irq/settings.h | 17 +++++++++++++++++ 3 files changed, 30 insertions(+), 1 deletion(-)
include/linux/irq.h +10 -1
@@ -76,6 +76,7 @@ enum irqchip_irq_state;* IRQ_DISABLE_UNLAZY - Disable lazy irq disable * IRQ_HIDDEN - Don't show up in /proc/interrupts * IRQ_NO_DEBUG - Exclude from note_interrupt() debugging +* IRQ_MODERATABLE - Can do software interrupt moderation */ enum {IRQ_TYPE_NONE =0x00000000 ,
@@ -104,13 +105,14 @@ enum {IRQ_HIDDEN = (1 <<20 ),IRQ_NO_DEBUG = (1 <<21 ),IRQ_RESERVED = (1 <<22 ), + IRQ_MODERATABLE = (1 << 23), };#define IRQF_MODIFY_MASK \ (IRQ_TYPE_SENSE_MASK | IRQ_NOPROBE | IRQ_NOREQUEST | \ IRQ_NOAUTOEN | IRQ_LEVEL | IRQ_NO_BALANCING | \ IRQ_PER_CPU | IRQ_NESTED_THREAD | IRQ_NOTHREAD | IRQ_PER_CPU_DEVID | \ - IRQ_IS_POLLED | IRQ_DISABLE_UNLAZY | IRQ_HIDDEN) + IRQ_IS_POLLED | IRQ_DISABLE_UNLAZY | IRQ_HIDDEN | IRQ_MODERATABLE)#define IRQ_NO_BALANCING_MASK (IRQ_PER_CPU | IRQ_NO_BALANCING)
@@ -224,6 +226,7 @@ struct irq_data {* irqchip have flag IRQCHIP_ENABLE_WAKEUP_ON_SUSPEND set. * IRQD_RESEND_WHEN_IN_PROGRESS - Interrupt may fire when already in progress in which * case it must be resent at the next available opportunity. +* IRQD_MODERATED - Interrupt is currently moderated. */ enum {IRQD_TRIGGER_MASK =0xf ,
@@ -249,6 +252,7 @@ enum {IRQD_AFFINITY_ON_ACTIVATE =BIT (28 ),IRQD_IRQ_ENABLED_ON_SUSPEND =BIT (29 ),IRQD_RESEND_WHEN_IN_PROGRESS =BIT (30 ), + IRQD_MODERATED = BIT(31), };#define __irqd_to_state (d) ACCESS_PRIVATE((d)->common, state_use_accessors)
@@ -438,6 +442,11 @@ static inline bool irqd_needs_resend_when_in_progress(struct irq_data *d)return __irqd_to_state (d) &IRQD_RESEND_WHEN_IN_PROGRESS ; } +static inline bool irqd_is_moderated (struct irq_data *d) +{ +return __irqd_to_state (d) &IRQD_MODERATED ; +} +#undef __irqd_to_statestatic inline irq_hw_number_t irqd_to_hwirq (struct irq_data *d)
kernel/irq/debugfs.c +3 -0
@@ -128,6 +128,8 @@ static const struct irq_bit_descr irqdata_states[] = {BIT_MASK_DESCR (IRQD_IRQ_ENABLED_ON_SUSPEND ),BIT_MASK_DESCR (IRQD_RESEND_WHEN_IN_PROGRESS ), + + BIT_MASK_DESCR(IRQD_MODERATED), };static const struct irq_bit_descr irqdesc_states[] = {
@@ -140,6 +142,7 @@ static const struct irq_bit_descr irqdesc_states[] = {BIT_MASK_DESCR (_IRQ_IS_POLLED),BIT_MASK_DESCR (_IRQ_DISABLE_UNLAZY ),BIT_MASK_DESCR (_IRQ_HIDDEN ), + BIT_MASK_DESCR(_IRQ_MODERATABLE), };static const struct irq_bit_descr irqdesc_istates[] = {
kernel/irq/settings.h +17 -0
@@ -19,6 +19,7 @@ enum { _IRQ_HIDDEN =IRQ_HIDDEN , _IRQ_NO_DEBUG =IRQ_NO_DEBUG , _IRQ_PROC_VALID =IRQ_RESERVED , + _IRQ_MODERATABLE =IRQ_MODERATABLE , _IRQF_MODIFY_MASK = IRQF_MODIFY_MASK, };
@@ -36,6 +37,7 @@ enum {#define IRQ_HIDDEN GOT_YOU_MORON#define IRQ_NO_DEBUG GOT_YOU_MORON#define IRQ_RESERVED GOT_YOU_MORON +#define IRQ_MODERATABLE GOT_YOU_MORON#undef IRQF_MODIFY_MASK#define IRQF_MODIFY_MASK GOT_YOU_MORON
@@ -193,3 +195,18 @@ static inline void irq_settings_update_proc_valid(struct irq_desc *desc, u32 set desc->status_use_accessors &= ~_IRQ_PROC_VALID; desc->status_use_accessors |= (set & _IRQ_PROC_VALID); } + +static inline bool irq_settings_moderatable (struct irq_desc *desc) +{ +return desc->status_use_accessors & _IRQ_MODERATABLE; +} + +static inline void irq_settings_set_moderatable (struct irq_desc *desc) +{ + desc->status_use_accessors |= _IRQ_MODERATABLE; +} + +static inline void irq_settings_clr_moderatable (struct irq_desc *desc) +{ + desc->status_use_accessors &= ~_IRQ_MODERATABLE; +}
Signature · 1 line
2.55.0.737.g08866a6d13-goog
[email protected]
AI reviewRe: [PATCH v5 1/7] genirq: Add flags for software interrupt moderation.
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines
Add two flags to support software interrupt moderation:
-
IRQ_MODERATABLEis an irqdesc flag indicating that an interrupt supports moderation. This is a capability that can be set for eligible sources.-
IRQD_MODERATEDis an internal irqdata flag indicating that the interrupt is currently being moderated. This is a state flag.Signed-off-by Luigi Rizzo <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
PATCH v5 2/7 genirq: Add GSIM infrastructure
In reply to Luigi RizzoIntroduce the Kconfig option CONFIG_IRQ_SW_MODERATION and GSIM fields in struct irq_desc to support Global Software Interrupt Moderation (GSIM).
Note: This commit only introduces the configuration option and basic data structures. The core moderation logic, procfs interface, and integration into the interrupt flow are implemented in subsequent commits. Enabling this option at this stage is a build-safe no-op.
--- include/linux/irqdesc.h | 12 ++++++++++++ kernel/irq/Kconfig | 11 +++++++++++ kernel/irq/internals.h | 8 ++++++++ kernel/irq/irqdesc.c | 1 + 4 files changed, 32 insertions(+)
include/linux/irqdesc.h +12 -0
@@ -18,6 +18,16 @@ struct irq_desc;struct irq_domain ;struct pt_regs ; +/** +* struct irq_desc_swmod - software interrupt moderation state +* @swmod_node: list head for per-CPU list of moderated irq_desc +*/ +struct irq_desc_swmod { +#ifdef CONFIG_IRQ_SW_MODERATION +struct list_head swmod_node; +#endif +}; +/** * struct irqstat - interrupt statistics * @cnt: real-time interrupt count
@@ -59,6 +69,7 @@ struct irq_redirect {* @threads_handled_last: comparator field for deferred spurious detection of threaded handlers * @lock: locking for SMP * @redirect: Facility for redirecting interrupts via irq_work +* @swmod_state: software interrupt moderation state * @affinity_hint: hint to user space for preferred irq affinity * @affinity_notify: context for notification of affinity changes * @pending_mask: pending rebalanced interrupts
@@ -95,6 +106,7 @@ struct irq_desc {atomic_t threads_handled;int threads_handled_last;raw_spinlock_t lock; +struct irq_desc_swmod swmod_state;struct cpumask *percpu_enabled;#ifdef CONFIG_SMP struct irq_redirect redirect;
kernel/irq/Kconfig +11 -0
@@ -150,6 +150,17 @@ config IRQ_KUNIT_TEST If unsure, say N. +config IRQ_SW_MODERATION + bool "Enable Global Software Interrupt Moderation" + depends on PROC_FS + help + Enable Global Software Interrupt Moderation. + Uses a local timer to delay interrupts in configurable ways + and depending on various global system load indicators + and targets. + + If you don't know what to do here, say N. + endmenu config GENERIC_IRQ_MULTI_HANDLER
kernel/irq/internals.h +8 -0
@@ -395,3 +395,11 @@ static inline struct irq_data *irqd_get_parent_data(struct irq_data *irqd)return NULL ;#endif } +#ifdef CONFIG_IRQ_SW_MODERATION +static inline void irq_moderation_init_fields (struct irq_desc *desc) +{ +INIT_LIST_HEAD (&desc->swmod_state.swmod_node); +} +#else +static inline void irq_moderation_init_fields (struct irq_desc *desc) {} +#endif
kernel/irq/irqdesc.c +1 -0
@@ -120,6 +120,7 @@ static inline void free_masks(struct irq_desc *desc) { }static void desc_set_defaults (unsigned int irq,struct irq_desc *desc,int node,const struct cpumask *affinity,struct module *owner) { +irq_moderation_init_fields (desc); desc->irq_common_data.handler_data =NULL ; desc->irq_common_data.msi_desc =NULL ;
Signature · 1 line
2.55.0.737.g08866a6d13-goog
[email protected]
AI reviewRe: [PATCH v5 2/7] genirq: Add GSIM infrastructure
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines
Introduce the
KconfigoptionCONFIG_IRQ_SW_MODERATIONand GSIM fields in structirq_descto support Global Software Interrupt Moderation (GSIM).Note: This commit only introduces the configuration option and basic data structures. The core moderation logic, procfs interface, and integration into the interrupt flow are implemented in subsequent commits. Enabling this option at this stage is a build-safe no-op.
Signed-off-by Luigi Rizzo <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
PATCH v5 3/7 genirq: Implement core GSIM moderation logic
In reply to Luigi RizzoIntroduce the core GSIM moderation logic, including full documentation of the architecture, per-CPU state tracking, basic list-draining functions and hooks for the interrupt handlers.
Note: The architecture documentation describes how GSIM integrates with the interrupt flow handlers (e.g. handle_edge_irq, handle_fasteoi_irq). These handlers are modified and GSIM is integrated in subsequent commits.
--- kernel/irq/Makefile | 1 + kernel/irq/irq_moderation.c | 296 ++++++++++++++++++++++++++++++++++++ kernel/irq/irq_moderation.h | 93 +++++++++++ 3 files changed, 390 insertions(+) create mode 100644 kernel/irq/irq_moderation.c create mode 100644 kernel/irq/irq_moderation.h
kernel/irq/Makefile +1 -0
@@ -5,6 +5,7 @@ obj-$(CONFIG_GENERIC_IRQ_CHIP) += generic-chip.o obj-$(CONFIG_GENERIC_IRQ_PROBE) += autoprobe.o obj-$(CONFIG_IRQ_DOMAIN) += irqdomain.o obj-$(CONFIG_IRQ_SIM) += irq_sim.o +obj-$(CONFIG_IRQ_SW_MODERATION) += irq_moderation.o obj-$(CONFIG_PROC_FS) += proc.o obj-$(CONFIG_GENERIC_PENDING_IRQ) += migration.o obj-$(CONFIG_GENERIC_IRQ_MIGRATION) += cpuhotplug.o
kernel/irq/irq_moderation.c new file +296 -0
296 lines not on this page · raw message
kernel/irq/irq_moderation.h new file +93 -0
93 lines not on this page · raw message
Signature · 1 line
2.55.0.737.g08866a6d13-goog
[email protected]
AI reviewRe: [PATCH v5 3/7] genirq: Implement core GSIM moderation logic
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines
Introduce the core GSIM moderation logic, including full documentation of the architecture, per-CPU state tracking, basic list-draining functions and hooks for the interrupt handlers.
Note: The architecture documentation describes how GSIM integrates with the interrupt flow handlers (e.g.
handle_edge_irq,handle_fasteoi_irq). These handlers are modified and GSIM is integrated in subsequent commits.Signed-off-by Luigi Rizzo <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
PATCH v5 4/7 genirq: Integrate GSIM into interrupt flow
In reply to Luigi RizzoHook GSIM into edge/fasteoi handlers and teardown.
Specifically:
- Hook GSIM into handle_edge_irq() and handle_fasteoi_irq().
- Introduce irq_moderation_allow(desc, allow) to validate if an IRQ is eligible for GSIM (non-oneshot, edge/fasteoi, single target, no bus lock) and set/clear the IRQ_MODERATABLE flag.
- Integrate GSIM setup in __setup_irq() using irq_moderation_allow(), defaulting to disabled (use_moderation = false).
- Clear GSIM state in __free_irq() and __cleanup_nmi().
Note: At this stage, GSIM is disabled by default (use_moderation = false) and cannot be enabled at runtime yet. Subsequent commits will introduce procfs configuration and a configurable default mode. For testing this commit, use_moderation can be manually set to true in the code.
--- kernel/irq/chip.c | 14 +++++++++++++ kernel/irq/internals.h | 8 +++++++ kernel/irq/irq_moderation.c | 42 +++++++++++++++++++++++++++++++++++++ kernel/irq/manage.c | 10 +++++++++ 4 files changed, 74 insertions(+)
kernel/irq/chip.c +14 -0
@@ -20,6 +20,7 @@#include <trace/events/irq.h> #include "internals.h" +#include "irq_moderation.h" static irqreturn_t bad_chained_irq (int irq,void *dev_id) {
@@ -500,6 +501,10 @@ static bool irq_can_handle_pm(struct irq_desc *desc)return false; } +/* Moderated interrupts have IRQD_IRQ_INPROGRESS and need early return. */ +if (irqd_is_moderated (irqd)) +return false; +/* Check whether the interrupt is polled on another CPU */ if (unlikely (desc->istate &IRQS_POLL_INPROGRESS )) { if (WARN_ONCE(irq_poll_cpu == smp_processor_id(),
@@ -749,6 +754,10 @@ void handle_fasteoi_irq(struct irq_desc *desc)* handling the previous one - it may need to be resent. */ if (!irq_can_handle_pm (desc)) { +if (irqd_is_moderated (&desc->irq_data)) { + desc->istate |=IRQS_PENDING ; +mask_irq (desc); + }if (irqd_needs_resend_when_in_progress (&desc->irq_data)) desc->istate |=IRQS_PENDING ;cond_eoi_irq (chip, &desc->irq_data);
@@ -769,6 +778,9 @@ void handle_fasteoi_irq(struct irq_desc *desc)cond_unmask_eoi_irq (desc, chip); +if (irq_start_moderation (desc)) +return ; +/* * When the race described above happens this will resend the interrupt. */
@@ -858,6 +870,8 @@ void handle_edge_irq(struct irq_desc *desc)handle_irq_event (desc); +if (irq_start_moderation (desc)) +break ; }while ((desc->istate &IRQS_PENDING ) && !irqd_irq_disabled (&desc->irq_data)); }EXPORT_SYMBOL (handle_edge_irq);
kernel/irq/internals.h +8 -0
@@ -400,6 +400,14 @@ static inline void irq_moderation_init_fields(struct irq_desc *desc) {INIT_LIST_HEAD (&desc->swmod_state.swmod_node); } + +int irq_moderation_allow (struct irq_desc *desc,bool allow); +bool irq_moderation_supported (struct irq_desc *desc);#else static inline void irq_moderation_init_fields (struct irq_desc *desc) {} +static inline int irq_moderation_allow (struct irq_desc *desc,bool allow) +{ +return allow ? -EOPNOTSUPP :0 ; +} +static inline bool irq_moderation_supported (struct irq_desc *desc) {return false; }#endif
kernel/irq/irq_moderation.c +42 -0
45 lines not on this page · raw message
kernel/irq/manage.c +10 -0
@@ -1470,6 +1470,12 @@ static bool valid_percpu_irqaction(struct irqaction *old, struct irqaction *new)static int __setup_irq (unsigned int irq,struct irq_desc *desc,struct irqaction *new) { +/* +* Choose the default moderation mode. +* Hardcoded to false for now; configurable defaults are added later. +* Set to true here to force-enable GSIM for testing this commit. +*/ +bool use_moderation = false;struct irqaction *old, **old_ptr;unsigned long flags, thread_mask =0 ;int ret, nested, shared =0 ;
@@ -1789,6 +1795,8 @@ __setup_irq(unsigned int irq, struct irq_desc *desc, struct irqaction *new)irq_pm_install_action (desc, new); +irq_moderation_allow (desc, use_moderation); +/* Reset broken irq detection when installing new handler */ desc->irq_count =0 ; desc->irqs_unhandled =0 ;
@@ -1897,6 +1905,7 @@ static struct irqaction *__free_irq(struct irq_desc *desc, void *dev_id)/* If this was the last handler, shut down the IRQ line: */ if (!desc->action) {irq_settings_clr_disable_unlazy (desc); +irq_settings_clr_moderatable (desc);/* Only shutdown. Deactivate after synchronize_hardirq() */ irq_shutdown (desc); }
@@ -2046,6 +2055,7 @@ static const void *__cleanup_nmi(unsigned int irq, struct irq_desc *desc) desc->action =NULL ;irq_settings_clr_disable_unlazy (desc); +irq_settings_clr_moderatable (desc);irq_shutdown_and_deactivate (desc); }
Signature · 1 line
2.55.0.737.g08866a6d13-goog
[email protected]
AI reviewRe: [PATCH v5 4/7] genirq: Integrate GSIM into interrupt flow
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Overloading IRQD_IRQ_INPROGRESS for GSIM breaks __synchronize_hardirq() by introducing prolonged…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Overloading IRQD_IRQ_INPROGRESS for GSIM breaks __synchronize_hardirq() by introducing prolonged busy-waiting and potential deadlocks. --
--- Patch [4]: [PATCH v5 4/7] genirq: Integrate GSIM into interrupt flow --- commit 8749b57516c52ba231f3249d6f52173005693e96
Author: Luigi Rizzo <[email protected]>
Subject: genirq: Integrate GSIM into interrupt flow
This commit integrates Generic Software Interrupt Moderation (GSIM) into the handle_edge_irq() and handle_fasteoi_irq() flows. It introduces irq_moderation_allow() to validate IRQ eligibility, defaults moderation to disabled during setup, and ensures the state is cleared during IRQ teardown.
Quoted 4 lines
kernel/irq/chip.c +0
-0
[ ... ]
kernel/irq/chip.c static bool irq_can_handle_pm(struct irq_desc *desc) · line 507
Quoted 8 lines
kernel/irq/chip.c +4
-0@@ -500,6 +501,10 @@ static bool irq_can_handle_pm(struct irq_desc *desc)return false; } +/* Moderated interrupts have IRQD_IRQ_INPROGRESS and need early return. */ +if (irqd_is_moderated (irqd)) +return false; +
[Severity: High]
The comment mentions that moderated interrupts have IRQD_IRQ_INPROGRESS set. Could this cause a prolonged busy-wait or deadlock in __synchronize_hardirq() when a driver calls disable_irq() or free_irq() while the moderation timer is active?
__synchronize_hardirq() uses a tight loop to wait for this flag to clear:
kernel/irq/manage.c:__synchronize_hardirq() {
...
while (irqd_irq_inprogress(&desc->irq_data))
cpu_relax();
...
}If the flag remains active for the entire duration of the moderation hrtimer, any teardown or suspend/resume operation would busy-spin for that entire duration.
Additionally, if synchronize_irq() is called with local interrupts disabled on the CPU where the moderation timer is pinned, would this hard-deadlock the system since the hrtimer can never fire to clear the flag?
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
PATCH v5 5/7 genirq: Add GSIM user space configuration (procfs)
In reply to Luigi RizzoIntroduce procfs interfaces to configure and monitor GSIM at runtime.
This adds:
- A global directory /proc/irq/sw_moderation/ containing:
- delay_us: Read/write interface to set/get the maximum moderation delay in microseconds (defaults to 0, GSIM disabled).
- stats: Read-only interface to monitor GSIM statistics per-CPU.
- files /proc/irq/NN/allow_sw_moderation (where NN is the IRQ number) to individually allow/disallow moderation. Created only for interrupts that support moderation (e.g., edge-triggered, single-target, etc.).
--- kernel/irq/internals.h | 4 + kernel/irq/irq_moderation.c | 256 +++++++++++++++++++++++++++++++++++- kernel/irq/proc.c | 2 + 3 files changed, 261 insertions(+), 1 deletion(-)
kernel/irq/internals.h +4 -0
@@ -403,6 +403,8 @@ static inline void irq_moderation_init_fields(struct irq_desc *desc)int irq_moderation_allow (struct irq_desc *desc,bool allow);bool irq_moderation_supported (struct irq_desc *desc); +void irq_moderation_procfs_add (struct irq_desc *desc,umode_t umode); +void irq_moderation_procfs_remove (struct irq_desc *desc);#else static inline void irq_moderation_init_fields (struct irq_desc *desc) {}static inline int irq_moderation_allow (struct irq_desc *desc,bool allow)
@@ -410,4 +412,6 @@ static inline int irq_moderation_allow(struct irq_desc *desc, bool allow) return allow ? -EOPNOTSUPP : 0; }static inline bool irq_moderation_supported (struct irq_desc *desc) {return false; } +static inline void irq_moderation_procfs_add (struct irq_desc *desc,umode_t umode) {} +static inline void irq_moderation_procfs_remove (struct irq_desc *desc) {}#endif
kernel/irq/irq_moderation.c +255 -1
299 lines not on this page · raw message
kernel/irq/proc.c +2 -0
@@ -379,6 +379,7 @@ void register_irq_proc(unsigned int irq, struct irq_desc *desc) irq_effective_aff_list_proc_show, irqp);# endif #endif +irq_moderation_procfs_add (desc,0644 );proc_create_single_data ("spurious" ,0444 , desc->dir, irq_spurious_proc_show, (void *)(long )irq);
@@ -400,6 +401,7 @@ void unregister_irq_proc(unsigned int irq, struct irq_desc *desc)remove_proc_entry ("effective_affinity_list" , desc->dir);# endif #endif +irq_moderation_procfs_remove (desc);remove_proc_entry ("spurious" , desc->dir);snprintf (name,MAX_NAMELEN ,"%u" , irq);
Signature · 1 line
2.55.0.737.g08866a6d13-goog
[email protected]
AI reviewRe: [PATCH v5 5/7] genirq: Add GSIM user space configuration (procfs)
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Unconditional removal of conditionally created `allow_sw_moderation` proc entry triggers…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Unconditional removal of conditionally created allow_sw_moderation proc entry triggers WARN(1) on unsupported IRQs. --
--- Patch [5]: [PATCH v5 5/7] genirq: Add GSIM user space configuration (procfs) --- commit fc9564358ed0238056ccb83c0aee7d6d588c4901
Author: Luigi Rizzo <[email protected]>
genirq: Add GSIM user space configuration (procfs)
Introduce procfs interfaces to configure and monitor GSIM at runtime. This adds a global directory /proc/irq/sw_moderation/ and per-IRQ files to individually allow or disallow moderation.
Quoted 4 lines
kernel/irq/irq_moderation.c +0
-0
[ ... ]
kernel/irq/irq_moderation.c bool irq_moderation_do_start(struct irq_desc *desc, struct irq_mod_state *m) · line 337
Quoted 11 lines
kernel/irq/irq_moderation.c +11
-0+void irq_moderation_procfs_add (struct irq_desc *desc,umode_t umode) +{ +if (!irq_moderation_supported (desc)) +return ; +proc_create_data ("allow_sw_moderation" , umode, desc->dir, + &allow_flag_ops, (void *)(long )desc->irq_data.irq); +} + +void irq_moderation_procfs_remove (struct irq_desc *desc) +{ +remove_proc_entry ("allow_sw_moderation" , desc->dir);
[Severity: Medium]
Will this unconditional call to remove_proc_entry() trigger a kernel warning when unregistering an unsupported IRQ?
In irq_moderation_procfs_add() just above, the "allow_sw_moderation" procfs entry is only created if irq_moderation_supported(desc) evaluates to true.
Because irq_moderation_procfs_remove() lacks the same check, freeing a dynamic IRQ that doesn't support moderation will cause remove_proc_entry() to print a warning stack trace about a missing entry.
Could we add the same !irq_moderation_supported(desc) check here to avoid the warning?
Quoted 1 line
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
PATCH v5 6/7 genirq: Adaptive Global Software Interrupt Moderation (GSIM).
In reply to Luigi RizzoGSIM helps keeping system load (as defined later) under control, but the use of a fixed moderation delay causes unnecessary latency when the system load is already low.
We focus on two types of system load related to interrupts:
- total device interrupt rates. Some platforms show severe I/O performance degradation with more than 1-2 Mintr/s across the entire system. This affects especially server-class hardware with hundreds of interrupt sources (NIC or SSD queues from many physical or virtualized devices).
- percentage of time spent in hardirq. This can become problematic in presence of large numbers of interrupt sources hitting individual CPUs (for instance, as a result of attempts to isolate application CPUs from interrupt load).
Make GSIM adaptive by measuring total and per-CPU interrupt rates, as well as time spent in hardirq on each CPU. The metrics are compared to configurable targets, and each CPU adjusts the moderation delay up or down depending on the result, using multiplicative increase/decrease.
Configuration of moderation parameters is done via procfs
echo ${VALUE} > /proc/irq/sw_moderation/${NAME}Parameters are:
delay_us (0 off, range 0-500)
Maximum moderation delay in microseconds.
target_intr_rate (0 off, range 0-50000000)
Total rate above which moderation should trigger.
hardirq_percent (0 off,range 0-100)
Percentage of time spent in hardirq above which moderation should trigger.
update_ms (range 1-100, default 5)
How often the metrics should be computed and moderation delay updated.
When target_intr_rate and hardirq_percent are both 0, GSIM uses delay_us as fixed moderation delay. Otherwise, the delay is dynamically adjusted up or down, independently on each CPU, based on how the total and per-CPU metrics compare with the targets.
Provided that delay_us suffices to bring the metrics within the target, the control loop will dynamically converge to the minimum actual moderation delay to stay within the target.
PERFORMANCE BENEFITS:
The tests below demonstrate how adaptive moderation allows improved throughput at high load (same as fixed moderation) and low latency ad moderate load (same as no moderation) without having to hand-tune the system based on load.
We run experiments on one x86 platform with 8 SSD (64 queues) capable of an aggregate of approximately 14M IOPS running fio with variable number of SSD devices (1 or 8), threads per disk (1 or 200), and IODEPTH (from 1 to 128 per thread) the actual command is below:
${FIO} --name=read_IOPs_test ${DEVICES} --iodepth=${IODEPTH} --numjobs=${JOBS} \
--bs=4K --rw=randread --filesize=1000G --ioengine=libaio --direct=1 \
--verify=0 --randrepeat=0 --time_based=1 --runtime=600s \
--cpus-allowed=1-119 --cpus_allowed_policy=split --group_reporting=1For each configuration we test three moderations settings:
- OFF: delay_us=0
- FIXED: delay_us=200 target_intr_rate=0 hardirq_percent=0
- ADAPTIVE: delay_us=200 target_intr_rate=1000000 hardirq_percent=70
The first set of measurements is for ONE DISK, ONE THREAD. At low IODEPTH the throughput is latency bound, and moderation is not necessary. A fixed moderation delay dominates the latency hence reducing throughput; adaptive moderation avoids the problem.
As the IODEPTH increases, the system becomes I/O bound, and even the fixed moderation delay does not harm.
Overall: adaptive moderation is better than fixed moderation and at least as good as moderation off.
------ OFF -------- ------ FIXED ------ ----- ADAPTIVE ----
IODEPTH IOPS p50 p90 p99 IOPS p50 p90 p99 IOPS p50 p90 p99
------------------- ------------------- -------------------
1: 12K 78 88 94 . 5K 208 210 215 . 12K 78 88 96
8: 94K 83 91 110 . 38K 210 212 221 . 94K 83 91 105
32: 423K 72 85 124 . 150K 210 219 235 . 424K 72 85 124
128: 698K 180 200 243 . 513K 251 306 347 . 718K 174 194 239A second set of measurements is with one disk and 200 threads. The system is I/O bound but without significant interrupt overhead. All three scenarios are basically equivalent.
--------- OFF -------- ------- FIXED ------- ------ ADAPTIVE -----
IODEPTH IOPS p50 p90 p99 IOPS p50 p90 p99 IOPS p50 p90 p99
---------------------- --------------------- ---------------------
1: 1581K 110 174 281 . 933K 208 223 363 . 1556K 114 176 277
8: 1768K 889 1516 1926 . 1768K 848 1516 2147 . 1768K 889 1516 1942
32: 1768K 3589 5735 7701 . 1768K 3589 5735 7635 . 1768K 3589 5735 7504
128: 1768K 14ms 24ms 31ms . 1768K 14ms 24ms 29ms . 1768K 14ms 24ms 30msFinally, we have one set of measurements with 8 disks and 200 threads per disk, all running on socket 0. The system would be I/O bound (and CPU/latency bound at low IODEPTH), but this platform is unable to cope with the high global interrupt rate and so moderation is really necessary to hit the disk limits.
As we see below, adaptive moderation gives more than 2X higher throughput at meaningful iodepths, and even latency is much better. The only case where we see a small regression is with iodepth=1, because the high interrupt rate triggers the control loop to increase the moderation delay.
--------- OFF -------- -------- FIXED ------- ------ ADAPTIVE ------
IODEPTH IOPS p50 p90 p99 IOPS p50 p90 p99 IOPS p50 p90 p99
---------------------- ---------------------- ----------------------
1: 2304K 82 94 128 . 1030K 208 277 293 . 1842K 97 149 188
8: 5240K 128 938 1680 . 7500K 208 233 343 . 10000K 151 210 281
32: 5251K 206 3621 3949 . 12300K 184 1106 5407 . 12100K 184 1139 5407
128: 5330K 4228 12ms 17ms . 13800K 1123 4883 7373 . 13800K 1074 4883 7635Finally, here are experiments indicating how throughput is affected by the various parameters (with 8 disks and 200 threads).
IOPS vs delay_us (target_intr_rate = 0, hardirq_percent=0)
delay_us 0 50 100 150 200 250
IODEPTH 1 2300 1860 1580 1300 1034 1066
IODEPTH 8 5254 9936 9645 8818 7500 6150
IODEPTH 32 5250 11300 13800 13800 13800 13800
IODEPTH 128 5900 13600 13900 13900 13900 13900
IOPS vs target_intr_rate (delay_us = 200, hardirq_percent=0, iodepth 128)
value 250K 500K 750K 1000K 1250K 1500K 1750K 2000K
socket0 13900 13900 13900 13800 13800 12900 11000 8808
both sockets 13900 13900 13900 13800 8600 8000 6900 6400
hardirq_percent (delay_us = 200, target_intr_rate=0, iodepth 128)
hardirq% 1 10 20 30 40 50 60 70
KIOPS 13900 13800 13300 12100 10400 8600 7400 6500
--- kernel/irq/irq_moderation.c | 391 ++++++++++++++++++++++++++++++++++-- kernel/irq/irq_moderation.h | 63 ++++++ 2 files changed, 442 insertions(+), 12 deletions(-)
kernel/irq/irq_moderation.c +379 -12
@@ -9,6 +9,9 @@#include <linux/cpuhotplug.h> #include <linux/irq.h> #include <linux/irqdesc.h> +#include <linux/kallsyms.h> +#include <linux/kernel_stat.h> +#include <linux/math64.h> #include <linux/mutex.h> #include <linux/notifier.h> #include <linux/proc_fs.h>
@@ -23,8 +26,9 @@* * Some platforms show reduced I/O performance when the total device interrupt * rate across the entire platform becomes too high. To address the problem, -* GSIM runs after the handler to implement software interrupt moderation -* with programmable delay. +* GSIM runs after the handler to measure global and per-CPU interrupt rates, +* compares them with configurable targets, and implements independent, per-CPU +* software moderation delays. * * Configuration is done at runtime via procfs * echo ${VALUE} > /proc/irq/sw_moderation/${NAME}
@@ -35,6 +39,19 @@* Maximum moderation delay. A reasonable range is 20-100. Higher values * can be useful if the hardirq handler has long runtimes. * +* target_intr_rate (default 0, suggested 1000000, 0 off, range 0-50000000) +* The total interrupt rate above which moderation kicks in. +* Not particularly critical, a value in the 500K-1M range is usually ok. +* +* hardirq_percent (default 0, suggested 70, 0 off, range 0-100) +* The hardirq percentage above which moderation kicks in. +* 50-90 is a reasonable range. +* +* FIXED MODERATION mode requires target_intr_rate=0, hardirq_percent=0 +* +* update_ms (default 5, range 1-100) +* How often the load is measured and moderation delay updated. +* * Moderation is allowed/disallowed dynamically for individual interrupts with * echo 1 > /proc/irq/NN/allow_sw_moderation # use 0 to disallow *
@@ -112,7 +129,47 @@* GSIM parameters. Initialize delay_ns here to statically enable moderation * (e.g. .delay_ns = 100000). */ -struct irq_mod_params irq_mod_params____cacheline_aligned ; + +/* +* Recommended values for the adaptive control loop. +* +* update_ns is documented earlier and can be modified via procfs +* +* The following two allow fine tuning of the control loop and should not +* be modified unless there is good understanding of their impact on the +* stability of the controller. +* +* scale_cpus (default 150, range 50-1000) +* Small update_ms may lead to underestimate the number of CPUs +* simultaneously handling interrupts, and the opposite can happen +* with very large values. This parameter may help correct the value, +* though it is not recommended to modify the default unless there are +* very strong reasons. +* +* increase_divisor (default 8, range 8-128) +* This is base parameter used for multiplicative increase/decrease. +*/ + +#define MIN_SCALING_DIVISOR 8 + +struct irq_mod_params irq_mod_params____cacheline_aligned = { + .update_ns =5 *NSEC_PER_MSEC , + .scale_cpus =150 , + .increase_divisor =MIN_SCALING_DIVISOR , + .seq =SEQCNT_ZERO (irq_mod_params.seq), +}; + +/* +* Accumulator for total interrupt and active CPUs, updated by all active +* CPUs on each epoch (update_ns or more). +* @total_intrs: running count of total interrupts +* @total_cpus: running count of total active CPUs +*/ +struct irq_mod_counters { +atomic_t total_intrs; +atomic_t total_cpus; +}; +static struct irq_mod_counters irq_mod_counters; DEFINE_PER_CPU_ALIGNED(struct irq_mod_state, irq_mod_state);
@@ -122,12 +179,215 @@ static DEFINE_MUTEX(swmod_mutex);static void update_enable_key (void ) { +lockdep_assert_held (&swmod_mutex); +if (irq_mod_params.delay_ns !=0 )static_branch_enable (&irq_moderation_enabled_key);else static_branch_disable (&irq_moderation_enabled_key); } +/* Slow path functions for interrupt moderation. */ + +/* +* Compute smoothed average between old and cur. 'steps' is used +* to approximate applying the smoothing multiple times. +*/ +static inline unsigned int smooth_avg (unsigned int old,unsigned int cur,unsigned int steps) +{ +const unsigned int smooth_factor =64 ; +u64 sum; + + steps =min (steps, smooth_factor -1 ); + sum = (u64)(smooth_factor - steps) * old + (u64 )steps * cur; +return div_u64 (sum, smooth_factor); +} + +/* Measure and assess time spent in hardirq. */ +static inline bool hardirq_high (struct irq_mod_state *m,unsigned int hardirq_percent, +u64 epoch_ns) +{ +bool above_threshold; +u64 irqtime, cur; + +if (!IS_ENABLED (CONFIG_IRQ_TIME_ACCOUNTING )) +return false; + + cur = kcpustat_this_cpu->cpustat[CPUTIME_IRQ ]; + irqtime = cur - m->last_irqtime; + m->last_irqtime = cur; + +if (hardirq_percent ==0 ) +return false; + + above_threshold = irqtime *100 > epoch_ns * hardirq_percent; + m->hardirq_high += above_threshold; +return above_threshold; +} + +/* Measure and assess total and per-CPU interrupt rates. */ +static inline bool irqrate_high (struct irq_mod_state *m,unsigned int target_rate, +unsigned int steps,u64 epoch_ns, +unsigned int update_ns,unsigned int scale_cpus) +{ +unsigned int global_intr_rate, local_intr_rate, delta_intrs, tmp; +bool local_rate_high, global_rate_high; +u64 num_local, num_global, num_cpus; +/* Use unsigned long to avoid overflow in intermediate results. */ +unsigned long active_cpus; +u32 denom = epoch_ns; +int shift =0 ; + + num_local = (u64 )m->intr_count *NSEC_PER_SEC ; +/* Scale denominator so we can avoid 64-bit division. */ +if (unlikely (epoch_ns >U32_MAX )) { + shift =fls64 (epoch_ns) -32 ; + denom = epoch_ns >> shift; + num_local >>= shift; + } + + local_intr_rate =div_u64 (num_local, denom); + +/* Accumulate global counter and compute global interrupt rate. */ + tmp =atomic_add_return (m->intr_count, &irq_mod_counters.total_intrs); + m->intr_count =0 ; + delta_intrs = tmp - m->last_total_intrs; + m->last_total_intrs = tmp; + num_global = (u64 )delta_intrs *NSEC_PER_SEC ; +if (unlikely (shift)) + num_global >>= shift; + global_intr_rate =div_u64 (num_global, denom); + +/* +* Count how many CPUs handled interrupts in the last epoch, needed +* to determine the per-CPU target (target_rate / active_cpus). +* Each active CPU increments the global counter approximately every +* update_ns. Scale the value by (update_ns / epoch_ns) to get the +* correct value. Also apply rounding and make sure active_cpus > 0. +*/ + tmp =atomic_add_return (1 , &irq_mod_counters.total_cpus); + active_cpus = tmp - m->last_total_cpus; + m->last_total_cpus = tmp; + num_cpus = (u64 )active_cpus * update_ns + (epoch_ns /2 ); +if (unlikely (shift)) + num_cpus >>= shift; + active_cpus =div_u64 (num_cpus, denom); +if (active_cpus <1 ) + active_cpus =1 ; + +/* Compare with global and per-CPU targets. */ + global_rate_high = global_intr_rate > target_rate; + +/* +* Short epochs may lead to underestimate the number of active CPUs. +* Apply a scaling factor to compensate. This may make the controller +* a bit more aggressive but does not harm system throughput. +*/ + local_rate_high = (u64 )local_intr_rate * active_cpus * + scale_cpus > (u64 )target_rate *100 ; + +/* Statistics. */ + m->global_intr_rate =smooth_avg (m->global_intr_rate, global_intr_rate, steps); + m->local_intr_rate =smooth_avg (m->local_intr_rate, local_intr_rate, steps); + m->scaled_cpu_count =smooth_avg (m->scaled_cpu_count, active_cpus *256 , steps); + +if (target_rate ==0 ) +return false; + + m->local_irq_high += local_rate_high; + m->global_irq_high += global_rate_high; + +/* Moderate on this CPU only if both global and local rates are high. */ +return global_rate_high && local_rate_high; +} + +/* Periodic adjustment, called once per epoch. */ +void irq_moderation_update_epoch (struct irq_mod_state *m,u64 epoch_ns) +{ +unsigned int hardirq_percent, target_rate, delay_ns, update_ns; +unsigned int increase_divisor, scale_cpus; +const unsigned int min_delay_ns =500 ; +bool above_target = false; +unsigned int steps, seq; + +do { + seq =read_seqcount_begin (&irq_mod_params.seq); + hardirq_percent =READ_ONCE (irq_mod_params.hardirq_percent); + target_rate =READ_ONCE (irq_mod_params.target_intr_rate); + delay_ns =READ_ONCE (irq_mod_params.delay_ns); + update_ns =READ_ONCE (irq_mod_params.update_ns); + increase_divisor =READ_ONCE (irq_mod_params.increase_divisor); + scale_cpus =READ_ONCE (irq_mod_params.scale_cpus); + }while (read_seqcount_retry (&irq_mod_params.seq, seq)); + +/* +* If one parameter changes, set the moderation delay to max, and rely +* on the adaptive mechanism to adjust it down if necessary. +* Otherwise the system may be stuck with an interrupt rate that is +* already below the threshold because of bus congestion (one of the +* problems that GSIM is trying to address), and the controller would +* have no signal react. Starting from a high value gives it a chance +* to converge if parameters allow it. +*/ +if (seq != m->seq) { + m->seq = seq; + m->mod_ns = delay_ns; + m->intr_count =0 ; + m->last_total_intrs =atomic_read (&irq_mod_counters.total_intrs); + m->last_total_cpus =atomic_read (&irq_mod_counters.total_cpus); +if (IS_ENABLED (CONFIG_IRQ_TIME_ACCOUNTING )) + m->last_irqtime = kcpustat_this_cpu->cpustat[CPUTIME_IRQ ]; +return ; + } + +if (target_rate ==0 && hardirq_percent ==0 ) { +/* Use fixed moderation delay. */ + m->mod_ns = delay_ns; + m->global_intr_rate =0 ; + m->local_intr_rate =0 ; + m->scaled_cpu_count =0 ; +return ; + } + +/* +* The controller wants to scale the delay mod_ns by (1 + 1/D) every "update_ns". +* Since we operate every epoch_ns >= update_ns, the formula becomes +* mod_ns = mod_ns * ((1 + 1/D) ** (epoch_ns / update_ns)) +* which we approximate with "mod_ns = mod_ns * (1 + steps/D)" +* where "steps = epoch_ns / update_ns" clamped to a value < D. +*/ + steps = (unsigned int )clamp_t (u64,div_u64 (epoch_ns, update_ns), +1ULL , (u64)(MIN_SCALING_DIVISOR -1u )); + +if (irqrate_high (m, target_rate, steps, epoch_ns, update_ns, scale_cpus)) + above_target = true; + +if (hardirq_high (m, hardirq_percent, epoch_ns)) + above_target = true; + +/* +* Controller: adjust delay with exponential increase or decrease. +* +* Following standard practices, we increase fast (smaller divisor) to +* aggressively slow down when the interrupt rate goes up, but decrease +* slowly (larger divisor) to reduce the chance of load spikes as the +* delay goes down. +*/ +if (above_target) { +/* Make sure the value is large enough for the exponential to grow. */ +if (m->mod_ns < min_delay_ns) + m->mod_ns = min_delay_ns; + m->mod_ns += m->mod_ns * steps / increase_divisor; +if (m->mod_ns > delay_ns) + m->mod_ns = delay_ns; + }else { + m->mod_ns -= m->mod_ns * steps / (2 * increase_divisor); +/* Round down to 0 values that are too small to bother. */ +if (m->mod_ns < min_delay_ns) + m->mod_ns =0 ; + } +} +/* Actually start moderation. */ bool irq_moderation_do_start (struct irq_desc *desc,struct irq_mod_state *m) {
@@ -138,7 +398,7 @@ bool irq_moderation_do_start(struct irq_desc *desc, struct irq_mod_state *m)const u64 slack_ns =2000 ;/* Accumulate sleep time, no moderation if too small. */ - m->sleep_ns +=READ_ONCE (irq_mod_params.delay_ns); + m->sleep_ns += m->mod_ns;if (m->sleep_ns < min_delay_ns)return false;/* We need moderation, start the timer. */
@@ -153,6 +413,11 @@ bool irq_moderation_do_start(struct irq_desc *desc, struct irq_mod_state *m)*/ m->enqueue++;list_add (&desc->swmod_state.swmod_node, &m->descs); +/* +* Set IRQD_IRQ_INPROGRESS so that synchronize_irq() called during +* free_irq() will block until the timer drains this descriptor from +* the moderation list. +*/ irqd_set (&desc->irq_data,IRQD_IRQ_INPROGRESS |IRQD_MODERATED );__disable_irq (desc);return true;
@@ -186,17 +451,31 @@ struct swmod_procfs_entry {struct var_info var; }; +static void write_param (unsigned int *ptr,unsigned int value) +{ +unsigned long flags; + +local_irq_save (flags); +write_seqcount_begin (&irq_mod_params.seq); +WRITE_ONCE (*ptr, value); +write_seqcount_end (&irq_mod_params.seq); +local_irq_restore (flags); +} +static ssize_t swmod_wr (struct var_info *v,const char __user *s,size_t count) {unsigned int value;int ret; +lockdep_assert_held (&swmod_mutex); + ret =kstrtouint_from_user (s, count,0 , &value);if (ret)return ret;if (value < v->min || value > v->max)return -ERANGE ; -WRITE_ONCE (*v->ptr, value * v->scale); +write_param (v->ptr, value * v->scale); +return count; }
@@ -216,34 +495,87 @@ static ssize_t swmod_wr_delay(struct var_info *v, const char __user *s, size_t creturn ret; } -#define HEAD_FMT "%5s %8s %11s %11s\n" -#define BODY_FMT "%5u %8u %11u %11u\n" +#define HEAD_FMT "%5s %8s %10s %4s %8s %11s %11s %11s %11s %11s\n" +#define BODY_FMT "%5u %8u %10u %4u %8u %11u %11u %11u %11u %11u\n"/* Print statistics */ static void rd_stats (struct seq_file *p) {unsigned int delay_ns =READ_ONCE (irq_mod_params.delay_ns); -int cpu; +unsigned long global_intr_rate =0 , global_irq_high =0 ; +unsigned long local_irq_high =0 , hardirq_high =0 ; +int recent_epoch_limit, cpu, active_cpus =0 ;if (delay_ns ==0 )return ;seq_printf (p,HEAD_FMT , -"# CPU" ,"delay_ns" ,"timer_set" ,"enqueue" ); +"# CPU" ,"irq/s" ,"loc_irq/s" ,"cpus" ,"delay_ns" , +"irq_hi" ,"loc_irq_hi" ,"hardirq_hi" ,"timer_set" , +"enqueue" ); + +/* +* Accumulate/print only entries updated within ~20-30s. The high 32 bits +* of timestamps give ~4s resolution, so we can use them without the need +* for 64bit atomics (because epoch_start_ns is updated concurrently). +*/ + recent_epoch_limit = (ktime_get_ns () -20ULL *NSEC_PER_SEC ) >>32 ;for_each_possible_cpu (cpu) {/* Copy statistics, will only use some unsigned int values; races ok. */ struct irq_mod_state cur =data_race (*per_cpu_ptr (&irq_mod_state, cpu)); +if (cur.epoch_start_ns && (int )(cur.epoch_start_ns >>32 ) >= recent_epoch_limit) { +/* Recent entry, accumulate in global rate. */ + active_cpus++; + global_intr_rate += cur.global_intr_rate; + }else { +/* Stale entries, print as 0. */ + cur.global_intr_rate =0 ; + cur.local_intr_rate =0 ; + cur.scaled_cpu_count =0 ; + cur.mod_ns =0 ; + } + + global_irq_high += cur.global_irq_high; + local_irq_high += cur.local_irq_high; + hardirq_high += cur.hardirq_high; +seq_printf (p,BODY_FMT , cpu, - delay_ns, + cur.global_intr_rate, + cur.local_intr_rate, + (cur.scaled_cpu_count +128 ) /256 , + cur.mod_ns, + cur.global_irq_high, + cur.local_irq_high, + cur.hardirq_high, cur.timer_set, cur.enqueue); }seq_printf (p,"\n" -"delay_us %lu\n" , - delay_ns /NSEC_PER_USEC ); +"delay_us %lu\n" +"target_intr_rate %u\n" +"hardirq_percent %u\n" +"update_ms %ld\n" +"scale_cpus %u\n" , + delay_ns /NSEC_PER_USEC , +READ_ONCE (irq_mod_params.target_intr_rate), +READ_ONCE (irq_mod_params.hardirq_percent), +READ_ONCE (irq_mod_params.update_ns) /NSEC_PER_MSEC , +READ_ONCE (irq_mod_params.scale_cpus)); + +seq_printf (p, +"intr_rate %lu\n" +"irq_high %lu\n" +"my_irq_high %lu\n" +"hardirq_percent_high %lu\n" +"total_interrupts %u\n" +"total_cpus %u\n" , + active_cpus ? global_intr_rate / active_cpus :0 , + global_irq_high, local_irq_high, hardirq_high, +atomic_read (&irq_mod_counters.total_intrs), +atomic_read (&irq_mod_counters.total_cpus)); }static int param_show (struct seq_file *p,void *v)
@@ -476,15 +808,41 @@ static struct swmod_procfs_entry procfs_entries[] = { .rd = swmod_rd, .var = SET_VAR(&irq_mod_params.delay_ns, 0, 500, NSEC_PER_USEC), }, + { + .name = "target_intr_rate", + .wr = swmod_wr, + .rd = swmod_rd, + .var = SET_VAR(&irq_mod_params.target_intr_rate, 0, 50000000, 1), + }, + { + .name = "hardirq_percent", + .wr = swmod_wr, + .rd = swmod_rd, + .var = SET_VAR(&irq_mod_params.hardirq_percent, 0, 100, 1), + }, + { + .name = "update_ms", + .wr = swmod_wr, + .rd = swmod_rd, + .var = SET_VAR(&irq_mod_params.update_ns, 1, 100, NSEC_PER_MSEC), + }, { .name = "stats", .rd = rd_stats, }, + /* The next parameters have no procfs entries, only range validation. */ + { + .var = SET_VAR(&irq_mod_params.increase_divisor, MIN_SCALING_DIVISOR, 128, 1), + }, + { + .var = SET_VAR(&irq_mod_params.scale_cpus, 50, 1000, 1), + }, }; static int __init init_irq_moderation(void) { struct proc_dir_entry *dir; + unsigned long flags; int cpuhp_state; int i, ret;
@@ -535,8 +893,17 @@ static int __init init_irq_moderation(void) goto cleanup_1; } +/* Increment sequence counter so per-CPU m->seq (0) mismatches on epoch 1 */ +local_irq_save (flags); +write_seqcount_begin (&irq_mod_params.seq); +write_seqcount_end (&irq_mod_params.seq); +local_irq_restore (flags); +/* Enable if the defaults require it. */ +/* Acquire swmod_mutex to satisfy lockdep assertions. */ +mutex_lock (&swmod_mutex);update_enable_key (); +mutex_unlock (&swmod_mutex);return 0 ; cleanup_1:
kernel/irq/irq_moderation.h +63 -0
110 lines not on this page · raw message
Signature · 1 line
2.55.0.737.g08866a6d13-goog
[email protected]
AI reviewRe: [PATCH v5 6/7] genirq: Adaptive Global Software Interrupt Moderation (GSIM).
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 15 lines
GSIM helps keeping system load (as defined later) under control, but the use of a fixed moderation delay causes unnecessary latency when the system load is already low.
We focus on two types of system load related to interrupts:
- total device interrupt rates. Some platforms show severe I/O performance degradation with more than 1-2 Mintr/s across the entire system. This affects especially server-class hardware with hundreds of interrupt sources (NIC or SSD queues from many physical or virtualized devices).
- percentage of time spent in hardirq. This can become problematic in presence of large numbers of interrupt sources hitting individual CPUs (for instance, as a result of attempts to isolate application CPUs from interrupt load).
Quoted 6 lines
[ ... ]
hardirq_percent(delay_us= 200, target_intr_rate=0, iodepth 128)
hardirq% 1 10 20 30 40 50 60 70
KIOPS 13900 13800 13300 12100 10400 8600 7400 6500Signed-off-by Luigi Rizzo <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
PATCH v5 7/7 PCI/MSI: re-enable conditional parent mask/unmask with sw moderation
In reply to Luigi RizzoRecent commits removed masking of interrupts at the PCIe level when the masking is also done at the parent (e.g. on ARM with GIC). This optimization saves the PCIe write, and especially the expensive readback, but causes the device to still send MSI interrupts that are ignored.
As a result, it very harmful on platforms (many, for multiple architectures) that are unable to deal with too many MSI interrupts, and defeats GSIM (CONFIG_IRQ_SW_MODERATION), which is esplicitely designed to address this problem, but depends on masking interrupts at the PCIe level.
Revert to the old behavior when GSIM (CONFIG_IRQ_SW_MODERATION) is compiled in.
--- drivers/irqchip/irq-msi-lib.c | 8 ++++++++ drivers/pci/msi/irqdomain.c | 20 ++++++++++++++++++++ 2 files changed, 28 insertions(+)
drivers/irqchip/irq-msi-lib.c +8 -0
@@ -116,6 +116,14 @@ bool msi_lib_init_dev_msi_info(struct device *dev, struct irq_domain *domain,if (!chip->irq_set_affinity && !(info->flags &MSI_FLAG_NO_AFFINITY )) chip->irq_set_affinity = msi_domain_set_affinity; +/* +* The problem addressed by software moderation depends on actually +* masking at the PCI level. irq_mask() can be optimized separately +* for the common cases outside the move/teardown paths. +*/ +if (IS_ENABLED (CONFIG_IRQ_SW_MODERATION )) +return true; +/* * If the parent domain insists on being in charge of masking, obey * blindly. The interrupt is un-masked at the PCI level on startup
drivers/pci/msi/irqdomain.c +20 -0
@@ -80,6 +80,22 @@ static unsigned int cond_startup_parent(struct irq_data *data)return 0 ; } +static __always_inline void cond_mask_parent(struct irq_data *data) +{ +struct msi_domain_info *info = data->domain->host_data; + +if (unlikely (info->flags &MSI_FLAG_PCI_MSI_MASK_PARENT )) +irq_chip_mask_parent (data); +} + +static __always_inline void cond_unmask_parent(struct irq_data *data) +{ +struct msi_domain_info *info = data->domain->host_data; + +if (unlikely (info->flags &MSI_FLAG_PCI_MSI_MASK_PARENT )) +irq_chip_unmask_parent (data); +} +static void pci_irq_shutdown_msi (struct irq_data *data) {struct msi_desc *desc =irq_data_get_msi_desc (data);
@@ -102,12 +118,14 @@ static void pci_irq_mask_msi(struct irq_data *data)struct msi_desc *desc =irq_data_get_msi_desc (data);pci_msi_mask (desc,BIT (data->irq - desc->irq)); +cond_mask_parent (data); }static void pci_irq_unmask_msi (struct irq_data *data) {struct msi_desc *desc =irq_data_get_msi_desc (data); +cond_unmask_parent (data);pci_msi_unmask (desc,BIT (data->irq - desc->irq)); }
@@ -160,10 +178,12 @@ static unsigned int pci_irq_startup_msix(struct irq_data *data)static void pci_irq_mask_msix (struct irq_data *data) {pci_msix_mask (irq_data_get_msi_desc (data)); +cond_mask_parent (data); }static void pci_irq_unmask_msix (struct irq_data *data) { +cond_unmask_parent (data);pci_msix_unmask (irq_data_get_msi_desc (data)); }
Signature · 1 line
2.55.0.737.g08866a6d13-goog
[email protected]
AI reviewRe: [PATCH v5 7/7] PCI/MSI: re-enable conditional parent mask/unmask with sw moderation
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 10 lines
Recent commits removed masking of interrupts at the PCIe level when the masking is also done at the parent (e.g. on ARM with GIC). This optimization saves the PCIe write, and especially the expensive readback, but causes the device to still send MSI interrupts that are ignored.
As a result, it very harmful on platforms (many, for multiple architectures) that are unable to deal with too many MSI interrupts, and defeats GSIM (
CONFIG_IRQ_SW_MODERATION), which is esplicitely designed to address this problem, but depends on masking interrupts at the PCIe level.
Quoted 4 lines
Revert to the old behavior when GSIM (
CONFIG_IRQ_SW_MODERATION) is compiled in.Signed-off-by Luigi Rizzo <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Quoted 5 lines · Luigi Rizzo
Configuration is easy and robust. System administrators specify the maximum targets (moderation delay; interrupt rate; percentage of time spent in hardirq), and which interrupt sources should be moderated. Independent per-CPU control loops adjust actual delays to try and keep metrics within the targets.
Can we find a way to autodetect and autoenable this? A magic go faster mode that needs very specific tuning is annoying compare to sensible defaults.
Also how does this interact with adaptive polling code inside drivers like NAPI or the upcoming nvme variant?
Quoted 108 lines
The system is adaptive. Moderation affects only latency and only in high load scenarios. Throughput and CPU efficiencly generally benefits significantly. Targets don't need to match precisely the platform limits, and one can make conservative and robust choices. Values like delay_us=100, target_intr_rate=1000000, hardirq_percent=70 are a very good starting point.
GSIM does not rely on any special hardware feature.
Global parameters can be modified at runtime with
echo ${VALUE} | sudo tee /proc/irq/sw_moderation/${NAME}
/proc/irq/sw_moderation/statsexports statistics when enabled.and moderation on individual interrupts can be turned on/off at runtime with
echo 1 | sudo tee
/proc/irq/NN/allow_moderation# use 0 to disableEXAMPLE:
# global configuration: 50us, target max 1M intr/s, and 70% in intr echo 50 | sudo tee /proc/irq/sw_moderation/delay_us echo 1000000 | sudo tee /proc/irq/sw_moderation/target_intr_rate echo 70 | sudo tee /proc/irq/sw_moderation/hardirq_percent# allow moderation on all interrupts that support it # remember to periodically check and set the flag for dynamically # created interrupts since the default is 0 echo 1 | sudo tee /proc/irq/*/allow_moderation# check the status
grep -r ./proc/irq/*/*/../allow_moderation# look at statistics
less/proc/irq/sw_moderation/statsPERFORMANCE BENEFITS:
Below are some experimental results under high load comparing conventional moderation with GSIM:- 100Gbps NIC, 32 queues: rx goes from 50 Gbps to 92.8 Gbps (line rate).
- 200Gbps NIC, 10 VMs (total 160 queues): rx goes from 30 Gbps to 190 Gbps (line rate).
- 12 SSD, 96 queues: 4K random read goes from 6M to 20.5M IOPS (device max).In all cases, with adaptive moderatrion, latency up to p95 is unaffected at low/moderate load, even if compared with no moderation at all.
Changes in v5:
- refactored the commits based on previous feedback
- various cleanups
- conditionally reverted parent IRQ mask/unmask, which would completely defeat the mechanism GSIM is based on.Changes in v4:
- added irqdesc and irqdata flags as suggested by maintainer
- parameters are only configured via independent procfs entries. No module parameters anymore.
- merged control and interrupt functions back into a single header/C files
- applied various annotations (lockdep,data_race())
- formatting and various renaming as suggested by maintainer.
- added performance measurements with adaptive moderation.
- removed the mechanism to conditionally enable moderation at interrupt creation. This can be done in userspace and suitable udev extensions will be handled separately.Changes in v3:
- clearly documented architecture inkernel/irq/irq_moderation.cincluding how to handle enable/disable/mask, interrupt migration, hotplug and suspend.
- split implementation in 4 files irq_moderation.[ch] and irq_moderation_hook.[ch] for better separation of control plane and "dataplane" (functions ran on each interrupt)
- limited scope tohandle_edge_irq()andhandle_fasteoi_irq()which have been tested on actual hardware.
- tested on Intel (also with intremap=posted_msi), AMD, ARM, with NIC, nvme, vfioChanges in v2:
- many style fixes (mostly on comments) based on reviewers' comments on v1
- removed background fromDocumentation/core-api/irq/irq-moderation.rst
- split procfs handlers
- moved internal details tokernel/irq/irq_moderation.h
- use cpu hotplug for per-CPU setup, removed unnecessary arch-specific changes
- select suitable irqs based on !irqd_is_level_type(irqd) &&irqd_is_single_target(irqd)
- use astatic_keyto enable/disable the featureLuigi Rizzo (7):
genirq: Add flags for software interrupt moderation.
genirq: Add GSIM infrastructure
genirq: Implement core GSIM moderation logic
genirq: Integrate GSIM into interrupt flow
genirq: Add GSIM user space configuration (procfs)
genirq: Adaptive Global Software Interrupt Moderation (GSIM). PCI/MSI: re-enable conditional parent mask/unmask with sw moderationdrivers/irqchip/irq-msi-lib.c | 8 + drivers/pci/msi/irqdomain.c | 20 + include/linux/irq.h | 11 +- include/linux/irqdesc.h | 12 + kernel/irq/Kconfig | 11 + kernel/irq/Makefile | 1 + kernel/irq/chip.c | 14 + kernel/irq/debugfs.c | 3 + kernel/irq/internals.h | 20 + kernel/irq/irq_moderation.c | 959 ++++++++++++++++++++++++++++++++++
Quoted 13 lines
kernel/irq/irq_moderation.h | 156 ++++++ kernel/irq/irqdesc.c | 1 + kernel/irq/manage.c | 10 + kernel/irq/proc.c | 2 + kernel/irq/settings.h | 17 + 15 files changed, 1244 insertions(+), 1 deletion(-) create mode 100644 kernel/irq/irq_moderation.c create mode 100644 kernel/irq/irq_moderation.hSignature · 1 line
2.55.0.737.g08866a6d13-goog
---end quoted text---
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · Luigi Rizzo
Configuration is easy and robust. System administrators specify the maximum targets (moderation delay; interrupt rate; percentage of time spent in hardirq), and which interrupt sources should be moderated. Independent per-CPU control loops adjust actual delays to try and keep metrics within the targets.
Can we find a way to autodetect and autoenable this? A magic go faster mode that needs very specific tuning is annoying compare to sensible defaults.
As mentioned later, the tunings do not need to be specific, because the control loop adjusts the delay to stay within the target, and the target is generally just a "don't overload me" value with little impact on performance.
Of course there are many cases (eg small systems with little I/O) where moderation is not needed at all so at least one bit of user input is necessary.
What I normally do is use another small patch to pass the initial settings via module parameters, using values similar to those below
(enable on all interrupts, delay_us=100, target_intr_rate=1000000, hardirq_percent=70)
Quoted 2 lines
Also how does this interact with adaptive polling code inside drivers like NAPI or the upcoming nvme variant?
GSIM acts at hardirq level (so below NAPI, and above device moderation e.g. the one in ethtool -C ...) and is completely orthogonal to those other mechanisms.
If NAPI etc manage to keep the interrupt rate/load below the targets, then the adaptive controller in GSIM ends up using zero additional delay. Otherwise, it gently adjusts the moderation delay so the combination suffices to stay within the target.
All the above assumes the parameters are sensible.
GSIM or any HW moderation allows up to 1/delay_us interrupts per second, so setting delay_us=10 allows up to 100Kintr/s per CPU handling interrupts, so there is no way we can throttle interrupts below 100K/s)
Quoted 8 lines
Quoted 8 lines
The system is adaptive. Moderation affects only latency and only in high load scenarios. Throughput and CPU efficiencly generally benefits significantly. Targets don't need to match precisely the platform limits, and one can make conservative and robust choices. Values like delay_us=100, target_intr_rate=1000000, hardirq_percent=70 are a very good starting point.
cheers
luigi
Quoted 32 lines
From: "Luigi Rizzo"<[email protected]>
Date: Thu, Aug 20, 2026, 15:35
Subject: Re: [PATCH v5 0/7] Global Software Interrupt Moderation (GSIM)
To: "Christoph Hellwig"<[email protected]>
Cc: "Thomas Gleixner"<[email protected]>, "Marc Zyngier"<[email protected]>, "Luigi Rizzo"<[email protected]>, "Paolo Abeni"<[email protected]>, <[email protected]>, <[email protected]>, "Bjorn Helgaas"<[email protected]>, <[email protected]>, <[email protected]>, "Fengnan Chang"<[email protected]>Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · Luigi Rizzo
Configuration is easy and robust. System administrators specify the maximum targets (moderation delay; interrupt rate; percentage of time spent in hardirq), and which interrupt sources should be moderated. Independent per-CPU control loops adjust actual delays to try and keep metrics within the targets.
Can we find a way to autodetect and autoenable this? A magic go faster mode that needs very specific tuning is annoying compare to sensible defaults.
As mentioned later, the tunings do not need to be specific, because the control loop adjusts the delay to stay within the target, and the target is generally just a "don't overload me" value with little impact on performance.
Of course there are many cases (eg small systems with little I/O) where moderation is not needed at all so at least one bit of user input is necessary.
What I normally do is use another small patch to pass the initial settings via module parameters, using values similar to those below
(enable on all interrupts, delay_us=100, target_intr_rate=1000000, hardirq_percent=70)
Quoted 6 lines
Quoted 2 lines
Also how does this interact with adaptive polling code inside drivers like NAPI or the upcoming nvme variant?
GSIM acts at hardirq level (so below NAPI, and above device moderation e.g. the one in ethtool -C ...) and is completely orthogonal to those other mechanisms.
Hi Luigi:
The NVMe variant that Christoph mentioned is this patch of mine.
https://lore.kernel.org/linux-nvme/d9210bcdf73fbe1ac8b6ec132865609a3ed68688.99d43a8b.dfff.41d1.b07d.0592a12473d2@bytedance.com/T/#m561d136611962055641f331e600e719de1e7c573
I took a quick look at your implementation, and I wouldn't say our two approaches are completely independent.
I’ve run some tests, the test methods, environment, and data are detailed in this document:
https://docs.google.com/spreadsheets/d/1p3XqVpKgx18QhqTFb0KlJfEW9giWDGSUuoKqCZzdZLg/edit?gid=1100001004#gid=1100001004
it appears that GSIM is only effective in scenarios where multi disks at very high IOPS; in some cases, there was a noticeable performance regression. If there’s something wrong with my configuration, please correct me.
Thanks.
Quoted 23 lines
If NAPI etc manage to keep the interrupt rate/load below the targets, then the adaptive controller in GSIM ends up using zero additional delay. Otherwise, it gently adjusts the moderation delay so the combination suffices to stay within the target.
All the above assumes the parameters are sensible.
GSIM or any HW moderation allows up to 1/delay_us interrupts per second, so setting delay_us=10 allows up to 100Kintr/s per CPU handling interrupts, so there is no way we can throttle interrupts below 100K/s)Quoted 8 lines
Quoted 8 lines
The system is adaptive. Moderation affects only latency and only in high load scenarios. Throughput and CPU efficiencly generally benefits significantly. Targets don't need to match precisely the platform limits, and one can make conservative and robust choices. Values like delay_us=100, target_intr_rate=1000000, hardirq_percent=70 are a very good starting point.
cheers
luigi
Quoted 6 lines · On Thu, Aug 20 2026 at 19:46, [email protected] wrote:
Quoted 6 lines
From: "Luigi Rizzo"<[email protected]>
Date: Thu, Aug 20, 2026, 15:35
Subject: Re: [PATCH v5 0/7] Global Software Interrupt Moderation (GSIM)
To: "Christoph Hellwig"<[email protected]>
Cc: "Thomas Gleixner"<[email protected]>, "Marc Zyngier"<[email protected]>, "Luigi Rizzo"<[email protected]>, "Paolo Abeni"<[email protected]>, <[email protected]>, <[email protected]>, "Bjorn Helgaas"<[email protected]>, <[email protected]>, <[email protected]>, "Fengnan Chang"<[email protected]> On Thu, Aug 20, 2026 at 9:09 AM Christoph Hellwig <[email protected]> wrote:
Can you please fix yuur mail client so it does not copy all that redundant headers into your reply?
Quoted 16 lines
Quoted 6 lines
Quoted 2 lines
Also how does this interact with adaptive polling code inside drivers like NAPI or the upcoming nvme variant?
GSIM acts at hardirq level (so below NAPI, and above device moderation e.g. the one in ethtool -C ...) and is completely orthogonal to those other mechanisms.Hi Luigi:
The NVMe variant that Christoph mentioned is this patch of mine.
https://lore.kernel.org/linux-nvme/d9210bcdf73fbe1ac8b6ec132865609a3ed68688.99d43a8b.dfff.41d1.b07d.0592a12473d2@bytedance.com/T/#m561d136611962055641f331e600e719de1e7c573
I took a quick look at your implementation, and I wouldn't say our two approaches are completely independent.
I’ve run some tests, the test methods, environment, and data are detailed in this document:
https://docs.google.com/spreadsheets/d/1p3XqVpKgx18QhqTFb0KlJfEW9giWDGSUuoKqCZzdZLg/edit?gid=1100001004#gid=1100001004
Quoted 3 lines
it appears that GSIM is only effective in scenarios where multi disks at very high IOPS; in some cases, there was a noticeable performance regression. If there’s something wrong with my configuration, please correct me.
So we have a NVME specific mechanism to tackle the same problem and a more generic version which is subsystem "independent".
Can you folks please coordinate and get your act together so that we don't end up with two competing mechanisms which make things worse than they are now.
TBH. I despise the NVME is special approach because it's fricking obvious that this is _NOT_ a NVME specific issue. But sure NVME is special as all other subsystems are special.
Q: When do driver folks actually start to look beyond the brim of their
tea cup?
A: Probably never ...
Thanks,
tglx
Quoted 6 lines · Thomas Gleixner
Quoted 3 lines
it appears that GSIM is only effective in scenarios where multi disks at very high IOPS; in some cases, there was a noticeable performance regression. If there’s something wrong with my configuration, please correct me.
So we have a NVME specific mechanism to tackle the same problem and a more generic version which is subsystem "independent".
I'm not sure they tackle the entirely same problem, although they are very related.
Quoted 3 lines
Can you folks please coordinate and get your act together so that we don't end up with two competing mechanisms which make things worse than they are now.
This is what I'm trying to get done here. This is the first time I've seen GSIM as I still try to read lkml, although I usuall fail. Unfortunately neither the nvme nor block lists were Cced on it, despite most of the numbers involving NVMe.
Quoted 3 lines
TBH. I despise the NVME is special approach because it's fricking obvious that this is _NOT_ a NVME specific issue. But sure NVME is special as all other subsystems are special.
Note that we tried to look into generic helpers, Keith tried various versions using DIMLIB, and we've also considered doing more work in the block core similar what networking does with NAPI. But we're always interested common code if it works, glad someone is looking. Although somewhat more productive suggestions would be helpful.
Quoted 11 lines · Christoph Hellwig
Quoted 6 lines · Thomas Gleixner
Quoted 3 lines
it appears that GSIM is only effective in scenarios where multi disks at very high IOPS; in some cases, there was a noticeable performance regression. If there’s something wrong with my configuration, please correct me.
So we have a NVME specific mechanism to tackle the same problem and a more generic version which is subsystem "independent".
I'm not sure they tackle the entirely same problem, although they are very related.
There is overlap but it is accidental.
The difference between the two mechanism is the following:
- GSIM addresses hardware limitations (SoCs that create huge PCIe backpressure on each MSIx interrupt, eventually reducing the available bandwidth too much, regardless of how many CPUs can process interrupts). The goal here is to throttle the total MSIx rate, blocking them at the source (PCIe device) rather than later in the interrupt controller.
- the NVME mechanism (and NAPI) aim to amortize interrupt processing overhead. It does so by recreating some fine grained single-source interrupt moderation (something that NICs do in hardware, but NVME does not, with 100us being way too coarse for modern SSDs) and an adaptive scheme to adjust the delay and decide when to kick in.
I think there is a reason for both.
GSIM because it is more general and addresses an issue the others cannot handle the subsystem-specific scheme because they can be more efficient (eg they can hook into existing helpers like the threaded irq handler, call directly the handler without going through the full irq chain, use more targeted heuristics to decide when to kick in).
cheers
luigi
Quoted 18 lines · Luigi Rizzo
Quoted 8 lines · Christoph Hellwig
Quoted 5 lines · Thomas Gleixner
Quoted 3 lines
it appears that GSIM is only effective in scenarios where multi disks at very high IOPS; in some cases, there was a noticeable performance regression. If there’s something wrong with my configuration, please correct me.
So we have a NVME specific mechanism to tackle the same problem and a more generic version which is subsystem "independent".
I'm not sure they tackle the entirely same problem, although they are very related.
There is overlap but it is accidental.
The difference between the two mechanism is the following:- GSIM addresses hardware limitations (SoCs that create huge PCIe backpressure on each MSIx interrupt, eventually reducing the available bandwidth too much, regardless of how many CPUs can process interrupts). The goal here is to throttle the total MSIx rate, blocking them at the source (PCIe device) rather than later in the interrupt controller.
Quoted 5 lines
- the NVME mechanism (and NAPI) aim to amortize interrupt processing overhead. It does so by recreating some fine grained single-source interrupt moderation (something that NICs do in hardware, but NVME does not, with 100us being way too coarse for modern SSDs) and an adaptive scheme to adjust the delay and decide when to kick in.
That's a great summary.
As for generic solutions, I've tried several approaches before, but none of them
could achieve the same or close results as NVMe adaptive interrupt polling. I'm still experimenting with different options.
Quoted 11 lines
I think there is a reason for both.
GSIM because it is more general and addresses an issue the others cannot handle the subsystem-specific scheme because they can be more efficient (eg they can hook into existing helpers like the threaded irq handler, call directly the handler without going through the full irq chain, use more targeted heuristics to decide when to kick in).
cheers
luigi
Quoted 24 lines · Fengnan
Quoted 18 lines · Luigi Rizzo
Quoted 8 lines · Christoph Hellwig
Quoted 5 lines · Thomas Gleixner
Quoted 3 lines
it appears that GSIM is only effective in scenarios where multi disks at very high IOPS; in some cases, there was a noticeable performance regression. If there’s something wrong with my configuration, please correct me.
So we have a NVME specific mechanism to tackle the same problem and a more generic version which is subsystem "independent".
I'm not sure they tackle the entirely same problem, although they are very related.
There is overlap but it is accidental.
The difference between the two mechanism is the following:- GSIM addresses hardware limitations (SoCs that create huge PCIe backpressure on each MSIx interrupt, eventually reducing the available bandwidth too much, regardless of how many CPUs can process interrupts). The goal here is to throttle the total MSIx rate, blocking them at the source (PCIe device) rather than later in the interrupt controller.
Quoted 5 lines
- the NVME mechanism (and NAPI) aim to amortize interrupt processing overhead. It does so by recreating some fine grained single-source interrupt moderation (something that NICs do in hardware, but NVME does not, with 100us being way too coarse for modern SSDs) and an adaptive scheme to adjust the delay and decide when to kick in.
Quoted 19 lines
That's a great summary.
As for generic solutions, I've tried several approaches before, but none of them
could achieve the same or close results as NVMe adaptive interrupt polling. I'm still experimenting with different options.Quoted 11 lines
I think there is a reason for both.
GSIM because it is more general and addresses an issue the others cannot handle the subsystem-specific scheme because they can be more efficient (eg they can hook into existing helpers like the threaded irq handler, call directly the handler without going through the full irq chain, use more targeted heuristics to decide when to kick in).
cheers
luigi
Sharing the test results with GSIM and other NVME specific approaches here as well. Previously discussed in
https://lore.kernel.org/all/[email protected]/
There are couple of implementations posted on NVME list, including mine, to solve nvme issues. Below data also suggests that while GSIM may bring generic improvements in certain cases, we may still need some NVME specific changes to address hardirq workload problems.
Background: I was debugging soft lockup issues where sufficiently high FIO workload can lockup the CPU cores on which high number of NVME PCI MSI interrupts are affined.
System: 128 ARM64 vCPUs, two NUMA nodes (CPUs 0-63 / 64-127), There are four data NVMe controllers, each with 14 interrupt-driven I/O queues. Corresponding queue vectors from the four controllers are routed to the same CPU. Therefore:
vectors per affected CPU: 4
affected interrupt CPUs: 14
total data-NVMe I/O vectors: 56
The affected CPUs are:
0, 10, 19, 28, 37, 46, 55,
64, 74, 83, 92, 101, 110, 119
The reproduced soft lockups happen on exactly those 14 CPUs.
The production fio command is equivalent to:
--filename=dev1:dev2:dev3:dev4
--numjobs=128
--iodepth=256
--group_reporting
so the potential total is:
128 x 256 = 32,768 requests
The NVMe hardware queue depth is 1,024, with 1,023 usable entries.
Default GSIM changes could not prevent soft lockup. Luigi suggested a timer push change in addition to GSIM changes, which prevents the lockups, but has an impact on latency and throughput. These are marked as "push" in below tables.
NVMe fallback: changes I proposed in NVME, here:
https://lore.kernel.org/all/[email protected]/
threaded NVMe IRQs: Default kernel with nvme.use_threaded_interrupts=1
High-load results are medians of three runs, you can find complete minimum-to-maximum range:
IOPS median complete range NVMe fallback 5.004M 5.002-5.004M threaded NVMe IRQs 3.140M 3.100-3.169M GSIM fixed 10 us + push 1.962M 1.935-2.025M GSIM fixed 30 us + push 1.790M 1.775-1.822M GSIM fixed 50 us + push 1.744M 1.736-1.746M
Completion latency:
mean p99 p99.9 NVMe fallback 6.533 ms 14.352 ms 16.908 ms threaded NVMe IRQs 10.421 ms 22.151 ms 24.510 ms GSIM fixed 10 us + push 16.570 ms 38.011 ms 270.533 ms GSIM fixed 30 us + push 18.283 ms 40.108 ms 492.831 ms GSIM fixed 50 us + push 18.743 ms 40.632 ms 541.065 ms
More details and numbers in my RFC series.
Regards,
Naman