Skip to contentSkip to conversation

Global Software Interrupt Moderation (GSIM)

PATCH In review Luigi Rizzo 6 participants 23 messages Aug 19 – Oct 19

Subsystems (6)
Networking Network drivers PCI NVMe +2 more subsystems
lore
Share & tools
lore
#0 lore raw

v5 cover Global Software Interrupt Moderation (GSIM)

Global Software Interrupt Moderation (GSIM) addresses a limitation of platforms, from many vendors, whose I/O performance drops significantly when the total rate of MSI-X interrupts is too high (e.g 1..3M intr/s depending on the platform).

Conventional interrupt moderation, typically implemented in hardware by NICs or storage devices, operates separately on each source (e.g. a completion queue). Large servers can have hundreds of sources, and without knowledge of global activity, keeping the total rate bounded would require moderation delays of 100-200us. Per-source adaptive moderation would reach those delays with as little as 10K intr/s per source. These values (high delay triggering very early) are unacceptable for RPC or transactional workloads.

To address this problem, GSIM measures efficiently the total and per-CPU interrupt rates, so that individual moderation delays can be dynamically adjusted based on current global and local load. This way, delays are normally 0 or very small except during actual local/global overload.

As an additional benefit, GSIM also monitors the percentage of time spent by each CPU in hardirq, and can use moderation to reserve some time for other, lower priority, tasks.

Configuration is easy and robust. System administrators specify the maximum targets (moderation delay; interrupt rate; percentage of time spent in hardirq), and which interrupt sources should be moderated. Independent per-CPU control loops adjust actual delays to try and keep metrics within the targets.

The system is adaptive. Moderation affects only latency and only in high load scenarios. Throughput and CPU efficiencly generally benefits significantly. Targets don't need to match precisely the platform limits, and one can make conservative and robust choices. Values like delay_us=100, target_intr_rate=1000000, hardirq_percent=70 are a very good starting point.

GSIM does not rely on any special hardware feature.

Global parameters can be modified at runtime with

    echo ${VALUE} | sudo tee /proc/irq/sw_moderation/${NAME}

/proc/irq/sw_moderation/stats exports statistics when enabled.

and moderation on individual interrupts can be turned on/off at runtime with

echo 1 | sudo tee /proc/irq/NN/allow_moderation # use 0 to disable

EXAMPLE:

    # global configuration: 50us, target max 1M intr/s, and 70% in intr
    echo 50 | sudo tee /proc/irq/sw_moderation/delay_us
    echo 1000000 | sudo tee /proc/irq/sw_moderation/target_intr_rate
    echo 70 | sudo tee /proc/irq/sw_moderation/hardirq_percent
    # allow moderation on all interrupts that support it
    # remember to periodically check and set the flag for dynamically
    # created interrupts since the default is 0
    echo 1 | sudo tee /proc/irq/*/allow_moderation

# check the status
grep -r . /proc/irq/*/*/../allow_moderation

# look at statistics
less /proc/irq/sw_moderation/stats

PERFORMANCE BENEFITS:
Below are some experimental results under high load comparing conventional moderation with GSIM:

- 100Gbps NIC, 32 queues: rx goes from 50 Gbps to 92.8 Gbps (line rate).
- 200Gbps NIC, 10 VMs (total 160 queues): rx goes from 30 Gbps to 190 Gbps (line rate).
- 12 SSD, 96 queues: 4K random read goes from 6M to 20.5M IOPS (device max).

In all cases, with adaptive moderatrion, latency up to p95 is unaffected at low/moderate load, even if compared with no moderation at all.

Changes in v5:
- refactored the commits based on previous feedback
- various cleanups
- conditionally reverted parent IRQ mask/unmask, which would completely defeat the mechanism GSIM is based on.

Changes in v4:
- added irqdesc and irqdata flags as suggested by maintainer
- parameters are only configured via independent procfs entries. No module parameters anymore.
- merged control and interrupt functions back into a single header/C files
- applied various annotations (lockdep, data_race())
- formatting and various renaming as suggested by maintainer.
- added performance measurements with adaptive moderation.
- removed the mechanism to conditionally enable moderation at interrupt creation. This can be done in userspace and suitable udev extensions will be handled separately.

Changes in v3:
- clearly documented architecture in kernel/irq/irq_moderation.c including how to handle enable/disable/mask, interrupt migration, hotplug and suspend.
- split implementation in 4 files irq_moderation.[ch] and irq_moderation_hook.[ch] for better separation of control plane and "dataplane" (functions ran on each interrupt)
- limited scope to handle_edge_irq() and handle_fasteoi_irq() which have been tested on actual hardware.
- tested on Intel (also with intremap=posted_msi), AMD, ARM, with NIC, nvme, vfio

Changes in v2:
- many style fixes (mostly on comments) based on reviewers' comments on v1
- removed background from Documentation/core-api/irq/irq-moderation.rst
- split procfs handlers
- moved internal details to kernel/irq/irq_moderation.h
- use cpu hotplug for per-CPU setup, removed unnecessary arch-specific changes
- select suitable irqs based on !irqd_is_level_type(irqd) && irqd_is_single_target(irqd)
- use a static_key to enable/disable the feature

Luigi Rizzo (7):
genirq: Add flags for software interrupt moderation.
genirq: Add GSIM infrastructure
genirq: Implement core GSIM moderation logic
genirq: Integrate GSIM into interrupt flow
genirq: Add GSIM user space configuration (procfs)
genirq: Adaptive Global Software Interrupt Moderation (GSIM). PCI/MSI: re-enable conditional parent mask/unmask with sw moderation

 drivers/irqchip/irq-msi-lib.c |   8 +
 drivers/pci/msi/irqdomain.c   |  20 +
 include/linux/irq.h           |  11 +-
 include/linux/irqdesc.h       |  12 +
 kernel/irq/Kconfig            |  11 +
 kernel/irq/Makefile           |   1 +
 kernel/irq/chip.c             |  14 +
 kernel/irq/debugfs.c          |   3 +
 kernel/irq/internals.h        |  20 +
 kernel/irq/irq_moderation.c   | 959 ++++++++++++++++++++++++++++++++++
 kernel/irq/irq_moderation.h   | 156 ++++++
 kernel/irq/irqdesc.c          |   1 +
 kernel/irq/manage.c           |  10 +
 kernel/irq/proc.c             |   2 +
 kernel/irq/settings.h         |  17 +
 15 files changed, 1244 insertions(+), 1 deletion(-)
 create mode 100644 kernel/irq/irq_moderation.c
 create mode 100644 kernel/irq/irq_moderation.h
Signature · 1 line
2.55.0.737.g08866a6d13-goog
#1 lore raw

PATCH v5 1/7 genirq: Add flags for software interrupt moderation.

Add two flags to support software interrupt moderation:

- IRQ_MODERATABLE is an irqdesc flag indicating that an interrupt supports moderation. This is a capability that can be set for eligible sources.

- IRQD_MODERATED is an internal irqdata flag indicating that the interrupt is currently being moderated. This is a state flag.

---
 include/linux/irq.h   | 11 ++++++++++-
 kernel/irq/debugfs.c  |  3 +++
 kernel/irq/settings.h | 17 +++++++++++++++++
 3 files changed, 30 insertions(+), 1 deletion(-)
include/linux/irq.h +10 -1
index f485369b1b4f7..232df18af1b66 100644
@@ -76,6 +76,7 @@ enum irqchip_irq_state;
  * IRQ_DISABLE_UNLAZY		- Disable lazy irq disable
  * IRQ_HIDDEN			- Don't show up in /proc/interrupts
  * IRQ_NO_DEBUG			- Exclude from note_interrupt() debugging
+ * IRQ_MODERATABLE		- Can do software interrupt moderation
  */
 enum {
 	IRQ_TYPE_NONE		= 0x00000000,
@@ -104,13 +105,14 @@ enum {
 	IRQ_HIDDEN		= (1 << 20),
 	IRQ_NO_DEBUG		= (1 << 21),
 	IRQ_RESERVED		= (1 << 22),
+	IRQ_MODERATABLE		= (1 << 23),
 };
 
 #define IRQF_MODIFY_MASK	\
 	(IRQ_TYPE_SENSE_MASK | IRQ_NOPROBE | IRQ_NOREQUEST | \
 	 IRQ_NOAUTOEN | IRQ_LEVEL | IRQ_NO_BALANCING | \
 	 IRQ_PER_CPU | IRQ_NESTED_THREAD | IRQ_NOTHREAD | IRQ_PER_CPU_DEVID | \
-	 IRQ_IS_POLLED | IRQ_DISABLE_UNLAZY | IRQ_HIDDEN)
+	 IRQ_IS_POLLED | IRQ_DISABLE_UNLAZY | IRQ_HIDDEN | IRQ_MODERATABLE)
 
 #define IRQ_NO_BALANCING_MASK	(IRQ_PER_CPU | IRQ_NO_BALANCING)
 
@@ -224,6 +226,7 @@ struct irq_data {
  *				  irqchip have flag IRQCHIP_ENABLE_WAKEUP_ON_SUSPEND set.
  * IRQD_RESEND_WHEN_IN_PROGRESS	- Interrupt may fire when already in progress in which
  *				  case it must be resent at the next available opportunity.
+ * IRQD_MODERATED		- Interrupt is currently moderated.
  */
 enum {
 	IRQD_TRIGGER_MASK		= 0xf,
@@ -249,6 +252,7 @@ enum {
 	IRQD_AFFINITY_ON_ACTIVATE	= BIT(28),
 	IRQD_IRQ_ENABLED_ON_SUSPEND	= BIT(29),
 	IRQD_RESEND_WHEN_IN_PROGRESS    = BIT(30),
+	IRQD_MODERATED			= BIT(31),
 };
 
 #define __irqd_to_state(d) ACCESS_PRIVATE((d)->common, state_use_accessors)
@@ -438,6 +442,11 @@ static inline bool irqd_needs_resend_when_in_progress(struct irq_data *d)
 	return __irqd_to_state(d) & IRQD_RESEND_WHEN_IN_PROGRESS;
 }
 
+static inline bool irqd_is_moderated(struct irq_data *d)
+{
+	return __irqd_to_state(d) & IRQD_MODERATED;
+}
+
 #undef __irqd_to_state
 
 static inline irq_hw_number_t irqd_to_hwirq(struct irq_data *d)
kernel/irq/debugfs.c +3 -0
index 5c5ebaee35f2c..634f00c0f0f59 100644
@@ -128,6 +128,8 @@ static const struct irq_bit_descr irqdata_states[] = {
 	BIT_MASK_DESCR(IRQD_IRQ_ENABLED_ON_SUSPEND),
 
 	BIT_MASK_DESCR(IRQD_RESEND_WHEN_IN_PROGRESS),
+
+	BIT_MASK_DESCR(IRQD_MODERATED),
 };
 
 static const struct irq_bit_descr irqdesc_states[] = {
@@ -140,6 +142,7 @@ static const struct irq_bit_descr irqdesc_states[] = {
 	BIT_MASK_DESCR(_IRQ_IS_POLLED),
 	BIT_MASK_DESCR(_IRQ_DISABLE_UNLAZY),
 	BIT_MASK_DESCR(_IRQ_HIDDEN),
+	BIT_MASK_DESCR(_IRQ_MODERATABLE),
 };
 
 static const struct irq_bit_descr irqdesc_istates[] = {
kernel/irq/settings.h +17 -0
index 0a0c027a5d342..1663bb46d3a99 100644
@@ -19,6 +19,7 @@ enum {
 	_IRQ_HIDDEN		= IRQ_HIDDEN,
 	_IRQ_NO_DEBUG		= IRQ_NO_DEBUG,
 	_IRQ_PROC_VALID		= IRQ_RESERVED,
+	_IRQ_MODERATABLE	= IRQ_MODERATABLE,
 	_IRQF_MODIFY_MASK	= IRQF_MODIFY_MASK,
 };
 
@@ -36,6 +37,7 @@ enum {
 #define IRQ_HIDDEN		GOT_YOU_MORON
 #define IRQ_NO_DEBUG		GOT_YOU_MORON
 #define IRQ_RESERVED		GOT_YOU_MORON
+#define IRQ_MODERATABLE		GOT_YOU_MORON
 #undef IRQF_MODIFY_MASK
 #define IRQF_MODIFY_MASK	GOT_YOU_MORON
 
@@ -193,3 +195,18 @@ static inline void irq_settings_update_proc_valid(struct irq_desc *desc, u32 set
 	desc->status_use_accessors &= ~_IRQ_PROC_VALID;
 	desc->status_use_accessors |= (set & _IRQ_PROC_VALID);
 }
+
+static inline bool irq_settings_moderatable(struct irq_desc *desc)
+{
+	return desc->status_use_accessors & _IRQ_MODERATABLE;
+}
+
+static inline void irq_settings_set_moderatable(struct irq_desc *desc)
+{
+	desc->status_use_accessors |= _IRQ_MODERATABLE;
+}
+
+static inline void irq_settings_clr_moderatable(struct irq_desc *desc)
+{
+	desc->status_use_accessors &= ~_IRQ_MODERATABLE;
+}
Signature · 1 line
2.55.0.737.g08866a6d13-goog
#2 lore raw

Re: [PATCH v5 1/7] genirq: Add flags for software interrupt moderation.

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines

Add two flags to support software interrupt moderation:

- IRQ_MODERATABLE is an irqdesc flag indicating that an interrupt supports moderation. This is a capability that can be set for eligible sources.

- IRQD_MODERATED is an internal irqdata flag indicating that the interrupt is currently being moderated. This is a state flag.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw

PATCH v5 2/7 genirq: Add GSIM infrastructure

In reply to Luigi Rizzo

Introduce the Kconfig option CONFIG_IRQ_SW_MODERATION and GSIM fields in struct irq_desc to support Global Software Interrupt Moderation (GSIM).

Note: This commit only introduces the configuration option and basic data structures. The core moderation logic, procfs interface, and integration into the interrupt flow are implemented in subsequent commits. Enabling this option at this stage is a build-safe no-op.

---
 include/linux/irqdesc.h | 12 ++++++++++++
 kernel/irq/Kconfig      | 11 +++++++++++
 kernel/irq/internals.h  |  8 ++++++++
 kernel/irq/irqdesc.c    |  1 +
 4 files changed, 32 insertions(+)
include/linux/irqdesc.h +12 -0
index 8080db17c1b1b..5ba130e224167 100644
@@ -18,6 +18,16 @@ struct irq_desc;
 struct irq_domain;
 struct pt_regs;
 
+/**
+ * struct irq_desc_swmod - software interrupt moderation state
+ * @swmod_node:		list head for per-CPU list of moderated irq_desc
+ */
+struct irq_desc_swmod {
+#ifdef CONFIG_IRQ_SW_MODERATION
+	struct list_head	swmod_node;
+#endif
+};
+
 /**
  * struct irqstat - interrupt statistics
  * @cnt:	real-time interrupt count
@@ -59,6 +69,7 @@ struct irq_redirect {
  * @threads_handled_last: comparator field for deferred spurious detection of threaded handlers
  * @lock:		locking for SMP
  * @redirect:		Facility for redirecting interrupts via irq_work
+ * @swmod_state:	software interrupt moderation state
  * @affinity_hint:	hint to user space for preferred irq affinity
  * @affinity_notify:	context for notification of affinity changes
  * @pending_mask:	pending rebalanced interrupts
@@ -95,6 +106,7 @@ struct irq_desc {
 	atomic_t		threads_handled;
 	int			threads_handled_last;
 	raw_spinlock_t		lock;
+	struct irq_desc_swmod	swmod_state;
 	struct cpumask		*percpu_enabled;
 #ifdef CONFIG_SMP
 	struct irq_redirect	redirect;
kernel/irq/Kconfig +11 -0
index 05cba4e16dad1..d8d556b4a279d 100644
@@ -150,6 +150,17 @@ config IRQ_KUNIT_TEST
 
 	  If unsure, say N.
 
+config IRQ_SW_MODERATION
+	bool "Enable Global Software Interrupt Moderation"
+	depends on PROC_FS
+	help
+	  Enable Global Software Interrupt Moderation.
+	  Uses a local timer to delay interrupts in configurable ways
+	  and depending on various global system load indicators
+	  and targets.
+
+	  If you don't know what to do here, say N.
+
 endmenu
 
 config GENERIC_IRQ_MULTI_HANDLER
kernel/irq/internals.h +8 -0
index 0ce21dd454047..716a7c2e3633a 100644
@@ -395,3 +395,11 @@ static inline struct irq_data *irqd_get_parent_data(struct irq_data *irqd)
 	return NULL;
 #endif
 }
+#ifdef CONFIG_IRQ_SW_MODERATION
+static inline void irq_moderation_init_fields(struct irq_desc *desc)
+{
+	INIT_LIST_HEAD(&desc->swmod_state.swmod_node);
+}
+#else
+static inline void irq_moderation_init_fields(struct irq_desc *desc) {}
+#endif
kernel/irq/irqdesc.c +1 -0
index 80ef4e27dcf47..e8488c9352d0c 100644
@@ -120,6 +120,7 @@ static inline void free_masks(struct irq_desc *desc) { }
 static void desc_set_defaults(unsigned int irq, struct irq_desc *desc, int node,
 			      const struct cpumask *affinity, struct module *owner)
 {
+	irq_moderation_init_fields(desc);
 	desc->irq_common_data.handler_data = NULL;
 	desc->irq_common_data.msi_desc = NULL;
 
Signature · 1 line
2.55.0.737.g08866a6d13-goog
#4 lore raw

Re: [PATCH v5 2/7] genirq: Add GSIM infrastructure

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines

Introduce the Kconfig option CONFIG_IRQ_SW_MODERATION and GSIM fields in struct irq_desc to support Global Software Interrupt Moderation (GSIM).

Note: This commit only introduces the configuration option and basic data structures. The core moderation logic, procfs interface, and integration into the interrupt flow are implemented in subsequent commits. Enabling this option at this stage is a build-safe no-op.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#5 lore raw

PATCH v5 3/7 genirq: Implement core GSIM moderation logic

In reply to Luigi Rizzo

Introduce the core GSIM moderation logic, including full documentation of the architecture, per-CPU state tracking, basic list-draining functions and hooks for the interrupt handlers.

Note: The architecture documentation describes how GSIM integrates with the interrupt flow handlers (e.g. handle_edge_irq, handle_fasteoi_irq). These handlers are modified and GSIM is integrated in subsequent commits.

---
 kernel/irq/Makefile         |   1 +
 kernel/irq/irq_moderation.c | 296 ++++++++++++++++++++++++++++++++++++
 kernel/irq/irq_moderation.h |  93 +++++++++++
 3 files changed, 390 insertions(+)
 create mode 100644 kernel/irq/irq_moderation.c
 create mode 100644 kernel/irq/irq_moderation.h
kernel/irq/Makefile +1 -0
index 86a2e5ae08f9a..07c19d4697712 100644
@@ -5,6 +5,7 @@ obj-$(CONFIG_GENERIC_IRQ_CHIP) += generic-chip.o
 obj-$(CONFIG_GENERIC_IRQ_PROBE) += autoprobe.o
 obj-$(CONFIG_IRQ_DOMAIN) += irqdomain.o
 obj-$(CONFIG_IRQ_SIM) += irq_sim.o
+obj-$(CONFIG_IRQ_SW_MODERATION) += irq_moderation.o
 obj-$(CONFIG_PROC_FS) += proc.o
 obj-$(CONFIG_GENERIC_PENDING_IRQ) += migration.o
 obj-$(CONFIG_GENERIC_IRQ_MIGRATION) += cpuhotplug.o
kernel/irq/irq_moderation.c new file +296 -0

296 lines not on this page · raw message

kernel/irq/irq_moderation.h new file +93 -0

93 lines not on this page · raw message

Signature · 1 line
2.55.0.737.g08866a6d13-goog
#6 lore raw

Re: [PATCH v5 3/7] genirq: Implement core GSIM moderation logic

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines

Introduce the core GSIM moderation logic, including full documentation of the architecture, per-CPU state tracking, basic list-draining functions and hooks for the interrupt handlers.

Note: The architecture documentation describes how GSIM integrates with the interrupt flow handlers (e.g. handle_edge_irq, handle_fasteoi_irq). These handlers are modified and GSIM is integrated in subsequent commits.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#7 lore raw

PATCH v5 4/7 genirq: Integrate GSIM into interrupt flow

In reply to Luigi Rizzo

Hook GSIM into edge/fasteoi handlers and teardown.

Specifically:
- Hook GSIM into handle_edge_irq() and handle_fasteoi_irq().
- Introduce irq_moderation_allow(desc, allow) to validate if an IRQ is eligible for GSIM (non-oneshot, edge/fasteoi, single target, no bus lock) and set/clear the IRQ_MODERATABLE flag.
- Integrate GSIM setup in __setup_irq() using irq_moderation_allow(), defaulting to disabled (use_moderation = false).
- Clear GSIM state in __free_irq() and __cleanup_nmi().

Note: At this stage, GSIM is disabled by default (use_moderation = false) and cannot be enabled at runtime yet. Subsequent commits will introduce procfs configuration and a configurable default mode. For testing this commit, use_moderation can be manually set to true in the code.

---
 kernel/irq/chip.c           | 14 +++++++++++++
 kernel/irq/internals.h      |  8 +++++++
 kernel/irq/irq_moderation.c | 42 +++++++++++++++++++++++++++++++++++++
 kernel/irq/manage.c         | 10 +++++++++
 4 files changed, 74 insertions(+)
kernel/irq/chip.c +14 -0
index de754db414d1d..04a30ba04e78b 100644
@@ -20,6 +20,7 @@
 #include <trace/events/irq.h>
 
 #include "internals.h"
+#include "irq_moderation.h"
 
 static irqreturn_t bad_chained_irq(int irq, void *dev_id)
 {
@@ -500,6 +501,10 @@ static bool irq_can_handle_pm(struct irq_desc *desc)
 		return false;
 	}
 
+	/* Moderated interrupts have IRQD_IRQ_INPROGRESS and need early return. */
+	if (irqd_is_moderated(irqd))
+		return false;
+
 	/* Check whether the interrupt is polled on another CPU */
 	if (unlikely(desc->istate & IRQS_POLL_INPROGRESS)) {
 		if (WARN_ONCE(irq_poll_cpu == smp_processor_id(),
@@ -749,6 +754,10 @@ void handle_fasteoi_irq(struct irq_desc *desc)
 	 * handling the previous one - it may need to be resent.
 	 */
 	if (!irq_can_handle_pm(desc)) {
+		if (irqd_is_moderated(&desc->irq_data)) {
+			desc->istate |= IRQS_PENDING;
+			mask_irq(desc);
+		}
 		if (irqd_needs_resend_when_in_progress(&desc->irq_data))
 			desc->istate |= IRQS_PENDING;
 		cond_eoi_irq(chip, &desc->irq_data);
@@ -769,6 +778,9 @@ void handle_fasteoi_irq(struct irq_desc *desc)
 
 	cond_unmask_eoi_irq(desc, chip);
 
+	if (irq_start_moderation(desc))
+		return;
+
 	/*
 	 * When the race described above happens this will resend the interrupt.
 	 */
@@ -858,6 +870,8 @@ void handle_edge_irq(struct irq_desc *desc)
 
 		handle_irq_event(desc);
 
+		if (irq_start_moderation(desc))
+			break;
 	} while ((desc->istate & IRQS_PENDING) && !irqd_irq_disabled(&desc->irq_data));
 }
 EXPORT_SYMBOL(handle_edge_irq);
kernel/irq/internals.h +8 -0
index 716a7c2e3633a..705c3bfe28e12 100644
@@ -400,6 +400,14 @@ static inline void irq_moderation_init_fields(struct irq_desc *desc)
 {
 	INIT_LIST_HEAD(&desc->swmod_state.swmod_node);
 }
+
+int irq_moderation_allow(struct irq_desc *desc, bool allow);
+bool irq_moderation_supported(struct irq_desc *desc);
 #else
 static inline void irq_moderation_init_fields(struct irq_desc *desc) {}
+static inline int irq_moderation_allow(struct irq_desc *desc, bool allow)
+{
+	return allow ? -EOPNOTSUPP : 0;
+}
+static inline bool irq_moderation_supported(struct irq_desc *desc) { return false; }
 #endif
kernel/irq/irq_moderation.c +42 -0

45 lines not on this page · raw message

kernel/irq/manage.c +10 -0
index 7eb07e3bdb4c2..ce92885e71323 100644
@@ -1470,6 +1470,12 @@ static bool valid_percpu_irqaction(struct irqaction *old, struct irqaction *new)
 static int
 __setup_irq(unsigned int irq, struct irq_desc *desc, struct irqaction *new)
 {
+	/*
+	 * Choose the default moderation mode.
+	 * Hardcoded to false for now; configurable defaults are added later.
+	 * Set to true here to force-enable GSIM for testing this commit.
+	 */
+	bool use_moderation = false;
 	struct irqaction *old, **old_ptr;
 	unsigned long flags, thread_mask = 0;
 	int ret, nested, shared = 0;
@@ -1789,6 +1795,8 @@ __setup_irq(unsigned int irq, struct irq_desc *desc, struct irqaction *new)
 
 	irq_pm_install_action(desc, new);
 
+	irq_moderation_allow(desc, use_moderation);
+
 	/* Reset broken irq detection when installing new handler */
 	desc->irq_count = 0;
 	desc->irqs_unhandled = 0;
@@ -1897,6 +1905,7 @@ static struct irqaction *__free_irq(struct irq_desc *desc, void *dev_id)
 	/* If this was the last handler, shut down the IRQ line: */
 	if (!desc->action) {
 		irq_settings_clr_disable_unlazy(desc);
+		irq_settings_clr_moderatable(desc);
 		/* Only shutdown. Deactivate after synchronize_hardirq() */
 		irq_shutdown(desc);
 	}
@@ -2046,6 +2055,7 @@ static const void *__cleanup_nmi(unsigned int irq, struct irq_desc *desc)
 		desc->action = NULL;
 
 		irq_settings_clr_disable_unlazy(desc);
+		irq_settings_clr_moderatable(desc);
 		irq_shutdown_and_deactivate(desc);
 	}
 
Signature · 1 line
2.55.0.737.g08866a6d13-goog
#8 lore raw

Re: [PATCH v5 4/7] genirq: Integrate GSIM into interrupt flow

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Overloading IRQD_IRQ_INPROGRESS for GSIM breaks __synchronize_hardirq() by introducing prolonged…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Overloading IRQD_IRQ_INPROGRESS for GSIM breaks __synchronize_hardirq() by introducing prolonged busy-waiting and potential deadlocks. --

--- Patch [4]: [PATCH v5 4/7] genirq: Integrate GSIM into interrupt flow --- commit 8749b57516c52ba231f3249d6f52173005693e96
Author: Luigi Rizzo <[email protected]>
Subject: genirq: Integrate GSIM into interrupt flow

This commit integrates Generic Software Interrupt Moderation (GSIM) into the handle_edge_irq() and handle_fasteoi_irq() flows. It introduces irq_moderation_allow() to validate IRQ eligibility, defaults moderation to disabled during setup, and ensures the state is cleared during IRQ teardown.

Quoted 4 lines
kernel/irq/chip.c +0 -0
index de754db414d1d..04a30ba04e78b 100644

[ ... ]

kernel/irq/chip.c static bool irq_can_handle_pm(struct irq_desc *desc) · line 507

Quoted 8 lines
kernel/irq/chip.c +4 -0
@@ -500,6 +501,10 @@ static bool irq_can_handle_pm(struct irq_desc *desc)
 		return false;
 	}
 
+	/* Moderated interrupts have IRQD_IRQ_INPROGRESS and need early return. */
+	if (irqd_is_moderated(irqd))
+		return false;
+

[Severity: High]
The comment mentions that moderated interrupts have IRQD_IRQ_INPROGRESS set. Could this cause a prolonged busy-wait or deadlock in __synchronize_hardirq() when a driver calls disable_irq() or free_irq() while the moderation timer is active?

__synchronize_hardirq() uses a tight loop to wait for this flag to clear:

kernel/irq/manage.c:__synchronize_hardirq() {
    ...
    while (irqd_irq_inprogress(&desc->irq_data))
        cpu_relax();
    ...
}

If the flag remains active for the entire duration of the moderation hrtimer, any teardown or suspend/resume operation would busy-spin for that entire duration.

Additionally, if synchronize_irq() is called with local interrupts disabled on the CPU where the moderation timer is pinned, would this hard-deadlock the system since the hrtimer can never fire to clear the flag?

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
#9 lore raw

PATCH v5 5/7 genirq: Add GSIM user space configuration (procfs)

In reply to Luigi Rizzo

Introduce procfs interfaces to configure and monitor GSIM at runtime.

This adds:
- A global directory /proc/irq/sw_moderation/ containing:
- delay_us: Read/write interface to set/get the maximum moderation delay in microseconds (defaults to 0, GSIM disabled).
- stats: Read-only interface to monitor GSIM statistics per-CPU.
- files /proc/irq/NN/allow_sw_moderation (where NN is the IRQ number) to individually allow/disallow moderation. Created only for interrupts that support moderation (e.g., edge-triggered, single-target, etc.).

---
 kernel/irq/internals.h      |   4 +
 kernel/irq/irq_moderation.c | 256 +++++++++++++++++++++++++++++++++++-
 kernel/irq/proc.c           |   2 +
 3 files changed, 261 insertions(+), 1 deletion(-)
kernel/irq/internals.h +4 -0
index 705c3bfe28e12..22a1926910f34 100644
@@ -403,6 +403,8 @@ static inline void irq_moderation_init_fields(struct irq_desc *desc)
 
 int irq_moderation_allow(struct irq_desc *desc, bool allow);
 bool irq_moderation_supported(struct irq_desc *desc);
+void irq_moderation_procfs_add(struct irq_desc *desc, umode_t umode);
+void irq_moderation_procfs_remove(struct irq_desc *desc);
 #else
 static inline void irq_moderation_init_fields(struct irq_desc *desc) {}
 static inline int irq_moderation_allow(struct irq_desc *desc, bool allow)
@@ -410,4 +412,6 @@ static inline int irq_moderation_allow(struct irq_desc *desc, bool allow)
 	return allow ? -EOPNOTSUPP : 0;
 }
 static inline bool irq_moderation_supported(struct irq_desc *desc) { return false; }
+static inline void irq_moderation_procfs_add(struct irq_desc *desc, umode_t umode) {}
+static inline void irq_moderation_procfs_remove(struct irq_desc *desc) {}
 #endif
kernel/irq/irq_moderation.c +255 -1

299 lines not on this page · raw message

kernel/irq/proc.c +2 -0
index 1b835725f7b1c..edffb30efab58 100644
@@ -379,6 +379,7 @@ void register_irq_proc(unsigned int irq, struct irq_desc *desc)
 				irq_effective_aff_list_proc_show, irqp);
 # endif
 #endif
+	irq_moderation_procfs_add(desc, 0644);
 	proc_create_single_data("spurious", 0444, desc->dir,
 				irq_spurious_proc_show, (void *)(long)irq);
 
@@ -400,6 +401,7 @@ void unregister_irq_proc(unsigned int irq, struct irq_desc *desc)
 	remove_proc_entry("effective_affinity_list", desc->dir);
 # endif
 #endif
+	irq_moderation_procfs_remove(desc);
 	remove_proc_entry("spurious", desc->dir);
 
 	snprintf(name, MAX_NAMELEN, "%u", irq);
Signature · 1 line
2.55.0.737.g08866a6d13-goog
#10 lore raw

Re: [PATCH v5 5/7] genirq: Add GSIM user space configuration (procfs)

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Unconditional removal of conditionally created `allow_sw_moderation` proc entry triggers…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Unconditional removal of conditionally created allow_sw_moderation proc entry triggers WARN(1) on unsupported IRQs. --

--- Patch [5]: [PATCH v5 5/7] genirq: Add GSIM user space configuration (procfs) --- commit fc9564358ed0238056ccb83c0aee7d6d588c4901
Author: Luigi Rizzo <[email protected]>

genirq: Add GSIM user space configuration (procfs)

Introduce procfs interfaces to configure and monitor GSIM at runtime. This adds a global directory /proc/irq/sw_moderation/ and per-IRQ files to individually allow or disallow moderation.

Quoted 4 lines
kernel/irq/irq_moderation.c +0 -0
index 2c75feb6634f3..1474d33455410 100644

[ ... ]

kernel/irq/irq_moderation.c bool irq_moderation_do_start(struct irq_desc *desc, struct irq_mod_state *m) · line 337

Quoted 11 lines
kernel/irq/irq_moderation.c +11 -0
+void irq_moderation_procfs_add(struct irq_desc *desc, umode_t umode)
+{
+	if (!irq_moderation_supported(desc))
+		return;
+	proc_create_data("allow_sw_moderation", umode, desc->dir,
+			 &allow_flag_ops, (void *)(long)desc->irq_data.irq);
+}
+
+void irq_moderation_procfs_remove(struct irq_desc *desc)
+{
+	remove_proc_entry("allow_sw_moderation", desc->dir);

[Severity: Medium]
Will this unconditional call to remove_proc_entry() trigger a kernel warning when unregistering an unsupported IRQ?

In irq_moderation_procfs_add() just above, the "allow_sw_moderation" procfs entry is only created if irq_moderation_supported(desc) evaluates to true.

Because irq_moderation_procfs_remove() lacks the same check, freeing a dynamic IRQ that doesn't support moderation will cause remove_proc_entry() to print a warning stack trace about a missing entry.

Could we add the same !irq_moderation_supported(desc) check here to avoid the warning?

Quoted 1 line

+}

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
#11 lore raw

PATCH v5 6/7 genirq: Adaptive Global Software Interrupt Moderation (GSIM).

In reply to Luigi Rizzo

GSIM helps keeping system load (as defined later) under control, but the use of a fixed moderation delay causes unnecessary latency when the system load is already low.

We focus on two types of system load related to interrupts:
- total device interrupt rates. Some platforms show severe I/O performance degradation with more than 1-2 Mintr/s across the entire system. This affects especially server-class hardware with hundreds of interrupt sources (NIC or SSD queues from many physical or virtualized devices).
- percentage of time spent in hardirq. This can become problematic in presence of large numbers of interrupt sources hitting individual CPUs (for instance, as a result of attempts to isolate application CPUs from interrupt load).

Make GSIM adaptive by measuring total and per-CPU interrupt rates, as well as time spent in hardirq on each CPU. The metrics are compared to configurable targets, and each CPU adjusts the moderation delay up or down depending on the result, using multiplicative increase/decrease.

Configuration of moderation parameters is done via procfs

  echo ${VALUE} > /proc/irq/sw_moderation/${NAME}

Parameters are:

delay_us (0 off, range 0-500)
Maximum moderation delay in microseconds.

target_intr_rate (0 off, range 0-50000000)
Total rate above which moderation should trigger.

hardirq_percent (0 off,range 0-100)
Percentage of time spent in hardirq above which moderation should trigger.

update_ms (range 1-100, default 5)
How often the metrics should be computed and moderation delay updated.

When target_intr_rate and hardirq_percent are both 0, GSIM uses delay_us as fixed moderation delay. Otherwise, the delay is dynamically adjusted up or down, independently on each CPU, based on how the total and per-CPU metrics compare with the targets.

Provided that delay_us suffices to bring the metrics within the target, the control loop will dynamically converge to the minimum actual moderation delay to stay within the target.

PERFORMANCE BENEFITS:

The tests below demonstrate how adaptive moderation allows improved throughput at high load (same as fixed moderation) and low latency ad moderate load (same as no moderation) without having to hand-tune the system based on load.

We run experiments on one x86 platform with 8 SSD (64 queues) capable of an aggregate of approximately 14M IOPS running fio with variable number of SSD devices (1 or 8), threads per disk (1 or 200), and IODEPTH (from 1 to 128 per thread) the actual command is below:

${FIO} --name=read_IOPs_test ${DEVICES} --iodepth=${IODEPTH} --numjobs=${JOBS} \
    --bs=4K --rw=randread  --filesize=1000G --ioengine=libaio --direct=1 \
    --verify=0 --randrepeat=0 --time_based=1 --runtime=600s \
    --cpus-allowed=1-119 --cpus_allowed_policy=split --group_reporting=1

For each configuration we test three moderations settings:
- OFF: delay_us=0
- FIXED: delay_us=200 target_intr_rate=0 hardirq_percent=0
- ADAPTIVE: delay_us=200 target_intr_rate=1000000 hardirq_percent=70

The first set of measurements is for ONE DISK, ONE THREAD. At low IODEPTH the throughput is latency bound, and moderation is not necessary. A fixed moderation delay dominates the latency hence reducing throughput; adaptive moderation avoids the problem.
As the IODEPTH increases, the system becomes I/O bound, and even the fixed moderation delay does not harm.

Overall: adaptive moderation is better than fixed moderation and at least as good as moderation off.

         ------ OFF --------   ------ FIXED ------   ----- ADAPTIVE ----
IODEPTH  IOPS  p50  p90  p99   IOPS  p50  p90  p99   IOPS  p50  p90  p99
         -------------------   -------------------   -------------------
1:        12K   78   88   94 .   5K  208  210  215 .  12K   78   88   96
8:        94K   83   91  110 .  38K  210  212  221 .  94K   83   91  105
32:      423K   72   85  124 . 150K  210  219  235 . 424K   72   85  124
128:     698K  180  200  243 . 513K  251  306  347 . 718K  174  194  239

A second set of measurements is with one disk and 200 threads. The system is I/O bound but without significant interrupt overhead. All three scenarios are basically equivalent.

         --------- OFF --------   ------- FIXED -------   ------ ADAPTIVE -----
IODEPTH  IOPS    p50  p90   p99   IOPS    p50  p90  p99   IOPS    p50  p90  p99
         ----------------------   ---------------------   ---------------------
1:       1581K   110  174   281 .  933K   208  223  363 . 1556K   114  176  277
8:       1768K   889 1516  1926 . 1768K   848 1516 2147 . 1768K   889 1516 1942
32:      1768K  3589 5735  7701 . 1768K  3589 5735 7635 . 1768K  3589 5735 7504
128:     1768K  14ms 24ms  31ms . 1768K  14ms 24ms 29ms . 1768K  14ms 24ms 30ms

Finally, we have one set of measurements with 8 disks and 200 threads per disk, all running on socket 0. The system would be I/O bound (and CPU/latency bound at low IODEPTH), but this platform is unable to cope with the high global interrupt rate and so moderation is really necessary to hit the disk limits.

As we see below, adaptive moderation gives more than 2X higher throughput at meaningful iodepths, and even latency is much better. The only case where we see a small regression is with iodepth=1, because the high interrupt rate triggers the control loop to increase the moderation delay.

         --------- OFF --------   -------- FIXED -------   ------ ADAPTIVE ------
IODEPTH  IOPS    p50  p90   p99   IOPS     p50  p90  p99   IOPS     p50  p90  p99
         ----------------------   ----------------------   ----------------------
1:       2304K    82   94   128 .  1030K   208  277  293 .  1842K    97  149  188
8:       5240K   128  938  1680 .  7500K   208  233  343 . 10000K   151  210  281
32:      5251K   206 3621  3949 . 12300K   184 1106 5407 . 12100K   184 1139 5407
128:     5330K  4228 12ms  17ms . 13800K  1123 4883 7373 . 13800K  1074 4883 7635

Finally, here are experiments indicating how throughput is affected by the various parameters (with 8 disks and 200 threads).

IOPS vs delay_us (target_intr_rate = 0, hardirq_percent=0)

delay_us 0 50 100 150 200 250
IODEPTH 1 2300 1860 1580 1300 1034 1066
IODEPTH 8 5254 9936 9645 8818 7500 6150
IODEPTH 32 5250 11300 13800 13800 13800 13800
IODEPTH 128 5900 13600 13900 13900 13900 13900

IOPS vs target_intr_rate (delay_us = 200, hardirq_percent=0, iodepth 128)
value 250K 500K 750K 1000K 1250K 1500K 1750K 2000K
socket0 13900 13900 13900 13800 13800 12900 11000 8808
both sockets 13900 13900 13900 13800 8600 8000 6900 6400

hardirq_percent (delay_us = 200, target_intr_rate=0, iodepth 128)
hardirq% 1 10 20 30 40 50 60 70
KIOPS 13900 13800 13300 12100 10400 8600 7400 6500

---
 kernel/irq/irq_moderation.c | 391 ++++++++++++++++++++++++++++++++++--
 kernel/irq/irq_moderation.h |  63 ++++++
 2 files changed, 442 insertions(+), 12 deletions(-)
kernel/irq/irq_moderation.c +379 -12
index 1474d33455410..74bd33c786c90 100644
@@ -9,6 +9,9 @@
 #include <linux/cpuhotplug.h>
 #include <linux/irq.h>
 #include <linux/irqdesc.h>
+#include <linux/kallsyms.h>
+#include <linux/kernel_stat.h>
+#include <linux/math64.h>
 #include <linux/mutex.h>
 #include <linux/notifier.h>
 #include <linux/proc_fs.h>
@@ -23,8 +26,9 @@
  *
  * Some platforms show reduced I/O performance when the total device interrupt
  * rate across the entire platform becomes too high. To address the problem,
- * GSIM runs after the handler to implement software interrupt moderation
- * with programmable delay.
+ * GSIM runs after the handler to measure global and per-CPU interrupt rates,
+ * compares them with configurable targets, and implements independent, per-CPU
+ * software moderation delays.
  *
  * Configuration is done at runtime via procfs
  *   echo ${VALUE} > /proc/irq/sw_moderation/${NAME}
@@ -35,6 +39,19 @@
  *       Maximum moderation delay. A reasonable range is 20-100. Higher values
  *       can be useful if the hardirq handler has long runtimes.
  *
+ *   target_intr_rate (default 0, suggested 1000000, 0 off, range 0-50000000)
+ *       The total interrupt rate above which moderation kicks in.
+ *       Not particularly critical, a value in the 500K-1M range is usually ok.
+ *
+ *   hardirq_percent (default 0, suggested 70, 0 off, range 0-100)
+ *       The hardirq percentage above which moderation kicks in.
+ *       50-90 is a reasonable range.
+ *
+ *       FIXED MODERATION mode requires target_intr_rate=0, hardirq_percent=0
+ *
+ *   update_ms (default 5, range 1-100)
+ *       How often the load is measured and moderation delay updated.
+ *
  * Moderation is allowed/disallowed dynamically for individual interrupts with
  *   echo 1 > /proc/irq/NN/allow_sw_moderation # use 0 to disallow
  *
@@ -112,7 +129,47 @@
  * GSIM parameters. Initialize delay_ns here to statically enable moderation
  * (e.g. .delay_ns = 100000).
  */
-struct irq_mod_params irq_mod_params ____cacheline_aligned;
+
+/*
+ * Recommended values for the adaptive control loop.
+ *
+ * update_ns is documented earlier and can be modified via procfs
+ *
+ * The following two allow fine tuning of the control loop and should not
+ * be modified unless there is good understanding of their impact on the
+ * stability of the controller.
+ *
+ * scale_cpus (default 150, range 50-1000)
+ *   Small update_ms may lead to underestimate the number of CPUs
+ *   simultaneously handling interrupts, and the opposite can happen
+ *   with very large values. This parameter may help correct the value,
+ *   though it is not recommended to modify the default unless there are
+ *   very strong reasons.
+ *
+ * increase_divisor (default 8, range 8-128)
+ *   This is base parameter used for multiplicative increase/decrease.
+ */
+
+#define MIN_SCALING_DIVISOR	8
+
+struct irq_mod_params irq_mod_params ____cacheline_aligned = {
+	.update_ns		= 5 * NSEC_PER_MSEC,
+	.scale_cpus		= 150,
+	.increase_divisor	= MIN_SCALING_DIVISOR,
+	.seq			= SEQCNT_ZERO(irq_mod_params.seq),
+};
+
+/*
+ * Accumulator for total interrupt and active CPUs, updated by all active
+ * CPUs on each epoch (update_ns or more).
+ * @total_intrs:	running count of total interrupts
+ * @total_cpus:		running count of total active CPUs
+ */
+struct irq_mod_counters {
+	atomic_t	total_intrs;
+	atomic_t	total_cpus;
+};
+static struct irq_mod_counters irq_mod_counters;
 
 DEFINE_PER_CPU_ALIGNED(struct irq_mod_state, irq_mod_state);
 
@@ -122,12 +179,215 @@ static DEFINE_MUTEX(swmod_mutex);
 
 static void update_enable_key(void)
 {
+	lockdep_assert_held(&swmod_mutex);
+
 	if (irq_mod_params.delay_ns != 0)
 		static_branch_enable(&irq_moderation_enabled_key);
 	else
 		static_branch_disable(&irq_moderation_enabled_key);
 }
 
+/* Slow path functions for interrupt moderation. */
+
+/*
+ * Compute smoothed average between old and cur. 'steps' is used
+ * to approximate applying the smoothing multiple times.
+ */
+static inline unsigned int smooth_avg(unsigned int old, unsigned int cur, unsigned int steps)
+{
+	const unsigned int smooth_factor = 64;
+	u64 sum;
+
+	steps = min(steps, smooth_factor - 1);
+	sum = (u64)(smooth_factor - steps) * old + (u64)steps * cur;
+	return div_u64(sum, smooth_factor);
+}
+
+/* Measure and assess time spent in hardirq. */
+static inline bool hardirq_high(struct irq_mod_state *m, unsigned int hardirq_percent,
+				u64 epoch_ns)
+{
+	bool above_threshold;
+	u64 irqtime, cur;
+
+	if (!IS_ENABLED(CONFIG_IRQ_TIME_ACCOUNTING))
+		return false;
+
+	cur = kcpustat_this_cpu->cpustat[CPUTIME_IRQ];
+	irqtime = cur - m->last_irqtime;
+	m->last_irqtime = cur;
+
+	if (hardirq_percent == 0)
+		return false;
+
+	above_threshold = irqtime * 100 > epoch_ns * hardirq_percent;
+	m->hardirq_high += above_threshold;
+	return above_threshold;
+}
+
+/* Measure and assess total and per-CPU interrupt rates. */
+static inline bool irqrate_high(struct irq_mod_state *m, unsigned int target_rate,
+				unsigned int steps, u64 epoch_ns,
+				unsigned int update_ns, unsigned int scale_cpus)
+{
+	unsigned int global_intr_rate, local_intr_rate, delta_intrs, tmp;
+	bool local_rate_high, global_rate_high;
+	u64 num_local, num_global, num_cpus;
+	/* Use unsigned long to avoid overflow in intermediate results. */
+	unsigned long active_cpus;
+	u32 denom = epoch_ns;
+	int shift = 0;
+
+	num_local = (u64)m->intr_count * NSEC_PER_SEC;
+	/* Scale denominator so we can avoid 64-bit division. */
+	if (unlikely(epoch_ns > U32_MAX)) {
+		shift = fls64(epoch_ns) - 32;
+		denom = epoch_ns >> shift;
+		num_local >>= shift;
+	}
+
+	local_intr_rate = div_u64(num_local, denom);
+
+	/* Accumulate global counter and compute global interrupt rate. */
+	tmp = atomic_add_return(m->intr_count, &irq_mod_counters.total_intrs);
+	m->intr_count = 0;
+	delta_intrs = tmp - m->last_total_intrs;
+	m->last_total_intrs = tmp;
+	num_global = (u64)delta_intrs * NSEC_PER_SEC;
+	if (unlikely(shift))
+		num_global >>= shift;
+	global_intr_rate = div_u64(num_global, denom);
+
+	/*
+	 * Count how many CPUs handled interrupts in the last epoch, needed
+	 * to determine the per-CPU target (target_rate / active_cpus).
+	 * Each active CPU increments the global counter approximately every
+	 * update_ns. Scale the value by (update_ns / epoch_ns) to get the
+	 * correct value. Also apply rounding and make sure active_cpus > 0.
+	 */
+	tmp = atomic_add_return(1, &irq_mod_counters.total_cpus);
+	active_cpus = tmp - m->last_total_cpus;
+	m->last_total_cpus = tmp;
+	num_cpus = (u64)active_cpus * update_ns + (epoch_ns / 2);
+	if (unlikely(shift))
+		num_cpus >>= shift;
+	active_cpus = div_u64(num_cpus, denom);
+	if (active_cpus < 1)
+		active_cpus = 1;
+
+	/* Compare with global and per-CPU targets. */
+	global_rate_high = global_intr_rate > target_rate;
+
+	/*
+	 * Short epochs may lead to underestimate the number of active CPUs.
+	 * Apply a scaling factor to compensate. This may make the controller
+	 * a bit more aggressive but does not harm system throughput.
+	 */
+	local_rate_high = (u64)local_intr_rate * active_cpus *
+			scale_cpus > (u64)target_rate * 100;
+
+	/* Statistics. */
+	m->global_intr_rate = smooth_avg(m->global_intr_rate, global_intr_rate, steps);
+	m->local_intr_rate = smooth_avg(m->local_intr_rate, local_intr_rate, steps);
+	m->scaled_cpu_count = smooth_avg(m->scaled_cpu_count, active_cpus * 256, steps);
+
+	if (target_rate == 0)
+		return false;
+
+	m->local_irq_high += local_rate_high;
+	m->global_irq_high += global_rate_high;
+
+	/* Moderate on this CPU only if both global and local rates are high. */
+	return global_rate_high && local_rate_high;
+}
+
+/* Periodic adjustment, called once per epoch. */
+void irq_moderation_update_epoch(struct irq_mod_state *m, u64 epoch_ns)
+{
+	unsigned int hardirq_percent, target_rate, delay_ns, update_ns;
+	unsigned int increase_divisor, scale_cpus;
+	const unsigned int min_delay_ns = 500;
+	bool above_target = false;
+	unsigned int steps, seq;
+
+	do {
+		seq = read_seqcount_begin(&irq_mod_params.seq);
+		hardirq_percent = READ_ONCE(irq_mod_params.hardirq_percent);
+		target_rate = READ_ONCE(irq_mod_params.target_intr_rate);
+		delay_ns = READ_ONCE(irq_mod_params.delay_ns);
+		update_ns = READ_ONCE(irq_mod_params.update_ns);
+		increase_divisor = READ_ONCE(irq_mod_params.increase_divisor);
+		scale_cpus = READ_ONCE(irq_mod_params.scale_cpus);
+	} while (read_seqcount_retry(&irq_mod_params.seq, seq));
+
+	/*
+	 * If one parameter changes, set the moderation delay to max, and rely
+	 * on the adaptive mechanism to adjust it down if necessary.
+	 * Otherwise the system may be stuck with an interrupt rate that is
+	 * already below the threshold because of bus congestion (one of the
+	 * problems that GSIM is trying to address), and the controller would
+	 * have no signal react. Starting from a high value gives it a chance
+	 * to converge if parameters allow it.
+	 */
+	if (seq != m->seq) {
+		m->seq = seq;
+		m->mod_ns = delay_ns;
+		m->intr_count = 0;
+		m->last_total_intrs = atomic_read(&irq_mod_counters.total_intrs);
+		m->last_total_cpus = atomic_read(&irq_mod_counters.total_cpus);
+		if (IS_ENABLED(CONFIG_IRQ_TIME_ACCOUNTING))
+			m->last_irqtime = kcpustat_this_cpu->cpustat[CPUTIME_IRQ];
+		return;
+	}
+
+	if (target_rate == 0 && hardirq_percent == 0) {
+		/* Use fixed moderation delay. */
+		m->mod_ns = delay_ns;
+		m->global_intr_rate = 0;
+		m->local_intr_rate = 0;
+		m->scaled_cpu_count = 0;
+		return;
+	}
+
+	/*
+	 * The controller wants to scale the delay mod_ns by (1 + 1/D) every "update_ns".
+	 * Since we operate every epoch_ns >= update_ns, the formula becomes
+	 *   mod_ns = mod_ns * ((1 + 1/D) ** (epoch_ns / update_ns))
+	 * which we approximate with "mod_ns = mod_ns * (1 + steps/D)"
+	 * where "steps = epoch_ns / update_ns" clamped to a value < D.
+	 */
+	steps = (unsigned int)clamp_t(u64, div_u64(epoch_ns, update_ns),
+				      1ULL, (u64)(MIN_SCALING_DIVISOR - 1u));
+
+	if (irqrate_high(m, target_rate, steps, epoch_ns, update_ns, scale_cpus))
+		above_target = true;
+
+	if (hardirq_high(m, hardirq_percent, epoch_ns))
+		above_target = true;
+
+	/*
+	 * Controller: adjust delay with exponential increase or decrease.
+	 *
+	 * Following standard practices, we increase fast (smaller divisor) to
+	 * aggressively slow down when the interrupt rate goes up, but decrease
+	 * slowly (larger divisor) to reduce the chance of load spikes as the
+	 * delay goes down.
+	 */
+	if (above_target) {
+		/* Make sure the value is large enough for the exponential to grow. */
+		if (m->mod_ns < min_delay_ns)
+			m->mod_ns = min_delay_ns;
+		m->mod_ns += m->mod_ns * steps / increase_divisor;
+		if (m->mod_ns > delay_ns)
+			m->mod_ns = delay_ns;
+	} else {
+		m->mod_ns -= m->mod_ns * steps / (2 * increase_divisor);
+		/* Round down to 0 values that are too small to bother. */
+		if (m->mod_ns < min_delay_ns)
+			m->mod_ns = 0;
+	}
+}
+
 /* Actually start moderation. */
 bool irq_moderation_do_start(struct irq_desc *desc, struct irq_mod_state *m)
 {
@@ -138,7 +398,7 @@ bool irq_moderation_do_start(struct irq_desc *desc, struct irq_mod_state *m)
 		const u64 slack_ns = 2000;
 
 		/* Accumulate sleep time, no moderation if too small. */
-		m->sleep_ns += READ_ONCE(irq_mod_params.delay_ns);
+		m->sleep_ns += m->mod_ns;
 		if (m->sleep_ns < min_delay_ns)
 			return false;
 		/* We need moderation, start the timer. */
@@ -153,6 +413,11 @@ bool irq_moderation_do_start(struct irq_desc *desc, struct irq_mod_state *m)
 	 */
 	m->enqueue++;
 	list_add(&desc->swmod_state.swmod_node, &m->descs);
+	/*
+	 * Set IRQD_IRQ_INPROGRESS so that synchronize_irq() called during
+	 * free_irq() will block until the timer drains this descriptor from
+	 * the moderation list.
+	 */
 	irqd_set(&desc->irq_data, IRQD_IRQ_INPROGRESS | IRQD_MODERATED);
 	__disable_irq(desc);
 	return true;
@@ -186,17 +451,31 @@ struct swmod_procfs_entry {
 	struct var_info	var;
 };
 
+static void write_param(unsigned int *ptr, unsigned int value)
+{
+	unsigned long flags;
+
+	local_irq_save(flags);
+	write_seqcount_begin(&irq_mod_params.seq);
+	WRITE_ONCE(*ptr, value);
+	write_seqcount_end(&irq_mod_params.seq);
+	local_irq_restore(flags);
+}
+
 static ssize_t swmod_wr(struct var_info *v, const char __user *s, size_t count)
 {
 	unsigned int value;
 	int ret;
 
+	lockdep_assert_held(&swmod_mutex);
+
 	ret = kstrtouint_from_user(s, count, 0, &value);
 	if (ret)
 		return ret;
 	if (value < v->min || value > v->max)
 		return -ERANGE;
-	WRITE_ONCE(*v->ptr, value * v->scale);
+	write_param(v->ptr, value * v->scale);
+
 	return count;
 }
 
@@ -216,34 +495,87 @@ static ssize_t swmod_wr_delay(struct var_info *v, const char __user *s, size_t c
 	return ret;
 }
 
-#define HEAD_FMT "%5s  %8s  %11s  %11s\n"
-#define BODY_FMT "%5u  %8u  %11u  %11u\n"
+#define HEAD_FMT "%5s  %8s  %10s  %4s  %8s  %11s  %11s  %11s  %11s  %11s\n"
+#define BODY_FMT "%5u  %8u  %10u  %4u  %8u  %11u  %11u  %11u  %11u  %11u\n"
 
 /* Print statistics */
 static void rd_stats(struct seq_file *p)
 {
 	unsigned int delay_ns = READ_ONCE(irq_mod_params.delay_ns);
-	int cpu;
+	unsigned long global_intr_rate = 0, global_irq_high = 0;
+	unsigned long local_irq_high = 0, hardirq_high = 0;
+	int recent_epoch_limit, cpu, active_cpus = 0;
 
 	if (delay_ns == 0)
 		return;
 	seq_printf(p, HEAD_FMT,
-		   "# CPU", "delay_ns", "timer_set", "enqueue");
+		   "# CPU", "irq/s", "loc_irq/s", "cpus", "delay_ns",
+		   "irq_hi", "loc_irq_hi", "hardirq_hi", "timer_set",
+		   "enqueue");
+
+	/*
+	 * Accumulate/print only entries updated within ~20-30s. The high 32 bits
+	 * of timestamps give ~4s resolution, so we can use them without the need
+	 * for 64bit atomics (because epoch_start_ns is updated concurrently).
+	 */
+	recent_epoch_limit = (ktime_get_ns() - 20ULL * NSEC_PER_SEC) >> 32;
 
 	for_each_possible_cpu(cpu) {
 		/* Copy statistics, will only use some unsigned int values; races ok. */
 		struct irq_mod_state cur = data_race(*per_cpu_ptr(&irq_mod_state, cpu));
 
+		if (cur.epoch_start_ns && (int)(cur.epoch_start_ns >> 32) >= recent_epoch_limit) {
+			/* Recent entry, accumulate in global rate. */
+			active_cpus++;
+			global_intr_rate += cur.global_intr_rate;
+		} else {
+			/* Stale entries, print as 0. */
+			cur.global_intr_rate = 0;
+			cur.local_intr_rate = 0;
+			cur.scaled_cpu_count = 0;
+			cur.mod_ns = 0;
+		}
+
+		global_irq_high += cur.global_irq_high;
+		local_irq_high += cur.local_irq_high;
+		hardirq_high += cur.hardirq_high;
+
 		seq_printf(p, BODY_FMT,
 			   cpu,
-			   delay_ns,
+			   cur.global_intr_rate,
+			   cur.local_intr_rate,
+			   (cur.scaled_cpu_count + 128) / 256,
+			   cur.mod_ns,
+			   cur.global_irq_high,
+			   cur.local_irq_high,
+			   cur.hardirq_high,
 			   cur.timer_set,
 			   cur.enqueue);
 	}
 
 	seq_printf(p, "\n"
-		   "delay_us             %lu\n",
-		   delay_ns / NSEC_PER_USEC);
+		   "delay_us             %lu\n"
+		   "target_intr_rate     %u\n"
+		   "hardirq_percent      %u\n"
+		   "update_ms            %ld\n"
+		   "scale_cpus           %u\n",
+		   delay_ns / NSEC_PER_USEC,
+		   READ_ONCE(irq_mod_params.target_intr_rate),
+		   READ_ONCE(irq_mod_params.hardirq_percent),
+		   READ_ONCE(irq_mod_params.update_ns) / NSEC_PER_MSEC,
+		   READ_ONCE(irq_mod_params.scale_cpus));
+
+	seq_printf(p,
+		   "intr_rate            %lu\n"
+		   "irq_high             %lu\n"
+		   "my_irq_high          %lu\n"
+		   "hardirq_percent_high %lu\n"
+		   "total_interrupts     %u\n"
+		   "total_cpus           %u\n",
+		   active_cpus ? global_intr_rate / active_cpus : 0,
+		   global_irq_high, local_irq_high, hardirq_high,
+		   atomic_read(&irq_mod_counters.total_intrs),
+		   atomic_read(&irq_mod_counters.total_cpus));
 }
 
 static int param_show(struct seq_file *p, void *v)
@@ -476,15 +808,41 @@ static struct swmod_procfs_entry procfs_entries[] = {
 		.rd	= swmod_rd,
 		.var	= SET_VAR(&irq_mod_params.delay_ns, 0, 500, NSEC_PER_USEC),
 	},
+	{
+		.name	= "target_intr_rate",
+		.wr	= swmod_wr,
+		.rd	= swmod_rd,
+		.var	= SET_VAR(&irq_mod_params.target_intr_rate, 0, 50000000, 1),
+	},
+	{
+		.name	= "hardirq_percent",
+		.wr	= swmod_wr,
+		.rd	= swmod_rd,
+		.var	= SET_VAR(&irq_mod_params.hardirq_percent, 0, 100, 1),
+	},
+	{
+		.name	= "update_ms",
+		.wr	= swmod_wr,
+		.rd	= swmod_rd,
+		.var	= SET_VAR(&irq_mod_params.update_ns, 1, 100, NSEC_PER_MSEC),
+	},
 	{
 		.name = "stats",
 		.rd = rd_stats,
 	},
+	/* The next parameters have no procfs entries, only range validation. */
+	{
+		.var	= SET_VAR(&irq_mod_params.increase_divisor, MIN_SCALING_DIVISOR, 128, 1),
+	},
+	{
+		.var	= SET_VAR(&irq_mod_params.scale_cpus, 50, 1000, 1),
+	},
 };
 
 static int __init init_irq_moderation(void)
 {
 	struct proc_dir_entry *dir;
+	unsigned long flags;
 	int cpuhp_state;
 	int i, ret;
 
@@ -535,8 +893,17 @@ static int __init init_irq_moderation(void)
 		goto cleanup_1;
 	}
 
+	/* Increment sequence counter so per-CPU m->seq (0) mismatches on epoch 1 */
+	local_irq_save(flags);
+	write_seqcount_begin(&irq_mod_params.seq);
+	write_seqcount_end(&irq_mod_params.seq);
+	local_irq_restore(flags);
+
 	/* Enable if the defaults require it. */
+	/* Acquire swmod_mutex to satisfy lockdep assertions. */
+	mutex_lock(&swmod_mutex);
 	update_enable_key();
+	mutex_unlock(&swmod_mutex);
 	return 0;
 
 cleanup_1:
kernel/irq/irq_moderation.h +63 -0

110 lines not on this page · raw message

Signature · 1 line
2.55.0.737.g08866a6d13-goog
#12 lore raw

Re: [PATCH v5 6/7] genirq: Adaptive Global Software Interrupt Moderation (GSIM).

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 15 lines

GSIM helps keeping system load (as defined later) under control, but the use of a fixed moderation delay causes unnecessary latency when the system load is already low.

We focus on two types of system load related to interrupts:
- total device interrupt rates. Some platforms show severe I/O performance degradation with more than 1-2 Mintr/s across the entire system. This affects especially server-class hardware with hundreds of interrupt sources (NIC or SSD queues from many physical or virtualized devices).
- percentage of time spent in hardirq. This can become problematic in presence of large numbers of interrupt sources hitting individual CPUs (for instance, as a result of attempts to isolate application CPUs from interrupt load).

Quoted 6 lines

[ ... ]
hardirq_percent (delay_us = 200, target_intr_rate=0, iodepth 128)
hardirq% 1 10 20 30 40 50 60 70
KIOPS 13900 13800 13300 12100 10400 8600 7400 6500

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
#13 lore raw

PATCH v5 7/7 PCI/MSI: re-enable conditional parent mask/unmask with sw moderation

In reply to Luigi Rizzo

Recent commits removed masking of interrupts at the PCIe level when the masking is also done at the parent (e.g. on ARM with GIC). This optimization saves the PCIe write, and especially the expensive readback, but causes the device to still send MSI interrupts that are ignored.

As a result, it very harmful on platforms (many, for multiple architectures) that are unable to deal with too many MSI interrupts, and defeats GSIM (CONFIG_IRQ_SW_MODERATION), which is esplicitely designed to address this problem, but depends on masking interrupts at the PCIe level.

Revert to the old behavior when GSIM (CONFIG_IRQ_SW_MODERATION) is compiled in.

---
 drivers/irqchip/irq-msi-lib.c |  8 ++++++++
 drivers/pci/msi/irqdomain.c   | 20 ++++++++++++++++++++
 2 files changed, 28 insertions(+)
drivers/irqchip/irq-msi-lib.c +8 -0
index 45e0ed3134ce1..453a5d5a2c106 100644
@@ -116,6 +116,14 @@ bool msi_lib_init_dev_msi_info(struct device *dev, struct irq_domain *domain,
 	if (!chip->irq_set_affinity && !(info->flags & MSI_FLAG_NO_AFFINITY))
 		chip->irq_set_affinity = msi_domain_set_affinity;
 
+	/*
+	 * The problem addressed by software moderation depends on actually
+	 * masking at the PCI level. irq_mask() can be optimized separately
+	 * for the common cases outside the move/teardown paths.
+	 */
+	if (IS_ENABLED(CONFIG_IRQ_SW_MODERATION))
+		return true;
+
 	/*
 	 * If the parent domain insists on being in charge of masking, obey
 	 * blindly. The interrupt is un-masked at the PCI level on startup
drivers/pci/msi/irqdomain.c +20 -0
index 6e65f0f44112e..44b022fee7f1a 100644
@@ -80,6 +80,22 @@ static unsigned int cond_startup_parent(struct irq_data *data)
 	return 0;
 }
 
+static __always_inline void cond_mask_parent(struct irq_data *data)
+{
+	struct msi_domain_info *info = data->domain->host_data;
+
+	if (unlikely(info->flags & MSI_FLAG_PCI_MSI_MASK_PARENT))
+		irq_chip_mask_parent(data);
+}
+
+static __always_inline void cond_unmask_parent(struct irq_data *data)
+{
+	struct msi_domain_info *info = data->domain->host_data;
+
+	if (unlikely(info->flags & MSI_FLAG_PCI_MSI_MASK_PARENT))
+		irq_chip_unmask_parent(data);
+}
+
 static void pci_irq_shutdown_msi(struct irq_data *data)
 {
 	struct msi_desc *desc = irq_data_get_msi_desc(data);
@@ -102,12 +118,14 @@ static void pci_irq_mask_msi(struct irq_data *data)
 	struct msi_desc *desc = irq_data_get_msi_desc(data);
 
 	pci_msi_mask(desc, BIT(data->irq - desc->irq));
+	cond_mask_parent(data);
 }
 
 static void pci_irq_unmask_msi(struct irq_data *data)
 {
 	struct msi_desc *desc = irq_data_get_msi_desc(data);
 
+	cond_unmask_parent(data);
 	pci_msi_unmask(desc, BIT(data->irq - desc->irq));
 }
 
@@ -160,10 +178,12 @@ static unsigned int pci_irq_startup_msix(struct irq_data *data)
 static void pci_irq_mask_msix(struct irq_data *data)
 {
 	pci_msix_mask(irq_data_get_msi_desc(data));
+	cond_mask_parent(data);
 }
 
 static void pci_irq_unmask_msix(struct irq_data *data)
 {
+	cond_unmask_parent(data);
 	pci_msix_unmask(irq_data_get_msi_desc(data));
 }
 
Signature · 1 line
2.55.0.737.g08866a6d13-goog
#14 lore raw

Re: [PATCH v5 7/7] PCI/MSI: re-enable conditional parent mask/unmask with sw moderation

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 10 lines

Recent commits removed masking of interrupts at the PCIe level when the masking is also done at the parent (e.g. on ARM with GIC). This optimization saves the PCIe write, and especially the expensive readback, but causes the device to still send MSI interrupts that are ignored.

As a result, it very harmful on platforms (many, for multiple architectures) that are unable to deal with too many MSI interrupts, and defeats GSIM (CONFIG_IRQ_SW_MODERATION), which is esplicitely designed to address this problem, but depends on masking interrupts at the PCIe level.

Quoted 4 lines

Revert to the old behavior when GSIM (CONFIG_IRQ_SW_MODERATION) is compiled in.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
#15 lore raw
In reply to Luigi Rizzo
Quoted 5 lines · Luigi Rizzo

Configuration is easy and robust. System administrators specify the maximum targets (moderation delay; interrupt rate; percentage of time spent in hardirq), and which interrupt sources should be moderated. Independent per-CPU control loops adjust actual delays to try and keep metrics within the targets.

Can we find a way to autodetect and autoenable this? A magic go faster mode that needs very specific tuning is annoying compare to sensible defaults.

Also how does this interact with adaptive polling code inside drivers like NAPI or the upcoming nvme variant?

Quoted 108 lines

The system is adaptive. Moderation affects only latency and only in high load scenarios. Throughput and CPU efficiencly generally benefits significantly. Targets don't need to match precisely the platform limits, and one can make conservative and robust choices. Values like delay_us=100, target_intr_rate=1000000, hardirq_percent=70 are a very good starting point.

GSIM does not rely on any special hardware feature.

Global parameters can be modified at runtime with

    echo ${VALUE} | sudo tee /proc/irq/sw_moderation/${NAME}

/proc/irq/sw_moderation/stats exports statistics when enabled.

and moderation on individual interrupts can be turned on/off at runtime with

echo 1 | sudo tee /proc/irq/NN/allow_moderation # use 0 to disable

EXAMPLE:

    # global configuration: 50us, target max 1M intr/s, and 70% in intr
    echo 50 | sudo tee /proc/irq/sw_moderation/delay_us
    echo 1000000 | sudo tee /proc/irq/sw_moderation/target_intr_rate
    echo 70 | sudo tee /proc/irq/sw_moderation/hardirq_percent
    # allow moderation on all interrupts that support it
    # remember to periodically check and set the flag for dynamically
    # created interrupts since the default is 0
    echo 1 | sudo tee /proc/irq/*/allow_moderation

# check the status
grep -r . /proc/irq/*/*/../allow_moderation

# look at statistics
less /proc/irq/sw_moderation/stats

PERFORMANCE BENEFITS:
Below are some experimental results under high load comparing conventional moderation with GSIM:

- 100Gbps NIC, 32 queues: rx goes from 50 Gbps to 92.8 Gbps (line rate).
- 200Gbps NIC, 10 VMs (total 160 queues): rx goes from 30 Gbps to 190 Gbps (line rate).
- 12 SSD, 96 queues: 4K random read goes from 6M to 20.5M IOPS (device max).

In all cases, with adaptive moderatrion, latency up to p95 is unaffected at low/moderate load, even if compared with no moderation at all.

Changes in v5:
- refactored the commits based on previous feedback
- various cleanups
- conditionally reverted parent IRQ mask/unmask, which would completely defeat the mechanism GSIM is based on.

Changes in v4:
- added irqdesc and irqdata flags as suggested by maintainer
- parameters are only configured via independent procfs entries. No module parameters anymore.
- merged control and interrupt functions back into a single header/C files
- applied various annotations (lockdep, data_race())
- formatting and various renaming as suggested by maintainer.
- added performance measurements with adaptive moderation.
- removed the mechanism to conditionally enable moderation at interrupt creation. This can be done in userspace and suitable udev extensions will be handled separately.

Changes in v3:
- clearly documented architecture in kernel/irq/irq_moderation.c including how to handle enable/disable/mask, interrupt migration, hotplug and suspend.
- split implementation in 4 files irq_moderation.[ch] and irq_moderation_hook.[ch] for better separation of control plane and "dataplane" (functions ran on each interrupt)
- limited scope to handle_edge_irq() and handle_fasteoi_irq() which have been tested on actual hardware.
- tested on Intel (also with intremap=posted_msi), AMD, ARM, with NIC, nvme, vfio

Changes in v2:
- many style fixes (mostly on comments) based on reviewers' comments on v1
- removed background from Documentation/core-api/irq/irq-moderation.rst
- split procfs handlers
- moved internal details to kernel/irq/irq_moderation.h
- use cpu hotplug for per-CPU setup, removed unnecessary arch-specific changes
- select suitable irqs based on !irqd_is_level_type(irqd) && irqd_is_single_target(irqd)
- use a static_key to enable/disable the feature

Luigi Rizzo (7):
genirq: Add flags for software interrupt moderation.
genirq: Add GSIM infrastructure
genirq: Implement core GSIM moderation logic
genirq: Integrate GSIM into interrupt flow
genirq: Add GSIM user space configuration (procfs)
genirq: Adaptive Global Software Interrupt Moderation (GSIM). PCI/MSI: re-enable conditional parent mask/unmask with sw moderation

 drivers/irqchip/irq-msi-lib.c |   8 +
 drivers/pci/msi/irqdomain.c   |  20 +
 include/linux/irq.h           |  11 +-
 include/linux/irqdesc.h       |  12 +
 kernel/irq/Kconfig            |  11 +
 kernel/irq/Makefile           |   1 +
 kernel/irq/chip.c             |  14 +
 kernel/irq/debugfs.c          |   3 +
 kernel/irq/internals.h        |  20 +
 kernel/irq/irq_moderation.c   | 959 ++++++++++++++++++++++++++++++++++
Quoted 13 lines
 kernel/irq/irq_moderation.h   | 156 ++++++
 kernel/irq/irqdesc.c          |   1 +
 kernel/irq/manage.c           |  10 +
 kernel/irq/proc.c             |   2 +
 kernel/irq/settings.h         |  17 +
 15 files changed, 1244 insertions(+), 1 deletion(-)
 create mode 100644 kernel/irq/irq_moderation.c
 create mode 100644 kernel/irq/irq_moderation.h
Signature · 1 line
2.55.0.737.g08866a6d13-goog

---end quoted text---

#16 lore raw
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · Luigi Rizzo

Configuration is easy and robust. System administrators specify the maximum targets (moderation delay; interrupt rate; percentage of time spent in hardirq), and which interrupt sources should be moderated. Independent per-CPU control loops adjust actual delays to try and keep metrics within the targets.

Can we find a way to autodetect and autoenable this? A magic go faster mode that needs very specific tuning is annoying compare to sensible defaults.

As mentioned later, the tunings do not need to be specific, because the control loop adjusts the delay to stay within the target, and the target is generally just a "don't overload me" value with little impact on performance.

Of course there are many cases (eg small systems with little I/O) where moderation is not needed at all so at least one bit of user input is necessary.

What I normally do is use another small patch to pass the initial settings via module parameters, using values similar to those below
(enable on all interrupts, delay_us=100, target_intr_rate=1000000, hardirq_percent=70)

Quoted 2 lines

Also how does this interact with adaptive polling code inside drivers like NAPI or the upcoming nvme variant?

GSIM acts at hardirq level (so below NAPI, and above device moderation e.g. the one in ethtool -C ...) and is completely orthogonal to those other mechanisms.

If NAPI etc manage to keep the interrupt rate/load below the targets, then the adaptive controller in GSIM ends up using zero additional delay. Otherwise, it gently adjusts the moderation delay so the combination suffices to stay within the target.

All the above assumes the parameters are sensible.
GSIM or any HW moderation allows up to 1/delay_us interrupts per second, so setting delay_us=10 allows up to 100Kintr/s per CPU handling interrupts, so there is no way we can throttle interrupts below 100K/s)

Quoted 8 lines
Quoted 8 lines

The system is adaptive. Moderation affects only latency and only in high load scenarios. Throughput and CPU efficiencly generally benefits significantly. Targets don't need to match precisely the platform limits, and one can make conservative and robust choices. Values like delay_us=100, target_intr_rate=1000000, hardirq_percent=70 are a very good starting point.

cheers
luigi

#17 lore raw
Quoted 32 lines

From: "Luigi Rizzo"<[email protected]>
Date:  Thu, Aug 20, 2026, 15:35
Subject:  Re: [PATCH v5 0/7] Global Software Interrupt Moderation (GSIM)
To: "Christoph Hellwig"<[email protected]>
Cc: "Thomas Gleixner"<[email protected]>, "Marc Zyngier"<[email protected]>, "Luigi Rizzo"<[email protected]>, "Paolo Abeni"<[email protected]>, <[email protected]>, <[email protected]>, "Bjorn Helgaas"<[email protected]>, <[email protected]>, <[email protected]>, "Fengnan Chang"<[email protected]>

Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · Luigi Rizzo

Configuration is easy and robust. System administrators specify the maximum targets (moderation delay; interrupt rate; percentage of time spent in hardirq), and which interrupt sources should be moderated. Independent per-CPU control loops adjust actual delays to try and keep metrics within the targets.

Can we find a way to autodetect and autoenable this?  A magic go faster mode that needs very specific tuning is annoying compare to sensible defaults.


As mentioned later, the tunings do not need to be specific, because the control loop adjusts the delay to stay within the target, and the target is generally just a "don't overload me" value with little impact on performance.

Of course there are many cases (eg small systems with little I/O) where moderation is not needed at all so at least one bit of user input is necessary.
What I normally do is use another small patch to pass the initial settings via module parameters, using values similar to those below
(enable on all interrupts, delay_us=100, target_intr_rate=1000000, hardirq_percent=70)

Quoted 6 lines
Quoted 2 lines

Also how does this interact with adaptive polling code inside drivers like NAPI or the upcoming nvme variant?


GSIM acts at hardirq level (so below NAPI, and above device moderation e.g. the one in ethtool -C ...) and is completely orthogonal to those other mechanisms.

Hi Luigi:
The NVMe variant that Christoph mentioned is this patch of mine.
https://lore.kernel.org/linux-nvme/d9210bcdf73fbe1ac8b6ec132865609a3ed68688.99d43a8b.dfff.41d1.b07d.0592a12473d2@bytedance.com/T/#m561d136611962055641f331e600e719de1e7c573
I took a quick look at your implementation, and I wouldn't say our two approaches are completely independent.
I’ve run some tests, the test methods, environment, and data are detailed in this document:
https://docs.google.com/spreadsheets/d/1p3XqVpKgx18QhqTFb0KlJfEW9giWDGSUuoKqCZzdZLg/edit?gid=1100001004#gid=1100001004

it appears that GSIM is only effective in scenarios where multi disks at very high IOPS; in some cases, there was a noticeable performance regression. If there’s something wrong with my configuration, please correct me.

Thanks.

Quoted 23 lines


If NAPI etc manage to keep the interrupt rate/load below the targets, then the adaptive controller in GSIM ends up using zero additional delay. Otherwise, it gently adjusts the moderation delay so the combination suffices to stay within the target.

All the above assumes the parameters are sensible.
GSIM or any HW moderation allows up to 1/delay_us interrupts per second, so setting delay_us=10 allows up to 100Kintr/s per CPU handling interrupts, so there is no way we can throttle interrupts below 100K/s)

Quoted 8 lines
Quoted 8 lines

The system is adaptive. Moderation affects only latency and only in high load scenarios.  Throughput and CPU efficiencly generally benefits significantly.  Targets don't need to match precisely the platform limits, and one can make conservative and robust choices. Values like delay_us=100, target_intr_rate=1000000, hardirq_percent=70 are a very good starting point.


cheers
luigi

#18 lore raw
Quoted 6 lines · On Thu, Aug 20 2026 at 19:46, [email protected] wrote:
Quoted 6 lines

From: "Luigi Rizzo"<[email protected]>
Date:  Thu, Aug 20, 2026, 15:35
Subject:  Re: [PATCH v5 0/7] Global Software Interrupt Moderation (GSIM)
To: "Christoph Hellwig"<[email protected]>
Cc: "Thomas Gleixner"<[email protected]>, "Marc Zyngier"<[email protected]>, "Luigi Rizzo"<[email protected]>, "Paolo Abeni"<[email protected]>, <[email protected]>, <[email protected]>, "Bjorn Helgaas"<[email protected]>, <[email protected]>, <[email protected]>, "Fengnan Chang"<[email protected]> On Thu, Aug 20, 2026 at 9:09 AM Christoph Hellwig <[email protected]> wrote:

Can you please fix yuur mail client so it does not copy all that redundant headers into your reply?

Quoted 16 lines
Quoted 6 lines
Quoted 2 lines

Also how does this interact with adaptive polling code inside drivers like NAPI or the upcoming nvme variant?


GSIM acts at hardirq level (so below NAPI, and above device moderation e.g. the one in ethtool -C ...) and is completely orthogonal to those other mechanisms.

Hi Luigi:
The NVMe variant that Christoph mentioned is this patch of mine.
https://lore.kernel.org/linux-nvme/d9210bcdf73fbe1ac8b6ec132865609a3ed68688.99d43a8b.dfff.41d1.b07d.0592a12473d2@bytedance.com/T/#m561d136611962055641f331e600e719de1e7c573
I took a quick look at your implementation, and I wouldn't say our two approaches are completely independent.
I’ve run some tests, the test methods, environment, and data are detailed in this document:
https://docs.google.com/spreadsheets/d/1p3XqVpKgx18QhqTFb0KlJfEW9giWDGSUuoKqCZzdZLg/edit?gid=1100001004#gid=1100001004

Quoted 3 lines

it appears that GSIM is only effective in scenarios where multi disks at very high IOPS; in some cases, there was a noticeable performance regression. If there’s something wrong with my configuration, please correct me.

So we have a NVME specific mechanism to tackle the same problem and a more generic version which is subsystem "independent".

Can you folks please coordinate and get your act together so that we don't end up with two competing mechanisms which make things worse than they are now.

TBH. I despise the NVME is special approach because it's fricking obvious that this is _NOT_ a NVME specific issue. But sure NVME is special as all other subsystems are special.

Q: When do driver folks actually start to look beyond the brim of their
tea cup?

A: Probably never ...

Thanks,

tglx

#19 lore raw
Quoted 6 lines · Thomas Gleixner
Quoted 3 lines

it appears that GSIM is only effective in scenarios where multi disks at very high IOPS; in some cases, there was a noticeable performance regression. If there’s something wrong with my configuration, please correct me.

So we have a NVME specific mechanism to tackle the same problem and a more generic version which is subsystem "independent".

I'm not sure they tackle the entirely same problem, although they are very related.

Quoted 3 lines

Can you folks please coordinate and get your act together so that we don't end up with two competing mechanisms which make things worse than they are now.

This is what I'm trying to get done here. This is the first time I've seen GSIM as I still try to read lkml, although I usuall fail. Unfortunately neither the nvme nor block lists were Cced on it, despite most of the numbers involving NVMe.

Quoted 3 lines

TBH. I despise the NVME is special approach because it's fricking obvious that this is _NOT_ a NVME specific issue. But sure NVME is special as all other subsystems are special.

Note that we tried to look into generic helpers, Keith tried various versions using DIMLIB, and we've also considered doing more work in the block core similar what networking does with NAPI. But we're always interested common code if it works, glad someone is looking. Although somewhat more productive suggestions would be helpful.

#20 lore raw
Quoted 11 lines · Christoph Hellwig
Quoted 6 lines · Thomas Gleixner
Quoted 3 lines

it appears that GSIM is only effective in scenarios where multi disks at very high IOPS; in some cases, there was a noticeable performance regression. If there’s something wrong with my configuration, please correct me.

So we have a NVME specific mechanism to tackle the same problem and a more generic version which is subsystem "independent".

I'm not sure they tackle the entirely same problem, although they are very related.

There is overlap but it is accidental.
The difference between the two mechanism is the following:

- GSIM addresses hardware limitations (SoCs that create huge PCIe backpressure on each MSIx interrupt, eventually reducing the available bandwidth too much, regardless of how many CPUs can process interrupts). The goal here is to throttle the total MSIx rate, blocking them at the source (PCIe device) rather than later in the interrupt controller.

- the NVME mechanism (and NAPI) aim to amortize interrupt processing overhead. It does so by recreating some fine grained single-source interrupt moderation (something that NICs do in hardware, but NVME does not, with 100us being way too coarse for modern SSDs) and an adaptive scheme to adjust the delay and decide when to kick in.

I think there is a reason for both.

GSIM because it is more general and addresses an issue the others cannot handle the subsystem-specific scheme because they can be more efficient (eg they can hook into existing helpers like the threaded irq handler, call directly the handler without going through the full irq chain, use more targeted heuristics to decide when to kick in).

cheers
luigi

#21 lore raw
Quoted 18 lines · Luigi Rizzo
Quoted 8 lines · Christoph Hellwig
Quoted 5 lines · Thomas Gleixner
Quoted 3 lines

it appears that GSIM is only effective in scenarios where multi disks at very high IOPS; in some cases, there was a noticeable performance regression. If there’s something wrong with my configuration, please correct me.

So we have a NVME specific mechanism to tackle the same problem and a more generic version which is subsystem "independent".

I'm not sure they tackle the entirely same problem, although they are very related.

There is overlap but it is accidental.
The difference between the two mechanism is the following:

- GSIM addresses hardware limitations (SoCs that create huge PCIe backpressure on each MSIx interrupt, eventually reducing the available bandwidth too much, regardless of how many CPUs can process interrupts). The goal here is to throttle the total MSIx rate, blocking them at the source (PCIe device) rather than later in the interrupt controller.

Quoted 5 lines

- the NVME mechanism (and NAPI) aim to amortize interrupt processing overhead. It does so by recreating some fine grained single-source interrupt moderation (something that NICs do in hardware, but NVME does not, with 100us being way too coarse for modern SSDs) and an adaptive scheme to adjust the delay and decide when to kick in.

That's a great summary.

As for generic solutions, I've tried several approaches before, but none of them
could achieve the same or close results as NVMe adaptive interrupt polling. I'm still experimenting with different options.

Quoted 11 lines

I think there is a reason for both.

GSIM because it is more general and addresses an issue the others cannot handle the subsystem-specific scheme because they can be more efficient (eg they can hook into existing helpers like the threaded irq handler, call directly the handler without going through the full irq chain, use more targeted heuristics to decide when to kick in).

cheers
luigi

#22 lore raw
Quoted 24 lines · Fengnan
Quoted 18 lines · Luigi Rizzo
Quoted 8 lines · Christoph Hellwig
Quoted 5 lines · Thomas Gleixner
Quoted 3 lines

it appears that GSIM is only effective in scenarios where multi disks at very high IOPS; in some cases, there was a noticeable performance regression. If there’s something wrong with my configuration, please correct me.

So we have a NVME specific mechanism to tackle the same problem and a more generic version which is subsystem "independent".

I'm not sure they tackle the entirely same problem, although they are very related.

There is overlap but it is accidental.
The difference between the two mechanism is the following:

- GSIM addresses hardware limitations (SoCs that create huge PCIe backpressure on each MSIx interrupt, eventually reducing the available bandwidth too much, regardless of how many CPUs can process interrupts). The goal here is to throttle the total MSIx rate, blocking them at the source (PCIe device) rather than later in the interrupt controller.

Quoted 5 lines

- the NVME mechanism (and NAPI) aim to amortize interrupt processing overhead. It does so by recreating some fine grained single-source interrupt moderation (something that NICs do in hardware, but NVME does not, with 100us being way too coarse for modern SSDs) and an adaptive scheme to adjust the delay and decide when to kick in.

Quoted 19 lines

That's a great summary.

As for generic solutions, I've tried several approaches before, but none of them
could achieve the same or close results as NVMe adaptive interrupt polling. I'm still experimenting with different options.

Quoted 11 lines

I think there is a reason for both.

GSIM because it is more general and addresses an issue the others cannot handle the subsystem-specific scheme because they can be more efficient (eg they can hook into existing helpers like the threaded irq handler, call directly the handler without going through the full irq chain, use more targeted heuristics to decide when to kick in).

cheers
luigi

Sharing the test results with GSIM and other NVME specific approaches here as well. Previously discussed in
https://lore.kernel.org/all/[email protected]/

There are couple of implementations posted on NVME list, including mine, to solve nvme issues. Below data also suggests that while GSIM may bring generic improvements in certain cases, we may still need some NVME specific changes to address hardirq workload problems.

Background: I was debugging soft lockup issues where sufficiently high FIO workload can lockup the CPU cores on which high number of NVME PCI MSI interrupts are affined.

System: 128 ARM64 vCPUs, two NUMA nodes (CPUs 0-63 / 64-127), There are four data NVMe controllers, each with 14 interrupt-driven I/O queues. Corresponding queue vectors from the four controllers are routed to the same CPU. Therefore:

vectors per affected CPU: 4
affected interrupt CPUs: 14
total data-NVMe I/O vectors: 56

The affected CPUs are:
0, 10, 19, 28, 37, 46, 55,
64, 74, 83, 92, 101, 110, 119
The reproduced soft lockups happen on exactly those 14 CPUs.

The production fio command is equivalent to:
--filename=dev1:dev2:dev3:dev4
--numjobs=128
--iodepth=256
--group_reporting

so the potential total is:
128 x 256 = 32,768 requests

The NVMe hardware queue depth is 1,024, with 1,023 usable entries.

Default GSIM changes could not prevent soft lockup. Luigi suggested a timer push change in addition to GSIM changes, which prevents the lockups, but has an impact on latency and throughput. These are marked as "push" in below tables.

NVMe fallback: changes I proposed in NVME, here:
https://lore.kernel.org/all/[email protected]/
threaded NVMe IRQs: Default kernel with nvme.use_threaded_interrupts=1

High-load results are medians of three runs, you can find complete minimum-to-maximum range:

                               IOPS median       complete range
   NVMe fallback                 5.004M          5.002-5.004M
   threaded NVMe IRQs            3.140M          3.100-3.169M
   GSIM fixed 10 us + push       1.962M          1.935-2.025M
   GSIM fixed 30 us + push       1.790M          1.775-1.822M
   GSIM fixed 50 us + push       1.744M          1.736-1.746M

Completion latency:

                               mean        p99          p99.9
   NVMe fallback              6.533 ms    14.352 ms     16.908 ms
   threaded NVMe IRQs        10.421 ms    22.151 ms     24.510 ms
   GSIM fixed 10 us + push   16.570 ms    38.011 ms    270.533 ms
   GSIM fixed 30 us + push   18.283 ms    40.108 ms    492.831 ms
   GSIM fixed 50 us + push   18.743 ms    40.632 ms    541.065 ms

More details and numbers in my RFC series.

Regards,
Naman