nvme-pci: adaptive interrupt coalescing
cover nvme-pci: adaptive interrupt coalescing
Background
==========
Nowadays, with the high performance PCIe 5.0 NVMe drives been used more increasingly and widely, it's important to maintain and fully realize NVMe drive's performance at the system level in scenarios where multiple NVMe drives are used simultaneously.
Problem
=======
With PCIe 5.0 NVMe drives, the interrupt count can reach very high levels in burst when running IOPS workloads (e.g., 4k/8k randread) with multiple queues across many NVMe drives. These high interrupts generated instantaneously may put pressure on current CPU handling mechanism.
Specifically, IOPS with small IO (e.g., 4k/8k randread) does not meet expectations when running with multiple queues among many NVMe drives (e.g., 6-8 NVMe drives per socket), even with each queue and job assigned to separate CPU cores.
The IOPS for small IO can be significantly improved when enabling interrupt coalescing in this case. However, in some cases (e.g., big IO for bandwidth or latency-sensitive tests with small IO and small queue depth), interrupt coalescing has side effects.
Solution
========
Monitor the IOPS and average queue depth of inflight IO periodically on each NVMe drive and enable/disable interrupt coalescing automatically without user intervention. This feature is enabled by default and can be disabled/enabled through debugfs during runtime.
Test Results
============
1. IOPS improved from < 20M to 55M for 4k randread with fio job=8, qd=128 across 16 PCIe Gen5 NVMe drives on Intel Xeon GNR CPUs.
- Latency not affected for 4k randread with job=1/2/4/8,
qd=1/2/4/8/16/32
- Bandwidth not affected for 32k/64k/128k read/write with
job=1/2/4/8, qd=1282. Tested with several mainstream PCIe Gen5 SSD vendors and Intel CPUs (including AMD) with improved 4K random read IOPS.
Notes
=====
1. Interrupt coalescing is a per-drive feature, not per-queue feature. Each queue has a separate interrupt, but coalescing is controlled globally per drive. We introduce thresholds for IO_DELTA, AVG_QUEUE_DEPTH, and TOTAL_PENDING_IO to judge workload status.
2. Tunable parameters exposed via debugfs at /sys/kernel/debug/nvme_pci/:
- disable_adaptive_interrupt_coalescing: 0=enabled, 1=disabled
- high_io_delta: IO count threshold per cycle (default: 45000)
- avg_qd_threshold: average queue depth threshold (default: 32)
- total_inflight_io_threshold: total inflight IO threshold
(default: 256)
3. The default threshold values are derived from empirical testing on various PCIe Gen5 NVMe drives.
Jun Zeng (1):
nvme-pci: adaptive interrupt coalescing
drivers/nvme/host/pci.c | 540 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 539 insertions(+), 1 deletion(-)
base-commit: 5d6919055dec134de3c40167a490f33c74c12581
Signature · 1 line
2.43.0
PATCH 1/1 nvme-pci: adaptive interrupt coalescing
Add adaptive interrupt coalescing to improve IOPS for high-throughput workloads. Monitor IO pressure periodically and enable/disable coalescing automatically based on IOPS, queue depth, and inflight IO thresholds. This feature is enabled by default, can be controlled through debugfs variable during runtime.
This significantly improves 4k random read IOPS on multi-disk Gen5 NVMe configurations without affecting latency-sensitive workloads (i.e., double the total IOPS with 16 NVMe disks).
--- drivers/nvme/host/pci.c | 540 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 539 insertions(+), 1 deletion(-)
drivers/nvme/host/pci.c +539 -1
@@ -9,6 +9,7 @@#include <linux/blkdev.h> #include <linux/blk-mq-dma.h> #include <linux/blk-integrity.h> +#include <linux/debugfs.h> #include <linux/dmi.h> #include <linux/init.h> #include <linux/interrupt.h>
@@ -19,6 +20,7 @@#include <linux/module.h> #include <linux/mutex.h> #include <linux/nodemask.h> +#include <linux/notifier.h> #include <linux/once.h> #include <linux/pci.h> #include <linux/suspend.h>
@@ -334,6 +336,20 @@ struct nvme_dev {unsigned int nr_allocated_queues;unsigned int nr_write_queues;unsigned int nr_poll_queues; + +/* adaptive interrupt coalescing support: */ +bool in_coalescing; +bool need_reset_coalesce; +bool irq_coalesce_supported; +bool coalesce_work_stop; +u32 coalesce_set_streak; +u32 coalesce_clear_streak; +unsigned long *active_q_bitmap; +u64 prev_coal_check_time; +struct delayed_work coalesce_work; +struct notifier_block coalesce_notifier; +bool coalesce_notifier_registered; +struct nvme_descriptor_pools descriptor_pools[]; };
@@ -391,6 +407,10 @@ struct nvme_queue {__le32 *dbbuf_sq_ei;__le32 *dbbuf_cq_ei;struct completion delete_done; +/* to count complete io number of this queue */ +atomic_long_t io_cnt; +/* Previous io_cnt for delta calculation */ +unsigned long prev_io_cnt; };/* bits for iod->flags */
@@ -1402,6 +1422,397 @@ static blk_status_t nvme_prep_rq(struct request *req)return ret; } +/* definition and logic for adaptive interrupt coalescing */ +#define NVME_ADAPTIVE_COALESCING_ENABLE 0x0001 +#define NVME_ADAPTIVE_COALESCING_DISABLE 0x0002 +#define COAL_CHECK_INTERVAL_MS 100 +#define COAL_CHECK_IO_DELTA_THRESH 45000/* 45k io num, in 100 ms */ +#define COAL_CHECK_STREAK_THRESH 5 +#define COAL_CHECK_AVG_INFLT_IO_THRESH 32 +#define COAL_CHECK_INFLT_IO_THRESH 256 +#define COAL_ACTIVE_Q_SAMPLE_INTERVAL 512 +#define COALESCE_SET_VALUE 0x10a +#define COALESCE_CLEAR_VALUE 0x000 + +static BLOCKING_NOTIFIER_HEAD (nvme_adaptive_coalesce_notifier_list); + +/* enable adaptive interrupt coalescing by default */ +static int adaptive_coalescing_disabled; +static u32 high_io_delta =COAL_CHECK_IO_DELTA_THRESH ; +static u32 avg_qd_threshold =COAL_CHECK_AVG_INFLT_IO_THRESH ; +static u32 total_inflight_io_threshold =COAL_CHECK_INFLT_IO_THRESH ; + +static int nvme_set_irq_coalesce (struct nvme_ctrl *ctrl,u32 v) +{ +return nvme_set_features (ctrl,NVME_FEAT_IRQ_COALESCE , v,NULL ,0 ,NULL ); +} + +static int nvme_get_irq_coalesce (struct nvme_ctrl *ctrl,u32 *v) +{ +return nvme_get_features (ctrl,NVME_FEAT_IRQ_COALESCE ,0 ,NULL ,0 , v); +} + +static bool nvme_check_irq_coalesce_supported (struct nvme_ctrl *ctrl) +{ +u32 value; + +if (nvme_get_irq_coalesce (ctrl, &value)) { +dev_info (ctrl->device,"interrupt coalescing not supported\n" ); +return false; + } +return true; +} + +static int disable_adaptive_coalescing_show (void *data,u64 *val) +{ + *val = adaptive_coalescing_disabled; +return 0 ; +} + +static int disable_adaptive_coalescing_store (void *data,u64 val) +{ +int old_val, new_val; + +if (val !=0 && val !=1 ) +return -EINVAL ; + + new_val = (int )val; + old_val =xchg (&adaptive_coalescing_disabled, new_val); + +/* If old_val == new_val, no change needed */ +if (old_val == new_val) +return 0 ; + +if (new_val) +blocking_notifier_call_chain (&nvme_adaptive_coalesce_notifier_list, +NVME_ADAPTIVE_COALESCING_DISABLE ,NULL ); +else +blocking_notifier_call_chain (&nvme_adaptive_coalesce_notifier_list, +NVME_ADAPTIVE_COALESCING_ENABLE ,NULL ); + +return 0 ; +} +DEFINE_DEBUGFS_ATTRIBUTE (disable_adaptive_coalescing_fops, + disable_adaptive_coalescing_show, + disable_adaptive_coalescing_store, +"%llu\n" ); + +static int nvme_coalesce_work_cpu (struct nvme_dev *dev) +{ +int node =dev_to_node (dev->dev); +int cpu; + +if (node !=NUMA_NO_NODE &&node_online (node)) { + cpu =cpumask_last (cpumask_of_node (node)); +if (cpu < nr_cpu_ids &&cpu_online (cpu)) +return cpu; + } + + cpu =cpumask_last (cpu_online_mask); +if (cpu < nr_cpu_ids &&cpu_online (cpu)) +return cpu; + +return WORK_CPU_UNBOUND ; +} + +/* +* Check if coalescing should be enabled based on IO pressure. +* Returns true if high IO pressure is detected. +*/ +static bool nvme_should_enable_coalesce (u64 io_delta,u64 time_interval, +u32 avg_inflight_io,u32 busy_queues) +{ +/* Check if IOPS exceeds threshold (adjusted for actual time interval) */ +if (io_delta *COAL_CHECK_INTERVAL_MS *NSEC_PER_MSEC <= + high_io_delta * time_interval) +return false; + +/* +* Enable coalescing when average number of inflight IOs on active queues +* is larger enough to indicate a throughput-oriented workload. +*/ +if (avg_inflight_io > avg_qd_threshold) +return true; + +/* +* Even when each active queue is relatively shallow, still enable +* coalescing if the aggregate inflight IO across busy queues is high. +*/ +return busy_queues * avg_inflight_io > total_inflight_io_threshold; +} + +/* Update coalesce streak counters based on current IO pressure */ +static void nvme_update_coalesce_streak (struct nvme_dev *dev,bool should_enable) +{ +if (should_enable) { + dev->coalesce_set_streak++; + dev->coalesce_clear_streak =0 ; + }else { + dev->coalesce_clear_streak++; + dev->coalesce_set_streak =0 ; + } +} + +/* Apply coalesce state change if streak threshold is reached */ +static void nvme_apply_coalesce_change (struct nvme_dev *dev) +{ +if (!dev->in_coalescing) { +if (dev->coalesce_set_streak <=COAL_CHECK_STREAK_THRESH ) +return ; + +if (nvme_set_irq_coalesce (&dev->ctrl,COALESCE_SET_VALUE )) { +dev_warn_ratelimited (dev->ctrl.device, +"set irq coalesce with value:%d failed.\n" , +COALESCE_SET_VALUE ); +return ; + } + + dev->in_coalescing = true; + dev->coalesce_set_streak =0 ; + dev->coalesce_clear_streak =0 ; +return ; + } + +if (dev->coalesce_clear_streak <=COAL_CHECK_STREAK_THRESH ) +return ; + +if (nvme_set_irq_coalesce (&dev->ctrl,COALESCE_CLEAR_VALUE )) { +dev_warn_ratelimited (dev->ctrl.device, +"clear irq coalesce with value:%d failed.\n" , +COALESCE_CLEAR_VALUE ); +return ; + } + + dev->in_coalescing = false; + dev->coalesce_set_streak =0 ; + dev->coalesce_clear_streak =0 ; +} + +/* Collect IO statistics from all queues */ +static void nvme_collect_io_stats (struct nvme_dev *dev,u64 *io_delta, +u32 *avg_inflight_io,u32 *busy_queues) +{ +struct nvme_queue *nvmeq; +u64 total_inflight_io =0 ; +unsigned int i; +u32 queues_counted =0 ; + + *io_delta =0 ; + +/* +* Count active queues during iteration rather than using bitmap_weight() +* upfront. The bitmap can be modified concurrently by nvme_mark_queue_active() +* from other CPUs, so we count the actual queues processed to ensure the +* divisor matches the accumulated total_inflight_io. +*/ +for_each_set_bit (i, dev->active_q_bitmap, dev->nr_allocated_queues) { +u32 sq_tail; +u32 cq_head; +u32 q_depth; + + nvmeq = &dev->queues[i]; +/* +* Read cq_head before sq_tail in source order for a conservative +* estimate. If we read sq_tail first and completions happen between +* reads, cq_head could advance past the observed sq_tail, causing +* the inflight calculation to produce an artificially high value. +* +* Note: READ_ONCE() only prevents compiler reordering; weakly ordered +* architectures may still reorder at hardware level. This is acceptable +* for approximate statistics - occasional inflated values don't affect +* adaptive coalescing decisions significantly. +*/ + cq_head =READ_ONCE (nvmeq->cq_head); + sq_tail =READ_ONCE (nvmeq->sq_tail); + q_depth =READ_ONCE (nvmeq->q_depth); +/* Skip uninitialized queue to avoid divide-by-zero */ +if (!q_depth) +continue ; +/* Use unsigned arithmetic to handle ring buffer wrap correctly */ + total_inflight_io += (sq_tail + q_depth - cq_head) % q_depth; + queues_counted++; + } + + *busy_queues = queues_counted; +if (queues_counted) + *avg_inflight_io =div_u64 (total_inflight_io, queues_counted); +else + *avg_inflight_io =0 ; + +/* +* Calculate io_delta and advance each queue's baseline every cycle, +* including cycles where no queue crossed an active sampling boundary. +* This keeps the completion delta aligned with the measurement interval. +* +* We must calculate delta in native word size first (unsigned long) +* before accumulating to 64-bit. On 32-bit platforms, if a queue's +* 32-bit counter wraps from 0xFFFFFFFF to 0, calculating delta in +* 32-bit unsigned arithmetic handles wrap correctly (e.g., 5 - 0xFFFFFFF0 +* = 21). Accumulating absolute 64-bit values would cause huge errors. +*/ +for (i = dev->nr_allocated_queues -1 ; i >0 ; i--) { +unsigned long curr_cnt, queue_delta; + + nvmeq = &dev->queues[i]; + curr_cnt =atomic_long_read (&nvmeq->io_cnt); +/* Unsigned subtraction handles wrap correctly */ + queue_delta = curr_cnt - nvmeq->prev_io_cnt; + nvmeq->prev_io_cnt = curr_cnt; + *io_delta += queue_delta; + } + +/* +* Use atomic clear_bit to avoid mixing atomic and non-atomic ops +* on the same memory, which could lose updates and trigger KCSAN. +*/ +for (i =0 ; i < dev->nr_allocated_queues; i++) +clear_bit (i, dev->active_q_bitmap); +} + +static void nvme_adjust_coalesce (struct nvme_dev *dev) +{ +u64 now =ktime_get_ns (); +u64 time_interval; +u64 io_delta; +u32 avg_inflight_io; +u32 busy_queues; +bool should_enable; + +if (READ_ONCE (adaptive_coalescing_disabled)) { +if (nvme_ctrl_state (&dev->ctrl) !=NVME_CTRL_LIVE ) +return ; + +/* Force clear coalescing when disabled */ + dev->coalesce_set_streak =0 ; + dev->coalesce_clear_streak +=COAL_CHECK_STREAK_THRESH +1 ; +dev_info (dev->ctrl.device,"clearing coalesce due to disable\n" ); +nvme_apply_coalesce_change (dev); +return ; + } + +/* If ctrlr state no LIVE, then skip this round */ +if (nvme_ctrl_state (&dev->ctrl) !=NVME_CTRL_LIVE ) +return ; + +/* +* Pairs with smp_wmb() in nvme_reset_work() to ensure we see +* need_reset_coalesce=true if we see state=LIVE after reset. +*/ +smp_rmb (); + +/* Check if controller been reset - coalescing config may be lost */ +if (dev->need_reset_coalesce) { +unsigned int j; + +/* +* Use atomic clear_bit to avoid data race with concurrent +* set_bit() from nvme_mark_queue_active() on other CPUs. +*/ +for (j =0 ; j < dev->nr_allocated_queues; j++) +clear_bit (j, dev->active_q_bitmap); + +/* +* Sync each queue's prev_io_cnt to current io_cnt, because queues +* may not be reallocated and io_cnt retains lifetime values. +* Otherwise io_delta would be erroneously huge on next cycle. +*/ +for (j = dev->nr_allocated_queues -1 ; j >0 ; j--) + dev->queues[j].prev_io_cnt =atomic_long_read (&dev->queues[j].io_cnt); + +/* +* Clear hardware coalescing state to ensure software and hardware +* states are synchronized after controller reset. +*/ +if (dev->in_coalescing) +nvme_set_irq_coalesce (&dev->ctrl,COALESCE_CLEAR_VALUE ); + + dev->in_coalescing = false; + dev->coalesce_set_streak =0 ; + dev->coalesce_clear_streak =0 ; + dev->need_reset_coalesce = false; + dev->prev_coal_check_time = now; +return ; + } + +nvme_collect_io_stats (dev, &io_delta, &avg_inflight_io, &busy_queues); + time_interval = now - dev->prev_coal_check_time; + dev->prev_coal_check_time = now; + + should_enable =nvme_should_enable_coalesce (io_delta, time_interval, + avg_inflight_io, busy_queues); +nvme_update_coalesce_streak (dev, should_enable); +nvme_apply_coalesce_change (dev); +} + +static void nvme_adjust_coalesce_work (struct work_struct *work) +{ + struct nvme_dev *dev = container_of(work, struct nvme_dev, coalesce_work.work); +/* +* Schedule the delayed work on the last CPU of the NUMA node the disk +* belongs to. +*/ +int cpu =nvme_coalesce_work_cpu (dev); + +/* Check stop flag before doing any work or rescheduling */ +if (READ_ONCE (dev->coalesce_work_stop)) +return ; + +if (READ_ONCE (adaptive_coalescing_disabled) && !dev->in_coalescing) +return ; + +nvme_adjust_coalesce (dev); + +/* Check again before rescheduling to avoid use-after-free */ +if (!READ_ONCE (dev->coalesce_work_stop)) +schedule_delayed_work_on (cpu, &dev->coalesce_work, +msecs_to_jiffies (COAL_CHECK_INTERVAL_MS )); +} + +static void nvme_mark_queue_active (struct nvme_queue *nvmeq,u16 sq_tail) +{ +if (READ_ONCE (adaptive_coalescing_disabled)) +return ; + +/* ignore admin op */ +if (nvmeq->qid ==0 ) +return ; + +/* +* Sample at COAL_ACTIVE_Q_SAMPLE_INTERVAL boundaries and whenever the +* submission queue wraps. This ensures queues shallower than the sample +* interval can still be recognized as active. +*/ +if ((sq_tail & (COAL_ACTIVE_Q_SAMPLE_INTERVAL -1 )) ==0 ) +set_bit (nvmeq->qid, nvmeq->dev->active_q_bitmap); +} + +static int nvme_pci_coalescing_notify (struct notifier_block *nb, +unsigned long action,void *data) +{ +struct nvme_dev *dev = + container_of(nb, struct nvme_dev, coalesce_notifier); +/* +* Schedule the delayed work on the last CPU of the NUMA node the disk +* belongs to. +*/ +int cpu =nvme_coalesce_work_cpu (dev); + +switch (action) { +case NVME_ADAPTIVE_COALESCING_ENABLE : +/* Use cached value to avoid blocking admin command in notifier */ +if (dev->irq_coalesce_supported) +schedule_delayed_work_on (cpu, &dev->coalesce_work, +msecs_to_jiffies (COAL_CHECK_INTERVAL_MS )); +break ; +case NVME_ADAPTIVE_COALESCING_DISABLE : +/* no need to do things now, the coalesce_work will exit automatically */ +break ; +default : +break ; + } +return NOTIFY_OK ; +} +static blk_status_t nvme_queue_rq (struct blk_mq_hw_ctx *hctx,const struct blk_mq_queue_data *bd) {
@@ -1424,8 +1835,10 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, ret =nvme_prep_rq (req);if (unlikely (ret))return ret; +spin_lock (&nvmeq->sq_lock);nvme_sq_copy_cmd (nvmeq, &iod->cmd); +nvme_mark_queue_active (nvmeq, nvmeq->sq_tail);nvme_write_sq_db (nvmeq, bd->last);spin_unlock (&nvmeq->sq_lock);return BLK_STS_OK ;
@@ -1443,6 +1856,7 @@ static void nvme_submit_cmds(struct nvme_queue *nvmeq, struct rq_list *rqlist)struct nvme_iod *iod =blk_mq_rq_to_pdu (req);nvme_sq_copy_cmd (nvmeq, &iod->cmd); +nvme_mark_queue_active (nvmeq, nvmeq->sq_tail); }nvme_write_sq_db (nvmeq, true);spin_unlock (&nvmeq->sq_lock);
@@ -1555,6 +1969,8 @@ static inline void nvme_handle_cqe(struct nvme_queue *nvmeq,return ; } +atomic_long_inc (&nvmeq->io_cnt); +trace_nvme_sq (req, cqe->sq_head, nvmeq->sq_tail);if (!nvme_try_complete_req (req, cqe->status, cqe->result) && !blk_mq_add_to_batch(req, iob,
@@ -2124,6 +2540,8 @@ static int nvme_alloc_queue(struct nvme_dev *dev, int qid, int depth) nvmeq->cq_phase =1 ; nvmeq->q_db = &dev->dbs[qid *2 * dev->db_stride]; nvmeq->qid = qid; +atomic_long_set (&nvmeq->io_cnt,0 ); + nvmeq->prev_io_cnt =0 ; dev->ctrl.queue_count++;return 0 ;
@@ -3342,8 +3760,18 @@ static void nvme_pci_free_ctrl(struct nvme_ctrl *ctrl) {struct nvme_dev *dev =to_nvme_dev (ctrl); +/* Signal work to stop before canceling to prevent rearming */ +WRITE_ONCE (dev->coalesce_work_stop, true); +if (dev->coalesce_notifier_registered) { +blocking_notifier_chain_unregister ( + &nvme_adaptive_coalesce_notifier_list, + &dev->coalesce_notifier); + dev->coalesce_notifier_registered = false; + } +cancel_delayed_work_sync (&dev->coalesce_work);nvme_free_tagset (dev);put_device (dev->dev); +bitmap_free (dev->active_q_bitmap);kfree (dev->queues);kfree (dev); }
@@ -3429,6 +3857,18 @@ static void nvme_reset_work(struct work_struct *work)nvme_free_tagset (dev); } +/* +* Set need_reset_coalesce before changing to LIVE state to avoid +* race with nvme_adjust_coalesce_work() which could run with stale +* bitmap and prev_io_cnt values causing erroneous IOPS calculation. +*/ + dev->need_reset_coalesce = true; +/* +* Ensure need_reset_coalesce is visible before the LIVE state change. +* Pairs with smp_rmb() in nvme_adjust_coalesce(). +*/ +smp_wmb (); +/* * If only admin queue live, keep it to do further investigation or * recovery.
@@ -3436,6 +3876,7 @@ static void nvme_reset_work(struct work_struct *work)if (!nvme_change_ctrl_state (&dev->ctrl,NVME_CTRL_LIVE )) {dev_warn (dev->ctrl.device,"failed to mark controller live state\n" ); + dev->need_reset_coalesce = false; result = -ENODEV ; goto out; }
@@ -3650,8 +4091,18 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev,if (!dev)return ERR_PTR (-ENOMEM );INIT_WORK (&dev->ctrl.reset_work, nvme_reset_work); +INIT_DELAYED_WORK (&dev->coalesce_work, nvme_adjust_coalesce_work); + dev->coalesce_notifier.notifier_call = nvme_pci_coalescing_notify; + dev->coalesce_notifier_registered = false;mutex_init (&dev->shutdown_lock); + dev->in_coalescing = false; + dev->need_reset_coalesce = false; + dev->coalesce_work_stop = false; + dev->coalesce_set_streak =0 ; + dev->coalesce_clear_streak =0 ; + dev->prev_coal_check_time =ktime_get_ns (); + dev->nr_write_queues = write_queues; dev->nr_poll_queues = poll_queues; dev->nr_allocated_queues =nvme_max_io_queues (dev) +1 ;
@@ -3660,6 +4111,12 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev,if (!dev->queues) goto out_free_dev; + dev->active_q_bitmap =bitmap_zalloc_node (dev->nr_allocated_queues,GFP_KERNEL , node); +if (!dev->active_q_bitmap) { + ret = -ENOMEM ; + goto out_free_queues; + } + dev->dev =get_device (&pdev->dev); quirks |=check_vendor_combination_bug (pdev);
@@ -3679,6 +4136,7 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev, quirks |= qentry->enabled_quirks; quirks &= ~qentry->disabled_quirks; } + ret =nvme_init_ctrl (&dev->ctrl, &pdev->dev, &nvme_pci_ctrl_ops, quirks);if (ret)
@@ -3704,6 +4162,8 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev, out_put_device:put_device (dev->dev); +kfree (dev->active_q_bitmap); +out_free_queues:kfree (dev->queues); out_free_dev:kfree (dev);
@@ -3790,14 +4250,39 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id) goto out_disable; } + result =blocking_notifier_chain_register ( + &nvme_adaptive_coalesce_notifier_list, + &dev->coalesce_notifier); +if (result) + goto out_disable; + dev->coalesce_notifier_registered = true; +pci_set_drvdata (pdev, dev);nvme_start_ctrl (&dev->ctrl);nvme_put_ctrl (&dev->ctrl);flush_work (&dev->ctrl.scan_work); +/* Cache irq coalesce support to avoid admin cmd in notifier callback */ + dev->irq_coalesce_supported =nvme_check_irq_coalesce_supported (&dev->ctrl); +if (dev->irq_coalesce_supported && !adaptive_coalescing_disabled) { +/* +* Schedule the delayed work on the last CPU of the +* NUMA node this NVMe disk belongs to. +*/ +schedule_delayed_work_on (nvme_coalesce_work_cpu (dev), + &dev->coalesce_work, +msecs_to_jiffies (COAL_CHECK_INTERVAL_MS )); + }return 0 ; out_disable: +if (dev->coalesce_notifier_registered) { +blocking_notifier_chain_unregister ( + &nvme_adaptive_coalesce_notifier_list, + &dev->coalesce_notifier); + dev->coalesce_notifier_registered = false; + } +cancel_delayed_work_sync (&dev->coalesce_work);nvme_change_ctrl_state (&dev->ctrl,NVME_CTRL_DELETING );nvme_dev_disable (dev, true);nvme_free_host_mem (dev);
@@ -3853,14 +4338,42 @@ static void nvme_remove(struct pci_dev *pdev) {struct nvme_dev *dev =pci_get_drvdata (pdev); +/* +* Set DELETING state first to fail pending admin commands quickly, +* preventing 60-second timeout stalls during abrupt removal. +*/ nvme_change_ctrl_state (&dev->ctrl,NVME_CTRL_DELETING ); + +/* +* Signal work to stop and unregister notifier early to prevent +* use-after-free: work could rearm itself, or debugfs write could +* trigger notifier callback which submits admin commands after +* admin queue is destroyed. +*/ +WRITE_ONCE (dev->coalesce_work_stop, true); +if (dev->coalesce_notifier_registered) { +blocking_notifier_chain_unregister ( + &nvme_adaptive_coalesce_notifier_list, + &dev->coalesce_notifier); + dev->coalesce_notifier_registered = false; + }pci_set_drvdata (pdev,NULL ); +/* +* Disable device BEFORE cancel_delayed_work_sync() to abort any +* pending admin commands. This prevents 60-second timeout stalls +* during surprise removal: if coalesce_work is blocked waiting +* for nvme_set_irq_coalesce() admin command on unresponsive +* hardware, nvme_dev_disable() will abort it immediately. +*/ if (!pci_device_is_present (pdev)) {nvme_change_ctrl_state (&dev->ctrl,NVME_CTRL_DEAD );nvme_dev_disable (dev, true); } +/* Now safe to sync - any pending admin commands are aborted */ +cancel_delayed_work_sync (&dev->coalesce_work); +flush_work (&dev->ctrl.reset_work);nvme_stop_ctrl (&dev->ctrl);nvme_remove_namespaces (&dev->ctrl);
@@ -4273,14 +4786,37 @@ static struct pci_driver nvme_driver = { .err_handler = &nvme_err_handler, }; +static struct dentry *nvme_dbg_root; +static int __init nvme_init (void ) { +int ret; + +/* +* debugfs is not essential for driver functionality. +* Don't check return values - debugfs API handles errors gracefully. +*/ + nvme_dbg_root =debugfs_create_dir ("nvme_pci" ,NULL ); +debugfs_create_file ("disable_adaptive_interrupt_coalescing" , +0644 , nvme_dbg_root,NULL , + &disable_adaptive_coalescing_fops); +debugfs_create_u32 ("high_io_delta" ,0644 , nvme_dbg_root, + &high_io_delta); +debugfs_create_u32 ("avg_qd_threshold" ,0644 , nvme_dbg_root, + &avg_qd_threshold); +debugfs_create_u32 ("total_inflight_io_threshold" ,0644 , + nvme_dbg_root, &total_inflight_io_threshold); +BUILD_BUG_ON (sizeof (struct nvme_create_cq ) !=64 );BUILD_BUG_ON (sizeof (struct nvme_create_sq ) !=64 );BUILD_BUG_ON (sizeof (struct nvme_delete_queue ) !=64 );BUILD_BUG_ON (IRQ_AFFINITY_MAX_SETS <2 ); -return pci_register_driver (&nvme_driver); + ret =pci_register_driver (&nvme_driver); +if (ret) +debugfs_remove_recursive (nvme_dbg_root); + +return ret; } static void __exit nvme_exit(void)
@@ -4288,6 +4824,8 @@ static void __exit nvme_exit(void)kfree (nvme_pci_quirk_list);pci_unregister_driver (&nvme_driver);flush_workqueue (nvme_wq); + +debugfs_remove_recursive (nvme_dbg_root); }MODULE_AUTHOR ("Matthew Wilcox <[email protected]>" );
Signature · 1 line
2.43.0
Keith Busch
maintainerQuoted 5 lines · Jun Zeng
Add adaptive interrupt coalescing to improve IOPS for high-throughput workloads. Monitor IO pressure periodically and enable/disable coalescing automatically based on IOPS, queue depth, and inflight IO thresholds. This feature is enabled by default, can be controlled through debugfs variable during runtime.
Can't you do all this from user space? You'd maybe need the blk-mq debugfs to get access to the individual hctx dispatch numbers, but maybe that's okay?
The atomic_long_inc where you placed it is especially harmful to polled queues.
Hi Busch,
Thanks for your reply. The main reason for implementing this in kernel space is the lack of a unified, deployable component in user space to support this feature. You see that there are sheer number of OS distributions. This will require separate adaptations for each, usability might be compromised, and-most importantly-ongoing maintenance and updates would be difficult. Meanwhile, nvme driver in kernel space is providing these unified capabilities for different kinds of usage. For the point of using atomic_long_inc in cqe polling we can discuss to do it in a proper way.
-----Original Message-----
From: Keith Busch <[email protected]>
Sent: Friday, July 17, 2026 4:53 AM
To: Zeng, Jun1 <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Shevchenko, Andriy <[email protected]>; Cao, Gang <[email protected]>; Jin, Jun I <[email protected]>; Hu, Yong <[email protected]>; Fang, Liang A <[email protected]>
Subject: Re: [PATCH v1 1/1] nvme-pci: adaptive interrupt coalescing
Quoted 5 lines · Jun Zeng
Add adaptive interrupt coalescing to improve IOPS for high-throughput workloads. Monitor IO pressure periodically and enable/disable coalescing automatically based on IOPS, queue depth, and inflight IO thresholds. This feature is enabled by default, can be controlled through debugfs variable during runtime.
Can't you do all this from user space? You'd maybe need the blk-mq debugfs to get access to the individual hctx dispatch numbers, but maybe that's okay?
The atomic_long_inc where you placed it is especially harmful to polled queues.
Quoted 9 lines · On Fri, Jul 17, 2026 at 04:10:08AM +0300, Zeng, Jun1 wrote:
Hi Busch,
Thanks for your reply. The main reason for implementing this in kernel space is the lack of a unified, deployable component in user space to support this feature. You see that there are sheer number of OS distributions. This will require separate adaptations for each, usability might be compromised, and-most importantly-ongoing maintenance and updates would be difficult. Meanwhile, nvme driver in kernel space is providing these unified capabilities for different kinds of usage. For the point of usingatomic_long_incin cqe polling we can discuss to do it in a proper way.
You need to stop top-posting!
Quoted 17 lines
-----Original Message-----
From: Keith Busch <[email protected]>
Sent: Friday, July 17, 2026 4:53 AM
To: Zeng, Jun1 <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Shevchenko, Andriy <[email protected]>; Cao, Gang <[email protected]>; Jin, Jun I <[email protected]>; Hu, Yong <[email protected]>; Fang, Liang A <[email protected]>
Subject: Re: [PATCH v1 1/1] nvme-pci: adaptive interrupt coalescingQuoted 5 lines · Jun Zeng
Add adaptive interrupt coalescing to improve IOPS for high-throughput workloads. Monitor IO pressure periodically and enable/disable coalescing automatically based on IOPS, queue depth, and inflight IO thresholds. This feature is enabled by default, can be controlled through debugfs variable during runtime.
Can't you do all this from user space? You'd maybe need the blk-mq debugfs to get access to the individual hctx dispatch numbers, but maybe that's okay?
The
atomic_long_incwhere you placed it is especially harmful to polled queues.
Signature · 2 lines
With Best Regards, Andy Shevchenko
Christoph Hellwig
maintainerQuoted 10 lines · Keith Busch
Quoted 5 lines · Jun Zeng
Add adaptive interrupt coalescing to improve IOPS for high-throughput workloads. Monitor IO pressure periodically and enable/disable coalescing automatically based on IOPS, queue depth, and inflight IO thresholds. This feature is enabled by default, can be controlled through debugfs variable during runtime.
Can't you do all this from user space? You'd maybe need the blk-mq debugfs to get access to the individual hctx dispatch numbers, but maybe that's okay?
Having these kinds of interfaces where we rely on a userspace daemon to be in sync with the kernel implementation are a mess, and really lock us out of future improvements in this area.
Quoted 2 lines
The
atomic_long_incwhere you placed it is especially harmful to polled queues.
We really should be able to do this without any counters, as blk-mq keeps more than enough statistics. And of course for polled queues nothing related to interrupt coalescing is actually need.
And we should take a page from the networking playbook and look into adaptively switching to polling under high load instead of just mitigating interrupts. An important part of that is to move the CQ reaping from irq context to thread context.
But modulo all these caveats adaptive interrupt coalescing / moderation is something we absolute need right now and even more so going foward.
Note that a few month ago there also was as an interesting patchset that does global interrupt moderation at the IRQ controller level. I don't remember what happened to it and can't find a link to it, but adding the interrupt maintainer in case he remembers.
Keith Busch
maintainerQuoted 8 lines · Christoph Hellwig
Quoted 7 lines · Keith Busch
Quoted 5 lines · Jun Zeng
Add adaptive interrupt coalescing to improve IOPS for high-throughput workloads. Monitor IO pressure periodically and enable/disable coalescing automatically based on IOPS, queue depth, and inflight IO thresholds. This feature is enabled by default, can be controlled through debugfs variable during runtime.
Quoted 7 lines
Quoted 3 lines
Can't you do all this from user space? You'd maybe need the blk-mq debugfs to get access to the individual hctx dispatch numbers, but maybe that's okay?
Having these kinds of interfaces where we rely on a userspace daemon to be in sync with the kernel implementation are a mess, and really lock us out of future improvements in this area.
We can't stop someone from doing this today, though.
Quoted 11 lines
Quoted 2 lines
The
atomic_long_incwhere you placed it is especially harmful to polled queues.We really should be able to do this without any counters, as blk-mq keeps more than enough statistics. And of course for polled queues nothing related to interrupt coalescing is actually need.
And we should take a page from the networking playbook and look into adaptively switching to polling under high load instead of just mitigating interrupts. An important part of that is to move the CQ reaping from irq context to thread context.
Totally. I had a proposal from I think 7 years ago to converge nvme's threaded IRQ handling: reap the first X entries from hard-irq context, and if there's more, return IRQ_WAKE_THREAD to poll the rest and future completions with that queue's interrupt masked (this is one scenario where NVMe's MSI masking is better than MSI-x, but it's not a big deal). I can't find the patch though, so I guess it was in the gap that infradead lost.
Anyway, I think combine that concept with the isolcpus proposal and you can have full control over which CPUs can dispatch new IO without being interrupted from CPUs reaping completions. Then we shouldn't need the spec's coalescing feature.
Quoted 8 lines
But modulo all these caveats adaptive interrupt coalescing / moderation is something we absolute need right now and even more so going foward.
Note that a few month ago there also was as an interesting patchset that does global interrupt moderation at the IRQ controller level. I don't remember what happened to it and can't find a link to it, but adding the interrupt maintainer in case he remembers.
Hi all,
I'd like to add some information. A few months ago, when I was testing Zeng's solution, I encountered some issues, primarily that enabling or disabling interrupt aggregation on certain disks could cause I/O latency of several tens of milliseconds.
Inspired by the design of network NAPI, I came up with a similar idea and implemented a patch, which performed well in testing. If you're interested in this approach, I can continue to improve it. The reason I didn't move forward with this approach is that our initial goal was to solve the issues encountered with QEMU, but disabling interrupts causes problems in VFIO+QEMU environments, and this approach only works in physical machine scenarios.
The patch estimates how busy a queue is from the submission queue tail and the SQ head reported in completion entries. When the number of pending commands crosses a high watermark, the hard IRQ handles the current completions, masks the queue's interrupt and wakes the IRQ thread. The IRQ thread then polls for more completions. It switches back to interrupts when the pending count falls below a low watermark, or when a bounded number of polling loops is reached. Using separate high and low watermarks avoids switching back and forth too often.
This is only enabled when the controller has separate MSI-X vectors, so masking one queue does not block completions from other queues.
I tested it with four Solidigm NVMe devices using 4K random reads. fio used regular io_uring, without hipri or SQ polling. These are the median results from three runs on a 7.2-rc1 based kernel:
baseline patched IOPS 7.61M 11.41M (+50%) mean latency 267 us 177 us (-34%) hard IRQs per I/O 0.52 0.21 (-60%)
Without the patch, three of the four devices were limited to about 1.49M IOPS each. With the patch, all four devices reached around 2.8M-2.9M IOPS.
The patch is included below.
Thanks,
Fengnan
From 1c84b5afc906e780e1dbd2ee186251f9de2716b5 Mon Sep 17 00:00:00 2001
From: Fengnan Chang <[email protected]>
Date: Tue, 21 Jul 2026 12:31:46 +0800
Subject: [PATCH] nvme-pci: adaptively poll busy queues from threaded
interrupts
On systems with multiple fast NVMe devices, hard IRQ completion processing can become CPU-bound and cap 4K random-read throughput before the devices are saturated.
Add adaptive polling for busy interrupt-driven queues. Track the SQ head reported by completions while the CQE is still owned by the host. After the hard IRQ reaps pending CQEs, wake the IRQ thread and mask the queue interrupt when the number of SQ entries not yet consumed reaches a high watermark. Poll in the IRQ thread until the queue drops below a low watermark or a bounded loop count is reached, then reenable the interrupt.
Only enable adaptive polling when the controller has independent vectors, as masking a shared single vector could suppress unrelated queues. Expose the enable switch, watermarks, loop bound, and sleep interval as module parameters.
--- drivers/nvme/host/pci.c | 212 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 198 insertions(+), 14 deletions(-)
drivers/nvme/host/pci.c +198 -14
@@ -9,6 +9,7 @@#include <linux/blkdev.h> #include <linux/blk-mq-dma.h> #include <linux/blk-integrity.h> +#include <linux/delay.h> #include <linux/dmi.h> #include <linux/init.h> #include <linux/interrupt.h>
@@ -79,8 +80,136 @@ struct quirk_entry {u32 disabled_quirks; }; -static int use_threaded_interrupts; -module_param (use_threaded_interrupts, int,0444 ); +static int use_adaptive_polling; +module_param (use_adaptive_polling, int,0444 ); +MODULE_PARM_DESC (use_adaptive_polling, +"use adaptive completion polling on busy interrupt queues" ); + +static unsigned int adaptive_poll_start_pending =10 ; +static unsigned int adaptive_poll_stop_pending =3 ; +static unsigned int adaptive_poll_max_loops =100 ; +static unsigned int adaptive_poll_sleep_min_us =20 ; +static unsigned int adaptive_poll_sleep_max_us =30 ; + +static int nvme_adaptive_poll_start_pending_set (const char *val,const struct kernel_param *kp) +{ +unsigned int pending; +int ret; + + ret =kstrtouint (val,10 , &pending); +if (ret) +return ret; +if (!pending || pending <READ_ONCE (adaptive_poll_stop_pending)) +return -EINVAL ; + +WRITE_ONCE (*(unsigned int *)kp->arg, pending); +return 0 ; +} + +static const struct kernel_param_ops nvme_adaptive_poll_start_pending_ops = { + .set = nvme_adaptive_poll_start_pending_set, + .get = param_get_uint, +}; + +module_param_cb (adaptive_poll_start_pending, &nvme_adaptive_poll_start_pending_ops, + &adaptive_poll_start_pending,0644 ); +MODULE_PARM_DESC (adaptive_poll_start_pending,"SQ pending high watermark for adaptive polling" ); + +static int nvme_adaptive_poll_stop_pending_set (const char *val,const struct kernel_param *kp) +{ +unsigned int pending; +int ret; + + ret =kstrtouint (val,10 , &pending); +if (ret) +return ret; +if (!pending || pending >READ_ONCE (adaptive_poll_start_pending)) +return -EINVAL ; + +WRITE_ONCE (*(unsigned int *)kp->arg, pending); +return 0 ; +} + +static const struct kernel_param_ops nvme_adaptive_poll_stop_pending_ops = { + .set = nvme_adaptive_poll_stop_pending_set, + .get = param_get_uint, +}; + +module_param_cb (adaptive_poll_stop_pending, &nvme_adaptive_poll_stop_pending_ops, + &adaptive_poll_stop_pending,0644 ); +MODULE_PARM_DESC (adaptive_poll_stop_pending,"SQ pending low watermark for adaptive polling" ); + +static int nvme_adaptive_poll_max_loops_set (const char *val,const struct kernel_param *kp) +{ +unsigned int loops; +int ret; + + ret =kstrtouint (val,10 , &loops); +if (ret) +return ret; +if (!loops) +return -EINVAL ; + +WRITE_ONCE (*(unsigned int *)kp->arg, loops); +return 0 ; +} + +static const struct kernel_param_ops nvme_adaptive_poll_max_loops_ops = { + .set = nvme_adaptive_poll_max_loops_set, + .get = param_get_uint, +}; + +module_param_cb (adaptive_poll_max_loops, &nvme_adaptive_poll_max_loops_ops, + &adaptive_poll_max_loops,0644 ); +MODULE_PARM_DESC (adaptive_poll_max_loops,"Maximum adaptive polling loops" ); + +static int nvme_adaptive_poll_sleep_min_us_set (const char *val,const struct kernel_param *kp) +{ +unsigned int usecs; +int ret; + + ret =kstrtouint (val,10 , &usecs); +if (ret) +return ret; +if (!usecs || usecs >READ_ONCE (adaptive_poll_sleep_max_us)) +return -EINVAL ; + +WRITE_ONCE (*(unsigned int *)kp->arg, usecs); +return 0 ; +} + +static const struct kernel_param_ops nvme_adaptive_poll_sleep_min_us_ops = { + .set = nvme_adaptive_poll_sleep_min_us_set, + .get = param_get_uint, +}; + +module_param_cb (adaptive_poll_sleep_min_us, &nvme_adaptive_poll_sleep_min_us_ops, + &adaptive_poll_sleep_min_us,0644 ); +MODULE_PARM_DESC (adaptive_poll_sleep_min_us,"Minimum adaptive polling delay in microseconds" ); + +static int nvme_adaptive_poll_sleep_max_us_set (const char *val,const struct kernel_param *kp) +{ +unsigned int usecs; +int ret; + + ret =kstrtouint (val,10 , &usecs); +if (ret) +return ret; +if (!usecs || usecs <READ_ONCE (adaptive_poll_sleep_min_us)) +return -EINVAL ; + +WRITE_ONCE (*(unsigned int *)kp->arg, usecs); +return 0 ; +} + +static const struct kernel_param_ops nvme_adaptive_poll_sleep_max_us_ops = { + .set = nvme_adaptive_poll_sleep_max_us_set, + .get = param_get_uint, +}; + +module_param_cb (adaptive_poll_sleep_max_us, &nvme_adaptive_poll_sleep_max_us_ops, + &adaptive_poll_sleep_max_us,0644 ); +MODULE_PARM_DESC (adaptive_poll_sleep_max_us,"Maximum adaptive polling delay in microseconds" );static bool use_cmb_sqes = true;module_param (use_cmb_sqes, bool,0444 );
@@ -377,6 +506,7 @@ struct nvme_queue {u16 cq_vector;u16 sq_tail;u16 last_sq_tail; +u16 sq_head;u16 cq_head;u16 qid;u8 cq_phase;
@@ -1607,7 +1737,7 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq) }static inline bool nvme_poll_cq (struct nvme_queue *nvmeq, -struct io_comp_batch *iob) +struct io_comp_batch *iob,u16 *sq_head) {bool found = false;
@@ -1618,6 +1748,8 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,* the cqe requires a full read memory barrier */ dma_rmb (); +if (sq_head) + *sq_head =le16_to_cpu (nvmeq->cqes[nvmeq->cq_head].sq_head);nvme_handle_cqe (nvmeq, iob, nvmeq->cq_head);nvme_update_cq_head (nvmeq); }
@@ -1631,8 +1763,11 @@ static irqreturn_t nvme_irq(int irq, void *data) {struct nvme_queue *nvmeq = data;DEFINE_IO_COMP_BATCH (iob); +u16 sq_head; + +if (nvme_poll_cq (nvmeq, &iob, &sq_head)) { +WRITE_ONCE (nvmeq->sq_head, sq_head); -if (nvme_poll_cq (nvmeq, &iob)) {if (!rq_list_empty (&iob.req_list))nvme_pci_complete_batch (&iob);return IRQ_HANDLED ;
@@ -1640,13 +1775,56 @@ static irqreturn_t nvme_irq(int irq, void *data)return IRQ_NONE ; } -static irqreturn_t nvme_irq_check (int irq,void *data) +static unsigned int nvme_sq_pending (struct nvme_queue *nvmeq) +{ +int pending =READ_ONCE (nvmeq->sq_tail) -READ_ONCE (nvmeq->sq_head); + +if (pending <0 ) + pending += nvmeq->q_depth; +return pending; +} + +static bool nvme_adaptive_poll_should_start (struct nvme_queue *nvmeq) +{ +return nvme_sq_pending (nvmeq) >=READ_ONCE (adaptive_poll_start_pending); +} + +static bool nvme_adaptive_poll_should_stop (struct nvme_queue *nvmeq) +{ +return nvme_sq_pending (nvmeq) <READ_ONCE (adaptive_poll_stop_pending); +} + +static irqreturn_t nvme_adaptive_poll (int irq,void *data) {struct nvme_queue *nvmeq = data; +unsigned int loops =0 ; +unsigned int max_loops =READ_ONCE (adaptive_poll_max_loops); +unsigned int sleep_min =READ_ONCE (adaptive_poll_sleep_min_us); +unsigned int sleep_max =READ_ONCE (adaptive_poll_sleep_max_us); -if (nvme_cqe_pending (nvmeq)) -return IRQ_WAKE_THREAD ; -return IRQ_NONE ; +do { +if (nvme_cqe_pending (nvmeq)) +nvme_irq (irq, data); +if (nvme_adaptive_poll_should_stop (nvmeq)) +break ; +usleep_range (sleep_min, sleep_max); + }while (++loops < max_loops); + +enable_irq (irq); +return IRQ_HANDLED ; +} + +static irqreturn_t nvme_irq_adaptive_poll (int irq,void *data) +{ +struct nvme_queue *nvmeq = data; +irqreturn_t ret; + + ret =nvme_irq (irq, data); +if (!nvme_adaptive_poll_should_start (nvmeq)) +return ret; + +disable_irq_nosync (irq); +return IRQ_WAKE_THREAD ; }/*
@@ -1663,7 +1841,7 @@ static void nvme_poll_irqdisable(struct nvme_queue *nvmeq) irq =pci_irq_vector (pdev, nvmeq->cq_vector);disable_irq (irq);spin_lock (&nvmeq->cq_poll_lock); -nvme_poll_cq (nvmeq,NULL ); +nvme_poll_cq (nvmeq,NULL ,NULL );spin_unlock (&nvmeq->cq_poll_lock);enable_irq (irq); }
@@ -1678,7 +1856,7 @@ static int nvme_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)return 0 ;spin_lock (&nvmeq->cq_poll_lock); - found =nvme_poll_cq (nvmeq, iob); + found =nvme_poll_cq (nvmeq, iob,NULL );spin_unlock (&nvmeq->cq_poll_lock);return found;
@@ -2075,7 +2253,7 @@ static void nvme_reap_pending_cqes(struct nvme_dev *dev)for (i = dev->ctrl.queue_count -1 ; i >0 ; i--) {spin_lock (&dev->queues[i].cq_poll_lock); -nvme_poll_cq (&dev->queues[i],NULL ); +nvme_poll_cq (&dev->queues[i],NULL ,NULL );spin_unlock (&dev->queues[i].cq_poll_lock); } }
@@ -2171,9 +2349,14 @@ static int queue_request_irq(struct nvme_queue *nvmeq)struct pci_dev *pdev =to_pci_dev (nvmeq->dev->dev);int nr = nvmeq->dev->ctrl.instance; -if (use_threaded_interrupts) { -return pci_request_irq (pdev, nvmeq->cq_vector, nvme_irq_check, - nvme_irq, nvmeq,"nvme%dq%d" , nr, nvmeq->qid); +/* +* Adaptive polling masks the entire vector, so don't use it when +* multiple queues share the single available vector. +*/ +if (use_adaptive_polling && nvmeq->dev->num_vecs >1 ) { +return pci_request_irq (pdev, nvmeq->cq_vector, + nvme_irq_adaptive_poll, nvme_adaptive_poll, nvmeq, +"nvme%dq%d" , nr, nvmeq->qid); }else {return pci_request_irq (pdev, nvmeq->cq_vector, nvme_irq,NULL , nvmeq,"nvme%dq%d" , nr, nvmeq->qid);
@@ -2186,6 +2369,7 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid) nvmeq->sq_tail =0 ; nvmeq->last_sq_tail =0 ; + nvmeq->sq_head =0 ; nvmeq->cq_head =0 ; nvmeq->cq_phase =1 ; nvmeq->q_db = &dev->dbs[qid *2 * dev->db_stride];
Quoted 1 line · Fengnan Chang
The patch is included below.
The above in this case can be moved to cover letter, in this form it's harder for maintainers to apply this.
...
Quoted 22 lines
From
1c84b5afc906e780e1dbd2ee186251f9de2716b5Mon Sep 17 00:00:00 2001
From: Fengnan Chang <[email protected]>
Date: Tue, 21 Jul 2026 12:31:46 +0800
Subject: [PATCH] nvme-pci: adaptively poll busy queues from threaded
interruptsOn systems with multiple fast NVMe devices, hard IRQ completion processing can become CPU-bound and cap 4K random-read throughput before the devices are saturated.
Add adaptive polling for busy interrupt-driven queues. Track the SQ head reported by completions while the CQE is still owned by the host. After the hard IRQ reaps pending CQEs, wake the IRQ thread and mask the queue interrupt when the number of SQ entries not yet consumed reaches a high watermark. Poll in the IRQ thread until the queue drops below a low watermark or a bounded loop count is reached, then reenable the interrupt.
Only enable adaptive polling when the controller has independent vectors, as masking a shared single vector could suppress unrelated queues. Expose the enable switch, watermarks, loop bound, and sleep interval as module parameters.
...
drivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data) · line 1800
Quoted 4 lines
drivers/nvme/host/pci.c +2
-0+static irqreturn_t nvme_adaptive_poll (int irq,void *data) {struct nvme_queue *nvmeq = data; +unsigned int loops =0 ;
Unneeded. See below how.
Quoted 4 lines
drivers/nvme/host/pci.c +3
-0+unsigned int max_loops =READ_ONCE (adaptive_poll_max_loops); +unsigned int sleep_min =READ_ONCE (adaptive_poll_sleep_min_us); +unsigned int sleep_max =READ_ONCE (adaptive_poll_sleep_max_us);
drivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data)
Quoted 10 lines
drivers/nvme/host/pci.c +6
-3-if (nvme_cqe_pending (nvmeq)) -return IRQ_WAKE_THREAD ; -return IRQ_NONE ; + do { + if (nvme_cqe_pending(nvmeq)) + nvme_irq(irq, data); + if (nvme_adaptive_poll_should_stop(nvmeq)) + break; + usleep_range(sleep_min, sleep_max);
fsleep() ? This automatically makes the upper bound ~25% of the requested value. Supplying max in this situation makes not much sense.
drivers/nvme/host/pci.c static irqreturn_t nvme_irq(int irq, void *data) · line 1811
Quoted 1 line
drivers/nvme/host/pci.c +1
-0+ }while (++loops < max_loops);
} while (--max_loops);
Quoted 3 lines
drivers/nvme/host/pci.c +3
-0+enable_irq (irq); +return IRQ_HANDLED ; +}
Signature · 2 lines
With Best Regards, Andy Shevchenko
Keith Busch
maintainerdrivers/nvme/host/pci.c static inline bool nvme_poll_cq(struct nvme_queue *nvmeq, · line 1752
Quoted 6 lines · Fengnan Chang
drivers/nvme/host/pci.c +2
-0@@ -1618,6 +1748,8 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,* the cqe requires a full read memory barrier */ dma_rmb (); +if (sq_head) + *sq_head =le16_to_cpu (nvmeq->cqes[nvmeq->cq_head].sq_head);
This is the wrong criteria to determine the outstanding depth. The spec allows the controller to move this forward after it has read an entry. The commands may still be in progress, so the depth of future completions to expect can't depend on this value.
But in general, I agree with the spirit of where this is going.
For MSI, you can make use of the INTMS/INTMC NVMe registers to make this even more efficient. Your patch disables the irq at the cpU level, but the device is still emitting those messages for no reason.
Quoted 7 lines
From: "Keith Busch"<[email protected]>
Date: Tue, Jul 21, 2026, 23:11
Subject: Re: [PATCH v1 1/1] nvme-pci: adaptive interrupt coalescing
To: "Fengnan Chang"<[email protected]>
Cc: <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]> On Tue, Jul 21, 2026 at 04:37:39PM +0800, Fengnan Chang wrote:
Quoted 13 lines
drivers/nvme/host/pci.c static inline bool nvme_poll_cq(struct nvme_queue *nvmeq, · line 1752
Quoted 6 lines
drivers/nvme/host/pci.c +2
-0@@ -1618,6 +1748,8 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,* the cqe requires a full read memory barrier */ dma_rmb(); + if (sq_head) + *sq_head = le16_to_cpu(nvmeq->cqes[nvmeq->cq_head].sq_head);
This is the wrong criteria to determine the outstanding depth. The spec allows the controller to move this forward after it has read an entry. The commands may still be in progress, so the depth of future completions to expect can't depend on this value.
But in general, I agree with the spirit of where this is going.
Thanks for your feedback. I’d like to hear more opinions from others. If this is the right direction, I’ll continue to improve this patch.
Hi Jens, Christoph, Sagi:
what do you think?
Thanks.
Fengnan.
Quoted 5 lines
For MSI, you can make use of the INTMS/INTMC NVMe registers to make this even more efficient. Your patch disables the irq at the cpU level, but the device is still emitting those messages for no reason.
Christoph Hellwig
maintainerQuoted 21 lines · changfengnan
Quoted 7 lines
From: "Keith Busch"<[email protected]>
Date: Tue, Jul 21, 2026, 23:11
Subject: Re: [PATCH v1 1/1] nvme-pci: adaptive interrupt coalescing
To: "Fengnan Chang"<[email protected]>
Cc: <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]> On Tue, Jul 21, 2026 at 04:37:39PM +0800, Fengnan Chang wrote:Quoted 13 lines
drivers/nvme/host/pci.c static inline bool nvme_poll_cq(struct nvme_queue *nvmeq, · line 1752
Quoted 6 lines
drivers/nvme/host/pci.c +2
-0@@ -1618,6 +1748,8 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,* the cqe requires a full read memory barrier */ dma_rmb(); + if (sq_head) + *sq_head = le16_to_cpu(nvmeq->cqes[nvmeq->cq_head].sq_head);
This is the wrong criteria to determine the outstanding depth. The spec allows the controller to move this forward after it has read an entry. The commands may still be in progress, so the depth of future completions to expect can't depend on this value.
But in general, I agree with the spirit of where this is going.
Quoted 5 lines
Thanks for your feedback. I’d like to hear more opinions from others. If this is the right direction, I’ll continue to improve this patch.
Hi Jens, Christoph, Sagi:
what do you think?
Yes, this is a useful idea, please continue with it if you can.
Quoted 5 lines · Jun Zeng
The IOPS for small IO can be significantly improved when enabling interrupt coalescing in this case. However, in some cases (e.g., big IO for bandwidth or latency-sensitive tests with small IO and small queue depth), interrupt coalescing has side effects.
Quoted 7 lines
Solution
========Monitor the IOPS and average queue depth of inflight IO periodically on each NVMe drive and enable/disable interrupt coalescing automatically without user intervention. This feature is enabled by default and can be disabled/enabled through debugfs during runtime.
An algorithm like the above can be implemented more efficiently in NVMe controller hardware than in software, isn't it?
Thanks,
Bart.
Quoted 9 lines
Quoted 9 lines · Jun Zeng
The IOPS for small IO can be significantly improved when enabling interrupt coalescing in this case. However, in some cases (e.g., big IO for bandwidth or latency-sensitive tests with small IO and small queue depth), interrupt coalescing has side effects.
Solution
========
Quoted 6 lines
Quoted 4 lines
Monitor the IOPS and average queue depth of inflight IO periodically on each NVMe drive and enable/disable interrupt coalescing automatically without user intervention. This feature is enabled by default and can be disabled/enabled through debugfs during runtime.
An algorithm like the above can be implemented more efficiently in NVMe controller hardware than in software, isn't it?
Theoretically, this can be implemented in the NVMe controller's firmware, but we have no control over the strategies and decisions of NVMe vendors. We are simply making the best possible optimizations at the software level.
BR,
Jun
Keith Busch
maintainerQuoted 14 lines · On Mon, Jul 20, 2026 at 05:24:12AM +0000, Zeng, Jun1 wrote:
Quoted 13 lines
Quoted 13 lines · Jun Zeng
The IOPS for small IO can be significantly improved when enabling interrupt coalescing in this case. However, in some cases (e.g., big IO for bandwidth or latency-sensitive tests with small IO and small queue depth), interrupt coalescing has side effects.
Solution
========Monitor the IOPS and average queue depth of inflight IO periodically on each NVMe drive and enable/disable interrupt coalescing automatically without user intervention. This feature is enabled by default and can be disabled/enabled through debugfs during runtime.
Quoted 6 lines
Quoted 1 line
An algorithm like the above can be implemented more efficiently in NVMe controller hardware than in software, isn't it?
Theoretically, this can be implemented in the NVMe controller's firmware, but we have no control over the strategies and decisions of NVMe vendors. We are simply making the best possible optimizations at the software level.
But the coalescing feature as it is is too coarse. There is/was some proposal going through the nvme technical workgroup to improve it. I really haven't had any time to participate in the standards work to say if it's any better, but that's probably the forum to improve such things across vendors.
I'm not sure what concern you have about distro support with respect to just controlling this from user space. AFAIK, everything usable has sysfs, and you can't stop passthrough ioctls, so everything you need should be there.
Christoph Hellwig
maintainerQuoted 7 lines · Bart Van Assche
Quoted 4 lines
Monitor the IOPS and average queue depth of inflight IO periodically on each NVMe drive and enable/disable interrupt coalescing automatically without user intervention. This feature is enabled by default and can be disabled/enabled through debugfs during runtime.
An algorithm like the above can be implemented more efficiently in NVMe controller hardware than in software, isn't it?
There is a lot of discussion on interrupt coalescing in the NVMe technical working group right now. I would suggest everyone working for a member company (and that should be everyone one the cc list here) to check that out before rushing anything.