iommu/riscv: Fix command queue publishing races
Subsystems (3)
v2 cover iommu/riscv: Fix command queue publishing races
From: Fangyu Yu <[email protected]>
This series fixes races in the RISC-V IOMMU command queue submission path.
The current command queue code reserves a producer index before the command is written and before the hardware tail is updated. Multiple CPUs can therefore reserve different producer indexes concurrently and then wait for the software tail to reach their index before publishing their own command.
That model is fragile when a command submission fails after reserving an index. The failed producer index is never published, so later submitters can wait behind a hole in the software producer stream. This can also lead to misleading IOFENCE.C completion timeouts, because the sync path may wait for a producer index whose command was never actually enqueued.
Fix this by serializing command queue publishing with a raw spinlock. Instead of reserving producer indexes ahead of time, the submission path uses the current software tail as the next command index, writes the command, publishes the hardware tail, and then advances the software tail while holding the lock. When the command queue is full, the code drops the lock and waits for hardware consumption before retrying, so other CPUs are not blocked behind a long hardware poll.
The final patch changes the queue submission helper to return an error when enqueue fails. The IOFENCE.C sync path then avoids waiting for a command that was never published to hardware.
This series does not attempt to add full RAS/error recovery for command queue failures. It keeps the existing local error reporting behavior and only fixes the software queue state and wait semantics.
---
Changes in v2:
- Avoid reacquiring the command queue lock after waiting for space times out.
- Link to v1:
https://lore.kernel.org/linux-riscv/[email protected]/
Fangyu Yu (3):
iommu/riscv: Add command queue lock iommu/riscv: Serialize command queue publishing iommu/riscv: Avoid waiting on failed command enqueue
drivers/iommu/riscv/iommu.c | 117 +++++++++++++++++++++--------------- drivers/iommu/riscv/iommu.h | 2 + 2 files changed, 72 insertions(+), 47 deletions(-)
Signature · 1 line
2.50.1
PATCH v2 1/3 iommu/riscv: Add command queue lock
From: Fangyu Yu <[email protected]>
Add a raw spinlock to the RISC-V IOMMU queue state so command queue publishing can be serialized by a later change.
--- drivers/iommu/riscv/iommu.c | 1 + drivers/iommu/riscv/iommu.h | 2 ++ 2 files changed, 3 insertions(+)
drivers/iommu/riscv/iommu.c +1 -0
@@ -1560,6 +1560,7 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu)int rc;RISCV_IOMMU_QUEUE_INIT (&iommu->cmdq,CQ ); +raw_spin_lock_init (&iommu->cmdq.lock);RISCV_IOMMU_QUEUE_INIT (&iommu->fltq,FQ ); rc =riscv_iommu_init_check (iommu);
drivers/iommu/riscv/iommu.h +2 -0
@@ -12,6 +12,7 @@#define _RISCV_IOMMU_H_#include <linux/iommu.h> +#include <linux/spinlock.h> #include <linux/types.h> #include <linux/iopoll.h>
@@ -23,6 +24,7 @@ struct riscv_iommu_queue {atomic_t prod;/* unbounded producer allocation index */ atomic_t head;/* unbounded shadow ring buffer consumer index */ atomic_t tail;/* unbounded shadow ring buffer producer index */ +raw_spinlock_t lock;/* serialize queue publishing */ unsigned int mask;/* index mask, queue length - 1 */ unsigned int irq;/* allocated interrupt number */ struct riscv_iommu_device *iommu;/* iommu device handling the queue when active */
Signature · 1 line
2.50.1
Re: [PATCH v2 1/3] iommu/riscv: Add command queue lock
Quoted 7 lines · On 9/1/2026 9:39 PM, [email protected] wrote:
From: Fangyu Yu <[email protected]>
Add a raw spinlock to the RISC-V IOMMU queue state so command queue publishing can be serialized by a later change.
Fixes 856c0cfe5c5f ("iommu/riscv: Command and fault queue support")Signed-off-by Fangyu Yu <[email protected]>
Thanks,
Nutty
Quoted 37 lines
--- drivers/iommu/riscv/iommu.c | 1 + drivers/iommu/riscv/iommu.h | 2 ++ 2 files changed, 3 insertions(+)drivers/iommu/riscv/iommu.c +1
-0@@ -1560,6 +1560,7 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu)int rc;RISCV_IOMMU_QUEUE_INIT (&iommu->cmdq,CQ ); +raw_spin_lock_init (&iommu->cmdq.lock);RISCV_IOMMU_QUEUE_INIT (&iommu->fltq,FQ ); rc =riscv_iommu_init_check (iommu);drivers/iommu/riscv/iommu.h +2
-0@@ -12,6 +12,7 @@#define _RISCV_IOMMU_H_#include <linux/iommu.h> +#include <linux/spinlock.h> #include <linux/types.h> #include <linux/iopoll.h> @@ -23,6 +24,7 @@ struct riscv_iommu_queue {atomic_t prod;/* unbounded producer allocation index */ atomic_t head;/* unbounded shadow ring buffer consumer index */ atomic_t tail;/* unbounded shadow ring buffer producer index */ +raw_spinlock_t lock;/* serialize queue publishing */ unsigned int mask;/* index mask, queue length - 1 */ unsigned int irq;/* allocated interrupt number */ struct riscv_iommu_device *iommu;/* iommu device handling the queue when active */
PATCH v2 2/3 iommu/riscv: Serialize command queue publishing
In reply to [email protected]From: Fangyu Yu <[email protected]>
Serialize command queue publishing so software producer state advances only after a command is written and the hardware tail is updated. Wait for hardware consumption outside the queue lock when the command queue is full so other CPUs are not blocked behind a long poll.
--- drivers/iommu/riscv/iommu.c | 100 +++++++++++++++++++++--------------- 1 file changed, 58 insertions(+), 42 deletions(-)
drivers/iommu/riscv/iommu.c +58 -42
@@ -382,77 +382,93 @@ static int riscv_iommu_queue_wait(struct riscv_iommu_queue *queue, (int)(cons - index) > 0, 0, timeout_us); } -/* Enqueue an entry and wait to be processed if timeout_us > 0 -* -* Error handling for IOMMU hardware not responding in reasonable time -* will be added as separate patch series along with other RAS features. -* For now, only report hardware failure and continue. -*/ +static int riscv_iommu_queue_wait_for_space (struct riscv_iommu_queue *queue, +unsigned int last) +{ +unsigned int head; +unsigned int tail; +unsigned int hw_head; +unsigned long flags; +int ret; + + ret =riscv_iommu_readl_timeout (queue->iommu,Q_HEAD (queue), hw_head, + !(hw_head & ~queue->mask) && hw_head != last, +0 ,RISCV_IOMMU_QUEUE_TIMEOUT ); +if (ret) +return ret; + +raw_spin_lock_irqsave (&queue->lock, flags); + head =atomic_read (&queue->head); + tail =atomic_read (&queue->tail); +if ((tail - head) >= queue->mask) { + last =Q_ITEM (queue, head); +/* +* Re-read hw_head under the lock so that it is consistent with +* the freshly computed 'last'. Using the pre-lock snapshot +* could produce a stale value that wraps around relative to the +* new 'last', advancing the shadow head past entries that have +* not yet been consumed by the hardware. +*/ + hw_head =riscv_iommu_readl (queue->iommu,Q_HEAD (queue)); +if (!(hw_head & ~queue->mask) && hw_head != last) +atomic_add ((hw_head - last) & queue->mask, &queue->head); + } +raw_spin_unlock_irqrestore (&queue->lock, flags); + +return 0 ; +} + +/* Enqueue an entry and publish it to the hardware queue. */ static unsigned int riscv_iommu_queue_send (struct riscv_iommu_queue *queue,void *entry,size_t entry_size) {unsigned int prod;unsigned int head; -unsigned int tail;unsigned long flags; +int ret; -/* Do not preempt submission flow. */ -local_irq_save (flags); +/* 1. Wait for space availability and reserve the next slot. */ +for (;;) { +raw_spin_lock_irqsave (&queue->lock, flags); -/* 1. Allocate some space in the queue */ - prod =atomic_inc_return (&queue->prod) -1 ; - head =atomic_read (&queue->head); + prod =atomic_read (&queue->tail); + head =atomic_read (&queue->head); -/* 2. Wait for space availability. */ -if ((prod - head) > queue->mask) { -if (readx_poll_timeout (atomic_read, &queue->head, - head, (prod - head) < queue->mask, -0 ,RISCV_IOMMU_QUEUE_TIMEOUT )) - goto err_busy; - }else if ((prod - head) == queue->mask) { -const unsigned int last =Q_ITEM (queue, head); +if ((prod - head) < queue->mask) +break ; + + head =Q_ITEM (queue, head); +raw_spin_unlock_irqrestore (&queue->lock, flags); -if (riscv_iommu_readl_timeout (queue->iommu,Q_HEAD (queue), head, - !(head & ~queue->mask) && head != last, -0 ,RISCV_IOMMU_QUEUE_TIMEOUT )) + ret =riscv_iommu_queue_wait_for_space (queue, head); +if (ret) goto err_busy; -atomic_add ((head - last) & queue->mask, &queue->head); } -/* 3. Store entry in the ring buffer */ +/* 2. Store entry in the ring buffer. */ memcpy (queue->base +Q_ITEM (queue, prod) * entry_size, entry, entry_size); -/* 4. Wait for all previous entries to be ready */ -if (readx_poll_timeout (atomic_read, &queue->tail, tail, prod == tail, -0 ,RISCV_IOMMU_QUEUE_TIMEOUT )) - goto err_busy; - -/* -* 5. Make sure the ring buffer update (whether in normal or I/O memory) is -* completed and visible before signaling the tail doorbell to fetch -* the next command. 'fence ow, ow' -*/ +/* 3. Make sure the entry is visible before updating the queue tail. */ dma_wmb ();riscv_iommu_writel (queue->iommu,Q_TAIL (queue),Q_ITEM (queue, prod +1 ));/* -* 6. Make sure the doorbell write to the device has finished before updating -* the shadow tail index in normal memory. 'fence o, w' +* 4. Make sure the doorbell write to the device has finished before +* updating the shadow tail index in normal memory. 'fence o, w' */ #ifdef CONFIG_MMIOWB mmiowb ();#endif -atomic_inc (&queue->tail); +atomic_set (&queue->tail, prod +1 ); +atomic_set (&queue->prod, prod +1 ); -/* 7. Complete submission and restore local interrupts */ -local_irq_restore (flags); +raw_spin_unlock_irqrestore (&queue->lock, flags);return prod; err_busy: -local_irq_restore (flags); +/* Report the failure and continue; full RAS recovery is not implemented. */ dev_err_once (queue->iommu->dev,"Hardware error: command enqueue failed\n" ); -return prod; }
Signature · 1 line
2.50.1
PATCH v2 3/3 iommu/riscv: Avoid waiting on failed command enqueue
In reply to [email protected]From: Fangyu Yu <[email protected]>
Do not wait for IOFENCE.C completion when the command failed to enter the queue. The command was not published to hardware, so waiting for its producer index can only report a misleading execution timeout.
--- drivers/iommu/riscv/iommu.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-)
drivers/iommu/riscv/iommu.c +13 -7
@@ -419,8 +419,9 @@ static int riscv_iommu_queue_wait_for_space(struct riscv_iommu_queue *queue, }/* Enqueue an entry and publish it to the hardware queue. */ -static unsigned int riscv_iommu_queue_send (struct riscv_iommu_queue *queue, -void *entry,size_t entry_size) +static int riscv_iommu_queue_send (struct riscv_iommu_queue *queue, +void *entry,size_t entry_size, +unsigned int *out_prod) {unsigned int prod;unsigned int head;
@@ -462,14 +463,16 @@ static unsigned int riscv_iommu_queue_send(struct riscv_iommu_queue *queue,atomic_set (&queue->tail, prod +1 );atomic_set (&queue->prod, prod +1 ); -raw_spin_unlock_irqrestore (&queue->lock, flags); +if (out_prod) + *out_prod = prod; -return prod; +raw_spin_unlock_irqrestore (&queue->lock, flags); +return 0 ; err_busy:/* Report the failure and continue; full RAS recovery is not implemented. */ dev_err_once (queue->iommu->dev,"Hardware error: command enqueue failed\n" ); -return prod; +return ret; }/*
@@ -508,7 +511,7 @@ static irqreturn_t riscv_iommu_cmdq_process(int irq, void *data)static void riscv_iommu_cmd_send (struct riscv_iommu_device *iommu,struct riscv_iommu_command *cmd) { -riscv_iommu_queue_send (&iommu->cmdq, cmd,sizeof (*cmd)); +riscv_iommu_queue_send (&iommu->cmdq, cmd,sizeof (*cmd),NULL ); }/* Send IOFENCE.C command and wait for all scheduled commands to complete. */
@@ -517,9 +520,12 @@ static void riscv_iommu_cmd_sync(struct riscv_iommu_device *iommu, {struct riscv_iommu_command cmd;unsigned int prod; +int ret;riscv_iommu_cmd_iofence (&cmd); - prod =riscv_iommu_queue_send (&iommu->cmdq, &cmd,sizeof (cmd)); + ret =riscv_iommu_queue_send (&iommu->cmdq, &cmd,sizeof (cmd), &prod); +if (ret) +return ;if (!timeout_us)return ;
Signature · 1 line
2.50.1
Quoted 6 lines · On Tue, Sep 01, 2026 at 09:39:17PM +0800, [email protected] wrote:
Changes in v2:
- Avoid reacquiring the command queue lock after waiting for space times out.
- Link to v1:
https://lore.kernel.org/linux-riscv/[email protected]/
Quoted 8 lines
Fangyu Yu (3):
iommu/riscv: Add command queue lock iommu/riscv: Serialize command queue publishing iommu/riscv: Avoid waiting on failed command enqueuedrivers/iommu/riscv/iommu.c | 117 +++++++++++++++++++++--------------- drivers/iommu/riscv/iommu.h | 2 + 2 files changed, 72 insertions(+), 47 deletions(-)
Queued for -rc, thanks.