Skip to contentSkip to conversation

aoe: clean device rq_list in aoedev_downdev()

PATCH Applied Justin Sanders 3 participants 4 messages Jun 10, 2025 – Jun 17, 2025

Subsystems (2)
Block layer ATA over Ethernet (aoe) driver
lore
Share & tools
lore

Justin Sanders

author
#0 lore raw

PATCH 1/2 aoe: clean device rq_list in aoedev_downdev()

An aoe device's rq_list contains accepted block requests that are waiting to be transmitted to the aoe target. This queue was added as part of the conversion to blk_mq. However, the queue was not cleaned out when an aoe device is downed which caused blk_mq_freeze_queue() to sleep indefinitely waiting for those requests to complete, causing a hang. This fix cleans out the queue before calling blk_mq_freeze_queue().

Link https://bugzilla.kernel.org/show_bug.cgi?id=212665
Fixes 3582dd291788 ("aoe: convert aoeblk to blk-mq")
Signed-off-by Justin Sanders <[email protected]>
---
 drivers/block/aoe/aoedev.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/block/aoe/aoedev.c +8 -0
index bba05f0c5bbd..edd4bae3b5a9 100644
@@ -198,6 +198,7 @@ aoedev_downdev(struct aoedev *d)
 {
 	struct aoetgt *t, **tt, **te;
 	struct list_head *head, *pos, *nx;
+	struct request *rq, *rqnext;
 	int i;
 
 	d->flags &= ~DEVFL_UP;

@@ -223,6 +224,13 @@ aoedev_downdev(struct aoedev *d)
 	/* clean out the in-process request (if any) */
 	aoe_failip(d);
 
+	/* clean out any queued block requests */
+	list_for_each_entry_safe(rq, rqnext, &d->rq_list, queuelist) {
+		list_del_init(&rq->queuelist);
+		blk_mq_start_request(rq);
+		blk_mq_end_request(rq, BLK_STS_IOERR);
+	}
+
 	/* fast fail all pending I/O */
 	if (d->blkq) {
 		/* UP is cleared, freeze+quiesce to insure all are errored */
Signature · 1 line
2.49.0
author
#1 lore raw

PATCH 2/2 aoe: defer rexmit timer downdev work to workqueue

When aoe's rexmit_timer() notices that an aoe target fails to respond to commands for more than aoe_deadsecs, it calls aoedev_downdev() which cleans the outstanding aoe and block queues. This can involve sleeping, such as in blk_mq_freeze_queue(), which should not occur in irq context.

This patch defers that aoedev_downdev() call to the aoe device's workqueue.

Link https://bugzilla.kernel.org/show_bug.cgi?id=212665
Signed-off-by Justin Sanders <[email protected]>
---
 drivers/block/aoe/aoe.h    | 1 +
 drivers/block/aoe/aoecmd.c | 8 ++++++--
 drivers/block/aoe/aoedev.c | 5 ++++-
 3 files changed, 11 insertions(+), 3 deletions(-)
drivers/block/aoe/aoe.h +1 -0
index 749ae1246f4c..d35caa3c69e1 100644
@@ -80,6 +80,7 @@ enum {
 	DEVFL_NEWSIZE = (1<<6),	/* need to update dev size in block layer */
 	DEVFL_FREEING = (1<<7),	/* set when device is being cleaned up */
 	DEVFL_FREED = (1<<8),	/* device has been cleaned up */
+	DEVFL_DEAD = (1<<9),	/* device has timed out of aoe_deadsecs */
 };
 
 enum {
drivers/block/aoe/aoecmd.c +6 -2
index 50cc90f6ab35..6298f8e271e3 100644
@@ -754,7 +754,7 @@ rexmit_timer(struct timer_list *timer)
 
 	utgts = count_targets(d, NULL);
 
-	if (d->flags & DEVFL_TKILL) {
+	if (d->flags & (DEVFL_TKILL | DEVFL_DEAD)) {
 		spin_unlock_irqrestore(&d->lock, flags);
 		return;
 	}

@@ -786,7 +786,8 @@ rexmit_timer(struct timer_list *timer)
 			 * to clean up.
 			 */
 			list_splice(&flist, &d->factive[0]);
-			aoedev_downdev(d);
+			d->flags |= DEVFL_DEAD;
+			queue_work(aoe_wq, &d->work);
 			goto out;
 		}
 

@@ -898,6 +899,9 @@ aoecmd_sleepwork(struct work_struct *work)
 {
 	struct aoedev *d = container_of(work, struct aoedev, work);
 
+	if (d->flags & DEVFL_DEAD)
+		aoedev_downdev(d);
+
 	if (d->flags & DEVFL_GDALLOC)
 		aoeblk_gdalloc(d);
drivers/block/aoe/aoedev.c +4 -1
index edd4bae3b5a9..3a240755045b 100644
@@ -200,8 +200,11 @@ aoedev_downdev(struct aoedev *d)
 	struct list_head *head, *pos, *nx;
 	struct request *rq, *rqnext;
 	int i;
+	unsigned long flags;
 
-	d->flags &= ~DEVFL_UP;
+	spin_lock_irqsave(&d->lock, flags);
+	d->flags &= ~(DEVFL_UP | DEVFL_DEAD);
+	spin_unlock_irqrestore(&d->lock, flags);
 
 	/* clean out active and to-be-retransmitted buffers */
 	for (i = 0; i < NFACTIVE; i++) {
Signature · 1 line
2.49.0
maintainer Applied
#2 lore raw
In reply to Justin Sanders
Quoted 8 lines · Justin Sanders

An aoe device's rq_list contains accepted block requests that are waiting to be transmitted to the aoe target. This queue was added as part of the conversion to blk_mq. However, the queue was not cleaned out when an aoe device is downed which caused blk_mq_freeze_queue() to sleep indefinitely waiting for those requests to complete, causing a hang. This fix cleans out the queue before calling blk_mq_freeze_queue().

[...]

Applied, thanks!

[1/2] aoe: clean device rq_list in aoedev_downdev()

      commit: a847c4a41630b38136e069aad82dd619c03e95b6
[2/2] aoe: defer rexmit timer downdev work to workqueue
      commit: 71437cf6208c63af6ba99cb42074d13d7b56b669

Best regards,

Signature · 1 line
Jens Axboe
Tested-by on v1 1/2
#3 lore raw
In reply to Justin Sanders
Quoted 10 lines · Justin Sanders

An aoe device's rq_list contains accepted block requests that are waiting to be transmitted to the aoe target. This queue was added as part of the conversion to blk_mq. However, the queue was not cleaned out when an aoe device is downed which caused blk_mq_freeze_queue() to sleep indefinitely waiting for those requests to complete, causing a hang. This fix cleans out the queue before calling blk_mq_freeze_queue().

Link https://bugzilla.kernel.org/show_bug.cgi?id=212665
Fixes 3582dd291788 ("aoe: convert aoeblk to blk-mq")
Signed-off-by Justin Sanders <[email protected]>

Thank you very much for the patches to fix this issue.
We have tested them in our environment and can confirm that they work as expected.

Tested-by Valentin Kleibel <[email protected]>

Best Regards,
Valentin