Skip to contentSkip to conversation

ide: Increment BB in-flight counter for TRIM BH

PATCH In review Hanna Reitz 3 participants 8 messages Jan 20, 2022 – Feb 21, 2022

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 ide: Increment BB in-flight counter for TRIM BH

When we still have an AIOCB registered for DMA operations, we try to settle the respective operation by draining the BlockBackend associated with the IDE device.

However, this assumes that every DMA operation is associated with an increment of the BlockBackend’s in-flight counter (e.g. through some ongoing I/O operation), so that draining the BB until its in-flight counter reaches 0 will settle all DMA operations. That is not the case:
For TRIM, the guest can issue a zero-length operation that will not result in any I/O operation forwarded to the BlockBackend, and also not increment the in-flight counter in any other way. In such a case, blk_drain() will be a no-op if no other operations are in flight.

It is clear that if blk_drain() is a no-op, the value of s->bus->dma->aiocb will not change between checking it in the if condition and asserting that it is NULL after blk_drain().

The particular problem is that ide_issue_trim() creates a BH (ide_trim_bh_cb()) to settle the TRIM request: iocb->common.cb() is ide_dma_cb(), which will either create a new request, or find the transfer to be done and call ide_set_inactive(), which clears s->bus->dma->aiocb. Therefore, the blk_drain() must wait for ide_trim_bh_cb() to run, which currently it will not always do.

To fix this issue, we increment the BlockBackend's in-flight counter when the TRIM operation begins (in ide_issue_trim(), when the ide_trim_bh_cb() BH is created) and decrement it when ide_trim_bh_cb() is done.

---
v1:
https://lists.nongnu.org/archive/html/qemu-block/2022-01/msg00024.html

v2:
- Increment BB’s in-flight counter while the BH is active so that blk_drain() will poll until the BH is done, as suggested by Paolo

(No git-backport-diff, because this patch was basically completely rewritten, so it wouldn’t be worth it.)

---
 hw/ide/core.c | 7 +++++++
 1 file changed, 7 insertions(+)
hw/ide/core.c +7 -0
index e28f8aad61..15138225be 100644
@@ -433,12 +433,16 @@ static const AIOCBInfo trim_aiocb_info = {
 static void ide_trim_bh_cb(void *opaque)
 {
     TrimAIOCB *iocb = opaque;
+    BlockBackend *blk = iocb->s->blk;
 
     iocb->common.cb(iocb->common.opaque, iocb->ret);
 
     qemu_bh_delete(iocb->bh);
     iocb->bh = NULL;
     qemu_aio_unref(iocb);
+
+    /* Paired with an increment in ide_issue_trim() */
+    blk_dec_in_flight(blk);
 }
 
 static void ide_issue_trim_cb(void *opaque, int ret)
@@ -508,6 +512,9 @@ BlockAIOCB *ide_issue_trim(
     IDEState *s = opaque;
     TrimAIOCB *iocb;
 
+    /* Paired with a decrement in ide_trim_bh_cb() */
+    blk_inc_in_flight(s->blk);
+
     iocb = blk_aio_get(&trim_aiocb_info, s->blk, cb, cb_opaque);
     iocb->s = s;
     iocb->bh = qemu_bh_new(ide_trim_bh_cb, iocb);
Signature · 1 line
2.34.1
Reviewed-by Tested-by on v2 1/1
#1 lore raw
Quoted 79 lines · Hanna Reitz

When we still have an AIOCB registered for DMA operations, we try to settle the respective operation by draining the BlockBackend associated with the IDE device.

However, this assumes that every DMA operation is associated with an increment of the BlockBackend’s in-flight counter (e.g. through some ongoing I/O operation), so that draining the BB until its in-flight counter reaches 0 will settle all DMA operations. That is not the case:
For TRIM, the guest can issue a zero-length operation that will not result in any I/O operation forwarded to the BlockBackend, and also not increment the in-flight counter in any other way. In such a case, blk_drain() will be a no-op if no other operations are in flight.

It is clear that if blk_drain() is a no-op, the value of s->bus->dma->aiocb will not change between checking it in the if condition and asserting that it is NULL after blk_drain().

The particular problem is that ide_issue_trim() creates a BH (ide_trim_bh_cb()) to settle the TRIM request: iocb->common.cb() is ide_dma_cb(), which will either create a new request, or find the transfer to be done and call ide_set_inactive(), which clears s->bus->dma->aiocb. Therefore, the blk_drain() must wait for ide_trim_bh_cb() to run, which currently it will not always do.

To fix this issue, we increment the BlockBackend's in-flight counter when the TRIM operation begins (in ide_issue_trim(), when the ide_trim_bh_cb() BH is created) and decrement it when ide_trim_bh_cb() is done.

---
v1:
https://lists.nongnu.org/archive/html/qemu-block/2022-01/msg00024.html

v2:
- Increment BB’s in-flight counter while the BH is active so that blk_drain() will poll until the BH is done, as suggested by Paolo

(No git-backport-diff, because this patch was basically completely rewritten, so it wouldn’t be worth it.)

---
 hw/ide/core.c | 7 +++++++
 1 file changed, 7 insertions(+)
hw/ide/core.c +7 -1
index e28f8aad61..15138225be 100644
@@ -433,12 +433,16 @@ static const AIOCBInfo trim_aiocb_info = {
 static void ide_trim_bh_cb(void *opaque)
 {
     TrimAIOCB *iocb = opaque;
+    BlockBackend *blk = iocb->s->blk;
 
     iocb->common.cb(iocb->common.opaque, iocb->ret);
 
     qemu_bh_delete(iocb->bh);
     iocb->bh = NULL;
     qemu_aio_unref(iocb);
+
+    /* Paired with an increment in ide_issue_trim() */
+    blk_dec_in_flight(blk);
 }
 
 static void ide_issue_trim_cb(void *opaque, int ret)
@@ -508,6 +512,9 @@ BlockAIOCB *ide_issue_trim(
     IDEState *s = opaque;
     TrimAIOCB *iocb;
 
+    /* Paired with a decrement in ide_trim_bh_cb() */
+    blk_inc_in_flight(s->blk);
+
     iocb = blk_aio_get(&trim_aiocb_info, s->blk, cb, cb_opaque);
     iocb->s = s;
     iocb->bh = qemu_bh_new(ide_trim_bh_cb, iocb);
--
Quoted 2 lines

2.34.1

Oh, this *wasn't* the same bug I thought it was.

There's no regression test, but I will trust you (and Paolo) that this solves the bug you were seeing. It makes sense.

#2 lore raw
Quoted 83 lines · John Snow
Quoted 78 lines · Hanna Reitz

When we still have an AIOCB registered for DMA operations, we try to settle the respective operation by draining the BlockBackend associated with the IDE device.

However, this assumes that every DMA operation is associated with an increment of the BlockBackend’s in-flight counter (e.g. through some ongoing I/O operation), so that draining the BB until its in-flight counter reaches 0 will settle all DMA operations. That is not the case:
For TRIM, the guest can issue a zero-length operation that will not result in any I/O operation forwarded to the BlockBackend, and also not increment the in-flight counter in any other way. In such a case, blk_drain() will be a no-op if no other operations are in flight.

It is clear that if blk_drain() is a no-op, the value of s->bus->dma->aiocb will not change between checking it in the if condition and asserting that it is NULL after blk_drain().

The particular problem is that ide_issue_trim() creates a BH (ide_trim_bh_cb()) to settle the TRIM request: iocb->common.cb() is ide_dma_cb(), which will either create a new request, or find the transfer to be done and call ide_set_inactive(), which clears s->bus->dma->aiocb. Therefore, the blk_drain() must wait for ide_trim_bh_cb() to run, which currently it will not always do.

To fix this issue, we increment the BlockBackend's in-flight counter when the TRIM operation begins (in ide_issue_trim(), when the ide_trim_bh_cb() BH is created) and decrement it when ide_trim_bh_cb() is done.

---
v1:
https://lists.nongnu.org/archive/html/qemu-block/2022-01/msg00024.html

v2:
- Increment BB’s in-flight counter while the BH is active so that blk_drain() will poll until the BH is done, as suggested by Paolo

(No git-backport-diff, because this patch was basically completely rewritten, so it wouldn’t be worth it.)

---
  hw/ide/core.c | 7 +++++++
  1 file changed, 7 insertions(+)
hw/ide/core.c +7 -1
index e28f8aad61..15138225be 100644
@@ -433,12 +433,16 @@ static const AIOCBInfo trim_aiocb_info = {
  static void ide_trim_bh_cb(void *opaque)
  {
      TrimAIOCB *iocb = opaque;
+    BlockBackend *blk = iocb->s->blk;
 
      iocb->common.cb(iocb->common.opaque, iocb->ret);
 
      qemu_bh_delete(iocb->bh);
      iocb->bh = NULL;
      qemu_aio_unref(iocb);
+
+    /* Paired with an increment in ide_issue_trim() */
+    blk_dec_in_flight(blk);
  }
 
  static void ide_issue_trim_cb(void *opaque, int ret)
@@ -508,6 +512,9 @@ BlockAIOCB *ide_issue_trim(
      IDEState *s = opaque;
      TrimAIOCB *iocb;
 
+    /* Paired with a decrement in ide_trim_bh_cb() */
+    blk_inc_in_flight(s->blk);
+
      iocb = blk_aio_get(&trim_aiocb_info, s->blk, cb, cb_opaque);
      iocb->s = s;
      iocb->bh = qemu_bh_new(ide_trim_bh_cb, iocb);
--
Quoted 2 lines

2.34.1

Oh, this *wasn't* the same bug I thought it was.

Quoted 2 lines

There's no regression test, but I will trust you (and Paolo) that this solves the bug you were seeing. It makes sense.

There is one in the BZ linked, but I don’t know where we’d put it into the qemu tree...  I’ve explained in v1 (https://lists.nongnu.org/archive/html/qemu-block/2022-01/msg00024.html) how I didn’t find a way to write a qtest for this, and so resorted to writing boot sector code to reproduce the assertion failure. Now, we could put that as a sample image into the iotests, but that’d just be... wrong.  (Is there a place where something like this would belong?)

Quoted 2 lines

Thanks!

Hanna

#3 lore raw
Quoted 6 lines · Hanna Reitz
Quoted 3 lines · John Snow

There's no regression test, but I will trust you (and Paolo) that this solves the bug you were seeing. It makes sense.

Quoted 8 lines

There is one in the BZ linked, but I don’t know where we’d put it into the qemu tree... I’ve explained in v1
(https://lists.nongnu.org/archive/html/qemu-block/2022-01/msg00024.html)
how I didn’t find a way to write a qtest for this, and so resorted to writing boot sector code to reproduce the assertion failure. Now, we could put that as a sample image into the iotests, but that’d just be... wrong. (Is there a place where something like this would belong?)

No idea. I guess it'd be more of an avacado-test level thing, but I'm not sure I know how to do it quickly. I'm worried there's lots of little things like this that'd be nice to test against, but "where do we put this" is a recurring problem.

(Definitely not insisting on this being solved, and also wise enough to not want to volunteer.)

Quoted 7 lines
Quoted 2 lines

Thanks!

Hanna

--js

Reviewed-by on v2 1/1
#4 lore raw
In reply to Hanna Reitz
Quoted 8 lines · Hanna Reitz

---
v1:
https://lists.nongnu.org/archive/html/qemu-block/2022-01/msg00024.html

Quoted 6 lines

v2:
- Increment BB’s in-flight counter while the BH is active so that blk_drain() will poll until the BH is done, as suggested by Paolo

(No git-backport-diff, because this patch was basically completely rewritten, so it wouldn’t be worth it.)

FWIW you can put a

Supersedes: <[email protected]>

anywhere in the message, and patchew will gladly produce the crossdiff link https://patchew.org/QEMU/[email protected]/diff/[email protected]/ for disbelievers---or for those who want to compare the commit messages.

Paolo

#5 lore raw
In reply to Hanna Reitz

Ping

(I can take it too, if you’d like, John, but you’re listed as the only maintainer for hw/ide, so...  Just say the word, though!)

Quoted 45 lines · Hanna Reitz

When we still have an AIOCB registered for DMA operations, we try to settle the respective operation by draining the BlockBackend associated with the IDE device.

However, this assumes that every DMA operation is associated with an increment of the BlockBackend’s in-flight counter (e.g. through some ongoing I/O operation), so that draining the BB until its in-flight counter reaches 0 will settle all DMA operations. That is not the case:
For TRIM, the guest can issue a zero-length operation that will not result in any I/O operation forwarded to the BlockBackend, and also not increment the in-flight counter in any other way. In such a case, blk_drain() will be a no-op if no other operations are in flight.

It is clear that if blk_drain() is a no-op, the value of s->bus->dma->aiocb will not change between checking it in the if condition and asserting that it is NULL after blk_drain().

The particular problem is that ide_issue_trim() creates a BH (ide_trim_bh_cb()) to settle the TRIM request: iocb->common.cb() is ide_dma_cb(), which will either create a new request, or find the transfer to be done and call ide_set_inactive(), which clears s->bus->dma->aiocb. Therefore, the blk_drain() must wait for ide_trim_bh_cb() to run, which currently it will not always do.

To fix this issue, we increment the BlockBackend's in-flight counter when the TRIM operation begins (in ide_issue_trim(), when the ide_trim_bh_cb() BH is created) and decrement it when ide_trim_bh_cb() is done.

---
v1:
https://lists.nongnu.org/archive/html/qemu-block/2022-01/msg00024.html

v2:
- Increment BB’s in-flight counter while the BH is active so that blk_drain() will poll until the BH is done, as suggested by Paolo

(No git-backport-diff, because this patch was basically completely rewritten, so it wouldn’t be worth it.)

---
  hw/ide/core.c | 7 +++++++
  1 file changed, 7 insertions(+)
#6 lore raw
Quoted 6 lines · Hanna Reitz

Ping

(I can take it too, if you’d like, John, but you’re listed as the only maintainer for hw/ide, so... Just say the word, though!)

Sorry, I sent you a mail off-list at the time where I said you were free to take it whenever you like. Why'd I send it off-list? I don't know....

Please feel free to send this with your next block PR.

--js

Quoted 47 lines · Hanna Reitz
Quoted 45 lines

When we still have an AIOCB registered for DMA operations, we try to settle the respective operation by draining the BlockBackend associated with the IDE device.

However, this assumes that every DMA operation is associated with an increment of the BlockBackend’s in-flight counter (e.g. through some ongoing I/O operation), so that draining the BB until its in-flight counter reaches 0 will settle all DMA operations. That is not the case:
For TRIM, the guest can issue a zero-length operation that will not result in any I/O operation forwarded to the BlockBackend, and also not increment the in-flight counter in any other way. In such a case, blk_drain() will be a no-op if no other operations are in flight.

It is clear that if blk_drain() is a no-op, the value of s->bus->dma->aiocb will not change between checking it in the if condition and asserting that it is NULL after blk_drain().

The particular problem is that ide_issue_trim() creates a BH (ide_trim_bh_cb()) to settle the TRIM request: iocb->common.cb() is ide_dma_cb(), which will either create a new request, or find the transfer to be done and call ide_set_inactive(), which clears s->bus->dma->aiocb. Therefore, the blk_drain() must wait for ide_trim_bh_cb() to run, which currently it will not always do.

To fix this issue, we increment the BlockBackend's in-flight counter when the TRIM operation begins (in ide_issue_trim(), when the ide_trim_bh_cb() BH is created) and decrement it when ide_trim_bh_cb() is done.

---
v1:
https://lists.nongnu.org/archive/html/qemu-block/2022-01/msg00024.html

v2:
- Increment BB’s in-flight counter while the BH is active so that blk_drain() will poll until the BH is done, as suggested by Paolo

(No git-backport-diff, because this patch was basically completely rewritten, so it wouldn’t be worth it.)

---
  hw/ide/core.c | 7 +++++++
  1 file changed, 7 insertions(+)
#7 lore raw
Quoted 11 lines · John Snow
Quoted 5 lines · Hanna Reitz

Ping

(I can take it too, if you’d like, John, but you’re listed as the only maintainer for hw/ide, so... Just say the word, though!)

Sorry, I sent you a mail off-list at the time where I said you were free to take it whenever you like. Why'd I send it off-list? I don't know....

Please feel free to send this with your next block PR.

Oh, oops.  You’re right.  I seemed to remember you saying something along those lines, but when I tried to support my memory with some factual email, I didn’t immediately see it, because I sorted that off-list mail somewhere else than where I was looking...

Thanks, will do!

Hanna