Skip to contentSkip to conversation

block/nvme: Fix VFIO_MAP_DMA failed: No space left on device

PATCH Superseded Philippe Mathieu-Daudé 2 participants 8 messages Jun 21, 2021 – Jun 22, 2021 Block layer

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 block/nvme: Fix VFIO_MAP_DMA failed: No space left on device

When the NVMe block driver was introduced (see commit bdd6a90a9e5, January 2018), Linux VFIO_IOMMU_MAP_DMA ioctl was only returning -ENOMEM in case of error. The driver was correctly handling the error path to recycle its volatile IOVA mappings.

To fix CVE-2019-3882, Linux commit 492855939bdb ("vfio/type1: Limit DMA mappings per container", April 2019) added the -ENOSPC error to signal the user exhausted the DMA mappings available for a container.

The block driver started to mis-behave:

qemu-system-x86_64: VFIO_MAP_DMA failed: No space left on device
(qemu)
(qemu) info status
VM status: paused (io-error)
(qemu) c
VFIO_MAP_DMA failed: No space left on device
qemu-system-x86_64: block/block-backend.c:1968: blk_get_aio_context: Assertion `ctx == blk->ctx' failed.

Fix by handling the new -ENOSPC error (when DMA mappings are exhausted) without any distinction to the current -ENOMEM error, so we don't change the behavior on old kernels where the CVE-2019-3882 fix is not present.

An easy way to reproduce this bug is to restrict the DMA mapping limit (65535 by default) when loading the VFIO IOMMU module:

  # modprobe vfio_iommu_type1 dma_entry_limit=666
Fixes bdd6a90a9e5 ("block: Add VFIO based NVMe driver")

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/65

---
v2: KISS checking both errors undistinguishedly (Maxim)

---
 block/nvme.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
block/nvme.c +1 -1
index 2b5421e7aa6..c3d2a49866c 100644
@@ -1030,7 +1030,7 @@ try_map:
         r = qemu_vfio_dma_map(s->vfio,
                               qiov->iov[i].iov_base,
                               len, true, &iova);
-        if (r == -ENOMEM && retry) {
+        if ((r == -ENOMEM || r == -ENOSPC) && retry) {
             retry = false;
             trace_nvme_dma_flush_queue_wait(s);
             if (s->dma_map_count) {
Signature · 1 line
2.31.1
Reviewed-by on v2 1/1
#1 lore raw
Quoted 15 lines · Philippe Mathieu-Daudé

When the NVMe block driver was introduced (see commit bdd6a90a9e5, January 2018), Linux VFIO_IOMMU_MAP_DMA ioctl was only returning -ENOMEM in case of error. The driver was correctly handling the error path to recycle its volatile IOVA mappings.

To fix CVE-2019-3882, Linux commit 492855939bdb ("vfio/type1: Limit DMA mappings per container", April 2019) added the -ENOSPC error to signal the user exhausted the DMA mappings available for a container.

The block driver started to mis-behave:

Quoted 7 lines

qemu-system-x86_64: VFIO_MAP_DMA failed: No space left on device
(qemu)
(qemu) info status
VM status: paused (io-error)
(qemu) c
VFIO_MAP_DMA failed: No space left on device
qemu-system-x86_64: block/block-backend.c:1968: blk_get_aio_context: Assertion `ctx == blk->ctx' failed.

Hi Phil,

The diff looks good to me, but I’m not sure what exactly caused the assertion failure. There is if (r) { goto fail; } that handles -ENOSPC before, so it should be treated as a general case. What am I missing?

Quoted 40 lines

Fix by handling the new -ENOSPC error (when DMA mappings are exhausted) without any distinction to the current -ENOMEM error, so we don't change the behavior on old kernels where the CVE-2019-3882 fix is not present.

An easy way to reproduce this bug is to restrict the DMA mapping limit (65535 by default) when loading the VFIO IOMMU module:

 # modprobe vfio_iommu_type1 dma_entry_limit=666
Fixes bdd6a90a9e5 ("block: Add VFIO based NVMe driver")

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/65

---
v2: KISS checking both errors undistinguishedly (Maxim)
---
block/nvme.c | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)
block/nvme.c +1 -1
index 2b5421e7aa6..c3d2a49866c 100644
@@ -1030,7 +1030,7 @@ try_map:
        r = qemu_vfio_dma_map(s->vfio,
                              qiov->iov[i].iov_base,
                              len, true, &iova);
-        if (r == -ENOMEM && retry) {
+        if ((r == -ENOMEM || r == -ENOSPC) && retry) {
            retry = false;
            trace_nvme_dma_flush_queue_wait(s);
            if (s->dma_map_count) {
Signature · 1 line
2.31.1
#2 lore raw
Quoted 23 lines · Fam Zheng
Quoted 15 lines · Philippe Mathieu-Daudé

When the NVMe block driver was introduced (see commit bdd6a90a9e5, January 2018), Linux VFIO_IOMMU_MAP_DMA ioctl was only returning -ENOMEM in case of error. The driver was correctly handling the error path to recycle its volatile IOVA mappings.

To fix CVE-2019-3882, Linux commit 492855939bdb ("vfio/type1: Limit DMA mappings per container", April 2019) added the -ENOSPC error to signal the user exhausted the DMA mappings available for a container.

The block driver started to mis-behave:

Quoted 7 lines

qemu-system-x86_64: VFIO_MAP_DMA failed: No space left on device
(qemu)
(qemu) info status
VM status: paused (io-error)
(qemu) c
VFIO_MAP_DMA failed: No space left on device
qemu-system-x86_64: block/block-backend.c:1968: blk_get_aio_context: Assertion `ctx == blk->ctx' failed.

Quoted 4 lines

Hi Phil,

The diff looks good to me, but I’m not sure what exactly caused the assertion failure. There is if (r) { goto fail; } that handles -ENOSPC before, so it should be treated as a general case. What am I missing?

Good catch, ENOSPC ends setting BLOCK_DEVICE_IO_STATUS_NOSPACE -> BLOCK_ERROR_ACTION_STOP, so the VM is paused with DMA mapping exhausted. I don't understand the full "VM resume" path, but this is not what we want (IO_NOSPACE is to warn the operator to add more storage and resume, which is pointless in our case, resuming won't help until we flush the mappings).

IIUC what we want is return ENOMEM to set BLOCK_DEVICE_IO_STATUS_FAILED.

#3 lore raw
Quoted 33 lines · Philippe Mathieu-Daudé
Quoted 23 lines · Fam Zheng
Quoted 15 lines · Philippe Mathieu-Daudé

When the NVMe block driver was introduced (see commit bdd6a90a9e5, January 2018), Linux VFIO_IOMMU_MAP_DMA ioctl was only returning -ENOMEM in case of error. The driver was correctly handling the error path to recycle its volatile IOVA mappings.

To fix CVE-2019-3882, Linux commit 492855939bdb ("vfio/type1: Limit DMA mappings per container", April 2019) added the -ENOSPC error to signal the user exhausted the DMA mappings available for a container.

The block driver started to mis-behave:

Quoted 7 lines

qemu-system-x86_64: VFIO_MAP_DMA failed: No space left on device
(qemu)
(qemu) info status
VM status: paused (io-error)
(qemu) c
VFIO_MAP_DMA failed: No space left on device
qemu-system-x86_64: block/block-backend.c:1968: blk_get_aio_context: Assertion `ctx == blk->ctx' failed.

Quoted 4 lines

Hi Phil,

The diff looks good to me, but I’m not sure what exactly caused the assertion failure. There is if (r) { goto fail; } that handles -ENOSPC before, so it should be treated as a general case. What am I missing?

Quoted 8 lines

Good catch, ENOSPC ends setting BLOCK_DEVICE_IO_STATUS_NOSPACE -> BLOCK_ERROR_ACTION_STOP, so the VM is paused with DMA mapping exhausted. I don't understand the full "VM resume" path, but this is not what we want (IO_NOSPACE is to warn the operator to add more storage and resume, which is pointless in our case, resuming won't help until we flush the mappings).

IIUC what we want is return ENOMEM to set BLOCK_DEVICE_IO_STATUS_FAILED.

I agree with that. It just makes me feel there’s another bug in the resuming code path. Can you get a backtrace?

Fam

#4 lore raw
Quoted 35 lines · Fam Zheng
Quoted 35 lines · Philippe Mathieu-Daudé
Quoted 21 lines · Fam Zheng
Quoted 13 lines · Philippe Mathieu-Daudé

When the NVMe block driver was introduced (see commit bdd6a90a9e5, January 2018), Linux VFIO_IOMMU_MAP_DMA ioctl was only returning -ENOMEM in case of error. The driver was correctly handling the error path to recycle its volatile IOVA mappings.

To fix CVE-2019-3882, Linux commit 492855939bdb ("vfio/type1: Limit DMA mappings per container", April 2019) added the -ENOSPC error to signal the user exhausted the DMA mappings available for a container.

The block driver started to mis-behave:

Quoted 7 lines

qemu-system-x86_64: VFIO_MAP_DMA failed: No space left on device
(qemu)
(qemu) info status
VM status: paused (io-error)
(qemu) c
VFIO_MAP_DMA failed: No space left on device
qemu-system-x86_64: block/block-backend.c:1968: blk_get_aio_context: Assertion `ctx == blk->ctx' failed.

Quoted 4 lines

Hi Phil,

The diff looks good to me, but I’m not sure what exactly caused the assertion failure. There is if (r) { goto fail; } that handles -ENOSPC before, so it should be treated as a general case. What am I missing?

Good catch, ENOSPC ends setting BLOCK_DEVICE_IO_STATUS_NOSPACE -> BLOCK_ERROR_ACTION_STOP, so the VM is paused with DMA mapping exhausted. I don't understand the full "VM resume" path, but this is not what we want (IO_NOSPACE is to warn the operator to add more storage and resume, which is pointless in our case, resuming won't help until we flush the mappings).

Quoted 3 lines
Quoted 1 line

IIUC what we want is return ENOMEM to set BLOCK_DEVICE_IO_STATUS_FAILED.

I agree with that. It just makes me feel there’s another bug in the resuming code path. Can you get a backtrace?

It seems the resuming code path bug has been fixed elsewhere:

(qemu) info status
info status
VM status: paused (io-error)
(qemu) c
c

2021-06-22T07:27:00.745466Z qemu-system-x86_64: VFIO_MAP_DMA failed: No

space left on device
(qemu) info status
info status
VM status: paused (io-error)
(qemu) c
c

2021-06-22T07:27:12.458137Z qemu-system-x86_64: VFIO_MAP_DMA failed: No
space left on device
(qemu) c
c
2021-06-22T07:27:13.439167Z qemu-system-x86_64: VFIO_MAP_DMA failed: No
space left on device
(qemu) c
c
2021-06-22T07:27:14.272071Z qemu-system-x86_64: VFIO_MAP_DMA failed: No

space left on device
(qemu)

#5 lore raw
Quoted 63 lines · Philippe Mathieu-Daudé
Quoted 35 lines · Fam Zheng
Quoted 35 lines · Philippe Mathieu-Daudé
Quoted 21 lines · Fam Zheng
Quoted 13 lines · Philippe Mathieu-Daudé

When the NVMe block driver was introduced (see commit bdd6a90a9e5, January 2018), Linux VFIO_IOMMU_MAP_DMA ioctl was only returning -ENOMEM in case of error. The driver was correctly handling the error path to recycle its volatile IOVA mappings.

To fix CVE-2019-3882, Linux commit 492855939bdb ("vfio/type1: Limit DMA mappings per container", April 2019) added the -ENOSPC error to signal the user exhausted the DMA mappings available for a container.

The block driver started to mis-behave:

Quoted 7 lines

qemu-system-x86_64: VFIO_MAP_DMA failed: No space left on device
(qemu)
(qemu) info status
VM status: paused (io-error)
(qemu) c
VFIO_MAP_DMA failed: No space left on device
qemu-system-x86_64: block/block-backend.c:1968: blk_get_aio_context: Assertion `ctx == blk->ctx' failed.

Quoted 4 lines

Hi Phil,

The diff looks good to me, but I’m not sure what exactly caused the assertion failure. There is if (r) { goto fail; } that handles -ENOSPC before, so it should be treated as a general case. What am I missing?

Good catch, ENOSPC ends setting BLOCK_DEVICE_IO_STATUS_NOSPACE -> BLOCK_ERROR_ACTION_STOP, so the VM is paused with DMA mapping exhausted. I don't understand the full "VM resume" path, but this is not what we want (IO_NOSPACE is to warn the operator to add more storage and resume, which is pointless in our case, resuming won't help until we flush the mappings).

Quoted 3 lines
Quoted 1 line

IIUC what we want is return ENOMEM to set BLOCK_DEVICE_IO_STATUS_FAILED.

I agree with that. It just makes me feel there’s another bug in the resuming code path. Can you get a backtrace?

It seems the resuming code path bug has been fixed elsewhere:

(qemu) info status
info status
VM status: paused (io-error)
(qemu) c
c

2021-06-22T07:27:00.745466Z qemu-system-x86_64: VFIO_MAP_DMA failed: No

space left on device
(qemu) info status
info status
VM status: paused (io-error)
(qemu) c
c

2021-06-22T07:27:12.458137Z qemu-system-x86_64: VFIO_MAP_DMA failed: No
space left on device
(qemu) c
c
2021-06-22T07:27:13.439167Z qemu-system-x86_64: VFIO_MAP_DMA failed: No
space left on device
(qemu) c
c
2021-06-22T07:27:14.272071Z qemu-system-x86_64: VFIO_MAP_DMA failed: No
Quoted 3 lines

space left on device
(qemu)

I tested all releases up to v4.1.0 and could not trigger the blk_get_aio_context() assertion. Building using --enable-debug. IIRC Gentoo is more aggressive, so I'll restart using -O2.

#6 lore raw
Quoted 68 lines · Philippe Mathieu-Daudé
Quoted 63 lines · Philippe Mathieu-Daudé
Quoted 35 lines · Fam Zheng
Quoted 35 lines · Philippe Mathieu-Daudé
Quoted 21 lines · Fam Zheng
Quoted 13 lines · Philippe Mathieu-Daudé

When the NVMe block driver was introduced (see commit bdd6a90a9e5, January 2018), Linux VFIO_IOMMU_MAP_DMA ioctl was only returning -ENOMEM in case of error. The driver was correctly handling the error path to recycle its volatile IOVA mappings.

To fix CVE-2019-3882, Linux commit 492855939bdb ("vfio/type1: Limit DMA mappings per container", April 2019) added the -ENOSPC error to signal the user exhausted the DMA mappings available for a container.

The block driver started to mis-behave:

Quoted 7 lines

qemu-system-x86_64: VFIO_MAP_DMA failed: No space left on device
(qemu)
(qemu) info status
VM status: paused (io-error)
(qemu) c
VFIO_MAP_DMA failed: No space left on device
qemu-system-x86_64: block/block-backend.c:1968: blk_get_aio_context: Assertion `ctx == blk->ctx' failed.

Quoted 4 lines

Hi Phil,

The diff looks good to me, but I’m not sure what exactly caused the assertion failure. There is if (r) { goto fail; } that handles -ENOSPC before, so it should be treated as a general case. What am I missing?

Good catch, ENOSPC ends setting BLOCK_DEVICE_IO_STATUS_NOSPACE -> BLOCK_ERROR_ACTION_STOP, so the VM is paused with DMA mapping exhausted. I don't understand the full "VM resume" path, but this is not what we want (IO_NOSPACE is to warn the operator to add more storage and resume, which is pointless in our case, resuming won't help until we flush the mappings).

Quoted 3 lines
Quoted 1 line

IIUC what we want is return ENOMEM to set BLOCK_DEVICE_IO_STATUS_FAILED.

I agree with that. It just makes me feel there’s another bug in the resuming code path. Can you get a backtrace?

It seems the resuming code path bug has been fixed elsewhere:

(qemu) info status
info status
VM status: paused (io-error)
(qemu) c
c

2021-06-22T07:27:00.745466Z qemu-system-x86_64: VFIO_MAP_DMA failed: No

space left on device
(qemu) info status
info status
VM status: paused (io-error)
(qemu) c
c

2021-06-22T07:27:12.458137Z qemu-system-x86_64: VFIO_MAP_DMA failed: No
space left on device
(qemu) c
c
2021-06-22T07:27:13.439167Z qemu-system-x86_64: VFIO_MAP_DMA failed: No
space left on device
(qemu) c
c
2021-06-22T07:27:14.272071Z qemu-system-x86_64: VFIO_MAP_DMA failed: No
Quoted 3 lines

space left on device
(qemu)

Quoted 3 lines

I tested all releases up to v4.1.0 and could not trigger the blk_get_aio_context() assertion. Building using --enable-debug. IIRC Gentoo is more aggressive, so I'll restart using -O2.

Took 4h30 to test all releases with -O3, couldn't reproduce :(

I wish I hadn't postponed writing an Ansible test script...

On v1 Michal said he doesn't have access to the machine anymore, so I'll assume the other issue got fixed elsewhere.

#7 lore raw
Quoted 77 lines · Philippe Mathieu-Daudé
Quoted 68 lines · Philippe Mathieu-Daudé
Quoted 63 lines · Philippe Mathieu-Daudé
Quoted 35 lines · Fam Zheng
Quoted 35 lines · Philippe Mathieu-Daudé
Quoted 21 lines · Fam Zheng
Quoted 13 lines · Philippe Mathieu-Daudé

When the NVMe block driver was introduced (see commit bdd6a90a9e5, January 2018), Linux VFIO_IOMMU_MAP_DMA ioctl was only returning -ENOMEM in case of error. The driver was correctly handling the error path to recycle its volatile IOVA mappings.

To fix CVE-2019-3882, Linux commit 492855939bdb ("vfio/type1: Limit DMA mappings per container", April 2019) added the -ENOSPC error to signal the user exhausted the DMA mappings available for a container.

The block driver started to mis-behave:

Quoted 7 lines

qemu-system-x86_64: VFIO_MAP_DMA failed: No space left on device
(qemu)
(qemu) info status
VM status: paused (io-error)
(qemu) c
VFIO_MAP_DMA failed: No space left on device
qemu-system-x86_64: block/block-backend.c:1968: blk_get_aio_context: Assertion `ctx == blk->ctx' failed.

Quoted 4 lines

Hi Phil,

The diff looks good to me, but I’m not sure what exactly caused the assertion failure. There is if (r) { goto fail; } that handles -ENOSPC before, so it should be treated as a general case. What am I missing?

Good catch, ENOSPC ends setting BLOCK_DEVICE_IO_STATUS_NOSPACE -> BLOCK_ERROR_ACTION_STOP, so the VM is paused with DMA mapping exhausted. I don't understand the full "VM resume" path, but this is not what we want (IO_NOSPACE is to warn the operator to add more storage and resume, which is pointless in our case, resuming won't help until we flush the mappings).

Quoted 3 lines
Quoted 1 line

IIUC what we want is return ENOMEM to set BLOCK_DEVICE_IO_STATUS_FAILED.

I agree with that. It just makes me feel there’s another bug in the resuming code path. Can you get a backtrace?

It seems the resuming code path bug has been fixed elsewhere:

(qemu) info status
info status
VM status: paused (io-error)
(qemu) c
c

2021-06-22T07:27:00.745466Z qemu-system-x86_64: VFIO_MAP_DMA failed: No

space left on device
(qemu) info status
info status
VM status: paused (io-error)
(qemu) c
c

2021-06-22T07:27:12.458137Z qemu-system-x86_64: VFIO_MAP_DMA failed: No
space left on device
(qemu) c
c
2021-06-22T07:27:13.439167Z qemu-system-x86_64: VFIO_MAP_DMA failed: No
space left on device
(qemu) c
c
2021-06-22T07:27:14.272071Z qemu-system-x86_64: VFIO_MAP_DMA failed: No
Quoted 3 lines

space left on device
(qemu)

Quoted 3 lines

I tested all releases up to v4.1.0 and could not trigger the blk_get_aio_context() assertion. Building using --enable-debug. IIRC Gentoo is more aggressive, so I'll restart using -O2.

Quoted 8 lines

Took 4h30 to test all releases with -O3, couldn't reproduce :(

I wish I hadn't postponed writing an Ansible test script...

On v1 Michal said he doesn't have access to the machine anymore, so I'll assume the other issue got fixed elsewhere.

Cool, then I think it’s just the commit message that should be updated.

Fam