Skip to contentSkip to conversation

block/nvme: Fix VFIO_MAP_DMA failed: No space left on device

PATCH Applied Philippe Mathieu-Daudé 2 participants 2 messages Jul 23, 2021 – Jul 26, 2021 Block layer

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 block/nvme: Fix VFIO_MAP_DMA failed: No space left on device

When the NVMe block driver was introduced (see commit bdd6a90a9e5, January 2018), Linux VFIO_IOMMU_MAP_DMA ioctl was only returning -ENOMEM in case of error. The driver was correctly handling the error path to recycle its volatile IOVA mappings.

To fix CVE-2019-3882, Linux commit 492855939bdb ("vfio/type1: Limit DMA mappings per container", April 2019) added the -ENOSPC error to signal the user exhausted the DMA mappings available for a container.

The block driver started to mis-behave:

qemu-system-x86_64: VFIO_MAP_DMA failed: No space left on device (qemu)
(qemu) info status
VM status: paused (io-error)
(qemu) c
VFIO_MAP_DMA failed: No space left on device
(qemu) c
VFIO_MAP_DMA failed: No space left on device

(The VM is not resumable from here, hence stuck.)

Fix by handling the new -ENOSPC error (when DMA mappings are exhausted) without any distinction to the current -ENOMEM error, so we don't change the behavior on old kernels where the CVE-2019-3882 fix is not present.

An easy way to reproduce this bug is to restrict the DMA mapping limit (65535 by default) when loading the VFIO IOMMU module:

  # modprobe vfio_iommu_type1 dma_entry_limit=666
Cc Fam Zheng <[email protected]>
Cc Maxim Levitsky <[email protected]>
Cc Alex Williamson <[email protected]>
Fixes bdd6a90a9e5 ("block: Add VFIO based NVMe driver")

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/65

---
v3: Reworded (Fam)
v2: KISS checking both errors undistinguishedly (Maxim)

---
 block/nvme.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
block/nvme.c +22 -0
index 2b5421e7aa6..e8dbbc23177 100644
@@ -1030,7 +1030,29 @@ try_map:
         r = qemu_vfio_dma_map(s->vfio,
                               qiov->iov[i].iov_base,
                               len, true, &iova);
+        if (r == -ENOSPC) {
+            /*
+             * In addition to the -ENOMEM error, the VFIO_IOMMU_MAP_DMA
+             * ioctl returns -ENOSPC to signal the user exhausted the DMA
+             * mappings available for a container since Linux kernel commit
+             * 492855939bdb ("vfio/type1: Limit DMA mappings per container",
+             * April 2019, see CVE-2019-3882).
+             *
+             * This block driver already handles this error path by checking
+             * for the -ENOMEM error, so we directly replace -ENOSPC by
+             * -ENOMEM. Beside, -ENOSPC has a specific meaning for blockdev
+             * coroutines: it triggers BLOCKDEV_ON_ERROR_ENOSPC and
+             * BLOCK_ERROR_ACTION_STOP which stops the VM, asking the operator
+             * to add more storage to the blockdev. Not something we can do
+             * easily with an IOMMU :)
+             */
+            r = -ENOMEM;
+        }
         if (r == -ENOMEM && retry) {
+            /*
+             * We exhausted the DMA mappings available for our container:
+             * recycle the volatile IOVA mappings.
+             */
             retry = false;
             trace_nvme_dma_flush_queue_wait(s);
             if (s->dma_map_count) {
Signature · 1 line
2.31.1
#1 lore raw
Quoted 39 lines · Philippe Mathieu-Daudé

When the NVMe block driver was introduced (see commit bdd6a90a9e5, January 2018), Linux VFIO_IOMMU_MAP_DMA ioctl was only returning -ENOMEM in case of error. The driver was correctly handling the error path to recycle its volatile IOVA mappings.

To fix CVE-2019-3882, Linux commit 492855939bdb ("vfio/type1: Limit DMA mappings per container", April 2019) added the -ENOSPC error to signal the user exhausted the DMA mappings available for a container.

The block driver started to mis-behave:

qemu-system-x86_64: VFIO_MAP_DMA failed: No space left on device (qemu)
(qemu) info status
VM status: paused (io-error)
(qemu) c
VFIO_MAP_DMA failed: No space left on device
(qemu) c
VFIO_MAP_DMA failed: No space left on device

(The VM is not resumable from here, hence stuck.)

Fix by handling the new -ENOSPC error (when DMA mappings are exhausted) without any distinction to the current -ENOMEM error, so we don't change the behavior on old kernels where the CVE-2019-3882 fix is not present.

An easy way to reproduce this bug is to restrict the DMA mapping limit (65535 by default) when loading the VFIO IOMMU module:

  # modprobe vfio_iommu_type1 dma_entry_limit=666
Cc Fam Zheng <[email protected]>
Cc Maxim Levitsky <[email protected]>
Cc Alex Williamson <[email protected]>
Fixes bdd6a90a9e5 ("block: Add VFIO based NVMe driver")
Quoted 8 lines

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/65

---
v3: Reworded (Fam)
v2: KISS checking both errors undistinguishedly (Maxim)

---
 block/nvme.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

Thanks, applied to my block tree:
https://gitlab.com/stefanha/qemu/commits/block

Stefan