Skip to contentSkip to conversation

nvme: introduce PMR support from NVMe 1.4 spec

PATCH RESEND Applied Andrzej Jakowski 4 participants 6 messages Mar 30, 2020 – Apr 8, 2020

lore
Share & tools
lore
Reviewed-by on v4 1/1
#1 lore raw
In reply to Andrzej Jakowski
Quoted 5 lines · Andrzej Jakowski

This patch introduces support for PMR that has been defined as part of NVMe 1.4 spec. User can now specify a pmrdev option that should point to HostMemoryBackend. pmrdev memory region will subsequently be exposed as PCI BAR 2 in emulated NVMe device. Guest OS can perform mmio read and writes to the PMR region that will stay persistent across system reboot.

Looks pretty good to me.

For a possible future extention, it could be interesting to select the BAR for PMR dynamically, so that you could have CMB and PMR enabled on the same device.

#0 lore raw

PATCH RESEND v4 1/1 nvme: introduce PMR support from NVMe 1.4 spec

This patch introduces support for PMR that has been defined as part of NVMe 1.4 spec. User can now specify a pmrdev option that should point to HostMemoryBackend. pmrdev memory region will subsequently be exposed as PCI BAR 2 in emulated NVMe device. Guest OS can perform mmio read and writes to the PMR region that will stay persistent across system reboot.

---
Changelog:

v4:
- replaced qemu_msync() use with qemu_ram_writeback() to allow pmem_persist() or qemu_msync() be called depending on configuration [4] (Stefan)

- rephrased comments to improve clarity and fixed code style issues [4] (Stefan, Klaus)

v3:
- reworked PMR to use HostMemoryBackend instead of directly mapping PMR backend file into qemu [1] (Stefan)

v2:
- provided support for Bit 1 from PMRWBM register instead of Bit 0 to ensure improved performance in virtualized environment [2] (Stefan)

- added check if pmr size is power of two in size [3] (David)

- addressed cross compilation build problems reported by CI environment

v1:
- inital push of PMR emulation

[1]: https://lore.kernel.org/qemu-devel/[email protected]/
[2]: https://nvmexpress.org/wp-content/uploads/NVM-Express-1_4-2019.06.10-Ratified.pdf
[3]: https://lore.kernel.org/qemu-devel/[email protected]/
[4]: https://lore.kernel.org/qemu-devel/[email protected]/

---
Persistent Memory Region (PMR) is a new optional feature provided in NVMe 1.4 specification. This patch implements initial support for it in NVMe driver.

---
 hw/block/Makefile.objs |   2 +-
 hw/block/nvme.c        | 109 ++++++++++++++++++++++++++
 hw/block/nvme.h        |   2 +
 hw/block/trace-events  |   4 +
 include/block/nvme.h   | 172 +++++++++++++++++++++++++++++++++++++++++
 5 files changed, 288 insertions(+), 1 deletion(-)
hw/block/Makefile.objs +1 -1
index 4b4a2b338d..47960b5f0d 100644
@@ -7,12 +7,12 @@ common-obj-$(CONFIG_PFLASH_CFI02) += pflash_cfi02.o
 common-obj-$(CONFIG_XEN) += xen-block.o
 common-obj-$(CONFIG_ECC) += ecc.o
 common-obj-$(CONFIG_ONENAND) += onenand.o
-common-obj-$(CONFIG_NVME_PCI) += nvme.o
 common-obj-$(CONFIG_SWIM) += swim.o
 
 common-obj-$(CONFIG_SH4) += tc58128.o
 
 obj-$(CONFIG_VIRTIO_BLK) += virtio-blk.o
 obj-$(CONFIG_VHOST_USER_BLK) += vhost-user-blk.o
+obj-$(CONFIG_NVME_PCI) += nvme.o
 
 obj-y += dataplane/
hw/block/nvme.c +109 -0

161 lines not on this page · raw message

hw/block/nvme.h +2 -0
index 557194ee19..6520a9f0be 100644
@@ -83,6 +83,8 @@ typedef struct NvmeCtrl {
     uint64_t    timestamp_set_qemu_clock_ms;    /* QEMU clock time */
 
     char            *serial;
+    HostMemoryBackend *pmrdev;
+
     NvmeNamespace   *namespaces;
     NvmeSQueue      **sq;
     NvmeCQueue      **cq;
hw/block/trace-events +4 -0
index bf6d11b58b..aca54bda14 100644
@@ -110,6 +110,10 @@ nvme_ub_mmiowr_ssreset_w1c_unsupported(void) "attempted to W1C CSTS.NSSRO but CA
 nvme_ub_mmiowr_ssreset_unsupported(void) "attempted NVM subsystem reset but CAP.NSSRS is zero (not supported)"
 nvme_ub_mmiowr_cmbloc_reserved(void) "invalid write to reserved CMBLOC when CMBSZ is zero, ignored"
 nvme_ub_mmiowr_cmbsz_readonly(void) "invalid write to read only CMBSZ, ignored"
+nvme_ub_mmiowr_pmrcap_readonly(void) "invalid write to read only PMRCAP, ignored"
+nvme_ub_mmiowr_pmrsts_readonly(void) "invalid write to read only PMRSTS, ignored"
+nvme_ub_mmiowr_pmrebs_readonly(void) "invalid write to read only PMREBS, ignored"
+nvme_ub_mmiowr_pmrswtp_readonly(void) "invalid write to read only PMRSWTP, ignored"
 nvme_ub_mmiowr_invalid(uint64_t offset, uint64_t data) "invalid MMIO write, offset=0x%"PRIx64", data=0x%"PRIx64""
 nvme_ub_mmiord_misaligned32(uint64_t offset) "MMIO read not 32-bit aligned, offset=0x%"PRIx64""
 nvme_ub_mmiord_toosmall(uint64_t offset) "MMIO read smaller than 32-bits, offset=0x%"PRIx64""
include/block/nvme.h +172 -0

202 lines not on this page · raw message

Signature · 1 line
2.21.1
#2 lore raw
In reply to Keith Busch
Quoted 7 lines · Keith Busch
Quoted 5 lines · Andrzej Jakowski

This patch introduces support for PMR that has been defined as part of NVMe 1.4 spec. User can now specify a pmrdev option that should point to HostMemoryBackend. pmrdev memory region will subsequently be exposed as PCI BAR 2 in emulated NVMe device. Guest OS can perform mmio read and writes to the PMR region that will stay persistent across system reboot.

Quoted 8 lines

Looks pretty good to me.

For a possible future extention, it could be interesting to select the BAR for PMR dynamically, so that you could have CMB and PMR enabled on the same device.

I thought about the same thing, but this would require disabling MSI-X right? Otherwise there is not enough room. AFAIU, the iomem takes up BAR0 and BAR1, CMB (or PMR) uses BAR2 and BAR3 and MSI-X uses BAR4 or 5.

#3 lore raw
Quoted 12 lines · Klaus Birkelund Jensen
Quoted 11 lines · Keith Busch
Quoted 5 lines · Andrzej Jakowski

This patch introduces support for PMR that has been defined as part of NVMe 1.4 spec. User can now specify a pmrdev option that should point to HostMemoryBackend. pmrdev memory region will subsequently be exposed as PCI BAR 2 in emulated NVMe device. Guest OS can perform mmio read and writes to the PMR region that will stay persistent across system reboot.

Looks pretty good to me.

Quoted 8 lines
Quoted 4 lines

For a possible future extention, it could be interesting to select the BAR for PMR dynamically, so that you could have CMB and PMR enabled on the same device.

I thought about the same thing, but this would require disabling MSI-X right? Otherwise there is not enough room. AFAIU, the iomem takes up BAR0 and BAR1, CMB (or PMR) uses BAR2 and BAR3 and MSI-X uses BAR4 or 5.

That's the way the emulated device is currently set, but there's no reason for CMB or MSIx to use an exlusive bar. Both of those can be be offsets into BAR 0/1, for example. PMR is the only one that can't share.

#4 lore raw
Quoted 18 lines · Keith Busch
Quoted 17 lines · Klaus Birkelund Jensen
Quoted 15 lines · Keith Busch
Quoted 5 lines · Andrzej Jakowski

This patch introduces support for PMR that has been defined as part of NVMe 1.4 spec. User can now specify a pmrdev option that should point to HostMemoryBackend. pmrdev memory region will subsequently be exposed as PCI BAR 2 in emulated NVMe device. Guest OS can perform mmio read and writes to the PMR region that will stay persistent across system reboot.

Looks pretty good to me.

For a possible future extention, it could be interesting to select the BAR for PMR dynamically, so that you could have CMB and PMR enabled on the same device.

Quoted 9 lines
Quoted 3 lines

I thought about the same thing, but this would require disabling MSI-X right? Otherwise there is not enough room. AFAIU, the iomem takes up BAR0 and BAR1, CMB (or PMR) uses BAR2 and BAR3 and MSI-X uses BAR4 or 5.

That's the way the emulated device is currently set, but there's no reason for CMB or MSIx to use an exlusive bar. Both of those can be be offsets into BAR 0/1, for example. PMR is the only one that can't share.

Oh, I did not consider that at all. Thanks!

Applied
#5 lore raw
In reply to Keith Busch

Am 30.03.2020 um 18:55 hat Keith Busch geschrieben:

Quoted 10 lines · Andrzej Jakowski
Quoted 5 lines

This patch introduces support for PMR that has been defined as part of NVMe 1.4 spec. User can now specify a pmrdev option that should point to HostMemoryBackend. pmrdev memory region will subsequently be exposed as PCI BAR 2 in emulated NVMe device. Guest OS can perform mmio read and writes to the PMR region that will stay persistent across system reboot.

Looks pretty good to me.

Thanks, applied to the block-next branch for 5.1.

Kevin