nvme: introduce PMR support from NVMe 1.4 spec
PATCH RESEND v4 1/1 nvme: introduce PMR support from NVMe 1.4 spec
This patch introduces support for PMR that has been defined as part of NVMe 1.4 spec. User can now specify a pmrdev option that should point to HostMemoryBackend. pmrdev memory region will subsequently be exposed as PCI BAR 2 in emulated NVMe device. Guest OS can perform mmio read and writes to the PMR region that will stay persistent across system reboot.
---
Changelog:
v4:
- replaced qemu_msync() use with qemu_ram_writeback() to allow pmem_persist() or qemu_msync() be called depending on configuration [4] (Stefan)
- rephrased comments to improve clarity and fixed code style issues [4] (Stefan, Klaus)
v3:
- reworked PMR to use HostMemoryBackend instead of directly mapping PMR backend file into qemu [1] (Stefan)
v2:
- provided support for Bit 1 from PMRWBM register instead of Bit 0 to ensure improved performance in virtualized environment [2] (Stefan)
- added check if pmr size is power of two in size [3] (David)
- addressed cross compilation build problems reported by CI environment
v1:
- inital push of PMR emulation
[1]: https://lore.kernel.org/qemu-devel/[email protected]/
[2]: https://nvmexpress.org/wp-content/uploads/NVM-Express-1_4-2019.06.10-Ratified.pdf
[3]: https://lore.kernel.org/qemu-devel/[email protected]/
[4]: https://lore.kernel.org/qemu-devel/[email protected]/
---
Persistent Memory Region (PMR) is a new optional feature provided in NVMe 1.4 specification. This patch implements initial support for it in NVMe driver.
--- hw/block/Makefile.objs | 2 +- hw/block/nvme.c | 109 ++++++++++++++++++++++++++ hw/block/nvme.h | 2 + hw/block/trace-events | 4 + include/block/nvme.h | 172 +++++++++++++++++++++++++++++++++++++++++ 5 files changed, 288 insertions(+), 1 deletion(-)
hw/block/Makefile.objs +1 -1
@@ -7,12 +7,12 @@ common-obj-$(CONFIG_PFLASH_CFI02) += pflash_cfi02.o common-obj-$(CONFIG_XEN) += xen-block.o common-obj-$(CONFIG_ECC) += ecc.o common-obj-$(CONFIG_ONENAND) += onenand.o -common-obj-$(CONFIG_NVME_PCI) += nvme.o common-obj-$(CONFIG_SWIM) += swim.o common-obj-$(CONFIG_SH4) += tc58128.o obj-$(CONFIG_VIRTIO_BLK) += virtio-blk.o obj-$(CONFIG_VHOST_USER_BLK) += vhost-user-blk.o +obj-$(CONFIG_NVME_PCI) += nvme.o obj-y += dataplane/
hw/block/nvme.c +109 -0
161 lines not on this page · raw message
hw/block/nvme.h +2 -0
@@ -83,6 +83,8 @@ typedef struct NvmeCtrl {uint64_t timestamp_set_qemu_clock_ms;/* QEMU clock time */ char *serial; +HostMemoryBackend *pmrdev; +NvmeNamespace *namespaces;NvmeSQueue **sq;NvmeCQueue **cq;
hw/block/trace-events +4 -0
@@ -110,6 +110,10 @@ nvme_ub_mmiowr_ssreset_w1c_unsupported(void) "attempted to W1C CSTS.NSSRO but CA nvme_ub_mmiowr_ssreset_unsupported(void) "attempted NVM subsystem reset but CAP.NSSRS is zero (not supported)" nvme_ub_mmiowr_cmbloc_reserved(void) "invalid write to reserved CMBLOC when CMBSZ is zero, ignored" nvme_ub_mmiowr_cmbsz_readonly(void) "invalid write to read only CMBSZ, ignored" +nvme_ub_mmiowr_pmrcap_readonly(void) "invalid write to read only PMRCAP, ignored" +nvme_ub_mmiowr_pmrsts_readonly(void) "invalid write to read only PMRSTS, ignored" +nvme_ub_mmiowr_pmrebs_readonly(void) "invalid write to read only PMREBS, ignored" +nvme_ub_mmiowr_pmrswtp_readonly(void) "invalid write to read only PMRSWTP, ignored" nvme_ub_mmiowr_invalid(uint64_t offset, uint64_t data) "invalid MMIO write, offset=0x%"PRIx64", data=0x%"PRIx64"" nvme_ub_mmiord_misaligned32(uint64_t offset) "MMIO read not 32-bit aligned, offset=0x%"PRIx64"" nvme_ub_mmiord_toosmall(uint64_t offset) "MMIO read smaller than 32-bits, offset=0x%"PRIx64""
include/block/nvme.h +172 -0
202 lines not on this page · raw message
Signature · 1 line
2.21.1
Quoted 5 lines · Andrzej Jakowski
This patch introduces support for PMR that has been defined as part of NVMe 1.4 spec. User can now specify a pmrdev option that should point to HostMemoryBackend. pmrdev memory region will subsequently be exposed as PCI BAR 2 in emulated NVMe device. Guest OS can perform mmio read and writes to the PMR region that will stay persistent across system reboot.
Looks pretty good to me.
For a possible future extention, it could be interesting to select the BAR for PMR dynamically, so that you could have CMB and PMR enabled on the same device.
Quoted 7 lines · Keith Busch
Quoted 5 lines · Andrzej Jakowski
This patch introduces support for PMR that has been defined as part of NVMe 1.4 spec. User can now specify a pmrdev option that should point to HostMemoryBackend. pmrdev memory region will subsequently be exposed as PCI BAR 2 in emulated NVMe device. Guest OS can perform mmio read and writes to the PMR region that will stay persistent across system reboot.
Quoted 8 lines
Looks pretty good to me.
Reviewed-by Keith Busch <[email protected]>For a possible future extention, it could be interesting to select the BAR for PMR dynamically, so that you could have CMB and PMR enabled on the same device.
I thought about the same thing, but this would require disabling MSI-X right? Otherwise there is not enough room. AFAIU, the iomem takes up BAR0 and BAR1, CMB (or PMR) uses BAR2 and BAR3 and MSI-X uses BAR4 or 5.
Quoted 12 lines · Klaus Birkelund Jensen
Quoted 11 lines · Keith Busch
Quoted 5 lines · Andrzej Jakowski
This patch introduces support for PMR that has been defined as part of NVMe 1.4 spec. User can now specify a pmrdev option that should point to HostMemoryBackend. pmrdev memory region will subsequently be exposed as PCI BAR 2 in emulated NVMe device. Guest OS can perform mmio read and writes to the PMR region that will stay persistent across system reboot.
Looks pretty good to me.
Reviewed-by Keith Busch <[email protected]>
Quoted 8 lines
Quoted 4 lines
For a possible future extention, it could be interesting to select the BAR for PMR dynamically, so that you could have CMB and PMR enabled on the same device.
I thought about the same thing, but this would require disabling MSI-X right? Otherwise there is not enough room. AFAIU, the iomem takes up BAR0 and BAR1, CMB (or PMR) uses BAR2 and BAR3 and MSI-X uses BAR4 or 5.
That's the way the emulated device is currently set, but there's no reason for CMB or MSIx to use an exlusive bar. Both of those can be be offsets into BAR 0/1, for example. PMR is the only one that can't share.
Quoted 18 lines · Keith Busch
Quoted 17 lines · Klaus Birkelund Jensen
Quoted 15 lines · Keith Busch
Quoted 5 lines · Andrzej Jakowski
This patch introduces support for PMR that has been defined as part of NVMe 1.4 spec. User can now specify a pmrdev option that should point to HostMemoryBackend. pmrdev memory region will subsequently be exposed as PCI BAR 2 in emulated NVMe device. Guest OS can perform mmio read and writes to the PMR region that will stay persistent across system reboot.
Looks pretty good to me.
Reviewed-by Keith Busch <[email protected]>For a possible future extention, it could be interesting to select the BAR for PMR dynamically, so that you could have CMB and PMR enabled on the same device.
Quoted 9 lines
Quoted 3 lines
I thought about the same thing, but this would require disabling MSI-X right? Otherwise there is not enough room. AFAIU, the iomem takes up BAR0 and BAR1, CMB (or PMR) uses BAR2 and BAR3 and MSI-X uses BAR4 or 5.
That's the way the emulated device is currently set, but there's no reason for CMB or MSIx to use an exlusive bar. Both of those can be be offsets into BAR 0/1, for example. PMR is the only one that can't share.
Oh, I did not consider that at all. Thanks!
Kevin Wolf
AppliedAm 30.03.2020 um 18:55 hat Keith Busch geschrieben:
Quoted 10 lines · Andrzej Jakowski
Quoted 5 lines
This patch introduces support for PMR that has been defined as part of NVMe 1.4 spec. User can now specify a pmrdev option that should point to HostMemoryBackend. pmrdev memory region will subsequently be exposed as PCI BAR 2 in emulated NVMe device. Guest OS can perform mmio read and writes to the PMR region that will stay persistent across system reboot.
Looks pretty good to me.
Reviewed-by Keith Busch <[email protected]>
Thanks, applied to the block-next branch for 5.1.
Kevin