(unknown)
Subsystems (4)
Date: Tue, 19 Mar 2019 14:45:45 +0200
Subject: [PATCH 0/9] RFC: NVME VFIO mediated device
Hi everyone!
In this patch series, I would like to introduce my take on the problem of doing as fast as possible virtualization of storage with emphasis on low latency.
In this patch series I implemented a kernel vfio based, mediated device that allows the user to pass through a partition and/or whole namespace to a guest.
The idea behind this driver is based on paper you can find at
https://www.usenix.org/conference/atc18/presentation/peng,
Although note that I stared the development prior to reading this paper, independently.
In addition to that implementation is not based on code used in the paper as I wasn't being able at that time to make the source available to me.
***Key points about the implementation:***
* Polling kernel thread is used. The polling is stopped after a predefined timeout (1/2 sec by default).
Support for all interrupt driven mode is planned, and it shows promising results.
* Guest sees a standard NVME device - this allows to run guest with unmodified drivers, for example windows guests.
* The NVMe device is shared between host and guest. That means that even a single namespace can be split between host and guest based on different partitions.
* Simple configuration
*** Performance ***
Performance was tested on Intel DC P3700, With Xeon E5-2620 v2 and both latency and throughput is very similar to SPDK.
Soon I will test this on a better server and nvme device and provide more formal performance numbers.
Latency numbers:
~80ms - spdk with fio plugin on the host.
~84ms - nvme driver on the host
~87ms - mdev-nvme + nvme driver in the guest
Throughput was following similar pattern as well.
* Configuration example
$ modprobe nvme mdev_queues=4 $ modprobe nvme-mdev
$ UUID=$(uuidgen) $ DEVICE='device pci address' $ echo $UUID > /sys/bus/pci/devices/$DEVICE/mdev_supported_types/nvme-2Q_V1/create $ echo n1p3 > /sys/bus/mdev/devices/$UUID/namespaces/add_namespace #attach host namespace 1 parition 3 $ echo 11 > /sys/bus/mdev/devices/$UUID/settings/iothread_cpu #pin the io thread to cpu 11
Afterward boot qemu with
-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$UUID
Zero configuration on the guest.
*** FAQ ***
* Why to make this in the kernel? Why this is better that SPDK
-> Reuse the existing nvme kernel driver in the host. No new drivers in the guest.
-> Share the NVMe device between host and guest.
Even in fully virtualized configurations,
some partitions of nvme device could be used by guests as block devices while others passed through with nvme-mdev to achieve balance between all features of full IO stack emulation and performance.
-> NVME-MDEV is a bit faster due to the fact that in-kernel driver
can send interrupts to the guest directly without a context switch that can be expensive due to meltdown mitigation.
-> Is able to utilize interrupts to get reasonable performance.
This is only implemented
as a proof of concept and not included in the patches, but interrupt driven mode shows reasonable performance
-> This is a framework that later can be used to support NVMe devices
with more of the IO virtualization built-in
(IOMMU with PASID support coupled with device that supports it)
* Why to attach directly to nvme-pci driver and not use block layer IO -> The direct attachment allows for better performance, but I will check the possibility of using block IO, especially for fabrics drivers.
*** Implementation notes ***
* All guest memory is mapped into the physical nvme device but not 1:1 as vfio-pci would do this.
This allows very efficient DMA.
To support this, patch 2 adds ability for a mdev device to listen on guest's memory map events.
Any such memory is immediately pinned and then DMA mapped. (Support for fabric drivers where this is not possible exits too,
in which case the fabric driver will do its own DMA mapping)
* nvme core driver is modified to announce the appearance and disappearance of nvme controllers and namespaces, to which the nvme-mdev driver is subscribed.
* nvme-pci driver is modified to expose raw interface of attaching to and sending/polling the IO queues.
This allows the mdev driver very efficiently to submit/poll for the IO. By default one host queue is used per each mediated device. (support for other fabric based host drivers is planned)
* The nvme-mdev doesn't assume presence of KVM, thus any VFIO user, including SPDK, a qemu running with tccg, ... can use this virtual device.
*** Testing ***
The device was tested with stock QEMU 3.0 on the host,
with host was using 5.0 kernel with nvme-mdev added and the following hardware:
* QEMU nvme virtual device (with nested guest)
* Intel DC P3700 on Xeon E5-2620 v2 server
* Samsung SM981 (in a Thunderbolt enclosure, with my laptop)
* Lenovo NVME device found in my laptop
The guest was tested with kernel 4.16, 4.18, 4.20 and
the same custom complied kernel 5.0
Windows 10 guest was tested too with both Microsoft's inbox driver and open source community NVME driver
(https://lists.openfabrics.org/pipermail/nvmewin/2016-December/001420.html)
Testing was mostly done on x86_64, but 32 bit host/guest combination was lightly tested too.
In addition to that, the virtual device was tested with nested guest, by passing the virtual device to it,
using pci passthrough, qemu userspace nvme driver, and spdk
PS: I used to contribute to the kernel as a hobby using the
[email protected] address
Maxim Levitsky (9):
vfio/mdev: add .request callback
nvme/core: add some more values from the spec
nvme/core: add NVME_CTRL_SUSPENDED controller state nvme/pci: use the NVME_CTRL_SUSPENDED state
nvme/pci: add known admin effects to augument admin effects log page nvme/pci: init shadow doorbell after each reset nvme/core: add mdev interfaces
nvme/core: add nvme-mdev core driver
nvme/pci: implement the mdev external queue allocation interface
MAINTAINERS | 5 + drivers/nvme/Kconfig | 1 + drivers/nvme/Makefile | 1 + drivers/nvme/host/core.c | 149 +++++- drivers/nvme/host/nvme.h | 55 ++- drivers/nvme/host/pci.c | 385 ++++++++++++++- drivers/nvme/mdev/Kconfig | 16 + drivers/nvme/mdev/Makefile | 5 + drivers/nvme/mdev/adm.c | 873 ++++++++++++++++++++++++++++++++++ drivers/nvme/mdev/events.c | 142 ++++++ drivers/nvme/mdev/host.c | 491 +++++++++++++++++++ drivers/nvme/mdev/instance.c | 802 +++++++++++++++++++++++++++++++ drivers/nvme/mdev/io.c | 563 ++++++++++++++++++++++ drivers/nvme/mdev/irq.c | 264 ++++++++++ drivers/nvme/mdev/mdev.h | 56 +++ drivers/nvme/mdev/mmio.c | 591 +++++++++++++++++++++++ drivers/nvme/mdev/pci.c | 247 ++++++++++ drivers/nvme/mdev/priv.h | 700 +++++++++++++++++++++++++++ drivers/nvme/mdev/udata.c | 390 +++++++++++++++ drivers/nvme/mdev/vcq.c | 207 ++++++++ drivers/nvme/mdev/vctrl.c | 514 ++++++++++++++++++++ drivers/nvme/mdev/viommu.c | 322 +++++++++++++ drivers/nvme/mdev/vns.c | 356 ++++++++++++++ drivers/nvme/mdev/vsq.c | 178 +++++++ drivers/vfio/mdev/vfio_mdev.c | 11 + include/linux/mdev.h | 4 + include/linux/nvme.h | 88 +++- 27 files changed, 7375 insertions(+), 41 deletions(-) create mode 100644 drivers/nvme/mdev/Kconfig create mode 100644 drivers/nvme/mdev/Makefile create mode 100644 drivers/nvme/mdev/adm.c create mode 100644 drivers/nvme/mdev/events.c create mode 100644 drivers/nvme/mdev/host.c create mode 100644 drivers/nvme/mdev/instance.c create mode 100644 drivers/nvme/mdev/io.c create mode 100644 drivers/nvme/mdev/irq.c create mode 100644 drivers/nvme/mdev/mdev.h create mode 100644 drivers/nvme/mdev/mmio.c create mode 100644 drivers/nvme/mdev/pci.c create mode 100644 drivers/nvme/mdev/priv.h create mode 100644 drivers/nvme/mdev/udata.c create mode 100644 drivers/nvme/mdev/vcq.c create mode 100644 drivers/nvme/mdev/vctrl.c create mode 100644 drivers/nvme/mdev/viommu.c create mode 100644 drivers/nvme/mdev/vns.c create mode 100644 drivers/nvme/mdev/vsq.c
Signature · 1 line
2.17.2
PATCH 1/9 vfio/mdev: add .request callback
This will allow the hotplug to be enabled for mediated devices
--- drivers/vfio/mdev/vfio_mdev.c | 11 +++++++++++ include/linux/mdev.h | 4 ++++ 2 files changed, 15 insertions(+)
drivers/vfio/mdev/vfio_mdev.c +11 -0
@@ -101,6 +101,16 @@ static int vfio_mdev_mmap(void *device_data, struct vm_area_struct *vma)return parent->ops->mmap (mdev, vma); } +static void vfio_mdev_request (void *device_data,unsigned int count) +{ +struct mdev_device *mdev = device_data; +struct mdev_parent *parent = mdev->parent; + +if (unlikely (!parent->ops->request)) +return ; + parent->ops->request (mdev, count); +} +static const struct vfio_device_ops vfio_mdev_dev_ops = { .name ="vfio-mdev" , .open = vfio_mdev_open,
@@ -109,6 +119,7 @@ static const struct vfio_device_ops vfio_mdev_dev_ops = { .read = vfio_mdev_read, .write = vfio_mdev_write, .mmap = vfio_mdev_mmap, + .request = vfio_mdev_request, };static int vfio_mdev_probe (struct device *dev)
include/linux/mdev.h +4 -0
@@ -13,6 +13,9 @@#ifndef MDEV_H #define MDEV_H +#include <linux/uuid.h> +#include <linux/device.h> +struct mdev_device ;/**
@@ -81,6 +84,7 @@ struct mdev_parent_ops {long (*ioctl)(struct mdev_device *mdev,unsigned int cmd,unsigned long arg);int (*mmap)(struct mdev_device *mdev,struct vm_area_struct *vma); +void (*request)(struct mdev_device *mdev,unsigned int count); };/* interface for exporting mdev supported type attributes */
Signature · 1 line
2.17.2
PATCH 2/9 nvme/core: add some more values from the spec
In reply to Maxim LevitskyThis adds few defines from the spec,
that will be used in the nvme-mdev driver
--- include/linux/nvme.h | 88 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 68 insertions(+), 20 deletions(-)
include/linux/nvme.h +68 -20
@@ -152,32 +152,42 @@ enum {#define NVME_NVM_IOCQES 4enum { -NVME_CC_ENABLE =1 <<0 , -NVME_CC_CSS_NVM =0 <<4 ,NVME_CC_EN_SHIFT =0 , +NVME_CC_ENABLE =1 <<NVME_CC_EN_SHIFT , +NVME_CC_CSS_SHIFT =4 , +NVME_CC_CSS_NVM =0 <<NVME_CC_CSS_SHIFT , +NVME_CC_MPS_SHIFT =7 , +NVME_CC_MPS_MASK =0xF <<NVME_CC_MPS_SHIFT , +NVME_CC_AMS_SHIFT =11 , -NVME_CC_SHN_SHIFT =14 , -NVME_CC_IOSQES_SHIFT =16 , -NVME_CC_IOCQES_SHIFT =20 ,NVME_CC_AMS_RR =0 <<NVME_CC_AMS_SHIFT ,NVME_CC_AMS_WRRU =1 <<NVME_CC_AMS_SHIFT ,NVME_CC_AMS_VS =7 <<NVME_CC_AMS_SHIFT , +NVME_CC_AMS_MASK =0x7 <<NVME_CC_AMS_SHIFT , + +NVME_CC_SHN_SHIFT =14 ,NVME_CC_SHN_NONE =0 <<NVME_CC_SHN_SHIFT ,NVME_CC_SHN_NORMAL =1 <<NVME_CC_SHN_SHIFT ,NVME_CC_SHN_ABRUPT =2 <<NVME_CC_SHN_SHIFT ,NVME_CC_SHN_MASK =3 <<NVME_CC_SHN_SHIFT , + +NVME_CC_IOSQES_SHIFT =16 , +NVME_CC_IOCQES_SHIFT =20 ,NVME_CC_IOSQES =NVME_NVM_IOSQES <<NVME_CC_IOSQES_SHIFT ,NVME_CC_IOCQES =NVME_NVM_IOCQES <<NVME_CC_IOCQES_SHIFT , +NVME_CSTS_RDY =1 <<0 ,NVME_CSTS_CFS =1 <<1 ,NVME_CSTS_NSSRO =1 <<4 ,NVME_CSTS_PP =1 <<5 , -NVME_CSTS_SHST_NORMAL =0 <<2 , -NVME_CSTS_SHST_OCCUR =1 <<2 , -NVME_CSTS_SHST_CMPLT =2 <<2 , -NVME_CSTS_SHST_MASK =3 <<2 , + +NVME_CSTS_SHST_SHIFT =2 , +NVME_CSTS_SHST_NORMAL =0 <<NVME_CSTS_SHST_SHIFT , +NVME_CSTS_SHST_OCCUR =1 <<NVME_CSTS_SHST_SHIFT , +NVME_CSTS_SHST_CMPLT =2 <<NVME_CSTS_SHST_SHIFT , +NVME_CSTS_SHST_MASK =3 <<NVME_CSTS_SHST_SHIFT , };struct nvme_id_power_state {
@@ -404,6 +414,20 @@ enum { NVME_NIDT_UUID = 0x03, }; +struct nvme_err_log_entry { +__u8 err_count[8 ]; +__le16 sqid; +__le16 cid; +__le16 status; +__le16 location; +__u8 lba[8 ]; +__le32 ns; +__u8 vnd; +__u8 rsvd1[3 ]; +__u8 cmd_specific[8 ]; +__u8 rsvd2[24 ]; +}; +struct nvme_smart_log {__u8 critical_warning;__u8 temperature[2 ];
@@ -491,13 +515,30 @@ enum { NVME_AER_VS = 7, }; -enum { -NVME_AER_NOTICE_NS_CHANGED =0x00 , -NVME_AER_NOTICE_FW_ACT_STARTING =0x01 , -NVME_AER_NOTICE_ANA =0x03 , -NVME_AER_NOTICE_DISC_CHANGED =0xf0 , +enum nvme_async_event_type { +NVME_AER_TYPE_ERROR =0 , +NVME_AER_TYPE_SMART =1 , +NVME_AER_TYPE_NOTICE =2 , +NVME_AER_TYPE_MAX =7 , }; +enum nvme_async_event { +NVME_AER_ERROR_INVALID_DB_REG =0 , +NVME_AER_ERROR_INVALID_DB_VALUE =1 , +NVME_AER_ERROR_DIAG_FAILURE =2 , +NVME_AER_ERROR_PERSISTENT_INT_ERR =3 , +NVME_AER_ERROR_TRANSIENT_INT_ERR =4 , +NVME_AER_ERROR_FW_IMAGE_LOAD_ERR =5 , + +NVME_AER_SMART_SUBSYS_RELIABILITY =0 , +NVME_AER_SMART_TEMP_THRESH =1 , +NVME_AER_SMART_SPARE_BELOW_THRESH =2 , + +NVME_AER_NOTICE_NS_CHANGED =0 , +NVME_AER_NOTICE_FW_ACT_STARTING =1 , +NVME_AER_NOTICE_ANA =3 , +NVME_AER_NOTICE_DISC_CHANGED =0xf0 , +};enum {NVME_AEN_BIT_NS_ATTR =8 ,NVME_AEN_BIT_FW_ACT =9 ,
@@ -548,12 +589,6 @@ struct nvme_reservation_status { } regctl_ds[]; }; -enum nvme_async_event_type { -NVME_AER_TYPE_ERROR =0 , -NVME_AER_TYPE_SMART =1 , -NVME_AER_TYPE_NOTICE =2 , -}; -/* I/O commands */ enum nvme_opcode {
@@ -705,10 +740,19 @@ enum {NVME_RW_DSM_LATENCY_LOW =3 <<4 ,NVME_RW_DSM_SEQ_REQ =1 <<6 ,NVME_RW_DSM_COMPRESSED =1 <<7 , + +NVME_WZ_DEAC =1 <<9 ,NVME_RW_PRINFO_PRCHK_REF =1 <<10 ,NVME_RW_PRINFO_PRCHK_APP =1 <<11 ,NVME_RW_PRINFO_PRCHK_GUARD =1 <<12 ,NVME_RW_PRINFO_PRACT =1 <<13 , + +NVME_RW_PRINFO = +NVME_RW_PRINFO_PRCHK_REF | +NVME_RW_PRINFO_PRCHK_APP | +NVME_RW_PRINFO_PRCHK_GUARD | +NVME_RW_PRINFO_PRACT , + NVME_RW_DTYPE_STREAMS = 1 << 4, };
@@ -809,6 +853,7 @@ enum {NVME_SQ_PRIO_HIGH = (1 <<1 ),NVME_SQ_PRIO_MEDIUM = (2 <<1 ),NVME_SQ_PRIO_LOW = (3 <<1 ), +NVME_SQ_PRIO_MASK = (3 <<1 ),NVME_FEAT_ARBITRATION =0x01 ,NVME_FEAT_POWER_MGMT =0x02 , NVME_FEAT_LBA_RANGE = 0x03,
@@ -1146,6 +1191,7 @@ struct streams_directive_params {struct nvme_command {union { +__le32 dwords[16 ];struct nvme_common_command common;struct nvme_rw_command rw;struct nvme_identify identify;
@@ -1217,6 +1263,8 @@ enum {NVME_SC_SGL_INVALID_METADATA =0x10 ,NVME_SC_SGL_INVALID_TYPE =0x11 , +NVME_SC_PRP_OFFSET_INVALID =0x13 , +NVME_SC_SGL_INVALID_OFFSET =0x16 , NVME_SC_SGL_INVALID_SUBTYPE = 0x17,
Signature · 1 line
2.17.2
PATCH 3/9 nvme/core: add NVME_CTRL_SUSPENDED controller state
In reply to Maxim LevitskyThis state will be used by a controller that is going to suspended state, and will later be used by mdev framework to detect this and flush its queues
--- drivers/nvme/host/core.c | 15 +++++++++++++++ drivers/nvme/host/nvme.h | 1 + 2 files changed, 16 insertions(+)
drivers/nvme/host/core.c +15 -0
@@ -320,6 +320,19 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,switch (old_state) {case NVME_CTRL_NEW :case NVME_CTRL_RESETTING : +case NVME_CTRL_CONNECTING : +case NVME_CTRL_SUSPENDED : + changed = true; +/* FALLTHRU */ +default : +break ; + } +break ; +case NVME_CTRL_SUSPENDED : +switch (old_state) { +case NVME_CTRL_NEW : +case NVME_CTRL_LIVE : +case NVME_CTRL_RESETTING :case NVME_CTRL_CONNECTING : changed = true;/* FALLTHRU */
@@ -332,6 +345,7 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,case NVME_CTRL_NEW :case NVME_CTRL_LIVE :case NVME_CTRL_ADMIN_ONLY : +case NVME_CTRL_SUSPENDED : changed = true;/* FALLTHRU */ default :
@@ -354,6 +368,7 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,case NVME_CTRL_ADMIN_ONLY :case NVME_CTRL_RESETTING :case NVME_CTRL_CONNECTING : +case NVME_CTRL_SUSPENDED : changed = true;/* FALLTHRU */ default :
drivers/nvme/host/nvme.h +1 -0
@@ -142,6 +142,7 @@ static inline u16 nvme_req_qid(struct request *req)enum nvme_ctrl_state {NVME_CTRL_NEW ,NVME_CTRL_LIVE , +NVME_CTRL_SUSPENDED ,NVME_CTRL_ADMIN_ONLY ,/* Only admin queue live */ NVME_CTRL_RESETTING ,NVME_CTRL_CONNECTING ,
Signature · 1 line
2.17.2
PATCH 4/9 nvme/pci: use the NVME_CTRL_SUSPENDED state
In reply to Maxim LevitskyWhen enteriing low power state, the nvme
driver will now inform the core with the NVME_CTRL_SUSPENDED state which will allow mdev driver to act on this information
--- drivers/nvme/host/pci.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/nvme/host/pci.c +5 -1
@@ -2451,7 +2451,8 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown)u32 csts =readl (dev->bar +NVME_REG_CSTS );if (dev->ctrl.state ==NVME_CTRL_LIVE || - dev->ctrl.state ==NVME_CTRL_RESETTING ) + dev->ctrl.state ==NVME_CTRL_RESETTING || + dev->ctrl.state ==NVME_CTRL_SUSPENDED )nvme_start_freeze (&dev->ctrl); dead = !!((csts &NVME_CSTS_CFS ) || !(csts &NVME_CSTS_RDY ) || pdev->error_state != pci_channel_io_normal);
@@ -2897,6 +2898,9 @@ static int nvme_suspend(struct device *dev)struct pci_dev *pdev =to_pci_dev (dev);struct nvme_dev *ndev =pci_get_drvdata (pdev); +if (!nvme_change_ctrl_state (&ndev->ctrl,NVME_CTRL_SUSPENDED )) +WARN_ON (1 ); +nvme_dev_disable (ndev, true);return 0 ; }
Signature · 1 line
2.17.2
Re: [PATCH 4/9] nvme/pci: use the NVME_CTRL_SUSPENDED state
Quoted 1 line · Maxim Levitsky
When enteriing low power state, the nvme
Typo: "entering".
Quoted 2 lines
driver will now inform the core with the
NVME_CTRL_SUSPENDEDstate which will allow mdev driver to act on this information
[snip]
Fam
PATCH 5/9 nvme/pci: add known admin effects to augument admin effects log page
In reply to Maxim LevitskyAdd known admin effects even if hardware has known admin effects page, since hardware can't be ever trusted to report sane values. (on my Intel DC P3700, it reports no side effects for namespace format)
--- drivers/nvme/host/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/nvme/host/core.c +2 -2
@@ -1260,8 +1260,8 @@ static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,if (ctrl->effects) effects =le32_to_cpu (ctrl->effects->acs[opcode]); -else - effects =nvme_known_admin_effects (opcode); + + effects |=nvme_known_admin_effects (opcode);/* * For simplicity, IO to all namespaces is quiesced even if the command
Signature · 1 line
2.17.2
PATCH 6/9 nvme/pci: init shadow doorbell after each reset
In reply to Maxim LevitskyThe spec states
"The settings are not retained across a Controller Level Reset" Therefore the driver must enable the shadow doorbell, after each reset.
This was caught while testing the nvme driver over upcoming nvme-mdev device
--- drivers/nvme/host/pci.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/nvme/host/pci.c +1 -2
@@ -2347,8 +2347,6 @@ static int nvme_dev_add(struct nvme_dev *dev)return ret; } dev->ctrl.tagset = &dev->tagset; - -nvme_dbbuf_set (dev); }else {blk_mq_update_nr_hw_queues (&dev->tagset, dev->online_queues -1 );
@@ -2356,6 +2354,7 @@ static int nvme_dev_add(struct nvme_dev *dev)nvme_free_queues (dev, dev->online_queues); } +nvme_dbbuf_set (dev);return 0 ; }
Signature · 1 line
2.17.2
PATCH 7/9 nvme/core: add mdev interfaces
In reply to Maxim LevitskyThis adds infrastructure for a nvme-mdev
to attach to the core driver, to be able to
know which nvme controllers are present and which
namespaces they have.
It also adds an interface to nvme device drivers which expose the its queues in a controlled manner to the nvme mdev core driver. A driver must opt-in for this using a new flag NVME_F_MDEV_SUPPORTED
If the mdev device driver also sets the NVME_F_MDEV_DMA_SUPPORTED, the mdev core will
dma map all the guest memory into the nvme device, so that nvme device driver can use dma addresses as passed from the mdev core driver
--- drivers/nvme/host/core.c | 125 ++++++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 54 +++++++++++++++-- 2 files changed, 172 insertions(+), 7 deletions(-)
drivers/nvme/host/core.c +122 -3
@@ -97,11 +97,111 @@ static dev_t nvme_chr_devt;static struct class *nvme_class;static struct class *nvme_subsys_class; +static void nvme_ns_remove (struct nvme_ns *ns);static int nvme_revalidate_disk (struct gendisk *disk);static void nvme_put_subsystem (struct nvme_subsystem *subsys);static void nvme_remove_invalid_namespaces (struct nvme_ctrl *ctrl,unsigned nsid); +#ifdef CONFIG_NVME_MDEV + +static struct nvme_mdev_driver *mdev_driver_interface; +static DEFINE_MUTEX (mdev_ctrl_lock); +static LIST_HEAD (mdev_ctrl_list); + +static bool nvme_ctrl_has_mdev (struct nvme_ctrl *ctrl) +{ +return (ctrl->ops->flags &NVME_F_MDEV_SUPPORTED ) !=0 ; +} + +static void nvme_mdev_add_ctrl (struct nvme_ctrl *ctrl) +{ +if (nvme_ctrl_has_mdev (ctrl)) { +mutex_lock (&mdev_ctrl_lock); +list_add_tail (&ctrl->link, &mdev_ctrl_list); +mutex_unlock (&mdev_ctrl_lock); + } +} + +static void nvme_mdev_remove_ctrl (struct nvme_ctrl *ctrl) +{ +if (nvme_ctrl_has_mdev (ctrl)) { +mutex_lock (&mdev_ctrl_lock); +list_del_init (&ctrl->link); +mutex_unlock (&mdev_ctrl_lock); + } +} + +int nvme_core_register_mdev_driver (struct nvme_mdev_driver *driver_ops) +{ +struct nvme_ctrl *ctrl; + +if (mdev_driver_interface) +return -EEXIST ; + + mdev_driver_interface = driver_ops; + +mutex_lock (&mdev_ctrl_lock); +list_for_each_entry (ctrl, &mdev_ctrl_list, link) + mdev_driver_interface->nvme_ctrl_state_changed (ctrl); + +mutex_unlock (&mdev_ctrl_lock); +return 0 ; +} +EXPORT_SYMBOL_GPL (nvme_core_register_mdev_driver); + +void nvme_core_unregister_mdev_driver (struct nvme_mdev_driver *driver_ops) +{ +if (WARN_ON (driver_ops != mdev_driver_interface)) +return ; + mdev_driver_interface =NULL ; +} +EXPORT_SYMBOL_GPL (nvme_core_unregister_mdev_driver); + +static void nvme_mdev_ctrl_state_changed (struct nvme_ctrl *ctrl) +{ +if (!mdev_driver_interface || !nvme_ctrl_has_mdev (ctrl)) +return ; +if (!try_module_get (mdev_driver_interface->owner)) +return ; + + mdev_driver_interface->nvme_ctrl_state_changed (ctrl); +module_put (mdev_driver_interface->owner); +} + +static void nvme_mdev_ns_state_changed (struct nvme_ctrl *ctrl, +struct nvme_ns *ns,bool removed) +{ +if (!mdev_driver_interface || !nvme_ctrl_has_mdev (ctrl)) +return ; +if (!try_module_get (mdev_driver_interface->owner)) +return ; + + mdev_driver_interface->nvme_ns_state_changed (ctrl, + ns->head->ns_id, removed); +module_put (mdev_driver_interface->owner); +} + +#else +static void nvme_mdev_ctrl_state_changed (struct nvme_ctrl *ctrl) +{ +} + +static void nvme_mdev_ns_state_changed (struct nvme_ctrl *ctrl, +struct nvme_ns *ns,bool removed) +{ +} + +static void nvme_mdev_add_ctrl (struct nvme_ctrl *ctrl) +{ +} + +static void nvme_mdev_remove_ctrl (struct nvme_ctrl *ctrl) +{ +} + +#endif +static void nvme_set_queue_dying (struct nvme_ns *ns) {/*
@@ -390,10 +490,13 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,if (changed) ctrl->state = new_state; -spin_unlock_irqrestore (&ctrl->lock, flags); + +nvme_mdev_ctrl_state_changed (ctrl); +if (changed && ctrl->state ==NVME_CTRL_LIVE )nvme_kick_requeue_lists (ctrl); +return changed; }EXPORT_SYMBOL_GPL (nvme_change_ctrl_state);
@@ -429,10 +532,11 @@ static void nvme_free_ns(struct kref *kref)kfree (ns); } -static void nvme_put_ns (struct nvme_ns *ns) +void nvme_put_ns (struct nvme_ns *ns) {kref_put (&ns->kref, nvme_free_ns); } +EXPORT_SYMBOL_GPL (nvme_put_ns);static inline void nvme_clear_nvme_request (struct request *req) {
@@ -1275,6 +1379,11 @@ static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,return effects; } +static void nvme_update_ns (struct nvme_ctrl *ctrl,struct nvme_ns *ns) +{ +nvme_mdev_ns_state_changed (ctrl, ns, false); +} +static void nvme_update_formats (struct nvme_ctrl *ctrl) {struct nvme_ns *ns;
@@ -1283,6 +1392,8 @@ static void nvme_update_formats(struct nvme_ctrl *ctrl)list_for_each_entry (ns, &ctrl->namespaces, list)if (ns->disk &&nvme_revalidate_disk (ns->disk))nvme_set_queue_dying (ns); +else +nvme_update_ns (ctrl, ns);up_read (&ctrl->namespaces_rwsem);nvme_remove_invalid_namespaces (ctrl,NVME_NSID_ALL );
@@ -3133,7 +3244,7 @@ static int ns_cmp(void *priv, struct list_head *a, struct list_head *b)return nsa->head->ns_id - nsb->head->ns_id; } -static struct nvme_ns *nvme_find_get_ns (struct nvme_ctrl *ctrl,unsigned nsid) +struct nvme_ns *nvme_find_get_ns (struct nvme_ctrl *ctrl,unsigned int nsid) {struct nvme_ns *ns, *ret =NULL ;
@@ -3151,6 +3262,7 @@ static struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid)up_read (&ctrl->namespaces_rwsem);return ret; } +EXPORT_SYMBOL_GPL (nvme_find_get_ns);static int nvme_setup_streams_ns (struct nvme_ctrl *ctrl,struct nvme_ns *ns) {
@@ -3271,6 +3383,8 @@ static void nvme_ns_remove(struct nvme_ns *ns)if (test_and_set_bit (NVME_NS_REMOVING , &ns->flags))return ; +nvme_mdev_ns_state_changed (ns->ctrl, ns, true); +nvme_fault_inject_fini (ns);if (ns->disk && ns->disk->flags &GENHD_FL_UP ) {del_gendisk (ns->disk);
@@ -3301,6 +3415,8 @@ static void nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid)if (ns) {if (ns->disk &&revalidate_disk (ns->disk))nvme_ns_remove (ns); +else +nvme_update_ns (ctrl, ns);nvme_put_ns (ns); }else nvme_alloc_ns (ctrl, nsid);
@@ -3654,6 +3770,7 @@ static void nvme_free_ctrl(struct device *dev)sysfs_remove_link (&subsys->dev.kobj,dev_name (ctrl->device)); } +nvme_mdev_remove_ctrl (ctrl); ctrl->ops->free_ctrl (ctrl);if (subsys)
@@ -3726,6 +3843,8 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,dev_pm_qos_update_user_latency_tolerance (ctrl->device,min (default_ps_max_latency_us, (unsigned long )S32_MAX )); +nvme_mdev_add_ctrl (ctrl); +return 0 ; out_free_name:kfree_const (ctrl->device->kobj.name);
drivers/nvme/host/nvme.h +50 -4
94 lines not on this page · raw message
Signature · 1 line
2.17.2
Re: [PATCH 7/9] nvme/core: add mdev interfaces
drivers/nvme/host/core.c static dev_t nvme_chr_devt; · line 142
Quoted 8 lines · Maxim Levitsky
drivers/nvme/host/core.c +8
-0+int nvme_core_register_mdev_driver (struct nvme_mdev_driver *driver_ops) +{ +struct nvme_ctrl *ctrl; + +if (mdev_driver_interface) +return -EEXIST ; + + mdev_driver_interface = driver_ops;
Can mdev_driver_interface be accessed from two CPUs at the same time? mdev_driver_interface isn't protected by the mutex. The state_changed functions below also don't protect mdev_driver_interface.
Re: [PATCH 7/9] nvme/core: add mdev interfaces
Quoted 13 lines · Stefan Hajnoczi
drivers/nvme/host/core.c static dev_t nvme_chr_devt; · line 142
Quoted 8 lines · Maxim Levitsky
drivers/nvme/host/core.c +8
-0+int nvme_core_register_mdev_driver (struct nvme_mdev_driver *driver_ops) +{ +struct nvme_ctrl *ctrl; + +if (mdev_driver_interface) +return -EEXIST ; + + mdev_driver_interface = driver_ops;Can
mdev_driver_interfacebe accessed from two CPUs at the same time?mdev_driver_interfaceisn't protected by the mutex. Thestate_changedfunctions below also don't protectmdev_driver_interface.
It can be for sure.
However the only time it is updated is when the mdev core module load/unload routines.
On module load the interface flips from NULL to a pointer to inside of the module, so this should be safe, and when mdev module unloads, its reference counter is 0, and all the callers first try to increase it and fail, they don't call using this interface.
I might still be wrong with this reasoning though.
Best regards,
Maxim Levitsky
PATCH 8/9 nvme/core: add nvme-mdev core driver
In reply to Maxim LevitskyThis is main commit in the series, adding the mediated nvme driver.
The idea behind this driver is based on paper you can find at
https://www.usenix.org/conference/atc18/presentation/peng
But this is an independent implementation.
This mdev device exposes a NVME 1.3 virtual device to any VFIO user which represents an partition (or whole namespace) of a host NVME device.
Unlike the paper, the driver/device uses
one polling thread per mediated device,
and only needs one hw queue per it to achieve near native performance (but can use more that one hw queue, in which case it splits the queues between virtual nvme queues thus supporting n:m mapping between host hw queues and the guest virtual queues).
The nvme-mdev can't yet be used after this commit, as no nvme device drivers support mediation (support is added in the next commit to nvme-pci)
The driver can use the shadow doorbell nvme optional feature, to stop polling after a timeout.
Currently the device has redhat pci vendor ID, and 0x1234 pci device id,
which is only a placeholder till a real device id is allocated.
Use example:
# load the nvme-mdev driver $ modprobe nvme-mdev
# load the nvme pci driver with 4 polling queues reserved # (will work with the next patch) $ modprobe nvme mdev_queues=4
# generate random UUID for the mediated device $ UUID=$(uuidgen) $ MDEV_DEVICE=/sys/bus/mdev/devices/$UUID
# the location of the real nvme device (replace with yours) $ PCI_DEVICE=/sys/bus/pci/devices/0000:44:00.0
# create the mediated device using 2 host polling queues $ echo $UUID > $PCI_DEVICE/mdev_supported_types/nvme-2Q_V1/create
# attach partition 1 of namespace 1 to a free virtual namespace # (use n1 to attach whole namespace) # you can attach up to 16 virtual namespaces for now $ echo n1p1 > $MDEV_DEVICE/namespaces/add_namespace
# move the polling thread to cpu 11 $ echo 11 > $MDEV_DEVICE/settings/iothread_cpu
# now you can boot qemu with # -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$UUID
Note that you can attach and detach virtual namespaces even while the guest is running which will make the device sent namespace changed AEN notifications to the guest prior to attach/detach action.
--- MAINTAINERS | 5 + drivers/nvme/Kconfig | 1 + drivers/nvme/Makefile | 1 + drivers/nvme/host/core.c | 5 +- drivers/nvme/mdev/Kconfig | 16 + drivers/nvme/mdev/Makefile | 5 + drivers/nvme/mdev/adm.c | 873 +++++++++++++++++++++++++++++++++++ drivers/nvme/mdev/events.c | 142 ++++++ drivers/nvme/mdev/host.c | 491 ++++++++++++++++++++ drivers/nvme/mdev/instance.c | 802 ++++++++++++++++++++++++++++++++ drivers/nvme/mdev/io.c | 563 ++++++++++++++++++++++ drivers/nvme/mdev/irq.c | 264 +++++++++++ drivers/nvme/mdev/mdev.h | 56 +++ drivers/nvme/mdev/mmio.c | 591 ++++++++++++++++++++++++ drivers/nvme/mdev/pci.c | 247 ++++++++++ drivers/nvme/mdev/priv.h | 700 ++++++++++++++++++++++++++++ drivers/nvme/mdev/udata.c | 390 ++++++++++++++++ drivers/nvme/mdev/vcq.c | 209 +++++++++ drivers/nvme/mdev/vctrl.c | 515 +++++++++++++++++++++ drivers/nvme/mdev/viommu.c | 322 +++++++++++++ drivers/nvme/mdev/vns.c | 356 ++++++++++++++ drivers/nvme/mdev/vsq.c | 181 ++++++++ 22 files changed, 6733 insertions(+), 2 deletions(-) create mode 100644 drivers/nvme/mdev/Kconfig create mode 100644 drivers/nvme/mdev/Makefile create mode 100644 drivers/nvme/mdev/adm.c create mode 100644 drivers/nvme/mdev/events.c create mode 100644 drivers/nvme/mdev/host.c create mode 100644 drivers/nvme/mdev/instance.c create mode 100644 drivers/nvme/mdev/io.c create mode 100644 drivers/nvme/mdev/irq.c create mode 100644 drivers/nvme/mdev/mdev.h create mode 100644 drivers/nvme/mdev/mmio.c create mode 100644 drivers/nvme/mdev/pci.c create mode 100644 drivers/nvme/mdev/priv.h create mode 100644 drivers/nvme/mdev/udata.c create mode 100644 drivers/nvme/mdev/vcq.c create mode 100644 drivers/nvme/mdev/vctrl.c create mode 100644 drivers/nvme/mdev/viommu.c create mode 100644 drivers/nvme/mdev/vns.c create mode 100644 drivers/nvme/mdev/vsq.c
MAINTAINERS +5 -0
@@ -10896,6 +10896,11 @@ W: http://git.infradead.org/nvme.git S: Supported F: drivers/nvme/target/ +NVM EXPRESS MDEV DRIVER +M: Maxim Levitsky <[email protected]> +S: Supported +F: drivers/nvme/mdev/ + NVMEM FRAMEWORK M: Srinivas Kandagatla <[email protected]> S: Maintained
drivers/nvme/Kconfig +1 -0
@@ -2,5 +2,6 @@ menu "NVME Support" source "drivers/nvme/host/Kconfig" source "drivers/nvme/target/Kconfig" +source "drivers/nvme/mdev/Kconfig" endmenu
drivers/nvme/Makefile +1 -0
@@ -1,3 +1,4 @@ obj-y += host/ obj-y += target/ +obj-y += mdev/
drivers/nvme/host/core.c +3 -2
@@ -1687,6 +1687,7 @@ static void nvme_update_disk_info(struct gendisk *disk,if (ns->ms && !ns->ext && (ns->ctrl->ops->flags &NVME_F_METADATA_SUPPORTED ))nvme_init_integrity (disk, ns->ms, ns->pi_type); +if (ns->ms && !nvme_ns_has_pi (ns) && !blk_get_integrity (disk)) capacity =0 ;
@@ -2302,7 +2303,7 @@ static void nvme_init_subnqn(struct nvme_subsystem *subsys, struct nvme_ctrl *ctsize_t nqnlen;int off; -if (!(ctrl->quirks &NVME_QUIRK_IGNORE_DEV_SUBNQN )) { +if (!(ctrl->quirks &NVME_QUIRK_IGNORE_DEV_SUBNQN )) { nqnlen =strnlen (id->subnqn,NVMF_NQN_SIZE );if (nqnlen >0 && nqnlen <NVMF_NQN_SIZE ) {strlcpy (subsys->subnqn, id->subnqn,NVMF_NQN_SIZE );
@@ -3361,8 +3362,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)nvme_mpath_add_disk (ns, id);nvme_fault_inject_init (ns); -kfree (id); +kfree (id);return ; out_put_disk:put_disk (ns->disk);
drivers/nvme/mdev/Kconfig new file +16 -0
@@ -0,0 +1,16 @@ + +config NVME_MDEV + bool + +config NVME_MDEV_VFIO + tristate "NVME Mediated VFIO virtual device" + select NVME_MDEV + depends on BLOCK + depends on VFIO_MDEV + depends on NVME_CORE + help + This provides EXPEREMENTAL support for lightweight software + passthrough of an partition on a NVME storage device to + guest, also as a NVME namespace, attached to a virtual NVME + controller + If unsure, say N.
drivers/nvme/mdev/Makefile new file +5 -0
@@ -0,0 +1,5 @@ + +obj-$(CONFIG_NVME_MDEV_VFIO) += nvme-mdev.o + +nvme-mdev-y += adm.o events.o instance.o host.o io.o irq.o \ + udata.o viommu.o vns.o vsq.o vcq.o vctrl.o mmio.o pci.o
drivers/nvme/mdev/adm.c new file +873 -0
873 lines not on this page · raw message
drivers/nvme/mdev/events.c new file +142 -0
142 lines not on this page · raw message
drivers/nvme/mdev/host.c new file +491 -0
491 lines not on this page · raw message
drivers/nvme/mdev/instance.c new file +802 -0
802 lines not on this page · raw message
drivers/nvme/mdev/io.c new file +563 -0
563 lines not on this page · raw message
drivers/nvme/mdev/irq.c new file +264 -0
63 lines not on this page · raw message
drivers/nvme/mdev/mdev.h new file +56 -0
56 lines not on this page · raw message
drivers/nvme/mdev/mmio.c new file +591 -0
591 lines not on this page · raw message
drivers/nvme/mdev/pci.c new file +247 -0
247 lines not on this page · raw message
drivers/nvme/mdev/priv.h new file +700 -0
700 lines not on this page · raw message
drivers/nvme/mdev/udata.c new file +390 -0
390 lines not on this page · raw message
drivers/nvme/mdev/vcq.c new file +209 -0
209 lines not on this page · raw message
drivers/nvme/mdev/vctrl.c new file +515 -0
515 lines not on this page · raw message
drivers/nvme/mdev/viommu.c new file +322 -0
322 lines not on this page · raw message
drivers/nvme/mdev/vns.c new file +356 -0
356 lines not on this page · raw message
drivers/nvme/mdev/vsq.c new file +181 -0
181 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.17.2
Part of the diff is not shown here. View the raw message.
PATCH 9/9 nvme/pci: implement the mdev external queue allocation interface
In reply to Maxim LevitskyNote that currently the number of hw queues reserved for mdev, has to be pre determined on module load.
(I used to allocate the queues dynamicaly on demand, but recent changes to allocate polled/read queues made this somewhat difficult, so I dropped this for now)
--- drivers/nvme/host/pci.c | 376 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 369 insertions(+), 7 deletions(-)
drivers/nvme/host/pci.c +369 -7
@@ -31,6 +31,7 @@#include <linux/io-64-nonatomic-lo-hi.h> #include <linux/sed-opal.h> #include <linux/pci-p2pdma.h> +#include "../mdev/mdev.h" #include "trace.h" #include "nvme.h"
@@ -40,6 +41,7 @@#define SGES_PER_PAGE (PAGE_SIZE / sizeof(struct nvme_sgl_desc)) +#define USE_SMALL_PRP_POOL (nprps) ((nprps) < (256 / 8))/* * These can be higher, but we need to ensure that any command doesn't * require an sg allocation that needs more than a page of data.
@@ -91,12 +93,24 @@ static int poll_queues = 0;module_param_cb (poll_queues, &queue_count_ops, &poll_queues,0644 );MODULE_PARM_DESC (poll_queues,"Number of queues to use for polled IO." ); +static int mdev_queues; +#ifdef CONFIG_NVME_MDEV +module_param_cb (mdev_queues, &queue_count_ops, &mdev_queues,0644 ); +MODULE_PARM_DESC (mdev_queues,"Number of queues to use for mediated VFIO" ); +#endif +struct nvme_dev ;struct nvme_queue ;static void nvme_dev_disable (struct nvme_dev *dev,bool shutdown);static bool __nvme_disable_io_queues (struct nvme_dev *dev,u8 opcode); +#ifdef CONFIG_NVME_MDEV +static void nvme_ext_queue_reset (struct nvme_dev *dev,u16 qid); +#else +static void nvme_ext_queue_reset (struct nvme_dev *dev,u16 qid) {} +#endif +/* * Represents an NVM Express device. Each nvme_dev is a PCI function. */
@@ -111,6 +125,7 @@ struct nvme_dev {unsigned online_queues;unsigned max_qid;unsigned io_queues[HCTX_MAX_TYPES ]; +unsigned int mdev_queues;unsigned int num_vecs;int q_depth;u32 db_stride;
@@ -118,6 +133,7 @@ struct nvme_dev {unsigned long bar_mapped_size;struct work_struct remove_work;struct mutex shutdown_lock; +struct mutex ext_dev_lock;bool subsystem;u64 cmb_size;bool cmb_use_sqes;
@@ -178,6 +194,16 @@ static inline struct nvme_dev *to_nvme_dev(struct nvme_ctrl *ctrl) return container_of(ctrl, struct nvme_dev, ctrl); } +/* Simplified IO descriptor for MDEV use */ +struct nvme_ext_iod { +struct list_head link; +u32 user_tag; +int nprps; +struct nvme_ext_data_iter *saved_iter; +dma_addr_t first_prplist_dma; +__le64 *prpslists[NVME_MAX_SEGS ]; +}; +/* * An NVM Express queue. Each device has at least two (one for admin * commands and one for I/O commands).
@@ -203,14 +229,25 @@ struct nvme_queue {u16 qid;u8 cq_phase;unsigned long flags; +#define NVMEQ_ENABLED 0#define NVMEQ_SQ_CMB 1#define NVMEQ_DELETE_ERROR 2 +#define NVMEQ_EXTERNAL 4 +u32 *dbbuf_sq_db;u32 *dbbuf_cq_db;u32 *dbbuf_sq_ei;u32 *dbbuf_cq_ei;struct completion delete_done; + +/* queue passthrough for external use */ +struct { +int inflight; +struct nvme_ext_iod *iods; +struct list_head free_iods; +struct list_head used_iods; + } ext; };/*
@@ -255,7 +292,7 @@ static inline void _nvme_check_size(void)static unsigned int max_io_queues (void ) { -return num_possible_cpus () + write_queues + poll_queues; +return num_possible_cpus () + write_queues + poll_queues + mdev_queues; }static unsigned int max_queue_count (void )
@@ -1057,6 +1094,7 @@ static irqreturn_t nvme_irq(int irq, void *data)* the irq handler, even if that was on another CPU. */ rmb (); +if (nvmeq->cq_head != nvmeq->last_cq_head) ret =IRQ_HANDLED ;nvme_process_cq (nvmeq, &start, &end,-1 );
@@ -1167,6 +1205,7 @@ static int adapter_alloc_cq(struct nvme_dev *dev, u16 qid, c.create_cq.cqid =cpu_to_le16 (qid); c.create_cq.qsize =cpu_to_le16 (nvmeq->q_depth -1 ); c.create_cq.cq_flags =cpu_to_le16 (flags); +if (vector !=-1 ) c.create_cq.irq_vector =cpu_to_le16 (vector);else
@@ -1551,7 +1590,11 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)memset ((void *)nvmeq->cqes,0 ,CQ_SIZE (nvmeq->q_depth));nvme_dbbuf_init (dev, nvmeq, qid); dev->online_queues++; +wmb ();/* ensure the first interrupt sees the initialization */ + +if (test_bit (NVMEQ_EXTERNAL , &nvmeq->flags)) +nvme_ext_queue_reset (nvmeq->dev, qid); }static int nvme_create_queue (struct nvme_queue *nvmeq,int qid,bool polled)
@@ -1757,7 +1800,7 @@ static int nvme_create_io_queues(struct nvme_dev *dev) } max =min (dev->max_qid, dev->ctrl.queue_count -1 ); -if (max !=1 && dev->io_queues[HCTX_TYPE_POLL ]) { +if (max !=1 ) { rw_queues = dev->io_queues[HCTX_TYPE_DEFAULT ] + dev->io_queues[HCTX_TYPE_READ ]; }else {
@@ -2094,14 +2137,23 @@ static int nvme_setup_irqs(struct nvme_dev *dev, unsigned int nr_io_queues)* Poll queues don't need interrupts, but we need at least one IO * queue left over for non-polled IO. */ - this_p_queues = poll_queues; + this_p_queues = poll_queues + mdev_queues;if (this_p_queues >= nr_io_queues) { this_p_queues = nr_io_queues -1 ; irq_queues =1 ; }else { irq_queues = nr_io_queues - this_p_queues +1 ; } + +if (mdev_queues > this_p_queues) { + mdev_queues = this_p_queues; + this_p_queues =0 ; + }else { + this_p_queues -= mdev_queues; + } + dev->io_queues[HCTX_TYPE_POLL ] = this_p_queues; + dev->mdev_queues = mdev_queues;/* * For irq sets, we have to ask for minvec == maxvec. This passes
@@ -2208,7 +2260,8 @@ static int nvme_setup_io_queues(struct nvme_dev *dev) dev->num_vecs = result; result =max (result -1 ,1 ); - dev->max_qid = result + dev->io_queues[HCTX_TYPE_POLL ]; + dev->max_qid = result + dev->io_queues[HCTX_TYPE_POLL ] + + dev->mdev_queues;/* * Should investigate if there's a performance win from allocating
@@ -2233,10 +2286,11 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)nvme_suspend_io_queues (dev); goto retry; } -dev_info (dev->ctrl.device,"%d/%d/%d default/read/poll queues\n" , +dev_info (dev->ctrl.device,"%d/%d/%d/%d default/read/poll/mdev queues\n" , dev->io_queues[HCTX_TYPE_DEFAULT ], dev->io_queues[HCTX_TYPE_READ ], - dev->io_queues[HCTX_TYPE_POLL ]); + dev->io_queues[HCTX_TYPE_POLL ], + dev->mdev_queues);return 0 ; }
@@ -2667,6 +2721,301 @@ static void nvme_remove_dead_ctrl_work(struct work_struct *work)nvme_put_ctrl (&dev->ctrl); } +#ifdef CONFIG_NVME_MDEV +static void nvme_ext_free_iod (struct nvme_dev *dev,struct nvme_ext_iod *iod) +{ +int i =0 , max_prp, nprps = iod->nprps; +dma_addr_t dma = iod->first_prplist_dma; + +if (iod->saved_iter) { + iod->saved_iter->release (iod->saved_iter); + iod->saved_iter =NULL ; + } + +if (--nprps <2 ) { + goto out; + }else if (USE_SMALL_PRP_POOL (nprps)) { +dma_pool_free (dev->prp_small_pool, iod->prpslists[0 ], dma); + goto out; + } + + max_prp = (dev->ctrl.page_size >>3 ) -1 ; +while (nprps >0 ) { +if (i >0 ) { + dma = iod->prpslists[i -1 ][max_prp]; +if (nprps ==1 ) +break ; + } +dma_pool_free (dev->prp_page_pool, iod->prpslists[i++], dma); + nprps -= max_prp; + } +out: + iod->nprps =-1 ; + iod->first_prplist_dma =0 ; + iod->user_tag =0xDEADDEAD ; +} + +static int nvme_ext_setup_iod (struct nvme_dev *dev,struct nvme_ext_iod *iod, +struct nvme_common_command *cmd, +struct nvme_ext_data_iter *iter) +{ +int ret, i, j; +__le64 *prp_list; +dma_addr_t prp_dma; +struct dma_pool *pool; +int max_prp = (dev->ctrl.page_size >>3 ) -1 ; + + iod->saved_iter = iter && iter->release ? iter :NULL ; + iod->nprps = iter ? iter->count :0 ; + cmd->dptr.prp1 =0 ; + cmd->dptr.prp2 =0 ; + cmd->metadata =0 ; + +if (!iter) +return 0 ; + +/* put first pointer*/ + cmd->dptr.prp1 =cpu_to_le64 (iter->host_iova); +if (iter->count ==1 ) +return 0 ; + + ret = iter->next (iter); +if (ret) + goto error; + +/* if only have one more pointer, put it to second data pointer*/ +if (iter->count ==1 ) { + cmd->dptr.prp2 =cpu_to_le64 (iter->host_iova); +return 0 ; + } + + pool =USE_SMALL_PRP_POOL (iter->count) ? dev->prp_small_pool : + dev->prp_page_pool; + +/* Allocate prp lists as needed and fill them */ +for (i =0 ; i <NVME_MAX_SEGS && iter->count ; i++) { + prp_list =dma_pool_alloc (pool,GFP_ATOMIC , &prp_dma); +if (!prp_list) { + ret = -ENOMEM ; + goto error; + } + + iod->prpslists[i++] = prp_list; + +if (i ==1 ) { + iod->first_prplist_dma = prp_dma; + cmd->dptr.prp2 =cpu_to_le64 (prp_dma); + j =0 ; + }else { + prp_list[0 ] = iod->prpslists[i -1 ][max_prp]; + iod->prpslists[i -1 ][max_prp] = prp_dma; + j =1 ; + } + +while (j <= max_prp && iter->count) { + prp_list[j++] = iter->host_iova; + ret = iter->next (iter); +if (ret) + goto error; + } + } + +if (iter->count) { + ret = -ENOSPC ; + goto error; + } +return 0 ; +error: + iod->nprps -= iter->count; +nvme_ext_free_iod (dev, iod); +return ret; +} + +static int nvme_ext_queues_available (struct nvme_ctrl *ctrl) +{ +struct nvme_dev *dev =to_nvme_dev (ctrl); +unsigned int ret =0 , qid; +unsigned int first_mdev_q = dev->online_queues - dev->mdev_queues; + +for (qid = first_mdev_q; qid < dev->online_queues; qid++) { +struct nvme_queue *nvmeq = &dev->queues[qid]; + +if (!test_bit (NVMEQ_EXTERNAL , &nvmeq->flags)) + ret++; + } +return ret; +} + +static void nvme_ext_queue_reset (struct nvme_dev *dev,u16 qid) +{ +struct nvme_queue *nvmeq = &dev->queues[qid]; +struct nvme_ext_iod *iod, *tmp; + +list_for_each_entry_safe (iod, tmp, &nvmeq->ext.used_iods, link) { +if (iod->saved_iter && iod->saved_iter->release) { + iod->saved_iter->release (iod->saved_iter); + iod->saved_iter =NULL ; +list_move (&iod->link, &nvmeq->ext.free_iods); + } + } + + nvmeq->ext.inflight =0 ; +} + +static int nvme_ext_queue_alloc (struct nvme_ctrl *ctrl,u16 *ret_qid) +{ +struct nvme_dev *dev =to_nvme_dev (ctrl); +struct nvme_queue *nvmeq; +int ret =0 , qid, i; +unsigned int first_mdev_q = dev->online_queues - dev->mdev_queues; + +mutex_lock (&dev->ext_dev_lock); + +/* find a polled queue to allocate */ +for (qid = dev->online_queues -1 ; qid >= first_mdev_q ; qid--) { + nvmeq = &dev->queues[qid]; +if (!test_bit (NVMEQ_EXTERNAL , &nvmeq->flags)) +break ; + } + +if (qid < first_mdev_q) { + ret = -ENOSPC ; + goto out; + } + +INIT_LIST_HEAD (&nvmeq->ext.free_iods); +INIT_LIST_HEAD (&nvmeq->ext.used_iods); + + nvmeq->ext.iods = +vzalloc_node (sizeof (struct nvme_ext_iod ) * nvmeq->q_depth, +dev_to_node (dev->dev)); + +if (!nvmeq->ext.iods) { + ret = -ENOMEM ; + goto out; + } + +for (i =0 ; i < nvmeq->q_depth ; i++) +list_add_tail (&nvmeq->ext.iods[i].link, &nvmeq->ext.free_iods); + +set_bit (NVMEQ_EXTERNAL , &nvmeq->flags); + *ret_qid = qid; +out: +mutex_unlock (&dev->ext_dev_lock); +return ret; +} + +static void nvme_ext_queue_free (struct nvme_ctrl *ctrl,u16 qid) +{ +struct nvme_dev *dev =to_nvme_dev (ctrl); +struct nvme_queue *nvmeq; + +mutex_lock (&dev->ext_dev_lock); + nvmeq = &dev->queues[qid]; + +if (WARN_ON (!test_bit (NVMEQ_EXTERNAL , &nvmeq->flags))) +return ; + +nvme_ext_queue_reset (dev, qid); + +vfree (nvmeq->ext.iods); + nvmeq->ext.iods =NULL ; +INIT_LIST_HEAD (&nvmeq->ext.free_iods); +INIT_LIST_HEAD (&nvmeq->ext.used_iods); + +clear_bit (NVMEQ_EXTERNAL , &nvmeq->flags); +mutex_unlock (&dev->ext_dev_lock); +} + +static int nvme_ext_queue_submit (struct nvme_ctrl *ctrl,u16 qid,u32 user_tag, +struct nvme_command *command, +struct nvme_ext_data_iter *iter) +{ +struct nvme_dev *dev =to_nvme_dev (ctrl); +struct nvme_queue *nvmeq = &dev->queues[qid]; +struct nvme_ext_iod *iod; +int ret; + +if (WARN_ON (!test_bit (NVMEQ_EXTERNAL , &nvmeq->flags))) +return -EINVAL ; + +if (list_empty (&nvmeq->ext.free_iods)) +return -1 ; + + iod =list_first_entry (&nvmeq->ext.free_iods, + struct nvme_ext_iod, link); + +list_move (&iod->link, &nvmeq->ext.used_iods); + + command->common.command_id =cpu_to_le16 (iod - nvmeq->ext.iods); + iod->user_tag = user_tag; + + ret =nvme_ext_setup_iod (dev, iod, &command->common, iter); +if (ret) { +list_move (&iod->link, &nvmeq->ext.free_iods); +return ret; + } + + nvmeq->ext.inflight++; +nvme_submit_cmd (nvmeq, command, true); +return 0 ; +} + +static int nvme_ext_queue_poll (struct nvme_ctrl *ctrl,u16 qid, +struct nvme_ext_cmd_result *results, +unsigned int max_len) +{ +struct nvme_dev *dev =to_nvme_dev (ctrl); +struct nvme_queue *nvmeq = &dev->queues[qid]; +u16 old_head; +int i, j; + +if (WARN_ON (!test_bit (NVMEQ_EXTERNAL , &nvmeq->flags))) +return -EINVAL ; + +if (nvmeq->ext.inflight ==0 ) +return -1 ; + + old_head = nvmeq->cq_head; + +for (i =0 ;nvme_cqe_pending (nvmeq) && i < max_len ; i++) { +u16 status =le16_to_cpu (nvmeq->cqes[nvmeq->cq_head].status); +u16 tag =le16_to_cpu (nvmeq->cqes[nvmeq->cq_head].command_id); + + results[i].status = status >>1 ; + results[i].tag = (u32 )tag; +nvme_update_cq_head (nvmeq); + } + +if (old_head != nvmeq->cq_head) +nvme_ring_cq_doorbell (nvmeq); + +for (j =0 ; j < i ; j++) { +u16 tag = results[j].tag &0xFFFF ; +struct nvme_ext_iod *iod = &nvmeq->ext.iods[tag]; + +if (WARN_ON (tag >= nvmeq->q_depth || iod->nprps ==-1 )) +continue ; + + results[j].tag = iod->user_tag; +nvme_ext_free_iod (dev, iod); +list_move (&iod->link, &nvmeq->ext.free_iods); + nvmeq->ext.inflight--; + } + +WARN_ON (nvmeq->ext.inflight <0 ); +return i; +} + +static bool nvme_ext_queue_full (struct nvme_ctrl *ctrl,u16 qid) +{ +struct nvme_dev *dev =to_nvme_dev (ctrl); +struct nvme_queue *nvmeq = &dev->queues[qid]; + +return nvmeq->ext.inflight < nvmeq->q_depth -1 ; +} +#endif +static int nvme_pci_reg_read32 (struct nvme_ctrl *ctrl,u32 off,u32 *val) { *val =readl (to_nvme_dev (ctrl)->bar + off);
@@ -2696,13 +3045,25 @@ static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = { .name = "pcie", .module = THIS_MODULE, .flags =NVME_F_METADATA_SUPPORTED | - NVME_F_PCI_P2PDMA, +NVME_F_PCI_P2PDMA | +NVME_F_MDEV_SUPPORTED | + NVME_F_MDEV_DMA_SUPPORTED, + .reg_read32 = nvme_pci_reg_read32, .reg_write32 = nvme_pci_reg_write32, .reg_read64 = nvme_pci_reg_read64, .free_ctrl = nvme_pci_free_ctrl, .submit_async_event = nvme_pci_submit_async_event, .get_address = nvme_pci_get_address, + +#ifdef CONFIG_NVME_MDEV + .ext_queues_available = nvme_ext_queues_available, + .ext_queue_alloc = nvme_ext_queue_alloc, + .ext_queue_free = nvme_ext_queue_free, + .ext_queue_submit = nvme_ext_queue_submit, + .ext_queue_poll = nvme_ext_queue_poll, + .ext_queue_full = nvme_ext_queue_full, +#endif };static int nvme_dev_map (struct nvme_dev *dev)
@@ -2791,6 +3152,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)INIT_WORK (&dev->ctrl.reset_work, nvme_reset_work);INIT_WORK (&dev->remove_work, nvme_remove_dead_ctrl_work);mutex_init (&dev->shutdown_lock); +mutex_init (&dev->ext_dev_lock); result =nvme_setup_prp_pools (dev);if (result)
Signature · 1 line
2.17.2
cover RFC: NVME VFIO mediated device
In reply to Maxim LevitskyOops, I placed the subject in the wrong place.
Best regards,
Maxim Levitsky
Quoted 209 lines · Maxim Levitsky
Date: Tue, 19 Mar 2019 14:45:45 +0200
Subject: [PATCH 0/9] RFC: NVME VFIO mediated deviceHi everyone!
In this patch series, I would like to introduce my take on the problem of doing
as fast as possible virtualization of storage with emphasis on low latency.In this patch series I implemented a kernel vfio based, mediated device that allows the user to pass through a partition and/or whole namespace to a guest.
The idea behind this driver is based on paper you can find at
https://www.usenix.org/conference/atc18/presentation/peng,Although note that I stared the development prior to reading this paper, independently.
In addition to that implementation is not based on code used in the paper as I wasn't being able at that time to make the source available to me.
***Key points about the implementation:***
* Polling kernel thread is used. The polling is stopped after a predefined timeout (1/2 sec by default).
Support for all interrupt driven mode is planned, and it shows promising results.* Guest sees a standard NVME device - this allows to run guest with unmodified drivers, for example windows guests.
* The NVMe device is shared between host and guest. That means that even a single namespace can be split between host and guest based on different partitions.
* Simple configuration
*** Performance ***
Performance was tested on Intel DC P3700, With Xeon E5-2620 v2 and both latency and throughput is very similar to SPDK.
Soon I will test this on a better server and nvme device and provide more formal performance numbers.
Latency numbers:
~80ms - spdk with fio plugin on the host.
~84ms - nvme driver on the host
~87ms - mdev-nvme + nvme driver in the guestThroughput was following similar pattern as well.
* Configuration example
$ modprobe nvme mdev_queues=4 $ modprobe nvme-mdev$ UUID=$(uuidgen) $ DEVICE='device pci address' $ echo $UUID > /sys/bus/pci/devices/$DEVICE/mdev_supported_types/nvme- 2Q_V1/create $ echo n1p3 > /sys/bus/mdev/devices/$UUID/namespaces/add_namespace #attach host namespace 1 parition 3 $ echo 11 > /sys/bus/mdev/devices/$UUID/settings/iothread_cpu #pin the io thread to cpu 11Afterward boot qemu with
-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$UUIDZero configuration on the guest.
*** FAQ ***
* Why to make this in the kernel? Why this is better that SPDK
-> Reuse the existing nvme kernel driver in the host. No new drivers in the
guest.-> Share the NVMe device between host and guest.
Even in fully virtualized configurations,
some partitions of nvme device could be used by guests as block devices while others passed through with nvme-mdev to achieve balance between all features of full IO stack emulation and performance.-> NVME-MDEV is a bit faster due to the fact that in-kernel driver
can send interrupts to the guest directly without a context switch that can be expensive due to meltdown mitigation.-> Is able to utilize interrupts to get reasonable performance.
This is only implemented
as a proof of concept and not included in the patches, but interrupt driven mode shows reasonable performance-> This is a framework that later can be used to support NVMe devices
with more of the IO virtualization built-in
(IOMMU with PASID support coupled with device that supports it)* Why to attach directly to nvme-pci driver and not use block layer IO -> The direct attachment allows for better performance, but I will check the possibility of using block IO, especially for fabrics drivers.
*** Implementation notes ***
* All guest memory is mapped into the physical nvme device but not 1:1 as vfio-pci would do this.
This allows very efficient DMA.
To support this, patch 2 adds ability for a mdev device to listen on guest's memory map events.
Any such memory is immediately pinned and then DMA mapped. (Support for fabric drivers where this is not possible exits too,
in which case the fabric driver will do its own DMA mapping)* nvme core driver is modified to announce the appearance and disappearance of nvme controllers and namespaces, to which the nvme-mdev driver is subscribed.
* nvme-pci driver is modified to expose raw interface of attaching to and sending/polling the IO queues.
This allows the mdev driver very efficiently to submit/poll for the IO. By default one host queue is used per each mediated device. (support for other fabric based host drivers is planned)* The nvme-mdev doesn't assume presence of KVM, thus any VFIO user, including SPDK, a qemu running with tccg, ... can use this virtual device.
*** Testing ***
The device was tested with stock QEMU 3.0 on the host, with host was using 5.0 kernel with nvme-mdev added and the following hardware:
* QEMU nvme virtual device (with nested guest)
* Intel DC P3700 on Xeon E5-2620 v2 server
* Samsung SM981 (in a Thunderbolt enclosure, with my laptop)
* Lenovo NVME device found in my laptopThe guest was tested with kernel 4.16, 4.18, 4.20 and
the same custom complied kernel 5.0
Windows 10 guest was tested too with both Microsoft's inbox driver and open source community NVME driver
(https://lists.openfabrics.org/pipermail/nvmewin/2016-December/001420.html)Testing was mostly done on
x86_64, but 32 bit host/guest combination was lightly tested too.In addition to that, the virtual device was tested with nested guest, by passing the virtual device to it,
using pci passthrough, qemu userspace nvme driver, and spdkPS: I used to contribute to the kernel as a hobby using the
[email protected] addressMaxim Levitsky (9):
vfio/mdev: add .request callback
nvme/core: add some more values from the spec
nvme/core: addNVME_CTRL_SUSPENDEDcontroller state nvme/pci: use theNVME_CTRL_SUSPENDEDstate
nvme/pci: add known admin effects to augument admin effects log page nvme/pci: init shadow doorbell after each reset nvme/core: add mdev interfaces
nvme/core: add nvme-mdev core driver
nvme/pci: implement the mdev external queue allocation interfaceMAINTAINERS | 5 + drivers/nvme/Kconfig | 1 + drivers/nvme/Makefile | 1 + drivers/nvme/host/core.c | 149 +++++- drivers/nvme/host/nvme.h | 55 ++- drivers/nvme/host/pci.c | 385 ++++++++++++++- drivers/nvme/mdev/Kconfig | 16 + drivers/nvme/mdev/Makefile | 5 + drivers/nvme/mdev/adm.c | 873 ++++++++++++++++++++++++++++++++++ drivers/nvme/mdev/events.c | 142 ++++++ drivers/nvme/mdev/host.c | 491 +++++++++++++++++++ drivers/nvme/mdev/instance.c | 802 +++++++++++++++++++++++++++++++ drivers/nvme/mdev/io.c | 563 ++++++++++++++++++++++ drivers/nvme/mdev/irq.c | 264 ++++++++++ drivers/nvme/mdev/mdev.h | 56 +++ drivers/nvme/mdev/mmio.c | 591 +++++++++++++++++++++++ drivers/nvme/mdev/pci.c | 247 ++++++++++ drivers/nvme/mdev/priv.h | 700 +++++++++++++++++++++++++++ drivers/nvme/mdev/udata.c | 390 +++++++++++++++ drivers/nvme/mdev/vcq.c | 207 ++++++++ drivers/nvme/mdev/vctrl.c | 514 ++++++++++++++++++++ drivers/nvme/mdev/viommu.c | 322 +++++++++++++ drivers/nvme/mdev/vns.c | 356 ++++++++++++++ drivers/nvme/mdev/vsq.c | 178 +++++++ drivers/vfio/mdev/vfio_mdev.c | 11 + include/linux/mdev.h | 4 + include/linux/nvme.h | 88 +++- 27 files changed, 7375 insertions(+), 41 deletions(-) create mode 100644 drivers/nvme/mdev/Kconfig create mode 100644 drivers/nvme/mdev/Makefile create mode 100644 drivers/nvme/mdev/adm.c create mode 100644 drivers/nvme/mdev/events.c create mode 100644 drivers/nvme/mdev/host.c create mode 100644 drivers/nvme/mdev/instance.c create mode 100644 drivers/nvme/mdev/io.c create mode 100644 drivers/nvme/mdev/irq.c create mode 100644 drivers/nvme/mdev/mdev.h create mode 100644 drivers/nvme/mdev/mmio.c create mode 100644 drivers/nvme/mdev/pci.c create mode 100644 drivers/nvme/mdev/priv.h create mode 100644 drivers/nvme/mdev/udata.c create mode 100644 drivers/nvme/mdev/vcq.c create mode 100644 drivers/nvme/mdev/vctrl.c create mode 100644 drivers/nvme/mdev/viommu.c create mode 100644 drivers/nvme/mdev/vns.c create mode 100644 drivers/nvme/mdev/vsq.c
Re: [PATCH 0/9] RFC: NVME VFIO mediated device [BENCHMARKS]
Hi
This is first round of benchmarks.
The system is Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
The system has 2 numa nodes, but only cpus and memory from node 0 were used to avoid noise from numa.
The SSD is Intel® Optane™ SSD 900P Series, 280 GB version
** Latency benchmark with no interrupts at all **
spdk was complited with fio plugin in the host and in the guest. spdk was first run in the host
then vm was started with one of spdk,pci passthrough, mdev and inside the vm spdk was run with fio plugin.
spdk was taken from my branch on gitlab, and fio was complied from source for 3.4 branch as needed by the spdk fio plugin.
The following spdk command line was used:
$WORK/fio/fio \ --name=job --runtime=40 --ramp_time=0 --time_based \ --filename="trtype=PCIe traddr=$DEVICE_FOR_FIO ns=1" --ioengine=spdk \ --direct=1 --rw=randread --bs=4K --cpus_allowed=0 \ --iodepth=1 --thread
The average values for slat (submission latency), clat (completion latency) and its sum (slat+clat) were noted.
The results:
spdk fio host:
573 Mib/s - slat 112.00ns, clat 6.400us, lat 6.52ms 573 Mib/s - slat 111.50ns, clat 6.406us, lat 6.52ms
pci passthough host/
spdk fio guest
571 Mib/s - slat 124.56ns, clat 6.422us lat 6.55ms 571 Mib/s - slat 122.86ns, clat 6.410us lat 6.53ms 570 Mib/s - slat 124.95ns, clat 6.425us lat 6.55ms
spdk host/
spdk fio guest:
535 Mib/s - slat 125.00ns, clat 6.895us lat 7.02ms 534 Mib/s - slat 125.36ns, clat 6.896us lat 7.02ms 534 Mib/s - slat 125.82ns, clat 6.892us lat 7.02ms
mdev host/
spdk fio guest:
534 Mib/s - slat 128.04ns, clat 6.902us lat 7.03ms 535 Mib/s - slat 126.97ns, clat 6.900us lat 7.03ms 535 Mib/s - slat 127.00ns, clat 6.898us lat 7.03ms
As you see, native latency is 6.52ms, pci passthrough barely adds any latency, while both mdev/spdk added about (7.03/2 - 6.52) - 0.51ms/0.50ms of latency.
In addtion to that I added few 'rdtsc' into my mdev driver to strategically capture the cycle count it takes it to do 3 things:
1. translate a just received command (till it is copied to the hardware submission queue)
2. receive a completion (divided by the number of completion received in one round of polling)
3. deliver an interupt to the guest (call to eventfd_signal)
This is not the whole latency as there is also a latency between the point the submission entry is written and till it is visible on the polling cpu, plus latency till polling cpu gets to the code which reads the submission entry, and of course latency of interrupt delivery, but the above measurements mostly capture the latency I can control.
The results are:
commands translated : avg cycles: 459.844 avg time(usec): 0.135
commands completed : avg cycles: 354.61 avg time(usec): 0.104
interrupts sent : avg cycles: 590.227 avg time(usec): 0.174
avg time total: 0.413 usec
All measurmenets done in the host kernel. the time calculated using tsc_khz kernel variable.
The biggest take from this is that both spdk and my driver are very fast and overhead is just a thousand of cpu cycles give it or take.
*** Throughput benchmarks ***
https://paste.fedoraproject.org/paste/ecijclLMG2B11MVCVIst-w
Here you can find the throughput benchmarks.
The biggest take is that when using no interrupts (spdk fio in guest or spdk fio in host), the bottelneck is in the device, and througput is about 2290 Mib/s
And mdev vs spdk, with interrupts, my driver sligly wins by giving throughput of about 2015 Mib/s while spdk is about 2005 Mib/s
mostly due to slightly different timings as the latency of both is about the same.
Disabling meltdown mitigation didn't had much effect on the performance.
Best regards,
Maxim Levitsky
Re: [PATCH 0/9] RFC: NVME VFIO mediated device [BENCHMARKS]
Quoted 57 lines · Maxim Levitsky
Hi
This is first round of benchmarks.
The system is Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
The system has 2 numa nodes, but only cpus and memory from node 0 were used to avoid noise from numa.
The SSD is Intel® Optane™ SSD 900P Series, 280 GB version
** Latency benchmark with no interrupts at all **
spdk was complited with fio plugin in the host and in the guest. spdk was first run in the host
then vm was started with one of spdk,pci passthrough, mdev and inside the vm spdk was run with fio plugin.spdk was taken from my branch on gitlab, and fio was complied from source for 3.4 branch as needed by the spdk fio plugin.
The following spdk command line was used:
$WORK/fio/fio \ --name=job --runtime=40 --ramp_time=0 --time_based \ --filename="trtype=PCIe traddr=$DEVICE_FOR_FIO ns=1" --ioengine=spdk \ --direct=1 --rw=randread --bs=4K --cpus_allowed=0 \ --iodepth=1 --threadThe average values for slat (submission latency), clat (completion latency) and its sum (slat+clat) were noted.
The results:
spdk fio host:
573 Mib/s - slat 112.00ns, clat 6.400us, lat 6.52ms 573 Mib/s - slat 111.50ns, clat 6.406us, lat 6.52mspci passthough host/
spdk fio guest571 Mib/s - slat 124.56ns, clat 6.422us lat 6.55ms 571 Mib/s - slat 122.86ns, clat 6.410us lat 6.53ms 570 Mib/s - slat 124.95ns, clat 6.425us lat 6.55msspdk host/
spdk fio guest:535 Mib/s - slat 125.00ns, clat 6.895us lat 7.02ms 534 Mib/s - slat 125.36ns, clat 6.896us lat 7.02ms 534 Mib/s - slat 125.82ns, clat 6.892us lat 7.02msmdev host/
spdk fio guest:
Quoted 7 lines
534 Mib/s - slat 128.04ns, clat 6.902us lat 7.03ms 535 Mib/s - slat 126.97ns, clat 6.900us lat 7.03ms 535 Mib/s - slat 127.00ns, clat 6.898us lat 7.03msAs you see, native latency is 6.52ms, pci passthrough barely adds any latency, while both mdev/spdk added about (7.03/2 - 6.52) - 0.51ms/0.50ms of latency.
Milliseconds is surprising. The SSD's spec says 10us read/write latency. Did you mean microseconds?
Quoted 24 lines
In addtion to that I added few 'rdtsc' into my mdev driver to strategically capture the cycle count it takes it to do 3 things:
1. translate a just received command (till it is copied to the hardware submission queue)
2. receive a completion (divided by the number of completion received in one round of polling)
3. deliver an interupt to the guest (call to
eventfd_signal)This is not the whole latency as there is also a latency between the point the submission entry is written and till it is visible on the polling cpu, plus latency till polling cpu gets to the code which reads the submission entry, and of course latency of interrupt delivery, but the above measurements mostly capture the latency I can control.
The results are:
commands translated : avg cycles: 459.844 avg time(usec): 0.135
commands completed : avg cycles: 354.61 avg time(usec): 0.104
interrupts sent : avg cycles: 590.227 avg time(usec): 0.174
Quoted 7 lines
avg time total: 0.413 usec
All measurmenets done in the host kernel. the time calculated using
tsc_khzkernel variable.The biggest take from this is that both spdk and my driver are very fast and overhead is just a thousand of cpu cycles give it or take.
Nice!
Stefan
Re: [PATCH 0/9] RFC: NVME VFIO mediated device [BENCHMARKS]
Quoted 63 lines · Stefan Hajnoczi
Quoted 62 lines · Maxim Levitsky
Hi
This is first round of benchmarks.
The system is Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
The system has 2 numa nodes, but only cpus and memory from node 0 were used to avoid noise from numa.
The SSD is Intel® Optane™ SSD 900P Series, 280 GB version
** Latency benchmark with no interrupts at all **
spdk was complited with fio plugin in the host and in the guest. spdk was first run in the host
then vm was started with one of spdk,pci passthrough, mdev and inside the vm spdk was run with fio plugin.spdk was taken from my branch on gitlab, and fio was complied from source for 3.4 branch as needed by the spdk fio plugin.
The following spdk command line was used:
$WORK/fio/fio \ --name=job --runtime=40 --ramp_time=0 --time_based \ --filename="trtype=PCIe traddr=$DEVICE_FOR_FIO ns=1" --ioengine=spdk \ --direct=1 --rw=randread --bs=4K --cpus_allowed=0 \ --iodepth=1 --threadThe average values for slat (submission latency), clat (completion latency) and its sum (slat+clat) were noted.
The results:
spdk fio host:
573 Mib/s - slat 112.00ns, clat 6.400us, lat 6.52ms 573 Mib/s - slat 111.50ns, clat 6.406us, lat 6.52mspci passthough host/
spdk fio guest571 Mib/s - slat 124.56ns, clat 6.422us lat 6.55ms 571 Mib/s - slat 122.86ns, clat 6.410us lat 6.53ms 570 Mib/s - slat 124.95ns, clat 6.425us lat 6.55msspdk host/
spdk fio guest:535 Mib/s - slat 125.00ns, clat 6.895us lat 7.02ms 534 Mib/s - slat 125.36ns, clat 6.896us lat 7.02ms 534 Mib/s - slat 125.82ns, clat 6.892us lat 7.02msmdev host/
spdk fio guest:534 Mib/s - slat 128.04ns, clat 6.902us lat 7.03ms 535 Mib/s - slat 126.97ns, clat 6.900us lat 7.03ms 535 Mib/s - slat 127.00ns, clat 6.898us lat 7.03ms
Quoted 5 lines
Quoted 2 lines
As you see, native latency is 6.52ms, pci passthrough barely adds any latency, while both mdev/spdk added about (7.03/2 - 6.52) - 0.51ms/0.50ms of latency.
Milliseconds is surprising. The SSD's spec says 10us read/write latency. Did you mean microseconds?
Yea, this is typo - all of this is microseconds.
Quoted 30 lines
Quoted 30 lines
In addtion to that I added few 'rdtsc' into my mdev driver to strategically capture the cycle count it takes it to do 3 things:
1. translate a just received command (till it is copied to the hardware submission queue)
2. receive a completion (divided by the number of completion received in one round of polling)
3. deliver an interupt to the guest (call to
eventfd_signal)This is not the whole latency as there is also a latency between the point the submission entry is written and till it is visible on the polling cpu, plus latency till polling cpu gets to the code which reads the submission entry, and of course latency of interrupt delivery, but the above measurements mostly capture the latency I can control.
The results are:
commands translated : avg cycles: 459.844 avg time(usec): 0.135
commands completed : avg cycles: 354.61 avg time(usec): 0.104
interrupts sent : avg cycles: 590.227 avg time(usec): 0.174avg time total: 0.413 usec
All measurmenets done in the host kernel. the time calculated using
tsc_khzkernel variable.
Quoted 6 lines
Quoted 2 lines
The biggest take from this is that both spdk and my driver are very fast and overhead is just a thousand of cpu cycles give it or take.
Nice!
Stefan
Best regards,
Maxim Levitsky
Keith Busch
maintainerRe: your mail
In reply to Maxim LevitskyQuoted 10 lines · Maxim Levitsky
-> Share the NVMe device between host and guest.
Even in fully virtualized configurations,
some partitions of nvme device could be used by guests as block devices while others passed through with nvme-mdev to achieve balance between all features of full IO stack emulation and performance.-> NVME-MDEV is a bit faster due to the fact that in-kernel driver
can send interrupts to the guest directly without a context switch that can be expensive due to meltdown mitigation.
Quoted 8 lines
-> Is able to utilize interrupts to get reasonable performance.
This is only implemented
as a proof of concept and not included in the patches, but interrupt driven mode shows reasonable performance-> This is a framework that later can be used to support NVMe devices
with more of the IO virtualization built-in
(IOMMU with PASID support coupled with device that supports it)
Would be very interested to see the PASID support. You wouldn't even need to mediate the IO doorbells or translations if assigning entire namespaces, and should be much faster than the shadow doorbells.
I think you should send 6/9 "nvme/pci: init shadow doorbell after each reset" separately for immediate inclusion.
I like the idea in principle, but it will take me a little time to get through reviewing your implementation. I would have guessed we could have leveraged something from the existing nvme/target for the mediating controller register access and admin commands. Maybe even start with implementing an nvme passthrough namespace target type (we currently have block and file).
Re: your mail
Hi Keith,
Quoted 27 lines · Keith Busch
Quoted 10 lines · Maxim Levitsky
-> Share the NVMe device between host and guest.
Even in fully virtualized configurations,
some partitions of nvme device could be used by guests as block devices while others passed through with nvme-mdev to achieve balance between all features of full IO stack emulation and performance.-> NVME-MDEV is a bit faster due to the fact that in-kernel driver
can send interrupts to the guest directly without a context switch that can be expensive due to meltdown mitigation.Quoted 8 lines
-> Is able to utilize interrupts to get reasonable performance.
This is only implemented
as a proof of concept and not included in the patches, but interrupt driven mode shows reasonable performance-> This is a framework that later can be used to support NVMe devices
with more of the IO virtualization built-in
(IOMMU with PASID support coupled with device that supports it)Would be very interested to see the PASID support. You wouldn't even need to mediate the IO doorbells or translations if assigning entire namespaces, and should be much faster than the shadow doorbells.
I think you should send 6/9 "nvme/pci: init shadow doorbell after each reset" separately for immediate inclusion.
Quoted 6 lines
I like the idea in principle, but it will take me a little time to get through reviewing your implementation. I would have guessed we could have leveraged something from the existing nvme/target for the mediating controller register access and admin commands. Maybe even start with implementing an nvme passthrough namespace target type (we currently have block and file).
I have the code for the NVMeOf target passthru-ctrl, I think we can use that as it is if you are looking for the passthru for NVMeOF.
I'll post patch-series based on the latest code base soon.
Quoted 6 lines
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: your mail
Quoted 38 lines · Chaitanya Kulkarni
Hi Keith,
Quoted 29 lines · Keith Busch
Quoted 12 lines · Maxim Levitsky
-> Share the NVMe device between host and guest.
Even in fully virtualized configurations,
some partitions of nvme device could be used by guests as block
devices
while others passed through with nvme-mdev to achieve balance
between
all features of full IO stack emulation and performance.-> NVME-MDEV is a bit faster due to the fact that in-kernel driver
can send interrupts to the guest directly without a context switch that can be expensive due to meltdown mitigation.Quoted 8 lines
-> Is able to utilize interrupts to get reasonable performance.
This is only implemented
as a proof of concept and not included in the patches, but interrupt driven mode shows reasonable performance-> This is a framework that later can be used to support NVMe devices
with more of the IO virtualization built-in
(IOMMU with PASID support coupled with device that supports it)Would be very interested to see the PASID support. You wouldn't even need to mediate the IO doorbells or translations if assigning entire namespaces, and should be much faster than the shadow doorbells.
I think you should send 6/9 "nvme/pci: init shadow doorbell after each reset" separately for immediate inclusion.
Quoted 6 lines
I like the idea in principle, but it will take me a little time to get through reviewing your implementation. I would have guessed we could have leveraged something from the existing nvme/target for the mediating controller register access and admin commands. Maybe even start with implementing an nvme passthrough namespace target type (we currently have block and file).
Quoted 4 lines
I have the code for the NVMeOf target passthru-ctrl, I think we can use that as it is if you are looking for the passthru for NVMeOF.
I'll post patch-series based on the latest code base soon.
I am very intersing in this code.
Could you explain how your NVMeOF target passthrough works? Which components of the NVME stack does it involve?
Best regards,
Maxim Levitsky
Quoted 12 lines
Quoted 6 lines
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: your mail
In reply to Keith BuschQuoted 17 lines · Keith Busch
Quoted 16 lines · Maxim Levitsky
-> Share the NVMe device between host and guest.
Even in fully virtualized configurations,
some partitions of nvme device could be used by guests as block
devices
while others passed through with nvme-mdev to achieve balance between all features of full IO stack emulation and performance.-> NVME-MDEV is a bit faster due to the fact that in-kernel driver
can send interrupts to the guest directly without a context switch that can be expensive due to meltdown mitigation.-> Is able to utilize interrupts to get reasonable performance.
This is only implemented
as a proof of concept and not included in the patches, but interrupt driven mode shows reasonable performance
Quoted 8 lines
Quoted 3 lines
-> This is a framework that later can be used to support NVMe devices
with more of the IO virtualization built-in
(IOMMU with PASID support coupled with device that supports it)Would be very interested to see the PASID support. You wouldn't even need to mediate the IO doorbells or translations if assigning entire namespaces, and should be much faster than the shadow doorbells.
I fully agree with that.
Note that to enable PASID support two things have to happen in this vendor.
1. Mature support for IOMMU with PASID support. On Intel side I know that they only have a spec released and currently the kernel bits to support it are placed.
I still don't know when a product actually supporting this spec is going to be released. For other vendors (ARM/AMD/) I haven't done yet a research on state of PASID based IOMMU support on their platforms.
2. NVMe spec has to be extended to support PASID. At minimum, we need an ability to assign an PASID to a sq/cq queue pair and ability to relocate the doorbells, such as each guest would get its own (hardware backed) MMIO page with its own doorbells. Plus of course the hardware vendors have to embrace the spec. I guess these two things will happen in collaborative manner.
Quoted 3 lines
I think you should send 6/9 "nvme/pci: init shadow doorbell after each reset" separately for immediate inclusion.
I'll do this soon.
Also '5/9 nvme/pci: add known admin effects to augment admin effects log page' can be considered for immediate inclusion as well, as it works around a flaw in the NVMe controller badly done admin side effects page with no side effects (pun intended) for spec compliant controllers (I think so).
This can be fixed with a quirk if you prefer though.
Quoted 7 lines
I like the idea in principle, but it will take me a little time to get through reviewing your implementation. I would have guessed we could have leveraged something from the existing nvme/target for the mediating controller register access and admin commands. Maybe even start with implementing an nvme passthrough namespace target type (we currently have block and file).
I fully agree with you on that I could have used some of the nvme/target code, and I am planning to do so eventually.
For that I would need to make my driver, to be one of the target drivers, and I would need to add another target back end, like you said to allow my target driver to talk directly to the nvme hardware bypassing the block layer.
Or instead I can use the block backend,
(but note that currently the block back-end doesn't support polling which is critical for the performance).
Switch to the target code might though have some (probably minor) performance impact, as it would probably lengthen the critical code path a bit (I might need for instance to translate the PRP lists I am getting from the virtual controller to a scattergather list and back).
This is why I did this the way I did, but now knowing that probably I can afford to loose a bit of performance, I can look at doing that.
Best regards,
Thanks in advance for the review,
Maxim Levitsky
PS:
For reference currently the IO path looks more or less like that:
My IO thread notices a doorbell write, reads a command from a submission queue, translates it (without even looking at the data pointer) and sends it to the nvme pci driver together with pointer to data iterator'.
The nvme pci driver calls the data iterator N times, which makes the iterator translate and fetch the DMA addresses where the data is already mapped on the its pci nvme device (the mdev driver maps all the guest memory to the nvme pci device).
The nvme pci driver uses these addresses it receives, to create a prp list, which it puts into the data pointer.
The nvme pci driver also allocates an free command id, from a list, puts it into the command ID and sends the command to the real hardware.
Later the IO thread calls to the nvme pci driver to poll the queue. When completions arrive, the nvme pci driver returns them back to the IO thread.
Quoted 5 lines
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerRe: your mail
Quoted 3 lines · Maxim Levitsky
Or instead I can use the block backend,
(but note that currently the block back-end doesn't support polling which is critical for the performance).
Oh, I think you can do polling through there. For reference, fs/io_uring.c has a pretty good implementation that aligns with how you could use it.
Re: your mail
Quoted 7 lines · Keith Busch
Quoted 3 lines · Maxim Levitsky
Or instead I can use the block backend,
(but note that currently the block back-end doesn't support polling which is critical for the performance).Oh, I think you can do polling through there. For reference,
fs/io_uring.chas a pretty good implementation that aligns with how you could use it.
That is exactly my thought. The polling recently got lot of improvements in the block layer, which migh make this feasable.
I will give it a try.
Best regards,
Maxim Levitsky
Re: your mail
In reply to Keith BuschQuoted 28 lines · Keith Busch
Quoted 11 lines · Maxim Levitsky
-> Share the NVMe device between host and guest.
Even in fully virtualized configurations,
some partitions of nvme device could be used by guests as block
devices
while others passed through with nvme-mdev to achieve balance between all features of full IO stack emulation and performance.-> NVME-MDEV is a bit faster due to the fact that in-kernel driver
can send interrupts to the guest directly without a context switch that can be expensive due to meltdown mitigation.Quoted 8 lines
-> Is able to utilize interrupts to get reasonable performance.
This is only implemented
as a proof of concept and not included in the patches, but interrupt driven mode shows reasonable performance-> This is a framework that later can be used to support NVMe devices
with more of the IO virtualization built-in
(IOMMU with PASID support coupled with device that supports it)Would be very interested to see the PASID support. You wouldn't even need to mediate the IO doorbells or translations if assigning entire namespaces, and should be much faster than the shadow doorbells.
I think you should send 6/9 "nvme/pci: init shadow doorbell after each reset" separately for immediate inclusion.
Quoted 6 lines
I like the idea in principle, but it will take me a little time to get through reviewing your implementation. I would have guessed we could have leveraged something from the existing nvme/target for the mediating controller register access and admin commands. Maybe even start with implementing an nvme passthrough namespace target type (we currently have block and file).
Hi!
Sorry to bother you, but any update?
I was somewhat sick for the last week, now finally back in shape to continue working on this and other tasks I have.
I am studing now the nvme target code and the io_uring to evaluate the difficultiy of using something similiar to talk to the block device instead of / in addtion to the direct connection I implemented.
I would be glad to hear more feedback on this project.
I will also soon post the few fixes separately as you suggested.
Best regards,
Maxim Levitskky
Re:
In reply to Maxim LevitskyQuoted 6 lines · Maxim Levitsky
Date: Tue, 19 Mar 2019 14:45:45 +0200
Subject: [PATCH 0/9] RFC: NVME VFIO mediated device
Quoted 7 lines
Hi everyone!
In this patch series, I would like to introduce my take on the problem of doing as fast as possible virtualization of storage with emphasis on low latency.
In this patch series I implemented a kernel vfio based, mediated device that allows the user to pass through a partition and/or whole namespace to a guest.
Hey Maxim!
I'm really excited to see this series, as it aligns to some extent with what we discussed in last year's KVM Forum VFIO BoF.
There's no arguing that we need a better story to efficiently virtualise NVMe devices. So far, for Qemu-based VMs, Changpeng's vhost-user-nvme is the best attempt at that. However, I seem to recall there was some pushback from qemu-devel in the sense that they would rather see investment in virtio-blk. I'm not sure what's the latest on that work and what are the next steps.
The pushback drove the discussion towards pursuing an mdev approach, which is why I'm excited to see your patches.
What I'm thinking is that passing through namespaces or partitions is very restrictive. It leaves no room to implement more elaborate virtualisation stacks like replicating data across multiple devices (local or remote), storage migration, software-managed thin provisioning, encryption, deduplication, compression, etc. In summary, anything that requires software intervention in the datapath. (Worth noting: vhost-user-nvme allows all of that to be easily done in SPDK's bdev layer.)
These complicated stacks should probably not be implemented in the kernel, though. So I'm wondering whether we could talk about mechanisms to allow efficient and performant userspace datapath intervention in your approach or pursue a mechanism to completely offload the device emulation to userspace (and align with what SPDK has to offer).
Thoughts welcome!
Felipe
Re:
Quoted 18 lines · Felipe Franciosi
Quoted 7 lines · Maxim Levitsky
Date: Tue, 19 Mar 2019 14:45:45 +0200
Subject: [PATCH 0/9] RFC: NVME VFIO mediated deviceHi everyone!
Quoted 8 lines
In this patch series, I would like to introduce my take on the problem of doing
as fast as possible virtualization of storage with emphasis on low latency.In this patch series I implemented a kernel vfio based, mediated device that
allows the user to pass through a partition and/or whole namespace to a guest.Hey Maxim!
Quoted 8 lines
I'm really excited to see this series, as it aligns to some extent with what we discussed in last year's KVM Forum VFIO BoF.
There's no arguing that we need a better story to efficiently virtualise NVMe devices. So far, for Qemu-based VMs, Changpeng's vhost-user-nvme is the best attempt at that. However, I seem to recall there was some pushback from qemu- devel in the sense that they would rather see investment in virtio-blk. I'm not sure what's the latest on that work and what are the next steps.
I agree with that. All my benchmarks were agains his vhost-user-nvme driver, and I am able to get pretty much the same througput and latency.
The ssd I tested on died just recently (Murphy law), not due to bug in my driver but some internal fault (even though most of my tests were reads, plus occassional 'nvme format's.
We are in process of buying an replacement.
Quoted 11 lines
The pushback drove the discussion towards pursuing an mdev approach, which is why I'm excited to see your patches.
What I'm thinking is that passing through namespaces or partitions is very restrictive. It leaves no room to implement more elaborate virtualisation stacks like replicating data across multiple devices (local or remote), storage migration, software-managed thin provisioning, encryption, deduplication, compression, etc. In summary, anything that requires software intervention in the datapath. (Worth noting: vhost-user-nvme allows all of that to be easily done in SPDK's bdev layer.)
Hi Felipe!
I guess that my driver is not geared toward more complicated use cases like you mentioned, but instead it is focused to get as fast as possible performance for the common case.
One thing that I can do which would solve several of the above problems is to accept an map betwent virtual and real logical blocks, pretty much in exactly the same way as EPT does it.
Then userspace can map any portions of the device anywhere, while still keeping the dataplane in the kernel, and having minimal overhead.
On top of that, note that the direction of IO virtualization is to do dataplane in hardware, which will probably give you even worse partition granuality / features but will be the fastest option aviable,
like for instance SR-IOV which alrady exists and just allows to split by namespaces without any more fine grained control.
Think of nvme-mdev as a very low level driver, which currntly uses polling, but eventually will use PASID based IOMMU to provide the guest with raw PCI device. The userspace / qemu can build on top of that with varios software layers.
On top of that I am thinking to solve the problem of migration in Qemu, by creating a 'vfio-nvme' driver which would bind vfio to bind to device exposed by the kernel, and would pass through all the doorbells and queues to the guest, while intercepting the admin queue. Such driver I think can be made to support migration while beeing able to run on top both SR-IOV device, my vfio-nvme abit with double admin queue emulation (its a bit ugly but won't affect performance at all) and on top of even regular NVME device vfio assigned to guest.
Best regards,
Maxim Levitsky
Quoted 13 lines
These complicated stacks should probably not be implemented in the kernel, though. So I'm wondering whether we could talk about mechanisms to allow efficient and performant userspace datapath intervention in your approach or pursue a mechanism to completely offload the device emulation to userspace (and align with what SPDK has to offer).
Thoughts welcome!
Felipe
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re:
Quoted 69 lines · Maxim Levitsky
Quoted 18 lines · Felipe Franciosi
Quoted 7 lines · Maxim Levitsky
Date: Tue, 19 Mar 2019 14:45:45 +0200
Subject: [PATCH 0/9] RFC: NVME VFIO mediated deviceHi everyone!
Quoted 8 lines
In this patch series, I would like to introduce my take on the problem of doing
as fast as possible virtualization of storage with emphasis on low latency.In this patch series I implemented a kernel vfio based, mediated device that
allows the user to pass through a partition and/or whole namespace to a guest.Hey Maxim!
Quoted 8 lines
I'm really excited to see this series, as it aligns to some extent with what we discussed in last year's KVM Forum VFIO BoF.
There's no arguing that we need a better story to efficiently virtualise NVMe devices. So far, for Qemu-based VMs, Changpeng's vhost-user-nvme is the best attempt at that. However, I seem to recall there was some pushback from qemu- devel in the sense that they would rather see investment in virtio-blk. I'm not sure what's the latest on that work and what are the next steps.
I agree with that. All my benchmarks were agains his vhost-user-nvme driver, and I am able to get pretty much the same througput and latency.
The ssd I tested on died just recently (Murphy law), not due to bug in my driver but some internal fault (even though most of my tests were reads, plus occassional 'nvme format's.
We are in process of buying an replacement.Quoted 11 lines
The pushback drove the discussion towards pursuing an mdev approach, which is why I'm excited to see your patches.
What I'm thinking is that passing through namespaces or partitions is very restrictive. It leaves no room to implement more elaborate virtualisation stacks like replicating data across multiple devices (local or remote), storage migration, software-managed thin provisioning, encryption, deduplication, compression, etc. In summary, anything that requires software intervention in the datapath. (Worth noting: vhost-user-nvme allows all of that to be easily done in SPDK's bdev layer.)
Hi Felipe!
I guess that my driver is not geared toward more complicated use cases like you mentioned, but instead it is focused to get as fast as possible performance for the common case.
One thing that I can do which would solve several of the above problems is to accept an map betwent virtual and real logical blocks, pretty much in exactly the same way as EPT does it.
Then userspace can map any portions of the device anywhere, while still keeping the dataplane in the kernel, and having minimal overhead.On top of that, note that the direction of IO virtualization is to do dataplane in hardware, which will probably give you even worse partition granuality / features but will be the fastest option aviable,
like for instance SR-IOV which alrady exists and just allows to split by namespaces without any more fine grained control.Think of nvme-mdev as a very low level driver, which currntly uses polling, but eventually will use PASID based IOMMU to provide the guest with raw PCI device. The userspace / qemu can build on top of that with varios software layers.
Quoted 7 lines
On top of that I am thinking to solve the problem of migration in Qemu, by creating a 'vfio-nvme' driver which would bind vfio to bind to device exposed by the kernel, and would pass through all the doorbells and queues to the guest, while intercepting the admin queue. Such driver I think can be made to support migration while beeing able to run on top both SR-IOV device, my vfio-nvme abit with double admin queue emulation (its a bit ugly but won't affect performance at all) and on top of even regular NVME device vfio assigned to guest.
mdev-nvme seems like a duplication of SPDK. The performance is not better and the features are more limited, so why focus on this approach?
One argument might be that the kernel NVMe subsystem wants to offer this functionality and loading the kernel module is more convenient than managing SPDK to some users.
Thoughts?
Stefan
Keith Busch
maintainerRe:
Quoted 8 lines · Stefan Hajnoczi
mdev-nvme seems like a duplication of SPDK. The performance is not better and the features are more limited, so why focus on this approach?
One argument might be that the kernel NVMe subsystem wants to offer this functionality and loading the kernel module is more convenient than managing SPDK to some users.
Thoughts?
Doesn't SPDK bind a controller to a single process? mdev binds to namespaces (or their partitions), so you could have many mdev's assigned to many VMs accessing a single controller.
Re:
Quoted 12 lines · Keith Busch
Quoted 7 lines · Stefan Hajnoczi
mdev-nvme seems like a duplication of SPDK. The performance is not better and the features are more limited, so why focus on this approach?
One argument might be that the kernel NVMe subsystem wants to offer this functionality and loading the kernel module is more convenient than managing SPDK to some users.
Quoted 5 lines
Quoted 1 line
Thoughts?
Doesn't SPDK bind a controller to a single process? mdev binds to namespaces (or their partitions), so you could have many mdev's assigned to many VMs accessing a single controller.
Yes, it binds to a single process which can drive the datapath of multiple virtual controllers for multiple VMs (similar to what you described for mdev). You can therefore efficiently poll multiple VM submission queues (and multiple device completion queues) from a single physical CPU.
The same could be done in the kernel, but the code gets complicated as you add more functionality to it. As this is a direct interface with an untrusted front-end (the guest), it's also arguably safer to do in userspace.
Worth noting: you can eventually have a single physical core polling all sorts of virtual devices (eg. virtual storage or network controllers) very efficiently. And this is quite configurable, too. In the interest of fairness, performance or efficiency, you can choose to dynamically add or remove queues to the poll thread or spawn more threads and redistribute the work.
F.
Re:
Quoted 25 lines · Felipe Franciosi
Quoted 10 lines · Keith Busch
Quoted 7 lines · Stefan Hajnoczi
mdev-nvme seems like a duplication of SPDK. The performance is not better and the features are more limited, so why focus on this approach?
One argument might be that the kernel NVMe subsystem wants to offer this functionality and loading the kernel module is more convenient than managing SPDK to some users.
Quoted 5 lines
Quoted 1 line
Thoughts?
Doesn't SPDK bind a controller to a single process? mdev binds to namespaces (or their partitions), so you could have many mdev's assigned to many VMs accessing a single controller.
Yes, it binds to a single process which can drive the datapath of multiple virtual controllers for multiple VMs (similar to what you described for mdev). You can therefore efficiently poll multiple VM submission queues (and multiple device completion queues) from a single physical CPU.
The same could be done in the kernel, but the code gets complicated as you add more functionality to it. As this is a direct interface with an untrusted front-end (the guest), it's also arguably safer to do in userspace.
Quoted 7 lines
Worth noting: you can eventually have a single physical core polling all sorts of virtual devices (eg. virtual storage or network controllers) very efficiently. And this is quite configurable, too. In the interest of fairness, performance or efficiency, you can choose to dynamically add or remove queues to the poll thread or spawn more threads and redistribute the work.
F.
Note though that SPDK doesn't support sharing the device between host and the guests, it takes over the nvme device, thus it makes the kernel nvme driver unbind from it.
My driver creates a polling thread per guest, but its trivial to add option to use the same polling thread for many guests if there need for that.
Best regards,
Maxim Levitsky
Re:
Quoted 36 lines · Maxim Levitsky
Quoted 31 lines · Felipe Franciosi
Quoted 10 lines · Keith Busch
Quoted 7 lines · Stefan Hajnoczi
mdev-nvme seems like a duplication of SPDK. The performance is not better and the features are more limited, so why focus on this approach?
One argument might be that the kernel NVMe subsystem wants to offer this functionality and loading the kernel module is more convenient than managing SPDK to some users.
Quoted 5 lines
Quoted 1 line
Thoughts?
Doesn't SPDK bind a controller to a single process? mdev binds to namespaces (or their partitions), so you could have many mdev's assigned to many VMs accessing a single controller.
Yes, it binds to a single process which can drive the datapath of multiple virtual controllers for multiple VMs (similar to what you described for mdev). You can therefore efficiently poll multiple VM submission queues (and multiple device completion queues) from a single physical CPU.
The same could be done in the kernel, but the code gets complicated as you add more functionality to it. As this is a direct interface with an untrusted front-end (the guest), it's also arguably safer to do in userspace.
Worth noting: you can eventually have a single physical core polling all sorts of virtual devices (eg. virtual storage or network controllers) very efficiently. And this is quite configurable, too. In the interest of fairness, performance or efficiency, you can choose to dynamically add or remove queues to the poll thread or spawn more threads and redistribute the work.
Quoted 5 lines
Quoted 1 line
F.
Note though that SPDK doesn't support sharing the device between host and the guests, it takes over the nvme device, thus it makes the kernel nvme driver unbind from it.
That is absolutely true. However, I find it not to be a problem in practice.
Hypervisor products, specially those caring about performance, efficiency and fairness, will dedicate NVMe devices for a particular purpose (eg. vDisk storage, cache, metadata) and will not share these devices for other use cases. That's because these products want to deterministically control the performance aspects of the device, which you just cannot do if you are sharing the device with a subsystem you do not control.
For scenarios where the device must be shared and such fine grained control is not required, it looks like using the kernel driver with io_uring offers very good performance with flexibility.
Cheers,
Felipe
Re:
Quoted 58 lines · Felipe Franciosi
Quoted 42 lines · Maxim Levitsky
Quoted 39 lines · Felipe Franciosi
Quoted 12 lines · Keith Busch
Quoted 9 lines · Stefan Hajnoczi
mdev-nvme seems like a duplication of SPDK. The performance is not better and the features are more limited, so why focus on this approach?
One argument might be that the kernel NVMe subsystem wants to offer this
functionality and loading the kernel module is more convenient than managing SPDK to some users.Quoted 5 lines
Quoted 1 line
Thoughts?
Doesn't SPDK bind a controller to a single process? mdev binds to namespaces (or their partitions), so you could have many mdev's assigned to many VMs accessing a single controller.
Yes, it binds to a single process which can drive the datapath of multiple virtual controllers for multiple VMs (similar to what you described for mdev).
You can therefore efficiently poll multiple VM submission queues (and multiple
device completion queues) from a single physical CPU.The same could be done in the kernel, but the code gets complicated as you add
more functionality to it. As this is a direct interface with an untrusted front-end (the guest), it's also arguably safer to do in userspace.Worth noting: you can eventually have a single physical core polling all sorts
of virtual devices (eg. virtual storage or network controllers) very efficiently. And this is quite configurable, too. In the interest of fairness,
performance or efficiency, you can choose to dynamically add or remove queues
to the poll thread or spawn more threads and redistribute the work.Quoted 6 lines
Quoted 1 line
F.
Note though that SPDK doesn't support sharing the device between host and the
guests, it takes over the nvme device, thus it makes the kernel nvme driver unbind from it.That is absolutely true. However, I find it not to be a problem in practice.
Hypervisor products, specially those caring about performance, efficiency and fairness, will dedicate NVMe devices for a particular purpose (eg. vDisk storage, cache, metadata) and will not share these devices for other use cases. That's because these products want to deterministically control the performance aspects of the device, which you just cannot do if you are sharing the device with a subsystem you do not control.
Quoted 3 lines
For scenarios where the device must be shared and such fine grained control is not required, it looks like using the kernel driver with
io_uringoffers very good performance with flexibility
I see the host/guest parition in the following way:
The guest assigned partitions are for guests that need lowest possible latency, and in between these guests it is possible to guarantee good enough level of fairness in my driver.
For example, in the current implementation of my driver, each guest gets its own host submission queue.
On the other hand, the host assigned partitions are for significantly higher latency IO, with no guarantees, and/or for guests that need all the more advanced features of full IO virtualization, for instance snapshots, thin provisioning, replication/backup over network, etc.io_uring can be used here to speed things up but it won't reach the nvme-mdev levels of latency.
Furthermore on NVME drives that support WRRU, its possible to let queues of guest's assigned partitions to belong to the high priority class and let the host queues use the regular medium/low priority class.
For drives that don't support WRRU, the IO throttling can be done in software on the host queues.
Host assigned partitions also don't need polling, thus allowing polling to be used only for guests that actually need low latency IO.
This reduces the number of cores that would be otherwise lost to polling, because the less work the polling core does, the less latency it contributes to overall latency, thus with less users, you could use less cores to achieve the same levels of latency.
For Stefan's argument, we can look at it in a slightly different way too:
While the nvme-mdev can be seen as a duplication of SPDK, the SPDK can also be seen as duplication of an existing kernel functionality which nvme-mdev can reuse for free.
Best regards,
Maxim Levitsky
Keith Busch
maintainerRe:
In reply to Felipe FranciosiQuoted 8 lines · Felipe Franciosi
Quoted 4 lines
Note though that SPDK doesn't support sharing the device between host and the guests, it takes over the nvme device, thus it makes the kernel nvme driver unbind from it.
That is absolutely true. However, I find it not to be a problem in practice.
Hypervisor products, specially those caring about performance, efficiency and fairness, will dedicate NVMe devices for a particular purpose (eg. vDisk storage, cache, metadata) and will not share these devices for other use cases. That's because these products want to deterministically control the performance aspects of the device, which you just cannot do if you are sharing the device with a subsystem you do not control.
I don't know, it sounds like you've traded kernel syscalls for IPC, and I don't think one performs better than the other.
Quoted 1 line
For scenarios where the device must be shared and such fine grained control is not required, it looks like using the kernel driver with
io_uringoffers very good performance with flexibility.
NVMe's IO Determinism features provide fine grained control for shared devices. It's still uncommon to find hardware supporting that, though.
Re:
Hi Keith,
Quoted 8 lines · Keith Busch
Quoted 5 lines · Felipe Franciosi
Quoted 4 lines
Note though that SPDK doesn't support sharing the device between host and the guests, it takes over the nvme device, thus it makes the kernel nvme driver unbind from it.
Quoted 6 lines
Quoted 3 lines
That is absolutely true. However, I find it not to be a problem in practice.
Hypervisor products, specially those caring about performance, efficiency and fairness, will dedicate NVMe devices for a particular purpose (eg. vDisk storage, cache, metadata) and will not share these devices for other use cases. That's because these products want to deterministically control the performance aspects of the device, which you just cannot do if you are sharing the device with a subsystem you do not control.
I don't know, it sounds like you've traded kernel syscalls for IPC, and I don't think one performs better than the other.
Sorry, I'm not sure I understand. My point is that if you are packaging a distro to be a hypervisor and you want to use a storage device for VM data, you _most likely_ won't be using that device for anything else. To that end, driving the device directly from your application definitely gives you more deterministic control.
Quoted 5 lines
Quoted 2 lines
For scenarios where the device must be shared and such fine grained control is not required, it looks like using the kernel driver with
io_uringoffers very good performance with flexibility.NVMe's IO Determinism features provide fine grained control for shared devices. It's still uncommon to find hardware supporting that, though.
Sure, but then your hypervisor needs to certify devices that support that. This will limit your HCL. Moreover, unless the feature is solid, well-established and works reliably on all devices you support, it's arguably preferable to have an architecture which gives you that control in software.
Cheers,
Felipe
Re: [PATCH 0/9] RFC: NVME VFIO mediated device
In reply to Maxim LevitskyQuoted 3 lines · Maxim Levitsky
*Polling kernel thread is used. The polling is stopped after a predefined timeout (1/2 sec by default).
Support for all interrupt driven mode is planned, and it shows promising results.
Which cgroup will the CPU cycles used for polling be attributed to? Can the polling code be moved into user space such that it becomes easy to identify which process needs most CPU cycles for polling and such that the polling CPU cycles are attributed to the proper cgroup?
Thanks,
Bart.
Re: [PATCH 0/9] RFC: NVME VFIO mediated device
Quoted 10 lines · Bart Van Assche
Quoted 4 lines · Maxim Levitsky
* Polling kernel thread is used. The polling is stopped after a predefined timeout (1/2 sec by default).
Support for all interrupt driven mode is planned, and it shows promising results.Which cgroup will the CPU cycles used for polling be attributed to? Can the polling code be moved into user space such that it becomes easy to identify which process needs most CPU cycles for polling and such that the polling CPU cycles are attributed to the proper cgroup?
Currently there is a single IO thread per each virtual controller instance. I would prefer to keep all the driver in the kernel, but I think I can make it cgroup aware, in a simiar way this is done in vhost-net, and vhost-scsi.
Best regards,
Maxim Levitsky
Quoted 3 lines
Thanks,
Bart.
Re: [PATCH 0/9] RFC: NVME VFIO mediated device
In reply to Maxim LevitskyQuoted 8 lines · Maxim Levitsky
* All guest memory is mapped into the physical nvme device but not 1:1 as vfio-pci would do this.
This allows very efficient DMA.
To support this, patch 2 adds ability for a mdev device to listen on guest's memory map events.
Any such memory is immediately pinned and then DMA mapped. (Support for fabric drivers where this is not possible exits too,
in which case the fabric driver will do its own DMA mapping)
Does this mean that all guest memory is pinned all the time? If so, are you sure that's acceptable?
Additionally, what is the performance overhead of the IOMMU notifier added by patch 8/9? How often was that notifier called per second in your tests and how much time was spent per call in the notifier callbacks?
Thanks,
Bart.
Re: [PATCH 0/9] RFC: NVME VFIO mediated device
Quoted 10 lines · Bart Van Assche
Quoted 8 lines · Maxim Levitsky
* All guest memory is mapped into the physical nvme device but not 1:1 as vfio-pci would do this.
This allows very efficient DMA.
To support this, patch 2 adds ability for a mdev device to listen on guest's memory map events.
Any such memory is immediately pinned and then DMA mapped. (Support for fabric drivers where this is not possible exits too,
in which case the fabric driver will do its own DMA mapping)
Quoted 2 lines
Does this mean that all guest memory is pinned all the time? If so, are you sure that's acceptable?
I think so. The VFIO pci passthrough also pins all the guest memory. SPDK also does this (pins and dma maps) all the guest memory.
I agree that this is not an ideal solution but this is a fastest and simplest solution possible.
Quoted 4 lines
Additionally, what is the performance overhead of the IOMMU notifier added by patch 8/9? How often was that notifier called per second in your tests and how much time was spent per call in the notifier callbacks?
To be honest I haven't optimized my IOMMU notifier at all, so when it is called, it stops the IO thread, does its work and then restarts it which is very slow.
Fortunelly it is not called at all during normal operation as VFIO dma map/unmap events are really rare and happen only on guest boot.
The same is even true for nested guests, as nested guest startup causes a wave of map unmap events while shadow IOMMU updates, but then it just uses these mapping without changing them.
The only case when performance is really bad is when you boot a guest with iommu=on intel_iommu=on and then use the nvme driver there. In this case, the driver in the guest does itself IOMMU maps/unmaps (on the virtual IOMMU) and for each such event my VFIO map/unmap callback is called.
This can be optimized though to be much better using also some kind of queued invalidation in my driver. iommu=pt meanwhile in the guest solves that issue.
Best regards,
Maxim Levitsky
Quoted 9 lines
Thanks,
Bart.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 0/9] RFC: NVME VFIO mediated device
Quoted 11 lines · On Wed, 20 Mar 2019 18:42:02 +0200 Maxim Levitsky <[email protected]> wrote:
Quoted 10 lines · Bart Van Assche
Quoted 8 lines · Maxim Levitsky
* All guest memory is mapped into the physical nvme device but not 1:1 as vfio-pci would do this.
This allows very efficient DMA.
To support this, patch 2 adds ability for a mdev device to listen on guest's memory map events.
Any such memory is immediately pinned and then DMA mapped. (Support for fabric drivers where this is not possible exits too,
in which case the fabric driver will do its own DMA mapping)
Quoted 7 lines
Quoted 2 lines
Does this mean that all guest memory is pinned all the time? If so, are you sure that's acceptable?
I think so. The VFIO pci passthrough also pins all the guest memory. SPDK also does this (pins and dma maps) all the guest memory.
I agree that this is not an ideal solution but this is a fastest and simplest solution possible.
FWIW, the pinned memory request up through the vfio iommu driver count against the user's locked memory limits, if that's the concern. Thanks,
Alex
Re: your mail
In reply to Maxim LevitskyQuoted 40 lines · Maxim Levitsky
Date: Tue, 19 Mar 2019 14:45:45 +0200
Subject: [PATCH 0/9] RFC: NVME VFIO mediated deviceHi everyone!
In this patch series, I would like to introduce my take on the problem of doing as fast as possible virtualization of storage with emphasis on low latency.
In this patch series I implemented a kernel vfio based, mediated device that allows the user to pass through a partition and/or whole namespace to a guest.
The idea behind this driver is based on paper you can find at
https://www.usenix.org/conference/atc18/presentation/peng,Although note that I stared the development prior to reading this paper, independently.
In addition to that implementation is not based on code used in the paper as I wasn't being able at that time to make the source available to me.
***Key points about the implementation:***
* Polling kernel thread is used. The polling is stopped after a predefined timeout (1/2 sec by default).
Support for all interrupt driven mode is planned, and it shows promising results.* Guest sees a standard NVME device - this allows to run guest with unmodified drivers, for example windows guests.
* The NVMe device is shared between host and guest. That means that even a single namespace can be split between host and guest based on different partitions.
* Simple configuration
*** Performance ***
Performance was tested on Intel DC P3700, With Xeon E5-2620 v2 and both latency and throughput is very similar to SPDK.
Quoted 7 lines
Soon I will test this on a better server and nvme device and provide more formal performance numbers.
Latency numbers:
~80ms - spdk with fio plugin on the host.
~84ms - nvme driver on the host
~87ms - mdev-nvme + nvme driver in the guest
You mentioned the spdk numbers are with vhost-user-nvme. Have you measured SPDK's vhost-user-blk?
Stefan
Re: your mail
Quoted 49 lines · Stefan Hajnoczi
Quoted 48 lines · Maxim Levitsky
Date: Tue, 19 Mar 2019 14:45:45 +0200
Subject: [PATCH 0/9] RFC: NVME VFIO mediated deviceHi everyone!
In this patch series, I would like to introduce my take on the problem of doing
as fast as possible virtualization of storage with emphasis on low latency.In this patch series I implemented a kernel vfio based, mediated device that
allows the user to pass through a partition and/or whole namespace to a guest.The idea behind this driver is based on paper you can find at
https://www.usenix.org/conference/atc18/presentation/peng,Although note that I stared the development prior to reading this paper, independently.
In addition to that implementation is not based on code used in the paper as
I wasn't being able at that time to make the source available to me.***Key points about the implementation:***
* Polling kernel thread is used. The polling is stopped after a predefined timeout (1/2 sec by default).
Support for all interrupt driven mode is planned, and it shows promising results.* Guest sees a standard NVME device - this allows to run guest with unmodified drivers, for example windows guests.
* The NVMe device is shared between host and guest. That means that even a single namespace can be split between host and guest based on different partitions.
* Simple configuration
*** Performance ***
Performance was tested on Intel DC P3700, With Xeon E5-2620 v2 and both latency and throughput is very similar to SPDK.
Soon I will test this on a better server and nvme device and provide more formal performance numbers.
Quoted 7 lines
Quoted 4 lines
Latency numbers:
~80ms - spdk with fio plugin on the host.
~84ms - nvme driver on the host
~87ms - mdev-nvme + nvme driver in the guestYou mentioned the spdk numbers are with vhost-user-nvme. Have you measured SPDK's vhost-user-blk?
I had lot of measuments of vhost-user-blk vs vhost-user-nvme. vhost-user-nvme was always a bit faster but only a bit. Thus I don't think it makes sense to benchamrk against vhost-user-blk.
Best regards,
Maxim Levitsky
Re: your mail
Quoted 55 lines · Maxim Levitsky
Quoted 54 lines · Stefan Hajnoczi
Quoted 52 lines · Maxim Levitsky
Date: Tue, 19 Mar 2019 14:45:45 +0200
Subject: [PATCH 0/9] RFC: NVME VFIO mediated deviceHi everyone!
In this patch series, I would like to introduce my take on the problem of doing
as fast as possible virtualization of storage with emphasis on low latency.In this patch series I implemented a kernel vfio based, mediated device that
allows the user to pass through a partition and/or whole namespace to a guest.The idea behind this driver is based on paper you can find at
https://www.usenix.org/conference/atc18/presentation/peng,Although note that I stared the development prior to reading this paper, independently.
In addition to that implementation is not based on code used in the paper as
I wasn't being able at that time to make the source available to me.***Key points about the implementation:***
* Polling kernel thread is used. The polling is stopped after a predefined timeout (1/2 sec by default).
Support for all interrupt driven mode is planned, and it shows promising results.* Guest sees a standard NVME device - this allows to run guest with unmodified drivers, for example windows guests.
* The NVMe device is shared between host and guest. That means that even a single namespace can be split between host and guest based on different partitions.
* Simple configuration
*** Performance ***
Performance was tested on Intel DC P3700, With Xeon E5-2620 v2 and both latency and throughput is very similar to SPDK.
Soon I will test this on a better server and nvme device and provide more formal performance numbers.
Latency numbers:
~80ms - spdk with fio plugin on the host.
~84ms - nvme driver on the host
~87ms - mdev-nvme + nvme driver in the guest
Quoted 6 lines
Quoted 2 lines
You mentioned the spdk numbers are with vhost-user-nvme. Have you measured SPDK's vhost-user-blk?
I had lot of measuments of vhost-user-blk vs vhost-user-nvme. vhost-user-nvme was always a bit faster but only a bit. Thus I don't think it makes sense to benchamrk against vhost-user-blk.
It's interesting because mdev-nvme is closest to the hardware while vhost-user-blk is closest to software. Doing things at the NVMe level isn't buying much performance because it's still going through a software path comparable to vhost-user-blk.
From what you say it sounds like there isn't much to optimize away :(.
Stefan