RFC virtio nvme
Get series with b4
b4 am 1443049097.28503.13.camel@ssiVersions newest first
- v1 RFC Latest0 of 2 patches 1 replyIn review
Patches in v1 No applies recorded yet
None of the 2 patches the cover letter announces are in this archive.
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.
Cover letter [RFC PATCH 0/2] virtio nvme
Quoted 80 lines · Nicholas A. Bellinger
Quoted 4 lines · Ming Lin
Quoted 3 lines · Nicholas A. Bellinger
Quoted 2 lines · Ming Lin
Quoted 1 line · Nicholas A. Bellinger
Hi Ming & Co,
<SNIP>
Quoted 62 lines
Quoted 57 lines
Quoted 47 lines
Quoted 18 lines
Quoted 9 lines
I think the future "LIO NVMe target" only speaks NVMe protocol.
Nick(CCed), could you correct me if I'm wrong?
For SCSI stack, we have:
virtio-scsi(guest)tcm_vhost(or vhost_scsi, host)
LIO-scsi-targetQuoted 5 lines
For NVMe stack, we'll have similar components:
virtio-nvme(guest)vhost_nvme(host)
LIO-NVMe-targetI think it's more interesting to consider a 'vhost style' driver that can be used with unmodified nvme host OS drivers.
Quoted 8 lines
Dr. Hannes (CC'ed) had done something like this for megasas a few years back using specialized QEMU emulation + eventfd based LIO fabric driver, and got it working with Linux + MSFT guests.
Doing something similar for nvme would (potentially) be on par with current virtio-scsi+vhost-scsi small-block performance for scsi-mq guests, without the extra burden of a new command set specific virtio driver.
Trying to understand it.
Is it like below?.------------------------. MMIO .---------------------------------------. | Guest |--------> | Qemu | | Unmodified NVMe driver |<-------- | NVMe device simulation(eventfd based) | '------------------------' '---------------------------------------' | ^ write NVMe | | notify command command | | completion to eventfd | | to eventfd v | .--------------------------------------. | Host: | | eventfd based LIO NVMe fabric driver | '--------------------------------------' | | nvme_queue_rq() v .--------------------------------------.Quoted 9 lines
| NVMe driver | '--------------------------------------' | | v .-------------------------------------. | NVMe device | '-------------------------------------'Quoted 4 lines
Correct. The LIO driver on KVM host would be handling some amount of NVMe host interface emulation in kernel code, and would be able to decode nvme Read/Write/Flush operations and translate -> submit to existing backend drivers.
Quoted 9 lines
Let me call the "eventfd based LIO NVMe fabric driver" as "
tcm_eventfd_nvme"Currently, LIO frontend
driver(iscsi, fc, vhost-scsi etc)talk to LIO backenddriver(fileio, iblock etc)with SCSI commands.Did you mean the "
tcm_eventfd_nvme" driver need to translate NVMe commands to SCSI commands and then submit to backend driver?
Quoted 8 lines
IBLOCK + FILEIO +
RD_MCPdon't speak SCSI, they simply process I/Os with LBA + length based on SGL memory or pass along a FLUSH with LBA + length.So once the '
tcm_eventfd_nvme' driver on KVM host receives a nvme host hardware frame via eventfd, it would decode the frame and send along the Read/Write/Flush when exposing existing (non nvme native) backend drivers.
Learned vhost architecture:
http://blog.vmsplice.net/2011/09/qemu-internals-vhost-architecture.html
The nice thing is it is not tied to KVM in any way.
For SCSI, there are "virtio-scsi" in guest kernel and "vhost-scsi" in host kernel.
For NVMe, there is no "virtio-nvme" in guest kernel(just unmodified NVMe driver), but I'll do similar thing in Qemu with vhost infrastructure. And there is "vhost_nvme" in host kernel.
For the "virtqueue" implementation in qemu-nvme, I'll possibly just use/copy drivers/virtio/virtio_ring.c, same as what linux/tools/virtio/virtio_test.c does.
A bit more detail graph as below. What do you think?
.-----------------------------------------. .------------------------.
| Guest(Linux, Windows, FreeBSD, Solaris) | NVMe | qemu |
| unmodified NVMe driver | command | NVMe device emulation |
| | -------> | vhost + virtqueue |
'-----------------------------------------' '------------------------'
| | ^
passthrough | kick/notify
NVMe command | via eventfd
userspace via virtqueue | | |
v v |
----------------------------------------------------------------------------------
.-----------------------------------------------------------------------.
kernel | LIO frontend driver |
| - vhost_nvme | '-----------------------------------------------------------------------'
| translate ^
| (NVMe command) |
| to |
v (LBA, length) |
.----------------------------------------------------------------------.
| LIO backend driver |
| - fileio (/mnt/xxx.file) |
| - iblock (/dev/sda1, /dev/nvme0n1, ...) |
'----------------------------------------------------------------------'
| ^
| submit_bio() |
v |
.----------------------------------------------------------------------.
| block layer |
| |
'----------------------------------------------------------------------'
| ^
| |
v |
.----------------------------------------------------------------------.
| block device driver |
| |
'----------------------------------------------------------------------'
| | | |
| | | |
v v v v
.------------. .-----------. .------------. .---------------.
| SATA | | SCSI | | NVMe | | .... |
'------------' '-----------' '------------' '---------------'