Skip to contentSkip to conversation

Multi level NVMe stack

RFC Ming Lin 2 participants 4 messages Jul 29, 2015 – Aug 18, 2015 NVMe

lore
Share & tools
lore

Ming Lin

author
#0 lore raw

Hi,

I'm working on a virtio-nvme driver.
NVMe over Fabrics initiator will probably be available some time.

I wonder does it make sense to have a multi level NVMe stack(similar as SCSI). So we can re-use the NVMe core code.

.-------------------------.
| /dev/nvme0n1 register |

  Upper level           | NVMe protocol process   |
                        |                         |
                        '-------------------------'



              .-----------.    .-----------.    .------------------.
 Lower level  |   PCIe    |    | VIRTIO    |    |NVMe over Fabrics |
              |           |    |           |    |initiator         |
              '-----------'    '-----------'    '------------------'

Thanks,
Ming

#1 lore raw
Quoted 5 lines · Ming Lin

I'm working on a virtio-nvme driver.
NVMe over Fabrics initiator will probably be available some time.

I wonder does it make sense to have a multi level NVMe stack(similar as SCSI). So we can re-use the NVMe core code.

Hi Ming,

I've put some work in for abstraction, see the work to submit the internal NVMe command through struct request.

Once we actually get other transports we can split main data structures, and move the parts sitting above that abstraction out into a separate module.

Quoted 1 line

From both looking at the I/O code I'd prefer not to add any abstractions

in the I/O code for.

I'd suggest you prototype the code without big refactoring first, and then compare your implementation against the NVMe virtualization extensions from Google and check if it's worth it:

http://lists.infradead.org/pipermail/linux-nvme/2014-July/001076.html

author
#2 lore raw
Quoted 16 lines · Christoph Hellwig
Quoted 5 lines · Ming Lin

I'm working on a virtio-nvme driver.
NVMe over Fabrics initiator will probably be available some time.

I wonder does it make sense to have a multi level NVMe stack(similar as SCSI). So we can re-use the NVMe core code.

Hi Ming,

I've put some work in for abstraction, see the work to submit the internal NVMe command through struct request.

Once we actually get other transports we can split main data structures, and move the parts sitting above that abstraction out into a separate module.

Quoted 8 lines

From both looking at the I/O code I'd prefer not to add any abstractions in the I/O code for.

I'd suggest you prototype the code without big refactoring first, and then compare your implementation against the NVMe virtualization extensions from Google and check if it's worth it:

http://lists.infradead.org/pipermail/linux-nvme/2014-July/001076.html

I'll prototype the code.

Thanks.

author
#3 lore raw
Quoted 20 lines · Ming Lin
Quoted 19 lines · Christoph Hellwig
Quoted 5 lines · Ming Lin

I'm working on a virtio-nvme driver.
NVMe over Fabrics initiator will probably be available some time.

I wonder does it make sense to have a multi level NVMe stack(similar as SCSI). So we can re-use the NVMe core code.

Hi Ming,

I've put some work in for abstraction, see the work to submit the internal NVMe command through struct request.

Once we actually get other transports we can split main data structures, and move the parts sitting above that abstraction out into a separate module.

From both looking at the I/O code I'd prefer not to add any abstractions in the I/O code for.

Quoted 7 lines
Quoted 5 lines

I'd suggest you prototype the code without big refactoring first, and then compare your implementation against the NVMe virtualization extensions from Google and check if it's worth it:

http://lists.infradead.org/pipermail/linux-nvme/2014-July/001076.html

I'll prototype the code.

So here is the super draft prototype code.
It's basically modified from linux/nvme-core.c and qemu/nvme.c.

http://minggr.noip.me/cgit/cgit.cgi/linux/log/?h=virtio-nvme/prototype
http://minggr.noip.me/cgit/cgit.cgi/qemu/log/?h=virtio-nvme/prototype

I haven't tried Google extensions yet.
A quick comparison with qemu-nvme:

qemu-nvme qemu-virtio-nvme
4k read: 95.7MB/s 105.3MB/s +10%
4k write: 90.2MB/s 96.3MB/s +6.7%

I'll post the patches after I make it in good shape and do more optimizations.