Skip to contentSkip to conversation

NVMe devices with CMB

SPDK Kit Chow 3 participants 4 messages Nov 2, 2021 – Nov 8, 2021

lore
Share & tools
lore

Kit Chow

author
#0 lore raw

Hi,

Do you know of any NVMe SSDs that have CMBs? I've asked Samsung but the rep did not seem to know and asked me to send my question to the samsung support email. I have not heard back yet.

If NVMe SSDs w/ CMBs aren't currently/readily available, what NVMe devices are being used by SPDK to do p2p copies.

Thanks

#1 lore raw
Quoted 11 lines

From: Kit Chow <kchow(a)maxlinear.com>

Hi,

Do you know of any NVMe SSDs that have CMBs? I've asked Samsung but the rep did not seem to know and asked me to send my question to the samsung support email. I have not heard back yet.

If NVMe SSDs w/ CMBs aren't currently/readily available, what NVMe devices are being used by SPDK to do p2p copies.

I believe you can use the QEMU emulated NVMe SSD to test CMB. That's probably the easiest way.

author
#2 lore raw

Hi Benjamin,

I found the following qemu nvme blog - http://blog.frankenmichl.de/2018/02/13/add-nvme-device-to-vm/

Following the instructions, I get the NVMe device listed below with a bar4 size of 4K. Is this the CMB and is it large enough for small p2p copies?

Thanks
Kit

----

00:03.0 Non-Volatile memory controller: Intel Corporation QEMU NVM Express Controller (rev 02) (prog-if 02 [NVM Express])
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 23
NUMA node: 0
Region 0: Memory at fc070000 (64-bit, non-prefetchable) [size=8K]
Region 4: Memory at fc073000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00000800
Capabilities: [80] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Kernel driver in use: nvme
Kernel modules: nvme

#3 lore raw
Quoted 6 lines · Kit Chow

Hi Benjamin,

I found the following qemu nvme blog - http://blog.frankenmichl.de/2018/02/13/add-nvme-device-to-vm/

Following the instructions, I get the NVMe device listed below with a bar4 size of 4K. Is this the CMB and is it large enough for small p2p copies?

Hi Kit,

You need to specifically enable the CMB with the cmb_size_mb parameters (argument in MBs).

-device "nvme,id=nvme0,serial=deadbeef,cmb_size_mb=64"

Looks like you are using a pretty old version of QEMU. In current versions the CMB is exclusively using BAR 4 - I think what you are seeing is an older version where the MSI-X tables were in BAR4.