Skip to contentSkip to conversation

virtio pmem: user document

PATCH In review Pankaj Gupta 2 participants 4 messages Jul 30, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 virtio pmem: user document

This patch documents the steps to use virtio pmem. It also documents other useful information about virtio pmem e.g use-case, comparison with Qemu NVDIMM backend and current limitations.

---
 docs/virtio-pmem.txt | 65 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)
 create mode 100644 docs/virtio-pmem.txt
docs/virtio-pmem.txt new file +65 -0
new file mode 100644
index 0000000000..fc61eebb20
@@ -0,0 +1,65 @@
+
+QEMU virtio pmem
+===================
+
+ This document explains the usage of virtio pmem device 
+ which is available since QEMU v4.1.0.
+
+ The virtio pmem is paravirtualized persistent memory device
+ on regular(non-NVDIMM) storage. 
+
+Usecase
+--------
+  Allows to bypass the guest page cache and directly use host page cache.
+  This reduces guest memory footprint as host can make efficient memory
+  reclaim decisions under memory pressure.
+
+o How does virtio-pmem compare to the nvdimm emulation supported by QEMU?
+
+  NVDIMM emulation on regular(non-NVDIMM) host storage does not persists
+  the guest writes as there are no defined semantecs in the device specification.
+  With virtio pmem device, guest write persistence on non-NVDIMM storage is
+  supported.
+
+virtio pmem usage
+-----------------
+  virtio pmem device is created with a memory-backend-file with the below
+  options:
+
+  -machine pc -m 8G,slots=$N,maxmem=$MAX_SIZE
+  -object memory-backend-file,id=mem1,share,mem-path=$PATH,size=$SIZE
+  -device virtio-pmem-pci,memdev=mem1,id=nv1
+
+   where:
+   - "object memory-backend-file,id=mem1,share,mem-path=$PATH,size=$VIRTIO_PMEM_SIZE"
+     creates a backend storage of size $SIZE on a file $PATH. All
+     accesses to the virtio pmem device go to the file $PATH.
+
+   - "device virtio-pmem-pci,id=nvdimm1,memdev=mem1" creates a virtio pmem
+     device whose storage is provided by above memory backend device.
+
+  Multiple virtio pmem devices can be created if multiple pairs of "-object"
+  and "-device" are provided.
+
+Hotplug
+-------
+Accomplished by two monitor commands "object_add" and "device_add".
+
+For example, the following commands add another 4GB virtio pmem device to
+the guest:
+
+ (qemu) object_add memory-backend-file,id=mem2,share=on,mem-path=virtio_pmem2.img,size=4G
+ (qemu) device_add virtio-pmem-pci,id=virtio_pmem2,memdev=mem2
+
+Guest Data Persistence
+----------------------
+Guest data persistence on non-NVDIMM requires guest userspace application to 
+perform fsync/msync. This is different than real nvdimm backend where no additional
+fsync/msync is required for data persistence.
+
+Limitations
+------------
+- Real nvdimm device backend is not supported.
+- virtio pmem hotunplug is not supported.
+- ACPI NVDIMM features like regions/namespaces are not supported.
+- ndctl command is not supported.
Signature · 1 line
2.21.0
#1 lore raw

PATCH 1/1 virtio pmem: user document

This patch documents the steps to use virtio pmem. It also documents other useful information about virtio pmem e.g use-case, comparison with Qemu NVDIMM backend and current limitations.

---
 docs/virtio-pmem.txt | 65 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)
 create mode 100644 docs/virtio-pmem.txt
docs/virtio-pmem.txt new file +65 -0
new file mode 100644
index 0000000000..fc61eebb20
@@ -0,0 +1,65 @@
+
+QEMU virtio pmem
+===================
+
+ This document explains the usage of virtio pmem device 
+ which is available since QEMU v4.1.0.
+
+ The virtio pmem is paravirtualized persistent memory device
+ on regular(non-NVDIMM) storage. 
+
+Usecase
+--------
+  Allows to bypass the guest page cache and directly use host page cache.
+  This reduces guest memory footprint as host can make efficient memory
+  reclaim decisions under memory pressure.
+
+o How does virtio-pmem compare to the nvdimm emulation supported by QEMU?
+
+  NVDIMM emulation on regular(non-NVDIMM) host storage does not persists
+  the guest writes as there are no defined semantecs in the device specification.
+  With virtio pmem device, guest write persistence on non-NVDIMM storage is
+  supported.
+
+virtio pmem usage
+-----------------
+  virtio pmem device is created with a memory-backend-file with the below
+  options:
+
+  -machine pc -m 8G,slots=$N,maxmem=$MAX_SIZE
+  -object memory-backend-file,id=mem1,share,mem-path=$PATH,size=$SIZE
+  -device virtio-pmem-pci,memdev=mem1,id=nv1
+
+   where:
+   - "object memory-backend-file,id=mem1,share,mem-path=$PATH,size=$VIRTIO_PMEM_SIZE"
+     creates a backend storage of size $SIZE on a file $PATH. All
+     accesses to the virtio pmem device go to the file $PATH.
+
+   - "device virtio-pmem-pci,id=nvdimm1,memdev=mem1" creates a virtio pmem
+     device whose storage is provided by above memory backend device.
+
+  Multiple virtio pmem devices can be created if multiple pairs of "-object"
+  and "-device" are provided.
+
+Hotplug
+-------
+Accomplished by two monitor commands "object_add" and "device_add".
+
+For example, the following commands add another 4GB virtio pmem device to
+the guest:
+
+ (qemu) object_add memory-backend-file,id=mem2,share=on,mem-path=virtio_pmem2.img,size=4G
+ (qemu) device_add virtio-pmem-pci,id=virtio_pmem2,memdev=mem2
+
+Guest Data Persistence
+----------------------
+Guest data persistence on non-NVDIMM requires guest userspace application to 
+perform fsync/msync. This is different than real nvdimm backend where no additional
+fsync/msync is required for data persistence.
+
+Limitations
+------------
+- Real nvdimm device backend is not supported.
+- virtio pmem hotunplug is not supported.
+- ACPI NVDIMM features like regions/namespaces are not supported.
+- ndctl command is not supported.
Signature · 1 line
2.21.0
#2 lore raw
In reply to Pankaj Gupta

docs/virtio-pmem.txt

Quoted 12 lines · On Tue, 30 Jul 2019 12:16:57 +0530 Pankaj Gupta <[email protected]> wrote:

This patch documents the steps to use virtio pmem. It also documents other useful information about virtio pmem e.g use-case, comparison with Qemu NVDIMM backend and current limitations.

---
 docs/virtio-pmem.txt | 65 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)
 create mode 100644 docs/virtio-pmem.txt
docs/virtio-pmem.txt +0 -0

Maybe make this ReST from the start? Should be trivial enough.

docs/virtio-pmem.txt line 5

Quoted 10 lines

new file mode 100644
index 0000000000..fc61eebb20

docs/virtio-pmem.txt new file +5 -0
@@ -0,0 +1,65 @@
+
+QEMU virtio pmem
+===================
+
+ This document explains the usage of virtio pmem device 

"setup and usage" ?

docs/virtio-pmem.txt line 8

Quoted 3 lines
docs/virtio-pmem.txt new file +3 -0
+ which is available since QEMU v4.1.0.
+
+ The virtio pmem is paravirtualized persistent memory device

"The virtio pmem device is a paravirtualized..."

docs/virtio-pmem.txt line 14

Quoted 6 lines
docs/virtio-pmem.txt new file +6 -0
+ on regular(non-NVDIMM) storage. 
+
+Usecase
+--------
+  Allows to bypass the guest page cache and directly use host page cache.
+  This reduces guest memory footprint as host can make efficient memory

s/as host/,as the host/

docs/virtio-pmem.txt line 19

Quoted 5 lines
docs/virtio-pmem.txt new file +5 -0
+  reclaim decisions under memory pressure.
+
+o How does virtio-pmem compare to the nvdimm emulation supported by QEMU?
+
+  NVDIMM emulation on regular(non-NVDIMM) host storage does not persists

s/regular(non-NVDIMM)/regular (i.e. non-NVDIMM)/ ? s/persists/persist/

docs/virtio-pmem.txt line 20

Quoted 1 line
docs/virtio-pmem.txt new file +1 -0
+  the guest writes as there are no defined semantecs in the device specification.

s/semantecs/semantics/

docs/virtio-pmem.txt line 22

Quoted 2 lines
docs/virtio-pmem.txt new file +2 -0
+  With virtio pmem device, guest write persistence on non-NVDIMM storage is
+  supported.

"The virtio pmem device provides a way to support guest write persistence on non-NVDIMM storage." ?

docs/virtio-pmem.txt line 27

Quoted 5 lines
docs/virtio-pmem.txt new file +5 -0
+
+virtio pmem usage
+-----------------
+  virtio pmem device is created with a memory-backend-file with the below
+  options:

"A virtio pmem device backed by a memory-backend-file can be created on the QEMU command line as in the following example:" ?

docs/virtio-pmem.txt line 29

Quoted 2 lines
docs/virtio-pmem.txt new file +2 -0
+
+  -machine pc -m 8G,slots=$N,maxmem=$MAX_SIZE

I'm not sure you should explicitly specify the machine type in this example. I think it is fine to say that something is only supported on a subset of machine types, but it should not make its way into an example on how to configure a device and its backing.

Also, maybe fill in more concrete values here? Or split it into a part specifying the syntax (where I'd use <max_size> instead of $MAX_SIZE etc.), and a more concrete example?

docs/virtio-pmem.txt line 39

Quoted 10 lines
docs/virtio-pmem.txt new file +10 -0
+  -object memory-backend-file,id=mem1,share,mem-path=$PATH,size=$SIZE
+  -device virtio-pmem-pci,memdev=mem1,id=nv1
+
+   where:
+   - "object memory-backend-file,id=mem1,share,mem-path=$PATH,size=$VIRTIO_PMEM_SIZE"
+     creates a backend storage of size $SIZE on a file $PATH. All
+     accesses to the virtio pmem device go to the file $PATH.
+
+   - "device virtio-pmem-pci,id=nvdimm1,memdev=mem1" creates a virtio pmem
+     device whose storage is provided by above memory backend device.

"a virtio pmem PCI device" ?

docs/virtio-pmem.txt line 46

Quoted 7 lines
docs/virtio-pmem.txt new file +7 -0
+
+  Multiple virtio pmem devices can be created if multiple pairs of "-object"
+  and "-device" are provided.
+
+Hotplug
+-------
+Accomplished by two monitor commands "object_add" and "device_add".

Hm... what about the following instead:

"Virtio pmem devices can be hotplugged via the QEMU monitor. First, the memory backing has to be added via 'object_add'; afterwards, the virtio pmem device can be added via 'device_add'."

docs/virtio-pmem.txt line 56

Quoted 10 lines
docs/virtio-pmem.txt new file +10 -0
+
+For example, the following commands add another 4GB virtio pmem device to
+the guest:
+
+ (qemu) object_add memory-backend-file,id=mem2,share=on,mem-path=virtio_pmem2.img,size=4G
+ (qemu) device_add virtio-pmem-pci,id=virtio_pmem2,memdev=mem2
+
+Guest Data Persistence
+----------------------
+Guest data persistence on non-NVDIMM requires guest userspace application to 

s/application/applications/ ?

docs/virtio-pmem.txt line 57

Quoted 1 line
docs/virtio-pmem.txt new file +1 -0
+perform fsync/msync. This is different than real nvdimm backend where no additional

s/than/from a/ ?

docs/virtio-pmem.txt line 58

Quoted 1 line
docs/virtio-pmem.txt new file +1 -0
+fsync/msync is required for data persistence.

Should we be a bit more verbose on what which guest applications are supposed to do? I.e., how do they know they need to do fsync/msync, when should they do it, and what are the consequences if they don't?

Quoted 7 lines
docs/virtio-pmem.txt new file +7 -0
+
+Limitations
+------------
+- Real nvdimm device backend is not supported.
+- virtio pmem hotunplug is not supported.
+- ACPI NVDIMM features like regions/namespaces are not supported.
+- ndctl command is not supported.
#3 lore raw
Quoted 16 lines
Quoted 11 lines · On Tue, 30 Jul 2019 12:16:57 +0530 Pankaj Gupta <[email protected]> wrote:

This patch documents the steps to use virtio pmem. It also documents other useful information about virtio pmem e.g use-case, comparison with Qemu NVDIMM backend and current limitations.

---
 docs/virtio-pmem.txt | 65 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)
 create mode 100644 docs/virtio-pmem.txt
Quoted 3 lines

docs/virtio-pmem.txt

Quoted 1 line
docs/virtio-pmem.txt +0 -0

Maybe make this ReST from the start? Should be trivial enough.

o.k

Quoted 13 lines

docs/virtio-pmem.txt line 5

Quoted 11 lines

new file mode 100644
index 0000000000..fc61eebb20

docs/virtio-pmem.txt new file +5 -0
@@ -0,0 +1,65 @@
+
+QEMU virtio pmem
+===================
+
+ This document explains the usage of virtio pmem device

"setup and usage" ?

o.k

Quoted 6 lines

docs/virtio-pmem.txt line 8

Quoted 4 lines
docs/virtio-pmem.txt new file +3 -0
+ which is available since QEMU v4.1.0.
+
+ The virtio pmem is paravirtualized persistent memory device

"The virtio pmem device is a paravirtualized..."

sure.

Quoted 9 lines

docs/virtio-pmem.txt line 14

Quoted 7 lines
docs/virtio-pmem.txt new file +6 -0
+ on regular(non-NVDIMM) storage.
+
+Usecase
+--------
+  Allows to bypass the guest page cache and directly use host page cache.
+  This reduces guest memory footprint as host can make efficient memory

s/as host/,as the host/

sure

Quoted 9 lines

docs/virtio-pmem.txt line 19

Quoted 6 lines
docs/virtio-pmem.txt new file +5 -0
+  reclaim decisions under memory pressure.
+
+o How does virtio-pmem compare to the nvdimm emulation supported by QEMU?
+
+  NVDIMM emulation on regular(non-NVDIMM) host storage does not persists

s/regular(non-NVDIMM)/regular (i.e. non-NVDIMM)/ ? s/persists/persist/

yes, to both.

Quoted 5 lines
Quoted 3 lines
docs/virtio-pmem.txt new file +1 -0
+  the guest writes as there are no defined semantecs in the device

specification.

s/semantecs/semantics/

ah...spell checker :(

Quoted 8 lines
Quoted 4 lines

+ With virtio pmem device, guest write persistence on non-NVDIMM storage is
+ supported.

"The virtio pmem device provides a way to support guest write persistence on non-NVDIMM storage." ?

Quoted 8 lines

docs/virtio-pmem.txt line 27

Quoted 5 lines
docs/virtio-pmem.txt new file +5 -0
+
+virtio pmem usage
+-----------------
+  virtio pmem device is created with a memory-backend-file with the below
+  options:

"A virtio pmem device backed by a memory-backend-file can be created on the QEMU command line as in the following example:" ?

o.k

Quoted 8 lines

docs/virtio-pmem.txt

Quoted 3 lines
docs/virtio-pmem.txt new file +2 -0
+
+  -machine pc -m 8G,slots=$N,maxmem=$MAX_SIZE

I'm not sure you should explicitly specify the machine type in this example. I think it is fine to say that something is only supported on a subset of machine types, but it should not make its way into an example on how to configure a device and its backing.

o.k

Quoted 4 lines

Also, maybe fill in more concrete values here? Or split it into a part specifying the syntax (where I'd use <max_size> instead of $MAX_SIZE etc.), and a more concrete example?

o.k

Quoted 14 lines

docs/virtio-pmem.txt line 39

Quoted 12 lines
docs/virtio-pmem.txt new file +5 -0
+  -object memory-backend-file,id=mem1,share,mem-path=$PATH,size=$SIZE
+  -device virtio-pmem-pci,memdev=mem1,id=nv1
+
+   where:
+   - "object

memory-backend-file,id=mem1,share,mem-path=$PATH,size=$VIRTIO_PMEM_SIZE"

docs/virtio-pmem.txt new file +5 -0
+     creates a backend storage of size $SIZE on a file $PATH. All
+     accesses to the virtio pmem device go to the file $PATH.
+
+   - "device virtio-pmem-pci,id=nvdimm1,memdev=mem1" creates a virtio pmem
+     device whose storage is provided by above memory backend device.

"a virtio pmem PCI device" ?

o.k

Quoted 12 lines

docs/virtio-pmem.txt line 46

Quoted 9 lines
docs/virtio-pmem.txt new file +2 -0
+
+  Multiple virtio pmem devices can be created if multiple pairs of

"-object"

docs/virtio-pmem.txt new file +5 -0
+  and "-device" are provided.
+
+Hotplug
+-------
+Accomplished by two monitor commands "object_add" and "device_add".

Hm... what about the following instead:

Quoted 3 lines

"Virtio pmem devices can be hotplugged via the QEMU monitor. First, the memory backing has to be added via 'object_add'; afterwards, the virtio pmem device can be added via 'device_add'."

o.k

Quoted 16 lines
Quoted 12 lines
docs/virtio-pmem.txt new file +5 -0
+
+For example, the following commands add another 4GB virtio pmem device to
+the guest:
+
+ (qemu) object_add

memory-backend-file,id=mem2,share=on,mem-path=virtio_pmem2.img,size=4G
+ (qemu) device_add virtio-pmem-pci,id=virtio_pmem2,memdev=mem2

docs/virtio-pmem.txt new file +4 -0
+
+Guest Data Persistence
+----------------------
+Guest data persistence on non-NVDIMM requires guest userspace application
Quoted 1 line

to

s/application/applications/ ?

Quoted 4 lines
Quoted 2 lines

+perform fsync/msync. This is different than real nvdimm backend where no additional

s/than/from a/ ?

yes.

Quoted 6 lines
Quoted 2 lines

+fsync/msync is required for data persistence.

Should we be a bit more verbose on what which guest applications are supposed to do? I.e., how do they know they need to do fsync/msync, when should they do it, and what are the consequences if they don't?

o.k.

Quoted 8 lines

docs/virtio-pmem.txt line 65

Quoted 8 lines
docs/virtio-pmem.txt new file +7 -0
+
+Limitations
+------------
+- Real nvdimm device backend is not supported.
+- virtio pmem hotunplug is not supported.
+- ACPI NVDIMM features like regions/namespaces are not supported.
+- ndctl command is not supported.

Thank you for the review.

Best regards,
Pankaj

Quoted 3 lines