Questions about vNVDIMM on qemu/KVM
Hello,
I'm investigating status of vNVDIMM on qemu/KVM,
and I have some questions about it. I'm glad if anyone answer them.
In my understanding, qemu/KVM has a feature to show NFIT for guest, and it will be still updated about platform capability with this patch set.
https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg04756.html
And libvirt also supports this feature with <memory model='nvdimm'>
https://libvirt.org/formatdomain.html#elementsMemory
However, virtio-pmem is developing now, and it is better for archtectures to detect regions of NVDIMM without ACPI (like s390x) In addition, It is also necessary to flush guest contents on vNVDIMM who has a backend-file.
Q1) Does ACPI.NFIT bus of qemu/kvm remain with virtio-pmem?
How do each roles become it if both NFIT and virtio-pmem will be available? If my understanding is correct, both NFIT and virtio-pmem is used to detect vNVDIMM regions, but only one seems to be necessary....
Otherwize, is the NFIT bus just for keeping compatibility, and virtio-pmem is promising way?
Q2) What bus is(will be?) created for virtio-pmem?
I could confirm the bus of NFIT is created with <memory model='nvdimm'>, and I heard other bus will be created for virtio-pmem, but I could not find what bus is created concretely.
---
# ndctl list -B
{
"provider":"ACPI.NFIT",
"dev":"ndbus0"
}
---
I think it affects what operations user will be able to, and what notification is necessary for vNVDIMM.
ACPI defines some operations like namespace controll, and notification for NVDIMM health status or others.
(I suppose that other status notification might be necessary for vNVDIMM,
but I'm not sure yet...)
If my understanding is wrong, please correct me.
Thanks,
---
Yasunori Goto
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 9 lines · Yasunori Goto
Hello,
I'm investigating status of vNVDIMM on qemu/KVM,
and I have some questions about it. I'm glad if anyone answer them.In my understanding, qemu/KVM has a feature to show NFIT for guest, and it will be still updated about platform capability with this patch set.
https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg04756.html
Quoted 6 lines
And libvirt also supports this feature with <memory model='nvdimm'>
https://libvirt.org/formatdomain.html#elementsMemoryHowever, virtio-pmem is developing now, and it is better for archtectures to detect regions of NVDIMM without ACPI (like s390x)
I think you are confusing virtio-pmem (patches from Pankaj) and virtio-mem (patches from David)? ...or I'm confused.
Quoted 2 lines
In addition, It is also necessary to flush guest contents on vNVDIMM who has a backend-file.
virtio-pmem is a mechanism to use host page cache as pmem in a guest. It does not support high performance memory applications because it requires fsync/msync. I.e. it is not DAX it is the traditional mmap I/O model, but moving page cache management to the host rather than duplicating it in guests.
Quoted 4 lines
Q1) Does ACPI.NFIT bus of qemu/kvm remain with virtio-pmem?
How do each roles become it if both NFIT and virtio-pmem will be available? If my understanding is correct, both NFIT and virtio-pmem is used to detect vNVDIMM regions, but only one seems to be necessary....
We need both because they are different. Guest DAX should not be using virtio-pmem.
Quoted 16 lines
Otherwize, is the NFIT bus just for keeping compatibility, and virtio-pmem is promising way?
Q2) What bus is(will be?) created for virtio-pmem?
I could confirm the bus of NFIT is created with <memory model='nvdimm'>, and I heard other bus will be created for virtio-pmem, but I could not find what bus is created concretely.
---
# ndctl list -B
{
"provider":"ACPI.NFIT",
"dev":"ndbus0"
}
---
Quoted 8 lines
I think it affects what operations user will be able to, and what notification is necessary for vNVDIMM.
ACPI defines some operations like namespace controll, and notification for NVDIMM health status or others.
(I suppose that other status notification might be necessary for vNVDIMM,
but I'm not sure yet...)If my understanding is wrong, please correct me.
The current plan, per my understanding, is a virtio-pmem SPA UUID added to the virtual NFIT so that the guest driver can load the pmem driver but also hook up the virtio command ring for forwarding WRITE_{FUA,FLUSH} commands as host fsync operations.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 14 lines · Yasunori Goto
Quoted 13 lines
Hello,
I'm investigating status of vNVDIMM on qemu/KVM,
and I have some questions about it. I'm glad if anyone answer them.In my understanding, qemu/KVM has a feature to show NFIT for guest, and it will be still updated about platform capability with this patch set.
https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg04756.htmlAnd libvirt also supports this feature with <memory model='nvdimm'>
https://libvirt.org/formatdomain.html#elementsMemory
Quoted 5 lines
Quoted 2 lines
However, virtio-pmem is developing now, and it is better for archtectures to detect regions of NVDIMM without ACPI (like s390x)
I think you are confusing virtio-pmem (patches from Pankaj) and virtio-mem (patches from David)? ...or I'm confused.
Probably, "I" am confusing.
So, your clarification is very helpful for me.
Quoted 9 lines
Quoted 3 lines
In addition, It is also necessary to flush guest contents on vNVDIMM who has a backend-file.
virtio-pmem is a mechanism to use host page cache as pmem in a guest. It does not support high performance memory applications because it requires fsync/msync. I.e. it is not DAX it is the traditional mmap I/O model, but moving page cache management to the host rather than duplicating it in guests.
Ah, ok.
Quoted 8 lines
Quoted 5 lines
Q1) Does ACPI.NFIT bus of qemu/kvm remain with virtio-pmem?
How do each roles become it if both NFIT and virtio-pmem will be available? If my understanding is correct, both NFIT and virtio-pmem is used to detect vNVDIMM regions, but only one seems to be necessary....We need both because they are different. Guest DAX should not be using virtio-pmem.
Hmm. Ok.
But ,I would like understand one more thing.
In the following mail, it seems that e820 bus will be used for fake DAX.
https://lists.01.org/pipermail/linux-nvdimm/2018-January/013926.html
Could you tell me what is relationship between "fake DAX" in this mail and Guest DAX?
Why e820 is necessary for this case?
(Probably, it may be one of the reason why I'm confusing....)
Quoted 24 lines
Quoted 24 lines
Otherwize, is the NFIT bus just for keeping compatibility, and virtio-pmem is promising way?
Q2) What bus is(will be?) created for virtio-pmem?
I could confirm the bus of NFIT is created with <memory model='nvdimm'>, and I heard other bus will be created for virtio-pmem, but I could not find what bus is created concretely.
---
# ndctl list -B
{
"provider":"ACPI.NFIT",
"dev":"ndbus0"
}
---I think it affects what operations user will be able to, and what notification is necessary for vNVDIMM.
ACPI defines some operations like namespace controll, and notification for NVDIMM health status or others.
(I suppose that other status notification might be necessary for vNVDIMM,
but I'm not sure yet...)
Quoted 6 lines
Quoted 1 line
If my understanding is wrong, please correct me.
The current plan, per my understanding, is a virtio-pmem SPA UUID added to the virtual NFIT so that the guest driver can load the pmem driver but also hook up the virtio command ring for forwarding WRITE_{FUA,FLUSH} commands as host fsync operations.
Ok.
Thank you very much for your answer!
---
Yasunori Goto
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 48 lines · Yasunori Goto
Quoted 14 lines · Yasunori Goto
Quoted 13 lines
Hello,
I'm investigating status of vNVDIMM on qemu/KVM,
and I have some questions about it. I'm glad if anyone answer them.In my understanding, qemu/KVM has a feature to show NFIT for guest, and it will be still updated about platform capability with this patch set.
https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg04756.htmlAnd libvirt also supports this feature with <memory model='nvdimm'>
https://libvirt.org/formatdomain.html#elementsMemoryQuoted 5 lines
Quoted 2 lines
However, virtio-pmem is developing now, and it is better for archtectures to detect regions of NVDIMM without ACPI (like s390x)
I think you are confusing virtio-pmem (patches from Pankaj) and virtio-mem (patches from David)? ...or I'm confused.
Probably, "I" am confusing.
So, your clarification is very helpful for me.Quoted 9 lines
Quoted 3 lines
In addition, It is also necessary to flush guest contents on vNVDIMM who has a backend-file.
virtio-pmem is a mechanism to use host page cache as pmem in a guest. It does not support high performance memory applications because it requires fsync/msync. I.e. it is not DAX it is the traditional mmap I/O model, but moving page cache management to the host rather than duplicating it in guests.
Ah, ok.
Quoted 8 lines
Quoted 5 lines
Q1) Does ACPI.NFIT bus of qemu/kvm remain with virtio-pmem?
How do each roles become it if both NFIT and virtio-pmem will be available? If my understanding is correct, both NFIT and virtio-pmem is used to detect vNVDIMM regions, but only one seems to be necessary....We need both because they are different. Guest DAX should not be using virtio-pmem.
Hmm. Ok.
Quoted 9 lines
But ,I would like understand one more thing.
In the following mail, it seems that e820 bus will be used for fake DAX.https://lists.01.org/pipermail/linux-nvdimm/2018-January/013926.html
Could you tell me what is relationship between "fake DAX" in this mail and Guest DAX?
Why e820 is necessary for this case?
It was proposed as a starting template for writing a new nvdimm bus driver. All we need is a way to communicate both the address range and the flush interface. This could be done with a new SPA Range GUID with the NFIT, or a custom virtio-pci device that registers a special nvdimm region with this property. My preference is whichever approach minimizes the code duplication, because the pmem driver should be re-used as much as possible.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 12 lines
Quoted 11 lines
But ,I would like understand one more thing.
In the following mail, it seems that e820 bus will be used for fake DAX.https://lists.01.org/pipermail/linux-nvdimm/2018-January/013926.html
Could you tell me what is relationship between "fake DAX" in this mail and Guest DAX?
Why e820 is necessary for this case?
Quoted 7 lines
It was proposed as a starting template for writing a new nvdimm bus driver. All we need is a way to communicate both the address range and the flush interface. This could be done with a new SPA Range GUID with the NFIT, or a custom virtio-pci device that registers a special nvdimm region with this property. My preference is whichever approach minimizes the code duplication, because the pmem driver should be re-used as much as possible.
Ok, I see.
Thank you very much for your explanation.
Bye,
---
Yasunori Goto
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 1 line · Yasunori Goto
Hello,
CCing Pankaj, who is developing virtio-pmem and may have comments beyond what has already been discussed.
Quoted 53 lines
I'm investigating status of vNVDIMM on qemu/KVM,
and I have some questions about it. I'm glad if anyone answer them.In my understanding, qemu/KVM has a feature to show NFIT for guest, and it will be still updated about platform capability with this patch set.
https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg04756.htmlAnd libvirt also supports this feature with <memory model='nvdimm'>
https://libvirt.org/formatdomain.html#elementsMemoryHowever, virtio-pmem is developing now, and it is better for archtectures to detect regions of NVDIMM without ACPI (like s390x) In addition, It is also necessary to flush guest contents on vNVDIMM who has a backend-file.
Q1) Does ACPI.NFIT bus of qemu/kvm remain with virtio-pmem?
How do each roles become it if both NFIT and virtio-pmem will be available? If my understanding is correct, both NFIT and virtio-pmem is used to detect vNVDIMM regions, but only one seems to be necessary....Otherwize, is the NFIT bus just for keeping compatibility, and virtio-pmem is promising way?
Q2) What bus is(will be?) created for virtio-pmem?
I could confirm the bus of NFIT is created with <memory model='nvdimm'>, and I heard other bus will be created for virtio-pmem, but I could not find what bus is created concretely.
---
# ndctl list -B
{
"provider":"ACPI.NFIT",
"dev":"ndbus0"
}
---I think it affects what operations user will be able to, and what notification is necessary for vNVDIMM.
ACPI defines some operations like namespace controll, and notification for NVDIMM health status or others.
(I suppose that other status notification might be necessary for vNVDIMM,
but I'm not sure yet...)If my understanding is wrong, please correct me.
Thanks,
---
Yasunori Goto
Hi,
Quoted 12 lines
Quoted 12 lines
I'm investigating status of vNVDIMM on qemu/KVM,
and I have some questions about it. I'm glad if anyone answer them.In my understanding, qemu/KVM has a feature to show NFIT for guest, and it will be still updated about platform capability with this patch set.
https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg04756.htmlAnd libvirt also supports this feature with <memory model='nvdimm'>
https://libvirt.org/formatdomain.html#elementsMemory
Quoted 7 lines
Quoted 7 lines
However, virtio-pmem is developing now, and it is better for archtectures to detect regions of NVDIMM without ACPI (like s390x) In addition, It is also necessary to flush guest contents on vNVDIMM who has a backend-file.
Q1) Does ACPI.NFIT bus of qemu/kvm remain with virtio-pmem?
No.
Quoted 2 lines
Quoted 2 lines
How do each roles become it if both NFIT and virtio-pmem will be available?
There are two main use cases:
1] DAX memory region pass-through to guest:
-------------------------------------------
As this region is present in actual physical NVDIMM device and exposed to guest, ACPI/NFIT way is used. If all the persistent memory is used by only this way we don't need 'virtio-pmem'.
2] Emulated DAX memory region in host passed to guest:
--------------------------------------------------------
If this type of region is exposed to guest, it will be preferable to use 'virtio-pmem'.
This is regular host memory which is mmaped in guest address space for emulating persistent memory. Guest writes are present in host page cache and not assured to be written on backing disk without an explicit flush/sync call.
'virtio-pmem' will solve the problem of flushing guest writes present in host page cache. With filesystems at host which use journal-ling like (ext4, xfs), they automatically call 'fsync' at regular intervals. but still there is not 100% assurance of all write persistence until an explicit flush is done from guest. So, we need an additional fsync to flush guest writes to backing disk. We are using this approach to avoid using guest page cache and keep page cache management of all the guests at host side.
If both ACPI NFIT and virtio-pmem are present, both will have their corresponding memory regions and defined by memory type "Persistent shared Memory" in case of virtio-pmem and "Persistent Memory" for ACPI NVDIMM. This is to differentiate both the memory types.
Quoted 10 lines
Quoted 10 lines
If my understanding is correct, both NFIT and virtio-pmem is used to detect vNVDIMM regions, but only one seems to be necessary....
Otherwize, is the NFIT bus just for keeping compatibility, and virtio-pmem is promising way?
Q2) What bus is(will be?) created for virtio-pmem?
I could confirm the bus of NFIT is created with <memory model='nvdimm'>,
For virtio-pmem also its nvdimm bus.
Quoted 10 lines
Quoted 10 lines
and I heard other bus will be created for virtio-pmem, but I could not find what bus is created concretely. --- # ndctl list -B { "provider":"ACPI.NFIT", "dev":"ndbus0" } ---
Quoted 7 lines
Quoted 7 lines
I think it affects what operations user will be able to, and what notification is necessary for vNVDIMM.
ACPI defines some operations like namespace controll, and notification for NVDIMM health status or others.
(I suppose that other status notification might be necessary for vNVDIMM,
but I'm not sure yet...)
For virtio-pmem, we are not providing advance features like namespace and various other features which ACPI/NVDIMM hardware provides. This is just to keep paravirt device simple.
Moreover I have not yet looked at ndctl side of things. I am not 100% sure how ndctl will handle 'virtio-pmem'.
Adding 'Dan' in loop, he can add his thoughts.
Thanks,
Pankaj
Quoted 10 lines
Quoted 9 lines
If my understanding is wrong, please correct me.
Thanks,
---
Yasunori Goto
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 50 lines
Hi,
Quoted 12 lines
Quoted 12 lines
I'm investigating status of vNVDIMM on qemu/KVM,
and I have some questions about it. I'm glad if anyone answer them.In my understanding, qemu/KVM has a feature to show NFIT for guest, and it will be still updated about platform capability with this patch set.
https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg04756.htmlAnd libvirt also supports this feature with <memory model='nvdimm'>
https://libvirt.org/formatdomain.html#elementsMemoryQuoted 7 lines
Quoted 7 lines
However, virtio-pmem is developing now, and it is better for archtectures to detect regions of NVDIMM without ACPI (like s390x) In addition, It is also necessary to flush guest contents on vNVDIMM who has a backend-file.
Q1) Does ACPI.NFIT bus of qemu/kvm remain with virtio-pmem?
No.Quoted 2 lines
Quoted 2 lines
How do each roles become it if both NFIT and virtio-pmem will be available?
There are two main use cases:
1] DAX memory region pass-through to guest:
-------------------------------------------
As this region is present in actual physical NVDIMM device and exposed to guest, ACPI/NFIT way is used. If all the persistent memory is used by only this way we don't need 'virtio-pmem'.2] Emulated DAX memory region in host passed to guest:
--------------------------------------------------------
If this type of region is exposed to guest, it will be preferable to use 'virtio-pmem'.This is regular host memory which is mmaped in guest address space for emulating persistent memory. Guest writes are present in host page cache and not assured to be written on backing disk without an explicit flush/sync call.
'virtio-pmem' will solve the problem of flushing guest writes present in host page cache. With filesystems at host which use journal-ling like (ext4, xfs), they automatically call 'fsync' at regular intervals. but still there is not 100% assurance of all write persistence until an explicit flush is done from guest. So, we need an additional fsync to flush guest writes to backing disk. We are using this approach to avoid using guest page cache and keep page cache management of all the guests at host side.
Quoted 3 lines
If both ACPI NFIT and virtio-pmem are present, both will have their corresponding memory regions and defined by memory type "Persistent shared Memory" in case of virtio-pmem and "Persistent Memory" for ACPI NVDIMM. This is to differentiate both the memory types.
Ok.
Quoted 32 lines
Quoted 11 lines
Quoted 11 lines
If my understanding is correct, both NFIT and virtio-pmem is used to detect vNVDIMM regions, but only one seems to be necessary....
Otherwize, is the NFIT bus just for keeping compatibility, and virtio-pmem is promising way?
Q2) What bus is(will be?) created for virtio-pmem?
I could confirm the bus of NFIT is created with <memory model='nvdimm'>,For virtio-pmem also its nvdimm bus.
Quoted 10 lines
Quoted 10 lines
and I heard other bus will be created for virtio-pmem, but I could not find what bus is created concretely. --- # ndctl list -B { "provider":"ACPI.NFIT", "dev":"ndbus0" } ---Quoted 7 lines
Quoted 7 lines
I think it affects what operations user will be able to, and what notification is necessary for vNVDIMM.
ACPI defines some operations like namespace controll, and notification for NVDIMM health status or others.
(I suppose that other status notification might be necessary for vNVDIMM,
but I'm not sure yet...)
Quoted 3 lines
For virtio-pmem, we are not providing advance features like namespace and various other features which ACPI/NVDIMM hardware provides. This is just to keep paravirt device simple.
Hmm, I see.
Thank you for your explanation.
Bye.
---
Yasunori Goto
Quoted 9 lines
Moreover I have not yet looked at ndctl side of things. I am not 100% sure how ndctl will handle 'virtio-pmem'.
Adding 'Dan' in loop, he can add his thoughts.
Thanks,
Pankaj
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm