Skip to contentSkip to conversation

Questions about vNVDIMM on qemu/KVM

DISCUSSION Yasunori Goto 4 participants 8 messages May 23, 2018 – Jun 6, 2018 libnvdimm: non-volatile memory device

lore
Share & tools
lore
#0 lore raw

Hello,

I'm investigating status of vNVDIMM on qemu/KVM,
and I have some questions about it. I'm glad if anyone answer them.

In my understanding, qemu/KVM has a feature to show NFIT for guest, and it will be still updated about platform capability with this patch set.
https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg04756.html

And libvirt also supports this feature with <memory model='nvdimm'>
https://libvirt.org/formatdomain.html#elementsMemory

However, virtio-pmem is developing now, and it is better for archtectures to detect regions of NVDIMM without ACPI (like s390x) In addition, It is also necessary to flush guest contents on vNVDIMM who has a backend-file.

Q1) Does ACPI.NFIT bus of qemu/kvm remain with virtio-pmem?
How do each roles become it if both NFIT and virtio-pmem will be available? If my understanding is correct, both NFIT and virtio-pmem is used to detect vNVDIMM regions, but only one seems to be necessary....

Otherwize, is the NFIT bus just for keeping compatibility, and virtio-pmem is promising way?

Q2) What bus is(will be?) created for virtio-pmem?
I could confirm the bus of NFIT is created with <memory model='nvdimm'>, and I heard other bus will be created for virtio-pmem, but I could not find what bus is created concretely.
---
# ndctl list -B
{
"provider":"ACPI.NFIT",
"dev":"ndbus0"
}
---

I think it affects what operations user will be able to, and what notification is necessary for vNVDIMM.
ACPI defines some operations like namespace controll, and notification for NVDIMM health status or others.
(I suppose that other status notification might be necessary for vNVDIMM,
but I'm not sure yet...)

If my understanding is wrong, please correct me.

Thanks,
---
Yasunori Goto

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#1 lore raw
Quoted 9 lines · Yasunori Goto

Hello,

I'm investigating status of vNVDIMM on qemu/KVM,
and I have some questions about it. I'm glad if anyone answer them.

In my understanding, qemu/KVM has a feature to show NFIT for guest, and it will be still updated about platform capability with this patch set.
https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg04756.html

Quoted 6 lines

And libvirt also supports this feature with <memory model='nvdimm'>
https://libvirt.org/formatdomain.html#elementsMemory

However, virtio-pmem is developing now, and it is better for archtectures to detect regions of NVDIMM without ACPI (like s390x)

I think you are confusing virtio-pmem (patches from Pankaj) and virtio-mem (patches from David)? ...or I'm confused.

Quoted 2 lines

In addition, It is also necessary to flush guest contents on vNVDIMM who has a backend-file.

virtio-pmem is a mechanism to use host page cache as pmem in a guest. It does not support high performance memory applications because it requires fsync/msync. I.e. it is not DAX it is the traditional mmap I/O model, but moving page cache management to the host rather than duplicating it in guests.

Quoted 4 lines

Q1) Does ACPI.NFIT bus of qemu/kvm remain with virtio-pmem?
How do each roles become it if both NFIT and virtio-pmem will be available? If my understanding is correct, both NFIT and virtio-pmem is used to detect vNVDIMM regions, but only one seems to be necessary....

We need both because they are different. Guest DAX should not be using virtio-pmem.

Quoted 16 lines

Otherwize, is the NFIT bus just for keeping compatibility, and virtio-pmem is promising way?

Q2) What bus is(will be?) created for virtio-pmem?
I could confirm the bus of NFIT is created with <memory model='nvdimm'>, and I heard other bus will be created for virtio-pmem, but I could not find what bus is created concretely.
---
# ndctl list -B
{
"provider":"ACPI.NFIT",
"dev":"ndbus0"
}
---

Quoted 8 lines

I think it affects what operations user will be able to, and what notification is necessary for vNVDIMM.
ACPI defines some operations like namespace controll, and notification for NVDIMM health status or others.
(I suppose that other status notification might be necessary for vNVDIMM,
but I'm not sure yet...)

If my understanding is wrong, please correct me.

The current plan, per my understanding, is a virtio-pmem SPA UUID added to the virtual NFIT so that the guest driver can load the pmem driver but also hook up the virtio command ring for forwarding WRITE_{FUA,FLUSH} commands as host fsync operations.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#2 lore raw
Quoted 14 lines · Yasunori Goto
Quoted 13 lines

Hello,

I'm investigating status of vNVDIMM on qemu/KVM,
and I have some questions about it. I'm glad if anyone answer them.

In my understanding, qemu/KVM has a feature to show NFIT for guest, and it will be still updated about platform capability with this patch set.
https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg04756.html

And libvirt also supports this feature with <memory model='nvdimm'>
https://libvirt.org/formatdomain.html#elementsMemory

Quoted 5 lines
Quoted 2 lines

However, virtio-pmem is developing now, and it is better for archtectures to detect regions of NVDIMM without ACPI (like s390x)

I think you are confusing virtio-pmem (patches from Pankaj) and virtio-mem (patches from David)? ...or I'm confused.

Probably, "I" am confusing.
So, your clarification is very helpful for me.

Quoted 9 lines
Quoted 3 lines

In addition, It is also necessary to flush guest contents on vNVDIMM who has a backend-file.

virtio-pmem is a mechanism to use host page cache as pmem in a guest. It does not support high performance memory applications because it requires fsync/msync. I.e. it is not DAX it is the traditional mmap I/O model, but moving page cache management to the host rather than duplicating it in guests.

Ah, ok.

Quoted 8 lines
Quoted 5 lines

Q1) Does ACPI.NFIT bus of qemu/kvm remain with virtio-pmem?
How do each roles become it if both NFIT and virtio-pmem will be available? If my understanding is correct, both NFIT and virtio-pmem is used to detect vNVDIMM regions, but only one seems to be necessary....

We need both because they are different. Guest DAX should not be using virtio-pmem.

Hmm. Ok.

But ,I would like understand one more thing.
In the following mail, it seems that e820 bus will be used for fake DAX.

https://lists.01.org/pipermail/linux-nvdimm/2018-January/013926.html

Could you tell me what is relationship between "fake DAX" in this mail and Guest DAX?
Why e820 is necessary for this case?

(Probably, it may be one of the reason why I'm confusing....)

Quoted 24 lines
Quoted 24 lines

Otherwize, is the NFIT bus just for keeping compatibility, and virtio-pmem is promising way?

Q2) What bus is(will be?) created for virtio-pmem?
I could confirm the bus of NFIT is created with <memory model='nvdimm'>, and I heard other bus will be created for virtio-pmem, but I could not find what bus is created concretely.
---
# ndctl list -B
{
"provider":"ACPI.NFIT",
"dev":"ndbus0"
}
---

I think it affects what operations user will be able to, and what notification is necessary for vNVDIMM.
ACPI defines some operations like namespace controll, and notification for NVDIMM health status or others.
(I suppose that other status notification might be necessary for vNVDIMM,
but I'm not sure yet...)

Quoted 6 lines
Quoted 1 line

If my understanding is wrong, please correct me.

The current plan, per my understanding, is a virtio-pmem SPA UUID added to the virtual NFIT so that the guest driver can load the pmem driver but also hook up the virtio command ring for forwarding WRITE_{FUA,FLUSH} commands as host fsync operations.

Ok.

Thank you very much for your answer!

---
Yasunori Goto

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#3 lore raw
Quoted 48 lines · Yasunori Goto
Quoted 14 lines · Yasunori Goto
Quoted 13 lines

Hello,

I'm investigating status of vNVDIMM on qemu/KVM,
and I have some questions about it. I'm glad if anyone answer them.

In my understanding, qemu/KVM has a feature to show NFIT for guest, and it will be still updated about platform capability with this patch set.
https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg04756.html

And libvirt also supports this feature with <memory model='nvdimm'>
https://libvirt.org/formatdomain.html#elementsMemory

Quoted 5 lines
Quoted 2 lines

However, virtio-pmem is developing now, and it is better for archtectures to detect regions of NVDIMM without ACPI (like s390x)

I think you are confusing virtio-pmem (patches from Pankaj) and virtio-mem (patches from David)? ...or I'm confused.

Probably, "I" am confusing.
So, your clarification is very helpful for me.

Quoted 9 lines
Quoted 3 lines

In addition, It is also necessary to flush guest contents on vNVDIMM who has a backend-file.

virtio-pmem is a mechanism to use host page cache as pmem in a guest. It does not support high performance memory applications because it requires fsync/msync. I.e. it is not DAX it is the traditional mmap I/O model, but moving page cache management to the host rather than duplicating it in guests.

Ah, ok.

Quoted 8 lines
Quoted 5 lines

Q1) Does ACPI.NFIT bus of qemu/kvm remain with virtio-pmem?
How do each roles become it if both NFIT and virtio-pmem will be available? If my understanding is correct, both NFIT and virtio-pmem is used to detect vNVDIMM regions, but only one seems to be necessary....

We need both because they are different. Guest DAX should not be using virtio-pmem.

Hmm. Ok.

Quoted 9 lines

But ,I would like understand one more thing.
In the following mail, it seems that e820 bus will be used for fake DAX.

https://lists.01.org/pipermail/linux-nvdimm/2018-January/013926.html

Could you tell me what is relationship between "fake DAX" in this mail and Guest DAX?
Why e820 is necessary for this case?

It was proposed as a starting template for writing a new nvdimm bus driver. All we need is a way to communicate both the address range and the flush interface. This could be done with a new SPA Range GUID with the NFIT, or a custom virtio-pci device that registers a special nvdimm region with this property. My preference is whichever approach minimizes the code duplication, because the pmem driver should be re-used as much as possible.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#4 lore raw
Quoted 12 lines
Quoted 11 lines

But ,I would like understand one more thing.
In the following mail, it seems that e820 bus will be used for fake DAX.

https://lists.01.org/pipermail/linux-nvdimm/2018-January/013926.html

Could you tell me what is relationship between "fake DAX" in this mail and Guest DAX?
Why e820 is necessary for this case?

Quoted 7 lines

It was proposed as a starting template for writing a new nvdimm bus driver. All we need is a way to communicate both the address range and the flush interface. This could be done with a new SPA Range GUID with the NFIT, or a custom virtio-pci device that registers a special nvdimm region with this property. My preference is whichever approach minimizes the code duplication, because the pmem driver should be re-used as much as possible.

Ok, I see.
Thank you very much for your explanation.

Bye,
---
Yasunori Goto

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#5 lore raw
In reply to Yasunori Goto
Quoted 1 line · Yasunori Goto

Hello,

CCing Pankaj, who is developing virtio-pmem and may have comments beyond what has already been discussed.

Quoted 53 lines

I'm investigating status of vNVDIMM on qemu/KVM,
and I have some questions about it. I'm glad if anyone answer them.

In my understanding, qemu/KVM has a feature to show NFIT for guest, and it will be still updated about platform capability with this patch set.
https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg04756.html

And libvirt also supports this feature with <memory model='nvdimm'>
https://libvirt.org/formatdomain.html#elementsMemory

However, virtio-pmem is developing now, and it is better for archtectures to detect regions of NVDIMM without ACPI (like s390x) In addition, It is also necessary to flush guest contents on vNVDIMM who has a backend-file.

Q1) Does ACPI.NFIT bus of qemu/kvm remain with virtio-pmem?
How do each roles become it if both NFIT and virtio-pmem will be available? If my understanding is correct, both NFIT and virtio-pmem is used to detect vNVDIMM regions, but only one seems to be necessary....

Otherwize, is the NFIT bus just for keeping compatibility, and virtio-pmem is promising way?

Q2) What bus is(will be?) created for virtio-pmem?
I could confirm the bus of NFIT is created with <memory model='nvdimm'>, and I heard other bus will be created for virtio-pmem, but I could not find what bus is created concretely.
---
# ndctl list -B
{
"provider":"ACPI.NFIT",
"dev":"ndbus0"
}
---

I think it affects what operations user will be able to, and what notification is necessary for vNVDIMM.
ACPI defines some operations like namespace controll, and notification for NVDIMM health status or others.
(I suppose that other status notification might be necessary for vNVDIMM,
but I'm not sure yet...)

If my understanding is wrong, please correct me.

Thanks,
---
Yasunori Goto

#6 lore raw

Hi,

Quoted 12 lines
Quoted 12 lines

I'm investigating status of vNVDIMM on qemu/KVM,
and I have some questions about it. I'm glad if anyone answer them.

In my understanding, qemu/KVM has a feature to show NFIT for guest, and it will be still updated about platform capability with this patch set.
https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg04756.html

And libvirt also supports this feature with <memory model='nvdimm'>
https://libvirt.org/formatdomain.html#elementsMemory

Quoted 7 lines
Quoted 7 lines

However, virtio-pmem is developing now, and it is better for archtectures to detect regions of NVDIMM without ACPI (like s390x) In addition, It is also necessary to flush guest contents on vNVDIMM who has a backend-file.

Q1) Does ACPI.NFIT bus of qemu/kvm remain with virtio-pmem?

        No.
Quoted 2 lines
Quoted 2 lines

How do each roles become it if both NFIT and virtio-pmem will be available?

There are two main use cases:

1] DAX memory region pass-through to guest:
-------------------------------------------
As this region is present in actual physical NVDIMM device and exposed to guest, ACPI/NFIT way is used. If all the persistent memory is used by only this way we don't need 'virtio-pmem'.

2] Emulated DAX memory region in host passed to guest:
--------------------------------------------------------
If this type of region is exposed to guest, it will be preferable to use 'virtio-pmem'.

This is regular host memory which is mmaped in guest address space for emulating persistent memory. Guest writes are present in host page cache and not assured to be written on backing disk without an explicit flush/sync call.

'virtio-pmem' will solve the problem of flushing guest writes present in host page cache. With filesystems at host which use journal-ling like (ext4, xfs), they automatically call 'fsync' at regular intervals. but still there is not 100% assurance of all write persistence until an explicit flush is done from guest. So, we need an additional fsync to flush guest writes to backing disk. We are using this approach to avoid using guest page cache and keep page cache management of all the guests at host side.

If both ACPI NFIT and virtio-pmem are present, both will have their corresponding memory regions and defined by memory type "Persistent shared Memory" in case of virtio-pmem and "Persistent Memory" for ACPI NVDIMM. This is to differentiate both the memory types.

Quoted 10 lines
Quoted 10 lines

If my understanding is correct, both NFIT and virtio-pmem is used to detect vNVDIMM regions, but only one seems to be necessary....

Otherwize, is the NFIT bus just for keeping compatibility, and virtio-pmem is promising way?

Q2) What bus is(will be?) created for virtio-pmem?
I could confirm the bus of NFIT is created with <memory model='nvdimm'>,

For virtio-pmem also its nvdimm bus.

Quoted 10 lines
Quoted 10 lines
    and I heard other bus will be created for virtio-pmem, but I could not
    find what bus is created concretely.
    ---
      # ndctl list -B
      {
         "provider":"ACPI.NFIT",
         "dev":"ndbus0"
      }
    ---
Quoted 7 lines
Quoted 7 lines

I think it affects what operations user will be able to, and what notification is necessary for vNVDIMM.
ACPI defines some operations like namespace controll, and notification for NVDIMM health status or others.
(I suppose that other status notification might be necessary for vNVDIMM,
but I'm not sure yet...)

For virtio-pmem, we are not providing advance features like namespace and various other features which ACPI/NVDIMM hardware provides. This is just to keep paravirt device simple.

Moreover I have not yet looked at ndctl side of things. I am not 100% sure how ndctl will handle 'virtio-pmem'.

Adding 'Dan' in loop, he can add his thoughts.

Thanks,
Pankaj

Quoted 10 lines
Quoted 9 lines

If my understanding is wrong, please correct me.

Thanks,
---
Yasunori Goto

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#7 lore raw
Quoted 50 lines

Hi,

Quoted 12 lines
Quoted 12 lines

I'm investigating status of vNVDIMM on qemu/KVM,
and I have some questions about it. I'm glad if anyone answer them.

In my understanding, qemu/KVM has a feature to show NFIT for guest, and it will be still updated about platform capability with this patch set.
https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg04756.html

And libvirt also supports this feature with <memory model='nvdimm'>
https://libvirt.org/formatdomain.html#elementsMemory

Quoted 7 lines
Quoted 7 lines

However, virtio-pmem is developing now, and it is better for archtectures to detect regions of NVDIMM without ACPI (like s390x) In addition, It is also necessary to flush guest contents on vNVDIMM who has a backend-file.

Q1) Does ACPI.NFIT bus of qemu/kvm remain with virtio-pmem?

        No.
Quoted 2 lines
Quoted 2 lines

How do each roles become it if both NFIT and virtio-pmem will be available?

There are two main use cases:

1] DAX memory region pass-through to guest:
-------------------------------------------
As this region is present in actual physical NVDIMM device and exposed to guest, ACPI/NFIT way is used. If all the persistent memory is used by only this way we don't need 'virtio-pmem'.

2] Emulated DAX memory region in host passed to guest:
--------------------------------------------------------
If this type of region is exposed to guest, it will be preferable to use 'virtio-pmem'.

This is regular host memory which is mmaped in guest address space for emulating persistent memory. Guest writes are present in host page cache and not assured to be written on backing disk without an explicit flush/sync call.

'virtio-pmem' will solve the problem of flushing guest writes present in host page cache. With filesystems at host which use journal-ling like (ext4, xfs), they automatically call 'fsync' at regular intervals. but still there is not 100% assurance of all write persistence until an explicit flush is done from guest. So, we need an additional fsync to flush guest writes to backing disk. We are using this approach to avoid using guest page cache and keep page cache management of all the guests at host side.

Quoted 3 lines

If both ACPI NFIT and virtio-pmem are present, both will have their corresponding memory regions and defined by memory type "Persistent shared Memory" in case of virtio-pmem and "Persistent Memory" for ACPI NVDIMM. This is to differentiate both the memory types.

Ok.

Quoted 32 lines
Quoted 11 lines
Quoted 11 lines

If my understanding is correct, both NFIT and virtio-pmem is used to detect vNVDIMM regions, but only one seems to be necessary....

Otherwize, is the NFIT bus just for keeping compatibility, and virtio-pmem is promising way?

Q2) What bus is(will be?) created for virtio-pmem?
I could confirm the bus of NFIT is created with <memory model='nvdimm'>,

For virtio-pmem also its nvdimm bus.

Quoted 10 lines
Quoted 10 lines
    and I heard other bus will be created for virtio-pmem, but I could not
    find what bus is created concretely.
    ---
      # ndctl list -B
      {
         "provider":"ACPI.NFIT",
         "dev":"ndbus0"
      }
    ---
Quoted 7 lines
Quoted 7 lines

I think it affects what operations user will be able to, and what notification is necessary for vNVDIMM.
ACPI defines some operations like namespace controll, and notification for NVDIMM health status or others.
(I suppose that other status notification might be necessary for vNVDIMM,
but I'm not sure yet...)

Quoted 3 lines

For virtio-pmem, we are not providing advance features like namespace and various other features which ACPI/NVDIMM hardware provides. This is just to keep paravirt device simple.

Hmm, I see.
Thank you for your explanation.

Bye.
---
Yasunori Goto

Quoted 9 lines

Moreover I have not yet looked at ndctl side of things. I am not 100% sure how ndctl will handle 'virtio-pmem'.

Adding 'Dan' in loop, he can add his thoughts.

Thanks,
Pankaj

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm