Skip to contentSkip to conversation

QEMU NVDIMM as type 7 in e820 table

DISCUSSION Ross Zwisler 5 participants 8 messages Jul 28, 2017 – Jul 31, 2017 libnvdimm: non-volatile memory device

lore
Share & tools
lore
#0 lore raw

I've been using the virtualized NVDIMM support in QEMU for testing, and I noticed that the physical addresses used by the virtual NVDIMMs aren't present in the guest's e820 table.

Here is the e820 table on my QEMU instance where I have one 32 GiB virtual NVDIMM:

[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable

The physical addresses used by the virtual NVDIMM are 0x240000000-0xA40000000. You can see this by looking at ndctl and the values we get from the NFIT:

# ndctl list -R 
{
  "dev":"region0",
  "size":34359738368,
  "available_size":0,
  "type":"pmem"
}
# grep . /sys/bus/nd/devices/region0/{resource,size}
region0/resource:0x240000000
region0/size:34359738368

Or you can see the same info by using iasl to dump /sys/firmware/acpi/tables/NFIT:

[028h 0040 2] Subtable Type : 0000 [System Physical Address Range]
[02Ah 0042 2] Length : 0038

[02Ch 0044 2] Range Index : 0002
[02Eh 0046 2] Flags (decoded below) : 0003
Add/Online Operation Only : 1
Proximity Domain Valid : 1
[030h 0048 4] Reserved : 00000000
[034h 0052 4] Proximity Domain : 00000000
[038h 0056 16] Address Range GUID : 66F0D379-B4F3-4074-AC43-0D3318B78CDB

[048h 0072   8]           Address Range Base : 0000000240000000
[050h 0080   8]         Address Range Length : 0000000800000000
[058h 0088   8]         Memory Map Attribute : 0000000000008008

I expected to see a type 7 region for the NVDIMM physical address range in the e820 table, so something like:

[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000240000000-0x0000000A40000000] persistent (type 7)

Thanks,
- Ross
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#1 lore raw
Quoted 52 lines · Ross Zwisler

I've been using the virtualized NVDIMM support in QEMU for testing, and I noticed that the physical addresses used by the virtual NVDIMMs aren't present in the guest's e820 table.

Here is the e820 table on my QEMU instance where I have one 32 GiB virtual NVDIMM:

[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable

The physical addresses used by the virtual NVDIMM are 0x240000000-0xA40000000. You can see this by looking at ndctl and the values we get from the NFIT:

# ndctl list -R
{
  "dev":"region0",
  "size":34359738368,
  "available_size":0,
  "type":"pmem"
}
# grep . /sys/bus/nd/devices/region0/{resource,size}
region0/resource:0x240000000
region0/size:34359738368

Or you can see the same info by using iasl to dump /sys/firmware/acpi/tables/NFIT:

[028h 0040 2] Subtable Type : 0000 [System Physical Address Range]
[02Ah 0042 2] Length : 0038

[02Ch 0044 2] Range Index : 0002
[02Eh 0046 2] Flags (decoded below) : 0003
Add/Online Operation Only : 1
Proximity Domain Valid : 1
[030h 0048 4] Reserved : 00000000
[034h 0052 4] Proximity Domain : 00000000
[038h 0056 16] Address Range GUID : 66F0D379-B4F3-4074-AC43-0D3318B78CDB

[048h 0072   8]           Address Range Base : 0000000240000000
[050h 0080   8]         Address Range Length : 0000000800000000
[058h 0088   8]         Memory Map Attribute : 0000000000008008

I expected to see a type 7 region for the NVDIMM physical address range in the e820 table, so something like:

Quoted 11 lines
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000240000000-0x0000000A40000000] persistent (type 7)

Do you need that informationin e820? Linux effectively ignores type-7. As long as the range is treated as reserved it's not clear that you need the e820 entry. We also infect the persistent type back into the memory map when the NFIT driver loads. /proc/iomem should show the right data.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#2 lore raw
Quoted 66 lines · Dan Williams
Quoted 52 lines · Ross Zwisler

I've been using the virtualized NVDIMM support in QEMU for testing, and I noticed that the physical addresses used by the virtual NVDIMMs aren't present in the guest's e820 table.

Here is the e820 table on my QEMU instance where I have one 32 GiB virtual NVDIMM:

[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable

The physical addresses used by the virtual NVDIMM are 0x240000000-0xA40000000. You can see this by looking at ndctl and the values we get from the NFIT:

# ndctl list -R
{
  "dev":"region0",
  "size":34359738368,
  "available_size":0,
  "type":"pmem"
}
# grep . /sys/bus/nd/devices/region0/{resource,size}
region0/resource:0x240000000
region0/size:34359738368

Or you can see the same info by using iasl to dump /sys/firmware/acpi/tables/NFIT:

[028h 0040 2] Subtable Type : 0000 [System Physical Address Range]
[02Ah 0042 2] Length : 0038

[02Ch 0044 2] Range Index : 0002
[02Eh 0046 2] Flags (decoded below) : 0003
Add/Online Operation Only : 1
Proximity Domain Valid : 1
[030h 0048 4] Reserved : 00000000
[034h 0052 4] Proximity Domain : 00000000
[038h 0056 16] Address Range GUID : 66F0D379-B4F3-4074-AC43-0D3318B78CDB

[048h 0072   8]           Address Range Base : 0000000240000000
[050h 0080   8]         Address Range Length : 0000000800000000
[058h 0088   8]         Memory Map Attribute : 0000000000008008

I expected to see a type 7 region for the NVDIMM physical address range in the e820 table, so something like:

Quoted 11 lines
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000240000000-0x0000000A40000000] persistent (type 7)
Quoted 5 lines

Do you need that informationin e820? Linux effectively ignores type-7. As long as the range is treated as reserved it's not clear that you need the e820 entry. We also infect the persistent type back into the memory map when the NFIT driver loads. /proc/iomem should show the right data.

[ Adding Linda & Toshi to see if they have an opinion. ]

I guess maybe we don't need it. Yep, /proc/iomem looks good:

  # cat /proc/iomem
  00000000-00000fff : Reserved
  00001000-0009fbff : System RAM
  ...
  100000000-23fffffff : System RAM
  240000000-a3fffffff : Persistent Memory
    240000000-a3fffffff : namespace0.0

I was just worried that this was an inconsistency between the way that virtual NVDIMMs are presented vs the way that they will be presented on bare metal. I at least look at the e820 table to get my bearings of how memory is laid out - maybe I just need to look at /proc/iomem instead?
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#3 lore raw
Quoted 2 lines · Ross Zwisler
Quoted 1 line · Dan Williams

On Fri, Jul 28, 2017 at 11:04 AM, Ross Zwisler

[..]

Quoted 12 lines

[ Adding Linda & Toshi to see if they have an opinion. ]

I guess maybe we don't need it. Yep, /proc/iomem looks good:

  # cat /proc/iomem
  00000000-00000fff : Reserved
  00001000-0009fbff : System RAM
  ...
  100000000-23fffffff : System RAM
  240000000-a3fffffff : Persistent Memory
    240000000-a3fffffff : namespace0.0
Quoted 4 lines

I was just worried that this was an inconsistency between the way that virtual NVDIMMs are presented vs the way that they will be presented on bare metal. I at least look at the e820 table to get my bearings of how memory is laid out - maybe I just need to look at /proc/iomem instead?

I don't think e820 matters as long as /proc/iomem ends up correct, and the effective type is "reserved" . There's nothing in the specification that requires the OS to validate e820 ranges vs NFIT.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#4 lore raw
In reply to Ross Zwisler
Quoted 3 lines · Ross Zwisler
Quoted 2 lines · Dan Williams

On Fri, Jul 28, 2017 at 11:04 AM, Ross Zwisler
<[email protected]> wrote:

:

Quoted 18 lines
Quoted 5 lines

Do you need that informationin e820? Linux effectively ignores type-7. As long as the range is treated as reserved it's not clear that you need the e820 entry. We also infect the persistent type back into the memory map when the NFIT driver loads. /proc/iomem should show the right data.

[ Adding Linda & Toshi to see if they have an opinion. ]

I guess maybe we don't need it.  Yep, /proc/iomem looks good:

  # cat /proc/iomem
  00000000-00000fff : Reserved
  00001000-0009fbff : System RAM
  ...
  100000000-23fffffff : System RAM
  240000000-a3fffffff : Persistent Memory
    240000000-a3fffffff : namespace0.0
Quoted 5 lines

I was just worried that this was an inconsistency between the way that virtual NVDIMMs are presented vs the way that they will be presented on bare metal.  I at least look at the e820 table to get my bearings of how memory is laid out - maybe I just need to look at /proc/iomem instead?

FW should present a persistent memory range in e820 or UEFI memory descriptor table. So, it's a good practice for QEMU to do it as well.

That said, the NFIT driver inserts a persistent memory range to the kernel IO resource table from NFIT, so we are OK without this info. Yes, /proc/iomem shows how the resources are managed by the kernel.

Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#5 lore raw
In reply to Ross Zwisler
Quoted 85 lines · Ross Zwisler
Quoted 66 lines · Dan Williams
Quoted 52 lines · Ross Zwisler

I've been using the virtualized NVDIMM support in QEMU for testing, and I noticed that the physical addresses used by the virtual NVDIMMs aren't present in the guest's e820 table.

Here is the e820 table on my QEMU instance where I have one 32 GiB virtual NVDIMM:

[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable

The physical addresses used by the virtual NVDIMM are 0x240000000-0xA40000000. You can see this by looking at ndctl and the values we get from the NFIT:

# ndctl list -R
{
  "dev":"region0",
  "size":34359738368,
  "available_size":0,
  "type":"pmem"
}
# grep . /sys/bus/nd/devices/region0/{resource,size}
region0/resource:0x240000000
region0/size:34359738368

Or you can see the same info by using iasl to dump /sys/firmware/acpi/tables/NFIT:

[028h 0040 2] Subtable Type : 0000 [System Physical Address Range]
[02Ah 0042 2] Length : 0038

[02Ch 0044 2] Range Index : 0002
[02Eh 0046 2] Flags (decoded below) : 0003
Add/Online Operation Only : 1
Proximity Domain Valid : 1
[030h 0048 4] Reserved : 00000000
[034h 0052 4] Proximity Domain : 00000000
[038h 0056 16] Address Range GUID : 66F0D379-B4F3-4074-AC43-0D3318B78CDB

[048h 0072   8]           Address Range Base : 0000000240000000
[050h 0080   8]         Address Range Length : 0000000800000000
[058h 0088   8]         Memory Map Attribute : 0000000000008008

I expected to see a type 7 region for the NVDIMM physical address range in the e820 table, so something like:

Quoted 11 lines
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000240000000-0x0000000A40000000] persistent (type 7)
Quoted 5 lines

Do you need that informationin e820? Linux effectively ignores type-7. As long as the range is treated as reserved it's not clear that you need the e820 entry. We also infect the persistent type back into the memory map when the NFIT driver loads. /proc/iomem should show the right data.

[ Adding Linda & Toshi to see if they have an opinion. ]

I guess maybe we don't need it. Yep, /proc/iomem looks good:

  # cat /proc/iomem
  00000000-00000fff : Reserved
  00001000-0009fbff : System RAM
  ...
  100000000-23fffffff : System RAM
  240000000-a3fffffff : Persistent Memory
    240000000-a3fffffff : namespace0.0
Quoted 4 lines

I was just worried that this was an inconsistency between the way that virtual NVDIMMs are presented vs the way that they will be presented on bare metal. I at least look at the e820 table to get my bearings of how memory is laid out - maybe I just need to look at /proc/iomem instead?

Do any OS or applications rely on the E820 information or the consistency between E820 and NFIT to properly work? If any, I can make a QEMU patch to build type-7 e820 entries.

Haozhong
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#6 lore raw
Quoted 86 lines · Haozhong Zhang
Quoted 85 lines · Ross Zwisler
Quoted 66 lines · Dan Williams
Quoted 52 lines · Ross Zwisler

I've been using the virtualized NVDIMM support in QEMU for testing, and I noticed that the physical addresses used by the virtual NVDIMMs aren't present in the guest's e820 table.

Here is the e820 table on my QEMU instance where I have one 32 GiB virtual NVDIMM:

[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable

The physical addresses used by the virtual NVDIMM are 0x240000000-0xA40000000. You can see this by looking at ndctl and the values we get from the NFIT:

# ndctl list -R
{
  "dev":"region0",
  "size":34359738368,
  "available_size":0,
  "type":"pmem"
}
# grep . /sys/bus/nd/devices/region0/{resource,size}
region0/resource:0x240000000
region0/size:34359738368

Or you can see the same info by using iasl to dump /sys/firmware/acpi/tables/NFIT:

[028h 0040 2] Subtable Type : 0000 [System Physical Address Range]
[02Ah 0042 2] Length : 0038

[02Ch 0044 2] Range Index : 0002
[02Eh 0046 2] Flags (decoded below) : 0003
Add/Online Operation Only : 1
Proximity Domain Valid : 1
[030h 0048 4] Reserved : 00000000
[034h 0052 4] Proximity Domain : 00000000
[038h 0056 16] Address Range GUID : 66F0D379-B4F3-4074-AC43-0D3318B78CDB

[048h 0072   8]           Address Range Base : 0000000240000000
[050h 0080   8]         Address Range Length : 0000000800000000
[058h 0088   8]         Memory Map Attribute : 0000000000008008

I expected to see a type 7 region for the NVDIMM physical address range in the e820 table, so something like:

Quoted 11 lines
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000240000000-0x0000000A40000000] persistent (type 7)
Quoted 5 lines

Do you need that informationin e820? Linux effectively ignores type-7. As long as the range is treated as reserved it's not clear that you need the e820 entry. We also infect the persistent type back into the memory map when the NFIT driver loads. /proc/iomem should show the right data.

[ Adding Linda & Toshi to see if they have an opinion. ]

I guess maybe we don't need it. Yep, /proc/iomem looks good:

  # cat /proc/iomem
  00000000-00000fff : Reserved
  00001000-0009fbff : System RAM
  ...
  100000000-23fffffff : System RAM
  240000000-a3fffffff : Persistent Memory
    240000000-a3fffffff : namespace0.0
Quoted 8 lines
Quoted 4 lines

I was just worried that this was an inconsistency between the way that virtual NVDIMMs are presented vs the way that they will be presented on bare metal. I at least look at the e820 table to get my bearings of how memory is laid out - maybe I just need to look at /proc/iomem instead?

Do any OS or applications rely on the E820 information or the consistency between E820 and NFIT to properly work? If any, I can make a QEMU patch to build type-7 e820 entries.

I don't know of any off hand, but IMO it would be good to have this consistency.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#7 lore raw
Quoted 92 lines · On Mon, 31 Jul 2017 09:48:08 -0600 Ross Zwisler <[email protected]>…
Quoted 91 lines · Haozhong Zhang
Quoted 89 lines · Ross Zwisler
Quoted 66 lines · Dan Williams
Quoted 52 lines · Ross Zwisler

I've been using the virtualized NVDIMM support in QEMU for testing, and I noticed that the physical addresses used by the virtual NVDIMMs aren't present in the guest's e820 table.

Here is the e820 table on my QEMU instance where I have one 32 GiB virtual NVDIMM:

[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable

The physical addresses used by the virtual NVDIMM are 0x240000000-0xA40000000. You can see this by looking at ndctl and the values we get from the NFIT:

# ndctl list -R
{
  "dev":"region0",
  "size":34359738368,
  "available_size":0,
  "type":"pmem"
}
# grep . /sys/bus/nd/devices/region0/{resource,size}
region0/resource:0x240000000
region0/size:34359738368

Or you can see the same info by using iasl to dump /sys/firmware/acpi/tables/NFIT:

[028h 0040 2] Subtable Type : 0000 [System Physical Address Range]
[02Ah 0042 2] Length : 0038

[02Ch 0044 2] Range Index : 0002
[02Eh 0046 2] Flags (decoded below) : 0003
Add/Online Operation Only : 1
Proximity Domain Valid : 1
[030h 0048 4] Reserved : 00000000
[034h 0052 4] Proximity Domain : 00000000
[038h 0056 16] Address Range GUID : 66F0D379-B4F3-4074-AC43-0D3318B78CDB

[048h 0072   8]           Address Range Base : 0000000240000000
[050h 0080   8]         Address Range Length : 0000000800000000
[058h 0088   8]         Memory Map Attribute : 0000000000008008

I expected to see a type 7 region for the NVDIMM physical address range in the e820 table, so something like:

Quoted 11 lines
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000240000000-0x0000000A40000000] persistent (type 7)
Quoted 5 lines

Do you need that informationin e820? Linux effectively ignores type-7. As long as the range is treated as reserved it's not clear that you need the e820 entry. We also infect the persistent type back into the memory map when the NFIT driver loads. /proc/iomem should show the right data.

[ Adding Linda & Toshi to see if they have an opinion. ]

I guess maybe we don't need it. Yep, /proc/iomem looks good:

  # cat /proc/iomem
  00000000-00000fff : Reserved
  00001000-0009fbff : System RAM
  ...
  100000000-23fffffff : System RAM
  240000000-a3fffffff : Persistent Memory
    240000000-a3fffffff : namespace0.0

I was just worried that this was an inconsistency between the way that virtual NVDIMMs are presented vs the way that they will be presented on bare metal. I at least look at the e820 table to get my bearings of how memory is laid out - maybe I just need to look at /proc/iomem instead?

Quoted 6 lines
Quoted 3 lines

Do any OS or applications rely on the E820 information or the consistency between E820 and NFIT to properly work? If any, I can make a QEMU patch to build type-7 e820 entries.

I don't know of any off hand, but IMO it would be good to have this consistency.

Maybe we should wait till there is an actual hardware that does it and OS which uses it.

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm