Patches for PCI passthrough with modified E820 (v3) - resent.

# HG changeset patch
# User Konrad Rzeszutek Wilk <[email protected]>
# Date 1304518568 14400
# Node ID ba218fa1a48ed682651fd90dd8eeb68eeab68e7a
# Parent b6af9b428bb16c4c5364ace0617923ffa44ad887
libxl: Add support for passing in the machine's E820 for PCI passthrough

The code that populates E820 is unconditionally triggered by the guest configuration having "pci=['<BDF>,..']", being a PV guest, and if b_info->u.pv.machine_e820 is set.

The code do_domain_create calls the libxl__e820_alloc when it notices that the guest is PV, has at least one PCI devices, and has the machine_e820 flag set.

libxl__e820_alloc calls the xc_get_machine_memory_map to retrieve the systems E820. Then the E820 is sanitized to weed out E820 entries below 16MB, and as well remove any E820_RAM or E820_UNUSED regions as the guest does not need to know about them. The guest only needs the E820_ACPI, E820_NVS, E820_RESERVED to get an idea of where the PCI I/O space is. Mostly.. The Linux kernel assumes that any gap in the E820 is considered PCI I/O space which means that if we pass in the guest 2GB, and the E820_ACPI, and its friend start at 3GB, the gap between 2GB and 3GB will be considered as PCI I/O space. To guard against that we also create an E820_UNUSABLE between the region of 'target_kb' (called ram_end in the code) up to the first E820_[ACPI,NVS,RESERVED] region. Lastly, the xc_domain_set_memory_map is called to install the new E820.

When tested with another PV guest (NetBSD 5.1) the modified E820 gave it no trouble. The code has also been tested with older "classic" Xen Linux and with the newer "pvops" with success (SLES11, RHEL5, Ubuntu Lucid, Debian Squeeze, 2.6.37, 2.6.38, 2.6.39).

Memory that is slack or for balloon (so 'maxmem' in guest configuration) is put behind the machine E820. Which in most cases is after the 4GB.

The reason for doing the fetching of the E820 using the hypercall in the toolstack (instead of the guest doing it) is that when a guest would do a hypercall to 'XENMEM_machine_memory_map' it would retrieve an E820 with I/O range caps added in. Meaning that the region after 4GB up to end of possible memory would be marked as unusable and the kernel would not have any space to allocate a balloon region.

Signed-off-by Konrad Rzeszutek Wilk <[email protected]>

diff -r b6af9b428bb1 -r ba218fa1a48e tools/libxl/libxl.idl

tools/libxl/libxl.idl +1 -0

@@ -180,6 +180,7 @@ libxl_domain_build_info = Struct("domain
                                         ("cmdline", string),
                                         ("ramdisk", libxl_file_reference),
                                         ("features", string, True),
+                                        ("machine_e820", bool, False, "Use machine's E820 for PCI passthrough."),
                                         ])),
                  ])),
     ],

diff -r b6af9b428bb1 -r ba218fa1a48e tools/libxl/libxl_create.c

tools/libxl/libxl_create.c +1 -0

+    }
     if ( cb && (d_config->c_info.hvm || d_config->b_info.u.pv.bootloader )) {
         if ( (*cb)(ctx, domid, priv) )
             goto error_out;

diff -r b6af9b428bb1 -r ba218fa1a48e tools/libxl/libxl_pci.c

tools/libxl/libxl_pci.c +13 -0

@@ -1047,3 +1047,157 @@ int libxl_device_pci_shutdown(libxl_ctx 
     free(pcidevs);
     return 0;
 }
+
+static int e820_sanitize(libxl_ctx *ctx, struct e820entry src[],
+                         uint32_t *nr_entries,
+                         unsigned long map_limitkb,
+                         unsigned long balloon_kb)
+{
+    uint64_t delta_kb = 0, start = 0, start_kb = 0, last = 0, ram_end;
+    uint32_t i, idx = 0, nr;
+    struct e820entry e820[E820MAX];
+
+    if (!src || !map_limitkb || !balloon_kb || !nr_entries)
+        return ERROR_INVAL;
+

tools/libxl/libxl_pci.c +6 -0

+    start = -1ULL;
+    last = 0;
+    for (i = 0; i < nr; i++) {
+        if ((src[i].type == E820_RAM) ||
+            (src[i].type == E820_UNUSABLE) ||
+            (src[i].type == 0)) 

Quoted 15 lines · Konrad Rzeszutek Wilk

# HG changeset patch
# User Konrad Rzeszutek Wilk <[email protected]>
# Date 1304518568 14400
# Node ID ba218fa1a48ed682651fd90dd8eeb68eeab68e7a
# Parent b6af9b428bb16c4c5364ace0617923ffa44ad887
libxl: Add support for passing in the machine's E820 for PCI passthrough

The code that populates E820 is unconditionally triggered by the guest configuration having "pci=['<BDF>,..']", being a PV guest, and if b_info->u.pv.machine_e820 is set.

The code do_domain_create calls the libxl__e820_alloc when it notices that the guest is PV, has at least one PCI devices, and has the machine_e820 flag set.

Quoted 11 lines

libxl__e820_alloc calls the xc_get_machine_memory_map to retrieve the systems E820. Then the E820 is sanitized to weed out E820 entries below 16MB, and as well remove any E820_RAM or E820_UNUSED regions as the guest does not need to know about them. The guest only needs the E820_ACPI, E820_NVS, E820_RESERVED to get an idea of where the PCI I/O space is. Mostly.. The Linux kernel assumes that any gap in the E820 is considered PCI I/O space which means that if we pass in the guest 2GB, and the E820_ACPI, and its friend start at 3GB, the gap between 2GB and 3GB will be considered as PCI I/O space. To guard against that we also create an E820_UNUSABLE between the region of 'target_kb' (called ram_end in the code) up to the first E820_[ACPI,NVS,RESERVED] region. Lastly, the xc_domain_set_memory_map is called to install the new E820.

Do we need to document the requirements this places on a PV guest somewhere more prominent?

Phrases like "up to the first E820...." make me worry that this will only work on "sensible" machines, but I suppose we can cross those bridges when we come to them...

Quoted 37 lines

When tested with another PV guest (NetBSD 5.1) the modified E820 gave it no trouble. The code has also been tested with older "classic" Xen Linux and with the newer "pvops" with success (SLES11, RHEL5, Ubuntu Lucid, Debian Squeeze, 2.6.37, 2.6.38, 2.6.39).

Memory that is slack or for balloon (so 'maxmem' in guest configuration) is put behind the machine E820. Which in most cases is after the 4GB.

The reason for doing the fetching of the E820 using the hypercall in the toolstack (instead of the guest doing it) is that when a guest would do a hypercall to 'XENMEM_machine_memory_map' it would retrieve an E820 with I/O range caps added in. Meaning that the region after 4GB up to end of possible memory would be marked as unusable and the kernel would not have any space to allocate a balloon region.

Signed-off-by Konrad Rzeszutek Wilk <[email protected]>

diff -r b6af9b428bb1 -r ba218fa1a48e tools/libxl/libxl.idl

tools/libxl/libxl.idl +1 -0

@@ -180,6 +180,7 @@ libxl_domain_build_info = Struct("domain
                                         ("cmdline", string),
                                         ("ramdisk", libxl_file_reference),
                                         ("features", string, True),
+                                        ("machine_e820", bool, False, "Use machine's E820 for PCI passthrough."),
                                         ])),
                  ])),
     ],

diff -r b6af9b428bb1 -r ba218fa1a48e tools/libxl/libxl_create.c

tools/libxl/libxl_create.c +0 -0

@@ -519,6 +519,14 @@ static int do_domain_create(libxl__gc *g
     for (i = 0; i < d_config->num_pcidevs; i++)
         libxl__device_pci_add(gc, domid, &d_config->pcidevs[i], 1);
 

tools/libxl/libxl_create.c static int do_domain_create(libxl__gc *g · line 524

Quoted 3 lines

tools/libxl/libxl_create.c +3 -0

+    if (!d_config->c_info.hvm && d_config->b_info.u.pv.machine_e820) {
+        int rc = 0;
+        rc = libxl__e820_alloc(ctx, domid, d_config);

rc is pretty comprehensively initialised ;-0

tools/libxl/libxl_create.c static int do_domain_create(libxl__gc *g · line 528

Quoted 5 lines

tools/libxl/libxl_create.c +4 -0

+        if (rc)
+            LIBXL__LOG(ctx, LIBXL__LOG_WARNING,
+                      "Failed while collecting E820 with: %d (errno:%d)\n",
+                      rc, errno);

LIBXL__LOG_ERRNO takes care of logging errno for you.

Quoted 31 lines

tools/libxl/libxl_create.c +1 -0

+    }
     if ( cb && (d_config->c_info.hvm || d_config->b_info.u.pv.bootloader )) {
         if ( (*cb)(ctx, domid, priv) )
             goto error_out;

diff -r b6af9b428bb1 -r ba218fa1a48e tools/libxl/libxl_pci.c

tools/libxl/libxl_pci.c +18 -0

@@ -1047,3 +1047,157 @@ int libxl_device_pci_shutdown(libxl_ctx 
     free(pcidevs);
     return 0;
 }
+
+static int e820_sanitize(libxl_ctx *ctx, struct e820entry src[],
+                         uint32_t *nr_entries,
+                         unsigned long map_limitkb,
+                         unsigned long balloon_kb)
+{
+    uint64_t delta_kb = 0, start = 0, start_kb = 0, last = 0, ram_end;
+    uint32_t i, idx = 0, nr;
+    struct e820entry e820[E820MAX];
+
+    if (!src || !map_limitkb || !balloon_kb || !nr_entries)
+        return ERROR_INVAL;
+
+    nr = *nr_entries;
+    if (!nr)
+        return ERROR_INVAL;
+
+    if (nr > E820MAX)

tools/libxl/libxl_pci.c int libxl_device_pci_shutdown(libxl_ctx · line 1072

Quoted 5 lines

tools/libxl/libxl_pci.c +5 -0

+        return ERROR_NOMEM;
+
+    /* Weed out anything under 16MB */
+    for (i = 0; i < nr; i++) {
+      if (src[i].addr > 0x100000)

0x100000 is 1MB not 16MB. I'm not sure if the code or the comment is correct.

tools/libxl/libxl_pci.c int libxl_device_pci_shutdown(libxl_ctx · line 1081

Quoted 10 lines

tools/libxl/libxl_pci.c +9 -0

+        continue;
+
+      src[i].type = 0;
+      src[i].size = 0;
+      src[i].addr = -1ULL;
+    }
+
+    /* Find the lowest and highest entry in E820, skipping over
+     * undersired entries. */

undesired ?

Quoted 6 lines

tools/libxl/libxl_pci.c +6 -0

+    start = -1ULL;
+    last = 0;
+    for (i = 0; i < nr; i++) {
+        if ((src[i].type == E820_RAM) ||
+            (src[i].type == E820_UNUSABLE) ||
+            (src[i].type == 0)) 

[...]

tools/libxl/libxl_pci.c int libxl_device_pci_shutdown(libxl_ctx · line 1160

Quoted 11 lines

    nr = idx;

tools/libxl/libxl_pci.c +10 -0

+
+    for (i = 0; i < nr; i++) {
+      LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, ":%s\t[%lx -> %lx]",
+	e820[i].type == E820_RAM ? "RAM " :
+	(e820[i].type == E820_RESERVED ? "RSV " :
+	 e820[i].type == E820_ACPI ? "ACPI" :
+         (e820[i].type == E820_NVS ? "NVS " :
+          (e820[i].type == E820_UNUSABLE ? "UNU " : "----"))),
+          e820[i].addr >> 12,
+         (e820[i].addr + e820[i].size) >> 12);

This screams out for a e820_type_as_string style function or a lookup table, or just about anything else really ;-).

Also you can use "%-4s" instead of manually aligning all the strings.

tools/libxl/libxl_pci.c int libxl_device_pci_shutdown(libxl_ctx · line 1168

Quoted 9 lines

tools/libxl/libxl_pci.c +8 -0

+    }
+
+    /* Done: copy the sanitized version. */
+    *nr_entries = nr;
+    memcpy(src, e820, nr * sizeof(struct e820entry));
+    return 0;
+}
+

Ian.

diff -r b6af9b428bb1 -r ba218fa1a48e tools/libxl/xl_cmdimpl.c

Sorry for the delay replying. But:

Konrad Rzeszutek Wilk writes ("[PATCH 2 of 3] libxl: Add support for passing in the machine's E820 for PCI passthrough"):

Quoted 1 line

diff -r b6af9b428bb1 -r ba218fa1a48e tools/libxl/xl_cmdimpl.c

...

tools/libxl/xl_cmdimpl.c skip_vfb: · line 1000

Quoted 7 lines

tools/libxl/xl_cmdimpl.c +2 -0

@@ -994,6 +995,8 @@ skip_vfb:
             if (!libxl_device_pci_parse_bdf(ctx, pcidev, buf))
                 d_config->num_pcidevs++;
         }
+        if (d_config->num_pcidevs && !c_info->hvm)
+          b_info->u.pv.machine_e820 = true;
     }

This doesn't seem to provide a way to force this behaviour on, which is (I assume) necessary if PCI devices are to be hotplugged later ?

Quoted 5 lines · Ian Jackson

Sorry for the delay replying. But:

Konrad Rzeszutek Wilk writes ("[PATCH 2 of 3] libxl: Add support for passing in the machine's E820 for PCI passthrough"):

Quoted 1 line

diff -r b6af9b428bb1 -r ba218fa1a48e tools/libxl/xl_cmdimpl.c

...

Quoted 10 lines

tools/libxl/xl_cmdimpl.c skip_vfb: · line 1000

Quoted 7 lines

tools/libxl/xl_cmdimpl.c +2 -0

@@ -994,6 +995,8 @@ skip_vfb:
             if (!libxl_device_pci_parse_bdf(ctx, pcidev, buf))
                 d_config->num_pcidevs++;
         }
+        if (d_config->num_pcidevs && !c_info->hvm)
+          b_info->u.pv.machine_e820 = true;
     }

This doesn't seem to provide a way to force this behaviour on, which is (I assume) necessary if PCI devices are to be hotplugged later ?

Right. We never resolved that question. One idea was that if you had 'pci=[]' it would trigger this. But that just seemed wrong.

Hello,

This set of v3 patches allows a PV domain to see the machine's E820 and figure out where the "PCI I/O" gap is and match it with the reality.

Changelog since v2 posting:
- Moved 'libxl__e820_alloc' to be called from do_domain_create and if machine_e820 == true.
- Made no_machine_e820 be set to true, if the guest has no PCI devices (and is PV)
- Used Keir's re-worked code for E820 creation.
Changelog since v1 posting:
- Squashed the "x86: make the pv-only e820 array be dynamic" and "x86: adjust the size of the e820 for pv guest to be dynamic" together.
- Made xc_domain_set_memmap_limit use the 'xc_domain_set_memory_map'
- Moved 'libxl_e820_alloc' and 'libxl_e820_sanitize' to be an internal operation and called from 'libxl_device_pci_parse_bdf'.
- Expanded 'libxl_device_pci_parse_bdf' API call to have an extra argument (optional).

The short end is that with these patches a PV domain can:

- Use the correct PCI I/O gap. Before these patches, Linux guest would boot up and would tell:

   [    0.000000] Allocating PCI resources starting at 40000000 (gap: 40000000:c0000000)
   while in actuality the PCI I/O gap should have been:
   [    0.000000] Allocating PCI resources starting at b0000000 (gap: b0000000:4c000000)

- The PV domain with PCI devices was limited to 3GB. It now can be booted with 4GB, 8GB, or whatever number you want. The PCI devices will now _not_ conflict with System RAM. Meaning the drivers can load.

- With 2.6.39 kernels (which has the 1-1 mapping code), the VM_IO flag will be now automatically applied to regions that are considerd PCI I/O regions. You can find out which those are by looking for '1-1' in the kernel bootup.

To use this patchset, the guest config file has to have the parameter 'pci=['<BDF>',...]' enabled.

This has been tested with 2.6.18 (RHEL5), 2.6.27(SLES11), 2.6.36, 2.6.37, 2.6.38, and 2.6.39 kernels. Also tested with PV NetBSD 5.1.

Tested this with the PCI devices (NIC, MSI), and with 2GB, 4GB, and 6GB guests with success.

 libxc/xc_domain.c      |   77 +++++++++++-----
 libxc/xc_e820.h        |    3 
 libxc/xenctrl.h        |   11 ++
 libxl/libxl.idl        |    1 
 libxl/libxl_create.c   |    8 +
 libxl/libxl_internal.h |    1 
 libxl/libxl_pci.c      |  230 +++++++++++++++++++++++++++++++++++++++++++++++++
 libxl/xl_cmdimpl.c     |    3 
 8 files changed, 309 insertions(+), 25 deletions(-)

Quoted 19 lines · Konrad Rzeszutek Wilk

Hello,

This set of v3 patches allows a PV domain to see the machine's E820 and figure out where the "PCI I/O" gap is and match it with the reality.

Changelog since v2 posting:
- Moved 'libxl__e820_alloc' to be called from do_domain_create and if machine_e820 == true.
- Made no_machine_e820 be set to true, if the guest has no PCI devices (and is PV)
- Used Keir's re-worked code for E820 creation.
Changelog since v1 posting:
- Squashed the "x86: make the pv-only e820 array be dynamic" and "x86: adjust the size of the e820 for pv guest to be dynamic" together.
- Made xc_domain_set_memmap_limit use the 'xc_domain_set_memory_map'
- Moved 'libxl_e820_alloc' and 'libxl_e820_sanitize' to be an internal operation and called from 'libxl_device_pci_parse_bdf'.
- Expanded 'libxl_device_pci_parse_bdf' API call to have an extra argument (optional).

Quoted 7 lines

The short end is that with these patches a PV domain can:

- Use the correct PCI I/O gap. Before these patches, Linux guest would boot up and would tell:

   [    0.000000] Allocating PCI resources starting at 40000000 (gap: 40000000:c0000000)
   while in actuality the PCI I/O gap should have been:
   [    0.000000] Allocating PCI resources starting at b0000000 (gap: b0000000:4c000000)

The reason it needs to be a particular gap is that we can't (easily? at all?) rewrite the device BARs to match the guest's idea of the hole, is that right? So it needs to be consistent with the underlying host hole.

I wonder if it is time to enable IOMMU for PV guests by default.

Presumably in that case we can manufacture any hole we like in the e820, which is useful e.g. when migrating to not-completely-homogeneous hosts.

Quoted 18 lines

- The PV domain with PCI devices was limited to 3GB. It now can be booted with 4GB, 8GB, or whatever number you want. The PCI devices will now _not_ conflict with System RAM. Meaning the drivers can load.

- With 2.6.39 kernels (which has the 1-1 mapping code), the VM_IO flag will be now automatically applied to regions that are considerd PCI I/O regions. You can find out which those are by looking for '1-1' in the kernel bootup.

To use this patchset, the guest config file has to have the parameter 'pci=['<BDF>',...]' enabled.

This has been tested with 2.6.18 (RHEL5), 2.6.27(SLES11), 2.6.36, 2.6.37, 2.6.38, and 2.6.39 kernels. Also tested with PV NetBSD 5.1.

Tested this with the PCI devices (NIC, MSI), and with 2GB, 4GB, and 6GB guests with success.

Quoted 11 lines

 libxc/xc_domain.c      |   77 +++++++++++-----
 libxc/xc_e820.h        |    3 
 libxc/xenctrl.h        |   11 ++
 libxl/libxl.idl        |    1 
 libxl/libxl_create.c   |    8 +
 libxl/libxl_internal.h |    1 
 libxl/libxl_pci.c      |  230 +++++++++++++++++++++++++++++++++++++++++++++++++
 libxl/xl_cmdimpl.c     |    3 
 8 files changed, 309 insertions(+), 25 deletions(-)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Quoted 28 lines · Ian Campbell

Quoted 19 lines · Konrad Rzeszutek Wilk

Hello,

This set of v3 patches allows a PV domain to see the machine's E820 and figure out where the "PCI I/O" gap is and match it with the reality.

Changelog since v2 posting:
- Moved 'libxl__e820_alloc' to be called from do_domain_create and if machine_e820 == true.
- Made no_machine_e820 be set to true, if the guest has no PCI devices (and is PV)
- Used Keir's re-worked code for E820 creation.
Changelog since v1 posting:
- Squashed the "x86: make the pv-only e820 array be dynamic" and "x86: adjust the size of the e820 for pv guest to be dynamic" together.
- Made xc_domain_set_memmap_limit use the 'xc_domain_set_memory_map'
- Moved 'libxl_e820_alloc' and 'libxl_e820_sanitize' to be an internal operation and called from 'libxl_device_pci_parse_bdf'.
- Expanded 'libxl_device_pci_parse_bdf' API call to have an extra argument (optional).

Quoted 7 lines

The short end is that with these patches a PV domain can:

- Use the correct PCI I/O gap. Before these patches, Linux guest would boot up and would tell:

   [    0.000000] Allocating PCI resources starting at 40000000 (gap: 40000000:c0000000)
   while in actuality the PCI I/O gap should have been:
   [    0.000000] Allocating PCI resources starting at b0000000 (gap: b0000000:4c000000)

Quoted 3 lines

The reason it needs to be a particular gap is that we can't (easily? at all?) rewrite the device BARs to match the guest's idea of the hole, is that right? So it needs to be consistent with the underlying host hole.

correct.

Quoted 2 lines

I wonder if it is time to enable IOMMU for PV guests by default.

Would be nice. I thought if the IOMMU was present it wouldautomatically do that?

Quoted 2 lines

Presumably in that case we can manufacture any hole we like in the e820, which is useful e.g. when migrating to not-completely-homogeneous hosts.

Hmm. I want to say yes, but not entirely sure what are all the pieces that this would entail.

Quoted 2 lines

Quoted 2 lines

This has been tested with 2.6.18 (RHEL5), 2.6.27(SLES11), 2.6.36, 2.6.37, 2.6.38, and 2.6.39 kernels. Also tested with PV NetBSD 5.1.

Quoted 22 lines

Quoted 4 lines

Tested this with the PCI devices (NIC, MSI), and with 2GB, 4GB, and 6GB guests with success.

Quoted 11 lines

 libxc/xc_domain.c      |   77 +++++++++++-----
 libxc/xc_e820.h        |    3 
 libxc/xenctrl.h        |   11 ++
 libxl/libxl.idl        |    1 
 libxl/libxl_create.c   |    8 +
 libxl/libxl_internal.h |    1 
 libxl/libxl_pci.c      |  230 +++++++++++++++++++++++++++++++++++++++++++++++++
 libxl/xl_cmdimpl.c     |    3 
 8 files changed, 309 insertions(+), 25 deletions(-)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Quoted 5 lines

Quoted 2 lines

Presumably in that case we can manufacture any hole we like in the e820, which is useful e.g. when migrating to not-completely-homogeneous hosts.

Hmm. I want to say yes, but not entirely sure what are all the pieces that this would entail.

I think it's a decision which will be internal to libxl (in particular the important thing is that the option isn't exposed in the guest cfg in a non-forward compatible way) so we can implement it as and when we get

round to it and not block this series on anything along these lines.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

This has been tested with 2.6.18 (RHEL5), 2.6.27(SLES11), 2.6.36, 2.6.37, 2.6.38, and 2.6.39 kernels. Also tested with PV NetBSD 5.1.

They all saw the full amount of RAM? Since the domain builder does not

obey the e820 I'd have thought they would end up with RAM in their I/O holes which needs to be swizzled around, which at least some of those guests won't do...

Quoted 5 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

This has been tested with 2.6.18 (RHEL5), 2.6.27(SLES11), 2.6.36, 2.6.37, 2.6.38, and 2.6.39 kernels. Also tested with PV NetBSD 5.1.

They all saw the full amount of RAM? Since the domain builder does not

Correct. So if the guest had 6GB passed in, roughly 3GB ended below the 4GB mark, and then rest - 3GB got stuck behind the 4GB.

Quoted 6 lines · Konrad Rzeszutek Wilk

Quoted 5 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

This has been tested with 2.6.18 (RHEL5), 2.6.27(SLES11), 2.6.36, 2.6.37, 2.6.38, and 2.6.39 kernels. Also tested with PV NetBSD 5.1.

They all saw the full amount of RAM? Since the domain builder does not

Quoted 2 lines

Correct. So if the guest had 6GB passed in, roughly 3GB ended below the 4GB mark, and then rest - 3GB got stuck behind the 4GB.

Do you happen to know what caused that to work? I wasn't aware of any code which moved the p2m around based on e820 in older guests (and I can't find any right now) and I didn't think the builder itself paid any attention to the e820 either.

Quoted 10 lines · Ian Campbell

Quoted 8 lines · Konrad Rzeszutek Wilk

Quoted 5 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

This has been tested with 2.6.18 (RHEL5), 2.6.27(SLES11), 2.6.36, 2.6.37, 2.6.38, and 2.6.39 kernels. Also tested with PV NetBSD 5.1.

They all saw the full amount of RAM? Since the domain builder does not

Correct. So if the guest had 6GB passed in, roughly 3GB ended below the 4GB mark, and then rest - 3GB got stuck behind the 4GB.

Quoted 4 lines

Do you happen to know what caused that to work? I wasn't aware of any code which moved the p2m around based on e820 in older guests (and I can't find any right now) and I didn't think the builder itself paid any attention to the e820 either.

Correct. The build does not pay any attention to it. But the Xen setup code in the older kernels (or perhaps it is the balloon code itself), does inflate back to what was deflated.

The PVOPS did not do that (argh, I didn't say
that in the earlier email - sorry about misleading you about that).

But just to make sure I am not feeding false information let me re-run this with an RHEL5 guest and double-check.

Quoted 8 lines

Quoted 8 lines

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

This has been tested with 2.6.18 (RHEL5), 2.6.27(SLES11), 2.6.36, 2.6.37, 2.6.38, and 2.6.39 kernels. Also tested with PV NetBSD 5.1.

They all saw the full amount of RAM? Since the domain builder does not

Correct. So if the guest had 6GB passed in, roughly 3GB ended below the 4GB mark, and then rest - 3GB got stuck behind the 4GB.

Quoted 8 lines

Quoted 4 lines

Do you happen to know what caused that to work? I wasn't aware of any code which moved the p2m around based on e820 in older guests (and I can't find any right now) and I didn't think the builder itself paid any attention to the e820 either.

Correct. The build does not pay any attention to it. But the Xen setup code in the older kernels (or perhaps it is the balloon code itself), does inflate back to what was deflated.

I'm not concerned with the ballooned case, I'm thinking of the normal "boot with all memory" case. If the builder has populated 0..4G with RAM but the E820 says 0..3G,4G..5G (i.e. an IO hole at 3G..4G) then somebody must move the mfns from pfn-space 3G..4G to 4G..5G, mustn't they? Or at least deallocate the ones in 3G..4G so that ballooning will subsequently take over.

Quoted 6 lines

The PVOPS did not do that (argh, I didn't say
that in the earlier email - sorry about misleading you about that).

But just to make sure I am not feeding false information let me re-run this with an RHEL5 guest and double-check.

Quoted 17 lines

Quoted 8 lines

Quoted 8 lines

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

This has been tested with 2.6.18 (RHEL5), 2.6.27(SLES11), 2.6.36, 2.6.37, 2.6.38, and 2.6.39 kernels. Also tested with PV NetBSD 5.1.

They all saw the full amount of RAM? Since the domain builder does not

Correct. So if the guest had 6GB passed in, roughly 3GB ended below the 4GB mark, and then rest - 3GB got stuck behind the 4GB.

Quoted 8 lines

Quoted 4 lines

Do you happen to know what caused that to work? I wasn't aware of any code which moved the p2m around based on e820 in older guests (and I can't find any right now) and I didn't think the builder itself paid any attention to the e820 either.

Correct. The build does not pay any attention to it. But the Xen setup code in the older kernels (or perhaps it is the balloon code itself), does inflate back to what was deflated.

Quoted 6 lines

I'm not concerned with the ballooned case, I'm thinking of the normal "boot with all memory" case. If the builder has populated 0..4G with RAM but the E820 says 0..3G,4G..5G (i.e. an IO hole at 3G..4G) then somebody must move the mfns from pfn-space 3G..4G to 4G..5G, mustn't they? Or at least deallocate the ones in 3G..4G so that ballooning will subsequently take over.

Right - and in the case of RHEL5, SLES11 ..

Quoted 7 lines

Quoted 6 lines

The PVOPS did not do that (argh, I didn't say
that in the earlier email - sorry about misleading you about that).

But just to make sure I am not feeding false information let me re-run this with an RHEL5 guest and double-check.

.. the kernels do all the magic stuff. I provided 4G to the guest, with a PCI device (and hence enabled the E820 patches), and while the bootup at the start initially finds:

Linux version 2.6.18-194.el5xen ([email protected]) (gcc version 4. 1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Fri Apr 2 15:34:40 EDT 2010 BIOS-provided physical RAM map:
Xen: 0000000000000000 - 00000000bffb0000 (usable)
Xen: 00000000bffb0000 - 00000000bffbe000 (ACPI data)
Xen: 00000000bffbe000 - 00000000bffe0000 (ACPI NVS)
Xen: 00000000bffe0000 - 00000000c0000000 (reserved)
Xen: 00000000fec00000 - 00000000fec01000 (reserved)
Xen: 00000000fee00000 - 00000000fee01000 (reserved)

Xen: 00000000fff00000 - 0000000100000000 (reserved)
Xen: 0000000100000000 - 0000000140850000 (usable)
..
Memory: 3026548k/5251392k available (2512k kernel code, 118176k reserved, 1396k ..
when it finished booting, I found that 4GBs are available:
[root@g-rhel5-amd64 ~]# cat /proc/meminfo | grep MemTotal
MemTotal: 4186112 kB

With the PVOPS one, it does almost all of the same thing except it does not balloon back to the initial size (4GB), so the user has to manually inflate. But that seems like a Linux kernel feature is missing: I get the same thing when my 8GB machine boots the PVOPS kernel. I end up with 6GB (no dom0_mem thing). If I try to 'inflate' past the 6GB it ignores me - while if I boot with the older Xen-O-Linux I get 8GB to Dom0.

NetBSD seems to do the same thing as RHEL5, SLES11 (note, I could not actually give it 4096MB, b/c the NetBSD unstable kernel would crash - irregardless if a PCI device was passed in. But passing in 3900MB made it work). With 3900MB and a PCI device (so with a modified E820), the guest told me I've 3891 MB free (the rest seem to be used by the kernel).

Quoted 9 lines

Linux version 2.6.18-194.el5xen ([email protected]) (gcc version 4. 1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Fri Apr 2 15:34:40 EDT 2010 BIOS-provided physical RAM map:
Xen: 0000000000000000 - 00000000bffb0000 (usable)
Xen: 00000000bffb0000 - 00000000bffbe000 (ACPI data)
Xen: 00000000bffbe000 - 00000000bffe0000 (ACPI NVS)
Xen: 00000000bffe0000 - 00000000c0000000 (reserved)
Xen: 00000000fec00000 - 00000000fec01000 (reserved)
Xen: 00000000fee00000 - 00000000fee01000 (reserved)

Quoted 8 lines

Xen: 00000000fff00000 - 0000000100000000 (reserved)
Xen: 0000000100000000 - 0000000140850000 (usable)
..
Memory: 3026548k/5251392k available (2512k kernel code, 118176k reserved, 1396k ..
when it finished booting, I found that 4GBs are available:
[root@g-rhel5-amd64 ~]# cat /proc/meminfo | grep MemTotal
MemTotal: 4186112 kB

Can you actually allocate and use (nearly) all of that 4G? (modulo kernel allocations/overheads etc). The above doesn't really show that

the p2m above 4G is actually sane (I admit I'd be surprised if we got away with it enough to boot if it wasn't...).

Quoted 2 lines

Can you actually allocate and use (nearly) all of that 4G? (modulo kernel allocations/overheads etc). The above doesn't really show that

memhog 4G worked great.. but then I noticed it started slowing down and it was using the swap disk?

Anyhow, seems that if you are using RHEL5, SLES11, you need to be carefull to use 'memory' and 'maxmem'.

With the PVOPS, need to balloon up and you are OK.

Thought I do want to see about writting the code that would automatically balloon back to the amount of memory that was deflated.

Quoted 2 lines

memhog 4G worked great.. but then I noticed it started slowing down and it was using the swap disk?

I guess the I/O holes shadowed the RAM and hence it is basically wasted.

Quoted 2 lines

Anyhow, seems that if you are using RHEL5, SLES11, you need to be carefull to use 'memory' and 'maxmem'.

Hrm, changing behaviour for existing guests isn't so nice, at least not without a way to turn the behaviour off, perhaps we do need an explicit cfg file variable to control this after all?

Quoted 5 lines

With the PVOPS, need to balloon up and you are OK.

Thought I do want to see about writting the code that would automatically balloon back to the amount of memory that was deflated.

I wonder if just writing the correct balloon target to xenstore while building the guest would be sufficient for the guest to balloon up once it's up?

Quoted 4 lines

Quoted 2 lines

memhog 4G worked great.. but then I noticed it started slowing down and it was using the swap disk?

I guess the I/O holes shadowed the RAM and hence it is basically wasted.

<nods>

Quoted 3 lines

Quoted 2 lines

Anyhow, seems that if you are using RHEL5, SLES11, you need to be carefull to use 'memory' and 'maxmem'.

Quoted 3 lines

Hrm, changing behaviour for existing guests isn't so nice, at least not without a way to turn the behaviour off, perhaps we do need an explicit cfg file variable to control this after all?

We could do that, and then once your idea below has been completly working we can rip out the parameter?

Quoted 9 lines · Konrad Rzeszutek Wilk

Quoted 4 lines

Quoted 2 lines

memhog 4G worked great.. but then I noticed it started slowing down and it was using the swap disk?

I guess the I/O holes shadowed the RAM and hence it is basically wasted.

<nods>

Quoted 3 lines

Quoted 2 lines

Anyhow, seems that if you are using RHEL5, SLES11, you need to be carefull to use 'memory' and 'maxmem'.

Quoted 6 lines

Quoted 3 lines

Hrm, changing behaviour for existing guests isn't so nice, at least not without a way to turn the behaviour off, perhaps we do need an explicit cfg file variable to control this after all?

We could do that, and then once your idea below has been completly working we can rip out the parameter?

How does this patch look to your eyes:

# HG changeset patch
# Parent c6fa04014d6e99ca4e62d04132180338403c0478
libxl: Add 'e820_host' option to config file.

.. which will be removed once the auto-ballooning of guests
with PCI devices works. During testing of the patches which provide a host E820 in a PV guest, certain inconsistencies were found with guests. When launching a RHEL5 or SLES11 PV guest with 4GB and a PCI device, the kernel would report 4GB, but have 1.5G "used". What happend was that the P2M that fall within the E820 I/O holes would never be used and was just wasted. The mechanism to go around this is to shrink the size of the guest before launch (say memory=2048, maxmem=4096) and then balloon back to 4096M after start. For PVOPS type kernels it would detect the E820 I/O holes and deflate by the correct amount but would not inflate back to 4GB. Manually inflating makes it work.

The fix in the future for guests where the memory amount flows over the PCI hole, is to launch the guest with decreased amount right up to the cusp of where the E820 PCI hole starts. Also increase the 'maxmem' by the delta and then when the guest has launched, balloon up to the delta number.

This will require some careful surgery so for right now this parameter will guard against unsuspecting users seeing their PV guests memory "vanish."

Signed-off-by Konrad Rzeszutek Wilk <[email protected]>

diff -r c6fa04014d6e tools/libxl/xl_cmdimpl.c

tools/libxl/xl_cmdimpl.c +6 -1

@@ -626,6 +626,7 @@ static void parse_config_data(const char
     XLU_ConfigList *vbds, *nics, *pcis, *cvfbs, *cpuids;
     int pci_power_mgmt = 0;
     int pci_msitranslate = 1;
+    int e820_host_override = 0;
     int e;
 
     libxl_domain_create_info *c_info = &d_config->c_info;

@@ -979,6 +980,10 @@ skip_vfb:
     if (!xlu_cfg_get_long (config, "pci_power_mgmt", &l))
         pci_power_mgmt = l;
 
+    /* To be removed once the auto ballooning after guest starts is done. */
+    if (!xlu_cfg_get_long (config, "e820_host", &l))
+        e820_host_override = l;
+
     if (!xlu_cfg_get_list (config, "pci", &pcis, 0, 0)) {
         int i;
         d_config->num_pcidevs = 0;

@@ -995,7 +1000,7 @@ skip_vfb:
             if (!libxl_device_pci_parse_bdf(ctx, pcidev, buf))
                 d_config->num_pcidevs++;
         }
-        if (d_config->num_pcidevs && !c_info->hvm)
+        if (d_config->num_pcidevs && !c_info->hvm && e820_host_override)
           b_info->u.pv.machine_e820 = true;
     }
 

Quoted 17 lines · Konrad Rzeszutek Wilk

Quoted 15 lines · Konrad Rzeszutek Wilk

Quoted 4 lines

Quoted 2 lines

memhog 4G worked great.. but then I noticed it started slowing down and it was using the swap disk?

I guess the I/O holes shadowed the RAM and hence it is basically wasted.

<nods>

Quoted 6 lines

Quoted 2 lines

Anyhow, seems that if you are using RHEL5, SLES11, you need to be carefull to use 'memory' and 'maxmem'.

Hrm, changing behaviour for existing guests isn't so nice, at least not without a way to turn the behaviour off, perhaps we do need an explicit cfg file variable to control this after all?

We could do that, and then once your idea below has been completly working we can rip out the parameter?

Quoted 1 line

How does this patch look to your eyes:

Looks ok to me.

We've been using the _override suffix for the cfg visible symbol, not just the internal variables, so if we think this is something the user typically should not touch then we should call it e820_host_override in the cfg file too. Although see my earlier comment about this option also

enabling hotplug -- perhaps this is an option user will want to care about in the long run?