Skip to contentSkip to conversation

exclude hyperv synic sections from vhost

PATCH Superseded Dr. David Alan Gilbert (git) 5 participants 13 messages Jan 13, 2020 – Jan 17, 2020

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover exclude hyperv synic sections from vhost

From: "Dr. David Alan Gilbert" <[email protected]>

Hyperv's synic (that we emulate) is a feature that allows the guest to place some magic (4k) pages of RAM anywhere it likes in GPA. This confuses vhost's RAM section merging when these pages land over the top of hugepages.

Since they're not normal RAM, and they shouldn't have vhost DMAing into them, exclude them from the vhost set.

This v2 is a complete rework after the v1 review; I've now got a flag on MemoryRegion's that we set.

bz: https://bugzilla.redhat.com/show_bug.cgi?id=1779041

Dr. David Alan Gilbert (3):
vhost: Add names to section rounded warning
memory: Allow a MemoryRegion to be marked no_vhost hyperv/synic: Mark regions as no vhost

 hw/hyperv/hyperv.c    |  8 ++++++++
 hw/virtio/vhost.c     | 10 ++++++----
 include/exec/memory.h | 21 +++++++++++++++++++++
 memory.c              | 15 +++++++++++++++
 4 files changed, 50 insertions(+), 4 deletions(-)
Signature · 1 line
2.24.1
#1 lore raw

PATCH v2 1/3 vhost: Add names to section rounded warning

From: "Dr. David Alan Gilbert" <[email protected]>

Add the memory region names to section rounding/alignment warnings.

---
 hw/virtio/vhost.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
hw/virtio/vhost.c +4 -3
index 4da0d5a6c5..774d87d98e 100644
@@ -590,9 +590,10 @@ static void vhost_region_add_section(struct vhost_dev *dev,
              * match up in the same RAMBlock if they do.
              */
             if (mrs_gpa < prev_gpa_start) {
-                error_report("%s:Section rounded to %"PRIx64
-                             " prior to previous %"PRIx64,
-                             __func__, mrs_gpa, prev_gpa_start);
+                error_report("%s:Section '%s' rounded to %"PRIx64
+                             " prior to previous '%s' %"PRIx64,
+                             __func__, section->mr->name, mrs_gpa,
+                             prev_sec->mr->name, prev_gpa_start);
                 /* A way to cleanly fail here would be better */
                 return;
             }
Signature · 1 line
2.24.1
#2 lore raw

PATCH v2 2/3 memory: Allow a MemoryRegion to be marked no_vhost

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Allow a memory region to be marked as 'no_vhost' and exclude that region from vhost's list build.

---
 hw/virtio/vhost.c     |  3 ++-
 include/exec/memory.h | 21 +++++++++++++++++++++
 memory.c              | 15 +++++++++++++++
 3 files changed, 38 insertions(+), 1 deletion(-)
hw/virtio/vhost.c +2 -1
index 774d87d98e..462498bfe6 100644
@@ -402,7 +402,8 @@ static bool vhost_section(struct vhost_dev *dev, MemoryRegionSection *section)
     bool log_dirty = memory_region_get_dirty_log_mask(section->mr) &
                      ~(1 << DIRTY_MEMORY_MIGRATION);
     result = memory_region_is_ram(section->mr) &&
-        !memory_region_is_rom(section->mr);
+             !memory_region_is_rom(section->mr) &&
+             !memory_region_get_no_vhost(section->mr);
 
     /* Vhost doesn't handle any block which is doing dirty-tracking other
      * than migration; this typically fires on VGA areas.
include/exec/memory.h +21 -0
index aef8123d48..f475c06d63 100644
@@ -394,6 +394,7 @@ struct MemoryRegion {
     bool ram_device;
     bool enabled;
     bool warning_printed; /* For reservations */
+    bool no_vhost;
     uint8_t vga_logging_count;
     MemoryRegion *alias;
     hwaddr alias_offset;
@@ -1625,6 +1626,26 @@ void memory_region_set_readonly(MemoryRegion *mr, bool readonly);
  */
 void memory_region_set_nonvolatile(MemoryRegion *mr, bool nonvolatile);
 
+/**
+ * memory_region_set_no_vhost: Make vhost ignore a memory region
+ *
+ * Makes vhost ignore a memory region, useful if it isn't real
+ * DMAble memory and is at inconvenient addresses
+ *
+ * @mr: the region being updated.
+ * @no_vhost: true to ignore
+ */
+void memory_region_set_no_vhost(MemoryRegion *mr, bool no_vhost);
+
+/**
+ * memory_region_set_no_vhost: Test if memory region is marked no vhost
+ *
+ * Test if the no_vhost flag is set on the memory region
+ *
+ * @mr: the region being tested.
+ */
+bool memory_region_get_no_vhost(const MemoryRegion *mr);
+
 /**
  * memory_region_rom_device_set_romd: enable/disable ROMD mode
  *
memory.c +15 -0
index d7b9bb6951..9371998e30 100644
@@ -2136,6 +2136,21 @@ void memory_region_set_nonvolatile(MemoryRegion *mr, bool nonvolatile)
     }
 }
 
+void memory_region_set_no_vhost(MemoryRegion *mr, bool no_vhost)
+{
+    if (mr->no_vhost != no_vhost) {
+        memory_region_transaction_begin();
+        mr->no_vhost = no_vhost;
+        memory_region_update_pending |= mr->enabled;
+        memory_region_transaction_commit();
+    }
+}
+
+bool memory_region_get_no_vhost(const MemoryRegion *mr)
+{
+    return mr->no_vhost;
+}
+
 void memory_region_rom_device_set_romd(MemoryRegion *mr, bool romd_mode)
 {
     if (mr->romd_mode != romd_mode) {
Signature · 1 line
2.24.1
#3 lore raw

Re: [PATCH v2 2/3] memory: Allow a MemoryRegion to be marked no_vhost

Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Allow a memory region to be marked as 'no_vhost' and exclude that region from vhost's list build.

I thought we agreed vfio needs this as well? I'd rather this had some meaning not just "no vhost" ... no_dma?

Quoted 87 lines
---
 hw/virtio/vhost.c     |  3 ++-
 include/exec/memory.h | 21 +++++++++++++++++++++
 memory.c              | 15 +++++++++++++++
 3 files changed, 38 insertions(+), 1 deletion(-)
hw/virtio/vhost.c +2 -1
index 774d87d98e..462498bfe6 100644
@@ -402,7 +402,8 @@ static bool vhost_section(struct vhost_dev *dev, MemoryRegionSection *section)
     bool log_dirty = memory_region_get_dirty_log_mask(section->mr) &
                      ~(1 << DIRTY_MEMORY_MIGRATION);
     result = memory_region_is_ram(section->mr) &&
-        !memory_region_is_rom(section->mr);
+             !memory_region_is_rom(section->mr) &&
+             !memory_region_get_no_vhost(section->mr);
 
     /* Vhost doesn't handle any block which is doing dirty-tracking other
      * than migration; this typically fires on VGA areas.
include/exec/memory.h +21 -0
index aef8123d48..f475c06d63 100644
@@ -394,6 +394,7 @@ struct MemoryRegion {
     bool ram_device;
     bool enabled;
     bool warning_printed; /* For reservations */
+    bool no_vhost;
     uint8_t vga_logging_count;
     MemoryRegion *alias;
     hwaddr alias_offset;
@@ -1625,6 +1626,26 @@ void memory_region_set_readonly(MemoryRegion *mr, bool readonly);
  */
 void memory_region_set_nonvolatile(MemoryRegion *mr, bool nonvolatile);
 
+/**
+ * memory_region_set_no_vhost: Make vhost ignore a memory region
+ *
+ * Makes vhost ignore a memory region, useful if it isn't real
+ * DMAble memory and is at inconvenient addresses
+ *
+ * @mr: the region being updated.
+ * @no_vhost: true to ignore
+ */
+void memory_region_set_no_vhost(MemoryRegion *mr, bool no_vhost);
+
+/**
+ * memory_region_set_no_vhost: Test if memory region is marked no vhost
+ *
+ * Test if the no_vhost flag is set on the memory region
+ *
+ * @mr: the region being tested.
+ */
+bool memory_region_get_no_vhost(const MemoryRegion *mr);
+
 /**
  * memory_region_rom_device_set_romd: enable/disable ROMD mode
  *
memory.c +15 -0
index d7b9bb6951..9371998e30 100644
@@ -2136,6 +2136,21 @@ void memory_region_set_nonvolatile(MemoryRegion *mr, bool nonvolatile)
     }
 }
 
+void memory_region_set_no_vhost(MemoryRegion *mr, bool no_vhost)
+{
+    if (mr->no_vhost != no_vhost) {
+        memory_region_transaction_begin();
+        mr->no_vhost = no_vhost;
+        memory_region_update_pending |= mr->enabled;
+        memory_region_transaction_commit();
+    }
+}
+
+bool memory_region_get_no_vhost(const MemoryRegion *mr)
+{
+    return mr->no_vhost;
+}
+
 void memory_region_rom_device_set_romd(MemoryRegion *mr, bool romd_mode)
 {
     if (mr->romd_mode != romd_mode) {
Signature · 1 line
2.24.1
#4 lore raw

Re: [PATCH v2 2/3] memory: Allow a MemoryRegion to be marked no_vhost

Quoted 11 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Allow a memory region to be marked as 'no_vhost' and exclude that region from vhost's list build.

I thought we agreed vfio needs this as well? I'd rather this had some meaning not just "no vhost" ... no_dma?

VFIO tends to be unplugged on migration anyway (at the moment); but I'm happy to change the name if everyone is happy with the rest of the series and we can agree on the new name,

Dave

Quoted 79 lines
Quoted 79 lines
---
 hw/virtio/vhost.c     |  3 ++-
 include/exec/memory.h | 21 +++++++++++++++++++++
 memory.c              | 15 +++++++++++++++
 3 files changed, 38 insertions(+), 1 deletion(-)
hw/virtio/vhost.c +2 -1
index 774d87d98e..462498bfe6 100644
@@ -402,7 +402,8 @@ static bool vhost_section(struct vhost_dev *dev, MemoryRegionSection *section)
     bool log_dirty = memory_region_get_dirty_log_mask(section->mr) &
                      ~(1 << DIRTY_MEMORY_MIGRATION);
     result = memory_region_is_ram(section->mr) &&
-        !memory_region_is_rom(section->mr);
+             !memory_region_is_rom(section->mr) &&
+             !memory_region_get_no_vhost(section->mr);
 
     /* Vhost doesn't handle any block which is doing dirty-tracking other
      * than migration; this typically fires on VGA areas.
include/exec/memory.h +21 -0
index aef8123d48..f475c06d63 100644
@@ -394,6 +394,7 @@ struct MemoryRegion {
     bool ram_device;
     bool enabled;
     bool warning_printed; /* For reservations */
+    bool no_vhost;
     uint8_t vga_logging_count;
     MemoryRegion *alias;
     hwaddr alias_offset;
@@ -1625,6 +1626,26 @@ void memory_region_set_readonly(MemoryRegion *mr, bool readonly);
  */
 void memory_region_set_nonvolatile(MemoryRegion *mr, bool nonvolatile);
 
+/**
+ * memory_region_set_no_vhost: Make vhost ignore a memory region
+ *
+ * Makes vhost ignore a memory region, useful if it isn't real
+ * DMAble memory and is at inconvenient addresses
+ *
+ * @mr: the region being updated.
+ * @no_vhost: true to ignore
+ */
+void memory_region_set_no_vhost(MemoryRegion *mr, bool no_vhost);
+
+/**
+ * memory_region_set_no_vhost: Test if memory region is marked no vhost
+ *
+ * Test if the no_vhost flag is set on the memory region
+ *
+ * @mr: the region being tested.
+ */
+bool memory_region_get_no_vhost(const MemoryRegion *mr);
+
 /**
  * memory_region_rom_device_set_romd: enable/disable ROMD mode
  *
memory.c +12 -0
index d7b9bb6951..9371998e30 100644
@@ -2136,6 +2136,21 @@ void memory_region_set_nonvolatile(MemoryRegion *mr, bool nonvolatile)
     }
 }
 
+void memory_region_set_no_vhost(MemoryRegion *mr, bool no_vhost)
+{
+    if (mr->no_vhost != no_vhost) {
+        memory_region_transaction_begin();
+        mr->no_vhost = no_vhost;
+        memory_region_update_pending |= mr->enabled;
+        memory_region_transaction_commit();
+    }
+}
+
+bool memory_region_get_no_vhost(const MemoryRegion *mr)
+{
Quoted 9 lines
Quoted 8 lines
memory.c +3 -0
+    return mr->no_vhost;
+}
+
 void memory_region_rom_device_set_romd(MemoryRegion *mr, bool romd_mode)
 {
     if (mr->romd_mode != romd_mode) {
Signature · 1 line
2.24.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#5 lore raw

Re: [PATCH v2 2/3] memory: Allow a MemoryRegion to be marked no_vhost

Quoted 7 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Allow a memory region to be marked as 'no_vhost' and exclude that region from vhost's list build.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

I thought we agreed vfio needs this as well? I'd rather this had some meaning not just "no vhost" ... no_dma?

VFIO tends to be unplugged on migration anyway (at the moment);

But it still should ignore synic, right?

Quoted 95 lines

but I'm happy to change the name if everyone is happy with the rest of the series and we can agree on the new name,

Dave

Quoted 79 lines
Quoted 79 lines
---
 hw/virtio/vhost.c     |  3 ++-
 include/exec/memory.h | 21 +++++++++++++++++++++
 memory.c              | 15 +++++++++++++++
 3 files changed, 38 insertions(+), 1 deletion(-)
hw/virtio/vhost.c +2 -1
index 774d87d98e..462498bfe6 100644
@@ -402,7 +402,8 @@ static bool vhost_section(struct vhost_dev *dev, MemoryRegionSection *section)
     bool log_dirty = memory_region_get_dirty_log_mask(section->mr) &
                      ~(1 << DIRTY_MEMORY_MIGRATION);
     result = memory_region_is_ram(section->mr) &&
-        !memory_region_is_rom(section->mr);
+             !memory_region_is_rom(section->mr) &&
+             !memory_region_get_no_vhost(section->mr);
 
     /* Vhost doesn't handle any block which is doing dirty-tracking other
      * than migration; this typically fires on VGA areas.
include/exec/memory.h +21 -0
index aef8123d48..f475c06d63 100644
@@ -394,6 +394,7 @@ struct MemoryRegion {
     bool ram_device;
     bool enabled;
     bool warning_printed; /* For reservations */
+    bool no_vhost;
     uint8_t vga_logging_count;
     MemoryRegion *alias;
     hwaddr alias_offset;
@@ -1625,6 +1626,26 @@ void memory_region_set_readonly(MemoryRegion *mr, bool readonly);
  */
 void memory_region_set_nonvolatile(MemoryRegion *mr, bool nonvolatile);
 
+/**
+ * memory_region_set_no_vhost: Make vhost ignore a memory region
+ *
+ * Makes vhost ignore a memory region, useful if it isn't real
+ * DMAble memory and is at inconvenient addresses
+ *
+ * @mr: the region being updated.
+ * @no_vhost: true to ignore
+ */
+void memory_region_set_no_vhost(MemoryRegion *mr, bool no_vhost);
+
+/**
+ * memory_region_set_no_vhost: Test if memory region is marked no vhost
+ *
+ * Test if the no_vhost flag is set on the memory region
+ *
+ * @mr: the region being tested.
+ */
+bool memory_region_get_no_vhost(const MemoryRegion *mr);
+
 /**
  * memory_region_rom_device_set_romd: enable/disable ROMD mode
  *
memory.c +12 -0
index d7b9bb6951..9371998e30 100644
@@ -2136,6 +2136,21 @@ void memory_region_set_nonvolatile(MemoryRegion *mr, bool nonvolatile)
     }
 }
 
+void memory_region_set_no_vhost(MemoryRegion *mr, bool no_vhost)
+{
+    if (mr->no_vhost != no_vhost) {
+        memory_region_transaction_begin();
+        mr->no_vhost = no_vhost;
+        memory_region_update_pending |= mr->enabled;
+        memory_region_transaction_commit();
+    }
+}
+
+bool memory_region_get_no_vhost(const MemoryRegion *mr)
+{
Quoted 9 lines
Quoted 8 lines
memory.c +3 -0
+    return mr->no_vhost;
+}
+
 void memory_region_rom_device_set_romd(MemoryRegion *mr, bool romd_mode)
 {
     if (mr->romd_mode != romd_mode) {
Signature · 1 line
2.24.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#6 lore raw

Re: [PATCH v2 2/3] memory: Allow a MemoryRegion to be marked no_vhost

In reply to Michael S. Tsirkin
Quoted 11 lines · On Tue, 14 Jan 2020 02:19:39 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Allow a memory region to be marked as 'no_vhost' and exclude that region from vhost's list build.

I thought we agreed vfio needs this as well? I'd rather this had some meaning not just "no vhost" ... no_dma?

Sorry, why does vfio need this? Are we intending to disallow peer-to-peer DMA? A vfio device MMIO range could certainly be a DMA target, but we need to stick to basic CPU operations to access it (ie. no multimedia extensions like used in memcpy), that's what brought about the ram device memory region type. Thanks,

Alex

Quoted 89 lines
Quoted 87 lines
---
 hw/virtio/vhost.c     |  3 ++-
 include/exec/memory.h | 21 +++++++++++++++++++++
 memory.c              | 15 +++++++++++++++
 3 files changed, 38 insertions(+), 1 deletion(-)
hw/virtio/vhost.c +2 -1
index 774d87d98e..462498bfe6 100644
@@ -402,7 +402,8 @@ static bool vhost_section(struct vhost_dev *dev, MemoryRegionSection *section)
     bool log_dirty = memory_region_get_dirty_log_mask(section->mr) &
                      ~(1 << DIRTY_MEMORY_MIGRATION);
     result = memory_region_is_ram(section->mr) &&
-        !memory_region_is_rom(section->mr);
+             !memory_region_is_rom(section->mr) &&
+             !memory_region_get_no_vhost(section->mr);
 
     /* Vhost doesn't handle any block which is doing dirty-tracking other
      * than migration; this typically fires on VGA areas.
include/exec/memory.h +21 -0
index aef8123d48..f475c06d63 100644
@@ -394,6 +394,7 @@ struct MemoryRegion {
     bool ram_device;
     bool enabled;
     bool warning_printed; /* For reservations */
+    bool no_vhost;
     uint8_t vga_logging_count;
     MemoryRegion *alias;
     hwaddr alias_offset;
@@ -1625,6 +1626,26 @@ void memory_region_set_readonly(MemoryRegion *mr, bool readonly);
  */
 void memory_region_set_nonvolatile(MemoryRegion *mr, bool nonvolatile);
 
+/**
+ * memory_region_set_no_vhost: Make vhost ignore a memory region
+ *
+ * Makes vhost ignore a memory region, useful if it isn't real
+ * DMAble memory and is at inconvenient addresses
+ *
+ * @mr: the region being updated.
+ * @no_vhost: true to ignore
+ */
+void memory_region_set_no_vhost(MemoryRegion *mr, bool no_vhost);
+
+/**
+ * memory_region_set_no_vhost: Test if memory region is marked no vhost
+ *
+ * Test if the no_vhost flag is set on the memory region
+ *
+ * @mr: the region being tested.
+ */
+bool memory_region_get_no_vhost(const MemoryRegion *mr);
+
 /**
  * memory_region_rom_device_set_romd: enable/disable ROMD mode
  *
memory.c +15 -0
index d7b9bb6951..9371998e30 100644
@@ -2136,6 +2136,21 @@ void memory_region_set_nonvolatile(MemoryRegion *mr, bool nonvolatile)
     }
 }
 
+void memory_region_set_no_vhost(MemoryRegion *mr, bool no_vhost)
+{
+    if (mr->no_vhost != no_vhost) {
+        memory_region_transaction_begin();
+        mr->no_vhost = no_vhost;
+        memory_region_update_pending |= mr->enabled;
+        memory_region_transaction_commit();
+    }
+}
+
+bool memory_region_get_no_vhost(const MemoryRegion *mr)
+{
+    return mr->no_vhost;
+}
+
 void memory_region_rom_device_set_romd(MemoryRegion *mr, bool romd_mode)
 {
     if (mr->romd_mode != romd_mode) {
Signature · 1 line
2.24.1  
#7 lore raw

PATCH v2 3/3 hyperv/synic: Mark regions as no vhost

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Mark synic regions as novhost; they are splitting huge page memory and breaking vhost region joining.

---
 hw/hyperv/hyperv.c | 8 ++++++++
 1 file changed, 8 insertions(+)
hw/hyperv/hyperv.c +8 -0
index da8ce82725..5dbbb14fdd 100644
@@ -99,6 +99,14 @@ static void synic_realize(DeviceState *dev, Error **errp)
                            sizeof(*synic->msg_page), &error_abort);
     memory_region_init_ram(&synic->event_page_mr, obj, eventp_name,
                            sizeof(*synic->event_page), &error_abort);
+    /*
+     * The guest can put the synic pages anywhere, including
+     * fragmenting something the host might want to keep as a huge
+     * page.
+     */
+    memory_region_set_no_vhost(&synic->msg_page_mr, true);
+    memory_region_set_no_vhost(&synic->event_page_mr, true);
+
     synic->msg_page = memory_region_get_ram_ptr(&synic->msg_page_mr);
     synic->event_page = memory_region_get_ram_ptr(&synic->event_page_mr);
 
Signature · 1 line
2.24.1
#8 lore raw
In reply to Dr. David Alan Gilbert (git)
Quoted 5 lines · Dr. David Alan Gilbert (git)

Hyperv's synic (that we emulate) is a feature that allows the guest to place some magic (4k) pages of RAM anywhere it likes in GPA. This confuses vhost's RAM section merging when these pages land over the top of hugepages.

Can you explain what is the confusion like? The memory API should just tell vhost to treat it as three sections (RAM before synIC, synIC region, RAM after synIC) and it's not clear to me why postcopy breaks either.

Paolo

Quoted 2 lines

Since they're not normal RAM, and they shouldn't have vhost DMAing into them, exclude them from the vhost set.

#9 lore raw
Quoted 11 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)

Hyperv's synic (that we emulate) is a feature that allows the guest to place some magic (4k) pages of RAM anywhere it likes in GPA. This confuses vhost's RAM section merging when these pages land over the top of hugepages.

Can you explain what is the confusion like? The memory API should just tell vhost to treat it as three sections (RAM before synIC, synIC region, RAM after synIC) and it's not clear to me why postcopy breaks either.

There's two separate problems:
a) For vhost-user there's a limited size for the 'mem table' message containing the number of regions to send; that's small - so an attempt is made to coalesce regions that all refer to the same underlying RAMblock. If things split the region up you use more slots. (it's why the coalescing code was originally there.)

b) With postcopy + vhost-user life gets more complex because of userfault. We require that the vhost-user client can mmap the memory areas on host page granularity (i.e. hugepage granularity if it's hugepage backed). To do that we tweak the aggregation code to align the blocks to page size boundaries and then perform aggregation - as long as nothing else important gets in the way we're OK.
In this case the guest is programming synic to land at the 512k boundary (in 16 separate 4k pages next to each other). So we end up with 0-512k (stretched to 0..2MB alignment) - then we see synic (512k-+4k ...) then we see RAM at 640k - and when we try to align that we error because we realise the synic mapping is in the way and we can't merge the 640k ram chunk with the base 0-512k aligned chunk.

Note the reported failure here is kernel vhost, not vhost-user; so actually it probably doesn't need the alignment, and vhost-user would probably filter out the synic mappings anyway due to the fact they've not got an fd ( vhost_user_mem_section_filter ). But the alignment code always runs.

Dave

Quoted 5 lines

Paolo

Quoted 2 lines

Since they're not normal RAM, and they shouldn't have vhost DMAing into them, exclude them from the vhost set.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#10 lore raw
Quoted 35 lines · Dr. David Alan Gilbert
Quoted 11 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)

Hyperv's synic (that we emulate) is a feature that allows the guest to place some magic (4k) pages of RAM anywhere it likes in GPA. This confuses vhost's RAM section merging when these pages land over the top of hugepages.

Can you explain what is the confusion like? The memory API should just tell vhost to treat it as three sections (RAM before synIC, synIC region, RAM after synIC) and it's not clear to me why postcopy breaks either.

There's two separate problems:
a) For vhost-user there's a limited size for the 'mem table' message containing the number of regions to send; that's small - so an attempt is made to coalesce regions that all refer to the same underlying RAMblock. If things split the region up you use more slots. (it's why the coalescing code was originally there.)

b) With postcopy + vhost-user life gets more complex because of userfault. We require that the vhost-user client can mmap the memory areas on host page granularity (i.e. hugepage granularity if it's hugepage backed). To do that we tweak the aggregation code to align the blocks to page size boundaries and then perform aggregation - as long as nothing else important gets in the way we're OK.
In this case the guest is programming synic to land at the 512k boundary (in 16 separate 4k pages next to each other). So we end up with 0-512k (stretched to 0..2MB alignment) - then we see synic (512k-+4k ...) then we see RAM at 640k - and when we try to align that we error because we realise the synic mapping is in the way and we can't merge the 640k ram chunk with the base 0-512k aligned chunk.

Quoted 2 lines

Note the reported failure here is kernel vhost, not vhost-user; so actually it probably doesn't need the alignment,

Yea vhost in the kernel just does copy from/to user. No alignment requirements.

Quoted 16 lines

and vhost-user would
probably filter out the synic mappings anyway due to the fact they've not got an fd ( vhost_user_mem_section_filter ). But the alignment code always runs.

Dave

Quoted 5 lines

Paolo

Quoted 2 lines

Since they're not normal RAM, and they shouldn't have vhost DMAing into them, exclude them from the vhost set.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#11 lore raw
In reply to Paolo Bonzini
Quoted 11 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)

Hyperv's synic (that we emulate) is a feature that allows the guest to place some magic (4k) pages of RAM anywhere it likes in GPA. This confuses vhost's RAM section merging when these pages land over the top of hugepages.

Can you explain what is the confusion like? The memory API should just tell vhost to treat it as three sections (RAM before synIC, synIC region, RAM after synIC) and it's not clear to me why postcopy breaks either.

See my v3 I posted yesterday; I've made this a lot simpler by just turning the alignment off for vhost-kernel and only enabling it for vhost-user; vhost-user skips any section without a backing fd anyway, so the synic problem goes away, as does another problem reported by Peter Lieven that he was seeing that seemed like one of the VGA regions getting in the way (which I'd not seen before).

Dave

Quoted 5 lines

Paolo

Quoted 2 lines

Since they're not normal RAM, and they shouldn't have vhost DMAing into them, exclude them from the vhost set.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#12 lore raw
In reply to Dr. David Alan Gilbert (git)
Quoted 9 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Hyperv's synic (that we emulate) is a feature that allows the guest to place some magic (4k) pages of RAM anywhere it likes in GPA. This confuses vhost's RAM section merging when these pages land over the top of hugepages.

Since they're not normal RAM, and they shouldn't have vhost DMAing into them, exclude them from the vhost set.

I still don't think this is correct assessment. These pages are normal RAM, perfectly eligible for DMA and what not.

It was a thinko to implement them this way, taking the Hyper-V spec too literally. Among the downsides is the excessive consumption of KVM memslots, and unnecessary large page splits or conflicts with unsplittable ones. I'm working on an alternative approach that doesn't suffer from these issues; struggling to preserve compatibility ATM.

Thanks,
Roman.