Skip to contentSkip to conversation

Fix hyperv synic on vhost

PATCH In review Dr. David Alan Gilbert (git) 4 participants 11 messages Jan 16, 2020 – Jan 17, 2020

lore
Share & tools
lore
#0 lore raw

v3 cover Fix hyperv synic on vhost

From: "Dr. David Alan Gilbert" <[email protected]>

Hyperv's synic (that we emulate) is a feature that allows the guest to place some magic (4k) pages of RAM anywhere it likes in GPA. This confuses vhost's RAM section merging when these pages land over the top of hugepages.

This v3 takes a different approach to v2 and v1.
It avoids doing the hugepage alignment except for vhost-user:

a) Vhost kernel : doesn't need alignment, it's turned off synic won't cause a problem.

b) vhost user : Already filters out anything without an fd synic won't cause a problem.

(Not tried vhost-user yet, it currently seems broken even without this).

This might also cause some other reported problems with vga pages causing similar issues.

bz: https://bugzilla.redhat.com/show_bug.cgi?id=1779041

Dr. David Alan Gilbert (2):
vhost: Add names to section rounded warning
vhost: Only align sections for vhost-user

 hw/virtio/vhost.c | 39 +++++++++++++++++++++------------------
 1 file changed, 21 insertions(+), 18 deletions(-)
Signature · 1 line
2.24.1
#1 lore raw

PATCH v3 1/2 vhost: Add names to section rounded warning

From: "Dr. David Alan Gilbert" <[email protected]>

Add the memory region names to section rounding/alignment warnings.

---
 hw/virtio/vhost.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
hw/virtio/vhost.c +4 -3
index 4da0d5a6c5..774d87d98e 100644
@@ -590,9 +590,10 @@ static void vhost_region_add_section(struct vhost_dev *dev,
              * match up in the same RAMBlock if they do.
              */
             if (mrs_gpa < prev_gpa_start) {
-                error_report("%s:Section rounded to %"PRIx64
-                             " prior to previous %"PRIx64,
-                             __func__, mrs_gpa, prev_gpa_start);
+                error_report("%s:Section '%s' rounded to %"PRIx64
+                             " prior to previous '%s' %"PRIx64,
+                             __func__, section->mr->name, mrs_gpa,
+                             prev_sec->mr->name, prev_gpa_start);
                 /* A way to cleanly fail here would be better */
                 return;
             }
Signature · 1 line
2.24.1
#2 lore raw

PATCH v3 2/2 vhost: Only align sections for vhost-user

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

I added hugepage alignment code in c1ece84e7c9 to deal with vhost-user + postcopy which needs aligned pages when using userfault. However, on x86 the lower 2MB of address space tends to be shotgun'd with small fragments around the 512-640k range - e.g. video RAM, and with HyperV synic pages tend to sit around there - again splitting it up. The alignment code complains with a 'Section rounded to ...' error and gives up.

Since vhost-user already filters out devices without an fd (see vhost-user.c vhost_user_mem_section_filter) it shouldn't be affected by those overlaps.

Turn the alignment off on vhost-kernel so that it doesn't try and align, and thus won't hit the rounding issues.

---
 hw/virtio/vhost.c | 34 ++++++++++++++++++----------------
 1 file changed, 18 insertions(+), 16 deletions(-)
hw/virtio/vhost.c +18 -16
index 774d87d98e..25fd469179 100644
@@ -547,26 +547,28 @@ static void vhost_region_add_section(struct vhost_dev *dev,
     uintptr_t mrs_host = (uintptr_t)memory_region_get_ram_ptr(section->mr) +
                          section->offset_within_region;
     RAMBlock *mrs_rb = section->mr->ram_block;
-    size_t mrs_page = qemu_ram_pagesize(mrs_rb);
 
     trace_vhost_region_add_section(section->mr->name, mrs_gpa, mrs_size,
                                    mrs_host);
 
-    /* Round the section to it's page size */
-    /* First align the start down to a page boundary */
-    uint64_t alignage = mrs_host & (mrs_page - 1);
-    if (alignage) {
-        mrs_host -= alignage;
-        mrs_size += alignage;
-        mrs_gpa  -= alignage;
-    }
-    /* Now align the size up to a page boundary */
-    alignage = mrs_size & (mrs_page - 1);
-    if (alignage) {
-        mrs_size += mrs_page - alignage;
-    }
-    trace_vhost_region_add_section_aligned(section->mr->name, mrs_gpa, mrs_size,
-                                           mrs_host);
+    if (dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_USER) {   
+        /* Round the section to it's page size */
+        /* First align the start down to a page boundary */
+        size_t mrs_page = qemu_ram_pagesize(mrs_rb);
+        uint64_t alignage = mrs_host & (mrs_page - 1);
+        if (alignage) {
+            mrs_host -= alignage;
+            mrs_size += alignage;
+            mrs_gpa  -= alignage;
+        }
+        /* Now align the size up to a page boundary */
+        alignage = mrs_size & (mrs_page - 1);
+        if (alignage) {
+            mrs_size += mrs_page - alignage;
+        }
+        trace_vhost_region_add_section_aligned(section->mr->name, mrs_gpa, mrs_size,
+                                               mrs_host);
+    }
 
     if (dev->n_tmp_sections) {
         /* Since we already have at least one section, lets see if
Signature · 1 line
2.24.1
Reviewed-by on v3 2/2
#3 lore raw

Re: [PATCH v3 2/2] vhost: Only align sections for vhost-user

Quoted 8 lines · Dr. David Alan Gilbert (git)
hw/virtio/vhost.c +8 -0
+    if (dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_USER) {   
+        /* Round the section to it's page size */
+        /* First align the start down to a page boundary */
+        size_t mrs_page = qemu_ram_pagesize(mrs_rb);
+        uint64_t alignage = mrs_host & (mrs_page - 1);
+        if (alignage) {
+            mrs_host -= alignage;
+            mrs_size += alignage;

hw/virtio/vhost.c static void vhost_region_add_section(struct vhost_dev *dev, · line 571

Quoted 10 lines
hw/virtio/vhost.c +10 -0
+            mrs_gpa  -= alignage;
+        }
+        /* Now align the size up to a page boundary */
+        alignage = mrs_size & (mrs_page - 1);
+        if (alignage) {
+            mrs_size += mrs_page - alignage;
+        }
+        trace_vhost_region_add_section_aligned(section->mr->name, mrs_gpa, mrs_size,
+                                               mrs_host);
+    }

Ok, now I understand!

So it seems to me that the vhost-user protocol is deficient, it should have had two different fields for the region size and the region alignment (so that mmap does not fail). But I guess that's just yet another thing to remember for vhost-user v2.

I would add a comment to explain why the alignment is needed in the first place, but this fix is certainly much more pleasant. Thanks very much.

Paolo

#4 lore raw

Re: [PATCH v3 2/2] vhost: Only align sections for vhost-user

Quoted 20 lines · Paolo Bonzini
Quoted 8 lines · Dr. David Alan Gilbert (git)
hw/virtio/vhost.c +8 -0
+    if (dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_USER) {   
+        /* Round the section to it's page size */
+        /* First align the start down to a page boundary */
+        size_t mrs_page = qemu_ram_pagesize(mrs_rb);
+        uint64_t alignage = mrs_host & (mrs_page - 1);
+        if (alignage) {
+            mrs_host -= alignage;
+            mrs_size += alignage;

hw/virtio/vhost.c static void vhost_region_add_section(struct vhost_dev *dev, · line 571

Quoted 10 lines
hw/virtio/vhost.c +10 -0
+            mrs_gpa  -= alignage;
+        }
+        /* Now align the size up to a page boundary */
+        alignage = mrs_size & (mrs_page - 1);
+        if (alignage) {
+            mrs_size += mrs_page - alignage;
+        }
+        trace_vhost_region_add_section_aligned(section->mr->name, mrs_gpa, mrs_size,
+                                               mrs_host);
+    }
Quoted 6 lines

Ok, now I understand!

So it seems to me that the vhost-user protocol is deficient, it should have had two different fields for the region size and the region alignment (so that mmap does not fail). But I guess that's just yet another thing to remember for vhost-user v2.

We don't really need v2 just to add a field. Compatibility is maintained using feature bits. Adding that is a subject for another patch. But I'm not sure I understand why does remote need to know about alignment. This patch seems to handle it locally ...

Quoted 7 lines

I would add a comment to explain why the alignment is needed in the first place, but this fix is certainly much more pleasant. Thanks very much.

Paolo

#5 lore raw

Re: [PATCH v3 2/2] vhost: Only align sections for vhost-user

Quoted 4 lines · Michael S. Tsirkin

We don't really need v2 just to add a field. Compatibility is maintained using feature bits. Adding that is a subject for another patch. But I'm not sure I understand why does remote need to know about alignment. This patch seems to handle it locally ...

Because the remote vhost here will not be able to use the synic regions.
If it did, it would have the same overlap problem as vhost-kernel.

The alignment is needed because, even if you are mapping only [768k,1M) of a 2M hugepage, you need to mmap [0,2M). You can then discard the rest, but IIUC if you only mmap [768k,1M) then the kernel will fail the mmap.

Paolo

#6 lore raw

Re: [PATCH v3 2/2] vhost: Only align sections for vhost-user

Quoted 9 lines · Paolo Bonzini
Quoted 4 lines · Michael S. Tsirkin

We don't really need v2 just to add a field. Compatibility is maintained using feature bits. Adding that is a subject for another patch. But I'm not sure I understand why does remote need to know about alignment. This patch seems to handle it locally ...

Because the remote vhost here will not be able to use the synic regions.
If it did, it would have the same overlap problem as vhost-kernel.

Quoted 6 lines

The alignment is needed because, even if you are mapping only [768k,1M) of a 2M hugepage, you need to mmap [0,2M). You can then discard the rest, but IIUC if you only mmap [768k,1M) then the kernel will fail the mmap.

Paolo

So right now remote will query the fd passed to get the alignment. You are basically saying it's not enough in some cases?

Signature · 1 line
MST
#7 lore raw

Re: [PATCH v3 2/2] vhost: Only align sections for vhost-user

Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Paolo Bonzini
Quoted 4 lines · Michael S. Tsirkin

We don't really need v2 just to add a field. Compatibility is maintained using feature bits. Adding that is a subject for another patch. But I'm not sure I understand why does remote need to know about alignment. This patch seems to handle it locally ...

Because the remote vhost here will not be able to use the synic regions.
If it did, it would have the same overlap problem as vhost-kernel.

Quoted 6 lines
Quoted 4 lines

The alignment is needed because, even if you are mapping only [768k,1M) of a 2M hugepage, you need to mmap [0,2M). You can then discard the rest, but IIUC if you only mmap [768k,1M) then the kernel will fail the mmap.

So right now remote will query the fd passed to get the alignment.

It should, but will it? It's not in the spec and I assume QEMU is doing this alignment work because some server is not doing it. But indeed we could use a feature bit to say "don't worry I will be doing the right thing".

Paolo

Quoted 2 lines

You are basically saying it's not enough in some cases?

#8 lore raw

Re: [PATCH v3 2/2] vhost: Only align sections for vhost-user

In reply to Paolo Bonzini
Quoted 6 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 4 lines · Michael S. Tsirkin

We don't really need v2 just to add a field. Compatibility is maintained using feature bits. Adding that is a subject for another patch. But I'm not sure I understand why does remote need to know about alignment. This patch seems to handle it locally ...

Quoted 7 lines

Because the remote vhost here will not be able to use the synic regions.
If it did, it would have the same overlap problem as vhost-kernel.

The alignment is needed because, even if you are mapping only [768k,1M) of a 2M hugepage, you need to mmap [0,2M). You can then discard the rest, but IIUC if you only mmap [768k,1M) then the kernel will fail the mmap.

The vhost-user problem is worse than that.

Lets ignore the synic regions for a minute; imagine a normal PC, 0-512k ish, and a 512k-1GB map; the client side can do the alignment to allow an mmap to work for each of those two mappings, you then end up with two mmap's on the client, both aligned at 0, each covering part of the same range. If the client is careful in normal use it can make sure it's OK, but you do end up with two mappings which is odd. Once you add userfault for psotcopy, having those two mappings of the same range gets even more problematic when we wait for one of the pages to become available.

The other problem is that there's a limited number of slots for mappings in the vhost-user protocol; and they easily get filled up if you break up the memory ranges.

Doing the aggregation on the qemu side keeps your slot usage down and also removes the problem of the double mappings.

The client can't easily resolve the double mapping alignment because it can't tell if the two fd's it's been passed actually refer to the same backing file; so it can't aggregate (from memory).

Dave

Quoted 2 lines

Paolo

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#9 lore raw
In reply to Dr. David Alan Gilbert (git)
Patchew URL: https://patchew.org/QEMU/[email protected]/

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series seems to have some coding style problems. See output below for more information:

Message-id: [email protected]
Type: series
Subject: [PATCH v3 0/2] Fix hyperv synic on vhost

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

From https://github.com/patchew-project/qemu
* [new tag] patchew/[email protected] -> patchew/[email protected]
Switched to a new branch 'test'
f7aeff2 vhost: Only align sections for vhost-user 5bb467f vhost: Add names to section rounded warning

=== OUTPUT BEGIN ===
1/2 Checking commit 5bb467f4ac3b (vhost: Add names to section rounded warning) 2/2 Checking commit f7aeff24a99a (vhost: Only align sections for vhost-user)
ERROR: trailing whitespace
#45: FILE: hw/virtio/vhost.c:554:
+ if (dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_USER) { $

WARNING: line over 80 characters
#60: FILE: hw/virtio/vhost.c:569:
+ trace_vhost_region_add_section_aligned(section->mr->name, mrs_gpa, mrs_size,

total: 1 errors, 1 warnings, 43 lines checked

Patch 2/2 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1

The full log is available at
http://patchew.org/logs/[email protected]/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]

#10 lore raw
Quoted 35 lines · * [email protected] ([email protected]) wrote:

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series seems to have some coding style problems. See output below for more information:

Message-id: [email protected]
Type: series
Subject: [PATCH v3 0/2] Fix hyperv synic on vhost

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

From https://github.com/patchew-project/qemu
* [new tag] patchew/[email protected] -> patchew/[email protected]
Switched to a new branch 'test'
f7aeff2 vhost: Only align sections for vhost-user 5bb467f vhost: Add names to section rounded warning

=== OUTPUT BEGIN ===
1/2 Checking commit 5bb467f4ac3b (vhost: Add names to section rounded warning) 2/2 Checking commit f7aeff24a99a (vhost: Only align sections for vhost-user)
ERROR: trailing whitespace
#45: FILE: hw/virtio/vhost.c:554:
+ if (dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_USER) { $

Quoted 5 lines

WARNING: line over 80 characters
#60: FILE: hw/virtio/vhost.c:569:
+ trace_vhost_region_add_section_aligned(section->mr->name, mrs_gpa, mrs_size,

total: 1 errors, 1 warnings, 43 lines checked

Oops, minor whitespace; easy to fix if people actually like this fix.

Dave

Quoted 6 lines

Patch 2/2 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Quoted 8 lines

Test command exited with code: 1

The full log is available at
http://patchew.org/logs/[email protected]/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK