exclude hyperv synic sections from vhost
cover exclude hyperv synic sections from vhost
From: "Dr. David Alan Gilbert" <[email protected]>
Hyperv's synic (that we emulate) is a feature that allows the guest to place some magic (4k) pages of RAM anywhere it likes in GPA. This confuses vhost's RAM section merging when these pages land over the top of hugepages.
Since they're not normal RAM, and they shouldn't have vhost DMAing into them, exclude them from the vhost set.
I do that by marking them as device-ram and then excluding device-ram from vhost.
bz: https://bugzilla.redhat.com/show_bug.cgi?id=1779041
Dr. David Alan Gilbert (2):
vhost: Don't pass ram device sections
hyperv/synic: Allocate as ram_device
hw/hyperv/hyperv.c | 14 ++++++++------ hw/virtio/vhost.c | 1 + 2 files changed, 9 insertions(+), 6 deletions(-)
Signature · 1 line
2.24.1
PATCH 1/2 vhost: Don't pass ram device sections
From: "Dr. David Alan Gilbert" <[email protected]>
Don't pass RAM blocks that are marked as ram devices to vhost. There's normally something special about them and they're not normally just shared memory.
--- hw/virtio/vhost.c | 1 + 1 file changed, 1 insertion(+)
hw/virtio/vhost.c +1 -0
@@ -402,6 +402,7 @@ static bool vhost_section(struct vhost_dev *dev, MemoryRegionSection *section)bool log_dirty =memory_region_get_dirty_log_mask (section->mr) & ~(1 <<DIRTY_MEMORY_MIGRATION ); result =memory_region_is_ram (section->mr) && + !memory_region_is_ram_device (section->mr) && !memory_region_is_rom (section->mr);/* Vhost doesn't handle any block which is doing dirty-tracking other
Signature · 1 line
2.24.1
Re: [PATCH 1/2] vhost: Don't pass ram device sections
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Don't pass RAM blocks that are marked as ram devices to vhost. There's normally something special about them and they're not normally just shared memory.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
So I think this is good by itself.
Quoted 18 lines
--- hw/virtio/vhost.c | 1 + 1 file changed, 1 insertion(+)hw/virtio/vhost.c +1
-0@@ -402,6 +402,7 @@ static bool vhost_section(struct vhost_dev *dev, MemoryRegionSection *section)bool log_dirty =memory_region_get_dirty_log_mask (section->mr) & ~(1 <<DIRTY_MEMORY_MIGRATION ); result =memory_region_is_ram (section->mr) && + !memory_region_is_ram_device (section->mr) && !memory_region_is_rom (section->mr);/* Vhost doesn't handle any block which is doing dirty-tracking other Signature · 1 line
2.24.1
Re: [PATCH 1/2] vhost: Don't pass ram device sections
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Don't pass RAM blocks that are marked as ram devices to vhost. There's normally something special about them and they're not normally just shared memory.
Quoted 5 lines
Quoted 1 line
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Michael S. Tsirkin <[email protected]>So I think this is good by itself.
Hmm second thoughts.
vhost kernel can handle any pointer.
And what we care about for vhost-user is that memory_region_get_fd
gives us an fd.
So why does it matter that there's something special about it? I worry that this will break things like pass through of virtio-pmem for nested virt.
Quoted 18 lines
Quoted 18 lines
--- hw/virtio/vhost.c | 1 + 1 file changed, 1 insertion(+)hw/virtio/vhost.c +1
-0@@ -402,6 +402,7 @@ static bool vhost_section(struct vhost_dev *dev, MemoryRegionSection *section)bool log_dirty =memory_region_get_dirty_log_mask (section->mr) & ~(1 <<DIRTY_MEMORY_MIGRATION ); result =memory_region_is_ram (section->mr) && + !memory_region_is_ram_device (section->mr) && !memory_region_is_rom (section->mr);/* Vhost doesn't handle any block which is doing dirty-tracking other Signature · 1 line
2.24.1
Re: [PATCH 1/2] vhost: Don't pass ram device sections
In reply to Dr. David Alan Gilbert (git)Quoted 3 lines · Dr. David Alan Gilbert (git)
Don't pass RAM blocks that are marked as ram devices to vhost. There's normally something special about them and they're not normally just shared memory.
Does this something special about them, whatever it is, make them automatically ineligible for DMA? Why should vhost ignore them?
/me goes reading what ram_device memory regions are...
Roman.
Quoted 21 lines
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- hw/virtio/vhost.c | 1 + 1 file changed, 1 insertion(+)hw/virtio/vhost.c +1
-0@@ -402,6 +402,7 @@ static bool vhost_section(struct vhost_dev *dev, MemoryRegionSection *section)bool log_dirty =memory_region_get_dirty_log_mask (section->mr) & ~(1 <<DIRTY_MEMORY_MIGRATION ); result =memory_region_is_ram (section->mr) && + !memory_region_is_ram_device (section->mr) && !memory_region_is_rom (section->mr);/* Vhost doesn't handle any block which is doing dirty-tracking other Signature · 1 line
2.24.1
Re: [PATCH 1/2] vhost: Don't pass ram device sections
Quoted 10 lines · Roman Kagan
Quoted 3 lines · Dr. David Alan Gilbert (git)
Don't pass RAM blocks that are marked as ram devices to vhost. There's normally something special about them and they're not normally just shared memory.
Does this something special about them, whatever it is, make them automatically ineligible for DMA? Why should vhost ignore them? /me goes reading what
ram_devicememory regions are...Roman.
Well device RAM can be DMA'd into.
But vhost is a host side of virtio as opposed to a random device, we know what it's doing, so skipping specific areas that we know are of no interest to virtio devices for performance/internal implementation reasons is OK.
Quoted 22 lines
Quoted 22 lines
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- hw/virtio/vhost.c | 1 + 1 file changed, 1 insertion(+)hw/virtio/vhost.c +1
-0@@ -402,6 +402,7 @@ static bool vhost_section(struct vhost_dev *dev, MemoryRegionSection *section)bool log_dirty =memory_region_get_dirty_log_mask (section->mr) & ~(1 <<DIRTY_MEMORY_MIGRATION ); result =memory_region_is_ram (section->mr) && + !memory_region_is_ram_device (section->mr) && !memory_region_is_rom (section->mr);/* Vhost doesn't handle any block which is doing dirty-tracking other Signature · 1 line
2.24.1
PATCH 2/2 hyperv/synic: Allocate as ram_device
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Mark the synic pages as ram_device so that they won't be visible to vhost.
--- hw/hyperv/hyperv.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
hw/hyperv/hyperv.c +8 -6
@@ -95,12 +95,14 @@ static void synic_realize(DeviceState *dev, Error **errp) msgp_name =g_strdup_printf ("synic-%u-msg-page" , vp_index); eventp_name =g_strdup_printf ("synic-%u-event-page" , vp_index); -memory_region_init_ram (&synic->msg_page_mr, obj, msgp_name, -sizeof (*synic->msg_page), &error_abort); -memory_region_init_ram (&synic->event_page_mr, obj, eventp_name, -sizeof (*synic->event_page), &error_abort); - synic->msg_page =memory_region_get_ram_ptr (&synic->msg_page_mr); - synic->event_page =memory_region_get_ram_ptr (&synic->event_page_mr); + synic->msg_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->msg_page)); + synic->event_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->event_page)); +memory_region_init_ram_device_ptr (&synic->msg_page_mr, obj, msgp_name, +sizeof (*synic->msg_page), synic->msg_page); +memory_region_init_ram_device_ptr (&synic->event_page_mr, obj, eventp_name, +sizeof (*synic->event_page), synic->event_page);g_free (msgp_name);g_free (eventp_name);
Signature · 1 line
2.24.1
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Mark the synic pages as
ram_deviceso that they won't be visible to vhost.Signed-off-by Dr. David Alan Gilbert <[email protected]>
I think I disagree with this one.
* A RAM device represents a mapping to a physical device, such as to a PCI
* MMIO BAR of an vfio-pci assigned device. The memory region may be mapped
* into the VM address space and access to the region will modify memory
* directly. However, the memory region should not be included in a memory
* dump (device may not be enabled/mapped at the time of the dump), and
* operations incompatible with manipulating MMIO should be avoided. Replaces
* skip_dump flag.
Looks like an abuse of notation.
Quoted 31 lines
--- hw/hyperv/hyperv.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)hw/hyperv/hyperv.c +8
-6@@ -95,12 +95,14 @@ static void synic_realize(DeviceState *dev, Error **errp) msgp_name =g_strdup_printf ("synic-%u-msg-page" , vp_index); eventp_name =g_strdup_printf ("synic-%u-event-page" , vp_index); -memory_region_init_ram (&synic->msg_page_mr, obj, msgp_name, -sizeof (*synic->msg_page), &error_abort); -memory_region_init_ram (&synic->event_page_mr, obj, eventp_name, -sizeof (*synic->event_page), &error_abort); - synic->msg_page =memory_region_get_ram_ptr (&synic->msg_page_mr); - synic->event_page =memory_region_get_ram_ptr (&synic->event_page_mr); + synic->msg_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->msg_page)); + synic->event_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->event_page)); +memory_region_init_ram_device_ptr (&synic->msg_page_mr, obj, msgp_name, +sizeof (*synic->msg_page), synic->msg_page); +memory_region_init_ram_device_ptr (&synic->event_page_mr, obj, eventp_name, +sizeof (*synic->event_page), synic->event_page);g_free (msgp_name);g_free (eventp_name);Signature · 1 line
2.24.1
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 18 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Mark the synic pages as
ram_deviceso that they won't be visible to vhost.Signed-off-by Dr. David Alan Gilbert <[email protected]>I think I disagree with this one.
* A RAM device represents a mapping to a physical device, such as to a PCI
* MMIO BAR of an vfio-pci assigned device. The memory region may be mapped
* into the VM address space and access to the region will modify memory
* directly. However, the memory region should not be included in a memory
* dump (device may not be enabled/mapped at the time of the dump), and
* operations incompatible with manipulating MMIO should be avoided. Replaces
*skip_dumpflag.
Quoted 1 line
Looks like an abuse of notation.
OK, it did feel a bit like that - any suggestions of another way to do it?
This clearly isn't normal RAM.
Dave
Quoted 29 lines
Quoted 29 lines
--- hw/hyperv/hyperv.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)hw/hyperv/hyperv.c +8
-6@@ -95,12 +95,14 @@ static void synic_realize(DeviceState *dev, Error **errp) msgp_name =g_strdup_printf ("synic-%u-msg-page" , vp_index); eventp_name =g_strdup_printf ("synic-%u-event-page" , vp_index); -memory_region_init_ram (&synic->msg_page_mr, obj, msgp_name, -sizeof (*synic->msg_page), &error_abort); -memory_region_init_ram (&synic->event_page_mr, obj, eventp_name, -sizeof (*synic->event_page), &error_abort); - synic->msg_page =memory_region_get_ram_ptr (&synic->msg_page_mr); - synic->event_page =memory_region_get_ram_ptr (&synic->event_page_mr); + synic->msg_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->msg_page)); + synic->event_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->event_page)); +memory_region_init_ram_device_ptr (&synic->msg_page_mr, obj, msgp_name, +sizeof (*synic->msg_page), synic->msg_page); +memory_region_init_ram_device_ptr (&synic->event_page_mr, obj, eventp_name, +sizeof (*synic->event_page), synic->event_page);
Quoted 5 lines
Quoted 4 lines
hw/hyperv/hyperv.c +0
-0g_free (msgp_name);g_free (eventp_name);Signature · 1 line
2.24.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 19 lines · Dr. David Alan Gilbert
Quoted 18 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Mark the synic pages as
ram_deviceso that they won't be visible to vhost.Signed-off-by Dr. David Alan Gilbert <[email protected]>I think I disagree with this one.
* A RAM device represents a mapping to a physical device, such as to a PCI
* MMIO BAR of an vfio-pci assigned device. The memory region may be mapped
* into the VM address space and access to the region will modify memory
* directly. However, the memory region should not be included in a memory
* dump (device may not be enabled/mapped at the time of the dump), and
* operations incompatible with manipulating MMIO should be avoided. Replaces
*skip_dumpflag.
Quoted 7 lines
Quoted 1 line
Looks like an abuse of notation.
OK, it did feel a bit like that - any suggestions of another way to do it?
This clearly isn't normal RAM.Dave
If it's just an optimization for vhost/postcopy/etc, then I think an API that says how this isn't normal ram would be ok. E.g. it's not DMA'd into? Then maybe _nodma?
Quoted 36 lines
Quoted 29 lines
Quoted 29 lines
--- hw/hyperv/hyperv.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)hw/hyperv/hyperv.c +8
-6@@ -95,12 +95,14 @@ static void synic_realize(DeviceState *dev, Error **errp) msgp_name =g_strdup_printf ("synic-%u-msg-page" , vp_index); eventp_name =g_strdup_printf ("synic-%u-event-page" , vp_index); -memory_region_init_ram (&synic->msg_page_mr, obj, msgp_name, -sizeof (*synic->msg_page), &error_abort); -memory_region_init_ram (&synic->event_page_mr, obj, eventp_name, -sizeof (*synic->event_page), &error_abort); - synic->msg_page =memory_region_get_ram_ptr (&synic->msg_page_mr); - synic->event_page =memory_region_get_ram_ptr (&synic->event_page_mr); + synic->msg_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->msg_page)); + synic->event_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->event_page)); +memory_region_init_ram_device_ptr (&synic->msg_page_mr, obj, msgp_name, +sizeof (*synic->msg_page), synic->msg_page); +memory_region_init_ram_device_ptr (&synic->event_page_mr, obj, eventp_name, +sizeof (*synic->event_page), synic->event_page);Quoted 5 lines
Quoted 4 lines
hw/hyperv/hyperv.c +0
-0g_free (msgp_name);g_free (eventp_name);Signature · 1 line
2.24.1--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 22 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 21 lines · Dr. David Alan Gilbert
Quoted 19 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Mark the synic pages as
ram_deviceso that they won't be visible to vhost.Signed-off-by Dr. David Alan Gilbert <[email protected]>I think I disagree with this one.
* A RAM device represents a mapping to a physical device, such as to a PCI
* MMIO BAR of an vfio-pci assigned device. The memory region may be mapped
* into the VM address space and access to the region will modify memory
* directly. However, the memory region should not be included in a memory
* dump (device may not be enabled/mapped at the time of the dump), and
* operations incompatible with manipulating MMIO should be avoided. Replaces
*skip_dumpflag.Looks like an abuse of notation.
Quoted 7 lines
Quoted 5 lines
OK, it did feel a bit like that - any suggestions of another way to do it?
This clearly isn't normal RAM.Dave
If it's just an optimization for vhost/postcopy/etc, then I think
Note it's not an optimisation; postcopy fails unless you can aggregate the members of the hugepage.
And I think vhost-user will fail if you have too many sections - and the 16 sections from synic I think will blow the slots available.
Quoted 2 lines
an API that says how this isn't normal ram would be ok. E.g. it's not DMA'd into? Then maybe
_nodma?
Do we want a new memory_region_init for that or just to be able to add a flag?
Dave
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
--- hw/hyperv/hyperv.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)hw/hyperv/hyperv.c +8
-6@@ -95,12 +95,14 @@ static void synic_realize(DeviceState *dev, Error **errp) msgp_name =g_strdup_printf ("synic-%u-msg-page" , vp_index); eventp_name =g_strdup_printf ("synic-%u-event-page" , vp_index); -memory_region_init_ram (&synic->msg_page_mr, obj, msgp_name, -sizeof (*synic->msg_page), &error_abort); -memory_region_init_ram (&synic->event_page_mr, obj, eventp_name, -sizeof (*synic->event_page), &error_abort); - synic->msg_page =memory_region_get_ram_ptr (&synic->msg_page_mr); - synic->event_page =memory_region_get_ram_ptr (&synic->event_page_mr); + synic->msg_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->msg_page)); + synic->event_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->event_page)); +memory_region_init_ram_device_ptr (&synic->msg_page_mr, obj, msgp_name, +sizeof (*synic->msg_page), synic->msg_page); +memory_region_init_ram_device_ptr (&synic->event_page_mr, obj, eventp_name, +sizeof (*synic->event_page), synic->event_page);
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
hw/hyperv/hyperv.c +0
-0g_free (msgp_name);g_free (eventp_name);Signature · 1 line
2.24.1--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 3 lines · Dr. David Alan Gilbert
And I think vhost-user will fail if you have too many sections - and the 16 sections from synic I think will blow the slots available.
SynIC is percpu, it will allocate two 4k pages for every vCPU the guest has so we're potentially looking at hundreds of such regions.
Signature · 1 line
Vitaly
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 8 lines · Vitaly Kuznetsov
Quoted 3 lines · Dr. David Alan Gilbert
And I think vhost-user will fail if you have too many sections - and the 16 sections from synic I think will blow the slots available.
SynIC is percpu, it will allocate two 4k pages for every vCPU the guest has so we're potentially looking at hundreds of such regions.
Indeed.
I think my original idea to implement overlay pages word-for-word to the HyperV spec was a mistake, as it lead to fragmentation and memslot waste.
I'll look into reworking it without actually mapping extra pages over the existing RAM, but achieving overlay semantics by just shoving the *content* of the "overlaid" memory somewhere.
That said, I haven't yet fully understood how the reported issue came about, and thus whether the proposed approach would resolve it too.
Thanks,
Roman.
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 16 lines · * Roman Kagan ([email protected]) wrote:
Quoted 8 lines · Vitaly Kuznetsov
Quoted 3 lines · Dr. David Alan Gilbert
And I think vhost-user will fail if you have too many sections - and the 16 sections from synic I think will blow the slots available.
SynIC is percpu, it will allocate two 4k pages for every vCPU the guest has so we're potentially looking at hundreds of such regions.
Indeed.
I think my original idea to implement overlay pages word-for-word to the HyperV spec was a mistake, as it lead to fragmentation and memslot waste.
Quoted 6 lines
I'll look into reworking it without actually mapping extra pages over the existing RAM, but achieving overlay semantics by just shoving the *content* of the "overlaid" memory somewhere.
That said, I haven't yet fully understood how the reported issue came about, and thus whether the proposed approach would resolve it too.
The problem happens when we end up with:
a) 0-512k RAM b) 512k + synic c) 570kish-640k RAM
the page alignment code rounds
(a) to 0-2MB - aligning to the hugepage it's in
(b) leaves as is
(c) aligns to 0-2MB
it then tries to coalesce (c) and (a) and notices (b) got in the way
and fails it.
Given the guest can put Synic anywhere I'm not sure that changing it's implementatino would help here.
(And changing it's implementation would probably break migration compatibility).
Dave
Quoted 3 lines
Thanks,
Roman.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 30 lines · Dr. David Alan Gilbert
Quoted 16 lines · * Roman Kagan ([email protected]) wrote:
Quoted 8 lines · Vitaly Kuznetsov
Quoted 3 lines · Dr. David Alan Gilbert
And I think vhost-user will fail if you have too many sections - and the 16 sections from synic I think will blow the slots available.
SynIC is percpu, it will allocate two 4k pages for every vCPU the guest has so we're potentially looking at hundreds of such regions.
Indeed.
I think my original idea to implement overlay pages word-for-word to the HyperV spec was a mistake, as it lead to fragmentation and memslot waste.
Quoted 6 lines
I'll look into reworking it without actually mapping extra pages over the existing RAM, but achieving overlay semantics by just shoving the *content* of the "overlaid" memory somewhere.
That said, I haven't yet fully understood how the reported issue came about, and thus whether the proposed approach would resolve it too.
The problem happens when we end up with:
a) 0-512k RAM b) 512k + synic c) 570kish-640k RAM
Quoted 7 lines
the page alignment code rounds
(a) to 0-2MB - aligning to the hugepage it's in
(b) leaves as is
(c) aligns to 0-2MBit then tries to coalesce (c) and (a) and notices (b) got in the way
and fails it.
I see, thanks. The only bit I still haven't quite followed is how this failure results in a quiet vhost malfunction rather than a refusal to start vhost.
Quoted 2 lines
Given the guest can put Synic anywhere I'm not sure that changing it's implementatino would help here.
There would be no (b) nor (separate) (c): synic would just refer to some memory straight from (a), regardless of its paging granularity.
Quoted 2 lines
(And changing it's implementation would probably break migration compatibility).
I'm afraid I see no better option.
Thanks,
Roman.
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 36 lines · * Roman Kagan ([email protected]) wrote:
Quoted 35 lines · Dr. David Alan Gilbert
Quoted 16 lines · * Roman Kagan ([email protected]) wrote:
Quoted 8 lines · Vitaly Kuznetsov
Quoted 3 lines · Dr. David Alan Gilbert
And I think vhost-user will fail if you have too many sections - and the 16 sections from synic I think will blow the slots available.
SynIC is percpu, it will allocate two 4k pages for every vCPU the guest has so we're potentially looking at hundreds of such regions.
Indeed.
I think my original idea to implement overlay pages word-for-word to the HyperV spec was a mistake, as it lead to fragmentation and memslot waste.
Quoted 6 lines
I'll look into reworking it without actually mapping extra pages over the existing RAM, but achieving overlay semantics by just shoving the *content* of the "overlaid" memory somewhere.
That said, I haven't yet fully understood how the reported issue came about, and thus whether the proposed approach would resolve it too.
The problem happens when we end up with:
a) 0-512k RAM b) 512k + synic c) 570kish-640k RAMthe page alignment code rounds
(a) to 0-2MB - aligning to the hugepage it's in
(b) leaves as is
(c) aligns to 0-2MB
Quoted 6 lines
Quoted 2 lines
it then tries to coalesce (c) and (a) and notices (b) got in the way
and fails it.I see, thanks. The only bit I still haven't quite followed is how this failure results in a quiet vhost malfunction rather than a refusal to start vhost.
Because there's no way to fail in vhost_region_add_section other than to abort;
if (mrs_gpa < prev_gpa_start) {
error_report("%s:Section rounded to %"PRIx64
" prior to previous %"PRIx64,
__func__, mrs_gpa, prev_gpa_start);
/* A way to cleanly fail here would be better */
return;
}Quoted 5 lines
Quoted 2 lines
Given the guest can put Synic anywhere I'm not sure that changing it's implementatino would help here.
There would be no (b) nor (separate) (c): synic would just refer to some memory straight from (a), regardless of its paging granularity.
Oh, if it's actually memory from main RAM, then sure, but I guess you'd have to reserve that somehow to stop the OS using it.
Quoted 4 lines
Quoted 2 lines
(And changing it's implementation would probably break migration compatibility).
I'm afraid I see no better option.
Migration compatibility!
Dave
Quoted 3 lines
Thanks,
Roman.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 55 lines · Dr. David Alan Gilbert
Quoted 36 lines · * Roman Kagan ([email protected]) wrote:
Quoted 35 lines · Dr. David Alan Gilbert
Quoted 16 lines · * Roman Kagan ([email protected]) wrote:
Quoted 8 lines · Vitaly Kuznetsov
Quoted 3 lines · Dr. David Alan Gilbert
And I think vhost-user will fail if you have too many sections - and the 16 sections from synic I think will blow the slots available.
SynIC is percpu, it will allocate two 4k pages for every vCPU the guest has so we're potentially looking at hundreds of such regions.
Indeed.
I think my original idea to implement overlay pages word-for-word to the HyperV spec was a mistake, as it lead to fragmentation and memslot waste.
Quoted 6 lines
I'll look into reworking it without actually mapping extra pages over the existing RAM, but achieving overlay semantics by just shoving the *content* of the "overlaid" memory somewhere.
That said, I haven't yet fully understood how the reported issue came about, and thus whether the proposed approach would resolve it too.
The problem happens when we end up with:
a) 0-512k RAM b) 512k + synic c) 570kish-640k RAMthe page alignment code rounds
(a) to 0-2MB - aligning to the hugepage it's in
(b) leaves as is
(c) aligns to 0-2MBQuoted 6 lines
Quoted 2 lines
it then tries to coalesce (c) and (a) and notices (b) got in the way
and fails it.I see, thanks. The only bit I still haven't quite followed is how this failure results in a quiet vhost malfunction rather than a refusal to start vhost.
Because there's no way to fail in
vhost_region_add_sectionother than to abort;if (mrs_gpa < prev_gpa_start) { error_report("%s:Section rounded to %"PRIx64 " prior to previous %"PRIx64, __func__, mrs_gpa, prev_gpa_start); /* A way to cleanly fail here would be better */ return; }
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Given the guest can put Synic anywhere I'm not sure that changing it's implementatino would help here.
There would be no (b) nor (separate) (c): synic would just refer to some memory straight from (a), regardless of its paging granularity.
Oh, if it's actually memory from main RAM, then sure, but I guess you'd have to reserve that somehow to stop the OS using it.
It's up to the OS to use it.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
(And changing it's implementation would probably break migration compatibility).
I'm afraid I see no better option.
Migration compatibility!
Hmm, good point!
I think I should be able to achieve that, by keeping the current scheme of allocating a one-page RAM memory region for every synic page, but, instead of mapping it on top of the general RAM, swap the content with the page being "overlaid". Let me see if it works out (and hope the QEMU gang won't shoot me for such an (ab)use of memory regions).
Thanks,
Roman.
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
In reply to Dr. David Alan GilbertQuoted 27 lines · Dr. David Alan Gilbert
Quoted 26 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 25 lines · Dr. David Alan Gilbert
Quoted 18 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Mark the synic pages as
ram_deviceso that they won't be visible to vhost.Signed-off-by Dr. David Alan Gilbert <[email protected]>I think I disagree with this one.
* A RAM device represents a mapping to a physical device, such as to a PCI
* MMIO BAR of an vfio-pci assigned device. The memory region may be mapped
* into the VM address space and access to the region will modify memory
* directly. However, the memory region should not be included in a memory
* dump (device may not be enabled/mapped at the time of the dump), and
* operations incompatible with manipulating MMIO should be avoided. Replaces
*skip_dumpflag.Quoted 1 line
Looks like an abuse of notation.
OK, it did feel a bit like that - any suggestions of another way to do it?
This clearly isn't normal RAM.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Dave
If it's just an optimization for vhost/postcopy/etc, then I think
Note it's not an optimisation; postcopy fails unless you can aggregate the members of the hugepage.
And I think vhost-user will fail if you have too many sections - and the 16 sections from synic I think will blow the slots available.
Right, so both are internal reasons.
Quoted 7 lines
Quoted 2 lines
an API that says how this isn't normal ram would be ok. E.g. it's not DMA'd into? Then maybe
_nodma?Do we want a new
memory_region_initfor that or just to be able to add a flag?Dave
I think a flag API is preferable since this can apply to any kind of region. But can go either way, Paolo's the maintainer there.
Quoted 39 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
--- hw/hyperv/hyperv.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)hw/hyperv/hyperv.c +8
-6@@ -95,12 +95,14 @@ static void synic_realize(DeviceState *dev, Error **errp) msgp_name =g_strdup_printf ("synic-%u-msg-page" , vp_index); eventp_name =g_strdup_printf ("synic-%u-event-page" , vp_index); -memory_region_init_ram (&synic->msg_page_mr, obj, msgp_name, -sizeof (*synic->msg_page), &error_abort); -memory_region_init_ram (&synic->event_page_mr, obj, eventp_name, -sizeof (*synic->event_page), &error_abort); - synic->msg_page =memory_region_get_ram_ptr (&synic->msg_page_mr); - synic->event_page =memory_region_get_ram_ptr (&synic->event_page_mr); + synic->msg_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->msg_page)); + synic->event_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->event_page)); +memory_region_init_ram_device_ptr (&synic->msg_page_mr, obj, msgp_name, +sizeof (*synic->msg_page), synic->msg_page); +memory_region_init_ram_device_ptr (&synic->event_page_mr, obj, eventp_name, +sizeof (*synic->event_page), synic->event_page);Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
hw/hyperv/hyperv.c +0
-0g_free (msgp_name);g_free (eventp_name);Signature · 1 line
2.24.1--
Dr. David Alan Gilbert / [email protected] / Manchester, UK--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 42 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 27 lines · Dr. David Alan Gilbert
Quoted 26 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 25 lines · Dr. David Alan Gilbert
Quoted 18 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Mark the synic pages as
ram_deviceso that they won't be visible to vhost.Signed-off-by Dr. David Alan Gilbert <[email protected]>I think I disagree with this one.
* A RAM device represents a mapping to a physical device, such as to a PCI
* MMIO BAR of an vfio-pci assigned device. The memory region may be mapped
* into the VM address space and access to the region will modify memory
* directly. However, the memory region should not be included in a memory
* dump (device may not be enabled/mapped at the time of the dump), and
* operations incompatible with manipulating MMIO should be avoided. Replaces
*skip_dumpflag.Quoted 1 line
Looks like an abuse of notation.
OK, it did feel a bit like that - any suggestions of another way to do it?
This clearly isn't normal RAM.Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Dave
If it's just an optimization for vhost/postcopy/etc, then I think
Note it's not an optimisation; postcopy fails unless you can aggregate the members of the hugepage.
And I think vhost-user will fail if you have too many sections - and the 16 sections from synic I think will blow the slots available.Right, so both are internal reasons.
Quoted 3 lines
Quoted 2 lines
an API that says how this isn't normal ram would be ok. E.g. it's not DMA'd into? Then maybe
_nodma?
Quoted 7 lines
Quoted 4 lines
Do we want a new
memory_region_initfor that or just to be able to add a flag?Dave
I think a flag API is preferable since this can apply to any kind of region. But can go either way, Paolo's the maintainer there.
(Copying Paolo in)
So what exactly does this flag mean; to me it's 'no vhost' - but is it actually more general?
Dave
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
--- hw/hyperv/hyperv.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)hw/hyperv/hyperv.c +8
-6@@ -95,12 +95,14 @@ static void synic_realize(DeviceState *dev, Error **errp) msgp_name =g_strdup_printf ("synic-%u-msg-page" , vp_index); eventp_name =g_strdup_printf ("synic-%u-event-page" , vp_index); -memory_region_init_ram (&synic->msg_page_mr, obj, msgp_name, -sizeof (*synic->msg_page), &error_abort); -memory_region_init_ram (&synic->event_page_mr, obj, eventp_name, -sizeof (*synic->event_page), &error_abort); - synic->msg_page =memory_region_get_ram_ptr (&synic->msg_page_mr); - synic->event_page =memory_region_get_ram_ptr (&synic->event_page_mr); + synic->msg_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->msg_page)); + synic->event_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->event_page)); +memory_region_init_ram_device_ptr (&synic->msg_page_mr, obj, msgp_name, +sizeof (*synic->msg_page), synic->msg_page); +memory_region_init_ram_device_ptr (&synic->event_page_mr, obj, eventp_name, +sizeof (*synic->event_page), synic->event_page);g_free (msgp_name);g_free (eventp_name);Signature · 1 line
2.24.1
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 48 lines · Dr. David Alan Gilbert
Quoted 47 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 27 lines · Dr. David Alan Gilbert
Quoted 26 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 25 lines · Dr. David Alan Gilbert
Quoted 18 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Mark the synic pages as
ram_deviceso that they won't be visible to vhost.Signed-off-by Dr. David Alan Gilbert <[email protected]>I think I disagree with this one.
* A RAM device represents a mapping to a physical device, such as to a PCI
* MMIO BAR of an vfio-pci assigned device. The memory region may be mapped
* into the VM address space and access to the region will modify memory
* directly. However, the memory region should not be included in a memory
* dump (device may not be enabled/mapped at the time of the dump), and
* operations incompatible with manipulating MMIO should be avoided. Replaces
*skip_dumpflag.Quoted 1 line
Looks like an abuse of notation.
OK, it did feel a bit like that - any suggestions of another way to do it?
This clearly isn't normal RAM.Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Dave
If it's just an optimization for vhost/postcopy/etc, then I think
Note it's not an optimisation; postcopy fails unless you can aggregate the members of the hugepage.
And I think vhost-user will fail if you have too many sections - and the 16 sections from synic I think will blow the slots available.Right, so both are internal reasons.
Quoted 7 lines
Quoted 2 lines
an API that says how this isn't normal ram would be ok. E.g. it's not DMA'd into? Then maybe
_nodma?Do we want a new
memory_region_initfor that or just to be able to add a flag?Dave
Quoted 8 lines
Quoted 2 lines
I think a flag API is preferable since this can apply to any kind of region. But can go either way, Paolo's the maintainer there.
(Copying Paolo in)
So what exactly does this flag mean; to me it's 'no vhost' - but is it actually more general?Dave
I think it's also handy for VFIO, that should skip it too.
Quoted 42 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
--- hw/hyperv/hyperv.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)hw/hyperv/hyperv.c +8
-6@@ -95,12 +95,14 @@ static void synic_realize(DeviceState *dev, Error **errp) msgp_name =g_strdup_printf ("synic-%u-msg-page" , vp_index); eventp_name =g_strdup_printf ("synic-%u-event-page" , vp_index); -memory_region_init_ram (&synic->msg_page_mr, obj, msgp_name, -sizeof (*synic->msg_page), &error_abort); -memory_region_init_ram (&synic->event_page_mr, obj, eventp_name, -sizeof (*synic->event_page), &error_abort); - synic->msg_page =memory_region_get_ram_ptr (&synic->msg_page_mr); - synic->event_page =memory_region_get_ram_ptr (&synic->event_page_mr); + synic->msg_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->msg_page)); + synic->event_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->event_page)); +memory_region_init_ram_device_ptr (&synic->msg_page_mr, obj, msgp_name, +sizeof (*synic->msg_page), synic->msg_page); +memory_region_init_ram_device_ptr (&synic->event_page_mr, obj, eventp_name, +sizeof (*synic->event_page), synic->event_page);g_free (msgp_name);g_free (eventp_name);Signature · 1 line
2.24.1Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK--
Dr. David Alan Gilbert / [email protected] / Manchester, UK--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 52 lines · Michael S. Tsirkin
Quoted 51 lines · Dr. David Alan Gilbert
Quoted 49 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 27 lines · Dr. David Alan Gilbert
Quoted 26 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 25 lines · Dr. David Alan Gilbert
Quoted 18 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Mark the synic pages as
ram_deviceso that they won't be visible to vhost.Signed-off-by Dr. David Alan Gilbert <[email protected]>I think I disagree with this one.
* A RAM device represents a mapping to a physical device, such as to a PCI
* MMIO BAR of an vfio-pci assigned device. The memory region may be mapped
* into the VM address space and access to the region will modify memory
* directly. However, the memory region should not be included in a memory
* dump (device may not be enabled/mapped at the time of the dump), and
* operations incompatible with manipulating MMIO should be avoided. Replaces
*skip_dumpflag.Quoted 1 line
Looks like an abuse of notation.
OK, it did feel a bit like that - any suggestions of another way to do it?
This clearly isn't normal RAM.Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Dave
If it's just an optimization for vhost/postcopy/etc, then I think
Note it's not an optimisation; postcopy fails unless you can aggregate the members of the hugepage.
And I think vhost-user will fail if you have too many sections - and the 16 sections from synic I think will blow the slots available.Right, so both are internal reasons.
Quoted 7 lines
Quoted 2 lines
an API that says how this isn't normal ram would be ok. E.g. it's not DMA'd into? Then maybe
_nodma?Do we want a new
memory_region_initfor that or just to be able to add a flag?Dave
I think a flag API is preferable since this can apply to any kind of region. But can go either way, Paolo's the maintainer there.
Quoted 7 lines
Quoted 5 lines
(Copying Paolo in)
So what exactly does this flag mean; to me it's 'no vhost' - but is it actually more general?Dave
I think it's also handy for VFIO, that should skip it too.
BTW if it's "per cpu" then that is another way to put it. Neither vfio nor vhost have a concept of cpu so neither can support accessing per cpu things.
Quoted 43 lines
Quoted 43 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 34 lines
--- hw/hyperv/hyperv.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)hw/hyperv/hyperv.c +8
-6@@ -95,12 +95,14 @@ static void synic_realize(DeviceState *dev, Error **errp) msgp_name =g_strdup_printf ("synic-%u-msg-page" , vp_index); eventp_name =g_strdup_printf ("synic-%u-event-page" , vp_index); -memory_region_init_ram (&synic->msg_page_mr, obj, msgp_name, -sizeof (*synic->msg_page), &error_abort); -memory_region_init_ram (&synic->event_page_mr, obj, eventp_name, -sizeof (*synic->event_page), &error_abort); - synic->msg_page =memory_region_get_ram_ptr (&synic->msg_page_mr); - synic->event_page =memory_region_get_ram_ptr (&synic->event_page_mr); + synic->msg_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->msg_page)); + synic->event_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->event_page)); +memory_region_init_ram_device_ptr (&synic->msg_page_mr, obj, msgp_name, +sizeof (*synic->msg_page), synic->msg_page); +memory_region_init_ram_device_ptr (&synic->event_page_mr, obj, eventp_name, +sizeof (*synic->event_page), synic->event_page);g_free (msgp_name);g_free (eventp_name);Signature · 1 line
2.24.1Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK--
Dr. David Alan Gilbert / [email protected] / Manchester, UK--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 57 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 56 lines · Michael S. Tsirkin
Quoted 55 lines · Dr. David Alan Gilbert
Quoted 42 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 27 lines · Dr. David Alan Gilbert
Quoted 26 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 25 lines · Dr. David Alan Gilbert
Quoted 18 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Mark the synic pages as
ram_deviceso that they won't be visible to vhost.Signed-off-by Dr. David Alan Gilbert <[email protected]>I think I disagree with this one.
* A RAM device represents a mapping to a physical device, such as to a PCI
* MMIO BAR of an vfio-pci assigned device. The memory region may be mapped
* into the VM address space and access to the region will modify memory
* directly. However, the memory region should not be included in a memory
* dump (device may not be enabled/mapped at the time of the dump), and
* operations incompatible with manipulating MMIO should be avoided. Replaces
*skip_dumpflag.Quoted 1 line
Looks like an abuse of notation.
OK, it did feel a bit like that - any suggestions of another way to do it?
This clearly isn't normal RAM.Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Dave
If it's just an optimization for vhost/postcopy/etc, then I think
Note it's not an optimisation; postcopy fails unless you can aggregate the members of the hugepage.
And I think vhost-user will fail if you have too many sections - and the 16 sections from synic I think will blow the slots available.Right, so both are internal reasons.
Quoted 3 lines
Quoted 2 lines
an API that says how this isn't normal ram would be ok. E.g. it's not DMA'd into? Then maybe
_nodma?Quoted 7 lines
Quoted 4 lines
Do we want a new
memory_region_initfor that or just to be able to add a flag?Dave
I think a flag API is preferable since this can apply to any kind of region. But can go either way, Paolo's the maintainer there.
(Copying Paolo in)
So what exactly does this flag mean; to me it's 'no vhost' - but is it actually more general?
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Dave
I think it's also handy for VFIO, that should skip it too.
BTW if it's "per cpu" then that is another way to put it. Neither vfio nor vhost have a concept of cpu so neither can support accessing per cpu things.
In this case it's not really per-cpu in that sense; there's one of these regions per cpu, but I think each region is mapped by all cpus.
Dave
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 34 lines
--- hw/hyperv/hyperv.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)hw/hyperv/hyperv.c +8
-6@@ -95,12 +95,14 @@ static void synic_realize(DeviceState *dev, Error **errp) msgp_name =g_strdup_printf ("synic-%u-msg-page" , vp_index); eventp_name =g_strdup_printf ("synic-%u-event-page" , vp_index); -memory_region_init_ram (&synic->msg_page_mr, obj, msgp_name, -sizeof (*synic->msg_page), &error_abort); -memory_region_init_ram (&synic->event_page_mr, obj, eventp_name, -sizeof (*synic->event_page), &error_abort); - synic->msg_page =memory_region_get_ram_ptr (&synic->msg_page_mr); - synic->event_page =memory_region_get_ram_ptr (&synic->event_page_mr); + synic->msg_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->msg_page)); + synic->event_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->event_page)); +memory_region_init_ram_device_ptr (&synic->msg_page_mr, obj, msgp_name, +sizeof (*synic->msg_page), synic->msg_page); +memory_region_init_ram_device_ptr (&synic->event_page_mr, obj, eventp_name, +sizeof (*synic->event_page), synic->event_page);g_free (msgp_name);g_free (eventp_name);Signature · 1 line
2.24.1
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK--
Dr. David Alan Gilbert / [email protected] / Manchester, UK--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
In reply to Dr. David Alan GilbertQuoted 11 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 3 lines · Dr. David Alan Gilbert
Do we want a new
memory_region_initfor that or just to be able to add a flag?I think a flag API is preferable since this can apply to any kind of region. But can go either way, Paolo's the maintainer there.
(Copying Paolo in)
So what exactly does this flag mean; to me it's 'no vhost' - but is it actually more general?
It has two more effects in addition to no vhost:
1) it is skipped when dumping the guest (is this a good or bad idea for SynIC?)
2) accesses to the region will use the specified size (e.g. 4-bytes for address_space_stl, 1-byte for address_space_stb) instead of a memcpy. Doesn't really matter for SynIC regions.
If (1) is a good idea, then it's 2 out of 3 and I guess the patch is okay.
Paolo
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 15 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 3 lines · Dr. David Alan Gilbert
Do we want a new
memory_region_initfor that or just to be able to add a flag?I think a flag API is preferable since this can apply to any kind of region. But can go either way, Paolo's the maintainer there.
(Copying Paolo in)
So what exactly does this flag mean; to me it's 'no vhost' - but is it actually more general?It has two more effects in addition to no vhost:
Quoted 8 lines
1) it is skipped when dumping the guest (is this a good or bad idea for SynIC?)
2) accesses to the region will use the specified size (e.g. 4-bytes for
address_space_stl, 1-byte foraddress_space_stb) instead of a memcpy. Doesn't really matter for SynIC regions.If (1) is a good idea, then it's 2 out of 3 and I guess the patch is okay.
It's probably best to keep them in the dump because they give some info on the current state of the windows guest and interrupts. Also, as Roman points out the ram-device pages aren't migrated, so we need to fix that as well.
So, do we add a new flag? If so, is no-vhost what we want?
Dave
Quoted 2 lines
Paolo
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
In reply to Paolo BonziniQuoted 9 lines · Paolo Bonzini
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 3 lines · Dr. David Alan Gilbert
Do we want a new
memory_region_initfor that or just to be able to add a flag?I think a flag API is preferable since this can apply to any kind of region. But can go either way, Paolo's the maintainer there.
Quoted 8 lines
Quoted 3 lines
(Copying Paolo in)
So what exactly does this flag mean; to me it's 'no vhost' - but is it actually more general?It has two more effects in addition to no vhost:
1) it is skipped when dumping the guest (is this a good or bad idea for SynIC?)
Imagine we have an not yet consumed message sitting in message page, or a signalled event, do I understand correctly that these are going to get lost upon migration? This may not work then -- unless we transfer in-QEMU synic state somehow separately.
Signature · 1 line
Vitaly
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 8 lines · Vitaly Kuznetsov
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Do we want a new
memory_region_initfor that or just to be able to add a flag?I think a flag API is preferable since this can apply to any kind of region. But can go either way, Paolo's the maintainer there.
(Copying Paolo in)
So what exactly does this flag mean; to me it's 'no vhost' - but is it actually more general?
Quoted 8 lines
Quoted 4 lines
It has two more effects in addition to no vhost:
1) it is skipped when dumping the guest (is this a good or bad idea for SynIC?)
Imagine we have an not yet consumed message sitting in message page, or a signalled event, do I understand correctly that these are going to get lost upon migration? This may not work then -- unless we transfer in-QEMU synic state somehow separately.
(Thread necromancy).
This is just dumping (and it should probably be extended to core dumping, i.e. MADV_DONTDUMP). Migration is controlled separately with vmstate_register_ram.
Paolo
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
In reply to Dr. David Alan Gilbert (git)Quoted 2 lines · Dr. David Alan Gilbert (git)
Mark the synic pages as
ram_deviceso that they won't be visible to vhost.
Unless I'm misreading the code this also makes them invisible to migration.
I need some more reading on the ram_device region behavior to better understand other potential consequences.
Thanks,
Roman.
Quoted 34 lines
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- hw/hyperv/hyperv.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)hw/hyperv/hyperv.c +8
-6@@ -95,12 +95,14 @@ static void synic_realize(DeviceState *dev, Error **errp) msgp_name =g_strdup_printf ("synic-%u-msg-page" , vp_index); eventp_name =g_strdup_printf ("synic-%u-event-page" , vp_index); -memory_region_init_ram (&synic->msg_page_mr, obj, msgp_name, -sizeof (*synic->msg_page), &error_abort); -memory_region_init_ram (&synic->event_page_mr, obj, eventp_name, -sizeof (*synic->event_page), &error_abort); - synic->msg_page =memory_region_get_ram_ptr (&synic->msg_page_mr); - synic->event_page =memory_region_get_ram_ptr (&synic->event_page_mr); + synic->msg_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->msg_page)); + synic->event_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->event_page)); +memory_region_init_ram_device_ptr (&synic->msg_page_mr, obj, msgp_name, +sizeof (*synic->msg_page), synic->msg_page); +memory_region_init_ram_device_ptr (&synic->event_page_mr, obj, eventp_name, +sizeof (*synic->event_page), synic->event_page);g_free (msgp_name);g_free (eventp_name);Signature · 1 line
2.24.1
Re: [PATCH 2/2] hyperv/synic: Allocate as ram_device
Quoted 9 lines · * Roman Kagan ([email protected]) wrote:
Quoted 2 lines · Dr. David Alan Gilbert (git)
Mark the synic pages as
ram_deviceso that they won't be visible to vhost.Unless I'm misreading the code this also makes them invisible to migration.
I need some more reading on the
ram_deviceregion behavior to better understand other potential consequences.
Hmm you might be right; that's not intended.
Looking at memory.c I see we have vmstate_register_ram calls in
memory_region_init_ram
memory_region_init_rom memory_region_init_rom_device
but not
memory_region_init_ram_device_ptr
hmm; OK this needs changing then.
Dave
Quoted 31 lines
Thanks,
Roman.Quoted 28 lines
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- hw/hyperv/hyperv.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)hw/hyperv/hyperv.c +8
-6@@ -95,12 +95,14 @@ static void synic_realize(DeviceState *dev, Error **errp) msgp_name =g_strdup_printf ("synic-%u-msg-page" , vp_index); eventp_name =g_strdup_printf ("synic-%u-event-page" , vp_index); -memory_region_init_ram (&synic->msg_page_mr, obj, msgp_name, -sizeof (*synic->msg_page), &error_abort); -memory_region_init_ram (&synic->event_page_mr, obj, eventp_name, -sizeof (*synic->event_page), &error_abort); - synic->msg_page =memory_region_get_ram_ptr (&synic->msg_page_mr); - synic->event_page =memory_region_get_ram_ptr (&synic->event_page_mr); + synic->msg_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->msg_page)); + synic->event_page =qemu_memalign (qemu_real_host_page_size, +sizeof (*synic->event_page)); +memory_region_init_ram_device_ptr (&synic->msg_page_mr, obj, msgp_name, +sizeof (*synic->msg_page), synic->msg_page); +memory_region_init_ram_device_ptr (&synic->event_page_mr, obj, eventp_name, +sizeof (*synic->event_page), synic->event_page);
Quoted 7 lines
Quoted 6 lines
hw/hyperv/hyperv.c +0
-0g_free (msgp_name);g_free (eventp_name);Signature · 1 line
2.24.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Hyperv's synic (that we emulate) is a feature that allows the guest to place some magic (4k) pages of RAM anywhere it likes in GPA. This confuses vhost's RAM section merging when these pages land over the top of hugepages.
Quoted 8 lines
Since they're not normal RAM, and they shouldn't have vhost DMAing into them, exclude them from the vhost set.
I do that by marking them as device-ram and then excluding device-ram from vhost.
Signature · 1 line
Vitaly
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Hyperv's synic (that we emulate) is a feature that allows the guest to place some magic (4k) pages of RAM anywhere it likes in GPA. This confuses vhost's RAM section merging when these pages land over the top of hugepages.
Hi David:
A silly question, is this because the alignment when adding sections? If yes, what's the reason for doing alignment which is not a must for vhost memory table.
Thanks
Quoted 17 lines
Since they're not normal RAM, and they shouldn't have vhost DMAing into them, exclude them from the vhost set.
I do that by marking them as device-ram and then excluding device-ram from vhost.
bz: https://bugzilla.redhat.com/show_bug.cgi?id=1779041
Dr. David Alan Gilbert (2):
vhost: Don't pass ram device sections
hyperv/synic: Allocate asram_devicehw/hyperv/hyperv.c | 14 ++++++++------ hw/virtio/vhost.c | 1 + 2 files changed, 9 insertions(+), 6 deletions(-)
Quoted 9 lines · Jason Wang
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Hyperv's synic (that we emulate) is a feature that allows the guest to place some magic (4k) pages of RAM anywhere it likes in GPA. This confuses vhost's RAM section merging when these pages land over the top of hugepages.
Quoted 5 lines
Hi David:
A silly question, is this because the alignment when adding sections? If yes, what's the reason for doing alignment which is not a must for vhost memory table.
SynIC regions are two 4k pages and they are picked by the guest, not the host. These can be anywhere in guest's ram.
Signature · 1 line
Vitaly
Quoted 10 lines · * Jason Wang ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Hyperv's synic (that we emulate) is a feature that allows the guest to place some magic (4k) pages of RAM anywhere it likes in GPA. This confuses vhost's RAM section merging when these pages land over the top of hugepages.
Quoted 5 lines
Hi David:
A silly question, is this because the alignment when adding sections? If yes, what's the reason for doing alignment which is not a must for vhost memory table.
Page alignment is a bit odd with vhost-user - it ends up having to mmap each of the sections itself; and still has to map them as hugepages to be able to mmap - in the old world you could sometimes have the daemon mmaping the same chunk of memory twice into the vhost-user process; without the aggregation you'd get a hugepage mapping for the 0-2MB chunk for the 0-512K mapping, and then maybe another 0-2MB chunk for some of the other parts over 512K.
With postcopy we can't have the multiple mappings of the same part of guest memory; we need to have one mapping for userfault.
Also, given the 16 separate synic regions, you'd probably end up having a lot of wasted vhost-sections.
Dave
Quoted 20 lines
Thanks
Quoted 17 lines
Since they're not normal RAM, and they shouldn't have vhost DMAing into them, exclude them from the vhost set.
I do that by marking them as device-ram and then excluding device-ram from vhost.
bz: https://bugzilla.redhat.com/show_bug.cgi?id=1779041
Dr. David Alan Gilbert (2):
vhost: Don't pass ram device sections
hyperv/synic: Allocate asram_devicehw/hyperv/hyperv.c | 14 ++++++++------ hw/virtio/vhost.c | 1 + 2 files changed, 9 insertions(+), 6 deletions(-)
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 27 lines · Dr. David Alan Gilbert
Quoted 10 lines · * Jason Wang ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Hyperv's synic (that we emulate) is a feature that allows the guest to place some magic (4k) pages of RAM anywhere it likes in GPA. This confuses vhost's RAM section merging when these pages land over the top of hugepages.
Quoted 5 lines
Hi David:
A silly question, is this because the alignment when adding sections? If yes, what's the reason for doing alignment which is not a must for vhost memory table.
Page alignment is a bit odd with vhost-user - it ends up having to mmap each of the sections itself; and still has to map them as hugepages to be able to mmap - in the old world you could sometimes have the daemon mmaping the same chunk of memory twice into the vhost-user process; without the aggregation you'd get a hugepage mapping for the 0-2MB chunk for the 0-512K mapping, and then maybe another 0-2MB chunk for some of the other parts over 512K.
With postcopy we can't have the multiple mappings of the same part of guest memory; we need to have one mapping for userfault.
Quoted 4 lines
Also, given the 16 separate synic regions, you'd probably end up having a lot of wasted vhost-sections.
Dave
So I'd worry that this is more an abuse of an interface. E.g. this means it's skipped from dumps, which is not nice.
And for vhost I worry these patches will break pass-through of PCI attached memory.
Quoted 25 lines
Quoted 23 lines
Thanks
Quoted 17 lines
Since they're not normal RAM, and they shouldn't have vhost DMAing into them, exclude them from the vhost set.
I do that by marking them as device-ram and then excluding device-ram from vhost.
bz: https://bugzilla.redhat.com/show_bug.cgi?id=1779041
Dr. David Alan Gilbert (2):
vhost: Don't pass ram device sections
hyperv/synic: Allocate asram_devicehw/hyperv/hyperv.c | 14 ++++++++------ hw/virtio/vhost.c | 1 + 2 files changed, 9 insertions(+), 6 deletions(-)--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 7 lines · Dr. David Alan Gilbert (git)
Hyperv's synic (that we emulate) is a feature that allows the guest to place some magic (4k) pages of RAM anywhere it likes in GPA. This confuses vhost's RAM section merging when these pages land over the top of hugepages.
Since they're not normal RAM, and they shouldn't have vhost DMAing into them, exclude them from the vhost set.
But they *are* normal RAM. If the guest driver sets up the device to DMA to a SynIC page so be it, and the guest deserves what it gets.
Quoted 4 lines
I do that by marking them as device-ram and then excluding device-ram from vhost.
I was pointed a while back by Vitaly at
https://bugs.launchpad.net/qemu/+bug/1811533 which appeared to be the same issue, but failed to reproduce the problem. Can you please provide some more detail as to how it's triggered?
Thanks,
Roman.
Quoted 11 lines · * Roman Kagan ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)
Hyperv's synic (that we emulate) is a feature that allows the guest to place some magic (4k) pages of RAM anywhere it likes in GPA. This confuses vhost's RAM section merging when these pages land over the top of hugepages.
Since they're not normal RAM, and they shouldn't have vhost DMAing into them, exclude them from the vhost set.
But they *are* normal RAM. If the guest driver sets up the device to DMA to a SynIC page so be it, and the guest deserves what it gets.
I don't think that's guaranteed to work.
However, in our case the guest isn't doing anything that crazy; it's just setting the GPA of these pages in an inconveninent place for us.
Quoted 9 lines
Quoted 4 lines
I do that by marking them as device-ram and then excluding device-ram from vhost.
I was pointed a while back by Vitaly at
https://bugs.launchpad.net/qemu/+bug/1811533 which appeared to be the same issue, but failed to reproduce the problem. Can you please provide some more detail as to how it's triggered?
Our test script is:
/usr/libexec/qemu-kvm \
-name 'dgilbert-vm1' \
-machine q35 \
-nodefaults \
-device VGA,bus=pcie.0,addr=0x1 \
-m 4096 \
-object memory-backend-file,size=1024M,prealloc=no,mem-path=/mnt/kvm_hugepage,policy=default,id=mem-mem0 \
-object memory-backend-file,size=3072M,prealloc=no,mem-path=/mnt/kvm_hugepage,policy=default,id=mem-mem1 \
-smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2 \
-numa node,memdev=mem-mem0 \
-numa node,memdev=mem-mem1 \
-cpu SandyBridge,hv_stimer,hv_time,hv_synic,hv_vpindex \
-device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
-device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
-device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \
-drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=./win2019-64-virtio-scsi.qcow2 \
-device scsi-hd,id=image1,drive=drive_image1 \
-device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
-device virtio-net-pci,mac=9a:5c:d7:9f:cd:48,id=id7ex9m8,netdev=idim5Sro,bus=pcie.0-root-port-4,addr=0x0 \
-netdev tap,id=idim5Sro,vhost=on \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-vnc :1 \
-rtc base=localtime,clock=host,driftfix=slew \
-boot order=cdn,once=c,menu=off,strict=off \
-enable-kvm \
-device pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
-monitor stdio -qmp tcp:0:4444,server,nowaitDave
Quoted 3 lines
Thanks,
Roman.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK