Enable SEV VM to boot with assigned PCI device
cover Enable SEV VM to boot with assigned PCI device
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1667249
On an AMD SEV enabled host with an SEV enabled guest, attaching an assigned device to the VM results in a failure to start the VM:
qemu-kvm: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x0: sev_ram_block_added: failed to register region (0x7fd96e6bb000+0x20000) error 'Cannot allocate memory'
In this example the assigned device is a simple Intel 82574L NIC:
01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
Subsystem: Intel Corporation Gigabit CT Desktop Adapter Flags: bus master, fast devsel, latency 0, IRQ 89, NUMA node 0 Memory at fb9c0000 (32-bit, non-prefetchable) [size=128K] Memory at fb900000 (32-bit, non-prefetchable) [size=512K]
Note that the error indicates the region as (base+size) where a size of 0x20000 is 128K, which matches that of BAR0 for the device. dmesg on the host also reports:
SVM: SEV: Failure locking 32 pages.
SEV guests make use of the RAMBlock notifier in QEMU to add page pinnings for SEV; the kernel side of the call only knows how to pin pages with get_user_pages(), and this currently faults on non-page backed mappings (e.g. the mmap of an MMIO BAR).
To resolve this failure, change the order of the memory region type assignment and avoid pinning device memory regions.
Danilo C. L. de Paula (2):
redhat: branching qemu-kvm to rhel-8.1.0
redhat: renaming branch to rhel-8.1.0
Gary R Hook (2):
Subject: memory: Fix the memory region type assignment order
Subject: target/i386: sev: Do not pin the ram device memory region
.gitpublish | 6 +++--- memory.c | 9 ++++++++- target/i386/sev.c | 11 +++++++++++ 3 files changed, 22 insertions(+), 4 deletions(-)
Signature · 1 line
2.18.1
PATCH 1/2 memory: Fix the memory region type assignment order
BZ: 1667249
Branch: rhel-8.1.0
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1667249
Upstream Status: 4.0.0-rc1
Build Info: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=20980582
Conflicts: None
commit 2ddb89b00f947f785c9ca6742f28f954e3b75e62
Author: Singh, Brijesh <[email protected]> Date: Mon Feb 4 22:23:39 2019 +0000
memory: Fix the memory region type assignment order
Currently, a callback registered through the RAMBlock notifier is not able to get the memory region type (i.e callback is not able to use memory_region_is_ram_device function). This is because mr->ram assignment happens _after_ the memory is allocated whereas the callback is executed during allocation.
Message-Id: <[email protected]>
--- memory.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
memory.c +8 -1
@@ -1631,10 +1631,17 @@ void memory_region_init_ram_device_ptr(MemoryRegion *mr,uint64_t size, void *ptr) { -memory_region_init_ram_ptr (mr ,owner ,name ,size ,ptr ); +memory_region_init (mr ,owner ,name ,size ); + mr->ram = true; + mr->terminates = true; mr->ram_device = true; mr->ops = &ram_device_mem_ops; mr->opaque = mr; + mr->destructor = memory_region_destructor_ram; + mr->dirty_log_mask =tcg_enabled () ? (1 <<DIRTY_MEMORY_CODE ) :0 ; +/* qemu_ram_alloc_from_ptr cannot fail with ptr != NULL. */ +assert (ptr !=NULL ); + mr->ram_block =qemu_ram_alloc_from_ptr (size, ptr, mr, &error_fatal); } void memory_region_init_alias(MemoryRegion *mr,
Signature · 1 line
2.18.1
Re: [Qemu-devel] [RHEL-8.1 virt 1/2] memory: Fix the memory region type assignment order
Quoted 24 lines · Gary R Hook
BZ: 1667249
Branch: rhel-8.1.0
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1667249
Upstream Status: 4.0.0-rc1
Build Info: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=20980582
Conflicts: Nonecommit
2ddb89b00f947f785c9ca6742f28f954e3b75e62Author: Singh, Brijesh <[email protected]> Date: Mon Feb 4 22:23:39 2019 +0000memory: Fix the memory region type assignment order
Currently, a callback registered through the RAMBlock notifier is not able to get the memory region type (i.e callback is not able to use
memory_region_is_ram_devicefunction). This is becausemr->ramassignment happens _after_ the memory is allocated whereas the callback is executed during allocation.Suggested-by Alex Williamson <[email protected]>Cc Paolo Bonzini <[email protected]>Reviewed-by Alex Williamson <[email protected]>Signed-off-by Brijesh Singh <[email protected]>
Quoted 5 lines
Message-Id: <[email protected]>
Signed-off-by Paolo Bonzini <[email protected]>Cc Paolo Bonzini <[email protected]>
Faithful backport.
Signature · 1 line
Eduardo
PATCH 2/2 target/i386: sev: Do not pin the ram device memory region
In reply to Gary R HookBZ: 1667249
Branch: rhel-8.1.0
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1667249
Upstream Status: 4.0.0-rc1
Build Info: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=20980582
Conflicts: None
commit cedc0ad539afbbb669dba9e73dfad2915bc1c25b
Author: Singh, Brijesh <[email protected]> Date: Mon Feb 4 22:23:40 2019 +0000
target/i386: sev: Do not pin the ram device memory region
The RAM device presents a memory region that should be handled as an IO region and should not be pinned.
In the case of the vfio-pci, RAM device represents a MMIO BAR and the memory region is not backed by pages hence KVM_MEMORY_ENCRYPT_REG_REGION fails to lock the memory range.
Message-Id: <[email protected]>
--- target/i386/sev.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
target/i386/sev.c +11 -0
@@ -130,6 +130,17 @@ sev_ram_block_added(RAMBlockNotifier *n, void *host, size_t size) {int r;struct kvm_enc_region range; +ram_addr_t offset; +MemoryRegion *mr; + +/* +* The RAM device presents a memory region that should be treated +* as IO region and should not be pinned. +*/ + mr =memory_region_from_host (host, &offset); +if (mr &&memory_region_is_ram_device (mr)) { +return ; + } range.addr = (__u64 )(unsigned long )host; range.size = size;
Signature · 1 line
2.18.1
Re: [Qemu-devel] [RHEL-8.1 virt 2/2] target/i386: sev: Do not pin the ram device memory region
Quoted 24 lines · Gary R Hook
BZ: 1667249
Branch: rhel-8.1.0
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1667249
Upstream Status: 4.0.0-rc1
Build Info: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=20980582
Conflicts: Nonecommit
cedc0ad539afbbb669dba9e73dfad2915bc1c25bAuthor: Singh, Brijesh <[email protected]> Date: Mon Feb 4 22:23:40 2019 +0000target/i386: sev: Do not pin the ram device memory region
The RAM device presents a memory region that should be handled as an IO region and should not be pinned.
In the case of the vfio-pci, RAM device represents a MMIO BAR and the memory region is not backed by pages hence
KVM_MEMORY_ENCRYPT_REG_REGIONfails to lock the memory range.Cc Alex Williamson <[email protected]>Cc Paolo Bonzini <[email protected]>Signed-off-by Brijesh Singh <[email protected]>
Quoted 7 lines
Message-Id: <[email protected]>
Signed-off-by Paolo Bonzini <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Richard Henderson <[email protected]>Cc Eduardo Habkost <[email protected]>
Signature · 1 line
Eduardo
Re: [RHEL-8.1 virt 2/2] target/i386: sev: Do not pin the ram device memory region
In reply to Gary R Hook----- Original Message -----
Quoted 32 lines
From: "Gary R Hook" <[email protected]>
To: [email protected]
Cc: [email protected], "Paolo Bonzini" <[email protected]>, [email protected], "Eduardo Habkost" <[email protected]>, "Richard Henderson" <[email protected]>
Sent: Tuesday, April 9, 2019 7:08:03 PM
Subject: [RHEL-8.1 virt 2/2] target/i386: sev: Do not pin the ram device memory regionBZ: 1667249
Branch: rhel-8.1.0
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1667249
Upstream Status: 4.0.0-rc1
Build Info:
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=20980582
Conflicts: Nonecommit
cedc0ad539afbbb669dba9e73dfad2915bc1c25bAuthor: Singh, Brijesh <[email protected]> Date: Mon Feb 4 22:23:40 2019 +0000target/i386: sev: Do not pin the ram device memory region
The RAM device presents a memory region that should be handled as an IO region and should not be pinned.
In the case of the vfio-pci, RAM device represents a MMIO BAR and the memory region is not backed by pages hence
KVM_MEMORY_ENCRYPT_REG_REGIONfails to lock the memory range.Cc Alex Williamson <[email protected]>Cc Paolo Bonzini <[email protected]>Signed-off-by Brijesh Singh <[email protected]>
Quoted 7 lines
Message-Id: <[email protected]>
Signed-off-by Paolo Bonzini <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Richard Henderson <[email protected]>Cc Eduardo Habkost <[email protected]>
Quoted 31 lines
--- target/i386/sev.c | 11 +++++++++++ 1 file changed, 11 insertions(+)target/i386/sev.c +11
-0@@ -130,6 +130,17 @@ sev_ram_block_added(RAMBlockNotifier *n, void *host, size_t size) {int r;struct kvm_enc_region range; +ram_addr_t offset; +MemoryRegion *mr; + +/* +* The RAM device presents a memory region that should be treated +* as IO region and should not be pinned. +*/ + mr =memory_region_from_host (host, &offset); +if (mr &&memory_region_is_ram_device (mr)) { +return ; + } range.addr = (__u64 )(unsigned long )host;range.size = size;--
2.18.1
Quoted 39 lines · Gary R Hook
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1667249
On an AMD SEV enabled host with an SEV enabled guest, attaching an assigned device to the VM results in a failure to start the VM:
qemu-kvm: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x0:
sev_ram_block_added: failed to register region (0x7fd96e6bb000+0x20000) error 'Cannot allocate memory'In this example the assigned device is a simple Intel 82574L NIC:
01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
Subsystem: Intel Corporation Gigabit CT Desktop Adapter Flags: bus master, fast devsel, latency 0, IRQ 89, NUMA node 0 Memory at fb9c0000 (32-bit, non-prefetchable) [size=128K] Memory at fb900000 (32-bit, non-prefetchable) [size=512K]Note that the error indicates the region as (base+size) where a size of
0x20000is 128K, which matches that of BAR0 for the device. dmesg on the host also reports:SVM: SEV: Failure locking 32 pages.
SEV guests make use of the RAMBlock notifier in QEMU to add page pinnings for SEV; the kernel side of the call only knows how to pin pages with
get_user_pages(), and this currently faults on non-page backed mappings (e.g. the mmap of an MMIO BAR).To resolve this failure, change the order of the memory region type assignment and avoid pinning device memory regions.
Cc "Danilo C. L. de Paula" <[email protected]>Cc Eduardo Habkost <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Richard Henderson <[email protected]>Danilo C. L. de Paula (2):
redhat: branching qemu-kvm to rhel-8.1.0
redhat: renaming branch to rhel-8.1.0
Quoted 9 lines
Gary R Hook (2):
Subject: memory: Fix the memory region type assignment order
Subject: target/i386: sev: Do not pin the ram device memory region.gitpublish | 6 +++--- memory.c | 9 ++++++++- target/i386/sev.c | 11 +++++++++++ 3 files changed, 22 insertions(+), 4 deletions(-)