Skip to contentSkip to conversation

Enable SEV VM to boot with assigned PCI device

PATCH NAK Gary R Hook 3 participants 7 messages Apr 10, 2019 – Nov 6, 2019

lore
Share & tools
lore
#0 lore raw

cover Enable SEV VM to boot with assigned PCI device

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1667249

On an AMD SEV enabled host with an SEV enabled guest, attaching an assigned device to the VM results in a failure to start the VM:

qemu-kvm: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x0: sev_ram_block_added: failed to register region (0x7fd96e6bb000+0x20000) error 'Cannot allocate memory'

In this example the assigned device is a simple Intel 82574L NIC:

01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

	Subsystem: Intel Corporation Gigabit CT Desktop Adapter
	Flags: bus master, fast devsel, latency 0, IRQ 89, NUMA node 0
	Memory at fb9c0000 (32-bit, non-prefetchable) [size=128K]
	Memory at fb900000 (32-bit, non-prefetchable) [size=512K]

Note that the error indicates the region as (base+size) where a size of 0x20000 is 128K, which matches that of BAR0 for the device. dmesg on the host also reports:

SVM: SEV: Failure locking 32 pages.

SEV guests make use of the RAMBlock notifier in QEMU to add page pinnings for SEV; the kernel side of the call only knows how to pin pages with get_user_pages(), and this currently faults on non-page backed mappings (e.g. the mmap of an MMIO BAR).

To resolve this failure, change the order of the memory region type assignment and avoid pinning device memory regions.

Cc "Danilo C. L. de Paula" <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>

Danilo C. L. de Paula (2):
redhat: branching qemu-kvm to rhel-8.1.0
redhat: renaming branch to rhel-8.1.0

Gary R Hook (2):
Subject: memory: Fix the memory region type assignment order
Subject: target/i386: sev: Do not pin the ram device memory region

 .gitpublish       |  6 +++---
 memory.c          |  9 ++++++++-
 target/i386/sev.c | 11 +++++++++++
 3 files changed, 22 insertions(+), 4 deletions(-)
Signature · 1 line
2.18.1
#3 lore raw

PATCH 2/2 target/i386: sev: Do not pin the ram device memory region

BZ: 1667249
Branch: rhel-8.1.0
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1667249
Upstream Status: 4.0.0-rc1
Build Info: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=20980582
Conflicts: None

commit cedc0ad539afbbb669dba9e73dfad2915bc1c25b

Author: Singh, Brijesh <[email protected]>
Date:   Mon Feb 4 22:23:40 2019 +0000

target/i386: sev: Do not pin the ram device memory region

The RAM device presents a memory region that should be handled as an IO region and should not be pinned.

In the case of the vfio-pci, RAM device represents a MMIO BAR and the memory region is not backed by pages hence KVM_MEMORY_ENCRYPT_REG_REGION fails to lock the memory range.

Message-Id: <[email protected]>

Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Eduardo Habkost <[email protected]>
---
 target/i386/sev.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
target/i386/sev.c +11 -0
index 2395171acf..b8009b001a 100644
@@ -130,6 +130,17 @@ sev_ram_block_added(RAMBlockNotifier *n, void *host, size_t size)
 {
     int r;
     struct kvm_enc_region range;
+    ram_addr_t offset;
+    MemoryRegion *mr;
+
+    /*
+     * The RAM device presents a memory region that should be treated
+     * as IO region and should not be pinned.
+     */
+    mr = memory_region_from_host(host, &offset);
+    if (mr && memory_region_is_ram_device(mr)) {
+	return;
+    }
 
     range.addr = (__u64)(unsigned long)host;
     range.size = size;
Signature · 1 line
2.18.1
#1 lore raw

PATCH 1/2 memory: Fix the memory region type assignment order

In reply to Gary R Hook

BZ: 1667249
Branch: rhel-8.1.0
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1667249
Upstream Status: 4.0.0-rc1
Build Info: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=20980582
Conflicts: None

commit 2ddb89b00f947f785c9ca6742f28f954e3b75e62

Author: Singh, Brijesh <[email protected]>
Date:   Mon Feb 4 22:23:39 2019 +0000

memory: Fix the memory region type assignment order

Currently, a callback registered through the RAMBlock notifier is not able to get the memory region type (i.e callback is not able to use memory_region_is_ram_device function). This is because mr->ram assignment happens _after_ the memory is allocated whereas the callback is executed during allocation.

Message-Id: <[email protected]>

---
 memory.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
memory.c +8 -1
index 4974f972d5..04ff5e9108 100644
@@ -1631,10 +1631,17 @@ void memory_region_init_ram_device_ptr(MemoryRegion *mr,
                                        uint64_t size,
                                        void *ptr)
 {
-    memory_region_init_ram_ptr(mr, owner, name, size, ptr);
+    memory_region_init(mr, owner, name, size);
+    mr->ram = true;
+    mr->terminates = true;
     mr->ram_device = true;
     mr->ops = &ram_device_mem_ops;
     mr->opaque = mr;
+    mr->destructor = memory_region_destructor_ram;
+    mr->dirty_log_mask = tcg_enabled() ? (1 << DIRTY_MEMORY_CODE) : 0;
+    /* qemu_ram_alloc_from_ptr cannot fail with ptr != NULL.  */
+    assert(ptr != NULL);
+    mr->ram_block = qemu_ram_alloc_from_ptr(size, ptr, mr, &error_fatal);
 }
 
 void memory_region_init_alias(MemoryRegion *mr,
Signature · 1 line
2.18.1
Acked-by on v1 1/2
#2 lore raw

Re: [Qemu-devel] [RHEL-8.1 virt 1/2] memory: Fix the memory region type assignment order

Quoted 24 lines · Gary R Hook

BZ: 1667249
Branch: rhel-8.1.0
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1667249
Upstream Status: 4.0.0-rc1
Build Info: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=20980582
Conflicts: None

commit 2ddb89b00f947f785c9ca6742f28f954e3b75e62

Author: Singh, Brijesh <[email protected]>
Date:   Mon Feb 4 22:23:39 2019 +0000

memory: Fix the memory region type assignment order

Currently, a callback registered through the RAMBlock notifier is not able to get the memory region type (i.e callback is not able to use memory_region_is_ram_device function). This is because mr->ram assignment happens _after_ the memory is allocated whereas the callback is executed during allocation.

Quoted 5 lines

Message-Id: <[email protected]>

Faithful backport.

Signature · 1 line
Eduardo
Acked-by on v1 2/2
#4 lore raw

Re: [Qemu-devel] [RHEL-8.1 virt 2/2] target/i386: sev: Do not pin the ram device memory region

In reply to Gary R Hook
Quoted 24 lines · Gary R Hook

BZ: 1667249
Branch: rhel-8.1.0
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1667249
Upstream Status: 4.0.0-rc1
Build Info: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=20980582
Conflicts: None

commit cedc0ad539afbbb669dba9e73dfad2915bc1c25b

Author: Singh, Brijesh <[email protected]>
Date:   Mon Feb 4 22:23:40 2019 +0000

target/i386: sev: Do not pin the ram device memory region

The RAM device presents a memory region that should be handled as an IO region and should not be pinned.

In the case of the vfio-pci, RAM device represents a MMIO BAR and the memory region is not backed by pages hence KVM_MEMORY_ENCRYPT_REG_REGION fails to lock the memory range.

Quoted 7 lines

Message-Id: <[email protected]>

Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Eduardo Habkost <[email protected]>
Signature · 1 line
Eduardo
Acked-by on the v1 series
#6 lore raw
In reply to Gary R Hook
Quoted 39 lines · Gary R Hook

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1667249

On an AMD SEV enabled host with an SEV enabled guest, attaching an assigned device to the VM results in a failure to start the VM:

qemu-kvm: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x0: sev_ram_block_added: failed to register region (0x7fd96e6bb000+0x20000) error 'Cannot allocate memory'

In this example the assigned device is a simple Intel 82574L NIC:

01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

	Subsystem: Intel Corporation Gigabit CT Desktop Adapter
	Flags: bus master, fast devsel, latency 0, IRQ 89, NUMA node 0
	Memory at fb9c0000 (32-bit, non-prefetchable) [size=128K]
	Memory at fb900000 (32-bit, non-prefetchable) [size=512K]

Note that the error indicates the region as (base+size) where a size of 0x20000 is 128K, which matches that of BAR0 for the device. dmesg on the host also reports:

SVM: SEV: Failure locking 32 pages.

SEV guests make use of the RAMBlock notifier in QEMU to add page pinnings for SEV; the kernel side of the call only knows how to pin pages with get_user_pages(), and this currently faults on non-page backed mappings (e.g. the mmap of an MMIO BAR).

To resolve this failure, change the order of the memory region type assignment and avoid pinning device memory regions.

Cc "Danilo C. L. de Paula" <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>

Danilo C. L. de Paula (2):
redhat: branching qemu-kvm to rhel-8.1.0
redhat: renaming branch to rhel-8.1.0

Quoted 9 lines

Gary R Hook (2):
Subject: memory: Fix the memory region type assignment order
Subject: target/i386: sev: Do not pin the ram device memory region

 .gitpublish       |  6 +++---
 memory.c          |  9 ++++++++-
 target/i386/sev.c | 11 +++++++++++
 3 files changed, 22 insertions(+), 4 deletions(-)
author NACKed-by on v1 2/2 NAK
#5 lore raw

Re: [RHEL-8.1 virt 2/2] target/i386: sev: Do not pin the ram device memory region

In reply to Gary R Hook

----- Original Message -----

Quoted 32 lines

From: "Gary R Hook" <[email protected]>
To: [email protected]
Cc: [email protected], "Paolo Bonzini" <[email protected]>, [email protected], "Eduardo Habkost" <[email protected]>, "Richard Henderson" <[email protected]>
Sent: Tuesday, April 9, 2019 7:08:03 PM
Subject: [RHEL-8.1 virt 2/2] target/i386: sev: Do not pin the ram device memory region

BZ: 1667249
Branch: rhel-8.1.0
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1667249
Upstream Status: 4.0.0-rc1
Build Info:
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=20980582
Conflicts: None

commit cedc0ad539afbbb669dba9e73dfad2915bc1c25b

Author: Singh, Brijesh <[email protected]>
Date:   Mon Feb 4 22:23:40 2019 +0000

target/i386: sev: Do not pin the ram device memory region

The RAM device presents a memory region that should be handled as an IO region and should not be pinned.

In the case of the vfio-pci, RAM device represents a MMIO BAR and the memory region is not backed by pages hence KVM_MEMORY_ENCRYPT_REG_REGION fails to lock the memory range.

Quoted 7 lines

Message-Id: <[email protected]>

Cc Paolo Bonzini <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Eduardo Habkost <[email protected]>
Quoted 31 lines
---
 target/i386/sev.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
target/i386/sev.c +11 -0
index 2395171acf..b8009b001a 100644
@@ -130,6 +130,17 @@ sev_ram_block_added(RAMBlockNotifier *n, void *host,
 size_t size)
 {
     int r;
     struct kvm_enc_region range;
+    ram_addr_t offset;
+    MemoryRegion *mr;
+
+    /*
+     * The RAM device presents a memory region that should be treated
+     * as IO region and should not be pinned.
+     */
+    mr = memory_region_from_host(host, &offset);
+    if (mr && memory_region_is_ram_device(mr)) {
+	return;
+    }
 
     range.addr = (__u64)(unsigned long)host;
     range.size = size;

--
2.18.1