Skip to contentSkip to conversation

kvm/x86: reserve bit KVM_HINTS_PHYS_ADDRESS_SIZE_DATA_VALID

PATCH In review Gerd Hoffmann 6 participants 11 messages Sep 8, 2022 – Sep 22, 2022

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kvm/x86: reserve bit KVM_HINTS_PHYS_ADDRESS_SIZE_DATA_VALID

The KVM_HINTS_PHYS_ADDRESS_SIZE_DATA_VALID bit hints to the guest that the size of the physical address space as advertised by CPUID leaf 0x80000008 is actually valid and can be used.

Unfortunately this is not the case today with qemu. Default behavior is to advertise 40 address bits (which I think comes from the very first x64 opteron processors). There are lots of intel desktop processors around which support less than that (36 or 39 depending on age), and when trying to use the full 40 bit address space on those things go south quickly.

This renders the physical address size information effectively useless for guests. This patch paves the way to fix that by adding a hint for the guest so it knows whenever the physical address size is usable or not.

The plan for qemu is to set the bit when the physical address size is valid. That is the case when qemu is started with the host-phys-bits=on option set for the cpu. Eventually qemu can also flip the default for that option from off to on, unfortunately that isn't easy for backward compatibility reasons.

The plan for the firmware is to check that bit and when it is set just query and use the available physical address space. When the bit is not set be conservative and try not exceed 36 bits (aka 64G) address space. The latter is what the firmware does today unconditionally.

---
 arch/x86/include/uapi/asm/kvm_para.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/include/uapi/asm/kvm_para.h +2 -1
index 6e64b27b2c1e..115bb34413cf 100644
@@ -37,7 +37,8 @@
 #define KVM_FEATURE_HC_MAP_GPA_RANGE	16
 #define KVM_FEATURE_MIGRATION_CONTROL	17
 
-#define KVM_HINTS_REALTIME      0
+#define KVM_HINTS_REALTIME                      0
+#define KVM_HINTS_PHYS_ADDRESS_SIZE_DATA_VALID  1
 
 /* The last 8 bits are used to indicate how to interpret the flags field
  * in pvclock structure. If no bits are set, all flags are ignored.
Signature · 1 line
2.37.3
#1 lore raw
Quoted 36 lines · Gerd Hoffmann

The KVM_HINTS_PHYS_ADDRESS_SIZE_DATA_VALID bit hints to the guest that the size of the physical address space as advertised by CPUID leaf 0x80000008 is actually valid and can be used.

Unfortunately this is not the case today with qemu. Default behavior is to advertise 40 address bits (which I think comes from the very first x64 opteron processors). There are lots of intel desktop processors around which support less than that (36 or 39 depending on age), and when trying to use the full 40 bit address space on those things go south quickly.

This renders the physical address size information effectively useless for guests. This patch paves the way to fix that by adding a hint for the guest so it knows whenever the physical address size is usable or not.

The plan for qemu is to set the bit when the physical address size is valid. That is the case when qemu is started with the host-phys-bits=on option set for the cpu. Eventually qemu can also flip the default for that option from off to on, unfortunately that isn't easy for backward compatibility reasons.

The plan for the firmware is to check that bit and when it is set just query and use the available physical address space. When the bit is not set be conservative and try not exceed 36 bits (aka 64G) address space. The latter is what the firmware does today unconditionally.

---
 arch/x86/include/uapi/asm/kvm_para.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/include/uapi/asm/kvm_para.h +0 -0
index 6e64b27b2c1e..115bb34413cf 100644

arch/x86/include/uapi/asm/kvm_para.h line 41

Quoted 6 lines
arch/x86/include/uapi/asm/kvm_para.h +2 -1
@@ -37,7 +37,8 @@
 #define KVM_FEATURE_HC_MAP_GPA_RANGE	16
 #define KVM_FEATURE_MIGRATION_CONTROL	17
 
-#define KVM_HINTS_REALTIME      0
+#define KVM_HINTS_REALTIME                      0
+#define KVM_HINTS_PHYS_ADDRESS_SIZE_DATA_VALID  1

Why does KVM need to get involved? This is purely a userspace problem. E.g. why not use QEMU's fw_cfg to communicate this information to the guest?

Defining this flag arguably breaks backwards compatibility for VMMs that already accurately advertise MAXPHYADDR. The absence of the flag would imply that MAXPHYADDR is invalid, which is not the case.

#2 lore raw
Quoted 40 lines · Sean Christopherson
Quoted 39 lines · Gerd Hoffmann

The KVM_HINTS_PHYS_ADDRESS_SIZE_DATA_VALID bit hints to the guest that the size of the physical address space as advertised by CPUID leaf 0x80000008 is actually valid and can be used.

Unfortunately this is not the case today with qemu. Default behavior is to advertise 40 address bits (which I think comes from the very first x64 opteron processors). There are lots of intel desktop processors around which support less than that (36 or 39 depending on age), and when trying to use the full 40 bit address space on those things go south quickly.

This renders the physical address size information effectively useless for guests. This patch paves the way to fix that by adding a hint for the guest so it knows whenever the physical address size is usable or not.

The plan for qemu is to set the bit when the physical address size is valid. That is the case when qemu is started with the host-phys-bits=on option set for the cpu. Eventually qemu can also flip the default for that option from off to on, unfortunately that isn't easy for backward compatibility reasons.

The plan for the firmware is to check that bit and when it is set just query and use the available physical address space. When the bit is not set be conservative and try not exceed 36 bits (aka 64G) address space. The latter is what the firmware does today unconditionally.

---
 arch/x86/include/uapi/asm/kvm_para.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/include/uapi/asm/kvm_para.h +0 -0
index 6e64b27b2c1e..115bb34413cf 100644
@@ -37,7 +37,8 @@
 #define KVM_FEATURE_HC_MAP_GPA_RANGE	16
 #define KVM_FEATURE_MIGRATION_CONTROL	17
 
Quoted 5 lines

arch/x86/include/uapi/asm/kvm_para.h line 41

Quoted 3 lines
arch/x86/include/uapi/asm/kvm_para.h +2 -1
-#define KVM_HINTS_REALTIME      0
+#define KVM_HINTS_REALTIME                      0
+#define KVM_HINTS_PHYS_ADDRESS_SIZE_DATA_VALID  1

Why does KVM need to get involved? This is purely a userspace problem.

It doesn't. I only need reserve a hints bit, and the canonical source for that happens to live in the kernel. That's why this patch doesn't touch any actual code ;)

Quoted 2 lines

E.g. why not use QEMU's fw_cfg to communicate this information to the guest?

That is indeed the other obvious way to implement this. Given this information will be needed in code paths which already do CPUID queries using CPUID to transport that information looked like the better option to me.

Quoted 3 lines

Defining this flag arguably breaks backwards compatibility for VMMs that already accurately advertise MAXPHYADDR. The absence of the flag would imply that MAXPHYADDR is invalid, which is not the case.

That is true no matter how we try to transport that information from the host to the guest (even with fw_cfg because other hypervisors start using that interface too).

In practice it is not much of a problem though. The firmware needs to know the exact platform it runs on anyway to initialize everything properly, so the logic can easily be restricted to qemu.

take care,
Gerd

#3 lore raw
Quoted 2 lines · Gerd Hoffmann
Quoted 1 line · Sean Christopherson

On Thu, Sep 08, 2022, Gerd Hoffmann wrote:

...

Quoted 21 lines
Quoted 11 lines
Quoted 11 lines
 arch/x86/include/uapi/asm/kvm_para.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/include/uapi/asm/kvm_para.h +0 -0
index 6e64b27b2c1e..115bb34413cf 100644
@@ -37,7 +37,8 @@
 #define KVM_FEATURE_HC_MAP_GPA_RANGE	16
 #define KVM_FEATURE_MIGRATION_CONTROL	17
 
Quoted 5 lines

arch/x86/include/uapi/asm/kvm_para.h line 41

Quoted 3 lines
arch/x86/include/uapi/asm/kvm_para.h +2 -1
-#define KVM_HINTS_REALTIME      0
+#define KVM_HINTS_REALTIME                      0
+#define KVM_HINTS_PHYS_ADDRESS_SIZE_DATA_VALID  1

Why does KVM need to get involved? This is purely a userspace problem.

It doesn't. I only need reserve a hints bit, and the canonical source for that happens to live in the kernel. That's why this patch doesn't touch any actual code ;)

Quoted 7 lines
Quoted 2 lines

E.g. why not use QEMU's fw_cfg to communicate this information to the guest?

That is indeed the other obvious way to implement this. Given this information will be needed in code paths which already do CPUID queries using CPUID to transport that information looked like the better option to me.

While this certainly looks like an overkill here, we could probably add new, VMM-spefific CPUID leaves to KVM, e.g.

0x4000000A: VMM signature
0x4000000B: VMM features
0x4000000C: VMM quirks
...

this way VMMs (like QEMU) could identify themselves and suggest VMM specific things to guests without KVM's involvement. Just if 'fw_cfg' is not enough)

Signature · 1 line
Vitaly
#4 lore raw
Quoted 39 lines · Vitaly Kuznetsov
Quoted 2 lines · Gerd Hoffmann
Quoted 1 line · Sean Christopherson

On Thu, Sep 08, 2022, Gerd Hoffmann wrote:

...

Quoted 21 lines
Quoted 11 lines
Quoted 11 lines
 arch/x86/include/uapi/asm/kvm_para.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/include/uapi/asm/kvm_para.h +0 -0
index 6e64b27b2c1e..115bb34413cf 100644
@@ -37,7 +37,8 @@
 #define KVM_FEATURE_HC_MAP_GPA_RANGE	16
 #define KVM_FEATURE_MIGRATION_CONTROL	17
 
Quoted 5 lines

arch/x86/include/uapi/asm/kvm_para.h line 41

Quoted 3 lines
arch/x86/include/uapi/asm/kvm_para.h +2 -1
-#define KVM_HINTS_REALTIME      0
+#define KVM_HINTS_REALTIME                      0
+#define KVM_HINTS_PHYS_ADDRESS_SIZE_DATA_VALID  1

Why does KVM need to get involved? This is purely a userspace problem.

It doesn't. I only need reserve a hints bit, and the canonical source for that happens to live in the kernel. That's why this patch doesn't touch any actual code ;)

Quoted 7 lines
Quoted 2 lines

E.g. why not use QEMU's fw_cfg to communicate this information to the guest?

That is indeed the other obvious way to implement this. Given this information will be needed in code paths which already do CPUID queries using CPUID to transport that information looked like the better option to me.

While this certainly looks like an overkill here, we could probably add new, VMM-spefific CPUID leaves to KVM, e.g.

Quoted 8 lines

0x4000000A: VMM signature
0x4000000B: VMM features
0x4000000C: VMM quirks
...

this way VMMs (like QEMU) could identify themselves and suggest VMM specific things to guests without KVM's involvement. Just if 'fw_cfg' is not enough)

I don't think KVM needs to get involved in that either. The de facto hypervisor CPUID standard already allows for multiple hypervisors/VMMs to announce themselves to the guest, e.g. QEMU could add itself as another VMM using 0x40000100 (shifted as necessary to accomodate KVM+Hyper-V).

#5 lore raw
Quoted 1 line · Sean Christopherson

On Fri, Sep 09, 2022, Vitaly Kuznetsov wrote:

...

Quoted 8 lines
Quoted 8 lines

While this certainly looks like an overkill here, we could probably add new, VMM-spefific CPUID leaves to KVM, e.g.

0x4000000A: VMM signature
0x4000000B: VMM features
0x4000000C: VMM quirks
...

Quoted 8 lines
Quoted 3 lines

this way VMMs (like QEMU) could identify themselves and suggest VMM specific things to guests without KVM's involvement. Just if 'fw_cfg' is not enough)

I don't think KVM needs to get involved in that either. The de facto hypervisor CPUID standard already allows for multiple hypervisors/VMMs to announce themselves to the guest, e.g. QEMU could add itself as another VMM using 0x40000100 (shifted as necessary to accomodate KVM+Hyper-V).

True, VMM can just use another hypervisor space (+0x100) but we can view it from a slightly different angle: KVM itself is insufficient to run VMs, there's always a VMM in the background, we may want to provide a "standard" for its information meaning guests won't need to search for VMMs signature[s] but can directly refer to "standardized" leaves.

(All this is a purely theoretical discussion at this point, we need a good reason to introduce this first).

Signature · 1 line
Vitaly
#6 lore raw
In reply to Gerd Hoffmann
Quoted 12 lines · Gerd Hoffmann
Quoted 6 lines · Sean Christopherson

arch/x86/include/uapi/asm/kvm_para.h line 41

Quoted 3 lines · Gerd Hoffmann
arch/x86/include/uapi/asm/kvm_para.h +2 -1
-#define KVM_HINTS_REALTIME      0
+#define KVM_HINTS_REALTIME                      0
+#define KVM_HINTS_PHYS_ADDRESS_SIZE_DATA_VALID  1

Why does KVM need to get involved? This is purely a userspace problem.

It doesn't. I only need reserve a hints bit, and the canonical source for that happens to live in the kernel. That's why this patch doesn't touch any actual code ;)

Quoted 7 lines
Quoted 2 lines

E.g. why not use QEMU's fw_cfg to communicate this information to the guest?

That is indeed the other obvious way to implement this. Given this information will be needed in code paths which already do CPUID queries using CPUID to transport that information looked like the better option to me.

I'd like to move forward with this.

So, any comment further comments and opinions? Is it ok to grab a hints bit given the explanation above? Or should I go for the fw_cfg approach?

thanks & take care,
Gerd

#7 lore raw
Quoted 9 lines · Gerd Hoffmann
Quoted 8 lines · Gerd Hoffmann
Quoted 6 lines · Sean Christopherson

arch/x86/include/uapi/asm/kvm_para.h line 41

Quoted 3 lines · Gerd Hoffmann
arch/x86/include/uapi/asm/kvm_para.h +2 -1
-#define KVM_HINTS_REALTIME      0
+#define KVM_HINTS_REALTIME                      0
+#define KVM_HINTS_PHYS_ADDRESS_SIZE_DATA_VALID  1

Why does KVM need to get involved? This is purely a userspace problem.

Quoted 3 lines
Quoted 3 lines

It doesn't. I only need reserve a hints bit, and the canonical source for that happens to live in the kernel. That's why this patch doesn't touch any actual code ;)

The issue is that this "hint" effectively breaks other VMMs that already provide an accurate guest.MAXPHYADDR.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

E.g. why not use QEMU's fw_cfg to communicate this information to the guest?

That is indeed the other obvious way to implement this. Given this information will be needed in code paths which already do CPUID queries using CPUID to transport that information looked like the better option to me.

Quoted 5 lines

I'd like to move forward with this.

So, any comment further comments and opinions? Is it ok to grab a hints bit given the explanation above? Or should I go for the fw_cfg approach?

My strong preference is the fw_cfg approach, or if the guest side really wants to use CPUID, have QEMU define it's own CPUID signature and provide QEMU-specific hints/quirks that way.

#8 lore raw
Quoted 11 lines · Sean Christopherson
Quoted 9 lines · Gerd Hoffmann
Quoted 8 lines · Gerd Hoffmann
Quoted 6 lines · Sean Christopherson

arch/x86/include/uapi/asm/kvm_para.h line 41

Quoted 3 lines · Gerd Hoffmann
arch/x86/include/uapi/asm/kvm_para.h +2 -1
-#define KVM_HINTS_REALTIME      0
+#define KVM_HINTS_REALTIME                      0
+#define KVM_HINTS_PHYS_ADDRESS_SIZE_DATA_VALID  1

Why does KVM need to get involved? This is purely a userspace problem.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

It doesn't. I only need reserve a hints bit, and the canonical source for that happens to live in the kernel. That's why this patch doesn't touch any actual code ;)

The issue is that this "hint" effectively breaks other VMMs that already provide an accurate guest.MAXPHYADDR.

Any VMM that doesn't provide an accurate guest.MAXPHYADDR is broken. Why do we need a "hint" that the virtual processor works?

Quoted 17 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

E.g. why not use QEMU's fw_cfg to communicate this information to the guest?

That is indeed the other obvious way to implement this. Given this information will be needed in code paths which already do CPUID queries using CPUID to transport that information looked like the better option to me.

Quoted 5 lines

I'd like to move forward with this.

So, any comment further comments and opinions? Is it ok to grab a hints bit given the explanation above? Or should I go for the fw_cfg approach?

My strong preference is the fw_cfg approach, or if the guest side really wants to use CPUID, have QEMU define it's own CPUID signature and provide QEMU-specific hints/quirks that way.

#9 lore raw
Quoted 12 lines · Jim Mattson
Quoted 11 lines · Sean Christopherson
Quoted 9 lines · Gerd Hoffmann
Quoted 8 lines · Gerd Hoffmann
Quoted 6 lines · Sean Christopherson

arch/x86/include/uapi/asm/kvm_para.h line 41

Quoted 3 lines · Gerd Hoffmann
arch/x86/include/uapi/asm/kvm_para.h +2 -1
-#define KVM_HINTS_REALTIME      0
+#define KVM_HINTS_REALTIME                      0
+#define KVM_HINTS_PHYS_ADDRESS_SIZE_DATA_VALID  1

Why does KVM need to get involved? This is purely a userspace problem.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

It doesn't. I only need reserve a hints bit, and the canonical source for that happens to live in the kernel. That's why this patch doesn't touch any actual code ;)

The issue is that this "hint" effectively breaks other VMMs that already provide an accurate guest.MAXPHYADDR.

Any VMM that doesn't provide an accurate guest.MAXPHYADDR is broken.

I stand for it as well.

To me, it looks an old QEMU bug and firmware provided a workaround for the bug that doesn't trust guest's CPUID.0x80000008. IMHO, (guest) firmware should always trust CPUID and error out if MAXPHYADDR reported from CPUID is broken to force VMM fixing itself to provide correct CPUID info. But letting firmware drop the workaround surely breaks backwards compatibility.

Quoted 1 line

Why do we need a "hint" that the virtual processor works?

maintainer
#10 lore raw
In reply to Jim Mattson
Quoted 5 lines · Jim Mattson
Quoted 2 lines

The issue is that this "hint" effectively breaks other VMMs that already provide an accurate guest.MAXPHYADDR.

Any VMM that doesn't provide an accurate guest.MAXPHYADDR is broken. Why do we need a "hint" that the virtual processor works?

I agree. Since old (and current) versions of QEMU wouldn't get the bit anyway, just fix the next one. I'll follow up on the QEMU mailing list.

Paolo