Skip to contentSkip to conversation

TDX module configurability of 0x80000008

RFC Edgecombe, Rick P 4 participants 18 messages Apr 24, 2024 – May 8, 2024 KVM

lore
Share & tools
lore
#0 lore raw

Hi,

This is a new effort to solicit community feedback for potential future TDX module features. There are two features in different stages of development around the configurability of the max physical address exposed in 0x80000008.EAX. I was hoping to get some comments on them and share the current plans on whether to implement them in KVM.

One of the TDX module features is called MAXPA_VIRT. In short, it is similar to KVM’s allow_smaller_maxphyaddr. It requires an explicit opt-in by the VMM, and allows a TD’s 0x80000008.EAX[7:0] to be configured by the VMM. Accesses to physical addresses above the specified value by the TD will cause the TDX module to inject a mostly correct #PF with the RSVD error code set. It has to deal with the same problems as allow_smaller_maxphyaddr for correctly setting the RSVD bit. I wasn’t thinking to push this feature for KVM due the movement away from allow_smaller_maxphyaddr and towards 0x80000008.EAX[23:16].

There is also a potential future TDX module feature currently being evaluated around the configurability of 0x80000008.EAX[23:16]. I wanted to get some community comments on the feature while it is still in the early stages of development.

0x80000008[7:0] is defined by the SDM as MAXPHYADDR. KVM is designed to work with guest MAXPHYADDR set to host MAXPHYADDR. In the future there is work for KVM to also accommodate a potentially smaller value in 0x80000008.EAX[23:16] for normal VMs. This value is defined by AMD spec as GuestPhysAddrSize:
Maximum guest physical address size in bits. This number applies only to guests using nested paging. When this field is zero, refer to the PhysAddrSize field for the maximum guest physical address size.

The idea is that TDX module could add the capability to configure these bits as well, so that TDs could match normal VMs for cases where there is a desire for the guests MAXPA to be smaller than the hosts. The requirements would be, roughly:
- The VMM specifies the 0x80000008.EAX[23:16] when creating a TD.
- The TDX module does sanity checking.
- The 0x80000008.EAX[23:16] field is used to communicate the max addressable GPA to  the guest. It will be used by the guest firmware to make sure resources like PCI bars are mapped into the addressable GPA.
- If the guest attempts to access memory beyond the max addressable GPA, then the TDX module generates EPT violation to the VMM. For the VMM, this case means that the guest attempted to access "invalid" (I/O) memory.
- The VMM will be expected to terminate the TD guest. The VMM may send a notification, but the TDX module doesn't necessarily need to know how.

Glad to hear any comments. Thanks.

Rick

#1 lore raw
Quoted 30 lines · Rick P

Hi,

This is a new effort to solicit community feedback for potential future TDX module features. There are two features in different stages of development around the configurability of the max physical address exposed in 0x80000008.EAX. I was hoping to get some comments on them and share the current plans on whether to implement them in KVM.

One of the TDX module features is called MAXPA_VIRT. In short, it is similar to KVM’s allow_smaller_maxphyaddr. It requires an explicit opt-in by the VMM, and allows a TD’s 0x80000008.EAX[7:0] to be configured by the VMM. Accesses to physical addresses above the specified value by the TD will cause the TDX module to inject a mostly correct #PF with the RSVD error code set. It has to deal with the same problems as allow_smaller_maxphyaddr for correctly setting the RSVD bit. I wasn’t thinking to push this feature for KVM due the movement away from allow_smaller_maxphyaddr and towards 0x80000008.EAX[23:16].

There is also a potential future TDX module feature currently being evaluated around the configurability of 0x80000008.EAX[23:16]. I wanted to get some community comments on the feature while it is still in the early stages of development.

0x80000008[7:0] is defined by the SDM as MAXPHYADDR. KVM is designed to work with guest MAXPHYADDR set to host MAXPHYADDR. In the future there is work for KVM to also accommodate a potentially smaller value in 0x80000008.EAX[23:16] for normal VMs. This value is defined by AMD spec as GuestPhysAddrSize:
Maximum guest physical address size in bits. This number applies only to guests using nested paging. When this field is zero, refer to the PhysAddrSize field for the maximum guest physical address size.

Quoted 14 lines

The idea is that TDX module could add the capability to configure these bits as well, so that TDs could match normal VMs for cases where there is a desire for the guests MAXPA to be smaller than the hosts. The requirements would be, roughly:
- The VMM specifies the 0x80000008.EAX[23:16] when creating a TD.
- The TDX module does sanity checking.
- The 0x80000008.EAX[23:16] field is used to communicate the max addressable GPA to  the guest. It will be used by the guest firmware to make sure resources like PCI bars are mapped into the addressable GPA.
- If the guest attempts to access memory beyond the max addressable GPA, then the TDX module generates EPT violation to the VMM. For the VMM, this case means that the guest attempted to access "invalid" (I/O) memory.
- The VMM will be expected to terminate the TD guest. The VMM may send a notification, but the TDX module doesn't necessarily need to know how.

This is not the same as how it works for normal (non-TDX) VMs.

For normal VMs, when userspace configures a smaller one than what hardware EPT/NPT supports, it doesn't cause any issue if guest accesses GPA beyond [23:16] but within hardware EPT/NTP capability.

It's more a hint to guest that KVM doesn't enforce the semantics of it. However, for TDX case, you are proposing to make it a hard rule.

Quoted 3 lines

Glad to hear any comments. Thanks.

Rick

#2 lore raw
Quoted 19 lines · Xiaoyao Li
Quoted 10 lines

The idea is that TDX module could add the capability to configure these bits as
well, so that TDs could match normal VMs for cases where there is a desire for
the guests MAXPA to be smaller than the hosts. The requirements would be, roughly:
   - The VMM specifies the 0x80000008.EAX[23:16] when creating a TD.
   - The TDX module does sanity checking.
   - The 0x80000008.EAX[23:16] field is used to communicate the max addressable

Quoted 8 lines

   GPA to  the guest. It will be used by the guest firmware to make sure resources like PCI bars are mapped into the addressable GPA.
   - If the guest attempts to access memory beyond the max addressable GPA, then
   the TDX module generates EPT violation to the VMM. For the VMM, this case means that the guest attempted to access "invalid" (I/O) memory.
   - The VMM will be expected to terminate the TD guest. The VMM may send a notification, but the TDX module doesn't necessarily need to know how.

Quoted 8 lines

This is not the same as how it works for normal (non-TDX) VMs.

For normal VMs, when userspace configures a smaller one than what hardware EPT/NPT supports, it doesn't cause any issue if guest accesses GPA beyond [23:16] but within hardware EPT/NTP capability.

It's more a hint to guest that KVM doesn't enforce the semantics of it. However, for TDX case, you are proposing to make it a hard rule.

If we limit ourselves to worrying about valid configurations, accessing a GPA beyond [23:16] is similar to accessing a GPA with no memslot. Like you say, [23:16] is a hint, so there is really no change from KVM's perspective. It behaves like normal based on the [7:0] MAXPA.

What do you think should happen in the case a TD accesses a GPA with no memslot? KVM/QEMU don't have a lot of options to recover. So are the differences here just the existing differences between normal VMs and TDX?

#3 lore raw
Quoted 21 lines · Rick P Edgecombe
Quoted 20 lines · Xiaoyao Li
Quoted 9 lines

The idea is that TDX module could add the capability to configure these bits as well, so that TDs could match normal VMs for cases where there is a desire for the guests MAXPA to be smaller than the hosts. The requirements would be,
roughly:
   - The VMM specifies the 0x80000008.EAX[23:16] when creating a TD.
   - The TDX module does sanity checking.
   - The 0x80000008.EAX[23:16] field is used to communicate the max addressable

Quoted 8 lines

   GPA to  the guest. It will be used by the guest firmware to make sure resources like PCI bars are mapped into the addressable GPA.
   - If the guest attempts to access memory beyond the max addressable GPA, then
   the TDX module generates EPT violation to the VMM. For the VMM, this case means that the guest attempted to access "invalid" (I/O) memory.
   - The VMM will be expected to terminate the TD guest. The VMM may send a notification, but the TDX module doesn't necessarily need to know how.

This is not the same as how it works for normal (non-TDX) VMs.

Quoted 8 lines
Quoted 6 lines

For normal VMs, when userspace configures a smaller one than what hardware EPT/NPT supports, it doesn't cause any issue if guest accesses GPA beyond [23:16] but within hardware EPT/NTP capability.

It's more a hint to guest that KVM doesn't enforce the semantics of it. However, for TDX case, you are proposing to make it a hard rule.

If we limit ourselves to worrying about valid configurations,

Define "valid configurations".

Quoted 1 line

accessing a GPA beyond [23:16] is similar to accessing a GPA with no memslot.

No, it's not. A GPA without a memslot has *very* well-defined semantics in KVM, and KVM can provide those semantics for all guest-legal GPAs regardless of hardware EPT/NPT support.

Quoted 4 lines

Like you say, [23:16] is a hint, so there is really no change from KVM's perspective. It behaves like normal based on the [7:0] MAXPA.

What do you think should happen in the case a TD accesses a GPA with no memslot?

Synthesize a #VE into the guest. The GPA isn't a violation of the "real" MAXPHYADDR, so killing the guest isn't warranted. And that also means the VMM could legitimately want to put emulated MMIO above the max addressable GPA. Synthesizing a #VE is also aligned with KVM's non-memslot behavior for TDX (configured to trigger #VE).

And most importantly, as you note above, the VMM *can't* resolve the problem. On the other hand, the guest *might* be able to resolve the issue, e.g. it could request MMIO, which may or may not succeed. Even if the guest panics, that's far better than it being terminated by the host as it gives the guest a chance to capture what led to the panic/crash.

The only downside is that the VMM doesn't have a chance to "bless" the #VE, but since the VMM literally cannot handle the "bad" access in any other than killing the guest, I don't see that as a major problem.

Quoted 2 lines

KVM/QEMU don't have a lot of options to recover. So are the differences here just the existing differences between normal VMs and TDX?

#4 lore raw
Quoted 3 lines · Sean Christopherson
Quoted 1 line

If we limit ourselves to worrying about valid configurations,

Define "valid configurations".

I meant configurations with no memslots above guest max pa. If there are memslots in that region, I don't know. Maybe valid is the wrong word.

Quoted 8 lines
Quoted 3 lines

accessing a GPA beyond [23:16] is similar to accessing a GPA with no memslot.

No, it's not.  A GPA without a memslot has *very* well-defined semantics in KVM,
and KVM can provide those semantics for all guest-legal GPAs regardless of hardware EPT/NPT support.

Sorry, not following. Are we expecting there to be memslots above the guest maxpa 23:16? If there are no memslots in that region, it seems exactly like accessing a GPA with no memslots. What is the difference between before and after the introduction of guest MAXPA? (there will be normal VMs and TDX differences of course).

Quoted 23 lines
Quoted 6 lines

Like you say, [23:16] is a hint, so there is really no change from KVM's perspective. It behaves like normal based on the [7:0] MAXPA.

What do you think should happen in the case a TD accesses a GPA with no memslot?


Synthesize a #VE into the guest.  The GPA isn't a violation of the "real" MAXPHYADDR,
so killing the guest isn't warranted.  And that also means the VMM could legitimately
want to put emulated MMIO above the max addressable GPA.  Synthesizing a #VE is
also aligned with KVM's non-memslot behavior for TDX (configured to trigger #VE).

And most importantly, as you note above, the VMM *can't* resolve the problem. On
the other hand, the guest *might* be able to resolve the issue, e.g. it could request MMIO, which may or may not succeed.  Even if the guest panics, that's far better than it being terminated by the host as it gives the guest a chance to capture what led to the panic/crash.

Quoted 5 lines

The only downside is that the VMM doesn't have a chance to "bless" the #VE, but
since the VMM literally cannot handle the "bad" access in any other than killing
the guest, I don't see that as a major problem.

Ok, so we want the TDX module to expect the TD to continue to live. Then we need to handle two things:
1. Trigger #VE for a GPA that is mappable by the EPT level (we can already do this)
2. Trigger #VE for a GPA that is not mappable by the EPT level

We could ask the TDX module to just handle both of these cases. But this means KVM loses a bit of control and debug-ability from the host side. Also, it adds complexity for cases where KVM maps GPAs above guest maxpa anyway. So maybe we want it to just handle 2? It might have some nuances still.

Another question, should we just tie guest maxpa to GPAW? Either enforce they are the same, or expose 23:16 based on GPAW.

#5 lore raw
Quoted 8 lines · Rick P Edgecombe
Quoted 6 lines · Sean Christopherson
Quoted 2 lines

accessing a GPA beyond [23:16] is similar to accessing a GPA with no memslot.

No, it's not.  A GPA without a memslot has *very* well-defined semantics in KVM, and KVM can provide those semantics for all guest-legal GPAs regardless of hardware EPT/NPT support.

Quoted 5 lines

Sorry, not following. Are we expecting there to be memslots above the guest maxpa 23:16? If there are no memslots in that region, it seems exactly like accessing a GPA with no memslots. What is the difference between before and after the introduction of guest MAXPA? (there will be normal VMs and TDX differences of course).

If there are no memslots, nothing from a functional perspectives, just a very slight increase in latency. Pre-TDX, KVM can always emulate in reponse to an EPT violation on an unmappable GPA. I.e. as long as there is no memslot, KVM doesn't *need* to create SPTEs, and so whether or not a GPA is mappable is completely irrelevant.

Enter TDX, and suddenly that doesn't work because KVM can't emulate without guest cooperation. And to get guest cooperation, _something_ needs to kick the guest with a #VE.

Quoted 22 lines
Quoted 18 lines
Quoted 5 lines

Like you say, [23:16] is a hint, so there is really no change from KVM's perspective. It behaves like normal based on the [7:0] MAXPA.

What do you think should happen in the case a TD accesses a GPA with no memslot?


Synthesize a #VE into the guest.  The GPA isn't a violation of the "real" MAXPHYADDR, so killing the guest isn't warranted.  And that also means the VMM could legitimately want to put emulated MMIO above the max addressable GPA.  Synthesizing a #VE is also aligned with KVM's non-memslot behavior for TDX (configured to trigger #VE).

And most importantly, as you note above, the VMM *can't* resolve the problem.  On the other hand, the guest *might* be able to resolve the issue, e.g. it could request MMIO, which may or may not succeed.  Even if the guest panics, that's far better than it being terminated by the host as it gives the guest a chance to capture what led to the panic/crash.

Quoted 3 lines

The only downside is that the VMM doesn't have a chance to "bless" the #VE, but since the VMM literally cannot handle the "bad" access in any other than killing the guest, I don't see that as a major problem.

Quoted 8 lines

Ok, so we want the TDX module to expect the TD to continue to live. Then we need to handle two things:
1. Trigger #VE for a GPA that is mappable by the EPT level (we can already do this)
2. Trigger #VE for a GPA that is not mappable by the EPT level

We could ask the TDX module to just handle both of these cases. But this means KVM loses a bit of control and debug-ability from the host side.

Why would the TDX module touch #1? Just leave it as is.

Quoted 2 lines

Also, it adds complexity for cases where KVM maps GPAs above guest maxpa anyway.

That should be disallowed. If KVM tries to map an address that it told the guest was impossible to map, then the TDX module should throw an error.

Quoted 1 line

So maybe we want it to just handle 2? It might have some nuances still.

I'm sure there are nuances, but I don't know that we care. I see three options:

1. Resume the guest without doing anything and hang the guest.

2. Punt the issue to the VMM and kill the guest.

3. Inject #VE into the guest and maybe the guest lives.

#1 is terrible for obvious reasons, so given the choice between guaranteed death and a slim chance of survival, I'll take that slim chance of survival :-)

Quoted 1 line

Another question, should we just tie guest maxpa to GPAW?

Yes

Quoted 1 line

Either enforce they are the same, or expose 23:16 based on GPAW.

I can't think of any reason not to derive 23:16 from GPAW, unless I'm missing some subtlety, they're quite literally the same thing.

#6 lore raw
Quoted 16 lines · Sean Christopherson
Quoted 9 lines · Rick P Edgecombe
Quoted 7 lines · Sean Christopherson
Quoted 2 lines

accessing a GPA beyond [23:16] is similar to accessing a GPA with no memslot.

No, it's not.  A GPA without a memslot has *very* well-defined semantics in
KVM, and KVM can provide those semantics for all guest-legal GPAs regardless of hardware EPT/NPT support.

Quoted 5 lines

Sorry, not following. Are we expecting there to be memslots above the guest maxpa 23:16? If there are no memslots in that region, it seems exactly like accessing a GPA with no memslots. What is the difference between before and after the introduction of guest MAXPA? (there will be normal VMs and TDX differences of course).

Quoted 7 lines

If there are no memslots, nothing from a functional perspectives, just a very slight increase in latency.  Pre-TDX, KVM can always emulate in reponse to an EPT
violation on an unmappable GPA.  I.e. as long as there is no memslot, KVM doesn't
*need* to create SPTEs, and so whether or not a GPA is mappable is completely irrelevant.

Right, although there are gaps in emulation that could fail. If the emulation succeeds and there is an MMIO exit targeting a totally unknown GPA, then I guess it's up to userspace to decide what to do.

KVM's done its job. But userspace still has to handle it. It can, but I was under the impression it didn't (maybe bad assumption).

Quoted 39 lines

Enter TDX, and suddenly that doesn't work because KVM can't emulate without guest
cooperation.  And to get guest cooperation, _something_ needs to kick the guest
with a #VE.

Quoted 32 lines
Quoted 19 lines
Quoted 5 lines

Like you say, [23:16] is a hint, so there is really no change from KVM's perspective. It behaves like normal based on the [7:0] MAXPA.

What do you think should happen in the case a TD accesses a GPA with no memslot?


Synthesize a #VE into the guest.  The GPA isn't a violation of the "real" MAXPHYADDR, so killing the guest isn't warranted.  And that also means the VMM could legitimately want to put emulated MMIO above the max addressable GPA.  Synthesizing a #VE is also aligned with KVM's non-memslot behavior for TDX (configured to trigger #VE).

And most importantly, as you note above, the VMM *can't* resolve the problem.  On the other hand, the guest *might* be able to resolve the issue, e.g. it could request MMIO, which may or may not succeed.  Even if the guest panics, that's far better than it being terminated by the host as
it gives the guest a chance to capture what led to the panic/crash.

Quoted 4 lines

The only downside is that the VMM doesn't have a chance to "bless" the #VE,
but since the VMM literally cannot handle the "bad" access in any other than killing the guest, I don't see that as a major problem.

Ok, so we want the TDX module to expect the TD to continue to live. Then we need
to handle two things:
1. Trigger #VE for a GPA that is mappable by the EPT level (we can already do
this)
2. Trigger #VE for a GPA that is not mappable by the EPT level

Quoted 5 lines
Quoted 3 lines

We could ask the TDX module to just handle both of these cases. But this means
KVM loses a bit of control and debug-ability from the host side.

Why would the TDX module touch #1?  Just leave it as is.

I think it won't even come up if GPAW is locked to 23:16 like discussed below. (and the current plan for picking EPT level).

Quoted 7 lines
Quoted 3 lines

Also, it adds complexity for cases where KVM maps GPAs above guest maxpa anyway.

That should be disallowed.  If KVM tries to map an address that it told the guest
was impossible to map, then the TDX module should throw an error.

Hmm. I'll mention this, but I don't see why KVM needs the TDX module to filter it. It seems in the range of userspace being allowed to create nonsense configurations that only hurt its own guest.

If we think the TDX module should do it, then maybe we should have KVM sanity filter these out today in preparation.

Quoted 8 lines
Quoted 2 lines

So maybe we want it to just handle 2? It might have some nuances still.

I'm sure there are nuances, but I don't know that we care.  I see three options:

 1. Resume the guest without doing anything and hang the guest.

Quoted 7 lines

 2. Punt the issue to the VMM and kill the guest.

 3. Inject #VE into the guest and maybe the guest lives.

#1 is terrible for obvious reasons, so given the choice between guaranteed death
and a slim chance of survival, I'll take that slim chance of survival :-)

Yes, maybe this proposal was being a bit lazy.

Quoted 9 lines
Quoted 2 lines

Another question, should we just tie guest maxpa to GPAW?

Yes

Quoted 1 line

Either enforce they are the same, or expose 23:16 based on GPAW.

I can't think of any reason not to derive 23:16 from GPAW, unless I'm missing some subtlety, they're quite literally the same thing.

So we have:
- Expose GPAW in 23:16
- Inject #VE if ept violation is for gpa that can't be mapped by EPT level

Seems relatively simple. I'll wait a bit for more comments, and circle back with TDX module folks.

#7 lore raw
Quoted 22 lines · Rick P Edgecombe
Quoted 15 lines · Sean Christopherson
Quoted 8 lines · Rick P Edgecombe
Quoted 6 lines · Sean Christopherson
Quoted 2 lines

accessing a GPA beyond [23:16] is similar to accessing a GPA with no memslot.

No, it's not.  A GPA without a memslot has *very* well-defined semantics in KVM, and KVM can provide those semantics for all guest-legal GPAs regardless of hardware EPT/NPT support.

Quoted 5 lines

Sorry, not following. Are we expecting there to be memslots above the guest maxpa 23:16? If there are no memslots in that region, it seems exactly like accessing a GPA with no memslots. What is the difference between before and after the introduction of guest MAXPA? (there will be normal VMs and TDX differences of course).

Quoted 5 lines

If there are no memslots, nothing from a functional perspectives, just a very slight increase in latency.  Pre-TDX, KVM can always emulate in reponse to an EPT violation on an unmappable GPA.  I.e. as long as there is no memslot, KVM doesn't *need* to create SPTEs, and so whether or not a GPA is mappable is completely irrelevant.

Quoted 5 lines

Right, although there are gaps in emulation that could fail. If the emulation succeeds and there is an MMIO exit targeting a totally unknown GPA, then I guess it's up to userspace to decide what to do.

KVM's done its job.

Yep.

Quoted 2 lines

But userspace still has to handle it. It can, but I was under the impression it didn't (maybe bad assumption).

I'm pretty sure QEMU handles accesses to non-existent MMIO with PCI abort semantics, i.e. ignores writes and returns all FFs for reads.

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

Also, it adds complexity for cases where KVM maps GPAs above guest maxpa anyway.

That should be disallowed.  If KVM tries to map an address that it told the guest was impossible to map, then the TDX module should throw an error.

Hmm. I'll mention this, but I don't see why KVM needs the TDX module to filter it. It seems in the range of userspace being allowed to create nonsense configurations that only hurt its own guest.

Because the whole point of TDX is to protect the guest from the bad, naughty host?

Quoted 2 lines

If we think the TDX module should do it, then maybe we should have KVM sanity filter these out today in preparation.

Nope. KVM isn't in the guest's TCB, TDX is. KVM's stance is that userspace is responsible for providing a sane vCPU model, because defining what is "sane" is extremely difficult unless the definition is super prescriptive, a la TDX.

E.g. letting the host map something that TDX's spec says will cause #VE would create a novel attack surface.

#8 lore raw
Quoted 7 lines · Sean Christopherson
Quoted 4 lines

Hmm. I'll mention this, but I don't see why KVM needs the TDX module to filter
it. It seems in the range of userspace being allowed to create nonsense configurations that only hurt its own guest.

Because the whole point of TDX is to protect the guest from the bad, naughty host?

DOS naughtiness by the host is allowed though.

Quoted 5 lines
Quoted 4 lines

If we think the TDX module should do it, then maybe we should have KVM sanity
filter these out today in preparation.

Quoted 8 lines

Nope.  KVM isn't in the guest's TCB, TDX is.
  KVM's stance is that userspace is
responsible for providing a sane vCPU model, because defining what is "sane" is
extremely difficult unless the definition is super prescriptive, a la TDX.

E.g. letting the host map something that TDX's spec says will cause #VE would create a novel attack surface.

I thought that the shared half could be mapped in that range unless KVM gets involved. But, no, as long as we tie GPAW, 23:16, ept-level all together, then mapping something above it won't even make sense.

I don't see attack surface risk immediately. I expect this will get more internal scrutiny in that regard though.

#9 lore raw
Quoted 23 lines · Rick P Edgecombe
Quoted 7 lines · Sean Christopherson
Quoted 4 lines

Hmm. I'll mention this, but I don't see why KVM needs the TDX module to filter
it. It seems in the range of userspace being allowed to create nonsense configurations that only hurt its own guest.

Because the whole point of TDX is to protect the guest from the bad, naughty host?

DOS naughtiness by the host is allowed though.

Quoted 11 lines
Quoted 3 lines

If we think the TDX module should do it, then maybe we should have KVM sanity filter these out today in preparation.

Nope.  KVM isn't in the guest's TCB, TDX is.   KVM's stance is that userspace is responsible for providing a sane vCPU model, because defining what is "sane" is extremely difficult unless the definition is super prescriptive, a la TDX.

E.g. letting the host map something that TDX's spec says will cause #VE would create a novel attack surface.

Quoted 6 lines

I thought that the shared half could be mapped in that range unless KVM gets involved. But, no, as long as we tie GPAW, 23:16, ept-level all together, then mapping something above it won't even make sense.

I don't see attack surface risk immediately. I expect this will get more internal scrutiny in that regard though.

Oooh, I thought you were talking about KVM mapping a private GPA address in S-EPT above the reported GPAW. In hindsight, I don't know _why_ I thought that.

Yeah, trying to sanity check the shared EPT in the TDX module would be comically pointless.

#10 lore raw
Quoted 20 lines · Sean Christopherson
Quoted 12 lines
Quoted 11 lines
Quoted 2 lines

If we think the TDX module should do it, then maybe we should have KVM sanity filter these out today in preparation.

Nope.  KVM isn't in the guest's TCB, TDX is.    KVM's stance is that userspace is responsible for providing a sane vCPU model, because defining what is "sane" is extremely difficult unless the definition is super prescriptive, a la TDX.

E.g. letting the host map something that TDX's spec says will cause #VE would
create a novel attack surface.

Quoted 7 lines

I thought that the shared half could be mapped in that range unless KVM gets involved. But, no, as long as we tie GPAW, 23:16, ept-level all together, then
mapping something above it won't even make sense.

I don't see attack surface risk immediately. I expect this will get more internal scrutiny in that regard though.

Quoted 7 lines

Oooh, I thought you were talking about KVM mapping a private GPA address in S- EPT
above the reported GPAW.  In hindsight, I don't know _why_ I thought that.

Yeah, trying to sanity check the shared EPT in the TDX module would be comically
pointless.

I might have been thinking that as well? Wasn't the fullest thought. Sorry for the confusion.

In any case it should be moot for the solution we are going for in KVM. I'll mention it to them though, because just because KVM will not do GPA_48 and 5- level EPT, doesn't mean another VMM wont.

#11 lore raw

Follow up on this:

1. The plan is to just always inject the #VEs for private and shared GPAs that exceed GPAW. (i.e. not pass the subset of EPT violations that could be handled by the VMM by clearing suppress #VE)

2. There was some concern that exposing non-zero bits in [23:16] could confuse existing TDs. Of course KVM doesn't support any TDs today, but if this feature comes after initial KVM support for TDX and KVM wants to set it by default, then it could be an issue.

For normal VMs, is there any concern that guests might not be masking the bits correctly?

TDX module folks were pushing for a guest opt-in out of concern some breakages could result. Of course it requires additional enabling in the guest OS and vBIOS then. I was thinking it should be a host opt-in without guest control. If there was a problem it could be a host userspace opt-in. Any concerns there?

Thanks,

Rick

#12 lore raw
Quoted 11 lines · Rick P

Follow up on this:

1. The plan is to just always inject the #VEs for private and shared GPAs that exceed GPAW. (i.e. not pass the subset of EPT violations that could be handled by the VMM by clearing suppress #VE)

2. There was some concern that exposing non-zero bits in [23:16] could confuse existing TDs. Of course KVM doesn't support any TDs today, but if this feature comes after initial KVM support for TDX and KVM wants to set it by default, then it could be an issue.

Do you mean some TDs may assert that [23:16] are 0s? A future-proof design won't have this assertion. And this case (i.e., some CPUID bits become non-zero) happens on every new generation of CPUs and doesn't confuse existing OSes. I don't understand why it would be a problem for TDs.

Quoted 12 lines

For normal VMs, is there any concern that guests might not be masking the bits correctly?

TDX module folks were pushing for a guest opt-in out of concern some breakages could result. Of course it requires additional enabling in the guest OS and vBIOS then. I was thinking it should be a host opt-in without guest control. If there was a problem it could be a host userspace opt-in. Any concerns there?

Thanks,

Rick

#13 lore raw
Quoted 8 lines · Chao Gao
Quoted 7 lines

2. There was some concern that exposing non-zero bits in [23:16] could confuse
existing TDs. Of course KVM doesn't support any TDs today, but if this feature
comes after initial KVM support for TDX and KVM wants to set it by default, then
it could be an issue.

Quoted 5 lines

Do you mean some TDs may assert that [23:16] are 0s? A future-proof design won't have this assertion. And this case (i.e., some CPUID bits become non- zero)
happens on every new generation of CPUs and doesn't confuse existing OSes. I don't understand why it would be a problem for TDs.

Intel defined these as reserved. AMD defined them for guest MAXPA. So, yes, OSs should be masking them. I'm not suggesting that any are not, but TDX module folks were concerned about this, and that then KVM would not be able to turn this on later without breaking them. So just circling back here to double check.

#14 lore raw
Quoted 11 lines · Rick P Edgecombe
Quoted 9 lines · Chao Gao
Quoted 4 lines

2. There was some concern that exposing non-zero bits in [23:16] could confuse existing TDs. Of course KVM doesn't support any TDs today, but if this feature comes after initial KVM support for TDX and KVM wants to set it by default, then it could be an issue.

Do you mean some TDs may assert that [23:16] are 0s? A future-proof design won't have this assertion. And this case (i.e., some CPUID bits become non- zero) happens on every new generation of CPUs and doesn't confuse existing OSes. I don't understand why it would be a problem for TDs.

Quoted 4 lines

Intel defined these as reserved. AMD defined them for guest MAXPA. So, yes, OSs should be masking them. I'm not suggesting that any are not, but TDX module folks were concerned about this, and that then KVM would not be able to turn this on later without breaking them. So just circling back here to double check.

I'm with Chao, a kernel/firmware implementation that asserts some CPUID bits that are currently reserved on _some_ CPUs are always zero deserves to be broken.

#15 lore raw
In reply to Edgecombe, Rick P
Quoted 7 lines · Rick P

Hi,

This is a new effort to solicit community feedback for potential future TDX module features. There are two features in different stages of development around the configurability of the max physical address exposed in 0x80000008.EAX. I was hoping to get some comments on them and share the current plans on whether to implement them in KVM.

Sean and Paolo,

Quoted 9 lines

One of the TDX module features is called MAXPA_VIRT. In short, it is similar to KVM’s allow_smaller_maxphyaddr. It requires an explicit opt-in by the VMM, and allows a TD’s 0x80000008.EAX[7:0] to be configured by the VMM. Accesses to physical addresses above the specified value by the TD will cause the TDX module to inject a mostly correct #PF with the RSVD error code set. It has to deal with the same problems as allow_smaller_maxphyaddr for correctly setting the RSVD bit. I wasn’t thinking to push this feature for KVM due the movement away from allow_smaller_maxphyaddr and towards 0x80000008.EAX[23:16].

I would like to get your opinion of the MAXPA_VIRT feature of TDX. What is likely the KVM's decision on it? Won't support it due to it has the same limitation of allow_smaller_maxphyaddr?

#16 lore raw
Quoted 11 lines · Xiaoyao Li
Quoted 9 lines · Rick P

One of the TDX module features is called MAXPA_VIRT. In short, it is similar to KVM’s allow_smaller_maxphyaddr. It requires an explicit opt-in by the VMM, and allows a TD’s 0x80000008.EAX[7:0] to be configured by the VMM. Accesses to physical addresses above the specified value by the TD will cause the TDX module to inject a mostly correct #PF with the RSVD error code set. It has to deal with the same problems as allow_smaller_maxphyaddr for correctly setting the RSVD bit. I wasn’t thinking to push this feature for KVM due the movement away from allow_smaller_maxphyaddr and towards 0x80000008.EAX[23:16].

Quoted 3 lines

I would like to get your opinion of the MAXPA_VIRT feature of TDX. What is likely the KVM's decision on it? Won't support it due to it has the same limitation of allow_smaller_maxphyaddr?

Not supporting MAXPA_VIRT has my vote. I'm of the opinion that allow_smaller_maxphyaddr should die a horrible, fiery death :-)

#17 lore raw
Quoted 12 lines · Sean Christopherson
Quoted 11 lines · Xiaoyao Li
Quoted 9 lines · Rick P

One of the TDX module features is called MAXPA_VIRT. In short, it is similar to KVM’s allow_smaller_maxphyaddr. It requires an explicit opt-in by the VMM, and allows a TD’s 0x80000008.EAX[7:0] to be configured by the VMM. Accesses to physical addresses above the specified value by the TD will cause the TDX module to inject a mostly correct #PF with the RSVD error code set. It has to deal with the same problems as allow_smaller_maxphyaddr for correctly setting the RSVD bit. I wasn’t thinking to push this feature for KVM due the movement away from allow_smaller_maxphyaddr and towards 0x80000008.EAX[23:16].

Quoted 6 lines
Quoted 3 lines

I would like to get your opinion of the MAXPA_VIRT feature of TDX. What is likely the KVM's decision on it? Won't support it due to it has the same limitation of allow_smaller_maxphyaddr?

Not supporting MAXPA_VIRT has my vote. I'm of the opinion that allow_smaller_maxphyaddr should die a horrible, fiery death :-)

Thanks for the response. It's good to know your preference.

I'm not sure if there is any user of "allow_smaller_maxphyaddr". On QEMU side, it doesn't check it nor rely on it. QEMU always allow the user to configure a smaller PA.