Bug? Incompatible APF for 4.14 guest on 5.10 and later host
Thanks, Vitaly, Paolo for your replies!
I'll reply just to this message to avoid branching the conversation too much.
Quoted 27 lines
-----Original Message-----
From: Paolo Bonzini <[email protected]>
Sent: 05 October 2023 17:15
To: Mancini, Riccardo <[email protected]>; [email protected]
Cc: [email protected]; Graf (AWS), Alexander <[email protected]>; Teragni, Matias <[email protected]>; Batalov, Eugene <[email protected]>
Subject: RE: [EXTERNAL] Bug? Incompatible APF for 4.14 guest on 5.10 and later hostQuoted 5 lines · Riccardo
Hi,
when a 4.14 guest runs on a 5.10 host (and later), it cannot use APF (despite CPUID advertising
KVM_FEATURE_ASYNC_PF) due to the new interrupt-based mechanism2635b5c4a0(KVM: x86: interrupt based APF'page ready' event delivery).
Quoted 2 lines
Kernels after 5.9 won't satisfy the guest request to enable APF through
KVM_ASYNC_PF_ENABLED, requiring also
KVM_ASYNC_PF_DELIVERY_AS_INTto be set.Quoted 5 lines
Furthermore, the patch set seems to be dropping parts of the legacy #PF handling as well.
I consider this as a bug as it breaks APF compatibility for older guests running on newer kernels, by breaking the underlying ABI. What do you think? Was this a deliberate decision?
Quoted 4 lines
Yes, this is intentional. It is not a breakage because the APF interface only tells how asynchronous page faults are delivered; it doesn't promise that they are actually delivered. However, I admit that the change was unfortunate.
:(
Makes sense, thanks for the explanation.
Quoted 21 lines
Apart from the concerns about reentrancy, there were two more issues with the old API:
- the page-ready notification lacked an acknowledge mechanism if many pages became ready at the same time (see commit
557a961abbe0, "KVM: x86:
acknowledgment mechanism for async pf page ready notifications"). This delayed the notifications of pages after the first. The new API usesMSR_KVM_ASYNC_PF_ACKto fix the problem.- the old API confused synchronous events (exceptions) with asynchronous events (interrupts); this created a unique case where a page fault was generated on a page that is not accessed by the instruction. (The new API only fixes half of this, because it also has a bogus CR2, but it's a bit better). It also meant that page-ready events were suppressed by disabled interrupts---but they were not necessarily injected when IF became 1, because KVM did not enable the interrupt window. This is solved automatically by just injecting an interrupt. On the theoretical side, it's also just ugly that page-ready events could only be enabled/disabled with CLI/STI and not APIC (TPR).
Quoted 7 lines
Quoted 4 lines
Was this already reported in the past (I couldn't find anything in the mailing list but I might have missed it!)?
Would it be much effort to support the legacy #PF based mechanism for older guests that choose to only setKVM_ASYNC_PF_ENABLED?It is not hard. However, I don't think we should accept such a patch upstream.
Regarding also Vitaly comment about backporting the changes to 4.14, I think supporting both modes in 5.10 (at least) might be the least effort path (fewer changes), at least to my naive untrained eye.
I tried to playing around by partially reverting some of the changes to handle both cases but only got kernel panics in the guest so far, so I might be missing something.
However, I have absolutely no experience with KVM code, so I wasn't expecting to get far in any case.
Quoted 4 lines
I do have a question for you. Can you describe the context in which you are using APF, and would you be interested in ARM support? We (Red Hat, not me the maintainer :)) have been trying to understand for a long time if cloud providers use or need APF.
Keeping it short, we resume "remote" VM snapshots so page faults might be very expensive on local cache misses. We have a few optimizations to work around some of the issues, but even on local hits there are still a lot of expensive page faults compared to a normal VM use-case, I believe. To be fair, I didn't even realise the benefits we were getting from APF until it actually broke :)
It indeed plays a big role in keeping the resumption quick and efficient in our use-case.
I didn't know that it wasn't available for ARM, as we don't use it at the moment, but that would be interesting for the future.
Thanks,
Riccardo
Quoted 8 lines
Paolo
Quoted 4 lines
The reason this is an issue for us now is that not having APF for older guests introduces a significant performance regression on 4.14 guests when paired to uffd handling of "remote" page-faults (similar to a live migration scenario) when we update from a 4.14 host kernel to
a 5.10 host kernel.
Quoted 2 lines · Riccardo
Quoted 2 lines
From: Paolo Bonzini <[email protected]>
Sent: 05 October 2023 17:15
[...]
Quoted 5 lines
Quoted 4 lines
I do have a question for you. Can you describe the context in which you are using APF, and would you be interested in ARM support? We (Red Hat, not me the maintainer :)) have been trying to understand for a long time if cloud providers use or need APF.
Quoted 11 lines
Keeping it short, we resume "remote" VM snapshots so page faults might be very expensive on local cache misses. We have a few optimizations to work around some of the issues, but even on local hits there are still a lot of expensive page faults compared to a normal VM use-case, I believe. To be fair, I didn't even realise the benefits we were getting from APF until it actually broke :)
It indeed plays a big role in keeping the resumption quick and efficient in our use-case.
I didn't know that it wasn't available for ARM, as we don't use it at the moment, but that would be interesting for the future.
Adding Marc, Oliver and [email protected]
I tried to make the feature available to ARM64 long time ago, but the efforts were discontinued as the significant concern was no users demanding for it [1]. It's definitely exciting news to know it's a important feature to AWS. I guess it's probably another chance to re-evaluate the feature for ARM64?
[1] https://lore.kernel.org/kvmarm/[email protected]/
Async PF needs two signals sent from host to guest, SDEI (Software Delegated Exception Interface) is leveraged for that. So there were two series to support SDEI virtualization [1] and Async PF on ARM64 [2].
[1] https://lore.kernel.org/kvmarm/[email protected]/
[2] https://lore.kernel.org/kvmarm/[email protected]/
I got several questions for Mancini to answer, helpful understand the situation better.
- VM shapshot is stored somewhere remotely. It means the page fault on instruction fetch becomes expensive. Do we have benchmarks how much benefits brought by Async PF on x86 in AWS environment?
- I'm wandering if the data can be fetched from somewhere remotely in AWS environment?
- The data can be stored in local DRAM or swapping space, the page fault to fetch data becomes expensive if the data is stored in swapping space. I'm not sure if it's possible the data resides in the swapping space in AWS environment? Note that the swapping space, corresponding to disk, could be somewhere remotely seated.
Thanks,
Gavin