Skip to content

KVM: x86/xen: Fix Xen/GP/PREEMPT_RT issues with rwlock_t

Sean Christopherson kvm lkml sashiko-reviews 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v2 Latest20 patches 33 repliesIn review

Changes in v2 from the cover letter

- Add the CLASS() APIs.
- Move the steal time change to the very end.
- "Fix" a dirty logging inconsistency with the Xen vCPU info page.

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/20locking/rt: Use raw_spin_lock_irqsave() in __rwbase_read_unlock()kernel/locking/rwbase_rt.c+3 -225 replies
2/20KVM: x86/xen: Use read_trylock() for GPC locks in hardirq/atomic pathsarch/x86/kvm/xen.c+23 -91 reply
3/20KVM: x86/xen: Remove unnecessary irqsave from GPC lock usage in xen.carch/x86/kvm/xen.c+25 -350 replies
4/20KVM: x86: Remove unnecessary irqsave from kvm_setup_guest_pvclock()arch/x86/kvm/x86.c+4 -50 replies
5/20KVM: Remove unnecessary IRQ disabling from GPC lock in pfncache.cvirt/kvm/pfncache.c+14 -140 replies
6/20KVM: x86/xen: Use guard() to grab kvm->srcu around gpc critical sectionsarch/x86/kvm/xen.c+14 -180 replies
7/20KVM: x86/xen: Extract delivery of event to vCPU into a separate helperarch/x86/kvm/xen.c+54 -460 replies
8/20KVM: x86/xen: Explicitly tag "shared info" page as never being dirty trackedarch/x86/kvm/xen.c include/linux/kvm_host.h include/linux/kvm_types.h +1 more+15 -50 replies
9/20KVM: x86/xen: Don't dirty track "vCPU info" pagearch/x86/kvm/xen.c+1 -20 replies
10/20KVM: Move {g,p}fn <=> {g,h}pa conversion helpers to kvm_types.hinclude/linux/kvm_host.h include/linux/kvm_types.h+16 -150 replies
11/20KVM: Add CLASS() constructs to automagically handle lock+check of gpcarch/x86/kvm/x86.c include/linux/kvm_host.h virt/kvm/pfncache.c+78 -230 replies
12/20KVM: x86/xen: Convert kvm_xen_shared_info_init() to gpc's CLASS() APIsarch/x86/kvm/xen.c+17 -170 replies
13/20KVM: x86/xen: Don't bother waiting on gpc->lock in SCHEDOP_pollarch/x86/kvm/xen.c+3 -10 replies
14/20KVM: x86/xen: Convert wait_pending_event() to gpc's CLASS() APIsarch/x86/kvm/xen.c+8 -160 replies
15/20KVM: x86/xen: Convert kvm_xen_set_evtchn_fast() to gpc's CLASS() APIsarch/x86/kvm/xen.c+16 -263 replies
16/20KVM: x86/xen: Convert xen_get_guest_pvclock() to gpc's CLASS() APIsarch/x86/kvm/xen.c+4 -140 replies
17/20KVM: x86/xen: Drop local "kick_vcpu" from __kvm_xen_set_evtchn_fast()arch/x86/kvm/xen.c+15 -180 replies
18/20KVM: x86/xen: Convert event injection to gpc's CLASS() APIsarch/x86/kvm/xen.c+5 -190 replies
19/20KVM: Add "extended" gpc CLASS() APIs for sometimes-atomic casesarch/x86/kvm/xen.c include/linux/kvm_host.h+26 -360 replies
20/20KVM: x86: Use gfn_to_pfn_cache for record_steal_timearch/x86/include/asm/kvm_host.h arch/x86/kvm/x86.c+54 -704 replies

Reviews trailers per patch

1 of 20 patches has a Reviewed-by or Acked-by.

PatchSebastian Andrzej Siewior
1/20R-b
2/20
3/20
4/20
5/20
6/20
7/20
8/20
9/20
10/20
11/20
12/20
13/20
14/20
15/20
16/20
17/20
18/20
19/20
20/20

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v2 thread

Cover letter [PATCH v2 00/20] KVM: x86/xen: Fix Xen/GP/PREEMPT_RT issues with rwlock_t

This series fixes sleeping-in-hardirq bugs in KVM's Xen emulation on PREEMPT_RT, cleans up the now-unnecessary IRQ disabling in GPC lock usage throughout KVM, and then adds CLASS()-based APIs for utilizing GPCs mappings to dedup code and (hopefully) make it easier to use GPCs in other places.

The core issue is that kvm_xen_set_evtchn_fast() and the Xen timer callback are called from hardirq/atomic context, but on PREEMPT_RT the GPC rwlock_t is a sleeping lock.

Assuming I can get an Ack on patch 1, I'm planning on grabbing at least these

KVM: Move {g,p}fn <=> {g,h}pa conversion helpers to kvm_types.h
KVM: x86/xen: Don't dirty track "vCPU info" page
KVM: x86/xen: Explicitly tag "shared info" page as never being dirty tracked
KVM: x86/xen: Extract delivery of event to vCPU into a separate helper
KVM: x86/xen: Use guard() to grab kvm->srcu around gpc critical sections
KVM: Remove unnecessary IRQ disabling from GPC lock in pfncache.c
KVM: x86: Remove unnecessary irqsave from kvm_setup_guest_pvclock()
KVM: x86/xen: Remove unnecessary irqsave from GPC lock usage in xen.c
KVM: x86/xen: Use read_trylock() for GPC locks in hardirq/atomic paths locking/rt: Use raw_spin_lock_irqsave() in __rwbase_read_unlock()

for 7.2. If people like the CLASS() stuff, I'll also probably grab these:

KVM: Add "extended" gpc CLASS() APIs for sometimes-atomic cases
KVM: x86/xen: Convert event injection to gpc's CLASS() APIs
KVM: x86/xen: Drop local "kick_vcpu" from __kvm_xen_set_evtchn_fast()
KVM: x86/xen: Convert xen_get_guest_pvclock() to gpc's CLASS() APIs
KVM: x86/xen: Convert kvm_xen_set_evtchn_fast() to gpc's CLASS() APIs
KVM: x86/xen: Convert wait_pending_event() to gpc's CLASS() APIs
KVM: x86/xen: Don't bother waiting on gpc->lock in SCHEDOP_poll
KVM: x86/xen: Convert kvm_xen_shared_info_init() to gpc's CLASS() APIs
KVM: Add CLASS() constructs to automagically handle lock+check of gpc

I do NOT plan on grabbing the record_steal_time change for 7.2 no matter what, even though I do like the end result, as I still have concerns over the lack of range-based invalidation for GPCs. I 100% agree that such problems are really only due to flawed VMMs and/or setups, but unfortunately history has shown that there are a suprising number of deployments running what I would consider flawed setups, e.g. run with NUMA autobalancing and KSM.

I realize I'm being somewhat paranoid, as KVM already uses a GPC for PV clocks. But for modern setups, KVM_REQ_CLOCK_UPDATE is a rare event, whereas KVM will update steal time (when enabled) on every vCPU load. So I want a high level of confidence that KVM won't regress "imperfect" setups before switching to a GPC for steal time (though again, I definitely like the end result and want to do so).

[*] https://lore.kernel.org/all/[email protected]

v2:
- Add the CLASS() APIs.
- Move the steal time change to the very end.
- "Fix" a dirty logging inconsistency with the Xen vCPU info page.

v1: https://lore.kernel.org/all/[email protected]

Carsten Stollmaier (1):
KVM: x86: Use gfn_to_pfn_cache for record_steal_time

David Woodhouse (5):
locking/rt: Use raw_spin_lock_irqsave() in __rwbase_read_unlock()
KVM: x86/xen: Use read_trylock() for GPC locks in hardirq/atomic paths
KVM: x86/xen: Remove unnecessary irqsave from GPC lock usage in xen.c
KVM: x86: Remove unnecessary irqsave from kvm_setup_guest_pvclock()
KVM: Remove unnecessary IRQ disabling from GPC lock in pfncache.c

Sean Christopherson (14):
KVM: x86/xen: Use guard() to grab kvm->srcu around gpc critical
sections
KVM: x86/xen: Extract delivery of event to vCPU into a separate helper
KVM: x86/xen: Explicitly tag "shared info" page as never being dirty
tracked
KVM: x86/xen: Don't dirty track "vCPU info" page
KVM: Move {g,p}fn <=> {g,h}pa conversion helpers to kvm_types.h
KVM: Add CLASS() constructs to automagically handle lock+check of gpc
KVM: x86/xen: Convert kvm_xen_shared_info_init() to gpc's CLASS() APIs
KVM: x86/xen: Don't bother waiting on gpc->lock in SCHEDOP_poll
KVM: x86/xen: Convert wait_pending_event() to gpc's CLASS() APIs
KVM: x86/xen: Convert kvm_xen_set_evtchn_fast() to gpc's CLASS() APIs
KVM: x86/xen: Convert xen_get_guest_pvclock() to gpc's CLASS() APIs
KVM: x86/xen: Drop local "kick_vcpu" from __kvm_xen_set_evtchn_fast()
KVM: x86/xen: Convert event injection to gpc's CLASS() APIs
KVM: Add "extended" gpc CLASS() APIs for sometimes-atomic cases

 arch/x86/include/asm/kvm_host.h |   2 +-
 arch/x86/kvm/x86.c              | 140 +++++++---------
 arch/x86/kvm/xen.c              | 288 +++++++++++++-------------------
 include/linux/kvm_host.h        |  84 +++++++---
 include/linux/kvm_types.h       |  17 ++
 kernel/locking/rwbase_rt.c      |   5 +-
 virt/kvm/pfncache.c             |  68 ++++++--
 7 files changed, 304 insertions(+), 300 deletions(-)

base-commit: d1568b1332b6b3b36b222c2868fc102727c12a34

Signature · 1 line
2.54.0.823.g6e5bcc1fc9-goog