srcu: KVM: Add, export and use call_srcu_expedited()
Subsystems (3)
Sean Christopherson
reviewercover srcu: KVM: Add, export and use call_srcu_expedited()
We've got a conundrum in KVM where we have multiple use cases that generally want the same thing (eliminate waiting on guest configuration changes whenever possible), but use KVM uAPIs in slightly different ways and effectively create competing requirements.
The crux of the problem is that one use case wants KVM to free an object via call_srcu() so that the task doesn't risk getting stalled waiting for a grace period. But for the other use case, using call_srcu() can trigger a non-expedited grace period and cause a synchronize_srcu_expedited() in a different ioctl (that must do a full sync, i.e. can't use call_srcu()) to stall waiting for the non-expedited grace period.
Tagged RFC because while having the call_srcu() request do an expedited grace period eliminates the unwanted synchronize_srcu_expedited() stalls, this feels
like a very crude fix. That said, I'm definitely not opposed to this being a
final solution if it's the best option available.
Sean Christopherson (3):
srcu: Declare exported symbols before including srcu{tiny,tree}.h
srcu: Add and export call_srcu_expedited() to avoid transferring grace
periods
KVM: Expedite SRCU callbacks when freeing objects during I/O bus
registration
include/linux/srcu.h | 10 +++++----- include/linux/srcutiny.h | 8 ++++++-- include/linux/srcutree.h | 2 ++ kernel/rcu/srcutree.c | 7 +++++++ virt/kvm/kvm_main.c | 2 +- 5 files changed, 21 insertions(+), 8 deletions(-)
base-commit: 5128b972fb2801ad9aca54d990a75611ab5283a9
Signature · 1 line
2.53.0.473.g4a7958ca14-goog
Sean Christopherson
reviewerRFC PATCH 1/3 srcu: Declare exported symbols before including srcu{tiny,tree}.h
Move the declarations of call_srcu(), cleanup_srcu_struct(), and synchronize_srcu() above the inclusion of the implementation specific SRCU header so that tiny SRCU can provide inline wrappers, e.g. for expedited versions, without needing to re-declare synchronize_srcu() and call_srcu().
Opportunsitically use rcu_callback_t in the call_srcu() declaration instead of an open coded equivalent (all implementations already use rcu_callback_t).
No functional change intended.
--- include/linux/srcu.h | 10 +++++----- include/linux/srcutiny.h | 2 -- 2 files changed, 5 insertions(+), 7 deletions(-)
include/linux/srcu.h +5 -5
@@ -79,6 +79,11 @@ int init_srcu_struct_fast_updown(struct srcu_struct *ssp);// instead of smp_mb(). void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases_shared(ssp); +void call_srcu (struct srcu_struct *ssp,struct rcu_head *head, +rcu_callback_t func); +void cleanup_srcu_struct (struct srcu_struct *ssp); +void synchronize_srcu (struct srcu_struct *ssp); +#ifdef CONFIG_TINY_SRCU #include <linux/srcutiny.h> #elif defined(CONFIG_TREE_SRCU )
@@ -87,11 +92,6 @@ void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases_shared(ssp)#error "Unknown SRCU implementation specified to kernel configuration"#endif -void call_srcu (struct srcu_struct *ssp,struct rcu_head *head, -void (*func)(struct rcu_head *head)); -void cleanup_srcu_struct (struct srcu_struct *ssp); -void synchronize_srcu (struct srcu_struct *ssp); -#define SRCU_GET_STATE_COMPLETED 0x1/**
include/linux/srcutiny.h +0 -2
@@ -64,8 +64,6 @@ struct srcu_usage { };#define init_srcu_struct_fast_updown init_srcu_struct#endif // #ifndef CONFIG_DEBUG_LOCK_ALLOC -void synchronize_srcu (struct srcu_struct *ssp); -/* * Counts the new reader in the appropriate per-CPU element of the * srcu_struct. Can be invoked from irq/bh handlers, but the matching
Signature · 1 line
2.53.0.473.g4a7958ca14-goog
Sean Christopherson
reviewerRFC PATCH 2/3 srcu: Add and export call_srcu_expedited() to avoid transferring grace periods
In reply to Sean ChristophersonAdd and export an expedited version of call_srcu() so that users of synchronize_srcu_expedited() can avoid "transferring" non-expedited grace periods. In response to userspace changes to guest devices and memory, KVM uses call_srcu() when freeing an object to avoid having to wait for readers to go away, but then often emits a synchronize_srcu_expedited() in a largely unrelated path shortly thereafter (on the same SRCU object).
Due to differences in how VMMs manage guest devices, and in the architecture being emulated by userspace, some updates trigger call_srcu() with concurrent readers (i.e. while the VM is active), while others occur without readers, e.g. when configuring devices during a pre-boot setup. For the later case (no concurrent readers), using the vanilla call_srcu() is problematic, as it can kick off a normal grace period (totally fine for freeing the object) and effectively transfer the non-expedited grace period to the upcoming synchronize_srcu_expedited().
For micro-VM use cases with CONFIG_HZ=100 kernels, the resulting ~20ms delay on the would-be-expedited sync can increase the boot time of the VM by 15% or more.
--- include/linux/srcutiny.h | 6 ++++++ include/linux/srcutree.h | 2 ++ kernel/rcu/srcutree.c | 7 +++++++ 3 files changed, 15 insertions(+)
include/linux/srcutiny.h +6 -0
@@ -130,6 +130,12 @@ static inline void srcu_barrier(struct srcu_struct *ssp)synchronize_srcu (ssp); } +static inline void call_srcu_expedited (struct srcu_struct *ssp,struct rcu_head *rhp, +rcu_callback_t func) +{ +call_srcu (ssp, rhp, func); +} +static inline void srcu_expedite_current (struct srcu_struct *ssp) { }#define srcu_check_read_flavor (ssp, read_flavor) do { } while (0)
include/linux/srcutree.h +2 -0
@@ -234,6 +234,8 @@ struct srcu_struct { __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST_UPDOWN, static) int __srcu_read_lock(struct srcu_struct *ssp) __acquires_shared(ssp); +void call_srcu_expedited (struct srcu_struct *ssp,struct rcu_head *head, +rcu_callback_t func);void synchronize_srcu_expedited (struct srcu_struct *ssp);void srcu_barrier (struct srcu_struct *ssp);void srcu_expedite_current (struct srcu_struct *ssp);
kernel/rcu/srcutree.c +7 -0
@@ -1495,6 +1495,13 @@ void call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp, }EXPORT_SYMBOL_GPL (call_srcu); +void call_srcu_expedited (struct srcu_struct *ssp,struct rcu_head *rhp, +rcu_callback_t func) +{ +__call_srcu (ssp, rhp, func,rcu_gp_is_normal ()); +} +EXPORT_SYMBOL_GPL (call_srcu_expedited); +/* * Helper function for synchronize_srcu() and synchronize_srcu_expedited(). */
Signature · 1 line
2.53.0.473.g4a7958ca14-goog
Sean Christopherson
reviewerRFC PATCH 3/3 KVM: Expedite SRCU callbacks when freeing objects during I/O bus registration
In reply to Sean ChristophersonUse the recently introduced call_srcu_expedited() when freeing the old I/O bug during device registration to avoid triggering a non-expedited grace period. Delaying the freeing of the object by a full grace period is a complete non-issue, but the grace period also gets transferred to future synchronizations, e.g. to the synchronize_srcu_expedited() invocation in kvm_swap_active_memslots().
For micro-VM use cases, effectively transferring the non-expedited grace period to memslot updates results in a meaningful delay in overall boot time. E.g. with a CONFIG_HZ=100 kernel, the sync triggers a ~20ms delay, increasing boot times by 15% or more.
--- virt/kvm/kvm_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
virt/kvm/kvm_main.c +1 -1
@@ -6021,7 +6021,7 @@ int kvm_io_bus_register_dev(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr,memcpy (new_bus->range + i +1 , bus->range + i, (bus->dev_count - i) *sizeof (struct kvm_io_range ));rcu_assign_pointer (kvm->buses[bus_idx], new_bus); -call_srcu (&kvm->srcu, &bus->rcu, __free_bus); +call_srcu_expedited (&kvm->srcu, &bus->rcu, __free_bus);return 0 ; }
Signature · 1 line
2.53.0.473.g4a7958ca14-goog
Quoted 17 lines · Sean Christopherson
We've got a conundrum in KVM where we have multiple use cases that generally want the same thing (eliminate waiting on guest configuration changes whenever possible), but use KVM uAPIs in slightly different ways and effectively create competing requirements.
The crux of the problem is that one use case wants KVM to free an object via
call_srcu()so that the task doesn't risk getting stalled waiting for a grace period. But for the other use case, usingcall_srcu()can trigger a non-expedited grace period and cause asynchronize_srcu_expedited()in a different ioctl (that must do a full sync, i.e. can't usecall_srcu()) to stall waiting for the non-expedited grace period.Tagged RFC because while having the
call_srcu()request do an expedited grace period eliminates the unwantedsynchronize_srcu_expedited()stalls, this feels
like a very crude fix. That said, I'm definitely not opposed to this being a
final solution if it's the best option available.
Quoted 4 lines
Sean Christopherson (3):
srcu: Declare exported symbols before including srcu{tiny,tree}.h
srcu: Add and exportcall_srcu_expedited()to avoid transferring grace
periods
Hi,
Thanks for writing this up.
The scenario you describe looks plausible.
That said, the cover letter wording might be a bit stronger than current SRCU behavior warrants. A later synchronize_srcu_expedited() can attempt to expedite an in-flight grace period, but it cannot avoid delay already incurred (for example, if the GP has already gone to sleep).
More generally, before adding an exported call_srcu_expedited() helper, should we consider improving existing in-flight promotion or delay behavior, or otherwise making the "expedite current GP" case more explicit without introducing a new callback-facing API?
Thanx, Kunwu
Quoted 12 lines
KVM: Expedite SRCU callbacks when freeing objects during I/O bus
registrationinclude/linux/srcu.h | 10 +++++----- include/linux/srcutiny.h | 8 ++++++-- include/linux/srcutree.h | 2 ++ kernel/rcu/srcutree.c | 7 +++++++ virt/kvm/kvm_main.c | 2 +- 5 files changed, 21 insertions(+), 8 deletions(-)base-commit:
5128b972fb2801ad9aca54d990a75611ab5283a9
Sean Christopherson
reviewerQuoted 35 lines · Kunwu Chan
Quoted 17 lines · Sean Christopherson
We've got a conundrum in KVM where we have multiple use cases that generally want the same thing (eliminate waiting on guest configuration changes whenever possible), but use KVM uAPIs in slightly different ways and effectively create competing requirements.
The crux of the problem is that one use case wants KVM to free an object via
call_srcu()so that the task doesn't risk getting stalled waiting for a grace period. But for the other use case, usingcall_srcu()can trigger a non-expedited grace period and cause asynchronize_srcu_expedited()in a different ioctl (that must do a full sync, i.e. can't usecall_srcu()) to stall waiting for the non-expedited grace period.Tagged RFC because while having the
call_srcu()request do an expedited grace period eliminates the unwantedsynchronize_srcu_expedited()stalls, this feels
like a very crude fix. That said, I'm definitely not opposed to this being a
final solution if it's the best option available.Quoted 4 lines
Sean Christopherson (3):
srcu: Declare exported symbols before including srcu{tiny,tree}.h
srcu: Add and exportcall_srcu_expedited()to avoid transferring grace
periodsHi,
Thanks for writing this up.
The scenario you describe looks plausible.
That said, the cover letter wording might be a bit stronger than current SRCU behavior warrants. A later
synchronize_srcu_expedited()can attempt to expedite an in-flight grace period, but it cannot avoid delay already incurred (for example, if the GP has already gone to sleep).
Quoted 4 lines
More generally, before adding an exported
call_srcu_expedited()helper, should we consider improving existing in-flight promotion or delay behavior, or otherwise making the "expedite current GP" case more explicit without introducing a new callback-facing API?
I'm all for a general solution, but that's far, far beyond my SRCU knowledge level. I was quite proud of myself for piecing together the incurred-delay. :-)
FYI, I'm going to be unavailable for ~2 weeks. Nikita (Cc'd) can likely help test potential fixes.
Thanks!
[...]
Quoted 6 lines
I'm all for a general solution, but that's far, far beyond my SRCU knowledge level. I was quite proud of myself for piecing together the incurred-delay. :-)
FYI, I'm going to be unavailable for ~2 weeks. Nikita (Cc'd) can likely help test potential fixes.
Hi Sean, hi Nikita,
Some data points on the RFC scenario.
The setup follows the pattern discussed in the RFC:
a call_srcu()-like path that can start a non-expedited GP, followed by a synchronize_srcu_expedited()-like wait on the same SRCU instance.
KVM selftest setup, BPF tracing.Single VM, single vCPU. Guest holds a long measured window (~8s, fixed in selftest).
Baseline:
In 5 recent 10-run batches, the no-preemption case produced a single >1ms sample in total.
With same-CPU preemption:
Quoted 1 line
1ms reproduces consistently and shows up in most runs.
For >1ms samples, sched_switch/off-CPU attribution lines up with switch-out in the same measured window.
In this setup, >1ms is mostly off-CPU time,not continuous in-CPU execution.
Question on the original report:
when synchronize_srcu_expedited() delay was observed, was the host mostly idle/bursty, or under steady load?
Can share repro details if needed.
Thanks,
Kunwu Chan