Expedite srcu_schedule_cbs_snp() callback invocation

Although Tree SRCU does reduce delays when there is at least one synchronize_srcu_expedited() invocation pending, srcu_schedule_cbs_snp() still waits for SRCU_INTERVAL before invoking callbacks. Since synchronize_srcu_expedited() now posts a callback and waits for that callback to do a wakeup, this destroys the expedited nature of synchronize_srcu_expedited().

This commit therefore invokes callbacks immediately at the end of the grace period when there is at least one synchronize_srcu_expedited() invocation pending.

Reported-by Marc Zyngier <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

kernel/rcu/srcutree.c +0 -0

index 1e54f91c2ff7..744ffdb79d23 100644

Quoted 19 lines · Paul E. McKenney

Although Tree SRCU does reduce delays when there is at least one synchronize_srcu_expedited() invocation pending, srcu_schedule_cbs_snp() still waits for SRCU_INTERVAL before invoking callbacks. Since synchronize_srcu_expedited() now posts a callback and waits for that callback to do a wakeup, this destroys the expedited nature of synchronize_srcu_expedited().

This commit therefore invokes callbacks immediately at the end of the grace period when there is at least one synchronize_srcu_expedited() invocation pending.

Reported-by Marc Zyngier <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

kernel/rcu/srcutree.c +0 -0

index 1e54f91c2ff7..744ffdb79d23 100644

kernel/rcu/srcutree.c static void srcu_schedule_cbs_snp(struct srcu_struct *sp, struct srcu_node *snp) · line 454

Quoted 9 lines

kernel/rcu/srcutree.c +2 -1

@@ -447,7 +447,8 @@ static void srcu_schedule_cbs_snp(struct srcu_struct *sp, struct srcu_node *snp)
 	int cpu;
 
 	for (cpu = snp->grplo; cpu <= snp->grphi; cpu++)
-		srcu_schedule_cbs_sdp(per_cpu_ptr(sp->sda, cpu), SRCU_INTERVAL);
+		srcu_schedule_cbs_sdp(per_cpu_ptr(sp->sda, cpu),
+				      atomic_read(&sp->srcu_exp_cnt) ? 0 : SRCU_INTERVAL);
 }
 
 /*

This fixes the issue I was observing, where UEFI in a KVM guest would take a x8 slowdown while it messes with some NOR flash emulation on arm64 (don't ask, this is horrible...). Anyway:

Tested-by Marc Zyngier <[email protected]>

Thanks again Paul!