rcu: speed up quiescent state detection

On top of Manfred's 'rcu: simplify quiescent state detection', see
http://marc.theaimsgroup.com/?l=linux-kernel&m=110433412126392

Let's suppose that cpu is running idle thread or user level process, and the grace period was started.

Afaics, currently we need 2 local timer interrupts to happen before this cpu can end its grace period.

1st timer_interrupt:

	__rcu_pending():
		if (rdp->quiescbatch != rcp->cur)
			return 1;
	rcu_check_callbacks():
		if (user || idle)
			->passed_quiesc = 1;
		tasklet_schedule(rcu_tasklet)
		do_softirq():
			__rcu_process_callbacks():
				rcu_check_quiescent_state()
					if (rdp->quiescbatch != rcp->cur) {
						->qs_pending = 1;
						->passed_quiesc = 0;
						return;
					}

Only the next interrupt will notice ->qs_pending in __rcu_pending() and increment ->completed in rcu_check_quiescent_state().

I think it is better to set ->qs_pending = 1 directly in __rcu_pending():

	int __rcu_pending()
	{
		if (qs_pending) return 1;

		if (quiescbatch != rcp->cur) {
			quiescbatch = rcp->cur;
			passed_quiesc = 0;
			barrier();
			qs_pending = 1;
			return 1;
		}
		... other checks ...
	}

	void rcu_check_quiescent_state()
	{
		if (!qs_pending) return;
		barrier();
		if (!passed_quiesc) return;

		cpu_quiet();

		qs_pending = 0;
	}

This way grace period for that cpu will be completed after 1st interupt.

Note that when fork()->scheduler_tick()->rcu_pending() runs in process context, it does this with interrupts disabled, so there is no race with timer interrupt (Although i think that it is better to call rcu_pending() directly from update_process_times() instead).

What do you think?

Oleg.

@@ -124,30 +124,7 @@ static inline void rcu_bh_qsctr_inc(int 
 	rdp->passed_quiesc = 1;
 }
 
-static inline int __rcu_pending(struct rcu_ctrlblk *rcp,
-						struct rcu_data *rdp)
-{
-	/* This cpu has pending rcu entries and the grace period
-	 * for them has completed.
-	 */
-	if (rdp->curlist && !rcu_batch_before(rcp->completed, rdp->batch))
-		return 1;
-
-	/* This cpu has no pending entries, but there are new entries */
-	if (!rdp->curlist && rdp->nxtlist)
-		return 1;
-
-	/* This cpu has finished callbacks to invoke */
-	if (rdp->donelist)
-		return 1;
-
-	/* The rcu core waits for a quiescent state from the cpu */
-	if (rdp->quiescbatch != rcp->cur || rdp->qs_pending)
-		return 1;
-
-	/* nothing to do */
-	return 0;
-}
+extern int __rcu_pending(struct rcu_ctrlblk *rcp, struct rcu_data *rdp);
 
 static inline int rcu_pending(int cpu)
 {

I agree with you that the design is not optimal, but I must think about it a bit more. Right now the grace period handling consists out of three steps for each cpu:
- start grace period (from timer irq)
- log quiescent state in ->passed_quiesc (from timer or schedule())
- end grace period (from timer irq)

This requires at least two timer ticks. Your patch would reduce that to one timer tick, without changing the approach. I'm still thinking about the design change I proposed: In each quiescent state: check if there is a running grace period and if there is one, then mark the current cpu as done.

Dipankar: What do you think?

--
Manfred

I must admit, i have misunderstood your patch after the first reading. Now i think your idea is better.

Oleg.