Skip to contentSkip to conversation

rcu batch tuning

PATCH Oleg Nesterov 3 participants 5 messages Jan 27, 2006 – Jan 30, 2006

lore
Share & tools
lore
#0 lore raw
Quoted 23 lines · Dipankar Sarma
(quoted diff) +18 -0
+static void force_quiescent_state(struct rcu_data *rdp,
+			struct rcu_ctrlblk *rcp)
+{
+	int cpu;
+	cpumask_t cpumask;
+	set_need_resched();
+	if (unlikely(rdp->qlen - rdp->last_rs_qlen > rsinterval)) {
+		rdp->last_rs_qlen = rdp->qlen;
+		/*
+		 * Don't send IPI to itself. With irqs disabled,
+		 * rdp->cpu is the current cpu.
+		 */
+		cpumask = rcp->cpumask;
+		cpu_clear(rdp->cpu, cpumask);
+		for_each_cpu_mask(cpu, cpumask)
+			smp_send_reschedule(cpu);
+	}
+}
[ ... snip ... ]
@@ -92,17 +128,13 @@ void fastcall call_rcu(struct rcu_head *
 	rdp = &__get_cpu_var(rcu_data);

Review void fastcall call_rcu(struct rcu_head * · line 134

Quoted 10 lines
(quoted diff) +4 -4
 	*rdp->nxttail = head;
 	rdp->nxttail = &head->next;
-
-	if (unlikely(++rdp->count > 10000))
-		set_need_resched();
-
+	if (unlikely(++rdp->qlen > qhimark)) {
+		rdp->blimit = INT_MAX;
+		force_quiescent_state(rdp, &rcu_ctrlblk);
+	}

When ->qlen exceeds qhimark for the first time we send reschedule IPI to other CPUs and force_quiescent_state() records ->last_rs_qlen = ->qlen. But we don't reset ->last_rs_qlen when ->qlen goes to 0, this means that next time we need ++rdp->qlen > qhimark + rsinterval to force other CPUS to pass quiescent state, no?

Also, it seems to me it's better to have 2 counters, one for length(->donelist) and another for length(->curlist + ->nxtlist). I think we don't need force_quiescent_state() when all rcu callbacks are placed in ->donelist, we only need to increase rdp->blimit in this case.

Oleg.

#1 lore raw
Quoted 32 lines · Oleg Nesterov
Quoted 31 lines · Dipankar Sarma
(quoted diff) +20 -4
+static void force_quiescent_state(struct rcu_data *rdp,
+			struct rcu_ctrlblk *rcp)
+{
+	int cpu;
+	cpumask_t cpumask;
+	set_need_resched();
+	if (unlikely(rdp->qlen - rdp->last_rs_qlen > rsinterval)) {
+		rdp->last_rs_qlen = rdp->qlen;
+		/*
+		 * Don't send IPI to itself. With irqs disabled,
+		 * rdp->cpu is the current cpu.
+		 */
+		cpumask = rcp->cpumask;
+		cpu_clear(rdp->cpu, cpumask);
+		for_each_cpu_mask(cpu, cpumask)
+			smp_send_reschedule(cpu);
+	}
+}
[ ... snip ... ]
@@ -92,17 +128,13 @@ void fastcall call_rcu(struct rcu_head *
 	rdp = &__get_cpu_var(rcu_data);
 	*rdp->nxttail = head;
 	rdp->nxttail = &head->next;
-
-	if (unlikely(++rdp->count > 10000))
-		set_need_resched();
-
+	if (unlikely(++rdp->qlen > qhimark)) {
+		rdp->blimit = INT_MAX;
Quoted 8 lines

Review void fastcall call_rcu(struct rcu_head * · line 134

Quoted 2 lines
(quoted diff) +2 -0
+		force_quiescent_state(rdp, &rcu_ctrlblk);
+	}

When ->qlen exceeds qhimark for the first time we send reschedule IPI to other CPUs and force_quiescent_state() records ->last_rs_qlen = ->qlen. But we don't reset ->last_rs_qlen when ->qlen goes to 0, this means that next time we need ++rdp->qlen > qhimark + rsinterval to force other CPUS to pass quiescent state, no?

Good catch -- this could well explain Lee's continuing to hit latency problems. Although this would not cause the first latency event, only subsequent ones, it seems to me that ->last_rs_qlen should be reset whenever ->blimit is reset.

Dipankar, thoughts?

Quoted 4 lines

Also, it seems to me it's better to have 2 counters, one for length(->donelist) and another for length(->curlist + ->nxtlist). I think we don't need force_quiescent_state() when all rcu callbacks are placed in ->donelist, we only need to increase rdp->blimit in this case.

True, currently the patch keeps the sum of the length of all three lists, and takes both actions when the sum gets too large. But the only way you would get unneeded IPIs would be if callback processing was stalled, but callback generation and grace-period processing was still proceeding. Seems at first glance to be an unusual corner case, with the only downside being some extra IPIs. Or am I missing some aspect?

Thanx, Paul

#2 lore raw
Quoted 9 lines · Paul E. McKenney
Quoted 6 lines · Oleg Nesterov

When ->qlen exceeds qhimark for the first time we send reschedule IPI to other CPUs and force_quiescent_state() records ->last_rs_qlen = ->qlen. But we don't reset ->last_rs_qlen when ->qlen goes to 0, this means that next time we need ++rdp->qlen > qhimark + rsinterval to force other CPUS to pass quiescent state, no?

Quoted 4 lines

Good catch -- this could well explain Lee's continuing to hit latency problems. Although this would not cause the first latency event, only subsequent ones, it seems to me that ->last_rs_qlen should be reset whenever ->blimit is reset.

May be it's better to do it in other way?

struct rcu_ctrlblk {
	...
	int signaled;
	...
};

void force_quiescent_state(rdp, rcp)

{
	if (!rcp->signaled) {
		// racy, but tolerable
		rcp->signaled = 1;

		for_each_cpu_mask(cpu, cpumask)
			smp_send_reschedule(cpu);
	}
}

void rcu_start_batch(rcp, rdp)

{
	if (->next_pending && ->completed == ->cur) {
		...
		rcp->signaled = 0;
		...
	}
}

Probably it is also makes sense to tasklet_schedule(rcu_tasklet) in call_rcu() when ++rdp->qlen > qhimark, this way we can detect that we need to start the next batch earlier.

Quoted 5 lines
Quoted 4 lines

Also, it seems to me it's better to have 2 counters, one for length(->donelist) and another for length(->curlist + ->nxtlist). I think we don't need force_quiescent_state() when all rcu callbacks are placed in ->donelist, we only need to increase rdp->blimit in this case.

Quoted 7 lines

True, currently the patch keeps the sum of the length of all three lists, and takes both actions when the sum gets too large. But the only way you would get unneeded IPIs would be if callback processing was stalled, but callback generation and grace-period processing was still proceeding. Seems at first glance to be an unusual corner case, with the only downside being some extra IPIs. Or am I missing some aspect?

Yes, it is probably not worth to complicate the code.

Oleg.

#3 lore raw
Quoted 34 lines · Oleg Nesterov
Quoted 9 lines · Paul E. McKenney
Quoted 6 lines · Oleg Nesterov

When ->qlen exceeds qhimark for the first time we send reschedule IPI to other CPUs and force_quiescent_state() records ->last_rs_qlen = ->qlen. But we don't reset ->last_rs_qlen when ->qlen goes to 0, this means that next time we need ++rdp->qlen > qhimark + rsinterval to force other CPUS to pass quiescent state, no?

Quoted 4 lines

Good catch -- this could well explain Lee's continuing to hit latency problems. Although this would not cause the first latency event, only subsequent ones, it seems to me that ->last_rs_qlen should be reset whenever ->blimit is reset.

May be it's better to do it in other way?

struct rcu_ctrlblk {
	...
	int signaled;
	...
};

void force_quiescent_state(rdp, rcp)

{
	if (!rcp->signaled) {
		// racy, but tolerable
		rcp->signaled = 1;

		for_each_cpu_mask(cpu, cpumask)
			smp_send_reschedule(cpu);
	}
}
Quoted 8 lines

void rcu_start_batch(rcp, rdp)

{
	if (->next_pending && ->completed == ->cur) {
		...
		rcp->signaled = 0;
		...
	}
}

Possibly... But the best thing would be for you and Dipankar to get together to work out the best strategy for this.

Thanx, Paul

Quoted 21 lines

Probably it is also makes sense to tasklet_schedule(rcu_tasklet) in call_rcu() when ++rdp->qlen > qhimark, this way we can detect that we need to start the next batch earlier.

Quoted 5 lines
Quoted 4 lines

Also, it seems to me it's better to have 2 counters, one for length(->donelist) and another for length(->curlist + ->nxtlist). I think we don't need force_quiescent_state() when all rcu callbacks are placed in ->donelist, we only need to increase rdp->blimit in this case.

Quoted 7 lines

True, currently the patch keeps the sum of the length of all three lists, and takes both actions when the sum gets too large. But the only way you would get unneeded IPIs would be if callback processing was stalled, but callback generation and grace-period processing was still proceeding. Seems at first glance to be an unusual corner case, with the only downside being some extra IPIs. Or am I missing some aspect?

Yes, it is probably not worth to complicate the code.

Oleg.

#4 lore raw
In reply to Oleg Nesterov
Quoted 12 lines · Oleg Nesterov
Quoted 11 lines · Dipankar Sarma
(quoted diff) +2 -4
@@ -92,17 +128,13 @@ void fastcall call_rcu(struct rcu_head *
 	rdp = &__get_cpu_var(rcu_data);
 	*rdp->nxttail = head;
 	rdp->nxttail = &head->next;
-
-	if (unlikely(++rdp->count > 10000))
-		set_need_resched();
-
+	if (unlikely(++rdp->qlen > qhimark)) {
+		rdp->blimit = INT_MAX;
Quoted 8 lines

Review void fastcall call_rcu(struct rcu_head * · line 134

Quoted 2 lines
(quoted diff) +2 -0
+		force_quiescent_state(rdp, &rcu_ctrlblk);
+	}

When ->qlen exceeds qhimark for the first time we send reschedule IPI to other CPUs and force_quiescent_state() records ->last_rs_qlen = ->qlen. But we don't reset ->last_rs_qlen when ->qlen goes to 0, this means that next time we need ++rdp->qlen > qhimark + rsinterval to force other CPUS to pass quiescent state, no?

Yes. I have fixed it my code.

Quoted 5 lines

Also, it seems to me it's better to have 2 counters, one for length(->donelist) and another for length(->curlist + ->nxtlist). I think we don't need force_quiescent_state() when all rcu callbacks are placed in ->donelist, we only need to increase rdp->blimit in this case.

We could, but I am not sure that ->qlen is not a good measure of grace period not happening. In fact, in the past, long donelists often resulted in longer grace periods. So, no harm in forcing reschedule and allowing ksoftirq to relinquish cpu.

Thanks
Dipankar