rcu batch tuning
Quoted 23 lines · Dipankar Sarma
(quoted diff) +18
-0+static void force_quiescent_state(struct rcu_data *rdp, + struct rcu_ctrlblk *rcp) +{ + int cpu; + cpumask_t cpumask; + set_need_resched(); + if (unlikely(rdp->qlen - rdp->last_rs_qlen > rsinterval)) { + rdp->last_rs_qlen = rdp->qlen; + /* + * Don't send IPI to itself. With irqs disabled, + * rdp->cpu is the current cpu. + */ + cpumask = rcp->cpumask; + cpu_clear(rdp->cpu, cpumask); + for_each_cpu_mask(cpu, cpumask) + smp_send_reschedule(cpu); + } +}@@ -92,17 +128,13 @@ void fastcall call_rcu(struct rcu_head * rdp = &__get_cpu_var(rcu_data);
Review void fastcall call_rcu(struct rcu_head * · line 134
Quoted 10 lines
(quoted diff) +4
-4*rdp->nxttail = head; rdp->nxttail = &head->next; - - if (unlikely(++rdp->count > 10000)) - set_need_resched(); - + if (unlikely(++rdp->qlen > qhimark)) { + rdp->blimit = INT_MAX; + force_quiescent_state(rdp, &rcu_ctrlblk); + }
When ->qlen exceeds qhimark for the first time we send reschedule IPI to other CPUs and force_quiescent_state() records ->last_rs_qlen = ->qlen. But we don't reset ->last_rs_qlen when ->qlen goes to 0, this means that next time we need ++rdp->qlen > qhimark + rsinterval to force other CPUS to pass quiescent state, no?
Also, it seems to me it's better to have 2 counters, one for length(->donelist) and another for length(->curlist + ->nxtlist). I think we don't need force_quiescent_state() when all rcu callbacks are placed in ->donelist, we only need to increase rdp->blimit in this case.
Oleg.
Quoted 32 lines · Oleg Nesterov
Quoted 31 lines · Dipankar Sarma
(quoted diff) +20
-4+static void force_quiescent_state(struct rcu_data *rdp, + struct rcu_ctrlblk *rcp) +{ + int cpu; + cpumask_t cpumask; + set_need_resched(); + if (unlikely(rdp->qlen - rdp->last_rs_qlen > rsinterval)) { + rdp->last_rs_qlen = rdp->qlen; + /* + * Don't send IPI to itself. With irqs disabled, + * rdp->cpu is the current cpu. + */ + cpumask = rcp->cpumask; + cpu_clear(rdp->cpu, cpumask); + for_each_cpu_mask(cpu, cpumask) + smp_send_reschedule(cpu); + } +}@@ -92,17 +128,13 @@ void fastcall call_rcu(struct rcu_head * rdp = &__get_cpu_var(rcu_data); *rdp->nxttail = head; rdp->nxttail = &head->next; - - if (unlikely(++rdp->count > 10000)) - set_need_resched(); - + if (unlikely(++rdp->qlen > qhimark)) { + rdp->blimit = INT_MAX;
Quoted 8 lines
Review void fastcall call_rcu(struct rcu_head * · line 134
Quoted 2 lines
(quoted diff) +2
-0+ force_quiescent_state(rdp, &rcu_ctrlblk); + }When ->qlen exceeds qhimark for the first time we send reschedule IPI to other CPUs and
force_quiescent_state()records ->last_rs_qlen = ->qlen. But we don't reset ->last_rs_qlen when ->qlen goes to 0, this means that next time we need ++rdp->qlen > qhimark + rsinterval to force other CPUS to pass quiescent state, no?
Good catch -- this could well explain Lee's continuing to hit latency problems. Although this would not cause the first latency event, only subsequent ones, it seems to me that ->last_rs_qlen should be reset whenever ->blimit is reset.
Dipankar, thoughts?
Quoted 4 lines
Also, it seems to me it's better to have 2 counters, one for
length(->donelist)and another forlength(->curlist + ->nxtlist). I think we don't needforce_quiescent_state()when all rcu callbacks are placed in ->donelist, we only need to increaserdp->blimitin this case.
True, currently the patch keeps the sum of the length of all three lists, and takes both actions when the sum gets too large. But the only way you would get unneeded IPIs would be if callback processing was stalled, but callback generation and grace-period processing was still proceeding. Seems at first glance to be an unusual corner case, with the only downside being some extra IPIs. Or am I missing some aspect?
Thanx, Paul
Quoted 9 lines · Paul E. McKenney
Quoted 6 lines · Oleg Nesterov
When ->qlen exceeds qhimark for the first time we send reschedule IPI to other CPUs and
force_quiescent_state()records ->last_rs_qlen = ->qlen. But we don't reset ->last_rs_qlen when ->qlen goes to 0, this means that next time we need ++rdp->qlen > qhimark + rsinterval to force other CPUS to pass quiescent state, no?
Quoted 4 lines
Good catch -- this could well explain Lee's continuing to hit latency problems. Although this would not cause the first latency event, only subsequent ones, it seems to me that ->last_rs_qlen should be reset whenever ->blimit is reset.
May be it's better to do it in other way?
struct rcu_ctrlblk {
...
int signaled;
...
};void force_quiescent_state(rdp, rcp)
{
if (!rcp->signaled) {
// racy, but tolerable
rcp->signaled = 1;
for_each_cpu_mask(cpu, cpumask)
smp_send_reschedule(cpu);
}
}void rcu_start_batch(rcp, rdp)
{
if (->next_pending && ->completed == ->cur) {
...
rcp->signaled = 0;
...
}
}Probably it is also makes sense to tasklet_schedule(rcu_tasklet) in call_rcu() when ++rdp->qlen > qhimark, this way we can detect that we need to start the next batch earlier.
Quoted 5 lines
Quoted 4 lines
Also, it seems to me it's better to have 2 counters, one for
length(->donelist)and another forlength(->curlist + ->nxtlist). I think we don't needforce_quiescent_state()when all rcu callbacks are placed in ->donelist, we only need to increaserdp->blimitin this case.
Quoted 7 lines
True, currently the patch keeps the sum of the length of all three lists, and takes both actions when the sum gets too large. But the only way you would get unneeded IPIs would be if callback processing was stalled, but callback generation and grace-period processing was still proceeding. Seems at first glance to be an unusual corner case, with the only downside being some extra IPIs. Or am I missing some aspect?
Yes, it is probably not worth to complicate the code.
Oleg.
Quoted 34 lines · Oleg Nesterov
Quoted 9 lines · Paul E. McKenney
Quoted 6 lines · Oleg Nesterov
When ->qlen exceeds qhimark for the first time we send reschedule IPI to other CPUs and
force_quiescent_state()records ->last_rs_qlen = ->qlen. But we don't reset ->last_rs_qlen when ->qlen goes to 0, this means that next time we need ++rdp->qlen > qhimark + rsinterval to force other CPUS to pass quiescent state, no?Quoted 4 lines
Good catch -- this could well explain Lee's continuing to hit latency problems. Although this would not cause the first latency event, only subsequent ones, it seems to me that ->last_rs_qlen should be reset whenever ->blimit is reset.
May be it's better to do it in other way?
struct rcu_ctrlblk { ... int signaled; ... };void
force_quiescent_state(rdp, rcp){ if (!rcp->signaled) { // racy, but tolerable rcp->signaled = 1; for_each_cpu_mask(cpu, cpumask) smp_send_reschedule(cpu); } }
Quoted 8 lines
void
rcu_start_batch(rcp, rdp){ if (->next_pending && ->completed == ->cur) { ... rcp->signaled = 0; ... } }
Possibly... But the best thing would be for you and Dipankar to get together to work out the best strategy for this.
Thanx, Paul
Quoted 21 lines
Probably it is also makes sense to
tasklet_schedule(rcu_tasklet)incall_rcu()when ++rdp->qlen > qhimark, this way we can detect that we need to start the next batch earlier.Quoted 5 lines
Quoted 4 lines
Also, it seems to me it's better to have 2 counters, one for
length(->donelist)and another forlength(->curlist + ->nxtlist). I think we don't needforce_quiescent_state()when all rcu callbacks are placed in ->donelist, we only need to increaserdp->blimitin this case.Quoted 7 lines
True, currently the patch keeps the sum of the length of all three lists, and takes both actions when the sum gets too large. But the only way you would get unneeded IPIs would be if callback processing was stalled, but callback generation and grace-period processing was still proceeding. Seems at first glance to be an unusual corner case, with the only downside being some extra IPIs. Or am I missing some aspect?
Yes, it is probably not worth to complicate the code.
Oleg.
Quoted 12 lines · Oleg Nesterov
Quoted 11 lines · Dipankar Sarma
(quoted diff) +2
-4@@ -92,17 +128,13 @@ void fastcall call_rcu(struct rcu_head * rdp = &__get_cpu_var(rcu_data); *rdp->nxttail = head; rdp->nxttail = &head->next; - - if (unlikely(++rdp->count > 10000)) - set_need_resched(); - + if (unlikely(++rdp->qlen > qhimark)) { + rdp->blimit = INT_MAX;
Quoted 8 lines
Review void fastcall call_rcu(struct rcu_head * · line 134
Quoted 2 lines
(quoted diff) +2
-0+ force_quiescent_state(rdp, &rcu_ctrlblk); + }When ->qlen exceeds qhimark for the first time we send reschedule IPI to other CPUs and
force_quiescent_state()records ->last_rs_qlen = ->qlen. But we don't reset ->last_rs_qlen when ->qlen goes to 0, this means that next time we need ++rdp->qlen > qhimark + rsinterval to force other CPUS to pass quiescent state, no?
Yes. I have fixed it my code.
Quoted 5 lines
Also, it seems to me it's better to have 2 counters, one for
length(->donelist)and another forlength(->curlist + ->nxtlist). I think we don't needforce_quiescent_state()when all rcu callbacks are placed in ->donelist, we only need to increaserdp->blimitin this case.
We could, but I am not sure that ->qlen is not a good measure of grace period not happening. In fact, in the past, long donelists often resulted in longer grace periods. So, no harm in forcing reschedule and allowing ksoftirq to relinquish cpu.
Thanks
Dipankar