A question on RCU vs. preempt-RCU
Hello, guys.
Kent recently implemented a generic percpu reference counter. It's scheduled to be merged in the coming merge window and some part of cgroup refcnting is already converted to it.
https://git.kernel.org/cgit/linux/kernel/git/tj/percpu.git/tree/lib/percpu-refcount.c?h=for-3.11
It's essentially a generalized form of module refcnting but uses regular RCU instead of toggling preemption for local atomicity.
I've been running some performance tests with different preemption levels and, with CONFIG_PREEMPT, the percpu ref could be slower by around 10% or at the worst contrived case maybe even close to 20% when compared to simple atomic_t on a single CPU (when hit by multiple CPUs concurrently, it of course destroys atomic_t). Most of the slow down seems to come from the preempt tree RCU calls and there no longer seems to be a way to opt out of that RCU implementation when CONFIG_PREEMPT.
For most use cases, the trade-off should be fine. With any kind of cross-cpu traffic, which there usually will be, it should be an easy win for the percpu-refcount even when CONFIG_PREEMPT; however, I've been looking to replace the module ref with the generic one and the performance degradation there has low but existing possibility of being noticeable in some edge use cases.
We can convert the percpu-refcount to use preempt_disable/enable() paired with call_rcu_sched() but IIUC that would have latency implications from the callback processing side, right? Given that module ref killing would be very low-frequency, it shouldn't contribute significant amount of callbacks but I'd like to avoid providing two separate implementations if at all possible.
So, what would be the right thing to do here? How bad would converting percpu-refcount to sched-RCU by default be? Would the extra overhead on module ref be acceptable when CONFIG_PREEMPT? What do you guys think?
Thanks!
Signature · 1 line
tejun
Quoted 13 lines · Tejun Heo
Hello, guys.
Kent recently implemented a generic percpu reference counter. It's scheduled to be merged in the coming merge window and some part of cgroup refcnting is already converted to it.
https://git.kernel.org/cgit/linux/kernel/git/tj/percpu.git/tree/lib/percpu-refcount.c?h=for-3.11
It's essentially a generalized form of module refcnting but uses regular RCU instead of toggling preemption for local atomicity.
Quoted 8 lines
I've been running some performance tests with different preemption levels and, with
CONFIG_PREEMPT, the percpu ref could be slower by around 10% or at the worst contrived case maybe even close to 20% when compared to simpleatomic_ton a single CPU (when hit by multiple CPUs concurrently, it of course destroysatomic_t). Most of the slow down seems to come from the preempt tree RCU calls and there no longer seems to be a way to opt out of that RCU implementation whenCONFIG_PREEMPT.
CONFIG_TREE_PREEMPT_RCU does have an increment, decrement (sort of), and check in its rcu_read_lock() and rcu_read_unlock(), which will add overhead that might well be noticeable compared to CONFIG_TREE_RCU's zero-code implementation of rcu_read_lock() and rcu_read_unlock().
Quoted 7 lines
For most use cases, the trade-off should be fine. With any kind of cross-cpu traffic, which there usually will be, it should be an easy win for the percpu-refcount even when
CONFIG_PREEMPT; however, I've been looking to replace the module ref with the generic one and the performance degradation there has low but existing possibility of being noticeable in some edge use cases.
Quoted 6 lines
We can convert the percpu-refcount to use preempt_disable/enable() paired with
call_rcu_sched()but IIUC that would have latency implications from the callback processing side, right? Given that module ref killing would be very low-frequency, it shouldn't contribute significant amount of callbacks but I'd like to avoid providing two separate implementations if at all possible.
The main source of longer latency from preempt_disable/enable() (or rcu_read_{,un}lock_sched()) will be on the read side. The callback-processing is very nearly identical.
Quoted 4 lines
So, what would be the right thing to do here? How bad would converting percpu-refcount to sched-RCU by default be? Would the extra overhead on module ref be acceptable when
CONFIG_PREEMPT? What do you guys think?
The big question is "how long are the RCU read-side critical sections?" My guess is that module references can have arbitrarily long lifetimes, which would argue strongly against use of RCU-sched. But if the lifetimes are always short (say, sub-microsecond), then RCU-sched should be fine.
Thanx, Paul
Hello, Paul.
Quoted 4 lines · Paul E. McKenney
CONFIG_TREE_PREEMPT_RCUdoes have an increment, decrement (sort of), and check in itsrcu_read_lock()andrcu_read_unlock(), which will add overhead that might well be noticeable compared to CONFIG_TREE_RCU's zero-code implementation ofrcu_read_lock()andrcu_read_unlock().
Yeah, I should have added one more data point. I was testing atomic_t vs. percpu-ref and saw the overhead and worrying that it would show regression against preempt_disable/enable() implementation.
Just ran some tests and preempt_disable/enable() based implementation is about 18% faster than rcu_read_lock/unlock() based one.
Compared to atomic_t, in a horribly contrived test case, normal RCU would be slower by around 20% while the preemption one would be slower by 7.5%.
Quoted 3 lines
The main source of longer latency from preempt_disable/enable() (or rcu_read_{,un}lock_sched()) will be on the read side. The callback-processing is very nearly identical.
Ah, right. I was completely confused there. The goal of CONFIG_TREE_PREEMPT_RCU is to allow preemption inside RCU read critical sections. I knew that at one point and completely forgot about it, so using preemption based one is fine as long as the length of critical section is short.
Quoted 1 line
The big question is "how long are the RCU read-side critical sections?"
Extremely short. It's gonna be like five instructions.
Quoted 1 line
My guess is that module references can have arbitrarily long lifetimes,
Preemption is disabled only while the refcnt operations are actually going on.
Quoted 2 lines
which would argue strongly against use of RCU-sched. But if the lifetimes are always short (say, sub-microsecond), then RCU-sched should be fine.
So, RCU-sched, it is.
Thanks a lot for the help!
Signature · 1 line
tejun
Quoted 9 lines · Tejun Heo
I've been running some performance tests with different preemption levels and, with
CONFIG_PREEMPT, the percpu ref could be slower by around 10% or at the worst contrived case maybe even close to 20% when compared to simpleatomic_ton a single CPU (when hit by multiple CPUs concurrently, it of course destroysatomic_t). Most of the slow down seems to come from the preempt tree RCU calls and there no longer seems to be a way to opt out of that RCU implementation whenCONFIG_PREEMPT.
Quoted 6 lines
For most use cases, the trade-off should be fine. With any kind of cross-cpu traffic, which there usually will be, it should be an easy win for the percpu-refcount even when
CONFIG_PREEMPT; however, I've been looking to replace the module ref with the generic one and the performance degradation there has low but existing possibility of being noticeable in some edge use cases.
I'm confused: is it actually 10% slower than the existing module refcount code, or 10% slower than atomic inc?
Quoted 11 lines
We can convert the percpu-refcount to use preempt_disable/enable() paired with
call_rcu_sched()but IIUC that would have latency implications from the callback processing side, right? Given that module ref killing would be very low-frequency, it shouldn't contribute significant amount of callbacks but I'd like to avoid providing two separate implementations if at all possible.So, what would be the right thing to do here? How bad would converting percpu-refcount to sched-RCU by default be? Would the extra overhead on module ref be acceptable when
CONFIG_PREEMPT? What do you guys think?
CONFIG_PREEMPT, now with more preempt! Sure, that has a cost, but you're arguably fixing a bug.
If we want to improve CONFIG_PREEMPT performance, we can probably use a trick I wanted to try long ago:
1) Use a per-cpu counter rather than a per-task counter for preempt.
2) Lay out preempt_counter so it covers NR_CPU pages, one per page.
3) When you want to preempt a CPU and counter isn't zero, make the page RO.
4) Handle preemption enable in the fault handler.
Then there's no branch in preempt_enable().
At a glance, the same trick could apply to t->rcu_read_unlock_special, but I'd have to offload that to my RCU coprocessor. Paul? :)
Cheers,
Rusty.
Hello, Rusty.
Quoted 9 lines · Rusty Russell
Quoted 6 lines
For most use cases, the trade-off should be fine. With any kind of cross-cpu traffic, which there usually will be, it should be an easy win for the percpu-refcount even when
CONFIG_PREEMPT; however, I've been looking to replace the module ref with the generic one and the performance degradation there has low but existing possibility of being noticeable in some edge use cases.I'm confused: is it actually 10% slower than the existing module refcount code, or 10% slower than atomic inc?
Heh, sorry about the confusion. I was comparing percpu_ref to atomic_t and then worrying about the rcu flipping overhead as it definitely seemed higher than flipping preemption. As I wrote in a reply to Paul, if I compare perpcu-ref with normal RCU against RCU-sched, the performance difference is around 18% in favor of RCU-sched.
Quoted 2 lines
CONFIG_PREEMPT, now with more preempt! Sure, that has a cost, but you're arguably fixing a bug.
It seems that using RCU-sched is the right flavor for perpcu_ref. In theory, we shouldn't see any performance degradation when converting module ref to percpu_ref.
Quoted 2 lines
If we want to improve
CONFIG_PREEMPTperformance, we can probably use a trick I wanted to try long ago:
So, this is a slight digression.
Quoted 6 lines
1) Use a per-cpu counter rather than a per-task counter for preempt.
2) Lay outpreempt_counterso it coversNR_CPUpages, one per page.
3) When you want to preempt a CPU and counter isn't zero, make the page RO.
4) Handle preemption enable in the fault handler.Then there's no branch in
preempt_enable().
Buth yeah, interesting trick. We'll be doing IPIs, flushing TLB and taking faults until it hits zero. It'll all depend on the frequency of preemption but given that branches don't tend to be too expensive on modern processors, maybe it'd be a bit too hairy for possibly marginal gain?
Thanks.
Signature · 1 line
tejun
Quoted 5 lines · Tejun Heo
Buth yeah, interesting trick. We'll be doing IPIs, flushing TLB and taking faults until it hits zero. It'll all depend on the frequency of preemption but given that branches don't tend to be too expensive on modern processors, maybe it'd be a bit too hairy for possibly marginal gain?
Yeah, I'm not convinced either, but I am hoping some enthusiast will run with the idea in hope of Fame and Glory :)
Cheers,
Rusty.
Quoted 9 lines · Rusty Russell
Quoted 5 lines · Tejun Heo
Buth yeah, interesting trick. We'll be doing IPIs, flushing TLB and taking faults until it hits zero. It'll all depend on the frequency of preemption but given that branches don't tend to be too expensive on modern processors, maybe it'd be a bit too hairy for possibly marginal gain?
Yeah, I'm not convinced either, but I am hoping some enthusiast will run with the idea in hope of Fame and Glory :)
This actually looks like a trick I think would be fun to implement. But as it being beneficial... the preempt count is used quite often, the fact that this page will be constantly in the TLB would have a bigger impact on general performance than saving the lousy branch.
-- Steve