IPC locks breaking down with RCU
Quoted 3 lines · Hugh
I had toyed with the idea of never
freeing entries once allocated, which is a similarly simple solution;
Not possible, the structure size for ipc sem depends on the number of semaphores in the semaphore set.
Probably the best approach is to add a "deleted" flag into the ipc_id structure, and to check that flag after acquiring the spinlock. And perform the actual free operations for the ipc element in a rcu callback.
At which context do the rcu callbacks run? The semaphore sets are allocated with vmalloc for large sets, and that function is only permitted from process context, not from bh or irq context. According to a comment in rcupdate.c, rcu_process_callbacks runs in a tasklet, i.e. at bh context.
Quoted 4 lines
I'm happy to be overruled by someone who understands these cacheline bounce issues better than we do, but nobody else has spoken up yet.
Are there any good documents about cacheline bouncing, or rules how to reduce it?
For example, should a spinlock and the data it protects be in the same cacheline, or in different cachelines?
I guess that "same cacheline" means that only one cacheline is transfered if a cpu acquires the spinlock and touches the data. But OTHO a spinning cpu would probably force the cacheline into shared state, and that'll slow down the data access for the cpu that owns the spinlock.
--
Manfred
Quoted 4 lines · Manfred Spraul
Probably the best approach is to add a "deleted" flag into the
ipc_idstructure, and to check that flag after acquiring the spinlock. And perform the actual free operations for the ipc element in a rcu callback.
Yes, that's what I was proposing.
Quoted 5 lines
At which context do the rcu callbacks run? The semaphore sets are allocated with vmalloc for large sets, and that function is only permitted from process context, not from bh or irq context. According to a comment in
rcupdate.c,rcu_process_callbacksruns in a tasklet, i.e. at bh context.
Hah! Very good point. Seems we need to think again.
Quoted 7 lines
For example, should a spinlock and the data it protects be in the same cacheline, or in different cachelines?
I guess that "same cacheline" means that only one cacheline is transfered if a cpu acquires the spinlock and touches the data. But OTHO a spinning cpu would probably force the cacheline into shared state, and that'll slow down the data access for the cpu that owns the spinlock.
Well, yes, but that's not the issue as I understand it. There you're thinking of contention on the spinlock, and its effect spreading to the data protected by that lock. Whereas the more pernicious effect of cacheline bouncing is when there is no particular contention on a spinlock, but data is (mis)distributed in such a way that mods to the same cacheline are likely to occur from different cpus at at about the same time.
In the original design, Mingming nicely split up the locks (greatly reducing contention), but had them in an array (causing lots of bounce, I believe): I'm resisting a return to that design.
Hugh
Quoted 4 lines · Hugh Dickins
In the original design, Mingming nicely split up the locks (greatly reducing contention), but had them in an array (causing lots of bounce, I believe):
I am not an expert of cacheline bouncing, so please point me if I miss something. I wonder if we could reduce the bounce even with current design (the spinlock is in the data it protects). We have to go through that array anyway to get access to the data (and the spinlock).
Quoted 1 line
I'm resisting a return to that design.
OK, I will back to original design.