Skip to contentSkip to conversation

RCU fanout leaf balancing

DISCUSSION Daniel J Blueman 2 participants 3 messages Oct 25, 2014 – Oct 27, 2014

lore
Share & tools
lore
#0 lore raw

Hi Paul,

Finding earlier reference to increasing RCU fanout leaf for the purpose of "decrease[ing] cache-miss overhead for large systems", would your suggestion be to increase the value to the next hierarchy core-count above 16?

If we have say 32 interconnected 48-core servers; 3 sockets of dual-node 8-core Opteron 6300s, so 1536 cores in all. Latency across the coherent interconnect is O(100x) higher than the internal Hypertransport interconnect, so if we set RCU_FANOUT_LEAF to 48 to keep leaf-checking local to one Hypertransport fabric, what wisdom would one use for RCU_FANOUT? 4x leaf?

Or, would it be more cache-friendly to set RCU_FANOUT_LEAF to 8 and RCU_FANOUT to 48?

Many thanks,
Daniel

Signature · 2 lines
Daniel J Blueman
Principal Software Engineer, Numascale
#1 lore raw
Quoted 14 lines · Daniel J Blueman

Hi Paul,

Finding earlier reference to increasing RCU fanout leaf for the purpose of "decrease[ing] cache-miss overhead for large systems", would your suggestion be to increase the value to the next hierarchy core-count above 16?

If we have say 32 interconnected 48-core servers; 3 sockets of dual-node 8-core Opteron 6300s, so 1536 cores in all. Latency across the coherent interconnect is O(100x) higher than the internal Hypertransport interconnect, so if we set RCU_FANOUT_LEAF to 48 to keep leaf-checking local to one Hypertransport fabric, what wisdom would one use for RCU_FANOUT? 4x leaf?

Quoted 2 lines

Or, would it be more cache-friendly to set RCU_FANOUT_LEAF to 8 and RCU_FANOUT to 48?

The easiest approach would be to use the default of 16. Assuming consecutive CPU numbering within each 48-core server, this would mean that you would have three rcu_node structures per 48-core server. The next level up would of course span servers, but that level is accessed much less frequently than is the root level, so this should still work.

If you also have hyperthreading, so that there are 96 hardware threads per server, and if you are using the same "interesting" numbering scheme that Intel uses, then this still works. You have three leaf rcu_node structure for the first set of hardware threads and another set of three for the second set of hardware threads.

Or are you seeing some problem with the default? If so, please tell me what that problem is.

You can of course increase RCU_FANOUT to 24 or 48 (this latter assuming a 64-bit kernel), at least if you are using a recent kernel. However, the penalty for too large a value for RCU_FANOUT is lock contention at scheduling-clock-interrupt time. So if you are setting RCU_FANOUT to 48, you probably also want to boot with skew_tick set.

But the best approach is to try it. I bet that the default will work just fine for you. ;-)

Thanx, Paul

#2 lore raw
Quoted 33 lines · Paul E. McKenney
Quoted 14 lines · Daniel J Blueman

Hi Paul,

Finding earlier reference to increasing RCU fanout leaf for the purpose of "decrease[ing] cache-miss overhead for large systems", would your suggestion be to increase the value to the next hierarchy core-count above 16?

If we have say 32 interconnected 48-core servers; 3 sockets of dual-node 8-core Opteron 6300s, so 1536 cores in all. Latency across the coherent interconnect is O(100x) higher than the internal Hypertransport interconnect, so if we set RCU_FANOUT_LEAF to 48 to keep leaf-checking local to one Hypertransport fabric, what wisdom would one use for RCU_FANOUT? 4x leaf?

Quoted 2 lines

Or, would it be more cache-friendly to set RCU_FANOUT_LEAF to 8 and RCU_FANOUT to 48?

The easiest approach would be to use the default of 16. Assuming consecutive CPU numbering within each 48-core server, this would mean that you would have three rcu_node structures per 48-core server. The next level up would of course span servers, but that level is accessed much less frequently than is the root level, so this should still work.

If you also have hyperthreading, so that there are 96 hardware threads per server, and if you are using the same "interesting" numbering scheme that Intel uses, then this still works. You have three leaf rcu_node structure for the first set of hardware threads and another set of three for the second set of hardware threads.

Or are you seeing some problem with the default? If so, please tell me what that problem is.

Quoted 8 lines

You can of course increase RCU_FANOUT to 24 or 48 (this latter assuming a 64-bit kernel), at least if you are using a recent kernel. However, the penalty for too large a value for RCU_FANOUT is lock contention at scheduling-clock-interrupt time. So if you are setting RCU_FANOUT to 48, you probably also want to boot with skew_tick set.

But the best approach is to try it. I bet that the default will work just fine for you. ;-)

Good info. I'll stick with the defaults of 16/64, will schedule some tuning later, and let you know if I find anything significant.

Thanks Paul!

Daniel

Signature · 2 lines
Daniel J Blueman
Principal Software Engineer, Numascale