cacheline alignment and per-cpu data
Someone asked me a question that I couldn't answer, so I thought I'd pass it on to here.
Suppose I declare an array of a struct type, where the size of the struct is not a multiple of the cacheline size. Each element in the array is used by a different cpu.
If I understand it, this would mean that the last member in the data belonging to one cpu shares a cacheline with the first member in the data belonging to the next cpu.
Will this cause cacheline pingpong? If I do this sort of thing do I need to ensure that the struct is a multiple of cacheline size (or specify cacheline alignement)?
Thanks,
Chris
Quoted 6 lines · Chris Friesen
Someone asked me a question that I couldn't answer, so I thought I'd pass it on to here.
Suppose I declare an array of a struct type, where the size of the struct is not a multiple of the cacheline size. Each element in the array is used by a different cpu.
If I understand it, this would mean that the last member in the data belonging to one cpu shares a cacheline with the first member in the data belonging to the next cpu.
Yes. Since an array element position is essentially base_ptr + sizeof(each_element) * index, if sizeof(each_element) is not a multiple of the cache line, the next element will start in the middle of a cache line.
Quoted 1 line
Will this cause cacheline pingpong? If I do this sort of thing do I need to ensure that the struct is a multiple of cacheline size (or specify cacheline alignement)?
Yes. If CPU 2 tries to write to struct member 1 of array element 2, and array element 1 is in CPU 1's cache, it must be invalidated.
GCC (and kernel macros) provide good support for enforced cacheline alignment, but it's also possible to pad your structures.
Quoted 3 lines
Thanks,
Chris
Chase
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Chase" == Chase Venters <[email protected]> writes:
Quoted 3 lines · Chris Friesen
Quoted 3 lines
Will this cause cacheline pingpong? If I do this sort of thing do I need to ensure that the struct is a multiple of cacheline size (or specify cacheline alignement)?
Chase> Yes. If CPU 2 tries to write to struct member 1 of array Chase> element 2, and array element 1 is in CPU 1's cache, it must be Chase> invalidated.
Chase> GCC (and kernel macros) provide good support for enforced Chase> cacheline alignment, but it's also possible to pad your Chase> structures.
Best way to go is to use __cacheline_aligned rather than try to get the padding right manually. Someone will add a field somewhere in the middle and mess it up and nobody notices.
Cheers,
Jes