Skip to contentSkip to conversation

cacheline alignment and per-cpu data

DISCUSSION Chris Friesen 3 participants 3 messages Jun 9, 2006 – Jun 12, 2006

lore
Share & tools
lore
#0 lore raw

Someone asked me a question that I couldn't answer, so I thought I'd pass it on to here.

Suppose I declare an array of a struct type, where the size of the struct is not a multiple of the cacheline size. Each element in the array is used by a different cpu.

If I understand it, this would mean that the last member in the data belonging to one cpu shares a cacheline with the first member in the data belonging to the next cpu.

Will this cause cacheline pingpong? If I do this sort of thing do I need to ensure that the struct is a multiple of cacheline size (or specify cacheline alignement)?

Thanks,

Chris

#1 lore raw
Quoted 6 lines · Chris Friesen

Someone asked me a question that I couldn't answer, so I thought I'd pass it on to here.

Suppose I declare an array of a struct type, where the size of the struct is not a multiple of the cacheline size. Each element in the array is used by a different cpu.

If I understand it, this would mean that the last member in the data belonging to one cpu shares a cacheline with the first member in the data belonging to the next cpu.

Yes. Since an array element position is essentially base_ptr + sizeof(each_element) * index, if sizeof(each_element) is not a multiple of the cache line, the next element will start in the middle of a cache line.

Quoted 1 line

Will this cause cacheline pingpong? If I do this sort of thing do I need to ensure that the struct is a multiple of cacheline size (or specify cacheline alignement)?

Yes. If CPU 2 tries to write to struct member 1 of array element 2, and array element 1 is in CPU 1's cache, it must be invalidated.

GCC (and kernel macros) provide good support for enforced cacheline alignment, but it's also possible to pad your structures.

Quoted 3 lines

Thanks,

Chris

Chase

#2 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Chase" == Chase Venters <[email protected]> writes:

Quoted 3 lines · Chris Friesen
Quoted 3 lines

Will this cause cacheline pingpong? If I do this sort of thing do I need to ensure that the struct is a multiple of cacheline size (or specify cacheline alignement)?

Chase> Yes. If CPU 2 tries to write to struct member 1 of array Chase> element 2, and array element 1 is in CPU 1's cache, it must be Chase> invalidated.

Chase> GCC (and kernel macros) provide good support for enforced Chase> cacheline alignment, but it's also possible to pad your Chase> structures.

Best way to go is to use __cacheline_aligned rather than try to get the padding right manually. Someone will add a field somewhere in the middle and mess it up and nobody notices.

Cheers,
Jes