mm, slub: Use prefetchw instead of prefetch
Subsystems (2)
PATCH 1/1 mm, slub: Use prefetchw instead of prefetch
It's certain that an object will be not only read, but also written after allocation.
Use prefetchw instead of prefetchw. On supported architecture like x86, it helps to invalidate cache line when the object exists in other processors' cache.
--- mm/slub.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
mm/slub.c +3 -4
@@ -352,9 +352,9 @@ static inline void *get_freepointer(struct kmem_cache *s, void *object)return freelist_dereference (s, object + s->offset); } -static void prefetch_freepointer (const struct kmem_cache *s,void *object) +static void prefetchw_freepointer (const struct kmem_cache *s,void *object) { -prefetch (object + s->offset); +prefetchw (object + s->offset); }static inline void *get_freepointer_safe (struct kmem_cache *s,void *object)
@@ -3195,10 +3195,9 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,note_cmpxchg_failure ("slab_alloc" , s, tid); goto redo; } -prefetch_freepointer (s, next_object); +prefetchw_freepointer (s, next_object);stat (s,ALLOC_FASTPATH ); } -maybe_wipe_obj_freeptr (s, object); init =slab_want_init_on_alloc (gfpflags, s);
Signature · 1 line
2.27.0
David Rientjes
reviewerQuoted 3 lines · Hyeonggon Yoo
It's certain that an object will be not only read, but also written after allocation.
Why is it certain? I think perhaps what you meant to say is that if we are doing any prefetching here, then access will benefit from prefetchw instead of prefetch. But it's not "certain" that allocated memory will be accessed at all.
Quoted 1 line
Use prefetchw instead of prefetchw. On supported architecture
If we're using prefetchw instead of prefetchw, I think the diff would be 0 lines changed :)
Quoted 40 lines
like x86, it helps to invalidate cache line when the object exists in other processors' cache.
Signed-off-by Hyeonggon Yoo <[email protected]>--- mm/slub.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)mm/slub.c +3
-4@@ -352,9 +352,9 @@ static inline void *get_freepointer(struct kmem_cache *s, void *object)return freelist_dereference (s, object + s->offset); } -static void prefetch_freepointer (const struct kmem_cache *s,void *object) +static void prefetchw_freepointer (const struct kmem_cache *s,void *object) { -prefetch (object + s->offset); +prefetchw (object + s->offset); }static inline void *get_freepointer_safe (struct kmem_cache *s,void *object)@@ -3195,10 +3195,9 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,note_cmpxchg_failure ("slab_alloc" , s, tid); goto redo; } -prefetch_freepointer (s, next_object); +prefetchw_freepointer (s, next_object);stat (s,ALLOC_FASTPATH ); } -maybe_wipe_obj_freeptr (s, object); init =slab_want_init_on_alloc (gfpflags, s);Signature · 1 line
2.27.0
Quoted 10 lines · David Rientjes
Quoted 3 lines · Hyeonggon Yoo
It's certain that an object will be not only read, but also written after allocation.
Why is it certain? I think perhaps what you meant to say is that if we are doing any prefetching here, then access will benefit from prefetchw instead of prefetch. But it's not "certain" that allocated memory will be accessed at all.
I think the primary reason there's a prefetch is freelist traversal. The cacheline we prefetch will be read during the next allocation, so if we expect there to be one soon, prefetch might help. That the freepointer is part of object itself and thus the cache line will be probably accessed also after the allocation, is secondary. Yeah this might help some workloads, but perhaps hurt others - these things might look obvious in theory but be rather unpredictable in practice. At least some hackbench results would help...
Quoted 21 lines
Quoted 1 line
Use prefetchw instead of prefetchw. On supported architecture
If we're using prefetchw instead of prefetchw, I think the diff would be 0 lines changed :)
Quoted 16 lines
like x86, it helps to invalidate cache line when the object exists in other processors' cache.
Signed-off-by Hyeonggon Yoo <[email protected]>--- mm/slub.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)mm/slub.c +0
-0@@ -352,9 +352,9 @@ static inline void *get_freepointer(struct kmem_cache *s, void *object)return freelist_dereference (s, object + s->offset); }
Quoted 2 lines
mm/slub.c static inline void *get_freepointer(struct kmem_cache *s, void *object) · line 355
Quoted 2 lines
mm/slub.c +1
-1-static void prefetch_freepointer (const struct kmem_cache *s,void *object) +static void prefetchw_freepointer (const struct kmem_cache *s,void *object)
I wouldn't rename the function itself, unless we have both variants for different situations (we don't). That it uses prefetchw() is internal detail at this point.
Quoted 23 lines
Quoted 22 lines
mm/slub.c +2
-3{ -prefetch (object + s->offset); +prefetchw (object + s->offset); }static inline void *get_freepointer_safe (struct kmem_cache *s,void *object)@@ -3195,10 +3195,9 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,note_cmpxchg_failure ("slab_alloc" , s, tid); goto redo; } -prefetch_freepointer (s, next_object); +prefetchw_freepointer (s, next_object);stat (s,ALLOC_FASTPATH ); } -maybe_wipe_obj_freeptr (s, object); init =slab_want_init_on_alloc (gfpflags, s);Signature · 1 line
2.27.0
Perf and Hackbench results on my machine
Hello Vlastimil.
Quoted 7 lines · Vlastimil Babka
Quoted 6 lines · David Rientjes
Quoted 3 lines · Hyeonggon Yoo
It's certain that an object will be not only read, but also written after allocation.
Quoted 8 lines
Quoted 4 lines
Why is it certain? I think perhaps what you meant to say is that if we are doing any prefetching here, then access will benefit from prefetchw instead of prefetch. But it's not "certain" that allocated memory will be accessed at all.
I think the primary reason there's a prefetch is freelist traversal. The cacheline we prefetch will be read during the next allocation, so if we expect there to be one soon, prefetch might help.
I agree that.
Quoted 3 lines
That the freepointer is
part of object itself and thus the cache line will be probably accessed also after the allocation, is secondary.
Right. it depends on cache line size and whether first cache line of an object is frequently accessed or not.
Quoted 4 lines
Yeah this might help some workloads, but
perhaps hurt others - these things might look obvious in theory but be rather unpredictable in practice. At least some hackbench results would help...
Below is my measurement. it seems prefetch(w) is not making things worse at least on hackbench.
Measured on 16 CPUs (ARM64) / 16G RAM
Without prefetch:
Time: 91.989
Performance counter stats for 'hackbench -g 100 -l 10000':
1467926.03 msec cpu-clock # 15.907 CPUs utilized
17782076 context-switches # 12.114 K/sec
957523 cpu-migrations # 652.296 /sec
104561 page-faults # 71.230 /sec
1622117569931 cycles # 1.105 GHz (54.54%)
2002981132267 instructions # 1.23 insn per cycle (54.32%)
5600876429 branch-misses (54.28%)
642657442307 cache-references # 437.800 M/sec (54.27%)
19404890844 cache-misses # 3.019 % of all cache refs (54.28%)
640413686039 L1-dcache-loads # 436.271 M/sec (46.85%)
19110650580 L1-dcache-load-misses # 2.98% of all L1-dcache accesses (46.83%)
651556334841 dTLB-loads # 443.862 M/sec (46.63%)
3193647402 dTLB-load-misses # 0.49% of all dTLB cache accesses (46.84%)
538927659684 iTLB-loads # 367.135 M/sec (54.31%)
118503839 iTLB-load-misses # 0.02% of all iTLB cache accesses (54.35%)
625750168840 L1-icache-loads # 426.282 M/sec (46.80%)
24348083282 L1-icache-load-misses # 3.89% of all L1-icache accesses (46.78%) 92.284351157 seconds time elapsed
44.524693000 seconds user
1426.214006000 seconds sysWith prefetch:
Time: 91.677
Performance counter stats for 'hackbench -g 100 -l 10000':
1462938.07 msec cpu-clock # 15.908 CPUs utilized
18072550 context-switches # 12.354 K/sec
1018814 cpu-migrations # 696.416 /sec
104558 page-faults # 71.471 /sec
2003670016013 instructions # 1.27 insn per cycle (54.31%)
5702204863 branch-misses (54.28%)
643368500985 cache-references # 439.778 M/sec (54.26%)
18475582235 cache-misses # 2.872 % of all cache refs (54.28%)
642206796636 L1-dcache-loads # 438.984 M/sec (46.87%)
18215813147 L1-dcache-load-misses # 2.84% of all L1-dcache accesses (46.83%)
653842996501 dTLB-loads # 446.938 M/sec (46.63%)
3227179675 dTLB-load-misses # 0.49% of all dTLB cache accesses (46.85%)
537531951350 iTLB-loads # 367.433 M/sec (54.33%)
114750630 iTLB-load-misses # 0.02% of all iTLB cache accesses (54.37%)
630135543177 L1-icache-loads # 430.733 M/sec (46.80%)
22923237620 L1-icache-load-misses # 3.64% of all L1-icache accesses (46.76%) 91.964452802 seconds time elapsed
43.416742000 seconds user
1422.441123000 seconds sysWith prefetchw:
Time: 90.220
Performance counter stats for 'hackbench -g 100 -l 10000':
1437418.48 msec cpu-clock # 15.880 CPUs utilized
17694068 context-switches # 12.310 K/sec
958257 cpu-migrations # 666.651 /sec
100604 page-faults # 69.989 /sec
1583259429428 cycles # 1.101 GHz (54.57%)
2004002484935 instructions # 1.27 insn per cycle (54.37%)
5594202389 branch-misses (54.36%)
643113574524 cache-references # 447.409 M/sec (54.39%)
18233791870 cache-misses # 2.835 % of all cache refs (54.37%)
640205852062 L1-dcache-loads # 445.386 M/sec (46.75%)
17968160377 L1-dcache-load-misses # 2.81% of all L1-dcache accesses (46.79%)
651747432274 dTLB-loads # 453.415 M/sec (46.59%)
3127124271 dTLB-load-misses # 0.48% of all dTLB cache accesses (46.75%)
535395273064 iTLB-loads # 372.470 M/sec (54.38%)
113500056 iTLB-load-misses # 0.02% of all iTLB cache accesses (54.35%)
628871845924 L1-icache-loads # 437.501 M/sec (46.80%)
22585641203 L1-icache-load-misses # 3.59% of all L1-icache accesses (46.79%) 90.514819303 seconds time elapsed
43.877656000 seconds user
1397.176001000 seconds sysThanks,
Hyeonggon
Re: Perf and Hackbench results on my machine
Quoted 19 lines · Hyeonggon Yoo
Hello Vlastimil.
Quoted 7 lines · Vlastimil Babka
Quoted 6 lines · David Rientjes
Quoted 3 lines · Hyeonggon Yoo
It's certain that an object will be not only read, but also written after allocation.
Quoted 8 lines
Quoted 4 lines
Why is it certain? I think perhaps what you meant to say is that if we are doing any prefetching here, then access will benefit from prefetchw instead of prefetch. But it's not "certain" that allocated memory will be accessed at all.
I think the primary reason there's a prefetch is freelist traversal. The cacheline we prefetch will be read during the next allocation, so if we expect there to be one soon, prefetch might help.
Quoted 8 lines
I agree that.
Quoted 3 lines
That the freepointer is
part of object itself and thus the cache line will be probably accessed also after the allocation, is secondary.Right. it depends on cache line size and whether first cache line of an object is frequently accessed or not.
Not first cache line because free pointer is in the middle of object or out of object area. my mistake.
Quoted 21 lines
Quoted 21 lines
Quoted 1 line
Use prefetchw instead of prefetchw. On supported architecture
If we're using prefetchw instead of prefetchw, I think the diff would be 0 lines changed :)
Quoted 16 lines
like x86, it helps to invalidate cache line when the object exists in other processors' cache.
Signed-off-by Hyeonggon Yoo <[email protected]>--- mm/slub.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)mm/slub.c +0
-0@@ -352,9 +352,9 @@ static inline void *get_freepointer(struct kmem_cache *s, void *object)return freelist_dereference (s, object + s->offset); }
Quoted 6 lines
Quoted 2 lines
mm/slub.c static inline void *get_freepointer(struct kmem_cache *s, void *object) · line 355
Quoted 2 lines
mm/slub.c +1
-1-static void prefetch_freepointer (const struct kmem_cache *s,void *object) +static void prefetchw_freepointer (const struct kmem_cache *s,void *object)I wouldn't rename the function itself, unless we have both variants for different situations (we don't). That it uses
prefetchw()is internal detail at this point.
looks good. that is simpler.
Quoted 11 lines · David Rientjes
Quoted 3 lines · Hyeonggon Yoo
It's certain that an object will be not only read, but also written after allocation.
Why is it certain? I think perhaps what you meant to say is that if we are doing any prefetching here, then access will benefit from prefetchw instead of prefetch. But it's not "certain" that allocated memory will be accessed at all.
Blame my english skill :(
When I wrote I thought it was ok, but it was unclear. Thank you for pointing them!
What I meant was "When accessing an object, it must be written before read. So There's no situation that caller only reads an object and does not write. Thus it's better to use prefetchw instead of prefetch.".
Let's rephrase:
commit 0ad9500e16fe ("slub: prefetch next freelist pointer in slab_alloc()") introduced prefetch_freepointer() because when other cpu(s) freed objects into a page that current cpu owns, the freelist link is hot on cpu(s) which freed objects and possibly very cold on current cpu.
But if freelist link chain is hot on cpu(s) which freed objects,
it's better to invalidate that chain because they're not going to access again within a short time.
So use prefetchw instead of prefetch. On supported architectures like x86, it invalidates other copied instances of a cache line when prefetching it.
Quoted 5 lines
Quoted 1 line
Use prefetchw instead of prefetchw. On supported architecture
If we're using prefetchw instead of prefetchw, I think the diff would be 0 lines changed :)
That was my typo. thankfully Andrew fixed that.
Quoted 40 lines
Quoted 40 lines
like x86, it helps to invalidate cache line when the object exists in other processors' cache.
Signed-off-by Hyeonggon Yoo <[email protected]>--- mm/slub.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)mm/slub.c +3
-4@@ -352,9 +352,9 @@ static inline void *get_freepointer(struct kmem_cache *s, void *object)return freelist_dereference (s, object + s->offset); } -static void prefetch_freepointer (const struct kmem_cache *s,void *object) +static void prefetchw_freepointer (const struct kmem_cache *s,void *object) { -prefetch (object + s->offset); +prefetchw (object + s->offset); }static inline void *get_freepointer_safe (struct kmem_cache *s,void *object)@@ -3195,10 +3195,9 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,note_cmpxchg_failure ("slab_alloc" , s, tid); goto redo; } -prefetch_freepointer (s, next_object); +prefetchw_freepointer (s, next_object);stat (s,ALLOC_FASTPATH ); } -maybe_wipe_obj_freeptr (s, object); init =slab_want_init_on_alloc (gfpflags, s);Signature · 1 line
2.27.0
Quoted 2 lines · Hyeonggon Yoo
It's certain that an object will be not only read, but also written after allocation.
get_freepointer is used in multiple code path not only in allocation. It is for example used when scanning through a freelist.
With this change all objects get needlessly dirtied and the cross cpu cache contention is increased.
Quoted 10 lines · Christoph Lameter
Quoted 2 lines · Hyeonggon Yoo
It's certain that an object will be not only read, but also written after allocation.
get_freepointeris used in multiple code path not only in allocation. It is for example used when scanning through a freelist.With this change all objects get needlessly dirtied and the cross cpu cache contention is increased.
I didn't touch get_freepointer and there's
only one caller of prefetch_freepointer.
My change was not adding additional prefetch on get_freepointer, but changing existing prefetch into prefetchw.
The prefetch was introcued by commit 0ad9500e16fe ("slub: prefetch next freelist pointer in slab_alloc()") that you ACKed in 2011.
Do you think removeing existing prefetch is better than changing it from prefetch to prefetchw?