Skip to contentSkip to conversation

mm, slub: Use prefetchw instead of prefetch

PATCH Superseded Hyeonggon Yoo 4 participants 8 messages Oct 8, 2021 – Oct 11, 2021

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm, slub: Use prefetchw instead of prefetch

It's certain that an object will be not only read, but also written after allocation.

Use prefetchw instead of prefetchw. On supported architecture like x86, it helps to invalidate cache line when the object exists in other processors' cache.

---
 mm/slub.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
mm/slub.c +3 -4
index 3d2025f7163b..2aca7523165e 100644
@@ -352,9 +352,9 @@ static inline void *get_freepointer(struct kmem_cache *s, void *object)
 	return freelist_dereference(s, object + s->offset);
 }
 
-static void prefetch_freepointer(const struct kmem_cache *s, void *object)
+static void prefetchw_freepointer(const struct kmem_cache *s, void *object)
 {
-	prefetch(object + s->offset);
+	prefetchw(object + s->offset);
 }
 
 static inline void *get_freepointer_safe(struct kmem_cache *s, void *object)
@@ -3195,10 +3195,9 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,
 			note_cmpxchg_failure("slab_alloc", s, tid);
 			goto redo;
 		}
-		prefetch_freepointer(s, next_object);
+		prefetchw_freepointer(s, next_object);
 		stat(s, ALLOC_FASTPATH);
 	}
-
 	maybe_wipe_obj_freeptr(s, object);
 	init = slab_want_init_on_alloc(gfpflags, s);
 
Signature · 1 line
2.27.0
#1 lore raw
Quoted 3 lines · Hyeonggon Yoo

It's certain that an object will be not only read, but also written after allocation.

Why is it certain? I think perhaps what you meant to say is that if we are doing any prefetching here, then access will benefit from prefetchw instead of prefetch. But it's not "certain" that allocated memory will be accessed at all.

Quoted 1 line

Use prefetchw instead of prefetchw. On supported architecture

If we're using prefetchw instead of prefetchw, I think the diff would be 0 lines changed :)

Quoted 40 lines

like x86, it helps to invalidate cache line when the object exists in other processors' cache.

---
 mm/slub.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
mm/slub.c +3 -4
index 3d2025f7163b..2aca7523165e 100644
@@ -352,9 +352,9 @@ static inline void *get_freepointer(struct kmem_cache *s, void *object)
 	return freelist_dereference(s, object + s->offset);
 }
 
-static void prefetch_freepointer(const struct kmem_cache *s, void *object)
+static void prefetchw_freepointer(const struct kmem_cache *s, void *object)
 {
-	prefetch(object + s->offset);
+	prefetchw(object + s->offset);
 }
 
 static inline void *get_freepointer_safe(struct kmem_cache *s, void *object)
@@ -3195,10 +3195,9 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,
 			note_cmpxchg_failure("slab_alloc", s, tid);
 			goto redo;
 		}
-		prefetch_freepointer(s, next_object);
+		prefetchw_freepointer(s, next_object);
 		stat(s, ALLOC_FASTPATH);
 	}
-
 	maybe_wipe_obj_freeptr(s, object);
 	init = slab_want_init_on_alloc(gfpflags, s);
 
Signature · 1 line
2.27.0
#2 lore raw
Quoted 10 lines · David Rientjes
Quoted 3 lines · Hyeonggon Yoo

It's certain that an object will be not only read, but also written after allocation.

Why is it certain? I think perhaps what you meant to say is that if we are doing any prefetching here, then access will benefit from prefetchw instead of prefetch. But it's not "certain" that allocated memory will be accessed at all.

I think the primary reason there's a prefetch is freelist traversal. The cacheline we prefetch will be read during the next allocation, so if we expect there to be one soon, prefetch might help. That the freepointer is part of object itself and thus the cache line will be probably accessed also after the allocation, is secondary. Yeah this might help some workloads, but perhaps hurt others - these things might look obvious in theory but be rather unpredictable in practice. At least some hackbench results would help...

Quoted 21 lines
Quoted 1 line

Use prefetchw instead of prefetchw. On supported architecture

If we're using prefetchw instead of prefetchw, I think the diff would be 0 lines changed :)

Quoted 16 lines

like x86, it helps to invalidate cache line when the object exists in other processors' cache.

---
 mm/slub.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
mm/slub.c +0 -0
index 3d2025f7163b..2aca7523165e 100644
@@ -352,9 +352,9 @@ static inline void *get_freepointer(struct kmem_cache *s, void *object)
 	return freelist_dereference(s, object + s->offset);
 }
 
Quoted 2 lines

mm/slub.c static inline void *get_freepointer(struct kmem_cache *s, void *object) · line 355

Quoted 2 lines
mm/slub.c +1 -1
-static void prefetch_freepointer(const struct kmem_cache *s, void *object)
+static void prefetchw_freepointer(const struct kmem_cache *s, void *object)

I wouldn't rename the function itself, unless we have both variants for different situations (we don't). That it uses prefetchw() is internal detail at this point.

Quoted 23 lines
Quoted 22 lines
mm/slub.c +2 -3
 {
-	prefetch(object + s->offset);
+	prefetchw(object + s->offset);
 }
 
 static inline void *get_freepointer_safe(struct kmem_cache *s, void *object)
@@ -3195,10 +3195,9 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,
 			note_cmpxchg_failure("slab_alloc", s, tid);
 			goto redo;
 		}
-		prefetch_freepointer(s, next_object);
+		prefetchw_freepointer(s, next_object);
 		stat(s, ALLOC_FASTPATH);
 	}
-
 	maybe_wipe_obj_freeptr(s, object);
 	init = slab_want_init_on_alloc(gfpflags, s);
 
Signature · 1 line
2.27.0
#3 lore raw

Perf and Hackbench results on my machine

Hello Vlastimil.

Quoted 7 lines · Vlastimil Babka
Quoted 6 lines · David Rientjes
Quoted 3 lines · Hyeonggon Yoo

It's certain that an object will be not only read, but also written after allocation.

Quoted 8 lines
Quoted 4 lines

Why is it certain? I think perhaps what you meant to say is that if we are doing any prefetching here, then access will benefit from prefetchw instead of prefetch. But it's not "certain" that allocated memory will be accessed at all.

I think the primary reason there's a prefetch is freelist traversal. The cacheline we prefetch will be read during the next allocation, so if we expect there to be one soon, prefetch might help.

I agree that.

Quoted 3 lines

That the freepointer is
part of object itself and thus the cache line will be probably accessed also after the allocation, is secondary.

Right. it depends on cache line size and whether first cache line of an object is frequently accessed or not.

Quoted 4 lines

Yeah this might help some workloads, but
perhaps hurt others - these things might look obvious in theory but be rather unpredictable in practice. At least some hackbench results would help...

Below is my measurement. it seems prefetch(w) is not making things worse at least on hackbench.

Measured on 16 CPUs (ARM64) / 16G RAM
Without prefetch:

Time: 91.989
Performance counter stats for 'hackbench -g 100 -l 10000':

        1467926.03 msec cpu-clock                 #   15.907 CPUs utilized          
          17782076      context-switches          #   12.114 K/sec                  
            957523      cpu-migrations            #  652.296 /sec                   
            104561      page-faults               #   71.230 /sec                   
     1622117569931      cycles                    #    1.105 GHz                      (54.54%)
     2002981132267      instructions              #    1.23  insn per cycle           (54.32%)
        5600876429      branch-misses                                                 (54.28%)
      642657442307      cache-references          #  437.800 M/sec                    (54.27%)
       19404890844      cache-misses              #    3.019 % of all cache refs      (54.28%)
      640413686039      L1-dcache-loads           #  436.271 M/sec                    (46.85%)
       19110650580      L1-dcache-load-misses     #    2.98% of all L1-dcache accesses  (46.83%)
      651556334841      dTLB-loads                #  443.862 M/sec                    (46.63%)
        3193647402      dTLB-load-misses          #    0.49% of all dTLB cache accesses  (46.84%)
      538927659684      iTLB-loads                #  367.135 M/sec                    (54.31%)
         118503839      iTLB-load-misses          #    0.02% of all iTLB cache accesses  (54.35%)
      625750168840      L1-icache-loads           #  426.282 M/sec                    (46.80%)
       24348083282      L1-icache-load-misses     #    3.89% of all L1-icache accesses  (46.78%)
      92.284351157 seconds time elapsed

      44.524693000 seconds user
    1426.214006000 seconds sys

With prefetch:

Time: 91.677

Performance counter stats for 'hackbench -g 100 -l 10000':

        1462938.07 msec cpu-clock                 #   15.908 CPUs utilized          
          18072550      context-switches          #   12.354 K/sec                  
           1018814      cpu-migrations            #  696.416 /sec                   
            104558      page-faults               #   71.471 /sec                   
     2003670016013      instructions              #    1.27  insn per cycle           (54.31%)
        5702204863      branch-misses                                                 (54.28%)
      643368500985      cache-references          #  439.778 M/sec                    (54.26%)
       18475582235      cache-misses              #    2.872 % of all cache refs      (54.28%)
      642206796636      L1-dcache-loads           #  438.984 M/sec                    (46.87%)
       18215813147      L1-dcache-load-misses     #    2.84% of all L1-dcache accesses  (46.83%)
      653842996501      dTLB-loads                #  446.938 M/sec                    (46.63%)
        3227179675      dTLB-load-misses          #    0.49% of all dTLB cache accesses  (46.85%)
      537531951350      iTLB-loads                #  367.433 M/sec                    (54.33%)
         114750630      iTLB-load-misses          #    0.02% of all iTLB cache accesses  (54.37%)
      630135543177      L1-icache-loads           #  430.733 M/sec                    (46.80%)
       22923237620      L1-icache-load-misses     #    3.64% of all L1-icache accesses  (46.76%)
      91.964452802 seconds time elapsed

      43.416742000 seconds user
    1422.441123000 seconds sys

With prefetchw:

Time: 90.220

Performance counter stats for 'hackbench -g 100 -l 10000':

        1437418.48 msec cpu-clock                 #   15.880 CPUs utilized          
          17694068      context-switches          #   12.310 K/sec                  
            958257      cpu-migrations            #  666.651 /sec                   
            100604      page-faults               #   69.989 /sec                   
     1583259429428      cycles                    #    1.101 GHz                      (54.57%)
     2004002484935      instructions              #    1.27  insn per cycle           (54.37%)
        5594202389      branch-misses                                                 (54.36%)
      643113574524      cache-references          #  447.409 M/sec                    (54.39%)
       18233791870      cache-misses              #    2.835 % of all cache refs      (54.37%)
      640205852062      L1-dcache-loads           #  445.386 M/sec                    (46.75%)
       17968160377      L1-dcache-load-misses     #    2.81% of all L1-dcache accesses  (46.79%)
      651747432274      dTLB-loads                #  453.415 M/sec                    (46.59%)
        3127124271      dTLB-load-misses          #    0.48% of all dTLB cache accesses  (46.75%)
      535395273064      iTLB-loads                #  372.470 M/sec                    (54.38%)
         113500056      iTLB-load-misses          #    0.02% of all iTLB cache accesses  (54.35%)
      628871845924      L1-icache-loads           #  437.501 M/sec                    (46.80%)
       22585641203      L1-icache-load-misses     #    3.59% of all L1-icache accesses  (46.79%)
      90.514819303 seconds time elapsed
 
      43.877656000 seconds user
    1397.176001000 seconds sys

Thanks,
Hyeonggon

#4 lore raw

Re: Perf and Hackbench results on my machine

Quoted 19 lines · Hyeonggon Yoo

Hello Vlastimil.

Quoted 7 lines · Vlastimil Babka
Quoted 6 lines · David Rientjes
Quoted 3 lines · Hyeonggon Yoo

It's certain that an object will be not only read, but also written after allocation.

Quoted 8 lines
Quoted 4 lines

Why is it certain? I think perhaps what you meant to say is that if we are doing any prefetching here, then access will benefit from prefetchw instead of prefetch. But it's not "certain" that allocated memory will be accessed at all.

I think the primary reason there's a prefetch is freelist traversal. The cacheline we prefetch will be read during the next allocation, so if we expect there to be one soon, prefetch might help.

Quoted 8 lines

I agree that.

Quoted 3 lines

That the freepointer is
part of object itself and thus the cache line will be probably accessed also after the allocation, is secondary.

Right. it depends on cache line size and whether first cache line of an object is frequently accessed or not.

Not first cache line because free pointer is in the middle of object or out of object area. my mistake.

Quoted 21 lines
Quoted 21 lines
Quoted 1 line

Use prefetchw instead of prefetchw. On supported architecture

If we're using prefetchw instead of prefetchw, I think the diff would be 0 lines changed :)

Quoted 16 lines

like x86, it helps to invalidate cache line when the object exists in other processors' cache.

---
 mm/slub.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
mm/slub.c +0 -0
index 3d2025f7163b..2aca7523165e 100644
@@ -352,9 +352,9 @@ static inline void *get_freepointer(struct kmem_cache *s, void *object)
 	return freelist_dereference(s, object + s->offset);
 }
 
Quoted 6 lines
Quoted 2 lines

mm/slub.c static inline void *get_freepointer(struct kmem_cache *s, void *object) · line 355

Quoted 2 lines
mm/slub.c +1 -1
-static void prefetch_freepointer(const struct kmem_cache *s, void *object)
+static void prefetchw_freepointer(const struct kmem_cache *s, void *object)

I wouldn't rename the function itself, unless we have both variants for different situations (we don't). That it uses prefetchw() is internal detail at this point.

looks good. that is simpler.

#5 lore raw
In reply to David Rientjes
Quoted 11 lines · David Rientjes
Quoted 3 lines · Hyeonggon Yoo

It's certain that an object will be not only read, but also written after allocation.

Why is it certain? I think perhaps what you meant to say is that if we are doing any prefetching here, then access will benefit from prefetchw instead of prefetch. But it's not "certain" that allocated memory will be accessed at all.

Blame my english skill :(

When I wrote I thought it was ok, but it was unclear. Thank you for pointing them!

What I meant was "When accessing an object, it must be written before read. So There's no situation that caller only reads an object and does not write. Thus it's better to use prefetchw instead of prefetch.".

Let's rephrase:

commit 0ad9500e16fe ("slub: prefetch next freelist pointer in slab_alloc()") introduced prefetch_freepointer() because when other cpu(s) freed objects into a page that current cpu owns, the freelist link is hot on cpu(s) which freed objects and possibly very cold on current cpu.

But if freelist link chain is hot on cpu(s) which freed objects,
it's better to invalidate that chain because they're not going to access again within a short time.

So use prefetchw instead of prefetch. On supported architectures like x86, it invalidates other copied instances of a cache line when prefetching it.

Quoted 5 lines
Quoted 1 line

Use prefetchw instead of prefetchw. On supported architecture

If we're using prefetchw instead of prefetchw, I think the diff would be 0 lines changed :)

That was my typo. thankfully Andrew fixed that.

Quoted 40 lines
Quoted 40 lines

like x86, it helps to invalidate cache line when the object exists in other processors' cache.

---
 mm/slub.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
mm/slub.c +3 -4
index 3d2025f7163b..2aca7523165e 100644
@@ -352,9 +352,9 @@ static inline void *get_freepointer(struct kmem_cache *s, void *object)
 	return freelist_dereference(s, object + s->offset);
 }
 
-static void prefetch_freepointer(const struct kmem_cache *s, void *object)
+static void prefetchw_freepointer(const struct kmem_cache *s, void *object)
 {
-	prefetch(object + s->offset);
+	prefetchw(object + s->offset);
 }
 
 static inline void *get_freepointer_safe(struct kmem_cache *s, void *object)
@@ -3195,10 +3195,9 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,
 			note_cmpxchg_failure("slab_alloc", s, tid);
 			goto redo;
 		}
-		prefetch_freepointer(s, next_object);
+		prefetchw_freepointer(s, next_object);
 		stat(s, ALLOC_FASTPATH);
 	}
-
 	maybe_wipe_obj_freeptr(s, object);
 	init = slab_want_init_on_alloc(gfpflags, s);
 
Signature · 1 line
2.27.0
#6 lore raw
In reply to Hyeonggon Yoo
Quoted 2 lines · Hyeonggon Yoo

It's certain that an object will be not only read, but also written after allocation.

get_freepointer is used in multiple code path not only in allocation. It is for example used when scanning through a freelist.

With this change all objects get needlessly dirtied and the cross cpu cache contention is increased.

#7 lore raw
Quoted 10 lines · Christoph Lameter
Quoted 2 lines · Hyeonggon Yoo

It's certain that an object will be not only read, but also written after allocation.

get_freepointer is used in multiple code path not only in allocation. It is for example used when scanning through a freelist.

With this change all objects get needlessly dirtied and the cross cpu cache contention is increased.

I didn't touch get_freepointer and there's
only one caller of prefetch_freepointer.

My change was not adding additional prefetch on get_freepointer, but changing existing prefetch into prefetchw.

The prefetch was introcued by commit 0ad9500e16fe ("slub: prefetch next freelist pointer in slab_alloc()") that you ACKed in 2011.

Do you think removeing existing prefetch is better than changing it from prefetch to prefetchw?