Skip to contentSkip to conversation

mm/slub: add missing TID updates on slab deactivation

PATCH In review Jann Horn 6 participants 8 messages Jun 8, 2022 – Jun 15, 2022

Subsystems (2)
lore
Share & tools
lore

Jann Horn

author
#0 lore raw

PATCH 1/1 mm/slub: add missing TID updates on slab deactivation

The fastpath in slab_alloc_node() assumes that c->slab is stable as long as the TID stays the same. However, two places in __slab_alloc() currently don't update the TID when deactivating the CPU slab.

If multiple operations race the right way, this could lead to an object getting lost; or, in an even more unlikely situation, it could even lead to an object being freed onto the wrong slab's freelist, messing up the inuse counter and eventually causing a page to be freed to the page allocator while it still contains slab objects.

(I haven't actually tested these cases though, this is just based on looking at the code. Writing testcases for this stuff seems like it'd be a pain...)

The race leading to state inconsistency is (all operations on the same CPU and kmem_cache):

- task A: begin do_slab_free():
- read TID
- read pcpu freelist (==NULL)
- check slab == c->slab (true)
- [PREEMPT A->B]
- task B: begin slab_alloc_node():
- fastpath fails (c->freelist is NULL)
- enter __slab_alloc()
- slub_get_cpu_ptr() (disables preemption)
- enter ___slab_alloc()
- take local_lock_irqsave()
- read c->freelist as NULL
- get_freelist() returns NULL
- write c->slab = NULL
- drop local_unlock_irqrestore()
- goto new_slab
- slub_percpu_partial() is NULL
- get_partial() returns NULL
- slub_put_cpu_ptr() (enables preemption)
- [PREEMPT B->A]
- task A: finish do_slab_free():
- this_cpu_cmpxchg_double() succeeds()
- [CORRUPT STATE: c->slab==NULL, c->freelist!=NULL]

From there, the object on c->freelist will get lost if task B is allowed to continue from here: It will proceed to the retry_load_slab label, set c->slab, then jump to load_freelist, which clobbers c->freelist.

But if we instead continue as follows, we get worse corruption:

- task A: run __slab_free() on object from other struct slab:
- CPU_PARTIAL_FREE case (slab was on no list, is now on pcpu partial)
- task A: run slab_alloc_node() with NUMA node constraint:
- fastpath fails (c->slab is NULL)
- call __slab_alloc()
- slub_get_cpu_ptr() (disables preemption)
- enter ___slab_alloc()
- c->slab is NULL: goto new_slab
- slub_percpu_partial() is non-NULL
- set c->slab to slub_percpu_partial(c)
- [CORRUPT STATE: c->slab points to slab-1, c->freelist has objects from slab-2]
- goto redo
- node_match() fails
- goto deactivate_slab
- existing c->freelist is passed into deactivate_slab()
- inuse count of slab-1 is decremented to account for object from slab-2

At this point, the inuse count of slab-1 is 1 lower than it should be. This means that if we free all allocated objects in slab-1 except for one, SLUB will think that slab-1 is completely unused, and may free its page, leading to use-after-free.

Fixes c17dda40a6a4e ("slub: Separate out kmem_cache_cpu processing from deactivate_slab")
Fixes 03e404af26dc2 ("slub: fast release on full slab")
---
 mm/slub.c | 2 ++
 1 file changed, 2 insertions(+)
mm/slub.c +2 -0
index e5535020e0fdf..b97fa5e210469 100644
@@ -2936,6 +2936,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 
 	if (!freelist) {
 		c->slab = NULL;
+		c->tid = next_tid(c->tid);
 		local_unlock_irqrestore(&s->cpu_slab->lock, flags);
 		stat(s, DEACTIVATE_BYPASS);
 		goto new_slab;
@@ -2968,6 +2969,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 	freelist = c->freelist;
 	c->slab = NULL;
 	c->freelist = NULL;
+	c->tid = next_tid(c->tid);
 	local_unlock_irqrestore(&s->cpu_slab->lock, flags);
 	deactivate_slab(s, slab, freelist);
 

base-commit: 9886142c7a2226439c1e3f7d9b69f9c7094c3ef6

Signature · 1 line
2.36.1.476.g0c4daa206d-goog
Acked-by on v1 1/1
#1 lore raw
Quoted 3 lines · Jann Horn

The fastpath in slab_alloc_node() assumes that c->slab is stable as long as the TID stays the same. However, two places in __slab_alloc() currently don't update the TID when deactivating the CPU slab.

Looks ok.

reviewer Acked-by on v1 1/1
#2 lore raw
In reply to Jann Horn
Quoted 73 lines · Jann Horn

The fastpath in slab_alloc_node() assumes that c->slab is stable as long as the TID stays the same. However, two places in __slab_alloc() currently don't update the TID when deactivating the CPU slab.

If multiple operations race the right way, this could lead to an object getting lost; or, in an even more unlikely situation, it could even lead to an object being freed onto the wrong slab's freelist, messing up the inuse counter and eventually causing a page to be freed to the page allocator while it still contains slab objects.

(I haven't actually tested these cases though, this is just based on looking at the code. Writing testcases for this stuff seems like it'd be a pain...)

The race leading to state inconsistency is (all operations on the same CPU and kmem_cache):

- task A: begin do_slab_free():
- read TID
- read pcpu freelist (==NULL)
- check slab == c->slab (true)
- [PREEMPT A->B]
- task B: begin slab_alloc_node():
- fastpath fails (c->freelist is NULL)
- enter __slab_alloc()
- slub_get_cpu_ptr() (disables preemption)
- enter ___slab_alloc()
- take local_lock_irqsave()
- read c->freelist as NULL
- get_freelist() returns NULL
- write c->slab = NULL
- drop local_unlock_irqrestore()
- goto new_slab
- slub_percpu_partial() is NULL
- get_partial() returns NULL
- slub_put_cpu_ptr() (enables preemption)
- [PREEMPT B->A]
- task A: finish do_slab_free():
- this_cpu_cmpxchg_double() succeeds()
- [CORRUPT STATE: c->slab==NULL, c->freelist!=NULL]

From there, the object on c->freelist will get lost if task B is allowed to continue from here: It will proceed to the retry_load_slab label, set c->slab, then jump to load_freelist, which clobbers c->freelist.

But if we instead continue as follows, we get worse corruption:

- task A: run __slab_free() on object from other struct slab:
- CPU_PARTIAL_FREE case (slab was on no list, is now on pcpu partial)
- task A: run slab_alloc_node() with NUMA node constraint:
- fastpath fails (c->slab is NULL)
- call __slab_alloc()
- slub_get_cpu_ptr() (disables preemption)
- enter ___slab_alloc()
- c->slab is NULL: goto new_slab
- slub_percpu_partial() is non-NULL
- set c->slab to slub_percpu_partial(c)
- [CORRUPT STATE: c->slab points to slab-1, c->freelist has objects from slab-2]
- goto redo
- node_match() fails
- goto deactivate_slab
- existing c->freelist is passed into deactivate_slab()
- inuse count of slab-1 is decremented to account for object from slab-2

At this point, the inuse count of slab-1 is 1 lower than it should be. This means that if we free all allocated objects in slab-1 except for one, SLUB will think that slab-1 is completely unused, and may free its page, leading to use-after-free.

Quoted 4 lines
Fixes c17dda40a6a4e ("slub: Separate out kmem_cache_cpu processing from deactivate_slab")
Fixes 03e404af26dc2 ("slub: fast release on full slab")
Reviewed-by on v1 1/1
#3 lore raw
In reply to Jann Horn
Quoted 73 lines · Jann Horn

The fastpath in slab_alloc_node() assumes that c->slab is stable as long as the TID stays the same. However, two places in __slab_alloc() currently don't update the TID when deactivating the CPU slab.

If multiple operations race the right way, this could lead to an object getting lost; or, in an even more unlikely situation, it could even lead to an object being freed onto the wrong slab's freelist, messing up the inuse counter and eventually causing a page to be freed to the page allocator while it still contains slab objects.

(I haven't actually tested these cases though, this is just based on looking at the code. Writing testcases for this stuff seems like it'd be a pain...)

The race leading to state inconsistency is (all operations on the same CPU and kmem_cache):

- task A: begin do_slab_free():
- read TID
- read pcpu freelist (==NULL)
- check slab == c->slab (true)
- [PREEMPT A->B]
- task B: begin slab_alloc_node():
- fastpath fails (c->freelist is NULL)
- enter __slab_alloc()
- slub_get_cpu_ptr() (disables preemption)
- enter ___slab_alloc()
- take local_lock_irqsave()
- read c->freelist as NULL
- get_freelist() returns NULL
- write c->slab = NULL
- drop local_unlock_irqrestore()
- goto new_slab
- slub_percpu_partial() is NULL
- get_partial() returns NULL
- slub_put_cpu_ptr() (enables preemption)
- [PREEMPT B->A]
- task A: finish do_slab_free():
- this_cpu_cmpxchg_double() succeeds()
- [CORRUPT STATE: c->slab==NULL, c->freelist!=NULL]

From there, the object on c->freelist will get lost if task B is allowed to continue from here: It will proceed to the retry_load_slab label, set c->slab, then jump to load_freelist, which clobbers c->freelist.

But if we instead continue as follows, we get worse corruption:

- task A: run __slab_free() on object from other struct slab:
- CPU_PARTIAL_FREE case (slab was on no list, is now on pcpu partial)
- task A: run slab_alloc_node() with NUMA node constraint:
- fastpath fails (c->slab is NULL)
- call __slab_alloc()
- slub_get_cpu_ptr() (disables preemption)
- enter ___slab_alloc()
- c->slab is NULL: goto new_slab
- slub_percpu_partial() is non-NULL
- set c->slab to slub_percpu_partial(c)
- [CORRUPT STATE: c->slab points to slab-1, c->freelist has objects from slab-2]
- goto redo
- node_match() fails
- goto deactivate_slab
- existing c->freelist is passed into deactivate_slab()
- inuse count of slab-1 is decremented to account for object from slab-2

At this point, the inuse count of slab-1 is 1 lower than it should be. This means that if we free all allocated objects in slab-1 except for one, SLUB will think that slab-1 is completely unused, and may free its page, leading to use-after-free.

Quoted 4 lines
Fixes c17dda40a6a4e ("slub: Separate out kmem_cache_cpu processing from deactivate_slab")
Fixes 03e404af26dc2 ("slub: fast release on full slab")

Thanks.

Tested-by on v1 1/1
#4 lore raw
In reply to Jann Horn
Quoted 32 lines · Jann Horn

The fastpath in slab_alloc_node() assumes that c->slab is stable as long as the TID stays the same. However, two places in __slab_alloc() currently don't update the TID when deactivating the CPU slab.

If multiple operations race the right way, this could lead to an object getting lost; or, in an even more unlikely situation, it could even lead to an object being freed onto the wrong slab's freelist, messing up the inuse counter and eventually causing a page to be freed to the page allocator while it still contains slab objects.

(I haven't actually tested these cases though, this is just based on looking at the code. Writing testcases for this stuff seems like it'd be a pain...)

The race leading to state inconsistency is (all operations on the same CPU and kmem_cache):

- task A: begin do_slab_free():
- read TID
- read pcpu freelist (==NULL)
- check slab == c->slab (true)
- [PREEMPT A->B]
- task B: begin slab_alloc_node():
- fastpath fails (c->freelist is NULL)
- enter __slab_alloc()
- slub_get_cpu_ptr() (disables preemption)
- enter ___slab_alloc()
- take local_lock_irqsave()
- read c->freelist as NULL
- get_freelist() returns NULL
- write c->slab = NULL
- drop local_unlock_irqrestore()

Quoted 8 lines

- goto new_slab
- slub_percpu_partial() is NULL
- get_partial() returns NULL
- slub_put_cpu_ptr() (enables preemption)
- [PREEMPT B->A]
- task A: finish do_slab_free():
- this_cpu_cmpxchg_double() succeeds()
- [CORRUPT STATE: c->slab==NULL, c->freelist!=NULL]

I can see this happening (!c->slab && c->freelist becoming true) when I synthetically add scheduling points in the code:

mm/slub.c +12 -0
index b97fa5e21046..b8012fdf2607 100644
@@ -3001,6 +3001,10 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 		goto check_new_slab;
 
 	slub_put_cpu_ptr(s->cpu_slab);
+
+	if (!in_atomic())
+		schedule();
+
 	slab = new_slab(s, gfpflags, node);
 	c = slub_get_cpu_ptr(s->cpu_slab);
 
@@ -3456,9 +3460,13 @@ static __always_inline void do_slab_free(struct kmem_cache *s,
 	if (likely(slab == c->slab)) {
 #ifndef CONFIG_PREEMPT_RT
 		void **freelist = READ_ONCE(c->freelist);
+		unsigned long flags;
 
 		set_freepointer(s, tail_obj, freelist);
 
+		if (!in_atomic())
+			schedule();
+
 		if (unlikely(!this_cpu_cmpxchg_double(
 				s->cpu_slab->freelist, s->cpu_slab->tid,
 				freelist, tid,
@@ -3467,6 +3475,10 @@ static __always_inline void do_slab_free(struct kmem_cache *s,
 			note_cmpxchg_failure("slab_free", s, tid);
 			goto redo;
 		}
+
+		local_irq_save(flags);
+		WARN_ON(!READ_ONCE(c->slab) && READ_ONCE(c->freelist));
+		local_irq_restore(flags);
 #else /* CONFIG_PREEMPT_RT */
 		/*
 		 * We cannot use the lockless fastpath on PREEMPT_RT because if
Quoted 16 lines

From there, the object on c->freelist will get lost if task B is allowed to continue from here: It will proceed to the retry_load_slab label, set c->slab, then jump to load_freelist, which clobbers c->freelist.

But if we instead continue as follows, we get worse corruption:

- task A: run __slab_free() on object from other struct slab:
- CPU_PARTIAL_FREE case (slab was on no list, is now on pcpu partial)
- task A: run slab_alloc_node() with NUMA node constraint:
- fastpath fails (c->slab is NULL)
- call __slab_alloc()
- slub_get_cpu_ptr() (disables preemption)
- enter ___slab_alloc()
- c->slab is NULL: goto new_slab
- slub_percpu_partial() is non-NULL
- set c->slab to slub_percpu_partial(c)

Quoted 8 lines

- [CORRUPT STATE: c->slab points to slab-1, c->freelist has objects from slab-2]
- goto redo
- node_match() fails
- goto deactivate_slab
- existing c->freelist is passed into deactivate_slab()
- inuse count of slab-1 is decremented to account for object from slab-2

I didn't try to reproduce this -- but I agree SLUB can be fooled by the condition (!c->slab && c->freelist).

Quoted 34 lines

At this point, the inuse count of slab-1 is 1 lower than it should be. This means that if we free all allocated objects in slab-1 except for one, SLUB will think that slab-1 is completely unused, and may free its page, leading to use-after-free.

Fixes c17dda40a6a4e ("slub: Separate out kmem_cache_cpu processing from deactivate_slab")
Fixes 03e404af26dc2 ("slub: fast release on full slab")
---
 mm/slub.c | 2 ++
 1 file changed, 2 insertions(+)
mm/slub.c +2 -0
index e5535020e0fdf..b97fa5e210469 100644
@@ -2936,6 +2936,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 
 	if (!freelist) {
 		c->slab = NULL;
+		c->tid = next_tid(c->tid);
 		local_unlock_irqrestore(&s->cpu_slab->lock, flags);
 		stat(s, DEACTIVATE_BYPASS);
 		goto new_slab;
@@ -2968,6 +2969,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 	freelist = c->freelist;
 	c->slab = NULL;
 	c->freelist = NULL;
+	c->tid = next_tid(c->tid);
 	local_unlock_irqrestore(&s->cpu_slab->lock, flags);
 	deactivate_slab(s, slab, freelist);
 
Quoted 3 lines

base-commit: 9886142c7a2226439c1e3f7d9b69f9c7094c3ef6

Signature · 1 line
2.36.1.476.g0c4daa206d-goog

With this patch I couldn't reproduce it.
This work is really nice. Thanks!

BTW I wonder how much this race will affect machines in the real world. Maybe just rare and undetectable memory leak?

Signature · 2 lines
Thanks,
Hyeonggon
#5 lore raw
In reply to Jann Horn
Quoted 68 lines · Jann Horn

The fastpath in slab_alloc_node() assumes that c->slab is stable as long as the TID stays the same. However, two places in __slab_alloc() currently don't update the TID when deactivating the CPU slab.

If multiple operations race the right way, this could lead to an object getting lost; or, in an even more unlikely situation, it could even lead to an object being freed onto the wrong slab's freelist, messing up the inuse counter and eventually causing a page to be freed to the page allocator while it still contains slab objects.

(I haven't actually tested these cases though, this is just based on looking at the code. Writing testcases for this stuff seems like it'd be a pain...)

The race leading to state inconsistency is (all operations on the same CPU and kmem_cache):

- task A: begin do_slab_free():
- read TID
- read pcpu freelist (==NULL)
- check slab == c->slab (true)
- [PREEMPT A->B]
- task B: begin slab_alloc_node():
- fastpath fails (c->freelist is NULL)
- enter __slab_alloc()
- slub_get_cpu_ptr() (disables preemption)
- enter ___slab_alloc()
- take local_lock_irqsave()
- read c->freelist as NULL
- get_freelist() returns NULL
- write c->slab = NULL
- drop local_unlock_irqrestore()
- goto new_slab
- slub_percpu_partial() is NULL
- get_partial() returns NULL
- slub_put_cpu_ptr() (enables preemption)
- [PREEMPT B->A]
- task A: finish do_slab_free():
- this_cpu_cmpxchg_double() succeeds()
- [CORRUPT STATE: c->slab==NULL, c->freelist!=NULL]

From there, the object on c->freelist will get lost if task B is allowed to continue from here: It will proceed to the retry_load_slab label, set c->slab, then jump to load_freelist, which clobbers c->freelist.

But if we instead continue as follows, we get worse corruption:

- task A: run __slab_free() on object from other struct slab:
- CPU_PARTIAL_FREE case (slab was on no list, is now on pcpu partial)
- task A: run slab_alloc_node() with NUMA node constraint:
- fastpath fails (c->slab is NULL)
- call __slab_alloc()
- slub_get_cpu_ptr() (disables preemption)
- enter ___slab_alloc()
- c->slab is NULL: goto new_slab
- slub_percpu_partial() is non-NULL
- set c->slab to slub_percpu_partial(c)
- [CORRUPT STATE: c->slab points to slab-1, c->freelist has objects from slab-2]
- goto redo
- node_match() fails
- goto deactivate_slab
- existing c->freelist is passed into deactivate_slab()
- inuse count of slab-1 is decremented to account for object from slab-2

Quoted 8 lines

At this point, the inuse count of slab-1 is 1 lower than it should be. This means that if we free all allocated objects in slab-1 except for one, SLUB will think that slab-1 is completely unused, and may free its page, leading to use-after-free.

Fixes c17dda40a6a4e ("slub: Separate out kmem_cache_cpu processing from deactivate_slab")
Fixes 03e404af26dc2 ("slub: fast release on full slab")

Hmm these are old commits, and currently oldest LTS is 4.9, so this will be fun. Worth doublechecking if it's not recent changes that actually introduced the bug... but seems not, AFAICS.

Quoted 10 lines
---
 mm/slub.c | 2 ++
 1 file changed, 2 insertions(+)
mm/slub.c +0 -0
index e5535020e0fdf..b97fa5e210469 100644

mm/slub.c static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, · line 2940

Quoted 5 lines
mm/slub.c +1 -0
@@ -2936,6 +2936,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 
 	if (!freelist) {
 		c->slab = NULL;
+		c->tid = next_tid(c->tid);
 		local_unlock_irqrestore(&s->cpu_slab->lock, flags);

So this immediate unlock after setting NULL is new from the 5.15 preempt-rt changes. However even in older versions we could goto new_slab, new_slab_objects(), new_slab(), allocate_slab(), where if (gfpflags_allow_blocking()) local_irq_enable(); (there's no extra disabled preemption besides the irq disable) so I'd say the bug was possible before too, but less often?

mm/slub.c static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, · line 2971

Quoted 6 lines
mm/slub.c +0 -0
 		stat(s, DEACTIVATE_BYPASS);
 		goto new_slab;
@@ -2968,6 +2969,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 	freelist = c->freelist;
 	c->slab = NULL;
 	c->freelist = NULL;

Previously these were part of deactivate_slab(), which does that at the very end, but also without bumping tid.
I just wonder if it's necessary too, because IIUC the scenario you described relies on the missing bump above. This alone doesn't cause the c->slab vs c->freelist mismatch?
But I guess it won't hurt to just bump tid on each c->freelist assignment. In backports we would just add it to deactivate_slab() instead.

Thanks. Applying to slab/for-5.19-rc3/fixes branch.

Quoted 6 lines
mm/slub.c +1 -0
+	c->tid = next_tid(c->tid);
 	local_unlock_irqrestore(&s->cpu_slab->lock, flags);
 	deactivate_slab(s, slab, freelist);
 

base-commit: 9886142c7a2226439c1e3f7d9b69f9c7094c3ef6

author
#6 lore raw
Quoted 10 lines · Vlastimil Babka
Quoted 9 lines · Jann Horn

The fastpath in slab_alloc_node() assumes that c->slab is stable as long as the TID stays the same. However, two places in __slab_alloc() currently don't update the TID when deactivating the CPU slab.

If multiple operations race the right way, this could lead to an object getting lost; or, in an even more unlikely situation, it could even lead to an object being freed onto the wrong slab's freelist, messing up the inuse counter and eventually causing a page to be freed to the page allocator while it still contains slab objects.

[...]

Quoted 7 lines
Quoted 3 lines
Fixes c17dda40a6a4e ("slub: Separate out kmem_cache_cpu processing from deactivate_slab")
Fixes 03e404af26dc2 ("slub: fast release on full slab")

Hmm these are old commits, and currently oldest LTS is 4.9, so this will be fun. Worth doublechecking if it's not recent changes that actually introduced the bug... but seems not, AFAICS.

[...]

Quoted 13 lines

mm/slub.c static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, · line 2940

Quoted 6 lines
mm/slub.c +1 -0
@@ -2936,6 +2936,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 
      if (!freelist) {
              c->slab = NULL;
+             c->tid = next_tid(c->tid);
              local_unlock_irqrestore(&s->cpu_slab->lock, flags);

So this immediate unlock after setting NULL is new from the 5.15 preempt-rt changes. However even in older versions we could goto new_slab, new_slab_objects(), new_slab(), allocate_slab(), where if (gfpflags_allow_blocking()) local_irq_enable(); (there's no extra disabled preemption besides the irq disable) so I'd say the bug was possible before too, but less often?

Yeah, I think so too.

Quoted 4 lines
Quoted 4 lines
mm/slub.c +0 -0
              stat(s, DEACTIVATE_BYPASS);
              goto new_slab;
@@ -2968,6 +2969,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
      freelist = c->freelist;
Quoted 8 lines

mm/slub.c static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, · line 2971

Quoted 2 lines
mm/slub.c +0 -0
      c->slab = NULL;
      c->freelist = NULL;

Previously these were part of deactivate_slab(), which does that at the very end, but also without bumping tid.
I just wonder if it's necessary too, because IIUC the scenario you described relies on the missing bump above. This alone doesn't cause the c->slab vs c->freelist mismatch?

It's a different scenario, but at least in the current version, the ALLOC_NODE_MISMATCH case jumps straight to the deactivate_slab label, which takes the local_lock, grabs the old c->freelist, NULLs out ->slab and ->freelist, then drops the local_lock again. If the c->freelist was non-NULL, then this will prevent concurrent cmpxchg success; but there is no reason why c->freelist has to be non-NULL here. So if c->freelist is already NULL, we basically just take the local_lock, set c->slab to NULL, and drop the local_lock. And IIUC the local_lock is the only protection we have here against concurrency, since the slub_get_cpu_ptr() in __slab_alloc() only disables migration? So again a concurrent fastpath free should be able to set c->freelist to non-NULL after c->slab has been set to NULL.

So I think this TID bump is also necessary for correctness in the current version.

And looking back at older kernels, back to at least 4.9, the ALLOC_NODE_MISMATCH case looks similarly broken - except that again, as you pointed out, we don't have the fine-grained locking, so it only becomes racy if we hit new_slab_objects() -> new_slab() -> allocate_slab() and then either we do local_irq_enable() or the allocation fails.

Quoted 1 line

Thanks. Applying to slab/for-5.19-rc3/fixes branch.

Thanks!

#7 lore raw
Quoted 15 lines · Jann Horn
Quoted 4 lines · Vlastimil Babka
Quoted 4 lines
              stat(s, DEACTIVATE_BYPASS);
              goto new_slab;
mm/slub.c +0 -0
@@ -2968,6 +2969,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
      freelist = c->freelist;
Quoted 8 lines

mm/slub.c static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, · line 2971

Quoted 2 lines
mm/slub.c +0 -0
      c->slab = NULL;
      c->freelist = NULL;

Previously these were part of deactivate_slab(), which does that at the very end, but also without bumping tid.
I just wonder if it's necessary too, because IIUC the scenario you described relies on the missing bump above. This alone doesn't cause the c->slab vs c->freelist mismatch?

Quoted 8 lines

It's a different scenario, but at least in the current version, the ALLOC_NODE_MISMATCH case jumps straight to the deactivate_slab label, which takes the local_lock, grabs the old c->freelist, NULLs out ->slab and ->freelist, then drops the local_lock again. If the c->freelist was non-NULL, then this will prevent concurrent cmpxchg success; but there is no reason why c->freelist has to be non-NULL here. So if c->freelist is already NULL, we basically just take the local_lock, set c->slab to NULL, and drop the local_lock. And IIUC the

Ah, right. Thanks for the explanation.

Quoted 3 lines

local_lock is the only protection we have here against concurrency, since the slub_get_cpu_ptr() in __slab_alloc() only disables migration?

On PREEMPT_RT it disables migration, but on !PREEMPT_RT it's a plain get_cpu_ptr() that does preempt_disable(). But that's an implementation detail, disabling migration would be sufficient on !PREEMPT_RT too, but right now it's cheaper to disable migration.

Quoted 5 lines

So again a concurrent fastpath free should be able to set c->freelist to non-NULL after c->slab has been set to NULL.

So I think this TID bump is also necessary for correctness in the current version.

OK.

Quoted 10 lines

And looking back at older kernels, back to at least 4.9, the ALLOC_NODE_MISMATCH case looks similarly broken - except that again, as you pointed out, we don't have the fine-grained locking, so it only becomes racy if we hit new_slab_objects() -> new_slab() -> allocate_slab() and then either we do local_irq_enable() or the allocation fails.

Quoted 1 line

Thanks. Applying to slab/for-5.19-rc3/fixes branch.

Thanks!