Skip to contentSkip to conversation

Corruption with MMOTS slub-bulk-allocation-from-per-cpu-partial-pages.patch

PATCH Jesper Dangaard Brouer 2 participants 3 messages Jun 8, 2015 – Jun 10, 2015

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

It seems the patch from (inserted below):
http://ozlabs.org/~akpm/mmots/broken-out/slub-bulk-allocation-from-per-cpu-partial-pages.patch

Is not protecting access to c->partial "enough" (section is under local_irq_disable/enable). When exercising bulk API I can make it crash/corrupt memory when compiled with CONFIG_SLUB_CPU_PARTIAL=y

First I suspected:
object = get_freelist(s, c->page);
But the problem goes way with CONFIG_SLUB_CPU_PARTIAL=n

From: Christoph Lameter <[email protected]>
Subject: slub: bulk allocation from per cpu partial pages

Cover all of the per cpu objects available.

Expand the bulk allocation support to drain the per cpu partial pages while interrupts are off.

Cc Jesper Dangaard Brouer <[email protected]>
Cc Pekka Enberg <[email protected]>
Cc David Rientjes <[email protected]>
Cc Joonsoo Kim <[email protected]>

---

 mm/slub.c |   36 +++++++++++++++++++++++++++++++++---
 1 file changed, 33 insertions(+), 3 deletions(-)
mm/slub.c renamed from mm/slub.c~slub-bulk-allocation-from-per-cpu-partial-pages +33 -3
diff -puN mm/slub.c~slub-bulk-allocation-from-per-cpu-partial-pages mm/slub.c
@@ -2769,15 +2769,45 @@ bool kmem_cache_alloc_bulk(struct kmem_c
 		while (size) {
 			void *object = c->freelist;
 
-			if (!object)
-				break;
+			if (unlikely(!object)) {
+				/*
+				 * Check if there remotely freed objects
+				 * availalbe in the page.
+				 */
+				object = get_freelist(s, c->page);
+
+				if (!object) {
+					/*
+					 * All objects in use lets check if
+					 * we have other per cpu partial
+					 * pages that have available
+					 * objects.
+					 */
+					c->page = c->partial;
+					if (!c->page) {
+						/* No per cpu objects left */
+						c->freelist = NULL;
+						break;
+					}
+
+					/* Next per cpu partial page */
+					c->partial = c->page->next;
+					c->freelist = get_freelist(s,
+							c->page);
+					continue;
+				}
+
+			}
+
 
-			c->freelist = get_freepointer(s, object);
 			*p++ = object;
 			size--;
 
 			if (unlikely(flags & __GFP_ZERO))
 				memset(object, 0, s->object_size);
+
+			c->freelist = get_freepointer(s, object);
+
 		}
 		c->tid = next_tid(c->tid);
 

_

Signature · 11 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#1 lore raw
Quoted 71 lines · Jesper Dangaard Brouer

It seems the patch from (inserted below):
http://ozlabs.org/~akpm/mmots/broken-out/slub-bulk-allocation-from-per-cpu-partial-pages.patch

Is not protecting access to c->partial "enough" (section is under local_irq_disable/enable). When exercising bulk API I can make it crash/corrupt memory when compiled with CONFIG_SLUB_CPU_PARTIAL=y

First I suspected:
object = get_freelist(s, c->page);
But the problem goes way with CONFIG_SLUB_CPU_PARTIAL=n

From: Christoph Lameter <[email protected]>
Subject: slub: bulk allocation from per cpu partial pages

Cover all of the per cpu objects available.

Expand the bulk allocation support to drain the per cpu partial pages while interrupts are off.

Cc Jesper Dangaard Brouer <[email protected]>
Cc Pekka Enberg <[email protected]>
Cc David Rientjes <[email protected]>
Cc Joonsoo Kim <[email protected]>

---

 mm/slub.c |   36 +++++++++++++++++++++++++++++++++---
 1 file changed, 33 insertions(+), 3 deletions(-)

diff -puN mm/slub.c~slub-bulk-allocation-from-per-cpu-partial-pages mm/slub.c

mm/slub.c renamed from mm/slub.c~slub-bulk-allocation-from-per-cpu-partial-pages +30 -2
@@ -2769,15 +2769,45 @@ bool kmem_cache_alloc_bulk(struct kmem_c
 		while (size) {
 			void *object = c->freelist;
 
-			if (!object)
-				break;
+			if (unlikely(!object)) {
+				/*
+				 * Check if there remotely freed objects
+				 * availalbe in the page.
+				 */
+				object = get_freelist(s, c->page);
+
+				if (!object) {
+					/*
+					 * All objects in use lets check if
+					 * we have other per cpu partial
+					 * pages that have available
+					 * objects.
+					 */
+					c->page = c->partial;
+					if (!c->page) {
+						/* No per cpu objects left */
+						c->freelist = NULL;
+						break;
+					}
+
+					/* Next per cpu partial page */
+					c->partial = c->page->next;
+					c->freelist = get_freelist(s,
+							c->page);
+					continue;
+				}
+
+			}
+

mm/slub.c bool kmem_cache_alloc_bulk(struct kmem_c · line 2810

Quoted 10 lines
mm/slub.c renamed from mm/slub.c~slub-bulk-allocation-from-per-cpu-partial-pages +3 -1
 
-			c->freelist = get_freepointer(s, object);
 			*p++ = object;
 			size--;
 
 			if (unlikely(flags & __GFP_ZERO))
 				memset(object, 0, s->object_size);
+
+			c->freelist = get_freepointer(s, object);
+

Hello,

get_freepointer() should be called before zeroing object. It may help your problem.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw

To Andrew/Christoph, can we drop this patch? Then, I'll base my work on top of the previous patch. Which also need some bug fixes, as pointed out by Joonsoo.

(p.s. iif then also drop slub-bulk-allocation-from-per-cpu-partial-pages-fix.patch)

Quoted 85 lines · On Tue, 9 Jun 2015 09:22:59 +0900 Joonsoo Kim <[email protected]> wrote:
Quoted 71 lines · Jesper Dangaard Brouer

It seems the patch from (inserted below):
http://ozlabs.org/~akpm/mmots/broken-out/slub-bulk-allocation-from-per-cpu-partial-pages.patch

Is not protecting access to c->partial "enough" (section is under local_irq_disable/enable). When exercising bulk API I can make it crash/corrupt memory when compiled with CONFIG_SLUB_CPU_PARTIAL=y

First I suspected:
object = get_freelist(s, c->page);
But the problem goes way with CONFIG_SLUB_CPU_PARTIAL=n

From: Christoph Lameter <[email protected]>
Subject: slub: bulk allocation from per cpu partial pages

Cover all of the per cpu objects available.

Expand the bulk allocation support to drain the per cpu partial pages while interrupts are off.

Cc Jesper Dangaard Brouer <[email protected]>
Cc Pekka Enberg <[email protected]>
Cc David Rientjes <[email protected]>
Cc Joonsoo Kim <[email protected]>

---

 mm/slub.c |   36 +++++++++++++++++++++++++++++++++---
 1 file changed, 33 insertions(+), 3 deletions(-)

diff -puN mm/slub.c~slub-bulk-allocation-from-per-cpu-partial-pages mm/slub.c

mm/slub.c renamed from mm/slub.c~slub-bulk-allocation-from-per-cpu-partial-pages +30 -2
@@ -2769,15 +2769,45 @@ bool kmem_cache_alloc_bulk(struct kmem_c
 		while (size) {
 			void *object = c->freelist;
 
-			if (!object)
-				break;
+			if (unlikely(!object)) {
+				/*
+				 * Check if there remotely freed objects
+				 * availalbe in the page.
+				 */
+				object = get_freelist(s, c->page);
+
+				if (!object) {
+					/*
+					 * All objects in use lets check if
+					 * we have other per cpu partial
+					 * pages that have available
+					 * objects.
+					 */
+					c->page = c->partial;
+					if (!c->page) {
+						/* No per cpu objects left */
+						c->freelist = NULL;
+						break;
+					}
+
+					/* Next per cpu partial page */
+					c->partial = c->page->next;
+					c->freelist = get_freelist(s,
+							c->page);
+					continue;
+				}
+
+			}
+

mm/slub.c bool kmem_cache_alloc_bulk(struct kmem_c · line 2810

Quoted 10 lines
mm/slub.c renamed from mm/slub.c~slub-bulk-allocation-from-per-cpu-partial-pages +3 -1
 
-			c->freelist = get_freepointer(s, object);
 			*p++ = object;
 			size--;
 
 			if (unlikely(flags & __GFP_ZERO))
 				memset(object, 0, s->object_size);
+
+			c->freelist = get_freepointer(s, object);
+

Hello,

Quoted 2 lines

get_freepointer() should be called before zeroing object. It may help your problem.

That is a bug, but I'm not invoking with __GFP_ZERO...

Signature · 11 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>