Skip to contentSkip to conversation

mm: slub: add panic_on_error to the debug facilities

PATCH In review Rafael Aquini 4 participants 11 messages May 1, 2020 – May 8, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm: slub: add panic_on_error to the debug facilities

Sometimes it is desirable to override SLUB's debug facilities default behavior upon stumbling on a cache or object error and just stop the execution in order to grab a coredump, at the error-spotting time, instead of trying to fix the issue and report in an attempt to keep the system rolling.

This patch introduces a new debug flag SLAB_PANIC_ON_ERROR, along with its related SLUB-machinery, in order to extend current slub_debug facilites and provide the aforementioned behavior override.

---
 Documentation/vm/slub.rst |  2 ++
 include/linux/slab.h      |  2 ++
 mm/slab.h                 |  3 ++-
 mm/slub.c                 | 44 ++++++++++++++++++++++++++++++---------
 4 files changed, 40 insertions(+), 11 deletions(-)
Documentation/vm/slub.rst +2 -0
index 933ada4368ff..51b18c28ec78 100644
@@ -54,6 +54,8 @@ Possible debug options are::
 			caused higher minimum slab orders
 	-		Switch all debugging off (useful if the kernel is
 			configured with CONFIG_SLUB_DEBUG_ON)
+	C		Toggle panic on error (crash) to allow for post-mortem
+			analysis of a coredump taken at the error-spotting time
 
 F.e. in order to boot just with sanity checks and red zoning one would specify::
 
include/linux/slab.h +2 -0
index 6d454886bcaf..e3496ad7859f 100644
@@ -25,6 +25,8 @@
  */
 /* DEBUG: Perform (expensive) checks on alloc/free */
 #define SLAB_CONSISTENCY_CHECKS	((slab_flags_t __force)0x00000100U)
+/* DEBUG: panic on error (forced crash) */
+#define SLAB_PANIC_ON_ERROR	((slab_flags_t __force)0x00000200U)
 /* DEBUG: Red zone objs in a cache */
 #define SLAB_RED_ZONE		((slab_flags_t __force)0x00000400U)
 /* DEBUG: Poison objects */
mm/slab.h +2 -1
index 207c83ef6e06..27116f8683a1 100644
@@ -198,7 +198,8 @@ static inline slab_flags_t kmem_cache_flags(unsigned int object_size,
 #define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER)
 #elif defined(CONFIG_SLUB_DEBUG)
 #define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \
-			  SLAB_TRACE | SLAB_CONSISTENCY_CHECKS)
+			  SLAB_TRACE | SLAB_CONSISTENCY_CHECKS | \
+			  SLAB_PANIC_ON_ERROR)
 #else
 #define SLAB_DEBUG_FLAGS (0)
 #endif
mm/slub.c +34 -10

92 lines not on this page · raw message

Signature · 1 line
2.25.4
#1 lore raw
Quoted 10 lines · Rafael Aquini

Sometimes it is desirable to override SLUB's debug facilities default behavior upon stumbling on a cache or object error and just stop the execution in order to grab a coredump, at the error-spotting time, instead of trying to fix the issue and report in an attempt to keep the system rolling.

Quoted 4 lines

This patch introduces a new debug flag SLAB_PANIC_ON_ERROR, along with its related SLUB-machinery, in order to extend current slub_debug facilites and provide the aforementioned behavior override.

Instead of adding those things everywhere. How about adding something like panic_on_taint? Then, you could write specific taint flags you are interested in to that file because slab_bug() will taint it TAINT_BAD_PAGE.

#2 lore raw
Quoted 10 lines · Qian Cai
Quoted 10 lines · Rafael Aquini

Sometimes it is desirable to override SLUB's debug facilities default behavior upon stumbling on a cache or object error and just stop the execution in order to grab a coredump, at the error-spotting time, instead of trying to fix the issue and report in an attempt to keep the system rolling.

Quoted 7 lines
Quoted 4 lines

This patch introduces a new debug flag SLAB_PANIC_ON_ERROR, along with its related SLUB-machinery, in order to extend current slub_debug facilites and provide the aforementioned behavior override.

Instead of adding those things everywhere. How about adding something like panic_on_taint? Then, you could write specific taint flags you are interested in to that file because slab_bug() will taint it TAINT_BAD_PAGE.

It seems like a good idea which also would required "adding things" elsewhere, but doesn't look mutually exclusive with the approach here.

Thanks
-- Rafael

#3 lore raw
Quoted 20 lines · Rafael Aquini
Quoted 15 lines · Qian Cai
Quoted 14 lines · Rafael Aquini

Sometimes it is desirable to override SLUB's debug facilities default behavior upon stumbling on a cache or object error and just stop the execution in order to grab a coredump, at the error-spotting time, instead of trying to fix the issue and report in an attempt to keep the system rolling.

This patch introduces a new debug flag SLAB_PANIC_ON_ERROR, along with its related SLUB-machinery, in order to extend current slub_debug facilites and provide the aforementioned behavior override.

Quoted 4 lines
Quoted 2 lines

Instead of adding those things everywhere. How about adding something like panic_on_taint? Then, you could write specific taint flags you are interested in to that file because slab_bug() will taint it TAINT_BAD_PAGE.

It seems like a good idea which also would required "adding things" elsewhere, but doesn't look mutually exclusive with the approach here.

No, it is mutually exclusive because panic_on_taint would do this same thing but saner.

The thing is that this request came up over and over again where people may want to panic the kernel because of TAINT_BAD_PAGE or some other places due to tainted.

#4 lore raw
In reply to Rafael Aquini
Quoted 6 lines · Rafael Aquini

It seems like a good idea which also would required "adding things" elsewhere, but doesn't look mutually exclusive with the approach here.

Also, what’s so special about these bad pages here that deserve a crash dump, but not other TAINT_BAD_PAGE places?

#5 lore raw
Quoted 9 lines · Qian Cai
Quoted 6 lines · Rafael Aquini

It seems like a good idea which also would required "adding things" elsewhere, but doesn't look mutually exclusive with the approach here.

Also, what’s so special about these bad pages here that deserve a crash dump, but not other TAINT_BAD_PAGE places?

In that light, yes they're not as different and we could better leverage a panic_on_taint mechanism (similar to panic_on_warn).

I'll get that posted soon.

Thanks.
-- Rafael

maintainer
#6 lore raw
In reply to Rafael Aquini
Quoted 11 lines · On Fri, 1 May 2020 17:15:40 -0400 Rafael Aquini <[email protected]> wrote:

Sometimes it is desirable to override SLUB's debug facilities default behavior upon stumbling on a cache or object error and just stop the execution in order to grab a coredump, at the error-spotting time, instead of trying to fix the issue and report in an attempt to keep the system rolling.

This patch introduces a new debug flag SLAB_PANIC_ON_ERROR, along with its related SLUB-machinery, in order to extend current slub_debug facilites and provide the aforementioned behavior override.

Sees reasonable.

Documentation/vm/slub.rst Possible debug options are:: · line 58

Quoted 8 lines
Documentation/vm/slub.rst +2 -0
@@ -54,6 +54,8 @@ Possible debug options are::
 			caused higher minimum slab orders
 	-		Switch all debugging off (useful if the kernel is
 			configured with CONFIG_SLUB_DEBUG_ON)
+	C		Toggle panic on error (crash) to allow for post-mortem
+			analysis of a coredump taken at the error-spotting time

nit: "toggle" means to switch to the other state. But what we're doing here is to set to the "on" state. This:

Documentation/vm/slub.rst renamed from Documentation/vm/slub.rst~mm-slub-add-panic_on_error-to-the-debug-facilities-fix +2 -2
@@ -49,12 +49,12 @@ Possible debug options are::
 	P		Poisoning (object and padding)
 	U		User tracking (free and alloc)
 	T		Trace (please only use on single slabs)
-	A		Toggle failslab filter mark for the cache
+	A		Enable failslab filter mark for the cache
 	O		Switch debugging off for caches that would have
 			caused higher minimum slab orders
 	-		Switch all debugging off (useful if the kernel is
 			configured with CONFIG_SLUB_DEBUG_ON)
-	C		Toggle panic on error (crash) to allow for post-mortem
+	C		Enable panic on error (crash) to allow for post-mortem
 			analysis of a coredump taken at the error-spotting time
 
 F.e. in order to boot just with sanity checks and red zoning one would specify::

_

#7 lore raw
Quoted 16 lines · Andrew Morton
Quoted 11 lines · On Fri, 1 May 2020 17:15:40 -0400 Rafael Aquini <[email protected]> wrote:

Sometimes it is desirable to override SLUB's debug facilities default behavior upon stumbling on a cache or object error and just stop the execution in order to grab a coredump, at the error-spotting time, instead of trying to fix the issue and report in an attempt to keep the system rolling.

This patch introduces a new debug flag SLAB_PANIC_ON_ERROR, along with its related SLUB-machinery, in order to extend current slub_debug facilites and provide the aforementioned behavior override.

Sees reasonable.

Quoted 12 lines

Documentation/vm/slub.rst Possible debug options are:: · line 58

Quoted 8 lines
Documentation/vm/slub.rst +2 -0
@@ -54,6 +54,8 @@ Possible debug options are::
 			caused higher minimum slab orders
 	-		Switch all debugging off (useful if the kernel is
 			configured with CONFIG_SLUB_DEBUG_ON)
+	C		Toggle panic on error (crash) to allow for post-mortem
+			analysis of a coredump taken at the error-spotting time

nit: "toggle" means to switch to the other state. But what we're doing here is to set to the "on" state. This:

Thanks Andrew, that's indeed much better.

Quoted 20 lines
Documentation/vm/slub.rst renamed from Documentation/vm/slub.rst~mm-slub-add-panic_on_error-to-the-debug-facilities-fix +2 -2
@@ -49,12 +49,12 @@ Possible debug options are::
 	P		Poisoning (object and padding)
 	U		User tracking (free and alloc)
 	T		Trace (please only use on single slabs)
-	A		Toggle failslab filter mark for the cache
+	A		Enable failslab filter mark for the cache
 	O		Switch debugging off for caches that would have
 			caused higher minimum slab orders
 	-		Switch all debugging off (useful if the kernel is
 			configured with CONFIG_SLUB_DEBUG_ON)
-	C		Toggle panic on error (crash) to allow for post-mortem
+	C		Enable panic on error (crash) to allow for post-mortem
 			analysis of a coredump taken at the error-spotting time
 
 F.e. in order to boot just with sanity checks and red zoning one would specify::

_

#8 lore raw
In reply to Rafael Aquini
Quoted 5 lines · Rafael Aquini

Sometimes it is desirable to override SLUB's debug facilities default behavior upon stumbling on a cache or object error and just stop the execution in order to grab a coredump, at the error-spotting time, instead of trying to fix the issue and report in an attempt to keep the system rolling.

The stopping of execution on an error is the default behavior. Usually you get some OOPS somewhere when data is corrupted and that causes a core dump.

SLUB can fix the issue and continue if enabled by specifying special options on boot. That is *not* the default.

#9 lore raw
Quoted 8 lines · Christopher Lameter
Quoted 5 lines · Rafael Aquini

Sometimes it is desirable to override SLUB's debug facilities default behavior upon stumbling on a cache or object error and just stop the execution in order to grab a coredump, at the error-spotting time, instead of trying to fix the issue and report in an attempt to keep the system rolling.

Quoted 7 lines

The stopping of execution on an error is the default behavior. Usually you get some OOPS somewhere when data is corrupted and that causes a core dump.

SLUB can fix the issue and continue if enabled by specifying special options on boot. That is *not* the default.

It is the default behavior when slub_debug is turned on, which is what this patch is trying to override, when needed. We've been seeing the need for such feature as, most often than not, by letting the system running to crash somewhere else after hitting occurrences reported by slub_debug ends up clobbering clues to the original issue.

-- Rafael

#10 lore raw
Quoted 13 lines · Rafael Aquini
Quoted 12 lines · Christopher Lameter
Quoted 5 lines · Rafael Aquini

Sometimes it is desirable to override SLUB's debug facilities default behavior upon stumbling on a cache or object error and just stop the execution in order to grab a coredump, at the error-spotting time, instead of trying to fix the issue and report in an attempt to keep the system rolling.

The stopping of execution on an error is the default behavior. Usually you get some OOPS somewhere when data is corrupted and that causes a core dump.

Quoted 8 lines
Quoted 3 lines

SLUB can fix the issue and continue if enabled by specifying special options on boot. That is *not* the default.

It is the default behavior when slub_debug is turned on, which is what this patch is trying to override, when needed. We've been seeing the need for such feature as, most often than not, by letting the system running to crash somewhere else after hitting occurrences reported by slub_debug ends up clobbering clues to the original issue.

Ok. The backtrace is not sufficient in that case?