Skip to contentSkip to conversation

discriminate single bit error hardware failure from slab corruption.

PATCH Dave Jones 9 participants 14 messages Feb 2, 2006 – Feb 6, 2006 Memory management

lore
Share & tools
lore

Dave Jones

author
#0 lore raw

In the case where we detect a single bit has been flipped, we spew the usual slab corruption message, which users instantly think is a kernel bug. In a lot of cases, single bit errors are down to bad memory, or other hardware failure.

This patch adds an extra line to the slab debug messages in those cases, in the hope that users will try memtest before they report a bug.

000: 6b 6b 6b 6b 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Single bit error detected. Possibly bad RAM. Please run memtest86.

mm/slab.c renamed from mm/slab.c~ +15 -0
@@ -1313,8 +1313,11 @@ static void poison_obj(kmem_cache_t *cac
 static void dump_line(char *data, int offset, int limit)
 {
 	int i;
+	unsigned char total=0;
 	printk(KERN_ERR "%03x:", offset);
 	for (i = 0; i < limit; i++) {
+		if (data[offset+i] != POISON_FREE)
+			total += data[offset+i];
 		printk(" %02x", (unsigned char)data[offset + i]);
 	}
 	printk("\n");
@@ -1019,6 +1023,18 @@ static void dump_line(char *data, int of
 		}
 	}
 	printk("\n");
+	switch (total) {
+		case 0x36:
+		case 0x6a:
+		case 0x6f:
+		case 0x81:
+		case 0xac:
+		case 0xd3:
+		case 0xd5:
+		case 0xea:
+			printk (KERN_ERR "Single bit error detected. Possibly bad RAM. Please run memtest86.\n");
+			return;
+	}
 }
 #endif
 
#1 lore raw
Quoted 10 lines · Dave Jones

In the case where we detect a single bit has been flipped, we spew the usual slab corruption message, which users instantly think is a kernel bug. In a lot of cases, single bit errors are down to bad memory, or other hardware failure.

This patch adds an extra line to the slab debug messages in those cases, in the hope that users will try memtest before they report a bug.

000: 6b 6b 6b 6b 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Single bit error detected. Possibly bad RAM. Please run memtest86.

does memtest86 run on all $ARCHes ?
or this is good for <large percentage>, so it's Good. :) Just checking; it is a good idea.

Quoted 41 lines
mm/slab.c renamed from mm/slab.c~ +15 -1
@@ -1313,8 +1313,11 @@ static void poison_obj(kmem_cache_t *cac
 static void dump_line(char *data, int offset, int limit)
 {
 	int i;
+	unsigned char total=0;
 	printk(KERN_ERR "%03x:", offset);
 	for (i = 0; i < limit; i++) {
+		if (data[offset+i] != POISON_FREE)
+			total += data[offset+i];
 		printk(" %02x", (unsigned char)data[offset + i]);
 	}
 	printk("\n");
@@ -1019,6 +1023,18 @@ static void dump_line(char *data, int of
 		}
 	}
 	printk("\n");
+	switch (total) {
+		case 0x36:
+		case 0x6a:
+		case 0x6f:
+		case 0x81:
+		case 0xac:
+		case 0xd3:
+		case 0xd5:
+		case 0xea:
+			printk (KERN_ERR "Single bit error detected. Possibly bad RAM. Please run memtest86.\n");
+			return;
+	}
 }
 #endif
 
-

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 1 line
~Randy
#2 lore raw
In reply to Dave Jones
Quoted 11 lines · Dave Jones

In the case where we detect a single bit has been flipped, we spew the usual slab corruption message, which users instantly think is a kernel bug. In a lot of cases, single bit errors are down to bad memory, or other hardware failure.

This patch adds an extra line to the slab debug messages in those cases, in the hope that users will try memtest before they report a bug.

000: 6b 6b 6b 6b 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Single bit error detected. Possibly bad RAM. Please run memtest86.

May I suggest that the text be
Single bit error detected. Possibly bad RAM. Please run memtest86
and/or memtest86+.

both programs are good memory testers, but they are different and sometimes one finds problems not detected by the other.

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#3 lore raw
In reply to Dave Jones
Quoted 5 lines · Dave Jones

In the case where we detect a single bit has been flipped, we spew the usual slab corruption message, which users instantly think is a kernel bug. In a lot of cases, single bit errors are down to bad memory, or other hardware failure.

Quoted 7 lines

This patch adds an extra line to the slab debug messages in those cases, in the hope that users will try memtest before they report a bug.

000: 6b 6b 6b 6b 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Single bit error detected. Possibly bad RAM. Please run memtest86.

Looks good to me.

Pekka

#4 lore raw
In reply to Dave Jones
Quoted 16 lines · Dave Jones

In the case where we detect a single bit has been flipped, we spew the usual slab corruption message, which users instantly think is a kernel bug. In a lot of cases, single bit errors are down to bad memory, or other hardware failure.

This patch adds an extra line to the slab debug messages in those cases, in the hope that users will try memtest before they report a bug.

000: 6b 6b 6b 6b 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Single bit error detected. Possibly bad RAM. Please run memtest86.

mm/slab.c static void poison_obj(kmem_cache_t *cac · line 1322

Quoted 10 lines
mm/slab.c renamed from mm/slab.c~ +3 -0
@@ -1313,8 +1313,11 @@ static void poison_obj(kmem_cache_t *cac
 static void dump_line(char *data, int offset, int limit)
 {
 	int i;
+	unsigned char total=0;
 	printk(KERN_ERR "%03x:", offset);
 	for (i = 0; i < limit; i++) {
+		if (data[offset+i] != POISON_FREE)
+			total += data[offset+i];
 
 

how about

        total += hweight8(data[offset+i] ^ POISON_FREE);
Quoted 11 lines
mm/slab.c renamed from mm/slab.c~ +0 -0
 		printk(" %02x", (unsigned char)data[offset + i]);
	}
	printk("\n");
mm/slab.c renamed from mm/slab.c~ +4 -0
@@ -1019,6 +1023,18 @@ static void dump_line(char *data, int of
 		}
 	}
 	printk("\n");
+	switch (total) {
+		case 0x36:
+		case 0x6a:
+		case 0x6f:

mm/slab.c static void dump_line(char *data, int of · line 1039

Quoted 10 lines
mm/slab.c renamed from mm/slab.c~ +8 -0
+		case 0x81:
+		case 0xac:
+		case 0xd3:
+		case 0xd5:
+		case 0xea:
+			printk (KERN_ERR "Single bit error detected. Possibly bad RAM. Please run memtest86.\n");
+			return;
+	}
 
 

and a

    if (total == 1)
          printk(...);

here? it seems more readable and more correct as well.

Quoted 5 lines
mm/slab.c renamed from mm/slab.c~ +0 -0
 }

#endif

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
author
#5 lore raw
Quoted 26 lines · Avi Kivity
        total += hweight8(data[offset+i] ^ POISON_FREE);
Quoted 12 lines
		printk(" %02x", (unsigned char)data[offset + i]);
	}
	printk("\n");
mm/slab.c renamed from mm/slab.c~ +5 -0
@@ -1019,6 +1023,18 @@ static void dump_line(char *data, int of
 		}
 	}
 	printk("\n");
+	switch (total) {
+		case 0x36:
+		case 0x6a:
+		case 0x6f:
+		case 0x81:

mm/slab.c static void dump_line(char *data, int of · line 1040

Quoted 10 lines
mm/slab.c renamed from mm/slab.c~ +7 -0
+		case 0xac:
+		case 0xd3:
+		case 0xd5:
+		case 0xea:
+			printk (KERN_ERR "Single bit error detected. 
 Possibly bad RAM. Please run memtest86.\n");
+			return;
+	}
 
 

and a

Quoted 4 lines
    if (total == 1)
          printk(...);

here? it seems more readable and more correct as well.

More readable ? Are you kidding ?
What I wrote is smack-you-in-the-face-obvious what it's doing. With your variant, I have to sit down and think it through.

wrt correctness, what do you see wrong with my approach?

Dave

#6 lore raw
Quoted 28 lines · Dave Jones
Quoted 26 lines · Avi Kivity
        total += hweight8(data[offset+i] ^ POISON_FREE);
Quoted 12 lines
		printk(" %02x", (unsigned char)data[offset + i]);
	}
	printk("\n");
mm/slab.c renamed from mm/slab.c~ +5 -0
@@ -1019,6 +1023,18 @@ static void dump_line(char *data, int of
 		}
 	}
 	printk("\n");
+	switch (total) {
+		case 0x36:
+		case 0x6a:
+		case 0x6f:
+		case 0x81:

mm/slab.c static void dump_line(char *data, int of · line 1040

Quoted 10 lines
mm/slab.c renamed from mm/slab.c~ +7 -0
+		case 0xac:
+		case 0xd3:
+		case 0xd5:
+		case 0xea:
+			printk (KERN_ERR "Single bit error detected. 
 Possibly bad RAM. Please run memtest86.\n");
+			return;
+	}
 
 

and a

Quoted 10 lines
Quoted 4 lines
    if (total == 1)
          printk(...);

here? it seems more readable and more correct as well.

More readable ? Are you kidding ?
What I wrote is smack-you-in-the-face-obvious what it's doing. With your variant, I have to sit down and think it through.

Looks like we have mirror image brains :) - I had to scratch my scalp to figure out where all the magic numbers in the switch came from.

Perhaps well named variables will help:

   unsigned char modified_bits = data[offset+i] ^ POSION_FREE;
   int modified_bits_count = hweight8(modified_bits);
   total += modified_bits_count;
Quoted 3 lines

wrt correctness, what do you see wrong with my approach?

Your code will generate a false positive 8 times in 256 runs, or 1 in 32. A 3% false positive rate seems excessive, It's also sensitive to changes to POISON_FREE.

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
author
#7 lore raw
Quoted 8 lines · Avi Kivity
   unsigned char modified_bits = data[offset+i] ^ POSION_FREE;
   int modified_bits_count = hweight8(modified_bits);
   total += modified_bits_count;
Quoted 1 line

wrt correctness, what do you see wrong with my approach?

Your code will generate a false positive 8 times in 256 runs, or 1 in
32. A 3% false positive rate seems excessive, It's also sensitive to changes to POISON_FREE.

Hmm, I made a mistake in my maths somewhere, and some of those values are incorrect, so having the compiler do the work would have stopped me screwing up, but once the correct values are used, I doubt there's ever a really compelling reason to change the slab poison pattern.

Dave

In case where we detect a single bit has been flipped, we spew the usual slab corruption message, which users instantly think is a kernel bug. In a lot of cases, single bit errors are down to bad memory, or other hardware failure.

This patch adds an extra line to the slab debug messages in those cases, in the hope that users will try memtest before they report a bug.

000: 6b 6b 6b 6b 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Single bit error detected. Possibly bad RAM. Run memtest86.

mm/slab.c renamed from mm/slab.c~ +19 -0
@@ -1313,8 +1313,11 @@ static void poison_obj(kmem_cache_t *cac
 static void dump_line(char *data, int offset, int limit)
 {
 	int i;
+	unsigned char total=0;
 	printk(KERN_ERR "%03x:", offset);
 	for (i = 0; i < limit; i++) {
+		if (data[offset+i] != POISON_FREE)
+			total += data[offset+i];
 		printk(" %02x", (unsigned char)data[offset + i]);
 	}
 	printk("\n");
@@ -1019,6 +1023,22 @@ static void dump_line(char *data, int of
 		}
 	}
 	printk("\n");
+	switch (total) {
+					/* 01101011 (0x6b - SLAB_POISON) */
+		case 0x6a:	/* 01101010 bit 0 flipped */
+		case 0x69:	/* 01101001 bit 1 flipped */
+		case 0x6f:	/* 01101111 bit 2 flipped */
+		case 0x63:	/* 01100011 bit 3 flipped */
+		case 0x7b:	/* 01111011 bit 4 flipped */
+		case 0x4b:	/* 01001011 bit 5 flipped */
+		case 0x2b:	/* 00101011 bit 6 flipped */
+		case 0xeb:	/* 11101011 bit 7 flipped */
+			printk (KERN_ERR "Single bit error detected. Possibly bad RAM\n"
+#ifdef CONFIG_X86
+			printk (KERN_ERR "Run memtest86 or other memory test tool.\n");
+#endif
+			return;
+	}
 }
 #endif
 
#8 lore raw

Dave> Hmm, I made a mistake in my maths somewhere, and some of Dave> those values are incorrect, so having the compiler do the Dave> work would have stopped me screwing up, but once the correct Dave> values are used, I doubt there's ever a really compelling Dave> reason to change the slab poison pattern.

But Avi is still correct about false positives. For example, if something stomps on the slab poison and leaves it as

    e0 08 03 00

then that will add up to eb and still trigger your message, even though it's far from a single bit error.

Maybe making the loop be something like

	unsigned char total = 0, bad_count = 0;
	printk(KERN_ERR "%03x:", offset);
	for (i = 0; i < limit; i++) {
		if (data[offset+i] != POISON_FREE) {
			total += data[offset+i];
			++bad_count;
		}
		printk(" %02x", (unsigned char)data[offset + i]);
	}

and then you can put

	if (bad_count == 1)

before the switch statement.

I have to admit that Avi's code seems clearer to me too, though.

- R.

author
#9 lore raw
Quoted 6 lines · Roland Dreier

Dave> Hmm, I made a mistake in my maths somewhere, and some of Dave> those values are incorrect, so having the compiler do the Dave> work would have stopped me screwing up, but once the correct Dave> values are used, I doubt there's ever a really compelling Dave> reason to change the slab poison pattern.

Quoted 7 lines

But Avi is still correct about false positives. For example, if something stomps on the slab poison and leaves it as

    e0 08 03 00

then that will add up to eb and still trigger your message, even though it's far from a single bit error.

Ah, now I see the point Avi was making.

Quoted 12 lines

Maybe making the loop be something like

	unsigned char total = 0, bad_count = 0;
	printk(KERN_ERR "%03x:", offset);
	for (i = 0; i < limit; i++) {
		if (data[offset+i] != POISON_FREE) {
			total += data[offset+i];
			++bad_count;
		}
		printk(" %02x", (unsigned char)data[offset + i]);
	}
Quoted 7 lines

and then you can put

	if (bad_count == 1)

before the switch statement.

I have to admit that Avi's code seems clearer to me too, though.

I'm easily persuaded either way really, as long as we arrive at a desirable end-result ;)

Dave

#11 lore raw
In reply to Dave Jones

mm/slab.c static void dump_line(char *data, int of · line 1035

Quoted 8 lines · Dave Jones
mm/slab.c renamed from mm/slab.c~ +8 -0
+		case 0x6a:	/* 01101010 bit 0 flipped */
+		case 0x69:	/* 01101001 bit 1 flipped */
+		case 0x6f:	/* 01101111 bit 2 flipped */
+		case 0x63:	/* 01100011 bit 3 flipped */
+		case 0x7b:	/* 01111011 bit 4 flipped */
+		case 0x4b:	/* 01001011 bit 5 flipped */
+		case 0x2b:	/* 00101011 bit 6 flipped */
+		case 0xeb:	/* 11101011 bit 7 flipped */

What about simply:
case 0x6b^0x01:
case 0x6b^0x02:
case 0x6b^0x04:
case 0x6b^0x08:
case 0x6b^0x10:
case 0x6b^0x20:
case 0x6b^0x40:
case 0x6b^0x80:

OG.

#13 lore raw
In reply to Dave Jones

mm/slab.c

Quoted 6 lines

000: 6b 6b 6b 6b 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Single bit error detected. Possibly bad RAM. Please run memtest86.

mm/slab.c renamed from mm/slab.c~ +0 -0

So, and what do non-x86 users use?

Jan Engelhardt

#12 lore raw
In reply to Dave Jones

Hi!

Quoted 3 lines
Quoted 1 line

here? it seems more readable and more correct as well.

More readable ? Are you kidding ?

Well, his method really counts flipped bits. It looks very obvious to me -- unlike magic numbers.

Signature · 1 line
Thanks, Sharp!