x86_64: early memtest to find bad ram

PATCH In review Yinghai Lu 8 participants 20 messages Mar 21, 2008 – Mar 25, 2008

Subsystems (2)

x86 x86 MM

lore

Share & tools

lore

Yinghai Lu

author

2008-03-21 06:56 UTC #0 lore raw

PATCH 1/1 x86_64: early memtest to find bad ram

do simple memtest after init_memory_mapping

use find_e820_area_size to find all ram range that is not reserved.

and do some simple bits test to find some bad ram.

if find some bad ram, use reserve_early to exclude that range.

Signed-off-by Yinghai Lu <[email protected]>

arch/x86/kernel/e820_64.c +69 -1

Index: linux-2.6/arch/x86/kernel/e820_64.c
===================================================================
@@ -119,6 +119,40 @@ again:
 	return changed;
 }
 
+/* Check for already reserved areas */
+static inline int
+bad_addr_size(unsigned long *addrp, unsigned long *sizep, unsigned long align)
+{
+	int i;
+	unsigned long addr = *addrp, last;
+	unsigned long size = *sizep;
+	int changed = 0;
+again:
+	last = addr + size;
+	for (i = 0; i < MAX_EARLY_RES && early_res[i].end; i++) {
+		struct early_res *r = &early_res[i];
+		if (last > r->start && addr < r->start) {
+			size = r->start - addr;
+			changed = 1;
+			goto again;
+		}
+		if (last > r->end && addr < r->end) {
+			addr = round_up(r->end, align);
+			size = last - addr;
+			changed = 1;
+			goto again;
+		}
+		if (last <= r->end && addr >= r->start) {
+			(*sizep)++;
+			return 0;
+		}
+	}
+	if (changed) {
+		*addrp = addr;
+		*sizep = size;
+	}
+	return changed;
+}
 /*
  * This function checks if any part of the range <start,end> is mapped
  * with type.

@@ -195,7 +229,7 @@ unsigned long __init find_e820_area(unsi
 		ei_last = ei->addr + ei->size;
 		if (addr < start)
 			addr = round_up(start, align);
-		if (addr > ei_last)
+		if (addr >= ei_last)
 			continue;
 		while (bad_addr(&addr, size, align) && addr+size <= ei_last)
 			;

@@ -210,6 +244,40 @@ unsigned long __init find_e820_area(unsi
 }
 
 /*
+ * Find next free range after *start
+ */
+unsigned long __init find_e820_area_size(unsigned long start, unsigned long *sizep, unsigned long align)
+{
+	int i;
+
+	for (i = 0; i < e820.nr_map; i++) {
+		struct e820entry *ei = &e820.map[i];
+		unsigned long addr, last;
+		unsigned long ei_last;
+
+		if (ei->type != E820_RAM)
+			continue;
+		addr = round_up(ei->addr, align);
+		ei_last = ei->addr + ei->size;
+//		printk(KERN_DEBUG "find_e820_area_size : e820 %d [%llx, %lx]\n", i, ei->addr, ei_last);
+		if (addr < start)
+			addr = round_up(start, align);
+//		printk(KERN_DEBUG "find_e820_area_size : 0 [%lx, %lx]\n", addr, ei_last);
+		if (addr >= ei_last)
+			continue;
+		*sizep = ei_last - addr;
+		while (bad_addr_size(&addr, sizep, align) && addr+ *sizep <= ei_last)
+			;
+		last = addr + *sizep;
+//		printk(KERN_DEBUG "find_e820_area_size : 1 [%lx, %lx]\n", addr, last);
+		if (last > ei_last)
+			continue;
+		return addr;
+	}
+	return -1UL;
+
+}
+/*
  * Find the highest page frame number we have available
  */
 unsigned long __init e820_end_of_ram(void)

arch/x86/mm/init_64.c +105 -1

125 lines not on this page · raw message

include/asm-x86/e820_64.h +3 -0

Index: linux-2.6/include/asm-x86/e820_64.h
===================================================================
@@ -16,6 +16,9 @@
 #ifndef __ASSEMBLY__
 extern unsigned long find_e820_area(unsigned long start, unsigned long end, 
 				    unsigned long size, unsigned long align);
+extern unsigned long find_e820_area_size(unsigned long start,
+					 unsigned long *sizep,
+					 unsigned long align);
 extern void add_memory_region(unsigned long start, unsigned long size, 
 			      int type);
 extern void update_memory_range(u64 start, u64 size, unsigned old_type,

Ingo Molnar

2008-03-21 12:03 UTC #1 lore raw

Quoted 7 lines · Yinghai Lu

do simple memtest after init_memory_mapping
use find_e820_area_size to find all ram range that is not reserved.
and do some simple bits test to find some bad ram.
if find some bad ram, use reserve_early to exclude that range.

very nice patch! I always thought that this was the proper way to do memtest - and we could in fact also do something like this after SMP bringup, and hit the memory bus via multiple CPUs. [that will need a different enumeration though than e820 maps]

one structural observation: please make this unified functionality, so that 32-bit kernels can make use of it too.

a small style nit from scripts/checkpatch.pl:

Quoted 1 line

+// printk(KERN_DEBUG "find_e820_area_size : e820 %d [%llx, %lx]\n", i, ei->addr, ei_last);

remove such lines or make them pr_debug(). (checkpatch also found more such cases)

also, please add a CONFIG_BOOTPARAM_MEMTEST=y option so that distributions can enable this by default in their debug kernels.

i've applied your current version to get some testing, please send delta patches against x86/latest.

Ingo

H. Peter Anvin

reviewer

2008-03-21 13:09 UTC #2 lore raw

Quoted 5 lines · Ingo Molnar

very nice patch! I always thought that this was the proper way to do memtest - and we could in fact also do something like this after SMP bringup, and hit the memory bus via multiple CPUs. [that will need a different enumeration though than e820 maps]
one structural observation: please make this unified functionality, so that 32-bit kernels can make use of it too.

Indeed. Of course, it would also be nice if distros shipped bootloader-invoked prekernel test software, like memtest86+, by default.

-hpa

Ingo Molnar

2008-03-21 14:29 UTC #3 lore raw

Quoted 5 lines · H. Peter Anvin

Quoted 5 lines
very nice patch! I always thought that this was the proper way to do memtest - and we could in fact also do something like this after SMP bringup, and hit the memory bus via multiple CPUs. [that will need a different enumeration though than e820 maps]

Quoted 7 lines

Quoted 3 lines
one structural observation: please make this unified functionality, so that 32-bit kernels can make use of it too.
Indeed. Of course, it would also be nice if distros shipped bootloader-invoked prekernel test software, like memtest86+, by default.

some do (Fedora for example), but it's still a bit quirky for users to invoke and it would be nice to see those results in the kernel log as well and flag possibly flaky systems that way. (add a taint bit, etc., etc.)

Ingo

Yinghai Lu

2008-03-21 19:08 UTC #6 lore raw

In reply to H. Peter Anvin

Quoted 7 lines · H. Peter Anvin

Quoted 6 lines · Ingo Molnar
very nice patch! I always thought that this was the proper way to do memtest - and we could in fact also do something like this after SMP bringup, and hit the memory bus via multiple CPUs. [that will need a different enumeration though than e820 maps]

Quoted 6 lines

Quoted 3 lines
one structural observation: please make this unified functionality, so that 32-bit kernels can make use of it too.
Indeed. Of course, it would also be nice if distros shipped bootloader-invoked prekernel test software, like memtest86+, by default.

the current memtest86 is running in 32 bit mode, and only support 64G ram.

I tried to expand that a bit, to support 1024g, but it only works on some machine.
could be stack provide is not big enough?

H. Peter Anvin

reviewer

2008-03-21 19:59 UTC #7 lore raw

Quoted 12 lines · Yinghai Lu

Quoted 11 lines · H. Peter Anvin
Quoted 9 lines · Ingo Molnar
very nice patch! I always thought that this was the proper way to do memtest - and we could in fact also do something like this after SMP bringup, and hit the memory bus via multiple CPUs. [that will need a different enumeration though than e820 maps]
one structural observation: please make this unified functionality, so that 32-bit kernels can make use of it too.

Quoted 9 lines

Quoted 2 lines
Indeed. Of course, it would also be nice if distros shipped bootloader-invoked prekernel test software, like memtest86+, by default.
the current memtest86 is running in 32 bit mode, and only support 64G ram.
I tried to expand that a bit, to support 1024g, but it only works on some machine.
could be stack provide is not big enough?

Wonder how hard it would be to make it run 64 bits...

-hpa

Yinghai Lu

2008-03-21 20:09 UTC #8 lore raw

Quoted 17 lines · H. Peter Anvin

Quoted 15 lines · Yinghai Lu
Quoted 13 lines · H. Peter Anvin
Quoted 9 lines · Ingo Molnar
very nice patch! I always thought that this was the proper way to do memtest - and we could in fact also do something like this after SMP bringup, and hit the memory bus via multiple CPUs. [that will need a different enumeration though than e820 maps]
one structural observation: please make this unified functionality, so that 32-bit kernels can make use of it too.
Indeed. Of course, it would also be nice if distros shipped bootloader-invoked prekernel test software, like memtest86+, by default.

Quoted 8 lines

Quoted 6 lines
the current memtest86 is running in 32 bit mode, and only support 64G ram.
I tried to expand that a bit, to support 1024g, but it only works on some machine.
could be stack provide is not big enough?
Wonder how hard it would be to make it run 64 bits...

1. in 32 bit test less than 4g in 32 bit mode
2. switch to 64 bit, set page table under 4g to cover all ram...

Jan Engelhardt

2008-03-21 21:22 UTC #9 lore raw

In reply to Ingo Molnar

Quoted 11 lines · Ingo Molnar

Quoted 4 lines · Yinghai Lu
do simple memtest after init_memory_mapping
use find_e820_area_size to find all ram range that is not reserved. and do some simple bits test to find some bad ram.
if find some bad ram, use reserve_early to exclude that range.
very nice patch! I always thought that this was the proper way to do memtest - and we could in fact also do something like this after SMP bringup, and hit the memory bus via multiple CPUs. [that will need a different enumeration though than e820 maps]

Perhaps this can even be used to provide on-the-fly badram patch semantics?

Yinghai Lu

2008-03-21 21:43 UTC #10 lore raw

Quoted 9 lines · Jan Engelhardt

Quoted 7 lines · Ingo Molnar
Quoted 4 lines · Yinghai Lu
do simple memtest after init_memory_mapping
use find_e820_area_size to find all ram range that is not reserved. and do some simple bits test to find some bad ram.
if find some bad ram, use reserve_early to exclude that range.

Quoted 7 lines

Quoted 4 lines
very nice patch! I always thought that this was the proper way to do memtest - and we could in fact also do something like this after SMP bringup, and hit the memory bus via multiple CPUs. [that will need a different enumeration though than e820 maps]
Perhaps this can even be used to provide on-the-fly badram patch semantics?

yes. but bad ranges can not be too many. otherwise early_res array will overflow. then need to use memmap=nn$ss to exclude range already found.

Sami Farin

2008-03-21 22:53 UTC #15 lore raw

In reply to Yinghai Lu

Quoted 8 lines · Yinghai Lu

do simple memtest after init_memory_mapping
use find_e820_area_size to find all ram range that is not reserved.
and do some simple bits test to find some bad ram.
if find some bad ram, use reserve_early to exclude that range.

Does somebody still remember the bug report in which the fault was found to be in hardware? By bisecting, the user got "working" kernel when movnti was not used to copy data.
Would be neat if also non-temporal moves were done in this early memtest.

Having the memtest feature in kernel is useful,
considering my grub can not load memtest86+ binary, failing with error "Selected item cannot fit into memory",
with or without the patch at
https://bugzilla.redhat.com/show_bug.cgi?id=237279

I have user-space app to test movnti... ask if you want it.

Signature · 1 line

Do what you love because life is too short for anything else.

Yinghai Lu

2008-03-22 00:04 UTC #11 lore raw

In reply to Ingo Molnar

Quoted 7 lines · Ingo Molnar

Quoted 4 lines · Yinghai Lu
do simple memtest after init_memory_mapping
use find_e820_area_size to find all ram range that is not reserved.

Quoted 8 lines

Quoted 3 lines
and do some simple bits test to find some bad ram.
if find some bad ram, use reserve_early to exclude that range.
very nice patch! I always thought that this was the proper way to do memtest - and we could in fact also do something like this after SMP bringup, and hit the memory bus via multiple CPUs. [that will need a different enumeration though than e820 maps]

or
1. core0/node0 check all memory at first
2. every core0 will check all memory later one by one.

Quoted 3 lines

one structural observation: please make this unified functionality, so that 32-bit kernels can make use of it too.

that will need to use PAE to switch 2G windows ....

Quoted 6 lines

a small style nit from scripts/checkpatch.pl:
Quoted 1 line
+// printk(KERN_DEBUG "find_e820_area_size : e820 %d [%llx, %lx]\n", i, ei->addr, ei_last);

Quoted 8 lines

remove such lines or make them pr_debug(). (checkpatch also found more such cases)
also, please add a CONFIG_BOOTPARAM_MEMTEST=y option so that distributions can enable this by default in their debug kernels.
i've applied your current version to get some testing, please send delta patches against x86/latest.

thanks. will submit delta patch.

Willy Tarreau

2008-03-22 05:46 UTC #4 lore raw

In reply to Ingo Molnar

Quoted 12 lines · Ingo Molnar

Quoted 9 lines · H. Peter Anvin
Quoted 8 lines
very nice patch! I always thought that this was the proper way to do memtest - and we could in fact also do something like this after SMP bringup, and hit the memory bus via multiple CPUs. [that will need a different enumeration though than e820 maps]
one structural observation: please make this unified functionality, so that 32-bit kernels can make use of it too.

Quoted 8 lines

Quoted 3 lines
Indeed. Of course, it would also be nice if distros shipped bootloader-invoked prekernel test software, like memtest86+, by default.
some do (Fedora for example), but it's still a bit quirky for users to invoke and it would be nice to see those results in the kernel log as well and flag possibly flaky systems that way. (add a taint bit, etc., etc.)

It may even make sense to merge in the full memtest86. The code is small (both source and binary) and IIRC it shares a lot of init code with x86. The remaining problem would then be how to maintain its tests up do date.

Willy

Yinghai Lu

2008-03-22 06:48 UTC #5 lore raw

Quoted 18 lines · Willy Tarreau

Quoted 16 lines · Ingo Molnar
Quoted 12 lines · H. Peter Anvin
Quoted 8 lines
very nice patch! I always thought that this was the proper way to do memtest - and we could in fact also do something like this after SMP bringup, and hit the memory bus via multiple CPUs. [that will need a different enumeration though than e820 maps]
one structural observation: please make this unified functionality, so that 32-bit kernels can make use of it too.
Indeed. Of course, it would also be nice if distros shipped bootloader-invoked prekernel test software, like memtest86+, by default.

Quoted 8 lines

Quoted 4 lines
some do (Fedora for example), but it's still a bit quirky for users to invoke and it would be nice to see those results in the kernel log as well and flag possibly flaky systems that way. (add a taint bit, etc., etc.)
It may even make sense to merge in the full memtest86. The code is small (both source and binary) and IIRC it shares a lot of init code with x86. The remaining problem would then be how to maintain its tests up do date.

memtester is another choice, and it is more easy to be merged.

Ingo Molnar

2008-03-22 12:04 UTC #12 lore raw

In reply to Yinghai Lu

Quoted 4 lines · Yinghai Lu

Quoted 2 lines
one structural observation: please make this unified functionality, so that 32-bit kernels can make use of it too.
that will need to use PAE to switch 2G windows ....

well, please try some non-PAE, checks-direct-mappings approach - if someone wants to extend it to the highmem bits i'm sure it will be done.

Ingo

Yinghai Lu

2008-03-22 17:00 UTC #13 lore raw

Quoted 10 lines · Ingo Molnar

Quoted 4 lines · Yinghai Lu
Quoted 2 lines
one structural observation: please make this unified functionality, so that 32-bit kernels can make use of it too.
that will need to use PAE to switch 2G windows ....
well, please try some non-PAE, checks-direct-mappings approach - if someone wants to extend it to the highmem bits i'm sure it will be done.

OK, First need to move some early_res code from e820_64.c to e820_32.c

or we can start to merge them. anyone is working on that?

Arjan van de Ven

2008-03-22 20:51 UTC #16 lore raw

In reply to Yinghai Lu

Quoted 4 lines · On Thu, 20 Mar 2008 23:58:33 -0700 Yinghai Lu <[email protected]>…

do simple memtest after init_memory_mapping
use find_e820_area_size to find all ram range that is not reserved.

be careful, there's some special memory that e820 right now says is not reserved, but still has bios data (the first 4Kb of memory come to mind)

Signature · 3 lines

If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

H. Peter Anvin

reviewer

2008-03-22 21:05 UTC #17 lore raw

Quoted 10 lines · Arjan van de Ven

Quoted 3 lines · On Thu, 20 Mar 2008 23:58:33 -0700 Yinghai Lu <[email protected]>…
do simple memtest after init_memory_mapping
use find_e820_area_size to find all ram range that is not reserved.
be careful, there's some special memory that e820 right now says is not reserved, but still has bios data (the first 4Kb of memory come to mind)

Is that true even after Yinghai's changes? I have lost track of all the patches...

-hpa

Yinghai Lu

2008-03-22 21:21 UTC #19 lore raw

In reply to Arjan van de Ven

Quoted 6 lines · Arjan van de Ven

On Thu, 20 Mar 2008 23:58:33 -0700
Quoted 1 line · Yinghai Lu

Quoted 6 lines

Quoted 3 lines
do simple memtest after init_memory_mapping
use find_e820_area_size to find all ram range that is not reserved.
be careful, there's some special memory that e820 right now says is not reserved, but still has bios data (the first 4Kb of memory come to mind)

only test ranges that have E820_RAM and exclude range that is early_reserved... so it is safe.

Yinghai Lu

2008-03-22 21:21 UTC #18 lore raw

In reply to H. Peter Anvin

Quoted 6 lines · H. Peter Anvin

Quoted 5 lines · Arjan van de Ven
Quoted 2 lines · On Thu, 20 Mar 2008 23:58:33 -0700 Yinghai Lu <[email protected]>…
do simple memtest after init_memory_mapping

Quoted 8 lines

Quoted 5 lines
Quoted 1 line
use find_e820_area_size to find all ram range that is not reserved.
be careful, there's some special memory that e820 right now says is not reserved, but still has bios data (the first 4Kb of memory come to mind)
Is that true even after Yinghai's changes? I have lost track of all the patches...

find_e820_area_size should be safe.

Ingo Molnar

2008-03-25 11:01 UTC #14 lore raw

In reply to Yinghai Lu

Quoted 7 lines · Yinghai Lu

Quoted 6 lines · Ingo Molnar
Quoted 3 lines · Yinghai Lu
Quoted 2 lines
one structural observation: please make this unified functionality, so that 32-bit kernels can make use of it too.

Quoted 7 lines

Quoted 5 lines
Quoted 1 line
that will need to use PAE to switch 2G windows ....
well, please try some non-PAE, checks-direct-mappings approach - if someone wants to extend it to the highmem bits i'm sure it will be done.
OK, First need to move some early_res code from e820_64.c to e820_32.c

yeah, please do that.

Quoted 1 line

or we can start to merge them. anyone is working on that?

not that i know of - feel free.

Ingo