Skip to contentSkip to conversation

2.6.18-rc1-git4 and 2.6.18-rc1-mm1 OOM's on boot

DISCUSSION Martin Bligh 4 participants 8 messages Jul 12, 2006 – Jul 14, 2006 VFS

lore
Share & tools
lore
#0 lore raw

-git3 was fine
(bootlog for git3: http://test.kernel.org/abat/40748/debug/console.log)

-mm1 has the same issue

Slightly different manifestations across 2 boots

http://test.kernel.org/abat/40760/debug/console.log
http://test.kernel.org/abat/40837/debug/console.log

32GB NUMA-Q system w/16 processors.

Swap cache: add 0, delete 0, find 0/0, race 0+0 Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
8321024 pages of RAM
8159232 pages of HIGHMEM
133127 reserved pages
3739 pages shared
0 pages swap cached
208 pages dirty
0 pages writeback
1135 pages mapped
24266 pages slab
76 pages pagetables
Out of Memory: Kill process 1 (init) score 0 and children. No available memory (MPOL_BIND): Killed process 1267 (rc). -- 0:conmux-control -- time-stamp -- Jul/12/06 2:00:36 -- -- 0:conmux-control -- time-stamp -- Jul/12/06 2:09:55 -- (bot:conmon-payload) disconnected

#1 lore raw
Quoted 9 lines · On Wed, 12 Jul 2006 09:53:08 -0700 Martin Bligh <[email protected]> wrote:

-git3 was fine
(bootlog for git3: http://test.kernel.org/abat/40748/debug/console.log)

-mm1 has the same issue

Slightly different manifestations across 2 boots

http://test.kernel.org/abat/40760/debug/console.log
http://test.kernel.org/abat/40837/debug/console.log

 [<c0136fcf>] out_of_memory+0x29/0xf6
 [<c0137f48>] __alloc_pages+0x1ed/0x276
 [<c014db73>] kmem_getpages+0x63/0xc1
 [<c014e960>] cache_grow+0xaa/0x139
 [<c014eb6a>] cache_alloc_refill+0x17b/0x1c0
 [<c014f1ef>] __kmalloc+0x83/0x93
 [<c0168cf5>] alloc_fd_array+0x19/0x24
 [<c0169122>] alloc_fdtable+0xb2/0xef
 [<c016917f>] expand_fdtable+0x20/0x7d
 [<c0169221>] expand_files+0x45/0x50
 [<c0161263>] locate_fd+0x70/0x8e
 [<c01612aa>] dupfd+0x29/0x61
 [<c01613dc>] sys_dup+0x1b/0x23
 [<c01027d3>] syscall_call+0x7/0xb

I suspect that's because I had me a little mistake.

fs/file.c renamed from fs/file.c~alloc_fdtable-expansion-fix +1 -1
@@ -240,7 +240,7 @@ static struct fdtable *alloc_fdtable(int
 	if (!fdt)
   		goto out;
 
-	nfds = max_t(int, 8 * L1_CACHE_BYTES, roundup_pow_of_two(nfds));
+	nfds = max_t(int, 8 * L1_CACHE_BYTES, roundup_pow_of_two(nr + 1));
 	if (nfds > NR_OPEN)
 		nfds = NR_OPEN;
 

_

#2 lore raw
Quoted 43 lines · Andrew Morton
Quoted 9 lines · On Wed, 12 Jul 2006 09:53:08 -0700 Martin Bligh <[email protected]> wrote:

-git3 was fine
(bootlog for git3: http://test.kernel.org/abat/40748/debug/console.log)

-mm1 has the same issue

Slightly different manifestations across 2 boots

http://test.kernel.org/abat/40760/debug/console.log
http://test.kernel.org/abat/40837/debug/console.log

 [<c0136fcf>] out_of_memory+0x29/0xf6
 [<c0137f48>] __alloc_pages+0x1ed/0x276
 [<c014db73>] kmem_getpages+0x63/0xc1
 [<c014e960>] cache_grow+0xaa/0x139
 [<c014eb6a>] cache_alloc_refill+0x17b/0x1c0
 [<c014f1ef>] __kmalloc+0x83/0x93
 [<c0168cf5>] alloc_fd_array+0x19/0x24
 [<c0169122>] alloc_fdtable+0xb2/0xef
 [<c016917f>] expand_fdtable+0x20/0x7d
 [<c0169221>] expand_files+0x45/0x50
 [<c0161263>] locate_fd+0x70/0x8e
 [<c01612aa>] dupfd+0x29/0x61
 [<c01613dc>] sys_dup+0x1b/0x23
 [<c01027d3>] syscall_call+0x7/0xb

I suspect that's because I had me a little mistake.

fs/file.c renamed from fs/file.c~alloc_fdtable-expansion-fix +1 -1
@@ -240,7 +240,7 @@ static struct fdtable *alloc_fdtable(int
 	if (!fdt)
   		goto out;
 
-	nfds = max_t(int, 8 * L1_CACHE_BYTES, roundup_pow_of_two(nfds));
+	nfds = max_t(int, 8 * L1_CACHE_BYTES, roundup_pow_of_two(nr + 1));
 	if (nfds > NR_OPEN)
 		nfds = NR_OPEN;
 
Quoted 2 lines

_

Thanks, that was affecting several machines.

Andy, any chance we can do an across-all-machines run of that one on top of -mm1? Thanks,

M.

#3 lore raw
Quoted 49 lines · Martin Bligh
Quoted 43 lines · Andrew Morton
Quoted 9 lines · On Wed, 12 Jul 2006 09:53:08 -0700 Martin Bligh <[email protected]> wrote:

-git3 was fine
(bootlog for git3: http://test.kernel.org/abat/40748/debug/console.log)

-mm1 has the same issue

Slightly different manifestations across 2 boots

http://test.kernel.org/abat/40760/debug/console.log
http://test.kernel.org/abat/40837/debug/console.log

 [<c0136fcf>] out_of_memory+0x29/0xf6
 [<c0137f48>] __alloc_pages+0x1ed/0x276
 [<c014db73>] kmem_getpages+0x63/0xc1
 [<c014e960>] cache_grow+0xaa/0x139
 [<c014eb6a>] cache_alloc_refill+0x17b/0x1c0
 [<c014f1ef>] __kmalloc+0x83/0x93
 [<c0168cf5>] alloc_fd_array+0x19/0x24
 [<c0169122>] alloc_fdtable+0xb2/0xef
 [<c016917f>] expand_fdtable+0x20/0x7d
 [<c0169221>] expand_files+0x45/0x50
 [<c0161263>] locate_fd+0x70/0x8e
 [<c01612aa>] dupfd+0x29/0x61
 [<c01613dc>] sys_dup+0x1b/0x23
 [<c01027d3>] syscall_call+0x7/0xb

I suspect that's because I had me a little mistake.

fs/file.c renamed from fs/file.c~alloc_fdtable-expansion-fix +1 -1
@@ -240,7 +240,7 @@ static struct fdtable *alloc_fdtable(int
     if (!fdt)
           goto out;
 
-    nfds = max_t(int, 8 * L1_CACHE_BYTES, roundup_pow_of_two(nfds));
+    nfds = max_t(int, 8 * L1_CACHE_BYTES, roundup_pow_of_two(nr + 1));
     if (nfds > NR_OPEN)
         nfds = NR_OPEN;
 
Quoted 2 lines

_

Thanks, that was affecting several machines.

Quoted 4 lines

Andy, any chance we can do an across-all-machines run of that one on top of -mm1? Thanks,

M.

Yep, I've run this with badari's fix as a set across the whole family. I did all dbenchall runs for now as this example is showing on that and badari's is triggered same. If there is any measure of success there I'll throw in the externals too.

-apw

#4 lore raw
Quoted 50 lines · Andy Whitcroft
Quoted 49 lines · Martin Bligh
Quoted 43 lines · Andrew Morton
Quoted 9 lines · On Wed, 12 Jul 2006 09:53:08 -0700 Martin Bligh <[email protected]> wrote:

-git3 was fine
(bootlog for git3: http://test.kernel.org/abat/40748/debug/console.log)

-mm1 has the same issue

Slightly different manifestations across 2 boots

http://test.kernel.org/abat/40760/debug/console.log
http://test.kernel.org/abat/40837/debug/console.log

 [<c0136fcf>] out_of_memory+0x29/0xf6
 [<c0137f48>] __alloc_pages+0x1ed/0x276
 [<c014db73>] kmem_getpages+0x63/0xc1
 [<c014e960>] cache_grow+0xaa/0x139
 [<c014eb6a>] cache_alloc_refill+0x17b/0x1c0
 [<c014f1ef>] __kmalloc+0x83/0x93
 [<c0168cf5>] alloc_fd_array+0x19/0x24
 [<c0169122>] alloc_fdtable+0xb2/0xef
 [<c016917f>] expand_fdtable+0x20/0x7d
 [<c0169221>] expand_files+0x45/0x50
 [<c0161263>] locate_fd+0x70/0x8e
 [<c01612aa>] dupfd+0x29/0x61
 [<c01613dc>] sys_dup+0x1b/0x23
 [<c01027d3>] syscall_call+0x7/0xb

I suspect that's because I had me a little mistake.

fs/file.c renamed from fs/file.c~alloc_fdtable-expansion-fix +1 -1
@@ -240,7 +240,7 @@ static struct fdtable *alloc_fdtable(int
     if (!fdt)
           goto out;
 
-    nfds = max_t(int, 8 * L1_CACHE_BYTES, roundup_pow_of_two(nfds));
+    nfds = max_t(int, 8 * L1_CACHE_BYTES, roundup_pow_of_two(nr + 1));
     if (nfds > NR_OPEN)
         nfds = NR_OPEN;
 
Quoted 2 lines

_

Thanks, that was affecting several machines.

Quoted 6 lines
Quoted 4 lines

Andy, any chance we can do an across-all-machines run of that one on top of -mm1? Thanks,

M.

Yep, I've run this with badari's fix as a set across the whole family. I did all dbenchall runs for now as this example is showing on that and badari's is triggered same. If there is any measure of success there I'll throw in the externals too.

General goodness from this one. Except where we're getting issues with the e1000's. That seems to be fixed up by backing out some driver changes.

All moot, as -mm2 is showing similar goodness.

-apw

#5 lore raw
Quoted 9 lines · On Fri, 14 Jul 2006 09:00:36 +0100 Andy Whitcroft <[email protected]> wrote:
Quoted 4 lines

Yep, I've run this with badari's fix as a set across the whole family. I did all dbenchall runs for now as this example is showing on that and badari's is triggered same. If there is any measure of success there I'll throw in the externals too.

General goodness from this one. Except where we're getting issues with the e1000's. That seems to be fixed up by backing out some driver changes.

All moot, as -mm2 is showing similar goodness.

Is -mm2's e1000 OK?

#6 lore raw
Quoted 9 lines · Andrew Morton
Quoted 4 lines · On Fri, 14 Jul 2006 09:00:36 +0100 Andy Whitcroft <[email protected]> wrote:
Quoted 1 line

Yep, I've run this with badari's fix as a set across the whole family. I did all dbenchall runs for now as this example is showing on that and badari's is triggered same. If there is any measure of success there I'll throw in the externals too.

General goodness from this one. Except where we're getting issues with the e1000's. That seems to be fixed up by backing out some driver changes.

All moot, as -mm2 is showing similar goodness.

Is -mm2's e1000 OK?

Whilst calling it the e1000 problem (that was how it was originally reported) I should say that this was related to the sysfs change in the following patches:

gregkh-driver-network-class_device-to-device.patch gregkh-driver-class_device_rename-remove.patch

I have two boxes under test which were failing on -mm1 similar to teh following (from userland):

    eth-id-00:02:55:d3:37:4a            No interface found

Both are booting -mm2 fine.

I can only see two outstanding issues. An IDE lost interrupt issue on a blade we have under test which I believe benh is looking at, and what looks like an s390 tool chain issue which I am told is being looked at.

-apw

#7 lore raw
Quoted 4 lines

I can only see two outstanding issues. An IDE lost interrupt issue on a blade we have under test which I believe benh is looking at, and what looks like an s390 tool chain issue which I am told is being looked at.

Yeah, well, I'm trying to look at it... between flights :)

Ben.