Skip to contentSkip to conversation

2.6.17-rc5-mm1

DISCUSSION Martin Bligh 10 participants 56 messages May 31, 2006 – Jun 7, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

The x86_65 panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.

M

http://test.kernel.org/abat/33807/debug/console.log

nable to handle kernel paging request at 0000000000001034 RIP:
[<ffffffff8048c5c4>] __sched_text_start+0x51c/0x745
PGD 1d836a067 PUD 1e2704067 PMD 0

Oops: 0002 [1] SMP
last sysfs file: /devices/pci0000:00/0000:00:06.0/resource
CPU 0
Modules linked in:
Pid: 230, comm: kswapd0 Not tainted 2.6.17-rc5-mm1-autokern1 #1
RIP: 0010:[<ffffffff8048c5c4>]  [<ffffffff8048c5c4>] __sched_text_start+0x51c/0x745
RSP: 0000:ffff810100237b48  EFLAGS: 00010093
RAX: ffff8100e3c592b0 RBX: 0000000000000d84 RCX: ffffffff805f0cd8
RDX: 000000003b9a078a RSI: ffff8100e3c590d0 RDI: 000000000007989e
RBP: ffff810100237be8 R08: ffff81007bfec930 R09: ffff81007bfec8e8
R10: ffff81007bfec8e8 R11: ffff81007ff3c2d8 R12: 0000000000000000
R13: ffff8101fd8ec140 R14: ffff810001011680 R15: 000002b3bb3b33a4
FS:  0000000000000000(0000) GS:ffffffff8061c000(0000) knlGS:00000000f7db6460
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000001034 CR3: 00000001f18fa000 CR4: 00000000000006e0
Process kswapd0 (pid: 230, threadinfo ffff810100236000, task ffff8100e3c590d0)
Stack: ffff81007bfec8e8 ffff81007bfec8e8 ffffffff805f0cd8 000000007ff3c2d8
       ffff8100e3c590d0 000000000000c276 ffffffff805f0cd8 ffff8100e3c592b0
       ffff81007be407b0 ffff81007be407b0
Call Trace:
 [<ffffffff8024d24c>] __remove_from_page_cache+0x1c/0x65
 [<ffffffff8048d16c>] cond_resched+0x4c/0x7b
 [<ffffffff8025783d>] shrink_inactive_list+0x125/0x882
 [<ffffffff8048c7ed>] thread_return+0x0/0xd1
 [<ffffffff80258493>] shrink_zone+0xd2/0xf3
 [<ffffffff80258a4b>] kswapd+0x359/0x49b
 [<ffffffff8023e51c>] autoremove_wake_function+0x0/0x37
 [<ffffffff802586f2>] kswapd+0x0/0x49b
 [<ffffffff8023dff8>] kthread+0xd0/0xfc
 [<ffffffff8020a34e>] child_rip+0x8/0x12
 [<ffffffff8023df28>] kthread+0x0/0xfc
 [<ffffffff8020a346>] child_rip+0x0/0x12
#1 lore raw
Quoted 38 lines · Martin Bligh

The x86_65 panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.

M

http://test.kernel.org/abat/33807/debug/console.log

nable to handle kernel paging request at 0000000000001034 RIP:
[<ffffffff8048c5c4>] __sched_text_start+0x51c/0x745
PGD 1d836a067 PUD 1e2704067 PMD 0

Oops: 0002 [1] SMP
last sysfs file: /devices/pci0000:00/0000:00:06.0/resource
CPU 0
Modules linked in:
Pid: 230, comm: kswapd0 Not tainted 2.6.17-rc5-mm1-autokern1 #1
RIP: 0010:[<ffffffff8048c5c4>]  [<ffffffff8048c5c4>] 
__sched_text_start+0x51c/0x745
RSP: 0000:ffff810100237b48  EFLAGS: 00010093
RAX: ffff8100e3c592b0 RBX: 0000000000000d84 RCX: ffffffff805f0cd8
RDX: 000000003b9a078a RSI: ffff8100e3c590d0 RDI: 000000000007989e
RBP: ffff810100237be8 R08: ffff81007bfec930 R09: ffff81007bfec8e8
R10: ffff81007bfec8e8 R11: ffff81007ff3c2d8 R12: 0000000000000000
R13: ffff8101fd8ec140 R14: ffff810001011680 R15: 000002b3bb3b33a4
FS:  0000000000000000(0000) GS:ffffffff8061c000(0000) knlGS:00000000f7db6460
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000001034 CR3: 00000001f18fa000 CR4: 00000000000006e0
Process kswapd0 (pid: 230, threadinfo ffff810100236000, task 
ffff8100e3c590d0)
Stack: ffff81007bfec8e8 ffff81007bfec8e8 ffffffff805f0cd8 000000007ff3c2d8
        ffff8100e3c590d0 000000000000c276 ffffffff805f0cd8 ffff8100e3c592b0
        ffff81007be407b0 ffff81007be407b0
Call Trace:
  [<ffffffff8024d24c>] __remove_from_page_cache+0x1c/0x65
  [<ffffffff8048d16c>] cond_resched+0x4c/0x7b
  [<ffffffff8025783d>] shrink_inactive_list+0x125/0x882
  [<ffffffff8048c7ed>] thread_return+0x0/0xd1
Quoted 8 lines
  [<ffffffff80258493>] shrink_zone+0xd2/0xf3
  [<ffffffff80258a4b>] kswapd+0x359/0x49b
  [<ffffffff8023e51c>] autoremove_wake_function+0x0/0x37
  [<ffffffff802586f2>] kswapd+0x0/0x49b
  [<ffffffff8023dff8>] kthread+0xd0/0xfc
  [<ffffffff8020a34e>] child_rip+0x8/0x12
  [<ffffffff8023df28>] kthread+0x0/0xfc
  [<ffffffff8020a346>] child_rip+0x0/0x12

What are you doing to make this happen?

#2 lore raw
Quoted 6 lines · Andrew Morton
Quoted 2 lines · Martin Bligh

The x86_65 panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.

What are you doing to make this happen?

runalltests on LTP

M.

#3 lore raw
Quoted 11 lines · Martin J. Bligh
Quoted 6 lines · Andrew Morton
Quoted 2 lines · Martin Bligh

The x86_65 panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.

What are you doing to make this happen?

runalltests on LTP

We have to get to the bottom of this - there's a shadow over about 500 patches and we don't know which.

iirc I tried to reproduce this a couple of weeks back and failed.

Are you able to narrow it down to a particular LTP test? It was mtest01 or something like that? Perhaps we can identify a particular command line which triggers the fault in a standalone fashion?

And why can't I make it happen? Perhaps it's a memory initialisation problem, and it only happens to hit in that stage of LTP because that's when you started doing page reclaim, or something? Perhaps just try putting a heap of memory pressure on the machine, see what that does?

Being unable to reproduce it and not having a theory to go on leaves us kinda stuck. Help, please?

#4 lore raw
Quoted 10 lines · Andrew Morton
Quoted 8 lines · Martin J. Bligh
Quoted 6 lines · Andrew Morton
Quoted 2 lines · Martin Bligh

The x86_65 panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

What are you doing to make this happen?

runalltests on LTP

We have to get to the bottom of this - there's a shadow over about 500 patches and we don't know which.

We did do one chop, and concluded it wasn't the x86_64 patches.

Quoted 1 line

iirc I tried to reproduce this a couple of weeks back and failed.

It looks like a different panic to me. It was a double-fault before.

Quoted 3 lines

Are you able to narrow it down to a particular LTP test? It was mtest01 or something like that? Perhaps we can identify a particular command line which triggers the fault in a standalone fashion?

I can't do much from here - it's running on an IBM machine. Have to beg Andy, or one of the other IBMers, for help.

Quoted 3 lines

And why can't I make it happen? Perhaps it's a memory initialisation problem, and it only happens to hit in that stage of LTP because that's when you started doing page reclaim, or something?

It consistently happens on -mm, and not mainline, flicking back and forth over time. So if you mean h/w mem init, I don't think so. if you mena some patch in -mm, then yes.

Quoted 2 lines

Perhaps just try putting a heap of memory pressure on the machine,
see what that does?

Yes, the other stuff might not be swapping.

Quoted 2 lines

Being unable to reproduce it and not having a theory to go on leaves us kinda stuck. Help, please?

Yeah, we have a sniff-testing mechanism that works well. However, drill-down still requires significant amounts of human intervention. The next gen of stuff should help do more intelligent stuff, but we're kind of stuck with human-ness for now.

M.

#5 lore raw
Quoted 55 lines · Martin J. Bligh
Quoted 14 lines · Andrew Morton
Quoted 12 lines · Martin J. Bligh
Quoted 8 lines · Andrew Morton
Quoted 2 lines · Martin Bligh

The x86_65 panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.

What are you doing to make this happen?

Quoted 6 lines
Quoted 2 lines

runalltests on LTP

We have to get to the bottom of this - there's a shadow over about 500 patches and we don't know which.

We did do one chop, and concluded it wasn't the x86_64 patches.

Quoted 1 line

iirc I tried to reproduce this a couple of weeks back and failed.

It looks like a different panic to me. It was a double-fault before.

Quoted 4 lines

Are you able to narrow it down to a particular LTP test? It was mtest01 or
something like that? Perhaps we can identify a particular command line which triggers the fault in a standalone fashion?

I can't do much from here - it's running on an IBM machine. Have to beg Andy, or one of the other IBMers, for help.

Quoted 3 lines

And why can't I make it happen? Perhaps it's a memory initialisation problem, and it only happens to hit in that stage of LTP because that's when you started doing page reclaim, or something?

It consistently happens on -mm, and not mainline, flicking back and forth over time. So if you mean h/w mem init, I don't think so. if you mena some patch in -mm, then yes.

Quoted 3 lines

Perhaps just try putting a heap of memory pressure on the machine,

see what that does?

Yes, the other stuff might not be swapping.

Quoted 8 lines
Quoted 2 lines

Being unable to reproduce it and not having a theory to go on leaves us kinda stuck. Help, please?

Yeah, we have a sniff-testing mechanism that works well. However, drill-down still requires significant amounts of human intervention. The next gen of stuff should help do more intelligent stuff, but we're kind of stuck with human-ness for now.

I am sure I got half way through diagnosing this one. We were context switching to a bad thread. I've been meaning to get back to it. Its at the top of my list for the AM.

-apw

#6 lore raw
In reply to Martin J. Bligh
Quoted 9 lines · Martin J. Bligh
Quoted 7 lines · Andrew Morton
Quoted 5 lines · Martin J. Bligh
Quoted 3 lines · Andrew Morton

Martin Bligh <[email protected]> wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

The x86_65 panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.

What are you doing to make this happen?

runalltests on LTP

Ok, I think this could well be the same problem I got half way through tracking last time round. We are still handing off threads with non-initialised stacks.

APW: schedule: ffffffff805f0cd8 bad rsp flags=00000000

Kernel panic - not syncing: BAD STACK POINTER

Interestingly this has remained unchanged for dispite the major churn -mm takes so this could well be the underlying issue as we almost always blow up randomly when we do this, in a mess with no stack to work with.

Last time I tried to split search -mm1 and she was being a hideous pig, I just couldn't get any bit of it to compile without the rest. Will try and track this down with the new -mm.

-apw

#7 lore raw
Quoted 2 lines · On Thu, 01 Jun 2006 17:34:10 +0100 Andy Whitcroft <[email protected]> wrote:

Last time I tried to split search -mm1 and she was being a hideous pig, I just couldn't get any bit of it to compile without the rest.

You shouldn't bisect blindly.

Looking at the series file (ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.17-rc5/2.6.17-rc5-mm2/patch-series):

...
add-scsi_add_host-failure-handling-for-nsp32.patch qla1280-fix-section-mismatch-warnings.patch bogus-disk-geometry-on-large-disks.patch bogus-disk-geometry-on-large-disks-warning-fix.patch megaraid_sas-switch-fw_outstanding-to-an-atomic_t.patch megaraid_sas-add-support-for-zcr-controller.patch megaraid_sas-add-support-for-zcr-controller-fix.patch
...

megaraid_sas-add-support-for-zcr-controller-fix.patch is a fix against megaraid_sas-add-support-for-zcr-controller.patch, so if the bisection lands you on megaraid_sas-add-support-for-zcr-controller.patch then obviously one should apply megaraid_sas-add-support-for-zcr-controller-fix.patch by hand as well.

You'll often find great streams of similarly-named contiguous patches - you should apply either all of them or none of them. Otherwise you're testing a known-to-be-broken kernel which has an already-available fix ;)

http://www.zip.com.au/~akpm/linux/patches/stuff/bisecting-mm-trees.txt
describes the bisection technique I use.

If you're using git-bisect, well, try to have a nice day anyway.

Quoted 1 line

Will try and track this down with the new -mm.

Would be great, thanks.

#8 lore raw
In reply to Andy Whitcroft
Quoted 32 lines · Andy Whitcroft
Quoted 13 lines · Martin J. Bligh
Quoted 10 lines · Andrew Morton
Quoted 7 lines · Martin J. Bligh
Quoted 4 lines · Andrew Morton

Martin Bligh <[email protected]> wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

The x86_65 panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.

What are you doing to make this happen?

runalltests on LTP

Ok, I think this could well be the same problem I got half way through tracking last time round. We are still handing off threads with non-initialised stacks.

APW: schedule: ffffffff805f0cd8 bad rsp flags=00000000

Kernel panic - not syncing: BAD STACK POINTER
Quoted 7 lines

Interestingly this has remained unchanged for dispite the major churn -mm takes so this could well be the underlying issue as we almost always blow up randomly when we do this, in a mess with no stack to work with.

Last time I tried to split search -mm1 and she was being a hideous pig, I just couldn't get any bit of it to compile without the rest. Will try and track this down with the new -mm.

Ok. Did a split search on -mm2 for this. With the full stack I was still tripping up on the bad thread hand-off trigger above. However, when split searching I seemed to get somewhat different panics pretty commonly in the allocator. My split search led me to the start of the swapless page migration patches:

GOOD:page-migration-cleanup-pass-mapping-to-migration-functions.patch GOOD:page-migration-cleanup-move-fallback-handling-into-special-function.patch ----:swapless-pm-add-r-w-migration-entries.patch -BAD:swapless-pm-add-r-w-migration-entries-fix-2.patch

I tried pretty hard to slide this out of -mm and can't say with cirtainty that I did it right. I backed out the following in order to get the first two off (obviously from the bottom up):

swapless-pm-add-r-w-migration-entries.patch swapless-pm-add-r-w-migration-entries-fix-2.patch swapless-page-migration-rip-out-swap-based-logic.patch swapless-page-migration-modify-core-logic.patch more-page-migration-do-not-inc-dec-rss-counters.patch more-page-migration-use-migration-entries-for-file-pages.patch page-migration-update-documentation.patch page-migration-simplify-migrate_pages.patch page-migration-simplify-migrate_pages-tweaks.patch page-migration-handle-freeing-of-pages-in-migrate_pages.patch page-migration-use-allocator-function-for-migrate_pages.patch page-migration-support-moving-of-individual-pages.patch page-migration-detailed-status-for-moving-of-individual-pages.patch page-migration-support-moving-of-individual-pages-fixes.patch page-migration-support-moving-of-individual-pages-x86_64-support.patch page-migration-support-moving-of-individual-pages-x86-support.patch
page-migration-support-moving-of-individual-pages-x86-support-fix.patch
page-migration-support-a-vma-migration-function.patch allow-migration-of-mlocked-pages.patch

That did seem to get rid of the original error. However, I'm now
experiencing a reiserfs4 panic so I'm not 100% confident the problem really is gone, but the reiserfs panic isn't anywhere as near hard as the panic's I have been experiencing; I was able to reboot the machine even though the tests were not completing correctly. Full panic for that one is below:

general protection fault: 0000 [1] SMP
last sysfs file: /devices/pci0000:00/0000:00:06.0/resource
CPU 1
Modules linked in:
Pid: 19163, comm: mkdir09 Not tainted 2.6.17-rc5-mm2-autokern1 #1
RIP: 0010:[<ffffffff802412b0>]  [<ffffffff802412b0>]
check_deadlock+0x1f/0x117
RSP: 0000:ffff8101fb803dd8  EFLAGS: 00010002
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
RDX: ffff8101f3e64000 RSI: 0000000000000001 RDI: 2222222222222222
RBP: ffff8101fb803df8 R08: 0000000000000000 R09: ffff8101fb803e68
R10: ffff8101fb802000 R11: 00000000ffffff9c R12: ffff8101fcafa7b0
R13: 2222222222222222 R14: 2222222222222222 R15: 0000000000000000
FS:  000000000804a020(0000) GS:ffff8100e3f27e40(0063) knlGS:00000000f7e15460
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00000000f7ee2090 CR3: 00000001f4dce000 CR4: 00000000000006e0
Process mkdir09 (pid: 19163, threadinfo ffff8101fb802000, task
ffff8101fdd4b0d0)
Stack: 0000000000000000 ffff8101fcafa7b0 ffff8101f3ae5988 2222222222222222
       ffff8101fb803e28 ffffffff80241361 ffff8101f3ae5988 ffff8101fb802000
       ffff8101fb803e68 ffff8101fdd4b0d0
Call Trace:
 [<ffffffff80241361>] check_deadlock+0xd0/0x117
 [<ffffffff80241574>] debug_mutex_add_waiter+0x57/0x6f
 [<ffffffff8048cf47>] __mutex_lock_slowpath+0xc7/0x232
 [<ffffffff802800bc>] do_path_lookup+0x258/0x29c
 [<ffffffff8048d0bb>] mutex_lock+0x9/0xb
 [<ffffffff802816db>] do_rmdir+0x84/0xfc
 [<ffffffff80281764>] sys_rmdir+0x11/0x13
 [<ffffffff8021c2d6>] ia32_sysret+0x0/0xa


Code: 48 8b 57 18 48 85 d2 0f 84 e2 00 00 00 4c 8b 22 45 31 f6 49
RIP  [<ffffffff802412b0>] check_deadlock+0x1f/0x117 RSP <ffff8101fb803dd8>
 NMI Watchdog detected LOCKUP on CPU 0
CPU 0
Modules linked in:
Pid: 19164, comm: mkdir09 Not tainted 2.6.17-rc5-mm2-autokern1 #1
RIP: 0010:[<ffffffff8048d626>]  [<ffffffff8048d626>]
.text.lock.mutex+0x2f/0x65
RSP: 0000:ffff8101f3e65e98  EFLAGS: 00000086
RAX: 0000000000000000 RBX: ffff81007b8ec748 RCX: 0000000000000012
RDX: 0000000000000000 RSI: 0000000000000012 RDI: ffff8101f3ae5988
RBP: ffff8101f3e65eb8 R08: ffff8101803f6cf8 R09: ffff81007b8ec748
R10: 000000000000002f R11: 0000000000000000 R12: ffff8101f3ae5988
R13: 0000000000000246 R14: 0000000000000000 R15: 0000000000000000
FS:  000000000804a020(0000) GS:ffffffff8061b000(0063) knlGS:00000000f7e15460
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00000000f7ed5cc0 CR3: 00000001f52e4000 CR4: 00000000000006e0
Process mkdir09 (pid: 19164, threadinfo ffff8101f3e64000, task
ffff8101fcafa7b0)
Stack: 0000000000000012 ffff81007b8ec748 0000000000000000 ffff81007b92a000
       ffff8101f3e65ec8 ffffffff8048d2a3 ffff8101f3e65f68 ffffffff8028172a
       ffff8101f4dd49b8 ffff810180051ec0
Call Trace:
 [<ffffffff8048d2a3>] mutex_unlock+0x9/0xb
 [<ffffffff8028172a>] do_rmdir+0xd3/0xfc
 [<ffffffff80281764>] sys_rmdir+0x11/0x13
 [<ffffffff8021c2d6>] ia32_sysret+0x0/0xa

-apw

#9 lore raw
Quoted 7 lines

That did seem to get rid of the original error. However, I'm now
experiencing a reiserfs4 panic so I'm not 100% confident the problem really is gone, but the reiserfs panic isn't anywhere as near hard as the panic's I have been experiencing; I was able to reboot the machine even though the tests were not completing correctly. Full panic for that one is below:

Get the same exact panic on ia32 NUMA-Q (just posted it, but for reference, there's a copy here:
http://test.kernel.org/abat/34623/debug/console.log
). But only in -mm3, not -mm2, so
probably it's intermittent if you were already seeing it on -mm1?

PS. I think the reiser4 thing was a typo, it's generic by the looks of it, and I don't think we test reiser4, is probably ext3 or 2.

Quoted 61 lines
general protection fault: 0000 [1] SMP
last sysfs file: /devices/pci0000:00/0000:00:06.0/resource
CPU 1
Modules linked in:
Pid: 19163, comm: mkdir09 Not tainted 2.6.17-rc5-mm2-autokern1 #1
RIP: 0010:[<ffffffff802412b0>]  [<ffffffff802412b0>]
check_deadlock+0x1f/0x117
RSP: 0000:ffff8101fb803dd8  EFLAGS: 00010002
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
RDX: ffff8101f3e64000 RSI: 0000000000000001 RDI: 2222222222222222
RBP: ffff8101fb803df8 R08: 0000000000000000 R09: ffff8101fb803e68
R10: ffff8101fb802000 R11: 00000000ffffff9c R12: ffff8101fcafa7b0
R13: 2222222222222222 R14: 2222222222222222 R15: 0000000000000000
FS:  000000000804a020(0000) GS:ffff8100e3f27e40(0063) knlGS:00000000f7e15460
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00000000f7ee2090 CR3: 00000001f4dce000 CR4: 00000000000006e0
Process mkdir09 (pid: 19163, threadinfo ffff8101fb802000, task
ffff8101fdd4b0d0)
Stack: 0000000000000000 ffff8101fcafa7b0 ffff8101f3ae5988 2222222222222222
       ffff8101fb803e28 ffffffff80241361 ffff8101f3ae5988 ffff8101fb802000
       ffff8101fb803e68 ffff8101fdd4b0d0
Call Trace:
 [<ffffffff80241361>] check_deadlock+0xd0/0x117
 [<ffffffff80241574>] debug_mutex_add_waiter+0x57/0x6f
 [<ffffffff8048cf47>] __mutex_lock_slowpath+0xc7/0x232
 [<ffffffff802800bc>] do_path_lookup+0x258/0x29c
 [<ffffffff8048d0bb>] mutex_lock+0x9/0xb
 [<ffffffff802816db>] do_rmdir+0x84/0xfc
 [<ffffffff80281764>] sys_rmdir+0x11/0x13
 [<ffffffff8021c2d6>] ia32_sysret+0x0/0xa


Code: 48 8b 57 18 48 85 d2 0f 84 e2 00 00 00 4c 8b 22 45 31 f6 49
RIP  [<ffffffff802412b0>] check_deadlock+0x1f/0x117 RSP <ffff8101fb803dd8>
 NMI Watchdog detected LOCKUP on CPU 0
CPU 0
Modules linked in:
Pid: 19164, comm: mkdir09 Not tainted 2.6.17-rc5-mm2-autokern1 #1
RIP: 0010:[<ffffffff8048d626>]  [<ffffffff8048d626>]
.text.lock.mutex+0x2f/0x65
RSP: 0000:ffff8101f3e65e98  EFLAGS: 00000086
RAX: 0000000000000000 RBX: ffff81007b8ec748 RCX: 0000000000000012
RDX: 0000000000000000 RSI: 0000000000000012 RDI: ffff8101f3ae5988
RBP: ffff8101f3e65eb8 R08: ffff8101803f6cf8 R09: ffff81007b8ec748
R10: 000000000000002f R11: 0000000000000000 R12: ffff8101f3ae5988
R13: 0000000000000246 R14: 0000000000000000 R15: 0000000000000000
FS:  000000000804a020(0000) GS:ffffffff8061b000(0063) knlGS:00000000f7e15460
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00000000f7ed5cc0 CR3: 00000001f52e4000 CR4: 00000000000006e0
Process mkdir09 (pid: 19164, threadinfo ffff8101f3e64000, task
ffff8101fcafa7b0)
Stack: 0000000000000012 ffff81007b8ec748 0000000000000000 ffff81007b92a000
       ffff8101f3e65ec8 ffffffff8048d2a3 ffff8101f3e65f68 ffffffff8028172a
       ffff8101f4dd49b8 ffff810180051ec0
Call Trace:
 [<ffffffff8048d2a3>] mutex_unlock+0x9/0xb
 [<ffffffff8028172a>] do_rmdir+0xd3/0xfc
 [<ffffffff80281764>] sys_rmdir+0x11/0x13
 [<ffffffff8021c2d6>] ia32_sysret+0x0/0xa

-apw

#10 lore raw
In reply to Andy Whitcroft
Quoted 83 lines · Andy Whitcroft
Quoted 37 lines · Andy Whitcroft
Quoted 17 lines · Martin J. Bligh
Quoted 13 lines · Andrew Morton
Quoted 9 lines · Martin J. Bligh
Quoted 5 lines · Andrew Morton

Martin Bligh <[email protected]> wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

The x86_65 panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.

What are you doing to make this happen?

runalltests on LTP

Ok, I think this could well be the same problem I got half way through tracking last time round. We are still handing off threads with non-initialised stacks.

APW: schedule: ffffffff805f0cd8 bad rsp flags=00000000

Kernel panic - not syncing: BAD STACK POINTER
Quoted 7 lines

Interestingly this has remained unchanged for dispite the major churn -mm takes so this could well be the underlying issue as we almost always blow up randomly when we do this, in a mess with no stack to work with.

Last time I tried to split search -mm1 and she was being a hideous pig, I just couldn't get any bit of it to compile without the rest. Will try and track this down with the new -mm.

Ok. Did a split search on -mm2 for this. With the full stack I was still tripping up on the bad thread hand-off trigger above. However, when split searching I seemed to get somewhat different panics pretty commonly in the allocator. My split search led me to the start of the swapless page migration patches:

GOOD:page-migration-cleanup-pass-mapping-to-migration-functions.patch GOOD:page-migration-cleanup-move-fallback-handling-into-special-function.patch ----:swapless-pm-add-r-w-migration-entries.patch -BAD:swapless-pm-add-r-w-migration-entries-fix-2.patch

I tried pretty hard to slide this out of -mm and can't say with cirtainty that I did it right. I backed out the following in order to get the first two off (obviously from the bottom up):

swapless-pm-add-r-w-migration-entries.patch swapless-pm-add-r-w-migration-entries-fix-2.patch swapless-page-migration-rip-out-swap-based-logic.patch swapless-page-migration-modify-core-logic.patch more-page-migration-do-not-inc-dec-rss-counters.patch more-page-migration-use-migration-entries-for-file-pages.patch page-migration-update-documentation.patch page-migration-simplify-migrate_pages.patch page-migration-simplify-migrate_pages-tweaks.patch page-migration-handle-freeing-of-pages-in-migrate_pages.patch page-migration-use-allocator-function-for-migrate_pages.patch page-migration-support-moving-of-individual-pages.patch page-migration-detailed-status-for-moving-of-individual-pages.patch page-migration-support-moving-of-individual-pages-fixes.patch page-migration-support-moving-of-individual-pages-x86_64-support.patch page-migration-support-moving-of-individual-pages-x86-support.patch
page-migration-support-moving-of-individual-pages-x86-support-fix.patch
page-migration-support-a-vma-migration-function.patch allow-migration-of-mlocked-pages.patch

Quoted 6 lines

That did seem to get rid of the original error. However, I'm now
experiencing a reiserfs4 panic so I'm not 100% confident the problem really is gone, but the reiserfs panic isn't anywhere as near hard as the panic's I have been experiencing; I was able to reboot the machine even though the tests were not completing correctly. Full panic for that one is below:

Ok, who can say what I am taking, but its strong. Anyhow, the panic is as below. The reiser4 reference is another issue, not this one.

Quoted 62 lines
general protection fault: 0000 [1] SMP
last sysfs file: /devices/pci0000:00/0000:00:06.0/resource
CPU 1
Modules linked in:
Pid: 19163, comm: mkdir09 Not tainted 2.6.17-rc5-mm2-autokern1 #1
RIP: 0010:[<ffffffff802412b0>]  [<ffffffff802412b0>]
check_deadlock+0x1f/0x117
RSP: 0000:ffff8101fb803dd8  EFLAGS: 00010002
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
RDX: ffff8101f3e64000 RSI: 0000000000000001 RDI: 2222222222222222
RBP: ffff8101fb803df8 R08: 0000000000000000 R09: ffff8101fb803e68
R10: ffff8101fb802000 R11: 00000000ffffff9c R12: ffff8101fcafa7b0
R13: 2222222222222222 R14: 2222222222222222 R15: 0000000000000000
FS:  000000000804a020(0000) GS:ffff8100e3f27e40(0063) knlGS:00000000f7e15460
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00000000f7ee2090 CR3: 00000001f4dce000 CR4: 00000000000006e0
Process mkdir09 (pid: 19163, threadinfo ffff8101fb802000, task
ffff8101fdd4b0d0)
Stack: 0000000000000000 ffff8101fcafa7b0 ffff8101f3ae5988 2222222222222222
       ffff8101fb803e28 ffffffff80241361 ffff8101f3ae5988 ffff8101fb802000
       ffff8101fb803e68 ffff8101fdd4b0d0
Call Trace:
 [<ffffffff80241361>] check_deadlock+0xd0/0x117
 [<ffffffff80241574>] debug_mutex_add_waiter+0x57/0x6f
 [<ffffffff8048cf47>] __mutex_lock_slowpath+0xc7/0x232
 [<ffffffff802800bc>] do_path_lookup+0x258/0x29c
 [<ffffffff8048d0bb>] mutex_lock+0x9/0xb
 [<ffffffff802816db>] do_rmdir+0x84/0xfc
 [<ffffffff80281764>] sys_rmdir+0x11/0x13
 [<ffffffff8021c2d6>] ia32_sysret+0x0/0xa


Code: 48 8b 57 18 48 85 d2 0f 84 e2 00 00 00 4c 8b 22 45 31 f6 49
RIP  [<ffffffff802412b0>] check_deadlock+0x1f/0x117 RSP <ffff8101fb803dd8>
 NMI Watchdog detected LOCKUP on CPU 0
CPU 0
Modules linked in:
Pid: 19164, comm: mkdir09 Not tainted 2.6.17-rc5-mm2-autokern1 #1
RIP: 0010:[<ffffffff8048d626>]  [<ffffffff8048d626>]
.text.lock.mutex+0x2f/0x65
RSP: 0000:ffff8101f3e65e98  EFLAGS: 00000086
RAX: 0000000000000000 RBX: ffff81007b8ec748 RCX: 0000000000000012
RDX: 0000000000000000 RSI: 0000000000000012 RDI: ffff8101f3ae5988
RBP: ffff8101f3e65eb8 R08: ffff8101803f6cf8 R09: ffff81007b8ec748
R10: 000000000000002f R11: 0000000000000000 R12: ffff8101f3ae5988
R13: 0000000000000246 R14: 0000000000000000 R15: 0000000000000000
FS:  000000000804a020(0000) GS:ffffffff8061b000(0063) knlGS:00000000f7e15460
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00000000f7ed5cc0 CR3: 00000001f52e4000 CR4: 00000000000006e0
Process mkdir09 (pid: 19164, threadinfo ffff8101f3e64000, task
ffff8101fcafa7b0)
Stack: 0000000000000012 ffff81007b8ec748 0000000000000000 ffff81007b92a000
       ffff8101f3e65ec8 ffffffff8048d2a3 ffff8101f3e65f68 ffffffff8028172a
       ffff8101f4dd49b8 ffff810180051ec0
Call Trace:
 [<ffffffff8048d2a3>] mutex_unlock+0x9/0xb
 [<ffffffff8028172a>] do_rmdir+0xd3/0xfc
 [<ffffffff80281764>] sys_rmdir+0x11/0x13
 [<ffffffff8021c2d6>] ia32_sysret+0x0/0xa

-apw

#11 lore raw
In reply to Andy Whitcroft
Quoted 10 lines · Andy Whitcroft

Ok. Did a split search on -mm2 for this. With the full stack I was still tripping up on the bad thread hand-off trigger above. However, when split searching I seemed to get somewhat different panics pretty commonly in the allocator. My split search led me to the start of the swapless page migration patches:

GOOD:page-migration-cleanup-pass-mapping-to-migration-functions.patch GOOD:page-migration-cleanup-move-fallback-handling-into-special-function.patch ----:swapless-pm-add-r-w-migration-entries.patch -BAD:swapless-pm-add-r-w-migration-entries-fix-2.patch

So it happens with r-w-migration-entries but not without fix-2? Or does it require the fix-2 in order to happen? Without the fix page migration is broken.

Does LTP include any page migration tests? Guess not? If not then this could be simply due to code rearrangement making the problem appear in a different way.

Which test did this?

Swap migration uses the two higest numbers of swap types (30 and 31). If those are generated by something then we could have trouble but I do not see how the patches could hurt otherwise.

#12 lore raw
Quoted 22 lines · Christoph Lameter
Quoted 10 lines · Andy Whitcroft

Ok. Did a split search on -mm2 for this. With the full stack I was still tripping up on the bad thread hand-off trigger above. However, when split searching I seemed to get somewhat different panics pretty commonly in the allocator. My split search led me to the start of the swapless page migration patches:

GOOD:page-migration-cleanup-pass-mapping-to-migration-functions.patch GOOD:page-migration-cleanup-move-fallback-handling-into-special-function.patch ----:swapless-pm-add-r-w-migration-entries.patch -BAD:swapless-pm-add-r-w-migration-entries-fix-2.patch

So it happens with r-w-migration-entries but not without fix-2? Or does it require the fix-2 in order to happen? Without the fix page migration is broken.

Does LTP include any page migration tests? Guess not? If not then this could be simply due to code rearrangement making the problem appear in a different way.

Quoted 5 lines

Which test did this?

Swap migration uses the two higest numbers of swap types (30 and 31). If those are generated by something then we could have trouble but I do not see how the patches could hurt otherwise.

ltp has many weird swap tests

(and I assume you're not talking about the pte's that are swapped, but only the file type in the swap signature; since the higher bits in the pte are used by the non-linear vma feature afaik)

#13 lore raw
Quoted 8 lines · Arjan van de Ven
Quoted 3 lines

Swap migration uses the two higest numbers of swap types (30 and 31). If those are generated by something then we could have trouble but I do not see how the patches could hurt otherwise.

ltp has many weird swap tests

(and I assume you're not talking about the pte's that are swapped, but only the file type in the swap signature; since the higher bits in the pte are used by the non-linear vma feature afaik)

How was the kernel compiled? It must have been running on a NUMA system for page migration to be enabled. Otherwise the fallback definitions in include/linux/swapops.h should remove all the swap migration entry handling.

#14 lore raw
Quoted 5 lines · Christoph Lameter
Quoted 2 lines · Arjan van de Ven
Quoted 1 line

Swap migration uses the two higest numbers of swap types (30 and 31). If those are generated by something then we could have trouble but I do not see how the patches could hurt otherwise.

Quoted 8 lines
Quoted 5 lines

ltp has many weird swap tests

(and I assume you're not talking about the pte's that are swapped, but only the file type in the swap signature; since the higher bits in the pte are used by the non-linear vma feature afaik)

How was the kernel compiled? It must have been running on a NUMA system for page migration to be enabled. Otherwise the fallback definitions in include/linux/swapops.h should remove all the swap migration entry handling.

x86_64 NUMA. Config is here:
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/config/abat/amd64

mtest01 was what killed it, IIRC.

#15 lore raw
Quoted 8 lines · Martin Bligh
Quoted 3 lines

How was the kernel compiled? It must have been running on a NUMA system for page migration to be enabled. Otherwise the fallback definitions in include/linux/swapops.h should remove all the swap migration entry handling.

x86_64 NUMA. Config is here:
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/config/abat/amd64

mtest01 was what killed it, IIRC.

CONFIG_MIGRATION is not set, no CONFIG_MIGRATION entry is in .config, so page migration was not compiled in. You also forgot to do "make oldconfig" when building the kernel. If you do a "make oldconfig" then your kernel will be build with page migration support.

Either what we see is due to code rearrangement or there is something wrong with the fallback definitions in include/linux/swapops.h.

The swapless-pm-add-r-w-migration-entries.patch also introduces Hugh's reversal of the anon_vma list.

#16 lore raw
Quoted 7 lines · Christoph Lameter
Quoted 4 lines · Martin Bligh
Quoted 3 lines

How was the kernel compiled? It must have been running on a NUMA system for page migration to be enabled. Otherwise the fallback definitions in include/linux/swapops.h should remove all the swap migration entry handling.

Quoted 7 lines
Quoted 4 lines

x86_64 NUMA. Config is here:
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/config/abat/amd64

mtest01 was what killed it, IIRC.

CONFIG_MIGRATION is not set, no CONFIG_MIGRATION entry is in .config, so page migration was not compiled in. You also forgot to do "make oldconfig" when building the kernel. If you do a "make oldconfig" then your kernel will be build with page migration support.

No, that is the config it starts with, sorry ... that was misleading. It then does makeoldconfig from there:

http://test.kernel.org/abat/34624/build/dotconfig

is teh one one from the -mm3 build

CONFIG_MIGRATION=y

Quoted 3 lines

Either what we see is due to code rearrangement or there is something wrong with the fallback definitions in include/linux/swapops.h.

The swapless-pm-add-r-w-migration-entries.patch also introduces Hugh's reversal of the anon_vma list.

OK, are those easily separable? Sounds like one good test would be to force CONFIG_MIGRATION to =n, right?

M.

#17 lore raw
Quoted 4 lines · Martin Bligh

No, that is the config it starts with, sorry ... that was misleading. It then does makeoldconfig from there:

http://test.kernel.org/abat/34624/build/dotconfig

Ok. So page migration is enabled.

Quoted 7 lines
Quoted 4 lines

Either what we see is due to code rearrangement or there is something wrong with the fallback definitions in include/linux/swapops.h.
The swapless-pm-add-r-w-migration-entries.patch also introduces Hugh's reversal of the anon_vma list.

OK, are those easily separable? Sounds like one good test would be to force CONFIG_MIGRATION to =n, right?

CONFIG_MIGRATION off will disable the page migration stuff and make swap types 30 / 31 usable as before. The reversal of the order for the anon_vma list will be preserved.

So yes it may lead us further. If the problem persists with CONFIG_MIGRATION=n then the anon_vma reversal may be an issue.

#18 lore raw
Quoted 21 lines · Christoph Lameter
Quoted 4 lines · Martin Bligh

No, that is the config it starts with, sorry ... that was misleading. It then does makeoldconfig from there:

http://test.kernel.org/abat/34624/build/dotconfig

Ok. So page migration is enabled.

Quoted 7 lines
Quoted 4 lines

Either what we see is due to code rearrangement or there is something wrong with the fallback definitions in include/linux/swapops.h.
The swapless-pm-add-r-w-migration-entries.patch also introduces Hugh's reversal of the anon_vma list.

OK, are those easily separable? Sounds like one good test would be to force CONFIG_MIGRATION to =n, right?

Quoted 6 lines

CONFIG_MIGRATION off will disable the page migration stuff and make swap types 30 / 31 usable as before. The reversal of the order for the anon_vma list will be preserved.

So yes it may lead us further. If the problem persists with CONFIG_MIGRATION=n then the anon_vma reversal may be an issue.

Ok. Will test that when I get a minute.

-apw

#19 lore raw
In reply to Arjan van de Ven
Quoted 3 lines · Arjan van de Ven

(and I assume you're not talking about the pte's that are swapped, but only the file type in the swap signature; since the higher bits in the pte are used by the non-linear vma feature afaik)

migration entries are swap entries of type 30 and 31. They use the format for swap entries.

#20 lore raw
In reply to Christoph Lameter
Quoted 16 lines · Christoph Lameter
Quoted 10 lines · Andy Whitcroft

Ok. Did a split search on -mm2 for this. With the full stack I was still tripping up on the bad thread hand-off trigger above. However, when split searching I seemed to get somewhat different panics pretty commonly in the allocator. My split search led me to the start of the swapless page migration patches:

GOOD:page-migration-cleanup-pass-mapping-to-migration-functions.patch GOOD:page-migration-cleanup-move-fallback-handling-into-special-function.patch ----:swapless-pm-add-r-w-migration-entries.patch -BAD:swapless-pm-add-r-w-migration-entries-fix-2.patch

Quoted 4 lines

So it happens with r-w-migration-entries but not without fix-2? Or does it require the fix-2 in order to happen? Without the fix page migration is broken.

I took the swapless-pm-add-r-w-migration-entries.patch and -fix-2.patch as a single patch and didn't test the ----'d level. It worked anywhere I tested up to and including the last one marked GOOD above. Anywhere below that was a mess. Backing out those two patches (and a bunch of dependant ones) seemed to make the problems we get very different, and if Martin is right the same as problems we are seeing on other architectures. So I am leaning to the feeling that this part of -mm is introducing a problem.

Quoted 9 lines

Does LTP include any page migration tests? Guess not? If not then this could be simply due to code rearrangement making the problem appear in a different way.

Which test did this?

Swap migration uses the two higest numbers of swap types (30 and 31). If those are generated by something then we could have trouble but I do not see how the patches could hurt otherwise.

I uses an awful lot of very wierd swap things. So possibly there is a
tie in there?

-apw

Adding 65528k swap on ./swapfile01.  Priority:-2 extents:116 across:360044k
Adding 65528k swap on ./swapfile01.  Priority:-3 extents:104 across:569492k
Adding 65528k swap on ./swapfile01.  Priority:-4 extents:172 across:520952k
Unable to find swap-space signature
Adding 32k swap on swapfile02.  Priority:-5 extents:2 across:60k
Adding 32k swap on swapfile03.  Priority:-6 extents:1 across:32k
Adding 32k swap on swapfile04.  Priority:-7 extents:1 across:32k
Adding 32k swap on swapfile05.  Priority:-8 extents:3 across:44k
Adding 32k swap on swapfile06.  Priority:-9 extents:1 across:32k
Adding 32k swap on swapfile07.  Priority:-10 extents:1 across:32k
Adding 32k swap on swapfile08.  Priority:-11 extents:1 across:32k
Adding 32k swap on swapfile09.  Priority:-12 extents:1 across:32k
Adding 32k swap on swapfile10.  Priority:-13 extents:1 across:32k
Adding 32k swap on swapfile11.  Priority:-14 extents:1 across:32k
Adding 32k swap on swapfile12.  Priority:-15 extents:1 across:32k
Adding 32k swap on swapfile13.  Priority:-16 extents:1 across:32k
Adding 32k swap on swapfile14.  Priority:-17 extents:1 across:32k
Adding 32k swap on swapfile15.  Priority:-18 extents:1 across:32k
Adding 32k swap on swapfile16.  Priority:-19 extents:1 across:32k
Adding 32k swap on swapfile17.  Priority:-20 extents:1 across:32k
Adding 32k swap on swapfile18.  Priority:-21 extents:1 across:32k
Adding 32k swap on swapfile19.  Priority:-22 extents:1 across:32k
Adding 32k swap on swapfile20.  Priority:-23 extents:1 across:32k
Adding 32k swap on swapfile21.  Priority:-24 extents:1 across:32k
Adding 32k swap on swapfile22.  Priority:-25 extents:1 across:32k
Adding 32k swap on swapfile23.  Priority:-26 extents:1 across:32k
Adding 32k swap on swapfile24.  Priority:-27 extents:1 across:32k
Adding 32k swap on swapfile25.  Priority:-28 extents:1 across:32k
Adding 32k swap on swapfile26.  Priority:-29 extents:1 across:32k
Adding 32k swap on swapfile27.  Priority:-30 extents:1 across:32k
Adding 32k swap on swapfile28.  Priority:-31 extents:1 across:32k
Adding 32k swap on swapfile29.  Priority:-32 extents:1 across:32k
Adding 32k swap on swapfile30.  Priority:-33 extents:1 across:32k
Adding 32k swap on swapfile31.  Priority:-34 extents:1 across:32k
Adding 32k swap on swapfile32.  Priority:-35 extents:1 across:32k
#21 lore raw
Quoted 28 lines · Andy Whitcroft
Adding 65528k swap on ./swapfile01.  Priority:-2 extents:116 across:360044k
Adding 65528k swap on ./swapfile01.  Priority:-3 extents:104 across:569492k
Adding 65528k swap on ./swapfile01.  Priority:-4 extents:172 across:520952k
Unable to find swap-space signature
Adding 32k swap on swapfile02.  Priority:-5 extents:2 across:60k
Adding 32k swap on swapfile03.  Priority:-6 extents:1 across:32k
Adding 32k swap on swapfile04.  Priority:-7 extents:1 across:32k
Adding 32k swap on swapfile05.  Priority:-8 extents:3 across:44k
Adding 32k swap on swapfile06.  Priority:-9 extents:1 across:32k
Adding 32k swap on swapfile07.  Priority:-10 extents:1 across:32k
Adding 32k swap on swapfile08.  Priority:-11 extents:1 across:32k
Adding 32k swap on swapfile09.  Priority:-12 extents:1 across:32k
Adding 32k swap on swapfile10.  Priority:-13 extents:1 across:32k
Adding 32k swap on swapfile11.  Priority:-14 extents:1 across:32k
Adding 32k swap on swapfile12.  Priority:-15 extents:1 across:32k
Adding 32k swap on swapfile13.  Priority:-16 extents:1 across:32k
Adding 32k swap on swapfile14.  Priority:-17 extents:1 across:32k
Adding 32k swap on swapfile15.  Priority:-18 extents:1 across:32k
Adding 32k swap on swapfile16.  Priority:-19 extents:1 across:32k
Adding 32k swap on swapfile17.  Priority:-20 extents:1 across:32k
Adding 32k swap on swapfile18.  Priority:-21 extents:1 across:32k
Adding 32k swap on swapfile19.  Priority:-22 extents:1 across:32k
Adding 32k swap on swapfile20.  Priority:-23 extents:1 across:32k
Adding 32k swap on swapfile21.  Priority:-24 extents:1 across:32k
Adding 32k swap on swapfile22.  Priority:-25 extents:1 across:32k
Adding 32k swap on swapfile23.  Priority:-26 extents:1 across:32k
Adding 32k swap on swapfile24.  Priority:-27 extents:1 across:32k
Quoted 8 lines
Adding 32k swap on swapfile25.  Priority:-28 extents:1 across:32k
Adding 32k swap on swapfile26.  Priority:-29 extents:1 across:32k
Adding 32k swap on swapfile27.  Priority:-30 extents:1 across:32k
Adding 32k swap on swapfile28.  Priority:-31 extents:1 across:32k
Adding 32k swap on swapfile29.  Priority:-32 extents:1 across:32k
Adding 32k swap on swapfile30.  Priority:-33 extents:1 across:32k
Adding 32k swap on swapfile31.  Priority:-34 extents:1 across:32k
Adding 32k swap on swapfile32.  Priority:-35 extents:1 across:32k

That should not work at all. It should bomb out at 30 swap files with page migration on.

#22 lore raw
Quoted 41 lines · Christoph Lameter
Quoted 27 lines · Andy Whitcroft
Adding 65528k swap on ./swapfile01.  Priority:-2 extents:116 across:360044k
Adding 65528k swap on ./swapfile01.  Priority:-3 extents:104 across:569492k
Adding 65528k swap on ./swapfile01.  Priority:-4 extents:172 across:520952k
Unable to find swap-space signature
Adding 32k swap on swapfile02.  Priority:-5 extents:2 across:60k
Adding 32k swap on swapfile03.  Priority:-6 extents:1 across:32k
Adding 32k swap on swapfile04.  Priority:-7 extents:1 across:32k
Adding 32k swap on swapfile05.  Priority:-8 extents:3 across:44k
Adding 32k swap on swapfile06.  Priority:-9 extents:1 across:32k
Adding 32k swap on swapfile07.  Priority:-10 extents:1 across:32k
Adding 32k swap on swapfile08.  Priority:-11 extents:1 across:32k
Adding 32k swap on swapfile09.  Priority:-12 extents:1 across:32k
Adding 32k swap on swapfile10.  Priority:-13 extents:1 across:32k
Adding 32k swap on swapfile11.  Priority:-14 extents:1 across:32k
Adding 32k swap on swapfile12.  Priority:-15 extents:1 across:32k
Adding 32k swap on swapfile13.  Priority:-16 extents:1 across:32k
Adding 32k swap on swapfile14.  Priority:-17 extents:1 across:32k
Adding 32k swap on swapfile15.  Priority:-18 extents:1 across:32k
Adding 32k swap on swapfile16.  Priority:-19 extents:1 across:32k
Adding 32k swap on swapfile17.  Priority:-20 extents:1 across:32k
Adding 32k swap on swapfile18.  Priority:-21 extents:1 across:32k
Adding 32k swap on swapfile19.  Priority:-22 extents:1 across:32k
Adding 32k swap on swapfile20.  Priority:-23 extents:1 across:32k
Adding 32k swap on swapfile21.  Priority:-24 extents:1 across:32k
Adding 32k swap on swapfile22.  Priority:-25 extents:1 across:32k
Adding 32k swap on swapfile23.  Priority:-26 extents:1 across:32k
Adding 32k swap on swapfile24.  Priority:-27 extents:1 across:32k
Quoted 8 lines
Adding 32k swap on swapfile25.  Priority:-28 extents:1 across:32k
Adding 32k swap on swapfile26.  Priority:-29 extents:1 across:32k
Adding 32k swap on swapfile27.  Priority:-30 extents:1 across:32k
Adding 32k swap on swapfile28.  Priority:-31 extents:1 across:32k
Adding 32k swap on swapfile29.  Priority:-32 extents:1 across:32k
Adding 32k swap on swapfile30.  Priority:-33 extents:1 across:32k
Adding 32k swap on swapfile31.  Priority:-34 extents:1 across:32k
Adding 32k swap on swapfile32.  Priority:-35 extents:1 across:32k
Quoted 2 lines

That should not work at all. It should bomb out at 30 swap files with page migration on.

The implication here is that there can only be 32 entries in-toto ... it feels like we have at least 33/34 as the machine has swap by default ... more to look at!

-apw

#23 lore raw
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Adding 32k swap on swapfile31. Priority:-34 extents:1 across:32k Adding 32k swap on swapfile32. Priority:-35 extents:1 across:32k

That should not work at all. It should bomb out at 30 swap files with page migration on.

The implication here is that there can only be 32 entries in-toto ... it feels like we have at least 33/34 as the machine has swap by default ... more to look at!

Either way, random panics are not the appropriate response ;-)

if it can't cope with that, why isn't it failing the request ???

M.

#24 lore raw
Quoted 11 lines · Martin Bligh
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

Adding 32k swap on swapfile31. Priority:-34 extents:1 across:32k Adding 32k swap on swapfile32. Priority:-35 extents:1 across:32k

That should not work at all. It should bomb out at 30 swap files with page migration on.

The implication here is that there can only be 32 entries in-toto ... it feels like we have at least 33/34 as the machine has swap by default ... more to look at!

Either way, random panics are not the appropriate response ;-)

Seems that the check for too many swapfiles is busted somehow. It should never go beyond 32 without page migration. Looking into it.

#25 lore raw
In reply to Martin Bligh
Quoted 4 lines

Either way, random panics are not the appropriate response ;-)

if it can't cope with that, why isn't it failing the request ???

There is a crappy test in swap_on(). It should check against MAX_SWAPFILES and not do this conversion back and forth. Some architectures may not check if we are beyond the boundaries of what a swap entry can take.

Why is this strange this in there? Are there architectures that support less than 32 swap devices?

mm/swapfile.c +3 -16
Index: linux-2.6.17-rc5-mm2/mm/swapfile.c
===================================================================
@@ -1384,6 +1384,9 @@ asmlinkage long sys_swapon(const char __
 	struct inode *inode = NULL;
 	int did_down = 0;
 
+	if (nr_swapfiles >= MAX_SWAPFILES)
+		return -E2BIG;
+
 	if (!capable(CAP_SYS_ADMIN))
 		return -EPERM;
 	spin_lock(&swap_lock);
@@ -1392,22 +1395,6 @@ asmlinkage long sys_swapon(const char __
 		if (!(p->flags & SWP_USED))
 			break;
 	error = -EPERM;
-	/*
-	 * Test if adding another swap device is possible. There are
-	 * two limiting factors: 1) the number of bits for the swap
-	 * type swp_entry_t definition and 2) the number of bits for
-	 * the swap type in the swap ptes as defined by the different
-	 * architectures. To honor both limitations a swap entry
-	 * with swap offset 0 and swap type ~0UL is created, encoded
-	 * to a swap pte, decoded to a swp_entry_t again and finally
-	 * the swap type part is extracted. This will mask all bits
-	 * from the initial ~0UL that can't be encoded in either the
-	 * swp_entry_t or the architecture definition of a swap pte.
-	 */
-	if (type > swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0))))) {
-		spin_unlock(&swap_lock);
-		goto out;
-	}
 	if (type >= nr_swapfiles)
 		nr_swapfiles = type+1;
 	INIT_LIST_HEAD(&p->extent_list);
#26 lore raw

Sigh the patch that I sent earlier will make swapon fail when adding more entries if 32 entries have been defined before even if some of these are later freed. Plus maybe we better leave the probing intact for arches that support less than 32 swap devices and also return -EPERM like before. I guess we need this one instead:

Do proper boundary checking in sys_swapon().

sys_swapon currently does not limit the number of swap devices. It may as a result overwrite memory following the swap_info array and get into entanglements with page migration since it may usethe swap types reserved for page migration.

Fix this by limiting the number of swap devices in swapon to MAX_SWAPFILES

mm/swapfile.c +6 -1
Index: linux-2.6.17-rc5-mm2/mm/swapfile.c
===================================================================
@@ -1408,8 +1408,13 @@ asmlinkage long sys_swapon(const char __
 		spin_unlock(&swap_lock);
 		goto out;
 	}
-	if (type >= nr_swapfiles)
+	if (type >= nr_swapfiles) {
+		if (nr_swapfiles >= MAX_SWAPFILES) {
+			spin_unlock(&swap_lock);
+			goto out;
+		}
 		nr_swapfiles = type+1;
+	}
 	INIT_LIST_HEAD(&p->extent_list);
 	p->flags = SWP_USED;
 	p->swap_file = NULL;
#27 lore raw
Quoted 28 lines · On Mon, 5 Jun 2006 13:43:47 -0700 (PDT) Christoph Lameter <[email protected]>…

Sigh the patch that I sent earlier will make swapon fail when adding more entries if 32 entries have been defined before even if some of these are later freed. Plus maybe we better leave the probing intact for arches that support less than 32 swap devices and also return -EPERM like before. I guess we need this one instead:

Do proper boundary checking in sys_swapon().

sys_swapon currently does not limit the number of swap devices. It may as a result overwrite memory following the swap_info array and get into entanglements with page migration since it may usethe swap types reserved for page migration.

Fix this by limiting the number of swap devices in swapon to MAX_SWAPFILES

Index: linux-2.6.17-rc5-mm2/mm/swapfile.c
===================================================================

mm/swapfile.c +0 -1
@@ -1408,8 +1408,13 @@ asmlinkage long sys_swapon(const char __
 		spin_unlock(&swap_lock);
 		goto out;
 	}
-	if (type >= nr_swapfiles)

mm/swapfile.c asmlinkage long sys_swapon(const char __ · line 1420

Quoted 10 lines
mm/swapfile.c +6 -0
+	if (type >= nr_swapfiles) {
+		if (nr_swapfiles >= MAX_SWAPFILES) {
+			spin_unlock(&swap_lock);
+			goto out;
+		}
 		nr_swapfiles = type+1;
+	}
 	INIT_LIST_HEAD(&p->extent_list);
 	p->flags = SWP_USED;
 	p->swap_file = NULL;

Thanks, Christoph. So we get to keep the crappy test?

I wonder what LTP was corrupting before it started to corrupt page migration data?

The above looks like a 2.6.17 thing to me.

#28 lore raw
Quoted 2 lines · Andrew Morton

I wonder what LTP was corrupting before it started to corrupt page migration data?

how about mutex debug data structures? ;)

#29 lore raw
In reply to Andrew Morton
Quoted 1 line · Andrew Morton

Thanks, Christoph. So we get to keep the crappy test?

Well at least as long as we have not established that it is not needed.

Quoted 2 lines

I wonder what LTP was corrupting before it started to corrupt page migration data?

Whatever came after the swap_info array. What does the System map say on that platform?

Quoted 1 line

The above looks like a 2.6.17 thing to me.

Right.

#30 lore raw
In reply to Andrew Morton
Quoted 21 lines · Andrew Morton
Quoted 18 lines · On Mon, 5 Jun 2006 13:43:47 -0700 (PDT) Christoph Lameter <[email protected]>…

Fix this by limiting the number of swap devices in swapon to MAX_SWAPFILES

Index: linux-2.6.17-rc5-mm2/mm/swapfile.c
===================================================================

mm/swapfile.c +4 -1
@@ -1408,8 +1408,13 @@ asmlinkage long sys_swapon(const char __
 		spin_unlock(&swap_lock);
 		goto out;
 	}
-	if (type >= nr_swapfiles)
+	if (type >= nr_swapfiles) {
+		if (nr_swapfiles >= MAX_SWAPFILES) {
+			spin_unlock(&swap_lock);
+			goto out;
Quoted 8 lines

mm/swapfile.c asmlinkage long sys_swapon(const char __ · line 1420

Quoted 6 lines
mm/swapfile.c +2 -0
+		}
 		nr_swapfiles = type+1;
+	}
 	INIT_LIST_HEAD(&p->extent_list);
 	p->flags = SWP_USED;
 	p->swap_file = NULL;

Thanks, Christoph. So we get to keep the crappy test?

Christoph's patch looks like it will fix the corruption to me (though I'd have thought the alternative below a little cleaner, keeping the associated tests together and avoiding a second unlock: whatever).

Whether it's correct depends on what Martin was trying to achieve with his test. I'm surprised to find a very ordinary
#define __swp_type(entry) (((entry).val >> 2) & 0x1f)
in include/asm-s390/pgtable.h, and no architecture with a more limiting mask.

I had expected that s390 (or some other) was short of bits and was having to use a more limiting mask: in which case neither Christoph's patch nor the one below would be appropriate, since the migration types 30 and 31 would then fall outside the mask (though it'd only be a problem if page migration were required on such an architecture).

So, the patch should do for now, but it'd be helpful if Martin can explain why he needed to use that test in 2.6.5. (Perhaps he went through a phase of reducing the swap type and extending the swap offset, and this is a remnant of that phase?)

Quoted 4 lines

I wonder what LTP was corrupting before it started to corrupt page migration data?

The above looks like a 2.6.17 thing to me.

Not really (though the clarity and reassurance of the additional MAX_SWAPFILES test is good). We went over it a year or two back, and the macro contortions do involve MAX_SWAPFILES_SHIFT: which up to and including 2.6.17 has enforced the MAX_SWAPFILES limit.

But swapless migration has changed the relationship between MAX_SWAPFILES_SHIFT and MAX_SWAPFILES (-2 when CONFIG_MIGRATION), hence this corruption now surfacing in -mm.

Hugh

mm/swapfile.c +2 -1
@@ -1404,7 +1404,8 @@ asmlinkage long sys_swapon(const char __
 	 * from the initial ~0UL that can't be encoded in either the
 	 * swp_entry_t or the architecture definition of a swap pte.
 	 */
-	if (type > swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0))))) {
+	if (type >= MAX_SWAPFILES ||
+	    type > swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0))))) {
 		spin_unlock(&swap_lock);
 		goto out;
 	}
#31 lore raw
Quoted 3 lines · Hugh Dickins

Christoph's patch looks like it will fix the corruption to me (though I'd have thought the alternative below a little cleaner, keeping the associated tests together and avoiding a second unlock: whatever).

Yup that looks cleaner.

Quoted 5 lines

Whether it's correct depends on what Martin was trying to achieve with his test. I'm surprised to find a very ordinary
#define __swp_type(entry) (((entry).val >> 2) & 0x1f)
in include/asm-s390/pgtable.h, and no architecture with a more limiting mask.

Yes we need to hear from Martin.

Quoted 4 lines

Not really (though the clarity and reassurance of the additional MAX_SWAPFILES test is good). We went over it a year or two back, and the macro contortions do involve MAX_SWAPFILES_SHIFT: which up to and including 2.6.17 has enforced the MAX_SWAPFILES limit.

It looks though as if the testers were able to define more than 32 swap devices. So there is the danger of overwriting the memory following the swap info if we do not fix this.

Where are the macro contortions? No arch uses MAX_SWAPFILES_SHIFT for its definitions and the only other significant use is in swapops.h to determine the shift.

#32 lore raw
Quoted 7 lines · Christoph Lameter
Quoted 5 lines

Whether it's correct depends on what Martin was trying to achieve with his test. I'm surprised to find a very ordinary
#define __swp_type(entry) (((entry).val >> 2) & 0x1f)
in include/asm-s390/pgtable.h, and no architecture with a more limiting mask.

Yes we need to hear from Martin.

s390 supports up to 32 swap devices. At the same time on 31 bit systems only up to 4GB of swap is supported per device. That is because there are three reserved bits in the 31 bit pte, (pte & 0x80000900) must be zero. Bit 2^10 is the invalid bit, bit 2^9 the read-only bit. Two more bits are needed to distinguish between the different pte types. That makes 32-7 = 25 bits in total for offset and swap device. 5 bits are used for the swap device number, and 20 for the offset.
I have chosen this split because for 31 the bit systems in 1999 the standard dasd device has been a model 3 with a capacity of 2.6 GB.

So much for the history lesson.

Quoted 5 lines
Quoted 4 lines

Not really (though the clarity and reassurance of the additional MAX_SWAPFILES test is good). We went over it a year or two back, and the macro contortions do involve MAX_SWAPFILES_SHIFT: which up to and including 2.6.17 has enforced the MAX_SWAPFILES limit.

Quoted 7 lines

It looks though as if the testers were able to define more than 32 swap devices. So there is the danger of overwriting the memory following the swap info if we do not fix this.

Where are the macro contortions? No arch uses MAX_SWAPFILES_SHIFT for its definitions and the only other significant use is in swapops.h to determine the shift.

swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0)))) evaluates to 31 for s390. The idea has been that the creation of a swap entry with ~0UL as the type should mask off all bits that can not be represented in a swap entry. Convert it back and you have the maximum number of swap devices. That should be true for all architectures.

Signature · 8 lines
blue skies,
  Martin.

Martin Schwidefsky
Linux for zSeries Development & Services
IBM Deutschland Entwicklung GmbH

"Reality continues to ruin my life." - Calvin.
#33 lore raw
Quoted 5 lines · Martin Schwidefsky

swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0)))) evaluates to 31 for s390. The idea has been that the creation of a swap entry with ~0UL as the type should mask off all bits that can not be represented in a swap entry. Convert it back and you have the maximum number of swap devices. That should be true for all architectures.

It evaluates to 31 it seems for all platforms. Could we replace this expression with MAX_SWAPFILES? While we at it get rid of the "type" variable because we are usually using pointers to swap_info.

Note that there is still another similar check in there for an arch specific test of the number of pages available per swap device.

mm/swapfile.c +10 -22
Index: linux-2.6.17-rc5-mm2/mm/swapfile.c
===================================================================
@@ -1363,12 +1363,11 @@ __initcall(procswaps_init);
  */
 asmlinkage long sys_swapon(const char __user * specialfile, int swap_flags)
 {
-	struct swap_info_struct * p;
+	struct swap_info_struct *p,*q;
 	char *name = NULL;
 	struct block_device *bdev = NULL;
 	struct file *swap_file = NULL;
 	struct address_space *mapping;
-	unsigned int type;
 	int i, prev;
 	int error;
 	static int least_priority;
@@ -1387,29 +1386,19 @@ asmlinkage long sys_swapon(const char __
 	if (!capable(CAP_SYS_ADMIN))
 		return -EPERM;
 	spin_lock(&swap_lock);
-	p = swap_info;
-	for (type = 0 ; type < nr_swapfiles ; type++,p++)
+
+	for (p = swap_info; p < swap_info + nr_swapfiles; p++)
 		if (!(p->flags & SWP_USED))
 			break;
 	error = -EPERM;
-	/*
-	 * Test if adding another swap device is possible. There are
-	 * two limiting factors: 1) the number of bits for the swap
-	 * type swp_entry_t definition and 2) the number of bits for
-	 * the swap type in the swap ptes as defined by the different
-	 * architectures. To honor both limitations a swap entry
-	 * with swap offset 0 and swap type ~0UL is created, encoded
-	 * to a swap pte, decoded to a swp_entry_t again and finally
-	 * the swap type part is extracted. This will mask all bits
-	 * from the initial ~0UL that can't be encoded in either the
-	 * swp_entry_t or the architecture definition of a swap pte.
-	 */
-	if (type > swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0))))) {
+
+	if (p == swap_info + MAX_SWAPFILES) {
 		spin_unlock(&swap_lock);
 		goto out;
 	}
-	if (type >= nr_swapfiles)
-		nr_swapfiles = type+1;
+	if (p >= swap_info + nr_swapfiles)
+		nr_swapfiles++;
+
 	INIT_LIST_HEAD(&p->extent_list);
 	p->flags = SWP_USED;
 	p->swap_file = NULL;
@@ -1445,10 +1434,9 @@ asmlinkage long sys_swapon(const char __
 	inode = mapping->host;
 
 	error = -EBUSY;
-	for (i = 0; i < nr_swapfiles; i++) {
-		struct swap_info_struct *q = &swap_info[i];
+	for (q = swap_info; q < swap_info + nr_swapfiles; q++) {
 
-		if (i == type || !q->swap_file)
+		if (q == p || !q->swap_file)
 			continue;
 		if (mapping == q->swap_file->f_mapping)
 			goto bad_swap;
#34 lore raw
Quoted 4 lines · Christoph Lameter

It evaluates to 31 it seems for all platforms. Could we replace this expression with MAX_SWAPFILES? While we at it get rid of the "type" variable because we are usually using pointers to swap_info.

Well, if you want to go through all of swapfile.c, changing it over to use pointers rather than type/indexes. But I don't think it's worth it, and I don't think it should be mixed in with this fix.

Quoted 2 lines

Note that there is still another similar check in there for an arch specific test of the number of pages available per swap device.

And that check, also Martin's I believe, has very good justification:
it will vary from arch to arch how big a swap area they can handle, and his check is the right way to do it - no more obscure than it has to be.

Hugh

#35 lore raw
Quoted 6 lines · Hugh Dickins
Quoted 2 lines

Note that there is still another similar check in there for an arch specific test of the number of pages available per swap device.

And that check, also Martin's I believe, has very good justification:
it will vary from arch to arch how big a swap area they can handle, and his check is the right way to do it - no more obscure than it has to be.

Yes, I added that fix since we managed to crash s390 with a swap disk bigger than 4GB. To keep the symmetry I changed the check for the swap type as well.

Signature · 8 lines
blue skies,
  Martin.

Martin Schwidefsky
Linux for zSeries Development & Services
IBM Deutschland Entwicklung GmbH

"Reality continues to ruin my life." - Calvin.
#36 lore raw
In reply to Christoph Lameter
Quoted 7 lines · Christoph Lameter
Quoted 4 lines · Hugh Dickins

Not really (though the clarity and reassurance of the additional MAX_SWAPFILES test is good). We went over it a year or two back, and the macro contortions do involve MAX_SWAPFILES_SHIFT: which up to and including 2.6.17 has enforced the MAX_SWAPFILES limit.

Quoted 7 lines

It looks though as if the testers were able to define more than 32 swap devices. So there is the danger of overwriting the memory following the swap info if we do not fix this.

Where are the macro contortions? No arch uses MAX_SWAPFILES_SHIFT for its definitions and the only other significant use is in swapops.h to determine the shift.

I'll go mad if I try to work it out again: I was as worried as you when I discovered that test in sys_swapon a year or so ago, apparently without any check on MAX_SWAPFILES; and went moaning to Andrew. But once I'd worked through swp_type, pte_to_swp_entry, swp_entry_to_pte, swp_entry, I did come to the conclusion that the MAX_SWAPFILES bound was actually safely built in there.

It's those four in swapops.h, and their concatenation in sys_swapon, that I meant by "macro contortions"; and I think they are safe (or were prior to swapless migration's -2), whatever the arch does for the __ones (which drive me even further insane).

Have you no mercy? Oh, wasn't it rendered safe by the:

#define SWP_TYPE_SHIFT(e) (sizeof(e.val) * 8 - MAX_SWAPFILES_SHIFT) static inline unsigned swp_type(swp_entry_t entry)

{
	return (entry.val >> SWP_TYPE_SHIFT(entry));
}

Hugh

#37 lore raw
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines

Not really (though the clarity and reassurance of the additional MAX_SWAPFILES test is good). We went over it a year or two back, and the macro contortions do involve MAX_SWAPFILES_SHIFT: which up to and including 2.6.17 has enforced the MAX_SWAPFILES limit.

It looks though as if the testers were able to define more than 32 swap devices. So there is the danger of overwriting the memory following the swap info if we do not fix this.

Where are the macro contortions? No arch uses MAX_SWAPFILES_SHIFT for its definitions and the only other significant use is in swapops.h to determine the shift.

Quoted 6 lines

I'll go mad if I try to work it out again: I was as worried as you when I discovered that test in sys_swapon a year or so ago, apparently without any check on MAX_SWAPFILES; and went moaning to Andrew. But once I'd worked through swp_type, pte_to_swp_entry, swp_entry_to_pte, swp_entry, I did come to the conclusion that the MAX_SWAPFILES bound was actually safely built in there.

If it's that difficult to figure out, is that not reason enough to rip it all out and replace it? ;-) Life seems quite complicated enough as it is.

M.

#38 lore raw
Quoted 10 lines · Martin Bligh
Quoted 6 lines

I'll go mad if I try to work it out again: I was as worried as you when I discovered that test in sys_swapon a year or so ago, apparently without any check on MAX_SWAPFILES; and went moaning to Andrew. But once I'd worked through swp_type, pte_to_swp_entry, swp_entry_to_pte, swp_entry, I did come to the conclusion that the MAX_SWAPFILES bound was actually safely built in there.

If it's that difficult to figure out, is that not reason enough to rip it all out and replace it? ;-) Life seems quite complicated enough as it is.

Is it that complicated? You convert UINT_MAX as the type to a swap entry and back to a number. __swp_entry_to_pte() needs to do the correct masking. If it doesn't it is broken. The alternative would be to add architecture defines with the maximum number of swap devices and the maximum swap offset.

Signature · 8 lines
blue skies,
  Martin.

Martin Schwidefsky
Linux for zSeries Development & Services
IBM Deutschland Entwicklung GmbH

"Reality continues to ruin my life." - Calvin.
#39 lore raw
In reply to Martin Bligh
Quoted 11 lines · Martin Bligh
Quoted 7 lines

I'll go mad if I try to work it out again: I was as worried as you when I discovered that test in sys_swapon a year or so ago, apparently without any check on MAX_SWAPFILES; and went moaning to Andrew. But once I'd worked through swp_type, pte_to_swp_entry, swp_entry_to_pte, swp_entry, I did come to the conclusion that the MAX_SWAPFILES bound was actually safely built in there.

If it's that difficult to figure out, is that not reason enough to rip it all out and replace it? ;-) Life seems quite complicated enough as it is.

Not everyone is as deficient as I am. But I do agree with you.

I think Martin had good reason to do his "maxpages = swp_offset(....)" to work out the maximum pgoff expressible in an architecture's swap offset bits; but just went overboard in doing likewise with swap type.

MAX_SWAPFILES and MAX_SWAPFILES_SHIFT are defined in the common include/linux/swap.h, and every architecture supports that many (some could manage more, but swp_type doesn't let them). If an architecture came along which somehow couldn't support that many (seems unlikely to me), then we'd move MAX_SWAPFILES_SHIFT into into arch-dependent files, wouldn't we?

Yes, I'm all for sys_swapon just saying "if (type >= MAX_SWAPFILES)".

Hugh

#40 lore raw
In reply to Hugh Dickins
Quoted 6 lines · Hugh Dickins

I'll go mad if I try to work it out again: I was as worried as you when I discovered that test in sys_swapon a year or so ago, apparently without any check on MAX_SWAPFILES; and went moaning to Andrew. But once I'd worked through swp_type, pte_to_swp_entry, swp_entry_to_pte, swp_entry, I did come to the conclusion that the MAX_SWAPFILES bound was actually safely built in there.

I do not think we want anyone go mad so could be simplify this whole m*** so that mere mortals can comprehend what is going on while reading the source?

Quoted 1 line

Have you no mercy? Oh, wasn't it rendered safe by the:

Ok. So we have a mystical expression here that needs to be explained:

swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0))))

swp_entry(~0L,0) ->
swp_entry(0xffffffffffffffff, 0) ->
0xfffffffffffffff << SWP_TYPE_SHIFT ->
0x1f

Ok we got the highest swap type and this encodes MAX_SWAPFILES_SHIFT and therefore limits the number of swap

But now we convert this to a arch specific pte. The hope is that the arch will mask off any unsupported bits? This is done by s390?

Then we convert the result back into a swap entry and take its type which will hopefully give us the maximum number of swap devices supported by the arch?

Would it not be better for an arch to explicitly tell us how many swap devices are supported? Then we could make the swap_info array shorter and get rid of this cryptic test.

This test also creates a problem for the migration entries: It is really not clear until runtime how many swap devices are supported so we cannot take the last 2 for page migration. In order for page migration to work all NUMA arches that do not support 32 swap devices need to define CONFIG_MIGRATION=n. Seems that this is only s390?

#41 lore raw
Quoted 9 lines · Christoph Lameter

Ok. So we have a mystical expression here that needs to be explained:

swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0))))

swp_entry(~0L,0) ->
swp_entry(0xffffffffffffffff, 0) ->
0xfffffffffffffff << SWP_TYPE_SHIFT ->
0x1f

Quoted 5 lines

Ok we got the highest swap type and this encodes MAX_SWAPFILES_SHIFT and therefore limits the number of swap

But now we convert this to a arch specific pte. The hope is that the arch will mask off any unsupported bits? This is done by s390?

Yes.

Quoted 3 lines

Then we convert the result back into a swap entry and take its type which will hopefully give us the maximum number of swap devices supported by the arch?

Yes.

Quoted 3 lines

Would it not be better for an arch to explicitly tell us how many swap devices are supported? Then we could make the swap_info array shorter and get rid of this cryptic test.

Do you want to calculate the correct number for all architectures? Have fun trying to decrypt the different pte formats.

Quoted 5 lines

This test also creates a problem for the migration entries: It is really not clear until runtime how many swap devices are supported so we cannot take the last 2 for page migration. In order for page migration to work all NUMA arches that do not support 32 swap devices need to define CONFIG_MIGRATION=n. Seems that this is only s390?

s390 is not a NUMA system, but it does support 32 swap devices.

Signature · 8 lines
blue skies,
  Martin.

Martin Schwidefsky
Linux for zSeries Development & Services
IBM Deutschland Entwicklung GmbH

"Reality continues to ruin my life." - Calvin.
#42 lore raw
In reply to Christoph Lameter
Quoted 4 lines · Christoph Lameter

Would it not be better for an arch to explicitly tell us how many swap devices are supported? Then we could make the swap_info array shorter and get rid of this cryptic test.

If they differed at all, yes.

Quoted 5 lines

This test also creates a problem for the migration entries: It is really not clear until runtime how many swap devices are supported so we cannot take the last 2 for page migration. In order for page migration to work all NUMA arches that do not support 32 swap devices need to define CONFIG_MIGRATION=n. Seems that this is only s390?

Even s390 is okay, isn't it? MartinS's swp_type returns the highest type, 31, corresponding to 32 types, as on every other architecture. You and I and Martin Bligh would prefer this patch...

Remove unnecessary obfuscation from sys_swapon's range check on swap type, which blew up causing memory corruption once swapless migration made MAX_SWAPFILES no longer 2 ^ MAX_SWAPFILES_SHIFT.

mm/swapfile.c +1 -1
@@ -1404,7 +1404,7 @@ asmlinkage long sys_swapon(const char __
 	 * from the initial ~0UL that can't be encoded in either the
 	 * swp_entry_t or the architecture definition of a swap pte.
 	 */
-	if (type > swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0))))) {
+	if (type >= MAX_SWAPFILES) {
 		spin_unlock(&swap_lock);
 		goto out;
 	}
#43 lore raw
Quoted 14 lines · Hugh Dickins

Even s390 is okay, isn't it? MartinS's swp_type returns the highest type, 31, corresponding to 32 types, as on every other architecture. You and I and Martin Bligh would prefer this patch...

Remove unnecessary obfuscation from sys_swapon's range check on swap type, which blew up causing memory corruption once swapless migration made MAX_SWAPFILES no longer 2 ^ MAX_SWAPFILES_SHIFT.

mm/swapfile.c asmlinkage long sys_swapon(const char __ · line 1410

Quoted 8 lines
mm/swapfile.c +1 -1
@@ -1404,7 +1404,7 @@ asmlinkage long sys_swapon(const char __
 	 * from the initial ~0UL that can't be encoded in either the
 	 * swp_entry_t or the architecture definition of a swap pte.
 	 */
-	if (type > swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0))))) {
+	if (type >= MAX_SWAPFILES) {
 		spin_unlock(&swap_lock);
 		goto out;
 	}

As long as nobody will have the smart idea to increase MAX_SWAPFILES to more than 32 I'm fine with it.

Signature · 8 lines
blue skies,
  Martin.

Martin Schwidefsky
Linux for zSeries Development & Services
IBM Deutschland Entwicklung GmbH

"Reality continues to ruin my life." - Calvin.
Acked-by Martin Schwidefsky
#44 lore raw
Quoted 7 lines · Martin Schwidefsky
Quoted 3 lines · Hugh Dickins

Even s390 is okay, isn't it? MartinS's swp_type returns the highest type, 31, corresponding to 32 types, as on every other architecture. You and I and Martin Bligh would prefer this patch...

As long as nobody will have the smart idea to increase MAX_SWAPFILES to more than 32 I'm fine with it.

Thanks, Martin: we know very well that we cannot increase MAX_SWAPFILES without going through all the architectures adjusting their swap ptes (and maybe making MAX_SWAPFILES_SHIFT per-arch), so I don't see any problem there. But Christoph has observed that the comment becomes out-of-date, so updated patch (adding your ack) below...

Remove unnecessary obfuscation from sys_swapon's range check on swap type, which blew up causing memory corruption once swapless migration made MAX_SWAPFILES no longer 2 ^ MAX_SWAPFILES_SHIFT.

mm/swapfile.c +1 -13
@@ -1392,19 +1392,7 @@ asmlinkage long sys_swapon(const char __
 		if (!(p->flags & SWP_USED))
 			break;
 	error = -EPERM;
-	/*
-	 * Test if adding another swap device is possible. There are
-	 * two limiting factors: 1) the number of bits for the swap
-	 * type swp_entry_t definition and 2) the number of bits for
-	 * the swap type in the swap ptes as defined by the different
-	 * architectures. To honor both limitations a swap entry
-	 * with swap offset 0 and swap type ~0UL is created, encoded
-	 * to a swap pte, decoded to a swp_entry_t again and finally
-	 * the swap type part is extracted. This will mask all bits
-	 * from the initial ~0UL that can't be encoded in either the
-	 * swp_entry_t or the architecture definition of a swap pte.
-	 */
-	if (type > swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0))))) {
+	if (type >= MAX_SWAPFILES) {
 		spin_unlock(&swap_lock);
 		goto out;
 	}
#45 lore raw

So it seems that this issue was page migration specific. Patch would need to go with the swapless-pm-add-r-w-migration-entries.patch

Quoted 31 lines

Remove unnecessary obfuscation from sys_swapon's range check on swap type, which blew up causing memory corruption once swapless migration made MAX_SWAPFILES no longer 2 ^ MAX_SWAPFILES_SHIFT.

mm/swapfile.c +1 -13
@@ -1392,19 +1392,7 @@ asmlinkage long sys_swapon(const char __
 		if (!(p->flags & SWP_USED))
 			break;
 	error = -EPERM;
-	/*
-	 * Test if adding another swap device is possible. There are
-	 * two limiting factors: 1) the number of bits for the swap
-	 * type swp_entry_t definition and 2) the number of bits for
-	 * the swap type in the swap ptes as defined by the different
-	 * architectures. To honor both limitations a swap entry
-	 * with swap offset 0 and swap type ~0UL is created, encoded
-	 * to a swap pte, decoded to a swp_entry_t again and finally
-	 * the swap type part is extracted. This will mask all bits
-	 * from the initial ~0UL that can't be encoded in either the
-	 * swp_entry_t or the architecture definition of a swap pte.
-	 */
-	if (type > swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0))))) {
+	if (type >= MAX_SWAPFILES) {
 		spin_unlock(&swap_lock);
 		goto out;
 	}
#46 lore raw
In reply to Hugh Dickins
Quoted 76 lines · Hugh Dickins
Quoted 22 lines · Andrew Morton
Quoted 18 lines · On Mon, 5 Jun 2006 13:43:47 -0700 (PDT) Christoph Lameter <[email protected]>…

Fix this by limiting the number of swap devices in swapon to MAX_SWAPFILES

Index: linux-2.6.17-rc5-mm2/mm/swapfile.c
===================================================================

mm/swapfile.c +4 -1
@@ -1408,8 +1408,13 @@ asmlinkage long sys_swapon(const char __
 		spin_unlock(&swap_lock);
 		goto out;
 	}
-	if (type >= nr_swapfiles)
+	if (type >= nr_swapfiles) {
+		if (nr_swapfiles >= MAX_SWAPFILES) {
+			spin_unlock(&swap_lock);
+			goto out;
Quoted 8 lines

mm/swapfile.c asmlinkage long sys_swapon(const char __ · line 1420

Quoted 6 lines
mm/swapfile.c +2 -0
+		}
 		nr_swapfiles = type+1;
+	}
 	INIT_LIST_HEAD(&p->extent_list);
 	p->flags = SWP_USED;
 	p->swap_file = NULL;

Thanks, Christoph. So we get to keep the crappy test?

Christoph's patch looks like it will fix the corruption to me (though I'd have thought the alternative below a little cleaner, keeping the associated tests together and avoiding a second unlock: whatever).

Whether it's correct depends on what Martin was trying to achieve with his test. I'm surprised to find a very ordinary
#define __swp_type(entry) (((entry).val >> 2) & 0x1f)
in include/asm-s390/pgtable.h, and no architecture with a more limiting mask.

I had expected that s390 (or some other) was short of bits and was having to use a more limiting mask: in which case neither Christoph's patch nor the one below would be appropriate, since the migration types 30 and 31 would then fall outside the mask (though it'd only be a problem if page migration were required on such an architecture).

So, the patch should do for now, but it'd be helpful if Martin can explain why he needed to use that test in 2.6.5. (Perhaps he went through a phase of reducing the swap type and extending the swap offset, and this is a remnant of that phase?)

Quoted 4 lines

I wonder what LTP was corrupting before it started to corrupt page migration data?

The above looks like a 2.6.17 thing to me.

Not really (though the clarity and reassurance of the additional MAX_SWAPFILES test is good). We went over it a year or two back, and the macro contortions do involve MAX_SWAPFILES_SHIFT: which up to and including 2.6.17 has enforced the MAX_SWAPFILES limit.

But swapless migration has changed the relationship between MAX_SWAPFILES_SHIFT and MAX_SWAPFILES (-2 when CONFIG_MIGRATION), hence this corruption now surfacing in -mm.

Hugh

mm/swapfile.c asmlinkage long sys_swapon(const char __ · line 1411

Quoted 9 lines
mm/swapfile.c +2 -1
@@ -1404,7 +1404,8 @@ asmlinkage long sys_swapon(const char __
 	 * from the initial ~0UL that can't be encoded in either the
 	 * swp_entry_t or the architecture definition of a swap pte.
 	 */
-	if (type > swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0))))) {
+	if (type >= MAX_SWAPFILES ||
+	    type > swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0))))) {
 		spin_unlock(&swap_lock);
 		goto out;
 	}

Ok, this with Ingo's 2222 deadlock fix has passed across the board.

-apw

#47 lore raw
In reply to Christoph Lameter
Quoted 28 lines · Christoph Lameter

Sigh the patch that I sent earlier will make swapon fail when adding more entries if 32 entries have been defined before even if some of these are later freed. Plus maybe we better leave the probing intact for arches that support less than 32 swap devices and also return -EPERM like before. I guess we need this one instead:

Do proper boundary checking in sys_swapon().

sys_swapon currently does not limit the number of swap devices. It may as a result overwrite memory following the swap_info array and get into entanglements with page migration since it may usethe swap types reserved for page migration.

Fix this by limiting the number of swap devices in swapon to MAX_SWAPFILES

Index: linux-2.6.17-rc5-mm2/mm/swapfile.c
===================================================================

mm/swapfile.c +0 -1
@@ -1408,8 +1408,13 @@ asmlinkage long sys_swapon(const char __
 		spin_unlock(&swap_lock);
 		goto out;
 	}
-	if (type >= nr_swapfiles)

mm/swapfile.c asmlinkage long sys_swapon(const char __ · line 1420

Quoted 10 lines
mm/swapfile.c +6 -0
+	if (type >= nr_swapfiles) {
+		if (nr_swapfiles >= MAX_SWAPFILES) {
+			spin_unlock(&swap_lock);
+			goto out;
+		}
 		nr_swapfiles = type+1;
+	}
 	INIT_LIST_HEAD(&p->extent_list);
 	p->flags = SWP_USED;
 	p->swap_file = NULL;

I'll shove this one onto the machine causing all the trouble.

-apw

#48 lore raw
In reply to Christoph Lameter
Quoted 28 lines · Christoph Lameter

Sigh the patch that I sent earlier will make swapon fail when adding more entries if 32 entries have been defined before even if some of these are later freed. Plus maybe we better leave the probing intact for arches that support less than 32 swap devices and also return -EPERM like before. I guess we need this one instead:

Do proper boundary checking in sys_swapon().

sys_swapon currently does not limit the number of swap devices. It may as a result overwrite memory following the swap_info array and get into entanglements with page migration since it may usethe swap types reserved for page migration.

Fix this by limiting the number of swap devices in swapon to MAX_SWAPFILES

Index: linux-2.6.17-rc5-mm2/mm/swapfile.c
===================================================================

mm/swapfile.c +0 -1
@@ -1408,8 +1408,13 @@ asmlinkage long sys_swapon(const char __
 		spin_unlock(&swap_lock);
 		goto out;
 	}
-	if (type >= nr_swapfiles)

mm/swapfile.c asmlinkage long sys_swapon(const char __ · line 1420

Quoted 10 lines
mm/swapfile.c +6 -0
+	if (type >= nr_swapfiles) {
+		if (nr_swapfiles >= MAX_SWAPFILES) {
+			spin_unlock(&swap_lock);
+			goto out;
+		}
 		nr_swapfiles = type+1;
+	}
 	INIT_LIST_HEAD(&p->extent_list);
 	p->flags = SWP_USE
 	p->swap_file = NULL;

Ok, this patch in combination with your other 2222 deadlock fix are showing passes across the board.

-apw

#49 lore raw
In reply to Christoph Lameter
Quoted 41 lines · On Mon, 5 Jun 2006 13:27:11 -0700 (PDT) Christoph Lameter <[email protected]>…
Quoted 4 lines

Either way, random panics are not the appropriate response ;-)

if it can't cope with that, why isn't it failing the request ???

There is a crappy test in swap_on(). It should check against MAX_SWAPFILES and not do this conversion back and forth. Some architectures may not check if we are beyond the boundaries of what a swap entry can take.

Why is this strange this in there? Are there architectures that support less than 32 swap devices?

Index: linux-2.6.17-rc5-mm2/mm/swapfile.c
===================================================================

mm/swapfile.c +3 -9
@@ -1384,6 +1384,9 @@ asmlinkage long sys_swapon(const char __
 	struct inode *inode = NULL;
 	int did_down = 0;
 
+	if (nr_swapfiles >= MAX_SWAPFILES)
+		return -E2BIG;
+
 	if (!capable(CAP_SYS_ADMIN))
 		return -EPERM;
 	spin_lock(&swap_lock);
@@ -1392,22 +1395,6 @@ asmlinkage long sys_swapon(const char __
 		if (!(p->flags & SWP_USED))
 			break;
 	error = -EPERM;
-	/*
-	 * Test if adding another swap device is possible. There are
-	 * two limiting factors: 1) the number of bits for the swap
-	 * type swp_entry_t definition and 2) the number of bits for
-	 * the swap type in the swap ptes as defined by the different
-	 * architectures. To honor both limitations a swap entry
-	 * with swap offset 0 and swap type ~0UL is created, encoded
-	 * to a swap pte, decoded to a swp_entry_t again and finally
-	 * the swap type part is extracted. This will mask all bits

mm/swapfile.c asmlinkage long sys_swapon(const char __ · line 1400

Quoted 10 lines
mm/swapfile.c +0 -7
-	 * from the initial ~0UL that can't be encoded in either the
-	 * swp_entry_t or the architecture definition of a swap pte.
-	 */
-	if (type > swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0))))) {
-		spin_unlock(&swap_lock);
-		goto out;
-	}
 	if (type >= nr_swapfiles)
 		nr_swapfiles = type+1;
 	INIT_LIST_HEAD(&p->extent_list);

Added [email protected] to cc.

#50 lore raw
In reply to Andy Whitcroft
Quoted 9 lines

I took the swapless-pm-add-r-w-migration-entries.patch and -fix-2.patch as a single patch and didn't test the ----'d level. It worked anywhere I tested up to and including the last one marked GOOD above. Anywhere below that was a mess. Backing out those two patches (and a bunch of dependant ones) seemed to make the problems we get very different, and if Martin is right the same as problems we are seeing on other architectures. So I am leaning to the feeling that this part of -mm is introducing a problem.

I'm not convinced this is all the same problem. I think we have at least 2 bugs, personally ... but possibly a run across everything with CONFIG_MIGRATION off would be helpful if you could do that ? Your beer-owed tab seems to be growing distressingly large though.

Thanks,

M.

#51 lore raw
In reply to Andrew Morton
Quoted 5 lines · Andrew Morton
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

The x86_65 panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.

What are you doing to make this happen?

Quoted 7 lines
Quoted 2 lines

runalltests on LTP

We have to get to the bottom of this - there's a shadow over about 500 patches and we don't know which.

iirc I tried to reproduce this a couple of weeks back and failed.

Martin, is the box still somewhat operational after such a crash? If yes then we could use my crash-tracer to see the kernel function call history leading up to the crash:

http://redhat.com/~mingo/lockdep-patches/latency-tracing-lockdep.patch

just apply the patch, accept the offered Kconfig defaults and it will be configured to do the trace-crashes thing. Reproduce the crash and save /proc/latency_trace - it contains the execution history leading up to the crash. (on the CPU that crashes) Should work on i386 and x86_64.

the trace is saved upon the first crash or lockdep assert that occurs on the box. (but you'll have lockdep disabled, so it's the crash that matters)

if the box dies after a crash then there's a possibility to print the execution history to the serial console - but that takes around 10-15 minutes even on 115200 baud. If you want/need to do this then edit kernel/latency.c and change "trace_print_at_crash = 0" to "trace_print_at_crash = 1".

(btw., the tracer has another neat feature as well: if a kernel crashes or triple faults (and reboots) early during bootup, then the tracer can be configured to print all function calls to the serial console, via early_printk() - right when the function calls happen. I debugged numerous nasty boot-time bugs via this. To set it, change "print_functions = 0" to "print_functions = 1" in kernel/latency.c.)

Ingo

#52 lore raw
Quoted 12 lines · Ingo Molnar

Martin, is the box still somewhat operational after such a crash? If yes then we could use my crash-tracer to see the kernel function call history leading up to the crash:

http://redhat.com/~mingo/lockdep-patches/latency-tracing-lockdep.patch

just apply the patch, accept the offered Kconfig defaults and it will be configured to do the trace-crashes thing. Reproduce the crash and save /proc/latency_trace - it contains the execution history leading up to the crash. (on the CPU that crashes) Should work on i386 and x86_64.

Quoted 3 lines

the trace is saved upon the first crash or lockdep assert that occurs on the box. (but you'll have lockdep disabled, so it's the crash that matters)

i just provoked a NULL pointer dereference with the tracer applied, and /proc/latency_trace contained the proper trace, leading up to the crash:

gettimeo-2333  0D... 2210us : trace_hardirqs_on (restore_nocheck)
gettimeo-2333  0.... 2210us > sys_gettimeofday (00000000 00000000 0000007b)
gettimeo-2333  0.... 2210us : sys_gettimeofday (sysenter_past_esp)
gettimeo-2333  0D... 2211us : do_page_fault (error_code)
gettimeo-2333  0D... 2211us : do_page_fault (c0123238 0 2)
gettimeo-2333  0D... 2211us : do_page_fault (10202 202 7b)
gettimeo-2333  0D... 2211us : trace_hardirqs_on (do_page_fault)
gettimeo-2333  0.... 2211us : lockdep_acquire (do_page_fault)

for best trace output you should have KALLSYMS and KALLSYMS_ALL enabled.

of course it could happen that tracing makes your crash go away ...

Ingo

#53 lore raw
In reply to Ingo Molnar
Quoted 11 lines

Martin, is the box still somewhat operational after such a crash? If yes then we could use my crash-tracer to see the kernel function call history leading up to the crash:

http://redhat.com/~mingo/lockdep-patches/latency-tracing-lockdep.patch

just apply the patch, accept the offered Kconfig defaults and it will be configured to do the trace-crashes thing. Reproduce the crash and save /proc/latency_trace - it contains the execution history leading up to the crash. (on the CPU that crashes) Should work on i386 and x86_64.

the trace is saved upon the first crash or lockdep assert that occurs on the box. (but you'll have lockdep disabled, so it's the crash that matters)

if the box dies after a crash then there's a possibility to print the execution history to the serial console - but that takes around 10-15 minutes even on 115200 baud. If you want/need to do this then edit kernel/latency.c and change "trace_print_at_crash = 0" to "trace_print_at_crash = 1".

(btw., the tracer has another neat feature as well: if a kernel crashes or triple faults (and reboots) early during bootup, then the tracer can be configured to print all function calls to the serial console, via early_printk() - right when the function calls happen. I debugged numerous nasty boot-time bugs via this. To set it, change "print_functions = 0" to "print_functions = 1" in kernel/latency.c.)

Looks cool, but we'll have to beg the IBM guys to run it, I don't have access to that box anymore. Dumping to serial console would be cool, even if it's slow (the test boxes all have that logged).

M.

#54 lore raw
In reply to Andrew Morton
Quoted 16 lines · Andrew Morton
Quoted 5 lines · Martin J. Bligh
Quoted 3 lines · Andrew Morton

Martin Bligh <[email protected]> wrote:

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

The x86_65 panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.

What are you doing to make this happen?

runalltests on LTP

Quoted 8 lines

We have to get to the bottom of this - there's a shadow over about 500 patches and we don't know which.

iirc I tried to reproduce this a couple of weeks back and failed.

Are you able to narrow it down to a particular LTP test? It was mtest01 or something like that? Perhaps we can identify a particular command line which triggers the fault in a standalone fashion?

The LTP output is here:

http://test.kernel.org/abat/33803/debug/test.log.1

The last test run was memset01

From a good test run (http://test.kernel.org/abat/33964/debug/test.log.1)
the one after memset01 is a second instance of the same.

Which is bad I suppose, in that it's likely an intermittent failure. Perhaps you can try running memset01 in a loop? I don't have such a box set up here right now, I'm afraid ... will see what I can do.

OTOH, it looks like this might be a different failure than the double fault we saw in previous -mm's, which was consistently in mtest01, IIRC.

M.

#55 lore raw
In reply to Martin Bligh

Hi!

Quoted 1 line

The x86_65 panic in LTP has changed a bit. Looks more

What? IBM adding one bit to x86_64 so it can run AS/400 on it? :-))))))))))))))))

Pavel

Signature · 1 line
Thanks for all the (sleeping) penguins.