2.6.17-rc5-mm1
Subsystems (2)
The x86_65 panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.
M
http://test.kernel.org/abat/33807/debug/console.log
nable to handle kernel paging request at 0000000000001034 RIP:
[<ffffffff8048c5c4>] __sched_text_start+0x51c/0x745
PGD 1d836a067 PUD 1e2704067 PMD 0
Oops: 0002 [1] SMP
last sysfs file: /devices/pci0000:00/0000:00:06.0/resource
CPU 0
Modules linked in:
Pid: 230, comm: kswapd0 Not tainted 2.6.17-rc5-mm1-autokern1 #1
RIP: 0010:[<ffffffff8048c5c4>] [<ffffffff8048c5c4>] __sched_text_start+0x51c/0x745
RSP: 0000:ffff810100237b48 EFLAGS: 00010093
RAX: ffff8100e3c592b0 RBX: 0000000000000d84 RCX: ffffffff805f0cd8
RDX: 000000003b9a078a RSI: ffff8100e3c590d0 RDI: 000000000007989e
RBP: ffff810100237be8 R08: ffff81007bfec930 R09: ffff81007bfec8e8
R10: ffff81007bfec8e8 R11: ffff81007ff3c2d8 R12: 0000000000000000
R13: ffff8101fd8ec140 R14: ffff810001011680 R15: 000002b3bb3b33a4
FS: 0000000000000000(0000) GS:ffffffff8061c000(0000) knlGS:00000000f7db6460
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000001034 CR3: 00000001f18fa000 CR4: 00000000000006e0
Process kswapd0 (pid: 230, threadinfo ffff810100236000, task ffff8100e3c590d0)
Stack: ffff81007bfec8e8 ffff81007bfec8e8 ffffffff805f0cd8 000000007ff3c2d8
ffff8100e3c590d0 000000000000c276 ffffffff805f0cd8 ffff8100e3c592b0
ffff81007be407b0 ffff81007be407b0
Call Trace:
[<ffffffff8024d24c>] __remove_from_page_cache+0x1c/0x65
[<ffffffff8048d16c>] cond_resched+0x4c/0x7b
[<ffffffff8025783d>] shrink_inactive_list+0x125/0x882
[<ffffffff8048c7ed>] thread_return+0x0/0xd1
[<ffffffff80258493>] shrink_zone+0xd2/0xf3
[<ffffffff80258a4b>] kswapd+0x359/0x49b
[<ffffffff8023e51c>] autoremove_wake_function+0x0/0x37
[<ffffffff802586f2>] kswapd+0x0/0x49b
[<ffffffff8023dff8>] kthread+0xd0/0xfc
[<ffffffff8020a34e>] child_rip+0x8/0x12
[<ffffffff8023df28>] kthread+0x0/0xfc
[<ffffffff8020a346>] child_rip+0x0/0x12Quoted 38 lines · Martin Bligh
The
x86_65panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.M
http://test.kernel.org/abat/33807/debug/console.log
nable to handle kernel paging request at 0000000000001034 RIP:
[<ffffffff8048c5c4>]__sched_text_start+0x51c/0x745
PGD1d836a067PUD1e2704067PMD 0Oops: 0002 [1] SMP last sysfs file: /devices/pci0000:00/0000:00:06.0/resource CPU 0 Modules linked in: Pid: 230, comm: kswapd0 Not tainted 2.6.17-rc5-mm1-autokern1 #1 RIP: 0010:[<ffffffff8048c5c4>] [<ffffffff8048c5c4>] __sched_text_start+0x51c/0x745 RSP: 0000:ffff810100237b48 EFLAGS: 00010093 RAX: ffff8100e3c592b0 RBX: 0000000000000d84 RCX: ffffffff805f0cd8 RDX: 000000003b9a078a RSI: ffff8100e3c590d0 RDI: 000000000007989e RBP: ffff810100237be8 R08: ffff81007bfec930 R09: ffff81007bfec8e8 R10: ffff81007bfec8e8 R11: ffff81007ff3c2d8 R12: 0000000000000000 R13: ffff8101fd8ec140 R14: ffff810001011680 R15: 000002b3bb3b33a4 FS: 0000000000000000(0000) GS:ffffffff8061c000(0000) knlGS:00000000f7db6460 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000000000001034 CR3: 00000001f18fa000 CR4: 00000000000006e0 Process kswapd0 (pid: 230, threadinfo ffff810100236000, task ffff8100e3c590d0) Stack: ffff81007bfec8e8 ffff81007bfec8e8 ffffffff805f0cd8 000000007ff3c2d8 ffff8100e3c590d0 000000000000c276 ffffffff805f0cd8 ffff8100e3c592b0 ffff81007be407b0 ffff81007be407b0 Call Trace: [<ffffffff8024d24c>] __remove_from_page_cache+0x1c/0x65 [<ffffffff8048d16c>] cond_resched+0x4c/0x7b [<ffffffff8025783d>] shrink_inactive_list+0x125/0x882 [<ffffffff8048c7ed>] thread_return+0x0/0xd1
Quoted 8 lines
[<ffffffff80258493>] shrink_zone+0xd2/0xf3 [<ffffffff80258a4b>] kswapd+0x359/0x49b [<ffffffff8023e51c>] autoremove_wake_function+0x0/0x37 [<ffffffff802586f2>] kswapd+0x0/0x49b [<ffffffff8023dff8>] kthread+0xd0/0xfc [<ffffffff8020a34e>] child_rip+0x8/0x12 [<ffffffff8023df28>] kthread+0x0/0xfc [<ffffffff8020a346>] child_rip+0x0/0x12
What are you doing to make this happen?
Quoted 6 lines · Andrew Morton
Quoted 2 lines · Martin Bligh
The
x86_65panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.What are you doing to make this happen?
runalltests on LTP
M.
Quoted 11 lines · Martin J. Bligh
Quoted 6 lines · Andrew Morton
Quoted 2 lines · Martin Bligh
The
x86_65panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.What are you doing to make this happen?
runalltests on LTP
We have to get to the bottom of this - there's a shadow over about 500 patches and we don't know which.
iirc I tried to reproduce this a couple of weeks back and failed.
Are you able to narrow it down to a particular LTP test? It was mtest01 or something like that? Perhaps we can identify a particular command line which triggers the fault in a standalone fashion?
And why can't I make it happen? Perhaps it's a memory initialisation problem, and it only happens to hit in that stage of LTP because that's when you started doing page reclaim, or something? Perhaps just try putting a heap of memory pressure on the machine, see what that does?
Being unable to reproduce it and not having a theory to go on leaves us kinda stuck. Help, please?
Quoted 10 lines · Andrew Morton
Quoted 8 lines · Martin J. Bligh
Quoted 6 lines · Andrew Morton
Quoted 2 lines · Martin Bligh
The
x86_65panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
What are you doing to make this happen?
runalltests on LTP
We have to get to the bottom of this - there's a shadow over about 500 patches and we don't know which.
We did do one chop, and concluded it wasn't the x86_64 patches.
Quoted 1 line
iirc I tried to reproduce this a couple of weeks back and failed.
It looks like a different panic to me. It was a double-fault before.
Quoted 3 lines
Are you able to narrow it down to a particular LTP test? It was mtest01 or something like that? Perhaps we can identify a particular command line which triggers the fault in a standalone fashion?
I can't do much from here - it's running on an IBM machine. Have to beg Andy, or one of the other IBMers, for help.
Quoted 3 lines
And why can't I make it happen? Perhaps it's a memory initialisation problem, and it only happens to hit in that stage of LTP because that's when you started doing page reclaim, or something?
It consistently happens on -mm, and not mainline, flicking back and forth over time. So if you mean h/w mem init, I don't think so. if you mena some patch in -mm, then yes.
Quoted 2 lines
Perhaps just try putting a heap of memory pressure on the machine,
see what that does?
Yes, the other stuff might not be swapping.
Quoted 2 lines
Being unable to reproduce it and not having a theory to go on leaves us kinda stuck. Help, please?
Yeah, we have a sniff-testing mechanism that works well. However, drill-down still requires significant amounts of human intervention. The next gen of stuff should help do more intelligent stuff, but we're kind of stuck with human-ness for now.
M.
Quoted 55 lines · Martin J. Bligh
Quoted 14 lines · Andrew Morton
Quoted 12 lines · Martin J. Bligh
Quoted 8 lines · Andrew Morton
Quoted 2 lines · Martin Bligh
The
x86_65panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.What are you doing to make this happen?
Quoted 6 lines
Quoted 2 lines
runalltests on LTP
We have to get to the bottom of this - there's a shadow over about 500 patches and we don't know which.
We did do one chop, and concluded it wasn't the
x86_64patches.Quoted 1 line
iirc I tried to reproduce this a couple of weeks back and failed.
It looks like a different panic to me. It was a double-fault before.
Quoted 4 lines
Are you able to narrow it down to a particular LTP test? It was mtest01 or
something like that? Perhaps we can identify a particular command line which triggers the fault in a standalone fashion?I can't do much from here - it's running on an IBM machine. Have to beg Andy, or one of the other IBMers, for help.
Quoted 3 lines
And why can't I make it happen? Perhaps it's a memory initialisation problem, and it only happens to hit in that stage of LTP because that's when you started doing page reclaim, or something?
It consistently happens on -mm, and not mainline, flicking back and forth over time. So if you mean h/w mem init, I don't think so. if you mena some patch in -mm, then yes.
Quoted 3 lines
Perhaps just try putting a heap of memory pressure on the machine,
see what that does?
Yes, the other stuff might not be swapping.
Quoted 8 lines
Quoted 2 lines
Being unable to reproduce it and not having a theory to go on leaves us kinda stuck. Help, please?
Yeah, we have a sniff-testing mechanism that works well. However, drill-down still requires significant amounts of human intervention. The next gen of stuff should help do more intelligent stuff, but we're kind of stuck with human-ness for now.
I am sure I got half way through diagnosing this one. We were context switching to a bad thread. I've been meaning to get back to it. Its at the top of my list for the AM.
-apw
Quoted 9 lines · Martin J. Bligh
Quoted 7 lines · Andrew Morton
Quoted 5 lines · Martin J. Bligh
Quoted 3 lines · Andrew Morton
Martin Bligh <[email protected]> wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
The
x86_65panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.What are you doing to make this happen?
runalltests on LTP
Ok, I think this could well be the same problem I got half way through tracking last time round. We are still handing off threads with non-initialised stacks.
APW: schedule: ffffffff805f0cd8 bad rsp flags=00000000
Kernel panic - not syncing: BAD STACK POINTER
Interestingly this has remained unchanged for dispite the major churn -mm takes so this could well be the underlying issue as we almost always blow up randomly when we do this, in a mess with no stack to work with.
Last time I tried to split search -mm1 and she was being a hideous pig, I just couldn't get any bit of it to compile without the rest. Will try and track this down with the new -mm.
-apw
Quoted 2 lines · On Thu, 01 Jun 2006 17:34:10 +0100 Andy Whitcroft <[email protected]> wrote:
Last time I tried to split search -mm1 and she was being a hideous pig, I just couldn't get any bit of it to compile without the rest.
You shouldn't bisect blindly.
Looking at the series file (ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.17-rc5/2.6.17-rc5-mm2/patch-series):
...
add-scsi_add_host-failure-handling-for-nsp32.patch qla1280-fix-section-mismatch-warnings.patch bogus-disk-geometry-on-large-disks.patch bogus-disk-geometry-on-large-disks-warning-fix.patch megaraid_sas-switch-fw_outstanding-to-an-atomic_t.patch megaraid_sas-add-support-for-zcr-controller.patch megaraid_sas-add-support-for-zcr-controller-fix.patch
...
megaraid_sas-add-support-for-zcr-controller-fix.patch is a fix against megaraid_sas-add-support-for-zcr-controller.patch, so if the bisection lands you on megaraid_sas-add-support-for-zcr-controller.patch then obviously one should apply megaraid_sas-add-support-for-zcr-controller-fix.patch by hand as well.
You'll often find great streams of similarly-named contiguous patches - you should apply either all of them or none of them. Otherwise you're testing a known-to-be-broken kernel which has an already-available fix ;)
http://www.zip.com.au/~akpm/linux/patches/stuff/bisecting-mm-trees.txt
describes the bisection technique I use.
If you're using git-bisect, well, try to have a nice day anyway.
Quoted 1 line
Will try and track this down with the new -mm.
Would be great, thanks.
Quoted 32 lines · Andy Whitcroft
Quoted 13 lines · Martin J. Bligh
Quoted 10 lines · Andrew Morton
Quoted 7 lines · Martin J. Bligh
Quoted 4 lines · Andrew Morton
Martin Bligh <[email protected]> wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
The
x86_65panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.What are you doing to make this happen?
runalltests on LTP
Ok, I think this could well be the same problem I got half way through tracking last time round. We are still handing off threads with non-initialised stacks.
APW: schedule:
ffffffff805f0cd8bad rsp flags=00000000Kernel panic - not syncing: BAD STACK POINTER
Quoted 7 lines
Interestingly this has remained unchanged for dispite the major churn -mm takes so this could well be the underlying issue as we almost always blow up randomly when we do this, in a mess with no stack to work with.
Last time I tried to split search -mm1 and she was being a hideous pig, I just couldn't get any bit of it to compile without the rest. Will try and track this down with the new -mm.
Ok. Did a split search on -mm2 for this. With the full stack I was still tripping up on the bad thread hand-off trigger above. However, when split searching I seemed to get somewhat different panics pretty commonly in the allocator. My split search led me to the start of the swapless page migration patches:
GOOD:page-migration-cleanup-pass-mapping-to-migration-functions.patch GOOD:page-migration-cleanup-move-fallback-handling-into-special-function.patch ----:swapless-pm-add-r-w-migration-entries.patch -BAD:swapless-pm-add-r-w-migration-entries-fix-2.patch
I tried pretty hard to slide this out of -mm and can't say with cirtainty that I did it right. I backed out the following in order to get the first two off (obviously from the bottom up):
swapless-pm-add-r-w-migration-entries.patch swapless-pm-add-r-w-migration-entries-fix-2.patch swapless-page-migration-rip-out-swap-based-logic.patch swapless-page-migration-modify-core-logic.patch more-page-migration-do-not-inc-dec-rss-counters.patch more-page-migration-use-migration-entries-for-file-pages.patch page-migration-update-documentation.patch page-migration-simplify-migrate_pages.patch page-migration-simplify-migrate_pages-tweaks.patch page-migration-handle-freeing-of-pages-in-migrate_pages.patch page-migration-use-allocator-function-for-migrate_pages.patch page-migration-support-moving-of-individual-pages.patch page-migration-detailed-status-for-moving-of-individual-pages.patch page-migration-support-moving-of-individual-pages-fixes.patch page-migration-support-moving-of-individual-pages-x86_64-support.patch page-migration-support-moving-of-individual-pages-x86-support.patch
page-migration-support-moving-of-individual-pages-x86-support-fix.patch
page-migration-support-a-vma-migration-function.patch allow-migration-of-mlocked-pages.patch
That did seem to get rid of the original error. However, I'm now
experiencing a reiserfs4 panic so I'm not 100% confident the problem really is gone, but the reiserfs panic isn't anywhere as near hard as the panic's I have been experiencing; I was able to reboot the machine even though the tests were not completing correctly. Full panic for that one is below:
general protection fault: 0000 [1] SMP
last sysfs file: /devices/pci0000:00/0000:00:06.0/resource
CPU 1
Modules linked in:
Pid: 19163, comm: mkdir09 Not tainted 2.6.17-rc5-mm2-autokern1 #1
RIP: 0010:[<ffffffff802412b0>] [<ffffffff802412b0>]
check_deadlock+0x1f/0x117
RSP: 0000:ffff8101fb803dd8 EFLAGS: 00010002
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
RDX: ffff8101f3e64000 RSI: 0000000000000001 RDI: 2222222222222222
RBP: ffff8101fb803df8 R08: 0000000000000000 R09: ffff8101fb803e68
R10: ffff8101fb802000 R11: 00000000ffffff9c R12: ffff8101fcafa7b0
R13: 2222222222222222 R14: 2222222222222222 R15: 0000000000000000
FS: 000000000804a020(0000) GS:ffff8100e3f27e40(0063) knlGS:00000000f7e15460
CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00000000f7ee2090 CR3: 00000001f4dce000 CR4: 00000000000006e0
Process mkdir09 (pid: 19163, threadinfo ffff8101fb802000, task
ffff8101fdd4b0d0)
Stack: 0000000000000000 ffff8101fcafa7b0 ffff8101f3ae5988 2222222222222222
ffff8101fb803e28 ffffffff80241361 ffff8101f3ae5988 ffff8101fb802000
ffff8101fb803e68 ffff8101fdd4b0d0
Call Trace:
[<ffffffff80241361>] check_deadlock+0xd0/0x117
[<ffffffff80241574>] debug_mutex_add_waiter+0x57/0x6f
[<ffffffff8048cf47>] __mutex_lock_slowpath+0xc7/0x232
[<ffffffff802800bc>] do_path_lookup+0x258/0x29c
[<ffffffff8048d0bb>] mutex_lock+0x9/0xb
[<ffffffff802816db>] do_rmdir+0x84/0xfc
[<ffffffff80281764>] sys_rmdir+0x11/0x13
[<ffffffff8021c2d6>] ia32_sysret+0x0/0xa
Code: 48 8b 57 18 48 85 d2 0f 84 e2 00 00 00 4c 8b 22 45 31 f6 49
RIP [<ffffffff802412b0>] check_deadlock+0x1f/0x117 RSP <ffff8101fb803dd8>
NMI Watchdog detected LOCKUP on CPU 0
CPU 0
Modules linked in:
Pid: 19164, comm: mkdir09 Not tainted 2.6.17-rc5-mm2-autokern1 #1
RIP: 0010:[<ffffffff8048d626>] [<ffffffff8048d626>]
.text.lock.mutex+0x2f/0x65
RSP: 0000:ffff8101f3e65e98 EFLAGS: 00000086
RAX: 0000000000000000 RBX: ffff81007b8ec748 RCX: 0000000000000012
RDX: 0000000000000000 RSI: 0000000000000012 RDI: ffff8101f3ae5988
RBP: ffff8101f3e65eb8 R08: ffff8101803f6cf8 R09: ffff81007b8ec748
R10: 000000000000002f R11: 0000000000000000 R12: ffff8101f3ae5988
R13: 0000000000000246 R14: 0000000000000000 R15: 0000000000000000
FS: 000000000804a020(0000) GS:ffffffff8061b000(0063) knlGS:00000000f7e15460
CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00000000f7ed5cc0 CR3: 00000001f52e4000 CR4: 00000000000006e0
Process mkdir09 (pid: 19164, threadinfo ffff8101f3e64000, task
ffff8101fcafa7b0)
Stack: 0000000000000012 ffff81007b8ec748 0000000000000000 ffff81007b92a000
ffff8101f3e65ec8 ffffffff8048d2a3 ffff8101f3e65f68 ffffffff8028172a
ffff8101f4dd49b8 ffff810180051ec0
Call Trace:
[<ffffffff8048d2a3>] mutex_unlock+0x9/0xb
[<ffffffff8028172a>] do_rmdir+0xd3/0xfc
[<ffffffff80281764>] sys_rmdir+0x11/0x13
[<ffffffff8021c2d6>] ia32_sysret+0x0/0xa-apw
Quoted 7 lines
That did seem to get rid of the original error. However, I'm now
experiencing a reiserfs4 panic so I'm not 100% confident the problem really is gone, but the reiserfs panic isn't anywhere as near hard as the panic's I have been experiencing; I was able to reboot the machine even though the tests were not completing correctly. Full panic for that one is below:
Get the same exact panic on ia32 NUMA-Q (just posted it, but for reference, there's a copy here:
http://test.kernel.org/abat/34623/debug/console.log
). But only in -mm3, not -mm2, so
probably it's intermittent if you were already seeing it on -mm1?
PS. I think the reiser4 thing was a typo, it's generic by the looks of it, and I don't think we test reiser4, is probably ext3 or 2.
Quoted 61 lines
general protection fault: 0000 [1] SMP last sysfs file: /devices/pci0000:00/0000:00:06.0/resource CPU 1 Modules linked in: Pid: 19163, comm: mkdir09 Not tainted 2.6.17-rc5-mm2-autokern1 #1 RIP: 0010:[<ffffffff802412b0>] [<ffffffff802412b0>] check_deadlock+0x1f/0x117 RSP: 0000:ffff8101fb803dd8 EFLAGS: 00010002 RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000 RDX: ffff8101f3e64000 RSI: 0000000000000001 RDI: 2222222222222222 RBP: ffff8101fb803df8 R08: 0000000000000000 R09: ffff8101fb803e68 R10: ffff8101fb802000 R11: 00000000ffffff9c R12: ffff8101fcafa7b0 R13: 2222222222222222 R14: 2222222222222222 R15: 0000000000000000 FS: 000000000804a020(0000) GS:ffff8100e3f27e40(0063) knlGS:00000000f7e15460 CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b CR2: 00000000f7ee2090 CR3: 00000001f4dce000 CR4: 00000000000006e0 Process mkdir09 (pid: 19163, threadinfo ffff8101fb802000, task ffff8101fdd4b0d0) Stack: 0000000000000000 ffff8101fcafa7b0 ffff8101f3ae5988 2222222222222222 ffff8101fb803e28 ffffffff80241361 ffff8101f3ae5988 ffff8101fb802000 ffff8101fb803e68 ffff8101fdd4b0d0 Call Trace: [<ffffffff80241361>] check_deadlock+0xd0/0x117 [<ffffffff80241574>] debug_mutex_add_waiter+0x57/0x6f [<ffffffff8048cf47>] __mutex_lock_slowpath+0xc7/0x232 [<ffffffff802800bc>] do_path_lookup+0x258/0x29c [<ffffffff8048d0bb>] mutex_lock+0x9/0xb [<ffffffff802816db>] do_rmdir+0x84/0xfc [<ffffffff80281764>] sys_rmdir+0x11/0x13 [<ffffffff8021c2d6>] ia32_sysret+0x0/0xa Code: 48 8b 57 18 48 85 d2 0f 84 e2 00 00 00 4c 8b 22 45 31 f6 49 RIP [<ffffffff802412b0>] check_deadlock+0x1f/0x117 RSP <ffff8101fb803dd8> NMI Watchdog detected LOCKUP on CPU 0 CPU 0 Modules linked in: Pid: 19164, comm: mkdir09 Not tainted 2.6.17-rc5-mm2-autokern1 #1 RIP: 0010:[<ffffffff8048d626>] [<ffffffff8048d626>] .text.lock.mutex+0x2f/0x65 RSP: 0000:ffff8101f3e65e98 EFLAGS: 00000086 RAX: 0000000000000000 RBX: ffff81007b8ec748 RCX: 0000000000000012 RDX: 0000000000000000 RSI: 0000000000000012 RDI: ffff8101f3ae5988 RBP: ffff8101f3e65eb8 R08: ffff8101803f6cf8 R09: ffff81007b8ec748 R10: 000000000000002f R11: 0000000000000000 R12: ffff8101f3ae5988 R13: 0000000000000246 R14: 0000000000000000 R15: 0000000000000000 FS: 000000000804a020(0000) GS:ffffffff8061b000(0063) knlGS:00000000f7e15460 CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b CR2: 00000000f7ed5cc0 CR3: 00000001f52e4000 CR4: 00000000000006e0 Process mkdir09 (pid: 19164, threadinfo ffff8101f3e64000, task ffff8101fcafa7b0) Stack: 0000000000000012 ffff81007b8ec748 0000000000000000 ffff81007b92a000 ffff8101f3e65ec8 ffffffff8048d2a3 ffff8101f3e65f68 ffffffff8028172a ffff8101f4dd49b8 ffff810180051ec0 Call Trace: [<ffffffff8048d2a3>] mutex_unlock+0x9/0xb [<ffffffff8028172a>] do_rmdir+0xd3/0xfc [<ffffffff80281764>] sys_rmdir+0x11/0x13 [<ffffffff8021c2d6>] ia32_sysret+0x0/0xa-apw
Quoted 83 lines · Andy Whitcroft
Quoted 37 lines · Andy Whitcroft
Quoted 17 lines · Martin J. Bligh
Quoted 13 lines · Andrew Morton
Quoted 9 lines · Martin J. Bligh
Quoted 5 lines · Andrew Morton
Martin Bligh <[email protected]> wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
The
x86_65panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.What are you doing to make this happen?
runalltests on LTP
Ok, I think this could well be the same problem I got half way through tracking last time round. We are still handing off threads with non-initialised stacks.
APW: schedule:
ffffffff805f0cd8bad rsp flags=00000000Kernel panic - not syncing: BAD STACK POINTERQuoted 7 lines
Interestingly this has remained unchanged for dispite the major churn -mm takes so this could well be the underlying issue as we almost always blow up randomly when we do this, in a mess with no stack to work with.
Last time I tried to split search -mm1 and she was being a hideous pig, I just couldn't get any bit of it to compile without the rest. Will try and track this down with the new -mm.
Ok. Did a split search on -mm2 for this. With the full stack I was still tripping up on the bad thread hand-off trigger above. However, when split searching I seemed to get somewhat different panics pretty commonly in the allocator. My split search led me to the start of the swapless page migration patches:
GOOD:page-migration-cleanup-pass-mapping-to-migration-functions.patch GOOD:page-migration-cleanup-move-fallback-handling-into-special-function.patch ----:swapless-pm-add-r-w-migration-entries.patch -BAD:swapless-pm-add-r-w-migration-entries-fix-2.patch
I tried pretty hard to slide this out of -mm and can't say with cirtainty that I did it right. I backed out the following in order to get the first two off (obviously from the bottom up):
swapless-pm-add-r-w-migration-entries.patch swapless-pm-add-r-w-migration-entries-fix-2.patch swapless-page-migration-rip-out-swap-based-logic.patch swapless-page-migration-modify-core-logic.patch more-page-migration-do-not-inc-dec-rss-counters.patch more-page-migration-use-migration-entries-for-file-pages.patch page-migration-update-documentation.patch page-migration-simplify-migrate_pages.patch page-migration-simplify-migrate_pages-tweaks.patch page-migration-handle-freeing-of-pages-in-migrate_pages.patch page-migration-use-allocator-function-for-migrate_pages.patch page-migration-support-moving-of-individual-pages.patch page-migration-detailed-status-for-moving-of-individual-pages.patch page-migration-support-moving-of-individual-pages-fixes.patch page-migration-support-moving-of-individual-pages-x86_64-support.patch page-migration-support-moving-of-individual-pages-x86-support.patch
page-migration-support-moving-of-individual-pages-x86-support-fix.patch
page-migration-support-a-vma-migration-function.patch allow-migration-of-mlocked-pages.patch
Quoted 6 lines
That did seem to get rid of the original error. However, I'm now
experiencing a reiserfs4 panic so I'm not 100% confident the problem really is gone, but the reiserfs panic isn't anywhere as near hard as the panic's I have been experiencing; I was able to reboot the machine even though the tests were not completing correctly. Full panic for that one is below:
Ok, who can say what I am taking, but its strong. Anyhow, the panic is as below. The reiser4 reference is another issue, not this one.
Quoted 62 lines
general protection fault: 0000 [1] SMP last sysfs file: /devices/pci0000:00/0000:00:06.0/resource CPU 1 Modules linked in: Pid: 19163, comm: mkdir09 Not tainted 2.6.17-rc5-mm2-autokern1 #1 RIP: 0010:[<ffffffff802412b0>] [<ffffffff802412b0>] check_deadlock+0x1f/0x117 RSP: 0000:ffff8101fb803dd8 EFLAGS: 00010002 RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000 RDX: ffff8101f3e64000 RSI: 0000000000000001 RDI: 2222222222222222 RBP: ffff8101fb803df8 R08: 0000000000000000 R09: ffff8101fb803e68 R10: ffff8101fb802000 R11: 00000000ffffff9c R12: ffff8101fcafa7b0 R13: 2222222222222222 R14: 2222222222222222 R15: 0000000000000000 FS: 000000000804a020(0000) GS:ffff8100e3f27e40(0063) knlGS:00000000f7e15460 CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b CR2: 00000000f7ee2090 CR3: 00000001f4dce000 CR4: 00000000000006e0 Process mkdir09 (pid: 19163, threadinfo ffff8101fb802000, task ffff8101fdd4b0d0) Stack: 0000000000000000 ffff8101fcafa7b0 ffff8101f3ae5988 2222222222222222 ffff8101fb803e28 ffffffff80241361 ffff8101f3ae5988 ffff8101fb802000 ffff8101fb803e68 ffff8101fdd4b0d0 Call Trace: [<ffffffff80241361>] check_deadlock+0xd0/0x117 [<ffffffff80241574>] debug_mutex_add_waiter+0x57/0x6f [<ffffffff8048cf47>] __mutex_lock_slowpath+0xc7/0x232 [<ffffffff802800bc>] do_path_lookup+0x258/0x29c [<ffffffff8048d0bb>] mutex_lock+0x9/0xb [<ffffffff802816db>] do_rmdir+0x84/0xfc [<ffffffff80281764>] sys_rmdir+0x11/0x13 [<ffffffff8021c2d6>] ia32_sysret+0x0/0xa Code: 48 8b 57 18 48 85 d2 0f 84 e2 00 00 00 4c 8b 22 45 31 f6 49 RIP [<ffffffff802412b0>] check_deadlock+0x1f/0x117 RSP <ffff8101fb803dd8> NMI Watchdog detected LOCKUP on CPU 0 CPU 0 Modules linked in: Pid: 19164, comm: mkdir09 Not tainted 2.6.17-rc5-mm2-autokern1 #1 RIP: 0010:[<ffffffff8048d626>] [<ffffffff8048d626>] .text.lock.mutex+0x2f/0x65 RSP: 0000:ffff8101f3e65e98 EFLAGS: 00000086 RAX: 0000000000000000 RBX: ffff81007b8ec748 RCX: 0000000000000012 RDX: 0000000000000000 RSI: 0000000000000012 RDI: ffff8101f3ae5988 RBP: ffff8101f3e65eb8 R08: ffff8101803f6cf8 R09: ffff81007b8ec748 R10: 000000000000002f R11: 0000000000000000 R12: ffff8101f3ae5988 R13: 0000000000000246 R14: 0000000000000000 R15: 0000000000000000 FS: 000000000804a020(0000) GS:ffffffff8061b000(0063) knlGS:00000000f7e15460 CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b CR2: 00000000f7ed5cc0 CR3: 00000001f52e4000 CR4: 00000000000006e0 Process mkdir09 (pid: 19164, threadinfo ffff8101f3e64000, task ffff8101fcafa7b0) Stack: 0000000000000012 ffff81007b8ec748 0000000000000000 ffff81007b92a000 ffff8101f3e65ec8 ffffffff8048d2a3 ffff8101f3e65f68 ffffffff8028172a ffff8101f4dd49b8 ffff810180051ec0 Call Trace: [<ffffffff8048d2a3>] mutex_unlock+0x9/0xb [<ffffffff8028172a>] do_rmdir+0xd3/0xfc [<ffffffff80281764>] sys_rmdir+0x11/0x13 [<ffffffff8021c2d6>] ia32_sysret+0x0/0xa-apw
Quoted 10 lines · Andy Whitcroft
Ok. Did a split search on -mm2 for this. With the full stack I was still tripping up on the bad thread hand-off trigger above. However, when split searching I seemed to get somewhat different panics pretty commonly in the allocator. My split search led me to the start of the swapless page migration patches:
GOOD:page-migration-cleanup-pass-mapping-to-migration-functions.patch GOOD:page-migration-cleanup-move-fallback-handling-into-special-function.patch ----:swapless-pm-add-r-w-migration-entries.patch -BAD:swapless-pm-add-r-w-migration-entries-fix-2.patch
So it happens with r-w-migration-entries but not without fix-2? Or does it require the fix-2 in order to happen? Without the fix page migration is broken.
Does LTP include any page migration tests? Guess not? If not then this could be simply due to code rearrangement making the problem appear in a different way.
Which test did this?
Swap migration uses the two higest numbers of swap types (30 and 31). If those are generated by something then we could have trouble but I do not see how the patches could hurt otherwise.
Quoted 22 lines · Christoph Lameter
Quoted 10 lines · Andy Whitcroft
Ok. Did a split search on -mm2 for this. With the full stack I was still tripping up on the bad thread hand-off trigger above. However, when split searching I seemed to get somewhat different panics pretty commonly in the allocator. My split search led me to the start of the swapless page migration patches:
GOOD:page-migration-cleanup-pass-mapping-to-migration-functions.patch GOOD:page-migration-cleanup-move-fallback-handling-into-special-function.patch ----:swapless-pm-add-r-w-migration-entries.patch -BAD:swapless-pm-add-r-w-migration-entries-fix-2.patch
So it happens with r-w-migration-entries but not without fix-2? Or does it require the fix-2 in order to happen? Without the fix page migration is broken.
Does LTP include any page migration tests? Guess not? If not then this could be simply due to code rearrangement making the problem appear in a different way.
Quoted 5 lines
Which test did this?
Swap migration uses the two higest numbers of swap types (30 and 31). If those are generated by something then we could have trouble but I do not see how the patches could hurt otherwise.
ltp has many weird swap tests
(and I assume you're not talking about the pte's that are swapped, but only the file type in the swap signature; since the higher bits in the pte are used by the non-linear vma feature afaik)
Quoted 8 lines · Arjan van de Ven
Quoted 3 lines
Swap migration uses the two higest numbers of swap types (30 and 31). If those are generated by something then we could have trouble but I do not see how the patches could hurt otherwise.
ltp has many weird swap tests
(and I assume you're not talking about the pte's that are swapped, but only the file type in the swap signature; since the higher bits in the pte are used by the non-linear vma feature afaik)
How was the kernel compiled? It must have been running on a NUMA system for page migration to be enabled. Otherwise the fallback definitions in include/linux/swapops.h should remove all the swap migration entry handling.
Quoted 5 lines · Christoph Lameter
Quoted 2 lines · Arjan van de Ven
Quoted 1 line
Swap migration uses the two higest numbers of swap types (30 and 31). If those are generated by something then we could have trouble but I do not see how the patches could hurt otherwise.
Quoted 8 lines
Quoted 5 lines
ltp has many weird swap tests
(and I assume you're not talking about the pte's that are swapped, but only the file type in the swap signature; since the higher bits in the pte are used by the non-linear vma feature afaik)
How was the kernel compiled? It must have been running on a NUMA system for page migration to be enabled. Otherwise the fallback definitions in
include/linux/swapops.hshould remove all the swap migration entry handling.
x86_64 NUMA. Config is here:
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/config/abat/amd64
mtest01 was what killed it, IIRC.
Quoted 8 lines · Martin Bligh
Quoted 3 lines
How was the kernel compiled? It must have been running on a NUMA system for page migration to be enabled. Otherwise the fallback definitions in
include/linux/swapops.hshould remove all the swap migration entry handling.
x86_64NUMA. Config is here:
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/config/abat/amd64mtest01 was what killed it, IIRC.
CONFIG_MIGRATION is not set, no CONFIG_MIGRATION entry is in .config, so page migration was not compiled in. You also forgot to do "make oldconfig" when building the kernel. If you do a "make oldconfig" then your kernel will be build with page migration support.
Either what we see is due to code rearrangement or there is something wrong with the fallback definitions in include/linux/swapops.h.
The swapless-pm-add-r-w-migration-entries.patch also introduces Hugh's reversal of the anon_vma list.
Quoted 7 lines · Christoph Lameter
Quoted 4 lines · Martin Bligh
Quoted 3 lines
How was the kernel compiled? It must have been running on a NUMA system for page migration to be enabled. Otherwise the fallback definitions in
include/linux/swapops.hshould remove all the swap migration entry handling.
Quoted 7 lines
Quoted 4 lines
x86_64NUMA. Config is here:
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/config/abat/amd64mtest01 was what killed it, IIRC.
CONFIG_MIGRATIONis not set, noCONFIG_MIGRATIONentry is in .config, so page migration was not compiled in. You also forgot to do "make oldconfig" when building the kernel. If you do a "make oldconfig" then your kernel will be build with page migration support.
No, that is the config it starts with, sorry ... that was misleading. It then does makeoldconfig from there:
http://test.kernel.org/abat/34624/build/dotconfig
is teh one one from the -mm3 build
CONFIG_MIGRATION=y
Quoted 3 lines
Either what we see is due to code rearrangement or there is something wrong with the fallback definitions in
include/linux/swapops.h.The swapless-pm-add-r-w-migration-entries.patch also introduces Hugh's reversal of the
anon_vmalist.
OK, are those easily separable? Sounds like one good test would be to force CONFIG_MIGRATION to =n, right?
M.
Quoted 4 lines · Martin Bligh
No, that is the config it starts with, sorry ... that was misleading. It then does makeoldconfig from there:
Ok. So page migration is enabled.
Quoted 7 lines
Quoted 4 lines
Either what we see is due to code rearrangement or there is something wrong with the fallback definitions in
include/linux/swapops.h.
The swapless-pm-add-r-w-migration-entries.patch also introduces Hugh's reversal of theanon_vmalist.OK, are those easily separable? Sounds like one good test would be to force
CONFIG_MIGRATIONto =n, right?
CONFIG_MIGRATION off will disable the page migration stuff and make swap types 30 / 31 usable as before. The reversal of the order for the anon_vma list will be preserved.
So yes it may lead us further. If the problem persists with CONFIG_MIGRATION=n then the anon_vma reversal may be an issue.
Quoted 21 lines · Christoph Lameter
Quoted 4 lines · Martin Bligh
No, that is the config it starts with, sorry ... that was misleading. It then does makeoldconfig from there:
Ok. So page migration is enabled.
Quoted 7 lines
Quoted 4 lines
Either what we see is due to code rearrangement or there is something wrong with the fallback definitions in
include/linux/swapops.h.
The swapless-pm-add-r-w-migration-entries.patch also introduces Hugh's reversal of theanon_vmalist.OK, are those easily separable? Sounds like one good test would be to force
CONFIG_MIGRATIONto =n, right?
Quoted 6 lines
CONFIG_MIGRATIONoff will disable the page migration stuff and make swap types 30 / 31 usable as before. The reversal of the order for theanon_vmalist will be preserved.So yes it may lead us further. If the problem persists with
CONFIG_MIGRATION=nthen theanon_vmareversal may be an issue.
Ok. Will test that when I get a minute.
-apw
Quoted 3 lines · Arjan van de Ven
(and I assume you're not talking about the pte's that are swapped, but only the file type in the swap signature; since the higher bits in the pte are used by the non-linear vma feature afaik)
migration entries are swap entries of type 30 and 31. They use the format for swap entries.
Quoted 16 lines · Christoph Lameter
Quoted 10 lines · Andy Whitcroft
Ok. Did a split search on -mm2 for this. With the full stack I was still tripping up on the bad thread hand-off trigger above. However, when split searching I seemed to get somewhat different panics pretty commonly in the allocator. My split search led me to the start of the swapless page migration patches:
GOOD:page-migration-cleanup-pass-mapping-to-migration-functions.patch GOOD:page-migration-cleanup-move-fallback-handling-into-special-function.patch ----:swapless-pm-add-r-w-migration-entries.patch -BAD:swapless-pm-add-r-w-migration-entries-fix-2.patch
Quoted 4 lines
So it happens with r-w-migration-entries but not without fix-2? Or does it require the fix-2 in order to happen? Without the fix page migration is broken.
I took the swapless-pm-add-r-w-migration-entries.patch and -fix-2.patch as a single patch and didn't test the ----'d level. It worked anywhere I tested up to and including the last one marked GOOD above. Anywhere below that was a mess. Backing out those two patches (and a bunch of dependant ones) seemed to make the problems we get very different, and if Martin is right the same as problems we are seeing on other architectures. So I am leaning to the feeling that this part of -mm is introducing a problem.
Quoted 9 lines
Does LTP include any page migration tests? Guess not? If not then this could be simply due to code rearrangement making the problem appear in a different way.
Which test did this?
Swap migration uses the two higest numbers of swap types (30 and 31). If those are generated by something then we could have trouble but I do not see how the patches could hurt otherwise.
I uses an awful lot of very wierd swap things. So possibly there is a
tie in there?
-apw
Adding 65528k swap on ./swapfile01. Priority:-2 extents:116 across:360044k Adding 65528k swap on ./swapfile01. Priority:-3 extents:104 across:569492k Adding 65528k swap on ./swapfile01. Priority:-4 extents:172 across:520952k Unable to find swap-space signature Adding 32k swap on swapfile02. Priority:-5 extents:2 across:60k Adding 32k swap on swapfile03. Priority:-6 extents:1 across:32k Adding 32k swap on swapfile04. Priority:-7 extents:1 across:32k Adding 32k swap on swapfile05. Priority:-8 extents:3 across:44k Adding 32k swap on swapfile06. Priority:-9 extents:1 across:32k Adding 32k swap on swapfile07. Priority:-10 extents:1 across:32k Adding 32k swap on swapfile08. Priority:-11 extents:1 across:32k Adding 32k swap on swapfile09. Priority:-12 extents:1 across:32k Adding 32k swap on swapfile10. Priority:-13 extents:1 across:32k Adding 32k swap on swapfile11. Priority:-14 extents:1 across:32k Adding 32k swap on swapfile12. Priority:-15 extents:1 across:32k Adding 32k swap on swapfile13. Priority:-16 extents:1 across:32k Adding 32k swap on swapfile14. Priority:-17 extents:1 across:32k Adding 32k swap on swapfile15. Priority:-18 extents:1 across:32k Adding 32k swap on swapfile16. Priority:-19 extents:1 across:32k Adding 32k swap on swapfile17. Priority:-20 extents:1 across:32k Adding 32k swap on swapfile18. Priority:-21 extents:1 across:32k Adding 32k swap on swapfile19. Priority:-22 extents:1 across:32k Adding 32k swap on swapfile20. Priority:-23 extents:1 across:32k Adding 32k swap on swapfile21. Priority:-24 extents:1 across:32k Adding 32k swap on swapfile22. Priority:-25 extents:1 across:32k Adding 32k swap on swapfile23. Priority:-26 extents:1 across:32k Adding 32k swap on swapfile24. Priority:-27 extents:1 across:32k Adding 32k swap on swapfile25. Priority:-28 extents:1 across:32k Adding 32k swap on swapfile26. Priority:-29 extents:1 across:32k Adding 32k swap on swapfile27. Priority:-30 extents:1 across:32k Adding 32k swap on swapfile28. Priority:-31 extents:1 across:32k Adding 32k swap on swapfile29. Priority:-32 extents:1 across:32k Adding 32k swap on swapfile30. Priority:-33 extents:1 across:32k Adding 32k swap on swapfile31. Priority:-34 extents:1 across:32k Adding 32k swap on swapfile32. Priority:-35 extents:1 across:32k
Quoted 28 lines · Andy Whitcroft
Adding 65528k swap on ./swapfile01. Priority:-2 extents:116 across:360044k Adding 65528k swap on ./swapfile01. Priority:-3 extents:104 across:569492k Adding 65528k swap on ./swapfile01. Priority:-4 extents:172 across:520952k Unable to find swap-space signature Adding 32k swap on swapfile02. Priority:-5 extents:2 across:60k Adding 32k swap on swapfile03. Priority:-6 extents:1 across:32k Adding 32k swap on swapfile04. Priority:-7 extents:1 across:32k Adding 32k swap on swapfile05. Priority:-8 extents:3 across:44k Adding 32k swap on swapfile06. Priority:-9 extents:1 across:32k Adding 32k swap on swapfile07. Priority:-10 extents:1 across:32k Adding 32k swap on swapfile08. Priority:-11 extents:1 across:32k Adding 32k swap on swapfile09. Priority:-12 extents:1 across:32k Adding 32k swap on swapfile10. Priority:-13 extents:1 across:32k Adding 32k swap on swapfile11. Priority:-14 extents:1 across:32k Adding 32k swap on swapfile12. Priority:-15 extents:1 across:32k Adding 32k swap on swapfile13. Priority:-16 extents:1 across:32k Adding 32k swap on swapfile14. Priority:-17 extents:1 across:32k Adding 32k swap on swapfile15. Priority:-18 extents:1 across:32k Adding 32k swap on swapfile16. Priority:-19 extents:1 across:32k Adding 32k swap on swapfile17. Priority:-20 extents:1 across:32k Adding 32k swap on swapfile18. Priority:-21 extents:1 across:32k Adding 32k swap on swapfile19. Priority:-22 extents:1 across:32k Adding 32k swap on swapfile20. Priority:-23 extents:1 across:32k Adding 32k swap on swapfile21. Priority:-24 extents:1 across:32k Adding 32k swap on swapfile22. Priority:-25 extents:1 across:32k Adding 32k swap on swapfile23. Priority:-26 extents:1 across:32k Adding 32k swap on swapfile24. Priority:-27 extents:1 across:32k
Quoted 8 lines
Adding 32k swap on swapfile25. Priority:-28 extents:1 across:32k Adding 32k swap on swapfile26. Priority:-29 extents:1 across:32k Adding 32k swap on swapfile27. Priority:-30 extents:1 across:32k Adding 32k swap on swapfile28. Priority:-31 extents:1 across:32k Adding 32k swap on swapfile29. Priority:-32 extents:1 across:32k Adding 32k swap on swapfile30. Priority:-33 extents:1 across:32k Adding 32k swap on swapfile31. Priority:-34 extents:1 across:32k Adding 32k swap on swapfile32. Priority:-35 extents:1 across:32k
That should not work at all. It should bomb out at 30 swap files with page migration on.
Quoted 41 lines · Christoph Lameter
Quoted 27 lines · Andy Whitcroft
Adding 65528k swap on ./swapfile01. Priority:-2 extents:116 across:360044k Adding 65528k swap on ./swapfile01. Priority:-3 extents:104 across:569492k Adding 65528k swap on ./swapfile01. Priority:-4 extents:172 across:520952k Unable to find swap-space signature Adding 32k swap on swapfile02. Priority:-5 extents:2 across:60k Adding 32k swap on swapfile03. Priority:-6 extents:1 across:32k Adding 32k swap on swapfile04. Priority:-7 extents:1 across:32k Adding 32k swap on swapfile05. Priority:-8 extents:3 across:44k Adding 32k swap on swapfile06. Priority:-9 extents:1 across:32k Adding 32k swap on swapfile07. Priority:-10 extents:1 across:32k Adding 32k swap on swapfile08. Priority:-11 extents:1 across:32k Adding 32k swap on swapfile09. Priority:-12 extents:1 across:32k Adding 32k swap on swapfile10. Priority:-13 extents:1 across:32k Adding 32k swap on swapfile11. Priority:-14 extents:1 across:32k Adding 32k swap on swapfile12. Priority:-15 extents:1 across:32k Adding 32k swap on swapfile13. Priority:-16 extents:1 across:32k Adding 32k swap on swapfile14. Priority:-17 extents:1 across:32k Adding 32k swap on swapfile15. Priority:-18 extents:1 across:32k Adding 32k swap on swapfile16. Priority:-19 extents:1 across:32k Adding 32k swap on swapfile17. Priority:-20 extents:1 across:32k Adding 32k swap on swapfile18. Priority:-21 extents:1 across:32k Adding 32k swap on swapfile19. Priority:-22 extents:1 across:32k Adding 32k swap on swapfile20. Priority:-23 extents:1 across:32k Adding 32k swap on swapfile21. Priority:-24 extents:1 across:32k Adding 32k swap on swapfile22. Priority:-25 extents:1 across:32k Adding 32k swap on swapfile23. Priority:-26 extents:1 across:32k Adding 32k swap on swapfile24. Priority:-27 extents:1 across:32kQuoted 8 lines
Adding 32k swap on swapfile25. Priority:-28 extents:1 across:32k Adding 32k swap on swapfile26. Priority:-29 extents:1 across:32k Adding 32k swap on swapfile27. Priority:-30 extents:1 across:32k Adding 32k swap on swapfile28. Priority:-31 extents:1 across:32k Adding 32k swap on swapfile29. Priority:-32 extents:1 across:32k Adding 32k swap on swapfile30. Priority:-33 extents:1 across:32k Adding 32k swap on swapfile31. Priority:-34 extents:1 across:32k Adding 32k swap on swapfile32. Priority:-35 extents:1 across:32k
Quoted 2 lines
That should not work at all. It should bomb out at 30 swap files with page migration on.
The implication here is that there can only be 32 entries in-toto ... it feels like we have at least 33/34 as the machine has swap by default ... more to look at!
-apw
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Adding 32k swap on swapfile31. Priority:-34 extents:1 across:32k Adding 32k swap on swapfile32. Priority:-35 extents:1 across:32k
That should not work at all. It should bomb out at 30 swap files with page migration on.
The implication here is that there can only be 32 entries in-toto ... it feels like we have at least 33/34 as the machine has swap by default ... more to look at!
Either way, random panics are not the appropriate response ;-)
if it can't cope with that, why isn't it failing the request ???
M.
Quoted 11 lines · Martin Bligh
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
Adding 32k swap on swapfile31. Priority:-34 extents:1 across:32k Adding 32k swap on swapfile32. Priority:-35 extents:1 across:32k
That should not work at all. It should bomb out at 30 swap files with page migration on.
The implication here is that there can only be 32 entries in-toto ... it feels like we have at least 33/34 as the machine has swap by default ... more to look at!
Either way, random panics are not the appropriate response ;-)
Seems that the check for too many swapfiles is busted somehow. It should never go beyond 32 without page migration. Looking into it.
Quoted 4 lines
Either way, random panics are not the appropriate response ;-)
if it can't cope with that, why isn't it failing the request ???
There is a crappy test in swap_on(). It should check against MAX_SWAPFILES and not do this conversion back and forth. Some architectures may not check if we are beyond the boundaries of what a swap entry can take.
Why is this strange this in there? Are there architectures that support less than 32 swap devices?
mm/swapfile.c +3 -16
@@ -1384,6 +1384,9 @@ asmlinkage long sys_swapon(const char __struct inode *inode =NULL ;int did_down =0 ; +if (nr_swapfiles >=MAX_SWAPFILES ) +return -E2BIG ; +if (!capable (CAP_SYS_ADMIN ))return -EPERM ;spin_lock (&swap_lock);
@@ -1392,22 +1395,6 @@ asmlinkage long sys_swapon(const char __if (!(p->flags &SWP_USED ))break ; error = -EPERM ; -/* -* Test if adding another swap device is possible. There are -* two limiting factors: 1) the number of bits for the swap -* type swp_entry_t definition and 2) the number of bits for -* the swap type in the swap ptes as defined by the different -* architectures. To honor both limitations a swap entry -* with swap offset 0 and swap type ~0UL is created, encoded -* to a swap pte, decoded to a swp_entry_t again and finally -* the swap type part is extracted. This will mask all bits -* from the initial ~0UL that can't be encoded in either the -* swp_entry_t or the architecture definition of a swap pte. -*/ -if (type >swp_type (pte_to_swp_entry (swp_entry_to_pte (swp_entry (~0UL ,0 ))))) { -spin_unlock (&swap_lock); - goto out; - }if (type >= nr_swapfiles) nr_swapfiles = type+1 ;INIT_LIST_HEAD (&p->extent_list);
Sigh the patch that I sent earlier will make swapon fail when adding more entries if 32 entries have been defined before even if some of these are later freed. Plus maybe we better leave the probing intact for arches that support less than 32 swap devices and also return -EPERM like before. I guess we need this one instead:
Do proper boundary checking in sys_swapon().
sys_swapon currently does not limit the number of swap devices. It may as a result overwrite memory following the swap_info array and get into entanglements with page migration since it may usethe swap types reserved for page migration.
Fix this by limiting the number of swap devices in swapon to MAX_SWAPFILES
mm/swapfile.c +6 -1
@@ -1408,8 +1408,13 @@ asmlinkage long sys_swapon(const char __spin_unlock (&swap_lock); goto out; } -if (type >= nr_swapfiles) +if (type >= nr_swapfiles) { +if (nr_swapfiles >=MAX_SWAPFILES ) { +spin_unlock (&swap_lock); + goto out; + } nr_swapfiles = type+1 ; + }INIT_LIST_HEAD (&p->extent_list); p->flags =SWP_USED ; p->swap_file =NULL ;
Quoted 28 lines · On Mon, 5 Jun 2006 13:43:47 -0700 (PDT) Christoph Lameter <[email protected]>…
Sigh the patch that I sent earlier will make swapon fail when adding more entries if 32 entries have been defined before even if some of these are later freed. Plus maybe we better leave the probing intact for arches that support less than 32 swap devices and also return
-EPERMlike before. I guess we need this one instead:Do proper boundary checking in
sys_swapon().
sys_swaponcurrently does not limit the number of swap devices. It may as a result overwrite memory following theswap_infoarray and get into entanglements with page migration since it may usethe swap types reserved for page migration.Fix this by limiting the number of swap devices in swapon to
MAX_SWAPFILESSigned-off-by Christoph Lameter <[email protected]>Index:
linux-2.6.17-rc5-mm2/mm/swapfile.c
===================================================================mm/swapfile.c +0
-1@@ -1408,8 +1408,13 @@ asmlinkage long sys_swapon(const char __spin_unlock (&swap_lock); goto out; } -if (type >= nr_swapfiles)
mm/swapfile.c asmlinkage long sys_swapon(const char __ · line 1420
Quoted 10 lines
mm/swapfile.c +6
-0+if (type >= nr_swapfiles) { +if (nr_swapfiles >=MAX_SWAPFILES ) { +spin_unlock (&swap_lock); + goto out; + } nr_swapfiles = type+1 ; + }INIT_LIST_HEAD (&p->extent_list); p->flags =SWP_USED ; p->swap_file =NULL ;
Thanks, Christoph. So we get to keep the crappy test?
I wonder what LTP was corrupting before it started to corrupt page migration data?
The above looks like a 2.6.17 thing to me.
Quoted 2 lines · Andrew Morton
I wonder what LTP was corrupting before it started to corrupt page migration data?
how about mutex debug data structures? ;)
Quoted 1 line · Andrew Morton
Thanks, Christoph. So we get to keep the crappy test?
Well at least as long as we have not established that it is not needed.
Quoted 2 lines
I wonder what LTP was corrupting before it started to corrupt page migration data?
Whatever came after the swap_info array. What does the System map say on that platform?
Quoted 1 line
The above looks like a 2.6.17 thing to me.
Right.
Quoted 21 lines · Andrew Morton
Quoted 18 lines · On Mon, 5 Jun 2006 13:43:47 -0700 (PDT) Christoph Lameter <[email protected]>…
Fix this by limiting the number of swap devices in swapon to
MAX_SWAPFILESSigned-off-by Christoph Lameter <[email protected]>Index:
linux-2.6.17-rc5-mm2/mm/swapfile.c
===================================================================mm/swapfile.c +4
-1@@ -1408,8 +1408,13 @@ asmlinkage long sys_swapon(const char __spin_unlock (&swap_lock); goto out; } -if (type >= nr_swapfiles) +if (type >= nr_swapfiles) { +if (nr_swapfiles >=MAX_SWAPFILES ) { +spin_unlock (&swap_lock); + goto out;
Quoted 8 lines
mm/swapfile.c asmlinkage long sys_swapon(const char __ · line 1420
Quoted 6 lines
mm/swapfile.c +2
-0+ } nr_swapfiles = type+1 ; + }INIT_LIST_HEAD (&p->extent_list); p->flags =SWP_USED ; p->swap_file =NULL ;Thanks, Christoph. So we get to keep the crappy test?
Christoph's patch looks like it will fix the corruption to me (though I'd have thought the alternative below a little cleaner, keeping the associated tests together and avoiding a second unlock: whatever).
Whether it's correct depends on what Martin was trying to achieve with his test. I'm surprised to find a very ordinary
#define __swp_type(entry) (((entry).val >> 2) & 0x1f)
in include/asm-s390/pgtable.h, and no architecture with a more limiting mask.
I had expected that s390 (or some other) was short of bits and was having to use a more limiting mask: in which case neither Christoph's patch nor the one below would be appropriate, since the migration types 30 and 31 would then fall outside the mask (though it'd only be a problem if page migration were required on such an architecture).
So, the patch should do for now, but it'd be helpful if Martin can explain why he needed to use that test in 2.6.5. (Perhaps he went through a phase of reducing the swap type and extending the swap offset, and this is a remnant of that phase?)
Quoted 4 lines
I wonder what LTP was corrupting before it started to corrupt page migration data?
The above looks like a 2.6.17 thing to me.
Not really (though the clarity and reassurance of the additional MAX_SWAPFILES test is good). We went over it a year or two back, and the macro contortions do involve MAX_SWAPFILES_SHIFT: which up to and including 2.6.17 has enforced the MAX_SWAPFILES limit.
But swapless migration has changed the relationship between MAX_SWAPFILES_SHIFT and MAX_SWAPFILES (-2 when CONFIG_MIGRATION), hence this corruption now surfacing in -mm.
Hugh
mm/swapfile.c +2 -1
@@ -1404,7 +1404,8 @@ asmlinkage long sys_swapon(const char __* from the initial ~0UL that can't be encoded in either the * swp_entry_t or the architecture definition of a swap pte. */ -if (type >swp_type (pte_to_swp_entry (swp_entry_to_pte (swp_entry (~0UL ,0 ))))) { +if (type >=MAX_SWAPFILES || + type >swp_type (pte_to_swp_entry (swp_entry_to_pte (swp_entry (~0UL ,0 ))))) {spin_unlock (&swap_lock); goto out; }
Quoted 3 lines · Hugh Dickins
Christoph's patch looks like it will fix the corruption to me (though I'd have thought the alternative below a little cleaner, keeping the associated tests together and avoiding a second unlock: whatever).
Yup that looks cleaner.
Quoted 5 lines
Whether it's correct depends on what Martin was trying to achieve with his test. I'm surprised to find a very ordinary
#define__swp_type(entry)(((entry).val >> 2) &0x1f)
ininclude/asm-s390/pgtable.h, and no architecture with a more limiting mask.
Yes we need to hear from Martin.
Quoted 4 lines
Not really (though the clarity and reassurance of the additional
MAX_SWAPFILEStest is good). We went over it a year or two back, and the macro contortions do involveMAX_SWAPFILES_SHIFT: which up to and including 2.6.17 has enforced theMAX_SWAPFILESlimit.
It looks though as if the testers were able to define more than 32 swap devices. So there is the danger of overwriting the memory following the swap info if we do not fix this.
Where are the macro contortions? No arch uses MAX_SWAPFILES_SHIFT for its definitions and the only other significant use is in swapops.h to determine the shift.
Quoted 7 lines · Christoph Lameter
Quoted 5 lines
Whether it's correct depends on what Martin was trying to achieve with his test. I'm surprised to find a very ordinary
#define__swp_type(entry)(((entry).val >> 2) &0x1f)
ininclude/asm-s390/pgtable.h, and no architecture with a more limiting mask.Yes we need to hear from Martin.
s390 supports up to 32 swap devices. At the same time on 31 bit systems only up to 4GB of swap is supported per device. That is because there are three reserved bits in the 31 bit pte, (pte & 0x80000900) must be zero. Bit 2^10 is the invalid bit, bit 2^9 the read-only bit. Two more bits are needed to distinguish between the different pte types. That makes 32-7 = 25 bits in total for offset and swap device. 5 bits are used for the swap device number, and 20 for the offset.
I have chosen this split because for 31 the bit systems in 1999 the standard dasd device has been a model 3 with a capacity of 2.6 GB.
So much for the history lesson.
Quoted 5 lines
Quoted 4 lines
Not really (though the clarity and reassurance of the additional
MAX_SWAPFILEStest is good). We went over it a year or two back, and the macro contortions do involveMAX_SWAPFILES_SHIFT: which up to and including 2.6.17 has enforced theMAX_SWAPFILESlimit.
Quoted 7 lines
It looks though as if the testers were able to define more than 32 swap devices. So there is the danger of overwriting the memory following the swap info if we do not fix this.
Where are the macro contortions? No arch uses
MAX_SWAPFILES_SHIFTfor its definitions and the only other significant use is inswapops.hto determine the shift.
swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0)))) evaluates to 31 for s390. The idea has been that the creation of a swap entry with ~0UL as the type should mask off all bits that can not be represented in a swap entry. Convert it back and you have the maximum number of swap devices. That should be true for all architectures.
Signature · 8 lines
blue skies, Martin. Martin Schwidefsky Linux for zSeries Development & Services IBM Deutschland Entwicklung GmbH "Reality continues to ruin my life." - Calvin.
Quoted 5 lines · Martin Schwidefsky
swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0)))) evaluates to 31 for s390. The idea has been that the creation of a swap entry with ~0UL as the type should mask off all bits that can not be represented in a swap entry. Convert it back and you have the maximum number of swap devices. That should be true for all architectures.
It evaluates to 31 it seems for all platforms. Could we replace this expression with MAX_SWAPFILES? While we at it get rid of the "type" variable because we are usually using pointers to swap_info.
Note that there is still another similar check in there for an arch specific test of the number of pages available per swap device.
mm/swapfile.c +10 -22
@@ -1363,12 +1363,11 @@ __initcall(procswaps_init);*/ asmlinkage long sys_swapon (const char __user * specialfile,int swap_flags) { -struct swap_info_struct * p; +struct swap_info_struct *p,*q;char *name =NULL ;struct block_device *bdev =NULL ;struct file *swap_file =NULL ;struct address_space *mapping; -unsigned int type;int i, prev;int error;static int least_priority;
@@ -1387,29 +1386,19 @@ asmlinkage long sys_swapon(const char __if (!capable (CAP_SYS_ADMIN ))return -EPERM ;spin_lock (&swap_lock); - p = swap_info; -for (type =0 ; type < nr_swapfiles ; type++,p++) + +for (p = swap_info; p < swap_info + nr_swapfiles; p++)if (!(p->flags &SWP_USED ))break ; error = -EPERM ; -/* -* Test if adding another swap device is possible. There are -* two limiting factors: 1) the number of bits for the swap -* type swp_entry_t definition and 2) the number of bits for -* the swap type in the swap ptes as defined by the different -* architectures. To honor both limitations a swap entry -* with swap offset 0 and swap type ~0UL is created, encoded -* to a swap pte, decoded to a swp_entry_t again and finally -* the swap type part is extracted. This will mask all bits -* from the initial ~0UL that can't be encoded in either the -* swp_entry_t or the architecture definition of a swap pte. -*/ -if (type >swp_type (pte_to_swp_entry (swp_entry_to_pte (swp_entry (~0UL ,0 ))))) { + +if (p == swap_info +MAX_SWAPFILES ) {spin_unlock (&swap_lock); goto out; } -if (type >= nr_swapfiles) - nr_swapfiles = type+1 ; +if (p >= swap_info + nr_swapfiles) + nr_swapfiles++; +INIT_LIST_HEAD (&p->extent_list); p->flags =SWP_USED ; p->swap_file =NULL ;
@@ -1445,10 +1434,9 @@ asmlinkage long sys_swapon(const char __ inode = mapping->host; error = -EBUSY ; -for (i =0 ; i < nr_swapfiles; i++) { -struct swap_info_struct *q = &swap_info[i]; +for (q = swap_info; q < swap_info + nr_swapfiles; q++) { -if (i == type || !q->swap_file) +if (q == p || !q->swap_file)continue ;if (mapping == q->swap_file->f_mapping) goto bad_swap;
Quoted 4 lines · Christoph Lameter
It evaluates to 31 it seems for all platforms. Could we replace this expression with
MAX_SWAPFILES? While we at it get rid of the "type" variable because we are usually using pointers toswap_info.
Well, if you want to go through all of swapfile.c, changing it over to use pointers rather than type/indexes. But I don't think it's worth it, and I don't think it should be mixed in with this fix.
Quoted 2 lines
Note that there is still another similar check in there for an arch specific test of the number of pages available per swap device.
And that check, also Martin's I believe, has very good justification:
it will vary from arch to arch how big a swap area they can handle, and his check is the right way to do it - no more obscure than it has to be.
Hugh
Quoted 6 lines · Hugh Dickins
Quoted 2 lines
Note that there is still another similar check in there for an arch specific test of the number of pages available per swap device.
And that check, also Martin's I believe, has very good justification:
it will vary from arch to arch how big a swap area they can handle, and his check is the right way to do it - no more obscure than it has to be.
Yes, I added that fix since we managed to crash s390 with a swap disk bigger than 4GB. To keep the symmetry I changed the check for the swap type as well.
Signature · 8 lines
blue skies, Martin. Martin Schwidefsky Linux for zSeries Development & Services IBM Deutschland Entwicklung GmbH "Reality continues to ruin my life." - Calvin.
Quoted 7 lines · Christoph Lameter
Quoted 4 lines · Hugh Dickins
Not really (though the clarity and reassurance of the additional
MAX_SWAPFILEStest is good). We went over it a year or two back, and the macro contortions do involveMAX_SWAPFILES_SHIFT: which up to and including 2.6.17 has enforced theMAX_SWAPFILESlimit.
Quoted 7 lines
It looks though as if the testers were able to define more than 32 swap devices. So there is the danger of overwriting the memory following the swap info if we do not fix this.
Where are the macro contortions? No arch uses
MAX_SWAPFILES_SHIFTfor its definitions and the only other significant use is inswapops.hto determine the shift.
I'll go mad if I try to work it out again: I was as worried as you when I discovered that test in sys_swapon a year or so ago, apparently without any check on MAX_SWAPFILES; and went moaning to Andrew. But once I'd worked through swp_type, pte_to_swp_entry, swp_entry_to_pte, swp_entry, I did come to the conclusion that the MAX_SWAPFILES bound was actually safely built in there.
It's those four in swapops.h, and their concatenation in sys_swapon, that I meant by "macro contortions"; and I think they are safe (or were prior to swapless migration's -2), whatever the arch does for the __ones (which drive me even further insane).
Have you no mercy? Oh, wasn't it rendered safe by the:
#define SWP_TYPE_SHIFT(e) (sizeof(e.val) * 8 - MAX_SWAPFILES_SHIFT) static inline unsigned swp_type(swp_entry_t entry)
{
return (entry.val >> SWP_TYPE_SHIFT(entry));
}Hugh
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines
Not really (though the clarity and reassurance of the additional
MAX_SWAPFILEStest is good). We went over it a year or two back, and the macro contortions do involveMAX_SWAPFILES_SHIFT: which up to and including 2.6.17 has enforced theMAX_SWAPFILESlimit.It looks though as if the testers were able to define more than 32 swap devices. So there is the danger of overwriting the memory following the swap info if we do not fix this.
Where are the macro contortions? No arch uses
MAX_SWAPFILES_SHIFTfor its definitions and the only other significant use is inswapops.hto determine the shift.
Quoted 6 lines
I'll go mad if I try to work it out again: I was as worried as you when I discovered that test in
sys_swapona year or so ago, apparently without any check onMAX_SWAPFILES; and went moaning to Andrew. But once I'd worked throughswp_type,pte_to_swp_entry,swp_entry_to_pte,swp_entry, I did come to the conclusion that theMAX_SWAPFILESbound was actually safely built in there.
If it's that difficult to figure out, is that not reason enough to rip it all out and replace it? ;-) Life seems quite complicated enough as it is.
M.
Quoted 10 lines · Martin Bligh
Quoted 6 lines
I'll go mad if I try to work it out again: I was as worried as you when I discovered that test in
sys_swapona year or so ago, apparently without any check onMAX_SWAPFILES; and went moaning to Andrew. But once I'd worked throughswp_type,pte_to_swp_entry,swp_entry_to_pte,swp_entry, I did come to the conclusion that theMAX_SWAPFILESbound was actually safely built in there.If it's that difficult to figure out, is that not reason enough to rip it all out and replace it? ;-) Life seems quite complicated enough as it is.
Is it that complicated? You convert UINT_MAX as the type to a swap entry and back to a number. __swp_entry_to_pte() needs to do the correct masking. If it doesn't it is broken. The alternative would be to add architecture defines with the maximum number of swap devices and the maximum swap offset.
Signature · 8 lines
blue skies, Martin. Martin Schwidefsky Linux for zSeries Development & Services IBM Deutschland Entwicklung GmbH "Reality continues to ruin my life." - Calvin.
Quoted 11 lines · Martin Bligh
Quoted 7 lines
I'll go mad if I try to work it out again: I was as worried as you when I discovered that test in
sys_swapona year or so ago, apparently without any check onMAX_SWAPFILES; and went moaning to Andrew. But once I'd worked throughswp_type,pte_to_swp_entry,swp_entry_to_pte,swp_entry, I did come to the conclusion that theMAX_SWAPFILESbound was actually safely built in there.If it's that difficult to figure out, is that not reason enough to rip it all out and replace it? ;-) Life seems quite complicated enough as it is.
Not everyone is as deficient as I am. But I do agree with you.
I think Martin had good reason to do his "maxpages = swp_offset(....)" to work out the maximum pgoff expressible in an architecture's swap offset bits; but just went overboard in doing likewise with swap type.
MAX_SWAPFILES and MAX_SWAPFILES_SHIFT are defined in the common include/linux/swap.h, and every architecture supports that many (some could manage more, but swp_type doesn't let them). If an architecture came along which somehow couldn't support that many (seems unlikely to me), then we'd move MAX_SWAPFILES_SHIFT into into arch-dependent files, wouldn't we?
Yes, I'm all for sys_swapon just saying "if (type >= MAX_SWAPFILES)".
Hugh
Quoted 6 lines · Hugh Dickins
I'll go mad if I try to work it out again: I was as worried as you when I discovered that test in
sys_swapona year or so ago, apparently without any check onMAX_SWAPFILES; and went moaning to Andrew. But once I'd worked throughswp_type,pte_to_swp_entry,swp_entry_to_pte,swp_entry, I did come to the conclusion that theMAX_SWAPFILESbound was actually safely built in there.
I do not think we want anyone go mad so could be simplify this whole m*** so that mere mortals can comprehend what is going on while reading the source?
Quoted 1 line
Have you no mercy? Oh, wasn't it rendered safe by the:
Ok. So we have a mystical expression here that needs to be explained:
swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0))))
swp_entry(~0L,0) ->swp_entry(0xffffffffffffffff, 0) ->0xfffffffffffffff << SWP_TYPE_SHIFT ->0x1f
Ok we got the highest swap type and this encodes MAX_SWAPFILES_SHIFT and therefore limits the number of swap
But now we convert this to a arch specific pte. The hope is that the arch will mask off any unsupported bits? This is done by s390?
Then we convert the result back into a swap entry and take its type which will hopefully give us the maximum number of swap devices supported by the arch?
Would it not be better for an arch to explicitly tell us how many swap devices are supported? Then we could make the swap_info array shorter and get rid of this cryptic test.
This test also creates a problem for the migration entries: It is really not clear until runtime how many swap devices are supported so we cannot take the last 2 for page migration. In order for page migration to work all NUMA arches that do not support 32 swap devices need to define CONFIG_MIGRATION=n. Seems that this is only s390?
Quoted 9 lines · Christoph Lameter
Ok. So we have a mystical expression here that needs to be explained:
swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0))))
swp_entry(~0L,0)->swp_entry(0xffffffffffffffff, 0)->0xfffffffffffffff<<SWP_TYPE_SHIFT->0x1f
Quoted 5 lines
Ok we got the highest swap type and this encodes
MAX_SWAPFILES_SHIFTand therefore limits the number of swapBut now we convert this to a arch specific pte. The hope is that the arch will mask off any unsupported bits? This is done by s390?
Yes.
Quoted 3 lines
Then we convert the result back into a swap entry and take its type which will hopefully give us the maximum number of swap devices supported by the arch?
Yes.
Quoted 3 lines
Would it not be better for an arch to explicitly tell us how many swap devices are supported? Then we could make the
swap_infoarray shorter and get rid of this cryptic test.
Do you want to calculate the correct number for all architectures? Have fun trying to decrypt the different pte formats.
Quoted 5 lines
This test also creates a problem for the migration entries: It is really not clear until runtime how many swap devices are supported so we cannot take the last 2 for page migration. In order for page migration to work all NUMA arches that do not support 32 swap devices need to define
CONFIG_MIGRATION=n. Seems that this is only s390?
s390 is not a NUMA system, but it does support 32 swap devices.
Signature · 8 lines
blue skies, Martin. Martin Schwidefsky Linux for zSeries Development & Services IBM Deutschland Entwicklung GmbH "Reality continues to ruin my life." - Calvin.
Quoted 4 lines · Christoph Lameter
Would it not be better for an arch to explicitly tell us how many swap devices are supported? Then we could make the
swap_infoarray shorter and get rid of this cryptic test.
If they differed at all, yes.
Quoted 5 lines
This test also creates a problem for the migration entries: It is really not clear until runtime how many swap devices are supported so we cannot take the last 2 for page migration. In order for page migration to work all NUMA arches that do not support 32 swap devices need to define
CONFIG_MIGRATION=n. Seems that this is only s390?
Even s390 is okay, isn't it? MartinS's swp_type returns the highest type, 31, corresponding to 32 types, as on every other architecture. You and I and Martin Bligh would prefer this patch...
Remove unnecessary obfuscation from sys_swapon's range check on swap type, which blew up causing memory corruption once swapless migration made MAX_SWAPFILES no longer 2 ^ MAX_SWAPFILES_SHIFT.
mm/swapfile.c +1 -1
@@ -1404,7 +1404,7 @@ asmlinkage long sys_swapon(const char __* from the initial ~0UL that can't be encoded in either the * swp_entry_t or the architecture definition of a swap pte. */ -if (type >swp_type (pte_to_swp_entry (swp_entry_to_pte (swp_entry (~0UL ,0 ))))) { +if (type >=MAX_SWAPFILES ) {spin_unlock (&swap_lock); goto out; }
Quoted 14 lines · Hugh Dickins
Even s390 is okay, isn't it? MartinS's
swp_typereturns the highest type, 31, corresponding to 32 types, as on every other architecture. You and I and Martin Bligh would prefer this patch...Remove unnecessary obfuscation from sys_swapon's range check on swap type, which blew up causing memory corruption once swapless migration made
MAX_SWAPFILESno longer 2 ^MAX_SWAPFILES_SHIFT.Signed-off-by Hugh Dickins <[email protected]>
mm/swapfile.c asmlinkage long sys_swapon(const char __ · line 1410
Quoted 8 lines
mm/swapfile.c +1
-1@@ -1404,7 +1404,7 @@ asmlinkage long sys_swapon(const char __* from the initial ~0UL that can't be encoded in either the * swp_entry_t or the architecture definition of a swap pte. */ -if (type >swp_type (pte_to_swp_entry (swp_entry_to_pte (swp_entry (~0UL ,0 ))))) { +if (type >=MAX_SWAPFILES ) {spin_unlock (&swap_lock); goto out; }
As long as nobody will have the smart idea to increase MAX_SWAPFILES to more than 32 I'm fine with it.
Signature · 8 lines
blue skies, Martin. Martin Schwidefsky Linux for zSeries Development & Services IBM Deutschland Entwicklung GmbH "Reality continues to ruin my life." - Calvin.
Hugh Dickins
Acked-by Martin SchwidefskyQuoted 7 lines · Martin Schwidefsky
Quoted 3 lines · Hugh Dickins
Even s390 is okay, isn't it? MartinS's
swp_typereturns the highest type, 31, corresponding to 32 types, as on every other architecture. You and I and Martin Bligh would prefer this patch...As long as nobody will have the smart idea to increase
MAX_SWAPFILESto more than 32 I'm fine with it.
Thanks, Martin: we know very well that we cannot increase MAX_SWAPFILES without going through all the architectures adjusting their swap ptes (and maybe making MAX_SWAPFILES_SHIFT per-arch), so I don't see any problem there. But Christoph has observed that the comment becomes out-of-date, so updated patch (adding your ack) below...
Remove unnecessary obfuscation from sys_swapon's range check on swap type, which blew up causing memory corruption once swapless migration made MAX_SWAPFILES no longer 2 ^ MAX_SWAPFILES_SHIFT.
mm/swapfile.c +1 -13
@@ -1392,19 +1392,7 @@ asmlinkage long sys_swapon(const char __if (!(p->flags &SWP_USED ))break ; error = -EPERM ; -/* -* Test if adding another swap device is possible. There are -* two limiting factors: 1) the number of bits for the swap -* type swp_entry_t definition and 2) the number of bits for -* the swap type in the swap ptes as defined by the different -* architectures. To honor both limitations a swap entry -* with swap offset 0 and swap type ~0UL is created, encoded -* to a swap pte, decoded to a swp_entry_t again and finally -* the swap type part is extracted. This will mask all bits -* from the initial ~0UL that can't be encoded in either the -* swp_entry_t or the architecture definition of a swap pte. -*/ -if (type >swp_type (pte_to_swp_entry (swp_entry_to_pte (swp_entry (~0UL ,0 ))))) { +if (type >=MAX_SWAPFILES ) {spin_unlock (&swap_lock); goto out; }
Christoph Lameter
Acked-bySo it seems that this issue was page migration specific. Patch would need to go with the swapless-pm-add-r-w-migration-entries.patch
Quoted 31 lines
Remove unnecessary obfuscation from sys_swapon's range check on swap type, which blew up causing memory corruption once swapless migration made
MAX_SWAPFILESno longer 2 ^MAX_SWAPFILES_SHIFT.Signed-off-by Hugh Dickins <[email protected]>mm/swapfile.c +1
-13@@ -1392,19 +1392,7 @@ asmlinkage long sys_swapon(const char __if (!(p->flags &SWP_USED ))break ; error = -EPERM ; -/* -* Test if adding another swap device is possible. There are -* two limiting factors: 1) the number of bits for the swap -* type swp_entry_t definition and 2) the number of bits for -* the swap type in the swap ptes as defined by the different -* architectures. To honor both limitations a swap entry -* with swap offset 0 and swap type ~0UL is created, encoded -* to a swap pte, decoded to a swp_entry_t again and finally -* the swap type part is extracted. This will mask all bits -* from the initial ~0UL that can't be encoded in either the -* swp_entry_t or the architecture definition of a swap pte. -*/ -if (type >swp_type (pte_to_swp_entry (swp_entry_to_pte (swp_entry (~0UL ,0 ))))) { +if (type >=MAX_SWAPFILES ) {spin_unlock (&swap_lock); goto out; }
Andy Whitcroft
Acked-byQuoted 76 lines · Hugh Dickins
Quoted 22 lines · Andrew Morton
Quoted 18 lines · On Mon, 5 Jun 2006 13:43:47 -0700 (PDT) Christoph Lameter <[email protected]>…
Fix this by limiting the number of swap devices in swapon to
MAX_SWAPFILESSigned-off-by Christoph Lameter <[email protected]>Index:
linux-2.6.17-rc5-mm2/mm/swapfile.c
===================================================================mm/swapfile.c +4
-1@@ -1408,8 +1408,13 @@ asmlinkage long sys_swapon(const char __spin_unlock (&swap_lock); goto out; } -if (type >= nr_swapfiles) +if (type >= nr_swapfiles) { +if (nr_swapfiles >=MAX_SWAPFILES ) { +spin_unlock (&swap_lock); + goto out;Quoted 8 lines
mm/swapfile.c asmlinkage long sys_swapon(const char __ · line 1420
Quoted 6 lines
mm/swapfile.c +2
-0+ } nr_swapfiles = type+1 ; + }INIT_LIST_HEAD (&p->extent_list); p->flags =SWP_USED ; p->swap_file =NULL ;Thanks, Christoph. So we get to keep the crappy test?
Christoph's patch looks like it will fix the corruption to me (though I'd have thought the alternative below a little cleaner, keeping the associated tests together and avoiding a second unlock: whatever).
Whether it's correct depends on what Martin was trying to achieve with his test. I'm surprised to find a very ordinary
#define__swp_type(entry)(((entry).val >> 2) &0x1f)
ininclude/asm-s390/pgtable.h, and no architecture with a more limiting mask.I had expected that s390 (or some other) was short of bits and was having to use a more limiting mask: in which case neither Christoph's patch nor the one below would be appropriate, since the migration types 30 and 31 would then fall outside the mask (though it'd only be a problem if page migration were required on such an architecture).
So, the patch should do for now, but it'd be helpful if Martin can explain why he needed to use that test in 2.6.5. (Perhaps he went through a phase of reducing the swap type and extending the swap offset, and this is a remnant of that phase?)
Quoted 4 lines
I wonder what LTP was corrupting before it started to corrupt page migration data?
The above looks like a 2.6.17 thing to me.
Not really (though the clarity and reassurance of the additional
MAX_SWAPFILEStest is good). We went over it a year or two back, and the macro contortions do involveMAX_SWAPFILES_SHIFT: which up to and including 2.6.17 has enforced theMAX_SWAPFILESlimit.But swapless migration has changed the relationship between
MAX_SWAPFILES_SHIFTandMAX_SWAPFILES(-2 whenCONFIG_MIGRATION), hence this corruption now surfacing in -mm.Hugh
mm/swapfile.c asmlinkage long sys_swapon(const char __ · line 1411
Quoted 9 lines
mm/swapfile.c +2
-1@@ -1404,7 +1404,8 @@ asmlinkage long sys_swapon(const char __* from the initial ~0UL that can't be encoded in either the * swp_entry_t or the architecture definition of a swap pte. */ -if (type >swp_type (pte_to_swp_entry (swp_entry_to_pte (swp_entry (~0UL ,0 ))))) { +if (type >=MAX_SWAPFILES || + type >swp_type (pte_to_swp_entry (swp_entry_to_pte (swp_entry (~0UL ,0 ))))) {spin_unlock (&swap_lock); goto out; }
Ok, this with Ingo's 2222 deadlock fix has passed across the board.
-apw
Quoted 28 lines · Christoph Lameter
Sigh the patch that I sent earlier will make swapon fail when adding more entries if 32 entries have been defined before even if some of these are later freed. Plus maybe we better leave the probing intact for arches that support less than 32 swap devices and also return
-EPERMlike before. I guess we need this one instead:Do proper boundary checking in
sys_swapon().
sys_swaponcurrently does not limit the number of swap devices. It may as a result overwrite memory following theswap_infoarray and get into entanglements with page migration since it may usethe swap types reserved for page migration.Fix this by limiting the number of swap devices in swapon to
MAX_SWAPFILESSigned-off-by Christoph Lameter <[email protected]>Index:
linux-2.6.17-rc5-mm2/mm/swapfile.c
===================================================================mm/swapfile.c +0
-1@@ -1408,8 +1408,13 @@ asmlinkage long sys_swapon(const char __spin_unlock (&swap_lock); goto out; } -if (type >= nr_swapfiles)
mm/swapfile.c asmlinkage long sys_swapon(const char __ · line 1420
Quoted 10 lines
mm/swapfile.c +6
-0+if (type >= nr_swapfiles) { +if (nr_swapfiles >=MAX_SWAPFILES ) { +spin_unlock (&swap_lock); + goto out; + } nr_swapfiles = type+1 ; + }INIT_LIST_HEAD (&p->extent_list); p->flags =SWP_USED ; p->swap_file =NULL ;
I'll shove this one onto the machine causing all the trouble.
-apw
Andy Whitcroft
Acked-byQuoted 28 lines · Christoph Lameter
Sigh the patch that I sent earlier will make swapon fail when adding more entries if 32 entries have been defined before even if some of these are later freed. Plus maybe we better leave the probing intact for arches that support less than 32 swap devices and also return
-EPERMlike before. I guess we need this one instead:Do proper boundary checking in
sys_swapon().
sys_swaponcurrently does not limit the number of swap devices. It may as a result overwrite memory following theswap_infoarray and get into entanglements with page migration since it may usethe swap types reserved for page migration.Fix this by limiting the number of swap devices in swapon to
MAX_SWAPFILESSigned-off-by Christoph Lameter <[email protected]>Index:
linux-2.6.17-rc5-mm2/mm/swapfile.c
===================================================================mm/swapfile.c +0
-1@@ -1408,8 +1408,13 @@ asmlinkage long sys_swapon(const char __spin_unlock (&swap_lock); goto out; } -if (type >= nr_swapfiles)
mm/swapfile.c asmlinkage long sys_swapon(const char __ · line 1420
Quoted 10 lines
mm/swapfile.c +6
-0+if (type >= nr_swapfiles) { +if (nr_swapfiles >=MAX_SWAPFILES ) { +spin_unlock (&swap_lock); + goto out; + } nr_swapfiles = type+1 ; + }INIT_LIST_HEAD (&p->extent_list); p->flags =SWP_USE p->swap_file = NULL;
Ok, this patch in combination with your other 2222 deadlock fix are showing passes across the board.
-apw
Quoted 41 lines · On Mon, 5 Jun 2006 13:27:11 -0700 (PDT) Christoph Lameter <[email protected]>…
Quoted 4 lines
Either way, random panics are not the appropriate response ;-)
if it can't cope with that, why isn't it failing the request ???
There is a crappy test in
swap_on(). It should check againstMAX_SWAPFILESand not do this conversion back and forth. Some architectures may not check if we are beyond the boundaries of what a swap entry can take.Why is this strange this in there? Are there architectures that support less than 32 swap devices?
Signed-off-by Christoph Lameter <[email protected]>Index:
linux-2.6.17-rc5-mm2/mm/swapfile.c
===================================================================mm/swapfile.c +3
-9@@ -1384,6 +1384,9 @@ asmlinkage long sys_swapon(const char __struct inode *inode =NULL ;int did_down =0 ; +if (nr_swapfiles >=MAX_SWAPFILES ) +return -E2BIG ; +if (!capable (CAP_SYS_ADMIN ))return -EPERM ;spin_lock (&swap_lock);@@ -1392,22 +1395,6 @@ asmlinkage long sys_swapon(const char __if (!(p->flags &SWP_USED ))break ; error = -EPERM ; -/* -* Test if adding another swap device is possible. There are -* two limiting factors: 1) the number of bits for the swap -* type swp_entry_t definition and 2) the number of bits for -* the swap type in the swap ptes as defined by the different -* architectures. To honor both limitations a swap entry -* with swap offset 0 and swap type ~0UL is created, encoded -* to a swap pte, decoded to a swp_entry_t again and finally -* the swap type part is extracted. This will mask all bits
mm/swapfile.c asmlinkage long sys_swapon(const char __ · line 1400
Quoted 10 lines
mm/swapfile.c +0
-7-* from the initial ~0UL that can't be encoded in either the -* swp_entry_t or the architecture definition of a swap pte. -*/ -if (type >swp_type (pte_to_swp_entry (swp_entry_to_pte (swp_entry (~0UL ,0 ))))) { -spin_unlock (&swap_lock); - goto out; - }if (type >= nr_swapfiles) nr_swapfiles = type+1 ;INIT_LIST_HEAD (&p->extent_list);
Added [email protected] to cc.
Quoted 9 lines
I took the swapless-pm-add-r-w-migration-entries.patch and -fix-2.patch as a single patch and didn't test the ----'d level. It worked anywhere I tested up to and including the last one marked GOOD above. Anywhere below that was a mess. Backing out those two patches (and a bunch of dependant ones) seemed to make the problems we get very different, and if Martin is right the same as problems we are seeing on other architectures. So I am leaning to the feeling that this part of -mm is introducing a problem.
I'm not convinced this is all the same problem. I think we have at least 2 bugs, personally ... but possibly a run across everything with CONFIG_MIGRATION off would be helpful if you could do that ? Your beer-owed tab seems to be growing distressingly large though.
Thanks,
M.
Quoted 5 lines · Andrew Morton
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
The
x86_65panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.What are you doing to make this happen?
Quoted 7 lines
Quoted 2 lines
runalltests on LTP
We have to get to the bottom of this - there's a shadow over about 500 patches and we don't know which.
iirc I tried to reproduce this a couple of weeks back and failed.
Martin, is the box still somewhat operational after such a crash? If yes then we could use my crash-tracer to see the kernel function call history leading up to the crash:
http://redhat.com/~mingo/lockdep-patches/latency-tracing-lockdep.patch
just apply the patch, accept the offered Kconfig defaults and it will be configured to do the trace-crashes thing. Reproduce the crash and save /proc/latency_trace - it contains the execution history leading up to the crash. (on the CPU that crashes) Should work on i386 and x86_64.
the trace is saved upon the first crash or lockdep assert that occurs on the box. (but you'll have lockdep disabled, so it's the crash that matters)
if the box dies after a crash then there's a possibility to print the execution history to the serial console - but that takes around 10-15 minutes even on 115200 baud. If you want/need to do this then edit kernel/latency.c and change "trace_print_at_crash = 0" to "trace_print_at_crash = 1".
(btw., the tracer has another neat feature as well: if a kernel crashes or triple faults (and reboots) early during bootup, then the tracer can be configured to print all function calls to the serial console, via early_printk() - right when the function calls happen. I debugged numerous nasty boot-time bugs via this. To set it, change "print_functions = 0" to "print_functions = 1" in kernel/latency.c.)
Ingo
Quoted 12 lines · Ingo Molnar
Martin, is the box still somewhat operational after such a crash? If yes then we could use my crash-tracer to see the kernel function call history leading up to the crash:
http://redhat.com/~mingo/lockdep-patches/latency-tracing-lockdep.patch
just apply the patch, accept the offered
Kconfigdefaults and it will be configured to do the trace-crashes thing. Reproduce the crash and save/proc/latency_trace- it contains the execution history leading up to the crash. (on the CPU that crashes) Should work on i386 andx86_64.
Quoted 3 lines
the trace is saved upon the first crash or lockdep assert that occurs on the box. (but you'll have lockdep disabled, so it's the crash that matters)
i just provoked a NULL pointer dereference with the tracer applied, and /proc/latency_trace contained the proper trace, leading up to the crash:
gettimeo-2333 0D... 2210us : trace_hardirqs_on (restore_nocheck) gettimeo-2333 0.... 2210us > sys_gettimeofday (00000000 00000000 0000007b) gettimeo-2333 0.... 2210us : sys_gettimeofday (sysenter_past_esp) gettimeo-2333 0D... 2211us : do_page_fault (error_code) gettimeo-2333 0D... 2211us : do_page_fault (c0123238 0 2) gettimeo-2333 0D... 2211us : do_page_fault (10202 202 7b) gettimeo-2333 0D... 2211us : trace_hardirqs_on (do_page_fault) gettimeo-2333 0.... 2211us : lockdep_acquire (do_page_fault)
for best trace output you should have KALLSYMS and KALLSYMS_ALL enabled.
of course it could happen that tracing makes your crash go away ...
Ingo
Quoted 11 lines
Martin, is the box still somewhat operational after such a crash? If yes then we could use my crash-tracer to see the kernel function call history leading up to the crash:
http://redhat.com/~mingo/lockdep-patches/latency-tracing-lockdep.patch
just apply the patch, accept the offered
Kconfigdefaults and it will be configured to do the trace-crashes thing. Reproduce the crash and save/proc/latency_trace- it contains the execution history leading up to the crash. (on the CPU that crashes) Should work on i386 andx86_64.the trace is saved upon the first crash or lockdep assert that occurs on the box. (but you'll have lockdep disabled, so it's the crash that matters)
if the box dies after a crash then there's a possibility to print the execution history to the serial console - but that takes around 10-15 minutes even on 115200 baud. If you want/need to do this then edit
kernel/latency.cand change "trace_print_at_crash= 0" to "trace_print_at_crash= 1".(btw., the tracer has another neat feature as well: if a kernel crashes or triple faults (and reboots) early during bootup, then the tracer can be configured to print all function calls to the serial console, via
early_printk()- right when the function calls happen. I debugged numerous nasty boot-time bugs via this. To set it, change "print_functions= 0" to "print_functions= 1" inkernel/latency.c.)
Looks cool, but we'll have to beg the IBM guys to run it, I don't have access to that box anymore. Dumping to serial console would be cool, even if it's slow (the test boxes all have that logged).
M.
Quoted 16 lines · Andrew Morton
Quoted 5 lines · Martin J. Bligh
Quoted 3 lines · Andrew Morton
Martin Bligh <[email protected]> wrote:
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
The
x86_65panic in LTP has changed a bit. Looks more useful now. Possibly just unrelated new stuff. Possibly we got lucky.What are you doing to make this happen?
runalltests on LTP
Quoted 8 lines
We have to get to the bottom of this - there's a shadow over about 500 patches and we don't know which.
iirc I tried to reproduce this a couple of weeks back and failed.
Are you able to narrow it down to a particular LTP test? It was mtest01 or something like that? Perhaps we can identify a particular command line which triggers the fault in a standalone fashion?
The LTP output is here:
http://test.kernel.org/abat/33803/debug/test.log.1
The last test run was memset01
From a good test run (http://test.kernel.org/abat/33964/debug/test.log.1)
the one after memset01 is a second instance of the same.
Which is bad I suppose, in that it's likely an intermittent failure. Perhaps you can try running memset01 in a loop? I don't have such a box set up here right now, I'm afraid ... will see what I can do.
OTOH, it looks like this might be a different failure than the double fault we saw in previous -mm's, which was consistently in mtest01, IIRC.
M.
Hi!
Quoted 1 line
The
x86_65panic in LTP has changed a bit. Looks more
What? IBM adding one bit to x86_64 so it can run AS/400 on it? :-))))))))))))))))
Pavel
Signature · 1 line
Thanks for all the (sleeping) penguins.