2.6.22-rc1-mm1 huge pages VM freeze (maybe?)
I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:
ps D ffff81001e57ed40 0 103558 103483
ffff81001f061dc8 0000000000000096 ffff81003d8586e8 ffff81001cbadc00 0000000000000006 ffffffff80537009 0000000000000030 ffffffff807ff700 ffffffff807ff700 ffffffff807ff700 ffffffff807ff700 ffffffff807ff700
Call Trace: [<ffffffff80537009>] _spin_unlock+0x29/0x50 [<ffffffff80536425>] __down_read+0x75/0xaf [<ffffffff80295f49>] access_process_vm+0x49/0x190 [<ffffffff802f3003>] proc_pid_cmdline+0xa3/0x130 [<ffffffff802f4cea>] proc_info_read+0xba/0x100 [<ffffffff802b0085>] vfs_read+0xc5/0x180 [<ffffffff802b0583>] sys_read+0x53/0x90 [<ffffffff8020c1de>] system_call+0x7e/0x83
and nothing will touch it after that.
Here's my kernel command line:
root=/dev/sda2 rootfstype=reiser4 rootflags=no_write_barrier ro i8042.nomux elevator=cfq resume=/dev/sda3 panic=5 nmi_watchdog=2,panic debug hugepages=32
Here's the "huge" script I was using to run programs:
#!/bin/sh
export LD_PRELOAD=libhugetlbfs.so
export HUGETLB_MORECORE=yes
export HUGETLB_PATH=/mnt/huge
export HUGETLB_VERBOSE=1
exec "$@"
I don't have any more info than that at the moment but I could reproduce it with whatever, on request.
Signature · 1 line
Zan Lynx <[email protected]>
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:
Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
There was a hugepage problem fixed very recently, in 2.6.23-rc1 IIRC.
Quoted 29 lines
ps D
ffff81001e57ed400 103558 103483
ffff81001f061dc80000000000000096ffff81003d8586e8ffff81001cbadc000000000000000006ffffffff805370090000000000000030ffffffff807ff700ffffffff807ff700ffffffff807ff700ffffffff807ff700ffffffff807ff700Call Trace: [<ffffffff80537009>] _spin_unlock+0x29/0x50 [<ffffffff80536425>] __down_read+0x75/0xaf [<ffffffff80295f49>] access_process_vm+0x49/0x190 [<ffffffff802f3003>] proc_pid_cmdline+0xa3/0x130 [<ffffffff802f4cea>] proc_info_read+0xba/0x100 [<ffffffff802b0085>] vfs_read+0xc5/0x180 [<ffffffff802b0583>] sys_read+0x53/0x90 [<ffffffff8020c1de>] system_call+0x7e/0x83and nothing will touch it after that.
Here's my kernel command line:
root=/dev/sda2 rootfstype=reiser4 rootflags=no_write_barrier ro i8042.nomux elevator=cfq resume=/dev/sda3 panic=5 nmi_watchdog=2,panic debug hugepages=32Here's the "huge" script I was using to run programs:
#!/bin/sh
export LD_PRELOAD=libhugetlbfs.so
export HUGETLB_MORECORE=yes
export HUGETLB_PATH=/mnt/huge
export HUGETLB_VERBOSE=1
exec "$@"
Quoted 2 lines
I don't have any more info than that at the moment but I could reproduce it with whatever, on request.
---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***
Quoted 8 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:
Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
D'oh! I mean 2.6.23-rc1-mm1, the 22 was a typo. Cut & paste to be sure:
Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007 x86_64 AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux
Signature · 1 line
Zan Lynx <[email protected]>
Quoted 8 lines · Zan Lynx
Quoted 7 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:
Quoted 6 lines
Quoted 1 line
Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
D'oh! I mean 2.6.23-rc1-mm1, the 22 was a typo. Cut & paste to be sure:
Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007x86_64AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux
Hrm -- if you kill Evolution does the system come back? Or is it unkillable/unusable. I guess you were able to run ps at the same time.
What is Evolution doing (sysrq+t)? For that matter, what was the output from libhuge?
Thanks,
Nish
Quoted 8 lines · Zan Lynx
Quoted 7 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:
Quoted 6 lines
Quoted 1 line
Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
D'oh! I mean 2.6.23-rc1-mm1, the 22 was a typo. Cut & paste to be sure:
Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007x86_64AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux
Just to confirm, still happens with -mm2?
Thanks,
Nish
Quoted 9 lines · Nish Aravamudan
Quoted 8 lines · Zan Lynx
Quoted 7 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
D'oh! I mean 2.6.23-rc1-mm1, the 22 was a typo. Cut & paste to be sure:
Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007x86_64AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/LinuxJust to confirm, still happens with -mm2?
No, it does not seem to. Evolution runs OK. ps, top, pmap all work fine.
However, a couple of other things happened. Could be unrelated or only loosely related.
Evolution launches spamd (spamassassin) to filter junk mail. spamd died and I have this in dmesg to show for it:
VM: killing process spamd
spamd would have inherited the libhugetlbfs.so environment variables. There are no other clues as to why it died though.
Also, immediately after launching evolution with libhugetlbfs, I got that USB bug where the mouse starts creating keyboard input. I got some of these in dmesg:keyboard.c: can't emulate rawmode for keycode 240
That could be pure coincidence, although I had been using the system almost all day before that, and it hadn't happened.
Signature · 1 line
Zan Lynx <[email protected]>
Quoted 17 lines · Zan Lynx
Quoted 16 lines · Nish Aravamudan
Quoted 14 lines · Zan Lynx
Quoted 8 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:
Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
D'oh! I mean 2.6.23-rc1-mm1, the 22 was a typo. Cut & paste to be sure:
Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007x86_64AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux
Quoted 4 lines
Quoted 1 line
Just to confirm, still happens with -mm2?
No, it does not seem to. Evolution runs OK. ps, top, pmap all work fine.
Interesting.
Quoted 10 lines
However, a couple of other things happened. Could be unrelated or only loosely related.
Evolution launches spamd (spamassassin) to filter junk mail. spamd died and I have this in dmesg to show for it:
VM: killing process spamd
spamd would have inherited the libhugetlbfs.so environment variables. There are no other clues as to why it died though.
Interesting. Any chance spamd can be run manually with those env variables, but with HUGETLB_VERBOSE=99 (and/or in gdb) to see what happens to it?
Quoted 7 lines
Also, immediately after launching evolution with libhugetlbfs, I got that USB bug where the mouse starts creating keyboard input. I got some of these in dmesg:
keyboard.c: can't emulate rawmode for keycode 240That could be pure coincidence, although I had been using the system almost all day before that, and it hadn't happened.
Had you started evolution w/o libhugetlbs at all before that?
It does seem like that would be coincidence.
Thanks,
Nish
Quoted 8 lines · Zan Lynx
Quoted 7 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:
Quoted 6 lines
Quoted 1 line
Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
D'oh! I mean 2.6.23-rc1-mm1, the 22 was a typo. Cut & paste to be sure:
Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007x86_64AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux
Also, are we at all sure this isn't a reiser4 issue? I assume you're able to use Evolution w/o libhuge on rc1-mm1 ok? Any chance to remove reiser4 from the picture? Have you been using libhuge this way regularly? Any chance you know it worked ok with some recent kernel (say 2.6.23-rc1?).
Thanks,
Nish
Quoted 10 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:
Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
There was a hugepage problem fixed very recently, in 2.6.23-rc1 IIRC.
Actually fixed just after 2.6.23-rc1:
git describe 5ab3ee7b1cd5c91eb2272764f9d7d1fe4749681e v2.6.23-rc1-14-g5ab3ee7
Thanks,
Nish
Quoted 7 lines · Nish Aravamudan
Quoted 6 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:
Quoted 8 lines
Quoted 3 lines
Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
There was a hugepage problem fixed very recently, in 2.6.23-rc1 IIRC.
Actually fixed just after 2.6.23-rc1:
git describe
5ab3ee7b1cd5c91eb2272764f9d7d1fe4749681ev2.6.23-rc1-14-g5ab3ee7
Looks to me like Andrew included Ken's patch in his rc1-mm1 anyway, so that shouldn't be the issue. Or did I not read mm/hugetlb.c correctly?
Signature · 2 lines
~Randy *** Remember to use Documentation/SubmitChecklist when testing your code ***
Quoted 12 lines · Randy Dunlap
Quoted 11 lines · Nish Aravamudan
Quoted 9 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:
Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
There was a hugepage problem fixed very recently, in 2.6.23-rc1 IIRC.
Quoted 7 lines
Quoted 4 lines
Actually fixed just after 2.6.23-rc1:
git describe
5ab3ee7b1cd5c91eb2272764f9d7d1fe4749681ev2.6.23-rc1-14-g5ab3ee7Looks to me like Andrew included Ken's patch in his rc1-mm1 anyway, so that shouldn't be the issue. Or did I not read
mm/hugetlb.ccorrectly?
Yeah you're right, the -mm tree has that bug fixed.
Thanks,
Nish