Skip to contentSkip to conversation

2.6.22-rc1-mm1 huge pages VM freeze (maybe?)

DISCUSSION Zan Lynx 3 participants 11 messages Jul 31, 2007 – Aug 3, 2007

lore
Share & tools
lore

Zan Lynx

author
#0 lore raw

I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:

ps D ffff81001e57ed40 0 103558 103483
ffff81001f061dc8 0000000000000096 ffff81003d8586e8 ffff81001cbadc00 0000000000000006 ffffffff80537009 0000000000000030 ffffffff807ff700 ffffffff807ff700 ffffffff807ff700 ffffffff807ff700 ffffffff807ff700

Call Trace:
 [<ffffffff80537009>] _spin_unlock+0x29/0x50
 [<ffffffff80536425>] __down_read+0x75/0xaf
 [<ffffffff80295f49>] access_process_vm+0x49/0x190
 [<ffffffff802f3003>] proc_pid_cmdline+0xa3/0x130
 [<ffffffff802f4cea>] proc_info_read+0xba/0x100
 [<ffffffff802b0085>] vfs_read+0xc5/0x180
 [<ffffffff802b0583>] sys_read+0x53/0x90
 [<ffffffff8020c1de>] system_call+0x7e/0x83

and nothing will touch it after that.

Here's my kernel command line:
root=/dev/sda2 rootfstype=reiser4 rootflags=no_write_barrier ro i8042.nomux elevator=cfq resume=/dev/sda3 panic=5 nmi_watchdog=2,panic debug hugepages=32

Here's the "huge" script I was using to run programs:
#!/bin/sh
export LD_PRELOAD=libhugetlbfs.so
export HUGETLB_MORECORE=yes
export HUGETLB_PATH=/mnt/huge
export HUGETLB_VERBOSE=1
exec "$@"

I don't have any more info than that at the moment but I could reproduce it with whatever, on request.

Signature · 1 line
Zan Lynx <[email protected]>
#1 lore raw
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:

I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:

Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?

There was a hugepage problem fixed very recently, in 2.6.23-rc1 IIRC.

Quoted 29 lines

ps D ffff81001e57ed40 0 103558 103483
ffff81001f061dc8 0000000000000096 ffff81003d8586e8 ffff81001cbadc00 0000000000000006 ffffffff80537009 0000000000000030 ffffffff807ff700 ffffffff807ff700 ffffffff807ff700 ffffffff807ff700 ffffffff807ff700

Call Trace:
 [<ffffffff80537009>] _spin_unlock+0x29/0x50
 [<ffffffff80536425>] __down_read+0x75/0xaf
 [<ffffffff80295f49>] access_process_vm+0x49/0x190
 [<ffffffff802f3003>] proc_pid_cmdline+0xa3/0x130
 [<ffffffff802f4cea>] proc_info_read+0xba/0x100
 [<ffffffff802b0085>] vfs_read+0xc5/0x180
 [<ffffffff802b0583>] sys_read+0x53/0x90
 [<ffffffff8020c1de>] system_call+0x7e/0x83

and nothing will touch it after that.

Here's my kernel command line:
root=/dev/sda2 rootfstype=reiser4 rootflags=no_write_barrier ro i8042.nomux elevator=cfq resume=/dev/sda3 panic=5 nmi_watchdog=2,panic debug hugepages=32

Here's the "huge" script I was using to run programs:
#!/bin/sh
export LD_PRELOAD=libhugetlbfs.so
export HUGETLB_MORECORE=yes
export HUGETLB_PATH=/mnt/huge
export HUGETLB_VERBOSE=1
exec "$@"

Quoted 2 lines

I don't have any more info than that at the moment but I could reproduce it with whatever, on request.

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

author
#2 lore raw
Quoted 8 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:

I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:

Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?

D'oh! I mean 2.6.23-rc1-mm1, the 22 was a typo. Cut & paste to be sure:
Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007 x86_64 AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux

Signature · 1 line
Zan Lynx <[email protected]>
#3 lore raw
Quoted 8 lines · Zan Lynx
Quoted 7 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:

I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:

Quoted 6 lines
Quoted 1 line

Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?

D'oh! I mean 2.6.23-rc1-mm1, the 22 was a typo. Cut & paste to be sure:
Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007 x86_64 AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux

Hrm -- if you kill Evolution does the system come back? Or is it unkillable/unusable. I guess you were able to run ps at the same time.

What is Evolution doing (sysrq+t)? For that matter, what was the output from libhuge?

Thanks,
Nish

#4 lore raw
In reply to Zan Lynx
Quoted 8 lines · Zan Lynx
Quoted 7 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:

I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:

Quoted 6 lines
Quoted 1 line

Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?

D'oh! I mean 2.6.23-rc1-mm1, the 22 was a typo. Cut & paste to be sure:
Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007 x86_64 AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux

Just to confirm, still happens with -mm2?

Thanks,
Nish

author
#5 lore raw
Quoted 9 lines · Nish Aravamudan
Quoted 8 lines · Zan Lynx
Quoted 7 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:

I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?

D'oh! I mean 2.6.23-rc1-mm1, the 22 was a typo. Cut & paste to be sure:
Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007 x86_64 AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux

Just to confirm, still happens with -mm2?

No, it does not seem to. Evolution runs OK. ps, top, pmap all work fine.

However, a couple of other things happened. Could be unrelated or only loosely related.

Evolution launches spamd (spamassassin) to filter junk mail. spamd died and I have this in dmesg to show for it:

VM: killing process spamd

spamd would have inherited the libhugetlbfs.so environment variables. There are no other clues as to why it died though.

Also, immediately after launching evolution with libhugetlbfs, I got that USB bug where the mouse starts creating keyboard input. I got some of these in dmesg:
keyboard.c: can't emulate rawmode for keycode 240

That could be pure coincidence, although I had been using the system almost all day before that, and it hadn't happened.

Signature · 1 line
Zan Lynx <[email protected]>
#6 lore raw
Quoted 17 lines · Zan Lynx
Quoted 16 lines · Nish Aravamudan
Quoted 14 lines · Zan Lynx
Quoted 8 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:

I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:

Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?

D'oh! I mean 2.6.23-rc1-mm1, the 22 was a typo. Cut & paste to be sure:
Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007 x86_64 AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux

Quoted 4 lines
Quoted 1 line

Just to confirm, still happens with -mm2?

No, it does not seem to. Evolution runs OK. ps, top, pmap all work fine.

Interesting.

Quoted 10 lines

However, a couple of other things happened. Could be unrelated or only loosely related.

Evolution launches spamd (spamassassin) to filter junk mail. spamd died and I have this in dmesg to show for it:

VM: killing process spamd

spamd would have inherited the libhugetlbfs.so environment variables. There are no other clues as to why it died though.

Interesting. Any chance spamd can be run manually with those env variables, but with HUGETLB_VERBOSE=99 (and/or in gdb) to see what happens to it?

Quoted 7 lines

Also, immediately after launching evolution with libhugetlbfs, I got that USB bug where the mouse starts creating keyboard input. I got some of these in dmesg:
keyboard.c: can't emulate rawmode for keycode 240

That could be pure coincidence, although I had been using the system almost all day before that, and it hadn't happened.

Had you started evolution w/o libhugetlbs at all before that?

It does seem like that would be coincidence.

Thanks,
Nish

#7 lore raw
In reply to Zan Lynx
Quoted 8 lines · Zan Lynx
Quoted 7 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:

I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:

Quoted 6 lines
Quoted 1 line

Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?

D'oh! I mean 2.6.23-rc1-mm1, the 22 was a typo. Cut & paste to be sure:
Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007 x86_64 AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux

Also, are we at all sure this isn't a reiser4 issue? I assume you're able to use Evolution w/o libhuge on rc1-mm1 ok? Any chance to remove reiser4 from the picture? Have you been using libhuge this way regularly? Any chance you know it worked ok with some recent kernel (say 2.6.23-rc1?).

Thanks,
Nish

#8 lore raw
In reply to Randy Dunlap
Quoted 10 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:

I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:

Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?

There was a hugepage problem fixed very recently, in 2.6.23-rc1 IIRC.

Actually fixed just after 2.6.23-rc1:

git describe 5ab3ee7b1cd5c91eb2272764f9d7d1fe4749681e v2.6.23-rc1-14-g5ab3ee7

Thanks,
Nish

#9 lore raw
Quoted 7 lines · Nish Aravamudan
Quoted 6 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:

I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:

Quoted 8 lines
Quoted 3 lines

Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?

There was a hugepage problem fixed very recently, in 2.6.23-rc1 IIRC.

Actually fixed just after 2.6.23-rc1:

git describe 5ab3ee7b1cd5c91eb2272764f9d7d1fe4749681e v2.6.23-rc1-14-g5ab3ee7

Looks to me like Andrew included Ken's patch in his rc1-mm1 anyway, so that shouldn't be the issue. Or did I not read mm/hugetlb.c correctly?

Signature · 2 lines
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***
#10 lore raw
Quoted 12 lines · Randy Dunlap
Quoted 11 lines · Nish Aravamudan
Quoted 9 lines · Randy Dunlap
Quoted 4 lines · On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:

I was playing with huge pages and libhugetlbfs. Small programs like "ls" work fine. I tried running Evolution through libhugetlbfs and the system slowly stops running. One interesting thing is the "ps" command, it gets stuck like this:

Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?

There was a hugepage problem fixed very recently, in 2.6.23-rc1 IIRC.

Quoted 7 lines
Quoted 4 lines

Actually fixed just after 2.6.23-rc1:

git describe 5ab3ee7b1cd5c91eb2272764f9d7d1fe4749681e v2.6.23-rc1-14-g5ab3ee7

Looks to me like Andrew included Ken's patch in his rc1-mm1 anyway, so that shouldn't be the issue. Or did I not read mm/hugetlb.c correctly?

Yeah you're right, the -mm tree has that bug fixed.

Thanks,
Nish