Skip to contentSkip to conversation

tasks getting stuck on mmap_sem?

DISCUSSION David Miller 4 participants 11 messages Aug 17, 2010

lore
Share & tools
lore
#0 lore raw

I just started seeing this with current GIT, is it related to the page table locking shuffle of a few days ago?

[21029.265646] INFO: task git:16643 blocked for more than 120 seconds.
[21029.267054] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[21029.268412] git           D 000000000072f804     0 16643  16273 0x210001101000080
[21029.269809] Call Trace:
[21029.271062]  [000000000072f778] rwsem_down_failed_common+0x90/0xb8
[21029.272390]  [000000000072f804] __down_read+0x30/0x40
[21029.273689]  [0000000000731148] do_sparc64_fault+0x238/0x788
[21029.274996]  [00000000004079e8] sparc64_realfault_common+0x10/0x20
[21029.276347] INFO: task git:16644 blocked for more than 120 seconds.
[21029.277655] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[21029.278993] git           D 000000000072f874     0 16644  16273 0x210041101000080
[21029.280348] Call Trace:
[21029.281542]  [000000000072f778] rwsem_down_failed_common+0x90/0xb8
[21029.282818]  [000000000072f874] __down_write+0x34/0x44
[21029.284057]  [00000000004d9c44] SyS_mmap_pgoff+0xc8/0x118
[21029.285293]  [0000000000406154] linux_sparc_syscall32+0x34/0x40
 ...

etc. it's a set of git threads computing deltas for a remote pull request to the machine.

#1 lore raw
Quoted 3 lines · David Miller

I just started seeing this with current GIT, is it related to the page table locking shuffle of a few days ago?

No. Looks like the rwsem changes broke sparc too. ia64 had some problems too.

ia64 had the type problems, but even after they were fixed there was reports of something similar. So I am starting to suspect that there is something more serious going on here - possibly hidden on x86 due to the strong memory ordering. And possibly just because mmap_sem is _mostly_ just used by a single thread. Threaded apps are still uncommon.

Adding the guilty parties to the cc, leaving your backtrace quoted.

Linus

---

Quoted 21 lines
[21029.265646] INFO: task git:16643 blocked for more than 120 seconds.
[21029.267054] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[21029.268412] git           D 000000000072f804     0 16643  16273 0x210001101000080
[21029.269809] Call Trace:
[21029.271062]  [000000000072f778] rwsem_down_failed_common+0x90/0xb8
[21029.272390]  [000000000072f804] __down_read+0x30/0x40
[21029.273689]  [0000000000731148] do_sparc64_fault+0x238/0x788
[21029.274996]  [00000000004079e8] sparc64_realfault_common+0x10/0x20
[21029.276347] INFO: task git:16644 blocked for more than 120 seconds.
[21029.277655] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[21029.278993] git           D 000000000072f874     0 16644  16273 0x210041101000080
[21029.280348] Call Trace:
[21029.281542]  [000000000072f778] rwsem_down_failed_common+0x90/0xb8
[21029.282818]  [000000000072f874] __down_write+0x34/0x44
[21029.284057]  [00000000004d9c44] SyS_mmap_pgoff+0xc8/0x118
[21029.285293]  [0000000000406154] linux_sparc_syscall32+0x34/0x40

 ...

etc.  it's a set of git threads computing deltas for a remote pull request to the machine.

#2 lore raw
Quoted 1 line · Linus Torvalds

No. Looks like the rwsem changes broke sparc too. ia64 had some problems too.

I did have some similar mmap_sem issues - but the combination of fixing the types
of the RWSEM_* defines to be unsigned, and the return value of ia64_atomic64_add()
to be "long" rather than "int" looks to have cleared up the problems I was seeing. I
could generally see the hung processes in less than 10 consecutive kernel builds, but
ran a few thousand builds over the weekend with no issues.

If git is multi-threaded, it may be hitting some different code path ... but it isn't trivial
for me to try this out (my systems are on an isolated lab network segment).

-Tony

#3 lore raw
Quoted 2 lines · Tony Luck

... but it isn't trivial
for me to try this out (my systems are on an isolated lab network segment).

Setting up for "git pull" would have been messy - but "git grep" is multi-threaded in newer versions of git. So I've set a loop to keep running "git grep" over and over. Load average is above 7, so there must be some threading going on, even though I only see one "git" process in "top(1)". Seems to be running fine so far.

-Tony

#4 lore raw
Quoted 6 lines · Tony Luck

Setting up for "git pull" would have been messy - but "git grep" is multi-threaded in newer versions of git. So I've set a loop to keep running "git grep" over and over.  Load average is above 7, so there must be some threading going on, even though I only see one "git" process in "top(1)".  Seems to be running fine so far.

I'm not sure how page-fault intensive that 'git grep' thing is - I think it does mainly open/read/close on the files and probably doesn't take that many faults once malloc() has created the main heap. In contrast, the delta compute code (that david hit) is probably going to do a lot more page faults due to being way more data-intensive (many more mmap/munmap calls - it's how git tends to like to map git objects).

So a more fault-intensive - and still threaded and all local - git load is likely to do something like

git repack -adf

and do it on something smaller than the whole kernel (git itself is probably a good test-case, and probably isn't that far off from a medium-sized kernel "pull" request).

Linus

#5 lore raw
Quoted 1 line · Linus Torvalds

  git repack -adf

$ /usr/bin/time git repack -adf
Counting objects: 125946, done.
Delta compression using up to 32 threads.
Compressing objects: 100% (124132/124132), done.
Writing objects: 100% (125946/125946), done.
Total 125946 (delta 92181), reused 0 (delta 0)
75.80user 5.83system 0:28.59elapsed 285%CPU (0avgtext+0avgdata
12414976maxresident)k
0inputs+0outputs (9major+17615minor)pagefaults 0swaps

Some threading, some page faults, no processes stuck on rwsem things.

Left that command in a while loop ... 10 repeats so far.

-Tony

#6 lore raw
In reply to Linus Torvalds

Oh, and it strikes me when looking at the sparc rwsem-const.h that it has some unsigned constants in it. I think

  #define RWSEM_WAITING_BIAS              0xffff0000

is actually unsigned, because it's a hex constant that doesn't fit in a signed int. So it's of type 'unsigned int' if I recall the C type rules right.

(That's one of the really _subtle_ C type rules: hex constants actually act differently from decimal constants).

So now RWSEM_ACTIVE_WRITE_BIAS is unsigned too.

The x86 constants are all explicitly signed.

Btw, Michel sent out an email saying:

FYI, I just verified that RWSEM_WAITING_BIAS is defined as signed on all architectures except for ia64. So, this would be consistent with the issue being observed only on ia64.

but I think he missed that subtlety of the C type system. Other architectures do

  #define RWSEM_WAITING_BIAS             (-0x00010000)

which is indeed signed, but Sparc is unsigned despite the lack of a "u".

Of course, maybe I only _think_ I know all the C typing rules. I might be full of sh*t. It happens..I didn't actually look them up.

Linus

Linus

#7 lore raw
Quoted 8 lines · Linus Torvalds

Oh, and it strikes me when looking at the sparc rwsem-const.h that it has some unsigned constants in it. I think

 #define RWSEM_WAITING_BIAS              0xffff0000

is actually unsigned, because it's a hex constant that doesn't fit in a signed int. So it's of type 'unsigned int' if I recall the C type rules right.

Eeeek, this totally escaped me when I had a quick look at constant signed-ness across architectures. This is definitely a problem.

Quoted 1 line

#define RWSEM_WAITING_BIAS (-0x00010000)

David, do you still see failures with v2.6.36-rc1 plus that one line fix ?

Signature · 2 lines
Michel "Walken" Lespinasse
A program is never fully debugged until the last user dies.
#8 lore raw

From: Michel Lespinasse <[email protected]>
Date: Tue, 17 Aug 2010 14:47:05 -0700

Quoted 11 lines
Quoted 8 lines · Linus Torvalds

Oh, and it strikes me when looking at the sparc rwsem-const.h that it has some unsigned constants in it. I think

 #define RWSEM_WAITING_BIAS              0xffff0000

is actually unsigned, because it's a hex constant that doesn't fit in a signed int. So it's of type 'unsigned int' if I recall the C type rules right.

Quoted 6 lines

Eeeek, this totally escaped me when I had a quick look at constant signed-ness across architectures. This is definitely a problem.

Quoted 1 line

#define RWSEM_WAITING_BIAS (-0x00010000)

David, do you still see failures with v2.6.36-rc1 plus that one line fix ?

As I said in a reply to Tony I don't even think your commit is to blame, because even after reverting it I still have the problems.

I'll keep digging into this.

#9 lore raw
In reply to Linus Torvalds

From: Linus Torvalds <[email protected]>
Date: Tue, 17 Aug 2010 14:28:49 -0700

Quoted 8 lines

Oh, and it strikes me when looking at the sparc rwsem-const.h that it has some unsigned constants in it. I think

  #define RWSEM_WAITING_BIAS              0xffff0000

is actually unsigned, because it's a hex constant that doesn't fit in a signed int. So it's of type 'unsigned int' if I recall the C type rules right.

You were right all along, of course. Nice spotting.

Something was messed up with my testing methodology last night such that I didn't test the revert of the lib/rwsem.c changes properly. Those actually did in fact introduce the problem.

And fixing these sparc rwsem values to be signed constants fixes the bug.

Thanks guys.

#10 lore raw
In reply to Tony Luck

From: Tony Luck <[email protected]>
Date: Tue, 17 Aug 2010 11:24:19 -0700

Quoted 4 lines
Quoted 1 line · Linus Torvalds

No. Looks like the rwsem changes broke sparc too. ia64 had some problems too.

Quoted 9 lines

I did have some similar mmap_sem issues - but the combination of fixing the types
of the RWSEM_* defines to be unsigned, and the return value of ia64_atomic64_add()
to be "long" rather than "int" looks to have cleared up the problems I was seeing. I
could generally see the hung processes in less than 10 consecutive kernel builds, but
ran a few thousand builds over the weekend with no issues.

You might be triggering it via threading as well, since make uses vfork() for running sub-jobs.

Quoted 3 lines

If git is multi-threaded, it may be hitting some different code path ... but it isn't trivial
for me to try this out (my systems are on an isolated lab network segment).

Like you I tried fixing atomic64, but as I mentioned it turns out sparc64 (like powerpc) always uses a 32-bit 'int' semaphore count.

I thought perhaps that for some reason the rwsem generic code had a dependency on 'long' so I switched sparc64's rwsems over to 'long' counters last night too, but I still get the problem.

Even reverting the rwsem commit that added the "<" test didn't fix things, so now I'm simply going to bisect.