Skip to contentSkip to conversation

Stupid futex question - 2.6.33-rc7-mmotm0210

DISCUSSION [email protected] 4 participants 6 messages Feb 18, 2010 – Feb 19, 2010

lore
Share & tools
lore
#0 lore raw

Kernel: x86_64 2.6.33-rc7-mmotm0210

I'm debugging a problem where pulseaudio is getting killed with a SIGKILL out of the blue. It appears to be a problem where pulseaudio sets RLIMIT_RTTIME and the bound gets exceeded. Analysis with 'top' shows a short spike of 96% system time, and the tail end of strace shows this:

[pid 25065] 01:50:20.371484 ioctl(28, USBDEVFS_CONTROL, 0x7fd3d76f630c) = 0 <0.000015> [pid 25065] 01:50:20.371548 ioctl(28, 0x40045532, 0x7fd3d76f636c) = 0 <0.000016> [pid 25065] 01:50:20.371611 open("/dev/snd/pcmC0D0p", O_RDWR|O_NONBLOCK|O_CLOEXEC <unfinished ...> [pid 25064] 01:50:20.371678 <... write resumed> ) = 8 <0.002104>
[pid 25064] 01:50:20.371718 futex(0xc2ec00, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...> [pid 25066] 01:50:21.408392 +++ killed by SIGKILL +++
PANIC: handle_group_exit: 25066 leader 25064
[pid 25065] 01:50:21.408442 +++ killed by SIGKILL +++
PANIC: handle_group_exit: 25065 leader 25064
01:50:21.420354 +++ killed by SIGKILL +++

thread 25064 apparently gets gunned down due to RTTIME because it spent a whole second in a futex() call - is it reasonable for futex() to not return for that long?

In other words - kernel bug because futex() should return, or pulseaudio bug for not understanding futex() can snooze a while?

If a kernel bug, anybody got a better idea than nuking the RLIMIT_RTTIME call, waiting for it to repeat (takes between 1 minute and 1 hour or so), and whomping it a few times with sysrq-T?

#1 lore raw
Quoted 22 lines · On Thu, 2010-02-18 at 09:04 -0500, [email protected] wrote:

Kernel: x86_64 2.6.33-rc7-mmotm0210

I'm debugging a problem where pulseaudio is getting killed with a SIGKILL out of the blue. It appears to be a problem where pulseaudio sets RLIMIT_RTTIME and the bound gets exceeded. Analysis with 'top' shows a short spike of 96% system time, and the tail end of strace shows this:

[pid 25065] 01:50:20.371484 ioctl(28, USBDEVFS_CONTROL, 0x7fd3d76f630c) = 0 <0.000015> [pid 25065] 01:50:20.371548 ioctl(28, 0x40045532, 0x7fd3d76f636c) = 0 <0.000016> [pid 25065] 01:50:20.371611 open("/dev/snd/pcmC0D0p", O_RDWR|O_NONBLOCK|O_CLOEXEC <unfinished ...> [pid 25064] 01:50:20.371678 <... write resumed> ) = 8 <0.002104>
[pid 25064] 01:50:20.371718 futex(0xc2ec00, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...> [pid 25066] 01:50:21.408392 +++ killed by SIGKILL +++
PANIC: handle_group_exit: 25066 leader 25064
[pid 25065] 01:50:21.408442 +++ killed by SIGKILL +++
PANIC: handle_group_exit: 25065 leader 25064
01:50:21.420354 +++ killed by SIGKILL +++

thread 25064 apparently gets gunned down due to RTTIME because it spent a whole second in a futex() call - is it reasonable for futex() to not return for that long?

Quoted 6 lines

In other words - kernel bug because futex() should return, or pulseaudio bug for not understanding futex() can snooze a while?

If a kernel bug, anybody got a better idea than nuking the RLIMIT_RTTIME call, waiting for it to repeat (takes between 1 minute and 1 hour or so), and whomping it a few times with sysrq-T?

is that second spend in processing sysrq-t?

#2 lore raw

(Adding some cc: to the list)

Quoted 23 lines · Peter Zijlstra
Quoted 22 lines · On Thu, 2010-02-18 at 09:04 -0500, [email protected] wrote:

Kernel: x86_64 2.6.33-rc7-mmotm0210

I'm debugging a problem where pulseaudio is getting killed with a SIGKILL out of the blue. It appears to be a problem where pulseaudio sets RLIMIT_RTTIME and the bound gets exceeded. Analysis with 'top' shows a short spike of 96% system time, and the tail end of strace shows this:

[pid 25065] 01:50:20.371484 ioctl(28, USBDEVFS_CONTROL, 0x7fd3d76f630c) = 0 <0.000015> [pid 25065] 01:50:20.371548 ioctl(28, 0x40045532, 0x7fd3d76f636c) = 0 <0.000016> [pid 25065] 01:50:20.371611 open("/dev/snd/pcmC0D0p", O_RDWR|O_NONBLOCK|O_CLOEXEC <unfinished ...> [pid 25064] 01:50:20.371678 <... write resumed> ) = 8 <0.002104>
[pid 25064] 01:50:20.371718 futex(0xc2ec00, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...> [pid 25066] 01:50:21.408392 +++ killed by SIGKILL +++
PANIC: handle_group_exit: 25066 leader 25064
[pid 25065] 01:50:21.408442 +++ killed by SIGKILL +++
PANIC: handle_group_exit: 25065 leader 25064
01:50:21.420354 +++ killed by SIGKILL +++

thread 25064 apparently gets gunned down due to RTTIME because it spent a whole second in a futex() call - is it reasonable for futex() to not return for that long?

Quoted 8 lines
Quoted 6 lines

In other words - kernel bug because futex() should return, or pulseaudio bug for not understanding futex() can snooze a while?

If a kernel bug, anybody got a better idea than nuking the RLIMIT_RTTIME call, waiting for it to repeat (takes between 1 minute and 1 hour or so), and whomping it a few times with sysrq-T?

is that second spend in processing sysrq-t?

No, currently that second is spent in a futex() syscall - I'm wondering:

1) should it get killed for RLIMIT_RTTIME because it's been in a futex() for multiple seconds? It seems suspicious - docs say a blocking syscall resets RTTIME - so if futex() blocks it shouldn't kill, and if it's in the kernel for a second without blocking it's a bug too.

2) Is sysrq-T my best bet here, or should I be trying something else first?

#3 lore raw
Quoted 4 lines · On Thu, 2010-02-18 at 11:42 -0500, [email protected] wrote:

1) should it get killed for RLIMIT_RTTIME because it's been in a futex() for multiple seconds? It seems suspicious - docs say a blocking syscall resets RTTIME - so if futex() blocks it shouldn't kill, and if it's in the kernel for a second without blocking it's a bug too.

sys_futex() shouldn't be taking that long indeed, and if its not caused by the sysrq interrupt then something funny is going on indeed.

You could poke around in kernel/futex.c:futex_wait() to see where it thinks its spending it time? There's a single retry loop in there, maybe instrument that to see if its retrying too many times..

#4 lore raw
In reply to [email protected]
Quoted 25 lines · On Thu, 18 Feb 2010, [email protected] wrote:

(Adding some cc: to the list)

Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · On Thu, 2010-02-18 at 09:04 -0500, [email protected] wrote:

Kernel: x86_64 2.6.33-rc7-mmotm0210

I'm debugging a problem where pulseaudio is getting killed with a SIGKILL out of the blue. It appears to be a problem where pulseaudio sets RLIMIT_RTTIME and the bound gets exceeded. Analysis with 'top' shows a short spike of 96% system time, and the tail end of strace shows this:

[pid 25065] 01:50:20.371484 ioctl(28, USBDEVFS_CONTROL, 0x7fd3d76f630c) = 0 <0.000015> [pid 25065] 01:50:20.371548 ioctl(28, 0x40045532, 0x7fd3d76f636c) = 0 <0.000016> [pid 25065] 01:50:20.371611 open("/dev/snd/pcmC0D0p", O_RDWR|O_NONBLOCK|O_CLOEXEC <unfinished ...> [pid 25064] 01:50:20.371678 <... write resumed> ) = 8 <0.002104>
[pid 25064] 01:50:20.371718 futex(0xc2ec00, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...> [pid 25066] 01:50:21.408392 +++ killed by SIGKILL +++
PANIC: handle_group_exit: 25066 leader 25064
[pid 25065] 01:50:21.408442 +++ killed by SIGKILL +++
PANIC: handle_group_exit: 25065 leader 25064
01:50:21.420354 +++ killed by SIGKILL +++

thread 25064 apparently gets gunned down due to RTTIME because it spent a whole second in a futex() call - is it reasonable for futex() to not return for that long?

Well, it's in futex_wait(). If nothing unlocks the futex, then it stays there forever.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

In other words - kernel bug because futex() should return, or pulseaudio bug for not understanding futex() can snooze a while?

If a kernel bug, anybody got a better idea than nuking the RLIMIT_RTTIME call, waiting for it to repeat (takes between 1 minute and 1 hour or so), and whomping it a few times with sysrq-T?

Quoted 8 lines
Quoted 1 line

is that second spend in processing sysrq-t?

No, currently that second is spent in a futex() syscall - I'm wondering:

1) should it get killed for RLIMIT_RTTIME because it's been in a futex() for multiple seconds? It seems suspicious - docs say a blocking syscall resets RTTIME - so if futex() blocks it shouldn't kill, and if it's in the kernel for a second without blocking it's a bug too.

If it schedules out, then the RLIMIT_RTTIME should not be hit.

There are several possibilities why this happens:

- the futex code has a bug which causes it to busy loop
- the rlimit code is wreckaged

Quoted 1 line

2) Is sysrq-T my best bet here, or should I be trying something else first?

Can you enable the function tracer and check whether it reproduces with the function tracer. If yes, then we can put a tracing_off() into the code which handles the rlimit, so we can see in the trace what happened before it triggered.

Thanks,

tglx

#5 lore raw
In reply to [email protected]
Quoted 7 lines · [email protected] wrote:

Kernel: x86_64 2.6.33-rc7-mmotm0210

I'm debugging a problem where pulseaudio is getting killed with a SIGKILL out of the blue. It appears to be a problem where pulseaudio sets RLIMIT_RTTIME and the bound gets exceeded. Analysis with 'top' shows a short spike of 96% system time, and the tail end of strace shows this:

Quoted 5 lines

[pid 25065] 01:50:20.371484 ioctl(28, USBDEVFS_CONTROL, 0x7fd3d76f630c) = 0 <0.000015> [pid 25065] 01:50:20.371548 ioctl(28, 0x40045532, 0x7fd3d76f636c) = 0 <0.000016> [pid 25065] 01:50:20.371611 open("/dev/snd/pcmC0D0p", O_RDWR|O_NONBLOCK|O_CLOEXEC <unfinished ...> [pid 25064] 01:50:20.371678 <... write resumed> ) = 8 <0.002104>
[pid 25064] 01:50:20.371718 futex(0xc2ec00, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>

So it received the signal and the task waiting in sys_futex(FUTEX_WAIT_PRIVATE...) wakes up to handle the signal and return right? What do you see here that suggests the task is spinning inside the futex syscall?

This is a non-rt kernel right? So spinning could occur on the hash bucket lock if something else is really horked. That seems unlikely.

Quoted 9 lines

[pid 25066] 01:50:21.408392 +++ killed by SIGKILL +++
PANIC: handle_group_exit: 25066 leader 25064 [pid 25065] 01:50:21.408442 +++ killed by SIGKILL +++
PANIC: handle_group_exit: 25065 leader 25064 01:50:21.420354 +++ killed by SIGKILL +++

thread 25064 apparently gets gunned down due to RTTIME because it spent a whole second in a futex() call - is it reasonable for futex() to not return for that long?

Certainly if a corresponding FUTEX_WAKE op was not sent.

--
Darrem

Quoted 7 lines

In other words - kernel bug because futex() should return, or pulseaudio bug for not understanding futex() can snooze a while?

If a kernel bug, anybody got a better idea than nuking the RLIMIT_RTTIME call, waiting for it to repeat (takes between 1 minute and 1 hour or so), and whomping it a few times with sysrq-T?

Signature · 3 lines
Darren Hart
IBM Linux Technology Center
Real-Time Linux Team