Skip to contentSkip to conversation

asterisk hangs with RT priority

DISCUSSION Patrick McHardy 4 participants 8 messages May 18, 2008 – Dec 25, 2008

lore
Share & tools
lore
#0 lore raw

I'm seeing hanging asterisk processes on startup when using RT priority with current -git. Unfortunately I'm not sure about the last working version, but it was one of the final 2.6.25 rcs. The hang stops when executing "schedtool -N $(pidof asterisk)".

The last thing seen in strace (tried multiple times, output is identical each time) is:

...
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)

close(3)                                = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 3
fcntl64(3, F_SETFL, O_RDWR|O_NONBLOCK)  = 0

connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(3) = 0
open("/etc/group", O_RDONLY|0x80000 /* O_??? */) = 3
_llseek(3, 0, [0], SEEK_CUR) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=504, ...}) = 0
mmap2(NULL, 504, PROT_READ, MAP_SHARED, 3, 0) = 0xb7f2e000
_llseek(3, 504, [504], SEEK_SET) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=504, ...}) = 0

munmap(0xb7f2e000, 504)                 = 0
close(3)                                = 0
setgroups32(3, [103, 20, 29])           = 0
setuid32(102)                           = 0

when executing the schedtool command above at this point, it continues with

capset(0x19980330, 0, {CAP_NET_ADMIN, CAP_NET_ADMIN, 0}) = 0
geteuid32() = 102
stat64("/etc/asterisk/extconfig.conf", {st_mode=S_IFREG|0640, st_size=1506, ...}) = 0 open("/etc/asterisk/extconfig.conf", O_RDONLY|O_LARGEFILE) = 3
...

sysrq-t shows:
...

[   32.032337]  =======================
[   32.032337] S21asterisk   S 00000000     0  1220   1148
[   32.032337]        c7d7cf00 00000282 c0103158 00000000 c10e1800 c7cbbc00 c7cbbd6c ffffffea
[   32.032337]        00000004 c7cbbbf8 c7d7cf78 c012097f c7cbaa80 070c0065 070c0065 c7d7cf3c
[   32.032337]        00000000 c7cbbd00 0000000c 00000000 00000003 c7cbbc00 00000001 c7cbbc00
[   32.032337] Call Trace:
[   32.032337]  [<c0103158>] ? do_notify_resume+0x51/0x78e
[   32.032337]  [<c012097f>] do_wait+0x5b1/0xb8c
[   32.032337]  [<c0119ce2>] ? default_wake_function+0x0/0xd
[   32.032337]  [<c0120fbf>] sys_wait4+0x65/0xa2
[   32.032337]  [<c0121023>] sys_waitpid+0x27/0x29
[   32.032337]  [<c0103c4a>] syscall_call+0x7/0xb
[   32.032337]  [<c0320000>] ? serial_pci_guess_board+0x170/0x1ce
[   32.032337]  =======================
[   32.032337] asterisk      R running      0  1229   1220
[   32.032337]        c7d7afb0 00000286 00000000 00001000 00000000 c7cbaa80 c7cbabf0 b7fa9ff4
[   32.032337]        b7faa668 b7f68b40 c7d7a000 c0103cee b7fa9ff4 00000007 0000035f b7faa668
[   32.032337]        b7f68b40 bfca8b04 0804abc8 0000007b 0000007b c0100000 ffffffff b7f97ff6
[   32.032337] Call Trace:
[   32.032337]  [<c0103cee>] work_resched+0x5/0x28
[   32.032337]  =======================
...
[   32.032337] Sched Debug Version: v0.07, 2.6.26-rc1 #28
[   32.032337] now at 31553.299102 msecs
[   32.032337]   .sysctl_sched_latency                    : 20.000000
[   32.032337]   .sysctl_sched_min_granularity            : 4.000000
[   32.032337]   .sysctl_sched_wakeup_granularity         : 10.000000
[   32.032337]   .sysctl_sched_child_runs_first           : 0.000001
[   32.032337]   .sysctl_sched_features                   : 895
[   32.032337]
[   32.032337] cpu#0, 2667.480 MHz
[   32.032337]   .nr_running                    : 3
[   32.032337]   .load                          : 179570
[   32.032337]   .nr_switches                   : 7594
[   32.032337]   .nr_load_updates               : 6015
[   32.032337]   .nr_uninterruptible            : 0
[   32.032337]   .jiffies                       : 4294698635
[   32.032337]   .next_balance                  : 0.000000
[   32.032337]   .curr->pid                     : 1235
[   32.032337]   .clock                         : 31339.548336
[   32.032337]   .cpu_load[0]                   : 179570
[   32.032337]   .cpu_load[1]                   : 178546
[   32.032337]   .cpu_load[2]                   : 178043
[   32.032337]   .cpu_load[3]                   : 177875
[   32.032337]   .cpu_load[4]                   : 178008
[   32.032337]
[   32.032337] cfs_rq[0]:
[   32.032337]   .exec_clock                    : 0.000000
[   32.032337]   .MIN_vruntime                  : 0.000001
[   32.032337]   .min_vruntime                  : 6015.548989
[   32.032337]   .max_vruntime                  : 0.000001
[   32.032337]   .spread                        : 0.000000
[   32.032337]   .spread0                       : 0.000000
[   32.032337]   .nr_running                    : 0
[   32.032337]   .load                          : 0
[   32.032337]   .nr_spread_over                : 0
[   32.032337]
[   32.032337] cfs_rq[0]:
[   32.032337]   .exec_clock                    : 0.000000
[   32.032337]   .MIN_vruntime                  : 0.000001
[   32.032337]   .min_vruntime                  : 6015.548989
[   32.032337]   .max_vruntime                  : 0.000001
[   32.032337]   .spread                        : 0.000000
[   32.032337]   .spread0                       : 0.000000
[   32.032337]   .nr_running                    : 0
[   32.032337]   .load                          : 0
[   32.032337]   .nr_spread_over                : 0
[   32.032337]
[   32.032337] cfs_rq[0]:
[   32.032337]   .exec_clock                    : 0.000000
[   32.032337]   .MIN_vruntime                  : 0.000001
[   32.032337]   .min_vruntime                  : 6015.548989
[   32.032337]   .max_vruntime                  : 0.000001
[   32.032337]   .spread                        : 0.000000
[   32.032337]   .spread0                       : 0.000000
[   32.032337]   .nr_running                    : 0
[   32.032337]   .load                          : 0
[   32.032337]   .nr_spread_over                : 0
[   32.032337]
[   32.032337] cfs_rq[0]:
[   32.032337]   .exec_clock                    : 0.000000
[   32.032337]   .MIN_vruntime                  : 0.000001
[   32.032337]   .min_vruntime                  : 6015.548989
[   32.032337]   .max_vruntime                  : 0.000001
[   32.032337]   .spread                        : 0.000000
[   32.032337]   .spread0                       : 0.000000
[   32.032337]   .nr_running                    : 0
[   32.032337]   .load                          : 0
[   32.032337]   .nr_spread_over                : 0
[   32.032337]
[   32.032337] cfs_rq[0]:
[   32.032337]   .exec_clock                    : 0.000000
[   32.032337]   .MIN_vruntime                  : 26351.612585
[   32.032337]   .min_vruntime                  : 6015.548989
[   32.032337]   .max_vruntime                  : 26351.612585
[   32.032337]   .spread                        : 0.000000
[   32.032337]   .spread0                       : 0.000000
[   32.032337]   .nr_running                    : 2
[   32.032337]   .load                          : 2048
[   32.032337]   .nr_spread_over                : 0
[   32.032337]
[   32.032337] runnable tasks:
[   32.032337]             task   PID         tree-key  switches  prio 
   exec-runtime
 sum-exec        sum-sleep
[   32.032337] --------------------------------------------------------------------------------
--------------------------
[   32.032337]            klogd  1168     26351.612585        14   120 
             0
      0               0.000000               0.000000 
0.000000
[   32.032337]         asterisk  1229     25780.061231        22    89 
             0
      0               0.000000               0.000000 
0.000000
[   32.032337] R           bash  1235     26362.344107        89   120 
             0
      0               0.000000               0.000000 
0.000000
[   32.032337]

The above is from inside lguest, but the problem happens on a real system as well. If more information is needed, please ask.

#1 lore raw
Quoted 4 lines · Patrick McHardy

I'm seeing hanging asterisk processes on startup when using RT priority with current -git. Unfortunately I'm not sure about the last working version, but it was one of the final 2.6.25 rcs. The hang stops when executing "schedtool -N $(pidof asterisk)".

The problem is gone in the current -git tree.

#2 lore raw
In reply to Patrick McHardy
Quoted 4 lines · Patrick McHardy

I'm seeing hanging asterisk processes on startup when using RT priority with current -git. Unfortunately I'm not sure about the last working version, but it was one of the final 2.6.25 rcs. The hang stops when executing "schedtool -N $(pidof asterisk)".

I'm now hitting exactly the same problem with 2.6.28-rc9 (I had it with 2.6.27 too but thought it might go away on its own :)

I've made a small program to attempt to reproduce this based on the asterisk code. Unfortunately it doesn't work on its own. However, if you run it under strace then it does hang in the same away as asterisk (which seems to happen after the setuid call, as strace shows it before hanging and the printk after it in asterisk gets executed).

Note that the hang goes away if either setscheduler or setuid is removed.

#include <sched.h>

int main(void)

{
	struct sched_param sched = {.sched_priority = 10};

	sched_setscheduler(0, SCHED_RR, &sched);
	setuid(106);
	return 0;
}

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#3 lore raw
Quoted 16 lines · Herbert Xu
Quoted 4 lines · Patrick McHardy

I'm seeing hanging asterisk processes on startup when using RT priority with current -git. Unfortunately I'm not sure about the last working version, but it was one of the final 2.6.25 rcs. The hang stops when executing "schedtool -N $(pidof asterisk)".

I'm now hitting exactly the same problem with 2.6.28-rc9 (I had it with 2.6.27 too but thought it might go away on its own :)

I've made a small program to attempt to reproduce this based on the asterisk code. Unfortunately it doesn't work on its own. However, if you run it under strace then it does hang in the same away as asterisk (which seems to happen after the setuid call, as strace shows it before hanging and the printk after it in asterisk gets executed).

Quoted 3 lines

Note that the hang goes away if either setscheduler or setuid is removed.

Hmm. Do you have CONFIG_FAIR_USER_SCHED set on?

Thanks,

Signature · 2 lines
regards,
Dhaval
#4 lore raw
Quoted 2 lines · Dhaval Giani

Hmm. Do you have CONFIG_FAIR_USER_SCHED set on?

I can't find such a config option, but

$ grep SCHED .config
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_GROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_USER_SCHED=y
# CONFIG_CGROUP_SCHED is not set
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=m
CONFIG_IOSCHED_DEADLINE=m
CONFIG_IOSCHED_CFQ=m
CONFIG_DEFAULT_IOSCHED="noop"
CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y
CONFIG_SCHED_HRTICK=y
CONFIG_NET_SCHED=y
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
$ 

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#5 lore raw
Quoted 14 lines · Herbert Xu
Quoted 2 lines · Dhaval Giani

Hmm. Do you have CONFIG_FAIR_USER_SCHED set on?

I can't find such a config option, but

$ grep SCHED .config
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_GROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_USER_SCHED=y
# CONFIG_CGROUP_SCHED is not set
Quoted 2 lines

CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y
CONFIG_SCHED_HRTICK=y

So you have uid-group scheduling and RT-group scheduling enabled (a feature that's experimental for real and has never been enabled by default), looking at the sys_setuid() code, the real uid change is done by switch_uid() and that doesn't have a failable scheduler hook.

The thing is, I suspect the uid you switch to doesn't have a RT runtime quota configured, therefore the RT task that gets placed in it by switch_uid() doesn't get to run.

[ Please read Documentation/scheduler/sched-rt-group.txt
when you enable RT group scheduling ]

The correct thing would be for switch_uid() (or set_user) to fail with -EINVAL, much like cpu_cgroup_can_attach() currently does for cgroup grouping.

After that it demonstrates a bug in your test program, which fails to check errors ;-)

#6 lore raw
Quoted 5 lines · Peter Zijlstra

So you have uid-group scheduling and RT-group scheduling enabled (a feature that's experimental for real and has never been enabled by default), looking at the sys_setuid() code, the real uid change is done by switch_uid() and that doesn't have a failable scheduler hook.

An experimental marking is no excuse for being broken.

Quoted 6 lines

The thing is, I suspect the uid you switch to doesn't have a RT runtime quota configured, therefore the RT task that gets placed in it by switch_uid() doesn't get to run.

[ Please read Documentation/scheduler/sched-rt-group.txt
when you enable RT group scheduling ]

This seems broken to me. The only way for a process to get into RR mode is if it had been set by someone with the right privileges or if it was inherited from its parent.

So having a default where such a process stops executing altogether after performing a setuid is wrong.

The default should be to give each user a non-zero allotment.

Quoted 6 lines

The correct thing would be for switch_uid() (or set_user) to fail with -EINVAL, much like cpu_cgroup_can_attach() currently does for cgroup grouping.

After that it demonstrates a bug in your test program, which fails to check errors ;-)

Well the program which this was based on, asterisk does check for errors on setuid. However, even if setuid did return an error this isn't much better than the status quo since the user will be left with the question "why on earth is asterisk failing on setuid?".

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#7 lore raw
Quoted 8 lines · Herbert Xu
Quoted 5 lines · Peter Zijlstra

So you have uid-group scheduling and RT-group scheduling enabled (a feature that's experimental for real and has never been enabled by default), looking at the sys_setuid() code, the real uid change is done by switch_uid() and that doesn't have a failable scheduler hook.

An experimental marking is no excuse for being broken.

True, and we strive to fix them -- so thanks for finding this one.

Quoted 7 lines
Quoted 6 lines

The thing is, I suspect the uid you switch to doesn't have a RT runtime quota configured, therefore the RT task that gets placed in it by switch_uid() doesn't get to run.

[ Please read Documentation/scheduler/sched-rt-group.txt
when you enable RT group scheduling ]

Quoted 6 lines

This seems broken to me. The only way for a process to get into RR mode is if it had been set by someone with the right privileges or if it was inherited from its parent.

So having a default where such a process stops executing altogether after performing a setuid is wrong.

True, therefore the setuid must fail.

Quoted 1 line

The default should be to give each user a non-zero allotment.

That's sadly impossible. The thing is, you cannot over-commit this time -- nor change an active configuration. So the only possibility left is a default of 0.

Quoted 11 lines
Quoted 6 lines

The correct thing would be for switch_uid() (or set_user) to fail with -EINVAL, much like cpu_cgroup_can_attach() currently does for cgroup grouping.

After that it demonstrates a bug in your test program, which fails to check errors ;-)

Well the program which this was based on, asterisk does check for errors on setuid. However, even if setuid did return an error this isn't much better than the status quo since the user will be left with the question "why on earth is asterisk failing on setuid?".

Maybe a more descriptive error like -ENOTIME ?