asterisk hangs with RT priority
I'm seeing hanging asterisk processes on startup when using RT priority with current -git. Unfortunately I'm not sure about the last working version, but it was one of the final 2.6.25 rcs. The hang stops when executing "schedtool -N $(pidof asterisk)".
The last thing seen in strace (tried multiple times, output is identical each time) is:
...connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(3) = 0 socket(PF_FILE, SOCK_STREAM, 0) = 3 fcntl64(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(3) = 0open("/etc/group", O_RDONLY|0x80000 /* O_??? */) = 3_llseek(3, 0, [0], SEEK_CUR) = 0fstat64(3, {st_mode=S_IFREG|0644, st_size=504, ...}) = 0mmap2(NULL, 504, PROT_READ, MAP_SHARED, 3, 0) = 0xb7f2e000_llseek(3, 504, [504], SEEK_SET) = 0fstat64(3, {st_mode=S_IFREG|0644, st_size=504, ...}) = 0
munmap(0xb7f2e000, 504) = 0 close(3) = 0 setgroups32(3, [103, 20, 29]) = 0 setuid32(102) = 0
when executing the schedtool command above at this point, it continues with
capset(0x19980330, 0, {CAP_NET_ADMIN, CAP_NET_ADMIN, 0}) = 0geteuid32() = 102stat64("/etc/asterisk/extconfig.conf", {st_mode=S_IFREG|0640, st_size=1506, ...}) = 0 open("/etc/asterisk/extconfig.conf", O_RDONLY|O_LARGEFILE) = 3
...
sysrq-t shows:
...
[ 32.032337] =======================
[ 32.032337] S21asterisk S 00000000 0 1220 1148
[ 32.032337] c7d7cf00 00000282 c0103158 00000000 c10e1800 c7cbbc00 c7cbbd6c ffffffea
[ 32.032337] 00000004 c7cbbbf8 c7d7cf78 c012097f c7cbaa80 070c0065 070c0065 c7d7cf3c
[ 32.032337] 00000000 c7cbbd00 0000000c 00000000 00000003 c7cbbc00 00000001 c7cbbc00
[ 32.032337] Call Trace:
[ 32.032337] [<c0103158>] ? do_notify_resume+0x51/0x78e
[ 32.032337] [<c012097f>] do_wait+0x5b1/0xb8c
[ 32.032337] [<c0119ce2>] ? default_wake_function+0x0/0xd
[ 32.032337] [<c0120fbf>] sys_wait4+0x65/0xa2
[ 32.032337] [<c0121023>] sys_waitpid+0x27/0x29
[ 32.032337] [<c0103c4a>] syscall_call+0x7/0xb
[ 32.032337] [<c0320000>] ? serial_pci_guess_board+0x170/0x1ce
[ 32.032337] =======================
[ 32.032337] asterisk R running 0 1229 1220
[ 32.032337] c7d7afb0 00000286 00000000 00001000 00000000 c7cbaa80 c7cbabf0 b7fa9ff4
[ 32.032337] b7faa668 b7f68b40 c7d7a000 c0103cee b7fa9ff4 00000007 0000035f b7faa668
[ 32.032337] b7f68b40 bfca8b04 0804abc8 0000007b 0000007b c0100000 ffffffff b7f97ff6
[ 32.032337] Call Trace:
[ 32.032337] [<c0103cee>] work_resched+0x5/0x28
[ 32.032337] =======================
...
[ 32.032337] Sched Debug Version: v0.07, 2.6.26-rc1 #28
[ 32.032337] now at 31553.299102 msecs
[ 32.032337] .sysctl_sched_latency : 20.000000
[ 32.032337] .sysctl_sched_min_granularity : 4.000000
[ 32.032337] .sysctl_sched_wakeup_granularity : 10.000000
[ 32.032337] .sysctl_sched_child_runs_first : 0.000001
[ 32.032337] .sysctl_sched_features : 895
[ 32.032337]
[ 32.032337] cpu#0, 2667.480 MHz
[ 32.032337] .nr_running : 3
[ 32.032337] .load : 179570
[ 32.032337] .nr_switches : 7594
[ 32.032337] .nr_load_updates : 6015
[ 32.032337] .nr_uninterruptible : 0
[ 32.032337] .jiffies : 4294698635
[ 32.032337] .next_balance : 0.000000
[ 32.032337] .curr->pid : 1235
[ 32.032337] .clock : 31339.548336
[ 32.032337] .cpu_load[0] : 179570
[ 32.032337] .cpu_load[1] : 178546
[ 32.032337] .cpu_load[2] : 178043
[ 32.032337] .cpu_load[3] : 177875
[ 32.032337] .cpu_load[4] : 178008
[ 32.032337]
[ 32.032337] cfs_rq[0]:
[ 32.032337] .exec_clock : 0.000000
[ 32.032337] .MIN_vruntime : 0.000001
[ 32.032337] .min_vruntime : 6015.548989
[ 32.032337] .max_vruntime : 0.000001
[ 32.032337] .spread : 0.000000
[ 32.032337] .spread0 : 0.000000
[ 32.032337] .nr_running : 0
[ 32.032337] .load : 0
[ 32.032337] .nr_spread_over : 0
[ 32.032337]
[ 32.032337] cfs_rq[0]:
[ 32.032337] .exec_clock : 0.000000
[ 32.032337] .MIN_vruntime : 0.000001
[ 32.032337] .min_vruntime : 6015.548989
[ 32.032337] .max_vruntime : 0.000001
[ 32.032337] .spread : 0.000000
[ 32.032337] .spread0 : 0.000000
[ 32.032337] .nr_running : 0
[ 32.032337] .load : 0
[ 32.032337] .nr_spread_over : 0
[ 32.032337]
[ 32.032337] cfs_rq[0]:
[ 32.032337] .exec_clock : 0.000000
[ 32.032337] .MIN_vruntime : 0.000001
[ 32.032337] .min_vruntime : 6015.548989
[ 32.032337] .max_vruntime : 0.000001
[ 32.032337] .spread : 0.000000
[ 32.032337] .spread0 : 0.000000
[ 32.032337] .nr_running : 0
[ 32.032337] .load : 0
[ 32.032337] .nr_spread_over : 0
[ 32.032337]
[ 32.032337] cfs_rq[0]:
[ 32.032337] .exec_clock : 0.000000
[ 32.032337] .MIN_vruntime : 0.000001
[ 32.032337] .min_vruntime : 6015.548989
[ 32.032337] .max_vruntime : 0.000001
[ 32.032337] .spread : 0.000000
[ 32.032337] .spread0 : 0.000000
[ 32.032337] .nr_running : 0
[ 32.032337] .load : 0
[ 32.032337] .nr_spread_over : 0
[ 32.032337]
[ 32.032337] cfs_rq[0]:
[ 32.032337] .exec_clock : 0.000000
[ 32.032337] .MIN_vruntime : 26351.612585
[ 32.032337] .min_vruntime : 6015.548989
[ 32.032337] .max_vruntime : 26351.612585
[ 32.032337] .spread : 0.000000
[ 32.032337] .spread0 : 0.000000
[ 32.032337] .nr_running : 2
[ 32.032337] .load : 2048
[ 32.032337] .nr_spread_over : 0
[ 32.032337]
[ 32.032337] runnable tasks:
[ 32.032337] task PID tree-key switches prio
exec-runtime
sum-exec sum-sleep
[ 32.032337] --------------------------------------------------------------------------------
--------------------------
[ 32.032337] klogd 1168 26351.612585 14 120
0
0 0.000000 0.000000
0.000000
[ 32.032337] asterisk 1229 25780.061231 22 89
0
0 0.000000 0.000000
0.000000
[ 32.032337] R bash 1235 26362.344107 89 120
0
0 0.000000 0.000000
0.000000
[ 32.032337]The above is from inside lguest, but the problem happens on a real system as well. If more information is needed, please ask.
Quoted 4 lines · Patrick McHardy
I'm seeing hanging asterisk processes on startup when using RT priority with current -git. Unfortunately I'm not sure about the last working version, but it was one of the final 2.6.25 rcs. The hang stops when executing "schedtool -N $(pidof asterisk)".
The problem is gone in the current -git tree.
Quoted 4 lines · Patrick McHardy
I'm seeing hanging asterisk processes on startup when using RT priority with current -git. Unfortunately I'm not sure about the last working version, but it was one of the final 2.6.25 rcs. The hang stops when executing "schedtool -N $(pidof asterisk)".
I'm now hitting exactly the same problem with 2.6.28-rc9 (I had it with 2.6.27 too but thought it might go away on its own :)
I've made a small program to attempt to reproduce this based on the asterisk code. Unfortunately it doesn't work on its own. However, if you run it under strace then it does hang in the same away as asterisk (which seems to happen after the setuid call, as strace shows it before hanging and the printk after it in asterisk gets executed).
Note that the hang goes away if either setscheduler or setuid is removed.
#include <sched.h>
int main(void)
{
struct sched_param sched = {.sched_priority = 10};
sched_setscheduler(0, SCHED_RR, &sched);
setuid(106);
return 0;
}Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 16 lines · Herbert Xu
Quoted 4 lines · Patrick McHardy
I'm seeing hanging asterisk processes on startup when using RT priority with current -git. Unfortunately I'm not sure about the last working version, but it was one of the final 2.6.25 rcs. The hang stops when executing "schedtool -N $(pidof asterisk)".
I'm now hitting exactly the same problem with 2.6.28-rc9 (I had it with 2.6.27 too but thought it might go away on its own :)
I've made a small program to attempt to reproduce this based on the asterisk code. Unfortunately it doesn't work on its own. However, if you run it under strace then it does hang in the same away as asterisk (which seems to happen after the setuid call, as strace shows it before hanging and the printk after it in asterisk gets executed).
Quoted 3 lines
Note that the hang goes away if either setscheduler or setuid is removed.
Hmm. Do you have CONFIG_FAIR_USER_SCHED set on?
Thanks,
Signature · 2 lines
regards, Dhaval
Quoted 2 lines · Dhaval Giani
Hmm. Do you have
CONFIG_FAIR_USER_SCHEDset on?
I can't find such a config option, but
$ grep SCHED .config CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y CONFIG_GROUP_SCHED=y CONFIG_FAIR_GROUP_SCHED=y CONFIG_RT_GROUP_SCHED=y CONFIG_USER_SCHED=y # CONFIG_CGROUP_SCHED is not set CONFIG_IOSCHED_NOOP=y CONFIG_IOSCHED_AS=m CONFIG_IOSCHED_DEADLINE=m CONFIG_IOSCHED_CFQ=m CONFIG_DEFAULT_IOSCHED="noop" CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y CONFIG_SCHED_HRTICK=y CONFIG_NET_SCHED=y # CONFIG_USB_EHCI_TT_NEWSCHED is not set $
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 14 lines · Herbert Xu
Quoted 2 lines · Dhaval Giani
Hmm. Do you have
CONFIG_FAIR_USER_SCHEDset on?I can't find such a config option, but
$ grep SCHED .config CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y CONFIG_GROUP_SCHED=y CONFIG_FAIR_GROUP_SCHED=y CONFIG_RT_GROUP_SCHED=y CONFIG_USER_SCHED=y # CONFIG_CGROUP_SCHED is not set
Quoted 2 lines
CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=yCONFIG_SCHED_HRTICK=y
So you have uid-group scheduling and RT-group scheduling enabled (a feature that's experimental for real and has never been enabled by default), looking at the sys_setuid() code, the real uid change is done by switch_uid() and that doesn't have a failable scheduler hook.
The thing is, I suspect the uid you switch to doesn't have a RT runtime quota configured, therefore the RT task that gets placed in it by switch_uid() doesn't get to run.
[ Please read Documentation/scheduler/sched-rt-group.txt
when you enable RT group scheduling ]
The correct thing would be for switch_uid() (or set_user) to fail with -EINVAL, much like cpu_cgroup_can_attach() currently does for cgroup grouping.
After that it demonstrates a bug in your test program, which fails to check errors ;-)
Quoted 5 lines · Peter Zijlstra
So you have uid-group scheduling and RT-group scheduling enabled (a feature that's experimental for real and has never been enabled by default), looking at the
sys_setuid()code, the real uid change is done byswitch_uid()and that doesn't have a failable scheduler hook.
An experimental marking is no excuse for being broken.
Quoted 6 lines
The thing is, I suspect the uid you switch to doesn't have a RT runtime quota configured, therefore the RT task that gets placed in it by
switch_uid()doesn't get to run.[ Please read
Documentation/scheduler/sched-rt-group.txt
when you enable RT group scheduling ]
This seems broken to me. The only way for a process to get into RR mode is if it had been set by someone with the right privileges or if it was inherited from its parent.
So having a default where such a process stops executing altogether after performing a setuid is wrong.
The default should be to give each user a non-zero allotment.
Quoted 6 lines
The correct thing would be for
switch_uid()(orset_user) to fail with-EINVAL, much likecpu_cgroup_can_attach()currently does for cgroup grouping.After that it demonstrates a bug in your test program, which fails to check errors ;-)
Well the program which this was based on, asterisk does check for errors on setuid. However, even if setuid did return an error this isn't much better than the status quo since the user will be left with the question "why on earth is asterisk failing on setuid?".
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 8 lines · Herbert Xu
Quoted 5 lines · Peter Zijlstra
So you have uid-group scheduling and RT-group scheduling enabled (a feature that's experimental for real and has never been enabled by default), looking at the
sys_setuid()code, the real uid change is done byswitch_uid()and that doesn't have a failable scheduler hook.An experimental marking is no excuse for being broken.
True, and we strive to fix them -- so thanks for finding this one.
Quoted 7 lines
Quoted 6 lines
The thing is, I suspect the uid you switch to doesn't have a RT runtime quota configured, therefore the RT task that gets placed in it by
switch_uid()doesn't get to run.[ Please read
Documentation/scheduler/sched-rt-group.txt
when you enable RT group scheduling ]
Quoted 6 lines
This seems broken to me. The only way for a process to get into RR mode is if it had been set by someone with the right privileges or if it was inherited from its parent.
So having a default where such a process stops executing altogether after performing a setuid is wrong.
True, therefore the setuid must fail.
Quoted 1 line
The default should be to give each user a non-zero allotment.
That's sadly impossible. The thing is, you cannot over-commit this time -- nor change an active configuration. So the only possibility left is a default of 0.
Quoted 11 lines
Quoted 6 lines
The correct thing would be for
switch_uid()(orset_user) to fail with-EINVAL, much likecpu_cgroup_can_attach()currently does for cgroup grouping.After that it demonstrates a bug in your test program, which fails to check errors ;-)
Well the program which this was based on, asterisk does check for errors on setuid. However, even if setuid did return an error this isn't much better than the status quo since the user will be left with the question "why on earth is asterisk failing on setuid?".
Maybe a more descriptive error like -ENOTIME ?