2.6.15-rt18, alsa sequencer, rosegarden -> alsa hangs
Hi all, I don't know where to go from here...
I'm trying to get a 2.6.15-rt18 kernel going and I'm finding a big showstopper in the alsa sequencer, presumably at the kernel module level. This is -rt18 with PREEMPT_RT and high resolution timers enabled, plus either alsa kernel drivers from 1.0.10 (latest stable release) or 1.0.11rc3. Same problem for both. I initially thought that the snd-rtctimer kernel module was a problem so the latest tests were done with that module disabled so that the alsa sequencer is using the system timer for timing (HZ == 1000 for this kernel).
The symptoms are as follows:
- start jack using qjackctl
- start qsynth (gui front end for fluidsynth, a synth)
- start rosegarden (midi sequencer and audio recorder)
- load a midi file into rosegarden
- midi file plays successfully
- close rosegarden
at this point one of the threads of rosegarden fails to exit and stays forever in the process list, in a ps axuw it shows as:
nando 5484 0.0 0.0 0 0 pts/1 D 13:32 0:00 [rosegardenseque]
Anything else that I try to stop that touches the alsa sequencer never dies (qjackctl, vkeybd, qsynth, etc). Anything I try to start that tries to use it does not start. This happened with two widely different versions of rosegarden (1.0 and 1.2.3) - I have not yet tried with other alsa sequencer clients - I tried different kernel builds and alsa builds before giving up, the cycle is long when I include the frequent reboots. I can browse around in /proc/asound/seq but a:
cat /proc/asound/clients
hangs forever (unkillable). Obviously I can't unload the alsa subsystem to try again. At this point my only recourse to recover is a reboot (which hangs when trying to stop the alsa subsystem - so the reset key is the only option).
There are no kernel oops or messages either in dmesg or /var/log/messages that I can see.
Booting into 2.6.14 with the latest -rt patch for that branch does not show the problem (using alsa kernel modules from 1.0.10 stable).
The problem happens with at least three completely different soundcards and two different kernel builds (one kind of vanilla, the other derived from the latest fc4 kernel configuration and patches).
So it would look like an interaction between the newest -rt patches and the alsa sequencer timing. Weird because while rosegarden is up and running it plays midi sequences with no apparent problems, it is only when it exits that things get completely messed up.
Suggestions appreciated...
-- Fernando
# cat /proc/asound/seq/timer Timer for queue 0 : system timer Period time : 0.001000000 Skew : 65536 / 65536
[rosegarden is still there.....]
# cat /proc/asound/seq/queues queue 0: [Rosegarden queue] owned by client : 130 lock status : Locked queued time events : 0 queued tick events : 0 timer state : Running timer PPQ : 96 current tempo : 500000 current BPM : 120 current time : 1617.680000000 s current tick : 318130
# cat /proc/asound/seq/drivers snd-seq-oss,loaded,0 snd-seq-midi,loaded,1
Quoted 10 lines · Fernando Lopez-Lezcano
The symptoms are as follows:
- start jack using qjackctl
- start qsynth (gui front end for fluidsynth, a synth)
- start rosegarden (midi sequencer and audio recorder)
- load a midi file into rosegarden
- midi file plays successfully
- close rosegarden
at this point one of the threads of rosegarden fails to exit and stays forever in the process list, in a ps axuw it shows as:
Quoted 5 lines
nando 5484 0.0 0.0 0 0 pts/1 D 13:32 0:00 [rosegardenseque]
Anything else that I try to stop that touches the alsa sequencer never dies (qjackctl, vkeybd, qsynth, etc). Anything I try to start that tries to use it does not start. This happened with two widely different
could you get a tasklist-dump? It's either SysRq-T, or:
echo t > /proc/sysrq-trigger
that should dump all tasks and their backtraces - including the hung rosegardensequencer task.
Ingo
Quoted 27 lines · --- Ingo Molnar
Quoted 3 lines · Fernando Lopez-Lezcano
The symptoms are as follows:
- start jack using qjackctl
- start qsynth (gui front end for fluidsynth, asynth)
Quoted 1 line
- start rosegarden (midi sequencer and audio
recorder)
Quoted 4 lines
- load a midi file into rosegarden
- midi file plays successfully
- close rosegarden
at this point one of the threads of rosegarden fails toexit and stays
Quoted 3 lines
forever in the process list, in a ps axuw it shows as:
nando 5484 0.0 0.0 0 0 pts/1 D 13:32 0:00
[rosegardenseque]
Quoted 2 lines
Anything else that I try to stop that touches the alsa
sequencer never
Quoted 1 line
dies (qjackctl, vkeybd, qsynth, etc). Anything I try to
start that tries
Quoted 1 line
to use it does not start. This happened with two widely
different
Quoted 8 lines
could you get a tasklist-dump? It's either SysRq-T, or:
echo t >
/proc/sysrq-triggerthat should dump all tasks and their backtraces -
including the hung
rosegardensequencer task.
I had similar symptoms here (FC4) and cured them with attached patch against linux/2.6.15/rt18/kernel/softirq.c. Its in rt19 and ++ i think.
Karsten
___________________________________________________________
Telefonate ohne weitere Kosten vom PC zum PC: http://messenger.yahoo.de
tmp/softirq.c renamed from tmp/softirq.c~ +4 -4
@@ -351,13 +351,13 @@ static DEFINE_PER_CPU(struct tasklet_head, tasklet_hi_vec) = { NULL };static void inline -__tasklet_common_schedule (struct tasklet_struct *t,struct tasklet_head *head) +__tasklet_common_schedule (struct tasklet_struct *t,struct tasklet_head *head,unsigned int nr) {if (tasklet_trylock (t)) {WARN_ON (t->next !=NULL ); t->next = head->list; head->list = t; -raise_softirq_irqoff (TASKLET_SOFTIRQ ); +raise_softirq_irqoff (nr);tasklet_unlock (t); } }
@@ -367,7 +367,7 @@unsigned long flags;raw_local_irq_save (flags); -__tasklet_common_schedule (t, &__get_cpu_var (tasklet_vec)); +__tasklet_common_schedule (t, &__get_cpu_var (tasklet_vec),TASKLET_SOFTIRQ );raw_local_irq_restore (flags); }
@@ -378,7 +378,7 @@unsigned long flags;raw_local_irq_save (flags); -__tasklet_common_schedule (t, &__get_cpu_var (tasklet_hi_vec)); +__tasklet_common_schedule (t, &__get_cpu_var (tasklet_hi_vec),HI_SOFTIRQ );raw_local_irq_restore (flags); }
Quoted 18 lines · Ingo Molnar
Quoted 10 lines · Fernando Lopez-Lezcano
The symptoms are as follows:
- start jack using qjackctl
- start qsynth (gui front end for fluidsynth, a synth)
- start rosegarden (midi sequencer and audio recorder)
- load a midi file into rosegarden
- midi file plays successfully
- close rosegarden
at this point one of the threads of rosegarden fails to exit and stays forever in the process list, in a ps axuw it shows as:Quoted 5 lines
nando 5484 0.0 0.0 0 0 pts/1 D 13:32 0:00 [rosegardenseque]
Anything else that I try to stop that touches the alsa sequencer never dies (qjackctl, vkeybd, qsynth, etc). Anything I try to start that tries to use it does not start. This happened with two widely different
Quoted 6 lines
could you get a tasklist-dump? It's either SysRq-T, or:
echo t >
/proc/sysrq-triggerthat should dump all tasks and their backtraces - including the hung rosegardensequencer task.
I'll try to do that tomorrow.
In the meanwhile I think this might be the same problem that was solved in -rt20? (building it as I type). I have been trying to catch up to the subset of lkml that I manage to read and found some reports that apparently point to similar problems with the alsa sequencer (Rui, in particular, and then Karsten's post in this thread).
-- Fernando
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
At Tue, 07 Mar 2006 22:53:55 -0800,
Quoted 20 lines · Fernando Lopez-Lezcano
Quoted 18 lines · Ingo Molnar
Quoted 15 lines · Fernando Lopez-Lezcano
The symptoms are as follows:
- start jack using qjackctl
- start qsynth (gui front end for fluidsynth, a synth)
- start rosegarden (midi sequencer and audio recorder)
- load a midi file into rosegarden
- midi file plays successfully
- close rosegarden
at this point one of the threads of rosegarden fails to exit and stays forever in the process list, in a ps axuw it shows as:nando 5484 0.0 0.0 0 0 pts/1 D 13:32 0:00 [rosegardenseque]
Anything else that I try to stop that touches the alsa sequencer never dies (qjackctl, vkeybd, qsynth, etc). Anything I try to start that tries to use it does not start. This happened with two widely different
Quoted 8 lines
Quoted 6 lines
could you get a tasklist-dump? It's either SysRq-T, or:
echo t >
/proc/sysrq-triggerthat should dump all tasks and their backtraces - including the hung rosegardensequencer task.
I'll try to do that tomorrow.
That'll be helpful. A deadlocked mutex could be easily found by a backtrace.
Takashi
Quoted 11 lines · Takashi Iwai
Quoted 8 lines
Quoted 6 lines
could you get a tasklist-dump? It's either SysRq-T, or:
echo t >
/proc/sysrq-triggerthat should dump all tasks and their backtraces - including the hung rosegardensequencer task.
I'll try to do that tomorrow.
That'll be helpful. A deadlocked mutex could be easily found by a backtrace.
furthermore, in the -rt kernel (if PREEMPT_RT and DEBUG_DEADLOCKS is enabled), all locks will also be printed, and their owners, and the site where they got acquired, etc. A deadlocked mutex/semaphore should in fact be detected by the kernel, if those options are enabled.
Ingo
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
Quoted 11 lines
Quoted 6 lines
could you get a tasklist-dump? It's either SysRq-T, or:
echo t >
/proc/sysrq-triggerthat should dump all tasks and their backtraces -
including the hung
Quoted 1 line
rosegardensequencer task.
I'll try to do that tomorrow.
Quoted 9 lines
In the meanwhile I think this might be the same problem that was solved
in -rt20? (building it as I type). I have been trying to catch up to the
subset of lkml that I manage to read and found some reports that
apparently point to similar problems with the alsa sequencer (Rui, in
particular, and then Karsten's post in this thread).
ALSA Midi sequencer uses tasklets. there are 2 kinds of
them: lo and hi.
In rt-18 PREEMPT-RT, tasklet_hi_schedule() didn't work, 'cause it woke up tasklet_lo's thread.
Thats what my patch fixed.
Before finding the solution, i reduced my testcase to a simple "aconnect -d" call:
aconnect sends a midi-reset to the client to disconnect via tasklet_hi then. subsequently aconnect wants to finish, and
cannot: it has to wait for tasklet_hi cleanup, which doesn't happen, 'cause the wrong thread had been triggered.
Find the corresponding sysrq-t excerpt attached.
Karsten
___________________________________________________________
Telefonate ohne weitere Kosten vom PC zum PC: http://messenger.yahoo.de
[ 998.876000] aconnect [f5867010]R C013CF4C [on rq #0] 0 2708 2272 (NOTLB)
[ 998.876000] f3ca1d1c f5867010 c0452ee0 c013cf4c f3ca1000 00000000 f5867148 c0494740
[ 998.876000] f3ca1d08 f5867010 8ec9c300 000000e8 003d0900 00000000 f3ca1000 0002aa75
[ 998.876000] 00000000 f3ca1d44 c034a6e5 c01279b9 c0494740 f3ca1d58 0002aa73 60000000
[ 998.876000] Call Trace:
[ 998.876000] [<c034a6e5>] schedule+0xa5/0x140 (40)
[ 998.876000] [<c034b0cc>] schedule_timeout+0x4c/0xc0 (52)
[ 998.876000] [<c034b17a>] schedule_timeout_uninterruptible+0x1a/0x20 (8)
[ 998.876000] [<c0128bd8>] msleep+0x28/0x40 (12)
[ 998.876000] [<c0123ddc>] tasklet_kill+0x3c/0x70 (16)
[ 998.876000] [<f88372a3>] snd_rawmidi_drop_output+0x43/0x90 [snd_rawmidi] (32)
[ 998.876000] [<f8837372>] snd_rawmidi_drain_output+0x82/0x180 [snd_rawmidi] (68)
[ 998.876000] [<f8a4c5a7>] midisynth_unuse+0x37/0x80 [snd_seq_midi] (32)
[ 998.876000] [<f8aafb3e>] unsubscribe_port+0x4e/0x100 [snd_seq] (36)
[ 998.876000] [<f8ab0004>] snd_seq_port_disconnect+0x134/0x180 [snd_seq] (60)
[ 998.876000] [<f8aa9eb6>] snd_seq_ioctl_unsubscribe_port+0xe6/0x140 [snd_seq] (132)
[ 998.876000] [<f8aaaf9a>] snd_seq_do_ioctl+0x5a/0xd0 [snd_seq] (36)
[ 998.876000] [<f8aab039>] snd_seq_ioctl+0x29/0x60 [snd_seq] (24)
[ 998.876000] [<c01838a8>] do_ioctl+0x38/0xa0 (36)
[ 998.876000] [<c0183ad0>] vfs_ioctl+0x60/0x200 (40)
[ 998.876000] [<c0183cea>] sys_ioctl+0x7a/0x90 (40)
[ 998.876000] [<c01032db>] sysenter_past_esp+0x54/0x75 (-4020)
[ 998.876000] ---------------------------
[ 998.876000] | preempt count: 00000002 ]
[ 998.876000] | 2-level deep critical section nesting:
[ 998.876000] ----------------------------------------
[ 998.876000] .. [<c013cf4c>] .... add_preempt_count+0x1c/0x20
[ 998.876000] .....[<c0349f7b>] .. ( <= __schedule+0x4b/0x710)
[ 998.876000] .. [<c013cf4c>] .... add_preempt_count+0x1c/0x20
[ 998.876000] .....[<c034a021>] .. ( <= __schedule+0xf1/0x710)
[ 998.876000]
[ 998.876000] ------------------------------
[ 998.876000] | showing all locks held by: | (aconnect/2708 [f5867010, 116]):
[ 998.876000] ------------------------------
[ 998.876000]
[ 998.876000] #001: [f0ee1264] {(struct rw_semaphore *)(&grp->list_mutex)}
[ 998.876000] ... acquired at: snd_seq_port_disconnect+0x27/0x180 [snd_seq]
[ 998.876000]
[ 998.876000] #002: [f0ee16f8] {(struct rw_semaphore *)(&grp->list_mutex)}
[ 998.876000] ... acquired at: snd_seq_port_disconnect+0x3a/0x180 [snd_seq]
[ 998.876000]
[ 998.876000] Quoted 5 lines · karsten wiese
ALSA Midi sequencer uses tasklets. there are 2 kinds of
them: lo and hi.
In rt-18 PREEMPT-RT,tasklet_hi_schedule()didn't work, 'cause it woke up tasklet_lo's thread.
Thats what my patch fixed.
ah - i thought that -rt18 had your fix already - but indeed it's -rt19 that added it. So it's probably the same bug.
Ingo
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
Quoted 10 lines · Ingo Molnar
Quoted 5 lines · karsten wiese
ALSA Midi sequencer uses tasklets. there are 2 kinds of
them: lo and hi.
In rt-18 PREEMPT-RT,tasklet_hi_schedule()didn't work, 'cause it woke up tasklet_lo's thread.
Thats what my patch fixed.ah - i thought that -rt18 had your fix already - but indeed it's -rt19 that added it. So it's probably the same bug.
I agree, I just booted into -rt20 and the problem seems to be gone. Let me know if you still want the trace, I can always boot again into the buggy kernel.
Thanks for the help!
-- Fernando