Skip to contentSkip to conversation

2.6.15-rt18, alsa sequencer, rosegarden -> alsa hangs

DISCUSSION Fernando Lopez-Lezcano 4 participants 9 messages Mar 7, 2006 – Mar 8, 2006

lore
Share & tools
lore
#0 lore raw

Hi all, I don't know where to go from here...

I'm trying to get a 2.6.15-rt18 kernel going and I'm finding a big showstopper in the alsa sequencer, presumably at the kernel module level. This is -rt18 with PREEMPT_RT and high resolution timers enabled, plus either alsa kernel drivers from 1.0.10 (latest stable release) or 1.0.11rc3. Same problem for both. I initially thought that the snd-rtctimer kernel module was a problem so the latest tests were done with that module disabled so that the alsa sequencer is using the system timer for timing (HZ == 1000 for this kernel).

The symptoms are as follows:
- start jack using qjackctl
- start qsynth (gui front end for fluidsynth, a synth)
- start rosegarden (midi sequencer and audio recorder)
- load a midi file into rosegarden
- midi file plays successfully
- close rosegarden
at this point one of the threads of rosegarden fails to exit and stays forever in the process list, in a ps axuw it shows as:

nando 5484 0.0 0.0 0 0 pts/1 D 13:32 0:00 [rosegardenseque]

Anything else that I try to stop that touches the alsa sequencer never dies (qjackctl, vkeybd, qsynth, etc). Anything I try to start that tries to use it does not start. This happened with two widely different versions of rosegarden (1.0 and 1.2.3) - I have not yet tried with other alsa sequencer clients - I tried different kernel builds and alsa builds before giving up, the cycle is long when I include the frequent reboots. I can browse around in /proc/asound/seq but a:
cat /proc/asound/clients
hangs forever (unkillable). Obviously I can't unload the alsa subsystem to try again. At this point my only recourse to recover is a reboot (which hangs when trying to stop the alsa subsystem - so the reset key is the only option).

There are no kernel oops or messages either in dmesg or /var/log/messages that I can see.

Booting into 2.6.14 with the latest -rt patch for that branch does not show the problem (using alsa kernel modules from 1.0.10 stable).

The problem happens with at least three completely different soundcards and two different kernel builds (one kind of vanilla, the other derived from the latest fc4 kernel configuration and patches).

So it would look like an interaction between the newest -rt patches and the alsa sequencer timing. Weird because while rosegarden is up and running it plays midi sequences with no apparent problems, it is only when it exits that things get completely messed up.

Suggestions appreciated...
-- Fernando

# cat /proc/asound/seq/timer
Timer for queue 0 : system timer
  Period time : 0.001000000
  Skew : 65536 / 65536

[rosegarden is still there.....]

# cat /proc/asound/seq/queues
queue 0: [Rosegarden queue]
owned by client    : 130
lock status        : Locked
queued time events : 0
queued tick events : 0
timer state        : Running
timer PPQ          : 96
current tempo      : 500000
current BPM        : 120
current time       : 1617.680000000 s
current tick       : 318130
# cat /proc/asound/seq/drivers
snd-seq-oss,loaded,0
snd-seq-midi,loaded,1
#1 lore raw
Quoted 10 lines · Fernando Lopez-Lezcano

The symptoms are as follows:
- start jack using qjackctl
- start qsynth (gui front end for fluidsynth, a synth)
- start rosegarden (midi sequencer and audio recorder)
- load a midi file into rosegarden
- midi file plays successfully
- close rosegarden
at this point one of the threads of rosegarden fails to exit and stays forever in the process list, in a ps axuw it shows as:

Quoted 5 lines

nando 5484 0.0 0.0 0 0 pts/1 D 13:32 0:00 [rosegardenseque]

Anything else that I try to stop that touches the alsa sequencer never dies (qjackctl, vkeybd, qsynth, etc). Anything I try to start that tries to use it does not start. This happened with two widely different

could you get a tasklist-dump? It's either SysRq-T, or:

echo t > /proc/sysrq-trigger

that should dump all tasks and their backtraces - including the hung rosegardensequencer task.

Ingo

#2 lore raw
Quoted 27 lines · --- Ingo Molnar
Quoted 3 lines · Fernando Lopez-Lezcano

The symptoms are as follows:
- start jack using qjackctl
- start qsynth (gui front end for fluidsynth, a

synth)

Quoted 1 line

- start rosegarden (midi sequencer and audio

recorder)

Quoted 4 lines

- load a midi file into rosegarden
- midi file plays successfully
- close rosegarden
at this point one of the threads of rosegarden fails to

exit and stays

Quoted 3 lines

forever in the process list, in a ps axuw it shows as:

nando 5484 0.0 0.0 0 0 pts/1 D 13:32 0:00

[rosegardenseque]

Quoted 2 lines

Anything else that I try to stop that touches the alsa

sequencer never

Quoted 1 line

dies (qjackctl, vkeybd, qsynth, etc). Anything I try to

start that tries

Quoted 1 line

to use it does not start. This happened with two widely

different

Quoted 8 lines

could you get a tasklist-dump? It's either SysRq-T, or:

echo t > /proc/sysrq-trigger

that should dump all tasks and their backtraces -
including the hung
rosegardensequencer task.

I had similar symptoms here (FC4) and cured them with attached patch against linux/2.6.15/rt18/kernel/softirq.c. Its in rt19 and ++ i think.

Karsten

___________________________________________________________
Telefonate ohne weitere Kosten vom PC zum PC: http://messenger.yahoo.de

tmp/softirq.c renamed from tmp/softirq.c~ +4 -4
@@ -351,13 +351,13 @@
 static DEFINE_PER_CPU(struct tasklet_head, tasklet_hi_vec) = { NULL };
 
 static void inline
-__tasklet_common_schedule(struct tasklet_struct *t, struct tasklet_head *head)
+__tasklet_common_schedule(struct tasklet_struct *t, struct tasklet_head *head, unsigned int nr)
 {
 	if (tasklet_trylock(t)) {
 		WARN_ON(t->next != NULL);
 		t->next = head->list;
 		head->list = t;
-		raise_softirq_irqoff(TASKLET_SOFTIRQ);
+		raise_softirq_irqoff(nr);
 		tasklet_unlock(t);
 	}
 }
@@ -367,7 +367,7 @@
 	unsigned long flags;
 
 	raw_local_irq_save(flags);
-	__tasklet_common_schedule(t, &__get_cpu_var(tasklet_vec));
+	__tasklet_common_schedule(t, &__get_cpu_var(tasklet_vec), TASKLET_SOFTIRQ);
 	raw_local_irq_restore(flags);
 }
 
@@ -378,7 +378,7 @@
 	unsigned long flags;
 
 	raw_local_irq_save(flags);
-	__tasklet_common_schedule(t, &__get_cpu_var(tasklet_hi_vec));
+	__tasklet_common_schedule(t, &__get_cpu_var(tasklet_hi_vec), HI_SOFTIRQ);
 	raw_local_irq_restore(flags);
 }
 
#3 lore raw
In reply to Ingo Molnar
Quoted 18 lines · Ingo Molnar
Quoted 10 lines · Fernando Lopez-Lezcano

The symptoms are as follows:
- start jack using qjackctl
- start qsynth (gui front end for fluidsynth, a synth)
- start rosegarden (midi sequencer and audio recorder)
- load a midi file into rosegarden
- midi file plays successfully
- close rosegarden
at this point one of the threads of rosegarden fails to exit and stays forever in the process list, in a ps axuw it shows as:

Quoted 5 lines

nando 5484 0.0 0.0 0 0 pts/1 D 13:32 0:00 [rosegardenseque]

Anything else that I try to stop that touches the alsa sequencer never dies (qjackctl, vkeybd, qsynth, etc). Anything I try to start that tries to use it does not start. This happened with two widely different

Quoted 6 lines

could you get a tasklist-dump? It's either SysRq-T, or:

echo t > /proc/sysrq-trigger

that should dump all tasks and their backtraces - including the hung rosegardensequencer task.

I'll try to do that tomorrow.

In the meanwhile I think this might be the same problem that was solved in -rt20? (building it as I type). I have been trying to catch up to the subset of lkml that I manage to read and found some reports that apparently point to similar problems with the alsa sequencer (Rui, in particular, and then Karsten's post in this thread).

-- Fernando

-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

#4 lore raw

At Tue, 07 Mar 2006 22:53:55 -0800,

Quoted 20 lines · Fernando Lopez-Lezcano
Quoted 18 lines · Ingo Molnar
Quoted 15 lines · Fernando Lopez-Lezcano

The symptoms are as follows:
- start jack using qjackctl
- start qsynth (gui front end for fluidsynth, a synth)
- start rosegarden (midi sequencer and audio recorder)
- load a midi file into rosegarden
- midi file plays successfully
- close rosegarden
at this point one of the threads of rosegarden fails to exit and stays forever in the process list, in a ps axuw it shows as:

nando 5484 0.0 0.0 0 0 pts/1 D 13:32 0:00 [rosegardenseque]

Anything else that I try to stop that touches the alsa sequencer never dies (qjackctl, vkeybd, qsynth, etc). Anything I try to start that tries to use it does not start. This happened with two widely different

Quoted 8 lines
Quoted 6 lines

could you get a tasklist-dump? It's either SysRq-T, or:

echo t > /proc/sysrq-trigger

that should dump all tasks and their backtraces - including the hung rosegardensequencer task.

I'll try to do that tomorrow.

That'll be helpful. A deadlocked mutex could be easily found by a backtrace.

Takashi

#5 lore raw
Quoted 11 lines · Takashi Iwai
Quoted 8 lines
Quoted 6 lines

could you get a tasklist-dump? It's either SysRq-T, or:

echo t > /proc/sysrq-trigger

that should dump all tasks and their backtraces - including the hung rosegardensequencer task.

I'll try to do that tomorrow.

That'll be helpful. A deadlocked mutex could be easily found by a backtrace.

furthermore, in the -rt kernel (if PREEMPT_RT and DEBUG_DEADLOCKS is enabled), all locks will also be printed, and their owners, and the site where they got acquired, etc. A deadlocked mutex/semaphore should in fact be detected by the kernel, if those options are enabled.

Ingo

-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

#6 lore raw
In reply to Fernando Lopez-Lezcano
Quoted 11 lines
Quoted 6 lines

could you get a tasklist-dump? It's either SysRq-T, or:

echo t > /proc/sysrq-trigger

that should dump all tasks and their backtraces -

including the hung

Quoted 1 line

rosegardensequencer task.

I'll try to do that tomorrow.

Quoted 9 lines

In the meanwhile I think this might be the same problem that was solved
in -rt20? (building it as I type). I have been trying to catch up to the
subset of lkml that I manage to read and found some reports that
apparently point to similar problems with the alsa sequencer (Rui, in
particular, and then Karsten's post in this thread).

ALSA Midi sequencer uses tasklets. there are 2 kinds of
them: lo and hi.
In rt-18 PREEMPT-RT, tasklet_hi_schedule() didn't work, 'cause it woke up tasklet_lo's thread.
Thats what my patch fixed.

Before finding the solution, i reduced my testcase to a simple "aconnect -d" call:
aconnect sends a midi-reset to the client to disconnect via tasklet_hi then. subsequently aconnect wants to finish, and
cannot: it has to wait for tasklet_hi cleanup, which doesn't happen, 'cause the wrong thread had been triggered.

Find the corresponding sysrq-t excerpt attached.

Karsten

___________________________________________________________
Telefonate ohne weitere Kosten vom PC zum PC: http://messenger.yahoo.de

[  998.876000] aconnect      [f5867010]R C013CF4C [on rq #0]     0  2708   2272                     (NOTLB)
[  998.876000] f3ca1d1c f5867010 c0452ee0 c013cf4c f3ca1000 00000000 f5867148 c0494740 
[  998.876000]        f3ca1d08 f5867010 8ec9c300 000000e8 003d0900 00000000 f3ca1000 0002aa75 
[  998.876000]        00000000 f3ca1d44 c034a6e5 c01279b9 c0494740 f3ca1d58 0002aa73 60000000 
[  998.876000] Call Trace:
[  998.876000]  [<c034a6e5>] schedule+0xa5/0x140 (40)
[  998.876000]  [<c034b0cc>] schedule_timeout+0x4c/0xc0 (52)
[  998.876000]  [<c034b17a>] schedule_timeout_uninterruptible+0x1a/0x20 (8)
[  998.876000]  [<c0128bd8>] msleep+0x28/0x40 (12)
[  998.876000]  [<c0123ddc>] tasklet_kill+0x3c/0x70 (16)
[  998.876000]  [<f88372a3>] snd_rawmidi_drop_output+0x43/0x90 [snd_rawmidi] (32)
[  998.876000]  [<f8837372>] snd_rawmidi_drain_output+0x82/0x180 [snd_rawmidi] (68)
[  998.876000]  [<f8a4c5a7>] midisynth_unuse+0x37/0x80 [snd_seq_midi] (32)
[  998.876000]  [<f8aafb3e>] unsubscribe_port+0x4e/0x100 [snd_seq] (36)
[  998.876000]  [<f8ab0004>] snd_seq_port_disconnect+0x134/0x180 [snd_seq] (60)
[  998.876000]  [<f8aa9eb6>] snd_seq_ioctl_unsubscribe_port+0xe6/0x140 [snd_seq] (132)
[  998.876000]  [<f8aaaf9a>] snd_seq_do_ioctl+0x5a/0xd0 [snd_seq] (36)
[  998.876000]  [<f8aab039>] snd_seq_ioctl+0x29/0x60 [snd_seq] (24)
[  998.876000]  [<c01838a8>] do_ioctl+0x38/0xa0 (36)
[  998.876000]  [<c0183ad0>] vfs_ioctl+0x60/0x200 (40)
[  998.876000]  [<c0183cea>] sys_ioctl+0x7a/0x90 (40)
[  998.876000]  [<c01032db>] sysenter_past_esp+0x54/0x75 (-4020)
[  998.876000] ---------------------------
[  998.876000] | preempt count: 00000002 ]
[  998.876000] | 2-level deep critical section nesting:
[  998.876000] ----------------------------------------
[  998.876000] .. [<c013cf4c>] .... add_preempt_count+0x1c/0x20
[  998.876000] .....[<c0349f7b>] ..   ( <= __schedule+0x4b/0x710)
[  998.876000] .. [<c013cf4c>] .... add_preempt_count+0x1c/0x20
[  998.876000] .....[<c034a021>] ..   ( <= __schedule+0xf1/0x710)
[  998.876000] 
[  998.876000] ------------------------------
[  998.876000] | showing all locks held by: |  (aconnect/2708 [f5867010, 116]):
[  998.876000] ------------------------------
[  998.876000] 
[  998.876000] #001:             [f0ee1264] {(struct rw_semaphore *)(&grp->list_mutex)}
[  998.876000] ... acquired at:               snd_seq_port_disconnect+0x27/0x180 [snd_seq]
[  998.876000] 
[  998.876000] #002:             [f0ee16f8] {(struct rw_semaphore *)(&grp->list_mutex)}
[  998.876000] ... acquired at:               snd_seq_port_disconnect+0x3a/0x180 [snd_seq]
[  998.876000] 
[  998.876000] 
#7 lore raw
Quoted 5 lines · karsten wiese

ALSA Midi sequencer uses tasklets. there are 2 kinds of
them: lo and hi.
In rt-18 PREEMPT-RT, tasklet_hi_schedule() didn't work, 'cause it woke up tasklet_lo's thread.
Thats what my patch fixed.

ah - i thought that -rt18 had your fix already - but indeed it's -rt19 that added it. So it's probably the same bug.

Ingo

-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

#8 lore raw
Quoted 10 lines · Ingo Molnar
Quoted 5 lines · karsten wiese

ALSA Midi sequencer uses tasklets. there are 2 kinds of
them: lo and hi.
In rt-18 PREEMPT-RT, tasklet_hi_schedule() didn't work, 'cause it woke up tasklet_lo's thread.
Thats what my patch fixed.

ah - i thought that -rt18 had your fix already - but indeed it's -rt19 that added it. So it's probably the same bug.

I agree, I just booted into -rt20 and the problem seems to be gone. Let me know if you still want the trace, I can always boot again into the buggy kernel.

Thanks for the help!
-- Fernando