Skip to contentSkip to conversation

Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-19

DISCUSSION Andrew Burgess 5 participants 12 messages Dec 2, 2004

lore
Share & tools
lore
#0 lore raw
Quoted 12 lines · Ingo Molnar
Quoted 5 lines · Florian Schmidt

Hmm, i wonder if there's a way to detect non RT behaviour in jackd clients. I mean AFAIK the only thing allowed for the process callback of on is the FIFO it waits on to be woken, right? Every other sleeping is to be considered a bug.

there's such a feature in -RT kernels. If a user process calls:

	gettimeofday(1,1);
then the kernel turns 'atomic mode' on. To turn it off, call:
	gettimeofday(1,0);
Quoted 7 lines

while in atomic-mode, any non-atomic activity (scheduling) will produce a kernel message and a SIGUSR2 sent to the offending process (once, atomic mode has to be re-enabled again for the next message). Preemption by a higher-prio task does not trigger a message/signal.

If you run the client under gdb you should be able to catch the SIGUSR2 signal and then you can see the offending code's backtrace via 'bt'.

Might be handy to have the option to send a SIGABRT, then you don't need to guess which app to run under gdb and the offending code is there in the core file.

Also, I'm cc-ing jack-devel. This could fit into libjack so no client mods would be needed I think. After the thread_init_callback is run libjack could run 'gettimeofday(1,1);' for each client thread. Then if any client breaks the rules you get a core showing where.

On further thought, I suppose libjack could install a SIGUSR2 handler and have that call abort for all the rt client threads. Still no client mods needed, only an RT-aware libjack.

A big thank you to Ingo and everyone else involved on behalf of all the linux audio users!

#1 lore raw
Quoted 26 lines · On Thu, 2 Dec 2004 07:46:20 -0800 Andrew Burgess <[email protected]> wrote:
Quoted 12 lines · Ingo Molnar
Quoted 5 lines · Florian Schmidt

Hmm, i wonder if there's a way to detect non RT behaviour in jackd clients. I mean AFAIK the only thing allowed for the process callback of on is the FIFO it waits on to be woken, right? Every other sleeping is to be considered a bug.

there's such a feature in -RT kernels. If a user process calls:

	gettimeofday(1,1);
then the kernel turns 'atomic mode' on. To turn it off, call:
	gettimeofday(1,0);
Quoted 7 lines

while in atomic-mode, any non-atomic activity (scheduling) will produce a kernel message and a SIGUSR2 sent to the offending process (once, atomic mode has to be re-enabled again for the next message). Preemption by a higher-prio task does not trigger a message/signal.

If you run the client under gdb you should be able to catch the SIGUSR2 signal and then you can see the offending code's backtrace via 'bt'.

Might be handy to have the option to send a SIGABRT, then you don't need to guess which app to run under gdb and the offending code is there in the core file.

Quoted 8 lines

Also, I'm cc-ing jack-devel. This could fit into libjack so no client mods would be needed I think. After the thread_init_callback is run libjack could run 'gettimeofday(1,1);' for each client thread. Then if any client breaks the rules you get a core showing where.

On further thought, I suppose libjack could install a SIGUSR2 handler and have that call abort for all the rt client threads. Still no client mods needed, only an RT-aware libjack.

right. Or instead of aborting jackd might print a debug output (like "client foo violated RT constraints").

But the calls to gettimeofday would need to be done right before and after every process callback as each client's RT thread does wait on the FIFO to get woken by jackd. This waiting would appear as RT constraints violation if the gettimeofday would be done only once per thread lifetime at thread startup.

Quoted 3 lines

A big thank you to Ingo and everyone else involved on behalf of all the linux audio users!

BTW: i suppose pretty much every jack client except for very simple ones do break the RT constraints.

Flo

#2 lore raw
Quoted 6 lines
Quoted 3 lines · On Thu, 2 Dec 2004 07:46:20 -0800 Andrew Burgess <[email protected]> wrote:

On further thought, I suppose libjack could install a SIGUSR2 handler and have that call abort for all the rt client threads. Still no client mods needed, only an RT-aware libjack.

Quoted 2 lines · Florian Schmidt

right. Or instead of aborting jackd might print a debug output (like "client foo violated RT constraints").

Libjack cannot assume the client has no SIGUSR2 handler of its own.

Quoted 2 lines
Quoted 2 lines

A big thank you to Ingo and everyone else involved on behalf of all the linux audio users!

thanks++ :-)

Quoted 2 lines

BTW: i suppose pretty much every jack client except for very simple ones do break the RT constraints.

True.

It would be wonderful to have a reliable mechanism for debugging them.

Signature · 1 line
  joq
#3 lore raw
Quoted 7 lines · On 02 Dec 2004 10:26:55 -0600 "Jack O'Quin" <[email protected]> wrote:
Quoted 6 lines
Quoted 3 lines · On Thu, 2 Dec 2004 07:46:20 -0800 Andrew Burgess <[email protected]> wrote:

On further thought, I suppose libjack could install a SIGUSR2 handler and have that call abort for all the rt client threads. Still no client mods needed, only an RT-aware libjack.

Quoted 5 lines
Quoted 2 lines · Florian Schmidt

right. Or instead of aborting jackd might print a debug output (like "client foo violated RT constraints").

Libjack cannot assume the client has no SIGUSR2 handler of its own.

i see..

Quoted 1 line

It would be wonderful to have a reliable mechanism for debugging them.

I suppose instead of catching the signal the user might just monitor the syslog. I'm not sure there's printk's triggered by thisalready , but i'm sure if not, ingo might add them. So a trivial patch for jackd would probably look like this:

client.c 2004-12-02 17:56:23.000000000 +0100 renamed from client.c.orig 2004-12-02 17:55:04.000000000 +0100 +7 -1
@@ -1238,6 +1238,9 @@
                        if (control->sync_cb)
                                jack_call_sync_client (client);
 
+                       // enable atomicity check for RP kernels
+                       gettimeofday(1,1);
+
                        if (control->process) {
                                if (control->process (control->nframes,
                                                      control->process_arg)
@@ -1247,7 +1250,10 @@
                        } else {
                                control->state = Finished;
                        }
-
+
+                       // disable atomicity check
+                       gettimeofday(0,1);
+
                        if (control->timebase_cb)
                                jack_call_timebase_master (client);
 

flo

#4 lore raw
Quoted 39 lines · On Thu, 2 Dec 2004 17:57:56 +0100 Florian Schmidt <[email protected]> wrote:
Quoted 7 lines · On 02 Dec 2004 10:26:55 -0600 "Jack O'Quin" <[email protected]> wrote:
Quoted 6 lines
Quoted 3 lines · On Thu, 2 Dec 2004 07:46:20 -0800 Andrew Burgess <[email protected]> wrote:

On further thought, I suppose libjack could install a SIGUSR2 handler and have that call abort for all the rt client threads. Still no client mods needed, only an RT-aware libjack.

Quoted 5 lines
Quoted 2 lines · Florian Schmidt

right. Or instead of aborting jackd might print a debug output (like "client foo violated RT constraints").

Libjack cannot assume the client has no SIGUSR2 handler of its own.

i see..

Quoted 1 line

It would be wonderful to have a reliable mechanism for debugging them.

I suppose instead of catching the signal the user might just monitor the syslog. I'm not sure there's printk's triggered by thisalready , but i'm sure if not, ingo might add them. So a trivial patch for jackd would probably look like this:

client.c 2004-12-02 17:56:23.000000000 +0100 renamed from client.c.orig 2004-12-02 17:55:04.000000000 +0100 +3 -0
@@ -1238,6 +1238,9 @@
                        if (control->sync_cb)
                                jack_call_sync_client (client);
 
+                       // enable atomicity check for RP kernels
+                       gettimeofday(1,1);
+
                        if (control->process) {
                                if (control->process (control->nframes,
                                                      control->process_arg)
@@ -1247,7 +1250,10 @@
                        } else {

client.c 2004-12-02 17:56:23.000000000 +0100 line 1259

Quoted 10 lines
client.c 2004-12-02 17:56:23.000000000 +0100 renamed from client.c.orig 2004-12-02 17:55:04.000000000 +0100 +4 -1
                                control->state = Finished;
                        }
-
+
+                       // disable atomicity check
+                       gettimeofday(0,1);
+
                        if (control->timebase_cb)
                                jack_call_timebase_master (client);
 

Well, i do get syslog output with this patch like this:

Dec  2 18:01:11 mango kernel: jack_test:22645 userspace BUG: scheduling in user-atomic context!
Dec  2 18:01:11 mango kernel:  [<c02a38b6>] schedule+0x76/0x130 (8)
Dec  2 18:01:11 mango kernel:  [<c02a44c5>] schedule_timeout+0x85/0xe0 (36)
Dec  2 18:01:11 mango kernel:  [<c016677f>] do_pollfd+0x4f/0x90 (48)
Dec  2 18:01:11 mango kernel:  [<c011ceb0>] process_timeout+0x0/0x10 (8)
Dec  2 18:01:11 mango kernel:  [<c016686a>] do_poll+0xaa/0xd0 (20)
Dec  2 18:01:11 mango kernel:  [<c01669e2>] sys_poll+0x152/0x230 (48)
Dec  2 18:01:11 mango kernel:  [<c0165db0>] __pollwait+0x0/0xd0 (36)
Dec  2 18:01:11 mango kernel:  [<c01025cb>] syscall_call+0x7/0xb (32)

even if the client's process callback is a noop (except for returning 0). Hmm, i must have missed something in jackd's source. i thought control->process() directly calls the clients process callback..

hmm..

#5 lore raw
Quoted 2 lines · Florian Schmidt

Hmm, i must have missed something in jackd's source. i thought control->process() directly calls the clients process callback..

It does.

Signature · 1 line
  joq
#6 lore raw
Quoted 6 lines · On 02 Dec 2004 11:32:49 -0600 "Jack O'Quin" <[email protected]> wrote:
Quoted 2 lines · Florian Schmidt

Hmm, i must have missed something in jackd's source. i thought control->process() directly calls the clients process callback..

It does.

Then either i have misunderstood how it works, or the mechanism is still buggy.. Wrote mail to Ingo. Will report to jackit-devel when i get an answer.

Flo

#7 lore raw
Quoted 5 lines · On Thu, 2 Dec 2004 21:03:18 +0100 Florian Schmidt <[email protected]> wrote:
Quoted 2 lines · On 02 Dec 2004 11:32:49 -0600 "Jack O'Quin" <[email protected]> wrote:

Florian Schmidt <[email protected]> writes:

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Hmm, i must have missed something in jackd's source. i thought control->process() directly calls the clients process callback..

It does.

Then either i have misunderstood how it works, or the mechanism is still buggy.. Wrote mail to Ingo. Will report to jackit-devel when i get an answer.

PEBCAK :) Here's the corrected patch:

client.c renamed from client.c.orig +7 -1
@@ -1238,6 +1238,9 @@
 			if (control->sync_cb)
 				jack_call_sync_client (client);
 
+			// enable atomicity check for RP kernels
+			gettimeofday(1,1);
+			
 			if (control->process) {
 				if (control->process (control->nframes,
 						      control->process_arg)
@@ -1247,7 +1250,10 @@
 			} else {
 				control->state = Finished;
 			}
-
+			
+			// disable atomicity check
+			gettimeofday(1,0);
+			
 			if (control->timebase_cb)
 				jack_call_timebase_master (client);
 

seems to work well with my changed jack_test client (this one sleeps in the 1000th call of the process callback and thuis triggers this trace and aborts as it doesn't handle SIGUSR2). test client attached.

Dec  2 22:05:10 mango kernel: jack_test:3043 userspace BUG: scheduling in user-atomic context!
Dec  2 22:05:10 mango kernel:  [<c02a38b6>] schedule+0x76/0x130 (8)
Dec  2 22:05:10 mango kernel:  [<c02a44c5>] schedule_timeout+0x85/0xe0 (36)
Dec  2 22:05:10 mango kernel:  [<c01e2f02>] copy_from_user+0x42/0x80 (48)
Dec  2 22:05:10 mango kernel:  [<c011ceb0>] process_timeout+0x0/0x10 (8)
Dec  2 22:05:10 mango kernel:  [<c011cfae>] sys_nanosleep+0xde/0x170 (20)
Dec  2 22:05:10 mango kernel:  [<c01025cb>] syscall_call+0x7/0xb (52)

flo

#include <jack/jack.h>
#include <iostream>
#include <sstream>
#include <unistd.h>
#include <signal.h>

jack_client_t *client;
jack_port_t *iport;
jack_port_t *oport;

int wasted_loops = 0;

int sleep_seconds = 1;
int sleep_in_period = 1000;
int counter = 0;

int process(jack_nframes_t frames, void *arg) {
	
	// std::cout << "process callback" << std::endl;
	jack_default_audio_sample_t *ibuf;
	ibuf = (jack_default_audio_sample_t*)jack_port_get_buffer(iport, frames);

	jack_default_audio_sample_t *obuf;
	obuf = (jack_default_audio_sample_t*)jack_port_get_buffer(oport, frames);

	for (jack_nframes_t frame = 0; frame < frames; frame++) {
		for (int i = 0; i < wasted_loops; ++i) {
			// do nothing
		}
		obuf[frame] = ibuf[frame];
	}
	counter++;
	if (counter == sleep_in_period) {
	  sleep(sleep_seconds);
	}

        return 0;
}

void signalled(int sig) {
std::cout << "signalled" << std::endl;
}

int main(int argc, char *argv[]) {
	// default = 60 seconds
	unsigned int seconds_to_run = 60;
	if (argc > 1) {
		std::stringstream sec_stream;
		sec_stream << argv[1];
		sec_stream >> seconds_to_run;
		if (argc > 2) {
			std::stringstream waste_stream;
			waste_stream << argv[2];
			waste_stream >> wasted_loops;
			std::cout << "wasted loops: " << wasted_loops << std::endl;
		}
	}

	// signal(SIGUSR2, signalled);

	std::cout << "seconds to run: " << seconds_to_run << std::endl;
	
	std::stringstream pid_stream;
	pid_stream << getpid();
	
        std::cout << "client_new" << std::endl;
        client = jack_client_new(pid_stream.str().c_str());

        std::cout << "port_register." << std::endl;
        iport = jack_port_register(client, "in", JACK_DEFAULT_AUDIO_TYPE, JackPortIsInput|JackPortIsTerminal, 0);
	oport = jack_port_register(client, "out", JACK_DEFAULT_AUDIO_TYPE, JackPortIsTerminal|JackPortIsOutput, 0);

        std::cout << "set_process_callback" << std::endl;
        jack_set_process_callback(client, process, 0);

        std::cout << "activate" << std::endl;
        jack_activate(client);

        std::cout << "running" << std::endl;

        // while(1) {sleep(1);};
	sleep(seconds_to_run);

	jack_deactivate(client);
	jack_client_close(client);
}
#8 lore raw
In reply to Florian Schmidt
Quoted 19 lines · Florian Schmidt

I suppose instead of catching the signal the user might just monitor the syslog. I'm not sure there's printk's triggered by thisalready , but i'm sure if not, ingo might add them. So a trivial patch for jackd would probably look like this:

client.c 2004-12-02 17:56:23.000000000 +0100 renamed from client.c.orig 2004-12-02 17:55:04.000000000 +0100 +3 -0
@@ -1238,6 +1238,9 @@
                        if (control->sync_cb)
                                jack_call_sync_client (client);
 
+                       // enable atomicity check for RP kernels
+                       gettimeofday(1,1);
+
                        if (control->process) {
                                if (control->process (control->nframes,
                                                      control->process_arg)
@@ -1247,7 +1250,10 @@
                        } else {

client.c 2004-12-02 17:56:23.000000000 +0100 line 1259

Quoted 10 lines
client.c 2004-12-02 17:56:23.000000000 +0100 renamed from client.c.orig 2004-12-02 17:55:04.000000000 +0100 +4 -1
                                control->state = Finished;
                        }
-
+
+                       // disable atomicity check
+                       gettimeofday(0,1);
+
                        if (control->timebase_cb)
                                jack_call_timebase_master (client);
 

The sync_cb and timebase_cb callbacks actually need to be RT-safe, too. ;-)

Is printk() guaranteed not to wait inside the kernel? I am not familiar with its internal implementation.

Signature · 1 line
  joq
#9 lore raw
Quoted 2 lines · Jack O'Quin

Is printk() guaranteed not to wait inside the kernel? I am not familiar with its internal implementation.

Yes. It just writes to a ring buffer and klogd dumps this to syslog. So if you really start to spew printk's they don't all make it to the log but you never get blocked.

The implementation probably looks a lot like a correct solution to fix the printf-from-RT-context issue in JACK would.

Lee

#10 lore raw
Quoted 10 lines · Lee Revell
Quoted 2 lines · Jack O'Quin

Is printk() guaranteed not to wait inside the kernel? I am not familiar with its internal implementation.

Yes. It just writes to a ring buffer and klogd dumps this to syslog. So if you really start to spew printk's they don't all make it to the log but you never get blocked.

The implementation probably looks a lot like a correct solution to fix the printf-from-RT-context issue in JACK would.

Right. That's exactly what I have in mind, whenever I find time to work on it. :-)

There's some similar code I wrote for JAMin, which we could adapt.

Signature · 1 line
  joq
#11 lore raw
In reply to Andrew Burgess
Quoted 2 lines · Andrew Burgess

A big thank you to Ingo and everyone else involved on behalf of all the linux audio users!

This is going to have a lot of ripple effect throughout the Linux community in that game developers will quite possibly have the ability to do low latency OpenGL, frame accurate video and other things along those lines. The previous Linux kernels without these mods couldn't allow this level of temporal precious for application developers. It's going to push applications like jackd and others in ways that will flush out bugs and general techniques that aren't typically apart of a proper RT applications.

We have the possibility of providing a first class gaming platform and other things, that Longhorn can't beging to touch if the middleware falls into place. :)

bill