sched: Isochronous class for unprivileged soft rt scheduling

This patch for 2.6.11-rc1 provides a method of providing real time scheduling to unprivileged users which increasingly is desired for multimedia workloads.

It does this by adding a new scheduling class called SCHED_ISO or Isochronous scheduling which means "same time" scheduling. This class does not require superuser privileges and is starvation free. The scheduling class no. 4 was chosen since there are quite a few userspace applications already supporting 3 and 4 for SCHED_BATCH and SCHED_ISO respectively on non-mainline kernels. As a way of immediately providing support for current userspace apps, any unprivileged user starting an application requesting SCHED_RR or SCHED_FIFO will be demoted to SCHED_ISO. This may or may not be worth removing later.

The SCHED_ISO class runs as SCHED_RR effectively at a priority just above all SCHED_NORMAL tasks and below all true real time tasks. Once a cpu usage limit is exceeded by tasks of this class (per cpu), SCHED_ISO tasks will then run as SCHED_NORMAL until the cpu usage drops to 90% of the cpu limit.

By default the cpu limit is set to 70% which literature suggests should provide good real time behaviour for most applications without gross unfairness. This cpu limit is calculated as a decaying average over 5 seconds. These limits are configurable with the tunables /proc/sys/kernel/iso_cpu
/proc/sys/kernel/iso_period

iso_cpu can be set to 100 which would allow all unprivileged users access to unrestricted SCHED_RR behaviour. OSX provides a similar class to SCHED_ISO and uses 90% as its cpu limit.

The sysrq-n combination which converts all user real-time tasks to SCHED_NORMAL also will affect SCHED_ISO tasks.

Currently the round robin interval is set to 10ms which is a cache friendly timeslice. It may be worth making this configurable or smaller, and it would also be possible to implement SCHED_ISO of a FIFO nature as well.

For testing, the userspace tool schedtool available here:
http://freequaos.host.sk/schedtool/
can be used as a wrapper to start SCHED_ISO tasks schedtool -I -e xmms
for example

Patch also available here:
http://ck.kolivas.org/patches/SCHED_ISO/

Comments and testing welcome.

Index: linux-2.6.11-rc1/include/linux/init_task.h
===================================================================
@@ -80,6 +80,7 @@
 	.mm		= NULL,						\
 	.active_mm	= &init_mm,					\
 	.run_list	= LIST_HEAD_INIT(tsk.run_list),			\
+	.iso_list	= LIST_HEAD_INIT(tsk.iso_list),			\
 	.time_slice	= HZ,						\
 	.tasks		= LIST_HEAD_INIT(tsk.tasks),			\
 	.ptrace_children= LIST_HEAD_INIT(tsk.ptrace_children),		\

Index: linux-2.6.11-rc1/include/linux/sched.h
===================================================================
@@ -130,6 +130,18 @@
 #define SCHED_NORMAL		0
 #define SCHED_FIFO		1
 #define SCHED_RR		2
+/* policy 3 reserved for SCHED_BATCH */
+#define SCHED_ISO		4
+
+extern int iso_cpu, iso_period;
+
+#define SCHED_RANGE(policy)	((policy) == SCHED_NORMAL || \
+				(policy) == SCHED_FIFO || \
+				(policy) == SCHED_RR || \
+				(policy) == SCHED_ISO)
+
+#define SCHED_RT(policy)	((policy) == SCHED_FIFO || \
+				(policy) == SCHED_RR)
 
 struct sched_param {
 	int sched_priority;

@@ -359,6 +371,7 @@
 #define MAX_PRIO		(MAX_RT_PRIO + 40)
 
 #define rt_task(p)		(unlikely((p)->prio < MAX_RT_PRIO))
+#define iso_task(p)		(unlikely((p)->policy == SCHED_ISO))
 
 /*
  * Some day this will be a full-fledged user tracking system..

@@ -536,6 +549,7 @@
 
 	int prio, static_prio;
 	struct list_head run_list;
+	struct list_head iso_list;
 	prio_array_t *array;
 
 	unsigned long sleep_avg;

Index: linux-2.6.11-rc1/include/linux/sysctl.h
===================================================================
@@ -135,6 +135,8 @@
 	KERN_HZ_TIMER=65,	/* int: hz timer on or off */
 	KERN_UNKNOWN_NMI_PANIC=66, /* int: unknown nmi panic flag */
 	KERN_BOOTLOADER_TYPE=67, /* int: boot loader type */
+	KERN_ISO_CPU=68,	/* int: cpu% allowed by SCHED_ISO class */
+	KERN_ISO_PERIOD=69,	/* int: seconds over which SCHED_ISO cpu is decayed */
 };
 
 

Index: linux-2.6.11-rc1/kernel/sysctl.c
===================================================================
@@ -219,6 +219,11 @@
 	{ .ctl_name = 0 }
 };
 
+/* Constants for minimum and maximum testing in vm_table.
+   We use these as one-element integer vectors. */
+static int zero;
+static int one_hundred = 100;
+
 static ctl_table kern_table[] = {
 	{
 		.ctl_name	= KERN_OSTYPE,

@@ -633,15 +638,28 @@
 		.proc_handler	= &proc_dointvec,
 	},
 #endif
+	{
+		.ctl_name	= KERN_ISO_CPU,
+		.procname	= "iso_cpu",
+		.data		= &iso_cpu,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec_minmax,
+		.strategy	= &sysctl_intvec,
+		.extra1		= &zero,
+		.extra2		= &one_hundred,
+	},
+	{
+		.ctl_name	= KERN_ISO_PERIOD,
+		.procname	= "iso_period",
+		.data		= &iso_period,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec,
+	},
 	{ .ctl_name = 0 }
 };
 
-/* Constants for minimum and maximum testing in vm_table.
-   We use these as one-element integer vectors. */
-static int zero;
-static int one_hundred = 100;
-
-
 static ctl_table vm_table[] = {
 	{
 		.ctl_name	= VM_OVERCOMMIT_MEMORY,

I should have mentioned. Many thanks to Alex Nyberg for generous debugging help.

Cheers,
Con

There's a collection of test summaries from jack_test3.2 runs at
<http://www.graggrag.com/ck-tests/ck-tests-0501182249.txt>

Tests were run with iso_cpu at 70, 90, 99, 100, each test was run twice. The discrepancies between consecutive runs (with same parameters) is puzzling. Also recorded were tests with SCHED_FIFO and SCHED_RR.

Before drawing any hardball conclusions, verification of the results would be nice. At first glance, it does seem that we still have that fateful gap between "harm minimisation" (policy) and "zero tolerance" (audio reality requirement).

cheers, Cal

Thanks.

SCHED_ISO
/proc/sys/kernel/iso_cpu . . .: 70
/proc/sys/kernel/iso_period . : 5
XRUN Count . . . . . . . . . : 110

vs

SCHED_FIFO

XRUN Count  . . . . . . . . . :   114
XRUN Count  . . . . . . . . . :   187

vs

SCHED_RR

XRUN Count  . . . . . . . . . :     0
XRUN Count  . . . . . . . . . :     0

Something funny going on here... You had more xruns with SCHED_FIFO than the default SCHED_ISO settings, and had none with SCHED_RR. Even in the absence of the SCHED_ISO results, the other results dont make a lot of sense.

Con

P.S. If you're running on SMP it may be worth booting on UP or using cpu affinity (schedtool -a 0x1 will bind you to 1st cpu only) and see what effect that is having. There are some interesting things that can adversely affect latency on SMP.

Actually it makes perfect sense. Running non-realtime JACK threads SCHED_FIFO will do the most harm. The others less.

I predict that using normal jackd -R (without schedtool) will produce the same results running SCHED_FIFO and SCHED_ISO (within the normal variance).

I think schedtool is too blunt and instrument for making these measurements.

  joq

It's probably suffering from some of the same problems of thread granularity we saw running nice --20. It looks like you used schedtool to start jackd. IIUC, that will cause all jackd processes to run in the specified scheduling class. JACK is carefully written not to do that. Did you also use schedtool to start all the clients?

I think your puzzling discrepancies are probably due to interference from non-realtime JACK threads running at elevated priority.

I still have not found time to run Con's latest version. I've been too busy trying to hack JACK into working with nice(-20), which has turned out to be quite difficult. The interfaces we need don't work, and the ones that do are not what we need. :-(

I believe Con's latest sched patch should work fine with the normal jack_test3.2 using the -R option of jackd. That should produce more consistent results.

  joq

Isn't this going to be a showstopper? If I understand the scheduler correctly, a nice -20 task is not guaranteed to preempt a nice -19 task, if the scheduler decides that one is more CPU bound than the other and lowers its dynamic priority. The design of JACK, however, requires the higher priority threads to *always* preempt the lower ones.

Lee

The point was the application was started in a manner which would not make best use of this policy. The way it was started put everything under the same policy, and had equal performance with doing the same thing as SCHED_FIFO. So if it's a showstopper for SCHED_ISO then it is a showstopper for SCHED_FIFO. Which is, of course, not the case. The test needs to be performed again with the rt threads running SCHED_ISO, which
Jack has pointed out is trivial. Nice -n -20 on the other hand will
suffer from this problem even if only the real time thread was run at -20.

Cheers,
Con

Hi Con

I just played with your SCHED_ISO patch and found a simple way to crash my machine.

I'm running this as unprivileged user with SCHED_ISO:

main ()

{
        while(1) {
                sched_yield();
        }
}

The system hangs about 3s and then reboots itself. 2.6.11-rc1 + 2.6.11-rc1-iso-0501182249 on an UP Athlon XP.

With real SCHED_FIFO it just lockup the system.

utz

Thanks I'll look into it.

Cheers,
Con

Just as a headsup there is a new cleaned up patch here:
http://ck.kolivas.org/patches/SCHED_ISO/2.6.11-rc1-iso-0501192240.diff

The yield bug is still eluding my capture but appears to be related to the array switch of yielding and rescheduling as ISO after the fact. Still working on it.

Cheers,
Con

I ran some jack_test3.2 runs with this, using all the default settings. The results of three runs differ quite significantly for no obvious reason. I can't figure out why the DSP load should vary so much.

These may be bogus results. It looks like a libjack bug sometimes causes clients to crash when deactivating. I will investigate more tomorrow, and come up with a fix.

For comparison, I also made a couple of runs using the realtime-lsm to grant SCHED_FIFO privileges. There was some variablility, but nowhere near as much (and no crashes). I used schedtool to verify that the jackd threads actually have the expected scheduler type.

============================================
Unprivileged, realtime threads are SCHED_ISO
============================================

*** Terminated Tue Jan 18 23:54:55 CST 2005 ***
************* SUMMARY RESULT ****************

Total seconds ran . . . . . . :   300
Number of clients . . . . . . :    20
Ports per client  . . . . . . :     4
Frames per buffer . . . . . . :    64
*********************************************
Timeout Count . . . . . . . . :(    3)          (   14)         (    2)        
XRUN Count  . . . . . . . . . :    10               42               3         
Delay Count (>spare time) . . :     1                0               0         
Delay Count (>1000 usecs) . . :     0                0               0         
Delay Maximum . . . . . . . . : 307419   usecs    6492   usecs   19339   usecs 
Cycle Maximum . . . . . . . . :   858   usecs      866   usecs     860   usecs 
Average DSP Load. . . . . . . :    37.3 %           14.5 %          37.7 %     
Average CPU System Load . . . :    10.2 %            4.5 %          10.0 %     
Average CPU User Load . . . . :    26.6 %           11.4 %          23.8 %     
Average CPU Nice Load . . . . :     0.0 %            0.0 %           0.0 %     
Average CPU I/O Wait Load . . :     2.0 %            0.7 %           0.2 %     
Average CPU IRQ Load  . . . . :     0.8 %            0.7 %           0.7 %     
Average CPU Soft-IRQ Load . . :     0.0 %            0.0 %           0.0 %     
Average Interrupt Rate  . . . :  1730.3 /sec      1695.5 /sec     1694.8 /sec  
Average Context-Switch Rate . : 11523.1 /sec      6151.1 /sec    11672.2 /sec  

*********************************************

==================================================
With CAP_SYS_NICE, realtime threads are SCHED_FIFO
==================================================

*** Terminated Tue Jan 18 23:41:42 CST 2005 ***
************* SUMMARY RESULT ****************

Total seconds ran . . . . . . :   300
Number of clients . . . . . . :    20
Ports per client  . . . . . . :     4
Frames per buffer . . . . . . :    64
*********************************************
Timeout Count . . . . . . . . :(    0)          (    0)        
XRUN Count  . . . . . . . . . :     0                0         
Delay Count (>spare time) . . :     0                0         
Delay Count (>1000 usecs) . . :     0                0         
Delay Maximum . . . . . . . . :   331   usecs      201   usecs 
Cycle Maximum . . . . . . . . :   882   usecs     1017   usecs 
Average DSP Load. . . . . . . :    40.7 %           41.7 %     
Average CPU System Load . . . :    11.1 %           10.9 %     
Average CPU User Load . . . . :    26.7 %           27.7 %     
Average CPU Nice Load . . . . :     0.0 %            0.0 %     
Average CPU I/O Wait Load . . :     0.6 %            1.0 %     
Average CPU IRQ Load  . . . . :     0.7 %            0.7 %     
Average CPU Soft-IRQ Load . . :     0.0 %            0.0 %     
Average Interrupt Rate  . . . :  1708.0 /sec      1697.1 /sec  
Average Context-Switch Rate . : 13297.0 /sec     13314.8 /sec  

*********************************************

  joq

Thanks for those. If you don't know what to make of the dsp variation and the crashing then I'm not sure what I should make of it either. It's highly likely that my code still needs fixing to ensure it behaves as expected. Already one bug has been picked up in testing with respect to yield() so there may be others. By design, if you set iso_cpu to 100 it should be as good as SCHED_RR. If not, then the implementation is still buggy.

Cheers,
Con

I fixed that bug in libjack, eliminating the crashes on disconnect.

They must have been perturbing the numbers quite a bit. Here are three more runs (all SCHED_ISO). The results are much more consistent. The only significant anomaly I see now is that small Delay Max in the first run.

*** Terminated Wed Jan 19 01:04:55 CST 2005 ***
************* SUMMARY RESULT ****************

Total seconds ran . . . . . . :   300
Number of clients . . . . . . :    20
Ports per client  . . . . . . :     4
Frames per buffer . . . . . . :    64
*********************************************
Timeout Count . . . . . . . . :(    1)          (    5)         (    3)         
XRUN Count  . . . . . . . . . :     2               16              15          
Delay Count (>spare time) . . :     0                0               0          
Delay Count (>1000 usecs) . . :     0                0               0          
Delay Maximum . . . . . . . . : 11932   usecs    101053  usecs   98719   usecs  
Cycle Maximum . . . . . . . . :   868   usecs     1099   usecs     887   usecs  
Average DSP Load. . . . . . . :    37.9 %           33.6 %          36.0 %      
Average CPU System Load . . . :    10.2 %            9.0 %           9.9 %      
Average CPU User Load . . . . :    24.5 %           22.7 %          23.0 %      
Average CPU Nice Load . . . . :     0.0 %            0.0 %           0.0 %      
Average CPU I/O Wait Load . . :     0.6 %            3.4 %           0.4 %      
Average CPU IRQ Load  . . . . :     0.7 %            0.7 %           0.7 %      
Average CPU Soft-IRQ Load . . :     0.0 %            0.0 %           0.0 %      
Average Interrupt Rate  . . . :  1688.1 /sec      1696.1 /sec     1685.2 /sec   
Average Context-Switch Rate . : 11727.9 /sec     10642.4 /sec    11568.3 /sec

*********************************************

I should probably experiment with higher thresholds on the SCHED_ISO class.

  joq

I installed a fresh jack installation and got the test suite. I tried running the test suite and found it only ran to completion if I changed the run time right down to 30 seconds from 300. Otherwise it bombed out almost instantly at the default of 300. I don't know if that helps you debug the problem or not but it might be worth mentioning.

As for my own results I gave it a run on the weak SCHED_ISO implementation in 2.6.10-ck5 (P4HT 3.06):

SCHED_NORMAL:
*********************************************
Timeout Count . . . . . . . . :( 0)
XRUN Count . . . . . . . . . : 74

Delay Count (>spare time) . . :     0
Delay Count (>1000 usecs) . . :     0
Delay Maximum . . . . . . . . :     0   usecs
Cycle Maximum . . . . . . . . :  1046   usecs
Average DSP Load. . . . . . . :    18.0 %
Average CPU System Load . . . :     2.5 %
Average CPU User Load . . . . :     7.8 %
Average CPU Nice Load . . . . :     0.1 %
Average CPU I/O Wait Load . . :     0.1 %
Average CPU IRQ Load  . . . . :     0.1 %
Average CPU Soft-IRQ Load . . :     0.0 %
Average Interrupt Rate  . . . :  1776.0 /sec
Average Context-Switch Rate . : 10290.4 /sec

*********************************************

SCHED_NORMAL nice -n -20:
*********************************************
Timeout Count . . . . . . . . :( 0)
XRUN Count . . . . . . . . . : 266

Delay Count (>spare time) . . :     0
Delay Count (>1000 usecs) . . :     0
Delay Maximum . . . . . . . . :     0   usecs
Cycle Maximum . . . . . . . . :  2239   usecs
Average DSP Load. . . . . . . :    28.6 %
Average CPU System Load . . . :     2.9 %
Average CPU User Load . . . . :    10.2 %
Average CPU Nice Load . . . . :     0.0 %
Average CPU I/O Wait Load . . :     1.0 %
Average CPU IRQ Load  . . . . :     0.2 %
Average CPU Soft-IRQ Load . . :     0.1 %
Average Interrupt Rate  . . . :  2049.7 /sec
Average Context-Switch Rate . : 10145.1 /sec

*********************************************

SCHED_ISO:
*********************************************

Timeout Count . . . . . . . . :(    0)
XRUN Count  . . . . . . . . . :     1
Delay Count (>spare time) . . :     0
Delay Count (>1000 usecs) . . :     0
Delay Maximum . . . . . . . . :     0   usecs
Cycle Maximum . . . . . . . . :   687   usecs
Average DSP Load. . . . . . . :    19.9 %
Average CPU System Load . . . :     2.6 %
Average CPU User Load . . . . :    10.3 %
Average CPU Nice Load . . . . :     0.0 %
Average CPU I/O Wait Load . . :     0.0 %
Average CPU IRQ Load  . . . . :     0.2 %
Average CPU Soft-IRQ Load . . :     0.3 %
Average Interrupt Rate  . . . :  2166.2 /sec
Average Context-Switch Rate . : 10117.3 /sec

*********************************************

SCHED_FIFO:
*********************************************

Timeout Count . . . . . . . . :(    0)
XRUN Count  . . . . . . . . . :     2
Delay Count (>spare time) . . :     0
Delay Count (>1000 usecs) . . :     0
Delay Maximum . . . . . . . . :     0   usecs
Cycle Maximum . . . . . . . . :   544   usecs
Average DSP Load. . . . . . . :    19.5 %
Average CPU System Load . . . :     3.1 %
Average CPU User Load . . . . :    12.6 %
Average CPU Nice Load . . . . :     0.0 %
Average CPU I/O Wait Load . . :     0.0 %
Average CPU IRQ Load  . . . . :     1.0 %
Average CPU Soft-IRQ Load . . :     1.1 %
Average Interrupt Rate  . . . :  5018.4 /sec
Average Context-Switch Rate . : 10902.5 /sec

*********************************************

It occasionally would segfault on client exit as well (as you've already mentioned). I think we're still in the dark here to be honest.

Con

[including Rui Nuno Capela in cc: list]

Here are a couple of bug fixes for the test package. I still had not gotten them to Rui (the test author). He has created several newer versions, but I'm still using this modified jack_test3.2, so the number comparisons will continue to make sense.

@@ -13,20 +13,25 @@
 unsigned int seconds_to_run = 60;
 unsigned int num_of_ports   = 4;
 
+/* mix all the input ports to each output port */
 int process(jack_nframes_t frames, void *arg) 
 {
 	// std::cout << "process callback" << std::endl;
 	jack_default_audio_sample_t *ibuf;
 	jack_default_audio_sample_t *obuf;
 	for (int i = 0; i < num_of_ports; i++) {
-		obuf = (jack_default_audio_sample_t*) jack_port_get_buffer(oports[i], frames);
+		obuf = (jack_default_audio_sample_t*)
+			jack_port_get_buffer(oports[i], frames);
 		for (int j = 0; j < num_of_ports; j++) {
-			ibuf = (jack_default_audio_sample_t*) jack_port_get_buffer(iports[j], frames);
-			for (jack_nframes_t frame = 0; frame < frames; frame++) {
+			ibuf = (jack_default_audio_sample_t*)
+				jack_port_get_buffer(iports[j], frames);
+			for (jack_nframes_t frame = 0;
+			     frame < frames; frame++) {
 				if (j == 0)
 					obuf[frame] = 0.0; 
 				if (ibuf[frame] < -1E-6 || ibuf[frame] > +1E-6)
-					obuf[frame] += ibuf[frame] / (float) num_of_ports;
+					obuf[frame] += ibuf[frame]
+						/ (float) num_of_ports;
 			}
 		}
 	}

@@ -68,10 +73,23 @@
 	for (int i = 0; i < num_of_ports; i++) {
 		std::stringstream iport_name;
 		iport_name << "in_" << i;
-		iports[i] = jack_port_register(client,  iport_name.str().c_str(), JACK_DEFAULT_AUDIO_TYPE, JackPortIsInput|JackPortIsTerminal, 0);
+		iports[i] =
+			jack_port_register(client,
+					   iport_name.str().c_str(),
+					   JACK_DEFAULT_AUDIO_TYPE,
+					   JackPortIsInput|JackPortIsTerminal,
+					   0);
 		std::stringstream oport_name;
 		oport_name << "out_" << i;
-		oports[i] = jack_port_register(client, oport_name.str().c_str(), JACK_DEFAULT_AUDIO_TYPE, JackPortIsTerminal|JackPortIsOutput, 0);
+		oports[i] = jack_port_register(client,
+					       oport_name.str().c_str(),
+					       JACK_DEFAULT_AUDIO_TYPE,
+					       JackPortIsTerminal|JackPortIsOutput,
+					       0);
+		if ((iports[i] == NULL) || (oports[i] == NULL)) {
+			std::cout << "port registration failed" << std::endl;
+			goto exit;
+		}
 	}
 	std::cout << "set_process_callback" << std::endl;
 	jack_set_process_callback(client, process, 0);

@@ -85,6 +103,8 @@
 	sleep(seconds_to_run);
 
 	jack_deactivate(client);
+
+exit:
 	jack_client_close(client);
 	
 	delete [] iports;

@@ -21,7 +21,9 @@
 //==============
 #define NL "\n"
 typedef unsigned long long ullong;
+#ifndef __USE_MISC
 typedef unsigned long ulong;
+#endif
 
 typedef ulong sample_t;
 

@@ -6,12 +6,14 @@
 CLIENTS=$2
 PORTS=$3
 PERIOD=$4
+PLYBK_PORTS=$5
 
 # Parameter defaults.
 [ -z "${SECS}"    ] && SECS=300
 [ -z "${CLIENTS}" ] && CLIENTS=20
 [ -z "${PORTS}"   ] && PORTS=4
 [ -z "${PERIOD}"  ] && PERIOD=64
+[ -z "${PLYBK_PORTS}"  ] && PLYBK_PORTS=32
 
 # Local tools must be on same directory...
 BASEDIR=`dirname $0`

@@ -24,13 +26,13 @@
 NMETER_ARGS="t c i x b m"
 
 # jackd configuration.
-JACKD="/usr/bin/jackd"
+JACKD="jackd"
 JACKD_DRIVER="alsa"
 JACKD_DEVICE="hw:0"
 JACKD_PRIO=60
 JACKD_RATE=44100
-JACKD_PORTS=$(((${CLIENTS} + 1) * ${PORTS} * 2))
-JACKD_ARGS="-v -R -P${JACKD_PRIO} -p${JACKD_PORTS} -d${JACKD_DRIVER} -d${JACKD_DEVICE} -r${JACKD_RATE} -p${PERIOD} -n2 -S -P"
+JACKD_PORTS=$((${CLIENTS} * ${PORTS} * 2 + ${PLYBK_PORTS}))
+JACKD_ARGS="-Rv -p${JACKD_PORTS} -d${JACKD_DRIVER} -d${JACKD_DEVICE} -r${JACKD_RATE} -p${PERIOD} -n2 -P"
 
 # client test configuration.
 CLIENT="${BASEDIR}/jack_test3_client"

@@ -85,6 +87,7 @@
 echo "Number of clients  (CLIENTS) = ${CLIENTS}"	>> ${LOG} 2>&1
 echo "Ports per client     (PORTS) = ${PORTS}"		>> ${LOG} 2>&1
 echo "Frames per buffer   (PERIOD) = ${PERIOD}"		>> ${LOG} 2>&1
+echo "Playback ports (PLYBK_PORTS) = ${PLYBK_PORTS}"	>> ${LOG} 2>&1
 echo >> ${LOG} 2>&1
 
 exec_log "uname -a"

@@ -102,7 +105,7 @@
 ilist_log 
 
 #
-# Lauch nmeter in the background...
+# Launch nmeter in the background...
 #
 echo -n "Launching `basename ${NMETER}`..."
 (${NMETER} ${NMETER_ARGS} >> ${LOG} 2>&1) &

@@ -110,7 +113,7 @@
 echo "done."
 sleep 1
 
-# Lauch the test client suite...
+# Launch the test client suite...
 SLEEP=6
 echo -n "Launching ${CLIENTS} ${CLIENT}(s) instance(s)..."
 for NUM in `seq 1 ${CLIENTS}`; do

@@ -129,7 +132,7 @@
 (sleep ${SLEEP}; killall `basename ${JACKD}`) &
 
 #
-# Lauch jackd and wait for it...
+# Launch jackd and wait for it...
 #
 echo -n "Running `basename ${JACKD}`..."
 exec_log ${JACKD} ${JACKD_ARGS}

@@ -149,7 +152,7 @@
 sleep 1
 sync
 
-# Summary log analynis...
+# Summary log analysis...
 cat ${LOG} | awk -f ${BASEDIR}/jack_test3_summary.awk | tee -a ${LOG}
 
 # Finally, plot log output...

@@ -1,10 +1,10 @@
 all:	jack_test3_nmeter jack_test3_client
 
 jack_test3_nmeter:	jack_test3_nmeter.c
-	gcc -o jack_test3_nmeter jack_test3_nmeter.c
+	$(CC) -g -o jack_test3_nmeter jack_test3_nmeter.c
 
 jack_test3_client:	jack_test3_client.cpp
-	g++ -o jack_test3_client jack_test3_client.cpp -ljack
+	$(CXX) -g -o jack_test3_client jack_test3_client.cpp -ljack
 
 clean:
 	rm -vf jack_test3_nmeter jack_test3_client

For completeness, here's Rui's unmodified test package against which these diff's were generated.

There's also a modified script for running with nice --20, called jack_test3_nice.sh. I hacked it for Ingo's tests. It really should be integrated into the main jack_test3_run.sh as an option, but I have not bothered. I think we're beyond that at this point, anyway. Mainly, it proves that per-thread granularity is essential for making this stuff work.

#!/bin/sh
#

# Command line arguments.
SECS=$1
CLIENTS=$2
PORTS=$3
PERIOD=$4
PLYBK_PORTS=$5

# Parameter defaults.
[ -z "${SECS}" ] && SECS=300
[ -z "${CLIENTS}" ] && CLIENTS=20
[ -z "${PORTS}" ] && PORTS=4
[ -z "${PERIOD}" ] && PERIOD=64
[ -z "${PLYBK_PORTS}" ] && PLYBK_PORTS=32

# Local tools must be on same directory...
BASEDIR=dirname $0

# Make sure our local tools are in place.
(cd ${BASEDIR}; make all > /dev/null) || exit 1

# nmeter configuration.
NMETER="${BASEDIR}/jack_test3_nmeter"
NMETER_ARGS="t c i x b m"

# jackd configuration.
JACKD="jackd"
JACKD_DRIVER="alsa"
JACKD_DEVICE="hw:0"
JACKD_PRIO=60
JACKD_RATE=44100
JACKD_PORTS=$((${CLIENTS} * ${PORTS} * 2 + ${PLYBK_PORTS}))
JACKD_ARGS="-v -p${JACKD_PORTS} -d${JACKD_DRIVER} -d${JACKD_DEVICE} -r${JACKD_RATE} -p${PERIOD} -n2 -P"

# client test configuration.
CLIENT="${BASEDIR}/jack_test3_client"
CLIENT_ARGS="${SECS} ${PORTS}"

# process/thread list configuration.
PIDOF="/sbin/pidof"
PLIST="ps -o pid,tid,class,rtprio,ni,pri,pcpu,stat,comm"

# Log filename.
LOG="jack_test3-uname -r-date +'%Y%m%d%H%M'.log"

# Command executive and logger.
exec_log ()

{
	CMD="$*"
	echo "-----------------------"		>> ${LOG} 2>&1
	echo "# ${CMD}"				>> ${LOG} 2>&1
	${CMD}					>> ${LOG} 2>&1
	echo					>> ${LOG} 2>&1
}

# Process/thread status loggers.
plist_log ()

{
	PIDS="$*"
	if [ -n "${PIDS}" ]; then
		echo "- - - - - - - - - - - -"	>> ${LOG} 2>&1
		${PLIST} -m ${PIDS}		>> ${LOG} 2>&1
		echo				>> ${LOG} 2>&1
	fi
}

# IRQ thread status loggers.
ilist_log ()

{
	echo "- - - - - - - - - - - -"	>> ${LOG} 2>&1
	${PLIST} --sort -rtprio -e \
	| egrep '(^[ ]+PID|IRQ|jack)'	>> ${LOG} 2>&1
	echo				>> ${LOG} 2>&1
}

#
# Log headings -- show some relevant info...
#
echo "*** Started date ***" >> ${LOG} 2>&1

echo >> ${LOG} 2>&1
echo "Seconds to run (SECS) = ${SECS}" >> ${LOG} 2>&1
echo "Number of clients (CLIENTS) = ${CLIENTS}" >> ${LOG} 2>&1
echo "Ports per client (PORTS) = ${PORTS}" >> ${LOG} 2>&1
echo "Frames per buffer (PERIOD) = ${PERIOD}" >> ${LOG} 2>&1
echo "Playback ports (PLYBK_PORTS) = ${PLYBK_PORTS}" >> ${LOG} 2>&1
echo >> ${LOG} 2>&1

exec_log "uname -a"

exec_log "cat /proc/asound/version"
exec_log "cat /proc/asound/cards"

#exec_log "grep . /proc/sys/kernel/*_preemption" #exec_log "grep . /proc/irq/*/*/threaded" #exec_log "grep . /sys/block/hd*/queue/max_sectors_kb"

exec_log "cat /proc/interrupts"

# This is just about to be sure...
ilist_log

#
# Launch nmeter in the background...
#
echo -n "Launching basename ${NMETER}..." (nice -15 ${NMETER} ${NMETER_ARGS} >> ${LOG} 2>&1) & sleep 2
echo "done."
sleep 1

# Launch the test client suite...
SLEEP=6
echo -n "Launching ${CLIENTS} ${CLIENT}(s) instance(s)..." for NUM in seq 1 ${CLIENTS}; do

	CLIENT_CMDLINE="${CLIENT} ${CLIENT_ARGS}"
	SLEEP=$((${SLEEP} + 3))
	(sleep ${SLEEP}; echo -n "$NUM..."; exec_log ${CLIENT_CMDLINE}) &

done
echo "done."

# Let there be some evidence of process status...
SLEEP=$((${SLEEP} + 6))
(sleep ${SLEEP}; plist_log -C basename ${JACKD}; plist_log -C basename ${CLIENT}) &

# Let jackd live for extended but limited time...
SLEEP=$((${SLEEP} + ${SECS}))
(sleep ${SLEEP}; killall basename ${JACKD}) &

#
# Launch jackd and wait for it...
#
echo -n "Running basename ${JACKD}..."
exec_log ${JACKD} ${JACKD_ARGS}
echo "done."
sleep 1

#echo -n "Killing basename ${CLIENT}..."
#killall basename ${CLIENT} && echo "OK." || echo "FAILED." #sleep 1

echo -n "Killing basename ${NMETER}..."
killall basename ${NMETER} && echo "OK." || echo "FAILED."

echo "*** Terminated date ***" >> ${LOG} 2>&1

sync
sleep 1
sync

# Summary log analynis...
cat ${LOG} | awk -f ${BASEDIR}/jack_test3_summary.awk | tee -a ${LOG}

# Finally, plot log output...
${BASEDIR}/jack_test3_plot.sh ${LOG}

What version of JACK do you have? (jackd --version)

You need a recent CVS version to get all the data collection Rui and Lee have added. Looks like you're missing that. Delay Max is an important statistic.

You need a *very* recent version (about 10 minutes ago) to fix that jack_deactivate() segfault. (see URL below)

Try again with JACK 0.99.48. It's in CVS now, but you probably need this tarball to get around the dreaded SourceForge anon CVS lag...

http://www.joq.us/jack/tarballs/jack-audio-connection-kit-0.99.48.tar.gz

The results I get with these versions are a lot more stable. But, there are still some puzzles about the scheduling. Do you distinguish different SCHED_ISO priorities?

JACK runs with three different SCHED_FIFO priorities:

(1) The main jackd audio thread uses the -P parameter. The JACK default is 10, but Rui's script sets it with -P60. I don't think the absolute value matters, but the value relative to the other JACK realtime threads probably does.

(2) The clients' process threads run at a priority one less (59).

(3) The watchdog timer thread runs at a priority 10 greater (70).

(4) LinuxThreads creates a manager thread in each process running one level higher than the highest user realtime thread priority.

  joq

Thanks it finally ran to completion. By the way the patch you sent with the test suite did not apply so I had to do it manually (booraroom..)

It was not clear whether that would be required or not. This is why testing is so critical because if you are having latency issues it wouldn't be a problem of getting cpu time since your cpu usage is well below the 70% limit.

Since I (finally) have it running at this end at last I'll do some benchmarking of my own to see how (lack of) priorities affects SCHED_ISO. If it is inadequate, it wont be too difficult to add them to the design. The problem with priorities is that once you go over the cpu limit everyone suffers equally; but that's a failsafe that you shouldn't actually hit in normal usage so it probably doesn't matter... Hmm come to think of it, it probably _is_ a good idea to implement priority support afterall.

Cheers,
Con

Oops! Sorry. I generated those by hand using some rather crude `diff -u .... >> xxx.diff' commands.

We should just add Rui's latest version to JACK CVS.

I'd be surprised if we're hitting it in this test. AFAICT, our "Average DSP Load" should approximate your CPU limit. That's running in the 30% to 40% range. Is there any way to verify this? Is your running average readable in /proc/sys/kernel somewhere?

We do need to test that the system degrades gracefully when the CPU is overloaded. That *will* happen (the dreaded P4 float denormal problems quickly come to mind). At some point, the user should be allowed to choose how much CPU to consume, possibly shutting down plugins as needed.

Hard to say for sure without trying it. These threads are dealing with a realtime cycle that is smaller than normal scheduler time slices. Getting all the work done is important. But, getting it done in the right order might make a difference for important statistics like Max Delay.

  joq