New scheduling latencies during audio playing + heavy disk I/O on various kernels
Hi,
I performed new interesting tests of the scheduling latencies while playing audio samples from RAM.
The test was mainly performed to see if Linux can provide a stable low-latency ( <20ms) audio I/O enviroment, on modern machines ( PII + UDMA / SCSI drives)
for realtime applications like MIDI software synthesizers, audio FX, harddisk recording .. , DURING HIGH DISK ACTIVITY, at applcation level.
Unfortunately the answer is NO, with current kernels.
The benchmark is scheduled with SCHED_FIFO , with max priority. ( schp.sched_priority=sched_get_priority_max(SCHED_FIFO); )
My hardware: PII 400 Motherboard ASUS P2B , 256 MB RAM HD IBM EDIE 16GB UDMA
---
Just for your information: under Windows I can play sound using a *20ms* buffer and do a copy of a large file on the disk without any dropout. ( the test was performed using th software synthesizer NI Generator).
This to ensure that the high scheduling latencies during disk I/O are not a hardware limitation, but a Linux limitation.
I know that the Windows programs uses some hacks like IRQ programming, but I think with the right process/disk scheduling policies, we could archieve similar results under Linux using rt-scheduled user processes.
---
I wrote a new scheduling benchmark which measures the maximum scheduling latencies between two write()s to /dev/dsp , and
the number of overruns (check if the time diff is > than the audio buffer size)
The benchmark runs with several different fragmentnum/fragmentsize numbers and summarizes the results in form of a nice table.
I tested various kernel versions: 2.0.36 , 2.2.9 , 2.2.9+andrea-patch , 2.3.5
I stressed the system in 4 ways:
- running "top" with an update frequency of 0.01 sec (this is a mix of accessing to /proc filesystem and gfx output)
- writing a large file
- copying the file to a second file
- cat both files >/dev/null (to simulate disk read access)
( I did a separate test stressing my machine with "x11perf -all"
and using a 3x1024 bytes audio buffer = 17.5ms buffer ,
I let x11perf run for about 1 hour, and there were no dropouts, pretty good !)
(I know of some PCI gfx-cards, which freeze the PCI for too long time during heavy graphics activity, causing sound dropouts)
Fortunately my Matrox G100 AGP, has not such problems.
to summarize:
/proc (top) test:
2.0.36 no dropouts with buffer > 17ms 2.2.5 no dropouts with buffer > 69ms *WORSE* than 2.0.36 :-( 2.2.9 no dropouts with buffer > 23ms 2.2.9-arca no dropouts with buffer > 46ms 2.3.5 no dropouts with buffer > 46ms
disk write test:
2.0.36 seems safe with buffer > 75ms 2.2.5 seems safe with buffer > 120ms 2.2.9 seems safe with buffer > 90ms 2.2.9-arca seems safe with buffer > 50-60ms 2.3.5 seems safe with buffer > 100ms
the arca patch seems to make the disk write situation a bit better
disk copy tests:
2.0.36 seems safe with buffer > 40-50ms 2.2.5 seems safe with buffer > 70-90ms 2.2.9 seems safe with buffer > 50-70ms 2.2.9-arca seems safe with buffer > 60-70ms 2.3.5 seems safe with buffer > 60-80ms
disk read test:
2.0.36 seems safe with buffer > 180-200ms 2.2.5 seems safe with buffer > 100-140ms 2.2.9 seems safe with buffer > 90-120ms 2.2.9-arca seems safe with buffer > 90-120ms 2.3.5 seems safe with buffer > 100-115ms
Detailed tables are included in the benchmark tgz (link below)
So it seems that must use at least 100-130ms of audio buffer, during high disk activity , to ensure that there are no sound drop-outs.
I tested even a 2.2.9 kernel with a HZ@0 patch to increase scheduling frequency , but there are no differences from a stock 2.2.9 kernel.
what helped alot is the remount of the disk in sync mode
( mount / -oremount,sync )
in this case the average scheduling latencies are very good between 16-19ms during the disk stressing tests.
Unfortunately the disk performance sucks, the file operations are several times slower than in async mode, therefore we can not use the sync mode in real world.
Do you think that there exists a way to get these low scheduling latencies, in async mode ?
I tested even the QLinux 2.2.0 , just for cursiousity, and QLinux showed the worst latencies of all kernels, about 200ms, maybe the cause of this was that the QLinux is a patch for kernel 2.2.0 , and not 2.2.9 ?
If you send me some patches, I can test these on my machine, to see if we made some progress by bringing down the scheduling latencies to the magic 20ms.
Do you think that to archieve these results, there are some modifications necessary, to the locking-scheme of the I/O layer ?
I'm not a kernel hacker but IMO the problem seems the kernel which holds some locks for too much time during heavy disk I/O, which results in a blocking of the write() call (to /dev/dsp).
Please correct me if I'm wrong.
Or could the locking be made more modular on a per device basis ? ( writing to /dev/dsp , has nothing to do with doing I/O on /dev/hd* , but it seems that some critical section locks the entire device layer for too much time >100ms))
again, correct me if I'm wrong.
Just for curiousity, has someone made similar tests under Solaris x86, just to have some numbers to compare ?
You can get my benchmark at
http://www.gardena.net/benno/linux/latencytest0.3.tgz
send me your results , comments and feedback, please !
regards,
Benno.
Re: New scheduling latencies during audio playing + heavy disk I/O
Quoted 1 line · Benno Senoner
I tested various kernel versions: 2.0.36 , 2.2.9 , 2.2.9+andrea-patch , 2.3.5
Which is exactly the andrea-patch you mentioned? I suppose the one that I posted here some time ago that will avoid too much stall during a `cp /dev/zero /tmp', right? (the one that only adds the wait_for_IO variable)
If the above is true I would be interested to get some number also for:
ftp://ftp.suse.com/pub/people/andrea/kernel-patches/2.2.10_andrea-VM5.bz2
Andrea Arcangeli
PS. Thanks for your very interesting reports :).
Quoted 2 lines
Alan, do you know if MIDI I/O or serial I/O suffer of high scheduling latencies in a similar manner like audio I/O when heavy disk I/O is performed ?
I would expect them to show similar latencies to the app in such cases when heavy disk I/O occured
Quoted 2 lines
what if the note is read from a MIDI interface connected to a standard serial port ? ( are there some disadvantages over the soundcard's own MIDI port ?)
You would want to set the low latency flags on the serial port. The serial code by default does processing 100 times a second in batches
Alan
Quoted 10 lines · Juhana Sadeharju
Quoted 5 lines
From: Benno Senoner <[email protected]>
audio FX, harddisk recording .. , DURING HIGH DISK ACTIVITY,
Unfortunately the answer is NO, with current kernels.
Your last test had an unlimited disk activity but in an audio application the disk activity is limited by number of channels times the sample rate. (We just have to make sure that the audio application is only serious disk user.)
The point is that your audio application can to control external programs.
example: assume you play on your MIDI keyboard notes which go into a linux software synthesizer which must use <20ms audio buffers so that you don't notice the the delay (between a MIDI keystroke and the output of the sound from the soundcard) while playing your song.
If for some reason disk activity occurs, (for example because GIMP is just saving your 10MB JPG picture on disk),
there is a high probability that sound dropouts may occur.
Quoted 3 lines
How about testing it too? Does it really make difference? I think it makes because your current tests seems to not be limited in any reasonable way.
Yes, if you write 150Kb/sec (in background) during audio playing, you will get fewer sound dropouts, than writing at maximum rate,
but actually the kernel give no guarantee that the RT scheduled ram audio player, will not block <20ms
Quoted 11 lines
Quoted 5 lines
what helped alot is the remount of the disk in sync mode ( mount / -oremount,sync )
in this case the average scheduling latencies are very good between 16-19ms during the disk stressing tests.Did you measure the wall clock time used to copy (whatever) the file in both cases? I have feeling that it took longer to copy the file, i.e., the same effect what is gained by limiting the disk activity. So, perhaps we reach the < 20 ms latencies just by using the disk the way audio software uses it.
in sync mode the disk really SUCKS, I don't made any benchmarks but I can say the disk is SEVERAL times slower ( >5 or so).
But I think that the reason because in sync mode you get these nice low-latency results, is that the kernel blocks for fewer time, because it writes smaller pieces at time but more often.
Quoted 3 lines
But surely a change to kernel would make it possible to limit the disk activity in favor of RT scheduling.
This is the main thing which must happen to enable Linux to run realtime audio-applications.
I think it's a very important feature, which should be included before kernel 2.4 comes out, to enable vendors to port their nice audio apps to Linux.
Alan said:
"One day in the future the audio layer has to run without the kernel lock. However by then its likely to be ALSA. Getting the SMP locking right will be fun, I dont think its worth repeating the effort for dead code (ie OSS), nor will it help you anyway as your lock understanding isnt quite right"
Alan, do you know if MIDI I/O or serial I/O suffer of high scheduling latencies in a similar manner like audio I/O when heavy disk I/O is performed ?
I ask this because a classic MIDI software synthesizer works as follows:
while(1)
{
read() MIDI note from MIDI interface
render the audio in a small buffer
write() buffer to /dev/dsp
}
therefore even if the write() to /dev/dsp has no locks anymore, the read() of the MIDI note need low latency too,
what if the note is read from a MIDI interface connected to a standard serial port ? ( are there some disadvantages over the soundcard's own MIDI port ?)
regards,
Benno.
Hi!
Quoted 5 lines
Quoted 2 lines
what if the note is read from a MIDI interface connected to a standard serial port ? ( are there some disadvantages over the soundcard's own MIDI port ?)
You would want to set the low latency flags on the serial port. The serial code by default does processing 100 times a second in batches
Well - other possibility is to set HZ to 1000 ;-).
Pavel
Signature · 2 lines
I'm really [email protected]. Look at http://195.113.31.123/~pavel. Pavel Hi! I'm a .signature virus! Copy me into your ~/.signature, please!