Preemption Latency Measurement Tool
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · Dieter Nützel
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
Right, the patch is returning the length preemption was unavailable (which is when a lock is held) in us. So it is indded 4ms.
But, I think Dieter is saying he _sees_ 0.5~1s latencies (in the form of audio skips). This is despite the 4ms locks being held.
Yes, that's the case. During dbench 16,32,40,48, etc...
You might actually be waiting on disk I/O and not blocked.
Quoted 7 lines
Quoted 2 lines
Does your audio source depend on any files (eg mp3s) and if so, could they be moved to a ramfs? Do the skips go away then?
Good point.
I've copied one video (MP2) and one Ogg-Vorbis file into
/dev/shm. Little bit better but hiccup still there :-(
Is anything else freezing up? Do you see your mouse stop moving, for instance? Do other apps stop getting scheduled (eg, ico)?
You might try stracing artsd to see if it hangs at a particular syscall. Use -tt or -r for timestamps and pipe the output through tee (to a file on your ramfs).
--
"Love the dolphins," she advised him. "Write by W.A.S.T.E.."
Quoted 19 lines · Oliver Xymoron
Quoted 10 lines · Dieter Nützel
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
Right, the patch is returning the length preemption was unavailable (which is when a lock is held) in us. So it is indded 4ms.
But, I think Dieter is saying he _sees_ 0.5~1s latencies (in the form of audio skips). This is despite the 4ms locks being held.
Yes, that's the case. During dbench 16,32,40,48, etc...
You might actually be waiting on disk I/O and not blocked.
Quoted 7 lines
Quoted 2 lines
Does your audio source depend on any files (eg mp3s) and if so, could they be moved to a ramfs? Do the skips go away then?
Good point.
I've copied one video (MP2) and one Ogg-Vorbis file into
/dev/shm. Little bit better but hiccup still there :-(
Quoted 6 lines
Is anything else freezing up? Do you see your mouse stop moving, for instance? Do other apps stop getting scheduled (eg, ico)?
You might try stracing artsd to see if it hangs at a particular syscall. Use -tt or -r for timestamps and pipe the output through tee (to a file on your ramfs).
Hi,
I tried playing a mp3 with noatun via artsd. Starting dbench 32 I get the same kind of dropouts - and no indication with my latency profiling patch => no process with higher prio is waiting to run!
One of noatun or artsd is waiting for something else! (That is why I included Stefan Westerfeld... artsd)
I noticed very nice improvement then reniceing (all) artsd and noatun. (I did also change the buffer size in artsd down to 40 ms)
(This part most for Stefan...
So I thought - lets try to run artsd with RT prio - changed the option HOW can it get RT prio when it is not suid? I guess it can not...
So I manually added suid bit - but then noatun could not connect with artsd... bug?, backed out the suid change... but is behaves as it works, could be that it has so short bursts that prio never get a chance to drop)
I have been doing some stracing but on noatun. It does some strange things... It does some non blocking IO...
Then I tried to run Bennos latencytest, with generated output (sine). pages locked, SCHED_FIFO prio - killed artsd before testing... (so it won't do secondary buffering)
Result - did only get some very short skips.
Let artsd start - tried again after artsd had started.
Still getting very good results - I have a SB Live (kernel), it can mix on board...
am I using that or do the sound pass artsd in all cases???
My conclusion is that it is noatun that is waiting for something. Maybe the priorities can be the cause:
noatun runs => priority goes down over time (increased when all running processes has zero prio)
many dbench some with more prio than noatun => blocks the bursty, but CPU consuming, noatun
=> dropouts...
Look at top - one artsd stays on top (of top). But noatun drops... dbenches comes and goes in between...
If this analysis is correct:
We really need to run RT processes with RT priorities!
It is also possible that multimedia applications needs to be rewritten to this new situation, i.e. remove some workarounds...
/RogerL
Signature · 3 lines
Roger Larsson Skellefteå Sweden
Quoted 5 lines
If this analysis is correct:
We really need to run RT processes with RT priorities!It is also possible that multimedia applications needs to be rewritten = to
I dont believe this is an application problem. Applications allocating memory can end up doing page outs for other people. Its really important they dont get stuck doing a huge amount of pageout work for someone else. Thats one thing I seem to be seeing with the 10pre11 VM.
Sound cards have a lot of buffering, we are talking 64-128Kbytes + on card buffers. Thats 0.25-0.5 seconds at 48Khz 16bit stereo
Alan
Quoted 6 lines · Alan Cox
Quoted 5 lines
If this analysis is correct:
We really need to run RT processes with RT priorities!It is also possible that multimedia applications needs to be rewritten = to
Quoted 8 lines
I dont believe this is an application problem. Applications allocating memory can end up doing page outs for other people. Its really important they dont get stuck doing a huge amount of pageout work for someone else. Thats one thing I seem to be seeing with the 10pre11 VM.
Sound cards have a lot of buffering, we are talking 64-128Kbytes + on card buffers. Thats 0.25-0.5 seconds at 48Khz 16bit stereo
Hmm..
The yield we do in __alloc_pages is it really correct?
1. Suppose it is not we that are the problem.
2. Suppose there are lots of other processes that gets to run.
3. If kswapd had higher prio, it would start to run without us giving up...
I have to try...
/RogerL
Signature · 3 lines
Roger Larsson Skellefteå Sweden
Quoted 15 lines · Roger Larsson
Quoted 6 lines · Alan Cox
Quoted 5 lines
If this analysis is correct:
We really need to run RT processes with RT priorities!It is also possible that multimedia applications needs to be rewritten = to
Quoted 7 lines
I dont believe this is an application problem. Applications allocating memory can end up doing page outs for other people. Its really important they dont get stuck doing a huge amount of pageout work for someone else. Thats one thing I seem to be seeing with the 10pre11 VM.
Sound cards have a lot of buffering, we are talking 64-128Kbytes + on card buffers. Thats 0.25-0.5 seconds at 48Khz 16bit stereo
Quoted 9 lines
Hmm..
The yield we do in
__alloc_pagesis it really correct?
1. Suppose it is not we that are the problem.
2. Suppose there are lots of other processes that gets to run.
3. If kswapd had higher prio, it would start to run without us giving up...I have to try...
They were already gone in -pre12!
But I was running -pre10...
Others?
Dieter is running -pre12...
I will test it anyways...
/RogerL
Signature · 3 lines
Roger Larsson Skellefteå Sweden
Continuing to follow up to myself...
Quoted 9 lines
Quoted 9 lines
Hmm..
The yield we do in
__alloc_pagesis it really correct?
1. Suppose it is not we that are the problem.
2. Suppose there are lots of other processes that gets to run.
3. If kswapd had higher prio, it would start to run without us giving up...
Quoted 8 lines
Quoted 1 line
I have to try...
They were already gone in -pre12!
But I was running -pre10...
Others?
Dieter is running -pre12...
I will test it anyways...
Unexpected result, but interesting.
Things got MUCH WORSE!
Quoted 1 line
From nearly no problem to hardly enjoyable.
(pre10 with preemptive kernel but no schedule in __alloc_pages,
actual if statement removed)
Maybe it is closer to what Dieter gets...
/RogerL
Signature · 3 lines
Roger Larsson Skellefteå Sweden
Quoted 6 lines · Roger Larsson
Quoted 3 lines
Quoted 3 lines
The yield we do in
__alloc_pagesis it really correct?I have to try...
Unexpected result, but interesting.
Things got MUCH WORSE!
Not at all unexpected, this thing has been tested
extensively in the 2.4.0-test* kernels.
Is it really so hard to learn from the past if it's more than 6 weeks ago ?
regards,
Rik
--
IA64: a worthy successor to the i860.
http://www.surriel.com/
Quoted 3 lines · Alan Cox
Sound cards have a lot of buffering, we are talking 64-128Kbytes + on card buffers. Thats 0.25-0.5 seconds at 48Khz 16bit stereo
Only "soundcards", that cheap crap like SoundBlaster. Professional lowlatency soundcards usually have something like 128-512 samples per channel for 24-bit (32-bit data) 96 kHz 8 channels...
- Jussi Laako
Signature · 2 lines
PGP key fingerprint: 161D 6FED 6A92 39E2 EB5B 39DD A4DE 63EB C216 1E4B Available at PGP keyservers
Quoted 3 lines
Only "soundcards", that cheap crap like SoundBlaster. Professional lowlatency soundcards usually have something like 128-512 samples per channel for 24-bit (32-bit data) 96 kHz 8 channels...
Then the problem space isnt interesting. Many graphics cards will hog the PCI bus for longer than that
Quoted 9 lines · Jussi Laako
Quoted 3 lines · Alan Cox
Sound cards have a lot of buffering, we are talking 64-128Kbytes + on card buffers. Thats 0.25-0.5 seconds at 48Khz 16bit stereo
Only "soundcards", that cheap crap like SoundBlaster. Professional lowlatency soundcards usually have something like 128-512 samples per channel for 24-bit (32-bit data) 96 kHz 8 channels...
This doesn't make sense. Even professional soundcards provide hundreds of kBytes of buffer space. And even with most cheap PCI cards you need not wait until the whole buffer is full before start reading samples. Even with those soundcards the latency is mostly caused by the digital filters in the codec.
Tom
Quoted 9 lines · Thomas Sailer
Quoted 3 lines
lowlatency soundcards usually have something like 128-512 samples per channel for 24-bit (32-bit data) 96 kHz 8 channels...
This doesn't make sense. Even professional soundcards provide hundreds of kBytes of buffer space. And even with most cheap PCI cards you need not wait until the whole buffer is full before start reading samples. Even with those soundcards the latency is mostly caused by the digital filters in the codec.
RME Digi 96/8 Pro has 128 kB buffer. This means 2048 samples per channel for 8 channel full-duplex operation which is 21.3 ms for 96 kHz. If interrupt is generated at 50% buffer fill then the time between interrupts is 10.7 ms.
(1)
ICEnsemble ICE1712 chip can use max 256 kB for record and playback so it's maximum latency is 56.9 ms for 96 kHz and 28.4 ms between interrupts. (2)
As a side note, from ICE's Windows control panel I can select 2-28 ms for DMA buffer size.
Those times are maximum, usually lower values are used. I use 10 ms buffer under Windows.
- Jussi Laako
In-Reply-To:
1) RME Digi96 series datasheet
2) ICEnsemble ICE1712 datasheet
Signature · 2 lines
PGP key fingerprint: 161D 6FED 6A92 39E2 EB5B 39DD A4DE 63EB C216 1E4B Available at PGP keyservers
Quoted 2 lines · Alan Cox
they dont get stuck doing a huge amount of pageout work for someone else. Thats one thing I seem to be seeing with the 10pre11 VM.
actually one feature of the 10pre11 VM is that it will avoid a task to give to other people the pages that it is freeing for itself. The previous VM didn't has such a feature. So (in theory :) it should be the other way around. see the implementation of page_alloc.c::balance_classzone().
Actually Linus found just a few minutes ago a possible source for high latencies in the pre12 VM, my silly mistake, he will certainly fix it in pre13 somehow just in case that was the problem.
Andrea
Quoted 9 lines · Andrea Arcangeli
Quoted 2 lines · Alan Cox
they dont get stuck doing a huge amount of pageout work for someone else. Thats one thing I seem to be seeing with the 10pre11 VM.
actually one feature of the 10pre11 VM is that it will avoid a task to give to other people the pages that it is freeing for itself. The previous VM didn't has such a feature. So (in theory :) it should be the other way around. see the implementation of
page_alloc.c::balance_classzone().
The key word here seems to be "in theory" ;)
Without testing the thing to death there's no way what it really does. Yes, this can make development go at a slower speed, but at least you'll end up with some idea of what's going on...
regards,
Rik
--
IA64: a worthy successor to the i860.
http://www.surriel.com/
Hi!
Quoted 14 lines · Roger Larsson
Quoted 3 lines
You might try stracing artsd to see if it hangs at a particular syscall. Use -tt or -r for timestamps and pipe the output through tee (to a file on your ramfs).
I tried playing a mp3 with noatun via artsd. Starting dbench 32 I get the same kind of dropouts - and no indication with my latency profiling patch => no process with higher prio is waiting to run!
One of noatun or artsd is waiting for something else! (That is why I included Stefan Westerfeld... artsd)
I noticed very nice improvement then reniceing (all) artsd and noatun. (I did also change the buffer size in artsd down to 40 ms)
Quoted 7 lines
(This part most for Stefan...
So I thought - lets try to run artsd with RT prio - changed the option HOW can it get RT prio when it is not suid? I guess it can not...So I manually added suid bit - but then noatun could not connect with artsd... bug?, backed out the suid change... but is behaves as it works, could be that it has so short bursts that prio never get a chance to drop)
If you want to run artsd with RT prio, add suid root to artswrapper if it is not already there - it should by default, but some distributors/packagers don't get this right. If you start artsd via kcminit, check the realtime checkbox, if you do manually, run artswrapper instead of artsd, i.e.
stefan@voyager:~/src/kdelibs/arts$ artswrapper -F10 -S4096
Quoted 1 line
Quoted 1 line
running as realtime process now (priority 50)
Artsd itself will not acquire RT prio, only artswrapper will. Noatun doesn't contain time critical code, so there it doesn't matter.
Cu... Stefan
Signature · 2 lines
-* Stefan Westerfeld, [email protected] (PGP!), Hamburg/Germany KDE Developer, project infos at http://space.twc.de/~stefan/kde *-
Am Freitag, 21. September 2001 18:18 schrieb Stefan Westerfeld:
Quoted 26 lines
Hi!Quoted 14 lines · Roger Larsson
Quoted 3 lines
You might try stracing artsd to see if it hangs at a particular syscall. Use -tt or -r for timestamps and pipe the output through tee (to a file on your ramfs).
I tried playing a mp3 with noatun via artsd. Starting dbench 32 I get the same kind of dropouts - and no indication with my latency profiling patch => no process with higher prio is waiting to run!
One of noatun or artsd is waiting for something else! (That is why I included Stefan Westerfeld... artsd)
I noticed very nice improvement then reniceing (all) artsd and noatun. (I did also change the buffer size in artsd down to 40 ms)
Quoted 8 lines
(This part most for Stefan...
So I thought - lets try to run artsd with RT prio - changed the option HOW can it get RT prio when it is not suid? I guess it can not...So I manually added suid bit - but then noatun could not connect with artsd... bug?, backed out the suid change... but is behaves as it works, could be that it has so short bursts that prio never get a chance to drop)
Quoted 5 lines
If you want to run artsd with RT prio, add suid root to artswrapper if it is not already there - it should by default, but some distributors/packagers don't get this right. If you start artsd via kcminit, check the realtime checkbox, if you do manually, run artswrapper instead of artsd, i.e.
Thank you very much Stefan!
I've an updated SuSE 7.1 (KDE-2.2.1, etc.) apart from my kernel and XFree86 DRI hacking running, here.
SuSE missed the suid root on artswrapper (security). I've changed it.
-rwxr-xr-x 1 root root 211892 Sep 15 22:59/opt/kde2/bin/artsbuilder
-rwxr-xr-x 1 root root 690 Sep 15 20:19/opt/kde2/bin/artsc-config
-rwxr-xr-x 1 root root 30388 Sep 15 20:48 /opt/kde2/bin/artscat -rwxr-xr-x 1 root root 157576 Sep 15 23:00 /opt/kde2/bin/artscontrol -rwxr-xr-x 1 root root 125040 Sep 15 20:48 /opt/kde2/bin/artsd -rwxr-xr-x 1 root root 2271 Sep 15 20:19 /opt/kde2/bin/artsdsp -rwxr-xr-x 1 root root 8568 Sep 15 20:50 /opt/kde2/bin/artsmessage -rwxr-xr-x 1 root root 17924 Sep 15 20:48 /opt/kde2/bin/artsplay -rwxr-xr-x 1 root root 30396 Sep 15 20:49
/opt/kde2/bin/artsshell
-rwsr-xr-x 1 root root 4452 Sep 15 20:48/opt/kde2/bin/artswrapper
After that I've tested, again.
This time 2.4.10-pre10 + patch-rml-2.4.10-pre12-preempt-kernel-1 + patch-rml-2.4.10-pre12-preempt-stats-1 + journal.c (from Chris Mason)
dbench 32
Throughput 26.485 MB/sec (NB=33.1062 MB/sec 264.85 MBit/sec)
15.070u 60.340s 2:40.50 46.9% 0+0k 0+0io 911pf+0w
max load: 2936
Artsd daemon wait in "wait_on_b" (WCHAN), D state (STAT) during the hiccup (for nearly 3 seconds).
The hiccup appears at the beginning of dbench (after 9-10 seconds). I've noticed no other hiccup.
snapshot
9:33pm up 2:03, 1 user, load average: 27.28, 10.53, 5.87
122 processes: 112 sleeping, 10 running, 0 zombie, 0 stopped CPU states: 15.4% user, 43.2% system, 0.0% nice, 41.3% idle
Mem: 642840K av, 607640K used, 35200K free, 0K shrd, 20244K buff
Swap: 1028120K av, 196K used, 1027924K free 465668K
cached
PID USER PRI NI PAGEIN SIZE SWAP RSS SHARE WCHAN STAT %CPU %MEM
TIME COMMA
7529 nuetzel 20 0 1274 7128 0 7128 4252 do_select S 2.7 1.1
0:05 artsd
7682 nuetzel 20 0 17 7128 0 7128 4252 rt_sigsus S 2.7 1.1
0:05 artsd
7725 nuetzel 18 0 25 524 0 524 400 wait_on_b D 1.9 0.0
0:01 dbench
7726 nuetzel 19 0 25 524 0 524 400 do_journa D 1.9 0.0
0:01 dbench
7727 nuetzel 18 0 25 524 0 524 400 wait_on_b D 1.9 0.0
0:01 dbench
7724 nuetzel 18 0 25 524 0 524 400 do_journa D 1.7 0.0
0:01 dbench
7729 nuetzel 18 0 25 524 0 524 400 wait_on_b D 1.7 0.0
0:01 dbench
7730 nuetzel 18 0 25 524 0 524 400 do_journa D 1.7 0.0
0:01 dbench
7737 nuetzel 18 0 25 524 0 524 400 do_journa D 1.7 0.0
0:01 dbench
7738 nuetzel 18 0 25 524 0 524 400 R 1.7 0.0
0:01 dbench
7731 nuetzel 18 0 25 524 0 524 400 wait_on_b D 1.5 0.0
0:01 dbench
7732 nuetzel 18 0 25 524 0 524 400 wait_on_b D 1.5 0.0
0:01 dbench
7733 nuetzel 18 0 25 524 0 524 400 do_journa D 1.5 0.0
0:01 dbench
7734 nuetzel 18 0 25 524 0 524 400 do_journa D 1.5 0.0
0:01 dbench
7735 nuetzel 18 0 25 524 0 524 400 do_journa D 1.5 0.0
0:01 dbench
7736 nuetzel 18 0 25 524 0 524 400 wait_on_b D 1.5 0.0
0:01 dbench
7740 nuetzel 18 0 25 524 0 524 400 R 1.5 0.0
0:01 dbench
7741 nuetzel 18 0 25 524 0 524 400 do_journa D 1.5 0.0
0:01 dbench
7747 nuetzel 18 0 25 524 0 524 400 R 1.5 0.0
0:01 dbench
7748 nuetzel 18 0 25 524 0 524 400 do_journa D 1.5 0.0
0:01 dbench
Worst 20 latency times of 6314 measured in this period.
usec cause mask start line/file address end line/file
6927 BKL 1 1302/inode.c c016de79 1306/inode.c 6622 spin_lock 1 547/sched.c c0111b04 697/sched.c 4823 BKL 1 1302/inode.c c016de79 842/inode.c 4421 reacqBKL 1 1375/sched.c c01138b4 697/sched.c 3546 reacqBKL 1 1375/sched.c c01138b4 1873/namei.c 3302 BKL 1 1302/inode.c c016de79 52/inode.c 2651 reacqBKL 0 1375/sched.c c01138b4 1439/namei.c 2597 BKL 0 533/inode.c c016c4ed 1380/sched.c 2585 BKL 0 1302/inode.c c016de79 1381/sched.c 2517 BKL 1 1302/inode.c c016de79 697/sched.c 2473 BKL 0 30/inode.c c016b971 1380/sched.c 2442 BKL 0 1437/namei.c c014af4f 1306/inode.c 2428 reacqBKL 0 1375/sched.c c01138b4 1381/sched.c 2409 BKL 0 1302/inode.c c016de79 1380/sched.c 2294 spin_lock 0 547/sched.c c0111b04 1380/sched.c 2289 BKL 1 927/namei.c c0149ddf 697/sched.c 2235 BKL 1 30/inode.c c016b971 697/sched.c 2225 BKL 0 1437/namei.c c014af4f 697/sched.c 2158 BKL 0 533/inode.c c016c4ed 1381/sched.c 2136 BKL 1 1366/namei.c c014ac4c 1381/sched.c
c016de20 T reiserfs_dirty_inodec016de79 ..... <<<<<c016df10 T reiserfs_sync_inode
c0111ad0 T schedulec0111b04 ..... <<<<<c0112020 T __wake_up
c016de20 T reiserfs_dirty_inodec016de79 ..... <<<<<c016df10 T reiserfs_sync_inode
Regards,
Dieter