Skip to contentSkip to conversation

Preemption Latency Measurement Tool

PATCH Oliver Xymoron 9 participants 15 messages Sep 20, 2001 – Sep 22, 2001 1 message not in the archive

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive
Quoted 10 lines · Dieter Nützel
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines

Right, the patch is returning the length preemption was unavailable (which is when a lock is held) in us. So it is indded 4ms.

But, I think Dieter is saying he _sees_ 0.5~1s latencies (in the form of audio skips). This is despite the 4ms locks being held.

Yes, that's the case. During dbench 16,32,40,48, etc...

You might actually be waiting on disk I/O and not blocked.

Quoted 7 lines
Quoted 2 lines

Does your audio source depend on any files (eg mp3s) and if so, could they be moved to a ramfs? Do the skips go away then?

Good point.

I've copied one video (MP2) and one Ogg-Vorbis file into /dev/shm. Little bit better but hiccup still there :-(

Is anything else freezing up? Do you see your mouse stop moving, for instance? Do other apps stop getting scheduled (eg, ico)?

You might try stracing artsd to see if it hangs at a particular syscall. Use -tt or -r for timestamps and pipe the output through tee (to a file on your ramfs).

--
"Love the dolphins," she advised him. "Write by W.A.S.T.E.."

#2 lore raw
Quoted 19 lines · Oliver Xymoron
Quoted 10 lines · Dieter Nützel
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines

Right, the patch is returning the length preemption was unavailable (which is when a lock is held) in us. So it is indded 4ms.

But, I think Dieter is saying he _sees_ 0.5~1s latencies (in the form of audio skips). This is despite the 4ms locks being held.

Yes, that's the case. During dbench 16,32,40,48, etc...

You might actually be waiting on disk I/O and not blocked.

Quoted 7 lines
Quoted 2 lines

Does your audio source depend on any files (eg mp3s) and if so, could they be moved to a ramfs? Do the skips go away then?

Good point.

I've copied one video (MP2) and one Ogg-Vorbis file into /dev/shm. Little bit better but hiccup still there :-(

Quoted 6 lines

Is anything else freezing up? Do you see your mouse stop moving, for instance? Do other apps stop getting scheduled (eg, ico)?

You might try stracing artsd to see if it hangs at a particular syscall. Use -tt or -r for timestamps and pipe the output through tee (to a file on your ramfs).

Hi,

I tried playing a mp3 with noatun via artsd. Starting dbench 32 I get the same kind of dropouts - and no indication with my latency profiling patch => no process with higher prio is waiting to run!

One of noatun or artsd is waiting for something else! (That is why I included Stefan Westerfeld... artsd)

I noticed very nice improvement then reniceing (all) artsd and noatun. (I did also change the buffer size in artsd down to 40 ms)

(This part most for Stefan...
So I thought - lets try to run artsd with RT prio - changed the option HOW can it get RT prio when it is not suid? I guess it can not...

So I manually added suid bit - but then noatun could not connect with artsd... bug?, backed out the suid change... but is behaves as it works, could be that it has so short bursts that prio never get a chance to drop)

I have been doing some stracing but on noatun. It does some strange things... It does some non blocking IO...

Then I tried to run Bennos latencytest, with generated output (sine). pages locked, SCHED_FIFO prio - killed artsd before testing... (so it won't do secondary buffering)
Result - did only get some very short skips.

Let artsd start - tried again after artsd had started.

Still getting very good results - I have a SB Live (kernel), it can mix on board...
am I using that or do the sound pass artsd in all cases???

My conclusion is that it is noatun that is waiting for something. Maybe the priorities can be the cause:

noatun runs => priority goes down over time (increased when all running processes has zero prio)
many dbench some with more prio than noatun => blocks the bursty, but CPU consuming, noatun
=> dropouts...

Look at top - one artsd stays on top (of top). But noatun drops... dbenches comes and goes in between...

If this analysis is correct:
We really need to run RT processes with RT priorities!

It is also possible that multimedia applications needs to be rewritten to this new situation, i.e. remove some workarounds...

/RogerL

Signature · 3 lines
Roger Larsson
Skellefteå
Sweden
#3 lore raw
Quoted 5 lines

If this analysis is correct:
We really need to run RT processes with RT priorities!

It is also possible that multimedia applications needs to be rewritten = to

I dont believe this is an application problem. Applications allocating memory can end up doing page outs for other people. Its really important they dont get stuck doing a huge amount of pageout work for someone else. Thats one thing I seem to be seeing with the 10pre11 VM.

Sound cards have a lot of buffering, we are talking 64-128Kbytes + on card buffers. Thats 0.25-0.5 seconds at 48Khz 16bit stereo

Alan

#12 lore raw
In reply to Roger Larsson
Quoted 2 lines · Alan Cox

they dont get stuck doing a huge amount of pageout work for someone else. Thats one thing I seem to be seeing with the 10pre11 VM.

actually one feature of the 10pre11 VM is that it will avoid a task to give to other people the pages that it is freeing for itself. The previous VM didn't has such a feature. So (in theory :) it should be the other way around. see the implementation of page_alloc.c::balance_classzone().

Actually Linus found just a few minutes ago a possible source for high latencies in the pre12 VM, my silly mistake, he will certainly fix it in pre13 somehow just in case that was the problem.

Andrea

#4 lore raw
In reply to Alan Cox
Quoted 6 lines · Alan Cox
Quoted 5 lines

If this analysis is correct:
We really need to run RT processes with RT priorities!

It is also possible that multimedia applications needs to be rewritten = to

Quoted 8 lines

I dont believe this is an application problem. Applications allocating memory can end up doing page outs for other people. Its really important they dont get stuck doing a huge amount of pageout work for someone else. Thats one thing I seem to be seeing with the 10pre11 VM.

Sound cards have a lot of buffering, we are talking 64-128Kbytes + on card buffers. Thats 0.25-0.5 seconds at 48Khz 16bit stereo

Hmm..

The yield we do in __alloc_pages is it really correct?
1. Suppose it is not we that are the problem.
2. Suppose there are lots of other processes that gets to run.
3. If kswapd had higher prio, it would start to run without us giving up...

I have to try...

/RogerL

Signature · 3 lines
Roger Larsson
Skellefteå
Sweden
#13 lore raw
In reply to Andrea Arcangeli
Quoted 9 lines · Andrea Arcangeli
Quoted 2 lines · Alan Cox

they dont get stuck doing a huge amount of pageout work for someone else. Thats one thing I seem to be seeing with the 10pre11 VM.

actually one feature of the 10pre11 VM is that it will avoid a task to give to other people the pages that it is freeing for itself. The previous VM didn't has such a feature. So (in theory :) it should be the other way around. see the implementation of page_alloc.c::balance_classzone().

The key word here seems to be "in theory" ;)

Without testing the thing to death there's no way what it really does. Yes, this can make development go at a slower speed, but at least you'll end up with some idea of what's going on...

regards,

Rik
--
IA64: a worthy successor to the i860.

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/

#5 lore raw
In reply to Alan Cox
Quoted 15 lines · Roger Larsson
Quoted 6 lines · Alan Cox
Quoted 5 lines

If this analysis is correct:
We really need to run RT processes with RT priorities!

It is also possible that multimedia applications needs to be rewritten = to

Quoted 7 lines

I dont believe this is an application problem. Applications allocating memory can end up doing page outs for other people. Its really important they dont get stuck doing a huge amount of pageout work for someone else. Thats one thing I seem to be seeing with the 10pre11 VM.

Sound cards have a lot of buffering, we are talking 64-128Kbytes + on card buffers. Thats 0.25-0.5 seconds at 48Khz 16bit stereo

Quoted 9 lines

Hmm..

The yield we do in __alloc_pages is it really correct?
1. Suppose it is not we that are the problem.
2. Suppose there are lots of other processes that gets to run.
3. If kswapd had higher prio, it would start to run without us giving up...

I have to try...

They were already gone in -pre12!
But I was running -pre10...
Others?
Dieter is running -pre12...
I will test it anyways...

/RogerL

Signature · 3 lines
Roger Larsson
Skellefteå
Sweden
#6 lore raw
In reply to Alan Cox

Continuing to follow up to myself...

Quoted 9 lines
Quoted 9 lines

Hmm..

The yield we do in __alloc_pages is it really correct?
1. Suppose it is not we that are the problem.
2. Suppose there are lots of other processes that gets to run.
3. If kswapd had higher prio, it would start to run without us giving up...

Quoted 8 lines
Quoted 1 line

I have to try...

They were already gone in -pre12!
But I was running -pre10...
Others?
Dieter is running -pre12...
I will test it anyways...

Unexpected result, but interesting.
Things got MUCH WORSE!

Quoted 1 line

From nearly no problem to hardly enjoyable.

(pre10 with preemptive kernel but no schedule in __alloc_pages,
actual if statement removed)
Maybe it is closer to what Dieter gets...

/RogerL

Signature · 3 lines
Roger Larsson
Skellefteå
Sweden
#7 lore raw
Quoted 6 lines · Roger Larsson
Quoted 3 lines
Quoted 3 lines

The yield we do in __alloc_pages is it really correct?

I have to try...

Unexpected result, but interesting.
Things got MUCH WORSE!

Not at all unexpected, this thing has been tested
extensively in the 2.4.0-test* kernels.

Is it really so hard to learn from the past if it's more than 6 weeks ago ?

regards,

Rik
--
IA64: a worthy successor to the i860.

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/

#8 lore raw
In reply to Alan Cox
Quoted 3 lines · Alan Cox

Sound cards have a lot of buffering, we are talking 64-128Kbytes + on card buffers. Thats 0.25-0.5 seconds at 48Khz 16bit stereo

Only "soundcards", that cheap crap like SoundBlaster. Professional lowlatency soundcards usually have something like 128-512 samples per channel for 24-bit (32-bit data) 96 kHz 8 channels...

- Jussi Laako

Signature · 2 lines
PGP key fingerprint: 161D 6FED 6A92 39E2 EB5B  39DD A4DE 63EB C216 1E4B
Available at PGP keyservers
#14 lore raw
In reply to Oliver Xymoron

Hi!

Quoted 14 lines · Roger Larsson
Quoted 3 lines

You might try stracing artsd to see if it hangs at a particular syscall. Use -tt or -r for timestamps and pipe the output through tee (to a file on your ramfs).

I tried playing a mp3 with noatun via artsd. Starting dbench 32 I get the same kind of dropouts - and no indication with my latency profiling patch => no process with higher prio is waiting to run!

One of noatun or artsd is waiting for something else! (That is why I included Stefan Westerfeld... artsd)

I noticed very nice improvement then reniceing (all) artsd and noatun. (I did also change the buffer size in artsd down to 40 ms)

Quoted 7 lines

(This part most for Stefan...
So I thought - lets try to run artsd with RT prio - changed the option HOW can it get RT prio when it is not suid? I guess it can not...

So I manually added suid bit - but then noatun could not connect with artsd... bug?, backed out the suid change... but is behaves as it works, could be that it has so short bursts that prio never get a chance to drop)

If you want to run artsd with RT prio, add suid root to artswrapper if it is not already there - it should by default, but some distributors/packagers don't get this right. If you start artsd via kcminit, check the realtime checkbox, if you do manually, run artswrapper instead of artsd, i.e.

stefan@voyager:~/src/kdelibs/arts$ artswrapper -F10 -S4096
Quoted 1 line
Quoted 1 line

running as realtime process now (priority 50)

Artsd itself will not acquire RT prio, only artswrapper will. Noatun doesn't contain time critical code, so there it doesn't matter.

Cu... Stefan

Signature · 2 lines
  -* Stefan Westerfeld, [email protected] (PGP!), Hamburg/Germany
     KDE Developer, project infos at http://space.twc.de/~stefan/kde *-         
#9 lore raw
In reply to Jussi Laako
Quoted 3 lines

Only "soundcards", that cheap crap like SoundBlaster. Professional lowlatency soundcards usually have something like 128-512 samples per channel for 24-bit (32-bit data) 96 kHz 8 channels...

Then the problem space isnt interesting. Many graphics cards will hog the PCI bus for longer than that

#10 lore raw
In reply to Alan Cox
Quoted 9 lines · Jussi Laako
Quoted 3 lines · Alan Cox

Sound cards have a lot of buffering, we are talking 64-128Kbytes + on card buffers. Thats 0.25-0.5 seconds at 48Khz 16bit stereo

Only "soundcards", that cheap crap like SoundBlaster. Professional lowlatency soundcards usually have something like 128-512 samples per channel for 24-bit (32-bit data) 96 kHz 8 channels...

This doesn't make sense. Even professional soundcards provide hundreds of kBytes of buffer space. And even with most cheap PCI cards you need not wait until the whole buffer is full before start reading samples. Even with those soundcards the latency is mostly caused by the digital filters in the codec.

Tom

#15 lore raw
In reply to Oliver Xymoron

Am Freitag, 21. September 2001 18:18 schrieb Stefan Westerfeld:

Quoted 26 lines
   Hi!
Quoted 14 lines · Roger Larsson
Quoted 3 lines

You might try stracing artsd to see if it hangs at a particular syscall. Use -tt or -r for timestamps and pipe the output through tee (to a file on your ramfs).

I tried playing a mp3 with noatun via artsd. Starting dbench 32 I get the same kind of dropouts - and no indication with my latency profiling patch => no process with higher prio is waiting to run!

One of noatun or artsd is waiting for something else! (That is why I included Stefan Westerfeld... artsd)

I noticed very nice improvement then reniceing (all) artsd and noatun. (I did also change the buffer size in artsd down to 40 ms)

Quoted 8 lines

(This part most for Stefan...
So I thought - lets try to run artsd with RT prio - changed the option HOW can it get RT prio when it is not suid? I guess it can not...

So I manually added suid bit - but then noatun could not connect with artsd... bug?, backed out the suid change... but is behaves as it works, could be that it has so short bursts that prio never get a chance to drop)

Quoted 5 lines

If you want to run artsd with RT prio, add suid root to artswrapper if it is not already there - it should by default, but some distributors/packagers don't get this right. If you start artsd via kcminit, check the realtime checkbox, if you do manually, run artswrapper instead of artsd, i.e.

Thank you very much Stefan!

I've an updated SuSE 7.1 (KDE-2.2.1, etc.) apart from my kernel and XFree86 DRI hacking running, here.

SuSE missed the suid root on artswrapper (security). I've changed it.

-rwxr-xr-x 1 root root 211892 Sep 15 22:59
/opt/kde2/bin/artsbuilder
-rwxr-xr-x 1 root root 690 Sep 15 20:19
/opt/kde2/bin/artsc-config

-rwxr-xr-x    1 root     root        30388 Sep 15 20:48 /opt/kde2/bin/artscat
-rwxr-xr-x    1 root     root       157576 Sep 15 23:00 
/opt/kde2/bin/artscontrol
-rwxr-xr-x    1 root     root       125040 Sep 15 20:48 /opt/kde2/bin/artsd
-rwxr-xr-x    1 root     root         2271 Sep 15 20:19 /opt/kde2/bin/artsdsp
-rwxr-xr-x    1 root     root         8568 Sep 15 20:50 
/opt/kde2/bin/artsmessage
-rwxr-xr-x    1 root     root        17924 Sep 15 20:48 /opt/kde2/bin/artsplay
-rwxr-xr-x    1 root     root        30396 Sep 15 20:49 

/opt/kde2/bin/artsshell
-rwsr-xr-x 1 root root 4452 Sep 15 20:48
/opt/kde2/bin/artswrapper

After that I've tested, again.

This time 2.4.10-pre10 + patch-rml-2.4.10-pre12-preempt-kernel-1 + patch-rml-2.4.10-pre12-preempt-stats-1 + journal.c (from Chris Mason)

dbench 32
Throughput 26.485 MB/sec (NB=33.1062 MB/sec 264.85 MBit/sec)
15.070u 60.340s 2:40.50 46.9% 0+0k 0+0io 911pf+0w
max load: 2936

Artsd daemon wait in "wait_on_b" (WCHAN), D state (STAT) during the hiccup (for nearly 3 seconds).
The hiccup appears at the beginning of dbench (after 9-10 seconds). I've noticed no other hiccup.

snapshot

9:33pm up 2:03, 1 user, load average: 27.28, 10.53, 5.87
122 processes: 112 sleeping, 10 running, 0 zombie, 0 stopped CPU states: 15.4% user, 43.2% system, 0.0% nice, 41.3% idle
Mem: 642840K av, 607640K used, 35200K free, 0K shrd, 20244K buff
Swap: 1028120K av, 196K used, 1027924K free 465668K
cached

PID USER PRI NI PAGEIN SIZE SWAP RSS SHARE WCHAN STAT %CPU %MEM
TIME COMMA
7529 nuetzel 20 0 1274 7128 0 7128 4252 do_select S 2.7 1.1
0:05 artsd
7682 nuetzel 20 0 17 7128 0 7128 4252 rt_sigsus S 2.7 1.1
0:05 artsd
7725 nuetzel 18 0 25 524 0 524 400 wait_on_b D 1.9 0.0
0:01 dbench
7726 nuetzel 19 0 25 524 0 524 400 do_journa D 1.9 0.0
0:01 dbench
7727 nuetzel 18 0 25 524 0 524 400 wait_on_b D 1.9 0.0
0:01 dbench
7724 nuetzel 18 0 25 524 0 524 400 do_journa D 1.7 0.0
0:01 dbench
7729 nuetzel 18 0 25 524 0 524 400 wait_on_b D 1.7 0.0
0:01 dbench
7730 nuetzel 18 0 25 524 0 524 400 do_journa D 1.7 0.0
0:01 dbench
7737 nuetzel 18 0 25 524 0 524 400 do_journa D 1.7 0.0
0:01 dbench
7738 nuetzel 18 0 25 524 0 524 400 R 1.7 0.0
0:01 dbench
7731 nuetzel 18 0 25 524 0 524 400 wait_on_b D 1.5 0.0
0:01 dbench
7732 nuetzel 18 0 25 524 0 524 400 wait_on_b D 1.5 0.0
0:01 dbench
7733 nuetzel 18 0 25 524 0 524 400 do_journa D 1.5 0.0
0:01 dbench
7734 nuetzel 18 0 25 524 0 524 400 do_journa D 1.5 0.0
0:01 dbench
7735 nuetzel 18 0 25 524 0 524 400 do_journa D 1.5 0.0
0:01 dbench
7736 nuetzel 18 0 25 524 0 524 400 wait_on_b D 1.5 0.0
0:01 dbench
7740 nuetzel 18 0 25 524 0 524 400 R 1.5 0.0
0:01 dbench
7741 nuetzel 18 0 25 524 0 524 400 do_journa D 1.5 0.0
0:01 dbench
7747 nuetzel 18 0 25 524 0 524 400 R 1.5 0.0
0:01 dbench
7748 nuetzel 18 0 25 524 0 524 400 do_journa D 1.5 0.0
0:01 dbench

Worst 20 latency times of 6314 measured in this period.
usec cause mask start line/file address end line/file

  6927        BKL        1  1302/inode.c         c016de79  1306/inode.c
  6622  spin_lock        1   547/sched.c         c0111b04   697/sched.c
  4823        BKL        1  1302/inode.c         c016de79   842/inode.c
  4421   reacqBKL        1  1375/sched.c         c01138b4   697/sched.c
  3546   reacqBKL        1  1375/sched.c         c01138b4  1873/namei.c
  3302        BKL        1  1302/inode.c         c016de79    52/inode.c
  2651   reacqBKL        0  1375/sched.c         c01138b4  1439/namei.c
  2597        BKL        0   533/inode.c         c016c4ed  1380/sched.c
  2585        BKL        0  1302/inode.c         c016de79  1381/sched.c
  2517        BKL        1  1302/inode.c         c016de79   697/sched.c
  2473        BKL        0    30/inode.c         c016b971  1380/sched.c
  2442        BKL        0  1437/namei.c         c014af4f  1306/inode.c
  2428   reacqBKL        0  1375/sched.c         c01138b4  1381/sched.c
  2409        BKL        0  1302/inode.c         c016de79  1380/sched.c
  2294  spin_lock        0   547/sched.c         c0111b04  1380/sched.c
  2289        BKL        1   927/namei.c         c0149ddf   697/sched.c
  2235        BKL        1    30/inode.c         c016b971   697/sched.c
  2225        BKL        0  1437/namei.c         c014af4f   697/sched.c
  2158        BKL        0   533/inode.c         c016c4ed  1381/sched.c
  2136        BKL        1  1366/namei.c         c014ac4c  1381/sched.c

c016de20 T reiserfs_dirty_inode
c016de79 ..... <<<<<
c016df10 T reiserfs_sync_inode

c0111ad0 T schedule
c0111b04 ..... <<<<<
c0112020 T __wake_up

c016de20 T reiserfs_dirty_inode
c016de79 ..... <<<<<
c016df10 T reiserfs_sync_inode

Regards,
Dieter

#11 lore raw
In reply to Alan Cox
Quoted 9 lines · Thomas Sailer
Quoted 3 lines

lowlatency soundcards usually have something like 128-512 samples per channel for 24-bit (32-bit data) 96 kHz 8 channels...

This doesn't make sense. Even professional soundcards provide hundreds of kBytes of buffer space. And even with most cheap PCI cards you need not wait until the whole buffer is full before start reading samples. Even with those soundcards the latency is mostly caused by the digital filters in the codec.

RME Digi 96/8 Pro has 128 kB buffer. This means 2048 samples per channel for 8 channel full-duplex operation which is 21.3 ms for 96 kHz. If interrupt is generated at 50% buffer fill then the time between interrupts is 10.7 ms.
(1)

ICEnsemble ICE1712 chip can use max 256 kB for record and playback so it's maximum latency is 56.9 ms for 96 kHz and 28.4 ms between interrupts. (2)

As a side note, from ICE's Windows control panel I can select 2-28 ms for DMA buffer size.

Those times are maximum, usually lower values are used. I use 10 ms buffer under Windows.

- Jussi Laako

In-Reply-To:

1) RME Digi96 series datasheet
2) ICEnsemble ICE1712 datasheet

Signature · 2 lines
PGP key fingerprint: 161D 6FED 6A92 39E2 EB5B  39DD A4DE 63EB C216 1E4B
Available at PGP keyservers