latency performance of 2.4 and 2.5...

hi andrew, hi everyone !

as per your request, i fearlessly built and booted 2.5.41 with preemption enabled.
although it was my first take at the dev kernel, it went very smoothly :-D

anyway, it seems to perform worse than 2.4.20-pre1-lowlatency. it feels very snappy on startup, but under load it is a lot less responsive to user interaction.

i have run benno senoner's latencytest on both kernels. you can see the results at

http://spunk.dnsalias.org/latencytest .

for me, the biggest problem seems to be soft-raid. i have my /var partition striped over 2 scsi disks, and accessing them causes huge latency peaks.
is there any chance to get this under control ?
i don't have a spare disk around, so i can't easily compare to non-raid, but i'm quite sure it's the source of the problem, because people with similar hardware are doing fine.

other than that, there were a few peaks under /proc stress, but they are not easily reproducible.

i have not yet tested your mm-patches due to a compile failure (log is attached below). i used
http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.41/2.5.41-mm3/2.5.41-mm3.gz.

if there's any patches to try or more benchmarks to run, let me know.

best,

jörn

***
Hardware: Dual-P3 @ 600Mhz, 512MB PC100 SDRAM, ASUS P2B-DS one 4G IBM DCAS SCSI disk (/), 2 9G COMPAQ SCSI disks (/var, soft-RAID 0),
SCSI CDROM and CDR, no IDE devices.

Soundcards: SB Live (used during the tests) and Ensoniq 1371 (idle) The audio file that was played back during the tests came over NFS.

***
make -f init/Makefile
Generating include/linux/compile.h (updated)
gcc -Wp,-MD,init/.version.o.d -D__KERNEL__ -Iinclude -Wall
-Wstrict-prototypes -Wno-trigraphs -O2 -fno-strict-aliasing -fno-common -pipe -mpreferred-stack-boundary=2 -march=i686 -Iarch/i386/mach-generic -fomit-frame-pointer -nostdinc -iwithprefix include
-DKBUILD_BASENAME=version -c -o init/version.o init/version.c
ld -m elf_i386 -r -o init/built-in.o init/main.o init/version.o
init/do_mounts.o

        ld -m elf_i386 -e stext -T arch/i386/vmlinux.lds.s

arch/i386/kernel/head.o arch/i386/kernel/init_task.o init/built-in.o --start-group arch/i386/kernel/built-in.o arch/i386/mm/built-in.o arch/i386/mach-generic/built-in.o kernel/built-in.o mm/built-in.o fs/built-in.o ipc/built-in.o security/built-in.o lib/lib.a arch/i386/lib/lib.a drivers/built-in.o sound/built-in.o arch/i386/pci/built-in.o net/built-in.o --end-group -o .tmp_vmlinux1 fs/built-in.o: In function `isofs_put_super':
fs/built-in.o(.text+0x390e5): undefined reference to unload_nls' fs/built-in.o: In function isofs_fill_super':
fs/built-in.o(.text+0x39f0b): undefined reference to load_nls' fs/built-in.o(.text+0x39f25): undefined reference to load_nls_default' fs/built-in.o(.text+0x3a0c6): undefined reference to unload_nls' fs/built-in.o: In function wcsntombs_be':
fs/built-in.o(.text+0x3c60b): undefined reference to `utf8_wctomb'
make: *** [.tmp_vmlinux1] Error 1

Jörn Nettingsmeier     
Kurfürstenstr 49, 45138 Essen, Germany      
http://spunk.dnsalias.org (my server)
http://www.linuxdj.com/audio/lad/ (Linux Audio Developers)

some new interesting results with 2.5.42:

http://spunk.dnsalias.org/latencytest/2.5.42/2x256.html

overall much worse, *but* greatly reduced latency peaks (max. 6 ms) as compared to 2.5.41:

http://spunk.dnsalias.org/latencytest/2.5.41/2x256.html

here the peaks easily reach 13 ms.
i'm not really sure what to make of this....
can someone explain ?

andrew, it seems part of your mm patch was merged - is there an updated patch around that will add the missing hunks ?

best,

jörn

Jörn Nettingsmeier     
Kurfürstenstr 49, 45138 Essen, Germany      
http://spunk.dnsalias.org (my server)
http://www.linuxdj.com/audio/lad/ (Linux Audio Developers)

Rather depends on the filesystem. ext3 does its own write scheduling, and does stuff inside lock_kernel(). It needs a couple of scheduling points I guess.

I'd expect ext2 to work OK with preemption, but nobody has really looked yet. Unless you're using ftruncate() (grr.)

Well 2.5.41 had the big change to the truncate code, which was a rather important part of this whole effort. I can't immediately think of a 2.5.42 change which would do this...

But nobody has really got down and dug out the toolkit on 2.5 scheduling latency. I guess I should do that soon.

oh, i should have stated i'm using reiserfs on /, /usr and /var (var being a softraid-0).

Jörn Nettingsmeier     
Kurfürstenstr 49, 45138 Essen, Germany      
http://spunk.dnsalias.org (my server)
http://www.linuxdj.com/audio/lad/ (Linux Audio Developers)

Jörn,
I did a little work on this file system choice a couple of weeks ago. My
results said stay away from ext3. I'm now using reiserfs.

There is a directory with some results on the same drive (different
partitions) using ext2, ext3 and reiserfs. ext3 and reiserfs also have data on how ext2 performed on that partition, but it didn't matter. Each partition was mostly the same.

http://www.controlnet.com/~makeMusic/disklatency/fstest/

FYI - I had to drop Gnome as it had too many things going on in the
background. These results were done using fluxbox.

Mark

-----Original Message-----
From: [email protected] [mailto:[email protected]]On Behalf Of Joern Nettingsmeier
Sent: Monday, October 14, 2002 1:53 AM
To: Andrew Morton
Cc: [email protected]; [email protected]
Subject: Re: [linux-audio-dev] latency performance of 2.4 and 2.5...

oh, i should have stated i'm using reiserfs on /, /usr and /var (var being a softraid-0).

--
Jörn Nettingsmeier
Kurfürstenstr 49, 45138 Essen, Germany
http://spunk.dnsalias.org (my server)
http://www.linuxdj.com/audio/lad/ (Linux Audio Developers)