Significant performace improvements on reiserfs systems
Quoted 25 lines · Beau Kuiper
Quoted 3 lines · Chris Mason
Quoted 1 line
On Thursday, September 20, 2001 03:12:44 PM +0800 Beau Kuiper
Quoted 21 lines · <[email protected]> wrote:
Quoted 7 lines
Hi,
Resierfs on 2.4 has always been bog slow.
I have identified kupdated as the culprit, and have 3 patches that fix the peformance problems I have had been suffering.
Thanks for sending these along.
Quoted 10 lines
I would like these patches to be reviewed an put into the mainline kernel so that others can testthe changes.
Patch 1.
This patch fixes reiserfs to use the kupdated code path when told to resync its super block, like it did in 2.2.19. This is the culpit for bad reiserfs performace in 2.4. Unfortunately, this fix relies on the second patch to work properly.
Quoted 5 lines
Quoted 5 lines
I promised linus I would never reactivate this code, it is just too nasty ;-) The problem is that
write_superdoesn't know if it is called from sync or from kupdated. The right fix is to have an extra param towrite_super, or anothersuper_blockmethod that gets called instead ofwrite_superwhen an immediate commit is not required.
I examined that ReiserFS suffer from kupdated since 2.4.7-ac3. When ever I do "kill -STOP kupdated" the performance is much better. I know this is unsafe...
Quoted 32 lines
I don't think that this could happen until 2.5.x though, as either solution touches every file system. However, if we added an extra methed, we could do this while only slightly touching the other filesystems (where kupdated sync == real sync) Simply see if the method exists (is non-null) and call that instead with a kupdate sync instead of the normal
super_sync. Are you interested in me writing a patch to do this?Quoted 4 lines
It is possible to get almost the same behaviour as 2.2.x by changing the metadata sync interval in bdflush to 30 seconds.
But then kupdate doesn't flush normal data as regularly as it should, plus it is almost as messy as Patch 1 :-)
Quoted 12 lines
Quoted 11 lines
Patch 2
This patch implements a simple mechinism to ensure that each superblock only gets told to be flushed once. With reiserfs and the first patch, the superblock is still dirty after being told to sync (probably becasue it doesn't want to write out the entire journal every 5 seconds when kupdate calls it). This caused an infinite loop because
sync_superswould always find the reiserfs superblock dirty when called from kupdated. I am not convinced that this patch is the best one for this problem (suggestions?)Quoted 4 lines
It is ok to leave the superblock dirty, after all, since the commit wasn't done, the super is still dirty. If the checks from
reiserfs_write_superare actually slowing things down, then it is probably best to fix the checks.
Quoted 8 lines
I meant, there might be better wway to prevent the endless loop than adding an extra field to the superblock data structure. I beleive (I havn't explored reiserfs code much) the slowdown is caused by the journal being synced with the superblock, thus causing:
1) Too much contention for disk resources.
2) A huge increase in the number of times programs must be suspended to wait for the disk
Please have a look at Robert Love's Linux kernel preemption patches and the conversation about my reported latency results.
It seems that ReiserFS is involved in the poor audio behavior (hiccups during MP2/MP3/Ogg-Vorbis playback).
Re: [PATCH] Preemption Latency Measurement Tool
http://marc.theaimsgroup.com/?l=linux-kernel&m=100097432006605&w=2
Taken from Andrea's latest post:
Quoted 6 lines
those are kernel addresses, can you resolve them via System.map rather than trying to find their start/end line number?
Quoted 3 lines
Worst 20 latency times of 8033 measured in this period.
usec cause mask start line/file address end line/file
10856spin_lock11376/sched.cc0114db3697/sched.c
I can (with Randy Dunlap's ksysmap,
http://www.osdlab.org/sw_resources/scripts/ksysmap).
SunWave1>./ksysmap /boot/System.map c0114db3
ksysmap: searching '/boot/System.map' for 'c0114db3'
c0114d60 T preempt_schedulec0114db3 ..... <<<<<c0114e10 T wake_up_process
Quoted 8 lines
with dbench 48 we gone to 10msec latency as far I can see (still far from 0.5~1 sec). dbench 48 is longer so more probability to get the higher latency, and it does more I/O, probably also seeks more, so thre are many variables (slower insection in I/O queues first of all, etcll). However 10msec isn't that bad, it means 100hz, something that the human eye cannot see. 0.5~1 sec would been horribly bad latency instead.. :)
Quoted 1 line
10705 BKL 1
1302/inode.cc016f359697/sched.c
c016f300 T reiserfs_dirty_inodec016f359 ..... <<<<<c016f3f0 T reiserfs_sync_inode
Quoted 1 line
Quoted 1 line
10577
spin_lock11376/sched.cc0114db3303/namei.c
c0114d60 T preempt_schedulec0114db3 ..... <<<<<c0114e10 T wake_up_process
Quoted 1 line
Quoted 1 line
9427
spin_lock1547/sched.cc0112fe4697/sched.c
c0112fb0 T schedulec0112fe4 ..... <<<<<c0113500 T __wake_up
Quoted 1 line
Quoted 1 line
8526 reacqBKL 1
1375/sched.cc0114d94697/sched.c
c0114d60 T preempt_schedulec0114d94 ..... <<<<<c0114e10 T wake_up_process
Quoted 1 line
Quoted 1 line
4492 reacqBKL 1
1375/sched.cc0114d941381/sched.c
c0114d60 T preempt_schedulec0114d94 ..... <<<<<c0114e10 T wake_up_process
Quoted 1 line
Quoted 1 line
4171 BKL 1
1302/inode.cc016f3591381/sched.c
c016f300 T reiserfs_dirty_inodec016f359 ..... <<<<<c016f3f0 T reiserfs_sync_inode
Quoted 1 line
Quoted 1 line
3902 reacqBKL 0
1375/sched.cc0114d941306/inode.c
c0114d60 T preempt_schedulec0114d94 ..... <<<<<c0114e10 T wake_up_process
Quoted 1 line
Quoted 1 line
3376
spin_lock01376/sched.cc0114db31380/sched.c
c0114d60 T preempt_schedulec0114db3 ..... <<<<<c0114e10 T wake_up_process
Quoted 1 line
Quoted 1 line
3132 BKL 0
1302/inode.cc016f3591380/sched.c
c016f300 T reiserfs_dirty_inodec016f359 ..... <<<<<c016f3f0 T reiserfs_sync_inode
Quoted 1 line
Quoted 1 line
3096
spin_lock1547/sched.cc0112fe41380/sched.c
c0112fb0 T schedulec0112fe4 ..... <<<<<c0113500 T __wake_up
Quoted 1 line
Quoted 1 line
2808 BKL 0
30/inode.cc016ce511381/sched.c
c016ce20 T reiserfs_delete_inodec016ce51 ..... <<<<<c016cf30 t _make_cpu_key
Quoted 1 line
Quoted 1 line
2807
spin_lock0547/sched.cc0112fe41381/sched.c
c0112fb0 T schedulec0112fe4 ..... <<<<<c0113500 T __wake_up
Quoted 1 line
Quoted 1 line
2782 BKL 0
452/exit.cc011af611380/sched.c
c011ae30 T do_exitc011af61 ..... <<<<<c011b190 T complete_and_exit
Quoted 1 line
Quoted 1 line
2631
spin_lock0483/dcache.cc0153efa520/dcache.c
c0153ec0 t select_parentc0153efa ..... <<<<<c0153fc0 T shrink_dcache_parent
Quoted 1 line
Quoted 1 line
2533 BKL 0
533/inode.cc016d9cd1380/sched.c
c016d930 T reiserfs_get_blockc016d9cd ..... <<<<<c016e860 t init_inode
Quoted 1 line
Quoted 1 line
2489 BKL 0
927/namei.cc014b2bf1380/sched.c
c014b210 T vfs_createc014b2bf ..... <<<<<c014b360 T open_namei
Quoted 1 line
Quoted 1 line
2389 BKL 1
452/exit.cc011af6152/inode.c
c011ae30 T do_exitc011af61 ..... <<<<<c011b190 T complete_and_exit
Quoted 1 line
Quoted 1 line
2369 BKL 1
1302/inode.cc016f359842/inode.c
c016f300 T reiserfs_dirty_inodec016f359 ..... <<<<<c016f3f0 T reiserfs_sync_inode
Quoted 1 line
Quoted 1 line
2327 BKL 1
30/inode.cc016ce511380/sched.c
c016ce20 T reiserfs_delete_inodec016ce51 ..... <<<<<c016cf30 t _make_cpu_key
Quoted 2 lines
3) Poor CPU utilization in code that uses the filesystem regularly (like compiling)
Unneeded kernel locks/stalls which hurt latency and (global) throughput.
Quoted 31 lines
Quoted 15 lines
Quoted 15 lines
Patch 3
This patch was generated as I was exploring the buffer cache, wondering why reiserfs was so slow on 2.4. I found that kupdated may write buffers that are not actually old back to disk. Eg
Imagine that there are 20 dirty buffers. 16 of them are more that 30 seconds old (and should be written back), but the other 4 are younger than 30 seconds. The current code would force all 20 out to disk, interrupting
programs still using the young 4 until the disk write was complete.Quoted 8 lines
Quoted 6 lines
I know that it isn't a major problem, but I found it and I have written the patch for it :-)
Please try out these patches and give comments about style, performace ect. They fixed my problems, sliced almost a minute off 2.2.19 kernel compile time on my duron 700 (from 4min 30sec to 3min 45sec)
Doe you have the results of the individual fixes?
Patch 3 doesn't improve performace much (even in theory the number of dirty buffers being wrongly flushed is pretty low)
Patch 2 doesn't improve performace at all (unless you apply patch 1, without it, the computer will bog itself into the ground on the first kupdated)
Quoted 5 lines
Patch 1 makes a huge difference because it stops reiserfs from reacting badly on a kupdated.
Are there any good benchmarks you want me to run, on the plain and modded kernels.
I will do some benchmarks against Andrea's VM 2.4.10-pre12 + patch-rml-2.4.10-pre12-preempt-kernel-1 + patch-rml-2.4.10-pre12-preempt-stats-1
Hope this post isn't to long and nonbody feels offended.
Regards,
Dieter