Huge disk performance degradation in 2.4.9
Sorry if this is already known or fixed.
I did not find something related in the last 2.4.10-pre changelog.
I recently upgraded from 2.4.6 to 2.4.9 and since then noticed that my hd (old noisy thing in an old P100 box) repeatedly
sounds strange (read on, please!) when dealing with lots of small files. (No physical problem; it's a kernel flushing issue.)
This turned out to be the sign of a quite huge performance loss anywhere between 2.4.7 and 2.4.9 (sorry, using a 56k Modem, I cannot test them all and it's a shame that I already deleted 2.4.6 sources).
I verified this on a PII-350 (440BX, 196Mb) using 600 files (instead of 300). I only tested things on reiserfs (v3.6. using 3.5 partitions) becuase I don't have ext2 around any longer.
For testing purposes, I used:
sync ; cat /proc/version ; sleep 1 ; /usr/bin/time sh -c 'declare -i cnt=0 ; while test $cnt -lt 300 ; do echo -en "\b\b\b\b\b\b\b$cnt " ;
dd if=/dev/zero of=file-$cnt bs=1 count=16 >/dev/null 2>&1 ; cnt=$cnt+1 ; done ; echo' ; rm file-*
This creates lots of 16-byte files using 16 write() calls [second test with 160kb files using 16 write() calls] and prints out the time (On P-100; 72Mb, 1Gb hd; besides kernel: equal setup for
all tests; machine idle; all tests ran several times; all kernels compiled with gcc version 2.95.2 19991024 (release)):
| (dd bs=1 count=16) | (dd bs=10240 count=16) kernel | user system elapsed CPU | user system elapsed CPU 2.4.9 | 3.12 5.10 32.46 25% | 3.84 14.31 73.09 24% 2.4.6 | 3.28 4.17 8.17 91% | 3.96 14.76 25.76 72% 2.2.19 | 2.82 3.75 7.12 92% | 4.42 12.90 19.26 89%
(user, system, elapsed time in seconds)
Look at the elapsed times! 2.4.9 takes >=3 times as long as 2.4.6 (and 2.2.19 performs even better).
This is a huge performance issue and I actually notice it using when squid or doing a CVS checkout with lots of small files.
When listening to the hd you note a difference. While 2.4.6 does a clustered write call from time to time, 2.4.9 starts to burst out in accessing the hd (always moving the hd's head) and does not finish until the test is over (same with my PII-350 and IBM 12Gb)
Anyone reproducing this? Reiserfs issue or cache/buffer/flush issue?
Regards,
Wolly
Quoted 3 lines
Anyone reproducing this? Reiserfs issue or
cache/buffer/flush issue?
I can not reproduced it witn ext2 2.4.10pre12aa1
I noticed the performance is much improved (to 2.4.9 and < )
I am gonna try it with ext3 later...
Cheers,
Quoted 9 lines
Regards,
Wolly
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
=====
S.KIEU
http://travel.yahoo.com.au - Yahoo! Travel
- Got Itchy feet? Get inspired!
Re: Huge disk performance degradation STILL IN 2.4.10
In reply to WollyHi kernel hackers,
As soon as 2.4.10 was out, I got the patch and tested it again. The problem is still there and did not get better at all.
Please have a look at that:
How to reproduce: Get the slowest computer you have and execute the file creation script as mentioned in my previous e-mail (appended below).
The faster the computer and the more RAM, the more files you have to create. Try increasing their number from 300 to 600 or 1000. Try running it several times; turn off swap.
The symptomes are as follows: The file counter raises quickly until at some point it begins to raise slowly only (2-4 files/sec on a P100 with 5 year old hardware). The hd is then accessed all the time and the hd's head is moved around continuously making things really slow.
If you still cannot reproduce it, make sure you have some applications running (X, windowmanager, xterm) and try executing `sync' in another XTerm while the file creation script is running.
When I tried, sync did not return until all files were created and the creation was awefully slow (steady hd accessed & head move).
Quoted 1 line
Is there nobody out there who can try & reproduce this?!
To me, it seems this is either a cache/buffer/flush issue or a reiserfs journal congestion issue.
Regards,
Wolly
Quoted 47 lines · On Friday 21 September 2001 19:23, I wrote:
I recently upgraded from 2.4.6 to 2.4.9 and since then noticed that my hd (old noisy thing in an old P100 box) repeatedly
sounds strange (read on, please!) when dealing with lots of small files. (No physical problem; it's a kernel flushing issue.)This turned out to be the sign of a quite huge performance loss anywhere between 2.4.7 and 2.4.9 (sorry, using a 56k Modem, I cannot test them all and it's a shame that I already deleted 2.4.6 sources).
I verified this on a PII-350 (440BX, 196Mb) using 600 files (instead of 300). I only tested things on reiserfs (v3.6. using 3.5 partitions) becuase I don't have ext2 around any longer.
For testing purposes, I used:
sync ; cat/proc/version; sleep 1 ;/usr/bin/timesh -c 'declare -i cnt=0 ; while test $cnt -lt 300 ; do echo -en "\b\b\b\b\b\b\b$cnt " ; dd if=/dev/zero of=file-$cnt bs=1 count=16 >/dev/null 2>&1 ; cnt=$cnt+1 ; done ; echo' ; rm file-*This creates lots of 16-byte files using 16
write()calls [second test with 160kb files using 16write()calls] and prints out the time (On P-100; 72Mb, 1Gb hd; besides kernel: equal setup for
all tests; machine idle; all tests ran several times; all kernels compiledwith gcc version 2.95.2 19991024 (release)):
| (dd bs=1 count=16) | (dd bs=10240 count=16)kernel | user system elapsed CPU | user system elapsed CPU 2.4.9 | 3.12 5.10 32.46 25% | 3.84 14.31 73.09 24% 2.4.6 | 3.28 4.17 8.17 91% | 3.96 14.76 25.76 72% 2.2.19 | 2.82 3.75 7.12 92% | 4.42 12.90 19.26 89%(user, system, elapsed time in seconds)
Look at the elapsed times! 2.4.9 takes >=3 times as long as 2.4.6 (and 2.2.19 performs even better).
This is a huge performance issue and I actually notice it using when squid or doing a CVS checkout with lots of small files.When listening to the hd you note a difference. While 2.4.6 does a clustered write call from time to time, 2.4.9 starts to burst out in accessing the hd (always moving the hd's head) and does not finish until the test is over (same with my PII-350 and IBM 12Gb)
Anyone reproducing this? Reiserfs issue or cache/buffer/flush issue?
Regards,
Wolly
Re: Huge disk performance degradation STILL IN 2.4.10
Quoted 5 lines · On Monday, September 24, 2001 01:19:18 AM +0200 Wolly <[email protected]> wrote:
Hi kernel hackers,
As soon as 2.4.10 was out, I got the patch and tested it again. The problem is still there and did not get better at all.
Could you please mount the FS with -o notail and try again? I think your test is hitting a worst case in the 2.4.x reiserfs tail code.
-chris
Re: Huge disk performance degradation STILL IN 2.4.10
In reply to WollyQuoted 10 lines · Chris Mason
Quoted 4 lines · On Monday, September 24, 2001 01:19:18 AM +0200 Wolly <[email protected]> wrote:
Hi kernel hackers,
As soon as 2.4.10 was out, I got the patch and tested it again. The problem is still there and did not get better at all.
Could you please mount the FS with -o notail and try again? I think your test is hitting a worst case in the 2.4.x reiserfs tail code.
Okay, I plugged some old hd into my computer and formatted one half with ext2, the other half with reiserfs.
It seems to be definitely a reiserfs issue because I cannot trigger the performance loss (permament hd head positioning) with ext2. However, passing -o notail when mounting does not help. (-o notail is accepted and bash# mount' displays /dev/hdxy on /mnt type reiserfs (rw,notail)')
Could you reproduce this? Did you even try?
And why does this not show up on kernels <=2.4.6 and shows up with linux>=2.4.9 (2.4.7, 2.4.8: unknown)?
-Wolly
Re: Huge disk performance degradation STILL IN 2.4.10
In reply to WollyQuoted 4 lines · Wolly
Okay, I plugged some old hd into my computer and formatted one half with ext2, the other half with reiserfs. It seems to be definitely a reiserfs issue because I cannot trigger the performance loss (permament hd head positioning) with ext2.
This might sound pedantic, but is ext2 on the first half of your hard disk and reiserfs on the second half? If so, that could skew your results. Try putting the reiserfs in the first half of the disk and see if that makes a difference.
See http://www.coker.com.au/bonnie++/zcav/ for why.
Signature · 1 line
#ozone/algorithm <[email protected]> - trust.in.love.to.save
Re: Huge disk performance degradation STILL IN 2.4.10
In reply to WollyQuoted 11 lines · Andre Pang
Quoted 4 lines · Wolly
Okay, I plugged some old hd into my computer and formatted one half with ext2, the other half with reiserfs. It seems to be definitely a reiserfs issue because I cannot trigger the performance loss (permament hd head positioning) with ext2.
This might sound pedantic, but is ext2 on the first half of your hard disk and reiserfs on the second half? If so, that could skew your results. Try putting the reiserfs in the first half of the disk and see if that makes a difference.
Well, actually this lead me to another idea: I created a disk image on my reiserfs home partition (the one I usually use to trigger the problem; note however that I can also reproduce it on my PII-350).
Firstly, I formatted the image with ext2 and run the file creation script. No problems, it works fine.
Then, I formatted the image with reiserfs and tried the file creation script. No problems either.. Strange.
I doubt it's a fragmentation issue (as kernel-2.4.6 does not have the problem and the image file would be fragmented as well).
Could this be some sort of reiserfs flushing issue? <speculation> I mean reiserfs wants to flush the newly created files too frequently. If reiserfs is on a disk partition, then this results in the performance degradation, if it is on an image, all requests to write to the image are cached again (by the reiserfs one level below which works fine because there is just manipulation of one large file) and the problem does not show up. </speculation>
Okay, as just a few people on the list seem to worry about the issue, I spent an hour online and downloaded kernel 2.4.6 + patches for 2.4.[789] again to see exactly where the problem begins.
Wolly
Re: Huge disk performance degradation STILL IN 2.4.10
In reply to Wollyi noticed it in 2.4.7-ac3 -> ac4, iirc.
Quoted 3 lines · Wolly
Okay, as just a few people on the list seem to worry about the issue, I spent an hour online and downloaded kernel 2.4.6 + patches for 2.4.[789] again to see exactly where the problem begins.
Signature · 2 lines
Tom Vier <[email protected]> DSA Key id 0x27371A2C