Abysmal interactive performance on 2.4.linus
On a 768 MB SMP box (2x466 MHz Celeron), I see some weird problems with interactive performance on 2.4.15pre{1,2}. A good example of this is the following scenario:
- copy a large file (eg. an iso image file) to a directory on the same (reiserfs in this case) filesystem, or...
- do a filesystem comparison between a CD and the original file (with cmp /mnt/cdrom/<filename> /mnt/reiserfs/1/data/<original_file_location>, using a PLEXTOR Model: CD-ROM PX-40TS SCSI CD-ROM drive),
- and THEN (while the copy or comparison runs) try any simple command (like 'ls /mnt/reiserfs/1/data' or 'top' or anything else...).
Response time is abysmal, a simple 'ls /some/dir' takes tens of seconds to start. Once the command is running, performance is normal. Try this when a cdrecord session is running and you'll get a buffer underrun.
The box has 768 MB of RAM, 512 MB of swap. There is no significant load on the system (according to an already running copy of top) neither before nor during the test. Try tab-completing a command in a terminal, and that terminal freezes for tens of seconds, usually until after the file system load has gone down.
In a few words, heavy filesystem activity seems to wreak havoc on the system. Not by loading the CPU (it hardly breaks out a sweat at 178% idle (SMP...)).
Turning off swap (swapoff -a) does not change the observed behaviour.
Anyone else seen something like this?
Signature · 9 lines
WWWWW _______________________
## o o\ / Frank de Lange \
}# \| / \
##---# _/ <Hacker for Hire> \
#### \ +31-320-252965 /
\ [email protected] /
-------------------------
[ "Omnis enim res, quae dando non deficit, dum habetur
et non datur, nondum habetur, quomodo habenda est." ]Quoted 11 lines · Frank de Lange
On a 768 MB SMP box (2x466 MHz Celeron), I see some weird problems with interactive performance on 2.4.15pre{1,2}. A good example of this is the following scenario:
- copy a large file (eg. an iso image file) to a directory on the same (reiserfs in this case) filesystem, or...
- do a filesystem comparison between a CD and the original file (with cmp /mnt/cdrom/<filename> /mnt/reiserfs/1/data/<original_file_location>, using a PLEXTOR Model: CD-ROM PX-40TS SCSI CD-ROM drive),
Quoted 6 lines
- and THEN (while the copy or comparison runs) try any simple command (like 'ls
/mnt/reiserfs/1/data' or 'top' or anything else...).Response time is abysmal, a simple 'ls /some/dir' takes tens of seconds to start. Once the command is running, performance is normal. Try this when a cdrecord session is running and you'll get a buffer underrun.
Can you try 2.4.13ac6 (not 7/8), and 2.2.20, and post a comparison?
Signature · 3 lines
Jeff Garzik | Only so many songs can be sung Building 1024 | with two lips, two lungs, and one tongue. MandrakeSoft | - nomeansno
I have tried both 2.4.13ac6 and 2.4.15-pre2, and am getting the same behaviour in 2.4.15-pre2, but not in the ac kernel.
mike
--
My Operat~1 System supports long filena~1, does yours?
Quoted 32 lines · Jeff Garzik
Quoted 11 lines · Frank de Lange
On a 768 MB SMP box (2x466 MHz Celeron), I see some weird problems with interactive performance on 2.4.15pre{1,2}. A good example of this is the following scenario:
- copy a large file (eg. an iso image file) to a directory on the same (reiserfs in this case) filesystem, or...
- do a filesystem comparison between a CD and the original file (with cmp /mnt/cdrom/<filename> /mnt/reiserfs/1/data/<original_file_location>, using a PLEXTOR Model: CD-ROM PX-40TS SCSI CD-ROM drive),Quoted 6 lines
- and THEN (while the copy or comparison runs) try any simple command (like 'ls
/mnt/reiserfs/1/data' or 'top' or anything else...).Response time is abysmal, a simple 'ls /some/dir' takes tens of seconds to start. Once the command is running, performance is normal. Try this when a cdrecord session is running and you'll get a buffer underrun.
Can you try 2.4.13ac6 (not 7/8), and 2.2.20, and post a comparison?
--
Jeff Garzik | Only so many songs can be sung Building 1024 | with two lips, two lungs, and one tongue. MandrakeSoft | - nomeansno-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 22 lines · Frank de Lange
On a 768 MB SMP box (2x466 MHz Celeron), I see some weird problems with interactive performance on 2.4.15pre{1,2}. A good example of this is the following scenario:
- copy a large file (eg. an iso image file) to a directory on the same (reiserfs in this case) filesystem, or...
- do a filesystem comparison between a CD and the original file (with cmp /mnt/cdrom/<filename> /mnt/reiserfs/1/data/<original_file_location>, using a PLEXTOR Model: CD-ROM PX-40TS SCSI CD-ROM drive),- and THEN (while the copy or comparison runs) try any simple command (like 'ls
/mnt/reiserfs/1/data' or 'top' or anything else...).Response time is abysmal, a simple 'ls /some/dir' takes tens of seconds to start. Once the command is running, performance is normal. Try this when a cdrecord session is running and you'll get a buffer underrun.
The box has 768 MB of RAM, 512 MB of swap. There is no significant load on the system (according to an already running copy of top) neither before nor during the test. Try tab-completing a command in a terminal, and that terminal freezes for tens of seconds, usually until after the file system load has gone down.
Quoted 6 lines
In a few words, heavy filesystem activity seems to wreak havoc on the system. Not by loading the CPU (it hardly breaks out a sweat at 178% idle (SMP...)).
Turning off swap (swapoff -a) does not change the observed behaviour.
Anyone else seen something like this?
I have 2.4.10-ac12 here and reiserfs filesystems and I have to say that performance is terrible when doing anything diskintensive. It seems like diskscheduling is very broken for my IDE disk, or it's a reiserfs problem. maybe it pushes a _lot_ into the diskscheduling at once?
I've heard that read-ahead for IDE has been broken for a while but's fixed in -ac and in Andre Hedrick's IDE-patches.
I'm going to upgrade to a more recent kernel and see how it behaves.
/Martin
Never argue with an idiot. They drag you down to their level, then beat you with experience.
Quoted 1 line · Jeff Garzik
Can you try 2.4.13ac6 (not 7/8), and 2.2.20, and post a comparison?
Here's the results from some tests I did:
2.2.20
======
without filesystem activity
no slowdowns observed
time ls -al /usr/|sort -k 5 -n
real 0m0.121s
user 0m0.000s
sys 0m0.090s
with filesystem activity on ext2
no slowdowns observed
time ls -al /opt/|sort -k 5 -n
real 0m0.079s
user 0m0.010s
sys 0m0.100s
2.4.13-ac5
==========
no slowdowns observed
without filesystem activity
time ls -al /usr/|sort -k 5 -n
real 0m0.142s
user 0m0.000s
sys 0m0.000s
with filesystem activity on ext2
no slowdowns observed
time ls -al /opt/|sort -k 5 -n
real 0m0.022s
user 0m0.020s
sys 0m0.010s
with filesystem activity on reiserfs
- it took 31 seconds to just open this small ( < 1 kb) text file (which resides in my home directory, on an ext2 filesystem) in vi...
time ls -al /usr/|sort -k 5 -n
real 0m6.136s user 0m0.020s sys 0m0.020s
2.4.15-pre4
===========
without filesystem activity
no slowdowns observed
time ls -al /usr/|sort -k 5 -n
real 0m0.081s
user 0m0.010s
sys 0m0.010s
with filesystem activity on ext2
no slowdowns observed
time ls -al /usr/|sort -k 5 -n
real 0m0.146s user 0m0.000s sys 0m0.020s
with filesystem activity on reiserfs
system behaviour erratic, some slowdowns
time ls -al /opt|sort -k5 -n
real 0m13.232s user 0m0.020s sys 0m0.010s
Seems that reiserfs is the common factor here, at least on my box. This is a 35 GB reiserfs filesystem, app 80% used, both large and small files.
As said in my previous message, the numbers themselves don't mean squat. It is the large delays (the fact that user+sys <<< real) which are the problem here.
Any other magic anyone wants me to perform? Hans, you reading this?
Cheers//Frank
Signature · 9 lines
WWWWW _______________________
## o o\ / Frank de Lange \
}# \| / \
##---# _/ <Hacker for Hire> \
#### \ +31-320-252965 /
\ [email protected] /
-------------------------
[ "Omnis enim res, quae dando non deficit, dum habetur
et non datur, nondum habetur, quomodo habenda est." ]Quoted 6 lines
Seems that reiserfs is the common factor here, at least on my box. This is a 35 GB reiserfs filesystem, app 80% used, both large and small files.
As said in my previous message, the numbers themselves don't mean squat. It is the large delays (the fact that user+sys <<< real) which are the problem here.
This was also reported as
Suspected bug - System slowdown under unexplained excessive disk I/O - 2.4.13 with huge delays during compiles (sasha Pachev) or mysql-benchmarks (me). But today I do not find this reiser-specific, this also seems to happen with ext3.
But as you wrote, not with ext2. I see that there is more disk-activity due to journaling in both cases, but waiting 30 seconds for simple tasks or waking from screen-apm seems not to be right.
Oktay Akbal
On Mon, Nov 12, 2001 at 11:56:42PM +0100, Frank de Lange wrote:
[snip]
Quoted 8 lines
Seems that reiserfs is the common factor here, at least on my box. This is a 35 GB reiserfs filesystem, app 80% used, both large and small files.
As said in my previous message, the numbers themselves don't mean squat. It is the large delays (the fact that user+sys <<< real) which are the problem here.
Any other magic anyone wants me to perform? Hans, you reading this?
Do you see/hear a lot of seeking happing during the delays?
If so, your Reiser partition is probably fragmented to hell...
IIRC this problem is being looked at, check some archives of lkml or reiser...
Mike
Quoted 5 lines · Mike Fedyk
On Mon, Nov 12, 2001 at 11:56:42PM +0100, Frank de Lange wrote:
[snip]Quoted 3 lines
Seems that reiserfs is the common factor here, at least on my box. This is a 35 GB reiserfs filesystem, app 80% used, both large and small files.
Quoted 7 lines
Quoted 5 lines
As said in my previous message, the numbers themselves don't mean squat. It is the large delays (the fact that user+sys <<< real) which are the problem here.
Any other magic anyone wants me to perform? Hans, you reading this?
Do you see/hear a lot of seeking happing during the delays?
Yup this is probably what's happening to me. I didn't think a harddrive could do so many seeks so fast :)
Quoted 1 line
If so, your Reiser partition is probably fragmented to hell...
:(
Quoted 1 line
IIRC this problem is being looked at, check some archives of lkml or reiser...
Ok, thanks
/Martin
Never argue with an idiot. They drag you down to their level, then beat you with experience.
Quoted 10 lines · Martin Josefsson
Quoted 8 lines · Mike Fedyk
On Mon, Nov 12, 2001 at 11:56:42PM +0100, Frank de Lange wrote:
[snip]Quoted 6 lines
Seems that reiserfs is the common factor here, at least on my box. This is a 35 GB reiserfs filesystem, app 80% used, both large and small files.
As said in my previous message, the numbers themselves don't mean squat. It is the large delays (the fact that user+sys <<< real) which are the problem here.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Any other magic anyone wants me to perform? Hans, you reading this?
Do you see/hear a lot of seeking happing during the delays?
Yup this is probably what's happening to me. I didn't think a harddrive could do so many seeks so fast :)
Check out the thread "reiserfs performance loss" back in oct 12 and 13...
Mike
Quoted 5 lines · you
Seems that reiserfs is the common factor here, at least on my box. This is a 35 GB reiserfs filesystem, app 80% used, both large and small files.
As said in my previous message, the numbers themselves don't mean squat. It is the large delays (the fact that user+sys <<< real) which are the problem here.
As another data point, I'm seeing the exact same thing. I haven't tried any non-Linus kernels, though. But recent 2.4.x (x >= 10?) linus kernels with reiserfs have these several-second delays during moderate-to-heavy disk i/o, exactly as you've described. I've seen this on both an SMP PIII system and a UP Athlon.
Signature · 2 lines
Jason Lunz Trellis Network Security [email protected] http://www.trellisinc.com/
Quoted 74 lines · Frank de Lange
Quoted 2 lines · Jeff Garzik
Can you try 2.4.13ac6 (not 7/8), and 2.2.20, and post a comparison?
Here's the results from some tests I did:
2.2.20
======
without filesystem activity
no slowdowns observed
time ls -al /usr/|sort -k 5 -n
real 0m0.121s
user 0m0.000s
sys 0m0.090swith filesystem activity on ext2
no slowdowns observed
time ls -al /opt/|sort -k 5 -n
real 0m0.079s
user 0m0.010s
sys 0m0.100s2.4.13-ac5
==========
no slowdowns observed
without filesystem activity
time ls -al /usr/|sort -k 5 -n
real 0m0.142s
user 0m0.000s
sys 0m0.000swith filesystem activity on ext2
no slowdowns observed
time ls -al /opt/|sort -k 5 -n
real 0m0.022s
user 0m0.020s
sys 0m0.010swith filesystem activity on reiserfs
- it took 31 seconds to just open this small ( < 1 kb) text file (which resides in my home directory, on an ext2 filesystem) in vi...
time ls -al /usr/|sort -k 5 -nreal 0m6.136s user 0m0.020s sys 0m0.020s2.4.15-pre4
===========
without filesystem activity
no slowdowns observed
time ls -al /usr/|sort -k 5 -n
real 0m0.081s
user 0m0.010s
sys 0m0.010swith filesystem activity on ext2
no slowdowns observed
time ls -al /usr/|sort -k 5 -nreal 0m0.146s user 0m0.000s sys 0m0.020swith filesystem activity on reiserfs
system behaviour erratic, some slowdowns
time ls -al /opt|sort -k5 -nreal 0m13.232s user 0m0.020s sys 0m0.010sSeems that reiserfs is the common factor here, at least on my box. This is a 35 GB reiserfs filesystem, app 80% used, both large and small files.
Quoted 7 lines
As said in my previous message, the numbers themselves don't mean squat. It is the large delays (the fact that user+sys <<< real) which are the problem here.
Any other magic anyone wants me to perform? Hans, you reading this?
Cheers//Frank
Yura, see if you can reproduce this and analyze the cause. If I understand correctly, he is saying the problem is not throughput but latency. Is that correct Frank? Once Yura reproduces it, I will speculate as to the cause.
Hans
Quoted 4 lines · Hans Reiser
Yura, see if you can reproduce this and analyze the cause. If I understand correctly, he is saying the problem is not throughput but latency. Is that correct Frank? Once Yura reproduces it, I will speculate as to the cause.
Correct.
Signature · 9 lines
WWWWW _______________________
## o o\ / Frank de Lange \
}# \| / \
##---# _/ <Hacker for Hire> \
#### \ +31-320-252965 /
\ [email protected] /
-------------------------
[ "Omnis enim res, quae dando non deficit, dum habetur
et non datur, nondum habetur, quomodo habenda est." ]Quoted 81 lines · Hans Reiser
Quoted 79 lines · Frank de Lange
Quoted 2 lines · Jeff Garzik
Can you try 2.4.13ac6 (not 7/8), and 2.2.20, and post a comparison?
Here's the results from some tests I did:
2.2.20
======
without filesystem activity
no slowdowns observed
time ls -al /usr/|sort -k 5 -nreal 0m0.121s user 0m0.000s sys 0m0.090swith filesystem activity on ext2
no slowdowns observed
time ls -al /opt/|sort -k 5 -nreal 0m0.079s user 0m0.010s sys 0m0.100s2.4.13-ac5
==========
no slowdowns observed
without filesystem activity
time ls -al /usr/|sort -k 5 -nreal 0m0.142s user 0m0.000s sys 0m0.000swith filesystem activity on ext2
no slowdowns observed
time ls -al /opt/|sort -k 5 -nreal 0m0.022s user 0m0.020s sys 0m0.010swith filesystem activity on reiserfs
- it took 31 seconds to just open this small ( < 1 kb) text file (which resides in my home directory, on an ext2 filesystem) in vi...
time ls -al /usr/|sort -k 5 -nreal 0m6.136s user 0m0.020s sys 0m0.020s2.4.15-pre4
===========
without filesystem activity
no slowdowns observed
time ls -al /usr/|sort -k 5 -nreal 0m0.081s user 0m0.010s sys 0m0.010swith filesystem activity on ext2
no slowdowns observed
time ls -al /usr/|sort -k 5 -nreal 0m0.146s user 0m0.000s sys 0m0.020swith filesystem activity on reiserfs
system behaviour erratic, some slowdowns
time ls -al /opt|sort -k5 -nreal 0m13.232s user 0m0.020s sys 0m0.010sSeems that reiserfs is the common factor here, at least on my box. This is a 35 GB reiserfs filesystem, app 80% used, both large and small files.
As said in my previous message, the numbers themselves don't mean squat. It is the large delays (the fact that user+sys <<< real) which are the problem here.
Any other magic anyone wants me to perform? Hans, you reading this?
Quoted 8 lines
Quoted 2 lines
Cheers//Frank
Yura, see if you can reproduce this and analyze the cause. If I understand correctly, he is saying the problem is not throughput but latency. Is that correct Frank? Once Yura reproduces it, I will speculate as to the cause.
Hans
Hello,
Yes, the latency problem exist. I was using "dd" and "cp" commands to create and copy 1 GB file as "filesystem activity".
In both cases the set of :
"time ls -al /opt|sort -k5 -n" show the same delay.
One way to improve the situation is to use the patch below, suggested by Chris Mason :
fs/buffer.c Sun, 18 Nov 2001 12:44:40 -0500 renamed from fs/buffer.c Fri, 16 Nov 2001 10:58:28 -0500 +3 -2
@@ -1020,9 +1020,10 @@ struct buffer_head * bh; bh = get_hash_table(dev, block, size); - if (bh) + if (bh) { + touch_buffer(bh) ; return bh; - + } if (!grow_buffers(dev, block, size)) free_more_memory(); }
Thanks,
Yura.