Skip to contentSkip to conversation

Abysmal interactive performance on 2.4.linus

DISCUSSION Frank de Lange 9 participants 13 messages Nov 12, 2001 – Nov 21, 2001 VFS

lore
Share & tools
lore
#0 lore raw

On a 768 MB SMP box (2x466 MHz Celeron), I see some weird problems with interactive performance on 2.4.15pre{1,2}. A good example of this is the following scenario:

- copy a large file (eg. an iso image file) to a directory on the same (reiserfs in this case) filesystem, or...
- do a filesystem comparison between a CD and the original file (with cmp /mnt/cdrom/<filename> /mnt/reiserfs/1/data/<original_file_location>, using a PLEXTOR Model: CD-ROM PX-40TS SCSI CD-ROM drive),

- and THEN (while the copy or comparison runs) try any simple command (like 'ls /mnt/reiserfs/1/data' or 'top' or anything else...).

Response time is abysmal, a simple 'ls /some/dir' takes tens of seconds to start. Once the command is running, performance is normal. Try this when a cdrecord session is running and you'll get a buffer underrun.

The box has 768 MB of RAM, 512 MB of swap. There is no significant load on the system (according to an already running copy of top) neither before nor during the test. Try tab-completing a command in a terminal, and that terminal freezes for tens of seconds, usually until after the file system load has gone down.

In a few words, heavy filesystem activity seems to wreak havoc on the system. Not by loading the CPU (it hardly breaks out a sweat at 178% idle (SMP...)).

Turning off swap (swapoff -a) does not change the observed behaviour.

Anyone else seen something like this?

Signature · 9 lines
  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \ [email protected]  /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]
#1 lore raw
Quoted 11 lines · Frank de Lange

On a 768 MB SMP box (2x466 MHz Celeron), I see some weird problems with interactive performance on 2.4.15pre{1,2}. A good example of this is the following scenario:

- copy a large file (eg. an iso image file) to a directory on the same (reiserfs in this case) filesystem, or...
- do a filesystem comparison between a CD and the original file (with cmp /mnt/cdrom/<filename> /mnt/reiserfs/1/data/<original_file_location>, using a PLEXTOR Model: CD-ROM PX-40TS SCSI CD-ROM drive),

Quoted 6 lines

- and THEN (while the copy or comparison runs) try any simple command (like 'ls /mnt/reiserfs/1/data' or 'top' or anything else...).

Response time is abysmal, a simple 'ls /some/dir' takes tens of seconds to start. Once the command is running, performance is normal. Try this when a cdrecord session is running and you'll get a buffer underrun.

Can you try 2.4.13ac6 (not 7/8), and 2.2.20, and post a comparison?

Signature · 3 lines
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno
#2 lore raw

I have tried both 2.4.13ac6 and 2.4.15-pre2, and am getting the same behaviour in 2.4.15-pre2, but not in the ac kernel.

mike

--

My Operat~1 System supports long filena~1, does yours?

Quoted 32 lines · Jeff Garzik
Quoted 11 lines · Frank de Lange

On a 768 MB SMP box (2x466 MHz Celeron), I see some weird problems with interactive performance on 2.4.15pre{1,2}. A good example of this is the following scenario:

- copy a large file (eg. an iso image file) to a directory on the same (reiserfs in this case) filesystem, or...
- do a filesystem comparison between a CD and the original file (with cmp /mnt/cdrom/<filename> /mnt/reiserfs/1/data/<original_file_location>, using a PLEXTOR Model: CD-ROM PX-40TS SCSI CD-ROM drive),

Quoted 6 lines

- and THEN (while the copy or comparison runs) try any simple command (like 'ls /mnt/reiserfs/1/data' or 'top' or anything else...).

Response time is abysmal, a simple 'ls /some/dir' takes tens of seconds to start. Once the command is running, performance is normal. Try this when a cdrecord session is running and you'll get a buffer underrun.

Can you try 2.4.13ac6 (not 7/8), and 2.2.20, and post a comparison?

--

Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#3 lore raw
In reply to Frank de Lange
Quoted 22 lines · Frank de Lange

On a 768 MB SMP box (2x466 MHz Celeron), I see some weird problems with interactive performance on 2.4.15pre{1,2}. A good example of this is the following scenario:

- copy a large file (eg. an iso image file) to a directory on the same (reiserfs in this case) filesystem, or...
- do a filesystem comparison between a CD and the original file (with cmp /mnt/cdrom/<filename> /mnt/reiserfs/1/data/<original_file_location>, using a PLEXTOR Model: CD-ROM PX-40TS SCSI CD-ROM drive),

- and THEN (while the copy or comparison runs) try any simple command (like 'ls /mnt/reiserfs/1/data' or 'top' or anything else...).

Response time is abysmal, a simple 'ls /some/dir' takes tens of seconds to start. Once the command is running, performance is normal. Try this when a cdrecord session is running and you'll get a buffer underrun.

The box has 768 MB of RAM, 512 MB of swap. There is no significant load on the system (according to an already running copy of top) neither before nor during the test. Try tab-completing a command in a terminal, and that terminal freezes for tens of seconds, usually until after the file system load has gone down.

Quoted 6 lines

In a few words, heavy filesystem activity seems to wreak havoc on the system. Not by loading the CPU (it hardly breaks out a sweat at 178% idle (SMP...)).

Turning off swap (swapoff -a) does not change the observed behaviour.

Anyone else seen something like this?

I have 2.4.10-ac12 here and reiserfs filesystems and I have to say that performance is terrible when doing anything diskintensive. It seems like diskscheduling is very broken for my IDE disk, or it's a reiserfs problem. maybe it pushes a _lot_ into the diskscheduling at once?

I've heard that read-ahead for IDE has been broken for a while but's fixed in -ac and in Andre Hedrick's IDE-patches.

I'm going to upgrade to a more recent kernel and see how it behaves.

/Martin

Never argue with an idiot. They drag you down to their level, then beat you with experience.

#4 lore raw
In reply to Frank de Lange
Quoted 1 line · Jeff Garzik

Can you try 2.4.13ac6 (not 7/8), and 2.2.20, and post a comparison?

Here's the results from some tests I did:

2.2.20
======
without filesystem activity
no slowdowns observed
time ls -al /usr/|sort -k 5 -n
real 0m0.121s
user 0m0.000s
sys 0m0.090s

with filesystem activity on ext2
no slowdowns observed
time ls -al /opt/|sort -k 5 -n
real 0m0.079s
user 0m0.010s
sys 0m0.100s

2.4.13-ac5
==========
no slowdowns observed
without filesystem activity
time ls -al /usr/|sort -k 5 -n
real 0m0.142s
user 0m0.000s
sys 0m0.000s

with filesystem activity on ext2
no slowdowns observed
time ls -al /opt/|sort -k 5 -n
real 0m0.022s
user 0m0.020s
sys 0m0.010s

with filesystem activity on reiserfs
- it took 31 seconds to just open this small ( < 1 kb) text file (which resides in my home directory, on an ext2 filesystem) in vi...
time ls -al /usr/|sort -k 5 -n

real    0m6.136s
user    0m0.020s
sys     0m0.020s

2.4.15-pre4
===========
without filesystem activity
no slowdowns observed
time ls -al /usr/|sort -k 5 -n
real 0m0.081s
user 0m0.010s
sys 0m0.010s

with filesystem activity on ext2
no slowdowns observed
time ls -al /usr/|sort -k 5 -n

real    0m0.146s
user    0m0.000s
sys     0m0.020s

with filesystem activity on reiserfs
system behaviour erratic, some slowdowns
time ls -al /opt|sort -k5 -n

real    0m13.232s
user    0m0.020s
sys     0m0.010s

Seems that reiserfs is the common factor here, at least on my box. This is a 35 GB reiserfs filesystem, app 80% used, both large and small files.

As said in my previous message, the numbers themselves don't mean squat. It is the large delays (the fact that user+sys <<< real) which are the problem here.

Any other magic anyone wants me to perform? Hans, you reading this?

Cheers//Frank

Signature · 9 lines
  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \ [email protected]  /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]
#5 lore raw
Quoted 6 lines

Seems that reiserfs is the common factor here, at least on my box. This is a 35 GB reiserfs filesystem, app 80% used, both large and small files.

As said in my previous message, the numbers themselves don't mean squat. It is the large delays (the fact that user+sys <<< real) which are the problem here.

This was also reported as

Suspected bug - System slowdown under unexplained excessive disk I/O - 2.4.13 with huge delays during compiles (sasha Pachev) or mysql-benchmarks (me). But today I do not find this reiser-specific, this also seems to happen with ext3.
But as you wrote, not with ext2. I see that there is more disk-activity due to journaling in both cases, but waiting 30 seconds for simple tasks or waking from screen-apm seems not to be right.

Oktay Akbal

#6 lore raw
In reply to Frank de Lange

On Mon, Nov 12, 2001 at 11:56:42PM +0100, Frank de Lange wrote:
[snip]

Quoted 8 lines

Seems that reiserfs is the common factor here, at least on my box. This is a 35 GB reiserfs filesystem, app 80% used, both large and small files.

As said in my previous message, the numbers themselves don't mean squat. It is the large delays (the fact that user+sys <<< real) which are the problem here.

Any other magic anyone wants me to perform? Hans, you reading this?

Do you see/hear a lot of seeking happing during the delays?

If so, your Reiser partition is probably fragmented to hell...

IIRC this problem is being looked at, check some archives of lkml or reiser...

Mike

#7 lore raw
Quoted 5 lines · Mike Fedyk

On Mon, Nov 12, 2001 at 11:56:42PM +0100, Frank de Lange wrote:
[snip]

Quoted 3 lines

Seems that reiserfs is the common factor here, at least on my box. This is a 35 GB reiserfs filesystem, app 80% used, both large and small files.

Quoted 7 lines
Quoted 5 lines

As said in my previous message, the numbers themselves don't mean squat. It is the large delays (the fact that user+sys <<< real) which are the problem here.

Any other magic anyone wants me to perform? Hans, you reading this?

Do you see/hear a lot of seeking happing during the delays?

Yup this is probably what's happening to me. I didn't think a harddrive could do so many seeks so fast :)

Quoted 1 line

If so, your Reiser partition is probably fragmented to hell...

:(

Quoted 1 line

IIRC this problem is being looked at, check some archives of lkml or reiser...

Ok, thanks

/Martin

Never argue with an idiot. They drag you down to their level, then beat you with experience.

#8 lore raw
In reply to Mike Fedyk
Quoted 10 lines · Martin Josefsson
Quoted 8 lines · Mike Fedyk

On Mon, Nov 12, 2001 at 11:56:42PM +0100, Frank de Lange wrote:
[snip]

Quoted 6 lines

Seems that reiserfs is the common factor here, at least on my box. This is a 35 GB reiserfs filesystem, app 80% used, both large and small files.

As said in my previous message, the numbers themselves don't mean squat. It is the large delays (the fact that user+sys <<< real) which are the problem here.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Any other magic anyone wants me to perform? Hans, you reading this?

Do you see/hear a lot of seeking happing during the delays?

Yup this is probably what's happening to me. I didn't think a harddrive could do so many seeks so fast :)

Check out the thread "reiserfs performance loss" back in oct 12 and 13...

Mike

#9 lore raw
In reply to Frank de Lange
Quoted 5 lines · you

Seems that reiserfs is the common factor here, at least on my box. This is a 35 GB reiserfs filesystem, app 80% used, both large and small files.

As said in my previous message, the numbers themselves don't mean squat. It is the large delays (the fact that user+sys <<< real) which are the problem here.

As another data point, I'm seeing the exact same thing. I haven't tried any non-Linus kernels, though. But recent 2.4.x (x >= 10?) linus kernels with reiserfs have these several-second delays during moderate-to-heavy disk i/o, exactly as you've described. I've seen this on both an SMP PIII system and a UP Athlon.

Signature · 2 lines
Jason Lunz		Trellis Network Security
[email protected]	http://www.trellisinc.com/
#10 lore raw
In reply to Frank de Lange
Quoted 74 lines · Frank de Lange
Quoted 2 lines · Jeff Garzik

Can you try 2.4.13ac6 (not 7/8), and 2.2.20, and post a comparison?

Here's the results from some tests I did:

2.2.20
======
without filesystem activity
no slowdowns observed
time ls -al /usr/|sort -k 5 -n
real 0m0.121s
user 0m0.000s
sys 0m0.090s

with filesystem activity on ext2
no slowdowns observed
time ls -al /opt/|sort -k 5 -n
real 0m0.079s
user 0m0.010s
sys 0m0.100s

2.4.13-ac5
==========
no slowdowns observed
without filesystem activity
time ls -al /usr/|sort -k 5 -n
real 0m0.142s
user 0m0.000s
sys 0m0.000s

with filesystem activity on ext2
no slowdowns observed
time ls -al /opt/|sort -k 5 -n
real 0m0.022s
user 0m0.020s
sys 0m0.010s

with filesystem activity on reiserfs
- it took 31 seconds to just open this small ( < 1 kb) text file (which resides in my home directory, on an ext2 filesystem) in vi...
time ls -al /usr/|sort -k 5 -n

real    0m6.136s
user    0m0.020s
sys     0m0.020s

2.4.15-pre4
===========
without filesystem activity
no slowdowns observed
time ls -al /usr/|sort -k 5 -n
real 0m0.081s
user 0m0.010s
sys 0m0.010s

with filesystem activity on ext2
no slowdowns observed
time ls -al /usr/|sort -k 5 -n

real    0m0.146s
user    0m0.000s
sys     0m0.020s

with filesystem activity on reiserfs
system behaviour erratic, some slowdowns
time ls -al /opt|sort -k5 -n

real    0m13.232s
user    0m0.020s
sys     0m0.010s

Seems that reiserfs is the common factor here, at least on my box. This is a 35 GB reiserfs filesystem, app 80% used, both large and small files.

Quoted 7 lines

As said in my previous message, the numbers themselves don't mean squat. It is the large delays (the fact that user+sys <<< real) which are the problem here.

Any other magic anyone wants me to perform? Hans, you reading this?

Cheers//Frank

Yura, see if you can reproduce this and analyze the cause. If I understand correctly, he is saying the problem is not throughput but latency. Is that correct Frank? Once Yura reproduces it, I will speculate as to the cause.

Hans

#11 lore raw
In reply to Frank de Lange
Quoted 4 lines · Hans Reiser

Yura, see if you can reproduce this and analyze the cause. If I understand correctly, he is saying the problem is not throughput but latency. Is that correct Frank? Once Yura reproduces it, I will speculate as to the cause.

Correct.

Signature · 9 lines
  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \ [email protected]  /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]
#12 lore raw
In reply to Frank de Lange
Quoted 81 lines · Hans Reiser
Quoted 79 lines · Frank de Lange
Quoted 2 lines · Jeff Garzik

Can you try 2.4.13ac6 (not 7/8), and 2.2.20, and post a comparison?

Here's the results from some tests I did:

2.2.20
======
without filesystem activity
no slowdowns observed
time ls -al /usr/|sort -k 5 -n

real   0m0.121s
user   0m0.000s
sys    0m0.090s

with filesystem activity on ext2
no slowdowns observed
time ls -al /opt/|sort -k 5 -n

real   0m0.079s
user   0m0.010s
sys    0m0.100s

2.4.13-ac5
==========
no slowdowns observed
without filesystem activity
time ls -al /usr/|sort -k 5 -n

real   0m0.142s
user   0m0.000s
sys    0m0.000s

with filesystem activity on ext2
no slowdowns observed
time ls -al /opt/|sort -k 5 -n

real   0m0.022s
user   0m0.020s
sys    0m0.010s

with filesystem activity on reiserfs
- it took 31 seconds to just open this small ( < 1 kb) text file (which resides in my home directory, on an ext2 filesystem) in vi...
time ls -al /usr/|sort -k 5 -n

real    0m6.136s
user    0m0.020s
sys     0m0.020s

2.4.15-pre4
===========
without filesystem activity
no slowdowns observed
time ls -al /usr/|sort -k 5 -n

real   0m0.081s
user   0m0.010s
sys    0m0.010s

with filesystem activity on ext2
no slowdowns observed
time ls -al /usr/|sort -k 5 -n

real    0m0.146s
user    0m0.000s
sys     0m0.020s

with filesystem activity on reiserfs
system behaviour erratic, some slowdowns
time ls -al /opt|sort -k5 -n

real    0m13.232s
user    0m0.020s
sys     0m0.010s

Seems that reiserfs is the common factor here, at least on my box. This is a 35 GB reiserfs filesystem, app 80% used, both large and small files.

As said in my previous message, the numbers themselves don't mean squat. It is the large delays (the fact that user+sys <<< real) which are the problem here.

Any other magic anyone wants me to perform? Hans, you reading this?

Quoted 8 lines
Quoted 2 lines

Cheers//Frank

Yura, see if you can reproduce this and analyze the cause. If I understand correctly, he is saying the problem is not throughput but latency. Is that correct Frank? Once Yura reproduces it, I will speculate as to the cause.

Hans

Hello,

Yes, the latency problem exist. I was using "dd" and "cp" commands to create and copy 1 GB file as "filesystem activity".

In both cases the set of :
"time ls -al /opt|sort -k5 -n" show the same delay.

One way to improve the situation is to use the patch below, suggested by Chris Mason :

fs/buffer.c Sun, 18 Nov 2001 12:44:40 -0500 renamed from fs/buffer.c Fri, 16 Nov 2001 10:58:28 -0500 +3 -2
@@ -1020,9 +1020,10 @@
                struct buffer_head * bh;
 
                bh = get_hash_table(dev, block, size);
-               if (bh)
+               if (bh) {
+                       touch_buffer(bh) ;
                        return bh;
-
+               }
                if (!grow_buffers(dev, block, size))
                        free_more_memory();
        }

Thanks,
Yura.