severe slowdown with 2.4 series w/heavy disk access
I have been having this problem with the whole 2.4 kernel series.. Under heavy disk access, the entire system will slow down and almost all of my memory, save 5 megs, gets used up, never to return. I am currently running 2.4.17 on a machine with 416 megs of ram, Duron 750, not overclocked. Cpu temp does not exceed 107 deg F ever, so I don't think its a heat issue. I have an MSI K7T Pro2 motherboard, using the KT133 chipset. Anyway, the following is how I can duplicate the problem, in about 5 minutes. I only used mozilla for this because I was working with it when I decided to isolate the problem. Also note this happens more than just with tar, and sometimes it happens for no apparent reason at all.
*** free and ps -ax, before the slowdown:
total used free shared buffers
cached
Mem: 417472 30104 387368 0 1264
21748
-/+ buffers/cache: 7092 410380
Swap: 136544 0 136544
PID TTY STAT TIME COMMAND
1 ? S 0:04 init
2 ? SW 0:00 [keventd]
3 ? SWN 0:00 [ksoftirqd_CPU0]
4 ? SW 0:00 [kswapd]
5 ? SW 0:00 [bdflush]
6 ? SW 0:00 [kupdated]
63 ? S 0:00 /usr/sbin/syslogd
66 ? S 0:00 /usr/sbin/klogd -c 3
78 ? S 0:00 /usr/sbin/atd -b 15 -l 1
90 tty1 S 0:00 -bash
91 tty2 S 0:00 -bash
92 tty3 S 0:00 /sbin/agetty 38400 tty3 linux
93 tty4 S 0:00 /sbin/agetty 38400 tty4 linux
94 tty5 S 0:00 /sbin/agetty 38400 tty5 linux
95 tty6 S 0:00 /sbin/agetty 38400 tty6 linux
192 tty2 S 0:00 top
200 tty1 R 0:00 ps -ax*** I then rm -rf'd mozilla, and tar -zxvf mozilla-source-0.9.7.tar.gz, *** and immediately after, ran free and ps -ax again. The system started
*** losing memory and slowing down about 10 seconds into the *** decompression.
total used free shared buffers
cached
Mem: 417472 412192 5280 0 20632
315680
-/+ buffers/cache: 75880 341592
Swap: 136544 0 136544
PID TTY STAT TIME COMMAND
1 ? S 0:04 init
2 ? SW 0:00 [keventd]
3 ? SWN 0:00 [ksoftirqd_CPU0]
4 ? SW 0:00 [kswapd]
5 ? SW 0:00 [bdflush]
6 ? SW 0:02 [kupdated]
63 ? S 0:00 /usr/sbin/syslogd
66 ? S 0:00 /usr/sbin/klogd -c 3
78 ? S 0:00 /usr/sbin/atd -b 15 -l 1
90 tty1 S 0:00 -bash
91 tty2 S 0:00 -bash
92 tty3 S 0:00 /sbin/agetty 38400 tty3 linux
93 tty4 S 0:00 /sbin/agetty 38400 tty4 linux
94 tty5 S 0:00 /sbin/agetty 38400 tty5 linux
95 tty6 S 0:00 /sbin/agetty 38400 tty6 linux
192 tty2 S 0:01 top
207 tty1 R 0:00 ps -axIt is quite possible your MB routes all disk access straight through the processor, while using system ram to cache itself. Tar does not (to my knowledge) limit itself, and is quite nasty on such things. My MB has had troubles as such before. I know of no workaround, except get a new MB.
Quoted 1 line
<><>< Idrigal of Imladris
----- Original Message -----
From: "Paul Boley" <[email protected]>
To: <[email protected]>
Sent: Tuesday, 25 December, 2001 11:17 PM
Subject: severe slowdown with 2.4 series w/heavy disk access
Quoted 79 lines
I have been having this problem with the whole 2.4 kernel series.. Under heavy disk access, the entire system will slow down and almost all of my memory, save 5 megs, gets used up, never to return. I am currently running 2.4.17 on a machine with 416 megs of ram, Duron 750, not overclocked. Cpu temp does not exceed 107 deg F ever, so I don't think its a heat issue. I have an MSI K7T Pro2 motherboard, using the KT133 chipset. Anyway, the following is how I can duplicate the problem, in about 5 minutes. I only used mozilla for this because I was working with it when I decided to isolate the problem. Also note this happens more than just with tar, and sometimes it happens for no apparent reason at all.
*** free and ps -ax, before the slowdown:
total used free shared buffers
cached
Mem: 417472 30104 387368 0 1264
21748
-/+ buffers/cache: 7092 410380
Swap: 136544 0 136544PID TTY STAT TIME COMMAND 1 ? S 0:04 init 2 ? SW 0:00 [keventd] 3 ? SWN 0:00 [ksoftirqd_CPU0] 4 ? SW 0:00 [kswapd] 5 ? SW 0:00 [bdflush] 6 ? SW 0:00 [kupdated] 63 ? S 0:00 /usr/sbin/syslogd 66 ? S 0:00 /usr/sbin/klogd -c 3 78 ? S 0:00 /usr/sbin/atd -b 15 -l 1 90 tty1 S 0:00 -bash 91 tty2 S 0:00 -bash 92 tty3 S 0:00 /sbin/agetty 38400 tty3 linux 93 tty4 S 0:00 /sbin/agetty 38400 tty4 linux 94 tty5 S 0:00 /sbin/agetty 38400 tty5 linux 95 tty6 S 0:00 /sbin/agetty 38400 tty6 linux 192 tty2 S 0:00 top 200 tty1 R 0:00 ps -ax*** I then rm -rf'd mozilla, and tar -zxvf mozilla-source-0.9.7.tar.gz, *** and immediately after, ran free and ps -ax again. The system started
*** losing memory and slowing down about 10 seconds into the *** decompression.total used free shared buffers
cached
Mem: 417472 412192 5280 0 20632
315680
-/+ buffers/cache: 75880 341592
Swap: 136544 0 136544PID TTY STAT TIME COMMAND 1 ? S 0:04 init 2 ? SW 0:00 [keventd] 3 ? SWN 0:00 [ksoftirqd_CPU0] 4 ? SW 0:00 [kswapd] 5 ? SW 0:00 [bdflush] 6 ? SW 0:02 [kupdated] 63 ? S 0:00 /usr/sbin/syslogd 66 ? S 0:00 /usr/sbin/klogd -c 3 78 ? S 0:00 /usr/sbin/atd -b 15 -l 1 90 tty1 S 0:00 -bash 91 tty2 S 0:00 -bash 92 tty3 S 0:00 /sbin/agetty 38400 tty3 linux 93 tty4 S 0:00 /sbin/agetty 38400 tty4 linux 94 tty5 S 0:00 /sbin/agetty 38400 tty5 linux 95 tty6 S 0:00 /sbin/agetty 38400 tty6 linux 192 tty2 S 0:01 top 207 tty1 R 0:00 ps -ax-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 4 lines
total used free shared buffers
cached
Mem: 417472 412192 5280 0 20632
315680
Thyose values dont show any problems. In fact your machine seems to think it had a ton of free memory to waste and has let it fill up with stuff that has been accessed - just on the chance that it may be reused.
It hasn't even felt enough memory pressure to start swapping. When you say it "becomes slow", what precisely becomes slow ?
Also what disks do you have and how are they set up ?
Quoted 6 lines · Alan Cox
Quoted 5 lines
total used free shared buffers
cached
Mem: 417472 412192 5280 0 20632
315680
Quoted 6 lines
Thyose values dont show any problems. In fact your machine seems to think it had a ton of free memory to waste and has let it fill up with stuff that has been accessed - just on the chance that it may be reused.
It hasn't even felt enough memory pressure to start swapping. When you say it "becomes slow", what precisely becomes slow ?
When this happens in X, the mouse drags and skips, any processes running (like tar/gzip. ls in an empty dir takes about 10 seconds) slow down, and it happens usually for about 10sec-2min, often for no apparent reason. The big decompression was just a way I can easily duplicate it. Oddly enough though, according to top, it caches all that memory at once, and my free goes down to 5 megs, with the system hanging/slow to respond, for 10sec-2min. Even typing in the console has delay before the characters appear, and according to top, tar and gz are both using under 1% cpu while this happens, and about 50% of the cpu is in use by the system (not by any processes that I can see. kupdated goes up to about 0.3% during this)
Quoted 3 lines
Also what disks do you have and how are they set up ? -
/dev/hdb3 on / type ext2 (rw)/dev/hdb4 on /home type ext2 (rw)/dev/hda1 on /dos/c type vfat (rw)/dev/hdb1 on /dos/d type vfat (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
none on /proc type proc (rw)
and my swap is /dev/hdb2
Quoted 16 lines · Paul Boley
When this happens in X, the mouse drags and skips, any processes running (like tar/gzip. ls in an empty dir takes about 10 seconds) slow down, and it happens usually for about 10sec-2min, often for no apparent reason. The big decompression was just a way I can easily duplicate it. Oddly enough though, according to top, it caches all that memory at once, and my free goes down to 5 megs, with the system hanging/slow to respond, for 10sec-2min. Even typing in the console has delay before the characters appear, and according to top, tar and gz are both using under 1% cpu while this happens, and about 50% of the cpu is in use by the system (not by any processes that I can see. kupdated goes up to about 0.3% during this)
Quoted 3 lines
Also what disks do you have and how are they set up ? -
Quoted 6 lines
/dev/hdb3on / type ext2 (rw)/dev/hdb4on /home type ext2 (rw)/dev/hda1on /dos/c type vfat (rw)/dev/hdb1on /dos/d type vfat (rw)
none on/dev/ptstype devpts (rw,gid=5,mode=620)
none on /proc type proc (rw)
Are your hard drives configured to use DMA? I noticed that when using PIO in linux, you definitely get some user responsiveness problems whenever you are doing heavy disk IO. Try turning dma mode on. man hdparm. However, if you say you are getting like 10sec-2min lags in responsiveness.. i somehow doubt that is the true culprit in this case...
Also, are you using NFS and/or NIS? These things can also really slow down some apparently low-cost things like running ls and the like. You might also have bad sectors on your hard drives. I know some hard drives (or is it just the drivers?) internally try to repeat a read or write request several times before reporting an error. Sometimes your hard drives' media are slightly bad and thus exhibit huge slowdowns whenever they operate on the offending sectors... the drive keeps retrying and then it succeeds with the only visible symptom being that you waited a really long time for the request to complete. During this time, of course, the whole drive is retarded and cannot be accessed at all by the driver...
These are just some ideas. Of course, since it's not so likely to be the kernel, I thought we could rule out other more likely suspects. :)
-Calin
Quoted 5 lines
When this happens in X, the mouse drags and skips, any processes running (like tar/gzip. ls in an empty dir takes about 10 seconds) slow down, and it happens usually for about 10sec-2min, often for no apparent reason. The big decompression was just a way I can easily duplicate it. Oddly enough though, according to top, it caches all that memory at
Ok
Quoted 1 line
once, and my free goes down to 5 megs, with the system hanging/slow to
The free behaviour is correct (free memory is wasted memory). The delays are obviously not
Quoted 6 lines
under 1% cpu while this happens, and about 50% of the cpu is in use by the system (not by any processes that I can see. kupdated goes up to about 0.3% during this)
Quoted 2 lines
Also what disks do you have and how are they set up ? -
[I meant are they in DMA / UDMA modes ?]
Alan
Quoted 11 lines · vda
Quoted 7 lines
total used free shared buffers
cached
Mem: 417472 412192 5280 0 20632
315680
-/+ buffers/cache: 75880 341592
Swap: 136544 0 136544It seems you think your memory is used for no purpose, but kernel just keeps page cache in your RAM (kernel bugs are indeed possible, but your test does not show any buggy behavior IMHO).
My whole system slows down (commands take a long time to execute, decompression slows, ls in an empty dir takes 10 sec)
Quoted 5 lines
To verify this, you may repeat this experiment on a separate partition:
1) mount a partition
2) do the test as you described
3) umount the partition
I did this, and it uncached some, but I only had 60megs (out of 416) free after unmounting the partition. The cache went down to about 5 megs, and in-use was at 350 megs.
Quoted 6 lines
I believe you should see tons of free memory then, especially if your tarfile is also on that partition.
Please report back if you would do the test.
--
vda