I/O causes performance problem with 2.4.8-ac3

This probably belongs in the "use-once" thread...

I ran into a significant (lack of) performance situation with 2.4.8-ac3 that does not exist with 2.4.8. Perhaps someone can shed some light on what happened and how to avoid it in the future.

The system is a new dual PIII 866MHz VALinux server, 2 GB of RAM, with two eight-disk JBODs connected through a Mylex ExtremeRAID 2000 (i.e., not a pig). I am doing testing of various kernels and filesystems (ext3, reiserfs) to see which will be most appropriate for production use (mostly as an NFS server). Using SMP-enabled kernels.

The system had been up for several hours, I had netscape going, a few emacs windows, untarred and compiled a bunch of linux source trees, built a few RPMS, did some NFS write testing, etc. In other words, system had cached a bunch of buffers.

Performance was excellent until I decided to "dd bs=1024k </dev/cdrom >somefile" a 600+MB cdrom while a kernel build was going on. It took nearly 7 minutes to complete the dd and near the end, the cdrom drive light was only occasionally flickering activity (not "on" as it was at the start), keystrokes were delayed, refreshes were sluggish. "top" showed the loadavg was hovering around 4-5, and the top-runnng processes were kswapd, kreclaimd, and kjournald and they were taking 30-60% of the processor time. Top also showed 3.7 MB free, 425 MB buff, and about 1.5 GB cached. I think the first was a "dd" to an ext3 fs, then I tried dumping to a reiserfs, with similar performance problems.

So this morning, I built a 2.4.8 SMP kernel with only the ext3 patches applied and have been unable to replicate the problem. The "dd" completed in under 3 minutes, while free memory dropped to lowish-levels, (fluctuates between 5-20 MB), and system response and loadavg were "reasonable." It appears when kswapd kicks in, I get another 5-10 MB free.

So, what is one in my situation to gather from this?

--Chris

[...]

[...]
Such a big copy operation is exactly what "use-once" does well. 2.4.8 has use-once, 2.4.8ac3 don't have use-once.

One can construct scenarios where use-once performs worse too, I believe this is why Alan Cox didn't want it yet.

A big copy without use-once will push everything else out of cache, and push a lot of programs into swap in order to cache a lot of the big copy. That's bad if the big
copy is done once and you don't really need the stuff again. Then you want the "other" stuff to stay in cache instead.

Use-once may perform worse if stuff falls out of cache and has to be re-read from disk.

Helge Hafting

Most of the current use once code is for the page cache, dd if=/dev/cdrom will read from the buffer cache, where the pages will take longer to age out.
One of the patches between 2.4.8pre7 and 2.4.8final seems to have helped a
lot here, but I'm not sure which one it was (check out the 'still buffer cache problems' thread). I'm also not sure if the speedup was intentional, so it might go away during future tweaks.

Rik also posted a patch in 2.4.8-ac5 that deactivates buffer cache pages after a threshold, which should help as well (and benefit reiserfs/ext3 in general).

-chris