why swap at all?

(Disclaimer: I don't know all that much about the current scheme.)

First, I don't believe this works. A couple weeks ago I did

# cp -a <~100GB> <different physical disk>

and my system was nearly unusable for a few hours. This is Athlon 64 3200+, 512MB RAM, DMA on on both drives, iowait time around 90%. So this was an io/pagecache problem.

The benchmark involved was ls. It took several seconds. If I ran it again in 5 seconds or so, it was fine. Much longer and it would take several seconds again. Sounds like pages getting evicted in LRU order.

I have this problem not only on every linux kernel I've ever tried (on different computers) but on other OS's as well. It's not an easy one to solve.

For kicks, I checked out vmstat 1 (I don't have a copy right of the output). It looked like cp -a dirtied pages as long as it could get them, and they got written out as quickly as they could. And, for whatever reason, the writes lag behind the reads by an amount comparable to the size of my physical memory.

It seems like some kind of limiting/balancing of what gets to use the cache is needed. I bet that most workloads that touch data much larger than RAM don't benefit that much from caching it all. (Yes, that kernel-tree-grep from cache is nice, but having glibc in cache is also nice.)

Should there be something like a (small) limit to how many dirty, non-mmaped pages a task can have? I have no objection to a program taking longer to finish because the 100MB it writes need to mostly hit the platter before it returns, since, in return, I get a usable system while it's running and it's not taking any more CPU time.

Second (IMHO) a "used once" heuristic has a fundamental problem:

If there are more pages "used more than once" _in roughly sequential order_ than fit in memory, then trying to cache them all is absurd. That is, if some program makes _multiple passes_ over that 100GB (mkisofs?), the system should never try to cache it all. It would be better off taking a guess (even a wild-ass-guess) of which 200MB to cache plus a few MB for readahead, leaving pages from other programs in cache for more than a few seconds, and probably getting better performance (i.e. those 200MB are at least cached next time around).

Is any of this reasonable?

--Andy

By what magic system can know that you are going to do ls again in 2 minutes?

Does is happen if you do ls several times in a row (to make needed pages not-once-used), then wait a bit and do ls again?

cp should use fadvise() and say that it _really_ does not need those pages.

Easier said than done... Why 200MB and not 400? etc...

If you think you see VM misbehavior,
1) verify that it is indeed MISbehaving
2) produce useful bug report
3) report it and track it until fixed

Apps take ages to start after cache being trashed because they are bloated. Fight bloat. Join uclibc/dietlibc/etc efforts.

Random example: why on earth ntpd daemon have RSS of ~1.5 Mb???! --
vda

The problem is more about the blocks cp touches, less about predicting the ls workload.

Yes, indeed. On the other hand the sequential read could be detected by the kernel, too.

Greetings
Bernd

eckes privat - http://www.eckes.org/
Project Freefire - http://www.freefire.org/

Looks like it was. ls' read was sequential, too, so it did not get any advantage. If you can definitely show that streaming io (say, cat hugefile >/dev/null) flushes _non_ sequentially read data (pages with program/library code, data of e.g. your Mozilla, etc), please submit a report to lkml. VM gurus said more than once that they _want_ to fix things, but need to know how to reproduce. --
vda

Yep.

Thanks to everyone's input I was able to test and adapt my mm work. It is hopefully at a stage where it can have wider testing now. It is stable on my SMP system under very heavy swapping, but the usual caution applies.

Test is 4 x cat 8GB > /dev/null (aggregate 100-200MB/s!) while in X, with xterms and mozilla open browsing and grepping kernel tree, etc.

Plain 2.6.7-rc1-mm1 swapped 200MB then completely froze up the system within 9 seconds of starting the read load. Things remained fairly responsive with my patch applied. A bit of swap out, but very little swap in, which is good. The entire 32GB went through the pagecache no problem.

A couple of concurrent mkisofs's writing 4 GB isos don't seem to be any problem either with the patched kernel. Haven't tried plain -mm yet.

http://www.kerneltrap.org/~npiggin/nickvm-267r1m1.gz

It is a cocktail of cleanups, simplification, and enhancements. The main ones that applie here is my split active lists patch (search archives for details), and explicit use-once logic.

Known issue: page reclaim can get a little bit lumpy (ie lots of memory freed up at once), but that is just a matter of teaching things not to bite off massive chunks at a time when it starts hitting memory pressure.

This isn't the "very heavy swapping" load, BTW :)

The very heavy swapping load is make -j15 in 64MB of memory.

Thanks for testing Matt. The patch I posted is the newest rollup I have. I haven't run into any significant problems with it yet, but I will notify lkml if I have found a bad bug or made significant changes.

Nick

I didn't have time to personally test it but just want to share some thoughts. This kind of improvement is absolutely necessary for 2.6 to be usefull on the desktop. It continues to escape me how come that this kind of, almost, bugfix arrives so late during stable period.

Unfortunately it doesn't apply on top of Con's autoregulate swappines (AM from now on) which I am currently testing. AM also does an excellent job in preventing swap trashing while doing a lot of filesystem reading.

I am curious on how does your patch technically relates to Con's AM, and is it possible to merge the two? I know that they do their job on completely different ways, so the real question would be: Is there a need for something like AM if using your patch, or it completely obsoletes it?

I had a quick look at Nick's patches to see for you and it appears that Nick's work completely removes the swappiness dial and introduces a different metric called vm_mapped_page_cost which is not interchangeable with the swappiness value. ie they cannot work together, sorry.

Con

Thanks!

This feels quite a bit better on my system. I'll try and stress it a bit more later today or tomorrow, but my system is now usable under heavy io load.

--Andy

I'm not sure. Copying a file is a pretty good indication that you are about to do something with either the new or the old file.

Thanks,

Jim

Like taking the new file with me on USB dongle and deleting old one? Caching the file really doesn't help in this case.

Lenar

No, and most file copies are not to be used in the "near" future. I.e. on my machine. Caching on the second read (close in time) is ok, or if there are unused ram, but paging out things in use is bad. It's much more likely that the page allocated to a program will be used than a newly read or written file.

Ofcourse your milega may vary. I'm thinking of my desktop now.

-- robin

No, it does not help in this case.

Not putting things in cache is a solution for the problem of having useful stuff pushed out of the cache. However, fixing the problem this way may create other problems if it causes us to fail to put useful things into the cache.

The point I was trying (perhaps unsuccessfully) to make, is that we should be careful about not caching things. We are likely to break other corner cases by fixing the ones we are discussing.

Thanks,

Jim

Hi,

From a pure *users* perspective the most evil part of my computer is the harddrive, because it has three bad properties;

-- its slowing stuff down
-- its very noisy
-- its sucks my battery dry (on laptops)

I dont care as much about how fast or efficient the swapping system really is, what does annoy me is when it lets me down. Say I do some action repeatedly every once in a while and it always takes about X milliseconds, when after a while that same action takes 2*X milliseconds. Now I'm DISAPPOINTED and I got a grudge with kswapd.

Its *emotionally* more 'okay' for the harddrive to its thing when I initiate it through some action (launching program, copying files) because then I expect it.

From what I've read previously in this thread, it seems to me that the only major problem with swapping that not all users want file system cache to swap out actual applications (thus making that somewhat aged mozilla window abit laggy).

Maybe we could just have a "Allow file system cache to swap out applications checkbox somewhere"?

Or, Am I missing something?

Sincerly,
/m

Isn't that what /proc/sys/vm/swappiness is for? (And yes, I know that it's not a total solution to the whole range of conflicting requirements that we have - I suspect that it's really difficult to make it work much better than it is without adding a lot more history-tracking instrumentation).

In practice, that would make no difference at all. Once physical memory is
full (and it pretty much will always be so), every memory request (whether due to the file system cache or application usage) will require discarding some page or other. So even if all memory requests due to file system cache usage were prohibited from forcing out application pages, you're launching enough other application that need pages that application pages will still be evicted.

Now, if you make the rule "don't ever swap out application pages", what
exactly is the swap going to do? Swap is for dirty pages. Dirty file pages get written back to their ultimate home, not swap.

DS

No.

Many people have machines with plenty of RAM (512MB or more is pretty much standard on new machines), much of which is only used to cache files. The file cache is the reason the memory is full.

-- robin

Huh?

Of course. That's why I said, "once physical memory is full (and it pretty
much will always be so)". Physical memory is always full, so every memory request requires that a page be evicted.

DS

I've experienced the problem where applications need to be swapped back in. It's mainly caused by the dual role my machine has (desktop machine when I'm using it, server when it is serving files). Whenever my machine has been sitting idly serving files for a while, when I get back, the desktop is slow. However, there is no need for that, as the files are served at low speeds -- there's no real point in caching them apart from maybe preventing harddisk wear... the harddisk itself can serve these files again faster than they will be needed.

So perhaps it is possible to reduce caching of data that is simply not putting stress on the system (the harddisk in this case). If the harddisk is not the bottleneck, it is probably not worth caching. Typical examples are letting a box play music all day (and then trying to read your mail...), having a webserver on a slow connection or watching a large movie file. None of these really require much caching beyond a bit of read-ahead.

I'm not sure how best to distinguish when something is fast I/O that would benefit from caching and when something is slow I/O that the harddisk can handle well enough on its own.

--John

It is?

Sorry for butting in folks, but I've been reading this thread hoping to see some possible solutions. Seems that a survey of best practices might have been suggested, however, I haven't seen such a suggestion.

So here goes, might it not be of some benefit to see how other operating systems (there are rather large number) handle the use of memory. For just one example you could look at MVS, where the application can request through various means how and how much memory it uses. Most defaults can be overridden by the scripting language used to run the application. This also true of other operating systems.

I would be more willing to say that the folks setting up the running of systems should have far more control over the use or non use of cache backed I/O data.

Now that I've said that you have to consider how and where this control should be based.

<soapbox>

SWAP is a solution for the age old whine, I caused the system to run out of memory and the big mean operating system terminated my application.

These days it allows the performance of the system to degrade to the point that the whine goes, The big mean operating system is taking forever to run my 10 TB backup and, by the way, it takes 3 days to wake up my openoffice application that I started a week ago.

Ain't progress grand ;-)

</soapbox>

Cheers,
Dave