why swap at all?

As a general question about ram/swap and relating to some of the issues in this thread:

~500 megs cached yet 2.6.5 goes into swap hell

Consider this: I have a desktop system with 256MB ram, so I make a 256MB swap partition. So I have 512MB "memory" and if some process wants more, too bad, there is no more.

Now I buy another 256MB of ram, so I have 512MB of real memory. Why not just disable my swap completely now? I won't have increased my memory's size at all, but won't I have increased its performance lots?

Or, to make it more appealing, say I initially had 512MB ram and now I have 1GB. Wouldn't I much rather not use swap at all anymore, in this case, on my desktop?

-Anthony
http://nodivisions.com/

This is a really good point. I think the bar should be set at max performance for systems that never need to use the swap device.

If someone wants to tune swap performance to their hearts content, so be it. But given cheap prices for memory, and the horrible best case performance for swap, an increase in swap performance should never, ever come at the expense of performance for a system that has been sized such that executable address spaces, libraries and anonymous memory will fit easily within physical ram.

This of course doesn't address the VM paging storms that happen due to large amounts of file system writes. Once the pagecache fills up, dirty pages must be evicted from the pagecache so that new pages can be added to the pagecache.

--Buddy

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Anthony DiSante
Sent: Tuesday, May 25, 2004 11:38 PM
To: [email protected]
Subject: why swap at all?

As a general question about ram/swap and relating to some of the issues in this thread:

~500 megs cached yet 2.6.5 goes into swap hell

Consider this: I have a desktop system with 256MB ram, so I make a 256MB swap partition. So I have 512MB "memory" and if some process wants more, too bad, there is no more.

Now I buy another 256MB of ram, so I have 512MB of real memory. Why not just disable my swap completely now? I won't have increased my memory's size at all, but won't I have increased its performance lots?

Or, to make it more appealing, say I initially had 512MB ram and now I have 1GB. Wouldn't I much rather not use swap at all anymore, in this case, on my desktop?

-Anthony
http://nodivisions.com/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

If you've got a real performance issue, please describe it properly instead of asserting without evidence the existence of one.

-- wli

No. I am not making any assertions whatsoever. I am just calling out that systems that run happily from physical memory and are not in need of swap should never sacrifice an ounce of performance for even drastic improvements to swap performance. Swap is a band-aid for saving money on memory and a few years ago, it allowed you to save a substantial amount of money.

Whether the cost savings for utilizing swap vs buying more memory are substantial as of late is subject to opinion, but I cannot think of a system that I have sized in the last three years where swap was expected to be used except in un-anticipated memory shortfalls. In fact, if I didn't plan to store crash dumps on the swap device, I think I would have omitted swap all together in many configurations.

I have worked at large fortune 500 companies with deep pockets though, so this may not be the case for many. I make this point though because I think if it isn't the case yet, it will be in the near future as memory becomes even cheaper because the trend certainly exists.

As for your short, two sentence comment below, let me save you the energy of insinuations and translate your message the way I read it:

-------------------------------------------------------------------------
I don't recognize your name, therefore you can't possibly have a valuable opinion on the direction VM system development should go. I doubt you have an actual performance problem to share, but if you do, please share it and go away so that we can work on solving the problem.
--------------------------------------------------------------------------

My response:

Get over yourself.

Regards,

--Buddy

-----Original Message-----
From: William Lee Irwin III [mailto:[email protected]]
Sent: Wednesday, May 26, 2004 12:55 AM
To: Buddy Lumpkin
Cc: [email protected]; [email protected]
Subject: Re: why swap at all?

it.

executable

large

must

If you've got a real performance issue, please describe it properly instead of asserting without evidence the existence of one.

-- wli

Swap serves another (often underrated) purpose: Graceful degradation.

If you have a reasonably amount of swap space mounted, you will know you are running out of RAM because your system will become noticeably slower. If you have no swap whatsoever, your first warning will quite possibly be an application OOM killed or losing data due to a failed memory allocation.

Think of the slowness of swap as a _feature_.

Roger

Yes, you can run swapless. Nothing wrong with that. --
vda

Hi Buddy,
Even for systems that don't *need* the extra memory space, swap can actually provide performance improvements by allowing unused memory to be replaced with often-used memory.

For example, I have 57MB swapped right now. It allows me to instantly grep the kernel tree. If I turned swap off, each grep would probably take 30 seconds.

The VM doesn't always get it right, and to make matters worse, desktop users don't appreciate their long running jobs finishing earlier, but *hate* having to wait a few seconds for a window to appear if it hasn't been used for 24 hours.

[snip]

I think the comment was rather directed at a specific problem you described:

This sounds like you are having a serious problem, and it would be great if you could describe it in detail. kernel version, workload, description of the system, vmstat output, etc.

Let's keep it nice.

Best regards
Nick

This is correct. You now have 512M of fast memory instead of
256M fast memory and 256M "slow" memory. You don't _need_ to have additional swap, but it is usually a good idea. If you keep your 256M of swap, then you now have 512M fast memory + 256M slow memory for a total of 768M. This is even better.

Please note that your machine _will_ do one kind of swapping even if you don't configure any swap: Executable files are a kind of swap-files, if memory pressure happens then (part of) your programs will be evicted from memory _because_ they can be reloaded from their executables.

This cause the same sort of performance degradations as swapping to a swap partition. Actually, it is worse because swapping to a swap partition allows swapping out little-used writeable memory before discarding program code that might see more use. So if swapping happens, then you're better off with a swap partition because then it is the least used stuff that goes first. Without a swap partition, the least used program code goes, but it may or may not be the least used memory overall.

Helge Hafting

Quote from Anthony DiSante <[email protected]>:

In my experience, it's perfectly possible to run a typical desktop system with no swap at all. Certainly the 'double the amount of physical RAM' guideline has been taken far too literally in my opinion.

As you point out, if a typical system works fine with 512 Mb of storage, it shouldn't matter what the mix of physical and virtual memory is. Of course, it will make a difference to performance, and there is a minimum practical amount of real RAM because some things are unswapable, but in my opinion it is absolutely wrong to size swap partitions simply by looking at the amount of physical RAM in a system and not considering the requirements of the workload.

Double the physical RAM is usually more than enough these days, and because most of the time the negative effects of too little swap are more noticable than too much swap, this 'rule' seems to work well enough.

See my recent posts in another thread about solving computing problems by learning solutions to common scenarios and not learning about computers in a generic way - in my opinion, the widespread inefficiency of swap space is a classic example of how this simply doesn't work very well.

Instead of trying to work out how much swap space is needed for a particular hardware configuation, I suggest that users look at the workload first.

Infact, it's generally sensible to think about the workload before buying any hardware.

John.

What the Hell? I have enough bugs I'm paid to fix that I'm not going to tolerate harassment for requesting that claims that the kernel behaves pathologically in some scenario be cast as comprehensible bugreports. It's also worth noting that paying customers don't respond so uncouthly.

-- wli

Quote from Roger Luethi <[email protected]>:

There is a very negative side to this approach as well, especially if users allocate excessive swap space.

A run-away process on a server with too much swap can cause it to grind to almost a complete halt, and become almost compltely unresponsive to remote connections.

If the total amount of storage is just enough for the tasks the server is expected to deal with, then a run-away process will likely be terminated quickly stopping it from causing the machine to grind to a halt.

If, on the other hand, there is excessive storage, it can continue running for a long time, often consuming a lot of CPU.

When the excess storage is physical RAM, this might not be particularly disasterous, but if it's swap space, it's much more likely to cause a serious drop in performance.

For a desktop system, it might not be a big deal, but when it's an ISP's server in a remote data centre, it can create a lot of unnecessary work.

John.

Quote from Nick Piggin <[email protected]>:

That's true, but it's not a magical property of swap space - extra physical RAM would do more or less the same thing.

John.

I'm not sure your optimism about the correct (run-away) process being terminated is justified. Granted, there are definitely scenarios where swapless operation is preferable, but in most circumstances -- especially workstations as the original poster described -- I'd rather minimize the risk of losing data.

Roger

Quote from Helge Hafting <[email protected]>:

I strongly disagree on the last point. It may be better, but it may also be a lot worse. Too much swap can be a bad thing - see my example in another post about run-away processes on remote machines.

Again, the user _may_ be better off swapping to a swap partition rather than having executable code paged out, but this is not necessarily true in all circumstances.

John.

Come on, that is quite an exaggeration. It can happen in a span of minutes -- after rsyncing a dir to a backup dir, for example, which fills ram rather quickly with cache I'll never use again. Or after configuring and compiling a package, which does the same thing.

As you said, the VM doesn't, in fact, always get it right. If 512MB worked before when it was half swap, 512MB of pure ram will work too, only faster.
I don't see how adding more swap at that point could increase performance
unless you are keeping your ram full of non-cached pages, and that's never the case for me -- my ram is almost always half cached pages.

-Anthony
http://nodivisions.com/

Well it is a magical property of swap space, because extra RAM doesn't allow you to replace unused memory with often used memory.

The theory holds true no matter how much RAM you have. Swap can improve performance. It can be trivially demonstrated.

rsync is something known to break the VM's use-once heuristics. I'm looking at that.

It can.

I do that on embedded systems. no swap. when there's no more, the oomkiller kicks in and removes a few extraneous processes...

The other way around can be "demonstrated" equally trivially.

In my personal machine i have 3GB of RAM and i regularly create DVD-ISO-Images (about 2 per day). After creating an image (reading up to 4,4GB and writing up to 4,4GB) the cache is 100% trashed(1). With swap it would be even more trashed then it is without swap(1).

1: This has "always(tm)" been so since i began burning DVDs 3 years ago. Beginning from kernel 2.4.4-2.4.25 and 2.6.4-2.6.6. Currently i use 2.6.5. (This is no typo!)

I have only tested the "with swap"-case with 2.4.4 as i didn't use swap after 2.4.4 trashed so badly with swap enabled. But i don't think that things have changed so fundamentaly that the "with swap"-case is better(FOR ME!) than the "without swap"-case.

Bis denn

Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

Quote from Roger Luethi <[email protected]>:

Well, I am basing this on experience. I know an ISP who had their main customer webserver down for hours because of this kind of problem - the whole thing created a lot of work and wasted a lot of time.

In this particular scenario, I think the run-away process was probably using up almost two thirds of the total RAM, so I'm pretty confident the correct process would have been terminated.

I know that trusting the kernel to terminate the correct run-away process might seem like a bit of a risky approach to take with respect to loosing data, especially where a little bit of swap space might come to the rescue.

However, in my opinion, if a machine has insufficient storage for the intended task then that's an error condition straight away. So, I am not really concerned with trying to make sure that a desktop system running an application which the user has underestimated the memory usage of doesn't crash no matter what. The machine is operating in an error condition, so data loss should be expected.

No, I am more concerned about preventing unexpected usage of a machine from causing large scale slowdowns, and unavailability to other users.

For example, if a run-away process occurs, or one user on a multi-user system uses up excessive resources.

Excessive swap space might create an illusion of protecting against data loss, by allowing things to continue working no matter what, just a bit slower, but for multi user systems, it's preventing normal usage of the system. This can indirectly lead to data loss if the machine is not accessible over the network to perform a critical function.

Ultimately, once a machine is spending 99% of it's time swapping, it's likely to be well past the point where it's practical to log in remotely and fix it.

However, I think that there are probably more machines using excessive swap which would benefit from reducing it, than the other way round, though, simply because users are not as aware of the potential problems.

My opinion was that the machine was already in an error condition the minute I couldn't access it remotely - a significant number of customer's webpages were inaccessible, which potentially means lost business for them.

I assume that the scenario you were thinking of when you mentioned data loss above was a system running a critical process which is using, for example, 90% of the available storage - in that case if another process starts up, and uses up the rest of the available storage, then the first process will probably be terminated, whereas if you increase the amount of storage, (either by adding swap or physical RAM), then the second process can continue for longer.

However, in this situation, I can see two possibilities - either the second process is genuine, (I.E. not a run-away process), in which case the machine has insufficient storage for it's intended purpose, which is an operator error in my opinion, or the process is a run-away process, in which case a little extra storage isn't going to do much other than buy time before the first process is terminated. This may give an operator chance to log in and fix the problem, (probably by terminating the run-away process), but if this extra storage is swap space, the machine may well become unresponsive very quickly, making it virtually impossible to log in remotely, and making other network services on that machine virtually inaccessible. Eventually the run-away process may use up the swap space, and then the first process will probably be terminated as before, just not as quickly. If instead of a little extra storage, a lot was added such that the first process was no longer using more storage than the run-away process was when storage was full, then the kernel will hopefully terminate the run-away process, but probably only after the machine has been unresponsive for a long time, possibly causing other problems.

Basically, I would be skeptical about using a desktop system where one terminated process could cause data loss to the extent that I couldn't easily restore the data.

John.

I don't disagree that you could find a situation where swap is worse than no swap. I don't understand what you mean by trashed and more trashed though :)

Creating your ISOs makes your system swap a lot when swap is enabled?

The 2.6 VM has changed pretty fundamentally. It would be good if you could retest.

Your analogy is flawed. There are many reasons why this doesn't work in the real world.

I don't think any modern and popular OS contains mechanisms that silently stage old pages to disk. The constant twitching of the hard drive this causes for no apparent reason drives people insane and drains precious battery life on laptops. (see description for the pages_min, pages_low and pages_high watermarks for clarity)

Pages are evicted from memory due to a memory shortfall, plain and simple. If your actually benefiting from the 57mb of anonymous memory that was evicted during a memory shortfall on your system then your in the unique position of not needing to do any more filesystem I/O, or allocating any more anonymous memory space.

The fact is, if your doing filesystem I/O, you will eventually exhaust all available physical memory on the system. At that point, you have to evict pages before you can read or write another page to/from the filesystem. The page replacement algorithms being somewhat LRU based make this better than FILO, but only as long as they don't get too clever and kill the corner cases due to complexity.

Your grep analogy incorrectly assumes that you have a bunch of vacant memory just waiting to store those filesystem pages, but that simply isn't the case. Rather 57MB of anonymous memory was evicted to make room for 57MB of anonymous or file system backed pages. Unless you have freed anonymous memory on the system by closing applications. Your physical memory pages are still mostly occupied.

This means your grep is only going to run faster if you already read those files recently and they are already in the pagecache. You still have the burdon of pushing pages that have not been used recently out of ram before you can read in the new ones. And as long as you are performing a sufficient amount of file system I/O, this is guaranteed to happen.

One thing that can be done to minimize the problem where heavy filesystem I/O flushes important pages from memory like pages from shared libraries and executables only for them to fault back in as soon as they become runnable, is to implement something similar to what Sun implemented in Solaris 8 called the cyclical page cache. The idea is that the pagecache pages against itself and is actually considered free memory from an anonymous memory perspective. The pagecache is free to grow all it wants, but since it is counted as free memory, anonymous memory allocation will cause the pagecache to shrink because it is considered free memory.

As these pages are evicted from the pagecache, they are placed on the opposite side of the cachelist (linked list that stores pages that have a vnode+offset already) than the side where pages are being overwritten. This way frequently re-accessed pages that were placed on the cache list and were eligible to be reclaimed, are found when the next minor fault occurs for that vnode+offset and moved back to the opposite side of the list so that they are not evicted.

Since the cache list is counted as free memory, there is no way to wake up the LRU mechanism to scan physical memory until 1/64 of physical memory is consumed by anonymous memory.

Again, if you have had enough file system I/O during that time, it would eventually cause pages from your application to be paged to the swap device as the processes that represent your window slept.

--Buddy

I think this is somewhat orthogonal to whether swap should be used or not.

What we should be doing here is enforcing the RSS rlimit. I have a patch from Rik to do this which needs to be merged.

Hopefully this would give you the best case situation of having only the runaway process really slow down, without killing anything until the admin arrives.

Quote from Nick Piggin <[email protected]>:

Strictly speaking no, but instead of replacing unused memory with often used memory, the often used memory has it's own silicon, so the unused memory can stay paged in as well.

Or to put it another way, however much swap a machine has, installing that much extra physical RAM, and removing the swap space will almost never cause a loss in performance. There are some theoretical cases where it would, such as where adding extra physical RAM requires the use of different memory addressing schemes, and some data which would have, by chance, resided in more quickly accessible RAM before the upgrade no longer does, but those scenarios are not really anything to do with the original discussion.

John.

Quote from Nick Piggin <[email protected]>:

Ideally, yes - by the way, this was some time ago, (I think the machine was running a 2.2 kernel).

John.

Hi Linux-gurus,

I agree with Anthony Disante, maybe not all users want swapping. I have myself felt very annoying with swapping lately but I've not yet tried to disable it.

In school I've studied the swapping concept from a theoretical point
of view, and I fully understand the fact that swapping, if used properly, can both increase performance and provide a safe way to get out of a bad situation when the box runs out of memory. The problem is that in reality this does not work, not on Linux nor on Windows 2000 which I use at home. Unfortunately I cannot provide a specific reason why it does not work, I'm very much a end-user/desktop-user, I'm not a kernel hacker (yet). But I see two things that needs improvement atm:

A) when I do large data processing operations the computer is always very very slow afterwards

B) if I have X Mb of RAM then there should not be imho a single swap read/write until the whole of my X Mb RAM is completely stuffed, is this so today?

---

Also, imagine that I disable swap today and start a large data processing operation. During this operation I try to start a new process, here ideally the program should not OOM but instead the memory allocated for the data processing operation should be decreased. Is this possible using today's technology? Can be divide memory into two sorts, one for processes (here to stay memory) and another sort for batch operations (where the amount of memory does not really matter but less memory means less performance). I see the problem with "taking memory back" though, I guess its impossible.

Regards,
/m

"640k will be enough for anyone" ?

No. Unfortunately, userspace programs grow in size as fast as your RAM. Because typically developers do not think about size of their program until it starts to outgrow their RAM.

Today, 128M RAM swapless is barely enough to run full spectrum of apps. OpenOffice and Mozilla "lead" the pack, followed by KDE/Gnome etc.

vda

He was asking for proper bugreport.

Preparing bug report:
=====================
How To Ask Questions The Smart Way:
http://www.catb.org/~esr/faqs/smart-questions.html
Anybody who has written software for public use will probably have received at least one bad bug report. Reports that say nothing ("It doesn't work!"); reports that make no sense; reports that don't give enough information; reports that give wrong information.
How to Report Bugs Effectively:
http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
Before asking a technical question by email, or in a newsgroup, or on a website chat board, do the following:
* Try to find an answer by searching the Web.
* Try to find an answer by reading the manual.
* Try to find an answer by reading a FAQ.
* Try to find an answer by inspection or experimentation.
* Try to find an answer by reading the source code.
Compile problems: report GCC output and result of

        "grep '^CONFIG_' .config"
Oops: decode it with ksymoops (or use 2.6 with kksymoops enabled ;).

Unkillable process: Alt-SysRq-T and ksymoops relevant part. Yes it means you should have ksymoops installed and tested, which is easy to get wrong. I've done that too often.

vda

trashed means "everything i need(tm)" is paged out (mozilla/konsole/xine ...)

with swap the data-part of running programs was swapped out, without swap only the program-part is thrown out of memory as the data-part can't be moved anywhere else.

I have a 10KPRM SCSI-HDD, i can here what my system is doing. :-)

Transfering up to 8,8GB tends to trash the cache.

Bis denn

Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

OK, this is obviously bad. Do you get this behaviour with 2.6.5 or 2.6.6? If so, can you strace the program while it is writing an ISO? (just send 20 lines or so). Or tell me what program you use to create them and how to create one?

Thanks

You don't need to provide a specific reason, a report would be valuable too.

Time how long the large data processing operations take, then turn swap off and time them again.

Yes, Linux doesn't start swapping or reclaiming at all until your RAM is full.

File backed data will be able to be reclaimed, yes.

-----Original Message-----
From: William Lee Irwin III [mailto:[email protected]]
Sent: Wednesday, May 26, 2004 2:09 AM
To: Buddy Lumpkin
Cc: [email protected]; [email protected]
Subject: Re: why swap at all?

of

If you follow the thread, you will see no claim from me that there is anything wrong with the kernel. I simply stated that the priority of VM system development should focus on physical memory, and that physical memory access should not suffer as a result of some tradeoff that improves the performance of the VM system when free physical memory is low and there is heavy use of the swap device.

I can't speak whether or not a case like this currently exists, but I know optimizing swap performance is a very complicated yet captivating subject that has consumed many a posts on this list. People have tried to optimize every part of the VM before, so I was just calling out what I believe to be a very reasonable and practical goal and put a little bit of substance around why I think it's practical.

Anthony DiSante's post was merely a catalyst for discussion as far as I was conserned, I wasn't implying that I had witnessed any VM system performance problems as of late.

To address your ranting about paying customers, etc ...

After reading your message I had to check whether my original post was addressed to you directly (it wasn't). One might gain the impression that you were actually directly solicited for your opinion the way your carrying on about harassment and paying customers ... sheesh, give me a break.

I have seen many cases, where one or more persons create a free application that many people like, then after a while some of the user base starts to demand features, and show various signs that they have become too comfortable with the expectation that the application will continue to improve and forget that they should be greatful.

This situation isn't even remotely similar. In this case, you (a contributor to a very, very large FREE software project) misread a thread and made some surly comments that you ended up eating, and are so used to telling people that you owe them nothing, that you have some how conjured up the image that I actually want something from you.

This is classic, you have managed to put yourself in a position where you spend the majority of your time working on a free project that has some very ambitious goals. It has afforded you the ability to forfill your own personal and professional goals as well, yet you reserve the right to discard all accountability for your actions when it's convenient because you get some frank feedback from someone that is not a paying customer.

What a crutch.

I can picture where this is going. Here is an interview between you and a popular Linux magazine in two years:

Linux Magazine: You have contributed to linux for quite some time, correct?

William: Oh yes, it is my hobby and occupation. I love my work.

Linux Magazine: You have done all these wonderful things!

William: Thanks, I am very proud of that

Linux Magazine: Why did you make such and such decision that backfired?

William: I don't have to answer that, I don't owe you anything and your not a paying customer.

Give me a break.

--Buddy

Exactly ...

-----Original Message-----
From: John Bradford [mailto:[email protected]]
Sent: Wednesday, May 26, 2004 2:35 AM
To: Nick Piggin; Buddy Lumpkin
Cc: 'William Lee Irwin III'; [email protected]; [email protected]
Subject: Re: why swap at all?

Quote from Nick Piggin <[email protected]>:

That's true, but it's not a magical property of swap space - extra physical RAM would do more or less the same thing.

John.

I bet you have demonstrated this. It strikes me of an observation that could be made in a lab environment. But your failing to realize that:

1) you will fill physical memory with pages eventually or your not doing work.

2) pages do not just silently move to the swap device. They move as a result of a memory shortfall

3) once physical memory is full, file system I/O will only benefit from reads that incur a minor fault. All other file system operations are bound by the rate you can reclaim pages from physical memory.

4) non-filesystem backed pages are still effected the same way, nothing has changed. When you run your next filesystem related operation, those pages will be faulted into physical memory, and something will be evicted to it's backing store (remember, memory is full).

--Buddy

I have no bug to report.

-----Original Message-----
From: Denis Vlasenko [mailto:[email protected]]
Sent: Wednesday, May 26, 2004 3:45 AM
To: Buddy Lumpkin; 'William Lee Irwin III'
Cc: [email protected]; [email protected]
Subject: RE: why swap at all?

He was asking for proper bugreport.

Preparing bug report:
=====================
How To Ask Questions The Smart Way:
http://www.catb.org/~esr/faqs/smart-questions.html
Anybody who has written software for public use will probably have received at least one bad bug report. Reports that say nothing ("It doesn't work!"); reports that make no sense; reports that don't give enough information; reports that give wrong information.
How to Report Bugs Effectively:
http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
Before asking a technical question by email, or in a newsgroup, or on a website chat board, do the following:
* Try to find an answer by searching the Web.
* Try to find an answer by reading the manual.
* Try to find an answer by reading a FAQ.
* Try to find an answer by inspection or experimentation.
* Try to find an answer by reading the source code.
Compile problems: report GCC output and result of

        "grep '^CONFIG_' .config"
Oops: decode it with ksymoops (or use 2.6 with kksymoops enabled ;).

Unkillable process: Alt-SysRq-T and ksymoops relevant part. Yes it means you should have ksymoops installed and tested, which is easy to get wrong. I've done that too often.

vda

640k? who wrote that?

In the last three years, I have witnessed many large Oracle databases where the maximum SGA size of roughly 4GB + all shadow processes, parallel slaves, dbwr, etc.. all run completely within physical memory with the most aggressive settings available. Previously, Oracle databases were much smaller, but I never saw databases sized this way such that they could exist entirely in physical memory.

In fact, the SGA is commonly configured to use large 4m, locked pages (ISM in Solaris, not sure if hugepages are swappable in linux) that couldn't be swapped to disk even if you wanted to.

Again, we are not talking about the bloatware that is developed using some rad tool for a workstation that has continued to grow over the years. I am talking about where the industry is dumping tons of money on performance where it really, really counts. The middle-ware that connects to a database may continue to grow in terms of bloat, but people are happily scaling those environments horizontally in most cases. The biggest performance problems to solve (that people care about and are willing to pay $$ to solve) are for the large databases that run Corporate America. There are certainly scientific applications where performance is critical and there are dollars to fund improvement as well, but their numbers don't compare to the number of Oracle instances out there running in the Enterprise.

Optimizing the performance of swap operations for even a small tradeoff in performance for memory operations that take place entirely in physical memory is just a broke minded, brain dead direction in the year 2004 IMHO.

--Buddy

-----Original Message-----
From: Denis Vlasenko [mailto:[email protected]]
Sent: Wednesday, May 26, 2004 3:41 AM
To: Buddy Lumpkin; 'William Lee Irwin III'
Cc: [email protected]; [email protected]
Subject: RE: why swap at all?

think

"640k will be enough for anyone" ?

No. Unfortunately, userspace programs grow in size as fast as your RAM. Because typically developers do not think about size of their program until it starts to outgrow their RAM.

Today, 128M RAM swapless is barely enough to run full spectrum of apps. OpenOffice and Mozilla "lead" the pack, followed by KDE/Gnome etc.

vda

On Wednesday 26 May 2004 14:07, Buddy Lumpkin wrote:

Hi Buddy,

Bill Gates, who else ...

ciao, Marc

I really should not feed the trolls, but...

Conclusion:

You really should learn how to read :)

This is a *technical* discussion list. So far you been able to post 7 mails about vague ramblings about what you think the VM should do and what swapping is (not to mention unjustified personal attacks).

If you really think there is a problem with the VM post benchmarks demonstrating the problem.

If you don't think there is a problem, don't waste our time.

If you want to continue this discussion, please do so off-list.

Best regards,

Paulo Marques - www.grupopie.com
"In a world without walls and fences who needs windows and gates?"

You also went on to say:

By and large, Linux doesn't reclaim dirty pages from the pagecache, and it should not have paging storms due to large amounts of file system writes.

If you had a workload where it does, we would be interested to see it. I pointed out to you that this is what Bill was asking you to file a detailed report about.

Actually, during the 2.5 development cycle, swapping performance got fairly neglected to the point where we were performing twice as bad as 2.4 for most things. I (and others) recently improved this because real people doing real things were complaining.

[snip rant]

What?? Give *you* a break? From a fictional interview you concocted?

Give me a break.

Executables and shared libraries live in the filesystem cache. Evicting those from memory - because swapping is disabled and "the VM should remove something from cache instead" - will feel exactly the same as swapping ...

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

program: mkisofs
kernel: 2.4.4-2.4.25, 2.6.4-2.6.6
(To say it in other words, i never (seen/felt) a difference in 3 years. So if there is a difference i just didn't realized there is one) The current kernel is 2.6.5 as 2.6.6 sometimes just "hangs"

Just throw together some lage files (My files are all >= 350MB, the "typical" case is about 4-5files with 800-1000MB each) and then mkisofs -J -r -o <image> <source-dir>
I store the image files on another HDD to get best possibel throughput. My HDDs (these are "normal" IDE-HDDs) are capable of delivering about 35-40MB/s, the last time i measured i got about 70MB/s aggregated throughput while creating an image-file.

Bis denn

Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

--------snip---------

In older BSD systems like SunOS 4.x, malloc would literally fail if you did not have enough physical memory and backing store (swap) to store that anonymous memory segment.

This meant that if you wanted to leverage swap to get additional virtual memory beyond the amount of installed physical memory, you needed more than 1x physical memory. This is where the old 2x physical memory came from.

--Buddy

Gentlemen,

There is not enough RAM address-space in even 64-bit machines to do a sort/merge of even a typical inventory with all the keys present in RAM. So you need multiple tasks, each with as much of the 64-bit address-space occupied by RAM, as possible. Even then, you need to do partial sorts, etc.

It's not "bloat-ware" that requires getting as much free RAM as possible for an application, but the business of doing business. So, performance of data-intensive work such as the sort/merge is improved by writing the contents of sleeping tasks RAM to a storage device and using that RAM. It's just that simple.

Many years ago, there was a small company that tried to sell a sort/merge engine (a dedicated CPU) to Digital because the problems with handling large databases was well known and interactive performance on VAX/11-750 machines sucked when database applications were being run (because their pages were being swapped). Of course the NIH syndrome took its toll and nobody ever got such an engine. The result being that everybody has performance problems when database operations are being run --even today, with different machines.

Any data-intensive application needs as much RAM as possible and that's never quite enough for best performance.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
Note 96.31% of all statistics are fiction.

It's worse than that. Way worse.

The speed of hard disks doesn't grow anywhere near as fast as the size of memory and applications. This means that over the last years, swapping in any particular application has gotten SLOWER than it used to be ...

This means that even though the VM is way smarter than it used to be, the visibility of any wrong decision has increased.

I wonder if there's a way we could change the VM so it could recover faster from any mistakes it made, instead of trying to prevent it from making any mistakes (those will happen anyway, the VM can't predict the future).

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Oh yeah, poor Corporate America. That what we should care most of.

Sorry Buddy. I am _not_ Corporate America.
I have 4 boxes at work and 5 boxes at home,
and only one of them can be safely run swapless. It's a router. --
vda

To use other words, this is the typical case where a "hint" would be useful.

program to kernel: "i read ONCE though this file caching not useful".

The last thing i knew in this area is that there exist a thing to tell the kernel to drop all cache after the file is closed. (IIRC!)

But this doesn't help in this case as the image-file is up to 4,4GB in whole which means that it ALONE can fill up the whole cache. Taking aside the files the image was created from, which can (with a size of up to 2GB (size-limit of iso9660-filesystem/linux-kernel)) also fill a lot of cache until they are closed.

(The/My) typical case is this.
1 create image-file
2 remove source-files
3 burn image
4 remove image-file

Step 1 and 3 trash the cache without ANY positive effect.

Bis denn

Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

You were very wrong here. He did not say that. You pervert his words.

wli, understandably, become angry.

...

...

<joke>
Aha!
Now we all know that wli is evil. Thanks for your crystall ball. </joke>
--
vda

I totally agree, you can't get away from evicting pages from memory to disk if your doing file system I/O because you eventually fill up memory. Any additional file system I/O requires evictions, period.

Trying to preference executables and shared libraries is difficult because they are backed by named files, hence they also pagecache. (kind of reminds me of that little white speck on chicken poop. If it came out of the chicken, it's chicken poop too :)

But there is the case where massive amounts of file system I/O (consider several fibre cards connecting to SAN attached storage that saturates the centerplane on some insanely large system) will force pages from running executables to be evicted, only to be faulted back a few milliseconds later. It's this thrashing effect that a separation eliminates if you have the ability to distinguish between executables (not just files mapped in executable but actual running processes) and non-executable pages.

Consider how silly it would be for a system running a single process that consumes only 100k that generates so much filesystem I/O that the process is constantly paged out. When it needs to wake up again, becomes runnable and the program counter starts to access pages within the process address space, it gets faulted back in.

Lather, Rinse, Repeat ...

--Buddy

No. Something will be terminated, not necessarily the "evil" process. A runaway process can have quite a few server processes
killed before it eventually dies. An attacker will of course
use processes that forks, so some remain and can keep spending memory and process time.

ulimit is the way of limiting how much memory a user can use, preventing this scenario. Not setting up swap isn't.

Well, the process (or processes) can consume lots of cpu time without swap too. It can cause server processes to get killed (or not get memory they need) and it can cause slowdowns ny evicting nearly all program code from memory.

Definitely use ulimit on such a machine.

Helge Hafting

in

have

on

I really agree. And I think swappoff is not enough.

Some of my customers have over 4GB of memory. RDMS, Java Virtual Machine or Grid system (like Globus tool kit) run on the servers.
Those kinds of application make a lot of threads and they have huge amount of shared memory. And those shared memory is sometimes mlocked.

I think, in those systems, memory aging itself is useless or obstructive in worst case. Because mlocked pages which can't be swapped off are on the LRU list.

In such case, aging-off (relevant to process) is effective, I think.

Of course, I agree that swap-off or aging-off is NEVER always useful. On the contrary, these functions may be required by very small number of user.

But it is very important that we can choose how we use the OS.

Satoshi Oshima

Well, way too much swap is of course not good. A swap that is half the size of memory isn't that bad - at any time, at least two thirds of what you want in memory is bound to be there. 10x as much swap as memory might be bad though.

The problem is that swapping happens. A small swap (no more than the amount you accept being swapped out) ensures that the paging code can select the best page for eviction, rather than being forced to evict code.

If you worry about runaway processes and/or troublesome users, use ulimit.

Helge Hafting

Linux is modern and popular . . .

Stupid people then. If they really expect the disk to work only when they hit save or start up something. Sheesh.

This is a valid concern. Laptop users may want to sacrifice performance for battery life. Linux can be tweaked quite a bit for this, more development is probably a good idea. We who use AC power don't want a performance loss on our machines though, so any such tweaks must be optional.

[...]

Linux counts cache as free memory too, of course. Allocate memory, and cache will go away. It has been like this for many years.

Helge Hafting

Wasn't their an O_STREAMING patch thrown around towards the beginning of the 2.5 development cycle?

// Gianni Tedesco (gianni at scaramanga dot co dot uk)
lynx --source www.scaramanga.co.uk/scaramanga.asc | gpg --import
8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D

Ironically, a fortune 500 company I left very recently is famous for their inventory system that has been implemented in the last three years. If someone were to assume I am exaggerating, a search for my name in google groups would likely reveal what company that is, and looking up news about their at finance.yahoo.com would likely churn up a few articles about the adda-boys they have received for their inventory system and what it has done for the company.

I was the primary system admin/engineer for this system and it only occupied roughly 1TB in a single database instance. 1TB would certainly fit in a 64-bit address space. While they didn't have a zillion sku's like a company like Walmart would, their skus change on a regular basis and change at the store level while information about a jar of mayonnaise or a desk in most companies can stay quite static. Where I am going with this is I doubt there are many inventory systems out there that run much in excess of a few Terabytes.

Again, my expectation is that most large database instances out there will happily fit in a 64-bit address space. Ironically, while code tends to run slower on 64-bit architectures because of reasons like having half as many cache lines because of the larger word size, byte packing, etc.. The ability to do a hash join in memory of two insanely large tables that wouldn't fit into a 32-bit address space easily negates this small issue. So in that regard, a 64-bit address space results in a performance boost provided the DBA knows how to leverage the features.

--Buddy

Could you try CONFIG_SWAP=n to see if that makes a difference? More aggressive non-paging methods could be devised if not, e.g. CONFIG_MMU=n support of various kinds for hardware supporting paging and virtual memory (this is a suggestion, not an offer to implement).

-- wli

I believe it was a 2.4 patch , its still around somewhere. I can find it and post it , if it's still relevant.

Matt H.

I like that, pull out the Hobby OS/small purpose card when it's convenient, but a lot of the plans I see talked about for the kernel revolve around features that are needed to support the type of applications that are being stressed by Corporate America. That's where a significant amount of money being poured (directly or indirectly) into linux right now, and that is where the bulk of the next level of challenges in terms of kernel development are going to come from.

Linux is already a modern OS in most respects, the next logical programming challenges are solving some of the vertical scaling issues, even as most of these applications are moving in a growing trend toward scaling horizontally.

--Buddy

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Denis Vlasenko
Sent: Wednesday, May 26, 2004 5:20 AM
To: Buddy Lumpkin
Cc: [email protected]; [email protected]
Subject: RE: why swap at all?

to

Oh yeah, poor Corporate America. That what we should care most of.

Sorry Buddy. I am _not_ Corporate America.
I have 4 boxes at work and 5 boxes at home,
and only one of them can be safely run swapless. It's a router. --
vda
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Well for mmapped pages, man madvise. Specifically look at MADV_SEQUENTIAL and MADV_DONTNEED.

--Buddy

http://lxr.linux.no/source/mm/madvise.c?v=2.6.5#L92

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Matthias Schniedermeyer
Sent: Wednesday, May 26, 2004 5:38 AM
To: Nick Piggin
Cc: [email protected]
Subject: Re: why swap at all?

To use other words, this is the typical case where a "hint" would be useful.

program to kernel: "i read ONCE though this file caching not useful".

The last thing i knew in this area is that there exist a thing to tell the kernel to drop all cache after the file is closed. (IIRC!)

But this doesn't help in this case as the image-file is up to 4,4GB in whole which means that it ALONE can fill up the whole cache. Taking aside the files the image was created from, which can (with a size of up to 2GB (size-limit of iso9660-filesystem/linux-kernel)) also fill a lot of cache until they are closed.

(The/My) typical case is this.
1 create image-file
2 remove source-files
3 burn image
4 remove image-file

Step 1 and 3 trash the cache without ANY positive effect.

Bis denn

Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

You're probably really just annoyed with physical memory that's too small
to hold your working set. Believe it or not, having swap delays the onset of this problem.

I don't think you really do understand it from a theoretical point of view,
because you say:

Of course, this is because the working set has changed. However, with swap,
the least used pages can be evicted from physical memory. Without it, there may be no place to put the least used pages and more frequently used pages have to be evicted.

It depends what you mean by "stuffed". On a modern operating system like
Linux, pretty much all of your physical memory is in use all the time. Without swap, dirty pages cannot be evicted from physical memory, even if they haven't been used for days. If your physical memory exceeds your working set size, you win no matter what. But without swap, every dirty page is part of your working set, even if it hasn't been read/written for days.

No, it's not difficult. The OS takes physical memory back all the time by
swapping.

You seem to be missing a fundamental concept. Physical memory will always
get full because the OS will always keep copies of file data in memory just in case it needs them again. Because new pages are always being read in and processes are always allocating new memory, the OS will have to make a decision of what pages to evict from physical memory. If a page is dirty, it can only be evicted if there's swap. So if you have dirty pages that are very rarely used, swap allows you to keep more hot, clean pages in memory.

A lot of people feel subjectively that swap makes a system slow. There's
anecdotal evidence that swap does horrible things or "must be badly broken because the machine gets slow" on almost every operating system that supports swapping. In most cases, it's just a case where the real working set has exceeded physical memory, and in that case, swap is just doing what it's supposed to be doing.

DS

It's true that physical RAM or swap, over and above the minimum needed for the working set is usually beneficial. However where there is physical RAM which will never be touched during normal usage, adding swap will not be beneficial.

John.

I have a question about that. I keep a debian mirror on one of my machines. there is over 70000 files. If I run find on that tree while it's downloading the file list, it doesn't take as long. I thought it would be nice if there was some way I could keep that in memory. The box has 256mb ram no swap. It is configured as diskless.

 Lab tests show that use of micro$oft causes cancer in lab animals

If your RAM happens to be large enough to contain not only everything on disk you ever want to even read *and* all the space you need for calculations, then you have nothing to gain from using swap. On the other hand, if you are say, grepping through a kernel source tree, the first time it is read from disk, but after that it is stored in cache in your RAM. If you have swap, anonymous pages of RAM that are not in use can be paged out while you do your grepping, even if you are grepping through a 900MB+ dataset and only have 1GB RAM. Swap allows non-filesystem-backed pages to be pushed to disk for some filesystem backed pages to be loaded and used.

Cheers,
Kyle Moffett

Very true. The system is based on the assumption that just-used pages are more useful that older pages, and it slows when this isn't true. We need ways to tell the kernel whether the assumption holds.

(What follows are progressively more impossible ideas that I have no idea how to implement.)

O_STREAMING and a flag to not cache a file when it closes are a good start.

It would also be useful to do this on a per-process basis. For example, you could set a running shell so that its (and it's children's) files are O_STREAMING, and use that shell to launch your one-time greps.

Ulimit could set a limit on how much cache a process and its children could use. (How much overhead this would this entail?) That would take the place of the above, and it might also be useful for shell server admins who don't want one user trashing everyone's interactivity.

Most drastic would be to change the way to choose pages to throw out. Different processes or pages could have different priorities, so you could mark interactive processes as keepers even if you haven't used them in days.

It's probably impossible because the kernel only knows about faults, but you could give frequently but not recently used pages (your day-old browser window) priority over recently but not frequently used pages (your one-time grep). You'd also need a way to allow cache to grow, which this would otherwise curtail.

Tom Felker, <[email protected]>
<http://vlevel.sourceforge.net> - Stop fiddling with the volume knob.

Alchemists became chemists when they stopped keeping secrets.

Thanks. I'll see if I can reproduce.

You mean that if you prime the cache by running find on the tree, your actual operation doesn't take as long?

I don't doubt this. Slab cache is shrunk aggressively compared to page cache. Traditionally I think this has been due at least in part to some failure cases in the balancing there resulting in slab growing out of control with some systems.

These failure cases should be fixed now, and slab vs pagecache is probably something that should be looked at again. I really need to get my hands on a 2GB+ system before I'd be game to start fiddling with too much stuff though.

A streaming flag is great, but we usually do OK without it. There is a "used once" heuristic that often gets it right as far as I know. Basically, new pages that are only used once put almost zero pressure on the rest of the memory.

It has a few corner cases where it breaks down. Hopefully they can be improved...

Win32 API has a FILE_ATTRIBTE_TEMPORARY to mark files which should be prefered be served from buffercache, FIL_FLAG_NO_BUFFERING allows raw access (required block boundary reads). FILE_FLAG_RANDOM_ACCESS is used to hint the cache (dont know what it does, maybe reduce prefetching?) as well als FILE_FLAG_SEQUENTIAL_SCAN as a hint for the other case where you read the stream. There is also a writethrough flag, which does not affect caching. So basically I think the hints Win32 API offers are not the perfect set of flags one can think about. Unless SEQUENTIAL_ACCESS implies also "forget blocks vefore current read position".

Greetings
Bernd

eckes privat - http://www.eckes.org/
Project Freefire - http://www.freefire.org/

Am Donnerstag, 27. Mai 2004 07:14 schrieb Tom Felker:

Do you really want that? Wouldn't you rather want pages of such tasks swapped in very aggresively once the first page fault happens? Or even preemptively?

Regards
Oliver

Quote from Kyle Moffett <[email protected]>:

Think about it - you seem to be suggesting that adding more and more swap will free up more and more physical RAM to be used as cache, but that it not really true, because once you've freed up all of the physical RAM, there is no more to free up.

That it not to say that there is no point in having more swap than physical RAM at all, rather that once all non-filesystem-backed pages apart from cache have been pushed out to swap, (and note that executables can and will be pushed out to swap independently of swap space anyway), all that additional swap space will allow is to run more processes, or move cache out to swap, which admittedly could give a performance benefit in some instances, but in most cases I think it would be minimal.

John.

Yup. Running the mirror doesn't matter really. I start that before I retire at the end of the day.

Where it gets me is the 2nd mirror I have on a usb disk. Updating it takes a while. Although priming the cache on the machine where the usb disk is is a bit quicker than where the mirror is (rsync over tcp/ip). Both disks use ext3, but the machine the usb is on has way more memory, usb2, and overall quicker than the other.

I've been wanting to upgrade that machine to 768mb, but I don't know if it'll handle it.

 Lab tests show that use of micro$oft causes cancer in lab animals

In the "why swap at all" thread, there was mention of the /proc/sys/vm/swappiness tunable, and some people suggested echoing a zero to there if you want to minimize/disable swap usage, or echoing a 100 to maximize swap usage, etc.

But on my 2.6.5 system, I can echo a zero to there, then cat it back to make sure... then 30 seconds later cat it again, and it's been changed to something else (50, 60, 80something).

Is this supposed to be a value that can be manually adjusted, as some have claimed, or is it something the kernel manages automatically? I definitely can't manually set it without having it overwritten shortly thereafter.

-Anthony
http://nodivisions.com/

On Tuesday 01 June 2004 09:23, Anthony DiSante wrote:

Hi Anthony,

I bet you have /proc/sys/vm/autoswappiness or the previous version of it w/o /proc stuff.

ciao, Marc

Ah, yes, I do, and it's set to one. So if I set that to zero, then the kernel won't automatically adjust /proc/sys/vm/swappiness?

-Anthony
http://nodivisions.com/

On Tuesday 01 June 2004 21:38, Anthony DiSante wrote:

Hi Anthony,

right :)

ciao, Marc

What option do I need to enable so I can get this control (to disable it)? I have sysctl enabled in 2.6.7-rc1 and no autoswappiness to be found.

   -bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
 last possible moment - but no longer"  -me

This only exists if you have it patched with my autoregulated swappiness patch.

http://ck.kolivas.org/patches/2.6/2.6.7-rc2/patch-2.6.7-rc2-am11

The mainline kernel has a static value for swappiness you can set any time.

echo 60 > /proc/sys/vm/swappiness

Con