Swap
--8<--
Quoted 4 lines
Except that openoffice and mozilla can be swapped out in BOTH cases: the kernel can discard mapped pages and reread as needed, whether you have a swap partition or not.
No they can't without swap, nothing can be SWAPPED out. The code pages can be paged out (discarded), but no SWAPPING takes place.
Quoted 1 line
Whereas without swapspace, only the read-only mapped pages can be swapped out.
Again, pages do not gat swapped out, only applications can get swapped out. Swapping is per definition the process of removing all pages used by one application from RAM, and moving ALL pages to swap.
Quoted 9 lines
Provided the VM is doing its job properly, adding swap will always be a net win for efficiency: the kernel is able to dump unused pages to make more room for others. Of course, you tend to "feel" the response times to interactive events, rather than the overall throughput, so a change which slows the system down but makes it more "responsive" to mouse clicks etc feels like a net win...
James.
With any properly sized system, it will NEVER SWAP. Paging is a completely different thing. A little paging is not a problem. Up to 70 pagescans/s on occasion is quite acceptable. If paging activety grows above that, you may have a real problem. I don't know about the current VM, but with most unixes when you hit this mark, the system actually starts swapping, and your responsiveness goes down the drain....
Signature · 10 lines
Met vriendelijke groeten, Remco Post SARA - Stichting Academisch Rekencentrum Amsterdam High Performance Computing Tel. +31 20 592 8008 Fax. +31 20 668 3167 "I really didn't foresee the Internet. But then, neither did the computer industry. Not that that tells us very much of course - the computer industry didn't even foresee that the century was going to end." -- Douglas Adams
Quoted 8 lines · Remco Post
--8<--
Quoted 3 lines
Except that openoffice and mozilla can be swapped out in BOTH cases: the kernel can discard mapped pages and reread as needed, whether you have a swap partition or not.
No they can't without swap, nothing can be SWAPPED out. The code pages can be paged out (discarded), but no SWAPPING takes place.
OK, s/swapped/paged/.
Quoted 6 lines
Quoted 2 lines
Whereas without swapspace, only the read-only mapped pages can be swapped out.
Again, pages do not gat swapped out, only applications can get swapped out. Swapping is per definition the process of removing all pages used by one application from RAM, and moving ALL pages to swap.
So in effect, Linux never ever swaps. At all. Under any circumstances. (Using your interpretation of the word). Which does raise the question of WTF that "swap space" is for, and why it's really used for "paging"...
Quoted 7 lines
Quoted 6 lines
Provided the VM is doing its job properly, adding swap will always be a net win for efficiency: the kernel is able to dump unused pages to make more room for others. Of course, you tend to "feel" the response times to interactive events, rather than the overall throughput, so a change which slows the system down but makes it more "responsive" to mouse clicks etc feels like a net win...
Quoted 6 lines
With any properly sized system, it will NEVER SWAP. Paging is a completely different thing. A little paging is not a problem. Up to 70 pagescans/s on occasion is quite acceptable. If paging activety grows above that, you may have a real problem. I don't know about the current VM, but with most unixes when you hit this mark, the system actually starts swapping, and your responsiveness goes down the drain....
By your definition, Linux does not swap, ever. It only "pages". This is what I was referring to as swapping, since this involves the SWAPspace/partition, rather than PAGEfile :)
James.
Quoted 15 lines · Remco Post
Quoted 8 lines
--8<--
Quoted 3 lines
Except that openoffice and mozilla can be swapped out in BOTH cases: the kernel can discard mapped pages and reread as needed, whether you have a swap partition or not.
No they can't without swap, nothing can be SWAPPED out. The code pages can be paged out (discarded), but no SWAPPING takes place.
OK, s/swapped/paged/.
Quoted 3 lines
Quoted 2 lines
Whereas without swapspace, only the read-only mapped pages can be swapped out.
Quoted 8 lines
Quoted 3 lines
Again, pages do not gat swapped out, only applications can get swapped out. Swapping is per definition the process of removing all pages used by one application from RAM, and moving ALL pages to swap.
So in effect, Linux never ever swaps. At all. Under any circumstances. (Using your interpretation of the word). Which does raise the question of WTF that "swap space" is for, and why it's really used for "paging"...
Linux does swap (I guess), swapping is a very extreem measure, "I need memory now, and the paging algorithm does not work any more", this is quite rare, but a few runaway netscape processes can easily cause this....
Quoted 14 lines
Quoted 7 lines
Quoted 6 lines
Provided the VM is doing its job properly, adding swap will always be a net win for efficiency: the kernel is able to dump unused pages to make more room for others. Of course, you tend to "feel" the response times to interactive events, rather than the overall throughput, so a change which slows the system down but makes it more "responsive" to mouse clicks etc feels like a net win...
Quoted 6 lines
With any properly sized system, it will NEVER SWAP. Paging is a completely different thing. A little paging is not a problem. Up to 70 pagescans/s on occasion is quite acceptable. If paging activety grows above that, you may have a real problem. I don't know about the current VM, but with most unixes when you hit this mark, the system actually starts swapping, and your responsiveness goes down the drain....
Quoted 7 lines
By your definition, Linux does not swap, ever. It only "pages". This is what I was referring to as swapping, since this involves the SWAPspace/partition, rather than PAGEfile :)
James.
It is quite a common mistake. When discussing the VM, it is important to make the distinction. In the old days (about the time when I was born ;) swapping was the only thing Unixes ever did, no paging, which is quite a recent invention. As you'd expect, this is why you have a swapspace that is now also used for paging. As a test, you could quite simply build an application that uses so much memory (not only malloc it, but also USE it) that your system will start swapping, try using any interative application after that, and you'll feel why you really don't want a system to swap...
Signature · 10 lines
Met vriendelijke groeten, Remco Post SARA - Stichting Academisch Rekencentrum Amsterdam High Performance Computing Tel. +31 20 592 8008 Fax. +31 20 668 3167 "I really didn't foresee the Internet. But then, neither did the computer industry. Not that that tells us very much of course - the computer industry didn't even foresee that the century was going to end." -- Douglas Adams
James A Sutherland <[email protected]>:
Quoted 15 lines · Remco Post
Quoted 8 lines
--8<--
Quoted 3 lines
Except that openoffice and mozilla can be swapped out in BOTH cases: the kernel can discard mapped pages and reread as needed, whether you have a swap partition or not.
No they can't without swap, nothing can be SWAPPED out. The code pages can be paged out (discarded), but no SWAPPING takes place.
OK, s/swapped/paged/.
Quoted 3 lines
Quoted 2 lines
Whereas without swapspace, only the read-only mapped pages can be swapped out.
Quoted 7 lines
Quoted 3 lines
Again, pages do not gat swapped out, only applications can get swapped out. Swapping is per definition the process of removing all pages used by one application from RAM, and moving ALL pages to swap.
So in effect, Linux never ever swaps. At all. Under any circumstances. (Using your interpretation of the word). Which does raise the question of WTF that "swap space" is for, and why it's really used for "paging"...
Linux doesn't - but some UNIX systems do swap. This is when the kernel pages out the process header, page tables, process kernel stack ...
At this point the process is in the equivalent state as that of the system that only does "swapping".
The swap space is used when more physical memory is required than is available for user data. The modified pages of user data are written to the swap space and the physical page re-used for another purpose. Effectively "swapping" the use of the page... :-)
Quoted 14 lines
Quoted 7 lines
Quoted 6 lines
Provided the VM is doing its job properly, adding swap will always be a net win for efficiency: the kernel is able to dump unused pages to make more room for others. Of course, you tend to "feel" the response times to interactive events, rather than the overall throughput, so a change which slows the system down but makes it more "responsive" to mouse clicks etc feels like a net win...
Quoted 6 lines
With any properly sized system, it will NEVER SWAP. Paging is a completely different thing. A little paging is not a problem. Up to 70 pagescans/s on occasion is quite acceptable. If paging activety grows above that, you may have a real problem. I don't know about the current VM, but with most unixes when you hit this mark, the system actually starts swapping, and your responsiveness goes down the drain....
Quoted 3 lines
By your definition, Linux does not swap, ever. It only "pages". This is what I was referring to as swapping, since this involves the SWAPspace/partition, rather than PAGEfile :)
The problem is determining "properly sized system". Second - ALL linux systems will page in (or swap in) executables, if only at the start of executution (easiest/fasted way to load the program... mmap is quick, even if it does blur the distinction between process pages and I/O cache)
Linux uses RAM+SWAP for virtual memory operation, and swaps pages used for data to the "swap space" to use different "swapped pages" to load back into physical memory. Since this is effectively hidden from most activity (and measures), it becomes easy to oversubscribe memory, causing thrashing (lots of page activity for little gain), where a system with mixed paging + swapping (page out entire processes and disable scheduling them) CAN make significant progress.
The other use of RAM is for data caching. Usually is faster to keep file data loaded into RAM for use by programs. Runtime libraries are frequently where the majority of CPU time is spent - Instead of waiting for data to be transferred to RAM for use, Linux tries to "read ahead" accomplishing more throughput that way by not forcing the active process to wait for the data.
The tricky part is determining the balance between the data cache, and process memory.
The systems that use a combined pageing + swapping use a variety of measures to decide what should be paged or swapped. Some characteristics used by these systems are:
1. number of page faults/sec (swap if > watermark - reduces thrashing)
2. time elapsed since last completed I/O (if greater than some watermark, swap
- makes more RAM available)
3. idle processes (wait time > watermark, swap - discard executable pages, swap out data pages)
4. batch processes (operate at a lower priority - swap non-interactive processes - makes more RAM available)
5. High memory requirements (reduce resident set size; which invokes item 1)
6. Users priority (swap lower priority processes - make more RAM available)
Of course the sys admin must have control over all of the watermarks and/or resource allocations. These are more characteristics of a general computation or batch system than they are of a single user workstation, which is where Linux started.
Hope I've help clear up some things.
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]
Any opinions expressed are solely my own.
Quoted 4 lines · Remco Post
Linux does swap (I guess), swapping is a very extreem measure, "I need memory now, and the paging algorithm does not work any more", this is quite rare, but a few runaway netscape processes can easily cause this....
Guess again. Linux doesn't have load control implemented ...
Rik
Signature · 3 lines
DMCA, SSSCA, W3C? Who cares? http://thefreeworld.net/ http://www.surriel.com/ http://distro.conectiva.com/
Quoted 1 line · --On Monday, 19 November, 2001 2:58 PM -0200 Rik van Riel…
Guess again. Linux doesn't have load control implemented ...
Out of interest, is received wisdom that this is a good/bad thing?
--
Alex Bligh
Quoted 7 lines · Alex Bligh - linux-kernel
--On Monday, 19 November, 2001 2:58 PM -0200 Rik van Riel
Quoted 1 line · <[email protected]> wrote:
Guess again. Linux doesn't have load control implemented ...
Out of interest, is received wisdom that this is a good/bad thing?
Load control is a good thing since it means the box gets slower in a controlled way instead of running fine one minute and horribly falling over the next minute.
I'm certainly planning to implement some load control measures for 2.5.
regards,
Rik
Signature · 3 lines
DMCA, SSSCA, W3C? Who cares? http://thefreeworld.net/ http://www.surriel.com/ http://distro.conectiva.com/
Rik,
Quoted 10 lines · --On Monday, 19 November, 2001 7:17 PM -0200 Rik van Riel…
Quoted 2 lines
Out of interest, is received wisdom that this is a good/bad thing?
Load control is a good thing since it means the box gets slower in a controlled way instead of running fine one minute and horribly falling over the next minute.
I'm certainly planning to implement some load control measures for 2.5.
OK another potentially dumb question on this:
I had previously (mis?)understood load control to mean (say) clustering page out requests to pages from specific processes, then altering the scheduler to avoid scheduling these processes for extended periods of time, then moving onto the next set of processes to victimize, and so forth; i.e. increasing scheduler granularity to cope with increased average virtual memory access times by decreasing VM footprint used per second.
The original poster seemed to be talking about the old-UNIX definition of swapping, which, if I remember right, was releasing /all/ clean pages for an app (I guess this has already been done by the time we want to do this) and paging /all/ dirty pages & freeing the memory there and then.
I'd have thought swapping was a pretty coarsely-grained form of load control (and difficulted with shared mem etc.); do you believe there is a requirement to implement (old UNIX) swapping per-se, or merely to intelligently tweak the scheduler to cope better with high VM system loads? [the absence of the former was what I was suggesting might have been considered a good thing]
--
Alex Bligh
Quoted 12 lines · On 20011119 James A Sutherland wrote:
Quoted 8 lines · Remco Post
--8<--
Quoted 3 lines
Except that openoffice and mozilla can be swapped out in BOTH cases: the kernel can discard mapped pages and reread as needed, whether you have a swap partition or not.
No they can't without swap, nothing can be SWAPPED out. The code pages can be paged out (discarded), but no SWAPPING takes place.
OK, s/swapped/paged/.
Not so OK.
AFAIK, that is all a question of names. All is the same. Old systems like MacOS do SWAP, because when they send something to disk they send the whole app with its data space to disk. Linux does not send a whole app to disk, but individual pages, so it does SWAP AT PAGE LEVEL, or paging. When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out. A page is paged-out if it is written to disk.
So _swaping_ and _paging_ are the same, but with different granularity.
(of course, flame and correct me if I'm wrong...)
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Whereas without swapspace, only the read-only mapped pages can be swapped out.
They are not swapped-out, just discarded to be re-read.
Quoted 5 lines
By your definition, Linux does not swap, ever. It only "pages". This is what I was referring to as swapping, since this involves the SWAPspace/partition, rather than PAGEfile :)
It is the same. You can page-out (because Linux never do swap, as the process of sending a whole app to disk), to an specially formatted partition or to a file. If you are going to be pedantic, linux really uses _page_partitions_ and _page_files_, instead of swap-partitions and swap-files.
BTW, there is soft for mac that changes the swap algorithm from app level to page level and they called it "RamDoubler", and people still thinks its magic...
Signature · 4 lines
J.A. Magallon # Let the source be with you... mailto:[email protected] Mandrake Linux release 8.2 (Cooker) for i586 Linux werewolf 2.4.15-pre6-beo #1 SMP Sun Nov 18 10:25:01 CET 2001 i686
Quoted 2 lines · J.A. Magallon
When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out.
What happens if the on-disk binary has changed since loading the program?
Quoted 6 lines · J.A. Magallon
Quoted 2 lines
When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out.
What happens if the on-disk binary has changed since loading the program?
The application usually crashes, but in theory it may run with just some 'strange' behaviour. (Don't worry, apps usually just crash ;)
Signature · 10 lines
Met vriendelijke groeten, Remco Post SARA - Stichting Academisch Rekencentrum Amsterdam High Performance Computing Tel. +31 20 592 8008 Fax. +31 20 668 3167 "I really didn't foresee the Internet. But then, neither did the computer industry. Not that that tells us very much of course - the computer industry didn't even foresee that the century was going to end." -- Douglas Adams
Quoted 5 lines · Wolfgang Rohdewald
Quoted 2 lines · J.A. Magallon
When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out.
What happens if the on-disk binary has changed since loading the program?
In general, you can't... You get a ETXTBSY 'text file busy' error. If you try to do this over NFS (where the system can't stop you), the running image will almost certainly crash if it tries to page in text.
-Nick
Quoted 6 lines · Wolfgang Rohdewald
Quoted 2 lines · J.A. Magallon
When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out.
What happens if the on-disk binary has changed since loading the program? -
It can't. That's the reason for install and other methods of changing execututable files (mv exe-file exe-file.old ; cp newfile exe-file). The currently open, and possibly mapped file can be re-named, but it can't be overwritten.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).
I was going to compile a list of innovations that could be attributed to Microsoft. Once I realized that Ctrl-Alt-Del was handled in the BIOS, I found that there aren't any.
Quoted 7 lines · Richard B. Johnson
Quoted 4 lines · Wolfgang Rohdewald
Quoted 2 lines · J.A. Magallon
When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out.
Quoted 7 lines
Quoted 2 lines
What happens if the on-disk binary has changed since loading the program? -
It can't. That's the reason for
installand other methods of changing execututable files (mv exe-file exe-file.old ; cp newfile exe-file). The currently open, and possibly mapped file can be re-named, but it can't be overwritten.
Actually, with NFS (and probably others) it can. Suppose I change the file on the server, and it's swapped out on a client that has it mounted. When it swaps back in, it can get the new information.
Chris
Signature · 4 lines
Chris Friesen | MailStop: 043/33/F10 Nortel Networks | work: (613) 765-0557 3500 Carling Avenue | fax: (613) 765-2986 Nepean, ON K2H 8E9 Canada | email: [email protected]
Quoted 16 lines · Christopher Friesen
Quoted 7 lines · Richard B. Johnson
Quoted 4 lines · Wolfgang Rohdewald
Quoted 2 lines · J.A. Magallon
When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out.
Quoted 7 lines
Quoted 2 lines
What happens if the on-disk binary has changed since loading the program? -
It can't. That's the reason for
installand other methods of changing execututable files (mv exe-file exe-file.old ; cp newfile exe-file). The currently open, and possibly mapped file can be re-named, but it can't be overwritten.
Quoted 5 lines
Actually, with NFS (and probably others) it can. Suppose I change the file on the server, and it's swapped out on a client that has it mounted. When it swaps back in, it can get the new information.
Chris
I note that NFS files don't currently return ETXTBSY, but this is a bug. It is 'known' to the OS that the NFS mounted file-system is busy because you can't unmount the file-system while an executable is running. If you can trash it (as you can on Linux), it is surely a bug.
Alan explained a few years ago that NFS was "stateless". Nevertheless it is still a bug.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).
I was going to compile a list of innovations that could be attributed to Microsoft. Once I realized that Ctrl-Alt-Del was handled in the BIOS, I found that there aren't any.
Quoted 9 lines · Christopher Friesen
Quoted 8 lines · Richard B. Johnson
Quoted 6 lines · Wolfgang Rohdewald
Quoted 2 lines · J.A. Magallon
When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out.
What happens if the on-disk binary has changed since loading the program? -
Quoted 8 lines
Quoted 4 lines
It can't. That's the reason for
installand other methods of changing execututable files (mv exe-file exe-file.old ; cp newfile exe-file). The currently open, and possibly mapped file can be re-named, but it can't be overwritten.Actually, with NFS (and probably others) it can. Suppose I change the file on the server, and it's swapped out on a client that has it mounted. When it swaps back in, it can get the new information.
I am quite sure this is also possible if the binary is emulated by the linux-abi modules like my old SCO binaries. I just cannot check right now because I did not yet get linux-abi working with 2.4.15-pre7 (worked with 2.4.15-pre4, but pre4 had a seemingly VM related OOPS when starting VMware3 which is gone with pre7)
<snip>
Quoted 7 lines
I note that NFS files don't currently return ETXTBSY, but this is a bug. It is 'known' to the OS that the NFS mounted file-system is busy because you can't unmount the file-system while an executable is running. If you can trash it (as you can on Linux), it is surely a bug.
Alan explained a few years ago that NFS was "stateless". Nevertheless it is still a bug.
Correct me if I'm wrong, but I think that it's more a bug in the NFS protocol than in the Linux (or Solaris, etc) NFS implementation. The problem is that NFS itself just doesn't pass that information along. The NFS server has no idea that the 'text' file is being executed, so it doesn't know that it should "return" ETXTBSY.
Now, this might be different in NFS v3, but I'm pretty sure that this applies for v2, at least.
-Nick
Quoted 4 lines · J.A. Magallon
BTW, there is soft for mac that changes the swap algorithm from app level to page level and they called it "RamDoubler", and people still thinks its magic...
Ahh, so that's what it does, in addition to compression...
Quoted 12 lines · Christopher Friesen
Quoted 10 lines · Richard B. Johnson
Quoted 6 lines · Wolfgang Rohdewald
Quoted 2 lines · J.A. Magallon
When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out.
What happens if the on-disk binary has changed since loading the program? -
Quoted 9 lines
Quoted 4 lines
It can't. That's the reason for
installand other methods of changing execututable files (mv exe-file exe-file.old ; cp newfile exe-file). The currently open, and possibly mapped file can be re-named, but it can't be overwritten.Actually, with NFS (and probably others) it can. Suppose I change the file on the server, and it's swapped out on a client that has it mounted. When it swaps back in, it can get the new information.
This sounds really dangerous... What about shared libraries ??
Regards,
Signature · 4 lines
Steffen Persvold | Scalable Linux Systems | Try out the world's best mailto:[email protected] | http://www.scali.com | performing MPI implementation: Tel: (+47) 2262 8950 | Olaf Helsets vei 6 | - ScaMPI 1.12.2 - Fax: (+47) 2262 8951 | N0621 Oslo, NORWAY | >300MBytes/s and <4uS latency
Quoted 18 lines · Steffen Persvold
Quoted 17 lines · Christopher Friesen
Quoted 14 lines · Richard B. Johnson
Quoted 6 lines · Wolfgang Rohdewald
Quoted 2 lines · J.A. Magallon
When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out.
What happens if the on-disk binary has changed since loading the program? -
It can't. That's the reason for
installand other methods of changing execututable files (mv exe-file exe-file.old ; cp newfile exe-file). The currently open, and possibly mapped file can be re-named, but it can't be overwritten.
Quoted 7 lines
Quoted 4 lines
Actually, with NFS (and probably others) it can. Suppose I change the file on the server, and it's swapped out on a client that has it mounted. When it swaps back in, it can get the new information.
This sounds really dangerous... What about shared libraries ??
IIRC (if wrong flame...)
When you delete an open file, the entry is removed from the directory, but not unlinked until the file is closed. This is a standard UNIX semantic.
Now, if you have a set of processes with shared memory, and one closes, and another is created to replace, the new process will get the new libraries, or even new version of the process. This could/will bring down the entire set of processes.
Apps like samba come to mind...
Mike
Quoted 15 lines · Steffen Persvold
Quoted 14 lines · Christopher Friesen
Quoted 12 lines · Richard B. Johnson
Quoted 6 lines · Wolfgang Rohdewald
Quoted 2 lines · J.A. Magallon
When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out.
What happens if the on-disk binary has changed since loading the program? -
It can't. That's the reason for
installand other methods of changing execututable files (mv exe-file exe-file.old ; cp newfile exe-file). The currently open, and possibly mapped file can be re-named, but it can't be overwritten.
Quoted 5 lines
Quoted 3 lines
Actually, with NFS (and probably others) it can. Suppose I change the file on the server, and it's swapped out on a client that has it mounted.
When it swaps back in, it can get the new information.This sounds really dangerous... What about shared libraries ??
It is. Usually it ends with a loud 'boom' the process crashes & burns.
-Nick
Quoted 32 lines · Mike Fedyk
Quoted 18 lines · Steffen Persvold
Quoted 17 lines · Christopher Friesen
Quoted 14 lines · Richard B. Johnson
Quoted 6 lines · Wolfgang Rohdewald
Quoted 2 lines · J.A. Magallon
When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out.
What happens if the on-disk binary has changed since loading the program? -
It can't. That's the reason for
installand other methods of changing execututable files (mv exe-file exe-file.old ; cp newfile exe-file). The currently open, and possibly mapped file can be re-named, but it can't be overwritten.Quoted 7 lines
Quoted 4 lines
Actually, with NFS (and probably others) it can. Suppose I change the file on the server, and it's swapped out on a client that has it mounted. When it swaps back in, it can get the new information.
This sounds really dangerous... What about shared libraries ??
IIRC (if wrong flame...)
When you delete an open file, the entry is removed from the directory, but not unlinked until the file is closed. This is a standard UNIX semantic.
Quoted 8 lines
Now, if you have a set of processes with shared memory, and one closes, and another is created to replace, the new process will get the new libraries, or even new version of the process. This could/will bring down the entire set of processes.
Apps like samba come to mind...
Mike
If the file is local, everything is fine. A file won't actually be deleted until the last access is closed. However, the long-standing problem with NFS is that it's phony. Basically, we send a message to a server that says "Give me a directory listing...". The server does the opendir() etc., and returns the results. If I want to open a file on the server, the server has no knowledge of the open. The client's software just emulated a file-system open(). When the client wants to read data from a server's file, it sends a message; "Gimmie data from file xxx, offset x, length y.". The server responds with that data. To get that data, the server did an open/lseek/read/close.
So, as far as the server is concerned, that file is closed. Somebody else (with privilege) can delete the file and replace it. The client, the one that got the data for an executable, doesn't even know it.
This is 'nice' for the server, it doesn't have the overhead of maintaining a file-system state. That's why servers are supposed to be read-only. However, somebody has got to write the stuff to the file-system that's going to (eventually) be read-only. Beware when such access occurs.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).
I was going to compile a list of innovations that could be attributed to Microsoft. Once I realized that Ctrl-Alt-Del was handled in the BIOS, I found that there aren't any.
Quoted 40 lines · Nick LeRoy
Quoted 27 lines · Mike Fedyk
Quoted 15 lines · Steffen Persvold
Quoted 14 lines · Christopher Friesen
Quoted 12 lines · Richard B. Johnson
Quoted 6 lines · Wolfgang Rohdewald
Quoted 2 lines · J.A. Magallon
When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out.
What happens if the on-disk binary has changed since loading the program? -
It can't. That's the reason for
installand other methods of changing execututable files (mv exe-file exe-file.old ; cp newfile exe-file). The currently open, and possibly mapped file can be re-named, but it can't be overwritten.Quoted 5 lines
Quoted 3 lines
Actually, with NFS (and probably others) it can. Suppose I change the file on the server, and it's swapped out on a client that has it mounted. When it swaps back in, it can get the new information.
This sounds really dangerous... What about shared libraries ??
IIRC (if wrong flame...)
When you delete an open file, the entry is removed from the directory, but not unlinked until the file is closed. This is a standard UNIX semantic.
Quoted 6 lines
Now, if you have a set of processes with shared memory, and one closes, and another is created to replace, the new process will get the new libraries, or even new version of the process. This could/will bring down the entire set of processes.
Apps like samba come to mind...
*Any* time that you write to an executing executable, all bets are off. The most likely outcome is a big 'ol crash & burn. With a local FS, Unix prevents you from shooting yourself in the foot, but with NFS, fire away.. I've done it. It *does* let you, but...
Quoted 9 lines
Solution: Don't do that. Shut them all down, on all clients, upgrade the binaries, then restart the processes on the clients.
As far as the scenerio that you've described, I *think* that it would actually work. When the new process is fork()ed, it gets a copy of the file descriptors from it's parent, so the file is still open to it. If it the exec()s, the new image no longer has any real ties to it's parent (at least, not that are relevant to this).
What about processes with shared memory such as samba 2.0?
Quoted 27 lines · Mike Fedyk
Quoted 15 lines · Steffen Persvold
Quoted 14 lines · Christopher Friesen
Quoted 12 lines · Richard B. Johnson
Quoted 6 lines · Wolfgang Rohdewald
Quoted 2 lines · J.A. Magallon
When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out.
What happens if the on-disk binary has changed since loading the program? -
It can't. That's the reason for
installand other methods of changing execututable files (mv exe-file exe-file.old ; cp newfile exe-file). The currently open, and possibly mapped file can be re-named, but it can't be overwritten.Quoted 5 lines
Quoted 3 lines
Actually, with NFS (and probably others) it can. Suppose I change the file on the server, and it's swapped out on a client that has it mounted. When it swaps back in, it can get the new information.
This sounds really dangerous... What about shared libraries ??
IIRC (if wrong flame...)
When you delete an open file, the entry is removed from the directory, but not unlinked until the file is closed. This is a standard UNIX semantic.
Quoted 6 lines
Now, if you have a set of processes with shared memory, and one closes, and another is created to replace, the new process will get the new libraries, or even new version of the process. This could/will bring down the entire set of processes.
Apps like samba come to mind...
*Any* time that you write to an executing executable, all bets are off. The most likely outcome is a big 'ol crash & burn. With a local FS, Unix prevents you from shooting yourself in the foot, but with NFS, fire away.. I've done it. It *does* let you, but...
Solution: Don't do that. Shut them all down, on all clients, upgrade the binaries, then restart the processes on the clients.
As far as the scenerio that you've described, I *think* that it would actually work. When the new process is fork()ed, it gets a copy of the file descriptors from it's parent, so the file is still open to it. If it the exec()s, the new image no longer has any real ties to it's parent (at least, not that are relevant to this).
If it's created via clone(), then, once again, it's got it's parents descriptors still open, so no problem.
I think the real problems only exist over NFS and NFS-like scenerios.
Did I miss something here, or am I actually correct? I was correct once, let's see... Ooops. That was a mistake too.
-Nick
NFS, Paging & Installing [was: Re: Swap]
In reply to Mike FedykQuoted 26 lines · Richard B. Johnson
Quoted 5 lines · Mike Fedyk
IIRC (if wrong flame...)
When you delete an open file, the entry is removed from the directory, but not unlinked until the file is closed. This is a standard UNIX semantic.
Quoted 8 lines
Now, if you have a set of processes with shared memory, and one closes, and another is created to replace, the new process will get the new libraries, or even new version of the process. This could/will bring down the entire set of processes.
Apps like samba come to mind...
Mike
If the file is local, everything is fine. A file won't actually be deleted until the last access is closed. However, the long-standing problem with NFS is that it's
phony. Basically, we send a message to a server that says "Give me a directory listing...". The server does theopendir()etc., and returns the results. If I want to open a file on the server, the server has no knowledge of theopen. The client's software just emulated a file-systemopen(). When the client wants to read data from a server's file, it sends a message; "Gimmie data from file xxx, offset x, length y.". The server responds with that data. To get that data, the server did an open/lseek/read/close.
Quoted 9 lines
So, as far as the server is concerned, that file is closed. Somebody else (with privilege) can delete the file and replace it. The client, the one that got the data for an executable, doesn't even know it.
This is 'nice' for the server, it doesn't have the overhead of maintaining a file-system state. That's why servers are supposed to be read-only. However, somebody has got to write the stuff to the file-system that's going to (eventually) be read-only. Beware when such access occurs.
Do any newer versions of NFS fix the stateless server problem?
If not, are there any drop in (at least for linux) replacements that do keep state on the server?
SMB is out because it doesn't propagate the unix uid/gid
Striped down (auth wise) AFS?
Intermezzo?
On Tuesday 20 November 2001 15:44, Mike Fedyk wrote:
<SNIP>
Quoted 8 lines
Quoted 8 lines
*Any* time that you write to an executing executable, all bets are off. The most likely outcome is a big 'ol crash & burn. With a local FS, Unix prevents you from shooting yourself in the foot, but with NFS, fire away.. I've done it. It *does* let you, but...
Solution: Don't do that. Shut them all down, on all clients, upgrade the binaries, then restart the processes on the clients.
Quoted 7 lines
Quoted 5 lines
As far as the scenerio that you've described, I *think* that it would actually work. When the new process is fork()ed, it gets a copy of the file descriptors from it's parent, so the file is still open to it. If it the exec()s, the new image no longer has any real ties to it's parent (at least, not that are relevant to this).
What about processes with shared memory such as samba 2.0?
fork()ed processes are *identical* to their parents execept for the return value from fork(). They have the same shared memory handles, file descriptors, etc. The kernel "knows" that there's an extra copy of each, and updates it's link counts, etc.
Actually, the real point is that it'll still be the old executable running with the old libraries, until you shut down the whole group. Each of the processes are "linked" to the original file, so the new version will never run 'til the whole group is restarted.
It should just work. I can't think of any reason why it shouldn't.
-Nick
Quoted 22 lines · Richard B. Johnson
Quoted 20 lines · Christopher Friesen
Quoted 7 lines · Richard B. Johnson
Quoted 4 lines · Wolfgang Rohdewald
Quoted 2 lines · J.A. Magallon
When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out.
Quoted 7 lines
Quoted 2 lines
What happens if the on-disk binary has changed since loading the program? -
It can't. That's the reason for
installand other methods of changing execututable files (mv exe-file exe-file.old ; cp newfile exe-file). The currently open, and possibly mapped file can be re-named, but it can't be overwritten.Actually, with NFS (and probably others) it can. Suppose I change the file on the server, and it's swapped out on a client that has it mounted. When it swaps back in, it can get the new information.
Quoted 7 lines
Quoted 1 line
Chris
I note that NFS files don't currently return ETXTBSY, but this is a bug. It is 'known' to the OS that the NFS mounted file-system is busy because you can't unmount the file-system while an executable is running. If you can trash it (as you can on Linux), it is surely a bug.
In most of the cases, the process on the client simply dies....
Re: NFS, Paging & Installing [was: Re: Swap]
In reply to Mike FedykQuoted 1 line · Mike Fedyk
Do any newer versions of NFS fix the stateless server problem?
This is an _extremely_ hard problem: The server has to know somehow what the client thinks the state is... and either one (or both) may have been rebooted in between without the other one knowing.
Signature · 2 lines
Horst von Brand [email protected] Casilla 9G, Vin~a del Mar, Chile +56 32 672616
Re: NFS, Paging & Installing [was: Re: Swap]
In reply to Mike FedykQuoted 6 lines · Horst von Brand
Quoted 1 line · Mike Fedyk
Do any newer versions of NFS fix the stateless server problem?
This is an _extremely_ hard problem: The server has to know somehow what the client thinks the state is... and either one (or both) may have been rebooted in between without the other one knowing.
Yep, but there are currently protocols (SMB) that do that, but not necessarily in a unix way.
Are there any that do this now with linux? Locking over the network just like it is locally?
Quoted 11 lines · Nick LeRoy
Quoted 2 lines
Alan explained a few years ago that NFS was "stateless". Nevertheless it is still a bug.
Correct me if I'm wrong, but I think that it's more a bug in the NFS protocol than in the Linux (or Solaris, etc) NFS implementation. The problem is that NFS itself just doesn't pass that information along. The NFS server has no idea that the 'text' file is being executed, so it doesn't know that it should "return" ETXTBSY.
Now, this might be different in NFS v3, but I'm pretty sure that this applies for v2, at least.
Consider the above mentioned statelessness. You can't get what you want as long as you want a stateless server - it is simply impossible.
Your client can be tweaked so that you can't write via NFS to a file executing on the same host - but nothing can prevent another client from writing to that file - because the server is stateless.
A stateless server means it don't actually know if a file is opened by anyone. The good part of this is that the server may crash and reboot, and the client will only see a delay. Open files will still work as soon as the server comes back up. No state were lost in the crash - because there were no state at all. But then you can't block writes because you don't know that someone is executing the file.
It is not a design bug - it is a design tradeoff. A stateful server might work if you have years of uptime or at least no unplanned downtime. But such implementations tend to force clients to remount if the server ever go down. That may be really annoying if you're accessing lots of servers.
Helge Hafting
Re: NFS, Paging & Installing [was: Re: Swap]
In reply to Mike FedykQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Mike Fedyk <[email protected]> writes:
> On Tue, Nov 20, 2001 at 10:22:58PM -0300, Horst von Brand > wrote:
>> Mike Fedyk <[email protected]> said:
>> > Do any newer versions of NFS fix the stateless server >> > problem?
>>
>> This is an _extremely_ hard problem: The server has to know >> somehow what the client thinks the state is... and either one >> (or both) may have been rebooted in between without the other >> one knowing.
> Yep, but there are currently protocols (SMB) that do that, but > not necessarily in a unix way.
<Cough, choke>
Exactly how, pray tell, does SMB cope with recovering the full state
info after client/server crashes?
Cheers,
Trond
Quoted 5 lines
It is not a design bug - it is a design tradeoff. A stateful server might work if you have years of uptime or at least no unplanned downtime. But such implementations tend to force clients to remount if the server ever go down. That may be really annoying if you're accessing lots of servers.
NFS is at best "imitation stateless". You can do good stateful servers that recover across both client and server machine failure. You can do far better with them than with NFS - its just a bit harder.
Alan
Quoted 29 lines · Christopher Friesen
Quoted 17 lines
Quoted 7 lines · Richard B. Johnson
Quoted 4 lines · Wolfgang Rohdewald
Quoted 2 lines · J.A. Magallon
When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out.
Quoted 7 lines
Quoted 2 lines
What happens if the on-disk binary has changed since loading the program? -
It can't. That's the reason for
installand other methods of changing execututable files (mv exe-file exe-file.old ; cp newfile exe-file). The currently open, and possibly mapped file can be re-named, but it can't be overwritten.Quoted 5 lines
Actually, with NFS (and probably others) it can. Suppose I change the file on the server, and it's swapped out on a client that has it mounted. When it swaps back in, it can get the new information.
Chris
I note that NFS files don't currently return ETXTBSY, but this is a bug. It is 'known' to the OS that the NFS mounted file-system is busy because you can't unmount the file-system while an executable is running. If you can trash it (as you can on Linux), it is surely a bug.
Quoted 6 lines
Alan explained a few years ago that NFS was "stateless". Nevertheless it is still a bug.
Cheers,
Dick Johnson
The Client OS knows the fs is busy, the server does not, so from the server side, I can change a file, unmount parts of the exported fs (nfs does not see fs boudries), or even mount a completely different fs on the exported fs, breaking the nfs client and the nfs server. Been there, done that. Yes, this is not userfriendly, but then again, NFS in not the best networked filesystem in the world, not was it designed to be handled by non-administrators. (and I think it shouldn't have to be).
Signature · 10 lines
Met vriendelijke groeten, Remco Post SARA - Stichting Academisch Rekencentrum Amsterdam High Performance Computing Tel. +31 20 592 8008 Fax. +31 20 668 3167 "I really didn't foresee the Internet. But then, neither did the computer industry. Not that that tells us very much of course - the computer industry didn't even foresee that the century was going to end." -- Douglas Adams
Quoted 18 lines
Quoted 17 lines · Christopher Friesen
Quoted 14 lines · Richard B. Johnson
Quoted 6 lines · Wolfgang Rohdewald
Quoted 2 lines · J.A. Magallon
When a page is deleted for one executable (because we can re-read it from on-disk binary), it is discarded, not paged out.
What happens if the on-disk binary has changed since loading the program? -
It can't. That's the reason for
installand other methods of changing execututable files (mv exe-file exe-file.old ; cp newfile exe-file). The currently open, and possibly mapped file can be re-named, but it can't be overwritten.
Quoted 7 lines
Quoted 4 lines
Actually, with NFS (and probably others) it can. Suppose I change the file on the server, and it's swapped out on a client that has it mounted. When it swaps back in, it can get the new information.
This sounds really dangerous... What about shared libraries ??
Same problem. This is why most Unix distros tell you to reboot after each patch applied and each OS upgrade. just to be sure that all mmapped files and page-demand loaded bins are all restarted.
Signature · 10 lines
Met vriendelijke groeten, Remco Post SARA - Stichting Academisch Rekencentrum Amsterdam High Performance Computing Tel. +31 20 592 8008 Fax. +31 20 668 3167 "I really didn't foresee the Internet. But then, neither did the computer industry. Not that that tells us very much of course - the computer industry didn't even foresee that the century was going to end." -- Douglas Adams
Re: NFS, Paging & Installing [was: Re: Swap]
In reply to Trond MyklebustQuoted 2 lines · In article <[email protected]> you wrote:
Exactly how, pray tell, does SMB cope with recovering the full state
info after client/server crashes?
Not doing that is the better solution.
Greetings
Bernd
Re: NFS, Paging & Installing [was: Re: Swap]
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Bernd Eckenfels <[email protected]> writes:
> In article <[email protected]> you wrote:
>> Exactly how, pray tell, does SMB cope with recovering the full >> state info after client/server crashes?
> Not doing that is the better solution.
...and is why stateless filesystems are the norm. The claim that SMB was different wasn't mine.
Cheers,
Trond
Re: NFS, Paging & Installing [was: Re: Swap]
In reply to Mike FedykQuoted 13 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Mike Fedyk <[email protected]> writes:
> On Tue, Nov 20, 2001 at 10:22:58PM -0300, Horst von Brand > wrote:
>> Mike Fedyk <[email protected]> said:
>> > Do any newer versions of NFS fix the stateless server >> > problem?
>>
>> This is an _extremely_ hard problem: The server has to know >> somehow what the client thinks the state is... and either one >> (or both) may have been rebooted in between without the other >> one knowing.
Quoted 8 lines
> Yep, but there are currently protocols (SMB) that do that, but > not necessarily in a unix way.
<Cough, choke>
Exactly how, pray tell, does SMB cope with recovering the full state
info after client/server crashes?
No, I wasn't claiming that SMB will recover from a server crash gracefully. If your SMB server goes down (upgrade being likely with samba instead of crash...) for whatever reason, any open file connections are hosed.
I was just stating that there are Network FSes that are stateful, and work good when the server stays up.
As stated by Alan, you can make a stateful Net FS that deals gracefully with crash recovery, it's just harder.
Also, SMB deals with crashed clients pretty well most of the time by querying the client with the write lock to see if it's still there...
Mike
Quoted 2 lines · Wolfgang Rohdewald
I am quite sure this is also possible if the binary is emulated by the linux-abi modules like my old SCO binaries.
Linux-ABI mmaps binaries if they are page-aligned, otherwise they are read completly at startup. Note that Linux-ABI uses the normal binfmt_elf for foreign ELF binaries, so the above applies only to COFF and X.out (Microsoft x.out) binaries.
Christoph
Signature · 1 line
Of course it doesn't work. We've performed a software upgrade.