Question: memory management and QoS
Hello,
I suppose you missed some points or I do not understand you needs. For general computation (and for almost all other workloads),
I think you do not need "reserved" memory - "reserved memory == wasted memory".
With a single memory-hungry computing hog per node in cluster, you may be happy with current Linux MM. As long as working set for this process fits in RAM you'll get top performance, and the system will handle sporadic memory allocations for other process more or less well. If application working set does not fit in RAM, you'll get huge (1000+ times) performance drop and no OS algorithms helps you.
If, additionally, you want guaranteed low latency on data access (for example for doing real-time feed of audio/video/whatever samples), you may lock all process memory to be resident in RAM: mlock(), mlockall() interfaces calls in mind.
Other memory related points on performance gain lay into your application. You should really take into account hierarchical memory structure, and make your application cache-friendly and swap-friendly. For some of my work, I found cache simulator from http://www.cacheprof.org/ to be useful.
QoS issues come to play, if multiple process instances fights with each other for memory resourse. Even when per-user swapfiles sounds overkill for me, fills with many drawbacks and a little benefits:
What you actually suggest is an obscure and inefficient per-user limits of VM usage (to the size of RAM + swapfile size). Beancounters (or other counters) based implementation is both faster and straightforward.
Per-user OOM is again just a per-user VM / whatever resource limit. System OOM can still be triggered in a number of not-so-trivially-to-fix ways:
- many small processes allocated multiple unswapable kernel memory for kernel objects (sockets, signals, locks, descriptors, ...);
- large fragmented network traffic from a fast media.
There is no point in reserving RAM or swap for possible future
allocations: this memory will become wasted memory if no such allocation occurs in near future, and we cannot predict this situation. Additionally, memory reservation policy does not scale well, specifically for systems with many idle users and a couple of active users, where active set of users is often changed.
What will the beancounter patch http://www.asplinux.com.sg/install/ubpatch.shtml
trying to guarantee, is a _minimal_ resident memory for a group of processes. I.e., if some group of processes behaves "well" and do not overcome their limits, their pages are protected from being swapped out due to activity of over processes. This should at least protect from swap-out attacks while one user trashing all memory and other users suffer from heavy swapping.
Quoted 4 lines
Concept of personal swapfiles:
The benefits (among others):
- there wouldn't be system OOM (only per user OOM)
there will be, see above
Quoted 1 line
- user would be able to check his available memory
This buys nothing for users - users will be happy checking his limits/guaranties, and the system will be happy allocating *all* availiable memory to *any* user that need it with a beancounter / swapout guarantee approach while provides you quality of service for "well-behaved" objects.
Quoted 1 line
- no limits for VM address space
?
Your VM is limited by your hardware/software implementation only, and hard disk space. All other limits (per-process, per-users, per-system - the ammount of disk space allocated for swap) are actually administrative constraints.
Quoted 2 lines
- there could be more policies for sharing of physical memory by users (and system)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Quoted 6 lines · Yuri Pudgorodsky
Hello,
I suppose you missed some points or I do not understand you needs. For general computation (and for almost all other workloads),
I think you do not need "reserved" memory - "reserved memory == wasted memory".
Only if reserved == unused. If reserved means 'available when needed' it's completely different question. Other users can use it but when the owner will reclaim it, system will swap them out.
Quoted 7 lines
With a single memory-hungry computing hog per node in cluster, you may be happy with current Linux MM. As long as working set for this process fits in RAM you'll get top performance, and the system will handle sporadic memory allocations for other process more or less well. If application working set does not fit in RAM, you'll get huge (1000+ times) performance drop and no OS algorithms helps you.
You are talking about single user dedicated cluster. I'm talking about multi user resource sharing... To desing application that fits into a cluster (and uses 100 % of resources) is non-trivial task. Of course unless your app is embarrassingly parallel. In computational grid (multiple clusters and single machines with different computing power and resources) application _must_ have guaranteed requested memory or some of nodes will get into the state you described. I didn't heard anybody speak that nodes would be allocated to application exclusively. But I don't want to discuss whether QoS is important or not... (I don't have time to waste on flamewar)
Quoted 4 lines
If, additionally, you want guaranteed low latency on data access (for example for doing real-time feed of audio/video/whatever samples), you may lock all process memory to be resident in RAM:
mlock(),mlockall()interfaces calls in mind.
No, user should have all used physical memory pages (inside his allocation) always resident.
Quoted 5 lines
Other memory related points on performance gain lay into your application. You should really take into account hierarchical memory structure, and make your application cache-friendly and swap-friendly. For some of my work, I found cache simulator from http://www.cacheprof.org/ to be useful.
There's really no doubt that application should be well designed. Is it ? :-)
Quoted 4 lines
QoS issues come to play, if multiple process instances fights with each other for memory resourse. Even when per-user swapfiles sounds overkill for me, fills with many drawbacks and a little benefits:
We'll see...
Quoted 3 lines
What you actually suggest is an obscure and inefficient per-user limits of VM usage (to the size of RAM + swapfile size).
Really ? If user would be able to set the size of his swapfile (according to his needs) or not use swapfile at all where are the limits (except his disk quota set by sysadmin) ? And btw, I would think twice before saying it would be inefficient (see below).
Quoted 2 lines
Beancounters (or other counters) based implementation is both faster and straightforward.
My test on limiting VM space by beancounter showed that mmapping of larger files than VM limit was impossible. IMO that's not the right way...
Quoted 6 lines
Per-user OOM is again just a per-user VM / whatever resource limit. System OOM can still be triggered in a number of not-so-trivially-to-fix ways:
- many small processes allocated multiple unswapable kernel memory for kernel objects (sockets, signals, locks, descriptors, ...);
- large fragmented network traffic from a fast media.
Do you have some numbers how much of these objects will hog 128MB of RAM ? (I could use your numbers for setting limits...) Is there any reason why not to account also this kind of memory to user ?
Quoted 3 lines
There is no point in reserving RAM or swap for possible future
allocations: this memory will become wasted memory if no such allocation
Again. By reserved, I don't mean unused... My fault, I thought it's obvious.
Quoted 9 lines
occurs in near future, and we cannot predict this situation. Additionally, memory reservation policy does not scale well, specifically for systems with many idle users and a couple of active users, where active set of users is often changed.
What will the beancounter patch http://www.asplinux.com.sg/install/ubpatch.shtml
trying to guarantee, is a _minimal_ resident memory for a group of processes. I.e., if some group of processes behaves "well" and do not overcome their limits, their pages are protected from being swapped out due to activity of over processes
I don't claim that beancounter is bad. On the level of physical memory, MM should work exactly this way. Another question is level of virtual memory.
Quoted 2 lines
This should at least protect from swap-out attacks while one user trashing all memory and other users suffer from heavy swapping.
Impossible with per user swapfiles. If process would be outside his allocation and will want another page and system won't have any, page for swapout will be selected from the pages of its owner. So user could trash only his own processes.
Quoted 7 lines
Quoted 5 lines
Concept of personal swapfiles:
The benefits (among others):
- there wouldn't be system OOM (only per user OOM)there will be, see above
:-)
Quoted 8 lines
Quoted 2 lines
- user would be able to check his available memory
This buys nothing for users - users will be happy checking his limits/guaranties, and the system will be happy allocating *all* availiable memory to *any* user that need it with a beancounter / swapout guarantee approach while provides you quality of service for "well-behaved" objects.
and without VM limits system will let users hog all available VM... and with limits there will be no mmap for larger files than VM memory limit. OK, some people can live with it... (Maybe it can be fixed in beancounter.)
I think that marking processes/users as '[well/bad]-behaved' is very unfortunate. There should be strictly defined rules and system just should not let users break them.
Quoted 9 lines
Quoted 2 lines
- no limits for VM address space
?
Your VM is limited by your hardware/software implementation only, and hard disk space. All other limits (per-process, per-users, per-system - the ammount of disk space allocated for swap) are actually administrative constraints.
So you suggests that
A: making a swap disk and setting per user VM memory limit to size/maximal_number_of_simultaneous_users is TheRightWay (tm) how to avoid memory pressure...
I suggest that
B: system swapfile would have only necessary size and won't
be ever touched by users (no system OOM). Only users that _need_ VM would have swapfiles. And possibly
.login
create_swapfile
swapon swapfile
.logout
swapoff swapfile
destroy_swapfile
Q: What approach is more inefficient and wasting disk space ?
And another question: How the size of swapfile (partition) affects the performance of swapping (with regard to fragmentation) ? Has anyone some numbers ?
Cheers,
Jan
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Quoted 7 lines · Jan Astalos
Quoted 3 lines
I suppose you missed some points or I do not understand you needs. For general computation (and for almost all other workloads),
I think you do not need "reserved" memory - "reserved memory == wasted memory".Only if reserved == unused. If reserved means 'available when needed' it's completely different question. Other users can use it but when the owner will reclaim it, system will swap them out.
Yes, I did not read your previous post in such a way. But think of reclaiming swap space, what does it mean? Copying pages from one swapfile to another?
If we want to reclaim used physical pages, we're speaking about "page replacement policy". It is not important where we will store swapped pages: on a partition or a file or multiple files. What is important that's an algorithm used to choose what page to replace.
Moreover, speaking about performance I'm sure fragmenting swap space to multiple files is bad unless these files are on seperate physical disks.
What I wanted to say in previous post too, you buy nothing interesting using swapfile per user. What you need to change to provide a "fairness" to each user is a page replacement strategy.
There are algorithms for local page replacement policy, implemented for years on mainframe OS-es (OS/390, MVS, VMS also has it AFAIK). With such design, if process needs page, OS replaces a page from the process itself, or from another process of the same user. Of course thare are many configurable parameters :-)
Unixes on contast traditionally implement global page replacement polices, using some kind of LRU strategy. This proves to be more better for overall system performance with a large number of users, while local policy allow you to fine tune system for a specific tasks. Global page replacement however often suffer from a page trashing problems when several processes actively replace each other pages. I'd like to see trash protecting algorithms find its way into Linux.
Unfortunatly I did not see good papers comparing one system with another. However I hear OS/390 is able to run 100+ Linux kernels on top of it CP, while Linux cannot :-)
Quoted 5 lines
In computational grid (multiple clusters and single machines with different computing power and resources) application _must_ have guaranteed requested memory or some of nodes will get into the state you described. I didn't heard anybody speak that nodes would be allocated to application exclusively. But I don't want to discuss whether QoS is important or not... (I don't have time to waste on flamewar)
I suppose you want "background" MM policy, to run memory hungry applications.
With swap-out guaranties, you may want to divide all processes to (a) system processes,
(b) user A processes, (c) user B processes, ... And setup reasonable minimal guaranteed RSS for each group.
For true QoS, once in a while I found interesting this paper:
http://www.dcs.gla.ac.uk/~ian/papers/memh.ps
Quoted 6 lines
Quoted 3 lines
If, additionally, you want guaranteed low latency on data access (for example for doing real-time feed of audio/video/whatever samples), you may lock all process memory to be resident in RAM:
mlock(),mlockall()interfaces calls in mind.No, user should have all used physical memory pages (inside his allocation) always resident.
Surely it always better to have your page in RAM then on disk :-) But for most systems this is expensive... or there would no VMM systems at all.
Quoted 7 lines
Quoted 2 lines
What you actually suggest is an obscure and inefficient per-user limits of VM usage (to the size of RAM + swapfile size).
Really ? If user would be able to set the size of his swapfile (according to his needs) or not use swapfile at all where are the limits (except his disk quota set by sysadmin) ? And btw, I would think twice before saying it would be inefficient (see below).
Will, user will set a swapfile to all available disk quota and start trashing VM with " *p = 0; p+= PAGE_SIZE ". With global LRU page replacement,
we will end with all RAM occupied by his pages, and unusable performance for each other.
Or you suggest user disk quota < RAM?
Quoted 2 lines
My test on limiting VM space by beancounter showed that mmapping of larger files than VM limit was impossible. IMO that's not the right way...
I also agreed here. Limiting VM is not practical, there're better resourses to be limited. For example, unswappable memory used for PTE/PGD.
Quoted 9 lines
Quoted 5 lines
Per-user OOM is again just a per-user VM / whatever resource limit. System OOM can still be triggered in a number of not-so-trivially-to-fix ways:
- many small processes allocated multiple unswapable kernel memory for kernel objects (sockets, signals, locks, descriptors, ...);
- large fragmented network traffic from a fast media.Do you have some numbers how much of these objects will hog 128MB of RAM ? (I could use your numbers for setting limits...) Is there any reason why not to account also this kind of memory to user ?
Userbeancounters are for that accounting. The problem is there are many different objects in play here, and sometimes it is not possible to associate them with particular user.
Quoted 63 lines
Quoted 2 lines
There is no point in reserving RAM or swap for possible future
allocations: this memory will become wasted memory if no such allocationAgain. By reserved, I don't mean unused... My fault, I thought it's obvious.
Quoted 9 lines
occurs in near future, and we cannot predict this situation. Additionally, memory reservation policy does not scale well, specifically for systems with many idle users and a couple of active users, where active set of users is often changed.
What will the beancounter patch http://www.asplinux.com.sg/install/ubpatch.shtml
trying to guarantee, is a _minimal_ resident memory for a group of processes. I.e., if some group of processes behaves "well" and do not overcome their limits, their pages are protected from being swapped out due to activity of over processesI don't claim that beancounter is bad. On the level of physical memory, MM should work exactly this way. Another question is level of virtual memory.
Quoted 2 lines
This should at least protect from swap-out attacks while one user trashing all memory and other users suffer from heavy swapping.
Impossible with per user swapfiles. If process would be outside his allocation and will want another page and system won't have any, page for swapout will be selected from the pages of its owner. So user could trash only his own processes.
Quoted 7 lines
Quoted 5 lines
Concept of personal swapfiles:
The benefits (among others):
- there wouldn't be system OOM (only per user OOM)there will be, see above
:-)
Quoted 8 lines
Quoted 2 lines
- user would be able to check his available memory
This buys nothing for users - users will be happy checking his limits/guaranties, and the system will be happy allocating *all* availiable memory to *any* user that need it with a beancounter / swapout guarantee approach while provides you quality of service for "well-behaved" objects.
and without VM limits system will let users hog all available VM... and with limits there will be no mmap for larger files than VM memory limit. OK, some people can live with it... (Maybe it can be fixed in beancounter.)
I think that marking processes/users as '[well/bad]-behaved' is very unfortunate. There should be strictly defined rules and system just should not let users break them.
Quoted 9 lines
Quoted 2 lines
- no limits for VM address space
?
Your VM is limited by your hardware/software implementation only, and hard disk space. All other limits (per-process, per-users, per-system - the ammount of disk space allocated for swap) are actually administrative constraints.
Quoted 5 lines
So you suggests that
A: making a swap disk and setting per user VM memory limit to size/maximal_number_of_simultaneous_users is TheRightWay (tm) how to avoid memory pressure...
No. Overcommiting memory is a normal practice within UNIX world. So you should not want to restrict users in VM space. However you want to protect users using low VM set (actually, low resident pages) from users trashing large VM area. Regardless of swap space used.
So,
1) Allocation a large VM area is OK, even overcommitted (and mmaping a large file is OK too).
You should not want restrict user VM space.
2) If there is a plenty of RAM, active usage of VM by a single user/process should use all available RAM (even 99,9% of overall system memory).
Stealing not used for a long time pages from other users is OK too. Stealing an active page from a user with low memory usage (lower then some configurable value) is not OK.
3) When low-memory-usage user wants a page, we should first try to take it from users with high (over quota) memory usage.
Quoted 10 lines
I suggest that
B: system swapfile would have only necessary size and won't
be ever touched by users (no system OOM). Only users that _need_ VM would have swapfiles. And possibly.login
create_swapfile
swapon swapfile
Quoted 5 lines
.logout
swapoff swapfile
destroy_swapfileQ: What approach is more inefficient and wasting disk space ?
I suppose system will loose performance with each logged in user due to swap space fragmentation.
(A)
+ guarantee physically continuous space
+ less overhead
- difficult to resize, usually resulting in admin decision to waste more disk space
(B)
+ may use less disk space for many usage patterns
+ easy to add/delete space
- is less efficient due to overhead of additional block bitmap operations,
(to convert fs block to pages) but I did not see any actual numbers;
- may be less efficient (and even to order of magnitute) if we take into account swap clustering: we loose ability to cluster swap requests from different users.
Both (A) and (B) needs some QoS-aware "fairness" page replacement algorithms. But IMO (A) has more advantages :-) and using (B) has little to do with QoS.
Quoted 2 lines
And another question: How the size of swapfile (partition) affects the performance of swapping (with regard to fragmentation) ?
Even for unfragmented files, there should be difference. I'm interesting in this numbers too :-)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Quoted 7 lines · Yuri Pudgorodsky
Quoted 4 lines · Jan Astalos
Quoted 3 lines
I suppose you missed some points or I do not understand you needs. For general computation (and for almost all other workloads),
I think you do not need "reserved" memory - "reserved memory == wasted memory".
Quoted 6 lines
Quoted 3 lines
Only if reserved == unused. If reserved means 'available when needed' it's completely different question. Other users can use it but when the owner will reclaim it, system will swap them out.
Yes, I did not read your previous post in such a way. But think of reclaiming swap space, what does it mean? Copying pages from one swapfile to another?
Reclaiming swap space ? What would be that good for ? Pages would change swap file only if they change owner (swapin, chown, swapout).
Quoted 5 lines
If we want to reclaim used physical pages, we're speaking about "page replacement policy". It is not important where we will store swapped pages: on a partition or a file or multiple files. What is important that's an algorithm used to choose what page to replace.
Incorrect. It is _very_ important where we store swapped pages. IMO it makes big difference whether we scatter pages of single process across large swap file(s) or whether we keep it inside relatively small (continuous) disk space. I agree that LRU policy can improve _overall_ performance. But it sacrifices _per_user_ performance.
Quoted 7 lines
Moreover, speaking about performance I'm sure fragmenting swap space to multiple files is bad unless these files are on seperate physical disks.
What I wanted to say in previous post too, you buy nothing interesting using swapfile per user. What you need to change to provide a "fairness" to each user is a page replacement strategy.
How about to split memory QoS into:
- guarantied amount of physical memory
- guarantied amount of virtual memory
The former is much more complicated and includes page replacement policies along with fair sharing of physical memory (true core of QoS).
The latter should gurantee users requested amount of VM. I.e. avoid this kind of situation: successful malloc, a lot of work, killed in action due to OOM ( out of munition^H^H^H^H^H^H^H^Hmemory), RIP...
In the current state it's the problem of system administration. In my approach it will become user's problem. So user would be able to satisfy his need for VM himself and system would only take care of fair management of physical memory.
Quoted 7 lines
There are algorithms for local page replacement policy, implemented for years on mainframe OS-es (OS/390, MVS, VMS also has it AFAIK). With such design, if process needs page, OS replaces a page from the process itself, or from another process of the same user. Of course thare are many configurable parameters :-)
Quoted 6 lines
Unixes on contast traditionally implement global page replacement polices, using some kind of LRU strategy. This proves to be more better for overall system performance with a large number of users, while local policy allow you to fine tune system for a specific tasks. Global page replacement however often suffer from a page trashing problems when several processes actively replace each other pages. I'd like to see trash protecting algorithms find its way into Linux.
Me too...
Quoted 6 lines
Will, user will set a swapfile to all available disk quota and start trashing VM with " *p = 0; p+=
PAGE_SIZE". With global LRU page replacement,
we will end with all RAM occupied by his pages, and unusable performance for each other.Or you suggest user disk quota < RAM?
I think I didn't get your point here...
Quoted 2 lines
Userbeancounters are for that accounting. The problem is there are many different objects in play here, and sometimes it is not possible to associate them with particular user.
But that's not a design flaw, it's a problem of implementation.
Quoted 10 lines
Quoted 5 lines
A: making a swap disk and setting per user VM memory limit to size/maximal_number_of_simultaneous_users is TheRightWay (tm) how to avoid memory pressure...
No. Overcommiting memory is a normal practice within UNIX world. So you should not want to restrict users in VM space. However you want to protect users using low VM set (actually, low resident pages) from users trashing large VM area. Regardless of swap space used.
If you want to implement QoS, you really should avoid overcommiting of memory. Otherwise your clever page replacement techniques will be absolutely useless.
Quoted 6 lines
So,
1) Allocation a large VM area is OK, even overcommitted (and mmaping a large file is OK too).
You should not want restrict user VM space.
100% agreed. But it makes a difference in mapped VM and actually used VM. I'd like to avoid even mention of limits on VM address space (RLIMIT_AS). But it's necessary to limit the number of used VM pages (in order to prevent OOM).
Quoted 4 lines
2) If there is a plenty of RAM, active usage of VM by a single user/process should use all available RAM (even 99,9% of overall system memory).
agreed.
Quoted 4 lines
Stealing not used for a long time pages from other users is OK too. Stealing an active page from a user with low memory usage (lower then some configurable value) is not OK.
In other words, used memory(meaning physical) pages inside the area allocated to a process should be protected from swapout. See ? Not a limit on RSS (beancounter's notation), because it's actually not a limit. Limit will be determined by MM policy (see below).
Quoted 4 lines
3) When low-memory-usage user wants a page, we should first try to take it from users with high (over quota) memory usage.
Exactly. (quota == allocation). But not with LRU. Free memory should be divided between users fairly. I mean, according to the ratio of their allocations.
Quoted 3 lines
I suppose system will loose performance with each logged in user due to swap space fragmentation.
I disagree.
- users with low memory usage (inside their allocation) wont swap at all.
- users with high memory usage will swap into their own (relatively compact) swap file. Therefore it would be possible to exchange a bunch of LRU pages (of this user) with his swapped pages without even moving disk heads.
Quoted 3 lines
(A)
+ guarantee physically continuous space
not at all... even with clustering. Reading the code, the pages of one process can get scattered over swap device. (Correct me if I missed something).
Quoted 1 line
+ less overhead
questionable. Finding free swap page would be much easier. But that's not the point. _per_user_ swapping performance depends on the location of his pages in swap device.
Quoted 1 line
- difficult to resize, usually resulting in admin decision to waste more disk space
i.e. money... (which could be otherwise spend on physical memory ;-)
Quoted 6 lines
(B)
+ may use less disk space for many usage patterns
+ easy to add/delete space
- is less efficient due to overhead of additional block bitmap operations,(to convert fs block to pages) but I did not see any actual numbers;
I'm really interested how large is this overhead comparing to disk seek times. (with regard to increasing CPU speed, readahead stuff in disks and still larger swapfiles)
Quoted 3 lines
- may be less efficient (and even to order of magnitute) if we take into account swap clustering: we loose ability to cluster swap requests from different users.
Maybe, maybe not. I can't see right now how clustering of pages from different users can guarantee (we still talking about QoS) _per_user_ performance...
Quoted 3 lines
Both (A) and (B) needs some QoS-aware "fairness" page replacement algorithms. But IMO (A) has more advantages :-) and using (B) has little to do with QoS.
You should ask users whether they accept QoS that forces them to use calloc(3) to ensure that they really will get requested amount of VM.
IMO, QoS should be transparent to users. If I'll pay for some service, I'll expect some guarantied quality. Not to force me to rewrite my code. There is _much_ more users than developers...
Quoted 7 lines
Quoted 3 lines
And another question: How the size of swapfile (partition) affects the performance of swapping (with regard to fragmentation) ?
Even for unfragmented files, there should be difference. I'm interesting in this numbers too :-)
By swap fragmentation I don't mean fragmentation of swap device, but the fragmentation of swap area.
see the section about swap clustering in http://lwn.net/1999/0121/a/vmreview.html
Jan
PS: Memory management of top-ranking OS of 21st century is not a counting of beans...
Sorry, I could not help myself :-)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
On Mon, Aug 28, 2000 at 10:36:53AM +0200, Jan Astalos wrote:
[snip]
Quoted 7 lines
How about to split memory QoS into:
- guarantied amount of physical memory
- guarantied amount of virtual memoryThe former is much more complicated and includes page replacement policies along with fair sharing of physical memory (true core of QoS).
Quoted 6 lines
The latter should gurantee users requested amount of VM. I.e. avoid this kind of situation: successful malloc, a lot of work, killed in action due to OOM ( out of munition^H^H^H^H^H^H^H^Hmemory), RIP...
In the current state it's the problem of system administration. In my approach it will become user's problem. So user would be able to satisfy his need for VM himself and system would only take care of fair management of physical memory.
That's what user beancounter patch is about.
Except that I'm not so strong in the judgements.
For example, I don't think that overcommits are evil. They are quite ok if
1. the system can provide guarantee that certain processes can never be killed because of OOM;
2. the whole system reaction to OOM situation is well predictable. It's a part of quality of service: some processes/groups of processes have better service, some others only best effort.
It's simply impossible to run Internet servers without overcommits.
I encourage you to take a look at ftp://ftp.sw.com.sg/pub/Linux/people/saw/kernel/user_beancounter/MemoryManagement.html, especially Overcommits section.
I need real guarantees only to some of processes, and I can bear overcommits and 0.01%/year chances for other processes being killed if it saves me the cost of 10Gygabytes of RAM (and the cost of motherboard which supports this amount of memory).
[snip]
Quoted 5 lines
Quoted 3 lines
Userbeancounters are for that accounting. The problem is there are many different objects in play here, and sometimes it is not possible to associate them with particular user.
But that's not a design flaw, it's a problem of implementation.
No.
How do you propose to associate shared pages (or unmapped page cache) with a particular user?
Regards
Andrey V. Savochkin
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Quoted 17 lines · Andrey Savochkin
On Mon, Aug 28, 2000 at 10:36:53AM +0200, Jan Astalos wrote:
[snip]Quoted 7 lines
How about to split memory QoS into:
- guarantied amount of physical memory
- guarantied amount of virtual memoryThe former is much more complicated and includes page replacement policies along with fair sharing of physical memory (true core of QoS).
Quoted 6 lines
The latter should gurantee users requested amount of VM. I.e. avoid this kind of situation: successful malloc, a lot of work, killed in action due to OOM ( out of munition^H^H^H^H^H^H^H^Hmemory), RIP...
In the current state it's the problem of system administration. In my approach it will become user's problem. So user would be able to satisfy his need for VM himself and system would only take care of fair management of physical memory.
Quoted 3 lines
That's what user beancounter patch is about.
Except that I'm not so strong in the judgements.
For example, I don't think that overcommits are evil. They are quite ok if
Did you ever asked your users ? Whether they like to see their apps (possibly running for quite a long time) to be killed (no matter whether with or without warning) ?
Quoted 2 lines
1. the system can provide guarantee that certain processes can never be killed because of OOM;
Again. I wonder how beancounter would prevent overcommit of virtual memory if you don't set limits...
Quoted 3 lines
2. the whole system reaction to OOM situation is well predictable. It's a part of quality of service: some processes/groups of processes have better service, some others only best effort.
I wont repeat it again. With personal swapfiles _all_ users would be guarantied to get the amount of virtual memory provided by _themselves_.
Quoted 2 lines
It's simply impossible to run Internet servers without overcommits.
Which kind of Internet server ? Web server or e-mail server with 100+ active users... Its questionable in what case QoS is more important. (sorry for flamebait)
Quoted 7 lines
I encourage you to take a look at ftp://ftp.sw.com.sg/pub/Linux/people/saw/kernel/user_beancounter/MemoryManagement.html, especially Overcommits section.
I need real guarantees only to some of processes, and I can bear overcommits and 0.01%/year chances for other processes being killed if it saves me the cost of 10Gygabytes of RAM (and the cost of motherboard which supports this amount of memory).
As a user, I won't bear _any_ overcommits at all. Once service is paid, I expect guarantied level of quality. In the case of VM, all the memory I paid for. For all of my processes.
Quoted 12 lines
[snip]
Quoted 5 lines
Quoted 3 lines
Userbeancounters are for that accounting. The problem is there are many different objects in play here, and sometimes it is not possible to associate them with particular user.
But that's not a design flaw, it's a problem of implementation.
No.
How do you propose to associate shared pages (or unmapped page cache) with a particular user?
Do you mean "pages shared between processes of particular user" ? Where's the problem ? If you mean "pages provided by user to another user", I still don't see the problem...
If you mean anonymous pages not owned by any user, I'm really interested why this should be allowed (to let some trash to pollute system resources. Is it common practice ?). OK, this can be solved by allocating some amount of memory (along with swapfile) to anonymous user.
This kind of pages can be (and should be) avoided by communicating via shared files...
(Btw, the best argument I saw so far. I'm really happy that we finally got to real arguments why personal swapfiles wouldn't work. The efficiency question can be solved only with implementation under heavy fire).
Thank you for suggestion...
Jan
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Quoted 1 line · Jan Astalos
Andrey Savochkin wrote:
[snip]
Quoted 7 lines
Quoted 4 lines
That's what user beancounter patch is about.
Except that I'm not so strong in the judgements.
For example, I don't think that overcommits are evil. They are quite ok ifDid you ever asked your users ? Whether they like to see their apps (possibly running for quite a long time) to be killed (no matter whether with or without warning) ?
Well, I was the person responsible for the work of servers (HTTP, FTP, mail, proxy, statistic and accounting etc).
Yes, I want some applications to be killed under certain conditions.
Quoted 6 lines
Quoted 3 lines
1. the system can provide guarantee that certain processes can never be killed because of OOM;
Again. I wonder how beancounter would prevent overcommit of virtual memory if you don't set limits...
It doesn't prevent overcommit.
And it doesn't prevent out-of-memory situations.
It prevents processes that stays below preconfigured threshold to face negative consequences of overcommits, OOM or whatever else.
If OOM happens then _someone_ is over the threshold, and this very one will face the consequences.
You propose exactly the same: user whose swap file ends is the only one who faces problems. The difference is that with my code he likely avoids problems if some other user doesn't consumed all his resources. What you propose is just punish the user unconditionally, even if there are some spare resources...
Quoted 7 lines
Quoted 4 lines
2. the whole system reaction to OOM situation is well predictable. It's a part of quality of service: some processes/groups of processes have better service, some others only best effort.
I wont repeat it again. With personal swapfiles _all_ users would be guarantied to get the amount of virtual memory provided by _themselves_.
Yes, personal swapfiles solve this problem, too. They are just a waste of resources, to be very frank.
[snip]
Quoted 3 lines
As a user, I won't bear _any_ overcommits at all. Once service is paid, I expect guarantied level of quality. In the case of VM, all the memory I paid for. For all of my processes.
It means that you pay orders of magnitude more for it.
Quoted 5 lines
Do you mean "pages shared between processes of particular user" ? Where's the problem ? If you mean "pages provided by user to another user", I still don't see the problem...
If you mean anonymous pages not owned by any user, I'm really interested why this should be allowed (to let some trash to pollute system resources. Is it common practice ?).
Well, you're speaking about private pages only.
I speak about all memory resources, in-core and swap, and all kinds of memory, shared and private, file mapped and anonymous.
Regards
Andrey V. Savochkin
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Andrey Savochkin wrote:
[snip]
Quoted 5 lines
Quoted 3 lines
As a user, I won't bear _any_ overcommits at all. Once service is paid, I expect guarantied level of quality. In the case of VM, all the memory I paid for. For all of my processes.
It means that you pay orders of magnitude more for it.
If I got it right you are speaking about disk space. About sum of disk quotas "orders of magnitude" higher than actual available disk space, right ? You will sell users more disk space than you have for the price of your actual space (and you'll hope that they won't use whole disk).
But you must get the disk space when users will need it (QoS), so in disk shortage, you'll need to buy next one. You'll then send an additional bill to them ?
Quoted 8 lines
Quoted 6 lines
Do you mean "pages shared between processes of particular user" ? Where's the problem ? If you mean "pages provided by user to another user", I still don't see the problem...
If you mean anonymous pages not owned by any user, I'm really interested why this should be allowed (to let some trash to pollute system resources. Is it common practice ?).
Well, you're speaking about private pages only.
No.
Quoted 3 lines
I speak about all memory resources, in-core and swap, and all kinds of memory, shared and private, file mapped and anonymous.
I don't think it's a problem to associate private memory (or private file map) with user. Shared memory should have its owner and permissions. Otherwise I don't know what would be the permissions good for.
Mapped files I didn't considered at all. I thought that they have private swap space (the file). So it's not a problem of personal swapfiles. It's a problem of accounting of physical memory (as I said, I know that this part of MM is much more complicated and I'm not going to write whole MM myself and from scratch :). I hope that beancounter would become more discussed as 2.5 will fork and all the physical memory accounting problems will be touched then.
So from the point of implementation of personal swapfiles it is important to select the right swapfile for swapin/swapout. And solve the cases when a page changes owner. And of course swapon/swapoff. Anything else is in the layer of physical memory management.
Can you be more concrete for whose memory objects (swappable) it is a problem to find owner and why, it would help me a lot (maybe in private e-mail) ? Thanx.
Regards,
Jan
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Quoted 9 lines · Jan Astalos
Andrey Savochkin wrote:
[snip]Quoted 5 lines
Quoted 3 lines
As a user, I won't bear _any_ overcommits at all. Once service is paid, I expect guarantied level of quality. In the case of VM, all the memory I paid for. For all of my processes.
It means that you pay orders of magnitude more for it.
Quoted 7 lines
If I got it right you are speaking about disk space. About sum of disk quotas "orders of magnitude" higher than actual available disk space, right ? You will sell users more disk space than you have for the price of your actual space (and you'll hope that they won't use whole disk).
But you must get the disk space when users will need it (QoS), so in disk shortage, you'll need to buy next one. You'll then send an additional bill to them ?
Well, IMHO it's a matter of numbers. If you're going to use 1/2 of the system resources, you may afford the cost of a whole dedicated system. No quotas, no users, no problems.
If you're going to use 1/1000 of them (so we're speaking of a huge system) you may consider that, on average, no all users will be using their resources, and it makes a lot of sense to overcommit.
If all citizens in a big town want to use their phone at the same time, most of them won't get the service. But it almost never happens. The bigger the numbers involved, the safer to overcommit. It allows the service provider to lower costs a lot. I think no one is selling a phone line that is garanteed to *always* work (it works only for p-o-p links, i.e. leased lines). It would cost too much, and, in practice, give no real advantage...
That's the whole idea behind time(and resource)-sharing systems... Otherwise, that 1/1000 of the huge system will cost *much* more than a personal workstation with better performances. And also you'll see other users, paying 1/100 of what you're paying, get almost the same service (the system almost never fails to fulfill their requests).
And you're not selling "more disk space than you have". You're selling 10MB and the user gets up to 10MB. Shortage *almost* never happens, so you *almost* always provide the service...
.TM.
Signature · 10 lines
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ [email protected]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/Quoted 3 lines · Jan Astalos
I wont repeat it again. With personal swapfiles _all_ users would be guarantied to get the amount of virtual memory provided by _themselves_.
This is STUPID.
Suppose that one user has a 10MB swapfile and a 32MB physical memory quota (quite reasonable or even low nowadays).
Now suppose that user is away from the console (drinking coffee) and has 20MB of IDLE processes sitting around.
In the mean time, another user is running something that could really need a bit more physical memory, but it CANNOT get the memory because the first (coffee drinking) user doesn't have the swap space available...
This is a rediculously inefficient situation that should (and can) be easily avoided by simply having per-user VM and RSS _quotas_, but sharing one system-wide swap area.
regards,
Rik
--
"What you're running that piece of shit Gnome?!?!"
-- Miguel de Icaza, UKUUG 2000
http://www.conectiva.com/ http://www.surriel.com/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/