Arch option to touch newly allocated pages
Subsystems (2)
What I'd like is for the arch to have __alloc_pages touch all pages that it has allocated and is about to return.
The reason for this is that for UML, those pages are backed by host memory, which may or may not be available when they are finally touched at some arbitrary place in the kernel. I hit this by tmpfs running out of room because my UMLs have their memory backed by tmpfs mounted on /tmp. So, I want to be able to dirty those pages before they are seen by any other code.
My first guess at what I want in the code is for all the places that __alloc_pages says this:
if (page) return page;
to change to this:
if (page) return arch_validate(page);
arch_validate would be defined as basically empty somewhere in a include/linux/*.h unless the arch has defined one already. And I may want to add order to the arg list if it can't be inferred from the page alignment.
My arch_validate would look something like this:
struct page_struct *arch_validate(page_struct *page)
{
unsigned long zero = 0;
unsigned long addr = page_address(page);
set_fs(USER_DS);
for(i = 0; i < 1 << order; i++){
if(copy_to_user(addr + i * PAGE_SIZE, &zero, sizeof(zero))){
set_fs(KERNEL_DS);
free_pages(addr, order);
return(NULL);
}
}
set_fs(KERNEL_DS);
return(page);
}The use of set_fs/copy_to_user is somewhat hokey, but that's exactly the effect that I want. Is there a better way of doing that?
So, is this a reasonable thing to do, and is the above the right way of getting it?
Jeff
Quoted 5 lines
The reason for this is that for UML, those pages are backed by host memory, which may or may not be available when they are finally touched at some arbitrary place in the kernel. I hit this by tmpfs running out of room because my UMLs have their memory backed by tmpfs mounted on /tmp. So, I want to be able to dirty those pages before they are seen by any other code.
No - you think you want to dirty the pages - you want to account the address space. What you want to do is run 2.4.18ac3 and do
echo "2" > /proc/sys/vm/overcommit_memory
which on a good day will give you overcommit protection. Your map requests will fail without the pages being dirtied and the extra swap that would cause. It knows about tmpfs too but not ramfs, ramdisk or ptrace yet
Alan
Quoted 6 lines · [email protected] said:
No - you think you want to dirty the pages - you want to account the address space. What you want to do is run 2.4.18ac3 and do
echo "2" >/proc/sys/vm/overcommit_memory
which on a good day will give you overcommit protection. Your map requests will fail without the pages being dirtied and the extra swap that would cause.
That doesn't sound right to me.
I don't have individual little map requests going on here. I have a single large map happening at boot time which creates the UML "physical" memory area.
So, say I have a 128M UML which is only ever going to use 32M of that. If there isn't 128M of address space, but there is 32M, this UML will never get off the ground, even though it really deserved to.
About the swap allocation, I'd bet essentially all the time when a page is allocated, its dirtiness is imminent anyway. So, I'm not adding anything to swap. It'll be there a usec later anyway. What I want is for the dirtying to happen in a controlled place where something sane can be done if the page isn't really there.
Jeff
Quoted 3 lines
I don't have individual little map requests going on here. I have a single large map happening at boot time which creates the UML "physical" memory area.
Doesn't matter
Quoted 3 lines
So, say I have a 128M UML which is only ever going to use 32M of that. If there isn't 128M of address space, but there is 32M, this UML will never get off the ground, even though it really deserved to.
Well thats up to you on how you implement it. mmap will tell you the truth in overcommit mode 2 or 3. Nothing will get killed off when you try and mmap too much or dirty pages you have.
Quoted 2 lines
About the swap allocation, I'd bet essentially all the time when a page is allocated, its dirtiness is imminent anyway. So, I'm not adding anything
Nothing of the sort. Sitting in a gnome desktop I'm showing a 41200Kb worst case swap requirement, but it appears under half of that is used.
Quoted 3 lines
to swap. It'll be there a usec later anyway. What I want is for the dirtying to happen in a controlled place where something sane can be done if the page isn't really there.
Like randomly killing another process off ? If you want to dirty the pages pray and catch the sigbus then see memset(3). If you want to be told "sorry you can't have that" and write a simple loop to pick a good memory size, you need the address space accounting.
Alan
Quoted 4 lines · [email protected] said:
Like randomly killing another process off ? If you want to dirty the pages pray and catch the sigbus then see memset(3). If you want to be told "sorry you can't have that" and write a simple loop to pick a good memory size, you need the address space accounting.
OK, this sounds right if the machine is short of memory. Random hacks to do something reasonable if a SIGBUS manages to gets through aren't the way to go when random process deaths are what happen if it doesn't.
However, the host wasn't under a global memory shortage. The UML hit the tmpfs size limit.
Does address space accounting enforce tmpfs limits (and other limits, like RSS, when it happens)? Or is it enforcing a global limit?
When the host isn't in a memory shortage and UML is running under a sub-limit (as with tmpfs), either of those gives me worse behavior than I get by being able to trap the SIGBUS. It will arrive reliably without accompanying process deaths. The first case means that the UML won't get off the ground even though it would be able to deal semi-gracefully with tmpfs running out of room. The second means that the mmap will succeed and I'm back to SIGBUS anyway.
Quoted 3 lines
Nothing of the sort. Sitting in a gnome desktop I'm showing a 41200Kb worst case swap requirement, but it appears under half of that is used.
This I don't get. I'm assuming that the vast majority of the time when a set of pages is returned by __alloc_pages, they all are going to be written pretty soon. This being the case, how can it possibly affect anything to touch them at the end of __alloc_pages?
Jeff
Quoted 2 lines
Does address space accounting enforce tmpfs limits (and other limits, like RSS, when it happens)? Or is it enforcing a global limit?
It ensures that the total number of anonymous and/or tmpfs (eg anon shared) pages that are mappable will fit in swap (or in mode 2 swap + 0.5*ram). You never get a SIGBUS. Writes to tmpfs for new blocks will fail if that would place the system in a potential overcommit situation.
Quoted 8 lines
Quoted 3 lines
Nothing of the sort. Sitting in a gnome desktop I'm showing a 41200Kb worst case swap requirement, but it appears under half of that is used.
This I don't get. I'm assuming that the vast majority of the time when a set of pages is returned by
__alloc_pages, they all are going to be written pretty soon. This being the case, how can it possibly affect anything to touch them at the end of__alloc_pages?
It isnt the alloc pages that is the problem.
You mmap - no pages are allocated. You use them , pages get allocated. If you look at the actual maps you'll find a lot of people allocate an area of address space but don't use it all. Without the address overcommit management nothing guarantees that when you touch those pages you won't fault. Furthermore unless you are very careful you may fault again on the stack push for the SIGBUS and if that faults - SIGKILL->OOM time
Alan
Quoted 2 lines · [email protected] said:
You never get a SIGBUS. Writes to tmpfs for new blocks will fail if that would place the system in a potential overcommit situation.
How will writes to tmpfs fail if we're not in an overcommit situation, but tmpfs is full? Unless tmpfs is changed, it looks to me like you get a SIGBUS.
Quoted 1 line
It isnt the alloc pages that is the problem.
We are somehow failing to communicate...
Quoted 1 line
You mmap - no pages are allocated.
I understand this.
Quoted 1 line
You use them , pages get allocated.
This too.
Quoted 2 lines
If you look at the actual maps you'll find a lot of people allocate an area of address space but don't use it all.
Yes.
Quoted 3 lines
Without the address
overcommit management nothing guarantees that when you touch those pages you won't fault.
Even with address overcommit management, I can fault if I touch pages when tmpfs is full but the system is not near overcommit.
Quoted 3 lines
Furthermore unless you are very careful you may
fault again on the stack push for the SIGBUS and if that faults -SIGKILL->OOMtime
We are talking about UML kernel stacks. If they have been allocated the way I'm proposing with the UML __alloc_pages touching each page on the way out, they are allocated on the host, and therefore can't fault.
This seems to me to be sufficiently careful.
One of us is missing something, who is it?
Jeff
Quoted 2 lines
Even with address overcommit management, I can fault if I touch pages when tmpfs is full but the system is not near overcommit.
That is what mmap defines for a file based mapping yes. Thats a case where there isnt much else you can do
Quoted 2 lines · [email protected] said:
That is what mmap defines for a file based mapping yes. Thats a case where there isnt much else you can do
Except the whole point of me starting this thread is that there is something sane that UML can do *if* it can trap those bus errors in a controlled way.
If UML can detect pages which tmpfs can't back as they leave the allocator, then it can prevent the rest of the UML kernel from getting randomly SIGBUSed as it touches those pages.
To recap in case it got lost in the confusion, I want __alloc_pages to call an arch hook before it return memory, turning every instance of
if (page) return page;
into
if (page) return arch_validate(page);
Unless the arch defines its own arch_validate(), a generic header would define it as
static inline arch_validate(struct page_struct *page){ return page; }or the equivalent macro.
On the other hand, UML would define it to touch each page in the allocation, trapping SIGBUS there. If any do SIGBUS, then my orginal proposal was to free the block back to the allocator and return NULL. This would cause a flurry of allocation failures to things that weren't willing to sleep, and if that causes trouble, then the caller needed fixing anyway.
A more interesting idea is to hang on to the block and maybe unmap it. Umapping would free any backed pages in the block back to tmpfs, giving it (and the other UMLs, if any) some breathing room. Even if the entire block was unbacked, the UML would lose it as being allocatable and would eventually be restricted to handing out pages that it had managed to touch before tmpfs ran out.
This is way more sane than the current get-a-SIGBUS-someplace-random-and-panic situation I have now.
Given that we are talking about tmpfs running out of space, the host still has plenty of free memory, and UML kernel stacks can receive the SIGBUS (because they've been allocated with this mechanism), is this still objectionable?
Jeff
Quoted 3 lines
If UML can detect pages which tmpfs can't back as they leave the allocator, then it can prevent the rest of the UML kernel from getting randomly SIGBUSed as it touches those pages.
Yes I follow this. I don't understand how it is related to your intended solution.
Quoted 2 lines
To recap in case it got lost in the confusion, I want
__alloc_pagesto call an arch hook before it return memory, turning every instance of
alloc_pages is only called at the time the backing page is created - by then it doesnt matter - its too late. You'd need to hack up the same code areas that are used for mlock MCL_FUTURE not alloc_pages
Quoted 4 lines
Given that we are talking about tmpfs running out of space, the host still has plenty of free memory, and UML kernel stacks can receive the SIGBUS (because they've been allocated with this mechanism), is this still objectionable?
With the vm no overcommit code the tmpfs cannot run out of space filling in pages, only when you make a tmpfs file larger. The code guarantees there are swap pages available to back between offset 0 and the file size. A write extending a tmpfs file may fail reporting the disk full.
The code guarantees (modulo bugs of course!) that the total number of pages that could be created by touching addresses that have already been mapped including accounting for tmpfs on the basis above never exceeds the number of pages available.
The bugs at the moment being
1. ptrace isnt accounted for its special weirdnesses
2. MAP_NORESERVE isnt forcibly accounted in these modes as required
Quoted 2 lines · [email protected] said:
alloc_pagesis only called at the time the backing page is created - by then it doesnt matter - its too late.
*My* (i.e. the one inside UML) alloc_pages, not the host's would do the dirtying. That's the whole point. The UML alloc_pages would make sure that the pages it hands out are backed on the host before they are handed out to the rest of UML.
Jeff
Quoted 7 lines
Quoted 2 lines
alloc_pagesis only called at the time the backing page is created - by then it doesnt matter - its too late.*My* (i.e. the one inside UML)
alloc_pages, not the host's would do the dirtying. That's the whole point. The UMLalloc_pageswould make sure that the pages it hands out are backed on the host before they are handed out to the rest of UML.
Ok got you - so its merely grossly ineffecient and downright rude to other users of the system ?
Quoted 1 line · [email protected] said:
Ok got you -
Good, if that's not being sarcastic...
Quoted 2 lines
so its merely grossly ineffecient and downright rude to other users of the system ?
OK, when something calls alloc_pages and gets back some pages, it's almost always going to modify them immediately, right?
If this is true, then what I'm proposing would force the host to find backing memory for those pages a tiny bit earlier than it would have had to otherwise.
This is the only possibility for inefficiency and rudeness that I can see. If I'm totally missing what you are referring to, please be a little bit more specific.
Jeff
Quoted 2 lines
OK, when something calls
alloc_pagesand gets back some pages, it's almost always going to modify them immediately, right?
Yes. Which is why we don't allocate the when you map an object or create a shmem fs file
Quoted 2 lines
If this is true, then what I'm proposing would force the host to find backing memory for those pages a tiny bit earlier than it would have had to otherwise.
In the normal case about half of the pages are never allocated that are mapped. In other words no alloc_pages was ever done for them or will ever be needed.
Followup to: <[email protected]> By author: Jeff Dike <[email protected]> In newsgroup: linux.dev.kernel
Quoted 8 lines
Even with address overcommit management, I can fault if I touch pages when tmpfs is full but the system is not near overcommit.
Quoted 3 lines
Furthermore unless you are very careful you may
fault again on the stack push for the SIGBUS and if that faults -SIGKILL->OOMtime
Quoted 8 lines
We are talking about UML kernel stacks. If they have been allocated the way I'm proposing with the UML
__alloc_pagestouching each page on the way out, they are allocated on the host, and therefore can't fault.This seems to me to be sufficiently careful.
One of us is missing something, who is it?
I think it's you -- you seem to suffer from the "my application is the only one that counts" syndrome. If you want to pages dirtied, then dirty them using memset() or similar.
-hpa
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
Quoted 3 lines · [email protected] said:
I think it's you -- you seem to suffer from the "my application is the only one that counts" syndrome. If you want to pages dirtied, then dirty them using
memset()or similar.
I think you and Alan think I want the host kernel to do the dirtying. Not so, I want no changes on the host. I want a hook that UML can use to make sure that all pages that it allocates are backed.
And memset or something similar is exactly what I have in mind.
Jeff
Quoted 7 lines · Jeff Dike
Quoted 4 lines · [email protected] said:
I think it's you -- you seem to suffer from the "my application is the only one that counts" syndrome. If you want to pages dirtied, then dirty them using
memset()or similar.
Quoted 6 lines
I think you and Alan think I want the host kernel to do the dirtying. Not so, I want no changes on the host. I want a hook that UML can use to make sure that all pages that it allocates are backed.
And memset or something similar is exactly what I have in mind.
So why, then, phrase this as a feature request???
-hpa
Quoted 1 line · [email protected] said:
So why, then, phrase this as a feature request???
Because it requires a hook in the generic kernel allocator that UML can use to make sure that all allocated pages are backed on the host.
Jeff
Quoted 5 lines
Quoted 1 line · [email protected] said:
So why, then, phrase this as a feature request???
Because it requires a hook in the generic kernel allocator that UML can use to make sure that all allocated pages are backed on the host.
At the point you actually allocate pages they are being allocated. No hook is needed. You seem to misunderstand the way the allocation works - we allocate address space not memory in things like mmap. We allocate pages on demand when referenced. The page allocator is only called after a page is referenced
Quoted 4 lines · [email protected] said:
You seem to misunderstand the way the allocation works - we allocate address space not memory in things like mmap. We allocate pages on demand when referenced. The page allocator is only called after a page is referenced
I understand perfectly well how it works.
You still don't understand what I'm talking about. To make this a bit more concrete, the patch below implements what I want (plus a couple of bug fixes needed to make it work).
If you want to run it, apply the 2.4.18-2 UML patch (available at
http://prdownloads.sourceforge.net/user-mode-linux/uml-patch-2.4.18-2.bz2) to a stock 2.4.18 pool. Copy the pool, apply the patch below to one of them, and build both.
Mount a 64M tmpfs on /tmp, boot up two 64M UMLs without the patch, run a -j 2 kernel build in each and watch them hang (see http://user-mode-linux/sf.net
for lots of docs, filesystem images, etc if you haven't run UML before). If you have gdb running on them, you will see that they're stuck at some random place in the kernel taking an infinite stream of SIGBUSes on a page that tmpfs can't back. If you apply the relay_signal piece of the patch to this pool, you will get panics instead of hangs.
Now do the same with two 64M UMLs with the patch. You will see the build die like this, but the UMLs stay up and they're fairly healthy:
gcc -D__KERNEL__ -I/kernel/include -Wall -Wstrict-prototypes -Wno-trigraphs -O2 -fomit-frame-pointer -fno-strict-aliasing -fno-common -pipe -mpreferred-stack-boundary=2 -march=i686 -c -o dma.o dma.c
cpp: output pipe has been closed
gcc: Internal compiler error: program cc1 got fatal signal 11 make[2]: *** [dma.o] Error 1
Note the following:
the host is not short of memory, so address space accounting and the
possibility of random process deaths do not come into play
you did not build or reboot the host kernel - all this is strictly
inside UML
the code added to mm.h is a no-op for every arch but UML
So, does this make things at all clearer? Without the patch I get random UML deaths when tmpfs can't back a page. With it, tmpfs is forced to back newly allocated pages when they're allocated, and the allocation returns NULL if it can't. The result being I get no UML deaths and fairly reasonable behavior.
Jeff
arch/um/kernel/exec_kern.c +6 -0
@@ -38,6 +38,12 @@int new_pid; stack =alloc_stack (); +if (stack ==0 ){ +printk (KERN_ERR +"flush_thread : failed to allocate temporary stack\n" ); +do_exit (SIGKILL ); + } + new_pid =start_fork_tramp ((void *) current->thread.kernel_stack, stack,0 , exec_tramp);if (new_pid <0 ){
arch/um/kernel/mem.c +26 -0
@@ -212,6 +212,32 @@" just be swapped out.\n Example: mem=64M\n\n" ); +struct page *arch_validate (struct page *page,int order) +{ +unsigned long addr, zero =0 ; +int i; + + addr = (unsigned long )page_address (page); +for (i =0 ; i < (1 << order); i++){ + current->thread.fault_addr = (void *) addr; +if (__do_copy_to_user ((void *) addr, &zero,sizeof (zero), + ¤t->thread.fault_addr, + ¤t->thread.fault_catcher)) +return (NULL ); + addr +=PAGE_SIZE ; + } +return (page); +} + +extern void relay_signal (int sig,void *sc,int usermode); + +void bus_handler (int sig,void *sc,int usermode) +{ +if (current->thread.fault_catcher !=NULL ) +do_longjmp (current->thread.fault_catcher); +else relay_signal (sig, sc, usermode); +} +/* * Overrides for Emacs so that we follow Linus's tabbing style. * Emacs will notice this stuff at the end of the file and automatically
arch/um/kernel/process_kern.c +6 -1
@@ -141,7 +141,7 @@unsigned long page;if ((page =__get_free_page (GFP_KERNEL )) ==0 ) -panic ("Couldn't allocate new stack" ); +return (0 );stack_protections (page);return (page); }
@@ -318,6 +318,11 @@panic ("copy_thread : pipe failed" );if (current->thread.forking){ stack =alloc_stack (); +if (stack ==0 ){ +printk (KERN_ERR "copy_thread : failed to allocate " +"temporary stack\n" ); +return (-ENOMEM ); + } clone_vm = (p->mm == current->mm); p->thread.temp_stack = stack; new_pid = start_fork_tramp((void *) p->thread.kernel_stack,
arch/um/kernel/trap_kern.c +5 -2
@@ -30,6 +30,7 @@struct mm_struct *mm = current->mm;struct vm_area_struct *vma;struct siginfo si; +void *catcher;pgd_t *pgd;pmd_t *pmd;pte_t *pte;
@@ -40,6 +41,7 @@return (0 ); }if (mm ==NULL )panic ("Segfault with no mm" ); + catcher = current->thread.fault_catcher; si.si_code =SEGV_MAPERR ;down_read (&mm->mmap_sem); vma =find_vma (mm, address);
@@ -84,10 +86,10 @@up_read (&mm->mmap_sem);return (0 ); bad: -if (current->thread.fault_catcher !=NULL ) { +if (catcher !=NULL ) { current->thread.fault_addr = (void *) address;up_read (&mm->mmap_sem); -do_longjmp (current->thread.fault_catcher); +do_longjmp (catcher); }else if (current->thread.fault_addr !=NULL ){panic ("fault_addr set but no fault catcher" );
@@ -120,6 +122,7 @@void relay_signal (int sig,void *sc,int usermode) { +if (!usermode)panic ("Kernel mode signal %d" , sig);force_sig (sig, current); }
arch/um/kernel/trap_user.c +3 -1
@@ -420,11 +420,13 @@extern int timer_ready, timer_on; +extern void bus_handler (int sig,void *sc,int usermode); +static void (*handlers[])(int ,void *,int ) = { [ SIGTRAP ] relay_signal, [ SIGFPE ] relay_signal, [ SIGILL ] relay_signal, - [ SIGBUS ] relay_signal, + [ SIGBUS ] bus_handler, [ SIGSEGV] segv_handler, [ SIGIO ] sigio_handler, [ SIGVTALRM ] timer_handler,
include/asm-um/page.h +3 -0
@@ -42,4 +42,7 @@#define virt_to_page (kaddr) (mem_map + (__pa(kaddr) >> PAGE_SHIFT))#define VALID_PAGE (page) ((page - mem_map) < max_mapnr) +extern struct page *arch_validate (struct page *page,int order); +#define HAVE_ARCH_VALIDATE +#endif
include/linux/mm.h +8 -1
@@ -358,6 +358,13 @@extern struct page *FASTCALL (__alloc_pages (unsigned int gfp_mask,unsigned int order,zonelist_t *zonelist));extern struct page *alloc_pages_node (int nid,unsigned int gfp_mask,unsigned int order); +#ifndef HAVE_ARCH_VALIDATE +static inline struct page *arch_validate (struct page *page,int order) +{ +return (page); +} +#endif +static inline struct page *alloc_pages (unsigned int gfp_mask,unsigned int order) {/*
@@ -365,7 +372,7 @@*/ if (order >=MAX_ORDER )return NULL ; -return _alloc_pages (gfp_mask, order); +return arch_validate (_alloc_pages (gfp_mask, order), order); }#define alloc_page (gfp_mask) alloc_pages(gfp_mask, 0)
Quoted 7 lines · Jeff Dike
So, does this make things at all clearer? Without the patch I get random UML deaths when tmpfs can't back a page. With it, tmpfs is forced to back newly allocated pages when they're allocated, and the allocation returns NULL if it can't. The result being I get no UML deaths and fairly reasonable behavior.
From your explanation of things, you only need to do the memsets once at
startup of UML where the ram is allocated -> a uml booted with 64MB of ram would write into every page of the backing store file before even running the kernel. Doesn't that accomplish the same thing?
-ben
Quoted 5 lines · [email protected] said:
From your explanation of things, you only need to do the memsets once at startup of UML where the ram is allocated -> a uml booted with 64MB of ram would write into every page of the backing store file before even running the kernel. Doesn't that accomplish the same thing?
Sort of, but it's very heavy-handed. The UML will force memory to be allocated on the host long before it will ever be needed, and it may never be needed. This patch doesn't waste memory like that.
Jeff
Quoted 9 lines · Jeff Dike
Quoted 6 lines · [email protected] said:
From your explanation of things, you only need to do the memsets once at startup of UML where the ram is allocated -> a uml booted with 64MB of ram would write into every page of the backing store file before even running the kernel. Doesn't that accomplish the same thing?
Quoted 4 lines
Sort of, but it's very heavy-handed. The UML will force memory to be allocated on the host long before it will ever be needed, and it may never be needed. This patch doesn't waste memory like that.
This is not necessarily a bad thing, however. If the user hadn't set up enough swap, they're probably better off getting the error message early.
-hpa
Quoted 3 lines · [email protected] said:
This is not necessarily a bad thing, however. If the user hadn't set up enough swap, they're probably better off getting the error message early.
This is not a situation in which a lack of swap or a lack of RAM is a problem.
The problem is a tmpfs filling up.
You think that UML refusing to run if it can't get every bit of memory it might ever need is preferable to UML running fine in somewhat less memory?
Jeff
Quoted 7 lines · Jeff Dike
Quoted 4 lines · [email protected] said:
This is not necessarily a bad thing, however. If the user hadn't set up enough swap, they're probably better off getting the error message early.
Quoted 7 lines
This is not a situation in which a lack of swap or a lack of RAM is a problem.
The problem is a tmpfs filling up.
You think that UML refusing to run if it can't get every bit of memory it might ever need is preferable to UML running fine in somewhat less memory?
Actually, yes, esp. since the only case you have been able to bring up is one of the sysadmin being a moron.
-hpa
Quoted 7 lines · H. Peter Avin
Quoted 2 lines · Jeff Dike
You think that UML refusing to run if it can't get every bit of memory it might ever need is preferable to UML running fine in somewhat less memory?
Actually, yes, esp. since the only case you have been able to bring up is one of the sysadmin being a moron.
I could easily imagine it being useful to run multiple UMLs on one machine (to simulate a network, say), and that one's application causes each UML to occasionally spike in its memory requirements. Then it would be disappointing for the number of UMLs one could run to be determined by this maximum memory requirement, rather than by the average memory requirement (minus some leeway for a few spiking UMLs).
The hook Jeff asks for seems harmless enough. If there is some disagreement about how UML interacts with the host kernel on memory allocation, the two different modes could be a configuration option of UML. The "touch it all at startup" option could be the default, as it does make alot of sense for the single UML case.
Cheers,
Wayne
Quoted 2 lines · [email protected] said:
Actually, yes, esp. since the only case you have been able to bring up is one of the sysadmin being a moron.
Really? And you're unconcerned about the impact on the rest of the system of a UML grabbing (say) 128M of memory when it starts up? Especially if it may never use it?
And I don't see anything wrong with starting a bunch of UMLs with a total maximum memory exceeding the available tmpfs as long as they don't all need all that memory at once. And, if they do, the patch I just posted will let them deal fairly sanely with the situation.
Jeff
Quoted 6 lines · Jeff Dike
Really? And you're unconcerned about the impact on the rest of the system of a UML grabbing (say) 128M of memory when it starts up? Especially if it may never use it?
It doesn't grab memory, it grabs backing store. The kernel will swap it out as necessary.
Quoted 5 lines
And I don't see anything wrong with starting a bunch of UMLs with a total maximum memory exceeding the available tmpfs as long as they don't all need all that memory at once. And, if they do, the patch I just posted will let them deal fairly sanely with the situation.
Bullshit. It means you have moved your system into an insane corner case, and you would have been better off denying access in the first place.
-hpa
Quoted 3 lines
maximum memory exceeding the available tmpfs as long as they don't all need all that memory at once. And, if they do, the patch I just posted will let them deal fairly sanely with the situation.
And the address space management stuff in the -ac tree will do all that and more without force allocating pages and regardless of what other apps do including without allowing your rude app to kill them.
You are using an axe to batter down a door. Worse than that I fitted a perfectly good door handle.
Quoted 4 lines · [email protected] said:
And I don't see anything wrong with starting a bunch of UMLs with a
total maximum memory exceeding the available tmpfs as long as they don't all need all that memory at once. And, if they do, the patch I just posted will let them deal fairly sanely with the situation.
Going off at a slight tangent...
You say 'at once'. Does UML somehow give pages back to the host when they're freed, so the pages that are no longer used by UML can be discarded by the host instead of getting swapped?
--
dwmw2
Quoted 3 lines · [email protected] said:
Does UML somehow give pages back to the host when they're freed, so the pages that are no longer used by UML can be discarded by the host instead of getting swapped?
No, but it could. Given another hook (in free_pages this time) I could unmap pages as they're freed, allowing them to be discarded on the host.
Jeff
Quoted 3 lines
You say 'at once'. Does UML somehow give pages back to the host when they're freed, so the pages that are no longer used by UML can be discarded by the host instead of getting swapped?
Doesn't seem to but it looks like madvise might be enough to make that happen. That BTW is an issue for more than UML - it has a bearing on running lots of Linux instances on any supervisor/virtualising system like S/390
Quoted 2 lines · [email protected] said:
Doesn't seem to but it looks like madvise might be enough to make that happen.
Yeah, MADV_DONTNEED looks right. UML and Linux/s390 (assuming VM has the equivalent of MADV_DONTNEED) would need a hook in free_pages to make that happen.
Quoted 2 lines
That BTW is an issue for more than UML - it has a bearing on running lots of Linux instances on any supervisor/virtualising system like S/390
On a side note, the "unused memory is wasted memory" behavior that UML and Linux/s390 inherit is also less than optimal for the host.
Jeff
Quoted 3 lines
Yeah,
MADV_DONTNEEDlooks right. UML and Linux/s390 (assuming VM has the equivalent ofMADV_DONTNEED) would need a hook infree_pagesto make that happen.
VM allows you to give it back a page and if you use it again you get a clean copy. What it seems to lack is the more ideal "here have this page and if I reuse it trap if you did throw it out" semantic.
Quoted 5 lines
Quoted 2 lines
That BTW is an issue for more than UML - it has a bearing on running lots of Linux instances on any supervisor/virtualising system like S/390
On a side note, the "unused memory is wasted memory" behavior that UML and Linux/s390 inherit is also less than optimal for the host.
Yes. I believe IBM folks are studying that
Quoted 6 lines · Alan Cox
Quoted 3 lines
Yeah,
MADV_DONTNEEDlooks right. UML and Linux/s390 (assuming VM has the equivalent ofMADV_DONTNEED) would need a hook infree_pagesto make that happen.VM allows you to give it back a page and if you use it again you get a clean copy.
Yep, clean as in a page of zeroes when you touch it. (DIAGNOSE X'10' as documented in the "CP Programming Services" manual, to be precise).
Quoted 2 lines
What it seems to lack is the more ideal "here have this page
and if I reuse it trap if you did throw it out" semantic.
We're looking at ways of having fancier memory management information pass between Linux and CP (it's safer to say CP (the "kernel" part of VM/ESA and z/VM) than VM, given the ambiguous and confusing dual meaning of "VM" otherwise :-).
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
That BTW is an issue for more than UML - it has a bearing on running lots of Linux instances on any supervisor/virtualising system like S/390
On a side note, the "unused memory is wasted memory" behavior that UML and Linux/s390 inherit is also less than optimal for the host.
Yes. I believe IBM folks are studying that
Indeed. A "quich hack" that turns out to have rather useful, fun properties is to have a little device driver (can be a module) which stores "negative pages" in the page cache by allocating page cache pages for the device's inode and then invoking the CP "release page" call mentioned above. Linux thinks the page is "useful" and so keeps it around until memory pressure kicks it out whereas the underlying CP knows it's a hole making the resident size and working set of the Linux image reduce. Add in a bit of feedback to get Linux re-reading the "device" into cache proportionally to how much CP wants to kick *out* resident pages from the image. Fun... However, closer integration with the main mm system is the "proper" way to do it (but depends on stuff like the latency, overheads and information shared with CP so is a little more than an afternoon hack.)
--Malcolm
Signature · 4 lines
Malcolm Beattie <[email protected]> Linux Technical Consultant IBM EMEA Enterprise Server Group... ...from home, speaking only for myself
Quoted 5 lines · [email protected] said:
A "quich hack" that turns out to have rather useful, fun properties is to have a little device driver (can be a module) which stores "negative pages" in the page cache by allocating page cache pages for the device's inode and then invoking the CP "release page" call mentioned above.
Yeah, I was thinking about something like that. It's unclear how it should figure out how much memory to grab, though. You'd have to get some idea how desperate the host is for memory and balance that off against how desperate the VM is.
And you want to avoid doing things that just aggravate the host's situation, i.e. if it is swapping its brains out, you want the VM to just drop some clean pages and you definitely don't want it swapping dirty ones and add to the host's IO load.
Quoted 4 lines
However, closer
integration with the main mm system is the "proper" way to do it (but depends on stuff like the latency, overheads and information shared with CP so is a little more than an afternoon hack.)
Yup.
Is any of your (you or IBM in general) thinking on this written down publically anywhere?
Jeff
Quoted 3 lines · [email protected] said:
Yeah,
MADV_DONTNEEDlooks right. UML and Linux/s390 (assuming VM has
the equivalent ofMADV_DONTNEED) would need a hook infree_pagesto make that happen.
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources asso ciated with it.)
It's not clear from that that the host kernel is actually permitted to discard the data.
Quoted 3 lines · [email protected] said:
VM allows you to give it back a page and if you use it again you get
a clean copy. What it seems to lack is the more ideal "here have this page and if I reuse it trap if you did throw it out" semantic.
I've wittered on occasion about other situations where such semantics might be useful -- essentially 'drop these pages if you need to as if they were clean, and tell me when I next touch them so I can recreate their data'.
UML might want that kind of thing for its (clean) page cache pages or something, but for pages allocated for kernel stack and task struct we really want the opposite -- we want to make sure they're present when we allocate them, and explicitly discard them when we're done.
--
dwmw2
Quoted 16 lines · David Woodhouse
Quoted 3 lines · [email protected] said:
Yeah,
MADV_DONTNEEDlooks right. UML and Linux/s390 (assuming VM has
the equivalent ofMADV_DONTNEED) would need a hook infree_pagesto make that happen.
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources asso� ciated with it.)It's not clear from that that the host kernel is actually permitted to discard the data.
[email protected] said:
Quoted 8 lines
Quoted 3 lines
VM allows you to give it back a page and if you use it again you get
a clean copy. What it seems to lack is the more ideal "here have this page and if I reuse it trap if you did throw it out" semantic.I've wittered on occasion about other situations where such semantics might be useful -- essentially 'drop these pages if you need to as if they were clean, and tell me when I next touch them so I can recreate their data'.
Indeed. I'd love such a feature. It's got applications in numerical/scientific code, not just UML.
Regards, Richard.... Permanent: [email protected] Current: [email protected]
Quoted 7 lines · [email protected] said:
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources asso ciated with it.)
It's not clear from that that the host kernel is actually permitted to discard the data.
Hmmm, you have better man pages than me. I don't have an madvise man page on either of my boxes (RH 6.2 and 7.2 :-)
Quoted 1 line
From that description, you're right. The code is very clear on what happens,
as is the comment above sys_madvise:
* MADV_DONTNEED - the application is finished with the given range,
* so the kernel can free resources associated with it.
Quoted 4 lines
UML might want that kind of thing for its (clean) page cache pages or something, but for pages allocated for kernel stack and task struct we
really want the opposite -- we want to make sure they're present when
we allocate them, and explicitly discard them when we're done.
Yeah, that's a decent idea. If you were going to make it fancier, you could cover the case that the UML's clean pages are all busy but it has lots of old dirty pages lying around. But then you'd need some way for the host to tell the UML that I/O would be a really bad idea and it should just dump clean pages.
Jeff
Quoted 3 lines
Hmmm, you have better man pages than me. I don't have an madvise man pag= e
on either of my boxes (RH 6.2 and 7.2 :-)
Curious. I have one on my 7.2 box 8)
Quoted 6 lines
Quoted 3 lines · [email protected] said:
Yeah,
MADV_DONTNEEDlooks right. UML and Linux/s390 (assuming VM has
the equivalent ofMADV_DONTNEED) would need a hook infree_pagesto make that happen.
Quoted 8 lines
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources asso ciated with it.)It's not clear from that that the host kernel is actually permitted to discard the data.
Solaris has MADV_FREE to say that the data can be discarded...
-joe
Hi!
Quoted 8 lines
Quoted 3 lines
You say 'at once'. Does UML somehow give pages back to the host when they're freed, so the pages that are no longer used by UML can be discarded by the host instead of getting swapped?
Doesn't seem to but it looks like madvise might be enough to make that happen. That BTW is an issue for more than UML - it has a bearing on running lots of Linux instances on any supervisor/virtualising system like S/390
I just imagined hardware which supports freeing memory -- just do not refresh it any more to conserve power ;-))).
Granted, it would probably only make sense in big chunks, like 2MB or so... It might make sense for a PDA...
Pavel
Signature · 2 lines
(about SSSCA) "I don't say this lightly. However, I really think that the U.S. no longer is classifiable as a democracy, but rather as a plutocracy." --hpa
Quoted 4 lines · Pavel Machek
I just imagined hardware which supports freeing memory -- just do not refresh it any more to conserve power ;-))).
Granted, it would probably only make sense in big chunks, like 2MB or so... It might make sense for a PDA...
ISTR reading about one handheld that did something like this (possibly psion) The hardware has the ability to migrate data from one memory bank to another and power down the least used one.
Signature · 2 lines
| Dave Jones. http://www.codemonkey.org.uk | SuSE Labs
Quoted 11 lines · Pavel Machek
Hi!
Quoted 7 lines
Quoted 2 lines
You say 'at once'. Does UML somehow give pages back to the host when they're freed, so the pages that are no longer used by UML can be discarded by the host instead of getting swapped?
Doesn't seem to but it looks like madvise might be enough to make that happen. That BTW is an issue for more than UML - it has a bearing on running lots of Linux instances on any supervisor/virtualising system like S/390
Quoted 7 lines
I just imagined hardware which supports freeing memory -- just do not refresh it any more to conserve power ;-))).
Granted, it would probably only make sense in big chunks, like 2MB or so... It might make sense for a PDA...
Pavel
Unlikely. Also, if you're using ECC, then that really screws with you.
However, if it is an issue for more than UML (I still consider the particular UML case "in case you have a UML on a tmpfs set up by an idiot admin" completely bogus) then it's another issue. The S/390 issue is real.
-hpa
Quoted 3 lines · Jeff Dike
Really? And you're unconcerned about the impact on the rest of the system of a UML grabbing (say) 128M of memory when it starts up? Especially if it may never use it?
Honestly, I think that most people want to know if the system they've setup is overcommited at as early a point as possible: a UML failing at startup with out of memory is better than random segvs at some later point when the system is under load. Refer to the principle of least surprise. And if the user truely wants to disable that, well, you can give them a command line option to shoot themselves in the foot with.
-ben
Signature · 3 lines
"A man with a bass just walked in, and he's putting it down on the floor."
Quoted 11 lines · Benjamin LaHaise
Quoted 3 lines · Jeff Dike
Really? And you're unconcerned about the impact on the rest of the system of a UML grabbing (say) 128M of memory when it starts up? Especially if it may never use it?
Honestly, I think that most people want to know if the system they've setup is overcommited at as early a point as possible: a UML failing at startup with out of memory is better than random segvs at some later point when the system is under load. Refer to the principle of least surprise. And if the user truely wants to disable that, well, you can give them a command line option to shoot themselves in the foot with.
Suppose you have 512 MB memory and an equal amount of swap. You start 8 umls with 64 MB each. With your and Peter's suggestion, the system always goes into swap. Whereas if the memory is only allocated on demand it probably doesn't.
Signature · 1 line
Daniel
Daniel Phillips <[email protected]>:
Quoted 13 lines · Benjamin LaHaise
Quoted 11 lines · Jeff Dike
Quoted 3 lines
Really? And you're unconcerned about the impact on the rest of the system of a UML grabbing (say) 128M of memory when it starts up? Especially if it may never use it?
Honestly, I think that most people want to know if the system they've setup is overcommited at as early a point as possible: a UML failing at startup with out of memory is better than random segvs at some later point when the system is under load. Refer to the principle of least surprise. And if the user truely wants to disable that, well, you can give them a command line option to shoot themselves in the foot with.
Quoted 4 lines
Suppose you have 512 MB memory and an equal amount of swap. You start 8 umls with 64 MB each. With your and Peter's suggestion, the system always goes into swap. Whereas if the memory is only allocated on demand it probably doesn't.
Not unless the VM is really bad... All that is called for is that the virtual space be available. Each umls gets 64 MB, but the rest is guaranteed available via swap. Nothing has to swap until all processes have expanded to use all available ram. Currently the only way to ensure that the memory IS available is to modify every page at startup. Yes it will swap the modified pages.
But it should only do so once, until the pages are really needed.
Otherwise the umls run until the system goes OOM - then somebody gets killed. Much nicer to have it die at the beginning instead of after 4-5 hours of operation when it needs just "one more page" only to find out that the system lied when it said it was available.
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]
Any opinions expressed are solely my own.
Quoted 2 lines · [email protected] said:
Currently the only way to ensure that the memory IS available is to modify every page at startup. Yes it will swap the modified pages.
Currently, yes.
But with Alan says his address space accounting will prevent mmaps from succeeding if populating them would OOM the system, which gives you want you want and which sounds like the right thing. The 8 64M UMLs will run without needing to touch all their pages at bootup and without fear of being killed later. If the 9th UML would be in danger of random death, then it will never get off the ground.
Note that this doesn't help when the UMLs are under a smaller limit than RAM + .5 * swap or whatever as happens when they are mmapping from tmpfs. That's the situation that I'm concerned about.
Jeff
Quoted 3 lines
Note that this doesn't help when the UMLs are under a smaller limit than RAM + .5 * swap or whatever as happens when they are mmapping from tmpfs. That's the situation that I'm concerned about.
Making tmpfs enforce the policy in those modes both checking the global overcommit and also enforcing a "must be able to fill in the pages between start and end of file" for the tmpfs file size itself is not hard from inspection. If its needed I can add that next update to the address accounting.
Quoted 3 lines · [email protected] said:
and also enforcing a "must be able to fill in the pages between start and end of file" for the tmpfs file size itself is not hard from inspection.
So if I mapped a single page from file offset 65M on a 64M tmpfs, that would fail?
I'd prefer maps to fail when they make the total maps exceed the tmpfs limit.
Then I can map in smaller chunks, PAGE_SIZE if necessary. That has the disadvantage that the vmas in the host would be even uglier than they are now because we don't have vma merging any more.
UML would still need that page_alloc hook, except it would map the allocated pages instead of touching them.
Jeff
Quoted 1 line
I'd prefer maps to fail when they make the total maps exceed the tmpfs limit.
That makes more sense and can be done yes. Probably it wants to be a tmpfs option
Quoted 4 lines · Daniel Phillips
Suppose you have 512 MB memory and an equal amount of swap. You start 8 umls with 64 MB each. With your and Peter's suggestion, the system always goes into swap. Whereas if the memory is only allocated on demand it probably doesn't.
As I said previously, going into swap is preferable over randomly killing new tasks under heavy load.
-ben
Signature · 3 lines
"A man with a bass just walked in, and he's putting it down on the floor."
Quoted 8 lines · Benjamin LaHaise
Quoted 4 lines · Daniel Phillips
Suppose you have 512 MB memory and an equal amount of swap. You start 8 umls with 64 MB each. With your and Peter's suggestion, the system always goes into swap. Whereas if the memory is only allocated on demand it probably doesn't.
As I said previously, going into swap is preferable over randomly killing new tasks under heavy load.
Huh? In the example I gave, you will never oom but with your suggestion, you will always go needlessly go into swap. I'm suprised that you and Peter are aguing in favor of wasting resources.
Signature · 1 line
Daniel
Quoted 7 lines · Daniel Phillips
Quoted 6 lines · Benjamin LaHaise
Quoted 4 lines · Daniel Phillips
Suppose you have 512 MB memory and an equal amount of swap. You start 8 umls with 64 MB each. With your and Peter's suggestion, the system always goes into swap. Whereas if the memory is only allocated on demand it probably doesn't.
Quoted 6 lines
Quoted 2 lines
As I said previously, going into swap is preferable over randomly killing new tasks under heavy load.
Huh? In the example I gave, you will never oom but with your suggestion, you will always go needlessly go into swap. I'm suprised that you and Peter are aguing in favor of wasting resources.
I'm arguing in favour of predictable behaviour. Stability and reliability are more important than a bit of swap space.
-ben
Signature · 3 lines
"A man with a bass just walked in, and he's putting it down on the floor."
Quoted 11 lines · Benjamin LaHaise
Quoted 10 lines · Daniel Phillips
Quoted 8 lines · Benjamin LaHaise
Quoted 4 lines · Daniel Phillips
Suppose you have 512 MB memory and an equal amount of swap. You start 8 umls with 64 MB each. With your and Peter's suggestion, the system always goes into swap. Whereas if the memory is only allocated on demand it probably doesn't.
As I said previously, going into swap is preferable over randomly killing new tasks under heavy load.
Quoted 6 lines
Quoted 3 lines
Huh? In the example I gave, you will never oom but with your suggestion, you will always go needlessly go into swap. I'm suprised that you and Peter are aguing in favor of wasting resources.
I'm arguing in favour of predictable behaviour. Stability and reliability are more important than a bit of swap space.
That's the same argument that says memory overcommit should not be allowed.
Signature · 1 line
Daniel
Quoted 12 lines · Daniel Phillips
Quoted 11 lines · Benjamin LaHaise
Quoted 10 lines · Daniel Phillips
Quoted 8 lines · Benjamin LaHaise
Quoted 4 lines · Daniel Phillips
Suppose you have 512 MB memory and an equal amount of swap. You start 8 umls with 64 MB each. With your and Peter's suggestion, the system always goes into swap. Whereas if the memory is only allocated on demand it probably doesn't.
As I said previously, going into swap is preferable over randomly killing new tasks under heavy load.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Huh? In the example I gave, you will never oom but with your suggestion, you will always go needlessly go into swap. I'm suprised that you and Peter are aguing in favor of wasting resources.
I'm arguing in favour of predictable behaviour. Stability and reliability are more important than a bit of swap space.
That's the same argument that says memory overcommit should not be allowed.
Go back in the thread: I suggested making it an option that the user has to turn on to allow his foot to be shot. Remember: the common case in the kernel is to be using all memory.
-ben
Signature · 3 lines
"A man with a bass just walked in, and he's putting it down on the floor."
Quoted 2 lines · [email protected] said:
Go back in the thread: I suggested making it an option that the user has to turn on to allow his foot to be shot.
OK, this seems to be the relevant quote (and you seem to be referring to the kernel build segfaults - correct me if I'm wrong):
Quoted 2 lines · [email protected] said:
a UML failing at startup with out of memory is better than random segvs at some later point when the system is under load.
I showed the kernel build segfaulting as an improvement over UML hanging, which is the alternative behavior.
The segfaults were caused by me implementing the simplest possible response to alloc_pages returning unbacked pages, which is to return NULL to the caller. This is actually wrong because in this failure case, it effectively changes the semantics of GFP_USER, GFP_KERNEL, and the other blocking GFP_* allocations to GFP_ATOMIC. And that's what forced UML to segfault the compilations.
A slightly fancier recovery would loop calling alloc_pages until it got a set of already-backed pages (with some possible sleeping in alloc_pages in there). That would preserve the blocking semantics of GFP_USER, GFP_KERNEL, et al, and would have allowed the UML userspace (the kernel build) to continue working as it should.
So, a slightly improved version of the patch (which I can write up if you're interested in seeing it) would have allowed UML and its userspace to continue running fine (albeit in less memory than it expected) in the presence of an overcommited tmpfs.
Jeff
Quoted 4 lines
caller. This is actually wrong because in this failure case, it effectively changes the semantics of
GFP_USER,GFP_KERNEL, and the other blocking GFP_* allocations toGFP_ATOMIC. And that's what forced UML to segfault the compilations.
GFP_KERNEL will sometimes return NULL.
Quoted 7 lines · Alan Cox
Jeff Dike Apparently wrote
Quoted 4 lines
caller. This is actually wrong because in this failure case, it effectively changes the semantics of
GFP_USER,GFP_KERNEL, and the other blocking GFP_* allocations toGFP_ATOMIC. And that's what forced UML to segfault the compilations.
GFP_KERNELwill sometimes return NULL.
Sad but true. IMHO we are on track to fix that in this kernel cycle, with better locked/dirty accounting and rmap to forcibly unmap pages when necessary.
Signature · 1 line
Daniel
Quoted 11 lines · Daniel Phillips
Quoted 7 lines · Alan Cox
Jeff Dike Apparently wrote
Quoted 4 lines
caller. This is actually wrong because in this failure case, it effectively changes the semantics of
GFP_USER,GFP_KERNEL, and the other blocking GFP_* allocations toGFP_ATOMIC. And that's what forced UML to segfault the compilations.
GFP_KERNELwill sometimes return NULL.Sad but true. IMHO we are on track to fix that in this kernel cycle, with better locked/dirty accounting and rmap to forcibly unmap pages when necessary.
Why is that a fix? And how can it work?
Signature · 4 lines
--------------------------------------------------------- Victor Yodaiken Finite State Machine Labs: The RTLinux Company. www.fsmlabs.com www.rtlinux.com
Quoted 8 lines · On March 7, 2002 03:04 pm, [email protected] wrote:
Quoted 7 lines · Daniel Phillips
Quoted 6 lines · Alan Cox
Jeff Dike Apparently wrote
Quoted 4 lines
caller. This is actually wrong because in this failure case, it effectively changes the semantics of
GFP_USER,GFP_KERNEL, and the other blocking GFP_* allocations toGFP_ATOMIC. And that's what forced UML to segfault the compilations.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
GFP_KERNELwill sometimes return NULL.Sad but true. IMHO we are on track to fix that in this kernel cycle, with better locked/dirty accounting and rmap to forcibly unmap pages when necessary.
Why is that a fix? And how can it work?
Since there is always at least one freeable page in the system (or we're oom) then we just have to find it and we know we can forcibly unmap it. We do need to know the total of pinned pages, I should have said locked/dirty/pinned.
Since GFP_KERNEL includes __GFP_WAIT, we are even allowed to wait for dirty page writeout.
Signature · 1 line
Daniel
Quoted 3 lines
Since there is always at least one freeable page in the system (or we're oom) then we just have to find it and we know we can forcibly unmap it. We do need to know the total of pinned pages, I should have said locked/dirty/pinned.
What if I did a 4 page allocation ?
And if we are OOM - we want to return NULL
Quoted 5 lines · Alan Cox
Quoted 3 lines
Since there is always at least one freeable page in the system (or we're oom) then we just have to find it and we know we can forcibly unmap it. We do need to know the total of pinned pages, I should have said locked/dirty/pinned.
What if I did a 4 page allocation ?
Higher order allocation - imho we can fix that too, eventually, however it's a lot more work. First we have to have reliable physical defragmentation.
Quoted 1 line
And if we are OOM - we want to return NULL
What good does that do?
Signature · 1 line
Daniel
Quoted 6 lines
Higher order allocation - imho we can fix that too, eventually, however it's a lot more work. First we have to have reliable physical defragmentation.
Quoted 1 line
And if we are OOM - we want to return NULL
What good does that do?
It allows us to continue. It avoids the deadlocks. It lets the caller make an intelligent decision.
Quoted 8 lines · Alan Cox
Quoted 6 lines
Higher order allocation - imho we can fix that too, eventually, however it's a lot more work. First we have to have reliable physical defragmentation.
Quoted 1 line
And if we are OOM - we want to return NULL
What good does that do?
It allows us to continue. It avoids the deadlocks.
Could you describe the deadlock, please?
Quoted 1 line
It lets the caller make an intelligent decision.
I maintain it's the wrong interface, we're mixing two concepts together there:
- VM can't find blocks that are freeable, so fails and dumps the problem on the caller, which has to busy wait. This sucks.
- The VM is under heavy load and the caller doesn't really need the memory that badly because it has a fallback, the VM somehow knows this, so fails the allocation and everybody is happy.
These should be separated, and we should fix the former.
Signature · 1 line
Daniel
Quoted 1 line · Alan Cox
And if we are OOM - we want to return NULL
Oh, right, it lets an allocator that didn't 100% need the page use a fallback strategy, but for that we probably want a different interface anyway, such as a GFP flag that says 'fail if this looks hard to get'.
Signature · 1 line
Daniel
Quoted 2 lines · Daniel Phillips
a GFP flag that says 'fail if this looks hard to get'.
Something like that would provide a solution to the readahead thrashing problem.
-
Quoted 6 lines · Andrew Morton
Quoted 2 lines · Daniel Phillips
a GFP flag that says 'fail if this looks hard to get'.
Something like that would provide a solution to the readahead thrashing problem.
Nope. Readahead pages are clean and very easy to evict, so it's still trivial to evict all the pages from another readahead window because everybody's readahead window is too large.
regards,
Rik
Signature · 3 lines
<insert bitkeeper endorsement here> http://www.surriel.com/ http://distro.conectiva.com/
Quoted 4 lines · Rik van Riel
Quoted 2 lines · Andrew Morton
Quoted 1 line · Daniel Phillips
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
a GFP flag that says 'fail if this looks hard to get'.
Something like that would provide a solution to the readahead thrashing problem.
Nope. Readahead pages are clean and very easy to evict, so it's still trivial to evict all the pages from another readahead window because everybody's readahead window is too large.
I was thinking an explicit GFP_READAHEAD and PG_readahead. Where a GFP_READAHEAD allocation would fail if it can't find any non-readahead pages. And it would fail if it had to perform I/O.
That's not nice - it'd result in large LRU walks. But it'd be better than the 10x slowdown which readahead thrashing causes.
Any clever ideas?
-
Quoted 5 lines · Andrew Morton
Quoted 3 lines
Nope. Readahead pages are clean and very easy to evict, so it's still trivial to evict all the pages from another readahead window because everybody's readahead window is too large.
Any clever ideas?
1) keep track of which pages we are reading ahead
... the readahead code already does this
2) at read() or fault time, see if the page
(a) is resident
(b) is in the current readahead window,
ie. already read ahead
3) if the page is in the current readahead window
but NOT resident, the page was read in and
evicted before we got around to using it, so
readahead window thrashing is going on
... in that case, collapse the size of the
readahead window TCP-style
4) slowly growing the readahead window when there is enough memory available, in order to minimise the number of disk seeks
5) the growing in (3) and shrinking in (4) mean that the readahead size of all streaming IO in the system gets automatically balanced against each other and against other memory demand in the system
regards,
Rik
Signature · 3 lines
<insert bitkeeper endorsement here> http://www.surriel.com/ http://distro.conectiva.com/
Quoted 17 lines · Rik van Riel
Quoted 5 lines · Andrew Morton
Quoted 3 lines
Nope. Readahead pages are clean and very easy to evict, so it's still trivial to evict all the pages from another readahead window because everybody's readahead window is too large.
Any clever ideas?
1) keep track of which pages we are reading ahead
... the readahead code already does this2) at
read()or fault time, see if the page
(a) is resident
(b) is in the current readahead window,
ie. already read ahead
Quoted 6 lines
3) if the page is in the current readahead window
but NOT resident, the page was read in and
evicted before we got around to using it, so
readahead window thrashing is going on
... in that case, collapse the size of the
readahead window TCP-style
I have all that. See handle_ra_thrashing() in
http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.6-pre2/dallocbase-10-readahead.patch
Quoted 8 lines
4) slowly growing the readahead window when there is enough memory available, in order to minimise the number of disk seeks
5) the growing in (3) and shrinking in (4) mean that the readahead size of all streaming IO in the system gets automatically balanced against each other and against other memory demand in the system
Doesn't work.
Ah, this is hard to describe.
umm.
a) Suppose that we're getting readahead thrashing. readahead pages are getting dropped. So we keep seeking to each file to get new data, so we do a ton of seeking.
b) Suppose that we nicely detect thrashing and reduce the readahead window. Well, we *still* need to seek to each file to read some blocks.
See? They're equivalent. In case a) we're doing more (pointless) I/O, but the cost of that is vanishingly small because it's just one request.
So what *is* a solution. Well, there's only so much memory available. In either case a) or case b) we're "fairly" distributing that memory between all files. And that's the problem. *All* the files have too small a readahead window. Which points one at: we need to stop being fair. We need to give some files a good readahead window and others
not. The "soft pinning" which I propose with GFP_READAHEAD andPG_readhead might have that effect, I think.
I'll try it, see how it feels.
-
Quoted 23 lines · Andrew Morton
Quoted 4 lines
5) the growing in (3) and shrinking in (4) mean that the readahead size of all streaming IO in the system gets automatically balanced against each other and against other memory demand in the system
Doesn't work.
Ah, this is hard to describe.
umm.
a) Suppose that we're getting readahead thrashing. readahead pages are getting dropped. So we keep seeking to each file to get new data, so we do a ton of seeking.
b) Suppose that we nicely detect thrashing and reduce the readahead window. Well, we *still* need to seek to each file to read some blocks.
See? They're equivalent. In case a) we're doing more (pointless) I/O, but the cost of that is vanishingly small because it's just one request.
Quoted 7 lines
So what *is* a solution. Well, there's only so much memory available. In either case a) or case b) we're "fairly" distributing that memory between all files. And that's the problem. *All* the files have too small a readahead window. Which points one at: we need to stop being fair. We need to give some files a good readahead window and others
not. The "soft pinning" which I propose withGFP_READAHEADandPG_readheadmight have that effect, I think.
Actually, it could boil down to something more:
use-once reduces the VM to FIFO order, which suffers from belady's anomaly so it doesn't matter much how much memory you throw at it
drop-behind will suffer the same problem once the readahead memory is too large to keep in the system, but at least the already-used pages won't kick out readahead pages
regards,
Rik
Signature · 3 lines
<insert bitkeeper endorsement here> http://www.surriel.com/ http://distro.conectiva.com/
Quoted 11 lines · Rik van Riel
Quoted 8 lines
So what *is* a solution. Well, there's only so much memory available. In either case a) or case b) we're "fairly" distributing that memory between all files. And that's the problem. *All* the files have too small a readahead window. Which points one at: we need to stop being fair. We need to give some files a good readahead window and others
not. The "soft pinning" which I propose withGFP_READAHEADandPG_readheadmight have that effect, I think.Actually, it could boil down to something more:
Quoted 7 lines
use-once reduces the VM to FIFO order, which suffers from belady's anomaly so it doesn't matter much how much memory you throw at it
drop-behind will suffer the same problem once the readahead memory is too large to keep in the system, but at least the already-used pages won't kick out readahead pages
err.. Was there a fix in there somewhere, or are we stuck?
-
Quoted 9 lines · Andrew Morton
Quoted 7 lines
use-once reduces the VM to FIFO order, which suffers from belady's anomaly so it doesn't matter much how much memory you throw at it
drop-behind will suffer the same problem once the readahead memory is too large to keep in the system, but at least the already-used pages won't kick out readahead pages
err.. Was there a fix in there somewhere, or are we stuck?
Imagine how TCP backoff would work if it kept old packets around and would drop random packets because of too many old packets in the buffers.
I suspect that the readahead window resizing might work when we throw away the already-used streaming IO pages before we start throwing away any pages we're about to use.
regards,
Rik
Signature · 3 lines
<insert bitkeeper endorsement here> http://www.surriel.com/ http://distro.conectiva.com/
in addition by rducing the amount of readahead you do for each file you can stabilize into a mode where you are doing _some_ readahead and not thrashing so this will reduce your seeks.
David Lang
Quoted 41 lines · Rik van Riel
Date: Thu, 7 Mar 2002 19:27:49 -0300 (BRT)
From: Rik van Riel <[email protected]>
To: Andrew Morton <[email protected]>
Cc: [email protected]
Subject: Re: [RFC] Arch option to touch newly allocated pagesQuoted 9 lines · Andrew Morton
Quoted 7 lines
use-once reduces the VM to FIFO order, which suffers from belady's anomaly so it doesn't matter much how much memory you throw at it
drop-behind will suffer the same problem once the readahead memory is too large to keep in the system, but at least the already-used pages won't kick out readahead pages
err.. Was there a fix in there somewhere, or are we stuck?
Imagine how TCP backoff would work if it kept old packets around and would drop random packets because of too many old packets in the buffers.
I suspect that the readahead window resizing might work when we throw away the already-used streaming IO pages before we start throwing away any pages we're about to use.
regards,
Rik
--
<insert bitkeeper endorsement here>http://www.surriel.com/ http://distro.conectiva.com/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 13 lines · Rik van Riel
Quoted 9 lines · Andrew Morton
Quoted 7 lines
use-once reduces the VM to FIFO order, which suffers from belady's anomaly so it doesn't matter much how much memory you throw at it
drop-behind will suffer the same problem once the readahead memory is too large to keep in the system, but at least the already-used pages won't kick out readahead pages
err.. Was there a fix in there somewhere, or are we stuck?
Quoted 8 lines
Imagine how TCP backoff would work if it kept old packets around and would drop random packets because of too many old packets in the buffers.
I suspect that the readahead window resizing might work when we throw away the already-used streaming IO pages before we start throwing away any pages we're about to use.
ewww.. You seem to be implying that when the readahead code goes to get a new page, it's reclaiming unused readahead pages *in preference to* already-used pages.
That would be awful, wouldn't it?
Perhaps an algorithm would be:
a) Call mark_page_accessed once against readahead pages.
b) If thrashing is detected, call mark_page_accessed twice against readahead pages, to move them onto the active list.
The intent being to say "this page is important. Throw something else away".
Seems this would delay the onset of the problem significantly?
-
Quoted 11 lines · Daniel Phillips
Quoted 10 lines · On March 7, 2002 03:04 pm, [email protected] wrote:
Quoted 9 lines · Daniel Phillips
Quoted 7 lines · Alan Cox
Jeff Dike Apparently wrote
Quoted 4 lines
caller. This is actually wrong because in this failure case, it effectively changes the semantics of
GFP_USER,GFP_KERNEL, and the other blocking GFP_* allocations toGFP_ATOMIC. And that's what forced UML to segfault the compilations.
GFP_KERNELwill sometimes return NULL.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Sad but true. IMHO we are on track to fix that in this kernel cycle, with better locked/dirty accounting and rmap to forcibly unmap pages when necessary.
Why is that a fix? And how can it work?
Since there is always at least one freeable page in the system (or we're oom) then we just have to find it and we know we can forcibly unmap it. We do need to know the total of pinned pages, I should have said locked/dirty/pinned.
What if we are oom?
What if we are on our way to deadlock?
What if the caller of kmalloc will make less good use of the page than the current owner of the page?
page_t *x,*p;for(i = 0; i < SOME_MADE_UP_NUMBER_THAT_SEEMS_GOOD;i++)
if( p = kmalloc(..)){
copyfromuser(x++,p);
dispatch_to_output(p);
}
else {//do the rest later
...
}Quoted 6 lines
Since
GFP_KERNELincludes__GFP_WAIT, we are even allowed to wait for dirty page writeout.Signature · 1 line
Daniel
Signature · 4 lines
--------------------------------------------------------- Victor Yodaiken Finite State Machine Labs: The RTLinux Company. www.fsmlabs.com www.rtlinux.com
Quoted 15 lines · On March 7, 2002 03:38 pm, [email protected] wrote:
Quoted 14 lines · Daniel Phillips
Quoted 13 lines · On March 7, 2002 03:04 pm, [email protected] wrote:
Quoted 11 lines · Daniel Phillips
Quoted 7 lines · Alan Cox
Jeff Dike Apparently wrote
Quoted 4 lines
caller. This is actually wrong because in this failure case, it effectively changes the semantics of
GFP_USER,GFP_KERNEL, and the other blocking GFP_* allocations toGFP_ATOMIC. And that's what forced UML to segfault the compilations.
GFP_KERNELwill sometimes return NULL.Sad but true. IMHO we are on track to fix that in this kernel cycle, with better locked/dirty accounting and rmap to forcibly unmap pages when necessary.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Why is that a fix? And how can it work?
Since there is always at least one freeable page in the system (or we're oom) then we just have to find it and we know we can forcibly unmap it. We do need to know the total of pinned pages, I should have said locked/dirty/pinned.
What if we are oom?
This problem didn't get any worse, we still have to deal with it. We can wait, so we deal with it in the standard way (i.e., we puke, have to do something about that.)
Quoted 1 line
What if we are on our way to deadlock?
huh??
Quoted 2 lines
What if the caller of kmalloc will make less good use of the page than the current owner of the page?
That's life, that's what lrus are for.
Quoted 9 lines
page_t*x,*p;for(i = 0; i < SOME_MADE_UP_NUMBER_THAT_SEEMS_GOOD;i++)if( p = kmalloc(..)){ copyfromuser(x++,p); dispatch_to_output(p); } else {//do the rest later ... }
Please put your thinking cap on and come up with a less borked interface for doing that ;-)
You won't find one if you don't look for it.
Signature · 1 line
Daniel
Quoted 11 lines · Daniel Phillips
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Why is that a fix? And how can it work?
Since there is always at least one freeable page in the system (or we're oom) then we just have to find it and we know we can forcibly unmap it. We do need to know the total of pinned pages, I should have said locked/dirty/pinned.
What if we are oom?
This problem didn't get any worse, we still have to deal with it. We can wait, so we deal with it in the standard way (i.e., we puke, have to do something about that.)
So it can return NULL?
Quoted 4 lines
Quoted 2 lines
What if we are on our way to deadlock?
huh??
Process A needs 4 pages, Process B needs 4 pages, each grabs 3. One easy, traditional unix algorithm for dealing with this is
for(i=0; i < 4; i++)if !(p[i]=kmallloc(...))
free all that we have so farQuoted 4 lines
Quoted 2 lines
What if the caller of kmalloc will make less good use of the page than the current owner of the page?
That's life, that's what lrus are for.
Really? I thought LRUs were to approximate working sets. Obviously if a program is kmallocing its working set is changing but that does not tell us anything about whether it is a correct decision to rip a page from the working set of another process.
Quoted 11 lines
Quoted 10 lines
page_t*x,*p;for(i = 0; i < SOME_MADE_UP_NUMBER_THAT_SEEMS_GOOD;i++)if( p = kmalloc(..)){ copyfromuser(x++,p); dispatch_to_output(p); } else {//do the rest later ... }
Quoted 4 lines
Please put your thinking cap on and come up with a less borked interface for doing that ;-)
You won't find one if you don't look for it.
I'm too dumb to come up with a solution here, but you are the one changing the interface, so surely you have a couple of "less borked" solutions in mind - right?
Signature · 4 lines
--------------------------------------------------------- Victor Yodaiken Finite State Machine Labs: The RTLinux Company. www.fsmlabs.com www.rtlinux.com
Quoted 7 lines · On March 7, 2002 05:50 pm, [email protected] wrote:
Quoted 6 lines · Daniel Phillips
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
Why is that a fix? And how can it work?
Since there is always at least one freeable page in the system (or we're oom) then we just have to find it and we know we can forcibly unmap it. We do need to know the total of pinned pages, I should have said locked/dirty/pinned.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
What if we are oom?
This problem didn't get any worse, we still have to deal with it. We can wait, so we deal with it in the standard way (i.e., we puke, have to do something about that.)
So it can return NULL?
Returning null here won't help if the caller doesn't have a fallback, or if the fallback is unacceptable, such as losing a filesystem transaction.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
What if we are on our way to deadlock?
huh??
Process A needs 4 pages, Process B needs 4 pages, each grabs 3.
This is no new deadlock. Supposing each has successfully grabbed 3, what good does it do if the process is too clueless to release the pages it's already grabbed, because the 4th page alloc fails? (The first 3 may have been alloced in a completely different part of the program.) And if the process does know how to do this, it should tell the VM that *then* the VM should feel free to fail it.
Quoted 3 lines
One easy, traditional unix algorithm for dealing with this is
for(i=0; i < 4; i++)if !(p[i]=kmallloc(...)) free all that we have so far
Just or in GFP_ok_to_fail there.
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
What if the caller of kmalloc will make less good use of the page than the current owner of the page?
That's life, that's what lrus are for.
Really? I thought LRUs were to approximate working sets. Obviously if a program is kmallocing its working set is changing but that does not tell us anything about whether it is a correct decision to rip a page from the working set of another process.
We're getting way far from the original question here. Our lru has no concept of working set, it's completely global. That's not so great and it's another problem to tackle. Sometime.
Quoted 5 lines
Quoted 1 line
You won't find one if you don't look for it.
I'm too dumb to come up with a solution here, but you are the one changing the interface, so surely you have a couple of "less borked" solutions in mind - right?
Yes. Well, I'm not alone here, ping Marcelo on that if you like. This is known borkness that's been deferred while more pressing borkness is dealt with.
Signature · 1 line
Daniel
Quoted 4 lines
Quoted 1 line
So it can return NULL?
Returning null here won't help if the caller doesn't have a fallback, or if the fallback is unacceptable, such as losing a filesystem transaction.
Not having a fallback is unacceptable. Thats the real problem. You can't go around pandering to sloppy coders who can't work a memory allocator
Quoted 3 lines · [email protected] said:
Not having a fallback is unacceptable. Thats the real problem. You
can't go around pandering to sloppy coders who can't work a memory allocator
OTOH there is perhaps some justification for distinguishing between 'If you fail this I'll tell the user -ENOMEM and continue happily on my way' allocations and 'If you fail this I lose track of hardware state and all is fucked till we reboot' ones.
--
dwmw2
Quoted 9 lines
Quoted 3 lines · [email protected] said:
Not having a fallback is unacceptable. Thats the real problem. You
can't go around pandering to sloppy coders who can't work a memory allocatorOTOH there is perhaps some justification for distinguishing between 'If you fail this I'll tell the user
-ENOMEMand continue happily on my way' allocations and 'If you fail this I lose track of hardware state and all is fucked till we reboot' ones.
None at all. If you needed the memory before you committed to an operation you should have reserved it before you started. See "sloppy coders"
Quoted 3 lines · [email protected] said:
None at all. If you needed the memory before you committed to an
operation you should have reserved it before you started. See "sloppy coders"
This is true. I must admit I was having trouble trying to think of a real case where the latter applied in _sane_ code.
--
dwmw2
Quoted 8 lines · Daniel Phillips
Quoted 4 lines
Really? I thought LRUs were to approximate working sets. Obviously if a program is kmallocing its working set is changing but that does not tell us anything about whether it is a correct decision to rip a page from the working set of another process.
We're getting way far from the original question here. Our lru has no concept of working set, it's completely global. That's not so great and it's another problem to tackle. Sometime.
Global lru is an approximation of per-task working set. That's why it works. But it's not perfect.
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
You won't find one if you don't look for it.
I'm too dumb to come up with a solution here, but you are the one changing the interface, so surely you have a couple of "less borked" solutions in mind - right?
Yes. Well, I'm not alone here, ping Marcelo on that if you like. This is known borkness that's been deferred while more pressing borkness is dealt with.
So you and Marcelo are planning on making changes to the semantics of primitive memory allocation modules in the production kernel?
Can that be true? I hope not.
Signature · 4 lines
--------------------------------------------------------- Victor Yodaiken Finite State Machine Labs: The RTLinux Company. www.fsmlabs.com www.rtlinux.com
Quoted 2 lines · Jeff Dike
I showed the kernel build segfaulting as an improvement over UML hanging, which is the alternative behavior.
Versus fully allocating the backing store, which would neither hang nor cause segfaults. This is the behaviour that one expects by default, and should be the first line of defense before going to the overcommit model. Get that aspect of reliability in place, then add the overcommit support. What is better: having uml fail before attempting to boot with an unable to allocate backing store message, or a random oops during early kernel init? As I see it, supporting the safe mode of operation first makes more sense before adding yet another arch hook.
-ben
Signature · 3 lines
"A man with a bass just walked in, and he's putting it down on the floor."
Quoted 5 lines · [email protected] said:
Versus fully allocating the backing store, which would neither hang nor cause segfaults. This is the behaviour that one expects by default, and should be the first line of defense before going to the
overcommit model. Get that aspect of reliability in place, then add
the overcommit support.
OK, the patch below (against UML 2.4.18-2) implements reliable overcommit for UML.
The test was the same as before -
64M tmpfs on /tmp
two 64M UMLs
one -j 2 kernel build running in each
tmpfs was exhausted nearly immediately. Both builds ran to completion. At the end, the 64M tmpfs was divided roughly 30M/35M between the two UMLs.
The first chunk of the patch (mm.h) is the hook that I started this thread talking about. It's a noop for all arches except UML (or s390 if they decide they can use it).
The next two (asm/page.h and mem.c) implement the hook for UML. I believe it correctly preserves the failure semantics of alloc_pages. Please let me know if I missed something.
It tests for unbacked pages by writing to them and catching the resulting SIGBUS. On a host with address space accounting, it would instead map the page and catch the map failures.
The rest of the patch is UML bug fixes which you're only interested in if you want to boot it up.
One bug - if alloc_pages returns a combination of backed and unbacked pages for an order > 0 allocation, the backed pages will effectively be leaked.
TBD -
a corresponding arch hook in free_pages which UML can use for MADV_DONTNEED
some way of poking at unbacked pages to see if they are now backed and can be released back to free_pages
These two items would go some way to allowing multiple UMLs to pass host memory back and forth as needed when it gets scarce.
Jeff
include/linux/mm.h +8 -1
@@ -358,6 +358,13 @@extern struct page *FASTCALL (__alloc_pages (unsigned int gfp_mask,unsigned int order,zonelist_t *zonelist));extern struct page *alloc_pages_node (int nid,unsigned int gfp_mask,unsigned int order); +#ifndef HAVE_ARCH_VALIDATE +static inline struct page *arch_validate (struct page *page,unsigned int gfp_mask,int order) +{ +return (page); +} +#endif +static inline struct page *alloc_pages (unsigned int gfp_mask,unsigned int order) {/*
@@ -365,7 +372,7 @@*/ if (order >=MAX_ORDER )return NULL ; -return _alloc_pages (gfp_mask, order); +return arch_validate (_alloc_pages (gfp_mask, order), gfp_mask, order); }#define alloc_page (gfp_mask) alloc_pages(gfp_mask, 0)
include/asm-um/page.h +3 -0
@@ -42,4 +42,7 @@#define virt_to_page (kaddr) (mem_map + (__pa(kaddr) >> PAGE_SHIFT))#define VALID_PAGE (page) ((page - mem_map) < max_mapnr) +extern struct page *arch_validate (struct page *page,int mask,int order); +#define HAVE_ARCH_VALIDATE +#endif
arch/um/kernel/mem.c +33 -0
@@ -212,6 +212,39 @@" just be swapped out.\n Example: mem=64M\n\n" ); +struct page *arch_validate (struct page *page,int mask,int order) +{ +unsigned long addr, zero =0 ; +int i; + + again: +if (page ==NULL )return (page); + addr = (unsigned long )page_address (page); +for (i =0 ; i < (1 << order); i++){ + current->thread.fault_addr = (void *) addr; +if (__do_copy_to_user ((void *) addr, &zero, +sizeof (zero), + ¤t->thread.fault_addr, + ¤t->thread.fault_catcher)){ +if (!(mask & __GFP_WAIT))return (NULL ); +else break ; + } + addr +=PAGE_SIZE ; + } +if (i == (1 << order))return (page); + page =_alloc_pages (mask, order); + goto again; +} + +extern void relay_signal (int sig,void *sc,int usermode); + +void bus_handler (int sig,void *sc,int usermode) +{ +if (current->thread.fault_catcher !=NULL ) +do_longjmp (current->thread.fault_catcher); +else relay_signal (sig, sc, usermode); +} +/* * Overrides for Emacs so that we follow Linus's tabbing style. * Emacs will notice this stuff at the end of the file and automatically
arch/um/kernel/exec_kern.c +6 -0
@@ -38,6 +38,12 @@int new_pid; stack =alloc_stack (); +if (stack ==0 ){ +printk (KERN_ERR +"flush_thread : failed to allocate temporary stack\n" ); +do_exit (SIGKILL ); + } + new_pid =start_fork_tramp ((void *) current->thread.kernel_stack, stack,0 , exec_tramp);if (new_pid <0 ){
arch/um/kernel/process_kern.c +6 -1
@@ -141,7 +141,7 @@unsigned long page;if ((page =__get_free_page (GFP_KERNEL )) ==0 ) -panic ("Couldn't allocate new stack" ); +return (0 );stack_protections (page);return (page); }
@@ -318,6 +318,11 @@panic ("copy_thread : pipe failed" );if (current->thread.forking){ stack =alloc_stack (); +if (stack ==0 ){ +printk (KERN_ERR "copy_thread : failed to allocate " +"temporary stack\n" ); +return (-ENOMEM ); + } clone_vm = (p->mm == current->mm); p->thread.temp_stack = stack; new_pid = start_fork_tramp((void *) p->thread.kernel_stack,
arch/um/kernel/trap_kern.c +5 -2
@@ -30,6 +30,7 @@struct mm_struct *mm = current->mm;struct vm_area_struct *vma;struct siginfo si; +void *catcher;pgd_t *pgd;pmd_t *pmd;pte_t *pte;
@@ -40,6 +41,7 @@return (0 ); }if (mm ==NULL )panic ("Segfault with no mm" ); + catcher = current->thread.fault_catcher; si.si_code =SEGV_MAPERR ;down_read (&mm->mmap_sem); vma =find_vma (mm, address);
@@ -84,10 +86,10 @@up_read (&mm->mmap_sem);return (0 ); bad: -if (current->thread.fault_catcher !=NULL ) { +if (catcher !=NULL ) { current->thread.fault_addr = (void *) address;up_read (&mm->mmap_sem); -do_longjmp (current->thread.fault_catcher); +do_longjmp (catcher); }else if (current->thread.fault_addr !=NULL ){panic ("fault_addr set but no fault catcher" );
@@ -120,6 +122,7 @@void relay_signal (int sig,void *sc,int usermode) { +if (!usermode)panic ("Kernel mode signal %d" , sig);force_sig (sig, current); }
arch/um/kernel/trap_user.c +3 -1
@@ -420,11 +420,13 @@extern int timer_ready, timer_on; +extern void bus_handler (int sig,void *sc,int usermode); +static void (*handlers[])(int ,void *,int ) = { [ SIGTRAP ] relay_signal, [ SIGFPE ] relay_signal, [ SIGILL ] relay_signal, - [ SIGBUS ] relay_signal, + [ SIGBUS ] bus_handler, [ SIGSEGV] segv_handler, [ SIGIO ] sigio_handler, [ SIGVTALRM ] timer_handler,
Quoted 2 lines · Jeff Dike
OK, the patch below (against UML 2.4.18-2) implements reliable overcommit for UML.
Well, I still dislike it, but I guess it'll have to do. The only nits I see about the patch are: could you make the inline function a #define for the no-arch_validate case? Also, the format of if statements is a bit abnormal:
please add line breaks as appropriate. Aside from that, go ahead.
-ben
Quoted 17 lines · Benjamin LaHaise
Quoted 16 lines · Daniel Phillips
Quoted 15 lines · Benjamin LaHaise
Quoted 13 lines · Daniel Phillips
Quoted 8 lines · Benjamin LaHaise
Quoted 4 lines · Daniel Phillips
Suppose you have 512 MB memory and an equal amount of swap. You start 8 umls with 64 MB each. With your and Peter's suggestion, the system always goes into swap. Whereas if the memory is only allocated on demand it probably doesn't.
As I said previously, going into swap is preferable over randomly killing new tasks under heavy load.
Huh? In the example I gave, you will never oom but with your suggestion, you will always go needlessly go into swap. I'm suprised that you and Peter are aguing in favor of wasting resources.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
I'm arguing in favour of predictable behaviour. Stability and reliability are more important than a bit of swap space.
That's the same argument that says memory overcommit should not be allowed.
Go back in the thread: I suggested making it an option that the user has to turn on to allow his foot to be shot. Remember: the common case in the kernel is to be using all memory.
OK, now suppose the user has turned on that option (I think it should be on by default, like memory overcommit). How is Jeff going to support it? That's his whole point as I understand it.
Instead of providing constructive suggestions on how to solve the problem so that memory overcommit works properly in this case, I see people telling Jeff there is no problem. I think Jeff has a little more of a clue than that.
Signature · 1 line
Daniel
Quoted 5 lines · Daniel Phillips
Instead of providing constructive suggestions on how to solve the problem so that memory overcommit works properly in this case, I see people telling Jeff there is no problem. I think Jeff has a little more of a clue than that.
Jeff has clue, but you, Daniel, quite frankly could take a cue. You nseem to be jumping into arguments just for the sake of them, but without ever contribute anything useful.
Please do us all a favour and shut up for once.
-hpa
Quoted 9 lines · H. Peter Anvin
Quoted 3 lines · Daniel Phillips
Instead of providing constructive suggestions on how to solve the problem so that memory overcommit works properly in this case, I see people telling Jeff there is no problem. I think Jeff has a little more of a clue than that.
Jeff has clue, but you, Daniel, quite frankly could take a cue. You nseem to be jumping into arguments just for the sake of them, but without ever contribute anything useful.
The useful contribution is to stop you and Ben from beating up on Jeff. Thankyou, I think I've accomplished that purpose. Feel free to attack me for that if you feel the need.
(objectionable comment removed)
Signature · 1 line
Daniel
Quoted 4 lines · [email protected] said:
you only need to do the memsets once at startup of UML where the ram is allocated -> a uml booted with 64MB of ram would write into every page of the backing store file before even running the kernel. Doesn't that accomplish the same thing?
The other reason I don't like this is that, at some point, I'd like to start thinking about userspace cooperating with the kernel on memory management. UML looks like a perfect place to start since it's essentially identical to the host making it easier for the two to bargain over memory.
Having UML react sanely to unbacked pages is a step in that direction, having UML preemptively grab all the memory it could ever use isn't.
Jeff
Followup to: <[email protected]> By author: Jeff Dike <[email protected]> In newsgroup: linux.dev.kernel
Quoted 9 lines
The other reason I don't like this is that, at some point, I'd like to start thinking about userspace cooperating with the kernel on memory management. UML looks like a perfect place to start since it's essentially identical to the host making it easier for the two to bargain over memory.
Having UML react sanely to unbacked pages is a step in that direction, having UML preemptively grab all the memory it could ever use isn't.
Until you can come up with a sane application for it, this is just featuritis.
-hpa
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
Quoted 2 lines · [email protected] said:
Until you can come up with a sane application for it, this is just featuritis.
Having the system better manage its memory is "featuritis"?
Jeff
Quoted 8 lines · Jeff Dike
Quoted 3 lines · [email protected] said:
Until you can come up with a sane application for it, this is just featuritis.
Having the system better manage its memory is "featuritis"?
s/better/insanely/
Your proposed application is, quite frankly, bullshit.
-hpa
Quoted 10 lines · Jeff Dike
Quoted 4 lines · [email protected] said:
you only need to do the memsets once at startup of UML where the ram is allocated -> a uml booted with 64MB of ram would write into every page of the backing store file before even running the kernel. Doesn't that accomplish the same thing?
The other reason I don't like this is that, at some point, I'd like to start thinking about userspace cooperating with the kernel on memory management. UML looks like a perfect place to start since it's essentially identical to the host making it easier for the two to bargain over memory.
I could use the same thing in Coda, we have large private memory mappings that are backed by a file which isn't always up-to-date. But we can make it so by applying the logged modifications. If there is some 'memory pressure' signal we could apply the log and remap the memory to reduce swap usage.
On the other hand, applying the logged modifications generates a lot of write activity which could push the system over the edge, so the current method of having a large amount of swap available is probably more reliable. Otherwise we'll get the whole OOM killer debate again (the pre-OOM signaller?).
Jan