out_of_memory called to often
Hi,
We are running an embedded linux (2.4.19) with no swap sapce. And we haven't applied any rmap yet.
Currely when the system runs low (reached the pages_low water-mark), kswapd kicks in. But soon after kswapd kicks in, the out_of_memory is called due to swapd not being able to reclaim enough pages from the page cache from each run of it. I don't understand why sometimes swapd can't reclaim the SWAP_CLUSTER_MAX number of pages.
We know there are still some number of cache pages that can be reclaimed if we run the following c program:
int main() {
int i, j;
char *tmp[10000]
for (i=0; i<NUMBER_OF_PAGES_LEFT_BEFORE_pages_low;
i++) tmp[i]=malloc(4k);
for (i=0; i<NUMBER_OF_PAGES_LEFT_BEFORE_pages_low;
i++) free(tmp[i]);
}
To solve this problem, I am trying to make a conditional call to the out_of_memory() -- only if the pages_min is reached, then the out_of_memory() can be called.
If we can solve the swapd livelock problem, is this a feasible solution? If you have a better idea, can you please share with us?
Any input will be greatly appreciated.
Thanks very much in advance! Please include my email address in your reply.
-Jason
__________________________________________________
Do you Yahoo!?
Yahoo! Web Hosting - establish your business online
http://webhosting.yahoo.com
Quoted 5 lines · Jason Li
Hi,
We are running an embedded linux (2.4.19) with no swap sapce. And we haven't applied any rmap yet.
I believe you may be seeing the same issue I brought up in a previous email. Have you tried what I suggested? I've appended the email to this message with out the test code.
Ross
I've verified this in 2.4.21-pre5 by code inspection and can trigger the problem on 2.4.18. It appears to have been fixed in 2.5.
The folowing code vmscan.c assumes that there is available swap space.
/*
* this is the non-racy check for busy page.
*/
if (!page->mapping || !is_page_cache_freeable(page)) {
spin_unlock(&pagecache_lock);
UnlockPage(page);
page_mapped:
if (--max_mapped >= 0)
continue;
/*
* Alert! We've found too many mapped pages on the
* inactive list, so we start swapping out now!
*/
spin_unlock(&pagemap_lru_lock);
swap_out(priority, gfp_mask, classzone);
return nr_pages;
}If there is no swap space, then unfreeable pages are left on the inactive queue and the vmtree is walked rather than going through the rest of the inactive queue. I believe something like
/*
* this is the non-racy check for busy page.
*/
if (!page->mapping || !is_page_cache_freeable(page)) {
spin_unlock(&pagecache_lock);
UnlockPage(page);
page_mapped:
/* If we don't have any swap space left, there
is no reason to worry about pages that do
not have swap associated with them, there
is nothing we can do about it. */
if (!page->mapping && !swap_avail()) {
/* Let's make the page active since we
cannot swap it out. It get's it off
the inactive list. */
spin_unlock(&pagemap_lru_lock);
activate_page(page);
ClearPageReferenced(page);
spin_lock(&pagemap_lru_lock);
continue;
}
if (--max_mapped >= 0)
continue;
/*
* Alert! We've found too many mapped pages on the
* inactive list, so we start swapping out now!
*/
spin_unlock(&pagemap_lru_lock);
swap_out(priority, gfp_mask, classzone);
return nr_pages;
}will work better when there is no swap space available. If this change is made, it may also be necessary to limit refill_inactive to prevent it from using too much cpu. This bug can be triggered with the attached code and the correct parameters. In particular on a 3 gigabyte machine with no swap,
for i in $(seq 0 9); do dd if=/dev/zero of=file$i bs=1024k count=512; done killmm 1032735283 2 9
Usually causes an out of memory error when there is hundreds of megabytes of cache.
Ross