OOM related fixes

Hello everyone,

this is a set of fixes done in the context of a quite evil workload reading from nfs large files with big read buffers in parallel from many tasks at the same time until the system goes oom. Mostly all of these fixes seems to be required to fix the customer workload on top of an older sles kernel. The forward port of the fixes has been already tested successfully on similar evil workloads.

mainline vanilla running a somewhat simulated workload:

Jun  8 06:06:56 kvm kernel: Out of memory: Killed process 3282 (klauncher).
Jun  8 06:17:35 kvm kernel: Out of memory: kill process 3002 (qmgr) score 11225 or a child
Jun  8 06:17:35 kvm kernel: Out of memory: kill process 3001 (pickup) score 11216 or a child
Jun  8 06:17:35 kvm kernel: Out of memory: kill process 2186 (hald) score 11004 or a child
Jun  8 06:17:35 kvm kernel: Out of memory: kill process 3515 (bash) score 9447 or a child
Jun  8 06:17:35 kvm kernel: Out of memory: kill process 2186 (hald) score 8558 or a child
Jun  8 06:17:35 kvm kernel: Out of memory: kill process 2142 (dbus-daemon) score 5591 or a child
Jun  8 06:17:35 kvm kernel: Out of memory: kill process 3549 (recursive_readd) score 4597 or a child
Jun  8 06:17:43 kvm kernel: Out of memory: kill process 3591 (pickup) score 9756 or a child
Jun  8 06:17:43 kvm kernel: Out of memory: kill process 2204 (hald-addon-acpi) score 4121 or a child
Jun  8 06:17:43 kvm kernel: Out of memory: kill process 3515 (bash) score 3808 or a child
Jun  8 06:17:45 kvm kernel: Out of memory: kill process 3555 (recursive_readd) score 2330 or a child
Jun  8 06:17:53 kvm kernel: Out of memory: kill process 3554 (recursive_readd) score 2605 or a child
Jun  8 06:18:00 kvm kernel: Out of memory: kill process 3170 (nscd) score 1985 or a child
Jun  8 06:18:00 kvm kernel: Out of memory: kill process 3187 (nscd) score 1985 or a child
Jun  8 06:18:00 kvm kernel: Out of memory: kill process 3188 (nscd) score 1985 or a child
Jun  8 06:18:00 kvm kernel: Out of memory: kill process 2855 (portmap) score 1965 or a child
Jun  8 06:18:00 kvm kernel: Out of memory: kill process 3551 (recursive_readd) score 859 or a child

[ eventually it deadlocks and stops killing new tasks ]

mainline + fixes running the same simulated workload:

Jun  8 13:35:32 kvm kernel: Out of memory: kill process 3494 (recursive_readd) score 3822 or a child
Jun  8 13:35:33 kvm kernel: Out of memory: kill process 3494 (recursive_readd) score 3822 or a child
Jun  8 13:35:33 kvm kernel: Out of memory: kill process 3494 (recursive_readd) score 3822 or a child
Jun  8 13:37:33 kvm kernel: Out of memory: kill process 3505 (recursive_readd) score 622 or a child
Jun  8 13:37:34 kvm kernel: Out of memory: kill process 3510 (recursive_readd) score 418 or a child
Jun  8 13:37:36 kvm kernel: Out of memory: kill process 3535 (recursive_readd) score 377 or a child
Jun  8 13:37:36 kvm kernel: Out of memory: kill process 3498 (recursive_readd) score 370 or a child
Jun  8 13:37:36 kvm kernel: Out of memory: kill process 3516 (recursive_readd) score 364 or a child
Jun  8 13:37:36 kvm kernel: Out of memory: kill process 3515 (recursive_readd) score 357 or a child
Jun  8 13:40:49 kvm kernel: Out of memory: kill process 3537 (recursive_readd) score 2391 or a child
Jun  8 13:40:50 kvm kernel: Out of memory: kill process 3537 (recursive_readd) score 2391 or a child
Jun  8 13:40:50 kvm kernel: Out of memory: kill process 3537 (recursive_readd) score 2391 or a child
Jun  8 13:40:50 kvm kernel: Out of memory: kill process 3537 (recursive_readd) score 2391 or a child
Jun  8 13:40:50 kvm kernel: Out of memory: kill process 3537 (recursive_readd) score 2391 or a child
Jun  8 13:40:50 kvm kernel: Out of memory: kill process 3537 (recursive_readd) score 2391 or a child
Jun  8 13:40:50 kvm kernel: Out of memory: kill process 3537 (recursive_readd) score 2391 or a child
Jun  8 13:40:50 kvm kernel: Out of memory: kill process 3537 (recursive_readd) score 2391 or a child
Jun  8 13:40:50 kvm kernel: Out of memory: kill process 3537 (recursive_readd) score 2391 or a child
Jun  8 13:40:50 kvm kernel: Out of memory: kill process 3537 (recursive_readd) score 2391 or a child
Jun  8 13:40:51 kvm kernel: Out of memory: kill process 3537 (recursive_readd) score 2391 or a child
Jun  8 13:40:51 kvm kernel: Out of memory: kill process 3537 (recursive_readd) score 2391 or a child
Jun  8 13:40:51 kvm kernel: Out of memory: kill process 3537 (recursive_readd) score 2391 or a child
Jun  8 13:40:51 kvm kernel: Out of memory: kill process 3537 (recursive_readd) score 2391 or a child
Jun  8 13:41:55 kvm kernel: Out of memory: kill process 3558 (recursive_readd) score 356 or a child
Jun  8 13:41:56 kvm kernel: Out of memory: kill process 3578 (recursive_readd) score 355 or a child
Jun  8 13:41:56 kvm kernel: Out of memory: kill process 3577 (recursive_readd) score 350 or a child
Jun  8 13:41:56 kvm kernel: Out of memory: kill process 3572 (recursive_readd) score 347 or a child
Jun  8 13:41:56 kvm kernel: Out of memory: kill process 3568 (recursive_readd) score 346 or a child

The oom deadlock detection triggers a couple of times against the PG_locked deadlock:

Jun  8 13:51:19 kvm kernel: Killed process 3504 (recursive_readd)
Jun  8 13:51:19 kvm kernel: detected probable OOM deadlock, so killing another task
Jun  8 13:51:19 kvm kernel: Out of memory: kill process 3532 (recursive_readd) score 1225 or a child

Example of stack trace of TIF_MEMDIE killed task (not literally verified that this was the one with TIF_MEMDIE set but it's the same as before with the verified one):

recursive_rea D ffff810001056418 0 3548 3544 (NOTLB)
ffff81000e57dba8 0000000000000082 ffff8100010af5e8 ffff8100148df730 ffff81001ff3ea10 0000000000bd2e1b ffff8100148df908 0000000000000046 ffff81001fd5f170 ffffffff8031c36d ffff81001fd5f170 ffff810001056418

Call Trace:
 [<ffffffff8031c36d>] __generic_unplug_device+0x13/0x24
 [<ffffffff80244163>] sync_page+0x0/0x40
 [<ffffffff804cdf5b>] io_schedule+0xf/0x17
 [<ffffffff8024419e>] sync_page+0x3b/0x40
 [<ffffffff804ce162>] __wait_on_bit_lock+0x36/0x65
 [<ffffffff80244150>] __lock_page+0x5e/0x64
 [<ffffffff802321f1>] wake_bit_function+0x0/0x23
 [<ffffffff802440c0>] find_get_page+0xe/0x40
 [<ffffffff80244a33>] do_generic_mapping_read+0x200/0x450
 [<ffffffff80243f26>] file_read_actor+0x0/0x11d
 [<ffffffff80247fd4>] get_page_from_freelist+0x2d3/0x36e
 [<ffffffff802464d0>] generic_file_aio_read+0x11d/0x159
 [<ffffffff80260bdc>] do_sync_read+0xc9/0x10c
 [<ffffffff80252adb>] vma_merge+0x10c/0x195
 [<ffffffff802321c3>] autoremove_wake_function+0x0/0x2e
 [<ffffffff80253a06>] do_mmap_pgoff+0x5e1/0x74c
 [<ffffffff8026134d>] vfs_read+0xaa/0x132                                                                                         
 [<ffffffff80261662>] sys_read+0x45/0x6e
 [<ffffffff8020991e>] system_call+0x7e/0x83

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

# HG changeset patch
# User Andrea Arcangeli <[email protected]>
# Date 1181332959 -7200
# Node ID 8e38f7656968417dfee09fbb6450a8f1e70f8b21 # Parent 8b84ac74c8464bb6e4a2c08ff2a656d06c8667ca remove nr_scan_inactive/active

The older atomic_add/atomic_set were pointless (atomic_set vs atomic_add would race), but removing them didn't actually remove the race, the race is still there, for the same reasons atomic_add/set couldn't prevent it. This is really the kind of code that I dislike because it's sort of buggy, and it shouldn't be making any measurable difference and when it does something for real it can only hurt!

The real focus is on shrink_zone (ignore the other places where it's being used that are even less interesting). Assume two tasks adds to nr_scan_*active at the same time (first line of the old buggy code), they'll effectively double their scan rate, for no good reason. What can happen is that instead of scanning nr_entries each, they'll scan nr_entries*2 each. The more CPUs the bigger the race and the higher the multiplication effect and the harder it will be to detect oom. In the case that nr_*active < sc->swap_cluster_max, regardless of whatever future invocation of alloc_pages, we'll be going down in the priorities in the current alloc_pages invocation if the DEF_PRIORITY was too high to make any work, so again accumulating the nr_scan_*active doesn't seem interesting even when it's smaller than sc->swap_cluster_max. Each task should work for itself without much care of what the others are doing.

@@ -220,8 +220,6 @@ struct zone {
 	spinlock_t		lru_lock;	
 	struct list_head	active_list;
 	struct list_head	inactive_list;
-	unsigned long		nr_scan_active;
-	unsigned long		nr_scan_inactive;
 	unsigned long		pages_scanned;	   /* since last reclaim */
 	int			all_unreclaimable; /* All pages pinned */
 

@@ -2649,8 +2649,6 @@ static void __meminit free_area_init_cor
 		zone_pcp_init(zone);
 		INIT_LIST_HEAD(&zone->active_list);
 		INIT_LIST_HEAD(&zone->inactive_list);
-		zone->nr_scan_active = 0;
-		zone->nr_scan_inactive = 0;
 		zap_zone_vm_stats(zone);
 		atomic_set(&zone->reclaim_in_progress, 0);
 		if (!size)

@@ -915,20 +915,11 @@ static unsigned long shrink_zone(int pri
 	 * Add one to `nr_to_scan' just to make sure that the kernel will
 	 * slowly sift through the active list.
 	 */
-	zone->nr_scan_active +=
-		(zone_page_state(zone, NR_ACTIVE) >> priority) + 1;
-	nr_active = zone->nr_scan_active;
-	if (nr_active >= sc->swap_cluster_max)
-		zone->nr_scan_active = 0;
-	else
+	nr_active = zone_page_state(zone, NR_ACTIVE) >> priority;
+	if (nr_active < sc->swap_cluster_max)
 		nr_active = 0;
-
-	zone->nr_scan_inactive +=
-		(zone_page_state(zone, NR_INACTIVE) >> priority) + 1;
-	nr_inactive = zone->nr_scan_inactive;
-	if (nr_inactive >= sc->swap_cluster_max)
-		zone->nr_scan_inactive = 0;
-	else
+	nr_inactive = zone_page_state(zone, NR_INACTIVE) >> priority;
+	if (nr_inactive < sc->swap_cluster_max)
 		nr_inactive = 0;
 
 	while (nr_active || nr_inactive) {

@@ -1392,22 +1383,14 @@ static unsigned long shrink_all_zones(un
 
 		/* For pass = 0 we don't shrink the active list */
 		if (pass > 0) {
-			zone->nr_scan_active +=
-				(zone_page_state(zone, NR_ACTIVE) >> prio) + 1;
-			if (zone->nr_scan_active >= nr_pages || pass > 3) {
-				zone->nr_scan_active = 0;
-				nr_to_scan = min(nr_pages,
-					zone_page_state(zone, NR_ACTIVE));
+			nr_to_scan = (zone_page_state(zone, NR_ACTIVE) >> prio) + 1;
+			if (nr_to_scan >= nr_pages || pass > 3) {
 				shrink_active_list(nr_to_scan, zone, sc, prio);
 			}
 		}
 
-		zone->nr_scan_inactive +=
-			(zone_page_state(zone, NR_INACTIVE) >> prio) + 1;
-		if (zone->nr_scan_inactive >= nr_pages || pass > 3) {
-			zone->nr_scan_inactive = 0;
-			nr_to_scan = min(nr_pages,
-				zone_page_state(zone, NR_INACTIVE));
+		nr_to_scan = (zone_page_state(zone, NR_INACTIVE) >> prio) + 1;
+		if (nr_to_scan >= nr_pages || pass > 3) {
 			ret += shrink_inactive_list(nr_to_scan, zone, sc);
 			if (ret >= nr_pages)
 				return ret;

@@ -554,7 +554,7 @@ static int zoneinfo_show(struct seq_file
 			   "\n        min      %lu"
 			   "\n        low      %lu"
 			   "\n        high     %lu"
-			   "\n        scanned  %lu (a: %lu i: %lu)"
+			   "\n        scanned  %lu"
 			   "\n        spanned  %lu"
 			   "\n        present  %lu",
 			   zone_page_state(zone, NR_FREE_PAGES),

@@ -562,7 +562,6 @@ static int zoneinfo_show(struct seq_file
 			   zone->pages_low,
 			   zone->pages_high,
 			   zone->pages_scanned,
-			   zone->nr_scan_active, zone->nr_scan_inactive,
 			   zone->spanned_pages,
 			   zone->present_pages);
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

# HG changeset patch
# User Andrea Arcangeli <[email protected]>
# Date 1181332960 -7200
# Node ID d64cb81222748354bf5b16258197217465f35aeb # Parent 8e38f7656968417dfee09fbb6450a8f1e70f8b21 avoid oom deadlock in nfs_create_request

When sigkill is pending after the oom killer set TIF_MEMDIE, the task must go away or the VM will malfunction.

@@ -61,16 +61,20 @@ nfs_create_request(struct nfs_open_conte
 	struct nfs_server *server = NFS_SERVER(inode);
 	struct nfs_page		*req;
 
-	for (;;) {
-		/* try to allocate the request struct */
-		req = nfs_page_alloc();
-		if (req != NULL)
-			break;
-
-		if (signalled() && (server->flags & NFS_MOUNT_INTR))
-			return ERR_PTR(-ERESTARTSYS);
-		yield();
-	}
+	/* try to allocate the request struct */
+	req = nfs_page_alloc();
+	if (unlikely(!req)) {
+		/*
+		 * -ENOMEM will be returned only when TIF_MEMDIE is set
+		 * so userland shouldn't risk to get confused by a new
+		 * unhandled ENOMEM errno.
+		 */
+		WARN_ON(!test_thread_flag(TIF_MEMDIE));
+		return ERR_PTR(-ENOMEM);
+	}
+
+	if (signalled() && (server->flags & NFS_MOUNT_INTR))
+		return ERR_PTR(-ERESTARTSYS);
 
 	/* Initialize the request struct. Initially, we assume a
 	 * long write-back delay. This will be adjusted in

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

# HG changeset patch
# User Andrea Arcangeli <[email protected]>
# Date 1181332960 -7200
# Node ID 532a5f712848ee75d827bfe233b9364a709e1fc1 # Parent d64cb81222748354bf5b16258197217465f35aeb prevent oom deadlocks during read/write operations

We need to react to SIGKILL during read/write with huge buffers or it becomes too easy to prevent a SIGKILLED task to run do_exit promptly after it has been selected for oom-killage.

@@ -894,6 +894,13 @@ void do_generic_mapping_read(struct addr
 		struct page *page;
 		unsigned long nr, ret;
 
+		if (unlikely(sigismember(&current->pending.signal, SIGKILL)))
+			/*
+			 * Must not hang almost forever in D state in presence of sigkill
+			 * and lots of ram/swap (think during OOM).
+			 */
+			break;
+
 		/* nr is the maximum number of bytes to copy from this page */
 		nr = PAGE_CACHE_SIZE;
 		if (index >= end_index) {

@@ -2105,6 +2112,13 @@ generic_file_buffered_write(struct kiocb
 		unsigned long index;
 		unsigned long offset;
 		size_t copied;
+
+		if (unlikely(sigismember(&current->pending.signal, SIGKILL)))
+			/*
+			 * Must not hang almost forever in D state in presence of sigkill
+			 * and lots of ram/swap (think during OOM).
+			 */
+			break;
 
 		offset = (pos & (PAGE_CACHE_SIZE -1)); /* Within page */
 		index = pos >> PAGE_CACHE_SHIFT;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

# HG changeset patch
# User Andrea Arcangeli <[email protected]>
# Date 1181332960 -7200
# Node ID baa866fedc79cb333b90004da2730715c145f1d5 # Parent 532a5f712848ee75d827bfe233b9364a709e1fc1 serialize oom killer

It's risky and useless to run two oom killers in parallel, let serialize it to reduce the probability of spurious oom-killage.

@@ -400,12 +400,15 @@ void out_of_memory(struct zonelist *zone
 	unsigned long points = 0;
 	unsigned long freed = 0;
 	int constraint;
+	static DECLARE_MUTEX(OOM_lock);
 
 	blocking_notifier_call_chain(&oom_notify_list, 0, &freed);
 	if (freed > 0)
 		/* Got some memory back in the last second. */
 		return;
 
+	if (down_trylock(&OOM_lock))
+		return;
 	if (printk_ratelimit()) {
 		printk(KERN_WARNING "%s invoked oom-killer: "
 			"gfp_mask=0x%x, order=%d, oomkilladj=%d\n",

@@ -472,4 +475,6 @@ out:
 	 */
 	if (!test_thread_flag(TIF_MEMDIE))
 		schedule_timeout_uninterruptible(1);
-}
+
+	up(&OOM_lock);
+}

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

# HG changeset patch
# User Andrea Arcangeli <[email protected]>
# Date 1181332960 -7200
# Node ID 2ebc46595ead0f1790c6ec1d0302dd60ffbb1978 # Parent baa866fedc79cb333b90004da2730715c145f1d5 avoid selecting already killed tasks

If the killed task doesn't go away because it's waiting on some other task who needs to allocate memory, to release the i_sem or some other lock, we must fallback to killing some other task in order to kill the original selected and already oomkilled task, but the logic that kills the childs first, would deadlock, if the already oom-killed task was actually the first child of the newly oom-killed task.

@@ -366,6 +366,12 @@ static int oom_kill_process(struct task_
 		c = list_entry(tsk, struct task_struct, sibling);
 		if (c->mm == p->mm)
 			continue;
+		/*
+		 * We cannot select tasks with TIF_MEMDIE already set
+		 * or we'll hard deadlock.
+		 */
+		if (unlikely(test_tsk_thread_flag(c, TIF_MEMDIE)))
+			continue;
 		if (!oom_kill_task(c))
 			return 0;
 	}

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

# HG changeset patch
# User Andrea Arcangeli <[email protected]>
# Date 1181332961 -7200
# Node ID fe82f6d082c859c641664990c6e14de8d16dcb5d # Parent 2ebc46595ead0f1790c6ec1d0302dd60ffbb1978 reduce the probability of an OOM livelock

There's no need to loop way too many times over the lrus in order to declare defeat and decide to kill a task. The more loops we do the more likely there we'll run in a livelock with a page bouncing back and forth between tasks. The maximum number of entries to check in a loop that returns less than swap-cluster-max pages freed, should be the size of the list (or at most twice the size of the list if you want to be really paranoid about the PG_referenced bit).

Our objective there is to know reliably when it's time that we kill a task, tring to free a few more pages at that already ciritical point is worthless.

This seems to have the effect of reducing the "hang" time during oom killing.

@@ -1007,7 +1007,7 @@ unsigned long try_to_free_pages(struct z
 	int priority;
 	int ret = 0;
 	unsigned long total_scanned = 0;
-	unsigned long nr_reclaimed = 0;
+	unsigned long nr_reclaimed;
 	struct reclaim_state *reclaim_state = current->reclaim_state;
 	unsigned long lru_pages = 0;
 	int i;

@@ -1035,12 +1035,12 @@ unsigned long try_to_free_pages(struct z
 		sc.nr_scanned = 0;
 		if (!priority)
 			disable_swap_token();
-		nr_reclaimed += shrink_zones(priority, zones, &sc);
+		nr_reclaimed = shrink_zones(priority, zones, &sc);
+		if (reclaim_state)
+			reclaim_state->reclaimed_slab = 0;
 		shrink_slab(sc.nr_scanned, gfp_mask, lru_pages);
-		if (reclaim_state) {
+		if (reclaim_state)
 			nr_reclaimed += reclaim_state->reclaimed_slab;
-			reclaim_state->reclaimed_slab = 0;
-		}
 		total_scanned += sc.nr_scanned;
 		if (nr_reclaimed >= sc.swap_cluster_max) {
 			ret = 1;

@@ -1131,7 +1131,6 @@ static unsigned long balance_pgdat(pg_da
 
 loop_again:
 	total_scanned = 0;
-	nr_reclaimed = 0;
 	sc.may_writepage = !laptop_mode;
 	count_vm_event(PAGEOUTRUN);
 

@@ -1186,6 +1185,7 @@ loop_again:
 		 * pages behind kswapd's direction of progress, which would
 		 * cause too much scanning of the lower zones.
 		 */
+		nr_reclaimed = 0;
 		for (i = 0; i <= end_zone; i++) {
 			struct zone *zone = pgdat->node_zones + i;
 			int nr_slab;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

# HG changeset patch
# User Andrea Arcangeli <[email protected]>
# Date 1181332961 -7200
# Node ID aafcc5c9057f11d88c43b823c241f14a5ebdd638 # Parent fe82f6d082c859c641664990c6e14de8d16dcb5d balance_pgdat doesn't return the number of pages freed

nr_reclaimed would be the number of pages freed in the last pass.

@@ -1092,8 +1092,6 @@ out:
  * For kswapd, balance_pgdat() will work across all this node's zones until
  * they are all at pages_high.
  *
- * Returns the number of pages which were actually freed.
- *
  * There is special handling here for zones which are full of pinned pages.
  * This can happen if the pages are all mlocked, or if they are all used by
  * device drivers (say, ZONE_DMA).  Or if they are all in use by hugetlb.

@@ -1109,7 +1107,7 @@ out:
  * the page allocator fallback scheme to ensure that aging of pages is balanced
  * across the zones.
  */
-static unsigned long balance_pgdat(pg_data_t *pgdat, int order)
+static void balance_pgdat(pg_data_t *pgdat, int order)
 {
 	int all_zones_ok;
 	int priority;

@@ -1259,8 +1257,6 @@ out:
 
 		goto loop_again;
 	}
-
-	return nr_reclaimed;
 }
 
 /*

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

# HG changeset patch
# User Andrea Arcangeli <[email protected]>
# Date 1181332961 -7200
# Node ID 60059913ab07906fceda14ffa72f2c77ef282fca # Parent aafcc5c9057f11d88c43b823c241f14a5ebdd638 don't depend on PF_EXITING tasks to go away

A PF_EXITING task don't have TIF_MEMDIE set so it might get stuck in memory allocations without access to the PF_MEMALLOC pool (said that ideally do_exit would better not require memory allocations, especially not before calling exit_mm). The same way we raise its privilege to TIF_MEMDIE if it's the current task, we should do it even if it's not the current task to speedup oom killing.

@@ -233,27 +233,13 @@ static struct task_struct *select_bad_pr
 		 * Note: this may have a chance of deadlock if it gets
 		 * blocked waiting for another task which itself is waiting
 		 * for memory. Is there a better alternative?
+		 *
+		 * Better not to skip PF_EXITING tasks, since they
+		 * don't have access to the PF_MEMALLOC pool until
+		 * we select them here first.
 		 */
 		if (test_tsk_thread_flag(p, TIF_MEMDIE))
 			return ERR_PTR(-1UL);
-
-		/*
-		 * This is in the process of releasing memory so wait for it
-		 * to finish before killing some other task by mistake.
-		 *
-		 * However, if p is the current task, we allow the 'kill' to
-		 * go ahead if it is exiting: this will simply set TIF_MEMDIE,
-		 * which will allow it to gain access to memory reserves in
-		 * the process of exiting and releasing its resources.
-		 * Otherwise we could get an easy OOM deadlock.
-		 */
-		if (p->flags & PF_EXITING) {
-			if (p != current)
-				return ERR_PTR(-1UL);
-
-			chosen = p;
-			*ppoints = ULONG_MAX;
-		}
 
 		if (p->oomkilladj == OOM_DISABLE)
 			continue;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

# HG changeset patch
# User Andrea Arcangeli <[email protected]>
# Date 1181332961 -7200
# Node ID 4a70e6a4142230fa161dd37202cd62fede122880 # Parent 60059913ab07906fceda14ffa72f2c77ef282fca fallback killing more tasks if tif-memdie doesn't go away

Waiting indefinitely for a TIF_MEMDIE task to go away will deadlock. Two tasks reading from the same inode at the same time and both going out of memory inside a read(largebuffer) syscall, will even deadlock through contention over the PG_locked bitflag. The task holding the semaphore detects oom but the oom killer decides to kill the task blocked in wait_on_page_locked(). The task holding the semaphore will hang inside alloc_pages that will never return because it will wait the TIF_MEMDIE task to go away, but the TIF_MEMDIE task can't go away until the task holding the semaphore is killed in the first place.

It's quite unpractical to teach the oom killer the locking dependencies across running tasks, so the feasible fix is to develop a logic that after waiting a long time for a TIF_MEMDIE tasks goes away, fallbacks on killing one more task. This also eliminates the possibility of suprious oom killage (i.e. two tasks killed despite only one had to be killed). It's not a math guarantee because we can't demonstrate that if a TIF_MEMDIE SIGKILLED task didn't mange to complete do_exit within 10sec, it never will. But the current probability of suprious oom killing is sure much higher than the probability of suprious oom killing with this patch applied.

The whole locking is around the tasklist_lock. On one side do_exit reads TIF_MEMDIE and clears VM_is_OOM under the lock, on the other side the oom killer accesses VM_is_OOM and TIF_MEMDIE under the lock. This is a read_lock in the oom killer but it's actually a write lock thanks to the OOM_lock semaphore running one oom killer at once (the locking rule is, either use write_lock_irq or read_lock+OOM_lock).

@@ -845,6 +845,15 @@ static void exit_notify(struct task_stru
 	     unlikely(tsk->parent->signal->flags & SIGNAL_GROUP_EXIT)))
 		state = EXIT_DEAD;
 	tsk->exit_state = state;
+
+	/*
+	 * Read TIF_MEMDIE and set VM_is_OOM to 0 atomically inside
+	 * the tasklist_lock_lock.
+	 */
+	if (unlikely(test_tsk_thread_flag(tsk, TIF_MEMDIE))) {
+		extern unsigned long VM_is_OOM;
+		clear_bit(0, &VM_is_OOM);
+	}
 
 	write_unlock_irq(&tasklist_lock);
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

# HG changeset patch
# User Andrea Arcangeli <[email protected]>
# Date 1181332962 -7200
# Node ID 24250f0be1aa26e5c6e33fd97b9eae125db9fbde
# Parent 4a70e6a4142230fa161dd37202cd62fede122880
stop useless vm trashing while we wait the TIF_MEMDIE task to exit

There's no point in trying to free memory if we're oom.

@@ -159,6 +159,8 @@ struct swap_list_t {
 #define vm_swap_full() (nr_swap_pages*2 < total_swap_pages)
 
 /* linux/mm/oom_kill.c */
+extern unsigned long VM_is_OOM;
+#define is_VM_OOM() unlikely(test_bit(0, &VM_is_OOM))
 extern void out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask, int order);
 extern int register_oom_notifier(struct notifier_block *nb);
 extern int unregister_oom_notifier(struct notifier_block *nb);

@@ -923,6 +923,8 @@ static unsigned long shrink_zone(int pri
 		nr_inactive = 0;
 
 	while (nr_active || nr_inactive) {
+		if (is_VM_OOM())
+			break;
 		if (nr_active) {
 			nr_to_scan = min(nr_active,
 					(unsigned long)sc->swap_cluster_max);

@@ -1032,6 +1034,17 @@ unsigned long try_to_free_pages(struct z
 	}
 
 	for (priority = DEF_PRIORITY; priority >= 0; priority--) {
+		if (is_VM_OOM()) {
+			if (!test_thread_flag(TIF_MEMDIE)) {
+				/* get out of the way */
+				schedule_timeout_interruptible(1);
+				/* don't waste cpu if we're still oom */
+				if (is_VM_OOM())
+					goto out;
+			} else
+				goto out;
+		}
+
 		sc.nr_scanned = 0;
 		if (!priority)
 			disable_swap_token();

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

# HG changeset patch
# User Andrea Arcangeli <[email protected]>
# Date 1181332962 -7200
# Node ID c6dfb528f53eaac2188b49f67eed51c1a33ce7cd
# Parent 24250f0be1aa26e5c6e33fd97b9eae125db9fbde the oom schedule timeout isn't needed with the VM_is_OOM logic

VM_is_OOM whole point is to give a proper time to the TIF_MEMDIE task in order to exit.

@@ -468,12 +468,5 @@ out:
 	read_unlock(&tasklist_lock);
 	cpuset_unlock();
 
-	/*
-	 * Give "p" a good chance of killing itself before we
-	 * retry to allocate memory unless "p" is current
-	 */
-	if (!test_thread_flag(TIF_MEMDIE))
-		schedule_timeout_uninterruptible(1);
-
 	up(&OOM_lock);
 }

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

# HG changeset patch
# User Andrea Arcangeli <[email protected]>
# Date 1181332962 -7200
# Node ID db4c0ce6754d7838713eda1851aef43c2fb52fca
# Parent c6dfb528f53eaac2188b49f67eed51c1a33ce7cd show mem information only when a task is actually being killed

Don't show garbage while VM_is_OOM and the timeout didn't trigger.

@@ -286,7 +286,7 @@ static void __oom_kill_task(struct task_
 	force_sig(SIGKILL, p);
 }
 
-static int oom_kill_task(struct task_struct *p)
+static int oom_kill_task(struct task_struct *p, gfp_t gfp_mask, int order)
 {
 	struct mm_struct *mm;
 	struct task_struct *g, *q;

@@ -313,93 +313,6 @@ static int oom_kill_task(struct task_str
 			return 1;
 	} while_each_thread(g, q);
 
-	__oom_kill_task(p, 1);
-
-	/*
-	 * kill all processes that share the ->mm (i.e. all threads),
-	 * but are in a different thread group. Don't let them have access
-	 * to memory reserves though, otherwise we might deplete all memory.
-	 */
-	do_each_thread(g, q) {
-		if (q->mm == mm && q->tgid != p->tgid)
-			force_sig(SIGKILL, q);
-	} while_each_thread(g, q);
-
-	return 0;
-}
-
-static int oom_kill_process(struct task_struct *p, unsigned long points,
-		const char *message)
-{
-	struct task_struct *c;
-	struct list_head *tsk;
-
-	/*
-	 * If the task is already exiting, don't alarm the sysadmin or kill
-	 * its children or threads, just set TIF_MEMDIE so it can die quickly
-	 */
-	if (p->flags & PF_EXITING) {
-		__oom_kill_task(p, 0);
-		return 0;
-	}
-
-	printk(KERN_ERR "%s: kill process %d (%s) score %li or a child\n",
-					message, p->pid, p->comm, points);
-
-	/* Try to kill a child first */
-	list_for_each(tsk, &p->children) {
-		c = list_entry(tsk, struct task_struct, sibling);
-		if (c->mm == p->mm)
-			continue;
-		/*
-		 * We cannot select tasks with TIF_MEMDIE already set
-		 * or we'll hard deadlock.
-		 */
-		if (unlikely(test_tsk_thread_flag(c, TIF_MEMDIE)))
-			continue;
-		if (!oom_kill_task(c))
-			return 0;
-	}
-	return oom_kill_task(p);
-}
-
-static BLOCKING_NOTIFIER_HEAD(oom_notify_list);
-
-int register_oom_notifier(struct notifier_block *nb)
-{
-	return blocking_notifier_chain_register(&oom_notify_list, nb);
-}
-EXPORT_SYMBOL_GPL(register_oom_notifier);
-
-int unregister_oom_notifier(struct notifier_block *nb)
-{
-	return blocking_notifier_chain_unregister(&oom_notify_list, nb);
-}
-EXPORT_SYMBOL_GPL(unregister_oom_notifier);
-
-/**
- * out_of_memory - kill the "best" process when we run out of memory
- *
- * If we run out of memory, we have the choice between either
- * killing a random task (bad), letting the system crash (worse)
- * OR try to be smart about which process to kill. Note that we
- * don't have to be perfect here, we just have to be good.
- */
-void out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask, int order)
-{
-	struct task_struct *p;
-	unsigned long points = 0;
-	unsigned long freed = 0;
-	int constraint;
-	static DECLARE_MUTEX(OOM_lock);
-
-	blocking_notifier_call_chain(&oom_notify_list, 0, &freed);
-	if (freed > 0)
-		/* Got some memory back in the last second. */
-		return;
-
-	if (down_trylock(&OOM_lock))
-		return;
 	if (printk_ratelimit()) {
 		printk(KERN_WARNING "%s invoked oom-killer: "
 			"gfp_mask=0x%x, order=%d, oomkilladj=%d\n",

@@ -408,6 +321,94 @@ void out_of_memory(struct zonelist *zone
 		show_mem();
 	}
 
+	__oom_kill_task(p, 1);
+
+	/*
+	 * kill all processes that share the ->mm (i.e. all threads),
+	 * but are in a different thread group. Don't let them have access
+	 * to memory reserves though, otherwise we might deplete all memory.
+	 */
+	do_each_thread(g, q) {
+		if (q->mm == mm && q->tgid != p->tgid)
+			force_sig(SIGKILL, q);
+	} while_each_thread(g, q);
+
+	return 0;
+}
+
+static int oom_kill_process(struct task_struct *p, unsigned long points,
+			    const char *message, gfp_t gfp_mask, int order)
+{
+	struct task_struct *c;
+	struct list_head *tsk;
+
+	/*
+	 * If the task is already exiting, don't alarm the sysadmin or kill
+	 * its children or threads, just set TIF_MEMDIE so it can die quickly
+	 */
+	if (p->flags & PF_EXITING) {
+		__oom_kill_task(p, 0);
+		return 0;
+	}
+
+	printk(KERN_ERR "%s: kill process %d (%s) score %li or a child\n",
+					message, p->pid, p->comm, points);
+
+	/* Try to kill a child first */
+	list_for_each(tsk, &p->children) {
+		c = list_entry(tsk, struct task_struct, sibling);
+		if (c->mm == p->mm)
+			continue;
+		/*
+		 * We cannot select tasks with TIF_MEMDIE already set
+		 * or we'll hard deadlock.
+		 */
+		if (unlikely(test_tsk_thread_flag(c, TIF_MEMDIE)))
+			continue;
+		if (!oom_kill_task(c, gfp_mask, order))
+			return 0;
+	}
+	return oom_kill_task(p, gfp_mask, order);
+}
+
+static BLOCKING_NOTIFIER_HEAD(oom_notify_list);
+
+int register_oom_notifier(struct notifier_block *nb)
+{
+	return blocking_notifier_chain_register(&oom_notify_list, nb);
+}
+EXPORT_SYMBOL_GPL(register_oom_notifier);
+
+int unregister_oom_notifier(struct notifier_block *nb)
+{
+	return blocking_notifier_chain_unregister(&oom_notify_list, nb);
+}
+EXPORT_SYMBOL_GPL(unregister_oom_notifier);
+
+/**
+ * out_of_memory - kill the "best" process when we run out of memory
+ *
+ * If we run out of memory, we have the choice between either
+ * killing a random task (bad), letting the system crash (worse)
+ * OR try to be smart about which process to kill. Note that we
+ * don't have to be perfect here, we just have to be good.
+ */
+void out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask, int order)
+{
+	struct task_struct *p;
+	unsigned long points = 0;
+	unsigned long freed = 0;
+	int constraint;
+	static DECLARE_MUTEX(OOM_lock);
+
+	blocking_notifier_call_chain(&oom_notify_list, 0, &freed);
+	if (freed > 0)
+		/* Got some memory back in the last second. */
+		return;
+
+	if (down_trylock(&OOM_lock))
+		return;
+
 	if (sysctl_panic_on_oom == 2)
 		panic("out of memory. Compulsory panic_on_oom is selected.\n");
 

@@ -434,12 +435,12 @@ void out_of_memory(struct zonelist *zone
 	switch (constraint) {
 	case CONSTRAINT_MEMORY_POLICY:
 		oom_kill_process(current, points,
-				"No available memory (MPOL_BIND)");
+				 "No available memory (MPOL_BIND)", gfp_mask, order);
 		break;
 
 	case CONSTRAINT_CPUSET:
 		oom_kill_process(current, points,
-				"No available memory in cpuset");
+				 "No available memory in cpuset", gfp_mask, order);
 		break;
 
 	case CONSTRAINT_NONE:

@@ -458,7 +459,7 @@ retry:
 			panic("Out of memory and no killable processes...\n");
 		}
 
-		if (oom_kill_process(p, points, "Out of memory"))
+		if (oom_kill_process(p, points, "Out of memory", gfp_mask, order))
 			goto retry;
 
 		break;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

# HG changeset patch
# User Andrea Arcangeli <[email protected]>
# Date 1181332962 -7200
# Node ID dfac333eb29032dab87dd2c46f71a22037a6dc4a # Parent db4c0ce6754d7838713eda1851aef43c2fb52fca simplify oom heuristics

Over time somebody had the good idea to remove the rcvd_sigterm points, this removes more of them. The selected task should be the one that if we don't kill, it will turn the system oom again sooner than later. These informations tell us nothing about which task is best to kill so they should be removed.

@@ -52,7 +52,7 @@ static unsigned long last_tif_memdie_jif
 
 unsigned long badness(struct task_struct *p, unsigned long uptime)
 {
-	unsigned long points, cpu_time, run_time, s;
+	unsigned long points;
 	struct mm_struct *mm;
 	struct task_struct *child;
 

@@ -93,26 +93,6 @@ unsigned long badness(struct task_struct
 			points += child->mm->total_vm/2 + 1;
 		task_unlock(child);
 	}
-
-	/*
-	 * CPU time is in tens of seconds and run time is in thousands
-         * of seconds. There is no particular reason for this other than
-         * that it turned out to work very well in practice.
-	 */
-	cpu_time = (cputime_to_jiffies(p->utime) + cputime_to_jiffies(p->stime))
-		>> (SHIFT_HZ + 3);
-
-	if (uptime >= p->start_time.tv_sec)
-		run_time = (uptime - p->start_time.tv_sec) >> 10;
-	else
-		run_time = 0;
-
-	s = int_sqrt(cpu_time);
-	if (s)
-		points /= s;
-	s = int_sqrt(int_sqrt(run_time));
-	if (s)
-		points /= s;
 
 	/*
 	 * Niced processes are most likely less important, so double

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

# HG changeset patch
# User Andrea Arcangeli <[email protected]>
# Date 1181332962 -7200
# Node ID dbd70ffd95f34cd12f1fd2f05a9cc0f9a50edb4a # Parent dfac333eb29032dab87dd2c46f71a22037a6dc4a oom select should only take rss into account

Running workloads where many tasks grow their virtual memory simultaneously, so they all have a relatively small virtual memory when oom triggers (if compared to innocent longstanding tasks), the oom killer then selects mysql/apache and other things with very large VM but very small RSS. RSS is the only thing that matters, killing a task with huge VM but zero RSS is not useful. Many apps tend to have large VM but small RSS in the first place (regardless of swapping activity) and they shouldn't be penalized like this.

@@ -66,7 +66,7 @@ unsigned long badness(struct task_struct
 	/*
 	 * The memory size of the process is the basis for the badness.
 	 */
-	points = mm->total_vm;
+	points = get_mm_rss(mm);
 
 	/*
 	 * After this unlock we can no longer dereference local variable `mm'

@@ -90,7 +90,7 @@ unsigned long badness(struct task_struct
 	list_for_each_entry(child, &p->children, sibling) {
 		task_lock(child);
 		if (child->mm != mm && child->mm)
-			points += child->mm->total_vm/2 + 1;
+			points += get_mm_rss(child->mm)/2 + 1;
 		task_unlock(child);
 	}
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

# HG changeset patch
# User Andrea Arcangeli <[email protected]>
# Date 1181332962 -7200
# Node ID 31ef5d0bf924fb47da144321f692f4fefebf5cf5 # Parent dbd70ffd95f34cd12f1fd2f05a9cc0f9a50edb4a limit reclaim if enough pages have been freed

No need to wipe out an huge chunk of the cache.

@@ -938,6 +938,8 @@ static unsigned long shrink_zone(int pri
 			nr_inactive -= nr_to_scan;
 			nr_reclaimed += shrink_inactive_list(nr_to_scan, zone,
 								sc);
+			if (nr_reclaimed >= sc->swap_cluster_max)
+				break;
 		}
 	}
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

# HG changeset patch
# User Andrea Arcangeli <[email protected]>
# Date 1181332962 -7200
# Node ID 19fb832beb3c83b7bed13c1a2f54ec4e077cfc0d # Parent 31ef5d0bf924fb47da144321f692f4fefebf5cf5 avoid some lock operation in vm fast path

Let's not bloat the kernel for numa. Not nice, but at least this way perhaps somebody will clean it up instead of hiding the inefficiency in there.

@@ -223,8 +223,10 @@ struct zone {
 	unsigned long		pages_scanned;	   /* since last reclaim */
 	int			all_unreclaimable; /* All pages pinned */
 
+#ifdef CONFIG_NUMA
 	/* A count of how many reclaimers are scanning this zone */
 	atomic_t		reclaim_in_progress;
+#endif
 
 	/* Zone statistics */
 	atomic_long_t		vm_stat[NR_VM_ZONE_STAT_ITEMS];

@@ -2650,7 +2650,9 @@ static void __meminit free_area_init_cor
 		INIT_LIST_HEAD(&zone->active_list);
 		INIT_LIST_HEAD(&zone->inactive_list);
 		zap_zone_vm_stats(zone);
+#ifdef CONFIG_NUMA
 		atomic_set(&zone->reclaim_in_progress, 0);
+#endif
 		if (!size)
 			continue;
 

@@ -909,7 +909,9 @@ static unsigned long shrink_zone(int pri
 	unsigned long nr_to_scan;
 	unsigned long nr_reclaimed = 0;
 
+#ifdef CONFIG_NUMA
 	atomic_inc(&zone->reclaim_in_progress);
+#endif
 
 	/*
 	 * Add one to `nr_to_scan' just to make sure that the kernel will

@@ -945,7 +947,9 @@ static unsigned long shrink_zone(int pri
 
 	throttle_vm_writeout(sc->gfp_mask);
 
+#ifdef CONFIG_NUMA
 	atomic_dec(&zone->reclaim_in_progress);
+#endif
 	return nr_reclaimed;
 }
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

[...]

Interesting. This seems to demonstrate a need for file IO to handle fatal signals, beyond just people wanting faster responses to kill -9. Perhaps it's the case that fatal signals should always be handled, and there should be no waiting primitives excluding them. __GFP_NOFAIL is also "interesting."

-- wli

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

OOMs can occur because we are in a cpuset or have a memory policy that restricts the allocations. So I guess that OOMness is a per node property and not a global one.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I'm sorry to inform you that the oom killing in current mainline has always been a global event not a per-node one, regardless of the fixes I just posted.

if (test_tsk_thread_flag(p, TIF_MEMDIE))
return ERR_PTR(-1UL);
[..]

		if (PTR_ERR(p) == -1UL)
	   	       goto out;

Best would be for you to send me more changes at the end of the patchbomb so that for that the first time _ever_, the oom will become a per-node event and not a global one anymore.

Said that it's not entirely obvious to me, that it makes any sense to disrupt functionality instead of just running slower but safely (I would generally prefer printk a warning instead of killing a task if we've to override the restriction on the memory policy). But that's your call, I'm fine either ways...

Thanks!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Wrong. The oom killling is a local event if we are in a constrained allocation. The allocating task is killed not a random task. That call to kill the allocating task should not set any global flags.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I just showed the global flag that is being checked. TIF_MEMDIE affects the whole system, not just your node-constrained allocating task. If your local constrained task fails to exit because it's running in the nfs path that loops forever even if NULL is returned from alloc_pages, it will deadlock the whole system if later a regular oom happens (alloc_pages isn't guaranteed to be called by a page fault where we know do_exit will guaranteed to be called if a sigkill is pending). This is just an example.

Amittedly my fixes made things worse for your "local" oom killing, but your code was only apparently "local" because TIF_MEMDIE is a _global_ flag in the mainline kernel. So again, I'm very willing to improve the local oom killing, so that it will really become a local event for the first time ever. Infact with my fixes applied the whole system will stop waiting for the TIF_MEMDIE flag to go away, so it'll be much easier to really make the global oom killing independent from the local one. I didn't look into the details of the local oom killing yet (exactly because it wasn't so local in the first place) but it may be enough to set VM_is_OOM only for tasks that are not being locally killed and then those new changes will automatically prevent TIF_MEMDIE being set on a local-oom to affect the global-oom event.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

On a side note about the current way you select the task to kill if a constrained alloc failure triggers, I think it would have been better if you simply extended the oom-selector by filtering tasks in function of the current->mems_allowed. Now I agree the current badness is quite bad, now with rss instead of the virtual space, it works a bit better at least, but the whole point is that if you integrate the cpuset task filtering in the oom-selector algorithm, then once we fix the badness algorithm to actually do something more meaningful than to check static values, you'll get the better algorithm working for your local-oom killing too. This if you really care about the huge-numa niche to get node-partitioning working really like if this was a virtualized environment. If you just have kill something to release memory, killing the current task is always the safest choice obviously, so as your customers are ok with it I'm certainly fine with the current approach too.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Wil,

Clearly the sooner we respond to a SIGKILL the better. We tried to catch the two critical points to solve the evil read(huge)->oom. BTW, the first suggestion that we had to also break out of read to make progress substantially quicker, was from Petr so I'm cc'ing him. I'm unsure what else of more generic we could do to solve more of those troubles at the same time without having to pollute the code with sigkill checks. For example we're not yet covering the o-direct paths but I did the minimal changes to resolve the current workload and that used buffered io of course ;). BTW, I could have checked the TIF_MEMDIE instead of seeing if sigkill was pending, but since I had to check the task structure anyway, I preferred to check for the sigkill so that kill -9 will now work for the first time against a large read/write syscall, besides allowing the TIF_MEMDIE task to exit in reasonable time without triggering the deadlock detection in the later patches.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Ok, so it should be changed to DEFINE_MUTEX. I have to trust you on this because there's not a sign of warning in asm-i386/semaphore.h that DECLARE_MUTEX has been deprecated and tons of code is still using it in the current kernel. I couldn't imagine that somebody duplicated it somewhere else for whatever reason without removing DECLARE_MUTEX. It's not like we have to keep deprecated and redundant interfaces in the kernel for no good reason, especially if sed can fix it without human intervention. Let's say it's a low priority to rename it, if I've to generate a new diff, I'd probably prefer to generate one that drops DECLARE_MUTEX all over the other places too.

On this I disagree, the whole point of static/private variables is to decrease visibility where it's unnecessary. A static variable function-local is even less visible so it's a good thing and it helps self-documenting the code. So I very much like to keep it there, coding strict improves readability (you immediately know that no other code could ever try to acquire that lock).

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I've seen recent upstream kernels free up to 75% of memory on my test system, when pushed hard enough.

It is not hard to get hundreds of tasks into the pageout code simultaneously, all starting out at priority 12 and not freeing anything until they all get to much lower priorities.

A workload that is dominated by anonymous memory will trigger this. All anonymous memory starts out on the active list and tasks will not even try to shrink the inactive list because nr_inactive >> priority is 0.

This patch is a step in the right direction.

However, I believe that your [PATCH 01 of 16] is a step in the wrong direction for these workloads...

Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Agreed.

Thanks for the Ack.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

BTW, this reminds me that I've been wondering if 2**12 is a too small fraction of the lru to start the scan with.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

However, if the sigkill is pending against ANOTHER task, this patch looks like it could introduce an IO error where the system would recover fine before.

Tasks that do not have a pending SIGKILL should retry the allocation, shouldn't they?

Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

If the system has 1 TB of RAM, it's probably too big of a fraction :)

We need something smarter.

Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

If all tasks spend 10 minutes in shrink_active_list before the first call to shrink_inactive_list that could mean you hit the race that I'm just trying to fix with this very patch. (i.e. nr_*active going totally huge because of the race triggering, and trashing over the few pages left in the *active_list until the artificially boosted nr_*active finally goes down to zero in all tasks that read it at the unlucky time when it got huge) So my patch may actually fix your situation completely if your trouble was nr_scan_active becoming huge for no good reason, just because many tasks entered the VM at the same time on big-SMP systems. Did you monitor the real sizes of the active lists during those 10 min and compared it to the nr_active stored in the stack?

Normally if the highest priority passes only calls into shrink_active_list that's because the two lists needs rebalancing. But I fail to see how it could ever take 10min for the first shrink_inactive_list to trigger with my patch applied, while if it happens in current vanilla that could be the race triggering, or anyway something unrelated is going wrong in the VM.

Overall this code seems quite flakey in its current "racy" form, so I doubt it can be allowed to live as-is. Infact even if we fix the race with a slow-shared-lock in a fast path or if we only make sure not to avoid exacerbate your situation with something a simple and lock-less as "nr_active = min(sizeof_active_list, nr_scan_active)", I think it would still wrong to do more work in the current tasks, if we've other tasks helping us at the same time. We should do nothing more, nothing less. So I think if we want those counters to avoid restarting from zero at each priority step (what I understand is your worry), those counters should be in the stack, task-local. That will still take into account the previously not scanned "nr_inactive" value.

Not sure what's best. I've the feeling that introducing a task-local *nr_active *nr_inactive counter shared by all priority steps, won't move the VM needle much, but I sure wouldn't be against it. It will change the balancing to be more fair, but in practice I don't expect huge differences, there are only 12 steps anyway, very quickly the inactive list should be shrunk even if the active list is huge.

I'm only generally against the current per-zone global and racy approach without limits, so potentially exacerbating your situation when nr_active becomes very huge despite the active list being very small.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

The error being returned would be -ENOMEM. But even that should not be returned because do_exit will run before userland runs again. When I told about this to Neil he didn't seem to object that do_exit will be called first so I hope we didn't get it wrong.

The only risk would be if we set TIF_MEMDIE but we kill a task with SIGTERM, then the I/O error could reach userland if the user catched the sigterm signal in userland.

I didn't add the warn-on for sigkill, because even if we decide to send sigterm first, in theory it wouldn't be a kernel issue if we correctly return -ENOMEM to userland if that is the task that must exit (we don't support a graceful exit path today, perhaps we never will). But clearly we don't know if all userland code is capable of coping with a -ENOMEM, so for now we don't have to worry thanks to the sigkill.

All tasks not having TIF_MEMDIE set (and currently sigkill pending as well) should retry yes.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Nope. In this case it spends its time in shrink_active_list because the active list is 99% of memory (several GB) while the inactive list is so small that nr_inactive_pages >> priority is zero.

Yeah, I have no real objection to your patch, but was just pointing out that it does not fix the big problem with this code.

All Rights Reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

TIF_MEMDIE affects the task that attempted to perform an constrained allocation. The effects are global for that task but there are not as severe as setting a global OOM flag!

TIF_MEMDIE is confined to a process.

TIF_MEMDIE must be set in order for the task to die properly even if its a constrained allocation because TIF_MEMDIE relaxes the constraints so that the task can terminate.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Filtering tasks is a very expensive operation on huge systems. We have had cases where it took an hour or so for the OOM to complete. OOM usually occurs under heavy processing loads which makes the taking of global locks quite expensive.

The "kill-the-current-process" approach is most effective in hitting the process that is allocating the most. And as far as I can tell its easiest to understand for our customer.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Well this value is depending on a nodes memory not on the systems total memory. So I think we are fine. 1TB systems (at least ours) are comprised of nodes with 4GB/8GB/16GB of memory.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Come on, oom_kill.c only happens at oom time, after the huge complex processing has figured out it's time to call into oom_kill.c, how can you care about the performance of oom_kill.c? Apparently some folks prefer to panic when oom triggers go figure...

Since you mean that a _global_ OOM took one hour (you just used it as the comparison of the slow-one, the local-oom is supposed to be the fast one instead) I'd appreciate if you could try again with all my fixes applied and see if the time to recover the global oom is reduced (which is the whole objective of most of the fixes I've just posted).

In general whatever you do inside oom_kill.c has nothing to do with the "expensive operations" (the expensive operations are infact halted with my fixes).

In turn killing the current task so that oom_kill.c is faster, is quite a dubious argument.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yours are fine, because currently the very large system customers tend to run fine tuned workloads.

We are seeing some other users throwing random workloads at systems with 256GB of RAM in a single zone. General purpose computing is moving up, VM explosions are becoming more spectacular :)

All Rights Reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Its pretty bad if a large system sits for hours just because it cannot finish its OOM processing. We have reports of that taking 4 hours!

It avoids repeated scans over a super sized tasklist with heavy lock contention. 4 loops for every OOM kill! If a number of processes will be OOM killed then it will take hours to sort out the lock contention.

Want this as a a SUSE bug?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Which is why I posted these fixes, so it will hopefully take much less than 4 hours. Even normal production systems takes far too long today. Most of these fixes are meant to reduce the complexity involved in detecting when the system is oom (starting from number 01). Keep in mind the whole 4 hours are spent _outside_ oom_kill.c.

Once the tasklist_lock has been taken, what else is going to trash inside oom_kill.c?

Did you measure it or this is just your imagination? I don't buy your hypothetical "several hours spent in oom_kill.c" numbers. How long does "ls /proc" takes? Can your run top at all?

Feel free to file a SUSE bugreport so I hope you will back your claim with some real profiling data and so we can check if this can be fixed in software of it's the hardware to blame (in which case we need a CONFIG_SLOW_NUMA, since other hardware implementations may prefer to use the oom-selector during local-oom killing too and not only during the global ones).

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

These are customer reports. 4 hours one and another 2 hours. I can certainly get more reports if I ask them for more details. I will get this on your SUSE radar.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

How long does "ls /proc" take? Can you run top at all on such a system (I mean before it reaches the oom point, then it'll hang for those 4 hours with the mainline kernel, I know this and that's why I worked to fix it and posted 18 patches so far about it).

If it takes 4 hours for the function out_of_memory to return, please report it. If instead as I start to suspect, you're going to show me the function out_of_memory called one million times and taking a few seconds for each invocation, please test all my fixes before reporting, there's a reason I made those changes...

Back to the local-oom: if out_of_memory takes a couple of seconds at most as I expect (it'll be the same order of ls /proc, actually ls /proc will be a lot slower), killing the current task in the local-oom as a performance optimization remains a very dubious argument.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

These are big systems and it would take some time to reproduce these issues. Thanks for your work. I'd really like to see improvements there. If you take care of not worsening the local kill path then I am okay with the rest.

out_of_memory takes about 5-10 minutes each (according to one report). An OOM storm will then take the machine out for 4 hours. The on site SE can likely tell you more details in the bugzilla.

Another reporter had been waiting for 2 hours after an oom without any messages indicating that a single OOM was processed.

Killing the local process avoids 4 slow scans over a pretty large tasklist. But I agree that there may be additionial other issues lurking there fore large systems.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Sure I understand.

I appreciate and hope it already helps for your oom troubles too.

The slight regression I introduced for the numa local oom path clearly needed correction. Let me know if you still see problems after the incremental patch I posted today of course. I think that should be enough to correct the local-oom without altering the global-oom. I tested it on non-numa and it still works fine.

Even 10 minutes is way beyond what I expected (but with the background trashing of the mainline kernel, I can imagine it happening).

Ok, then I think you really want to try my patchset for the oom storm since at least that one should be gone. When the first oom starts, the whole VM will stop, no other oom_kill will be called, and even if they're on their way to call a spurious out_of_memory, the semaphore trylock will put them back in S state immediately inside try_to_free_pages. Especially in systems like yours where trashing cachelines is practically forbidden, I suspect this could make a substantial difference and perhaps then out_of_memory will return in less than 10 minutes by the fact of practically running single threaded.

This is the case I'm dealing with more commonly, normally the more swap more more it takes, and that's expectable. It should have improved too with the patchset.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Do you have a SLES10 kernel with these fixes?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Late as always... :((
It's not only about getting it quicker - the loop wouldn't break until the whole chunk has been read, which couldn't be accomplished until some memory was freed first, but the memory would be freed by killing this task which wouldn't terminate until everything is read, and so on... We obviously need to break the vicious circle somewhere.

If we want to resolve all such cases we would have to ensure that delivering a SIGKILL can't fail on OOM conditions, i.e. that SIGKILL can always be handled without memory allocation. I'm planning to do some investigations on which places in the kernel are (worst) affected and then think about ways of fixing them. I don't expect we can fix them all, or at least not in the first round, but this looks like the only way to go...

Cheers,
Petr Tesarik

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGbmBRjpY2ODFi2ogRAseoAKCV+rX+PTmdGdjmjdObBwmdYDlqXACfXI9f BT5dOXg5qPVhH7Wj/DlHCP4=
=ZlW9
-----END PGP SIGNATURE-----

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I got around to testing it now. I am using AIM7 since it is a very anonymous memory heavy workload.

Unfortunately your patch does not fix the problem, but behaves as I had feared :(

Both the normal kernel and your kernel fall over once memory pressure gets big enough, but they explode differently and at different points.

I am running the test on a quad core x86-64 system with 2GB memory. I am "zooming in" on the 4000 user range, because that is where they start to diverge. I am running aim7 to cross-over, which is the point at which fewer than 1 jobs/min/user are being completed.

First vanilla 2.6.22-rc5-git8:

Num     Parent   Child   Child  Jobs per   Jobs/min/  Std_dev  Std_dev  JTI
Forked  Time     SysTime UTime   Minute     Child      Time     Percent
4000    119.97   432.86  47.17   204051.01  51.01      11.52    9.99 

90
4100 141.59 517.31 48.92 177215.91 43.22 6.67 4.84
95
4200 154.95 569.16 50.51 165885.77 39.50 5.07 3.35
96
4300 166.24 613.40 51.58 158301.25 36.81 10.59 6.51
93
4400 170.40 628.63 52.72 158028.17 35.92 5.46 3.27
96
4500 188.88 701.84 54.06 145806.86 32.40 6.13 3.31
96
4600 200.37 745.73 55.55 140500.07 30.54 4.98 2.54
97
4700 219.25 819.80 57.01 131192.70 27.91 5.38 2.51
97
4800 219.70 820.36 58.22 133709.60 27.86 5.40 2.52
97
4900 232.45 870.08 59.56 129008.39 26.33 4.65 2.02
97
5105 1704.46 5406.56 64.03 18329.91 3.59 264.38 18.85
81
Crossover achieved
Max Jobs per Minute 204051.01

Now 2.6.22-rc5-git8 with your patches 01/16 and 15/16:

Num     Parent   Child   Child  Jobs per   Jobs/min/  Std_dev  Std_dev  JTI
Forked  Time     SysTime UTime   Minute     Child      Time     Percent
4000    141.51   518.37  47.46   172991.31  43.25      5.20     3.75 

96
4100 147.07 539.16 48.91 170612.63 41.61 5.11 3.58
96
4200 155.43 571.36 50.18 165373.48 39.37 5.42 3.58
96
4300 1317.89 4558.95 52.53 19968.28 4.64 219.76 18.42
81
Crossover achieved
Max Jobs per Minute 172991.31

One thing I noticed is that with the vanilla kernel, the lower numbers of users allowed the system to still run fine, while with your patches the system seemed to get stuck at ~90% system time pretty quickly...

All Rights Reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

with what command line and config scripts does one run aim7 to reproduce this?

Where's the system time being spent?

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

reaim -x -i 100 -s 5000

Using the default reaim.config and workfile.shared

I will run the tests again with profiling enabled.

All Rights Reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Ok, at some point it's normal they start trashing. What is strange is that it seems patch 01 requires the VM to do more work and in turn more memory to be free. The only explanation I could have is that the race has the side effect of in average reducing the amount of vm activity for each task instead of increasing it (this in turn reduces thrashing and free memory level requirements before the workload halts).

Even if it may have a positive effect in practice, I still think the current racy behavior (randomly overstimating and randomly understimating the amount of work each task has to do depending of who adds and read the zone values first) isn't good.

Perhaps if you change the DEF_PRIORITY you'll get closer to the current mainline but without any race. You can try to halve it and see what happens. If the initial passes fails, it'll start swapping and performance will go down quick. So perhaps once we fix the race we've to decrease DEF_PRIORITY to get the same vm-tune.

It'd also be interesting to see what we get between 3000 and 4000.

Where exactly we get to the halting point (4300 vs 5105) isn't crucial, otherwise one can win by simply decreasing min_free_kbytes as well, which clearly shows "when" we hang isn't the real interest. OTOH I agree the difference between 4300 and 5105 seems way too big but if this was between 5000 and 5105 I wouldn't worry too much (5000 instead of 5105 would result in more memory to be free at the oom point which isn't a net-negative). Hope the benchmark is repeatable. This week I've been working on another project but I'll shortly try to install AIM and reproduce and see what happens by decreasing DEF_PRIORITY. Thanks for the testing!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yes, but I would hope that the system would be disk bound at that time instead of CPU bound.

There was no swap IO going on yet, the system was just wasting CPU time in the VM.

Oh, I like your simplification of the code, too.

I was running the test to see if that patch could be merged without any negative side effects, because I would have liked to see it.

However, neither of the two seems to be IO bound
at that point...

Not only is the AIM7 test perfectly repeatable, it also causes the VM to show some of the same behaviour that customers are seeing in the field with large JVM workloads.

All Rights Reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

That seems a separate problem, 01 starts wasting cpu sooner and that's the regression you discovered, but mainline wastes cpu the same way too later on. We should do some profiling like Andrew suggested to see what's going on when it starts trashing cpu (perhaps it's some smp lock? you said you've only 4 cores so it must be some highly contended one if it's really a lock).

I see. Good that you tested this with this workload so we noticed this regression. At the moment I hope it's only a tuning knob in the DEF_PRIORITY (or similar), it'd be really sad if this had a magic racy behavior that wouldn't be reproducible with a static non-racy algorithm.

If nothing else, if we want to stick with this explicit smp race in the vm core, somebody should at least attempt to document how they can predict what the race will do at runtime, because to me it seems quite an unpredictable beast. On average it will probably reach a stable state, but this stable state will depend on the speed of the cpu caches and on the number of cpus, on the architecture and on the assembly generated by gcc, and then the race will trigger more or less or in a different way...

Yes. For now I'd be happy to see the same results for both to eliminate the regression.

Sounds good, thanks!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

OK, it turns out that there is quite a bit of variability in where the system spends its time. I did a number of reaim runs and averaged the time the system spent in the top functions.

This is with the Fedora rawhide kernel config, which has quite a few debugging options enabled.

_raw_spin_lock 32.0%
page_check_address 12.7%
__delay 10.8%
mwait_idle 10.4%
anon_vma_unlink 5.7%
__anon_vma_link 5.3%
lockdep_reset_lock 3.5%
__kmalloc_node_track_caller 2.8%
security_port_sid 1.8%
kfree 1.6%
anon_vma_link 1.2%
page_referenced_one 1.1%

In short, the system is waiting on the anon_vma lock.

I wonder if Lee Schemmerhorn's patch to turn that
spinlock into an rwlock would help this workload,
or if we simply should scan fewer pages in the
pageout code.

Andrea, with your VM patches for some reason the number of users where reaim has its crossover point is also somewhat variable, between 4200 and 5100 users, with 9 out of 10 runs under 4500 on my system.

A kernel without your patches is not as variable,
but has visibly more unfairness between tasks, as
seen in the reaim "Std_dev" columns.

Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Sigh. We had a workload (forget which, still unfixed) in which things would basically melt down in that linear anon_vma walk, walking 10,000 or more vma's. I wonder if that's what's happening here?

Also, one thing to watch out for here is a problem with the spinlocks
themselves: the problem wherein the cores in one package keep rattling the lock around between them and never let it out for the cores in another package to grab.

Maybe. I'm thinking that the problem here is really due to the huge amount of processing which needs to occur when we are in the "all pages active, referenced" state and then we hit pages_low. Panic time, we need to scan and deactivate a huge amount of stuff.

Would it not be better to prevent that situation from occurring by doing a bit of scanning and balancing when adding pages to the LRU? Make sure that the lists will be in reasonable shape for when reclaim starts?

That'd deoptimise those workloads which allocate and free pages but never enter reclaim. Probably liveable with.

We would want to avoid needlessly unmapping pages and causing more minor faults.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

That would be a large multi-threaded application that fills up memory. Customers are reproducing this with JVMs on some very large systems.

This is a single package quad core system, though.

Agreed, we need to simply scan fewer pages.

Doing something like SEQ replacement on the anonymous (and other swap backed) pages might just do the trick here. Page cache, of course, should continue using a used-once scheme.

I suspect we want to split out the lists for many other reasons anyway, as detailed on http://linux-mm.org/PageoutFailureModes

I'll whip up a patch that does this...

If we do true SEQ replacement for anonymous pages (deactivating active pages without regard to the referenced bit) and keep the inactive list reasonably small that penalty should be negligable.

That's a minor issue, the page fault path is pretty cheap and very scalable.

Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

So.... does that mean "yes, it's scanning a lot of vmas"?

If so, I expect there will still be failure modes, whatever we do outside of this. A locked, linear walk of a list whose length is application-controlled is going to be a problem. Could be that we'll need an O(n) -> O(log(n)) conversion, which will be tricky in there.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Not necessarily.

The problem can also be reproduced if you have many threads, from "enough" CPUs, all scanning pages in the same huge VMA.

Scanning fewer pages in the pageout path is probably the way to go.

No matter how efficient we make the scanning of one individual page, we simply cannot scan through 1TB worth of anonymous pages (which are all referenced because they've been there for a week) in order to deactivate something.

Systems that big are only a year or two away from
general purpose use.

Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

There's no swapping, so are we sure we need to scan the pte? This might be as well the unmapping code being invoked too early despite there's still clean cache to free. If I/O would start because swapping is really needed, the O(N) walk wouldn't hog the cpu so much because lots of time would be spent waiting for I/O too. Decreasing DEF_PRIORITY should defer the invocation of the unmapping code too.

Conversion to O(log(N)) like for the filebacked mappings shouldn't be a big problem but it'll waste more static memory for each vma and anon_vma.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I wouldn't have expected the anon_vma lock to be the main problem for a single vma.

If it _is_ the problem then significant improvements could probably be obtained by passing the whole isolate_lru_pages() pile of pages into the rmap code rather than doing them one-at-a-time.

I don't see why that would help. The bottom-line steady-state case is that we need to reclaim N pages per second, and we need to scan N*M vmas per second to do so. How we chunk that up won't affect the aggregate amount of work which needs to be done.

Or maybe you're referring to the ongoing LRU balancing thing. Or to something else.

Sure. And we could avoid that sudden transition by balancing the LRU prior to hitting the great pages_high wall.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yes, I am indeed talking about LRU balancing.

We pretty much *know* that an anonymous page on the active list is accessed, so why bother scanning them all?

We could just deactivate the oldest ones and clear their referenced bits.

Once they reach the end of the inactive list, we
check for the referenced bit again. If the page
was accessed, we move it back to the active list.

The only problem with this is that anonymous
pages could be easily pushed out of memory by
the page cache, because the page cache has
totally different locality of reference.

The page cache also benefits from the use-once
scheme we have in place today.

Because of these three reasons, I want to split
the page cache LRU lists from the anonymous
memory LRU lists.

Does this make sense to you?

Yes, we will need to do some preactive balancing.

Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

well, for better or for worse, that's the design. We need to run page_referenced() when considering whether to deactivate the page and that involves a scan of all the ptes.

Might be so, but even if we ade changes there, failure modes will remain.

yup. The *total* amount of CPu we spend in there shouldn't matter a lot:
unless something else is bust, it'll be relatively low. I think the problem here is that a) we do it all in a big burst and b) we do it on lots of CPUs at the same time, so that burst is quite an inefficient one.

We _could_ teach kswapd to keep the lists in balance in some fashion even when we're above pages_high. But I suspect that'll have corner-cases and probably it'd be better to do it synchronously. There's not much point in having multiple CPUs doing this so some per-zone trylock could perhaps be used.

hm, OK, I haven't looked at what would be involved there.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Because there might well be pages in there which haven't been accessed in days. Confused.

ok.

I don't immediately see why we need to change the fundamental aging design
at all. The problems afacit are

a) that huge burst of activity when we hit pages_high and

b) the fact that this huge burst happens on lots of CPUs at the same time.

And balancing the LRUs _prior_ to hitting pages_high can address both problems?

It will I guess impact the page aging a bit though.

Could do, don't know. What new problems will it introduce? :(

OK..

And that huge anon-vma walk might need attention. At the least we could do something to prevent lots of CPUs from piling up in there.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

We won't know that unless we actually did some background scanning. Currently hours old (or days old) referenced bits are not cleared from anonymous pages.

That may work on systems with up to a few GB of memory, but customers are already rolling out systems with 256GB of RAM for general purpose use, that's 64 million pages!

Even doing a background scan on that many pages will take insane amounts of CPU time.

In a few years, they will be deploying systems with 1TB of memory and throwing random workloads at them.

Yes, it will. However, I believe that the current system of page aging is simply not sustainable when memory size gets insanely large.

The obvious problem is how to balance the eviction of page cache backed pages versus the eviction of swap backed pages.

The "good news" here is that the current VM does not really balance this either, but relies on system administrators to tweak /proc/sys/vm/swappiness on systems that run a "corner case" workload.

Speaking of which, I have also seen a thousand processes waiting to grab the iprune_mutex in prune_icache.

Maybe direct reclaim processes should not dive into this cache at all, but simply increase some variable indicating that kswapd might want to prune some extra pages from this cache on its next run?

Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I don't see how the amount of memory changes anything here: if there are more pages, more work needs to be done regardless of when we do it.

Still confused.

It would make sense to only permit one cpu at a time to go in and do reclaimation against a particular zone (or even node).

But the problem with the vfs caches is that they aren't node/zone-specific. We wouldn't want to get into the situation where 1023 CPUs are twiddling thumbs waiting for one CPU to free stuff up (or less extreme variants of this).

Tell the node's kswapd to go off and do VFS reclaim while the CPUs on that node wait for it? That would help I guess, but those thousand processes would still need to block _somewhere_ waiting for the memory to come back.

Of course, iprune_mutex is a particularly dumb place in which to do that, because the memory may get freed up from somewhere else.

The general design here could/should be to back off to the top-level when there's contention (that's presently congestion_wait()) and to poll for memory-became-allocatable.

So what we could do here is to back off when iprune_mutex is busy and, if nothing else works out, block in congestion_wait() (which is becoming increasingly misnamed). Then, add some more smarts to congestion_wait():
deliver a wakeup when "enough" memory got freed from the VFS caches.

One suspects that at some stage, congestion_wait() will need to be told what the calling task is actually waiting for (perhaps a zonelist) so that the wakup delivery can become smarter.

But for now, the question is: is this a reasonable overall design? Back off from contention points, block at the top-level, polling for allocatable memory to turn up?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

If we deactivate some of the active pages regardless of whether or not they were recently referenced, you end up with "hey, I need to deactivate 1GB worth of pages", instead of with "I need to scan through 1TB worth of pages to find 1GB of not recently accessed ones".

Note that is the exact same argument used against the used-once cleanups that have been proposed in the past:
it is more work to scan through the whole list than to have pages end up in a "reclaimable" state by default.

The direct reclaimers can free something else. Chances are they don't care about the little bit of memory coming out of these caches.

We just need to make sure the pressure gets evened out later.

Not for the VFS memory. They can just recycle some page cache memory or start IO on anonymous memory going into swap.

Yeah, that sounds doable. Not sure if they should wait in congestion_wait() though, or if they should just return to __alloc_pages() since they may already have reclaimed enough pages from the anonymous list.

I'm not convinced. If we have already reclaimed some pages from the inactive list, why wait in congestion_wait() AT ALL?

Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Well by top-level I meant top-level. The point where we either block or declare oom.

We do that now in alloc_pages(), correctly I believe.

The congestion_wait()s in vmscan.c might be misplaced (ie: too far down) because they could lead to us blocking when some memory actually got freed up (or became freeable?) somewhere else.

To fix that we'd need to take a global look at things from within direct-reclaim, or back out of direct-reclaim back up to alloc_pages(), but remember where we were up to for the next pass. Perhaps by extending scan_control a bit and moving its instantiation up to __alloc_pages().

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Rik:

Here's a fairly recent version of the patch if you want to try it on your workload. We've seen mixed results on somewhat larger systems, with and without your split LRU patch. I've started writing up those results. I'll try to get back to finishing up the writeup after OLS and vacation.

Regards,
Lee

-----------
Patch against 2.6.22-rc4-mm2

Make the anon_vma list lock a read/write lock. Heaviest use of this lock is in the page_referenced()/try_to_unmap() calls from vmscan [shrink_page_list()]. These functions can use a read lock to allow some parallelism for different cpus trying to reclaim pages mapped via the same set of vmas.

This change should not change the footprint of the anon_vma in the non-debug case.

 include/linux/rmap.h |    9 ++++++---
 mm/migrate.c         |    4 ++--
 mm/mmap.c            |    4 ++--
 mm/rmap.c            |   20 ++++++++++----------
 4 files changed, 20 insertions(+), 17 deletions(-)

Index: Linux/include/linux/rmap.h
===================================================================
@@ -24,7 +24,7 @@
  * pointing to this anon_vma once its vma list is empty.
  */
 struct anon_vma {
-	spinlock_t lock;	/* Serialize access to vma list */
+	rwlock_t rwlock;	/* Serialize access to vma list */
 	struct list_head head;	/* List of private "related" vmas */
 };
 

@@ -42,18 +42,21 @@ static inline void anon_vma_free(struct 
 	kmem_cache_free(anon_vma_cachep, anon_vma);
 }
 
+/*
+ * This needs to be a write lock for __vma_link()
+ */
 static inline void anon_vma_lock(struct vm_area_struct *vma)
 {
 	struct anon_vma *anon_vma = vma->anon_vma;
 	if (anon_vma)
-		spin_lock(&anon_vma->lock);
+		write_lock(&anon_vma->rwlock);
 }
 
 static inline void anon_vma_unlock(struct vm_area_struct *vma)
 {
 	struct anon_vma *anon_vma = vma->anon_vma;
 	if (anon_vma)
-		spin_unlock(&anon_vma->lock);
+		write_unlock(&anon_vma->rwlock);
 }
 
 /*

Index: Linux/mm/rmap.c
===================================================================
@@ -25,7 +25,7 @@
  *   mm->mmap_sem
  *     page->flags PG_locked (lock_page)
  *       mapping->i_mmap_lock
- *         anon_vma->lock
+ *         anon_vma->rwlock
  *           mm->page_table_lock or pte_lock
  *             zone->lru_lock (in mark_page_accessed, isolate_lru_page)
  *             swap_lock (in swap_duplicate, swap_info_get)

@@ -85,7 +85,7 @@ int anon_vma_prepare(struct vm_area_stru
 		if (anon_vma) {
 			allocated = NULL;
 			locked = anon_vma;
-			spin_lock(&locked->lock);
+			write_lock(&locked->rwlock);
 		} else {
 			anon_vma = anon_vma_alloc();
 			if (unlikely(!anon_vma))

@@ -104,7 +104,7 @@ int anon_vma_prepare(struct vm_area_stru
 		spin_unlock(&mm->page_table_lock);
 
 		if (locked)
-			spin_unlock(&locked->lock);
+			write_unlock(&locked->rwlock);
 		if (unlikely(allocated))
 			anon_vma_free(allocated);
 	}

@@ -132,10 +132,10 @@ void anon_vma_link(struct vm_area_struct
 	struct anon_vma *anon_vma = vma->anon_vma;
 
 	if (anon_vma) {
-		spin_lock(&anon_vma->lock);
+		write_lock(&anon_vma->rwlock);
 		list_add_tail(&vma->anon_vma_node, &anon_vma->head);
 		validate_anon_vma(vma);
-		spin_unlock(&anon_vma->lock);
+		write_unlock(&anon_vma->rwlock);
 	}
 }
 

@@ -147,13 +147,13 @@ void anon_vma_unlink(struct vm_area_stru
 	if (!anon_vma)
 		return;
 
-	spin_lock(&anon_vma->lock);
+	write_lock(&anon_vma->rwlock);
 	validate_anon_vma(vma);
 	list_del(&vma->anon_vma_node);
 
 	/* We must garbage collect the anon_vma if it's empty */
 	empty = list_empty(&anon_vma->head);
-	spin_unlock(&anon_vma->lock);
+	write_unlock(&anon_vma->rwlock);
 
 	if (empty)
 		anon_vma_free(anon_vma);

@@ -164,7 +164,7 @@ static void anon_vma_ctor(void *data, st
 {
 	struct anon_vma *anon_vma = data;
 
-	spin_lock_init(&anon_vma->lock);
+ 	rwlock_init(&anon_vma->rwlock);
 	INIT_LIST_HEAD(&anon_vma->head);
 }
 

@@ -191,7 +191,7 @@ static struct anon_vma *page_lock_anon_v
 		goto out;
 
 	anon_vma = (struct anon_vma *) (anon_mapping - PAGE_MAPPING_ANON);
-	spin_lock(&anon_vma->lock);
+	read_lock(&anon_vma->rwlock);
 	return anon_vma;
 out:
 	rcu_read_unlock();

@@ -200,7 +200,7 @@ out:
 
 static void page_unlock_anon_vma(struct anon_vma *anon_vma)
 {
-	spin_unlock(&anon_vma->lock);
+	read_unlock(&anon_vma->rwlock);
 	rcu_read_unlock();
 }
 

Index: Linux/mm/mmap.c
===================================================================
@@ -571,7 +571,7 @@ again:			remove_next = 1 + (end > next->
 	if (vma->anon_vma)
 		anon_vma = vma->anon_vma;
 	if (anon_vma) {
-		spin_lock(&anon_vma->lock);
+		write_lock(&anon_vma->rwlock);
 		/*
 		 * Easily overlooked: when mprotect shifts the boundary,
 		 * make sure the expanding vma has anon_vma set if the

@@ -625,7 +625,7 @@ again:			remove_next = 1 + (end > next->
 	}
 
 	if (anon_vma)
-		spin_unlock(&anon_vma->lock);
+		write_unlock(&anon_vma->rwlock);
 	if (mapping)
 		spin_unlock(&mapping->i_mmap_lock);
 

Index: Linux/mm/migrate.c
===================================================================
@@ -228,12 +228,12 @@ static void remove_anon_migration_ptes(s
 	 * We hold the mmap_sem lock. So no need to call page_lock_anon_vma.
 	 */
 	anon_vma = (struct anon_vma *) (mapping - PAGE_MAPPING_ANON);
-	spin_lock(&anon_vma->lock);
+	read_lock(&anon_vma->rwlock);
 
 	list_for_each_entry(vma, &anon_vma->head, anon_vma_node)
 		remove_migration_pte(vma, old, new);
 
-	spin_unlock(&anon_vma->lock);
+	read_unlock(&anon_vma->rwlock);
 }
 
 /*

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

BTW, hope the above numbers are measured before the trashing stage when the number of jobs per second is lower than 10. It'd be nice not to spend all that time in system time but after that point the system will shortly reach oom. It's more important to be fast and save cpu in "useful" conditions (like with <4000 tasks).

This looks a very good idea indeed.

Overall the O(log(N)) change I doubt would help, being able to give an efficient answer to "give me only the vmas that maps this anon page" won't be helpful here since the answer will be the same as the current question "give me any vma that may be mapping this anon page". Only for the filebacked mappings it matters.

Also I'm stunned this is being compared to a java workload, java is a threaded beast (unless you're capable of understanding async-io in which case it's still threaded but with tons less threads, but anyway you code it won't create any anonymous related overhead). What we deal with isn't really an issue with anon-vma but just with the fact the system is trying to unmap pages that are mapped in 4000-5000 pte, so no matter how you code it, there will be still 4000-5000 ptes to check for each page that we want to know if it's referenced and it will take system time, this is an hardware issue not a software one. And the other suspect thing is to do all that pte-mangling work without doing any I/O at all.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

If the numbers were measured only in the thrashing stage, mwait_idle would be the top CPU "user", not the scanning code.

What I am trying to measure is more a question of system robustness than performance. We have seen a few cases where the system took 2 hours to recover to a useful state after running out of RAM, with enough free swap.

Linux needs to deal better with memory filling up. It should start to swap instead of scanning pages for very long periods of time and not recovering for a while.

I'm definately going to give Lee's patch a spin.

Interestingly enough, both a heavy Java workload and this AIM7 test block on the anon_vma lock contention.

Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

<snip>

Andrea:

Yes, the patch is not a panacea. At best, it allows different kswapd's to attempt to unmap different pages associated with the same VMA. But, as you say, you still have to unmap X000 ptes. On one of the smaller ia64 systems we've been testing, we hit this state in the 15000-20000 range of AIM jobs. This patch, along with Rik's split LRU patch allowed us to make forward progress at saturation, and we were actually swapping, instead of just spinning around in page_referenced() and try_to_unmap(). [Actually, I don't think we get past page_referenced() much w/o this patch--have to check.]

I have experimented with another "noreclaim" infrastructure, based on some patches by Larry Woodman at Red Hat, to keep non-reclaimable pages off the active/inactive list. I envisioned this as a general infrastructure to handle this case--pages whose anon_vmas have excessively long vma lists, swap-backed pages for which no swap space is available and mlock()ed pages [a la Nick Piggin's patch].

I will include the patch overview here and send along the 2 infrastructure patches and one "client" patch--the excessively referenced anon_vma case. I'm not proposing that these be considered for inclusion. Just another take on this issue.

The patches are against 2.6.21-rc6. I have been distracted by other issues lately, so they have languished, and even the overview is a bit out of date relative to on-going activity in this area. I did integrate this series with Rik's split LRU patch at one time, and it all "worked" for some definition thereof.

One final note before the "noreclaim overview": I have seen similar behavior on the i_mmap_lock for file back pages running a [too] heavy Oracle/TPC-C workload--on a larger ia64 system with ~8TB of storage. System hung/unresponsive, spitting out "Soft lockup" messages. Stack traces showed cpus in spinlock contention called from page_referenced_file. So, it's not limited to anon pages.

Lee
-----------------

This series of patches introduces support for mananaging "non-reclaimable"
pages off the LRU active and inactive list. In this rather long-winded
overview, I attempt to provide the motivation for this work, describe how it relates to other recent patches that address different aspects of the "problem", and give an overview of the mechanism. I'll try not to repeat too much of this in the patch descriptions.

We have seen instances of large linux servers [10s/100s of GB of memory => millions of pages] apparently hanging for extended periods [10s or minutes or more] while all processors attempt to reclaim memory. For various reasons many of the pages on the LRU lists become difficult or impossible to reclaim. The system spends a lot time trying to reclaim [unmap] the difficult pages and/or shuffling through the impossible ones.

Some of the conditions that make pages difficult or impossible to reclaim:
1) page is anon or shmem, but no swap space available
2) page is mlocked into memory
3) page is anon with an excessive number of related vmas [on the anon_vma list]. More on this below.

The basic noreclaim mechanism, described below, is based
on a patch developed by Larry Woodman of Red Hat for RHEL4 [2.6.9+ based kernel] to address the first condition above--an x86_64 non-NUMA system with 64G-128G memory [16M-32M 4k pages] with very little swap space-- ~2GB. The majority of the memory on the system was consumed by large database shared memory areas. A file IO intensive operation, such as backup, causes remaining free memory to be consumed by the page cache, initiating reclaim.

vmscan then spends a great deal of time shuffling non-swappable anon and shmem pages between the active to the inactive lists, only to find that it can't move them to the swap cache. The pages get reactivated and round and round it goes. Because pages cannot be easily reclaimed, eventually other processors need to allocate pages and enter direct reclaim, only to compete for the zone lru lock. The single [normal] zone on the non-numa platform exacerbates this problem, but it can also arise, per zone, on numa platforms.

Larry's patch alleviates this problem by maintaining anon and shmem pages for which no swap space exists on a per zone noreclaim list. Once the pages have been parked there, vmscan deals only with page cache pages, and anon/shmem pages to which space space has already been assigned. Pages move from the noreclaim list back to the LRU when swap space becomes available.

Upstream developers have been addressing some of these issues in other ways:

Christoph Lameter posted a patch to keep anon pages off the LRU when SWAP support not configured into the kernel. With Christoph's patch, these pages are left out "in limbo"--not on any list. Because of this, Christoph's patch does not address the more common situation of kernels with SWAP configured in, but insufficient or no swap added. I think this is a more common situation because most distros will ship kernels with the SWAP support configured in--at least for "enterprise" use. Maintaining these pages on a noreclaim list, will make it possible to restore these pages to the [in]active lists when/if swap is added.

Nick Piggin's patch to keep mlock'ed pages [condition 2 above] off the LRU list also lets the mlocked/non-reclaimable pages float, not on any list. While Nick's patch does allow these pages to become reclaimable when all memory locks are removed, there is another reason to keep pages on a separate list.

We want to be able to migrate anon pages that have no swap space backing them, and those that are mlocked. Indeed, the migration infrastructure supports this. However, the LRU lists, via the zone lru locks, arbitrate between tasks attempting to migrate the same pages simultaneously. To migrate a page, we must isolate it from the LRU. If the page cannot be isolated, migration gives up and moves on to another page. Which ever task is successful in isolating the page proceeds with the migration. Keeping the nonreclaimable pages on a separate list, protected by the zone lru lock, would preserve this arbitration function. isolate_page_lru(), used by both migration and Nick's mlock patch, can be enhanced to find pages on the noreclaim list, as well as on the [in]active lists.

What's the probability that tasks will race on migrating the same page? Fairly high if auto-migration ever makes it into the kernel, but non-zero in any case.

Rik van Reil's patch to split the active and inactive lists can address the non-swappable page problem by throttling the scan of the anon LRU lists, that contain both anon and shmem pages. However, if the system supports any swap space at all, one still needs to scan the anon lists to free up memory consumed by pages already in the swap cache. On large memory systems, the anon lists can still be millions of pages long and contain a large per centage of non-swappable and mlocked pages.

This series attempts to unify this work into a general mechanism for managing non-reclaimable pages. The basic objective is to make vmscan as productive as possible on very large memory systems, by eliminating non-productive page shuffling.

Like Larry's patch, the noreclaim infrastructure maintains "non-reclaimable" pages on a separate per-zone list. This noreclaim list is, conceptually, another LRU list--a sibling of the active and inactive lists. A page on the noreclaim list will have the PG_lru and PG_noreclaim flags set. The PG_noreclaim flag is analogous to, and mutually exclusive with, the PG_active flag--it specifies which LRU list the page resides on. The noreclaim list supports a pagevec cache, like the active and inactive lists to reduce contention on the zone lru lock in vmscan and in the fault path.

Pages on the noreclaim list are "hidden" from page reclaim scanning. Thus, reclaim will not spend time attempting to reclaim the pages, only to find that they can't be unmapped, have no swap space available, are locked into memory, ... However, vmscan may find pages on the [in]active lists that have become non-reclaimable since they were put on the list. It will move them to the noreclaim list at that time.

This series of patches includes the basic noreclaim list support and one patch, as a proof of concept, to address the 3rd condition listed above:
the excessively long anon_vma list of related vmas. This seemed to be the easiest of the 3 conditions to address, and I have a test case handy [AIM7--see below]. Additional patches to handle anon pages for which no swap exists and to layer Nick Piggin's patch to keep "mlock pages off the LRU" will be forthcoming, if feedback indicates that this approach is worth pursuing.

Now, about those anon pages with really long "related vma" lists:
We have only seen this in AIM7 benchmarks on largish servers. The situation occurs when a single task fork()s many [10s of] thousands of children, and the the system needs to reclaim memory. We've seen all processors on a system spinning on the anon_vma lock attempting to unmap pages mapped by these thousands of children--for 10s of minutes or until we give up and reboot.

I discussed this issue at LCA'07 in a kernel miniconf presentation. Linus questioned whether this was a problem that really needs solving. After all, AIM7 is only a synthetic benchmark. Does any real application
behave this way? After the presentation, someone came up to me and told
me that Apache also fork()s for each incoming connection and can fork thousands of children. However, I have not witnessed this, nor do I know how long lived these children are.

I have included another patch that makes the anon_vma lock a reader/write lock. This allows different cpus to attempt to reclaim, in parallel, different pages that point to the same anon_vma. However, this doesn't solve the problem of trying to unmap pages that are [potentially] mapped into thousands of vmas.

The last patch in this series counts the number of related vmas on an anon_vma's list and, when it exceeds a tunable threshold, pages that reference that anon_vma are declared nonreclaimable. We detect these non-reclaimable pages either on fault [COW or new anon page in a vma with an excessively shared anon_vma] or when vmscan encounters such a page on the LRU list.

The patch/series does not [yet] support moving such a page back to the [in]active lists when it's anon_vma sharing drops below the threshold. This usually occurs when a task exits or explicitly unmapps the area. Any COWed private pages will be freed at this time, but anon pages that are still shared will remain nonreclaimable even though the related vma count is below the no-reclaim limit. Again, I will address this if the overall approach is deemed worth pursuing.

Additional considerations:

If the noreclaim list contains mlocked pages, they can be directly deleted from the noreclaim list without scanning when the become unlocked. But, note that we can't use one of the lru link fields to contain the mlocked vma count in this case.

If the noreclaim list contains anon/shmem pages for which no swap space exists, it will be necessary to scan the list when swap space becomes available, either because it has been freed from other pages, or because additional swap has been added. The latter case should not occur frequently enough to be a problem. We should be able to defer the scanning when swap space is freed from other pages until a sufficient number become available or system is under severe pressure.

If the list contains pages that are merely difficult to reclaim because of the excessive anon_vma sharing, and if we want to make them reclaimable again when the anon_vma related vma count drops to an acceptable value, one would have to scan the list at some point. Again, this could be deferred until there are a sufficient number of such pages to make it worth while or until the system is under severe memory pressure.

The above considerations suggest that one consider separate lists for
non-reclaimable [no swap, mlocked] and difficult to reclaim. Or,
maybe not...

Interaction of noreclaim list and LRU lists: My current patch moves pages to the noreclaim list as soon as they are detected, either on the active or inactive list. I could change this such that non-reclaimable pages found on the active list go to the inactive list first, and take a ride there before being declared non-reclaimable. However, we still have the issue of where to place the pages when then come off the no reclaim list: back to the active list? the inactive list? head or tail thereof? My current mechanism, with the PG_active and PG_noreclaim flags being mutually exclusive, does not track activeness of pages on the noreclaim list. To do so would require additional scanning of the list, I think, sort of defeating the purpose of the list. But, maybe acceptable if we scan just to test/modify the active flags.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Patch against 2.6.21-rc5/6

Infrastructure to manage pages excluded from reclaim--i.e., hidden from vmscan. Based on a patch by Larry Woodman of Red Hat.

Applies atop two patches from Nick Piggin's "mlock pages off the LRU"
series: move-and-rework-isolate_lru_page and move-and-rename-install_arg_page

Maintain "nonreclaimable" pages on a separate per-zone list, to "hide" them from vmscan.

Although this patch series does not support it, the noreclaim list could be scanned at a lower rate--for example to attempt to reclaim the "difficult to reclaim" pages when pages are REALLY needed, such as when reserves are exhausted and a critical need arises.

A new function 'page_reclaimable(page, vma)' in vmscan.c tests whether or not a page is reclaimable. Subsequent patches will add the various !reclaimable tests. Reclaimable pages are placed on the appropriate LRU list; non-reclaimable pages on the new noreclaim list.

Notes:

1. Not sure I need the 'vma' arg to page_reclaimable(). I did in an earlier incarnation. Don't seem to now

2. for now, use bit 20 in page flags. Could restrict to 64-bit
systems only and use one of bits 21-30 [ia64 uses bit 31; other archs ???].

 include/linux/mm_inline.h  |   34 +++++++++++++++++++-
 include/linux/mmzone.h     |    6 +++
 include/linux/page-flags.h |   20 ++++++++++++
 include/linux/pagevec.h    |    5 +++
 include/linux/swap.h       |   11 ++++++
 mm/Kconfig                 |    8 ++++
 mm/mempolicy.c             |    2 -
 mm/migrate.c               |    8 ++++
 mm/page_alloc.c            |    6 +++
 mm/swap.c                  |   73 +++++++++++++++++++++++++++++++++++++++----
 mm/vmscan.c                |   75 ++++++++++++++++++++++++++++++++++++++++++++-
 11 files changed, 237 insertions(+), 11 deletions(-)

Index: Linux/mm/Kconfig
===================================================================
@@ -163,3 +163,11 @@ config ZONE_DMA_FLAG
 	default "0" if !ZONE_DMA
 	default "1"
 
+config NORECLAIM
+	bool "Track non-reclaimable pages"
+	help
+	  Supports tracking of non-reclaimable pages off the [in]active lists
+	  to avoid excessive reclaim overhead on large memory systems.  Pages
+	  may be non-reclaimable because:  they are locked into memory, they
+	  are anonymous pages for which no swap space exists, or they are anon
+	  pages that are expensive to unmap [long anon_vma "related vma" list.]

Index: Linux/include/linux/page-flags.h
===================================================================
@@ -91,6 +91,9 @@
 #define PG_nosave_free		18	/* Used for system suspend/resume */
 #define PG_buddy		19	/* Page is free, on buddy lists */
 
+#define PG_noreclaim		20	/* Page is "non-reclaimable"  */
+
+
 /* PG_owner_priv_1 users should have descriptive aliases */
 #define PG_checked		PG_owner_priv_1 /* Used by some filesystems */
 

@@ -249,6 +252,23 @@ static inline void SetPageUptodate(struc
 #define PageSwapCache(page)	0
 #endif
 
+#ifdef CONFIG_NORECLAIM
+#define PageNoreclaim(page)	test_bit(PG_noreclaim, &(page)->flags)
+#define SetPageNoreclaim(page)	set_bit(PG_noreclaim, &(page)->flags)
+#define ClearPageNoreclaim(page) clear_bit(PG_noreclaim, &(page)->flags)
+#define __ClearPageNoreclaim(page) __clear_bit(PG_noreclaim, &(page)->flags)
+//TODO:   need test versions?
+#define TestSetPageNoreclaim(page) \
+				test_and_set_bit(PG_noreclaim, &(page)->flags)
+#define TestClearPageNoreclaim(page) \
+				test_and_clear_bit(PG_noreclaim, &(page)->flags)
+#else
+#define PageNoreclaim(page)	0
+#define SetPageNoreclaim(page)
+#define ClearPageNoreclaim(page)
+#define __ClearPageNoreclaim(page)
+#endif
+
 #define PageUncached(page)	test_bit(PG_uncached, &(page)->flags)
 #define SetPageUncached(page)	set_bit(PG_uncached, &(page)->flags)
 #define ClearPageUncached(page)	clear_bit(PG_uncached, &(page)->flags)

Index: Linux/include/linux/mmzone.h
===================================================================
@@ -51,6 +51,9 @@ enum zone_stat_item {
 	NR_FREE_PAGES,
 	NR_INACTIVE,
 	NR_ACTIVE,
+#ifdef CONFIG_NORECLAIM
+	NR_NORECLAIM,
+#endif
 	NR_ANON_PAGES,	/* Mapped anonymous pages */
 	NR_FILE_MAPPED,	/* pagecache pages mapped into pagetables.
 			   only modified from process context */

@@ -217,6 +220,9 @@ struct zone {
 	spinlock_t		lru_lock;	
 	struct list_head	active_list;
 	struct list_head	inactive_list;
+#ifdef CONFIG_NORECLAIM
+	struct list_head	noreclaim_list;
+#endif
 	unsigned long		nr_scan_active;
 	unsigned long		nr_scan_inactive;
 	unsigned long		pages_scanned;	   /* since last reclaim */

Index: Linux/mm/page_alloc.c
===================================================================
@@ -198,6 +198,7 @@ static void bad_page(struct page *page)
 			1 << PG_private |
 			1 << PG_locked	|
 			1 << PG_active	|
+			1 << PG_noreclaim	|
 			1 << PG_dirty	|
 			1 << PG_reclaim |
 			1 << PG_slab    |

@@ -433,6 +434,7 @@ static inline int free_pages_check(struc
 			1 << PG_private |
 			1 << PG_locked	|
 			1 << PG_active	|
+			1 << PG_noreclaim	|
 			1 << PG_reclaim	|
 			1 << PG_slab	|
 			1 << PG_swapcache |

@@ -582,6 +584,7 @@ static int prep_new_page(struct page *pa
 			1 << PG_private	|
 			1 << PG_locked	|
 			1 << PG_active	|
+			1 << PG_noreclaim	|
 			1 << PG_dirty	|
 			1 << PG_reclaim	|
 			1 << PG_slab    |

@@ -2673,6 +2676,9 @@ static void __meminit free_area_init_cor
 		zone_pcp_init(zone);
 		INIT_LIST_HEAD(&zone->active_list);
 		INIT_LIST_HEAD(&zone->inactive_list);
+#ifdef CONFIG_NORECLAIM
+		INIT_LIST_HEAD(&zone->noreclaim_list);
+#endif
 		zone->nr_scan_active = 0;
 		zone->nr_scan_inactive = 0;
 		zap_zone_vm_stats(zone);

Index: Linux/include/linux/mm_inline.h
===================================================================
@@ -26,11 +26,43 @@ del_page_from_inactive_list(struct zone 
 	__dec_zone_state(zone, NR_INACTIVE);
 }
 
+#ifdef CONFIG_NORECLAIM
+static inline void __dec_zone_noreclaim(struct zone *zone)
+{
+	__dec_zone_state(zone, NR_NORECLAIM);
+}
+
+static inline void
+add_page_to_noreclaim_list(struct zone *zone, struct page *page)
+{
+	list_add(&page->lru, &zone->noreclaim_list);
+	__inc_zone_state(zone, NR_NORECLAIM);
+}
+
+static inline void
+del_page_from_noreclaim_list(struct zone *zone, struct page *page)
+{
+	list_del(&page->lru);
+	__dec_zone_noreclaim(zone);
+}
+#else
+static inline void __dec_zone_noreclaim(struct zone *zone) { }
+
+static inline void
+add_page_to_noreclaim_list(struct zone *zone, struct page *page) { }
+
+static inline void
+del_page_from_noreclaim_list(struct zone *zone, struct page *page) { }
+#endif
+
 static inline void
 del_page_from_lru(struct zone *zone, struct page *page)
 {
 	list_del(&page->lru);
-	if (PageActive(page)) {
+	if (PageNoreclaim(page)) {
+		__ClearPageNoreclaim(page);
+		__dec_zone_noreclaim(zone);
+	} else if (PageActive(page)) {
 		__ClearPageActive(page);
 		__dec_zone_state(zone, NR_ACTIVE);
 	} else {

Index: Linux/include/linux/swap.h
===================================================================
@@ -186,6 +186,11 @@ extern void lru_add_drain(void);
 extern int lru_add_drain_all(void);
 extern int rotate_reclaimable_page(struct page *page);
 extern void swap_setup(void);
+#ifdef CONFIG_NORECLAIM
+extern void FASTCALL(lru_cache_add_noreclaim(struct page *page));
+#else
+static inline void lru_cache_add_noreclaim(struct page *page) { }
+#endif
 
 /* linux/mm/vmscan.c */
 extern unsigned long try_to_free_pages(struct zone **, gfp_t);

@@ -207,6 +212,12 @@ static inline int zone_reclaim(struct zo
 }
 #endif
 
+#ifdef CONFIG_NORECLAIM
+extern int page_reclaimable(struct page *page, struct vm_area_struct *vma);
+#else
+#define page_reclaimable(P, V) 1
+#endif
+
 extern int kswapd_run(int nid);
 
 #ifdef CONFIG_MMU

Index: Linux/include/linux/pagevec.h
===================================================================
@@ -25,6 +25,11 @@ void __pagevec_release_nonlru(struct pag
 void __pagevec_free(struct pagevec *pvec);
 void __pagevec_lru_add(struct pagevec *pvec);
 void __pagevec_lru_add_active(struct pagevec *pvec);
+#ifdef CONFIG_NORECLAIM
+void __pagevec_lru_add_noreclaim(struct pagevec *pvec);
+#else
+static inline void __pagevec_lru_add_noreclaim(struct pagevec *pvec) { }
+#endif
 void pagevec_strip(struct pagevec *pvec);
 unsigned pagevec_lookup(struct pagevec *pvec, struct address_space *mapping,
 		pgoff_t start, unsigned nr_pages);

Index: Linux/mm/migrate.c
===================================================================
@@ -52,7 +52,10 @@ int migrate_prep(void)
 
 static inline void move_to_lru(struct page *page)
 {
-	if (PageActive(page)) {
+	if (PageNoreclaim(page)) {
+		ClearPageNoreclaim(page);
+		lru_cache_add_noreclaim(page);
+	} else if (PageActive(page)) {
 		/*
 		 * lru_cache_add_active checks that
 		 * the PG_active bit is off.

@@ -322,6 +325,9 @@ static void migrate_page_copy(struct pag
 		SetPageUptodate(newpage);
 	if (PageActive(page))
 		SetPageActive(newpage);
+	else
+		if (PageNoreclaim(page))
+			SetPageNoreclaim(newpage);
 	if (PageChecked(page))
 		SetPageChecked(newpage);
 	if (PageMappedToDisk(page))

Index: Linux/mm/mempolicy.c
===================================================================
@@ -1790,7 +1790,7 @@ static void gather_stats(struct page *pa
 	if (PageSwapCache(page))
 		md->swapcache++;
 
-	if (PageActive(page))
+	if (PageActive(page) || PageNoreclaim(page))
 		md->active++;
 
 	if (PageWriteback(page))

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Patch against 2.6.21-rc5

n/m in the noreclaim series

Report non-reclaimable pages per zone and system wide.

 drivers/base/node.c |    6 ++++++
 fs/proc/proc_misc.c |    6 ++++++
 mm/page_alloc.c     |   16 +++++++++++++++-
 mm/vmstat.c         |    3 +++
 4 files changed, 30 insertions(+), 1 deletion(-)

Index: Linux/mm/page_alloc.c
===================================================================
@@ -1574,10 +1574,18 @@ void show_free_areas(void)
 		}
 	}
 
-	printk("Active:%lu inactive:%lu dirty:%lu writeback:%lu unstable:%lu\n"
+//TODO:  check/adjust line lengths
+	printk("Active:%lu inactive:%lu"
+#ifdef CONFIG_NORECLAIM
+		" noreclaim:%lu"
+#endif
+		" dirty:%lu writeback:%lu unstable:%lu\n"
 		" free:%lu slab:%lu mapped:%lu pagetables:%lu bounce:%lu\n",
 		global_page_state(NR_ACTIVE),
 		global_page_state(NR_INACTIVE),
+#ifdef CONFIG_NORECLAIM
+		global_page_state(NR_NORECLAIM),
+#endif
 		global_page_state(NR_FILE_DIRTY),
 		global_page_state(NR_WRITEBACK),
 		global_page_state(NR_UNSTABLE_NFS),

@@ -1602,6 +1610,9 @@ void show_free_areas(void)
 			" high:%lukB"
 			" active:%lukB"
 			" inactive:%lukB"
+#ifdef CONFIG_NORECLAIM
+			" noreclaim:%lukB"
+#endif
 			" present:%lukB"
 			" pages_scanned:%lu"
 			" all_unreclaimable? %s"

@@ -1613,6 +1624,9 @@ void show_free_areas(void)
 			K(zone->pages_high),
 			K(zone_page_state(zone, NR_ACTIVE)),
 			K(zone_page_state(zone, NR_INACTIVE)),
+#ifdef CONFIG_NORECLAIM
+			K(zone_page_state(zone, NR_NORECLAIM)),
+#endif
 			K(zone->present_pages),
 			zone->pages_scanned,
 			(zone->all_unreclaimable ? "yes" : "no")

Index: Linux/mm/vmstat.c
===================================================================
@@ -434,6 +434,9 @@ static const char * const vmstat_text[] 
 	"nr_free_pages",
 	"nr_active",
 	"nr_inactive",
+#ifdef CONFIG_NORECLAIM
+	"nr_noreclaim",
+#endif
 	"nr_anon_pages",
 	"nr_mapped",
 	"nr_file_pages",

Index: Linux/drivers/base/node.c
===================================================================
@@ -49,6 +49,9 @@ static ssize_t node_read_meminfo(struct 
 		       "Node %d MemUsed:      %8lu kB\n"
 		       "Node %d Active:       %8lu kB\n"
 		       "Node %d Inactive:     %8lu kB\n"
+#ifdef CONFIG_NORECLAIM
+		       "Node %d Noreclaim:    %8lu kB\n"
+#endif
 #ifdef CONFIG_HIGHMEM
 		       "Node %d HighTotal:    %8lu kB\n"
 		       "Node %d HighFree:     %8lu kB\n"

@@ -71,6 +74,9 @@ static ssize_t node_read_meminfo(struct 
 		       nid, K(i.totalram - i.freeram),
 		       nid, node_page_state(nid, NR_ACTIVE),
 		       nid, node_page_state(nid, NR_INACTIVE),
+#ifdef CONFIG_NORECLAIM
+		       nid, node_page_state(nid, NR_NORECLAIM),
+#endif
 #ifdef CONFIG_HIGHMEM
 		       nid, K(i.totalhigh),
 		       nid, K(i.freehigh),

Index: Linux/fs/proc/proc_misc.c
===================================================================
@@ -154,6 +154,9 @@ static int meminfo_read_proc(char *page,
 		"SwapCached:   %8lu kB\n"
 		"Active:       %8lu kB\n"
 		"Inactive:     %8lu kB\n"
+#ifdef CONFIG_NORECLAIM
+		"Noreclaim:    %8lu kB\n"
+#endif
 #ifdef CONFIG_HIGHMEM
 		"HighTotal:    %8lu kB\n"
 		"HighFree:     %8lu kB\n"

@@ -184,6 +187,9 @@ static int meminfo_read_proc(char *page,
 		K(total_swapcache_pages),
 		K(global_page_state(NR_ACTIVE)),
 		K(global_page_state(NR_INACTIVE)),
+#ifdef CONFIG_NORECLAIM
+		K(global_page_state(NR_NORECLAIM)),
+#endif
 #ifdef CONFIG_HIGHMEM
 		K(i.totalhigh),
 		K(i.freehigh),

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Here's the last one for now. I have a couple more in this series that handle the swap-backed w/ no swap space avail, but that's a different topic, right?

----

Patch m/n against 2.6.21-rc5 - track anon_vma "related vmas" == list length

When a single parent forks a large number [thousands, 10s of thousands] of children, the anon_vma list of related vmas becomes very long. In reclaim, this list must be traversed twice--once in page_referenced_anon() and once in try_to_unmap_anon()--under a spin lock to reclaim the page. Multiple cpus can end up spinning behind the same anon_vma spinlock and traversing the lists. This patch, part of the "noreclaim" series, treats anon pages with list lengths longer than a tunable threshold as non- reclaimable.

1) add mm Kconfig option NORECLAIM_ANON_VMA, dependent on NORECLAIM. 32-bit systems may not want/need this features.

2) add a counter of related vmas to the anon_vma structure. This won't increase the size of the structure on 64-bit systems, as it will fit in a padding slot.

3) In [__]anon_vma_[un]link(), track number of related vmas. The count is only incremented/decremented while the anon_vma lock is held, so regular, non-atomic, increment/decrement is used.

4) in page_reclaimable(), check anon_vma count in vma's anon_vma, if vma supplied, or in page's anon_vma. In fault path, new anon pages are placed on the LRU before adding the anon rmap, so we need to check the vma's anon_vma. Fortunately, the vma is available at that point. In vmscan, we can just check the page's anon_vma for any anon pages that made it onto the [in]active list before the anon_vma list length became "excessive".

5) make the threshold tunable via /proc/sys/vm/anon_vma_reclaim_limit. Default value of 64 is totally arbitrary, but should be high enough that most applications won't hit it.

6) In the fault paths that install new anonymous pages, check whether the page is reclaimable or not [#4 above]. If it is, just add it to the active lru list [via the pagevec cache], else add it to the noreclaim list.

Notes:

1) a separate patch makes the anon_vma lock a reader/writer lock. This allows some parallelism--different cpus can work on different pages that reference the same anon_vma--but this does not address the problem of long lists and potentially many pte's to unmap.

2) I moved the call to page_add_new_anon_rmap() to before the test for page_reclaimable() and thus before the calls to lru_cache_add_{active|noreclaim}(), so that page_reclaimable() could recognize the page as anon, thus obviating, I think, the vma arg to page_reclaimable(). TBD I think this reordering is OK, but the previous order may have existed to close some obscure race?

 include/linux/rmap.h   |   57 ++++++++++++++++++++++++++++++++++++++++++++++++-
 include/linux/swap.h   |    3 ++
 include/linux/sysctl.h |    1 
 kernel/sysctl.c        |   12 ++++++++++
 mm/Kconfig             |   11 +++++++++
 mm/memory.c            |   20 +++++++++++++----
 mm/rmap.c              |    9 ++++++-
 mm/vmscan.c            |   22 +++++++++++++++++-
 8 files changed, 127 insertions(+), 8 deletions(-)

Index: Linux/mm/Kconfig
===================================================================
@@ -171,3 +171,14 @@ config NORECLAIM
 	  may be non-reclaimable because:  they are locked into memory, they
 	  are anonymous pages for which no swap space exists, or they are anon
 	  pages that are expensive to unmap [long anon_vma "related vma" list.]
+
+config NORECLAIM_ANON_VMA
+	bool "Exclude pages with excessively long anon_vma lists"
+	depends on NORECLAIM
+	help
+	  Treats anonymous pages with excessively long anon_vma lists as
+	  non-reclaimable.  Long anon_vma lists results from fork()ing
+	  many [hundreds, thousands] of children from a single parent.  The
+	  anonymous pages in such tasks are very expensive [sometimes almost
+	  impossible] to reclaim.  Treating them as non-reclaimable avoids
+	  the overhead of attempting to reclaim them.

Index: Linux/mm/rmap.c
===================================================================
@@ -99,6 +99,7 @@ int anon_vma_prepare(struct vm_area_stru
 		if (likely(!vma->anon_vma)) {
 			vma->anon_vma = anon_vma;
 			list_add_tail(&vma->anon_vma_node, &anon_vma->head);
+			add_related_vma(anon_vma);
 			allocated = NULL;
 		}
 		spin_unlock(&mm->page_table_lock);

@@ -113,8 +114,11 @@ int anon_vma_prepare(struct vm_area_stru
 
 void __anon_vma_merge(struct vm_area_struct *vma, struct vm_area_struct *next)
 {
-	BUG_ON(vma->anon_vma != next->anon_vma);
+	struct anon_vma *anon_vma = vma->anon_vma;
+
+	BUG_ON(anon_vma != next->anon_vma);
 	list_del(&next->anon_vma_node);
+	remove_related_vma(anon_vma);
 }
 
 void __anon_vma_link(struct vm_area_struct *vma)

@@ -123,6 +127,7 @@ void __anon_vma_link(struct vm_area_stru
 
 	if (anon_vma) {
 		list_add_tail(&vma->anon_vma_node, &anon_vma->head);
+		add_related_vma(anon_vma);
 		validate_anon_vma(vma);
 	}
 }

@@ -134,6 +139,7 @@ void anon_vma_link(struct vm_area_struct
 	if (anon_vma) {
 		write_lock(&anon_vma->rwlock);
 		list_add_tail(&vma->anon_vma_node, &anon_vma->head);
+		add_related_vma(anon_vma);
 		validate_anon_vma(vma);
 		write_unlock(&anon_vma->rwlock);
 	}

@@ -150,6 +156,7 @@ void anon_vma_unlink(struct vm_area_stru
 	write_lock(&anon_vma->rwlock);
 	validate_anon_vma(vma);
 	list_del(&vma->anon_vma_node);
+	remove_related_vma(anon_vma);
 
 	/* We must garbage collect the anon_vma if it's empty */
 	empty = list_empty(&anon_vma->head);

Index: Linux/include/linux/swap.h
===================================================================
@@ -214,6 +214,9 @@ static inline int zone_reclaim(struct zo
 
 #ifdef CONFIG_NORECLAIM
 extern int page_reclaimable(struct page *page, struct vm_area_struct *vma);
+#ifdef CONFIG_NORECLAIM_ANON_VMA
+extern int anon_vma_reclaim_limit;
+#endif
 #else
 #define page_reclaimable(P, V) 1
 #endif

Index: Linux/mm/vmscan.c
===================================================================
@@ -1812,6 +1812,10 @@ int zone_reclaim(struct zone *zone, gfp_
 #endif
 
 #ifdef CONFIG_NORECLAIM
+
+#ifdef CONFIG_NORECLAIM_ANON_VMA
+int anon_vma_reclaim_limit = DEFAULT_ANON_VMA_RECLAIM_LIMIT;
+#endif
 /*
  * page_reclaimable(struct page *page, struct vm_area_struct *vma)
  * Test whether page is reclaimable--i.e., should be placed on active/inactive

@@ -1822,7 +1826,8 @@ int zone_reclaim(struct zone *zone, gfp_
  *               If !NULL, called from fault path.
  *
  * Reasons page might not be reclaimable:
- * TODO - later patches
+ * 1) anon_vma [if any] has too many related vmas
+ * [more TBD.  e.g., anon page and no swap available, page mlocked, ...]
  *
  * TODO:  specify locking assumptions
  */

@@ -1832,7 +1837,20 @@ int page_reclaimable(struct page *page, 
 
 	VM_BUG_ON(PageNoreclaim(page));
 
-	/* TODO:  test page [!]reclaimable conditions */
+#ifdef CONFIG_NORECLAIM_ANON_VMA
+	if (PageAnon(page)) {
+		struct anon_vma *anon_vma;
+
+		/*
+		 * anon page with too many related vmas?
+		 */
+		anon_vma = page_anon_vma(page);
+		VM_BUG_ON(!anon_vma);
+		if (anon_vma_reclaim_limit &&
+			anon_vma->count > anon_vma_reclaim_limit)
+			reclaimable = 0;
+	}
+#endif
 
 	return reclaimable;
 }

Index: Linux/include/linux/sysctl.h
===================================================================
@@ -207,6 +207,7 @@ enum
 	VM_PANIC_ON_OOM=33,	/* panic at out-of-memory */
 	VM_VDSO_ENABLED=34,	/* map VDSO into new processes? */
 	VM_MIN_SLAB=35,		 /* Percent pages ignored by zone reclaim */
+	VM_ANON_VMA_RECLAIM_LIMIT=36, /* max "related vmas" for reclaim  */
 
 	/* s390 vm cmm sysctls */
 	VM_CMM_PAGES=1111,

Index: Linux/kernel/sysctl.c
===================================================================
@@ -859,6 +859,18 @@ static ctl_table vm_table[] = {
 		.extra1		= &zero,
 	},
 #endif
+#ifdef CONFIG_NORECLAIM_ANON_VMA
+	{
+		.ctl_name	= VM_ANON_VMA_RECLAIM_LIMIT,
+		.procname	= "anon_vma_reclaim_limit",
+		.data		= &anon_vma_reclaim_limit,
+		.maxlen		= sizeof(anon_vma_reclaim_limit),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec,
+		.strategy	= &sysctl_intvec,
+		.extra1		= &zero,
+	},
+#endif
 	{ .ctl_name = 0 }
 };
 

Index: Linux/mm/memory.c
===================================================================
@@ -1650,8 +1650,11 @@ gotten:
 		ptep_clear_flush(vma, address, page_table);
 		set_pte_at(mm, address, page_table, entry);
 		update_mmu_cache(vma, address, entry);
-		lru_cache_add_active(new_page);
 		page_add_new_anon_rmap(new_page, vma, address);
+		if (page_reclaimable(new_page, vma))
+			lru_cache_add_active(new_page);
+		else
+			lru_cache_add_noreclaim(new_page);
 
 		/* Free the old page.. */
 		new_page = old_page;

@@ -2149,8 +2152,11 @@ int install_new_anon_page(struct vm_area
 	inc_mm_counter(mm, anon_rss);
 	set_pte_at(mm, address, pte, pte_mkdirty(pte_mkwrite(mk_pte(
 					page, vma->vm_page_prot))));
-	lru_cache_add_active(page);
 	page_add_new_anon_rmap(page, vma, address);
+	if (page_reclaimable(page, vma))
+		lru_cache_add_active(page);
+	else
+		lru_cache_add_noreclaim(page);
 	pte_unmap_unlock(pte, ptl);
 
 	/* no need for flush_tlb */

@@ -2187,8 +2193,11 @@ static int do_anonymous_page(struct mm_s
 		if (!pte_none(*page_table))
 			goto release;
 		inc_mm_counter(mm, anon_rss);
-		lru_cache_add_active(page);
 		page_add_new_anon_rmap(page, vma, address);
+		if (page_reclaimable(page, vma))
+			lru_cache_add_active(page);
+		else
+			lru_cache_add_noreclaim(page);
 	} else {
 		/* Map the ZERO_PAGE - vm_page_prot is readonly */
 		page = ZERO_PAGE(address);

@@ -2334,8 +2343,11 @@ retry:
 		set_pte_at(mm, address, page_table, entry);
 		if (anon) {
 			inc_mm_counter(mm, anon_rss);
-			lru_cache_add_active(new_page);
 			page_add_new_anon_rmap(new_page, vma, address);
+			if (page_reclaimable(new_page, vma))
+				lru_cache_add_active(new_page);
+			else
+				lru_cache_add_noreclaim(new_page);
 		} else {
 			inc_mm_counter(mm, file_rss);
 			page_add_file_rmap(new_page);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>