Skip to contentSkip to conversation

memcg: per cgroup background reclaim

RFC PATCH In review Ying Han 6 participants 52 messages Nov 30, 2010 – Dec 8, 2010

Subsystems (7)
lore
Share & tools
lore

Ying Han

author
#0 lore raw

cover memcg: per cgroup background reclaim

The current implementation of memcg only supports direct reclaim and this patchset adds the support for background reclaim. Per cgroup background reclaim is needed which spreads out the memory pressure over longer period of time and smoothes out the system performance.

The current implementation is not a stable version, and it crashes sometimes on my NUMA machine. Before going further for debugging, I would like to start the discussion and hear the feedbacks of the initial design.

Current status:
I run through some simple tests which reads/writes a large file and makes sure it triggers per cgroup kswapd on the low_wmark. Also, I compared at pg_steal/pg_scan ratio w/o background reclaim.

Step1: Create a cgroup with 500M memory_limit and set the min_free_kbytes to 1024.

$ mount -t cgroup -o cpuset,memory cpuset /dev/cgroup
$ mkdir /dev/cgroup/A
$ echo 0 >/dev/cgroup/A/cpuset.cpus
$ echo 0 >/dev/cgroup/A/cpuset.mems
$ echo 500m >/dev/cgroup/A/memory.limit_in_bytes
$ echo 1024 >/dev/cgroup/A/memory.min_free_kbytes
$ echo $$ >/dev/cgroup/A/tasks

Step2: Check the wmarks.

$ cat /dev/cgroup/A/memory.reclaim_wmarks
memcg_low_wmark 98304000
memcg_high_wmark 81920000

Step3: Dirty the pages by creating a 20g file on hard drive.

$ ddtest -D /export/hdc3/dd -b 1024 -n 20971520 -t 1

Checked the memory.stat w/o background reclaim. It used to be all the pages are reclaimed from direct reclaim, and now about half of them are reclaimed at background. (note: writing '0' to min_free_kbytes disables per cgroup kswapd)

Only direct reclaim With background reclaim:
kswapd_steal 0 kswapd_steal 2751822
pg_pgsteal 5100401 pg_pgsteal 2476676
kswapd_pgscan 0 kswapd_pgscan 6019373
pg_scan 5542464 pg_scan 3851281
pgrefill 304505 pgrefill 348077
pgoutrun 0 pgoutrun 44568
allocstall 159278 allocstall 75669

Step4: Cleanup

$ echo $$ >/dev/cgroup/tasks
$ echo 0 > /dev/cgroup/A/memory.force_empty

Step5: Read the 20g file into the pagecache.

$ cat /export/hdc3/dd/tf0 > /dev/zero;

Checked the memory.stat w/o background reclaim. All the clean pages are reclaimed at background instead of direct reclaim.

Only direct reclaim With background reclaim
kswapd_steal 0 kswapd_steal 3512424
pg_pgsteal 3461280 pg_pgsteal 0
kswapd_pgscan 0 kswapd_pgscan 3512440
pg_scan 3461280 pg_scan 0
pgrefill 0 pgrefill 0
pgoutrun 0 pgoutrun 74973
allocstall 108165 allocstall 0

Ying Han (4):
Add kswapd descriptor.
Add per cgroup reclaim watermarks.
Per cgroup background reclaim.
Add more per memcg stats.

 include/linux/memcontrol.h  |  112 +++++++++++
 include/linux/mmzone.h      |    3 +-
 include/linux/res_counter.h |   88 +++++++++-
 include/linux/swap.h        |   10 +
 kernel/res_counter.c        |   26 ++-
 mm/memcontrol.c             |  447 ++++++++++++++++++++++++++++++++++++++++++-
 mm/mmzone.c                 |    2 +-
 mm/page_alloc.c             |   11 +-
 mm/vmscan.c                 |  346 ++++++++++++++++++++++++++++++----
 9 files changed, 994 insertions(+), 51 deletions(-)
Signature · 8 lines
1.7.3.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#1 lore raw

PATCH 1/4 Add kswapd descriptor.

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup. The kswapd is sleeping in the wait queue headed at kswapd_wait field of a kswapd descriptor. The kswapd descriptor stores information of node or cgroup and it allows the global and per cgroup background reclaim to share common reclaim algorithms.

This patch addes the kswapd descriptor and changes per zone kswapd_wait to the common data structure.

---
 include/linux/mmzone.h |    3 +-
 include/linux/swap.h   |   10 +++++
 mm/memcontrol.c        |    2 +
 mm/mmzone.c            |    2 +-
 mm/page_alloc.c        |    9 +++-
 mm/vmscan.c            |   98 +++++++++++++++++++++++++++++++++--------------
 6 files changed, 90 insertions(+), 34 deletions(-)
include/linux/mmzone.h +1 -2
index 39c24eb..c77dfa2 100644
@@ -642,8 +642,7 @@ typedef struct pglist_data {
 	unsigned long node_spanned_pages; /* total size of physical page
 					     range, including holes */
 	int node_id;
-	wait_queue_head_t kswapd_wait;
-	struct task_struct *kswapd;
+	wait_queue_head_t *kswapd_wait;
 	int kswapd_max_order;
 } pg_data_t;
 
include/linux/swap.h +10 -0
index eba53e7..2e6cb58 100644
@@ -26,6 +26,16 @@ static inline int current_is_kswapd(void)
 	return current->flags & PF_KSWAPD;
 }
 
+struct kswapd {
+	struct task_struct *kswapd_task;
+	wait_queue_head_t kswapd_wait;
+	struct mem_cgroup *kswapd_mem;
+	pg_data_t *kswapd_pgdat;
+};
+
+#define MAX_KSWAPDS MAX_NUMNODES
+extern struct kswapd kswapds[MAX_KSWAPDS];
+int kswapd(void *p);
 /*
  * MAX_SWAPFILES defines the maximum number of swaptypes: things which can
  * be swapped to.  The swap type and the offset into that swap type are
mm/memcontrol.c +2 -0
index a4034b6..dca3590 100644
@@ -263,6 +263,8 @@ struct mem_cgroup {
 	 */
 	struct mem_cgroup_stat_cpu nocpu_base;
 	spinlock_t pcp_counter_lock;
+
+	wait_queue_head_t *kswapd_wait;
 };
 
 /* Stuffs for move charges at task migration. */
mm/mmzone.c +1 -1
index e35bfb8..c7cbed5 100644
@@ -102,7 +102,7 @@ unsigned long zone_nr_free_pages(struct zone *zone)
 	 * free pages are low, get a better estimate for free pages
 	 */
 	if (nr_free_pages < zone->percpu_drift_mark &&
-			!waitqueue_active(&zone->zone_pgdat->kswapd_wait))
+			!waitqueue_active(zone->zone_pgdat->kswapd_wait))
 		return zone_page_state_snapshot(zone, NR_FREE_PAGES);
 
 	return nr_free_pages;
mm/page_alloc.c +7 -2
index b48dea2..a15bc1c 100644
@@ -4070,13 +4070,18 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat,
 	int nid = pgdat->node_id;
 	unsigned long zone_start_pfn = pgdat->node_start_pfn;
 	int ret;
+	struct kswapd *kswapd_p;
 
 	pgdat_resize_init(pgdat);
 	pgdat->nr_zones = 0;
-	init_waitqueue_head(&pgdat->kswapd_wait);
 	pgdat->kswapd_max_order = 0;
 	pgdat_page_cgroup_init(pgdat);
-	
+
+	kswapd_p = &kswapds[nid];
+	init_waitqueue_head(&kswapd_p->kswapd_wait);
+	pgdat->kswapd_wait = &kswapd_p->kswapd_wait;
+	kswapd_p->kswapd_pgdat = pgdat;
+
 	for (j = 0; j < MAX_NR_ZONES; j++) {
 		struct zone *zone = pgdat->node_zones + j;
 		unsigned long size, realsize, memmap_pages;
mm/vmscan.c +69 -29

192 lines not on this page · raw message

Signature · 8 lines
1.7.3.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#2 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 41 lines · On Mon, 29 Nov 2010 22:49:42 -0800 Ying Han <[email protected]> wrote:

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup. The kswapd is sleeping in the wait queue headed at kswapd_wait field of a kswapd descriptor. The kswapd descriptor stores information of node or cgroup and it allows the global and per cgroup background reclaim to share common reclaim algorithms.

This patch addes the kswapd descriptor and changes per zone kswapd_wait to the common data structure.

---
 include/linux/mmzone.h |    3 +-
 include/linux/swap.h   |   10 +++++
 mm/memcontrol.c        |    2 +
 mm/mmzone.c            |    2 +-
 mm/page_alloc.c        |    9 +++-
 mm/vmscan.c            |   98 +++++++++++++++++++++++++++++++++--------------
 6 files changed, 90 insertions(+), 34 deletions(-)
include/linux/mmzone.h +1 -2
index 39c24eb..c77dfa2 100644
@@ -642,8 +642,7 @@ typedef struct pglist_data {
 	unsigned long node_spanned_pages; /* total size of physical page
 					     range, including holes */
 	int node_id;
-	wait_queue_head_t kswapd_wait;
-	struct task_struct *kswapd;
+	wait_queue_head_t *kswapd_wait;
 	int kswapd_max_order;
 } pg_data_t;
 
include/linux/swap.h +0 -0
index eba53e7..2e6cb58 100644
@@ -26,6 +26,16 @@ static inline int current_is_kswapd(void)
 	return current->flags & PF_KSWAPD;
 }
 

include/linux/swap.h static inline int current_is_kswapd(void) · line 38

Quoted 10 lines
include/linux/swap.h +10 -0
+struct kswapd {
+	struct task_struct *kswapd_task;
+	wait_queue_head_t kswapd_wait;
+	struct mem_cgroup *kswapd_mem;
+	pg_data_t *kswapd_pgdat;
+};
+
+#define MAX_KSWAPDS MAX_NUMNODES
+extern struct kswapd kswapds[MAX_KSWAPDS];
+int kswapd(void *p);

Why this is required ? Can't we allocate this at boot (if necessary) ? Why exsiting kswapd is also controlled under this structure ? At the 1st look, this just seem to increase the size of changes....

IMHO, implementing background-reclaim-for-memcg is cleaner than reusing kswapd.. kswapd has tons of unnecessary checks.

Regards,
-Kame

Quoted 270 lines
include/linux/swap.h +0 -0
 /*
  * MAX_SWAPFILES defines the maximum number of swaptypes: things which can
  * be swapped to.  The swap type and the offset into that swap type are
mm/memcontrol.c +2 -0
index a4034b6..dca3590 100644
@@ -263,6 +263,8 @@ struct mem_cgroup {
 	 */
 	struct mem_cgroup_stat_cpu nocpu_base;
 	spinlock_t pcp_counter_lock;
+
+	wait_queue_head_t *kswapd_wait;
 };
 
 /* Stuffs for move charges at task migration. */
mm/mmzone.c +1 -1
index e35bfb8..c7cbed5 100644
@@ -102,7 +102,7 @@ unsigned long zone_nr_free_pages(struct zone *zone)
 	 * free pages are low, get a better estimate for free pages
 	 */
 	if (nr_free_pages < zone->percpu_drift_mark &&
-			!waitqueue_active(&zone->zone_pgdat->kswapd_wait))
+			!waitqueue_active(zone->zone_pgdat->kswapd_wait))
 		return zone_page_state_snapshot(zone, NR_FREE_PAGES);
 
 	return nr_free_pages;
mm/page_alloc.c +7 -2
index b48dea2..a15bc1c 100644
@@ -4070,13 +4070,18 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat,
 	int nid = pgdat->node_id;
 	unsigned long zone_start_pfn = pgdat->node_start_pfn;
 	int ret;
+	struct kswapd *kswapd_p;
 
 	pgdat_resize_init(pgdat);
 	pgdat->nr_zones = 0;
-	init_waitqueue_head(&pgdat->kswapd_wait);
 	pgdat->kswapd_max_order = 0;
 	pgdat_page_cgroup_init(pgdat);
-	
+
+	kswapd_p = &kswapds[nid];
+	init_waitqueue_head(&kswapd_p->kswapd_wait);
+	pgdat->kswapd_wait = &kswapd_p->kswapd_wait;
+	kswapd_p->kswapd_pgdat = pgdat;
+
 	for (j = 0; j < MAX_NR_ZONES; j++) {
 		struct zone *zone = pgdat->node_zones + j;
 		unsigned long size, realsize, memmap_pages;
mm/vmscan.c +69 -29
index b8a6fdc..e08005e 100644
@@ -2115,12 +2115,18 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem_cont,
 
 	return nr_reclaimed;
 }
+
 #endif
 
+DEFINE_SPINLOCK(kswapds_spinlock);
+struct kswapd kswapds[MAX_KSWAPDS];
+
 /* is kswapd sleeping prematurely? */
-static int sleeping_prematurely(pg_data_t *pgdat, int order, long remaining)
+static int sleeping_prematurely(struct kswapd *kswapd, int order,
+				long remaining)
 {
 	int i;
+	pg_data_t *pgdat = kswapd->kswapd_pgdat;
 
 	/* If a direct reclaimer woke kswapd within HZ/10, it's premature */
 	if (remaining)
@@ -2377,21 +2383,28 @@ out:
  * If there are applications that are active memory-allocators
  * (most normal use), this basically shouldn't matter.
  */
-static int kswapd(void *p)
+int kswapd(void *p)
 {
 	unsigned long order;
-	pg_data_t *pgdat = (pg_data_t*)p;
+	struct kswapd *kswapd_p = (struct kswapd *)p;
+	pg_data_t *pgdat = kswapd_p->kswapd_pgdat;
+	struct mem_cgroup *mem = kswapd_p->kswapd_mem;
+	wait_queue_head_t *wait_h = &kswapd_p->kswapd_wait;
 	struct task_struct *tsk = current;
 	DEFINE_WAIT(wait);
 	struct reclaim_state reclaim_state = {
 		.reclaimed_slab = 0,
 	};
-	const struct cpumask *cpumask = cpumask_of_node(pgdat->node_id);
+	const struct cpumask *cpumask;
 
 	lockdep_set_current_reclaim_state(GFP_KERNEL);
 
-	if (!cpumask_empty(cpumask))
-		set_cpus_allowed_ptr(tsk, cpumask);
+	if (pgdat) {
+		BUG_ON(pgdat->kswapd_wait != wait_h);
+		cpumask = cpumask_of_node(pgdat->node_id);
+		if (!cpumask_empty(cpumask))
+			set_cpus_allowed_ptr(tsk, cpumask);
+	}
 	current->reclaim_state = &reclaim_state;
 
 	/*
@@ -2414,9 +2427,13 @@ static int kswapd(void *p)
 		unsigned long new_order;
 		int ret;
 
-		prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
-		new_order = pgdat->kswapd_max_order;
-		pgdat->kswapd_max_order = 0;
+		prepare_to_wait(wait_h, &wait, TASK_INTERRUPTIBLE);
+		if (pgdat) {
+			new_order = pgdat->kswapd_max_order;
+			pgdat->kswapd_max_order = 0;
+		} else
+			new_order = 0;
+
 		if (order < new_order) {
 			/*
 			 * Don't sleep if someone wants a larger 'order'
@@ -2428,10 +2445,12 @@ static int kswapd(void *p)
 				long remaining = 0;
 
 				/* Try to sleep for a short interval */
-				if (!sleeping_prematurely(pgdat, order, remaining)) {
+				if (!sleeping_prematurely(kswapd_p, order,
+							remaining)) {
 					remaining = schedule_timeout(HZ/10);
-					finish_wait(&pgdat->kswapd_wait, &wait);
-					prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
+					finish_wait(wait_h, &wait);
+					prepare_to_wait(wait_h, &wait,
+							TASK_INTERRUPTIBLE);
 				}
 
 				/*
@@ -2439,20 +2458,25 @@ static int kswapd(void *p)
 				 * premature sleep. If not, then go fully
 				 * to sleep until explicitly woken up
 				 */
-				if (!sleeping_prematurely(pgdat, order, remaining)) {
-					trace_mm_vmscan_kswapd_sleep(pgdat->node_id);
+				if (!sleeping_prematurely(kswapd_p, order,
+								remaining)) {
+					if (pgdat)
+						trace_mm_vmscan_kswapd_sleep(
+								pgdat->node_id);
 					schedule();
 				} else {
 					if (remaining)
-						count_vm_event(KSWAPD_LOW_WMARK_HIT_QUICKLY);
+						count_vm_event(
+						KSWAPD_LOW_WMARK_HIT_QUICKLY);
 					else
-						count_vm_event(KSWAPD_HIGH_WMARK_HIT_QUICKLY);
+						count_vm_event(
+						KSWAPD_HIGH_WMARK_HIT_QUICKLY);
 				}
 			}
-
-			order = pgdat->kswapd_max_order;
+			if (pgdat)
+				order = pgdat->kswapd_max_order;
 		}
-		finish_wait(&pgdat->kswapd_wait, &wait);
+		finish_wait(wait_h, &wait);
 
 		ret = try_to_freeze();
 		if (kthread_should_stop())
@@ -2476,6 +2500,7 @@ static int kswapd(void *p)
 void wakeup_kswapd(struct zone *zone, int order)
 {
 	pg_data_t *pgdat;
+	wait_queue_head_t *wait;
 
 	if (!populated_zone(zone))
 		return;
@@ -2488,9 +2513,10 @@ void wakeup_kswapd(struct zone *zone, int order)
 	trace_mm_vmscan_wakeup_kswapd(pgdat->node_id, zone_idx(zone), order);
 	if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL))
 		return;
-	if (!waitqueue_active(&pgdat->kswapd_wait))
+	wait = pgdat->kswapd_wait;
+	if (!waitqueue_active(wait))
 		return;
-	wake_up_interruptible(&pgdat->kswapd_wait);
+	wake_up_interruptible(wait);
 }
 
 /*
@@ -2587,7 +2613,10 @@ static int __devinit cpu_callback(struct notifier_block *nfb,
 
 			if (cpumask_any_and(cpu_online_mask, mask) < nr_cpu_ids)
 				/* One of our CPUs online: restore mask */
-				set_cpus_allowed_ptr(pgdat->kswapd, mask);
+				if (kswapds[nid].kswapd_task)
+					set_cpus_allowed_ptr(
+						kswapds[nid].kswapd_task,
+						mask);
 		}
 	}
 	return NOTIFY_OK;
@@ -2599,19 +2628,20 @@ static int __devinit cpu_callback(struct notifier_block *nfb,
  */
 int kswapd_run(int nid)
 {
-	pg_data_t *pgdat = NODE_DATA(nid);
+	struct task_struct *thr;
 	int ret = 0;
 
-	if (pgdat->kswapd)
+	if (kswapds[nid].kswapd_task)
 		return 0;
 
-	pgdat->kswapd = kthread_run(kswapd, pgdat, "kswapd%d", nid);
-	if (IS_ERR(pgdat->kswapd)) {
+	thr = kthread_run(kswapd, &kswapds[nid], "kswapd%d", nid);
+	if (IS_ERR(thr)) {
 		/* failure at boot is fatal */
 		BUG_ON(system_state == SYSTEM_BOOTING);
 		printk("Failed to start kswapd on node %d\n",nid);
 		ret = -1;
 	}
+	kswapds[nid].kswapd_task = thr;
 	return ret;
 }
 
@@ -2620,10 +2650,20 @@ int kswapd_run(int nid)
  */
 void kswapd_stop(int nid)
 {
-	struct task_struct *kswapd = NODE_DATA(nid)->kswapd;
+	struct task_struct *thr;
+	struct kswapd *kswapd_p;
+	wait_queue_head_t *wait;
+
+	pg_data_t *pgdat = NODE_DATA(nid);
+
+	spin_lock(&kswapds_spinlock);
+	wait = pgdat->kswapd_wait;
+	kswapd_p = container_of(wait, struct kswapd, kswapd_wait);
+	thr = kswapd_p->kswapd_task;
+	spin_unlock(&kswapds_spinlock);
 
-	if (kswapd)
-		kthread_stop(kswapd);
+	if (thr)
+		kthread_stop(thr);
 }
 
 static int __init kswapd_init(void)
Signature · 8 lines
1.7.3.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 59 lines · KAMEZAWA Hiroyuki
Quoted 41 lines · On Mon, 29 Nov 2010 22:49:42 -0800 Ying Han <[email protected]> wrote:

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup. The kswapd is sleeping in the wait queue headed at kswapd_wait field of a kswapd descriptor. The kswapd descriptor stores information of node or cgroup and it allows the global and per cgroup background reclaim to share common reclaim algorithms.

This patch addes the kswapd descriptor and changes per zone kswapd_wait to the common data structure.

---
 include/linux/mmzone.h |    3 +-
 include/linux/swap.h   |   10 +++++
 mm/memcontrol.c        |    2 +
 mm/mmzone.c            |    2 +-
 mm/page_alloc.c        |    9 +++-
 mm/vmscan.c            |   98 +++++++++++++++++++++++++++++++++--------------

 6 files changed, 90 insertions(+), 34 deletions(-)
include/linux/mmzone.h +0 -0
index 39c24eb..c77dfa2 100644
@@ -642,8 +642,7 @@ typedef struct pglist_data {

      unsigned long node_spanned_pages; /* total size of physical page range, including holes */ int node_id;
-     wait_queue_head_t kswapd_wait;
-     struct task_struct *kswapd;
+     wait_queue_head_t *kswapd_wait;
      int kswapd_max_order;
 } pg_data_t;

include/linux/swap.h +0 -0
index eba53e7..2e6cb58 100644
@@ -26,6 +26,16 @@ static inline int current_is_kswapd(void)
       return current->flags & PF_KSWAPD;
  }
 

include/linux/swap.h static inline int current_is_kswapd(void) · line 38

Quoted 10 lines
include/linux/swap.h +10 -0
+struct kswapd {
+     struct task_struct *kswapd_task;
+     wait_queue_head_t kswapd_wait;
+     struct mem_cgroup *kswapd_mem;
+     pg_data_t *kswapd_pgdat;
+};
+
+#define MAX_KSWAPDS MAX_NUMNODES
+extern struct kswapd kswapds[MAX_KSWAPDS];
+int kswapd(void *p);

Why this is required ? Can't we allocate this at boot (if necessary) ? Why exsiting kswapd is also controlled under this structure ? At the 1st look, this just seem to increase the size of changes....

Quoted 2 lines

IMHO, implementing background-reclaim-for-memcg is cleaner than reusing kswapd.. kswapd has tons of unnecessary checks.

Ideally, I hope we unify global and memcg of kswapd for easy maintainance if it's not a big problem.
When we make patches about lru pages, we always have to consider what I should do for memcg.
And when we review patches, we also should consider what the patch is missing for memcg.
It makes maintainance cost big. Of course, if memcg maintainers is involved with all patches, it's no problem as it is.

If it is impossible due to current kswapd's spaghetti, we can clean up it first. I am not sure whether my suggestion make sense or not. Kame can know it much rather than me. But please consider such the voice.

Quoted 4 lines

Regards,
-Kame

Signature · 9 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#4 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 9 lines · On Tue, 30 Nov 2010 17:15:37 +0900 Minchan Kim <[email protected]> wrote:

Ideally, I hope we unify global and memcg of kswapd for easy maintainance if it's not a big problem.
When we make patches about lru pages, we always have to consider what I should do for memcg.
And when we review patches, we also should consider what the patch is missing for memcg.
It makes maintainance cost big. Of course, if memcg maintainers is involved with all patches, it's no problem as it is.

I know it's not. But thread control of kswapd will not have much merging point. And balance_pgdat() is fully replaced in patch/3. The effort for merging seems not big.

Quoted 2 lines

If it is impossible due to current kswapd's spaghetti, we can clean up it first. I am not sure whether my suggestion make sense or not.

make sense.

Quoted 1 line

Kame can know it much rather than me. But please consider such the voice.

Unifying is ok in general but this patch seems uglier than imagined. Implementing a simple memcg one and considering how-to-merge is a way. But it's a long way.

For now. we have to check the design/function of patch before how beautiful it is.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#5 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 12 lines · On Tue, 30 Nov 2010 17:27:10 +0900 KAMEZAWA Hiroyuki…
Quoted 9 lines · On Tue, 30 Nov 2010 17:15:37 +0900 Minchan Kim <[email protected]> wrote:

Ideally, I hope we unify global and memcg of kswapd for easy maintainance if it's not a big problem.
When we make patches about lru pages, we always have to consider what I should do for memcg.
And when we review patches, we also should consider what the patch is missing for memcg.
It makes maintainance cost big. Of course, if memcg maintainers is involved with all patches, it's no problem as it is.

Quoted 4 lines

I know it's not. But thread control of kswapd will not have much merging point. And balance_pgdat() is fully replaced in patch/3. The effort for merging seems not big.

kswapd's balance_pgdat() is for following
- reclaim pages within a node.
- balancing zones in a pgdat.

memcg's background reclaim needs followings.
- reclaim pages within a memcg
- reclaim pages from arbitrary zones, if it's fair, it's good. But it's not important from which zone the pages are reclaimed from. (I'm not sure we can select "the oldest" pages from divided LRU.)

Then, merging will put 2 _very_ different functionalities into 1 function.

So, I thought it's simpler to implement

1. a victim node selector (This algorithm will never be in kswapd.)
2. call _existing_ try_to_free_pages_mem_cgroup() with node local zonelist. Sharing is enough.

kswapd stop/go routine may be able to be shared. But this patch itself seems not very good to me.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#6 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 15 lines · KAMEZAWA Hiroyuki
Quoted 12 lines · On Tue, 30 Nov 2010 17:27:10 +0900 KAMEZAWA Hiroyuki…
Quoted 9 lines · On Tue, 30 Nov 2010 17:15:37 +0900 Minchan Kim <[email protected]> wrote:

Ideally, I hope we unify global and memcg of kswapd for easy maintainance if it's not a big problem.
When we make patches about lru pages, we always have to consider what I should do for memcg.
And when we review patches, we also should consider what the patch is missing for memcg.
It makes maintainance cost big. Of course, if memcg maintainers is involved with all patches, it's no problem as it is.

Quoted 3 lines
Quoted 3 lines

I know it's not. But thread control of kswapd will not have much merging point. And balance_pgdat() is fully replaced in patch/3. The effort for merging seems not big.

I intended to separate out the logic of per-memcg kswapd logics and not having it
interfere with existing code. This should help for merging.

Quoted 11 lines
Quoted 1 line

kswapd's balance_pgdat() is for following
 - reclaim pages within a node.
 - balancing zones in a pgdat.

memcg's background reclaim needs followings.
 - reclaim pages within a memcg
 - reclaim pages from arbitrary zones, if it's fair, it's good. But it's not important from which zone the pages are reclaimed from. (I'm not sure we can select "the oldest" pages from divided LRU.)

The current implementation is simple, which it iterates all the nodes and reclaims pages from the per-memcg-per-zone LRU. As long as the wmarks is ok, the kswapd is done. Meanwhile, in order to not wasting cputime on "unreclaimable: nodes ( a node is unreclaimable if all the zones are unreclaimable), I used the nodemask to record that from the last scan, and the bit is reset as long as a page is returned back. This is a similar logic used in the global kswapd.

A potential improvement is to remember the last node we reclaimed from, and starting from the next node for the next kswapd wake_up. This avoids the case all the memcg kswapds are reclaiming from the small node ids on large numa machines.

Quoted 2 lines

Then, merging will put 2 _very_ different functionalities into 1 function

#7 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 17 lines · On Tue, 30 Nov 2010 12:40:16 -0800 Ying Han <[email protected]> wrote:
Quoted 15 lines · KAMEZAWA Hiroyuki
Quoted 12 lines · On Tue, 30 Nov 2010 17:27:10 +0900 KAMEZAWA Hiroyuki…
Quoted 9 lines · On Tue, 30 Nov 2010 17:15:37 +0900 Minchan Kim <[email protected]> wrote:

Ideally, I hope we unify global and memcg of kswapd for easy maintainance if it's not a big problem.
When we make patches about lru pages, we always have to consider what I should do for memcg.
And when we review patches, we also should consider what the patch is missing for memcg.
It makes maintainance cost big. Of course, if memcg maintainers is involved with all patches, it's no problem as it is.

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

I know it's not. But thread control of kswapd will not have much merging point. And balance_pgdat() is fully replaced in patch/3. The effort for merging seems not big.

I intended to separate out the logic of per-memcg kswapd logics and not having it
interfere with existing code. This should help for merging.

yes.

Quoted 20 lines
Quoted 11 lines
Quoted 1 line

kswapd's balance_pgdat() is for following
A - reclaim pages within a node.
A - balancing zones in a pgdat.

memcg's background reclaim needs followings.
A - reclaim pages within a memcg
A - reclaim pages from arbitrary zones, if it's fair, it's good. A A But it's not important from which zone the pages are reclaimed from. A A (I'm not sure we can select "the oldest" pages from divided LRU.)

The current implementation is simple, which it iterates all the nodes and reclaims pages from the per-memcg-per-zone LRU. As long as the wmarks is ok, the kswapd is done. Meanwhile, in order to not wasting cputime on "unreclaimable: nodes ( a node is unreclaimable if all the zones are unreclaimable), I used the nodemask to record that from the last scan, and the bit is reset as long as a page is returned back. This is a similar logic used in the global kswapd.

Quoted 5 lines

A potential improvement is to remember the last node we reclaimed from, and starting from the next node for the next kswapd wake_up. This avoids the case all the memcg kswapds are reclaiming from the small node ids on large numa machines.

Yes, that's helpful.

Quoted 12 lines
Quoted 2 lines

Then, merging will put 2 _very_ different functionalities into 1 function.

Agree.

Quoted 4 lines

So, I thought it's simpler to implement

A 1. a victim node selector (This algorithm will never be in kswapd.)

Yeah, or round robin as I replied above ?

I think it's good to have.

Quoted 5 lines
Quoted 2 lines

A 2. call _existing_ try_to_free_pages_mem_cgroup() with node local zonelist. A Sharing is enough.

That will in turn use direct reclaim logic which has no notion of wmarks.

 do {
	node = select_victim_node();
	do_try_to_free_pages_mem_cgroup(node);
	check watermark

}

or If we need to check priority at el, your new balance_pgdat_mem_cgroup() will be good.

Quoted 3 lines
Quoted 2 lines

kswapd stop/go routine may be able to be shared. But this patch itself seems not very good to me.

This looks feasible change, I will double check with it.

Thanks.

Regards,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#8 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

In reply to KAMEZAWA Hiroyuki
Quoted 24 lines · KAMEZAWA Hiroyuki
Quoted 12 lines · On Tue, 30 Nov 2010 17:27:10 +0900 KAMEZAWA Hiroyuki…
Quoted 9 lines · On Tue, 30 Nov 2010 17:15:37 +0900 Minchan Kim <[email protected]> wrote:

Ideally, I hope we unify global and memcg of kswapd for easy maintainance if it's not a big problem.
When we make patches about lru pages, we always have to consider what I should do for memcg.
And when we review patches, we also should consider what the patch is missing for memcg.
It makes maintainance cost big. Of course, if memcg maintainers is involved with all patches, it's no problem as it is.

Quoted 4 lines

I know it's not. But thread control of kswapd will not have much merging point. And balance_pgdat() is fully replaced in patch/3. The effort for merging seems not big.

kswapd's balance_pgdat() is for following
- reclaim pages within a node.
- balancing zones in a pgdat.

Quoted 6 lines

memcg's background reclaim needs followings.
- reclaim pages within a memcg
- reclaim pages from arbitrary zones, if it's fair, it's good. But it's not important from which zone the pages are reclaimed from. (I'm not sure we can select "the oldest" pages from divided LRU.)

Yes, if it is fair, then we don't break what kswapd tries to do, so fairness is quite important, in that we don't leaves zones unbalanced (at least by very much) as we try to do background reclaim. But sometimes it cannot be helped, specially if there are policies that bias the allocation.

Quoted 5 lines

Then, merging will put 2 _very_ different functionalities into 1 function.

So, I thought it's simpler to implement

1. a victim node selector (This algorithm will never be in kswapd.)

A victim node selector per memcg? Could you clarify the context of node here?

Quoted 9 lines

2. call _existing_ try_to_free_pages_mem_cgroup() with node local zonelist. Sharing is enough.

kswapd stop/go routine may be able to be shared. But this patch itself seems not very good to me.

Thanks,
-Kame

Signature · 9 lines
	Three Cheers,
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#9 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 39 lines · On Tue, 7 Dec 2010 11:45:03 +0530 Balbir Singh <[email protected]>…
Quoted 24 lines · KAMEZAWA Hiroyuki
Quoted 12 lines · On Tue, 30 Nov 2010 17:27:10 +0900 KAMEZAWA Hiroyuki…
Quoted 9 lines · On Tue, 30 Nov 2010 17:15:37 +0900 Minchan Kim <[email protected]> wrote:

Ideally, I hope we unify global and memcg of kswapd for easy maintainance if it's not a big problem.
When we make patches about lru pages, we always have to consider what I should do for memcg.
And when we review patches, we also should consider what the patch is missing for memcg.
It makes maintainance cost big. Of course, if memcg maintainers is involved with all patches, it's no problem as it is.

Quoted 4 lines

I know it's not. But thread control of kswapd will not have much merging point. And balance_pgdat() is fully replaced in patch/3. The effort for merging seems not big.

kswapd's balance_pgdat() is for following
- reclaim pages within a node.
- balancing zones in a pgdat.

Quoted 6 lines

memcg's background reclaim needs followings.
- reclaim pages within a memcg
- reclaim pages from arbitrary zones, if it's fair, it's good. But it's not important from which zone the pages are reclaimed from. (I'm not sure we can select "the oldest" pages from divided LRU.)

Yes, if it is fair, then we don't break what kswapd tries to do, so fairness is quite important, in that we don't leaves zones unbalanced (at least by very much) as we try to do background reclaim. But sometimes it cannot be helped, specially if there are policies that bias the allocation.

Quoted 9 lines
Quoted 5 lines

Then, merging will put 2 _very_ different functionalities into 1 function.

So, I thought it's simpler to implement

1. a victim node selector (This algorithm will never be in kswapd.)

A victim node selector per memcg? Could you clarify the context of node here?

An argument to balance_pgdat_for_memcg() or a start point of zonelist[]. i.e.

	zone_list = NODE_DATA(victim)->zonelist[0 or 1]

	for_each_zone_zonelist(z, zone_list)....

But, this is just an example, we just need to determine where we reclaim page from before start walking.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#10 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 51 lines · KAMEZAWA Hiroyuki
Quoted 39 lines · On Tue, 7 Dec 2010 11:45:03 +0530 Balbir Singh <[email protected]>…
Quoted 24 lines · KAMEZAWA Hiroyuki
Quoted 12 lines · On Tue, 30 Nov 2010 17:27:10 +0900 KAMEZAWA Hiroyuki…
Quoted 9 lines · On Tue, 30 Nov 2010 17:15:37 +0900 Minchan Kim <[email protected]> wrote:

Ideally, I hope we unify global and memcg of kswapd for easy maintainance if it's not a big problem.
When we make patches about lru pages, we always have to consider what I should do for memcg.
And when we review patches, we also should consider what the patch is missing for memcg.
It makes maintainance cost big. Of course, if memcg maintainers is involved with all patches, it's no problem as it is.

Quoted 4 lines

I know it's not. But thread control of kswapd will not have much merging point. And balance_pgdat() is fully replaced in patch/3. The effort for merging seems not big.

kswapd's balance_pgdat() is for following
- reclaim pages within a node.
- balancing zones in a pgdat.

Quoted 6 lines

memcg's background reclaim needs followings.
- reclaim pages within a memcg
- reclaim pages from arbitrary zones, if it's fair, it's good. But it's not important from which zone the pages are reclaimed from. (I'm not sure we can select "the oldest" pages from divided LRU.)

Yes, if it is fair, then we don't break what kswapd tries to do, so fairness is quite important, in that we don't leaves zones unbalanced (at least by very much) as we try to do background reclaim. But sometimes it cannot be helped, specially if there are policies that bias the allocation.

Quoted 9 lines
Quoted 5 lines

Then, merging will put 2 _very_ different functionalities into 1 function.

So, I thought it's simpler to implement

1. a victim node selector (This algorithm will never be in kswapd.)

A victim node selector per memcg? Could you clarify the context of node here?

Quoted 9 lines

An argument to balance_pgdat_for_memcg() or a start point of zonelist[]. i.e.

	zone_list = NODE_DATA(victim)->zonelist[0 or 1]

	for_each_zone_zonelist(z, zone_list)....

But, this is just an example, we just need to determine where we reclaim page from before start walking.

OK, I understand. BTW, I am not against integration with kswapd for watermark based reclaim, the advantage I see is that as we balance zone/node watermarks, we also balance per memcg watermark. The cost would be proportional to the size of memcg's that have allocated from that zone/node. kswapd is not fast path and already optimized in terms of when to wake up, so it makes sense to reuse all of that.

Signature · 9 lines
	Three Cheers,
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#11 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 11 lines · On Tue, 7 Dec 2010 12:29:29 +0530 Balbir Singh <[email protected]>…
Quoted 9 lines · KAMEZAWA Hiroyuki

An argument to balance_pgdat_for_memcg() or a start point of zonelist[]. i.e.

	zone_list = NODE_DATA(victim)->zonelist[0 or 1]

	for_each_zone_zonelist(z, zone_list)....

But, this is just an example, we just need to determine where we reclaim page from before start walking.

Quoted 7 lines

OK, I understand. BTW, I am not against integration with kswapd for watermark based reclaim, the advantage I see is that as we balance zone/node watermarks, we also balance per memcg watermark. The cost would be proportional to the size of memcg's that have allocated from that zone/node. kswapd is not fast path and already optimized in terms of when to wake up, so it makes sense to reuse all of that.

But we cannot use balance_pgdat() as it is because we don't need almost all checks in it and I don't want to add hooks into it because it's updated frequently. And, I doubt how cleanly we can do merging.

As Ying Han did, adding balance_pgdat_for_memcg() is a clean way for now. kswapd wakeup, sleep routine may be able to be reused.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#12 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

In reply to Minchan Kim
Quoted 61 lines · Minchan Kim
Quoted 59 lines · KAMEZAWA Hiroyuki
Quoted 41 lines · On Mon, 29 Nov 2010 22:49:42 -0800 Ying Han <[email protected]> wrote:

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup. The kswapd is sleeping in the wait queue headed at kswapd_wait field of a kswapd descriptor. The kswapd descriptor stores information of node or cgroup and it allows the global and per cgroup background reclaim to share common reclaim algorithms.

This patch addes the kswapd descriptor and changes per zone kswapd_wait to the common data structure.

---
 include/linux/mmzone.h |    3 +-
 include/linux/swap.h   |   10 +++++
 mm/memcontrol.c        |    2 +
 mm/mmzone.c            |    2 +-
 mm/page_alloc.c        |    9 +++-
 mm/vmscan.c            |   98 +++++++++++++++++++++++++++++++++--------------

 6 files changed, 90 insertions(+), 34 deletions(-)
include/linux/mmzone.h +0 -0
index 39c24eb..c77dfa2 100644
@@ -642,8 +642,7 @@ typedef struct pglist_data {

      unsigned long node_spanned_pages; /* total size of physical page range, including holes */ int node_id;
-     wait_queue_head_t kswapd_wait;
-     struct task_struct *kswapd;
+     wait_queue_head_t *kswapd_wait;
      int kswapd_max_order;
 } pg_data_t;

include/linux/swap.h +0 -0
index eba53e7..2e6cb58 100644
@@ -26,6 +26,16 @@ static inline int current_is_kswapd(void)
       return current->flags & PF_KSWAPD;
  }
 

include/linux/swap.h static inline int current_is_kswapd(void) · line 38

Quoted 10 lines
include/linux/swap.h +10 -0
+struct kswapd {
+     struct task_struct *kswapd_task;
+     wait_queue_head_t kswapd_wait;
+     struct mem_cgroup *kswapd_mem;
+     pg_data_t *kswapd_pgdat;
+};
+
+#define MAX_KSWAPDS MAX_NUMNODES
+extern struct kswapd kswapds[MAX_KSWAPDS];
+int kswapd(void *p);

Why this is required ? Can't we allocate this at boot (if necessary) ? Why exsiting kswapd is also controlled under this structure ? At the 1st look, this just seem to increase the size of changes....

Quoted 5 lines
Quoted 2 lines

IMHO, implementing background-reclaim-for-memcg is cleaner than reusing kswapd.. kswapd has tons of unnecessary checks.

Ideally, I hope we unify global and memcg of kswapd for easy maintainance if it's not a big problem.

I intended not doing so in this patchset since the algorithm and reclaiming target are
different for global and per-memcg kswapd. I would prefer not having the new changes
to affect existing logic.

Quoted 4 lines

When we make patches about lru pages, we always have to consider what I should do for memcg.
And when we review patches, we also should consider what the patch is missing for memcg.

The per-memcg LRU is there and that needs to be considered differently as global one. This
patchset doesn't change that part but is based on that. I don't see by merging the kswapd will
help the maintainance in that sense. All the following changes to the per-memcg LRU should take
effect automatically to the per-memcg kswapd later on.

Quoted 7 lines

It makes maintainance cost big. Of course, if memcg maintainers is involved with all patches, it's no problem as it is.

If it is impossible due to current kswapd's spaghetti, we can clean up it first. I am not sure whether my suggestion make sense or not. Kame can know it much rather than me. But please consider such the voice.

The global kswapd is working on node and zones on the node. Its target is to bring
all the zones above high wmarks unless the zones are "unreclaimable". The logic is
different for per-memcg kswapd which scans all the nodes and zones on the system and tries to bring the per-memcg wmark above the threshold. Lots of heuristics are
not shared at this moment, and I am not sure if this is a good idea to merge them.

--Ying

Quoted 8 lines
Quoted 5 lines

Regards,
-Kame

Quoted 4 lines

--
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#13 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

In reply to KAMEZAWA Hiroyuki
Quoted 44 lines · KAMEZAWA Hiroyuki
Quoted 41 lines · On Mon, 29 Nov 2010 22:49:42 -0800 Ying Han <[email protected]> wrote:

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup. The kswapd is sleeping in the wait queue headed at kswapd_wait field of a kswapd descriptor. The kswapd descriptor stores information of node or cgroup and it allows the global and per cgroup background reclaim to share common reclaim algorithms.

This patch addes the kswapd descriptor and changes per zone kswapd_wait to the common data structure.

---
 include/linux/mmzone.h |    3 +-
 include/linux/swap.h   |   10 +++++
 mm/memcontrol.c        |    2 +
 mm/mmzone.c            |    2 +-
 mm/page_alloc.c        |    9 +++-
 mm/vmscan.c            |   98 +++++++++++++++++++++++++++++++++--------------

 6 files changed, 90 insertions(+), 34 deletions(-)
include/linux/mmzone.h +0 -0
index 39c24eb..c77dfa2 100644
@@ -642,8 +642,7 @@ typedef struct pglist_data {

      unsigned long node_spanned_pages; /* total size of physical page range, including holes */ int node_id;
-     wait_queue_head_t kswapd_wait;
-     struct task_struct *kswapd;
+     wait_queue_head_t *kswapd_wait;
      int kswapd_max_order;
 } pg_data_t;

include/linux/swap.h +0 -0
index eba53e7..2e6cb58 100644
@@ -26,6 +26,16 @@ static inline int current_is_kswapd(void)
       return current->flags & PF_KSWAPD;
  }
 
Quoted 12 lines

include/linux/swap.h static inline int current_is_kswapd(void) · line 38

Quoted 10 lines
include/linux/swap.h +10 -0
+struct kswapd {
+     struct task_struct *kswapd_task;
+     wait_queue_head_t kswapd_wait;
+     struct mem_cgroup *kswapd_mem;
+     pg_data_t *kswapd_pgdat;
+};
+
+#define MAX_KSWAPDS MAX_NUMNODES
+extern struct kswapd kswapds[MAX_KSWAPDS];
+int kswapd(void *p);

Why this is required ? Can't we allocate this at boot (if necessary) ?

I can double check with that.

Quoted 1 line

Why exsiting kswapd is also controlled under this structure ?

Some of the reclaim algorithm could be shared after we unifying the API for global/memcg
background reclaim. One of the examples is the kswapd() daemon function itself.

Quoted 4 lines

At the 1st look, this just seem to increase the size of changes....

IMHO, implementing background-reclaim-for-memcg is cleaner than reusing kswapd.. kswapd has tons of unnecessary checks.

Sorry I am not aware of "background-reclaim-for-memcg", can you specify little bit more? Also,
the unnecessary checks here refers to the kswapd() or balance_pgdat()? If the latter one, the
logic is not being shared at all included in patch/3.

--Ying

Quoted 267 lines

Regards,
-Kame

Quoted 263 lines
include/linux/swap.h +0 -0
  /*
   * MAX_SWAPFILES defines the maximum number of swaptypes: things which can
   * be swapped to.  The swap type and the offset into that swap type are
mm/memcontrol.c +2 -0
index a4034b6..dca3590 100644
@@ -263,6 +263,8 @@ struct mem_cgroup {
        */
       struct mem_cgroup_stat_cpu nocpu_base;
       spinlock_t pcp_counter_lock;
+
+     wait_queue_head_t *kswapd_wait;
  };
 
  /* Stuffs for move charges at task migration. */
mm/mmzone.c +0 -0
index e35bfb8..c7cbed5 100644
@@ -102,7 +102,7 @@ unsigned long zone_nr_free_pages(struct zone *zone)
        * free pages are low, get a better estimate for free pages
        */

      if (nr_free_pages < zone->percpu_drift_mark &&
-                     !waitqueue_active(&zone->zone_pgdat->kswapd_wait))
+                     !waitqueue_active(zone->zone_pgdat->kswapd_wait)) return zone_page_state_snapshot(zone, NR_FREE_PAGES);

      return nr_free_pages;

mm/page_alloc.c +7 -2
index b48dea2..a15bc1c 100644
@@ -4070,13 +4070,18 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat,
       int nid = pgdat->node_id;
       unsigned long zone_start_pfn = pgdat->node_start_pfn;
       int ret;
+     struct kswapd *kswapd_p;
 
       pgdat_resize_init(pgdat);
       pgdat->nr_zones = 0;
-     init_waitqueue_head(&pgdat->kswapd_wait);
       pgdat->kswapd_max_order = 0;
       pgdat_page_cgroup_init(pgdat);
-
+
+     kswapd_p = &kswapds[nid];
+     init_waitqueue_head(&kswapd_p->kswapd_wait);
+     pgdat->kswapd_wait = &kswapd_p->kswapd_wait;
+     kswapd_p->kswapd_pgdat = pgdat;
+
       for (j = 0; j < MAX_NR_ZONES; j++) {
               struct zone *zone = pgdat->node_zones + j;
               unsigned long size, realsize, memmap_pages;
mm/vmscan.c +38 -15
index b8a6fdc..e08005e 100644
@@ -2115,12 +2115,18 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem_cont,
 
       return nr_reclaimed;
  }
+
  #endif
 
+DEFINE_SPINLOCK(kswapds_spinlock);
+struct kswapd kswapds[MAX_KSWAPDS];
+
  /* is kswapd sleeping prematurely? */
-static int sleeping_prematurely(pg_data_t *pgdat, int order, long remaining)
+static int sleeping_prematurely(struct kswapd *kswapd, int order,
+                             long remaining)
  {
       int i;
+     pg_data_t *pgdat = kswapd->kswapd_pgdat;
 
       /* If a direct reclaimer woke kswapd within HZ/10, it's premature */
       if (remaining)
@@ -2377,21 +2383,28 @@ out:
   * If there are applications that are active memory-allocators
   * (most normal use), this basically shouldn't matter.
   */
-static int kswapd(void *p)
+int kswapd(void *p)
  {
       unsigned long order;
-     pg_data_t *pgdat = (pg_data_t*)p;
+     struct kswapd *kswapd_p = (struct kswapd *)p;
+     pg_data_t *pgdat = kswapd_p->kswapd_pgdat;
+     struct mem_cgroup *mem = kswapd_p->kswapd_mem;
+     wait_queue_head_t *wait_h = &kswapd_p->kswapd_wait;
       struct task_struct *tsk = current;
       DEFINE_WAIT(wait);
       struct reclaim_state reclaim_state = {
               .reclaimed_slab = 0,
       };
-     const struct cpumask *cpumask = cpumask_of_node(pgdat->node_id);
+     const struct cpumask *cpumask;
 
       lockdep_set_current_reclaim_state(GFP_KERNEL);
 
-     if (!cpumask_empty(cpumask))
-             set_cpus_allowed_ptr(tsk, cpumask);
+     if (pgdat) {
+             BUG_ON(pgdat->kswapd_wait != wait_h);
+             cpumask = cpumask_of_node(pgdat->node_id);
+             if (!cpumask_empty(cpumask))
+                     set_cpus_allowed_ptr(tsk, cpumask);
+     }
       current->reclaim_state = &reclaim_state;
 
       /*
@@ -2414,9 +2427,13 @@ static int kswapd(void *p)
               unsigned long new_order;
               int ret;
 
-             prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
-             new_order = pgdat->kswapd_max_order;
-             pgdat->kswapd_max_order = 0;
+             prepare_to_wait(wait_h, &wait, TASK_INTERRUPTIBLE);
+             if (pgdat) {
+                     new_order = pgdat->kswapd_max_order;
+                     pgdat->kswapd_max_order = 0;
+             } else
+                     new_order = 0;
+
               if (order < new_order) {
                       /*
                        * Don't sleep if someone wants a larger 'order'
@@ -2428,10 +2445,12 @@ static int kswapd(void *p)
                               long remaining = 0;
 
                               /* Try to sleep for a short interval */
-                             if (!sleeping_prematurely(pgdat, order, remaining)) {
+                             if (!sleeping_prematurely(kswapd_p, order,
+                                                     remaining)) {
                                       remaining = schedule_timeout(HZ/10);
-                                     finish_wait(&pgdat->kswapd_wait, &wait);
-                                     prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
+                                     finish_wait(wait_h, &wait);
+                                     prepare_to_wait(wait_h, &wait,
+                                                     TASK_INTERRUPTIBLE);
                               }
 
                               /*
@@ -2439,20 +2458,25 @@ static int kswapd(void *p)
                                * premature sleep. If not, then go fully
                                * to sleep until explicitly woken up
                                */
-                             if (!sleeping_prematurely(pgdat, order, remaining)) {
-                                     trace_mm_vmscan_kswapd_sleep(pgdat->node_id);
+                             if (!sleeping_prematurely(kswapd_p, order,
+                                                             remaining)) {
+                                     if (pgdat)
+                                             trace_mm_vmscan_kswapd_sleep(
+                                                             pgdat->node_id);
                                       schedule();
                               } else {
                                       if (remaining)
-                                             count_vm_event(KSWAPD_LOW_WMARK_HIT_QUICKLY);
+                                             count_vm_event(
+                                             KSWAPD_LOW_WMARK_HIT_QUICKLY);

                                      else
-                                             count_vm_event(KSWAPD_HIGH_WMARK_HIT_QUICKLY);
+                                             count_vm_event(
+                                             KSWAPD_HIGH_WMARK_HIT_QUICKLY); }
                      }
-
-                     order = pgdat->kswapd_max_order;
+                     if (pgdat)
+                             order = pgdat->kswapd_max_order;
              }
-             finish_wait(&pgdat->kswapd_wait, &wait);
+             finish_wait(wait_h, &wait);

              ret = try_to_freeze();
              if (kthread_should_stop())

mm/vmscan.c +26 -11
@@ -2476,6 +2500,7 @@ static int kswapd(void *p)
  void wakeup_kswapd(struct zone *zone, int order)
  {
       pg_data_t *pgdat;
+     wait_queue_head_t *wait;
 
       if (!populated_zone(zone))
               return;
@@ -2488,9 +2513,10 @@ void wakeup_kswapd(struct zone *zone, int order)
       trace_mm_vmscan_wakeup_kswapd(pgdat->node_id, zone_idx(zone), order);
       if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL))
               return;
-     if (!waitqueue_active(&pgdat->kswapd_wait))
+     wait = pgdat->kswapd_wait;
+     if (!waitqueue_active(wait))
               return;
-     wake_up_interruptible(&pgdat->kswapd_wait);
+     wake_up_interruptible(wait);
  }
 
  /*
@@ -2587,7 +2613,10 @@ static int __devinit cpu_callback(struct notifier_block *nfb,
 
                       if (cpumask_any_and(cpu_online_mask, mask) < nr_cpu_ids)
                               /* One of our CPUs online: restore mask */
-                             set_cpus_allowed_ptr(pgdat->kswapd, mask);
+                             if (kswapds[nid].kswapd_task)
+                                     set_cpus_allowed_ptr(
+                                             kswapds[nid].kswapd_task,
+                                             mask);
               }
       }
       return NOTIFY_OK;
@@ -2599,19 +2628,20 @@ static int __devinit cpu_callback(struct notifier_block *nfb,
   */
  int kswapd_run(int nid)
  {
-     pg_data_t *pgdat = NODE_DATA(nid);
+     struct task_struct *thr;
       int ret = 0;
 
-     if (pgdat->kswapd)
+     if (kswapds[nid].kswapd_task)
               return 0;
 
-     pgdat->kswapd = kthread_run(kswapd, pgdat, "kswapd%d", nid);
-     if (IS_ERR(pgdat->kswapd)) {
+     thr = kthread_run(kswapd, &kswapds[nid], "kswapd%d", nid);
+     if (IS_ERR(thr)) {
               /* failure at boot is fatal */
               BUG_ON(system_state == SYSTEM_BOOTING);
               printk("Failed to start kswapd on node %d\n",nid);
               ret = -1;
       }
+     kswapds[nid].kswapd_task = thr;
       return ret;
  }
 
@@ -2620,10 +2650,20 @@ int kswapd_run(int nid)
   */
  void kswapd_stop(int nid)
  {
-     struct task_struct *kswapd = NODE_DATA(nid)->kswapd;
+     struct task_struct *thr;
+     struct kswapd *kswapd_p;
+     wait_queue_head_t *wait;
+
+     pg_data_t *pgdat = NODE_DATA(nid);
+
+     spin_lock(&kswapds_spinlock);
+     wait = pgdat->kswapd_wait;
+     kswapd_p = container_of(wait, struct kswapd, kswapd_wait);
+     thr = kswapd_p->kswapd_task;
+     spin_unlock(&kswapds_spinlock);
 
-     if (kswapd)
-             kthread_stop(kswapd);
+     if (thr)
+             kthread_stop(thr);
  }
 
  static int __init kswapd_init(void)
--

1.7.3.1

Quoted 9 lines
Quoted 7 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#14 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 11 lines · On Tue, 30 Nov 2010 12:17:33 -0800 Ying Han <[email protected]> wrote:
Quoted 4 lines

At the 1st look, this just seem to increase the size of changes....

IMHO, implementing background-reclaim-for-memcg is cleaner than reusing kswapd.. kswapd has tons of unnecessary checks.

Sorry I am not aware of "background-reclaim-for-memcg", can you specify little bit more? Also,
the unnecessary checks here refers to the kswapd() or balance_pgdat()? If the latter one, the
logic is not being shared at all included in patch/3.

Yes, now I read patch/3 and I'm sorry to say that.

Some nits.

At 1st. I just coudln't undestand idea of array of kswapd descriptor. Hmm, dynamic allocation isn't possible ? as

==

struct kswapd_param {
	pg_data_t	*pgdat;
	struct mem_cgroup *memcg;
	struct wait_queue *waitq;
};

int kswapd_run(int nid, struct mem_cgroup *memcg)

{
	struct kswapd_param *param;

	param = kzalloc(); /* freed by kswapd */

	if (!memcg) { /* per-node kswapd */
		param->pgdat = NODE_DATA(nid);
		if (param->pgdat->kswapd)
			return;
		pgdat->kswapd = kthread_run(param);
		..../* fatal error check */
		return;
	}
		
	/* per-memcg kswapd */
	kthread_run(param);
}

==

Secondaly, I think some macro is necessary.

How about
==
#define is_node_kswapd(param) (!param->memcg)

int kswapd(void *p)

{
	struct kswapd_param *param = p;

	if (is_node_kswapd(param))
		param->waitq = &param->pgdat->kswapd_wait;
	else
		param->waitq = mem_cgroup_get_kswapd_waitq(param->memcg);
		/* Here, we can notify the memcg which thread is for yours. */

or some ?

I think a macro like scanning_global_lru() is necessary.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#15 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

In reply to Ying Han
Quoted 2 lines · Ying Han

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup.

Could you please elaborate on this, what is adding? creating a thread?

The kswapd is sleeping in the wait queue headed at kswapd_wait

Quoted 9 lines

field of a kswapd descriptor. The kswapd descriptor stores information of node or cgroup and it allows the global and per cgroup background reclaim to share common reclaim algorithms.

This patch addes the kswapd descriptor and changes per zone kswapd_wait to the common data structure.

---

The performance data you posted earlier is helpful, do you have any additional insights on the the CPU overheads if any?

My general overall comment is that this patch needs to be refactored to bring out the change the patch makes.

Quoted 27 lines
 include/linux/mmzone.h |    3 +-
 include/linux/swap.h   |   10 +++++
 mm/memcontrol.c        |    2 +
 mm/mmzone.c            |    2 +-
 mm/page_alloc.c        |    9 +++-
 mm/vmscan.c            |   98 +++++++++++++++++++++++++++++++++--------------
 6 files changed, 90 insertions(+), 34 deletions(-)
include/linux/mmzone.h +1 -2
index 39c24eb..c77dfa2 100644
@@ -642,8 +642,7 @@ typedef struct pglist_data {
 	unsigned long node_spanned_pages; /* total size of physical page
 					     range, including holes */
 	int node_id;
-	wait_queue_head_t kswapd_wait;
-	struct task_struct *kswapd;
+	wait_queue_head_t *kswapd_wait;
 	int kswapd_max_order;
 } pg_data_t;
 
include/linux/swap.h +0 -0
index eba53e7..2e6cb58 100644

include/linux/swap.h static inline int current_is_kswapd(void) · line 32

Quoted 7 lines
include/linux/swap.h +4 -0
@@ -26,6 +26,16 @@ static inline int current_is_kswapd(void)
 	return current->flags & PF_KSWAPD;
 }
 
+struct kswapd {
+	struct task_struct *kswapd_task;
+	wait_queue_head_t kswapd_wait;
+	struct mem_cgroup *kswapd_mem;

Is this field being used anywhere in this patch?

Quoted 40 lines
include/linux/swap.h +6 -0
+	pg_data_t *kswapd_pgdat;
+};
+
+#define MAX_KSWAPDS MAX_NUMNODES
+extern struct kswapd kswapds[MAX_KSWAPDS];
+int kswapd(void *p);
 /*
  * MAX_SWAPFILES defines the maximum number of swaptypes: things which can
  * be swapped to.  The swap type and the offset into that swap type are
mm/memcontrol.c +2 -0
index a4034b6..dca3590 100644
@@ -263,6 +263,8 @@ struct mem_cgroup {
 	 */
 	struct mem_cgroup_stat_cpu nocpu_base;
 	spinlock_t pcp_counter_lock;
+
+	wait_queue_head_t *kswapd_wait;
 };
 
 /* Stuffs for move charges at task migration. */
mm/mmzone.c +1 -1
index e35bfb8..c7cbed5 100644
@@ -102,7 +102,7 @@ unsigned long zone_nr_free_pages(struct zone *zone)
 	 * free pages are low, get a better estimate for free pages
 	 */
 	if (nr_free_pages < zone->percpu_drift_mark &&
-			!waitqueue_active(&zone->zone_pgdat->kswapd_wait))
+			!waitqueue_active(zone->zone_pgdat->kswapd_wait))
 		return zone_page_state_snapshot(zone, NR_FREE_PAGES);
 
 	return nr_free_pages;
mm/page_alloc.c +0 -0
index b48dea2..a15bc1c 100644

mm/page_alloc.c static void __paginginit free_area_init_core(struct pglist_data *pgdat, · line 4073

Quoted 4 lines
mm/page_alloc.c +1 -0
@@ -4070,13 +4070,18 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat,
 	int nid = pgdat->node_id;
 	unsigned long zone_start_pfn = pgdat->node_start_pfn;
 	int ret;
+	struct kswapd *kswapd_p;

_p is sort of ugly, do we really need it?

mm/page_alloc.c static void __paginginit free_area_init_core(struct pglist_data *pgdat, · line 4079

Quoted 7 lines
mm/page_alloc.c +0 -2
 	pgdat_resize_init(pgdat);
 	pgdat->nr_zones = 0;
-	init_waitqueue_head(&pgdat->kswapd_wait);
 	pgdat->kswapd_max_order = 0;
 	pgdat_page_cgroup_init(pgdat);
-	

Thanks for the whitspace cleanup, but I don't know if that should be done here.

Quoted 35 lines
mm/page_alloc.c +6 -0
+
+	kswapd_p = &kswapds[nid];
+	init_waitqueue_head(&kswapd_p->kswapd_wait);
+	pgdat->kswapd_wait = &kswapd_p->kswapd_wait;
+	kswapd_p->kswapd_pgdat = pgdat;
+
 	for (j = 0; j < MAX_NR_ZONES; j++) {
 		struct zone *zone = pgdat->node_zones + j;
 		unsigned long size, realsize, memmap_pages;
mm/vmscan.c +7 -1
index b8a6fdc..e08005e 100644
@@ -2115,12 +2115,18 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem_cont,
 
 	return nr_reclaimed;
 }
+
 #endif
 
+DEFINE_SPINLOCK(kswapds_spinlock);
+struct kswapd kswapds[MAX_KSWAPDS];
+
 /* is kswapd sleeping prematurely? */
-static int sleeping_prematurely(pg_data_t *pgdat, int order, long remaining)
+static int sleeping_prematurely(struct kswapd *kswapd, int order,
+				long remaining)
 {
 	int i;
+	pg_data_t *pgdat = kswapd->kswapd_pgdat;
 
 	/* If a direct reclaimer woke kswapd within HZ/10, it's premature */
 	if (remaining)
@@ -2377,21 +2383,28 @@ out:
  * If there are applications that are active memory-allocators

mm/vmscan.c out: · line 2391

Quoted 10 lines
mm/vmscan.c +4 -2
  * (most normal use), this basically shouldn't matter.
  */
-static int kswapd(void *p)
+int kswapd(void *p)
 {
 	unsigned long order;
-	pg_data_t *pgdat = (pg_data_t*)p;
+	struct kswapd *kswapd_p = (struct kswapd *)p;
+	pg_data_t *pgdat = kswapd_p->kswapd_pgdat;
+	struct mem_cgroup *mem = kswapd_p->kswapd_mem;

Do we use mem anywhere?

mm/vmscan.c out: · line 2392

Quoted 1 line
mm/vmscan.c +1 -0
+	wait_queue_head_t *wait_h = &kswapd_p->kswapd_wait;

_p, _h almost look like hungarian notation in reverse :)

Quoted 65 lines
mm/vmscan.c +24 -11
 	struct task_struct *tsk = current;
 	DEFINE_WAIT(wait);
 	struct reclaim_state reclaim_state = {
 		.reclaimed_slab = 0,
 	};
-	const struct cpumask *cpumask = cpumask_of_node(pgdat->node_id);
+	const struct cpumask *cpumask;
 
 	lockdep_set_current_reclaim_state(GFP_KERNEL);
 
-	if (!cpumask_empty(cpumask))
-		set_cpus_allowed_ptr(tsk, cpumask);
+	if (pgdat) {
+		BUG_ON(pgdat->kswapd_wait != wait_h);
+		cpumask = cpumask_of_node(pgdat->node_id);
+		if (!cpumask_empty(cpumask))
+			set_cpus_allowed_ptr(tsk, cpumask);
+	}
 	current->reclaim_state = &reclaim_state;
 
 	/*
@@ -2414,9 +2427,13 @@ static int kswapd(void *p)
 		unsigned long new_order;
 		int ret;
 
-		prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
-		new_order = pgdat->kswapd_max_order;
-		pgdat->kswapd_max_order = 0;
+		prepare_to_wait(wait_h, &wait, TASK_INTERRUPTIBLE);
+		if (pgdat) {
+			new_order = pgdat->kswapd_max_order;
+			pgdat->kswapd_max_order = 0;
+		} else
+			new_order = 0;
+
 		if (order < new_order) {
 			/*
 			 * Don't sleep if someone wants a larger 'order'
@@ -2428,10 +2445,12 @@ static int kswapd(void *p)
 				long remaining = 0;
 
 				/* Try to sleep for a short interval */
-				if (!sleeping_prematurely(pgdat, order, remaining)) {
+				if (!sleeping_prematurely(kswapd_p, order,
+							remaining)) {
 					remaining = schedule_timeout(HZ/10);
-					finish_wait(&pgdat->kswapd_wait, &wait);
-					prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
+					finish_wait(wait_h, &wait);
+					prepare_to_wait(wait_h, &wait,
+							TASK_INTERRUPTIBLE);
 				}
 
 				/*
@@ -2439,20 +2458,25 @@ static int kswapd(void *p)
 				 * premature sleep. If not, then go fully
 				 * to sleep until explicitly woken up
 				 */
-				if (!sleeping_prematurely(pgdat, order, remaining)) {
-					trace_mm_vmscan_kswapd_sleep(pgdat->node_id);
+				if (!sleeping_prematurely(kswapd_p, order,
+								remaining)) {
+					if (pgdat)
+						trace_mm_vmscan_kswapd_sleep(
+								pgdat->node_id);

mm/vmscan.c static int kswapd(void *p) · line 2473

Quoted 10 lines
mm/vmscan.c +4 -2
 					schedule();
 				} else {
 					if (remaining)
-						count_vm_event(KSWAPD_LOW_WMARK_HIT_QUICKLY);
+						count_vm_event(
+						KSWAPD_LOW_WMARK_HIT_QUICKLY);
 					else
-						count_vm_event(KSWAPD_HIGH_WMARK_HIT_QUICKLY);
+						count_vm_event(
+						KSWAPD_HIGH_WMARK_HIT_QUICKLY);

Sorry, but the coding style hits me here do we really need to change this?

Quoted 46 lines
mm/vmscan.c +11 -6
 				}
 			}
-
-			order = pgdat->kswapd_max_order;
+			if (pgdat)
+				order = pgdat->kswapd_max_order;
 		}
-		finish_wait(&pgdat->kswapd_wait, &wait);
+		finish_wait(wait_h, &wait);
 
 		ret = try_to_freeze();
 		if (kthread_should_stop())
@@ -2476,6 +2500,7 @@ static int kswapd(void *p)
 void wakeup_kswapd(struct zone *zone, int order)
 {
 	pg_data_t *pgdat;
+	wait_queue_head_t *wait;
 
 	if (!populated_zone(zone))
 		return;
@@ -2488,9 +2513,10 @@ void wakeup_kswapd(struct zone *zone, int order)
 	trace_mm_vmscan_wakeup_kswapd(pgdat->node_id, zone_idx(zone), order);
 	if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL))
 		return;
-	if (!waitqueue_active(&pgdat->kswapd_wait))
+	wait = pgdat->kswapd_wait;
+	if (!waitqueue_active(wait))
 		return;
-	wake_up_interruptible(&pgdat->kswapd_wait);
+	wake_up_interruptible(wait);
 }
 
 /*
@@ -2587,7 +2613,10 @@ static int __devinit cpu_callback(struct notifier_block *nfb,
 
 			if (cpumask_any_and(cpu_online_mask, mask) < nr_cpu_ids)
 				/* One of our CPUs online: restore mask */
-				set_cpus_allowed_ptr(pgdat->kswapd, mask);
+				if (kswapds[nid].kswapd_task)
+					set_cpus_allowed_ptr(
+						kswapds[nid].kswapd_task,
+						mask);
 		}
 	}
 	return NOTIFY_OK;

mm/vmscan.c static int __devinit cpu_callback(struct notifier_block *nfb, · line 2631

Quoted 5 lines
mm/vmscan.c +1 -1
@@ -2599,19 +2628,20 @@ static int __devinit cpu_callback(struct notifier_block *nfb,
  */
 int kswapd_run(int nid)
 {
-	pg_data_t *pgdat = NODE_DATA(nid);
+	struct task_struct *thr;

thr is an ugly name for task_struct instance

Quoted 4 lines
mm/vmscan.c +1 -1
 	int ret = 0;
 
-	if (pgdat->kswapd)
+	if (kswapds[nid].kswapd_task)

mm/vmscan.c static int __devinit cpu_callback(struct notifier_block *nfb, · line 2642

Quoted 10 lines
mm/vmscan.c +2 -2
 		return 0;
 
-	pgdat->kswapd = kthread_run(kswapd, pgdat, "kswapd%d", nid);
-	if (IS_ERR(pgdat->kswapd)) {
+	thr = kthread_run(kswapd, &kswapds[nid], "kswapd%d", nid);
+	if (IS_ERR(thr)) {
 		/* failure at boot is fatal */
 		BUG_ON(system_state == SYSTEM_BOOTING);
 		printk("Failed to start kswapd on node %d\n",nid);
 		ret = -1;

What happens to the threads started?

Quoted 10 lines
mm/vmscan.c +1 -1
 	}
+	kswapds[nid].kswapd_task = thr;
 	return ret;
 }
 
@@ -2620,10 +2650,20 @@ int kswapd_run(int nid)
  */
 void kswapd_stop(int nid)
 {
-	struct task_struct *kswapd = NODE_DATA(nid)->kswapd;

mm/vmscan.c int kswapd_run(int nid) · line 2662

Quoted 10 lines
mm/vmscan.c +10 -0
+	struct task_struct *thr;
+	struct kswapd *kswapd_p;
+	wait_queue_head_t *wait;
+
+	pg_data_t *pgdat = NODE_DATA(nid);
+
+	spin_lock(&kswapds_spinlock);
+	wait = pgdat->kswapd_wait;
+	kswapd_p = container_of(wait, struct kswapd, kswapd_wait);
+	thr = kswapd_p->kswapd_task;

Sorry, but thr is just an ugly name to use.

Quoted 18 lines
mm/vmscan.c +3 -2
+	spin_unlock(&kswapds_spinlock);
 
-	if (kswapd)
-		kthread_stop(kswapd);
+	if (thr)
+		kthread_stop(thr);
 }
 
 static int __init kswapd_init(void)
Signature · 8 lines
1.7.3.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Signature · 9 lines
	Three Cheers,
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#16 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 6 lines · Balbir Singh
Quoted 2 lines · Ying Han

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup.

Could you please elaborate on this, what is adding? creating a thread?

Ok, better descriptor on the V2.

Quoted 6 lines

The kswapd is sleeping in the wait queue headed at kswapd_wait

Quoted 4 lines

field of a kswapd descriptor. The kswapd descriptor stores information of node or cgroup and it allows the global and per cgroup background reclaim to share common reclaim algorithms.

Quoted 8 lines
Quoted 5 lines

This patch addes the kswapd descriptor and changes per zone kswapd_wait to the common data structure.

---

The performance data you posted earlier is helpful, do you have any additional insights on the the CPU overheads if any?

I haven't measured the kswapd cputime overhead, numbers will be posted on the next patch.

Quoted 34 lines

My general overall comment is that this patch needs to be refactored to bring out the change the patch makes.

Quoted 30 lines

 include/linux/mmzone.h |    3 +-
 include/linux/swap.h   |   10 +++++
 mm/memcontrol.c        |    2 +
 mm/mmzone.c            |    2 +-
 mm/page_alloc.c        |    9 +++-
 mm/vmscan.c            |   98 +++++++++++++++++++++++++++++++++--------------

 6 files changed, 90 insertions(+), 34 deletions(-)
include/linux/mmzone.h +0 -0
index 39c24eb..c77dfa2 100644
@@ -642,8 +642,7 @@ typedef struct pglist_data {

      unsigned long node_spanned_pages; /* total size of physical page range, including holes */ int node_id;
-     wait_queue_head_t kswapd_wait;
-     struct task_struct *kswapd;
+     wait_queue_head_t *kswapd_wait;
      int kswapd_max_order;
 } pg_data_t;

include/linux/swap.h +0 -0
index eba53e7..2e6cb58 100644
@@ -26,6 +26,16 @@ static inline int current_is_kswapd(void)
       return current->flags & PF_KSWAPD;
  }
 
Quoted 6 lines

include/linux/swap.h static inline int current_is_kswapd(void) · line 32

Quoted 4 lines
include/linux/swap.h +4 -0
+struct kswapd {
+     struct task_struct *kswapd_task;
+     wait_queue_head_t kswapd_wait;
+     struct mem_cgroup *kswapd_mem;

Is this field being used anywhere in this patch?

will move this to the patch3.

Quoted 35 lines
Quoted 35 lines
include/linux/swap.h +6 -0
+     pg_data_t *kswapd_pgdat;
+};
+
+#define MAX_KSWAPDS MAX_NUMNODES
+extern struct kswapd kswapds[MAX_KSWAPDS];
+int kswapd(void *p);
  /*
   * MAX_SWAPFILES defines the maximum number of swaptypes: things which can
   * be swapped to.  The swap type and the offset into that swap type are
mm/memcontrol.c +2 -0
index a4034b6..dca3590 100644
@@ -263,6 +263,8 @@ struct mem_cgroup {
        */
       struct mem_cgroup_stat_cpu nocpu_base;
       spinlock_t pcp_counter_lock;
+
+     wait_queue_head_t *kswapd_wait;
  };
 
  /* Stuffs for move charges at task migration. */
mm/mmzone.c +0 -0
index e35bfb8..c7cbed5 100644
@@ -102,7 +102,7 @@ unsigned long zone_nr_free_pages(struct zone *zone)
        * free pages are low, get a better estimate for free pages
        */

      if (nr_free_pages < zone->percpu_drift_mark &&
-                     !waitqueue_active(&zone->zone_pgdat->kswapd_wait))
+                     !waitqueue_active(zone->zone_pgdat->kswapd_wait)) return zone_page_state_snapshot(zone, NR_FREE_PAGES);

Quoted 12 lines

mm/page_alloc.c static void __paginginit free_area_init_core(struct pglist_data *pgdat, · line 4073

Quoted 10 lines

      return nr_free_pages;

mm/page_alloc.c +1 -0
index b48dea2..a15bc1c 100644
@@ -4070,13 +4070,18 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat,
       int nid = pgdat->node_id;
       unsigned long zone_start_pfn = pgdat->node_start_pfn;
       int ret;
+     struct kswapd *kswapd_p;

_p is sort of ugly, do we really need it?

will change.

Quoted 11 lines

mm/page_alloc.c static void __paginginit free_area_init_core(struct pglist_data *pgdat, · line 4079

Quoted 8 lines
mm/page_alloc.c +0 -2
       pgdat_resize_init(pgdat);
       pgdat->nr_zones = 0;
-     init_waitqueue_head(&pgdat->kswapd_wait);
       pgdat->kswapd_max_order = 0;
       pgdat_page_cgroup_init(pgdat);
-

Thanks for the whitspace cleanup, but I don't know if that should be done here.

done.

Quoted 32 lines
Quoted 32 lines
mm/page_alloc.c +6 -0
+
+     kswapd_p = &kswapds[nid];
+     init_waitqueue_head(&kswapd_p->kswapd_wait);
+     pgdat->kswapd_wait = &kswapd_p->kswapd_wait;
+     kswapd_p->kswapd_pgdat = pgdat;
+
       for (j = 0; j < MAX_NR_ZONES; j++) {
               struct zone *zone = pgdat->node_zones + j;
               unsigned long size, realsize, memmap_pages;
mm/vmscan.c +7 -1
index b8a6fdc..e08005e 100644
@@ -2115,12 +2115,18 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem_cont,
 
       return nr_reclaimed;
  }
+
  #endif
 
+DEFINE_SPINLOCK(kswapds_spinlock);
+struct kswapd kswapds[MAX_KSWAPDS];
+
  /* is kswapd sleeping prematurely? */
-static int sleeping_prematurely(pg_data_t *pgdat, int order, long remaining)
+static int sleeping_prematurely(struct kswapd *kswapd, int order,
+                             long remaining)
  {
       int i;
+     pg_data_t *pgdat = kswapd->kswapd_pgdat;
 
Quoted 16 lines
Quoted 4 lines
mm/vmscan.c +0 -0
       /* If a direct reclaimer woke kswapd within HZ/10, it's premature */
       if (remaining)
@@ -2377,21 +2383,28 @@ out:
   * If there are applications that are active memory-allocators

mm/vmscan.c out: · line 2391

Quoted 10 lines
mm/vmscan.c +4 -2
   * (most normal use), this basically shouldn't matter.
   */
-static int kswapd(void *p)
+int kswapd(void *p)
  {
       unsigned long order;
-     pg_data_t *pgdat = (pg_data_t*)p;
+     struct kswapd *kswapd_p = (struct kswapd *)p;
+     pg_data_t *pgdat = kswapd_p->kswapd_pgdat;
+     struct mem_cgroup *mem = kswapd_p->kswapd_mem;

Do we use mem anywhere?

move to patch3.

Quoted 75 lines

mm/vmscan.c out: · line 2392

Quoted 2 lines
mm/vmscan.c +1 -0
+     wait_queue_head_t *wait_h = &kswapd_p->kswapd_wait;

_p, _h almost look like hungarian notation in reverse :)

Quoted 70 lines
mm/vmscan.c +25 -12
       struct task_struct *tsk = current;
       DEFINE_WAIT(wait);
       struct reclaim_state reclaim_state = {
               .reclaimed_slab = 0,
       };
-     const struct cpumask *cpumask = cpumask_of_node(pgdat->node_id);
+     const struct cpumask *cpumask;
 
       lockdep_set_current_reclaim_state(GFP_KERNEL);
 
-     if (!cpumask_empty(cpumask))
-             set_cpus_allowed_ptr(tsk, cpumask);
+     if (pgdat) {
+             BUG_ON(pgdat->kswapd_wait != wait_h);
+             cpumask = cpumask_of_node(pgdat->node_id);
+             if (!cpumask_empty(cpumask))
+                     set_cpus_allowed_ptr(tsk, cpumask);
+     }
       current->reclaim_state = &reclaim_state;
 
       /*
@@ -2414,9 +2427,13 @@ static int kswapd(void *p)
               unsigned long new_order;
               int ret;
 
-             prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
-             new_order = pgdat->kswapd_max_order;
-             pgdat->kswapd_max_order = 0;
+             prepare_to_wait(wait_h, &wait, TASK_INTERRUPTIBLE);
+             if (pgdat) {
+                     new_order = pgdat->kswapd_max_order;
+                     pgdat->kswapd_max_order = 0;
+             } else
+                     new_order = 0;
+
               if (order < new_order) {
                       /*
                        * Don't sleep if someone wants a larger 'order'
@@ -2428,10 +2445,12 @@ static int kswapd(void *p)
                               long remaining = 0;
 
                               /* Try to sleep for a short interval */
-                             if (!sleeping_prematurely(pgdat, order, remaining)) {
+                             if (!sleeping_prematurely(kswapd_p, order,
+                                                     remaining)) {
                                       remaining = schedule_timeout(HZ/10);
-                                     finish_wait(&pgdat->kswapd_wait, &wait);
-                                     prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
+                                     finish_wait(wait_h, &wait);
+                                     prepare_to_wait(wait_h, &wait,
+                                                     TASK_INTERRUPTIBLE);
                               }
 
                               /*
@@ -2439,20 +2458,25 @@ static int kswapd(void *p)
                                * premature sleep. If not, then go fully
                                * to sleep until explicitly woken up
                                */
-                             if (!sleeping_prematurely(pgdat, order, remaining)) {
-                                     trace_mm_vmscan_kswapd_sleep(pgdat->node_id);
+                             if (!sleeping_prematurely(kswapd_p, order,
+                                                             remaining)) {
+                                     if (pgdat)
+                                             trace_mm_vmscan_kswapd_sleep(
+                                                             pgdat->node_id);
                                       schedule();
                               } else {
                                       if (remaining)
-                                             count_vm_event(KSWAPD_LOW_WMARK_HIT_QUICKLY);
+                                             count_vm_event(
Quoted 8 lines
Quoted 5 lines
mm/vmscan.c +1 -0
+                                             KSWAPD_LOW_WMARK_HIT_QUICKLY);

                                      else
-                                             count_vm_event(KSWAPD_HIGH_WMARK_HIT_QUICKLY);
+                                             count_vm_event(
+                                             KSWAPD_HIGH_WMARK_HIT_QUICKLY);

Sorry, but the coding style hits me here do we really need to change this?

done.

Quoted 71 lines
Quoted 47 lines

                              }
                      }
-
-                     order = pgdat->kswapd_max_order;
+                     if (pgdat)
+                             order = pgdat->kswapd_max_order; }
-             finish_wait(&pgdat->kswapd_wait, &wait);
+             finish_wait(wait_h, &wait);

              ret = try_to_freeze();
              if (kthread_should_stop())

mm/vmscan.c +8 -3
@@ -2476,6 +2500,7 @@ static int kswapd(void *p)
  void wakeup_kswapd(struct zone *zone, int order)
  {
       pg_data_t *pgdat;
+     wait_queue_head_t *wait;
 
       if (!populated_zone(zone))
               return;
@@ -2488,9 +2513,10 @@ void wakeup_kswapd(struct zone *zone, int order)
       trace_mm_vmscan_wakeup_kswapd(pgdat->node_id, zone_idx(zone), order);
       if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL))
               return;
-     if (!waitqueue_active(&pgdat->kswapd_wait))
+     wait = pgdat->kswapd_wait;
+     if (!waitqueue_active(wait))
               return;
-     wake_up_interruptible(&pgdat->kswapd_wait);
+     wake_up_interruptible(wait);
  }
 
  /*
@@ -2587,7 +2613,10 @@ static int __devinit cpu_callback(struct notifier_block *nfb,
 
                       if (cpumask_any_and(cpu_online_mask, mask) < nr_cpu_ids)
                               /* One of our CPUs online: restore mask */
-                             set_cpus_allowed_ptr(pgdat->kswapd, mask);
+                             if (kswapds[nid].kswapd_task)
+                                     set_cpus_allowed_ptr(
+                                             kswapds[nid].kswapd_task,
+                                             mask);
               }
       }
       return NOTIFY_OK;

mm/vmscan.c static int __devinit cpu_callback(struct notifier_block *nfb, · line 2631

Quoted 5 lines
mm/vmscan.c +1 -1
@@ -2599,19 +2628,20 @@ static int __devinit cpu_callback(struct notifier_block *nfb,
   */
  int kswapd_run(int nid)
  {
-     pg_data_t *pgdat = NODE_DATA(nid);
+     struct task_struct *thr;

thr is an ugly name for task_struct instance

Quoted 4 lines
mm/vmscan.c +1 -1
       int ret = 0;
 
-     if (pgdat->kswapd)
+     if (kswapds[nid].kswapd_task)

mm/vmscan.c static int __devinit cpu_callback(struct notifier_block *nfb, · line 2642

Quoted 10 lines
mm/vmscan.c +2 -2
               return 0;
 
-     pgdat->kswapd = kthread_run(kswapd, pgdat, "kswapd%d", nid);
-     if (IS_ERR(pgdat->kswapd)) {
+     thr = kthread_run(kswapd, &kswapds[nid], "kswapd%d", nid);
+     if (IS_ERR(thr)) {
               /* failure at boot is fatal */
               BUG_ON(system_state == SYSTEM_BOOTING);
               printk("Failed to start kswapd on node %d\n",nid);
               ret = -1;
Quoted 1 line

What happens to the threads started?

Can you elaborate this little bit more?

Quoted 44 lines
Quoted 11 lines
mm/vmscan.c +1 -1
       }
+     kswapds[nid].kswapd_task = thr;
       return ret;
  }
 
@@ -2620,10 +2650,20 @@ int kswapd_run(int nid)
   */
  void kswapd_stop(int nid)
  {
-     struct task_struct *kswapd = NODE_DATA(nid)->kswapd;

mm/vmscan.c int kswapd_run(int nid) · line 2662

Quoted 10 lines
mm/vmscan.c +10 -0
+     struct task_struct *thr;
+     struct kswapd *kswapd_p;
+     wait_queue_head_t *wait;
+
+     pg_data_t *pgdat = NODE_DATA(nid);
+
+     spin_lock(&kswapds_spinlock);
+     wait = pgdat->kswapd_wait;
+     kswapd_p = container_of(wait, struct kswapd, kswapd_wait);
+     thr = kswapd_p->kswapd_task;

Sorry, but thr is just an ugly name to use.

Quoted 12 lines
mm/vmscan.c +3 -3
+     spin_unlock(&kswapds_spinlock);
 
-     if (kswapd)
-             kthread_stop(kswapd);
+     if (thr)
+             kthread_stop(thr);
  }
 
  static int __init kswapd_init(void)
--

1.7.3.1

Quoted 6 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 4 lines

--
       Three Cheers,
       Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#17 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

In reply to Ying Han
Quoted 2 lines · Ying Han

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup.

What is considered a normal number of cgroups in production? 10, 50, 10000? If it's a really large number and all the cgroups kswapds wake at the same time, the zone LRU lock will be very heavily contended. Potentially there will also be a very large number of new IO sources. I confess I haven't read the thread yet so maybe this has already been thought of but it might make sense to have a 1:N relationship between kswapd and memcgroups and cycle between containers. The difficulty will be a latency between when kswapd wakes up and when a particular container is scanned. The closer the ratio is to 1:1, the less the latency will be but the higher the contenion on the LRU lock and IO will be.

Quoted 39 lines

The kswapd is sleeping in the wait queue headed at kswapd_wait field of a kswapd descriptor. The kswapd descriptor stores information of node or cgroup and it allows the global and per cgroup background reclaim to share common reclaim algorithms.

This patch addes the kswapd descriptor and changes per zone kswapd_wait to the common data structure.

---
 include/linux/mmzone.h |    3 +-
 include/linux/swap.h   |   10 +++++
 mm/memcontrol.c        |    2 +
 mm/mmzone.c            |    2 +-
 mm/page_alloc.c        |    9 +++-
 mm/vmscan.c            |   98 +++++++++++++++++++++++++++++++++--------------
 6 files changed, 90 insertions(+), 34 deletions(-)
include/linux/mmzone.h +1 -2
index 39c24eb..c77dfa2 100644
@@ -642,8 +642,7 @@ typedef struct pglist_data {
 	unsigned long node_spanned_pages; /* total size of physical page
 					     range, including holes */
 	int node_id;
-	wait_queue_head_t kswapd_wait;
-	struct task_struct *kswapd;
+	wait_queue_head_t *kswapd_wait;
 	int kswapd_max_order;
 } pg_data_t;
 
include/linux/swap.h +0 -0
index eba53e7..2e6cb58 100644
@@ -26,6 +26,16 @@ static inline int current_is_kswapd(void)
 	return current->flags & PF_KSWAPD;
 }

include/linux/swap.h static inline int current_is_kswapd(void) · line 37

Quoted 10 lines
include/linux/swap.h +9 -0
 
+struct kswapd {
+	struct task_struct *kswapd_task;
+	wait_queue_head_t kswapd_wait;
+	struct mem_cgroup *kswapd_mem;
+	pg_data_t *kswapd_pgdat;
+};
+
+#define MAX_KSWAPDS MAX_NUMNODES
+extern struct kswapd kswapds[MAX_KSWAPDS];

This is potentially very large for a static structure. Can they not be dynamically allocated and kept on a list? Yes, there will be a list walk involved if yonu need a particular structure but that looks like it's a rare operation at this point.

Quoted 60 lines
include/linux/swap.h +1 -0
+int kswapd(void *p);
 /*
  * MAX_SWAPFILES defines the maximum number of swaptypes: things which can
  * be swapped to.  The swap type and the offset into that swap type are
mm/memcontrol.c +2 -0
index a4034b6..dca3590 100644
@@ -263,6 +263,8 @@ struct mem_cgroup {
 	 */
 	struct mem_cgroup_stat_cpu nocpu_base;
 	spinlock_t pcp_counter_lock;
+
+	wait_queue_head_t *kswapd_wait;
 };
 
 /* Stuffs for move charges at task migration. */
mm/mmzone.c +1 -1
index e35bfb8..c7cbed5 100644
@@ -102,7 +102,7 @@ unsigned long zone_nr_free_pages(struct zone *zone)
 	 * free pages are low, get a better estimate for free pages
 	 */
 	if (nr_free_pages < zone->percpu_drift_mark &&
-			!waitqueue_active(&zone->zone_pgdat->kswapd_wait))
+			!waitqueue_active(zone->zone_pgdat->kswapd_wait))
 		return zone_page_state_snapshot(zone, NR_FREE_PAGES);
 
 	return nr_free_pages;
mm/page_alloc.c +7 -2
index b48dea2..a15bc1c 100644
@@ -4070,13 +4070,18 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat,
 	int nid = pgdat->node_id;
 	unsigned long zone_start_pfn = pgdat->node_start_pfn;
 	int ret;
+	struct kswapd *kswapd_p;
 
 	pgdat_resize_init(pgdat);
 	pgdat->nr_zones = 0;
-	init_waitqueue_head(&pgdat->kswapd_wait);
 	pgdat->kswapd_max_order = 0;
 	pgdat_page_cgroup_init(pgdat);
-	
+
+	kswapd_p = &kswapds[nid];
+	init_waitqueue_head(&kswapd_p->kswapd_wait);
+	pgdat->kswapd_wait = &kswapd_p->kswapd_wait;
+	kswapd_p->kswapd_pgdat = pgdat;
+
 	for (j = 0; j < MAX_NR_ZONES; j++) {
 		struct zone *zone = pgdat->node_zones + j;
 		unsigned long size, realsize, memmap_pages;
mm/vmscan.c +0 -0
index b8a6fdc..e08005e 100644

mm/vmscan.c unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem_cont, · line 2120

Quoted 6 lines
mm/vmscan.c +1 -0
@@ -2115,12 +2115,18 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem_cont,
 
 	return nr_reclaimed;
 }
+
 #endif
 

Unnecessary whitespace there.

mm/vmscan.c unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem_cont, · line 2130

Quoted 11 lines
mm/vmscan.c +6 -1
+DEFINE_SPINLOCK(kswapds_spinlock);
+struct kswapd kswapds[MAX_KSWAPDS];
+
 /* is kswapd sleeping prematurely? */
-static int sleeping_prematurely(pg_data_t *pgdat, int order, long remaining)
+static int sleeping_prematurely(struct kswapd *kswapd, int order,
+				long remaining)
 {
 	int i;
+	pg_data_t *pgdat = kswapd->kswapd_pgdat;
 

This will behave strangely. You are using information from a *node* to determine if the kswapd belonging to a cgroup should sleep or not. The risk is that a cgroup kswapd never goes to sleep because even when all of its pages are discarded, the node itself is still not balanced.

Quoted 56 lines
mm/vmscan.c +19 -8
 	/* If a direct reclaimer woke kswapd within HZ/10, it's premature */
 	if (remaining)
@@ -2377,21 +2383,28 @@ out:
  * If there are applications that are active memory-allocators
  * (most normal use), this basically shouldn't matter.
  */
-static int kswapd(void *p)
+int kswapd(void *p)
 {
 	unsigned long order;
-	pg_data_t *pgdat = (pg_data_t*)p;
+	struct kswapd *kswapd_p = (struct kswapd *)p;
+	pg_data_t *pgdat = kswapd_p->kswapd_pgdat;
+	struct mem_cgroup *mem = kswapd_p->kswapd_mem;
+	wait_queue_head_t *wait_h = &kswapd_p->kswapd_wait;
 	struct task_struct *tsk = current;
 	DEFINE_WAIT(wait);
 	struct reclaim_state reclaim_state = {
 		.reclaimed_slab = 0,
 	};
-	const struct cpumask *cpumask = cpumask_of_node(pgdat->node_id);
+	const struct cpumask *cpumask;
 
 	lockdep_set_current_reclaim_state(GFP_KERNEL);
 
-	if (!cpumask_empty(cpumask))
-		set_cpus_allowed_ptr(tsk, cpumask);
+	if (pgdat) {
+		BUG_ON(pgdat->kswapd_wait != wait_h);
+		cpumask = cpumask_of_node(pgdat->node_id);
+		if (!cpumask_empty(cpumask))
+			set_cpus_allowed_ptr(tsk, cpumask);
+	}
 	current->reclaim_state = &reclaim_state;
 
 	/*
@@ -2414,9 +2427,13 @@ static int kswapd(void *p)
 		unsigned long new_order;
 		int ret;
 
-		prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
-		new_order = pgdat->kswapd_max_order;
-		pgdat->kswapd_max_order = 0;
+		prepare_to_wait(wait_h, &wait, TASK_INTERRUPTIBLE);
+		if (pgdat) {
+			new_order = pgdat->kswapd_max_order;
+			pgdat->kswapd_max_order = 0;
+		} else
+			new_order = 0;
+
 		if (order < new_order) {
 			/*
 			 * Don't sleep if someone wants a larger 'order'
@@ -2428,10 +2445,12 @@ static int kswapd(void *p)
 				long remaining = 0;
 

mm/vmscan.c static int kswapd(void *p) · line 2453

Quoted 10 lines
mm/vmscan.c +5 -3
 				/* Try to sleep for a short interval */
-				if (!sleeping_prematurely(pgdat, order, remaining)) {
+				if (!sleeping_prematurely(kswapd_p, order,
+							remaining)) {
 					remaining = schedule_timeout(HZ/10);
-					finish_wait(&pgdat->kswapd_wait, &wait);
-					prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
+					finish_wait(wait_h, &wait);
+					prepare_to_wait(wait_h, &wait,
+							TASK_INTERRUPTIBLE);

It would be nice if patch 1 did nothing but move the wait queue outside of the node structure without any other functional change. It'll be then be far easier to review a patch that introduces background reclaim for containers.

Quoted 121 lines
mm/vmscan.c +38 -17
 				}
 
 				/*
@@ -2439,20 +2458,25 @@ static int kswapd(void *p)
 				 * premature sleep. If not, then go fully
 				 * to sleep until explicitly woken up
 				 */
-				if (!sleeping_prematurely(pgdat, order, remaining)) {
-					trace_mm_vmscan_kswapd_sleep(pgdat->node_id);
+				if (!sleeping_prematurely(kswapd_p, order,
+								remaining)) {
+					if (pgdat)
+						trace_mm_vmscan_kswapd_sleep(
+								pgdat->node_id);
 					schedule();
 				} else {
 					if (remaining)
-						count_vm_event(KSWAPD_LOW_WMARK_HIT_QUICKLY);
+						count_vm_event(
+						KSWAPD_LOW_WMARK_HIT_QUICKLY);
 					else
-						count_vm_event(KSWAPD_HIGH_WMARK_HIT_QUICKLY);
+						count_vm_event(
+						KSWAPD_HIGH_WMARK_HIT_QUICKLY);
 				}
 			}
-
-			order = pgdat->kswapd_max_order;
+			if (pgdat)
+				order = pgdat->kswapd_max_order;
 		}
-		finish_wait(&pgdat->kswapd_wait, &wait);
+		finish_wait(wait_h, &wait);
 
 		ret = try_to_freeze();
 		if (kthread_should_stop())
@@ -2476,6 +2500,7 @@ static int kswapd(void *p)
 void wakeup_kswapd(struct zone *zone, int order)
 {
 	pg_data_t *pgdat;
+	wait_queue_head_t *wait;
 
 	if (!populated_zone(zone))
 		return;
@@ -2488,9 +2513,10 @@ void wakeup_kswapd(struct zone *zone, int order)
 	trace_mm_vmscan_wakeup_kswapd(pgdat->node_id, zone_idx(zone), order);
 	if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL))
 		return;
-	if (!waitqueue_active(&pgdat->kswapd_wait))
+	wait = pgdat->kswapd_wait;
+	if (!waitqueue_active(wait))
 		return;
-	wake_up_interruptible(&pgdat->kswapd_wait);
+	wake_up_interruptible(wait);
 }
 
 /*
@@ -2587,7 +2613,10 @@ static int __devinit cpu_callback(struct notifier_block *nfb,
 
 			if (cpumask_any_and(cpu_online_mask, mask) < nr_cpu_ids)
 				/* One of our CPUs online: restore mask */
-				set_cpus_allowed_ptr(pgdat->kswapd, mask);
+				if (kswapds[nid].kswapd_task)
+					set_cpus_allowed_ptr(
+						kswapds[nid].kswapd_task,
+						mask);
 		}
 	}
 	return NOTIFY_OK;
@@ -2599,19 +2628,20 @@ static int __devinit cpu_callback(struct notifier_block *nfb,
  */
 int kswapd_run(int nid)
 {
-	pg_data_t *pgdat = NODE_DATA(nid);
+	struct task_struct *thr;
 	int ret = 0;
 
-	if (pgdat->kswapd)
+	if (kswapds[nid].kswapd_task)
 		return 0;
 
-	pgdat->kswapd = kthread_run(kswapd, pgdat, "kswapd%d", nid);
-	if (IS_ERR(pgdat->kswapd)) {
+	thr = kthread_run(kswapd, &kswapds[nid], "kswapd%d", nid);
+	if (IS_ERR(thr)) {
 		/* failure at boot is fatal */
 		BUG_ON(system_state == SYSTEM_BOOTING);
 		printk("Failed to start kswapd on node %d\n",nid);
 		ret = -1;
 	}
+	kswapds[nid].kswapd_task = thr;
 	return ret;
 }
 
@@ -2620,10 +2650,20 @@ int kswapd_run(int nid)
  */
 void kswapd_stop(int nid)
 {
-	struct task_struct *kswapd = NODE_DATA(nid)->kswapd;
+	struct task_struct *thr;
+	struct kswapd *kswapd_p;
+	wait_queue_head_t *wait;
+
+	pg_data_t *pgdat = NODE_DATA(nid);
+
+	spin_lock(&kswapds_spinlock);
+	wait = pgdat->kswapd_wait;
+	kswapd_p = container_of(wait, struct kswapd, kswapd_wait);
+	thr = kswapd_p->kswapd_task;
+	spin_unlock(&kswapds_spinlock);
 
-	if (kswapd)
-		kthread_stop(kswapd);
+	if (thr)
+		kthread_stop(thr);
 }
 
 static int __init kswapd_init(void)
Signature · 1 line
1.7.3.1
Signature · 10 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#18 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 5 lines · Mel Gorman
Quoted 2 lines · Ying Han

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup.

What is considered a normal number of cgroups in production? 10, 50, 10000?

Normally it is less than 100. I assume there is a cap of number of cgroups can be created
per system.

If it's a really large number and all the cgroups kswapds wake at the same time,

Quoted 1 line

the zone LRU lock will be very heavily contended.

Thanks for reviewing the patch~

Agree. The zone->lru_lock is another thing we are looking at. Eventually, we need to break the lock to
per-zone per-memcg lru.

Potentially there will

Quoted 7 lines

also be a very large number of new IO sources. I confess I haven't read the thread yet so maybe this has already been thought of but it might make sense to have a 1:N relationship between kswapd and memcgroups and cycle between containers. The difficulty will be a latency between when kswapd wakes up and when a particular container is scanned. The closer the ratio is to 1:1, the less the latency will be but the higher the contenion on the LRU lock and IO will be.

No, we weren't talked about the mapping anywhere in the thread. Having many kswapd threads
at the same time isn't a problem as long as no locking contention ( ext, 1k kswapd threads on
1k fake numa node system). So breaking the zone->lru_lock should work.

Quoted 125 lines
Quoted 40 lines

The kswapd is sleeping in the wait queue headed at kswapd_wait field of a kswapd descriptor. The kswapd descriptor stores information of node or cgroup and it allows the global and per cgroup background reclaim to share common reclaim algorithms.

This patch addes the kswapd descriptor and changes per zone kswapd_wait to the common data structure.

---
 include/linux/mmzone.h |    3 +-
 include/linux/swap.h   |   10 +++++
 mm/memcontrol.c        |    2 +
 mm/mmzone.c            |    2 +-
 mm/page_alloc.c        |    9 +++-
 mm/vmscan.c            |   98 +++++++++++++++++++++++++++++++++--------------

 6 files changed, 90 insertions(+), 34 deletions(-)
include/linux/mmzone.h +0 -0
index 39c24eb..c77dfa2 100644
@@ -642,8 +642,7 @@ typedef struct pglist_data {

      unsigned long node_spanned_pages; /* total size of physical page range, including holes */ int node_id;
-     wait_queue_head_t kswapd_wait;
-     struct task_struct *kswapd;
+     wait_queue_head_t *kswapd_wait;
      int kswapd_max_order;
 } pg_data_t;

include/linux/swap.h +0 -0
index eba53e7..2e6cb58 100644
@@ -26,6 +26,16 @@ static inline int current_is_kswapd(void)
       return current->flags & PF_KSWAPD;
  }

include/linux/swap.h static inline int current_is_kswapd(void) · line 37

Quoted 10 lines
include/linux/swap.h +9 -0
 
+struct kswapd {
+     struct task_struct *kswapd_task;
+     wait_queue_head_t kswapd_wait;
+     struct mem_cgroup *kswapd_mem;
+     pg_data_t *kswapd_pgdat;
+};
+
+#define MAX_KSWAPDS MAX_NUMNODES
+extern struct kswapd kswapds[MAX_KSWAPDS];

This is potentially very large for a static structure. Can they not be dynamically allocated and kept on a list? Yes, there will be a list walk involved if yonu need a particular structure but that looks like it's a rare operation at this point.

Quoted 60 lines
include/linux/swap.h +1 -0
+int kswapd(void *p);
  /*
   * MAX_SWAPFILES defines the maximum number of swaptypes: things which can
   * be swapped to.  The swap type and the offset into that swap type are
mm/memcontrol.c +2 -0
index a4034b6..dca3590 100644
@@ -263,6 +263,8 @@ struct mem_cgroup {
        */
       struct mem_cgroup_stat_cpu nocpu_base;
       spinlock_t pcp_counter_lock;
+
+     wait_queue_head_t *kswapd_wait;
  };
 
  /* Stuffs for move charges at task migration. */
mm/mmzone.c +0 -0
index e35bfb8..c7cbed5 100644
@@ -102,7 +102,7 @@ unsigned long zone_nr_free_pages(struct zone *zone)
        * free pages are low, get a better estimate for free pages
        */

      if (nr_free_pages < zone->percpu_drift_mark &&
-                     !waitqueue_active(&zone->zone_pgdat->kswapd_wait))
+                     !waitqueue_active(zone->zone_pgdat->kswapd_wait)) return zone_page_state_snapshot(zone, NR_FREE_PAGES);

      return nr_free_pages;

mm/page_alloc.c +7 -2
index b48dea2..a15bc1c 100644
@@ -4070,13 +4070,18 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat,
       int nid = pgdat->node_id;
       unsigned long zone_start_pfn = pgdat->node_start_pfn;
       int ret;
+     struct kswapd *kswapd_p;
 
       pgdat_resize_init(pgdat);
       pgdat->nr_zones = 0;
-     init_waitqueue_head(&pgdat->kswapd_wait);
       pgdat->kswapd_max_order = 0;
       pgdat_page_cgroup_init(pgdat);
-
+
+     kswapd_p = &kswapds[nid];
+     init_waitqueue_head(&kswapd_p->kswapd_wait);
+     pgdat->kswapd_wait = &kswapd_p->kswapd_wait;
+     kswapd_p->kswapd_pgdat = pgdat;
+
       for (j = 0; j < MAX_NR_ZONES; j++) {
               struct zone *zone = pgdat->node_zones + j;
               unsigned long size, realsize, memmap_pages;
mm/vmscan.c +0 -0
index b8a6fdc..e08005e 100644

mm/vmscan.c unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem_cont, · line 2120

Quoted 6 lines
mm/vmscan.c +1 -0
@@ -2115,12 +2115,18 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem_cont,
 
       return nr_reclaimed;
  }
+
  #endif
 

Unnecessary whitespace there.

Quoted 14 lines

mm/vmscan.c unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem_cont, · line 2130

Quoted 11 lines
mm/vmscan.c +6 -1
+DEFINE_SPINLOCK(kswapds_spinlock);
+struct kswapd kswapds[MAX_KSWAPDS];
+
  /* is kswapd sleeping prematurely? */
-static int sleeping_prematurely(pg_data_t *pgdat, int order, long remaining)
+static int sleeping_prematurely(struct kswapd *kswapd, int order,
+                             long remaining)
  {
       int i;
+     pg_data_t *pgdat = kswapd->kswapd_pgdat;
 

This will behave strangely. You are using information from a *node* to determine if the kswapd belonging to a cgroup should sleep or not.

The

Quoted 2 lines

risk is that a cgroup kswapd never goes to sleep because even when all of its pages are discarded, the node itself is still not balanced.

The kswapd descriptor is one per-node and one per-cgroup. I believe I have the logic on the later patch to separate them out. And per-cgroup kswapd is using the wmark calculated based on the limits. Like this:

static int sleeping_prematurely(struct kswapd *kswapd, int order,

Quoted 65 lines

------->------->------->-------long remaining)

-------if (mem) {
------->-------if (!mem_cgroup_watermark_ok(mem, CHARGE_WMARK_HIGH)) ------->------->-------return 1;
------->-------return 0;
-------}

Quoted 56 lines
mm/vmscan.c +19 -8
       /* If a direct reclaimer woke kswapd within HZ/10, it's premature */
       if (remaining)
@@ -2377,21 +2383,28 @@ out:
   * If there are applications that are active memory-allocators
   * (most normal use), this basically shouldn't matter.
   */
-static int kswapd(void *p)
+int kswapd(void *p)
  {
       unsigned long order;
-     pg_data_t *pgdat = (pg_data_t*)p;
+     struct kswapd *kswapd_p = (struct kswapd *)p;
+     pg_data_t *pgdat = kswapd_p->kswapd_pgdat;
+     struct mem_cgroup *mem = kswapd_p->kswapd_mem;
+     wait_queue_head_t *wait_h = &kswapd_p->kswapd_wait;
       struct task_struct *tsk = current;
       DEFINE_WAIT(wait);
       struct reclaim_state reclaim_state = {
               .reclaimed_slab = 0,
       };
-     const struct cpumask *cpumask = cpumask_of_node(pgdat->node_id);
+     const struct cpumask *cpumask;
 
       lockdep_set_current_reclaim_state(GFP_KERNEL);
 
-     if (!cpumask_empty(cpumask))
-             set_cpus_allowed_ptr(tsk, cpumask);
+     if (pgdat) {
+             BUG_ON(pgdat->kswapd_wait != wait_h);
+             cpumask = cpumask_of_node(pgdat->node_id);
+             if (!cpumask_empty(cpumask))
+                     set_cpus_allowed_ptr(tsk, cpumask);
+     }
       current->reclaim_state = &reclaim_state;
 
       /*
@@ -2414,9 +2427,13 @@ static int kswapd(void *p)
               unsigned long new_order;
               int ret;
 
-             prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
-             new_order = pgdat->kswapd_max_order;
-             pgdat->kswapd_max_order = 0;
+             prepare_to_wait(wait_h, &wait, TASK_INTERRUPTIBLE);
+             if (pgdat) {
+                     new_order = pgdat->kswapd_max_order;
+                     pgdat->kswapd_max_order = 0;
+             } else
+                     new_order = 0;
+
               if (order < new_order) {
                       /*
                        * Don't sleep if someone wants a larger 'order'
@@ -2428,10 +2445,12 @@ static int kswapd(void *p)
                               long remaining = 0;
 
Quoted 14 lines

mm/vmscan.c static int kswapd(void *p) · line 2453

Quoted 10 lines
mm/vmscan.c +5 -3
                               /* Try to sleep for a short interval */
-                             if (!sleeping_prematurely(pgdat, order, remaining)) {
+                             if (!sleeping_prematurely(kswapd_p, order,
+                                                     remaining)) {
                                       remaining = schedule_timeout(HZ/10);
-                                     finish_wait(&pgdat->kswapd_wait, &wait);
-                                     prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
+                                     finish_wait(wait_h, &wait);
+                                     prepare_to_wait(wait_h, &wait,
+                                                     TASK_INTERRUPTIBLE);

It would be nice if patch 1 did nothing but move the wait queue outside of the node structure without any other functional change. It'll be then be far easier to review a patch that introduces background reclaim for containers.

I will see if to splitting this patch into two.

--Ying

Quoted 118 lines
Quoted 118 lines
mm/vmscan.c +7 -3
                               }
 
                               /*
@@ -2439,20 +2458,25 @@ static int kswapd(void *p)
                                * premature sleep. If not, then go fully
                                * to sleep until explicitly woken up
                                */
-                             if (!sleeping_prematurely(pgdat, order, remaining)) {
-                                     trace_mm_vmscan_kswapd_sleep(pgdat->node_id);
+                             if (!sleeping_prematurely(kswapd_p, order,
+                                                             remaining)) {
+                                     if (pgdat)
+                                             trace_mm_vmscan_kswapd_sleep(
+                                                             pgdat->node_id);
                                       schedule();
                               } else {
                                       if (remaining)
-                                             count_vm_event(KSWAPD_LOW_WMARK_HIT_QUICKLY);
+                                             count_vm_event(
+                                             KSWAPD_LOW_WMARK_HIT_QUICKLY);

                                      else
-                                             count_vm_event(KSWAPD_HIGH_WMARK_HIT_QUICKLY);
+                                             count_vm_event(
+                                             KSWAPD_HIGH_WMARK_HIT_QUICKLY); }
                      }
-
-                     order = pgdat->kswapd_max_order;
+                     if (pgdat)
+                             order = pgdat->kswapd_max_order;
              }
-             finish_wait(&pgdat->kswapd_wait, &wait);
+             finish_wait(wait_h, &wait);

              ret = try_to_freeze();
              if (kthread_should_stop())

mm/vmscan.c +26 -10
@@ -2476,6 +2500,7 @@ static int kswapd(void *p)
  void wakeup_kswapd(struct zone *zone, int order)
  {
       pg_data_t *pgdat;
+     wait_queue_head_t *wait;
 
       if (!populated_zone(zone))
               return;
@@ -2488,9 +2513,10 @@ void wakeup_kswapd(struct zone *zone, int order)
       trace_mm_vmscan_wakeup_kswapd(pgdat->node_id, zone_idx(zone), order);
       if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL))
               return;
-     if (!waitqueue_active(&pgdat->kswapd_wait))
+     wait = pgdat->kswapd_wait;
+     if (!waitqueue_active(wait))
               return;
-     wake_up_interruptible(&pgdat->kswapd_wait);
+     wake_up_interruptible(wait);
  }
 
  /*
@@ -2587,7 +2613,10 @@ static int __devinit cpu_callback(struct notifier_block *nfb,
 
                       if (cpumask_any_and(cpu_online_mask, mask) < nr_cpu_ids)
                               /* One of our CPUs online: restore mask */
-                             set_cpus_allowed_ptr(pgdat->kswapd, mask);
+                             if (kswapds[nid].kswapd_task)
+                                     set_cpus_allowed_ptr(
+                                             kswapds[nid].kswapd_task,
+                                             mask);
               }
       }
       return NOTIFY_OK;
@@ -2599,19 +2628,20 @@ static int __devinit cpu_callback(struct notifier_block *nfb,
   */
  int kswapd_run(int nid)
  {
-     pg_data_t *pgdat = NODE_DATA(nid);
+     struct task_struct *thr;
       int ret = 0;
 
-     if (pgdat->kswapd)
+     if (kswapds[nid].kswapd_task)
               return 0;
 
-     pgdat->kswapd = kthread_run(kswapd, pgdat, "kswapd%d", nid);
-     if (IS_ERR(pgdat->kswapd)) {
+     thr = kthread_run(kswapd, &kswapds[nid], "kswapd%d", nid);
+     if (IS_ERR(thr)) {
               /* failure at boot is fatal */
               BUG_ON(system_state == SYSTEM_BOOTING);
               printk("Failed to start kswapd on node %d\n",nid);
               ret = -1;
       }
+     kswapds[nid].kswapd_task = thr;
       return ret;
  }
 
@@ -2620,10 +2650,20 @@ int kswapd_run(int nid)
   */
  void kswapd_stop(int nid)
  {
-     struct task_struct *kswapd = NODE_DATA(nid)->kswapd;
+     struct task_struct *thr;
+     struct kswapd *kswapd_p;
+     wait_queue_head_t *wait;
+
+     pg_data_t *pgdat = NODE_DATA(nid);
+
+     spin_lock(&kswapds_spinlock);
+     wait = pgdat->kswapd_wait;
+     kswapd_p = container_of(wait, struct kswapd, kswapd_wait);
+     thr = kswapd_p->kswapd_task;
+     spin_unlock(&kswapds_spinlock);
 
-     if (kswapd)
-             kthread_stop(kswapd);
+     if (thr)
+             kthread_stop(thr);
  }
 
Quoted 10 lines
Quoted 4 lines
mm/vmscan.c +0 -1
  static int __init kswapd_init(void)
--

1.7.3.1

--
Mel Gorman
Part-time Phd Student                          Linux Technology Center University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#19 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 11 lines · On Tue, 7 Dec 2010 09:28:01 -0800 Ying Han <[email protected]> wrote:

On Tue, Dec 7, 2010 at 4:33 AM, Mel Gorman <[email protected]> wrote:

Potentially there will

Quoted 7 lines

also be a very large number of new IO sources. I confess I haven't read the thread yet so maybe this has already been thought of but it might make sense to have a 1:N relationship between kswapd and memcgroups and cycle between containers. The difficulty will be a latency between when kswapd wakes up and when a particular container is scanned. The closer the ratio is to 1:1, the less the latency will be but the higher the contenion on the LRU lock and IO will be.

Quoted 6 lines

No, we weren't talked about the mapping anywhere in the thread. Having many kswapd threads
at the same time isn't a problem as long as no locking contention ( ext, 1k kswapd threads on
1k fake numa node system). So breaking the zone->lru_lock should work.

That's me who make zone->lru_lock be shared. And per-memcg lock will makes the maintainance of memcg very bad. That will add many races.
Or we need to make memcg's LRU not synchronized with zone's LRU, IOW, we need to have completely independent LRU.

I'd like to limit the number of kswapd-for-memcg if zone->lru lock contention is problematic. memcg _can_ work without background reclaim.

How about adding per-node kswapd-for-memcg it will reclaim pages by a memcg's request ? as

	memcg_wake_kswapd(struct mem_cgroup *mem) 
	{
		do {
			nid = select_victim_node(mem);
			/* ask kswapd to reclaim memcg's memory */
			ret = memcg_kswapd_queue_work(nid, mem); /* may return -EBUSY if very busy*/
		} while()
	}

This will make lock contention minimum. Anyway, using too much cpu for this unnecessary_but_good_for_performance_function is bad. Throttoling is required.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#20 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 42 lines · KAMEZAWA Hiroyuki
Quoted 11 lines · On Tue, 7 Dec 2010 09:28:01 -0800 Ying Han <[email protected]> wrote:

On Tue, Dec 7, 2010 at 4:33 AM, Mel Gorman <[email protected]> wrote:

Potentially there will

Quoted 7 lines

also be a very large number of new IO sources. I confess I haven't read the thread yet so maybe this has already been thought of but it might make sense to have a 1:N relationship between kswapd and memcgroups and cycle between containers. The difficulty will be a latency between when kswapd wakes up and when a particular container is scanned. The closer the ratio is to 1:1, the less the latency will be but the higher the contenion on the LRU lock and IO will be.

Quoted 6 lines

No, we weren't talked about the mapping anywhere in the thread. Having many kswapd threads
at the same time isn't a problem as long as no locking contention ( ext, 1k kswapd threads on
1k fake numa node system). So breaking the zone->lru_lock should work.

That's me who make zone->lru_lock be shared. And per-memcg lock will makes the maintainance of memcg very bad. That will add many races.
Or we need to make memcg's LRU not synchronized with zone's LRU, IOW, we need to have completely independent LRU.

I'd like to limit the number of kswapd-for-memcg if zone->lru lock contention is problematic. memcg _can_ work without background reclaim.

How about adding per-node kswapd-for-memcg it will reclaim pages by a memcg's request ? as

       memcg_wake_kswapd(struct mem_cgroup *mem)
       {
               do {
                       nid = select_victim_node(mem);
                       /* ask kswapd to reclaim memcg's memory */
                       ret = memcg_kswapd_queue_work(nid, mem); /* may return -EBUSY if very busy*/ } while()
       }

Quoted 2 lines

This will make lock contention minimum. Anyway, using too much cpu for this unnecessary_but_good_for_performance_function is bad. Throttoling is required.

I don't see the problem of one-kswapd-per-cgroup here since there will be no performance cost if they are not running.

I haven't measured the lock contention and cputime for each kswapd running. Theoretically it would be a problem
if thousands of cgroups are configured on the the host and all of them are under memory pressure.

We can either optimize the locking or make each kswapd smarter (hold the lock less time). My current plan is to have the one-kswapd-per-cgroup on the V2 patch w/ select_victim_node, and the optimization for this comes as following patchset.

--Ying

Quoted 5 lines

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#21 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 44 lines · On Tue, 7 Dec 2010 17:24:12 -0800 Ying Han <[email protected]> wrote:
Quoted 42 lines · KAMEZAWA Hiroyuki
Quoted 11 lines · On Tue, 7 Dec 2010 09:28:01 -0800 Ying Han <[email protected]> wrote:

On Tue, Dec 7, 2010 at 4:33 AM, Mel Gorman <[email protected]> wrote:

Potentially there will

Quoted 7 lines

also be a very large number of new IO sources. I confess I haven't read the thread yet so maybe this has already been thought of but it might make sense to have a 1:N relationship between kswapd and memcgroups and cycle between containers. The difficulty will be a latency between when kswapd wakes up and when a particular container is scanned. The closer the ratio is to 1:1, the less the latency will be but the higher the contenion on the LRU lock and IO will be.

Quoted 6 lines

No, we weren't talked about the mapping anywhere in the thread. Having many kswapd threads
at the same time isn't a problem as long as no locking contention ( ext, 1k kswapd threads on
1k fake numa node system). So breaking the zone->lru_lock should work.

That's me who make zone->lru_lock be shared. And per-memcg lock will makes the maintainance of memcg very bad. That will add many races.
Or we need to make memcg's LRU not synchronized with zone's LRU, IOW, we need to have completely independent LRU.

I'd like to limit the number of kswapd-for-memcg if zone->lru lock contention is problematic. memcg _can_ work without background reclaim.

How about adding per-node kswapd-for-memcg it will reclaim pages by a memcg's request ? as

A  A  A  A memcg_wake_kswapd(struct mem_cgroup *mem)
A  A  A  A {
A  A  A  A  A  A  A  A do {
A  A  A  A  A  A  A  A  A  A  A  A nid = select_victim_node(mem);
A  A  A  A  A  A  A  A  A  A  A  A /* ask kswapd to reclaim memcg's memory */
A  A  A  A  A  A  A  A  A  A  A  A ret = memcg_kswapd_queue_work(nid, mem); /* may return -EBUSY if very busy*/
A  A  A  A  A  A  A  A } while()
A  A  A  A }
Quoted 6 lines
Quoted 2 lines

This will make lock contention minimum. Anyway, using too much cpu for this unnecessary_but_good_for_performance_function is bad. Throttoling is required.

I don't see the problem of one-kswapd-per-cgroup here since there will be no performance cost if they are not running.

Yes. But we've got a report from user who uses 2000+ cgroups on his host, one year ago. (in libcgroup mailing list.)

So, running 2000+ deadly thread will be bad. It's cost. In theory, the number of memcg can be 65534.

Quoted 5 lines

I haven't measured the lock contention and cputime for each kswapd running. Theoretically it would be a problem
if thousands of cgroups are configured on the the host and all of them are under memory pressure.

I think that's a configuration mistake.

Quoted 5 lines

We can either optimize the locking or make each kswapd smarter (hold the lock less time). My current plan is to have the one-kswapd-per-cgroup on the V2 patch w/ select_victim_node, and the optimization for this comes as following patchset.

My point above is holding remove node's lock, touching remote node's page increases memory reclaim cost very much. Then, I like per-node approach.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#22 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 24 lines · KAMEZAWA Hiroyuki
Quoted 21 lines · On Tue, 7 Dec 2010 17:24:12 -0800 Ying Han <[email protected]> wrote:
Quoted 19 lines · KAMEZAWA Hiroyuki
Quoted 16 lines · On Tue, 7 Dec 2010 09:28:01 -0800 Ying Han <[email protected]> wrote:

On Tue, Dec 7, 2010 at 4:33 AM, Mel Gorman <[email protected]> wrote:

Potentially there will

Quoted 7 lines

also be a very large number of new IO sources. I confess I haven't read the thread yet so maybe this has already been thought of but it might make sense to have a 1:N relationship between kswapd and memcgroups and cycle between containers. The difficulty will be a latency between when kswapd wakes up and when a particular container is scanned. The closer the ratio is to 1:1, the less the latency will be but the higher the contenion on the LRU lock and IO will be.

No, we weren't talked about the mapping anywhere in the thread. Having many kswapd threads
at the same time isn't a problem as long as no locking contention ( ext, 1k kswapd threads on
1k fake numa node system). So breaking the zone->lru_lock should work

#23 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 17 lines · On Tue, 7 Dec 2010 18:10:11 -0800 Ying Han <[email protected]> wrote:
Quoted 8 lines
Quoted 6 lines

I haven't measured the lock contention and cputime for each kswapd running. Theoretically it would be a problem
if thousands of cgroups are configured on the the host and all of them are under memory pressure.

I think that's a configuration mistake.

Quoted 8 lines
Quoted 5 lines

We can either optimize the locking or make each kswapd smarter (hold the lock less time). My current plan is to have the one-kswapd-per-cgroup on the V2 patch w/ select_victim_node, and the optimization for this comes as following patchset.

My point above is holding remove node's lock, touching remote node's page increases memory reclaim cost very much. Then, I like per-node approach.

Quoted 6 lines

So in a case of one physical node and thousands of cgroups, we are queuing all the works into single kswapd
which is doing the global background reclaim as well. This could be a problem on a multi-core system where
all the cgroups queuing behind the current work being throttle which might not be necessary.

percpu thread is enough. And there is direct reclaim, absense of kswapd will not be critical (because memcg doesn't need 'zone balancing'). And as you said, 'usual' users will not use 100+ cgroups. Queueing will not be fatal, I think.

Quoted 5 lines

I am not sure which way is better at this point. I would like to keep the current implementation for the next post V2
since smaller changes between versions sounds better to me.

yes, please go ahread. I'm not against the functionality itself.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#24 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

In reply to Ying Han
Quoted 44 lines · Ying Han
Quoted 42 lines · KAMEZAWA Hiroyuki
Quoted 11 lines · On Tue, 7 Dec 2010 09:28:01 -0800 Ying Han <[email protected]> wrote:

On Tue, Dec 7, 2010 at 4:33 AM, Mel Gorman <[email protected]> wrote:

Potentially there will

Quoted 7 lines

also be a very large number of new IO sources. I confess I haven't read the thread yet so maybe this has already been thought of but it might make sense to have a 1:N relationship between kswapd and memcgroups and cycle between containers. The difficulty will be a latency between when kswapd wakes up and when a particular container is scanned. The closer the ratio is to 1:1, the less the latency will be but the higher the contenion on the LRU lock and IO will be.

Quoted 6 lines

No, we weren't talked about the mapping anywhere in the thread. Having many kswapd threads
at the same time isn't a problem as long as no locking contention ( ext, 1k kswapd threads on
1k fake numa node system). So breaking the zone->lru_lock should work.

That's me who make zone->lru_lock be shared. And per-memcg lock will makes the maintainance of memcg very bad. That will add many races.
Or we need to make memcg's LRU not synchronized with zone's LRU, IOW, we need to have completely independent LRU.

I'd like to limit the number of kswapd-for-memcg if zone->lru lock contention is problematic. memcg _can_ work without background reclaim.

How about adding per-node kswapd-for-memcg it will reclaim pages by a memcg's request ? as

       memcg_wake_kswapd(struct mem_cgroup *mem)
       {
               do {
                       nid = select_victim_node(mem);
                       /* ask kswapd to reclaim memcg's memory */
                       ret = memcg_kswapd_queue_work(nid, mem); /* may return -EBUSY if very busy*/
               } while()
       }
Quoted 6 lines
Quoted 2 lines

This will make lock contention minimum. Anyway, using too much cpu for this unnecessary_but_good_for_performance_function is bad. Throttoling is required.

I don't see the problem of one-kswapd-per-cgroup here since there will be no performance cost if they are not running.

*If* they are not running. There is potentially a massive cost here.

Quoted 5 lines

I haven't measured the lock contention and cputime for each kswapd running. Theoretically it would be a problem
if thousands of cgroups are configured on the the host and all of them are under memory pressure.

It's not just the locking. If all of these kswapds are running and each container has a small number of dirty pages, we potentially have tens or hundreds of kswapd each queueing a small number of pages for IO. Granted, if we reach the point where these IO sources are delegated to flusher threads it would be less of a problem but it's not how things currently behave.

Quoted 2 lines

We can either optimize the locking or make each kswapd smarter (hold the lock less time).

Holding the lock less time might allow other kswapd instances to make small amounts of progress but they'll still be wasting a lot of CPU spinning on the lock. It's not a simple issue which is why I think we need either a) a means of telling kswapd which containers it should be reclaiming from or b) a 1:N mapping of kswapd instances to containers from the outset. Otherwise users with large numbers of containers will see severe slowdowns under memory pressure where as previously they would have experienced stalls in individual containers.

Quoted 4 lines

My current plan is to have the
one-kswapd-per-cgroup on the V2 patch w/ select_victim_node, and the optimization for this comes as following patchset.

Will read when they come out :)

Signature · 10 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#25 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

In reply to Ying Han
Quoted 10 lines · Mel Gorman
Quoted 5 lines · Ying Han
Quoted 2 lines

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup.

What is considered a normal number of cgroups in production? 10, 50, 10000?

Normally it is less than 100. I assume there is a cap of number of cgroups can be created
per system.

Quoted 8 lines

If it's a really large number and all the cgroups kswapds wake at the same time,

Quoted 1 line

the zone LRU lock will be very heavily contended.

Thanks for reviewing the patch~

Agree. The zone->lru_lock is another thing we are looking at. Eventually, we need to break the lock to
per-zone per-memcg lru.

This may make following bad scenario. That's the reason why now we are using zone->lru_lock.

1) start memcg reclaim
2) found the lru tail page has pte access bit
3) memcg reclaim decided that the page move to active list of memcg-lru.
Also, pte access bit was cleaned. But, the page still remain inactive list of global-lru.
4) Sadly, global reclaim discard the page quickly because it has been lost accessed bit by memcg.

But, if we have to modify both memcg and global LRU, we can't avoid zone->lru_lock anyway. Then, we don't use memcg special lock.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#26 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

In reply to Mel Gorman
Quoted 55 lines · Mel Gorman
Quoted 2 lines · Ying Han

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup.

What is considered a normal number of cgroups in production? 10, 50, 10000? If it's a really large number and all the cgroups kswapds wake at the same time, the zone LRU lock will be very heavily contended.  Potentially there will also be a very large number of new IO sources. I confess I haven't read the thread yet so maybe this has already been thought of but it might make sense to have a 1:N relationship between kswapd and memcgroups and cycle between containers. The difficulty will be a latency between when kswapd wakes up and when a particular container is scanned. The closer the ratio is to 1:1, the less the latency will be but the higher the contenion on the LRU lock and IO will be.

Quoted 40 lines

The kswapd is sleeping in the wait queue headed at kswapd_wait field of a kswapd descriptor. The kswapd descriptor stores information of node or cgroup and it allows the global and per cgroup background reclaim to share common reclaim algorithms.

This patch addes the kswapd descriptor and changes per zone kswapd_wait to the common data structure.

---
 include/linux/mmzone.h |    3 +-
 include/linux/swap.h   |   10 +++++
 mm/memcontrol.c        |    2 +
 mm/mmzone.c            |    2 +-
 mm/page_alloc.c        |    9 +++-
 mm/vmscan.c            |   98 +++++++++++++++++++++++++++++++++--------------

 6 files changed, 90 insertions(+), 34 deletions(-)
include/linux/mmzone.h +0 -0
index 39c24eb..c77dfa2 100644
@@ -642,8 +642,7 @@ typedef struct pglist_data {

      unsigned long node_spanned_pages; /* total size of physical page range, including holes */ int node_id;
-     wait_queue_head_t kswapd_wait;
-     struct task_struct *kswapd;
+     wait_queue_head_t *kswapd_wait;
      int kswapd_max_order;
 } pg_data_t;

include/linux/swap.h +0 -0
index eba53e7..2e6cb58 100644
@@ -26,6 +26,16 @@ static inline int current_is_kswapd(void)
       return current->flags & PF_KSWAPD;
  }
 
Quoted 14 lines

include/linux/swap.h static inline int current_is_kswapd(void) · line 37

Quoted 9 lines
include/linux/swap.h +9 -0
+struct kswapd {
+     struct task_struct *kswapd_task;
+     wait_queue_head_t kswapd_wait;
+     struct mem_cgroup *kswapd_mem;
+     pg_data_t *kswapd_pgdat;
+};
+
+#define MAX_KSWAPDS MAX_NUMNODES
+extern struct kswapd kswapds[MAX_KSWAPDS];

This is potentially very large for a static structure. Can they not be dynamically allocated and kept on a list? Yes, there will be a list walk involved if yonu need a particular structure but that looks like it's a rare operation at this point.

This has been changed to dynamic allocation on the V2 I am working on. The kswapd descriptor is dynamic allocated at kswapd_run for both per-node and per-cgroup kswapd, and there is no list walking since it could be identified by container_of(wait, struct kswapd, kswapd_wait);

--Ying

Quoted 275 lines
Quoted 61 lines
include/linux/swap.h +1 -0
+int kswapd(void *p);
  /*
   * MAX_SWAPFILES defines the maximum number of swaptypes: things which can
   * be swapped to.  The swap type and the offset into that swap type are
mm/memcontrol.c +2 -0
index a4034b6..dca3590 100644
@@ -263,6 +263,8 @@ struct mem_cgroup {
        */
       struct mem_cgroup_stat_cpu nocpu_base;
       spinlock_t pcp_counter_lock;
+
+     wait_queue_head_t *kswapd_wait;
  };
 
  /* Stuffs for move charges at task migration. */
mm/mmzone.c +0 -0
index e35bfb8..c7cbed5 100644
@@ -102,7 +102,7 @@ unsigned long zone_nr_free_pages(struct zone *zone)
        * free pages are low, get a better estimate for free pages
        */

      if (nr_free_pages < zone->percpu_drift_mark &&
-                     !waitqueue_active(&zone->zone_pgdat->kswapd_wait))
+                     !waitqueue_active(zone->zone_pgdat->kswapd_wait)) return zone_page_state_snapshot(zone, NR_FREE_PAGES);

      return nr_free_pages;

mm/page_alloc.c +7 -2
index b48dea2..a15bc1c 100644
@@ -4070,13 +4070,18 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat,
       int nid = pgdat->node_id;
       unsigned long zone_start_pfn = pgdat->node_start_pfn;
       int ret;
+     struct kswapd *kswapd_p;
 
       pgdat_resize_init(pgdat);
       pgdat->nr_zones = 0;
-     init_waitqueue_head(&pgdat->kswapd_wait);
       pgdat->kswapd_max_order = 0;
       pgdat_page_cgroup_init(pgdat);
-
+
+     kswapd_p = &kswapds[nid];
+     init_waitqueue_head(&kswapd_p->kswapd_wait);
+     pgdat->kswapd_wait = &kswapd_p->kswapd_wait;
+     kswapd_p->kswapd_pgdat = pgdat;
+
       for (j = 0; j < MAX_NR_ZONES; j++) {
               struct zone *zone = pgdat->node_zones + j;
               unsigned long size, realsize, memmap_pages;
mm/vmscan.c +0 -0
index b8a6fdc..e08005e 100644

mm/vmscan.c unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem_cont, · line 2120

Quoted 6 lines
mm/vmscan.c +1 -0
@@ -2115,12 +2115,18 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem_cont,
 
       return nr_reclaimed;
  }
+
  #endif
 

Unnecessary whitespace there.

mm/vmscan.c unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem_cont, · line 2130

Quoted 11 lines
mm/vmscan.c +6 -1
+DEFINE_SPINLOCK(kswapds_spinlock);
+struct kswapd kswapds[MAX_KSWAPDS];
+
  /* is kswapd sleeping prematurely? */
-static int sleeping_prematurely(pg_data_t *pgdat, int order, long remaining)
+static int sleeping_prematurely(struct kswapd *kswapd, int order,
+                             long remaining)
  {
       int i;
+     pg_data_t *pgdat = kswapd->kswapd_pgdat;
 

This will behave strangely. You are using information from a *node* to determine if the kswapd belonging to a cgroup should sleep or not. The risk is that a cgroup kswapd never goes to sleep because even when all of its pages are discarded, the node itself is still not balanced.

Quoted 56 lines
mm/vmscan.c +19 -8
       /* If a direct reclaimer woke kswapd within HZ/10, it's premature */
       if (remaining)
@@ -2377,21 +2383,28 @@ out:
   * If there are applications that are active memory-allocators
   * (most normal use), this basically shouldn't matter.
   */
-static int kswapd(void *p)
+int kswapd(void *p)
  {
       unsigned long order;
-     pg_data_t *pgdat = (pg_data_t*)p;
+     struct kswapd *kswapd_p = (struct kswapd *)p;
+     pg_data_t *pgdat = kswapd_p->kswapd_pgdat;
+     struct mem_cgroup *mem = kswapd_p->kswapd_mem;
+     wait_queue_head_t *wait_h = &kswapd_p->kswapd_wait;
       struct task_struct *tsk = current;
       DEFINE_WAIT(wait);
       struct reclaim_state reclaim_state = {
               .reclaimed_slab = 0,
       };
-     const struct cpumask *cpumask = cpumask_of_node(pgdat->node_id);
+     const struct cpumask *cpumask;
 
       lockdep_set_current_reclaim_state(GFP_KERNEL);
 
-     if (!cpumask_empty(cpumask))
-             set_cpus_allowed_ptr(tsk, cpumask);
+     if (pgdat) {
+             BUG_ON(pgdat->kswapd_wait != wait_h);
+             cpumask = cpumask_of_node(pgdat->node_id);
+             if (!cpumask_empty(cpumask))
+                     set_cpus_allowed_ptr(tsk, cpumask);
+     }
       current->reclaim_state = &reclaim_state;
 
       /*
@@ -2414,9 +2427,13 @@ static int kswapd(void *p)
               unsigned long new_order;
               int ret;
 
-             prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
-             new_order = pgdat->kswapd_max_order;
-             pgdat->kswapd_max_order = 0;
+             prepare_to_wait(wait_h, &wait, TASK_INTERRUPTIBLE);
+             if (pgdat) {
+                     new_order = pgdat->kswapd_max_order;
+                     pgdat->kswapd_max_order = 0;
+             } else
+                     new_order = 0;
+
               if (order < new_order) {
                       /*
                        * Don't sleep if someone wants a larger 'order'
@@ -2428,10 +2445,12 @@ static int kswapd(void *p)
                               long remaining = 0;
 

mm/vmscan.c static int kswapd(void *p) · line 2453

Quoted 10 lines
mm/vmscan.c +5 -3
                               /* Try to sleep for a short interval */
-                             if (!sleeping_prematurely(pgdat, order, remaining)) {
+                             if (!sleeping_prematurely(kswapd_p, order,
+                                                     remaining)) {
                                       remaining = schedule_timeout(HZ/10);
-                                     finish_wait(&pgdat->kswapd_wait, &wait);
-                                     prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
+                                     finish_wait(wait_h, &wait);
+                                     prepare_to_wait(wait_h, &wait,
+                                                     TASK_INTERRUPTIBLE);

It would be nice if patch 1 did nothing but move the wait queue outside of the node structure without any other functional change. It'll be then be far easier to review a patch that introduces background reclaim for containers.

Quoted 117 lines
mm/vmscan.c +7 -3
                               }
 
                               /*
@@ -2439,20 +2458,25 @@ static int kswapd(void *p)
                                * premature sleep. If not, then go fully
                                * to sleep until explicitly woken up
                                */
-                             if (!sleeping_prematurely(pgdat, order, remaining)) {
-                                     trace_mm_vmscan_kswapd_sleep(pgdat->node_id);
+                             if (!sleeping_prematurely(kswapd_p, order,
+                                                             remaining)) {
+                                     if (pgdat)
+                                             trace_mm_vmscan_kswapd_sleep(
+                                                             pgdat->node_id);
                                       schedule();
                               } else {
                                       if (remaining)
-                                             count_vm_event(KSWAPD_LOW_WMARK_HIT_QUICKLY);
+                                             count_vm_event(
+                                             KSWAPD_LOW_WMARK_HIT_QUICKLY);

                                      else
-                                             count_vm_event(KSWAPD_HIGH_WMARK_HIT_QUICKLY);
+                                             count_vm_event(
+                                             KSWAPD_HIGH_WMARK_HIT_QUICKLY); }
                      }
-
-                     order = pgdat->kswapd_max_order;
+                     if (pgdat)
+                             order = pgdat->kswapd_max_order;
              }
-             finish_wait(&pgdat->kswapd_wait, &wait);
+             finish_wait(wait_h, &wait);

              ret = try_to_freeze();
              if (kthread_should_stop())

mm/vmscan.c +26 -10
@@ -2476,6 +2500,7 @@ static int kswapd(void *p)
  void wakeup_kswapd(struct zone *zone, int order)
  {
       pg_data_t *pgdat;
+     wait_queue_head_t *wait;
 
       if (!populated_zone(zone))
               return;
@@ -2488,9 +2513,10 @@ void wakeup_kswapd(struct zone *zone, int order)
       trace_mm_vmscan_wakeup_kswapd(pgdat->node_id, zone_idx(zone), order);
       if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL))
               return;
-     if (!waitqueue_active(&pgdat->kswapd_wait))
+     wait = pgdat->kswapd_wait;
+     if (!waitqueue_active(wait))
               return;
-     wake_up_interruptible(&pgdat->kswapd_wait);
+     wake_up_interruptible(wait);
  }
 
  /*
@@ -2587,7 +2613,10 @@ static int __devinit cpu_callback(struct notifier_block *nfb,
 
                       if (cpumask_any_and(cpu_online_mask, mask) < nr_cpu_ids)
                               /* One of our CPUs online: restore mask */
-                             set_cpus_allowed_ptr(pgdat->kswapd, mask);
+                             if (kswapds[nid].kswapd_task)
+                                     set_cpus_allowed_ptr(
+                                             kswapds[nid].kswapd_task,
+                                             mask);
               }
       }
       return NOTIFY_OK;
@@ -2599,19 +2628,20 @@ static int __devinit cpu_callback(struct notifier_block *nfb,
   */
  int kswapd_run(int nid)
  {
-     pg_data_t *pgdat = NODE_DATA(nid);
+     struct task_struct *thr;
       int ret = 0;
 
-     if (pgdat->kswapd)
+     if (kswapds[nid].kswapd_task)
               return 0;
 
-     pgdat->kswapd = kthread_run(kswapd, pgdat, "kswapd%d", nid);
-     if (IS_ERR(pgdat->kswapd)) {
+     thr = kthread_run(kswapd, &kswapds[nid], "kswapd%d", nid);
+     if (IS_ERR(thr)) {
               /* failure at boot is fatal */
               BUG_ON(system_state == SYSTEM_BOOTING);
               printk("Failed to start kswapd on node %d\n",nid);
               ret = -1;
       }
+     kswapds[nid].kswapd_task = thr;
       return ret;
  }
 
@@ -2620,10 +2650,20 @@ int kswapd_run(int nid)
   */
  void kswapd_stop(int nid)
  {
-     struct task_struct *kswapd = NODE_DATA(nid)->kswapd;
+     struct task_struct *thr;
+     struct kswapd *kswapd_p;
+     wait_queue_head_t *wait;
+
+     pg_data_t *pgdat = NODE_DATA(nid);
+
+     spin_lock(&kswapds_spinlock);
+     wait = pgdat->kswapd_wait;
+     kswapd_p = container_of(wait, struct kswapd, kswapd_wait);
+     thr = kswapd_p->kswapd_task;
+     spin_unlock(&kswapds_spinlock);
 
-     if (kswapd)
-             kthread_stop(kswapd);
+     if (thr)
+             kthread_stop(thr);
  }
 
Quoted 10 lines
Quoted 4 lines
mm/vmscan.c +0 -1
  static int __init kswapd_init(void)
--

1.7.3.1

--
Mel Gorman
Part-time Phd Student                          Linux Technology Center University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#27 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

In reply to Mel Gorman
Quoted 6 lines
Quoted 6 lines
include/linux/swap.h +6 -0
+struct kswapd {
+	struct task_struct *kswapd_task;
+	wait_queue_head_t kswapd_wait;
+	struct mem_cgroup *kswapd_mem;
+	pg_data_t *kswapd_pgdat;
+};
Quoted 8 lines

include/linux/swap.h static inline int current_is_kswapd(void) · line 37

Quoted 3 lines
include/linux/swap.h +3 -0
+
+#define MAX_KSWAPDS MAX_NUMNODES
+extern struct kswapd kswapds[MAX_KSWAPDS];

This is potentially very large for a static structure. Can they not be dynamically allocated and kept on a list? Yes, there will be a list walk involved if yonu need a particular structure but that looks like it's a rare operation at this point.

Why can't we use normal workqueue mechanism?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#28 lore raw

Re: [PATCH 1/4] Add kswapd descriptor.

Quoted 15 lines · On Wed, 8 Dec 2010 16:22:30 +0900 (JST) KOSAKI Motohiro…
Quoted 6 lines
Quoted 6 lines
include/linux/swap.h +6 -0
+struct kswapd {
+	struct task_struct *kswapd_task;
+	wait_queue_head_t kswapd_wait;
+	struct mem_cgroup *kswapd_mem;
+	pg_data_t *kswapd_pgdat;
+};
Quoted 8 lines

include/linux/swap.h static inline int current_is_kswapd(void) · line 37

Quoted 3 lines
include/linux/swap.h +3 -0
+
+#define MAX_KSWAPDS MAX_NUMNODES
+extern struct kswapd kswapds[MAX_KSWAPDS];

This is potentially very large for a static structure. Can they not be dynamically allocated and kept on a list? Yes, there will be a list walk involved if yonu need a particular structure but that looks like it's a rare operation at this point.

Quoted 2 lines

Why can't we use normal workqueue mechanism?

Sounds much simpler than self management of threads.

My only concern is cpu-time-stealing by background works but ....

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#29 lore raw

PATCH 2/4 Add per cgroup reclaim watermarks.

In reply to Ying Han

The per cgroup kswapd is invoked at mem_cgroup_charge when the cgroup's memory usage above a threshold--low_wmark. Then the kswapd thread starts to reclaim pages in a priority loop similar to global algorithm. The kswapd is done if the memory usage below a threshold--high_wmark.

The per cgroup background reclaim is based on the per cgroup LRU and also adds per cgroup watermarks. There are two watermarks including "low_wmark" and "high_wmark", and they are calculated based on the limit_in_bytes(hard_limit) for each cgroup. Each time the hard_limit is change, the corresponding wmarks are re-calculated. Since memory controller charges only user pages, there is no need for a "min_wmark". The current calculation of wmarks is a function of "memory.min_free_kbytes" which could be adjusted by writing different values into the new api. This is added mainly for debugging purpose.

---
 include/linux/memcontrol.h  |    1 +
 include/linux/res_counter.h |   88 ++++++++++++++++++++++++++++++-
 kernel/res_counter.c        |   26 ++++++++--
 mm/memcontrol.c             |  123 +++++++++++++++++++++++++++++++++++++++++--
 mm/vmscan.c                 |   10 ++++
 5 files changed, 238 insertions(+), 10 deletions(-)
include/linux/memcontrol.h +1 -0
index 159a076..90fe7fe 100644
@@ -76,6 +76,7 @@ int task_in_mem_cgroup(struct task_struct *task, const struct mem_cgroup *mem);
 
 extern struct mem_cgroup *try_get_mem_cgroup_from_page(struct page *page);
 extern struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p);
+extern int mem_cgroup_watermark_ok(struct mem_cgroup *mem, int charge_flags);
 
 static inline
 int mm_match_cgroup(const struct mm_struct *mm, const struct mem_cgroup *cgroup)
include/linux/res_counter.h +86 -2

135 lines not on this page · raw message

kernel/res_counter.c +22 -4
index c7eaa37..a524349 100644
@@ -19,12 +19,26 @@ void res_counter_init(struct res_counter *counter, struct res_counter *parent)
 	spin_lock_init(&counter->lock);
 	counter->limit = RESOURCE_MAX;
 	counter->soft_limit = RESOURCE_MAX;
+	counter->low_wmark_limit = RESOURCE_MAX;
+	counter->high_wmark_limit = RESOURCE_MAX;
 	counter->parent = parent;
 }
 
-int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
+int res_counter_charge_locked(struct res_counter *counter, unsigned long val,
+				int charge_flags)
 {
-	if (counter->usage + val > counter->limit) {
+	unsigned long long limit = 0;
+
+	if (charge_flags & CHARGE_WMARK_LOW)
+		limit = counter->low_wmark_limit;
+
+	if (charge_flags & CHARGE_WMARK_HIGH)
+		limit = counter->high_wmark_limit;
+
+	if (charge_flags & CHARGE_WMARK_MIN)
+		limit = counter->limit;
+
+	if (counter->usage + val > limit) {
 		counter->failcnt++;
 		return -ENOMEM;
 	}
@@ -36,7 +50,7 @@ int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
 }
 
 int res_counter_charge(struct res_counter *counter, unsigned long val,
-			struct res_counter **limit_fail_at)
+			int charge_flags, struct res_counter **limit_fail_at)
 {
 	int ret;
 	unsigned long flags;
@@ -46,7 +60,7 @@ int res_counter_charge(struct res_counter *counter, unsigned long val,
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
-		ret = res_counter_charge_locked(c, val);
+		ret = res_counter_charge_locked(c, val, charge_flags);
 		spin_unlock(&c->lock);
 		if (ret < 0) {
 			*limit_fail_at = c;
@@ -103,6 +117,10 @@ res_counter_member(struct res_counter *counter, int member)
 		return &counter->failcnt;
 	case RES_SOFT_LIMIT:
 		return &counter->soft_limit;
+	case RES_LOW_WMARK_LIMIT:
+		return &counter->low_wmark_limit;
+	case RES_HIGH_WMARK_LIMIT:
+		return &counter->high_wmark_limit;
 	};
 
 	BUG();
mm/memcontrol.c +119 -4

189 lines not on this page · raw message

mm/vmscan.c +10 -0
index e08005e..6d5702b 100644
@@ -46,6 +46,8 @@
 
 #include <linux/swapops.h>
 
+#include <linux/res_counter.h>
+
 #include "internal.h"
 
 #define CREATE_TRACE_POINTS
@@ -2127,11 +2129,19 @@ static int sleeping_prematurely(struct kswapd *kswapd, int order,
 {
 	int i;
 	pg_data_t *pgdat = kswapd->kswapd_pgdat;
+	struct mem_cgroup *mem = kswapd->kswapd_mem;
 
 	/* If a direct reclaimer woke kswapd within HZ/10, it's premature */
 	if (remaining)
 		return 1;
 
+	if (mem) {
+		if (!mem_cgroup_watermark_ok(kswapd->kswapd_mem,
+						CHARGE_WMARK_HIGH))
+			return 1;
+		return 0;
+	}
+
 	/* If after HZ/10, a zone is below the high mark, it's premature */
 	for (i = 0; i < pgdat->nr_zones; i++) {
 		struct zone *zone = pgdat->node_zones + i;
Signature · 8 lines
1.7.3.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#30 lore raw

Re: [PATCH 2/4] Add per cgroup reclaim watermarks.

Quoted 14 lines · On Mon, 29 Nov 2010 22:49:43 -0800 Ying Han <[email protected]> wrote:

The per cgroup kswapd is invoked at mem_cgroup_charge when the cgroup's memory usage above a threshold--low_wmark. Then the kswapd thread starts to reclaim pages in a priority loop similar to global algorithm. The kswapd is done if the memory usage below a threshold--high_wmark.

The per cgroup background reclaim is based on the per cgroup LRU and also adds per cgroup watermarks. There are two watermarks including "low_wmark" and "high_wmark", and they are calculated based on the limit_in_bytes(hard_limit) for each cgroup. Each time the hard_limit is change, the corresponding wmarks are re-calculated. Since memory controller charges only user pages, there is no need for a "min_wmark". The current calculation of wmarks is a function of "memory.min_free_kbytes" which could be adjusted by writing different values into the new api. This is added mainly for debugging purpose.

Quoted 1 line

A few points.

1. I can understand the motivation for including low/high watermark to res_coutner. But, sadly, compareing all charge will make the counter slow. IMHO, as memory controller threshold-check or soft limit, checking usage periodically based on event counter is enough. It will be low cost.

2. min_free_kbytes must be automatically calculated. For example, max(3% of limit, 20MB) or some.

3. When you allow min_free_kbytes to be set by users, please compare it with the limit.
I think min_free_kbyte interface itself should be in another patch... interface code tends to make patch bigger.

Quoted 340 lines
---
 include/linux/memcontrol.h  |    1 +
 include/linux/res_counter.h |   88 ++++++++++++++++++++++++++++++-
 kernel/res_counter.c        |   26 ++++++++--
 mm/memcontrol.c             |  123 +++++++++++++++++++++++++++++++++++++++++--
 mm/vmscan.c                 |   10 ++++
 5 files changed, 238 insertions(+), 10 deletions(-)
include/linux/memcontrol.h +1 -0
index 159a076..90fe7fe 100644
@@ -76,6 +76,7 @@ int task_in_mem_cgroup(struct task_struct *task, const struct mem_cgroup *mem);
 
 extern struct mem_cgroup *try_get_mem_cgroup_from_page(struct page *page);
 extern struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p);
+extern int mem_cgroup_watermark_ok(struct mem_cgroup *mem, int charge_flags);
 
 static inline
 int mm_match_cgroup(const struct mm_struct *mm, const struct mem_cgroup *cgroup)
include/linux/res_counter.h +86 -2
index fcb9884..eed12c5 100644
@@ -39,6 +39,16 @@ struct res_counter {
 	 */
 	unsigned long long soft_limit;
 	/*
+	 * the limit that reclaim triggers. TODO: res_counter in mem
+	 * or wmark_limit.
+	 */
+	unsigned long long low_wmark_limit;
+	/*
+	 * the limit that reclaim stops. TODO: res_counter in mem or
+	 * wmark_limit.
+	 */
+	unsigned long long high_wmark_limit;
+	/*
 	 * the number of unsuccessful attempts to consume the resource
 	 */
 	unsigned long long failcnt;
@@ -55,6 +65,10 @@ struct res_counter {
 
 #define RESOURCE_MAX (unsigned long long)LLONG_MAX
 
+#define CHARGE_WMARK_MIN	0x01
+#define CHARGE_WMARK_LOW	0x02
+#define CHARGE_WMARK_HIGH	0x04
+
 /**
  * Helpers to interact with userspace
  * res_counter_read_u64() - returns the value of the specified member.
@@ -92,6 +106,8 @@ enum {
 	RES_LIMIT,
 	RES_FAILCNT,
 	RES_SOFT_LIMIT,
+	RES_LOW_WMARK_LIMIT,
+	RES_HIGH_WMARK_LIMIT
 };
 
 /*
@@ -112,9 +128,10 @@ void res_counter_init(struct res_counter *counter, struct res_counter *parent);
  */
 
 int __must_check res_counter_charge_locked(struct res_counter *counter,
-		unsigned long val);
+		unsigned long val, int charge_flags);
 int __must_check res_counter_charge(struct res_counter *counter,
-		unsigned long val, struct res_counter **limit_fail_at);
+		unsigned long val, int charge_flags,
+		struct res_counter **limit_fail_at);
 
 /*
  * uncharge - tell that some portion of the resource is released
@@ -145,6 +162,24 @@ static inline bool res_counter_soft_limit_check_locked(struct res_counter *cnt)
 	return false;
 }
 
+static inline bool
+res_counter_high_wmark_limit_check_locked(struct res_counter *cnt)
+{
+	if (cnt->usage < cnt->high_wmark_limit)
+		return true;
+
+	return false;
+}
+
+static inline bool
+res_counter_low_wmark_limit_check_locked(struct res_counter *cnt)
+{
+	if (cnt->usage < cnt->low_wmark_limit)
+		return true;
+
+	return false;
+}
+
 /**
  * Get the difference between the usage and the soft limit
  * @cnt: The counter
@@ -193,6 +228,30 @@ static inline bool res_counter_check_under_soft_limit(struct res_counter *cnt)
 	return ret;
 }
 
+static inline bool
+res_counter_check_under_low_wmark_limit(struct res_counter *cnt)
+{
+	bool ret;
+	unsigned long flags;
+
+	spin_lock_irqsave(&cnt->lock, flags);
+	ret = res_counter_low_wmark_limit_check_locked(cnt);
+	spin_unlock_irqrestore(&cnt->lock, flags);
+	return ret;
+}
+
+static inline bool
+res_counter_check_under_high_wmark_limit(struct res_counter *cnt)
+{
+	bool ret;
+	unsigned long flags;
+
+	spin_lock_irqsave(&cnt->lock, flags);
+	ret = res_counter_high_wmark_limit_check_locked(cnt);
+	spin_unlock_irqrestore(&cnt->lock, flags);
+	return ret;
+}
+
 static inline void res_counter_reset_max(struct res_counter *cnt)
 {
 	unsigned long flags;
@@ -220,6 +279,8 @@ static inline int res_counter_set_limit(struct res_counter *cnt,
 	spin_lock_irqsave(&cnt->lock, flags);
 	if (cnt->usage <= limit) {
 		cnt->limit = limit;
+		cnt->low_wmark_limit = limit;
+		cnt->high_wmark_limit = limit;
 		ret = 0;
 	}
 	spin_unlock_irqrestore(&cnt->lock, flags);
@@ -238,4 +299,27 @@ res_counter_set_soft_limit(struct res_counter *cnt,
 	return 0;
 }
 
+static inline int
+res_counter_set_high_wmark_limit(struct res_counter *cnt,
+				unsigned long long wmark_limit)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&cnt->lock, flags);
+	cnt->high_wmark_limit = wmark_limit;
+	spin_unlock_irqrestore(&cnt->lock, flags);
+	return 0;
+}
+
+static inline int
+res_counter_set_low_wmark_limit(struct res_counter *cnt,
+				unsigned long long wmark_limit)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&cnt->lock, flags);
+	cnt->low_wmark_limit = wmark_limit;
+	spin_unlock_irqrestore(&cnt->lock, flags);
+	return 0;
+}
 #endif
kernel/res_counter.c +22 -4
index c7eaa37..a524349 100644
@@ -19,12 +19,26 @@ void res_counter_init(struct res_counter *counter, struct res_counter *parent)
 	spin_lock_init(&counter->lock);
 	counter->limit = RESOURCE_MAX;
 	counter->soft_limit = RESOURCE_MAX;
+	counter->low_wmark_limit = RESOURCE_MAX;
+	counter->high_wmark_limit = RESOURCE_MAX;
 	counter->parent = parent;
 }
 
-int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
+int res_counter_charge_locked(struct res_counter *counter, unsigned long val,
+				int charge_flags)
 {
-	if (counter->usage + val > counter->limit) {
+	unsigned long long limit = 0;
+
+	if (charge_flags & CHARGE_WMARK_LOW)
+		limit = counter->low_wmark_limit;
+
+	if (charge_flags & CHARGE_WMARK_HIGH)
+		limit = counter->high_wmark_limit;
+
+	if (charge_flags & CHARGE_WMARK_MIN)
+		limit = counter->limit;
+
+	if (counter->usage + val > limit) {
 		counter->failcnt++;
 		return -ENOMEM;
 	}
@@ -36,7 +50,7 @@ int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
 }
 
 int res_counter_charge(struct res_counter *counter, unsigned long val,
-			struct res_counter **limit_fail_at)
+			int charge_flags, struct res_counter **limit_fail_at)
 {
 	int ret;
 	unsigned long flags;
@@ -46,7 +60,7 @@ int res_counter_charge(struct res_counter *counter, unsigned long val,
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
-		ret = res_counter_charge_locked(c, val);
+		ret = res_counter_charge_locked(c, val, charge_flags);
 		spin_unlock(&c->lock);
 		if (ret < 0) {
 			*limit_fail_at = c;
@@ -103,6 +117,10 @@ res_counter_member(struct res_counter *counter, int member)
 		return &counter->failcnt;
 	case RES_SOFT_LIMIT:
 		return &counter->soft_limit;
+	case RES_LOW_WMARK_LIMIT:
+		return &counter->low_wmark_limit;
+	case RES_HIGH_WMARK_LIMIT:
+		return &counter->high_wmark_limit;
 	};
 
 	BUG();
mm/memcontrol.c +62 -2
index dca3590..a0c6ed9 100644
@@ -265,6 +265,7 @@ struct mem_cgroup {
 	spinlock_t pcp_counter_lock;
 
 	wait_queue_head_t *kswapd_wait;
+	unsigned long min_free_kbytes;
 };
 
 /* Stuffs for move charges at task migration. */
@@ -370,6 +371,7 @@ static void mem_cgroup_get(struct mem_cgroup *mem);
 static void mem_cgroup_put(struct mem_cgroup *mem);
 static struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *mem);
 static void drain_all_stock_async(void);
+static unsigned long get_min_free_kbytes(struct mem_cgroup *mem);
 
 static struct mem_cgroup_per_zone *
 mem_cgroup_zoneinfo(struct mem_cgroup *mem, int nid, int zid)
@@ -796,6 +798,32 @@ static inline bool mem_cgroup_is_root(struct mem_cgroup *mem)
 	return (mem == root_mem_cgroup);
 }
 
+void setup_per_memcg_wmarks(struct mem_cgroup *mem)
+{
+	u64 limit;
+	unsigned long min_free_kbytes;
+
+	min_free_kbytes = get_min_free_kbytes(mem);
+	limit = mem_cgroup_get_limit(mem);
+	if (min_free_kbytes == 0) {
+		res_counter_set_low_wmark_limit(&mem->res, limit);
+		res_counter_set_high_wmark_limit(&mem->res, limit);
+	} else {
+		unsigned long page_min = min_free_kbytes >> (PAGE_SHIFT - 10);
+		unsigned long lowmem_pages = 2048;
+		unsigned long low_wmark, high_wmark;
+		u64 tmp;
+
+		tmp = (u64)page_min * limit;
+		do_div(tmp, lowmem_pages);
+
+		low_wmark = tmp + (tmp >> 1);
+		high_wmark = tmp + (tmp >> 2);
+		res_counter_set_low_wmark_limit(&mem->res, low_wmark);
+		res_counter_set_high_wmark_limit(&mem->res, high_wmark);
+	}
+}
+
 /*
  * Following LRU functions are allowed to be used without PCG_LOCK.
  * Operations are called by routine of global LRU independently from memcg.
@@ -1148,6 +1176,22 @@ static unsigned int get_swappiness(struct mem_cgroup *memcg)
 	return swappiness;
 }
 
+static unsigned long get_min_free_kbytes(struct mem_cgroup *memcg)
+{
+	struct cgroup *cgrp = memcg->css.cgroup;
+	unsigned long min_free_kbytes;
+
+	/* root ? */
+	if (cgrp == NULL || cgrp->parent == NULL)
+		return 0;
+
+	spin_lock(&memcg->reclaim_param_lock);
+	min_free_kbytes = memcg->min_free_kbytes;
+	spin_unlock(&memcg->reclaim_param_lock);
+
+	return min_free_kbytes;
+}
+
 static void mem_cgroup_start_move(struct mem_cgroup *mem)
 {
 	int cpu;
@@ -1844,12 +1888,13 @@ static int __mem_cgroup_do_charge(struct mem_cgroup *mem, gfp_t gfp_mask,
 	unsigned long flags = 0;
 	int ret;
 
-	ret = res_counter_charge(&mem->res, csize, &fail_res);
+	ret = res_counter_charge(&mem->res, csize, CHARGE_WMARK_MIN, &fail_res);
 
 	if (likely(!ret)) {
 		if (!do_swap_account)
 			return CHARGE_OK;
-		ret = res_counter_charge(&mem->memsw, csize, &fail_res);
+		ret = res_counter_charge(&mem->memsw, csize, CHARGE_WMARK_MIN,
+					&fail_res);
 		if (likely(!ret))
 			return CHARGE_OK;
 
@@ -3733,6 +3778,37 @@ static int mem_cgroup_swappiness_write(struct cgroup *cgrp, struct cftype *cft,
 	return 0;
 }
 
+static u64 mem_cgroup_min_free_read(struct cgroup *cgrp, struct cftype *cft)
+{
+	struct mem_cgroup *memcg = mem_cgroup_from_cont(cgrp);
+
+	return get_min_free_kbytes(memcg);
+}
+
+static int mem_cgroup_min_free_write(struct cgroup *cgrp, struct cftype *cfg,
+				     u64 val)
+{
+	struct mem_cgroup *memcg = mem_cgroup_from_cont(cgrp);
+	struct mem_cgroup *parent;
+
+	if (cgrp->parent == NULL)
+		return -EINVAL;

mm/memcontrol.c static int mem_cgroup_swappiness_write(struct cgroup *cgrp, struct cftype *cft, · line 3805

Quoted 10 lines
mm/memcontrol.c +10 -0
+
+	parent = mem_cgroup_from_cont(cgrp->parent);
+
+	cgroup_lock();
+
+	spin_lock(&memcg->reclaim_param_lock);
+	memcg->min_free_kbytes = val;
+	spin_unlock(&memcg->reclaim_param_lock);
+
+	cgroup_unlock();

Why cgroup_lock is required ?

Thanks,
-Kame

Quoted 119 lines
mm/memcontrol.c +47 -2
+
+	setup_per_memcg_wmarks(memcg);
+	return 0;
+
+}
+
 static void __mem_cgroup_threshold(struct mem_cgroup *memcg, bool swap)
 {
 	struct mem_cgroup_threshold_ary *t;
@@ -4024,6 +4100,21 @@ static void mem_cgroup_oom_unregister_event(struct cgroup *cgrp,
 	mutex_unlock(&memcg_oom_mutex);
 }
 
+static int mem_cgroup_wmark_read(struct cgroup *cgrp,
+	struct cftype *cft,  struct cgroup_map_cb *cb)
+{
+	struct mem_cgroup *mem = mem_cgroup_from_cont(cgrp);
+	unsigned long low_wmark, high_wmark;
+
+	low_wmark = res_counter_read_u64(&mem->res, RES_LOW_WMARK_LIMIT);
+	high_wmark = res_counter_read_u64(&mem->res, RES_HIGH_WMARK_LIMIT);
+
+	cb->fill(cb, "memcg_low_wmark", low_wmark);
+	cb->fill(cb, "memcg_high_wmark", high_wmark);
+
+	return 0;
+}
+
 static int mem_cgroup_oom_control_read(struct cgroup *cgrp,
 	struct cftype *cft,  struct cgroup_map_cb *cb)
 {
@@ -4127,6 +4218,15 @@ static struct cftype mem_cgroup_files[] = {
 		.unregister_event = mem_cgroup_oom_unregister_event,
 		.private = MEMFILE_PRIVATE(_OOM_TYPE, OOM_CONTROL),
 	},
+	{
+		.name = "min_free_kbytes",
+		.write_u64 = mem_cgroup_min_free_write,
+		.read_u64 = mem_cgroup_min_free_read,
+	},
+	{
+		.name = "reclaim_wmarks",
+		.read_map = mem_cgroup_wmark_read,
+	},
 };
 
 #ifdef CONFIG_CGROUP_MEM_RES_CTLR_SWAP
@@ -4308,6 +4408,19 @@ static void __init enable_swap_cgroup(void)
 }
 #endif
 
+int mem_cgroup_watermark_ok(struct mem_cgroup *mem,
+				int charge_flags)
+{
+	long ret = 0;
+
+	if (charge_flags & CHARGE_WMARK_LOW)
+		ret = res_counter_check_under_low_wmark_limit(&mem->res);
+	if (charge_flags & CHARGE_WMARK_HIGH)
+		ret = res_counter_check_under_high_wmark_limit(&mem->res);
+
+	return ret;
+}
+
 static int mem_cgroup_soft_limit_tree_init(void)
 {
 	struct mem_cgroup_tree_per_node *rtpn;
@@ -4450,10 +4563,12 @@ static int mem_cgroup_do_precharge(unsigned long count)
 		 * are still under the same cgroup_mutex. So we can postpone
 		 * css_get().
 		 */
-		if (res_counter_charge(&mem->res, PAGE_SIZE * count, &dummy))
+		if (res_counter_charge(&mem->res, PAGE_SIZE * count,
+					CHARGE_WMARK_MIN, &dummy))
 			goto one_by_one;
 		if (do_swap_account && res_counter_charge(&mem->memsw,
-						PAGE_SIZE * count, &dummy)) {
+						PAGE_SIZE * count,
+						CHARGE_WMARK_MIN, &dummy)) {
 			res_counter_uncharge(&mem->res, PAGE_SIZE * count);
 			goto one_by_one;
 		}
mm/vmscan.c +10 -0
index e08005e..6d5702b 100644
@@ -46,6 +46,8 @@
 
 #include <linux/swapops.h>
 
+#include <linux/res_counter.h>
+
 #include "internal.h"
 
 #define CREATE_TRACE_POINTS
@@ -2127,11 +2129,19 @@ static int sleeping_prematurely(struct kswapd *kswapd, int order,
 {
 	int i;
 	pg_data_t *pgdat = kswapd->kswapd_pgdat;
+	struct mem_cgroup *mem = kswapd->kswapd_mem;
 
 	/* If a direct reclaimer woke kswapd within HZ/10, it's premature */
 	if (remaining)
 		return 1;
 
+	if (mem) {
+		if (!mem_cgroup_watermark_ok(kswapd->kswapd_mem,
+						CHARGE_WMARK_HIGH))
+			return 1;
+		return 0;
+	}
+
 	/* If after HZ/10, a zone is below the high mark, it's premature */
 	for (i = 0; i < pgdat->nr_zones; i++) {
 		struct zone *zone = pgdat->node_zones + i;
Signature · 1 line
1.7.3.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#31 lore raw

Re: [PATCH 2/4] Add per cgroup reclaim watermarks.

Quoted 17 lines · KAMEZAWA Hiroyuki
Quoted 14 lines · On Mon, 29 Nov 2010 22:49:43 -0800 Ying Han <[email protected]> wrote:

The per cgroup kswapd is invoked at mem_cgroup_charge when the cgroup's memory usage above a threshold--low_wmark. Then the kswapd thread starts to reclaim pages in a priority loop similar to global algorithm. The kswapd is done if the memory usage below a threshold--high_wmark.

The per cgroup background reclaim is based on the per cgroup LRU and also adds per cgroup watermarks. There are two watermarks including "low_wmark" and "high_wmark", and they are calculated based on the limit_in_bytes(hard_limit) for each cgroup. Each time the hard_limit is change, the corresponding wmarks are re-calculated. Since memory controller charges only user pages, there is no need for a "min_wmark". The current calculation of wmarks is a function of "memory.min_free_kbytes" which could be adjusted by writing different values into the new api. This is added mainly for debugging purpose.

Quoted 8 lines
Quoted 1 line

A few points.

1. I can understand the motivation for including low/high watermark to res_coutner. But, sadly, compareing all charge will make the counter slow. IMHO, as memory controller threshold-check or soft limit, checking usage periodically based on event counter is enough. It will be low cost.

If we have other limits using the event counter, this sounds a feasible try for the
wmarks. I can look into that.

Quoted 3 lines

2. min_free_kbytes must be automatically calculated. For example, max(3% of limit, 20MB) or some.

Now the wmark is automatically calculated based on the limit. Adding the min_free_kbytes gives
us more flexibility to adjust the portion of the threshold. This could just be a performance tuning
parameter later. I need it now at least at the beginning before figuring out a reasonable calculation
formula.

Quoted 5 lines

3. When you allow min_free_kbytes to be set by users, please compare it with the limit.
  I think min_free_kbyte interface itself should be in another patch... interface code tends to make patch bigger.

Sounds feasible.

--Ying

Quoted 54 lines
Quoted 54 lines

---
 include/linux/memcontrol.h  |    1 +
 include/linux/res_counter.h |   88 ++++++++++++++++++++++++++++++- kernel/res_counter.c        |   26 ++++++++--
 mm/memcontrol.c             |  123 +++++++++++++++++++++++++++++++++++++++++-- mm/vmscan.c                 |   10 ++++

 5 files changed, 238 insertions(+), 10 deletions(-)
include/linux/memcontrol.h +1 -0
index 159a076..90fe7fe 100644
@@ -76,6 +76,7 @@ int task_in_mem_cgroup(struct task_struct *task, const struct mem_cgroup *mem);
 
  extern struct mem_cgroup *try_get_mem_cgroup_from_page(struct page *page);
  extern struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p);
+extern int mem_cgroup_watermark_ok(struct mem_cgroup *mem, int charge_flags);
 

 static inline
 int mm_match_cgroup(const struct mm_struct *mm, const struct mem_cgroup *cgroup)

include/linux/res_counter.h +14 -0
index fcb9884..eed12c5 100644
@@ -39,6 +39,16 @@ struct res_counter {
        */
       unsigned long long soft_limit;
       /*
+      * the limit that reclaim triggers. TODO: res_counter in mem
+      * or wmark_limit.
+      */
+     unsigned long long low_wmark_limit;
+     /*
+      * the limit that reclaim stops. TODO: res_counter in mem or
+      * wmark_limit.
+      */
+     unsigned long long high_wmark_limit;
+     /*
        * the number of unsuccessful attempts to consume the resource
        */
       unsigned long long failcnt;
@@ -55,6 +65,10 @@ struct res_counter {
 
  #define RESOURCE_MAX (unsigned long long)LLONG_MAX
 
+#define CHARGE_WMARK_MIN     0x01
+#define CHARGE_WMARK_LOW     0x02
+#define CHARGE_WMARK_HIGH    0x04
+
  /**
   * Helpers to interact with userspace
   * res_counter_read_u64() - returns the value of the specified member
#32 lore raw

Re: [PATCH 2/4] Add per cgroup reclaim watermarks.

Quoted 36 lines · On Tue, 30 Nov 2010 12:44:13 -0800 Ying Han <[email protected]> wrote:
Quoted 17 lines · KAMEZAWA Hiroyuki
Quoted 14 lines · On Mon, 29 Nov 2010 22:49:43 -0800 Ying Han <[email protected]> wrote:

The per cgroup kswapd is invoked at mem_cgroup_charge when the cgroup's memory usage above a threshold--low_wmark. Then the kswapd thread starts to reclaim pages in a priority loop similar to global algorithm. The kswapd is done if the memory usage below a threshold--high_wmark.

The per cgroup background reclaim is based on the per cgroup LRU and also adds per cgroup watermarks. There are two watermarks including "low_wmark" and "high_wmark", and they are calculated based on the limit_in_bytes(hard_limit) for each cgroup. Each time the hard_limit is change, the corresponding wmarks are re-calculated. Since memory controller charges only user pages, there is no need for a "min_wmark". The current calculation of wmarks is a function of "memory.min_free_kbytes" which could be adjusted by writing different values into the new api. This is added mainly for debugging purpose.

Quoted 8 lines
Quoted 1 line

A few points.

1. I can understand the motivation for including low/high watermark to

A  res_coutner. But, sadly, compareing all charge will make the counter slow.
A  IMHO, as memory controller threshold-check or soft limit, checking usage
A  periodically based on event counter is enough. It will be low cost.

If we have other limits using the event counter, this sounds a feasible try for the
wmarks. I can look into that.

Quoted 3 lines

2. min_free_kbytes must be automatically calculated. A For example, max(3% of limit, 20MB) or some.

Quoted 8 lines

Now the wmark is automatically calculated based on the limit. Adding the min_free_kbytes gives
us more flexibility to adjust the portion of the threshold. This could just be a performance tuning
parameter later. I need it now at least at the beginning before figuring out a reasonable calculation
formula.

mm/page_alloc.c::init_per_zone_wmark_min() can be reused.

My question is.

Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+void setup_per_memcg_wmarks(struct mem_cgroup *mem)
+{
+ A  A  u64 limit;
+ A  A  unsigned long min_free_kbytes;
+
+ A  A  min_free_kbytes = get_min_free_kbytes(mem);
+ A  A  limit = mem_cgroup_get_limit(mem);
+ A  A  if (min_free_kbytes == 0) {

I think this min_free_kbyte is always 0 until a user set it. Please set this when the limit is changed, automatically.

I wonder

	struct mem_cgroup {

		unsigned long min_free_kbytes;
		unsigned long min_free_kbytes_user_set; /* use this always if set */
	}

may be necessary if we never adjust min_free_kbytes once a user set it.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#33 lore raw

Re: [PATCH 2/4] Add per cgroup reclaim watermarks.

In reply to Ying Han
Quoted 28 lines · Ying Han

The per cgroup kswapd is invoked at mem_cgroup_charge when the cgroup's memory usage above a threshold--low_wmark. Then the kswapd thread starts to reclaim pages in a priority loop similar to global algorithm. The kswapd is done if the memory usage below a threshold--high_wmark.

The per cgroup background reclaim is based on the per cgroup LRU and also adds per cgroup watermarks. There are two watermarks including "low_wmark" and "high_wmark", and they are calculated based on the limit_in_bytes(hard_limit) for each cgroup. Each time the hard_limit is change, the corresponding wmarks are re-calculated. Since memory controller charges only user pages, there is no need for a "min_wmark". The current calculation of wmarks is a function of "memory.min_free_kbytes" which could be adjusted by writing different values into the new api. This is added mainly for debugging purpose.

---
 include/linux/memcontrol.h  |    1 +
 include/linux/res_counter.h |   88 ++++++++++++++++++++++++++++++-
 kernel/res_counter.c        |   26 ++++++++--
 mm/memcontrol.c             |  123 +++++++++++++++++++++++++++++++++++++++++--
 mm/vmscan.c                 |   10 ++++
 5 files changed, 238 insertions(+), 10 deletions(-)
include/linux/memcontrol.h +0 -0
index 159a076..90fe7fe 100644

include/linux/memcontrol.h int task_in_mem_cgroup(struct task_struct *task, const struct mem_cgroup *mem); · line 80

Quoted 5 lines
include/linux/memcontrol.h +1 -0
@@ -76,6 +76,7 @@ int task_in_mem_cgroup(struct task_struct *task, const struct mem_cgroup *mem);
 
 extern struct mem_cgroup *try_get_mem_cgroup_from_page(struct page *page);
 extern struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p);
+extern int mem_cgroup_watermark_ok(struct mem_cgroup *mem, int charge_flags);
 

bool. I know zone_watermark_ok is int, but it should be bool too.

Quoted 24 lines
include/linux/memcontrol.h +0 -0
 static inline
 int mm_match_cgroup(const struct mm_struct *mm, const struct mem_cgroup *cgroup)
include/linux/res_counter.h +10 -0
index fcb9884..eed12c5 100644
@@ -39,6 +39,16 @@ struct res_counter {
 	 */
 	unsigned long long soft_limit;
 	/*
+	 * the limit that reclaim triggers. TODO: res_counter in mem
+	 * or wmark_limit.
+	 */
+	unsigned long long low_wmark_limit;
+	/*
+	 * the limit that reclaim stops. TODO: res_counter in mem or
+	 * wmark_limit.
+	 */
+	unsigned long long high_wmark_limit;
+	/*
 	 * the number of unsuccessful attempts to consume the resource
 	 */
 	unsigned long long failcnt;

include/linux/res_counter.h struct res_counter { · line 68

Quoted 4 lines
include/linux/res_counter.h +1 -0
@@ -55,6 +65,10 @@ struct res_counter {
 
 #define RESOURCE_MAX (unsigned long long)LLONG_MAX
 
+#define CHARGE_WMARK_MIN	0x01

Comment that CHARGE_WMARK_MIN is the maximum limit of the container.

Quoted 7 lines
include/linux/res_counter.h +3 -0
+#define CHARGE_WMARK_LOW	0x02
+#define CHARGE_WMARK_HIGH	0x04
+
 /**
  * Helpers to interact with userspace
  * res_counter_read_u64() - returns the value of the specified member.

include/linux/res_counter.h enum { · line 112

Quoted 7 lines
include/linux/res_counter.h +2 -0
@@ -92,6 +106,8 @@ enum {
 	RES_LIMIT,
 	RES_FAILCNT,
 	RES_SOFT_LIMIT,
+	RES_LOW_WMARK_LIMIT,
+	RES_HIGH_WMARK_LIMIT
 };
 

I'm a little concerns that memcg watermarks feel like the opposite of the core VM watermarks. In the core VM, the number of "free pages" are compared against a watermark and we are wary of "going below" a watermark. In memory containers, the number of "used pages" is checked and should not "be above" a watermark.

It means one has to think differently about the core VM and containers. The more examples of this that exist, the harder maintainership will be in the future.

Quoted 15 lines
include/linux/res_counter.h +3 -2
 /*
@@ -112,9 +128,10 @@ void res_counter_init(struct res_counter *counter, struct res_counter *parent);
  */
 
 int __must_check res_counter_charge_locked(struct res_counter *counter,
-		unsigned long val);
+		unsigned long val, int charge_flags);
 int __must_check res_counter_charge(struct res_counter *counter,
-		unsigned long val, struct res_counter **limit_fail_at);
+		unsigned long val, int charge_flags,
+		struct res_counter **limit_fail_at);
 
 /*
  * uncharge - tell that some portion of the resource is released

include/linux/res_counter.h static inline bool res_counter_soft_limit_check_locked(struct res_counter *cnt) · line 167

Quoted 6 lines
include/linux/res_counter.h +3 -0
@@ -145,6 +162,24 @@ static inline bool res_counter_soft_limit_check_locked(struct res_counter *cnt)
 	return false;
 }
 
+static inline bool
+res_counter_high_wmark_limit_check_locked(struct res_counter *cnt)
+{

You only call this from one place and it takes the lock. Just collapse the functions together. Instead of postfixing this with _locked, you could have prefixed it with __ . There is a loose convension that similar named functions with __ imply that the caller is expected to acquire the necessary locks.

Second, the naming of this function does not give the reader a clue as to what the function is for. It needs a comment explaining what "true" means. However, collapsing res_counter_check_under_low_wmark_limit and this function together would give a better clue.

It'd probably be easier overall if you used the same pattern as zone_watermark_ok and pass in the watermark as a parameter.

Quoted 61 lines
include/linux/res_counter.h +41 -0
+	if (cnt->usage < cnt->high_wmark_limit)
+		return true;
+
+	return false;
+}
+
+static inline bool
+res_counter_low_wmark_limit_check_locked(struct res_counter *cnt)
+{
+	if (cnt->usage < cnt->low_wmark_limit)
+		return true;
+
+	return false;
+}
+
 /**
  * Get the difference between the usage and the soft limit
  * @cnt: The counter
@@ -193,6 +228,30 @@ static inline bool res_counter_check_under_soft_limit(struct res_counter *cnt)
 	return ret;
 }
 
+static inline bool
+res_counter_check_under_low_wmark_limit(struct res_counter *cnt)
+{
+	bool ret;
+	unsigned long flags;
+
+	spin_lock_irqsave(&cnt->lock, flags);
+	ret = res_counter_low_wmark_limit_check_locked(cnt);
+	spin_unlock_irqrestore(&cnt->lock, flags);
+	return ret;
+}
+
+static inline bool
+res_counter_check_under_high_wmark_limit(struct res_counter *cnt)
+{
+	bool ret;
+	unsigned long flags;
+
+	spin_lock_irqsave(&cnt->lock, flags);
+	ret = res_counter_high_wmark_limit_check_locked(cnt);
+	spin_unlock_irqrestore(&cnt->lock, flags);
+	return ret;
+}
+
 static inline void res_counter_reset_max(struct res_counter *cnt)
 {
 	unsigned long flags;
@@ -220,6 +279,8 @@ static inline int res_counter_set_limit(struct res_counter *cnt,
 	spin_lock_irqsave(&cnt->lock, flags);
 	if (cnt->usage <= limit) {
 		cnt->limit = limit;
+		cnt->low_wmark_limit = limit;
+		cnt->high_wmark_limit = limit;
 		ret = 0;
 	}
 	spin_unlock_irqrestore(&cnt->lock, flags);
@@ -238,4 +299,27 @@ res_counter_set_soft_limit(struct res_counter *cnt,
 	return 0;
 }

include/linux/res_counter.h res_counter_set_soft_limit(struct res_counter *cnt, · line 310

Quoted 10 lines
include/linux/res_counter.h +9 -0
 
+static inline int
+res_counter_set_high_wmark_limit(struct res_counter *cnt,
+				unsigned long long wmark_limit)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&cnt->lock, flags);
+	cnt->high_wmark_limit = wmark_limit;
+	spin_unlock_irqrestore(&cnt->lock, flags);

Is a full IRQ-safe lock here *really* necessary? Do interrupts call this function? Even then, is a spinlock necessary at all? What are the consequences if a parallel reader sees a temporarily stale value?

include/linux/res_counter.h res_counter_set_soft_limit(struct res_counter *cnt, · line 312

Quoted 2 lines
include/linux/res_counter.h +2 -0
+	return 0;
+}

The return value is never anything but 0 so why return anything?

Quoted 32 lines
include/linux/res_counter.h +12 -0
+
+static inline int
+res_counter_set_low_wmark_limit(struct res_counter *cnt,
+				unsigned long long wmark_limit)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&cnt->lock, flags);
+	cnt->low_wmark_limit = wmark_limit;
+	spin_unlock_irqrestore(&cnt->lock, flags);
+	return 0;
+}
 #endif
kernel/res_counter.c +5 -2
index c7eaa37..a524349 100644
@@ -19,12 +19,26 @@ void res_counter_init(struct res_counter *counter, struct res_counter *parent)
 	spin_lock_init(&counter->lock);
 	counter->limit = RESOURCE_MAX;
 	counter->soft_limit = RESOURCE_MAX;
+	counter->low_wmark_limit = RESOURCE_MAX;
+	counter->high_wmark_limit = RESOURCE_MAX;
 	counter->parent = parent;
 }
 
-int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
+int res_counter_charge_locked(struct res_counter *counter, unsigned long val,
+				int charge_flags)
 {
-	if (counter->usage + val > counter->limit) {
+	unsigned long long limit = 0;

kernel/res_counter.c void res_counter_init(struct res_counter *counter, struct res_counter *parent) · line 40

Quoted 10 lines
kernel/res_counter.c +10 -0
+
+	if (charge_flags & CHARGE_WMARK_LOW)
+		limit = counter->low_wmark_limit;
+
+	if (charge_flags & CHARGE_WMARK_HIGH)
+		limit = counter->high_wmark_limit;
+
+	if (charge_flags & CHARGE_WMARK_MIN)
+		limit = counter->limit;
+

Similar to zones, you can make the watermarks an array and use the WMARK flags in charge_flags to index it. It'll reduce the number of branches.

Quoted 57 lines
kernel/res_counter.c +7 -2
+	if (counter->usage + val > limit) {
 		counter->failcnt++;
 		return -ENOMEM;
 	}
@@ -36,7 +50,7 @@ int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
 }
 
 int res_counter_charge(struct res_counter *counter, unsigned long val,
-			struct res_counter **limit_fail_at)
+			int charge_flags, struct res_counter **limit_fail_at)
 {
 	int ret;
 	unsigned long flags;
@@ -46,7 +60,7 @@ int res_counter_charge(struct res_counter *counter, unsigned long val,
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
-		ret = res_counter_charge_locked(c, val);
+		ret = res_counter_charge_locked(c, val, charge_flags);
 		spin_unlock(&c->lock);
 		if (ret < 0) {
 			*limit_fail_at = c;
@@ -103,6 +117,10 @@ res_counter_member(struct res_counter *counter, int member)
 		return &counter->failcnt;
 	case RES_SOFT_LIMIT:
 		return &counter->soft_limit;
+	case RES_LOW_WMARK_LIMIT:
+		return &counter->low_wmark_limit;
+	case RES_HIGH_WMARK_LIMIT:
+		return &counter->high_wmark_limit;
 	};
 
 	BUG();
mm/memcontrol.c +2 -0
index dca3590..a0c6ed9 100644
@@ -265,6 +265,7 @@ struct mem_cgroup {
 	spinlock_t pcp_counter_lock;
 
 	wait_queue_head_t *kswapd_wait;
+	unsigned long min_free_kbytes;
 };
 
 /* Stuffs for move charges at task migration. */
@@ -370,6 +371,7 @@ static void mem_cgroup_get(struct mem_cgroup *mem);
 static void mem_cgroup_put(struct mem_cgroup *mem);
 static struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *mem);
 static void drain_all_stock_async(void);
+static unsigned long get_min_free_kbytes(struct mem_cgroup *mem);
 
 static struct mem_cgroup_per_zone *
 mem_cgroup_zoneinfo(struct mem_cgroup *mem, int nid, int zid)
@@ -796,6 +798,32 @@ static inline bool mem_cgroup_is_root(struct mem_cgroup *mem)
 	return (mem == root_mem_cgroup);
 }
 

mm/memcontrol.c static inline bool mem_cgroup_is_root(struct mem_cgroup *mem) · line 810

Quoted 10 lines
mm/memcontrol.c +10 -0
+void setup_per_memcg_wmarks(struct mem_cgroup *mem)
+{
+	u64 limit;
+	unsigned long min_free_kbytes;
+
+	min_free_kbytes = get_min_free_kbytes(mem);
+	limit = mem_cgroup_get_limit(mem);
+	if (min_free_kbytes == 0) {
+		res_counter_set_low_wmark_limit(&mem->res, limit);
+		res_counter_set_high_wmark_limit(&mem->res, limit);

This needs a comment stating that a min_free_kbytes of 0 means that kswapd is never woken up.

mm/memcontrol.c static inline bool mem_cgroup_is_root(struct mem_cgroup *mem) · line 813

Quoted 3 lines
mm/memcontrol.c +3 -0
+	} else {
+		unsigned long page_min = min_free_kbytes >> (PAGE_SHIFT - 10);
+		unsigned long lowmem_pages = 2048;

What if the container is less than 8M?

mm/memcontrol.c static inline bool mem_cgroup_is_root(struct mem_cgroup *mem) · line 824

Quoted 11 lines
mm/memcontrol.c +11 -0
+		unsigned long low_wmark, high_wmark;
+		u64 tmp;
+
+		tmp = (u64)page_min * limit;
+		do_div(tmp, lowmem_pages);
+
+		low_wmark = tmp + (tmp >> 1);
+		high_wmark = tmp + (tmp >> 2);
+		res_counter_set_low_wmark_limit(&mem->res, low_wmark);
+		res_counter_set_high_wmark_limit(&mem->res, high_wmark);
+	}

If mem_free_kbytes happens to be larger than the container, it may trigger OOM. Mind you, the core VM suffers the same problem but in the case of tuning containers it might be a lot easier for sysadmins to fall into the trap.

Quoted 12 lines
mm/memcontrol.c +5 -0
+}
+
 /*
  * Following LRU functions are allowed to be used without PCG_LOCK.
  * Operations are called by routine of global LRU independently from memcg.
@@ -1148,6 +1176,22 @@ static unsigned int get_swappiness(struct mem_cgroup *memcg)
 	return swappiness;
 }
 
+static unsigned long get_min_free_kbytes(struct mem_cgroup *memcg)
+{
+	struct cgroup *cgrp = memcg->css.cgroup;

mm/memcontrol.c static unsigned int get_swappiness(struct mem_cgroup *memcg) · line 1191

Quoted 10 lines
mm/memcontrol.c +10 -0
+	unsigned long min_free_kbytes;
+
+	/* root ? */
+	if (cgrp == NULL || cgrp->parent == NULL)
+		return 0;
+
+	spin_lock(&memcg->reclaim_param_lock);
+	min_free_kbytes = memcg->min_free_kbytes;
+	spin_unlock(&memcg->reclaim_param_lock);
+

Is the lock really necessary? Again, reading a stale value seems harmless.

Quoted 102 lines
mm/memcontrol.c +61 -2
+	return min_free_kbytes;
+}
+
 static void mem_cgroup_start_move(struct mem_cgroup *mem)
 {
 	int cpu;
@@ -1844,12 +1888,13 @@ static int __mem_cgroup_do_charge(struct mem_cgroup *mem, gfp_t gfp_mask,
 	unsigned long flags = 0;
 	int ret;
 
-	ret = res_counter_charge(&mem->res, csize, &fail_res);
+	ret = res_counter_charge(&mem->res, csize, CHARGE_WMARK_MIN, &fail_res);
 
 	if (likely(!ret)) {
 		if (!do_swap_account)
 			return CHARGE_OK;
-		ret = res_counter_charge(&mem->memsw, csize, &fail_res);
+		ret = res_counter_charge(&mem->memsw, csize, CHARGE_WMARK_MIN,
+					&fail_res);
 		if (likely(!ret))
 			return CHARGE_OK;
 
@@ -3733,6 +3778,37 @@ static int mem_cgroup_swappiness_write(struct cgroup *cgrp, struct cftype *cft,
 	return 0;
 }
 
+static u64 mem_cgroup_min_free_read(struct cgroup *cgrp, struct cftype *cft)
+{
+	struct mem_cgroup *memcg = mem_cgroup_from_cont(cgrp);
+
+	return get_min_free_kbytes(memcg);
+}
+
+static int mem_cgroup_min_free_write(struct cgroup *cgrp, struct cftype *cfg,
+				     u64 val)
+{
+	struct mem_cgroup *memcg = mem_cgroup_from_cont(cgrp);
+	struct mem_cgroup *parent;
+
+	if (cgrp->parent == NULL)
+		return -EINVAL;
+
+	parent = mem_cgroup_from_cont(cgrp->parent);
+
+	cgroup_lock();
+
+	spin_lock(&memcg->reclaim_param_lock);
+	memcg->min_free_kbytes = val;
+	spin_unlock(&memcg->reclaim_param_lock);
+
+	cgroup_unlock();
+
+	setup_per_memcg_wmarks(memcg);
+	return 0;
+
+}
+
 static void __mem_cgroup_threshold(struct mem_cgroup *memcg, bool swap)
 {
 	struct mem_cgroup_threshold_ary *t;
@@ -4024,6 +4100,21 @@ static void mem_cgroup_oom_unregister_event(struct cgroup *cgrp,
 	mutex_unlock(&memcg_oom_mutex);
 }
 
+static int mem_cgroup_wmark_read(struct cgroup *cgrp,
+	struct cftype *cft,  struct cgroup_map_cb *cb)
+{
+	struct mem_cgroup *mem = mem_cgroup_from_cont(cgrp);
+	unsigned long low_wmark, high_wmark;
+
+	low_wmark = res_counter_read_u64(&mem->res, RES_LOW_WMARK_LIMIT);
+	high_wmark = res_counter_read_u64(&mem->res, RES_HIGH_WMARK_LIMIT);
+
+	cb->fill(cb, "memcg_low_wmark", low_wmark);
+	cb->fill(cb, "memcg_high_wmark", high_wmark);
+
+	return 0;
+}
+
 static int mem_cgroup_oom_control_read(struct cgroup *cgrp,
 	struct cftype *cft,  struct cgroup_map_cb *cb)
 {
@@ -4127,6 +4218,15 @@ static struct cftype mem_cgroup_files[] = {
 		.unregister_event = mem_cgroup_oom_unregister_event,
 		.private = MEMFILE_PRIVATE(_OOM_TYPE, OOM_CONTROL),
 	},
+	{
+		.name = "min_free_kbytes",
+		.write_u64 = mem_cgroup_min_free_write,
+		.read_u64 = mem_cgroup_min_free_read,
+	},
+	{
+		.name = "reclaim_wmarks",
+		.read_map = mem_cgroup_wmark_read,
+	},
 };
 
 #ifdef CONFIG_CGROUP_MEM_RES_CTLR_SWAP
@@ -4308,6 +4408,19 @@ static void __init enable_swap_cgroup(void)
 }
 #endif
 

mm/memcontrol.c static void __init enable_swap_cgroup(void) · line 4420

Quoted 10 lines
mm/memcontrol.c +10 -0
+int mem_cgroup_watermark_ok(struct mem_cgroup *mem,
+				int charge_flags)
+{
+	long ret = 0;
+
+	if (charge_flags & CHARGE_WMARK_LOW)
+		ret = res_counter_check_under_low_wmark_limit(&mem->res);
+	if (charge_flags & CHARGE_WMARK_HIGH)
+		ret = res_counter_check_under_high_wmark_limit(&mem->res);
+

The naming here is odd as well. We are using flags called CHARGE in situations where nothing is being charged at all. If the enum zone_watermark was reused and the watermarks were checked in a similar way to the core VM (preferably identically), it will be easier to understand both core VM and container behaviour.

Quoted 44 lines
mm/memcontrol.c +7 -2
+	return ret;
+}
+
 static int mem_cgroup_soft_limit_tree_init(void)
 {
 	struct mem_cgroup_tree_per_node *rtpn;
@@ -4450,10 +4563,12 @@ static int mem_cgroup_do_precharge(unsigned long count)
 		 * are still under the same cgroup_mutex. So we can postpone
 		 * css_get().
 		 */
-		if (res_counter_charge(&mem->res, PAGE_SIZE * count, &dummy))
+		if (res_counter_charge(&mem->res, PAGE_SIZE * count,
+					CHARGE_WMARK_MIN, &dummy))
 			goto one_by_one;
 		if (do_swap_account && res_counter_charge(&mem->memsw,
-						PAGE_SIZE * count, &dummy)) {
+						PAGE_SIZE * count,
+						CHARGE_WMARK_MIN, &dummy)) {
 			res_counter_uncharge(&mem->res, PAGE_SIZE * count);
 			goto one_by_one;
 		}
mm/vmscan.c +3 -0
index e08005e..6d5702b 100644
@@ -46,6 +46,8 @@
 
 #include <linux/swapops.h>
 
+#include <linux/res_counter.h>
+
 #include "internal.h"
 
 #define CREATE_TRACE_POINTS
@@ -2127,11 +2129,19 @@ static int sleeping_prematurely(struct kswapd *kswapd, int order,
 {
 	int i;
 	pg_data_t *pgdat = kswapd->kswapd_pgdat;
+	struct mem_cgroup *mem = kswapd->kswapd_mem;
 
 	/* If a direct reclaimer woke kswapd within HZ/10, it's premature */
 	if (remaining)
 		return 1;
 

mm/vmscan.c static int sleeping_prematurely(struct kswapd *kswapd, int order, · line 2147

Quoted 10 lines
mm/vmscan.c +7 -0
+	if (mem) {
+		if (!mem_cgroup_watermark_ok(kswapd->kswapd_mem,
+						CHARGE_WMARK_HIGH))
+			return 1;
+		return 0;
+	}
+
 	/* If after HZ/10, a zone is below the high mark, it's premature */
 	for (i = 0; i < pgdat->nr_zones; i++) {
 		struct zone *zone = pgdat->node_zones + i;

Very broadly speaking, I'd be happier if watermarks for containers behaved similarly to watermarks in the core VM.

Signature · 10 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#34 lore raw

PATCH 3/4 Per cgroup background reclaim.

In reply to Ying Han

The current implementation of memcg only supports direct reclaim and this patch adds the support for background reclaim. Per cgroup background reclaim is needed which spreads out the memory pressure over longer period of time and smoothes out the system performance.

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup. The kswapd is sleeping in the wait queue headed at kswapd_wait field of a kswapd descriptor.

The kswapd() function now is shared between global and per cgroup kswapd thread. It is passed in with the kswapd descriptor which contains the information of either node or cgroup. Then the new function balance_mem_cgroup_pgdat is invoked if it is per cgroup kswapd thread. The balance_mem_cgroup_pgdat performs a priority loop similar to global reclaim. In each iteration it invokes balance_pgdat_node for all nodes on the system, which is a new function performs background reclaim per node. After reclaiming each node, it checks mem_cgroup_watermark_ok() and breaks the priority loop if returns true. A per memcg zone will be marked as "unreclaimable" if the scanning rate is much greater than the reclaiming rate on the per cgroup LRU. The bit is cleared when there is a page charged to the cgroup being freed. Kswapd breaks the priority loop if all the zones are marked as "unreclaimable".

---
 include/linux/memcontrol.h |   30 +++++++
 mm/memcontrol.c            |  182 ++++++++++++++++++++++++++++++++++++++-
 mm/page_alloc.c            |    2 +
 mm/vmscan.c                |  205 +++++++++++++++++++++++++++++++++++++++++++-
 4 files changed, 416 insertions(+), 3 deletions(-)
include/linux/memcontrol.h +30 -0
index 90fe7fe..dbed45d 100644
@@ -127,6 +127,12 @@ unsigned long mem_cgroup_soft_limit_reclaim(struct zone *zone, int order,
 						gfp_t gfp_mask);
 u64 mem_cgroup_get_limit(struct mem_cgroup *mem);
 
+void mem_cgroup_clear_unreclaimable(struct page *page, struct zone *zone);
+bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid, int zid);
+bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem, struct zone *zone);
+void mem_cgroup_mz_set_unreclaimable(struct mem_cgroup *mem, struct zone *zone);
+void mem_cgroup_mz_pages_scanned(struct mem_cgroup *mem, struct zone* zone,
+					unsigned long nr_scanned);
 #else /* CONFIG_CGROUP_MEM_RES_CTLR */
 struct mem_cgroup;
 
@@ -299,6 +305,25 @@ static inline void mem_cgroup_update_file_mapped(struct page *page,
 {
 }
 
+static inline void mem_cgroup_mz_pages_scanned(struct mem_cgroup *mem,
+						struct zone *zone,
+						unsigned long nr_scanned)
+{
+}
+
+static inline void mem_cgroup_clear_unreclaimable(struct page *page,
+							struct zone *zone)
+{
+}
+static inline void mem_cgroup_mz_set_unreclaimable(struct mem_cgroup *mem,
+		struct zone *zone)
+{
+}
+static inline bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem,
+						struct zone *zone)
+{
+}
+
 static inline
 unsigned long mem_cgroup_soft_limit_reclaim(struct zone *zone, int order,
 					    gfp_t gfp_mask)
@@ -312,6 +337,11 @@ u64 mem_cgroup_get_limit(struct mem_cgroup *mem)
 	return 0;
 }
 
+static inline bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid,
+								int zid)
+{
+	return false;
+}
 #endif /* CONFIG_CGROUP_MEM_CONT */
 
 #endif /* _LINUX_MEMCONTROL_H */
mm/memcontrol.c +181 -1

261 lines not on this page · raw message

mm/page_alloc.c +2 -0
index a15bc1c..dc61f2a 100644
@@ -615,6 +615,7 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 
 		do {
 			page = list_entry(list->prev, struct page, lru);
+			mem_cgroup_clear_unreclaimable(page, zone);
 			/* must delete as __free_one_page list manipulates */
 			list_del(&page->lru);
 			/* MIGRATE_MOVABLE list may include MIGRATE_RESERVEs */
@@ -632,6 +633,7 @@ static void free_one_page(struct zone *zone, struct page *page, int order,
 	spin_lock(&zone->lock);
 	zone->all_unreclaimable = 0;
 	zone->pages_scanned = 0;
+	mem_cgroup_clear_unreclaimable(page, zone);
 
 	__free_one_page(page, zone, order, migratetype);
 	__mod_zone_page_state(zone, NR_FREE_PAGES, 1 << order);
mm/vmscan.c +203 -2

223 lines not on this page · raw message

Signature · 8 lines
1.7.3.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#35 lore raw

Re: [PATCH 3/4] Per cgroup background reclaim.

Quoted 197 lines · On Mon, 29 Nov 2010 22:49:44 -0800 Ying Han <[email protected]> wrote:

The current implementation of memcg only supports direct reclaim and this patch adds the support for background reclaim. Per cgroup background reclaim is needed which spreads out the memory pressure over longer period of time and smoothes out the system performance.

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup. The kswapd is sleeping in the wait queue headed at kswapd_wait field of a kswapd descriptor.

The kswapd() function now is shared between global and per cgroup kswapd thread. It is passed in with the kswapd descriptor which contains the information of either node or cgroup. Then the new function balance_mem_cgroup_pgdat is invoked if it is per cgroup kswapd thread. The balance_mem_cgroup_pgdat performs a priority loop similar to global reclaim. In each iteration it invokes balance_pgdat_node for all nodes on the system, which is a new function performs background reclaim per node. After reclaiming each node, it checks mem_cgroup_watermark_ok() and breaks the priority loop if returns true. A per memcg zone will be marked as "unreclaimable" if the scanning rate is much greater than the reclaiming rate on the per cgroup LRU. The bit is cleared when there is a page charged to the cgroup being freed. Kswapd breaks the priority loop if all the zones are marked as "unreclaimable".

---
 include/linux/memcontrol.h |   30 +++++++
 mm/memcontrol.c            |  182 ++++++++++++++++++++++++++++++++++++++-
 mm/page_alloc.c            |    2 +
 mm/vmscan.c                |  205 +++++++++++++++++++++++++++++++++++++++++++-
 4 files changed, 416 insertions(+), 3 deletions(-)
include/linux/memcontrol.h +30 -0
index 90fe7fe..dbed45d 100644
@@ -127,6 +127,12 @@ unsigned long mem_cgroup_soft_limit_reclaim(struct zone *zone, int order,
 						gfp_t gfp_mask);
 u64 mem_cgroup_get_limit(struct mem_cgroup *mem);
 
+void mem_cgroup_clear_unreclaimable(struct page *page, struct zone *zone);
+bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid, int zid);
+bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem, struct zone *zone);
+void mem_cgroup_mz_set_unreclaimable(struct mem_cgroup *mem, struct zone *zone);
+void mem_cgroup_mz_pages_scanned(struct mem_cgroup *mem, struct zone* zone,
+					unsigned long nr_scanned);
 #else /* CONFIG_CGROUP_MEM_RES_CTLR */
 struct mem_cgroup;
 
@@ -299,6 +305,25 @@ static inline void mem_cgroup_update_file_mapped(struct page *page,
 {
 }
 
+static inline void mem_cgroup_mz_pages_scanned(struct mem_cgroup *mem,
+						struct zone *zone,
+						unsigned long nr_scanned)
+{
+}
+
+static inline void mem_cgroup_clear_unreclaimable(struct page *page,
+							struct zone *zone)
+{
+}
+static inline void mem_cgroup_mz_set_unreclaimable(struct mem_cgroup *mem,
+		struct zone *zone)
+{
+}
+static inline bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem,
+						struct zone *zone)
+{
+}
+
 static inline
 unsigned long mem_cgroup_soft_limit_reclaim(struct zone *zone, int order,
 					    gfp_t gfp_mask)
@@ -312,6 +337,11 @@ u64 mem_cgroup_get_limit(struct mem_cgroup *mem)
 	return 0;
 }
 
+static inline bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid,
+								int zid)
+{
+	return false;
+}
 #endif /* CONFIG_CGROUP_MEM_CONT */
 
 #endif /* _LINUX_MEMCONTROL_H */
mm/memcontrol.c +82 -0
index a0c6ed9..1d39b65 100644
@@ -48,6 +48,8 @@
 #include <linux/page_cgroup.h>
 #include <linux/cpu.h>
 #include <linux/oom.h>
+#include <linux/kthread.h>
+
 #include "internal.h"
 
 #include <asm/uaccess.h>
@@ -118,7 +120,10 @@ struct mem_cgroup_per_zone {
 	bool			on_tree;
 	struct mem_cgroup	*mem;		/* Back pointer, we cannot */
 						/* use container_of	   */
+	unsigned long		pages_scanned;	/* since last reclaim */
+	int			all_unreclaimable;	/* All pages pinned */
 };
+
 /* Macro for accessing counter */
 #define MEM_CGROUP_ZSTAT(mz, idx)	((mz)->count[(idx)])
 
@@ -372,6 +377,7 @@ static void mem_cgroup_put(struct mem_cgroup *mem);
 static struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *mem);
 static void drain_all_stock_async(void);
 static unsigned long get_min_free_kbytes(struct mem_cgroup *mem);
+static inline void wake_memcg_kswapd(struct mem_cgroup *mem);
 
 static struct mem_cgroup_per_zone *
 mem_cgroup_zoneinfo(struct mem_cgroup *mem, int nid, int zid)
@@ -1086,6 +1092,106 @@ mem_cgroup_get_reclaim_stat_from_page(struct page *page)
 	return &mz->reclaim_stat;
 }
 
+unsigned long mem_cgroup_zone_reclaimable_pages(
+					struct mem_cgroup_per_zone *mz)
+{
+	int nr;
+	nr = MEM_CGROUP_ZSTAT(mz, LRU_ACTIVE_FILE) +
+		MEM_CGROUP_ZSTAT(mz, LRU_INACTIVE_FILE);
+
+	if (nr_swap_pages > 0)
+		nr += MEM_CGROUP_ZSTAT(mz, LRU_ACTIVE_ANON) +
+			MEM_CGROUP_ZSTAT(mz, LRU_INACTIVE_ANON);
+
+	return nr;
+}
+
+void mem_cgroup_mz_pages_scanned(struct mem_cgroup *mem, struct zone* zone,
+						unsigned long nr_scanned)
+{
+	struct mem_cgroup_per_zone *mz = NULL;
+	int nid = zone_to_nid(zone);
+	int zid = zone_idx(zone);
+
+	if (!mem)
+		return;
+
+	mz = mem_cgroup_zoneinfo(mem, nid, zid);
+	if (mz)
+		mz->pages_scanned += nr_scanned;
+}
+
+bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid, int zid)
+{
+	struct mem_cgroup_per_zone *mz = NULL;
+
+	if (!mem)
+		return 0;
+
+	mz = mem_cgroup_zoneinfo(mem, nid, zid);
+	if (mz)
+		return mz->pages_scanned <
+				mem_cgroup_zone_reclaimable_pages(mz) * 6;
+	return 0;
+}
+
+bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem, struct zone *zone)
+{
+	struct mem_cgroup_per_zone *mz = NULL;
+	int nid = zone_to_nid(zone);
+	int zid = zone_idx(zone);
+
+	if (!mem)
+		return 0;
+
+	mz = mem_cgroup_zoneinfo(mem, nid, zid);
+	if (mz)
+		return mz->all_unreclaimable;
+
+	return 0;
+}
+
+void mem_cgroup_mz_set_unreclaimable(struct mem_cgroup *mem, struct zone *zone)
+{
+	struct mem_cgroup_per_zone *mz = NULL;
+	int nid = zone_to_nid(zone);
+	int zid = zone_idx(zone);
+
+	if (!mem)
+		return;
+
+	mz = mem_cgroup_zoneinfo(mem, nid, zid);
+	if (mz)
+		mz->all_unreclaimable = 1;
+}
+
+void mem_cgroup_clear_unreclaimable(struct page *page, struct zone *zone)
+{
+	struct mem_cgroup_per_zone *mz = NULL;

mm/memcontrol.c mem_cgroup_get_reclaim_stat_from_page(struct page *page) · line 1180

Quoted 10 lines
mm/memcontrol.c +10 -0
+	struct mem_cgroup *mem = NULL;
+	int nid = zone_to_nid(zone);
+	int zid = zone_idx(zone);
+	struct page_cgroup *pc = lookup_page_cgroup(page);
+
+	if (unlikely(!pc))
+		return;
+
+	rcu_read_lock();
+	mem = pc->mem_cgroup;

This is incorrect. you have to do css_tryget(&mem->css) before rcu_read_unlock.

Quoted 25 lines
mm/memcontrol.c +18 -0
+	rcu_read_unlock();
+
+	if (!mem)
+		return;
+
+	mz = mem_cgroup_zoneinfo(mem, nid, zid);
+	if (mz) {
+		mz->pages_scanned = 0;
+		mz->all_unreclaimable = 0;
+	}
+
+	return;
+}
+
 unsigned long mem_cgroup_isolate_pages(unsigned long nr_to_scan,
 					struct list_head *dst,
 					unsigned long *scanned, int order,
@@ -1887,6 +1993,20 @@ static int __mem_cgroup_do_charge(struct mem_cgroup *mem, gfp_t gfp_mask,
 	struct res_counter *fail_res;
 	unsigned long flags = 0;
 	int ret;
+	unsigned long min_free_kbytes = 0;
+
+	min_free_kbytes = get_min_free_kbytes(mem);
+	if (min_free_kbytes) {

mm/memcontrol.c static int __mem_cgroup_do_charge(struct mem_cgroup *mem, gfp_t gfp_mask, · line 2009

Quoted 10 lines
mm/memcontrol.c +10 -0
+		ret = res_counter_charge(&mem->res, csize, CHARGE_WMARK_LOW,
+					&fail_res);
+		if (likely(!ret)) {
+			return CHARGE_OK;
+		} else {
+			mem_over_limit = mem_cgroup_from_res_counter(fail_res,
+									res);
+			wake_memcg_kswapd(mem_over_limit);
+		}
+	}

I think this check can be moved out to periodic-check as threshould notifiers.

Quoted 12 lines
mm/memcontrol.c +1 -0
 	ret = res_counter_charge(&mem->res, csize, CHARGE_WMARK_MIN, &fail_res);
 
@@ -3037,6 +3157,7 @@ static int mem_cgroup_resize_limit(struct mem_cgroup *memcg,
 			else
 				memcg->memsw_is_minimum = false;
 		}
+		setup_per_memcg_wmarks(memcg);
 		mutex_unlock(&set_limit_mutex);
 
 		if (!ret)

mm/memcontrol.c static int mem_cgroup_resize_limit(struct mem_cgroup *memcg, · line 3173

Quoted 8 lines
mm/memcontrol.c +1 -1
@@ -3046,7 +3167,7 @@ static int mem_cgroup_resize_limit(struct mem_cgroup *memcg,
 						MEM_CGROUP_RECLAIM_SHRINK);
 		curusage = res_counter_read_u64(&memcg->res, RES_USAGE);
 		/* Usage is reduced ? */
-  		if (curusage >= oldusage)
+		if (curusage >= oldusage)
 			retry_count--;
 		else
 			oldusage = curusage;

What's changed here ?

Quoted 47 lines
mm/memcontrol.c +22 -0
@@ -3096,6 +3217,7 @@ static int mem_cgroup_resize_memsw_limit(struct mem_cgroup *memcg,
 			else
 				memcg->memsw_is_minimum = false;
 		}
+		setup_per_memcg_wmarks(memcg);
 		mutex_unlock(&set_limit_mutex);
 
 		if (!ret)
@@ -4352,6 +4474,8 @@ static struct mem_cgroup *mem_cgroup_alloc(void)
 static void __mem_cgroup_free(struct mem_cgroup *mem)
 {
 	int node;
+	struct kswapd *kswapd_p;
+	wait_queue_head_t *wait;
 
 	mem_cgroup_remove_from_trees(mem);
 	free_css_id(&mem_cgroup_subsys, &mem->css);
@@ -4360,6 +4484,15 @@ static void __mem_cgroup_free(struct mem_cgroup *mem)
 		free_mem_cgroup_per_zone_info(mem, node);
 
 	free_percpu(mem->stat);
+
+	wait = mem->kswapd_wait;
+	kswapd_p = container_of(wait, struct kswapd, kswapd_wait);
+	if (kswapd_p) {
+		if (kswapd_p->kswapd_task)
+			kthread_stop(kswapd_p->kswapd_task);
+		kfree(kswapd_p);
+	}
+
 	if (sizeof(struct mem_cgroup) < PAGE_SIZE)
 		kfree(mem);
 	else
@@ -4421,6 +4554,39 @@ int mem_cgroup_watermark_ok(struct mem_cgroup *mem,
 	return ret;
 }
 
+static inline
+void wake_memcg_kswapd(struct mem_cgroup *mem)
+{
+	wait_queue_head_t *wait;
+	struct kswapd *kswapd_p;
+	struct task_struct *thr;
+	static char memcg_name[PATH_MAX];
+
+	if (!mem)
+		return;

mm/memcontrol.c int mem_cgroup_watermark_ok(struct mem_cgroup *mem, · line 4576

Quoted 10 lines
mm/memcontrol.c +10 -0
+
+	wait = mem->kswapd_wait;
+	kswapd_p = container_of(wait, struct kswapd, kswapd_wait);
+	if (!kswapd_p->kswapd_task) {
+		if (mem->css.cgroup)
+			cgroup_path(mem->css.cgroup, memcg_name, PATH_MAX);
+		else
+			sprintf(memcg_name, "no_name");
+
+		thr = kthread_run(kswapd, kswapd_p, "kswapd%s", memcg_name);

I don't think reusing the name of "kswapd" isn't good. and this name cannot be long as PATH_MAX...IIUC, this name is for comm[] field which is 16bytes long.

So, how about naming this as

"memcg%d", mem->css.id ?

Exporing css.id will be okay if necessary.

mm/memcontrol.c int mem_cgroup_watermark_ok(struct mem_cgroup *mem, · line 4582

Quoted 6 lines
mm/memcontrol.c +6 -0
+		if (IS_ERR(thr))
+			printk(KERN_INFO "Failed to start kswapd on memcg %d\n",
+				0);
+		else
+			kswapd_p->kswapd_task = thr;
+	}

Hmm, ok, then, kswapd-for-memcg is created when someone go over watermark. Why this new kswapd will not exit() until memcg destroy ?

I think there are several approaches.

1. create/destroy a thread at memcg create/destroy
2. create/destroy a thread at watermarks.
3. use thread pool for watermarks.
4. use workqueue for watermaks.

The good point of "1" is that we can control a-thread-for-kswapd by cpu controller but it will use some resource.
The good point of "2" is that we can avoid unnecessary resource usage.

3 and 4 is not very good, I think.

I'd like to vote for "1"...I want to avoid "stealing" other container's cpu by bad application in a container uses up memory.

Quoted 72 lines
mm/memcontrol.c +21 -0
+
+	if (!waitqueue_active(wait)) {
+		return;
+	}
+	wake_up_interruptible(wait);
+}
+
 static int mem_cgroup_soft_limit_tree_init(void)
 {
 	struct mem_cgroup_tree_per_node *rtpn;
@@ -4452,6 +4618,7 @@ mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cont)
 	struct mem_cgroup *mem, *parent;
 	long error = -ENOMEM;
 	int node;
+	struct kswapd *kswapd_p = NULL;
 
 	mem = mem_cgroup_alloc();
 	if (!mem)
@@ -4499,6 +4666,19 @@ mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cont)
 	spin_lock_init(&mem->reclaim_param_lock);
 	INIT_LIST_HEAD(&mem->oom_notify);
 
+
+	if (!mem_cgroup_is_root(mem)) {
+		kswapd_p = kmalloc(sizeof(struct kswapd), GFP_KERNEL);
+		if (!kswapd_p) {
+			printk(KERN_INFO "Failed to kmalloc kswapd_p %d\n", 0);
+			goto free_out;
+		}
+		memset(kswapd_p, 0, sizeof(struct kswapd));
+		init_waitqueue_head(&kswapd_p->kswapd_wait);
+		mem->kswapd_wait = &kswapd_p->kswapd_wait;
+		kswapd_p->kswapd_mem = mem;
+	}
+
 	if (parent)
 		mem->swappiness = get_swappiness(parent);
 	atomic_set(&mem->refcnt, 1);
mm/page_alloc.c +2 -0
index a15bc1c..dc61f2a 100644
@@ -615,6 +615,7 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 
 		do {
 			page = list_entry(list->prev, struct page, lru);
+			mem_cgroup_clear_unreclaimable(page, zone);
 			/* must delete as __free_one_page list manipulates */
 			list_del(&page->lru);
 			/* MIGRATE_MOVABLE list may include MIGRATE_RESERVEs */
@@ -632,6 +633,7 @@ static void free_one_page(struct zone *zone, struct page *page, int order,
 	spin_lock(&zone->lock);
 	zone->all_unreclaimable = 0;
 	zone->pages_scanned = 0;
+	mem_cgroup_clear_unreclaimable(page, zone);
 
 	__free_one_page(page, zone, order, migratetype);
 	__mod_zone_page_state(zone, NR_FREE_PAGES, 1 << order);
mm/vmscan.c +2 -0
index 6d5702b..f8430c4 100644
@@ -100,6 +100,8 @@ struct scan_control {
 	 * are scanned.
 	 */
 	nodemask_t	*nodemask;
+
+	int priority;
 };
 
 #define lru_to_page(_head) (list_entry((_head)->prev, struct page, lru))

mm/vmscan.c out: · line 2384

Quoted 3 lines
mm/vmscan.c +0 -0
@@ -2380,6 +2382,201 @@ out:
 	return sc.nr_reclaimed;
 }
 

Because you write all codes below, I don't think merging with kswapd is not necessary..

Quoted 107 lines
mm/vmscan.c +107 -0
+#ifdef CONFIG_CGROUP_MEM_RES_CTLR
+/*
+ * TODO: the same function is used for global LRU and memcg LRU. For global
+ * LRU, the kswapd is done until all this node's zones are at
+ * high_wmark_pages(zone) or zone->all_unreclaimable.
+ */
+static void balance_pgdat_node(pg_data_t *pgdat, int order,
+					struct scan_control *sc)
+{
+	int i, end_zone;
+	unsigned long total_scanned;
+	struct mem_cgroup *mem_cont = sc->mem_cgroup;
+	int priority = sc->priority;
+	int nid = pgdat->node_id;
+
+	/*
+	 * Scan in the highmem->dma direction for the highest
+	 * zone which needs scanning
+	 */
+	for (i = pgdat->nr_zones - 1; i >= 0; i--) {
+		struct zone *zone = pgdat->node_zones + i;
+
+		if (!populated_zone(zone))
+			continue;
+
+		if (mem_cgroup_mz_unreclaimable(mem_cont, zone) &&
+				priority != DEF_PRIORITY)
+			continue;
+		/*
+		 * Do some background aging of the anon list, to give
+		 * pages a chance to be referenced before reclaiming.
+		 */
+		if (inactive_anon_is_low(zone, sc))
+			shrink_active_list(SWAP_CLUSTER_MAX, zone,
+							sc, priority, 0);
+
+		end_zone = i;
+		goto scan;
+	}
+	return;
+
+scan:
+	total_scanned = 0;
+	/*
+	 * Now scan the zone in the dma->highmem direction, stopping
+	 * at the last zone which needs scanning.
+	 *
+	 * We do this because the page allocator works in the opposite
+	 * direction.  This prevents the page allocator from allocating
+	 * pages behind kswapd's direction of progress, which would
+	 * cause too much scanning of the lower zones.
+	 */
+	for (i = 0; i <= end_zone; i++) {
+		struct zone *zone = pgdat->node_zones + i;
+
+		if (!populated_zone(zone))
+			continue;
+
+		if (mem_cgroup_mz_unreclaimable(mem_cont, zone) &&
+			priority != DEF_PRIORITY)
+			continue;
+
+		sc->nr_scanned = 0;
+		shrink_zone(priority, zone, sc);
+		total_scanned += sc->nr_scanned;
+
+		if (mem_cgroup_mz_unreclaimable(mem_cont, zone))
+			continue;
+
+		if (!mem_cgroup_zone_reclaimable(mem_cont, nid, i))
+			mem_cgroup_mz_set_unreclaimable(mem_cont, zone);
+
+		/*
+		 * If we've done a decent amount of scanning and
+		 * the reclaim ratio is low, start doing writepage
+		 * even in laptop mode
+		 */
+		if (total_scanned > SWAP_CLUSTER_MAX * 2 &&
+		    total_scanned > sc->nr_reclaimed + sc->nr_reclaimed / 2) {
+			sc->may_writepage = 1;
+		}
+	}
+
+	sc->nr_scanned = total_scanned;
+	return;
+}
+
+static unsigned long balance_mem_cgroup_pgdat(struct mem_cgroup *mem_cont,
+					      int order)
+{
+	unsigned long total_scanned = 0;
+	int i;
+	int priority;
+	int wmark_ok, nid;
+	struct scan_control sc = {
+		.gfp_mask = GFP_KERNEL,
+		.may_unmap = 1,
+		.may_swap = 1,
+		/*
+		 * kswapd doesn't want to be bailed out while reclaim. because
+		 * we want to put equal scanning pressure on each zone.
+		 * TODO: this might not be true for the memcg background
+		 * reclaim.
+		 */
+		.nr_to_reclaim = ULONG_MAX,
+		.swappiness = vm_swappiness,
+		.order = order,

mm/vmscan.c out: · line 2501

Quoted 10 lines
mm/vmscan.c +10 -0
+		.mem_cgroup = mem_cont,
+	};
+	DECLARE_BITMAP(do_nodes, MAX_NUMNODES);
+
+	/*
+	 * bitmap to indicate which node to reclaim pages from. Initially we
+	 * assume all nodes need reclaim.
+	 */
+	bitmap_fill(do_nodes, MAX_NUMNODES);
+

Hmm..

Quoted 11 lines
mm/vmscan.c +11 -0
+loop_again:
+	sc.may_writepage = !laptop_mode;
+	sc.nr_reclaimed = 0;
+	total_scanned = 0;
+
+	for (priority = DEF_PRIORITY; priority >= 0; priority--) {
+		sc.priority = priority;
+
+		/* The swap token gets in the way of swapout... */
+		if (!priority)
+			disable_swap_token();

mm/vmscan.c out: · line 2522

Quoted 10 lines
mm/vmscan.c +10 -0
+
+
+		for_each_online_node(nid) {
+			pg_data_t *pgdat = NODE_DATA(nid);
+
+			wmark_ok = 1;
+
+			if (!test_bit(nid, do_nodes))
+				continue;
+

Then, always start reclaim from node "0"....it's not good.

If using bitmap, could you add fairness among nodes ?

as:
node = select_next_victim_node(mem);

This function will select the next scan node considering fairness between nodes.
(Because memcg doesn't take care of NODE placement and just takes care of
"amount", we don't know the best node to be reclaimed.)

Quoted 62 lines
mm/vmscan.c +57 -0
+			balance_pgdat_node(pgdat, order, &sc);
+			total_scanned += sc.nr_scanned;
+
+			for (i = pgdat->nr_zones - 1; i >= 0; i++) {
+				struct zone *zone = pgdat->node_zones + i;
+
+				if (!populated_zone(zone))
+					continue;
+
+				if (!mem_cgroup_mz_unreclaimable(mem_cont,
+								zone)) {
+					__set_bit(nid, do_nodes);
+					break;
+				}
+			}
+
+			if (i < 0)
+				__clear_bit(nid, do_nodes);
+
+			if (!mem_cgroup_watermark_ok(sc.mem_cgroup,
+							CHARGE_WMARK_HIGH))
+				wmark_ok = 0;
+
+			if (wmark_ok) {
+				goto out;
+			}
+		}
+
+		if (wmark_ok)
+			break;
+
+		if (total_scanned && priority < DEF_PRIORITY - 2)
+			congestion_wait(WRITE, HZ/10);
+
+		if (sc.nr_reclaimed >= SWAP_CLUSTER_MAX)
+			break;
+	}
+
+out:
+	if (!wmark_ok) {
+		cond_resched();
+
+		try_to_freeze();
+
+		goto loop_again;
+	}
+
+	return sc.nr_reclaimed;
+}
+#else
+static unsigned long balance_mem_cgroup_pgdat(struct mem_cgroup *mem_cont,
+							int order)
+{
+	return 0;
+}
+#endif
+
 /*
  * The background pageout daemon, started as a kernel thread
  * from the init process.
@@ -2497,8 +2694,12 @@ int kswapd(void *p)
 		 * after returning from the refrigerator

mm/vmscan.c int kswapd(void *p) · line 2702

Quoted 10 lines
mm/vmscan.c +6 -2
 		 */
 		if (!ret) {
-			trace_mm_vmscan_kswapd_wake(pgdat->node_id, order);
-			balance_pgdat(pgdat, order);
+			if (pgdat) {
+				trace_mm_vmscan_kswapd_wake(pgdat->node_id,
+								order);
+				balance_pgdat(pgdat, order);
+			} else
+				balance_mem_cgroup_pgdat(mem, order);

mem_cgroup's order is always 0.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#36 lore raw

Re: [PATCH 3/4] Per cgroup background reclaim.

Quoted 17 lines · On Tue, 30 Nov 2010 16:51:42 +0900 KAMEZAWA Hiroyuki…

mm/memcontrol.c int mem_cgroup_watermark_ok(struct mem_cgroup *mem, · line 4582

Quoted 6 lines
mm/memcontrol.c +6 -0
+		if (IS_ERR(thr))
+			printk(KERN_INFO "Failed to start kswapd on memcg %d\n",
+				0);
+		else
+			kswapd_p->kswapd_task = thr;
+	}

Hmm, ok, then, kswapd-for-memcg is created when someone go over watermark. Why this new kswapd will not exit() until memcg destroy ?

I think there are several approaches.

1. create/destroy a thread at memcg create/destroy
2. create/destroy a thread at watermarks.
3. use thread pool for watermarks.
4. use workqueue for watermaks.

Quoted 9 lines

The good point of "1" is that we can control a-thread-for-kswapd by cpu controller but it will use some resource.
The good point of "2" is that we can avoid unnecessary resource usage.

3 and 4 is not very good, I think.

I'd like to vote for "1"...I want to avoid "stealing" other container's cpu by bad application in a container uses up memory.

One more point, one-thread-per-hierarchy is enough. So, please check memory.use_hierarchy==1 or not at creating a thread.

Thanks,
-kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#37 lore raw

Re: [PATCH 3/4] Per cgroup background reclaim.

Quoted 24 lines · KAMEZAWA Hiroyuki
Quoted 21 lines · On Tue, 30 Nov 2010 16:51:42 +0900 KAMEZAWA Hiroyuki…
Quoted 6 lines

+           if (IS_ERR(thr))
+                   printk(KERN_INFO "Failed to start kswapd on memcg %d\n",
+                           0);
+           else
+                   kswapd_p->kswapd_task = thr;
+   }

Hmm, ok, then, kswapd-for-memcg is created when someone go over watermark. Why this new kswapd will not exit() until memcg destroy ?

I think there are several approaches.

  1. create/destroy a thread at memcg create/destroy
  2. create/destroy a thread at watermarks.
  3. use thread pool for watermarks.
  4. use workqueue for watermaks.

The good point of "1" is that we can control a-thread-for-kswapd by cpu controller but it will use some resource.
The good point of "2" is that we can avoid unnecessary resource usage.

Quoted 8 lines
Quoted 5 lines

3 and 4 is not very good, I think.

I'd like to vote for "1"...I want to avoid "stealing" other container's cpu by bad application in a container uses up memory.

One more point, one-thread-per-hierarchy is enough. So, please check memory.use_hierarchy==1 or not at creating a thread.

Thanks. Will take a look at it.

--Ying

Quoted 5 lines

Thanks,
-kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#38 lore raw

Re: [PATCH 3/4] Per cgroup background reclaim.

In reply to KAMEZAWA Hiroyuki
Quoted 204 lines · KAMEZAWA Hiroyuki
Quoted 201 lines · On Mon, 29 Nov 2010 22:49:44 -0800 Ying Han <[email protected]> wrote:

The current implementation of memcg only supports direct reclaim and this patch adds the support for background reclaim. Per cgroup background reclaim is needed which spreads out the memory pressure over longer period of time and smoothes out the system performance.

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup. The kswapd is sleeping in the wait queue headed at kswapd_wait field of a kswapd descriptor.

The kswapd() function now is shared between global and per cgroup kswapd thread. It is passed in with the kswapd descriptor which contains the information of either node or cgroup. Then the new function balance_mem_cgroup_pgdat is invoked if it is per cgroup kswapd thread. The balance_mem_cgroup_pgdat performs a priority loop similar to global reclaim. In each iteration it invokes balance_pgdat_node for all nodes on the system, which is a new function performs background reclaim per node. After reclaiming each node, it checks mem_cgroup_watermark_ok() and breaks the priority loop if returns true. A per memcg zone will be marked as "unreclaimable" if the scanning rate is much greater than the reclaiming rate on the per cgroup LRU. The bit is cleared when there is a page charged to the cgroup being freed. Kswapd breaks the priority loop if all the zones are marked as "unreclaimable".

---
 include/linux/memcontrol.h |   30 +++++++
 mm/memcontrol.c            |  182 ++++++++++++++++++++++++++++++++++++++- mm/page_alloc.c            |    2 +
 mm/vmscan.c                |  205 +++++++++++++++++++++++++++++++++++++++++++-

 4 files changed, 416 insertions(+), 3 deletions(-)
include/linux/memcontrol.h +25 -0
index 90fe7fe..dbed45d 100644
@@ -127,6 +127,12 @@ unsigned long mem_cgroup_soft_limit_reclaim(struct zone *zone, int order,
                                               gfp_t gfp_mask);
  u64 mem_cgroup_get_limit(struct mem_cgroup *mem);
 
+void mem_cgroup_clear_unreclaimable(struct page *page, struct zone *zone);
+bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid, int zid);
+bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem, struct zone *zone);
+void mem_cgroup_mz_set_unreclaimable(struct mem_cgroup *mem, struct zone *zone);
+void mem_cgroup_mz_pages_scanned(struct mem_cgroup *mem, struct zone* zone,
+                                     unsigned long nr_scanned);
  #else /* CONFIG_CGROUP_MEM_RES_CTLR */
  struct mem_cgroup;
 
@@ -299,6 +305,25 @@ static inline void mem_cgroup_update_file_mapped(struct page *page,
  {
  }
 
+static inline void mem_cgroup_mz_pages_scanned(struct mem_cgroup *mem,
+                                             struct zone *zone,
+                                             unsigned long nr_scanned)
+{
+}
+
+static inline void mem_cgroup_clear_unreclaimable(struct page *page,
+                                                     struct zone *zone)
+{
+}
+static inline void mem_cgroup_mz_set_unreclaimable(struct mem_cgroup *mem,
+             struct zone *zone)
+{
+}
+static inline bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem,
+                                             struct zone *zone)
+{
+}
+

 static inline
 unsigned long mem_cgroup_soft_limit_reclaim(struct zone *zone, int order, gfp_t gfp_mask)

include/linux/memcontrol.h +5 -0
@@ -312,6 +337,11 @@ u64 mem_cgroup_get_limit(struct mem_cgroup *mem)
       return 0;
  }
 
+static inline bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid,
+                                                             int zid)
+{
+     return false;
+}
  #endif /* CONFIG_CGROUP_MEM_CONT */
 
  #endif /* _LINUX_MEMCONTROL_H */
mm/memcontrol.c +2 -0
index a0c6ed9..1d39b65 100644
@@ -48,6 +48,8 @@
  #include <linux/page_cgroup.h>
  #include <linux/cpu.h>
  #include <linux/oom.h>
+#include <linux/kthread.h>
+
  #include "internal.h"
 
  #include <asm/uaccess.h>
@@ -118,7 +120,10 @@ struct mem_cgroup_per_zone {
       bool                    on_tree;

      struct mem_cgroup       *mem;           /* Back pointer, we cannot */ /* use container_of        */
+     unsigned long           pages_scanned;  /* since last reclaim */
+     int                     all_unreclaimable;      /* All pages pinned */ };
+
 /* Macro for accessing counter */
 #define MEM_CGROUP_ZSTAT(mz, idx)    ((mz)->count[(idx)])

mm/memcontrol.c +1 -0
@@ -372,6 +377,7 @@ static void mem_cgroup_put(struct mem_cgroup *mem);
  static struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *mem);
  static void drain_all_stock_async(void);
  static unsigned long get_min_free_kbytes(struct mem_cgroup *mem);
+static inline void wake_memcg_kswapd(struct mem_cgroup *mem);
 

 static struct mem_cgroup_per_zone * mem_cgroup_zoneinfo(struct mem_cgroup *mem, int nid, int zid)

mm/memcontrol.c +80 -0
@@ -1086,6 +1092,106 @@ mem_cgroup_get_reclaim_stat_from_page(struct page *page)
       return &mz->reclaim_stat;
  }
 
+unsigned long mem_cgroup_zone_reclaimable_pages(
+                                     struct mem_cgroup_per_zone *mz)
+{
+     int nr;
+     nr = MEM_CGROUP_ZSTAT(mz, LRU_ACTIVE_FILE) +
+             MEM_CGROUP_ZSTAT(mz, LRU_INACTIVE_FILE);
+
+     if (nr_swap_pages > 0)
+             nr += MEM_CGROUP_ZSTAT(mz, LRU_ACTIVE_ANON) +
+                     MEM_CGROUP_ZSTAT(mz, LRU_INACTIVE_ANON);
+
+     return nr;
+}
+
+void mem_cgroup_mz_pages_scanned(struct mem_cgroup *mem, struct zone* zone,
+                                             unsigned long nr_scanned)
+{
+     struct mem_cgroup_per_zone *mz = NULL;
+     int nid = zone_to_nid(zone);
+     int zid = zone_idx(zone);
+
+     if (!mem)
+             return;
+
+     mz = mem_cgroup_zoneinfo(mem, nid, zid);
+     if (mz)
+             mz->pages_scanned += nr_scanned;
+}
+
+bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid, int zid)
+{
+     struct mem_cgroup_per_zone *mz = NULL;
+
+     if (!mem)
+             return 0;
+
+     mz = mem_cgroup_zoneinfo(mem, nid, zid);
+     if (mz)
+             return mz->pages_scanned <
+                             mem_cgroup_zone_reclaimable_pages(mz) * 6;
+     return 0;
+}
+
+bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem, struct zone *zone)
+{
+     struct mem_cgroup_per_zone *mz = NULL;
+     int nid = zone_to_nid(zone);
+     int zid = zone_idx(zone);
+
+     if (!mem)
+             return 0;
+
+     mz = mem_cgroup_zoneinfo(mem, nid, zid);
+     if (mz)
+             return mz->all_unreclaimable;
+
+     return 0;
+}
+
+void mem_cgroup_mz_set_unreclaimable(struct mem_cgroup *mem, struct zone *zone)
+{
+     struct mem_cgroup_per_zone *mz = NULL;
+     int nid = zone_to_nid(zone);
+     int zid = zone_idx(zone);
+
+     if (!mem)
+             return;
+
+     mz = mem_cgroup_zoneinfo(mem, nid, zid);
+     if (mz)
+             mz->all_unreclaimable = 1;
+}
+
+void mem_cgroup_clear_unreclaimable(struct page *page, struct zone *zone)
+{
+     struct mem_cgroup_per_zone *mz = NULL;
+     struct mem_cgroup *mem = NULL;
+     int nid = zone_to_nid(zone);
+     int zid = zone_idx(zone);
+     struct page_cgroup *pc = lookup_page_cgroup(page);
Quoted 8 lines

mm/memcontrol.c mem_cgroup_get_reclaim_stat_from_page(struct page *page) · line 1180

Quoted 6 lines
mm/memcontrol.c +6 -0
+
+     if (unlikely(!pc))
+             return;
+
+     rcu_read_lock();
+     mem = pc->mem_cgroup;

This is incorrect. you have to do css_tryget(&mem->css) before rcu_read_unlock.

Thanks. This will be changed in the next post.

Quoted 30 lines
Quoted 30 lines
mm/memcontrol.c +22 -0
+     rcu_read_unlock();
+
+     if (!mem)
+             return;
+
+     mz = mem_cgroup_zoneinfo(mem, nid, zid);
+     if (mz) {
+             mz->pages_scanned = 0;
+             mz->all_unreclaimable = 0;
+     }
+
+     return;
+}
+
  unsigned long mem_cgroup_isolate_pages(unsigned long nr_to_scan,
                                       struct list_head *dst,
                                       unsigned long *scanned, int order,
@@ -1887,6 +1993,20 @@ static int __mem_cgroup_do_charge(struct mem_cgroup *mem, gfp_t gfp_mask,
       struct res_counter *fail_res;
       unsigned long flags = 0;
       int ret;
+     unsigned long min_free_kbytes = 0;
+
+     min_free_kbytes = get_min_free_kbytes(mem);
+     if (min_free_kbytes) {
+             ret = res_counter_charge(&mem->res, csize, CHARGE_WMARK_LOW,
+                                     &fail_res);
+             if (likely(!ret)) {
+                     return CHARGE_OK;
Quoted 8 lines

mm/memcontrol.c static int __mem_cgroup_do_charge(struct mem_cgroup *mem, gfp_t gfp_mask, · line 2009

Quoted 6 lines
mm/memcontrol.c +6 -0
+             } else {
+                     mem_over_limit = mem_cgroup_from_res_counter(fail_res,
+                                                                     res);
+                     wake_memcg_kswapd(mem_over_limit);
+             }
+     }

I think this check can be moved out to periodic-check as threshould notifiers.

I have to check how the threshold notifier works. If the periodic-check causes delay of triggering kswapd, we might end up relying on ttfp as we do now.

Quoted 13 lines
Quoted 13 lines
mm/memcontrol.c +0 -0
       ret = res_counter_charge(&mem->res, csize, CHARGE_WMARK_MIN, &fail_res);
 
@@ -3037,6 +3157,7 @@ static int mem_cgroup_resize_limit(struct mem_cgroup *memcg,

                      else
                              memcg->memsw_is_minimum = false; }
+             setup_per_memcg_wmarks(memcg); mutex_unlock(&set_limit_mutex);

Quoted 12 lines
Quoted 10 lines

              if (!ret)

mm/memcontrol.c +1 -1
@@ -3046,7 +3167,7 @@ static int mem_cgroup_resize_limit(struct mem_cgroup *memcg,
                                               MEM_CGROUP_RECLAIM_SHRINK);
               curusage = res_counter_read_u64(&memcg->res, RES_USAGE);
               /* Usage is reduced ? */
-             if (curusage >= oldusage)
+             if (curusage >= oldusage)
                       retry_count--;

              else
                      oldusage = curusage;

What's changed here ?

Hmm. will change in the next patch.

Quoted 62 lines
Quoted 48 lines
mm/memcontrol.c +0 -0
@@ -3096,6 +3217,7 @@ static int mem_cgroup_resize_memsw_limit(struct mem_cgroup *memcg,

                      else
                              memcg->memsw_is_minimum = false; }
+             setup_per_memcg_wmarks(memcg); mutex_unlock(&set_limit_mutex);

              if (!ret)

mm/memcontrol.c +11 -0
@@ -4352,6 +4474,8 @@ static struct mem_cgroup *mem_cgroup_alloc(void)
  static void __mem_cgroup_free(struct mem_cgroup *mem)
  {
       int node;
+     struct kswapd *kswapd_p;
+     wait_queue_head_t *wait;
 
       mem_cgroup_remove_from_trees(mem);
       free_css_id(&mem_cgroup_subsys, &mem->css);
@@ -4360,6 +4484,15 @@ static void __mem_cgroup_free(struct mem_cgroup *mem)
               free_mem_cgroup_per_zone_info(mem, node);
 
       free_percpu(mem->stat);
+
+     wait = mem->kswapd_wait;
+     kswapd_p = container_of(wait, struct kswapd, kswapd_wait);
+     if (kswapd_p) {
+             if (kswapd_p->kswapd_task)
+                     kthread_stop(kswapd_p->kswapd_task);
+             kfree(kswapd_p);
+     }
+
       if (sizeof(struct mem_cgroup) < PAGE_SIZE)
               kfree(mem);

      else

mm/memcontrol.c +10 -0
@@ -4421,6 +4554,39 @@ int mem_cgroup_watermark_ok(struct mem_cgroup *mem,
       return ret;
  }
 
+static inline
+void wake_memcg_kswapd(struct mem_cgroup *mem)
+{
+     wait_queue_head_t *wait;
+     struct kswapd *kswapd_p;
+     struct task_struct *thr;
+     static char memcg_name[PATH_MAX];
+
+     if (!mem)
+             return;

mm/memcontrol.c int mem_cgroup_watermark_ok(struct mem_cgroup *mem, · line 4576

Quoted 10 lines
mm/memcontrol.c +10 -0
+
+     wait = mem->kswapd_wait;
+     kswapd_p = container_of(wait, struct kswapd, kswapd_wait);
+     if (!kswapd_p->kswapd_task) {
+             if (mem->css.cgroup)
+                     cgroup_path(mem->css.cgroup, memcg_name, PATH_MAX);
+             else
+                     sprintf(memcg_name, "no_name");
+
+             thr = kthread_run(kswapd, kswapd_p, "kswapd%s", memcg_name);

I don't think reusing the name of "kswapd" isn't good. and this name cannot be long as PATH_MAX...IIUC, this name is for comm[] field which is 16bytes long.

Quoted 3 lines

So, how about naming this as

 "memcg%d", mem->css.id ?

No strong objection with the name. :)

Quoted 13 lines

Exporing css.id will be okay if necessary.

mm/memcontrol.c int mem_cgroup_watermark_ok(struct mem_cgroup *mem, · line 4582

Quoted 6 lines
mm/memcontrol.c +6 -0
+             if (IS_ERR(thr))
+                     printk(KERN_INFO "Failed to start kswapd on memcg %d\n",
+                             0);
+             else
+                     kswapd_p->kswapd_task = thr;
+     }

Hmm, ok, then, kswapd-for-memcg is created when someone go over watermark

author
#39 lore raw

Re: [PATCH 3/4] Per cgroup background reclaim.

In reply to KAMEZAWA Hiroyuki
Quoted 242 lines · KAMEZAWA Hiroyuki
Quoted 197 lines · On Mon, 29 Nov 2010 22:49:44 -0800 Ying Han <[email protected]> wrote:

The current implementation of memcg only supports direct reclaim and this patch adds the support for background reclaim. Per cgroup background reclaim is needed which spreads out the memory pressure over longer period of time and smoothes out the system performance.

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup. The kswapd is sleeping in the wait queue headed at kswapd_wait field of a kswapd descriptor.

The kswapd() function now is shared between global and per cgroup kswapd thread. It is passed in with the kswapd descriptor which contains the information of either node or cgroup. Then the new function balance_mem_cgroup_pgdat is invoked if it is per cgroup kswapd thread. The balance_mem_cgroup_pgdat performs a priority loop similar to global reclaim. In each iteration it invokes balance_pgdat_node for all nodes on the system, which is a new function performs background reclaim per node. After reclaiming each node, it checks mem_cgroup_watermark_ok() and breaks the priority loop if returns true. A per memcg zone will be marked as "unreclaimable" if the scanning rate is much greater than the reclaiming rate on the per cgroup LRU. The bit is cleared when there is a page charged to the cgroup being freed. Kswapd breaks the priority loop if all the zones are marked as "unreclaimable".

---
 include/linux/memcontrol.h |   30 +++++++
 mm/memcontrol.c            |  182 ++++++++++++++++++++++++++++++++++++++- mm/page_alloc.c            |    2 +
 mm/vmscan.c                |  205 +++++++++++++++++++++++++++++++++++++++++++-

 4 files changed, 416 insertions(+), 3 deletions(-)
include/linux/memcontrol.h +25 -0
index 90fe7fe..dbed45d 100644
@@ -127,6 +127,12 @@ unsigned long mem_cgroup_soft_limit_reclaim(struct zone *zone, int order,
                                               gfp_t gfp_mask);
  u64 mem_cgroup_get_limit(struct mem_cgroup *mem);
 
+void mem_cgroup_clear_unreclaimable(struct page *page, struct zone *zone);
+bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid, int zid);
+bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem, struct zone *zone);
+void mem_cgroup_mz_set_unreclaimable(struct mem_cgroup *mem, struct zone *zone);
+void mem_cgroup_mz_pages_scanned(struct mem_cgroup *mem, struct zone* zone,
+                                     unsigned long nr_scanned);
  #else /* CONFIG_CGROUP_MEM_RES_CTLR */
  struct mem_cgroup;
 
@@ -299,6 +305,25 @@ static inline void mem_cgroup_update_file_mapped(struct page *page,
  {
  }
 
+static inline void mem_cgroup_mz_pages_scanned(struct mem_cgroup *mem,
+                                             struct zone *zone,
+                                             unsigned long nr_scanned)
+{
+}
+
+static inline void mem_cgroup_clear_unreclaimable(struct page *page,
+                                                     struct zone *zone)
+{
+}
+static inline void mem_cgroup_mz_set_unreclaimable(struct mem_cgroup *mem,
+             struct zone *zone)
+{
+}
+static inline bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem,
+                                             struct zone *zone)
+{
+}
+

 static inline
 unsigned long mem_cgroup_soft_limit_reclaim(struct zone *zone, int order, gfp_t gfp_mask)

include/linux/memcontrol.h +5 -0
@@ -312,6 +337,11 @@ u64 mem_cgroup_get_limit(struct mem_cgroup *mem)
       return 0;
  }
 
+static inline bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid,
+                                                             int zid)
+{
+     return false;
+}
  #endif /* CONFIG_CGROUP_MEM_CONT */
 
  #endif /* _LINUX_MEMCONTROL_H */
mm/memcontrol.c +2 -0
index a0c6ed9..1d39b65 100644
@@ -48,6 +48,8 @@
  #include <linux/page_cgroup.h>
  #include <linux/cpu.h>
  #include <linux/oom.h>
+#include <linux/kthread.h>
+
  #include "internal.h"
 
  #include <asm/uaccess.h>
@@ -118,7 +120,10 @@ struct mem_cgroup_per_zone {
       bool                    on_tree;

      struct mem_cgroup       *mem;           /* Back pointer, we cannot */ /* use container_of        */
+     unsigned long           pages_scanned;  /* since last reclaim */
+     int                     all_unreclaimable;      /* All pages pinned */ };
+
 /* Macro for accessing counter */
 #define MEM_CGROUP_ZSTAT(mz, idx)    ((mz)->count[(idx)])

mm/memcontrol.c +1 -0
@@ -372,6 +377,7 @@ static void mem_cgroup_put(struct mem_cgroup *mem);
  static struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *mem);
  static void drain_all_stock_async(void);
  static unsigned long get_min_free_kbytes(struct mem_cgroup *mem);
+static inline void wake_memcg_kswapd(struct mem_cgroup *mem);
 

 static struct mem_cgroup_per_zone * mem_cgroup_zoneinfo(struct mem_cgroup *mem, int nid, int zid)

mm/memcontrol.c +76 -0
@@ -1086,6 +1092,106 @@ mem_cgroup_get_reclaim_stat_from_page(struct page *page)
       return &mz->reclaim_stat;
  }
 
+unsigned long mem_cgroup_zone_reclaimable_pages(
+                                     struct mem_cgroup_per_zone *mz)
+{
+     int nr;
+     nr = MEM_CGROUP_ZSTAT(mz, LRU_ACTIVE_FILE) +
+             MEM_CGROUP_ZSTAT(mz, LRU_INACTIVE_FILE);
+
+     if (nr_swap_pages > 0)
+             nr += MEM_CGROUP_ZSTAT(mz, LRU_ACTIVE_ANON) +
+                     MEM_CGROUP_ZSTAT(mz, LRU_INACTIVE_ANON);
+
+     return nr;
+}
+
+void mem_cgroup_mz_pages_scanned(struct mem_cgroup *mem, struct zone* zone,
+                                             unsigned long nr_scanned)
+{
+     struct mem_cgroup_per_zone *mz = NULL;
+     int nid = zone_to_nid(zone);
+     int zid = zone_idx(zone);
+
+     if (!mem)
+             return;
+
+     mz = mem_cgroup_zoneinfo(mem, nid, zid);
+     if (mz)
+             mz->pages_scanned += nr_scanned;
+}
+
+bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid, int zid)
+{
+     struct mem_cgroup_per_zone *mz = NULL;
+
+     if (!mem)
+             return 0;
+
+     mz = mem_cgroup_zoneinfo(mem, nid, zid);
+     if (mz)
+             return mz->pages_scanned <
+                             mem_cgroup_zone_reclaimable_pages(mz) * 6;
+     return 0;
+}
+
+bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem, struct zone *zone)
+{
+     struct mem_cgroup_per_zone *mz = NULL;
+     int nid = zone_to_nid(zone);
+     int zid = zone_idx(zone);
+
+     if (!mem)
+             return 0;
+
+     mz = mem_cgroup_zoneinfo(mem, nid, zid);
+     if (mz)
+             return mz->all_unreclaimable;
+
+     return 0;
+}
+
+void mem_cgroup_mz_set_unreclaimable(struct mem_cgroup *mem, struct zone *zone)
+{
+     struct mem_cgroup_per_zone *mz = NULL;
+     int nid = zone_to_nid(zone);
+     int zid = zone_idx(zone);
+
+     if (!mem)
+             return;
+
+     mz = mem_cgroup_zoneinfo(mem, nid, zid);
+     if (mz)
+             mz->all_unreclaimable = 1;
+}
+
+void mem_cgroup_clear_unreclaimable(struct page *page, struct zone *zone)
+{
+     struct mem_cgroup_per_zone *mz = NULL;

mm/memcontrol.c mem_cgroup_get_reclaim_stat_from_page(struct page *page) · line 1180

Quoted 10 lines
mm/memcontrol.c +10 -0
+     struct mem_cgroup *mem = NULL;
+     int nid = zone_to_nid(zone);
+     int zid = zone_idx(zone);
+     struct page_cgroup *pc = lookup_page_cgroup(page);
+
+     if (unlikely(!pc))
+             return;
+
+     rcu_read_lock();
+     mem = pc->mem_cgroup;

This is incorrect. you have to do css_tryget(&mem->css) before rcu_read_unlock.

Quoted 29 lines
mm/memcontrol.c +22 -0
+     rcu_read_unlock();
+
+     if (!mem)
+             return;
+
+     mz = mem_cgroup_zoneinfo(mem, nid, zid);
+     if (mz) {
+             mz->pages_scanned = 0;
+             mz->all_unreclaimable = 0;
+     }
+
+     return;
+}
+
  unsigned long mem_cgroup_isolate_pages(unsigned long nr_to_scan,
                                       struct list_head *dst,
                                       unsigned long *scanned, int order,
@@ -1887,6 +1993,20 @@ static int __mem_cgroup_do_charge(struct mem_cgroup *mem, gfp_t gfp_mask,
       struct res_counter *fail_res;
       unsigned long flags = 0;
       int ret;
+     unsigned long min_free_kbytes = 0;
+
+     min_free_kbytes = get_min_free_kbytes(mem);
+     if (min_free_kbytes) {
+             ret = res_counter_charge(&mem->res, csize, CHARGE_WMARK_LOW,
+                                     &fail_res);
+             if (likely(!ret)) {
+                     return CHARGE_OK;
Quoted 8 lines

mm/memcontrol.c static int __mem_cgroup_do_charge(struct mem_cgroup *mem, gfp_t gfp_mask, · line 2009

Quoted 6 lines
mm/memcontrol.c +6 -0
+             } else {
+                     mem_over_limit = mem_cgroup_from_res_counter(fail_res,
+                                                                     res);
+                     wake_memcg_kswapd(mem_over_limit);
+             }
+     }

I think this check can be moved out to periodic-check as threshould notifiers.

Yes. This will be changed in V2.

Quoted 87 lines
Quoted 21 lines
mm/memcontrol.c +0 -0
       ret = res_counter_charge(&mem->res, csize, CHARGE_WMARK_MIN, &fail_res);
 
@@ -3037,6 +3157,7 @@ static int mem_cgroup_resize_limit(struct mem_cgroup *memcg,

                      else
                              memcg->memsw_is_minimum = false; }
+             setup_per_memcg_wmarks(memcg); mutex_unlock(&set_limit_mutex);

              if (!ret)

mm/memcontrol.c +1 -1
@@ -3046,7 +3167,7 @@ static int mem_cgroup_resize_limit(struct mem_cgroup *memcg,
                                               MEM_CGROUP_RECLAIM_SHRINK);
               curusage = res_counter_read_u64(&memcg->res, RES_USAGE);
               /* Usage is reduced ? */
-             if (curusage >= oldusage)
+             if (curusage >= oldusage)
                       retry_count--;
Quoted 2 lines

              else
                      oldusage = curusage;

What's changed here ?

Quoted 47 lines
mm/memcontrol.c +0 -0
@@ -3096,6 +3217,7 @@ static int mem_cgroup_resize_memsw_limit(struct mem_cgroup *memcg,

                      else
                              memcg->memsw_is_minimum = false; }
+             setup_per_memcg_wmarks(memcg); mutex_unlock(&set_limit_mutex);

              if (!ret)

mm/memcontrol.c +11 -0
@@ -4352,6 +4474,8 @@ static struct mem_cgroup *mem_cgroup_alloc(void)
  static void __mem_cgroup_free(struct mem_cgroup *mem)
  {
       int node;
+     struct kswapd *kswapd_p;
+     wait_queue_head_t *wait;
 
       mem_cgroup_remove_from_trees(mem);
       free_css_id(&mem_cgroup_subsys, &mem->css);
@@ -4360,6 +4484,15 @@ static void __mem_cgroup_free(struct mem_cgroup *mem)
               free_mem_cgroup_per_zone_info(mem, node);
 
       free_percpu(mem->stat);
+
+     wait = mem->kswapd_wait;
+     kswapd_p = container_of(wait, struct kswapd, kswapd_wait);
+     if (kswapd_p) {
+             if (kswapd_p->kswapd_task)
+                     kthread_stop(kswapd_p->kswapd_task);
+             kfree(kswapd_p);
+     }
+
       if (sizeof(struct mem_cgroup) < PAGE_SIZE)
               kfree(mem);

      else

mm/memcontrol.c +10 -0
@@ -4421,6 +4554,39 @@ int mem_cgroup_watermark_ok(struct mem_cgroup *mem,
       return ret;
  }
 
+static inline
+void wake_memcg_kswapd(struct mem_cgroup *mem)
+{
+     wait_queue_head_t *wait;
+     struct kswapd *kswapd_p;
+     struct task_struct *thr;
+     static char memcg_name[PATH_MAX];
+
+     if (!mem)
+             return;

mm/memcontrol.c int mem_cgroup_watermark_ok(struct mem_cgroup *mem, · line 4576

Quoted 10 lines
mm/memcontrol.c +10 -0
+
+     wait = mem->kswapd_wait;
+     kswapd_p = container_of(wait, struct kswapd, kswapd_wait);
+     if (!kswapd_p->kswapd_task) {
+             if (mem->css.cgroup)
+                     cgroup_path(mem->css.cgroup, memcg_name, PATH_MAX);
+             else
+                     sprintf(memcg_name, "no_name");
+
+             thr = kthread_run(kswapd, kswapd_p, "kswapd%s", memcg_name);

I don't think reusing the name of "kswapd" isn't good. and this name cannot be long as PATH_MAX...IIUC, this name is for comm[] field which is 16bytes long.

Quoted 5 lines

So, how about naming this as

 "memcg%d", mem->css.id ?

Exporing css.id will be okay if necessary.

I am not if that is working since the mem->css hasn't been initialized during mem_cgroup_create(). So that is one of the reasons that I put the kswapd creation at triggering wmarks instead of creating cgroup, since I have all that information ready by the time.

However, I agree that adding into the cgroup creation is better for performance perspective since we won't add the overhead for the page allocation. ( Although only the first wmark triggering ). Any suggestion?

--Ying

Quoted 11 lines

mm/memcontrol.c int mem_cgroup_watermark_ok(struct mem_cgroup *mem, · line 4582

Quoted 9 lines
mm/memcontrol.c +6 -0
+             if (IS_ERR(thr))
+                     printk(KERN_INFO "Failed to start kswapd on memcg %d\n",
+                             0);
+             else
+                     kswapd_p->kswapd_task = thr;
+     }

Hmm, ok, then, kswapd-for-memcg is created when someone go over watermark

#40 lore raw

Re: [PATCH 3/4] Per cgroup background reclaim.

Quoted 353 lines · On Mon, 6 Dec 2010 18:25:55 -0800 Ying Han <[email protected]> wrote:
Quoted 242 lines · KAMEZAWA Hiroyuki
Quoted 197 lines · On Mon, 29 Nov 2010 22:49:44 -0800 Ying Han <[email protected]> wrote:

The current implementation of memcg only supports direct reclaim and this patch adds the support for background reclaim. Per cgroup background reclaim is needed which spreads out the memory pressure over longer period of time and smoothes out the system performance.

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup. The kswapd is sleeping in the wait queue headed at kswapd_wait field of a kswapd descriptor.

The kswapd() function now is shared between global and per cgroup kswapd thread. It is passed in with the kswapd descriptor which contains the information of either node or cgroup. Then the new function balance_mem_cgroup_pgdat is invoked if it is per cgroup kswapd thread. The balance_mem_cgroup_pgdat performs a priority loop similar to global reclaim. In each iteration it invokes balance_pgdat_node for all nodes on the system, which is a new function performs background reclaim per node. After reclaiming each node, it checks mem_cgroup_watermark_ok() and breaks the priority loop if returns true. A per memcg zone will be marked as "unreclaimable" if the scanning rate is much greater than the reclaiming rate on the per cgroup LRU. The bit is cleared when there is a page charged to the cgroup being freed. Kswapd breaks the priority loop if all the zones are marked as "unreclaimable".

---
A include/linux/memcontrol.h | A 30 +++++++

A mm/memcontrol.c A  A  A  A  A  A | A 182 ++++++++++++++++++++++++++++++++++++++-
A mm/page_alloc.c A  A  A  A  A  A | A  A 2 +
A mm/vmscan.c A  A  A  A  A  A  A  A | A 205 +++++++++++++++++++++++++++++++++++++++++++-

A 4 files changed, 416 insertions(+), 3 deletions(-)

include/linux/memcontrol.h +6 -0
index 90fe7fe..dbed45d 100644
@@ -127,6 +127,12 @@ unsigned long mem_cgroup_soft_limit_reclaim(struct zone *zone, int order,
 A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  gfp_t gfp_mask);
 A u64 mem_cgroup_get_limit(struct mem_cgroup *mem);
 
+void mem_cgroup_clear_unreclaimable(struct page *page, struct zone *zone);
+bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid, int zid);
+bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem, struct zone *zone);
+void mem_cgroup_mz_set_unreclaimable(struct mem_cgroup *mem, struct zone *zone);
+void mem_cgroup_mz_pages_scanned(struct mem_cgroup *mem, struct zone* zone,
+ A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  unsigned long nr_scanned);

A #else /* CONFIG_CGROUP_MEM_RES_CTLR */
A struct mem_cgroup;

include/linux/memcontrol.h +19 -0
@@ -299,6 +305,25 @@ static inline void mem_cgroup_update_file_mapped(struct page *page,
 A {
 A }
 
+static inline void mem_cgroup_mz_pages_scanned(struct mem_cgroup *mem,
+ A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  struct zone *zone,
+ A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  unsigned long nr_scanned)
+{
+}
+
+static inline void mem_cgroup_clear_unreclaimable(struct page *page,
+ A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  struct zone *zone)
+{
+}
+static inline void mem_cgroup_mz_set_unreclaimable(struct mem_cgroup *mem,
+ A  A  A  A  A  A  struct zone *zone)
+{
+}
+static inline bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem,
+ A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  struct zone *zone)
+{
+}
+

A static inline
A unsigned long mem_cgroup_soft_limit_reclaim(struct zone *zone, int order, A A A A A A A A A A A A A A A A A A A A A gfp_t gfp_mask)

include/linux/memcontrol.h +5 -0
@@ -312,6 +337,11 @@ u64 mem_cgroup_get_limit(struct mem_cgroup *mem)
 A  A  A  return 0;
 A }
 
+static inline bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid,
+ A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  int zid)
+{
+ A  A  return false;
+}

A #endif /* CONFIG_CGROUP_MEM_CONT */

A #endif /* _LINUX_MEMCONTROL_H */

mm/memcontrol.c +0 -0
index a0c6ed9..1d39b65 100644
@@ -48,6 +48,8 @@

A #include <linux/page_cgroup.h>
A #include <linux/cpu.h>
A #include <linux/oom.h>

mm/memcontrol.c +2 -0
+#include <linux/kthread.h>
+

A #include "internal.h"

A #include <asm/uaccess.h>

mm/memcontrol.c +0 -0
@@ -118,7 +120,10 @@ struct mem_cgroup_per_zone {
 A  A  A  bool A  A  A  A  A  A  A  A  A  A on_tree;
A  A  A  struct mem_cgroup A  A  A  *mem; A  A  A  A  A  /* Back pointer, we cannot */
A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  /* use container_of A  A  A  A */
+ A  A  unsigned long A  A  A  A  A  pages_scanned; A /* since last reclaim */
+ A  A  int A  A  A  A  A  A  A  A  A  A  all_unreclaimable; A  A  A /* All pages pinned */

A };
+
A /* Macro for accessing counter */
A #define MEM_CGROUP_ZSTAT(mz, idx) A A ((mz)->count[(idx)])

mm/memcontrol.c +1 -0
@@ -372,6 +377,7 @@ static void mem_cgroup_put(struct mem_cgroup *mem);
 A static struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *mem);
 A static void drain_all_stock_async(void);
 A static unsigned long get_min_free_kbytes(struct mem_cgroup *mem);
+static inline void wake_memcg_kswapd(struct mem_cgroup *mem);
 

A static struct mem_cgroup_per_zone *
A mem_cgroup_zoneinfo(struct mem_cgroup *mem, int nid, int zid)

mm/memcontrol.c +76 -0
@@ -1086,6 +1092,106 @@ mem_cgroup_get_reclaim_stat_from_page(struct page *page)
 A  A  A  return &mz->reclaim_stat;
 A }
 
+unsigned long mem_cgroup_zone_reclaimable_pages(
+ A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  struct mem_cgroup_per_zone *mz)
+{
+ A  A  int nr;
+ A  A  nr = MEM_CGROUP_ZSTAT(mz, LRU_ACTIVE_FILE) +
+ A  A  A  A  A  A  MEM_CGROUP_ZSTAT(mz, LRU_INACTIVE_FILE);
+
+ A  A  if (nr_swap_pages > 0)
+ A  A  A  A  A  A  nr += MEM_CGROUP_ZSTAT(mz, LRU_ACTIVE_ANON) +
+ A  A  A  A  A  A  A  A  A  A  MEM_CGROUP_ZSTAT(mz, LRU_INACTIVE_ANON);
+
+ A  A  return nr;
+}
+
+void mem_cgroup_mz_pages_scanned(struct mem_cgroup *mem, struct zone* zone,
+ A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  unsigned long nr_scanned)
+{
+ A  A  struct mem_cgroup_per_zone *mz = NULL;
+ A  A  int nid = zone_to_nid(zone);
+ A  A  int zid = zone_idx(zone);
+
+ A  A  if (!mem)
+ A  A  A  A  A  A  return;
+
+ A  A  mz = mem_cgroup_zoneinfo(mem, nid, zid);
+ A  A  if (mz)
+ A  A  A  A  A  A  mz->pages_scanned += nr_scanned;
+}
+
+bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid, int zid)
+{
+ A  A  struct mem_cgroup_per_zone *mz = NULL;
+
+ A  A  if (!mem)
+ A  A  A  A  A  A  return 0;
+
+ A  A  mz = mem_cgroup_zoneinfo(mem, nid, zid);
+ A  A  if (mz)
+ A  A  A  A  A  A  return mz->pages_scanned <
+ A  A  A  A  A  A  A  A  A  A  A  A  A  A  mem_cgroup_zone_reclaimable_pages(mz) * 6;
+ A  A  return 0;
+}
+
+bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem, struct zone *zone)
+{
+ A  A  struct mem_cgroup_per_zone *mz = NULL;
+ A  A  int nid = zone_to_nid(zone);
+ A  A  int zid = zone_idx(zone);
+
+ A  A  if (!mem)
+ A  A  A  A  A  A  return 0;
+
+ A  A  mz = mem_cgroup_zoneinfo(mem, nid, zid);
+ A  A  if (mz)
+ A  A  A  A  A  A  return mz->all_unreclaimable;
+
+ A  A  return 0;
+}
+
+void mem_cgroup_mz_set_unreclaimable(struct mem_cgroup *mem, struct zone *zone)
+{
+ A  A  struct mem_cgroup_per_zone *mz = NULL;
+ A  A  int nid = zone_to_nid(zone);
+ A  A  int zid = zone_idx(zone);
+
+ A  A  if (!mem)
+ A  A  A  A  A  A  return;
+
+ A  A  mz = mem_cgroup_zoneinfo(mem, nid, zid);
+ A  A  if (mz)
+ A  A  A  A  A  A  mz->all_unreclaimable = 1;
+}
+
+void mem_cgroup_clear_unreclaimable(struct page *page, struct zone *zone)
+{
+ A  A  struct mem_cgroup_per_zone *mz = NULL;

mm/memcontrol.c mem_cgroup_get_reclaim_stat_from_page(struct page *page) · line 1180

Quoted 10 lines
mm/memcontrol.c +10 -0
+ A  A  struct mem_cgroup *mem = NULL;
+ A  A  int nid = zone_to_nid(zone);
+ A  A  int zid = zone_idx(zone);
+ A  A  struct page_cgroup *pc = lookup_page_cgroup(page);
+
+ A  A  if (unlikely(!pc))
+ A  A  A  A  A  A  return;
+
+ A  A  rcu_read_lock();
+ A  A  mem = pc->mem_cgroup;

This is incorrect. you have to do css_tryget(&mem->css) before rcu_read_unlock.

Quoted 29 lines
mm/memcontrol.c +22 -0
+ A  A  rcu_read_unlock();
+
+ A  A  if (!mem)
+ A  A  A  A  A  A  return;
+
+ A  A  mz = mem_cgroup_zoneinfo(mem, nid, zid);
+ A  A  if (mz) {
+ A  A  A  A  A  A  mz->pages_scanned = 0;
+ A  A  A  A  A  A  mz->all_unreclaimable = 0;
+ A  A  }
+
+ A  A  return;
+}
+
 A unsigned long mem_cgroup_isolate_pages(unsigned long nr_to_scan,
 A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  struct list_head *dst,
 A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  unsigned long *scanned, int order,
@@ -1887,6 +1993,20 @@ static int __mem_cgroup_do_charge(struct mem_cgroup *mem, gfp_t gfp_mask,
 A  A  A  struct res_counter *fail_res;
 A  A  A  unsigned long flags = 0;
 A  A  A  int ret;
+ A  A  unsigned long min_free_kbytes = 0;
+
+ A  A  min_free_kbytes = get_min_free_kbytes(mem);
+ A  A  if (min_free_kbytes) {
+ A  A  A  A  A  A  ret = res_counter_charge(&mem->res, csize, CHARGE_WMARK_LOW,
+ A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  &fail_res);
+ A  A  A  A  A  A  if (likely(!ret)) {
+ A  A  A  A  A  A  A  A  A  A  return CHARGE_OK;
Quoted 8 lines

mm/memcontrol.c static int __mem_cgroup_do_charge(struct mem_cgroup *mem, gfp_t gfp_mask, · line 2009

Quoted 6 lines
mm/memcontrol.c +6 -0
+ A  A  A  A  A  A  } else {
+ A  A  A  A  A  A  A  A  A  A  mem_over_limit = mem_cgroup_from_res_counter(fail_res,
+ A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  res);
+ A  A  A  A  A  A  A  A  A  A  wake_memcg_kswapd(mem_over_limit);
+ A  A  A  A  A  A  }
+ A  A  }

I think this check can be moved out to periodic-check as threshould notifiers.

Yes. This will be changed in V2.

Quoted 87 lines
Quoted 18 lines
mm/memcontrol.c +0 -0
 A  A  A  ret = res_counter_charge(&mem->res, csize, CHARGE_WMARK_MIN, &fail_res);
 
@@ -3037,6 +3157,7 @@ static int mem_cgroup_resize_limit(struct mem_cgroup *memcg,
A  A  A  A  A  A  A  A  A  A  A  else
A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  memcg->memsw_is_minimum = false;
A  A  A  A  A  A  A  }

+ A A A A A A setup_per_memcg_wmarks(memcg); A A A A A A A mutex_unlock(&set_limit_mutex);

A A A A A A A if (!ret)

mm/memcontrol.c +0 -0
@@ -3046,7 +3167,7 @@ static int mem_cgroup_resize_limit(struct mem_cgroup *memcg,
 A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  MEM_CGROUP_RECLAIM_SHRINK);
 A  A  A  A  A  A  A  curusage = res_counter_read_u64(&memcg->res, RES_USAGE);

A A A A A A A /* Usage is reduced ? */

Quoted 5 lines
- A  A  A  A  A  A  if (curusage >= oldusage)
+ A  A  A  A  A  A  if (curusage >= oldusage)
A  A  A  A  A  A  A  A  A  A  A  retry_count--;
A  A  A  A  A  A  A  else
A  A  A  A  A  A  A  A  A  A  A  oldusage = curusage;

What's changed here ?

Quoted 47 lines
mm/memcontrol.c +0 -0
@@ -3096,6 +3217,7 @@ static int mem_cgroup_resize_memsw_limit(struct mem_cgroup *memcg,
A  A  A  A  A  A  A  A  A  A  A  else
A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  memcg->memsw_is_minimum = false;
A  A  A  A  A  A  A  }

+ A A A A A A setup_per_memcg_wmarks(memcg); A A A A A A A mutex_unlock(&set_limit_mutex);

A A A A A A A if (!ret)

mm/memcontrol.c +11 -0
@@ -4352,6 +4474,8 @@ static struct mem_cgroup *mem_cgroup_alloc(void)
 A static void __mem_cgroup_free(struct mem_cgroup *mem)
 A {
 A  A  A  int node;
+ A  A  struct kswapd *kswapd_p;
+ A  A  wait_queue_head_t *wait;
 
 A  A  A  mem_cgroup_remove_from_trees(mem);
 A  A  A  free_css_id(&mem_cgroup_subsys, &mem->css);
@@ -4360,6 +4484,15 @@ static void __mem_cgroup_free(struct mem_cgroup *mem)
 A  A  A  A  A  A  A  free_mem_cgroup_per_zone_info(mem, node);
 
 A  A  A  free_percpu(mem->stat);
+
+ A  A  wait = mem->kswapd_wait;
+ A  A  kswapd_p = container_of(wait, struct kswapd, kswapd_wait);
+ A  A  if (kswapd_p) {
+ A  A  A  A  A  A  if (kswapd_p->kswapd_task)
+ A  A  A  A  A  A  A  A  A  A  kthread_stop(kswapd_p->kswapd_task);
+ A  A  A  A  A  A  kfree(kswapd_p);
+ A  A  }
+
 A  A  A  if (sizeof(struct mem_cgroup) < PAGE_SIZE)
 A  A  A  A  A  A  A  kfree(mem);

A A A else

mm/memcontrol.c +10 -0
@@ -4421,6 +4554,39 @@ int mem_cgroup_watermark_ok(struct mem_cgroup *mem,
 A  A  A  return ret;
 A }
 
+static inline
+void wake_memcg_kswapd(struct mem_cgroup *mem)
+{
+ A  A  wait_queue_head_t *wait;
+ A  A  struct kswapd *kswapd_p;
+ A  A  struct task_struct *thr;
+ A  A  static char memcg_name[PATH_MAX];
+
+ A  A  if (!mem)
+ A  A  A  A  A  A  return;

mm/memcontrol.c int mem_cgroup_watermark_ok(struct mem_cgroup *mem, · line 4576

Quoted 10 lines
mm/memcontrol.c +10 -0
+
+ A  A  wait = mem->kswapd_wait;
+ A  A  kswapd_p = container_of(wait, struct kswapd, kswapd_wait);
+ A  A  if (!kswapd_p->kswapd_task) {
+ A  A  A  A  A  A  if (mem->css.cgroup)
+ A  A  A  A  A  A  A  A  A  A  cgroup_path(mem->css.cgroup, memcg_name, PATH_MAX);
+ A  A  A  A  A  A  else
+ A  A  A  A  A  A  A  A  A  A  sprintf(memcg_name, "no_name");
+
+ A  A  A  A  A  A  thr = kthread_run(kswapd, kswapd_p, "kswapd%s", memcg_name);

I don't think reusing the name of "kswapd" isn't good. and this name cannot be long as PATH_MAX...IIUC, this name is for comm[] field which is 16bytes long.

Quoted 5 lines

So, how about naming this as

A "memcg%d", mem->css.id ?

Exporing css.id will be okay if necessary.

I am not if that is working since the mem->css hasn't been initialized during mem_cgroup_create(). So that is one of the reasons that I put the kswapd creation at triggering wmarks instead of creating cgroup, since I have all that information ready by the time.

Quoted 5 lines

However, I agree that adding into the cgroup creation is better for performance perspective since we won't add the overhead for the page allocation. ( Although only the first wmark triggering ). Any suggestion?

Hmm, my recommendation is to start the thread when the limit is set.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#41 lore raw

Re: [PATCH 3/4] Per cgroup background reclaim.

In reply to Ying Han
Quoted 13 lines
mm/page_alloc.c +1 -0
index a15bc1c..dc61f2a 100644
@@ -615,6 +615,7 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 
 		do {
 			page = list_entry(list->prev, struct page, lru);
+			mem_cgroup_clear_unreclaimable(page, zone);
 			/* must delete as __free_one_page list manipulates */
 			list_del(&page->lru);
 			/* MIGRATE_MOVABLE list may include MIGRATE_RESERVEs */

mm/page_alloc.c static void free_one_page(struct zone *zone, struct page *page, int order, · line 639

Quoted 7 lines
mm/page_alloc.c +1 -0
@@ -632,6 +633,7 @@ static void free_one_page(struct zone *zone, struct page *page, int order,
 	spin_lock(&zone->lock);
 	zone->all_unreclaimable = 0;
 	zone->pages_scanned = 0;
+	mem_cgroup_clear_unreclaimable(page, zone);
 
 	__free_one_page(page, zone, order, migratetype);
 	__mod_zone_page_state(zone, NR_FREE_PAGES, 1 << order);

Please don't do that. free page is one of fast path. We don't want to add additonal overhead here.

So I would like to explain why we clear zone->all_unreclaimable in free page path at first. Look, zone free pages are maintained by NR_FREE_PAGES and free_one_page modify it.

But, free_one_page() is unrelated to memory cgroup uncharge thing. If nobody does memcg uncharge, reclaim retrying is pointless. no? I think we have better place than here.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#42 lore raw

Re: [PATCH 3/4] Per cgroup background reclaim.

Quoted 24 lines · On Wed, 1 Dec 2010 11:18:45 +0900 (JST) KOSAKI Motohiro…
Quoted 13 lines
mm/page_alloc.c +1 -0
index a15bc1c..dc61f2a 100644
@@ -615,6 +615,7 @@ static void free_pcppages_bulk(struct zone *zone, int count,
 
 		do {
 			page = list_entry(list->prev, struct page, lru);
+			mem_cgroup_clear_unreclaimable(page, zone);
 			/* must delete as __free_one_page list manipulates */
 			list_del(&page->lru);
 			/* MIGRATE_MOVABLE list may include MIGRATE_RESERVEs */

mm/page_alloc.c static void free_one_page(struct zone *zone, struct page *page, int order, · line 639

Quoted 7 lines
mm/page_alloc.c +1 -0
@@ -632,6 +633,7 @@ static void free_one_page(struct zone *zone, struct page *page, int order,
 	spin_lock(&zone->lock);
 	zone->all_unreclaimable = 0;
 	zone->pages_scanned = 0;
+	mem_cgroup_clear_unreclaimable(page, zone);
 
 	__free_one_page(page, zone, order, migratetype);
 	__mod_zone_page_state(zone, NR_FREE_PAGES, 1 << order);

Please don't do that. free page is one of fast path. We don't want to add additonal overhead here.

Quoted 8 lines

So I would like to explain why we clear zone->all_unreclaimable in free page path at first. Look, zone free pages are maintained by NR_FREE_PAGES and free_one_page modify it.

But, free_one_page() is unrelated to memory cgroup uncharge thing. If nobody does memcg uncharge, reclaim retrying is pointless. no? I think we have better place than here.

I agree. Should be done in uncharge or event counter.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#43 lore raw

PATCH 4/4 Add more per memcg stats.

In reply to Ying Han

A bunch of statistics are added in memory.stat to monitor per cgroup kswapd performance.

---
 include/linux/memcontrol.h |   81 +++++++++++++++++++++++++
 mm/memcontrol.c            |  140 ++++++++++++++++++++++++++++++++++++++++++++
 mm/vmscan.c                |   33 +++++++++-
 3 files changed, 250 insertions(+), 4 deletions(-)
include/linux/memcontrol.h +81 -0
index dbed45d..893ca62 100644
@@ -127,6 +127,19 @@ unsigned long mem_cgroup_soft_limit_reclaim(struct zone *zone, int order,
 						gfp_t gfp_mask);
 u64 mem_cgroup_get_limit(struct mem_cgroup *mem);
 
+/* background reclaim stats */
+void mem_cgroup_kswapd_steal(struct mem_cgroup *memcg, int val);
+void mem_cgroup_pg_steal(struct mem_cgroup *memcg, int val);
+void mem_cgroup_kswapd_pgscan(struct mem_cgroup *memcg, int val);
+void mem_cgroup_pg_pgscan(struct mem_cgroup *memcg, int val);
+void mem_cgroup_pgrefill(struct mem_cgroup *memcg, int val);
+void mem_cgroup_pg_outrun(struct mem_cgroup *memcg, int val);
+void mem_cgroup_alloc_stall(struct mem_cgroup *memcg, int val);
+void mem_cgroup_balance_wmark_ok(struct mem_cgroup *memcg, int val);
+void mem_cgroup_balance_swap_max(struct mem_cgroup *memcg, int val);
+void mem_cgroup_kswapd_shrink_zone(struct mem_cgroup *memcg, int val);
+void mem_cgroup_kswapd_may_writepage(struct mem_cgroup *memcg, int val);
+
 void mem_cgroup_clear_unreclaimable(struct page *page, struct zone *zone);
 bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid, int zid);
 bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem, struct zone *zone);
@@ -337,6 +350,74 @@ u64 mem_cgroup_get_limit(struct mem_cgroup *mem)
 	return 0;
 }
 
+/* background reclaim stats */
+static inline void mem_cgroup_kswapd_steal(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_pg_steal(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_kswapd_pgscan(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_pg_pgscan(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_pgrefill(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_pg_outrun(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_alloc_stall(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_balance_wmark_ok(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_balance_swap_max(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_kswapd_shrink_zone(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+
+static inline void mem_cgroup_kswapd_may_writepage(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
 static inline bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid,
 								int zid)
 {
mm/memcontrol.c +140 -0

187 lines not on this page · raw message

mm/vmscan.c +29 -4
index f8430c4..5b0c349 100644
@@ -1389,10 +1389,15 @@ shrink_inactive_list(unsigned long nr_to_scan, struct zone *zone,
 					ISOLATE_INACTIVE : ISOLATE_BOTH,
 			zone, sc->mem_cgroup,
 			0, file);
+		mem_cgroup_mz_pages_scanned(sc->mem_cgroup, zone, nr_scanned);
 		/*
 		 * mem_cgroup_isolate_pages() keeps track of
 		 * scanned pages on its own.
 		 */
+		if (current_is_kswapd())
+			mem_cgroup_kswapd_pgscan(sc->mem_cgroup, nr_scanned);
+		else
+			mem_cgroup_pg_pgscan(sc->mem_cgroup, nr_scanned);
 	}
 
 	if (nr_taken == 0) {
@@ -1413,9 +1418,16 @@ shrink_inactive_list(unsigned long nr_to_scan, struct zone *zone,
 	}
 
 	local_irq_disable();
-	if (current_is_kswapd())
-		__count_vm_events(KSWAPD_STEAL, nr_reclaimed);
-	__count_zone_vm_events(PGSTEAL, zone, nr_reclaimed);
+	if (scanning_global_lru(sc)) {
+		if (current_is_kswapd())
+			__count_vm_events(KSWAPD_STEAL, nr_reclaimed);
+		__count_zone_vm_events(PGSTEAL, zone, nr_reclaimed);
+	} else {
+		if (current_is_kswapd())
+			mem_cgroup_kswapd_steal(sc->mem_cgroup, nr_reclaimed);
+		else
+			mem_cgroup_pg_steal(sc->mem_cgroup, nr_reclaimed);
+	}
 
 	putback_lru_pages(zone, sc, nr_anon, nr_file, &page_list);
 
@@ -1508,11 +1520,16 @@ static void shrink_active_list(unsigned long nr_pages, struct zone *zone,
 		 * mem_cgroup_isolate_pages() keeps track of
 		 * scanned pages on its own.
 		 */
+		mem_cgroup_mz_pages_scanned(sc->mem_cgroup, zone, pgscanned);
 	}
 
 	reclaim_stat->recent_scanned[file] += nr_taken;
 
-	__count_zone_vm_events(PGREFILL, zone, pgscanned);
+	if (scanning_global_lru(sc))
+		__count_zone_vm_events(PGREFILL, zone, pgscanned);
+	else
+		mem_cgroup_pgrefill(sc->mem_cgroup, pgscanned);
+
 	if (file)
 		__mod_zone_page_state(zone, NR_ACTIVE_FILE, -nr_taken);
 	else
@@ -1955,6 +1972,8 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,
 
 	if (scanning_global_lru(sc))
 		count_vm_event(ALLOCSTALL);
+	else
+		mem_cgroup_alloc_stall(sc->mem_cgroup, 1);
 
 	for (priority = DEF_PRIORITY; priority >= 0; priority--) {
 		sc->nr_scanned = 0;
@@ -2444,6 +2463,8 @@ scan:
 			priority != DEF_PRIORITY)
 			continue;
 
+		mem_cgroup_kswapd_shrink_zone(mem_cont, 1);
+
 		sc->nr_scanned = 0;
 		shrink_zone(priority, zone, sc);
 		total_scanned += sc->nr_scanned;
@@ -2462,6 +2483,7 @@ scan:
 		if (total_scanned > SWAP_CLUSTER_MAX * 2 &&
 		    total_scanned > sc->nr_reclaimed + sc->nr_reclaimed / 2) {
 			sc->may_writepage = 1;
+			mem_cgroup_kswapd_may_writepage(mem_cont, 1);
 		}
 	}
 
@@ -2504,6 +2526,8 @@ loop_again:
 	sc.nr_reclaimed = 0;
 	total_scanned = 0;
 
+	mem_cgroup_pg_outrun(mem_cont, 1);
+
 	for (priority = DEF_PRIORITY; priority >= 0; priority--) {
 		sc.priority = priority;
 
@@ -2544,6 +2568,7 @@ loop_again:
 				wmark_ok = 0;
 
 			if (wmark_ok) {
+				mem_cgroup_balance_wmark_ok(sc.mem_cgroup, 1);
 				goto out;
 			}
 		}
Signature · 8 lines
1.7.3.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#44 lore raw

Re: [PATCH 4/4] Add more per memcg stats.

Quoted 4 lines · On Mon, 29 Nov 2010 22:49:45 -0800 Ying Han <[email protected]> wrote:

A bunch of statistics are added in memory.stat to monitor per cgroup kswapd performance.

No objections. But please update the documenation and add more comments.

Thanks,
-Kame

Quoted 414 lines
---
 include/linux/memcontrol.h |   81 +++++++++++++++++++++++++
 mm/memcontrol.c            |  140 ++++++++++++++++++++++++++++++++++++++++++++
 mm/vmscan.c                |   33 +++++++++-
 3 files changed, 250 insertions(+), 4 deletions(-)
include/linux/memcontrol.h +81 -0
index dbed45d..893ca62 100644
@@ -127,6 +127,19 @@ unsigned long mem_cgroup_soft_limit_reclaim(struct zone *zone, int order,
 						gfp_t gfp_mask);
 u64 mem_cgroup_get_limit(struct mem_cgroup *mem);
 
+/* background reclaim stats */
+void mem_cgroup_kswapd_steal(struct mem_cgroup *memcg, int val);
+void mem_cgroup_pg_steal(struct mem_cgroup *memcg, int val);
+void mem_cgroup_kswapd_pgscan(struct mem_cgroup *memcg, int val);
+void mem_cgroup_pg_pgscan(struct mem_cgroup *memcg, int val);
+void mem_cgroup_pgrefill(struct mem_cgroup *memcg, int val);
+void mem_cgroup_pg_outrun(struct mem_cgroup *memcg, int val);
+void mem_cgroup_alloc_stall(struct mem_cgroup *memcg, int val);
+void mem_cgroup_balance_wmark_ok(struct mem_cgroup *memcg, int val);
+void mem_cgroup_balance_swap_max(struct mem_cgroup *memcg, int val);
+void mem_cgroup_kswapd_shrink_zone(struct mem_cgroup *memcg, int val);
+void mem_cgroup_kswapd_may_writepage(struct mem_cgroup *memcg, int val);
+
 void mem_cgroup_clear_unreclaimable(struct page *page, struct zone *zone);
 bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid, int zid);
 bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem, struct zone *zone);
@@ -337,6 +350,74 @@ u64 mem_cgroup_get_limit(struct mem_cgroup *mem)
 	return 0;
 }
 
+/* background reclaim stats */
+static inline void mem_cgroup_kswapd_steal(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_pg_steal(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_kswapd_pgscan(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_pg_pgscan(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_pgrefill(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_pg_outrun(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_alloc_stall(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_balance_wmark_ok(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_balance_swap_max(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+static inline void mem_cgroup_kswapd_shrink_zone(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
+
+static inline void mem_cgroup_kswapd_may_writepage(struct mem_cgroup *memcg,
+								int val)
+{
+	return 0;
+}
+
 static inline bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid,
 								int zid)
 {
mm/memcontrol.c +140 -0
index 1d39b65..97df6dd 100644
@@ -91,6 +91,21 @@ enum mem_cgroup_stat_index {
 	MEM_CGROUP_STAT_PGPGIN_COUNT,	/* # of pages paged in */
 	MEM_CGROUP_STAT_PGPGOUT_COUNT,	/* # of pages paged out */
 	MEM_CGROUP_STAT_SWAPOUT, /* # of pages, swapped out */
+	MEM_CGROUP_STAT_KSWAPD_INVOKE, /* # of times invokes kswapd */
+	MEM_CGROUP_STAT_KSWAPD_STEAL, /* # of pages reclaimed from kswapd */
+	MEM_CGROUP_STAT_PG_PGSTEAL, /* # of pages reclaimed from ttfp */
+	MEM_CGROUP_STAT_KSWAPD_PGSCAN, /* # of pages scanned from kswapd */
+	MEM_CGROUP_STAT_PG_PGSCAN, /* # of pages scanned from ttfp */
+	MEM_CGROUP_STAT_PGREFILL, /* # of pages scanned on active list */
+	MEM_CGROUP_STAT_WMARK_LOW_OK,
+	MEM_CGROUP_STAT_KSWAP_CREAT,
+	MEM_CGROUP_STAT_PGOUTRUN,
+	MEM_CGROUP_STAT_ALLOCSTALL,
+	MEM_CGROUP_STAT_BALANCE_WMARK_OK,
+	MEM_CGROUP_STAT_BALANCE_SWAP_MAX,
+	MEM_CGROUP_STAT_WAITQUEUE,
+	MEM_CGROUP_STAT_KSWAPD_SHRINK_ZONE,
+	MEM_CGROUP_STAT_KSWAPD_MAY_WRITEPAGE,
 	MEM_CGROUP_STAT_DATA, /* end of data requires synchronization */
 	/* incremented at every  pagein/pageout */
 	MEM_CGROUP_EVENTS = MEM_CGROUP_STAT_DATA,
@@ -619,6 +634,62 @@ static void mem_cgroup_swap_statistics(struct mem_cgroup *mem,
 	this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_SWAPOUT], val);
 }
 
+void mem_cgroup_kswapd_steal(struct mem_cgroup *mem, int val)
+{
+	this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_KSWAPD_STEAL], val);
+}
+
+void mem_cgroup_pg_steal(struct mem_cgroup *mem, int val)
+{
+	this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_PG_PGSTEAL], val);
+}
+
+void mem_cgroup_kswapd_pgscan(struct mem_cgroup *mem, int val)
+{
+	this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_KSWAPD_PGSCAN], val);
+}
+
+void mem_cgroup_pg_pgscan(struct mem_cgroup *mem, int val)
+{
+	this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_PG_PGSCAN], val);
+}
+
+void mem_cgroup_pgrefill(struct mem_cgroup *mem, int val)
+{
+	this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_PGREFILL], val);
+}
+
+void mem_cgroup_pg_outrun(struct mem_cgroup *mem, int val)
+{
+	this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_PGOUTRUN], val);
+}
+
+void mem_cgroup_alloc_stall(struct mem_cgroup *mem, int val)
+{
+	this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_ALLOCSTALL], val);
+}
+
+void mem_cgroup_balance_wmark_ok(struct mem_cgroup *mem, int val)
+{
+	this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_BALANCE_WMARK_OK], val);
+}
+
+void mem_cgroup_balance_swap_max(struct mem_cgroup *mem, int val)
+{
+	this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_BALANCE_SWAP_MAX], val);
+}
+
+void mem_cgroup_kswapd_shrink_zone(struct mem_cgroup *mem, int val)
+{
+	this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_KSWAPD_SHRINK_ZONE], val);
+}
+
+void mem_cgroup_kswapd_may_writepage(struct mem_cgroup *mem, int val)
+{
+	this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_KSWAPD_MAY_WRITEPAGE],
+			val);
+}
+
 static void mem_cgroup_charge_statistics(struct mem_cgroup *mem,
 					 struct page_cgroup *pc,
 					 bool charge)
@@ -2000,8 +2071,14 @@ static int __mem_cgroup_do_charge(struct mem_cgroup *mem, gfp_t gfp_mask,
 		ret = res_counter_charge(&mem->res, csize, CHARGE_WMARK_LOW,
 					&fail_res);
 		if (likely(!ret)) {
+			this_cpu_add(
+				mem->stat->count[MEM_CGROUP_STAT_WMARK_LOW_OK],
+				1);
 			return CHARGE_OK;
 		} else {
+			this_cpu_add(
+				mem->stat->count[MEM_CGROUP_STAT_KSWAPD_INVOKE],
+				1);
 			mem_over_limit = mem_cgroup_from_res_counter(fail_res,
 									res);
 			wake_memcg_kswapd(mem_over_limit);
@@ -3723,6 +3800,21 @@ enum {
 	MCS_PGPGIN,
 	MCS_PGPGOUT,
 	MCS_SWAP,
+	MCS_KSWAPD_INVOKE,
+	MCS_KSWAPD_STEAL,
+	MCS_PG_PGSTEAL,
+	MCS_KSWAPD_PGSCAN,
+	MCS_PG_PGSCAN,
+	MCS_PGREFILL,
+	MCS_WMARK_LOW_OK,
+	MCS_KSWAP_CREAT,
+	MCS_PGOUTRUN,
+	MCS_ALLOCSTALL,
+	MCS_BALANCE_WMARK_OK,
+	MCS_BALANCE_SWAP_MAX,
+	MCS_WAITQUEUE,
+	MCS_KSWAPD_SHRINK_ZONE,
+	MCS_KSWAPD_MAY_WRITEPAGE,
 	MCS_INACTIVE_ANON,
 	MCS_ACTIVE_ANON,
 	MCS_INACTIVE_FILE,
@@ -3745,6 +3837,21 @@ struct {
 	{"pgpgin", "total_pgpgin"},
 	{"pgpgout", "total_pgpgout"},
 	{"swap", "total_swap"},
+	{"kswapd_invoke", "total_kswapd_invoke"},
+	{"kswapd_steal", "total_kswapd_steal"},
+	{"pg_pgsteal", "total_pg_pgsteal"},
+	{"kswapd_pgscan", "total_kswapd_pgscan"},
+	{"pg_scan", "total_pg_scan"},
+	{"pgrefill", "total_pgrefill"},
+	{"wmark_low_ok", "total_wmark_low_ok"},
+	{"kswapd_create", "total_kswapd_create"},
+	{"pgoutrun", "total_pgoutrun"},
+	{"allocstall", "total_allocstall"},
+	{"balance_wmark_ok", "total_balance_wmark_ok"},
+	{"balance_swap_max", "total_balance_swap_max"},
+	{"waitqueue", "total_waitqueue"},
+	{"kswapd_shrink_zone", "total_kswapd_shrink_zone"},
+	{"kswapd_may_writepage", "total_kswapd_may_writepage"},
 	{"inactive_anon", "total_inactive_anon"},
 	{"active_anon", "total_active_anon"},
 	{"inactive_file", "total_inactive_file"},
@@ -3773,6 +3880,37 @@ mem_cgroup_get_local_stat(struct mem_cgroup *mem, struct mcs_total_stat *s)
 		val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_SWAPOUT);
 		s->stat[MCS_SWAP] += val * PAGE_SIZE;
 	}
+	/* kswapd stat */
+	val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_KSWAPD_INVOKE);
+	s->stat[MCS_KSWAPD_INVOKE] += val;
+	val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_KSWAPD_STEAL);
+	s->stat[MCS_KSWAPD_STEAL] += val;
+	val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_PG_PGSTEAL);
+	s->stat[MCS_PG_PGSTEAL] += val;
+	val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_KSWAPD_PGSCAN);
+	s->stat[MCS_KSWAPD_PGSCAN] += val;
+	val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_PG_PGSCAN);
+	s->stat[MCS_PG_PGSCAN] += val;
+	val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_PGREFILL);
+	s->stat[MCS_PGREFILL] += val;
+	val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_WMARK_LOW_OK);
+	s->stat[MCS_WMARK_LOW_OK] += val;
+	val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_KSWAP_CREAT);
+	s->stat[MCS_KSWAP_CREAT] += val;
+	val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_PGOUTRUN);
+	s->stat[MCS_PGOUTRUN] += val;
+	val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_ALLOCSTALL);
+	s->stat[MCS_ALLOCSTALL] += val;
+	val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_BALANCE_WMARK_OK);
+	s->stat[MCS_BALANCE_WMARK_OK] += val;
+	val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_BALANCE_SWAP_MAX);
+	s->stat[MCS_BALANCE_SWAP_MAX] += val;
+	val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_WAITQUEUE);
+	s->stat[MCS_WAITQUEUE] += val;
+	val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_KSWAPD_SHRINK_ZONE);
+	s->stat[MCS_KSWAPD_SHRINK_ZONE] += val;
+	val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_KSWAPD_MAY_WRITEPAGE);
+	s->stat[MCS_KSWAPD_MAY_WRITEPAGE] += val;
 
 	/* per zone stat */
 	val = mem_cgroup_get_local_zonestat(mem, LRU_INACTIVE_ANON);
@@ -4579,9 +4717,11 @@ void wake_memcg_kswapd(struct mem_cgroup *mem)
 				0);
 		else
 			kswapd_p->kswapd_task = thr;
+		this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_KSWAP_CREAT], 1);
 	}
 
 	if (!waitqueue_active(wait)) {
+		this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_WAITQUEUE], 1);
 		return;
 	}
 	wake_up_interruptible(wait);
mm/vmscan.c +29 -4
index f8430c4..5b0c349 100644
@@ -1389,10 +1389,15 @@ shrink_inactive_list(unsigned long nr_to_scan, struct zone *zone,
 					ISOLATE_INACTIVE : ISOLATE_BOTH,
 			zone, sc->mem_cgroup,
 			0, file);
+		mem_cgroup_mz_pages_scanned(sc->mem_cgroup, zone, nr_scanned);
 		/*
 		 * mem_cgroup_isolate_pages() keeps track of
 		 * scanned pages on its own.
 		 */
+		if (current_is_kswapd())
+			mem_cgroup_kswapd_pgscan(sc->mem_cgroup, nr_scanned);
+		else
+			mem_cgroup_pg_pgscan(sc->mem_cgroup, nr_scanned);
 	}
 
 	if (nr_taken == 0) {
@@ -1413,9 +1418,16 @@ shrink_inactive_list(unsigned long nr_to_scan, struct zone *zone,
 	}
 
 	local_irq_disable();
-	if (current_is_kswapd())
-		__count_vm_events(KSWAPD_STEAL, nr_reclaimed);
-	__count_zone_vm_events(PGSTEAL, zone, nr_reclaimed);
+	if (scanning_global_lru(sc)) {
+		if (current_is_kswapd())
+			__count_vm_events(KSWAPD_STEAL, nr_reclaimed);
+		__count_zone_vm_events(PGSTEAL, zone, nr_reclaimed);
+	} else {
+		if (current_is_kswapd())
+			mem_cgroup_kswapd_steal(sc->mem_cgroup, nr_reclaimed);
+		else
+			mem_cgroup_pg_steal(sc->mem_cgroup, nr_reclaimed);
+	}
 
 	putback_lru_pages(zone, sc, nr_anon, nr_file, &page_list);
 
@@ -1508,11 +1520,16 @@ static void shrink_active_list(unsigned long nr_pages, struct zone *zone,
 		 * mem_cgroup_isolate_pages() keeps track of
 		 * scanned pages on its own.
 		 */
+		mem_cgroup_mz_pages_scanned(sc->mem_cgroup, zone, pgscanned);
 	}
 
 	reclaim_stat->recent_scanned[file] += nr_taken;
 
-	__count_zone_vm_events(PGREFILL, zone, pgscanned);
+	if (scanning_global_lru(sc))
+		__count_zone_vm_events(PGREFILL, zone, pgscanned);
+	else
+		mem_cgroup_pgrefill(sc->mem_cgroup, pgscanned);
+
 	if (file)
 		__mod_zone_page_state(zone, NR_ACTIVE_FILE, -nr_taken);
 	else
@@ -1955,6 +1972,8 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,
 
 	if (scanning_global_lru(sc))
 		count_vm_event(ALLOCSTALL);
+	else
+		mem_cgroup_alloc_stall(sc->mem_cgroup, 1);
 
 	for (priority = DEF_PRIORITY; priority >= 0; priority--) {
 		sc->nr_scanned = 0;
@@ -2444,6 +2463,8 @@ scan:
 			priority != DEF_PRIORITY)
 			continue;
 
+		mem_cgroup_kswapd_shrink_zone(mem_cont, 1);
+
 		sc->nr_scanned = 0;
 		shrink_zone(priority, zone, sc);
 		total_scanned += sc->nr_scanned;
@@ -2462,6 +2483,7 @@ scan:
 		if (total_scanned > SWAP_CLUSTER_MAX * 2 &&
 		    total_scanned > sc->nr_reclaimed + sc->nr_reclaimed / 2) {
 			sc->may_writepage = 1;
+			mem_cgroup_kswapd_may_writepage(mem_cont, 1);
 		}
 	}
 
@@ -2504,6 +2526,8 @@ loop_again:
 	sc.nr_reclaimed = 0;
 	total_scanned = 0;
 
+	mem_cgroup_pg_outrun(mem_cont, 1);
+
 	for (priority = DEF_PRIORITY; priority >= 0; priority--) {
 		sc.priority = priority;
 
@@ -2544,6 +2568,7 @@ loop_again:
 				wmark_ok = 0;
 
 			if (wmark_ok) {
+				mem_cgroup_balance_wmark_ok(sc.mem_cgroup, 1);
 				goto out;
 			}
 		}
Signature · 8 lines
1.7.3.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#45 lore raw

Re: [PATCH 4/4] Add more per memcg stats.

Quoted 9 lines · KAMEZAWA Hiroyuki
Quoted 4 lines · On Mon, 29 Nov 2010 22:49:45 -0800 Ying Han <[email protected]> wrote:

A bunch of statistics are added in memory.stat to monitor per cgroup kswapd performance.

No objections. But please update the documenation and add more comments.

Sure. will do.

Thanks

--Ying

Quoted 411 lines

Thanks,
-Kame

Quoted 407 lines

---
 include/linux/memcontrol.h |   81 +++++++++++++++++++++++++ mm/memcontrol.c            |  140 ++++++++++++++++++++++++++++++++++++++++++++ mm/vmscan.c                |   33 +++++++++-

 3 files changed, 250 insertions(+), 4 deletions(-)
include/linux/memcontrol.h +81 -0
index dbed45d..893ca62 100644
@@ -127,6 +127,19 @@ unsigned long mem_cgroup_soft_limit_reclaim(struct zone *zone, int order,
                                               gfp_t gfp_mask);
  u64 mem_cgroup_get_limit(struct mem_cgroup *mem);
 
+/* background reclaim stats */
+void mem_cgroup_kswapd_steal(struct mem_cgroup *memcg, int val);
+void mem_cgroup_pg_steal(struct mem_cgroup *memcg, int val);
+void mem_cgroup_kswapd_pgscan(struct mem_cgroup *memcg, int val);
+void mem_cgroup_pg_pgscan(struct mem_cgroup *memcg, int val);
+void mem_cgroup_pgrefill(struct mem_cgroup *memcg, int val);
+void mem_cgroup_pg_outrun(struct mem_cgroup *memcg, int val);
+void mem_cgroup_alloc_stall(struct mem_cgroup *memcg, int val);
+void mem_cgroup_balance_wmark_ok(struct mem_cgroup *memcg, int val);
+void mem_cgroup_balance_swap_max(struct mem_cgroup *memcg, int val);
+void mem_cgroup_kswapd_shrink_zone(struct mem_cgroup *memcg, int val);
+void mem_cgroup_kswapd_may_writepage(struct mem_cgroup *memcg, int val);
+
  void mem_cgroup_clear_unreclaimable(struct page *page, struct zone *zone);
  bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid, int zid);
  bool mem_cgroup_mz_unreclaimable(struct mem_cgroup *mem, struct zone *zone);
@@ -337,6 +350,74 @@ u64 mem_cgroup_get_limit(struct mem_cgroup *mem)
       return 0;
  }
 
+/* background reclaim stats */
+static inline void mem_cgroup_kswapd_steal(struct mem_cgroup *memcg,
+                                                             int val)
+{
+     return 0;
+}
+
+static inline void mem_cgroup_pg_steal(struct mem_cgroup *memcg,
+                                                             int val)
+{
+     return 0;
+}
+
+static inline void mem_cgroup_kswapd_pgscan(struct mem_cgroup *memcg,
+                                                             int val)
+{
+     return 0;
+}
+
+static inline void mem_cgroup_pg_pgscan(struct mem_cgroup *memcg,
+                                                             int val)
+{
+     return 0;
+}
+
+static inline void mem_cgroup_pgrefill(struct mem_cgroup *memcg,
+                                                             int val)
+{
+     return 0;
+}
+
+static inline void mem_cgroup_pg_outrun(struct mem_cgroup *memcg,
+                                                             int val)
+{
+     return 0;
+}
+
+static inline void mem_cgroup_alloc_stall(struct mem_cgroup *memcg,
+                                                             int val)
+{
+     return 0;
+}
+
+static inline void mem_cgroup_balance_wmark_ok(struct mem_cgroup *memcg,
+                                                             int val)
+{
+     return 0;
+}
+
+static inline void mem_cgroup_balance_swap_max(struct mem_cgroup *memcg,
+                                                             int val)
+{
+     return 0;
+}
+
+static inline void mem_cgroup_kswapd_shrink_zone(struct mem_cgroup *memcg,
+                                                             int val)
+{
+     return 0;
+}
+
+
+static inline void mem_cgroup_kswapd_may_writepage(struct mem_cgroup *memcg,
+                                                             int val)
+{
+     return 0;
+}
+
  static inline bool mem_cgroup_zone_reclaimable(struct mem_cgroup *mem, int nid,
                                                               int zid)
  {
mm/memcontrol.c +0 -0
index 1d39b65..97df6dd 100644
@@ -91,6 +91,21 @@ enum mem_cgroup_stat_index {

      MEM_CGROUP_STAT_PGPGIN_COUNT,   /* # of pages paged in */ MEM_CGROUP_STAT_PGPGOUT_COUNT,  /* # of pages paged out */ MEM_CGROUP_STAT_SWAPOUT, /* # of pages, swapped out */
+     MEM_CGROUP_STAT_KSWAPD_INVOKE, /* # of times invokes kswapd */
+     MEM_CGROUP_STAT_KSWAPD_STEAL, /* # of pages reclaimed from kswapd */
+     MEM_CGROUP_STAT_PG_PGSTEAL, /* # of pages reclaimed from ttfp */
+     MEM_CGROUP_STAT_KSWAPD_PGSCAN, /* # of pages scanned from kswapd */
+     MEM_CGROUP_STAT_PG_PGSCAN, /* # of pages scanned from ttfp */
+     MEM_CGROUP_STAT_PGREFILL, /* # of pages scanned on active list */
+     MEM_CGROUP_STAT_WMARK_LOW_OK,
+     MEM_CGROUP_STAT_KSWAP_CREAT,
+     MEM_CGROUP_STAT_PGOUTRUN,
+     MEM_CGROUP_STAT_ALLOCSTALL,
+     MEM_CGROUP_STAT_BALANCE_WMARK_OK,
+     MEM_CGROUP_STAT_BALANCE_SWAP_MAX,
+     MEM_CGROUP_STAT_WAITQUEUE,
+     MEM_CGROUP_STAT_KSWAPD_SHRINK_ZONE,
+     MEM_CGROUP_STAT_KSWAPD_MAY_WRITEPAGE, MEM_CGROUP_STAT_DATA, /* end of data requires synchronization */ /* incremented at every  pagein/pageout */ MEM_CGROUP_EVENTS = MEM_CGROUP_STAT_DATA,

mm/memcontrol.c +123 -0
@@ -619,6 +634,62 @@ static void mem_cgroup_swap_statistics(struct mem_cgroup *mem,
       this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_SWAPOUT], val);
  }
 
+void mem_cgroup_kswapd_steal(struct mem_cgroup *mem, int val)
+{
+     this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_KSWAPD_STEAL], val);
+}
+
+void mem_cgroup_pg_steal(struct mem_cgroup *mem, int val)
+{
+     this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_PG_PGSTEAL], val);
+}
+
+void mem_cgroup_kswapd_pgscan(struct mem_cgroup *mem, int val)
+{
+     this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_KSWAPD_PGSCAN], val);
+}
+
+void mem_cgroup_pg_pgscan(struct mem_cgroup *mem, int val)
+{
+     this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_PG_PGSCAN], val);
+}
+
+void mem_cgroup_pgrefill(struct mem_cgroup *mem, int val)
+{
+     this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_PGREFILL], val);
+}
+
+void mem_cgroup_pg_outrun(struct mem_cgroup *mem, int val)
+{
+     this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_PGOUTRUN], val);
+}
+
+void mem_cgroup_alloc_stall(struct mem_cgroup *mem, int val)
+{
+     this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_ALLOCSTALL], val);
+}
+
+void mem_cgroup_balance_wmark_ok(struct mem_cgroup *mem, int val)
+{
+     this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_BALANCE_WMARK_OK], val);
+}
+
+void mem_cgroup_balance_swap_max(struct mem_cgroup *mem, int val)
+{
+     this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_BALANCE_SWAP_MAX], val);
+}
+
+void mem_cgroup_kswapd_shrink_zone(struct mem_cgroup *mem, int val)
+{
+     this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_KSWAPD_SHRINK_ZONE], val);
+}
+
+void mem_cgroup_kswapd_may_writepage(struct mem_cgroup *mem, int val)
+{
+     this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_KSWAPD_MAY_WRITEPAGE],
+                     val);
+}
+
  static void mem_cgroup_charge_statistics(struct mem_cgroup *mem,
                                        struct page_cgroup *pc,
                                        bool charge)
@@ -2000,8 +2071,14 @@ static int __mem_cgroup_do_charge(struct mem_cgroup *mem, gfp_t gfp_mask,
               ret = res_counter_charge(&mem->res, csize, CHARGE_WMARK_LOW,
                                       &fail_res);
               if (likely(!ret)) {
+                     this_cpu_add(
+                             mem->stat->count[MEM_CGROUP_STAT_WMARK_LOW_OK],
+                             1);
                       return CHARGE_OK;
               } else {
+                     this_cpu_add(
+                             mem->stat->count[MEM_CGROUP_STAT_KSWAPD_INVOKE],
+                             1);
                       mem_over_limit = mem_cgroup_from_res_counter(fail_res,
                                                                       res);
                       wake_memcg_kswapd(mem_over_limit);
@@ -3723,6 +3800,21 @@ enum {
       MCS_PGPGIN,
       MCS_PGPGOUT,
       MCS_SWAP,
+     MCS_KSWAPD_INVOKE,
+     MCS_KSWAPD_STEAL,
+     MCS_PG_PGSTEAL,
+     MCS_KSWAPD_PGSCAN,
+     MCS_PG_PGSCAN,
+     MCS_PGREFILL,
+     MCS_WMARK_LOW_OK,
+     MCS_KSWAP_CREAT,
+     MCS_PGOUTRUN,
+     MCS_ALLOCSTALL,
+     MCS_BALANCE_WMARK_OK,
+     MCS_BALANCE_SWAP_MAX,
+     MCS_WAITQUEUE,
+     MCS_KSWAPD_SHRINK_ZONE,
+     MCS_KSWAPD_MAY_WRITEPAGE,
       MCS_INACTIVE_ANON,
       MCS_ACTIVE_ANON,
       MCS_INACTIVE_FILE,
@@ -3745,6 +3837,21 @@ struct {
       {"pgpgin", "total_pgpgin"},
       {"pgpgout", "total_pgpgout"},
       {"swap", "total_swap"},
+     {"kswapd_invoke", "total_kswapd_invoke"},
+     {"kswapd_steal", "total_kswapd_steal"},
+     {"pg_pgsteal", "total_pg_pgsteal"},
+     {"kswapd_pgscan", "total_kswapd_pgscan"},
+     {"pg_scan", "total_pg_scan"},
+     {"pgrefill", "total_pgrefill"},
+     {"wmark_low_ok", "total_wmark_low_ok"},
+     {"kswapd_create", "total_kswapd_create"},
+     {"pgoutrun", "total_pgoutrun"},
+     {"allocstall", "total_allocstall"},
+     {"balance_wmark_ok", "total_balance_wmark_ok"},
+     {"balance_swap_max", "total_balance_swap_max"},
+     {"waitqueue", "total_waitqueue"},
+     {"kswapd_shrink_zone", "total_kswapd_shrink_zone"},
+     {"kswapd_may_writepage", "total_kswapd_may_writepage"},
       {"inactive_anon", "total_inactive_anon"},
       {"active_anon", "total_active_anon"},
       {"inactive_file", "total_inactive_file"},
@@ -3773,6 +3880,37 @@ mem_cgroup_get_local_stat(struct mem_cgroup *mem, struct mcs_total_stat *s)
               val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_SWAPOUT);
               s->stat[MCS_SWAP] += val * PAGE_SIZE;
       }
+     /* kswapd stat */
+     val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_KSWAPD_INVOKE);
+     s->stat[MCS_KSWAPD_INVOKE] += val;
+     val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_KSWAPD_STEAL);
+     s->stat[MCS_KSWAPD_STEAL] += val;
+     val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_PG_PGSTEAL);
+     s->stat[MCS_PG_PGSTEAL] += val;
+     val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_KSWAPD_PGSCAN);
+     s->stat[MCS_KSWAPD_PGSCAN] += val;
+     val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_PG_PGSCAN);
+     s->stat[MCS_PG_PGSCAN] += val;
+     val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_PGREFILL);
+     s->stat[MCS_PGREFILL] += val;
+     val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_WMARK_LOW_OK);
+     s->stat[MCS_WMARK_LOW_OK] += val;
+     val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_KSWAP_CREAT);
+     s->stat[MCS_KSWAP_CREAT] += val;
+     val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_PGOUTRUN);
+     s->stat[MCS_PGOUTRUN] += val;
+     val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_ALLOCSTALL);
+     s->stat[MCS_ALLOCSTALL] += val;
+     val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_BALANCE_WMARK_OK);
+     s->stat[MCS_BALANCE_WMARK_OK] += val;
+     val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_BALANCE_SWAP_MAX);
+     s->stat[MCS_BALANCE_SWAP_MAX] += val;
+     val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_WAITQUEUE);
+     s->stat[MCS_WAITQUEUE] += val;
+     val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_KSWAPD_SHRINK_ZONE);
+     s->stat[MCS_KSWAPD_SHRINK_ZONE] += val;
+     val = mem_cgroup_read_stat(mem, MEM_CGROUP_STAT_KSWAPD_MAY_WRITEPAGE);
+     s->stat[MCS_KSWAPD_MAY_WRITEPAGE] += val;
 
       /* per zone stat */
       val = mem_cgroup_get_local_zonestat(mem, LRU_INACTIVE_ANON);
@@ -4579,9 +4717,11 @@ void wake_memcg_kswapd(struct mem_cgroup *mem)
                               0);

              else
                      kswapd_p->kswapd_task = thr;
+             this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_KSWAP_CREAT], 1); }

      if (!waitqueue_active(wait)) {
+             this_cpu_add(mem->stat->count[MEM_CGROUP_STAT_WAITQUEUE], 1); return;
      }
      wake_up_interruptible(wait);

mm/vmscan.c +21 -4
index f8430c4..5b0c349 100644
@@ -1389,10 +1389,15 @@ shrink_inactive_list(unsigned long nr_to_scan, struct zone *zone,
                                       ISOLATE_INACTIVE : ISOLATE_BOTH,
                       zone, sc->mem_cgroup,
                       0, file);
+             mem_cgroup_mz_pages_scanned(sc->mem_cgroup, zone, nr_scanned);
               /*
                * mem_cgroup_isolate_pages() keeps track of
                * scanned pages on its own.
                */
+             if (current_is_kswapd())
+                     mem_cgroup_kswapd_pgscan(sc->mem_cgroup, nr_scanned);
+             else
+                     mem_cgroup_pg_pgscan(sc->mem_cgroup, nr_scanned);
       }
 
       if (nr_taken == 0) {
@@ -1413,9 +1418,16 @@ shrink_inactive_list(unsigned long nr_to_scan, struct zone *zone,
       }
 
       local_irq_disable();
-     if (current_is_kswapd())
-             __count_vm_events(KSWAPD_STEAL, nr_reclaimed);
-     __count_zone_vm_events(PGSTEAL, zone, nr_reclaimed);
+     if (scanning_global_lru(sc)) {
+             if (current_is_kswapd())
+                     __count_vm_events(KSWAPD_STEAL, nr_reclaimed);
+             __count_zone_vm_events(PGSTEAL, zone, nr_reclaimed);
+     } else {
+             if (current_is_kswapd())
+                     mem_cgroup_kswapd_steal(sc->mem_cgroup, nr_reclaimed);
+             else
+                     mem_cgroup_pg_steal(sc->mem_cgroup, nr_reclaimed);
+     }
 
       putback_lru_pages(zone, sc, nr_anon, nr_file, &page_list);
 
@@ -1508,11 +1520,16 @@ static void shrink_active_list(unsigned long nr_pages, struct zone *zone,
                * mem_cgroup_isolate_pages() keeps track of
                * scanned pages on its own.
                */
+             mem_cgroup_mz_pages_scanned(sc->mem_cgroup, zone, pgscanned);
       }
 
       reclaim_stat->recent_scanned[file] += nr_taken;
 
-     __count_zone_vm_events(PGREFILL, zone, pgscanned);
+     if (scanning_global_lru(sc))
+             __count_zone_vm_events(PGREFILL, zone, pgscanned);
+     else
+             mem_cgroup_pgrefill(sc->mem_cgroup, pgscanned);
+
       if (file)
               __mod_zone_page_state(zone, NR_ACTIVE_FILE, -nr_taken);

      else

mm/vmscan.c +4 -0
@@ -1955,6 +1972,8 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,
 
       if (scanning_global_lru(sc))
               count_vm_event(ALLOCSTALL);
+     else
+             mem_cgroup_alloc_stall(sc->mem_cgroup, 1);
 
       for (priority = DEF_PRIORITY; priority >= 0; priority--) {
               sc->nr_scanned = 0;
@@ -2444,6 +2463,8 @@ scan:
                       priority != DEF_PRIORITY)
                       continue;
 
+             mem_cgroup_kswapd_shrink_zone(mem_cont, 1);
+
               sc->nr_scanned = 0;
               shrink_zone(priority, zone, sc);
               total_scanned += sc->nr_scanned;
@@ -2462,6 +2483,7 @@ scan:

              if (total_scanned > SWAP_CLUSTER_MAX * 2 && total_scanned > sc->nr_reclaimed + sc->nr_reclaimed / 2) { sc->may_writepage = 1;
+                     mem_cgroup_kswapd_may_writepage(mem_cont, 1); }
      }

mm/vmscan.c +3 -1
@@ -2504,6 +2526,8 @@ loop_again:
       sc.nr_reclaimed = 0;
       total_scanned = 0;
 
+     mem_cgroup_pg_outrun(mem_cont, 1);
+
       for (priority = DEF_PRIORITY; priority >= 0; priority--) {
               sc.priority = priority;
 
@@ -2544,6 +2568,7 @@ loop_again:
                               wmark_ok = 0;
 
                       if (wmark_ok) {
+                             mem_cgroup_balance_wmark_ok(sc.mem_cgroup, 1);
                               goto out;
                       }
               }
--

1.7.3.1

Quoted 9 lines
Quoted 7 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#46 lore raw
In reply to Ying Han
Quoted 8 lines

The current implementation of memcg only supports direct reclaim and this patchset adds the support for background reclaim. Per cgroup background reclaim is needed which spreads out the memory pressure over longer period of time and smoothes out the system performance.

The current implementation is not a stable version, and it crashes sometimes on my NUMA machine. Before going further for debugging, I would like to start the discussion and hear the feedbacks of the initial design.

I haven't read your code at all. However I agree your claim that memcg also need background reclaim.

So if you post high level design memo, I'm happy.

Quoted 20 lines

Current status:
I run through some simple tests which reads/writes a large file and makes sure it triggers per cgroup kswapd on the low_wmark. Also, I compared at pg_steal/pg_scan ratio w/o background reclaim.

Step1: Create a cgroup with 500M memory_limit and set the min_free_kbytes to 1024.

$ mount -t cgroup -o cpuset,memory cpuset /dev/cgroup
$ mkdir /dev/cgroup/A
$ echo 0 >/dev/cgroup/A/cpuset.cpus
$ echo 0 >/dev/cgroup/A/cpuset.mems
$ echo 500m >/dev/cgroup/A/memory.limit_in_bytes
$ echo 1024 >/dev/cgroup/A/memory.min_free_kbytes
$ echo $$ >/dev/cgroup/A/tasks

Step2: Check the wmarks.

$ cat /dev/cgroup/A/memory.reclaim_wmarks
memcg_low_wmark 98304000
memcg_high_wmark 81920000
Quoted 6 lines

Step3: Dirty the pages by creating a 20g file on hard drive.

$ ddtest -D /export/hdc3/dd -b 1024 -n 20971520 -t 1

Checked the memory.stat w/o background reclaim. It used to be all the pages are reclaimed from direct reclaim, and now about half of them are reclaimed at background. (note: writing '0' to min_free_kbytes disables per cgroup kswapd)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#47 lore raw
Quoted 11 lines · KOSAKI Motohiro
Quoted 8 lines

The current implementation of memcg only supports direct reclaim and this patchset adds the support for background reclaim. Per cgroup background reclaim is needed which spreads out the memory pressure over longer period of time and smoothes out the system performance.

The current implementation is not a stable version, and it crashes sometimes on my NUMA machine. Before going further for debugging, I would like to start the discussion and hear the feedbacks of the initial design.

I haven't read your code at all. However I agree your claim that memcg also need background reclaim.

Thanks for your comment.

Quoted 2 lines

So if you post high level design memo, I'm happy.

My high level design is kind of spreading out into each patch, and here is the consolidated one. This is nothing more but cluing all the commits' messages for the following patches.

"
The current implementation of memcg only supports direct reclaim and this patchset adds the support for background reclaim. Per cgroup background reclaim is needed which spreads out the memory pressure over longer period of time and smoothes out the system performance.

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup. The kswapd is sleeping in the wait queue headed at kswapd_wait field of a kswapd descriptor. The kswapd descriptor stores information of node or cgroup and it allows the global and per cgroup background reclaim to share common reclaim algorithms. The per cgroup kswapd is invoked at mem_cgroup_charge when the cgroup's memory usage above a threshold--low_wmark. Then the kswapd thread starts to reclaim pages in a priority loop similar to global algorithm. The kswapd is done if the usage below a threshold--high_wmark.

The per cgroup background reclaim is based on the per cgroup LRU and also adds per cgroup watermarks. There are two watermarks including "low_wmark" and "high_wmark", and they are calculated based on the limit_in_bytes(hard_limit) for each cgroup. Each time the hard_limit is change, the corresponding wmarks are re-calculated. Since memory controller charges only user pages, there is no need for a "min_wmark". The current calculation of wmarks is a function of "memory.min_free_kbytes" which could be adjusted by writing different values into the new api. This is added mainly for debugging purpose.

The kswapd() function now is shared between global and per cgroup kswapd thread. It is passed in with the kswapd descriptor which contains the information of either node or cgroup. Then the new function balance_mem_cgroup_pgdat is invoked if it is per cgroup kswapd thread. The balance_mem_cgroup_pgdat performs a priority loop similar to global reclaim. In each iteration it invokes balance_pgdat_node for all nodes on the system, which is a new function performs background reclaim per node. After reclaiming each node, it checks mem_cgroup_watermark_ok() and breaks the priority loop if returns true. A per memcg zone will be marked as "unreclaimable" if the scanning rate is much greater than the reclaiming rate on the per cgroup LRU. The bit is cleared when there is a page charged to the cgroup being freed. Kswapd breaks the priority loop if all the zones are marked as "unreclaimable".
"

Also, I am happy to add more descriptions if anything not clear :)

thanks

--Ying

Quoted 21 lines
Quoted 21 lines

Current status:
I run through some simple tests which reads/writes a large file and makes sure it triggers per cgroup kswapd on the low_wmark. Also, I compared at pg_steal/pg_scan ratio w/o background reclaim.

Step1: Create a cgroup with 500M memory_limit and set the min_free_kbytes to 1024.

$ mount -t cgroup -o cpuset,memory cpuset /dev/cgroup
$ mkdir /dev/cgroup/A
$ echo 0 >/dev/cgroup/A/cpuset.cpus
$ echo 0 >/dev/cgroup/A/cpuset.mems
$ echo 500m >/dev/cgroup/A/memory.limit_in_bytes
$ echo 1024 >/dev/cgroup/A/memory.min_free_kbytes
$ echo $$ >/dev/cgroup/A/tasks

Step2: Check the wmarks.

$ cat /dev/cgroup/A/memory.reclaim_wmarks
memcg_low_wmark 98304000
memcg_high_wmark 81920000
Quoted 10 lines
Quoted 6 lines

Step3: Dirty the pages by creating a 20g file on hard drive.

$ ddtest -D /export/hdc3/dd -b 1024 -n 20971520 -t 1

Checked the memory.stat w/o background reclaim. It used to be all the pages are reclaimed from direct reclaim, and now about half of them are reclaimed at background. (note: writing '0' to min_free_kbytes disables per cgroup kswapd)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#48 lore raw
Quoted 28 lines · Ying Han
Quoted 11 lines · KOSAKI Motohiro
Quoted 8 lines

The current implementation of memcg only supports direct reclaim and this patchset adds the support for background reclaim. Per cgroup background reclaim is needed which spreads out the memory pressure over longer period of time and smoothes out the system performance.

The current implementation is not a stable version, and it crashes sometimes on my NUMA machine. Before going further for debugging, I would like to start the discussion and hear the feedbacks of the initial design.

I haven't read your code at all. However I agree your claim that memcg also need background reclaim.

Thanks for your comment.

Quoted 2 lines

So if you post high level design memo, I'm happy.

My high level design is kind of spreading out into each patch, and here is the consolidated one. This is nothing more but cluing all the commits' messages for the following patches.

"
The current implementation of memcg only supports direct reclaim and this patchset adds the support for background reclaim. Per cgroup background reclaim is needed which spreads out the memory pressure over longer period of time and smoothes out the system performance.

Quoted 9 lines

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup. The kswapd is sleeping in the wait queue headed at kswapd_wait field of a kswapd descriptor. The kswapd descriptor stores information of node or cgroup and it allows the global and per cgroup background reclaim to share common reclaim algorithms. The per cgroup kswapd is invoked at mem_cgroup_charge when the cgroup's memory usage above a threshold--low_wmark. Then the kswapd thread starts to reclaim pages in a priority loop similar to global algorithm. The kswapd is done if the usage below a threshold--high_wmark.

So the logic is per-node/per-zone/per-cgroup right?

Quoted 5 lines

The per cgroup background reclaim is based on the per cgroup LRU and also adds per cgroup watermarks. There are two watermarks including "low_wmark" and "high_wmark", and they are calculated based on the limit_in_bytes(hard_limit) for each cgroup. Each time the hard_limit is change, the corresponding wmarks are re-calculated. Since memory controller charges only user pages, there is

What about memsw limits, do they impact anything, I presume not.

Quoted 3 lines

no need for a "min_wmark". The current calculation of wmarks is a function of "memory.min_free_kbytes" which could be adjusted by writing different values into the new api. This is added mainly for debugging purpose.

When you say debugging, can you elaborate?

Quoted 15 lines

The kswapd() function now is shared between global and per cgroup kswapd thread. It is passed in with the kswapd descriptor which contains the information of either node or cgroup. Then the new function balance_mem_cgroup_pgdat is invoked if it is per cgroup kswapd thread. The balance_mem_cgroup_pgdat performs a priority loop similar to global reclaim. In each iteration it invokes balance_pgdat_node for all nodes on the system, which is a new function performs background reclaim per node. After reclaiming each node, it checks mem_cgroup_watermark_ok() and breaks the priority loop if returns true. A per memcg zone will be marked as "unreclaimable" if the scanning rate is much greater than the reclaiming rate on the per cgroup LRU. The bit is cleared when there is a page charged to the cgroup being freed. Kswapd breaks the priority loop if all the zones are marked as "unreclaimable".
"

Quoted 2 lines

Also, I am happy to add more descriptions if anything not clear :)

Thanks for explaining this in detail, it makes the review easier.

Signature · 9 lines
	Three Cheers,
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#49 lore raw
Quoted 40 lines · Balbir Singh
Quoted 28 lines · Ying Han
Quoted 11 lines · KOSAKI Motohiro
Quoted 8 lines

The current implementation of memcg only supports direct reclaim and this patchset adds the support for background reclaim. Per cgroup background reclaim is needed which spreads out the memory pressure over longer period of time and smoothes out the system performance.

The current implementation is not a stable version, and it crashes sometimes on my NUMA machine. Before going further for debugging, I would like to start the discussion and hear the feedbacks of the initial design.

I haven't read your code at all. However I agree your claim that memcg also need background reclaim.

Thanks for your comment.

Quoted 2 lines

So if you post high level design memo, I'm happy.

My high level design is kind of spreading out into each patch, and here is the consolidated one. This is nothing more but cluing all the commits' messages for the following patches.

"
The current implementation of memcg only supports direct reclaim and this patchset adds the support for background reclaim. Per cgroup background reclaim is needed which spreads out the memory pressure over longer period of time and smoothes out the system performance.

Quoted 9 lines

There is a kswapd kernel thread for each memory node. We add a different kswapd for each cgroup. The kswapd is sleeping in the wait queue headed at kswapd_wait field of a kswapd descriptor. The kswapd descriptor stores information of node or cgroup and it allows the global and per cgroup background reclaim to share common reclaim algorithms. The per cgroup kswapd is invoked at mem_cgroup_charge when the cgroup's memory usage above a threshold--low_wmark. Then the kswapd thread starts to reclaim pages in a priority loop similar to global algorithm. The kswapd is done if the usage below a threshold--high_wmark.

Quoted 1 line

So the logic is per-node/per-zone/per-cgroup right?

Thanks Balbir for your comments:

The kswapd thread is per-cgroup, and the scanning is on per-node and per-zone. The watermarks is calculated based on the per-cgroup limit_in_bytes, and kswapd is done whenever the usage_in_bytes is under the watermarks.

Quoted 7 lines
Quoted 6 lines

The per cgroup background reclaim is based on the per cgroup LRU and also adds per cgroup watermarks. There are two watermarks including "low_wmark" and "high_wmark", and they are calculated based on the limit_in_bytes(hard_limit) for each cgroup. Each time the hard_limit is change, the corresponding wmarks are re-calculated. Since memory controller charges only user pages, there is

Quoted 7 lines

What about memsw limits, do they impact anything, I presume not.

Quoted 3 lines

no need for a "min_wmark". The current calculation of wmarks is a function of "memory.min_free_kbytes" which could be adjusted by writing different values into the new api. This is added mainly for debugging purpose.

When you say debugging, can you elaborate?

I am not sure if we would like to keep the memory.min_free_kbytes for the final version, which is used
to adjust the calculation of per-cgroup wmarks. For now, I am adding it for performance testing purpose.

Quoted 16 lines
Quoted 16 lines

The kswapd() function now is shared between global and per cgroup kswapd thread. It is passed in with the kswapd descriptor which contains the information of either node or cgroup. Then the new function balance_mem_cgroup_pgdat is invoked if it is per cgroup kswapd thread. The balance_mem_cgroup_pgdat performs a priority loop similar to global reclaim. In each iteration it invokes balance_pgdat_node for all nodes on the system, which is a new function performs background reclaim per node. After reclaiming each node, it checks mem_cgroup_watermark_ok() and breaks the priority loop if returns true. A per memcg zone will be marked as "unreclaimable" if the scanning rate is much greater than the reclaiming rate on the per cgroup LRU. The bit is cleared when there is a page charged to the cgroup being freed. Kswapd breaks the priority loop if all the zones are marked as "unreclaimable".
"

Quoted 1 line
Quoted 1 line

Also, I am happy to add more descriptions if anything not clear :)

Sure. :)

--Ying

Quoted 8 lines
Quoted 1 line

Thanks for explaining this in detail, it makes the review easier.

--
       Three Cheers,
       Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#50 lore raw
In reply to Ying Han
Quoted 9 lines · On Mon, 29 Nov 2010 22:49:41 -0800 Ying Han <[email protected]> wrote:

The current implementation of memcg only supports direct reclaim and this patchset adds the support for background reclaim. Per cgroup background reclaim is needed which spreads out the memory pressure over longer period of time and smoothes out the system performance.

The current implementation is not a stable version, and it crashes sometimes on my NUMA machine. Before going further for debugging, I would like to start the discussion and hear the feedbacks of the initial design.

It's welcome but please wait until merge of dirty-ratio. And please post after you don't see crash ....

Description of design is appreciated.
Where the cost for "kswapd" is charged agaist if cpu cgroup is used at the same time ?

Quoted 6 lines

Current status:
I run through some simple tests which reads/writes a large file and makes sure it triggers per cgroup kswapd on the low_wmark. Also, I compared at pg_steal/pg_scan ratio w/o background reclaim.

Step1: Create a cgroup with 500M memory_limit and set the min_free_kbytes to 1024.

Quoted 39 lines
$ mount -t cgroup -o cpuset,memory cpuset /dev/cgroup
$ mkdir /dev/cgroup/A
$ echo 0 >/dev/cgroup/A/cpuset.cpus
$ echo 0 >/dev/cgroup/A/cpuset.mems
$ echo 500m >/dev/cgroup/A/memory.limit_in_bytes
$ echo 1024 >/dev/cgroup/A/memory.min_free_kbytes
$ echo $$ >/dev/cgroup/A/tasks

Step2: Check the wmarks.

$ cat /dev/cgroup/A/memory.reclaim_wmarks
memcg_low_wmark 98304000
memcg_high_wmark 81920000

Step3: Dirty the pages by creating a 20g file on hard drive.

$ ddtest -D /export/hdc3/dd -b 1024 -n 20971520 -t 1

Checked the memory.stat w/o background reclaim. It used to be all the pages are reclaimed from direct reclaim, and now about half of them are reclaimed at background. (note: writing '0' to min_free_kbytes disables per cgroup kswapd)

Only direct reclaim With background reclaim:
kswapd_steal 0 kswapd_steal 2751822
pg_pgsteal 5100401 pg_pgsteal 2476676
kswapd_pgscan 0 kswapd_pgscan 6019373
pg_scan 5542464 pg_scan 3851281
pgrefill 304505 pgrefill 348077
pgoutrun 0 pgoutrun 44568
allocstall 159278 allocstall 75669

Step4: Cleanup

$ echo $$ >/dev/cgroup/tasks
$ echo 0 > /dev/cgroup/A/memory.force_empty

Step5: Read the 20g file into the pagecache.

$ cat /export/hdc3/dd/tf0 > /dev/zero;

Checked the memory.stat w/o background reclaim. All the clean pages are reclaimed at background instead of direct reclaim.

Quoted 9 lines

Only direct reclaim With background reclaim
kswapd_steal 0 kswapd_steal 3512424
pg_pgsteal 3461280 pg_pgsteal 0
kswapd_pgscan 0 kswapd_pgscan 3512440
pg_scan 3461280 pg_scan 0
pgrefill 0 pgrefill 0
pgoutrun 0 pgoutrun 74973
allocstall 108165 allocstall 0

What is the trigger for starting background reclaim ?

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#51 lore raw
Quoted 8 lines · KAMEZAWA Hiroyuki
Quoted 5 lines · On Mon, 29 Nov 2010 22:49:41 -0800 Ying Han <[email protected]> wrote:

The current implementation of memcg only supports direct reclaim and this patchset adds the support for background reclaim. Per cgroup background reclaim is needed which spreads out the memory pressure over longer period of time and smoothes out the system performance.

Quoted 7 lines
Quoted 4 lines

The current implementation is not a stable version, and it crashes sometimes on my NUMA machine. Before going further for debugging, I would like to start the discussion and hear the feedbacks of the initial design.

It's welcome but please wait until merge of dirty-ratio. And please post after you don't see crash ....

Yeah, I will look into the crash and fix it. Besides, it runs fine so far on my single node
system.

Quoted 3 lines

Description of design is appreciated.
Where the cost for "kswapd" is charged agaist if cpu cgroup is used at the same time ?

There is no special treatment for that in the current implementation. Ideally it would be nice to charge the kswapd time to the corresponding cgroup. As a starting point, all the kswapd threads cputime could be charged to root.

Quoted 57 lines
Quoted 6 lines

Current status:
I run through some simple tests which reads/writes a large file and makes sure it triggers per cgroup kswapd on the low_wmark. Also, I compared at pg_steal/pg_scan ratio w/o background reclaim.

 Step1: Create a cgroup with 500M memory_limit and set the min_free_kbytes to 1024.

Quoted 39 lines
$ mount -t cgroup -o cpuset,memory cpuset /dev/cgroup
$ mkdir /dev/cgroup/A
$ echo 0 >/dev/cgroup/A/cpuset.cpus
$ echo 0 >/dev/cgroup/A/cpuset.mems
$ echo 500m >/dev/cgroup/A/memory.limit_in_bytes
$ echo 1024 >/dev/cgroup/A/memory.min_free_kbytes
$ echo $$ >/dev/cgroup/A/tasks

Step2: Check the wmarks.

$ cat /dev/cgroup/A/memory.reclaim_wmarks
memcg_low_wmark 98304000
memcg_high_wmark 81920000

Step3: Dirty the pages by creating a 20g file on hard drive.

$ ddtest -D /export/hdc3/dd -b 1024 -n 20971520 -t 1

Checked the memory.stat w/o background reclaim. It used to be all the pages are reclaimed from direct reclaim, and now about half of them are reclaimed at background. (note: writing '0' to min_free_kbytes disables per cgroup kswapd)

Only direct reclaim                                                With background reclaim:
kswapd_steal 0                                                     kswapd_steal 2751822 pg_pgsteal 5100401                                               pg_pgsteal 2476676 kswapd_pgscan 0                                                  kswapd_pgscan 6019373 pg_scan 5542464                                                   pg_scan 3851281 pgrefill 304505                                                       pgrefill 348077 pgoutrun 0                                                             pgoutrun 44568 allocstall 159278                                                    allocstall 75669

Step4: Cleanup

$ echo $$ >/dev/cgroup/tasks
$ echo 0 > /dev/cgroup/A/memory.force_empty

Step5: Read the 20g file into the pagecache.

$ cat /export/hdc3/dd/tf0 > /dev/zero;

Checked the memory.stat w/o background reclaim. All the clean pages are reclaimed at background instead of direct reclaim.

Quoted 9 lines

Only direct reclaim                                                With background reclaim kswapd_steal 0                                                      kswapd_steal 3512424 pg_pgsteal 3461280                                               pg_pgsteal 0 kswapd_pgscan 0                                                  kswapd_pgscan 3512440 pg_scan 3461280                                                   pg_scan 0 pgrefill 0                                                                pgrefill 0 pgoutrun 0                                                             pgoutrun 74973 allocstall 108165                                                    allocstall 0

Quoted 1 line

What is the trigger for starting background reclaim ?

The background reclaim is triggered when the usage_in_bytes above the watermark in mem_cgroup_do_charge.

--Ying

Quoted 5 lines

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>