introduce get_mm_hiwater_xxx(), fix taskstats->hiwater_xxx accounting

(changes: update the changelog/comments)

xacct_add_tsk() relies on do_exit()->update_hiwater_xxx() and uses mm->hiwater_xxx directly, this leads to 2 problems:

- taskstats_user_cmd() can call fill_pid()->xacct_add_tsk() at any moment before the task exits, so we should check the current values of rss/vm anyway.

- do_exit()->update_hiwater_xxx() calls are racy. An exiting thread can be preempted right before mm->hiwater_xxx = new_val, and another thread can use A_LOT of memory and exit in between. When the first thread resumes it can be the last thread in the thread group, in that case we report the wrong hiwater_xxx values which do not take A_LOT into account.

Introduce get_mm_hiwater_rss() and get_mm_hiwater_vm() helpers and change xacct_add_tsk() to use them. The first helper will also be used by rusage->ru_maxrss accounting.

Kill do_exit()->update_hiwater_xxx() calls. Unless we are going to decrease rss/vm there is no point to update mm->hiwater_xxx, and nobody can look at this mm_struct when exit_mmap() actually unmaps the memory.

Signed-off-by Oleg Nesterov <[email protected]>

include/linux/sched.h renamed from include/linux/sched.h~HIWATER +3 -0

@@ -388,6 +388,9 @@ extern void arch_unmap_area_topdown(stru
 		(mm)->hiwater_vm = (mm)->total_vm;	\
 } while (0)
 
+#define get_mm_hiwater_rss(mm)	max((mm)->hiwater_rss, get_mm_rss(mm))
+#define get_mm_hiwater_vm(mm)	max((mm)->hiwater_vm, (mm)->total_vm)
+
 extern void set_dumpable(struct mm_struct *mm, int value);
 extern int get_dumpable(struct mm_struct *mm);
 

kernel/tsacct.c renamed from kernel/tsacct.c~HIWATER +2 -2

@@ -90,8 +90,8 @@ void xacct_add_tsk(struct taskstats *sta
 	mm = get_task_mm(p);
 	if (mm) {
 		/* adjust to KB unit */
-		stats->hiwater_rss   = mm->hiwater_rss * PAGE_SIZE / KB;
-		stats->hiwater_vm    = mm->hiwater_vm * PAGE_SIZE / KB;
+		stats->hiwater_rss   = get_mm_hiwater_rss(mm) * PAGE_SIZE / KB;
+		stats->hiwater_vm    = get_mm_hiwater_vm(mm)  * PAGE_SIZE / KB;
 		mmput(mm);
 	}
 	stats->read_char	= p->ioac.rchar;

kernel/exit.c renamed from kernel/exit.c~HIWATER +1 -4

@@ -1048,10 +1048,7 @@ NORET_TYPE void do_exit(long code)
 				preempt_count());
 
 	acct_update_integrals(tsk);
-	if (tsk->mm) {
-		update_hiwater_rss(tsk->mm);
-		update_hiwater_vm(tsk->mm);
-	}
+
 	group_dead = atomic_dec_and_test(&tsk->signal->live);
 	if (group_dead) {
 		hrtimer_cancel(&tsk->signal->real_timer);

mm/mmap.c renamed from mm/mmap.c~HIWATER +1 -1

@@ -2103,7 +2103,7 @@ void exit_mmap(struct mm_struct *mm)
 	lru_add_drain();
 	flush_cache_mm(mm);
 	tlb = tlb_gather_mmu(mm, 1);
-	/* Don't update_hiwater_rss(mm) here, do_exit already did */
+	/* update_hiwater_rss(mm) here? but nobody should be looking */
 	/* Use -1 here to ensure all VMAs in the mm are unmapped */
 	end = unmap_vmas(&tlb, vma, 0, -1, &nr_accounted, NULL);
 	vm_unacct_memory(nr_accounted);

(changes: update the changelog/comments)

xacct_add_tsk() relies on do_exit()->update_hiwater_xxx() and uses mm->hiwater_xxx directly, this leads to 2 problems:

- taskstats_user_cmd() can call fill_pid()->xacct_add_tsk() at any moment before the task exits, so we should check the current values of rss/vm anyway.

- do_exit()->update_hiwater_xxx() calls are racy. An exiting thread can be preempted right before mm->hiwater_xxx = new_val, and another thread can use A_LOT of memory and exit in between. When the first thread resumes it can be the last thread in the thread group, in that case we report the wrong hiwater_xxx values which do not take A_LOT into account.

Introduce get_mm_hiwater_rss() and get_mm_hiwater_vm() helpers and change xacct_add_tsk() to use them. The first helper will also be used by rusage->ru_maxrss accounting.

Kill do_exit()->update_hiwater_xxx() calls. Unless we are going to decrease rss/vm there is no point to update mm->hiwater_xxx, and nobody can look at this mm_struct when exit_mmap() actually unmaps the memory.

Signed-off-by Oleg Nesterov <[email protected]>

Quoted 25 lines

(changes: update the changelog/comments)

xacct_add_tsk() relies on do_exit()->update_hiwater_xxx() and uses mm->hiwater_xxx directly, this leads to 2 problems:

- taskstats_user_cmd() can call fill_pid()->xacct_add_tsk() at any moment before the task exits, so we should check the current values of rss/vm anyway.

- do_exit()->update_hiwater_xxx() calls are racy. An exiting thread can be preempted right before mm->hiwater_xxx = new_val, and another thread can use A_LOT of memory and exit in between. When the first thread resumes it can be the last thread in the thread group, in that case we report the wrong hiwater_xxx values which do not take A_LOT into account.

Introduce get_mm_hiwater_rss() and get_mm_hiwater_vm() helpers and change xacct_add_tsk() to use them. The first helper will also be used by rusage->ru_maxrss accounting.

Kill do_exit()->update_hiwater_xxx() calls. Unless we are going to decrease rss/vm there is no point to update mm->hiwater_xxx, and nobody can look at this mm_struct when exit_mmap() actually unmaps the memory.

Quoted 1 line

Signed-off-by Oleg Nesterov <[email protected]>

Thanks! looks good to me.

Reviewed-by KOSAKI Motohiro <[email protected]>

+#define get_mm_hiwater_rss(mm) max((mm)->hiwater_rss, get_mm_rss(mm))

+#define get_mm_hiwater_vm(mm) max((mm)->hiwater_vm, (mm)->total_vm)

Quoted 1 line · On Fri, 12 Dec 2008 15:05:24 +0100 Oleg Nesterov <[email protected]> wrote:

+#define get_mm_hiwater_rss(mm) max((mm)->hiwater_rss, get_mm_rss(mm))

This evaluates its argument thrice.

Quoted 1 line

+#define get_mm_hiwater_vm(mm) max((mm)->hiwater_vm, (mm)->total_vm)

This evaluates its argument twice.

was sched.h the appropriate header in which to implement these? Maybe...

But they're only ever _used_ in kernel/tsacct.c, so do they actually need to be implemented in any .h file?

Quoted 2 lines

+#define get_mm_hiwater_rss(mm) max((mm)->hiwater_rss, get_mm_rss(mm))

This evaluates its argument thrice.

Quoted 1 line

+#define get_mm_hiwater_vm(mm) max((mm)->hiwater_vm, (mm)->total_vm)

This evaluates its argument twice.

was sched.h the appropriate header in which to implement these? Maybe...

But they're only ever _used_ in kernel/tsacct.c, so do they actually need to be implemented in any .h file?