Skip to contentSkip to conversation

patch mm-fix-committed_as-underflow-on-large-nr_cpus-environment.patch added to 2.6.29-stable tree

DISCUSSION Greg KH 4 participants 10 messages May 5, 2009 – May 6, 2009 2 messages not in the archive

Subsystems (4)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

Greg KH

author
#2 lore raw
Quoted 53 lines · Stefan Lippers-Hollmann

Hi

On Dienstag, 5. Mai 2009, [email protected] wrote:
[...]

Quoted 30 lines

From 00a62ce91e554198ef28234c91c36f850f5a3bc9 Mon Sep 17 00:00:00 2001
From: KOSAKI Motohiro <[email protected]>
Date: Thu, 30 Apr 2009 15:08:51 -0700
Subject: mm: fix Committed_AS underflow on large NR_CPUS environment

From: KOSAKI Motohiro <[email protected]>

commit 00a62ce91e554198ef28234c91c36f850f5a3bc9 upstream.

The Committed_AS field can underflow in certain situations:

Quoted 7 lines
        # while true; do cat /proc/meminfo  | grep _AS; sleep 1; done | uniq -c
              1 Committed_AS: 18446744073709323392 kB
             11 Committed_AS: 18446744073709455488 kB
              6 Committed_AS:    35136 kB
              5 Committed_AS: 18446744073709454400 kB
              7 Committed_AS:    35904 kB
              3 Committed_AS: 18446744073709453248 kB
Quoted 8 lines
              2 Committed_AS:    34752 kB
              9 Committed_AS: 18446744073709453248 kB
              8 Committed_AS:    34752 kB
              3 Committed_AS: 18446744073709320960 kB
              7 Committed_AS: 18446744073709454080 kB
              3 Committed_AS: 18446744073709320960 kB
              5 Committed_AS: 18446744073709454080 kB
              6 Committed_AS: 18446744073709320960 kB

Because NR_CPUS can be greater than 1000 and meminfo_proc_show() does not check for underflow.

Quoted 7 lines

But NR_CPUS proportional isn't good calculation. In general, possibility of lock contention is proportional to the number of online cpus, not theorical maximum cpus (NR_CPUS).

The current kernel has generic percpu-counter stuff. using it is right way. it makes code simplify and percpu_counter_read_positive() don't make underflow issue.

This patch reproducably panics on all systems I could test so far on 2.6.29.2 + the current stable queue:
- Acer Aspire One, Intel Atom N270, Intel 945GC chipset (i386)
- AMD64 3200+, nVidia nforce3 chipset (amd64)
- AMD64 X2 4200+, nVidia nforce4 chipset (amd64)

Kernel panic (sorry, this is from a bad webcam and only the last part of 

the messages scrolling by - I can try to get a serial console tomorrow),
taken from an AMD64 3200+, nVidia nforce3 chipset (amd64).
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/panic.gif [198 KB]

Quoted 5 lines

Reverting just this patch fixes the problem for me.

Kernel configurations:
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/config-2.6.29-2.slh.5.3-sidux-686 [96 KB]
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/config-2.6.29-2.slh.5.6-sidux-amd64 [91 KB]

Thanks for the report, I'm also getting some oopses on boot and am bisecting down the tree right now. The error looks a bit like yours, but not identical enough for me to declare that it is the same thing just yet.

Give me a few more hours...

thanks,

greg k-h

author
#3 lore raw
In reply to a message not in the archive
Quoted 53 lines · Stefan Lippers-Hollmann

Hi

On Dienstag, 5. Mai 2009, [email protected] wrote:
[...]

Quoted 30 lines

From 00a62ce91e554198ef28234c91c36f850f5a3bc9 Mon Sep 17 00:00:00 2001
From: KOSAKI Motohiro <[email protected]>
Date: Thu, 30 Apr 2009 15:08:51 -0700
Subject: mm: fix Committed_AS underflow on large NR_CPUS environment

From: KOSAKI Motohiro <[email protected]>

commit 00a62ce91e554198ef28234c91c36f850f5a3bc9 upstream.

The Committed_AS field can underflow in certain situations:

Quoted 7 lines
        # while true; do cat /proc/meminfo  | grep _AS; sleep 1; done | uniq -c
              1 Committed_AS: 18446744073709323392 kB
             11 Committed_AS: 18446744073709455488 kB
              6 Committed_AS:    35136 kB
              5 Committed_AS: 18446744073709454400 kB
              7 Committed_AS:    35904 kB
              3 Committed_AS: 18446744073709453248 kB
Quoted 8 lines
              2 Committed_AS:    34752 kB
              9 Committed_AS: 18446744073709453248 kB
              8 Committed_AS:    34752 kB
              3 Committed_AS: 18446744073709320960 kB
              7 Committed_AS: 18446744073709454080 kB
              3 Committed_AS: 18446744073709320960 kB
              5 Committed_AS: 18446744073709454080 kB
              6 Committed_AS: 18446744073709320960 kB

Because NR_CPUS can be greater than 1000 and meminfo_proc_show() does not check for underflow.

Quoted 7 lines

But NR_CPUS proportional isn't good calculation. In general, possibility of lock contention is proportional to the number of online cpus, not theorical maximum cpus (NR_CPUS).

The current kernel has generic percpu-counter stuff. using it is right way. it makes code simplify and percpu_counter_read_positive() don't make underflow issue.

This patch reproducably panics on all systems I could test so far on 2.6.29.2 + the current stable queue:
- Acer Aspire One, Intel Atom N270, Intel 945GC chipset (i386)
- AMD64 3200+, nVidia nforce3 chipset (amd64)
- AMD64 X2 4200+, nVidia nforce4 chipset (amd64)

Kernel panic (sorry, this is from a bad webcam and only the last part of 

the messages scrolling by - I can try to get a serial console tomorrow),
taken from an AMD64 3200+, nVidia nforce3 chipset (amd64).
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/panic.gif [198 KB]

Quoted 5 lines

Reverting just this patch fixes the problem for me.

Kernel configurations:
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/config-2.6.29-2.slh.5.3-sidux-686 [96 KB]
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/config-2.6.29-2.slh.5.6-sidux-amd64 [91 KB]

Yes, I just confirmed this on my machines as well, so I'm going to drop it from the stable queue.

If anyone thinks it is really necessary for the 2.6.29-stable tree, please resubmit it after fixing it and testing that it works :)

thanks,

greg k-h

#4 lore raw

2009/5/6 Greg KH <[email protected]>:

Quoted 60 lines · Stefan Lippers-Hollmann
Quoted 53 lines

Hi

On Dienstag, 5. Mai 2009, [email protected] wrote:
[...]

Quoted 30 lines

From 00a62ce91e554198ef28234c91c36f850f5a3bc9 Mon Sep 17 00:00:00 2001
From: KOSAKI Motohiro <[email protected]>
Date: Thu, 30 Apr 2009 15:08:51 -0700
Subject: mm: fix Committed_AS underflow on large NR_CPUS environment

From: KOSAKI Motohiro <[email protected]>

commit 00a62ce91e554198ef28234c91c36f850f5a3bc9 upstream.

The Committed_AS field can underflow in certain situations:

Quoted 7 lines
        # while true; do cat /proc/meminfo  | grep _AS; sleep 1; done | uniq -c
              1 Committed_AS: 18446744073709323392 kB
             11 Committed_AS: 18446744073709455488 kB
              6 Committed_AS:    35136 kB
              5 Committed_AS: 18446744073709454400 kB
              7 Committed_AS:    35904 kB
              3 Committed_AS: 18446744073709453248 kB
Quoted 8 lines
              2 Committed_AS:    34752 kB
              9 Committed_AS: 18446744073709453248 kB
              8 Committed_AS:    34752 kB
              3 Committed_AS: 18446744073709320960 kB
              7 Committed_AS: 18446744073709454080 kB
              3 Committed_AS: 18446744073709320960 kB
              5 Committed_AS: 18446744073709454080 kB
              6 Committed_AS: 18446744073709320960 kB

Because NR_CPUS can be greater than 1000 and meminfo_proc_show() does not check for underflow.

Quoted 7 lines

But NR_CPUS proportional isn't good calculation.  In general, possibility of lock contention is proportional to the number of online cpus, not theorical maximum cpus (NR_CPUS).

The current kernel has generic percpu-counter stuff.  using it is right way.  it makes code simplify and percpu_counter_read_positive() don't make underflow issue.

This patch reproducably panics on all systems I could test so far on 2.6.29.2 + the current stable queue:
- Acer Aspire One, Intel Atom N270, Intel 945GC chipset (i386)
- AMD64 3200+, nVidia nforce3 chipset (amd64)
- AMD64 X2 4200+, nVidia nforce4 chipset (amd64)

Kernel panic (sorry, this is from a bad webcam and only the last part of

the messages scrolling by - I can try to get a serial console tomorrow),
taken from an AMD64 3200+, nVidia nforce3 chipset (amd64).
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/panic.gif [198 KB]

Quoted 5 lines

Reverting just this patch fixes the problem for me.

Kernel configurations:
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/config-2.6.29-2.slh.5.3-sidux-686   [96 KB]
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/config-2.6.29-2.slh.5.6-sidux-amd64 [91 KB]

Quoted 7 lines

Yes, I just confirmed this on my machines as well, so I'm going to drop it from the stable queue.

If anyone thinks it is really necessary for the 2.6.29-stable tree, please resubmit it after fixing it and testing that it works :)

thanks,

Could you please post oops log?
current mmotm (include this patch) works fine on my x86_64 box. Then, I have no hint now ;-)

author
#5 lore raw
Quoted 64 lines · KOSAKI Motohiro

2009/5/6 Greg KH <[email protected]>:

Quoted 63 lines · Stefan Lippers-Hollmann
Quoted 53 lines

Hi

On Dienstag, 5. Mai 2009, [email protected] wrote:
[...]

Quoted 30 lines

From 00a62ce91e554198ef28234c91c36f850f5a3bc9 Mon Sep 17 00:00:00 2001
From: KOSAKI Motohiro <[email protected]>
Date: Thu, 30 Apr 2009 15:08:51 -0700
Subject: mm: fix Committed_AS underflow on large NR_CPUS environment

From: KOSAKI Motohiro <[email protected]>

commit 00a62ce91e554198ef28234c91c36f850f5a3bc9 upstream.

The Committed_AS field can underflow in certain situations:

Quoted 7 lines
        # while true; do cat /proc/meminfo  | grep _AS; sleep 1; done | uniq -c
              1 Committed_AS: 18446744073709323392 kB
             11 Committed_AS: 18446744073709455488 kB
              6 Committed_AS:    35136 kB
              5 Committed_AS: 18446744073709454400 kB
              7 Committed_AS:    35904 kB
              3 Committed_AS: 18446744073709453248 kB
Quoted 8 lines
              2 Committed_AS:    34752 kB
              9 Committed_AS: 18446744073709453248 kB
              8 Committed_AS:    34752 kB
              3 Committed_AS: 18446744073709320960 kB
              7 Committed_AS: 18446744073709454080 kB
              3 Committed_AS: 18446744073709320960 kB
              5 Committed_AS: 18446744073709454080 kB
              6 Committed_AS: 18446744073709320960 kB

Because NR_CPUS can be greater than 1000 and meminfo_proc_show() does not check for underflow.

Quoted 7 lines

But NR_CPUS proportional isn't good calculation.  In general, possibility of lock contention is proportional to the number of online cpus, not theorical maximum cpus (NR_CPUS).

The current kernel has generic percpu-counter stuff.  using it is right way.  it makes code simplify and percpu_counter_read_positive() don't make underflow issue.

This patch reproducably panics on all systems I could test so far on 2.6.29.2 + the current stable queue:
- Acer Aspire One, Intel Atom N270, Intel 945GC chipset (i386)
- AMD64 3200+, nVidia nforce3 chipset (amd64)
- AMD64 X2 4200+, nVidia nforce4 chipset (amd64)

Kernel panic (sorry, this is from a bad webcam and only the last part of

the messages scrolling by - I can try to get a serial console tomorrow),
taken from an AMD64 3200+, nVidia nforce3 chipset (amd64).
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/panic.gif [198 KB]

Quoted 5 lines

Reverting just this patch fixes the problem for me.

Kernel configurations:
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/config-2.6.29-2.slh.5.3-sidux-686   [96 KB]
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/config-2.6.29-2.slh.5.6-sidux-amd64 [91 KB]

Yes, I just confirmed this on my machines as well, so I'm going to drop it from the stable queue.

Quoted 8 lines
Quoted 4 lines

If anyone thinks it is really necessary for the 2.6.29-stable tree, please resubmit it after fixing it and testing that it works :)

thanks,

Could you please post oops log?
current mmotm (include this patch) works fine on my x86_64 box. Then, I have no hint now ;-)

Ok, I'll apply the patch and reboot and take a picture of it...

#6 lore raw
Quoted 5 lines
Quoted 3 lines

Could you please post oops log?
current mmotm (include this patch) works fine on my x86_64 box. Then, I have no hint now ;-)

Ok, I'll apply the patch and reboot and take a picture of it...

umm.

Greg, your stable queue has following function? 2.6.29.2 + this patch don't make correct mmap_init() on my box. (instead, changing unrelated function)

Please check.

--------------------------------------------------------------------------------------
/*
* initialise the VMA slab
*/
void __init mmap_init(void)

{
        int ret;

        ret = percpu_counter_init(&vm_committed_as, 0);
        VM_BUG_ON(ret);
        vm_area_cachep = kmem_cache_create("vm_area_struct",
                        sizeof(struct vm_area_struct), 0,
                        SLAB_PANIC, NULL);
}

--------------------------------------------------------------------------------------

author
#7 lore raw
In reply to KOSAKI Motohiro
Quoted 3 lines · KOSAKI Motohiro

Could you please post oops log?
current mmotm (include this patch) works fine on my x86_64 box. Then, I have no hint now ;-)

Attached is a picture of the oops.

Hope this helps,

greg k-h

#8 lore raw
Quoted 8 lines · KOSAKI Motohiro
Quoted 3 lines

Could you please post oops log?
current mmotm (include this patch) works fine on my x86_64 box. Then, I have no hint now ;-)

Attached is a picture of the oops.

Hope this helps,

OK, the reason is 99% mis-applied patch. I've made rebased patch.

in my environment

2.6.29.2 + original patch: panic
2.6.29.2 + rebased patch: works well

Could you please try it?

commit 12adffadf709dd675f736d7d848e24c3db03a1e4

Author: KOSAKI Motohiro <[email protected]>
Date:   Tue Apr 28 22:48:11 2009 +0200

The Committed_AS field can underflow in certain situations:

> # while true; do cat /proc/meminfo | grep _AS; sleep 1; done | uniq -c
> 1 Committed_AS: 18446744073709323392 kB
> 11 Committed_AS: 18446744073709455488 kB
> 6 Committed_AS: 35136 kB
> 5 Committed_AS: 18446744073709454400 kB
> 7 Committed_AS: 35904 kB
> 3 Committed_AS: 18446744073709453248 kB
> 2 Committed_AS: 34752 kB
> 9 Committed_AS: 18446744073709453248 kB
> 8 Committed_AS: 34752 kB
> 3 Committed_AS: 18446744073709320960 kB
> 7 Committed_AS: 18446744073709454080 kB
> 3 Committed_AS: 18446744073709320960 kB
> 5 Committed_AS: 18446744073709454080 kB
> 6 Committed_AS: 18446744073709320960 kB

Because NR_CPUS can be greater than 1000 and meminfo_proc_show() does not check for underflow.

But NR_CPUS proportional isn't good calculation. In general, possibility of lock contention is proportional to the number of online cpus, not theorical maximum cpus (NR_CPUS).

The current kernel has generic percpu-counter stuff. using it is right way. it makes code simplify and percpu_counter_read_positive() don't make underflow issue.

Cc Eric B Munson <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc <[email protected]> [All kernel versions]
---
 fs/proc/meminfo.c    |    2 +-
 include/linux/mman.h |    9 +++------
 mm/mmap.c            |   12 ++++++------
 mm/nommu.c           |   13 +++++++------
 mm/swap.c            |   46 ----------------------------------------------
 5 files changed, 17 insertions(+), 65 deletions(-)
fs/proc/meminfo.c +1 -1
Index: b/fs/proc/meminfo.c
===================================================================
@@ -35,7 +35,7 @@ static int meminfo_proc_show(struct seq_
 #define K(x) ((x) << (PAGE_SHIFT - 10))
 	si_meminfo(&i);
 	si_swapinfo(&i);
-	committed = atomic_long_read(&vm_committed_space);
+	committed = percpu_counter_read_positive(&vm_committed_as);
 	allowed = ((totalram_pages - hugetlb_total_pages())
 		* sysctl_overcommit_ratio / 100) + total_swap_pages;
 
include/linux/mman.h +3 -6
Index: b/include/linux/mman.h
===================================================================
@@ -12,21 +12,18 @@
 
 #ifdef __KERNEL__
 #include <linux/mm.h>
+#include <linux/percpu_counter.h>
 
 #include <asm/atomic.h>
 
 extern int sysctl_overcommit_memory;
 extern int sysctl_overcommit_ratio;
-extern atomic_long_t vm_committed_space;
+extern struct percpu_counter vm_committed_as;
 
-#ifdef CONFIG_SMP
-extern void vm_acct_memory(long pages);
-#else
 static inline void vm_acct_memory(long pages)
 {
-	atomic_long_add(pages, &vm_committed_space);
+	percpu_counter_add(&vm_committed_as, pages);
 }
-#endif
 
 static inline void vm_unacct_memory(long pages)
 {
mm/mmap.c +6 -6
Index: b/mm/mmap.c
===================================================================
@@ -84,7 +84,7 @@ EXPORT_SYMBOL(vm_get_page_prot);
 int sysctl_overcommit_memory = OVERCOMMIT_GUESS;  /* heuristic overcommit */
 int sysctl_overcommit_ratio = 50;	/* default is 50% */
 int sysctl_max_map_count __read_mostly = DEFAULT_MAX_MAP_COUNT;
-atomic_long_t vm_committed_space = ATOMIC_LONG_INIT(0);
+struct percpu_counter vm_committed_as;
 
 /*
  * Check that a process has enough memory to allocate a new virtual
@@ -178,11 +178,7 @@ int __vm_enough_memory(struct mm_struct 
 	if (mm)
 		allowed -= mm->total_vm / 32;
 
-	/*
-	 * cast `allowed' as a signed long because vm_committed_space
-	 * sometimes has a negative value
-	 */
-	if (atomic_long_read(&vm_committed_space) < (long)allowed)
+	if (percpu_counter_read_positive(&vm_committed_as) < allowed)
 		return 0;
 error:
 	vm_unacct_memory(pages);
@@ -2477,6 +2473,10 @@ void mm_drop_all_locks(struct mm_struct 
  */
 void __init mmap_init(void)
 {
+	int ret;
+
+	ret = percpu_counter_init(&vm_committed_as, 0);
+	VM_BUG_ON(ret);
 	vm_area_cachep = kmem_cache_create("vm_area_struct",
 			sizeof(struct vm_area_struct), 0,
 			SLAB_PANIC, NULL);
mm/nommu.c +7 -6
Index: b/mm/nommu.c
===================================================================
@@ -62,7 +62,7 @@ void *high_memory;
 struct page *mem_map;
 unsigned long max_mapnr;
 unsigned long num_physpages;
-atomic_long_t vm_committed_space = ATOMIC_LONG_INIT(0);
+struct percpu_counter vm_committed_as;
 int sysctl_overcommit_memory = OVERCOMMIT_GUESS; /* heuristic overcommit */
 int sysctl_overcommit_ratio = 50; /* default is 50% */
 int sysctl_max_map_count = DEFAULT_MAX_MAP_COUNT;
@@ -463,6 +463,10 @@ SYSCALL_DEFINE1(brk, unsigned long, brk)
  */
 void __init mmap_init(void)
 {
+	int ret;
+
+	ret = percpu_counter_init(&vm_committed_as, 0);
+	VM_BUG_ON(ret);
 	vm_region_jar = kmem_cache_create("vm_region_jar",
 					  sizeof(struct vm_region), 0,
 					  SLAB_PANIC, NULL);
@@ -1849,12 +1853,9 @@ int __vm_enough_memory(struct mm_struct 
 	if (mm)
 		allowed -= mm->total_vm / 32;
 
-	/*
-	 * cast `allowed' as a signed long because vm_committed_space
-	 * sometimes has a negative value
-	 */
-	if (atomic_long_read(&vm_committed_space) < (long)allowed)
+	if (percpu_counter_read_positive(&vm_committed_as) < allowed)
 		return 0;
+
 error:
 	vm_unacct_memory(pages);
 
mm/swap.c +0 -46

56 lines not on this page · raw message

author
#9 lore raw
Quoted 9 lines · KOSAKI Motohiro
Quoted 8 lines · KOSAKI Motohiro
Quoted 3 lines

Could you please post oops log?
current mmotm (include this patch) works fine on my x86_64 box. Then, I have no hint now ;-)

Attached is a picture of the oops.

Hope this helps,

Quoted 8 lines

OK, the reason is 99% mis-applied patch. I've made rebased patch.

in my environment

2.6.29.2 + original patch: panic
2.6.29.2 + rebased patch: works well

Could you please try it?

thank you for looking into this, I'll try it out tomorrow morning and let you know how it goes.

greg k-h

#10 lore raw
Quoted 14 lines · Greg KH
Quoted 13 lines · KOSAKI Motohiro
Quoted 8 lines · KOSAKI Motohiro
Quoted 3 lines

Could you please post oops log?
current mmotm (include this patch) works fine on my x86_64 box. Then, I have no hint now ;-)

Attached is a picture of the oops.

Hope this helps,

OK, the reason is 99% mis-applied patch. I've made rebased patch.

in my environment

Quoted 7 lines
Quoted 4 lines

2.6.29.2 + original patch: panic
2.6.29.2 + rebased patch: works well

Could you please try it?

thank you for looking into this, I'll try it out tomorrow morning and let you know how it goes.

Yes, this works!

Thanks so much for working so quickly to resolve this. I've added the updated patch to the .29-stable queue.

greg k-h

#11 lore raw
In reply to KOSAKI Motohiro

Hi

Quoted 9 lines · KOSAKI Motohiro
Quoted 8 lines · KOSAKI Motohiro
Quoted 3 lines

Could you please post oops log?
current mmotm (include this patch) works fine on my x86_64 box. Then, I have no hint now ;-)

Attached is a picture of the oops.

Hope this helps,

Quoted 8 lines

OK, the reason is 99% mis-applied patch. I've made rebased patch.

in my environment

2.6.29.2 + original patch: panic
2.6.29.2 + rebased patch: works well

Could you please try it?

This patch seems to fix the problems, it's working without any issues on all systems I could test so far.

Thank you
Stefan Lippers-Hollmann