patch mm-fix-committed_as-underflow-on-large-nr_cpus-environment.patch added to 2.6.29-stable tree
Subsystems (4)
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 53 lines · Stefan Lippers-Hollmann
Hi
On Dienstag, 5. Mai 2009, [email protected] wrote:
[...]Quoted 30 lines
From
00a62ce91e554198ef28234c91c36f850f5a3bc9Mon Sep 17 00:00:00 2001
From: KOSAKI Motohiro <[email protected]>
Date: Thu, 30 Apr 2009 15:08:51 -0700
Subject: mm: fixCommitted_ASunderflow on largeNR_CPUSenvironmentFrom: KOSAKI Motohiro <[email protected]>
commit
00a62ce91e554198ef28234c91c36f850f5a3bc9upstream.The
Committed_ASfield can underflow in certain situations:Quoted 7 lines
# while true; do cat /proc/meminfo | grep _AS; sleep 1; done | uniq -c 1 Committed_AS: 18446744073709323392 kB 11 Committed_AS: 18446744073709455488 kB 6 Committed_AS: 35136 kB 5 Committed_AS: 18446744073709454400 kB 7 Committed_AS: 35904 kB 3 Committed_AS: 18446744073709453248 kBQuoted 8 lines
2 Committed_AS: 34752 kB 9 Committed_AS: 18446744073709453248 kB 8 Committed_AS: 34752 kB 3 Committed_AS: 18446744073709320960 kB 7 Committed_AS: 18446744073709454080 kB 3 Committed_AS: 18446744073709320960 kB 5 Committed_AS: 18446744073709454080 kB 6 Committed_AS: 18446744073709320960 kBBecause
NR_CPUScan be greater than 1000 andmeminfo_proc_show()does not check for underflow.Quoted 7 lines
But
NR_CPUSproportional isn't good calculation. In general, possibility of lock contention is proportional to the number of online cpus, not theorical maximum cpus (NR_CPUS).The current kernel has generic percpu-counter stuff. using it is right way. it makes code simplify and
percpu_counter_read_positive()don't make underflow issue.This patch reproducably panics on all systems I could test so far on 2.6.29.2 + the current stable queue:
- Acer Aspire One, Intel Atom N270, Intel 945GC chipset (i386)
- AMD64 3200+, nVidia nforce3 chipset (amd64)
- AMD64 X2 4200+, nVidia nforce4 chipset (amd64)Kernel panic (sorry, this is from a bad webcam and only the last part ofthe messages scrolling by - I can try to get a serial console tomorrow),
taken from an AMD64 3200+, nVidia nforce3 chipset (amd64).
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/panic.gif [198 KB]
Quoted 5 lines
Reverting just this patch fixes the problem for me.
Kernel configurations:
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/config-2.6.29-2.slh.5.3-sidux-686 [96 KB]
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/config-2.6.29-2.slh.5.6-sidux-amd64 [91 KB]
Thanks for the report, I'm also getting some oopses on boot and am bisecting down the tree right now. The error looks a bit like yours, but not identical enough for me to declare that it is the same thing just yet.
Give me a few more hours...
thanks,
greg k-h
Quoted 53 lines · Stefan Lippers-Hollmann
Hi
On Dienstag, 5. Mai 2009, [email protected] wrote:
[...]Quoted 30 lines
From
00a62ce91e554198ef28234c91c36f850f5a3bc9Mon Sep 17 00:00:00 2001
From: KOSAKI Motohiro <[email protected]>
Date: Thu, 30 Apr 2009 15:08:51 -0700
Subject: mm: fixCommitted_ASunderflow on largeNR_CPUSenvironmentFrom: KOSAKI Motohiro <[email protected]>
commit
00a62ce91e554198ef28234c91c36f850f5a3bc9upstream.The
Committed_ASfield can underflow in certain situations:Quoted 7 lines
# while true; do cat /proc/meminfo | grep _AS; sleep 1; done | uniq -c 1 Committed_AS: 18446744073709323392 kB 11 Committed_AS: 18446744073709455488 kB 6 Committed_AS: 35136 kB 5 Committed_AS: 18446744073709454400 kB 7 Committed_AS: 35904 kB 3 Committed_AS: 18446744073709453248 kBQuoted 8 lines
2 Committed_AS: 34752 kB 9 Committed_AS: 18446744073709453248 kB 8 Committed_AS: 34752 kB 3 Committed_AS: 18446744073709320960 kB 7 Committed_AS: 18446744073709454080 kB 3 Committed_AS: 18446744073709320960 kB 5 Committed_AS: 18446744073709454080 kB 6 Committed_AS: 18446744073709320960 kBBecause
NR_CPUScan be greater than 1000 andmeminfo_proc_show()does not check for underflow.Quoted 7 lines
But
NR_CPUSproportional isn't good calculation. In general, possibility of lock contention is proportional to the number of online cpus, not theorical maximum cpus (NR_CPUS).The current kernel has generic percpu-counter stuff. using it is right way. it makes code simplify and
percpu_counter_read_positive()don't make underflow issue.This patch reproducably panics on all systems I could test so far on 2.6.29.2 + the current stable queue:
- Acer Aspire One, Intel Atom N270, Intel 945GC chipset (i386)
- AMD64 3200+, nVidia nforce3 chipset (amd64)
- AMD64 X2 4200+, nVidia nforce4 chipset (amd64)Kernel panic (sorry, this is from a bad webcam and only the last part ofthe messages scrolling by - I can try to get a serial console tomorrow),
taken from an AMD64 3200+, nVidia nforce3 chipset (amd64).
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/panic.gif [198 KB]
Quoted 5 lines
Reverting just this patch fixes the problem for me.
Kernel configurations:
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/config-2.6.29-2.slh.5.3-sidux-686 [96 KB]
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/config-2.6.29-2.slh.5.6-sidux-amd64 [91 KB]
Yes, I just confirmed this on my machines as well, so I'm going to drop it from the stable queue.
If anyone thinks it is really necessary for the 2.6.29-stable tree, please resubmit it after fixing it and testing that it works :)
thanks,
greg k-h
2009/5/6 Greg KH <[email protected]>:
Quoted 60 lines · Stefan Lippers-Hollmann
Quoted 53 lines
Hi
On Dienstag, 5. Mai 2009, [email protected] wrote:
[...]Quoted 30 lines
From
00a62ce91e554198ef28234c91c36f850f5a3bc9Mon Sep 17 00:00:00 2001
From: KOSAKI Motohiro <[email protected]>
Date: Thu, 30 Apr 2009 15:08:51 -0700
Subject: mm: fixCommitted_ASunderflow on largeNR_CPUSenvironmentFrom: KOSAKI Motohiro <[email protected]>
commit
00a62ce91e554198ef28234c91c36f850f5a3bc9upstream.The
Committed_ASfield can underflow in certain situations:Quoted 7 lines
# while true; do cat /proc/meminfo | grep _AS; sleep 1; done | uniq -c 1 Committed_AS: 18446744073709323392 kB 11 Committed_AS: 18446744073709455488 kB 6 Committed_AS: 35136 kB 5 Committed_AS: 18446744073709454400 kB 7 Committed_AS: 35904 kB 3 Committed_AS: 18446744073709453248 kBQuoted 8 lines
2 Committed_AS: 34752 kB 9 Committed_AS: 18446744073709453248 kB 8 Committed_AS: 34752 kB 3 Committed_AS: 18446744073709320960 kB 7 Committed_AS: 18446744073709454080 kB 3 Committed_AS: 18446744073709320960 kB 5 Committed_AS: 18446744073709454080 kB 6 Committed_AS: 18446744073709320960 kBBecause
NR_CPUScan be greater than 1000 andmeminfo_proc_show()does not check for underflow.Quoted 7 lines
But
NR_CPUSproportional isn't good calculation. In general, possibility of lock contention is proportional to the number of online cpus, not theorical maximum cpus (NR_CPUS).The current kernel has generic percpu-counter stuff. using it is right way. it makes code simplify and
percpu_counter_read_positive()don't make underflow issue.This patch reproducably panics on all systems I could test so far on 2.6.29.2 + the current stable queue:
- Acer Aspire One, Intel Atom N270, Intel 945GC chipset (i386)
- AMD64 3200+, nVidia nforce3 chipset (amd64)
- AMD64 X2 4200+, nVidia nforce4 chipset (amd64)Kernel panic (sorry, this is from a bad webcam and only the last part ofthe messages scrolling by - I can try to get a serial console tomorrow),
taken from an AMD64 3200+, nVidia nforce3 chipset (amd64).
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/panic.gif [198 KB]Quoted 5 lines
Reverting just this patch fixes the problem for me.
Kernel configurations:
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/config-2.6.29-2.slh.5.3-sidux-686 [96 KB]
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/config-2.6.29-2.slh.5.6-sidux-amd64 [91 KB]
Quoted 7 lines
Yes, I just confirmed this on my machines as well, so I'm going to drop it from the stable queue.
If anyone thinks it is really necessary for the 2.6.29-stable tree, please resubmit it after fixing it and testing that it works :)
thanks,
Could you please post oops log?
current mmotm (include this patch) works fine on my x86_64 box. Then, I have no hint now ;-)
Quoted 64 lines · KOSAKI Motohiro
2009/5/6 Greg KH <[email protected]>:
Quoted 63 lines · Stefan Lippers-Hollmann
Quoted 53 lines
Hi
On Dienstag, 5. Mai 2009, [email protected] wrote:
[...]Quoted 30 lines
From
00a62ce91e554198ef28234c91c36f850f5a3bc9Mon Sep 17 00:00:00 2001
From: KOSAKI Motohiro <[email protected]>
Date: Thu, 30 Apr 2009 15:08:51 -0700
Subject: mm: fixCommitted_ASunderflow on largeNR_CPUSenvironmentFrom: KOSAKI Motohiro <[email protected]>
commit
00a62ce91e554198ef28234c91c36f850f5a3bc9upstream.The
Committed_ASfield can underflow in certain situations:Quoted 7 lines
# while true; do cat /proc/meminfo | grep _AS; sleep 1; done | uniq -c 1 Committed_AS: 18446744073709323392 kB 11 Committed_AS: 18446744073709455488 kB 6 Committed_AS: 35136 kB 5 Committed_AS: 18446744073709454400 kB 7 Committed_AS: 35904 kB 3 Committed_AS: 18446744073709453248 kBQuoted 8 lines
2 Committed_AS: 34752 kB 9 Committed_AS: 18446744073709453248 kB 8 Committed_AS: 34752 kB 3 Committed_AS: 18446744073709320960 kB 7 Committed_AS: 18446744073709454080 kB 3 Committed_AS: 18446744073709320960 kB 5 Committed_AS: 18446744073709454080 kB 6 Committed_AS: 18446744073709320960 kBBecause
NR_CPUScan be greater than 1000 andmeminfo_proc_show()does not check for underflow.Quoted 7 lines
But
NR_CPUSproportional isn't good calculation. In general, possibility of lock contention is proportional to the number of online cpus, not theorical maximum cpus (NR_CPUS).The current kernel has generic percpu-counter stuff. using it is right way. it makes code simplify and
percpu_counter_read_positive()don't make underflow issue.This patch reproducably panics on all systems I could test so far on 2.6.29.2 + the current stable queue:
- Acer Aspire One, Intel Atom N270, Intel 945GC chipset (i386)
- AMD64 3200+, nVidia nforce3 chipset (amd64)
- AMD64 X2 4200+, nVidia nforce4 chipset (amd64)Kernel panic (sorry, this is from a bad webcam and only the last part ofthe messages scrolling by - I can try to get a serial console tomorrow),
taken from an AMD64 3200+, nVidia nforce3 chipset (amd64).
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/panic.gif [198 KB]Quoted 5 lines
Reverting just this patch fixes the problem for me.
Kernel configurations:
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/config-2.6.29-2.slh.5.3-sidux-686 [96 KB]
http://sidux.com/slh/mm-fix-committed_as-underflow-on-large-nr_cpus-environment/config-2.6.29-2.slh.5.6-sidux-amd64 [91 KB]Yes, I just confirmed this on my machines as well, so I'm going to drop it from the stable queue.
Quoted 8 lines
Quoted 4 lines
If anyone thinks it is really necessary for the 2.6.29-stable tree, please resubmit it after fixing it and testing that it works :)
thanks,
Could you please post oops log?
current mmotm (include this patch) works fine on myx86_64box. Then, I have no hint now ;-)
Ok, I'll apply the patch and reboot and take a picture of it...
Quoted 5 lines
Quoted 3 lines
Could you please post oops log?
current mmotm (include this patch) works fine on myx86_64box. Then, I have no hint now ;-)Ok, I'll apply the patch and reboot and take a picture of it...
umm.
Greg, your stable queue has following function? 2.6.29.2 + this patch don't make correct mmap_init() on my box. (instead, changing unrelated function)
Please check.
--------------------------------------------------------------------------------------
/*
* initialise the VMA slab
*/
void __init mmap_init(void)
{
int ret;
ret = percpu_counter_init(&vm_committed_as, 0);
VM_BUG_ON(ret);
vm_area_cachep = kmem_cache_create("vm_area_struct",
sizeof(struct vm_area_struct), 0,
SLAB_PANIC, NULL);
}--------------------------------------------------------------------------------------
Quoted 3 lines · KOSAKI Motohiro
Could you please post oops log?
current mmotm (include this patch) works fine on myx86_64box. Then, I have no hint now ;-)
Attached is a picture of the oops.
Hope this helps,
greg k-h
Quoted 8 lines · KOSAKI Motohiro
Quoted 3 lines
Could you please post oops log?
current mmotm (include this patch) works fine on myx86_64box. Then, I have no hint now ;-)Attached is a picture of the oops.
Hope this helps,
OK, the reason is 99% mis-applied patch. I've made rebased patch.
in my environment
2.6.29.2 + original patch: panic
2.6.29.2 + rebased patch: works well
Could you please try it?
commit 12adffadf709dd675f736d7d848e24c3db03a1e4
Author: KOSAKI Motohiro <[email protected]> Date: Tue Apr 28 22:48:11 2009 +0200
The Committed_AS field can underflow in certain situations:
> # while true; do cat /proc/meminfo | grep _AS; sleep 1; done | uniq -c
> 1 Committed_AS: 18446744073709323392 kB
> 11 Committed_AS: 18446744073709455488 kB
> 6 Committed_AS: 35136 kB
> 5 Committed_AS: 18446744073709454400 kB
> 7 Committed_AS: 35904 kB
> 3 Committed_AS: 18446744073709453248 kB
> 2 Committed_AS: 34752 kB
> 9 Committed_AS: 18446744073709453248 kB
> 8 Committed_AS: 34752 kB
> 3 Committed_AS: 18446744073709320960 kB
> 7 Committed_AS: 18446744073709454080 kB
> 3 Committed_AS: 18446744073709320960 kB
> 5 Committed_AS: 18446744073709454080 kB
> 6 Committed_AS: 18446744073709320960 kB
Because NR_CPUS can be greater than 1000 and meminfo_proc_show() does not check for underflow.
But NR_CPUS proportional isn't good calculation. In general, possibility of lock contention is proportional to the number of online cpus, not theorical maximum cpus (NR_CPUS).
The current kernel has generic percpu-counter stuff. using it is right way. it makes code simplify and percpu_counter_read_positive() don't make underflow issue.
--- fs/proc/meminfo.c | 2 +- include/linux/mman.h | 9 +++------ mm/mmap.c | 12 ++++++------ mm/nommu.c | 13 +++++++------ mm/swap.c | 46 ---------------------------------------------- 5 files changed, 17 insertions(+), 65 deletions(-)
fs/proc/meminfo.c +1 -1
@@ -35,7 +35,7 @@ static int meminfo_proc_show(struct seq_#define K (x) ((x) << (PAGE_SHIFT - 10))si_meminfo (&i);si_swapinfo (&i); - committed =atomic_long_read (&vm_committed_space); + committed =percpu_counter_read_positive (&vm_committed_as); allowed = ((totalram_pages -hugetlb_total_pages ()) * sysctl_overcommit_ratio /100 ) + total_swap_pages;
include/linux/mman.h +3 -6
@@ -12,21 +12,18 @@#ifdef __KERNEL__#include <linux/mm.h> +#include <linux/percpu_counter.h> #include <asm/atomic.h> extern int sysctl_overcommit_memory;extern int sysctl_overcommit_ratio; -extern atomic_long_t vm_committed_space; +extern struct percpu_counter vm_committed_as; -#ifdef CONFIG_SMP -extern void vm_acct_memory (long pages); -#else static inline void vm_acct_memory (long pages) { -atomic_long_add (pages, &vm_committed_space); +percpu_counter_add (&vm_committed_as, pages); } -#endif static inline void vm_unacct_memory (long pages) {
mm/mmap.c +6 -6
@@ -84,7 +84,7 @@ EXPORT_SYMBOL(vm_get_page_prot);int sysctl_overcommit_memory =OVERCOMMIT_GUESS ;/* heuristic overcommit */ int sysctl_overcommit_ratio =50 ;/* default is 50% */ int sysctl_max_map_count__read_mostly =DEFAULT_MAX_MAP_COUNT ; -atomic_long_t vm_committed_space =ATOMIC_LONG_INIT (0 ); +struct percpu_counter vm_committed_as;/* * Check that a process has enough memory to allocate a new virtual
@@ -178,11 +178,7 @@ int __vm_enough_memory(struct mm_structif (mm) allowed -= mm->total_vm /32 ; -/* -* cast `allowed' as a signed long because vm_committed_space -* sometimes has a negative value -*/ -if (atomic_long_read (&vm_committed_space) < (long )allowed) +if (percpu_counter_read_positive (&vm_committed_as) < allowed)return 0 ; error:vm_unacct_memory (pages);
@@ -2477,6 +2473,10 @@ void mm_drop_all_locks(struct mm_struct*/ void __init mmap_init (void ) { +int ret; + + ret =percpu_counter_init (&vm_committed_as,0 ); +VM_BUG_ON (ret); vm_area_cachep =kmem_cache_create ("vm_area_struct" ,sizeof (struct vm_area_struct ),0 ,SLAB_PANIC ,NULL );
mm/nommu.c +7 -6
@@ -62,7 +62,7 @@ void *high_memory;struct page *mem_map;unsigned long max_mapnr;unsigned long num_physpages; -atomic_long_t vm_committed_space =ATOMIC_LONG_INIT (0 ); +struct percpu_counter vm_committed_as;int sysctl_overcommit_memory =OVERCOMMIT_GUESS ;/* heuristic overcommit */ int sysctl_overcommit_ratio =50 ;/* default is 50% */ int sysctl_max_map_count =DEFAULT_MAX_MAP_COUNT ;
@@ -463,6 +463,10 @@ SYSCALL_DEFINE1(brk, unsigned long, brk)*/ void __init mmap_init (void ) { +int ret; + + ret =percpu_counter_init (&vm_committed_as,0 ); +VM_BUG_ON (ret); vm_region_jar =kmem_cache_create ("vm_region_jar" ,sizeof (struct vm_region ),0 ,SLAB_PANIC ,NULL );
@@ -1849,12 +1853,9 @@ int __vm_enough_memory(struct mm_structif (mm) allowed -= mm->total_vm /32 ; -/* -* cast `allowed' as a signed long because vm_committed_space -* sometimes has a negative value -*/ -if (atomic_long_read (&vm_committed_space) < (long )allowed) +if (percpu_counter_read_positive (&vm_committed_as) < allowed)return 0 ; + error:vm_unacct_memory (pages);
mm/swap.c +0 -46
56 lines not on this page · raw message
Quoted 9 lines · KOSAKI Motohiro
Quoted 8 lines · KOSAKI Motohiro
Quoted 3 lines
Could you please post oops log?
current mmotm (include this patch) works fine on myx86_64box. Then, I have no hint now ;-)Attached is a picture of the oops.
Hope this helps,
Quoted 8 lines
OK, the reason is 99% mis-applied patch. I've made rebased patch.
in my environment
2.6.29.2 + original patch: panic
2.6.29.2 + rebased patch: works wellCould you please try it?
thank you for looking into this, I'll try it out tomorrow morning and let you know how it goes.
greg k-h
Quoted 14 lines · Greg KH
Quoted 13 lines · KOSAKI Motohiro
Quoted 8 lines · KOSAKI Motohiro
Quoted 3 lines
Could you please post oops log?
current mmotm (include this patch) works fine on myx86_64box. Then, I have no hint now ;-)Attached is a picture of the oops.
Hope this helps,
OK, the reason is 99% mis-applied patch. I've made rebased patch.
in my environment
Quoted 7 lines
Quoted 4 lines
2.6.29.2 + original patch: panic
2.6.29.2 + rebased patch: works wellCould you please try it?
thank you for looking into this, I'll try it out tomorrow morning and let you know how it goes.
Yes, this works!
Thanks so much for working so quickly to resolve this. I've added the updated patch to the .29-stable queue.
greg k-h
Hi
Quoted 9 lines · KOSAKI Motohiro
Quoted 8 lines · KOSAKI Motohiro
Quoted 3 lines
Could you please post oops log?
current mmotm (include this patch) works fine on myx86_64box. Then, I have no hint now ;-)Attached is a picture of the oops.
Hope this helps,
Quoted 8 lines
OK, the reason is 99% mis-applied patch. I've made rebased patch.
in my environment
2.6.29.2 + original patch: panic
2.6.29.2 + rebased patch: works wellCould you please try it?
This patch seems to fix the problems, it's working without any issues on all systems I could test so far.
Thank you
Stefan Lippers-Hollmann