Skip to contentSkip to conversation

mm, page_alloc: do not depend on cpu hotplug locks inside the allocator

PATCH Applied Michal Hocko 4 participants 6 messages Feb 7, 2017 – Feb 8, 2017

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm, page_alloc: do not depend on cpu hotplug locks inside the allocator

From: Michal Hocko <[email protected]>

Dmitry has reported the following lockdep splat

[<ffffffff81571db1>] lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3753
[<ffffffff8436697e>] __mutex_lock_common kernel/locking/mutex.c:521 [inline]
[<ffffffff8436697e>] mutex_lock_nested+0x24e/0xff0 kernel/locking/mutex.c:621
[<ffffffff818f07ea>] pcpu_alloc+0xbda/0x1280 mm/percpu.c:896
[<ffffffff818f0ee4>] __alloc_percpu+0x24/0x30 mm/percpu.c:1075
[<ffffffff816543e3>] smpcfd_prepare_cpu+0x73/0xd0 kernel/smp.c:44
[<ffffffff814240b4>] cpuhp_invoke_callback+0x254/0x1480 kernel/cpu.c:136
[<ffffffff81425821>] cpuhp_up_callbacks+0x81/0x2a0 kernel/cpu.c:493
[<ffffffff81427bf3>] _cpu_up+0x1e3/0x2a0 kernel/cpu.c:1057
[<ffffffff81427d23>] do_cpu_up+0x73/0xa0 kernel/cpu.c:1087
[<ffffffff81427d68>] cpu_up+0x18/0x20 kernel/cpu.c:1095
[<ffffffff854ede84>] smp_init+0xe9/0xee kernel/smp.c:564
[<ffffffff85482f81>] kernel_init_freeable+0x439/0x690 init/main.c:1010
[<ffffffff84357083>] kernel_init+0x13/0x180 init/main.c:941
[<ffffffff84377baa>] ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:433

cpu_hotplug_begin
cpu_hotplug.lock
pcpu_alloc
pcpu_alloc_mutex

[<ffffffff81423012>] get_online_cpus+0x62/0x90 kernel/cpu.c:248
[<ffffffff8185fcf8>] drain_all_pages+0xf8/0x710 mm/page_alloc.c:2385
[<ffffffff81865e5d>] __alloc_pages_direct_reclaim mm/page_alloc.c:3440 [inline]
[<ffffffff81865e5d>] __alloc_pages_slowpath+0x8fd/0x2370 mm/page_alloc.c:3778
[<ffffffff818681c5>] __alloc_pages_nodemask+0x8f5/0xc60 mm/page_alloc.c:3980
[<ffffffff818ed0c1>] __alloc_pages include/linux/gfp.h:426 [inline]
[<ffffffff818ed0c1>] __alloc_pages_node include/linux/gfp.h:439 [inline]
[<ffffffff818ed0c1>] alloc_pages_node include/linux/gfp.h:453 [inline]
[<ffffffff818ed0c1>] pcpu_alloc_pages mm/percpu-vm.c:93 [inline]
[<ffffffff818ed0c1>] pcpu_populate_chunk+0x1e1/0x900 mm/percpu-vm.c:282
[<ffffffff818f0a11>] pcpu_alloc+0xe01/0x1280 mm/percpu.c:998
[<ffffffff818f0eb7>] __alloc_percpu_gfp+0x27/0x30 mm/percpu.c:1062
[<ffffffff817d25b2>] bpf_array_alloc_percpu kernel/bpf/arraymap.c:34 [inline]
[<ffffffff817d25b2>] array_map_alloc+0x532/0x710 kernel/bpf/arraymap.c:99
[<ffffffff817ba034>] find_and_alloc_map kernel/bpf/syscall.c:34 [inline]
[<ffffffff817ba034>] map_create kernel/bpf/syscall.c:188 [inline]
[<ffffffff817ba034>] SYSC_bpf kernel/bpf/syscall.c:870 [inline]
[<ffffffff817ba034>] SyS_bpf+0xd64/0x2500 kernel/bpf/syscall.c:827
[<ffffffff84377941>] entry_SYSCALL_64_fastpath+0x1f/0xc2

pcpu_alloc
pcpu_alloc_mutex
drain_all_pages
get_online_cpus
cpu_hotplug.lock

[<ffffffff81427876>] cpu_hotplug_begin+0x206/0x2e0 kernel/cpu.c:304
[<ffffffff81427ada>] _cpu_up+0xca/0x2a0 kernel/cpu.c:1011
[<ffffffff81427d23>] do_cpu_up+0x73/0xa0 kernel/cpu.c:1087
[<ffffffff81427d68>] cpu_up+0x18/0x20 kernel/cpu.c:1095
[<ffffffff854ede84>] smp_init+0xe9/0xee kernel/smp.c:564
[<ffffffff85482f81>] kernel_init_freeable+0x439/0x690 init/main.c:1010
[<ffffffff84357083>] kernel_init+0x13/0x180 init/main.c:941
[<ffffffff84377baa>] ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:433

cpu_hotplug_begin
cpu_hotplug.lock

Pulling cpu hotplug locks inside the page allocator is just too dangerous. Let's remove the dependency by dropping get_online_cpus() from drain_all_pages. This is not so simple though because now we do not have a protection against cpu hotplug which means 2 things:
- the work item might be executed on a different cpu in worker from unbound pool so it doesn't run on pinned on the cpu
- we have to make sure that we do not race with page_alloc_cpu_dead calling drain_pages_zone

Disabling preemption in drain_local_pages_wq will solve the first problem drain_local_pages will determine its local CPU from the WQ context which will be stable after that point, page_alloc_cpu_dead is pinned to the CPU already. The later condition is achieved by disabling IRQs in drain_pages_zone.

Fixes mm, page_alloc: drain per-cpu pages from workqueue context
---
 mm/page_alloc.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)
mm/page_alloc.c +9 -7
index c3358d4f7932..b6411816787a 100644
@@ -2343,7 +2343,16 @@ void drain_local_pages(struct zone *zone)
 
 static void drain_local_pages_wq(struct work_struct *work)
 {
+	/*
+	 * drain_all_pages doesn't use proper cpu hotplug protection so
+	 * we can race with cpu offline when the WQ can move this from
+	 * a cpu pinned worker to an unbound one. We can operate on a different
+	 * cpu which is allright but we also have to make sure to not move to
+	 * a different one.
+	 */
+	preempt_disable();
 	drain_local_pages(NULL);
+	preempt_enable();
 }
 
 /*
@@ -2379,12 +2388,6 @@ void drain_all_pages(struct zone *zone)
 	}
 
 	/*
-	 * As this can be called from reclaim context, do not reenter reclaim.
-	 * An allocation failure can be handled, it's simply slower
-	 */
-	get_online_cpus();
-
-	/*
 	 * We don't care about racing with CPU hotplug event
 	 * as offline notification will cause the notified
 	 * cpu to drain that CPU pcps and on_each_cpu_mask
@@ -2423,7 +2426,6 @@ void drain_all_pages(struct zone *zone)
 	for_each_cpu(cpu, &cpus_with_pcps)
 		flush_work(per_cpu_ptr(&pcpu_drain, cpu));
 
-	put_online_cpus();
 	mutex_unlock(&pcpu_drain_mutex);
 }
 
Signature · 7 lines
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#1 lore raw

PATCH 1/1 mm-page_alloc-use-static-global-work_struct-for-draining-per-cpu-pages-fix

From: Michal Hocko <[email protected]>

there is no need to have both pcpu_drain and pcpu_drain_mutex visible outside of drain_all_pages. This might just attract abuse.

---
Hi Andrew, Mel,
I think this would be a good cleanup to be folded into mm-page_alloc-use-static-global-work_struct-for-draining-per-cpu-pages.patch.

 mm/page_alloc.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
mm/page_alloc.c +2 -4
index b6411816787a..6c48053bcd81 100644
@@ -92,10 +92,6 @@ EXPORT_PER_CPU_SYMBOL(_numa_mem_);
 int _node_numa_mem_[MAX_NUMNODES];
 #endif
 
-/* work_structs for global per-cpu drains */
-DEFINE_MUTEX(pcpu_drain_mutex);
-DEFINE_PER_CPU(struct work_struct, pcpu_drain);
-
 #ifdef CONFIG_GCC_PLUGIN_LATENT_ENTROPY
 volatile unsigned long latent_entropy __latent_entropy;
 EXPORT_SYMBOL(latent_entropy);
@@ -2364,6 +2360,8 @@ static void drain_local_pages_wq(struct work_struct *work)
  */
 void drain_all_pages(struct zone *zone)
 {
+	static DEFINE_PER_CPU(struct work_struct, pcpu_drain);
+	static DEFINE_MUTEX(pcpu_drain_mutex);
 	int cpu;
 
 	/*
Signature · 7 lines
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#2 lore raw

Re: [PATCH] mm-page_alloc-use-static-global-work_struct-for-draining-per-cpu-pages-fix

reproduce:

Hi Michal,

[auto build test ERROR on mmotm/master]
[also build test ERROR on v4.10-rc7 next-20170207]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Michal-Hocko/mm-page_alloc-use-static-global-work_struct-for-draining-per-cpu-pages-fix/20170208-050036
base:   git://git.cmpxchg.org/linux-mmotm.git master
config: i386-randconfig-x001-201706 (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

All error/warnings (new ones prefixed by >>):

In file included from include/asm-generic/percpu.h:6:0,

                    from arch/x86/include/asm/percpu.h:542,
                    from arch/x86/include/asm/preempt.h:5,
                    from include/linux/preempt.h:59,
                    from include/linux/spinlock.h:50,
                    from include/linux/mmzone.h:7,
                    from include/linux/gfp.h:5,
                    from include/linux/mm.h:9,
                    from mm/page_alloc.c:18:
   mm/page_alloc.c: In function 'drain_all_pages':
>> include/linux/percpu-defs.h:91:33: error: section attribute cannot be specified for local variables
     extern __PCPU_DUMMY_ATTRS char __pcpu_unique_##name;  \
                                    ^
>> include/linux/percpu-defs.h:116:2: note: in expansion of macro 'DEFINE_PER_CPU_SECTION'
     DEFINE_PER_CPU_SECTION(type, name, "")
     ^~~~~~~~~~~~~~~~~~~~~~
>> mm/page_alloc.c:2354:9: note: in expansion of macro 'DEFINE_PER_CPU'
     static DEFINE_PER_CPU(struct work_struct, pcpu_drain);
            ^~~~~~~~~~~~~~
   include/linux/percpu-defs.h:92:26: error: section attribute cannot be specified for local variables
     __PCPU_DUMMY_ATTRS char __pcpu_unique_##name;   \
                             ^
>> include/linux/percpu-defs.h:116:2: note: in expansion of macro 'DEFINE_PER_CPU_SECTION'
     DEFINE_PER_CPU_SECTION(type, name, "")
     ^~~~~~~~~~~~~~~~~~~~~~
>> mm/page_alloc.c:2354:9: note: in expansion of macro 'DEFINE_PER_CPU'
     static DEFINE_PER_CPU(struct work_struct, pcpu_drain);
            ^~~~~~~~~~~~~~
>> include/linux/percpu-defs.h:92:26: error: declaration of '__pcpu_unique_pcpu_drain' with no linkage follows extern declaration
     __PCPU_DUMMY_ATTRS char __pcpu_unique_##name;   \
                             ^
>> include/linux/percpu-defs.h:116:2: note: in expansion of macro 'DEFINE_PER_CPU_SECTION'
     DEFINE_PER_CPU_SECTION(type, name, "")
     ^~~~~~~~~~~~~~~~~~~~~~
>> mm/page_alloc.c:2354:9: note: in expansion of macro 'DEFINE_PER_CPU'
     static DEFINE_PER_CPU(struct work_struct, pcpu_drain);
            ^~~~~~~~~~~~~~
   include/linux/percpu-defs.h:91:33: note: previous declaration of '__pcpu_unique_pcpu_drain' was here
     extern __PCPU_DUMMY_ATTRS char __pcpu_unique_##name;  \
                                    ^
>> include/linux/percpu-defs.h:116:2: note: in expansion of macro 'DEFINE_PER_CPU_SECTION'
     DEFINE_PER_CPU_SECTION(type, name, "")
     ^~~~~~~~~~~~~~~~~~~~~~
>> mm/page_alloc.c:2354:9: note: in expansion of macro 'DEFINE_PER_CPU'
     static DEFINE_PER_CPU(struct work_struct, pcpu_drain);
            ^~~~~~~~~~~~~~
>> mm/page_alloc.c:2354:44: error: section attribute cannot be specified for local variables
     static DEFINE_PER_CPU(struct work_struct, pcpu_drain);
                                               ^
   include/linux/percpu-defs.h:93:44: note: in definition of macro 'DEFINE_PER_CPU_SECTION'
     extern __PCPU_ATTRS(sec) __typeof__(type) name;   \
                                               ^~~~
>> mm/page_alloc.c:2354:9: note: in expansion of macro 'DEFINE_PER_CPU'
     static DEFINE_PER_CPU(struct work_struct, pcpu_drain);
            ^~~~~~~~~~~~~~
>> mm/page_alloc.c:2354:44: error: section attribute cannot be specified for local variables
     static DEFINE_PER_CPU(struct work_struct, pcpu_drain);
                                               ^
   include/linux/percpu-defs.h:95:19: note: in definition of macro 'DEFINE_PER_CPU_SECTION'
     __typeof__(type) name
                      ^~~~
>> mm/page_alloc.c:2354:9: note: in expansion of macro 'DEFINE_PER_CPU'
     static DEFINE_PER_CPU(struct work_struct, pcpu_drain);
            ^~~~~~~~~~~~~~
>> mm/page_alloc.c:2354:44: error: weak declaration of 'pcpu_drain' must be public
     static DEFINE_PER_CPU(struct work_struct, pcpu_drain);
                                               ^
   include/linux/percpu-defs.h:95:19: note: in definition of macro 'DEFINE_PER_CPU_SECTION'
     __typeof__(type) name
                      ^~~~
>> mm/page_alloc.c:2354:9: note: in expansion of macro 'DEFINE_PER_CPU'
     static DEFINE_PER_CPU(struct work_struct, pcpu_drain);
            ^~~~~~~~~~~~~~
>> mm/page_alloc.c:2354:44: error: declaration of 'pcpu_drain' with no linkage follows extern declaration
     static DEFINE_PER_CPU(struct work_struct, pcpu_drain);
                                               ^
   include/linux/percpu-defs.h:95:19: note: in definition of macro 'DEFINE_PER_CPU_SECTION'
     __typeof__(type) name
                      ^~~~
>> mm/page_alloc.c:2354:9: note: in expansion of macro 'DEFINE_PER_CPU'
     static DEFINE_PER_CPU(struct work_struct, pcpu_drain);
            ^~~~~~~~~~~~~~
   mm/page_alloc.c:2354:44: note: previous declaration of 'pcpu_drain' was here
     static DEFINE_PER_CPU(struct work_struct, pcpu_drain);
                                               ^
   include/linux/percpu-defs.h:93:44: note: in definition of macro 'DEFINE_PER_CPU_SECTION'
     extern __PCPU_ATTRS(sec) __typeof__(type) name;   \
                                               ^~~~
>> mm/page_alloc.c:2354:9: note: in expansion of macro 'DEFINE_PER_CPU'
     static DEFINE_PER_CPU(struct work_struct, pcpu_drain);
            ^~~~~~~~~~~~~~

vim +2354 mm/page_alloc.c

2348 * When zone parameter is non-NULL, spill just the single zone's pages.
2349 *
2350 * Note that this can be extremely slow as the draining happens in a workqueue.
2351 */
2352 void drain_all_pages(struct zone *zone)
2353 {

> 2354		static DEFINE_PER_CPU(struct work_struct, pcpu_drain);

2355 static DEFINE_MUTEX(pcpu_drain_mutex);
2356 int cpu;
2357

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

maintainer
#3 lore raw

Re: [PATCH] mm-page_alloc-use-static-global-work_struct-for-draining-per-cpu-pages-fix

Quoted 26 lines · On Wed, 8 Feb 2017 05:54:56 +0800 kbuild test robot <[email protected]> wrote:

Hi Michal,

[auto build test ERROR on mmotm/master]
[also build test ERROR on v4.10-rc7 next-20170207]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Michal-Hocko/mm-page_alloc-use-static-global-work_struct-for-draining-per-cpu-pages-fix/20170208-050036
base:   git://git.cmpxchg.org/linux-mmotm.git master
config: i386-randconfig-x001-201706 (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

All error/warnings (new ones prefixed by >>):

   In file included from include/asm-generic/percpu.h:6:0,
                    from arch/x86/include/asm/percpu.h:542,
                    from arch/x86/include/asm/preempt.h:5,
                    from include/linux/preempt.h:59,
                    from include/linux/spinlock.h:50,
                    from include/linux/mmzone.h:7,
                    from include/linux/gfp.h:5,
                    from include/linux/mm.h:9,
                    from mm/page_alloc.c:18:
   mm/page_alloc.c: In function 'drain_all_pages':
Quoted 3 lines
Quoted 1 line
Quoted 1 line

include/linux/percpu-defs.h:91:33: error: section attribute cannot be specified for local variables

     extern __PCPU_DUMMY_ATTRS char __pcpu_unique_##name;  \
                                    ^

huh, yes. The DEFINE_PER_CPU() macro is broken.

If you do

foo()

{
	static DEFINE_PER_CPU(int, bar);
}

then it won't compile, as described here. It should.

And if you do

static DEFINE_PER_CPU(int, bar);

then you still get global symbols (__pcpu_unique_bar).

The kernel does the above thing in, umm, 466 places and afaict they're all broken. If two code sites ever use the same identifier, they'll get linkage errors.

huh. Seems hard to fix.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw

Re: [PATCH] mm-page_alloc-use-static-global-work_struct-for-draining-per-cpu-pages-fix

Quoted 47 lines · Andrew Morton
Quoted 26 lines · On Wed, 8 Feb 2017 05:54:56 +0800 kbuild test robot <[email protected]> wrote:

Hi Michal,

[auto build test ERROR on mmotm/master]
[also build test ERROR on v4.10-rc7 next-20170207]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Michal-Hocko/mm-page_alloc-use-static-global-work_struct-for-draining-per-cpu-pages-fix/20170208-050036
base:   git://git.cmpxchg.org/linux-mmotm.git master
config: i386-randconfig-x001-201706 (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

All error/warnings (new ones prefixed by >>):

   In file included from include/asm-generic/percpu.h:6:0,
                    from arch/x86/include/asm/percpu.h:542,
                    from arch/x86/include/asm/preempt.h:5,
                    from include/linux/preempt.h:59,
                    from include/linux/spinlock.h:50,
                    from include/linux/mmzone.h:7,
                    from include/linux/gfp.h:5,
                    from include/linux/mm.h:9,
                    from mm/page_alloc.c:18:
   mm/page_alloc.c: In function 'drain_all_pages':
Quoted 3 lines
Quoted 1 line
Quoted 1 line

include/linux/percpu-defs.h:91:33: error: section attribute cannot be specified for local variables

     extern __PCPU_DUMMY_ATTRS char __pcpu_unique_##name;  \
                                    ^

huh, yes. The DEFINE_PER_CPU() macro is broken.

If you do

foo()

{
	static DEFINE_PER_CPU(int, bar);
}

then it won't compile, as described here. It should.

And if you do

static DEFINE_PER_CPU(int, bar);

Quoted 7 lines

then you still get global symbols (__pcpu_unique_bar).

The kernel does the above thing in, umm, 466 places and afaict they're all broken. If two code sites ever use the same identifier, they'll get linkage errors.

huh. Seems hard to fix.

Nasty! Unfortunately, I am not familiar with the static pcp code magic to come up with a fix.

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#5 lore raw

Re: [PATCH] mm-page_alloc-use-static-global-work_struct-for-draining-per-cpu-pages-fix

In reply to Andrew Morton

Hello, Andrew.

Quoted 4 lines · Andrew Morton
Quoted 2 lines
     extern __PCPU_DUMMY_ATTRS char __pcpu_unique_##name;  \
                                    ^

huh, yes. The DEFINE_PER_CPU() macro is broken.

Yeah, that was the trade off I had to take with percpu vars to force s390 and alpha to generate long references (GOT based addressing) for percpu variables; otherwise, they generate memory deref which is too limited to access the special percpu addresses. It's explained in include/linux/percpu-defs.h.

Quoted 11 lines

If you do

foo()

{
	static DEFINE_PER_CPU(int, bar);
}

then it won't compile, as described here. It should.

And if you do

Quoted 7 lines

static DEFINE_PER_CPU(int, bar);

then you still get global symbols (__pcpu_unique_bar).

The kernel does the above thing in, umm, 466 places and afaict they're all broken. If two code sites ever use the same identifier, they'll get linkage errors.

So, we have CONFIG_DEBUG_FORCE_WEAK_PER_CPU to catch those cases on archs other than s390 or alpha.

Quoted 1 line

huh. Seems hard to fix.

This was the only way I could come up with to support alpha and s390. All the restrictions are there to ensure that. If we can do s390 and alpha w/o the global weak reference, neither restriction is necessary.

Thanks.

Signature · 7 lines
tejun

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>