smp,cpumask: Don't call functions on offline CPUs
PATCH 1/1 smp,cpumask: Don't call functions on offline CPUs
When we are able to allocate a cpumask in on_each_cpu_cond_mask we call functions with on_each_cpu_mask - this masks out offline cpus via smp_call_function_many.
However when we fail to allocate a cpumask in on_each_cpu_cond_mask we call functions with smp_call_function_single - this will return -ENXIO from generic_exec_single if a CPU is offline which will result in a WARN_ON_ONCE.
Let's avoid the WARN by only calling smp_call_function_single when the CPU is online and thus making both paths consistent with each other.
--- kernel/smp.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
kernel/smp.c +6 -2
@@ -259,6 +259,7 @@ static void flush_smp_call_function_queue(bool warn_cpu_offline)/* * smp_call_function_single - Run a function on a specific CPU +* @cpu: The CPU to run on. * @func: The function to run. This must be fast and non-blocking. * @info: An arbitrary pointer to pass to the function. * @wait: If true, wait until function has completed on other CPUs.
@@ -657,6 +658,8 @@ EXPORT_SYMBOL(on_each_cpu_mask);* completed on other CPUs. * @gfp_flags: GFP flags to use when allocating the cpumask * used internally by the function. +* @mask: The set of cpus to run on (only runs on online +subset). * * The function might sleep if the GFP flags indicates a non * atomic allocation is allowed.
@@ -690,12 +693,13 @@ void on_each_cpu_cond_mask(bool (*cond_func)(int cpu, void *info),* just have to IPI them one by one. */ preempt_disable (); -for_each_cpu (cpu, mask) -if (cond_func (cpu, info)) { +for_each_cpu (cpu, mask) { +if (cpu_online (cpu) &&cond_func (cpu, info)) { ret =smp_call_function_single (cpu, func, info, wait);WARN_ON_ONCE (ret); } + }preempt_enable (); } }
Signature · 1 line
2.21.0
Quoted 4 lines · Andrew Murray
When we are able to allocate a cpumask in
on_each_cpu_cond_maskwe call functions withon_each_cpu_mask- this masks out offline cpus viasmp_call_function_many.
Quoted 8 lines
However when we fail to allocate a cpumask in
on_each_cpu_cond_maskwe call functions withsmp_call_function_single- this will return-ENXIOfromgeneric_exec_singleif a CPU is offline which will result in aWARN_ON_ONCE.Let's avoid the WARN by only calling
smp_call_function_singlewhen the CPU is online and thus making both paths consistent with each other.
I'm confused, why are you feeding it offline CPUs to begin with? @mask shouldn't include them.
Is perhaps the problem that on_each_cpu_cond() uses cpu_onlne_mask without protection?
Something like so?
kernel/smp.c +2 -0
@@ -705,8 +707,10 @@ void on_each_cpu_cond(bool (*cond_func)(int cpu, void *info), smp_call_func_t func, void *info, bool wait, gfp_t gfp_flags) { +cpus_read_lock ();on_each_cpu_cond_mask (cond_func, func, info, wait, gfp_flags, cpu_online_mask); +cpus_read_unlock (); }EXPORT_SYMBOL (on_each_cpu_cond);
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Andrew Murray
When we are able to allocate a cpumask in
on_each_cpu_cond_maskwe call functions withon_each_cpu_mask- this masks out offline cpus viasmp_call_function_many.However when we fail to allocate a cpumask in
on_each_cpu_cond_maskwe call functions withsmp_call_function_single- this will return-ENXIOfromgeneric_exec_singleif a CPU is offline which will result in aWARN_ON_ONCE.
Quoted 6 lines
Quoted 3 lines
Let's avoid the WARN by only calling
smp_call_function_singlewhen the CPU is online and thus making both paths consistent with each other.I'm confused, why are you feeding it offline CPUs to begin with? @mask shouldn't include them.
There are only two users of this function: on_each_cpu_cond which passes cpu_online_mask, and native_flush_tlb_others from arch/x86/mm/tlb.c. But I haven't followed all callers to native_flush_tlb_others to determine if they are only passing in online CPUs.
Whilst trying to understand the code I couldn't help but notice that the behaviour changes depending on the ability of memory allocation at the time. This seemed odd. And of course there may be future callers of this function that for some strange reason do pass in offline CPUs. (Maybe this is OK and it's something they shouldn't be doing).
I felt that this function should be consistent and document the behaviour via the @mask doc like some of the other functions in the file.
Quoted 3 lines
Is perhaps the problem that
on_each_cpu_cond()usescpu_onlne_maskwithout protection?
Does this prevent racing with a CPU going offline? I guess this prevents the warning at the expense of a lock - but is only beneficial in the unlikely path. (In the likely path this prevents new CPUs going offline but we don't care because we don't WARN if they aren't they when we attempt to call functions).
At least this is my limited understanding.
Thanks,
Andrew Murray
Quoted 19 lines
Something like so?
kernel/smp.c +2
-0@@ -705,8 +707,10 @@ void on_each_cpu_cond(bool (*cond_func)(int cpu, void *info), smp_call_func_t func, void *info, bool wait, gfp_t gfp_flags) { +cpus_read_lock ();on_each_cpu_cond_mask (cond_func, func, info, wait, gfp_flags, cpu_online_mask); +cpus_read_unlock (); }EXPORT_SYMBOL (on_each_cpu_cond);
Quoted 10 lines · Andrew Murray
Quoted 2 lines
Is perhaps the problem that
on_each_cpu_cond()usescpu_onlne_maskwithout protection?Does this prevent racing with a CPU going offline? I guess this prevents the warning at the expense of a lock - but is only beneficial in the unlikely path. (In the likely path this prevents new CPUs going offline but we don't care because we don't WARN if they aren't they when we attempt to call functions).
At least this is my limited understanding.
Hmm.. I don't think it could matter, we only use the mask when preempt_disable(), which would already block offline, due to it using stop_machine().
So the patch is a no-op.
What's the WARN you see? TLB invalidation should pass mm_cpumask(), which similarly should not contain offline CPUs I'm thinking.
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Andrew Murray
Quoted 2 lines
Is perhaps the problem that
on_each_cpu_cond()usescpu_onlne_maskwithout protection?Does this prevent racing with a CPU going offline? I guess this prevents the warning at the expense of a lock - but is only beneficial in the unlikely path. (In the likely path this prevents new CPUs going offline but we don't care because we don't WARN if they aren't they when we attempt to call functions).
Quoted 5 lines
Quoted 1 line
At least this is my limited understanding.
Hmm.. I don't think it could matter, we only use the mask when
preempt_disable(), which would already block offline, due to it usingstop_machine().
OK, so as long as all arches use stop_machine to bring down a CPU then preeempt_disable will stop racing with CPUs going offline (I don't know if they all do or not).
Quoted 5 lines
So the patch is a no-op.
What's the WARN you see? TLB invalidation should pass
mm_cpumask(), which similarly should not contain offline CPUs I'm thinking.
So the only remaining issue is if callers pass offline CPUs to the function (I guess there is still the chance of a race between calling on_each_cpu_cond_mask and entering the preempt disabled'd bit). As you suggest they probably don't.
Given the above, should we add a " @mask: The mask of cpus it can run on." to on_each_cpu_cond_mask? (Which is different to the wording of other functions in the same file that mask it with online CPUs, e.g. " @mask: The set of cpus to run on (only runs on online subset).".
(I haven't seen any WARN, but from looking at the code thought that it was possible.)
Thanks,
Andrew Murray
Quoted 13 lines · Andrew Murray
Quoted 12 lines · Peter Zijlstra
Quoted 10 lines · Andrew Murray
Quoted 2 lines
Is perhaps the problem that
on_each_cpu_cond()usescpu_onlne_maskwithout protection?Does this prevent racing with a CPU going offline? I guess this prevents the warning at the expense of a lock - but is only beneficial in the unlikely path. (In the likely path this prevents new CPUs going offline but we don't care because we don't WARN if they aren't they when we attempt to call functions).
At least this is my limited understanding.
Quoted 7 lines
Quoted 3 lines
Hmm.. I don't think it could matter, we only use the mask when
preempt_disable(), which would already block offline, due to it usingstop_machine().OK, so as long as all arches use
stop_machineto bring down a CPU thenpreeempt_disablewill stop racing with CPUs going offline (I don't know if they all do or not).
kernel/cpu.c:takedown_cpu() is generic code :-)
Quoted 9 lines
Quoted 4 lines
So the patch is a no-op.
What's the WARN you see? TLB invalidation should pass
mm_cpumask(), which similarly should not contain offline CPUs I'm thinking.So the only remaining issue is if callers pass offline CPUs to the function (I guess there is still the chance of a race between calling
on_each_cpu_cond_maskand entering the preempt disabled'd bit). As you suggest they probably don't.
Yes, it is possible to construct fail that way.
Quoted 16 lines · Peter Zijlstra
Quoted 13 lines · Andrew Murray
Quoted 3 lines
Is perhaps the problem that
on_each_cpu_cond()usescpu_onlne_mask
without protection?Does this prevent racing with a CPU going offline? I guess this prevents
the warning at the expense of a lock - but is only beneficial in the
unlikely path. (In the likely path this prevents new CPUs going offline
but we don't care because we don't WARN if they aren't they when we attempt to call functions).Quoted 1 line
At least this is my limited understanding.
Quoted 8 lines
Hmm.. I don't think it could matter, we only use the mask when
preempt_disable(), which would already block offline, due to it usingstop_machine().So the patch is a no-op.
What's the WARN you see? TLB invalidation should pass
mm_cpumask(), which similarly should not contain offline CPUs I'm thinking.
Does the TLB invalidation code have anything in it to prevent from racing with the CPU offline code?
In other words, could we end up with the TLB invalidation code building its bitmask, getting interrupted (eg. hypervisor preemption, NMI), and not sending out the IPI to that bitmask of CPUs until after one of the CPUs in the bitmap has gotten offlined?
Signature · 1 line
All Rights Reversed.
Quoted 29 lines · Rik van Riel
Quoted 16 lines · Peter Zijlstra
Quoted 13 lines · Andrew Murray
Quoted 3 lines
Is perhaps the problem that
on_each_cpu_cond()usescpu_onlne_mask
without protection?Does this prevent racing with a CPU going offline? I guess this prevents
the warning at the expense of a lock - but is only beneficial in the
unlikely path. (In the likely path this prevents new CPUs going offline
but we don't care because we don't WARN if they aren't they when we attempt to call functions).Quoted 1 line
At least this is my limited understanding.
Quoted 8 lines
Hmm.. I don't think it could matter, we only use the mask when
preempt_disable(), which would already block offline, due to it usingstop_machine().So the patch is a no-op.
What's the WARN you see? TLB invalidation should pass
mm_cpumask(), which similarly should not contain offline CPUs I'm thinking.Does the TLB invalidation code have anything in it to prevent from racing with the CPU offline code?
Quoted 6 lines
In other words, could we end up with the TLB invalidation code building its bitmask, getting interrupted (eg. hypervisor preemption, NMI), and not sending out the IPI to that bitmask of CPUs until after one of the CPUs in the bitmap has gotten offlined?
One possible thing would be if cpu-offline didn't remove the bit from mm_cpumask() because it entered lazy state earlier or something.
Then, mm_cpumask() would contain an offline CPU and the WARN could trigger, but I don't _think_ we do that, but I didn't check.