irq_work: A couple fixes v2
cover irq_work: A couple fixes v2
Hey,
After some discussion with Steve, this is a respin with changelogs and comments sanitized. The code itself hasn't changed.
Thanks.
Frederic Weisbecker (2):
irq_work: Fix racy IRQ_WORK_BUSY flag setting
irq_work: Fix racy check on work pending flag
kernel/irq_work.c | 21 +++++++++++++++------ 1 files changed, 15 insertions(+), 6 deletions(-)
Signature · 1 line
1.7.5.4
PATCH 1/2 irq_work: Fix racy IRQ_WORK_BUSY flag setting
The IRQ_WORK_BUSY flag is set right before we execute the work. Once this flag value is set, the work enters a claimable state again.
So if we have specific data to compute in our work, we ensure it's either handled by another CPU or locally by enqueuing the work again. This state machine is guanranteed by atomic operations on the flags.
So when we set IRQ_WORK_BUSY without using an xchg-like operation, we break this guarantee as in the following summarized scenario:
CPU 1 CPU 2
----- -----
(flags = 0)
old_flags = flags;
(flags = 0)
cmpxchg(flags, old_flags,
old_flags | IRQ_WORK_FLAGS)
(flags = 3)
[...]
flags = IRQ_WORK_BUSY
(flags = 2)
func()
(sees flags = 3)
cmpxchg(flags, old_flags,
old_flags | IRQ_WORK_FLAGS)
(give up)
cmpxchg(flags, 2, 0);
(flags = 0)CPU 1 claims a work and executes it, so it sets IRQ_WORK_BUSY and the work is again in a claimable state. Now CPU 2 has new data to process and try to claim that work but it may see a stale value of the flags and think the work is still pending somewhere that will handle our data. This is because CPU 1 doesn't set IRQ_WORK_BUSY atomically.
As a result, the data expected to be handle by CPU 2 won't get handled.
To fix this, use xchg() to set IRQ_WORK_BUSY, this way we ensure the CPU 2 will see the correct value with cmpxchg() using the expected ordering.
--- kernel/irq_work.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)
kernel/irq_work.c +4 -1
@@ -119,8 +119,11 @@ void irq_work_run(void)/* * Clear the PENDING bit, after this point the @work * can be re-used. +* Make it immediately visible so that other CPUs trying +* to claim that work don't rely on us to handle their data +* while we are in the middle of the func. */ - work->flags =IRQ_WORK_BUSY ; +xchg (&work->flags,IRQ_WORK_BUSY ); work->func (work);/* * Clear the BUSY bit and return to the free state if
Signature · 1 line
1.7.5.4
PATCH 2/2 irq_work: Fix racy check on work pending flag
Work claiming wants to be SMP-safe.
And by the time we try to claim a work, if it is already executing concurrently on another CPU, we want to succeed the claiming and queue the work again because the other CPU may have missed the data we wanted to handle in our work if it's about to complete there.
This scenario is summarized below:
CPU 1 CPU 2
----- -----
(flags = 0)
cmpxchg(flags, 0, IRQ_WORK_FLAGS)
(flags = 3)
[...]
xchg(flags, IRQ_WORK_BUSY)
(flags = 2)
func()
if (flags & IRQ_WORK_PENDING)
(not true)
cmpxchg(flags, flags, IRQ_WORK_FLAGS)
(flags = 3)
[...]
cmpxchg(flags, IRQ_WORK_BUSY, 0);
(fail, pending on CPU 2)This state machine is synchronized using [cmp]xchg() on the flags. As such, the early IRQ_WORK_PENDING check in CPU 2 above is racy. By the time we check it, we may be dealing with a stale value because we aren't using an atomic accessor. As a result, CPU 2 may "see" that the work is still pending on another CPU while it may be actually completing the work function exection already, leaving our data unprocessed.
To fix this, we start by speculating about the value we wish to be in the work->flags but we only make any conclusion after the value returned by the cmpxchg() call that either claims the work or let the current owner handle the pending work for us.
--- kernel/irq_work.c | 16 +++++++++++----- 1 files changed, 11 insertions(+), 5 deletions(-)
kernel/irq_work.c +11 -5
@@ -34,15 +34,21 @@ static DEFINE_PER_CPU(struct llist_head, irq_work_list);*/ static bool irq_work_claim (struct irq_work *work) { -unsigned long flags, nflags; +unsigned long flags, oflags, nflags; +/* +* Start with our best wish as a premise but only trust any +* flag value after cmpxchg() result. +*/ + flags = work->flags & ~IRQ_WORK_PENDING ;for (;;) { - flags = work->flags; -if (flags &IRQ_WORK_PENDING ) -return false; nflags = flags |IRQ_WORK_FLAGS ; -if (cmpxchg (&work->flags, flags, nflags) == flags) + oflags =cmpxchg (&work->flags, flags, nflags); +if (oflags == flags)break ; +if (oflags &IRQ_WORK_PENDING ) +return false; + flags = oflags;cpu_relax (); }
Signature · 1 line
1.7.5.4
Re: [PATCH 2/2] irq_work: Fix racy check on work pending flag
Quoted 39 lines · Frederic Weisbecker
Work claiming wants to be SMP-safe.
And by the time we try to claim a work, if it is already executing concurrently on another CPU, we want to succeed the claiming and queue the work again because the other CPU may have missed the data we wanted to handle in our work if it's about to complete there.
This scenario is summarized below:
CPU 1 CPU 2 ----- ----- (flags = 0) cmpxchg(flags, 0, IRQ_WORK_FLAGS) (flags = 3) [...] xchg(flags, IRQ_WORK_BUSY) (flags = 2) func() if (flags & IRQ_WORK_PENDING) (not true) cmpxchg(flags, flags, IRQ_WORK_FLAGS) (flags = 3) [...] cmpxchg(flags, IRQ_WORK_BUSY, 0); (fail, pending on CPU 2)This state machine is synchronized using [cmp]xchg() on the flags. As such, the early
IRQ_WORK_PENDINGcheck in CPU 2 above is racy. By the time we check it, we may be dealing with a stale value because we aren't using an atomic accessor. As a result, CPU 2 may "see" that the work is still pending on another CPU while it may be actually completing the work function exection already, leaving our data unprocessed.To fix this, we start by speculating about the value we wish to be in the
work->flagsbut we only make any conclusion after the value returned by thecmpxchg()call that either claims the work or let the current owner handle the pending work for us.
Quoted 6 lines
Changelog-heavily-inspired-by Steven Rostedt <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Ingo Molnar <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Andrew Morton <[email protected]>
-- Steve
Quoted 37 lines
Cc Paul Gortmaker <[email protected]>Cc Anish Kumar <[email protected]>--- kernel/irq_work.c | 16 +++++++++++----- 1 files changed, 11 insertions(+), 5 deletions(-)kernel/irq_work.c +11
-5@@ -34,15 +34,21 @@ static DEFINE_PER_CPU(struct llist_head, irq_work_list);*/ static bool irq_work_claim (struct irq_work *work) { -unsigned long flags, nflags; +unsigned long flags, oflags, nflags; +/* +* Start with our best wish as a premise but only trust any +* flag value after cmpxchg() result. +*/ + flags = work->flags & ~IRQ_WORK_PENDING ;for (;;) { - flags = work->flags; -if (flags &IRQ_WORK_PENDING ) -return false; nflags = flags |IRQ_WORK_FLAGS ; -if (cmpxchg (&work->flags, flags, nflags) == flags) + oflags =cmpxchg (&work->flags, flags, nflags); +if (oflags == flags)break ; +if (oflags &IRQ_WORK_PENDING ) +return false; + flags = oflags;cpu_relax (); }
Re: [PATCH 1/2] irq_work: Fix racy IRQ_WORK_BUSY flag setting
In reply to Frederic WeisbeckerQuoted 42 lines · Frederic Weisbecker
The
IRQ_WORK_BUSYflag is set right before we execute the work. Once this flag value is set, the work enters a claimable state again.So if we have specific data to compute in our work, we ensure it's either handled by another CPU or locally by enqueuing the work again. This state machine is guanranteed by atomic operations on the flags.
So when we set
IRQ_WORK_BUSYwithout using an xchg-like operation, we break this guarantee as in the following summarized scenario:CPU 1 CPU 2 ----- ----- (flags = 0) old_flags = flags; (flags = 0) cmpxchg(flags, old_flags, old_flags | IRQ_WORK_FLAGS) (flags = 3) [...] flags = IRQ_WORK_BUSY (flags = 2) func() (sees flags = 3) cmpxchg(flags, old_flags, old_flags | IRQ_WORK_FLAGS) (give up) cmpxchg(flags, 2, 0); (flags = 0)CPU 1 claims a work and executes it, so it sets
IRQ_WORK_BUSYand the work is again in a claimable state. Now CPU 2 has new data to process and try to claim that work but it may see a stale value of the flags and think the work is still pending somewhere that will handle our data. This is because CPU 1 doesn't setIRQ_WORK_BUSYatomically.As a result, the data expected to be handle by CPU 2 won't get handled.
To fix this, use
xchg()to setIRQ_WORK_BUSY, this way we ensure the CPU 2 will see the correct value withcmpxchg()using the expected ordering.
Quoted 6 lines
Changelog-heavily-inspired-by Steven Rostedt <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Ingo Molnar <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Andrew Morton <[email protected]>
-- Steve
Quoted 23 lines
Cc Paul Gortmaker <[email protected]>Cc Anish Kumar <[email protected]>--- kernel/irq_work.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)kernel/irq_work.c +4
-1@@ -119,8 +119,11 @@ void irq_work_run(void)/* * Clear the PENDING bit, after this point the @work * can be re-used. +* Make it immediately visible so that other CPUs trying +* to claim that work don't rely on us to handle their data +* while we are in the middle of the func. */ - work->flags =IRQ_WORK_BUSY ; +xchg (&work->flags,IRQ_WORK_BUSY ); work->func (work);/* * Clear the BUSY bit and return to the free state if