Skip to contentSkip to conversation

irq_work: Fix IRQ_WORK_BUZY bit clearing

PATCH Applied Frederic Weisbecker 4 participants 4 messages Nov 13, 2019 – Nov 15, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 irq_work: Fix IRQ_WORK_BUZY bit clearing

While attempting to clear the buzy bit at the end of a work execution, atomic_cmpxchg() expects the value of the flags with the pending bit cleared as the old value. However we are passing by mistake the value of the flags before we actually cleared the pending bit.

As a result, clearing the buzy bit fails and irq_work_sync() may stall:

	watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [blktrace:4948]
	CPU: 0 PID: 4948 Comm: blktrace Not tainted 5.4.0-rc7-00003-gfeb4a51323bab #1
	Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
	RIP: 0010:irq_work_sync+0x4/0x10
	Call Trace:
	  relay_close_buf+0x19/0x50
	  relay_close+0x64/0x100
	  blk_trace_free+0x1f/0x50
	  __blk_trace_remove+0x1e/0x30
	  blk_trace_ioctl+0x11b/0x140
	  blkdev_ioctl+0x6c1/0xa40
	  block_ioctl+0x39/0x40
	  do_vfs_ioctl+0xa5/0x700
	  ksys_ioctl+0x70/0x80
	  __x64_sys_ioctl+0x16/0x20
	  do_syscall_64+0x5b/0x1d0
	  entry_SYSCALL_64_after_hwframe+0x44/0xa9

So clear the appropriate bit before passing the old flags to cmpxchg().

Fixes feb4a51323ba ("irq_work: Slightly simplify IRQ_WORK_PENDING clearing")
Cc Paul E . McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/irq_work.c | 1 +
 1 file changed, 1 insertion(+)
kernel/irq_work.c +1 -0
index 49c53f80a13a..828cc30774bc 100644
@@ -158,6 +158,7 @@ static void irq_work_run_list(struct llist_head *list)
 		 * Clear the BUSY bit and return to the free state if
 		 * no-one else claimed it meanwhile.
 		 */
+		flags &= ~IRQ_WORK_PENDING;
 		(void)atomic_cmpxchg(&work->flags, flags, flags & ~IRQ_WORK_BUSY);
 	}
 }
Signature · 1 line
2.23.0
Tested-by on v1 1/1
#1 lore raw
Quoted 4 lines · Frederic Weisbecker

While attempting to clear the buzy bit at the end of a work execution, atomic_cmpxchg() expects the value of the flags with the pending bit cleared as the old value. However we are passing by mistake the value of the flags before we actually cleared the pending bit.

Busy is spelled with an S

Quoted 23 lines

As a result, clearing the buzy bit fails and irq_work_sync() may stall:

	watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [blktrace:4948]
	CPU: 0 PID: 4948 Comm: blktrace Not tainted 5.4.0-rc7-00003-gfeb4a51323bab #1
	Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
	RIP: 0010:irq_work_sync+0x4/0x10
	Call Trace:
	  relay_close_buf+0x19/0x50
	  relay_close+0x64/0x100
	  blk_trace_free+0x1f/0x50
	  __blk_trace_remove+0x1e/0x30
	  blk_trace_ioctl+0x11b/0x140
	  blkdev_ioctl+0x6c1/0xa40
	  block_ioctl+0x39/0x40
	  do_vfs_ioctl+0xa5/0x700
	  ksys_ioctl+0x70/0x80
	  __x64_sys_ioctl+0x16/0x20
	  do_syscall_64+0x5b/0x1d0
	  entry_SYSCALL_64_after_hwframe+0x44/0xa9

So clear the appropriate bit before passing the old flags to cmpxchg().

Quoted 8 lines
Fixes feb4a51323ba ("irq_work: Slightly simplify IRQ_WORK_PENDING clearing")
Cc Paul E . McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]> everywhere.
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>

Without this patch switching cpufreq governors hangs on arm64.

Quoted 17 lines
---
  kernel/irq_work.c | 1 +
  1 file changed, 1 insertion(+)
kernel/irq_work.c +1 -0
index 49c53f80a13a..828cc30774bc 100644
@@ -158,6 +158,7 @@ static void irq_work_run_list(struct llist_head *list)
  		 * Clear the BUSY bit and return to the free state if
  		 * no-one else claimed it meanwhile.
  		 */
+		flags &= ~IRQ_WORK_PENDING;
  		(void)atomic_cmpxchg(&work->flags, flags, flags & ~IRQ_WORK_BUSY);
  	}
  }
Tested-by
#2 lore raw

Hi Frederic,

Thanks for this fix patch.

Quoted 41 lines · Leonard Crestez
Quoted 4 lines · Frederic Weisbecker

While attempting to clear the buzy bit at the end of a work execution, atomic_cmpxchg() expects the value of the flags with the pending bit cleared as the old value. However we are passing by mistake the value of the flags before we actually cleared the pending bit.

Busy is spelled with an S

Quoted 23 lines

As a result, clearing the buzy bit fails and irq_work_sync() may stall:

      watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [blktrace:4948]
      CPU: 0 PID: 4948 Comm: blktrace Not tainted 5.4.0-rc7-00003-gfeb4a51323bab #1
      Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
      RIP: 0010:irq_work_sync+0x4/0x10
      Call Trace:
        relay_close_buf+0x19/0x50
        relay_close+0x64/0x100
        blk_trace_free+0x1f/0x50
        __blk_trace_remove+0x1e/0x30
        blk_trace_ioctl+0x11b/0x140
        blkdev_ioctl+0x6c1/0xa40
        block_ioctl+0x39/0x40
        do_vfs_ioctl+0xa5/0x700
        ksys_ioctl+0x70/0x80
        __x64_sys_ioctl+0x16/0x20
        do_syscall_64+0x5b/0x1d0
        entry_SYSCALL_64_after_hwframe+0x44/0xa9

So clear the appropriate bit before passing the old flags to cmpxchg().

Quoted 8 lines
Fixes feb4a51323ba ("irq_work: Slightly simplify IRQ_WORK_PENDING clearing")
Cc Paul E . McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]> everywhere.
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Quoted 3 lines

Without this patch switching cpufreq governors hangs on arm64.

Right.

This patch solved two problems,
1) juno-r2 boot pass now
2) rcu_sched self-detected stall on CPU on x86_64 problem is solved now.

Hope this will get merged into linux next.

ref:
https://lkft.validation.linaro.org/scheduler/job/1010542#L260
https://lkft.validation.linaro.org/scheduler/job/1010793#L493

- Naresh

#3 lore raw

[tip: irq/core] irq_work: Fix IRQ_WORK_BUSY bit clearing

In reply to Frederic Weisbecker
The following commit has been merged into the irq/core branch of tip:

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     e9838bd51169af87ae248336d4c3fc59184a0e46
Gitweb:        https://git.kernel.org/tip/e9838bd51169af87ae248336d4c3fc59184a0e46
Author:        Frederic Weisbecker <[email protected]>
AuthorDate:    Wed, 13 Nov 2019 18:12:01 +01:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Fri, 15 Nov 2019 10:48:37 +01:00

irq_work: Fix IRQ_WORK_BUSY bit clearing

While attempting to clear the busy bit at the end of a work execution, atomic_cmpxchg() expects the value of the flags with the pending bit cleared as the old value. However by mistake the value of the flags is passed without clearing the pending bit first.

As a result, clearing the busy bit fails and irq_work_sync() may stall:

 watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [blktrace:4948]
 CPU: 0 PID: 4948 Comm: blktrace Not tainted 5.4.0-rc7-00003-gfeb4a51323bab #1
 RIP: 0010:irq_work_sync+0x4/0x10
 Call Trace:
  relay_close_buf+0x19/0x50
  relay_close+0x64/0x100
  blk_trace_free+0x1f/0x50
  __blk_trace_remove+0x1e/0x30
  blk_trace_ioctl+0x11b/0x140
  blkdev_ioctl+0x6c1/0xa40
  block_ioctl+0x39/0x40
  do_vfs_ioctl+0xa5/0x700
  ksys_ioctl+0x70/0x80
  __x64_sys_ioctl+0x16/0x20
  do_syscall_64+0x5b/0x1d0
  entry_SYSCALL_64_after_hwframe+0x44/0xa9

So clear the appropriate bit before passing the old flags to cmpxchg().

---
 kernel/irq_work.c | 1 +
 1 file changed, 1 insertion(+)
kernel/irq_work.c +1 -0
index 49c53f8..828cc30 100644
@@ -158,6 +158,7 @@ static void irq_work_run_list(struct llist_head *list)
 		 * Clear the BUSY bit and return to the free state if
 		 * no-one else claimed it meanwhile.
 		 */
+		flags &= ~IRQ_WORK_PENDING;
 		(void)atomic_cmpxchg(&work->flags, flags, flags & ~IRQ_WORK_BUSY);
 	}
 }