unable to handle kernel paging request when running btrfs/011

Saw below warn leading to bug when running btrfs/011, not reproducible. Any idea ?

------
static void __queue_work(..)
{
::

       /* if draining, only works from the same workqueue are allowed */
         if (unlikely(wq->flags & __WQ_DRAINING) &&
             WARN_ON_ONCE(!is_chained_work(wq)))       <---
                 return;

and then btrfs reported..

   BUG: unable to handle kernel paging request

(gdb) list *btrfs_destroy_workqueue+0x1f
0x6a43f is in btrfs_destroy_workqueue (fs/btrfs/async-thread.c:350).
345 }
346

347    static inline void
348    __btrfs_destroy_workqueue(struct __btrfs_workqueue *wq)
349    {
350        destroy_workqueue(wq->normal_wq); <---
351        trace_btrfs_workqueue_destroy(wq);
352        kfree(wq);

353 }
354
------

Thanks, Anand

[   61.038867] ------------[ cut here ]------------
[   61.038876] WARNING: CPU: 0 PID: 117 at kernel/workqueue.c:1381
__queue_work+0x37f/0x470()
[   61.038878] Modules linked in: btrfs xor raid6_pq cfg80211 rfkill

ppdev pcspkr psmouse serio_raw evdev joydev mousedev mac_hid acpi_cpufreq processor e1000 battery intel_agp intel_gtt video parport_pc parport ac button sch_fq_codel nfs lockd grace sunrpc fscache sr_mod cdrom sd_mod ata_generic pata_acpi hid_generic usbhid hid atkbd libps2 ata_piix ahci libahci ohci_pci ohci_hcd usbcore usb_common libata scsi_mod i8042 serio [last unloaded: xor]

[   61.038908] CPU: 0 PID: 117 Comm: kworker/u2:5 Not tainted
4.5.0-rc3asj+ #10
[   61.038909] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS
VirtualBox 12/01/2006
[   61.038931] Workqueue: btrfs-readahead btrfs_readahead_helper [btrfs]
[   61.038933]  0000000000000086 00000000bc179d46 ffff88011f403c40
ffffffff813a2651
[   61.038935]  0000000000000000 ffffffff817f7268 ffff88011f403c78
ffffffff81079892
[   61.038937]  0000000000000246 0000000000000080 ffff8800da4fcb20
ffff88011a59db90
[   61.038939] Call Trace:
[   61.038941]  <IRQ>  [<ffffffff813a2651>] dump_stack+0x63/0x82
[   61.038946]  [<ffffffff81079892>] warn_slowpath_common+0x82/0xc0
[   61.038948]  [<ffffffff810799da>] warn_slowpath_null+0x1a/0x20
[   61.038949]  [<ffffffff810910ef>] __queue_work+0x37f/0x470
[   61.038951]  [<ffffffff81091207>] queue_work_on+0x27/0x40
[   61.038964]  [<ffffffffa00933a0>] btrfs_queue_work+0x90/0xe0 [btrfs]
[   61.038974]  [<ffffffffa0056785>] end_workqueue_bio+0x85/0xe0 [btrfs]
[   61.038978]  [<ffffffff8136c4af>] bio_endio+0x3f/0x60
[   61.038990]  [<ffffffffa008d075>] btrfs_end_bio+0x95/0x140 [btrfs]
[   61.038992]  [<ffffffff8136c4af>] bio_endio+0x3f/0x60
[   61.038995]  [<ffffffff8137390b>] blk_update_request+0x8b/0x310
[   61.039002]  [<ffffffffa016ab43>] scsi_end_request+0x33/0x1d0 [scsi_mod]
[   61.039008]  [<ffffffffa016d26a>] scsi_io_completion+0xba/0x5f0
[scsi_mod]
[   61.039012]  [<ffffffffa0164fae>] scsi_finish_command+0xbe/0xf0
[scsi_mod]
[   61.039017]  [<ffffffffa016cbc8>] scsi_softirq_done+0xf8/0x120
[scsi_mod]
[   61.039019]  [<ffffffff8137b36b>] blk_done_softirq+0x8b/0xb0
[   61.039021]  [<ffffffff8107dadf>] __do_softirq+0xcf/0x290
[   61.039023]  [<ffffffff8107de13>] irq_exit+0xa3/0xb0
[   61.039026]  [<ffffffff815f2bc4>] do_IRQ+0x54/0xd0
[   61.039028]  [<ffffffff815f0d42>] common_interrupt+0x82/0x82
[   61.039028]  <EOI>  [<ffffffffa00c09d8>] ?
reada_extent_put+0x128/0x1a0 [btrfs]
[   61.039043]  [<ffffffff81387101>] ? set_task_ioprio+0x21/0xb0
[   61.039054]  [<ffffffffa00c1b18>]
reada_start_machine_worker+0x228/0x360 [btrfs]
[   61.039065]  [<ffffffffa0092df7>] btrfs_scrubparity_helper+0x77/0x2f0
[btrfs]
[   61.039076]  [<ffffffffa00930ce>] btrfs_readahead_helper+0xe/0x10
[btrfs]
[   61.039078]  [<ffffffff8109223e>] process_one_work+0x14e/0x440
[   61.039080]  [<ffffffff81092578>] worker_thread+0x48/0x4e0
[   61.039081]  [<ffffffff81092530>] ? process_one_work+0x440/0x440
[   61.039083]  [<ffffffff810980c8>] kthread+0xd8/0xf0
[   61.039085]  [<ffffffff81097ff0>] ? kthread_worker_fn+0x180/0x180
[   61.039087]  [<ffffffff815f060f>] ret_from_fork+0x3f/0x70
[   61.039089]  [<ffffffff81097ff0>] ? kthread_worker_fn+0x180/0x180
[   61.039090] ---[ end trace 1531237816dde88b ]---
[   61.039119] BUG: unable to handle kernel paging request at
0000000100000000
[   61.039121] IP: [<ffffffffa009340f>]
btrfs_destroy_workqueue+0x1f/0xf0 [btrfs]
[   61.039140] PGD 116827067 PUD 0
[   61.039143] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[   61.039145] Modules linked in: btrfs xor raid6_pq cfg80211 rfkill

ppdev pcspkr psmouse serio_raw evdev joydev mousedev mac_hid acpi_cpufreq processor e1000 battery intel_agp intel_gtt video parport_pc parport ac button sch_fq_codel nfs lockd grace sunrpc fscache sr_mod cdrom sd_mod ata_generic pata_acpi hid_generic usbhid hid atkbd libps2 ata_piix ahci libahci ohci_pci ohci_hcd usbcore usb_common libata scsi_mod i8042 serio [last unloaded: xor]

[   61.039163] CPU: 0 PID: 11203 Comm: umount Tainted: G        W
4.5.0-rc3asj+ #10
[   61.039164] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS
VirtualBox 12/01/2006
[   61.039166] task: ffff8800d8d82ac0 ti: ffff8800be6a0000 task.ti:
ffff8800be6a0000
[   61.039167] RIP: 0010:[<ffffffffa009340f>]  [<ffffffffa009340f>]
btrfs_destroy_workqueue+0x1f/0xf0 [btrfs]
[   61.039178] RSP: 0018:ffff8800be6a3d70  EFLAGS: 00010206
[   61.039179] RAX: ffff8800da4fcf01 RBX: ffff8800d9372000 RCX:
000000000011b080
[   61.039180] RDX: 000000000011b000 RSI: ffff88011f418a40 RDI:
ffff8800da4fcf90
[   61.039181] RBP: ffff8800be6a3d88 R08: 0000000000018a40 R09:
ffffffffa0093447
[   61.039182] R10: ffffea0003693f00 R11: ffff8800da31d700 R12:
ffff8800da4fcf90
[   61.039183] R13: 0000000100000000 R14: 0000000000000000 R15:
ffff8800d8d82ac0
[   61.039185] FS:  00007f28c0315780(0000) GS:ffff88011f400000(0000)
knlGS:0000000000000000
[   61.039186] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   61.039187] CR2: 0000000100000000 CR3: 0000000119b0f000 CR4:
00000000000406f0
[   61.039190] Stack:
[   61.039191]  ffff8800d9372000 ffff8800d9372b58 ffff8800daf04800
ffff8800be6a3da0
[   61.039193]  ffffffffa0057d6d ffff8800d9372000 ffff8800be6a3df8
ffffffffa005b8f2
[   61.039194]  ffff8800daf06800 ffff8800daf06e00 ffff8800be6a4000
ffff8800be6a3dc8
[   61.039196] Call Trace:
[   61.039207]  [<ffffffffa0057d6d>] btrfs_stop_all_workers+0xcd/0xf0
[btrfs]
[   61.039218]  [<ffffffffa005b8f2>] close_ctree+0x182/0x360 [btrfs]
[   61.039225]  [<ffffffffa002ec69>] btrfs_put_super+0x19/0x20 [btrfs]
[   61.039231]  [<ffffffff811f695a>] generic_shutdown_super+0x6a/0xf0
[   61.039233]  [<ffffffff811f6c72>] kill_anon_super+0x12/0x20
[   61.039241]  [<ffffffffa002e908>] btrfs_kill_super+0x18/0x110 [btrfs]
[   61.039243]  [<ffffffff811f6e13>] deactivate_locked_super+0x43/0x70
[   61.039244]  [<ffffffff811f79fc>] deactivate_super+0x5c/0x60
[   61.039247]  [<ffffffff8121328f>] cleanup_mnt+0x3f/0x90
[   61.039248]  [<ffffffff81213322>] __cleanup_mnt+0x12/0x20
[   61.039250]  [<ffffffff810963a3>] task_work_run+0x73/0x90
[   61.039252]  [<ffffffff8100369e>] exit_to_usermode_loop+0xae/0xb0
[   61.039254]  [<ffffffff81003b9e>] syscall_return_slowpath+0x4e/0x60
[   61.039256]  [<ffffffff815f040c>] int_ret_from_sys_call+0x25/0x8f
[   61.039257] Code: ff e8 2a fc f6 e0 eb a1 0f 1f 40 00 0f 1f 44 00 00
48 85 ff 74 53 55 48 89 e5 41 55 41 54 53 4c 8b 6f 08 49 89 fc 4d 85 ed
74 16 <49> 8b 7d 00 e8 d8 e4 ff e0 0f 1f 44 00 00 4c 89 ef e8 0b 06 14
[   61.039276] RIP  [<ffffffffa009340f>]
btrfs_destroy_workqueue+0x1f/0xf0 [btrfs]
[   61.039287]  RSP <ffff8800be6a3d70>
[   61.039288] CR2: 0000000100000000
[   61.039290] ---[ end trace 1531237816dde88c ]---

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 5 lines · Anand Jain

Saw below warn leading to bug when running btrfs/011, not reproducible. Any idea ?

Seems like another wq_destroy race.

But it's hard to locate which wq is the cause from backtrace only.

Quoted 2 lines · Anand Jain

Quoted 3 lines

Saw below warn leading to bug when running btrfs/011, not reproducible. Any idea ?

Seems like another wq_destroy race.

But it's hard to locate which wq is the cause from backtrace only.

What's the wq btrfs_stop_all_workers+0xcd is going to free?

And what's the wq end_workqueue_bio+0x85 is going to add?

Quoted 10 lines · Anand Jain

Quoted 5 lines

Saw below warn leading to bug when running btrfs/011, not reproducible. Any idea ?

Seems like another wq_destroy race.

Quoted 3 lines

But it's hard to locate which wq is the cause from backtrace only.

What's the wq btrfs_stop_all_workers+0xcd is going to free?

You can always try to guess on your locally built sources, provided the configs do not enable "too much" debugging. What I get and looks reasonable:

(gdb) l *(btrfs_stop_all_workers+0xcd)
0x2bc7d is in btrfs_stop_all_workers (fs/btrfs/disk-io.c:2154).
warning: Source file is more recent than executable.

2149            btrfs_destroy_workqueue(fs_info->delayed_workers);
2150            btrfs_destroy_workqueue(fs_info->caching_workers);
2151            btrfs_destroy_workqueue(fs_info->readahead_workers);
2152            btrfs_destroy_workqueue(fs_info->flush_workers);
2153            btrfs_destroy_workqueue(fs_info->qgroup_rescan_workers);
2154            btrfs_destroy_workqueue(fs_info->extent_workers);

2155 }
2156
2157 static void free_root_extent_buffers(struct btrfs_root *root)
2158 {

Quoted 1 line

And what's the wq end_workqueue_bio+0x85 is going to add?

(gdb) l *(end_workqueue_bio+0x85)
0x2ab55 is in end_workqueue_bio (fs/btrfs/disk-io.c:726).
721
722 if (bio->bi_rw & REQ_WRITE) {
723 if (end_io_wq->metadata == BTRFS_WQ_ENDIO_METADATA) {
724 wq = fs_info->endio_meta_write_workers;
725 func = btrfs_endio_meta_write_helper;
726 } else if (end_io_wq->metadata == BTRFS_WQ_ENDIO_FREE_SPACE) {
727 wq = fs_info->endio_freespace_worker;
728 func = btrfs_freespace_write_helper;
729 } else if (end_io_wq->metadata == BTRFS_WQ_ENDIO_RAID56) {
730 wq = fs_info->endio_raid56_workers;
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html