soft lockup when using btrfs dev del
Hello,
I'm running Linux 4.3 and also btrfs-progs-v4.3
I'm replacing one drive with another in my system. Both drives are healthy as far as I know.
I can start btrfs dev del and it will run for a while and then I get a cpu soft lockup.
I've restarted the process a few times and each time it runs a while, getting a bit further and then I get the cpu soft lockup again. (at which point the I/O has stopped and I typically have to hard power the system)
I do this to monitor the progress.
root@x1:~# while sleep 60; do /apps/src/btrfs-progs-v4.3/btrfs fi show /mnt/btrfs-seagate2tbp3-root; done
Label: none uuid: eebd1291-7ccd-49da-b3c3-93942527ead4
Total devices 2 FS bytes used 1.48TiB
devid 1 size 0.00B used 420.04GiB path
/dev/mapper/seagate2tbp3_crypt
devid 2 size 1.82TiB used 1.08TiB path /dev/mapper/sde1_cryptbtrfs-progs v4.3
Label: none uuid: eebd1291-7ccd-49da-b3c3-93942527ead4
Total devices 2 FS bytes used 1.48TiB
devid 1 size 0.00B used 418.04GiB path/dev/mapper/seagate2tbp3_crypt
devid 2 size 1.82TiB used 1.08TiB path /dev/mapper/sde1_crypt
Any suggestions on how to prevent this from happening?
Thanks.
Nov 8 01:02:30 x1 kernel: [22954.038620] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [btrfs:3086] Nov 8 01:02:30 x1 kernel: [22954.038621] Modules linked in: pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables usblp snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic nls_utf8 nls_cp437 vfat intel_rapl fat iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_intel snd_hda_codec snd_hda_core snd_pcsp snd_hwdep serio_raw snd_pcm snd_timer snd soundcore i2c_i801 efivars i915 evdev i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm mei_me shpchp mei battery wmi video button acpi_pad processor fuse efivarfs autofs4 algif_skcipher af_alg hid_generic usbhid dm_crypt dm_mod sg sr_mod cdrom sd_mod crct10dif_pclmul crc32_pclmul crc32c_intel jitterentropy_rng sha256_ssse3 sha256_generic hmac drbg ansi_cprng aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ahci libahci xhci_pci libata xhci_hcd scsi_mod usbcore usb_common fan thermal i2c_hid i2c_core hid Nov 8 01:02:30 x1 kernel: [22954.038684] CPU: 1 PID: 3086 Comm: btrfs Tainted: G W O 4.3.0 #1 Nov 8 01:02:30 x1 kernel: [22954.038685] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./H170M-DS3H-CF, BIOS F2 10/14/2015 Nov 8 01:02:30 x1 kernel: [22954.038686] task: ffff8803f30e91c0 ti: ffff8803f2e28000 task.ti: ffff8803f2e28000 Nov 8 01:02:30 x1 kernel: [22954.038687] RIP: 0010:[<ffffffff815e5060>] [<ffffffff815e5060>] _raw_spin_lock+0x10/0x20 Nov 8 01:02:30 x1 kernel: [22954.038702] RSP: 0018:ffff8803f2e2baf8 EFLAGS: 00000246 Nov 8 01:02:30 x1 kernel: [22954.038703] RAX: 0000000000000000 RBX: ffff8803f2e2bb38 RCX: 0000000000000000 Nov 8 01:02:30 x1 kernel: [22954.038704] RDX: 0000000000000001 RSI: ffff880471c9cf00 RDI: ffff88028d94dd70 Nov 8 01:02:30 x1 kernel: [22954.038704] RBP: ffff880131e94800 R08: ffff88046108a000 R09: 00000000ffffffe2 Nov 8 01:02:30 x1 kernel: [22954.038705] R10: 000000000000002a R11: 000000000000000b R12: ffff88028d94d800 Nov 8 01:02:30 x1 kernel: [22954.038706] R13: ffff8803f2e2bb38 R14: ffff8801e6a48800 R15: ffff88028d94dd78 Nov 8 01:02:30 x1 kernel: [22954.038706] FS: 00007f9ad24948c0(0000) GS:ffff880471c80000(0000) knlGS:0000000000000000 Nov 8 01:02:30 x1 kernel: [22954.038707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 8 01:02:30 x1 kernel: [22954.038708] CR2: 00000000014e2d38 CR3: 00000003d8989000 CR4: 00000000003406e0 Nov 8 01:02:30 x1 kernel: [22954.038709] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Nov 8 01:02:30 x1 kernel: [22954.038709] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Nov 8 01:02:30 x1 kernel: [22954.038710] Stack: Nov 8 01:02:30 x1 kernel: [22954.038710] ffffffff812f41e7 ffff8803f2e2bb38 ffff880131e94800 ffff88028d94d800 Nov 8 01:02:30 x1 kernel: [22954.038711] ffffffff812f42dd ffff880131e94bc0 ffffffff812f8b00 000000005698f3c0 Nov 8 01:02:30 x1 kernel: [22954.038713] ffff880131e94bc0 ffff880131e94bc0 ffff88028d94d8e8 ffff8802a77e6f50 Nov 8 01:02:30 x1 kernel: [22954.038714] Call Trace: Nov 8 01:02:30 x1 kernel: [22954.038716] [<ffffffff812f41e7>] ? __del_reloc_root+0x27/0x100 Nov 8 01:02:30 x1 kernel: [22954.038718] [<ffffffff812f42dd>] ? free_reloc_roots+0x1d/0x30 Nov 8 01:02:30 x1 kernel: [22954.038719] [<ffffffff812f8b00>] ? merge_reloc_roots+0x170/0x230 Nov 8 01:02:30 x1 kernel: [22954.038720] [<ffffffff812f8ffc>] ? relocate_block_group+0x43c/0x670 Nov 8 01:02:30 x1 kernel: [22954.038721] [<ffffffff812f93d0>] ? btrfs_relocate_block_group+0x1a0/0x280 Nov 8 01:02:30 x1 kernel: [22954.038723] [<ffffffff812d1691>] ? btrfs_relocate_chunk.isra.36+0x41/0xc0 Nov 8 01:02:30 x1 kernel: [22954.038724] [<ffffffff812d203b>] ? btrfs_shrink_device+0x19b/0x520 Nov 8 01:02:30 x1 kernel: [22954.038725] [<ffffffff812d6058>] ? btrfs_rm_device+0x2a8/0x820 Nov 8 01:02:30 x1 kernel: [22954.038727] [<ffffffff812e1869>] ? btrfs_ioctl+0x19d9/0x27b0 Nov 8 01:02:30 x1 kernel: [22954.038729] [<ffffffff811b605a>] ? filename_lookup+0xda/0x150 Nov 8 01:02:30 x1 kernel: [22954.038731] [<ffffffff810ee1ae>] ? from_kgid_munged+0xe/0x20 Nov 8 01:02:30 x1 kernel: [22954.038733] [<ffffffff811ac053>] ? cp_new_stat+0x143/0x160 Nov 8 01:02:30 x1 kernel: [22954.038734] [<ffffffff811b9943>] ? do_vfs_ioctl+0x2c3/0x490 Nov 8 01:02:30 x1 kernel: [22954.038735] [<ffffffff811ac099>] ? SYSC_newstat+0x29/0x40 Nov 8 01:02:30 x1 kernel: [22954.038737] [<ffffffff811b9b4b>] ? SyS_ioctl+0x3b/0x70 Nov 8 01:02:30 x1 kernel: [22954.038738] [<ffffffff815e53b6>] ? entry_SYSCALL_64_fastpath+0x16/0x75 Nov 8 01:02:30 x1 kernel: [22954.038739] Code: 8b 07 85 c0 74 03 31 c0 c3 ba 01 00 00 00 f0 0f b1 17 85 c0 75 f0 b0 01 c3 66 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <85> c0 75 01 c3 89 c6 e9 a4 34 ac ff 0f 1f 40 00 0f 1f 44 00 00
Signature · 4 lines
Copyright © Charles Wright 2015 All Rights Reserved. No part of this message may be reproduced, forwarded or published without Charles Wright's express written consent. Thanks for your cooperation.
Charles Wright posted on Sun, 08 Nov 2015 11:59:41 -0500 as excerpted:
Since you claim full copyright and don't allow quotes without written consent... the normal behavior of quoting and replying in context can't be done, as the implied consent when posting to a list isn't enough to meet the higher legal barrier you claim.
So, I can't quote the problem, and thus, can't reply in context...
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Hi Duncan,
I give my written permission for people on this list to quote the
problem.
That's my standard email signature that I should have probably removed in this case and I wasn't intending it to prevent a reply.
Thanks.