device delete missing panic
openSUSE 13.1 i686 8 device raid 10
when replacing a failed disk (new device is added)
~ # uname -r
3.11.6-4-pae
~ # btrfs --version
Btrfs v3.12+20131125
~ # mount -o degraded /pool
~ # journalctl | tail
Feb 06 12:22:51 store03 kernel: device label pool devid 4 transid 55050 /dev/sde Feb 06 12:22:53 store03 kernel: btrfs: allowing degraded mounts Feb 06 12:22:53 store03 kernel: btrfs: disk space caching is enabled Feb 06 12:22:53 store03 kernel: btrfs: bdev (null) errs: wr 353, rd 1, flush 17, corrupt 0, gen 0 Feb 06 12:23:16 store03 kernel: BTRFS debug (device sde): unlinked 1 orphans
~ # btrfs filesystem show /dev/disk/by-label/pool
Label: pool uuid: 3e6ba20f-a4d0-40e4-88e7-a31c4930bcfe
Total devices 9 FS bytes used 5.19TiB
devid 1 size 1.36TiB used 169.50GiB path
devid 2 size 1.82TiB used 1.62TiB path /dev/sdc
devid 3 size 931.51GiB used 931.51GiB path /dev/sdd
devid 4 size 931.51GiB used 931.51GiB path /dev/sde
devid 6 size 1.82TiB used 1.62TiB path /dev/sdg
devid 7 size 1.82TiB used 1.62TiB path /dev/sdh
devid 8 size 931.51GiB used 931.51GiB path /dev/sdi
devid 9 size 1.82TiB used 1.62TiB path /dev/sdf
devid 10 size 1.82TiB used 1.01TiB path /dev/sdb
~ # btrfs device delete missing /pool
~ # journalctl -l | tail
Feb 06 12:25:43 store03 kernel: btrfs: relocating block group 10590585618432 flags 68 ... Feb 06 12:47:23 store03 kernel: [<c025ebd2>] kthread+0x92/0xa0 Feb 06 12:47:23 store03 kernel: [<c06ece67>] ret_from_kernel_thread+0x1b/0x28 Feb 06 12:47:23 store03 kernel: [<c025eb40>] kthread_create_on_node+0xd0/0xd0 Feb 06 12:47:23 store03 kernel: DWARF2 unwinder stuck at kthread+0x0/0xa0 Feb 06 12:47:23 store03 kernel: Feb 06 12:47:23 store03 kernel: Leftover inexact backtrace: Feb 06 12:47:23 store03 kernel: ---[ end trace c47f82d03f79250d ]--- Feb 06 12:47:23 store03 kernel: ------------[ cut here ]------------ Feb 06 12:47:23 store03 kernel: WARNING: CPU: 0 PID: 3028 at /home/abuild/rpmbuild/BUILD/kernel-pae-3.11.6/linux-3.11/fs/btrfs/disk-io.c:482 btree_csum_one_bio.isra.48+0x93/0x110 [btrfs]() Feb 06 12:47:23 store03 kernel: Modules linked in: bonding hwmon_vid
btrfs raid6_pq zlib_deflate xor libcrc32c joydev hid_generic iTCO_wdt iTCO_vendor_support coretemp pcspkr serio_raw i2c_i801 ata_generic lpc_ich mfd_core usbhid mvsas libsas scsi_transport_sas e1000e ptp pps_core shpchp mperf sg dm_mod autofs4 ata_piix uhci_hcd ehci_pci ehci_hcd usbcore usb_common i915 fan thermal processor drm_kms_helper drm i2c_algo_bit button video thermal_sys scsi_dh_hp_sw scsi_dh_emc scsi_dh_rdac scsi_dh_alua scsi_dh
Feb 06 12:47:23 store03 kernel: CPU: 0 PID: 3028 Comm: btrfs-worker-2 Tainted: G W 3.11.6-4-pae #1 Feb 06 12:47:23 store03 kernel: Hardware name: PhoenixAward 945GM/945GM, BIOS 6.00 PG 08/13/2008 Feb 06 12:47:23 store03 kernel: 00000009 c06e075a 00000000 c0242c5e c085dbc8 00000000 00000bd4 f8a06e34 Feb 06 12:47:23 store03 kernel: 000001e2 f8985503 f8985503 00000002 f5c60304 f2e606d8 c14ca4f0 c0242d1b Feb 06 12:47:23 store03 kernel: 00000009 00000000 f8985503 ef93d4a0 f5c60304 e62a6c00 16c1f682 f46fe86c Feb 06 12:47:23 store03 kernel: Call Trace: Feb 06 12:47:23 store03 kernel: [<c0204ef9>] try_stack_unwind+0x179/0x190 Feb 06 12:47:23 store03 kernel: [<c0203e17>] dump_trace+0x47/0xf0 Feb 06 12:47:23 store03 kernel: [<c0204f4f>] show_trace_log_lvl+0x3f/0x50 Feb 06 12:47:23 store03 kernel: [<c0203f10>] show_stack_log_lvl+0x50/0xd0 Feb 06 12:47:23 store03 kernel: [<c0204f9f>] show_stack+0x1f/0x40 Feb 06 12:47:23 store03 kernel: [<c06e075a>] dump_stack+0x3e/0x4e Feb 06 12:47:23 store03 kernel: [<c0242c5e>] warn_slowpath_common+0x7e/0xa0 Feb 06 12:47:23 store03 kernel: [<c0242d1b>] warn_slowpath_null+0x1b/0x20 Feb 06 12:47:23 store03 kernel: [<f8985503>] btree_csum_one_bio.isra.48+0x93/0x110 [btrfs] Feb 06 12:47:23 store03 kernel: [<f898261f>] run_one_async_start+0x2f/0x40 [btrfs] Feb 06 12:47:23 store03 kernel: [<f89bdcb7>] worker_loop+0x107/0x470 [btrfs] Feb 06 12:47:23 store03 kernel: [<c025ebd2>] kthread+0x92/0xa0 Feb 06 12:47:23 store03 kernel: [<c06ece67>] ret_from_kernel_thread+0x1b/0x28 Feb 06 12:47:23 store03 kernel: [<c025eb40>] kthread_create_on_node+0xd0/0xd0 Feb 06 12:47:23 store03 kernel: DWARF2 unwinder stuck at kthread+0x0/0xa0 Feb 06 12:47:23 store03 kernel: Feb 06 12:47:23 store03 kernel: Leftover inexact backtrace: Feb 06 12:47:23 store03 kernel: ---[ end trace c47f82d03f79250e ]--- Feb 06 12:47:23 store03 kernel: ------------[ cut here ]------------
...
kernel soon locks up, any advice on how to proceed? any other info needed?
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Yep, not planning on using btrfsck unless told ;)
hmm now I can't even mount it;
~ # dmesg | tail
[ 351.083338] btrfs: failed to read chunk tree on sdg [ 351.088283] btrfs: open_ctree failed [ 392.541465] device label pool devid 7 transid 55056 /dev/sdh [ 393.558496] btrfs: allowing degraded mounts [ 393.558503] btrfs: disk space caching is enabled [ 393.605461] BTRFS critical (device sdg): unable to find logical 1563838812160 len 4096 [ 393.605479] BTRFS critical (device sdg): unable to find logical 1563838812160 len 4096 [ 393.605491] BTRFS critical (device sdg): unable to find logical 1563838812160 len 4096 [ 393.605495] btrfs: failed to read tree root on sdg [ 393.612297] btrfs: open_ctree failed
Quoted 134 lines · Christian Robert
and also "cat
/proc/mounts| fgrep /mountpoint" # will show most mount options, rw/ro status and so.and *never* try to btrfsck this filesystem (fsck.btrfs is still broken) unless an expert developper tell you to try this.
Xtian.
Quoted 125 lines · Christian Robert
ps:
always add a "btrfs file df /mountpoint" to your reporting bug, so they can figure if it is single dup raid0 raid1 raid10 raid5
raid6 and so on... or a mixupmy 2 cents.
Quoted 116 lines · Thermionix
openSUSE 13.1 i686 8 device raid 10
when replacing a failed disk (new device is added)~ # uname -r
3.11.6-4-pae~ # btrfs --version
Btrfs v3.12+20131125~ # mount -o degraded /pool
~ # journalctl | tail
Feb 06 12:22:51 store03 kernel: device label pool devid 4 transid 55050 /dev/sde Feb 06 12:22:53 store03 kernel: btrfs: allowing degraded mounts Feb 06 12:22:53 store03 kernel: btrfs: disk space caching is enabled Feb 06 12:22:53 store03 kernel: btrfs: bdev (null) errs: wr 353, rd 1, flush 17, corrupt 0, gen 0 Feb 06 12:23:16 store03 kernel: BTRFS debug (device sde): unlinked 1orphans
~ # btrfs filesystem show
/dev/disk/by-label/pool
Label: pool uuid: 3e6ba20f-a4d0-40e4-88e7-a31c4930bcfe
Total devices 9 FS bytes used 5.19TiB
devid 1 size 1.36TiB used 169.50GiB path
devid 2 size 1.82TiB used 1.62TiB path/dev/sdc
devid 3 size 931.51GiB used 931.51GiB path/dev/sdd
devid 4 size 931.51GiB used 931.51GiB path/dev/sde
devid 6 size 1.82TiB used 1.62TiB path/dev/sdg
devid 7 size 1.82TiB used 1.62TiB path/dev/sdh
devid 8 size 931.51GiB used 931.51GiB path/dev/sdi
devid 9 size 1.82TiB used 1.62TiB path/dev/sdf
devid 10 size 1.82TiB used 1.01TiB path/dev/sdb~ # btrfs device delete missing /pool
~ # journalctl -l | tail
Feb 06 12:25:43 store03 kernel: btrfs: relocating block group 10590585618432 flags 68 ... Feb 06 12:47:23 store03 kernel: [<c025ebd2>] kthread+0x92/0xa0 Feb 06 12:47:23 store03 kernel: [<c06ece67>] ret_from_kernel_thread+0x1b/0x28 Feb 06 12:47:23 store03 kernel: [<c025eb40>] kthread_create_on_node+0xd0/0xd0 Feb 06 12:47:23 store03 kernel: DWARF2 unwinder stuck at kthread+0x0/0xa0 Feb 06 12:47:23 store03 kernel: Feb 06 12:47:23 store03 kernel: Leftover inexact backtrace: Feb 06 12:47:23 store03 kernel: ---[ end trace c47f82d03f79250d ]--- Feb 06 12:47:23 store03 kernel: ------------[ cut here ]------------ Feb 06 12:47:23 store03 kernel: WARNING: CPU: 0 PID: 3028 at
/home/abuild/rpmbuild/BUILD/kernel-pae-3.11.6/linux-3.11/fs/btrfs/disk-io.c:482btree_csum_one_bio.isra.48+0x93/0x110 [btrfs]()
Feb 06 12:47:23 store03 kernel: Modules linked in: bonding hwmon_vidbtrfs
raid6_pqzlib_deflatexor libcrc32c joydevhid_genericiTCO_wdtiTCO_vendor_supportcoretemp pcspkrserio_rawi2c_i801ata_genericlpc_ichmfd_coreusbhid mvsas libsasscsi_transport_sase1000e ptppps_coreshpchp mperf sgdm_modautofs4ata_piixuhci_hcdehci_pciehci_hcdusbcoreusb_commoni915 fan thermal processordrm_kms_helperdrmi2c_algo_bitbutton videothermal_sysscsi_dh_hp_swscsi_dh_emcscsi_dh_rdacscsi_dh_aluascsi_dhFeb 06 12:47:23 store03 kernel: CPU: 0 PID: 3028 Comm: btrfs-worker-2 Tainted: G W 3.11.6-4-pae #1 Feb 06 12:47:23 store03 kernel: Hardware name: PhoenixAward 945GM/945GM, BIOS 6.00 PG 08/13/2008 Feb 06 12:47:23 store03 kernel: 00000009 c06e075a 00000000 c0242c5e c085dbc8 00000000 00000bd4 f8a06e34 Feb 06 12:47:23 store03 kernel: 000001e2 f8985503 f8985503 00000002 f5c60304 f2e606d8 c14ca4f0 c0242d1b Feb 06 12:47:23 store03 kernel: 00000009 00000000 f8985503 ef93d4a0 f5c60304 e62a6c00 16c1f682 f46fe86c Feb 06 12:47:23 store03 kernel: Call Trace: Feb 06 12:47:23 store03 kernel: [<c0204ef9>] try_stack_unwind+0x179/0x190 Feb 06 12:47:23 store03 kernel: [<c0203e17>] dump_trace+0x47/0xf0 Feb 06 12:47:23 store03 kernel: [<c0204f4f>] show_trace_log_lvl+0x3f/0x50 Feb 06 12:47:23 store03 kernel: [<c0203f10>] show_stack_log_lvl+0x50/0xd0 Feb 06 12:47:23 store03 kernel: [<c0204f9f>] show_stack+0x1f/0x40 Feb 06 12:47:23 store03 kernel: [<c06e075a>] dump_stack+0x3e/0x4e Feb 06 12:47:23 store03 kernel: [<c0242c5e>] warn_slowpath_common+0x7e/0xa0 Feb 06 12:47:23 store03 kernel: [<c0242d1b>] warn_slowpath_null+0x1b/0x20 Feb 06 12:47:23 store03 kernel: [<f8985503>] btree_csum_one_bio.isra.48+0x93/0x110 [btrfs] Feb 06 12:47:23 store03 kernel: [<f898261f>] run_one_async_start+0x2f/0x40 [btrfs] Feb 06 12:47:23 store03 kernel: [<f89bdcb7>] worker_loop+0x107/0x470 [btrfs] Feb 06 12:47:23 store03 kernel: [<c025ebd2>] kthread+0x92/0xa0 Feb 06 12:47:23 store03 kernel: [<c06ece67>] ret_from_kernel_thread+0x1b/0x28 Feb 06 12:47:23 store03 kernel: [<c025eb40>] kthread_create_on_node+0xd0/0xd0 Feb 06 12:47:23 store03 kernel: DWARF2 unwinder stuck at kthread+0x0/0xa0 Feb 06 12:47:23 store03 kernel: Feb 06 12:47:23 store03 kernel: Leftover inexact backtrace: Feb 06 12:47:23 store03 kernel: ---[ end trace c47f82d03f79250e ]--- Feb 06 12:47:23 store03 kernel: ------------[ cut here ]------------...
kernel soon locks up, any advice on how to proceed? any other info needed?
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
your test case is same as in the patch below
and the panic was due to null bdev (which matches
in your logs).
[RFC PATCH] btrfs: fix null pointer deference atbtrfs_sysfs_add_one+0x105
But in your logs below, there isn't a panic right ? wrong cut and paste ? or what did I miss?
Thanks, Anand
Quoted 106 lines · Thermionix
openSUSE 13.1 i686 8 device raid 10
when replacing a failed disk (new device is added)~ # uname -r
3.11.6-4-pae~ # btrfs --version
Btrfs v3.12+20131125~ # mount -o degraded /pool
~ # journalctl | tail
Feb 06 12:22:51 store03 kernel: device label pool devid 4 transid 55050 /dev/sde Feb 06 12:22:53 store03 kernel: btrfs: allowing degraded mounts Feb 06 12:22:53 store03 kernel: btrfs: disk space caching is enabled Feb 06 12:22:53 store03 kernel: btrfs: bdev (null) errs: wr 353, rd 1, flush 17, corrupt 0, gen 0 Feb 06 12:23:16 store03 kernel: BTRFS debug (device sde): unlinked 1 orphans~ # btrfs filesystem show
/dev/disk/by-label/pool
Label: pool uuid: 3e6ba20f-a4d0-40e4-88e7-a31c4930bcfe
Total devices 9 FS bytes used 5.19TiB
devid 1 size 1.36TiB used 169.50GiB path
devid 2 size 1.82TiB used 1.62TiB path/dev/sdc
devid 3 size 931.51GiB used 931.51GiB path/dev/sdd
devid 4 size 931.51GiB used 931.51GiB path/dev/sde
devid 6 size 1.82TiB used 1.62TiB path/dev/sdg
devid 7 size 1.82TiB used 1.62TiB path/dev/sdh
devid 8 size 931.51GiB used 931.51GiB path/dev/sdi
devid 9 size 1.82TiB used 1.62TiB path/dev/sdf
devid 10 size 1.82TiB used 1.01TiB path/dev/sdb~ # btrfs device delete missing /pool
~ # journalctl -l | tail
Feb 06 12:25:43 store03 kernel: btrfs: relocating block group 10590585618432 flags 68 ... Feb 06 12:47:23 store03 kernel: [<c025ebd2>] kthread+0x92/0xa0 Feb 06 12:47:23 store03 kernel: [<c06ece67>] ret_from_kernel_thread+0x1b/0x28 Feb 06 12:47:23 store03 kernel: [<c025eb40>] kthread_create_on_node+0xd0/0xd0 Feb 06 12:47:23 store03 kernel: DWARF2 unwinder stuck at kthread+0x0/0xa0 Feb 06 12:47:23 store03 kernel: Feb 06 12:47:23 store03 kernel: Leftover inexact backtrace: Feb 06 12:47:23 store03 kernel: ---[ end trace c47f82d03f79250d ]--- Feb 06 12:47:23 store03 kernel: ------------[ cut here ]------------ Feb 06 12:47:23 store03 kernel: WARNING: CPU: 0 PID: 3028 at /home/abuild/rpmbuild/BUILD/kernel-pae-3.11.6/linux-3.11/fs/btrfs/disk-io.c:482 btree_csum_one_bio.isra.48+0x93/0x110 [btrfs]() Feb 06 12:47:23 store03 kernel: Modules linked in: bonding hwmon_vidbtrfs
raid6_pqzlib_deflatexor libcrc32c joydevhid_genericiTCO_wdtiTCO_vendor_supportcoretemp pcspkrserio_rawi2c_i801ata_genericlpc_ichmfd_coreusbhid mvsas libsasscsi_transport_sase1000e ptppps_coreshpchp mperf sgdm_modautofs4ata_piixuhci_hcdehci_pciehci_hcdusbcoreusb_commoni915 fan thermal processordrm_kms_helperdrmi2c_algo_bitbutton videothermal_sysscsi_dh_hp_swscsi_dh_emcscsi_dh_rdacscsi_dh_aluascsi_dhFeb 06 12:47:23 store03 kernel: CPU: 0 PID: 3028 Comm: btrfs-worker-2 Tainted: G W 3.11.6-4-pae #1 Feb 06 12:47:23 store03 kernel: Hardware name: PhoenixAward 945GM/945GM, BIOS 6.00 PG 08/13/2008 Feb 06 12:47:23 store03 kernel: 00000009 c06e075a 00000000 c0242c5e c085dbc8 00000000 00000bd4 f8a06e34 Feb 06 12:47:23 store03 kernel: 000001e2 f8985503 f8985503 00000002 f5c60304 f2e606d8 c14ca4f0 c0242d1b Feb 06 12:47:23 store03 kernel: 00000009 00000000 f8985503 ef93d4a0 f5c60304 e62a6c00 16c1f682 f46fe86c Feb 06 12:47:23 store03 kernel: Call Trace: Feb 06 12:47:23 store03 kernel: [<c0204ef9>] try_stack_unwind+0x179/0x190 Feb 06 12:47:23 store03 kernel: [<c0203e17>] dump_trace+0x47/0xf0 Feb 06 12:47:23 store03 kernel: [<c0204f4f>] show_trace_log_lvl+0x3f/0x50 Feb 06 12:47:23 store03 kernel: [<c0203f10>] show_stack_log_lvl+0x50/0xd0 Feb 06 12:47:23 store03 kernel: [<c0204f9f>] show_stack+0x1f/0x40 Feb 06 12:47:23 store03 kernel: [<c06e075a>] dump_stack+0x3e/0x4e Feb 06 12:47:23 store03 kernel: [<c0242c5e>] warn_slowpath_common+0x7e/0xa0 Feb 06 12:47:23 store03 kernel: [<c0242d1b>] warn_slowpath_null+0x1b/0x20 Feb 06 12:47:23 store03 kernel: [<f8985503>] btree_csum_one_bio.isra.48+0x93/0x110 [btrfs] Feb 06 12:47:23 store03 kernel: [<f898261f>] run_one_async_start+0x2f/0x40 [btrfs] Feb 06 12:47:23 store03 kernel: [<f89bdcb7>] worker_loop+0x107/0x470 [btrfs] Feb 06 12:47:23 store03 kernel: [<c025ebd2>] kthread+0x92/0xa0 Feb 06 12:47:23 store03 kernel: [<c06ece67>] ret_from_kernel_thread+0x1b/0x28 Feb 06 12:47:23 store03 kernel: [<c025eb40>] kthread_create_on_node+0xd0/0xd0 Feb 06 12:47:23 store03 kernel: DWARF2 unwinder stuck at kthread+0x0/0xa0 Feb 06 12:47:23 store03 kernel: Feb 06 12:47:23 store03 kernel: Leftover inexact backtrace: Feb 06 12:47:23 store03 kernel: ---[ end trace c47f82d03f79250e ]--- Feb 06 12:47:23 store03 kernel: ------------[ cut here ]------------...
kernel soon locks up, any advice on how to proceed?
any other info needed?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
those are the last useful log outputs before the server locks up
digging in /var/log/messages - you can see it stopped logging at 12:47, and I hard reset at 3:07
maybe I should have specified hard-lock-up instead of panic
2014-02-06T12:47:47.590784+11:00 store03 kernel: [ 4619.769346] ------------[ cut here ]------------ 2014-02-06T12:47:47.590785+11:00 store03 kernel: [ 4619.769369] WARNING: CPU: 0 PID: 3005 at /home/abuild/rpmbuild/BUILD/kernel-pae-3.11.6/l inux-3.11/fs/btrfs/disk-io.c:482 btree_csum_one_bio.isra.48+0x93/0x110 [btrfs]() 2014-02-06T12:47:47.590893+11:00 store03 kernel: [ 4619.769399] Modules linked in: bonding hwmon_vid btrfs raid6_pq zlib_deflate xor libcrc32c joydev hid_generic iTCO_wdt iTCO_vendor_support coretemp pcspkr serio_raw i2c_i801 ata_generic lpc_ich mfd_core usbhid mvsas libsas scsi_transport_sas e1000e ptp pps_core shpchp mperf sg dm_mod autofs4 ata_piix uhci_hcd ehci_pci ehci_hcd usbcore usb_common i915 fan thermal processor drm_kms_helper drm i2c_algo_bit button video thermal_sys scsi_dh_hp_sw scsi_dh_emc scsi_dh_rdac scsi_dh_alua scsi_dh 2014-02-06T12:47:47.590896+11:00 store03 kernel: [ 4619.769402] CPU: 0 PID: 3005 Comm: btrfs-worker-1 Tainted: G W 3.11.6-4-pae #1 2014-02-06T12:47:47.590898+11:00 store03 kernel: [ 4619.769403] Hardware name: PhoenixAward 945GM/945GM, BIOS 6.00 PG 08/13/2008 2014-02-06T12:47:47.590899+11:00 store03 kernel: [ 4619.769407] 00000009 c06e075a 00000000 c0242c5e c085dbc8 00000000 00000bbd f8a06e34 2014-02-06T12:47:47.590901+11:00 store03 kernel: [ 4619.769411] 000001e2 f8985503 f8985503 0000000d f5abaa5c d7c20a5c f1c97070 c0242d1b 2014-02-06T12:47:47.590903+11:00 store03 kernel: [ 4619.769415] 00000009 00000000 f8985503 e85afce0 f5abaa5c e62a6c00 16c205ed c6b5476c 2014-02-06T12:47:47.590905+11:00 store03 kernel: [ 4619.769415] Call Trace: 2014-02-06T12:47:47.590906+11:00 store03 kernel: [ 4619.769424] [<c0204ef9>] try_stack_unwind+0x179/0x190 2014-02-06T12:47:47.590908+11:00 store03 kernel: [ 4619.769430] [<c0203e17>] dump_trace+0x47/0xf0 2014-02-06T12:47:47.590910+11:00 store03 kernel: [ 4619.769434] [<c0204f4f>] show_trace_log_lvl+0x3f/0x50 2014-02-06T12:47:47.590911+11:00 store03 kernel: [ 4619.769437] [<c0203f10>] show_stack_log_lvl+0x50/0xd0 2014-02-06T12:47:47.590913+11:00 store03 kernel: [ 4619.769441] [<c0204f9f>] show_stack+0x1f/0x40 2014-02-06T12:47:47.590915+11:00 store03 kernel: [ 4619.769445] [<c06e075a>] dump_stack+0x3e/0x4e 2014-02-06T12:47:47.590917+11:00 store03 kernel: [ 4619.769450] [<c0242c5e>] warn_slowpath_common+0x7e/0xa0 2014-02-06T12:47:47.590918+11:00 store03 kernel: [ 4619.769454] [<c0242d1b>] warn_slowpath_null+0x1b/0x20 2014-02-06T12:47:47.590920+11:00 store03 kernel: [ 4619.769472] [<f8985503>] btree_csum_one_bio.isra.48+0x93/0x110 [btrfs] 2014-02-06T12:47:47.590922+11:00 store03 kernel: [ 4619.769555] [<f898261f>] run_one_async_start+0x2f/0x40 [btrfs] 2014-02-06T12:47:47.590924+11:00 store03 kernel: [ 4619.769630] [<f89bdcb7>] worker_loop+0x107/0x470 [btrfs] ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@2014-02-06T15:07:05.120258+11:00 store03 rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="418" x-info="http://www.rsyslog.com"] start 2014-02-06T15:07:05.127408+11:00 store03 kernel: [ 0.000000] Initializing cgroup subsys cpuset
as I now can't mount (open_ctree failed)
Should I be mounting with -o recovery ?
Quoted 132 lines · Anand Jain
your test case is same as in the patch below
and the panic was due to null bdev (which matches
in your logs).[RFC PATCH] btrfs: fix null pointer deference at
btrfs_sysfs_add_one+0x105But in your logs below, there isn't a panic right ? wrong cut and paste ? or what did I miss?
Thanks, Anand
Quoted 113 lines · Thermionix
openSUSE 13.1 i686 8 device raid 10
when replacing a failed disk (new device is added)~ # uname -r
3.11.6-4-pae~ # btrfs --version
Btrfs v3.12+20131125~ # mount -o degraded /pool
~ # journalctl | tail
Feb 06 12:22:51 store03 kernel: device label pool devid 4 transid 55050 /dev/sde Feb 06 12:22:53 store03 kernel: btrfs: allowing degraded mounts Feb 06 12:22:53 store03 kernel: btrfs: disk space caching is enabled Feb 06 12:22:53 store03 kernel: btrfs: bdev (null) errs: wr 353, rd 1, flush 17, corrupt 0, gen 0 Feb 06 12:23:16 store03 kernel: BTRFS debug (device sde): unlinked 1orphans
~ # btrfs filesystem show
/dev/disk/by-label/pool
Label: pool uuid: 3e6ba20f-a4d0-40e4-88e7-a31c4930bcfe
Total devices 9 FS bytes used 5.19TiB
devid 1 size 1.36TiB used 169.50GiB path
devid 2 size 1.82TiB used 1.62TiB path/dev/sdc
devid 3 size 931.51GiB used 931.51GiB path/dev/sdd
devid 4 size 931.51GiB used 931.51GiB path/dev/sde
devid 6 size 1.82TiB used 1.62TiB path/dev/sdg
devid 7 size 1.82TiB used 1.62TiB path/dev/sdh
devid 8 size 931.51GiB used 931.51GiB path/dev/sdi
devid 9 size 1.82TiB used 1.62TiB path/dev/sdf
devid 10 size 1.82TiB used 1.01TiB path/dev/sdb~ # btrfs device delete missing /pool
~ # journalctl -l | tail
Feb 06 12:25:43 store03 kernel: btrfs: relocating block group 10590585618432 flags 68 ... Feb 06 12:47:23 store03 kernel: [<c025ebd2>] kthread+0x92/0xa0 Feb 06 12:47:23 store03 kernel: [<c06ece67>] ret_from_kernel_thread+0x1b/0x28 Feb 06 12:47:23 store03 kernel: [<c025eb40>] kthread_create_on_node+0xd0/0xd0 Feb 06 12:47:23 store03 kernel: DWARF2 unwinder stuck at kthread+0x0/0xa0 Feb 06 12:47:23 store03 kernel: Feb 06 12:47:23 store03 kernel: Leftover inexact backtrace: Feb 06 12:47:23 store03 kernel: ---[ end trace c47f82d03f79250d ]--- Feb 06 12:47:23 store03 kernel: ------------[ cut here ]------------ Feb 06 12:47:23 store03 kernel: WARNING: CPU: 0 PID: 3028 at
/home/abuild/rpmbuild/BUILD/kernel-pae-3.11.6/linux-3.11/fs/btrfs/disk-io.c:482btree_csum_one_bio.isra.48+0x93/0x110 [btrfs]()
Feb 06 12:47:23 store03 kernel: Modules linked in: bonding hwmon_vidbtrfs
raid6_pqzlib_deflatexor libcrc32c joydevhid_genericiTCO_wdtiTCO_vendor_supportcoretemp pcspkrserio_rawi2c_i801ata_genericlpc_ichmfd_coreusbhid mvsas libsasscsi_transport_sase1000e ptppps_coreshpchp mperf sgdm_modautofs4ata_piixuhci_hcdehci_pciehci_hcdusbcoreusb_commoni915 fan thermal processordrm_kms_helperdrmi2c_algo_bitbutton videothermal_sysscsi_dh_hp_swscsi_dh_emcscsi_dh_rdacscsi_dh_aluascsi_dhFeb 06 12:47:23 store03 kernel: CPU: 0 PID: 3028 Comm: btrfs-worker-2 Tainted: G W 3.11.6-4-pae #1 Feb 06 12:47:23 store03 kernel: Hardware name: PhoenixAward 945GM/945GM, BIOS 6.00 PG 08/13/2008 Feb 06 12:47:23 store03 kernel: 00000009 c06e075a 00000000 c0242c5e c085dbc8 00000000 00000bd4 f8a06e34 Feb 06 12:47:23 store03 kernel: 000001e2 f8985503 f8985503 00000002 f5c60304 f2e606d8 c14ca4f0 c0242d1b Feb 06 12:47:23 store03 kernel: 00000009 00000000 f8985503 ef93d4a0 f5c60304 e62a6c00 16c1f682 f46fe86c Feb 06 12:47:23 store03 kernel: Call Trace: Feb 06 12:47:23 store03 kernel: [<c0204ef9>] try_stack_unwind+0x179/0x190 Feb 06 12:47:23 store03 kernel: [<c0203e17>] dump_trace+0x47/0xf0 Feb 06 12:47:23 store03 kernel: [<c0204f4f>] show_trace_log_lvl+0x3f/0x50 Feb 06 12:47:23 store03 kernel: [<c0203f10>] show_stack_log_lvl+0x50/0xd0 Feb 06 12:47:23 store03 kernel: [<c0204f9f>] show_stack+0x1f/0x40 Feb 06 12:47:23 store03 kernel: [<c06e075a>] dump_stack+0x3e/0x4e Feb 06 12:47:23 store03 kernel: [<c0242c5e>] warn_slowpath_common+0x7e/0xa0 Feb 06 12:47:23 store03 kernel: [<c0242d1b>] warn_slowpath_null+0x1b/0x20 Feb 06 12:47:23 store03 kernel: [<f8985503>] btree_csum_one_bio.isra.48+0x93/0x110 [btrfs] Feb 06 12:47:23 store03 kernel: [<f898261f>] run_one_async_start+0x2f/0x40 [btrfs] Feb 06 12:47:23 store03 kernel: [<f89bdcb7>] worker_loop+0x107/0x470 [btrfs] Feb 06 12:47:23 store03 kernel: [<c025ebd2>] kthread+0x92/0xa0 Feb 06 12:47:23 store03 kernel: [<c06ece67>] ret_from_kernel_thread+0x1b/0x28 Feb 06 12:47:23 store03 kernel: [<c025eb40>] kthread_create_on_node+0xd0/0xd0 Feb 06 12:47:23 store03 kernel: DWARF2 unwinder stuck at kthread+0x0/0xa0 Feb 06 12:47:23 store03 kernel: Feb 06 12:47:23 store03 kernel: Leftover inexact backtrace: Feb 06 12:47:23 store03 kernel: ---[ end trace c47f82d03f79250e ]--- Feb 06 12:47:23 store03 kernel: ------------[ cut here ]------------...
kernel soon locks up, any advice on how to proceed? any other info needed?
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines
as I now can't mount (
open_ctreefailed)
at around open_ctree failed message should help.
Thanks, Anand
~ # dmesg | tail
[ 351.083338] btrfs: failed to read chunk tree on sdg [ 351.088283] btrfs: open_ctree failed [ 392.541465] device label pool devid 7 transid 55056 /dev/sdh [ 393.558496] btrfs: allowing degraded mounts [ 393.558503] btrfs: disk space caching is enabled [ 393.605461] BTRFS critical (device sdg): unable to find logical 1563838812160 len 4096 [ 393.605479] BTRFS critical (device sdg): unable to find logical 1563838812160 len 4096 [ 393.605491] BTRFS critical (device sdg): unable to find logical 1563838812160 len 4096 [ 393.605495] btrfs: failed to read tree root on sdg [ 393.612297] btrfs: open_ctree failed
Quoted 8 lines · Anand Jain
Quoted 4 lines
as I now can't mount (
open_ctreefailed)at around
open_ctreefailed message should help.Thanks, Anand
Quoted 1 line · Thermionix
CPU: 0 PID: 3005 Comm: btrfs-worker-1 Tainted: G W 3.11.6-4-pae #1
Something happened even before the kernel messages you posted, as this kernel is already tainted. I would honestly try to get a prebuilt 3.14-rc1 kernel from your distro's dev branch, or build from mainline. There are a large pile of bug fixes from just 3.13, let alone 3.11.6.
And also make sure you're at btrfs-progs 3.12. If the kernel alone isn't able to work around it with a normal mount, or -o recovery, or -o recovery,degraded, then your next step is probably btrfs-zero-log based on the btrfs: failed to read tree root on sdg.
Let's see if others concur.
Chris Murphy
~ # uname -r
3.13.1-12.gfc9498b-pae
~ # btrfs --version
Btrfs v3.12+20131125
~ # mount -o degraded,recovery /pool
Killed
Feb 07 09:53:44 store03 kernel: btrfs: device label pool devid 4 transid 55056 /dev/sde Feb 07 09:53:44 store03 kernel: btrfs: allowing degraded mounts Feb 07 09:53:44 store03 kernel: btrfs: enabling auto recovery Feb 07 09:53:44 store03 kernel: btrfs: disk space caching is enabled Feb 07 09:53:44 store03 kernel: BTRFS critical (device sde): unable to find logical 1563803254784 len 4096 Feb 07 09:53:44 store03 kernel: BTRFS critical (device sde): unable to find logical 1563803254784 len 4096 Feb 07 09:53:44 store03 kernel: BTRFS critical (device sde): unable to find logical 1563803254784 len 4096 Feb 07 09:53:44 store03 kernel: btrfs: failed to read tree root on sde Feb 07 09:53:44 store03 kernel: BTRFS critical (device sde): unable to find logical 1563803254784 len 4096 Feb 07 09:53:44 store03 kernel: BTRFS critical (device sde): unable to find logical 1563803254784 len 4096 Feb 07 09:53:44 store03 kernel: BTRFS critical (device sde): unable to find logical 1563803254784 len 4096 Feb 07 09:53:44 store03 kernel: btrfs: failed to read tree root on sde Feb 07 09:53:44 store03 kernel: btrfs: bdev (null) errs: wr 353, rd 1, flush 17, corrupt 0, gen 0 Feb 07 09:53:44 store03 kernel: BUG: unable to handle kernel NULL pointer dereference at 00000050 Feb 07 09:53:44 store03 kernel: IP: [<f834b7b7>] btrfs_sysfs_add_one+0x1a7/0x280 [btrfs] Feb 07 09:53:44 store03 kernel: *pdpt = 0000000034fce001 *pde = 0000000000000000 Feb 07 09:53:44 store03 kernel: Oops: 0000 [#1] SMP Feb 07 09:53:44 store03 kernel: Modules linked in: bonding hwmon_vid iTCO_wdt joydev iTCO_vendor_support hid_generic serio_raw lpc_ich pcspk r coretemp mfd_core i2c_i801 ata_generic e1000e usbhid ptp pps_core mvsas libsas scsi_transport_sas shpchp sg dm_mod autofs4 btrfs raid6_pq xor libcrc32c ata_piix uhci_hcd ehci_pci ehci_hcd usbcore usb_common fan thermal processor i915 drm_kms_helper drm i2c_algo_bit button video thermal_sys scsi_dh_hp_sw scsi_dh_emc scsi_dh_rdac scsi_dh_alua scsi_dh Feb 07 09:53:44 store03 kernel: CPU: 0 PID: 1893 Comm: mount Not tainted 3.13.1-12.gfc9498b-pae #1 Feb 07 09:53:44 store03 kernel: Hardware name: PhoenixAward 945GM/945GM, BIOS 6.00 PG 08/13/2008 Feb 07 09:53:44 store03 kernel: task: f4fa80d0 ti: f1614000 task.ti: f1614000 Feb 07 09:53:44 store03 kernel: EIP: 0060:[<f834b7b7>] EFLAGS: 00010212 CPU: 0 Feb 07 09:53:44 store03 kernel: EIP is at btrfs_sysfs_add_one+0x1a7/0x280 [btrfs] Feb 07 09:53:44 store03 kernel: EAX: f4f8eec0 EBX: f6c0e800 ECX: f14063c0 EDX: 00000000 Feb 07 09:53:44 store03 kernel: ESI: f3454374 EDI: 00000000 EBP: f154e000 ESP: f1615d38 Feb 07 09:53:44 store03 kernel: DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 Feb 07 09:53:44 store03 kernel: CR0: 8005003b CR2: 00000050 CR3: 31470000 CR4: 000007f0 Feb 07 09:53:44 store03 kernel: Stack: Feb 07 09:53:44 store03 kernel: f154ea00 f83d10a0 00000000 f83aee16 f154e000 00000000 00000000 f154e000 Feb 07 09:53:44 store03 kernel: f83d3a60 f154ea00 f83aee1a 00000000 00000000 00000000 f6dbf800 f154e000 Feb 07 09:53:44 store03 kernel: f4fe3000 f1615e48 f832f099 00001000 0000d70f 00000000 00000003 00000000 Feb 07 09:53:44 store03 kernel: Call Trace: Feb 07 09:53:44 store03 kernel: [<f832f099>] open_ctree+0x1409/0x1d80 [btrfs] Feb 07 09:53:44 store03 kernel: [<f830675b>] btrfs_mount+0x6ab/0x880 [btrfs] Feb 07 09:53:44 store03 kernel: [<c035f1ad>] mount_fs+0x2d/0x170 Feb 07 09:53:44 store03 kernel: [<c0375af5>] vfs_kern_mount+0x45/0xd0 Feb 07 09:53:44 store03 kernel: [<c0377a74>] do_mount+0x1f4/0x8e0 Feb 07 09:53:44 store03 kernel: [<c03783fe>] SyS_mount+0x7e/0xc0 Feb 07 09:53:44 store03 kernel: [<c070977d>] sysenter_do_call+0x12/0x28 Feb 07 09:53:44 store03 kernel: [<b76ed424>] 0xb76ed423 Feb 07 09:53:44 store03 kernel: Code: 0c 8d 76 00 89 e8 e8 59 fd ff ff 89 f8 83 c4 38 5b 5e 5f 5d c3 8d b4 26 00 00 00 00 8b 1b 39 de 74 b1 8b 85 28 0a 00 00 8b 53 50 <8b> 52 50 8b 4a 28 83 c2 28 e8 7b fe 06 c8 85 c0 74 df 89 c7 eb Feb 07 09:53:44 store03 kernel: EIP: [<f834b7b7>] btrfs_sysfs_add_one+0x1a7/0x280 [btrfs] SS:ESP 0068:f1615d38 Feb 07 09:53:44 store03 kernel: CR2: 0000000000000050 Feb 07 09:53:44 store03 kernel: ---[ end trace 4b34a92ed1b54297 ]---
~ # btrfs-zero-log /dev/sde
warning, device 1 is missing
warning devid 1 not found already
checksum verify failed on 19155989299200 found 42FA71B2 wanted A7E1A39B checksum verify failed on 19155989299200 found 42FA71B2 wanted A7E1A39B checksum verify failed on 19155989299200 found 42FA71B2 wanted A7E1A39B checksum verify failed on 19155989299200 found 42FA71B2 wanted A7E1A39B Csum didn't match
Couldn't read tree root
~ # mount -o degraded,recovery /pool
Feb 07 10:04:55 store03 kernel: btrfs: device label pool devid 2 transid 55056 /dev/sdc Feb 07 10:04:55 store03 kernel: btrfs: device label pool devid 4 transid 55056 /dev/sde Feb 07 10:05:09 store03 kernel: btrfs: device label pool devid 4 transid 55056 /dev/sde Feb 07 10:06:25 store03 systemd[1]: pool.mount mounting timed out. Stopping. Feb 07 10:06:34 store03 systemd[1]: Starting Cleanup of Temporary Directories... Feb 07 10:06:34 store03 systemd[1]: Started Cleanup of Temporary Directories. Feb 07 10:07:55 store03 systemd[1]: pool.mount mounting timed out. Killing. Feb 07 10:09:25 store03 systemd[1]: pool.mount mount process still around after SIGKILL. Ignoring. Feb 07 10:09:25 store03 systemd[1]: Failed to mount /pool. Feb 07 10:09:25 store03 systemd[1]: Unit pool.mount entered failed state.
Quoted 15 lines · Chris Murphy
Quoted 1 line · Thermionix
CPU: 0 PID: 3005 Comm: btrfs-worker-1 Tainted: G W 3.11.6-4-pae #1Something happened even before the kernel messages you posted, as this kernel is already tainted. I would honestly try to get a prebuilt 3.14-rc1 kernel from your distro's dev branch, or build from mainline. There are a large pile of bug fixes from just 3.13, let alone 3.11.6.
And also make sure you're at btrfs-progs 3.12. If the kernel alone isn't able to work around it with a normal mount, or -o recovery, or -o recovery,degraded, then your next step is probably btrfs-zero-log based on the btrfs: failed to read tree root on sdg.
Let's see if others concur.
Chris Murphy
Quoted 8 lines
~ # mount -o degraded,recovery /pool
Killed
Feb 07 09:53:44 store03 kernel: BUG: unable to handle kernel NULL pointer dereference at 00000050 Feb 07 09:53:44 store03 kernel: IP: [<f834b7b7>] btrfs_sysfs_add_one+0x1a7/0x280 [btrfs]
Mostly the above Bug is fixed in the patch below..
[RFC PATCH] btrfs: fix null pointer deference atbtrfs_sysfs_add_one+0x105
If you are hitting the same panic again, could you test it ?
Thanks, Anand