RAID1: system stability

Hi list,
I'm regular on this list and I very like btrfs, I want use it on production server, and I want replace hw raid on it.

Test case: server with N scsi discs
2 SAS disks used for raid 1 root fs
If I just remove one disk physically, all okay, kernel show me write errors and system continue work some time. But after first sync call, example

# sync
# dd if=/Dev/zero of=/zero

Kernel will crush and system freeze.
Yes, after reboot, I can mount with degraded and recovery options, and I can add failed disk again, and btrfs will rebuild array. But kernel crush and reboot expected in this case, or I can skip it? How?

# mount -o remount, degraded -> kernel crush
Insert failed disk again -> kernel crush

May be I missing something? I just want avoid shutdown time or/and reboot =.=���{.n�+�������+%�����ݶ��w��{.n�+����{���k~���^n�r���z���h����&���z��z�ޗ�+��+zf���h���~����i��������z_��j:+v���)ߣ�m

Oh, i missing, i've test it on 3.19+ kernels
I can get trace from screen if it interesting for developers.

2015-05-26 14:23 GMT+03:00 Timofey Titovets <[email protected]>:

Have a nice day,
Timofey.

Without a complete dmesg it's hard to say what's going on. The call trace alone probably don't show the instigating factor so you may need to use remote ssh with journalctl -f, or use netconsole to continuously get kernel messages prior to the implosion.

Chris Murphy

Oh, thanks for advice, i'll get and attach it. i.e. as i understand behaviour like it, not expected, cool

2015-05-26 22:49 GMT+03:00 Chris Murphy <[email protected]>:

Have a nice day,
Timofey.

Okay, logs, i did release disk /dev/sde1 and get:

Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: Read(10): 28 00 11 1d 69
00 00 00 08 00
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: blk_update_request: I/O
error, dev sde, sector 287140096
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: mptbase: ioc0:
LogInfo(0x31010011): Originator={PL}, Code={Open Failure},
SubCode(0x0011) cb_idx mptscsih_io_done
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: sd 0:0:5:0: [sde] FAILED
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: sd 0:0:5:0: [sde] CDB:
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: Read(10): 28 00 11 1d 69
00 00 00 08 00
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: blk_update_request: I/O
error, dev sde, sector 287140096
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: Buffer I/O error on dev
sde1, logical block 35892256, async page read
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: mptbase: ioc0:
LogInfo(0x31010011): Originator={PL}, Code={Open Failure},
SubCode(0x0011) cb_idx mptscsih_io_done
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: mptbase: ioc0:
LogInfo(0x31010011): Originator={PL}, Code={Open Failure},
SubCode(0x0011) cb_idx mptscsih_io_done
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: sd 0:0:5:0: [sde] FAILED
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: sd 0:0:5:0: [sde] CDB:
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: Read(10): 28 00 11 1d 69
00 00 00 08 00
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: blk_update_request: I/O
error, dev sde, sector 287140096
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: Buffer I/O error on dev
sde1, logical block 35892256, async page read
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: mptbase: ioc0:
LogInfo(0x31010011): Originator={PL}, Code={Open Failure},
SubCode(0x0011) cb_idx mptscsih_io_done
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: mptbase: ioc0:
LogInfo(0x31010011): Originator={PL}, Code={Open Failure},
SubCode(0x0011) cb_idx mptscsih_io_done
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: sd 0:0:5:0: [sde] FAILED
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: sd 0:0:5:0: [sde] CDB:
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: Read(10): 28 00 11 1d 69
00 00 00 08 00
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: blk_update_request: I/O
error, dev sde, sector 287140096
Jun 22 14:28:40 srv-lab-ceph-node-01 kernel: Buffer I/O error on dev
sde1, logical block 35892256, async page read
Jun 22 14:28:41 srv-lab-ceph-node-01 kernel: mptbase: ioc0:
LogInfo(0x31010011): Originator={PL}, Code={Open Failure},
SubCode(0x0011) cb_idx mptscsih_io_done
Jun 22 14:28:41 srv-lab-ceph-node-01 kernel:  end_device-0:0:6:
mptsas: ioc0: removing ssp device: fw_channel 0, fw_id 16, phy
5,sas_addr 0x5000cca00d0514bd
Jun 22 14:28:41 srv-lab-ceph-node-01 kernel:  phy-0:0:9: mptsas: ioc0:
delete phy 5, phy-obj (0xffff880449541400)
Jun 22 14:28:41 srv-lab-ceph-node-01 kernel:  port-0:0:6: mptsas:
ioc0: delete port 6, sas_addr (0x5000cca00d0514bd)
Jun 22 14:28:41 srv-lab-ceph-node-01 kernel: scsi target0:0:5: mptsas:
ioc0: delete device: fw_channel 0, fw_id 16, phy 5, sas_addr
0x5000cca00d0514bd
Jun 22 14:28:44 srv-lab-ceph-node-01 kernel: BTRFS: lost page write
due to I/O error on /dev/sde1
Jun 22 14:28:44 srv-lab-ceph-node-01 kernel: BTRFS: bdev /dev/sde1
errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Jun 22 14:28:44 srv-lab-ceph-node-01 kernel: BTRFS: lost page write
due to I/O error on /dev/sde1
Jun 22 14:28:44 srv-lab-ceph-node-01 kernel: BTRFS: bdev /dev/sde1
errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
Jun 22 14:29:13 srv-lab-ceph-node-01 kernel: BTRFS: bdev /dev/sde1
errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
Jun 22 14:29:13 srv-lab-ceph-node-01 kernel: BTRFS: bdev /dev/sde1
errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
Jun 22 14:29:13 srv-lab-ceph-node-01 kernel: BTRFS: bdev /dev/sde1
errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
Jun 22 14:29:13 srv-lab-ceph-node-01 kernel: BTRFS: bdev /dev/sde1
errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
Jun 22 14:29:13 srv-lab-ceph-node-01 kernel: BTRFS: bdev /dev/sde1
errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
Jun 22 14:29:13 srv-lab-ceph-node-01 kernel: BTRFS: bdev /dev/sde1
errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
Jun 22 14:29:13 srv-lab-ceph-node-01 kernel: BTRFS: bdev /dev/sde1
errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
Jun 22 14:29:13 srv-lab-ceph-node-01 kernel: BTRFS: bdev /dev/sde1
errs: wr 11, rd 0, flush 0, corrupt 0, gen 0
Jun 22 14:29:13 srv-lab-ceph-node-01 kernel: BTRFS: bdev /dev/sde1
errs: wr 12, rd 0, flush 0, corrupt 0, gen 0
Jun 22 14:29:13 srv-lab-ceph-node-01 kernel: BTRFS: bdev /dev/sde1
errs: wr 13, rd 0, flush 0, corrupt 0, gen 0
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: BTRFS info (device
md127): csum failed ino 1039 extent 390332416 csum 2059524288 wanted
343582415 mirror 0
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: BUG: unable to handle
kernel paging request at ffff87fa7ff53430
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: IP: [<ffffffffc04709d9>]
__btrfs_map_block+0x2d9/0x1180 [btrfs]
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: PGD 0
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: Oops: 0000 [#1] SMP
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: Modules linked in: 8021q

garp mrp stp llc binfmt_misc amdkfd amd_iommu_v2 radeon ttm drm_kms_helper ipmi_ssif coretemp gpio_ich drm kvm_intel serio_raw i5000_edac kvm ipmi_si lpc_ich edac_core ioatdma joydev i2c_algo_bit 8250_fintek mac_hid ipmi_msghandler i5k_amb dca shpchp bonding autofs4 btrfs ses enclosure raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx mptsas mptscsih xor hid_generic raid6_pq raid1 usbhid e1000e mptbase raid0 psmouse ptp hid multipath scsi_transport_sas pps_core linear

Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: CPU: 1 PID: 2411 Comm:
kworker/u16:16 Not tainted 3.19.0-21-generic #21-Ubuntu
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: Hardware name: Intel
S5000VSA/S5000VSA, BIOS S5000.86B.15.00.0101.110920101604 11/09/2010
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: Workqueue: btrfs-endio
btrfs_endio_helper [btrfs]
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: task: ffff8803ef8ae220
ti: ffff8803efbe4000 task.ti: ffff8803efbe4000
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: RIP:
0010:[<ffffffffc04709d9>]  [<ffffffffc04709d9>]
__btrfs_map_block+0x2d9/0x1180 [btrfs]
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: RSP:
0018:ffff8803efbe79d8  EFLAGS: 00010287
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: RAX: 0000000000010000
RBX: ffff88009a80dd00 RCX: 0000000000001533
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: RDX: ffff87fa7ff53428
RSI: ffff88009a80dd70 RDI: 000000009a869e00
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: RBP: ffff8803efbe7ab8
R08: 000000000000c000 R09: ffff88009a80dd00
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: R10: 000000000000c000
R11: 0000000000000002 R12: 000000009a869e00
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: R13: ffff880403566420
R14: ffff880448e20000 R15: 0000000000000001
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: FS:
0000000000000000(0000) GS:ffff88045fc40000(0000)
knlGS:0000000000000000
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: CS:  0010 DS: 0000 ES:
0000 CR0: 000000008005003b
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: CR2: ffff87fa7ff53430
CR3: 000000034f878000 CR4: 00000000000407e0
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: Stack:
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  0000000000000000
0000000000001000 ffff8803efbe7a28 000000000000c000
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  0000000000000001
0000000000010000 0000000015340000 0000000000000000
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  0000000000001534
0000000000001533 ffff880448e20dd0 0000000000000000
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: Call Trace:
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffffc0420afa>] ?
btrfs_free_path+0x2a/0x40 [btrfs]
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffffc0476b5d>]
btrfs_map_bio+0x7d/0x530 [btrfs]
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffffc0493982>]
btrfs_submit_compressed_read+0x332/0x4d0 [btrfs]
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffffc044df51>]
btrfs_submit_bio_hook+0x1c1/0x1d0 [btrfs]
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffff8137cc6e>] ?
bio_add_page+0x5e/0x70
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffffc046be79>] ?
btrfs_create_repair_bio+0xe9/0x110 [btrfs]
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffffc046c38a>]
end_bio_extent_readpage+0x4ea/0x5e0 [btrfs]
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffffc046bea0>] ?
btrfs_create_repair_bio+0x110/0x110 [btrfs]
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffff8137f1eb>]
bio_endio+0x6b/0xa0
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffff811d5bce>] ?
kmem_cache_free+0x1be/0x200
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffff8137f232>]
bio_endio_nodec+0x12/0x20
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffffc0440f3f>]
end_workqueue_fn+0x3f/0x50 [btrfs]
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffffc047b4e2>]
normal_work_helper+0xc2/0x2b0 [btrfs]
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffffc047b7a2>]
btrfs_endio_helper+0x12/0x20 [btrfs]
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffff8108fc98>]
process_one_work+0x158/0x430
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffff810907db>]
worker_thread+0x5b/0x530
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffff81090780>] ?
rescuer_thread+0x3a0/0x3a0
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffff81095879>]
kthread+0xc9/0xe0
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffff810957b0>] ?
kthread_create_on_node+0x1c0/0x1c0
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffff817cae18>]
ret_from_fork+0x58/0x90
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  [<ffffffff810957b0>] ?
kthread_create_on_node+0x1c0/0x1c0
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: Code: 8d 68 ff ff ff 7e
3f 0f 1f 00 49 63 c4 4d 89 d0 41 83 c4 01 48 8d 04 40 48 83 c6 18 48
8d 14 c5 20 00 00 00 49 63 45 10 4c 01 ea <4c> 03 42 08 48 0f af c1 4c
01 c0 48 89 46 e8 48 8b 02 48 89 46
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: RIP  [<ffffffffc04709d9>]
__btrfs_map_block+0x2d9/0x1180 [btrfs]
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel:  RSP <ffff8803efbe79d8>
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: CR2: ffff87fa7ff53430
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: ---[ end trace
f8af5955ebefcf19 ]---
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: BUG: unable to handle
kernel paging request at ffffffffffffffd8
Jun 22 14:29:22 srv-lab-ceph-node-01 kernel: IP: [<ffffffff81095f80>]

kthread_data+0x10/0x20

Have a nice day,
Timofey.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in

And again if i've try
echo 1 > /sys/block/sdf/device/delete

Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: ------------[ cut here
]------------
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: kernel BUG at
/build/buildd/linux-3.19.0/fs/btrfs/extent_io.c:2056!
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: invalid opcode: 0000 [#1] SMP
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: Modules linked in: 8021q

garp mrp stp llc binfmt_misc ipmi_ssif amdkfd amd_iommu_v2 gpio_ich radeon ttm drm_kms_helper lpc_ich coretemp drm kvm_intel kvm i5000_edac i2c_algo_bit edac_core i5k_amb shpchp ipmi_si serio_raw 8250_fintek ioatdma dca joydev mac_hid ipmi_msghandler bonding autofs4 btrfs ses enclosure raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq hid_generic raid1 e1000e raid0 usbhid mptsas mptscsih multipath psmouse hid mptbase ptp scsi_transport_sas pps_core linear

Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: CPU: 0 PID: 1150 Comm:
kworker/u16:12 Not tainted 3.19.0-21-generic #21-Ubuntu
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: Hardware name: Intel
S5000VSA/S5000VSA, BIOS S5000.86B.15.00.0101.110920101604 11/09/2010
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: Workqueue: btrfs-endio
btrfs_endio_helper [btrfs]
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: task: ffff88044c603110
ti: ffff88044b4b8000 task.ti: ffff88044b4b8000
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: RIP:
0010:[<ffffffffc043fa80>]  [<ffffffffc043fa80>]
repair_io_failure+0x1a0/0x220 [btrfs]
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: RSP:
0018:ffff88044b4bbba8  EFLAGS: 00010202
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: RAX: 0000000000000000
RBX: 0000000000001000 RCX: 0000000000000000
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: RDX: 0000000000000000
RSI: ffff880449841b08 RDI: ffff880449841a80
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: RBP: ffff88044b4bbc08
R08: 0000000000109000 R09: ffff880449841a80
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: R10: 0000000000009000
R11: 0000000000000002 R12: ffff8803fa878068
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: R13: ffff880448f5d000
R14: ffff88044cde8d28 R15: 0000000524f09000
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: FS:
0000000000000000(0000) GS:ffff88045fc00000(0000)
knlGS:0000000000000000
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: CS:  0010 DS: 0000 ES:
0000 CR0: 000000008005003b
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: CR2: 00007fdcef9cafb8
CR3: 0000000001c13000 CR4: 00000000000407f0
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: Stack:
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  ffff880448f5d100
0000000000001000 000000004b4bbbd8 ffffea000fb66d40
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  0000000000007000
ffff880449841a80 ffff88044b4bbc08 ffff880439a44b58
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  0000000000001000
ffff880448f5d000 ffff88044cde8d28 ffff88044cde8bf0
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: Call Trace:
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffffc043fd7c>]
clean_io_failure+0x19c/0x1b0 [btrfs]
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffffc04401b0>]
end_bio_extent_readpage+0x310/0x5e0 [btrfs]
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffff811d5795>] ?
__slab_free+0xa5/0x320
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffff8101e74a>] ?
native_sched_clock+0x2a/0x90
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffff8137f1eb>]
bio_endio+0x6b/0xa0
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffff811d5bce>] ?
kmem_cache_free+0x1be/0x200
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffff8137f232>]
bio_endio_nodec+0x12/0x20
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffffc0414f3f>]
end_workqueue_fn+0x3f/0x50 [btrfs]
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffffc044f4e2>]
normal_work_helper+0xc2/0x2b0 [btrfs]
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffffc044f7a2>]
btrfs_endio_helper+0x12/0x20 [btrfs]
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffff8108fc98>]
process_one_work+0x158/0x430
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffff810907db>]
worker_thread+0x5b/0x530
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffff81090780>] ?
rescuer_thread+0x3a0/0x3a0
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffff81095879>]
kthread+0xc9/0xe0
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffff810957b0>] ?
kthread_create_on_node+0x1c0/0x1c0
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffff817cae18>]
ret_from_fork+0x58/0x90
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  [<ffffffff810957b0>] ?
kthread_create_on_node+0x1c0/0x1c0
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: Code: f4 fe ff ff 0f 1f
80 00 00 00 00 0f 0b 66 0f 1f 44 00 00 4c 89 e7 e8 e0 e4 f3 c0 41 b9
fb ff ff ff e9 d2 fe ff ff 0f 1f 44 00 00 <0f> 0b 66 0f 1f 44 00 00 4c
89 e7 e8 c0 e4 f3 c0 31 f6 4c 89 ef
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel: RIP  [<ffffffffc043fa80>]
repair_io_failure+0x1a0/0x220 [btrfs]
Jun 22 14:44:16 srv-lab-ceph-node-01 kernel:  RSP <ffff88044b4bbba8>

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in

So what's up with this? This only happens after you try to (software) remove /dev/sde1? Or is it happening also before that? Because this looks like some kind of hardware problem when the drive is reporting an error for a particular sector on read, as if it's a bad sector.

Again same sector as before. This is not a Btrfs error message, it's coming from the block layer.

I'm not a dev so take it with a grain of salt but because this references a logical block, this is the layer in between Btrfs and the physical device. Btrfs works on logical blocks and those have to be translated to device and physical sector. Maybe what's happening is there's confusion somewhere about this device not actually being unavailable so Btrfs or something else is trying to read this logical block again, which causes a read attempt to happen instead of a flat out "this device doesn't exist" type of error. So I don't know if this is a problem strictly in Btrfs missing device error handling, or if there's something else that's not really working correctly.

You could test by physically removing the device, if you have hot plug support (be certain all the hardware components support it), you can see if you get different results. Or you could try to reproduce the software delete of the device with mdraid or lvm raid with XFS and no Btrfs at all, and see if you get different results.

It's known that the btrfs multiple device failure use case is weak right now. Data isn't lost, but the error handling, notification, all that is almost non-existent compared to mdadm.

OK it looks like not until here does it actually get deleted (?) and then that results in piles of write errors to this device by btrfs:

So this makes sense in that it tries to write but can't because the device is now missing. So it's a case of Btrfs not doing very well handling suddenly missing device, I think.

And then oops. Not good. So yeah it's definitely a Btrfs bug that it oopses instead of gracefully handling the failure. The question is whether (and what) other mitigating circumstances contribute to this bad handling, there may be other bugs that instigate this. I've tested this in a ridiculously rudimentary way (with USB drives) just by hanging them during usage, and I don't get an oops. But I do get piles of read and or write errors and it seems Btrfs never really becomes aware of the fact there's a missing device until there's a remount or even a reboot. I haven't quantified what amount of data is lost, but the file system itself still works degraded in this case with the remaining drive (actually both drives work fine, but once they're each written to separately with degraded mount option, they can't be rejoined together; if you try it, serious fs corruption results.)

Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in

2015-06-22 19:03 GMT+03:00 Chris Murphy <[email protected]>:

Nope, i've physically remove device and as you see it's produce errors on block layer -.-
and this disks have 100% 'health'

Because it's hot-plug device, kernel see what device now missing and remove all kernel objects reletad to them.

So sad -.-
i've test this test case with md raid1 and system continue work without problem when i release one of two md device

You right about usb devices, it's not produce oops.
May be its because kernel use different modules for SAS/SATA disks and usb sticks.

Have a nice day,
Timofey.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in

OK I actually don't know what the intended block layer behavior is when unplugging a device, if it is supposed to vanish, or change state somehow so that thing that depend on it can know it's "missing" or what. So the question here is, is this working as intended? If the layer Btrfs depends on isn't working as intended, then Btrfs is probably going to do wild and crazy things. And I don't know that the part of the block layer Btrfs depends on for this is the same (or different) as what the md driver depends on.

OK well then it's either a Btrfs bug or something it directly depends on that md does not.

They appear as sd devices on my system, so they're using libata and as such they ultimately still depend on the SCSI block layer. But there may be a very different kind of missing device error handling for USB that somehow makes its way up to libata differently than SAS/SATA hotplug.

I'd say the oops is definitely a Btrfs bug. But it might also be worth while to post the kernel messages to linux-scsi@ list, listing the hardware details (logic board, SAS/SATA card, drives) and of course the full kernel messages along with reproduce steps and see if the fact the device doesn't actually drop out like with USB devices is intended behavior.

Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in

I disagree with that statement. BTRFS should be expected to not do wild and crazy things regardless of what happens with block devices.

A BTRFS RAID-1/5/6 array should cope with a single disk failing or returning any manner of corrupted data and should not lose data or panic the kernel.

A BTRFS RAID-0 or single disk setup should cope with a disk giving errors by mounting read-only or failing all operations on the filesystem. It should not affect any other filesystem or have any significant impact on the system unless it's the root filesystem.

My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

I would generally agree with this, although we really shouldn't be doing things like trying to handle hardware failures without user intervention. If a block device disappears from under us, we should throw a warning and if it's the last device in the FS, kill anything that is trying to read or write to that FS. At the very least, we should try to avoid hanging or panicking the system if all of the devices in an FS disappear out from under us.

It's debatable however whether the array should go read-only when degraded. MD/DM RAID (at least, AFAIK) and most hardware RAID controllers I've seen will still accept writes to degraded arrays, although there are arguments for forcing it read-only as well. Personally, I think that should be controlled by a mount option, so the sysadmin can decide, as it really is a policy decision.

Or some other critical filesystem (there are still people who put /usr and/or /var on separate filesystems). Ideally, I'd love to see some some kind of warning from the kernel if a filesystem gets mounted that has the metadata/system profile set to raid0 (and possibly have some of the tools spit out such a warning also).

Am Mittwoch, 5. August 2015, 13:32:41 schrieb Austin S Hemmelgarn:

The best solution I have ever seen for removable media is with AmigaOS. You remove a disk (or nowadays an usb stick) while it is being written to and AmigaDOS/AmigaOS pops up a dialog window saying "You MUST insert volume $VOLUMENAME again". And if you did, it just continued writing. I bet this may be difficult for do for Linux for all devices as unwritten changes pile up in memory until dirty limits are reached, unless one says "Okay, disk gone, we block all processes writing to it immediately or quite soon", but for removable media I never saw anything with that amount of sanity. There was some GSoC for NetBSD once to implement this, but I don´t know whether its implemented in there now. For AmigaOS and floppy disks with back then filesystem there was just one culprit: If you didn´t insert the disk again, it was often broken beyond repair. For journaling or COW filesystem it would just be like in any other sudden stop to writes.

On Linux with eSATA I saw I can also replug the disk if I didn´t yet hit the timeouts in block layer. After that the disk is gone.

Ciao,

Martin