Broken multi-device filesystem
Hi,
I have a bcachefs root filesystem over two SSDs, together about ~1TB of space (replicas=1). For historical reasons, one of them is an LVM volume, and the other one is a simple partition. This is with a 6.7 kernel (Debian 6.7.7-1, IIRC).
While doing a large-ish build, I noticed the machine got slower and slower, until it hung completely. I hit the physical reset button, and as the root filesystem started mounting, it hung in the step after journal replay (I believe it is called something with “deleted inodes”). After about two hours, I hit the physical reset button again, and ran bcachefs fsck (from bcachefs-tools 1.6.3) on the filesystem. It printed out the following (manually typed from a photo of the display, errors may occur):
fsck binary is version 1.6: btree_subvolume_children but filesystem is 1.3: rebalance_work and kernel is 1.3: rebalance_work mounting version 1.3: rebalance_work opts=ro,compression=lz4,degraded,fsck,fix_errors=ask,read_only recovering from unclean shutdown
Doing compatible version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children
running recovery passes: check_subvols,check_dirents
journal read done,replaying entries 7263879-7263879
alloc_read... done
stripes_read... done
snapshots_read... done
check_allocations...dev 0 has wrong free buckets: got 507491, should be 499853: fix?
(y,n, or Y,N for all errors of this type) Y
dev 0 has wrong sb buckets: got 0, should be 7, fixing
dev 0 has wrong sb sectors: got 0, should be 6152: fix?
(y,n, or Y,N for all errors of this type) Y
dev 0 has wrong sb fragmented: got 0, should be 1016: fix?
(y,n, or Y,N for all errors of this type) Y
dev 0 has wrong journal buckets: got 0, should be 7631, fixing
dev 0 has wrong journal buckets: got 0, should be 1192480: fix?
(y,n, or Y,N for all errors of this type) Y
done
going read-write
journal_replay... done
check_alloc_info... done
check_lrus... done
check_btree_backpointers... done
check_backpointers_to_extents... done
check_alloc_to_lru_refs... done
check_snapshot_trees... done
check_snapshots... done
check_subvols... done
check_subvol_children... done
delete_dead_snapshot... done
resume_logged_ops... done
check_inodes...inodes 759339:4294967288 unlinked, but not on deleted list: fix?
WARNING at libbcachefs/btree_iter.c:2827: btree trans held srcu lock (delaying memory reclaim) for 22 seconds
At this point, there was no visible I/O and no visible progress for about an hour (and being in initramfs, I couldn't check top), so I Ctrl-C-ed fsck and tried to mount again. It gave (more manual typing; the top and right is cut off in the screenshot, and I haven't bothered typing in the microseconds):
[ 729.446] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): alloc_read... done [ 729.452] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): stripes_read... done [ 729.457] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): snapshots_read... done [ 729.484] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): journal_replay... done [ 729.489] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): check_subvols... done [ 729.494] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): resume_logged_ops... done [ 729.499] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): check_dirents... [ 729.543] inode 1410398479:4294966699 has wrong backpointer: [ 729.543] got 604737456:6483958551636691117 [ 729.543] should be 23149:8508601757679713244, exiting [ 729.562] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): Unable to continue, halting [ 729.567] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): check_dirent_target(): error fsck_erro [ 729.572] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): check_dirent(): error fsck_errors_not_ [ 729.577] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): bch2_check_dirents(): error fsck_errors [ 729.583] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): bch2_fs_start(): error starting filesys [ 729.588] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): mounting version 1.6: (unknown version) [ 729.626] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): recovering from unclean shutdown [ 729.631] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): superblock required following recovery [ 729.636] check_subvols,check_dirents [ 729.646] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): Version downgrade required: [ 729.646] [ 729.657] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): Version upgrade from 1.3: rebalance_work [ 729.657] Doing compatible version upgrade from 1.3: rebalance_work to 1.6: (unknown version) [ 729.657] [ 731.722] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): journal read done, replaying entries 7263 [ 731.747] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): alloc_read... done [ 731.750] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): stripes_read... done [ 731.754] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): snapshots_read... done [ 731.759] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): journal_replay... done [ 731.786] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): check_subvols... done [ 731.791] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): resume_logged_ops... done [ 731.796] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): check_dirents... [ 731.846] inode 1410398479:4294966699 has wrong backpointer: [ 731.846] got 604737456:6483958551636691117 [ 731.846] should be 23149:8508601757679713244, exiting [ 731.867] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): Unable to continue, halting [ 731.872] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): check_dirent_target(): error fsck_errors_no [ 731.877] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): check_dirent(): error fsck_errors_not_fixed [ 731.883] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): bch2_check_dirents(): error fsck_errors_not [ 731.888] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): bch2_fs_recovery(): error fsck_errors_not_f [ 731.893] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): bch2_fs_start(): error starting filesys
ERROR - bcachefs::commands::cmd_mount Fatal error: Invalid argument
mount: mounting /dev/mapper/bigscreen--vg-tablebase:/dev/nvme1n1p1 on /root failed: Invalid argument
At this point, I am no longer allowed to try fsck, because it thinks the two devices have gone into split brain:
(initramfs) bcachefs fsck /dev/mapper/bigscreen--vg-tablebase:/dev/nvme1n1p1 bch2_dev_in_fs() Split brain detected between /dev/nvme1n1p1 and /dev/mapper/bigscreen--vg-tablebase:
/dev/mapper/bigscreen--vg-tablebase believes seq of /dev/nvme1n1p1 to be 126, but /dev/nvme1n1p1 has 130 Not using /dev/nvme1n1p1
bch2_fs_open() bcs_fs_open err opening /dev/mapper/bigscreen--vg-tablebase: insufficient_devices_to_start bch2_dev_in_fs() Split brain detected between /dev/nvme1n1p1 and /dev/mapper/bigscreen--vg-tablebase:
/dev/mapper/bigscreen--vg-tablebase believes seq of /dev/nvme1n1p1 to be 126, but /dev/nvme1n1p1 has 130 Not using /dev/nvme1n1p1
insufficient devices online (0) for replicas entry journal: 1/1 [0]
bch2_fs_open() bcs_fs_open err opening /dev/mapper/bigscreen--vg-tablebase: insufficient_devices_to_start
So the entire machine is hosed. (I still have an older copy of the filesystem in xfs form on a different partition, so I can boot into that, and I do have backups of all the user data, but it is still pretty inconvenient.) Is there any cure?
/* Steinar */
Signature · 1 line
Homepage: https://www.sesse.net/
Quoted 3 lines · Steinar H. Gunderson
So the entire machine is hosed. (I still have an older copy of the filesystem in xfs form on a different partition, so I can boot into that, and I do have backups of all the user data, but it is still pretty inconvenient.)
I did a dump of the filesystem to two .qcow2 files, overriding the split brain detection (and then repaired them with qemu-img, as qemu-nbd was complaining they were broken). fsck on the resulting block devices gave an assertion error:
root@bigscreen:~/bcachefs-tools# ./bcachefs fsck /dev/nbd0:/dev/nbd1
mounting version 1.6: btree_subvolume_children opts=ro,compression=lz4,degraded,fsck,fix_errors=ask,read_only
recovering from unclean shutdown
superblock requires following recovery passes to be run:
check_subvols,check_dirents
Version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children incomplete
Doing compatible version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children journal read done, replaying entries 7263903-7263903
error validating btree node on /dev/nbd0 at btree extents level 2/2
u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq b25c303a8b7a1163 written 112 min_key POS_MIN durability: 1 ptr: 0:441498:0 gen 1 node offset 64/112: btree node data missing: expected 112 sectors, found 64: fix?
(y,n, or Y,N for all errors of this type) Y
btree_node_read_work: rewriting btree node at btree=extents level=2 SPOS_MAX due to error
error validating btree node on /dev/nbd0 at btree dirents level 1/1
u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq 79b64c5dfcd3e63e written 459 min_key POS_MIN durability: 1 ptr: 0:351317:0 gen 2 node offset 0/459 bset u64s 17199: checksum error: got bb38ac4c should be fbddc60e type crc32c_nonzero, fixing
error validating btree node at btree dirents level 1/1
u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq 79b64c5dfcd3e63e written 459 min_key POS_MIN durability: 1 ptr: 0:351317:0 gen 2
node offset 0/459 bset u64s 17199: invalid bkey: u64s too small (2 < 5)
u64s 2 type deleted POS_MIN len 0 ver 0: , fixing error validating btree node on /dev/nbd0 at btree dirents level 1/1
u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq 79b64c5dfcd3e63e written 459 min_key POS_MIN durability: 1 ptr: 0:351317:0 gen 2 node offset 270/459: btree node data missing: expected 459 sectors, found 270, fixing
bcachefs: libbcachefs/extents.h:71: extent_entry_bytes: Assertion `0' failed.
/* Steinar */
Signature · 1 line
Homepage: https://www.sesse.net/
Quoted 9 lines · Steinar H. Gunderson
Quoted 3 lines · Steinar H. Gunderson
So the entire machine is hosed. (I still have an older copy of the filesystem in xfs form on a different partition, so I can boot into that, and I do have backups of all the user data, but it is still pretty inconvenient.)
I did a dump of the filesystem to two .qcow2 files, overriding the split brain detection (and then repaired them with qemu-img, as qemu-nbd was complaining they were broken). fsck on the resulting block devices gave an assertion error:
Can you hop on IRC and send me the dumps with magic wormhole?
Quoted 30 lines
root@bigscreen:~/bcachefs-tools# ./bcachefs fsck /dev/nbd0:/dev/nbd1 mounting version 1.6: btree_subvolume_children opts=ro,compression=lz4,degraded,fsck,fix_errors=ask,read_only recovering from unclean shutdown superblock requires following recovery passes to be run: check_subvols,check_dirents Version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children incomplete Doing compatible version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_childrenjournal read done, replaying entries 7263903-7263903
error validating btree node on/dev/nbd0at btree extents level 2/2
u64s 11 typebtree_ptr_v2SPOS_MAXlen 0 ver 0: seqb25c303a8b7a1163written 112min_keyPOS_MINdurability: 1 ptr: 0:441498:0 gen 1 node offset 64/112: btree node data missing: expected 112 sectors, found 64: fix?
(y,n, or Y,N for all errors of this type) Y
btree_node_read_work: rewriting btree node at btree=extents level=2SPOS_MAXdue to error
error validating btree node on/dev/nbd0at btree dirents level 1/1
u64s 11 typebtree_ptr_v2SPOS_MAXlen 0 ver 0: seq79b64c5dfcd3e63ewritten 459min_keyPOS_MINdurability: 1 ptr: 0:351317:0 gen 2 node offset 0/459 bset u64s 17199: checksum error: gotbb38ac4cshould befbddc60etypecrc32c_nonzero, fixing
error validating btree node at btree dirents level 1/1u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq 79b64c5dfcd3e63e written 459 min_key POS_MIN durability: 1 ptr: 0:351317:0 gen 2 node offset 0/459 bset u64s 17199: invalid bkey: u64s too small (2 < 5) u64s 2 type deleted POS_MIN len 0 ver 0: , fixingerror validating btree node on
/dev/nbd0at btree dirents level 1/1
u64s 11 typebtree_ptr_v2SPOS_MAXlen 0 ver 0: seq79b64c5dfcd3e63ewritten 459min_keyPOS_MINdurability: 1 ptr: 0:351317:0 gen 2 node offset 270/459: btree node data missing: expected 459 sectors, found 270, fixing
bcachefs:libbcachefs/extents.h:71:extent_entry_bytes: Assertion `0' failed./* Steinar */
Signature · 1 line
Homepage: https://www.sesse.net/