Skip to contentSkip to conversation

Broken multi-device filesystem

DISCUSSION Steinar H. Gunderson 2 participants 3 messages Mar 4, 2024 – Mar 8, 2024

lore
Share & tools
lore
#0 lore raw

Hi,

I have a bcachefs root filesystem over two SSDs, together about ~1TB of space (replicas=1). For historical reasons, one of them is an LVM volume, and the other one is a simple partition. This is with a 6.7 kernel (Debian 6.7.7-1, IIRC).

While doing a large-ish build, I noticed the machine got slower and slower, until it hung completely. I hit the physical reset button, and as the root filesystem started mounting, it hung in the step after journal replay (I believe it is called something with “deleted inodes”). After about two hours, I hit the physical reset button again, and ran bcachefs fsck (from bcachefs-tools 1.6.3) on the filesystem. It printed out the following (manually typed from a photo of the display, errors may occur):

fsck binary is version 1.6: btree_subvolume_children but filesystem is 1.3: rebalance_work and kernel is 1.3: rebalance_work mounting version 1.3: rebalance_work opts=ro,compression=lz4,degraded,fsck,fix_errors=ask,read_only recovering from unclean shutdown
Doing compatible version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children
running recovery passes: check_subvols,check_dirents
journal read done,replaying entries 7263879-7263879
alloc_read... done
stripes_read... done
snapshots_read... done
check_allocations...dev 0 has wrong free buckets: got 507491, should be 499853: fix?
(y,n, or Y,N for all errors of this type) Y
dev 0 has wrong sb buckets: got 0, should be 7, fixing
dev 0 has wrong sb sectors: got 0, should be 6152: fix?
(y,n, or Y,N for all errors of this type) Y
dev 0 has wrong sb fragmented: got 0, should be 1016: fix?
(y,n, or Y,N for all errors of this type) Y
dev 0 has wrong journal buckets: got 0, should be 7631, fixing
dev 0 has wrong journal buckets: got 0, should be 1192480: fix?
(y,n, or Y,N for all errors of this type) Y
done
going read-write
journal_replay... done
check_alloc_info... done
check_lrus... done
check_btree_backpointers... done
check_backpointers_to_extents... done
check_alloc_to_lru_refs... done
check_snapshot_trees... done
check_snapshots... done
check_subvols... done
check_subvol_children... done
delete_dead_snapshot... done
resume_logged_ops... done
check_inodes...inodes 759339:4294967288 unlinked, but not on deleted list: fix?
WARNING at libbcachefs/btree_iter.c:2827: btree trans held srcu lock (delaying memory reclaim) for 22 seconds

At this point, there was no visible I/O and no visible progress for about an hour (and being in initramfs, I couldn't check top), so I Ctrl-C-ed fsck and tried to mount again. It gave (more manual typing; the top and right is cut off in the screenshot, and I haven't bothered typing in the microseconds):

  [  729.446] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): alloc_read... done
  [  729.452] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): stripes_read... done
  [  729.457] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): snapshots_read... done
  [  729.484] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): journal_replay... done
  [  729.489] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): check_subvols... done
  [  729.494] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): resume_logged_ops... done
  [  729.499] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): check_dirents...
  [  729.543] inode 1410398479:4294966699 has wrong backpointer:
  [  729.543] got       604737456:6483958551636691117
  [  729.543] should be 23149:8508601757679713244, exiting
  [  729.562] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): Unable to continue, halting
  [  729.567] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): check_dirent_target(): error fsck_erro
  [  729.572] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): check_dirent(): error fsck_errors_not_
  [  729.577] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): bch2_check_dirents(): error fsck_errors
  [  729.583] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): bch2_fs_start(): error starting filesys
  [  729.588] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): mounting version 1.6: (unknown version)
  [  729.626] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): recovering from unclean shutdown
  [  729.631] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): superblock required following recovery 
  [  729.636]   check_subvols,check_dirents
  [  729.646] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): Version downgrade required:
  [  729.646]
  [  729.657] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): Version upgrade from 1.3: rebalance_work
  [  729.657] Doing compatible version upgrade from 1.3: rebalance_work to 1.6: (unknown version)
  [  729.657] 
  [  731.722] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): journal read done, replaying entries 7263
  [  731.747] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): alloc_read...  done
  [  731.750] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): stripes_read... done
  [  731.754] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): snapshots_read... done
  [  731.759] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): journal_replay... done
  [  731.786] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): check_subvols... done
  [  731.791] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): resume_logged_ops... done
  [  731.796] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): check_dirents...
  [  731.846] inode 1410398479:4294966699 has wrong backpointer:
  [  731.846] got       604737456:6483958551636691117
  [  731.846] should be 23149:8508601757679713244, exiting
  [  731.867] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): Unable to continue, halting
  [  731.872] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): check_dirent_target(): error fsck_errors_no
  [  731.877] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): check_dirent(): error fsck_errors_not_fixed
  [  731.883] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): bch2_check_dirents(): error fsck_errors_not
  [  731.888] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): bch2_fs_recovery(): error fsck_errors_not_f
  [  731.893] bcachefs (16494483-fbc6-4e2f-a316-d2777e730960): bch2_fs_start(): error starting filesys

ERROR - bcachefs::commands::cmd_mount Fatal error: Invalid argument
mount: mounting /dev/mapper/bigscreen--vg-tablebase:/dev/nvme1n1p1 on /root failed: Invalid argument

At this point, I am no longer allowed to try fsck, because it thinks the two devices have gone into split brain:

(initramfs) bcachefs fsck /dev/mapper/bigscreen--vg-tablebase:/dev/nvme1n1p1 bch2_dev_in_fs() Split brain detected between /dev/nvme1n1p1 and /dev/mapper/bigscreen--vg-tablebase:
/dev/mapper/bigscreen--vg-tablebase believes seq of /dev/nvme1n1p1 to be 126, but /dev/nvme1n1p1 has 130 Not using /dev/nvme1n1p1
bch2_fs_open() bcs_fs_open err opening /dev/mapper/bigscreen--vg-tablebase: insufficient_devices_to_start bch2_dev_in_fs() Split brain detected between /dev/nvme1n1p1 and /dev/mapper/bigscreen--vg-tablebase:
/dev/mapper/bigscreen--vg-tablebase believes seq of /dev/nvme1n1p1 to be 126, but /dev/nvme1n1p1 has 130 Not using /dev/nvme1n1p1
insufficient devices online (0) for replicas entry journal: 1/1 [0]
bch2_fs_open() bcs_fs_open err opening /dev/mapper/bigscreen--vg-tablebase: insufficient_devices_to_start

So the entire machine is hosed. (I still have an older copy of the filesystem in xfs form on a different partition, so I can boot into that, and I do have backups of all the user data, but it is still pretty inconvenient.) Is there any cure?

/* Steinar */

Signature · 1 line
Homepage: https://www.sesse.net/
#1 lore raw
Quoted 3 lines · Steinar H. Gunderson

So the entire machine is hosed. (I still have an older copy of the filesystem in xfs form on a different partition, so I can boot into that, and I do have backups of all the user data, but it is still pretty inconvenient.)

I did a dump of the filesystem to two .qcow2 files, overriding the split brain detection (and then repaired them with qemu-img, as qemu-nbd was complaining they were broken). fsck on the resulting block devices gave an assertion error:

  root@bigscreen:~/bcachefs-tools# ./bcachefs fsck /dev/nbd0:/dev/nbd1
  mounting version 1.6: btree_subvolume_children opts=ro,compression=lz4,degraded,fsck,fix_errors=ask,read_only
  recovering from unclean shutdown
  superblock requires following recovery passes to be run:
    check_subvols,check_dirents
  Version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children incomplete
  Doing compatible version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children

journal read done, replaying entries 7263903-7263903
error validating btree node on /dev/nbd0 at btree extents level 2/2
u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq b25c303a8b7a1163 written 112 min_key POS_MIN durability: 1 ptr: 0:441498:0 gen 1 node offset 64/112: btree node data missing: expected 112 sectors, found 64: fix?
(y,n, or Y,N for all errors of this type) Y
btree_node_read_work: rewriting btree node at btree=extents level=2 SPOS_MAX due to error
error validating btree node on /dev/nbd0 at btree dirents level 1/1
u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq 79b64c5dfcd3e63e written 459 min_key POS_MIN durability: 1 ptr: 0:351317:0 gen 2 node offset 0/459 bset u64s 17199: checksum error: got bb38ac4c should be fbddc60e type crc32c_nonzero, fixing
error validating btree node at btree dirents level 1/1

    u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq 79b64c5dfcd3e63e written 459 min_key POS_MIN durability: 1 ptr: 0:351317:0 gen 2
    node offset 0/459 bset u64s 17199: invalid bkey: u64s too small (2 < 5)
    u64s 2 type deleted POS_MIN len 0 ver 0: , fixing

error validating btree node on /dev/nbd0 at btree dirents level 1/1
u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq 79b64c5dfcd3e63e written 459 min_key POS_MIN durability: 1 ptr: 0:351317:0 gen 2 node offset 270/459: btree node data missing: expected 459 sectors, found 270, fixing
bcachefs: libbcachefs/extents.h:71: extent_entry_bytes: Assertion `0' failed.

/* Steinar */

Signature · 1 line
Homepage: https://www.sesse.net/
#2 lore raw
Quoted 9 lines · Steinar H. Gunderson
Quoted 3 lines · Steinar H. Gunderson

So the entire machine is hosed. (I still have an older copy of the filesystem in xfs form on a different partition, so I can boot into that, and I do have backups of all the user data, but it is still pretty inconvenient.)

I did a dump of the filesystem to two .qcow2 files, overriding the split brain detection (and then repaired them with qemu-img, as qemu-nbd was complaining they were broken). fsck on the resulting block devices gave an assertion error:

Can you hop on IRC and send me the dumps with magic wormhole?

Quoted 30 lines
  root@bigscreen:~/bcachefs-tools# ./bcachefs fsck /dev/nbd0:/dev/nbd1
  mounting version 1.6: btree_subvolume_children opts=ro,compression=lz4,degraded,fsck,fix_errors=ask,read_only
  recovering from unclean shutdown
  superblock requires following recovery passes to be run:
    check_subvols,check_dirents
  Version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children incomplete
  Doing compatible version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children

journal read done, replaying entries 7263903-7263903
error validating btree node on /dev/nbd0 at btree extents level 2/2
u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq b25c303a8b7a1163 written 112 min_key POS_MIN durability: 1 ptr: 0:441498:0 gen 1 node offset 64/112: btree node data missing: expected 112 sectors, found 64: fix?
(y,n, or Y,N for all errors of this type) Y
btree_node_read_work: rewriting btree node at btree=extents level=2 SPOS_MAX due to error
error validating btree node on /dev/nbd0 at btree dirents level 1/1
u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq 79b64c5dfcd3e63e written 459 min_key POS_MIN durability: 1 ptr: 0:351317:0 gen 2 node offset 0/459 bset u64s 17199: checksum error: got bb38ac4c should be fbddc60e type crc32c_nonzero, fixing
error validating btree node at btree dirents level 1/1

    u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq 79b64c5dfcd3e63e written 459 min_key POS_MIN durability: 1 ptr: 0:351317:0 gen 2
    node offset 0/459 bset u64s 17199: invalid bkey: u64s too small (2 < 5)
    u64s 2 type deleted POS_MIN len 0 ver 0: , fixing

error validating btree node on /dev/nbd0 at btree dirents level 1/1
u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq 79b64c5dfcd3e63e written 459 min_key POS_MIN durability: 1 ptr: 0:351317:0 gen 2 node offset 270/459: btree node data missing: expected 459 sectors, found 270, fixing
bcachefs: libbcachefs/extents.h:71: extent_entry_bytes: Assertion `0' failed.

/* Steinar */

Signature · 1 line
Homepage: https://www.sesse.net/