generic/243 occasional fsck SIGSEGV
Hi all,
So we've been seeing an occasional failure of generic/243 that seems to be caused by a userspace fsck crash. This is not the most reproducible failure as it seems to require hundreds of iterations. The test failure generally looks like this:
_check_generic_filesystem: filesystem on /dev/mapper/fedora_fs--i40c--06-scratch is inconsistent *** fsck.bcachefs output ***
fsck from util-linux 2.38.1
fsck: Warning... fsck.bcachefs for device /dev/mapper/fedora_fs--i40c--06-scratch exited with signal 11. *** end fsck.bcachefs output
I reproduced with a non-optimized bcachefs binary and managed to collect a core file from this test system. The initial crash signature looks like this:
Core was generated by `fsck.bcachefs -n /dev/mapper/fedora_fs--i40c--06-scratch'. Program terminated with signal SIGSEGV, Segmentation fault.
#0 memcpy_u64s_small (dst=0x7ffc02479e50, src=0x7fd846fab040, u64s=5) at libbcachefs/util.h:607
607 *d++ = *s++;
[Current thread is 1 (Thread 0x7fd8478cdc80 (LWP 562943))]
(gdb) bt
#0 memcpy_u64s_small (dst=0x7ffc02479e50, src=0x7fd846fab040, u64s=5) at libbcachefs/util.h:607 #1 0x000000000043a186 in bkey_reassemble (dst=0x7ffc02479e28, src=...) at libbcachefs/bkey.h:553 #2 0x000000000043bab0 in bch2_bkey_buf_reassemble (s=0x7ffc02479e20, c=0x7fd847804000, k=...) at libbcachefs/bkey_buf.h:28 #3 0x000000000043e05e in check_bp_exists (trans=0xccb000, bucket=..., bp=..., orig_k=..., bucket_start=..., bucket_end=..., last_flushed=0x7ffc02479e20) at libbcachefs/backpointers.c:450 #4 0x000000000043e464 in check_extent_to_backpointers (trans=0xccb000, btree=BTREE_ID_backpointers, level=1, bucket_start=..., bucket_end=..., last_flushed=0x7ffc02479e20, k=...) at libbcachefs/backpointers.c:501 #5 0x000000000043ed70 in bch2_check_extents_to_backpointers_pass (trans=0xccb000, bucket_start=..., bucket_end=...) at libbcachefs/backpointers.c:647 #6 0x000000000043fa17 in bch2_check_extents_to_backpointers (c=0x7fd847804000) at libbcachefs/backpointers.c:758 #7 0x0000000000536dc0 in bch2_run_recovery_pass (c=0x7fd847804000, pass=BCH_RECOVERY_PASS_check_extents_to_backpointers) at libbcachefs/recovery.c:664 #8 0x0000000000536e57 in bch2_run_recovery_passes (c=0x7fd847804000) at libbcachefs/recovery.c:681 #9 0x0000000000537775 in bch2_fs_recovery (c=0x7fd847804000) at libbcachefs/recovery.c:875
#10 0x000000000055d0ba in bch2_fs_start (c=0x7fd847804000) at libbcachefs/super.c:1010
#11 0x00000000005602d8 in bch2_fs_open (devices=0x7ffc0247a808, nr_devices=1, opts=...) at libbcachefs/super.c:2000 #12 0x000000000041a96d in cmd_fsck (argc=1, argv=0x7ffc0247a808) at cmd_fsck.c:145
#13 0x0000000000411dca in main (argc=3, argv=0x7ffc0247a7f8) at bcachefs.c:202
I've not grokked the high level algorithm for this particular fsck pass, but some quick mechanical analysis suggests that we somehow or another end up with a bkey with a value that points to inaccessible memory. For example, starting in the lowest frame and then jumping up to the frame where k is assigned via bch2_btree_iter_peek():
(gdb) p *d
$2 = 9131882230505012667
(gdb) p *s
Cannot access memory at address 0x7fd846fab048
(gdb) p src
$1 = (const void *) 0x7fd846fab040
(gdb) p *(u64 *) 0x7fd846fab040
Cannot access memory at address 0x7fd846fab040
(gdb) frame 5
#5 0x000000000043ed70 in bch2_check_extents_to_backpointers_pass (trans=0xccb000, bucket_start=..., bucket_end=...) at libbcachefs/backpointers.c:647
647 check_extent_to_backpointers(trans, btree_id, level,
(gdb) p k.k
$4 = (const struct bkey *) 0x7ffc02479ee0
(gdb) p k.v
$5 = (const struct bch_val *) 0x7fd846fab040
So I dont have enough data and context here to know root cause, but I wanted to put some of this data on the list for reference.
Brian
Quoted 38 lines · Brian Foster
Hi all,
So we've been seeing an occasional failure of generic/243 that seems to be caused by a userspace fsck crash. This is not the most reproducible failure as it seems to require hundreds of iterations. The test failure generally looks like this:
_check_generic_filesystem: filesystem on/dev/mapper/fedora_fs--i40c--06-scratchis inconsistent *** fsck.bcachefs output ***
fsck from util-linux 2.38.1
fsck: Warning... fsck.bcachefs for device/dev/mapper/fedora_fs--i40c--06-scratchexited with signal 11. *** end fsck.bcachefs outputI reproduced with a non-optimized bcachefs binary and managed to collect a core file from this test system. The initial crash signature looks like this:
Core was generated by `fsck.bcachefs -n
/dev/mapper/fedora_fs--i40c--06-scratch'. Program terminated with signal SIGSEGV, Segmentation fault.
#0memcpy_u64s_small(dst=0x7ffc02479e50, src=0x7fd846fab040, u64s=5) atlibbcachefs/util.h:607
607 *d++ = *s++;
[Current thread is 1 (Thread0x7fd8478cdc80(LWP 562943))]
(gdb) bt#0 memcpy_u64s_small (dst=0x7ffc02479e50, src=0x7fd846fab040, u64s=5) at libbcachefs/util.h:607 #1 0x000000000043a186 in bkey_reassemble (dst=0x7ffc02479e28, src=...) at libbcachefs/bkey.h:553 #2 0x000000000043bab0 in bch2_bkey_buf_reassemble (s=0x7ffc02479e20, c=0x7fd847804000, k=...) at libbcachefs/bkey_buf.h:28 #3 0x000000000043e05e in check_bp_exists (trans=0xccb000, bucket=..., bp=..., orig_k=..., bucket_start=..., bucket_end=..., last_flushed=0x7ffc02479e20) at libbcachefs/backpointers.c:450 #4 0x000000000043e464 in check_extent_to_backpointers (trans=0xccb000, btree=BTREE_ID_backpointers, level=1, bucket_start=..., bucket_end=..., last_flushed=0x7ffc02479e20, k=...) at libbcachefs/backpointers.c:501 #5 0x000000000043ed70 in bch2_check_extents_to_backpointers_pass (trans=0xccb000, bucket_start=..., bucket_end=...) at libbcachefs/backpointers.c:647 #6 0x000000000043fa17 in bch2_check_extents_to_backpointers (c=0x7fd847804000) at libbcachefs/backpointers.c:758 #7 0x0000000000536dc0 in bch2_run_recovery_pass (c=0x7fd847804000, pass=BCH_RECOVERY_PASS_check_extents_to_backpointers) at libbcachefs/recovery.c:664 #8 0x0000000000536e57 in bch2_run_recovery_passes (c=0x7fd847804000) at libbcachefs/recovery.c:681 #9 0x0000000000537775 in bch2_fs_recovery (c=0x7fd847804000) at libbcachefs/recovery.c:875#10
0x000000000055d0bainbch2_fs_start(c=0x7fd847804000) atlibbcachefs/super.c:1010
#110x00000000005602d8inbch2_fs_open(devices=0x7ffc0247a808, nr_devices=1, opts=...) atlibbcachefs/super.c:2000#120x000000000041a96dincmd_fsck(argc=1, argv=0x7ffc0247a808) at cmd_fsck.c:145
#130x0000000000411dcain main (argc=3, argv=0x7ffc0247a7f8) at bcachefs.c:202
Quoted 5 lines
I've not grokked the high level algorithm for this particular fsck pass, but some quick mechanical analysis suggests that we somehow or another end up with a bkey with a value that points to inaccessible memory. For example, starting in the lowest frame and then jumping up to the frame where k is assigned via
bch2_btree_iter_peek():
libbcachefs/backpointers.c:450, we're using orig_k right after we called write_buffer_flush() - but write_buffer_flush() internally handles transaction restarts, and transaction restart invalides all our old pointers...
this kind of thing - incorrect transaction restart handling - has bit us before; we've got a bunch of checks for "incorrectly kept going/didn't handle a transaction restart" but there's no easy way to guard against this one.
(Perhaps embedding a sequence number in bkey_s_c in debug mode? but then we'd have to check that against a sequence number in btree_trans on every access, so that would be a lot of plumbing.)