Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 309E982EC2 for ; Fri, 15 Dec 2023 17:13:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="TewKeuVu" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702660425; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=ThdJeZKJvn4FKQKnj2GVbHAKzt3jIUBgRnicNjm7sjk=; b=TewKeuVu5MnnPdIk+Dg4A6ltDVhe9blUeAnSmgEs5J5MsRD3FibWRwXHj7qhliQ9ar/BZP b4yJN6kVaIt2palthTnzRLD4X+0nQbsmEDU4cZfdMyofocQxIWw3voRxfZsojd1DATpGoX KlviEHM2E8BKs/xAC7DT6E7aMdIsp5M= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-684-0qei5iN9PU6itUWEf_TvvA-1; Fri, 15 Dec 2023 12:13:43 -0500 X-MC-Unique: 0qei5iN9PU6itUWEf_TvvA-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3AE9E29AB3EC for ; Fri, 15 Dec 2023 17:13:43 +0000 (UTC) Received: from bfoster (unknown [10.22.8.199]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1D7EF492BF0 for ; Fri, 15 Dec 2023 17:13:43 +0000 (UTC) Date: Fri, 15 Dec 2023 12:14:44 -0500 From: Brian Foster To: linux-bcachefs@vger.kernel.org Subject: generic/243 occasional fsck SIGSEGV Message-ID: Precedence: bulk X-Mailing-List: linux-bcachefs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.10 Hi all, So we've been seeing an occasional failure of generic/243 that seems to be caused by a userspace fsck crash. This is not the most reproducible failure as it seems to require hundreds of iterations. The test failure generally looks like this: _check_generic_filesystem: filesystem on /dev/mapper/fedora_fs--i40c--06-scratch is inconsistent *** fsck.bcachefs output *** fsck from util-linux 2.38.1 fsck: Warning... fsck.bcachefs for device /dev/mapper/fedora_fs--i40c--06-scratch exited with signal 11. *** end fsck.bcachefs output I reproduced with a non-optimized bcachefs binary and managed to collect a core file from this test system. The initial crash signature looks like this: Core was generated by `fsck.bcachefs -n /dev/mapper/fedora_fs--i40c--06-scratch'. Program terminated with signal SIGSEGV, Segmentation fault. #0 memcpy_u64s_small (dst=0x7ffc02479e50, src=0x7fd846fab040, u64s=5) at libbcachefs/util.h:607 607 *d++ = *s++; [Current thread is 1 (Thread 0x7fd8478cdc80 (LWP 562943))] (gdb) bt #0 memcpy_u64s_small (dst=0x7ffc02479e50, src=0x7fd846fab040, u64s=5) at libbcachefs/util.h:607 #1 0x000000000043a186 in bkey_reassemble (dst=0x7ffc02479e28, src=...) at libbcachefs/bkey.h:553 #2 0x000000000043bab0 in bch2_bkey_buf_reassemble (s=0x7ffc02479e20, c=0x7fd847804000, k=...) at libbcachefs/bkey_buf.h:28 #3 0x000000000043e05e in check_bp_exists (trans=0xccb000, bucket=..., bp=..., orig_k=..., bucket_start=..., bucket_end=..., last_flushed=0x7ffc02479e20) at libbcachefs/backpointers.c:450 #4 0x000000000043e464 in check_extent_to_backpointers (trans=0xccb000, btree=BTREE_ID_backpointers, level=1, bucket_start=..., bucket_end=..., last_flushed=0x7ffc02479e20, k=...) at libbcachefs/backpointers.c:501 #5 0x000000000043ed70 in bch2_check_extents_to_backpointers_pass (trans=0xccb000, bucket_start=..., bucket_end=...) at libbcachefs/backpointers.c:647 #6 0x000000000043fa17 in bch2_check_extents_to_backpointers (c=0x7fd847804000) at libbcachefs/backpointers.c:758 #7 0x0000000000536dc0 in bch2_run_recovery_pass (c=0x7fd847804000, pass=BCH_RECOVERY_PASS_check_extents_to_backpointers) at libbcachefs/recovery.c:664 #8 0x0000000000536e57 in bch2_run_recovery_passes (c=0x7fd847804000) at libbcachefs/recovery.c:681 #9 0x0000000000537775 in bch2_fs_recovery (c=0x7fd847804000) at libbcachefs/recovery.c:875 #10 0x000000000055d0ba in bch2_fs_start (c=0x7fd847804000) at libbcachefs/super.c:1010 #11 0x00000000005602d8 in bch2_fs_open (devices=0x7ffc0247a808, nr_devices=1, opts=...) at libbcachefs/super.c:2000 #12 0x000000000041a96d in cmd_fsck (argc=1, argv=0x7ffc0247a808) at cmd_fsck.c:145 #13 0x0000000000411dca in main (argc=3, argv=0x7ffc0247a7f8) at bcachefs.c:202 I've not grokked the high level algorithm for this particular fsck pass, but some quick mechanical analysis suggests that we somehow or another end up with a bkey with a value that points to inaccessible memory. For example, starting in the lowest frame and then jumping up to the frame where k is assigned via bch2_btree_iter_peek(): (gdb) p *d $2 = 9131882230505012667 (gdb) p *s Cannot access memory at address 0x7fd846fab048 (gdb) p src $1 = (const void *) 0x7fd846fab040 (gdb) p *(u64 *) 0x7fd846fab040 Cannot access memory at address 0x7fd846fab040 (gdb) frame 5 #5 0x000000000043ed70 in bch2_check_extents_to_backpointers_pass (trans=0xccb000, bucket_start=..., bucket_end=...) at libbcachefs/backpointers.c:647 647 check_extent_to_backpointers(trans, btree_id, level, (gdb) p k.k $4 = (const struct bkey *) 0x7ffc02479ee0 (gdb) p k.v $5 = (const struct bch_val *) 0x7fd846fab040 So I dont have enough data and context here to know root cause, but I wanted to put some of this data on the list for reference. Brian