general protection fault, probably for non-canonical address 0x280766500040001: 0000 [#1] PREEMPT SMP PTI
Hi Kent,
JFYI, I'm seeing the following splat pretty reliably via generic/361 on an 80xcpu test box. The CI doesn't seem to produce this failure for whatever reason. This bisects down to commit 023f9ac9f70f ("bcachefs:
Delete dio read alignment check"), before which the test still fails but the kernel doesn't explode.
Brian
--- 8< ---
general protection fault, probably for non-canonical address 0x280766500040001: 0000 [#1] PREEMPT SMP PTI CPU: 15 PID: 1078 Comm: kworker/15:1H Tainted: G OE 6.7.0-rc7+ #68 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc37 04/01/2014 Workqueue: events_highpri __bch2_read_endio [bcachefs] RIP: 0010:bio_copy_data_iter+0x186/0x260 Code: 29 f6 48 c1 f9 06 48 c1 fe 06 48 c1 e1 0c 48 c1 e6 0c 48 01 e9 48 01 ee 48 01 d9 4c 01 d6 83 fa 08 0f 82 b0 fe ff ff 48 8b 06 <48> 89 01 89 d0 48 8b 7c 06 f8 48 89 7c 01 f8 48 8d 79 08 48 83 e7 RSP: 0018:ffffa31644a63d38 EFLAGS: 00010212 RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0280766500040001 RDX: 0000000000000c00 RSI: ffff8d462b83d400 RDI: ffff8d460fc9edd0 RBP: ffff8d4500000000 R08: ffffa31644a63dc0 R09: ffffa31644a63dd4 R10: 0000000000000400 R11: 0000000000000c00 R12: ffff8d462beb9aa8 R13: ffff8d460fc9ed38 R14: fffffc5e80000000 R15: 0000000000001000 FS: 0000000000000000(0000) GS:ffff8d64fdd80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f7255016120 CR3: 000000012534e003 CR4: 0000000000770ef0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 PKRU: 55555554 Call Trace: <TASK> ? die_addr+0x32/0x80 ? exc_general_protection+0x19b/0x450 ? asm_exc_general_protection+0x22/0x30 ? bio_copy_data_iter+0x186/0x260 __bch2_read_endio+0x83d/0x920 [bcachefs] ? process_one_work+0x192/0x4d0 ? process_one_work+0x1fc/0x4d0 process_one_work+0x1fc/0x4d0 worker_thread+0x1dd/0x3d0 ? __pfx_worker_thread+0x10/0x10 kthread+0x100/0x130 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x2d/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1b/0x30 </TASK> Modules linked in: bcachefs(OE) lz4_compress(E) lz4hc_compress(E) nft_fib_inet(E) nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E) nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E) nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E) nf_tables(E) nfnetlink(E) sunrpc(E) intel_rapl_msr(E) intel_rapl_common(E) intel_uncore_frequency_common(E) isst_if_common(E) kvm_intel(E) kvm(E) iTCO_wdt(E) intel_pmc_bxt(E) iTCO_vendor_support(E) irqbypass(E) virtio_balloon(E) rapl(E) pktcdvd(E) i2c_i801(E) pcspkr(E) i2c_smbus(E) lpc_ich(E) joydev(E) loop(E) zram(E) crct10dif_pclmul(E) crc32_pclmul(E) crc32c_intel(E) virtiofs(E) virtio_net(E) net_failover(E) ghash_clmulni_intel(E) virtio_console(E) failover(E) fuse(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) qemu_fw_cfg(E) Unloaded tainted modules: bcachefs(E):3 intel_uncore_frequency(E):1 isst_if_mbox_msr(E):1 [last unloaded: bcachefs(OE)] ---[ end trace 0000000000000000 ]--- RIP: 0010:bio_copy_data_iter+0x186/0x260 Code: 29 f6 48 c1 f9 06 48 c1 fe 06 48 c1 e1 0c 48 c1 e6 0c 48 01 e9 48 01 ee 48 01 d9 4c 01 d6 83 fa 08 0f 82 b0 fe ff ff 48 8b 06 <48> 89 01 89 d0 48 8b 7c 06 f8 48 89 7c 01 f8 48 8d 79 08 48 83 e7 RSP: 0018:ffffa31644a63d38 EFLAGS: 00010212 RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0280766500040001 RDX: 0000000000000c00 RSI: ffff8d462b83d400 RDI: ffff8d460fc9edd0 RBP: ffff8d4500000000 R08: ffffa31644a63dc0 R09: ffffa31644a63dd4 R10: 0000000000000400 R11: 0000000000000c00 R12: ffff8d462beb9aa8 R13: ffff8d460fc9ed38 R14: fffffc5e80000000 R15: 0000000000001000 FS: 0000000000000000(0000) GS:ffff8d64fdd80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f7255016120 CR3: 000000012534e003 CR4: 0000000000770ef0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Quoted 7 lines · Brian Foster
Hi Kent,
JFYI, I'm seeing the following splat pretty reliably via generic/361 on an 80xcpu test box. The CI doesn't seem to produce this failure for whatever reason. This bisects down to commit
023f9ac9f70f("bcachefs:
Delete dio read alignment check"), before which the test still fails but the kernel doesn't explode.
Ahh, that probably needs to be a 512 byte alignment check; since the block layer alternates units between bytes and 512 byte sectors things will get very confused if the IO wasn't 512 byte aligned.
Quoted 59 lines
Brian
--- 8< ---
general protection fault, probably for non-canonical address 0x280766500040001: 0000 [#1] PREEMPT SMP PTI CPU: 15 PID: 1078 Comm: kworker/15:1H Tainted: G OE 6.7.0-rc7+ #68 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc37 04/01/2014 Workqueue: events_highpri __bch2_read_endio [bcachefs] RIP: 0010:bio_copy_data_iter+0x186/0x260 Code: 29 f6 48 c1 f9 06 48 c1 fe 06 48 c1 e1 0c 48 c1 e6 0c 48 01 e9 48 01 ee 48 01 d9 4c 01 d6 83 fa 08 0f 82 b0 fe ff ff 48 8b 06 <48> 89 01 89 d0 48 8b 7c 06 f8 48 89 7c 01 f8 48 8d 79 08 48 83 e7 RSP: 0018:ffffa31644a63d38 EFLAGS: 00010212 RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0280766500040001 RDX: 0000000000000c00 RSI: ffff8d462b83d400 RDI: ffff8d460fc9edd0 RBP: ffff8d4500000000 R08: ffffa31644a63dc0 R09: ffffa31644a63dd4 R10: 0000000000000400 R11: 0000000000000c00 R12: ffff8d462beb9aa8 R13: ffff8d460fc9ed38 R14: fffffc5e80000000 R15: 0000000000001000 FS: 0000000000000000(0000) GS:ffff8d64fdd80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f7255016120 CR3: 000000012534e003 CR4: 0000000000770ef0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 PKRU: 55555554 Call Trace: <TASK> ? die_addr+0x32/0x80 ? exc_general_protection+0x19b/0x450 ? asm_exc_general_protection+0x22/0x30 ? bio_copy_data_iter+0x186/0x260 __bch2_read_endio+0x83d/0x920 [bcachefs] ? process_one_work+0x192/0x4d0 ? process_one_work+0x1fc/0x4d0 process_one_work+0x1fc/0x4d0 worker_thread+0x1dd/0x3d0 ? __pfx_worker_thread+0x10/0x10 kthread+0x100/0x130 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x2d/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1b/0x30 </TASK> Modules linked in: bcachefs(OE) lz4_compress(E) lz4hc_compress(E) nft_fib_inet(E) nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E) nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E) nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E) nf_tables(E) nfnetlink(E) sunrpc(E) intel_rapl_msr(E) intel_rapl_common(E) intel_uncore_frequency_common(E) isst_if_common(E) kvm_intel(E) kvm(E) iTCO_wdt(E) intel_pmc_bxt(E) iTCO_vendor_support(E) irqbypass(E) virtio_balloon(E) rapl(E) pktcdvd(E) i2c_i801(E) pcspkr(E) i2c_smbus(E) lpc_ich(E) joydev(E) loop(E) zram(E) crct10dif_pclmul(E) crc32_pclmul(E) crc32c_intel(E) virtiofs(E) virtio_net(E) net_failover(E) ghash_clmulni_intel(E) virtio_console(E) failover(E) fuse(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) qemu_fw_cfg(E) Unloaded tainted modules: bcachefs(E):3 intel_uncore_frequency(E):1 isst_if_mbox_msr(E):1 [last unloaded: bcachefs(OE)] ---[ end trace 0000000000000000 ]--- RIP: 0010:bio_copy_data_iter+0x186/0x260 Code: 29 f6 48 c1 f9 06 48 c1 fe 06 48 c1 e1 0c 48 c1 e6 0c 48 01 e9 48 01 ee 48 01 d9 4c 01 d6 83 fa 08 0f 82 b0 fe ff ff 48 8b 06 <48> 89 01 89 d0 48 8b 7c 06 f8 48 89 7c 01 f8 48 8d 79 08 48 83 e7 RSP: 0018:ffffa31644a63d38 EFLAGS: 00010212 RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0280766500040001 RDX: 0000000000000c00 RSI: ffff8d462b83d400 RDI: ffff8d460fc9edd0 RBP: ffff8d4500000000 R08: ffffa31644a63dc0 R09: ffffa31644a63dd4 R10: 0000000000000400 R11: 0000000000000c00 R12: ffff8d462beb9aa8 R13: ffff8d460fc9ed38 R14: fffffc5e80000000 R15: 0000000000001000 FS: 0000000000000000(0000) GS:ffff8d64fdd80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f7255016120 CR3: 000000012534e003 CR4: 0000000000770ef0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400PKRU: 55555554
Quoted 10 lines · Brian Foster
Hi Kent,
JFYI, I'm seeing the following splat pretty reliably via generic/361 on an 80xcpu test box. The CI doesn't seem to produce this failure for whatever reason. This bisects down to commit
023f9ac9f70f("bcachefs:
Delete dio read alignment check"), before which the test still fails but the kernel doesn't explode.Brian
Can you test the following?
--- 8< ---
Subject: [PATCH] bcachefs: bios must be 512 byte algined
fs/bcachefs/fs-io-direct.c +4 -0
@@ -77,6 +77,10 @@ static int bch2_direct_IO_read(struct kiocb *req, struct iov_iter *iter)bch2_inode_opts_get (&opts, c, &inode->ei_inode); +/* bios must be 512 byte aligned: */ +if ((offset|iter->count) & (SECTOR_SIZE -1 )) +return -EINVAL ; + ret =min_t (loff_t, iter->count,max_t (loff_t,0 ,i_size_read (&inode->v) - offset));
Quoted 9 lines · Kent Overstreet
Quoted 8 lines · Brian Foster
Hi Kent,
JFYI, I'm seeing the following splat pretty reliably via generic/361 on an 80xcpu test box. The CI doesn't seem to produce this failure for whatever reason. This bisects down to commit
023f9ac9f70f("bcachefs:
Delete dio read alignment check"), before which the test still fails but the kernel doesn't explode.
Quoted 5 lines
Quoted 2 lines
Brian
Can you test the following?
Still blows up... repeated a couple times to be sure.
Brian
Quoted 23 lines
--- 8< ---
Subject: [PATCH] bcachefs: bios must be 512 byte alginedFixes 023f9ac9f70f bcachefs: Delete dio read alignment checkReported-by Brian Foster <[email protected]>Signed-off-by Kent Overstreet <[email protected]>fs/bcachefs/fs-io-direct.c +4
-0@@ -77,6 +77,10 @@ static int bch2_direct_IO_read(struct kiocb *req, struct iov_iter *iter)bch2_inode_opts_get (&opts, c, &inode->ei_inode); +/* bios must be 512 byte aligned: */ +if ((offset|iter->count) & (SECTOR_SIZE -1 )) +return -EINVAL ; + ret =min_t (loff_t, iter->count,max_t (loff_t,0 ,i_size_read (&inode->v) - offset));
Quoted 10 lines · Brian Foster
Quoted 9 lines · Kent Overstreet
Quoted 8 lines · Brian Foster
Hi Kent,
JFYI, I'm seeing the following splat pretty reliably via generic/361 on an 80xcpu test box. The CI doesn't seem to produce this failure for whatever reason. This bisects down to commit
023f9ac9f70f("bcachefs:
Delete dio read alignment check"), before which the test still fails but the kernel doesn't explode.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Brian
Can you test the following?
Still blows up... repeated a couple times to be sure.
That sounds like a driver bug then - what driver?
Quoted 14 lines · Kent Overstreet
Quoted 13 lines · Brian Foster
Quoted 12 lines · Kent Overstreet
Quoted 10 lines · Brian Foster
Hi Kent,
JFYI, I'm seeing the following splat pretty reliably via generic/361 on an 80xcpu test box. The CI doesn't seem to produce this failure for whatever reason. This bisects down to commit
023f9ac9f70f("bcachefs:
Delete dio read alignment check"), before which the test still fails but the kernel doesn't explode.Brian
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Can you test the following?
Still blows up... repeated a couple times to be sure.
That sounds like a driver bug then - what driver?
I think it's not a drive bug. It's related to bcachefs block_size. I can reproduce it by running generic/361 with block_size 4096.
The test devices are normal qemu disks backing by files in host. The bug disappears after hanging mkfs block_size to 512.
--
Su
Quoted 27 lines · Su Yue
Quoted 23 lines · Kent Overstreet
Quoted 21 lines · Brian Foster
Quoted 4 lines · Kent Overstreet
Quoted 3 lines · Brian Foster
Hi Kent,
JFYI, I'm seeing the following splat pretty reliably via > >
generic/361 on
Quoted 1 line
Quoted 1 line
an 80xcpu test box. The CI doesn't seem to produce this > >
failure for
Quoted 1 line
Quoted 1 line
whatever reason. This bisects down to commit
023f9ac9f70f> >("bcachefs:
Quoted 1 line
Quoted 1 line
Delete dio read alignment check"), before which the test > >
still fails but
Quoted 7 lines
Quoted 4 lines
the kernel doesn't explode.
Brian
Can you test the following?
Still blows up... repeated a couple times to be sure.
Quoted 9 lines
Quoted 1 line
That sounds like a driver bug then - what driver?
I think it's not a drive bug. It's related to bcachefs
block_size. I can reproduce it by running generic/361 withblock_size4096.The test devices are normal qemu disks backing by files in host. The bug disappears after hanging mkfs
block_sizeto 512.
Hi Su,
Yes, I think this is the issue as block_bytes(c) is 4k in my test. Thanks for testing/confirming.
The immediate reason for the crash appears to be bio_copy_data_iter() going off the rails trying to copy a larger source bio into a smaller destination bio with seemingly inconsistent size/bvec.
The broader context is we start with a sub-block sized read, so e.g. I see a request for 1024 bytes at offset 4096. This results in a bio with bi_size == 4096, however, because right after the alignment check that was removed we do this:
ret = min_t(loff_t, iter->count, max_t(loff_t, 0, i_size_read(&inode->v) - offset)); ... shorten = iov_iter_count(iter) - round_up(ret, block_bytes(c)); iter->count -= shorten;
... which appears to underflow the iter count and is presumably fixed up somewhere to cap at the block size. I'm assuming this ends up in a situation where an internally inconsistent iov_iter leads to a similarly broken bvec_iter on the bio, or otherwise this gets further confused when creating the bounce bio, but I haven't traced to that level of detail.
In any event, we end up in bio_copy_data_iter() with a destination bio of bi_size == 4k that only appears to have a single 1k sized bvec. The first 1k copies as expected, but this doesn't reduce bi_size to zero and so the iteration just continues to advance the bio vec index until it finds some garbage data it can infer as a non-zero bv_len bvec to copy into.
Brian
Quoted 3 lines
--
Su