Skip to contentSkip to conversation

general protection fault, probably for non-canonical address 0x280766500040001: 0000 [#1] PREEMPT SMP PTI

BUG Brian Foster 3 participants 7 messages Jan 16, 2024 – Jan 17, 2024

lore
Share & tools
lore
#0 lore raw

Hi Kent,

JFYI, I'm seeing the following splat pretty reliably via generic/361 on an 80xcpu test box. The CI doesn't seem to produce this failure for whatever reason. This bisects down to commit 023f9ac9f70f ("bcachefs:
Delete dio read alignment check"), before which the test still fails but the kernel doesn't explode.

Brian

--- 8< ---

general protection fault, probably for non-canonical address 0x280766500040001: 0000 [#1] PREEMPT SMP PTI
CPU: 15 PID: 1078 Comm: kworker/15:1H Tainted: G           OE      6.7.0-rc7+ #68 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc37 04/01/2014
Workqueue: events_highpri __bch2_read_endio [bcachefs]
RIP: 0010:bio_copy_data_iter+0x186/0x260
Code: 29 f6 48 c1 f9 06 48 c1 fe 06 48 c1 e1 0c 48 c1 e6 0c 48 01 e9 48 01 ee 48 01 d9 4c 01 d6 83 fa 08 0f 82 b0 fe ff ff 48 8b 06 <48> 89 01 89 d0 48 8b 7c 06 f8 48 89 7c 01 f8 48 8d 79 08 48 83 e7 
RSP: 0018:ffffa31644a63d38 EFLAGS: 00010212
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0280766500040001
RDX: 0000000000000c00 RSI: ffff8d462b83d400 RDI: ffff8d460fc9edd0
RBP: ffff8d4500000000 R08: ffffa31644a63dc0 R09: ffffa31644a63dd4
R10: 0000000000000400 R11: 0000000000000c00 R12: ffff8d462beb9aa8
R13: ffff8d460fc9ed38 R14: fffffc5e80000000 R15: 0000000000001000
FS:  0000000000000000(0000) GS:ffff8d64fdd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7255016120 CR3: 000000012534e003 CR4: 0000000000770ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554 
Call Trace:
 <TASK>
 ? die_addr+0x32/0x80
 ? exc_general_protection+0x19b/0x450
 ? asm_exc_general_protection+0x22/0x30
 ? bio_copy_data_iter+0x186/0x260
 __bch2_read_endio+0x83d/0x920 [bcachefs]
 ? process_one_work+0x192/0x4d0
 ? process_one_work+0x1fc/0x4d0
 process_one_work+0x1fc/0x4d0
 worker_thread+0x1dd/0x3d0
 ? __pfx_worker_thread+0x10/0x10
 kthread+0x100/0x130
 ? __pfx_kthread+0x10/0x10
 ret_from_fork+0x2d/0x50
 ? __pfx_kthread+0x10/0x10
 ret_from_fork_asm+0x1b/0x30
 </TASK>
Modules linked in: bcachefs(OE) lz4_compress(E) lz4hc_compress(E) nft_fib_inet(E) nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E) nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E) nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E) nf_tables(E) nfnetlink(E) sunrpc(E) intel_rapl_msr(E) intel_rapl_common(E) intel_uncore_frequency_common(E) isst_if_common(E) kvm_intel(E) kvm(E) iTCO_wdt(E) intel_pmc_bxt(E) iTCO_vendor_support(E) irqbypass(E) virtio_balloon(E) rapl(E) pktcdvd(E) i2c_i801(E) pcspkr(E) i2c_smbus(E) lpc_ich(E) joydev(E) loop(E) zram(E) crct10dif_pclmul(E) crc32_pclmul(E) crc32c_intel(E) virtiofs(E) virtio_net(E) net_failover(E) ghash_clmulni_intel(E) virtio_console(E) failover(E) fuse(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) qemu_fw_cfg(E) 
Unloaded tainted modules: bcachefs(E):3 intel_uncore_frequency(E):1 isst_if_mbox_msr(E):1 [last unloaded: bcachefs(OE)]
---[ end trace 0000000000000000 ]---
RIP: 0010:bio_copy_data_iter+0x186/0x260
Code: 29 f6 48 c1 f9 06 48 c1 fe 06 48 c1 e1 0c 48 c1 e6 0c 48 01 e9 48 01 ee 48 01 d9 4c 01 d6 83 fa 08 0f 82 b0 fe ff ff 48 8b 06 <48> 89 01 89 d0 48 8b 7c 06 f8 48 89 7c 01 f8 48 8d 79 08 48 83 e7 
RSP: 0018:ffffa31644a63d38 EFLAGS: 00010212
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0280766500040001
RDX: 0000000000000c00 RSI: ffff8d462b83d400 RDI: ffff8d460fc9edd0
RBP: ffff8d4500000000 R08: ffffa31644a63dc0 R09: ffffa31644a63dd4
R10: 0000000000000400 R11: 0000000000000c00 R12: ffff8d462beb9aa8
R13: ffff8d460fc9ed38 R14: fffffc5e80000000 R15: 0000000000001000
FS:  0000000000000000(0000) GS:ffff8d64fdd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7255016120 CR3: 000000012534e003 CR4: 0000000000770ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

PKRU: 55555554

#1 lore raw
Quoted 7 lines · Brian Foster

Hi Kent,

JFYI, I'm seeing the following splat pretty reliably via generic/361 on an 80xcpu test box. The CI doesn't seem to produce this failure for whatever reason. This bisects down to commit 023f9ac9f70f ("bcachefs:
Delete dio read alignment check"), before which the test still fails but the kernel doesn't explode.

Ahh, that probably needs to be a 512 byte alignment check; since the block layer alternates units between bytes and 512 byte sectors things will get very confused if the IO wasn't 512 byte aligned.

Quoted 59 lines

Brian

--- 8< ---

general protection fault, probably for non-canonical address 0x280766500040001: 0000 [#1] PREEMPT SMP PTI
CPU: 15 PID: 1078 Comm: kworker/15:1H Tainted: G           OE      6.7.0-rc7+ #68 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc37 04/01/2014
Workqueue: events_highpri __bch2_read_endio [bcachefs]
RIP: 0010:bio_copy_data_iter+0x186/0x260
Code: 29 f6 48 c1 f9 06 48 c1 fe 06 48 c1 e1 0c 48 c1 e6 0c 48 01 e9 48 01 ee 48 01 d9 4c 01 d6 83 fa 08 0f 82 b0 fe ff ff 48 8b 06 <48> 89 01 89 d0 48 8b 7c 06 f8 48 89 7c 01 f8 48 8d 79 08 48 83 e7 
RSP: 0018:ffffa31644a63d38 EFLAGS: 00010212
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0280766500040001
RDX: 0000000000000c00 RSI: ffff8d462b83d400 RDI: ffff8d460fc9edd0
RBP: ffff8d4500000000 R08: ffffa31644a63dc0 R09: ffffa31644a63dd4
R10: 0000000000000400 R11: 0000000000000c00 R12: ffff8d462beb9aa8
R13: ffff8d460fc9ed38 R14: fffffc5e80000000 R15: 0000000000001000
FS:  0000000000000000(0000) GS:ffff8d64fdd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7255016120 CR3: 000000012534e003 CR4: 0000000000770ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554 
Call Trace:
 <TASK>
 ? die_addr+0x32/0x80
 ? exc_general_protection+0x19b/0x450
 ? asm_exc_general_protection+0x22/0x30
 ? bio_copy_data_iter+0x186/0x260
 __bch2_read_endio+0x83d/0x920 [bcachefs]
 ? process_one_work+0x192/0x4d0
 ? process_one_work+0x1fc/0x4d0
 process_one_work+0x1fc/0x4d0
 worker_thread+0x1dd/0x3d0
 ? __pfx_worker_thread+0x10/0x10
 kthread+0x100/0x130
 ? __pfx_kthread+0x10/0x10
 ret_from_fork+0x2d/0x50
 ? __pfx_kthread+0x10/0x10
 ret_from_fork_asm+0x1b/0x30
 </TASK>
Modules linked in: bcachefs(OE) lz4_compress(E) lz4hc_compress(E) nft_fib_inet(E) nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E) nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E) nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E) nf_tables(E) nfnetlink(E) sunrpc(E) intel_rapl_msr(E) intel_rapl_common(E) intel_uncore_frequency_common(E) isst_if_common(E) kvm_intel(E) kvm(E) iTCO_wdt(E) intel_pmc_bxt(E) iTCO_vendor_support(E) irqbypass(E) virtio_balloon(E) rapl(E) pktcdvd(E) i2c_i801(E) pcspkr(E) i2c_smbus(E) lpc_ich(E) joydev(E) loop(E) zram(E) crct10dif_pclmul(E) crc32_pclmul(E) crc32c_intel(E) virtiofs(E) virtio_net(E) net_failover(E) ghash_clmulni_intel(E) virtio_console(E) failover(E) fuse(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) qemu_fw_cfg(E) 
Unloaded tainted modules: bcachefs(E):3 intel_uncore_frequency(E):1 isst_if_mbox_msr(E):1 [last unloaded: bcachefs(OE)]
---[ end trace 0000000000000000 ]---
RIP: 0010:bio_copy_data_iter+0x186/0x260
Code: 29 f6 48 c1 f9 06 48 c1 fe 06 48 c1 e1 0c 48 c1 e6 0c 48 01 e9 48 01 ee 48 01 d9 4c 01 d6 83 fa 08 0f 82 b0 fe ff ff 48 8b 06 <48> 89 01 89 d0 48 8b 7c 06 f8 48 89 7c 01 f8 48 8d 79 08 48 83 e7 
RSP: 0018:ffffa31644a63d38 EFLAGS: 00010212
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0280766500040001
RDX: 0000000000000c00 RSI: ffff8d462b83d400 RDI: ffff8d460fc9edd0
RBP: ffff8d4500000000 R08: ffffa31644a63dc0 R09: ffffa31644a63dd4
R10: 0000000000000400 R11: 0000000000000c00 R12: ffff8d462beb9aa8
R13: ffff8d460fc9ed38 R14: fffffc5e80000000 R15: 0000000000001000
FS:  0000000000000000(0000) GS:ffff8d64fdd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7255016120 CR3: 000000012534e003 CR4: 0000000000770ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

PKRU: 55555554

#2 lore raw
In reply to Brian Foster
Quoted 10 lines · Brian Foster

Hi Kent,

JFYI, I'm seeing the following splat pretty reliably via generic/361 on an 80xcpu test box. The CI doesn't seem to produce this failure for whatever reason. This bisects down to commit 023f9ac9f70f ("bcachefs:
Delete dio read alignment check"), before which the test still fails but the kernel doesn't explode.

Brian

Can you test the following?

--- 8< ---
Subject: [PATCH] bcachefs: bios must be 512 byte algined

Fixes 023f9ac9f70f bcachefs: Delete dio read alignment check
fs/bcachefs/fs-io-direct.c +4 -0
index fdd57c5785c9..e3b219e19e10 100644
@@ -77,6 +77,10 @@ static int bch2_direct_IO_read(struct kiocb *req, struct iov_iter *iter)
 
 	bch2_inode_opts_get(&opts, c, &inode->ei_inode);
 
+	/* bios must be 512 byte aligned: */
+	if ((offset|iter->count) & (SECTOR_SIZE - 1))
+		return -EINVAL;
+
 	ret = min_t(loff_t, iter->count,
 		    max_t(loff_t, 0, i_size_read(&inode->v) - offset));
 
#3 lore raw
Quoted 9 lines · Kent Overstreet
Quoted 8 lines · Brian Foster

Hi Kent,

JFYI, I'm seeing the following splat pretty reliably via generic/361 on an 80xcpu test box. The CI doesn't seem to produce this failure for whatever reason. This bisects down to commit 023f9ac9f70f ("bcachefs:
Delete dio read alignment check"), before which the test still fails but the kernel doesn't explode.

Quoted 5 lines
Quoted 2 lines

Brian

Can you test the following?

Still blows up... repeated a couple times to be sure.

Brian

Quoted 23 lines

--- 8< ---
Subject: [PATCH] bcachefs: bios must be 512 byte algined

Fixes 023f9ac9f70f bcachefs: Delete dio read alignment check
fs/bcachefs/fs-io-direct.c +4 -0
index fdd57c5785c9..e3b219e19e10 100644
@@ -77,6 +77,10 @@ static int bch2_direct_IO_read(struct kiocb *req, struct iov_iter *iter)
 
 	bch2_inode_opts_get(&opts, c, &inode->ei_inode);
 
+	/* bios must be 512 byte aligned: */
+	if ((offset|iter->count) & (SECTOR_SIZE - 1))
+		return -EINVAL;
+
 	ret = min_t(loff_t, iter->count,
 		    max_t(loff_t, 0, i_size_read(&inode->v) - offset));
 
#4 lore raw
Quoted 10 lines · Brian Foster
Quoted 9 lines · Kent Overstreet
Quoted 8 lines · Brian Foster

Hi Kent,

JFYI, I'm seeing the following splat pretty reliably via generic/361 on an 80xcpu test box. The CI doesn't seem to produce this failure for whatever reason. This bisects down to commit 023f9ac9f70f ("bcachefs:
Delete dio read alignment check"), before which the test still fails but the kernel doesn't explode.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Brian

Can you test the following?

Still blows up... repeated a couple times to be sure.

That sounds like a driver bug then - what driver?

#5 lore raw
Quoted 14 lines · Kent Overstreet
Quoted 13 lines · Brian Foster
Quoted 12 lines · Kent Overstreet
Quoted 10 lines · Brian Foster

Hi Kent,

JFYI, I'm seeing the following splat pretty reliably via generic/361 on an 80xcpu test box. The CI doesn't seem to produce this failure for whatever reason. This bisects down to commit 023f9ac9f70f ("bcachefs:
Delete dio read alignment check"), before which the test still fails but the kernel doesn't explode.

Brian

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Can you test the following?

Still blows up... repeated a couple times to be sure.

That sounds like a driver bug then - what driver?

I think it's not a drive bug. It's related to bcachefs block_size. I can reproduce it by running generic/361 with block_size 4096.

The test devices are normal qemu disks backing by files in host. The bug disappears after hanging mkfs block_size to 512.

--
Su

#6 lore raw
Quoted 27 lines · Su Yue
Quoted 23 lines · Kent Overstreet
Quoted 21 lines · Brian Foster
Quoted 4 lines · Kent Overstreet
Quoted 3 lines · Brian Foster

Hi Kent,

JFYI, I'm seeing the following splat pretty reliably via > >

generic/361 on

Quoted 1 line
Quoted 1 line

an 80xcpu test box. The CI doesn't seem to produce this > >

failure for

Quoted 1 line
Quoted 1 line

whatever reason. This bisects down to commit 023f9ac9f70f > >

("bcachefs:

Quoted 1 line
Quoted 1 line

Delete dio read alignment check"), before which the test > >

still fails but

Quoted 7 lines
Quoted 4 lines

the kernel doesn't explode.

Brian

Can you test the following?

Still blows up... repeated a couple times to be sure.

Quoted 9 lines
Quoted 1 line

That sounds like a driver bug then - what driver?

I think it's not a drive bug. It's related to bcachefs block_size. I can reproduce it by running generic/361 with block_size 4096.

The test devices are normal qemu disks backing by files in host. The bug disappears after hanging mkfs block_size to 512.

Hi Su,

Yes, I think this is the issue as block_bytes(c) is 4k in my test. Thanks for testing/confirming.

The immediate reason for the crash appears to be bio_copy_data_iter() going off the rails trying to copy a larger source bio into a smaller destination bio with seemingly inconsistent size/bvec.

The broader context is we start with a sub-block sized read, so e.g. I see a request for 1024 bytes at offset 4096. This results in a bio with bi_size == 4096, however, because right after the alignment check that was removed we do this:

	ret = min_t(loff_t, iter->count,
		    max_t(loff_t, 0, i_size_read(&inode->v) - offset));
	...
	shorten = iov_iter_count(iter) - round_up(ret, block_bytes(c));
	iter->count -= shorten;

... which appears to underflow the iter count and is presumably fixed up somewhere to cap at the block size. I'm assuming this ends up in a situation where an internally inconsistent iov_iter leads to a similarly broken bvec_iter on the bio, or otherwise this gets further confused when creating the bounce bio, but I haven't traced to that level of detail.

In any event, we end up in bio_copy_data_iter() with a destination bio of bi_size == 4k that only appears to have a single 1k sized bvec. The first 1k copies as expected, but this doesn't reduce bi_size to zero and so the iteration just continues to advance the bio vec index until it finds some garbage data it can infer as a non-zero bv_len bvec to copy into.

Brian

Quoted 3 lines

--
Su