Skip to contentSkip to conversation

Kernel Oops: kernel BUG at block/bio.c:52

DISCUSSION Victor Westerhuis 3 participants 7 messages Apr 20, 2021 – May 6, 2021 bcache (block layer cache)

lore
Share & tools
lore
#0 lore raw

Hi all!

Today I tried using bcache for the first time, but after a few minutes all accesses to my new bcache filesystem started hanging.

Looking in dmesg I found the following Oops:

[  533.649096] kernel BUG at block/bio.c:52!
[  533.649107] invalid opcode: 0000 [#1] SMP PTI
[  533.649111] CPU: 3 PID: 1161 Comm: transmission-qt Not tainted 5.12.0-rc8-personal #1
[  533.649115] Hardware name: LENOVO 20354/Lancer 5A5, BIOS 9BCN91WW 07/21/2015
[  533.649117] RIP: 0010:biovec_slab.part.0+0x5/0x10
[  533.649128] Code: 82 00 48 8b 7b f0 48 85 ff 74 e1 48 8b 07 48 89 43 f0 48 85 c0 75 ce 48 c7 43 f8 00 00 00 00 eb c4 5b 5d c3 90 0f 1f 44 00 00 <0f> 0b 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 83 7f 40 00 75
[  533.649132] RSP: 0018:ffffbc0cc3337950 EFLAGS: 00010206
[  533.649136] RAX: 000000000000017f RBX: ffffbc0cc3337984 RCX: 0000000000000100
[  533.649139] RDX: 0000000000000800 RSI: ffffbc0cc3337984 RDI: ffffa015d5300118
[  533.649141] RBP: 0000000000000800 R08: ffffa015d5300118 R09: ffffa015222d8d00
[  533.649143] R10: ffffa015d47d38f0 R11: 9000000002800000 R12: ffffa015d5300118
[  533.649145] R13: 0000000000000800 R14: ffffa015d53000d0 R15: 0000000000000800
[  533.649148] FS:  00007f68dab79700(0000) GS:ffffa01707380000(0000) knlGS:0000000000000000
[  533.649151] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  533.649153] CR2: 00007f84340df020 CR3: 0000000145538006 CR4: 00000000001706e0
[  533.649155] Call Trace:
[  533.649158]  bvec_alloc+0x90/0xc0
[  533.649166]  bio_alloc_bioset+0x130/0x1d0
[  533.649172]  cached_dev_cache_miss+0x100/0x300 [bcache]
[  533.649202]  cache_lookup_fn+0x119/0x300 [bcache]
[  533.649222]  ? bch_btree_iter_next_filter+0x1b0/0x2c0 [bcache]
[  533.649239]  ? bch_data_invalidate+0x160/0x160 [bcache]
[  533.649258]  bch_btree_map_keys_recurse+0x7e/0x180 [bcache]
[  533.649277]  ? mempool_alloc+0x62/0x170
[  533.649283]  bch_btree_map_keys+0x15c/0x1b0 [bcache]
[  533.649301]  ? bch_data_invalidate+0x160/0x160 [bcache]
[  533.649321]  cache_lookup+0x8a/0x140 [bcache]
[  533.649340]  cached_dev_submit_bio+0x964/0xc90 [bcache]
[  533.649361]  ? submit_bio_checks+0x1a2/0x570
[  533.649366]  ? mempool_alloc+0x62/0x170
[  533.649371]  submit_bio_noacct+0x122/0x4f0
[  533.649376]  ext4_mpage_readpages+0x1e0/0x8d0 [ext4]
[  533.649420]  ? __mod_memcg_lruvec_state+0x22/0xe0
[  533.649424]  ? __add_to_page_cache_locked+0x18f/0x370
[  533.649429]  read_pages+0x82/0x1e0
[  533.649433]  page_cache_ra_unbounded+0x197/0x1e0
[  533.649438]  force_page_cache_ra+0xda/0x140
[  533.649442]  generic_fadvise+0x190/0x280
[  533.649448]  __x64_sys_fadvise64+0x7c/0x90
[  533.649453]  do_syscall_64+0x33/0x40
[  533.649458]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[  533.649463] RIP: 0033:0x7f68eeab954a
[  533.649467] Code: ff eb 80 0f 1f 44 00 00 48 8b 15 41 a9 0c 00 f7 d8 64 89 02 b8 ff ff ff ff eb c3 e8 e0 a0 01 00 41 89 ca b8 dd 00 00 00 0f 05 <89> c2 f7 da 3d 00 f0 ff ff b8 00 00 00 00 0f 47 c2 c3 0f 1f 40 00
[  533.649470] RSP: 002b:00007f68dab787f8 EFLAGS: 00000246 ORIG_RAX: 00000000000000dd
[  533.649474] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f68eeab954a
[  533.649477] RDX: 0000000000800000 RSI: 0000000038ff17a8 RDI: 0000000000000028
[  533.649479] RBP: 0000000000000000 R08: 0000000000000003 R09: 0000000000000003
[  533.649481] R10: 0000000000000003 R11: 0000000000000246 R12: 0000000000000001
[  533.649483] R13: 0000000000000001 R14: 00007f68c434aaf8 R15: 00007f68c42dd950
[  533.649488] Modules linked in: des_generic libdes sha1_ssse3 sha1_generic md5 md4 bcache crc64 hid_generic usbhid ext4 crc32c_generic mbcache jbd2 pkcs8_key_parser fuse
[  533.649506] ---[ end trace 4d34ff6f2d05924c ]---
[  533.649509] RIP: 0010:biovec_slab.part.0+0x5/0x10
[  533.649515] Code: 82 00 48 8b 7b f0 48 85 ff 74 e1 48 8b 07 48 89 43 f0 48 85 c0 75 ce 48 c7 43 f8 00 00 00 00 eb c4 5b 5d c3 90 0f 1f 44 00 00 <0f> 0b 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 83 7f 40 00 75
[  533.649518] RSP: 0018:ffffbc0cc3337950 EFLAGS: 00010206
[  533.649522] RAX: 000000000000017f RBX: ffffbc0cc3337984 RCX: 0000000000000100
[  533.649524] RDX: 0000000000000800 RSI: ffffbc0cc3337984 RDI: ffffa015d5300118
[  533.649526] RBP: 0000000000000800 R08: ffffa015d5300118 R09: ffffa015222d8d00
[  533.649528] R10: ffffa015d47d38f0 R11: 9000000002800000 R12: ffffa015d5300118
[  533.649531] R13: 0000000000000800 R14: ffffa015d53000d0 R15: 0000000000000800
[  533.649533] FS:  00007f68dab79700(0000) GS:ffffa01707380000(0000) knlGS:0000000000000000
[  533.649536] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  533.649539] CR2: 00007f84340df020 CR3: 0000000145538006 CR4: 00000000001706e0

I've traced the error back to drivers/md/bcache/request.c:919, but I don't know the code well enough to trace it any further. It appears to me that bcache tries to allocate an invalid amount of memory, but like I said, I'm not sure what is going on exactly.

The problem hasn't appeared yet when running 5.11.15.

Regards,

Victor Westerhuis

#1 lore raw
Quoted 99 lines · Victor Westerhuis

Hi all!

Today I tried using bcache for the first time, but after a few minutes all accesses to my new bcache filesystem started hanging.

Looking in dmesg I found the following Oops:
[  533.649096] kernel BUG at block/bio.c:52!
[  533.649107] invalid opcode: 0000 [#1] SMP PTI
[  533.649111] CPU: 3 PID: 1161 Comm: transmission-qt Not tainted 5.12.0-rc8-personal #1
[  533.649115] Hardware name: LENOVO 20354/Lancer 5A5, BIOS 9BCN91WW 07/21/2015
[  533.649117] RIP: 0010:biovec_slab.part.0+0x5/0x10
[  533.649128] Code: 82 00 48 8b 7b f0 48 85 ff 74 e1 48 8b 07 48 89 43 f0 48 85 c0 75 ce 48 c7 43 f8 00 00 00 00 eb c4 5b 5d c3 90 0f 1f 44 00 00 <0f> 0b 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 83 7f 40 00 75 [  533.649132] RSP: 0018:ffffbc0cc3337950 EFLAGS: 00010206
[  533.649136] RAX: 000000000000017f RBX: ffffbc0cc3337984 RCX:
0000000000000100
[  533.649139] RDX: 0000000000000800 RSI: ffffbc0cc3337984 RDI:
ffffa015d5300118
[  533.649141] RBP: 0000000000000800 R08: ffffa015d5300118 R09:
ffffa015222d8d00
[  533.649143] R10: ffffa015d47d38f0 R11: 9000000002800000 R12:
ffffa015d5300118
[  533.649145] R13: 0000000000000800 R14: ffffa015d53000d0 R15:
0000000000000800
[  533.649148] FS:  00007f68dab79700(0000) GS:ffffa01707380000(0000) knlGS:0000000000000000
[  533.649151] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [  533.649153] CR2: 00007f84340df020 CR3: 0000000145538006 CR4:
00000000001706e0
[  533.649155] Call Trace:
[  533.649158]  bvec_alloc+0x90/0xc0
[  533.649166]  bio_alloc_bioset+0x130/0x1d0
[  533.649172]  cached_dev_cache_miss+0x100/0x300 [bcache]
[  533.649202]  cache_lookup_fn+0x119/0x300 [bcache]
[  533.649222]  ? bch_btree_iter_next_filter+0x1b0/0x2c0 [bcache] [  533.649239]  ? bch_data_invalidate+0x160/0x160 [bcache]
[  533.649258]  bch_btree_map_keys_recurse+0x7e/0x180 [bcache] [  533.649277]  ? mempool_alloc+0x62/0x170
[  533.649283]  bch_btree_map_keys+0x15c/0x1b0 [bcache]
[  533.649301]  ? bch_data_invalidate+0x160/0x160 [bcache]
[  533.649321]  cache_lookup+0x8a/0x140 [bcache]
[  533.649340]  cached_dev_submit_bio+0x964/0xc90 [bcache]
[  533.649361]  ? submit_bio_checks+0x1a2/0x570
[  533.649366]  ? mempool_alloc+0x62/0x170
[  533.649371]  submit_bio_noacct+0x122/0x4f0
[  533.649376]  ext4_mpage_readpages+0x1e0/0x8d0 [ext4]
[  533.649420]  ? __mod_memcg_lruvec_state+0x22/0xe0
[  533.649424]  ? __add_to_page_cache_locked+0x18f/0x370
[  533.649429]  read_pages+0x82/0x1e0
[  533.649433]  page_cache_ra_unbounded+0x197/0x1e0
[  533.649438]  force_page_cache_ra+0xda/0x140
[  533.649442]  generic_fadvise+0x190/0x280
[  533.649448]  __x64_sys_fadvise64+0x7c/0x90
[  533.649453]  do_syscall_64+0x33/0x40
[  533.649458]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[  533.649463] RIP: 0033:0x7f68eeab954a
[  533.649467] Code: ff eb 80 0f 1f 44 00 00 48 8b 15 41 a9 0c 00 f7 d8 64 89 02 b8 ff ff ff ff eb c3 e8 e0 a0 01 00 41 89 ca b8 dd 00 00 00 0f 05 <89> c2 f7 da 3d 00 f0 ff ff b8 00 00 00 00 0f 47 c2 c3 0f 1f 40 00 [  533.649470] RSP: 002b:00007f68dab787f8 EFLAGS: 00000246 ORIG_RAX:
00000000000000dd
[  533.649474] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
00007f68eeab954a
[  533.649477] RDX: 0000000000800000 RSI: 0000000038ff17a8 RDI:
0000000000000028
[  533.649479] RBP: 0000000000000000 R08: 0000000000000003 R09:
0000000000000003
[  533.649481] R10: 0000000000000003 R11: 0000000000000246 R12:
0000000000000001
[  533.649483] R13: 0000000000000001 R14: 00007f68c434aaf8 R15:
00007f68c42dd950
[  533.649488] Modules linked in: des_generic libdes sha1_ssse3 sha1_generic md5 md4 bcache crc64 hid_generic usbhid ext4 crc32c_generic mbcache jbd2 pkcs8_key_parser fuse
[  533.649506] ---[ end trace 4d34ff6f2d05924c ]---
[  533.649509] RIP: 0010:biovec_slab.part.0+0x5/0x10
[  533.649515] Code: 82 00 48 8b 7b f0 48 85 ff 74 e1 48 8b 07 48 89 43 f0 48 85 c0 75 ce 48 c7 43 f8 00 00 00 00 eb c4 5b 5d c3 90 0f 1f 44 00 00 <0f> 0b 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 83 7f 40 00 75 [  533.649518] RSP: 0018:ffffbc0cc3337950 EFLAGS: 00010206
[  533.649522] RAX: 000000000000017f RBX: ffffbc0cc3337984 RCX:
0000000000000100
[  533.649524] RDX: 0000000000000800 RSI: ffffbc0cc3337984 RDI:
ffffa015d5300118
[  533.649526] RBP: 0000000000000800 R08: ffffa015d5300118 R09:
ffffa015222d8d00
[  533.649528] R10: ffffa015d47d38f0 R11: 9000000002800000 R12:
ffffa015d5300118
[  533.649531] R13: 0000000000000800 R14: ffffa015d53000d0 R15:
0000000000000800
[  533.649533] FS:  00007f68dab79700(0000) GS:ffffa01707380000(0000) knlGS:0000000000000000
[  533.649536] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [  533.649539] CR2: 00007f84340df020 CR3: 0000000145538006 CR4:
00000000001706e0

Quoted 6 lines

I've traced the error back to  drivers/md/bcache/request.c:919, but I don't know the code well enough to trace it any further. It appears to me that bcache tries to allocate an invalid amount of memory, but like I said, I'm not sure what is going on exactly.

The problem hasn't appeared yet when running 5.11.15.

Could you please help to apply a debug patch and gather some debug information when it reproduces ?

Thanks.

Coly Li

#2 lore raw

Hi,

I'm getting the same issue on kernel 5.12.0 after upgrading from 5.11.16. For me, so far the error always occurs a short while after boot.

Quoted 2 lines

Could you please help to apply a debug patch and gather some debug information when it reproduces ?

I could do that as well, which patch should I apply?

Thanks,
Marco

#3 lore raw
Quoted 10 lines · Marco Rebhan

Hi,

I'm getting the same issue on kernel 5.12.0 after upgrading from 5.11.16. For me, so far the error always occurs a short while after boot.

Quoted 2 lines

Could you please help to apply a debug patch and gather some debug information when it reproduces ?

I could do that as well, which patch should I apply?

Could you please try the attached patch ? If a suspicious bio allocation happens, this patch will print out a warning kernel message and avoid the BUG() panic.

Thank you in advance.

Coly Li

From 6f2edee7100efabf2ccccb84e4a92ccbfbddd8c5 Mon Sep 17 00:00:00 2001
From: Coly Li <[email protected]>
Date: Thu, 6 May 2021 10:38:41 +0800
Subject: [PATCH] bcache: avoid oversized bio_alloc_bioset() call in
cached_dev_cache_miss()

Since Linux v5.12, calling bio_alloc_bioset() with oversized bio vectors number will cause a BUG() panic in biovec_slab(). There are 2 locations in bcache code calling bio_alloc_bioset(), and only the location in cached_dev_cache_miss() has such potential oversized risk.

In cached_dev_cache_miss() the bio vectors number is calculated by DIV_ROUND_UP(s->insert_bio_sectors, PAGE_SECTORS), this patch checks the calculated result, if it is larger than BIO_MAX_VECS, then give up the allocation of cache_bio and sending request to backing device directly.

By this restriction, the potential BUG() panic can be avoided from the cache missing code path.

---
 drivers/md/bcache/request.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
drivers/md/bcache/request.c +9 -4
index 29c231758293..a657d3a2b624 100644
@@ -879,7 +879,7 @@ static void cached_dev_read_done_bh(struct closure *cl)
 static int cached_dev_cache_miss(struct btree *b, struct search *s,
 				 struct bio *bio, unsigned int sectors)
 {
-	int ret = MAP_CONTINUE;
+	int ret = MAP_CONTINUE, nr_iovecs = 0;
 	unsigned int reada = 0;
 	struct cached_dev *dc = container_of(s->d, struct cached_dev, disk);
 	struct bio *miss, *cache_bio;
@@ -916,9 +916,14 @@ static int cached_dev_cache_miss(struct btree *b, struct search *s,
 	/* btree_search_recurse()'s btree iterator is no good anymore */
 	ret = miss == bio ? MAP_DONE : -EINTR;
 
-	cache_bio = bio_alloc_bioset(GFP_NOWAIT,
-			DIV_ROUND_UP(s->insert_bio_sectors, PAGE_SECTORS),
-			&dc->disk.bio_split);
+	nr_iovecs = DIV_ROUND_UP(s->insert_bio_sectors, PAGE_SECTORS);
+	if (nr_iovecs > BIO_MAX_VECS) {
+		pr_warn("inserting bio is too large: %d iovecs, not intsert.\n",
+			nr_iovecs);
+		goto out_submit;
+	}
+	cache_bio = bio_alloc_bioset(GFP_NOWAIT, nr_iovecs,
+				     &dc->disk.bio_split);
 	if (!cache_bio)
 		goto out_submit;
 
Signature · 1 line
2.26.2
#4 lore raw
Quoted 3 lines · On Thursday, 6 May 2021 04:50:06 CEST Coly Li wrote:

Could you please try the attached patch ? If a suspicious bio allocation happens, this patch will print out a warning kernel message and avoid the BUG() panic.

Looks like the patch works. Here's a dmesg log that comes from starting up a game with a bunch of large files (which I'm guessing are what makes this happen more often?)

[   39.284230] bcache: cached_dev_cache_miss() inserting bio is too large: 344 iovecs, not intsert.
[   65.415896] bcache: cached_dev_cache_miss() inserting bio is too large: 282 iovecs, not intsert.
[   65.446327] bcache: cached_dev_cache_miss() inserting bio is too large: 946 iovecs, not intsert.
[   88.116826] bcache: cached_dev_cache_miss() inserting bio is too large: 342 iovecs, not intsert.
[   88.957691] bcache: cached_dev_cache_miss() inserting bio is too large: 342 iovecs, not intsert.
[   89.020544] bcache: cached_dev_cache_miss() inserting bio is too large: 332 iovecs, not intsert.
[   90.531875] bcache: cached_dev_cache_miss() inserting bio is too large: 261 iovecs, not intsert.
[  111.464124] bcache: cached_dev_cache_miss() inserting bio is too large: 342 iovecs, not intsert.
[  111.497049] bcache: cached_dev_cache_miss() inserting bio is too large: 262 iovecs, not intsert.
[  111.638928] bcache: cached_dev_cache_miss() inserting bio is too large: 318 iovecs, not intsert.
[  155.884142] bcache: cached_dev_cache_miss() inserting bio is too large: 447 iovecs, not intsert.
[  156.146070] bcache: cached_dev_cache_miss() inserting bio is too large: 512 iovecs, not intsert.
[  156.223795] bcache: cached_dev_cache_miss() inserting bio is too large: 277 iovecs, not intsert.
[  156.326145] bcache: cached_dev_cache_miss() inserting bio is too large: 342 iovecs, not intsert.
[  156.602906] bcache: cached_dev_cache_miss() inserting bio is too large: 290 iovecs, not intsert.
[  156.646365] bcache: cached_dev_cache_miss() inserting bio is too large: 341 iovecs, not intsert.
[  156.671285] bcache: cached_dev_cache_miss() inserting bio is too large: 501 iovecs, not intsert.
[  157.216087] bcache: cached_dev_cache_miss() inserting bio is too large: 258 iovecs, not intsert.
[  165.010961] bcache: cached_dev_cache_miss() inserting bio is too large: 413 iovecs, not intsert.
[  165.386483] bcache: cached_dev_cache_miss() inserting bio is too large: 260 iovecs, not intsert.

Thanks,
Marco

#5 lore raw
Quoted 9 lines · Marco Rebhan
Quoted 3 lines · On Thursday, 6 May 2021 04:50:06 CEST Coly Li wrote:

Could you please try the attached patch ? If a suspicious bio allocation happens, this patch will print out a warning kernel message and avoid the BUG() panic.

Looks like the patch works. Here's a dmesg log that comes from starting up a game with a bunch of large files (which I'm guessing are what makes this happen more often?)

Before 5.12, the allocation failure returns a NULL pointer, so such issue was hidden. I will post a quick fix to emulate previous code logic without bothering bio_alloc_bioset().

Thanks for the quick test and verification.

Coly Li

Quoted 24 lines
[   39.284230] bcache: cached_dev_cache_miss() inserting bio is too large: 344 iovecs, not intsert.
[   65.415896] bcache: cached_dev_cache_miss() inserting bio is too large: 282 iovecs, not intsert.
[   65.446327] bcache: cached_dev_cache_miss() inserting bio is too large: 946 iovecs, not intsert.
[   88.116826] bcache: cached_dev_cache_miss() inserting bio is too large: 342 iovecs, not intsert.
[   88.957691] bcache: cached_dev_cache_miss() inserting bio is too large: 342 iovecs, not intsert.
[   89.020544] bcache: cached_dev_cache_miss() inserting bio is too large: 332 iovecs, not intsert.
[   90.531875] bcache: cached_dev_cache_miss() inserting bio is too large: 261 iovecs, not intsert.
[  111.464124] bcache: cached_dev_cache_miss() inserting bio is too large: 342 iovecs, not intsert.
[  111.497049] bcache: cached_dev_cache_miss() inserting bio is too large: 262 iovecs, not intsert.
[  111.638928] bcache: cached_dev_cache_miss() inserting bio is too large: 318 iovecs, not intsert.
[  155.884142] bcache: cached_dev_cache_miss() inserting bio is too large: 447 iovecs, not intsert.
[  156.146070] bcache: cached_dev_cache_miss() inserting bio is too large: 512 iovecs, not intsert.
[  156.223795] bcache: cached_dev_cache_miss() inserting bio is too large: 277 iovecs, not intsert.
[  156.326145] bcache: cached_dev_cache_miss() inserting bio is too large: 342 iovecs, not intsert.
[  156.602906] bcache: cached_dev_cache_miss() inserting bio is too large: 290 iovecs, not intsert.
[  156.646365] bcache: cached_dev_cache_miss() inserting bio is too large: 341 iovecs, not intsert.
[  156.671285] bcache: cached_dev_cache_miss() inserting bio is too large: 501 iovecs, not intsert.
[  157.216087] bcache: cached_dev_cache_miss() inserting bio is too large: 258 iovecs, not intsert.
[  165.010961] bcache: cached_dev_cache_miss() inserting bio is too large: 413 iovecs, not intsert.
[  165.386483] bcache: cached_dev_cache_miss() inserting bio is too large: 260 iovecs, not intsert.

Thanks,
Marco

#6 lore raw
Quoted 3 lines · On Thursday, 6 May 2021 12:04:00 CEST Coly Li wrote:

Before 5.12, the allocation failure returns a NULL pointer, so such issue was hidden. I will post a quick fix to emulate previous code logic without bothering bio_alloc_bioset().

Ah, I see. I guess that means then I can actually use this patch until that fix is mainlined :)

Quoted 1 line

Thanks for the quick test and verification.

Glad to help!

Thanks,
Marco