Skip to contentSkip to conversation

blk-mq crash with dm-multipath in for-3.20/core

PATCH In review Dongsu Park 4 participants 7 messages Feb 9, 2015 – Feb 10, 2015 Device mapper

lore
Share & tools
lore
#0 lore raw

Hi Jens,

during testing with the linux-block for-3.20/core branch, I hit a BUG like below. It's reproducible by running xfstests/xfs/279.

Bisecting showed that the first bad commit is 6d6285c45f5a ("block:
require blk_rq_prep_clone() be given an initialized clone request"). With reverting this commit, the crash disappears.
The linux-dm's branch dm-for-3.20 works fine without crash too.

As pointed out already by Keith Busch in a thread, [1] that commit should not be there in the first place. Commit 102e38b1030e ("dm: split request structure out from dm_rq_target_io structure") from linux-dm tree
[2] is going to move the blk_rq_init() call again to __clone_rq().

So that commit 6d6285c45f5a should be either reverted, or moved to linux-dm tree, doesn't it?

Cheers,
Dongsu

[1] https://www.redhat.com/archives/dm-devel/2015-January/msg00171.html
[2] https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-for-3.20&id=102e38b1030e883efc022dfdc7b7e7a3de70d1c5

------------[ cut here ]------------
kernel BUG at block/blk-core.c:2333!
RIP: 0010: [<ffffffff814c6858>] blk_dequeue_request+0x78/0x90
Call Trace:
 [<ffffffff814c6886>] blk_start_request+0x16/0x70
 [<ffffffff8169f9fa>] dm_start_request+0x1a/0x50
 [<ffffffff8169fce6>] dm_request_fn+0x2b6/0x3e0
 [<ffffffff814c0087>] __blk_run_queue+0x37/0x50
 [<ffffffff814c31ed>] queue_unplugged+0x5d/0x230
 [<ffffffff814c710c>] blk_flush_plug_list+0x1ac/0x230
 [<ffffffff814c7708>] blk_finish_plug+0x18/0x60
 [<ffffffff811baea1>] __do_page_cache_readahead+0x2b1/0x320
 [<ffffffff811bad55>] ? __do_page_cache_readahead+0x165/0x320
 [<ffffffff811baff2>] ondemand_readahead+0xe2/0x480
 [<ffffffff811ac3ff>] ? pagecache_get_page+0x2f/0x200
 [<ffffffff811bb4c1>] page_cache_sync_readahead+0x31/0x50
 [<ffffffff811ad5bc>] generic_file_read_iter+0x51c/0x630
 [<ffffffff811dd00e>] ? might_fault+0x5e/0xc0
 [<ffffffff81261e37>] blkdev_read_iter+0x37/0x40
 [<ffffffff8121fa4e>] new_sync_read+0x7e/0xb0
 [<ffffffff81220ce8>] __vfs_read+0x18/0x50
 [<ffffffff81220dad>] vfs_read+0x8d/0x150
 [<ffffffff81220eb9>] SyS_read+0x49/0xb0
 [<ffffffff817dce52>] system_call_fastpath+0x12/0x17
RIP  [<ffffffff814c6858>] blk_dequeue_request+0x78/0x90
 RSP <ffff88006e1eba68>
---[ end trace dcfc3d438518b1aa ]---
#1 lore raw
Quoted 15 lines · Dongsu Park

Hi Jens,

during testing with the linux-block for-3.20/core branch, I hit a BUG like below. It's reproducible by running xfstests/xfs/279.

Bisecting showed that the first bad commit is 6d6285c45f5a ("block:
require blk_rq_prep_clone() be given an initialized clone request"). With reverting this commit, the crash disappears.
The linux-dm's branch dm-for-3.20 works fine without crash too.

As pointed out already by Keith Busch in a thread, [1] that commit should not be there in the first place. Commit 102e38b1030e ("dm: split request structure out from dm_rq_target_io structure") from linux-dm tree
[2] is going to move the blk_rq_init() call again to __clone_rq().

Quoted 8 lines

So that commit 6d6285c45f5a should be either reverted, or moved to linux-dm tree, doesn't it?

Cheers,
Dongsu

[1] https://www.redhat.com/archives/dm-devel/2015-January/msg00171.html
[2] https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-for-3.20&id=102e38b1030e883efc022dfdc7b7e7a3de70d1c5

Right, we're aware of this typo in 6d6285c45f5a. Sorry about that, but as you noted, once both the linux-block and linux-dm branches for 3.20 are merged all is back to working.

So we're planning to just leave the block commit as broken and let the dm commit you noted fix it up. In the end 3.20-rc1 will have a working dm-multipath.

Mike

#2 lore raw
Quoted 11 lines · Mike Snitzer
Quoted 8 lines · Dongsu Park

So that commit 6d6285c45f5a should be either reverted, or moved to linux-dm tree, doesn't it?

Cheers,
Dongsu

[1] https://www.redhat.com/archives/dm-devel/2015-January/msg00171.html
[2] https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-for-3.20&id=102e38b1030e883efc022dfdc7b7e7a3de70d1c5

Quoted 7 lines

Right, we're aware of this typo in 6d6285c45f5a. Sorry about that, but as you noted, once both the linux-block and linux-dm branches for 3.20 are merged all is back to working.

So we're planning to just leave the block commit as broken and let the dm commit you noted fix it up. In the end 3.20-rc1 will have a working dm-multipath.

Oh, we're not going rebase the series with the correction? I'm concerned someone biscecting a completely unrelated problem might step on this commit. Up to you guys. It's my fault, so I'll deal with the consequences.

#3 lore raw
Quoted 16 lines · Keith Busch
Quoted 15 lines · Mike Snitzer
Quoted 8 lines · Dongsu Park

So that commit 6d6285c45f5a should be either reverted, or moved to linux-dm tree, doesn't it?

Cheers,
Dongsu

[1] https://www.redhat.com/archives/dm-devel/2015-January/msg00171.html
[2] https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-for-3.20&id=102e38b1030e883efc022dfdc7b7e7a3de70d1c5

Right, we're aware of this typo in 6d6285c45f5a. Sorry about that, but as you noted, once both the linux-block and linux-dm branches for 3.20 are merged all is back to working.

Quoted 7 lines
Quoted 3 lines

So we're planning to just leave the block commit as broken and let the dm commit you noted fix it up. In the end 3.20-rc1 will have a working dm-multipath.

Oh, we're not going rebase the series with the correction? I'm concerned someone biscecting a completely unrelated problem might step on this commit. Up to you guys. It's my fault, so I'll deal with the consequences.

Rebasing this late (3.20 merge already opened) is generally not done. Unfortunately we'll just have to suck it up and deal with the fallout during bisect -- would wager very few are bisecting with multipath as the focus of their test.

#4 lore raw

PATCH 1/1 dm: fix multipath regression due to initializing wrong request

Quoted 11 lines · Mike Snitzer
Quoted 3 lines · Keith Busch

Oh, we're not going rebase the series with the correction? I'm concerned someone biscecting a completely unrelated problem might step on this commit. Up to you guys. It's my fault, so I'll deal with the consequences.

Rebasing this late (3.20 merge already opened) is generally not done. Unfortunately we'll just have to suck it up and deal with the fallout during bisect -- would wager very few are bisecting with multipath as the focus of their test.

Jens and I discussed this further and given that linux-block breaks dm-multipath it is best to fix linux-block and let Linus resolve the merge when I send him the linux-dm pull.

Here is the patch to fix the regression:

From: Mike Snitzer <[email protected]>
Date: Mon, 9 Feb 2015 12:21:54 -0500
Subject: [PATCH] dm: fix multipath regression due to initializing wrong request

Commit febf715 ("block: require blk_rq_prep_clone() be given an initialized clone request") introduced a regression by calling blk_rq_init() on the original request rather than the clone request that is passed to setup_clone().

---
 drivers/md/dm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/md/dm.c +1 -1
index f251633..71e6b73 100644
@@ -1719,7 +1719,7 @@ static int setup_clone(struct request *clone, struct request *rq,
 {
 	int r;
 
-	blk_rq_init(NULL, rq);
+	blk_rq_init(NULL, clone);
 	r = blk_rq_prep_clone(clone, rq, tio->md->bs, GFP_ATOMIC,
 			      dm_rq_bio_constructor, tio);
 	if (r)
Signature · 1 line
1.9.3 (Apple Git-50)
#5 lore raw

Re: [PATCH] dm: fix multipath regression due to initializing wrong request

Quoted 15 lines · Mike Snitzer
Quoted 11 lines · Mike Snitzer
Quoted 3 lines · Keith Busch

Oh, we're not going rebase the series with the correction? I'm concerned someone biscecting a completely unrelated problem might step on this commit. Up to you guys. It's my fault, so I'll deal with the consequences.

Rebasing this late (3.20 merge already opened) is generally not done. Unfortunately we'll just have to suck it up and deal with the fallout during bisect -- would wager very few are bisecting with multipath as the focus of their test.

Quoted 5 lines

Jens and I discussed this further and given that linux-block breaks dm-multipath it is best to fix linux-block and let Linus resolve the merge when I send him the linux-dm pull.

Here is the patch to fix the regression:

Added, thanks. I don't think this is worth rebasing for, so just added to the top of for-3.20/core (since that's where the buggy commit was added).

Signature · 1 line
Jens Axboe
#6 lore raw

Re: [PATCH] dm: fix multipath regression due to initializing wrong request

Quoted 4 lines · Jens Axboe
Quoted 3 lines · Mike Snitzer

On Mon, Feb 09 2015 at 12:13P -0500,
Mike Snitzer <[email protected]> wrote:

Quoted 8 lines
Quoted 5 lines

Jens and I discussed this further and given that linux-block breaks dm-multipath it is best to fix linux-block and let Linus resolve the merge when I send him the linux-dm pull.

Here is the patch to fix the regression:

Added, thanks. I don't think this is worth rebasing for, so just added to the top of for-3.20/core (since that's where the buggy commit was added).

Thanks a lot. Now the branch for-3.20/core works without hitting the BUG.

Dongsu

Quoted 3 lines
Signature · 1 line
Jens Axboe