Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F324AC43387 for ; Thu, 20 Dec 2018 21:44:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id ADF6921903 for ; Thu, 20 Dec 2018 21:44:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="KIqWzVQS" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390076AbeLTVoO (ORCPT ); Thu, 20 Dec 2018 16:44:14 -0500 Received: from mail-io1-f44.google.com ([209.85.166.44]:42598 "EHLO mail-io1-f44.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732606AbeLTVoN (ORCPT ); Thu, 20 Dec 2018 16:44:13 -0500 Received: by mail-io1-f44.google.com with SMTP id x6so2047378ioa.9 for ; Thu, 20 Dec 2018 13:44:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=Kq2ooDdPuw9xoSNTMW5wE2teKWLRwUe0kZXisWKHUCo=; b=KIqWzVQSdQ8PICnAIc+20ZmfCUDJ2IWz0byAgPYpj6ID4izpZytk23/KB+TlRGPV+t LVogSqqKkF0iipGJdxI17kxfKfIBIUiv/5cvLg/mpvj0N5KWPuIVXPlYWnaRQDebtlqH 03/OkfkbCneBeCIiI6wO4FU26uIWxJD2npkA4uZz+MD/sFGka4DkEKkkuJHJsHFit8DM lsvU+y1R1T+W6y68w9VoN2qqYSs2Cdhv6tRHiAYaa0LJLELDu79Xg4L1uk21pLs+ZyG/ 74PW8W82qBQcxVAeNSNJFn7cfkB2/ccMexSJQ8d/NYsBYOF80Azw4ZsY1wNkFXqaYZvW JELw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=Kq2ooDdPuw9xoSNTMW5wE2teKWLRwUe0kZXisWKHUCo=; b=ArJzPYU75t+9xrvViMLQZNYaFkBxMMWshIbQCyKJJZJprMXNZ0LWlXb6sxjevwl6QY Lvij1m4+9fnz8ldrKZisBfBr4vbW9EAW/RuNhMAp9ea/rmcoLiSwhXBT6EckMeN5aRtp Qxj9MQgy6hEFcd35xDb7N5kbyT3p5vreKp53BIeSuhlo8FwwJozIwAxGVoPqN1m4u98b wnc/xR87p+kewLVmC/WBT9lGTwGyaqrdsSyr1Ee0kIT/7bWD3Rfyobc7q0eGNK76j8qd aeuM9FuYm3lruLZUnbsJHWxPcuDUA5p3LwAJ02T6b6kfP2KcBPS//ljl1rThkS5cmfoB YDDg== X-Gm-Message-State: AA+aEWa785lZmQoMRChWXk12Mpddc8DWGrU5RpxJ90l8L9zvP6sxZcEK /I/Bg0i1sFLdEJvXtcB7m+1tdlKwaeXUTw== X-Google-Smtp-Source: AFSGD/WUNqtQEae4fc1Y66V8hIvwsEzDYaWASaLcKk3rsnQ0JqEDsrQmx+O2620bZME0UYBRXnlb9A== X-Received: by 2002:a6b:4e14:: with SMTP id c20mr21109629iob.65.1545342250774; Thu, 20 Dec 2018 13:44:10 -0800 (PST) Received: from [192.168.1.56] ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id t64sm9064021itb.5.2018.12.20.13.44.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Dec 2018 13:44:09 -0800 (PST) Subject: Re: v4.20-rc6: Sporadic use-after-free in bt_iter() To: Bart Van Assche , "jianchao.wang" , "linux-block@vger.kernel.org" References: <1545261885.185366.488.camel@acm.org> <74c0280c-e557-ad03-cd75-98dd6d868da3@kernel.dk> <1545265001.185366.496.camel@acm.org> <1ca5cd37-87ce-8cff-6cb4-1fdb29bd4da2@kernel.dk> <0560802f-efc0-e9ec-99f7-4bdbdbc234f8@oracle.com> <372d2960-ff0c-1135-28f9-23eea8670463@oracle.com> <6ae35005-7ba9-91e1-f315-d128f410c12c@kernel.dk> <1545328865.185366.508.camel@acm.org> <1545339362.185366.511.camel@acm.org> <1545340987.185366.515.camel@acm.org> <120bb59a-af93-7d8c-9afc-7087973632bf@kernel.dk> <1545341470.185366.519.camel@acm.org> <61515137-0565-e3b7-e6de-554af7d49753@kernel.dk> <1545342043.185366.523.camel@acm.org> From: Jens Axboe Message-ID: <60b4819c-4c19-a4e3-41f3-e21b0544c9a4@kernel.dk> Date: Thu, 20 Dec 2018 14:44:08 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <1545342043.185366.523.camel@acm.org> Content-Type: text/plain; charset=UTF-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org On 12/20/18 2:40 PM, Bart Van Assche wrote: > On Thu, 2018-12-20 at 14:34 -0700, Jens Axboe wrote: >> Yeah, I don't think it's bullet proof either, it just closes the gap. >> I'm fine with fiddling with the tag iteration. On top of what I sent, we >> could have tag iteration hold the RCU read lock, and then we just need >> to ensure that the tags are freed with RCU. > > Do you mean using call_rcu() to free tags? Would that require to add a > struct rcu_head to every request? Would it be acceptable to increase the > size of struct request with an rcu_head? Additionally, could that reduce > the queue depth if the time between grace periods is larger than the time > between I/O submissions? No no, the requests are out of full pages, we just need one per blk_mq_tags. Something like the below... And then the tag iteration needs to grab the RCU read lock to prevent the page freeing from happening. This should essentially be free, as rcu read lock for tag iteration is basically a no-op. Need to handle the flush request on top of this as well. diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c index 2089c6c62f44..c39b58391ae8 100644 --- a/block/blk-mq-tag.c +++ b/block/blk-mq-tag.c @@ -228,13 +228,15 @@ static bool bt_iter(struct sbitmap *bitmap, unsigned int bitnr, void *data) if (!reserved) bitnr += tags->nr_reserved_tags; - rq = tags->rqs[bitnr]; + if (tags->rqs[bitnr].queue != hctx->queue) + return true; /* * We can hit rq == NULL here, because the tagging functions * test and set the bit before assigning ->rqs[]. */ - if (rq && rq->q == hctx->queue) + rq = tags->rqs[bitnr].rq; + if (rq) return iter_data->fn(hctx, rq, iter_data->data, reserved); return true; } @@ -263,7 +265,9 @@ static void bt_for_each(struct blk_mq_hw_ctx *hctx, struct sbitmap_queue *bt, .reserved = reserved, }; + rcu_read_lock(); sbitmap_for_each_set(&bt->sb, bt_iter, &iter_data); + rcu_read_unlock(); } struct bt_tags_iter_data { @@ -287,7 +291,7 @@ static bool bt_tags_iter(struct sbitmap *bitmap, unsigned int bitnr, void *data) * We can hit rq == NULL here, because the tagging functions * test and set the bit before assining ->rqs[]. */ - rq = tags->rqs[bitnr]; + rq = tags->rqs[bitnr].rq; if (rq && blk_mq_request_started(rq)) return iter_data->fn(rq, iter_data->data, reserved); @@ -317,8 +321,11 @@ static void bt_tags_for_each(struct blk_mq_tags *tags, struct sbitmap_queue *bt, .reserved = reserved, }; - if (tags->rqs) + if (tags->rqs) { + rcu_read_lock(); sbitmap_for_each_set(&bt->sb, bt_tags_iter, &iter_data); + rcu_read_unlock(); + } } /** diff --git a/block/blk-mq-tag.h b/block/blk-mq-tag.h index 61deab0b5a5a..bdd1bfc08c21 100644 --- a/block/blk-mq-tag.h +++ b/block/blk-mq-tag.h @@ -4,6 +4,11 @@ #include "blk-mq.h" +struct rq_tag_entry { + struct request_queue *queue; + struct request *rq; +}; + /* * Tag address space map. */ @@ -16,9 +21,11 @@ struct blk_mq_tags { struct sbitmap_queue bitmap_tags; struct sbitmap_queue breserved_tags; - struct request **rqs; + struct rq_tag_entry *rqs; struct request **static_rqs; struct list_head page_list; + + struct rcu_work rcu_work; }; @@ -78,7 +85,8 @@ static inline void blk_mq_tag_idle(struct blk_mq_hw_ctx *hctx) static inline void blk_mq_tag_set_rq(struct blk_mq_hw_ctx *hctx, unsigned int tag, struct request *rq) { - hctx->tags->rqs[tag] = rq; + hctx->tags->rqs[tag].queue = hctx->queue; + hctx->tags->rqs[tag].rq = rq; } static inline bool blk_mq_tag_is_reserved(struct blk_mq_tags *tags, diff --git a/block/blk-mq.c b/block/blk-mq.c index 2de972857496..4decd1e7d2d9 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -298,13 +298,16 @@ static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data, rq->tag = -1; rq->internal_tag = tag; } else { - if (data->hctx->flags & BLK_MQ_F_TAG_SHARED) { + struct blk_mq_hw_ctx *hctx = data->hctx; + + if (hctx->flags & BLK_MQ_F_TAG_SHARED) { rq_flags = RQF_MQ_INFLIGHT; - atomic_inc(&data->hctx->nr_active); + atomic_inc(&hctx->nr_active); } rq->tag = tag; rq->internal_tag = -1; - data->hctx->tags->rqs[rq->tag] = rq; + hctx->tags->rqs[rq->tag].queue = hctx->queue; + hctx->tags->rqs[rq->tag].rq = rq; } /* csd/requeue_work/fifo_time is initialized before use */ @@ -797,8 +800,8 @@ EXPORT_SYMBOL(blk_mq_delay_kick_requeue_list); struct request *blk_mq_tag_to_rq(struct blk_mq_tags *tags, unsigned int tag) { if (tag < tags->nr_tags) { - prefetch(tags->rqs[tag]); - return tags->rqs[tag]; + prefetch(tags->rqs[tag].rq); + return tags->rqs[tag].rq; } return NULL; @@ -809,10 +812,11 @@ static bool blk_mq_rq_inflight(struct blk_mq_hw_ctx *hctx, struct request *rq, void *priv, bool reserved) { /* - * If we find a request that is inflight and the queue matches, - * we know the queue is busy. Return false to stop the iteration. + * We're only called here if the queue matches. If the rq state is + * inflight, we know the queue is busy. Return false to stop the + * iteration. */ - if (rq->state == MQ_RQ_IN_FLIGHT && rq->q == hctx->queue) { + if (rq->state == MQ_RQ_IN_FLIGHT) { bool *busy = priv; *busy = true; @@ -1049,11 +1053,14 @@ bool blk_mq_get_driver_tag(struct request *rq) shared = blk_mq_tag_busy(data.hctx); rq->tag = blk_mq_get_tag(&data); if (rq->tag >= 0) { + struct blk_mq_hw_ctx *hctx = data.hctx; + if (shared) { rq->rq_flags |= RQF_MQ_INFLIGHT; atomic_inc(&data.hctx->nr_active); } - data.hctx->tags->rqs[rq->tag] = rq; + hctx->tags->rqs[rq->tag].queue = hctx->queue; + hctx->tags->rqs[rq->tag].rq = rq; } done: @@ -2020,11 +2027,27 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) return cookie; } -void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, - unsigned int hctx_idx) +static void blk_mq_rcu_free_pages(struct work_struct *work) { + struct blk_mq_tags *tags = container_of(to_rcu_work(work), + struct blk_mq_tags, rcu_work); struct page *page; + while (!list_empty(&tags->page_list)) { + page = list_first_entry(&tags->page_list, struct page, lru); + list_del_init(&page->lru); + /* + * Remove kmemleak object previously allocated in + * blk_mq_init_rq_map(). + */ + kmemleak_free(page_address(page)); + __free_pages(page, page->private); + } +} + +void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, + unsigned int hctx_idx) +{ if (tags->rqs && set->ops->exit_request) { int i; @@ -2038,16 +2061,9 @@ void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, } } - while (!list_empty(&tags->page_list)) { - page = list_first_entry(&tags->page_list, struct page, lru); - list_del_init(&page->lru); - /* - * Remove kmemleak object previously allocated in - * blk_mq_init_rq_map(). - */ - kmemleak_free(page_address(page)); - __free_pages(page, page->private); - } + /* Punt to RCU free, so we don't race with tag iteration */ + INIT_RCU_WORK(&tags->rcu_work, blk_mq_rcu_free_pages); + queue_rcu_work(system_wq, &tags->rcu_work); } void blk_mq_free_rq_map(struct blk_mq_tags *tags) @@ -2077,7 +2093,7 @@ struct blk_mq_tags *blk_mq_alloc_rq_map(struct blk_mq_tag_set *set, if (!tags) return NULL; - tags->rqs = kcalloc_node(nr_tags, sizeof(struct request *), + tags->rqs = kcalloc_node(nr_tags, sizeof(struct rq_tag_entry), GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, node); if (!tags->rqs) { -- Jens Axboe