Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8EE20CCA47E for ; Tue, 21 Jun 2022 20:55:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=mIH81nm3nFAi4tQSHDTwlcneBRjYDSCEOrECCaNXml0=; b=OL22Q8OYs6AWJr4zWpk2Nrnuog N91djMX/evZM2MraalXyJucwHr6wKr9KxMESrMK5oFMuYyba3FOC7SHHnB2wcOiJlyCres61mRUK/ 7ybW7UUL6IHMC+um1iwUvaqOyV7MlLauY88nZVfB7LCWsX5HuUN0WTAJpoF95dIkhT1dPZjpIObq6 W/Ndq/LylxnKXD55zvOt+V2+HsewayjNRKEvf5LtUHih4o2/Ku33UYC7DHgNvkGYV2lYtCcZllJNC doJHFX15SqwXj7CkTPRtj5G7NQu7xDQJFhSmFuDxY72Cxt5LA+yrujL4FzZDDOsy7qzeT+feIDtvw EcbzZbsw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o3ktl-0078lc-H0; Tue, 21 Jun 2022 20:54:57 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o3kpn-0076hJ-Gc for linux-nvme@lists.infradead.org; Tue, 21 Jun 2022 20:50:52 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 19DA961889; Tue, 21 Jun 2022 20:50:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 336FDC341C4; Tue, 21 Jun 2022 20:50:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1655844650; bh=JeBr1bFqE2A6C5zJTr/9zSU6USgH72iHiRljlSjX1VE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lQiEhxgm2O9Lnv7mVkec0arstDyR/qZXK8cXEyCJEVHCyL1bw5DU2tGoC6ATc6fIL uj0racNchQ07lSaEmHxvjxgPqkxxpYvHguubeJFkHcv/hgYNi7f9FC6wX01IAIIR67 sniEZM39DVupRAvkfw/IuPzuMvoBtpGaN8JD1xGmj1BRbEOcU9QftACmvwZ55Ku13C w7q4Q732vDdsm4l9p7x+NFK28KsOwBUglzfTguddGhFrYTzyJkHfXyS3M3uafPrey3 y/3DEQFz9ngjsIT7Xazy9874GbDeVB21agJ9lUDBgmy+botbhoOehNHN0hGPN3aeov rsMnmMptQP8Xg== From: Sasha Levin To: linux-kernel@vger.kernel.org, stable@vger.kernel.org Cc: Keith Busch , Chaitanya Kulkarni , Christoph Hellwig , Sasha Levin , axboe@fb.com, sagi@grimberg.me, linux-nvme@lists.infradead.org Subject: [PATCH AUTOSEL 5.15 05/17] nvme-pci: add trouble shooting steps for timeouts Date: Tue, 21 Jun 2022 16:50:28 -0400 Message-Id: <20220621205041.250426-5-sashal@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220621205041.250426-1-sashal@kernel.org> References: <20220621205041.250426-1-sashal@kernel.org> MIME-Version: 1.0 X-stable: review X-Patchwork-Hint: Ignore Content-Transfer-Encoding: 8bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220621_135051_681678_14C2FD07 X-CRM114-Status: GOOD ( 11.90 ) X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org From: Keith Busch [ Upstream commit 4641a8e6e145f595059e695f0f8dbbe608134086 ] Many users have encountered IO timeouts with a CSTS value of 0xffffffff, which indicates a failure to read the register. While there are various potential causes for this observation, faulty NVMe APST has been the culprit quite frequently. Add the recommended troubleshooting steps in the error output when this condition occurs. Signed-off-by: Keith Busch Reviewed-by: Chaitanya Kulkarni Signed-off-by: Christoph Hellwig Signed-off-by: Sasha Levin --- drivers/nvme/host/pci.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 3ddd24a42043..e0accbf40f49 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1241,6 +1241,14 @@ static void nvme_warn_reset(struct nvme_dev *dev, u32 csts) dev_warn(dev->ctrl.device, "controller is down; will reset: CSTS=0x%x, PCI_STATUS read failed (%d)\n", csts, result); + + if (csts != ~0) + return; + + dev_warn(dev->ctrl.device, + "Does your device have a faulty power saving mode enabled?\n"); + dev_warn(dev->ctrl.device, + "Try \"nvme_core.default_ps_max_latency_us=0 pcie_aspm=off\" and report a bug\n"); } static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved) -- 2.35.1