Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B16F2ECE58C for ; Mon, 14 Oct 2019 23:36:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 938C9217D9 for ; Mon, 14 Oct 2019 23:36:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726440AbfJNXgh (ORCPT ); Mon, 14 Oct 2019 19:36:37 -0400 Received: from mx2.suse.de ([195.135.220.15]:60978 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726497AbfJNXgg (ORCPT ); Mon, 14 Oct 2019 19:36:36 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 040B1AE2A; Mon, 14 Oct 2019 23:36:34 +0000 (UTC) From: NeilBrown To: "J. Bruce Fields" , Trond Myklebust , Anna Schumaker Date: Tue, 15 Oct 2019 10:36:27 +1100 Cc: Linux NFS Mailing List Subject: [PATCH] SUNRPC: backchannel RPC request must reference XPRT In-Reply-To: <20191011165603.GD19318@fieldses.org> References: <87tv8iqz3b.fsf@notabene.neil.brown.name> <20191011165603.GD19318@fieldses.org> Message-ID: <87imoqrjb8.fsf@notabene.neil.brown.name> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable The backchannel RPC requests - that are queued waiting for the reply to be sent by the "NFSv4 callback" thread - have a pointer to the xprt, but it is not reference counted. It is possible for the xprt to be freed while there are still queued requests. I think this has been a problem since Commit fb7a0b9addbd ("nfs41: New backchannel helper routines") when the code was introduced, but I suspect it became more of a problem after Commit 80b14d5e61ca ("SUNRPC: Add a structure to track multiple transports") (or there abouts). Before this second patch, the nfs client would hold a reference to the xprt to keep it alive. After multipath was introduced, a client holds a reference to a swtich, and the switch can have multiple xprts which can be added and removed. I'm not sure of all the causal issues, but this patch has fixed a customer problem were an NFSv4.1 client would run out of memory with tens of thousands of backchannel rpc requests queued for an xprt that had been freed. This was a 64K-page machine so each rpc_rqst consumed more than 128K of memory. Fixes: 80b14d5e61ca ("SUNRPC: Add a structure to track multiple transports") cc: stable@vger.kernel.org (v4.6) Signed-off-by: NeilBrown =2D-- net/sunrpc/backchannel_rqst.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/net/sunrpc/backchannel_rqst.c b/net/sunrpc/backchannel_rqst.c index 339e8c077c2d..c95ca39688b6 100644 =2D-- a/net/sunrpc/backchannel_rqst.c +++ b/net/sunrpc/backchannel_rqst.c @@ -61,6 +61,7 @@ static void xprt_free_allocation(struct rpc_rqst *req) free_page((unsigned long)xbufp->head[0].iov_base); xbufp =3D &req->rq_snd_buf; free_page((unsigned long)xbufp->head[0].iov_base); + xprt_put(req->rq_xprt); kfree(req); } =20 @@ -85,7 +86,7 @@ struct rpc_rqst *xprt_alloc_bc_req(struct rpc_xprt *xprt,= gfp_t gfp_flags) if (req =3D=3D NULL) return NULL; =20 =2D req->rq_xprt =3D xprt; + req->rq_xprt =3D xprt_get(xprt); INIT_LIST_HEAD(&req->rq_bc_list); =20 /* Preallocate one XDR receive buffer */ =2D-=20 2.23.0 --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEG8Yp69OQ2HB7X0l6Oeye3VZigbkFAl2lBnsACgkQOeye3VZi gbkktg//T+OnehuZeD5J3/192d+8xpWsk0q9VIYiuO+O+plTOuNOF2GEfR7vOAZW 5U23Kp/boE97HQXAachRv67FWqrlK3MtTiMv8wuOkqMaOXhbO/RSKSle7QExZvhP XWuFYCZ0QagUjnpvtEb29Imj7pupjxIlCxSokIqg6PJFh+iMljLxGhneC3c2xf0X CA4ClyhDV+MWaOtHb2fp9J8f8Ve+65lunW/EaJmR1XKoEpaqALqQbHtzcEc4QyIq yUH6DG+qIz5raK6i9yWMRRuHEMO+5CEMTrz4K97CCVTlu4tJ5ROb6SWLkwk3kayb /3+arMuHNDmsT2Jbv6PQ91n90zaF3CMMALgGl6R6Pn1oqYG4nA3zVQfbl5gAVOKA Q2ldPJmAWQiRgDR1nRsbXMakzEAaEBlAdZhbnKIcAnPEX4NgOCPgh00qtbj+I11L wPPI62+WBuSkBDyf13nLW00oGCnqInN9w6i/vdPCIHF6e7x8WxmcwEmrz8TYecEM oI2fNYC78caugclcaETpabTpenXI493rrYU3NCinSGM/GyGqokVRIdClDMYYIUdJ BEfyoY6bScx2zcs+jXhPWFoNUPIqvgr5Dl2AbmIrc/pXEh2N3XSsAXdGHMp0UjEU 2wEIcRoPAznwBGoF6+1HAlb2K0CpMQRObwStPauhswIBPTBBet8= =4db9 -----END PGP SIGNATURE----- --=-=-=--