Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9ADADC11D02 for ; Thu, 20 Feb 2020 08:29:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 529D7208C4 for ; Thu, 20 Feb 2020 08:29:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 529D7208C4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CD40F6B0005; Thu, 20 Feb 2020 03:29:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C5C6A6B0006; Thu, 20 Feb 2020 03:29:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B71976B0007; Thu, 20 Feb 2020 03:29:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0102.hostedemail.com [216.40.44.102]) by kanga.kvack.org (Postfix) with ESMTP id 9F5506B0005 for ; Thu, 20 Feb 2020 03:29:23 -0500 (EST) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 845862492 for ; Thu, 20 Feb 2020 08:29:23 +0000 (UTC) X-FDA: 76509831006.16.iron41_34d7b61e99234 X-HE-Tag: iron41_34d7b61e99234 X-Filterd-Recvd-Size: 6254 Received: from mail-wr1-f67.google.com (mail-wr1-f67.google.com [209.85.221.67]) by imf04.hostedemail.com (Postfix) with ESMTP for ; Thu, 20 Feb 2020 08:29:22 +0000 (UTC) Received: by mail-wr1-f67.google.com with SMTP id g3so3531639wrs.12 for ; Thu, 20 Feb 2020 00:29:22 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=pcVb7tYtjd5vJy2GdOdQXq1b2D4fCTq0JpAZZtr5yr0=; b=G2MvaXlYZwhN0xhS098fIudyKAkzaBjgLS85nvXoKFXBO4DLoj2ch6/4yszkGTx6zv vx0DJrYXgmbHEfbCHsVmvUqVgGFEn1DKZazvQlImunu9tcUb5qY+6BUd1XVonxn6Y9Ie 1FIzsTZdvYHzRYmHToYSBZJ3LVIYXo6+szuMnCXeOMM6joQUX2WsWdjELBAOgUVIhrJq Oq8yYKDjaJpHiLA4KoXQVBfj9TU4JvM92LhsB1XTP+cZKhLzGjmtnGSv76OJncfYylXU tm4n9kDFnMuIK7P82CstsSvlluNEp0EHQhPEMqvjPMENnQW1yj40+nmE8zNusk7JPZh4 2BJA== X-Gm-Message-State: APjAAAWWI0Cv8DisdIyY8Ytu5BSET9p+1kOAXNJ7V6l72xfcYfkWx3Zf G8ZGC8BhU3osy9rPz/Mh0pc= X-Google-Smtp-Source: APXvYqznloAc7/YvN2REE3Ur5hUHLeUc0Ogh8t5kKphHGasqgO/VXOdODTu2+evN2nivuKVMKGzQBA== X-Received: by 2002:a5d:6802:: with SMTP id w2mr40472712wru.353.1582187361702; Thu, 20 Feb 2020 00:29:21 -0800 (PST) Received: from localhost (ip-37-188-133-21.eurotel.cz. [37.188.133.21]) by smtp.gmail.com with ESMTPSA id h128sm3895837wmh.33.2020.02.20.00.29.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Feb 2020 00:29:20 -0800 (PST) Date: Thu, 20 Feb 2020 09:29:19 +0100 From: Michal Hocko To: Sultan Alsawaf Cc: Dave Hansen , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Mel Gorman , Johannes Weiner Subject: Re: [PATCH] mm: Stop kswapd early when nothing's waiting for it to free pages Message-ID: <20200220082919.GC20509@dhcp22.suse.cz> References: <20200219182522.1960-1-sultan@kerneltoast.com> <20200219194006.GA3075@sultan-book.localdomain> <20200219200527.GF11847@dhcp22.suse.cz> <20200219204220.GA3488@sultan-book.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200219204220.GA3488@sultan-book.localdomain> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed 19-02-20 12:42:20, Sultan Alsawaf wrote: > On Wed, Feb 19, 2020 at 09:05:27PM +0100, Michal Hocko wrote: [...] > > Again, do you have more details about the workload and what was the > > cause of responsiveness issues? Because I would expect that the > > situation would be quite opposite because it is usually the direct > > reclaim that is a source of stalls visible from userspace. Or is this > > about a single CPU situation where kswapd saturates the single CPU and > > all other tasks are just not getting enough CPU cycles? > > The workload was having lots of applications open at once. At a certain point > when memory ran low, my system became sluggish and kswapd CPU usage skyrocketed. Could you provide more details please? Is kswapd making a forward progress? Have you checked why other precesses are slugish? They do not get CPU time or they are blocked on something? > I added printks into kswapd with this patch, and my premature exit in kswapd > kicked in quite often. > > > > On systems with more memory I tested (>=4G), kswapd becomes more expensive to > > > run at its higher scan depths, so stopping kswapd prematurely when there aren't > > > any memory allocations waiting for it prevents it from reaching the *really* > > > expensive scan depths and burning through even more resources. > > > > > > Combine a large amount of memory with a slow CPU and the current problematic > > > behavior of kswapd at high memory pressure shows. My personal test scenario for > > > this was an arm64 CPU with a variable amount of memory (up to 4G RAM + 2G swap). > > > > But still, somebody has to put the system into balanced state so who is > > going to do all the work? > > All the work will be done by kswapd of course, but only if it's needed. > > The real problem is that a single memory allocation failure, and free memory > being some amount below the high watermark, are not good heuristics to predict > *future* memory allocation needs. They are good for determining how to steer > kswapd to help satisfy a failed allocation in the present, but anything more is > pure speculation (which turns out to be wrong speculation, since this behavior > causes problems). Well, you might be right that there might be better heuristics than the existing watermark based one. After all nobody can predict the future. The existing heuristic aims at providing min_free_kbytes of free memory as much as possible and that tends to work reasonably well for a large set of workloads. > If there are outstanding failed allocations that won't go away, then it's > perfectly reasonable to keep kswapd running until it frees pages up to the high > watermark. But beyond that is unnecessary, since there's no way to know if or > when kswapd will need to fire up again. This makes sense considering how kswapd > is currently invoked: it's fired up due to a failed allocation of some sort, not > because the amount of free memory dropped below the high watermark. Very broadly speaking (sorry if I am stating obvious here), the kswapd is woken up when the allocator hits low watermark or the reguested high order pages are depleted. Then allocator enters its slow path. That means that the background reclaim then aims at reclaiming the high-low watermark gap or invokes compaction to keep the balance. It takes to consume that gap to wake the kswapd again for order-0 (most common) requests. So this is usually not about a single allocation to trigger the background reclaim and counting failures on low watermark attempts is unlikely to work with the current code as you suggested. -- Michal Hocko SUSE Labs