Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3973C001B0 for ; Fri, 7 Jul 2023 19:06:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 480E46B0072; Fri, 7 Jul 2023 15:06:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 430268D0003; Fri, 7 Jul 2023 15:06:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D2008D0002; Fri, 7 Jul 2023 15:06:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 1D1F96B0072 for ; Fri, 7 Jul 2023 15:06:58 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 58DE11407B1 for ; Fri, 7 Jul 2023 19:06:57 +0000 (UTC) X-FDA: 80985748074.28.A9A419E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 195F3C0010 for ; Fri, 7 Jul 2023 19:06:53 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jQ05UvU4; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688756814; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=paIBB7sUf/Zqn/4VcL295pzVpCyEKoMCoSkthIK/cDY=; b=gRhnYNPCr6y5oXLmg/Gh6HZLNzNyLJPCm61pMXqFnOWQhGFyoT5nTqonXEbIPZ+HuDbITv GRleNwNTfqodto7XqD96fGRaF3ELpy27vb0y5b8lI3lxwwxLSbUdwlT8mfPN6s50pzFDwv GS/33FjM7MP7vBtH7+1Qdrg8OMv8WVg= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jQ05UvU4; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688756814; a=rsa-sha256; cv=none; b=015kAeO65WtOZEBIZutE5v9+Kmzmzm4oj8lYlQ5RBknZU8U1gKxdwqV/I/88K1Gy98BdNR /61wJuqFLz6aRQjrUYje9Rbsxqd9DyeNAeoOj/arrl53QryQ8NHioH75pivclpEUnqgRLB xcZ+5asuL0JwYOsA0ooLGM98x+2eaXE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688756813; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=paIBB7sUf/Zqn/4VcL295pzVpCyEKoMCoSkthIK/cDY=; b=jQ05UvU4OoIW3AEz0DucWBOUW8A0hS6NT+PgJGkEWp/TXOqa3OQpIPer+oqxg95rT3ifqI MEqwJg6fGmyBYbFrHguRiCkZfQkirbkYFzSPXA+XSozCed5npt01cb97Waw+hEFdVBJr1f 30uiWhjT7x0jusUuOUmz2nGJ+l0Kx5k= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-593-WNmHact0OmCby-1GiSL7PQ-1; Fri, 07 Jul 2023 15:06:52 -0400 X-MC-Unique: WNmHact0OmCby-1GiSL7PQ-1 Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-3fbefe1b402so12726425e9.1 for ; Fri, 07 Jul 2023 12:06:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688756811; x=1691348811; h=content-transfer-encoding:in-reply-to:subject:organization:from :references:cc:to:content-language:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=paIBB7sUf/Zqn/4VcL295pzVpCyEKoMCoSkthIK/cDY=; b=RLWFmVUIn2FqXjGA8VXpydfX3ZPqwaFgWwBE2gfpjLIN+6nZvfFv2BoO25jtkvZiAV 0axjmYWIu4ekY8+P/YoQNwIFpGEYocdyBx1aB9H9+NQC5Sy9PmC4HxnAY2oIGyXyK3n7 kE2SFPbtZyyb1N4+b8h3Ol0PoSvVoj3Ny2isrAFax1+9LVTpbfvwgNxOZTf59I7Swq4w 2VJoTVNg2pE0Ag9wc9sMHlhdQ0FfDnzBNB8BDKz+svNk/xTWBw3ThD6lEgjAPeJw2iCb vHsZpQug6CcIy9D+YKH71bsveyIeBh9N27pomqfSnIMXNOdh/cjYzDpgM9aPm2VCRDza Y7Qw== X-Gm-Message-State: ABy/qLaojzTUYzkBl2HmKwO2+CNDmgzjUc8V5bYPe3z+8HdmbkL1dR9L er8CW+VY9pMyB/PzxNfK/Fcsh2JfswlQ8+sP19Ruq9t8fRo4b+wCvIYihtJ+aRgyZxm7eGMpRLz yw4RNYtGVG3E= X-Received: by 2002:a7b:c3d5:0:b0:3f6:91c:4e86 with SMTP id t21-20020a7bc3d5000000b003f6091c4e86mr4429228wmj.3.1688756811017; Fri, 07 Jul 2023 12:06:51 -0700 (PDT) X-Google-Smtp-Source: APBJJlFZCPQ0FYZqAu9VBMOE7vXIKu8edRl0ks2ySLHo/6XMBlpctAYBJaE1EDQ7rgfWGU/ApHetcA== X-Received: by 2002:a7b:c3d5:0:b0:3f6:91c:4e86 with SMTP id t21-20020a7bc3d5000000b003f6091c4e86mr4429208wmj.3.1688756810620; Fri, 07 Jul 2023 12:06:50 -0700 (PDT) Received: from ?IPV6:2003:d8:2f04:3c00:248f:bf5b:b03e:aac7? (p200300d82f043c00248fbf5bb03eaac7.dip0.t-ipconnect.de. [2003:d8:2f04:3c00:248f:bf5b:b03e:aac7]) by smtp.gmail.com with ESMTPSA id c26-20020a7bc01a000000b003f7f475c3bcsm10392928wmb.1.2023.07.07.12.06.49 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 07 Jul 2023 12:06:49 -0700 (PDT) Message-ID: Date: Fri, 7 Jul 2023 21:06:48 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 To: Ryan Roberts , Matthew Wilcox Cc: "Huang, Ying" , Andrew Morton , "Kirill A. Shutemov" , Yin Fengwei , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org References: <20230703135330.1865927-1-ryan.roberts@arm.com> <20230703135330.1865927-5-ryan.roberts@arm.com> <87edlkgnfa.fsf@yhuang6-desk2.ccr.corp.intel.com> <44e60630-5e9d-c8df-ab79-cb0767de680e@arm.com> <524bacd2-4a47-2b8b-6685-c46e31a01631@redhat.com> <1e406f04-78ef-6573-e1f1-f0d0e0d5246a@redhat.com> <9dd036a8-9ba3-0cc4-b791-cb3178237728@arm.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 195F3C0010 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 1o4r9gbpnko7onha394fk9gkk6rqput9 X-HE-Tag: 1688756813-942607 X-HE-Meta: U2FsdGVkX1+HV5NIiRoZinpureFfTGuYR5NOJtssiEfALBAGs6KI2LypWf/+v286XVUvHpdhst24crnZ97w8Nq3Z3BYlv8Dpaq54Mf8vpqkcpV1JNSZGUSf2h2kY8uxDZLt9PaWlBmSdDbjKO+bPZIyCsj5prIpuNJXzt3r6qNFMWZQ4soiMc39P5eL9REfXuXWk5jjp9/0AlLGYggT/sjMuCFkYedL6BEDzNxYx6OgvrLnuXh1v+JRTJ6+N/fnAKHOF7ZUAXdb9angGooRREZzpI+8irpa2rLxNrvZF8o+sWA0RMedVWkIqou9uvoBT0hycABWSPwJ9VhtX92e0yRaclYmm+CjVhkVXCQ3EOj/TVfajHNRDB7uTf55FB3eEaFVk0/0C/Xfw/PGizuE6+BKa9VgFwewMjGVExvLVusvEqryIN0KlS7XUehv12/Ve+7yQPimPsfJoGBQ1AcXkEAIbktrii2PTGFSQkQNcq24TKNI2INzmERLrMt4eFvuSl4JQ13iy9VXOivFDxsOgvgwMdHBynl6UFlKnuwcqqB4Kj4HwTCHHXRdYo/D0Cb4RXm2rQqk3O4jukf0liR/XLF8jrBXn540qWFe+c/le9PjxothVswDrv4QzG7KA0vHArKqSWi1u1Qxc7EaC+/bfXEnca3F8+IbDgolzWAiFrGbZ9Qsp90GMMuzBPEOWB1+cq/qC0GVKaHqt0dbPq/mW4df1YEsQU8ZN6DfyKfvoDLWNg7Dj7P7Uf6GJOpYY1oweyIxpNyBiGj6MbGG4AwWv9uXr8IIZQY+vqXNwETPW2NS4GRlGi2LXRMrW2NmWeol+eoqcABDuWxEIMeolafvqb+uT3gW9/kbxEyFj2QlrSRnAzYOrT6ryWLrMzTd8Ew9jt3XB3pHltpPDkpjCOeNnpvvxbt1/HprBfYAeIJRNqjnM6+4CJolzC+xGhgryojf42MmmVCXMMrzwl28Ajge xR496Wb5 faSvBPSF6MWOgt7ydveCf2GKnE1OEPrkaHcgni1TLHHQu+2glpoKC3HTVw0lUVyEij9lbHO1EPc14MqTLUB3DSRpsKp/2ApbWWWdl/tVZMBtGHl7RUtbzO2untVj6sHhmCQGdLDBG5Polkxiy/DPAx2BOrwuH1gtS8cZX2gZhsorekeezpozufZ9Jm6qorQ8VvJgukwcJOe2QjdrHPyRgABwPYDYch7VROZH9vx7TJuxME+Yg9NfLhGujyz+RUb1+V2ZMHJ1M25+TC/vg2DDr9j/qEPmlm+I0WGDM56Pmx85j67iauDSCuTT4t9JgMu9zI+8T+N5UaXy74oa5eU0anmGjYLYoMLQ4X4xh7hz2Ie8jt+EWCGAzxXxM7cTxbTZmZpFtKxsCY6XABHMYgnoW6yN51pKwfW7R3EcoDC1KbB+VwbY3R42L2381KA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: >>> I still feel that it would be better for the thp and large anon folio controls >>> to be independent though - what's the argument for tying them together? >> >> Thinking about desired 2 MiB flexible THP on aarch64 (64k kernel) vs, 2 MiB PMD >> THP on aarch64 (4k kernel), how are they any different? Just the way they are >> mapped ... > > The last patch in the series shows my current approach to that: > > int arch_wants_pte_order(struct vm_area_struct *vma) > { > if (hugepage_vma_check(vma, vma->vm_flags, false, true, true)) > return CONFIG_ARM64_PTE_ORDER_THP; <<< always the contpte size > else > return CONFIG_ARM64_PTE_ORDER_NOTHP; <<< limited to 64K > } > > But Yu has raised concerns that this type of policy needs to be in the core mm. > So we could have the arch blindly return the preferred order from HW perspective > (which would be contpte size for arm64). Then for !hugepage_vma_check(), mm > could take the min of that value and some determined "acceptable" limit (which > in my mind is 64K ;-). Yeah, it's really tricky. Because why should arm64 with 64k base pages *not* return 2MiB (which is one possible cont-pte size IIRC) ? I share the idea that 64k might *currently* on *some platforms* be a reasonable choice. But that's where the "fun" begins. > >> >> It's easy to say "64k vs. 2 MiB" is a difference and we want separate controls, >> but how is "2MiB vs. 2 MiB" different? >> >> Having that said, I think we have to make up our mind how much control we want >> to give user space. Again, the "2MiB vs. 2 MiB" case nicely shows that it's not >> trivial: memory waste is a real issue on some systems where we limit THP to >> madvise(). >> >> >> Just throwing it out for discussing: >> >> What about keeping the "all / madvise / never" semantics (and MADV_NOHUGEPAGE >> ...) but having an additional config knob that specifies in which cases we >> *still* allow flexible THP even though the system was configured for "madvise". >> >> I can't come up with a good name for that, but something like >> "max_auto_size=64k" could be something reasonable to set. We could have an >> arch+hw specific default. > > Ahha, yes, that's essentially what I have above. I personally also like the idea > of the limit being an absolute value rather than an order. Although I know Yu > feels differently (see [1]). Exposed to user space I think it should be a human-readable value. Inside the kernel, I don't particularly care. (Having databases/VMs on arch64 with 64k in mind) I think it might be interesting to have something like the following: thp=madvise max_auto_size=64k/128k/256k So in MADV_HUGEPAGE VMAs (such as under QEMU), we'd happily take any flexible THP, especially ones < PMD THP (512 MiB) as well. 2 MiB or 4 MiB THP? sure, give them to my VM. You're barely going to find 512 MiB THP either way in practice .... But for the remainder of my system, just do something reasonable and don't go crazy on the memory waste. I'll try reading all the previous discussions next week. -- Cheers, David / dhildenb