Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA261EB64DC for ; Fri, 21 Jul 2023 10:57:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 60D1B2801B4; Fri, 21 Jul 2023 06:57:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5BDAE2801A0; Fri, 21 Jul 2023 06:57:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AC932801B4; Fri, 21 Jul 2023 06:57:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3AE982801A0 for ; Fri, 21 Jul 2023 06:57:29 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 07EB080207 for ; Fri, 21 Jul 2023 10:57:29 +0000 (UTC) X-FDA: 81035317818.01.E8D9FB5 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf01.hostedemail.com (Postfix) with ESMTP id 0102940009 for ; Fri, 21 Jul 2023 10:57:26 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf01.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689937047; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lQ9d1CLJIpYSjxOXkI35FspyNE6hTD7U5en3Vl1t774=; b=XyXJb+mN4581lHdXoXznbFA5k+KMZ8poKInh7Eq5N3XFXSTpie0s5jeIKJogPUvuZiQyHs FQAhS2NgUkalveW4zasdpbaQGqSc9V3P2FaINv0BhbaCz//6pLXpEHVj/ObHICdwdNJ2Es 3f4FdxZaXZ7qUOWdjWVmYfRBmhKGqAI= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf01.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689937047; a=rsa-sha256; cv=none; b=AVDBlC3eJMzX695tXs40SMiYHXmopTchh3r0/uduy+6h3AgfmAAoXBCoTTuMOKg9hfWPM9 XxJJGcns967RSIFXXSNrJ+qDQLCMtO01X8pqVzw8qfHDNGD1EkqHF7MTgjmNCzkmD2exhW sIGzZHUA4VJ7PlueHw8Acf2PsHjpAkg= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4A1912F4; Fri, 21 Jul 2023 03:58:09 -0700 (PDT) Received: from [10.57.64.194] (unknown [10.57.64.194]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9CD993F738; Fri, 21 Jul 2023 03:57:23 -0700 (PDT) Message-ID: Date: Fri, 21 Jul 2023 11:57:22 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: [PATCH v3 3/4] mm: FLEXIBLE_THP for improved performance To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org References: <20230714160407.4142030-1-ryan.roberts@arm.com> <20230714161733.4144503-3-ryan.roberts@arm.com> From: Ryan Roberts In-Reply-To: <20230714161733.4144503-3-ryan.roberts@arm.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 0102940009 X-Stat-Signature: 4fggqdduw13brryk9gdpenw5upgmm1y9 X-Rspam-User: X-HE-Tag: 1689937046-870401 X-HE-Meta: U2FsdGVkX1+eTSFEEsZ9R/Phq1T7poo3fYF0DGBXeDRP2k6vCL98QMF/bCYMwmZpMx5ykEW4YFzn+UPWy6UhZorRcGkK+PoMNItqnCmQGgpUIr8h02UM7VSoGWN8N9CCJbKFElTnZPGDR1UyanYWzg7l4k+mTNTOnz2D0ZUJPBwirBjTvwpY4Y4HkfwCGYUuJdeVnhPC+mwxQdhrM+QSzu59FtQPEROdg1OXMoxbPGjwPlGjC9smmk/Yav9b/NUzDRgxHmFzDxycZy4B6d1cHE5Z8Wd2s36fPG2WHP8yrxYN47ym0ziGoAd/ARXWCCeSq7EsPYYi+8c6CDI6y6rT/IsdsMItOdtnceCB84VgS5XMlg6dvAACN5WFQMy/iHBZYUcwgoaNhdLF6xqEAo3ey07QOkg6EKXOkyLHtZdRrTvpwP0hXi1Dqb2K+UbmMJziWkdXO8Gh55mTaayxvGsa1Z/Vzz0k5l0puUrRfelgZnudltwUHeQJ7dhGQcKzdeU+jKGeb5vOt3w1g5pl0ghOp6/PmaxBMmULI8aw1oyNT8B7zP0rm++uHQ8jhigY/I1+vFbt9QghpScEEJFLJEEAU0Xr3eWfWPur7EhfO83ed6Nx7+Zi0x4O6j30zBZ7cPxL9EDJnGgLxH8NcccsI8qBUYSEQBiO5J7eOxmhft+cE6ecexGHvzJ0y2VJ3baCr4nMHjGVf8S9ssRTLJB42VQw0YCGf9oiE7TKzFMzLsEwrgUBCf59Konlng6odx63/3k8/wL1OULVHSXxuOUNUqmZ50NJUUbezP07DnEMDD1i/ZpxTQz3t5UF+QH4IS4g5ipSMP9dopQgHh09LKisj08rs5agYayxKrPgRir1tdmpo1m8RTOeIoA6nBBzxFXre9/S3qb6YuXgMgA5WzREyALv6UyYEsp8SZRY2FRsq40uKwEEYE++LCLBskWbgq6uBJ4pysRqetA2Kb4eNmPAAjk kQFtUIuK 7AKkG3h77PiC1bpM5TMPj31xBWVU+RuHK7DvflQupDK2H7cjVRcyUnVbXuferh/6AXUPHsmfdEgUds3JlyFQcvyUSUmQeoJuOnKtfxeVp5b3dSaouuAQ6q6H5lmjMf5Y17YqbyELPtpoUqYyGF0gQiB0jT/8EItqrklH1zM89aVwxObFWOWUPVu8BW7ISfTE6w+7O9v1g0tuFTPBbILA+2f3lAA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 14/07/2023 17:17, Ryan Roberts wrote: > Introduce FLEXIBLE_THP feature, which allows anonymous memory to be > allocated in large folios of a determined order. All pages of the large > folio are pte-mapped during the same page fault, significantly reducing > the number of page faults. The number of per-page operations (e.g. ref > counting, rmap management lru list management) are also significantly > reduced since those ops now become per-folio. > > The new behaviour is hidden behind the new FLEXIBLE_THP Kconfig, which > defaults to disabled for now; The long term aim is for this to defaut to > enabled, but there are some risks around internal fragmentation that > need to be better understood first. > > When enabled, the folio order is determined as such: For a vma, process > or system that has explicitly disabled THP, we continue to allocate > order-0. THP is most likely disabled to avoid any possible internal > fragmentation so we honour that request. > > Otherwise, the return value of arch_wants_pte_order() is used. For vmas > that have not explicitly opted-in to use transparent hugepages (e.g. > where thp=madvise and the vma does not have MADV_HUGEPAGE), then > arch_wants_pte_order() is limited by the new cmdline parameter, > `flexthp_unhinted_max`. This allows for a performance boost without > requiring any explicit opt-in from the workload while allowing the > sysadmin to tune between performance and internal fragmentation. > > arch_wants_pte_order() can be overridden by the architecture if desired. > Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous > set of ptes map physically contigious, naturally aligned memory, so this > mechanism allows the architecture to optimize as required. > > If the preferred order can't be used (e.g. because the folio would > breach the bounds of the vma, or because ptes in the region are already > mapped) then we fall back to a suitable lower order; first > PAGE_ALLOC_COSTLY_ORDER, then order-0. > > Signed-off-by: Ryan Roberts ... > + > /* > * We enter with non-exclusive mmap_lock (to exclude vma changes, > * but allow concurrent faults), and pte mapped but not yet locked. > @@ -4057,11 +4199,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > */ > static vm_fault_t do_anonymous_page(struct vm_fault *vmf) > { > + int i = 0; > + int nr_pages = 1; > bool uffd_wp = vmf_orig_pte_uffd_wp(vmf); > struct vm_area_struct *vma = vmf->vma; > struct folio *folio; > vm_fault_t ret = 0; > pte_t entry; > + unsigned long addr; > > /* File mapping without ->vm_ops ? */ > if (vma->vm_flags & VM_SHARED) > @@ -4101,10 +4246,15 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) > /* Allocate our own private page. */ > if (unlikely(anon_vma_prepare(vma))) > goto oom; > - folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); > + ret = alloc_anon_folio(vmf, &folio); > + if (unlikely(ret == -EAGAIN)) > + return 0; > if (!folio) > goto oom; > > + nr_pages = folio_nr_pages(folio); > + addr = ALIGN_DOWN(vmf->address, nr_pages * PAGE_SIZE); > + > if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL)) > goto oom_free_page; > folio_throttle_swaprate(folio, GFP_KERNEL); > @@ -4116,17 +4266,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) > */ > __folio_mark_uptodate(folio); > > - entry = mk_pte(&folio->page, vma->vm_page_prot); > - entry = pte_sw_mkyoung(entry); > - if (vma->vm_flags & VM_WRITE) > - entry = pte_mkwrite(pte_mkdirty(entry)); > - > - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, > - &vmf->ptl); > + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); > if (!vmf->pte) > goto release; > - if (vmf_pte_changed(vmf)) { > - update_mmu_tlb(vma, vmf->address, vmf->pte); > + if (vmf_pte_range_changed(vmf, nr_pages)) { > + for (i = 0; i < nr_pages; i++) > + update_mmu_tlb(vma, addr + PAGE_SIZE * i, vmf->pte + i); > goto release; > } > > @@ -4141,16 +4286,24 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) > return handle_userfault(vmf, VM_UFFD_MISSING); > } > > - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); > - folio_add_new_anon_rmap(folio, vma, vmf->address); > + folio_ref_add(folio, nr_pages - 1); > + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); > + folio_add_new_anon_rmap(folio, vma, addr); > folio_add_lru_vma(folio, vma); > + > + for (i = 0; i < nr_pages; i++) { > + entry = mk_pte(folio_page(folio, i), vma->vm_page_prot); > + entry = pte_sw_mkyoung(entry); > + if (vma->vm_flags & VM_WRITE) > + entry = pte_mkwrite(pte_mkdirty(entry)); > setpte: > - if (uffd_wp) > - entry = pte_mkuffd_wp(entry); > - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); > + if (uffd_wp) > + entry = pte_mkuffd_wp(entry); > + set_pte_at(vma->vm_mm, addr + PAGE_SIZE * i, vmf->pte + i, entry); I've just spotted a bug here for the case where we arrive via goto setpte; in this case, addr is not initialized. This crept in during the refactoring and I have no idea how this could possibly have not fallen over in a heap when executed. Sorry about that. I'm fixing in v4. > > - /* No need to invalidate - it was non-present before */ > - update_mmu_cache(vma, vmf->address, vmf->pte); > + /* No need to invalidate - it was non-present before */ > + update_mmu_cache(vma, addr + PAGE_SIZE * i, vmf->pte + i); > + } > unlock: > if (vmf->pte) > pte_unmap_unlock(vmf->pte, vmf->ptl);