riscv: Introduce 64K base page
Subsystems (13)
cover riscv: Introduce 64K base page
Some existing architectures like ARM supports base page larger than 4K as their MMU supports more page sizes. Thus, besides hugetlb page and transparent huge page, there is another way for these architectures to enjoy the benefits of fewer TLB misses without worrying about cost of splitting and merging huge pages. However, on architectures with only 4K MMU, larger base page is unavailable now.
This patch series attempts to break through the limitation of MMU and supports larger base page on RISC-V, which only supports 4K page size now.
The key idea to implement larger base page based on 4K MMU is to decouple the MMU page from the base page in view of kernel mm, which we denote as software page. In contrary to software page, we denote the MMU page as hardware page. Below is the difference between these two kinds of pages.
1. Kernel memory management module manages, allocates and maps memory at a granularity of software page, which should not be restricted by MMU and can be larger than hardware page.
2. Architecture page table operations should be carried out from MMU's perspective and page table entries are encoded at a granularity of hardware page, which is 4K on RISC-V MMU now.
The main work to decouple these two kinds of pages lies in architecture code. For example, we turn the pte_t struct to an array of page table entries to match it with software page which can be larger than hardware page, and adapt the page table operations accordingly. For 64K software base page, the pte_t struct now contains 16 contiguous page table entries which point to 16 contiguous 4K hardware pages.
To achieve the benefits of large base page, we applies Svnapot for each base page's mapping. The Svnapot extension on RISC-V is like contiguous PTE on ARM64. It allows ptes of a naturally aligned power-of 2 size memory range be encoded in the same format to save the TLB space.
This patch series is the first version and is based on v6.7-rc1. This version supports both bare metal and virtualization scenarios.
In the next versions, we will continue on the following works:
1. Reduce the memory usage of page table page as it only uses 4K space while costs a whole base page.
2. When IMSIC interrupt file is smaller than 64K, extra isolation measures for the interrupt file are needed. (S)PMP and IOPMP may be good choices.
3. More consideration is needed to make this patch series collaborate with folios better.
4. Support 64K base page on IOMMU.
5. The performance test is on schedule to verify the actual performance improvement and the decrease in TLB miss rate.
Thanks in advance for comments.
Xu Lu (11):
mm: Fix misused APIs on huge pte
riscv: Introduce concept of hardware base page
riscv: Adapt pte struct to gap between hw page and sw page
riscv: Adapt pte operations to gap between hw page and sw page
riscv: Decouple pmd operations and pte operations
riscv: Distinguish pmd huge pte and napot huge pte
riscv: Adapt satp operations to gap between hw page and sw page
riscv: Apply Svnapot for base page mapping
riscv: Adjust fix_btmap slots number to match variable page size
riscv: kvm: Adapt kvm to gap between hw page and sw page
riscv: Introduce 64K page size
arch/Kconfig | 1 + arch/riscv/Kconfig | 28 +++ arch/riscv/include/asm/fixmap.h | 3 +- arch/riscv/include/asm/hugetlb.h | 71 ++++++- arch/riscv/include/asm/page.h | 16 +- arch/riscv/include/asm/pgalloc.h | 21 ++- arch/riscv/include/asm/pgtable-32.h | 2 +- arch/riscv/include/asm/pgtable-64.h | 45 +++-- arch/riscv/include/asm/pgtable.h | 282 +++++++++++++++++++++++----- arch/riscv/kernel/efi.c | 2 +- arch/riscv/kernel/head.S | 4 +- arch/riscv/kernel/hibernate.c | 3 +- arch/riscv/kvm/mmu.c | 198 +++++++++++++------ arch/riscv/mm/context.c | 7 +- arch/riscv/mm/fault.c | 1 + arch/riscv/mm/hugetlbpage.c | 42 +++-- arch/riscv/mm/init.c | 25 +-- arch/riscv/mm/kasan_init.c | 7 +- arch/riscv/mm/pageattr.c | 2 +- fs/proc/task_mmu.c | 2 +- include/asm-generic/hugetlb.h | 7 + include/asm-generic/pgtable-nopmd.h | 1 + include/linux/pgtable.h | 6 + mm/hugetlb.c | 2 +- mm/migrate.c | 5 +- mm/mprotect.c | 2 +- mm/rmap.c | 10 +- mm/vmalloc.c | 3 +- 28 files changed, 616 insertions(+), 182 deletions(-)
Signature · 7 lines
2.20.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 1/11 mm: Fix misused APIs on huge pte
There exist some paths that try to get value of huge pte via normal pte API ptep_get instead of huge pte API huge_ptep_get. This commit corrects these misused APIs.
--- arch/riscv/mm/hugetlbpage.c | 2 +- fs/proc/task_mmu.c | 2 +- include/asm-generic/hugetlb.h | 7 +++++++ mm/hugetlb.c | 2 +- mm/migrate.c | 5 ++++- mm/mprotect.c | 2 +- mm/rmap.c | 10 ++++++++-- mm/vmalloc.c | 3 ++- 8 files changed, 25 insertions(+), 8 deletions(-)
arch/riscv/mm/hugetlbpage.c +1 -1
@@ -74,7 +74,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, out:if (pte) { -pte_t pteval =ptep_get_lockless (pte); +pte_t pteval =huge_ptep_get_lockless (pte);WARN_ON_ONCE (pte_present (pteval) && !pte_huge (pteval)); }
fs/proc/task_mmu.c +1 -1
@@ -726,7 +726,7 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask,struct mem_size_stats *mss = walk->private;struct vm_area_struct *vma = walk->vma;struct page *page =NULL ; -pte_t ptent =ptep_get (pte); +pte_t ptent =huge_ptep_get (pte);if (pte_present (ptent)) { page =vm_normal_page (vma, addr, ptent);
include/asm-generic/hugetlb.h +7 -0
@@ -150,6 +150,13 @@ static inline pte_t huge_ptep_get(pte_t *ptep) }#endif +#ifndef __HAVE_ARCH_HUGE_PTEP_GET_LOCKLESS +static inline pte_t huge_ptep_get_lockless (pte_t *ptep) +{ +return huge_ptep_get (ptep); +} +#endif +#ifndef __HAVE_ARCH_GIGANTIC_PAGE_RUNTIME_SUPPORTEDstatic inline bool gigantic_page_runtime_supported (void ) {
mm/hugetlb.c +1 -1
@@ -7406,7 +7406,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, }if (pte) { -pte_t pteval =ptep_get_lockless (pte); +pte_t pteval =huge_ptep_get_lockless (pte);BUG_ON (pte_present (pteval) && !pte_huge (pteval)); }
mm/migrate.c +4 -1
@@ -210,7 +210,10 @@ static bool remove_migration_pte(struct folio *folio,folio_get (folio); pte =mk_pte (new,READ_ONCE (vma->vm_page_prot)); - old_pte =ptep_get (pvmw.pte); +if (folio_test_hugetlb (folio)) + old_pte =huge_ptep_get (pvmw.pte); +else + old_pte =ptep_get (pvmw.pte);if (pte_swp_soft_dirty (old_pte)) pte =pte_mksoft_dirty (pte);
mm/mprotect.c +1 -1
@@ -555,7 +555,7 @@ static int prot_none_hugetlb_entry(pte_t *pte, unsigned long hmask, unsigned long addr, unsigned long next, struct mm_walk *walk) { - return pfn_modify_allowed(pte_pfn(ptep_get(pte)), + return pfn_modify_allowed(pte_pfn(huge_ptep_get(pte)), *(pgprot_t *)(walk->private)) ? 0 : -EACCES; }
mm/rmap.c +8 -2
@@ -1577,7 +1577,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,break ; } - pfn =pte_pfn (ptep_get (pvmw.pte)); +if (folio_test_hugetlb (folio)) + pfn =pte_pfn (huge_ptep_get (pvmw.pte)); +else + pfn =pte_pfn (ptep_get (pvmw.pte)); subpage =folio_page (folio, pfn -folio_pfn (folio)); address = pvmw.address; anon_exclusive = folio_test_anon(folio) &&
@@ -1931,7 +1934,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,/* Unexpected PMD-mapped THP? */ VM_BUG_ON_FOLIO (!pvmw.pte, folio); - pfn =pte_pfn (ptep_get (pvmw.pte)); +if (folio_test_hugetlb (folio)) + pfn =pte_pfn (huge_ptep_get (pvmw.pte)); +else + pfn =pte_pfn (ptep_get (pvmw.pte));if (folio_is_zone_device (folio)) {/*
mm/vmalloc.c +2 -1
@@ -103,7 +103,6 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,if (!pte)return -ENOMEM ; do { - BUG_ON(!pte_none(ptep_get(pte))); #ifdef CONFIG_HUGETLB_PAGE size = arch_vmap_pte_range_map_size(addr, end, pfn, max_page_shift);
@@ -111,11 +110,13 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,pte_t entry =pfn_pte (pfn, prot); entry =arch_make_huge_pte (entry,ilog2 (size),0 ); +BUG_ON (!pte_none (huge_ptep_get (pte)));set_huge_pte_at (&init_mm, addr, pte, entry, size); pfn +=PFN_DOWN (size); continue; }#endif +BUG_ON (!pte_none (ptep_get (pte)));set_pte_at (&init_mm, addr, pte,pfn_pte (pfn, prot)); pfn++; }while (pte +=PFN_DOWN (size), addr += size, addr != end);
Signature · 1 line
2.20.1
kernel test robot
build botRe: [RFC PATCH V1 01/11] mm: Fix misused APIs on huge pte
kernel test robot noticed the following build errors:
Hi Xu,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on arnd-asm-generic/master linus/master v6.7-rc2 next-20231124] [cannot apply to efi/next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Xu-Lu/mm-Fix-misused-APIs-on-huge-pte/20231123-150128 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20231123065708.91345-2-luxu.kernel%40bytedance.com patch subject: [RFC PATCH V1 01/11] mm: Fix misused APIs on huge pte config: x86_64-allnoconfig (https://download.01.org/0day-ci/archive/20231124/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231124/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All error/warnings (new ones prefixed by >>):
mm/mprotect.c: In function 'prot_none_hugetlb_entry':
>> mm/mprotect.c:558:43: error: implicit declaration of function 'huge_ptep_get' [-Werror=implicit-function-declaration]
558 | return pfn_modify_allowed(pte_pfn(huge_ptep_get(pte)),
| ^~~~~~~~~~~~~
>> mm/mprotect.c:558:43: error: incompatible type for argument 1 of 'pte_pfn'
558 | return pfn_modify_allowed(pte_pfn(huge_ptep_get(pte)),
| ^~~~~~~~~~~~~~~~~~
| |
| int
In file included from include/linux/pgtable.h:6,
from include/linux/mm.h:29,
from include/linux/pagewalk.h:5,
from mm/mprotect.c:12:
arch/x86/include/asm/pgtable.h:223:43: note: expected 'pte_t' but argument is of type 'int'
223 | static inline unsigned long pte_pfn(pte_t pte)
| ~~~~~~^~~
>> mm/mprotect.c:561:1: warning: control reaches end of non-void function [-Wreturn-type]
561 | }
| ^
cc1: some warnings being treated as errors
--
mm/rmap.c: In function 'try_to_unmap_one':
>> mm/rmap.c:1611:39: error: implicit declaration of function 'huge_ptep_get' [-Werror=implicit-function-declaration]
1611 | pfn = pte_pfn(huge_ptep_get(pvmw.pte));
| ^~~~~~~~~~~~~
>> mm/rmap.c:1611:39: error: incompatible type for argument 1 of 'pte_pfn'
1611 | pfn = pte_pfn(huge_ptep_get(pvmw.pte));
| ^~~~~~~~~~~~~~~~~~~~~~~
| |
| int
In file included from include/linux/pgtable.h:6,
from include/linux/mm.h:29,
from mm/rmap.c:56:
arch/x86/include/asm/pgtable.h:223:43: note: expected 'pte_t' but argument is of type 'int'
223 | static inline unsigned long pte_pfn(pte_t pte)
| ~~~~~~^~~
mm/rmap.c: In function 'try_to_migrate_one':
mm/rmap.c:1968:39: error: incompatible type for argument 1 of 'pte_pfn'
1968 | pfn = pte_pfn(huge_ptep_get(pvmw.pte));
| ^~~~~~~~~~~~~~~~~~~~~~~
| |
| int arch/x86/include/asm/pgtable.h:223:43: note: expected 'pte_t' but argument is of type 'int'
223 | static inline unsigned long pte_pfn(pte_t pte)
| ~~~~~~^~~
cc1: some warnings being treated as errors
vim +/huge_ptep_get +558 mm/mprotect.c
553
554 static int prot_none_hugetlb_entry(pte_t *pte, unsigned long hmask,
555 unsigned long addr, unsigned long next,
556 struct mm_walk *walk)
557 {
> 558 return pfn_modify_allowed(pte_pfn(huge_ptep_get(pte)), 559 *(pgprot_t *)(walk->private)) ? 560 0 : -EACCES; > 561 }
562
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [RFC PATCH V1 01/11] mm: Fix misused APIs on huge pte
In reply to Xu Lukernel test robot noticed the following build errors:
Hi Xu,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on arnd-asm-generic/master linus/master v6.7-rc2 next-20231124] [cannot apply to efi/next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Xu-Lu/mm-Fix-misused-APIs-on-huge-pte/20231123-150128 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20231123065708.91345-2-luxu.kernel%40bytedance.com patch subject: [RFC PATCH V1 01/11] mm: Fix misused APIs on huge pte config: i386-allnoconfig (https://download.01.org/0day-ci/archive/20231124/[email protected]/config) compiler: clang version 16.0.4 (https://github.com/llvm/llvm-project.git ae42196bc493ffe877a7e3dff8be32035dea4d07) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231124/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> mm/mprotect.c:558:36: error: call to undeclared function 'huge_ptep_get'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
return pfn_modify_allowed(pte_pfn(huge_ptep_get(pte)),
^
>> mm/mprotect.c:558:36: error: passing 'int' to parameter of incompatible type 'pte_t'
return pfn_modify_allowed(pte_pfn(huge_ptep_get(pte)),
^~~~~~~~~~~~~~~~~~
arch/x86/include/asm/pgtable.h:223:43: note: passing argument to parameter 'pte' here
static inline unsigned long pte_pfn(pte_t pte)
^
2 errors generated.
--
>> mm/rmap.c:1611:18: error: call to undeclared function 'huge_ptep_get'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
pfn = pte_pfn(huge_ptep_get(pvmw.pte));
^
>> mm/rmap.c:1611:18: error: passing 'int' to parameter of incompatible type 'pte_t'
pfn = pte_pfn(huge_ptep_get(pvmw.pte));
^~~~~~~~~~~~~~~~~~~~~~~
arch/x86/include/asm/pgtable.h:223:43: note: passing argument to parameter 'pte' here
static inline unsigned long pte_pfn(pte_t pte)
^
mm/rmap.c:1968:18: error: call to undeclared function 'huge_ptep_get'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
pfn = pte_pfn(huge_ptep_get(pvmw.pte));
^
mm/rmap.c:1968:18: error: passing 'int' to parameter of incompatible type 'pte_t'
pfn = pte_pfn(huge_ptep_get(pvmw.pte));
^~~~~~~~~~~~~~~~~~~~~~~
arch/x86/include/asm/pgtable.h:223:43: note: passing argument to parameter 'pte' here
static inline unsigned long pte_pfn(pte_t pte)
^4 errors generated.
vim +/huge_ptep_get +558 mm/mprotect.c
553
554 static int prot_none_hugetlb_entry(pte_t *pte, unsigned long hmask,
555 unsigned long addr, unsigned long next,
556 struct mm_walk *walk)
557 {
> 558 return pfn_modify_allowed(pte_pfn(huge_ptep_get(pte)),
559 *(pgprot_t *)(walk->private)) ?
560 0 : -EACCES;
561 }
562
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Alexandre Ghiti
reviewerRe: [RFC PATCH V1 01/11] mm: Fix misused APIs on huge pte
In reply to Xu LuHi Xu,
Quoted 21 lines · Xu Lu
There exist some paths that try to get value of huge pte via normal pte API
ptep_getinstead of huge pte APIhuge_ptep_get. This commit corrects these misused APIs.Signed-off-by Xu Lu <[email protected]>--- arch/riscv/mm/hugetlbpage.c | 2 +- fs/proc/task_mmu.c | 2 +- include/asm-generic/hugetlb.h | 7 +++++++ mm/hugetlb.c | 2 +- mm/migrate.c | 5 ++++- mm/mprotect.c | 2 +- mm/rmap.c | 10 ++++++++-- mm/vmalloc.c | 3 ++- 8 files changed, 25 insertions(+), 8 deletions(-)arch/riscv/mm/hugetlbpage.c +0
-0
arch/riscv/mm/hugetlbpage.c pte_t *huge_pte_alloc(struct mm_struct *mm, · line 80
Quoted 8 lines
arch/riscv/mm/hugetlbpage.c +1
-1@@ -74,7 +74,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, out:if (pte) { -pte_t pteval =ptep_get_lockless (pte); +pte_t pteval =huge_ptep_get_lockless (pte);WARN_ON_ONCE (pte_present (pteval) && !pte_huge (pteval)); }
Only looking at riscv for this patch, this change does not look necessary as the pte value is only used to check if the pte is huge or not, it does not use the A/D bits.
Thanks,
Alex
Quoted 127 lines
fs/proc/task_mmu.c +1
-1@@ -726,7 +726,7 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask,struct mem_size_stats *mss = walk->private;struct vm_area_struct *vma = walk->vma;struct page *page =NULL ; -pte_t ptent =ptep_get (pte); +pte_t ptent =huge_ptep_get (pte);if (pte_present (ptent)) { page =vm_normal_page (vma, addr, ptent);include/asm-generic/hugetlb.h +7
-0@@ -150,6 +150,13 @@ static inline pte_t huge_ptep_get(pte_t *ptep) }#endif +#ifndef __HAVE_ARCH_HUGE_PTEP_GET_LOCKLESS +static inline pte_t huge_ptep_get_lockless (pte_t *ptep) +{ +return huge_ptep_get (ptep); +} +#endif +#ifndef __HAVE_ARCH_GIGANTIC_PAGE_RUNTIME_SUPPORTEDstatic inline bool gigantic_page_runtime_supported (void ) {mm/hugetlb.c +1
-1@@ -7406,7 +7406,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, }if (pte) { -pte_t pteval =ptep_get_lockless (pte); +pte_t pteval =huge_ptep_get_lockless (pte);BUG_ON (pte_present (pteval) && !pte_huge (pteval)); }mm/migrate.c +4
-1@@ -210,7 +210,10 @@ static bool remove_migration_pte(struct folio *folio,folio_get (folio); pte =mk_pte (new,READ_ONCE (vma->vm_page_prot)); - old_pte =ptep_get (pvmw.pte); +if (folio_test_hugetlb (folio)) + old_pte =huge_ptep_get (pvmw.pte); +else + old_pte =ptep_get (pvmw.pte);if (pte_swp_soft_dirty (old_pte)) pte =pte_mksoft_dirty (pte);mm/mprotect.c +1
-1@@ -555,7 +555,7 @@ static int prot_none_hugetlb_entry(pte_t *pte, unsigned long hmask, unsigned long addr, unsigned long next, struct mm_walk *walk) { - return pfn_modify_allowed(pte_pfn(ptep_get(pte)), + return pfn_modify_allowed(pte_pfn(huge_ptep_get(pte)), *(pgprot_t *)(walk->private)) ? 0 : -EACCES; }mm/rmap.c +8
-2@@ -1577,7 +1577,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,break ; } - pfn =pte_pfn (ptep_get (pvmw.pte)); +if (folio_test_hugetlb (folio)) + pfn =pte_pfn (huge_ptep_get (pvmw.pte)); +else + pfn =pte_pfn (ptep_get (pvmw.pte)); subpage =folio_page (folio, pfn -folio_pfn (folio)); address = pvmw.address; anon_exclusive = folio_test_anon(folio) &&@@ -1931,7 +1934,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,/* Unexpected PMD-mapped THP? */ VM_BUG_ON_FOLIO (!pvmw.pte, folio); - pfn =pte_pfn (ptep_get (pvmw.pte)); +if (folio_test_hugetlb (folio)) + pfn =pte_pfn (huge_ptep_get (pvmw.pte)); +else + pfn =pte_pfn (ptep_get (pvmw.pte));if (folio_is_zone_device (folio)) {/* mm/vmalloc.c +2
-1@@ -103,7 +103,6 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,if (!pte)return -ENOMEM ; do { - BUG_ON(!pte_none(ptep_get(pte))); #ifdef CONFIG_HUGETLB_PAGE size = arch_vmap_pte_range_map_size(addr, end, pfn, max_page_shift);@@ -111,11 +110,13 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,pte_t entry =pfn_pte (pfn, prot); entry =arch_make_huge_pte (entry,ilog2 (size),0 ); +BUG_ON (!pte_none (huge_ptep_get (pte)));set_huge_pte_at (&init_mm, addr, pte, entry, size); pfn +=PFN_DOWN (size); continue; }#endif +BUG_ON (!pte_none (ptep_get (pte)));set_pte_at (&init_mm, addr, pte,pfn_pte (pfn, prot)); pfn++; }while (pte +=PFN_DOWN (size), addr += size, addr != end);
Re: [External] Re: [RFC PATCH V1 01/11] mm: Fix misused APIs on huge pte
Hi Alexandre,
Thanks for your feedback!
Quoted 39 lines · Alexandre Ghiti
Hi Xu,
Quoted 21 lines · Xu Lu
There exist some paths that try to get value of huge pte via normal pte API
ptep_getinstead of huge pte APIhuge_ptep_get. This commit corrects these misused APIs.Signed-off-by Xu Lu <[email protected]>--- arch/riscv/mm/hugetlbpage.c | 2 +- fs/proc/task_mmu.c | 2 +- include/asm-generic/hugetlb.h | 7 +++++++ mm/hugetlb.c | 2 +- mm/migrate.c | 5 ++++- mm/mprotect.c | 2 +- mm/rmap.c | 10 ++++++++-- mm/vmalloc.c | 3 ++- 8 files changed, 25 insertions(+), 8 deletions(-)arch/riscv/mm/hugetlbpage.c +0
-0arch/riscv/mm/hugetlbpage.c pte_t *huge_pte_alloc(struct mm_struct *mm, · line 80
Quoted 8 lines
arch/riscv/mm/hugetlbpage.c +1
-1@@ -74,7 +74,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, out:if (pte) { -pte_t pteval =ptep_get_lockless (pte); +pte_t pteval =huge_ptep_get_lockless (pte);WARN_ON_ONCE (pte_present (pteval) && !pte_huge (pteval)); }Only looking at riscv for this patch, this change does not look necessary as the pte value is only used to check if the pte is huge or not, it does not use the A/D bits.
Quoted 4 lines
Thanks,
Alex
Your observation is very accurate. The pte value here does not use the A/D bits.
Actually, the modification here is to ensure the match between pte type and pte operating functions. In this patch version, 'huge_ptep_get_lockless' and 'ptep_get_lockless' won't traverse pte array in pte_t to check A/D bits. Instead, to ensure atomicity, they just get the value of the first pte element and use it to generate the whole pte_t struct. Thus they may lead to potential A/D bits loss. In our opinion, if one wants to check the A/D bit to indicate following page flushing or page swapping, he should acquire page table lock and use huge_ptep_get/ptep_get instead.
Regards,
Xu
Quoted 128 lines
Quoted 128 lines
fs/proc/task_mmu.c +1
-1@@ -726,7 +726,7 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask,struct mem_size_stats *mss = walk->private;struct vm_area_struct *vma = walk->vma;struct page *page =NULL ; -pte_t ptent =ptep_get (pte); +pte_t ptent =huge_ptep_get (pte);if (pte_present (ptent)) { page =vm_normal_page (vma, addr, ptent);include/asm-generic/hugetlb.h +7
-0@@ -150,6 +150,13 @@ static inline pte_t huge_ptep_get(pte_t *ptep) }#endif +#ifndef __HAVE_ARCH_HUGE_PTEP_GET_LOCKLESS +static inline pte_t huge_ptep_get_lockless (pte_t *ptep) +{ +return huge_ptep_get (ptep); +} +#endif +#ifndef __HAVE_ARCH_GIGANTIC_PAGE_RUNTIME_SUPPORTEDstatic inline bool gigantic_page_runtime_supported (void ) {mm/hugetlb.c +1
-1@@ -7406,7 +7406,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, }if (pte) { -pte_t pteval =ptep_get_lockless (pte); +pte_t pteval =huge_ptep_get_lockless (pte);BUG_ON (pte_present (pteval) && !pte_huge (pteval)); }mm/migrate.c +4
-1@@ -210,7 +210,10 @@ static bool remove_migration_pte(struct folio *folio,folio_get (folio); pte =mk_pte (new,READ_ONCE (vma->vm_page_prot)); - old_pte =ptep_get (pvmw.pte); +if (folio_test_hugetlb (folio)) + old_pte =huge_ptep_get (pvmw.pte); +else + old_pte =ptep_get (pvmw.pte);if (pte_swp_soft_dirty (old_pte)) pte =pte_mksoft_dirty (pte);mm/mprotect.c +1
-1@@ -555,7 +555,7 @@ static int prot_none_hugetlb_entry(pte_t *pte, unsigned long hmask, unsigned long addr, unsigned long next, struct mm_walk *walk) { - return pfn_modify_allowed(pte_pfn(ptep_get(pte)), + return pfn_modify_allowed(pte_pfn(huge_ptep_get(pte)), *(pgprot_t *)(walk->private)) ? 0 : -EACCES; }mm/rmap.c +8
-2@@ -1577,7 +1577,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,break ; } - pfn =pte_pfn (ptep_get (pvmw.pte)); +if (folio_test_hugetlb (folio)) + pfn =pte_pfn (huge_ptep_get (pvmw.pte)); +else + pfn =pte_pfn (ptep_get (pvmw.pte)); subpage =folio_page (folio, pfn -folio_pfn (folio)); address = pvmw.address; anon_exclusive = folio_test_anon(folio) &&@@ -1931,7 +1934,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,/* Unexpected PMD-mapped THP? */ VM_BUG_ON_FOLIO (!pvmw.pte, folio); - pfn =pte_pfn (ptep_get (pvmw.pte)); +if (folio_test_hugetlb (folio)) + pfn =pte_pfn (huge_ptep_get (pvmw.pte)); +else + pfn =pte_pfn (ptep_get (pvmw.pte));if (folio_is_zone_device (folio)) {/* mm/vmalloc.c +2
-1@@ -103,7 +103,6 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,if (!pte)return -ENOMEM ; do { - BUG_ON(!pte_none(ptep_get(pte))); #ifdef CONFIG_HUGETLB_PAGE size = arch_vmap_pte_range_map_size(addr, end, pfn, max_page_shift);@@ -111,11 +110,13 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,pte_t entry =pfn_pte (pfn, prot); entry =arch_make_huge_pte (entry,ilog2 (size),0 ); +BUG_ON (!pte_none (huge_ptep_get (pte)));set_huge_pte_at (&init_mm, addr, pte, entry, size); pfn +=PFN_DOWN (size); continue; }#endif +BUG_ON (!pte_none (ptep_get (pte)));set_pte_at (&init_mm, addr, pte,pfn_pte (pfn, prot)); pfn++; }while (pte +=PFN_DOWN (size), addr += size, addr != end);
RFC PATCH 2/11 riscv: Introduce concept of hardware base page
In reply to Xu LuThe key idea to implement larger base page based on MMU that only supports 4K page is to decouple the MMU page from the software page in view of kernel mm. In contrary to software page, we denote the MMU page as hardware page.
To decouple these two kinds of pages, we should manage, allocate and map memory at a granularity of software page, which is exactly what existing mm code does. The page table operations, however, should configure page table entries at a granularity of hardware page, which is the responsibility of arch code.
This commit introduces the concept of hardware base page for RISCV.
--- arch/riscv/Kconfig | 8 ++++++++ arch/riscv/include/asm/page.h | 6 +++++- 2 files changed, 13 insertions(+), 1 deletion(-)
arch/riscv/Kconfig +8 -0
@@ -221,6 +221,14 @@ config PAGE_OFFSET default 0x80000000 if !MMU default 0xff60000000000000 if 64BIT +config RISCV_HW_PAGE_SHIFT + int + default 12 + +config RISCV_PAGE_SHIFT + int + default 12 + config KASAN_SHADOW_OFFSET hex depends on KASAN_GENERIC
arch/riscv/include/asm/page.h +5 -1
@@ -12,7 +12,11 @@#include <linux/pfn.h> #include <linux/const.h> -#define PAGE_SHIFT (12) +#define HW_PAGE_SHIFT CONFIG_RISCV_HW_PAGE_SHIFT +#define HW_PAGE_SIZE (_AC(1, UL) << HW_PAGE_SHIFT) +#define HW_PAGE_MASK (~(HW_PAGE_SIZE - 1)) + +#define PAGE_SHIFT CONFIG_RISCV_PAGE_SHIFT#define PAGE_SIZE (_AC(1, UL) << PAGE_SHIFT)#define PAGE_MASK (~(PAGE_SIZE - 1))
Signature · 7 lines
2.20.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 3/11 riscv: Adapt pte struct to gap between hw page and sw page
In reply to Xu LuThe pte_t struct maps a virtual page to a physical page, both of which are the concept of software page in view of kernel mm. In contrary, per page table entry refers to a single hardware page. When sw page is larger than hw page, the existing pte_t struct with only one page table entry can not represent a software page anymore.
This commit extends pte_t struct to contain an array of page table entries. The pte_t struct now maps a software page to an exact number of hardware pages, the total size of which still matches sw page size.
--- arch/riscv/include/asm/page.h | 7 ++- arch/riscv/include/asm/pgtable-64.h | 3 +- arch/riscv/include/asm/pgtable.h | 91 ++++++++++++++++++++++++++--- arch/riscv/kernel/efi.c | 2 +- arch/riscv/mm/hugetlbpage.c | 2 +- arch/riscv/mm/pageattr.c | 2 +- 6 files changed, 92 insertions(+), 15 deletions(-)
arch/riscv/include/asm/page.h +4 -3
@@ -68,9 +68,11 @@ typedef struct {unsigned long pgd; } pgd_t; +#define PTES_PER_PAGE (1 << (PAGE_SHIFT - HW_PAGE_SHIFT)) +/* Page Table entry */ typedef struct { -unsigned long pte; +unsigned long ptes[PTES_PER_PAGE ]; }pte_t ; typedef struct {
@@ -79,11 +81,10 @@ typedef struct {typedef struct page *pgtable_t ; -#define pte_val (x) ((x).pte) +#define pte_val (x) ((x).ptes[0])#define pgd_val (x) ((x).pgd)#define pgprot_val (x) ((x).pgprot) -#define __pte (x) ((pte_t) { (x) })#define __pgd (x) ((pgd_t) { (x) })#define __pgprot (x) ((pgprot_t) { (x) })
arch/riscv/include/asm/pgtable-64.h +2 -1
@@ -99,7 +99,8 @@ enum napot_cont_order {#define for_each_napot_order_rev (order) \ for (order = NAPOT_ORDER_MAX - 1; \ order >= NAPOT_CONT_ORDER_BASE; order--) -#define napot_cont_order (val) (__builtin_ctzl((val.pte >> _PAGE_PFN_SHIFT) << 1)) +#define napot_cont_order (val) \ + (__builtin_ctzl((pte_val(val) >> _PAGE_PFN_SHIFT) << 1))#define napot_cont_shift (order) ((order) + PAGE_SHIFT)#define napot_cont_size (order) BIT(napot_cont_shift(order))
arch/riscv/include/asm/pgtable.h +83 -8
152 lines not on this page · raw message
arch/riscv/kernel/efi.c +1 -1
@@ -60,7 +60,7 @@ int __init efi_create_mapping(struct mm_struct *mm, efi_memory_desc_t *md)static int __init set_permissions (pte_t *ptep,unsigned long addr,void *data) {efi_memory_desc_t *md = data; -pte_t pte =READ_ONCE (*ptep); +pte_t pte =ptep_get (ptep);unsigned long val;if (md->attribute &EFI_MEMORY_RO ) {
arch/riscv/mm/hugetlbpage.c +1 -1
@@ -293,7 +293,7 @@ void huge_pte_clear(struct mm_struct *mm, pte_t *ptep, unsigned long sz) { -pte_t pte =READ_ONCE (*ptep); +pte_t pte =ptep_get (ptep);int i, pte_num;if (!pte_napot (pte)) {
arch/riscv/mm/pageattr.c +1 -1
@@ -68,7 +68,7 @@ static int pageattr_pmd_entry(pmd_t *pmd, unsigned long addr,static int pageattr_pte_entry (pte_t *pte,unsigned long addr,unsigned long next,struct mm_walk *walk) { -pte_t val =READ_ONCE (*pte); +pte_t val =ptep_get (pte); val =__pte (set_pageattr_masks (pte_val (val), walk));set_pte (pte, val);
Signature · 1 line
2.20.1
kernel test robot
build botRe: [RFC PATCH V1 03/11] riscv: Adapt pte struct to gap between hw page and sw page
kernel test robot noticed the following build errors:
Hi Xu,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on arnd-asm-generic/master linus/master v6.7-rc2 next-20231124] [cannot apply to efi/next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Xu-Lu/mm-Fix-misused-APIs-on-huge-pte/20231123-150128 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20231123065708.91345-4-luxu.kernel%40bytedance.com patch subject: [RFC PATCH V1 03/11] riscv: Adapt pte struct to gap between hw page and sw page config: riscv-rv32_defconfig (https://download.01.org/0day-ci/archive/20231124/[email protected]/config) compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project.git 4a5ac14ee968ff0ad5d2cc1ffa0299048db4c88a) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231124/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from arch/riscv/kernel/asm-offsets.c:10:
In file included from include/linux/mm.h:29:
In file included from include/linux/pgtable.h:6:
>> arch/riscv/include/asm/pgtable.h:222:18: error: use of undeclared identifier '_PAGE_NAPOT'
222 | return pteval & _PAGE_NAPOT;
| ^
In file included from arch/riscv/kernel/asm-offsets.c:10:
In file included from include/linux/mm.h:1095:
In file included from include/linux/huge_mm.h:8:
In file included from include/linux/fs.h:33:
In file included from include/linux/percpu-rwsem.h:7:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:97:11: warning: array index 3 is past the end of the array (that has type 'unsigned long[2]') [-Warray-bounds]
97 | return (set->sig[3] | set->sig[2] |
| ^ ~
include/uapi/asm-generic/signal.h:62:2: note: array 'sig' declared here
62 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/riscv/kernel/asm-offsets.c:10:
In file included from include/linux/mm.h:1095:
In file included from include/linux/huge_mm.h:8:
In file included from include/linux/fs.h:33:
In file included from include/linux/percpu-rwsem.h:7:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:97:25: warning: array index 2 is past the end of the array (that has type 'unsigned long[2]') [-Warray-bounds]
97 | return (set->sig[3] | set->sig[2] |
| ^ ~
include/uapi/asm-generic/signal.h:62:2: note: array 'sig' declared here
62 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/riscv/kernel/asm-offsets.c:10:
In file included from include/linux/mm.h:1095:
In file included from include/linux/huge_mm.h:8:
In file included from include/linux/fs.h:33:
In file included from include/linux/percpu-rwsem.h:7:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:113:11: warning: array index 3 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
113 | return (set1->sig[3] == set2->sig[3]) &&
| ^ ~
include/uapi/asm-generic/signal.h:62:2: note: array 'sig' declared here
62 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/riscv/kernel/asm-offsets.c:10:
In file included from include/linux/mm.h:1095:
In file included from include/linux/huge_mm.h:8:
In file included from include/linux/fs.h:33:
In file included from include/linux/percpu-rwsem.h:7:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:113:27: warning: array index 3 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
113 | return (set1->sig[3] == set2->sig[3]) &&
| ^ ~
include/uapi/asm-generic/signal.h:62:2: note: array 'sig' declared here
62 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/riscv/kernel/asm-offsets.c:10:
In file included from include/linux/mm.h:1095:
In file included from include/linux/huge_mm.h:8:
In file included from include/linux/fs.h:33:
In file included from include/linux/percpu-rwsem.h:7:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:114:5: warning: array index 2 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
114 | (set1->sig[2] == set2->sig[2]) &&
| ^ ~
include/uapi/asm-generic/signal.h:62:2: note: array 'sig' declared here
62 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/riscv/kernel/asm-offsets.c:10:
In file included from include/linux/mm.h:1095:
In file included from include/linux/huge_mm.h:8:
In file included from include/linux/fs.h:33:
In file included from include/linux/percpu-rwsem.h:7:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:114:21: warning: array index 2 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
114 | (set1->sig[2] == set2->sig[2]) &&
| ^ ~
include/uapi/asm-generic/signal.h:62:2: note: array 'sig' declared here
62 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/riscv/kernel/asm-offsets.c:10:
In file included from include/linux/mm.h:1095:
In file included from include/linux/huge_mm.h:8:
In file included from include/linux/fs.h:33:
In file included from include/linux/percpu-rwsem.h:7:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:156:1: warning: array index 3 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
156 | _SIG_SET_BINOP(sigorsets, _sig_or)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/linux/signal.h:137:8: note: expanded from macro '_SIG_SET_BINOP'
137 | a3 = a->sig[3]; a2 = a->sig[2]; \
| ^ ~
include/uapi/asm-generic/signal.h:62:2: note: array 'sig' declared here
62 | unsigned long sig[_NSIG_WORDS];
| ^ In file included from arch/riscv/kernel/asm-offsets.c:10:
In file included from include/linux/mm.h:1095:
In file included from include/linux/huge_mm.h:8:
In file included from include/linux/fs.h:33:
vim +/_PAGE_NAPOT +222 arch/riscv/include/asm/pgtable.h
219
220 static __always_inline unsigned long __pte_napot(unsigned long pteval)
221 {
> 222 return pteval & _PAGE_NAPOT;
223 }
224
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
RFC PATCH 4/11 riscv: Adapt pte operations to gap between hw page and sw page
In reply to Xu LuMMU handles pages at a granularity of hardware page. That is, for 4K MMU, the pfn decoded from page table entry will be regarded as 4K page frame number, no matter how large the software page is. Thus, page table entries should always be encoded at the granularity of hardware page.
This commit makes pte operations aware of the gap between hw page and sw page. All pte operations now configure page table entries via hardware page frame number.
--- arch/riscv/include/asm/page.h | 3 +++ arch/riscv/include/asm/pgalloc.h | 21 ++++++++++----- arch/riscv/include/asm/pgtable-32.h | 2 +- arch/riscv/include/asm/pgtable-64.h | 40 ++++++++++++++++++----------- arch/riscv/include/asm/pgtable.h | 19 +++++++------- arch/riscv/mm/init.c | 18 ++++++------- 6 files changed, 62 insertions(+), 41 deletions(-)
arch/riscv/include/asm/page.h +3 -0
@@ -177,6 +177,9 @@ extern phys_addr_t __phys_addr_symbol(unsigned long x);#define __pa (x) __virt_to_phys((unsigned long)(x))#define __va (x) ((void *)__pa_to_va_nodebug((phys_addr_t)(x))) +#define pfn_to_hwpfn (pfn) (pfn << (PAGE_SHIFT - HW_PAGE_SHIFT)) +#define hwpfn_to_pfn (hwpfn) (hwpfn >> (PAGE_SHIFT - HW_PAGE_SHIFT)) +#define phys_to_pfn (phys) (PFN_DOWN(phys))#define pfn_to_phys (pfn) (PFN_PHYS(pfn))
arch/riscv/include/asm/pgalloc.h +14 -7
@@ -19,32 +19,36 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte) {unsigned long pfn =virt_to_pfn (pte); +unsigned long hwpfn =pfn_to_hwpfn (pfn); -set_pmd (pmd,__pmd ((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); +set_pmd (pmd,__pmd ((hwpfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); }static inline void pmd_populate (struct mm_struct *mm,pmd_t *pmd,pgtable_t pte) {unsigned long pfn =virt_to_pfn (page_address (pte)); +unsigned long hwpfn =pfn_to_hwpfn (pfn); -set_pmd (pmd,__pmd ((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); +set_pmd (pmd,__pmd ((hwpfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); }#ifndef __PAGETABLE_PMD_FOLDEDstatic inline void pud_populate (struct mm_struct *mm,pud_t *pud,pmd_t *pmd) {unsigned long pfn =virt_to_pfn (pmd); +unsigned long hwpfn =pfn_to_hwpfn (pfn); -set_pud (pud,__pud ((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); +set_pud (pud,__pud ((hwpfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); }static inline void p4d_populate (struct mm_struct *mm,p4d_t *p4d,pud_t *pud) {if (pgtable_l4_enabled) {unsigned long pfn =virt_to_pfn (pud); +unsigned long hwpfn =pfn_to_hwpfn (pfn); -set_p4d (p4d,__p4d ((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); +set_p4d (p4d,__p4d ((hwpfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } }
@@ -53,9 +57,10 @@ static inline void p4d_populate_safe(struct mm_struct *mm, p4d_t *p4d, {if (pgtable_l4_enabled) {unsigned long pfn =virt_to_pfn (pud); +unsigned long hwpfn =pfn_to_hwpfn (pfn);set_p4d_safe (p4d, -__p4d ((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); +__p4d ((hwpfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } }
@@ -63,8 +68,9 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4d) {if (pgtable_l5_enabled) {unsigned long pfn =virt_to_pfn (p4d); +unsigned long hwpfn =pfn_to_hwpfn (pfn); -set_pgd (pgd,__pgd ((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); +set_pgd (pgd,__pgd ((hwpfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } }
@@ -73,9 +79,10 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, {if (pgtable_l5_enabled) {unsigned long pfn =virt_to_pfn (p4d); +unsigned long hwpfn =pfn_to_hwpfn (pfn);set_pgd_safe (pgd, -__pgd ((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); +__pgd ((hwpfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } }
arch/riscv/include/asm/pgtable-32.h +1 -1
@@ -20,7 +20,7 @@/* * rv32 PTE format: * | XLEN-1 10 | 9 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 -* PFN reserved for SW D A G U X W R V +* HW_PFN reserved for SW D A G U X W R V */ #define _PAGE_PFN_MASK GENMASK(31, 10)
arch/riscv/include/asm/pgtable-64.h +25 -15
@@ -50,7 +50,7 @@ typedef struct {#define p4d_val (x) ((x).p4d)#define __p4d (x) ((p4d_t) { (x) }) -#define PTRS_PER_P4D (PAGE_SIZE / sizeof(p4d_t)) +#define PTRS_PER_P4D (HW_PAGE_SIZE / sizeof(p4d_t))/* Page Upper Directory entry */ typedef struct {
@@ -59,7 +59,7 @@ typedef struct {#define pud_val (x) ((x).pud)#define __pud (x) ((pud_t) { (x) }) -#define PTRS_PER_PUD (PAGE_SIZE / sizeof(pud_t)) +#define PTRS_PER_PUD (HW_PAGE_SIZE / sizeof(pud_t))/* Page Middle Directory entry */ typedef struct {
@@ -69,12 +69,12 @@ typedef struct {#define pmd_val (x) ((x).pmd)#define __pmd (x) ((pmd_t) { (x) }) -#define PTRS_PER_PMD (PAGE_SIZE / sizeof(pmd_t)) +#define PTRS_PER_PMD (HW_PAGE_SIZE / sizeof(pmd_t))/* * rv64 PTE format: * | 63 | 62 61 | 60 54 | 53 10 | 9 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 -* N MT RSV PFN reserved for SW D A G U X W R V +* N MT RSV HW_PFN reserved for SW D A G U X W R V */ #define _PAGE_PFN_MASK GENMASK(53, 10)
@@ -94,13 +94,23 @@ enum napot_cont_order { NAPOT_ORDER_MAX, }; -#define for_each_napot_order (order) \ - for (order = NAPOT_CONT_ORDER_BASE; order < NAPOT_ORDER_MAX; order++) -#define for_each_napot_order_rev (order) \ - for (order = NAPOT_ORDER_MAX - 1; \ - order >= NAPOT_CONT_ORDER_BASE; order--) -#define napot_cont_order (val) \ - (__builtin_ctzl((pte_val(val) >> _PAGE_PFN_SHIFT) << 1)) +#define NAPOT_PAGE_ORDER_BASE \ + ((NAPOT_CONT_ORDER_BASE >= (PAGE_SHIFT - HW_PAGE_SHIFT)) ? \ + (NAPOT_CONT_ORDER_BASE - (PAGE_SHIFT - HW_PAGE_SHIFT)) : 1) +#define NAPOT_PAGE_ORDER_MAX \ + ((NAPOT_ORDER_MAX > (PAGE_SHIFT - HW_PAGE_SHIFT)) ? \ + (NAPOT_ORDER_MAX - (PAGE_SHIFT - HW_PAGE_SHIFT)) : \ + NAPOT_PAGE_ORDER_BASE) + +#define for_each_napot_order (order) \ + for (order = NAPOT_PAGE_ORDER_BASE; \ + order < NAPOT_PAGE_ORDER_MAX; order++) +#define for_each_napot_order_rev (order) \ + for (order = NAPOT_PAGE_ORDER_MAX - 1; \ + order >= NAPOT_PAGE_ORDER_BASE; order--) +#define napot_cont_order (val) \ + (__builtin_ctzl((pte_val(val) >> _PAGE_PFN_SHIFT) << 1) \ + - (PAGE_SHIFT - HW_PAGE_SHIFT))#define napot_cont_shift (order) ((order) + PAGE_SHIFT)#define napot_cont_size (order) BIT(napot_cont_shift(order))
@@ -108,7 +118,7 @@ enum napot_cont_order {#define napot_pte_num (order) BIT(order)#ifdef CONFIG_RISCV_ISA_SVNAPOT -#define HUGE_MAX_HSTATE (2 + (NAPOT_ORDER_MAX - NAPOT_CONT_ORDER_BASE)) +#define HUGE_MAX_HSTATE (2 + (NAPOT_ORDER_MAX - NAPOT_PAGE_ORDER_BASE))#else #define HUGE_MAX_HSTATE 2#endif
@@ -213,7 +223,7 @@ static inline void pud_clear(pud_t *pudp)static inline pud_t pfn_pud (unsigned long pfn,pgprot_t prot) { -return __pud ((pfn << _PAGE_PFN_SHIFT) |pgprot_val (prot)); +return __pud ((pfn_to_hwpfn (pfn) << _PAGE_PFN_SHIFT) |pgprot_val (prot)); }static inline unsigned long _pud_pfn (pud_t pud)
@@ -257,7 +267,7 @@ static inline pmd_t pfn_pmd(unsigned long pfn, pgprot_t prot)ALT_THEAD_PMA (prot_val); -return __pmd ((pfn << _PAGE_PFN_SHIFT) | prot_val); +return __pmd ((pfn_to_hwpfn (pfn) << _PAGE_PFN_SHIFT) | prot_val); }static inline unsigned long _pmd_pfn (pmd_t pmd)
@@ -316,7 +326,7 @@ static inline void p4d_clear(p4d_t *p4d)static inline p4d_t pfn_p4d (unsigned long pfn,pgprot_t prot) { -return __p4d ((pfn << _PAGE_PFN_SHIFT) |pgprot_val (prot)); +return __p4d ((pfn_to_hwpfn (pfn) << _PAGE_PFN_SHIFT) |pgprot_val (prot)); }static inline unsigned long _p4d_pfn (p4d_t p4d)
arch/riscv/include/asm/pgtable.h +10 -9
@@ -26,9 +26,9 @@#endif /* Number of entries in the page global directory */ -#define PTRS_PER_PGD (PAGE_SIZE / sizeof(pgd_t)) +#define PTRS_PER_PGD (HW_PAGE_SIZE / sizeof(pgd_t))/* Number of entries in the page table */ -#define PTRS_PER_PTE (PAGE_SIZE / sizeof(pte_t)) +#define PTRS_PER_PTE (HW_PAGE_SIZE / sizeof(pte_t))/* * Half of the kernel address space (1/4 of the entries of the page global
@@ -118,7 +118,8 @@#include <linux/mm_types.h> #include <asm/compat.h> -#define __page_val_to_pfn (_val) (((_val) & _PAGE_PFN_MASK) >> _PAGE_PFN_SHIFT) +#define __page_val_to_hwpfn (_val) (((_val) & _PAGE_PFN_MASK) >> _PAGE_PFN_SHIFT) +#define __page_val_to_pfn (_val) hwpfn_to_pfn(__page_val_to_hwpfn(_val))#ifdef CONFIG_64BIT #include <asm/pgtable-64.h>
@@ -287,7 +288,7 @@ static inline pgd_t pfn_pgd(unsigned long pfn, pgprot_t prot)ALT_THEAD_PMA (prot_val); -return __pgd ((pfn << _PAGE_PFN_SHIFT) | prot_val); +return __pgd ((pfn_to_hwpfn (pfn) << _PAGE_PFN_SHIFT) | prot_val); }static inline unsigned long _pgd_pfn (pgd_t pgd)
@@ -351,12 +352,12 @@ static inline unsigned long pte_napot(pte_t pte)/* Yields the page frame number (PFN) of a page table entry */ static inline unsigned long pte_pfn (pte_t pte) { -unsigned long res =__page_val_to_pfn (pte_val (pte)); +unsigned long res =__page_val_to_hwpfn (pte_val (pte));if (has_svnapot () &&pte_napot (pte)) res = res & (res -1UL ); -return res; +return hwpfn_to_pfn (res); }#define pte_page (x) pfn_to_page(pte_pfn(x))
@@ -368,7 +369,7 @@ static inline pte_t pfn_pte(unsigned long pfn, pgprot_t prot)ALT_THEAD_PMA (prot_val); -return __pte ((pfn << _PAGE_PFN_SHIFT) | prot_val); +return __pte ((pfn_to_hwpfn (pfn) << _PAGE_PFN_SHIFT) | prot_val); }#define mk_pte (page, prot) pfn_pte(page_to_pfn(page), prot)
@@ -723,14 +724,14 @@ static inline pmd_t pmd_mkinvalid(pmd_t pmd)return __pmd (pmd_val (pmd) & ~(_PAGE_PRESENT|_PAGE_PROT_NONE)); } -#define __pmd_to_phys (pmd) (__page_val_to_pfn(pmd_val(pmd)) << PAGE_SHIFT) +#define __pmd_to_phys (pmd) (__page_val_to_hwpfn(pmd_val(pmd)) << HW_PAGE_SHIFT)static inline unsigned long pmd_pfn (pmd_t pmd) {return ((__pmd_to_phys (pmd) &PMD_MASK ) >>PAGE_SHIFT ); } -#define __pud_to_phys (pud) (__page_val_to_pfn(pud_val(pud)) << PAGE_SHIFT) +#define __pud_to_phys (pud) (__page_val_to_hwpfn(pud_val(pud)) << HW_PAGE_SHIFT)static inline unsigned long pud_pfn (pud_t pud) {
arch/riscv/mm/init.c +9 -9
@@ -466,7 +466,7 @@ static void __init create_pmd_mapping(pmd_t *pmdp, pte_phys = pt_ops.alloc_pte (va); pmdp[pmd_idx] =pfn_pmd (PFN_DOWN (pte_phys),PAGE_TABLE ); ptep = pt_ops.get_pte_virt (pte_phys); -memset (ptep,0 ,PAGE_SIZE ); +memset (ptep,0 ,PTRS_PER_PTE *sizeof (pte_t)); }else { pte_phys =PFN_PHYS (_pmd_pfn (pmdp[pmd_idx])); ptep = pt_ops.get_pte_virt (pte_phys);
@@ -569,7 +569,7 @@ static void __init create_pud_mapping(pud_t *pudp, next_phys = pt_ops.alloc_pmd (va); pudp[pud_index] =pfn_pud (PFN_DOWN (next_phys),PAGE_TABLE ); nextp = pt_ops.get_pmd_virt (next_phys); -memset (nextp,0 ,PAGE_SIZE ); +memset (nextp,0 ,PTRS_PER_PMD *sizeof (pmd_t)); }else { next_phys =PFN_PHYS (_pud_pfn (pudp[pud_index])); nextp = pt_ops.get_pmd_virt (next_phys);
@@ -596,7 +596,7 @@ static void __init create_p4d_mapping(p4d_t *p4dp, next_phys = pt_ops.alloc_pud (va); p4dp[p4d_index] =pfn_p4d (PFN_DOWN (next_phys),PAGE_TABLE ); nextp = pt_ops.get_pud_virt (next_phys); -memset (nextp,0 ,PAGE_SIZE ); +memset (nextp,0 ,PTRS_PER_PUD *sizeof (pud_t)); }else { next_phys =PFN_PHYS (_p4d_pfn (p4dp[p4d_index])); nextp = pt_ops.get_pud_virt (next_phys);
@@ -654,7 +654,7 @@ void __init create_pgd_mapping(pgd_t *pgdp, next_phys =alloc_pgd_next (va); pgdp[pgd_idx] =pfn_pgd (PFN_DOWN (next_phys),PAGE_TABLE ); nextp =get_pgd_next_virt (next_phys); -memset (nextp,0 ,PAGE_SIZE ); +memset (nextp,0 ,PTRS_PER_P4D *sizeof (p4d_t)); }else { next_phys =PFN_PHYS (_pgd_pfn (pgdp[pgd_idx])); nextp =get_pgd_next_virt (next_phys);
@@ -815,16 +815,16 @@ static __init void set_satp_mode(uintptr_t dtb_pa)if (hw_satp != identity_satp) {if (pgtable_l5_enabled) {disable_pgtable_l5 (); -memset (early_pg_dir,0 ,PAGE_SIZE ); +memset (early_pg_dir,0 ,PTRS_PER_PGD *sizeof (pgd_t)); goto retry; }disable_pgtable_l4 (); } -memset (early_pg_dir,0 ,PAGE_SIZE ); -memset (early_p4d,0 ,PAGE_SIZE ); -memset (early_pud,0 ,PAGE_SIZE ); - memset(early_pmd, 0, PAGE_SIZE); +memset (early_pg_dir,0 ,PTRS_PER_PGD *sizeof (pgd_t)); +memset (early_p4d,0 ,PTRS_PER_P4D *sizeof (p4d_t)); +memset (early_pud,0 ,PTRS_PER_PUD *sizeof (pud_t)); + memset(early_pmd, 0, PTRS_PER_PMD * sizeof(pmd_t)); }#endif
Signature · 7 lines
2.20.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 5/11 riscv: Decouple pmd operations and pte operations
In reply to Xu LuExisting pmd operations are usually implemented via pte operations. For example, the pmd_mkdirty function, which is used to mark a pmd_t struct as dirty, will transfer pmd_t struct to pte_t struct via pmd_pte first, mark the generated pte_t as dirty then, and finally transfer it back to pmd_t struct via pte_pmd function. Such implementation introduces unnecessary overhead of struct transferring. Also, Now that pte_t struct is a number of page table entries, which can be larger than pmd_t struct, functions like set_pmd_at implemented via set_pte_at will cause write amplifications.
This commit decouples pmd operations and pte operations. Pmd operations are now implemented independently of pte operations.
--- arch/riscv/include/asm/pgtable-64.h | 6 ++ arch/riscv/include/asm/pgtable.h | 124 +++++++++++++++++++++------- include/asm-generic/pgtable-nopmd.h | 1 + include/linux/pgtable.h | 6 ++ 4 files changed, 108 insertions(+), 29 deletions(-)
arch/riscv/include/asm/pgtable-64.h +6 -0
@@ -206,6 +206,12 @@ static inline int pud_leaf(pud_t pud)return pud_present (pud) && (pud_val (pud) & _PAGE_LEAF); } +#define pud_exec pud_exec +static inline int pud_exec (pud_t pud) +{ +return pud_val (pud) & _PAGE_EXEC; +} +static inline int pud_user (pud_t pud) {return pud_val (pud) & _PAGE_USER;
arch/riscv/include/asm/pgtable.h +95 -29
219 lines not on this page · raw message
include/asm-generic/pgtable-nopmd.h +1 -0
@@ -32,6 +32,7 @@ static inline int pud_bad(pud_t pud) { return 0; }static inline int pud_present (pud_t pud) {return 1 ; }static inline int pud_user (pud_t pud) {return 0 ; }static inline int pud_leaf (pud_t pud) {return 0 ; } +static inline int pud_exec (pud_t pud) {return 0 ; }static inline void pud_clear (pud_t *pud) { }#define pmd_ERROR (pmd) (pud_ERROR((pmd).pud))
include/linux/pgtable.h +6 -0
@@ -1630,9 +1630,15 @@ typedef unsigned int pgtbl_mod_mask;#ifndef pud_leaf#define pud_leaf (x) 0#endif +#ifndef pud_exec +#define pud_exec (x) 0 +#endif #ifndef pmd_leaf#define pmd_leaf (x) 0#endif +#ifndef pmd_exec +#define pmd_exec (x) 0 +#endif #ifndef pgd_leaf_size#define pgd_leaf_size (x) (1ULL << PGDIR_SHIFT)
Signature · 7 lines
2.20.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 6/11 riscv: Distinguish pmd huge pte and napot huge pte
In reply to Xu LuThere exist two kinds of huge pte on RISC-V: pmd and napot pte. For pmd kind huge pte, the huge page it represents is much larger than base page. Thus pmd kind huge pte can be represented via a single pmd entry and needs no special handling. For napot kind huge pte, it is actually several normal pte entries encoded in Svnapot format. Thus napot kind huge pte should be represented via pte_t struct and handled via pte operations.
This commit distinguishes these two kinds of huge pte and handles them with different operations.
--- arch/riscv/include/asm/hugetlb.h | 71 +++++++++++++++++++++++++++++++- arch/riscv/mm/hugetlbpage.c | 40 ++++++++++++------ 2 files changed, 97 insertions(+), 14 deletions(-)
arch/riscv/include/asm/hugetlb.h +70 -1
@@ -4,6 +4,7 @@#include <asm/cacheflush.h> #include <asm/page.h> +#include <asm/pgtable.h> static inline void arch_clear_hugepage_flags (struct page *page) {
@@ -12,6 +13,7 @@ static inline void arch_clear_hugepage_flags(struct page *page)#define arch_clear_hugepage_flags arch_clear_hugepage_flags#ifdef CONFIG_RISCV_ISA_SVNAPOT +#define __HAVE_ARCH_HUGE_PTE_CLEARvoid huge_pte_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep,unsigned long sz);
@@ -41,10 +43,77 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,#define __HAVE_ARCH_HUGE_PTEP_GETpte_t huge_ptep_get (pte_t *ptep); +#define __HAVE_ARCH_HUGE_PTEP_GET_LOCKLESS +static inline pte_t huge_ptep_get_lockless (pte_t *ptep) +{ +unsigned long pteval =READ_ONCE (ptep->ptes[0 ]); + +return __pte (pteval); +} +pte_t arch_make_huge_pte (pte_t entry,unsigned int shift,vm_flags_t flags);#define arch_make_huge_pte arch_make_huge_pte -#endif /*CONFIG_RISCV_ISA_SVNAPOT*/ +#else /* CONFIG_RISCV_ISA_SVNAPOT */ + +#define __HAVE_ARCH_HUGE_PTEP_GET +static inline pte_t huge_ptep_get (pte_t *ptep) +{ +pmd_t *pmdp = (pmd_t *)ptep; + +return pmd_pte (pmdp_get (pdmp)); +} + +#define __HAVE_ARCH_HUGE_PTEP_GET_LOCKLESS +static inline pte_t huge_ptep_get_lockless (pte_t *ptep) +{ +return huge_ptep_get (ptep); +} + +#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT +static inline void set_huge_pte_at (struct mm_struct *mm,unsigned long addr, +pte_t *ptep,pte_t pte) +{ +set_pmd_at (mm, addr, (pmd_t *)ptep,pte_pmd (pte)); +} + +#define __HAVE_ARCH_HUGE_PTEP_SET_ACCESS_FLAGS +static inline int huge_ptep_set_access_flags (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep, +pte_t pte,int dirty) +{ +return pmdp_set_access_flags (vma, addr, (pmd_t *)ptep,pte_pmd (pte), dirty); +} + +#define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR +static inline pte_t huge_ptep_get_and_clear (struct mm_struct *mm, +unsigned long addr,pte_t *ptep) +{ +return pmd_pte (pmdp_get_and_clear (mm, addr, (pmd_t *)ptep)); +} + +#define __HAVE_ARCH_HUGE_PTEP_SET_WRPROTECT +static inline void huge_ptep_set_wrprotect (struct mm_struct *mm, +unsigned long addr,pte_t *ptep) +{ +pmdp_set_wrprotect (mm, addr, (pmd_t *)ptep); +} + +#define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSH +static inline pte_t huge_ptep_clear_flush (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep) +{ +return pmd_pte (pmdp_clear_flush (vma, addr, (pmd_t *)ptep)); +} + +#define __HAVE_ARCH_HUGE_PTE_CLEAR +static inline void huge_pte_clear (struct mm_struct *mm,unsigned long addr, +pte_t *ptep,unsigned long sz) +{ +pmd_clear ((pmd_t *)ptep); +} + +#endif /* CONFIG_RISCV_ISA_SVNAPOT */ #include <asm-generic/hugetlb.h>
arch/riscv/mm/hugetlbpage.c +27 -13
@@ -7,8 +7,13 @@ pte_t huge_ptep_get(pte_t *ptep) {unsigned long pte_num;int i; -pte_t orig_pte =ptep_get (ptep); +pmd_t *pmdp = (pmd_t *)ptep; +pte_t orig_pte =pmd_pte (pmdp_get (pmdp)); +/* +* Non napot pte indicates a middle page table entry and +* should be treated as a pmd. +*/ if (!pte_present (orig_pte) || !pte_napot (orig_pte))return orig_pte;
@@ -198,6 +203,8 @@ void set_huge_pte_at(struct mm_struct *mm, hugepage_shift =PAGE_SHIFT ; pte_num = sz >> hugepage_shift; +if (pte_num ==1 ) +set_pmd_at (mm, addr, (pmd_t *)ptep,pte_pmd (pte));for (i =0 ; i < pte_num; i++, ptep++, addr += (1 << hugepage_shift))set_pte_at (mm, addr, ptep, pte); }
@@ -214,7 +221,8 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,int i, pte_num;if (!pte_napot (pte)) -return ptep_set_access_flags (vma, addr, ptep, pte, dirty); +return pmdp_set_access_flags (vma, addr, (pmd_t *)ptep, +pte_pmd (pte), dirty); order =napot_cont_order (pte); pte_num =napot_pte_num (order);
@@ -237,11 +245,12 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm,unsigned long addr, pte_t *ptep) { - pte_t orig_pte = ptep_get(ptep); + pmd_t *pmdp = (pmd_t *)ptep; +pte_t orig_pte =pmd_pte (pmdp_get (pmdp));int pte_num;if (!pte_napot (orig_pte)) -return ptep_get_and_clear (mm, addr, ptep); +return pmd_pte (pmdp_get_and_clear (mm, addr, pmdp)); pte_num =napot_pte_num (napot_cont_order (orig_pte));
@@ -252,13 +261,14 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm,unsigned long addr, pte_t *ptep) { - pte_t pte = ptep_get(ptep); + pmd_t *pmdp = (pmd_t *)ptep; +pte_t pte =pmd_pte (pmdp_get (pmdp));unsigned long order;pte_t orig_pte;int i, pte_num;if (!pte_napot (pte)) { -ptep_set_wrprotect (mm, addr, ptep); +pmdp_set_wrprotect (mm, addr, pmdp);return ; }
@@ -277,11 +287,12 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma,unsigned long addr, pte_t *ptep) { - pte_t pte = ptep_get(ptep); + pmd_t *pmdp = (pmd_t *)ptep; +pte_t pte =pmd_pte (pmdp_get (pmdp));int pte_num;if (!pte_napot (pte)) -return ptep_clear_flush (vma, addr, ptep); +return pmd_pte (pmdp_clear_flush (vma, addr, pmdp)); pte_num =napot_pte_num (napot_cont_order (pte));
@@ -293,11 +304,12 @@ void huge_pte_clear(struct mm_struct *mm, pte_t *ptep, unsigned long sz) { -pte_t pte =ptep_get (ptep); +pmd_t *pmdp = (pmd_t *)ptep; +pte_t pte =pmd_pte (pmdp_get (pmdp));int i, pte_num;if (!pte_napot (pte)) { -pte_clear (mm, addr, ptep); +pmd_clear (pmdp);return ; }
@@ -325,8 +337,10 @@ static __init int napot_hugetlbpages_init(void)if (has_svnapot ()) {unsigned long order; -for_each_napot_order (order) -hugetlb_add_hstate (order); +for_each_napot_order (order) { +if (napot_cont_shift (order) >PAGE_SHIFT ) +hugetlb_add_hstate (order); + } }return 0 ; }
@@ -357,7 +371,7 @@ bool __init arch_hugetlb_valid_size(unsigned long size)return true; else if (IS_ENABLED(CONFIG_64BIT) && size == PUD_SIZE) return true; - else if (is_napot_size(size)) + else if (is_napot_size(size) && size > PAGE_SIZE) return true; else return false;
Signature · 7 lines
2.20.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 7/11 riscv: Adapt satp operations to gap between hw page and sw page
In reply to Xu LuThe control register CSR_SATP on RISC-V, which points to the root page table page, is used by MMU to translate va to pa when TLB miss happens. Thus it should be encoded at a granularity of hardware page, while existing code usually encodes it via software page frame number.
This commit corrects encoding operations of CSR_SATP register. To get developers rid of the annoying encoding format of CSR_SATP and the conversion between sw pfn and hw pfn, we abstract the encoding operations of CSR_SATP into a specific function.
--- arch/riscv/include/asm/pgtable.h | 7 +++++++ arch/riscv/kernel/head.S | 4 ++-- arch/riscv/kernel/hibernate.c | 3 ++- arch/riscv/mm/context.c | 7 +++---- arch/riscv/mm/fault.c | 1 + arch/riscv/mm/init.c | 7 +++++-- arch/riscv/mm/kasan_init.c | 7 +++++-- 7 files changed, 25 insertions(+), 11 deletions(-)
arch/riscv/include/asm/pgtable.h +7 -0
@@ -213,6 +213,13 @@ extern pgd_t swapper_pg_dir[];extern pgd_t trampoline_pg_dir[];extern pgd_t early_pg_dir[]; +static inline unsigned long make_satp (unsigned long pfn, +unsigned long asid,unsigned long satp_mode) +{ +return (pfn_to_hwpfn (pfn) | + ((asid &SATP_ASID_MASK ) <<SATP_ASID_SHIFT ) | satp_mode); +} + static __always_inline int __pte_present(unsigned long pteval) {return (pteval & (_PAGE_PRESENT | _PAGE_PROT_NONE));
arch/riscv/kernel/head.S +2 -2
@@ -87,7 +87,7 @@ relocate_enable_mmu: csrw CSR_TVEC, a2 /* Compute satp for kernel page tables, but don't load it yet */ - srl a2, a0, PAGE_SHIFT + srl a2, a0, HW_PAGE_SHIFT la a1, satp_mode REG_L a1, 0(a1) or a2, a2, a1
@@ -100,7 +100,7 @@ relocate_enable_mmu: */ la a0, trampoline_pg_dir XIP_FIXUP_OFFSET a0 - srl a0, a0, PAGE_SHIFT + srl a0, a0, HW_PAGE_SHIFT or a0, a0, a1 sfence.vma csrw CSR_SATP, a0
arch/riscv/kernel/hibernate.c +2 -1
@@ -395,7 +395,8 @@ int swsusp_arch_resume(void)if (ret)return ret; -hibernate_restore_image (resume_hdr.saved_satp, (PFN_DOWN (__pa (resume_pg_dir)) | satp_mode), +hibernate_restore_image (resume_hdr.saved_satp, +make_satp (PFN_DOWN (__pa (resume_pg_dir)),0 , satp_mode), resume_hdr.restore_cpu_addr);return 0 ;
arch/riscv/mm/context.c +3 -4
@@ -190,9 +190,8 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu)raw_spin_unlock_irqrestore (&context_lock, flags); switch_mm_fast: -csr_write (CSR_SATP ,virt_to_pfn (mm->pgd) | - ((cntx & asid_mask) <<SATP_ASID_SHIFT ) | - satp_mode); +csr_write (CSR_SATP ,make_satp (virt_to_pfn (mm->pgd), (cntx & asid_mask), + satp_mode));if (need_flush_tlb)local_flush_tlb_all ();
@@ -201,7 +200,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu)static void set_mm_noasid (struct mm_struct *mm) {/* Switch the page table and blindly nuke entire local TLB */ -csr_write (CSR_SATP ,virt_to_pfn (mm->pgd) | satp_mode); +csr_write (CSR_SATP ,make_satp (virt_to_pfn (mm->pgd),0 , satp_mode));local_flush_tlb_all (); }
arch/riscv/mm/fault.c +1 -0
@@ -133,6 +133,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a*/ index =pgd_index (addr); pfn =csr_read (CSR_SATP ) &SATP_PPN ; + pfn =hwpfn_to_pfn (pfn); pgd = (pgd_t *)pfn_to_virt (pfn) + index; pgd_k = init_mm.pgd + index;
arch/riscv/mm/init.c +5 -2
@@ -805,7 +805,7 @@ static __init void set_satp_mode(uintptr_t dtb_pa) (uintptr_t)early_p4d : (uintptr_t)early_pud, PGDIR_SIZE, PAGE_TABLE); - identity_satp =PFN_DOWN ((uintptr_t )&early_pg_dir) | satp_mode; + identity_satp =make_satp (PFN_DOWN ((uintptr_t )&early_pg_dir),0 , satp_mode);local_flush_tlb_all ();csr_write (CSR_SATP , identity_satp);
@@ -1285,6 +1285,8 @@ static void __init create_linear_mapping_page_table(void)static void __init setup_vm_final (void ) { +unsigned long satp; +/* Setup swapper PGD for fixmap */ #if !defined(CONFIG_64BIT )/*
@@ -1318,7 +1320,8 @@ static void __init setup_vm_final(void)clear_fixmap (FIX_P4D );/* Move to swapper page table */ -csr_write (CSR_SATP ,PFN_DOWN (__pa_symbol (swapper_pg_dir)) | satp_mode); + satp =make_satp (PFN_DOWN (__pa_symbol (swapper_pg_dir)),0 , satp_mode); +csr_write (CSR_SATP , satp);local_flush_tlb_all ();pt_ops_set_late ();
arch/riscv/mm/kasan_init.c +5 -2
@@ -471,11 +471,13 @@ static void __init create_tmp_mapping(void)void __init kasan_init (void ) { +unsigned long satp;phys_addr_t p_start, p_end;u64 i;create_tmp_mapping (); -csr_write (CSR_SATP ,PFN_DOWN (__pa (tmp_pg_dir)) | satp_mode); + satp =make_satp (PFN_DOWN (__pa (tmp_pg_dir)),0 , satp_mode); +csr_write (CSR_SATP , satp);kasan_early_clear_pgd (pgd_offset_k (KASAN_SHADOW_START ),KASAN_SHADOW_START ,KASAN_SHADOW_END );
@@ -520,6 +522,7 @@ void __init kasan_init(void)memset (kasan_early_shadow_page,KASAN_SHADOW_INIT ,PAGE_SIZE ); init_task.kasan_depth =0 ; -csr_write (CSR_SATP ,PFN_DOWN (__pa (swapper_pg_dir)) | satp_mode); + satp =make_satp (PFN_DOWN (__pa (swapper_pg_dir)),0 , satp_mode); +csr_write (CSR_SATP , satp);local_flush_tlb_all (); }
Signature · 7 lines
2.20.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 8/11 riscv: Apply Svnapot for base page mapping
In reply to Xu LuThe Svnapot extension on RISC-V is like contiguous PTE on ARM64. It allows ptes of a naturally aligned power-of 2 (NAPOT) memory range be encoded in the same format to save the TLB space.
This commit applies Svnapot for each base page's mapping. This commit is the key to achieving larger base page's performance optimization.
--- arch/riscv/include/asm/pgtable.h | 34 +++++++++++++++++++++++++++----- 1 file changed, 29 insertions(+), 5 deletions(-)
arch/riscv/include/asm/pgtable.h +29 -5
@@ -230,6 +230,16 @@ static __always_inline unsigned long __pte_napot(unsigned long pteval)return pteval & _PAGE_NAPOT; } +static __always_inline unsigned long __pte_mknapot (unsigned long pteval, +unsigned int order) +{ +int pos = order -1 + _PAGE_PFN_SHIFT; +unsigned long napot_bit =BIT (pos); +unsigned long napot_mask = ~GENMASK (pos, _PAGE_PFN_SHIFT); + +return (pteval & napot_mask) | napot_bit | _PAGE_NAPOT; +} +static inline pte_t __pte (unsigned long pteval) {pte_t pte;
@@ -348,13 +358,11 @@ static inline unsigned long pte_napot(pte_t pte) return __pte_napot(pte_val(pte)); } -static inline pte_t pte_mknapot (pte_t pte,unsigned int order) +static inline pte_t pte_mknapot (pte_t pte,unsigned int page_order) { -int pos = order -1 + _PAGE_PFN_SHIFT; -unsigned long napot_bit =BIT (pos); -unsigned long napot_mask = ~GENMASK (pos, _PAGE_PFN_SHIFT); +unsigned int hw_page_order = page_order + (PAGE_SHIFT -HW_PAGE_SHIFT ); -return __pte ((pte_val (pte) & napot_mask) | napot_bit | _PAGE_NAPOT); +return __pte (__pte_mknapot (pte_val (pte), hw_page_order)); }#else
@@ -366,6 +374,11 @@ static inline unsigned long pte_napot(pte_t pte) return 0; } +static inline pte_t pte_mknapot (pte_t pte,unsigned int page_order) +{ +return pte; +} +#endif /* CONFIG_RISCV_ISA_SVNAPOT */ /* Yields the page frame number (PFN) of a page table entry */
@@ -585,6 +598,17 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)*/ static inline void set_pte (pte_t *ptep,pte_t pteval) { +unsigned long order; + +/* +* has_svnapot() always return false before riscv_isa is initialized. +*/ +if (has_svnapot () &&pte_present (pteval) && !pte_napot (pteval)) { +for_each_napot_order (order) { +if (napot_cont_shift (order) ==PAGE_SHIFT ) + pteval =pte_mknapot (pteval, order); + } + } *ptep = pteval; }
Signature · 7 lines
2.20.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 9/11 riscv: Adjust fix_btmap slots number to match variable page size
In reply to Xu LuThe existing fixmap slot number will cause the fixmap size to exceed FIX_FDT_SIZE when base page becomes larger than 4K. This patch adjusts the slot number to make them always match.
--- arch/riscv/include/asm/fixmap.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/riscv/include/asm/fixmap.h +2 -1
@@ -44,7 +44,8 @@ enum fixed_addresses {* before ioremap() is functional. */ #define NR_FIX_BTMAPS (SZ_256K / PAGE_SIZE) -#define FIX_BTMAPS_SLOTS 7 +#define FIX_BTMAPS_SIZE (FIXADDR_SIZE - ((FIX_BTMAP_END + 1) << PAGE_SHIFT)) +#define FIX_BTMAPS_SLOTS (FIX_BTMAPS_SIZE / SZ_256K)#define TOTAL_FIX_BTMAPS (NR_FIX_BTMAPS * FIX_BTMAPS_SLOTS) FIX_BTMAP_END = __end_of_permanent_fixed_addresses,
Signature · 7 lines
2.20.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 10/11 riscv: kvm: Adapt kvm to gap between hw page and sw page
In reply to Xu LuExisting mmu code in kvm handles middle level page table entry and the last level page table entry in the same way, which is insufficient when base page becomes larger. For example, for 64K base page, per pte_t contains 16 page table entries while per pmd_t still contains one and thus needs to be handled in different ways.
This commit refines kvm mmu code to handle middle level page table entries and last level page table entries distinctively.
--- arch/riscv/include/asm/pgtable.h | 7 ++ arch/riscv/kvm/mmu.c | 198 +++++++++++++++++++++---------- 2 files changed, 145 insertions(+), 60 deletions(-)
arch/riscv/include/asm/pgtable.h +7 -0
@@ -220,6 +220,13 @@ static inline unsigned long make_satp(unsigned long pfn, ((asid & SATP_ASID_MASK) << SATP_ASID_SHIFT) | satp_mode); } +static inline unsigned long make_hgatp (unsigned long pfn, +unsigned long vmid,unsigned long hgatp_mode) +{ +return ((pfn_to_hwpfn (pfn) &HGATP_PPN ) | + ((vmid <<HGATP_VMID_SHIFT ) &HGATP_VMID ) | hgatp_mode); +} + static __always_inline int __pte_present(unsigned long pteval) {return (pteval & (_PAGE_PRESENT | _PAGE_PROT_NONE));
arch/riscv/kvm/mmu.c +138 -60
350 lines not on this page · raw message
Signature · 7 lines
2.20.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 11/11 riscv: Introduce 64K page size
In reply to Xu LuThis patch introduces new config to control whether enabling the 64K base page feature on RISC-V.
--- arch/Kconfig | 1 + arch/riscv/Kconfig | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+)
arch/Kconfig +1 -0
@@ -1087,6 +1087,7 @@ config HAVE_ARCH_COMPAT_MMAP_BASES config PAGE_SIZE_LESS_THAN_64KB def_bool y + depends on !RISCV_64K_PAGES depends on !ARM64_64K_PAGES depends on !PAGE_SIZE_64KB depends on !PARISC_PAGE_SIZE_64KB
arch/riscv/Kconfig +20 -0
@@ -227,6 +227,7 @@ config RISCV_HW_PAGE_SHIFT config RISCV_PAGE_SHIFT int + default 16 if RISCV_64K_PAGES default 12 config KASAN_SHADOW_OFFSET
@@ -692,6 +693,25 @@ config RISCV_BOOT_SPINWAIT If unsure what to do here, say N. +choice + prompt "Page size" + default RISCV_4K_PAGES + help + Page size (translation granule) configuration. + +config RISCV_4K_PAGES + bool "4KB" + help + This feature enables 4KB pages support. + +config RISCV_64K_PAGES + bool "64KB" + depends on ARCH_HAS_STRICT_KERNEL_RWX && 64BIT + help + This feature enables 64KB pages support (4KB by default) + +endchoice + config ARCH_SUPPORTS_KEXEC def_bool MMU
Signature · 7 lines
2.20.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Arnd Bergmann
maintainerQuoted 11 lines · Xu Lu
Some existing architectures like ARM supports base page larger than 4K as their MMU supports more page sizes. Thus, besides hugetlb page and transparent huge page, there is another way for these architectures to enjoy the benefits of fewer TLB misses without worrying about cost of splitting and merging huge pages. However, on architectures with only 4K MMU, larger base page is unavailable now.
This patch series attempts to break through the limitation of MMU and supports larger base page on RISC-V, which only supports 4K page size now.
Quoted 5 lines
The key idea to implement larger base page based on 4K MMU is to decouple the MMU page from the base page in view of kernel mm, which we denote as software page. In contrary to software page, we denote the MMU page as hardware page. Below is the difference between these two kinds of pages.
We have played with this on arm32, but the conclusion is that it's almost never worth the memory overhead, as most workloads end up using several times the amount of physical RAM after each small file in the page cache and any sparse populated anonymous memory area explodes to up to 16 times the size.
On ppc64, using 64KB pages was way to get around limitations in their hashed MMU design, which had a much bigger performance impact because any page table access ends up being a cache miss. On arm64, there are some CPUs like the Fujitsu A64FX that are really bad at 4KB pages and don't support 16KB pages, so this is the only real option.
You will see a notable performance benefit in synthetic benchmarks like speccpu with 64KB pages, or on specific computational workloads that have large densely packed memory chunks, but for real workloads, the usual answer is to just use transparent hugepages for larger mappings and a page size of no more than 16KB for the page cache.
With the work going into using folios in the kernel (see e.g.
https://lwn.net/Articles/932386/), even the workloads that benefit from 64KB base pages should be better off with 4KB pages and just using the TLB hints for large folios.
Arnd
Thanks a lot for your reply! And sorry for replying so late.
Quoted 32 lines · Arnd Bergmann
Quoted 11 lines · Xu Lu
Some existing architectures like ARM supports base page larger than 4K as their MMU supports more page sizes. Thus, besides hugetlb page and transparent huge page, there is another way for these architectures to enjoy the benefits of fewer TLB misses without worrying about cost of splitting and merging huge pages. However, on architectures with only 4K MMU, larger base page is unavailable now.
This patch series attempts to break through the limitation of MMU and supports larger base page on RISC-V, which only supports 4K page size now.
Quoted 5 lines
The key idea to implement larger base page based on 4K MMU is to decouple the MMU page from the base page in view of kernel mm, which we denote as software page. In contrary to software page, we denote the MMU page as hardware page. Below is the difference between these two kinds of pages.
We have played with this on arm32, but the conclusion is that it's almost never worth the memory overhead, as most workloads end up using several times the amount of physical RAM after each small file in the page cache and any sparse populated anonymous memory area explodes to up to 16 times the size.
On ppc64, using 64KB pages was way to get around limitations in their hashed MMU design, which had a much bigger performance impact because any page table access ends up being a cache miss. On arm64, there are some CPUs like the Fujitsu A64FX that are really bad at 4KB pages and don't support 16KB pages, so this is the only real option.
Quoted 6 lines
You will see a notable performance benefit in synthetic benchmarks like speccpu with 64KB pages, or on specific computational workloads that have large densely packed memory chunks, but for real workloads, the usual answer is to just use transparent hugepages for larger mappings and a page size of no more than 16KB for the page cache.
Actually we did find actual performance benefits brought by 64K page size in real business scenarios.
On the Ampere ARM server, when applying 64K base page size, we saw an improvement of 2.5x for both qps and latency on redis, a performance improvement of 10~20% on our own newsql database and 50% on object storage.
For mysql, the qps increases about 14%, 17.5% and 20% for read-only, write-only and random read/write workloads respectively. And the latency reduces about 13.7%, 15.8% and 14.5% on average. This is also why we chose to implement a similar feature on RISC-V in the beginning.
Quoted 5 lines
With the work going into using folios in the kernel (see e.g.
https://lwn.net/Articles/932386/), even the workloads that benefit from 64KB base pages should be better off with 4KB pages and just using the TLB hints for large folios.
Maybe 64K page size combined with large folios can achieve more benefits. As is mentioned in this patch[1], a 64K page size kernel combined with large folios and THPs via cont pte can achieve speedup of 10.5x on some memory-intensive workloads on arm64 SBSA server.
[1] https://lore.kernel.org/all/[email protected]/
Quoted 2 lines
Arnd
A gentle ping.
Quoted 103 lines · Xu Lu
Some existing architectures like ARM supports base page larger than 4K as their MMU supports more page sizes. Thus, besides hugetlb page and transparent huge page, there is another way for these architectures to enjoy the benefits of fewer TLB misses without worrying about cost of splitting and merging huge pages. However, on architectures with only 4K MMU, larger base page is unavailable now.
This patch series attempts to break through the limitation of MMU and supports larger base page on RISC-V, which only supports 4K page size now.
The key idea to implement larger base page based on 4K MMU is to decouple the MMU page from the base page in view of kernel mm, which we denote as software page. In contrary to software page, we denote the MMU page as hardware page. Below is the difference between these two kinds of pages.
1. Kernel memory management module manages, allocates and maps memory at a granularity of software page, which should not be restricted by MMU and can be larger than hardware page.
2. Architecture page table operations should be carried out from MMU's perspective and page table entries are encoded at a granularity of hardware page, which is 4K on RISC-V MMU now.
The main work to decouple these two kinds of pages lies in architecture code. For example, we turn the
pte_tstruct to an array of page table entries to match it with software page which can be larger than hardware page, and adapt the page table operations accordingly. For 64K software base page, thepte_tstruct now contains 16 contiguous page table entries which point to 16 contiguous 4K hardware pages.To achieve the benefits of large base page, we applies Svnapot for each base page's mapping. The Svnapot extension on RISC-V is like contiguous PTE on ARM64. It allows ptes of a naturally aligned power-of 2 size memory range be encoded in the same format to save the TLB space.
This patch series is the first version and is based on v6.7-rc1. This version supports both bare metal and virtualization scenarios.
In the next versions, we will continue on the following works:
1. Reduce the memory usage of page table page as it only uses 4K space while costs a whole base page.
2. When IMSIC interrupt file is smaller than 64K, extra isolation measures for the interrupt file are needed. (S)PMP and IOPMP may be good choices.
3. More consideration is needed to make this patch series collaborate with folios better.
4. Support 64K base page on IOMMU.
5. The performance test is on schedule to verify the actual performance improvement and the decrease in TLB miss rate.
Thanks in advance for comments.
Xu Lu (11):
mm: Fix misused APIs on huge pte
riscv: Introduce concept of hardware base page
riscv: Adapt pte struct to gap between hw page and sw page
riscv: Adapt pte operations to gap between hw page and sw page
riscv: Decouple pmd operations and pte operations
riscv: Distinguish pmd huge pte and napot huge pte
riscv: Adapt satp operations to gap between hw page and sw page
riscv: Apply Svnapot for base page mapping
riscv: Adjustfix_btmapslots number to match variable page size
riscv: kvm: Adapt kvm to gap between hw page and sw page
riscv: Introduce 64K page sizearch/Kconfig | 1 + arch/riscv/Kconfig | 28 +++ arch/riscv/include/asm/fixmap.h | 3 +- arch/riscv/include/asm/hugetlb.h | 71 ++++++- arch/riscv/include/asm/page.h | 16 +- arch/riscv/include/asm/pgalloc.h | 21 ++- arch/riscv/include/asm/pgtable-32.h | 2 +- arch/riscv/include/asm/pgtable-64.h | 45 +++-- arch/riscv/include/asm/pgtable.h | 282 +++++++++++++++++++++++----- arch/riscv/kernel/efi.c | 2 +- arch/riscv/kernel/head.S | 4 +- arch/riscv/kernel/hibernate.c | 3 +- arch/riscv/kvm/mmu.c | 198 +++++++++++++------ arch/riscv/mm/context.c | 7 +- arch/riscv/mm/fault.c | 1 + arch/riscv/mm/hugetlbpage.c | 42 +++-- arch/riscv/mm/init.c | 25 +-- arch/riscv/mm/kasan_init.c | 7 +- arch/riscv/mm/pageattr.c | 2 +- fs/proc/task_mmu.c | 2 +- include/asm-generic/hugetlb.h | 7 + include/asm-generic/pgtable-nopmd.h | 1 + include/linux/pgtable.h | 6 + mm/hugetlb.c | 2 +- mm/migrate.c | 5 +- mm/mprotect.c | 2 +- mm/rmap.c | 10 +- mm/vmalloc.c | 3 +- 28 files changed, 616 insertions(+), 182 deletions(-)
Quoted 3 lines
--
2.20.1
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv