riscv: mm: Pre-allocate PGD entries vmalloc/modules area
Subsystems (2)
PATCH 1/1 riscv: mm: Pre-allocate PGD entries vmalloc/modules area
From: Björn Töpel <[email protected]>
The RISC-V port requires that kernel PGD entries are to be synchronized between MMs. This is done via the vmalloc_fault() function, that simply copies the PGD entries from init_mm to the faulting one.
Historically, faulting in PGD entries have been a source for both bugs [1], and poor performance.
One way to get rid of vmalloc faults is by pre-allocating the PGD entries. Pre-allocating the entries potientially wastes 64 * 4K (65 on SV39). The pre-allocation function is pulled from Jörg Rödel's x86 work, with the addition of 3-level page tables (PMD allocations).
The pmd_alloc() function needs the ptlock cache to be initialized (when split page locks is enabled), so the pre-allocation is done in a RISC-V specific pgtable_cache_init() implementation.
Pre-allocate the kernel PGD entries for the vmalloc/modules area, but only for 64b platforms.
--- arch/riscv/mm/fault.c | 20 +++------------ arch/riscv/mm/init.c | 58 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+), 16 deletions(-)
arch/riscv/mm/fault.c +4 -16
@@ -230,32 +230,20 @@ void handle_page_fault(struct pt_regs *regs)return ;/* -* Fault-in kernel-space virtual memory on-demand. -* The 'reference' page table is init_mm.pgd. +* Fault-in kernel-space virtual memory on-demand, for 32-bit +* architectures. The 'reference' page table is init_mm.pgd. * * NOTE! We MUST NOT take any locks for this case. We may * be in an interrupt or a critical region, and should * only copy the information from the master page table, * nothing more. */ -if (unlikely ((addr >=VMALLOC_START ) && (addr <VMALLOC_END ))) { +if (!IS_ENABLED (CONFIG_64BIT ) && +unlikely (addr >=VMALLOC_START && addr <VMALLOC_END )) {vmalloc_fault (regs, code, addr);return ; } -#ifdef CONFIG_64BIT -/* -* Modules in 64bit kernels lie in their own virtual region which is not -* in the vmalloc region, but dealing with page faults in this region -* or the vmalloc region amounts to doing the same thing: checking that -* the mapping exists in init_mm.pgd and updating user page table, so -* just use vmalloc_fault. -*/ -if (unlikely (addr >=MODULES_VADDR && addr <MODULES_END )) { -vmalloc_fault (regs, code, addr); -return ; - } -#endif /* Enable interrupts if they were enabled in the parent context. */ if (!regs_irqs_disabled (regs))local_irq_enable ();
arch/riscv/mm/init.c +58 -0
61 lines not on this page · raw message
base-commit: ac9a78681b921877518763ba0e89202254349d1b
Signature · 1 line
2.39.2
Quoted 19 lines · Björn Töpel
From: Björn Töpel <[email protected]>
The RISC-V port requires that kernel PGD entries are to be synchronized between MMs. This is done via the
vmalloc_fault()function, that simply copies the PGD entries frominit_mmto the faulting one.Historically, faulting in PGD entries have been a source for both bugs [1], and poor performance.
One way to get rid of vmalloc faults is by pre-allocating the PGD entries. Pre-allocating the entries potientially wastes 64 * 4K (65 on SV39). The pre-allocation function is pulled from Jörg Rödel's x86 work, with the addition of 3-level page tables (PMD allocations).
The
pmd_alloc()function needs the ptlock cache to be initialized (when split page locks is enabled), so the pre-allocation is done in a RISC-V specificpgtable_cache_init()implementation.
Quoted 5 lines
Pre-allocate the kernel PGD entries for the vmalloc/modules area, but only for 64b platforms.
Signed-off-by Björn Töpel <[email protected]>
...and the build was bitten by nommu [1]. I'll hold off the v2 a bit, to see if there's any other comments.
Björn
[1] https://patchwork.kernel.org/project/linux-riscv/patch/[email protected]/
kernel test robot
build botkernel test robot noticed the following build warnings:
Hi Björn,
kernel test robot noticed the following build warnings:
[auto build test WARNING on ac9a78681b921877518763ba0e89202254349d1b]
url: https://github.com/intel-lab-lkp/linux/commits/Bj-rn-T-pel/riscv-mm-Pre-allocate-PGD-entries-vmalloc-modules-area/20230530-020212 base: ac9a78681b921877518763ba0e89202254349d1b patch link: https://lore.kernel.org/r/20230529180023.289904-1-bjorn%40kernel.org patch subject: [PATCH] riscv: mm: Pre-allocate PGD entries vmalloc/modules area config: riscv-randconfig-r042-20230530 (https://download.01.org/0day-ci/archive/20230531/[email protected]/config) compiler: riscv64-linux-gcc (GCC) 12.3.0
reproduce (this is a W=1 build):
mkdir -p ~/bin
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/9383adbcdfb7f850149742579615e8c174328a78
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Bj-rn-T-pel/riscv-mm-Pre-allocate-PGD-entries-vmalloc-modules-area/20230530-020212 git checkout 9383adbcdfb7f850149742579615e8c174328a78
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.3.0 ~/bin/make.cross W=1 O=build_dir ARCH=riscv olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.3.0 ~/bin/make.cross W=1 O=build_dir ARCH=riscv SHELL=/bin/bash arch/riscv/mm/
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
arch/riscv/mm/init.c: In function 'preallocate_pgd_pages_range':
arch/riscv/mm/init.c:1381:30: error: implicit declaration of function 'pgd_offset_k'; did you mean 'p4d_offset'? [-Werror=implicit-function-declaration]
1381 | pgd_t *pgd = pgd_offset_k(addr);
| ^~~~~~~~~~~~
| p4d_offset
>> arch/riscv/mm/init.c:1381:30: warning: initialization of 'pgd_t *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
arch/riscv/mm/init.c:1387:23: error: implicit declaration of function 'p4d_alloc'; did you mean 'd_alloc'? [-Werror=implicit-function-declaration]
1387 | p4d = p4d_alloc(&init_mm, pgd, addr);
| ^~~~~~~~~
| d_alloc
>> arch/riscv/mm/init.c:1387:21: warning: assignment to 'p4d_t *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
1387 | p4d = p4d_alloc(&init_mm, pgd, addr);
| ^
arch/riscv/mm/init.c:1395:23: error: implicit declaration of function 'pud_alloc'; did you mean 'd_alloc'? [-Werror=implicit-function-declaration]
1395 | pud = pud_alloc(&init_mm, p4d, addr);
| ^~~~~~~~~
| d_alloc
>> arch/riscv/mm/init.c:1395:21: warning: assignment to 'pud_t *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
1395 | pud = pud_alloc(&init_mm, p4d, addr);
| ^
arch/riscv/mm/init.c:1403:23: error: implicit declaration of function 'pmd_alloc'; did you mean 'mm_alloc'? [-Werror=implicit-function-declaration]
1403 | pmd = pmd_alloc(&init_mm, pud, addr);
| ^~~~~~~~~
| mm_alloc
>> arch/riscv/mm/init.c:1403:21: warning: assignment to 'pmd_t *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
1403 | pmd = pmd_alloc(&init_mm, pud, addr);
| ^
arch/riscv/mm/init.c: In function 'pgtable_cache_init':
arch/riscv/mm/init.c:1421:45: error: 'MODULES_VADDR' undeclared (first use in this function)
1421 | preallocate_pgd_pages_range(MODULES_VADDR, MODULES_END, "bpf/modules");
| ^~~~~~~~~~~~~
arch/riscv/mm/init.c:1421:45: note: each undeclared identifier is reported only once for each function it appears in
arch/riscv/mm/init.c:1421:60: error: 'MODULES_END' undeclared (first use in this function)
1421 | preallocate_pgd_pages_range(MODULES_VADDR, MODULES_END, "bpf/modules");
| ^~~~~~~~~~~cc1: some warnings being treated as errors
vim +1381 arch/riscv/mm/init.c
1366
1367 #ifdef CONFIG_64BIT
1368 /*
1369 * Pre-allocates page-table pages for a specific area in the kernel
1370 * page-table. Only the level which needs to be synchronized between
1371 * all page-tables is allocated because the synchronization can be
1372 * expensive.
1373 */
1374 static void __init preallocate_pgd_pages_range(unsigned long start, unsigned long end,
1375 const char *area)
1376 {
1377 unsigned long addr;
1378 const char *lvl;
1379
1380 for (addr = start; addr < end && addr >= start; addr = ALIGN(addr + 1, PGDIR_SIZE)) {
> 1381 pgd_t *pgd = pgd_offset_k(addr); 1382 p4d_t *p4d; 1383 pud_t *pud; 1384 pmd_t *pmd; 1385 1386 lvl = "p4d"; > 1387 p4d = p4d_alloc(&init_mm, pgd, addr); 1388 if (!p4d) 1389 goto failed; 1390 1391 if (pgtable_l5_enabled) 1392 continue; 1393 1394 lvl = "pud"; > 1395 pud = pud_alloc(&init_mm, p4d, addr); 1396 if (!pud) 1397 goto failed; 1398 1399 if (pgtable_l4_enabled) 1400 continue; 1401 1402 lvl = "pmd"; > 1403 pmd = pmd_alloc(&init_mm, pud, addr);
1404 if (!pmd)
1405 goto failed;
1406 }
1407 return;
1408
1409 failed:
1410 /*
1411 * The pages have to be there now or they will be missing in
1412 * process page-tables later.
1413 */
1414 panic("Failed to pre-allocate %s pages for %s area\n", lvl, area);
1415 }
1416
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botkernel test robot noticed the following build errors:
Hi Björn,
kernel test robot noticed the following build errors:
[auto build test ERROR on ac9a78681b921877518763ba0e89202254349d1b]
url: https://github.com/intel-lab-lkp/linux/commits/Bj-rn-T-pel/riscv-mm-Pre-allocate-PGD-entries-vmalloc-modules-area/20230530-020212 base: ac9a78681b921877518763ba0e89202254349d1b patch link: https://lore.kernel.org/r/20230529180023.289904-1-bjorn%40kernel.org patch subject: [PATCH] riscv: mm: Pre-allocate PGD entries vmalloc/modules area config: riscv-randconfig-r042-20230530 (https://download.01.org/0day-ci/archive/20230531/[email protected]/config) compiler: riscv64-linux-gcc (GCC) 12.3.0
reproduce (this is a W=1 build):
mkdir -p ~/bin
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/9383adbcdfb7f850149742579615e8c174328a78
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Bj-rn-T-pel/riscv-mm-Pre-allocate-PGD-entries-vmalloc-modules-area/20230530-020212 git checkout 9383adbcdfb7f850149742579615e8c174328a78
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.3.0 ~/bin/make.cross W=1 O=build_dir ARCH=riscv olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.3.0 ~/bin/make.cross W=1 O=build_dir ARCH=riscv SHELL=/bin/bash
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
arch/riscv/mm/init.c: In function 'preallocate_pgd_pages_range':
>> arch/riscv/mm/init.c:1381:30: error: implicit declaration of function 'pgd_offset_k'; did you mean 'p4d_offset'? [-Werror=implicit-function-declaration]
1381 | pgd_t *pgd = pgd_offset_k(addr);
| ^~~~~~~~~~~~
| p4d_offset
arch/riscv/mm/init.c:1381:30: warning: initialization of 'pgd_t *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
>> arch/riscv/mm/init.c:1387:23: error: implicit declaration of function 'p4d_alloc'; did you mean 'd_alloc'? [-Werror=implicit-function-declaration]
1387 | p4d = p4d_alloc(&init_mm, pgd, addr);
| ^~~~~~~~~
| d_alloc
arch/riscv/mm/init.c:1387:21: warning: assignment to 'p4d_t *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
1387 | p4d = p4d_alloc(&init_mm, pgd, addr);
| ^
>> arch/riscv/mm/init.c:1395:23: error: implicit declaration of function 'pud_alloc'; did you mean 'd_alloc'? [-Werror=implicit-function-declaration]
1395 | pud = pud_alloc(&init_mm, p4d, addr);
| ^~~~~~~~~
| d_alloc
arch/riscv/mm/init.c:1395:21: warning: assignment to 'pud_t *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
1395 | pud = pud_alloc(&init_mm, p4d, addr);
| ^
>> arch/riscv/mm/init.c:1403:23: error: implicit declaration of function 'pmd_alloc'; did you mean 'mm_alloc'? [-Werror=implicit-function-declaration]
1403 | pmd = pmd_alloc(&init_mm, pud, addr);
| ^~~~~~~~~
| mm_alloc
arch/riscv/mm/init.c:1403:21: warning: assignment to 'pmd_t *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
1403 | pmd = pmd_alloc(&init_mm, pud, addr);
| ^
arch/riscv/mm/init.c: In function 'pgtable_cache_init':
>> arch/riscv/mm/init.c:1421:45: error: 'MODULES_VADDR' undeclared (first use in this function)
1421 | preallocate_pgd_pages_range(MODULES_VADDR, MODULES_END, "bpf/modules");
| ^~~~~~~~~~~~~
arch/riscv/mm/init.c:1421:45: note: each undeclared identifier is reported only once for each function it appears in
>> arch/riscv/mm/init.c:1421:60: error: 'MODULES_END' undeclared (first use in this function)
1421 | preallocate_pgd_pages_range(MODULES_VADDR, MODULES_END, "bpf/modules");
| ^~~~~~~~~~~cc1: some warnings being treated as errors
vim +1381 arch/riscv/mm/init.c
1366
1367 #ifdef CONFIG_64BIT
1368 /*
1369 * Pre-allocates page-table pages for a specific area in the kernel
1370 * page-table. Only the level which needs to be synchronized between
1371 * all page-tables is allocated because the synchronization can be
1372 * expensive.
1373 */
1374 static void __init preallocate_pgd_pages_range(unsigned long start, unsigned long end,
1375 const char *area)
1376 {
1377 unsigned long addr;
1378 const char *lvl;
1379
1380 for (addr = start; addr < end && addr >= start; addr = ALIGN(addr + 1, PGDIR_SIZE)) {
> 1381 pgd_t *pgd = pgd_offset_k(addr);
1382 p4d_t *p4d;
1383 pud_t *pud;
1384 pmd_t *pmd;
1385
1386 lvl = "p4d";
> 1387 p4d = p4d_alloc(&init_mm, pgd, addr);
1388 if (!p4d)
1389 goto failed;
1390
1391 if (pgtable_l5_enabled)
1392 continue;
1393
1394 lvl = "pud";
> 1395 pud = pud_alloc(&init_mm, p4d, addr);
1396 if (!pud)
1397 goto failed;
1398
1399 if (pgtable_l4_enabled)
1400 continue;
1401
1402 lvl = "pmd";
> 1403 pmd = pmd_alloc(&init_mm, pud, addr);
1404 if (!pmd)
1405 goto failed;
1406 }
1407 return;
1408
1409 failed:
1410 /*
1411 * The pages have to be there now or they will be missing in
1412 * process page-tables later.
1413 */
1414 panic("Failed to pre-allocate %s pages for %s area\n", lvl, area);
1415 }
1416
1417 void __init pgtable_cache_init(void)
1418 {
1419 preallocate_pgd_pages_range(VMALLOC_START, VMALLOC_END, "vmalloc");
1420 if (IS_ENABLED(CONFIG_MODULES))
> 1421 preallocate_pgd_pages_range(MODULES_VADDR, MODULES_END, "bpf/modules");Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Quoted 34 lines · On Mon, 29 May 2023 11:00:23 PDT (-0700), [email protected] wrote:
From: Björn Töpel <[email protected]>
The RISC-V port requires that kernel PGD entries are to be synchronized between MMs. This is done via the
vmalloc_fault()function, that simply copies the PGD entries frominit_mmto the faulting one.Historically, faulting in PGD entries have been a source for both bugs [1], and poor performance.
One way to get rid of vmalloc faults is by pre-allocating the PGD entries. Pre-allocating the entries potientially wastes 64 * 4K (65 on SV39). The pre-allocation function is pulled from Jörg Rödel's x86 work, with the addition of 3-level page tables (PMD allocations).
The
pmd_alloc()function needs the ptlock cache to be initialized (when split page locks is enabled), so the pre-allocation is done in a RISC-V specificpgtable_cache_init()implementation.Pre-allocate the kernel PGD entries for the vmalloc/modules area, but only for 64b platforms.
Signed-off-by Björn Töpel <[email protected]>--- arch/riscv/mm/fault.c | 20 +++------------ arch/riscv/mm/init.c | 58 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+), 16 deletions(-)arch/riscv/mm/fault.c +0
-0
arch/riscv/mm/fault.c void handle_page_fault(struct pt_regs *regs) · line 234
Quoted 7 lines
arch/riscv/mm/fault.c +2
-2@@ -230,32 +230,20 @@ void handle_page_fault(struct pt_regs *regs)return ;/* -* Fault-in kernel-space virtual memory on-demand. -* The 'reference' page table is init_mm.pgd. +* Fault-in kernel-space virtual memory on-demand, for 32-bit +* architectures. The 'reference' page table is init_mm.pgd.
That wording seems a little odd to me: I think English allows for these "add something after the comma to change the meaning of a sentence" things, but they're kind of complicated. Maybe it's easier to just flip the order?
That said, it's very early so maybe it's fine...
Quoted 96 lines
arch/riscv/mm/fault.c +2
-14* * NOTE! We MUST NOT take any locks for this case. We may * be in an interrupt or a critical region, and should * only copy the information from the master page table, * nothing more. */ -if (unlikely ((addr >=VMALLOC_START ) && (addr <VMALLOC_END ))) { +if (!IS_ENABLED (CONFIG_64BIT ) && +unlikely (addr >=VMALLOC_START && addr <VMALLOC_END )) {vmalloc_fault (regs, code, addr);return ; } -#ifdef CONFIG_64BIT -/* -* Modules in 64bit kernels lie in their own virtual region which is not -* in the vmalloc region, but dealing with page faults in this region -* or the vmalloc region amounts to doing the same thing: checking that -* the mapping exists in init_mm.pgd and updating user page table, so -* just use vmalloc_fault. -*/ -if (unlikely (addr >=MODULES_VADDR && addr <MODULES_END )) { -vmalloc_fault (regs, code, addr); -return ; - } -#endif /* Enable interrupts if they were enabled in the parent context. */ if (!regs_irqs_disabled (regs))local_irq_enable ();arch/riscv/mm/init.c +58
-0@@ -1363,3 +1363,61 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,return vmemmap_populate_basepages (start, end, node,NULL ); }#endif + +#ifdef CONFIG_64BIT +/* +* Pre-allocates page-table pages for a specific area in the kernel +* page-table. Only the level which needs to be synchronized between +* all page-tables is allocated because the synchronization can be +* expensive. +*/ +static void __init preallocate_pgd_pages_range (unsigned long start,unsigned long end, +const char *area) +{ +unsigned long addr; +const char *lvl; + +for (addr = start; addr < end && addr >= start; addr =ALIGN (addr +1 ,PGDIR_SIZE )) { +pgd_t *pgd =pgd_offset_k (addr); +p4d_t *p4d; +pud_t *pud; +pmd_t *pmd; + + lvl ="p4d" ; + p4d =p4d_alloc (&init_mm, pgd, addr); +if (!p4d) + goto failed; + +if (pgtable_l5_enabled) +continue ; + + lvl ="pud" ; + pud =pud_alloc (&init_mm, p4d, addr); +if (!pud) + goto failed; + +if (pgtable_l4_enabled) +continue ; + + lvl ="pmd" ; + pmd =pmd_alloc (&init_mm, pud, addr); +if (!pmd) + goto failed; + } +return ; + +failed: +/* +* The pages have to be there now or they will be missing in +* process page-tables later. +*/ +panic ("Failed to pre-allocate %s pages for %s area\n" , lvl, area); +} + +void __init pgtable_cache_init (void ) +{ +preallocate_pgd_pages_range (VMALLOC_START ,VMALLOC_END ,"vmalloc" ); +if (IS_ENABLED (CONFIG_MODULES )) +preallocate_pgd_pages_range (MODULES_VADDR ,MODULES_END ,"bpf/modules" ); +} +#endif
Quoted 1 line
base-commit:
ac9a78681b921877518763ba0e89202254349d1b
aside from the build issue, which seems pretty straight-forward. I'm going to drop this from patchwork.
Quoted 146 lines · Palmer Dabbelt
Quoted 34 lines · On Mon, 29 May 2023 11:00:23 PDT (-0700), [email protected] wrote:
From: Björn Töpel <[email protected]>
The RISC-V port requires that kernel PGD entries are to be synchronized between MMs. This is done via the
vmalloc_fault()function, that simply copies the PGD entries frominit_mmto the faulting one.Historically, faulting in PGD entries have been a source for both bugs [1], and poor performance.
One way to get rid of vmalloc faults is by pre-allocating the PGD entries. Pre-allocating the entries potientially wastes 64 * 4K (65 on SV39). The pre-allocation function is pulled from Jörg Rödel's x86 work, with the addition of 3-level page tables (PMD allocations).
The
pmd_alloc()function needs the ptlock cache to be initialized (when split page locks is enabled), so the pre-allocation is done in a RISC-V specificpgtable_cache_init()implementation.Pre-allocate the kernel PGD entries for the vmalloc/modules area, but only for 64b platforms.
Signed-off-by Björn Töpel <[email protected]>--- arch/riscv/mm/fault.c | 20 +++------------ arch/riscv/mm/init.c | 58 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+), 16 deletions(-)arch/riscv/mm/fault.c +0
-0arch/riscv/mm/fault.c void handle_page_fault(struct pt_regs *regs) · line 234
Quoted 7 lines
arch/riscv/mm/fault.c +2
-2@@ -230,32 +230,20 @@ void handle_page_fault(struct pt_regs *regs)return ;/* -* Fault-in kernel-space virtual memory on-demand. -* The 'reference' page table is init_mm.pgd. +* Fault-in kernel-space virtual memory on-demand, for 32-bit +* architectures. The 'reference' page table is init_mm.pgd. That wording seems a little odd to me: I think English allows for these "add something after the comma to change the meaning of a sentence" things, but they're kind of complicated. Maybe it's easier to just flip the order?
That said, it's very early so maybe it's fine...
Quoted 96 lines
arch/riscv/mm/fault.c +2
-14* * NOTE! We MUST NOT take any locks for this case. We may * be in an interrupt or a critical region, and should * only copy the information from the master page table, * nothing more. */ -if (unlikely ((addr >=VMALLOC_START ) && (addr <VMALLOC_END ))) { +if (!IS_ENABLED (CONFIG_64BIT ) && +unlikely (addr >=VMALLOC_START && addr <VMALLOC_END )) {vmalloc_fault (regs, code, addr);return ; } -#ifdef CONFIG_64BIT -/* -* Modules in 64bit kernels lie in their own virtual region which is not -* in the vmalloc region, but dealing with page faults in this region -* or the vmalloc region amounts to doing the same thing: checking that -* the mapping exists in init_mm.pgd and updating user page table, so -* just use vmalloc_fault. -*/ -if (unlikely (addr >=MODULES_VADDR && addr <MODULES_END )) { -vmalloc_fault (regs, code, addr); -return ; - } -#endif /* Enable interrupts if they were enabled in the parent context. */ if (!regs_irqs_disabled (regs))local_irq_enable ();arch/riscv/mm/init.c +58
-0@@ -1363,3 +1363,61 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,return vmemmap_populate_basepages (start, end, node,NULL ); }#endif + +#ifdef CONFIG_64BIT +/* +* Pre-allocates page-table pages for a specific area in the kernel +* page-table. Only the level which needs to be synchronized between +* all page-tables is allocated because the synchronization can be +* expensive. +*/ +static void __init preallocate_pgd_pages_range (unsigned long start,unsigned long end, +const char *area) +{ +unsigned long addr; +const char *lvl; + +for (addr = start; addr < end && addr >= start; addr =ALIGN (addr +1 ,PGDIR_SIZE )) { +pgd_t *pgd =pgd_offset_k (addr); +p4d_t *p4d; +pud_t *pud; +pmd_t *pmd; + + lvl ="p4d" ; + p4d =p4d_alloc (&init_mm, pgd, addr); +if (!p4d) + goto failed; + +if (pgtable_l5_enabled) +continue ; + + lvl ="pud" ; + pud =pud_alloc (&init_mm, p4d, addr); +if (!pud) + goto failed; + +if (pgtable_l4_enabled) +continue ; + + lvl ="pmd" ; + pmd =pmd_alloc (&init_mm, pud, addr); +if (!pmd) + goto failed; + } +return ; + +failed: +/* +* The pages have to be there now or they will be missing in +* process page-tables later. +*/ +panic ("Failed to pre-allocate %s pages for %s area\n" , lvl, area); +} + +void __init pgtable_cache_init (void ) +{ +preallocate_pgd_pages_range (VMALLOC_START ,VMALLOC_END ,"vmalloc" ); +if (IS_ENABLED (CONFIG_MODULES )) +preallocate_pgd_pages_range (MODULES_VADDR ,MODULES_END ,"bpf/modules" ); +} +#endif
Quoted 6 lines
Quoted 1 line
base-commit:
ac9a78681b921877518763ba0e89202254349d1bReviewed-by Palmer Dabbelt <[email protected]>aside from the build issue, which seems pretty straight-forward. I'm going to drop this from patchwork.
Hmm, you applied the V2 a couple of days ago [1], which fixes the build issue. Did you drop the V2 from the queue?
Björn