x86: prefetch_page() vDSO call
Subsystems (10)
cover x86: prefetch_page() vDSO call
From: Nadav Amit <[email protected]>
Just as applications can use prefetch instructions to overlap computations and memory accesses, applications may want to overlap the page-faults and compute or overlap the I/O accesses that are required for page-faults of different pages.
Applications can use multiple threads and cores for this matter, by running one thread that prefetches the data (i.e., faults in the data) and another that does the compute, but this scheme is inefficient. Using mincore() can tell whether a page is mapped, but might not tell whether the page is in the page-cache and does not fault in the data.
Introduce prefetch_page() vDSO-call to prefetch, i.e. fault-in memory asynchronously. The semantic of this call is: try to prefetch a page of in a given address and return zero if the page is accessible following the call. Start I/O operations to retrieve the page if such operations are required and there is no high memory pressure that might introduce slowdowns.
Note that as usual the page might be paged-out at any point and therefore, similarly to mincore(), there is no guarantee that the page will be present at the time that the user application uses the data that resides on the page. Nevertheless, it is expected that in the vast majority of the cases this would not happen, since prefetch_page() accesses the page and therefore sets the PTE access-bit (if it is clear).
The implementation is as follows. The vDSO code accesses the data, triggering a page-fault it is not present. The handler detects based on the instruction pointer that this is an asynchronous-#PF, using the recently introduce vDSO exception tables. If the page can be brought without waiting (e.g., the page is already in the page-cache), the kernel handles the fault and returns success (zero). If there is memory pressure that prevents the proper handling of the fault (i.e., requires heavy-weight reclamation) it returns a failure. Otherwise, it starts an I/O to bring the page and returns failure.
Compilers can be extended to issue the prefetch_page() calls when needed.
Nadav Amit (6):
vdso/extable: fix calculation of base x86/vdso: add mask and flags to extable x86/vdso: introduce page_prefetch() mm/swap_state: respect FAULT_FLAG_RETRY_NOWAIT
mm: use lightweight reclaim on FAULT_FLAG_RETRY_NOWAIT testing/selftest: test vDSO prefetch_page()
arch/x86/Kconfig | 1 + arch/x86/entry/vdso/Makefile | 1 + arch/x86/entry/vdso/extable.c | 70 +++-- arch/x86/entry/vdso/extable.h | 21 +- arch/x86/entry/vdso/vdso.lds.S | 1 + arch/x86/entry/vdso/vprefetch.S | 39 +++ arch/x86/entry/vdso/vsgx.S | 9 +- arch/x86/include/asm/vdso.h | 38 ++- arch/x86/mm/fault.c | 11 +- lib/vdso/Kconfig | 5 + mm/memory.c | 47 +++- mm/shmem.c | 1 + mm/swap_state.c | 12 +- tools/testing/selftests/vDSO/Makefile | 2 + .../selftests/vDSO/vdso_test_prefetch_page.c | 265 ++++++++++++++++++ 15 files changed, 470 insertions(+), 53 deletions(-) create mode 100644 arch/x86/entry/vdso/vprefetch.S create mode 100644 tools/testing/selftests/vDSO/vdso_test_prefetch_page.c
Signature · 1 line
2.25.1
RFC 1/6 vdso/extable: fix calculation of base
From: Nadav Amit <[email protected]>
Apparently, the assembly considers __ex_table as the location when the pushsection directive was issued. Therefore when there is more than a single entry in the vDSO exception table, the calculations of the base and fixup are wrong.
Fix the calculations of the expected fault IP and new IP by adjusting the base after each entry.
--- arch/x86/entry/vdso/extable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/entry/vdso/extable.c +1 -1
@@ -32,7 +32,7 @@ bool fixup_vdso_exception(struct pt_regs *regs, int trapnr, nr_entries = image->extable_len / (sizeof (*extable)); extable = image->extable; -for (i =0 ; i < nr_entries; i++) { +for (i =0 ; i < nr_entries; i++, base +=sizeof (*extable)) {if (regs->ip == base + extable[i].insn) { regs->ip = base + extable[i].fixup; regs->di = trapnr;
Signature · 1 line
2.25.1
Re: [RFC 1/6] vdso/extable: fix calculation of base
Quoted 28 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Apparently, the assembly considers
__ex_tableas the location when the pushsection directive was issued. Therefore when there is more than a single entry in the vDSO exception table, the calculations of the base and fixup are wrong.Fix the calculations of the expected fault IP and new IP by adjusting the base after each entry.
Cc Andy Lutomirski <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Sean Christopherson <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Borislav Petkov <[email protected]>Cc Andrew Morton <[email protected]>Signed-off-by Nadav Amit <[email protected]>--- arch/x86/entry/vdso/extable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/entry/vdso/extable.c +0
-0
arch/x86/entry/vdso/extable.c bool fixup_vdso_exception(struct pt_regs *regs, int trapnr, · line 35
Quoted 5 lines
arch/x86/entry/vdso/extable.c +1
-1@@ -32,7 +32,7 @@ bool fixup_vdso_exception(struct pt_regs *regs, int trapnr, nr_entries = image->extable_len / (sizeof (*extable)); extable = image->extable; -for (i =0 ; i < nr_entries; i++) { +for (i =0 ; i < nr_entries; i++, base +=sizeof (*extable)) {
It's been literally years since I wrote this code, but I distinctly remember the addresses being relative to the base. I also remember testing multiple entries, but again, that was a long time ago.
Assuming things have changed, or I was flat out wrong, the comment above the macro magic should also be updated.
/*
* Inject exception fixup for vDSO code. Unlike normal exception fixup,
* vDSO uses a dedicated handler the addresses are relative to the overall
* exception table, not each individual entry.
*/
Quoted 6 lines
arch/x86/entry/vdso/extable.c +0
-0if (regs->ip == base + extable[i].insn) { regs->ip = base + extable[i].fixup; regs->di = trapnr;Signature · 1 line
2.25.1
Re: [RFC 1/6] vdso/extable: fix calculation of base
Quoted 43 lines · Sean Christopherson
Quoted 28 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Apparently, the assembly considers
__ex_tableas the location when the pushsection directive was issued. Therefore when there is more than a single entry in the vDSO exception table, the calculations of the base and fixup are wrong.Fix the calculations of the expected fault IP and new IP by adjusting the base after each entry.
Cc Andy Lutomirski <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Sean Christopherson <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Borislav Petkov <[email protected]>Cc Andrew Morton <[email protected]>Signed-off-by Nadav Amit <[email protected]>---
arch/x86/entry/vdso/extable.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/entry/vdso/extable.c +0
-0arch/x86/entry/vdso/extable.c bool fixup_vdso_exception(struct pt_regs *regs, int trapnr, · line 35
Quoted 5 lines
arch/x86/entry/vdso/extable.c +1
-1@@ -32,7 +32,7 @@ bool fixup_vdso_exception(struct pt_regs *regs, int trapnr, nr_entries = image->extable_len / (sizeof (*extable)); extable = image->extable; -for (i =0 ; i < nr_entries; i++) { +for (i =0 ; i < nr_entries; i++, base +=sizeof (*extable)) {It's been literally years since I wrote this code, but I distinctly remember the addresses being relative to the base. I also remember testing multiple entries, but again, that was a long time ago.
Quoted 8 lines
Assuming things have changed, or I was flat out wrong, the comment above the macro magic should also be updated.
/*
* Inject exception fixup for vDSO code. Unlike normal exception fixup,
* vDSO uses a dedicated handler the addresses are relative to the overall
* exception table, not each individual entry.
*/
I will update the comment. I am not very familiar with pushsection stuff, but the offsets were wrong.
Since you say you checked it, I wonder whether it can somehow be caused by having exception table entries defined from multiple object files.
Anyhow, this change follows the kernel’s (not vDSO) exception table scheme.
Re: [RFC 1/6] vdso/extable: fix calculation of base
Quoted 15 lines · Nadav Amit
Quoted 6 lines · Sean Christopherson
It's been literally years since I wrote this code, but I distinctly remember the addresses being relative to the base. I also remember testing multiple entries, but again, that was a long time ago.
Quoted 8 lines
Assuming things have changed, or I was flat out wrong, the comment above the macro magic should also be updated.
/*
* Inject exception fixup for vDSO code. Unlike normal exception fixup,
* vDSO uses a dedicated handler the addresses are relative to the overall
* exception table, not each individual entry.
*/
Quoted 5 lines
I will update the comment. I am not very familiar with pushsection stuff, but the offsets were wrong.
Since you say you checked it, I wonder whether it can somehow be caused by having exception table entries defined from multiple object files.
Oooh, I think that would do it. Have you checked what happens if there are multiple object files and multiple fixups within an object file?
Quoted 2 lines
Anyhow, this change follows the kernel’s (not vDSO) exception table scheme.
Re: [RFC 1/6] vdso/extable: fix calculation of base
Quoted 20 lines · Sean Christopherson
Quoted 15 lines · Nadav Amit
Quoted 14 lines · Sean Christopherson
It's been literally years since I wrote this code, but I distinctly remember the addresses being relative to the base. I also remember testing multiple entries, but again, that was a long time ago.
Assuming things have changed, or I was flat out wrong, the comment above the macro magic should also be updated.
/*
* Inject exception fixup for vDSO code. Unlike normal exception fixup,
* vDSO uses a dedicated handler the addresses are relative to the overall
* exception table, not each individual entry.
*/
Quoted 8 lines
Quoted 5 lines
I will update the comment. I am not very familiar with pushsection stuff, but the offsets were wrong.
Since you say you checked it, I wonder whether it can somehow be caused by having exception table entries defined from multiple object files.
Oooh, I think that would do it. Have you checked what happens if there are multiple object files and multiple fixups within an object file?
Good thing that you insisted...
I certainly do not know well enough the assembly section directives, but indeed it seems (after some experiments) that referring to the section provides different values from different objects.
So both the current (yours) and this patch (mine) are broken. I think the easiest thing is to fall back to the kernel exception table scheme. I checked the following with both entries in the same and different objects and it seems to work correctly:
-- >8 --
arch/x86/entry/vdso/extable.c +5 -3
@@ -32,9 +32,11 @@ bool fixup_vdso_exception(struct pt_regs *regs, int trapnr, nr_entries = image->extable_len / (sizeof (*extable)); extable = image->extable; -for (i =0 ; i < nr_entries; i++) { -if (regs->ip == base + extable[i].insn) { - regs->ip = base + extable[i].fixup; +for (i =0 ; i < nr_entries; i++, base +=sizeof (*extable)) { +if (regs->ip == base + extable[i].insn + + offsetof(struct vdso_exception_table_entry , insn)) { + regs->ip = base + extable[i].fixup + + offsetof(struct vdso_exception_table_entry , fixup); regs->di = trapnr; regs->si = error_code; regs->dx = fault_addr;
arch/x86/entry/vdso/extable.h +2 -2
@@ -13,8 +13,8 @@ .macro ASM_VDSO_EXTABLE_HANDLE from:req to:req .pushsection __ex_table, "a" - .long (\from) - __ex_table - .long (\to) - __ex_table + .long (\from) - . + .long (\to) - . .popsection .endm #else
--
2.25.1
RFC 2/6 x86/vdso: add mask and flags to extable
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Add a "mask" field to vDSO exception tables that says which exceptions should be handled.
Add a "flags" field to vDSO as well to provide additional information about the exception.
The existing preprocessor macro _ASM_VDSO_EXTABLE_HANDLE for assembly is not easy to use as it requires the user to stringify the expanded C macro. Remove _ASM_VDSO_EXTABLE_HANDLE and use a similar scheme to ALTERNATIVE, using assembly macros directly in assembly without wrapping them in C macros.
Move the vsgx supported exceptions test out of the generic C code into vsgx-specific assembly by setting vsgx supported exceptions in the mask.
--- arch/x86/entry/vdso/extable.c | 9 +-------- arch/x86/entry/vdso/extable.h | 21 +++++++++++++-------- arch/x86/entry/vdso/vsgx.S | 9 +++++++-- 3 files changed, 21 insertions(+), 18 deletions(-)
arch/x86/entry/vdso/extable.c +1 -8
@@ -7,6 +7,7 @@struct vdso_exception_table_entry {int insn, fixup; +unsigned int mask, flags; }; bool fixup_vdso_exception(struct pt_regs *regs, int trapnr,
@@ -17,14 +18,6 @@ bool fixup_vdso_exception(struct pt_regs *regs, int trapnr,unsigned int nr_entries, i;unsigned long base; -/* -* Do not attempt to fixup #DB or #BP. It's impossible to identify -* whether or not a #DB/#BP originated from within an SGX enclave and -* SGX enclaves are currently the only use case for vDSO fixup. -*/ -if (trapnr ==X86_TRAP_DB || trapnr ==X86_TRAP_BP ) -return false; -if (!current->mm->context.vdso)return false;
arch/x86/entry/vdso/extable.h +13 -8
@@ -2,26 +2,31 @@#ifndef __VDSO_EXTABLE_H#define __VDSO_EXTABLE_H +#include <asm/trapnr.h> + +#define ASM_VDSO_ASYNC_FLAGS (1 << 0) +/* * Inject exception fixup for vDSO code. Unlike normal exception fixup, * vDSO uses a dedicated handler the addresses are relative to the overall * exception table, not each individual entry. */ #ifdef __ASSEMBLY__ -#define _ASM_VDSO_EXTABLE_HANDLE (from, to) \ - ASM_VDSO_EXTABLE_HANDLE from to - -.macro ASM_VDSO_EXTABLE_HANDLE from:req to:req +.macro ASM_VDSO_EXTABLE_HANDLE from:req to:req mask:req flags:req .pushsection __ex_table, "a" .long (\from) - __ex_table .long (\to) - __ex_table + .long (\mask) + .long (\flags) .popsection .endm#else -#define _ASM_VDSO_EXTABLE_HANDLE (from, to) \ - ".pushsection __ex_table, \"a\"\n" \ - ".long (" #from ") - __ex_table\n" \ - ".long (" #to ") - __ex_table\n" \ +#define ASM_VDSO_EXTABLE_HANDLE (from, to, mask, flags) \ + ".pushsection __ex_table, \"a\"\n" \ + ".long (" #from ") - __ex_table\n" \ + ".long (" #to ") - __ex_table\n" \ + ".long (" #mask ")\n" \ + ".long (" #flags ")\n" \ ".popsection\n"#endif
arch/x86/entry/vdso/vsgx.S +7 -2
@@ -4,6 +4,7 @@ #include <asm/export.h> #include <asm/errno.h> #include <asm/enclu.h> +#include <asm/trapnr.h> #include "extable.h"
@@ -146,6 +147,10 @@ SYM_FUNC_START(__vdso_sgx_enter_enclave) .cfi_endproc -_ASM_VDSO_EXTABLE_HANDLE(.Lenclu_eenter_eresume, .Lhandle_exception) - +/* + * Do not attempt to fixup #DB or #BP. It's impossible to identify + * whether or not a #DB/#BP originated from within an SGX enclave. + */ +ASM_VDSO_EXTABLE_HANDLE .Lenclu_eenter_eresume, .Lhandle_exception, \ + ~((1<<X86_TRAP_DB)+(1<<X86_TRAP_BP)), 0 SYM_FUNC_END(__vdso_sgx_enter_enclave)
Signature · 1 line
2.25.1
RFC 3/6 x86/vdso: introduce page_prefetch()
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Introduce a new vDSO function: page_prefetch() which is to be used when certain memory, which might be paged out, is expected to be used soon. The function prefetches the page if needed. The function returns zero if the page is accessible after the call and -1 otherwise.
page_prefetch() is intended to be very lightweight both when the page is already present and when the page is prefetched.
The implementation leverages the new vDSO exception tables mechanism. page_prefetch() accesses the page for read and has a corresponding vDSO exception-table entry that indicates that a #PF might occur and that in such case the page should be brought asynchronously. If #PF indeed occurs, the page-fault handler sets the FAULT_FLAG_RETRY_NOWAIT flag.
If the page-fault was not resolved, the page-fault handler does not retry, and instead jumps to the new IP that is marked in the exception table. The vDSO part returns accordingly the return value.
--- arch/x86/Kconfig | 1 + arch/x86/entry/vdso/Makefile | 1 + arch/x86/entry/vdso/extable.c | 59 +++++++++++++++++++++++++-------- arch/x86/entry/vdso/vdso.lds.S | 1 + arch/x86/entry/vdso/vprefetch.S | 39 ++++++++++++++++++++++ arch/x86/include/asm/vdso.h | 38 +++++++++++++++++++-- arch/x86/mm/fault.c | 11 ++++-- lib/vdso/Kconfig | 5 +++ 8 files changed, 136 insertions(+), 19 deletions(-) create mode 100644 arch/x86/entry/vdso/vprefetch.S
arch/x86/Kconfig +1 -0
@@ -136,6 +136,7 @@ config X86 select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY select GENERIC_VDSO_TIME_NS + select GENERIC_VDSO_PREFETCH select GUP_GET_PTE_LOW_HIGH if X86_PAE select HARDIRQS_SW_RESEND select HARDLOCKUP_CHECK_TIMESTAMP if X86_64
arch/x86/entry/vdso/Makefile +1 -0
@@ -28,6 +28,7 @@ vobjs-y := vdso-note.o vclock_gettime.o vgetcpu.o vobjs32-y := vdso32/note.o vdso32/system_call.o vdso32/sigreturn.o vobjs32-y += vdso32/vclock_gettime.o vobjs-$(CONFIG_X86_SGX) += vsgx.o +vobjs-$(CONFIG_GENERIC_VDSO_PREFETCH) += vprefetch.o # files to link into kernel obj-y += vma.o extable.o
arch/x86/entry/vdso/extable.c +45 -14
81 lines not on this page · raw message
arch/x86/entry/vdso/vdso.lds.S +1 -0
@@ -28,6 +28,7 @@ VERSION { clock_getres; __vdso_clock_getres; __vdso_sgx_enter_enclave; + __vdso_prefetch_page; local: *; }; }
arch/x86/entry/vdso/vprefetch.S new file +39 -0
@@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include <linux/linkage.h> +#include <asm/export.h> +#include <asm/errno.h> +#include <asm/enclu.h> + +#include "extable.h" + +.code64 +.section .text, "ax" + +SYM_FUNC_START(__vdso_prefetch_page) + /* Prolog */ + .cfi_startproc + push %rbp + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %rbp, 0 + mov %rsp, %rbp + .cfi_def_cfa_register %rbp + + xor %rax, %rax +.Laccess_page: + movb (%rdi), %dil +.Lout: + + /* Epilog */ + pop %rbp + .cfi_def_cfa %rsp, 8 + ret + +.Lhandle_exception: + mov $-1ll, %rax + jmp .Lout + .cfi_endproc +ASM_VDSO_EXTABLE_HANDLE .Laccess_page, .Lhandle_exception, \ + (1<<X86_TRAP_PF), ASM_VDSO_ASYNC_FLAGS + +SYM_FUNC_END(__vdso_prefetch_page)
arch/x86/include/asm/vdso.h +35 -3
@@ -9,6 +9,7 @@ #ifndef __ASSEMBLER__ #include <linux/mm_types.h> +#include <linux/sched.h> struct vdso_image { void *data;
@@ -49,9 +50,40 @@ extern void __init init_vdso_image(const struct vdso_image *image);extern int map_vdso_once (const struct vdso_image *image,unsigned long addr); -extern bool fixup_vdso_exception (struct pt_regs *regs,int trapnr, -unsigned long error_code, -unsigned long fault_addr); +extern bool __fixup_vdso_exception (struct pt_regs *regs,int trapnr, +unsigned long error_code, +unsigned long fault_addr); + +extern __attribute_const__ bool __is_async_vdso_exception(struct pt_regs *regs, +int trapnr); + +static inline bool is_exception_in_vdso (struct pt_regs *regs) +{ +const struct vdso_image *image = current->mm->context.vdso_image; +unsigned long vdso_base = (unsigned long )current->mm->context.vdso; + +return regs->ip >= vdso_base && regs->ip < vdso_base + image->size && + vdso_base !=0 ; +} + +static inline bool is_async_vdso_exception (struct pt_regs *regs,int trapnr) +{ +if (!is_exception_in_vdso (regs)) +return false; + +return __is_async_vdso_exception (regs, trapnr); +} + +static inline bool fixup_vdso_exception (struct pt_regs *regs,int trapnr, +unsigned long error_code, +unsigned long fault_addr) +{ +if (is_exception_in_vdso (regs)) +return __fixup_vdso_exception (regs, trapnr, error_code, + fault_addr); +return false; +} +#endif /* __ASSEMBLER__ */ #endif /* _ASM_X86_VDSO_H */
arch/x86/mm/fault.c +9 -2
@@ -1289,6 +1289,10 @@ void do_user_addr_fault(struct pt_regs *regs,if (user_mode (regs)) {local_irq_enable (); flags |=FAULT_FLAG_USER ; +if (IS_ENABLED (CONFIG_GENERIC_VDSO_PREFETCH ) && +is_async_vdso_exception (regs,X86_TRAP_PF )) + flags |=FAULT_FLAG_ALLOW_RETRY | +FAULT_FLAG_RETRY_NOWAIT ; }else {if (regs->flags &X86_EFLAGS_IF )local_irq_enable ();
@@ -1407,8 +1411,11 @@ void do_user_addr_fault(struct pt_regs *regs,*/ if (unlikely ((fault &VM_FAULT_RETRY ) && (flags &FAULT_FLAG_ALLOW_RETRY ))) { - flags |=FAULT_FLAG_TRIED ; - goto retry; +if (!(flags &FAULT_FLAG_RETRY_NOWAIT )) { + flags |=FAULT_FLAG_TRIED ; + goto retry; + } +fixup_vdso_exception (regs,X86_TRAP_PF , hw_error_code, address); }mmap_read_unlock (mm);
lib/vdso/Kconfig +5 -0
@@ -30,4 +30,9 @@ config GENERIC_VDSO_TIME_NS Selected by architectures which support time namespaces in the VDSO +config GENERIC_VDSO_PREFETCH + bool + help + Selected by architectures which support page prefetch VDSO + endif
Signature · 1 line
2.25.1
RFC 4/6 mm/swap_state: respect FAULT_FLAG_RETRY_NOWAIT
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Certain use-cases (e.g., prefetch_page()) may want to avoid polling while a page is brought from the swap. Yet, swap_cluster_readahead() and swap_vma_readahead() do not respect FAULT_FLAG_RETRY_NOWAIT.
Add support to respect FAULT_FLAG_RETRY_NOWAIT by not polling in these cases.
--- mm/memory.c | 15 +++++++++++++-- mm/shmem.c | 1 + mm/swap_state.c | 12 +++++++++--- 3 files changed, 23 insertions(+), 5 deletions(-)
mm/memory.c +13 -2
@@ -3326,12 +3326,23 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) }if (!page) { +/* +* Back out if we failed to bring the page while we +* tried to avoid I/O. +*/ +if (fault_flag_allow_retry_first (vmf->flags) && + (vmf->flags &FAULT_FLAG_RETRY_NOWAIT )) { + ret =VM_FAULT_RETRY ; +delayacct_clear_flag (DELAYACCT_PF_SWAPIN ); + goto out; + } +/* * Back out if somebody else faulted in this pte * while we released the pte lock. */ - vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + vmf->pte =pte_offset_map_lock (vma->vm_mm, + vmf->pmd, vmf->address, &vmf->ptl);if (likely (pte_same (*vmf->pte, vmf->orig_pte))) ret =VM_FAULT_OOM ;delayacct_clear_flag (DELAYACCT_PF_SWAPIN );
mm/shmem.c +1 -0
@@ -1525,6 +1525,7 @@ static struct page *shmem_swapin(swp_entry_t swap, gfp_t gfp,shmem_pseudo_vma_init (&pvma, info, index); vmf.vma = &pvma; vmf.address =0 ; + vmf.flags =0 ; page =swap_cluster_readahead (swap, gfp, &vmf);shmem_pseudo_vma_destroy (&pvma);
mm/swap_state.c +9 -3
@@ -656,10 +656,13 @@ struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,unsigned long mask;struct swap_info_struct *si =swp_swap_info (entry);struct blk_plug plug; -bool do_poll = true, page_allocated; +bool page_allocated, do_poll;struct vm_area_struct *vma = vmf->vma;unsigned long addr = vmf->address; + do_poll = !fault_flag_allow_retry_first (vmf->flags) || + !(vmf->flags &FAULT_FLAG_RETRY_NOWAIT ); + mask =swapin_nr_pages (offset) -1 ;if (!mask) goto skip;
@@ -838,7 +841,7 @@ static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask,pte_t *pte, pentry;swp_entry_t entry;unsigned int i; -bool page_allocated; +bool page_allocated, do_poll;struct vma_swap_readahead ra_info = { .win =1 , };
@@ -873,9 +876,12 @@ static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask, }blk_finish_plug (&plug);lru_add_drain (); + skip: + do_poll = (!fault_flag_allow_retry_first (vmf->flags) || + !(vmf->flags &FAULT_FLAG_RETRY_NOWAIT )) && ra_info.win ==1 ;return read_swap_cache_async (fentry, gfp_mask, vma, vmf->address, - ra_info.win ==1 ); + do_poll); }/**
Signature · 1 line
2.25.1
RFC 5/6 mm: use lightweight reclaim on FAULT_FLAG_RETRY_NOWAIT
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
When FAULT_FLAG_RETRY_NOWAIT is set, the caller arguably wants only a lightweight reclaim to avoid a long reclamation, which would not respect the "NOWAIT" semantic. Regard the request in swap and file-backed page-faults accordingly during the first try.
--- mm/memory.c | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-)
mm/memory.c +22 -10
@@ -2679,18 +2679,31 @@ static inline bool cow_user_page(struct page *dst, struct page *src,return ret; } -static gfp_t __get_fault_gfp_mask (struct vm_area_struct *vma) +static gfp_t massage_page_gfp_mask (gfp_t gfp_mask,unsigned long vmf_flags) { -struct file *vm_file = vma->vm_file; +if (fault_flag_allow_retry_first (vmf_flags) && + (vmf_flags &FAULT_FLAG_RETRY_NOWAIT )) + gfp_mask |= __GFP_NORETRY | __GFP_NOWARN; -if (vm_file) -return mapping_gfp_mask (vm_file->f_mapping) | __GFP_FS | __GFP_IO; +return gfp_mask; +} + +static gfp_t __get_fault_gfp_mask (struct vm_area_struct *vma, +unsigned long flags) +{ +struct file *vm_file = vma->vm_file; +gfp_t gfp_mask;/* * Special mappings (e.g. VDSO) do not have any file so fake * a default GFP_KERNEL for them. */ -return GFP_KERNEL ; +if (!vm_file) +return GFP_KERNEL ; + + gfp_mask =mapping_gfp_mask (vm_file->f_mapping) | __GFP_FS | __GFP_IO; + +return massage_page_gfp_mask (gfp_mask, flags); }/*
@@ -3253,6 +3266,7 @@ EXPORT_SYMBOL(unmap_mapping_range);*/ vm_fault_t do_swap_page (struct vm_fault *vmf) { +gfp_t gfp_mask =massage_page_gfp_mask (GFP_HIGHUSER_MOVABLE , vmf->flags);struct vm_area_struct *vma = vmf->vma;struct page *page =NULL , *swapcache;swp_entry_t entry;
@@ -3293,8 +3307,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)if (data_race (si->flags &SWP_SYNCHRONOUS_IO ) &&__swap_count (entry) ==1 ) {/* skip swapcache */ - page =alloc_page_vma (GFP_HIGHUSER_MOVABLE , vma, - vmf->address); + page =alloc_page_vma (gfp_mask, vma, vmf->address);if (page) {int err;
@@ -3320,8 +3333,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)swap_readpage (page, true); } }else { - page =swapin_readahead (entry,GFP_HIGHUSER_MOVABLE , - vmf); + page =swapin_readahead (entry, gfp_mask, vmf); swapcache = page; }
@@ -4452,7 +4464,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, .address = address & PAGE_MASK, .flags = flags, .pgoff = linear_page_index(vma, address), - .gfp_mask = __get_fault_gfp_mask(vma), + .gfp_mask = __get_fault_gfp_mask(vma, flags), };unsigned int dirty = flags &FAULT_FLAG_WRITE ;struct mm_struct *mm = vma->vm_mm;
Signature · 1 line
2.25.1
PATCH 6/6 testing/selftest: test vDSO prefetch_page()
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Test prefetch_page() in cases of invalid pointer, file-mmap and anonymous memory. Partial checks are also done with mincore syscall to ensure the output of prefetch_page() is consistent with mincore (taking into account the different semantics of the two).
The tests are not fool-proof as they rely on the behavior of the page-cache and page reclamation mechanism to get a major page-fault. They should be robust in the sense of test being skipped if it failed.
There is a question though on how to know how much memory to access in the test of anonymous memory to force the eviction of a page and trigger a refault.
--- tools/testing/selftests/vDSO/Makefile | 2 + .../selftests/vDSO/vdso_test_prefetch_page.c | 265 ++++++++++++++++++ 2 files changed, 267 insertions(+) create mode 100644 tools/testing/selftests/vDSO/vdso_test_prefetch_page.c
tools/testing/selftests/vDSO/Makefile +2 -0
@@ -11,6 +11,7 @@ ifeq ($(ARCH),$(filter $(ARCH),x86 x86_64)) TEST_GEN_PROGS += $(OUTPUT)/vdso_standalone_test_x86 endif TEST_GEN_PROGS += $(OUTPUT)/vdso_test_correctness +TEST_GEN_PROGS += $(OUTPUT)/vdso_test_prefetch_page CFLAGS := -std=gnu99 CFLAGS_vdso_standalone_test_x86 := -nostdlib -fno-asynchronous-unwind-tables -fno-stack-protector
@@ -33,3 +34,4 @@ $(OUTPUT)/vdso_test_correctness: vdso_test_correctness.c vdso_test_correctness.c \ -o $@ \ $(LDFLAGS_vdso_test_correctness) +$(OUTPUT)/vdso_test_prefetch_page: vdso_test_prefetch_page.c parse_vdso.c
tools/testing/selftests/vDSO/vdso_test_prefetch_page.c new file +265 -0
265 lines not on this page · raw message
Signature · 1 line
2.25.1
Peter Zijlstra
maintainerQuoted 38 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Just as applications can use prefetch instructions to overlap computations and memory accesses, applications may want to overlap the page-faults and compute or overlap the I/O accesses that are required for page-faults of different pages.
Applications can use multiple threads and cores for this matter, by running one thread that prefetches the data (i.e., faults in the data) and another that does the compute, but this scheme is inefficient. Using
mincore()can tell whether a page is mapped, but might not tell whether the page is in the page-cache and does not fault in the data.Introduce
prefetch_page()vDSO-call to prefetch, i.e. fault-in memory asynchronously. The semantic of this call is: try to prefetch a page of in a given address and return zero if the page is accessible following the call. Start I/O operations to retrieve the page if such operations are required and there is no high memory pressure that might introduce slowdowns.Note that as usual the page might be paged-out at any point and therefore, similarly to
mincore(), there is no guarantee that the page will be present at the time that the user application uses the data that resides on the page. Nevertheless, it is expected that in the vast majority of the cases this would not happen, sinceprefetch_page()accesses the page and therefore sets the PTE access-bit (if it is clear).The implementation is as follows. The vDSO code accesses the data, triggering a page-fault it is not present. The handler detects based on the instruction pointer that this is an asynchronous-#PF, using the recently introduce vDSO exception tables. If the page can be brought without waiting (e.g., the page is already in the page-cache), the kernel handles the fault and returns success (zero). If there is memory pressure that prevents the proper handling of the fault (i.e., requires heavy-weight reclamation) it returns a failure. Otherwise, it starts an I/O to bring the page and returns failure.
Quoted 2 lines
Compilers can be extended to issue the
prefetch_page()calls when needed.
Interesting, but given we've been removing explicit prefetch from some parts of the kernel how useful is this in actual use? I'm thinking there should at least be a real user and performance numbers with this before merging.
Quoted 43 lines · Peter Zijlstra
Quoted 38 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Just as applications can use prefetch instructions to overlap computations and memory accesses, applications may want to overlap the page-faults and compute or overlap the I/O accesses that are required for page-faults of different pages.
Applications can use multiple threads and cores for this matter, by running one thread that prefetches the data (i.e., faults in the data) and another that does the compute, but this scheme is inefficient. Using
mincore()can tell whether a page is mapped, but might not tell whether the page is in the page-cache and does not fault in the data.Introduce
prefetch_page()vDSO-call to prefetch, i.e. fault-in memory asynchronously. The semantic of this call is: try to prefetch a page of in a given address and return zero if the page is accessible following the call. Start I/O operations to retrieve the page if such operations are required and there is no high memory pressure that might introduce slowdowns.Note that as usual the page might be paged-out at any point and therefore, similarly to
mincore(), there is no guarantee that the page will be present at the time that the user application uses the data that resides on the page. Nevertheless, it is expected that in the vast majority of the cases this would not happen, sinceprefetch_page()accesses the page and therefore sets the PTE access-bit (if it is clear).The implementation is as follows. The vDSO code accesses the data, triggering a page-fault it is not present. The handler detects based on the instruction pointer that this is an asynchronous-#PF, using the recently introduce vDSO exception tables. If the page can be brought without waiting (e.g., the page is already in the page-cache), the kernel handles the fault and returns success (zero). If there is memory pressure that prevents the proper handling of the fault (i.e., requires heavy-weight reclamation) it returns a failure. Otherwise, it starts an I/O to bring the page and returns failure.
Quoted 7 lines
Quoted 2 lines
Compilers can be extended to issue the
prefetch_page()calls when needed.Interesting, but given we've been removing explicit prefetch from some parts of the kernel how useful is this in actual use? I'm thinking there should at least be a real user and performance numbers with this before merging.
Can you give me a reference to the “removing explicit prefetch from some parts of the kernel”?
I will work on an llvm/gcc plugin to provide some performance numbers. I wanted to make sure that the idea is not a complete obscenity first.
Quoted 8 lines · Nadav Amit
Quoted 5 lines · Peter Zijlstra
Quoted 7 lines
Quoted 7 lines · Nadav Amit
Quoted 6 lines
From: Nadav Amit <[email protected]>
Just as applications can use prefetch instructions to overlap computations and memory accesses, applications may want to overlap the page-faults and compute or overlap the I/O accesses that are required for page-faults of different pages.
[
[ snip ]
Quoted 7 lines
Quoted 4 lines
Interesting, but given we've been removing explicit prefetch from some parts of the kernel how useful is this in actual use? I'm thinking there should at least be a real user and performance numbers with this before merging.
Can you give me a reference to the “removing explicit prefetch from some parts of the kernel”?
Oh. I get it - you mean we remove we remove the use of explicit memory prefetch from the kernel code. Well, I don’t think it is really related, but yes, performance numbers are needed.
Peter Zijlstra
maintainerQuoted 4 lines · Nadav Amit
Quoted 4 lines · Nadav Amit
Can you give me a reference to the “removing explicit prefetch from some parts of the kernel”?
75d65a425c01 ("hlist: remove software prefetching in hlist iterators") e66eed651fd1 ("list: remove prefetching from regular list iterators")
Quoted 3 lines
Oh. I get it - you mean we remove we remove the use of explicit memory prefetch from the kernel code. Well, I don’t think it is really related, but yes, performance numbers are needed.
Right, so my main worry was that use of the prefetch instruction actually hurt performance once the hardware prefetchers got smart enough, this being a very similar construct (just on a different level of the stack) should be careful not to suffer the same fate.
Quoted 4 lines · Nadav Amit
Just as applications can use prefetch instructions to overlap computations and memory accesses, applications may want to overlap the page-faults and compute or overlap the I/O accesses that are required for page-faults of different pages.
Isn't this madvise(MADV_WILLNEED)?
Quoted 10 lines · Matthew Wilcox
Quoted 4 lines · Nadav Amit
Just as applications can use prefetch instructions to overlap computations and memory accesses, applications may want to overlap the page-faults and compute or overlap the I/O accesses that are required for page-faults of different pages.
Isn't this
madvise(MADV_WILLNEED)?
Good point that I should have mentioned. In a way prefetch_page() a combination of mincore() and MADV_WILLNEED.
There are 4 main differences from MADV_WILLNEED:
1. Much lower invocation cost if the readahead is not needed: this allows to prefetch pages more abundantly.
2. Return value: return value tells you whether the page is accessible. This makes it usable for coroutines, for instance. In this regard the call is more similar to mincore() than MADV_WILLNEED.
3. The PTEs are mapped if the pages are already present in the swap/page-cache, preventing an additional page-fault just to map them.
4. Avoiding heavy-weight reclamation on low memory (this may need to be selective, and can be integrated with MADV_WILLNEED).
Quoted 11 lines · Nadav Amit
Quoted 10 lines · Matthew Wilcox
Quoted 4 lines · Nadav Amit
Just as applications can use prefetch instructions to overlap computations and memory accesses, applications may want to overlap the page-faults and compute or overlap the I/O accesses that are required for page-faults of different pages.
Isn't this
madvise(MADV_WILLNEED)?
Quoted 7 lines
Good point that I should have mentioned. In a way
prefetch_page()a combination ofmincore()andMADV_WILLNEED.There are 4 main differences from
MADV_WILLNEED:1. Much lower invocation cost if the readahead is not needed: this allows to prefetch pages more abundantly.
That seems like something that could be fixed in libc -- if we add a page prefetch vdso call, an application calling posix_madvise() could be implemented by calling this fast path. Assuming the performance increase justifies this extra complexity.
Quoted 3 lines
2. Return value: return value tells you whether the page is accessible. This makes it usable for coroutines, for instance. In this regard the call is more similar to
mincore()thanMADV_WILLNEED.
I don't quite understand the programming model you're describing here.
Quoted 2 lines
3. The PTEs are mapped if the pages are already present in the swap/page-cache, preventing an additional page-fault just to map them.
We could enhance madvise() to do this, no?
Quoted 2 lines
4. Avoiding heavy-weight reclamation on low memory (this may need to be selective, and can be integrated with
MADV_WILLNEED).
Likewise.
I don't want to add a new Linux-specific call when there's already a POSIX interface that communicates the exact same thing. The return value seems like the only problem.
https://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_madvise.html
Quoted 40 lines · Matthew Wilcox
Quoted 11 lines · Nadav Amit
Quoted 10 lines · Matthew Wilcox
Quoted 4 lines · Nadav Amit
Just as applications can use prefetch instructions to overlap computations and memory accesses, applications may want to overlap the page-faults and compute or overlap the I/O accesses that are required for page-faults of different pages.
Isn't this
madvise(MADV_WILLNEED)?Quoted 7 lines
Good point that I should have mentioned. In a way
prefetch_page()a combination ofmincore()andMADV_WILLNEED.There are 4 main differences from
MADV_WILLNEED:1. Much lower invocation cost if the readahead is not needed: this allows to prefetch pages more abundantly.
That seems like something that could be fixed in libc -- if we add a page prefetch vdso call, an application calling
posix_madvise()could be implemented by calling this fast path. Assuming the performance increase justifies this extra complexity.Quoted 3 lines
2. Return value: return value tells you whether the page is accessible. This makes it usable for coroutines, for instance. In this regard the call is more similar to
mincore()thanMADV_WILLNEED.I don't quite understand the programming model you're describing here.
Quoted 2 lines
3. The PTEs are mapped if the pages are already present in the swap/page-cache, preventing an additional page-fault just to map them.
We could enhance
madvise()to do this, no?
Quoted 8 lines
Quoted 2 lines
4. Avoiding heavy-weight reclamation on low memory (this may need to be selective, and can be integrated with
MADV_WILLNEED).Likewise.
I don't want to add a new Linux-specific call when there's already a POSIX interface that communicates the exact same thing. The return value seems like the only problem.
I agree that this call does not have to be exposed to the application.
I am not sure there is a lot of extra complexity now, but obviously some evaluations are needed.