*** SUBJECT HERE ***
Subsystems (8)
cover *** SUBJECT HERE ***
From: Jeremy Fitzhardinge <[email protected]>
Hi,
This series changes the paravirt-ops lazy update modes in the following ways:
1. Make paravirt_get_lazy_mode() always return PARAVIRT_LAZY_NONE in interrupt context. This has the effect of making all mmu and cpu state updates synchronous in interrupt routines, without the need to add explicit flushing.
2. De-generalize lazy cpu mode to a specific context-switch operation. Rather than dealing with fully general enter/leave lazy cpu update mode, make it specifically about start/end context switch, since that was the only use anyway. That means the implementations know that they're in the middle of a context switch.
3. On starting a context switch, if the task is in lazy mmu update mode, record that fact in thread_info->flags TIF_LAZY_MMU_UPDATE, and disable lazy updates (flushing anything pending). On ending a context switch, if the new task has TIF_LAZY_MMU_UPDATE set, then reenter lazy mmu update mode.
4. Remove all the preempt_disable/enable pairs and assertion that preemption must be disabled for lazy mmu updates.
This has the net effect of 1) making lazy mmu updates more robust, and
2) not requiring any changes to the current rules of how init_mm can be updated.
Thanks,
J
Jeremy Fitzhardinge (7):
x86/paravirt: remove lazy mode in interrupts
x86/pvops: replace arch_enter_lazy_cpu_mode with arch_start_context_switch x86/paravirt: flush pending mmu updates on context switch x86/paravirt: allow preemption with lazy mmu mode
mm: allow preemption in apply_to_pte_range
x86/paravirt: use percpu_ rather than __get_cpu_var x86/paravirt: finish change from lazy cpu to context switch start/end
PATCH 3/9 x86/paravirt: remove lazy mode in interrupts
From: Jeremy Fitzhardinge <[email protected]>
Make paravirt_lazy_mode() always return PARAVIRT_LAZY_NONE when in an interrupt. This prevents interrupt code from accidentally inheriting an outer lazy state, and instead does everything synchronously. Outer batched operations are left deferred.
--- arch/x86/kernel/paravirt.c | 3 +++ arch/x86/mm/fault.c | 5 ++--- arch/x86/mm/highmem_32.c | 3 --- arch/x86/mm/iomap_32.c | 2 -- arch/x86/mm/pageattr.c | 14 -------------- 5 files changed, 5 insertions(+), 22 deletions(-)
arch/x86/kernel/paravirt.c +3 -0
@@ -283,6 +283,9 @@ void paravirt_leave_lazy_cpu(void)enum paravirt_lazy_mode paravirt_get_lazy_mode (void ) { +if (in_interrupt ()) +return PARAVIRT_LAZY_NONE ; +return __get_cpu_var (paravirt_lazy_mode); }
arch/x86/mm/fault.c +2 -3
@@ -296,10 +296,9 @@ static inline pmd_t *vmalloc_sync_one(pgd_t *pgd, unsigned long address) pmd_k =pmd_offset (pud_k, address);if (!pmd_present (*pmd_k))return NULL ; -if (!pmd_present (*pmd)) { +if (!pmd_present (*pmd))set_pmd (pmd, *pmd_k); -arch_flush_lazy_mmu_mode (); - }else +else BUG_ON (pmd_page (*pmd) !=pmd_page (*pmd_k));return pmd_k; }
arch/x86/mm/highmem_32.c +0 -3
@@ -86,7 +86,6 @@ void *kmap_atomic_prot(struct page *page, enum km_type type, pgprot_t prot) vaddr =__fix_to_virt (FIX_KMAP_BEGIN + idx);BUG_ON (!pte_none (*(kmap_pte-idx)));set_pte (kmap_pte-idx,mk_pte (page, prot)); -arch_flush_lazy_mmu_mode ();return (void *)vaddr; }
@@ -116,7 +115,6 @@ void kunmap_atomic(void *kvaddr, enum km_type type)#endif } -arch_flush_lazy_mmu_mode ();pagefault_enable (); }
@@ -133,7 +131,6 @@ void *kmap_atomic_pfn(unsigned long pfn, enum km_type type) idx = type +KM_TYPE_NR *smp_processor_id (); vaddr =__fix_to_virt (FIX_KMAP_BEGIN + idx);set_pte (kmap_pte-idx,pfn_pte (pfn, kmap_prot)); -arch_flush_lazy_mmu_mode ();return (void *) vaddr; }
arch/x86/mm/iomap_32.c +0 -2
@@ -42,7 +42,6 @@ iomap_atomic_prot_pfn(unsigned long pfn, enum km_type type, pgprot_t prot) idx = type +KM_TYPE_NR *smp_processor_id (); vaddr =__fix_to_virt (FIX_KMAP_BEGIN + idx);set_pte (kmap_pte-idx,pfn_pte (pfn, prot)); -arch_flush_lazy_mmu_mode ();return (void *) vaddr; }
@@ -63,7 +62,6 @@ iounmap_atomic(void *kvaddr, enum km_type type)if (vaddr ==__fix_to_virt (FIX_KMAP_BEGIN +idx))kpte_clear_flush (kmap_pte-idx, vaddr); -arch_flush_lazy_mmu_mode ();pagefault_enable (); }EXPORT_SYMBOL_GPL (iounmap_atomic);
arch/x86/mm/pageattr.c +0 -14
@@ -811,13 +811,6 @@ static int change_page_attr_set_clr(unsigned long *addr, int numpages,vm_unmap_aliases (); -/* -* If we're called with lazy mmu updates enabled, the -* in-memory pte state may be stale. Flush pending updates to -* bring them up to date. -*/ -arch_flush_lazy_mmu_mode (); - cpa.vaddr = addr; cpa.numpages = numpages; cpa.mask_set = mask_set;
@@ -860,13 +853,6 @@ static int change_page_attr_set_clr(unsigned long *addr, int numpages, }else cpa_flush_all (cache); -/* -* If we've been called with lazy mmu updates enabled, then -* make sure that everything gets flushed out before we -* return. -*/ -arch_flush_lazy_mmu_mode (); - out:return ret; }
Signature · 1 line
1.6.0.6
Re: [PATCH 3/9] x86/paravirt: remove lazy mode in interrupts
Quoted 7 lines · Jeremy Fitzhardinge
From: Jeremy Fitzhardinge <[email protected]>
Make
paravirt_lazy_mode()always returnPARAVIRT_LAZY_NONEwhen in an interrupt. This prevents interrupt code from accidentally inheriting an outer lazy state, and instead does everything synchronously. Outer batched operations are left deferred.
This looks like a good solution, but can you make paravirt_lazy_mode() inline? It doesn't need to be a function call.
Zach
Re: [PATCH 3/9] x86/paravirt: remove lazy mode in interrupts
Quoted 11 lines · Zachary Amsden
Quoted 8 lines · Jeremy Fitzhardinge
From: Jeremy Fitzhardinge <[email protected]>
Make
paravirt_lazy_mode()always returnPARAVIRT_LAZY_NONEwhen in an interrupt. This prevents interrupt code from accidentally inheriting an outer lazy state, and instead does everything synchronously. Outer batched operations are left deferred.
Quoted 2 lines
This looks like a good solution, but can you make
paravirt_lazy_mode()inline? It doesn't need to be a function call.
Yeah. I'm not quite sure why it became out of line.
J
PATCH 4/9 x86/pvops: replace arch_enter_lazy_cpu_mode with arch_start_context_switch
In reply to Jeremy FitzhardingeFrom: Jeremy Fitzhardinge <[email protected]>
Make lazy cpu mode more specific to context switching, so that it makes sense to do more context-switch specific things in the callbacks.
--- arch/x86/include/asm/paravirt.h | 8 +++----- arch/x86/kernel/paravirt.c | 13 ------------- arch/x86/kernel/process_32.c | 2 +- arch/x86/kernel/process_64.c | 2 +- arch/x86/xen/mmu.c | 5 +---- include/asm-frv/pgtable.h | 4 ++-- include/asm-generic/pgtable.h | 21 +++++++++++---------- kernel/sched.c | 2 +- 8 files changed, 20 insertions(+), 37 deletions(-)
arch/x86/include/asm/paravirt.h +3 -5
@@ -1420,19 +1420,17 @@ void paravirt_enter_lazy_mmu(void);void paravirt_leave_lazy_mmu (void );void paravirt_leave_lazy (enum paravirt_lazy_mode mode); -#define __HAVE_ARCH_ENTER_LAZY_CPU_MODE -static inline void arch_enter_lazy_cpu_mode (void ) +#define __HAVE_ARCH_START_CONTEXT_SWITCH +static inline void arch_start_context_switch (void ) {PVOP_VCALL0 (pv_cpu_ops.lazy_mode.enter); } -static inline void arch_leave_lazy_cpu_mode (void ) +static inline void arch_end_context_switch (void ) {PVOP_VCALL0 (pv_cpu_ops.lazy_mode.leave); } -void arch_flush_lazy_cpu_mode (void ); -#define __HAVE_ARCH_ENTER_LAZY_MMU_MODEstatic inline void arch_enter_lazy_mmu_mode (void ) {
arch/x86/kernel/paravirt.c +0 -13
@@ -302,19 +302,6 @@ void arch_flush_lazy_mmu_mode(void)preempt_enable (); } -void arch_flush_lazy_cpu_mode (void ) -{ -preempt_disable (); - -if (paravirt_get_lazy_mode () ==PARAVIRT_LAZY_CPU ) { -WARN_ON (preempt_count () ==1 ); -arch_leave_lazy_cpu_mode (); -arch_enter_lazy_cpu_mode (); - } - -preempt_enable (); -} -struct pv_info pv_info = { .name ="bare hardware" , .paravirt_enabled =0 ,
arch/x86/kernel/process_32.c +1 -1
@@ -578,7 +578,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)* done before math_state_restore, so the TS bit is up * to date. */ -arch_leave_lazy_cpu_mode (); +arch_end_context_switch ();/* If the task has used fpu the last 5 timeslices, just do a full * restore of the math state immediately to avoid the trap; the
arch/x86/kernel/process_64.c +1 -1
@@ -595,7 +595,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)* done before math_state_restore, so the TS bit is up * to date. */ -arch_leave_lazy_cpu_mode (); +arch_end_context_switch ();/* * Switch FS and GS.
arch/x86/xen/mmu.c +1 -4
@@ -1112,10 +1112,8 @@ static void drop_other_mm_ref(void *info)/* If this cpu still has a stale cr3 reference, then make sure it has been flushed. */ -if (percpu_read (xen_current_cr3) ==__pa (mm->pgd)) { +if (percpu_read (xen_current_cr3) ==__pa (mm->pgd))load_cr3 (swapper_pg_dir); -arch_flush_lazy_cpu_mode (); - } }static void xen_drop_mm_ref (struct mm_struct *mm)
@@ -1128,7 +1126,6 @@ static void xen_drop_mm_ref(struct mm_struct *mm)load_cr3 (swapper_pg_dir); elseleave_mm (smp_processor_id ()); -arch_flush_lazy_cpu_mode (); }/* Get the "official" set of cpus referring to our pagetable. */
include/asm-frv/pgtable.h +2 -2
@@ -73,8 +73,8 @@ static inline int pte_file(pte_t pte) { return 0; }#define pgtable_cache_init () do {} while (0)#define arch_enter_lazy_mmu_mode () do {} while (0)#define arch_leave_lazy_mmu_mode () do {} while (0) -#define arch_enter_lazy_cpu_mode () do {} while (0) -#define arch_leave_lazy_cpu_mode () do {} while (0) + +#define arch_start_context_switch () do {} while (0)#else /* !CONFIG_MMU */ /*****************************************************************************/
include/asm-generic/pgtable.h +11 -10
@@ -280,17 +280,18 @@ static inline void ptep_modify_prot_commit(struct mm_struct *mm,#endif /* -* A facility to provide batching of the reload of page tables with the -* actual context switch code for paravirtualized guests. By convention, -* only one of the lazy modes (CPU, MMU) should be active at any given -* time, entry should never be nested, and entry and exits should always -* be paired. This is for sanity of maintaining and reasoning about the -* kernel code. +* A facility to provide batching of the reload of page tables and +* other process state with the actual context switch code for +* paravirtualized guests. By convention, only one of the batched +* update (lazy) modes (CPU, MMU) should be active at any given time, +* entry should never be nested, and entry and exits should always be +* paired. This is for sanity of maintaining and reasoning about the +* kernel code. In this case, the exit (end of the context switch) is +* in architecture-specific code, and so doesn't need a generic +* definition. */ -#ifndef __HAVE_ARCH_ENTER_LAZY_CPU_MODE -#define arch_enter_lazy_cpu_mode () do {} while (0) -#define arch_leave_lazy_cpu_mode () do {} while (0) -#define arch_flush_lazy_cpu_mode () do {} while (0) +#ifndef __HAVE_ARCH_START_CONTEXT_SWITCH +#define arch_start_context_switch () do {} while (0)#endif #ifndef __HAVE_PFNMAP_TRACKING
kernel/sched.c +1 -1
@@ -2782,7 +2782,7 @@ context_switch(struct rq *rq, struct task_struct *prev,* combine the page table reload and the switch backend into * one hypercall. */ -arch_enter_lazy_cpu_mode (); +arch_start_context_switch ();if (unlikely (!mm)) { next->active_mm = oldmm;
Signature · 1 line
1.6.0.6
PATCH 5/9 x86/paravirt: flush pending mmu updates on context switch
In reply to Jeremy FitzhardingeFrom: Jeremy Fitzhardinge <[email protected]>
If we're in lazy mmu mode when context switching, leave lazy mmu mode, but remember the task's state in TIF_LAZY_MMU_UPDATES. When we resume the task, check this flag and re-enter lazy mmu mode if its set.
This sets things up for allowing lazy mmu mode while preemptible, though that won't actually be active until the next change.
--- arch/x86/include/asm/paravirt.h | 1 - arch/x86/include/asm/thread_info.h | 2 ++ arch/x86/kernel/kvm.c | 2 +- arch/x86/kernel/paravirt.c | 13 ++++++++++--- arch/x86/kernel/vmi_32.c | 14 ++++++++++---- arch/x86/lguest/boot.c | 14 ++++++++++---- arch/x86/xen/enlighten.c | 6 +++--- arch/x86/xen/mmu.c | 7 ++++++- arch/x86/xen/xen-ops.h | 1 - 9 files changed, 42 insertions(+), 18 deletions(-)
arch/x86/include/asm/paravirt.h +0 -1
@@ -1418,7 +1418,6 @@ void paravirt_enter_lazy_cpu(void);void paravirt_leave_lazy_cpu (void );void paravirt_enter_lazy_mmu (void );void paravirt_leave_lazy_mmu (void ); -void paravirt_leave_lazy (enum paravirt_lazy_mode mode);#define __HAVE_ARCH_START_CONTEXT_SWITCHstatic inline void arch_start_context_switch (void )
arch/x86/include/asm/thread_info.h +2 -0
@@ -95,6 +95,7 @@ struct thread_info {#define TIF_FORCED_TF 24/* true if TF in eflags artificially */ #define TIF_DEBUGCTLMSR 25/* uses thread_struct.debugctlmsr */ #define TIF_DS_AREA_MSR 26/* uses thread_struct.ds_area_msr */ +#define TIF_LAZY_MMU_UPDATES 27/* task is updating the mmu lazily */ #define _TIF_SYSCALL_TRACE (1 << TIF_SYSCALL_TRACE)#define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME)
@@ -117,6 +118,7 @@ struct thread_info {#define _TIF_FORCED_TF (1 << TIF_FORCED_TF)#define _TIF_DEBUGCTLMSR (1 << TIF_DEBUGCTLMSR)#define _TIF_DS_AREA_MSR (1 << TIF_DS_AREA_MSR) +#define _TIF_LAZY_MMU_UPDATES (1 << TIF_LAZY_MMU_UPDATES)/* work to do in syscall_trace_enter() */ #define _TIF_WORK_SYSCALL_ENTRY \
arch/x86/kernel/kvm.c +1 -1
@@ -201,7 +201,7 @@ static void kvm_leave_lazy_mmu(void)struct kvm_para_state *state =kvm_para_state ();mmu_queue_flush (state); -paravirt_leave_lazy (paravirt_get_lazy_mode ()); +paravirt_leave_lazy_mmu (); state->mode =paravirt_get_lazy_mode (); }
arch/x86/kernel/paravirt.c +10 -3
@@ -253,7 +253,7 @@ static inline void enter_lazy(enum paravirt_lazy_mode mode)__get_cpu_var (paravirt_lazy_mode) = mode; } -void paravirt_leave_lazy (enum paravirt_lazy_mode mode) +static void leave_lazy (enum paravirt_lazy_mode mode) {BUG_ON (__get_cpu_var (paravirt_lazy_mode) != mode);BUG_ON (preemptible ());
@@ -268,17 +268,24 @@ void paravirt_enter_lazy_mmu(void)void paravirt_leave_lazy_mmu (void ) { -paravirt_leave_lazy (PARAVIRT_LAZY_MMU ); +leave_lazy (PARAVIRT_LAZY_MMU ); }void paravirt_enter_lazy_cpu (void ) { +if (percpu_read (paravirt_lazy_mode) ==PARAVIRT_LAZY_MMU ) { +arch_leave_lazy_mmu_mode (); +set_thread_flag (TIF_LAZY_MMU_UPDATES ); + }enter_lazy (PARAVIRT_LAZY_CPU ); }void paravirt_leave_lazy_cpu (void ) { -paravirt_leave_lazy (PARAVIRT_LAZY_CPU ); +leave_lazy (PARAVIRT_LAZY_CPU ); + +if (test_and_clear_thread_flag (TIF_LAZY_MMU_UPDATES )) +arch_enter_lazy_mmu_mode (); }enum paravirt_lazy_mode paravirt_get_lazy_mode (void )
arch/x86/kernel/vmi_32.c +10 -4
@@ -473,16 +473,22 @@ static void vmi_enter_lazy_cpu(void) vmi_ops.set_lazy_mode (2 ); } +static void vmi_leave_lazy_cpu (void ) +{ + vmi_ops.set_lazy_mode (0 ); +paravirt_leave_lazy_cpu (); +} +static void vmi_enter_lazy_mmu (void ) {paravirt_enter_lazy_mmu (); vmi_ops.set_lazy_mode (1 ); } -static void vmi_leave_lazy (void ) +static void vmi_leave_lazy_mmu (void ) { -paravirt_leave_lazy (paravirt_get_lazy_mode ()); vmi_ops.set_lazy_mode (0 ); +paravirt_leave_lazy_mmu (); }static inline int __init check_vmi_rom (struct vrom_header *rom)
@@ -718,12 +724,12 @@ static inline int __init activate_vmi(void)para_wrap (pv_cpu_ops.lazy_mode.enter, vmi_enter_lazy_cpu, set_lazy_mode, SetLazyMode); -para_wrap (pv_cpu_ops.lazy_mode.leave, vmi_leave_lazy, +para_wrap (pv_cpu_ops.lazy_mode.leave, vmi_leave_lazy_cpu, set_lazy_mode, SetLazyMode);para_wrap (pv_mmu_ops.lazy_mode.enter, vmi_enter_lazy_mmu, set_lazy_mode, SetLazyMode); -para_wrap (pv_mmu_ops.lazy_mode.leave, vmi_leave_lazy, +para_wrap (pv_mmu_ops.lazy_mode.leave, vmi_leave_lazy_mmu, set_lazy_mode, SetLazyMode);/* user and kernel flush are just handled with different flags to FlushTLB */
arch/x86/lguest/boot.c +10 -4
@@ -147,10 +147,16 @@ static void lazy_hcall(unsigned long call,/* When lazy mode is turned off reset the per-cpu lazy mode variable and then * issue the do-nothing hypercall to flush any stored calls. */ -static void lguest_leave_lazy_mode (void ) +static void lguest_leave_lazy_mmu_mode (void ) { -paravirt_leave_lazy (paravirt_get_lazy_mode ());hcall (LHCALL_FLUSH_ASYNC ,0 ,0 ,0 ); +paravirt_leave_lazy_mmu (); +} + +static void lguest_leave_lazy_cpu_mode (void ) +{ +hcall (LHCALL_FLUSH_ASYNC ,0 ,0 ,0 ); +paravirt_leave_lazy_cpu (); }/*G:033
@@ -1019,7 +1025,7 @@ __init void lguest_init(void) pv_cpu_ops.write_idt_entry = lguest_write_idt_entry; pv_cpu_ops.wbinvd = lguest_wbinvd; pv_cpu_ops.lazy_mode.enter = paravirt_enter_lazy_cpu; - pv_cpu_ops.lazy_mode.leave = lguest_leave_lazy_mode; + pv_cpu_ops.lazy_mode.leave = lguest_leave_lazy_cpu_mode;/* pagetable management */ pv_mmu_ops.write_cr3 = lguest_write_cr3;
@@ -1032,7 +1038,7 @@ __init void lguest_init(void) pv_mmu_ops.read_cr2 = lguest_read_cr2; pv_mmu_ops.read_cr3 = lguest_read_cr3; pv_mmu_ops.lazy_mode.enter = paravirt_enter_lazy_mmu; - pv_mmu_ops.lazy_mode.leave = lguest_leave_lazy_mode; + pv_mmu_ops.lazy_mode.leave = lguest_leave_lazy_mmu_mode;#ifdef CONFIG_X86_LOCAL_APIC /* apic read/write intercepts */
arch/x86/xen/enlighten.c +3 -3
@@ -203,10 +203,10 @@ static unsigned long xen_get_debugreg(int reg)return HYPERVISOR_get_debugreg (reg); } -void xen_leave_lazy (void ) +static void xen_leave_lazy_cpu (void ) { -paravirt_leave_lazy (paravirt_get_lazy_mode ());xen_mc_flush (); +paravirt_leave_lazy_cpu (); }static unsigned long xen_store_tr (void )
@@ -817,7 +817,7 @@ static const struct pv_cpu_ops xen_cpu_ops __initdata = { .lazy_mode = { .enter = paravirt_enter_lazy_cpu, - .leave = xen_leave_lazy, + .leave = xen_leave_lazy_cpu, }, };
arch/x86/xen/mmu.c +6 -1
@@ -1809,6 +1809,11 @@ __init void xen_post_allocator_init(void)xen_mark_init_mm_pinned (); } +static void xen_leave_lazy_mmu (void ) +{ +xen_mc_flush (); +paravirt_leave_lazy_mmu (); +}const struct pv_mmu_ops xen_mmu_ops__initdata = { .pagetable_setup_start = xen_pagetable_setup_start,
@@ -1884,7 +1889,7 @@ const struct pv_mmu_ops xen_mmu_ops __initdata = { .lazy_mode = { .enter = paravirt_enter_lazy_mmu, - .leave = xen_leave_lazy, + .leave = xen_leave_lazy_mmu, }, .set_fixmap = xen_set_fixmap,
arch/x86/xen/xen-ops.h +0 -1
@@ -30,7 +30,6 @@ pgd_t *xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn);void xen_ident_map_ISA (void );void xen_reserve_top (void ); -void xen_leave_lazy (void );void xen_post_allocator_init (void );char *__init xen_memory_setup (void );
Signature · 1 line
1.6.0.6
PATCH 6/9 x86/paravirt: allow preemption with lazy mmu mode
In reply to Jeremy FitzhardingeFrom: Jeremy Fitzhardinge <[email protected]>
Lift restrictions on preemption with lazy mmu mode, as it is now allowed.
--- arch/x86/kernel/paravirt.c | 7 ++++--- arch/x86/xen/mmu.c | 8 +------- 2 files changed, 5 insertions(+), 10 deletions(-)
arch/x86/kernel/paravirt.c +4 -3
@@ -248,7 +248,6 @@ static DEFINE_PER_CPU(enum paravirt_lazy_mode, paravirt_lazy_mode) = PARAVIRT_LAstatic inline void enter_lazy (enum paravirt_lazy_mode mode) {BUG_ON (__get_cpu_var (paravirt_lazy_mode) !=PARAVIRT_LAZY_NONE ); -BUG_ON (preemptible ());__get_cpu_var (paravirt_lazy_mode) = mode; }
@@ -256,7 +255,6 @@ static inline void enter_lazy(enum paravirt_lazy_mode mode)static void leave_lazy (enum paravirt_lazy_mode mode) {BUG_ON (__get_cpu_var (paravirt_lazy_mode) != mode); -BUG_ON (preemptible ());__get_cpu_var (paravirt_lazy_mode) =PARAVIRT_LAZY_NONE ; }
@@ -273,6 +271,8 @@ void paravirt_leave_lazy_mmu(void)void paravirt_enter_lazy_cpu (void ) { +BUG_ON (preemptible ()); +if (percpu_read (paravirt_lazy_mode) ==PARAVIRT_LAZY_MMU ) {arch_leave_lazy_mmu_mode ();set_thread_flag (TIF_LAZY_MMU_UPDATES );
@@ -282,6 +282,8 @@ void paravirt_enter_lazy_cpu(void)void paravirt_leave_lazy_cpu (void ) { +BUG_ON (preemptible ()); +leave_lazy (PARAVIRT_LAZY_CPU );if (test_and_clear_thread_flag (TIF_LAZY_MMU_UPDATES ))
@@ -301,7 +303,6 @@ void arch_flush_lazy_mmu_mode(void)preempt_disable ();if (paravirt_get_lazy_mode () ==PARAVIRT_LAZY_MMU ) { -WARN_ON (preempt_count () ==1 );arch_leave_lazy_mmu_mode ();arch_enter_lazy_mmu_mode (); }
arch/x86/xen/mmu.c +1 -7
@@ -412,10 +412,6 @@ void set_pte_mfn(unsigned long vaddr, unsigned long mfn, pgprot_t flags)void xen_set_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pteval) { -/* updates to init_mm may be done without lock */ -if (mm == &init_mm) -preempt_disable (); -ADD_STATS (set_pte_at,1 );// ADD_STATS(set_pte_at_pinned, xen_page_pinned(ptep)); ADD_STATS (set_pte_at_current, mm == current->mm);
@@ -436,9 +432,7 @@ void xen_set_pte_at(struct mm_struct *mm, unsigned long addr, }xen_set_pte (ptep, pteval); -out: -if (mm == &init_mm) -preempt_enable (); +out:return ; } pte_t xen_ptep_modify_prot_start(struct mm_struct *mm,
Signature · 1 line
1.6.0.6
PATCH 7/9 mm: allow preemption in apply_to_pte_range
In reply to Jeremy FitzhardingeFrom: Jeremy Fitzhardinge <[email protected]>
Preemption is now allowed for lazy mmu mode, so don't disable it for the inner loop of apply_to_pte_range.
--- mm/memory.c | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-)
mm/memory.c +0 -2
@@ -1718,7 +1718,6 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd,BUG_ON (pmd_huge (*pmd)); -preempt_disable ();arch_enter_lazy_mmu_mode (); token =pmd_pgtable (*pmd);
@@ -1730,7 +1729,6 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd, }while (pte++, addr +=PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode (); -preempt_enable ();if (mm != &init_mm)pte_unmap_unlock (pte-1 , ptl);
Signature · 1 line
1.6.0.6
PATCH 8/9 x86/paravirt: use percpu_ rather than __get_cpu_var
In reply to Jeremy FitzhardingeFrom: Jeremy Fitzhardinge <[email protected]>
percpu_read/write is a slightly more direct way of getting to percpu data.
--- arch/x86/kernel/paravirt.c | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/kernel/paravirt.c +5 -5
@@ -247,16 +247,16 @@ static DEFINE_PER_CPU(enum paravirt_lazy_mode, paravirt_lazy_mode) = PARAVIRT_LAstatic inline void enter_lazy (enum paravirt_lazy_mode mode) { -BUG_ON (__get_cpu_var (paravirt_lazy_mode) !=PARAVIRT_LAZY_NONE ); +BUG_ON (percpu_read (paravirt_lazy_mode) !=PARAVIRT_LAZY_NONE ); -__get_cpu_var (paravirt_lazy_mode) = mode; +percpu_write (paravirt_lazy_mode, mode); }static void leave_lazy (enum paravirt_lazy_mode mode) { -BUG_ON (__get_cpu_var (paravirt_lazy_mode) != mode); +BUG_ON (percpu_read (paravirt_lazy_mode) != mode); -__get_cpu_var (paravirt_lazy_mode) =PARAVIRT_LAZY_NONE ; +percpu_write (paravirt_lazy_mode,PARAVIRT_LAZY_NONE ); }void paravirt_enter_lazy_mmu (void )
@@ -295,7 +295,7 @@ enum paravirt_lazy_mode paravirt_get_lazy_mode(void)if (in_interrupt ())return PARAVIRT_LAZY_NONE ; -return __get_cpu_var (paravirt_lazy_mode); +return percpu_read (paravirt_lazy_mode); }void arch_flush_lazy_mmu_mode (void )
Signature · 1 line
1.6.0.6
PATCH 9/9 x86/paravirt: finish change from lazy cpu to context switch start/end
In reply to Jeremy FitzhardingeFrom: Jeremy Fitzhardinge <[email protected]>
Pass the previous and next tasks into the context switch start end calls, so that the called functions can properly access the task state (esp in end_context_switch, in which the next task is not yet completely current).
--- arch/x86/include/asm/paravirt.h | 17 ++++++++++------- arch/x86/include/asm/pgtable.h | 2 ++ arch/x86/kernel/paravirt.c | 14 ++++++-------- arch/x86/kernel/process_32.c | 2 +- arch/x86/kernel/process_64.c | 2 +- arch/x86/kernel/vmi_32.c | 12 ++++++------ arch/x86/lguest/boot.c | 8 ++++---- arch/x86/xen/enlighten.c | 10 ++++------ include/asm-frv/pgtable.h | 2 +- include/asm-generic/pgtable.h | 2 +- kernel/sched.c | 2 +- 11 files changed, 37 insertions(+), 36 deletions(-)
arch/x86/include/asm/paravirt.h +10 -7
@@ -56,6 +56,7 @@ struct desc_ptr;struct tss_struct ;struct mm_struct ;struct desc_struct ; +struct task_struct ;/* * Wrapper type for pointers to code which uses the non-standard
@@ -203,7 +204,8 @@ struct pv_cpu_ops {void (*swapgs)(void ); -struct pv_lazy_ops lazy_mode; +void (*start_context_switch)(struct task_struct *prev); +void (*end_context_switch)(struct task_struct *next); };struct pv_irq_ops {
@@ -1414,20 +1416,21 @@ enum paravirt_lazy_mode { };enum paravirt_lazy_mode paravirt_get_lazy_mode (void ); -void paravirt_enter_lazy_cpu (void ); -void paravirt_leave_lazy_cpu (void ); +void paravirt_start_context_switch (struct task_struct *prev); +void paravirt_end_context_switch (struct task_struct *next); +void paravirt_enter_lazy_mmu (void );void paravirt_leave_lazy_mmu (void );#define __HAVE_ARCH_START_CONTEXT_SWITCH -static inline void arch_start_context_switch (void ) +static inline void arch_start_context_switch (struct task_struct *prev) { -PVOP_VCALL0 (pv_cpu_ops.lazy_mode.enter); +PVOP_VCALL1 (pv_cpu_ops.start_context_switch, prev); } -static inline void arch_end_context_switch (void ) +static inline void arch_end_context_switch (struct task_struct *next) { -PVOP_VCALL0 (pv_cpu_ops.lazy_mode.leave); +PVOP_VCALL1 (pv_cpu_ops.end_context_switch, next); }#define __HAVE_ARCH_ENTER_LAZY_MMU_MODE
arch/x86/include/asm/pgtable.h +2 -0
@@ -83,6 +83,8 @@ static inline void __init paravirt_pagetable_setup_done(pgd_t *base)#define pte_val (x) native_pte_val(x)#define __pte (x) native_make_pte(x) +#define arch_end_context_switch (prev) do {} while(0) +#endif /* CONFIG_PARAVIRT */ /*
arch/x86/kernel/paravirt.c +6 -8
@@ -269,24 +269,24 @@ void paravirt_leave_lazy_mmu(void)leave_lazy (PARAVIRT_LAZY_MMU ); } -void paravirt_enter_lazy_cpu (void ) +void paravirt_start_context_switch (struct task_struct *prev) {BUG_ON (preemptible ());if (percpu_read (paravirt_lazy_mode) ==PARAVIRT_LAZY_MMU ) {arch_leave_lazy_mmu_mode (); -set_thread_flag (TIF_LAZY_MMU_UPDATES ); +set_ti_thread_flag (task_thread_info (prev),TIF_LAZY_MMU_UPDATES ); }enter_lazy (PARAVIRT_LAZY_CPU ); } -void paravirt_leave_lazy_cpu (void ) +void paravirt_end_context_switch (struct task_struct *next) {BUG_ON (preemptible ());leave_lazy (PARAVIRT_LAZY_CPU ); -if (test_and_clear_thread_flag (TIF_LAZY_MMU_UPDATES )) +if (test_and_clear_ti_thread_flag (task_thread_info (next),TIF_LAZY_MMU_UPDATES ))arch_enter_lazy_mmu_mode (); }
@@ -401,10 +401,8 @@ struct pv_cpu_ops pv_cpu_ops = { .set_iopl_mask = native_set_iopl_mask, .io_delay = native_io_delay, - .lazy_mode = { - .enter = paravirt_nop, - .leave = paravirt_nop, - }, + .start_context_switch = paravirt_nop, + .end_context_switch = paravirt_nop, };struct pv_apic_ops pv_apic_ops = {
arch/x86/kernel/process_32.c +1 -1
@@ -578,7 +578,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)* done before math_state_restore, so the TS bit is up * to date. */ -arch_end_context_switch (); +arch_end_context_switch (next_p);/* If the task has used fpu the last 5 timeslices, just do a full * restore of the math state immediately to avoid the trap; the
arch/x86/kernel/process_64.c +1 -1
@@ -595,7 +595,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)* done before math_state_restore, so the TS bit is up * to date. */ -arch_end_context_switch (); +arch_end_context_switch (next_p);/* * Switch FS and GS.
arch/x86/kernel/vmi_32.c +6 -6
@@ -467,16 +467,16 @@ vmi_startup_ipi_hook(int phys_apicid, unsigned long start_eip, }#endif -static void vmi_enter_lazy_cpu (void ) +static void vmi_start_context_switch (struct task_struct *prev) { -paravirt_enter_lazy_cpu (); +paravirt_start_context_switch (prev); vmi_ops.set_lazy_mode (2 ); } -static void vmi_leave_lazy_cpu (void ) +static void vmi_end_context_switch (struct task_struct *next) { vmi_ops.set_lazy_mode (0 ); -paravirt_leave_lazy_cpu (); +paravirt_end_context_switch (next); }static void vmi_enter_lazy_mmu (void )
@@ -722,9 +722,9 @@ static inline int __init activate_vmi(void)para_fill (pv_cpu_ops.set_iopl_mask, SetIOPLMask);para_fill (pv_cpu_ops.io_delay, IODelay); -para_wrap (pv_cpu_ops.lazy_mode.enter, vmi_enter_lazy_cpu, +para_wrap (pv_cpu_ops.start_context_switch, vmi_start_context_switch, set_lazy_mode, SetLazyMode); -para_wrap (pv_cpu_ops.lazy_mode.leave, vmi_leave_lazy_cpu, +para_wrap (pv_cpu_ops.end_context_switch, vmi_end_context_switch, set_lazy_mode, SetLazyMode); para_wrap(pv_mmu_ops.lazy_mode.enter, vmi_enter_lazy_mmu,
arch/x86/lguest/boot.c +4 -4
@@ -153,10 +153,10 @@ static void lguest_leave_lazy_mmu_mode(void)paravirt_leave_lazy_mmu (); } -static void lguest_leave_lazy_cpu_mode (void ) +static void lguest_end_context_switch (struct task_struct *next) {hcall (LHCALL_FLUSH_ASYNC ,0 ,0 ,0 ); -paravirt_leave_lazy_cpu (); +paravirt_end_context_switch (next); }/*G:033
@@ -1024,8 +1024,8 @@ __init void lguest_init(void) pv_cpu_ops.write_gdt_entry = lguest_write_gdt_entry; pv_cpu_ops.write_idt_entry = lguest_write_idt_entry; pv_cpu_ops.wbinvd = lguest_wbinvd; - pv_cpu_ops.lazy_mode.enter = paravirt_enter_lazy_cpu; - pv_cpu_ops.lazy_mode.leave = lguest_leave_lazy_cpu_mode; + pv_cpu_ops.start_context_switch = paravirt_start_context_switch; + pv_cpu_ops.end_context_switch = lguest_end_context_switch;/* pagetable management */ pv_mmu_ops.write_cr3 = lguest_write_cr3;
arch/x86/xen/enlighten.c +4 -6
@@ -203,10 +203,10 @@ static unsigned long xen_get_debugreg(int reg)return HYPERVISOR_get_debugreg (reg); } -static void xen_leave_lazy_cpu (void ) +static void xen_end_context_switch (struct task_struct *next) {xen_mc_flush (); -paravirt_leave_lazy_cpu (); +paravirt_end_context_switch (next); }static unsigned long xen_store_tr (void )
@@ -815,10 +815,8 @@ static const struct pv_cpu_ops xen_cpu_ops __initdata = {/* Xen takes care of %gs when switching to usermode for us */ .swapgs = paravirt_nop, - .lazy_mode = { - .enter = paravirt_enter_lazy_cpu, - .leave = xen_leave_lazy_cpu, - }, + .start_context_switch = paravirt_start_context_switch, + .end_context_switch = xen_end_context_switch, };static const struct pv_apic_ops xen_apic_ops__initdata = {
include/asm-frv/pgtable.h +1 -1
@@ -74,7 +74,7 @@ static inline int pte_file(pte_t pte) { return 0; }#define arch_enter_lazy_mmu_mode () do {} while (0)#define arch_leave_lazy_mmu_mode () do {} while (0) -#define arch_start_context_switch () do {} while (0) +#define arch_start_context_switch (prev) do {} while (0)#else /* !CONFIG_MMU */ /*****************************************************************************/
include/asm-generic/pgtable.h +1 -1
@@ -291,7 +291,7 @@ static inline void ptep_modify_prot_commit(struct mm_struct *mm,* definition. */ #ifndef __HAVE_ARCH_START_CONTEXT_SWITCH -#define arch_start_context_switch () do {} while (0) +#define arch_start_context_switch (prev) do {} while (0)#endif #ifndef __HAVE_PFNMAP_TRACKING
kernel/sched.c +1 -1
@@ -2782,7 +2782,7 @@ context_switch(struct rq *rq, struct task_struct *prev,* combine the page table reload and the switch backend into * one hypercall. */ -arch_start_context_switch (); +arch_start_context_switch (prev);if (unlikely (!mm)) { next->active_mm = oldmm;
Signature · 1 line
1.6.0.6
Andrew Morton
maintainerRe: [PATCH 0/9] Make lazy mmu update mode pre-empt safe
In reply to Jeremy FitzhardingeQuoted 1 line · On Wed, 18 Feb 2009 14:30:10 -0800 Jeremy Fitzhardinge <[email protected]> wrote:
This series changes the paravirt-ops lazy update modes in the following ways:
glumph. This doesn't look like a 2.6.29 quickfix.
Re: [PATCH 0/9] Make lazy mmu update mode pre-empt safe
Quoted 9 lines · Andrew Morton
Quoted 2 lines · On Wed, 18 Feb 2009 14:30:10 -0800 Jeremy Fitzhardinge <[email protected]> wrote:
This series changes the paravirt-ops lazy update modes in the following ways:
glumph. This doesn't look like a 2.6.29 quickfix.
You mean for apply_to_page_range? No, definitely not, but I don't think it has any users wanting to operate on init_mm in 2.6.29.
J
Re: [PATCH 0/9] Make lazy mmu update mode pre-empt safe
In reply to Jeremy Fitzhardinge[ Clearly some aspect of this mailbomb tool is eluding me. ]
Hi,
This series changes the paravirt-ops lazy update modes in the following ways:
1. Make paravirt_get_lazy_mode() always return PARAVIRT_LAZY_NONE in interrupt context. This has the effect of making all mmu and cpu state updates synchronous in interrupt routines, without the need to add explicit flushing.
2. De-generalize lazy cpu mode to a specific context-switch operation. Rather than dealing with fully general enter/leave lazy cpu update mode, make it specifically about start/end context switch, since that was the only use anyway. That means the implementations know that they're in the middle of a context switch.
3. On starting a context switch, if the task is in lazy mmu update mode, record that fact in thread_info->flags TIF_LAZY_MMU_UPDATE, and disable lazy updates (flushing anything pending). On ending a context switch, if the new task has TIF_LAZY_MMU_UPDATE set, then reenter lazy mmu update mode.
4. Remove all the preempt_disable/enable pairs and assertion that preemption must be disabled for lazy mmu updates.
This has the net effect of 1) making lazy mmu updates more robust, and
2) not requiring any changes to the current rules of how init_mm can be updated.
Thanks,
J
Re: [PATCH 0/9] Make lazy mmu update mode pre-empt safe
Quoted 1 line · Jeremy Fitzhardinge
[ Clearly some aspect of this mailbomb tool is eluding me. ]
Are you using "git format-patch" and "git send-email" ? Then perhaps the --compose option to git-send-email is what you are after?
Ingo
Re: [PATCH 0/9] Make lazy mmu update mode pre-empt safe
Quoted 1 line · Ingo Molnar
Are you using "git format-patch" and "git send-email" ?
Yes.
Quoted 2 lines
Then
perhaps the --compose option to git-send-email is what you are after?
I'm pre-editing the 0000-cover.patch - but it looks like its picking up the 0000-cover.patch~ file in preference to 0000-cover.patch. I think.
J
Re: [PATCH 0/9] Make lazy mmu update mode pre-empt safe
Quoted 10 lines · Jeremy Fitzhardinge
Quoted 1 line · Ingo Molnar
Are you using "git format-patch" and "git send-email" ?
Yes.
Quoted 2 lines
Then perhaps the --compose option to git-send-email is what you are
after?I'm pre-editing the 0000-cover.patch - but it looks like its picking up the 0000-cover.patch~ file in preference to 0000-cover.patch. I think.
try test-sending to yourself until it works :-)
Ingo