Skip to contentSkip to conversation

*** SUBJECT HERE ***

PATCH In review Jeremy Fitzhardinge 4 participants 16 messages Feb 18, 2009 – Feb 19, 2009

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

cover *** SUBJECT HERE ***

From: Jeremy Fitzhardinge <[email protected]>

Hi,

This series changes the paravirt-ops lazy update modes in the following ways:

1. Make paravirt_get_lazy_mode() always return PARAVIRT_LAZY_NONE in interrupt context. This has the effect of making all mmu and cpu state updates synchronous in interrupt routines, without the need to add explicit flushing.

2. De-generalize lazy cpu mode to a specific context-switch operation. Rather than dealing with fully general enter/leave lazy cpu update mode, make it specifically about start/end context switch, since that was the only use anyway. That means the implementations know that they're in the middle of a context switch.

3. On starting a context switch, if the task is in lazy mmu update mode, record that fact in thread_info->flags TIF_LAZY_MMU_UPDATE, and disable lazy updates (flushing anything pending). On ending a context switch, if the new task has TIF_LAZY_MMU_UPDATE set, then reenter lazy mmu update mode.

4. Remove all the preempt_disable/enable pairs and assertion that preemption must be disabled for lazy mmu updates.

This has the net effect of 1) making lazy mmu updates more robust, and
2) not requiring any changes to the current rules of how init_mm can be updated.

Thanks,
J

Jeremy Fitzhardinge (7):
x86/paravirt: remove lazy mode in interrupts
x86/pvops: replace arch_enter_lazy_cpu_mode with arch_start_context_switch x86/paravirt: flush pending mmu updates on context switch x86/paravirt: allow preemption with lazy mmu mode
mm: allow preemption in apply_to_pte_range
x86/paravirt: use percpu_ rather than __get_cpu_var x86/paravirt: finish change from lazy cpu to context switch start/end

#1 lore raw

PATCH 3/9 x86/paravirt: remove lazy mode in interrupts

From: Jeremy Fitzhardinge <[email protected]>

Make paravirt_lazy_mode() always return PARAVIRT_LAZY_NONE when in an interrupt. This prevents interrupt code from accidentally inheriting an outer lazy state, and instead does everything synchronously. Outer batched operations are left deferred.

---
 arch/x86/kernel/paravirt.c |    3 +++
 arch/x86/mm/fault.c        |    5 ++---
 arch/x86/mm/highmem_32.c   |    3 ---
 arch/x86/mm/iomap_32.c     |    2 --
 arch/x86/mm/pageattr.c     |   14 --------------
 5 files changed, 5 insertions(+), 22 deletions(-)
arch/x86/kernel/paravirt.c +3 -0
index 6dc4dca..83dadf9 100644
@@ -283,6 +283,9 @@ void paravirt_leave_lazy_cpu(void)
 
 enum paravirt_lazy_mode paravirt_get_lazy_mode(void)
 {
+	if (in_interrupt())
+		return PARAVIRT_LAZY_NONE;
+
 	return __get_cpu_var(paravirt_lazy_mode);
 }
 
arch/x86/mm/fault.c +2 -3
index 6b9239d..51b352a 100644
@@ -296,10 +296,9 @@ static inline pmd_t *vmalloc_sync_one(pgd_t *pgd, unsigned long address)
 	pmd_k = pmd_offset(pud_k, address);
 	if (!pmd_present(*pmd_k))
 		return NULL;
-	if (!pmd_present(*pmd)) {
+	if (!pmd_present(*pmd))
 		set_pmd(pmd, *pmd_k);
-		arch_flush_lazy_mmu_mode();
-	} else
+	else
 		BUG_ON(pmd_page(*pmd) != pmd_page(*pmd_k));
 	return pmd_k;
 }
arch/x86/mm/highmem_32.c +0 -3
index bcc079c..98c9ac7 100644
@@ -86,7 +86,6 @@ void *kmap_atomic_prot(struct page *page, enum km_type type, pgprot_t prot)
 	vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
 	BUG_ON(!pte_none(*(kmap_pte-idx)));
 	set_pte(kmap_pte-idx, mk_pte(page, prot));
-	arch_flush_lazy_mmu_mode();
 
 	return (void *)vaddr;
 }
@@ -116,7 +115,6 @@ void kunmap_atomic(void *kvaddr, enum km_type type)
 #endif
 	}
 
-	arch_flush_lazy_mmu_mode();
 	pagefault_enable();
 }
 
@@ -133,7 +131,6 @@ void *kmap_atomic_pfn(unsigned long pfn, enum km_type type)
 	idx = type + KM_TYPE_NR*smp_processor_id();
 	vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
 	set_pte(kmap_pte-idx, pfn_pte(pfn, kmap_prot));
-	arch_flush_lazy_mmu_mode();
 
 	return (void*) vaddr;
 }
arch/x86/mm/iomap_32.c +0 -2
index ca53224..a75f8bb 100644
@@ -42,7 +42,6 @@ iomap_atomic_prot_pfn(unsigned long pfn, enum km_type type, pgprot_t prot)
 	idx = type + KM_TYPE_NR*smp_processor_id();
 	vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
 	set_pte(kmap_pte-idx, pfn_pte(pfn, prot));
-	arch_flush_lazy_mmu_mode();
 
 	return (void*) vaddr;
 }
@@ -63,7 +62,6 @@ iounmap_atomic(void *kvaddr, enum km_type type)
 	if (vaddr == __fix_to_virt(FIX_KMAP_BEGIN+idx))
 		kpte_clear_flush(kmap_pte-idx, vaddr);
 
-	arch_flush_lazy_mmu_mode();
 	pagefault_enable();
 }
 EXPORT_SYMBOL_GPL(iounmap_atomic);
arch/x86/mm/pageattr.c +0 -14
index 8ca0d85..52460f9 100644
@@ -811,13 +811,6 @@ static int change_page_attr_set_clr(unsigned long *addr, int numpages,
 
 	vm_unmap_aliases();
 
-	/*
-	 * If we're called with lazy mmu updates enabled, the
-	 * in-memory pte state may be stale.  Flush pending updates to
-	 * bring them up to date.
-	 */
-	arch_flush_lazy_mmu_mode();
-
 	cpa.vaddr = addr;
 	cpa.numpages = numpages;
 	cpa.mask_set = mask_set;
@@ -860,13 +853,6 @@ static int change_page_attr_set_clr(unsigned long *addr, int numpages,
 	} else
 		cpa_flush_all(cache);
 
-	/*
-	 * If we've been called with lazy mmu updates enabled, then
-	 * make sure that everything gets flushed out before we
-	 * return.
-	 */
-	arch_flush_lazy_mmu_mode();
-
 out:
 	return ret;
 }
Signature · 1 line
1.6.0.6
#4 lore raw

PATCH 4/9 x86/pvops: replace arch_enter_lazy_cpu_mode with arch_start_context_switch

In reply to Jeremy Fitzhardinge

From: Jeremy Fitzhardinge <[email protected]>

Make lazy cpu mode more specific to context switching, so that it makes sense to do more context-switch specific things in the callbacks.

---
 arch/x86/include/asm/paravirt.h |    8 +++-----
 arch/x86/kernel/paravirt.c      |   13 -------------
 arch/x86/kernel/process_32.c    |    2 +-
 arch/x86/kernel/process_64.c    |    2 +-
 arch/x86/xen/mmu.c              |    5 +----
 include/asm-frv/pgtable.h       |    4 ++--
 include/asm-generic/pgtable.h   |   21 +++++++++++----------
 kernel/sched.c                  |    2 +-
 8 files changed, 20 insertions(+), 37 deletions(-)
arch/x86/include/asm/paravirt.h +3 -5
index 0617d5c..7b28aba 100644
@@ -1420,19 +1420,17 @@ void paravirt_enter_lazy_mmu(void);
 void paravirt_leave_lazy_mmu(void);
 void paravirt_leave_lazy(enum paravirt_lazy_mode mode);
 
-#define  __HAVE_ARCH_ENTER_LAZY_CPU_MODE
-static inline void arch_enter_lazy_cpu_mode(void)
+#define  __HAVE_ARCH_START_CONTEXT_SWITCH
+static inline void arch_start_context_switch(void)
 {
 	PVOP_VCALL0(pv_cpu_ops.lazy_mode.enter);
 }
 
-static inline void arch_leave_lazy_cpu_mode(void)
+static inline void arch_end_context_switch(void)
 {
 	PVOP_VCALL0(pv_cpu_ops.lazy_mode.leave);
 }
 
-void arch_flush_lazy_cpu_mode(void);
-
 #define  __HAVE_ARCH_ENTER_LAZY_MMU_MODE
 static inline void arch_enter_lazy_mmu_mode(void)
 {
arch/x86/kernel/paravirt.c +0 -13
index 83dadf9..ef3ca17 100644
@@ -302,19 +302,6 @@ void arch_flush_lazy_mmu_mode(void)
 	preempt_enable();
 }
 
-void arch_flush_lazy_cpu_mode(void)
-{
-	preempt_disable();
-
-	if (paravirt_get_lazy_mode() == PARAVIRT_LAZY_CPU) {
-		WARN_ON(preempt_count() == 1);
-		arch_leave_lazy_cpu_mode();
-		arch_enter_lazy_cpu_mode();
-	}
-
-	preempt_enable();
-}
-
 struct pv_info pv_info = {
 	.name = "bare hardware",
 	.paravirt_enabled = 0,
arch/x86/kernel/process_32.c +1 -1
index 8a09fa0..60d7b18 100644
@@ -578,7 +578,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)
 	 * done before math_state_restore, so the TS bit is up
 	 * to date.
 	 */
-	arch_leave_lazy_cpu_mode();
+	arch_end_context_switch();
 
 	/* If the task has used fpu the last 5 timeslices, just do a full
 	 * restore of the math state immediately to avoid the trap; the
arch/x86/kernel/process_64.c +1 -1
index 836ef65..d5156e7 100644
@@ -595,7 +595,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)
 	 * done before math_state_restore, so the TS bit is up
 	 * to date.
 	 */
-	arch_leave_lazy_cpu_mode();
+	arch_end_context_switch();
 
 	/*
 	 * Switch FS and GS.
arch/x86/xen/mmu.c +1 -4
index 319bd40..298b9f5 100644
@@ -1112,10 +1112,8 @@ static void drop_other_mm_ref(void *info)
 
 	/* If this cpu still has a stale cr3 reference, then make sure
 	   it has been flushed. */
-	if (percpu_read(xen_current_cr3) == __pa(mm->pgd)) {
+	if (percpu_read(xen_current_cr3) == __pa(mm->pgd))
 		load_cr3(swapper_pg_dir);
-		arch_flush_lazy_cpu_mode();
-	}
 }
 
 static void xen_drop_mm_ref(struct mm_struct *mm)
@@ -1128,7 +1126,6 @@ static void xen_drop_mm_ref(struct mm_struct *mm)
 			load_cr3(swapper_pg_dir);
 		else
 			leave_mm(smp_processor_id());
-		arch_flush_lazy_cpu_mode();
 	}
 
 	/* Get the "official" set of cpus referring to our pagetable. */
include/asm-frv/pgtable.h +2 -2
index 83c51ab..88bec28 100644
@@ -73,8 +73,8 @@ static inline int pte_file(pte_t pte) { return 0; }
 #define pgtable_cache_init()		do {} while (0)
 #define arch_enter_lazy_mmu_mode()	do {} while (0)
 #define arch_leave_lazy_mmu_mode()	do {} while (0)
-#define arch_enter_lazy_cpu_mode()	do {} while (0)
-#define arch_leave_lazy_cpu_mode()	do {} while (0)
+
+#define arch_start_context_switch()	do {} while (0)
 
 #else /* !CONFIG_MMU */
 /*****************************************************************************/
include/asm-generic/pgtable.h +11 -10
index 8e6d0ca..922f036 100644
@@ -280,17 +280,18 @@ static inline void ptep_modify_prot_commit(struct mm_struct *mm,
 #endif
 
 /*
- * A facility to provide batching of the reload of page tables with the
- * actual context switch code for paravirtualized guests.  By convention,
- * only one of the lazy modes (CPU, MMU) should be active at any given
- * time, entry should never be nested, and entry and exits should always
- * be paired.  This is for sanity of maintaining and reasoning about the
- * kernel code.
+ * A facility to provide batching of the reload of page tables and
+ * other process state with the actual context switch code for
+ * paravirtualized guests.  By convention, only one of the batched
+ * update (lazy) modes (CPU, MMU) should be active at any given time,
+ * entry should never be nested, and entry and exits should always be
+ * paired.  This is for sanity of maintaining and reasoning about the
+ * kernel code.  In this case, the exit (end of the context switch) is
+ * in architecture-specific code, and so doesn't need a generic
+ * definition.
  */
-#ifndef __HAVE_ARCH_ENTER_LAZY_CPU_MODE
-#define arch_enter_lazy_cpu_mode()	do {} while (0)
-#define arch_leave_lazy_cpu_mode()	do {} while (0)
-#define arch_flush_lazy_cpu_mode()	do {} while (0)
+#ifndef __HAVE_ARCH_START_CONTEXT_SWITCH
+#define arch_start_context_switch()	do {} while (0)
 #endif
 
 #ifndef __HAVE_PFNMAP_TRACKING
kernel/sched.c +1 -1
index a345d40..25d4c6b 100644
@@ -2782,7 +2782,7 @@ context_switch(struct rq *rq, struct task_struct *prev,
 	 * combine the page table reload and the switch backend into
 	 * one hypercall.
 	 */
-	arch_enter_lazy_cpu_mode();
+	arch_start_context_switch();
 
 	if (unlikely(!mm)) {
 		next->active_mm = oldmm;
Signature · 1 line
1.6.0.6
#5 lore raw

PATCH 5/9 x86/paravirt: flush pending mmu updates on context switch

In reply to Jeremy Fitzhardinge

From: Jeremy Fitzhardinge <[email protected]>

If we're in lazy mmu mode when context switching, leave lazy mmu mode, but remember the task's state in TIF_LAZY_MMU_UPDATES. When we resume the task, check this flag and re-enter lazy mmu mode if its set.

This sets things up for allowing lazy mmu mode while preemptible, though that won't actually be active until the next change.

---
 arch/x86/include/asm/paravirt.h    |    1 -
 arch/x86/include/asm/thread_info.h |    2 ++
 arch/x86/kernel/kvm.c              |    2 +-
 arch/x86/kernel/paravirt.c         |   13 ++++++++++---
 arch/x86/kernel/vmi_32.c           |   14 ++++++++++----
 arch/x86/lguest/boot.c             |   14 ++++++++++----
 arch/x86/xen/enlighten.c           |    6 +++---
 arch/x86/xen/mmu.c                 |    7 ++++++-
 arch/x86/xen/xen-ops.h             |    1 -
 9 files changed, 42 insertions(+), 18 deletions(-)
arch/x86/include/asm/paravirt.h +0 -1
index 7b28aba..58d2481 100644
@@ -1418,7 +1418,6 @@ void paravirt_enter_lazy_cpu(void);
 void paravirt_leave_lazy_cpu(void);
 void paravirt_enter_lazy_mmu(void);
 void paravirt_leave_lazy_mmu(void);
-void paravirt_leave_lazy(enum paravirt_lazy_mode mode);
 
 #define  __HAVE_ARCH_START_CONTEXT_SWITCH
 static inline void arch_start_context_switch(void)
arch/x86/include/asm/thread_info.h +2 -0
index ca7310e..b857399 100644
@@ -95,6 +95,7 @@ struct thread_info {
 #define TIF_FORCED_TF		24	/* true if TF in eflags artificially */
 #define TIF_DEBUGCTLMSR		25	/* uses thread_struct.debugctlmsr */
 #define TIF_DS_AREA_MSR		26      /* uses thread_struct.ds_area_msr */
+#define TIF_LAZY_MMU_UPDATES	27	/* task is updating the mmu lazily */
 
 #define _TIF_SYSCALL_TRACE	(1 << TIF_SYSCALL_TRACE)
 #define _TIF_NOTIFY_RESUME	(1 << TIF_NOTIFY_RESUME)
@@ -117,6 +118,7 @@ struct thread_info {
 #define _TIF_FORCED_TF		(1 << TIF_FORCED_TF)
 #define _TIF_DEBUGCTLMSR	(1 << TIF_DEBUGCTLMSR)
 #define _TIF_DS_AREA_MSR	(1 << TIF_DS_AREA_MSR)
+#define _TIF_LAZY_MMU_UPDATES	(1 << TIF_LAZY_MMU_UPDATES)
 
 /* work to do in syscall_trace_enter() */
 #define _TIF_WORK_SYSCALL_ENTRY	\
arch/x86/kernel/kvm.c +1 -1
index 478bca9..5d7f6e7 100644
@@ -201,7 +201,7 @@ static void kvm_leave_lazy_mmu(void)
 	struct kvm_para_state *state = kvm_para_state();
 
 	mmu_queue_flush(state);
-	paravirt_leave_lazy(paravirt_get_lazy_mode());
+	paravirt_leave_lazy_mmu();
 	state->mode = paravirt_get_lazy_mode();
 }
 
arch/x86/kernel/paravirt.c +10 -3
index ef3ca17..554dadd 100644
@@ -253,7 +253,7 @@ static inline void enter_lazy(enum paravirt_lazy_mode mode)
 	__get_cpu_var(paravirt_lazy_mode) = mode;
 }
 
-void paravirt_leave_lazy(enum paravirt_lazy_mode mode)
+static void leave_lazy(enum paravirt_lazy_mode mode)
 {
 	BUG_ON(__get_cpu_var(paravirt_lazy_mode) != mode);
 	BUG_ON(preemptible());
@@ -268,17 +268,24 @@ void paravirt_enter_lazy_mmu(void)
 
 void paravirt_leave_lazy_mmu(void)
 {
-	paravirt_leave_lazy(PARAVIRT_LAZY_MMU);
+	leave_lazy(PARAVIRT_LAZY_MMU);
 }
 
 void paravirt_enter_lazy_cpu(void)
 {
+	if (percpu_read(paravirt_lazy_mode) == PARAVIRT_LAZY_MMU) {
+		arch_leave_lazy_mmu_mode();
+		set_thread_flag(TIF_LAZY_MMU_UPDATES);
+	}
 	enter_lazy(PARAVIRT_LAZY_CPU);
 }
 
 void paravirt_leave_lazy_cpu(void)
 {
-	paravirt_leave_lazy(PARAVIRT_LAZY_CPU);
+	leave_lazy(PARAVIRT_LAZY_CPU);
+
+	if (test_and_clear_thread_flag(TIF_LAZY_MMU_UPDATES))
+		arch_enter_lazy_mmu_mode();
 }
 
 enum paravirt_lazy_mode paravirt_get_lazy_mode(void)
arch/x86/kernel/vmi_32.c +10 -4
index 2cc4a90..950929c 100644
@@ -473,16 +473,22 @@ static void vmi_enter_lazy_cpu(void)
 	vmi_ops.set_lazy_mode(2);
 }
 
+static void vmi_leave_lazy_cpu(void)
+{
+	vmi_ops.set_lazy_mode(0);
+	paravirt_leave_lazy_cpu();
+}
+
 static void vmi_enter_lazy_mmu(void)
 {
 	paravirt_enter_lazy_mmu();
 	vmi_ops.set_lazy_mode(1);
 }
 
-static void vmi_leave_lazy(void)
+static void vmi_leave_lazy_mmu(void)
 {
-	paravirt_leave_lazy(paravirt_get_lazy_mode());
 	vmi_ops.set_lazy_mode(0);
+	paravirt_leave_lazy_mmu();
 }
 
 static inline int __init check_vmi_rom(struct vrom_header *rom)
@@ -718,12 +724,12 @@ static inline int __init activate_vmi(void)
 
 	para_wrap(pv_cpu_ops.lazy_mode.enter, vmi_enter_lazy_cpu,
 		  set_lazy_mode, SetLazyMode);
-	para_wrap(pv_cpu_ops.lazy_mode.leave, vmi_leave_lazy,
+	para_wrap(pv_cpu_ops.lazy_mode.leave, vmi_leave_lazy_cpu,
 		  set_lazy_mode, SetLazyMode);
 
 	para_wrap(pv_mmu_ops.lazy_mode.enter, vmi_enter_lazy_mmu,
 		  set_lazy_mode, SetLazyMode);
-	para_wrap(pv_mmu_ops.lazy_mode.leave, vmi_leave_lazy,
+	para_wrap(pv_mmu_ops.lazy_mode.leave, vmi_leave_lazy_mmu,
 		  set_lazy_mode, SetLazyMode);
 
 	/* user and kernel flush are just handled with different flags to FlushTLB */
arch/x86/lguest/boot.c +10 -4
index f3a5305..804c170 100644
@@ -147,10 +147,16 @@ static void lazy_hcall(unsigned long call,
 
 /* When lazy mode is turned off reset the per-cpu lazy mode variable and then
  * issue the do-nothing hypercall to flush any stored calls. */
-static void lguest_leave_lazy_mode(void)
+static void lguest_leave_lazy_mmu_mode(void)
 {
-	paravirt_leave_lazy(paravirt_get_lazy_mode());
 	hcall(LHCALL_FLUSH_ASYNC, 0, 0, 0);
+	paravirt_leave_lazy_mmu();
+}
+
+static void lguest_leave_lazy_cpu_mode(void)
+{
+	hcall(LHCALL_FLUSH_ASYNC, 0, 0, 0);
+	paravirt_leave_lazy_cpu();
 }
 
 /*G:033
@@ -1019,7 +1025,7 @@ __init void lguest_init(void)
 	pv_cpu_ops.write_idt_entry = lguest_write_idt_entry;
 	pv_cpu_ops.wbinvd = lguest_wbinvd;
 	pv_cpu_ops.lazy_mode.enter = paravirt_enter_lazy_cpu;
-	pv_cpu_ops.lazy_mode.leave = lguest_leave_lazy_mode;
+	pv_cpu_ops.lazy_mode.leave = lguest_leave_lazy_cpu_mode;
 
 	/* pagetable management */
 	pv_mmu_ops.write_cr3 = lguest_write_cr3;
@@ -1032,7 +1038,7 @@ __init void lguest_init(void)
 	pv_mmu_ops.read_cr2 = lguest_read_cr2;
 	pv_mmu_ops.read_cr3 = lguest_read_cr3;
 	pv_mmu_ops.lazy_mode.enter = paravirt_enter_lazy_mmu;
-	pv_mmu_ops.lazy_mode.leave = lguest_leave_lazy_mode;
+	pv_mmu_ops.lazy_mode.leave = lguest_leave_lazy_mmu_mode;
 
 #ifdef CONFIG_X86_LOCAL_APIC
 	/* apic read/write intercepts */
arch/x86/xen/enlighten.c +3 -3
index 86497d5..fc6519a 100644
@@ -203,10 +203,10 @@ static unsigned long xen_get_debugreg(int reg)
 	return HYPERVISOR_get_debugreg(reg);
 }
 
-void xen_leave_lazy(void)
+static void xen_leave_lazy_cpu(void)
 {
-	paravirt_leave_lazy(paravirt_get_lazy_mode());
 	xen_mc_flush();
+	paravirt_leave_lazy_cpu();
 }
 
 static unsigned long xen_store_tr(void)
@@ -817,7 +817,7 @@ static const struct pv_cpu_ops xen_cpu_ops __initdata = {
 
 	.lazy_mode = {
 		.enter = paravirt_enter_lazy_cpu,
-		.leave = xen_leave_lazy,
+		.leave = xen_leave_lazy_cpu,
 	},
 };
 
arch/x86/xen/mmu.c +6 -1
index 298b9f5..2669c58 100644
@@ -1809,6 +1809,11 @@ __init void xen_post_allocator_init(void)
 	xen_mark_init_mm_pinned();
 }
 
+static void xen_leave_lazy_mmu(void)
+{
+	xen_mc_flush();
+	paravirt_leave_lazy_mmu();
+}
 
 const struct pv_mmu_ops xen_mmu_ops __initdata = {
 	.pagetable_setup_start = xen_pagetable_setup_start,
@@ -1884,7 +1889,7 @@ const struct pv_mmu_ops xen_mmu_ops __initdata = {
 
 	.lazy_mode = {
 		.enter = paravirt_enter_lazy_mmu,
-		.leave = xen_leave_lazy,
+		.leave = xen_leave_lazy_mmu,
 	},
 
 	.set_fixmap = xen_set_fixmap,
arch/x86/xen/xen-ops.h +0 -1
index 2f5ef26..f897cdf 100644
@@ -30,7 +30,6 @@ pgd_t *xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn);
 void xen_ident_map_ISA(void);
 void xen_reserve_top(void);
 
-void xen_leave_lazy(void);
 void xen_post_allocator_init(void);
 
 char * __init xen_memory_setup(void);
Signature · 1 line
1.6.0.6
#6 lore raw

PATCH 6/9 x86/paravirt: allow preemption with lazy mmu mode

In reply to Jeremy Fitzhardinge

From: Jeremy Fitzhardinge <[email protected]>

Lift restrictions on preemption with lazy mmu mode, as it is now allowed.

---
 arch/x86/kernel/paravirt.c |    7 ++++---
 arch/x86/xen/mmu.c         |    8 +-------
 2 files changed, 5 insertions(+), 10 deletions(-)
arch/x86/kernel/paravirt.c +4 -3
index 554dadd..1a81ce9 100644
@@ -248,7 +248,6 @@ static DEFINE_PER_CPU(enum paravirt_lazy_mode, paravirt_lazy_mode) = PARAVIRT_LA
 static inline void enter_lazy(enum paravirt_lazy_mode mode)
 {
 	BUG_ON(__get_cpu_var(paravirt_lazy_mode) != PARAVIRT_LAZY_NONE);
-	BUG_ON(preemptible());
 
 	__get_cpu_var(paravirt_lazy_mode) = mode;
 }
@@ -256,7 +255,6 @@ static inline void enter_lazy(enum paravirt_lazy_mode mode)
 static void leave_lazy(enum paravirt_lazy_mode mode)
 {
 	BUG_ON(__get_cpu_var(paravirt_lazy_mode) != mode);
-	BUG_ON(preemptible());
 
 	__get_cpu_var(paravirt_lazy_mode) = PARAVIRT_LAZY_NONE;
 }
@@ -273,6 +271,8 @@ void paravirt_leave_lazy_mmu(void)
 
 void paravirt_enter_lazy_cpu(void)
 {
+	BUG_ON(preemptible());
+
 	if (percpu_read(paravirt_lazy_mode) == PARAVIRT_LAZY_MMU) {
 		arch_leave_lazy_mmu_mode();
 		set_thread_flag(TIF_LAZY_MMU_UPDATES);
@@ -282,6 +282,8 @@ void paravirt_enter_lazy_cpu(void)
 
 void paravirt_leave_lazy_cpu(void)
 {
+	BUG_ON(preemptible());
+
 	leave_lazy(PARAVIRT_LAZY_CPU);
 
 	if (test_and_clear_thread_flag(TIF_LAZY_MMU_UPDATES))
@@ -301,7 +303,6 @@ void arch_flush_lazy_mmu_mode(void)
 	preempt_disable();
 
 	if (paravirt_get_lazy_mode() == PARAVIRT_LAZY_MMU) {
-		WARN_ON(preempt_count() == 1);
 		arch_leave_lazy_mmu_mode();
 		arch_enter_lazy_mmu_mode();
 	}
arch/x86/xen/mmu.c +1 -7
index 2669c58..92c6463 100644
@@ -412,10 +412,6 @@ void set_pte_mfn(unsigned long vaddr, unsigned long mfn, pgprot_t flags)
 void xen_set_pte_at(struct mm_struct *mm, unsigned long addr,
 		    pte_t *ptep, pte_t pteval)
 {
-	/* updates to init_mm may be done without lock */
-	if (mm == &init_mm)
-		preempt_disable();
-
 	ADD_STATS(set_pte_at, 1);
 //	ADD_STATS(set_pte_at_pinned, xen_page_pinned(ptep));
 	ADD_STATS(set_pte_at_current, mm == current->mm);
@@ -436,9 +432,7 @@ void xen_set_pte_at(struct mm_struct *mm, unsigned long addr,
 	}
 	xen_set_pte(ptep, pteval);
 
-out:
-	if (mm == &init_mm)
-		preempt_enable();
+out:	return;
 }
 
 pte_t xen_ptep_modify_prot_start(struct mm_struct *mm,
Signature · 1 line
1.6.0.6
#7 lore raw

PATCH 7/9 mm: allow preemption in apply_to_pte_range

In reply to Jeremy Fitzhardinge

From: Jeremy Fitzhardinge <[email protected]>

Preemption is now allowed for lazy mmu mode, so don't disable it for the inner loop of apply_to_pte_range.

---
 mm/memory.c |    2 --
 1 files changed, 0 insertions(+), 2 deletions(-)
mm/memory.c +0 -2
index b80cc31..baa999e 100644
@@ -1718,7 +1718,6 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd,
 
 	BUG_ON(pmd_huge(*pmd));
 
-	preempt_disable();
 	arch_enter_lazy_mmu_mode();
 
 	token = pmd_pgtable(*pmd);
@@ -1730,7 +1729,6 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd,
 	} while (pte++, addr += PAGE_SIZE, addr != end);
 
 	arch_leave_lazy_mmu_mode();
-	preempt_enable();
 
 	if (mm != &init_mm)
 		pte_unmap_unlock(pte-1, ptl);
Signature · 1 line
1.6.0.6
#8 lore raw

PATCH 8/9 x86/paravirt: use percpu_ rather than __get_cpu_var

In reply to Jeremy Fitzhardinge

From: Jeremy Fitzhardinge <[email protected]>

percpu_read/write is a slightly more direct way of getting to percpu data.

---
 arch/x86/kernel/paravirt.c |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/kernel/paravirt.c +5 -5
index 1a81ce9..0811730 100644
@@ -247,16 +247,16 @@ static DEFINE_PER_CPU(enum paravirt_lazy_mode, paravirt_lazy_mode) = PARAVIRT_LA
 
 static inline void enter_lazy(enum paravirt_lazy_mode mode)
 {
-	BUG_ON(__get_cpu_var(paravirt_lazy_mode) != PARAVIRT_LAZY_NONE);
+	BUG_ON(percpu_read(paravirt_lazy_mode) != PARAVIRT_LAZY_NONE);
 
-	__get_cpu_var(paravirt_lazy_mode) = mode;
+	percpu_write(paravirt_lazy_mode, mode);
 }
 
 static void leave_lazy(enum paravirt_lazy_mode mode)
 {
-	BUG_ON(__get_cpu_var(paravirt_lazy_mode) != mode);
+	BUG_ON(percpu_read(paravirt_lazy_mode) != mode);
 
-	__get_cpu_var(paravirt_lazy_mode) = PARAVIRT_LAZY_NONE;
+	percpu_write(paravirt_lazy_mode, PARAVIRT_LAZY_NONE);
 }
 
 void paravirt_enter_lazy_mmu(void)
@@ -295,7 +295,7 @@ enum paravirt_lazy_mode paravirt_get_lazy_mode(void)
 	if (in_interrupt())
 		return PARAVIRT_LAZY_NONE;
 
-	return __get_cpu_var(paravirt_lazy_mode);
+	return percpu_read(paravirt_lazy_mode);
 }
 
 void arch_flush_lazy_mmu_mode(void)
Signature · 1 line
1.6.0.6
#9 lore raw

PATCH 9/9 x86/paravirt: finish change from lazy cpu to context switch start/end

In reply to Jeremy Fitzhardinge

From: Jeremy Fitzhardinge <[email protected]>

Pass the previous and next tasks into the context switch start end calls, so that the called functions can properly access the task state (esp in end_context_switch, in which the next task is not yet completely current).

---
 arch/x86/include/asm/paravirt.h |   17 ++++++++++-------
 arch/x86/include/asm/pgtable.h  |    2 ++
 arch/x86/kernel/paravirt.c      |   14 ++++++--------
 arch/x86/kernel/process_32.c    |    2 +-
 arch/x86/kernel/process_64.c    |    2 +-
 arch/x86/kernel/vmi_32.c        |   12 ++++++------
 arch/x86/lguest/boot.c          |    8 ++++----
 arch/x86/xen/enlighten.c        |   10 ++++------
 include/asm-frv/pgtable.h       |    2 +-
 include/asm-generic/pgtable.h   |    2 +-
 kernel/sched.c                  |    2 +-
 11 files changed, 37 insertions(+), 36 deletions(-)
arch/x86/include/asm/paravirt.h +10 -7
index 58d2481..dfdee0c 100644
@@ -56,6 +56,7 @@ struct desc_ptr;
 struct tss_struct;
 struct mm_struct;
 struct desc_struct;
+struct task_struct;
 
 /*
  * Wrapper type for pointers to code which uses the non-standard
@@ -203,7 +204,8 @@ struct pv_cpu_ops {
 
 	void (*swapgs)(void);
 
-	struct pv_lazy_ops lazy_mode;
+	void (*start_context_switch)(struct task_struct *prev);
+	void (*end_context_switch)(struct task_struct *next);
 };
 
 struct pv_irq_ops {
@@ -1414,20 +1416,21 @@ enum paravirt_lazy_mode {
 };
 
 enum paravirt_lazy_mode paravirt_get_lazy_mode(void);
-void paravirt_enter_lazy_cpu(void);
-void paravirt_leave_lazy_cpu(void);
+void paravirt_start_context_switch(struct task_struct *prev);
+void paravirt_end_context_switch(struct task_struct *next);
+
 void paravirt_enter_lazy_mmu(void);
 void paravirt_leave_lazy_mmu(void);
 
 #define  __HAVE_ARCH_START_CONTEXT_SWITCH
-static inline void arch_start_context_switch(void)
+static inline void arch_start_context_switch(struct task_struct *prev)
 {
-	PVOP_VCALL0(pv_cpu_ops.lazy_mode.enter);
+	PVOP_VCALL1(pv_cpu_ops.start_context_switch, prev);
 }
 
-static inline void arch_end_context_switch(void)
+static inline void arch_end_context_switch(struct task_struct *next)
 {
-	PVOP_VCALL0(pv_cpu_ops.lazy_mode.leave);
+	PVOP_VCALL1(pv_cpu_ops.end_context_switch, next);
 }
 
 #define  __HAVE_ARCH_ENTER_LAZY_MMU_MODE
arch/x86/include/asm/pgtable.h +2 -0
index a00628a..d7dfb89 100644
@@ -83,6 +83,8 @@ static inline void __init paravirt_pagetable_setup_done(pgd_t *base)
 #define pte_val(x)	native_pte_val(x)
 #define __pte(x)	native_make_pte(x)
 
+#define arch_end_context_switch(prev)	do {} while(0)
+
 #endif	/* CONFIG_PARAVIRT */
 
 /*
arch/x86/kernel/paravirt.c +6 -8
index 0811730..cb406cb 100644
@@ -269,24 +269,24 @@ void paravirt_leave_lazy_mmu(void)
 	leave_lazy(PARAVIRT_LAZY_MMU);
 }
 
-void paravirt_enter_lazy_cpu(void)
+void paravirt_start_context_switch(struct task_struct *prev)
 {
 	BUG_ON(preemptible());
 
 	if (percpu_read(paravirt_lazy_mode) == PARAVIRT_LAZY_MMU) {
 		arch_leave_lazy_mmu_mode();
-		set_thread_flag(TIF_LAZY_MMU_UPDATES);
+		set_ti_thread_flag(task_thread_info(prev), TIF_LAZY_MMU_UPDATES);
 	}
 	enter_lazy(PARAVIRT_LAZY_CPU);
 }
 
-void paravirt_leave_lazy_cpu(void)
+void paravirt_end_context_switch(struct task_struct *next)
 {
 	BUG_ON(preemptible());
 
 	leave_lazy(PARAVIRT_LAZY_CPU);
 
-	if (test_and_clear_thread_flag(TIF_LAZY_MMU_UPDATES))
+	if (test_and_clear_ti_thread_flag(task_thread_info(next), TIF_LAZY_MMU_UPDATES))
 		arch_enter_lazy_mmu_mode();
 }
 
@@ -401,10 +401,8 @@ struct pv_cpu_ops pv_cpu_ops = {
 	.set_iopl_mask = native_set_iopl_mask,
 	.io_delay = native_io_delay,
 
-	.lazy_mode = {
-		.enter = paravirt_nop,
-		.leave = paravirt_nop,
-	},
+	.start_context_switch = paravirt_nop,
+	.end_context_switch = paravirt_nop,
 };
 
 struct pv_apic_ops pv_apic_ops = {
arch/x86/kernel/process_32.c +1 -1
index 60d7b18..a846b53 100644
@@ -578,7 +578,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)
 	 * done before math_state_restore, so the TS bit is up
 	 * to date.
 	 */
-	arch_end_context_switch();
+	arch_end_context_switch(next_p);
 
 	/* If the task has used fpu the last 5 timeslices, just do a full
 	 * restore of the math state immediately to avoid the trap; the
arch/x86/kernel/process_64.c +1 -1
index d5156e7..64fcb9f 100644
@@ -595,7 +595,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)
 	 * done before math_state_restore, so the TS bit is up
 	 * to date.
 	 */
-	arch_end_context_switch();
+	arch_end_context_switch(next_p);
 
 	/*
 	 * Switch FS and GS.
arch/x86/kernel/vmi_32.c +6 -6
index 950929c..55a5d69 100644
@@ -467,16 +467,16 @@ vmi_startup_ipi_hook(int phys_apicid, unsigned long start_eip,
 }
 #endif
 
-static void vmi_enter_lazy_cpu(void)
+static void vmi_start_context_switch(struct task_struct *prev)
 {
-	paravirt_enter_lazy_cpu();
+	paravirt_start_context_switch(prev);
 	vmi_ops.set_lazy_mode(2);
 }
 
-static void vmi_leave_lazy_cpu(void)
+static void vmi_end_context_switch(struct task_struct *next)
 {
 	vmi_ops.set_lazy_mode(0);
-	paravirt_leave_lazy_cpu();
+	paravirt_end_context_switch(next);
 }
 
 static void vmi_enter_lazy_mmu(void)
@@ -722,9 +722,9 @@ static inline int __init activate_vmi(void)
 	para_fill(pv_cpu_ops.set_iopl_mask, SetIOPLMask);
 	para_fill(pv_cpu_ops.io_delay, IODelay);
 
-	para_wrap(pv_cpu_ops.lazy_mode.enter, vmi_enter_lazy_cpu,
+	para_wrap(pv_cpu_ops.start_context_switch, vmi_start_context_switch,
 		  set_lazy_mode, SetLazyMode);
-	para_wrap(pv_cpu_ops.lazy_mode.leave, vmi_leave_lazy_cpu,
+	para_wrap(pv_cpu_ops.end_context_switch, vmi_end_context_switch,
 		  set_lazy_mode, SetLazyMode);
 
 	para_wrap(pv_mmu_ops.lazy_mode.enter, vmi_enter_lazy_mmu,
arch/x86/lguest/boot.c +4 -4
index 804c170..d70e80d 100644
@@ -153,10 +153,10 @@ static void lguest_leave_lazy_mmu_mode(void)
 	paravirt_leave_lazy_mmu();
 }
 
-static void lguest_leave_lazy_cpu_mode(void)
+static void lguest_end_context_switch(struct task_struct *next)
 {
 	hcall(LHCALL_FLUSH_ASYNC, 0, 0, 0);
-	paravirt_leave_lazy_cpu();
+	paravirt_end_context_switch(next);
 }
 
 /*G:033
@@ -1024,8 +1024,8 @@ __init void lguest_init(void)
 	pv_cpu_ops.write_gdt_entry = lguest_write_gdt_entry;
 	pv_cpu_ops.write_idt_entry = lguest_write_idt_entry;
 	pv_cpu_ops.wbinvd = lguest_wbinvd;
-	pv_cpu_ops.lazy_mode.enter = paravirt_enter_lazy_cpu;
-	pv_cpu_ops.lazy_mode.leave = lguest_leave_lazy_cpu_mode;
+	pv_cpu_ops.start_context_switch = paravirt_start_context_switch;
+	pv_cpu_ops.end_context_switch = lguest_end_context_switch;
 
 	/* pagetable management */
 	pv_mmu_ops.write_cr3 = lguest_write_cr3;
arch/x86/xen/enlighten.c +4 -6
index fc6519a..2af94de 100644
@@ -203,10 +203,10 @@ static unsigned long xen_get_debugreg(int reg)
 	return HYPERVISOR_get_debugreg(reg);
 }
 
-static void xen_leave_lazy_cpu(void)
+static void xen_end_context_switch(struct task_struct *next)
 {
 	xen_mc_flush();
-	paravirt_leave_lazy_cpu();
+	paravirt_end_context_switch(next);
 }
 
 static unsigned long xen_store_tr(void)
@@ -815,10 +815,8 @@ static const struct pv_cpu_ops xen_cpu_ops __initdata = {
 	/* Xen takes care of %gs when switching to usermode for us */
 	.swapgs = paravirt_nop,
 
-	.lazy_mode = {
-		.enter = paravirt_enter_lazy_cpu,
-		.leave = xen_leave_lazy_cpu,
-	},
+	.start_context_switch = paravirt_start_context_switch,
+	.end_context_switch = xen_end_context_switch,
 };
 
 static const struct pv_apic_ops xen_apic_ops __initdata = {
include/asm-frv/pgtable.h +1 -1
index 88bec28..b2107a5 100644
@@ -74,7 +74,7 @@ static inline int pte_file(pte_t pte) { return 0; }
 #define arch_enter_lazy_mmu_mode()	do {} while (0)
 #define arch_leave_lazy_mmu_mode()	do {} while (0)
 
-#define arch_start_context_switch()	do {} while (0)
+#define arch_start_context_switch(prev)	do {} while (0)
 
 #else /* !CONFIG_MMU */
 /*****************************************************************************/
include/asm-generic/pgtable.h +1 -1
index 922f036..e410f60 100644
@@ -291,7 +291,7 @@ static inline void ptep_modify_prot_commit(struct mm_struct *mm,
  * definition.
  */
 #ifndef __HAVE_ARCH_START_CONTEXT_SWITCH
-#define arch_start_context_switch()	do {} while (0)
+#define arch_start_context_switch(prev)	do {} while (0)
 #endif
 
 #ifndef __HAVE_PFNMAP_TRACKING
kernel/sched.c +1 -1
index 25d4c6b..2376a90 100644
@@ -2782,7 +2782,7 @@ context_switch(struct rq *rq, struct task_struct *prev,
 	 * combine the page table reload and the switch backend into
 	 * one hypercall.
 	 */
-	arch_start_context_switch();
+	arch_start_context_switch(prev);
 
 	if (unlikely(!mm)) {
 		next->active_mm = oldmm;
Signature · 1 line
1.6.0.6
maintainer
#10 lore raw

Re: [PATCH 0/9] Make lazy mmu update mode pre-empt safe

In reply to Jeremy Fitzhardinge
Quoted 1 line · On Wed, 18 Feb 2009 14:30:10 -0800 Jeremy Fitzhardinge <[email protected]> wrote:

This series changes the paravirt-ops lazy update modes in the following ways:

glumph. This doesn't look like a 2.6.29 quickfix.

#11 lore raw

Re: [PATCH 0/9] Make lazy mmu update mode pre-empt safe

Quoted 9 lines · Andrew Morton
Quoted 2 lines · On Wed, 18 Feb 2009 14:30:10 -0800 Jeremy Fitzhardinge <[email protected]> wrote:

This series changes the paravirt-ops lazy update modes in the following ways:

glumph. This doesn't look like a 2.6.29 quickfix.

You mean for apply_to_page_range? No, definitely not, but I don't think it has any users wanting to operate on init_mm in 2.6.29.

J

#12 lore raw

Re: [PATCH 0/9] Make lazy mmu update mode pre-empt safe

In reply to Jeremy Fitzhardinge

[ Clearly some aspect of this mailbomb tool is eluding me. ]

Hi,

This series changes the paravirt-ops lazy update modes in the following ways:

1. Make paravirt_get_lazy_mode() always return PARAVIRT_LAZY_NONE in interrupt context. This has the effect of making all mmu and cpu state updates synchronous in interrupt routines, without the need to add explicit flushing.

2. De-generalize lazy cpu mode to a specific context-switch operation. Rather than dealing with fully general enter/leave lazy cpu update mode, make it specifically about start/end context switch, since that was the only use anyway. That means the implementations know that they're in the middle of a context switch.

3. On starting a context switch, if the task is in lazy mmu update mode, record that fact in thread_info->flags TIF_LAZY_MMU_UPDATE, and disable lazy updates (flushing anything pending). On ending a context switch, if the new task has TIF_LAZY_MMU_UPDATE set, then reenter lazy mmu update mode.

4. Remove all the preempt_disable/enable pairs and assertion that preemption must be disabled for lazy mmu updates.

This has the net effect of 1) making lazy mmu updates more robust, and
2) not requiring any changes to the current rules of how init_mm can be updated.

Thanks,
J

#13 lore raw

Re: [PATCH 0/9] Make lazy mmu update mode pre-empt safe

Quoted 1 line · Jeremy Fitzhardinge

[ Clearly some aspect of this mailbomb tool is eluding me. ]

Are you using "git format-patch" and "git send-email" ? Then perhaps the --compose option to git-send-email is what you are after?

Ingo

#14 lore raw

Re: [PATCH 0/9] Make lazy mmu update mode pre-empt safe

Quoted 1 line · Ingo Molnar

Are you using "git format-patch" and "git send-email" ?

Yes.

Quoted 2 lines

Then
perhaps the --compose option to git-send-email is what you are after?

I'm pre-editing the 0000-cover.patch - but it looks like its picking up the 0000-cover.patch~ file in preference to 0000-cover.patch. I think.

J

#15 lore raw

Re: [PATCH 0/9] Make lazy mmu update mode pre-empt safe

Quoted 10 lines · Jeremy Fitzhardinge
Quoted 1 line · Ingo Molnar

Are you using "git format-patch" and "git send-email" ?

Yes.

Quoted 2 lines

Then perhaps the --compose option to git-send-email is what you are
after?

I'm pre-editing the 0000-cover.patch - but it looks like its picking up the 0000-cover.patch~ file in preference to 0000-cover.patch. I think.

try test-sending to yourself until it works :-)

Ingo

#2 lore raw

Re: [PATCH 3/9] x86/paravirt: remove lazy mode in interrupts

In reply to Jeremy Fitzhardinge
Quoted 7 lines · Jeremy Fitzhardinge

From: Jeremy Fitzhardinge <[email protected]>

Make paravirt_lazy_mode() always return PARAVIRT_LAZY_NONE when in an interrupt. This prevents interrupt code from accidentally inheriting an outer lazy state, and instead does everything synchronously. Outer batched operations are left deferred.

This looks like a good solution, but can you make paravirt_lazy_mode() inline? It doesn't need to be a function call.

Zach

#3 lore raw

Re: [PATCH 3/9] x86/paravirt: remove lazy mode in interrupts

Quoted 11 lines · Zachary Amsden
Quoted 8 lines · Jeremy Fitzhardinge

From: Jeremy Fitzhardinge <[email protected]>

Make paravirt_lazy_mode() always return PARAVIRT_LAZY_NONE when in an interrupt. This prevents interrupt code from accidentally inheriting an outer lazy state, and instead does everything synchronously. Outer batched operations are left deferred.

Quoted 2 lines

This looks like a good solution, but can you make paravirt_lazy_mode() inline? It doesn't need to be a function call.

Yeah. I'm not quite sure why it became out of line.

J