Skip to contentSkip to conversation

Runtime paravirt patching

RFC PATCH NAK Ankur Arora 4 participants 42 messages Apr 8, 2020 – Apr 10, 2020

Subsystems (13)
lore
Share & tools
lore
#0 lore raw

cover Runtime paravirt patching

A KVM host (or another hypervisor) might advertise paravirtualized features and optimization hints (ex KVM_HINTS_REALTIME) which might become stale over the lifetime of the guest. For instance, the host might go from being undersubscribed to being oversubscribed (or the other way round) and it would make sense for the guest switch pv-ops based on that.

This lockorture splat that I saw on the guest while testing this is indicative of the problem:

  [ 1136.461522] watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [lock_torture_wr:12865]
  [ 1136.461542] CPU: 8 PID: 12865 Comm: lock_torture_wr Tainted: G W L 5.4.0-rc7+ #77
  [ 1136.461546] RIP: 0010:native_queued_spin_lock_slowpath+0x15/0x220

(Caused by an oversubscribed host but using mismatched native pv_lock_ops on the gues.)

This series addresses the problem by doing paravirt switching at runtime.

We keep an interesting subset of pv-ops (pv_lock_ops only for now, but PV-TLB ops are also good candidates) in .parainstructions.runtime, while discarding the .parainstructions as usual at init. This is then used for switching back and forth between native and paravirt mode. ([1] lists some representative numbers of the increased memory footprint.)

Mechanism: the patching itself is done using stop_machine(). That is not ideal -- text_poke_stop_machine() was replaced with INT3+emulation via text_poke_bp(), but I'm using this to address two issues:
1) emulation in text_poke() can only easily handle a small set of instructions and this is problematic for inlined pv-ops (and see a possible alternatives use-case below.)
2) paravirt patching might have inter-dependendent ops (ex. lock.queued_lock_slowpath, lock.queued_lock_unlock are paired and need to be updated atomically.)

The alternative use-case is a runtime version of apply_alternatives() (not posted with this patch-set) that can be used for some safe subset of X86_FEATUREs. This could be useful in conjunction with the ongoing late microcode loading work that Mihai Carabas and others have been working on.

Also, there are points of similarity with the ongoing static_call work which does rewriting of indirect calls. The difference here is that we need to switch a group of calls atomically and given that some of them can be inlined, need to handle a wider variety of opcodes.

To patch safely we need to satisfy these constraints:

- No references to insn sequences under replacement on any kernel stack once replacement is in progress. Without this constraint we might end up returning to an address that is in the middle of an instruction.

- handle inter-dependent ops: as above, lock.queued_lock_unlock(), lock.queued_lock_slowpath() and the rest of the pv_lock_ops are a good example.

- handle a broader set of insns than CALL and JMP: some pv-ops end up getting inlined. Alternatives can contain arbitrary instructions.

- locking operations can be called from interrupt handlers which means we cannot trivially use IPIs for flushing.

Handling these, necessitates that target pv-ops not be preemptible. Once that is a given (for safety these need to be explicitly whitelisted in runtime_patch()), use a state-machine with the primary CPU doing the patching and secondary CPUs in a sync_core() loop.

In case we hit an INT3/BP (in NMI or thread-context) we makes forward progress by continuing the patching instead of emulating.

One remaining issue is inter-dependent pv-ops which are also executed in the NMI handler -- patching can potentially deadlock in case of multiple NMIs. Handle these by pushing some of this work in the NMI handler where we know it will be uninterrupted.

There are four main sets of patches in this series:

1. PV-ops management (patches 1-10, 20): mostly infrastructure and refactoring pieces to make paravirt patching usable at runtime. For the most part scoped under CONFIG_PARAVIRT_RUNTIME.

Patches 1-7, to persist part of parainstructions in memory:
"x86/paravirt: Specify subsection in PVOP macros" "x86/paravirt: Allow paravirt patching post-init" "x86/paravirt: PVRTOP macros for PARAVIRT_RUNTIME" "x86/alternatives: Refactor alternatives_smp_module* "x86/alternatives: Rename alternatives_smp*, smp_alt_module "x86/alternatives: Remove stale symbols "x86/paravirt: Persist .parainstructions.runtime"

Patches 8-10, develop the inerfaces to safely switch pv-ops:
"x86/paravirt: Stash native pv-ops" "x86/paravirt: Add runtime_patch()" "x86/paravirt: Add primitives to stage pv-ops"

Patch 20 enables switching of pv_lock_ops:
"x86/paravirt: Enable pv-spinlocks in runtime_patch()"

2. Non-emulated text poking (patches 11-19)

Patches 11-13 are mostly refactoring to split __text_poke() into map, unmap and poke/memcpy phases with the poke portion being re-entrant
"x86/alternatives: Remove return value of text_poke*()" "x86/alternatives: Use __get_unlocked_pte() in text_poke()" "x86/alternatives: Split __text_poke()"

Patches 15, 17 add the actual poking state-machine:
"x86/alternatives: Non-emulated text poking" "x86/alternatives: Add patching logic in text_poke_site()"

with patches 14 and 18 containing the pieces for BP handling:
"x86/alternatives: Handle native insns in text_poke_loc*()" "x86/alternatives: Handle BP in non-emulated text poking"

and patch 19 provides the ability to use the state-machine above in an NMI context (fixes some potential deadlocks when handling inter- dependent operations and multiple NMIs):
"x86/alternatives: NMI safe runtime patching".

Patch 16 provides the interface (paravirt_runtime_patch()) to use the poking mechanism developed above and patch 21 adds a selftest:
"x86/alternatives: Add paravirt patching at runtime" "x86/alternatives: Paravirt runtime selftest"

3. KVM guest changes to be able to use this (patches 22-23,25-26):
"kvm/paravirt: Encapsulate KVM pv switching logic" "x86/kvm: Add worker to trigger runtime patching" "x86/kvm: Guest support for dynamic hints"
"x86/kvm: Add hint change notifier for KVM_HINT_REALTIME".

4. KVM host changes to notify the guest of a change (patch 24):
"x86/kvm: Support dynamic CPUID hints"

Testing:
With paravirt patching, the code is mostly stable on Intel and AMD systems under kernbench and locktorture with paravirt toggling (with, without synthetic NMIs) in the background.

Queued spinlock performance for locktorture is also on expected lines:

 [ 1533.221563] Writes:  Total: 1048759000  Max/Min: 0/0   Fail: 0 
 # toggle PV spinlocks
 [ 1594.713699] Writes:  Total: 1111660545  Max/Min: 0/0   Fail: 0 
 # PV spinlocks (in ~60 seconds) = 62,901,545
 # toggle native spinlocks
 [ 1656.117175] Writes:  Total: 1113888840  Max/Min: 0/0   Fail: 0 
  # native spinlocks (in ~60 seconds) = 2,228,295

The alternatives testing is more limited with it being used to rewrite mostly harmless X86_FEATUREs with load in the background.

Patches also at:

ssh://[email protected]/terminus/linux.git alternatives-rfc-upstream-v1

Please review.

Thanks
Ankur

[1] The precise change in memory footprint depends on config options but the following example inlines queued_spin_unlock() (which forms the bulk of the added state). The added footprint is the size of the .parainstructions.runtime section:

 $ objdump -h vmlinux|grep .parainstructions
 Idx Name              		Size      VMA               
 	LMA                File-off  Algn
  27 .parainstructions 		0001013c  ffffffff82895000
  	0000000002895000   01c95000  2**3
  28 .parainstructions.runtime  0000cd2c  ffffffff828a5140
  	00000000028a5140   01ca5140  2**3
  $ size vmlinux                                         
  text       data       bss        dec      hex       filename
  13726196   12302814   14094336   40123346 2643bd2   vmlinux

Ankur Arora (26):
x86/paravirt: Specify subsection in PVOP macros x86/paravirt: Allow paravirt patching post-init x86/paravirt: PVRTOP macros for PARAVIRT_RUNTIME x86/alternatives: Refactor alternatives_smp_module* x86/alternatives: Rename alternatives_smp*, smp_alt_module x86/alternatives: Remove stale symbols x86/paravirt: Persist .parainstructions.runtime x86/paravirt: Stash native pv-ops
x86/paravirt: Add runtime_patch()
x86/paravirt: Add primitives to stage pv-ops x86/alternatives: Remove return value of text_poke*() x86/alternatives: Use __get_unlocked_pte() in text_poke() x86/alternatives: Split __text_poke() x86/alternatives: Handle native insns in text_poke_loc*() x86/alternatives: Non-emulated text poking x86/alternatives: Add paravirt patching at runtime x86/alternatives: Add patching logic in text_poke_site() x86/alternatives: Handle BP in non-emulated text poking x86/alternatives: NMI safe runtime patching x86/paravirt: Enable pv-spinlocks in runtime_patch() x86/alternatives: Paravirt runtime selftest kvm/paravirt: Encapsulate KVM pv switching logic x86/kvm: Add worker to trigger runtime patching x86/kvm: Support dynamic CPUID hints
x86/kvm: Guest support for dynamic hints x86/kvm: Add hint change notifier for KVM_HINT_REALTIME

 Documentation/virt/kvm/api.rst        |  17 +
 Documentation/virt/kvm/cpuid.rst      |   9 +-
 arch/x86/Kconfig                      |  14 +
 arch/x86/Kconfig.debug                |  13 +
 arch/x86/entry/entry_64.S             |   5 +
 arch/x86/include/asm/alternative.h    |  20 +-
 arch/x86/include/asm/kvm_host.h       |   6 +
 arch/x86/include/asm/kvm_para.h       |  17 +
 arch/x86/include/asm/paravirt.h       |  10 +-
 arch/x86/include/asm/paravirt_types.h | 230 ++++--
 arch/x86/include/asm/text-patching.h  |  18 +-
 arch/x86/include/uapi/asm/kvm_para.h  |   2 +
 arch/x86/kernel/Makefile              |   1 +
 arch/x86/kernel/alternative.c         | 987 +++++++++++++++++++++++---
 arch/x86/kernel/kvm.c                 | 191 ++++-
 arch/x86/kernel/module.c              |  42 +-
 arch/x86/kernel/paravirt.c            |  16 +-
 arch/x86/kernel/paravirt_patch.c      |  61 ++
 arch/x86/kernel/pv_selftest.c         | 264 +++++++
 arch/x86/kernel/pv_selftest.h         |  15 +
 arch/x86/kernel/setup.c               |   2 +
 arch/x86/kernel/vmlinux.lds.S         |  16 +
 arch/x86/kvm/cpuid.c                  |   3 +-
 arch/x86/kvm/x86.c                    |  39 +
 include/asm-generic/kvm_para.h        |  12 +
 include/asm-generic/vmlinux.lds.h     |   8 +
 include/linux/kvm_para.h              |   5 +
 include/linux/mm.h                    |  16 +-
 include/linux/preempt.h               |  17 +
 include/uapi/linux/kvm.h              |   4 +
 kernel/locking/lock_events.c          |   2 +-
 mm/memory.c                           |   9 +-
 32 files changed, 1850 insertions(+), 221 deletions(-)
 create mode 100644 arch/x86/kernel/pv_selftest.c
 create mode 100644 arch/x86/kernel/pv_selftest.h
Signature · 1 line
2.20.1
#1 lore raw

RFC PATCH 1/26 x86/paravirt: Specify subsection in PVOP macros

Allow PVOP macros to specify a subsection such that _paravirt_alt() can optionally put sites in .parainstructions.*.

---
 arch/x86/include/asm/paravirt_types.h | 158 +++++++++++++++++---------
 1 file changed, 102 insertions(+), 56 deletions(-)
arch/x86/include/asm/paravirt_types.h +102 -56
index 732f62e04ddb..37e8f27a3b9d 100644
@@ -337,6 +337,9 @@ struct paravirt_patch_template {
 extern struct pv_info pv_info;
 extern struct paravirt_patch_template pv_ops;
 
+/* Sub-section for .parainstructions */
+#define PV_SUFFIX ""
+
 #define PARAVIRT_PATCH(x)					\
 	(offsetof(struct paravirt_patch_template, x) / sizeof(void *))
 
@@ -350,9 +353,9 @@ extern struct paravirt_patch_template pv_ops;
  * Generate some code, and mark it as patchable by the
  * apply_paravirt() alternate instruction patcher.
  */
-#define _paravirt_alt(insn_string, type, clobber)	\
+#define _paravirt_alt(sec, insn_string, type, clobber)	\
 	"771:\n\t" insn_string "\n" "772:\n"		\
-	".pushsection .parainstructions,\"a\"\n"	\
+	".pushsection .parainstructions" sec ",\"a\"\n"	\
 	_ASM_ALIGN "\n"					\
 	_ASM_PTR " 771b\n"				\
 	"  .byte " type "\n"				\
@@ -361,8 +364,9 @@ extern struct paravirt_patch_template pv_ops;
 	".popsection\n"
 
 /* Generate patchable code, with the default asm parameters. */
-#define paravirt_alt(insn_string)					\
-	_paravirt_alt(insn_string, "%c[paravirt_typenum]", "%c[paravirt_clobber]")
+#define paravirt_alt(sec, insn_string)					\
+	_paravirt_alt(sec, insn_string, "%c[paravirt_typenum]",		\
+		      "%c[paravirt_clobber]")
 
 /* Simple instruction patching code. */
 #define NATIVE_LABEL(a,x,b) "\n\t.globl " a #x "_" #b "\n" a #x "_" #b ":\n\t"
@@ -414,7 +418,7 @@ int paravirt_disable_iospace(void);
  * unfortunately, are quite a bit (r8 - r11)
  *
  * The call instruction itself is marked by placing its start address
- * and size into the .parainstructions section, so that
+ * and size into the .parainstructions* sections, so that
  * apply_paravirt() in arch/i386/kernel/alternative.c can do the
  * appropriate patching under the control of the backend pv_init_ops
  * implementation.
@@ -512,7 +516,7 @@ int paravirt_disable_iospace(void);
 	})
 
 
-#define ____PVOP_CALL(rettype, op, clbr, call_clbr, extra_clbr,		\
+#define ____PVOP_CALL(sec, rettype, op, clbr, call_clbr, extra_clbr,	\
 		      pre, post, ...)					\
 	({								\
 		rettype __ret;						\
@@ -522,7 +526,7 @@ int paravirt_disable_iospace(void);
 		/* since this condition will never hold */		\
 		if (sizeof(rettype) > sizeof(unsigned long)) {		\
 			asm volatile(pre				\
-				     paravirt_alt(PARAVIRT_CALL)	\
+				     paravirt_alt(sec, PARAVIRT_CALL)	\
 				     post				\
 				     : call_clbr, ASM_CALL_CONSTRAINT	\
 				     : paravirt_type(op),		\
@@ -532,7 +536,7 @@ int paravirt_disable_iospace(void);
 			__ret = (rettype)((((u64)__edx) << 32) | __eax); \
 		} else {						\
 			asm volatile(pre				\
-				     paravirt_alt(PARAVIRT_CALL)	\
+				     paravirt_alt(sec, PARAVIRT_CALL)	\
 				     post				\
 				     : call_clbr, ASM_CALL_CONSTRAINT	\
 				     : paravirt_type(op),		\
@@ -544,22 +548,22 @@ int paravirt_disable_iospace(void);
 		__ret;							\
 	})
 
-#define __PVOP_CALL(rettype, op, pre, post, ...)			\
-	____PVOP_CALL(rettype, op, CLBR_ANY, PVOP_CALL_CLOBBERS,	\
+#define __PVOP_CALL(sec, rettype, op, pre, post, ...)			\
+	____PVOP_CALL(sec, rettype, op, CLBR_ANY, PVOP_CALL_CLOBBERS,	\
 		      EXTRA_CLOBBERS, pre, post, ##__VA_ARGS__)
 
-#define __PVOP_CALLEESAVE(rettype, op, pre, post, ...)			\
-	____PVOP_CALL(rettype, op.func, CLBR_RET_REG,			\
+#define __PVOP_CALLEESAVE(sec, rettype, op, pre, post, ...)		\
+	____PVOP_CALL(sec, rettype, op.func, CLBR_RET_REG,		\
 		      PVOP_CALLEE_CLOBBERS, ,				\
 		      pre, post, ##__VA_ARGS__)
 
 
-#define ____PVOP_VCALL(op, clbr, call_clbr, extra_clbr, pre, post, ...)	\
+#define ____PVOP_VCALL(sec, op, clbr, call_clbr, extra_clbr, pre, post, ...)	\
 	({								\
 		PVOP_VCALL_ARGS;					\
 		PVOP_TEST_NULL(op);					\
 		asm volatile(pre					\
-			     paravirt_alt(PARAVIRT_CALL)		\
+			     paravirt_alt(sec, PARAVIRT_CALL)		\
 			     post					\
 			     : call_clbr, ASM_CALL_CONSTRAINT		\
 			     : paravirt_type(op),			\
@@ -568,85 +572,127 @@ int paravirt_disable_iospace(void);
 			     : "memory", "cc" extra_clbr);		\
 	})
 
-#define __PVOP_VCALL(op, pre, post, ...)				\
-	____PVOP_VCALL(op, CLBR_ANY, PVOP_VCALL_CLOBBERS,		\
+#define __PVOP_VCALL(sec, op, pre, post, ...)				\
+	____PVOP_VCALL(sec, op, CLBR_ANY, PVOP_VCALL_CLOBBERS,		\
 		       VEXTRA_CLOBBERS,					\
 		       pre, post, ##__VA_ARGS__)
 
-#define __PVOP_VCALLEESAVE(op, pre, post, ...)				\
-	____PVOP_VCALL(op.func, CLBR_RET_REG,				\
+#define __PVOP_VCALLEESAVE(sec, op, pre, post, ...)			\
+	____PVOP_VCALL(sec, op.func, CLBR_RET_REG,			\
 		      PVOP_VCALLEE_CLOBBERS, ,				\
 		      pre, post, ##__VA_ARGS__)
 
 
 
-#define PVOP_CALL0(rettype, op)						\
-	__PVOP_CALL(rettype, op, "", "")
-#define PVOP_VCALL0(op)							\
-	__PVOP_VCALL(op, "", "")
+#define _PVOP_CALL0(sec, rettype, op)					\
+	__PVOP_CALL(sec, rettype, op, "", "")
+#define _PVOP_VCALL0(sec, op)						\
+	__PVOP_VCALL(sec, op, "", "")
 
-#define PVOP_CALLEE0(rettype, op)					\
-	__PVOP_CALLEESAVE(rettype, op, "", "")
-#define PVOP_VCALLEE0(op)						\
-	__PVOP_VCALLEESAVE(op, "", "")
+#define _PVOP_CALLEE0(sec, rettype, op)					\
+	__PVOP_CALLEESAVE(sec, rettype, op, "", "")
+#define _PVOP_VCALLEE0(sec, op)						\
+	__PVOP_VCALLEESAVE(sec, op, "", "")
 
 
-#define PVOP_CALL1(rettype, op, arg1)					\
-	__PVOP_CALL(rettype, op, "", "", PVOP_CALL_ARG1(arg1))
-#define PVOP_VCALL1(op, arg1)						\
-	__PVOP_VCALL(op, "", "", PVOP_CALL_ARG1(arg1))
+#define _PVOP_CALL1(sec, rettype, op, arg1)				\
+	__PVOP_CALL(sec, rettype, op, "", "", PVOP_CALL_ARG1(arg1))
+#define _PVOP_VCALL1(sec, op, arg1)					\
+	__PVOP_VCALL(sec, op, "", "", PVOP_CALL_ARG1(arg1))
 
-#define PVOP_CALLEE1(rettype, op, arg1)					\
-	__PVOP_CALLEESAVE(rettype, op, "", "", PVOP_CALL_ARG1(arg1))
-#define PVOP_VCALLEE1(op, arg1)						\
-	__PVOP_VCALLEESAVE(op, "", "", PVOP_CALL_ARG1(arg1))
+#define _PVOP_CALLEE1(sec, rettype, op, arg1)				\
+	__PVOP_CALLEESAVE(sec, rettype, op, "", "", PVOP_CALL_ARG1(arg1))
+#define _PVOP_VCALLEE1(sec, op, arg1)					\
+	__PVOP_VCALLEESAVE(sec, op, "", "", PVOP_CALL_ARG1(arg1))
 
-
-#define PVOP_CALL2(rettype, op, arg1, arg2)				\
-	__PVOP_CALL(rettype, op, "", "", PVOP_CALL_ARG1(arg1),		\
+#define _PVOP_CALL2(sec, rettype, op, arg1, arg2)			\
+	__PVOP_CALL(sec, rettype, op, "", "", PVOP_CALL_ARG1(arg1),	\
 		    PVOP_CALL_ARG2(arg2))
-#define PVOP_VCALL2(op, arg1, arg2)					\
-	__PVOP_VCALL(op, "", "", PVOP_CALL_ARG1(arg1),			\
+#define _PVOP_VCALL2(sec, op, arg1, arg2)				\
+	__PVOP_VCALL(sec, op, "", "", PVOP_CALL_ARG1(arg1),		\
 		     PVOP_CALL_ARG2(arg2))
 
-#define PVOP_CALLEE2(rettype, op, arg1, arg2)				\
-	__PVOP_CALLEESAVE(rettype, op, "", "", PVOP_CALL_ARG1(arg1),	\
+#define _PVOP_CALLEE2(sec, rettype, op, arg1, arg2)			\
+	__PVOP_CALLEESAVE(sec, rettype, op, "", "", PVOP_CALL_ARG1(arg1), \
 			  PVOP_CALL_ARG2(arg2))
-#define PVOP_VCALLEE2(op, arg1, arg2)					\
-	__PVOP_VCALLEESAVE(op, "", "", PVOP_CALL_ARG1(arg1),		\
+#define _PVOP_VCALLEE2(sec, op, arg1, arg2)				\
+	__PVOP_VCALLEESAVE(sec, op, "", "", PVOP_CALL_ARG1(arg1),	\
 			   PVOP_CALL_ARG2(arg2))
 
 
-#define PVOP_CALL3(rettype, op, arg1, arg2, arg3)			\
-	__PVOP_CALL(rettype, op, "", "", PVOP_CALL_ARG1(arg1),		\
+#define _PVOP_CALL3(sec, rettype, op, arg1, arg2, arg3)			\
+	__PVOP_CALL(sec, rettype, op, "", "", PVOP_CALL_ARG1(arg1),	\
 		    PVOP_CALL_ARG2(arg2), PVOP_CALL_ARG3(arg3))
-#define PVOP_VCALL3(op, arg1, arg2, arg3)				\
-	__PVOP_VCALL(op, "", "", PVOP_CALL_ARG1(arg1),			\
+#define _PVOP_VCALL3(sec, op, arg1, arg2, arg3)				\
+	__PVOP_VCALL(sec, op, "", "", PVOP_CALL_ARG1(arg1),		\
 		     PVOP_CALL_ARG2(arg2), PVOP_CALL_ARG3(arg3))
 
 /* This is the only difference in x86_64. We can make it much simpler */
 #ifdef CONFIG_X86_32
-#define PVOP_CALL4(rettype, op, arg1, arg2, arg3, arg4)			\
-	__PVOP_CALL(rettype, op,					\
+#define _PVOP_CALL4(sec, rettype, op, arg1, arg2, arg3, arg4)		\
+	__PVOP_CALL(sec, rettype, op,					\
 		    "push %[_arg4];", "lea 4(%%esp),%%esp;",		\
 		    PVOP_CALL_ARG1(arg1), PVOP_CALL_ARG2(arg2),		\
 		    PVOP_CALL_ARG3(arg3), [_arg4] "mr" ((u32)(arg4)))
-#define PVOP_VCALL4(op, arg1, arg2, arg3, arg4)				\
-	__PVOP_VCALL(op,						\
+#define _PVOP_VCALL4(sec, op, arg1, arg2, arg3, arg4)			\
+	__PVOP_VCALL(sec, op,						\
 		    "push %[_arg4];", "lea 4(%%esp),%%esp;",		\
 		    "0" ((u32)(arg1)), "1" ((u32)(arg2)),		\
 		    "2" ((u32)(arg3)), [_arg4] "mr" ((u32)(arg4)))
 #else
-#define PVOP_CALL4(rettype, op, arg1, arg2, arg3, arg4)			\
-	__PVOP_CALL(rettype, op, "", "",				\
+#define _PVOP_CALL4(sec, rettype, op, arg1, arg2, arg3, arg4)		\
+	__PVOP_CALL(sec, rettype, op, "", "",				\
 		    PVOP_CALL_ARG1(arg1), PVOP_CALL_ARG2(arg2),		\
 		    PVOP_CALL_ARG3(arg3), PVOP_CALL_ARG4(arg4))
-#define PVOP_VCALL4(op, arg1, arg2, arg3, arg4)				\
-	__PVOP_VCALL(op, "", "",					\
+#define _PVOP_VCALL4(sec, op, arg1, arg2, arg3, arg4)			\
+	__PVOP_VCALL(sec, op, "", "",					\
 		     PVOP_CALL_ARG1(arg1), PVOP_CALL_ARG2(arg2),	\
 		     PVOP_CALL_ARG3(arg3), PVOP_CALL_ARG4(arg4))
 #endif
 
+/*
+ * PVOP macros for .parainstructions
+ */
+#define PVOP_CALL0(rettype, op)						\
+	_PVOP_CALL0(PV_SUFFIX, rettype, op)
+#define PVOP_VCALL0(op)							\
+	_PVOP_VCALL0(PV_SUFFIX, op)
+
+#define PVOP_CALLEE0(rettype, op)					\
+	_PVOP_CALLEE0(PV_SUFFIX, rettype, op)
+#define PVOP_VCALLEE0(op)						\
+	_PVOP_VCALLEE0(PV_SUFFIX, op)
+
+#define PVOP_CALL1(rettype, op, arg1)					\
+	_PVOP_CALL1(PV_SUFFIX, rettype, op, arg1)
+#define PVOP_VCALL1(op, arg1)						\
+	_PVOP_VCALL1(PV_SUFFIX, op, arg1)
+
+#define PVOP_CALLEE1(rettype, op, arg1)					\
+	_PVOP_CALLEE1(PV_SUFFIX, rettype, op, arg1)
+#define PVOP_VCALLEE1(op, arg1)						\
+	_PVOP_VCALLEE1(PV_SUFFIX, op, arg1)
+
+#define PVOP_CALL2(rettype, op, arg1, arg2)				\
+	_PVOP_CALL2(PV_SUFFIX, rettype, op, arg1, arg2)
+#define PVOP_VCALL2(op, arg1, arg2)					\
+	_PVOP_VCALL2(PV_SUFFIX, op, arg1, arg2)
+
+#define PVOP_CALLEE2(rettype, op, arg1, arg2)				\
+	_PVOP_CALLEE2(PV_SUFFIX, rettype, op, arg1, arg2)
+#define PVOP_VCALLEE2(op, arg1, arg2)					\
+	_PVOP_VCALLEE2(PV_SUFFIX, op, arg1, arg2)
+
+#define PVOP_CALL3(rettype, op, arg1, arg2, arg3)			\
+	_PVOP_CALL3(PV_SUFFIX, rettype, op, arg1, arg2, arg3)
+#define PVOP_VCALL3(op, arg1, arg2, arg3)				\
+	_PVOP_VCALL3(PV_SUFFIX, op, arg1, arg2, arg3)
+
+#define PVOP_CALL4(rettype, op, arg1, arg2, arg3, arg4)			\
+	_PVOP_CALL4(PV_SUFFIX, rettype, op, arg1, arg2, arg3, arg4)
+#define PVOP_VCALL4(op, arg1, arg2, arg3, arg4)				\
+	_PVOP_VCALL4(PV_SUFFIX, op, arg1, arg2, arg3, arg4)
+
 /* Lazy mode for batching updates / context switch */
 enum paravirt_lazy_mode {
 	PARAVIRT_LAZY_NONE,
@@ -667,7 +713,7 @@ u64 _paravirt_ident_64(u64);
 
 #define paravirt_nop	((void *)_paravirt_nop)
 
-/* These all sit in the .parainstructions section to tell us what to patch. */
+/* These all sit in .parainstructions* sections to tell us what to patch. */
 struct paravirt_patch_site {
 	u8 *instr;		/* original instructions */
 	u8 type;		/* type of this instruction */
Signature · 1 line
2.20.1
#2 lore raw

RFC PATCH 2/26 x86/paravirt: Allow paravirt patching post-init

In reply to Ankur Arora

Paravirt-ops are patched at init to convert indirect calls into direct calls and in some cases, to inline the target at the call-site. This is done by way of PVOP* macros which save the call-site information via compile time annotations.

Pull this state out in .parainstructions.runtime for some pv-ops such that they can be used for runtime patching.

---
 arch/x86/Kconfig                      | 12 ++++++++++++
 arch/x86/include/asm/paravirt_types.h |  5 +++++
 arch/x86/include/asm/text-patching.h  |  5 +++++
 arch/x86/kernel/alternative.c         |  2 ++
 arch/x86/kernel/module.c              | 10 +++++++++-
 arch/x86/kernel/vmlinux.lds.S         | 16 ++++++++++++++++
 include/asm-generic/vmlinux.lds.h     |  8 ++++++++
 7 files changed, 57 insertions(+), 1 deletion(-)
arch/x86/Kconfig +12 -0
index 1edf788d301c..605619938f08 100644
@@ -764,6 +764,18 @@ config PARAVIRT
 	  over full virtualization.  However, when run without a hypervisor
 	  the kernel is theoretically slower and slightly larger.
 
+config PARAVIRT_RUNTIME
+	bool "Enable paravirtualized ops to be patched at runtime"
+	depends on PARAVIRT
+	help
+	  Enable the paravirtualized guest kernel to switch pv-ops based on
+	  changed host conditions, potentially improving performance
+	  significantly.
+
+	  This would increase the memory footprint of the running kernel
+	  slightly (depending mostly on whether lock and unlock are inlined
+	  or not.)
+
 config PARAVIRT_XXL
 	bool
 
arch/x86/include/asm/paravirt_types.h +5 -0
index 37e8f27a3b9d..00e4a062ca10 100644
@@ -723,6 +723,11 @@ struct paravirt_patch_site {
 extern struct paravirt_patch_site __parainstructions[],
 	__parainstructions_end[];
 
+#ifdef CONFIG_PARAVIRT_RUNTIME
+extern struct paravirt_patch_site __parainstructions_runtime[],
+	__parainstructions_runtime_end[];
+#endif
+
 #endif	/* __ASSEMBLY__ */
 
 #endif	/* _ASM_X86_PARAVIRT_TYPES_H */
arch/x86/include/asm/text-patching.h +5 -0
index 67315fa3956a..e2ef241c261e 100644
@@ -18,6 +18,11 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
 #define __parainstructions_end	NULL
 #endif
 
+#ifndef CONFIG_PARAVIRT_RUNTIME
+#define __parainstructions_runtime	NULL
+#define __parainstructions_runtime_end	NULL
+#endif
+
 /*
  * Currently, the max observed size in the kernel code is
  * JUMP_LABEL_NOP_SIZE/RELATIVEJUMP_SIZE, which are 5.
arch/x86/kernel/alternative.c +2 -0
index 7867dfb3963e..fdfda1375f82 100644
@@ -740,6 +740,8 @@ void __init alternative_instructions(void)
 #endif
 
 	apply_paravirt(__parainstructions, __parainstructions_end);
+	apply_paravirt(__parainstructions_runtime,
+		       __parainstructions_runtime_end);
 
 	restart_nmi();
 	alternatives_patched = 1;
arch/x86/kernel/module.c +9 -1
index d5c72cb877b3..658ea60ce324 100644
@@ -222,7 +222,7 @@ int module_finalize(const Elf_Ehdr *hdr,
 		    struct module *me)
 {
 	const Elf_Shdr *s, *text = NULL, *alt = NULL, *locks = NULL,
-		*para = NULL, *orc = NULL, *orc_ip = NULL;
+		*para = NULL, *para_run = NULL, *orc = NULL, *orc_ip = NULL;
 	char *secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset;
 
 	for (s = sechdrs; s < sechdrs + hdr->e_shnum; s++) {
@@ -234,6 +234,9 @@ int module_finalize(const Elf_Ehdr *hdr,
 			locks = s;
 		if (!strcmp(".parainstructions", secstrings + s->sh_name))
 			para = s;
+		if (!strcmp(".parainstructions.runtime",
+			    secstrings + s->sh_name))
+			para_run = s;
 		if (!strcmp(".orc_unwind", secstrings + s->sh_name))
 			orc = s;
 		if (!strcmp(".orc_unwind_ip", secstrings + s->sh_name))
@@ -257,6 +260,11 @@ int module_finalize(const Elf_Ehdr *hdr,
 		void *pseg = (void *)para->sh_addr;
 		apply_paravirt(pseg, pseg + para->sh_size);
 	}
+	if (para_run) {
+		void *pseg = (void *)para_run->sh_addr;
+
+		apply_paravirt(pseg, pseg + para_run->sh_size);
+	}
 
 	/* make jump label nops */
 	jump_label_apply_nops(me);
arch/x86/kernel/vmlinux.lds.S +16 -0
index 1bf7e312361f..7f5b8f6ab96e 100644
@@ -269,6 +269,7 @@ SECTIONS
 	.parainstructions : AT(ADDR(.parainstructions) - LOAD_OFFSET) {
 		__parainstructions = .;
 		*(.parainstructions)
+		PARAVIRT_DISCARD(.parainstructions.runtime)
 		__parainstructions_end = .;
 	}
 
@@ -348,6 +349,21 @@ SECTIONS
 		__smp_locks_end = .;
 	}
 
+#ifdef CONFIG_PARAVIRT_RUNTIME
+	/*
+	 * .parainstructions.runtime sticks around in memory after
+	 * init so it doesn't need to be page-aligned but everything
+	 * around us is so we will be too.
+	 */
+	. = ALIGN(8);
+	.parainstructions.runtime : AT(ADDR(.parainstructions.runtime) - \
+								LOAD_OFFSET) {
+		__parainstructions_runtime = .;
+		PARAVIRT_KEEP(.parainstructions.runtime)
+		__parainstructions_runtime_end = .;
+	}
+#endif
+
 #ifdef CONFIG_X86_64
 	.data_nosave : AT(ADDR(.data_nosave) - LOAD_OFFSET) {
 		NOSAVE_DATA
include/asm-generic/vmlinux.lds.h +8 -0
index 71e387a5fe90..6b009d5ce51f 100644
@@ -135,6 +135,14 @@
 #define MEM_DISCARD(sec) *(.mem##sec)
 #endif
 
+#if defined(CONFIG_PARAVIRT_RUNTIME)
+#define PARAVIRT_KEEP(sec)	*(sec)
+#define PARAVIRT_DISCARD(sec)
+#else
+#define PARAVIRT_KEEP(sec)
+#define PARAVIRT_DISCARD(sec)	*(sec)
+#endif
+
 #ifdef CONFIG_FTRACE_MCOUNT_RECORD
 /*
  * The ftrace call sites are logged to a section whose name depends on the
Signature · 1 line
2.20.1
#3 lore raw

RFC PATCH 3/26 x86/paravirt: PVRTOP macros for PARAVIRT_RUNTIME

In reply to Ankur Arora

Define PVRT* macros which can be used to put pv-ops in .parainstructions.runtime.

---
 arch/x86/include/asm/paravirt_types.h | 49 +++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)
arch/x86/include/asm/paravirt_types.h +49 -0
index 00e4a062ca10..f1153f53c529 100644
@@ -337,6 +337,12 @@ struct paravirt_patch_template {
 extern struct pv_info pv_info;
 extern struct paravirt_patch_template pv_ops;
 
+#ifdef CONFIG_PARAVIRT_RUNTIME
+#define PVRT_SUFFIX ".runtime"
+#else
+#define PVRT_SUFFIX ""
+#endif
+
 /* Sub-section for .parainstructions */
 #define PV_SUFFIX ""
 
@@ -693,6 +699,49 @@ int paravirt_disable_iospace(void);
 #define PVOP_VCALL4(op, arg1, arg2, arg3, arg4)				\
 	_PVOP_VCALL4(PV_SUFFIX, op, arg1, arg2, arg3, arg4)
 
+/*
+ * PVRTOP macros for .parainstructions.runtime
+ */
+#define PVRTOP_CALL0(rettype, op)					\
+	_PVOP_CALL0(PVRT_SUFFIX, rettype, op)
+#define PVRTOP_VCALL0(op)						\
+	_PVOP_VCALL0(PVRT_SUFFIX, op)
+
+#define PVRTOP_CALLEE0(rettype, op)					\
+	_PVOP_CALLEE0(PVRT_SUFFIX, rettype, op)
+#define PVRTOP_VCALLEE0(op)						\
+	_PVOP_VCALLEE0(PVRT_SUFFIX, op)
+
+#define PVRTOP_CALL1(rettype, op, arg1)					\
+	_PVOP_CALL1(PVRT_SUFFIX, rettype, op, arg1)
+#define PVRTOP_VCALL1(op, arg1)						\
+	_PVOP_VCALL1(PVRT_SUFFIX, op, arg1)
+
+#define PVRTOP_CALLEE1(rettype, op, arg1)				\
+	_PVOP_CALLEE1(PVRT_SUFFIX, rettype, op, arg1)
+#define PVRTOP_VCALLEE1(op, arg1)					\
+	_PVOP_VCALLEE1(PVRT_SUFFIX, op, arg1)
+
+#define PVRTOP_CALL2(rettype, op, arg1, arg2)				\
+	_PVOP_CALL2(PVRT_SUFFIX, rettype, op, arg1, arg2)
+#define PVRTOP_VCALL2(op, arg1, arg2)					\
+	_PVOP_VCALL2(PVRT_SUFFIX, op, arg1, arg2)
+
+#define PVRTOP_CALLEE2(rettype, op, arg1, arg2)				\
+	_PVOP_CALLEE2(PVRT_SUFFIX, rettype, op, arg1, arg2)
+#define PVRTOP_VCALLEE2(op, arg1, arg2)					\
+	_PVOP_VCALLEE2(PVRT_SUFFIX, op, arg1, arg2)
+
+#define PVRTOP_CALL3(rettype, op, arg1, arg2, arg3)			\
+	_PVOP_CALL3(PVRT_SUFFIX, rettype, op, arg1, arg2, arg3)
+#define PVRTOP_VCALL3(op, arg1, arg2, arg3)				\
+	_PVOP_VCALL3(PVRT_SUFFIX, op, arg1, arg2, arg3)
+
+#define PVRTOP_CALL4(rettype, op, arg1, arg2, arg3, arg4)		\
+	_PVOP_CALL4(PVRT_SUFFIX, rettype, op, arg1, arg2, arg3, arg4)
+#define PVRTOP_VCALL4(op, arg1, arg2, arg3, arg4)			\
+	_PVOP_VCALL4(PVRT_SUFFIX, op, arg1, arg2, arg3, arg4)
+
 /* Lazy mode for batching updates / context switch */
 enum paravirt_lazy_mode {
 	PARAVIRT_LAZY_NONE,
Signature · 1 line
2.20.1
#4 lore raw

RFC PATCH 4/26 x86/alternatives: Refactor alternatives_smp_module*

In reply to Ankur Arora

Refactor alternatives_smp_module* logic to make it available for holding generic late patching state.

Most of the changes are to pull the module handling logic out from CONFIG_SMP. In addition now we unconditionally call alternatives_smp_module_add() and make the decision on patching for UP or not there.

---
 arch/x86/include/asm/alternative.h | 13 ++-----
 arch/x86/kernel/alternative.c      | 55 ++++++++++++++++--------------
 2 files changed, 32 insertions(+), 36 deletions(-)
arch/x86/include/asm/alternative.h +2 -11
index 13adca37c99a..8235bbb746d9 100644
@@ -75,24 +75,15 @@ extern void apply_alternatives(struct alt_instr *start, struct alt_instr *end);
 
 struct module;
 
-#ifdef CONFIG_SMP
 extern void alternatives_smp_module_add(struct module *mod, char *name,
 					void *locks, void *locks_end,
 					void *text, void *text_end);
 extern void alternatives_smp_module_del(struct module *mod);
-extern void alternatives_enable_smp(void);
 extern int alternatives_text_reserved(void *start, void *end);
-extern bool skip_smp_alternatives;
+#ifdef CONFIG_SMP
+extern void alternatives_enable_smp(void);
 #else
-static inline void alternatives_smp_module_add(struct module *mod, char *name,
-					       void *locks, void *locks_end,
-					       void *text, void *text_end) {}
-static inline void alternatives_smp_module_del(struct module *mod) {}
 static inline void alternatives_enable_smp(void) {}
-static inline int alternatives_text_reserved(void *start, void *end)
-{
-	return 0;
-}
 #endif	/* CONFIG_SMP */
 
 #define b_replacement(num)	"664"#num
arch/x86/kernel/alternative.c +30 -25

115 lines not on this page · raw message

Signature · 1 line
2.20.1
#5 lore raw

RFC PATCH 5/26 x86/alternatives: Rename alternatives_smp*, smp_alt_module

In reply to Ankur Arora

Rename alternatives_smp_module_*(), smp_alt_module to reflect their new purpose.

---
 arch/x86/include/asm/alternative.h | 10 +++---
 arch/x86/kernel/alternative.c      | 54 +++++++++++++++---------------
 arch/x86/kernel/module.c           |  8 ++---
 3 files changed, 36 insertions(+), 36 deletions(-)
arch/x86/include/asm/alternative.h +5 -5
index 8235bbb746d9..db91a7731d87 100644
@@ -75,11 +75,11 @@ extern void apply_alternatives(struct alt_instr *start, struct alt_instr *end);
 
 struct module;
 
-extern void alternatives_smp_module_add(struct module *mod, char *name,
-					void *locks, void *locks_end,
-					void *text, void *text_end);
-extern void alternatives_smp_module_del(struct module *mod);
-extern int alternatives_text_reserved(void *start, void *end);
+void alternatives_module_add(struct module *mod, char *name,
+			     void *locks, void *locks_end,
+			     void *text, void *text_end);
+void alternatives_module_del(struct module *mod);
+int alternatives_text_reserved(void *start, void *end);
 #ifdef CONFIG_SMP
 extern void alternatives_enable_smp(void);
 #else
arch/x86/kernel/alternative.c +27 -27

121 lines not on this page · raw message

arch/x86/kernel/module.c +4 -4
index 658ea60ce324..fc3d35198b09 100644
@@ -251,9 +251,9 @@ int module_finalize(const Elf_Ehdr *hdr,
 	if (locks && text) {
 		void *lseg = (void *)locks->sh_addr;
 		void *tseg = (void *)text->sh_addr;
-		alternatives_smp_module_add(me, me->name,
-					    lseg, lseg + locks->sh_size,
-					    tseg, tseg + text->sh_size);
+		alternatives_module_add(me, me->name,
+					lseg, lseg + locks->sh_size,
+					tseg, tseg + text->sh_size);
 	}
 
 	if (para) {
@@ -278,5 +278,5 @@ int module_finalize(const Elf_Ehdr *hdr,
 
 void module_arch_cleanup(struct module *mod)
 {
-	alternatives_smp_module_del(mod);
+	alternatives_module_del(mod);
 }
Signature · 1 line
2.20.1
#6 lore raw

RFC PATCH 6/26 x86/alternatives: Remove stale symbols

In reply to Ankur Arora

__start_parainstructions and __stop_parainstructions aren't defined, remove them.

---
 arch/x86/kernel/alternative.c | 2 --
 1 file changed, 2 deletions(-)
arch/x86/kernel/alternative.c +0 -2
index 4157f848b537..09e4ee0e09a2 100644
@@ -623,8 +623,6 @@ void __init_or_module apply_paravirt(struct paravirt_patch_site *start,
 		text_poke_early(p->instr, insn_buff, p->len);
 	}
 }
-extern struct paravirt_patch_site __start_parainstructions[],
-	__stop_parainstructions[];
 #endif	/* CONFIG_PARAVIRT */
 
 /*
Signature · 1 line
2.20.1
#7 lore raw

RFC PATCH 7/26 x86/paravirt: Persist .parainstructions.runtime

In reply to Ankur Arora

Persist .parainstructions.runtime in memory. We will use it to patch paravirt-ops at runtime.

The extra memory footprint depends on chosen config options but the inlined queued_spin_unlock() presents an edge case:

 $ objdump -h vmlinux|grep .parainstructions
 Idx Name              		Size      VMA
 	LMA                File-off  Algn
  27 .parainstructions 		0001013c  ffffffff82895000
  	0000000002895000   01c95000  2**3
  28 .parainstructions.runtime  0000cd2c  ffffffff828a5140
  	00000000028a5140   01ca5140  2**3

(The added footprint is the size of the .parainstructions.runtime section.)

  $ size vmlinux
  text       data       bss        dec      hex       filename
  13726196   12302814   14094336   40123346 2643bd2   vmlinux
---
 arch/x86/include/asm/alternative.h |  1 +
 arch/x86/kernel/alternative.c      | 16 +++++++++++++++-
 arch/x86/kernel/module.c           | 28 +++++++++++++++++++++++-----
 3 files changed, 39 insertions(+), 6 deletions(-)
arch/x86/include/asm/alternative.h +1 -0
index db91a7731d87..d19546c14ff6 100644
@@ -76,6 +76,7 @@ extern void apply_alternatives(struct alt_instr *start, struct alt_instr *end);
 struct module;
 
 void alternatives_module_add(struct module *mod, char *name,
+			     void *para, void *para_end,
 			     void *locks, void *locks_end,
 			     void *text, void *text_end);
 void alternatives_module_del(struct module *mod);
arch/x86/kernel/alternative.c +15 -1
index 09e4ee0e09a2..8189ac21624c 100644
@@ -482,6 +482,12 @@ struct alt_module {
 	struct module	*mod;
 	char		*name;
 
+#ifdef CONFIG_PARAVIRT_RUNTIME
+	/* ptrs to paravirt sites */
+	struct paravirt_patch_site *para;
+	struct paravirt_patch_site *para_end;
+#endif
+
 	/* ptrs to lock prefixes */
 	const s32	*locks;
 	const s32	*locks_end;
@@ -496,6 +502,7 @@ struct alt_module {
 static LIST_HEAD(alt_modules);
 
 void __init_or_module alternatives_module_add(struct module *mod, char *name,
+					      void *para, void *para_end,
 					      void *locks, void *locks_end,
 					      void *text,  void *text_end)
 {
@@ -506,7 +513,7 @@ void __init_or_module alternatives_module_add(struct module *mod, char *name,
 	if (!noreplace_smp && (num_present_cpus() == 1 || setup_max_cpus <= 1))
 		uniproc_patched = true;
 #endif
-	if (!uniproc_patched)
+	if (!IS_ENABLED(CONFIG_PARAVIRT_RUNTIME) && !uniproc_patched)
 		return;
 
 	mutex_lock(&text_mutex);
@@ -516,6 +523,11 @@ void __init_or_module alternatives_module_add(struct module *mod, char *name,
 	alt->mod	= mod;
 	alt->name	= name;
 
+#ifdef CONFIG_PARAVIRT_RUNTIME
+	alt->para	= para;
+	alt->para_end	= para_end;
+#endif
+
 	if (num_possible_cpus() != 1 || uniproc_patched) {
 		/* Remember only if we'll need to undo it. */
 		alt->locks	= locks;
@@ -733,6 +745,8 @@ void __init alternative_instructions(void)
 	apply_alternatives(__alt_instructions, __alt_instructions_end);
 
 	alternatives_module_add(NULL, "core kernel",
+				__parainstructions_runtime,
+				__parainstructions_runtime_end,
 				__smp_locks, __smp_locks_end,
 				_text, _etext);
 
arch/x86/kernel/module.c +23 -5
index fc3d35198b09..7b2632184c11 100644
@@ -248,12 +248,30 @@ int module_finalize(const Elf_Ehdr *hdr,
 		void *aseg = (void *)alt->sh_addr;
 		apply_alternatives(aseg, aseg + alt->sh_size);
 	}
-	if (locks && text) {
-		void *lseg = (void *)locks->sh_addr;
-		void *tseg = (void *)text->sh_addr;
+	if (para_run || (locks && text)) {
+		void *pseg, *pseg_end;
+		void *lseg, *lseg_end;
+		void *tseg, *tseg_end;
+
+		pseg = pseg_end = NULL;
+		lseg = lseg_end = NULL;
+		tseg = tseg_end = NULL;
+		if (para_run) {
+			pseg = (void *)para_run->sh_addr;
+			pseg_end = pseg + para_run->sh_size;
+		}
+
+		if (locks && text) {
+			tseg = (void *)text->sh_addr;
+			tseg_end = tseg + text->sh_size;
+
+			lseg = (void *)locks->sh_addr;
+			lseg_end = lseg + locks->sh_size;
+		}
 		alternatives_module_add(me, me->name,
-					lseg, lseg + locks->sh_size,
-					tseg, tseg + text->sh_size);
+					pseg, pseg_end,
+					lseg, lseg_end,
+					tseg, tseg_end);
 	}
 
 	if (para) {
Signature · 1 line
2.20.1
#8 lore raw

RFC PATCH 8/26 x86/paravirt: Stash native pv-ops

In reply to Ankur Arora

Introduce native_pv_ops where we stash the pv_ops array before hypervisor specific hooks have modified it.

native_pv_ops get used when switching between paravirt and native pv-ops at runtime.

---
 arch/x86/include/asm/paravirt_types.h |  4 ++++
 arch/x86/kernel/paravirt.c            | 10 ++++++++++
 arch/x86/kernel/setup.c               |  2 ++
 3 files changed, 16 insertions(+)
arch/x86/include/asm/paravirt_types.h +4 -0
index f1153f53c529..bc935eec7ec6 100644
@@ -339,6 +339,7 @@ extern struct paravirt_patch_template pv_ops;
 
 #ifdef CONFIG_PARAVIRT_RUNTIME
 #define PVRT_SUFFIX ".runtime"
+extern struct paravirt_patch_template native_pv_ops;
 #else
 #define PVRT_SUFFIX ""
 #endif
@@ -775,6 +776,9 @@ extern struct paravirt_patch_site __parainstructions[],
 #ifdef CONFIG_PARAVIRT_RUNTIME
 extern struct paravirt_patch_site __parainstructions_runtime[],
 	__parainstructions_runtime_end[];
+void paravirt_ops_init(void);
+#else
+static inline void paravirt_ops_init(void) { }
 #endif
 
 #endif	/* __ASSEMBLY__ */
arch/x86/kernel/paravirt.c +10 -0
index c131ba4e70ef..8c511cc4d4f4 100644
@@ -458,5 +458,15 @@ NOKPROBE_SYMBOL(native_set_debugreg);
 NOKPROBE_SYMBOL(native_load_idt);
 #endif
 
+#ifdef CONFIG_PARAVIRT_RUNTIME
+__ro_after_init struct paravirt_patch_template native_pv_ops;
+
+void __init paravirt_ops_init(void)
+{
+	native_pv_ops = pv_ops;
+}
+EXPORT_SYMBOL(native_pv_ops);
+#endif
+
 EXPORT_SYMBOL(pv_ops);
 EXPORT_SYMBOL_GPL(pv_info);
arch/x86/kernel/setup.c +2 -0
index e6b545047f38..2746a6a78fe7 100644
@@ -43,6 +43,7 @@
 #include <asm/unwind.h>
 #include <asm/vsyscall.h>
 #include <linux/vmalloc.h>
+#include <asm/paravirt_types.h>
 
 /*
  * max_low_pfn_mapped: highest directly mapped pfn < 4 GB
@@ -831,6 +832,7 @@ void __init setup_arch(char **cmdline_p)
 	boot_cpu_data.x86_phys_bits = MAX_PHYSMEM_BITS;
 #endif
 
+	paravirt_ops_init();
 	/*
 	 * If we have OLPC OFW, we might end up relocating the fixmap due to
 	 * reserve_top(), so do this before touching the ioremap area.
Signature · 1 line
2.20.1
#9 lore raw

RFC PATCH 9/26 x86/paravirt: Add runtime_patch()

In reply to Ankur Arora

runtime_patch() generates insn sequences for patching supported pv_ops. It does this by calling paravirt_patch_default() or native_patch() dpending on if the target is a paravirt or native pv-op.

In addition, runtime_patch() also whitelists pv-ops that are safe to patch at runtime.

The static conditions that need to be satisfied to patch safely:
- Insn sequences under replacement need to execute without preemption. This is meant to avoid scenarios where a call-site (ex. lock.vcpu_is_preempted) switches between the following sequences:

lock.vcpu_is_preempted = __raw_callee_save___kvm_vcpu_is_preempted

    0: e8 31 e6 ff ff		callq  0xffffffffffffe636
    4: 66 90			xchg   %ax,%ax      # NOP2

lock.vcpu_is_preempted = __raw_callee_save___native_vcpu_is_preempted

    0: 31 c0			xor   %rax, %rax
    2: 0f 1f 44 00 00		nopl   0x0(%rax)    # NOP5

If kvm_vcpu_is_preempted() were preemptible, then, post patching we would return to address 4 above, which is in the middle of an instruction for native_vcpu_is_preempted().

Even if this were to be made safe (ex. by changing the NOP2 to be a prefix instead of a suffix), it would still not be enough -- since we do not want any code from the switched out pv-op to be executing after the pv-op has been switched out.

- Entered only at the beginning: this allows us to use text_poke() which uses INT3 as a barrier.

We don't store the address inside any call-sites so the second can be assumed.

Guaranteeing the first condition boils down to stating that any pv-op being patched cannot be present/referenced from any call-stack in the system. pv-ops that are not obviously non-preemptible need to be enclosed in preempt_disable_runtime_patch()/preempt_enable_runtime_patch().

This should be sufficient because runtime_patch() itself is called from a stop_machine() context which would would be enough to flush out any non-preemptible sequences.

Note that preemption in the host is okay: stop_machine() would unwind any pv-ops sleeping in the host.

---
 arch/x86/include/asm/paravirt_types.h |  8 +++++
 arch/x86/kernel/paravirt.c            |  6 +---
 arch/x86/kernel/paravirt_patch.c      | 49 +++++++++++++++++++++++++++
 include/linux/preempt.h               | 17 ++++++++++
 4 files changed, 75 insertions(+), 5 deletions(-)
arch/x86/include/asm/paravirt_types.h +8 -0
index bc935eec7ec6..3b9f6c105397 100644
@@ -350,6 +350,12 @@ extern struct paravirt_patch_template native_pv_ops;
 #define PARAVIRT_PATCH(x)					\
 	(offsetof(struct paravirt_patch_template, x) / sizeof(void *))
 
+/*
+ * Neat trick to map patch type back to the call within the
+ * corresponding structure.
+ */
+#define PARAVIRT_PATCH_OP(ops, type) (*(long *)(&((long **)&(ops))[type]))
+
 #define paravirt_type(op)				\
 	[paravirt_typenum] "i" (PARAVIRT_PATCH(op)),	\
 	[paravirt_opptr] "i" (&(pv_ops.op))
@@ -383,6 +389,8 @@ unsigned paravirt_patch_default(u8 type, void *insn_buff, unsigned long addr, un
 unsigned paravirt_patch_insns(void *insn_buff, unsigned len, const char *start, const char *end);
 
 unsigned native_patch(u8 type, void *insn_buff, unsigned long addr, unsigned len);
+int runtime_patch(u8 type, void *insn_buff, void *op, unsigned long addr,
+		  unsigned int len);
 
 int paravirt_disable_iospace(void);
 
arch/x86/kernel/paravirt.c +1 -5
index 8c511cc4d4f4..c4128436b05a 100644
@@ -117,11 +117,7 @@ void __init native_pv_lock_init(void)
 unsigned paravirt_patch_default(u8 type, void *insn_buff,
 				unsigned long addr, unsigned len)
 {
-	/*
-	 * Neat trick to map patch type back to the call within the
-	 * corresponding structure.
-	 */
-	void *opfunc = *((void **)&pv_ops + type);
+	void *opfunc = (void *)PARAVIRT_PATCH_OP(pv_ops, type);
 	unsigned ret;
 
 	if (opfunc == NULL)
arch/x86/kernel/paravirt_patch.c +49 -0

57 lines not on this page · raw message

include/linux/preempt.h +17 -0
index bc3f1aecaa19..c569d077aab2 100644
@@ -203,6 +203,13 @@ do { \
 		__preempt_schedule(); \
 } while (0)
 
+/*
+ * preempt_enable_no_resched() so we don't add any preemption points until
+ * after the caller has returned.
+ */
+#define preempt_enable_runtime_patch()	preempt_enable_no_resched()
+#define preempt_disable_runtime_patch()	preempt_disable()
+
 #else /* !CONFIG_PREEMPTION */
 #define preempt_enable() \
 do { \
@@ -217,6 +224,12 @@ do { \
 } while (0)
 
 #define preempt_check_resched() do { } while (0)
+
+/*
+ * NOP, if there's no preemption.
+ */
+#define preempt_disable_runtime_patch()	do { } while (0)
+#define preempt_enable_runtime_patch()	do { } while (0)
 #endif /* CONFIG_PREEMPTION */
 
 #define preempt_disable_notrace() \
@@ -250,6 +263,8 @@ do { \
 #define preempt_enable_notrace()		barrier()
 #define preemptible()				0
 
+#define preempt_disable_runtime_patch()	do { } while (0)
+#define preempt_enable_runtime_patch()	do { } while (0)
 #endif /* CONFIG_PREEMPT_COUNT */
 
 #ifdef MODULE
@@ -260,6 +275,8 @@ do { \
 #undef preempt_enable_no_resched
 #undef preempt_enable_no_resched_notrace
 #undef preempt_check_resched
+#undef preempt_disable_runtime_patch
+#undef preempt_enable_runtime_patch
 #endif
 
 #define preempt_set_need_resched() \
Signature · 1 line
2.20.1
maintainer NAK
#10 lore raw

Re: [RFC PATCH 09/26] x86/paravirt: Add runtime_patch()

include/linux/preempt.h do { \ · line 211

Quoted 6 lines · Ankur Arora
include/linux/preempt.h +6 -0
+/*
+ * preempt_enable_no_resched() so we don't add any preemption points until
+ * after the caller has returned.
+ */
+#define preempt_enable_runtime_patch()	preempt_enable_no_resched()
+#define preempt_disable_runtime_patch()	preempt_disable()

NAK, this is probably a stright preemption bug, also, afaict, there aren't actually any users of this in the patch-set.

#11 lore raw

RFC PATCH 10/26 x86/paravirt: Add primitives to stage pv-ops

In reply to Ankur Arora

Add paravirt_stage_alt() which conditionally selects between a paravirt or native pv-op and then stages it for later patching.

---
 arch/x86/include/asm/paravirt_types.h |  6 +++
 arch/x86/include/asm/text-patching.h  |  3 ++
 arch/x86/kernel/alternative.c         | 58 +++++++++++++++++++++++++++
 3 files changed, 67 insertions(+)
arch/x86/include/asm/paravirt_types.h +6 -0
index 3b9f6c105397..0c4ca7ad719c 100644
@@ -350,6 +350,12 @@ extern struct paravirt_patch_template native_pv_ops;
 #define PARAVIRT_PATCH(x)					\
 	(offsetof(struct paravirt_patch_template, x) / sizeof(void *))
 
+#define paravirt_stage_alt(do_stage, op, opfn)				\
+	(text_poke_pv_stage(PARAVIRT_PATCH(op),				\
+			    (do_stage) ? (opfn) : (native_pv_ops.op)))
+
+#define paravirt_stage_zero() text_poke_pv_stage_zero()
+
 /*
  * Neat trick to map patch type back to the call within the
  * corresponding structure.
arch/x86/include/asm/text-patching.h +3 -0
index e2ef241c261e..706e61e6967d 100644
@@ -55,6 +55,9 @@ extern void text_poke_bp(void *addr, const void *opcode, size_t len, const void
 extern void text_poke_queue(void *addr, const void *opcode, size_t len, const void *emulate);
 extern void text_poke_finish(void);
 
+bool text_poke_pv_stage(u8 type, void *opfn);
+void text_poke_pv_stage_zero(void);
+
 #define INT3_INSN_SIZE		1
 #define INT3_INSN_OPCODE	0xCC
 
arch/x86/kernel/alternative.c +58 -0

61 lines not on this page · raw message

Signature · 1 line
2.20.1
#12 lore raw

RFC PATCH 11/26 x86/alternatives: Remove return value of text_poke*()

In reply to Ankur Arora

Various text_poke() variants don't return a useful value. Remove it.

---
 arch/x86/include/asm/text-patching.h |  4 ++--
 arch/x86/kernel/alternative.c        | 11 +++++------
 2 files changed, 7 insertions(+), 8 deletions(-)
arch/x86/include/asm/text-patching.h +2 -2
index 706e61e6967d..04778c2bc34e 100644
@@ -46,9 +46,9 @@ extern void text_poke_early(void *addr, const void *opcode, size_t len);
  * On the local CPU you need to be protected against NMI or MCE handlers seeing
  * an inconsistent instruction while you patch.
  */
-extern void *text_poke(void *addr, const void *opcode, size_t len);
+extern void text_poke(void *addr, const void *opcode, size_t len);
 extern void text_poke_sync(void);
-extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len);
+extern void text_poke_kgdb(void *addr, const void *opcode, size_t len);
 extern int poke_int3_handler(struct pt_regs *regs);
 extern void text_poke_bp(void *addr, const void *opcode, size_t len, const void *emulate);
 
arch/x86/kernel/alternative.c +5 -6
index 0c335af9ee28..8c79a3dc5e72 100644
@@ -805,7 +805,7 @@ void __init_or_module text_poke_early(void *addr, const void *opcode,
 __ro_after_init struct mm_struct *poking_mm;
 __ro_after_init unsigned long poking_addr;
 
-static void *__text_poke(void *addr, const void *opcode, size_t len)
+static void __text_poke(void *addr, const void *opcode, size_t len)
 {
 	bool cross_page_boundary = offset_in_page(addr) + len > PAGE_SIZE;
 	struct page *pages[2] = {NULL};
@@ -906,7 +906,6 @@ static void *__text_poke(void *addr, const void *opcode, size_t len)
 
 	pte_unmap_unlock(ptep, ptl);
 	local_irq_restore(flags);
-	return addr;
 }
 
 /**
@@ -925,11 +924,11 @@ static void *__text_poke(void *addr, const void *opcode, size_t len)
  * by registering a module notifier, and ordering module removal and patching
  * trough a mutex.
  */
-void *text_poke(void *addr, const void *opcode, size_t len)
+void text_poke(void *addr, const void *opcode, size_t len)
 {
 	lockdep_assert_held(&text_mutex);
 
-	return __text_poke(addr, opcode, len);
+	__text_poke(addr, opcode, len);
 }
 
 /**
@@ -946,9 +945,9 @@ void *text_poke(void *addr, const void *opcode, size_t len)
  * Context: should only be used by kgdb, which ensures no other core is running,
  *	    despite the fact it does not hold the text_mutex.
  */
-void *text_poke_kgdb(void *addr, const void *opcode, size_t len)
+void text_poke_kgdb(void *addr, const void *opcode, size_t len)
 {
-	return __text_poke(addr, opcode, len);
+	__text_poke(addr, opcode, len);
 }
 
 static void do_sync_core(void *info)
Signature · 1 line
2.20.1
#13 lore raw

RFC PATCH 12/26 x86/alternatives: Use __get_unlocked_pte() in text_poke()

In reply to Ankur Arora

text_poke() uses get_locked_pte() to map poking_addr. However, this introduces a dependency on locking code which precludes using text_poke() to modify qspinlock primitives.

Accesses to this pte (and poking_addr) are protected by text_mutex so we can safely switch to __get_unlocked_pte() here. Note that we do need to be careful that we do not try to modify the poking_addr from multiple contexts simultaneously (ex. INT3 or NMI context.)

---
 arch/x86/kernel/alternative.c |  9 ++++-----
 include/linux/mm.h            | 16 ++++++++++++++--
 mm/memory.c                   |  9 ++++++---
 3 files changed, 24 insertions(+), 10 deletions(-)
arch/x86/kernel/alternative.c +4 -5
index 8c79a3dc5e72..0344e49a4ade 100644
@@ -812,7 +812,6 @@ static void __text_poke(void *addr, const void *opcode, size_t len)
 	temp_mm_state_t prev;
 	unsigned long flags;
 	pte_t pte, *ptep;
-	spinlock_t *ptl;
 	pgprot_t pgprot;
 
 	/*
@@ -846,10 +845,11 @@ static void __text_poke(void *addr, const void *opcode, size_t len)
 	pgprot = __pgprot(pgprot_val(PAGE_KERNEL) & ~_PAGE_GLOBAL);
 
 	/*
-	 * The lock is not really needed, but this allows to avoid open-coding.
+	 * text_poke() might be used to poke spinlock primitives so do this
+	 * unlocked. This does mean that we need to be careful that no other
+	 * context (ex. INT3 handler) is simultaneously writing to this pte.
 	 */
-	ptep = get_locked_pte(poking_mm, poking_addr, &ptl);
-
+	ptep = __get_unlocked_pte(poking_mm, poking_addr);
 	/*
 	 * This must not fail; preallocated in poking_init().
 	 */
@@ -904,7 +904,6 @@ static void __text_poke(void *addr, const void *opcode, size_t len)
 	 */
 	BUG_ON(memcmp(addr, opcode, len));
 
-	pte_unmap_unlock(ptep, ptl);
 	local_irq_restore(flags);
 }
 
include/linux/mm.h +14 -2
index 7dd5c4ccbf85..d4a652c2e269 100644
@@ -1895,8 +1895,20 @@ static inline int pte_devmap(pte_t pte)
 
 int vma_wants_writenotify(struct vm_area_struct *vma, pgprot_t vm_page_prot);
 
-extern pte_t *__get_locked_pte(struct mm_struct *mm, unsigned long addr,
-			       spinlock_t **ptl);
+pte_t *__get_pte(struct mm_struct *mm, unsigned long addr, spinlock_t **ptl);
+
+static inline pte_t *__get_unlocked_pte(struct mm_struct *mm,
+					unsigned long addr)
+{
+	return __get_pte(mm, addr, NULL);
+}
+
+static inline pte_t *__get_locked_pte(struct mm_struct *mm,
+				      unsigned long addr, spinlock_t **ptl)
+{
+	return __get_pte(mm, addr, ptl);
+}
+
 static inline pte_t *get_locked_pte(struct mm_struct *mm, unsigned long addr,
 				    spinlock_t **ptl)
 {
mm/memory.c +6 -3
index 586271f3efc6..7acfe9512084 100644
@@ -1407,8 +1407,8 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address,
 }
 EXPORT_SYMBOL_GPL(zap_vma_ptes);
 
-pte_t *__get_locked_pte(struct mm_struct *mm, unsigned long addr,
-			spinlock_t **ptl)
+pte_t *__get_pte(struct mm_struct *mm, unsigned long addr,
+		 spinlock_t **ptl)
 {
 	pgd_t *pgd;
 	p4d_t *p4d;
@@ -1427,7 +1427,10 @@ pte_t *__get_locked_pte(struct mm_struct *mm, unsigned long addr,
 		return NULL;
 
 	VM_BUG_ON(pmd_trans_huge(*pmd));
-	return pte_alloc_map_lock(mm, pmd, addr, ptl);
+	if (likely(ptl))
+		return pte_alloc_map_lock(mm, pmd, addr, ptl);
+	else
+		return pte_alloc_map(mm, pmd, addr);
 }
 
 /*
Signature · 1 line
2.20.1
#14 lore raw

RFC PATCH 13/26 x86/alternatives: Split __text_poke()

In reply to Ankur Arora

Separate __text_poke() into map, memcpy and unmap portions, (__text_poke_map(), __text_do_poke() and __text_poke_unmap().)

Do this to separate the non-reentrant bits from the reentrant __text_do_poke(). __text_poke_map()/_unmap() modify poking_mm, poking_addr and do the pte-mapping and thus are non-reentrant.

This allows __text_do_poke() to be safely called from an INT3 context with __text_poke_map()/_unmap() being called at the start and the end of the patching of a call-site instead of doing that for each stage of the three patching stages.

---
 arch/x86/kernel/alternative.c | 46 +++++++++++++++++++++++++----------
 1 file changed, 33 insertions(+), 13 deletions(-)
arch/x86/kernel/alternative.c +33 -13
index 0344e49a4ade..337aad8c2521 100644
@@ -805,13 +805,12 @@ void __init_or_module text_poke_early(void *addr, const void *opcode,
 __ro_after_init struct mm_struct *poking_mm;
 __ro_after_init unsigned long poking_addr;
 
-static void __text_poke(void *addr, const void *opcode, size_t len)
+static void __text_poke_map(void *addr, size_t len,
+			    temp_mm_state_t *prev_mm, pte_t **ptep)
 {
 	bool cross_page_boundary = offset_in_page(addr) + len > PAGE_SIZE;
 	struct page *pages[2] = {NULL};
-	temp_mm_state_t prev;
-	unsigned long flags;
-	pte_t pte, *ptep;
+	pte_t pte;
 	pgprot_t pgprot;
 
 	/*
@@ -836,8 +835,6 @@ static void __text_poke(void *addr, const void *opcode, size_t len)
 	 */
 	BUG_ON(!pages[0] || (cross_page_boundary && !pages[1]));
 
-	local_irq_save(flags);
-
 	/*
 	 * Map the page without the global bit, as TLB flushing is done with
 	 * flush_tlb_mm_range(), which is intended for non-global PTEs.
@@ -849,30 +846,42 @@ static void __text_poke(void *addr, const void *opcode, size_t len)
 	 * unlocked. This does mean that we need to be careful that no other
 	 * context (ex. INT3 handler) is simultaneously writing to this pte.
 	 */
-	ptep = __get_unlocked_pte(poking_mm, poking_addr);
+	*ptep = __get_unlocked_pte(poking_mm, poking_addr);
 	/*
 	 * This must not fail; preallocated in poking_init().
 	 */
-	VM_BUG_ON(!ptep);
+	VM_BUG_ON(!*ptep);
 
 	pte = mk_pte(pages[0], pgprot);
-	set_pte_at(poking_mm, poking_addr, ptep, pte);
+	set_pte_at(poking_mm, poking_addr, *ptep, pte);
 
 	if (cross_page_boundary) {
 		pte = mk_pte(pages[1], pgprot);
-		set_pte_at(poking_mm, poking_addr + PAGE_SIZE, ptep + 1, pte);
+		set_pte_at(poking_mm, poking_addr + PAGE_SIZE, *ptep + 1, pte);
 	}
 
 	/*
 	 * Loading the temporary mm behaves as a compiler barrier, which
 	 * guarantees that the PTE will be set at the time memcpy() is done.
 	 */
-	prev = use_temporary_mm(poking_mm);
+	*prev_mm = use_temporary_mm(poking_mm);
+}
 
+/*
+ * Do the actual poke. Needs to be re-entrant as this can be called
+ * via INT3 context as well.
+ */
+static void __text_do_poke(unsigned long offset, const void *opcode, size_t len)
+{
 	kasan_disable_current();
-	memcpy((u8 *)poking_addr + offset_in_page(addr), opcode, len);
+	memcpy((u8 *)poking_addr + offset, opcode, len);
 	kasan_enable_current();
+}
 
+static void __text_poke_unmap(void *addr, const void *opcode, size_t len,
+			      temp_mm_state_t *prev_mm, pte_t *ptep)
+{
+	bool cross_page_boundary = offset_in_page(addr) + len > PAGE_SIZE;
 	/*
 	 * Ensure that the PTE is only cleared after the instructions of memcpy
 	 * were issued by using a compiler barrier.
@@ -888,7 +897,7 @@ static void __text_poke(void *addr, const void *opcode, size_t len)
 	 * instruction that already allows the core to see the updated version.
 	 * Xen-PV is assumed to serialize execution in a similar manner.
 	 */
-	unuse_temporary_mm(prev);
+	unuse_temporary_mm(*prev_mm);
 
 	/*
 	 * Flushing the TLB might involve IPIs, which would require enabled
@@ -903,7 +912,18 @@ static void __text_poke(void *addr, const void *opcode, size_t len)
 	 * fundamentally screwy; there's nothing we can really do about that.
 	 */
 	BUG_ON(memcmp(addr, opcode, len));
+}
 
+static void __text_poke(void *addr, const void *opcode, size_t len)
+{
+	temp_mm_state_t prev_mm;
+	unsigned long flags;
+	pte_t *ptep;
+
+	local_irq_save(flags);
+	__text_poke_map(addr, len, &prev_mm, &ptep);
+	__text_do_poke(offset_in_page(addr), opcode, len);
+	__text_poke_unmap(addr, opcode, len, &prev_mm, ptep);
 	local_irq_restore(flags);
 }
 
Signature · 1 line
2.20.1
#15 lore raw

RFC PATCH 14/26 x86/alternatives: Handle native insns in text_poke_loc*()

In reply to Ankur Arora

Intended to handle scenarios where we might want to patch arbitrary instructions (ex. inlined opcodes in pv_lock_ops.)

Users for native mode (as opposed to emulated) are introduced in later patches.

---
 arch/x86/include/asm/text-patching.h |  4 +-
 arch/x86/kernel/alternative.c        | 61 ++++++++++++++++++++--------
 2 files changed, 45 insertions(+), 20 deletions(-)
arch/x86/include/asm/text-patching.h +2 -2
index 04778c2bc34e..c4b2814f2f9d 100644
@@ -25,10 +25,10 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
 
 /*
  * Currently, the max observed size in the kernel code is
- * JUMP_LABEL_NOP_SIZE/RELATIVEJUMP_SIZE, which are 5.
+ * NOP7 for indirect call, which is 7.
  * Raise it if needed.
  */
-#define POKE_MAX_OPCODE_SIZE	5
+#define POKE_MAX_OPCODE_SIZE	7
 
 extern void text_poke_early(void *addr, const void *opcode, size_t len);
 
arch/x86/kernel/alternative.c +43 -18

148 lines not on this page · raw message

Signature · 1 line
2.20.1
maintainer NAK
#16 lore raw

Re: [RFC PATCH 14/26] x86/alternatives: Handle native insns in text_poke_loc*()

Quoted 5 lines · Ankur Arora

struct text_poke_loc {

arch/x86/kernel/alternative.c +1 -2
 	s32 rel_addr; /* addr := _stext + rel_addr */
-	s32 rel32;
-	u8 opcode;
+	union {

arch/x86/kernel/alternative.c void text_poke_sync(void) · line 994

Quoted 10 lines
arch/x86/kernel/alternative.c +8 -0
+		struct {
+			s32 rel32;
+			u8 opcode;
+		} emulated;
+		struct {
+			u8 len;
+		} native;
+	};
 	const u8 text[POKE_MAX_OPCODE_SIZE];
 };

NAK, this grows the structure from 16 to 20 bytes.

maintainer
#17 lore raw

Re: [RFC PATCH 14/26] x86/alternatives: Handle native insns in text_poke_loc*()

In reply to Ankur Arora

arch/x86/kernel/alternative.c int notrace poke_int3_handler(struct pt_regs *regs) · line 1084

Quoted 8 lines · Ankur Arora
arch/x86/kernel/alternative.c +3 -1
@@ -1071,10 +1079,13 @@ int notrace poke_int3_handler(struct pt_regs *regs)
 			goto out_put;
 	}
 
-	len = text_opcode_size(tp->opcode);
+	if (desc->native)
+		BUG();
+

Subject: x86/alternatives: Handle native insns in text_poke_loc*()

That's not really handling, is it..

#18 lore raw

RFC PATCH 15/26 x86/alternatives: Non-emulated text poking

In reply to Ankur Arora

Patching at runtime needs to handle interdependent pv-ops: as an example, lock.queued_lock_slowpath(), lock.queued_lock_unlock() and the other pv_lock_ops are paired and so need to be updated atomically. This is difficult with emulation because non-patching CPUs could be executing in critical sections.
(We could apply INT3 everywhere first and then use RCU to force a barrier but given that spinlocks are everywhere, it still might mean a lot of time in emulation.)

Second, locking operations can be called from interrupt handlers which means we cannot trivially use IPIs to introduce a pipeline sync step on non-patching CPUs.

Third, some pv-ops can be inlined and so we would need to emulate a broader set of operations than CALL, JMP, NOP*.

Introduce the core state-machine with the actual poking and pipeline sync stubbed out. This executes via stop_machine() with the primary CPU carrying out a text_poke_bp() style three-staged algorithm.

The control flow diagram below shows CPU0 as the primary which does the patching, while the rest of the CPUs (CPUx) execute the sync loop in text_poke_sync_finish().

CPU0 CPUx
---- ----

patch_worker() patch_worker()

/* Traversal, insn-gen */ text_poke_sync_finish()
tps.patch_worker() /*
* wait until:

     /* for each patch-site */ 	       *  tps->state == PATCH_DONE
     text_poke_site()		       */
       poke_sync()

... ...

smp_store_release(&tps->state, PATCH_DONE)

Commits further on flesh out the rest of the code.

---
sync_one() uses the following for pipeline synchronization:

+ if (in_nmi())
+ cpuid_eax(1);
+ else
+ sync_core();

The if (in_nmi()) clause is meant to be executed from NMI contexts. Reading through past LKML discussions cpuid_eax() is probably a bad choice -- at least in so far as Xen PV is concerned. What would be a good primitive to use insead?

Also, given that we do handle the nested NMI case, does it make sense to just use native_iret() (via sync_core()) in NMI contexts well?

---
 arch/x86/kernel/alternative.c | 247 ++++++++++++++++++++++++++++++++++
 1 file changed, 247 insertions(+)
arch/x86/kernel/alternative.c +247 -0
index 004fe86f463f..452d4081eded 100644
@@ -979,6 +979,26 @@ void text_poke_sync(void)
 	on_each_cpu(do_sync_core, NULL, 1);
 }
 
+static void __maybe_unused sync_one(void)
+{
+	/*
+	 * We might be executing in NMI context, and so cannot use
+	 * IRET as a synchronizing instruction.
+	 *
+	 * We could use native_write_cr2() but that is not guaranteed
+	 * to work on Xen-PV -- it is emulated by Xen and might not
+	 * execute an iret (or similar synchronizing instruction)
+	 * internally.
+	 *
+	 * cpuid() would trap as well. Unclear if that's a solution
+	 * either.
+	 */
+	if (in_nmi())
+		cpuid_eax(1);
+	else
+		sync_core();
+}
+
 struct text_poke_loc {
 	s32 rel_addr; /* addr := _stext + rel_addr */
 	union {
@@ -1351,6 +1371,233 @@ void __ref text_poke_bp(void *addr, const void *opcode, size_t len, const void *
 	text_poke_bp_batch(&tp, 1);
 }
 
+struct text_poke_state;
+typedef void (*patch_worker_t)(struct text_poke_state *tps);
+
+/*
+ *                        +-----------possible-BP----------+
+ *                        |                                |
+ *         +--write-INT3--+   +--suffix--+   +-insn-prefix-+
+ *        /               | _/           |__/              |
+ *       /                v'             v                 v
+ * PATCH_SYNC_0    PATCH_SYNC_1    PATCH_SYNC_2   *PATCH_SYNC_DONE*
+ *       \                                                    |`----> PATCH_DONE
+ *        `----------<---------<---------<---------<----------+
+ *
+ * We start in state PATCH_SYNC_DONE and loop through PATCH_SYNC_* states
+ * to end at PATCH_DONE. The primary drives these in text_poke_site()
+ * with patch_worker() making the final transition to PATCH_DONE.
+ * All transitions but the last iteration need to be globally observed.
+ *
+ * On secondary CPUs, text_poke_sync_finish() waits in a cpu_relax()
+ * loop waiting for a transition to PATCH_SYNC_0 at which point it would
+ * start observing transitions until PATCH_SYNC_DONE.
+ * Eventually the master moves to PATCH_DONE and secondary CPUs finish.
+ */
+enum patch_state {
+	/*
+	 * Add an artificial state that we can do a bitwise operation
+	 * over all the PATCH_SYNC_* states.
+	 */
+	PATCH_SYNC_x = 4,
+	PATCH_SYNC_0 = PATCH_SYNC_x | 0,	/* Serialize INT3 */
+	PATCH_SYNC_1 = PATCH_SYNC_x | 1,	/* Serialize rest */
+	PATCH_SYNC_2 = PATCH_SYNC_x | 2,	/* Serialize first opcode */
+	PATCH_SYNC_DONE = PATCH_SYNC_x | 3,	/* Site done, and start state */
+
+	PATCH_DONE = 8,				/* End state */
+};
+
+/*
+ * State for driving text-poking via stop_machine().
+ */
+struct text_poke_state {
+	/* Whatever we are poking */
+	void *stage;
+
+	/* Modules to be processed. */
+	struct list_head *head;
+
+	/*
+	 * Accesses to sync_ack_map are ordered by the primary
+	 * via tps.state.
+	 */
+	struct cpumask sync_ack_map;
+
+	/*
+	 * Generates insn sequences for call-sites to be patched and
+	 * calls text_poke_site() to do the actual poking.
+	 */
+	patch_worker_t	patch_worker;
+
+	/*
+	 * Where are we in the patching state-machine.
+	 */
+	enum patch_state state;
+
+	unsigned int primary_cpu; /* CPU doing the patching. */
+	unsigned int num_acks; /* Number of Acks needed. */
+};
+
+static struct text_poke_state text_poke_state;
+
+/**
+ * poke_sync() - transitions to the specified state.
+ *
+ * @tps - struct text_poke_state *
+ * @state - one of PATCH_SYNC_* states
+ * @offset - offset to be patched
+ * @insns - insns to write
+ * @len - length of insn sequence
+ */
+static void poke_sync(struct text_poke_state *tps, int state, int offset,
+		      const char *insns, int len)
+{
+	/*
+	 * STUB: no patching or synchronization, just go through the
+	 * motions.
+	 */
+	smp_store_release(&tps->state, state);
+}
+
+/**
+ * text_poke_site() - called on the primary to patch a single call site.
+ *
+ * Returns after switching tps->state to PATCH_SYNC_DONE.
+ */
+static void __maybe_unused text_poke_site(struct text_poke_state *tps,
+					  struct text_poke_loc *tp)
+{
+	const unsigned char int3 = INT3_INSN_OPCODE;
+	temp_mm_state_t prev_mm;
+	pte_t *ptep;
+	int offset;
+
+	__text_poke_map(text_poke_addr(tp), tp->native.len, &prev_mm, &ptep);
+
+	offset = offset_in_page(text_poke_addr(tp));
+
+	/*
+	 * All secondary CPUs are waiting in tps->state == PATCH_SYNC_DONE
+	 * to move to PATCH_SYNC_0. Poke the INT3 and wait until all CPUs
+	 * are known to have observed PATCH_SYNC_0.
+	 *
+	 * The earliest we can hit an INT3 is just after the first poke.
+	 */
+	poke_sync(tps, PATCH_SYNC_0, offset, &int3, INT3_INSN_SIZE);
+
+	/* Poke remaining */
+	poke_sync(tps, PATCH_SYNC_1, offset + INT3_INSN_SIZE,
+		  tp->text + INT3_INSN_SIZE, tp->native.len - INT3_INSN_SIZE);
+
+	/*
+	 * Replace the INT3 with the first opcode and force the serializing
+	 * instruction for the last time. Any secondaries in the BP
+	 * handler should be able to move past the INT3 handler after this.
+	 * (See poke_int3_native() for details on this.)
+	 */
+	poke_sync(tps, PATCH_SYNC_2, offset, tp->text, INT3_INSN_SIZE);
+
+	/*
+	 * Force all CPUS to observe PATCH_SYNC_DONE (in the BP handler or
+	 * in text_poke_site()), so they know that this iteration is done
+	 * and it is safe to exit the wait-until-a-sync-is-required loop.
+	 */
+	poke_sync(tps, PATCH_SYNC_DONE, 0, NULL, 0);
+
+	/*
+	 * Unmap the poking_addr, poking_mm.
+	 */
+	__text_poke_unmap(text_poke_addr(tp), tp->text, tp->native.len,
+			  &prev_mm, ptep);
+}
+
+/**
+ * text_poke_sync_finish() -- called to synchronize the CPU pipeline
+ * on secondary CPUs for all patch sites.
+ *
+ * Called in thread context with tps->state == PATCH_SYNC_DONE.
+ * Returns with tps->state == PATCH_DONE.
+ */
+static void text_poke_sync_finish(struct text_poke_state *tps)
+{
+	while (true) {
+		enum patch_state state;
+
+		state = READ_ONCE(tps->state);
+
+		/*
+		 * We aren't doing any actual poking yet, so we don't
+		 * handle any other states.
+		 */
+		if (state == PATCH_DONE)
+			break;
+
+		/*
+		 * Relax here while the primary makes up its mind on
+		 * whether it is done or not.
+		 */
+		cpu_relax();
+	}
+}
+
+static int patch_worker(void *t)
+{
+	int cpu = smp_processor_id();
+	struct text_poke_state *tps = t;
+
+	if (cpu == tps->primary_cpu) {
+		/*
+		 * Generates insns and calls text_poke_site() to do the poking
+		 * and sync.
+		 */
+		tps->patch_worker(tps);
+
+		/*
+		 * We are done patching. Switch the state to PATCH_DONE
+		 * so the secondaries can exit.
+		 */
+		smp_store_release(&tps->state, PATCH_DONE);
+	} else {
+		/* Secondary CPUs spin in a sync_core() state-machine. */
+		text_poke_sync_finish(tps);
+	}
+	return 0;
+}
+
+/**
+ * text_poke_late() -- late patching via stop_machine().
+ *
+ * Called holding the text_mutex.
+ *
+ * Return: 0 on success, -errno on failure.
+ */
+static int __maybe_unused text_poke_late(patch_worker_t worker, void *stage)
+{
+	int ret;
+
+	lockdep_assert_held(&text_mutex);
+
+	if (system_state != SYSTEM_RUNNING)
+		return -EINVAL;
+
+	text_poke_state.stage = stage;
+	text_poke_state.num_acks = cpumask_weight(cpu_online_mask);
+	text_poke_state.head = &alt_modules;
+
+	text_poke_state.patch_worker = worker;
+	text_poke_state.state = PATCH_SYNC_DONE; /* Start state */
+	text_poke_state.primary_cpu = smp_processor_id();
+
+	/*
+	 * Run the worker on all online CPUs. Don't need to do anything
+	 * for offline CPUs as they come back online with a clean cache.
+	 */
+	ret = stop_machine(patch_worker, &text_poke_state, cpu_online_mask);
+
+	return ret;
+}
+
 #ifdef CONFIG_PARAVIRT_RUNTIME
 struct paravirt_stage_entry {
 	void *dest;	/* pv_op destination */
Signature · 1 line
2.20.1
maintainer
#19 lore raw

Re: [RFC PATCH 15/26] x86/alternatives: Non-emulated text poking

Quoted 9 lines · Ankur Arora
arch/x86/kernel/alternative.c +9 -0
+static void __maybe_unused sync_one(void)
+{
+	/*
+	 * We might be executing in NMI context, and so cannot use
+	 * IRET as a synchronizing instruction.
+	 *
+	 * We could use native_write_cr2() but that is not guaranteed
+	 * to work on Xen-PV -- it is emulated by Xen and might not
+	 * execute an iret (or similar synchronizing instruction)

arch/x86/kernel/alternative.c void text_poke_sync(void) · line 1000

Quoted 10 lines
arch/x86/kernel/alternative.c +10 -0
+	 * internally.
+	 *
+	 * cpuid() would trap as well. Unclear if that's a solution
+	 * either.
+	 */
+	if (in_nmi())
+		cpuid_eax(1);
+	else
+		sync_core();
+}

That's not thinking staight; what do you think the INT3 does when it happens inside an NMI ?

maintainer
#20 lore raw

Re: [RFC PATCH 15/26] x86/alternatives: Non-emulated text poking

In reply to Ankur Arora
Quoted 12 lines · Ankur Arora
arch/x86/kernel/alternative.c +12 -0
+static int __maybe_unused text_poke_late(patch_worker_t worker, void *stage)
+{
+	int ret;
+
+	lockdep_assert_held(&text_mutex);
+
+	if (system_state != SYSTEM_RUNNING)
+		return -EINVAL;
+
+	text_poke_state.stage = stage;
+	text_poke_state.num_acks = cpumask_weight(cpu_online_mask);
+	text_poke_state.head = &alt_modules;

arch/x86/kernel/alternative.c void __ref text_poke_bp(void *addr, const void *opcode, size_t len, const void * · line 1596

Quoted 10 lines
arch/x86/kernel/alternative.c +10 -0
+
+	text_poke_state.patch_worker = worker;
+	text_poke_state.state = PATCH_SYNC_DONE; /* Start state */
+	text_poke_state.primary_cpu = smp_processor_id();
+
+	/*
+	 * Run the worker on all online CPUs. Don't need to do anything
+	 * for offline CPUs as they come back online with a clean cache.
+	 */
+	ret = stop_machine(patch_worker, &text_poke_state, cpu_online_mask);

This.. that on its own is almost a reason to NAK the entire thing. We're all working very hard to get rid of stop_machine() and you're adding one.

Worse, stop_machine() is notoriously crap on over-committed virt, the exact scenario where you want it.

Quoted 3 lines
arch/x86/kernel/alternative.c +3 -0
+
+	return ret;
+}
#21 lore raw

RFC PATCH 16/26 x86/alternatives: Add paravirt patching at runtime

In reply to Ankur Arora

Add paravirt_patch_runtime() which uses text_poke_late() to patch paravirt sites.

Also add paravirt_worker() which does the actual insn generation generate_paravirt() (which uses runtime_patch() to generate the appropriate native or paravirt insn sequences) and then calls text_poke_site() to do the actual poking.

CPU0 CPUx
---- ----

patch_worker() patch_worker()

/* Traversal, insn-gen */ text_poke_sync_finish()
tps.patch_worker()

     /* = paravirt_worker() */         /*
                                        * wait until:
     /* for each patch-site */          *  tps->state == PATCH_DONE
     generate_paravirt()                */
       runtime_patch()
     text_poke_site()
       poke_sync()

           ...                                 ...

smp_store_release(&tps->state, PATCH_DONE)

---
 arch/x86/include/asm/text-patching.h |  2 +
 arch/x86/kernel/alternative.c        | 98 +++++++++++++++++++++++++++-
 2 files changed, 99 insertions(+), 1 deletion(-)
arch/x86/include/asm/text-patching.h +2 -0
index c4b2814f2f9d..e86709a8287e 100644
@@ -21,6 +21,8 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
 #ifndef CONFIG_PARAVIRT_RUNTIME
 #define __parainstructions_runtime	NULL
 #define __parainstructions_runtime_end	NULL
+#else
+int paravirt_runtime_patch(void);
 #endif
 
 /*
arch/x86/kernel/alternative.c +97 -1

114 lines not on this page · raw message

Signature · 1 line
2.20.1
#22 lore raw

RFC PATCH 17/26 x86/alternatives: Add patching logic in text_poke_site()

In reply to Ankur Arora

Add actual poking and pipeline sync logic in poke_sync(). This is called from text_poke_site()).

The patching logic is similar to that in text_poke_bp_batch() where we patch the first byte with an INT3, which serves as a barrier, then patch the remaining bytes and then come back and fixup the first byte.

The first and the last steps are single byte writes and are thus atomic, and the second step is protected because the INT3 serves as a barrier.

Between each of these steps is a global pipeline sync which ensures that remote pipelines flush out any stale opcodes that they might have cached. This is driven from poke_sync() where the primary introduces a sync_core() on secondary CPUs for every PATCH_SYNC_* state change. The corresponding loop on the secondary executes in text_poke_sync_site().

Note that breakpoints are not handled yet.

CPU0 CPUx
---- ----

patch_worker() patch_worker()

/* Traversal, insn-gen */ text_poke_sync_finish()
tps.patch_worker() /* wait until:

     /* = paravirt_worker() */            *  tps->state == PATCH_DONE
                                          */
                  /* for each patch-site */
     generate_paravirt()
       runtime_patch()
     text_poke_site()                    text_poke_sync_site()
        poke_sync()                       /* for each:
          __text_do_poke()                 *  PATCH_SYNC_[012] */
          sync_one()                       sync_one()
          ack()                            ack()
          wait_for_acks()

           ...                                 ...

smp_store_release(&tps->state, PATCH_DONE)

---
 arch/x86/kernel/alternative.c | 103 +++++++++++++++++++++++++++++++---
 1 file changed, 95 insertions(+), 8 deletions(-)
arch/x86/kernel/alternative.c +95 -8
index 1c5acdc4f349..7fdaae9edbf0 100644
@@ -1441,27 +1441,57 @@ struct text_poke_state {
 
 static struct text_poke_state text_poke_state;
 
+static void wait_for_acks(struct text_poke_state *tps)
+{
+	int cpu = smp_processor_id();
+
+	cpumask_set_cpu(cpu, &tps->sync_ack_map);
+
+	/* Wait until all CPUs are known to have observed the state change. */
+	while (cpumask_weight(&tps->sync_ack_map) < tps->num_acks)
+		cpu_relax();
+}
+
 /**
- * poke_sync() - transitions to the specified state.
+ * poke_sync() - carries out one poke-step for a single site and
+ * transitions to the specified state.
+ * Called with the target populated in poking_mm and poking_addr.
  *
  * @tps - struct text_poke_state *
  * @state - one of PATCH_SYNC_* states
  * @offset - offset to be patched
  * @insns - insns to write
  * @len - length of insn sequence
+ *
+ * Returns after all CPUs have observed the state change and called
+ * sync_core().
  */
 static void poke_sync(struct text_poke_state *tps, int state, int offset,
 		      const char *insns, int len)
 {
+	if (len)
+		__text_do_poke(offset, insns, len);
 	/*
-	 * STUB: no patching or synchronization, just go through the
-	 * motions.
+	 * Stores to tps.sync_ack_map are ordered with
+	 * smp_load_acquire(tps->state) in text_poke_sync_site()
+	 * so we can safely clear the cpumask.
 	 */
 	smp_store_release(&tps->state, state);
+
+	cpumask_clear(&tps->sync_ack_map);
+
+	/*
+	 * Introduce a synchronizing instruction in local and remote insn
+	 * streams. This flushes any stale cached uops from CPU pipelines.
+	 */
+	sync_one();
+
+	wait_for_acks(tps);
 }
 
 /**
  * text_poke_site() - called on the primary to patch a single call site.
+ * The interlocking sync work on the secondary is done in text_poke_sync_site().
  *
  * Called in thread context with tps->state == PATCH_SYNC_DONE where it
  * takes tps->state through different PATCH_SYNC_* states, returning
@@ -1514,6 +1544,43 @@ static void __maybe_unused text_poke_site(struct text_poke_state *tps,
 			  &prev_mm, ptep);
 }
 
+/**
+ * text_poke_sync_site() -- called to synchronize the CPU pipeline
+ * on secondary CPUs for each patch site.
+ *
+ * Called in thread context with tps->state == PATCH_SYNC_0.
+ *
+ * Returns after having observed tps->state == PATCH_SYNC_DONE.
+ */
+static void text_poke_sync_site(struct text_poke_state *tps)
+{
+	int cpu = smp_processor_id();
+	int prevstate = -1;
+	int acked;
+
+	/*
+	 * In thread context we arrive here expecting tps->state to move
+	 * in-order from PATCH_SYNC_{0 -> 1 -> 2} -> PATCH_SYNC_DONE.
+	 */
+	do {
+		/*
+		 * Wait until there's some work for us to do.
+		 */
+		smp_cond_load_acquire(&tps->state,
+				      prevstate != VAL);
+
+		prevstate = READ_ONCE(tps->state);
+
+		if (prevstate < PATCH_SYNC_DONE) {
+			acked = cpumask_test_cpu(cpu, &tps->sync_ack_map);
+
+			BUG_ON(acked);
+			sync_one();
+			cpumask_set_cpu(cpu, &tps->sync_ack_map);
+		}
+	} while (prevstate < PATCH_SYNC_DONE);
+}
+
 /**
  * text_poke_sync_finish() -- called to synchronize the CPU pipeline
  * on secondary CPUs for all patch sites.
@@ -1525,6 +1592,7 @@ static void text_poke_sync_finish(struct text_poke_state *tps)
 {
 	while (true) {
 		enum patch_state state;
+		int cpu = smp_processor_id();
 
 		state = READ_ONCE(tps->state);
 
@@ -1535,11 +1603,24 @@ static void text_poke_sync_finish(struct text_poke_state *tps)
 		if (state == PATCH_DONE)
 			break;
 
-		/*
-		 * Relax here while the primary makes up its mind on
-		 * whether it is done or not.
-		 */
-		cpu_relax();
+		if (state == PATCH_SYNC_DONE) {
+			/*
+			 * Ack that we've seen the end of this iteration
+			 * and then wait until everybody's ready to move
+			 * to the next iteration or exit.
+			 */
+			cpumask_set_cpu(cpu, &tps->sync_ack_map);
+			smp_cond_load_acquire(&tps->state,
+					      (state != VAL));
+		} else if (state == PATCH_SYNC_0) {
+			/*
+			 * PATCH_SYNC_1, PATCH_SYNC_2 are handled
+			 * inside text_poke_sync_site().
+			 */
+			text_poke_sync_site(tps);
+		} else {
+			BUG();
+		}
 	}
 }
 
@@ -1549,6 +1630,12 @@ static int patch_worker(void *t)
 	struct text_poke_state *tps = t;
 
 	if (cpu == tps->primary_cpu) {
+		/*
+		 * The init state is PATCH_SYNC_DONE. Wait until the
+		 * secondaries have assembled before we start patching.
+		 */
+		wait_for_acks(tps);
+
 		/*
 		 * Generates insns and calls text_poke_site() to do the poking
 		 * and sync.
Signature · 1 line
2.20.1
#23 lore raw

RFC PATCH 18/26 x86/alternatives: Handle BP in non-emulated text poking

In reply to Ankur Arora

Handle breakpoints if we hit an INT3 either by way of an NMI while patching a site in the NMI handling path, or if we are patching text in text_poke_site() (executes on the primary), or in the pipeline sync path in text_poke_sync_site() (executes on secondary CPUs.) (The last two are not expected to happen, but see below.)

The handling on the primary CPU is to update the insn stream locally such that we can return to the primary patching loop but not force the secondary CPUs to execute sync_core().

From my reading of the Intel spec and the thread which laid down the INT3 approach: https//lore.kernel.org/lkml/[email protected], skipping the sync_core() would mean that remote pipelines -- if they have relevant uops cached would not see the updated instruction and would continue to execute stale uops.

This is safe because the primary eventually gets back to the patching loop in text_poke_site() and resumes the state-machine, re-writing some of the insn sequences just written in the BP handling and forcing the secondary CPUs to execute sync_core().

The handling on the secondary, is to call text_poke_sync_site() just as in thread-context, so it contains acking the patch states such that the primary can continue making forward progress. This can be called in a re-entrant fashion.

Note that this does mean that we cannot handle any patches in text_poke_sync_site() itself since that would end up being called recursively in the BP handler.

Control flow diagram with the BP handler:

CPU0-BP CPUx-BP
------- -------

poke_int3_native() poke_int3_native()
__text_do_poke() text_poke_sync_site()
sync_one() /* for state in:

                                         *  [PATCH_SYNC_y.._SYNC_DONE) */
                                         sync_one()
                                         ack()

CPU0 CPUx
---- ----

patch_worker() patch_worker()

/* Traversal, insn-gen */ text_poke_sync_finish()
tps.patch_worker() /* wait until:

     /* = paravirt_worker() */            *  tps->state == PATCH_DONE
                                          */
                 /* for each patch-site */
     generate_paravirt()
       runtime_patch()
     text_poke_site()                    text_poke_sync_site()
        poke_sync()                       /* for state in:
          __text_do_poke()                 *  [PATCH_SYNC_0..PATCH_SYNC_y]
          sync_one()                       */
          ack()                            sync_one()
          wait_for_acks()                  ack()

           ...                                 ...

smp_store_release(&tps->state, PATCH_DONE)

---
 arch/x86/kernel/alternative.c | 145 ++++++++++++++++++++++++++++++++--
 1 file changed, 137 insertions(+), 8 deletions(-)
arch/x86/kernel/alternative.c +137 -8
index 7fdaae9edbf0..c68d940356a2 100644
@@ -1055,6 +1055,8 @@ static int notrace patch_cmp(const void *key, const void *elt)
 }
 NOKPROBE_SYMBOL(patch_cmp);
 
+static void poke_int3_native(struct pt_regs *regs,
+			     struct text_poke_loc *tp);
 int notrace poke_int3_handler(struct pt_regs *regs)
 {
 	struct bp_patching_desc *desc;
@@ -1099,8 +1101,11 @@ int notrace poke_int3_handler(struct pt_regs *regs)
 			goto out_put;
 	}
 
-	if (desc->native)
-		BUG();
+	if (desc->native) {
+		poke_int3_native(regs, tp);
+		ret = 1; /* handled */
+		goto out_put;
+	}
 
 	len = text_opcode_size(tp->emulated.opcode);
 	ip += len;
@@ -1469,8 +1474,15 @@ static void wait_for_acks(struct text_poke_state *tps)
 static void poke_sync(struct text_poke_state *tps, int state, int offset,
 		      const char *insns, int len)
 {
-	if (len)
+	if (len) {
+		/*
+		 * Note that we could hit a BP right after patching memory
+		 * below. This could happen before the state change further
+		 * down. The primary BP handler allows us to make
+		 * forward-progress in that case.
+		 */
 		__text_do_poke(offset, insns, len);
+	}
 	/*
 	 * Stores to tps.sync_ack_map are ordered with
 	 * smp_load_acquire(tps->state) in text_poke_sync_site()
@@ -1504,11 +1516,22 @@ static void __maybe_unused text_poke_site(struct text_poke_state *tps,
 	temp_mm_state_t prev_mm;
 	pte_t *ptep;
 	int offset;
+	struct bp_patching_desc desc = {
+		.vec = tp,
+		.nr_entries = 1,
+		.native = true,
+		.refs = ATOMIC_INIT(1),
+	};
 
 	__text_poke_map(text_poke_addr(tp), tp->native.len, &prev_mm, &ptep);
 
 	offset = offset_in_page(text_poke_addr(tp));
 
+	/*
+	 * For INT3 use the same exclusion logic as BP emulation path.
+	 */
+	smp_store_release(&bp_desc, &desc); /* rcu_assign_pointer */
+
 	/*
 	 * All secondary CPUs are waiting in tps->state == PATCH_SYNC_DONE
 	 * to move to PATCH_SYNC_0. Poke the INT3 and wait until all CPUs
@@ -1537,6 +1560,19 @@ static void __maybe_unused text_poke_site(struct text_poke_state *tps,
 	 */
 	poke_sync(tps, PATCH_SYNC_DONE, 0, NULL, 0);
 
+	/*
+	 * All CPUs have ack'd PATCH_SYNC_DONE. So there can be no
+	 * laggard CPUs executing BP handlers. Reset bp_desc.
+	 */
+	WRITE_ONCE(bp_desc, NULL); /* RCU_INIT_POINTER */
+
+	/*
+	 * We've already done the synchronization so this should not
+	 * race.
+	 */
+	if (!atomic_dec_and_test(&desc.refs))
+		atomic_cond_read_acquire(&desc.refs, !VAL);
+
 	/*
 	 * Unmap the poking_addr, poking_mm.
 	 */
@@ -1548,7 +1584,8 @@ static void __maybe_unused text_poke_site(struct text_poke_state *tps,
  * text_poke_sync_site() -- called to synchronize the CPU pipeline
  * on secondary CPUs for each patch site.
  *
- * Called in thread context with tps->state == PATCH_SYNC_0.
+ * Called in thread context with tps->state == PATCH_SYNC_0 and in
+ * BP context with tps->state < PATCH_SYNC_DONE.
  *
  * Returns after having observed tps->state == PATCH_SYNC_DONE.
  */
@@ -1561,6 +1598,26 @@ static void text_poke_sync_site(struct text_poke_state *tps)
 	/*
 	 * In thread context we arrive here expecting tps->state to move
 	 * in-order from PATCH_SYNC_{0 -> 1 -> 2} -> PATCH_SYNC_DONE.
+	 *
+	 * We could also arrive here in BP-context some point after having
+	 * observed bp_patching.nr_entries (and after poking the first INT3.)
+	 * This could happen by way of an NMI while we are patching a site
+	 * that'll get executed in the NMI handler, or if we hit a site
+	 * being patched in text_poke_sync_site().
+	 *
+	 * Just as thread-context, the BP handler calls text_poke_sync_site()
+	 * to keep the primary's state-machine moving forward until it has
+	 * finished patching the call-site. At that point it is safe to
+	 * unwind the contexts.
+	 *
+	 * The second case, where we are patching a site in
+	 * text_poke_sync_site(), could end up in recursive BP handlers
+	 * and is not handled.
+	 *
+	 * Note that unlike thread-context where the start state can only
+	 * be PATCH_SYNC_0, in the BP-context, the start state could be any
+	 * PATCH_SYNC_x, so long as (state < PATCH_SYNC_DONE) since once a
+	 * CPU has acked PATCH_SYNC_2, there is no INT3 left for it to observe.
 	 */
 	do {
 		/*
@@ -1571,16 +1628,88 @@ static void text_poke_sync_site(struct text_poke_state *tps)
 
 		prevstate = READ_ONCE(tps->state);
 
-		if (prevstate < PATCH_SYNC_DONE) {
-			acked = cpumask_test_cpu(cpu, &tps->sync_ack_map);
-
-			BUG_ON(acked);
+		/*
+		 * As described above, text_poke_sync_site() gets called
+		 * from both thread-context and potentially in a re-entrant
+		 * fashion in BP-context. Accordingly expect to potentially
+		 * enter and exit this loop twice.
+		 *
+		 * Concretely, this means we need to handle the case where we
+		 * see an already acked state at BP/NMI entry and, see a
+		 * state discontinuity when returning to thread-context from
+		 * BP-context which would return after having observed
+		 * tps->state == PATCH_SYNC_DONE.
+		 *
+		 * Help this along by always exiting with tps->state ==
+		 * PATCH_SYNC_DONE but without acking it. Not acking it in
+		 * text_poke_sync_site(), guarantees that the state can only
+		 * forward once all secondary CPUs have exited both thread
+		 * and BP-contexts.
+		 */
+		acked = cpumask_test_cpu(cpu, &tps->sync_ack_map);
+		if (prevstate < PATCH_SYNC_DONE && !acked) {
 			sync_one();
 			cpumask_set_cpu(cpu, &tps->sync_ack_map);
 		}
 	} while (prevstate < PATCH_SYNC_DONE);
 }
 
+static void poke_int3_native(struct pt_regs *regs,
+			     struct text_poke_loc *tp)
+{
+	int cpu = smp_processor_id();
+	struct text_poke_state *tps = &text_poke_state;
+
+	if (cpu != tps->primary_cpu) {
+		/*
+		 * We came here from the sync loop in text_poke_sync_site().
+		 * Continue syncing. The primary is waiting.
+		 */
+		text_poke_sync_site(tps);
+	} else {
+		int offset = offset_in_page(text_poke_addr(tp));
+
+		/*
+		 * We are in the primary context and have hit the INT3 barrier
+		 * either ourselves or via an NMI.
+		 *
+		 * The secondary CPUs at this time are either in the original
+		 * text_poke_sync_site() loop or after having hit an NMI->INT3
+		 * themselves in the BP text_poke_sync_site() loop.
+		 *
+		 * The minimum that we need to do here is to update the local
+		 * insn stream such that we can return to the primary loop.
+		 * Without executing sync_core() on the secondary CPUs it is
+		 * possible that some of them might be executing stale uops in
+		 * their respective pipelines.
+		 *
+		 * This should be safe because we will get back to the patching
+		 * loop in text_poke_site() in due course and will resume
+		 * the state-machine where we left off including by re-writing
+		 * some of the insns sequences just written here.
+		 *
+		 * Note that we continue to be in poking_mm context and so can
+		 * safely call __text_do_poke() here.
+		 */
+		__text_do_poke(offset + INT3_INSN_SIZE,
+			       tp->text + INT3_INSN_SIZE,
+			       tp->native.len - INT3_INSN_SIZE);
+		__text_do_poke(offset, tp->text, INT3_INSN_SIZE);
+
+		/*
+		 * We only introduce a serializing instruction locally. As
+		 * noted above, the secondary CPUs can stay where they are --
+		 * potentially executing in the now stale INT3.) This is fine
+		 * because the primary will force the sync_core() on the
+		 * secondary CPUs once it returns.
+		 */
+		sync_one();
+	}
+
+	/* A new start */
+	regs->ip -= INT3_INSN_SIZE;
+}
+
 /**
  * text_poke_sync_finish() -- called to synchronize the CPU pipeline
  * on secondary CPUs for all patch sites.
Signature · 1 line
2.20.1
#24 lore raw

RFC PATCH 19/26 x86/alternatives: NMI safe runtime patching

In reply to Ankur Arora

Runtime patching can deadlock with multiple simultaneous NMIs. This can happen while patching inter-dependent pv-ops which are used in the NMI path (ex pv_lock_ops):

CPU0 CPUx
---- ----

patch_worker() patch_worker()

/* Traversal, insn-gen */ text_poke_sync_finish()
tps.patch_worker() /* wait until:

     /* = paravirt_worker() */           *  tps->state == PATCH_DONE
                                         */
           /* start-patching:lock.spin_unlock */
      generate_paravirt()
        runtime_patch()

      text_poke_site()                  text_poke_sync_site()
        poke_sync()                      /* for state in:
          __text_do_poke()                *  PATCH_SYNC_[012]
	  ==NMI==                         */
	                                 ==NMI==
         tries-to-acquire:nmi_lock       acquires:nmi_lock
                                         tries-to-release:nmi_lock
					 ==BP==

text_poke_sync_site()

      /* waiting-for:nmi_lock */    /* waiting-for:patched-spin_unlock() */

A similar deadlock exists if two secondary CPUs get an NMI as well.

Fix this by patching NMI-unsafe ops in an NMI context. Given that the NMI entry code ensures that NMIs do not nest, we are guaranteed that this can be done atomically.

We do this by registering a local NMI handler (text_poke_nmi()) and triggering a local NMI on the primary (via patch_worker_nmi()) which then calls the same worker (tps->patch_worker()) as in thread-context.

On the secondary, we continue with the pipeline sync loop (via text_poke_sync_finish()) in thread-context; however, if there is an NMI on the secondary, we call text_poke_sync_finish() in the handler which continues the work that was being done in thread-context.

Also note that text_poke_nmi() always executes first so we know that it takes priority over any arbitrary code executing in the installed NMI handlers.

CPU0 CPUx
---- ----

patch_worker(nmi=true) patch_worker(nmi=true)

patch_worker_nmi() -> triggers NMI text_poke_sync_finish()
/* wait for return from NMI */ /* wait until:

            ...                            *  tps->state == PATCH_DONE
                                           */

smp_store_release(&tps->state,

                     PATCH_DONE)
                                          /* for each patch-site */

                                          text_poke_sync_site()
 CPU0-NMI                                 /* for each:
 --------                                  *  PATCH_SYNC_[012]
                                           */

text_poke_nmi() sync_one()
/* Traversal, insn-gen */ ack()
tps.patch_worker()
/* = paravirt_worker() */ ...

/* for each patch-site */

     generate_paravirt()
       runtime_patch()

     text_poke_site()
       poke_sync()
         __text_do_poke()
         sync_one()
         ack()
         wait_for_acks()

          ...
---
 arch/x86/include/asm/text-patching.h |   2 +-
 arch/x86/kernel/alternative.c        | 120 ++++++++++++++++++++++++++-
 2 files changed, 117 insertions(+), 5 deletions(-)
arch/x86/include/asm/text-patching.h +1 -1
index e86709a8287e..9ba329bf9479 100644
@@ -22,7 +22,7 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
 #define __parainstructions_runtime	NULL
 #define __parainstructions_runtime_end	NULL
 #else
-int paravirt_runtime_patch(void);
+int paravirt_runtime_patch(bool nmi);
 #endif
 
 /*
arch/x86/kernel/alternative.c +116 -4

179 lines not on this page · raw message

Signature · 1 line
2.20.1
maintainer
#25 lore raw

Re: [RFC PATCH 19/26] x86/alternatives: NMI safe runtime patching

Quoted 10 lines · Ankur Arora
arch/x86/kernel/alternative.c +5 -0
@@ -1807,12 +1911,20 @@ static int __maybe_unused text_poke_late(patch_worker_t worker, void *stage)
 	text_poke_state.state = PATCH_SYNC_DONE; /* Start state */
 	text_poke_state.primary_cpu = smp_processor_id();
 
+	text_poke_state.nmi_context = nmi;
+
+	if (nmi)
+		register_nmi_handler(NMI_LOCAL, text_poke_nmi,
+				     NMI_FLAG_FIRST, "text_poke_nmi");
 	/*

arch/x86/kernel/alternative.c static int __maybe_unused text_poke_late(patch_worker_t worker, void *stage) · line 1929

Quoted 10 lines
arch/x86/kernel/alternative.c +3 -0
 	 * Run the worker on all online CPUs. Don't need to do anything
 	 * for offline CPUs as they come back online with a clean cache.
 	 */
 	ret = stop_machine(patch_worker, &text_poke_state, cpu_online_mask);
 
+	if (nmi)
+		unregister_nmi_handler(NMI_LOCAL, "text_poke_nmi");
+
 	return ret;
 }

This is completely bonghits crazy.

#26 lore raw

RFC PATCH 20/26 x86/paravirt: Enable pv-spinlocks in runtime_patch()

In reply to Ankur Arora

Enable runtime patching of paravirt spinlocks. These can be trivially enabled because pv_lock_ops are never preemptible -- preemption is disabled at entry to spin_lock*().

Note that a particular CPU instance might get preempted in the host but because runtime_patching() is called via stop_machine(), the migration thread would flush out any kernel threads preempted in the host.

---
 arch/x86/include/asm/paravirt.h  | 10 +++++-----
 arch/x86/kernel/paravirt_patch.c | 12 ++++++++++++
 kernel/locking/lock_events.c     |  2 +-
 3 files changed, 18 insertions(+), 6 deletions(-)
arch/x86/include/asm/paravirt.h +5 -5
index 694d8daf4983..cb3d0a91c060 100644
@@ -642,27 +642,27 @@ static inline void __set_fixmap(unsigned /* enum fixed_addresses */ idx,
 static __always_inline void pv_queued_spin_lock_slowpath(struct qspinlock *lock,
 							u32 val)
 {
-	PVOP_VCALL2(lock.queued_spin_lock_slowpath, lock, val);
+	PVRTOP_VCALL2(lock.queued_spin_lock_slowpath, lock, val);
 }
 
 static __always_inline void pv_queued_spin_unlock(struct qspinlock *lock)
 {
-	PVOP_VCALLEE1(lock.queued_spin_unlock, lock);
+	PVRTOP_VCALLEE1(lock.queued_spin_unlock, lock);
 }
 
 static __always_inline void pv_wait(u8 *ptr, u8 val)
 {
-	PVOP_VCALL2(lock.wait, ptr, val);
+	PVRTOP_VCALL2(lock.wait, ptr, val);
 }
 
 static __always_inline void pv_kick(int cpu)
 {
-	PVOP_VCALL1(lock.kick, cpu);
+	PVRTOP_VCALL1(lock.kick, cpu);
 }
 
 static __always_inline bool pv_vcpu_is_preempted(long cpu)
 {
-	return PVOP_CALLEE1(bool, lock.vcpu_is_preempted, cpu);
+	return PVRTOP_CALLEE1(bool, lock.vcpu_is_preempted, cpu);
 }
 
 void __raw_callee_save___native_queued_spin_unlock(struct qspinlock *lock);
arch/x86/kernel/paravirt_patch.c +12 -0
index 3eb8c0e720b4..3f8606f2811c 100644
@@ -152,6 +152,18 @@ int runtime_patch(u8 type, void *insn_buff, void *op,
 
 	/* Nothing whitelisted for now. */
 	switch (type) {
+#ifdef CONFIG_PARAVIRT_SPINLOCKS
+	/*
+	 * Preemption is always disabled in the lifetime of a spinlock
+	 * (whether held or while waiting to acquire.)
+	 */
+	case PARAVIRT_PATCH(lock.queued_spin_lock_slowpath):
+	case PARAVIRT_PATCH(lock.queued_spin_unlock):
+	case PARAVIRT_PATCH(lock.wait):
+	case PARAVIRT_PATCH(lock.kick):
+	case PARAVIRT_PATCH(lock.vcpu_is_preempted):
+		break;
+#endif
 	default:
 		pr_warn("type=%d unsuitable for runtime-patching\n", type);
 		return -EINVAL;
kernel/locking/lock_events.c +1 -1
index fa2c2f951c6b..c3057e82e6f9 100644
@@ -115,7 +115,7 @@ static const struct file_operations fops_lockevent = {
 	.llseek = default_llseek,
 };
 
-#ifdef CONFIG_PARAVIRT_SPINLOCKS
+#if defined(CONFIG_PARAVIRT_SPINLOCKS) && !defined(CONFIG_PARAVIRT_RUNTIME)
 #include <asm/paravirt.h>
 
 static bool __init skip_lockevent(const char *name)
Signature · 1 line
2.20.1
#27 lore raw

RFC PATCH 21/26 x86/alternatives: Paravirt runtime selftest

In reply to Ankur Arora

Add a selftest that triggers paravirt_runtime_patch() which toggles between the paravirt and native pv_lock_ops.

The selftest also register an NMI handler, which exercises the patched pv-ops by spin-lock operations. These are triggered via artificially sent NMIs.

And last, introduce patch sites in the primary and secondary patching code which are hit while during the patching process.

---
 arch/x86/Kconfig.debug        |  13 ++
 arch/x86/kernel/Makefile      |   1 +
 arch/x86/kernel/alternative.c |  20 +++
 arch/x86/kernel/kvm.c         |   4 +-
 arch/x86/kernel/pv_selftest.c | 264 ++++++++++++++++++++++++++++++++++
 arch/x86/kernel/pv_selftest.h |  15 ++
 6 files changed, 315 insertions(+), 2 deletions(-)
 create mode 100644 arch/x86/kernel/pv_selftest.c
 create mode 100644 arch/x86/kernel/pv_selftest.h
arch/x86/Kconfig.debug +13 -0
index 2e74690b028a..82a8e3fa68c7 100644
@@ -252,6 +252,19 @@ config X86_DEBUG_FPU
 
 	  If unsure, say N.
 
+config DEBUG_PARAVIRT_SELFTEST
+	bool "Enable paravirt runtime selftest"
+	depends on PARAVIRT
+	depends on PARAVIRT_RUNTIME
+	depends on PARAVIRT_SPINLOCKS
+	depends on KVM_GUEST
+	help
+	  This option enables sanity testing of the runtime paravirtualized
+	  patching code. Triggered via debugfs.
+
+	  Might help diagnose patching problems in different
+	  configurations and loads.
+
 config PUNIT_ATOM_DEBUG
 	tristate "ATOM Punit debug driver"
 	depends on PCI
arch/x86/kernel/Makefile +1 -0
index ba89cabe5fcf..ed3c93681f12 100644
@@ -114,6 +114,7 @@ obj-$(CONFIG_APB_TIMER)		+= apb_timer.o
 
 obj-$(CONFIG_AMD_NB)		+= amd_nb.o
 obj-$(CONFIG_DEBUG_NMI_SELFTEST) += nmi_selftest.o
+obj-$(CONFIG_DEBUG_PARAVIRT_SELFTEST) += pv_selftest.o
 
 obj-$(CONFIG_KVM_GUEST)		+= kvm.o kvmclock.o
 obj-$(CONFIG_PARAVIRT)		+= paravirt.o paravirt_patch.o
arch/x86/kernel/alternative.c +20 -0
index 385c3e6ea925..26407d7a54db 100644
@@ -26,6 +26,7 @@
 #include <asm/insn.h>
 #include <asm/io.h>
 #include <asm/fixmap.h>
+#include "pv_selftest.h"
 
 int __read_mostly alternatives_patched;
 
@@ -1549,6 +1550,12 @@ static void __maybe_unused text_poke_site(struct text_poke_state *tps,
 	 */
 	poke_sync(tps, PATCH_SYNC_0, offset, &int3, INT3_INSN_SIZE);
 
+	/*
+	 * We have an INT3 in place; execute a contrived selftest that
+	 * has an insn sequence that is under patching.
+	 */
+	pv_selftest_primary();
+
 	/* Poke remaining */
 	poke_sync(tps, PATCH_SYNC_1, offset + INT3_INSN_SIZE,
 		  tp->text + INT3_INSN_SIZE, tp->native.len - INT3_INSN_SIZE);
@@ -1634,6 +1641,19 @@ static void text_poke_sync_site(struct text_poke_state *tps)
 		smp_cond_load_acquire(&tps->state,
 				      prevstate != VAL);
 
+		/*
+		 * Send an NMI to one of the other CPUs.
+		 */
+		pv_selftest_send_nmi();
+
+		/*
+		 * We have an INT3 in place; execute a contrived selftest that
+		 * has an insn sequence that is under patching.
+		 *
+		 * Note that this function is also called from BP fixup but
+		 * is just an NOP when called from there.
+		 */
+		pv_selftest_secondary();
 		prevstate = READ_ONCE(tps->state);
 
 		/*
arch/x86/kernel/kvm.c +2 -2
index 6efe0410fb72..e56d263159d7 100644
@@ -779,7 +779,7 @@ arch_initcall(kvm_alloc_cpumask);
 #ifdef CONFIG_PARAVIRT_SPINLOCKS
 
 /* Kick a cpu by its apicid. Used to wake up a halted vcpu */
-static void kvm_kick_cpu(int cpu)
+void kvm_kick_cpu(int cpu)
 {
 	int apicid;
 	unsigned long flags = 0;
@@ -790,7 +790,7 @@ static void kvm_kick_cpu(int cpu)
 
 #include <asm/qspinlock.h>
 
-static void kvm_wait(u8 *ptr, u8 val)
+void kvm_wait(u8 *ptr, u8 val)
 {
 	unsigned long flags;
 
arch/x86/kernel/pv_selftest.c new file +264 -0

264 lines not on this page · raw message

arch/x86/kernel/pv_selftest.h new file +15 -0
new file mode 100644
index 000000000000..5afa0f7db5cc
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _PVR_SELFTEST_H
+#define _PVR_SELFTEST_H
+
+#ifdef CONFIG_DEBUG_PARAVIRT_SELFTEST
+void pv_selftest_send_nmi(void);
+void pv_selftest_primary(void);
+void pv_selftest_secondary(void);
+#else
+static inline void pv_selftest_send_nmi(void) { }
+static inline void pv_selftest_primary(void) { }
+static inline void pv_selftest_secondary(void) { }
+#endif /*! CONFIG_DEBUG_PARAVIRT_SELFTEST */
+
+#endif /* _PVR_SELFTEST_H */
Signature · 1 line
2.20.1
#28 lore raw

RFC PATCH 22/26 kvm/paravirt: Encapsulate KVM pv switching logic

In reply to Ankur Arora

KVM pv-ops are dependent on KVM features/hints which are exposed via CPUID. Decouple the probing and the enabling of these ops from __init so they can be called post-init as well.

---
 arch/x86/Kconfig      |  1 +
 arch/x86/kernel/kvm.c | 87 ++++++++++++++++++++++++++++++-------------
 2 files changed, 63 insertions(+), 25 deletions(-)
arch/x86/Kconfig +1 -0
index 605619938f08..e0629558b6b5 100644
@@ -809,6 +809,7 @@ config KVM_GUEST
 	depends on PARAVIRT
 	select PARAVIRT_CLOCK
 	select ARCH_CPUIDLE_HALTPOLL
+	select PARAVIRT_RUNTIME
 	default y
 	---help---
 	  This option enables various optimizations for running under the KVM
arch/x86/kernel/kvm.c +62 -25

141 lines not on this page · raw message

Signature · 1 line
2.20.1
#29 lore raw

RFC PATCH 23/26 x86/kvm: Add worker to trigger runtime patching

In reply to Ankur Arora

Make __pv_init_lock_hash() conditional on either paravirt spinlocks being enabled (via kvm_pv_spinlock()) or if paravirt spinlocks might get enabled (runtime patching via CONFIG_PARAVIRT_RUNTIME.)

Also add a handler for CPUID reprobe which can trigger this patching.

---
 arch/x86/kernel/kvm.c | 34 +++++++++++++++++++++++++++++-----
 1 file changed, 29 insertions(+), 5 deletions(-)
arch/x86/kernel/kvm.c +29 -5
index 31f5ecfd3907..1cb7eab805a6 100644
@@ -35,6 +35,7 @@
 #include <asm/hypervisor.h>
 #include <asm/tlb.h>
 #include <asm/cpuidle_haltpoll.h>
+#include <asm/text-patching.h>
 
 static int kvmapf = 1;
 
@@ -909,12 +910,15 @@ void __init kvm_spinlock_init(void)
 	if (num_possible_cpus() == 1)
 		return;
 
-	if (!kvm_pv_spinlock())
-		return;
-
-	__pv_init_lock_hash();
+	/*
+	 * Allocate if pv_spinlocks are enabled or if we might
+	 * end up patching them in later.
+	 */
+	if (kvm_pv_spinlock() || IS_ENABLED(CONFIG_PARAVIRT_RUNTIME))
+		__pv_init_lock_hash();
 }
-
+#else	/* !CONFIG_PARAVIRT_SPINLOCKS */
+static inline bool kvm_pv_spinlock(void) { return false; }
 #endif	/* CONFIG_PARAVIRT_SPINLOCKS */
 
 #ifdef CONFIG_ARCH_CPUIDLE_HALTPOLL
@@ -952,3 +956,23 @@ void arch_haltpoll_disable(unsigned int cpu)
 }
 EXPORT_SYMBOL_GPL(arch_haltpoll_disable);
 #endif
+
+#ifdef CONFIG_PARAVIRT_RUNTIME
+void kvm_trigger_reprobe_cpuid(struct work_struct *work)
+{
+	mutex_lock(&text_mutex);
+
+	paravirt_stage_zero();
+
+	kvm_pv_steal_clock();
+	kvm_pv_tlb();
+	paravirt_runtime_patch(false);
+
+	paravirt_stage_zero();
+
+	kvm_pv_spinlock();
+	paravirt_runtime_patch(true);
+
+	mutex_unlock(&text_mutex);
+}
+#endif /* CONFIG_PARAVIRT_RUNTIME */
Signature · 1 line
2.20.1
#30 lore raw

RFC PATCH 24/26 x86/kvm: Support dynamic CPUID hints

In reply to Ankur Arora

Change in the state of a KVM hint like KVM_HINTS_REALTIME can lead to significant performance impact. Given that the hint might not be stable across the lifetime of a guest, dynamic hints allow the host to inform the guest if the hint changes.

Do this via KVM CPUID leaf in %ecx. If the guest has registered a callback via MSR_KVM_HINT_VECTOR, the hint change is notified to it by means of a callback triggered via vcpu ioctl KVM_CALLBACK.

---
The callback vector is currently tied in with the hint notification and can (should) be made more generic such that we could deliver arbitrary callbacks on it.

One use might be for TSC frequency switching notifications support for emulated Hyper-V guests.

---
 Documentation/virt/kvm/api.rst       | 17 ++++++++++++
 Documentation/virt/kvm/cpuid.rst     |  9 +++++--
 arch/x86/include/asm/kvm_host.h      |  6 +++++
 arch/x86/include/uapi/asm/kvm_para.h |  2 ++
 arch/x86/kvm/cpuid.c                 |  3 ++-
 arch/x86/kvm/x86.c                   | 39 ++++++++++++++++++++++++++++
 include/uapi/linux/kvm.h             |  4 +++
 7 files changed, 77 insertions(+), 3 deletions(-)
Documentation/virt/kvm/api.rst +17 -0
index efbbe570aa9b..40a9b22d6979 100644
@@ -4690,6 +4690,17 @@ KVM_PV_VM_VERIFY
   Verify the integrity of the unpacked image. Only if this succeeds,
   KVM is allowed to start protected VCPUs.
 
+4.126 KVM_CALLBACK
+------------------
+
+:Capability: KVM_CAP_CALLBACK
+:Architectures: x86
+:Type: vcpu ioctl
+:Parameters: none
+:Returns: 0 on success, -1 on error
+
+Queues a callback on the guess's vcpu if a callback has been regisered.
+
 
 5. The kvm_run structure
 ========================
@@ -6109,3 +6120,9 @@ KVM can therefore start protected VMs.
 This capability governs the KVM_S390_PV_COMMAND ioctl and the
 KVM_MP_STATE_LOAD MP_STATE. KVM_SET_MP_STATE can fail for protected
 guests when the state change is invalid.
+
+8.24 KVM_CAP_CALLBACK
+
+Architectures: x86_64
+
+This capability indicates that the ioctl KVM_CALLBACK is available.
Documentation/virt/kvm/cpuid.rst +7 -2
index 01b081f6e7ea..5a997c9e74c0 100644
@@ -86,6 +86,9 @@ KVM_FEATURE_PV_SCHED_YIELD        13          guest checks this feature bit
                                               before using paravirtualized
                                               sched yield.
 
+KVM_FEATURE_DYNAMIC_HINTS	  14	      guest handles feature hints
+					      changing under it.
+
 KVM_FEATURE_CLOCSOURCE_STABLE_BIT 24          host will warn if no guest-side
                                               per-cpu warps are expeced in
                                               kvmclock
@@ -93,9 +96,11 @@ KVM_FEATURE_CLOCSOURCE_STABLE_BIT 24          host will warn if no guest-side
 
 ::
 
-      edx = an OR'ed group of (1 << flag)
+      ecx, edx = an OR'ed group of (1 << flag)
 
-Where ``flag`` here is defined as below:
+Where the ``flag`` in ecx is currently applicable hints, and ``flag`` in
+edx is the union of all hints ever provided to the guest, both drawn from
+the set listed below:
 
 ================== ============ =================================
 flag               value        meaning
arch/x86/include/asm/kvm_host.h +6 -0
index 42a2d0d3984a..4f061550274d 100644
@@ -723,6 +723,8 @@ struct kvm_vcpu_arch {
 	bool nmi_injected;    /* Trying to inject an NMI this entry */
 	bool smi_pending;    /* SMI queued after currently running handler */
 
+	bool callback_pending;	/* Callback queued after running handler */
+
 	struct kvm_mtrr mtrr_state;
 	u64 pat;
 
@@ -982,6 +984,10 @@ struct kvm_arch {
 
 	struct kvm_pmu_event_filter *pmu_event_filter;
 	struct task_struct *nx_lpage_recovery_thread;
+
+	struct {
+		u8 vector;
+	} callback;
 };
 
 struct kvm_vm_stat {
arch/x86/include/uapi/asm/kvm_para.h +2 -0
index 2a8e0b6b9805..bf016e232f2f 100644
@@ -31,6 +31,7 @@
 #define KVM_FEATURE_PV_SEND_IPI	11
 #define KVM_FEATURE_POLL_CONTROL	12
 #define KVM_FEATURE_PV_SCHED_YIELD	13
+#define KVM_FEATURE_DYNAMIC_HINTS	14
 
 #define KVM_HINTS_REALTIME      0
 
@@ -50,6 +51,7 @@
 #define MSR_KVM_STEAL_TIME  0x4b564d03
 #define MSR_KVM_PV_EOI_EN      0x4b564d04
 #define MSR_KVM_POLL_CONTROL	0x4b564d05
+#define MSR_KVM_HINT_VECTOR	0x4b564d06
 
 struct kvm_steal_time {
 	__u64 steal;
arch/x86/kvm/cpuid.c +2 -1
index 901cd1fdecd9..db6a4c4d9430 100644
@@ -712,7 +712,8 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 			     (1 << KVM_FEATURE_ASYNC_PF_VMEXIT) |
 			     (1 << KVM_FEATURE_PV_SEND_IPI) |
 			     (1 << KVM_FEATURE_POLL_CONTROL) |
-			     (1 << KVM_FEATURE_PV_SCHED_YIELD);
+			     (1 << KVM_FEATURE_PV_SCHED_YIELD) |
+			     (1 << KVM_FEATURE_DYNAMIC_HINTS);
 
 		if (sched_info_on())
 			entry->eax |= (1 << KVM_FEATURE_STEAL_TIME);
arch/x86/kvm/x86.c +39 -0
index b8124b562dea..838d033bf5ba 100644
@@ -1282,6 +1282,7 @@ static const u32 emulated_msrs_all[] = {
 
 	MSR_K7_HWCR,
 	MSR_KVM_POLL_CONTROL,
+	MSR_KVM_HINT_VECTOR,
 };
 
 static u32 emulated_msrs[ARRAY_SIZE(emulated_msrs_all)];
@@ -2910,7 +2911,15 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 
 		vcpu->arch.msr_kvm_poll_control = data;
 		break;
+	case MSR_KVM_HINT_VECTOR: {
+		u8 vector = (u8)data;
 
+		if ((u64)data > 0xffUL)
+			return 1;
+
+		vcpu->kvm->arch.callback.vector = vector;
+		break;
+	}
 	case MSR_IA32_MCG_CTL:
 	case MSR_IA32_MCG_STATUS:
 	case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1:
@@ -3156,6 +3165,9 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	case MSR_KVM_POLL_CONTROL:
 		msr_info->data = vcpu->arch.msr_kvm_poll_control;
 		break;
+	case MSR_KVM_HINT_VECTOR:
+		msr_info->data = vcpu->kvm->arch.callback.vector;
+		break;
 	case MSR_IA32_P5_MC_ADDR:
 	case MSR_IA32_P5_MC_TYPE:
 	case MSR_IA32_MCG_CAP:
@@ -3373,6 +3385,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_GET_MSR_FEATURES:
 	case KVM_CAP_MSR_PLATFORM_INFO:
 	case KVM_CAP_EXCEPTION_PAYLOAD:
+	case KVM_CAP_CALLBACK:
 		r = 1;
 		break;
 	case KVM_CAP_SYNC_REGS:
@@ -3721,6 +3734,20 @@ static int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
+static int kvm_vcpu_ioctl_callback(struct kvm_vcpu *vcpu)
+{
+	/*
+	 * Has the guest setup a callback?
+	 */
+	if (vcpu->kvm->arch.callback.vector) {
+		vcpu->arch.callback_pending = true;
+		kvm_make_request(KVM_REQ_EVENT, vcpu);
+		return 0;
+	} else {
+		return -EINVAL;
+	}
+}
+
 static int kvm_vcpu_ioctl_nmi(struct kvm_vcpu *vcpu)
 {
 	kvm_inject_nmi(vcpu);
@@ -4611,6 +4638,10 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 		r = 0;
 		break;
 	}
+	case KVM_CALLBACK: {
+		r = kvm_vcpu_ioctl_callback(vcpu);
+		break;
+	}
 	default:
 		r = -EINVAL;
 	}
@@ -7737,6 +7768,14 @@ static int inject_pending_event(struct kvm_vcpu *vcpu)
 		--vcpu->arch.nmi_pending;
 		vcpu->arch.nmi_injected = true;
 		kvm_x86_ops.set_nmi(vcpu);
+	} else if (vcpu->arch.callback_pending) {
+		if (kvm_x86_ops.interrupt_allowed(vcpu)) {
+			vcpu->arch.callback_pending = false;
+			kvm_queue_interrupt(vcpu,
+					    vcpu->kvm->arch.callback.vector,
+					    false);
+			kvm_x86_ops.set_irq(vcpu);
+		}
 	} else if (kvm_cpu_has_injectable_intr(vcpu)) {
 		/*
 		 * Because interrupts can be injected asynchronously, we are
include/uapi/linux/kvm.h +4 -0
index 428c7dde6b4b..5401c056742c 100644
@@ -1017,6 +1017,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_S390_VCPU_RESETS 179
 #define KVM_CAP_S390_PROTECTED 180
 #define KVM_CAP_PPC_SECURE_GUEST 181
+#define KVM_CAP_CALLBACK	182
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
@@ -1518,6 +1519,9 @@ struct kvm_pv_cmd {
 /* Available with KVM_CAP_S390_PROTECTED */
 #define KVM_S390_PV_COMMAND		_IOWR(KVMIO, 0xc5, struct kvm_pv_cmd)
 
+/* Available with  KVM_CAP_CALLBACK */
+#define KVM_CALLBACK		  _IO(KVMIO,  0xc6)
+
 /* Secure Encrypted Virtualization command */
 enum sev_cmd_id {
 	/* Guest initialization commands */
Signature · 1 line
2.20.1
#31 lore raw

RFC PATCH 25/26 x86/kvm: Guest support for dynamic hints

In reply to Ankur Arora

If the hypervisor supports KVM_FEATURE_DYNAMIC_HINTS, then register a callback vector (currently chosen to be HYPERVISOR_CALLBACK_VECTOR.) The callback triggers on a change in the active hints which are are exported via KVM CPUID in %ecx.

Trigger re-evaluation of KVM_HINTS based on change in their active status.

---
 arch/x86/Kconfig                |  1 +
 arch/x86/entry/entry_64.S       |  5 +++
 arch/x86/include/asm/kvm_para.h |  7 ++++
 arch/x86/kernel/kvm.c           | 58 ++++++++++++++++++++++++++++++---
 include/asm-generic/kvm_para.h  |  4 +++
 include/linux/kvm_para.h        |  5 +++
 6 files changed, 76 insertions(+), 4 deletions(-)
arch/x86/Kconfig +1 -0
index e0629558b6b5..23b239d184fc 100644
@@ -810,6 +810,7 @@ config KVM_GUEST
 	select PARAVIRT_CLOCK
 	select ARCH_CPUIDLE_HALTPOLL
 	select PARAVIRT_RUNTIME
+	select X86_HV_CALLBACK_VECTOR
 	default y
 	---help---
 	  This option enables various optimizations for running under the KVM
arch/x86/entry/entry_64.S +5 -0
index 0e9504fabe52..96b2a243c54f 100644
@@ -1190,6 +1190,11 @@ apicinterrupt3 HYPERVISOR_CALLBACK_VECTOR \
 	acrn_hv_callback_vector acrn_hv_vector_handler
 #endif
 
+#if IS_ENABLED(CONFIG_KVM_GUEST)
+apicinterrupt3 HYPERVISOR_CALLBACK_VECTOR \
+	kvm_callback_vector kvm_do_callback
+#endif
+
 idtentry debug			do_debug		has_error_code=0	paranoid=1 shift_ist=IST_INDEX_DB ist_offset=DB_STACK_OFFSET
 idtentry int3			do_int3			has_error_code=0	create_gap=1
 idtentry stack_segment		do_stack_segment	has_error_code=1
arch/x86/include/asm/kvm_para.h +7 -0
index 9b4df6eaa11a..5a7ca5639c2e 100644
@@ -88,11 +88,13 @@ static inline long kvm_hypercall4(unsigned int nr, unsigned long p1,
 bool kvm_para_available(void);
 unsigned int kvm_arch_para_features(void);
 unsigned int kvm_arch_para_hints(void);
+unsigned int kvm_arch_para_active_hints(void);
 void kvm_async_pf_task_wait(u32 token, int interrupt_kernel);
 void kvm_async_pf_task_wake(u32 token);
 u32 kvm_read_and_reset_pf_reason(void);
 extern void kvm_disable_steal_time(void);
 void do_async_page_fault(struct pt_regs *regs, unsigned long error_code, unsigned long address);
+void kvm_callback_vector(struct pt_regs *regs);
 
 #ifdef CONFIG_PARAVIRT_SPINLOCKS
 void __init kvm_spinlock_init(void);
@@ -121,6 +123,11 @@ static inline unsigned int kvm_arch_para_hints(void)
 	return 0;
 }
 
+static inline unsigned int kvm_arch_para_active_hints(void)
+{
+	return 0;
+}
+
 static inline u32 kvm_read_and_reset_pf_reason(void)
 {
 	return 0;
arch/x86/kernel/kvm.c +54 -4

116 lines not on this page · raw message

include/asm-generic/kvm_para.h +4 -0
index 728e5c5706c4..4a575299ad62 100644
@@ -24,6 +24,10 @@ static inline unsigned int kvm_arch_para_hints(void)
 	return 0;
 }
 
+static inline unsigned int kvm_arch_para_active_hints(void)
+{
+	return 0;
+}
 static inline bool kvm_para_available(void)
 {
 	return false;
include/linux/kvm_para.h +5 -0
index f23b90b02898..c98d3944d25a 100644
@@ -14,4 +14,9 @@ static inline bool kvm_para_has_hint(unsigned int feature)
 {
 	return !!(kvm_arch_para_hints() & (1UL << feature));
 }
+
+static inline bool kvm_para_has_active_hint(unsigned int feature)
+{
+	return !!(kvm_arch_para_active_hints() & BIT(feature));
+}
 #endif /* __LINUX_KVM_PARA_H */
Signature · 1 line
2.20.1
#32 lore raw

RFC PATCH 26/26 x86/kvm: Add hint change notifier for KVM_HINT_REALTIME

In reply to Ankur Arora

Add a blocking notifier that triggers when the host sends a hint change notification.

---
 arch/x86/include/asm/kvm_para.h | 10 ++++++++++
 arch/x86/kernel/kvm.c           | 16 ++++++++++++++++
 include/asm-generic/kvm_para.h  |  8 ++++++++
 3 files changed, 34 insertions(+)
arch/x86/include/asm/kvm_para.h +12 -0
index 5a7ca5639c2e..54c3c7a3225e 100644
@@ -2,6 +2,7 @@
 #ifndef _ASM_X86_KVM_PARA_H
 #define _ASM_X86_KVM_PARA_H
 
+#include <linux/notifier.h>
 #include <asm/processor.h>
 #include <asm/alternative.h>
 #include <uapi/asm/kvm_para.h>
@@ -96,6 +97,9 @@ extern void kvm_disable_steal_time(void);
 void do_async_page_fault(struct pt_regs *regs, unsigned long error_code, unsigned long address);
 void kvm_callback_vector(struct pt_regs *regs);
 
+void kvm_realtime_notifier_register(struct notifier_block *nb);
+void kvm_realtime_notifier_unregister(struct notifier_block *nb);
+
 #ifdef CONFIG_PARAVIRT_SPINLOCKS
 void __init kvm_spinlock_init(void);
 #else /* !CONFIG_PARAVIRT_SPINLOCKS */
@@ -137,6 +141,14 @@ static inline void kvm_disable_steal_time(void)
 {
 	return;
 }
+
+static inline void kvm_realtime_notifier_register(struct notifier_block *nb)
+{
+}
+
+static inline void kvm_realtime_notifier_unregister(struct notifier_block *nb)
+{
+}
 #endif
 
 #endif /* _ASM_X86_KVM_PARA_H */
arch/x86/kernel/kvm.c +16 -0
index 163b7a7ec5f9..35ba4a837027 100644
@@ -951,6 +951,20 @@ void __init kvm_spinlock_init(void)
 static inline bool kvm_pv_spinlock(void) { return false; }
 #endif	/* CONFIG_PARAVIRT_SPINLOCKS */
 
+static BLOCKING_NOTIFIER_HEAD(realtime_notifier);
+
+void kvm_realtime_notifier_register(struct notifier_block *nb)
+{
+	blocking_notifier_chain_register(&realtime_notifier, nb);
+}
+EXPORT_SYMBOL_GPL(kvm_realtime_notifier_register);
+
+void kvm_realtime_notifier_unregister(struct notifier_block *nb)
+{
+	blocking_notifier_chain_unregister(&realtime_notifier, nb);
+}
+EXPORT_SYMBOL_GPL(kvm_realtime_notifier_unregister);
+
 #ifdef CONFIG_ARCH_CPUIDLE_HALTPOLL
 
 static void kvm_disable_host_haltpoll(void *i)
@@ -1004,6 +1018,8 @@ void kvm_trigger_reprobe_cpuid(struct work_struct *work)
 	paravirt_runtime_patch(true);
 
 	mutex_unlock(&text_mutex);
+
+	blocking_notifier_call_chain(&realtime_notifier, 0, NULL);
 }
 
 static DECLARE_WORK(trigger_reprobe, kvm_trigger_reprobe_cpuid);
include/asm-generic/kvm_para.h +8 -0
index 4a575299ad62..d443531b49ac 100644
@@ -33,4 +33,12 @@ static inline bool kvm_para_available(void)
 	return false;
 }
 
+static inline void kvm_realtime_notifier_register(struct notifier_block *nb)
+{
+}
+
+static inline void kvm_realtime_notifier_unregister(struct notifier_block *nb)
+{
+}
+
 #endif
Signature · 1 line
2.20.1
maintainer
#33 lore raw
In reply to Ankur Arora
Quoted 6 lines · Ankur Arora

A KVM host (or another hypervisor) might advertise paravirtualized features and optimization hints (ex KVM_HINTS_REALTIME) which might become stale over the lifetime of the guest. For instance, the host might go from being undersubscribed to being oversubscribed (or the other way round) and it would make sense for the guest switch pv-ops based on that.

So what, the paravirt spinlock stuff works just fine when you're not oversubscribed.

Quoted 2 lines

We keep an interesting subset of pv-ops (pv_lock_ops only for now, but PV-TLB ops are also good candidates)

The PV-TLB ops also work just fine when not oversubscribed. IIRC kvm_flush_tlb_others() is pretty much the same in that case.

Quoted 6 lines

in .parainstructions.runtime,
while discarding the .parainstructions as usual at init. This is then used for switching back and forth between native and paravirt mode. ([1] lists some representative numbers of the increased memory footprint.)

Quoted 9 lines

Mechanism: the patching itself is done using stop_machine(). That is not ideal -- text_poke_stop_machine() was replaced with INT3+emulation via text_poke_bp(), but I'm using this to address two issues:
1) emulation in text_poke() can only easily handle a small set of instructions and this is problematic for inlined pv-ops (and see a possible alternatives use-case below.)
2) paravirt patching might have inter-dependendent ops (ex. lock.queued_lock_slowpath, lock.queued_lock_unlock are paired and need to be updated atomically.)

And then you hope that the spinlock state transfers.. That is that both implementations agree what an unlocked spinlock looks like.

Suppose the native one was a ticket spinlock, where unlocked means 'head == tail' while the paravirt one is a test-and-set spinlock, where unlocked means 'val == 0'.

That just happens to not be the case now, but it was for a fair while.

Quoted 5 lines

The alternative use-case is a runtime version of apply_alternatives() (not posted with this patch-set) that can be used for some safe subset of X86_FEATUREs. This could be useful in conjunction with the ongoing late microcode loading work that Mihai Carabas and others have been working on.

The whole late-microcode loading stuff is crazy already; you're making it take double bonghits.

Quoted 2 lines

Also, there are points of similarity with the ongoing static_call work which does rewriting of indirect calls.

Only in so far as that code patching is involved. An analogy would be comparing having a beer with shooting dope. They're both 'drugs'.

Quoted 9 lines

The difference here is that
we need to switch a group of calls atomically and given that some of them can be inlined, need to handle a wider variety of opcodes.

To patch safely we need to satisfy these constraints:

- No references to insn sequences under replacement on any kernel stack once replacement is in progress. Without this constraint we might end up returning to an address that is in the middle of an instruction.

Both ftrace and optprobes have that issue, neither of them are quite as crazy as this.

Quoted 3 lines

- handle inter-dependent ops: as above, lock.queued_lock_unlock(), lock.queued_lock_slowpath() and the rest of the pv_lock_ops are a good example.

While I'm sure this is a fun problem, why are we solving it?

Quoted 2 lines

- handle a broader set of insns than CALL and JMP: some pv-ops end up getting inlined. Alternatives can contain arbitrary instructions.

So can optprobes.

Quoted 2 lines

- locking operations can be called from interrupt handlers which means we cannot trivially use IPIs for flushing.

Heck, some NMI handlers use locks..

Quoted 1 line

Handling these, necessitates that target pv-ops not be preemptible.

I don't think that is a correct inferrence.

Quoted 11 lines

Once that is a given (for safety these need to be explicitly whitelisted in runtime_patch()), use a state-machine with the primary CPU doing the patching and secondary CPUs in a sync_core() loop.

In case we hit an INT3/BP (in NMI or thread-context) we makes forward progress by continuing the patching instead of emulating.

One remaining issue is inter-dependent pv-ops which are also executed in the NMI handler -- patching can potentially deadlock in case of multiple NMIs. Handle these by pushing some of this work in the NMI handler where we know it will be uninterrupted.

I'm just seeing a lot of bonghits without sane rationale. Why is any of this important?

maintainer
#34 lore raw
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Ankur Arora

Mechanism: the patching itself is done using stop_machine(). That is not ideal -- text_poke_stop_machine() was replaced with INT3+emulation via text_poke_bp(), but I'm using this to address two issues:
1) emulation in text_poke() can only easily handle a small set of instructions and this is problematic for inlined pv-ops (and see a possible alternatives use-case below.)
2) paravirt patching might have inter-dependendent ops (ex. lock.queued_lock_slowpath, lock.queued_lock_unlock are paired and need to be updated atomically.)

Quoted 8 lines

And then you hope that the spinlock state transfers.. That is that both implementations agree what an unlocked spinlock looks like.

Suppose the native one was a ticket spinlock, where unlocked means 'head == tail' while the paravirt one is a test-and-set spinlock, where unlocked means 'val == 0'.

That just happens to not be the case now, but it was for a fair while.

Sure? This would mean that before spinlock-pvops are being set no lock is allowed to be used in the kernel, because this would block the boot time transition of the lock variant to use.

Another problem I'm seeing is that runtime pvops patching would rely on the fact that stop_machine() isn't guarded by a spinlock.

Juergen

maintainer
#35 lore raw
Quoted 19 lines · Jürgen Groß
Quoted 18 lines · Peter Zijlstra
Quoted 9 lines · Ankur Arora

Mechanism: the patching itself is done using stop_machine(). That is not ideal -- text_poke_stop_machine() was replaced with INT3+emulation via text_poke_bp(), but I'm using this to address two issues:
1) emulation in text_poke() can only easily handle a small set of instructions and this is problematic for inlined pv-ops (and see a possible alternatives use-case below.)
2) paravirt patching might have inter-dependendent ops (ex. lock.queued_lock_slowpath, lock.queued_lock_unlock are paired and need to be updated atomically.)

And then you hope that the spinlock state transfers.. That is that both implementations agree what an unlocked spinlock looks like.

Suppose the native one was a ticket spinlock, where unlocked means 'head == tail' while the paravirt one is a test-and-set spinlock, where unlocked means 'val == 0'.

Quoted 5 lines
Quoted 1 line

That just happens to not be the case now, but it was for a fair while.

Sure? This would mean that before spinlock-pvops are being set no lock is allowed to be used in the kernel, because this would block the boot time transition of the lock variant to use.

Hurm.. true. I suppose I completely forgot how paravirt spinlocks looked before it got rewritten.

Quoted 2 lines

Another problem I'm seeing is that runtime pvops patching would rely on the fact that stop_machine() isn't guarded by a spinlock.

It can't be, stop_machine() relies on scheduling. But yes, that another variation of 'stuff uses spinlocks'.

#36 lore raw
In reply to Peter Zijlstra
Quoted 42 lines · Peter Zijlstra
Quoted 6 lines · Ankur Arora

A KVM host (or another hypervisor) might advertise paravirtualized features and optimization hints (ex KVM_HINTS_REALTIME) which might become stale over the lifetime of the guest. For instance, the host might go from being undersubscribed to being oversubscribed (or the other way round) and it would make sense for the guest switch pv-ops based on that.

So what, the paravirt spinlock stuff works just fine when you're not oversubscribed.

Quoted 2 lines

We keep an interesting subset of pv-ops (pv_lock_ops only for now, but PV-TLB ops are also good candidates)

The PV-TLB ops also work just fine when not oversubscribed. IIRC kvm_flush_tlb_others() is pretty much the same in that case.

Quoted 6 lines

in .parainstructions.runtime,
while discarding the .parainstructions as usual at init. This is then used for switching back and forth between native and paravirt mode. ([1] lists some representative numbers of the increased memory footprint.)

Quoted 9 lines

Mechanism: the patching itself is done using stop_machine(). That is not ideal -- text_poke_stop_machine() was replaced with INT3+emulation via text_poke_bp(), but I'm using this to address two issues:
1) emulation in text_poke() can only easily handle a small set of instructions and this is problematic for inlined pv-ops (and see a possible alternatives use-case below.)
2) paravirt patching might have inter-dependendent ops (ex. lock.queued_lock_slowpath, lock.queued_lock_unlock are paired and need to be updated atomically.)

And then you hope that the spinlock state transfers.. That is that both implementations agree what an unlocked spinlock looks like.

Suppose the native one was a ticket spinlock, where unlocked means 'head == tail' while the paravirt one is a test-and-set spinlock, where unlocked means 'val == 0'.

That just happens to not be the case now, but it was for a fair while.

Quoted 8 lines
Quoted 5 lines

The alternative use-case is a runtime version of apply_alternatives() (not posted with this patch-set) that can be used for some safe subset of X86_FEATUREs. This could be useful in conjunction with the ongoing late microcode loading work that Mihai Carabas and others have been working on.

The whole late-microcode loading stuff is crazy already; you're making it take double bonghits.

That's fair. I was talking in a fairly limited sense, ex making static_cpu_has() catch up with boot_cpu_has() after a microcode update but I should have specified that.

Quoted 6 lines
Quoted 3 lines

Also, there are points of similarity with the ongoing static_call work which does rewriting of indirect calls.

Only in so far as that code patching is involved. An analogy would be comparing having a beer with shooting dope. They're both 'drugs'.

I meant closer to updating indirect pointers, like static_call_update() semantics. But of course I don't know static_call code well enough.

Quoted 5 lines
Quoted 5 lines

The difference here is that
we need to switch a group of calls atomically and given that some of them can be inlined, need to handle a wider variety of opcodes.

Quoted 8 lines
Quoted 5 lines

To patch safely we need to satisfy these constraints:

- No references to insn sequences under replacement on any kernel stack once replacement is in progress. Without this constraint we might end up returning to an address that is in the middle of an instruction.

Both ftrace and optprobes have that issue, neither of them are quite as crazy as this.

I did look at ftrace. Will look at optprobes. Thanks.

Quoted 7 lines
Quoted 4 lines

- handle inter-dependent ops: as above, lock.queued_lock_unlock(), lock.queued_lock_slowpath() and the rest of the pv_lock_ops are a good example.

While I'm sure this is a fun problem, why are we solving it?

Quoted 8 lines
Quoted 2 lines

- handle a broader set of insns than CALL and JMP: some pv-ops end up getting inlined. Alternatives can contain arbitrary instructions.

So can optprobes.>

Quoted 2 lines

- locking operations can be called from interrupt handlers which means we cannot trivially use IPIs for flushing.

Heck, some NMI handlers use locks..

This does handle the NMI locking problem. The solution -- doing it in the NMI handler was of course pretty ugly.

Quoted 3 lines
Quoted 1 line

Handling these, necessitates that target pv-ops not be preemptible.

I don't think that is a correct inferrence.The non-preemptibility requirement was to ensure that any pv-op under

replacement not be under execution after it is patched out. (Not a concern for pv_lock_ops.)

Ensuring that we don't return to an address in the middle of an instruction could be done by moving the NOPs in the prefix, but I couldn't think of any other way to ensure that a function not be under execution.

Thanks
Ankur

Quoted 15 lines
Quoted 11 lines

Once that is a given (for safety these need to be explicitly whitelisted in runtime_patch()), use a state-machine with the primary CPU doing the patching and secondary CPUs in a sync_core() loop.

In case we hit an INT3/BP (in NMI or thread-context) we makes forward progress by continuing the patching instead of emulating.

One remaining issue is inter-dependent pv-ops which are also executed in the NMI handler -- patching can potentially deadlock in case of multiple NMIs. Handle these by pushing some of this work in the NMI handler where we know it will be uninterrupted.

I'm just seeing a lot of bonghits without sane rationale. Why is any of this important?

maintainer
#37 lore raw
In reply to Ankur Arora
Quoted 3 lines · Ankur Arora

A KVM host (or another hypervisor) might advertise paravirtualized features and optimization hints (ex KVM_HINTS_REALTIME) which might become stale over the lifetime of the guest. For instance, the

Then this hint is wrong if it can't be guaranteed.

Quoted 3 lines

host might go from being undersubscribed to being oversubscribed (or the other way round) and it would make sense for the guest switch pv-ops based on that.

I think using pvops for such a feature change is just wrong.

What comes next? Using pvops for being able to migrate a guest from an Intel to an AMD machine?

...

Quoted 164 lines

There are four main sets of patches in this series:

1. PV-ops management (patches 1-10, 20): mostly infrastructure and refactoring pieces to make paravirt patching usable at runtime. For the most part scoped under CONFIG_PARAVIRT_RUNTIME.

Patches 1-7, to persist part of parainstructions in memory:
"x86/paravirt: Specify subsection in PVOP macros" "x86/paravirt: Allow paravirt patching post-init" "x86/paravirt: PVRTOP macros for PARAVIRT_RUNTIME" "x86/alternatives: Refactor alternatives_smp_module* "x86/alternatives: Rename alternatives_smp*, smp_alt_module "x86/alternatives: Remove stale symbols "x86/paravirt: Persist .parainstructions.runtime"

Patches 8-10, develop the inerfaces to safely switch pv-ops:
"x86/paravirt: Stash native pv-ops" "x86/paravirt: Add runtime_patch()" "x86/paravirt: Add primitives to stage pv-ops"

Patch 20 enables switching of pv_lock_ops:
"x86/paravirt: Enable pv-spinlocks in runtime_patch()"

2. Non-emulated text poking (patches 11-19)

Patches 11-13 are mostly refactoring to split __text_poke() into map, unmap and poke/memcpy phases with the poke portion being re-entrant
"x86/alternatives: Remove return value of text_poke*()" "x86/alternatives: Use __get_unlocked_pte() in text_poke()" "x86/alternatives: Split __text_poke()"

Patches 15, 17 add the actual poking state-machine:
"x86/alternatives: Non-emulated text poking" "x86/alternatives: Add patching logic in text_poke_site()"

with patches 14 and 18 containing the pieces for BP handling:
"x86/alternatives: Handle native insns in text_poke_loc*()" "x86/alternatives: Handle BP in non-emulated text poking"

and patch 19 provides the ability to use the state-machine above in an NMI context (fixes some potential deadlocks when handling inter- dependent operations and multiple NMIs):
"x86/alternatives: NMI safe runtime patching".

Patch 16 provides the interface (paravirt_runtime_patch()) to use the poking mechanism developed above and patch 21 adds a selftest:
"x86/alternatives: Add paravirt patching at runtime" "x86/alternatives: Paravirt runtime selftest"

3. KVM guest changes to be able to use this (patches 22-23,25-26):
"kvm/paravirt: Encapsulate KVM pv switching logic" "x86/kvm: Add worker to trigger runtime patching" "x86/kvm: Guest support for dynamic hints"
"x86/kvm: Add hint change notifier for KVM_HINT_REALTIME".

4. KVM host changes to notify the guest of a change (patch 24):
"x86/kvm: Support dynamic CPUID hints"

Testing:
With paravirt patching, the code is mostly stable on Intel and AMD systems under kernbench and locktorture with paravirt toggling (with, without synthetic NMIs) in the background.

Queued spinlock performance for locktorture is also on expected lines:

  [ 1533.221563] Writes:  Total: 1048759000  Max/Min: 0/0   Fail: 0
  # toggle PV spinlocks
  [ 1594.713699] Writes:  Total: 1111660545  Max/Min: 0/0   Fail: 0
  # PV spinlocks (in ~60 seconds) = 62,901,545
  # toggle native spinlocks
  [ 1656.117175] Writes:  Total: 1113888840  Max/Min: 0/0   Fail: 0
   # native spinlocks (in ~60 seconds) = 2,228,295

The alternatives testing is more limited with it being used to rewrite mostly harmless X86_FEATUREs with load in the background.

Patches also at:

ssh://[email protected]/terminus/linux.git alternatives-rfc-upstream-v1

Please review.

Thanks
Ankur

[1] The precise change in memory footprint depends on config options but the following example inlines queued_spin_unlock() (which forms the bulk of the added state). The added footprint is the size of the .parainstructions.runtime section:

  $ objdump -h vmlinux|grep .parainstructions
  Idx Name              		Size      VMA
  	LMA                File-off  Algn
   27 .parainstructions 		0001013c  ffffffff82895000
   	0000000002895000   01c95000  2**3
   28 .parainstructions.runtime  0000cd2c  ffffffff828a5140
   	00000000028a5140   01ca5140  2**3
   $ size vmlinux
   text       data       bss        dec      hex       filename
   13726196   12302814   14094336   40123346 2643bd2   vmlinux

Ankur Arora (26):
x86/paravirt: Specify subsection in PVOP macros x86/paravirt: Allow paravirt patching post-init x86/paravirt: PVRTOP macros for PARAVIRT_RUNTIME x86/alternatives: Refactor alternatives_smp_module* x86/alternatives: Rename alternatives_smp*, smp_alt_module x86/alternatives: Remove stale symbols x86/paravirt: Persist .parainstructions.runtime x86/paravirt: Stash native pv-ops
x86/paravirt: Add runtime_patch()
x86/paravirt: Add primitives to stage pv-ops x86/alternatives: Remove return value of text_poke*() x86/alternatives: Use __get_unlocked_pte() in text_poke() x86/alternatives: Split __text_poke() x86/alternatives: Handle native insns in text_poke_loc*() x86/alternatives: Non-emulated text poking x86/alternatives: Add paravirt patching at runtime x86/alternatives: Add patching logic in text_poke_site() x86/alternatives: Handle BP in non-emulated text poking x86/alternatives: NMI safe runtime patching x86/paravirt: Enable pv-spinlocks in runtime_patch() x86/alternatives: Paravirt runtime selftest kvm/paravirt: Encapsulate KVM pv switching logic x86/kvm: Add worker to trigger runtime patching x86/kvm: Support dynamic CPUID hints
x86/kvm: Guest support for dynamic hints x86/kvm: Add hint change notifier for KVM_HINT_REALTIME

  Documentation/virt/kvm/api.rst        |  17 +
  Documentation/virt/kvm/cpuid.rst      |   9 +-
  arch/x86/Kconfig                      |  14 +
  arch/x86/Kconfig.debug                |  13 +
  arch/x86/entry/entry_64.S             |   5 +
  arch/x86/include/asm/alternative.h    |  20 +-
  arch/x86/include/asm/kvm_host.h       |   6 +
  arch/x86/include/asm/kvm_para.h       |  17 +
  arch/x86/include/asm/paravirt.h       |  10 +-
  arch/x86/include/asm/paravirt_types.h | 230 ++++--
  arch/x86/include/asm/text-patching.h  |  18 +-
  arch/x86/include/uapi/asm/kvm_para.h  |   2 +
  arch/x86/kernel/Makefile              |   1 +
  arch/x86/kernel/alternative.c         | 987 +++++++++++++++++++++++---
  arch/x86/kernel/kvm.c                 | 191 ++++-
  arch/x86/kernel/module.c              |  42 +-
  arch/x86/kernel/paravirt.c            |  16 +-
  arch/x86/kernel/paravirt_patch.c      |  61 ++
  arch/x86/kernel/pv_selftest.c         | 264 +++++++
  arch/x86/kernel/pv_selftest.h         |  15 +
  arch/x86/kernel/setup.c               |   2 +
  arch/x86/kernel/vmlinux.lds.S         |  16 +
  arch/x86/kvm/cpuid.c                  |   3 +-
  arch/x86/kvm/x86.c                    |  39 +
  include/asm-generic/kvm_para.h        |  12 +
  include/asm-generic/vmlinux.lds.h     |   8 +
  include/linux/kvm_para.h              |   5 +
  include/linux/mm.h                    |  16 +-
  include/linux/preempt.h               |  17 +
  include/uapi/linux/kvm.h              |   4 +
  kernel/locking/lock_events.c          |   2 +-
  mm/memory.c                           |   9 +-
  32 files changed, 1850 insertions(+), 221 deletions(-)
Quoted 3 lines

create mode 100644 arch/x86/kernel/pv_selftest.c create mode 100644 arch/x86/kernel/pv_selftest.h

Quite a lot of code churn and hacks for a problem which should not occur on a well administrated machine.

Especially the NMI dependencies make me not wanting to Ack this series.

Juergen

#38 lore raw

So, first thanks for the quick comments even though some of my choices were straight NAKs (or maybe because of that!)

Second, I clearly did a bad job of motivating the series. Let me try to address the motivation comments first and then I can address the technical concerns separately.

[ I'm collating all the motivation comments below. ]

Quoted 3 lines
Quoted 3 lines

A KVM host (or another hypervisor) might advertise paravirtualized features and optimization hints (ex KVM_HINTS_REALTIME) which might become stale over the lifetime of the guest. For instance, the

Thomas> If your host changes his advertised behaviour then you want to Thomas> fix the host setup or find a competent admin.

Juergen> Then this hint is wrong if it can't be guaranteed.

I agree, the hint behaviour is wrong and the host shouldn't be giving hints it can only temporarily honor.
The host problem is hard to fix though: the behaviour change is either because of a guest migration or in case of a hosted guest, cloud economics -- customers want to go to a 2-1 or worse VCPU-CPU ratio at times of low load.

I had an offline discussion with Paolo Bonzini where he agreed that it makes sense to make KVM_HINTS_REALTIME a dynamic hint rather than static as it is now. (That was really the starting point for this series.)

Quoted 3 lines
Quoted 3 lines

host might go from being undersubscribed to being oversubscribed (or the other way round) and it would make sense for the guest switch pv-ops based on that.

Juergen> I think using pvops for such a feature change is just wrong. Juergen> What comes next? Using pvops for being able to migrate a guest Juergen> from an Intel to an AMD machine?

My statement about switching pv-ops was too broadly worded. What I meant to say was that KVM guests choose pv_lock_ops to be native or paravirt based on undersubscribed/oversubscribed hint at boot, and this choice should be available at run-time as well.

KVM chooses between native/paravirt spinlocks at boot based on this reasoning (from commit b2798ba0b8):
"Waiman Long mentioned that:

Quoted 3 lines

Generally speaking, unfair lock performs well for VMs with a small number of vCPUs. Native qspinlock may perform better than pvqspinlock if there is vCPU pinning and there is no vCPU over-commitment.

"

PeterZ> So what, the paravirt spinlock stuff works just fine when PeterZ> you're not oversubscribed.
Yeah, the paravirt spinlocks work fine for both under and oversubscribed hosts, but they are more expensive and that extra cost provides no benefits when CPUs are pinned.
For instance, pvqueued spin_unlock() is a call+locked cmpxchg as opposed to just a movb $0, (%rdi).

This difference shows up in kernbench running on a KVM guest with native and paravirt spinlocks. I ran with 8 and 64 CPU guests with CPUs pinned.

The native version performs same or better.

8 CPU       Native  (std-dev)  Paravirt (std-dev)
            -----------------  -----------------
-j  4: sys  151.89  ( 0.2462)  160.14   ( 4.8366)    +5.4%
-j 32: sys  162.715 (11.4129)  170.225  (11.1138)    +4.6%
-j  0: sys  164.193 ( 9.4063)  170.843  ( 8.9651)    +4.0%

64 CPU Native (std-dev) Paravirt (std-dev)

            -----------------  -----------------
-j  32: sys 209.448 (0.37009)  210.976   (0.4245)    +0.7%
-j 256: sys 267.401 (61.0928)  285.73   (78.8021)    +6.8%
-j   0: sys 286.313 (56.5978)  307.721  (70.9758)    +7.4%

In all cases the pv_kick, pv_wait numbers were minimal as expected. The lock_slowpath counts were higher with PV but AFAICS the native and paravirt lock_slowpath are not directly comparable.

Detailed kernbench numbers attached.

Thanks
Ankur

8-cpu-pinned,native
==================

Average Half load -j 4 Run (std deviation):
Elapsed Time 303.686 (0.737652)
User Time 1032.24 (2.8133)
System Time 151.89 (0.246272)
Percent CPU 389.2 (0.447214)
Context Switches 19350.4 (82.1785)
Sleeps 125885 (148.338)

Average Optimal load -j 32 Run (std deviation):
Elapsed Time 187.068 (0.358427)
User Time 1130.33 (103.405)
System Time 162.715 (11.4129)
Percent CPU 569.1 (189.633)
Context Switches 143301 (130656)
Sleeps 126938 (1132.83)

Average Maximal load -j Run (std deviation):
Elapsed Time 189.098 (0.316812)
User Time 1166.59 (98.4454)
System Time 164.193 (9.4063)
Percent CPU 627.133 (174.169)
Context Switches 222270 (156005)
Sleeps 122562 (6470.93)

8-cpu-pinned, pv
================

Average Half load -j 4 Run (std deviation):
Elapsed Time 309.872 (5.882)
User Time 1045.8 (18.5295)
System Time 160.14 (4.83669)
Percent CPU 388.8 (0.447214)
Context Switches 41215.4 (679.522)
Sleeps 122369 (477.593)

Average Optimal load -j 32 Run (std deviation):
Elapsed Time 190.1 (0.377823)
User Time 1144 (104.248)
System Time 170.225 (11.1138)
Percent CPU 568.2 (189.107)

Average Maximal load -j Run (std deviation):
Elapsed Time 191.606 (0.108305)
User Time 1178.83 (97.908)
System Time 170.843 (8.9651)
Percent CPU 625.8 (173.49)
Context Switches 234878 (149479)
Sleeps 120542 (6073.79)

64-cpu-pinned, native
=====================

Average Half load -j 32 Run (std deviation):
Elapsed Time 54.306 (0.134833)
User Time 1072.75 (1.34598)
System Time 209.448 (0.370095)
Percent CPU 2360.4 (4.03733)
Context Switches 26999 (99.5414)
Sleeps 122408 (184.87)

Average Optimal load -j 256 Run (std deviation):
Elapsed Time 39.424 (0.150599)
User Time 1140.91 (71.8722)
System Time 267.401 (61.0928)
Percent CPU 3125.9 (806.96)
Context Switches 129662 (108217)
Sleeps 121767 (699.198)

Average Maximal load -j Run (std deviation):
Elapsed Time 41.562 (0.206083)
User Time 1174.68 (75.9342)
System Time 286.313 (56.5978)
Percent CPU 3339.87 (719.062)
Context Switches 203428 (138536)
Sleeps 119066 (3993.58)

64-cpu-pinned, pv
================
Average Half load -j 32 Run (std deviation):
Elapsed Time 55.14 (0.0894427)
User Time 1071.99 (1.43335)
System Time 210.976 (0.424594)
Percent CPU 2326 (4.52769)
Context Switches 37544.8 (220.969)
Sleeps 115527 (94.7138)

Average Optimal load -j 256 Run (std deviation):
Elapsed Time 40.54 (0.246779)
User Time 1137.41 (68.9773)
System Time 285.73 (78.8021)
Percent CPU 3090.7 (806.218)
Context Switches 139059 (107006)
Sleeps 116962 (1518.56)

Average Maximal load -j Run (std deviation):
Elapsed Time 42.682 (0.170939)
User Time 1171.64 (74.6663)
System Time 307.721 (70.9758)
Percent CPU 3303.27 (717.418)
Context Switches 213430 (138616)
Sleeps 115143 (2930.03)

#39 lore raw
In reply to Jürgen Groß
Quoted 1 line · Jürgen Groß

On 08.04.20 07:02, Ankur Arora wrote:

[ snip ]

Quoted 3 lines

Quite a lot of code churn and hacks for a problem which should not occur on a well administrated machine.

Yeah, I agree the patch set is pretty large and clearly the NMI or the stop_machine() are completely out. That said, as I wrote in my other mail I think the problem is still worth solving.

Quoted 1 line

Especially the NMI dependencies make me not wanting to Ack this series.

The NMI solution did turn out to be pretty ugly.

I was using it to solve two problems: avoid a deadlock where an NMI handler could use a lock while the stop_machine() thread is trying to rewrite the corresponding call-sites. And, needed to ensure that we don't lock and unlock using mismatched primitives.

Thanks
Ankur

Quoted 3 lines

Juergen

#40 lore raw
In reply to Ankur Arora
Quoted 6 lines · Ankur Arora

A KVM host (or another hypervisor) might advertise paravirtualized features and optimization hints (ex KVM_HINTS_REALTIME) which might become stale over the lifetime of the guest. For instance, the host might go from being undersubscribed to being oversubscribed (or the other way round) and it would make sense for the guest switch pv-ops based on that.

If your host changes his advertised behaviour then you want to fix the host setup or find a competent admin.

Quoted 9 lines

This lockorture splat that I saw on the guest while testing this is indicative of the problem:

  [ 1136.461522] watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [lock_torture_wr:12865]
  [ 1136.461542] CPU: 8 PID: 12865 Comm: lock_torture_wr Tainted: G W L 5.4.0-rc7+ #77
  [ 1136.461546] RIP: 0010:native_queued_spin_lock_slowpath+0x15/0x220

(Caused by an oversubscribed host but using mismatched native pv_lock_ops on the gues.)

And this illustrates what? The fact that you used a misconfigured setup.

Quoted 2 lines

This series addresses the problem by doing paravirt switching at runtime.

You're not addressing the problem. Your fixing the symptom, which is wrong to begin with.

Quoted 5 lines

The alternative use-case is a runtime version of apply_alternatives() (not posted with this patch-set) that can be used for some safe subset of X86_FEATUREs. This could be useful in conjunction with the ongoing late microcode loading work that Mihai Carabas and others have been working on.

This has been discussed to death before and there is no safe subset as long as this hasn't been resolved:

https://lore.kernel.org/lkml/[email protected]/

Thanks,

tglx

#41 lore raw
Quoted 35 lines · Thomas Gleixner
Quoted 6 lines · Ankur Arora

A KVM host (or another hypervisor) might advertise paravirtualized features and optimization hints (ex KVM_HINTS_REALTIME) which might become stale over the lifetime of the guest. For instance, the host might go from being undersubscribed to being oversubscribed (or the other way round) and it would make sense for the guest switch pv-ops based on that.

If your host changes his advertised behaviour then you want to fix the host setup or find a competent admin.

Quoted 9 lines

This lockorture splat that I saw on the guest while testing this is indicative of the problem:

   [ 1136.461522] watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [lock_torture_wr:12865]
   [ 1136.461542] CPU: 8 PID: 12865 Comm: lock_torture_wr Tainted: G W L 5.4.0-rc7+ #77
   [ 1136.461546] RIP: 0010:native_queued_spin_lock_slowpath+0x15/0x220

(Caused by an oversubscribed host but using mismatched native pv_lock_ops on the gues.)

And this illustrates what? The fact that you used a misconfigured setup.

Quoted 2 lines

This series addresses the problem by doing paravirt switching at runtime.

You're not addressing the problem. Your fixing the symptom, which is wrong to begin with.

Quoted 5 lines

The alternative use-case is a runtime version of apply_alternatives() (not posted with this patch-set) that can be used for some safe subset of X86_FEATUREs. This could be useful in conjunction with the ongoing late microcode loading work that Mihai Carabas and others have been working on.

Quoted 4 lines

This has been discussed to death before and there is no safe subset as long as this hasn't been resolved:

   https://lore.kernel.org/lkml/[email protected]/

Thanks. I was thinking of fairly limited subset: ex re-evaluate X86_FEATURE_ALWAYS to make sure static_cpu_has() reflects reality but I guess that has second order effects here.

Ankur

Quoted 5 lines

Thanks,

tglx