Skip to contentSkip to conversation

KVM_GUEST support breaks page fault tracing

DISCUSSION Dave Hansen 3 participants 5 messages May 8, 2014 – May 9, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I noticed on some of my systems that page fault tracing doesn't work:

cd /sys/kernel/debug/tracing
echo 1 > events/exceptions/enable
cat trace;
# nothing shows up

I eventually traced it down to CONFIG_KVM_GUEST. At least in a KVM VM, enabling that option breaks page fault tracing, and disabling fixes it.
I tried on some old kernels and this does not appear to be a
regression: it never worked.

Anybody have any theories about what is going on?

I'm also having problems with the page fault tracing on bare metal (same symptom of no trace output). I'm unsure if it's related.

#1 lore raw

On Thu, 8 May 2014, Dave Hansen wrote:

You forgot to cc Steven :)

Quoted 17 lines

I noticed on some of my systems that page fault tracing doesn't work:

cd /sys/kernel/debug/tracing
echo 1 > events/exceptions/enable
cat trace;
# nothing shows up

I eventually traced it down to CONFIG_KVM_GUEST. At least in a KVM VM, enabling that option breaks page fault tracing, and disabling fixes it.
I tried on some old kernels and this does not appear to be a
regression: it never worked.

Anybody have any theories about what is going on?

I'm also having problems with the page fault tracing on bare metal (same symptom of no trace output). I'm unsure if it's related.

#2 lore raw
Quoted 7 lines · Thomas Gleixner
Quoted 7 lines
Quoted 7 lines

I noticed on some of my systems that page fault tracing doesn't work:

cd /sys/kernel/debug/tracing
echo 1 > events/exceptions/enable
cat trace;
# nothing shows up

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

I eventually traced it down to CONFIG_KVM_GUEST. At least in a KVM VM, enabling that option breaks page fault tracing, and disabling fixes it.
I tried on some old kernels and this does not appear to be a
regression: it never worked.

Anybody have any theories about what is going on?

Looks like the KVM code calls do_page_fault() directly:

Quoted 10 lines

dotraplinkage void __kprobes
do_async_page_fault(struct pt_regs *regs, unsigned long error_code)

{
        enum ctx_state prev_state;

        switch (kvm_read_and_reset_pf_reason()) {
        default:
                do_page_fault(regs, error_code);
                break;
        case KVM_PV_REASON_PAGE_NOT_PRESENT:

That seems to explain my problems in a VM. Any objections to doing something like the attached patch?

---

 b/arch/x86/include/asm/traps.h |    5 +++++
 b/arch/x86/kernel/kvm.c        |    2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)
arch/x86/kernel/kvm.c renamed from arch/x86/kernel/kvm.c~muck-with-kvm-guest-code +1 -1
diff -puN arch/x86/kernel/kvm.c~muck-with-kvm-guest-code arch/x86/kernel/kvm.c
@@ -259,7 +259,7 @@ do_async_page_fault(struct pt_regs *regs
 
 	switch (kvm_read_and_reset_pf_reason()) {
 	default:
-		do_page_fault(regs, error_code);
+		trace_do_page_fault(regs, error_code);
 		break;
 	case KVM_PV_REASON_PAGE_NOT_PRESENT:
 		/* page is swapped out by the host. */
arch/x86/include/asm/traps.h renamed from arch/x86/include/asm/traps.h~muck-with-kvm-guest-code +5 -0
diff -puN arch/x86/include/asm/traps.h~muck-with-kvm-guest-code arch/x86/include/asm/traps.h
@@ -74,6 +74,11 @@ dotraplinkage void do_general_protection
 dotraplinkage void do_page_fault(struct pt_regs *, unsigned long);
 #ifdef CONFIG_TRACING
 dotraplinkage void trace_do_page_fault(struct pt_regs *, unsigned long);
+#else
+static inline void trace_do_page_fault(struct pt_regs *regs, unsigned long error)
+{
+	do_page_fault(regs, error);
+}
 #endif
 dotraplinkage void do_spurious_interrupt_bug(struct pt_regs *, long);
 dotraplinkage void do_coprocessor_error(struct pt_regs *, long);

_

#3 lore raw

On Thu, 08 May 2014 16:05:12 -0700
Dave Hansen <[email protected]> wrote:

Added Seiji and hpa

Quoted 8 lines · Thomas Gleixner
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I noticed on some of my systems that page fault tracing doesn't work:

cd /sys/kernel/debug/tracing
echo 1 > events/exceptions/enable
cat trace;
# nothing shows up

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

I eventually traced it down to CONFIG_KVM_GUEST. At least in a KVM VM, enabling that option breaks page fault tracing, and disabling fixes it.
I tried on some old kernels and this does not appear to be a
regression: it never worked.

Anybody have any theories about what is going on?

Has to do with the IDT tricks done to keep page faults from getting zero overhead when tracing is disabled. It sets up two types of traps. One that encompasses the other. When all of the trap tracepoints are disabled, the traps are called directly. But when any of them are enabled, the IDT gets switched to have the traps point to exceptions that trace the entry and exit of the exception.

see commits 25c74b10bacead867478480170083f69cfc0db48 and d34603b07c4255b2b00a546d34f297ccd50ae4c6

Quoted 8 lines

Looks like the KVM code calls do_page_fault() directly:

Quoted 5 lines

dotraplinkage void __kprobes
do_async_page_fault(struct pt_regs *regs, unsigned long error_code)

{
        enum ctx_state prev_state;
Quoted 8 lines
Quoted 5 lines
        switch (kvm_read_and_reset_pf_reason()) {
        default:
                do_page_fault(regs, error_code);
                break;
        case KVM_PV_REASON_PAGE_NOT_PRESENT:

That seems to explain my problems in a VM. Any objections to doing something like the attached patch?

As your patch showed up as an attachment, I couldn't include it in my reply. But sure, that may work. But you could also play tricks to keep the overhead off when tracing is disabled like this one:

[ I booted this, but didn't test any kvm guests with it ]

arch/x86/include/asm/traps.h +2 -1
index 58d66fe..835ba5b 100644
@@ -39,6 +39,8 @@ asmlinkage void simd_coprocessor_error(void);
 
 #ifdef CONFIG_TRACING
 asmlinkage void trace_page_fault(void);
+asmlinkage void trace_async_page_fault(void);
+#define trace_async_page_fault async_page_fault
 #define trace_divide_error divide_error
 #define trace_bounds bounds
 #define trace_invalid_op invalid_op
@@ -51,7 +53,6 @@ asmlinkage void trace_page_fault(void);
 #define trace_coprocessor_error coprocessor_error
 #define trace_alignment_check alignment_check
 #define trace_simd_coprocessor_error simd_coprocessor_error
-#define trace_async_page_fault async_page_fault
 #endif
 
 dotraplinkage void do_divide_error(struct pt_regs *, long);
arch/x86/kernel/entry_32.S +10 -0
index a2a4f46..26e370a 100644
@@ -1451,7 +1451,17 @@ ENTRY(async_page_fault)
 	jmp error_code
 	CFI_ENDPROC
 END(async_page_fault)
+
+#ifdef CONFIG_TRACING
+ENTRY(trace_async_page_fault)
+	RING0_EC_FRAME
+	ASM_CLAC
+	pushl_cfi $trace_do_async_page_fault
+	jmp error_code
+	CFI_ENDPROC
+END(async_page_fault)
 #endif
+#endif /* CONFIG_KVM_GUEST */
 
 /*
  * End of kprobes section
arch/x86/kernel/entry_64.S +1 -1
index 3db806d..e1f8926 100644
@@ -1289,7 +1289,7 @@ errorentry xen_stack_segment do_stack_segment
 errorentry general_protection do_general_protection
 trace_errorentry page_fault do_page_fault
 #ifdef CONFIG_KVM_GUEST
-errorentry async_page_fault do_async_page_fault
+trace_errorentry async_page_fault do_async_page_fault
 #endif
 #ifdef CONFIG_X86_MCE
 paranoidzeroentry machine_check *machine_check_vector(%rip)
arch/x86/kernel/kvm.c +34 -3
index 0331cb3..baef30e 100644
@@ -252,14 +252,18 @@ u32 kvm_read_and_reset_pf_reason(void)
 }
 EXPORT_SYMBOL_GPL(kvm_read_and_reset_pf_reason);
 
-dotraplinkage void __kprobes
-do_async_page_fault(struct pt_regs *regs, unsigned long error_code)
+enum {
+	KVM_PV_DO_PAGE_FAULT,
+	KVM_PV_HANDLED,
+};
+
+static int kvm_pv_page_fault(struct pt_regs *regs, unsigned long error_code)
 {
 	enum ctx_state prev_state;
 
 	switch (kvm_read_and_reset_pf_reason()) {
 	default:
-		do_page_fault(regs, error_code);
+		return KVM_PV_DO_PAGE_FAULT;
 		break;
 	case KVM_PV_REASON_PAGE_NOT_PRESENT:
 		/* page is swapped out by the host. */
@@ -275,7 +279,34 @@ do_async_page_fault(struct pt_regs *regs, unsigned long error_code)
 		rcu_irq_exit();
 		break;
 	}
+	return KVM_PV_HANDLED;
+}
+
+dotraplinkage void __kprobes
+do_async_page_fault(struct pt_regs *regs, unsigned long error_code)
+{
+	int ret;
+
+	ret = kvm_pv_page_fault(regs, error_code);
+	if (ret == KVM_PV_HANDLED)
+		return;
+
+	do_page_fault(regs, error_code);
+}
+
+#ifdef CONFIG_TRACING
+dotraplinkage void __kprobes notrace
+trace_do_async_page_fault(struct pt_regs *regs, unsigned long error_code)
+{
+	int ret;
+
+	ret = kvm_pv_page_fault(regs, error_code);
+	if (ret == KVM_PV_HANDLED)
+		return;
+
+	trace_do_page_fault(regs, error_code);
 }
+#endif /* CONFIG_TRACING */
 
 static void __init paravirt_ops_setup(void)
 {
#4 lore raw
Quoted 3 lines · Steven Rostedt

As your patch showed up as an attachment, I couldn't include it in my reply. But sure, that may work. But you could also play tricks to keep the overhead off when tracing is disabled like this one:

...

How important is it to have zero-overhead? This bug was introduced by trying to do just that, and I'm worried about having an explosion of so many paths to get in to the page fault code. The overhead in this case is just a few noops, right?

I guess that's a question for the KVM folks. Do you guys prefer zero overhead or simpler code?