Skip to contentSkip to conversation

Drivers: hv: vmbus: Implement Direct Mode for stimer0

PATCH In review [email protected] 3 participants 4 messages Jan 22, 2018 – Jan 26, 2018

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 Drivers: hv: vmbus: Implement Direct Mode for stimer0

From: Michael Kelley <[email protected]>

The 2016 version of Hyper-V offers the option to operate the guest VM per-vcpu stimer's in Direct Mode, which means the timer interupts on its own vector rather than queueing a VMbus message. Direct Mode reduces timer processing overhead in both the hypervisor and the guest, and avoids having timer interrupts pollute the VMbus interrupt stream for the synthetic NIC and storage. This patch enables Direct Mode by default on stimer0 when running on a version of Hyper-V that supports it, with a hv_vmbus module parameter for disabling Direct Mode and reverting to the old behavior.

In prep for coming support of Hyper-V on ARM64, the arch independent portion of the code contains calls to routines that will be populated on ARM64 but are not needed and do nothing on x86.

---
Changes since v1:
* Major rework to allocate and use a fixed interrupt vector
* Fixed block comment style
* Removed minor changes unrelated to Direct Mode

---
 arch/x86/entry/entry_32.S          |  3 ++
 arch/x86/entry/entry_64.S          |  2 ++
 arch/x86/include/asm/hardirq.h     |  3 ++
 arch/x86/include/asm/irq_vectors.h |  7 +++-
 arch/x86/include/asm/mshyperv.h    | 12 +++++++
 arch/x86/include/uapi/asm/hyperv.h |  3 ++
 arch/x86/kernel/cpu/mshyperv.c     | 41 ++++++++++++++++++++++-
 arch/x86/kernel/irq.c              |  9 +++++
 drivers/hv/hv.c                    | 67 ++++++++++++++++++++++++++++++++++++--
 drivers/hv/hyperv_vmbus.h          |  4 ++-
 10 files changed, 146 insertions(+), 5 deletions(-)
arch/x86/entry/entry_32.S +3 -0
index ace8f32..1814991 100644
@@ -882,6 +882,9 @@ BUILD_INTERRUPT3(xen_hvm_callback_vector, HYPERVISOR_CALLBACK_VECTOR,
 BUILD_INTERRUPT3(hyperv_callback_vector, HYPERVISOR_CALLBACK_VECTOR,
 		 hyperv_vector_handler)
 
+BUILD_INTERRUPT3(hv_stimer0_callback_vector, HYPERV_STIMER0_VECTOR,
+		 hv_stimer0_vector_handler)
+
 #endif /* CONFIG_HYPERV */
 
 ENTRY(page_fault)
arch/x86/entry/entry_64.S +2 -0
index f048e38..23afac9 100644
@@ -1227,6 +1227,8 @@ apicinterrupt3 HYPERVISOR_CALLBACK_VECTOR \
 #if IS_ENABLED(CONFIG_HYPERV)
 apicinterrupt3 HYPERVISOR_CALLBACK_VECTOR \
 	hyperv_callback_vector hyperv_vector_handler
+apicinterrupt3 HYPERV_STIMER0_VECTOR \
+	hv_stimer0_callback_vector hv_stimer0_vector_handler
 #endif /* CONFIG_HYPERV */
 
 idtentry debug			do_debug		has_error_code=0	paranoid=1 shift_ist=DEBUG_STACK
arch/x86/include/asm/hardirq.h +3 -0
index 51cc979..c788343 100644
@@ -38,6 +38,9 @@
 #if IS_ENABLED(CONFIG_HYPERV) || defined(CONFIG_XEN)
 	unsigned int irq_hv_callback_count;
 #endif
+#if IS_ENABLED(CONFIG_HYPERV)
+	unsigned int hyperv_stimer0_count;
+#endif
 } ____cacheline_aligned irq_cpustat_t;
 
 DECLARE_PER_CPU_SHARED_ALIGNED(irq_cpustat_t, irq_stat);
arch/x86/include/asm/irq_vectors.h +6 -1
index 67421f6..6accf0b 100644
@@ -103,7 +103,12 @@
 #endif
 
 #define MANAGED_IRQ_SHUTDOWN_VECTOR	0xef
-#define LOCAL_TIMER_VECTOR		0xee
+
+#if IS_ENABLED(CONFIG_HYPERV)
+#define HYPERV_STIMER0_VECTOR		0xee
+#endif
+
+#define LOCAL_TIMER_VECTOR		0xed
 
 #define NR_VECTORS			 256
 
arch/x86/include/asm/mshyperv.h +12 -0
index b623a42..aed9185 100644
@@ -171,6 +171,18 @@ static inline void vmbus_signal_eom(struct hv_message *msg, u32 old_msg_type)
 void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
 void hv_remove_crash_handler(void);
 
+/*
+ * Routines for stimer0 Direct Mode handling.
+ * On x86/x64, there are no percpu actions to take.
+ */
+void hv_stimer0_callback_vector(void);
+int hv_setup_stimer0_irq(int *irq, int *vector, void (*handler)(void));
+void hv_remove_stimer0_irq(int irq);
+
+static inline void hv_enable_stimer0_percpu_irq(int irq) {}
+static inline void hv_disable_stimer0_percpu_irq(int irq) {}
+
+
 #if IS_ENABLED(CONFIG_HYPERV)
 extern struct clocksource *hyperv_cs;
 extern void *hv_hypercall_pg;
arch/x86/include/uapi/asm/hyperv.h +3 -0
index 1a5bfea..7213cb8 100644
@@ -74,6 +74,9 @@
 /* Crash MSR available */
 #define HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE (1 << 10)
 
+/* stimer Direct Mode is available */
+#define HV_X64_STIMER_DIRECT_MODE_AVAILABLE	(1 << 19)
+
 /*
  * Feature identification: EBX indicates which flags were specified at
  * partition creation. The format is the same as the partition creation
arch/x86/kernel/cpu/mshyperv.c +40 -1

60 lines not on this page · raw message

arch/x86/kernel/irq.c +9 -0
index 68e1867..e1154f0 100644
@@ -142,6 +142,15 @@ int arch_show_interrupts(struct seq_file *p, int prec)
 		seq_puts(p, "  Hypervisor callback interrupts\n");
 	}
 #endif
+#if IS_ENABLED(CONFIG_HYPERV)
+	if (test_bit(HYPERV_STIMER0_VECTOR, system_vectors)) {
+		seq_printf(p, "%*s: ", prec, "HVS");
+		for_each_online_cpu(j)
+			seq_printf(p, "%10u ",
+				   irq_stats(j)->hyperv_stimer0_count);
+		seq_puts(p, "  Hyper-V stimer0 interrupts\n");
+	}
+#endif
 	seq_printf(p, "%*s: %10u\n", prec, "ERR", atomic_read(&irq_err_count));
 #if defined(CONFIG_X86_IO_APIC)
 	seq_printf(p, "%*s: %10u\n", prec, "MIS", atomic_read(&irq_mis_count));
drivers/hv/hv.c +65 -2

118 lines not on this page · raw message

drivers/hv/hyperv_vmbus.h +3 -1
index 22300ec..36d34fe 100644
@@ -57,7 +57,9 @@
 		u64 periodic:1;
 		u64 lazy:1;
 		u64 auto_enable:1;
-		u64 reserved_z0:12;
+		u64 apic_vector:8;
+		u64 direct_mode:1;
+		u64 reserved_z0:3;
 		u64 sintx:4;
 		u64 reserved_z1:44;
 	};
Signature · 6 lines
1.8.3.1

_______________________________________________
devel mailing list
[email protected]
http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
#1 lore raw

drivers/hv/hv.c struct hv_context hv_context = { · line 53

Quoted 12 lines · On Mon, Jan 22, 2018 at 03:29:29PM -0700, [email protected] wrote:
drivers/hv/hv.c +12 -0
+/*
+ * If false, we're using the old mechanism for stimer0 interrupts
+ * where it sends a VMbus message when it expires. The old
+ * mechanism is used if Direct Mode is explicitly disabled
+ * by the module parameter, or on older versions of Hyper-V
+ * that don't support Direct Mode. While Hyper-V provides
+ * four stimer's per CPU, Linux uses only stimer0.
+ */
+static bool direct_mode_enabled = true;
+module_param(direct_mode_enabled, bool, 0444);
+MODULE_PARM_DESC(direct_mode_enabled,
+			"Set to N to disable stimer Direct Mode.");

Ick ick ick. How is a distro or user supposed to know if/when to enable this and not to enable it? This isn't ok, can you please make this "automatic" to always do the right thing based on what it is running on?

Module parameters are not a good idea as they are impossible to maintain and document and use over time. Please never add new ones to the kernel.

thanks,

greg k-h
_______________________________________________
devel mailing list
[email protected]
http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel

#2 lore raw
Quoted 12 lines

-----Original Message-----
From: Greg KH [mailto:[email protected]]
Sent: Thursday, January 25, 2018 2:00 AM
To: Michael Kelley (EOSG) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Stephen Hemminger <[email protected]>; KY Srinivasan <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH v2 char-misc 1/1] Drivers: hv: vmbus: Implement Direct Mode for stimer0

On Mon, Jan 22, 2018 at 03:29:29PM -0700, [email protected] wrote:

Quoted 16 lines

drivers/hv/hv.c struct hv_context hv_context = { · line 53

Quoted 12 lines
drivers/hv/hv.c +12 -0
+/*
+ * If false, we're using the old mechanism for stimer0 interrupts
+ * where it sends a VMbus message when it expires. The old
+ * mechanism is used if Direct Mode is explicitly disabled
+ * by the module parameter, or on older versions of Hyper-V
+ * that don't support Direct Mode. While Hyper-V provides
+ * four stimer's per CPU, Linux uses only stimer0.
+ */
+static bool direct_mode_enabled = true;
+module_param(direct_mode_enabled, bool, 0444);
+MODULE_PARM_DESC(direct_mode_enabled,
+			"Set to N to disable stimer Direct Mode.");

Ick ick ick. How is a distro or user supposed to know if/when to enable this and not to enable it? This isn't ok, can you please make this "automatic" to always do the right thing based on what it is running on?

To be clear, this patch already does the "automatic" thing based on the capabilities of the hypervisor. But it's often handy to be able to override the automatic behavior without having to rebuild the kernel.

Quoted 5 lines

Module parameters are not a good idea as they are impossible to maintain and document and use over time. Please never add new ones to the kernel.

Is there a better alternative for providing an "override" capability? Having an override isn't super important in this case, so I'll drop the parameter, but it would be nice to have something easier than a rebuild.

Michael

Quoted 3 lines

thanks,

greg k-h

#3 lore raw
Quoted 29 lines · Michael Kelley (EOSG)
Quoted 12 lines

-----Original Message-----
From: Greg KH [mailto:[email protected]]
Sent: Thursday, January 25, 2018 2:00 AM
To: Michael Kelley (EOSG) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Stephen Hemminger <[email protected]>; KY Srinivasan <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH v2 char-misc 1/1] Drivers: hv: vmbus: Implement Direct Mode for stimer0

On Mon, Jan 22, 2018 at 03:29:29PM -0700, [email protected] wrote:

Quoted 16 lines

drivers/hv/hv.c struct hv_context hv_context = { · line 53

Quoted 12 lines
drivers/hv/hv.c +12 -0
+/*
+ * If false, we're using the old mechanism for stimer0 interrupts
+ * where it sends a VMbus message when it expires. The old
+ * mechanism is used if Direct Mode is explicitly disabled
+ * by the module parameter, or on older versions of Hyper-V
+ * that don't support Direct Mode. While Hyper-V provides
+ * four stimer's per CPU, Linux uses only stimer0.
+ */
+static bool direct_mode_enabled = true;
+module_param(direct_mode_enabled, bool, 0444);
+MODULE_PARM_DESC(direct_mode_enabled,
+			"Set to N to disable stimer Direct Mode.");

Ick ick ick. How is a distro or user supposed to know if/when to enable this and not to enable it? This isn't ok, can you please make this "automatic" to always do the right thing based on what it is running on?

Quoted 3 lines

To be clear, this patch already does the "automatic" thing based on the capabilities of the hypervisor. But it's often handy to be able to override the automatic behavior without having to rebuild the kernel.

Then put the option somewhere else. sysfs, debugfs, somewhere else. Never a module parameter please, not for new code.

thanks,

greg k-h
_______________________________________________
devel mailing list
[email protected]
http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel