Skip to contentSkip to conversation

Drivers: hv: Move Hyper-V clock/timer code to separate clocksource driver

PATCH In review Michael Kelley 2 participants 7 messages Jun 28, 2019 – Jul 1, 2019

Subsystems (11)
x86 KVM Kselftest MIPS +7 more subsystems
lore
Share & tools
lore
#0 lore raw

v4 cover Drivers: hv: Move Hyper-V clock/timer code to separate clocksource driver

This patch series moves Hyper-V clock/timer code to a separate Hyper-V clocksource driver. Previously, Hyper-V clock/timer code and data structures were mixed in with other Hyper-V code in the ISA independent drivers/hv code as well as in ISA dependent code. The new Hyper-V clocksource driver is ISA independent, with a just few dependencies on ISA specific functions. The patch series does not change any behavior or functionality -- it only reorganizes the existing code and fixes up the linkages. A few places outside of Hyper-V code are fixed up to use the new #include file structure.

This restructuring is in response to Marc Zyngier's review comments on supporting Hyper-V running on ARM64, and is a good idea in general. It increases the amount of code shared between the x86 and ARM64 architectures, and reduces the size of the new code for supporting Hyper-V on ARM64. A new version of the Hyper-V on ARM64 patches will follow once this clocksource restructuring is accepted.

The code is diff'ed against the upstream tip tree:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers/vdso

Changes in v4:
* Revised commit messages
* Rebased to upstream tip tree

Changes in v3:
* Removed boolean argument to hv_init_clocksource(). Always call sched_clock_register, which is needed on ARM64 but a no-op on x86.
* Removed separate cpuhp setup in hv_stimer_alloc() and instead directly call hv_stimer_init() and hv_stimer_cleanup() from corresponding VMbus functions. This more closely matches original code and avoids clocksource stop/restart problems on ARM64 when VMbus code denies CPU offlining request.

Changes in v2:
* Revised commit short descriptions so the distinction between the first and second patches is clearer [GregKH]
* Renamed new clocksource driver files and functions to use existing "timer" and "stimer" names instead of introducing "syntimer". [Vitaly Kuznetsov]
* Introduced CONFIG_HYPER_TIMER to fix build problem when CONFIG_HYPERV=m [Vitaly Kuznetsov]
* Added "Suggested-by: Marc Zyngier"

Michael Kelley (2):
Drivers: hv: Create Hyper-V clocksource driver from existing
clockevents code
Drivers: hv: Move Hyper-V clocksource code to new clocksource driver

 MAINTAINERS                              |   2 +
 arch/x86/entry/vdso/vma.c                |   2 +-
 arch/x86/hyperv/hv_init.c                |  91 +--------
 arch/x86/include/asm/hyperv-tlfs.h       |   6 +
 arch/x86/include/asm/mshyperv.h          |  81 ++------
 arch/x86/include/asm/vdso/gettimeofday.h |   2 +-
 arch/x86/kernel/cpu/mshyperv.c           |   2 +
 arch/x86/kvm/x86.c                       |   1 +
 drivers/clocksource/Makefile             |   1 +
 drivers/clocksource/hyperv_timer.c       | 321 +++++++++++++++++++++++++++++++
 drivers/hv/Kconfig                       |   3 +
 drivers/hv/hv.c                          | 156 +--------------
 drivers/hv/hyperv_vmbus.h                |   3 -
 drivers/hv/vmbus_drv.c                   |  42 ++--
 include/clocksource/hyperv_timer.h       | 105 ++++++++++
 15 files changed, 483 insertions(+), 335 deletions(-)
 create mode 100644 drivers/clocksource/hyperv_timer.c
 create mode 100644 include/clocksource/hyperv_timer.h
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH v4 1/2 Drivers: hv: Create Hyper-V clocksource driver from existing clockevents code

Hyper-V clock/timer code and data structures are currently mixed in with other code in the ISA independent drivers/hv directory as well as the ISA dependent Hyper-V code under arch/x86. Consolidate this code and data structures into a Hyper-V clocksource driver to better follow the Linux model. In doing so, separate out the ISA dependent portions so the new clocksource driver works for x86 and for the in-process Hyper-V on ARM64 code.

To start, move the existing clockevents code to create the new clocksource driver. Update the VMbus driver to call initialization and cleanup routines since the Hyper-V synthetic timers are not independently enumerated in ACPI.

No behavior is changed and no new functionality is added.

---
 MAINTAINERS                        |   2 +
 arch/x86/include/asm/hyperv-tlfs.h |   6 ++
 arch/x86/kernel/cpu/mshyperv.c     |   2 +
 drivers/clocksource/Makefile       |   1 +
 drivers/clocksource/hyperv_timer.c | 191 +++++++++++++++++++++++++++++++++++++
 drivers/hv/Kconfig                 |   3 +
 drivers/hv/hv.c                    | 156 +-----------------------------
 drivers/hv/hyperv_vmbus.h          |   3 -
 drivers/hv/vmbus_drv.c             |  42 ++++----
 include/clocksource/hyperv_timer.h |  27 ++++++
 10 files changed, 258 insertions(+), 175 deletions(-)
 create mode 100644 drivers/clocksource/hyperv_timer.c
 create mode 100644 include/clocksource/hyperv_timer.h
MAINTAINERS +2 -0
index a281297..bfde42a 100644
@@ -7313,6 +7313,7 @@ F:	arch/x86/include/asm/trace/hyperv.h
 F:	arch/x86/include/asm/hyperv-tlfs.h
 F:	arch/x86/kernel/cpu/mshyperv.c
 F:	arch/x86/hyperv
+F:	drivers/clocksource/hyperv_timer.c
 F:	drivers/hid/hid-hyperv.c
 F:	drivers/hv/
 F:	drivers/input/serio/hyperv-keyboard.c
@@ -7323,6 +7324,7 @@ F:	drivers/uio/uio_hv_generic.c
 F:	drivers/video/fbdev/hyperv_fb.c
 F:	drivers/iommu/hyperv_iommu.c
 F:	net/vmw_vsock/hyperv_transport.c
+F:	include/clocksource/hyperv_timer.h
 F:	include/linux/hyperv.h
 F:	include/uapi/linux/hyperv.h
 F:	tools/hv/
arch/x86/include/asm/hyperv-tlfs.h +6 -0
index cdf44aa..af78cd7 100644
@@ -401,6 +401,12 @@ enum HV_GENERIC_SET_FORMAT {
 #define HV_STATUS_INVALID_CONNECTION_ID		18
 #define HV_STATUS_INSUFFICIENT_BUFFERS		19
 
+/*
+ * The Hyper-V TimeRefCount register and the TSC
+ * page provide a guest VM clock with 100ns tick rate
+ */
+#define HV_CLOCK_HZ (NSEC_PER_SEC/100)
+
 typedef struct _HV_REFERENCE_TSC_PAGE {
 	__u32 tsc_sequence;
 	__u32 res1;
arch/x86/kernel/cpu/mshyperv.c +2 -0
index 7df29f0..b822fa6 100644
@@ -17,6 +17,7 @@
 #include <linux/irq.h>
 #include <linux/kexec.h>
 #include <linux/i8253.h>
+#include <linux/random.h>
 #include <asm/processor.h>
 #include <asm/hypervisor.h>
 #include <asm/hyperv-tlfs.h>
@@ -80,6 +81,7 @@ __visible void __irq_entry hv_stimer0_vector_handler(struct pt_regs *regs)
 	inc_irq_stat(hyperv_stimer0_count);
 	if (hv_stimer0_handler)
 		hv_stimer0_handler();
+	add_interrupt_randomness(HYPERV_STIMER0_VECTOR, 0);
 	ack_APIC_irq();
 
 	exiting_irq();
drivers/clocksource/Makefile +1 -0
index 236858f..2b65c5f 100644
@@ -84,3 +84,4 @@ obj-$(CONFIG_ATCPIT100_TIMER)		+= timer-atcpit100.o
 obj-$(CONFIG_RISCV_TIMER)		+= timer-riscv.o
 obj-$(CONFIG_CSKY_MP_TIMER)		+= timer-mp-csky.o
 obj-$(CONFIG_GX6605S_TIMER)		+= timer-gx6605s.o
+obj-$(CONFIG_HYPERV_TIMER)		+= hyperv_timer.o
drivers/clocksource/hyperv_timer.c new file +191 -0

191 lines not on this page · raw message

drivers/hv/Kconfig +3 -0
index 1c1a251..c423e57 100644
@@ -10,6 +10,9 @@ config HYPERV
 	  Select this option to run Linux as a Hyper-V client operating
 	  system.
 
+config HYPERV_TIMER
+	def_bool HYPERV
+
 config HYPERV_TSCPAGE
        def_bool HYPERV && X86_64
 
drivers/hv/hv.c +4 -152

211 lines not on this page · raw message

drivers/hv/hyperv_vmbus.h +0 -3
index b8e1ff0..362e70e 100644
@@ -138,7 +138,6 @@ struct hv_per_cpu_context {
 	 * per-cpu list of the channels based on their CPU affinity.
 	 */
 	struct list_head chan_list;
-	struct clock_event_device *clk_evt;
 };
 
 struct hv_context {
@@ -176,8 +175,6 @@ extern int hv_post_message(union hv_connection_id connection_id,
 
 extern int hv_synic_cleanup(unsigned int cpu);
 
-extern void hv_synic_clockevents_cleanup(void);
-
 /* Interface */
 
 void hv_ringbuffer_pre_init(struct vmbus_channel *channel);
drivers/hv/vmbus_drv.c +22 -20

101 lines not on this page · raw message

include/clocksource/hyperv_timer.h new file +27 -0
new file mode 100644
index 0000000..ba5dc17
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Definitions for the clocksource provided by the Hyper-V
+ * hypervisor to guest VMs, as described in the Hyper-V Top
+ * Level Functional Spec (TLFS).
+ *
+ * Copyright (C) 2019, Microsoft, Inc.
+ *
+ * Author:  Michael Kelley <[email protected]>
+ */
+
+#ifndef __CLKSOURCE_HYPERV_TIMER_H
+#define __CLKSOURCE_HYPERV_TIMER_H
+
+#define HV_MAX_MAX_DELTA_TICKS 0xffffffff
+#define HV_MIN_DELTA_TICKS 1
+
+/* Routines called by the VMbus driver */
+extern int hv_stimer_alloc(int sint);
+extern void hv_stimer_free(void);
+extern int hv_stimer_init(unsigned int cpu);
+extern int hv_stimer_cleanup(unsigned int cpu);
+extern void hv_stimer_global_cleanup(void);
+extern void hv_stimer0_isr(void);
+
+#endif
Signature · 1 line
1.8.3.1
#2 lore raw

Re: [PATCH v4 1/2] Drivers: hv: Create Hyper-V clocksource driver from existing clockevents code

Micheal,

Quoted 1 line · Michael Kelley

[PATCH v4 1/2] Drivers: hv: Create Hyper-V clocksource driver from existing clockevents code

As this creates a new clocksource driver, please use the

'clocksource/drivers: '

prefix and provide a more concise summary phrase. Something like:

'clocksource/drivers: Make Hyper-V clocksource ISA agnostic'

The longer explanation is already in the changelog.

drivers/clocksource/hyperv_timer.c line 6

Quoted 8 lines

+++ b/drivers/clocksource/hyperv_timer.c

drivers/clocksource/hyperv_timer.c new file +6 -0
@@ -0,0 +1,191 @@
+static int hv_ce_set_next_event(unsigned long delta,
+				struct clock_event_device *evt)
+{
+	u64 current_tick;
+
+	WARN_ON(!clockevent_state_oneshot(evt));

If at all that wants to be a WARN_ON_ONCE, but there is really no point in sprinkling these WARN_ONs into functions which are called from generic infrastructure code.

Generic infrastructure wants to have warnings because the functions are called from random places.

Quoted 19 lines
drivers/clocksource/hyperv_timer.c new file +19 -0
+
+	current_tick = hyperv_cs->read(NULL);
+	current_tick += delta;
+	hv_init_timer(0, current_tick);
+	return 0;
+}
+/*
+ * hv_stimer_init - Per-cpu initialization of the clockevent
+ */
+int hv_stimer_init(unsigned int cpu)
+{
+	struct clock_event_device *ce;
+
+	if (ms_hyperv.features & HV_MSR_SYNTIMER_AVAILABLE) {
+		ce = per_cpu_ptr(hv_clock_event, cpu);
+		ce->name = "Hyper-V clockevent";
+		ce->features = CLOCK_EVT_FEAT_ONESHOT;
+		ce->cpumask = cpumask_of(cpu);
+		ce->rating = 1000;

drivers/clocksource/hyperv_timer.c line 123

Quoted 10 lines
drivers/clocksource/hyperv_timer.c new file +10 -0
+		ce->set_state_shutdown = hv_ce_shutdown;
+		ce->set_state_oneshot = hv_ce_set_oneshot;
+		ce->set_next_event = hv_ce_set_next_event;
+
+		clockevents_config_and_register(ce,
+						HV_CLOCK_HZ,
+						HV_MIN_DELTA_TICKS,
+						HV_MAX_MAX_DELTA_TICKS);
+	}
+	return 0;

Why is this returning success if there is no timer available?

Quoted 4 lines
drivers/clocksource/hyperv_timer.c new file +4 -0
+}
+EXPORT_SYMBOL_GPL(hv_stimer_init);
+
+/*

drivers/clocksource/hyperv_timer.c line 137

Quoted 10 lines
drivers/clocksource/hyperv_timer.c new file +10 -0
+ * hv_stimer_cleanup - Per-cpu cleanup of the clockevent
+ */
+int hv_stimer_cleanup(unsigned int cpu)
+{
+	struct clock_event_device *ce;
+
+	/* Turn off clockevent device */
+	if (ms_hyperv.features & HV_MSR_SYNTIMER_AVAILABLE) {
+		ce = per_cpu_ptr(hv_clock_event, cpu);
+		clockevents_unbind_device(ce, cpu);

Can you please explain why this unbind is needed?

drivers/clocksource/hyperv_timer.c line 141

Quoted 4 lines
drivers/clocksource/hyperv_timer.c new file +4 -0
+		hv_ce_shutdown(ce);
+	}
+
+	return 0;

Why does that function need a return value?

Quoted 6 lines
drivers/clocksource/hyperv_timer.c new file +6 -0
+}
+EXPORT_SYMBOL_GPL(hv_stimer_cleanup);
+
+/* hv_stimer_alloc - Global initialization of the clockevent and stimer0 */
+int hv_stimer_alloc(int sint)
+{

drivers/clocksource/hyperv_timer.c line 157

Quoted 10 lines
drivers/clocksource/hyperv_timer.c new file +10 -0
+	hv_clock_event = alloc_percpu(struct clock_event_device);
+	if (!hv_clock_event)
+		return -ENOMEM;
+
+	direct_mode_enabled = ms_hyperv.misc_features &
+			HV_STIMER_DIRECT_MODE_AVAILABLE;
+	if (direct_mode_enabled &&
+	    hv_setup_stimer0_irq(&stimer0_irq, &stimer0_vector,
+				hv_stimer0_isr)) {
+		free_percpu(hv_clock_event);

Leaves a dangling pointer around.

drivers/clocksource/hyperv_timer.c line 158

Quoted 1 line
drivers/clocksource/hyperv_timer.c new file +1 -0
+		return -EINVAL;

Hrm. Doesn't hv_setup_stimer0_irq() return a useful return value? -EINVAL looks pretty strange here.

drivers/clocksource/hyperv_timer.c line 171

Quoted 13 lines
drivers/clocksource/hyperv_timer.c new file +13 -0
+	}
+
+	stimer0_message_sint = sint;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(hv_stimer_alloc);
+
+/* hv_stimer_free - Free global resources allocated by hv_stimer_alloc() */
+void hv_stimer_free(void)
+{
+	if (direct_mode_enabled)
+		hv_remove_stimer0_irq(stimer0_irq);
+	free_percpu(hv_clock_event);

Again dangling pointer. Also is it guaranteed that stimer0_irq has been set up when this is invoked? As a general rule, make such functions defensive then you don't need state tracking at the call site and you can invoke them unconditionally.

Quoted 7 lines
drivers/clocksource/hyperv_timer.c new file +7 -0
+}
+EXPORT_SYMBOL_GPL(hv_stimer_free);
+
+/*
+ * Do a global cleanup of clockevents for the cases of kexec and
+ * vmbus exit
+ */

drivers/clocksource/hyperv_timer.c line 188

Quoted 10 lines
drivers/clocksource/hyperv_timer.c new file +10 -0
+void hv_stimer_global_cleanup(void)
+{
+	int	cpu;
+	struct clock_event_device *ce;
+
+	if (ms_hyperv.features & HV_MSR_SYNTIMER_AVAILABLE)
+		for_each_present_cpu(cpu) {
+			ce = per_cpu_ptr(hv_clock_event, cpu);
+			clockevents_unbind_device(ce, cpu);
+		}

Please put braces after the if () because the 4 lines following it do not qualify as a single line statement.

Thanks,

tglx

#3 lore raw

RE: [PATCH v4 1/2] Drivers: hv: Create Hyper-V clocksource driver from existing clockevents code

From: Thomas Gleixner <[email protected]> Sent: Friday, June 28, 2019 11:50 PM

Quoted 7 lines · Michael Kelley
Quoted 2 lines

[PATCH v4 1/2] Drivers: hv: Create Hyper-V clocksource driver from existing clockevents

code

As this creates a new clocksource driver, please use the

Quoted 7 lines

'clocksource/drivers: '

prefix and provide a more concise summary phrase. Something like:

'clocksource/drivers: Make Hyper-V clocksource ISA agnostic'

The longer explanation is already in the changelog.

Will do in v5.

Quoted 14 lines

drivers/clocksource/hyperv_timer.c line 6

Quoted 9 lines

+++ b/drivers/clocksource/hyperv_timer.c

drivers/clocksource/hyperv_timer.c new file +6 -0
@@ -0,0 +1,191 @@
+static int hv_ce_set_next_event(unsigned long delta,
+				struct clock_event_device *evt)
+{
+	u64 current_tick;
+
+	WARN_ON(!clockevent_state_oneshot(evt));

If at all that wants to be a WARN_ON_ONCE, but there is really no point in sprinkling these WARN_ONs into functions which are called from generic infrastructure code.

Quoted 2 lines

Generic infrastructure wants to have warnings because the functions are called from random places.

I'll remove it.

Quoted 24 lines
Quoted 24 lines
drivers/clocksource/hyperv_timer.c new file +23 -0
+
+	current_tick = hyperv_cs->read(NULL);
+	current_tick += delta;
+	hv_init_timer(0, current_tick);
+	return 0;
+}
+/*
+ * hv_stimer_init - Per-cpu initialization of the clockevent
+ */
+int hv_stimer_init(unsigned int cpu)
+{
+	struct clock_event_device *ce;
+
+	if (ms_hyperv.features & HV_MSR_SYNTIMER_AVAILABLE) {
+		ce = per_cpu_ptr(hv_clock_event, cpu);
+		ce->name = "Hyper-V clockevent";
+		ce->features = CLOCK_EVT_FEAT_ONESHOT;
+		ce->cpumask = cpumask_of(cpu);
+		ce->rating = 1000;
+		ce->set_state_shutdown = hv_ce_shutdown;
+		ce->set_state_oneshot = hv_ce_set_oneshot;
+		ce->set_next_event = hv_ce_set_next_event;
+
Quoted 8 lines

drivers/clocksource/hyperv_timer.c line 123

Quoted 6 lines
drivers/clocksource/hyperv_timer.c new file +6 -0
+		clockevents_config_and_register(ce,
+						HV_CLOCK_HZ,
+						HV_MIN_DELTA_TICKS,
+						HV_MAX_MAX_DELTA_TICKS);
+	}
+	return 0;

Why is this returning success if there is no timer available?

The synthetic timers are available on all Hyper-V versions on ARM64 and all but the oldest versions of Hyper-V on x86. For the latter case, it's not an error because Linux still runs with PIT and LAPIC-based clock events using the emulated hardware provided by Hyper-V. So the caller doesn't check the return value. I'll add a comment to this effect, and change this to a void function.

Quoted 17 lines
Quoted 5 lines
drivers/clocksource/hyperv_timer.c new file +4 -0
+}
+EXPORT_SYMBOL_GPL(hv_stimer_init);
+
+/*

drivers/clocksource/hyperv_timer.c line 137

Quoted 10 lines
drivers/clocksource/hyperv_timer.c new file +10 -0
+ * hv_stimer_cleanup - Per-cpu cleanup of the clockevent
+ */
+int hv_stimer_cleanup(unsigned int cpu)
+{
+	struct clock_event_device *ce;
+
+	/* Turn off clockevent device */
+	if (ms_hyperv.features & HV_MSR_SYNTIMER_AVAILABLE) {
+		ce = per_cpu_ptr(hv_clock_event, cpu);
+		clockevents_unbind_device(ce, cpu);

Can you please explain why this unbind is needed?

It's not needed. But it has been in the code since the beginning of the Hyper-V support and I was taking the approach of not changing code I didn't need to. I'll take it out.

Quoted 7 lines

drivers/clocksource/hyperv_timer.c line 141

Quoted 5 lines
drivers/clocksource/hyperv_timer.c new file +4 -0
+		hv_ce_shutdown(ce);
+	}
+
+	return 0;

Why does that function need a return value?

It doesn't. This should be a void function. I'll change it.

Quoted 11 lines
Quoted 11 lines
drivers/clocksource/hyperv_timer.c new file +10 -0
+}
+EXPORT_SYMBOL_GPL(hv_stimer_cleanup);
+
+/* hv_stimer_alloc - Global initialization of the clockevent and stimer0 */
+int hv_stimer_alloc(int sint)
+{
+	hv_clock_event = alloc_percpu(struct clock_event_device);
+	if (!hv_clock_event)
+		return -ENOMEM;
+
Quoted 8 lines

drivers/clocksource/hyperv_timer.c line 157

Quoted 6 lines
drivers/clocksource/hyperv_timer.c new file +6 -0
+	direct_mode_enabled = ms_hyperv.misc_features &
+			HV_STIMER_DIRECT_MODE_AVAILABLE;
+	if (direct_mode_enabled &&
+	    hv_setup_stimer0_irq(&stimer0_irq, &stimer0_vector,
+				hv_stimer0_isr)) {
+		free_percpu(hv_clock_event);

Leaves a dangling pointer around.

Will fix.

Quoted 5 lines

drivers/clocksource/hyperv_timer.c line 158

Quoted 2 lines
drivers/clocksource/hyperv_timer.c new file +1 -0
+		return -EINVAL;

Hrm. Doesn't hv_setup_stimer0_irq() return a useful return value? -EINVAL looks pretty strange here.

On x86, hv_setup_stimer0_irq() does nothing because Hyper-V stimer0 has a hardcoded interrupt vector across all CPUs. But on ARM64, it does real setup of a PPI and gives a useful return value. I'll change this to propagate the return value from hv_setup_stimer0_irq().

Quoted 11 lines
Quoted 11 lines
drivers/clocksource/hyperv_timer.c new file +10 -0
+	}
+
+	stimer0_message_sint = sint;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(hv_stimer_alloc);
+
+/* hv_stimer_free - Free global resources allocated by hv_stimer_alloc() */
+void hv_stimer_free(void)
+{
Quoted 8 lines

drivers/clocksource/hyperv_timer.c line 171

Quoted 3 lines
drivers/clocksource/hyperv_timer.c new file +3 -0
+	if (direct_mode_enabled)
+		hv_remove_stimer0_irq(stimer0_irq);
+	free_percpu(hv_clock_event);

Again dangling pointer. Also is it guaranteed that stimer0_irq has been set up when this is invoked? As a general rule, make such functions defensive then you don't need state tracking at the call site and you can invoke them unconditionally.

I'll fix the dangling pointer and add a check for stimer0_irq being 0.

Quoted 13 lines
Quoted 13 lines
drivers/clocksource/hyperv_timer.c new file +12 -0
+}
+EXPORT_SYMBOL_GPL(hv_stimer_free);
+
+/*
+ * Do a global cleanup of clockevents for the cases of kexec and
+ * vmbus exit
+ */
+void hv_stimer_global_cleanup(void)
+{
+	int	cpu;
+	struct clock_event_device *ce;
+
Quoted 8 lines

drivers/clocksource/hyperv_timer.c line 188

Quoted 5 lines
drivers/clocksource/hyperv_timer.c new file +5 -0
+	if (ms_hyperv.features & HV_MSR_SYNTIMER_AVAILABLE)
+		for_each_present_cpu(cpu) {
+			ce = per_cpu_ptr(hv_clock_event, cpu);
+			clockevents_unbind_device(ce, cpu);
+		}

Please put braces after the if () because the 4 lines following it do not qualify as a single line statement.

Will do.

Thanks for the comments.

Michael

#4 lore raw

PATCH v4 2/2 Drivers: hv: Move Hyper-V clocksource code to new clocksource driver

In reply to Michael Kelley

Continue consolidating Hyper-V clock and timer code into an ISA independent Hyper-V clocksource driver. Move the existing clocksource code under drivers/hv and arch/x86 to the new clocksource driver while separating out the ISA dependencies. Update Hyper-V initialization to call initialization and cleanup routines since the Hyper-V synthetic clock is not independently enumerated in ACPI. Update Hyper-V clocksource users in KVM and VDSO to get definitions from a new include file.

No behavior is changed and no new functionality is added.

---
 arch/x86/entry/vdso/vma.c                |   2 +-
 arch/x86/hyperv/hv_init.c                |  91 +---------------------
 arch/x86/include/asm/mshyperv.h          |  81 +++----------------
 arch/x86/include/asm/vdso/gettimeofday.h |   2 +-
 arch/x86/kvm/x86.c                       |   1 +
 drivers/clocksource/hyperv_timer.c       | 130 +++++++++++++++++++++++++++++++
 include/clocksource/hyperv_timer.h       |  78 +++++++++++++++++++
 7 files changed, 225 insertions(+), 160 deletions(-)
arch/x86/entry/vdso/vma.c +1 -1
index 8db1f59..349a61d 100644
@@ -22,7 +22,7 @@
 #include <asm/page.h>
 #include <asm/desc.h>
 #include <asm/cpufeature.h>
-#include <asm/mshyperv.h>
+#include <clocksource/hyperv_timer.h>
 
 #if defined(CONFIG_X86_64)
 unsigned int __read_mostly vdso64_enabled = 1;
arch/x86/hyperv/hv_init.c +3 -88

109 lines not on this page · raw message

arch/x86/include/asm/mshyperv.h +11 -70

97 lines not on this page · raw message

arch/x86/include/asm/vdso/gettimeofday.h +1 -1
index a14039a..ae91429 100644
@@ -18,7 +18,7 @@
 #include <asm/unistd.h>
 #include <asm/msr.h>
 #include <asm/pvclock.h>
-#include <asm/mshyperv.h>
+#include <clocksource/hyperv_timer.h>
 
 #define __vdso_data (VVAR(_vdso_data))
 
arch/x86/kvm/x86.c +1 -0
index 9857992..07ddb1e 100644
@@ -67,6 +67,7 @@
 #include <asm/mshyperv.h>
 #include <asm/hypervisor.h>
 #include <asm/intel_pt.h>
+#include <clocksource/hyperv_timer.h>
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
drivers/clocksource/hyperv_timer.c +130 -0

139 lines not on this page · raw message

include/clocksource/hyperv_timer.h +78 -0

88 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#5 lore raw

Re: [PATCH v4 2/2] Drivers: hv: Move Hyper-V clocksource code to new clocksource driver

Michael,

On Fri, 28 Jun 2019, Michael Kelley wrote:

See the other reply vs. prefix and summary

Quoted 1 line

Continue consolidating Hyper-V clock and timer code into an ISA

...

drivers/clocksource/hyperv_timer.c void hv_stimer_global_cleanup(void) · line 216

Quoted 11 lines
drivers/clocksource/hyperv_timer.c +10 -0
+#ifdef CONFIG_HYPERV_TSCPAGE
+
+static struct ms_hyperv_tsc_page *tsc_pg;
+
+struct ms_hyperv_tsc_page *hv_get_tsc_page(void)
+{
+	return tsc_pg;
+}
+EXPORT_SYMBOL_GPL(hv_get_tsc_page);
 
+static u64 read_hv_sched_clock_tsc(void)

Lacks a notrace annotation as it is used as sched clock.

Quoted 10 lines
drivers/clocksource/hyperv_timer.c +10 -0
+{
+	u64 current_tick = hv_read_tsc_page(tsc_pg);
+
+	if (current_tick == U64_MAX)
+		hv_get_time_ref_count(current_tick);
+
+	return current_tick;
+}
+
+static u64 read_hv_clock_tsc(struct clocksource *arg)

drivers/clocksource/hyperv_timer.c void hv_stimer_global_cleanup(void) · line 236

Quoted 10 lines
drivers/clocksource/hyperv_timer.c +10 -0
+{
+	return read_hv_sched_clock_tsc();
+}
+
+static struct clocksource hyperv_cs_tsc = {
+		.name		= "hyperv_clocksource_tsc_page",
+		.rating		= 400,
+		.read		= read_hv_clock_tsc,
+		.mask		= CLOCKSOURCE_MASK(64),
+		.flags		= CLOCK_SOURCE_IS_CONTINUOUS,

One tab indent is sufficient

drivers/clocksource/hyperv_timer.c void hv_stimer_global_cleanup(void) · line 267

Quoted 6 lines
drivers/clocksource/hyperv_timer.c +5 -0
+};
+#endif
 
+void __init hv_init_clocksource(void)
+{
+#ifdef CONFIG_HYPERV_TSCPAGE

Can you please avoid ifdef blocks in the code and just split out this into a helper function and provide an empty stub for the TSCPAGE=n case?

Then you can spare one indent and the ifdeffery is confined. Get's rid of the goto as well.

drivers/clocksource/hyperv_timer.c void hv_stimer_global_cleanup(void) · line 280

Quoted 13 lines
drivers/clocksource/hyperv_timer.c +13 -0
+	if (ms_hyperv.features & HV_MSR_REFERENCE_TSC_AVAILABLE) {
+
+		u64		tsc_msr;
+		phys_addr_t	phys_addr;
+
+		tsc_pg = vmalloc(PAGE_SIZE);
+		if (!tsc_pg)
+			goto register_msr_cs;
+
+		hyperv_cs = &hyperv_cs_tsc;
+		phys_addr = page_to_phys(vmalloc_to_page(tsc_pg));
+
+		/* The Hyper-V TLFS specifies to preserve the value of

Please use proper multi line comments formatting:

		/*
		 * This is a proper formatted multi-line comment.
		 * See?
		 */
Quoted 10 lines
drivers/clocksource/hyperv_timer.c +10 -0
+		 * reserved bits in registers.  So read the existing
+		 * value, preserve the low order 12 bits, and add in
+		 * the guest physical address (which already has at
+		 * least the low 12 bits set to zero since it is page
+		 * aligned). Also set the "enable" bit, which is bit 0.
+		 */
+		hv_get_reference_tsc(tsc_msr);
+		tsc_msr &= GENMASK_ULL(11, 0);
+		tsc_msr = tsc_msr | 0x1 | (u64)phys_addr;
+		hv_set_reference_tsc(tsc_msr);

drivers/clocksource/hyperv_timer.c void hv_stimer_global_cleanup(void) · line 300

Quoted 10 lines
drivers/clocksource/hyperv_timer.c +10 -0
+
+		hv_set_clocksource_vdso(hyperv_cs_tsc);
+		clocksource_register_hz(&hyperv_cs_tsc, NSEC_PER_SEC/100);
+
+		/*
+		 * sched_clock_register is needed on ARM64 but
+		 * is a no-op on x86
+		 */
+		sched_clock_register(read_hv_sched_clock_tsc,
+						64, HV_CLOCK_HZ);

If you have to use line breaks, then please align them properly

		sched_clock_register(read_hv_sched_clock_tsc,
				     64, HV_CLOCK_HZ);
Quoted 8 lines
drivers/clocksource/hyperv_timer.c +8 -0
+		return;
+	}
+register_msr_cs:
+#endif
+	/*
+	 * For 32 bit guests just use the MSR based mechanism for reading
+	 * the partition counter.
+	 */

drivers/clocksource/hyperv_timer.c void hv_stimer_global_cleanup(void) · line 318

Quoted 10 lines
drivers/clocksource/hyperv_timer.c +10 -0
+	hyperv_cs = &hyperv_cs_msr;
+	if (ms_hyperv.features & HV_MSR_TIME_REF_COUNT_AVAILABLE) {
+		clocksource_register_hz(&hyperv_cs_msr, NSEC_PER_SEC/100);
+
+		/*
+		 * sched_clock_register is needed on ARM64 but
+		 * is a no-op on x86
+		 */
+		sched_clock_register(read_hv_sched_clock_msr,
+						64, HV_CLOCK_HZ);

Ditto. But again if you return when the feature is not available you spare an indent and the line break.

Thanks,

tglx

#6 lore raw

RE: [PATCH v4 2/2] Drivers: hv: Move Hyper-V clocksource code to new clocksource driver

From: Thomas Gleixner <[email protected]> Sent: Saturday, June 29, 2019 12:04 AM

Quoted 18 lines · Michael Kelley

See the other reply vs. prefix and summary

Quoted 1 line

Continue consolidating Hyper-V clock and timer code into an ISA

...

Quoted 9 lines
drivers/clocksource/hyperv_timer.c +9 -0
+#ifdef CONFIG_HYPERV_TSCPAGE
+
+static struct ms_hyperv_tsc_page *tsc_pg;
+
+struct ms_hyperv_tsc_page *hv_get_tsc_page(void)
+{
+	return tsc_pg;
+}
+EXPORT_SYMBOL_GPL(hv_get_tsc_page);
Quoted 3 lines
Quoted 1 line

+static u64 read_hv_sched_clock_tsc(void)

Lacks a notrace annotation as it is used as sched clock.

Will add to this and to read_hv_sched_clock_msr().

Quoted 15 lines
Quoted 15 lines
drivers/clocksource/hyperv_timer.c +14 -0
+{
+	u64 current_tick = hv_read_tsc_page(tsc_pg);
+
+	if (current_tick == U64_MAX)
+		hv_get_time_ref_count(current_tick);
+
+	return current_tick;
+}
+
+static u64 read_hv_clock_tsc(struct clocksource *arg)
+{
+	return read_hv_sched_clock_tsc();
+}
+
Quoted 8 lines

drivers/clocksource/hyperv_timer.c void hv_stimer_global_cleanup(void) · line 236

Quoted 6 lines
drivers/clocksource/hyperv_timer.c +6 -0
+static struct clocksource hyperv_cs_tsc = {
+		.name		= "hyperv_clocksource_tsc_page",
+		.rating		= 400,
+		.read		= read_hv_clock_tsc,
+		.mask		= CLOCKSOURCE_MASK(64),
+		.flags		= CLOCK_SOURCE_IS_CONTINUOUS,

One tab indent is sufficient

Will fix.

Quoted 11 lines

drivers/clocksource/hyperv_timer.c void hv_stimer_global_cleanup(void) · line 267

Quoted 7 lines
drivers/clocksource/hyperv_timer.c +5 -0
+};
+#endif
 
+void __init hv_init_clocksource(void)
+{
+#ifdef CONFIG_HYPERV_TSCPAGE

Can you please avoid ifdef blocks in the code and just split out this into a helper function and provide an empty stub for the TSCPAGE=n case?

Quoted 2 lines

Then you can spare one indent and the ifdeffery is confined. Get's rid of the goto as well.

I was trying not to change code unrelated to reorganizing into a clocksource driver, but I've coded the approach you suggest and the result is a lot cleaner. I'll include the new approach in v5 of the patch.

Quoted 17 lines
Quoted 4 lines
drivers/clocksource/hyperv_timer.c +3 -0
+	if (ms_hyperv.features & HV_MSR_REFERENCE_TSC_AVAILABLE) {
+
+		u64		tsc_msr;

drivers/clocksource/hyperv_timer.c void hv_stimer_global_cleanup(void) · line 280

Quoted 10 lines
drivers/clocksource/hyperv_timer.c +10 -0
+		phys_addr_t	phys_addr;
+
+		tsc_pg = vmalloc(PAGE_SIZE);
+		if (!tsc_pg)
+			goto register_msr_cs;
+
+		hyperv_cs = &hyperv_cs_tsc;
+		phys_addr = page_to_phys(vmalloc_to_page(tsc_pg));
+
+		/* The Hyper-V TLFS specifies to preserve the value of

Please use proper multi line comments formatting:

Quoted 4 lines
		/*
		 * This is a proper formatted multi-line comment.
		 * See?
		 */

Argh. Old habits ...

Quoted 24 lines
Quoted 11 lines
drivers/clocksource/hyperv_timer.c +10 -0
+		 * reserved bits in registers.  So read the existing
+		 * value, preserve the low order 12 bits, and add in
+		 * the guest physical address (which already has at
+		 * least the low 12 bits set to zero since it is page
+		 * aligned). Also set the "enable" bit, which is bit 0.
+		 */
+		hv_get_reference_tsc(tsc_msr);
+		tsc_msr &= GENMASK_ULL(11, 0);
+		tsc_msr = tsc_msr | 0x1 | (u64)phys_addr;
+		hv_set_reference_tsc(tsc_msr);

drivers/clocksource/hyperv_timer.c void hv_stimer_global_cleanup(void) · line 300

Quoted 10 lines
drivers/clocksource/hyperv_timer.c +10 -0
+
+		hv_set_clocksource_vdso(hyperv_cs_tsc);
+		clocksource_register_hz(&hyperv_cs_tsc, NSEC_PER_SEC/100);
+
+		/*
+		 * sched_clock_register is needed on ARM64 but
+		 * is a no-op on x86
+		 */
+		sched_clock_register(read_hv_sched_clock_tsc,
+						64, HV_CLOCK_HZ);

If you have to use line breaks, then please align them properly

Quoted 3 lines
		sched_clock_register(read_hv_sched_clock_tsc,
				     64, HV_CLOCK_HZ);

Got it. The line break is no longer needed in the new version anyway.

Quoted 14 lines
Quoted 14 lines
drivers/clocksource/hyperv_timer.c +13 -0
+		return;
+	}
+register_msr_cs:
+#endif
+	/*
+	 * For 32 bit guests just use the MSR based mechanism for reading
+	 * the partition counter.
+	 */
+	hyperv_cs = &hyperv_cs_msr;
+	if (ms_hyperv.features & HV_MSR_TIME_REF_COUNT_AVAILABLE) {
+		clocksource_register_hz(&hyperv_cs_msr, NSEC_PER_SEC/100);
+
+		/*
Quoted 8 lines

drivers/clocksource/hyperv_timer.c void hv_stimer_global_cleanup(void) · line 318

Quoted 5 lines
drivers/clocksource/hyperv_timer.c +5 -0
+		 * sched_clock_register is needed on ARM64 but
+		 * is a no-op on x86
+		 */
+		sched_clock_register(read_hv_sched_clock_msr,
+						64, HV_CLOCK_HZ);

Ditto. But again if you return when the feature is not available you spare an indent and the line break.

Agreed.

Thanks,

Michael