Skip to contentSkip to conversation

clocksource: hyper-v: Use lapic timer in a TDX VM without paravisor

PATCH Superseded Dexuan Cui 3 participants 4 messages Jun 19, 2024 – Jun 20, 2024

Subsystems (5)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Dexuan Cui

author maintainer
#0 lore raw

PATCH 1/1 clocksource: hyper-v: Use lapic timer in a TDX VM without paravisor

In a TDX VM without paravisor, currently the default timer is the Hyper-V timer, which depends on the slow VM Reference Counter MSR: the Hyper-V TSC page is not enabled in such a VM because the VM uses Invariant TSC as a better clocksource and it's challenging to mark the Hyper-V TSC page shared in very early boot.

Lower the rating of the Hyper-V timer so the local APIC timer becomes the the default timer in such a VM. This change should cause no perceivable performance difference.

---
 arch/x86/kernel/cpu/mshyperv.c     |  6 +++++-
 drivers/clocksource/hyperv_timer.c | 16 +++++++++++++++-
 2 files changed, 20 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/mshyperv.c +5 -1
index e0fd57a8ba840..745af47ca0459 100644
@@ -449,9 +449,13 @@ static void __init ms_hyperv_init_platform(void)
 			ms_hyperv.hints &= ~HV_X64_APIC_ACCESS_RECOMMENDED;
 
 			if (!ms_hyperv.paravisor_present) {
-				/* To be supported: more work is required.  */
+				/* Use Invariant TSC as a better clocksource. */
 				ms_hyperv.features &= ~HV_MSR_REFERENCE_TSC_AVAILABLE;
 
+				/* Use the Ref Counter in case Invariant TSC is unavailable. */
+				if (!(ms_hyperv.features & HV_ACCESS_TSC_INVARIANT))
+					pr_warn("Hyper-V: Invariant TSC is unavailable\n");
+
 				/* HV_MSR_CRASH_CTL is unsupported. */
 				ms_hyperv.misc_features &= ~HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE;
 
drivers/clocksource/hyperv_timer.c +15 -1
index b2a080647e413..99177835cadec 100644
@@ -137,7 +137,21 @@ static int hv_stimer_init(unsigned int cpu)
 	ce->name = "Hyper-V clockevent";
 	ce->features = CLOCK_EVT_FEAT_ONESHOT;
 	ce->cpumask = cpumask_of(cpu);
-	ce->rating = 1000;
+
+	/*
+	 * Lower the rating of the Hyper-V timer in a TDX VM without paravisor,
+	 * so the local APIC timer (lapic_clockevent) is the default timer in
+	 * such a VM. The Hyper-V timer is not preferred in such a VM because
+	 * it depends on the slow VM Reference Counter MSR (the Hyper-V TSC
+	 * page is not enbled in such a VM because the VM uses Invariant TSC
+	 * as a better clocksource and it's challenging to mark the Hyper-V
+	 * TSC page shared in very early boot).
+	 */
+	if (!ms_hyperv.paravisor_present && hv_isolation_type_tdx())
+		ce->rating = 90;
+	else
+		ce->rating = 1000;
+
 	ce->set_state_shutdown = hv_ce_shutdown;
 	ce->set_state_oneshot = hv_ce_set_oneshot;
 	ce->set_next_event = hv_ce_set_next_event;
Signature · 1 line
2.25.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 53 lines · Dexuan Cui

In a TDX VM without paravisor, currently the default timer is the Hyper-V timer, which depends on the slow VM Reference Counter MSR: the Hyper-V TSC page is not enabled in such a VM because the VM uses Invariant TSC as a better clocksource and it's challenging to mark the Hyper-V TSC page shared in very early boot.

Lower the rating of the Hyper-V timer so the local APIC timer becomes the the default timer in such a VM. This change should cause no perceivable performance difference.

---
  arch/x86/kernel/cpu/mshyperv.c     |  6 +++++-
  drivers/clocksource/hyperv_timer.c | 16 +++++++++++++++-
  2 files changed, 20 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/mshyperv.c +5 -1
index e0fd57a8ba840..745af47ca0459 100644
@@ -449,9 +449,13 @@ static void __init ms_hyperv_init_platform(void)
  			ms_hyperv.hints &= ~HV_X64_APIC_ACCESS_RECOMMENDED;
 
  			if (!ms_hyperv.paravisor_present) {
-				/* To be supported: more work is required.  */
+				/* Use Invariant TSC as a better clocksource. */
  				ms_hyperv.features &= ~HV_MSR_REFERENCE_TSC_AVAILABLE;
 
+				/* Use the Ref Counter in case Invariant TSC is unavailable. */
+				if (!(ms_hyperv.features & HV_ACCESS_TSC_INVARIANT))
+					pr_warn("Hyper-V: Invariant TSC is unavailable\n");
+
  				/* HV_MSR_CRASH_CTL is unsupported. */
  				ms_hyperv.misc_features &= ~HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE;
 
drivers/clocksource/hyperv_timer.c +8 -1
index b2a080647e413..99177835cadec 100644
@@ -137,7 +137,21 @@ static int hv_stimer_init(unsigned int cpu)
  	ce->name = "Hyper-V clockevent";
  	ce->features = CLOCK_EVT_FEAT_ONESHOT;
  	ce->cpumask = cpumask_of(cpu);
-	ce->rating = 1000;
+
+	/*
+	 * Lower the rating of the Hyper-V timer in a TDX VM without paravisor,
+	 * so the local APIC timer (lapic_clockevent) is the default timer in
+	 * such a VM. The Hyper-V timer is not preferred in such a VM because
+	 * it depends on the slow VM Reference Counter MSR (the Hyper-V TSC
+	 * page is not enbled in such a VM because the VM uses Invariant TSC
+	 * as a better clocksource and it's challenging to mark the Hyper-V

drivers/clocksource/hyperv_timer.c static int hv_stimer_init(unsigned int cpu) · line 157

Quoted 10 lines
drivers/clocksource/hyperv_timer.c +7 -0
+	 * TSC page shared in very early boot).
+	 */
+	if (!ms_hyperv.paravisor_present && hv_isolation_type_tdx())
+		ce->rating = 90;
+	else
+		ce->rating = 1000;
+
  	ce->set_state_shutdown = hv_ce_shutdown;
  	ce->set_state_oneshot = hv_ce_set_oneshot;
  	ce->set_next_event = hv_ce_set_next_event;

LGTM.

Signature · 2 lines
Thank you,
Roman
#2 lore raw
In reply to Dexuan Cui

From: Dexuan Cui <[email protected]> Sent: Tuesday, June 18, 2024 5:25 PM

Quoted 23 lines

In a TDX VM without paravisor, currently the default timer is the Hyper-V timer, which depends on the slow VM Reference Counter MSR: the Hyper-V TSC page is not enabled in such a VM because the VM uses Invariant TSC as a better clocksource and it's challenging to mark the Hyper-V TSC page shared in very early boot.

Lower the rating of the Hyper-V timer so the local APIC timer becomes the the default timer in such a VM. This change should cause no perceivable performance difference.

---
 arch/x86/kernel/cpu/mshyperv.c     |  6 +++++-
 drivers/clocksource/hyperv_timer.c | 16 +++++++++++++++-
 2 files changed, 20 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/mshyperv.c +0 -0
index e0fd57a8ba840..745af47ca0459 100644

arch/x86/kernel/cpu/mshyperv.c static void __init ms_hyperv_init_platform(void) · line 452

Quoted 5 lines
arch/x86/kernel/cpu/mshyperv.c +1 -1
@@ -449,9 +449,13 @@ static void __init ms_hyperv_init_platform(void)
 			ms_hyperv.hints &= ~HV_X64_APIC_ACCESS_RECOMMENDED;
 
 			if (!ms_hyperv.paravisor_present) {
-				/* To be supported: more work is required.  */
+				/* Use Invariant TSC as a better clocksource. */

I got confused by this comment, partly because I've forgotten the meaning of the ms_hyperv.feature flags. :-( Perhaps you could be more explicit in the comment and say "Mark the Hyper-V TSC page feature as disabled in a TDX VM so that the Invariant TSC, which is a better clocksource anyway, is used instead."

arch/x86/kernel/cpu/mshyperv.c static void __init ms_hyperv_init_platform(void) · line 457

Quoted 5 lines
arch/x86/kernel/cpu/mshyperv.c +3 -0
 				ms_hyperv.features &= ~HV_MSR_REFERENCE_TSC_AVAILABLE;
 
+				/* Use the Ref Counter in case Invariant TSC is unavailable. */
+				if (!(ms_hyperv.features & HV_ACCESS_TSC_INVARIANT))
+					pr_warn("Hyper-V: Invariant TSC is unavailable\n");

The above comment was even more confusing, because the code block doesn't do anything except print a message. The code doesn't force the use of the Ref Counter. I'd suggest something like: "The Invariant TSC is expected to be available, but if not, print a warning message. The slower Hyper-V MSR-based Ref Counter should end up being the clocksource."

Michael

Quoted 34 lines
arch/x86/kernel/cpu/mshyperv.c +1 -0
+
 				/* HV_MSR_CRASH_CTL is unsupported. */
 				ms_hyperv.misc_features &= ~HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE;
 
drivers/clocksource/hyperv_timer.c +15 -2
index b2a080647e413..99177835cadec 100644
@@ -137,7 +137,21 @@ static int hv_stimer_init(unsigned int cpu)
 	ce->name = "Hyper-V clockevent";
 	ce->features = CLOCK_EVT_FEAT_ONESHOT;
 	ce->cpumask = cpumask_of(cpu);
-	ce->rating = 1000;
+
+	/*
+	 * Lower the rating of the Hyper-V timer in a TDX VM without paravisor,
+	 * so the local APIC timer (lapic_clockevent) is the default timer in
+	 * such a VM. The Hyper-V timer is not preferred in such a VM because
+	 * it depends on the slow VM Reference Counter MSR (the Hyper-V TSC
+	 * page is not enbled in such a VM because the VM uses Invariant TSC
+	 * as a better clocksource and it's challenging to mark the Hyper-V
+	 * TSC page shared in very early boot).
+	 */
+	if (!ms_hyperv.paravisor_present && hv_isolation_type_tdx())
+		ce->rating = 90;
+	else
+		ce->rating = 1000;
+
 	ce->set_state_shutdown = hv_ce_shutdown;
 	ce->set_state_oneshot = hv_ce_set_oneshot;
 	ce->set_next_event = hv_ce_set_next_event;
--

2.25.1

author maintainer
#3 lore raw
Quoted 32 lines

From: Michael Kelley <[email protected]>
Sent: Thursday, June 20, 2024 2:19 PM
To: Dexuan Cui <[email protected]>; KY Srinivasan [...]

arch/x86/kernel/cpu/mshyperv.c static void __init ms_hyperv_init_platform(void) · line 449

Quoted 4 lines
arch/x86/kernel/cpu/mshyperv.c +0 -0
@@ -449,9 +449,13 @@ static void __init ms_hyperv_init_platform(void)
 			ms_hyperv.hints &=

~HV_X64_APIC_ACCESS_RECOMMENDED;

arch/x86/kernel/cpu/mshyperv.c static void __init ms_hyperv_init_platform(void) · line 451

Quoted 3 lines
arch/x86/kernel/cpu/mshyperv.c +0 -1
 			if (!ms_hyperv.paravisor_present) {
-				/* To be supported: more work is required.

*/

arch/x86/kernel/cpu/mshyperv.c static void __init ms_hyperv_init_platform(void) · line 451

Quoted 1 line
arch/x86/kernel/cpu/mshyperv.c +1 -0
+				/* Use Invariant TSC as a better

clocksource. */

I got confused by this comment, partly because I've forgotten the meaning of the ms_hyperv.feature flags. :-( Perhaps you could be more explicit in the comment and say "Mark the Hyper-V TSC page feature as disabled in a TDX VM so that the Invariant TSC, which is a better clocksource anyway, is used instead."

arch/x86/kernel/cpu/mshyperv.c static void __init ms_hyperv_init_platform(void) · line 452

Quoted 1 line
arch/x86/kernel/cpu/mshyperv.c +0 -0
 				ms_hyperv.features &=

~HV_MSR_REFERENCE_TSC_AVAILABLE;

arch/x86/kernel/cpu/mshyperv.c static void __init ms_hyperv_init_platform(void) · line 453

Quoted 2 lines
arch/x86/kernel/cpu/mshyperv.c +1 -0
+				/* Use the Ref Counter in case Invariant

TSC is unavailable. */

arch/x86/kernel/cpu/mshyperv.c static void __init ms_hyperv_init_platform(void) · line 454

Quoted 1 line
arch/x86/kernel/cpu/mshyperv.c +1 -0
+				if (!(ms_hyperv.features &

HV_ACCESS_TSC_INVARIANT))

arch/x86/kernel/cpu/mshyperv.c static void __init ms_hyperv_init_platform(void) · line 455

Quoted 1 line
arch/x86/kernel/cpu/mshyperv.c +1 -0
+					pr_warn("Hyper-V: Invariant TSC is

unavailable\n");

Quoted 8 lines

The above comment was even more confusing, because the code block doesn't do anything except print a message. The code doesn't force the use of the Ref Counter. I'd suggest something like: "The Invariant TSC is expected to be available, but if not, print a warning message. The slower Hyper-V MSR-based Ref Counter should end up being the clocksource."

Michael

Thanks for the good "comments"! :-)

I'm going to post v2 with the change below.

arch/x86/kernel/cpu/mshyperv.c +15 -1
index e0fd57a8ba840..954b7cbfa2f02 100644
@@ -449,9 +449,23 @@ static void __init ms_hyperv_init_platform(void)
                        ms_hyperv.hints &= ~HV_X64_APIC_ACCESS_RECOMMENDED;
 
                        if (!ms_hyperv.paravisor_present) {
-                               /* To be supported: more work is required.  */
+                               /*
+                                * Mark the Hyper-V TSC page feature as disabled
+                                * in a TDX VM without paravisor so that the
+                                * Invariant TSC, which is a better clocksource
+                                * anyway, is used instead.
+                                */
                                ms_hyperv.features &= ~HV_MSR_REFERENCE_TSC_AVAILABLE;
 
+                               /*
+                                * The Invariant TSC is expected to be available
+                                * in a TDX VM without paravisor, but if not,
+                                * print a warning message. The slower Hyper-V MSR-based
+                                * Ref Counter should end up being the clocksource.
+                                */
+                               if (!(ms_hyperv.features & HV_ACCESS_TSC_INVARIANT))
+                                       pr_warn("Hyper-V: Invariant TSC is unavailable\n");
+
                                /* HV_MSR_CRASH_CTL is unsupported. */
                                ms_hyperv.misc_features &= ~HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE;