Skip to contentSkip to conversation

xen-time: decreasing the rating of the xen clocksource below that of the tsc clocksource for dom0's

RFC PATCH In review Imre Palik 5 participants 11 messages Jan 7, 2015 – Jan 12, 2015

Subsystems (2)
lore
Share & tools
lore

Imre Palik

author
#0 lore raw

RFC PATCH 1/1 xen-time: decreasing the rating of the xen clocksource below that of the tsc clocksource for dom0's

From: "Palik, Imre" <[email protected]>

In Dom0's the use of the TSC clocksource (whenever it is stable enough to be used) instead of the Xen clocksource should not cause any issues, as Dom0 VMs never live-migrated. The TSC clocksource is somewhat more efficient than the Xen paravirtualised clocksource, thus it should have higher rating.

Cc Anthony Liguori <[email protected]>
---
 arch/x86/xen/time.c |    4 ++++
 1 file changed, 4 insertions(+)
arch/x86/xen/time.c +4 -0
index f473d26..c768726 100644
@@ -487,6 +487,10 @@ static void __init xen_time_init(void)
 	int cpu = smp_processor_id();
 	struct timespec tp;
 
+	/* As Dom0 is never moved, no penalty on using TSC there */
+	if (xen_initial_domain())
+		xen_clocksource.rating = 275;
+
 	clocksource_register_hz(&xen_clocksource, NSEC_PER_SEC);
 
 	if (HYPERVISOR_vcpu_op(VCPUOP_stop_periodic_timer, cpu, NULL) == 0) {
Signature · 1 line
1.7.9.5
#1 lore raw
Quoted 5 lines · Imre Palik

From: "Palik, Imre" <[email protected]>

In Dom0's the use of the TSC clocksource (whenever it is stable enough to be used) instead of the Xen clocksource should not cause any issues, as Dom0 VMs never live-migrated.

Is this still true given that dom0's vcpus are migrated amongst pcpus on the host? The tsc are not synchronised on some generations of hardware so the result there would be the TSC appearing to do very odd things under dom0's feet. Does Linux cope with that or does it not matter for some other reason?

Quoted 22 lines
Cc Anthony Liguori <[email protected]>
---
 arch/x86/xen/time.c |    4 ++++
 1 file changed, 4 insertions(+)
arch/x86/xen/time.c +4 -0
index f473d26..c768726 100644
@@ -487,6 +487,10 @@ static void __init xen_time_init(void)
 	int cpu = smp_processor_id();
 	struct timespec tp;
 
+	/* As Dom0 is never moved, no penalty on using TSC there */
+	if (xen_initial_domain())
+		xen_clocksource.rating = 275;
+
 	clocksource_register_hz(&xen_clocksource, NSEC_PER_SEC);
 
 	if (HYPERVISOR_vcpu_op(VCPUOP_stop_periodic_timer, cpu, NULL) == 0) {
#2 lore raw
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 07.01.15 at 17:30, <[email protected]> wrote:
Quoted 5 lines · Imre Palik

From: "Palik, Imre" <[email protected]>

In Dom0's the use of the TSC clocksource (whenever it is stable enough to be used) instead of the Xen clocksource should not cause any issues, as Dom0 VMs never live-migrated.

Quoted 5 lines

Is this still true given that dom0's vcpus are migrated amongst pcpus on the host? The tsc are not synchronised on some generations of hardware so the result there would be the TSC appearing to do very odd things under dom0's feet. Does Linux cope with that or does it not matter for some other reason?

Indeed. The textual qualification above ("whenever it is stable enough") isn't being expressed in the code change at all.

Jan

#3 lore raw
In reply to Ian Campbell
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 07.01.15 at 17:30, <[email protected]> wrote:
Quoted 5 lines · Imre Palik

From: "Palik, Imre" <[email protected]>

In Dom0's the use of the TSC clocksource (whenever it is stable enough to be used) instead of the Xen clocksource should not cause any issues, as Dom0 VMs never live-migrated.

Quoted 5 lines

Is this still true given that dom0's vcpus are migrated amongst pcpus on the host? The tsc are not synchronised on some generations of hardware so the result there would be the TSC appearing to do very odd things under dom0's feet. Does Linux cope with that or does it not matter for some other reason?

Indeed. The textual qualification above ("whenever it is stable enough") isn't being expressed in the code change at all.

Jan

author
#4 lore raw
In reply to Ian Campbell
Quoted 7 lines · Ian Campbell
Quoted 5 lines · Imre Palik

From: "Palik, Imre" <[email protected]>

In Dom0's the use of the TSC clocksource (whenever it is stable enough to be used) instead of the Xen clocksource should not cause any issues, as Dom0 VMs never live-migrated.

Quoted 5 lines

Is this still true given that dom0's vcpus are migrated amongst pcpus on the host? The tsc are not synchronised on some generations of hardware so the result there would be the TSC appearing to do very odd things under dom0's feet. Does Linux cope with that or does it not matter for some other reason?

First of all, if the initial pcpus are not synchronised, linux won't consider TSC as a viable clocksource.

If the initial pcpus are synchronised, but then the dom0 vcpus are migrated to unsynchronised pcpus, then hopefully the tsc watchdog catches the issue, and the kernel falls back to the xen clocksource. The issue here is that the watchdog can only detect clock skews bigger than 0.0625s/0.5s. Hopefully this is enough to avoid the weirdest things.

Also, some parts of the kernel (e.g., scheduling) will always use the paravirt clock. No matter what priorities are set.

So it should be safe for some definition of safe.
But I was unable to test it on a hardware platform that would hit the problematic case described above.

Quoted 25 lines
Quoted 23 lines
Cc Anthony Liguori <[email protected]>
---
 arch/x86/xen/time.c |    4 ++++
 1 file changed, 4 insertions(+)
arch/x86/xen/time.c +4 -0
index f473d26..c768726 100644
@@ -487,6 +487,10 @@ static void __init xen_time_init(void)
 	int cpu = smp_processor_id();
 	struct timespec tp;
 
+	/* As Dom0 is never moved, no penalty on using TSC there */
+	if (xen_initial_domain())
+		xen_clocksource.rating = 275;
+
 	clocksource_register_hz(&xen_clocksource, NSEC_PER_SEC);
 
 	if (HYPERVISOR_vcpu_op(VCPUOP_stop_periodic_timer, cpu, NULL) == 0) {
author
#5 lore raw
In reply to Ian Campbell
Quoted 7 lines · Ian Campbell
Quoted 5 lines · Imre Palik

From: "Palik, Imre" <[email protected]>

In Dom0's the use of the TSC clocksource (whenever it is stable enough to be used) instead of the Xen clocksource should not cause any issues, as Dom0 VMs never live-migrated.

Quoted 5 lines

Is this still true given that dom0's vcpus are migrated amongst pcpus on the host? The tsc are not synchronised on some generations of hardware so the result there would be the TSC appearing to do very odd things under dom0's feet. Does Linux cope with that or does it not matter for some other reason?

First of all, if the initial pcpus are not synchronised, linux won't consider TSC as a viable clocksource.

If the initial pcpus are synchronised, but then the dom0 vcpus are migrated to unsynchronised pcpus, then hopefully the tsc watchdog catches the issue, and the kernel falls back to the xen clocksource. The issue here is that the watchdog can only detect clock skews bigger than 0.0625s/0.5s. Hopefully this is enough to avoid the weirdest things.

Also, some parts of the kernel (e.g., scheduling) will always use the paravirt clock. No matter what priorities are set.

So it should be safe for some definition of safe.
But I was unable to test it on a hardware platform that would hit the problematic case described above.

Quoted 25 lines
Quoted 23 lines
Cc Anthony Liguori <[email protected]>
---
 arch/x86/xen/time.c |    4 ++++
 1 file changed, 4 insertions(+)
arch/x86/xen/time.c +4 -0
index f473d26..c768726 100644
@@ -487,6 +487,10 @@ static void __init xen_time_init(void)
 	int cpu = smp_processor_id();
 	struct timespec tp;
 
+	/* As Dom0 is never moved, no penalty on using TSC there */
+	if (xen_initial_domain())
+		xen_clocksource.rating = 275;
+
 	clocksource_register_hz(&xen_clocksource, NSEC_PER_SEC);
 
 	if (HYPERVISOR_vcpu_op(VCPUOP_stop_periodic_timer, cpu, NULL) == 0) {
#6 lore raw
Quoted 14 lines · Imre Palik
Quoted 7 lines · Ian Campbell
Quoted 5 lines · Imre Palik

From: "Palik, Imre" <[email protected]>

In Dom0's the use of the TSC clocksource (whenever it is stable enough to be used) instead of the Xen clocksource should not cause any issues, as Dom0 VMs never live-migrated.

Quoted 5 lines

Is this still true given that dom0's vcpus are migrated amongst pcpus on the host? The tsc are not synchronised on some generations of hardware so the result there would be the TSC appearing to do very odd things under dom0's feet. Does Linux cope with that or does it not matter for some other reason?

Quoted 8 lines

First of all, if the initial pcpus are not synchronised, linux won't consider TSC as a viable clocksource.

If the initial pcpus are synchronised, but then the dom0 vcpus are migrated to unsynchronised pcpus, then hopefully the tsc watchdog catches the issue, and the kernel falls back to the xen clocksource. The issue here is that the watchdog can only detect clock skews bigger than 0.0625s/0.5s. Hopefully this is enough to avoid the weirdest things.

I don't think any such hardware exists. Either TSC is synchronized across all CPUs or none.

Quoted 6 lines

Also, some parts of the kernel (e.g., scheduling) will always use the paravirt clock. No matter what priorities are set.

So it should be safe for some definition of safe.
But I was unable to test it on a hardware platform that would hit the problematic case described above.

Ok. Can you list the various time sources and their ratings in the commit message for clarity. i.e, to justify 275 (below TSC = 300. above hpet = 250).

David

#7 lore raw
In reply to Imre Palik
Quoted 14 lines · Imre Palik
Quoted 7 lines · Ian Campbell
Quoted 5 lines · Imre Palik

From: "Palik, Imre" <[email protected]>

In Dom0's the use of the TSC clocksource (whenever it is stable enough to be used) instead of the Xen clocksource should not cause any issues, as Dom0 VMs never live-migrated.

Quoted 5 lines

Is this still true given that dom0's vcpus are migrated amongst pcpus on the host? The tsc are not synchronised on some generations of hardware so the result there would be the TSC appearing to do very odd things under dom0's feet. Does Linux cope with that or does it not matter for some other reason?

Quoted 8 lines

First of all, if the initial pcpus are not synchronised, linux won't consider TSC as a viable clocksource.

If the initial pcpus are synchronised, but then the dom0 vcpus are migrated to unsynchronised pcpus, then hopefully the tsc watchdog catches the issue, and the kernel falls back to the xen clocksource. The issue here is that the watchdog can only detect clock skews bigger than 0.0625s/0.5s. Hopefully this is enough to avoid the weirdest things.

I don't think any such hardware exists. Either TSC is synchronized across all CPUs or none.

Quoted 6 lines

Also, some parts of the kernel (e.g., scheduling) will always use the paravirt clock. No matter what priorities are set.

So it should be safe for some definition of safe.
But I was unable to test it on a hardware platform that would hit the problematic case described above.

Ok. Can you list the various time sources and their ratings in the commit message for clarity. i.e, to justify 275 (below TSC = 300. above hpet = 250).

David

#8 lore raw
In reply to Imre Palik
Quoted 5 lines · Imre Palik

From: "Palik, Imre" <[email protected]>

In Dom0's the use of the TSC clocksource (whenever it is stable enough to be used) instead of the Xen clocksource should not cause any issues, as Dom0 VMs never live-migrated.

Is this still true given that dom0's vcpus are migrated amongst pcpus on the host? The tsc are not synchronised on some generations of hardware so the result there would be the TSC appearing to do very odd things under dom0's feet. Does Linux cope with that or does it not matter for some other reason?

Quoted 22 lines
Cc Anthony Liguori <[email protected]>
---
 arch/x86/xen/time.c |    4 ++++
 1 file changed, 4 insertions(+)
arch/x86/xen/time.c +4 -0
index f473d26..c768726 100644
@@ -487,6 +487,10 @@ static void __init xen_time_init(void)
 	int cpu = smp_processor_id();
 	struct timespec tp;
 
+	/* As Dom0 is never moved, no penalty on using TSC there */
+	if (xen_initial_domain())
+		xen_clocksource.rating = 275;
+
 	clocksource_register_hz(&xen_clocksource, NSEC_PER_SEC);
 
 	if (HYPERVISOR_vcpu_op(VCPUOP_stop_periodic_timer, cpu, NULL) == 0) {
#9 lore raw
In reply to Imre Palik
Quoted 19 lines · Imre Palik

From: "Palik, Imre" <[email protected]>

In Dom0's the use of the TSC clocksource (whenever it is stable enough to be used) instead of the Xen clocksource should not cause any issues, as Dom0 VMs never live-migrated. The TSC clocksource is somewhat more efficient than the Xen paravirtualised clocksource, thus it should have higher rating.

Cc Anthony Liguori <[email protected]>
---
  arch/x86/xen/time.c |    4 ++++
  1 file changed, 4 insertions(+)
arch/x86/xen/time.c +0 -0
index f473d26..c768726 100644

arch/x86/xen/time.c static void __init xen_time_init(void) · line 492

Quoted 6 lines
arch/x86/xen/time.c +3 -0
@@ -487,6 +487,10 @@ static void __init xen_time_init(void)
  	int cpu = smp_processor_id();
  	struct timespec tp;
 
+	/* As Dom0 is never moved, no penalty on using TSC there */
+	if (xen_initial_domain())
+		xen_clocksource.rating = 275;

Should this only be limited to dom0? Can we do this for guests running with TSC_MODE_NEVER_EMULATE as well?

We also have disable_migrate flag for guests (but it doesn't appear to be accessible to a guest kernel).

-boris

Quoted 4 lines
arch/x86/xen/time.c +1 -0
+
  	clocksource_register_hz(&xen_clocksource, NSEC_PER_SEC);
 
  	if (HYPERVISOR_vcpu_op(VCPUOP_stop_periodic_timer, cpu, NULL) == 0) {
#10 lore raw
In reply to Imre Palik
Quoted 19 lines · Imre Palik

From: "Palik, Imre" <[email protected]>

In Dom0's the use of the TSC clocksource (whenever it is stable enough to be used) instead of the Xen clocksource should not cause any issues, as Dom0 VMs never live-migrated. The TSC clocksource is somewhat more efficient than the Xen paravirtualised clocksource, thus it should have higher rating.

Cc Anthony Liguori <[email protected]>
---
  arch/x86/xen/time.c |    4 ++++
  1 file changed, 4 insertions(+)
arch/x86/xen/time.c +0 -0
index f473d26..c768726 100644

arch/x86/xen/time.c static void __init xen_time_init(void) · line 492

Quoted 6 lines
arch/x86/xen/time.c +3 -0
@@ -487,6 +487,10 @@ static void __init xen_time_init(void)
  	int cpu = smp_processor_id();
  	struct timespec tp;
 
+	/* As Dom0 is never moved, no penalty on using TSC there */
+	if (xen_initial_domain())
+		xen_clocksource.rating = 275;

Should this only be limited to dom0? Can we do this for guests running with TSC_MODE_NEVER_EMULATE as well?

We also have disable_migrate flag for guests (but it doesn't appear to be accessible to a guest kernel).

-boris

Quoted 4 lines
arch/x86/xen/time.c +1 -0
+
  	clocksource_register_hz(&xen_clocksource, NSEC_PER_SEC);
 
  	if (HYPERVISOR_vcpu_op(VCPUOP_stop_periodic_timer, cpu, NULL) == 0) {