Skip to contentSkip to conversation

KVM: x86: Include host suspended time in steal time.

PATCH Superseded Suleiman Souhlal 3 participants 12 messages Aug 20, 2024 – Sep 25, 2024

Subsystems (4)

v2 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

v2 cover KVM: x86: Include host suspended time in steal time.

This series makes it so that the time that the host is suspended is included in guests' steal time.

When the host resumes from a suspend, the guest thinks any task that was running during the suspend ran for a long time, even though the effective run time was much shorter, which can end up having negative effects with scheduling. This can be particularly noticeable if the guest task was RT, as it can end up getting throttled for a long time.

To mitigate this issue, we include the time that the host was suspended in steal time, which lets the guest can subtract the duration from the tasks' runtime.

(v1 was at https://lore.kernel.org/kvm/[email protected]/)

v1 -> v2:
- Accumulate suspend time at machine-independent kvm layer and track per-VCPU instead of per-VM.
- Document changes.

Suleiman Souhlal (3):
KVM: Introduce kvm_total_suspend_ns().
KVM: x86: Include host suspended time in steal time.
KVM: x86: Document host suspend being included in steal time.

 Documentation/virt/kvm/x86/msr.rst |  6 ++++--
 arch/x86/include/asm/kvm_host.h    |  1 +
 arch/x86/kvm/x86.c                 | 11 ++++++++++-
 include/linux/kvm_host.h           |  2 ++
 virt/kvm/kvm_main.c                | 13 +++++++++++++
 5 files changed, 30 insertions(+), 3 deletions(-)
Signature · 1 line
2.46.0.184.g6999bdac58-goog
#1 lore raw

PATCH v2 1/3 KVM: Introduce kvm_total_suspend_ns().

It returns the cumulative nanoseconds that the host has been suspended. It is intended to be used for reporting host suspend time to the guest.

---
 include/linux/kvm_host.h |  2 ++
 virt/kvm/kvm_main.c      | 13 +++++++++++++
 2 files changed, 15 insertions(+)
include/linux/kvm_host.h +2 -0
index b23c6d48392f7c..8fec37b372d8c0 100644
@@ -2494,4 +2494,6 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
 				    struct kvm_pre_fault_memory *range);
 #endif
 
+u64 kvm_total_suspend_ns(void);
+
 #endif
virt/kvm/kvm_main.c +13 -0
index cb2b78e92910fb..2235933d9247bc 100644
@@ -5720,6 +5720,15 @@ static void kvm_shutdown(void)
 	on_each_cpu(hardware_disable_nolock, NULL, 1);
 }
 
+static u64 last_suspend;
+static u64 total_suspend_ns;
+
+u64
+kvm_total_suspend_ns(void)
+{
+	return total_suspend_ns;
+}
+
 static int kvm_suspend(void)
 {
 	/*
@@ -5735,6 +5744,8 @@ static int kvm_suspend(void)
 
 	if (kvm_usage_count)
 		hardware_disable_nolock(NULL);
+
+	last_suspend = ktime_get_boottime_ns();
 	return 0;
 }
 
@@ -5745,6 +5756,8 @@ static void kvm_resume(void)
 
 	if (kvm_usage_count)
 		WARN_ON_ONCE(__hardware_enable_nolock());
+
+	total_suspend_ns += ktime_get_boottime_ns() - last_suspend;
 }
 
 static struct syscore_ops kvm_syscore_ops = {
Signature · 1 line
2.46.0.184.g6999bdac58-goog
Reviewed-by on v2 1/3
#2 lore raw

Re: [PATCH v2 1/3] KVM: Introduce kvm_total_suspend_ns().

Quoted 4 lines · Suleiman Souhlal

It returns the cumulative nanoseconds that the host has been suspended. It is intended to be used for reporting host suspend time to the guest.

one nit below

Quoted 21 lines

---
include/linux/kvm_host.h | 2 ++
virt/kvm/kvm_main.c | 13 +++++++++++++

2 files changed, 15 insertions(+)
include/linux/kvm_host.h +2 -0
index b23c6d48392f7c..8fec37b372d8c0 100644
@@ -2494,4 +2494,6 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
 				    struct kvm_pre_fault_memory *range);
 #endif
 
+u64 kvm_total_suspend_ns(void);
+
 #endif
virt/kvm/kvm_main.c +0 -0
index cb2b78e92910fb..2235933d9247bc 100644

virt/kvm/kvm_main.c static void kvm_shutdown(void) · line 5727

Quoted 8 lines
virt/kvm/kvm_main.c +5 -0
@@ -5720,6 +5720,15 @@ static void kvm_shutdown(void)
 	on_each_cpu(hardware_disable_nolock, NULL, 1);
 }
 
+static u64 last_suspend;
+static u64 total_suspend_ns;
+
+u64
+kvm_total_suspend_ns(void)

nit: don't wrap before the function name.

Quoted 28 lines
virt/kvm/kvm_main.c +8 -0
+{
+	return total_suspend_ns;
+}
+
 static int kvm_suspend(void)
 {
 	/*
@@ -5735,6 +5744,8 @@ static int kvm_suspend(void)
 
 	if (kvm_usage_count)
 		hardware_disable_nolock(NULL);
+
+	last_suspend = ktime_get_boottime_ns();
 	return 0;
 }
 
@@ -5745,6 +5756,8 @@ static void kvm_resume(void)
 
 	if (kvm_usage_count)
 		WARN_ON_ONCE(__hardware_enable_nolock());
+
+	total_suspend_ns += ktime_get_boottime_ns() - last_suspend;
 }
 
 static struct syscore_ops kvm_syscore_ops = {
Signature · 1 line
2.46.0.184.g6999bdac58-goog
#3 lore raw

Re: [PATCH v2 1/3] KVM: Introduce kvm_total_suspend_ns().

Quoted 35 lines · Chao Gao
Quoted 4 lines · Suleiman Souhlal

It returns the cumulative nanoseconds that the host has been suspended. It is intended to be used for reporting host suspend time to the guest.

one nit below

Quoted 24 lines

---
include/linux/kvm_host.h | 2 ++
virt/kvm/kvm_main.c | 13 +++++++++++++

2 files changed, 15 insertions(+)
include/linux/kvm_host.h +2 -0
index b23c6d48392f7c..8fec37b372d8c0 100644
@@ -2494,4 +2494,6 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
                                  struct kvm_pre_fault_memory *range);
 #endif
 
+u64 kvm_total_suspend_ns(void);
+
 #endif
virt/kvm/kvm_main.c +0 -0
index cb2b78e92910fb..2235933d9247bc 100644
@@ -5720,6 +5720,15 @@ static void kvm_shutdown(void)
      on_each_cpu(hardware_disable_nolock, NULL, 1);
 }
 
Quoted 7 lines

virt/kvm/kvm_main.c static void kvm_shutdown(void) · line 5727

Quoted 5 lines
virt/kvm/kvm_main.c +5 -0
+static u64 last_suspend;
+static u64 total_suspend_ns;
+
+u64
+kvm_total_suspend_ns(void)

nit: don't wrap before the function name.

Sorry, I completely missed that, even after Sean told me. Force of habit (FreeBSD style(9) says you have to do it).

If I send another version I will fix that.

-- Suleiman

#4 lore raw

PATCH v2 2/3 KVM: x86: Include host suspended time in steal time.

In reply to Suleiman Souhlal

When the host resumes from a suspend, the guest thinks any task that was running during the suspend ran for a long time, even though the effective run time was much shorter, which can end up having negative effects with scheduling. This can be particularly noticeable if the guest task was RT, as it can end up getting throttled for a long time.

To mitigate this issue, we include the time that the host was suspended in steal time, which lets the guest subtract the duration from the tasks' runtime.

Note that the case of a suspend happening during a VM migration might not be accounted.

---
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/x86.c              | 11 ++++++++++-
 2 files changed, 11 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 4a68cb3eba78f8..728798decb6d12 100644
@@ -898,6 +898,7 @@ struct kvm_vcpu_arch {
 		u8 preempted;
 		u64 msr_val;
 		u64 last_steal;
+		u64 last_suspend_ns;
 		struct gfn_to_hva_cache cache;
 	} st;
 
arch/x86/kvm/x86.c +10 -1
index 70219e4069874a..104f3d318026fa 100644
@@ -3654,7 +3654,7 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
 	struct kvm_steal_time __user *st;
 	struct kvm_memslots *slots;
 	gpa_t gpa = vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS;
-	u64 steal;
+	u64 steal, suspend_ns;
 	u32 version;
 
 	if (kvm_xen_msr_enabled(vcpu->kvm)) {
@@ -3735,6 +3735,14 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
 	steal += current->sched_info.run_delay -
 		vcpu->arch.st.last_steal;
 	vcpu->arch.st.last_steal = current->sched_info.run_delay;
+	/*
+	 * Include the time that the host was suspended in steal time.
+	 * Note that the case of a suspend happening during a VM migration
+	 * might not be accounted.
+	 */
+	suspend_ns = kvm_total_suspend_ns();
+	steal += suspend_ns - vcpu->arch.st.last_suspend_ns;
+	vcpu->arch.st.last_suspend_ns = suspend_ns;
 	unsafe_put_user(steal, &st->steal, out);
 
 	version += 1;
@@ -12280,6 +12288,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 
 	vcpu->arch.arch_capabilities = kvm_get_arch_capabilities();
 	vcpu->arch.msr_platform_info = MSR_PLATFORM_INFO_CPUID_FAULT;
+	vcpu->arch.st.last_suspend_ns = kvm_total_suspend_ns();
 	kvm_xen_init_vcpu(vcpu);
 	vcpu_load(vcpu);
 	kvm_set_tsc_khz(vcpu, vcpu->kvm->arch.default_tsc_khz);
Signature · 1 line
2.46.0.184.g6999bdac58-goog
#5 lore raw
Quoted 47 lines · Suleiman Souhlal

When the host resumes from a suspend, the guest thinks any task that was running during the suspend ran for a long time, even though the effective run time was much shorter, which can end up having negative effects with scheduling. This can be particularly noticeable if the guest task was RT, as it can end up getting throttled for a long time.

To mitigate this issue, we include the time that the host was suspended in steal time, which lets the guest subtract the duration from the tasks' runtime.

Note that the case of a suspend happening during a VM migration might not be accounted.

---
arch/x86/include/asm/kvm_host.h | 1 +
arch/x86/kvm/x86.c | 11 ++++++++++-

2 files changed, 11 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 4a68cb3eba78f8..728798decb6d12 100644
@@ -898,6 +898,7 @@ struct kvm_vcpu_arch {
 		u8 preempted;
 		u64 msr_val;
 		u64 last_steal;
+		u64 last_suspend_ns;
 		struct gfn_to_hva_cache cache;
 	} st;
 
arch/x86/kvm/x86.c +1 -1
index 70219e4069874a..104f3d318026fa 100644
@@ -3654,7 +3654,7 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
 	struct kvm_steal_time __user *st;
 	struct kvm_memslots *slots;
 	gpa_t gpa = vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS;
-	u64 steal;
+	u64 steal, suspend_ns;
 	u32 version;
 
 	if (kvm_xen_msr_enabled(vcpu->kvm)) {
@@ -3735,6 +3735,14 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
 	steal += current->sched_info.run_delay -

arch/x86/kvm/x86.c static void record_steal_time(struct kvm_vcpu *vcpu) · line 3745

Quoted 10 lines
arch/x86/kvm/x86.c +8 -0
 		vcpu->arch.st.last_steal;
 	vcpu->arch.st.last_steal = current->sched_info.run_delay;
+	/*
+	 * Include the time that the host was suspended in steal time.
+	 * Note that the case of a suspend happening during a VM migration
+	 * might not be accounted.
+	 */
+	suspend_ns = kvm_total_suspend_ns();
+	steal += suspend_ns - vcpu->arch.st.last_suspend_ns;
+	vcpu->arch.st.last_suspend_ns = suspend_ns;

The document in patch 3 states:

Time during which the vcpu is idle, will not be reported as steal time

I'm wondering if all host suspend time should be reported as steal time, or if the suspend time during a vCPU halt should be excluded.

arch/x86/kvm/x86.c int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) · line 12291

Quoted 8 lines
arch/x86/kvm/x86.c +1 -0
 	unsafe_put_user(steal, &st->steal, out);
 
 	version += 1;
@@ -12280,6 +12288,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 
 	vcpu->arch.arch_capabilities = kvm_get_arch_capabilities();
 	vcpu->arch.msr_platform_info = MSR_PLATFORM_INFO_CPUID_FAULT;
+	vcpu->arch.st.last_suspend_ns = kvm_total_suspend_ns();

is this necessary? I doubt this because KVM doesn't capture current->sched_info.run_delay here.

Quoted 6 lines
arch/x86/kvm/x86.c +0 -0
 	kvm_xen_init_vcpu(vcpu);
 	vcpu_load(vcpu);
 	kvm_set_tsc_khz(vcpu, vcpu->kvm->arch.default_tsc_khz);
Signature · 1 line
2.46.0.184.g6999bdac58-goog
#6 lore raw
Quoted 62 lines · Chao Gao
Quoted 47 lines · Suleiman Souhlal

When the host resumes from a suspend, the guest thinks any task that was running during the suspend ran for a long time, even though the effective run time was much shorter, which can end up having negative effects with scheduling. This can be particularly noticeable if the guest task was RT, as it can end up getting throttled for a long time.

To mitigate this issue, we include the time that the host was suspended in steal time, which lets the guest subtract the duration from the tasks' runtime.

Note that the case of a suspend happening during a VM migration might not be accounted.

---
arch/x86/include/asm/kvm_host.h | 1 +
arch/x86/kvm/x86.c | 11 ++++++++++-

2 files changed, 11 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 4a68cb3eba78f8..728798decb6d12 100644
@@ -898,6 +898,7 @@ struct kvm_vcpu_arch {
              u8 preempted;
              u64 msr_val;
              u64 last_steal;
+              u64 last_suspend_ns;
              struct gfn_to_hva_cache cache;
      } st;
 
arch/x86/kvm/x86.c +1 -1
index 70219e4069874a..104f3d318026fa 100644
@@ -3654,7 +3654,7 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
      struct kvm_steal_time __user *st;
      struct kvm_memslots *slots;
      gpa_t gpa = vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS;
-      u64 steal;
+      u64 steal, suspend_ns;
      u32 version;
 
      if (kvm_xen_msr_enabled(vcpu->kvm)) {
@@ -3735,6 +3735,14 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
      steal += current->sched_info.run_delay -

arch/x86/kvm/x86.c static void record_steal_time(struct kvm_vcpu *vcpu) · line 3745

Quoted 10 lines
arch/x86/kvm/x86.c +8 -0
              vcpu->arch.st.last_steal;
      vcpu->arch.st.last_steal = current->sched_info.run_delay;
+      /*
+       * Include the time that the host was suspended in steal time.
+       * Note that the case of a suspend happening during a VM migration
+       * might not be accounted.
+       */
+      suspend_ns = kvm_total_suspend_ns();
+      steal += suspend_ns - vcpu->arch.st.last_suspend_ns;
+      vcpu->arch.st.last_suspend_ns = suspend_ns;

The document in patch 3 states:

Quoted 4 lines

Time during which the vcpu is idle, will not be reported as steal time

I'm wondering if all host suspend time should be reported as steal time, or if the suspend time during a vCPU halt should be excluded.

I think the statement about idle time not being reported as steal isn't completely accurate, so I'm not sure if it's worth the extra complexity.

Quoted 6 lines
Quoted 6 lines
arch/x86/kvm/x86.c +0 -0
      unsafe_put_user(steal, &st->steal, out);
 
      version += 1;
@@ -12280,6 +12288,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 
Quoted 6 lines

arch/x86/kvm/x86.c int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) · line 12291

Quoted 3 lines
arch/x86/kvm/x86.c +1 -0
      vcpu->arch.arch_capabilities = kvm_get_arch_capabilities();
      vcpu->arch.msr_platform_info = MSR_PLATFORM_INFO_CPUID_FAULT;
+      vcpu->arch.st.last_suspend_ns = kvm_total_suspend_ns();

is this necessary? I doubt this because KVM doesn't capture current->sched_info.run_delay here.

Isn't run_delay being captured by the scheduler at all time?

We need to initialize last_suspend_ns otherwise the first call to record_steal_time() for a VCPU would report a wrong value if the VCPU is started after the host has already had a suspend.

Thanks,
-- Suleiman

#7 lore raw
Quoted 77 lines · Suleiman Souhlal
Quoted 62 lines · Chao Gao
Quoted 47 lines · Suleiman Souhlal

When the host resumes from a suspend, the guest thinks any task that was running during the suspend ran for a long time, even though the effective run time was much shorter, which can end up having negative effects with scheduling. This can be particularly noticeable if the guest task was RT, as it can end up getting throttled for a long time.

To mitigate this issue, we include the time that the host was suspended in steal time, which lets the guest subtract the duration from the tasks' runtime.

Note that the case of a suspend happening during a VM migration might not be accounted.

---
arch/x86/include/asm/kvm_host.h | 1 +
arch/x86/kvm/x86.c | 11 ++++++++++-

2 files changed, 11 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 4a68cb3eba78f8..728798decb6d12 100644
@@ -898,6 +898,7 @@ struct kvm_vcpu_arch {
              u8 preempted;
              u64 msr_val;
              u64 last_steal;
+              u64 last_suspend_ns;
              struct gfn_to_hva_cache cache;
      } st;
 
arch/x86/kvm/x86.c +1 -1
index 70219e4069874a..104f3d318026fa 100644
@@ -3654,7 +3654,7 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
      struct kvm_steal_time __user *st;
      struct kvm_memslots *slots;
      gpa_t gpa = vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS;
-      u64 steal;
+      u64 steal, suspend_ns;
      u32 version;
 
      if (kvm_xen_msr_enabled(vcpu->kvm)) {
@@ -3735,6 +3735,14 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
      steal += current->sched_info.run_delay -

arch/x86/kvm/x86.c static void record_steal_time(struct kvm_vcpu *vcpu) · line 3745

Quoted 10 lines
arch/x86/kvm/x86.c +8 -0
              vcpu->arch.st.last_steal;
      vcpu->arch.st.last_steal = current->sched_info.run_delay;
+      /*
+       * Include the time that the host was suspended in steal time.
+       * Note that the case of a suspend happening during a VM migration
+       * might not be accounted.
+       */
+      suspend_ns = kvm_total_suspend_ns();
+      steal += suspend_ns - vcpu->arch.st.last_suspend_ns;
+      vcpu->arch.st.last_suspend_ns = suspend_ns;

The document in patch 3 states:

Quoted 4 lines

Time during which the vcpu is idle, will not be reported as steal time

I'm wondering if all host suspend time should be reported as steal time, or if the suspend time during a vCPU halt should be excluded.

I think the statement about idle time not being reported as steal isn't completely accurate, so I'm not sure if it's worth the extra complexity.

Quoted 6 lines
Quoted 6 lines
arch/x86/kvm/x86.c +0 -0
      unsafe_put_user(steal, &st->steal, out);
 
      version += 1;
@@ -12280,6 +12288,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 
Quoted 8 lines
Quoted 6 lines

arch/x86/kvm/x86.c int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) · line 12291

Quoted 3 lines
arch/x86/kvm/x86.c +1 -0
      vcpu->arch.arch_capabilities = kvm_get_arch_capabilities();
      vcpu->arch.msr_platform_info = MSR_PLATFORM_INFO_CPUID_FAULT;
+      vcpu->arch.st.last_suspend_ns = kvm_total_suspend_ns();

is this necessary? I doubt this because KVM doesn't capture current->sched_info.run_delay here.

Isn't run_delay being captured by the scheduler at all time?

I meant KVM doesn't do:

	vcpu->arch.st.last_steal = current->sched_info.run_delay;

at vCPU creation time.

Quoted 4 lines

We need to initialize last_suspend_ns otherwise the first call to record_steal_time() for a VCPU would report a wrong value if the VCPU is started after the host has already had a suspend.

But initializing last_suspend_ns here doesn't guarantee KVM won't report a "wrong" value because a suspend can happen after vCPU creation and before its first VM-enter.

#8 lore raw
Quoted 86 lines · Chao Gao
Quoted 84 lines · Suleiman Souhlal
Quoted 62 lines · Chao Gao
Quoted 47 lines · Suleiman Souhlal

When the host resumes from a suspend, the guest thinks any task that was running during the suspend ran for a long time, even though the effective run time was much shorter, which can end up having negative effects with scheduling. This can be particularly noticeable if the guest task was RT, as it can end up getting throttled for a long time.

To mitigate this issue, we include the time that the host was suspended in steal time, which lets the guest subtract the duration from the tasks' runtime.

Note that the case of a suspend happening during a VM migration might not be accounted.

---
arch/x86/include/asm/kvm_host.h | 1 +
arch/x86/kvm/x86.c | 11 ++++++++++-

2 files changed, 11 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 4a68cb3eba78f8..728798decb6d12 100644
@@ -898,6 +898,7 @@ struct kvm_vcpu_arch {
              u8 preempted;
              u64 msr_val;
              u64 last_steal;
+              u64 last_suspend_ns;
              struct gfn_to_hva_cache cache;
      } st;
 
arch/x86/kvm/x86.c +1 -1
index 70219e4069874a..104f3d318026fa 100644
@@ -3654,7 +3654,7 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
      struct kvm_steal_time __user *st;
      struct kvm_memslots *slots;
      gpa_t gpa = vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS;
-      u64 steal;
+      u64 steal, suspend_ns;
      u32 version;
 
      if (kvm_xen_msr_enabled(vcpu->kvm)) {
@@ -3735,6 +3735,14 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
      steal += current->sched_info.run_delay -

arch/x86/kvm/x86.c static void record_steal_time(struct kvm_vcpu *vcpu) · line 3745

Quoted 10 lines
arch/x86/kvm/x86.c +8 -0
              vcpu->arch.st.last_steal;
      vcpu->arch.st.last_steal = current->sched_info.run_delay;
+      /*
+       * Include the time that the host was suspended in steal time.
+       * Note that the case of a suspend happening during a VM migration
+       * might not be accounted.
+       */
+      suspend_ns = kvm_total_suspend_ns();
+      steal += suspend_ns - vcpu->arch.st.last_suspend_ns;
+      vcpu->arch.st.last_suspend_ns = suspend_ns;

The document in patch 3 states:

Quoted 4 lines

Time during which the vcpu is idle, will not be reported as steal time

I'm wondering if all host suspend time should be reported as steal time, or if the suspend time during a vCPU halt should be excluded.

I think the statement about idle time not being reported as steal isn't completely accurate, so I'm not sure if it's worth the extra complexity.

Quoted 12 lines

arch/x86/kvm/x86.c int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) · line 12291

Quoted 9 lines
arch/x86/kvm/x86.c +1 -0
      unsafe_put_user(steal, &st->steal, out);
 
      version += 1;
@@ -12280,6 +12288,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 
      vcpu->arch.arch_capabilities = kvm_get_arch_capabilities();
      vcpu->arch.msr_platform_info = MSR_PLATFORM_INFO_CPUID_FAULT;
+      vcpu->arch.st.last_suspend_ns = kvm_total_suspend_ns();

is this necessary? I doubt this because KVM doesn't capture current->sched_info.run_delay here.

Quoted 7 lines
Quoted 1 line

Isn't run_delay being captured by the scheduler at all time?

I meant KVM doesn't do:

        vcpu->arch.st.last_steal = current->sched_info.run_delay;

at vCPU creation time.

I think for run_delay it's different because run_delay is a time difference. It's something that gets added to steal, not relative to the previous steal value.
From what I can tell, it's correct for last_steal to be initialized to 0.

Quoted 9 lines
Quoted 5 lines

We need to initialize last_suspend_ns otherwise the first call to record_steal_time() for a VCPU would report a wrong value if the VCPU is started after the host has already had a suspend.

But initializing last_suspend_ns here doesn't guarantee KVM won't report a "wrong" value because a suspend can happen after vCPU creation and before its first VM-enter.

I see what you're saying.
I'm not sure how much this matters in practice.

-- Suleiman

#9 lore raw
In reply to Suleiman Souhlal
Quoted 5 lines · Suleiman Souhlal
arch/x86/kvm/x86.c +1 -0
@@ -3735,6 +3735,14 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
        steal += current->sched_info.run_delay -
                vcpu->arch.st.last_steal;
        vcpu->arch.st.last_steal = current->sched_info.run_delay;
+       /*

arch/x86/kvm/x86.c static void record_steal_time(struct kvm_vcpu *vcpu) · line 3748

Quoted 10 lines
arch/x86/kvm/x86.c +7 -0
+        * Include the time that the host was suspended in steal time.
+        * Note that the case of a suspend happening during a VM migration
+        * might not be accounted.
+        */
+       suspend_ns = kvm_total_suspend_ns();
+       steal += suspend_ns - vcpu->arch.st.last_suspend_ns;
+       vcpu->arch.st.last_suspend_ns = suspend_ns;
        unsafe_put_user(steal, &st->steal, out);
 
        version += 1;

There is an issue here: We are calling a function under UACCESS, which raises an objtool warning.
I'll be sending a v3 with that addressed (and the function return value wrapping in patch 1/3).

-- Suleiman

#10 lore raw

PATCH v2 3/3 KVM: x86: Document host suspend being included in steal time.

In reply to Suleiman Souhlal

Steal time now includes the time that the host was suspended.

---
 Documentation/virt/kvm/x86/msr.rst | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
Documentation/virt/kvm/x86/msr.rst +4 -2
index 3aecf2a70e7b43..81c17c2200ca2f 100644
@@ -294,8 +294,10 @@ data:
 
 	steal:
 		the amount of time in which this vCPU did not run, in
-		nanoseconds. Time during which the vcpu is idle, will not be
-		reported as steal time.
+		nanoseconds. This includes the time during which the host is
+		suspended. However, the case where the host suspends during a
+		VM migration might not be correctly accounted. Time during
+		which the vcpu is idle, will not be reported as steal time.
 
 	preempted:
 		indicate the vCPU who owns this struct is running or
Signature · 1 line
2.46.0.184.g6999bdac58-goog
#11 lore raw
In reply to Suleiman Souhlal

+David W for his input.

Quoted 36 lines · Suleiman Souhlal

This series makes it so that the time that the host is suspended is included in guests' steal time.

When the host resumes from a suspend, the guest thinks any task that was running during the suspend ran for a long time, even though the effective run time was much shorter, which can end up having negative effects with scheduling. This can be particularly noticeable if the guest task was RT, as it can end up getting throttled for a long time.

To mitigate this issue, we include the time that the host was suspended in steal time, which lets the guest can subtract the duration from the tasks' runtime.

(v1 was at https://lore.kernel.org/kvm/[email protected]/)

v1 -> v2:
- Accumulate suspend time at machine-independent kvm layer and track per-VCPU instead of per-VM.
- Document changes.

Suleiman Souhlal (3):
KVM: Introduce kvm_total_suspend_ns().
KVM: x86: Include host suspended time in steal time.
KVM: x86: Document host suspend being included in steal time.

 Documentation/virt/kvm/x86/msr.rst |  6 ++++--
 arch/x86/include/asm/kvm_host.h    |  1 +
 arch/x86/kvm/x86.c                 | 11 ++++++++++-
 include/linux/kvm_host.h           |  2 ++
 virt/kvm/kvm_main.c                | 13 +++++++++++++
 5 files changed, 30 insertions(+), 3 deletions(-)
Signature · 1 line
2.46.0.184.g6999bdac58-goog