Alter steal time reporting in KVM

Add a consigned field. This field will hold the time lost due to capping or overcommit. The rest of the time will still show up in the steal-time field.

Signed-off-by Michael Wolf <[email protected]>

---
 arch/x86/include/asm/paravirt.h       |    4 ++--
 arch/x86/include/asm/paravirt_types.h |    2 +-
 arch/x86/kernel/kvm.c                 |    7 ++-----
 kernel/sched/core.c                   |   10 +++++++++-
 kernel/sched/cputime.c                |    2 +-
 5 files changed, 15 insertions(+), 10 deletions(-)

arch/x86/include/asm/paravirt.h +0 -0

index a0facf3..a5f9f30 100644

arch/x86/include/asm/paravirt.h +1 -1

 {
-	return PVOP_CALL1(u64, pv_time_ops.steal_clock, cpu);
+	PVOP_VCALL2(pv_time_ops.steal_clock, cpu, steal);
 }
 
 static inline unsigned long long paravirt_read_pmc(int counter)

arch/x86/include/asm/paravirt_types.h +0 -0

index 142236e..5d4fc8b 100644

Quoted 17 lines · Michael Wolf

Add a consigned field. This field will hold the time lost due to capping or overcommit. The rest of the time will still show up in the steal-time field.

Signed-off-by Michael Wolf <[email protected]>

---
  arch/x86/include/asm/paravirt.h       |    4 ++--
  arch/x86/include/asm/paravirt_types.h |    2 +-
  arch/x86/kernel/kvm.c                 |    7 ++-----
  kernel/sched/core.c                   |   10 +++++++++-
  kernel/sched/cputime.c                |    2 +-
  5 files changed, 15 insertions(+), 10 deletions(-)

arch/x86/include/asm/paravirt.h +0 -0

index a0facf3..a5f9f30 100644

arch/x86/include/asm/paravirt.h struct static_key; · line 199

Quoted 5 lines

arch/x86/include/asm/paravirt.h +1 -1

@@ -196,9 +196,9 @@ struct static_key;
  extern struct static_key paravirt_steal_enabled;
  extern struct static_key paravirt_steal_rq_enabled;
 
-static inline u64 paravirt_steal_clock(int cpu)
+static inline u64 paravirt_steal_clock(int cpu, u64 *steal)

So its u64 here.

Quoted 12 lines

arch/x86/include/asm/paravirt.h +1 -1

  {
-	return PVOP_CALL1(u64, pv_time_ops.steal_clock, cpu);
+	PVOP_VCALL2(pv_time_ops.steal_clock, cpu, steal);
  }
 
  static inline unsigned long long paravirt_read_pmc(int counter)

arch/x86/include/asm/paravirt_types.h +0 -0

index 142236e..5d4fc8b 100644
@@ -95,7 +95,7 @@ struct pv_lazy_ops {
 

arch/x86/include/asm/paravirt_types.h struct pv_lazy_ops { · line 98

Quoted 4 lines

arch/x86/include/asm/paravirt_types.h +1 -1

  struct pv_time_ops {
  	unsigned long long (*sched_clock)(void);
-	unsigned long long (*steal_clock)(int cpu);
+	void (*steal_clock)(int cpu, unsigned long long *steal);

But not u64 here? Any particular reason?

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user. To ease the confusion this patch set adds the idea of consigned (expected steal) time. The host will separate the consigned time from the steal time. The consignment limit passed to the host will be the amount of steal time expected within a fixed period of time. Any other steal time accruing during that period will show as the traditional steal time.

---

Michael Wolf (5):
Alter the amount of steal time reported by the guest. Expand the steal time msr to also contain the consigned time. Add the code to send the consigned time from the host to the guest Add a timer to allow the separation of consigned from steal time. Add an ioctl to communicate the consign limit to the host.

 arch/x86/include/asm/kvm_host.h       |   11 +++++++
 arch/x86/include/asm/kvm_para.h       |    3 +-
 arch/x86/include/asm/paravirt.h       |    4 +--
 arch/x86/include/asm/paravirt_types.h |    2 +
 arch/x86/kernel/kvm.c                 |    8 ++---
 arch/x86/kernel/paravirt.c            |    4 +--
 arch/x86/kvm/x86.c                    |   50 ++++++++++++++++++++++++++++++++-
 fs/proc/stat.c                        |    9 +++++-
 include/linux/kernel_stat.h           |    2 +
 include/linux/kvm_host.h              |    2 +
 include/uapi/linux/kvm.h              |    2 +
 kernel/sched/core.c                   |   10 ++++++-
 kernel/sched/cputime.c                |   21 +++++++++++++-
 kernel/sched/sched.h                  |    2 +
 virt/kvm/kvm_main.c                   |    7 +++++
 15 files changed, 120 insertions(+), 17 deletions(-)

Hi,

Quoted 9 lines · Michael Wolf

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user. To ease the confusion this patch set adds the idea of consigned (expected steal) time. The host will separate the consigned time from the steal time. The consignment limit passed to the host will be the amount of steal time expected within a fixed period of time. Any other steal time accruing during that period will show as the traditional steal time.

If you submit this again, please include a version number in your series.

It would also be helpful to include a small changelog about what changed between last version and this version, so we could focus on that.

As for the rest, I answered your previous two submissions saying I don't agree with the concept. If you hadn't changed anything, resending it won't change my mind.

I could of course, be mistaken or misguided. But I had also not seen any wave of support in favor of this previously, so basically I have no new data to make me believe I should see it any differently.

Let's try this again:

* Rik asked you in your last submission how does ppc handle this. You said, and I quote: "In the case of lpar on POWER systems they simply report steal time and do not alter it in any way.
They do however report how much processor is assigned to the partition and that information is in /proc/ppc64/lparcfg."

Now, that is a *way* more sensible thing to do. Much more. "Confusing users" is something extremely subjective. This is specially true about concepts that are know for quite some time, like steal time. If you out of a sudden change the meaning of this, it is sure to confuse a lot more users than it would clarify.

Quoted 14 lines

---

Michael Wolf (5):
Alter the amount of steal time reported by the guest. Expand the steal time msr to also contain the consigned time. Add the code to send the consigned time from the host to the guest Add a timer to allow the separation of consigned from steal time. Add an ioctl to communicate the consign limit to the host.

Quoted 17 lines

  arch/x86/include/asm/kvm_host.h       |   11 +++++++
  arch/x86/include/asm/kvm_para.h       |    3 +-
  arch/x86/include/asm/paravirt.h       |    4 +--
  arch/x86/include/asm/paravirt_types.h |    2 +
  arch/x86/kernel/kvm.c                 |    8 ++---
  arch/x86/kernel/paravirt.c            |    4 +--
  arch/x86/kvm/x86.c                    |   50 ++++++++++++++++++++++++++++++++-
  fs/proc/stat.c                        |    9 +++++-
  include/linux/kernel_stat.h           |    2 +
  include/linux/kvm_host.h              |    2 +
  include/uapi/linux/kvm.h              |    2 +
  kernel/sched/core.c                   |   10 ++++++-
  kernel/sched/cputime.c                |   21 +++++++++++++-
  kernel/sched/sched.h                  |    2 +
  virt/kvm/kvm_main.c                   |    7 +++++
  15 files changed, 120 insertions(+), 17 deletions(-)

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 14 lines · Glauber Costa

Hi,

Quoted 11 lines · Michael Wolf

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user. To ease the confusion this patch set adds the idea of consigned (expected steal) time. The host will separate
the consigned time from the steal time. The consignment limit passed to the
host will be the amount of steal time expected within a fixed period of time. Any other steal time accruing during that period will show as the traditional steal time.

Quoted 1 line

If you submit this again, please include a version number in your series.

Will do. The patchset was sent twice yesterday by mistake. Got an error the first time and didn't
think the patches went out. This has been corrected.

Quoted 3 lines

It would also be helpful to include a small changelog about what changed between last version and this version, so we could focus on that.

yes, will do that. When I took the RFC off the patches I was looking at it as a new patchset which was
a mistake. I will make sure to add a changelog when I submit again.

Quoted 9 lines

As for the rest, I answered your previous two submissions saying I don't agree with the concept. If you hadn't changed anything, resending it won't change my mind.

I could of course, be mistaken or misguided. But I had also not seen any wave of support in favor of this previously, so basically I have no new data to make me believe I should see it any differently.

Quoted 7 lines

Let's try this again:

* Rik asked you in your last submission how does ppc handle this. You said, and I quote: "In the case of lpar on POWER systems they simply report steal time and do not alter it in any way.
They do however report how much processor is assigned to the partition and that information is in /proc/ppc64/lparcfg."

Yes, but we still get questions from users asking what is steal time? why am I seeing this?

Quoted 6 lines

Now, that is a *way* more sensible thing to do. Much more. "Confusing users" is something extremely subjective. This is specially true about concepts that are know for quite some time, like steal time. If you out of a sudden change the meaning of this, it is sure to confuse a lot more users than it would clarify.

Something like this could certainly be done. But when I was submitting the patch set as
an RFC then qemu was passing a cpu percentage that would be used by the guest kernel
to adjust the steal time. This percentage was being stored on the guest as a sysctl value.
Avi stated he didn't like that kind of coupling, and that the value could get out of sync. Anthony stated "The guest shouldn't need to know it's entitlement. Or at least, it's up to a management tool to report that in a way that's meaningful for the guest."

So perhaps I misunderstood what they were suggesting, but I took it to mean that they did not
want the guest to know what the entitlement was. That the host should take care of it and just
report the already adjusted data to the guest. So in this version of the code the host would use a set
period for a timer and be passed essentially a number of ticks of expected steal time. The host
would then use the timer to break out the steal time into consigned and steal buckets which would be
reported to the guest.

Both the consigned and the steal would be reported via /proc/stat. So anyone needing to see total
time away could add the two fields together. The user, however, when using tools like top or vmstat
would see the usage based on what the guest is entitled to.

Do you have suggestions for how I can build consensus around one of the two approaches?

Quoted 75 lines · Michael Wolf

Quoted 14 lines · Glauber Costa

Hi,

Quoted 11 lines · Michael Wolf

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user. To ease the confusion this patch set adds the idea of consigned (expected steal) time. The host will separate
the consigned time from the steal time. The consignment limit passed to the
host will be the amount of steal time expected within a fixed period of time. Any other steal time accruing during that period will show as the traditional steal time.

Quoted 1 line

If you submit this again, please include a version number in your series.

Will do. The patchset was sent twice yesterday by mistake. Got an error the first time and didn't
think the patches went out. This has been corrected.

Quoted 2 lines

It would also be helpful to include a small changelog about what changed between last version and this version, so we could focus on that.

yes, will do that. When I took the RFC off the patches I was looking at it as a new patchset which was
a mistake. I will make sure to add a changelog when I submit again.

Quoted 8 lines

As for the rest, I answered your previous two submissions saying I don't agree with the concept. If you hadn't changed anything, resending it won't change my mind.

I could of course, be mistaken or misguided. But I had also not seen any wave of support in favor of this previously, so basically I have no new data to make me believe I should see it any differently.

Quoted 7 lines

Let's try this again:

* Rik asked you in your last submission how does ppc handle this. You said, and I quote: "In the case of lpar on POWER systems they simply report steal time and do not alter it in any way.
They do however report how much processor is assigned to the partition and that information is in /proc/ppc64/lparcfg."

Yes, but we still get questions from users asking what is steal time? why am I seeing this?

Quoted 5 lines

Now, that is a *way* more sensible thing to do. Much more. "Confusing users" is something extremely subjective. This is specially true about concepts that are know for quite some time, like steal time. If you out of a sudden change the meaning of this, it is sure to confuse a lot more users than it would clarify.

Something like this could certainly be done. But when I was submitting the patch set as
an RFC then qemu was passing a cpu percentage that would be used by the guest kernel
to adjust the steal time. This percentage was being stored on the guest as a sysctl value.
Avi stated he didn't like that kind of coupling, and that the value could get out of sync. Anthony stated "The guest shouldn't need to know it's entitlement. Or at least, it's up to a management tool to report that in a way that's meaningful for the guest."

So perhaps I misunderstood what they were suggesting, but I took it to mean that they did not
want the guest to know what the entitlement was. That the host should take care of it and just
report the already adjusted data to the guest. So in this version of the code the host would use a set
period for a timer and be passed essentially a number of ticks of expected steal time. The host
would then use the timer to break out the steal time into consigned and steal buckets which would be
reported to the guest.

Both the consigned and the steal would be reported via /proc/stat. So anyone needing to see total
time away could add the two fields together. The user, however, when using tools like top or vmstat
would see the usage based on what the guest is entitled to.

Quoted 3 lines

Do you have suggestions for how I can build consensus around one of the two approaches?

Before I answer this, can you please detail which mechanism are you using to enforce the entitlement? Is it the cgroup cpu controller, or something else?

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi,

Quoted 9 lines · Michael Wolf

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user. To ease the confusion this patch set adds the idea of consigned (expected steal) time. The host will separate the consigned time from the steal time. The consignment limit passed to the host will be the amount of steal time expected within a fixed period of time. Any other steal time accruing during that period will show as the traditional steal time.

If you submit this again, please include a version number in your series.

It would also be helpful to include a small changelog about what changed between last version and this version, so we could focus on that.

As for the rest, I answered your previous two submissions saying I don't agree with the concept. If you hadn't changed anything, resending it won't change my mind.

I could of course, be mistaken or misguided. But I had also not seen any wave of support in favor of this previously, so basically I have no new data to make me believe I should see it any differently.

Let's try this again:

* Rik asked you in your last submission how does ppc handle this. You said, and I quote: "In the case of lpar on POWER systems they simply report steal time and do not alter it in any way.
They do however report how much processor is assigned to the partition and that information is in /proc/ppc64/lparcfg."

Now, that is a *way* more sensible thing to do. Much more. "Confusing users" is something extremely subjective. This is specially true about concepts that are know for quite some time, like steal time. If you out of a sudden change the meaning of this, it is sure to confuse a lot more users than it would clarify.

Quoted 16 lines

---

Michael Wolf (5):
Alter the amount of steal time reported by the guest. Expand the steal time msr to also contain the consigned time. Add the code to send the consigned time from the host to the guest Add a timer to allow the separation of consigned from steal time. Add an ioctl to communicate the consign limit to the host.

Quoted 17 lines

 arch/x86/include/asm/kvm_host.h       |   11 +++++++
 arch/x86/include/asm/kvm_para.h       |    3 +-
 arch/x86/include/asm/paravirt.h       |    4 +--
 arch/x86/include/asm/paravirt_types.h |    2 +
 arch/x86/kernel/kvm.c                 |    8 ++---
 arch/x86/kernel/paravirt.c            |    4 +--
 arch/x86/kvm/x86.c                    |   50 ++++++++++++++++++++++++++++++++-
 fs/proc/stat.c                        |    9 +++++-
 include/linux/kernel_stat.h           |    2 +
 include/linux/kvm_host.h              |    2 +
 include/uapi/linux/kvm.h              |    2 +
 kernel/sched/core.c                   |   10 ++++++-
 kernel/sched/cputime.c                |   21 +++++++++++++-
 kernel/sched/sched.h                  |    2 +
 virt/kvm/kvm_main.c                   |    7 +++++
 15 files changed, 120 insertions(+), 17 deletions(-)

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat.

This can
cause confusion for the end user. To ease the confusion this patch set adds the idea of consigned (expected steal) time. The host will separate the consigned time from the steal time. The consignment limit passed to the host will be the amount of steal time expected within a fixed period of time. Any other steal time accruing during that period will show as the traditional steal time.

---

Michael Wolf (5):
Alter the amount of steal time reported by the guest. Expand the steal time msr to also contain the consigned time. Add the code to send the consigned time from the host to the guest Add a timer to allow the separation of consigned from steal time. Add an ioctl to communicate the consign limit to the host.

 arch/x86/include/asm/kvm_host.h       |   11 +++++++
 arch/x86/include/asm/kvm_para.h       |    3 +-
 arch/x86/include/asm/paravirt.h       |    4 +--
 arch/x86/include/asm/paravirt_types.h |    2 +
 arch/x86/kernel/kvm.c                 |    8 ++---
 arch/x86/kernel/paravirt.c            |    4 +--
 arch/x86/kvm/x86.c                    |   50 ++++++++++++++++++++++++++++++++-
 fs/proc/stat.c                        |    9 +++++-
 include/linux/kernel_stat.h           |    2 +
 include/linux/kvm_host.h              |    2 +
 include/uapi/linux/kvm.h              |    2 +
 kernel/sched/core.c                   |   10 ++++++-
 kernel/sched/cputime.c                |   21 +++++++++++++-
 kernel/sched/sched.h                  |    2 +
 virt/kvm/kvm_main.c                   |    7 +++++
 15 files changed, 120 insertions(+), 17 deletions(-)

Signature · 6 lines

Signature

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 3 lines · Michael Wolf

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat.

The definition of stolen time is 'time during which the virtual CPU is runnable to not running'. Overcommit is the main scenario which steal time helps to detect.

Can you describe the 'capped' case?

Quoted 3 lines · Michael Wolf

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat.

The definition of stolen time is 'time during which the virtual CPU is runnable to not running'. Overcommit is the main scenario which steal time helps to detect.

Can you describe the 'capped' case?

Quoted 19 lines

This can
cause confusion for the end user. To ease the confusion this patch set adds the idea of consigned (expected steal) time. The host will separate the consigned time from the steal time. The consignment limit passed to the host will be the amount of steal time expected within a fixed period of time. Any other steal time accruing during that period will show as the traditional steal time.

---

Michael Wolf (5):
Alter the amount of steal time reported by the guest. Expand the steal time msr to also contain the consigned time. Add the code to send the consigned time from the host to the guest Add a timer to allow the separation of consigned from steal time. Add an ioctl to communicate the consign limit to the host.

Quoted 24 lines

  arch/x86/include/asm/kvm_host.h       |   11 +++++++
  arch/x86/include/asm/kvm_para.h       |    3 +-
  arch/x86/include/asm/paravirt.h       |    4 +--
  arch/x86/include/asm/paravirt_types.h |    2 +
  arch/x86/kernel/kvm.c                 |    8 ++---
  arch/x86/kernel/paravirt.c            |    4 +--
  arch/x86/kvm/x86.c                    |   50 ++++++++++++++++++++++++++++++++-
  fs/proc/stat.c                        |    9 +++++-
  include/linux/kernel_stat.h           |    2 +
  include/linux/kvm_host.h              |    2 +
  include/uapi/linux/kvm.h              |    2 +
  kernel/sched/core.c                   |   10 ++++++-
  kernel/sched/cputime.c                |   21 +++++++++++++-
  kernel/sched/sched.h                  |    2 +
  virt/kvm/kvm_main.c                   |    7 +++++
  15 files changed, 120 insertions(+), 17 deletions(-)

Signature · 6 lines

Signature

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 9 lines · Marcelo Tosatti

Quoted 3 lines · Michael Wolf

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat.

The definition of stolen time is 'time during which the virtual CPU is runnable to not running'. Overcommit is the main scenario which steal time helps to detect.

Can you describe the 'capped' case?

In the capped case, the time that the guest spends waiting due to it having used its full allottment of time shows up as steal time. The way my patchset currently stands is that you would set up the bandwidth control and you would have to pass it a matching value from qemu. In the future, it would
be possible to have something parse the bandwidth setting and automatically adjust the setting in the
host used for steal time reporting.

Quoted 1 line

Quoted 1 line

Alter the amount of steal time reported by the guest.

Quoted 1 line

Quoted 1 line

Expand the steal time msr to also contain the consigned time.

Quoted 2 lines

Quoted 2 lines

Add the code to send the consigned time from the host to the
guest

Quoted 1 line

Quoted 1 line

Add a timer to allow the separation of consigned from steal time.

Quoted 1 line

Quoted 1 line

Add an ioctl to communicate the consign limit to the host.

Quoted 10 lines · Michael Wolf

Quoted 9 lines · Marcelo Tosatti

Quoted 3 lines · Michael Wolf

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat.

The definition of stolen time is 'time during which the virtual CPU is runnable to not running'. Overcommit is the main scenario which steal time helps to detect.

Can you describe the 'capped' case?

Quoted 8 lines

In the capped case, the time that the guest spends waiting due to it having used its full allottment of time shows up as steal time. The way my patchset currently stands is that you would set up the bandwidth control and you would have to pass it a matching value from qemu. In the future, it would
be possible to have something parse the bandwidth setting and automatically adjust the setting in the
host used for steal time reporting.

Ok, so correct me if I am wrong, but I believe you would be using something like the bandwidth capper in the cpu cgroup to set those entitlements, right?

Some time has passed since I last looked into it, but IIRC, after you get are out of your quota, you should be out of the runqueue. In the lovely world of KVM, we approximate steal time as runqueue time:

arch/x86/kvm/x86.c:
delta = current->sched_info.run_delay - vcpu->arch.st.last_steal; vcpu->arch.st.last_steal = current->sched_info.run_delay; vcpu->arch.st.accum_steal = delta;

include/linux/sched.h:
unsigned long long run_delay; /* time spent waiting on a runqueue */

So if you are out of the runqueue, you won't get steal time accounted, and then I truly fail to understand what you are doing.

In case I am wrong, and run_delay also includes the time you can't run because you are out of capacity, then maybe what we should do, is to just subtract it from run_delay in kvm/x86.c before we pass it on. In summary:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Alter the amount of steal time reported by the guest.

Maybe this should go away.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Expand the steal time msr to also contain the consigned time.

Maybe this should go away

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Add the code to send the consigned time from the host to the
guest

This definitely should be heavily modified

Quoted 1 line

Quoted 1 line

Quoted 1 line

Add a timer to allow the separation of consigned from steal time.

Maybe this should go away

Quoted 1 line

Quoted 1 line

Quoted 1 line

Add an ioctl to communicate the consign limit to the host.

This definitely should go away.

More specifically, *whatever* way we use to cap the processor, the host system will have all the information at all times.

Quoted 6 lines

include/linux/sched.h:
unsigned long long run_delay; /* time spent waiting on a runqueue */

So if you are out of the runqueue, you won't get steal time accounted, and then I truly fail to understand what you are doing.

So I looked at something like this in the past. To make sure things haven't changed
I set up a cgroup on my test server running a kernel built from the latest tip tree.

[root]# cat cpu.cfs_quota_us
50000
[root]# cat cpu.cfs_period_us
100000
[root]# cat cpuset.cpus
1
[root]# cat cpuset.mems
0

Next I put the PID from the cpu thread into tasks. When I start a script that will hog the cpu I see the
following in top on the guest
Cpu(s): 1.9%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 48.3%hi, 0.0%si, 49.8%st

So the steal time here is in line with the bandwidth control settings.

So then the steal time did not show on the guest. You have no value that needs to be passed
around. What I did not like about this approach was
* only works for cfs bandwidth control. If another type of hard limit was added to the kernel
the code would potentially need to change.

* This approach doesn't help if the limits are set by overcommitting the cpus. It is my understanding
that this is a common approach.

Quoted 5 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Add an ioctl to communicate the consign limit to the host.

This definitely should go away.

More specifically, *whatever* way we use to cap the processor, the host system will have all the information at all times.

I'm not understanding that comment. If you are capping by simply controlling the amount of
overcommit on the host then wouldn't you still need some value to indicate the desired amount.

I am deeply sorry.

I was busy first time I read this, so I postponed answering and ended up forgetting.

Sorry

Quoted 19 lines

Quoted 5 lines

include/linux/sched.h:
unsigned long long run_delay; /* time spent waiting on a runqueue */

So if you are out of the runqueue, you won't get steal time accounted, and then I truly fail to understand what you are doing.

So I looked at something like this in the past. To make sure things haven't changed
I set up a cgroup on my test server running a kernel built from the latest tip tree.

[root]# cat cpu.cfs_quota_us
50000
[root]# cat cpu.cfs_period_us
100000
[root]# cat cpuset.cpus
1
[root]# cat cpuset.mems
0

Quoted 7 lines

Next I put the PID from the cpu thread into tasks. When I start a script that will hog the cpu I see the
following in top on the guest
Cpu(s): 1.9%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 48.3%hi, 0.0%si, 49.8%st

So the steal time here is in line with the bandwidth control settings.

Ok. So I was wrong in my hunch that it would be outside the runqueue, therefore work automatically. Still, the host kernel has all the information in cgroups.

Quoted 6 lines

So then the steal time did not show on the guest. You have no value that needs to be passed
around. What I did not like about this approach was
* only works for cfs bandwidth control. If another type of hard limit was added to the kernel
the code would potentially need to change.

This is true for almost everything we have in the kernel! It is *very* unlikely for other bandwidth control mechanism to ever appear. If it ever does, it's *their* burden to make sure it works for steal time (provided it is merged). Code in tree gets precedence.

Quoted 5 lines

* This approach doesn't help if the limits are set by overcommitting the cpus. It is my understanding
that this is a common approach.

I can't say anything about commonality, but common or not, it is a *crazy* approach.

When you simply overcommit, you have no way to differentiate between intended steal time and non-intended steal time. Moreover, when you overcommit, your cpu usage will vary over time. If two guests use the cpu to their full power, you will have 50 % each. But if one of them slows down, the other gets more. What is your entitlement value? How do you define this?

And then after you define it, you end up using more than this, what is your cpu usage? 130 %?

The only sane way to do it, is to communicate this value to the kernel somehow. The bandwidth controller is the interface we have for that. So everybody that wants to *intentionally* overcommit needs to communicate this to the controller. IOW: Any sane configuration should be explicit about your capping.

Quoted 9 lines

Quoted 5 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Add an ioctl to communicate the consign limit to the host.

This definitely should go away.

More specifically, *whatever* way we use to cap the processor, the host system will have all the information at all times.

I'm not understanding that comment. If you are capping by simply controlling the amount of
overcommit on the host then wouldn't you still need some value to indicate the desired amount.

No, that is just crazy, and I don't like it a single bit.

So in the light of it: Whatever capping mechanism we have, we need to be explicit about the expected entitlement. At this point, the kernel already knows what it is, and needs no extra ioctls or anything like that.

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html