Alter steal-time reporting in the guest

Expand the steal time msr to also contain the consigned time.

Signed-off-by Michael Wolf <[email protected]>

---
  arch/x86/include/asm/paravirt.h       |    4 ++--
  arch/x86/include/asm/paravirt_types.h |    2 +-
  arch/x86/kernel/kvm.c                 |    7 ++-----
  kernel/sched/core.c                   |   10 +++++++++-
  kernel/sched/cputime.c                |    2 +-
  5 files changed, 15 insertions(+), 10 deletions(-)

arch/x86/include/asm/paravirt.h +0 -0

index 5edd174..9b753ea 100644

Quoted 16 lines · Michael Wolf

Expand the steal time msr to also contain the consigned time.

Signed-off-by Michael Wolf <[email protected]>

---
  arch/x86/include/asm/paravirt.h       |    4 ++--
  arch/x86/include/asm/paravirt_types.h |    2 +-
  arch/x86/kernel/kvm.c                 |    7 ++-----
  kernel/sched/core.c                   |   10 +++++++++-
  kernel/sched/cputime.c                |    2 +-
  5 files changed, 15 insertions(+), 10 deletions(-)

arch/x86/include/asm/paravirt.h +0 -0

index 5edd174..9b753ea 100644

arch/x86/include/asm/paravirt.h struct static_key; · line 202

Quoted 9 lines

arch/x86/include/asm/paravirt.h +2 -2

@@ -196,9 +196,9 @@ struct static_key;
  extern struct static_key paravirt_steal_enabled;
  extern struct static_key paravirt_steal_rq_enabled;
 
-static inline u64 paravirt_steal_clock(int cpu)
+static inline void paravirt_steal_clock(int cpu, u64 *steal)
  {
-    return PVOP_CALL1(u64, pv_time_ops.steal_clock, cpu);
+    PVOP_VCALL2(pv_time_ops.steal_clock, cpu, steal);
  }

This may be a stupid question, but what happens if a KVM guest with this change, runs on a kernel that still has the old steal time interface?

What happens if the host has the new steal time interface, but the guest uses the old interface?

Will both cases continue to work as expected with your patch series?

If so, could you document (in the source code) why things continue to work?

Change the paravirt calls that retrieve the steal-time information from the host. Add to it getting the consigned value as well as the steal time.

Signed-off-by Michael Wolf <[email protected]>

arch/x86/include/uapi/asm/kvm_para.h +0 -0

index 06fdbd9..55d617f 100644

Quoted 11 lines · Michael Wolf

Change the paravirt calls that retrieve the steal-time information from the host. Add to it getting the consigned value as well as the steal time.

Signed-off-by Michael Wolf <[email protected]>

arch/x86/include/uapi/asm/kvm_para.h +0 -0

index 06fdbd9..55d617f 100644

arch/x86/include/uapi/asm/kvm_para.h line 49

Quoted 9 lines

arch/x86/include/uapi/asm/kvm_para.h +2 -1

@@ -42,9 +42,10 @@
 
  struct kvm_steal_time {
      __u64 steal;
+    __u64 consigned;
      __u32 version;
      __u32 flags;
-    __u32 pad[12];
+    __u32 pad[10];
  };

The function kvm_register_steal_time passes the address of such a structure to the host kernel, which then does something with it.

Could running a guest with the above patch, on top of a host with the old code, result in the values for "version" and "flags" being written into "consigned"?

Could that result in confusing the guest kernel to no end, and generally breaking things?

Add a helper routine to scheduler/core.c to allow the kvm module to retrieve the cpu hardlimit settings. The values will be used to set up a timer that is used to separate the consigned from the steal time.

Quoted 4 lines · Michael Wolf

Add a helper routine to scheduler/core.c to allow the kvm module to retrieve the cpu hardlimit settings. The values will be used to set up a timer that is used to separate the consigned from the steal time.

Sorry: What is the business of a timer in here?
Whenever we read steal time, we know how much time has passed and with that information we can know the entitlement for the period. This breaks if we suspend, but we know that we suspended, so this is not a problem.

Everything bigger the entitlement is steal time.

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 9 lines · Glauber Costa

Quoted 4 lines · Michael Wolf

Add a helper routine to scheduler/core.c to allow the kvm module to retrieve the cpu hardlimit settings. The values will be used to set up a timer that is used to separate the consigned from the steal time.

Sorry: What is the business of a timer in here?
Whenever we read steal time, we know how much time has passed and with that information we can know the entitlement for the period. This breaks if we suspend, but we know that we suspended, so this is not a problem.

I may be missing something, but how do we know how much time has passed? That is why
I had the timer in there. I will go look again at the code but I thought the data was collected
as ticks and passed at random times. The ticks are also accumulating so we are looking at the
difference in the count between reads.....

Quoted 10 lines · Michael Wolf

Quoted 9 lines · Glauber Costa

Quoted 4 lines · Michael Wolf

Add a helper routine to scheduler/core.c to allow the kvm module to retrieve the cpu hardlimit settings. The values will be used to set up a timer that is used to separate the consigned from the steal time.

Sorry: What is the business of a timer in here?
Whenever we read steal time, we know how much time has passed and with that information we can know the entitlement for the period. This breaks if we suspend, but we know that we suspended, so this is not a problem.

Quoted 7 lines

I may be missing something, but how do we know how much time has passed? That is why
I had the timer in there. I will go look again at the code but I thought the data was collected
as ticks and passed at random times. The ticks are also accumulating so we are looking at the
difference in the count between reads.....

They can be collected at random times, but you can of course record the time in which it happened.

Add a helper routine to scheduler/core.c to allow the kvm module to retrieve the cpu hardlimit settings. The values will be used to set up a timer that is used to separate the consigned from the steal time.

Quoted 4 lines · Michael Wolf

Add a helper routine to scheduler/core.c to allow the kvm module to retrieve the cpu hardlimit settings. The values will be used to set up a timer that is used to separate the consigned from the steal time.

1) Can you please describe, in english, the mechanics of subtracting cpu hardlimit values from steal time reported via run_delay supposed to work?

"The period and the quota used to separate the consigned time (expected steal) from the steal time are taken
from the cfs bandwidth control settings. Any other steal time accruing during that period will show as the traditional steal time."

There is no "expected steal time" over a fixed period of real time.

2) From the description of patch 1: "In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat."

This is outdated, right? Because overcommitted environment is exactly what steal time should report.

Thanks

Sorry for the delay in the response. I did not see your question.

Quoted 15 lines · Marcelo Tosatti

Quoted 4 lines · Michael Wolf

Add a helper routine to scheduler/core.c to allow the kvm module to retrieve the cpu hardlimit settings. The values will be used to set up a timer that is used to separate the consigned from the steal time.

1) Can you please describe, in english, the mechanics of subtracting cpu hardlimit values from steal time reported via run_delay supposed to work?

"The period and the quota used to separate the consigned time (expected steal) from the steal time are taken
from the cfs bandwidth control settings. Any other steal time accruing during that period will show as the traditional steal time."

Quoted 1 line

There is no "expected steal time" over a fixed period of real time.

There is expected steal time in the sense that the administrator of the system sets up guests on the host so that there will be cpu overcommitment.

The end user who is using the guest does not know this,
they only know they have been guaranteed a certain level of performance. So if steal time shows up the end user typically thinks they are not getting their guaranteed performance. So this patchset is meant to allow top to show 100% utilization and ONLY show steal time if it is over the

level of steal time that the host administrator setup. So take a simple example of a host with 1 cpu and two guest on it. If each guest is fully utilized a user will see 50% utilization and 50% steal in either of the guests. In this case the amount of steal time that the host administrator would expect to see is 50%. As long as the steal in the guest does not exceed 50% the guest is running as expected. If for some reason the steal increases to 60%, now something is wrong and the steal time needs to be reported and the end user will make inquiries?

Quoted 5 lines

2) From the description of patch 1: "In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat."

Quoted 2 lines

This is outdated, right? Because overcommitted environment is exactly what steal time should report.

I hope I'm not missing your point here. But again this comes down to the point of view. The end user is guaranteed a capability/level of performance that may not be a whole cpu. So only show steal time if the amount of steal time exceeds what the host admin expected when the guest was set up.

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user.

To ease the confusion this patch set
adds the idea of consigned (expected steal) time. The host will separate the consigned time from the steal time. Tthe steal time will only be altered if hard limits (cfs bandwidth control) is used. The period and the quota used to separate the consigned time (expected steal) from the steal time are taken from the cfs bandwidth control settings. Any other steal time accruing during that period will show as the traditional steal time.

2013/2/5 Michael Wolf <[email protected]>:

Quoted 4 lines

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user.

Sorry, I'm no expert in this area. But I don't really understand what is confusing for the end user here.

Quoted 7 lines

To ease the confusion this patch set
adds the idea of consigned (expected steal) time. The host will separate the consigned time from the steal time. Tthe steal time will only be altered if hard limits (cfs bandwidth control) is used. The period and the quota used to separate the consigned time (expected steal) from the steal time are taken from the cfs bandwidth control settings. Any other steal time accruing during that period will show as the traditional steal time.

I'm also a bit confused here. steal time will then only account the cpu time lost due to quotas from cfs bandwidth control? Also what do you exactly mean by "expected steal time"? Is it steal time due to overcommitting minus scheduler quotas?

Thanks.

Quoted 6 lines · Frederic Weisbecker

2013/2/5 Michael Wolf <[email protected]>:

Quoted 4 lines

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user.

Quoted 2 lines

Sorry, I'm no expert in this area. But I don't really understand what is confusing for the end user here.

I suppose that what is wanted is to subtract stolen time due to 'known reasons' from steal time reporting. 'Known reasons' being, for example, hard caps. So a vcpu executing instructions with no halt, but limited to 80% of available bandwidth, would not have 20% of stolen time reported.

But yes, a description of the scenario that is being dealt with, with details, is important.

Quoted 9 lines

Quoted 8 lines

To ease the confusion this patch set
adds the idea of consigned (expected steal) time. The host will separate the consigned time from the steal time. Tthe steal time will only be altered if hard limits (cfs bandwidth control) is used. The period and the quota used to separate the consigned time (expected steal) from the steal time are taken from the cfs bandwidth control settings. Any other steal time accruing during that period will show as the traditional steal time.

Quoted 6 lines

I'm also a bit confused here. steal time will then only account the cpu time lost due to quotas from cfs bandwidth control? Also what do you exactly mean by "expected steal time"? Is it steal time due to overcommitting minus scheduler quotas?

Thanks.

Sorry for the delay in the response. I did not see the email right away.

Quoted 7 lines · Marcelo Tosatti

Quoted 6 lines · Frederic Weisbecker

2013/2/5 Michael Wolf <[email protected]>:

Quoted 4 lines

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user.

Quoted 7 lines

Quoted 2 lines

Sorry, I'm no expert in this area. But I don't really understand what is confusing for the end user here.

I suppose that what is wanted is to subtract stolen time due to 'known reasons' from steal time reporting. 'Known reasons' being, for example, hard caps. So a vcpu executing instructions with no halt, but limited to 80% of available bandwidth, would not have 20% of stolen time reported.

Yes exactly and the end user many times did not set up the guest and is not aware of the capping. The end user is only aware of the performance level that they were told they would get with the guest.

Quoted 2 lines

But yes, a description of the scenario that is being dealt with, with details, is important.

I will add more detail to the description next time I submit the patches. How about something like,"In a cloud environment the user of a kvm guest is not aware of the underlying hardware or how many other guests are running on it. The end user is only aware of a level of
performance that they should see." or does that just muddy the picture
more??

Quoted 25 lines · Michael Wolf

Sorry for the delay in the response. I did not see the email right away.

Quoted 7 lines · Marcelo Tosatti

Quoted 6 lines · Frederic Weisbecker

2013/2/5 Michael Wolf <[email protected]>:

Quoted 4 lines

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user.

Quoted 7 lines

Quoted 2 lines

Sorry, I'm no expert in this area. But I don't really understand what is confusing for the end user here.

I suppose that what is wanted is to subtract stolen time due to 'known reasons' from steal time reporting. 'Known reasons' being, for example, hard caps. So a vcpu executing instructions with no halt, but limited to 80% of available bandwidth, would not have 20% of stolen time reported.

Yes exactly and the end user many times did not set up the guest and is not aware of the capping. The end user is only aware of the performance level that they were told they would get with the guest.

Quoted 2 lines

But yes, a description of the scenario that is being dealt with, with details, is important.

Quoted 6 lines

I will add more detail to the description next time I submit the patches. How about something like,"In a cloud environment the user of a kvm guest is not aware of the underlying hardware or how many other guests are running on it. The end user is only aware of a level of
performance that they should see." or does that just muddy the picture
more??

So the feature aims for is to report stolen time relative to hard capping. That is: stolen time should be counted as time stolen from the guest _beyond_ hard capping. Yes?

Probably don't need to report new data to the guest for that.

Quoted 33 lines · Marcelo Tosatti

Quoted 25 lines · Michael Wolf

Sorry for the delay in the response. I did not see the email right away.

Quoted 7 lines · Marcelo Tosatti

Quoted 6 lines · Frederic Weisbecker

2013/2/5 Michael Wolf <[email protected]>:

Quoted 4 lines

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user.

Quoted 7 lines

Quoted 2 lines

Sorry, I'm no expert in this area. But I don't really understand what is confusing for the end user here.

I suppose that what is wanted is to subtract stolen time due to 'known reasons' from steal time reporting. 'Known reasons' being, for example, hard caps. So a vcpu executing instructions with no halt, but limited to 80% of available bandwidth, would not have 20% of stolen time reported.

Yes exactly and the end user many times did not set up the guest and is not aware of the capping. The end user is only aware of the performance level that they were told they would get with the guest.

Quoted 2 lines

But yes, a description of the scenario that is being dealt with, with details, is important.

Quoted 6 lines

I will add more detail to the description next time I submit the patches. How about something like,"In a cloud environment the user of a kvm guest is not aware of the underlying hardware or how many other guests are running on it. The end user is only aware of a level of
performance that they should see." or does that just muddy the picture
more??

Quoted 6 lines

So the feature aims for is to report stolen time relative to hard capping. That is: stolen time should be counted as time stolen from the guest _beyond_ hard capping. Yes?

Probably don't need to report new data to the guest for that.

If we take into account that 1 second always have one second, I believe that you can just subtract the consigned time from the steal time the host passes to the guest.

During each second, the numbers won't sum up to 100. The delta to 100 is the consigned time, if anyone cares.

Adopting this would simplify this a lot. All you need to do, really, is to get your calculation right from the bandwidth given by the cpu controller. Subtract it in the host, and voila.

Quoted 47 lines · Glauber Costa

Quoted 33 lines · Marcelo Tosatti

Quoted 25 lines · Michael Wolf

Sorry for the delay in the response. I did not see the email right away.

Quoted 7 lines · Marcelo Tosatti

Quoted 6 lines · Frederic Weisbecker

2013/2/5 Michael Wolf <[email protected]>:

Quoted 4 lines

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user.

Quoted 7 lines

Quoted 2 lines

Sorry, I'm no expert in this area. But I don't really understand what is confusing for the end user here.

I suppose that what is wanted is to subtract stolen time due to 'known reasons' from steal time reporting. 'Known reasons' being, for example, hard caps. So a vcpu executing instructions with no halt, but limited to 80% of available bandwidth, would not have 20% of stolen time reported.

Yes exactly and the end user many times did not set up the guest and is not aware of the capping. The end user is only aware of the performance level that they were told they would get with the guest.

Quoted 2 lines

But yes, a description of the scenario that is being dealt with, with details, is important.

Quoted 6 lines

I will add more detail to the description next time I submit the patches. How about something like,"In a cloud environment the user of a kvm guest is not aware of the underlying hardware or how many other guests are running on it. The end user is only aware of a level of
performance that they should see." or does that just muddy the picture
more??

Quoted 6 lines

So the feature aims for is to report stolen time relative to hard capping. That is: stolen time should be counted as time stolen from the guest _beyond_ hard capping. Yes?

Probably don't need to report new data to the guest for that.

If we take into account that 1 second always have one second, I believe that you can just subtract the consigned time from the steal time the host passes to the guest.

During each second, the numbers won't sum up to 100. The delta to 100 is the consigned time, if anyone cares.

Quoted 3 lines

Adopting this would simplify this a lot. All you need to do, really, is to get your calculation right from the bandwidth given by the cpu controller. Subtract it in the host, and voila.

I looked at doing that once but was told that I was changing the interface in an unacceptable way, because now I was not reporting all of the elapsed time. I agree it would make things simpler.

Quoted 3 lines · Michael Wolf

I looked at doing that once but was told that I was changing the interface in an unacceptable way, because now I was not reporting all of the elapsed time. I agree it would make things simpler.

Pointer to that claim, please?

Quoted 6 lines · Marcelo Tosatti

Quoted 3 lines · Michael Wolf

I looked at doing that once but was told that I was changing the interface in an unacceptable way, because now I was not reporting all of the elapsed time. I agree it would make things simpler.

Pointer to that claim, please?

Back in about 2004 or 2005 or so I was looking at changing how user and system times were calculated (in the context of trying to find a better way to report resources used by a thread in an SMT processor). I found that utilities such as top expected the deltas in the /proc/stat numbers to add up to elapsed time, and would report strange and inconsistent results if that wasn't the case. Unfortunately at this distance I don't recall the exact details. I don't know whether the expectation that the deltas in the /proc/stat numbers over a period of time add up to the elapsed real time is documented anywhere, but I wouldn't be at all surprised if some programs depend on it, so it's better to maintain that property.

Quoted 25 lines · Michael Wolf

Sorry for the delay in the response. I did not see the email right away.

Quoted 7 lines · Marcelo Tosatti

Quoted 6 lines · Frederic Weisbecker

2013/2/5 Michael Wolf <[email protected]>:

Quoted 4 lines

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user.

Quoted 7 lines

Quoted 2 lines

Sorry, I'm no expert in this area. But I don't really understand what is confusing for the end user here.

I suppose that what is wanted is to subtract stolen time due to 'known reasons' from steal time reporting. 'Known reasons' being, for example, hard caps. So a vcpu executing instructions with no halt, but limited to 80% of available bandwidth, would not have 20% of stolen time reported.

Yes exactly and the end user many times did not set up the guest and is not aware of the capping. The end user is only aware of the performance level that they were told they would get with the guest.

Quoted 2 lines

But yes, a description of the scenario that is being dealt with, with details, is important.

Quoted 6 lines

I will add more detail to the description next time I submit the patches. How about something like,"In a cloud environment the user of a kvm guest is not aware of the underlying hardware or how many other guests are running on it. The end user is only aware of a level of
performance that they should see." or does that just muddy the picture
more??

So the feature aims for is to report stolen time relative to hard capping. That is: stolen time should be counted as time stolen from the guest _beyond_ hard capping. Yes?

Probably don't need to report new data to the guest for that.

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 33 lines · Marcelo Tosatti

Quoted 25 lines · Michael Wolf

Sorry for the delay in the response. I did not see the email right away.

Quoted 7 lines · Marcelo Tosatti

Quoted 6 lines · Frederic Weisbecker

2013/2/5 Michael Wolf <[email protected]>:

Quoted 4 lines

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user.

Quoted 7 lines

Quoted 2 lines

Sorry, I'm no expert in this area. But I don't really understand what is confusing for the end user here.

I suppose that what is wanted is to subtract stolen time due to 'known reasons' from steal time reporting. 'Known reasons' being, for example, hard caps. So a vcpu executing instructions with no halt, but limited to 80% of available bandwidth, would not have 20% of stolen time reported.

Yes exactly and the end user many times did not set up the guest and is not aware of the capping. The end user is only aware of the performance level that they were told they would get with the guest.

Quoted 2 lines

But yes, a description of the scenario that is being dealt with, with details, is important.

Quoted 6 lines

I will add more detail to the description next time I submit the patches. How about something like,"In a cloud environment the user of a kvm guest is not aware of the underlying hardware or how many other guests are running on it. The end user is only aware of a level of
performance that they should see." or does that just muddy the picture
more??

Quoted 3 lines

So the feature aims for is to report stolen time relative to hard capping. That is: stolen time should be counted as time stolen from the guest _beyond_ hard capping. Yes?

Yes, that is the goal.

Quoted 1 line

Quoted 1 line

Probably don't need to report new data to the guest for that.

Not sure I understand what you are saying here. Do you mean that I don't need to report the expected steal from the guest? If I don't do that then I'm not reporting all of the time and changing /proc/stat in a bigger way than adding another catagory. Also I thought I would need to provide the consigned time and the steal time for debugging purposes. Maybe I'm missing your point.....

From top man page:

Quoted 39 lines · Michael Wolf

Quoted 33 lines · Marcelo Tosatti

Quoted 25 lines · Michael Wolf

Sorry for the delay in the response. I did not see the email right away.

Quoted 7 lines · Marcelo Tosatti

Quoted 6 lines · Frederic Weisbecker

2013/2/5 Michael Wolf <[email protected]>:

Quoted 4 lines

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user.

Quoted 7 lines

Quoted 2 lines

Sorry, I'm no expert in this area. But I don't really understand what is confusing for the end user here.

I suppose that what is wanted is to subtract stolen time due to 'known reasons' from steal time reporting. 'Known reasons' being, for example, hard caps. So a vcpu executing instructions with no halt, but limited to 80% of available bandwidth, would not have 20% of stolen time reported.

Yes exactly and the end user many times did not set up the guest and is not aware of the capping. The end user is only aware of the performance level that they were told they would get with the guest.

Quoted 2 lines

But yes, a description of the scenario that is being dealt with, with details, is important.

Quoted 6 lines

I will add more detail to the description next time I submit the patches. How about something like,"In a cloud environment the user of a kvm guest is not aware of the underlying hardware or how many other guests are running on it. The end user is only aware of a level of
performance that they should see." or does that just muddy the picture
more??

Quoted 3 lines

So the feature aims for is to report stolen time relative to hard capping. That is: stolen time should be counted as time stolen from the guest _beyond_ hard capping. Yes?

Yes, that is the goal.

Quoted 1 line

Quoted 7 lines

Quoted 1 line

Probably don't need to report new data to the guest for that.

Not sure I understand what you are saying here. Do you mean that I don't need to report the expected steal from the guest? If I don't do that then I'm not reporting all of the time and changing /proc/stat in a bigger way than adding another catagory. Also I thought I would need to provide the consigned time and the steal time for debugging purposes. Maybe I'm missing your point.....

OK so the usefulness of steal time comes from the ability to measure CPU cycles that the guest is being deprived of, relative to some unit (implicitly the CPU frequency presented to the VM). That way, it becomes easier to properly allocate resources.

From top man page:
st : time stolen from this vm by the hypervisor

Not only its a problem for the lender, it is also confusing for the user (who has to subtract from the reported value himself), the hardcapping from reported steal time.

The problem with the algorithm in the patchset is the following (practical example):

- Hard capping set to 80% of available CPU.
- vcpu does not exceed its threshold, say workload with 40% CPU utilization.
- Under this scenario it is possible for vcpu to be deprived of cycles (because out of the 40% that workload uses, only 30% of actual CPU time are being provided).
- The algorithm in this patchset will not report any stolen time because it assumes 20% of stolen time reported via 'run_delay' is fixed at all times (which is false), therefore any valid stolen time below 20% will not be reported.

Makes sense?

Not sure what the concrete way to report stolen time relative to hard capping is (probably easier inside the scheduler, where run_delay is calculated).

Reporting the hard capping to the guest is a good idea (which saves the user from having to measure it themselves), but better done separately via new field.

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 39 lines · Michael Wolf

Quoted 33 lines · Marcelo Tosatti

Quoted 25 lines · Michael Wolf

Sorry for the delay in the response. I did not see the email right away.

Quoted 7 lines · Marcelo Tosatti

Quoted 6 lines · Frederic Weisbecker

2013/2/5 Michael Wolf <[email protected]>:

Quoted 4 lines

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user.

Quoted 7 lines

Quoted 2 lines

Sorry, I'm no expert in this area. But I don't really understand what is confusing for the end user here.

I suppose that what is wanted is to subtract stolen time due to 'known reasons' from steal time reporting. 'Known reasons' being, for example, hard caps. So a vcpu executing instructions with no halt, but limited to 80% of available bandwidth, would not have 20% of stolen time reported.

Yes exactly and the end user many times did not set up the guest and is not aware of the capping. The end user is only aware of the performance level that they were told they would get with the guest.

Quoted 2 lines

But yes, a description of the scenario that is being dealt with, with details, is important.

Quoted 6 lines

I will add more detail to the description next time I submit the patches. How about something like,"In a cloud environment the user of a kvm guest is not aware of the underlying hardware or how many other guests are running on it. The end user is only aware of a level of
performance that they should see." or does that just muddy the picture
more??

Quoted 3 lines

So the feature aims for is to report stolen time relative to hard capping. That is: stolen time should be counted as time stolen from the guest _beyond_ hard capping. Yes?

Yes, that is the goal.

Quoted 1 line

Quoted 7 lines

Quoted 1 line

Probably don't need to report new data to the guest for that.

Not sure I understand what you are saying here. Do you mean that I don't need to report the expected steal from the guest? If I don't do that then I'm not reporting all of the time and changing /proc/stat in a bigger way than adding another catagory. Also I thought I would need to provide the consigned time and the steal time for debugging purposes. Maybe I'm missing your point.....

OK so the usefulness of steal time comes from the ability to measure CPU cycles that the guest is being deprived of, relative to some unit (implicitly the CPU frequency presented to the VM). That way, it becomes easier to properly allocate resources.

From top man page:
st : time stolen from this vm by the hypervisor

Not only its a problem for the lender, it is also confusing for the user (who has to subtract from the reported value himself), the hardcapping from reported steal time.

The problem with the algorithm in the patchset is the following (practical example):

- Hard capping set to 80% of available CPU.
- vcpu does not exceed its threshold, say workload with 40% CPU utilization.
- Under this scenario it is possible for vcpu to be deprived of cycles (because out of the 40% that workload uses, only 30% of actual CPU time are being provided).
- The algorithm in this patchset will not report any stolen time because it assumes 20% of stolen time reported via 'run_delay' is fixed at all times (which is false), therefore any valid stolen time below 20% will not be reported.

Makes sense?

Not sure what the concrete way to report stolen time relative to hard capping is (probably easier inside the scheduler, where run_delay is calculated).

Reporting the hard capping to the guest is a good idea (which saves the user from having to measure it themselves), but better done separately via new field.

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 7 lines

Makes sense?

Not sure what the concrete way to report stolen time relative to hard capping is (probably easier inside the scheduler, where run_delay is calculated).

Quoted 3 lines

Reporting the hard capping to the guest is a good idea (which saves the user from having to measure it themselves), but better done separately via new field.

didnt respond to this in the previous response. I'm not sure I'm following you here. I thought this is what I was doing by having a consigned (expected steal) field add to the /proc/stat output. Are you looking for something else or a better naming convention?

Quoted 7 lines · Michael Wolf

Quoted 7 lines

Makes sense?

Not sure what the concrete way to report stolen time relative to hard capping is (probably easier inside the scheduler, where run_delay is calculated).

Quoted 8 lines

Quoted 3 lines

Reporting the hard capping to the guest is a good idea (which saves the user from having to measure it themselves), but better done separately via new field.

didnt respond to this in the previous response. I'm not sure I'm following you here. I thought this is what I was doing by having a consigned (expected steal) field add to the /proc/stat output. Are you looking for something else or a better naming convention?

Expected steal is not a good measure to use (because as mentioned in the previous email there is no expected steal over a fixed period of time).

It is fine to report 'maximum percentage of underlying physical CPU' (what percentage of the physical CPU time guest VM is allowed to make use of).

And then steal time is relative to maximum percentage of underlying physical CPU time allowed.

Quoted 20 lines · Marcelo Tosatti

Quoted 7 lines · Michael Wolf

Quoted 7 lines

Makes sense?

Not sure what the concrete way to report stolen time relative to hard capping is (probably easier inside the scheduler, where run_delay is calculated).

Quoted 8 lines

Quoted 3 lines

Reporting the hard capping to the guest is a good idea (which saves the user from having to measure it themselves), but better done separately via new field.

didnt respond to this in the previous response. I'm not sure I'm following you here. I thought this is what I was doing by having a consigned (expected steal) field add to the /proc/stat output. Are you looking for something else or a better naming convention?

Expected steal is not a good measure to use (because as mentioned in the previous email there is no expected steal over a fixed period of time).

Quoted 7 lines

It is fine to report 'maximum percentage of underlying physical CPU' (what percentage of the physical CPU time guest VM is allowed to make use of).

And then steal time is relative to maximum percentage of underlying physical CPU time allowed.

So last August I had sent out an RFC set of patches to do this. That patchset was meant to handle the general overcommit case as well as the capping case by having qemu pass a percentage to the host that would then be passed onto the guest and used to adjust the steal time. Here is the link to the discussion
http://lkml.indiana.edu/hypermail/linux/kernel/1208.3/01458.html

As you will see there Avi didn't like the idea of a percentage down in the guest, among other reasons he was concerned about migration. Also in the email thread you will see that Anthony Liguori was opposed to the idea of just changing the steal time, he wanted it split out.

What Glauber has suggested and I am working on implementing is taking out the timer and adding a last read field in the host. So in the host I can determine the total time that has passed and compute a percentage and apply that percentage to the steal time while the info is still on the host. Then pass the steal and consigned time to the guest.

Does that address your concerns?

Quoted 36 lines · Michael Wolf

Quoted 20 lines · Marcelo Tosatti

Quoted 7 lines · Michael Wolf

Quoted 7 lines

Makes sense?

Not sure what the concrete way to report stolen time relative to hard capping is (probably easier inside the scheduler, where run_delay is calculated).

Quoted 8 lines

Quoted 3 lines

Reporting the hard capping to the guest is a good idea (which saves the user from having to measure it themselves), but better done separately via new field.

didnt respond to this in the previous response. I'm not sure I'm following you here. I thought this is what I was doing by having a consigned (expected steal) field add to the /proc/stat output. Are you looking for something else or a better naming convention?

Expected steal is not a good measure to use (because as mentioned in the previous email there is no expected steal over a fixed period of time).

Quoted 7 lines

It is fine to report 'maximum percentage of underlying physical CPU' (what percentage of the physical CPU time guest VM is allowed to make use of).

And then steal time is relative to maximum percentage of underlying physical CPU time allowed.

So last August I had sent out an RFC set of patches to do this. That patchset was meant to handle the general overcommit case as well as the capping case by having qemu pass a percentage to the host that would then be passed onto the guest and used to adjust the steal time. Here is the link to the discussion
http://lkml.indiana.edu/hypermail/linux/kernel/1208.3/01458.html

Quoted 2 lines

As you will see there Avi didn't like the idea of a percentage down in the guest, among other reasons he was concerned about migration.

Quoted 3 lines

Also in the email thread you will see that Anthony Liguori was opposed to the idea of just changing the steal time, he wanted it split out.

Quoted 5 lines

What Glauber has suggested and I am working on implementing is taking out the timer and adding a last read field in the host. So in the host I can determine the total time that has passed and compute a percentage and apply that percentage to the steal time while the info is still on the host. Then pass the steal and consigned time to the guest.

Quoted 2 lines

Does that address your concerns?

I am not asking about passing percentage down the host - just pointing out a counter example to the correctness of the current algorithm.

I cannot see how you can report proper steal time value relative to hard cap without having that number calculated in the scheduler. IOW, "run_delay" must be split in two: you want to differentiate whether run delay was due to hard cap exhaustion or due to other reasons. Without that, steal time reporting is incorrect (as the example details). Now the question is, how to do that separation.

Sorry for the delay in the response. I did not see the email right away.

Quoted 7 lines · Marcelo Tosatti

Quoted 6 lines · Frederic Weisbecker

2013/2/5 Michael Wolf <[email protected]>:

Quoted 4 lines

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user.

Quoted 7 lines

Quoted 2 lines

Sorry, I'm no expert in this area. But I don't really understand what is confusing for the end user here.

I suppose that what is wanted is to subtract stolen time due to 'known reasons' from steal time reporting. 'Known reasons' being, for example, hard caps. So a vcpu executing instructions with no halt, but limited to 80% of available bandwidth, would not have 20% of stolen time reported.

Yes exactly and the end user many times did not set up the guest and is not aware of the capping. The end user is only aware of the performance level that they were told they would get with the guest.

Quoted 3 lines

But yes, a description of the scenario that is being dealt with, with details, is important.

I will add more detail to the description next time I submit the patches. How about something like,"In a cloud environment the user of a kvm guest is not aware of the underlying hardware or how many other guests are running on it. The end user is only aware of a level of
performance that they should see." or does that just muddy the picture
more??

2013/3/5 Michael Wolf <[email protected]>:

Quoted 27 lines

Sorry for the delay in the response. I did not see the email right away.

Quoted 7 lines · Marcelo Tosatti

Quoted 6 lines · Frederic Weisbecker

2013/2/5 Michael Wolf <[email protected]>:

Quoted 4 lines

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user.

Quoted 7 lines

Quoted 2 lines

Sorry, I'm no expert in this area. But I don't really understand what is confusing for the end user here.

I suppose that what is wanted is to subtract stolen time due to 'known reasons' from steal time reporting. 'Known reasons' being, for example, hard caps. So a vcpu executing instructions with no halt, but limited to 80% of available bandwidth, would not have 20% of stolen time reported.

Yes exactly and the end user many times did not set up the guest and is not aware of the capping. The end user is only aware of the performance level that they were told they would get with the guest.

Quoted 3 lines

But yes, a description of the scenario that is being dealt with, with details, is important.

Quoted 6 lines

I will add more detail to the description next time I submit the patches. How about something like,"In a cloud environment the user of a kvm guest is not aware of the underlying hardware or how many other guests are running on it. The end user is only aware of a level of
performance that they should see." or does that just muddy the picture
more??

That alone is probably not enough. But yeah, make sure you clearly state the difference between expected (caps, sched bandwidth...) and unexpected (overcommitting, competing load...) stolen time. Then add a practical example as you made above that explains why it matters to make that distinction and why you want to report it.

Quoted 6 lines

2013/2/5 Michael Wolf <[email protected]>:

Quoted 4 lines

In the case of where you have a system that is running in a capped or overcommitted environment the user may see steal time being reported in accounting tools such as top or vmstat. This can cause confusion for the end user.

Quoted 2 lines

Sorry, I'm no expert in this area. But I don't really understand what is confusing for the end user here.

I suppose that what is wanted is to subtract stolen time due to 'known reasons' from steal time reporting. 'Known reasons' being, for example, hard caps. So a vcpu executing instructions with no halt, but limited to 80% of available bandwidth, would not have 20% of stolen time reported.

But yes, a description of the scenario that is being dealt with, with details, is important.