CPU hard limits

2. Need for hard limiting CPU resource
--------------------------------------
- Pay-per-use: In enterprise systems that cater to multiple clients/customers where a customer demands a certain share of CPU resources and pays only that, CPU hard limits will be useful to hard limit the customer's job to consume only the specified amount of CPU resource.
- In container based virtualization environments running multiple containers, hard limits will be useful to ensure a container doesn't exceed its CPU entitlement.
- Hard limits can be used to provide guarantees.

- Hard limits can be used to provide guarantees.

2. Need for hard limiting CPU resource
--------------------------------------
- Pay-per-use: In enterprise systems that cater to multiple clients/customers where a customer demands a certain share of CPU resources and pays only that, CPU hard limits will be useful to hard limit the customer's job to consume only the specified amount of CPU resource.
- In container based virtualization environments running multiple containers, hard limits will be useful to ensure a container doesn't exceed its CPU entitlement.
- Hard limits can be used to provide guarantees.

Quoted 11 lines · Bharata B Rao

2. Need for hard limiting CPU resource
--------------------------------------
- Pay-per-use: In enterprise systems that cater to multiple clients/customers where a customer demands a certain share of CPU resources and pays only that, CPU hard limits will be useful to hard limit the customer's job to consume only the specified amount of CPU resource.
- In container based virtualization environments running multiple containers, hard limits will be useful to ensure a container doesn't exceed its CPU entitlement.
- Hard limits can be used to provide guarantees.

How can hard limits provide guarantees?

Quoted 11 lines · Bharata B Rao

2. Need for hard limiting CPU resource
--------------------------------------
- Pay-per-use: In enterprise systems that cater to multiple clients/customers where a customer demands a certain share of CPU resources and pays only that, CPU hard limits will be useful to hard limit the customer's job to consume only the specified amount of CPU resource.
- In container based virtualization environments running multiple containers, hard limits will be useful to ensure a container doesn't exceed its CPU entitlement.
- Hard limits can be used to provide guarantees.

How can hard limits provide guarantees?

Let's take an example where I have 1 group that I wish to guarantee a 20% share of the cpu, and anther 8 groups with no limits or guarantees.

One way to achieve the guarantee is to hard limit each of the 8 other groups to 10%; the sum total of the limits is 80%, leaving 20% for the guarantee group. The downside is the arbitrary limit imposed on the other groups.

Another way is to place the 8 groups in a container group, and limit that to 80%. But that doesn't work if I want to provide guarantees to several groups.

Quoted 11 lines · Bharata B Rao

2. Need for hard limiting CPU resource
--------------------------------------
- Pay-per-use: In enterprise systems that cater to multiple clients/customers where a customer demands a certain share of CPU resources and pays only that, CPU hard limits will be useful to hard limit the customer's job to consume only the specified amount of CPU resource.
- In container based virtualization environments running multiple containers, hard limits will be useful to ensure a container doesn't exceed its CPU entitlement.
- Hard limits can be used to provide guarantees.

How can hard limits provide guarantees?

Let's take an example where I have 1 group that I wish to guarantee a 20% share of the cpu, and anther 8 groups with no limits or guarantees.

One way to achieve the guarantee is to hard limit each of the 8 other groups to 10%; the sum total of the limits is 80%, leaving 20% for the guarantee group. The downside is the arbitrary limit imposed on the other groups.

Another way is to place the 8 groups in a container group, and limit that to 80%. But that doesn't work if I want to provide guarantees to several groups.

Quoted 11 lines · Bharata B Rao

2. Need for hard limiting CPU resource
--------------------------------------
- Pay-per-use: In enterprise systems that cater to multiple clients/customers where a customer demands a certain share of CPU resources and pays only that, CPU hard limits will be useful to hard limit the customer's job to consume only the specified amount of CPU resource.
- In container based virtualization environments running multiple containers, hard limits will be useful to ensure a container doesn't exceed its CPU entitlement.
- Hard limits can be used to provide guarantees.

How can hard limits provide guarantees?

Quoted 11 lines · Bharata B Rao

2. Need for hard limiting CPU resource
--------------------------------------
- Pay-per-use: In enterprise systems that cater to multiple clients/customers where a customer demands a certain share of CPU resources and pays only that, CPU hard limits will be useful to hard limit the customer's job to consume only the specified amount of CPU resource.
- In container based virtualization environments running multiple containers, hard limits will be useful to ensure a container doesn't exceed its CPU entitlement.
- Hard limits can be used to provide guarantees.

How can hard limits provide guarantees?

Let's take an example where I have 1 group that I wish to guarantee a 20% share of the cpu, and anther 8 groups with no limits or guarantees.

One way to achieve the guarantee is to hard limit each of the 8 other groups to 10%; the sum total of the limits is 80%, leaving 20% for the guarantee group. The downside is the arbitrary limit imposed on the other groups.

Another way is to place the 8 groups in a container group, and limit that to 80%. But that doesn't work if I want to provide guarantees to several groups.

Quoted 4 lines

Another way is to place the 8 groups in a container group, and limit that to 80%. But that doesn't work if I want to provide guarantees to several groups.

Hmm why not ? Reduce the guarantee of the container group and provide the same to additional groups ?

Quoted 2 lines

Another way is to place the 8 groups in a container group, and limit that to 80%. But that doesn't work if I want to provide guarantees to several groups.

Hmm why not ? Reduce the guarantee of the container group and provide the same to additional groups ?

Quoted 9 lines · Bharata B Rao

Quoted 5 lines

Another way is to place the 8 groups in a container group, and limit that to 80%. But that doesn't work if I want to provide guarantees to several groups.

Hmm why not ? Reduce the guarantee of the container group and provide the same to additional groups ?

This method produces suboptimal results:

$ cgroup-limits 10 10 0
[50.0, 50.0, 40.0]

I want to provide two 10% guaranteed groups and one best-effort group. Using the limits method, no group can now use more than 50% of the resources.  However, having the first group use 90% of the resources does not violate any guarantees, but it not allowed by the solution.

Quoted 13 lines · Avi Kivity

Quoted 9 lines · Bharata B Rao

Quoted 5 lines

Another way is to place the 8 groups in a container group, and limit
that to 80%. But that doesn't work if I want to provide guarantees to several groups.

Hmm why not ? Reduce the guarantee of the container group and provide the same to additional groups ?

This method produces suboptimal results:

Quoted 9 lines

$ cgroup-limits 10 10 0
[50.0, 50.0, 40.0]

I want to provide two 10% guaranteed groups and one best-effort group.
Using the limits method, no group can now use more than 50% of the
resources. However, having the first group use 90% of the resources does not violate any guarantees, but it not allowed by the solution.

How, it works out fine in my calculation

50 + 40 for G2 and G3, make sure that G1 gets 10%, since others are limited to 90%
50 + 40 for G1 and G3, make sure that G2 gets 10%, since others are limited to 90%
50 + 50 for G1 and G2, make sure that G3 gets 0%, since others are limited to 100%

Now if we really have zeros, I would recommend using

cgroup-limits 10 10 and you'll see that you'll get 90, 90 as output.

Adding zeros to the calcuation is not recommended. Does that help?

Quoted 13 lines · Avi Kivity

Quoted 9 lines · Bharata B Rao

Quoted 5 lines

Another way is to place the 8 groups in a container group, and limit
that to 80%. But that doesn't work if I want to provide guarantees to several groups.

Hmm why not ? Reduce the guarantee of the container group and provide the same to additional groups ?

This method produces suboptimal results:

Quoted 9 lines

$ cgroup-limits 10 10 0
[50.0, 50.0, 40.0]

I want to provide two 10% guaranteed groups and one best-effort group.
Using the limits method, no group can now use more than 50% of the
resources. However, having the first group use 90% of the resources does not violate any guarantees, but it not allowed by the solution.

How, it works out fine in my calculation

50 + 40 for G2 and G3, make sure that G1 gets 10%, since others are limited to 90%
50 + 40 for G1 and G3, make sure that G2 gets 10%, since others are limited to 90%
50 + 50 for G1 and G2, make sure that G3 gets 0%, since others are limited to 100%

Now if we really have zeros, I would recommend using

cgroup-limits 10 10 and you'll see that you'll get 90, 90 as output.

Adding zeros to the calcuation is not recommended. Does that help?

Quoted 25 lines · Balbir Singh

Quoted 13 lines · Avi Kivity

Quoted 9 lines · Bharata B Rao

Quoted 5 lines

Another way is to place the 8 groups in a container group, and limit
that to 80%. But that doesn't work if I want to provide guarantees to several groups.

Hmm why not ? Reduce the guarantee of the container group and provide the same to additional groups ?

This method produces suboptimal results:

Quoted 9 lines

$ cgroup-limits 10 10 0
[50.0, 50.0, 40.0]

I want to provide two 10% guaranteed groups and one best-effort group.
Using the limits method, no group can now use more than 50% of the
resources. However, having the first group use 90% of the resources does not violate any guarantees, but it not allowed by the solution.

Quoted 9 lines

How, it works out fine in my calculation

50 + 40 for G2 and G3, make sure that G1 gets 10%, since others are limited to 90%
50 + 40 for G1 and G3, make sure that G2 gets 10%, since others are limited to 90%
50 + 50 for G1 and G2, make sure that G3 gets 0%, since others are limited to 100%

It's fine in that it satisfies the guarantees, but it is deeply suboptimal. If I ran a cpu hog in the first group, while the other two were idle, it would be limited to 50% cpu. On the other hand, if it consumed all 100% cpu it would still satisfy the guarantees (as the other groups are idle).

The result is that in such a situation, wall clock time would double even though cpu resources are available.

Quoted 5 lines

Now if we really have zeros, I would recommend using

cgroup-limits 10 10 and you'll see that you'll get 90, 90 as output.

Adding zeros to the calcuation is not recommended. Does that help?

What do you mean, it is not recommended? I have two groups which need at least 10% and one which does not need any guarantee, how do I express it?

In any case, changing the zero to 1% does not materially change the results.

Quoted 25 lines · Balbir Singh

Quoted 13 lines · Avi Kivity

Quoted 9 lines · Bharata B Rao

Quoted 5 lines

Another way is to place the 8 groups in a container group, and limit
that to 80%. But that doesn't work if I want to provide guarantees to several groups.

Hmm why not ? Reduce the guarantee of the container group and provide the same to additional groups ?

This method produces suboptimal results:

Quoted 9 lines

$ cgroup-limits 10 10 0
[50.0, 50.0, 40.0]

I want to provide two 10% guaranteed groups and one best-effort group.
Using the limits method, no group can now use more than 50% of the
resources. However, having the first group use 90% of the resources does not violate any guarantees, but it not allowed by the solution.

Quoted 9 lines

How, it works out fine in my calculation

50 + 40 for G2 and G3, make sure that G1 gets 10%, since others are limited to 90%
50 + 40 for G1 and G3, make sure that G2 gets 10%, since others are limited to 90%
50 + 50 for G1 and G2, make sure that G3 gets 0%, since others are limited to 100%

It's fine in that it satisfies the guarantees, but it is deeply suboptimal. If I ran a cpu hog in the first group, while the other two were idle, it would be limited to 50% cpu. On the other hand, if it consumed all 100% cpu it would still satisfy the guarantees (as the other groups are idle).

The result is that in such a situation, wall clock time would double even though cpu resources are available.

Quoted 5 lines

Now if we really have zeros, I would recommend using

cgroup-limits 10 10 and you'll see that you'll get 90, 90 as output.

Adding zeros to the calcuation is not recommended. Does that help?

What do you mean, it is not recommended? I have two groups which need at least 10% and one which does not need any guarantee, how do I express it?

In any case, changing the zero to 1% does not materially change the results.

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

Quoted 42 lines · Avi Kivity

Quoted 25 lines · Balbir Singh

Quoted 13 lines · Avi Kivity

Quoted 9 lines · Bharata B Rao

Quoted 5 lines

Another way is to place the 8 groups in a container group, and limit
that to 80%. But that doesn't work if I want to provide guarantees to several groups.

Hmm why not ? Reduce the guarantee of the container group and provide the same to additional groups ?

This method produces suboptimal results:

Quoted 9 lines

$ cgroup-limits 10 10 0
[50.0, 50.0, 40.0]

I want to provide two 10% guaranteed groups and one best-effort group.
Using the limits method, no group can now use more than 50% of the
resources. However, having the first group use 90% of the resources does not violate any guarantees, but it not allowed by the solution.

Quoted 9 lines

How, it works out fine in my calculation

50 + 40 for G2 and G3, make sure that G1 gets 10%, since others are limited to 90%
50 + 40 for G1 and G3, make sure that G2 gets 10%, since others are limited to 90%
50 + 50 for G1 and G2, make sure that G3 gets 0%, since others are limited to 100%

It's fine in that it satisfies the guarantees, but it is deeply suboptimal. If I ran a cpu hog in the first group, while the other two were idle, it would be limited to 50% cpu. On the other hand, if it consumed all 100% cpu it would still satisfy the guarantees (as the other groups are idle).

Quoted 2 lines

The result is that in such a situation, wall clock time would double even though cpu resources are available.

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

Quoted 16 lines

Quoted 10 lines

How, it works out fine in my calculation

50 + 40 for G2 and G3, make sure that G1 gets 10%, since others are limited to 90%
50 + 40 for G1 and G3, make sure that G2 gets 10%, since others are limited to 90%
50 + 50 for G1 and G2, make sure that G3 gets 0%, since others are limited to 100%

It's fine in that it satisfies the guarantees, but it is deeply suboptimal. If I ran a cpu hog in the first group, while the other two were idle, it would be limited to 50% cpu. On the other hand, if it consumed all 100% cpu it would still satisfy the guarantees (as the other groups are idle).

Quoted 3 lines

The result is that in such a situation, wall clock time would double even though cpu resources are available.

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

Quoted 4 lines · Balbir Singh

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

The example given was two 10% guaranteed groups and one best-effort group. Why would this require idling resources?

If I have a hog in each group, the requirements would be met if the groups got 33, 33, and 33. (Or 10/10/80, for that matter.) If the second and third groups go idle, why not let the first group use 100% of the cpu?

The only hard restriction is that the sum of the guarantees must be less than 100%.

Quoted 4 lines · Balbir Singh

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

The example given was two 10% guaranteed groups and one best-effort group. Why would this require idling resources?

If I have a hog in each group, the requirements would be met if the groups got 33, 33, and 33. (Or 10/10/80, for that matter.) If the second and third groups go idle, why not let the first group use 100% of the cpu?

The only hard restriction is that the sum of the guarantees must be less than 100%.

Quoted 42 lines · Avi Kivity

Quoted 25 lines · Balbir Singh

Quoted 13 lines · Avi Kivity

Quoted 9 lines · Bharata B Rao

Quoted 5 lines

Another way is to place the 8 groups in a container group, and limit
that to 80%. But that doesn't work if I want to provide guarantees to several groups.

Hmm why not ? Reduce the guarantee of the container group and provide the same to additional groups ?

This method produces suboptimal results:

Quoted 9 lines

$ cgroup-limits 10 10 0
[50.0, 50.0, 40.0]

I want to provide two 10% guaranteed groups and one best-effort group.
Using the limits method, no group can now use more than 50% of the
resources. However, having the first group use 90% of the resources does not violate any guarantees, but it not allowed by the solution.

Quoted 9 lines

How, it works out fine in my calculation

50 + 40 for G2 and G3, make sure that G1 gets 10%, since others are limited to 90%
50 + 40 for G1 and G3, make sure that G2 gets 10%, since others are limited to 90%
50 + 50 for G1 and G2, make sure that G3 gets 0%, since others are limited to 100%

It's fine in that it satisfies the guarantees, but it is deeply suboptimal. If I ran a cpu hog in the first group, while the other two were idle, it would be limited to 50% cpu. On the other hand, if it consumed all 100% cpu it would still satisfy the guarantees (as the other groups are idle).

Quoted 2 lines

The result is that in such a situation, wall clock time would double even though cpu resources are available.

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

Quoted 5 lines

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

OK, I see part of your concern, but I think we could do some optimizations during design. For example if all groups have reached their hard-limit and the system is idle, should we do start a new hard limit interval and restart, so that idleness can be removed. Would that be an acceptable design point?

Quoted 5 lines

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

OK, I see part of your concern, but I think we could do some optimizations during design. For example if all groups have reached their hard-limit and the system is idle, should we do start a new hard limit interval and restart, so that idleness can be removed. Would that be an acceptable design point?

Quoted 6 lines · Balbir Singh

Quoted 5 lines

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

Quoted 5 lines

OK, I see part of your concern, but I think we could do some optimizations during design. For example if all groups have reached their hard-limit and the system is idle, should we do start a new hard limit interval and restart, so that idleness can be removed. Would that be an acceptable design point?

I think so. Given guarantees G1..Gn (0 <= Gi <= 1; sum(Gi) <= 1), and a cpu hog running in each group, how would the algorithm divide resources?

Quoted 6 lines · Balbir Singh

Quoted 5 lines

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

Quoted 5 lines

OK, I see part of your concern, but I think we could do some optimizations during design. For example if all groups have reached their hard-limit and the system is idle, should we do start a new hard limit interval and restart, so that idleness can be removed. Would that be an acceptable design point?

I think so. Given guarantees G1..Gn (0 <= Gi <= 1; sum(Gi) <= 1), and a cpu hog running in each group, how would the algorithm divide resources?

Quoted 13 lines · Avi Kivity

Quoted 11 lines · Balbir Singh

Quoted 5 lines

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

OK, I see part of your concern, but I think we could do some optimizations during design. For example if all groups have reached their hard-limit and the system is idle, should we do start a new hard limit interval and restart, so that idleness can be removed. Would that be an acceptable design point?

Quoted 3 lines

I think so. Given guarantees G1..Gn (0 <= Gi <= 1; sum(Gi) <= 1), and a cpu hog running in each group, how would the algorithm divide resources?

As per the matrix calculation, but as soon as we reach an idle point, we redistribute the b/w and start a new quantum so to speak, where all groups are charged up to their hard limits.

Quoted 4 lines

I think so. Given guarantees G1..Gn (0 <= Gi <= 1; sum(Gi) <= 1), and a cpu hog running in each group, how would the algorithm divide resources?

As per the matrix calculation, but as soon as we reach an idle point, we redistribute the b/w and start a new quantum so to speak, where all groups are charged up to their hard limits.

For your question, if there is a CPU hog running, it would be as per the matrix calculation, since the system has no idle point during the bandwidth period.

Quoted 13 lines · Avi Kivity

Quoted 11 lines · Balbir Singh

Quoted 5 lines

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

OK, I see part of your concern, but I think we could do some optimizations during design. For example if all groups have reached their hard-limit and the system is idle, should we do start a new hard limit interval and restart, so that idleness can be removed. Would that be an acceptable design point?

Quoted 3 lines

I think so. Given guarantees G1..Gn (0 <= Gi <= 1; sum(Gi) <= 1), and a cpu hog running in each group, how would the algorithm divide resources?

As per the matrix calculation, but as soon as we reach an idle point, we redistribute the b/w and start a new quantum so to speak, where all groups are charged up to their hard limits.

Quoted 21 lines · Balbir Singh

Quoted 18 lines · Avi Kivity

Quoted 6 lines · Balbir Singh

Quoted 6 lines

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

Quoted 6 lines

OK, I see part of your concern, but I think we could do some optimizations during design. For example if all groups have reached their hard-limit and the system is idle, should we do start a new hard limit interval and restart, so that idleness can be removed. Would that be an acceptable design point?

I think so. Given guarantees G1..Gn (0 <= Gi <= 1; sum(Gi) <= 1), and a cpu hog running in each group, how would the algorithm divide resources?

Quoted 4 lines

As per the matrix calculation, but as soon as we reach an idle point, we redistribute the b/w and start a new quantum so to speak, where all groups are charged up to their hard limits.

But could there be client models where you are required to strictly adhere to the limit within the bandwidth and not provide more (by advancing the bandwidth period) in the presence of idle cycles ?

Quoted 19 lines · Balbir Singh

Quoted 16 lines · Avi Kivity

Quoted 11 lines · Balbir Singh

Quoted 5 lines

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

OK, I see part of your concern, but I think we could do some optimizations during design. For example if all groups have reached their hard-limit and the system is idle, should we do start a new hard limit interval and restart, so that idleness can be removed. Would that be an acceptable design point?

I think so. Given guarantees G1..Gn (0 <= Gi <= 1; sum(Gi) <= 1), and a cpu hog running in each group, how would the algorithm divide resources?

Quoted 3 lines

As per the matrix calculation, but as soon as we reach an idle point, we redistribute the b/w and start a new quantum so to speak, where all groups are charged up to their hard limits.

But could there be client models where you are required to strictly adhere to the limit within the bandwidth and not provide more (by advancing the bandwidth period) in the presence of idle cycles ?

Quoted 20 lines · Balbir Singh

Quoted 17 lines · Avi Kivity

Quoted 6 lines · Balbir Singh

Quoted 6 lines

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

Quoted 6 lines

OK, I see part of your concern, but I think we could do some optimizations during design. For example if all groups have reached their hard-limit and the system is idle, should we do start a new hard limit interval and restart, so that idleness can be removed. Would that be an acceptable design point?

I think so. Given guarantees G1..Gn (0 <= Gi <= 1; sum(Gi) <= 1), and a cpu hog running in each group, how would the algorithm divide resources?

Quoted 4 lines

As per the matrix calculation, but as soon as we reach an idle point, we redistribute the b/w and start a new quantum so to speak, where all groups are charged up to their hard limits.

But could there be client models where you are required to strictly adhere to the limit within the bandwidth and not provide more (by advancing the bandwidth period) in the presence of idle cycles ?

Quoted 5 lines · Bharata B Rao

But could there be client models where you are required to strictly adhere to the limit within the bandwidth and not provide more (by advancing the bandwidth period) in the presence of idle cycles ?

That's the limit part. I'd like to be able to specify limits and guarantees on the same host and for the same groups; I don't think that works when you advance the bandwidth period.

I think we need to treat guarantees as first-class goals, not something derived from limits (in fact I think guarantees are more useful as they can be used to provide SLAs).

Quoted 11 lines · Avi Kivity

Quoted 5 lines · Bharata B Rao

But could there be client models where you are required to strictly adhere to the limit within the bandwidth and not provide more (by advancing the bandwidth period) in the presence of idle cycles ?

That's the limit part. I'd like to be able to specify limits and guarantees on the same host and for the same groups; I don't think that works when you advance the bandwidth period.

Quoted 4 lines

I think we need to treat guarantees as first-class goals, not something derived from limits (in fact I think guarantees are more useful as they can be used to provide SLAs).

I agree that guarantees are important, but I am not sure about

1. specifying both limits and guarantees for groups and

2. not deriving guarantees from limits.

Guarantees are met by some form of throttling or limiting and hence I think limiting should drive the guarantees

Quoted 7 lines · Avi Kivity

Quoted 5 lines · Bharata B Rao

But could there be client models where you are required to strictly adhere to the limit within the bandwidth and not provide more (by advancing the bandwidth period) in the presence of idle cycles ?

Quoted 4 lines

That's the limit part. I'd like to be able to specify limits and guarantees on the same host and for the same groups; I don't think that works when you advance the bandwidth period.

I think we need to treat guarantees as first-class goals, not something derived from limits (in fact I think guarantees are more useful as they can be used to provide SLAs).

I agree that guarantees are important, but I am not sure about

1. specifying both limits and guarantees for groups and

2. not deriving guarantees from limits.

Guarantees are met by some form of throttling or limiting and hence I think limiting should drive the guarantees

Quoted 17 lines · Bharata B Rao

Quoted 14 lines · Avi Kivity

Quoted 4 lines · Bharata B Rao

But could there be client models where you are required to strictly adhere to the limit within the bandwidth and not provide more (by advancing the bandwidth period) in the presence of idle cycles ?

That's the limit part. I'd like to be able to specify limits and guarantees on the same host and for the same groups; I don't think that works when you advance the bandwidth period.

I think we need to treat guarantees as first-class goals, not something derived from limits (in fact I think guarantees are more useful as they can be used to provide SLAs).

Quoted 4 lines

I agree that guarantees are important, but I am not sure about

1. specifying both limits and guarantees for groups and

Why would you allow specifying a lower bound for cpu usage (a guarantee), and upper bound (a limit), but not both?

Quoted 17 lines · Bharata B Rao

Quoted 14 lines · Avi Kivity

Quoted 4 lines · Bharata B Rao

But could there be client models where you are required to strictly adhere to the limit within the bandwidth and not provide more (by advancing the bandwidth period) in the presence of idle cycles ?

That's the limit part. I'd like to be able to specify limits and guarantees on the same host and for the same groups; I don't think that works when you advance the bandwidth period.

I think we need to treat guarantees as first-class goals, not something derived from limits (in fact I think guarantees are more useful as they can be used to provide SLAs).

Quoted 4 lines

I agree that guarantees are important, but I am not sure about

1. specifying both limits and guarantees for groups and

Why would you allow specifying a lower bound for cpu usage (a guarantee), and upper bound (a limit), but not both?

Quoted 28 lines · Bharata B Rao

Quoted 25 lines · Balbir Singh

Quoted 13 lines · Avi Kivity

Quoted 11 lines · Balbir Singh

Quoted 5 lines

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

OK, I see part of your concern, but I think we could do some optimizations during design. For example if all groups have reached their hard-limit and the system is idle, should we do start a new hard limit interval and restart, so that idleness can be removed. Would that be an acceptable design point?

Quoted 5 lines

I think so. Given guarantees G1..Gn (0 <= Gi <= 1; sum(Gi) <= 1), and a cpu hog running in each group, how would the algorithm divide resources?

As per the matrix calculation, but as soon as we reach an idle point, we redistribute the b/w and start a new quantum so to speak, where all groups are charged up to their hard limits.

Quoted 4 lines

But could there be client models where you are required to strictly adhere to the limit within the bandwidth and not provide more (by advancing the bandwidth period) in the presence of idle cycles ?

That's the limit part. I'd like to be able to specify limits and guarantees on the same host and for the same groups; I don't think that works when you advance the bandwidth period.

I think we need to treat guarantees as first-class goals, not something derived from limits (in fact I think guarantees are more useful as they can be used to provide SLAs).

Quoted 28 lines · Bharata B Rao

Quoted 25 lines · Balbir Singh

Quoted 13 lines · Avi Kivity

Quoted 11 lines · Balbir Singh

Quoted 5 lines

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

OK, I see part of your concern, but I think we could do some optimizations during design. For example if all groups have reached their hard-limit and the system is idle, should we do start a new hard limit interval and restart, so that idleness can be removed. Would that be an acceptable design point?

Quoted 5 lines

I think so. Given guarantees G1..Gn (0 <= Gi <= 1; sum(Gi) <= 1), and a cpu hog running in each group, how would the algorithm divide resources?

As per the matrix calculation, but as soon as we reach an idle point, we redistribute the b/w and start a new quantum so to speak, where all groups are charged up to their hard limits.

Quoted 4 lines

But could there be client models where you are required to strictly adhere to the limit within the bandwidth and not provide more (by advancing the bandwidth period) in the presence of idle cycles ?

That's the limit part. I'd like to be able to specify limits and guarantees on the same host and for the same groups; I don't think that works when you advance the bandwidth period.

I think we need to treat guarantees as first-class goals, not something derived from limits (in fact I think guarantees are more useful as they can be used to provide SLAs).

Quoted 4 lines

That's the limit part. I'd like to be able to specify limits and guarantees on the same host and for the same groups; I don't think that works when you advance the bandwidth period.

Yes, this feature needs to be configurable. But your use case for both limits and guarantees is interesting. We spoke to Peter and he was convinced only of the guarantee use case. Could you please help elaborate your use case, so that we can incorporate it into RFC v2 we send out. Peter is opposed to having hard limits and is convinced that they are not generally useful, so far I seen you and Paul say it is useful, any arguments you have or any +1 from you will help us. Peter I am not back stabbing you :)

Quoted 4 lines

I think we need to treat guarantees as first-class goals, not something derived from limits (in fact I think guarantees are more useful as they can be used to provide SLAs).

Even limits are useful for SLA's since your b/w available changes quite drastically as we add or remove groups. There are other use cases for limits as well

Quoted 2 lines

That's the limit part. I'd like to be able to specify limits and guarantees on the same host and for the same groups; I don't think that works when you advance the bandwidth period.

Yes, this feature needs to be configurable. But your use case for both limits and guarantees is interesting. We spoke to Peter and he was convinced only of the guarantee use case. Could you please help elaborate your use case, so that we can incorporate it into RFC v2 we send out. Peter is opposed to having hard limits and is convinced that they are not generally useful, so far I seen you and Paul say it is useful, any arguments you have or any +1 from you will help us. Peter I am not back stabbing you :)

Quoted 2 lines

I think we need to treat guarantees as first-class goals, not something derived from limits (in fact I think guarantees are more useful as they can be used to provide SLAs).

Even limits are useful for SLA's since your b/w available changes quite drastically as we add or remove groups. There are other use cases for limits as well

Quoted 6 lines · Balbir Singh

Quoted 5 lines

That's the limit part.  I'd like to be able to specify limits and guarantees on the same host and for the same groups; I don't think that works when you advance the bandwidth period.

Quoted 9 lines

Yes, this feature needs to be configurable. But your use case for both limits and guarantees is interesting. We spoke to Peter and he was convinced only of the guarantee use case. Could you please help elaborate your use case, so that we can incorporate it into RFC v2 we send out. Peter is opposed to having hard limits and is convinced that they are not generally useful, so far I seen you and Paul say it is useful, any arguments you have or any +1 from you will help us. Peter I am not back stabbing you :)

I am selling virtual private servers.  A 10% cpu share costs $x/month, and I guarantee you'll get that 10%, or your money back.  On the other hand, I want to limit cpu usage to that 10% (maybe a little more) so people don't buy 10% shares and use 100% on my underutilized servers.  If they want 100%, let them pay for 100%.

Quoted 6 lines

Quoted 5 lines

I think we need to treat guarantees as first-class goals, not something derived from limits (in fact I think guarantees are more useful as they can be used to provide SLAs).

Quoted 3 lines

Even limits are useful for SLA's since your b/w available changes quite drastically as we add or remove groups. There are other use cases for limits as well

SLAs are specified in terms of guarantees on a service, not on limits on others.  If we could use limits to provide guarantees, that would be fine, but it doesn't quite work out.

Quoted 6 lines

I am selling virtual private servers. A 10% cpu share costs $x/month, and I guarantee you'll get that 10%, or your money back. On the other hand, I want to limit cpu usage to that 10% (maybe a little more) so people don't buy 10% shares and use 100% on my underutilized servers. If they want 100%, let them pay for 100%.

Excellent examples, we've covered them in the RFC, could you see if we missed anything in terms of use cases? The real question is do we care enough to build hard limits control into the CFS group scheduler. I believe we should.

Quoted 8 lines

Quoted 4 lines

Even limits are useful for SLA's since your b/w available changes quite drastically as we add or remove groups. There are other use cases for limits as well

SLAs are specified in terms of guarantees on a service, not on limits on others. If we could use limits to provide guarantees, that would be fine, but it doesn't quite work out.

To be honest, I would disagree here, specifically if you start comparing how you would build guarantees in the kernel and compare it with the proposed approach. I don't want to harp on the technicality, but point out the feasibility for people who care for lower end of the guarantee without requiring density. I think the real technical discussion should be on here are the use cases, lets agree on the need for the feature and go ahead and start prototyping the feature.

Quoted 6 lines

I am selling virtual private servers. A 10% cpu share costs $x/month, and I guarantee you'll get that 10%, or your money back. On the other hand, I want to limit cpu usage to that 10% (maybe a little more) so people don't buy 10% shares and use 100% on my underutilized servers. If they want 100%, let them pay for 100%.

Excellent examples, we've covered them in the RFC, could you see if we missed anything in terms of use cases? The real question is do we care enough to build hard limits control into the CFS group scheduler. I believe we should.

Quoted 8 lines

Quoted 4 lines

Even limits are useful for SLA's since your b/w available changes quite drastically as we add or remove groups. There are other use cases for limits as well

SLAs are specified in terms of guarantees on a service, not on limits on others. If we could use limits to provide guarantees, that would be fine, but it doesn't quite work out.

To be honest, I would disagree here, specifically if you start comparing how you would build guarantees in the kernel and compare it with the proposed approach. I don't want to harp on the technicality, but point out the feasibility for people who care for lower end of the guarantee without requiring density. I think the real technical discussion should be on here are the use cases, lets agree on the need for the feature and go ahead and start prototyping the feature.

I am selling virtual private servers. A 10% cpu share costs $x/month, and I guarantee you'll get that 10%, or your money back. On the other hand, I want to limit cpu usage to that 10% (maybe a little more) so people don't buy 10% shares and use 100% on my underutilized servers. If they want 100%, let them pay for 100%.

I am selling virtual private servers. A 10% cpu share costs $x/month, and I guarantee you'll get that 10%, or your money back. On the other hand, I want to limit cpu usage to that 10% (maybe a little more) so people don't buy 10% shares and use 100% on my underutilized servers. If they want 100%, let them pay for 100%.

Quoted 6 lines · Avi Kivity

I am selling virtual private servers. A 10% cpu share costs $x/month, and I guarantee you'll get that 10%, or your money back. On the other hand, I want to limit cpu usage to that 10% (maybe a little more) so people don't buy 10% shares and use 100% on my underutilized servers. If they want 100%, let them pay for 100%.

What about taking a page from the networking folks and specifying cpu like a networking SLA?

Something like "group A is guaranteed X percent (or share) of the cpu, but it is allowed to burst up to Y percent for Z milliseconds"

If a rule of this form was the first-class citizen, it would provide both guarantees, limits, and flexible behaviour.

Quoted 2 lines · Avi Kivity

I am selling virtual private servers. A 10% cpu share costs $x/month, and I guarantee you'll get that 10%, or your money back. On the other hand, I want to limit cpu usage to that 10% (maybe a little more) so people don't buy 10% shares and use 100% on my underutilized servers. If they want 100%, let them pay for 100%.

What about taking a page from the networking folks and specifying cpu like a networking SLA?

Something like "group A is guaranteed X percent (or share) of the cpu, but it is allowed to burst up to Y percent for Z milliseconds"

If a rule of this form was the first-class citizen, it would provide both guarantees, limits, and flexible behaviour.

Quoted 19 lines · Balbir Singh

Quoted 16 lines · Avi Kivity

Quoted 11 lines · Balbir Singh

Quoted 5 lines

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

OK, I see part of your concern, but I think we could do some optimizations during design. For example if all groups have reached their hard-limit and the system is idle, should we do start a new hard limit interval and restart, so that idleness can be removed. Would that be an acceptable design point?

I think so. Given guarantees G1..Gn (0 <= Gi <= 1; sum(Gi) <= 1), and a cpu hog running in each group, how would the algorithm divide resources?

Quoted 3 lines

As per the matrix calculation, but as soon as we reach an idle point, we redistribute the b/w and start a new quantum so to speak, where all groups are charged up to their hard limits.

But could there be client models where you are required to strictly adhere to the limit within the bandwidth and not provide more (by advancing the bandwidth period) in the presence of idle cycles ?

Quoted 3 lines

I think so. Given guarantees G1..Gn (0 <= Gi <= 1; sum(Gi) <= 1), and a cpu hog running in each group, how would the algorithm divide resources?

As per the matrix calculation, but as soon as we reach an idle point, we redistribute the b/w and start a new quantum so to speak, where all groups are charged up to their hard limits.

For your question, if there is a CPU hog running, it would be as per the matrix calculation, since the system has no idle point during the bandwidth period.

Quoted 10 lines · Balbir Singh

Quoted 5 lines

I think so. Given guarantees G1..Gn (0 <= Gi <= 1; sum(Gi) <= 1), and a cpu hog running in each group, how would the algorithm divide resources?

As per the matrix calculation, but as soon as we reach an idle point, we redistribute the b/w and start a new quantum so to speak, where all groups are charged up to their hard limits.

Quoted 4 lines

For your question, if there is a CPU hog running, it would be as per the matrix calculation, since the system has no idle point during the bandwidth period.

So the groups with guarantees get a priority boost. That's not a good side effect.

Quoted 3 lines

So the groups with guarantees get a priority boost. That's not a good side effect.

That happens only in the presence of idle cycles when other groups [with or without guarantees] have nothing useful to do. So how would that matter since there is nothing else to run anyway ?

Quoted 2 lines

So the groups with guarantees get a priority boost. That's not a good side effect.

That happens only in the presence of idle cycles when other groups [with or without guarantees] have nothing useful to do. So how would that matter since there is nothing else to run anyway ?

Quoted 10 lines · Balbir Singh

Quoted 5 lines

I think so. Given guarantees G1..Gn (0 <= Gi <= 1; sum(Gi) <= 1), and a cpu hog running in each group, how would the algorithm divide resources?

As per the matrix calculation, but as soon as we reach an idle point, we redistribute the b/w and start a new quantum so to speak, where all groups are charged up to their hard limits.

Quoted 4 lines

For your question, if there is a CPU hog running, it would be as per the matrix calculation, since the system has no idle point during the bandwidth period.

So the groups with guarantees get a priority boost. That's not a good side effect.

Quoted 12 lines

Quoted 10 lines

How, it works out fine in my calculation

50 + 40 for G2 and G3, make sure that G1 gets 10%, since others are limited to 90%
50 + 40 for G1 and G3, make sure that G2 gets 10%, since others are limited to 90%
50 + 50 for G1 and G2, make sure that G3 gets 0%, since others are limited to 100%

It's fine in that it satisfies the guarantees, but it is deeply suboptimal. If I ran a cpu hog in the first group, while the other two were idle, it would be limited to 50% cpu. On the other hand, if it consumed all 100% cpu it would still satisfy the guarantees (as the other groups are idle).

Quoted 2 lines

The result is that in such a situation, wall clock time would double even though cpu resources are available.

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

Quoted 15 lines · Balbir Singh

Quoted 15 lines

Quoted 9 lines

How, it works out fine in my calculation

50 + 40 for G2 and G3, make sure that G1 gets 10%, since others are limited to 90%
50 + 40 for G1 and G3, make sure that G2 gets 10%, since others are limited to 90%
50 + 50 for G1 and G2, make sure that G3 gets 0%, since others are limited to 100%

It's fine in that it satisfies the guarantees, but it is deeply suboptimal. If I ran a cpu hog in the first group, while the other two were idle, it would be limited to 50% cpu. On the other hand, if it consumed all 100% cpu it would still satisfy the guarantees (as the other groups are idle).

Quoted 9 lines

Quoted 3 lines

The result is that in such a situation, wall clock time would double even though cpu resources are available.

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

Suppose in my example cgroup 1 consumed 100% of the cpu resources and cgroup 2 and 3 were completely idle. All of the guarantees are met (if cgroup 2 is idle, there's no need to give it the 10% cpu time it is guaranteed).

If your only tool to achieve the guarantees is a limit system, then yes, the equation yields the correct results. But given that it yields such inferior results, I think we need to look for a more involved solution.

I think the limits method fits cases where it is difficult to evict a resource (say, disk quotas -- if you want to guarantee 10% of space to cgroups 1, you must limit all others to 90%). But for processor usage, you can evict a cgroup instantly, so nothing prevents a cgroup from consuming all available resources as long as others do not contend for them.

Quoted 15 lines · Balbir Singh

Quoted 15 lines

Quoted 9 lines

How, it works out fine in my calculation

50 + 40 for G2 and G3, make sure that G1 gets 10%, since others are limited to 90%
50 + 40 for G1 and G3, make sure that G2 gets 10%, since others are limited to 90%
50 + 50 for G1 and G2, make sure that G3 gets 0%, since others are limited to 100%

It's fine in that it satisfies the guarantees, but it is deeply suboptimal. If I ran a cpu hog in the first group, while the other two were idle, it would be limited to 50% cpu. On the other hand, if it consumed all 100% cpu it would still satisfy the guarantees (as the other groups are idle).

Quoted 9 lines

Quoted 3 lines

The result is that in such a situation, wall clock time would double even though cpu resources are available.

But then there is no other way to make a *guarantee*, guarantees come at a cost of idling resources, no? Can you show me any other combination that will provide the guarantee and without idling the system for the specified guarantees?

Suppose in my example cgroup 1 consumed 100% of the cpu resources and cgroup 2 and 3 were completely idle. All of the guarantees are met (if cgroup 2 is idle, there's no need to give it the 10% cpu time it is guaranteed).

If your only tool to achieve the guarantees is a limit system, then yes, the equation yields the correct results. But given that it yields such inferior results, I think we need to look for a more involved solution.

I think the limits method fits cases where it is difficult to evict a resource (say, disk quotas -- if you want to guarantee 10% of space to cgroups 1, you must limit all others to 90%). But for processor usage, you can evict a cgroup instantly, so nothing prevents a cgroup from consuming all available resources as long as others do not contend for them.

Quoted 9 lines · Bharata B Rao

Quoted 5 lines

Another way is to place the 8 groups in a container group, and limit that to 80%. But that doesn't work if I want to provide guarantees to several groups.

Hmm why not ? Reduce the guarantee of the container group and provide the same to additional groups ?

This method produces suboptimal results:

$ cgroup-limits 10 10 0
[50.0, 50.0, 40.0]

I want to provide two 10% guaranteed groups and one best-effort group. Using the limits method, no group can now use more than 50% of the resources.  However, having the first group use 90% of the resources does not violate any guarantees, but it not allowed by the solution.

Quoted 11 lines · Bharata B Rao

2. Need for hard limiting CPU resource
--------------------------------------
- Pay-per-use: In enterprise systems that cater to multiple clients/customers where a customer demands a certain share of CPU resources and pays only that, CPU hard limits will be useful to hard limit the customer's job to consume only the specified amount of CPU resource.
- In container based virtualization environments running multiple containers, hard limits will be useful to ensure a container doesn't exceed its CPU entitlement.
- Hard limits can be used to provide guarantees.

How can hard limits provide guarantees?

Let's take an example where I have 1 group that I wish to guarantee a 20% share of the cpu, and anther 8 groups with no limits or guarantees.

One way to achieve the guarantee is to hard limit each of the 8 other groups to 10%; the sum total of the limits is 80%, leaving 20% for the guarantee group. The downside is the arbitrary limit imposed on the other groups.

Another way is to place the 8 groups in a container group, and limit that to 80%. But that doesn't work if I want to provide guarantees to several groups.

- Hard limits can be used to provide guarantees.

Quoted 2 lines · Bharata B Rao

- Hard limits can be used to provide guarantees.

This claim (and the subsequent long thread it generated on how limits can provide guarantees) confused me a bit.

Why do we need limits to provide guarantees when we can already provide guarantees via shares?

Suppose 10 cgroups each want 10% of the machine's CPU. We can just give each cgroup an equal share, and they're guaranteed 10% if they try to use it; if they don't use it, other cgroups can get access to the idle cycles.

Quoted 2 lines · Bharata B Rao

- Hard limits can be used to provide guarantees.

This claim (and the subsequent long thread it generated on how limits can provide guarantees) confused me a bit.

Why do we need limits to provide guarantees when we can already provide guarantees via shares?

Suppose 10 cgroups each want 10% of the machine's CPU. We can just give each cgroup an equal share, and they're guaranteed 10% if they try to use it; if they don't use it, other cgroups can get access to the idle cycles.

Suppose cgroup A wants a guarantee of 50% and two others, B and C, want guarantees of 15% each; give A 50 shares and B and C 15 shares each. In this case, if they all run flat out they'll get 62%/19%/19%, which is within their SLA.

That's not to say that hard limits can't be useful in their own right
- e.g. for providing reproducible loadtesting conditions by controlling how much CPU a service can use during the load test. But I don't see why using them to implement guarantees is either necessary or desirable.

(Unless I'm missing some crucial point ...)

This claim (and the subsequent long thread it generated on how limits can provide guarantees) confused me a bit.

Why do we need limits to provide guarantees when we can already provide guarantees via shares?

Quoted 3 lines · Bharata B Rao

- Hard limits can be used to provide guarantees.

This claim (and the subsequent long thread it generated on how limits can provide guarantees) confused me a bit.

Why do we need limits to provide guarantees when we can already provide guarantees via shares?

Suppose 10 cgroups each want 10% of the machine's CPU. We can just give each cgroup an equal share, and they're guaranteed 10% if they try to use it; if they don't use it, other cgroups can get access to the idle cycles.

Suppose cgroup A wants a guarantee of 50% and two others, B and C, want guarantees of 15% each; give A 50 shares and B and C 15 shares each. In this case, if they all run flat out they'll get 62%/19%/19%, which is within their SLA.

That's not to say that hard limits can't be useful in their own right
- e.g. for providing reproducible loadtesting conditions by controlling how much CPU a service can use during the load test. But I don't see why using them to implement guarantees is either necessary or desirable.

(Unless I'm missing some crucial point ...)

Quoted 2 lines · Bharata B Rao

- Hard limits can be used to provide guarantees.

This claim (and the subsequent long thread it generated on how limits can provide guarantees) confused me a bit.

Why do we need limits to provide guarantees when we can already provide guarantees via shares?

Suppose 10 cgroups each want 10% of the machine's CPU. We can just give each cgroup an equal share, and they're guaranteed 10% if they try to use it; if they don't use it, other cgroups can get access to the idle cycles.

Quoted 5 lines

Suppose 10 cgroups each want 10% of the machine's CPU. We can just give each cgroup an equal share, and they're guaranteed 10% if they try to use it; if they don't use it, other cgroups can get access to the idle cycles.

Now if 11th group with same shares comes in, then each group will now get 9% of CPU and that 10% guarantee breaks.

Quoted 5 lines

Suppose 10 cgroups each want 10% of the machine's CPU. We can just give each cgroup an equal share, and they're guaranteed 10% if they try to use it; if they don't use it, other cgroups can get access to the idle cycles.

Now if 11th group with same shares comes in, then each group will now get 9% of CPU and that 10% guarantee breaks.

Quoted 8 lines · Bharata B Rao

Quoted 5 lines

Suppose 10 cgroups each want 10% of the machine's CPU. We can just give each cgroup an equal share, and they're guaranteed 10% if they try to use it; if they don't use it, other cgroups can get access to the idle cycles.

Now if 11th group with same shares comes in, then each group will now get 9% of CPU and that 10% guarantee breaks.

So you're trying to guarantee 11 cgroups that they can each get 10% of the CPU? That's called over-committing, and while there's nothing wrong with doing that if you're confident that they'll not al need their 10% at the same time, there's no way to *guarantee* them all 10%. You can guarantee them all 9% and hope the extra 1% is spare for those that need it (over-committing), or you can guarantee 10 of them 10% and give the last one 0 shares.

How would you propose to guarantee 11 cgroups each 10% of the CPU using hard limits?

Quoted 8 lines · Bharata B Rao

Quoted 5 lines

Suppose 10 cgroups each want 10% of the machine's CPU. We can just give each cgroup an equal share, and they're guaranteed 10% if they try to use it; if they don't use it, other cgroups can get access to the idle cycles.

Now if 11th group with same shares comes in, then each group will now get 9% of CPU and that 10% guarantee breaks.

So you're trying to guarantee 11 cgroups that they can each get 10% of the CPU? That's called over-committing, and while there's nothing wrong with doing that if you're confident that they'll not al need their 10% at the same time, there's no way to *guarantee* them all 10%. You can guarantee them all 9% and hope the extra 1% is spare for those that need it (over-committing), or you can guarantee 10 of them 10% and give the last one 0 shares.

How would you propose to guarantee 11 cgroups each 10% of the CPU using hard limits?

Quoted 11 lines

Quoted 2 lines

Now if 11th group with same shares comes in, then each group will now get 9% of CPU and that 10% guarantee breaks.

So you're trying to guarantee 11 cgroups that they can each get 10% of the CPU? That's called over-committing, and while there's nothing wrong with doing that if you're confident that they'll not all need their 10% at the same time, there's no way to *guarantee* them all 10%. You can guarantee them all 9% and hope the extra 1% is spare for those that need it (over-committing), or you can guarantee 10 of them 10% and give the last one 0 shares.

Quoted 3 lines

How would you propose to guarantee 11 cgroups each 10% of the CPU using hard limits?

You cannot guarantee 10% to 11 groups on any system (unless I am missing something). The sum of guarantees cannot exceed 100%.

Quoted 2 lines

Now if 11th group with same shares comes in, then each group will now get 9% of CPU and that 10% guarantee breaks.

Quoted 11 lines

Quoted 2 lines

Now if 11th group with same shares comes in, then each group will now get 9% of CPU and that 10% guarantee breaks.

So you're trying to guarantee 11 cgroups that they can each get 10% of the CPU? That's called over-committing, and while there's nothing wrong with doing that if you're confident that they'll not all need their 10% at the same time, there's no way to *guarantee* them all 10%. You can guarantee them all 9% and hope the extra 1% is spare for those that need it (over-committing), or you can guarantee 10 of them 10% and give the last one 0 shares.

Quoted 3 lines

How would you propose to guarantee 11 cgroups each 10% of the CPU using hard limits?

You cannot guarantee 10% to 11 groups on any system (unless I am missing something). The sum of guarantees cannot exceed 100%.

Quoted 2 lines

Now if 11th group with same shares comes in, then each group will now get 9% of CPU and that 10% guarantee breaks.

Quoted 11 lines · Dhaval Giani

Quoted 11 lines

Quoted 2 lines

Now if 11th group with same shares comes in, then each group will now get 9% of CPU and that 10% guarantee breaks.

So you're trying to guarantee 11 cgroups that they can each get 10% of the CPU? That's called over-committing, and while there's nothing wrong with doing that if you're confident that they'll not all need their 10% at the same time, there's no way to *guarantee* them all 10%. You can guarantee them all 9% and hope the extra 1% is spare for those that need it (over-committing), or you can guarantee 10 of them 10% and give the last one 0 shares.

Quoted 6 lines

Quoted 3 lines

How would you propose to guarantee 11 cgroups each 10% of the CPU using hard limits?

You cannot guarantee 10% to 11 groups on any system (unless I am missing something). The sum of guarantees cannot exceed 100%.

That's exactly my point. I was trying to counter Bharata's statement, which was:

Quoted 2 lines

Quoted 2 lines

Now if 11th group with same shares comes in, then each group will now get 9% of CPU and that 10% guarantee breaks.

which seemed to be implying that this was a drawback of using shares to implement guarantees.

Quoted 11 lines · Dhaval Giani

Quoted 11 lines

Quoted 2 lines

Now if 11th group with same shares comes in, then each group will now get 9% of CPU and that 10% guarantee breaks.

So you're trying to guarantee 11 cgroups that they can each get 10% of the CPU? That's called over-committing, and while there's nothing wrong with doing that if you're confident that they'll not all need their 10% at the same time, there's no way to *guarantee* them all 10%. You can guarantee them all 9% and hope the extra 1% is spare for those that need it (over-committing), or you can guarantee 10 of them 10% and give the last one 0 shares.

Quoted 6 lines

Quoted 3 lines

How would you propose to guarantee 11 cgroups each 10% of the CPU using hard limits?

You cannot guarantee 10% to 11 groups on any system (unless I am missing something). The sum of guarantees cannot exceed 100%.

That's exactly my point. I was trying to counter Bharata's statement, which was:

Quoted 2 lines

Quoted 2 lines

Now if 11th group with same shares comes in, then each group will now get 9% of CPU and that 10% guarantee breaks.

which seemed to be implying that this was a drawback of using shares to implement guarantees.

I think we are focusing on the wrong use case here. Guarantees is just a useful side-effect we get by using hard limits. I think the more important use case is where the provider wants to limit the amount of time a user gets (such as in a cloud).

Maybe we should direct our attention in solving that problem? :)

Quoted 7 lines · Dhaval Giani

I think we are focusing on the wrong use case here. Guarantees is just a useful side-effect we get by using hard limits. I think the more important use case is where the provider wants to limit the amount of time a user gets (such as in a cloud).

Maybe we should direct our attention in solving that problem? :)

Yes, that case and the "predictable load test behaviour" case are both good reasons for hard limits.

Paul

Quoted 7 lines · Dhaval Giani

I think we are focusing on the wrong use case here. Guarantees is just a useful side-effect we get by using hard limits. I think the more important use case is where the provider wants to limit the amount of time a user gets (such as in a cloud).

Maybe we should direct our attention in solving that problem? :)

Yes, that case and the "predictable load test behaviour" case are both good reasons for hard limits.

Paul

I think we are focusing on the wrong use case here. Guarantees is just a useful side-effect we get by using hard limits. I think the more important use case is where the provider wants to limit the amount of time a user gets (such as in a cloud).

Maybe we should direct our attention in solving that problem? :)

Quoted 2 lines · Bharata B Rao

- Hard limits can be used to provide guarantees.

This claim (and the subsequent long thread it generated on how limits can provide guarantees) confused me a bit.

Why do we need limits to provide guarantees when we can already provide guarantees via shares?

Suppose 10 cgroups each want 10% of the machine's CPU. We can just give each cgroup an equal share, and they're guaranteed 10% if they try to use it; if they don't use it, other cgroups can get access to the idle cycles.

Suppose cgroup A wants a guarantee of 50% and two others, B and C, want guarantees of 15% each; give A 50 shares and B and C 15 shares each. In this case, if they all run flat out they'll get 62%/19%/19%, which is within their SLA.

That's not to say that hard limits can't be useful in their own right
- e.g. for providing reproducible loadtesting conditions by controlling how much CPU a service can use during the load test. But I don't see why using them to implement guarantees is either necessary or desirable.

(Unless I'm missing some crucial point ...)

The important scenario I have is adding and removing groups.

Consider 10 cgroups with shares of 10 each, what if 5 new are created with the same shares? We now start getting 100/15, even though we did not change our shares.

Quoted 6 lines · Balbir Singh

The important scenario I have is adding and removing groups.

Consider 10 cgroups with shares of 10 each, what if 5 new are created with the same shares? We now start getting 100/15, even though we did not change our shares.

Are you assuming that arbitrary users can create new cgroups whenever they like, with whatever shares they like? In that situation, how would you use hard limits to provide guarantees? Presumably if the user could create a cgroup with an arbitrary share, they could create one with an arbitrary hard limit too.

Can you explain a bit more about how you're envisaging cgroups being created, and how their shares and hard limits would get set? I was working on the assumption that (for any sub-tree of the CFS hierarchy) there's a single managing entity that gets to decide the shares given to the cgroups within that tree. That managing entity would be responsible for ensuring that the shares given out allowed guarantees to be met (or alternatively, that the probability of violating those guarantees based on the shares given out was within some tolerance threshold).

Quoted 6 lines · Balbir Singh

The important scenario I have is adding and removing groups.

Consider 10 cgroups with shares of 10 each, what if 5 new are created with the same shares? We now start getting 100/15, even though we did not change our shares.

Are you assuming that arbitrary users can create new cgroups whenever they like, with whatever shares they like? In that situation, how would you use hard limits to provide guarantees? Presumably if the user could create a cgroup with an arbitrary share, they could create one with an arbitrary hard limit too.

Can you explain a bit more about how you're envisaging cgroups being created, and how their shares and hard limits would get set? I was working on the assumption that (for any sub-tree of the CFS hierarchy) there's a single managing entity that gets to decide the shares given to the cgroups within that tree. That managing entity would be responsible for ensuring that the shares given out allowed guarantees to be met (or alternatively, that the probability of violating those guarantees based on the shares given out was within some tolerance threshold).

The point is that there is no single control entity for creating groups. if run a solution, it might create groups without telling the user. No one is arbitrating, not even libcgroup. What if someone changes the cpuset assignment and moves CPUS x to y in an exclusive cpuset all of a sudden. How do we arbitrate?

The point is that there is no single control entity for creating groups. if run a solution, it might create groups without telling the user. No one is arbitrating, not even libcgroup. What if someone changes the cpuset assignment and moves CPUS x to y in an exclusive cpuset all of a sudden. How do we arbitrate?

What about applications running as root, that can create their own groups? How about multiple instances of the same application started? Do applications need to know that creating a group will hurt guarantees provided to others?

What about applications running as root, that can create their own groups? How about multiple instances of the same application started? Do applications need to know that creating a group will hurt guarantees provided to others?

This claim (and the subsequent long thread it generated on how limits can provide guarantees) confused me a bit.

Why do we need limits to provide guarantees when we can already provide guarantees via shares?

I think the interval over which we need guarantee matters here. Shares can generally provide guaranteed share of resource over longer (sometimes minutes) intervals. For high-priority bursty workloads, the latency in achieving guaranteed resource usage matters.

By having hard-limits, we are
"reserving" (potentially idle) slots where the high-priority group can run and claim its guaranteed share almost immediately.

Quoted 5 lines · Paul Menage

This claim (and the subsequent long thread it generated on how limits can provide guarantees) confused me a bit.

Why do we need limits to provide guarantees when we can already provide guarantees via shares?

I think the interval over which we need guarantee matters here. Shares can generally provide guaranteed share of resource over longer (sometimes minutes) intervals. For high-priority bursty workloads, the latency in achieving guaranteed resource usage matters. By having hard-limits, we are "reserving" (potentially idle) slots where the high-priority group can run and claim its guaranteed share almost immediately.

I think the interval over which we need guarantee matters here. Shares can generally provide guaranteed share of resource over longer (sometimes minutes) intervals. For high-priority bursty workloads, the latency in achieving guaranteed resource usage matters.

By having hard-limits, we are
"reserving" (potentially idle) slots where the high-priority group can run and claim its guaranteed share almost immediately.

Well yes, it's true that you *could* just enforce shares over a granularity of minutes, and limits over a granularity of milliseconds. But why would you? It could well make sense that you can adjust the granularity over which shares are enforced - e.g. for batch jobs, only enforcing over minutes or tens of seconds might be fine. But if you're

doing the fine-grained accounting and scheduling required for the tight hard limit enforcement, it doesn't seem as though it should be much harder to enforce shares at the same granularity for those cgroups that matter. In fact I thought that's what CFS already did - updated the virtual time accounting at each context switch, and picked the runnable child with the oldest virtual time. (Maybe someone like Ingo or Peter who's more familiar than I with the CFS implementation could comment here?)

Quoted 3 lines

By having hard-limits, we are
"reserving" (potentially idle) slots where the high-priority group can run and claim its guaranteed share almost immediately.

Quoted 5 lines · Paul Menage

Well yes, it's true that you *could* just enforce shares over a granularity of minutes, and limits over a granularity of milliseconds. But why would you? It could well make sense that you can adjust the granularity over which shares are enforced - e.g. for batch jobs, only enforcing over minutes or tens of seconds might be fine. But if you're

Quoted 8 lines

doing the fine-grained accounting and scheduling required for the tight hard limit enforcement, it doesn't seem as though it should be much harder to enforce shares at the same granularity for those cgroups that matter. In fact I thought that's what CFS already did - updated the virtual time accounting at each context switch, and picked the runnable child with the oldest virtual time. (Maybe someone like Ingo or Peter who's more familiar than I with the CFS implementation could comment here?)

Using shares to guarantee resources over short period (<2-3 seconds) works just well on a single CPU. The complexity is with multi-cpu case, where CFS can take a long time to converge to a fair point. This is because fairness is based on rebalancing tasks equally across all CPUs.

For something like 4 tasks on 4 CPUs, it will converge pretty quickly (2-3 seconds):

[top o/p refreshed every 2sec on 2.6.30-rc5-tip]

14753 vatsa     20   0 63812 1072  924 R 99.9  0.0   0:39.54 hog 14754 vatsa     20   0 63812 1072  924 R 99.9  0.0   0:38.69 hog 14756 vatsa     20   0 63812 1076  924 R 99.9  0.0   0:38.27 hog 14755 vatsa     20   0 63812 1072  924 R 99.6  0.0   0:38.27 hog

whereas for something like 5 tasks on 4 CPUs, it will take a sufficiently longer time (>30 seconds)

[top o/p refreshed every 2sec]:

14754 vatsa     20   0 63812 1072  924 R 86.0  0.0   2:06.45 hog 14766 vatsa     20   0 63812 1072  924 R 83.0  0.0   0:07.95 hog 14756 vatsa     20   0 63812 1076  924 R 81.7  0.0   2:06.48 hog 14753 vatsa     20   0 63812 1072  924 R 78.7  0.0   2:07.10 hog 14755 vatsa     20   0 63812 1072  924 R 69.4  0.0   2:05.62 hog

[top o/p refreshed every 120sec]:

14766 vatsa     20   0 63812 1072  924 R 90.1  0.0   5:57.22 hog 14755 vatsa     20   0 63812 1072  924 R 84.8  0.0   8:01.61 hog 14754 vatsa     20   0 63812 1072  924 R 77.3  0.0   7:52.04 hog 14753 vatsa     20   0 63812 1072  924 R 74.1  0.0   7:29.01 hog 14756 vatsa     20   0 63812 1076  924 R 73.5  0.0   7:34.69 hog

[Note that even over 2min, we haven't achieved perfect fairness]

Quoted 3 lines

Quoted 3 lines

By having hard-limits, we are
"reserving" (potentially idle) slots where the high-priority group can run and claim its guaranteed share almost immediately.

On further thinking, this is not as simple as that. In above example of 5 tasks on 4 CPUs, we could cap each task at a hard limit of 80% (4 CPUs/5 tasks), which is still not sufficient to ensure that each task gets the perfect fairness of 80%! Not just that, hard-limit for a group (on each CPU) will have to be adjusted based on its task distribution. For ex: a group that has a hard-limit of 25% on a 4-cpu system and that has a single task, is entitled to claim a whole CPU. So the per-cpu hard-limit for the group should be 100% on whatever CPU the task is running. This adjustment of per-cpu hard-limit should happen whenever the task distribution of the group across CPUs change - which in theory would require you to monitor every task exit/migration event and readjust limits, making it very complex and high-overhead.

Balbir,
       I dont think guarantee can be met easily thr' hard-limits in case of CPU resource. Atleast its not as straightforward as in case of memory!

Quoted 5 lines · Paul Menage

Well yes, it's true that you *could* just enforce shares over a granularity of minutes, and limits over a granularity of milliseconds. But why would you? It could well make sense that you can adjust the granularity over which shares are enforced - e.g. for batch jobs, only enforcing over minutes or tens of seconds might be fine. But if you're

Quoted 8 lines

doing the fine-grained accounting and scheduling required for the tight hard limit enforcement, it doesn't seem as though it should be much harder to enforce shares at the same granularity for those cgroups that matter. In fact I thought that's what CFS already did - updated the virtual time accounting at each context switch, and picked the runnable child with the oldest virtual time. (Maybe someone like Ingo or Peter who's more familiar than I with the CFS implementation could comment here?)

Using shares to guarantee resources over short period (<2-3 seconds) works just well on a single CPU. The complexity is with multi-cpu case, where CFS can take a long time to converge to a fair point. This is because fairness is based on rebalancing tasks equally across all CPUs.

For something like 4 tasks on 4 CPUs, it will converge pretty quickly (2-3 seconds):

[top o/p refreshed every 2sec on 2.6.30-rc5-tip]

14753 vatsa     20   0 63812 1072  924 R 99.9  0.0   0:39.54 hog 14754 vatsa     20   0 63812 1072  924 R 99.9  0.0   0:38.69 hog 14756 vatsa     20   0 63812 1076  924 R 99.9  0.0   0:38.27 hog 14755 vatsa     20   0 63812 1072  924 R 99.6  0.0   0:38.27 hog

whereas for something like 5 tasks on 4 CPUs, it will take a sufficiently longer time (>30 seconds)

[top o/p refreshed every 2sec]:

14754 vatsa     20   0 63812 1072  924 R 86.0  0.0   2:06.45 hog 14766 vatsa     20   0 63812 1072  924 R 83.0  0.0   0:07.95 hog 14756 vatsa     20   0 63812 1076  924 R 81.7  0.0   2:06.48 hog 14753 vatsa     20   0 63812 1072  924 R 78.7  0.0   2:07.10 hog 14755 vatsa     20   0 63812 1072  924 R 69.4  0.0   2:05.62 hog

[top o/p refreshed every 120sec]:

14766 vatsa     20   0 63812 1072  924 R 90.1  0.0   5:57.22 hog 14755 vatsa     20   0 63812 1072  924 R 84.8  0.0   8:01.61 hog 14754 vatsa     20   0 63812 1072  924 R 77.3  0.0   7:52.04 hog 14753 vatsa     20   0 63812 1072  924 R 74.1  0.0   7:29.01 hog 14756 vatsa     20   0 63812 1076  924 R 73.5  0.0   7:34.69 hog

[Note that even over 2min, we haven't achieved perfect fairness]

Quoted 3 lines

Quoted 3 lines

By having hard-limits, we are
"reserving" (potentially idle) slots where the high-priority group can run and claim its guaranteed share almost immediately.

On further thinking, this is not as simple as that. In above example of 5 tasks on 4 CPUs, we could cap each task at a hard limit of 80% (4 CPUs/5 tasks), which is still not sufficient to ensure that each task gets the perfect fairness of 80%! Not just that, hard-limit for a group (on each CPU) will have to be adjusted based on its task distribution. For ex: a group that has a hard-limit of 25% on a 4-cpu system and that has a single task, is entitled to claim a whole CPU. So the per-cpu hard-limit for the group should be 100% on whatever CPU the task is running. This adjustment of per-cpu hard-limit should happen whenever the task distribution of the group across CPUs change - which in theory would require you to monitor every task exit/migration event and readjust limits, making it very complex and high-overhead.

Balbir,
       I dont think guarantee can be met easily thr' hard-limits in case of CPU resource. Atleast its not as straightforward as in case of memory!

Quoted 4 lines

On further thinking, this is not as simple as that. In above example of 5 tasks on 4 CPUs, we could cap each task at a hard limit of 80% (4 CPUs/5 tasks), which is still not sufficient to ensure that each task gets the perfect fairness of 80%! Not just that, hard-limit

Quoted 9 lines

for a group (on each CPU) will have to be adjusted based on its task distribution. For ex: a group that has a hard-limit of 25% on a 4-cpu system and that has a single task, is entitled to claim a whole CPU. So the per-cpu hard-limit for the group should be 100% on whatever CPU the task is running. This adjustment of per-cpu hard-limit should happen whenever the task distribution of the group across CPUs change - which in theory would require you to monitor every task exit/migration event and readjust limits, making it very complex and high-overhead.

We already do that for shares right? I mean instead of 25% hard limit, if the group had 25% of the shares the same thing would apply - no?

Quoted 4 lines

Balbir,
       I dont think guarantee can be met easily thr' hard-limits in case of CPU resource. Atleast its not as straightforward as in case of memory!

OK, based on the discussion - leaving implementation issues out, speaking of whether it is possible to implement guarantees using shares? My answer would be

1. Yes - but then the hard limits will prevent you and can cause idle times, some of those can be handled in the implementation. There might be other fairness and SMP concerns about the accuracy of the fairness, thank you for that data.
2. We'll update the RFC (second version) with the findings and send it out, so that the expectations are clearer
3. From what I've read and seen there seems to be no strong objection to hard limits, but some reservations (based on 1) about using them for guarantees and our RFC will reflect that.

Do you agree?

Quoted 4 lines

On further thinking, this is not as simple as that. In above example of 5 tasks on 4 CPUs, we could cap each task at a hard limit of 80% (4 CPUs/5 tasks), which is still not sufficient to ensure that each task gets the perfect fairness of 80%! Not just that, hard-limit

Quoted 9 lines

for a group (on each CPU) will have to be adjusted based on its task distribution. For ex: a group that has a hard-limit of 25% on a 4-cpu system and that has a single task, is entitled to claim a whole CPU. So the per-cpu hard-limit for the group should be 100% on whatever CPU the task is running. This adjustment of per-cpu hard-limit should happen whenever the task distribution of the group across CPUs change - which in theory would require you to monitor every task exit/migration event and readjust limits, making it very complex and high-overhead.

We already do that for shares right? I mean instead of 25% hard limit, if the group had 25% of the shares the same thing would apply - no?

Quoted 4 lines

Balbir,
       I dont think guarantee can be met easily thr' hard-limits in case of CPU resource. Atleast its not as straightforward as in case of memory!

OK, based on the discussion - leaving implementation issues out, speaking of whether it is possible to implement guarantees using shares? My answer would be

1. Yes - but then the hard limits will prevent you and can cause idle times, some of those can be handled in the implementation. There might be other fairness and SMP concerns about the accuracy of the fairness, thank you for that data.
2. We'll update the RFC (second version) with the findings and send it out, so that the expectations are clearer
3. From what I've read and seen there seems to be no strong objection to hard limits, but some reservations (based on 1) about using them for guarantees and our RFC will reflect that.

Do you agree?

Quoted 5 lines · Paul Menage

This claim (and the subsequent long thread it generated on how limits can provide guarantees) confused me a bit.

Why do we need limits to provide guarantees when we can already provide guarantees via shares?

I think the interval over which we need guarantee matters here. Shares can generally provide guaranteed share of resource over longer (sometimes minutes) intervals. For high-priority bursty workloads, the latency in achieving guaranteed resource usage matters. By having hard-limits, we are "reserving" (potentially idle) slots where the high-priority group can run and claim its guaranteed share almost immediately.

Quoted 3 lines · Bharata B Rao

- Hard limits can be used to provide guarantees.

This claim (and the subsequent long thread it generated on how limits can provide guarantees) confused me a bit.

Why do we need limits to provide guarantees when we can already provide guarantees via shares?

Suppose 10 cgroups each want 10% of the machine's CPU. We can just give each cgroup an equal share, and they're guaranteed 10% if they try to use it; if they don't use it, other cgroups can get access to the idle cycles.

Suppose cgroup A wants a guarantee of 50% and two others, B and C, want guarantees of 15% each; give A 50 shares and B and C 15 shares each. In this case, if they all run flat out they'll get 62%/19%/19%, which is within their SLA.

That's not to say that hard limits can't be useful in their own right
- e.g. for providing reproducible loadtesting conditions by controlling how much CPU a service can use during the load test. But I don't see why using them to implement guarantees is either necessary or desirable.

(Unless I'm missing some crucial point ...)

Quoted 29 lines · Paul Menage

Quoted 3 lines · Bharata B Rao

- Hard limits can be used to provide guarantees.

This claim (and the subsequent long thread it generated on how limits can provide guarantees) confused me a bit.

Why do we need limits to provide guarantees when we can already provide guarantees via shares?

Suppose 10 cgroups each want 10% of the machine's CPU. We can just give each cgroup an equal share, and they're guaranteed 10% if they try to use it; if they don't use it, other cgroups can get access to the idle cycles.

Suppose cgroup A wants a guarantee of 50% and two others, B and C, want guarantees of 15% each; give A 50 shares and B and C 15 shares each. In this case, if they all run flat out they'll get 62%/19%/19%, which is within their SLA.

That's not to say that hard limits can't be useful in their own right
- e.g. for providing reproducible loadtesting conditions by controlling how much CPU a service can use during the load test. But I don't see why using them to implement guarantees is either necessary or desirable.

Quoted 2 lines

(Unless I'm missing some crucial point ...)

How many shares does a cgroup with a 0% guarantee get?

Quoted 29 lines · Paul Menage

Quoted 3 lines · Bharata B Rao

- Hard limits can be used to provide guarantees.

This claim (and the subsequent long thread it generated on how limits can provide guarantees) confused me a bit.

Why do we need limits to provide guarantees when we can already provide guarantees via shares?

Suppose 10 cgroups each want 10% of the machine's CPU. We can just give each cgroup an equal share, and they're guaranteed 10% if they try to use it; if they don't use it, other cgroups can get access to the idle cycles.

Suppose cgroup A wants a guarantee of 50% and two others, B and C, want guarantees of 15% each; give A 50 shares and B and C 15 shares each. In this case, if they all run flat out they'll get 62%/19%/19%, which is within their SLA.

That's not to say that hard limits can't be useful in their own right
- e.g. for providing reproducible loadtesting conditions by controlling how much CPU a service can use during the load test. But I don't see why using them to implement guarantees is either necessary or desirable.

Quoted 2 lines

(Unless I'm missing some crucial point ...)

How many shares does a cgroup with a 0% guarantee get?

Shares cannot be used to provide guarantees. All they decide is what propotion groups can get CPU time. (yes, shares is a bad name, weight shows the intent better).

Shares cannot be used to provide guarantees. All they decide is what propotion groups can get CPU time. (yes, shares is a bad name, weight shows the intent better).

4. Existing solutions
---------------------
- Both Linux-VServer and OpenVZ virtualization solutions support CPU hard limiting.
- Per task limit can be enforced using rlimits, but it is not rate based.