cgroup, RT reservation per core(s)?
Hi,
I would like to have in a quad-core, 2 cores totally (100%) dedicated to RT tasks (SCHED_RR and SCHED_FIFO) and the other 2 cores with normal behavior, better said, allowing SCHED_OTHER & SCHED_FIFO & SCHED_RR tasks but still with a RT reservation. Follows an example:
# Setup first domain (cpu 0,1)
echo 0-1 > /dev/cgroup/0/cpuset.cpus
echo 0 > /dev/cgroup/0/cpuset.mems
# Setup RT bandwidth for firstdomain (80% for RT, 20% others) echo 1000000 > /dev/cgroup/0/cpu.rt_period_us
echo 800000 > /dev/cgroup/0/cpu.rt_runtime_us
# Setup second domain (cpu 2,3)
mkdir /dev/cgroup/1
echo 2-3 > /dev/cgroup/1/cpuset.cpus
echo 0 > /dev/cgroup/1/cpuset.mems
# Setup RT bandwidth for second domain (100% for RT) echo 1000000 > /dev/cgroup/1/cpu.rt_period_us echo 1000000 > /dev/cgroup/1/cpu.rt_runtime_us
Is there anyway for doing this?
Thanks,
Rol
Quoted 21 lines · Rolando Martins
Hi,
I would like to have in a quad-core, 2 cores totally (100%) dedicated to RT tasks (SCHED_RRandSCHED_FIFO) and the other 2 cores with normal behavior, better said, allowingSCHED_OTHER&SCHED_FIFO&SCHED_RRtasks but still with a RT reservation. Follows an example:# Setup first domain (cpu 0,1)
echo 0-1 >/dev/cgroup/0/cpuset.cpus
echo 0 >/dev/cgroup/0/cpuset.mems# Setup RT bandwidth for firstdomain (80% for RT, 20% others) echo 1000000 >
/dev/cgroup/0/cpu.rt_period_us
echo 800000 >/dev/cgroup/0/cpu.rt_runtime_us# Setup second domain (cpu 2,3)
mkdir/dev/cgroup/1
echo 2-3 >/dev/cgroup/1/cpuset.cpus
echo 0 >/dev/cgroup/1/cpuset.mems
Quoted 5 lines
# Setup RT bandwidth for second domain (100% for RT) echo 1000000 >
/dev/cgroup/1/cpu.rt_period_usecho 1000000 >/dev/cgroup/1/cpu.rt_runtime_usIs there anyway for doing this?
Nope, but why do you need bandwidth groups if all you want is a full cpu?
Just the cpuset should be plenty
Thanks for the quick reply.
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Rolando Martins
Hi,
I would like to have in a quad-core, 2 cores totally (100%) dedicated to RT tasks (SCHED_RRandSCHED_FIFO) and the other 2 cores with normal behavior, better said, allowingSCHED_OTHER&SCHED_FIFO&SCHED_RRtasks but still with a RT reservation. Follows an example:# Setup first domain (cpu 0,1)
echo 0-1 >/dev/cgroup/0/cpuset.cpus
echo 0 >/dev/cgroup/0/cpuset.mems# Setup RT bandwidth for firstdomain (80% for RT, 20% others) echo 1000000 >
/dev/cgroup/0/cpu.rt_period_us
echo 800000 >/dev/cgroup/0/cpu.rt_runtime_us# Setup second domain (cpu 2,3)
mkdir/dev/cgroup/1
echo 2-3 >/dev/cgroup/1/cpuset.cpus
echo 0 >/dev/cgroup/1/cpuset.mems# Setup RT bandwidth for second domain (100% for RT) echo 1000000 >
/dev/cgroup/1/cpu.rt_period_usecho 1000000 >/dev/cgroup/1/cpu.rt_runtime_us
Quoted 6 lines
Quoted 1 line
Is there anyway for doing this?
Nope, but why do you need bandwidth groups if all you want is a full cpu?
Just the cpuset should be plenty
You have a point;)
Quoted 2 lines
I should have elaborated this more:
root
----|----
| |
(0.5 mem) 0 1 (100% rt, 0.5 mem)
---------
| | |
2 3 4 (33% rt for each group, 33% memper group(0.165))
Rol
Quoted 11 lines · Rolando Martins
I should have elaborated this more:
root ----|---- | | (0.5 mem) 0 1 (100% rt, 0.5 mem) --------- | | | 2 3 4 (33% rt for each group, 33% memper group(0.165))
Rol
Right, i think this can be done.
You would indeed need cpusets and sched-cgroups.
Split the machine in 2 using cpusets.
___R___
/ \
A B
Where R is the root cpuset, and A and B are the siblings. Assign A one half the cpus, and B the other half. Disable load-balancing on R.
Then using sched cgroups create the hierarchy
____1____
/ | \
2 3 4
Where 1 can be the root group if you like.
Assign 1 a utilization limit of 100%, and 2,3 and 4 a utilization limit of 33% each.
Then place the tasks that get 100% cputime on your 2 cpus in cpuset A and sched group 1.
Place your other tasks in B,{2-4} respectively.
The reason this works is that bandwidth distribution is sched domain wide, and by disabling load-balancing on R, you split the schedule domain.
I've never actually tried anything like this, let me know if it works ;-)
Quoted 53 lines · Peter Zijlstra
Quoted 11 lines · Rolando Martins
I should have elaborated this more:
root ----|---- | | (0.5 mem) 0 1 (100% rt, 0.5 mem) --------- | | | 2 3 4 (33% rt for each group, 33% memper group(0.165))
RolRight, i think this can be done.
You would indeed need cpusets and sched-cgroups.
Split the machine in 2 using cpusets.
___R___ / \ A BWhere R is the root cpuset, and A and B are the siblings. Assign A one half the cpus, and B the other half. Disable load-balancing on R.
Then using sched cgroups create the hierarchy
____1____
/ | \
2 3 4Where 1 can be the root group if you like.
Assign 1 a utilization limit of 100%, and 2,3 and 4 a utilization limit of 33% each.
Then place the tasks that get 100% cputime on your 2 cpus in cpuset A and sched group 1.
Place your other tasks in B,{2-4} respectively.
The reason this works is that bandwidth distribution is sched domain wide, and by disabling load-balancing on R, you split the schedule domain.
I've never actually tried anything like this, let me know if it works ;-)
Quoted 44 lines · Peter Zijlstra
Quoted 11 lines · Rolando Martins
I should have elaborated this more:
root ----|---- | | (0.5 mem) 0 1 (100% rt, 0.5 mem) --------- | | | 2 3 4 (33% rt for each group, 33% memper group(0.165))
RolRight, i think this can be done.
You would indeed need cpusets and sched-cgroups.
Split the machine in 2 using cpusets.
___R___ / \ A BWhere R is the root cpuset, and A and B are the siblings. Assign A one half the cpus, and B the other half. Disable load-balancing on R.
Then using sched cgroups create the hierarchy
____1____
/ | \
2 3 4Where 1 can be the root group if you like.
Assign 1 a utilization limit of 100%, and 2,3 and 4 a utilization limit of 33% each.
Then place the tasks that get 100% cputime on your 2 cpus in cpuset A and sched group 1.
Quoted 9 lines
Place your other tasks in B,{2-4} respectively.
The reason this works is that bandwidth distribution is sched domain wide, and by disabling load-balancing on R, you split the schedule domain.
I've never actually tried anything like this, let me know if it works ;-)
Thanks Peter, it works!
I am thinking for different strategies to be used in my rt middleware project, and I think there is a limitation.
If I wanted to have some RT on the B cpuset, I couldn't because I assigned A.cpu.rt_runtime_ns = root.cpu.rt_runtime_ns (then subdivided the A cpuset, with 2,3,4, each one having A.cpu.rt_runtime_ns/3).
This happens because there is a global /proc/sys/kernel/sched_rt_runtime_us and /proc/sys/kernel/sched_rt_period_us.
What do you think about adding a separate tuple (runtime,period) for each core/cpu?
In this case:/proc/sys/kernel/sched_rt_runtime_us_0 /proc/sys/kernel/sched_rt_period_us_0
.../proc/sys/kernel/sched_rt_runtime_us_n (n, cpu count) /proc/sys/kernel/sched_rt_period_us_n
Given this, we could the following:
mkdir /dev/cgroup/A
echo 0-1 > /dev/cgroup/A/cpuset.cpus
echo 0 > /dev/cgroup/A/cpuset.mems
echo 1000000 > /dev/cgroup/A/cpu.rt_period_us
echo 1000000 > /dev/cgroup/A/cpu.rt_runtime_us
This would only work if we could allocate (cpu.rt_runtime_us,cpu.rt_period_us) in both CPU 0 and CPU 1, otherwise fail.
mkdir /dev/cgroup/B
echo 2-3 > /dev/cgroup/B/cpuset.cpus
echo 0 > /dev/cgroup/B/cpuset.mems
echo 1000000 > /dev/cgroup/B/cpu.rt_period_us
echo 800000 > /dev/cgroup/B/cpu.rt_runtime_us
The same here, failed if we couldn't allocate 0.8 in both CPU 2 and CPU 3.
Does this make sense? ;)
Rol
Quoted 5 lines · Rolando Martins
Quoted 4 lines
I've never actually tried anything like this, let me know if it works ;-)
Quoted 7 lines
Thanks Peter, it works!
I am thinking for different strategies to be used in my rt middleware project, and I think there is a limitation.
If I wanted to have some RT on the B cpuset, I couldn't because I assigned A.cpu.rt_runtime_ns = root.cpu.rt_runtime_ns (then subdivided the A cpuset, with 2,3,4, each one having A.cpu.rt_runtime_ns/3).
Try it, you can run RT proglets in B.
You get n*utilization per schedule domain, where n is the number of cpus in it.
So you still have 200% left in B, even if you use 200% of A.
Quoted 3 lines
This happens because there is a
global/proc/sys/kernel/sched_rt_runtime_usand/proc/sys/kernel/sched_rt_period_us.
These globals don't actually do much (except provide a global cap) in the cgroup case.
Quoted 4 lines
What do you think about adding a separate tuple (runtime,period) for each core/cpu?
Does this make sense? ;)
That's going to give me a horrible head-ache trying to load-balance stuff.
Quoted 32 lines · Peter Zijlstra
Quoted 5 lines · Rolando Martins
Quoted 4 lines
I've never actually tried anything like this, let me know if it works ;-)
Quoted 7 lines
Thanks Peter, it works!
I am thinking for different strategies to be used in my rt middleware project, and I think there is a limitation.
If I wanted to have some RT on the B cpuset, I couldn't because I assigned A.cpu.rt_runtime_ns = root.cpu.rt_runtime_ns (then subdivided the A cpuset, with 2,3,4, each one having A.cpu.rt_runtime_ns/3).Try it, you can run RT proglets in B.
You get n*utilization per schedule domain, where n is the number of cpus in it.
So you still have 200% left in B, even if you use 200% of A.
Quoted 3 lines
This happens because there is a
global/proc/sys/kernel/sched_rt_runtime_usand/proc/sys/kernel/sched_rt_period_us.These globals don't actually do much (except provide a global cap) in
the cgroup case.
Quoted 9 lines
Quoted 5 lines
What do you think about adding a separate tuple (runtime,period) for each core/cpu?
Does this make sense? ;)
That's going to give me a horrible head-ache trying to load-balance stuff.
Sorry Peter, but I didn't think before typing;)
I was looking at the cgroups as more integrated (rigid;) ) infrastructure, and therefore using only a mount point for all the operations... :x
Now I got everything working properly! Thanks for the support.
For helping others:
mkdir /dev/cpuset
mount -t cgroup -o cpuset none /dev/cpuset
cd /dev/cpuset
echo 0 > cpuset.sched_load_balance
mkdir A
echo 0-1 > A/cpuset.cpus
echo 0 > A/cpuset.mems
mkdir B
echo 2-3 > B/cpuset.cpus
echo 0 > B/cpuset.mems
mount -t cgroup -o cpu none /dev/sched_domain
cd /dev/sched_domain
mkdir 1
echo cpu.rt_runtime_ns > 1/cpu.rt_runtime_ns
mkdir 1/2
echo 33333 > 1/2/cpu.rt_runtime_ns
mkdir 1/3
echo 33333 > 1/3/cpu.rt_runtime_ns
mkdir 1/4
echo 33333 > 1/3/cpu.rt_runtime_ns
For example, setting the current shell to a specific cpuset(A) and sched(1/2):
echo $$ > /dev/cpuset/A/tasks
echo $$ > /dev/sched_domain/1/2/tasks
"execute program"
Peter, can you confirm this code? ;)
Thanks!
Rol
Quoted 7 lines · Rolando Martins
For helping others:
mkdir
/dev/cpuset
mount -t cgroup -o cpuset none/dev/cpuset
cd/dev/cpuset
echo 0 > cpuset.sched_load_balance
I'm not quite sure that its allowed to disable load-balance before creating children. Other than that it looks ok.
Quoted 25 lines
mkdir A
echo 0-1 > A/cpuset.cpus
echo 0 > A/cpuset.mems
mkdir B
echo 2-3 > B/cpuset.cpus
echo 0 > B/cpuset.memsmount -t cgroup -o cpu none
/dev/sched_domain
cd/dev/sched_domain
mkdir 1
echo cpu.rt_runtime_ns > 1/cpu.rt_runtime_ns
mkdir 1/2
echo 33333 > 1/2/cpu.rt_runtime_ns
mkdir 1/3
echo 33333 > 1/3/cpu.rt_runtime_ns
mkdir 1/4
echo 33333 > 1/3/cpu.rt_runtime_nsFor example, setting the current shell to a specific cpuset(A) and sched(1/2):
echo $$ >
/dev/cpuset/A/tasks
echo $$ >/dev/sched_domain/1/2/tasks
"execute program"
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines · Rolando Martins
For helping others:
Quoted 37 lines
Quoted 4 lines
mkdir
/dev/cpuset
mount -t cgroup -o cpuset none/dev/cpuset
cd/dev/cpuset
echo 0 > cpuset.sched_load_balanceI'm not quite sure that its allowed to disable load-balance before
creating children. Other than that it looks ok.Quoted 25 lines
mkdir A
echo 0-1 > A/cpuset.cpus
echo 0 > A/cpuset.mems
mkdir B
echo 2-3 > B/cpuset.cpus
echo 0 > B/cpuset.memsmount -t cgroup -o cpu none
/dev/sched_domain
cd/dev/sched_domain
mkdir 1
echo cpu.rt_runtime_ns > 1/cpu.rt_runtime_ns
mkdir 1/2
echo 33333 > 1/2/cpu.rt_runtime_ns
mkdir 1/3
echo 33333 > 1/3/cpu.rt_runtime_ns
mkdir 1/4
echo 33333 > 1/3/cpu.rt_runtime_nsFor example, setting the current shell to a specific cpuset(A) and sched(1/2):
echo $$ >
/dev/cpuset/A/tasks
echo $$ >/dev/sched_domain/1/2/tasks
"execute program"
Hi again,
is there any way to have multiple "distinct" sched domains, i.e.:
mount -t cgroup -o cpu none /dev/sched_domain_0
... setup sched_domain_0 (ex: 90% RT, 10% Others) mount -t cgroup -o cpu none /dev/sched_domain_1
... setup sched_domain_1 (ex: 20% RT, 80% Others)
Then give sched_domain_0 to cpuset A and sched_domain_1 to B?
Thanks,
Rol
Quoted 8 lines · Rolando Martins
Hi again,
is there any way to have multiple "distinct" sched domains, i.e.:
mount -t cgroup -o cpu none/dev/sched_domain_0
.... setupsched_domain_0(ex: 90% RT, 10% Others) mount -t cgroup -o cpu none/dev/sched_domain_1
.... setupsched_domain_1(ex: 20% RT, 80% Others) Then givesched_domain_0to cpuset A andsched_domain_1to B?
Nope.
We currently only support a single instance of a cgroup controller.
I see the use for what you propose, however implementing that will be 'interesting'.
Quoted 11 lines · Peter Zijlstra
Quoted 8 lines · Rolando Martins
Hi again,
is there any way to have multiple "distinct" sched domains, i.e.:
mount -t cgroup -o cpu none/dev/sched_domain_0
.... setupsched_domain_0(ex: 90% RT, 10% Others) mount -t cgroup -o cpu none/dev/sched_domain_1
.... setupsched_domain_1(ex: 20% RT, 80% Others) Then givesched_domain_0to cpuset A andsched_domain_1to B?
Quoted 6 lines
Nope.
We currently only support a single instance of a cgroup controller.
I see the use for what you propose, however implementing that will be 'interesting'.
I am confused, if you cpusets, you get your own sched_domain. If you mount cpusets and cpu controller together, you'll get what you want. Is this a figment of my imagination. You might need to use exclusive CPUsets though.
Signature · 1 line
Balbir
Quoted 17 lines · Balbir Singh
Quoted 15 lines · Peter Zijlstra
Quoted 8 lines · Rolando Martins
Hi again,
is there any way to have multiple "distinct" sched domains, i.e.:
mount -t cgroup -o cpu none/dev/sched_domain_0
.... setupsched_domain_0(ex: 90% RT, 10% Others) mount -t cgroup -o cpu none/dev/sched_domain_1
.... setupsched_domain_1(ex: 20% RT, 80% Others) Then givesched_domain_0to cpuset A andsched_domain_1to B?Nope.
We currently only support a single instance of a cgroup controller.
Quoted 7 lines
Quoted 2 lines
I see the use for what you propose, however implementing that will be 'interesting'.
I am confused, if you cpusets, you get your own
sched_domain. If you mount cpusets and cpu controller together, you'll get what you want. Is this a figment of my imagination. You might need to use exclusive CPUsets though.
afaiui he wants a cgroup hierarchy per exclusive sched domain.
Quoted 22 lines · Balbir Singh
Quoted 11 lines · Peter Zijlstra
Quoted 8 lines · Rolando Martins
Hi again,
is there any way to have multiple "distinct" sched domains, i.e.:
mount -t cgroup -o cpu none/dev/sched_domain_0
.... setupsched_domain_0(ex: 90% RT, 10% Others) mount -t cgroup -o cpu none/dev/sched_domain_1
.... setupsched_domain_1(ex: 20% RT, 80% Others) Then givesched_domain_0to cpuset A andsched_domain_1to B?Quoted 6 lines
Nope.
We currently only support a single instance of a cgroup controller.
I see the use for what you propose, however implementing that will be 'interesting'.
Quoted 9 lines
I am confused, if you cpusets, you get your own
sched_domain. If you
mount cpusets and cpu controller together, you'll get what you want. Is this a figment of my imagination. You might need to use exclusive CPUsets though.--
Balbir
I don't know if you meant the following situation (mounting cpuset and cpu together):
R
-----------------------
(80% RT, 20%Others) A B (100% RT, 0%Others)
(Cpus 0-2) (CPU 3)
If so, we can't do this because of the restriction imposed by global rt_runtime_ns.
Perhaps a "feasible" solution could be implemented by having distinct global rt_runtime_ns (one for each cpu, i.e.: rt_runtime_ns_0; ...; rt_runtime_n )
R
-----------------------
(80% RT, 20%Others) A B (100% RT, 0%Others)
(Cpus 0-2) (CPU 3)capacity_used_cpu_0_rt = 0.8 capacity_used_cpu_3_rt = 1capacity_used_cpu_1_rt = 0.8capacity_used_cpu_2_rt = 0.8
Given a i processor: we have the global restriction enforced:SUM(capacity_used_cpu_i_rt) < rt_runtime_i
Rol
Quoted 43 lines · Peter Zijlstra
Quoted 11 lines · Rolando Martins
I should have elaborated this more:
root
----|----
| |
(0.5 mem) 0 1 (100% rt, 0.5 mem)
---------
| | |
2 3 4 (33% rt for each group, 33% mem per group(0.165))
RolRight, i think this can be done.
You would indeed need cpusets and sched-cgroups.
Split the machine in 2 using cpusets.
___R___
/ \
A BWhere R is the root cpuset, and A and B are the siblings. Assign A one half the cpus, and B the other half. Disable load-balancing on R.
Then using sched cgroups create the hierarchy
____1____
/ | \
2 3 4Where 1 can be the root group if you like.
Assign 1 a utilization limit of 100%, and 2,3 and 4 a utilization limit of 33% each.
Then place the tasks that get 100% cputime on your 2 cpus in cpuset A and sched group 1.
Quoted 9 lines
Place your other tasks in B,{2-4} respectively.
The reason this works is that bandwidth distribution is sched domain wide, and by disabling load-balancing on R, you split the schedule domain.
I've never actually tried anything like this, let me know if it works ;-)
Just to confirm, cpuset.sched_load_balance doesn't work with RT, right? You cannot have tasks for sub-domain 2 to utilize bandwidth of sub-domain 3, right?
__1__
/ \
2 3
(50% rt) (50% rt )For my application domain;) it would be interesting to have rt_runtime_ns as a min. of allocated rt and not a max.
Ex. If an application of domain 2 needs to go up to 100% and domain 3 is idle, then it would be cool to let it utilize the full bandwidth. (we also could have a hard upper limit in each sub-domain, like hard_up=0.8, i.e. even if we could get 100%, we will only utilize 80%).
Does this make sense?