sched/cgroup: Does cpu-cgroup still works fine nowadays?
During our testing, we found that the cpu.shares doesn't work as expected, the testing is:
X86 HOST:
12 CPU
GUEST(KVM):
6 VCPU
We create 3 GUEST, each with 1024 shares, the workload inside them is:
GUEST_1:
dbench 6GUEST_2:
stress -c 6GUEST_3:
stress -c 6
So by theory, each GUEST will got (1024 / (3 * 1024)) * 1200% == 400% according to the group share (3 groups are created by virtual manager on same level, and they are the only groups heavily running in system).
Now if only GUEST_1 running, it got 300% CPU, which is 1/4 of the whole CPU resource.
So when all 3 GUEST running concurrently, we expect:
GUEST_1 GUEST_2 GUEST_3
CPU% 300% 450% 450%
That is the GUEST_1 got the 300% it required, and the unused 100% was shared by the rest group.
But the result is:
GUEST_1 GUEST_2 GUEST_3
CPU% 40% 580% 580%
GUEST_1 failed to gain the CPU it required, and the dbench inside it dropped a lot on performance.
So is this results expected (I really do not think so...)?
Or that imply the cpu-cgroup got some issue to be fixed?
Any comments are welcomed :)
Regards,
Michael Wang
Peter Zijlstra
maintainerQuoted 3 lines · Michael wang
During our testing, we found that the cpu.shares doesn't work as expected, the testing is:
/me zaps all the kvm nonsense as that's non reproducable and only serves to annoy.
Pro-tip: never use kvm to report cpu-cgroup issues.
Quoted 3 lines
So is this results expected (I really do not think so...)?
Or that imply the cpu-cgroup got some issue to be fixed?
So what I did (WSM-EP 2x6x2):
mount none /cgroup -t cgroup -o cpu
mkdir -p /cgroup/a
mkdir -p /cgroup/b
mkdir -p /cgroup/c
echo $$ > /cgroup/a/tasks ; for ((i=0; i<12; i++)) ; do A.sh & done echo $$ > /cgroup/b/tasks ; for ((i=0; i<12; i++)) ; do B.sh & done echo $$ > /cgroup/c/tasks ; for ((i=0; i<12; i++)) ; do C.sh & done
echo 2048 > /cgroup/c/cpu.shares
Where [ABC].sh are spinners:
---
#!/bin/bash
while :; do :; done
---
for i in A B C ; do ps -deo pcpu,cmd | grep "${i}\.sh" | awk '{t += $1} END {print t}' ; done 639.7
629.8
1127.4
That is of course not perfect, but it's close enough.
Now you again.. :-)
Quoted 21 lines · Peter Zijlstra
Quoted 3 lines · Michael wang
During our testing, we found that the cpu.shares doesn't work as expected, the testing is:
/me zaps all the kvm nonsense as that's non reproducable and only serves to annoy.
Pro-tip: never use kvm to report cpu-cgroup issues.
Quoted 3 lines
So is this results expected (I really do not think so...)?
Or that imply the cpu-cgroup got some issue to be fixed?
So what I did (WSM-EP 2x6x2):
mount none /cgroup -t cgroup -o cpu
mkdir -p /cgroup/a
mkdir -p /cgroup/b
mkdir -p /cgroup/c
Quoted 7 lines
echo $$ > /cgroup/a/tasks ; for ((i=0; i<12; i++)) ; do
A.sh& done echo $$ > /cgroup/b/tasks ; for ((i=0; i<12; i++)) ; doB.sh& done echo $$ > /cgroup/c/tasks ; for ((i=0; i<12; i++)) ; doC.sh& doneecho 2048 > /cgroup/c/cpu.shares
Where [ABC].sh are spinners:
I suspect the "are spinners" is key.
Infinite loops can run all the time, while dbench spends a lot of its time waiting for locks. That waiting may interfere with getting as much CPU as it wants.
Signature · 1 line
All rights reversed
Peter Zijlstra
maintainerQuoted 28 lines · Rik van Riel
Quoted 27 lines · Peter Zijlstra
Quoted 3 lines · Michael wang
During our testing, we found that the cpu.shares doesn't work as expected, the testing is:
/me zaps all the kvm nonsense as that's non reproducable and only serves to annoy.
Pro-tip: never use kvm to report cpu-cgroup issues.
Quoted 3 lines
So is this results expected (I really do not think so...)?
Or that imply the cpu-cgroup got some issue to be fixed?
So what I did (WSM-EP 2x6x2):
mount none /cgroup -t cgroup -o cpu
mkdir -p /cgroup/a
mkdir -p /cgroup/b
mkdir -p /cgroup/cecho $$ > /cgroup/a/tasks ; for ((i=0; i<12; i++)) ; do
A.sh& done echo $$ > /cgroup/b/tasks ; for ((i=0; i<12; i++)) ; doB.sh& done echo $$ > /cgroup/c/tasks ; for ((i=0; i<12; i++)) ; doC.sh& doneecho 2048 > /cgroup/c/cpu.shares
Quoted 7 lines
Quoted 1 line
Where [ABC].sh are spinners:
I suspect the "are spinners" is key.
Infinite loops can run all the time, while dbench spends a lot of its time waiting for locks. That waiting may interfere with getting as much CPU as it wants.
At which point it becomes an entirely different problem and the weight things become far more 'interesting'.
The point remains though, don't use massive and awkward software stacks that are impossible to operate.
I you want to investigate !spinners, replace the ABC with slightly more complex loads like: https://lkml.org/lkml/2012/6/18/212
On 05/13/2014 10:23 PM, Peter Zijlstra wrote:
[snip]
Quoted 6 lines
The point remains though, don't use massive and awkward software stacks that are impossible to operate.
I you want to investigate !spinners, replace the ABC with slightly more complex loads like: https://lkml.org/lkml/2012/6/18/212
That's what we need, may be a little reform to enable multi-threads, or may be add some locks... anyway, will redo the test and see what we could found :)
Regards,
Michael Wang
Quoted 6 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Hi, Peter
On 05/13/2014 10:23 PM, Peter Zijlstra wrote:
[snip]
Quoted 3 lines
I you want to investigate !spinners, replace the ABC with slightly more complex loads like: https://lkml.org/lkml/2012/6/18/212
I've done a little reform, enabled multi-threads and add a mutex, please check the code below for details.
I built it by:
gcc -o my_tool cgroup_tool.c -lpthread
distro mount cpu-subsys under '/sys/fs/cgroup/cpu', create group like:
mkdir /sys/fs/cgroup/cpu/A
mkdir /sys/fs/cgroup/cpu/B
mkdir /sys/fs/cgroup/cpu/C
and then:
echo $$ > /sys/fs/cgroup/cpu/A/tasks ; ./my_tool -l echo $$ > /sys/fs/cgroup/cpu/B/tasks ; ./my_tool -l echo $$ > /sys/fs/cgroup/cpu/C/tasks ; ./my_tool 50
the results in top is around:
A B C CPU% 550 550 100
While only './my_tool 50' was running, it require around 300%.
And this could also be reproduced by dbench, stress combination like:
echo $$ > /sys/fs/cgroup/cpu/A/tasks ; dbench 6 echo $$ > /sys/fs/cgroup/cpu/B/tasks ; stress -c 6 echo $$ > /sys/fs/cgroup/cpu/C/tasks ; stress -c 6
Now it seems more like a generic problem... will keep investigating, please let me know if there are any suggestions :)
Regards,
Michael Wang
#include <sys/time.h>
#include <unistd.h>
#include <stdio.h>
#include <pthread.h>
pthread_mutex_t my_mutex;
unsigned long long stamp(void)
{
struct timeval tv;
gettimeofday(&tv, NULL);
return (unsigned long long)tv.tv_sec * 1000000 + tv.tv_usec;
}
void consume(int spin, int total)
{
unsigned long long begin, now;
begin = stamp();
for (;;) {
pthread_mutex_lock(&my_mutex);
now = stamp();
if ((long long)(now - begin) > spin) {
pthread_mutex_unlock(&my_mutex);
usleep(total - spin);
pthread_mutex_lock(&my_mutex);
begin += total;
}
pthread_mutex_unlock(&my_mutex);
}
}struct my_data {
int spin;
int total;
};void *my_fn_sleepy(void *arg)
{
struct my_data *data = (struct my_data *)arg;
consume(data->spin, data->total);
return NULL;
}void *my_fn_loop(void *arg)
{
while (1) {};
return NULL;
}int main(int argc, char **argv)
{
int period = 100000; /* 100ms */
int frac;
struct my_data data;
pthread_t last_thread;
int thread_num = sysconf(_SC_NPROCESSORS_ONLN) / 2;
void *(*my_fn)(void *arg) = &my_fn_sleepy;
if (thread_num <= 0 || thread_num > 1024) {
fprintf(stderr, "insane processor(half) size %d\n", thread_num);
return -1;
}
if (argc == 2 && !strcmp(argv[1], "-l")) {
my_fn = &my_fn_loop;
printf("loop mode enabled\n");
goto loop_mode;
}
if (argc < 2) {
fprintf(stderr, "%s <frac> [<period>]\n"
" frac -- [1-100] %% of time to burn\n"
" period -- [usec] period of burn/sleep cycle\n",
argv[0]);
return -1;
}
frac = atoi(argv[1]);
if (argc > 2)
period = atoi(argv[2]);
if (frac > 100)
frac = 100;
if (frac < 1)
frac = 1;
data.spin = (period * frac) / 100;
data.total = period;loop_mode:
pthread_mutex_init(&my_mutex, NULL);
while (thread_num--) {
if (pthread_create(&last_thread, NULL, my_fn, &data)) {
fprintf(stderr, "Create thread failed\n");
return -1;
}
}
printf("Threads never stop, CTRL + C to terminate\n");
pthread_join(last_thread, NULL);
pthread_mutex_destroy(&my_mutex); //won't happen
return 0;
}Quoted 6 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
#include <sys/time.h>
#include <unistd.h>
#include <stdio.h>
#include <pthread.h>
pthread_mutex_t my_mutex;
unsigned long long stamp(void)
{
struct timeval tv;
gettimeofday(&tv, NULL);
return (unsigned long long)tv.tv_sec * 1000000 + tv.tv_usec;
}
void consume(int spin, int total)
{
unsigned long long begin, now;
begin = stamp();
for (;;) {
pthread_mutex_lock(&my_mutex);
now = stamp();
if ((long long)(now - begin) > spin) {
pthread_mutex_unlock(&my_mutex);
usleep(total - spin);
pthread_mutex_lock(&my_mutex);
begin += total;
}
pthread_mutex_unlock(&my_mutex);
}
}struct my_data {
int spin;
int total;
};void *my_fn_sleepy(void *arg)
{
struct my_data *data = (struct my_data *)arg;
consume(data->spin, data->total);
return NULL;
}void *my_fn_loop(void *arg)
{
while (1) {};
return NULL;
}int main(int argc, char **argv)
{
int period = 100000; /* 100ms */
int frac;
struct my_data data;
pthread_t last_thread;
int thread_num = sysconf(_SC_NPROCESSORS_ONLN) / 2;
void *(*my_fn)(void *arg) = &my_fn_sleepy;
if (thread_num <= 0 || thread_num > 1024) {
fprintf(stderr, "insane processor(half) size %d\n", thread_num);
return -1;
}
if (argc == 2 && !strcmp(argv[1], "-l")) {
my_fn = &my_fn_loop;
printf("loop mode enabled\n");
goto loop_mode;
}
if (argc < 2) {
fprintf(stderr, "%s <frac> [<period>]\n"
" frac -- [1-100] %% of time to burn\n"
" period -- [usec] period of burn/sleep cycle\n",
argv[0]);
return -1;
}
frac = atoi(argv[1]);
if (argc > 2)
period = atoi(argv[2]);
if (frac > 100)
frac = 100;
if (frac < 1)
frac = 1;
data.spin = (period * frac) / 100;
data.total = period;loop_mode:
pthread_mutex_init(&my_mutex, NULL);
while (thread_num--) {
if (pthread_create(&last_thread, NULL, my_fn, &data)) {
fprintf(stderr, "Create thread failed\n");
return -1;
}
}
printf("Threads never stop, CTRL + C to terminate\n");
pthread_join(last_thread, NULL);
pthread_mutex_destroy(&my_mutex); //won't happen
return 0;
}Peter Zijlstra
maintainerQuoted 4 lines · Michael wang
distro mount cpu-subsys under '
/sys/fs/cgroup/cpu', create group like:
mkdir/sys/fs/cgroup/cpu/A
mkdir/sys/fs/cgroup/cpu/B
mkdir/sys/fs/cgroup/cpu/C
Yeah, distro is on crack, nobody sane mounts anything there.
Quoted 9 lines
and then:
echo $$ > /sys/fs/cgroup/cpu/A/tasks ; ./my_tool -l echo $$ > /sys/fs/cgroup/cpu/B/tasks ; ./my_tool -l echo $$ > /sys/fs/cgroup/cpu/C/tasks ; ./my_tool 50the results in top is around:
A B C CPU% 550 550 100
top doesn't do per-cgroup accounting, so how do you get these numbers, per the above all instances of the prog are also called the same, further making it error prone and difficult to get sane numbers.
Quoted 23 lines
#include <
sys/time.h>
#include <unistd.h>
#include <stdio.h>
#include <pthread.h>
pthread_mutex_tmy_mutex;unsigned long long stamp(void)
{ struct timeval tv; gettimeofday(&tv, NULL); return (unsigned long long)tv.tv_sec * 1000000 + tv.tv_usec; } void consume(int spin, int total) { unsigned long long begin, now; begin = stamp(); for (;;) { pthread_mutex_lock(&my_mutex); now = stamp(); if ((long long)(now - begin) > spin) {
Quoted 8 lines
pthread_mutex_unlock(&my_mutex); usleep(total - spin); pthread_mutex_lock(&my_mutex); begin += total; } pthread_mutex_unlock(&my_mutex); } }
Uh,.. that's just insane.. what's the point of having a multi-threaded program do busy-wait loops if you then serialize the lot on a global mutex such that only 1 thread can run at any one time?
How can one such prog ever consume more than 100% cpu.
On 05/14/2014 05:44 PM, Peter Zijlstra wrote:
[snip]
Quoted 5 lines
Quoted 5 lines
and then:
echo $$ > /sys/fs/cgroup/cpu/A/tasks ; ./my_tool -l echo $$ > /sys/fs/cgroup/cpu/B/tasks ; ./my_tool -l echo $$ > /sys/fs/cgroup/cpu/C/tasks ; ./my_tool 50
Quoted 8 lines
Quoted 4 lines
the results in top is around:
A B C CPU% 550 550 100top doesn't do per-cgroup accounting, so how do you get these numbers, per the above all instances of the prog are also called the same, further making it error prone and difficult to get sane numbers.
Oh, my bad to make it confusing, I myself was checking the PID of my_tool instant inside top, like:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24968 root 20 0 55600 720 648 S 558.1 0.0 2:08.76 my_tool 24984 root 20 0 55600 720 648 S 536.2 0.0 1:10.29 my_tool 25001 root 20 0 55600 720 648 S 88.6 0.0 0:04.39 my_tool
By 'cat /sys/fs/cgroup/cpu/C/tasks' I got the PID of './my_tool 50' is 25001, and all it's pthread's %CPU was count in, could we check like that?
Quoted 2 lines
[snip]
Quoted 18 lines
Quoted 17 lines
void
consume(int spin, int total){ unsigned long long begin, now; begin = stamp(); for (;;) { pthread_mutex_lock(&my_mutex); now = stamp(); if ((long long)(now - begin) > spin) { pthread_mutex_unlock(&my_mutex); usleep(total - spin); pthread_mutex_lock(&my_mutex); begin += total; } pthread_mutex_unlock(&my_mutex); } }
Quoted 5 lines
Uh,.. that's just insane.. what's the point of having a multi-threaded program do busy-wait loops if you then serialize the lot on a global mutex such that only 1 thread can run at any one time?
How can one such prog ever consume more than 100% cpu.
That's a good point... however the top show that when only './my_tool 50' 25001 running, it used around 300%, like below:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 25001 root 20 0 55600 720 648 S 284.3 0.0 5:18.00 my_tool 2376 root 20 0 950m 85m 29m S 4.4 0.2 163:47.94 python 1658 root 20 0 1013m 19m 11m S 3.0 0.1 97:06.11 libvirtd
IMHO, if pthread-mutex was similar like the kernel one's behaviour, then it may not going to sleep when it's the only one running on CPU.
Oh, I think we got the reason here, when there are other task running, mutex will going to sleep and the %CPU dropped to serialized case that is around 100%.
But for the dbench, stress combination, that's not spin-wasted, dbench throughput do dropped, how could we explain that one?
Regards,
Michael Wang
Quoted 1 line
Peter Zijlstra
maintainerQuoted 2 lines · Michael wang
But for the dbench, stress combination, that's not spin-wasted, dbench throughput do dropped, how could we explain that one?
I've no clue what dbench does.. At this point you'll have to expose/trace the per-task runtime accounting for these tasks and ideally also the things the cgroup code does with them to see if it still makes sense.
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines · Michael wang
But for the dbench, stress combination, that's not spin-wasted, dbench throughput do dropped, how could we explain that one?
I've no clue what dbench does.. At this point you'll have to expose/trace the per-task runtime accounting for these tasks and ideally also the things the cgroup code does with them to see if it still makes sense.
I see :)
BTW, some interesting thing we found during the dbench/stress testing is, by doing:
echo 240000000 > /proc/sys/kernel/sched_latency_ns
echo NO_GENTLE_FAIR_SLEEPERS > /sys/kernel/debug/sched_featuresthat is sched_latency_ns increased around 10 times and GENTLE_FAIR_SLEEPERS was disabled, the dbench got it's CPU back.
However, when the group level is too deep, that doesn't works any more...
I'm not sure but seems like 'deep group level' and 'vruntime bonus for sleeper' is the keep points here, will try to list the root cause after more investigation, thanks for the hints and suggestions, really helpful ;-)
Regards,
Michael Wang
Quoted 1 line
Peter Zijlstra
maintainerQuoted 18 lines · Michael wang
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines · Michael wang
But for the dbench, stress combination, that's not spin-wasted, dbench throughput do dropped, how could we explain that one?
I've no clue what dbench does.. At this point you'll have to expose/trace the per-task runtime accounting for these tasks and ideally also the things the cgroup code does with them to see if it still makes sense.
I see :)
BTW, some interesting thing we found during the dbench/stress testing is, by doing:
echo 240000000 > /proc/sys/kernel/sched_latency_ns echo NO_GENTLE_FAIR_SLEEPERS > /sys/kernel/debug/sched_features
Quoted 8 lines
that is
sched_latency_nsincreased around 10 times andGENTLE_FAIR_SLEEPERSwas disabled, the dbench got it's CPU back.However, when the group level is too deep, that doesn't works any more...
I'm not sure but seems like 'deep group level' and 'vruntime bonus for sleeper' is the keep points here, will try to list the root cause after more investigation, thanks for the hints and suggestions, really helpful ;-)
How deep is deep? You run into numerical problems quite quickly, esp. when you've got lots of CPUs. We've only got 64bit to play with, that said there were some patches...
What happens if you do the below, Google has been running with that, and nobody was ever able to reproduce the report that got it disabled.
kernel/sched/sched.h +1 -1
@@ -40,7 +40,7 @@ extern void update_cpu_load_active(struct rq *this_rq);* when BITS_PER_LONG <= 32 are pretty high and the returns do not justify the * increased costs. */ -#if 0 /* BITS_PER_LONG > 32 -- currently broken: it increases power usage under light load */ +#if 1 /* BITS_PER_LONG > 32 -- currently broken: it increases power usage under light load */ # define SCHED_LOAD_RESOLUTION 10# define scale_load (w) ((w) << SCHED_LOAD_RESOLUTION)# define scale_load_down (w) ((w) >> SCHED_LOAD_RESOLUTION)
On 05/15/2014 05:06 PM, Peter Zijlstra wrote:
[snip]
Quoted 9 lines
Quoted 5 lines
However, when the group level is too deep, that doesn't works any more...
I'm not sure but seems like 'deep group level' and 'vruntime bonus for sleeper' is the keep points here, will try to list the root cause after more investigation, thanks for the hints and suggestions, really helpful ;-)
How deep is deep? You run into numerical problems quite quickly, esp. when you've got lots of CPUs. We've only got 64bit to play with, that said there were some patches...
It's like:
/cgroup/cpu/l1/l2/l3/l4/l5/l6/A
about level 7, the issue can not be solved any more.
Quoted 11 lines
What happens if you do the below, Google has been running with that, and nobody was ever able to reproduce the report that got it disabled.
kernel/sched/sched.h +0
-0
kernel/sched/sched.h extern void update_cpu_load_active(struct rq *this_rq); · line 43
Quoted 5 lines
kernel/sched/sched.h +1
-1@@ -40,7 +40,7 @@ extern void update_cpu_load_active(struct rq *this_rq);* when BITS_PER_LONG <= 32 are pretty high and the returns do not justify the * increased costs. */ -#if 0 /* BITS_PER_LONG > 32 -- currently broken: it increases power usage under light load */ +#if 1 /* BITS_PER_LONG > 32 -- currently broken: it increases power usage under light load */
That is trying to solve the load overflow issue, correct?
I'm not sure which account will turns to be huge when group get deeper, the load accumulation will suffer discount when passing up, isn't it?
Anyway, will give it a try and see what happened :)
Regards,
Michael Wang
Quoted 4 lines
kernel/sched/sched.h +0
-0# define SCHED_LOAD_RESOLUTION 10# define scale_load (w) ((w) << SCHED_LOAD_RESOLUTION)# define scale_load_down (w) ((w) >> SCHED_LOAD_RESOLUTION)
Peter Zijlstra
maintainerQuoted 12 lines · Michael wang
On 05/15/2014 05:06 PM, Peter Zijlstra wrote:
[snip]Quoted 9 lines
Quoted 5 lines
However, when the group level is too deep, that doesn't works any more...
I'm not sure but seems like 'deep group level' and 'vruntime bonus for sleeper' is the keep points here, will try to list the root cause after more investigation, thanks for the hints and suggestions, really helpful ;-)
How deep is deep? You run into numerical problems quite quickly, esp. when you've got lots of CPUs. We've only got 64bit to play with, that said there were some patches...
Quoted 5 lines
It's like:
/cgroup/cpu/l1/l2/l3/l4/l5/l6/Aabout level 7, the issue can not be solved any more.
That's pretty retarded and yeah, that's way past the point where things make sense. You might be lucky and have l1-5 as empty/pointless hierarchy so the effective depth is less and then things will work, but *shees*..
Quoted 8 lines
kernel/sched/sched.h extern void update_cpu_load_active(struct rq *this_rq); · line 43
Quoted 2 lines
kernel/sched/sched.h +1
-1-#if 0 /* BITS_PER_LONG > 32 -- currently broken: it increases power usage under light load */ +#if 1 /* BITS_PER_LONG > 32 -- currently broken: it increases power usage under light load */ That is trying to solve the load overflow issue, correct?
I'm not sure which account will turns to be huge when group get deeper, the load accumulation will suffer discount when passing up, isn't it?
It'll use 20 bits for precision instead of 10, so it gives a little more 'room' for deeper hierarchies/big cpu-count.
All assuming you're running 64bit kernels of course.
On 05/15/2014 07:57 PM, Peter Zijlstra wrote:
[snip]
Quoted 11 lines
Quoted 6 lines
It's like:
/cgroup/cpu/l1/l2/l3/l4/l5/l6/Aabout level 7, the issue can not be solved any more.
That's pretty retarded and yeah, that's way past the point where things make sense. You might be lucky and have l1-5 as empty/pointless hierarchy so the effective depth is less and then things will work, but *shees*..
Exactly, that's the simulation of cgroup topology setup by libvirt, really doesn't make sense... rather torture than deployment, but they do make things like that...
Quoted 1 line
[snip]
Quoted 6 lines
Quoted 3 lines
I'm not sure which account will turns to be huge when group get deeper, the load accumulation will suffer discount when passing up, isn't it?
It'll use 20 bits for precision instead of 10, so it gives a little more 'room' for deeper hierarchies/big cpu-count.
Got it :)
Quoted 2 lines
All assuming you're running 64bit kernels of course.
Yes, it's 64bit, I tried the testing with this feature on, seems like haven't address the issue...
But we found that one difference when group get deeper is the tasks of that group become to gathered on CPU more often, some time all the dbench instances was running on the same CPU, this won't happen for l1 group, may could explain why dbench could not get CPU more than 100% any more.
But why the gather happen when group get deeper is unclear... will try to make it out :)
Regards,
Michael Wang
Quoted 1 line
Quoted 5 lines · Michael wang
But we found that one difference when group get deeper is the tasks of that group become to gathered on CPU more often, some time all the dbench instances was running on the same CPU, this won't happen for l1 group, may could explain why dbench could not get CPU more than 100% any more.
Right. I played a little (sane groups), saw load balancing as well.
-Mike
Hey, Mike :)
Quoted 9 lines · Mike Galbraith
Quoted 5 lines · Michael wang
But we found that one difference when group get deeper is the tasks of that group become to gathered on CPU more often, some time all the dbench instances was running on the same CPU, this won't happen for l1 group, may could explain why dbench could not get CPU more than 100% any more.
Right. I played a little (sane groups), saw load balancing as well.
Yeah, now we found that even l2 groups will face the same issue, allow me to re-list the details here:
Firstly do workaround (10 times latency):
echo 240000000 > /proc/sys/kernel/sched_latency_ns
echo NO_GENTLE_FAIR_SLEEPERS > /sys/kernel/debug/sched_featuresThis workaround may related to another issue about vruntime bonus for sleeper, but let's put it down currently and focus on the gather issue.
Create groups like:
mkdir /sys/fs/cgroup/cpu/A
mkdir /sys/fs/cgroup/cpu/B
mkdir /sys/fs/cgroup/cpu/C
mkdir /sys/fs/cgroup/cpu/l1
mkdir /sys/fs/cgroup/cpu/l1/A
mkdir /sys/fs/cgroup/cpu/l1/B
mkdir /sys/fs/cgroup/cpu/l1/C
Run workload like (6 is half of the CPUS on my box):
echo $$ > /sys/fs/cgroup/cpu/A/tasks ; dbench 6 echo $$ > /sys/fs/cgroup/cpu/B/tasks ; stress 6 echo $$ > /sys/fs/cgroup/cpu/C/tasks ; stress 6
Check top, each dbench instance got around 45%, totally around 270%, this is close to the case when only dbench running (300%) since we use the workaround, otherwise we will see it to be around 100%, but that's another issue...
By sample /proc/sched_debug, rarely see more than 2 dbench instances on same rq.
Now re-run workload like:
echo $$ > /sys/fs/cgroup/cpu/l1/A/tasks ; dbench 6 echo $$ > /sys/fs/cgroup/cpu/l1/B/tasks ; stress 6 echo $$ > /sys/fs/cgroup/cpu/l1/C/tasks ; stress 6
Check top, each dbench instance got around 20%, totally around 120%, sometime dropped under 100%, and dbench throughput dropped.
By sample /proc/sched_debug, frequently see 4 or 5 dbench instances on same rq.
So just one level deeper from l1 to l2 and such a big difference, and groups with same shares not equally share the resources...
BTW, by bind each dbench instances to different CPU, dbench in l2 groups will regain all the CPU% which is 300%.
I'll keep investigation and try to figure out why l2 group's tasks starting to gather, please let me know if there are any suggestions ;-)
Regards,
Michael Wang
Quoted 9 lines
-Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Peter Zijlstra
maintainerQuoted 11 lines · Michael wang
Hey, Mike :)
Quoted 8 lines · Mike Galbraith
Quoted 5 lines · Michael wang
But we found that one difference when group get deeper is the tasks of that group become to gathered on CPU more often, some time all the dbench instances was running on the same CPU, this won't happen for l1 group, may could explain why dbench could not get CPU more than 100% any more.
Quoted 4 lines
Quoted 1 line
Right. I played a little (sane groups), saw load balancing as well.
Yeah, now we found that even l2 groups will face the same issue, allow me to re-list the details here:
Hmm, that _should_ more or less work and does indeed suggest there's something iffy.
On 05/16/2014 03:54 PM, Peter Zijlstra wrote:
[snip]
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Right. I played a little (sane groups), saw load balancing as well.
Yeah, now we found that even l2 groups will face the same issue, allow me to re-list the details here:
Hmm, that _should_ more or less work and does indeed suggest there's something iffy.
Yeah, sane group topology also issued... besides the sleeper bonus, it seems like the root cause is tasks starting to gather, I plan to check the difference on task load between two cases, see if there is a good way to solve this problem :)
Regards,
Michael Wang
Quoted 1 line
On 05/16/2014 03:54 PM, Peter Zijlstra wrote:
[snip]
Quoted 4 lines
Hmm, that _should_ more or less work and does indeed suggest there's something iffy.
I think we locate the reason why cpu-cgroup doesn't works well on dbench now... finally.
I'd like to link the reproduce way of the issue here since long time passed...
https://lkml.org/lkml/2014/5/16/4
Now here is the analysis:
So our problem is when put tasks like dbench which sleep and wakeup each other frequently into a deep-group, they will gathered on same CPU when workload like stress are running, which lead to that the whole group could gain no more than one CPU.
Basically there are two key points here, load-balance and wake-affine.
Wake-affine for sure pull tasks together for workload like dbench, what make it difference when put dbench into a group one level deeper is the load-balance, which happened less.
Usually, when system is busy, during the wakeup when we could not locate idle cpu, we pick the search point instead, whatever how busy it is since we count on the balance routine later to help balance the load.
However, in our cases the load balance could not help on that, since deeper the group is, less the load effect it means to root group.
By which means even tasks in deep group all gathered on one CPU, the load could still balanced from the view of root group, and the tasks lost the only chances (balance) to spread when they already on the same CPU...
Furthermore, for tasks flip frequently like dbench, it'll become far more harder for load balance to help, it could even rarely catch them on rq.
So in such cases, the only chance to do balance for these tasks is during the wakeup, however it will be expensive...
Thus the cheaper way is something just like select_idle_sibling(), the only difference is now we balance tasks inside the group to prevent them from gathered.
Below patch has solved the problem during the testing, I'd like to do more testing on other benchmarks before send out the formal patch, any comments are welcomed ;-)
Regards,
Michael Wang
kernel/sched/fair.c +81 -0
@@ -4409,6 +4409,62 @@ find_idlest_cpu(struct sched_group *group, struct task_struct *p, int this_cpu)return idlest; } +static inline int tg_idle_cpu (struct task_group *tg,int cpu) +{ +return !tg->cfs_rq[cpu]->nr_running; +} + +/* +* Try and locate an idle CPU in the sched_domain from tg's view. +* +* Although gathered on same CPU and spread accross CPUs could make +* no difference from highest group's view, this will cause the tasks +* starving, even they have enough share to fight for CPU, they only +* got one battle filed, which means whatever how big their weight is, +* they totally got one CPU at maximum. +* +* Thus when system is busy, we filtered out those tasks which couldn't +* gain help from balance routine, and try to balance them internally +* by this func, so they could stand a chance to show their power. +* +*/ +static int tg_idle_sibling (struct task_struct *p,int target) +{ +struct sched_domain *sd; +struct sched_group *sg; +int i =task_cpu (p); +struct task_group *tg =task_group (p); + +if (tg_idle_cpu (tg, target)) + goto done; + + sd =rcu_dereference (per_cpu (sd_llc, target)); +for_each_lower_domain (sd) { + sg = sd->groups; +do { +if (!cpumask_intersects (sched_group_cpus (sg), +tsk_cpus_allowed (p))) + goto next; + +for_each_cpu (i,sched_group_cpus (sg)) { +if (i == target || !tg_idle_cpu (tg, i)) + goto next; + } + + target =cpumask_first_and (sched_group_cpus (sg), +tsk_cpus_allowed (p)); + + goto done; +next: + sg = sg->next; + }while (sg != sd->groups); + } + +done: + +return target; +} +/* * Try and locate an idle CPU in the sched_domain. */
@@ -4417,6 +4473,7 @@ static int select_idle_sibling(struct task_struct *p, int target)struct sched_domain *sd;struct sched_group *sg;int i =task_cpu (p); +struct sched_entity *se =task_group (p)->se[i];if (idle_cpu (target))return target;
@@ -4451,6 +4508,30 @@ next: }while (sg != sd->groups); } done: + +if (!idle_cpu (target)) { +/* +* No idle cpu located imply the system is somewhat busy, +* usually we count on load balance routine's help and +* just pick the target whatever how busy it is. +* +* However, when task belong to a deep group (harder to +* make root imbalance) and flip frequently (harder to be +* caught during balance), load balance routine could help +* nothing, and these tasks will eventually gathered on same +* cpu when they wakeup each other, that is the chance of +* gathered stand far more higher than the chance of spread. +* +* Thus for such tasks, we need to handle them carefully +* during wakeup, since it's the very rarely chance for +* them to spread. +* +*/ +if (se && se->depth && + p->wakee_flips >this_cpu_read (sd_llc_size)) +return tg_idle_sibling (p, target); + } +return target; }
Peter Zijlstra
maintainerQuoted 22 lines · Michael wang
On 05/16/2014 03:54 PM, Peter Zijlstra wrote:
[snip]Quoted 4 lines
Hmm, that _should_ more or less work and does indeed suggest there's something iffy.
I think we locate the reason why cpu-cgroup doesn't works well on dbench now... finally.
I'd like to link the reproduce way of the issue here since long time passed...
https://lkml.org/lkml/2014/5/16/4Now here is the analysis:
So our problem is when put tasks like dbench which sleep and wakeup each other frequently into a deep-group, they will gathered on same CPU when workload like stress are running, which lead to that the whole group could gain no more than one CPU.
Quoted 5 lines
Basically there are two key points here, load-balance and wake-affine.
Wake-affine for sure pull tasks together for workload like dbench, what make it difference when put dbench into a group one level deeper is the load-balance, which happened less.
We load-balance less (frequently) or we migrate less tasks due to load-balancing ?
Quoted 3 lines
Usually, when system is busy, during the wakeup when we could not locate idle cpu, we pick the search point instead, whatever how busy it is since we count on the balance routine later to help balance the load.
But above you said that dbench usually triggers the wake-affine logic, but now you say it doesn't and we rely on select_idle_sibling?
Note that the comparison isn't fair, running dbench on an idle system vs running dbench on a busy system is the first step.
The second is adding the cgroup crap on.
Quoted 2 lines
However, in our cases the load balance could not help on that, since deeper the group is, less the load effect it means to root group.
But since all actual load is on the same depth, the relative threshold (imbalance pct) should work the same, the size of the values don't matter, the relative ratios do.
Quoted 3 lines
By which means even tasks in deep group all gathered on one CPU, the load could still balanced from the view of root group, and the tasks lost the only chances (balance) to spread when they already on the same CPU...
Sure, but see above.
Quoted 2 lines
Furthermore, for tasks flip frequently like dbench, it'll become far more harder for load balance to help, it could even rarely catch them on rq.
And I suspect that is the main problem; so see what it does on a busy
system: !cgroup: nr_cpus busy loops + dbench, because that's your benchmark for adding cgroups, the cgroup can only shift that behaviour around.
Quoted 10 lines
So in such cases, the only chance to do balance for these tasks is during the wakeup, however it will be expensive...
Thus the cheaper way is something just like
select_idle_sibling(), the only difference is now we balance tasks inside the group to prevent them from gathered.Below patch has solved the problem during the testing, I'd like to do more testing on other benchmarks before send out the formal patch, any comments are welcomed ;-)
So I think that approach is wrong, select_idle_siblings() works because we want to keep CPUs from being idle, but if they're not actually idle, pretending like they are (in a cgroup) is actively wrong and can skew load pretty bad.
Furthermore, if as I expect, dbench sucks on a busy system, then the proposed cgroup thing is wrong, as a cgroup isn't supposed to radically alter behaviour like that.
More so, I suspect that patch will tend to overload cpu0 (and lower cpu numbers in general -- because its scanning in the same direction for each cgroup) for other workloads. You can't just go pile more and more work on cpu0 just because there's nothing running in this particular cgroup.
So dbench is very sensitive to queueing, and select_idle_siblings() avoids a lot of queueing on an idle system. I don't think that's something we should fix with cgroups.
Hi, Peter
Thanks for the reply :)
On 06/10/2014 08:12 PM, Peter Zijlstra wrote:
[snip]
Quoted 6 lines
Quoted 3 lines
Wake-affine for sure pull tasks together for workload like dbench, what make it difference when put dbench into a group one level deeper is the load-balance, which happened less.
We load-balance less (frequently) or we migrate less tasks due to load-balancing ?
IMHO, when we put tasks one group deeper, in other word the totally weight of these tasks is 1024 (prev is 3072), the load become more balancing in root, which make bl-routine consider the system is balanced, which make we migrate less in lb-routine.
Quoted 7 lines
Quoted 4 lines
Usually, when system is busy, during the wakeup when we could not locate idle cpu, we pick the search point instead, whatever how busy it is since we count on the balance routine later to help balance the load.
But above you said that dbench usually triggers the wake-affine logic, but now you say it doesn't and we rely on
select_idle_sibling?
During wakeup, it triggered wake-affine, after that, go inside select_idle_sibling() and found no idle cpu, than pick the search point instead (curr cpu if wake-affine or prev cpu if not).
Quoted 3 lines
Note that the comparison isn't fair, running dbench on an idle system vs running dbench on a busy system is the first step.
Our comparison is based on the same busy-system, all the two cases have the same workload running, the only difference is that we put the same workload (dbench + stress) one group deeper, it's like:
Good case:
root
l1-A l1-B l1-C
dbench stress stress
results:
dbench got around 300%
each stress got around 450%
Bad case:
root
l1
l2-A l2-B l2-C
dbench stress stress
results:
dbench got around 100% (throughout dropped too) each stress got around 550%
Although the l1-group gain the same resources (1200%), it doesn't assign to l2-ABC correctly like the root-group did.
Quoted 9 lines
The second is adding the cgroup crap on.
Quoted 2 lines
However, in our cases the load balance could not help on that, since deeper the group is, less the load effect it means to root group.
But since all actual load is on the same depth, the relative threshold (imbalance pct) should work the same, the size of the values don't matter, the relative ratios do.
Exactly, however, when group is deep, the chance of it to make root imbalance reduced, in good case, gathered on cpu means 1024 load, while in bad case it dropped to 1024/3 ideally, that make it harder to trigger imbalance and gain help from the routine, please note that although dbench and stress are the only workload in system, there are still other tasks serve for the system need to be wakeup (some very actively since the dbench...), compared to them, deep group load means nothing...
Quoted 6 lines
Quoted 4 lines
By which means even tasks in deep group all gathered on one CPU, the load could still balanced from the view of root group, and the tasks lost the only chances (balance) to spread when they already on the same CPU...
Sure, but see above.
The lb-routine could not provide enough help for deep group, since the imbalance happened inside the group could not cause imbalance in root, ideally each l2-task will gain 1024/18 ~= 56 root-load, which could be easily ignored, but inside the l2-group, the gathered case could already means imbalance like (1024 * 5) : 1024.
Quoted 8 lines
Quoted 3 lines
Furthermore, for tasks flip frequently like dbench, it'll become far more harder for load balance to help, it could even rarely catch them on rq.
And I suspect that is the main problem; so see what it does on a busy
system: !cgroup:nr_cpusbusy loops + dbench, because that's your benchmark for adding cgroups, the cgroup can only shift that behaviour around.
There are busy loops in good case too, and dbench behaviour in l1-groups should not changed after put them to l2-group, what make things worse is the chance for them to spread after gathered become less.
Quoted 1 line
[snip]
Quoted 8 lines
Quoted 3 lines
Below patch has solved the problem during the testing, I'd like to do more testing on other benchmarks before send out the formal patch, any comments are welcomed ;-)
So I think that approach is wrong,
select_idle_siblings()works because we want to keep CPUs from being idle, but if they're not actually idle, pretending like they are (in a cgroup) is actively wrong and can skew load pretty bad.
We only choose the timing when no idle cpu located, and flips is somewhat high, also the group is deep.
In such cases, select_idle_siblings() doesn't works anyway, it return the target even it is very busy, we just check twice to prevent it from making some obviously bad decision ;-)
Quoted 4 lines
Furthermore, if as I expect, dbench sucks on a busy system, then the proposed cgroup thing is wrong, as a cgroup isn't supposed to radically alter behaviour like that.
That's true and that's why we currently still need to shut down the GENTLE_FAIR_SLEEPERS feature, but that's another problem we need to solve later...
What we currently expect is that the cgroup assign the resource according to the share, it works well in l1-groups, so we expect it to work the same well in l2-groups...
Quoted 6 lines
More so, I suspect that patch will tend to overload cpu0 (and lower cpu numbers in general -- because its scanning in the same direction for each cgroup) for other workloads. You can't just go pile more and more work on cpu0 just because there's nothing running in this particular cgroup.
That's a good point...
However during the testing, this doesn't happen on the 3 groups, tasks stay on high-cpu as often as low-cpu, IMHO the key point here is the lb-routine still works, although much less than before.
So the fix just make the result of lb-routine effect longer, since the higher cpu it picked is usually idle in group (directly pick later), in other word, tasks on high-cpu is harder to be wake-affine to low-cpu than before.
And when this apply to all the groups, each of them will be balanced both internally and externally, then we will see equal tasks on each cpu.
select_idle_sibling() do pick low-cpu more often, and combined with wake-affine, without enough load-balance, the tasks will gathered on low-cpu more often, but our solution will make the less load-balance become more valuable (when they need to be), IMHO, it could even contribute to balance work in some cases...
Quoted 4 lines
So dbench is very sensitive to queueing, and
select_idle_siblings()avoids a lot of queueing on an idle system. I don't think that's something we should fix with cgroups.
It has to queue anyway after wakeup, isn't it? we just want a good candidate which won't make things too bad inside group, and only do this when select_idle_siblings() give up on searching...
Regards,
Michael Wang
Quoted 1 line
Peter Zijlstra
maintainerQuoted 10 lines · Michael wang
Hi, Peter
Thanks for the reply :)
On 06/10/2014 08:12 PM, Peter Zijlstra wrote:
[snip]Quoted 4 lines
Quoted 3 lines
Wake-affine for sure pull tasks together for workload like dbench, what make it difference when put dbench into a group one level deeper is the load-balance, which happened less.
Quoted 7 lines
Quoted 2 lines
We load-balance less (frequently) or we migrate less tasks due to load-balancing ?
IMHO, when we put tasks one group deeper, in other word the totally weight of these tasks is 1024 (prev is 3072), the load become more balancing in root, which make bl-routine consider the system is balanced, which make we migrate less in lb-routine.
But how? The absolute value (1024 vs 3072) is of no effect to the imbalance, the imbalance is computed from relative differences between cpus.
Quoted 19 lines
Our comparison is based on the same busy-system, all the two cases have the same workload running, the only difference is that we put the same workload (dbench + stress) one group deeper, it's like:
Good case:
root
l1-A l1-B l1-C
dbench stress stressresults:
dbench got around 300%
each stress got around 450%Bad case:
root
l1
l2-A l2-B l2-C
dbench stress stress
Quoted 6 lines
results:
dbench got around 100% (throughout dropped too) each stress got around 550%Although the l1-group gain the same resources (1200%), it doesn't assign to l2-ABC correctly like the root-group did.
But in this case select_idle_sibling() should function identially, so that cannot be the problem.
Quoted 9 lines
Quoted 8 lines
The second is adding the cgroup crap on.
Quoted 2 lines
However, in our cases the load balance could not help on that, since deeper the group is, less the load effect it means to root group.
But since all actual load is on the same depth, the relative threshold (imbalance pct) should work the same, the size of the values don't matter, the relative ratios do.
Quoted 7 lines
Exactly, however, when group is deep, the chance of it to make root imbalance reduced, in good case, gathered on cpu means 1024 load, while in bad case it dropped to 1024/3 ideally, that make it harder to trigger imbalance and gain help from the routine, please note that although dbench and stress are the only workload in system, there are still other tasks serve for the system need to be wakeup (some very actively since the dbench...), compared to them, deep group load means nothing...
What tasks are these? And is it their interference that disturbs load-balancing?
Quoted 11 lines
Quoted 5 lines
Quoted 3 lines
By which means even tasks in deep group all gathered on one CPU, the load could still balanced from the view of root group, and the tasks lost the only chances (balance) to spread when they already on the same CPU...
Sure, but see above.
The lb-routine could not provide enough help for deep group, since the imbalance happened inside the group could not cause imbalance in root, ideally each l2-task will gain 1024/18 ~= 56 root-load, which could be easily ignored, but inside the l2-group, the gathered case could already means imbalance like (1024 * 5) : 1024.
your explanation is not making sense, we have 3 cgroups, so the total root weight is at least 3072, with 18 tasks you would get 3072/18 ~ 170.
And again, the absolute value doesn't matter, with (istr) 12 cpus the avg cpu load would be 3072/12 ~ 256, and 170 is significant on that scale.
Same with l2, total weight of 1024, giving a per task weight of ~56 and a per-cpu weight of ~85, which is again significant.
Also, you said load-balance doesn't usually participate much because dbench is too fast, so please make up your mind, does it or doesn't it matter?
Quoted 7 lines
Quoted 4 lines
So I think that approach is wrong,
select_idle_siblings()works because we want to keep CPUs from being idle, but if they're not actually idle, pretending like they are (in a cgroup) is actively wrong and can skew load pretty bad.We only choose the timing when no idle cpu located, and flips is somewhat high, also the group is deep.
-enotmakingsense
Quoted 3 lines
In such cases,
select_idle_siblings()doesn't works anyway, it return the target even it is very busy, we just check twice to prevent it from making some obviously bad decision ;-)
-emakinglesssense
Quoted 7 lines
Quoted 3 lines
Furthermore, if as I expect, dbench sucks on a busy system, then the proposed cgroup thing is wrong, as a cgroup isn't supposed to radically alter behaviour like that.
That's true and that's why we currently still need to shut down the
GENTLE_FAIR_SLEEPERSfeature, but that's another problem we need to solve later...
more confusion..
Quoted 3 lines
What we currently expect is that the cgroup assign the resource according to the share, it works well in l1-groups, so we expect it to work the same well in l2-groups...
Sure, but explain why it isn't? So far you're just saying words that don't compute.
On 06/11/2014 04:24 PM, Peter Zijlstra wrote:
[snip]
Quoted 9 lines
Quoted 5 lines
IMHO, when we put tasks one group deeper, in other word the totally weight of these tasks is 1024 (prev is 3072), the load become more balancing in root, which make bl-routine consider the system is balanced, which make we migrate less in lb-routine.
But how? The absolute value (1024 vs 3072) is of no effect to the imbalance, the imbalance is computed from relative differences between cpus.
Ok, forgive me for the confusion, please allow me to explain things again, for gathered cases like:
cpu 0 cpu 1 dbench task_sys dbench task_sys dbench dbench dbench dbench task_sys task_sys
task_sys is other tasks belong to root which is nice 0, so when dbench in l1:
cpu 0 cpu 1 load 1024 + 1024*2 1024*2 3072: 2048 imbalance %150
now when they belong to l2:
cpu 0 cpu 1 load 1024/3 + 1024*2 1024*2 2389 : 2048 imbalance %116
And it could be even less during my testing...
This is just try to explain that when 'group_load : rq_load' become lower, it's influence to 'rq_load' become lower too, and if the system is balanced with only 'rq_load' there, it will be considered still balanced even 'group_load' gathered on one cpu.
Please let me know if I missed something here...
Quoted 1 line
[snip]
Quoted 6 lines
Quoted 3 lines
Although the l1-group gain the same resources (1200%), it doesn't assign to l2-ABC correctly like the root-group did.
But in this case
select_idle_sibling()should function identially, so that cannot be the problem.
Yes, it's clean, select_idle_sibling() just return curr or prev cpu in this case.
Quoted 1 line
[snip]
Quoted 11 lines
Quoted 8 lines
Exactly, however, when group is deep, the chance of it to make root imbalance reduced, in good case, gathered on cpu means 1024 load, while in bad case it dropped to 1024/3 ideally, that make it harder to trigger imbalance and gain help from the routine, please note that although dbench and stress are the only workload in system, there are still other tasks serve for the system need to be wakeup (some very actively since the dbench...), compared to them, deep group load means nothing...
What tasks are these? And is it their interference that disturbs load-balancing?
These are dbench and stress with less root-load when put into l2-groups, that make it harder to trigger root-group imbalance like in the case above.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
By which means even tasks in deep group all gathered on one CPU, the load could still balanced from the view of root group, and the tasks lost the only chances (balance) to spread when they already on the same CPU...
Sure, but see above.
Quoted 8 lines
Quoted 5 lines
The lb-routine could not provide enough help for deep group, since the imbalance happened inside the group could not cause imbalance in root, ideally each l2-task will gain 1024/18 ~= 56 root-load, which could be easily ignored, but inside the l2-group, the gathered case could already means imbalance like (1024 * 5) : 1024.
your explanation is not making sense, we have 3 cgroups, so the total root weight is at least 3072, with 18 tasks you would get 3072/18 ~ 170.
I mean the l2-groups case here... since l1 share is 1024, the total load of l2-groups will be 1024 by theory.
Quoted 7 lines
And again, the absolute value doesn't matter, with (istr) 12 cpus the avg cpu load would be 3072/12 ~ 256, and 170 is significant on that scale.
Same with l2, total weight of 1024, giving a per task weight of ~56 and a per-cpu weight of ~85, which is again significant.
We have other tasks which has to running in the system, in order to serve dbench and others, and that also the case in real world, dbench and stress are not the only tasks on rq time to time.
May be we could focus on the case above and see if it could make things more clear firstly?
Regards,
Michael Wang
Quoted 38 lines
Also, you said load-balance doesn't usually participate much because dbench is too fast, so please make up your mind, does it or doesn't it matter?
Quoted 7 lines
Quoted 4 lines
So I think that approach is wrong,
select_idle_siblings()works because we want to keep CPUs from being idle, but if they're not actually idle, pretending like they are (in a cgroup) is actively wrong and can skew load pretty bad.We only choose the timing when no idle cpu located, and flips is somewhat high, also the group is deep.
-enotmakingsense
Quoted 3 lines
In such cases,
select_idle_siblings()doesn't works anyway, it return the target even it is very busy, we just check twice to prevent it from making some obviously bad decision ;-)-emakinglesssense
Quoted 7 lines
Quoted 3 lines
Furthermore, if as I expect, dbench sucks on a busy system, then the proposed cgroup thing is wrong, as a cgroup isn't supposed to radically alter behaviour like that.
That's true and that's why we currently still need to shut down the
GENTLE_FAIR_SLEEPERSfeature, but that's another problem we need to solve later...more confusion..
Quoted 3 lines
What we currently expect is that the cgroup assign the resource according to the share, it works well in l1-groups, so we expect it to work the same well in l2-groups...
Sure, but explain why it isn't? So far you're just saying words that don't compute.
Peter Zijlstra
maintainerQuoted 15 lines · Michael wang
On 06/11/2014 04:24 PM, Peter Zijlstra wrote:
[snip]Quoted 9 lines
Quoted 5 lines
IMHO, when we put tasks one group deeper, in other word the totally weight of these tasks is 1024 (prev is 3072), the load become more balancing in root, which make bl-routine consider the system is balanced, which make we migrate less in lb-routine.
But how? The absolute value (1024 vs 3072) is of no effect to the imbalance, the imbalance is computed from relative differences between cpus.
Ok, forgive me for the confusion, please allow me to explain things again, for gathered cases like:
Quoted 10 lines
cpu 0 cpu 1 dbench task_sys dbench task_sys dbench dbench dbench dbench task_sys task_sys
It might help if you prefix each task with the cgroup they're in; but I think I get it, its like:
cpu0
A/dbench
A/dbench
A/dbench
A/dbench
A/dbench
A/dbench
/task_sys
/task_sys
Quoted 9 lines
task_sysis other tasks belong to root which is nice 0, so when dbench in l1:cpu 0 cpu 1 load 1024 + 1024*2 1024*2 3072: 2048 imbalance %150now when they belong to l2:
That would be:
cpu0
A/B/dbench
A/B/dbench
A/B/dbench
A/B/dbench
A/B/dbench
A/B/dbench
/task_sys
/task_sys
Right?
Quoted 4 lines
cpu 0 cpu 1 load 1024/3 + 1024*2 1024*2 2389 : 2048 imbalance %116
Which should still end up with 3072, because A is still 1024 in total, and all its member tasks run on the one CPU.
Quoted 1 line
And it could be even less during my testing...
Well, yes, up to 1024/nr_cpus I imagine.
Quoted 6 lines
This is just try to explain that when '
group_load:rq_load' become lower, it's influence to 'rq_load' become lower too, and if the system is balanced with only 'rq_load' there, it will be considered still balanced even 'group_load' gathered on one cpu.Please let me know if I missed something here...
Yeah, what other tasks are these task_sys things? workqueue crap?
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Exactly, however, when group is deep, the chance of it to make root imbalance reduced, in good case, gathered on cpu means 1024 load, while in bad case it dropped to 1024/3 ideally, that make it harder to trigger imbalance and gain help from the routine, please note that although dbench and stress are the only workload in system, there are still other tasks serve for the system need to be wakeup (some very actively since the dbench...), compared to them, deep group load means nothing...
Quoted 5 lines
Quoted 2 lines
What tasks are these? And is it their interference that disturbs load-balancing?
These are dbench and stress with less root-load when put into l2-groups, that make it harder to trigger root-group imbalance like in the case above.
You're still not making sense here.. without the task_sys thingies in you get something like:
cpu0 cpu1
A/dbench A/dbench
B/stress B/stress
And the total loads are: 512+512 vs 512+512.
Quoted 9 lines
Quoted 2 lines
Same with l2, total weight of 1024, giving a per task weight of ~56 and a per-cpu weight of ~85, which is again significant.
We have other tasks which has to running in the system, in order to serve dbench and others, and that also the case in real world, dbench and stress are not the only tasks on rq time to time.
May be we could focus on the case above and see if it could make things more clear firstly?
Well, this all smells like you need some cgroup affinity for whatever system tasks are running. Not fuck up the scheduler for no sane reason.
Hi, Peter
Thanks for the reply :)
On 06/23/2014 05:42 PM, Peter Zijlstra wrote:
[snip]
Quoted 12 lines
Quoted 11 lines
cpu 0 cpu 1 dbench task_sys dbench task_sys dbench dbench dbench dbench task_sys task_sys
Quoted 1 line
It might help if you prefix each task with the cgroup they're in;
My bad...
but I
Quoted 4 lines
think I get it, its like:
cpu0
Quoted 8 lines
A/dbench
A/dbench
A/dbench
A/dbench
A/dbench
A/dbench
/task_sys
/task_sys
Yeah, it's like that.
Quoted 1 line
[snip]
Quoted 12 lines
cpu0
A/B/dbench
A/B/dbench
A/B/dbench
A/B/dbench
A/B/dbench
A/B/dbench
/task_sys
/task_sys
Quoted 1 line
Right?
My bad to missed the group symbol here... it's actually like:
cpu0 /l1/A/dbench /l1/A/dbench /l1/A/dbench /l1/A/dbench /l1/A/dbench /task_sys /task_sys
And we also have six:
/l1/B/stress
and six:
/l1/C/stress
running in system.
A, B, C is the child groups of l1.
Quoted 8 lines
Quoted 5 lines
cpu 0 cpu 1 load 1024/3 + 1024*2 1024*2 2389 : 2048 imbalance %116Which should still end up with 3072, because A is still 1024 in total, and all its member tasks run on the one CPU.
l1 have 3 child groups, each got 6 NICE 0 tasks, so ideally each task will got 1024/18, 6 dbench will means (1024/18)*6 == 1024/3.
Previously each of the 3 group got 1024 shares, now they need to share 1024 shares, it will become less for each of them.
Quoted 5 lines
Quoted 2 lines
And it could be even less during my testing...
Well, yes, up to 1024/nr_cpus I imagine.
Quoted 8 lines
Quoted 6 lines
This is just try to explain that when '
group_load:rq_load' become lower, it's influence to 'rq_load' become lower too, and if the system is balanced with only 'rq_load' there, it will be considered still balanced even 'group_load' gathered on one cpu.Please let me know if I missed something here...
Yeah, what other tasks are these
task_systhings? workqueue crap?
There are some other tasks but mostly showup are the kworkers, yes the workqueue stuff.
They rapidly showup on each CPU, in some period if they showup too much, they will eat some CPU% too, but not very much.
Quoted 1 line
[snip]
Quoted 7 lines
Quoted 3 lines
These are dbench and stress with less root-load when put into l2-groups, that make it harder to trigger root-group imbalance like in the case above.
You're still not making sense here.. without the
task_systhingies in you get something like:
Quoted 6 lines
cpu0 cpu1
A/dbench A/dbench
B/stress B/stressAnd the total loads are: 512+512 vs 512+512.
Without other task's influence, I believe the balance should be fine, but in our cases, at least these kworkers will join the battle anyway...
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Same with l2, total weight of 1024, giving a per task weight of ~56 and a per-cpu weight of ~85, which is again significant.
We have other tasks which has to running in the system, in order to serve dbench and others, and that also the case in real world, dbench and stress are not the only tasks on rq time to time.
Quoted 5 lines
Quoted 2 lines
May be we could focus on the case above and see if it could make things more clear firstly?
Well, this all smells like you need some cgroup affinity for whatever system tasks are running. Not fuck up the scheduler for no sane reason.
These kworkers are bind to their CPU already, I don't know how to handle them to prevent the issue, they just keep working on their CPU, and whenever they showup, dbench spreading inactively...
We just want a way which could help workload like dbench to work normally with cpu-group when there are stress likely workload running in the system.
We want dbench to gain more CPU% but cpu-shares doesn't work as expected... dbench can get no more than 100% whatever how big it's group's shares is, and we consider that cpu-group was broken in this cases...
I agree that this is not a generic requirement and scheduler should only be responsible for general situation, but since it's really a too big regression, could we at least provide some way to stop the damage? After all, most of the cpu-group logic is insider scheduler...
I'd like to list some real numbers in patch-thread, we really desired for some way to make cpu-group perform normally on workload like dbench, actually we also find some transaction workloads suffered from this issue too, in such cases, cpu-group just failed on managing the CPU resources...
Regards,
Michael Wang
Quoted 1 line
Peter Zijlstra
maintainerQuoted 9 lines · Michael wang
On 05/15/2014 07:57 PM, Peter Zijlstra wrote:
[snip]Quoted 7 lines
Quoted 6 lines
It's like:
/cgroup/cpu/l1/l2/l3/l4/l5/l6/Aabout level 7, the issue can not be solved any more.
Quoted 8 lines
Quoted 4 lines
That's pretty retarded and yeah, that's way past the point where things make sense. You might be lucky and have l1-5 as empty/pointless hierarchy so the effective depth is less and then things will work, but *shees*..
Exactly, that's the simulation of cgroup topology setup by libvirt, really doesn't make sense... rather torture than deployment, but they do make things like that...
I'm calling it broken and unfit for purpose if it does crazy shit like that.
There's really not much we can do to fix it either, barring softfloat in the load-balancer and I'm sure everybody but virt wankers will complain about _that_.
On 05/13/2014 09:36 PM, Rik van Riel wrote:
[snip]
Quoted 10 lines
Quoted 4 lines
echo 2048 > /cgroup/c/cpu.shares
Where [ABC].sh are spinners:
I suspect the "are spinners" is key.
Infinite loops can run all the time, while dbench spends a lot of its time waiting for locks. That waiting may interfere with getting as much CPU as it wants.
That's what we are thinking, also we assume that by introducing load decay mechanism, it become harder for the sleepy tasks to gain enough slice, well, that currently just imagination, more investigation is needed ;-)
Regards,
Michael Wang
Quoted 1 line
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Michael wang
During our testing, we found that the cpu.shares doesn't work as expected, the testing is:
/me zaps all the kvm nonsense as that's non reproducable and only serves to annoy.
Pro-tip: never use kvm to report cpu-cgroup issues.
Make sense.
Quoted 1 line
[snip]
Quoted 1 line
for i in A B C ; do ps -deo pcpu,cmd | grep "${i}\.sh" | awk '{t += $1} END {print t}' ; done
Enjoyable :)
Quoted 5 lines
639.7
629.8
1127.4That is of course not perfect, but it's close enough.
Yeah, for cpu intensive work load, the share do work very well, the issue only appeared when workload start to become some kind of...sleepy.
I will use the tool you mentioned for the following investigation, thanks for the suggestion.
Quoted 2 lines
Now you again.. :-)
And here I am ;-)
Regards,
Michael Wang