busted CFS group load balancer?
It appears that the fair-group load balancer in 2.6.27 does not work properly. It broke down severely with real-world application workload here at google. We are trying to use CFS cgroup scheduler as a proportional scheduler to control CPU cycle allocation for concurrent groups of tasks on the same server machine. For example, we have one cgroup named 'bee' which has a cfs weight of 1024, and another cgroup named 'ant' that has a cfs weight of 2. Ideally, I would expect 99.8% of CPU cycles goes into 'bee' cgroup while the remaining 0.2% goes into 'ant'. However, when evaluating 2.6.27, the result deviates significantly from the expectation. It appears that the fair-group load balancer is sub par and not doing its job properly.
Attached is a snapshot of /proc/sched_debug. Two CPUs were dominated by tasks of a very low weight 'ant' cgroup, while multiple tasks in higher weight 'bee' cgroup landed on the same CPU and competing among themselves. I think the load balancer does its job poorly. Have you seen similar l-b deficiency?
- Ken
Sched Debug Version: v0.07, 2.6.27 #1
now at 62931690.049344 msecs
.sysctl_sched_latency : 80.000000 .sysctl_sched_min_granularity : 16.000000 .sysctl_sched_wakeup_granularity : 20.000000 .sysctl_sched_child_runs_first : 0.000001 .sysctl_sched_features : 7935
cpu#0, 2333.417 MHz
.nr_running : 2 .load : 103 .nr_switches : 12943039 .nr_load_updates : 62738664 .nr_uninterruptible : -260 .jiffies : 4357598986 .next_balance : 4357.599094 .curr->pid : 12721 .clock : 62969376.294574 .cpu_load[0] : 103 .cpu_load[1] : 121 .cpu_load[2] : 143 .cpu_load[3] : 158 .cpu_load[4] : 166
cfs_rq[0]:/ant
.exec_clock : 53371626.919067 .MIN_vruntime : 60790257.707167 .min_vruntime : 44998074.291781 .max_vruntime : 60790257.707167 .spread : 0.000000 .spread0 : 0.000000 .nr_running : 1 .load : 1024 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 14251 .sched_switch : 0 .sched_count : 12981786 .sched_goidle : 288336 .ttwu_count : 6565365 .ttwu_local : 5442677 .bkl_count : 46781 .nr_spread_over : 12567 .shares : 0
cfs_rq[0]:/bee
.exec_clock : 9183712.055274 .MIN_vruntime : 0.000001 .min_vruntime : 44998074.291781 .max_vruntime : 0.000001 .spread : 0.000000 .spread0 : 0.000000 .nr_running : 1 .load : 1024 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 14251 .sched_switch : 0 .sched_count : 12981786 .sched_goidle : 288336 .ttwu_count : 6565365 .ttwu_local : 5442677 .bkl_count : 46781 .nr_spread_over : 442289 .shares : 101
cfs_rq[0]:/
.exec_clock : 62567144.106498 .MIN_vruntime : 44998074.291781 .min_vruntime : 44998074.291781 .max_vruntime : 44998074.291781 .spread : 0.000000 .spread0 : 0.000000 .nr_running : 2 .load : 103 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 14251 .sched_switch : 0 .sched_count : 12981786 .sched_goidle : 288336 .ttwu_count : 6565365 .ttwu_local : 5442677 .bkl_count : 46781 .nr_spread_over : 43480 .shares : 0
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
hog 4202 60790257.707167 5421910 120 60790257.707167 53371613.637974 0.000000 /ant R hpro 12721 10434396.891434 425169 120 10434396.891434 2351102.444642 1015320.568226 /bee
cpu#1, 2333.417 MHz
.nr_running : 2 .load : 102 .nr_switches : 35564399 .nr_load_updates : 62728095 .nr_uninterruptible : 371 .jiffies : 4357598986 .next_balance : 4357.599133 .curr->pid : 12727 .clock : 62969376.338300 .cpu_load[0] : 103 .cpu_load[1] : 109 .cpu_load[2] : 121 .cpu_load[3] : 136 .cpu_load[4] : 150
cfs_rq[1]:/ant
.exec_clock : 50553231.251769 .MIN_vruntime : 98877044.002500 .min_vruntime : 49317332.550674 .max_vruntime : 98877044.002500 .spread : 0.000000 .spread0 : 4319258.258893 .nr_running : 1 .load : 1024 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 19005 .sched_switch : 0 .sched_count : 35601762 .sched_goidle : 284437 .ttwu_count : 18517385 .ttwu_local : 16585787 .bkl_count : 24693 .nr_spread_over : 21321 .shares : 0
cfs_rq[1]:/bee
.exec_clock : 12002883.375364 .MIN_vruntime : 0.000001 .min_vruntime : 49317332.550674 .max_vruntime : 0.000001 .spread : 0.000000 .spread0 : 4319258.258893 .nr_running : 1 .load : 1024 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 19005 .sched_switch : 0 .sched_count : 35601762 .sched_goidle : 284437 .ttwu_count : 18517385 .ttwu_local : 16585787 .bkl_count : 24693 .nr_spread_over : 341849 .shares : 100
cfs_rq[1]:/
.exec_clock : 62568079.552049 .MIN_vruntime : 49317332.550674 .min_vruntime : 49317332.550674 .max_vruntime : 49317332.550674 .spread : 0.000000 .spread0 : 4319258.258893 .nr_running : 2 .load : 102 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 19005 .sched_switch : 0 .sched_count : 35601762 .sched_goidle : 284437 .ttwu_count : 18517385 .ttwu_local : 16585787 .bkl_count : 24693 .nr_spread_over : 50967 .shares : 0
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
hog 4200 98877044.002500 14797192 120 98877044.002500 50553231.092595 0.000000 /ant R hpro 12727 14039095.103079 422222 120 14039095.103079 2347785.641546 1017814.287309 /bee
cpu#2, 2333.417 MHz
.nr_running : 3 .load : 205 .nr_switches : 5452324 .nr_load_updates : 62703726 .nr_uninterruptible : 1143 .jiffies : 4357598986 .next_balance : 4357.599024 .curr->pid : 12728 .clock : 62969376.294927 .cpu_load[0] : 205 .cpu_load[1] : 198 .cpu_load[2] : 190 .cpu_load[3] : 184 .cpu_load[4] : 183
cfs_rq[2]:/ant
.exec_clock : 53265000.243513 .MIN_vruntime : 62505737.284348 .min_vruntime : 44986496.033419 .max_vruntime : 62505737.284348 .spread : 0.000000 .spread0 : -11578.258362 .nr_running : 1 .load : 1024 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 14697 .sched_switch : 0 .sched_count : 5470658 .sched_goidle : 47945 .ttwu_count : 2792641 .ttwu_local : 1831019 .bkl_count : 43426 .nr_spread_over : 16899 .shares : 0
cfs_rq[2]:/bee
.exec_clock : 9290734.241942 .MIN_vruntime : 10838701.944779 .min_vruntime : 44986496.033419 .max_vruntime : 10838701.944779 .spread : 0.000000 .spread0 : -11578.258362 .nr_running : 2 .load : 2048 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 14697 .sched_switch : 0 .sched_count : 5470658 .sched_goidle : 47945 .ttwu_count : 2792641 .ttwu_local : 1831019 .bkl_count : 43426 .nr_spread_over : 403151 .shares : 203
cfs_rq[2]:/
.exec_clock : 62557630.621219 .MIN_vruntime : 44986496.033419 .min_vruntime : 44986496.033419 .max_vruntime : 44986496.033419 .spread : 0.000000 .spread0 : -11578.258362 .nr_running : 2 .load : 205 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 14697 .sched_switch : 0 .sched_count : 5470658 .sched_goidle : 47945 .ttwu_count : 2792641 .ttwu_local : 1831019 .bkl_count : 43426 .nr_spread_over : 83787 .shares : 0
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
hog 4205 62505737.284348 1919921 120 62505737.284348 53264991.605391 0.000000 /ant
hpro 12726 10838701.944779 423772 120 10838701.944779 2344927.648792 1017455.081051 /bee
R hpro 12728 10838780.058709 421699 120 10838780.058709 2345123.009711 1019061.435473 /beecpu#3, 2333.417 MHz
.nr_running : 1 .load : 2 .nr_switches : 4682853 .nr_load_updates : 62711242 .nr_uninterruptible : -1414 .jiffies : 4357598986 .next_balance : 4357.599007 .curr->pid : 4204 .clock : 62969376.294206 .cpu_load[0] : 2 .cpu_load[1] : 48 .cpu_load[2] : 68 .cpu_load[3] : 78 .cpu_load[4] : 82
cfs_rq[3]:/ant
.exec_clock : 53314990.134015 .MIN_vruntime : 0.000001 .min_vruntime : 44976333.787158 .max_vruntime : 0.000001 .spread : 0.000000 .spread0 : -21740.504623 .nr_running : 1 .load : 1024 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 14720 .sched_switch : 0 .sched_count : 4699788 .sched_goidle : 23093 .ttwu_count : 2413046 .ttwu_local : 1516294 .bkl_count : 38944 .nr_spread_over : 36602 .shares : 0
cfs_rq[3]:/bee
.exec_clock : 9240891.195912 .MIN_vruntime : 0.000001 .min_vruntime : 44976333.787158 .max_vruntime : 0.000001 .spread : 0.000000 .spread0 : -21740.504623 .nr_running : 0 .load : 0 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 14720 .sched_switch : 0 .sched_count : 4699788 .sched_goidle : 23093 .ttwu_count : 2413046 .ttwu_local : 1516294 .bkl_count : 38944 .nr_spread_over : 368326 .shares : 101
cfs_rq[3]:/
.exec_clock : 62560353.760642 .MIN_vruntime : 0.000001 .min_vruntime : 44976333.787158 .max_vruntime : 0.000001 .spread : 0.000000 .spread0 : -21740.504623 .nr_running : 1 .load : 2 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 14720 .sched_switch : 0 .sched_count : 4699788 .sched_goidle : 23093 .ttwu_count : 2413046 .ttwu_local : 1516294 .bkl_count : 38944 .nr_spread_over : 278060 .shares : 0
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
R hog 4204 61265712.229407 1590630 120 61265712.229407 53314966.642964 0.000000 /ant
cpu#4, 2333.417 MHz
.nr_running : 3 .load : 205 .nr_switches : 6034988 .nr_load_updates : 62701982 .nr_uninterruptible : -489 .jiffies : 4357598986 .next_balance : 4357.599098 .curr->pid : 12722 .clock : 62969376.294969 .cpu_load[0] : 205 .cpu_load[1] : 198 .cpu_load[2] : 190 .cpu_load[3] : 181 .cpu_load[4] : 163
cfs_rq[4]:/ant
.exec_clock : 57463524.397256 .MIN_vruntime : 61439338.118634 .min_vruntime : 43422727.063373 .max_vruntime : 61439338.118634 .spread : 0.000000 .spread0 : -1575347.228408 .nr_running : 1 .load : 1024 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 7979 .sched_switch : 0 .sched_count : 6043766 .sched_goidle : 18552 .ttwu_count : 3180073 .ttwu_local : 2221510 .bkl_count : 8680 .nr_spread_over : 19946 .shares : 0
cfs_rq[4]:/bee
.exec_clock : 5093116.134219 .MIN_vruntime : 5520870.121561 .min_vruntime : 43422727.063373 .max_vruntime : 5520870.121561 .spread : 0.000000 .spread0 : -1575347.228408 .nr_running : 2 .load : 2048 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 7979 .sched_switch : 0 .sched_count : 6043766 .sched_goidle : 18552 .ttwu_count : 3180073 .ttwu_local : 2221510 .bkl_count : 8680 .nr_spread_over : 369128 .shares : 203
cfs_rq[4]:/
.exec_clock : 62557994.778412 .MIN_vruntime : 43422727.063373 .min_vruntime : 43422727.063373 .max_vruntime : 43422727.063373 .spread : 0.000000 .spread0 : -1575347.228408 .nr_running : 2 .load : 205 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 7979 .sched_switch : 0 .sched_count : 6043766 .sched_goidle : 18552 .ttwu_count : 3180073 .ttwu_local : 2221510 .bkl_count : 8680 .nr_spread_over : 191336 .shares : 0
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
hog 4199 61439338.118634 2252113 120 61439338.118634 57463477.501906 0.000000 /ant
R hpro 12722 5520893.049620 423940 120 5520893.049620 2353601.970167 1016247.261338 /bee
hpro 12725 5520870.121561 417763 120 5520870.121561 2352852.333553 1020343.133648 /beecpu#5, 2333.417 MHz
.nr_running : 2 .load : 102 .nr_switches : 36228963 .nr_load_updates : 62712542 .nr_uninterruptible : 1684 .jiffies : 4357598986 .next_balance : 4357.599094 .curr->pid : 12724 .clock : 62969376.294760 .cpu_load[0] : 103 .cpu_load[1] : 109 .cpu_load[2] : 115 .cpu_load[3] : 112 .cpu_load[4] : 105
cfs_rq[5]:/ant
.exec_clock : 50479803.539169 .MIN_vruntime : 97702539.108896 .min_vruntime : 49951986.086422 .max_vruntime : 97702539.108896 .spread : 0.000000 .spread0 : 4953911.794641 .nr_running : 1 .load : 1024 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 19372 .sched_switch : 0 .sched_count : 36250568 .sched_goidle : 40214 .ttwu_count : 19127707 .ttwu_local : 17723882 .bkl_count : 1632 .nr_spread_over : 17524 .shares : 0
cfs_rq[5]:/bee
.exec_clock : 12079214.598498 .MIN_vruntime : 0.000001 .min_vruntime : 49951986.086422 .max_vruntime : 0.000001 .spread : 0.000000 .spread0 : 4953911.794641 .nr_running : 1 .load : 1024 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 19372 .sched_switch : 0 .sched_count : 36250568 .sched_goidle : 40214 .ttwu_count : 19127707 .ttwu_local : 17723882 .bkl_count : 1632 .nr_spread_over : 257170 .shares : 100
cfs_rq[5]:/
.exec_clock : 62560310.280302 .MIN_vruntime : 49951986.086422 .min_vruntime : 49951986.086422 .max_vruntime : 49951986.086422 .spread : 0.000000 .spread0 : 4953911.794641 .nr_running : 2 .load : 102 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 19372 .sched_switch : 0 .sched_count : 36250568 .sched_goidle : 40214 .ttwu_count : 19127707 .ttwu_local : 17723882 .bkl_count : 1632 .nr_spread_over : 51023 .shares : 0
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
hog 4201 97702539.108896 15313291 120 97702539.108896 50479803.305704 0.000000 /ant R hpro 12724 13553228.752335 428968 120 13553228.752335 2359267.848257 1021475.119277 /bee
cpu#6, 2333.417 MHz
.nr_running : 2 .load : 103 .nr_switches : 5740323 .nr_load_updates : 62705164 .nr_uninterruptible : -483 .jiffies : 4357598986 .next_balance : 4357.599088 .curr->pid : 18280 .clock : 62969376.299341 .cpu_load[0] : 103 .cpu_load[1] : 78 .cpu_load[2] : 47 .cpu_load[3] : 26 .cpu_load[4] : 15
cfs_rq[6]:/ant
.exec_clock : 57412441.111827 .MIN_vruntime : 60648319.346763 .min_vruntime : 43681901.531907 .max_vruntime : 60648319.346763 .spread : 0.000000 .spread0 : -1316172.759874 .nr_running : 1 .load : 1024 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 8292 .sched_switch : 0 .sched_count : 5749245 .sched_goidle : 17290 .ttwu_count : 2988054 .ttwu_local : 2083371 .bkl_count : 8904 .nr_spread_over : 11085 .shares : 0
cfs_rq[6]:/bee
.exec_clock : 5144065.252998 .MIN_vruntime : 0.000001 .min_vruntime : 43681901.531907 .max_vruntime : 0.000001 .spread : 0.000000 .spread0 : -1316172.759874 .nr_running : 1 .load : 1024 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 8292 .sched_switch : 0 .sched_count : 5749245 .sched_goidle : 17290 .ttwu_count : 2988054 .ttwu_local : 2083371 .bkl_count : 8904 .nr_spread_over : 372629 .shares : 101
cfs_rq[6]:/
.exec_clock : 62561266.440406 .MIN_vruntime : 43681901.531907 .min_vruntime : 43681901.531907 .max_vruntime : 43681901.531907 .spread : 0.000000 .spread0 : -1316172.759874 .nr_running : 2 .load : 103 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 8292 .sched_switch : 0 .sched_count : 5749245 .sched_goidle : 17290 .ttwu_count : 2988054 .ttwu_local : 2083371 .bkl_count : 8904 .nr_spread_over : 73618 .shares : 0
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
hog 4198 60648319.346763 2155112 120 60648319.346763 57412425.922637 0.000000 /ant R cat 18280 5564927.823046 3 120 5564927.823046 1.995011 0.012258 /bee
cpu#7, 2333.417 MHz
.nr_running : 2 .load : 103 .nr_switches : 6910780 .nr_load_updates : 62705080 .nr_uninterruptible : -552 .jiffies : 4357598986 .next_balance : 4357.599016 .curr->pid : 12723 .clock : 62969376.305631 .cpu_load[0] : 103 .cpu_load[1] : 99 .cpu_load[2] : 94 .cpu_load[3] : 93 .cpu_load[4] : 103
cfs_rq[7]:/ant
.exec_clock : 57417395.762701 .MIN_vruntime : 61941066.080427 .min_vruntime : 43566817.296538 .max_vruntime : 61941066.080427 .spread : 0.000000 .spread0 : -1431256.995243 .nr_running : 1 .load : 1024 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 8673 .sched_switch : 0 .sched_count : 6928312 .sched_goidle : 24889 .ttwu_count : 3645419 .ttwu_local : 2666683 .bkl_count : 16970 .nr_spread_over : 12049 .shares : 0
cfs_rq[7]:/bee
.exec_clock : 5142675.452517 .MIN_vruntime : 0.000001 .min_vruntime : 43566817.296538 .max_vruntime : 0.000001 .spread : 0.000000 .spread0 : -1431256.995243 .nr_running : 1 .load : 1024 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 8673 .sched_switch : 0 .sched_count : 6928312 .sched_goidle : 24889 .ttwu_count : 3645419 .ttwu_local : 2666683 .bkl_count : 16970 .nr_spread_over : 389009 .shares : 101
cfs_rq[7]:/
.exec_clock : 62561246.337270 .MIN_vruntime : 43566817.296538 .min_vruntime : 43566817.296538 .max_vruntime : 43566817.296538 .spread : 0.000000 .spread0 : -1431256.995243 .nr_running : 2 .load : 103 .yld_exp_empty : 0 .yld_act_empty : 0 .yld_both_empty : 0 .yld_count : 8673 .sched_switch : 0 .sched_count : 6928312 .sched_goidle : 24889 .ttwu_count : 3645419 .ttwu_local : 2666683 .bkl_count : 16970 .nr_spread_over : 31384 .shares : 0
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
hog 4203 61941066.080427 2636297 120 61941066.080427 57417288.830170 0.000000 /ant
R hpro 12723 5580819.679520 424397 120 5580819.679520 2359524.675698 1020120.649702 /bee
Quoted 2 lines · Ken Chen
It appears that the fair-group load balancer in 2.6.27 does not work properly.
There was an issue fixed post 2.6.27 where the load balancer didn't work properly if there was one task per group per cpu. You might try backporting commit 38736f4 and see if that helps.
Also, caea8a0 fixes up list traversal in load_balance_fair() to use the _rcu() version. I'm not aware of this causing any problems in practice, but it might be worth a try.
Chris
Quoted 6 lines · Chris Friesen
Quoted 2 lines
It appears that the fair-group load balancer in 2.6.27 does not work properly.
There was an issue fixed post 2.6.27 where the load balancer didn't work properly if there was one task per group per cpu. You might try backporting commit 38736f4 and see if that helps.
Tested git commit 38736f4, it doesn't fix the problem I'm seeing.
Quoted 9 lines · Ken Chen
Quoted 5 lines · Chris Friesen
Quoted 2 lines
It appears that the fair-group load balancer in 2.6.27 does not work properly.
There was an issue fixed post 2.6.27 where the load balancer didn't work properly if there was one task per group per cpu. You might try backporting commit 38736f4 and see if that helps.
Tested git commit 38736f4, it doesn't fix the problem I'm seeing.
I plugged in the same weights into my test app (groups 1 and 2 instead of ant/bee) and got the results below for a 10-sec run. The "actual" numbers give the overall average and then the values for each hog separately. In this case we see that both tasks in group 2 ended up sharing a cpu with one of the tasks from group 1.
group actual(%) expected(%) ctx switches max_latency(ms)
1 99.69(99.38/99.99) 99.81 160/262 4/0
2 0.31( 0.31/0.31) 0.19 32/33 391/375I've only got a 2-way system. If the results really are that much worse on larger systems, then that's going to cause problems for us as well. I'll see if I can get some time on a bigger machine.
Chris
Quoted 17 lines · Chris Friesen
Quoted 9 lines · Ken Chen
Quoted 5 lines · Chris Friesen
Quoted 2 lines
It appears that the fair-group load balancer in 2.6.27 does not work properly.
There was an issue fixed post 2.6.27 where the load balancer didn't work properly if there was one task per group per cpu. You might try backporting commit 38736f4 and see if that helps.
Tested git commit 38736f4, it doesn't fix the problem I'm seeing.
I plugged in the same weights into my test app (groups 1 and 2 instead of ant/bee) and got the results below for a 10-sec run. The "actual" numbers give the overall average and then the values for each hog separately. In this case we see that both tasks in group 2 ended up sharing a cpu with one of the tasks from group 1.
Quoted 7 lines
group actual(%) expected(%) ctx switches max_latency(ms) 1 99.69(99.38/99.99) 99.81 160/262 4/0 2 0.31( 0.31/0.31) 0.19 32/33 391/375I've only got a 2-way system. If the results really are that much worse on larger systems, then that's going to cause problems for us as well. I'll see if I can get some time on a bigger machine.
Note that with larger cpu count and/or lower group weight we'll quickly run into numerical trouble...
I would recommend trying this with the minimum weight in the order of 8-16 times number of cpus on your system.
There is only so much one can do with 10 bit fixed precision math :/
Quoted 7 lines · Peter Zijlstra
Note that with larger cpu count and/or lower group weight we'll quickly run into numerical trouble...
I would recommend trying this with the minimum weight in the order of 8-16 times number of cpus on your system.
There is only so much one can do with 10 bit fixed precision math :/
That is probably one of the many problems. I also found that the updates to the per-cpu task_group's sched_entity load weight (tg->se[cpu]->load.weight) is very problematic and very erratic.
The total rq_weight is calculated at one beginning of tg_shares_up(),
for_each_cpu_mask(i, sd->span) {
rq_weight += tg->cfs_rq[i]->load.weight;
shares += tg->cfs_rq[i]->shares;
}However, the scaling of per-cpu se->load.weight in function __update_group_shares_cpu() takes another lookup of tg->cfs_rq[cpu]->load.weight at a different time. cfs_rq[cpu].load.weight aren't always consistent across these two times. Due to these inconsistency of value taken on per cpu cfs_rq, I've see tg->se[cpu]->load.weight jumping all over the place. In our environment, the cpu loads are very dynamic. Process queuing/dequeuing at high rate.
I'm also very troubled with this calculation in __update_group_shares_cpu():
shares = (sd_shares * rq_weight) / (sd_rq_weight + 1);
Won't you have rounding problem here? value 'shares' will gradually decrease for each iteration of __update_group_shares_cpu()?
- Ken
Quoted 20 lines · Ken Chen
Quoted 7 lines · Peter Zijlstra
Note that with larger cpu count and/or lower group weight we'll quickly run into numerical trouble...
I would recommend trying this with the minimum weight in the order of 8-16 times number of cpus on your system.
There is only so much one can do with 10 bit fixed precision math :/
That is probably one of the many problems. I also found that the updates to the per-cpu task_group's
sched_entityload weight (tg->se[cpu]->load.weight) is very problematic and very erratic.The total
rq_weightis calculated at one beginning oftg_shares_up(),for_each_cpu_mask(i, sd->span) { rq_weight += tg->cfs_rq[i]->load.weight; shares += tg->cfs_rq[i]->shares; }
Quoted 8 lines
However, the scaling of per-cpu
se->load.weightin function__update_group_shares_cpu()takes another lookup of tg->cfs_rq[cpu]->load.weight at a different time. cfs_rq[cpu].load.weight aren't always consistent across these two times. Due to these inconsistency of value taken on per cpucfs_rq, I've see tg->se[cpu]->load.weight jumping all over the place. In our environment, the cpu loads are very dynamic. Process queuing/dequeuing at high rate.
Ok, if your load values are very unstable in the order of the load-balance interval then you're hosed too, the same is true for the normal smp load-balancer.
The cgroup load-balancer makes that even more problematic.
Again, there's just very little you can do about that, except increase the coupling between cpus and thereby increase the overhead. Try decreasingsysctl_sched_shares_ratelimit.
Quoted 6 lines
I'm also very troubled with this calculation in
__update_group_shares_cpu():shares = (sd_shares * rq_weight) / (sd_rq_weight + 1);Won't you have rounding problem here? value 'shares' will gradually decrease for each iteration of
__update_group_shares_cpu()?
Yes it will, however at the top of the sched-domain tree its reset.
if (!sd->parent || !(sd->parent->flags & SD_LOAD_BALANCE))
shares = tg->shares;Quoted 34 lines · Peter Zijlstra
Quoted 20 lines · Ken Chen
Quoted 7 lines · Peter Zijlstra
Note that with larger cpu count and/or lower group weight we'll quickly run into numerical trouble...
I would recommend trying this with the minimum weight in the order of 8-16 times number of cpus on your system.
There is only so much one can do with 10 bit fixed precision math :/
That is probably one of the many problems. I also found that the updates to the per-cpu task_group's
sched_entityload weight (tg->se[cpu]->load.weight) is very problematic and very erratic.The total
rq_weightis calculated at one beginning oftg_shares_up(),for_each_cpu_mask(i, sd->span) { rq_weight += tg->cfs_rq[i]->load.weight; shares += tg->cfs_rq[i]->shares; }Quoted 8 lines
However, the scaling of per-cpu
se->load.weightin function__update_group_shares_cpu()takes another lookup of tg->cfs_rq[cpu]->load.weight at a different time. cfs_rq[cpu].load.weight aren't always consistent across these two times. Due to these inconsistency of value taken on per cpucfs_rq, I've see tg->se[cpu]->load.weight jumping all over the place. In our environment, the cpu loads are very dynamic. Process queuing/dequeuing at high rate.Ok, if your load values are very unstable in the order of the load-balance interval then you're hosed too, the same is true for the normal smp load-balancer.
Quoted 6 lines
The cgroup load-balancer makes that even more problematic.
Again, there's just very little you can do about that, except increase the coupling between cpus and thereby increase the overhead. Try decreasing
sysctl_sched_shares_ratelimit.
Also, lower sysctl_sched_shares_thresh to 1 or 0.
Quoted 11 lines · Peter Zijlstra
Quoted 6 lines
I'm also very troubled with this calculation in
__update_group_shares_cpu():shares = (sd_shares * rq_weight) / (sd_rq_weight + 1);Won't you have rounding problem here? value 'shares' will gradually decrease for each iteration of
__update_group_shares_cpu()?Yes it will, however at the top of the sched-domain tree its reset.
if (!sd->parent || !(sd->parent->flags & SD_LOAD_BALANCE)) shares = tg->shares;
Hmm? it is only true mostly on flat smp machine, or on numa system where task is woken up across a sched-domain.
There is this code in try_to_wake_up():
for_each_domain(this_cpu, sd) {
if (cpu_isset(cpu, sd->span)) {
update_shares(sd);
break;
}
}It doesn't iterate to the top half of the time on numa machine.
- Ken
Quoted 6 lines · Chris Friesen
Quoted 2 lines
It appears that the fair-group load balancer in 2.6.27 does not work properly.
There was an issue fixed post 2.6.27 where the load balancer didn't work properly if there was one task per group per cpu. You might try backporting commit 38736f4 and see if that helps.
Quoted 1 line · Ken Chen
Tested git commit 38736f4, it doesn't fix the problem I'm seeing.
I would also like to mention that commit 38736f4 has stability problems. I put a kernel with this commit on several test machines, and several of them had hard lockups within 2 hours of testing. I wasn't doing anything fancy. The workload was pure 'while (1)' loop. Though there are other background services running on those machines.
- Ken