Reconcile NUMA balancing decisions with the load balancer v6
cover Reconcile NUMA balancing decisions with the load balancer v6
The only differences in V6 are due to Vincent's latest patch series.
This is V5 which includes the latest versions of Vincent's patch addressing review feedback. Patches 4-9 are Vincent's work plus one important performance fix. Vincent's patches were retested and while not presented in detail, it was mostly an improvement.
Changelog since V5:
o Import Vincent's latest patch set
Changelog since V4:
o The V4 send was the completely wrong versions of the patches and is useless.
Changelog since V3:
o Remove stray tab (Valentin)
o Update comment about allowing a move when src is imbalanced (Hillf)
o Updated "sched/pelt: Add a new runnable average signal" (Vincent)
Changelog since V2:
o Rebase on top of Vincent's series again
o Fix a missed rcu_read_unlock
o Reduce overhead of tracepoint
Changelog since V1:
o Rebase on top of Vincent's series and rework
Note: The baseline for this series is tip/sched/core as of February
12th rebased on top of v5.6-rc1. The series includes patches from Vincent as I needed to add a fix and build on top of it. Vincent's series on its own introduces performance regressions for *some* but not *all* machines so it's easily missed. This series overall is close to performance-neutral with some gains depending on the machine. However, the end result does less work on NUMA balancing and the fact that both the NUMA balancer and load balancer uses similar logic makes it much easier to understand.
The NUMA balancer makes placement decisions on tasks that partially take the load balancer into account and vice versa but there are inconsistencies. This can result in placement decisions that override each other leading to unnecessary migrations -- both task placement and page placement. This series reconciles many of the decisions -- partially Vincent's work with some fixes and optimisations on top to merge our two series.
The first patch is unrelated. It's picked up by tip but was not present in the tree at the time of the fork. I'm including it here because I tested with it.
The second and third patches are tracing only and was needed to get sensible data out of ftrace with respect to task placement for NUMA balancing. The NUMA balancer is *far* easier to analyse with the patches and informed how the series should be developed.
Patches 4-5 are Vincent's and use very similar code patterns and logic between NUMA and load balancer. Patch 6 is a fix to Vincent's work that is necessary to avoid serious imbalances being introduced by the NUMA balancer. Patches 7-9 are also Vincents and while I have not reviewed them closely myself, others have.
The rest of the series are a mix of optimisations and improvements, one of which stops the NUMA balancer fighting with itself.
Note that this is not necessarily a universal performance win although performance results are generally ok (small gains/losses depending on the machine and workload). However, task migrations, page migrations, variability and overall overhead are generally reduced.
The main reference workload I used was specjbb running one JVM per node which typically would be expected to split evenly. It's an interesting workload because the number of "warehouses" does not linearly related to the number of running tasks due to the creation of GC threads and other interfering activity. The mmtests configuration used is jvm-specjbb2005-multi with two runs -- one with ftrace enabling relevant scheduler tracepoints.
An example of the headline performance of the series is below and the tested kernels are
baseline-v3r1 Patches 1-3 for the tracing
loadavg-v3 Patches 1-5 (Add half of Vincent's work)
lbidle-v6 Patches 1-6 Vincent's work with a fix on top
classify-v6 Patches 1-9 Rest of Vincent's work
stopsearch-v6 All patches
5.6.0-rc1 5.6.0-rc1 5.6.0-rc1 5.6.0-rc1 5.6.0-rc1
baseline-v3 loadavg-v3 lbidle-v3 classify-v6 stopsearch-v6
Hmean tput-1 43593.49 ( 0.00%) 41616.85 ( -4.53%) 43657.25 ( 0.15%) 38110.46 * -12.58%* 42213.29 ( -3.17%) Hmean tput-2 95692.84 ( 0.00%) 93196.89 * -2.61%* 92287.78 * -3.56%* 89077.29 ( -6.91%) 96474.49 * 0.82%* Hmean tput-3 143813.12 ( 0.00%) 134447.05 * -6.51%* 134587.84 * -6.41%* 133706.98 ( -7.03%) 144279.90 ( 0.32%) Hmean tput-4 190702.67 ( 0.00%) 176533.79 * -7.43%* 182278.42 * -4.42%* 181405.19 ( -4.88%) 189948.10 ( -0.40%) Hmean tput-5 230242.39 ( 0.00%) 209059.51 * -9.20%* 223219.06 ( -3.05%) 227188.16 ( -1.33%) 225220.39 ( -2.18%) Hmean tput-6 274868.74 ( 0.00%) 246470.42 * -10.33%* 258387.09 * -6.00%* 264252.76 ( -3.86%) 271429.49 ( -1.25%) Hmean tput-7 312281.15 ( 0.00%) 284564.06 * -8.88%* 296446.00 * -5.07%* 302682.72 ( -3.07%) 309187.26 ( -0.99%) Hmean tput-8 347261.31 ( 0.00%) 332019.39 * -4.39%* 331202.25 * -4.62%* 339469.52 ( -2.24%) 345504.60 ( -0.51%) Hmean tput-9 387336.25 ( 0.00%) 352219.62 * -9.07%* 370222.03 * -4.42%* 367077.01 ( -5.23%) 381610.17 ( -1.48%) Hmean tput-10 421586.76 ( 0.00%) 397304.22 ( -5.76%) 405458.01 ( -3.83%) 416689.66 ( -1.16%) 415549.97 ( -1.43%) Hmean tput-11 459422.43 ( 0.00%) 398023.51 * -13.36%* 441999.08 ( -3.79%) 449912.39 ( -2.07%) 454458.04 ( -1.08%) Hmean tput-12 499087.97 ( 0.00%) 400914.35 * -19.67%* 475755.59 ( -4.68%) 493678.32 ( -1.08%) 493936.79 ( -1.03%) Hmean tput-13 536335.59 ( 0.00%) 406101.41 * -24.28%* 514858.97 ( -4.00%) 528496.01 ( -1.46%) 530662.68 ( -1.06%) Hmean tput-14 571542.75 ( 0.00%) 478797.13 * -16.23%* 551716.00 ( -3.47%) 553771.29 ( -3.11%) 565915.55 ( -0.98%) Hmean tput-15 601412.81 ( 0.00%) 534776.98 * -11.08%* 580105.28 ( -3.54%) 597513.89 ( -0.65%) 596192.34 ( -0.87%) Hmean tput-16 629817.55 ( 0.00%) 407294.29 * -35.33%* 615606.40 ( -2.26%) 630044.12 ( 0.04%) 627806.13 ( -0.32%) Hmean tput-17 667025.18 ( 0.00%) 457416.34 * -31.42%* 626074.81 ( -6.14%) 659706.41 ( -1.10%) 658350.40 ( -1.30%) Hmean tput-18 688148.21 ( 0.00%) 518534.45 * -24.65%* 663161.87 ( -3.63%) 675616.08 ( -1.82%) 682224.35 ( -0.86%) Hmean tput-19 705092.87 ( 0.00%) 466530.37 * -33.83%* 689430.29 ( -2.22%) 691050.89 ( -1.99%) 705532.41 ( 0.06%) Hmean tput-20 711481.44 ( 0.00%) 564355.80 * -20.68%* 692170.67 ( -2.71%) 717866.36 ( 0.90%) 716243.50 ( 0.67%) Hmean tput-21 739790.92 ( 0.00%) 508542.10 * -31.26%* 712348.91 ( -3.71%) 724666.68 ( -2.04%) 723361.87 ( -2.22%) Hmean tput-22 730593.57 ( 0.00%) 540881.37 ( -25.97%) 709794.02 ( -2.85%) 727177.54 ( -0.47%) 721353.36 ( -1.26%) Hmean tput-23 738401.59 ( 0.00%) 561474.46 * -23.96%* 702869.93 ( -4.81%) 720954.73 ( -2.36%) 720813.53 ( -2.38%) Hmean tput-24 731301.95 ( 0.00%) 582929.73 * -20.29%* 704337.59 ( -3.69%) 717204.03 * -1.93%* 714131.38 * -2.35%* Hmean tput-25 734414.40 ( 0.00%) 591635.13 ( -19.44%) 702334.30 ( -4.37%) 720272.39 ( -1.93%) 714245.12 ( -2.75%) Hmean tput-26 724774.17 ( 0.00%) 701310.59 ( -3.24%) 700771.85 ( -3.31%) 718084.92 ( -0.92%) 712988.02 ( -1.63%) Hmean tput-27 713484.55 ( 0.00%) 632795.43 ( -11.31%) 692213.36 ( -2.98%) 710432.96 ( -0.43%) 703087.86 ( -1.46%) Hmean tput-28 723111.86 ( 0.00%) 697438.61 ( -3.55%) 695934.68 ( -3.76%) 708413.26 ( -2.03%) 703449.60 ( -2.72%) Hmean tput-29 714690.69 ( 0.00%) 675820.16 ( -5.44%) 689400.90 ( -3.54%) 698436.85 ( -2.27%) 699981.24 ( -2.06%) Hmean tput-30 711106.03 ( 0.00%) 699748.68 ( -1.60%) 688439.96 ( -3.19%) 698258.70 ( -1.81%) 691636.96 ( -2.74%) Hmean tput-31 701632.39 ( 0.00%) 698807.56 ( -0.40%) 682588.20 ( -2.71%) 696608.99 ( -0.72%) 691015.36 ( -1.51%) Hmean tput-32 703479.77 ( 0.00%) 679020.34 ( -3.48%) 674057.11 * -4.18%* 690706.86 ( -1.82%) 684958.62 ( -2.63%) Hmean tput-33 691594.71 ( 0.00%) 686583.04 ( -0.72%) 673382.64 ( -2.63%) 687319.97 ( -0.62%) 683367.65 ( -1.19%) Hmean tput-34 693435.51 ( 0.00%) 685137.16 ( -1.20%) 674883.97 ( -2.68%) 684897.97 ( -1.23%) 674923.39 ( -2.67%) Hmean tput-35 688036.06 ( 0.00%) 682612.92 ( -0.79%) 668159.93 ( -2.89%) 679301.53 ( -1.27%) 678117.69 ( -1.44%) Hmean tput-36 678957.95 ( 0.00%) 670160.33 ( -1.30%) 662395.36 ( -2.44%) 672165.17 ( -1.00%) 668512.57 ( -1.54%) Hmean tput-37 679748.70 ( 0.00%) 675428.41 ( -0.64%) 666970.33 ( -1.88%) 674127.70 ( -0.83%) 667644.78 ( -1.78%) Hmean tput-38 669969.62 ( 0.00%) 670976.06 ( 0.15%) 660499.74 ( -1.41%) 670848.38 ( 0.13%) 666646.89 ( -0.50%) Hmean tput-39 669291.41 ( 0.00%) 665367.66 ( -0.59%) 649337.71 ( -2.98%) 659685.61 ( -1.44%) 658818.08 ( -1.56%) Hmean tput-40 668074.80 ( 0.00%) 672478.06 ( 0.66%) 661273.87 ( -1.02%) 665147.36 ( -0.44%) 660279.43 ( -1.17%)
Note the regression with the first two patches of Vincent's work (loadavg-v3) followed by lbidle-v3 which mostly restores the performance and the final version keeping things close to performance neutral (showing a mix but within noise). This is not universal as a different 2-socket machine with fewer cores and older CPUs showed no difference. EPYC 1 and EPYC 2 were both affected by the regression as well as a 4-socket Intel box but again, the full series is mostly performance neutral for specjbb but with less NUMA balancing work.
While not presented here, the full series also shows that the throughput measured by each JVM is less variable.
The high-level NUMA stats from /proc/vmstat look like this
5.6.0-rc1 5.6.0-rc1 5.6.0-rc1 5.6.0-rc1 5.6.0-rc1
baseline-v3 loadavg-v3 lbidle-v3 classify-v3 stopsearch-v3
Ops NUMA alloc hit 878062.00 882981.00 957762.00 961630.00 880821.00
Ops NUMA alloc miss 0.00 0.00 0.00 0.00 0.00
Ops NUMA interleave hit 225582.00 237785.00 242554.00 234671.00 234818.00
Ops NUMA alloc local 764932.00 763850.00 835939.00 843950.00 763005.00
Ops NUMA base-page range updates 2517600.00 3707398.00 2889034.00 2442203.00 3303790.00
Ops NUMA PTE updates 1754720.00 1672198.00 1569610.00 1356763.00 1591662.00
Ops NUMA PMD updates 1490.00 3975.00 2577.00 2120.00 3344.00
Ops NUMA hint faults 1678620.00 1586860.00 1475303.00 1285152.00 1512208.00
Ops NUMA hint local faults % 1461203.00 1389234.00 1181790.00 1085517.00 1411194.00Ops NUMA hint local percent 87.05 87.55 80.10 84.47 93.32
Ops NUMA pages migrated 69473.00 62504.00 121893.00 80802.00 46266.00
Ops AutoNUMA cost 8412.04 7961.44 7399.05 6444.39 7585.05
Overall, the local hints percentage is slightly better but crucially, it's done with much less page migrations.
A separate run gathered information from ftrace and analysed it offline. This is based on an earlier version of the series but the changes are not significant enough to warrant a rerun as there are no changes in the NUMA balancing optimisations.
5.6.0-rc1 5.6.0-rc1
baseline-v2 stopsearch-v2Ops Migrate failed no CPU 1871.00 689.00
Ops Migrate failed move to idle 0.00 0.00
Ops Migrate failed swap task fail 872.00 568.00
Ops Task Migrated swapped 6702.00 3344.00
Ops Task Migrated swapped local NID 0.00 0.00
Ops Task Migrated swapped within group 1094.00 124.00
Ops Task Migrated idle CPU 14409.00 14610.00
Ops Task Migrated idle CPU local NID 0.00 0.00
Ops Task Migrate retry 2355.00 1074.00
Ops Task Migrate retry success 0.00 0.00
Ops Task Migrate retry failed 2355.00 1074.00
Ops Load Balance cross NUMA 1248401.00 1261853.00
"Migrate failed no CPU" is the times when NUMA balancing did not find a suitable page on a preferred node. This is increased because the series avoids making decisions that the LB would override.
"Migrate failed swap task fail" is when migrate_swap fails and it can fail for a lot of reasons.
"Task Migrated swapped" is lower which would would be a concern but in this test, locality was higher unlike the test with tracing disabled. This event triggers when two tasks are swapped to keep load neutral or improved from the perspective of the load balancer. The series attempts to swap tasks that both move to their preferred node.
"Task Migrated idle CPU" is similar and while the the series does try to avoid NUMA Balancer and LB fighting each other, it also continues to obey overall CPU load balancer.
"Task Migrate retry failed" happens when NUMA balancing makes multiple attempts to place a task on a preferred node. It is slightly reduced here but it would generally be expected to happen to maintain CPU load balance.
A variety of other workloads were evaluated and appear to be mostly neutral or improved. netperf running on localhost shows gains between 1-8% depending on the machine. hackbench is a mixed bag -- small regressions on one machine around 1-2% depending on the group count but up to 15% gain on another machine. dbench looks generally ok, very small performance gains. pgbench looks ok, small gains and losses, much of which is within the noise. schbench (Facebook workload that is sensitive to wakeup latencies) is mostly good. The autonuma benchmark also generally looks good, most differences are within the noise but with higher locality success rates and fewer page migrations. Other long lived workloads are still running but I'm not expecting many surprises.
include/linux/sched.h | 31 ++- include/trace/events/sched.h | 49 ++-- kernel/sched/core.c | 13 - kernel/sched/debug.c | 17 +- kernel/sched/fair.c | 626 ++++++++++++++++++++++++++++--------------- kernel/sched/pelt.c | 59 ++-- kernel/sched/sched.h | 42 ++- 7 files changed, 535 insertions(+), 302 deletions(-)
Signature · 1 line
2.16.4
PATCH 1/13 sched/fair: Allow a per-CPU kthread waking a task to stack on the same CPU, to fix XFS performance regression
The following XFS commit:
8ab39f11d974 ("xfs: prevent CIL push holdoff in log recovery")
changed the logic from using bound workqueues to using unbound workqueues. Functionally this makes sense but it was observed at the time that the dbench performance dropped quite a lot and CPU migrations were increased.
The current pattern of the task migration is straight-forward. With XFS, an IO issuer delegates work to xlog_cil_push_work ()on an unbound kworker. This runs on a nearby CPU and on completion, dbench wakes up on its old CPU as it is still idle and no migration occurs. dbench then queues the real IO on the blk_mq_requeue_work() work item which runs on a bound kworker which is forced to run on the same CPU as dbench. When IO completes, the bound kworker wakes dbench but as the kworker is a bound but, real task, the CPU is not considered idle and dbench gets migrated by select_idle_sibling() to a new CPU. dbench may ping-pong between two CPUs for a while but ultimately it starts a round-robin of all CPUs sharing the same LLC. High-frequency migration on each IO completion has poor performance overall. It has negative implications both in commication costs and power management. mpstat confirmed that at low thread counts that all CPUs sharing an LLC has low level of activity.
Note that even if the CIL patch was reverted, there still would be migrations but the impact is less noticeable. It turns out that individually the scheduler, XFS, blk-mq and workqueues all made sensible decisions but in combination, the overall effect was sub-optimal.
This patch special cases the IO issue/completion pattern and allows a bound kworker waker and a task wakee to stack on the same CPU if there is a strong chance they are directly related. The expectation is that the kworker is likely going back to sleep shortly. This is not guaranteed as the IO could be queued asynchronously but there is a very strong relationship between the task and kworker in this case that would justify stacking on the same CPU instead of migrating. There should be few concerns about kworker starvation given that the special casing is only when the kworker is the waker.
DBench on XFS
MMTests config: io-dbench4-async modified to run on a fresh XFS filesystem
UMA machine with 8 cores sharing LLC
5.5.0-rc7 5.5.0-rc7
tipsched-20200124 kworkerstack
Amean 1 22.63 ( 0.00%) 20.54 * 9.23%*
Amean 2 25.56 ( 0.00%) 23.40 * 8.44%*
Amean 4 28.63 ( 0.00%) 27.85 * 2.70%*
Amean 8 37.66 ( 0.00%) 37.68 ( -0.05%)
Amean 64 469.47 ( 0.00%) 468.26 ( 0.26%)
Stddev 1 1.00 ( 0.00%) 0.72 ( 28.12%)
Stddev 2 1.62 ( 0.00%) 1.97 ( -21.54%)
Stddev 4 2.53 ( 0.00%) 3.58 ( -41.19%)
Stddev 8 5.30 ( 0.00%) 5.20 ( 1.92%)
Stddev 64 86.36 ( 0.00%) 94.53 ( -9.46%)NUMA machine, 48 CPUs total, 24 CPUs share cache
5.5.0-rc7 5.5.0-rc7
tipsched-20200124 kworkerstack-v1r2
Amean 1 58.69 ( 0.00%) 30.21 * 48.53%*
Amean 2 60.90 ( 0.00%) 35.29 * 42.05%*
Amean 4 66.77 ( 0.00%) 46.55 * 30.28%*
Amean 8 81.41 ( 0.00%) 68.46 * 15.91%*
Amean 16 113.29 ( 0.00%) 107.79 * 4.85%*
Amean 32 199.10 ( 0.00%) 198.22 * 0.44%*
Amean 64 478.99 ( 0.00%) 477.06 * 0.40%*
Amean 128 1345.26 ( 0.00%) 1372.64 * -2.04%*
Stddev 1 2.64 ( 0.00%) 4.17 ( -58.08%)
Stddev 2 4.35 ( 0.00%) 5.38 ( -23.73%)
Stddev 4 6.77 ( 0.00%) 6.56 ( 3.00%)
Stddev 8 11.61 ( 0.00%) 10.91 ( 6.04%)
Stddev 16 18.63 ( 0.00%) 19.19 ( -3.01%)
Stddev 32 38.71 ( 0.00%) 38.30 ( 1.06%)
Stddev 64 100.28 ( 0.00%) 91.24 ( 9.02%)
Stddev 128 186.87 ( 0.00%) 160.34 ( 14.20%)Dbench has been modified to report the time to complete a single "load file". This is a more meaningful metric for dbench that a throughput metric as the benchmark makes many different system calls that are not throughput-related
Patch shows a 9.23% and 48.53% reduction in the time to process a load file with the difference partially explained by the number of CPUs sharing a LLC. In a separate run, task migrations were almost eliminated by the patch for low client counts. In case people have issue with the metric used for the benchmark, this is a comparison of the throughputs as reported by dbench on the NUMA machine.
dbench4 Throughput (misleading but traditional)
5.5.0-rc7 5.5.0-rc7
tipsched-20200124 kworkerstack-v1r2
Hmean 1 321.41 ( 0.00%) 617.82 * 92.22%*
Hmean 2 622.87 ( 0.00%) 1066.80 * 71.27%*
Hmean 4 1134.56 ( 0.00%) 1623.74 * 43.12%*
Hmean 8 1869.96 ( 0.00%) 2212.67 * 18.33%*
Hmean 16 2673.11 ( 0.00%) 2806.13 * 4.98%*
Hmean 32 3032.74 ( 0.00%) 3039.54 ( 0.22%)
Hmean 64 2514.25 ( 0.00%) 2498.96 * -0.61%*
Hmean 128 1778.49 ( 0.00%) 1746.05 * -1.82%*Note that this is somewhat specific to XFS and ext4 shows no performance difference as it does not rely on kworkers in the same way. No major problem was observed running other workloads on different machines although not all tests have completed yet.
--- kernel/sched/core.c | 11 ----------- kernel/sched/fair.c | 14 ++++++++++++++ kernel/sched/sched.h | 13 +++++++++++++ 3 files changed, 27 insertions(+), 11 deletions(-)
kernel/sched/core.c +0 -11
@@ -1447,17 +1447,6 @@ void check_preempt_curr(struct rq *rq, struct task_struct *p, int flags)#ifdef CONFIG_SMP -static inline bool is_per_cpu_kthread (struct task_struct *p) -{ -if (!(p->flags &PF_KTHREAD )) -return false; - -if (p->nr_cpus_allowed !=1 ) -return false; - -return true; -} -/* * Per-CPU kthreads are allowed to run on !active && online CPUs, see * __set_cpus_allowed_ptr() and select_fallback_rq().
kernel/sched/fair.c +14 -0
@@ -5970,6 +5970,20 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) (available_idle_cpu(prev) || sched_idle_cpu(prev)))return prev; +/* +* Allow a per-cpu kthread to stack with the wakee if the +* kworker thread and the tasks previous CPUs are the same. +* The assumption is that the wakee queued work for the +* per-cpu kthread that is now complete and the wakeup is +* essentially a sync wakeup. An obvious example of this +* pattern is IO completions. +*/ +if (is_per_cpu_kthread (current) && + prev ==smp_processor_id () && +this_rq ()->nr_running <=1 ) { +return prev; + } +/* Check a recently used CPU as a potential idle candidate: */ recent_used_cpu = p->recent_used_cpu; if (recent_used_cpu != prev &&
kernel/sched/sched.h +13 -0
@@ -2484,3 +2484,16 @@ static inline void membarrier_switch_mm(struct rq *rq, { }#endif + +#ifdef CONFIG_SMP +static inline bool is_per_cpu_kthread (struct task_struct *p) +{ +if (!(p->flags &PF_KTHREAD )) +return false; + +if (p->nr_cpus_allowed !=1 ) +return false; + +return true; +} +#endif
Signature · 1 line
2.16.4
PATCH 2/13 sched/numa: Trace when no candidate CPU was found on the preferred node
In reply to Mel Gormansched:sched_stick_numa is meant to fire when a task is unable to migrate to the preferred node. The case where no candidate CPU could be found is not traced which is an important gap. The tracepoint is not fired when the task is not allowed to run on any CPU on the preferred node or the task is already running on the target CPU but neither are interesting corner cases.
--- kernel/sched/fair.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/sched/fair.c +3 -1
@@ -1848,8 +1848,10 @@ static int task_numa_migrate(struct task_struct *p) }/* No better CPU than the current one was found. */ -if (env.best_cpu ==-1 ) +if (env.best_cpu ==-1 ) { +trace_sched_stick_numa (p, env.src_cpu,-1 );return -EAGAIN ; + } best_rq =cpu_rq (env.best_cpu);if (env.best_task ==NULL ) {
Signature · 1 line
2.16.4
[tip: sched/core] sched/numa: Trace when no candidate CPU was found on the preferred node
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: f22aef4afb0d6cc22e408d8254cf6d02d7982ef1 Gitweb: https://git.kernel.org/tip/f22aef4afb0d6cc22e408d8254cf6d02d7982ef1 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 24 Feb 2020 09:52:12 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 24 Feb 2020 11:36:32 +01:00
sched/numa: Trace when no candidate CPU was found on the preferred node
sched:sched_stick_numa is meant to fire when a task is unable to migrate to the preferred node. The case where no candidate CPU could be found is not traced which is an important gap. The tracepoint is not fired when the task is not allowed to run on any CPU on the preferred node or the task is already running on the target CPU but neither are interesting corner cases.
--- kernel/sched/fair.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/sched/fair.c +3 -1
@@ -1848,8 +1848,10 @@ static int task_numa_migrate(struct task_struct *p) }/* No better CPU than the current one was found. */ -if (env.best_cpu ==-1 ) +if (env.best_cpu ==-1 ) { +trace_sched_stick_numa (p, env.src_cpu,-1 );return -EAGAIN ; + } best_rq =cpu_rq (env.best_cpu);if (env.best_task ==NULL ) {
PATCH 3/13 sched/numa: Distinguish between the different task_numa_migrate failure cases
In reply to Mel Gormansched:sched_stick_numa is meant to fire when a task is unable to migrate to the preferred node but from the trace, it's possibile to tell the difference between "no CPU found", "migration to idle CPU failed" and "tasks could not be swapped". Extend the tracepoint accordingly.
--- include/trace/events/sched.h | 49 ++++++++++++++++++++++++-------------------- kernel/sched/fair.c | 6 +++--- 2 files changed, 30 insertions(+), 25 deletions(-)
include/trace/events/sched.h +27 -22
@@ -487,7 +487,11 @@ TRACE_EVENT(sched_process_hang, );#endif /* CONFIG_DETECT_HUNG_TASK */ -DECLARE_EVENT_CLASS(sched_move_task_template, +/* +* Tracks migration of tasks from one runqueue to another. Can be used to +* detect if automatic NUMA balancing is bouncing between nodes +*/ +TRACE_EVENT(sched_move_numa, TP_PROTO(struct task_struct *tsk, int src_cpu, int dst_cpu),
@@ -519,23 +523,7 @@ DECLARE_EVENT_CLASS(sched_move_task_template, __entry->dst_cpu, __entry->dst_nid) ); -/* -* Tracks migration of tasks from one runqueue to another. Can be used to -* detect if automatic NUMA balancing is bouncing between nodes -*/ -DEFINE_EVENT (sched_move_task_template, sched_move_numa, - TP_PROTO(struct task_struct *tsk, int src_cpu, int dst_cpu), - -TP_ARGS (tsk, src_cpu, dst_cpu) -); - -DEFINE_EVENT (sched_move_task_template, sched_stick_numa, - TP_PROTO(struct task_struct *tsk, int src_cpu, int dst_cpu), - -TP_ARGS (tsk, src_cpu, dst_cpu) -); - -TRACE_EVENT(sched_swap_numa, +DECLARE_EVENT_CLASS(sched_numa_pair_template, TP_PROTO(struct task_struct *src_tsk, int src_cpu, struct task_struct *dst_tsk, int dst_cpu),
@@ -561,11 +549,11 @@ TRACE_EVENT(sched_swap_numa, __entry->src_ngid =task_numa_group_id (src_tsk); __entry->src_cpu = src_cpu; __entry->src_nid =cpu_to_node (src_cpu); - __entry->dst_pid =task_pid_nr (dst_tsk); - __entry->dst_tgid =task_tgid_nr (dst_tsk); - __entry->dst_ngid =task_numa_group_id (dst_tsk); + __entry->dst_pid = dst_tsk ?task_pid_nr (dst_tsk) :0 ; + __entry->dst_tgid = dst_tsk ?task_tgid_nr (dst_tsk) :0 ; + __entry->dst_ngid = dst_tsk ?task_numa_group_id (dst_tsk) :0 ; __entry->dst_cpu = dst_cpu; - __entry->dst_nid =cpu_to_node (dst_cpu); + __entry->dst_nid = dst_cpu >=0 ?cpu_to_node (dst_cpu) :-1 ; ), TP_printk("src_pid=%d src_tgid=%d src_ngid=%d src_cpu=%d src_nid=%d dst_pid=%d dst_tgid=%d dst_ngid=%d dst_cpu=%d dst_nid=%d",
@@ -575,6 +563,23 @@ TRACE_EVENT(sched_swap_numa, __entry->dst_cpu, __entry->dst_nid) ); +DEFINE_EVENT (sched_numa_pair_template, sched_stick_numa, + + TP_PROTO(struct task_struct *src_tsk, int src_cpu, + struct task_struct *dst_tsk, int dst_cpu), + +TP_ARGS (src_tsk, src_cpu, dst_tsk, dst_cpu) +); + +DEFINE_EVENT (sched_numa_pair_template, sched_swap_numa, + + TP_PROTO(struct task_struct *src_tsk, int src_cpu, + struct task_struct *dst_tsk, int dst_cpu), + +TP_ARGS (src_tsk, src_cpu, dst_tsk, dst_cpu) +); + +/* * Tracepoint for waking a polling cpu without an IPI. */
kernel/sched/fair.c +3 -3
@@ -1849,7 +1849,7 @@ static int task_numa_migrate(struct task_struct *p)/* No better CPU than the current one was found. */ if (env.best_cpu ==-1 ) { -trace_sched_stick_numa (p, env.src_cpu,-1 ); +trace_sched_stick_numa (p, env.src_cpu,NULL ,-1 );return -EAGAIN ; }
@@ -1858,7 +1858,7 @@ static int task_numa_migrate(struct task_struct *p) ret =migrate_task_to (p, env.best_cpu);WRITE_ONCE (best_rq->numa_migrate_on,0 );if (ret !=0 ) -trace_sched_stick_numa (p, env.src_cpu, env.best_cpu); +trace_sched_stick_numa (p, env.src_cpu,NULL , env.best_cpu);return ret; }
@@ -1866,7 +1866,7 @@ static int task_numa_migrate(struct task_struct *p)WRITE_ONCE (best_rq->numa_migrate_on,0 );if (ret !=0 ) -trace_sched_stick_numa (p, env.src_cpu,task_cpu (env.best_task)); +trace_sched_stick_numa (p, env.src_cpu, env.best_task, env.best_cpu);put_task_struct (env.best_task);return ret; }
Signature · 1 line
2.16.4
[tip: sched/core] sched/numa: Distinguish between the different task_numa_migrate() failure cases
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: b2b2042b204796190af7c20069ab790a614c36d0 Gitweb: https://git.kernel.org/tip/b2b2042b204796190af7c20069ab790a614c36d0 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 24 Feb 2020 09:52:13 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 24 Feb 2020 11:36:33 +01:00
sched/numa: Distinguish between the different task_numa_migrate() failure cases
sched:sched_stick_numa is meant to fire when a task is unable to migrate to the preferred node but from the trace, it's possibile to tell the difference between "no CPU found", "migration to idle CPU failed" and "tasks could not be swapped". Extend the tracepoint accordingly.
--- include/trace/events/sched.h | 49 +++++++++++++++++++---------------- kernel/sched/fair.c | 6 ++-- 2 files changed, 30 insertions(+), 25 deletions(-)
include/trace/events/sched.h +27 -22
@@ -487,7 +487,11 @@ TRACE_EVENT(sched_process_hang, );#endif /* CONFIG_DETECT_HUNG_TASK */ -DECLARE_EVENT_CLASS(sched_move_task_template, +/* +* Tracks migration of tasks from one runqueue to another. Can be used to +* detect if automatic NUMA balancing is bouncing between nodes. +*/ +TRACE_EVENT(sched_move_numa, TP_PROTO(struct task_struct *tsk, int src_cpu, int dst_cpu),
@@ -519,23 +523,7 @@ DECLARE_EVENT_CLASS(sched_move_task_template, __entry->dst_cpu, __entry->dst_nid) ); -/* -* Tracks migration of tasks from one runqueue to another. Can be used to -* detect if automatic NUMA balancing is bouncing between nodes -*/ -DEFINE_EVENT (sched_move_task_template, sched_move_numa, - TP_PROTO(struct task_struct *tsk, int src_cpu, int dst_cpu), - -TP_ARGS (tsk, src_cpu, dst_cpu) -); - -DEFINE_EVENT (sched_move_task_template, sched_stick_numa, - TP_PROTO(struct task_struct *tsk, int src_cpu, int dst_cpu), - -TP_ARGS (tsk, src_cpu, dst_cpu) -); - -TRACE_EVENT(sched_swap_numa, +DECLARE_EVENT_CLASS(sched_numa_pair_template, TP_PROTO(struct task_struct *src_tsk, int src_cpu, struct task_struct *dst_tsk, int dst_cpu),
@@ -561,11 +549,11 @@ TRACE_EVENT(sched_swap_numa, __entry->src_ngid =task_numa_group_id (src_tsk); __entry->src_cpu = src_cpu; __entry->src_nid =cpu_to_node (src_cpu); - __entry->dst_pid =task_pid_nr (dst_tsk); - __entry->dst_tgid =task_tgid_nr (dst_tsk); - __entry->dst_ngid =task_numa_group_id (dst_tsk); + __entry->dst_pid = dst_tsk ?task_pid_nr (dst_tsk) :0 ; + __entry->dst_tgid = dst_tsk ?task_tgid_nr (dst_tsk) :0 ; + __entry->dst_ngid = dst_tsk ?task_numa_group_id (dst_tsk) :0 ; __entry->dst_cpu = dst_cpu; - __entry->dst_nid =cpu_to_node (dst_cpu); + __entry->dst_nid = dst_cpu >=0 ?cpu_to_node (dst_cpu) :-1 ; ), TP_printk("src_pid=%d src_tgid=%d src_ngid=%d src_cpu=%d src_nid=%d dst_pid=%d dst_tgid=%d dst_ngid=%d dst_cpu=%d dst_nid=%d",
@@ -575,6 +563,23 @@ TRACE_EVENT(sched_swap_numa, __entry->dst_cpu, __entry->dst_nid) ); +DEFINE_EVENT (sched_numa_pair_template, sched_stick_numa, + + TP_PROTO(struct task_struct *src_tsk, int src_cpu, + struct task_struct *dst_tsk, int dst_cpu), + +TP_ARGS (src_tsk, src_cpu, dst_tsk, dst_cpu) +); + +DEFINE_EVENT (sched_numa_pair_template, sched_swap_numa, + + TP_PROTO(struct task_struct *src_tsk, int src_cpu, + struct task_struct *dst_tsk, int dst_cpu), + +TP_ARGS (src_tsk, src_cpu, dst_tsk, dst_cpu) +); + +/* * Tracepoint for waking a polling cpu without an IPI. */
kernel/sched/fair.c +3 -3
@@ -1849,7 +1849,7 @@ static int task_numa_migrate(struct task_struct *p)/* No better CPU than the current one was found. */ if (env.best_cpu ==-1 ) { -trace_sched_stick_numa (p, env.src_cpu,-1 ); +trace_sched_stick_numa (p, env.src_cpu,NULL ,-1 );return -EAGAIN ; }
@@ -1858,7 +1858,7 @@ static int task_numa_migrate(struct task_struct *p) ret =migrate_task_to (p, env.best_cpu);WRITE_ONCE (best_rq->numa_migrate_on,0 );if (ret !=0 ) -trace_sched_stick_numa (p, env.src_cpu, env.best_cpu); +trace_sched_stick_numa (p, env.src_cpu,NULL , env.best_cpu);return ret; }
@@ -1866,7 +1866,7 @@ static int task_numa_migrate(struct task_struct *p)WRITE_ONCE (best_rq->numa_migrate_on,0 );if (ret !=0 ) -trace_sched_stick_numa (p, env.src_cpu,task_cpu (env.best_task)); +trace_sched_stick_numa (p, env.src_cpu, env.best_task, env.best_cpu);put_task_struct (env.best_task);return ret; }
PATCH 4/13 sched/fair: Reorder enqueue/dequeue_task_fair path
In reply to Mel GormanFrom: Vincent Guittot <[email protected]>
The walk through the cgroup hierarchy during the enqueue/dequeue of a task is split in 2 distinct parts for throttled cfs_rq without any added value but making code less readable.
Change the code ordering such that everything related to a cfs_rq (throttled or not) will be done in the same loop.
In addition, the same steps ordering is used when updating a cfs_rq:
- update_load_avg
- update_cfs_group
- update *h_nr_running
This reordering enables the use of h_nr_running in PELT algorithm.
No functional and performance changes are expected and have been noticed during tests.
--- kernel/sched/fair.c | 42 ++++++++++++++++++++---------------------- 1 file changed, 20 insertions(+), 22 deletions(-)
kernel/sched/fair.c +20 -22
@@ -5261,32 +5261,31 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) cfs_rq =cfs_rq_of (se);enqueue_entity (cfs_rq, se, flags); -/* -* end evaluation on encountering a throttled cfs_rq -* -* note: in the case of encountering a throttled cfs_rq we will -* post the final h_nr_running increment below. -*/ -if (cfs_rq_throttled (cfs_rq)) -break ; cfs_rq->h_nr_running++; cfs_rq->idle_h_nr_running += idle_h_nr_running; +/* end evaluation on encountering a throttled cfs_rq */ +if (cfs_rq_throttled (cfs_rq)) + goto enqueue_throttle; + flags =ENQUEUE_WAKEUP ; }for_each_sched_entity (se) { cfs_rq =cfs_rq_of (se); - cfs_rq->h_nr_running++; - cfs_rq->idle_h_nr_running += idle_h_nr_running; +/* end evaluation on encountering a throttled cfs_rq */ if (cfs_rq_throttled (cfs_rq)) -break ; + goto enqueue_throttle;update_load_avg (cfs_rq, se,UPDATE_TG );update_cfs_group (se); + + cfs_rq->h_nr_running++; + cfs_rq->idle_h_nr_running += idle_h_nr_running; } +enqueue_throttle:if (!se) {add_nr_running (rq,1 );/*
@@ -5347,17 +5346,13 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) cfs_rq =cfs_rq_of (se);dequeue_entity (cfs_rq, se, flags); -/* -* end evaluation on encountering a throttled cfs_rq -* -* note: in the case of encountering a throttled cfs_rq we will -* post the final h_nr_running decrement below. -*/ -if (cfs_rq_throttled (cfs_rq)) -break ; cfs_rq->h_nr_running--; cfs_rq->idle_h_nr_running -= idle_h_nr_running; +/* end evaluation on encountering a throttled cfs_rq */ +if (cfs_rq_throttled (cfs_rq)) + goto dequeue_throttle; +/* Don't dequeue parent if it has other entities besides us */ if (cfs_rq->load.weight) {/* Avoid re-evaluating load for this entity: */
@@ -5375,16 +5370,19 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)for_each_sched_entity (se) { cfs_rq =cfs_rq_of (se); - cfs_rq->h_nr_running--; - cfs_rq->idle_h_nr_running -= idle_h_nr_running; +/* end evaluation on encountering a throttled cfs_rq */ if (cfs_rq_throttled (cfs_rq)) -break ; + goto dequeue_throttle;update_load_avg (cfs_rq, se,UPDATE_TG );update_cfs_group (se); + + cfs_rq->h_nr_running--; + cfs_rq->idle_h_nr_running -= idle_h_nr_running; } +dequeue_throttle:if (!se)sub_nr_running (rq,1 );
Signature · 1 line
2.16.4
[tip: sched/core] sched/fair: Reorder enqueue/dequeue_task_fair path
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 6d4d22468dae3d8757af9f8b81b848a76ef4409d Gitweb: https://git.kernel.org/tip/6d4d22468dae3d8757af9f8b81b848a76ef4409d Author: Vincent Guittot <[email protected]> AuthorDate: Mon, 24 Feb 2020 09:52:14 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 24 Feb 2020 11:36:34 +01:00
sched/fair: Reorder enqueue/dequeue_task_fair path
The walk through the cgroup hierarchy during the enqueue/dequeue of a task is split in 2 distinct parts for throttled cfs_rq without any added value but making code less readable.
Change the code ordering such that everything related to a cfs_rq (throttled or not) will be done in the same loop.
In addition, the same steps ordering is used when updating a cfs_rq:
- update_load_avg
- update_cfs_group
- update *h_nr_running
This reordering enables the use of h_nr_running in PELT algorithm.
No functional and performance changes are expected and have been noticed during tests.
--- kernel/sched/fair.c | 42 ++++++++++++++++++++---------------------- 1 file changed, 20 insertions(+), 22 deletions(-)
kernel/sched/fair.c +20 -22
@@ -5260,32 +5260,31 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) cfs_rq =cfs_rq_of (se);enqueue_entity (cfs_rq, se, flags); -/* -* end evaluation on encountering a throttled cfs_rq -* -* note: in the case of encountering a throttled cfs_rq we will -* post the final h_nr_running increment below. -*/ -if (cfs_rq_throttled (cfs_rq)) -break ; cfs_rq->h_nr_running++; cfs_rq->idle_h_nr_running += idle_h_nr_running; +/* end evaluation on encountering a throttled cfs_rq */ +if (cfs_rq_throttled (cfs_rq)) + goto enqueue_throttle; + flags =ENQUEUE_WAKEUP ; }for_each_sched_entity (se) { cfs_rq =cfs_rq_of (se); - cfs_rq->h_nr_running++; - cfs_rq->idle_h_nr_running += idle_h_nr_running; +/* end evaluation on encountering a throttled cfs_rq */ if (cfs_rq_throttled (cfs_rq)) -break ; + goto enqueue_throttle;update_load_avg (cfs_rq, se,UPDATE_TG );update_cfs_group (se); + + cfs_rq->h_nr_running++; + cfs_rq->idle_h_nr_running += idle_h_nr_running; } +enqueue_throttle:if (!se) {add_nr_running (rq,1 );/*
@@ -5346,17 +5345,13 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) cfs_rq =cfs_rq_of (se);dequeue_entity (cfs_rq, se, flags); -/* -* end evaluation on encountering a throttled cfs_rq -* -* note: in the case of encountering a throttled cfs_rq we will -* post the final h_nr_running decrement below. -*/ -if (cfs_rq_throttled (cfs_rq)) -break ; cfs_rq->h_nr_running--; cfs_rq->idle_h_nr_running -= idle_h_nr_running; +/* end evaluation on encountering a throttled cfs_rq */ +if (cfs_rq_throttled (cfs_rq)) + goto dequeue_throttle; +/* Don't dequeue parent if it has other entities besides us */ if (cfs_rq->load.weight) {/* Avoid re-evaluating load for this entity: */
@@ -5374,16 +5369,19 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)for_each_sched_entity (se) { cfs_rq =cfs_rq_of (se); - cfs_rq->h_nr_running--; - cfs_rq->idle_h_nr_running -= idle_h_nr_running; +/* end evaluation on encountering a throttled cfs_rq */ if (cfs_rq_throttled (cfs_rq)) -break ; + goto dequeue_throttle;update_load_avg (cfs_rq, se,UPDATE_TG );update_cfs_group (se); + + cfs_rq->h_nr_running--; + cfs_rq->idle_h_nr_running -= idle_h_nr_running; } +dequeue_throttle:if (!se)sub_nr_running (rq,1 );
PATCH 5/13 sched/numa: Replace runnable_load_avg by load_avg
In reply to Mel GormanFrom: Vincent Guittot <[email protected]>
Similarly to what has been done for the normal load balancer, we can replace runnable_load_avg by load_avg in numa load balancing and track the other statistics like the utilization and the number of running tasks to get to better view of the current state of a node.
--- kernel/sched/fair.c | 102 +++++++++++++++++++++++++++++++++++----------------- 1 file changed, 70 insertions(+), 32 deletions(-)
kernel/sched/fair.c +70 -32
@@ -1473,38 +1473,35 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page,group_faults_cpu (ng, src_nid) *group_faults (p, dst_nid) *4 ; } -static inline unsigned long cfs_rq_runnable_load_avg (struct cfs_rq *cfs_rq); - -static unsigned long cpu_runnable_load (struct rq *rq) -{ -return cfs_rq_runnable_load_avg (&rq->cfs); -} +/* +* 'numa_type' describes the node at the moment of load balancing. +*/ +enum numa_type { +/* The node has spare capacity that can be used to run more tasks. */ + node_has_spare =0 , +/* +* The node is fully used and the tasks don't compete for more CPU +* cycles. Nevertheless, some tasks might wait before running. +*/ + node_fully_busy, +/* +* The node is overloaded and can't provide expected CPU cycles to all +* tasks. +*/ + node_overloaded +};/* Cached statistics for all CPUs within a node */ struct numa_stats {unsigned long load; - +unsigned long util;/* Total compute capacity of CPUs on a node */ unsigned long compute_capacity; +unsigned int nr_running; +unsigned int weight; +enum numa_type node_type; }; -/* -* XXX borrowed from update_sg_lb_stats -*/ -static void update_numa_stats (struct numa_stats *ns,int nid) -{ -int cpu; - -memset (ns,0 ,sizeof (*ns)); -for_each_cpu (cpu,cpumask_of_node (nid)) { -struct rq *rq =cpu_rq (cpu); - - ns->load +=cpu_runnable_load (rq); - ns->compute_capacity +=capacity_of (cpu); - } - -} -struct task_numa_env {struct task_struct *p;
@@ -1521,6 +1518,47 @@ struct task_numa_env {int best_cpu; }; +static unsigned long cpu_load (struct rq *rq); +static unsigned long cpu_util (int cpu); + +static inline enum +numa_type numa_classify (unsigned int imbalance_pct, +struct numa_stats *ns) +{ +if ((ns->nr_running > ns->weight) && + ((ns->compute_capacity *100 ) < (ns->util * imbalance_pct))) +return node_overloaded; + +if ((ns->nr_running < ns->weight) || + ((ns->compute_capacity *100 ) > (ns->util * imbalance_pct))) +return node_has_spare; + +return node_fully_busy; +} + +/* +* XXX borrowed from update_sg_lb_stats +*/ +static void update_numa_stats (struct task_numa_env *env, +struct numa_stats *ns,int nid) +{ +int cpu; + +memset (ns,0 ,sizeof (*ns)); +for_each_cpu (cpu,cpumask_of_node (nid)) { +struct rq *rq =cpu_rq (cpu); + + ns->load +=cpu_load (rq); + ns->util +=cpu_util (cpu); + ns->nr_running += rq->cfs.h_nr_running; + ns->compute_capacity +=capacity_of (cpu); + } + + ns->weight =cpumask_weight (cpumask_of_node (nid)); + + ns->node_type =numa_classify (env->imbalance_pct, ns); +} +static void task_numa_assign (struct task_numa_env *env,struct task_struct *p,long imp) {
@@ -1556,6 +1594,11 @@ static bool load_too_imbalanced(long src_load, long dst_load,long orig_src_load, orig_dst_load;long src_capacity, dst_capacity; + +/* If dst node has spare capacity, there is no real load imbalance */ +if (env->dst_stats.node_type == node_has_spare) +return false; +/* * The load is corrected for the CPU capacity available on each node. *
@@ -1788,10 +1831,10 @@ static int task_numa_migrate(struct task_struct *p) dist = env.dist =node_distance (env.src_nid, env.dst_nid); taskweight =task_weight (p, env.src_nid, dist); groupweight =group_weight (p, env.src_nid, dist); -update_numa_stats (&env.src_stats, env.src_nid); +update_numa_stats (&env, &env.src_stats, env.src_nid); taskimp =task_weight (p, env.dst_nid, dist) - taskweight; groupimp =group_weight (p, env.dst_nid, dist) - groupweight; -update_numa_stats (&env.dst_stats, env.dst_nid); +update_numa_stats (&env, &env.dst_stats, env.dst_nid);/* Try to find a spot on the preferred nid. */ task_numa_find_cpu (&env, taskimp, groupimp);
@@ -1824,7 +1867,7 @@ static int task_numa_migrate(struct task_struct *p) env.dist = dist; env.dst_nid = nid; -update_numa_stats (&env.dst_stats, env.dst_nid); +update_numa_stats (&env, &env.dst_stats, env.dst_nid);task_numa_find_cpu (&env, taskimp, groupimp); } }
@@ -3687,11 +3730,6 @@ static void remove_entity_load_avg(struct sched_entity *se)raw_spin_unlock_irqrestore (&cfs_rq->removed.lock, flags); } -static inline unsigned long cfs_rq_runnable_load_avg (struct cfs_rq *cfs_rq) -{ -return cfs_rq->avg.runnable_load_avg; -} -static inline unsigned long cfs_rq_load_avg (struct cfs_rq *cfs_rq) {return cfs_rq->avg.load_avg;
Signature · 1 line
2.16.4
[tip: sched/core] sched/numa: Replace runnable_load_avg by load_avg
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 6499b1b2dd1b8d404a16b9fbbf1af6b9b3c1d83d Gitweb: https://git.kernel.org/tip/6499b1b2dd1b8d404a16b9fbbf1af6b9b3c1d83d Author: Vincent Guittot <[email protected]> AuthorDate: Mon, 24 Feb 2020 09:52:15 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 24 Feb 2020 11:36:34 +01:00
sched/numa: Replace runnable_load_avg by load_avg
Similarly to what has been done for the normal load balancer, we can replace runnable_load_avg by load_avg in numa load balancing and track the other statistics like the utilization and the number of running tasks to get to better view of the current state of a node.
--- kernel/sched/fair.c | 102 +++++++++++++++++++++++++++++-------------- 1 file changed, 70 insertions(+), 32 deletions(-)
kernel/sched/fair.c +70 -32
@@ -1473,38 +1473,35 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page,group_faults_cpu (ng, src_nid) *group_faults (p, dst_nid) *4 ; } -static inline unsigned long cfs_rq_runnable_load_avg (struct cfs_rq *cfs_rq); - -static unsigned long cpu_runnable_load (struct rq *rq) -{ -return cfs_rq_runnable_load_avg (&rq->cfs); -} +/* +* 'numa_type' describes the node at the moment of load balancing. +*/ +enum numa_type { +/* The node has spare capacity that can be used to run more tasks. */ + node_has_spare =0 , +/* +* The node is fully used and the tasks don't compete for more CPU +* cycles. Nevertheless, some tasks might wait before running. +*/ + node_fully_busy, +/* +* The node is overloaded and can't provide expected CPU cycles to all +* tasks. +*/ + node_overloaded +};/* Cached statistics for all CPUs within a node */ struct numa_stats {unsigned long load; - +unsigned long util;/* Total compute capacity of CPUs on a node */ unsigned long compute_capacity; +unsigned int nr_running; +unsigned int weight; +enum numa_type node_type; }; -/* -* XXX borrowed from update_sg_lb_stats -*/ -static void update_numa_stats (struct numa_stats *ns,int nid) -{ -int cpu; - -memset (ns,0 ,sizeof (*ns)); -for_each_cpu (cpu,cpumask_of_node (nid)) { -struct rq *rq =cpu_rq (cpu); - - ns->load +=cpu_runnable_load (rq); - ns->compute_capacity +=capacity_of (cpu); - } - -} -struct task_numa_env {struct task_struct *p;
@@ -1521,6 +1518,47 @@ struct task_numa_env {int best_cpu; }; +static unsigned long cpu_load (struct rq *rq); +static unsigned long cpu_util (int cpu); + +static inline enum +numa_type numa_classify (unsigned int imbalance_pct, +struct numa_stats *ns) +{ +if ((ns->nr_running > ns->weight) && + ((ns->compute_capacity *100 ) < (ns->util * imbalance_pct))) +return node_overloaded; + +if ((ns->nr_running < ns->weight) || + ((ns->compute_capacity *100 ) > (ns->util * imbalance_pct))) +return node_has_spare; + +return node_fully_busy; +} + +/* +* XXX borrowed from update_sg_lb_stats +*/ +static void update_numa_stats (struct task_numa_env *env, +struct numa_stats *ns,int nid) +{ +int cpu; + +memset (ns,0 ,sizeof (*ns)); +for_each_cpu (cpu,cpumask_of_node (nid)) { +struct rq *rq =cpu_rq (cpu); + + ns->load +=cpu_load (rq); + ns->util +=cpu_util (cpu); + ns->nr_running += rq->cfs.h_nr_running; + ns->compute_capacity +=capacity_of (cpu); + } + + ns->weight =cpumask_weight (cpumask_of_node (nid)); + + ns->node_type =numa_classify (env->imbalance_pct, ns); +} +static void task_numa_assign (struct task_numa_env *env,struct task_struct *p,long imp) {
@@ -1556,6 +1594,11 @@ static bool load_too_imbalanced(long src_load, long dst_load,long orig_src_load, orig_dst_load;long src_capacity, dst_capacity; + +/* If dst node has spare capacity, there is no real load imbalance */ +if (env->dst_stats.node_type == node_has_spare) +return false; +/* * The load is corrected for the CPU capacity available on each node. *
@@ -1788,10 +1831,10 @@ static int task_numa_migrate(struct task_struct *p) dist = env.dist =node_distance (env.src_nid, env.dst_nid); taskweight =task_weight (p, env.src_nid, dist); groupweight =group_weight (p, env.src_nid, dist); -update_numa_stats (&env.src_stats, env.src_nid); +update_numa_stats (&env, &env.src_stats, env.src_nid); taskimp =task_weight (p, env.dst_nid, dist) - taskweight; groupimp =group_weight (p, env.dst_nid, dist) - groupweight; -update_numa_stats (&env.dst_stats, env.dst_nid); +update_numa_stats (&env, &env.dst_stats, env.dst_nid);/* Try to find a spot on the preferred nid. */ task_numa_find_cpu (&env, taskimp, groupimp);
@@ -1824,7 +1867,7 @@ static int task_numa_migrate(struct task_struct *p) env.dist = dist; env.dst_nid = nid; -update_numa_stats (&env.dst_stats, env.dst_nid); +update_numa_stats (&env, &env.dst_stats, env.dst_nid);task_numa_find_cpu (&env, taskimp, groupimp); } }
@@ -3686,11 +3729,6 @@ static void remove_entity_load_avg(struct sched_entity *se)raw_spin_unlock_irqrestore (&cfs_rq->removed.lock, flags); } -static inline unsigned long cfs_rq_runnable_load_avg (struct cfs_rq *cfs_rq) -{ -return cfs_rq->avg.runnable_load_avg; -} -static inline unsigned long cfs_rq_load_avg (struct cfs_rq *cfs_rq) {return cfs_rq->avg.load_avg;
PATCH 6/13 sched/numa: Use similar logic to the load balancer for moving between domains with spare capacity
In reply to Mel GormanThe standard load balancer generally tries to keep the number of running tasks or idle CPUs balanced between NUMA domains. The NUMA balancer allows tasks to move if there is spare capacity but this causes a conflict and utilisation between NUMA nodes gets badly skewed. This patch uses similar logic between the NUMA balancer and load balancer when deciding if a task migrating to its preferred node can use an idle CPU.
--- kernel/sched/fair.c | 81 +++++++++++++++++++++++++++++++++-------------------- 1 file changed, 50 insertions(+), 31 deletions(-)
kernel/sched/fair.c +50 -31
@@ -1520,6 +1520,7 @@ struct task_numa_env {static unsigned long cpu_load (struct rq *rq);static unsigned long cpu_util (int cpu); +static inline long adjust_numa_imbalance (int imbalance,int src_nr_running); static inline enum numa_type numa_classify(unsigned int imbalance_pct,
@@ -1594,11 +1595,6 @@ static bool load_too_imbalanced(long src_load, long dst_load,long orig_src_load, orig_dst_load;long src_capacity, dst_capacity; - -/* If dst node has spare capacity, there is no real load imbalance */ -if (env->dst_stats.node_type == node_has_spare) -return false; -/* * The load is corrected for the CPU capacity available on each node. *
@@ -1757,19 +1753,42 @@ static void task_numa_compare(struct task_numa_env *env,static void task_numa_find_cpu (struct task_numa_env *env,long taskimp,long groupimp) { -long src_load, dst_load, load;bool maymove = false;int cpu; - load =task_h_load (env->p); - dst_load = env->dst_stats.load + load; - src_load = env->src_stats.load - load; -/* -* If the improvement from just moving env->p direction is better -* than swapping tasks around, check if a move is possible. +* If dst node has spare capacity, then check if there is an +* imbalance that would be overruled by the load balancer. */ - maymove = !load_too_imbalanced (src_load, dst_load, env); +if (env->dst_stats.node_type == node_has_spare) { +unsigned int imbalance; +int src_running, dst_running; + +/* +* Would movement cause an imbalance? Note that if src has +* more running tasks that the imbalance is ignored as the +* move improves the imbalance from the perspective of the +* CPU load balancer. +* */ + src_running = env->src_stats.nr_running -1 ; + dst_running = env->dst_stats.nr_running +1 ; + imbalance =max (0 , dst_running - src_running); + imbalance =adjust_numa_imbalance (imbalance, src_running); + +/* Use idle CPU if there is no imbalance */ +if (!imbalance) + maymove = true; + }else { +long src_load, dst_load, load; +/* +* If the improvement from just moving env->p direction is better +* than swapping tasks around, check if a move is possible. +*/ + load =task_h_load (env->p); + dst_load = env->dst_stats.load + load; + src_load = env->src_stats.load - load; + maymove = !load_too_imbalanced (src_load, dst_load, env); + }for_each_cpu (cpu,cpumask_of_node (env->dst_nid)) {/* Skip this CPU if the source task cannot migrate */
@@ -8695,6 +8714,21 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd } } +static inline long adjust_numa_imbalance (int imbalance,int src_nr_running) +{ +unsigned int imbalance_min; + +/* +* Allow a small imbalance based on a simple pair of communicating +* tasks that remain local when the source domain is almost idle. +*/ + imbalance_min =2 ; +if (src_nr_running <= imbalance_min) +return 0 ; + +return imbalance; +} +/** * calculate_imbalance - Calculate the amount of imbalance present within the * groups of a given sched_domain during load balance.
@@ -8791,24 +8825,9 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *s }/* Consider allowing a small imbalance between NUMA groups */ -if (env->sd->flags &SD_NUMA ) { -unsigned int imbalance_min; - -/* -* Compute an allowed imbalance based on a simple -* pair of communicating tasks that should remain -* local and ignore them. -* -* NOTE: Generally this would have been based on -* the domain size and this was evaluated. However, -* the benefit is similar across a range of workloads -* and machines but scaling by the domain size adds -* the risk that lower domains have to be rebalanced. -*/ - imbalance_min =2 ; -if (busiest->sum_nr_running <= imbalance_min) - env->imbalance =0 ; - } +if (env->sd->flags &SD_NUMA ) + env->imbalance =adjust_numa_imbalance (env->imbalance, + busiest->sum_nr_running);return ; }
Signature · 1 line
2.16.4
[tip: sched/core] sched/numa: Use similar logic to the load balancer for moving between domains with spare capacity
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: fb86f5b2119245afd339280099b4e9417cc0b03a Gitweb: https://git.kernel.org/tip/fb86f5b2119245afd339280099b4e9417cc0b03a Author: Mel Gorman <[email protected]> AuthorDate: Mon, 24 Feb 2020 09:52:16 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 24 Feb 2020 11:36:35 +01:00
sched/numa: Use similar logic to the load balancer for moving between domains with spare capacity
The standard load balancer generally tries to keep the number of running tasks or idle CPUs balanced between NUMA domains. The NUMA balancer allows tasks to move if there is spare capacity but this causes a conflict and utilisation between NUMA nodes gets badly skewed. This patch uses similar logic between the NUMA balancer and load balancer when deciding if a task migrating to its preferred node can use an idle CPU.
--- kernel/sched/fair.c | 81 +++++++++++++++++++++++++++----------------- 1 file changed, 50 insertions(+), 31 deletions(-)
kernel/sched/fair.c +50 -31
@@ -1520,6 +1520,7 @@ struct task_numa_env {static unsigned long cpu_load (struct rq *rq);static unsigned long cpu_util (int cpu); +static inline long adjust_numa_imbalance (int imbalance,int src_nr_running); static inline enum numa_type numa_classify(unsigned int imbalance_pct,
@@ -1594,11 +1595,6 @@ static bool load_too_imbalanced(long src_load, long dst_load,long orig_src_load, orig_dst_load;long src_capacity, dst_capacity; - -/* If dst node has spare capacity, there is no real load imbalance */ -if (env->dst_stats.node_type == node_has_spare) -return false; -/* * The load is corrected for the CPU capacity available on each node. *
@@ -1757,19 +1753,42 @@ unlock:static void task_numa_find_cpu (struct task_numa_env *env,long taskimp,long groupimp) { -long src_load, dst_load, load;bool maymove = false;int cpu; - load =task_h_load (env->p); - dst_load = env->dst_stats.load + load; - src_load = env->src_stats.load - load; -/* -* If the improvement from just moving env->p direction is better -* than swapping tasks around, check if a move is possible. +* If dst node has spare capacity, then check if there is an +* imbalance that would be overruled by the load balancer. */ - maymove = !load_too_imbalanced (src_load, dst_load, env); +if (env->dst_stats.node_type == node_has_spare) { +unsigned int imbalance; +int src_running, dst_running; + +/* +* Would movement cause an imbalance? Note that if src has +* more running tasks that the imbalance is ignored as the +* move improves the imbalance from the perspective of the +* CPU load balancer. +* */ + src_running = env->src_stats.nr_running -1 ; + dst_running = env->dst_stats.nr_running +1 ; + imbalance =max (0 , dst_running - src_running); + imbalance =adjust_numa_imbalance (imbalance, src_running); + +/* Use idle CPU if there is no imbalance */ +if (!imbalance) + maymove = true; + }else { +long src_load, dst_load, load; +/* +* If the improvement from just moving env->p direction is better +* than swapping tasks around, check if a move is possible. +*/ + load =task_h_load (env->p); + dst_load = env->dst_stats.load + load; + src_load = env->src_stats.load - load; + maymove = !load_too_imbalanced (src_load, dst_load, env); + }for_each_cpu (cpu,cpumask_of_node (env->dst_nid)) {/* Skip this CPU if the source task cannot migrate */
@@ -8694,6 +8713,21 @@ next_group: } } +static inline long adjust_numa_imbalance (int imbalance,int src_nr_running) +{ +unsigned int imbalance_min; + +/* +* Allow a small imbalance based on a simple pair of communicating +* tasks that remain local when the source domain is almost idle. +*/ + imbalance_min =2 ; +if (src_nr_running <= imbalance_min) +return 0 ; + +return imbalance; +} +/** * calculate_imbalance - Calculate the amount of imbalance present within the * groups of a given sched_domain during load balance.
@@ -8790,24 +8824,9 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *s }/* Consider allowing a small imbalance between NUMA groups */ -if (env->sd->flags &SD_NUMA ) { -unsigned int imbalance_min; - -/* -* Compute an allowed imbalance based on a simple -* pair of communicating tasks that should remain -* local and ignore them. -* -* NOTE: Generally this would have been based on -* the domain size and this was evaluated. However, -* the benefit is similar across a range of workloads -* and machines but scaling by the domain size adds -* the risk that lower domains have to be rebalanced. -*/ - imbalance_min =2 ; -if (busiest->sum_nr_running <= imbalance_min) - env->imbalance =0 ; - } +if (env->sd->flags &SD_NUMA ) + env->imbalance =adjust_numa_imbalance (env->imbalance, + busiest->sum_nr_running);return ; }
PATCH 7/13 sched/pelt: Remove unused runnable load average
In reply to Mel GormanFrom: Vincent Guittot <[email protected]>
Now that runnable_load_avg is no more used, we can remove it to make space for a new signal.
--- include/linux/sched.h | 5 +- kernel/sched/core.c | 2 - kernel/sched/debug.c | 8 ---- kernel/sched/fair.c | 130 +++++++------------------------------------------- kernel/sched/pelt.c | 62 ++++++++++-------------- kernel/sched/sched.h | 7 +-- 6 files changed, 43 insertions(+), 171 deletions(-)
include/linux/sched.h +1 -4
@@ -357,7 +357,7 @@ struct util_est {/* * The load_avg/util_avg accumulates an infinite geometric series -* (see __update_load_avg() in kernel/sched/fair.c). +* (see __update_load_avg_cfs_rq() in kernel/sched/pelt.c). * * [load_avg definition] *
@@ -401,11 +401,9 @@ struct util_est {struct sched_avg {u64 last_update_time;u64 load_sum; -u64 runnable_load_sum;u32 util_sum;u32 period_contrib;unsigned long load_avg; -unsigned long runnable_load_avg;unsigned long util_avg;struct util_est util_est; }____cacheline_aligned ;
@@ -449,7 +447,6 @@ struct sched_statistics {struct sched_entity {/* For load-balancing: */ struct load_weight load; -unsigned long runnable_weight;struct rb_node run_node;struct list_head group_node;unsigned int on_rq;
kernel/sched/core.c +0 -2
@@ -761,7 +761,6 @@ static void set_load_weight(struct task_struct *p, bool update_load)if (task_has_idle_policy (p)) { load->weight =scale_load (WEIGHT_IDLEPRIO ); load->inv_weight =WMULT_IDLEPRIO ; - p->se.runnable_weight = load->weight;return ; }
@@ -774,7 +773,6 @@ static void set_load_weight(struct task_struct *p, bool update_load) }else { load->weight =scale_load (sched_prio_to_weight[prio]); load->inv_weight = sched_prio_to_wmult[prio]; - p->se.runnable_weight = load->weight; } }
kernel/sched/debug.c +0 -8
@@ -402,11 +402,9 @@ static void print_cfs_group_stats(struct seq_file *m, int cpu, struct task_group }P (se->load.weight); -P (se->runnable_weight);#ifdef CONFIG_SMP P (se->avg.load_avg);P (se->avg.util_avg); -P (se->avg.runnable_load_avg);#endif #undef PN_SCHEDSTAT
@@ -524,11 +522,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)SEQ_printf (m," .%-30s: %d\n" ,"nr_running" , cfs_rq->nr_running);SEQ_printf (m," .%-30s: %ld\n" ,"load" , cfs_rq->load.weight);#ifdef CONFIG_SMP -SEQ_printf (m," .%-30s: %ld\n" ,"runnable_weight" , cfs_rq->runnable_weight);SEQ_printf (m," .%-30s: %lu\n" ,"load_avg" , cfs_rq->avg.load_avg); -SEQ_printf (m," .%-30s: %lu\n" ,"runnable_load_avg" , - cfs_rq->avg.runnable_load_avg);SEQ_printf (m," .%-30s: %lu\n" ,"util_avg" , cfs_rq->avg.util_avg); SEQ_printf(m, " .%-30s: %u\n", "util_est_enqueued",
@@ -947,13 +942,10 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, "nr_involuntary_switches", (long long)p->nivcsw);P (se.load.weight); -P (se.runnable_weight);#ifdef CONFIG_SMP P (se.avg.load_sum); -P (se.avg.runnable_load_sum);P (se.avg.util_sum);P (se.avg.load_avg); -P (se.avg.runnable_load_avg);P (se.avg.util_avg);P (se.avg.last_update_time);P (se.avg.util_est.ewma);
kernel/sched/fair.c +16 -114
267 lines not on this page · raw message
kernel/sched/pelt.c +24 -38
195 lines not on this page · raw message
kernel/sched/sched.h +2 -5
@@ -489,7 +489,6 @@ struct cfs_bandwidth { };/* CFS-related fields in a runqueue */ struct cfs_rq {struct load_weight load; -unsigned long runnable_weight;unsigned int nr_running;unsigned int h_nr_running;/* SCHED_{NORMAL,BATCH,IDLE} */ unsigned int idle_h_nr_running;/* SCHED_IDLE */
@@ -688,8 +687,10 @@ struct dl_rq {#ifdef CONFIG_FAIR_GROUP_SCHED /* An entity is a task if it doesn't "own" a runqueue */ #define entity_is_task (se) (!se->my_q) +#else #define entity_is_task (se) 1 +#endif #ifdef CONFIG_SMP
@@ -701,10 +702,6 @@ static inline long se_weight(struct sched_entity *se)return scale_load_down (se->load.weight); } -static inline long se_runnable (struct sched_entity *se) -{ -return scale_load_down (se->runnable_weight); -}static inline bool sched_asym_prefer (int a,int b) {
Signature · 1 line
2.16.4
[tip: sched/core] sched/pelt: Remove unused runnable load average
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 0dacee1bfa70e171be3a12a30414c228453048d2 Gitweb: https://git.kernel.org/tip/0dacee1bfa70e171be3a12a30414c228453048d2 Author: Vincent Guittot <[email protected]> AuthorDate: Mon, 24 Feb 2020 09:52:17 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 24 Feb 2020 11:36:36 +01:00
sched/pelt: Remove unused runnable load average
Now that runnable_load_avg is no more used, we can remove it to make space for a new signal.
--- include/linux/sched.h | 5 +-- kernel/sched/core.c | 2 +- kernel/sched/debug.c | 8 +--- kernel/sched/fair.c | 130 +++++------------------------------------ kernel/sched/pelt.c | 62 +++++++------------- kernel/sched/sched.h | 7 +-- 6 files changed, 43 insertions(+), 171 deletions(-)
include/linux/sched.h +1 -4
@@ -357,7 +357,7 @@ struct util_est {/* * The load_avg/util_avg accumulates an infinite geometric series -* (see __update_load_avg() in kernel/sched/fair.c). +* (see __update_load_avg_cfs_rq() in kernel/sched/pelt.c). * * [load_avg definition] *
@@ -401,11 +401,9 @@ struct util_est {struct sched_avg {u64 last_update_time;u64 load_sum; -u64 runnable_load_sum;u32 util_sum;u32 period_contrib;unsigned long load_avg; -unsigned long runnable_load_avg;unsigned long util_avg;struct util_est util_est; }____cacheline_aligned ;
@@ -449,7 +447,6 @@ struct sched_statistics {struct sched_entity {/* For load-balancing: */ struct load_weight load; -unsigned long runnable_weight;struct rb_node run_node;struct list_head group_node;unsigned int on_rq;
kernel/sched/core.c +0 -2
@@ -761,7 +761,6 @@ static void set_load_weight(struct task_struct *p, bool update_load)if (task_has_idle_policy (p)) { load->weight =scale_load (WEIGHT_IDLEPRIO ); load->inv_weight =WMULT_IDLEPRIO ; - p->se.runnable_weight = load->weight;return ; }
@@ -774,7 +773,6 @@ static void set_load_weight(struct task_struct *p, bool update_load) }else { load->weight =scale_load (sched_prio_to_weight[prio]); load->inv_weight = sched_prio_to_wmult[prio]; - p->se.runnable_weight = load->weight; } }
kernel/sched/debug.c +0 -8
@@ -402,11 +402,9 @@ static void print_cfs_group_stats(struct seq_file *m, int cpu, struct task_group }P (se->load.weight); -P (se->runnable_weight);#ifdef CONFIG_SMP P (se->avg.load_avg);P (se->avg.util_avg); -P (se->avg.runnable_load_avg);#endif #undef PN_SCHEDSTAT
@@ -524,11 +522,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)SEQ_printf (m," .%-30s: %d\n" ,"nr_running" , cfs_rq->nr_running);SEQ_printf (m," .%-30s: %ld\n" ,"load" , cfs_rq->load.weight);#ifdef CONFIG_SMP -SEQ_printf (m," .%-30s: %ld\n" ,"runnable_weight" , cfs_rq->runnable_weight);SEQ_printf (m," .%-30s: %lu\n" ,"load_avg" , cfs_rq->avg.load_avg); -SEQ_printf (m," .%-30s: %lu\n" ,"runnable_load_avg" , - cfs_rq->avg.runnable_load_avg);SEQ_printf (m," .%-30s: %lu\n" ,"util_avg" , cfs_rq->avg.util_avg); SEQ_printf(m, " .%-30s: %u\n", "util_est_enqueued",
@@ -947,13 +942,10 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, "nr_involuntary_switches", (long long)p->nivcsw);P (se.load.weight); -P (se.runnable_weight);#ifdef CONFIG_SMP P (se.avg.load_sum); -P (se.avg.runnable_load_sum);P (se.avg.util_sum);P (se.avg.load_avg); -P (se.avg.runnable_load_avg);P (se.avg.util_avg);P (se.avg.last_update_time);P (se.avg.util_est.ewma);
kernel/sched/fair.c +16 -114
267 lines not on this page · raw message
kernel/sched/pelt.c +24 -38
192 lines not on this page · raw message
kernel/sched/sched.h +2 -5
@@ -489,7 +489,6 @@ struct cfs_bandwidth { };/* CFS-related fields in a runqueue */ struct cfs_rq {struct load_weight load; -unsigned long runnable_weight;unsigned int nr_running;unsigned int h_nr_running;/* SCHED_{NORMAL,BATCH,IDLE} */ unsigned int idle_h_nr_running;/* SCHED_IDLE */
@@ -688,8 +687,10 @@ struct dl_rq {#ifdef CONFIG_FAIR_GROUP_SCHED /* An entity is a task if it doesn't "own" a runqueue */ #define entity_is_task (se) (!se->my_q) +#else #define entity_is_task (se) 1 +#endif #ifdef CONFIG_SMP
@@ -701,10 +702,6 @@ static inline long se_weight(struct sched_entity *se)return scale_load_down (se->load.weight); } -static inline long se_runnable (struct sched_entity *se) -{ -return scale_load_down (se->runnable_weight); -}static inline bool sched_asym_prefer (int a,int b) {
PATCH 8/13 sched/pelt: Add a new runnable average signal
In reply to Mel GormanFrom: Vincent Guittot <[email protected]>
Now that runnable_load_avg has been removed, we can replace it by a new signal that will highlight the runnable pressure on a cfs_rq. This signal track the waiting time of tasks on rq and can help to better define the state of rqs.
At now, only util_avg is used to define the state of a rq:
A rq with more that around 80% of utilization and more than 1 tasks is considered as overloaded.
But the util_avg signal of a rq can become temporaly low after that a task migrated onto another rq which can bias the classification of the rq.
When tasks compete for the same rq, their runnable average signal will be higher than util_avg as it will include the waiting time and we can use this signal to better classify cfs_rqs.
The new runnable_avg will track the runnable time of a task which simply adds the waiting time to the running time. The runnable _avg of cfs_rq will be the /Sum of se's runnable_avg and the runnable_avg of group entity will follow the one of the rq similarly to util_avg.
--- include/linux/sched.h | 26 ++++++++++------- kernel/sched/debug.c | 9 ++++-- kernel/sched/fair.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++----- kernel/sched/pelt.c | 39 ++++++++++++++++++-------- kernel/sched/sched.h | 22 ++++++++++++++- 5 files changed, 142 insertions(+), 31 deletions(-)
include/linux/sched.h +16 -10
@@ -356,28 +356,30 @@ struct util_est { } __attribute__((__aligned__(sizeof(u64))));/* -* The load_avg/util_avg accumulates an infinite geometric series +* The load/runnable/util_avg accumulates an infinite geometric series * (see __update_load_avg_cfs_rq() in kernel/sched/pelt.c). * * [load_avg definition] * * load_avg = runnable% * scale_load_down(load) * -* where runnable% is the time ratio that a sched_entity is runnable. -* For cfs_rq, it is the aggregated load_avg of all runnable and -* blocked sched_entities. +* [runnable_avg definition] +* +* runnable_avg = runnable% * SCHED_CAPACITY_SCALE * * [util_avg definition] * * util_avg = running% * SCHED_CAPACITY_SCALE * -* where running% is the time ratio that a sched_entity is running on -* a CPU. For cfs_rq, it is the aggregated util_avg of all runnable -* and blocked sched_entities. +* where runnable% is the time ratio that a sched_entity is runnable and +* running% the time ratio that a sched_entity is running. +* +* For cfs_rq, they are the aggregated values of all runnable and blocked +* sched_entities. * -* load_avg and util_avg don't direcly factor frequency scaling and CPU -* capacity scaling. The scaling is done through the rq_clock_pelt that -* is used for computing those signals (see update_rq_clock_pelt()) +* The load/runnable/util_avg doesn't direcly factor frequency scaling and CPU +* capacity scaling. The scaling is done through the rq_clock_pelt that is used +* for computing those signals (see update_rq_clock_pelt()) * * N.B., the above ratios (runnable% and running%) themselves are in the * range of [0, 1]. To do fixed point arithmetics, we therefore scale them
@@ -401,9 +403,11 @@ struct util_est {struct sched_avg {u64 last_update_time;u64 load_sum; +u64 runnable_sum;u32 util_sum;u32 period_contrib;unsigned long load_avg; +unsigned long runnable_avg;unsigned long util_avg;struct util_est util_est; }____cacheline_aligned ;
@@ -467,6 +471,8 @@ struct sched_entity {struct cfs_rq *cfs_rq;/* rq "owned" by this entity/group: */ struct cfs_rq *my_q; +/* cached value of my_q->h_nr_running */ +unsigned long runnable_weight;#endif #ifdef CONFIG_SMP
kernel/sched/debug.c +7 -2
@@ -405,6 +405,7 @@ static void print_cfs_group_stats(struct seq_file *m, int cpu, struct task_group#ifdef CONFIG_SMP P (se->avg.load_avg);P (se->avg.util_avg); +P (se->avg.runnable_avg);#endif #undef PN_SCHEDSTAT
@@ -524,6 +525,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)#ifdef CONFIG_SMP SEQ_printf (m," .%-30s: %lu\n" ,"load_avg" , cfs_rq->avg.load_avg); +SEQ_printf (m," .%-30s: %lu\n" ,"runnable_avg" , + cfs_rq->avg.runnable_avg);SEQ_printf (m," .%-30s: %lu\n" ,"util_avg" , cfs_rq->avg.util_avg); SEQ_printf(m, " .%-30s: %u\n", "util_est_enqueued",
@@ -532,8 +535,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq) cfs_rq->removed.load_avg);SEQ_printf (m," .%-30s: %ld\n" ,"removed.util_avg" , cfs_rq->removed.util_avg); -SEQ_printf (m," .%-30s: %ld\n" ,"removed.runnable_sum" , - cfs_rq->removed.runnable_sum); +SEQ_printf (m," .%-30s: %ld\n" ,"removed.runnable_avg" , + cfs_rq->removed.runnable_avg);#ifdef CONFIG_FAIR_GROUP_SCHED SEQ_printf (m," .%-30s: %lu\n" ,"tg_load_avg_contrib" , cfs_rq->tg_load_avg_contrib);
@@ -944,8 +947,10 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,P (se.load.weight);#ifdef CONFIG_SMP P (se.avg.load_sum); +P (se.avg.runnable_sum);P (se.avg.util_sum);P (se.avg.load_avg); +P (se.avg.runnable_avg);P (se.avg.util_avg);P (se.avg.last_update_time);P (se.avg.util_est.ewma);
kernel/sched/fair.c +70 -7
192 lines not on this page · raw message
kernel/sched/pelt.c +28 -11
@@ -108,7 +108,7 @@ static u32 __accumulate_pelt_segments(u64 periods, u32 d1, u32 d3)*/ static __always_inline u32accumulate_sum (u64 delta,struct sched_avg *sa, -unsigned long load,int running) +unsigned long load,unsigned long runnable,int running) {u32 contrib = (u32 )delta;/* p == 0 -> delta < 1024 */ u64 periods;
@@ -121,6 +121,8 @@ accumulate_sum(u64 delta, struct sched_avg *sa,*/ if (periods) { sa->load_sum =decay_load (sa->load_sum, periods); + sa->runnable_sum = +decay_load (sa->runnable_sum, periods); sa->util_sum =decay_load ((u64)(sa->util_sum), periods);/*
@@ -146,6 +148,8 @@ accumulate_sum(u64 delta, struct sched_avg *sa,if (load) sa->load_sum += load * contrib; +if (runnable) + sa->runnable_sum += runnable * contrib <<SCHED_CAPACITY_SHIFT ;if (running) sa->util_sum += contrib <<SCHED_CAPACITY_SHIFT ;
@@ -182,7 +186,7 @@ accumulate_sum(u64 delta, struct sched_avg *sa,*/ static __always_inline int___update_load_sum (u64 now,struct sched_avg *sa, -unsigned long load,int running) +unsigned long load,unsigned long runnable,int running) {u64 delta;
@@ -218,7 +222,7 @@ ___update_load_sum(u64 now, struct sched_avg *sa,* Also see the comment in accumulate_sum(). */ if (!load) - running =0 ; + runnable = running =0 ;/* * Now we know we crossed measurement unit boundaries. The *_avg
@@ -227,7 +231,7 @@ ___update_load_sum(u64 now, struct sched_avg *sa,* Step 1: accumulate *_sum since last_update_time. If we haven't * crossed period boundaries, finish. */ -if (!accumulate_sum (delta, sa, load, running)) +if (!accumulate_sum (delta, sa, load, runnable, running))return 0 ;return 1 ;
@@ -242,6 +246,7 @@ ___update_load_avg(struct sched_avg *sa, unsigned long load)* Step 2: update *_avg. */ sa->load_avg =div_u64 (load * sa->load_sum, divider); + sa->runnable_avg =div_u64 (sa->runnable_sum, divider);WRITE_ONCE (sa->util_avg, sa->util_sum / divider); }
@@ -250,24 +255,30 @@ ___update_load_avg(struct sched_avg *sa, unsigned long load)* * task: * se_weight() = se->load.weight +* se_runnable() = !!on_rq * * group: [ see update_cfs_group() ] * se_weight() = tg->weight * grq->load_avg / tg->load_avg +* se_runnable() = grq->h_nr_running +* +* runnable_sum = se_runnable() * runnable = grq->runnable_sum +* runnable_avg = runnable_sum * * load_sum := runnable * load_avg = se_weight(se) * load_sum * -* XXX collapse load_sum and runnable_load_sum -* * cfq_rq: * +* runnable_sum = \Sum se->avg.runnable_sum +* runnable_avg = \Sum se->avg.runnable_avg +* * load_sum = \Sum se_weight(se) * se->avg.load_sum * load_avg = \Sum se->avg.load_avg */ int __update_load_avg_blocked_se (u64 now,struct sched_entity *se) { -if (___update_load_sum (now, &se->avg,0 ,0 )) { +if (___update_load_sum (now, &se->avg,0 ,0 ,0 )) {___update_load_avg (&se->avg,se_weight (se));trace_pelt_se_tp (se);return 1 ;
@@ -278,7 +289,8 @@ int __update_load_avg_blocked_se(u64 now, struct sched_entity *se)int __update_load_avg_se (u64 now,struct cfs_rq *cfs_rq,struct sched_entity *se) { -if (___update_load_sum (now, &se->avg, !!se->on_rq, cfs_rq->curr == se)) { +if (___update_load_sum (now, &se->avg, !!se->on_rq,se_runnable (se), + cfs_rq->curr == se)) {___update_load_avg (&se->avg,se_weight (se));cfs_se_util_change (&se->avg);
@@ -293,6 +305,7 @@ int __update_load_avg_cfs_rq(u64 now, struct cfs_rq *cfs_rq) {if (___update_load_sum (now, &cfs_rq->avg,scale_load_down (cfs_rq->load.weight), + cfs_rq->h_nr_running, cfs_rq->curr !=NULL )) {___update_load_avg (&cfs_rq->avg,1 );
@@ -310,13 +323,14 @@ int __update_load_avg_cfs_rq(u64 now, struct cfs_rq *cfs_rq)* util_sum = cpu_scale * load_sum * runnable_sum = util_sum * -* load_avg is not supported and meaningless. +* load_avg and runnable_avg are not supported and meaningless. * */ int update_rt_rq_load_avg (u64 now,struct rq *rq,int running) {if (___update_load_sum (now, &rq->avg_rt, + running, running, running)) {
@@ -335,13 +349,14 @@ int update_rt_rq_load_avg(u64 now, struct rq *rq, int running)* util_sum = cpu_scale * load_sum * runnable_sum = util_sum * -* load_avg is not supported and meaningless. +* load_avg and runnable_avg are not supported and meaningless. * */ int update_dl_rq_load_avg (u64 now,struct rq *rq,int running) {if (___update_load_sum (now, &rq->avg_dl, + running, running, running)) {
@@ -361,7 +376,7 @@ int update_dl_rq_load_avg(u64 now, struct rq *rq, int running)* util_sum = cpu_scale * load_sum * runnable_sum = util_sum * -* load_avg is not supported and meaningless. +* load_avg and runnable_avg are not supported and meaningless. * */
@@ -389,9 +404,11 @@ int update_irq_load_avg(struct rq *rq, u64 running)* rq->clock += delta with delta >= running */ ret =___update_load_sum (rq->clock - running, &rq->avg_irq, +0 ,0 ,0 ); ret +=___update_load_sum (rq->clock, &rq->avg_irq, +1 ,1 ,1 );
kernel/sched/sched.h +21 -1
@@ -527,7 +527,7 @@ struct cfs_rq {int nr;unsigned long load_avg;unsigned long util_avg; -unsigned long runnable_sum; +unsigned long runnable_avg; } removed;#ifdef CONFIG_FAIR_GROUP_SCHED
@@ -688,9 +688,29 @@ struct dl_rq {/* An entity is a task if it doesn't "own" a runqueue */ #define entity_is_task (se) (!se->my_q) +static inline void se_update_runnable (struct sched_entity *se) +{ +if (!entity_is_task (se)) + se->runnable_weight = se->my_q->h_nr_running; +} + +static inline long se_runnable (struct sched_entity *se) +{ +if (entity_is_task (se)) +return !!se->on_rq; +else +return se->runnable_weight; +} +#else #define entity_is_task (se) 1 +static inline void se_update_runnable (struct sched_entity *se) {} + +static inline long se_runnable (struct sched_entity *se) +{ +return !!se->on_rq; +}#endif #ifdef CONFIG_SMP
Signature · 1 line
2.16.4
[tip: sched/core] sched/pelt: Add a new runnable average signal
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 9f68395333ad7f5bfe2f83473fed363d4229f11c Gitweb: https://git.kernel.org/tip/9f68395333ad7f5bfe2f83473fed363d4229f11c Author: Vincent Guittot <[email protected]> AuthorDate: Mon, 24 Feb 2020 09:52:18 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 24 Feb 2020 11:36:36 +01:00
sched/pelt: Add a new runnable average signal
Now that runnable_load_avg has been removed, we can replace it by a new signal that will highlight the runnable pressure on a cfs_rq. This signal track the waiting time of tasks on rq and can help to better define the state of rqs.
At now, only util_avg is used to define the state of a rq:
A rq with more that around 80% of utilization and more than 1 tasks is considered as overloaded.
But the util_avg signal of a rq can become temporaly low after that a task migrated onto another rq which can bias the classification of the rq.
When tasks compete for the same rq, their runnable average signal will be higher than util_avg as it will include the waiting time and we can use this signal to better classify cfs_rqs.
The new runnable_avg will track the runnable time of a task which simply adds the waiting time to the running time. The runnable _avg of cfs_rq will be the /Sum of se's runnable_avg and the runnable_avg of group entity will follow the one of the rq similarly to util_avg.
--- include/linux/sched.h | 26 ++++++++------ kernel/sched/debug.c | 9 +++-- kernel/sched/fair.c | 77 ++++++++++++++++++++++++++++++++++++++---- kernel/sched/pelt.c | 39 +++++++++++++++------ kernel/sched/sched.h | 22 +++++++++++- 5 files changed, 142 insertions(+), 31 deletions(-)
include/linux/sched.h +16 -10
@@ -356,28 +356,30 @@ struct util_est { } __attribute__((__aligned__(sizeof(u64))));/* -* The load_avg/util_avg accumulates an infinite geometric series +* The load/runnable/util_avg accumulates an infinite geometric series * (see __update_load_avg_cfs_rq() in kernel/sched/pelt.c). * * [load_avg definition] * * load_avg = runnable% * scale_load_down(load) * -* where runnable% is the time ratio that a sched_entity is runnable. -* For cfs_rq, it is the aggregated load_avg of all runnable and -* blocked sched_entities. +* [runnable_avg definition] +* +* runnable_avg = runnable% * SCHED_CAPACITY_SCALE * * [util_avg definition] * * util_avg = running% * SCHED_CAPACITY_SCALE * -* where running% is the time ratio that a sched_entity is running on -* a CPU. For cfs_rq, it is the aggregated util_avg of all runnable -* and blocked sched_entities. +* where runnable% is the time ratio that a sched_entity is runnable and +* running% the time ratio that a sched_entity is running. +* +* For cfs_rq, they are the aggregated values of all runnable and blocked +* sched_entities. * -* load_avg and util_avg don't direcly factor frequency scaling and CPU -* capacity scaling. The scaling is done through the rq_clock_pelt that -* is used for computing those signals (see update_rq_clock_pelt()) +* The load/runnable/util_avg doesn't direcly factor frequency scaling and CPU +* capacity scaling. The scaling is done through the rq_clock_pelt that is used +* for computing those signals (see update_rq_clock_pelt()) * * N.B., the above ratios (runnable% and running%) themselves are in the * range of [0, 1]. To do fixed point arithmetics, we therefore scale them
@@ -401,9 +403,11 @@ struct util_est {struct sched_avg {u64 last_update_time;u64 load_sum; +u64 runnable_sum;u32 util_sum;u32 period_contrib;unsigned long load_avg; +unsigned long runnable_avg;unsigned long util_avg;struct util_est util_est; }____cacheline_aligned ;
@@ -467,6 +471,8 @@ struct sched_entity {struct cfs_rq *cfs_rq;/* rq "owned" by this entity/group: */ struct cfs_rq *my_q; +/* cached value of my_q->h_nr_running */ +unsigned long runnable_weight;#endif #ifdef CONFIG_SMP
kernel/sched/debug.c +7 -2
@@ -405,6 +405,7 @@ static void print_cfs_group_stats(struct seq_file *m, int cpu, struct task_group#ifdef CONFIG_SMP P (se->avg.load_avg);P (se->avg.util_avg); +P (se->avg.runnable_avg);#endif #undef PN_SCHEDSTAT
@@ -524,6 +525,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)#ifdef CONFIG_SMP SEQ_printf (m," .%-30s: %lu\n" ,"load_avg" , cfs_rq->avg.load_avg); +SEQ_printf (m," .%-30s: %lu\n" ,"runnable_avg" , + cfs_rq->avg.runnable_avg);SEQ_printf (m," .%-30s: %lu\n" ,"util_avg" , cfs_rq->avg.util_avg); SEQ_printf(m, " .%-30s: %u\n", "util_est_enqueued",
@@ -532,8 +535,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq) cfs_rq->removed.load_avg);SEQ_printf (m," .%-30s: %ld\n" ,"removed.util_avg" , cfs_rq->removed.util_avg); -SEQ_printf (m," .%-30s: %ld\n" ,"removed.runnable_sum" , - cfs_rq->removed.runnable_sum); +SEQ_printf (m," .%-30s: %ld\n" ,"removed.runnable_avg" , + cfs_rq->removed.runnable_avg);#ifdef CONFIG_FAIR_GROUP_SCHED SEQ_printf (m," .%-30s: %lu\n" ,"tg_load_avg_contrib" , cfs_rq->tg_load_avg_contrib);
@@ -944,8 +947,10 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,P (se.load.weight);#ifdef CONFIG_SMP P (se.avg.load_sum); +P (se.avg.runnable_sum);P (se.avg.util_sum);P (se.avg.load_avg); +P (se.avg.runnable_avg);P (se.avg.util_avg);P (se.avg.last_update_time);P (se.avg.util_est.ewma);
kernel/sched/fair.c +70 -7
192 lines not on this page · raw message
kernel/sched/pelt.c +28 -11
@@ -108,7 +108,7 @@ static u32 __accumulate_pelt_segments(u64 periods, u32 d1, u32 d3)*/ static __always_inline u32accumulate_sum (u64 delta,struct sched_avg *sa, -unsigned long load,int running) +unsigned long load,unsigned long runnable,int running) {u32 contrib = (u32 )delta;/* p == 0 -> delta < 1024 */ u64 periods;
@@ -121,6 +121,8 @@ accumulate_sum(u64 delta, struct sched_avg *sa,*/ if (periods) { sa->load_sum =decay_load (sa->load_sum, periods); + sa->runnable_sum = +decay_load (sa->runnable_sum, periods); sa->util_sum =decay_load ((u64)(sa->util_sum), periods);/*
@@ -146,6 +148,8 @@ accumulate_sum(u64 delta, struct sched_avg *sa,if (load) sa->load_sum += load * contrib; +if (runnable) + sa->runnable_sum += runnable * contrib <<SCHED_CAPACITY_SHIFT ;if (running) sa->util_sum += contrib <<SCHED_CAPACITY_SHIFT ;
@@ -182,7 +186,7 @@ accumulate_sum(u64 delta, struct sched_avg *sa,*/ static __always_inline int___update_load_sum (u64 now,struct sched_avg *sa, -unsigned long load,int running) +unsigned long load,unsigned long runnable,int running) {u64 delta;
@@ -218,7 +222,7 @@ ___update_load_sum(u64 now, struct sched_avg *sa,* Also see the comment in accumulate_sum(). */ if (!load) - running =0 ; + runnable = running =0 ;/* * Now we know we crossed measurement unit boundaries. The *_avg
@@ -227,7 +231,7 @@ ___update_load_sum(u64 now, struct sched_avg *sa,* Step 1: accumulate *_sum since last_update_time. If we haven't * crossed period boundaries, finish. */ -if (!accumulate_sum (delta, sa, load, running)) +if (!accumulate_sum (delta, sa, load, runnable, running))return 0 ;return 1 ;
@@ -242,6 +246,7 @@ ___update_load_avg(struct sched_avg *sa, unsigned long load)* Step 2: update *_avg. */ sa->load_avg =div_u64 (load * sa->load_sum, divider); + sa->runnable_avg =div_u64 (sa->runnable_sum, divider);WRITE_ONCE (sa->util_avg, sa->util_sum / divider); }
@@ -250,24 +255,30 @@ ___update_load_avg(struct sched_avg *sa, unsigned long load)* * task: * se_weight() = se->load.weight +* se_runnable() = !!on_rq * * group: [ see update_cfs_group() ] * se_weight() = tg->weight * grq->load_avg / tg->load_avg +* se_runnable() = grq->h_nr_running +* +* runnable_sum = se_runnable() * runnable = grq->runnable_sum +* runnable_avg = runnable_sum * * load_sum := runnable * load_avg = se_weight(se) * load_sum * -* XXX collapse load_sum and runnable_load_sum -* * cfq_rq: * +* runnable_sum = \Sum se->avg.runnable_sum +* runnable_avg = \Sum se->avg.runnable_avg +* * load_sum = \Sum se_weight(se) * se->avg.load_sum * load_avg = \Sum se->avg.load_avg */ int __update_load_avg_blocked_se (u64 now,struct sched_entity *se) { -if (___update_load_sum (now, &se->avg,0 ,0 )) { +if (___update_load_sum (now, &se->avg,0 ,0 ,0 )) {___update_load_avg (&se->avg,se_weight (se));trace_pelt_se_tp (se);return 1 ;
@@ -278,7 +289,8 @@ int __update_load_avg_blocked_se(u64 now, struct sched_entity *se)int __update_load_avg_se (u64 now,struct cfs_rq *cfs_rq,struct sched_entity *se) { -if (___update_load_sum (now, &se->avg, !!se->on_rq, cfs_rq->curr == se)) { +if (___update_load_sum (now, &se->avg, !!se->on_rq,se_runnable (se), + cfs_rq->curr == se)) {___update_load_avg (&se->avg,se_weight (se));cfs_se_util_change (&se->avg);
@@ -293,6 +305,7 @@ int __update_load_avg_cfs_rq(u64 now, struct cfs_rq *cfs_rq) {if (___update_load_sum (now, &cfs_rq->avg,scale_load_down (cfs_rq->load.weight), + cfs_rq->h_nr_running, cfs_rq->curr !=NULL )) {___update_load_avg (&cfs_rq->avg,1 );
@@ -310,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, struct cfs_rq *cfs_rq)* util_sum = cpu_scale * load_sum * runnable_sum = util_sum * -* load_avg is not supported and meaningless. +* load_avg and runnable_avg are not supported and meaningless. * */
@@ -318,6 +331,7 @@ int update_rt_rq_load_avg(u64 now, struct rq *rq, int running) {if (___update_load_sum (now, &rq->avg_rt, running, + running, running)) {___update_load_avg (&rq->avg_rt,1 );
@@ -335,7 +349,7 @@ int update_rt_rq_load_avg(u64 now, struct rq *rq, int running)* util_sum = cpu_scale * load_sum * runnable_sum = util_sum * -* load_avg is not supported and meaningless. +* load_avg and runnable_avg are not supported and meaningless. * */
@@ -343,6 +357,7 @@ int update_dl_rq_load_avg(u64 now, struct rq *rq, int running) {if (___update_load_sum (now, &rq->avg_dl, running, + running, running)) {___update_load_avg (&rq->avg_dl,1 );
@@ -361,7 +376,7 @@ int update_dl_rq_load_avg(u64 now, struct rq *rq, int running)* util_sum = cpu_scale * load_sum * runnable_sum = util_sum * -* load_avg is not supported and meaningless. +* load_avg and runnable_avg are not supported and meaningless. * */
@@ -390,9 +405,11 @@ int update_irq_load_avg(struct rq *rq, u64 running)*/ ret =___update_load_sum (rq->clock - running, &rq->avg_irq,0 , +0 ,0 ); ret +=___update_load_sum (rq->clock, &rq->avg_irq,1 , +1 ,1 );if (ret) {
kernel/sched/sched.h +21 -1
@@ -527,7 +527,7 @@ struct cfs_rq {int nr;unsigned long load_avg;unsigned long util_avg; -unsigned long runnable_sum; +unsigned long runnable_avg; } removed;#ifdef CONFIG_FAIR_GROUP_SCHED
@@ -688,9 +688,29 @@ struct dl_rq {/* An entity is a task if it doesn't "own" a runqueue */ #define entity_is_task (se) (!se->my_q) +static inline void se_update_runnable (struct sched_entity *se) +{ +if (!entity_is_task (se)) + se->runnable_weight = se->my_q->h_nr_running; +} + +static inline long se_runnable (struct sched_entity *se) +{ +if (entity_is_task (se)) +return !!se->on_rq; +else +return se->runnable_weight; +} +#else #define entity_is_task (se) 1 +static inline void se_update_runnable (struct sched_entity *se) {} + +static inline long se_runnable (struct sched_entity *se) +{ +return !!se->on_rq; +}#endif #ifdef CONFIG_SMP
Valentin Schneider
Reviewed-byRe: [tip: sched/core] sched/pelt: Add a new runnable average signal
Quoted 5 lines · tip-bot2 for Vincent Guittot writes:
Signed-off-by Vincent Guittot <[email protected]>Signed-off-by Mel Gorman <[email protected]>Signed-off-by Ingo Molnar <[email protected]>Reviewed-by "Dietmar Eggemann <[email protected]>"Acked-by Peter Zijlstra <[email protected]>
With the fork time initialization thing being sorted out, the rest of the runnable series can claim my
but I doubt any of that is worth the hassle since it's in tip already. Just figured I'd mention it, being in Cc and all :-)
Re: [tip: sched/core] sched/pelt: Add a new runnable average signal
Quoted 9 lines · Valentin Schneider
Quoted 5 lines · tip-bot2 for Vincent Guittot writes:
Signed-off-by Vincent Guittot <[email protected]>Signed-off-by Mel Gorman <[email protected]>Signed-off-by Ingo Molnar <[email protected]>Reviewed-by "Dietmar Eggemann <[email protected]>"Acked-by Peter Zijlstra <[email protected]>
Quoted 7 lines
With the fork time initialization thing being sorted out, the rest of the runnable series can claim my
Reviewed-by Valentin Schneider <[email protected]>but I doubt any of that is worth the hassle since it's in tip already. Just figured I'd mention it, being in Cc and all :-)
Whether the tag gets included or not, it's nice to have definite confirmation that you're ok with this version!
Signature · 2 lines
Mel Gorman SUSE Labs
Vincent Guittot
maintainerRe: [tip: sched/core] sched/pelt: Add a new runnable average signal
In reply to Valentin SchneiderQuoted 10 lines · Valentin Schneider
Quoted 5 lines · tip-bot2 for Vincent Guittot writes:
Signed-off-by Vincent Guittot <[email protected]>Signed-off-by Mel Gorman <[email protected]>Signed-off-by Ingo Molnar <[email protected]>Reviewed-by "Dietmar Eggemann <[email protected]>"Acked-by Peter Zijlstra <[email protected]>
Quoted 4 lines
With the fork time initialization thing being sorted out, the rest of the runnable series can claim my
Reviewed-by Valentin Schneider <[email protected]>
Thanks
Quoted 3 lines
but I doubt any of that is worth the hassle since it's in tip already. Just figured I'd mention it, being in Cc and all :-)
PATCH 9/13 sched/fair: Take into account runnable_avg to classify group
In reply to Mel GormanFrom: Vincent Guittot <[email protected]>
Take into account the new runnable_avg signal to classify a group and to mitigate the volatility of util_avg in face of intensive migration or new task with random utilization.
--- kernel/sched/fair.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)
kernel/sched/fair.c +30 -1
@@ -5470,6 +5470,24 @@ static unsigned long cpu_runnable(struct rq *rq)return cfs_rq_runnable_avg (&rq->cfs); } +static unsigned long cpu_runnable_without (struct rq *rq,struct task_struct *p) +{ +struct cfs_rq *cfs_rq; +unsigned int runnable; + +/* Task has no contribution or is new */ +if (cpu_of (rq) !=task_cpu (p) || !READ_ONCE (p->se.avg.last_update_time)) +return cpu_runnable (rq); + + cfs_rq = &rq->cfs; + runnable =READ_ONCE (cfs_rq->avg.runnable_avg); + +/* Discount task's runnable from CPU's runnable */ +lsub_positive (&runnable, p->se.avg.runnable_avg); + +return runnable; +} +static unsigned long capacity_of (int cpu) {return cpu_rq (cpu)->cpu_capacity;
@@ -7753,7 +7771,8 @@ struct sg_lb_stats {unsigned long avg_load;/*Avg load across the CPUs of the group */ unsigned long group_load;/* Total load over the CPUs of the group */ unsigned long group_capacity; -unsigned long group_util;/* Total utilization of the group */ +unsigned long group_util;/* Total utilization over the CPUs of the group */ +unsigned long group_runnable;/* Total runnable time over the CPUs of the group */ unsigned int sum_nr_running;/* Nr of tasks running in the group */ unsigned int sum_h_nr_running;/* Nr of CFS tasks running in the group */ unsigned int idle_cpus;
@@ -7974,6 +7993,10 @@ group_has_capacity(unsigned int imbalance_pct, struct sg_lb_stats *sgs)if (sgs->sum_nr_running < sgs->group_weight)return true; +if ((sgs->group_capacity * imbalance_pct) < + (sgs->group_runnable *100 )) +return false; +if ((sgs->group_capacity *100 ) > (sgs->group_util * imbalance_pct))return true;
@@ -7999,6 +8022,10 @@ group_is_overloaded(unsigned int imbalance_pct, struct sg_lb_stats *sgs) (sgs->group_util * imbalance_pct))return true; +if ((sgs->group_capacity * imbalance_pct) < + (sgs->group_runnable *100 )) +return true; +return false; }
@@ -8093,6 +8120,7 @@ static inline void update_sg_lb_stats(struct lb_env *env, sgs->group_load +=cpu_load (rq); sgs->group_util +=cpu_util (i); + sgs->group_runnable +=cpu_runnable (rq); sgs->sum_h_nr_running += rq->cfs.h_nr_running; nr_running = rq->nr_running;
@@ -8368,6 +8396,7 @@ static inline void update_sg_wakeup_stats(struct sched_domain *sd, sgs->group_load +=cpu_load_without (rq, p); sgs->group_util +=cpu_util_without (i, p); + sgs->group_runnable +=cpu_runnable_without (rq, p); local =task_running_on_cpu (i, p); sgs->sum_h_nr_running += rq->cfs.h_nr_running - local;
Signature · 1 line
2.16.4
[tip: sched/core] sched/fair: Take into account runnable_avg to classify group
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 070f5e860ee2bf588c99ef7b4c202451faa48236 Gitweb: https://git.kernel.org/tip/070f5e860ee2bf588c99ef7b4c202451faa48236 Author: Vincent Guittot <[email protected]> AuthorDate: Mon, 24 Feb 2020 09:52:19 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 24 Feb 2020 11:36:37 +01:00
sched/fair: Take into account runnable_avg to classify group
Take into account the new runnable_avg signal to classify a group and to mitigate the volatility of util_avg in face of intensive migration or new task with random utilization.
--- kernel/sched/fair.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)
kernel/sched/fair.c +30 -1
@@ -5469,6 +5469,24 @@ static unsigned long cpu_runnable(struct rq *rq)return cfs_rq_runnable_avg (&rq->cfs); } +static unsigned long cpu_runnable_without (struct rq *rq,struct task_struct *p) +{ +struct cfs_rq *cfs_rq; +unsigned int runnable; + +/* Task has no contribution or is new */ +if (cpu_of (rq) !=task_cpu (p) || !READ_ONCE (p->se.avg.last_update_time)) +return cpu_runnable (rq); + + cfs_rq = &rq->cfs; + runnable =READ_ONCE (cfs_rq->avg.runnable_avg); + +/* Discount task's runnable from CPU's runnable */ +lsub_positive (&runnable, p->se.avg.runnable_avg); + +return runnable; +} +static unsigned long capacity_of (int cpu) {return cpu_rq (cpu)->cpu_capacity;
@@ -7752,7 +7770,8 @@ struct sg_lb_stats {unsigned long avg_load;/*Avg load across the CPUs of the group */ unsigned long group_load;/* Total load over the CPUs of the group */ unsigned long group_capacity; -unsigned long group_util;/* Total utilization of the group */ +unsigned long group_util;/* Total utilization over the CPUs of the group */ +unsigned long group_runnable;/* Total runnable time over the CPUs of the group */ unsigned int sum_nr_running;/* Nr of tasks running in the group */ unsigned int sum_h_nr_running;/* Nr of CFS tasks running in the group */ unsigned int idle_cpus;
@@ -7973,6 +7992,10 @@ group_has_capacity(unsigned int imbalance_pct, struct sg_lb_stats *sgs)if (sgs->sum_nr_running < sgs->group_weight)return true; +if ((sgs->group_capacity * imbalance_pct) < + (sgs->group_runnable *100 )) +return false; +if ((sgs->group_capacity *100 ) > (sgs->group_util * imbalance_pct))return true;
@@ -7998,6 +8021,10 @@ group_is_overloaded(unsigned int imbalance_pct, struct sg_lb_stats *sgs) (sgs->group_util * imbalance_pct))return true; +if ((sgs->group_capacity * imbalance_pct) < + (sgs->group_runnable *100 )) +return true; +return false; }
@@ -8092,6 +8119,7 @@ static inline void update_sg_lb_stats(struct lb_env *env, sgs->group_load +=cpu_load (rq); sgs->group_util +=cpu_util (i); + sgs->group_runnable +=cpu_runnable (rq); sgs->sum_h_nr_running += rq->cfs.h_nr_running; nr_running = rq->nr_running;
@@ -8367,6 +8395,7 @@ static inline void update_sg_wakeup_stats(struct sched_domain *sd, sgs->group_load +=cpu_load_without (rq, p); sgs->group_util +=cpu_util_without (i, p); + sgs->group_runnable +=cpu_runnable_without (rq, p); local =task_running_on_cpu (i, p); sgs->sum_h_nr_running += rq->cfs.h_nr_running - local;
PATCH 10/13 sched/numa: Prefer using an idle cpu as a migration target instead of comparing tasks
In reply to Mel Gormantask_numa_find_cpu can scan a node multiple times. Minimally it scans to gather statistics and later to find a suitable target. In some cases, the second scan will simply pick an idle CPU if the load is not imbalanced.
This patch caches information on an idle core while gathering statistics and uses it immediately if load is not imbalanced to avoid a second scan of the node runqueues. Preference is given to an idle core rather than an idle SMT sibling to avoid packing HT siblings due to linearly scanning the node cpumask.
As a side-effect, even when the second scan is necessary, the importance of using select_idle_sibling is much reduced because information on idle CPUs is cached and can be reused.
Note that this patch actually makes is harder to move to an idle CPU as multiple tasks can race for the same idle CPU due to a race checking numa_migrate_on. This is addressed in the next patch.
--- kernel/sched/fair.c | 119 ++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 102 insertions(+), 17 deletions(-)
kernel/sched/fair.c +102 -17
@@ -1500,8 +1500,29 @@ struct numa_stats {unsigned int nr_running;unsigned int weight;enum numa_type node_type; +int idle_cpu; }; +static inline bool is_core_idle (int cpu) +{ +#ifdef CONFIG_SCHED_SMT +int sibling; + +for_each_cpu (sibling,cpu_smt_mask (cpu)) { +if (cpu == sibling) +continue ; + +if (!idle_cpu (cpu)) +return false; + } +#endif + +return true; +} + +/* Forward declarations of select_idle_sibling helpers */ +static inline bool test_idle_cores (int cpu,bool def); +struct task_numa_env {struct task_struct *p;
@@ -1537,15 +1558,39 @@ numa_type numa_classify(unsigned int imbalance_pct,return node_fully_busy; } +static inline int numa_idle_core (int idle_core,int cpu) +{ +#ifdef CONFIG_SCHED_SMT +if (!static_branch_likely (&sched_smt_present) || + idle_core >=0 || !test_idle_cores (cpu, false)) +return idle_core; + +/* +* Prefer cores instead of packing HT siblings +* and triggering future load balancing. +*/ +if (is_core_idle (cpu)) + idle_core = cpu; +#endif + +return idle_core; +} +/* -* XXX borrowed from update_sg_lb_stats +* Gather all necessary information to make NUMA balancing placement +* decisions that are compatible with standard load balancer. This +* borrows code and logic from update_sg_lb_stats but sharing a +* common implementation is impractical. */ static void update_numa_stats (struct task_numa_env *env, -struct numa_stats *ns,int nid) +struct numa_stats *ns,int nid, +bool find_idle) { -int cpu; +int cpu, idle_core =-1 ;memset (ns,0 ,sizeof (*ns)); + ns->idle_cpu =-1 ; +for_each_cpu (cpu,cpumask_of_node (nid)) {struct rq *rq =cpu_rq (cpu);
@@ -1553,11 +1598,25 @@ static void update_numa_stats(struct task_numa_env *env, ns->util +=cpu_util (cpu); ns->nr_running += rq->cfs.h_nr_running; ns->compute_capacity +=capacity_of (cpu); + +if (find_idle && !rq->nr_running &&idle_cpu (cpu)) { +if (READ_ONCE (rq->numa_migrate_on) || + !cpumask_test_cpu (cpu, env->p->cpus_ptr)) +continue ; + +if (ns->idle_cpu ==-1 ) + ns->idle_cpu = cpu; + + idle_core =numa_idle_core (idle_core, cpu); + } } ns->weight =cpumask_weight (cpumask_of_node (nid)); ns->node_type =numa_classify (env->imbalance_pct, ns); + +if (idle_core >=0 ) + ns->idle_cpu = idle_core; } static void task_numa_assign(struct task_numa_env *env,
@@ -1566,7 +1625,7 @@ static void task_numa_assign(struct task_numa_env *env,struct rq *rq =cpu_rq (env->dst_cpu);/* Bail out if run-queue part of active NUMA balance. */ -if (xchg (&rq->numa_migrate_on,1 )) +if (env->best_cpu != env->dst_cpu &&xchg (&rq->numa_migrate_on,1 ))return ;/*
@@ -1730,19 +1789,39 @@ static void task_numa_compare(struct task_numa_env *env, goto unlock; assign: -/* -* One idle CPU per node is evaluated for a task numa move. -* Call select_idle_sibling to maybe find a better one. -*/ +/* Evaluate an idle CPU for a task numa move. */ if (!cur) { +int cpu = env->dst_stats.idle_cpu; + +/* Nothing cached so current CPU went idle since the search. */ +if (cpu <0 ) + cpu = env->dst_cpu; +/* -* select_idle_siblings() uses an per-CPU cpumask that -* can be used from IRQ context. +* If the CPU is no longer truly idle and the previous best CPU +* is, keep using it. */ -local_irq_disable (); - env->dst_cpu =select_idle_sibling (env->p, env->src_cpu, +if (!idle_cpu (cpu) && env->best_cpu >=0 && +idle_cpu (env->best_cpu)) { + cpu = env->best_cpu; + } + +/* +* Use select_idle_sibling if the previously found idle CPU is +* not idle any more. +*/ +if (!idle_cpu (cpu)) { +/* +* select_idle_siblings() uses an per-CPU cpumask that +* can be used from IRQ context. +*/ +local_irq_disable (); + cpu =select_idle_sibling (env->p, env->src_cpu, env->dst_cpu); -local_irq_enable (); +local_irq_enable (); + } + + env->dst_cpu = cpu; }task_numa_assign (env, cur, imp);
@@ -1776,8 +1855,14 @@ static void task_numa_find_cpu(struct task_numa_env *env, imbalance =adjust_numa_imbalance (imbalance, src_running);/* Use idle CPU if there is no imbalance */ -if (!imbalance) +if (!imbalance) { maymove = true; +if (env->dst_stats.idle_cpu >=0 ) { + env->dst_cpu = env->dst_stats.idle_cpu; +task_numa_assign (env,NULL ,0 ); +return ; + } + } }else {long src_load, dst_load, load;/*
@@ -1850,10 +1935,10 @@ static int task_numa_migrate(struct task_struct *p) dist = env.dist =node_distance (env.src_nid, env.dst_nid); taskweight =task_weight (p, env.src_nid, dist); groupweight =group_weight (p, env.src_nid, dist); -update_numa_stats (&env, &env.src_stats, env.src_nid); +update_numa_stats (&env, &env.src_stats, env.src_nid, false); taskimp =task_weight (p, env.dst_nid, dist) - taskweight; groupimp =group_weight (p, env.dst_nid, dist) - groupweight; -update_numa_stats (&env, &env.dst_stats, env.dst_nid); +update_numa_stats (&env, &env.dst_stats, env.dst_nid, true);/* Try to find a spot on the preferred nid. */ task_numa_find_cpu (&env, taskimp, groupimp);
@@ -1886,7 +1971,7 @@ static int task_numa_migrate(struct task_struct *p) env.dist = dist; env.dst_nid = nid; -update_numa_stats (&env, &env.dst_stats, env.dst_nid); +update_numa_stats (&env, &env.dst_stats, env.dst_nid, true);task_numa_find_cpu (&env, taskimp, groupimp); } }
Signature · 1 line
2.16.4
[tip: sched/core] sched/numa: Prefer using an idle CPU as a migration target instead of comparing tasks
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: ff7db0bf24db919f69121bf5df8f3cb6d79f49af Gitweb: https://git.kernel.org/tip/ff7db0bf24db919f69121bf5df8f3cb6d79f49af Author: Mel Gorman <[email protected]> AuthorDate: Mon, 24 Feb 2020 09:52:20 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 24 Feb 2020 11:36:38 +01:00
sched/numa: Prefer using an idle CPU as a migration target instead of comparing tasks
task_numa_find_cpu() can scan a node multiple times. Minimally it scans to gather statistics and later to find a suitable target. In some cases, the second scan will simply pick an idle CPU if the load is not imbalanced.
This patch caches information on an idle core while gathering statistics and uses it immediately if load is not imbalanced to avoid a second scan of the node runqueues. Preference is given to an idle core rather than an idle SMT sibling to avoid packing HT siblings due to linearly scanning the node cpumask.
As a side-effect, even when the second scan is necessary, the importance of using select_idle_sibling is much reduced because information on idle CPUs is cached and can be reused.
Note that this patch actually makes is harder to move to an idle CPU as multiple tasks can race for the same idle CPU due to a race checking numa_migrate_on. This is addressed in the next patch.
--- kernel/sched/fair.c | 119 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 102 insertions(+), 17 deletions(-)
kernel/sched/fair.c +102 -17
@@ -1500,8 +1500,29 @@ struct numa_stats {unsigned int nr_running;unsigned int weight;enum numa_type node_type; +int idle_cpu; }; +static inline bool is_core_idle (int cpu) +{ +#ifdef CONFIG_SCHED_SMT +int sibling; + +for_each_cpu (sibling,cpu_smt_mask (cpu)) { +if (cpu == sibling) +continue ; + +if (!idle_cpu (cpu)) +return false; + } +#endif + +return true; +} + +/* Forward declarations of select_idle_sibling helpers */ +static inline bool test_idle_cores (int cpu,bool def); +struct task_numa_env {struct task_struct *p;
@@ -1537,15 +1558,39 @@ numa_type numa_classify(unsigned int imbalance_pct,return node_fully_busy; } +static inline int numa_idle_core (int idle_core,int cpu) +{ +#ifdef CONFIG_SCHED_SMT +if (!static_branch_likely (&sched_smt_present) || + idle_core >=0 || !test_idle_cores (cpu, false)) +return idle_core; + +/* +* Prefer cores instead of packing HT siblings +* and triggering future load balancing. +*/ +if (is_core_idle (cpu)) + idle_core = cpu; +#endif + +return idle_core; +} +/* -* XXX borrowed from update_sg_lb_stats +* Gather all necessary information to make NUMA balancing placement +* decisions that are compatible with standard load balancer. This +* borrows code and logic from update_sg_lb_stats but sharing a +* common implementation is impractical. */ static void update_numa_stats (struct task_numa_env *env, -struct numa_stats *ns,int nid) +struct numa_stats *ns,int nid, +bool find_idle) { -int cpu; +int cpu, idle_core =-1 ;memset (ns,0 ,sizeof (*ns)); + ns->idle_cpu =-1 ; +for_each_cpu (cpu,cpumask_of_node (nid)) {struct rq *rq =cpu_rq (cpu);
@@ -1553,11 +1598,25 @@ static void update_numa_stats(struct task_numa_env *env, ns->util +=cpu_util (cpu); ns->nr_running += rq->cfs.h_nr_running; ns->compute_capacity +=capacity_of (cpu); + +if (find_idle && !rq->nr_running &&idle_cpu (cpu)) { +if (READ_ONCE (rq->numa_migrate_on) || + !cpumask_test_cpu (cpu, env->p->cpus_ptr)) +continue ; + +if (ns->idle_cpu ==-1 ) + ns->idle_cpu = cpu; + + idle_core =numa_idle_core (idle_core, cpu); + } } ns->weight =cpumask_weight (cpumask_of_node (nid)); ns->node_type =numa_classify (env->imbalance_pct, ns); + +if (idle_core >=0 ) + ns->idle_cpu = idle_core; } static void task_numa_assign(struct task_numa_env *env,
@@ -1566,7 +1625,7 @@ static void task_numa_assign(struct task_numa_env *env,struct rq *rq =cpu_rq (env->dst_cpu);/* Bail out if run-queue part of active NUMA balance. */ -if (xchg (&rq->numa_migrate_on,1 )) +if (env->best_cpu != env->dst_cpu &&xchg (&rq->numa_migrate_on,1 ))return ;/*
@@ -1730,19 +1789,39 @@ static void task_numa_compare(struct task_numa_env *env, goto unlock; assign: -/* -* One idle CPU per node is evaluated for a task numa move. -* Call select_idle_sibling to maybe find a better one. -*/ +/* Evaluate an idle CPU for a task numa move. */ if (!cur) { +int cpu = env->dst_stats.idle_cpu; + +/* Nothing cached so current CPU went idle since the search. */ +if (cpu <0 ) + cpu = env->dst_cpu; +/* -* select_idle_siblings() uses an per-CPU cpumask that -* can be used from IRQ context. +* If the CPU is no longer truly idle and the previous best CPU +* is, keep using it. */ -local_irq_disable (); - env->dst_cpu =select_idle_sibling (env->p, env->src_cpu, +if (!idle_cpu (cpu) && env->best_cpu >=0 && +idle_cpu (env->best_cpu)) { + cpu = env->best_cpu; + } + +/* +* Use select_idle_sibling if the previously found idle CPU is +* not idle any more. +*/ +if (!idle_cpu (cpu)) { +/* +* select_idle_siblings() uses an per-CPU cpumask that +* can be used from IRQ context. +*/ +local_irq_disable (); + cpu =select_idle_sibling (env->p, env->src_cpu, env->dst_cpu); -local_irq_enable (); +local_irq_enable (); + } + + env->dst_cpu = cpu; }task_numa_assign (env, cur, imp);
@@ -1776,8 +1855,14 @@ static void task_numa_find_cpu(struct task_numa_env *env, imbalance =adjust_numa_imbalance (imbalance, src_running);/* Use idle CPU if there is no imbalance */ -if (!imbalance) +if (!imbalance) { maymove = true; +if (env->dst_stats.idle_cpu >=0 ) { + env->dst_cpu = env->dst_stats.idle_cpu; +task_numa_assign (env,NULL ,0 ); +return ; + } + } }else {long src_load, dst_load, load;/*
@@ -1850,10 +1935,10 @@ static int task_numa_migrate(struct task_struct *p) dist = env.dist =node_distance (env.src_nid, env.dst_nid); taskweight =task_weight (p, env.src_nid, dist); groupweight =group_weight (p, env.src_nid, dist); -update_numa_stats (&env, &env.src_stats, env.src_nid); +update_numa_stats (&env, &env.src_stats, env.src_nid, false); taskimp =task_weight (p, env.dst_nid, dist) - taskweight; groupimp =group_weight (p, env.dst_nid, dist) - groupweight; -update_numa_stats (&env, &env.dst_stats, env.dst_nid); +update_numa_stats (&env, &env.dst_stats, env.dst_nid, true);/* Try to find a spot on the preferred nid. */ task_numa_find_cpu (&env, taskimp, groupimp);
@@ -1886,7 +1971,7 @@ static int task_numa_migrate(struct task_struct *p) env.dist = dist; env.dst_nid = nid; -update_numa_stats (&env, &env.dst_stats, env.dst_nid); +update_numa_stats (&env, &env.dst_stats, env.dst_nid, true);task_numa_find_cpu (&env, taskimp, groupimp); } }
PATCH 11/13 sched/numa: Find an alternative idle CPU if the CPU is part of an active NUMA balance
In reply to Mel GormanMultiple tasks can attempt to select and idle CPU but fail because numa_migrate_on is already set and the migration fails. Instead of failing, scan for an alternative idle CPU. select_idle_sibling is not used because it requires IRQs to be disabled and it ignores numa_migrate_on allowing multiple tasks to stack. This scan may still fail if there are idle candidate CPUs due to races but if this occurs, it's best that a task stay on an available CPU that move to a contended one.
--- kernel/sched/fair.c | 40 ++++++++++++++++++++++------------------ 1 file changed, 22 insertions(+), 18 deletions(-)
kernel/sched/fair.c +22 -18
@@ -1624,15 +1624,34 @@ static void task_numa_assign(struct task_numa_env *env, {struct rq *rq =cpu_rq (env->dst_cpu); -/* Bail out if run-queue part of active NUMA balance. */ -if (env->best_cpu != env->dst_cpu &&xchg (&rq->numa_migrate_on,1 )) +/* Check if run-queue part of active NUMA balance. */ +if (env->best_cpu != env->dst_cpu &&xchg (&rq->numa_migrate_on,1 )) { +int cpu; +int start = env->dst_cpu; + +/* Find alternative idle CPU. */ +for_each_cpu_wrap (cpu,cpumask_of_node (env->dst_nid), start) { +if (cpu == env->best_cpu || !idle_cpu (cpu) || + !cpumask_test_cpu (cpu, env->p->cpus_ptr)) { +continue ; + } + + env->dst_cpu = cpu; + rq =cpu_rq (env->dst_cpu); +if (!xchg (&rq->numa_migrate_on,1 )) + goto assign; + } + +/* Failed to find an alternative idle CPU */ return ; + } +assign:/* * Clear previous best_cpu/rq numa-migrate flag, since task now * found a better CPU to move/swap. */ -if (env->best_cpu !=-1 ) { +if (env->best_cpu !=-1 && env->best_cpu != env->dst_cpu) { rq =cpu_rq (env->best_cpu);WRITE_ONCE (rq->numa_migrate_on,0 ); }
@@ -1806,21 +1825,6 @@ static void task_numa_compare(struct task_numa_env *env, cpu = env->best_cpu; } -/* -* Use select_idle_sibling if the previously found idle CPU is -* not idle any more. -*/ -if (!idle_cpu (cpu)) { -/* -* select_idle_siblings() uses an per-CPU cpumask that -* can be used from IRQ context. -*/ -local_irq_disable (); - cpu =select_idle_sibling (env->p, env->src_cpu, - env->dst_cpu); -local_irq_enable (); - } - env->dst_cpu = cpu; }
Signature · 1 line
2.16.4
[tip: sched/core] sched/numa: Find an alternative idle CPU if the CPU is part of an active NUMA balance
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 5fb52dd93a2fe9a738f730de9da108bd1f6c30d0 Gitweb: https://git.kernel.org/tip/5fb52dd93a2fe9a738f730de9da108bd1f6c30d0 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 24 Feb 2020 09:52:21 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 24 Feb 2020 11:36:39 +01:00
sched/numa: Find an alternative idle CPU if the CPU is part of an active NUMA balance
Multiple tasks can attempt to select and idle CPU but fail because numa_migrate_on is already set and the migration fails. Instead of failing, scan for an alternative idle CPU. select_idle_sibling is not used because it requires IRQs to be disabled and it ignores numa_migrate_on allowing multiple tasks to stack. This scan may still fail if there are idle candidate CPUs due to races but if this occurs, it's best that a task stay on an available CPU that move to a contended one.
--- kernel/sched/fair.c | 40 ++++++++++++++++++++++------------------ 1 file changed, 22 insertions(+), 18 deletions(-)
kernel/sched/fair.c +22 -18
@@ -1624,15 +1624,34 @@ static void task_numa_assign(struct task_numa_env *env, {struct rq *rq =cpu_rq (env->dst_cpu); -/* Bail out if run-queue part of active NUMA balance. */ -if (env->best_cpu != env->dst_cpu &&xchg (&rq->numa_migrate_on,1 )) +/* Check if run-queue part of active NUMA balance. */ +if (env->best_cpu != env->dst_cpu &&xchg (&rq->numa_migrate_on,1 )) { +int cpu; +int start = env->dst_cpu; + +/* Find alternative idle CPU. */ +for_each_cpu_wrap (cpu,cpumask_of_node (env->dst_nid), start) { +if (cpu == env->best_cpu || !idle_cpu (cpu) || + !cpumask_test_cpu (cpu, env->p->cpus_ptr)) { +continue ; + } + + env->dst_cpu = cpu; + rq =cpu_rq (env->dst_cpu); +if (!xchg (&rq->numa_migrate_on,1 )) + goto assign; + } + +/* Failed to find an alternative idle CPU */ return ; + } +assign:/* * Clear previous best_cpu/rq numa-migrate flag, since task now * found a better CPU to move/swap. */ -if (env->best_cpu !=-1 ) { +if (env->best_cpu !=-1 && env->best_cpu != env->dst_cpu) { rq =cpu_rq (env->best_cpu);WRITE_ONCE (rq->numa_migrate_on,0 ); }
@@ -1806,21 +1825,6 @@ assign: cpu = env->best_cpu; } -/* -* Use select_idle_sibling if the previously found idle CPU is -* not idle any more. -*/ -if (!idle_cpu (cpu)) { -/* -* select_idle_siblings() uses an per-CPU cpumask that -* can be used from IRQ context. -*/ -local_irq_disable (); - cpu =select_idle_sibling (env->p, env->src_cpu, - env->dst_cpu); -local_irq_enable (); - } - env->dst_cpu = cpu; }
PATCH 12/13 sched/numa: Bias swapping tasks based on their preferred node
In reply to Mel GormanWhen swapping tasks for NUMA balancing, it is preferred that tasks move to or remain on their preferred node. When considering an imbalance, encourage tasks to move to their preferred node and discourage tasks from moving away from their preferred node.
--- kernel/sched/fair.c | 43 +++++++++++++++++++++++++++++++++++++------ 1 file changed, 37 insertions(+), 6 deletions(-)
kernel/sched/fair.c +37 -6
@@ -1741,18 +1741,27 @@ static void task_numa_compare(struct task_numa_env *env, goto unlock; } +/* Skip this swap candidate if cannot move to the source cpu. */ +if (!cpumask_test_cpu (env->src_cpu, cur->cpus_ptr)) + goto unlock; + +/* +* Skip this swap candidate if it is not moving to its preferred +* node and the best task is. +*/ +if (env->best_task && + env->best_task->numa_preferred_nid == env->src_nid && + cur->numa_preferred_nid != env->src_nid) { + goto unlock; + } +/* * "imp" is the fault differential for the source task between the * source and destination node. Calculate the total differential for * the source task and potential destination task. The more negative * the value is, the more remote accesses that would be expected to * be incurred if the tasks were swapped. -*/ -/* Skip this swap candidate if cannot move to the source cpu */ -if (!cpumask_test_cpu (env->src_cpu, cur->cpus_ptr)) - goto unlock; - -/* +* * If dst and source tasks are in the same NUMA group, or not * in any group then look only at task weights. */
@@ -1779,12 +1788,34 @@ static void task_numa_compare(struct task_numa_env *env,task_weight (cur, env->dst_nid, dist); } +/* Discourage picking a task already on its preferred node */ +if (cur->numa_preferred_nid == env->dst_nid) + imp -= imp /16 ; + +/* +* Encourage picking a task that moves to its preferred node. +* This potentially makes imp larger than it's maximum of +* 1998 (see SMALLIMP and task_weight for why) but in this +* case, it does not matter. +*/ +if (cur->numa_preferred_nid == env->src_nid) + imp += imp /8 ; +if (maymove && moveimp > imp && moveimp > env->best_imp) { imp = moveimp; cur =NULL ; goto assign; } +/* +* Prefer swapping with a task moving to its preferred node over a +* task that is not. +*/ +if (env->best_task && cur->numa_preferred_nid == env->src_nid && + env->best_task->numa_preferred_nid != env->src_nid) { + goto assign; + } +/* * If the NUMA importance is less than SMALLIMP, * task migration might only result in ping pong
Signature · 1 line
2.16.4
[tip: sched/core] sched/numa: Bias swapping tasks based on their preferred node
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 88cca72c9673e631b63eca7a1dba4a9722a3f414 Gitweb: https://git.kernel.org/tip/88cca72c9673e631b63eca7a1dba4a9722a3f414 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 24 Feb 2020 09:52:22 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 24 Feb 2020 11:36:39 +01:00
sched/numa: Bias swapping tasks based on their preferred node
When swapping tasks for NUMA balancing, it is preferred that tasks move to or remain on their preferred node. When considering an imbalance, encourage tasks to move to their preferred node and discourage tasks from moving away from their preferred node.
--- kernel/sched/fair.c | 43 +++++++++++++++++++++++++++++++++++++------ 1 file changed, 37 insertions(+), 6 deletions(-)
kernel/sched/fair.c +37 -6
@@ -1741,18 +1741,27 @@ static void task_numa_compare(struct task_numa_env *env, goto unlock; } +/* Skip this swap candidate if cannot move to the source cpu. */ +if (!cpumask_test_cpu (env->src_cpu, cur->cpus_ptr)) + goto unlock; + +/* +* Skip this swap candidate if it is not moving to its preferred +* node and the best task is. +*/ +if (env->best_task && + env->best_task->numa_preferred_nid == env->src_nid && + cur->numa_preferred_nid != env->src_nid) { + goto unlock; + } +/* * "imp" is the fault differential for the source task between the * source and destination node. Calculate the total differential for * the source task and potential destination task. The more negative * the value is, the more remote accesses that would be expected to * be incurred if the tasks were swapped. -*/ -/* Skip this swap candidate if cannot move to the source cpu */ -if (!cpumask_test_cpu (env->src_cpu, cur->cpus_ptr)) - goto unlock; - -/* +* * If dst and source tasks are in the same NUMA group, or not * in any group then look only at task weights. */
@@ -1779,6 +1788,19 @@ static void task_numa_compare(struct task_numa_env *env,task_weight (cur, env->dst_nid, dist); } +/* Discourage picking a task already on its preferred node */ +if (cur->numa_preferred_nid == env->dst_nid) + imp -= imp /16 ; + +/* +* Encourage picking a task that moves to its preferred node. +* This potentially makes imp larger than it's maximum of +* 1998 (see SMALLIMP and task_weight for why) but in this +* case, it does not matter. +*/ +if (cur->numa_preferred_nid == env->src_nid) + imp += imp /8 ; +if (maymove && moveimp > imp && moveimp > env->best_imp) { imp = moveimp; cur =NULL ;
@@ -1786,6 +1808,15 @@ static void task_numa_compare(struct task_numa_env *env, }/* +* Prefer swapping with a task moving to its preferred node over a +* task that is not. +*/ +if (env->best_task && cur->numa_preferred_nid == env->src_nid && + env->best_task->numa_preferred_nid != env->src_nid) { + goto assign; + } + +/* * If the NUMA importance is less than SMALLIMP, * task migration might only result in ping pong * of tasks and also hurt performance due to cache
PATCH 13/13 sched/numa: Stop an exhastive search if a reasonable swap candidate or idle CPU is found
In reply to Mel GormanWhen domains are imbalanced or overloaded a search of all CPUs on the target domain is searched and compared with task_numa_compare. In some circumstances, a candidate is found that is an obvious win.
o A task can move to an idle CPU and an idle CPU is found
o A swap candidate is found that would move to its preferred domain
This patch terminates the search when either condition is met.
--- kernel/sched/fair.c | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-)
kernel/sched/fair.c +27 -4
@@ -1707,7 +1707,7 @@ static bool load_too_imbalanced(long src_load, long dst_load,* into account that it might be best if task running on the dst_cpu should * be exchanged with the source task */ -static void task_numa_compare (struct task_numa_env *env, +static bool task_numa_compare (struct task_numa_env *env,long taskimp,long groupimp,bool maymove) {struct numa_group *cur_ng, *p_ng =deref_curr_numa_group (env->p);
@@ -1718,9 +1718,10 @@ static void task_numa_compare(struct task_numa_env *env,int dist = env->dist;long moveimp = imp;long load; +bool stopsearch = false;if (READ_ONCE (dst_rq->numa_migrate_on)) -return ; +return false;rcu_read_lock (); cur =rcu_dereference (dst_rq->curr);
@@ -1731,8 +1732,10 @@ static void task_numa_compare(struct task_numa_env *env,* Because we have preemption enabled we can get migrated around and * end try selecting ourselves (current == env->p) as a swap candidate. */ -if (cur == env->p) +if (cur == env->p) { + stopsearch = true; goto unlock; + }if (!cur) {if (maymove && moveimp >= env->best_imp)
@@ -1860,8 +1863,27 @@ static void task_numa_compare(struct task_numa_env *env, }task_numa_assign (env, cur, imp); + +/* +* If a move to idle is allowed because there is capacity or load +* balance improves then stop the search. While a better swap +* candidate may exist, a search is not free. +*/ +if (maymove && !cur && env->best_cpu >=0 &&idle_cpu (env->best_cpu)) + stopsearch = true; + +/* +* If a swap candidate must be identified and the current best task +* moves its preferred node then stop the search. +*/ +if (!maymove && env->best_task && + env->best_task->numa_preferred_nid == env->src_nid) { + stopsearch = true; + } unlock:rcu_read_unlock (); + +return stopsearch; } static void task_numa_find_cpu(struct task_numa_env *env,
@@ -1916,7 +1938,8 @@ static void task_numa_find_cpu(struct task_numa_env *env,continue ; env->dst_cpu = cpu; -task_numa_compare (env, taskimp, groupimp, maymove); +if (task_numa_compare (env, taskimp, groupimp, maymove)) +break ; } }
Signature · 1 line
2.16.4
[tip: sched/core] sched/numa: Stop an exhastive search if a reasonable swap candidate or idle CPU is found
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: a0f03b617c3b2644d3d47bf7d9e60aed01bd5b10 Gitweb: https://git.kernel.org/tip/a0f03b617c3b2644d3d47bf7d9e60aed01bd5b10 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 24 Feb 2020 09:52:23 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 24 Feb 2020 11:36:40 +01:00
sched/numa: Stop an exhastive search if a reasonable swap candidate or idle CPU is found
When domains are imbalanced or overloaded a search of all CPUs on the target domain is searched and compared with task_numa_compare. In some circumstances, a candidate is found that is an obvious win.
o A task can move to an idle CPU and an idle CPU is found
o A swap candidate is found that would move to its preferred domain
This patch terminates the search when either condition is met.
--- kernel/sched/fair.c | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-)
kernel/sched/fair.c +27 -4
@@ -1707,7 +1707,7 @@ static bool load_too_imbalanced(long src_load, long dst_load,* into account that it might be best if task running on the dst_cpu should * be exchanged with the source task */ -static void task_numa_compare (struct task_numa_env *env, +static bool task_numa_compare (struct task_numa_env *env,long taskimp,long groupimp,bool maymove) {struct numa_group *cur_ng, *p_ng =deref_curr_numa_group (env->p);
@@ -1718,9 +1718,10 @@ static void task_numa_compare(struct task_numa_env *env,int dist = env->dist;long moveimp = imp;long load; +bool stopsearch = false;if (READ_ONCE (dst_rq->numa_migrate_on)) -return ; +return false;rcu_read_lock (); cur =rcu_dereference (dst_rq->curr);
@@ -1731,8 +1732,10 @@ static void task_numa_compare(struct task_numa_env *env,* Because we have preemption enabled we can get migrated around and * end try selecting ourselves (current == env->p) as a swap candidate. */ -if (cur == env->p) +if (cur == env->p) { + stopsearch = true; goto unlock; + }if (!cur) {if (maymove && moveimp >= env->best_imp)
@@ -1860,8 +1863,27 @@ assign: }task_numa_assign (env, cur, imp); + +/* +* If a move to idle is allowed because there is capacity or load +* balance improves then stop the search. While a better swap +* candidate may exist, a search is not free. +*/ +if (maymove && !cur && env->best_cpu >=0 &&idle_cpu (env->best_cpu)) + stopsearch = true; + +/* +* If a swap candidate must be identified and the current best task +* moves its preferred node then stop the search. +*/ +if (!maymove && env->best_task && + env->best_task->numa_preferred_nid == env->src_nid) { + stopsearch = true; + } unlock:rcu_read_unlock (); + +return stopsearch; } static void task_numa_find_cpu(struct task_numa_env *env,
@@ -1916,7 +1938,8 @@ static void task_numa_find_cpu(struct task_numa_env *env,continue ; env->dst_cpu = cpu; -task_numa_compare (env, taskimp, groupimp, maymove); +if (task_numa_compare (env, taskimp, groupimp, maymove)) +break ; } }
Ingo Molnar
AppliedQuoted 10 lines · Mel Gorman
The only differences in V6 are due to Vincent's latest patch series.
This is V5 which includes the latest versions of Vincent's patch addressing review feedback. Patches 4-9 are Vincent's work plus one important performance fix. Vincent's patches were retested and while not presented in detail, it was mostly an improvement.
Changelog since V5:
o Import Vincent's latest patch set
Quoted 8 lines
include/linux/sched.h | 31 ++- include/trace/events/sched.h | 49 ++-- kernel/sched/core.c | 13 - kernel/sched/debug.c | 17 +- kernel/sched/fair.c | 626 ++++++++++++++++++++++++++++--------------- kernel/sched/pelt.c | 59 ++-- kernel/sched/sched.h | 42 ++- 7 files changed, 535 insertions(+), 302 deletions(-)
Applied to tip:sched/core for v5.7 inclusion, thanks Mel and Vincent!
Please base future iterations on top of a0f03b617c3b (current sched/core).
Thanks!
Ingo
Quoted 22 lines · Ingo Molnar
Quoted 10 lines · Mel Gorman
The only differences in V6 are due to Vincent's latest patch series.
This is V5 which includes the latest versions of Vincent's patch addressing review feedback. Patches 4-9 are Vincent's work plus one important performance fix. Vincent's patches were retested and while not presented in detail, it was mostly an improvement.
Changelog since V5:
o Import Vincent's latest patch setQuoted 8 lines
include/linux/sched.h | 31 ++- include/trace/events/sched.h | 49 ++-- kernel/sched/core.c | 13 - kernel/sched/debug.c | 17 +- kernel/sched/fair.c | 626 ++++++++++++++++++++++++++++--------------- kernel/sched/pelt.c | 59 ++-- kernel/sched/sched.h | 42 ++- 7 files changed, 535 insertions(+), 302 deletions(-)
Quoted 2 lines
Applied to tip:sched/core for v5.7 inclusion, thanks Mel and Vincent!
Thanks!
Quoted 3 lines
Please base future iterations on top of
a0f03b617c3b(current sched/core).
Will do.
However I noticed that "sched/fair: Fix find_idlest_group() to handle CPU affinity" did not make it to tip/sched/core. Peter seemed to think it was fine. Was it rejected or is it just sitting in Peter's queue somewhere?
Signature · 2 lines
Mel Gorman SUSE Labs
Vincent Guittot
maintainerQuoted 33 lines · Mel Gorman
Quoted 22 lines · Ingo Molnar
Quoted 10 lines · Mel Gorman
The only differences in V6 are due to Vincent's latest patch series.
This is V5 which includes the latest versions of Vincent's patch addressing review feedback. Patches 4-9 are Vincent's work plus one important performance fix. Vincent's patches were retested and while not presented in detail, it was mostly an improvement.
Changelog since V5:
o Import Vincent's latest patch setQuoted 8 lines
include/linux/sched.h | 31 ++- include/trace/events/sched.h | 49 ++-- kernel/sched/core.c | 13 - kernel/sched/debug.c | 17 +- kernel/sched/fair.c | 626 ++++++++++++++++++++++++++++--------------- kernel/sched/pelt.c | 59 ++-- kernel/sched/sched.h | 42 ++- 7 files changed, 535 insertions(+), 302 deletions(-)Quoted 2 lines
Applied to tip:sched/core for v5.7 inclusion, thanks Mel and Vincent!
Thanks!
Quoted 3 lines
Please base future iterations on top of
a0f03b617c3b(current sched/core).
Quoted 5 lines
Will do.
However I noticed that "sched/fair: Fix
find_idlest_group()to handle CPU affinity" did not make it to tip/sched/core. Peter seemed to think it was fine. Was it rejected or is it just sitting in Peter's queue somewhere?
The patch has already reached mainline through tip/sched-urgent-for-linus
Quoted 4 lines
--
Mel Gorman
SUSE Labs
Quoted 9 lines · Vincent Guittot
Quoted 6 lines
Will do.
However I noticed that "sched/fair: Fix
find_idlest_group()to handle CPU affinity" did not make it to tip/sched/core. Peter seemed to think it was fine. Was it rejected or is it just sitting in Peter's queue somewhere?The patch has already reached mainline through tip/sched-urgent-for-linus
Bah, I pasted the wrong subject. I am thinking of your patch "sched/fair: fix statistics for find_idlest_group" --
https://lore.kernel.org/lkml/[email protected]/
It still appears to be relevant or did I miss something else?
Signature · 2 lines
Mel Gorman SUSE Labs
Vincent Guittot
maintainerQuoted 9 lines · Mel Gorman
Quoted 7 lines · Vincent Guittot
Quoted 6 lines
Will do.
However I noticed that "sched/fair: Fix
find_idlest_group()to handle CPU affinity" did not make it to tip/sched/core. Peter seemed to think it was fine. Was it rejected or is it just sitting in Peter's queue somewhere?
Quoted 7 lines
Quoted 2 lines
The patch has already reached mainline through tip/sched-urgent-for-linus
Bah, I pasted the wrong subject. I am thinking of your patch "sched/fair: fix statistics for
find_idlest_group" --
https://lore.kernel.org/lkml/[email protected]/
It still appears to be relevant or did I miss something else?
you're right, it's not yet in tip/sched/core but most probably in Peter's queue
Quoted 4 lines
--
Mel Gorman
SUSE Labs
Ingo Molnar
AppliedQuoted 8 lines · Mel Gorman
Quoted 7 lines · Vincent Guittot
Quoted 6 lines
Will do.
However I noticed that "sched/fair: Fix
find_idlest_group()to handle CPU affinity" did not make it to tip/sched/core. Peter seemed to think it was fine. Was it rejected or is it just sitting in Peter's queue somewhere?
Quoted 7 lines
Quoted 2 lines
The patch has already reached mainline through tip/sched-urgent-for-linus
Bah, I pasted the wrong subject. I am thinking of your patch "sched/fair: fix statistics for
find_idlest_group" --
https://lore.kernel.org/lkml/[email protected]/
It still appears to be relevant or did I miss something else?
Applied to tip:sched/urgent now, thanks guys!
Ingo
Hi Mel,
Quoted 213 lines · On Mon, Feb 24, 2020 at 09:52:10AM +0000 Mel Gorman wrote:
The only differences in V6 are due to Vincent's latest patch series.
This is V5 which includes the latest versions of Vincent's patch addressing review feedback. Patches 4-9 are Vincent's work plus one important performance fix. Vincent's patches were retested and while not presented in detail, it was mostly an improvement.
Changelog since V5:
o Import Vincent's latest patch setChangelog since V4:
o The V4 send was the completely wrong versions of the patches and is useless.Changelog since V3:
o Remove stray tab (Valentin)
o Update comment about allowing a move when src is imbalanced (Hillf)
o Updated "sched/pelt: Add a new runnable average signal" (Vincent)Changelog since V2:
o Rebase on top of Vincent's series again
o Fix a missedrcu_read_unlock
o Reduce overhead of tracepointChangelog since V1:
o Rebase on top of Vincent's series and reworkNote: The baseline for this series is tip/sched/core as of February
12th rebased on top of v5.6-rc1. The series includes patches from Vincent as I needed to add a fix and build on top of it. Vincent's series on its own introduces performance regressions for *some* but not *all* machines so it's easily missed. This series overall is close to performance-neutral with some gains depending on the machine. However, the end result does less work on NUMA balancing and the fact that both the NUMA balancer and load balancer uses similar logic makes it much easier to understand.The NUMA balancer makes placement decisions on tasks that partially take the load balancer into account and vice versa but there are inconsistencies. This can result in placement decisions that override each other leading to unnecessary migrations -- both task placement and page placement. This series reconciles many of the decisions -- partially Vincent's work with some fixes and optimisations on top to merge our two series.
The first patch is unrelated. It's picked up by tip but was not present in the tree at the time of the fork. I'm including it here because I tested with it.
The second and third patches are tracing only and was needed to get sensible data out of ftrace with respect to task placement for NUMA balancing. The NUMA balancer is *far* easier to analyse with the patches and informed how the series should be developed.
Patches 4-5 are Vincent's and use very similar code patterns and logic between NUMA and load balancer. Patch 6 is a fix to Vincent's work that is necessary to avoid serious imbalances being introduced by the NUMA balancer. Patches 7-9 are also Vincents and while I have not reviewed them closely myself, others have.
The rest of the series are a mix of optimisations and improvements, one of which stops the NUMA balancer fighting with itself.
Note that this is not necessarily a universal performance win although performance results are generally ok (small gains/losses depending on the machine and workload). However, task migrations, page migrations, variability and overall overhead are generally reduced.
The main reference workload I used was specjbb running one JVM per node which typically would be expected to split evenly. It's an interesting workload because the number of "warehouses" does not linearly related to the number of running tasks due to the creation of GC threads and other interfering activity. The mmtests configuration used is jvm-specjbb2005-multi with two runs -- one with ftrace enabling relevant scheduler tracepoints.
An example of the headline performance of the series is below and the tested kernels are
baseline-v3r1 Patches 1-3 for the tracing
loadavg-v3 Patches 1-5 (Add half of Vincent's work)
lbidle-v6 Patches 1-6 Vincent's work with a fix on top
classify-v6 Patches 1-9 Rest of Vincent's work
stopsearch-v6 All patches5.6.0-rc1 5.6.0-rc1 5.6.0-rc1 5.6.0-rc1 5.6.0-rc1
baseline-v3 loadavg-v3 lbidle-v3 classify-v6 stopsearch-v6Hmean tput-1 43593.49 ( 0.00%) 41616.85 ( -4.53%) 43657.25 ( 0.15%) 38110.46 * -12.58%* 42213.29 ( -3.17%) Hmean tput-2 95692.84 ( 0.00%) 93196.89 * -2.61%* 92287.78 * -3.56%* 89077.29 ( -6.91%) 96474.49 * 0.82%* Hmean tput-3 143813.12 ( 0.00%) 134447.05 * -6.51%* 134587.84 * -6.41%* 133706.98 ( -7.03%) 144279.90 ( 0.32%) Hmean tput-4 190702.67 ( 0.00%) 176533.79 * -7.43%* 182278.42 * -4.42%* 181405.19 ( -4.88%) 189948.10 ( -0.40%) Hmean tput-5 230242.39 ( 0.00%) 209059.51 * -9.20%* 223219.06 ( -3.05%) 227188.16 ( -1.33%) 225220.39 ( -2.18%) Hmean tput-6 274868.74 ( 0.00%) 246470.42 * -10.33%* 258387.09 * -6.00%* 264252.76 ( -3.86%) 271429.49 ( -1.25%) Hmean tput-7 312281.15 ( 0.00%) 284564.06 * -8.88%* 296446.00 * -5.07%* 302682.72 ( -3.07%) 309187.26 ( -0.99%) Hmean tput-8 347261.31 ( 0.00%) 332019.39 * -4.39%* 331202.25 * -4.62%* 339469.52 ( -2.24%) 345504.60 ( -0.51%) Hmean tput-9 387336.25 ( 0.00%) 352219.62 * -9.07%* 370222.03 * -4.42%* 367077.01 ( -5.23%) 381610.17 ( -1.48%) Hmean tput-10 421586.76 ( 0.00%) 397304.22 ( -5.76%) 405458.01 ( -3.83%) 416689.66 ( -1.16%) 415549.97 ( -1.43%) Hmean tput-11 459422.43 ( 0.00%) 398023.51 * -13.36%* 441999.08 ( -3.79%) 449912.39 ( -2.07%) 454458.04 ( -1.08%) Hmean tput-12 499087.97 ( 0.00%) 400914.35 * -19.67%* 475755.59 ( -4.68%) 493678.32 ( -1.08%) 493936.79 ( -1.03%) Hmean tput-13 536335.59 ( 0.00%) 406101.41 * -24.28%* 514858.97 ( -4.00%) 528496.01 ( -1.46%) 530662.68 ( -1.06%) Hmean tput-14 571542.75 ( 0.00%) 478797.13 * -16.23%* 551716.00 ( -3.47%) 553771.29 ( -3.11%) 565915.55 ( -0.98%) Hmean tput-15 601412.81 ( 0.00%) 534776.98 * -11.08%* 580105.28 ( -3.54%) 597513.89 ( -0.65%) 596192.34 ( -0.87%) Hmean tput-16 629817.55 ( 0.00%) 407294.29 * -35.33%* 615606.40 ( -2.26%) 630044.12 ( 0.04%) 627806.13 ( -0.32%) Hmean tput-17 667025.18 ( 0.00%) 457416.34 * -31.42%* 626074.81 ( -6.14%) 659706.41 ( -1.10%) 658350.40 ( -1.30%) Hmean tput-18 688148.21 ( 0.00%) 518534.45 * -24.65%* 663161.87 ( -3.63%) 675616.08 ( -1.82%) 682224.35 ( -0.86%) Hmean tput-19 705092.87 ( 0.00%) 466530.37 * -33.83%* 689430.29 ( -2.22%) 691050.89 ( -1.99%) 705532.41 ( 0.06%) Hmean tput-20 711481.44 ( 0.00%) 564355.80 * -20.68%* 692170.67 ( -2.71%) 717866.36 ( 0.90%) 716243.50 ( 0.67%) Hmean tput-21 739790.92 ( 0.00%) 508542.10 * -31.26%* 712348.91 ( -3.71%) 724666.68 ( -2.04%) 723361.87 ( -2.22%) Hmean tput-22 730593.57 ( 0.00%) 540881.37 ( -25.97%) 709794.02 ( -2.85%) 727177.54 ( -0.47%) 721353.36 ( -1.26%) Hmean tput-23 738401.59 ( 0.00%) 561474.46 * -23.96%* 702869.93 ( -4.81%) 720954.73 ( -2.36%) 720813.53 ( -2.38%) Hmean tput-24 731301.95 ( 0.00%) 582929.73 * -20.29%* 704337.59 ( -3.69%) 717204.03 * -1.93%* 714131.38 * -2.35%* Hmean tput-25 734414.40 ( 0.00%) 591635.13 ( -19.44%) 702334.30 ( -4.37%) 720272.39 ( -1.93%) 714245.12 ( -2.75%) Hmean tput-26 724774.17 ( 0.00%) 701310.59 ( -3.24%) 700771.85 ( -3.31%) 718084.92 ( -0.92%) 712988.02 ( -1.63%) Hmean tput-27 713484.55 ( 0.00%) 632795.43 ( -11.31%) 692213.36 ( -2.98%) 710432.96 ( -0.43%) 703087.86 ( -1.46%) Hmean tput-28 723111.86 ( 0.00%) 697438.61 ( -3.55%) 695934.68 ( -3.76%) 708413.26 ( -2.03%) 703449.60 ( -2.72%) Hmean tput-29 714690.69 ( 0.00%) 675820.16 ( -5.44%) 689400.90 ( -3.54%) 698436.85 ( -2.27%) 699981.24 ( -2.06%) Hmean tput-30 711106.03 ( 0.00%) 699748.68 ( -1.60%) 688439.96 ( -3.19%) 698258.70 ( -1.81%) 691636.96 ( -2.74%) Hmean tput-31 701632.39 ( 0.00%) 698807.56 ( -0.40%) 682588.20 ( -2.71%) 696608.99 ( -0.72%) 691015.36 ( -1.51%) Hmean tput-32 703479.77 ( 0.00%) 679020.34 ( -3.48%) 674057.11 * -4.18%* 690706.86 ( -1.82%) 684958.62 ( -2.63%) Hmean tput-33 691594.71 ( 0.00%) 686583.04 ( -0.72%) 673382.64 ( -2.63%) 687319.97 ( -0.62%) 683367.65 ( -1.19%) Hmean tput-34 693435.51 ( 0.00%) 685137.16 ( -1.20%) 674883.97 ( -2.68%) 684897.97 ( -1.23%) 674923.39 ( -2.67%) Hmean tput-35 688036.06 ( 0.00%) 682612.92 ( -0.79%) 668159.93 ( -2.89%) 679301.53 ( -1.27%) 678117.69 ( -1.44%) Hmean tput-36 678957.95 ( 0.00%) 670160.33 ( -1.30%) 662395.36 ( -2.44%) 672165.17 ( -1.00%) 668512.57 ( -1.54%) Hmean tput-37 679748.70 ( 0.00%) 675428.41 ( -0.64%) 666970.33 ( -1.88%) 674127.70 ( -0.83%) 667644.78 ( -1.78%) Hmean tput-38 669969.62 ( 0.00%) 670976.06 ( 0.15%) 660499.74 ( -1.41%) 670848.38 ( 0.13%) 666646.89 ( -0.50%) Hmean tput-39 669291.41 ( 0.00%) 665367.66 ( -0.59%) 649337.71 ( -2.98%) 659685.61 ( -1.44%) 658818.08 ( -1.56%) Hmean tput-40 668074.80 ( 0.00%) 672478.06 ( 0.66%) 661273.87 ( -1.02%) 665147.36 ( -0.44%) 660279.43 ( -1.17%)Note the regression with the first two patches of Vincent's work (loadavg-v3) followed by lbidle-v3 which mostly restores the performance and the final version keeping things close to performance neutral (showing a mix but within noise). This is not universal as a different 2-socket machine with fewer cores and older CPUs showed no difference. EPYC 1 and EPYC 2 were both affected by the regression as well as a 4-socket Intel box but again, the full series is mostly performance neutral for specjbb but with less NUMA balancing work.
While not presented here, the full series also shows that the throughput measured by each JVM is less variable.
The high-level NUMA stats from
/proc/vmstatlook like this5.6.0-rc1 5.6.0-rc1 5.6.0-rc1 5.6.0-rc1 5.6.0-rc1 baseline-v3 loadavg-v3 lbidle-v3 classify-v3 stopsearch-v3 Ops NUMA alloc hit 878062.00 882981.00 957762.00 961630.00 880821.00 Ops NUMA alloc miss 0.00 0.00 0.00 0.00 0.00 Ops NUMA interleave hit 225582.00 237785.00 242554.00 234671.00 234818.00 Ops NUMA alloc local 764932.00 763850.00 835939.00 843950.00 763005.00 Ops NUMA base-page range updates 2517600.00 3707398.00 2889034.00 2442203.00 3303790.00 Ops NUMA PTE updates 1754720.00 1672198.00 1569610.00 1356763.00 1591662.00 Ops NUMA PMD updates 1490.00 3975.00 2577.00 2120.00 3344.00 Ops NUMA hint faults 1678620.00 1586860.00 1475303.00 1285152.00 1512208.00 Ops NUMA hint local faults % 1461203.00 1389234.00 1181790.00 1085517.00 1411194.00Ops NUMA hint local percent 87.05 87.55 80.10 84.47 93.32
Ops NUMA pages migrated 69473.00 62504.00 121893.00 80802.00 46266.00
Ops AutoNUMA cost 8412.04 7961.44 7399.05 6444.39 7585.05Overall, the local hints percentage is slightly better but crucially, it's done with much less page migrations.
A separate run gathered information from ftrace and analysed it offline. This is based on an earlier version of the series but the changes are not significant enough to warrant a rerun as there are no changes in the NUMA balancing optimisations.
5.6.0-rc1 5.6.0-rc1 baseline-v2 stopsearch-v2Ops Migrate failed no CPU 1871.00 689.00
Ops Migrate failed move to idle 0.00 0.00
Ops Migrate failed swap task fail 872.00 568.00
Ops Task Migrated swapped 6702.00 3344.00
Ops Task Migrated swapped local NID 0.00 0.00
Ops Task Migrated swapped within group 1094.00 124.00
Ops Task Migrated idle CPU 14409.00 14610.00
Ops Task Migrated idle CPU local NID 0.00 0.00
Ops Task Migrate retry 2355.00 1074.00
Ops Task Migrate retry success 0.00 0.00
Ops Task Migrate retry failed 2355.00 1074.00
Ops Load Balance cross NUMA 1248401.00 1261853.00"Migrate failed no CPU" is the times when NUMA balancing did not find a suitable page on a preferred node. This is increased because the series avoids making decisions that the LB would override.
"Migrate failed swap task fail" is when
migrate_swapfails and it can fail for a lot of reasons."Task Migrated swapped" is lower which would would be a concern but in this test, locality was higher unlike the test with tracing disabled. This event triggers when two tasks are swapped to keep load neutral or improved from the perspective of the load balancer. The series attempts to swap tasks that both move to their preferred node.
"Task Migrated idle CPU" is similar and while the the series does try to avoid NUMA Balancer and LB fighting each other, it also continues to obey overall CPU load balancer.
"Task Migrate retry failed" happens when NUMA balancing makes multiple attempts to place a task on a preferred node. It is slightly reduced here but it would generally be expected to happen to maintain CPU load balance.
A variety of other workloads were evaluated and appear to be mostly neutral or improved. netperf running on localhost shows gains between 1-8% depending on the machine. hackbench is a mixed bag -- small regressions on one machine around 1-2% depending on the group count but up to 15% gain on another machine. dbench looks generally ok, very small performance gains. pgbench looks ok, small gains and losses, much of which is within the noise. schbench (Facebook workload that is sensitive to wakeup latencies) is mostly good. The autonuma benchmark also generally looks good, most differences are within the noise but with higher locality success rates and fewer page migrations. Other long lived workloads are still running but I'm not expecting many surprises.
Quoted 12 lines
include/linux/sched.h | 31 ++- include/trace/events/sched.h | 49 ++-- kernel/sched/core.c | 13 - kernel/sched/debug.c | 17 +- kernel/sched/fair.c | 626 ++++++++++++++++++++++++++++--------------- kernel/sched/pelt.c | 59 ++-- kernel/sched/sched.h | 42 ++- 7 files changed, 535 insertions(+), 302 deletions(-)Signature · 1 line
2.16.4
Our Perf team was been comparing tip/sched/core (v5.6-rc3 + lb_numa series) with upstream v5.6-rc3 and has noticed some regressions.
Here's a summary of what Jirka Hladky reported to me:
---
We see following problems when comparing 5.6.0_rc3.tip_lb_numa+-5 against 5.6.0-0.rc3.1.elrdy:
• performance drop by 20% - 40% across almost all benchmarks especially for low and medium thread counts and especially on 4 NUMA and 8 NUMA nodes servers
• 2 NUMA nodes servers are affected as well, but performance drop is less
significant (10-20%)
• servers with just one NUMA node are NOT affected
• we see big load imbalances between different NUMA nodes
---
The actual data reports are on an intranet web page so they are harder to share. I can create PDFs or screenshots but I didn't want to just blast those to the list. I'd be happy to send some direclty if you are interested.
Some data in text format I can easily include shows imbalances across the numa nodes. This is for NAS sp.C.x benchmark because it was easiest to pull and see the data in text. The regressions can be seen in other tests as well.
For example:
5.6.0_rc3.tip_lb_numa+
sp.C.x_008_02 - CPU load average across the individual NUMA nodes (timestep is 5 seconds)
# NUMA | AVR | Utilization over time in percentage 0 | 5 | 12 9 3 0 0 11 8 0 1 3 5 17 9 5 0 0 0 11 3 1 | 16 | 20 21 10 10 2 6 9 12 11 9 9 23 24 23 24 24 24 19 20 2 | 21 | 19 23 26 22 22 23 25 20 25 34 38 17 13 13 13 13 13 27 13 3 | 15 | 19 23 20 21 21 15 15 20 20 18 10 10 9 9 9 9 9 9 11 4 | 19 | 13 14 15 22 23 20 19 20 17 12 15 15 25 25 24 24 24 14 24 5 | 3 | 0 2 11 6 20 8 0 0 0 0 0 0 0 0 0 0 0 0 9 6 | 0 | 0 0 0 5 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 7 | 4 | 0 0 0 0 0 0 4 11 9 0 0 0 0 5 12 12 12 3 0
5.6.0-0.rc3.1.elrdy
sp.C.x_008_01 - CPU load average across the individual NUMA nodes (timestep is 5 seconds)
# NUMA | AVR | Utilization over time in percentage 0 | 13 | 6 8 10 10 11 10 18 13 20 17 14 15 1 | 11 | 10 10 11 11 9 16 12 14 9 11 11 10 2 | 17 | 25 19 16 11 13 12 11 16 17 22 22 16 3 | 21 | 21 22 22 23 21 23 23 21 21 17 22 21 4 | 14 | 20 23 11 12 15 18 12 10 9 13 12 18 5 | 4 | 0 0 8 10 7 0 8 2 0 0 8 2 6 | 1 | 0 5 1 0 0 0 0 0 0 1 0 0 7 | 7 | 7 3 10 10 10 11 3 8 10 4 0 5
mops/s sp_C_x
kernel threads 8 16 32 48 56 64
5.6.0-0.rc3.1.elrdy mean 22819.8 39955.3 34301.6 31086.8 30316.2 30089.2
max 23911.4 47185.6 37995.9 33742.6 33048.0 30853.4 min 20799.3 36518.0 31459.4 27559.9 28565.7 29306.3 stdev 1096.7 3965.3 2350.2 2134.7 1871.1 612.1 first_q 22780.4 37263.7 32081.7 29955.8 28609.8 29632.0 median 22936.7 37577.6 34866.0 32020.8 29299.9 29906.3 third_q 23671.0 41231.7 35105.1 32154.7 32057.8 30748.0 5.6.0_rc3.tip_lb_numa mean 12111.1 24712.6 30147.8 32560.7 31040.4 28219.4 max 17772.9 28934.4 33828.3 33659.3 32275.3 30434.9 min 9869.9 18407.9 25092.7 31512.9 29964.3 25652.8 stdev 2868.4 3673.6 2877.6 832.2 765.8 1800.6 first_q 10763.4 23346.1 29608.5 31827.2 30609.4 27008.8 median 10855.0 25415.4 30804.4 32462.1 31061.8 27992.6 third_q 11294.5 27459.2 31405.0 33341.8 31291.2 30007.9 Comparison mean -47 -38 -12 5 2 -6 median -53 -32 -12 1 6 -6
On 5.6.0-rc3.tip-lb_numa+ we see:
• BIG fluctuation in runtime
• NAS running up to 2x slower than on 5.6.0-0.rc3.1.elrdy
$ grep "Time in seconds" *log sp.C.x.defaultRun.008threads.loop01.log: Time in seconds = 125.73 sp.C.x.defaultRun.008threads.loop02.log: Time in seconds = 87.54 sp.C.x.defaultRun.008threads.loop03.log: Time in seconds = 86.93 sp.C.x.defaultRun.008threads.loop04.log: Time in seconds = 165.98 sp.C.x.defaultRun.008threads.loop05.log: Time in seconds = 114.78
On the other hand, runtime on 5.6.0-0.rc3.1.elrdy is stable:
$ grep "Time in seconds" *log sp.C.x.defaultRun.008threads.loop01.log: Time in seconds = 59.83 sp.C.x.defaultRun.008threads.loop02.log: Time in seconds = 67.72 sp.C.x.defaultRun.008threads.loop03.log: Time in seconds = 63.62 sp.C.x.defaultRun.008threads.loop04.log: Time in seconds = 55.01 sp.C.x.defaultRun.008threads.loop05.log: Time in seconds = 65.20
It looks like things are moving around a lot but not getting balanced as well across the numa nodes. I have a couple of nice heat maps that show this if you want to see them.
Thanks,
Phil
Quoted 34 lines · Phil Auld
Quoted 12 lines
A variety of other workloads were evaluated and appear to be mostly neutral or improved. netperf running on localhost shows gains between 1-8% depending on the machine. hackbench is a mixed bag -- small regressions on one machine around 1-2% depending on the group count but up to 15% gain on another machine. dbench looks generally ok, very small performance gains. pgbench looks ok, small gains and losses, much of which is within the noise. schbench (Facebook workload that is sensitive to wakeup latencies) is mostly good. The autonuma benchmark also generally looks good, most differences are within the noise but with higher locality success rates and fewer page migrations. Other long lived workloads are still running but I'm not expecting many surprises.
Quoted 12 lines
include/linux/sched.h | 31 ++- include/trace/events/sched.h | 49 ++-- kernel/sched/core.c | 13 - kernel/sched/debug.c | 17 +- kernel/sched/fair.c | 626 ++++++++++++++++++++++++++++--------------- kernel/sched/pelt.c | 59 ++-- kernel/sched/sched.h | 42 ++- 7 files changed, 535 insertions(+), 302 deletions(-)Signature · 1 line
2.16.4Our Perf team was been comparing tip/sched/core (v5.6-rc3 +
lb_numaseries) with upstream v5.6-rc3 and has noticed some regressions.Here's a summary of what Jirka Hladky reported to me:
---
We see following problems when comparing 5.6.0_rc3.tip_lb_numa+-5 against 5.6.0-0.rc3.1.elrdy:
Quoted 9 lines
??? performance drop by 20% - 40% across almost all benchmarks especially
for low and medium thread counts and especially on 4 NUMA and 8 NUMA nodes servers
??? 2 NUMA nodes servers are affected as well, but performance drop is lesssignificant (10-20%)??? servers with just one NUMA node are NOT affected ??? we see big load imbalances between different NUMA nodes
---
UMA being unaffected is not a surprise, the rest is obviously.
Quoted 4 lines
The actual data reports are on an intranet web page so they are harder to share. I can create PDFs or screenshots but I didn't want to just blast those to the list. I'd be happy to send some direclty if you are interested.
Send them to me privately please.
Quoted 5 lines
Some data in text format I can easily include shows imbalances across the numa nodes. This is for NAS sp.C.x benchmark because it was easiest to pull and see the data in text. The regressions can be seen in other tests as well.
What was the value for x?
I ask because I ran NAS across a variety of machines for C class in two configurations -- one using as many CPUs as possible and one running with a third of the available CPUs for both MPI and OMP. Generally there were small gains and losses across multiple kernels but often within the noise or within a few percent of each other.
The largest machine I had available was 4 sockets.
The other curiousity is that you used C class. On bigger machines, that is very short lived to the point of being almost useless. Is D class similarly affected?
Quoted 18 lines
For example:
5.6.0_rc3.tip_lb_numa+
sp.C.x_008_02 - CPU load average across the individual NUMA nodes (timestep is 5 seconds)# NUMA | AVR | Utilization over time in percentage 0 | 5 | 12 9 3 0 0 11 8 0 1 3 5 17 9 5 0 0 0 11 3 1 | 16 | 20 21 10 10 2 6 9 12 11 9 9 23 24 23 24 24 24 19 20 2 | 21 | 19 23 26 22 22 23 25 20 25 34 38 17 13 13 13 13 13 27 13 3 | 15 | 19 23 20 21 21 15 15 20 20 18 10 10 9 9 9 9 9 9 11 4 | 19 | 13 14 15 22 23 20 19 20 17 12 15 15 25 25 24 24 24 14 24 5 | 3 | 0 2 11 6 20 8 0 0 0 0 0 0 0 0 0 0 0 0 9 6 | 0 | 0 0 0 5 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 7 | 4 | 0 0 0 0 0 0 4 11 9 0 0 0 0 5 12 12 12 3 05.6.0-0.rc3.1.elrdy
sp.C.x_008_01 - CPU load average across the individual NUMA nodes (timestep is 5 seconds)
Quoted 10 lines
# NUMA | AVR | Utilization over time in percentage 0 | 13 | 6 8 10 10 11 10 18 13 20 17 14 15 1 | 11 | 10 10 11 11 9 16 12 14 9 11 11 10 2 | 17 | 25 19 16 11 13 12 11 16 17 22 22 16 3 | 21 | 21 22 22 23 21 23 23 21 21 17 22 21 4 | 14 | 20 23 11 12 15 18 12 10 9 13 12 18 5 | 4 | 0 0 8 10 7 0 8 2 0 0 8 2 6 | 1 | 0 5 1 0 0 0 0 0 0 1 0 0 7 | 7 | 7 3 10 10 10 11 3 8 10 4 0 5
A critical difference with the series is that large imbalances shouldn't happen but prior to the series the NUMA balancing would keep trying to move tasks to a node with load balancing moving them back. That should not happen any more but there are cases where it's actually faster to have the fight between NUMA balancing and load balancing. Ideally a degree of imbalance would be allowed but I haven't found a way of doing that without side effects.
Generally the utilisatiojn looks low in either kernel making me think the value for x is relatively small.
Quoted 4 lines
mops/s
sp_C_x
kernel threads 8 16 32 48 56 64
5.6.0-0.rc3.1.elrdy mean 22819.8 39955.3 34301.6 31086.8 30316.2 30089.2
Quoted 16 lines
max 23911.4 47185.6 37995.9 33742.6 33048.0 30853.4 min 20799.3 36518.0 31459.4 27559.9 28565.7 29306.3 stdev 1096.7 3965.3 2350.2 2134.7 1871.1 612.1 first_q 22780.4 37263.7 32081.7 29955.8 28609.8 29632.0 median 22936.7 37577.6 34866.0 32020.8 29299.9 29906.3 third_q 23671.0 41231.7 35105.1 32154.7 32057.8 30748.0 5.6.0_rc3.tip_lb_numa mean 12111.1 24712.6 30147.8 32560.7 31040.4 28219.4 max 17772.9 28934.4 33828.3 33659.3 32275.3 30434.9 min 9869.9 18407.9 25092.7 31512.9 29964.3 25652.8 stdev 2868.4 3673.6 2877.6 832.2 765.8 1800.6 first_q 10763.4 23346.1 29608.5 31827.2 30609.4 27008.8 median 10855.0 25415.4 30804.4 32462.1 31061.8 27992.6 third_q 11294.5 27459.2 31405.0 33341.8 31291.2 30007.9 Comparison mean -47 -38 -12 5 2 -6 median -53 -32 -12 1 6 -6
So I *think* this is observing the difference in imbalance. The range for 8 threads is massive but it stabilises when the thread count is higher. When fewer threads than a NUMA side is used, it can be beneficial to run everything one one node but the load balancer doesn't let that happen and the NUMA balancer no longer fights it.
Quoted 20 lines
On 5.6.0-rc3.tip-lb_numa+ we see:
??? BIG fluctuation in runtime
??? NAS running up to 2x slower than on 5.6.0-0.rc3.1.elrdy$ grep "Time in seconds" *log sp.C.x.defaultRun.008threads.loop01.log: Time in seconds = 125.73 sp.C.x.defaultRun.008threads.loop02.log: Time in seconds = 87.54 sp.C.x.defaultRun.008threads.loop03.log: Time in seconds = 86.93 sp.C.x.defaultRun.008threads.loop04.log: Time in seconds = 165.98 sp.C.x.defaultRun.008threads.loop05.log: Time in seconds = 114.78On the other hand, runtime on 5.6.0-0.rc3.1.elrdy is stable:
$ grep "Time in seconds" *log sp.C.x.defaultRun.008threads.loop01.log: Time in seconds = 59.83 sp.C.x.defaultRun.008threads.loop02.log: Time in seconds = 67.72 sp.C.x.defaultRun.008threads.loop03.log: Time in seconds = 63.62 sp.C.x.defaultRun.008threads.loop04.log: Time in seconds = 55.01 sp.C.x.defaultRun.008threads.loop05.log: Time in seconds = 65.20
Quoted 4 lines
It looks like things are moving around a lot but not getting balanced as well across the numa nodes. I have a couple of nice heat maps that show this if you want to see them.
I'd like to see the heatmap. I just looked at the ones I had for NAS and I'm not seeing a bad pattern with either all CPUs used or a third. What I'm looking for is a pattern showing higher utilisation on one node over another in the baseline kernel and showing relatively even utilisation in tip/sched/core.
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 13 lines · Mel Gorman
Quoted 4 lines
The actual data reports are on an intranet web page so they are harder to share. I can create PDFs or screenshots but I didn't want to just blast those to the list. I'd be happy to send some direclty if you are interested.
Send them to me privately please.
Quoted 5 lines
Some data in text format I can easily include shows imbalances across the numa nodes. This is for NAS sp.C.x benchmark because it was easiest to pull and see the data in text. The regressions can be seen in other tests as well.
Quoted 8 lines
What was the value for x?
I ask because I ran NAS across a variety of machines for C class in two configurations -- one using as many CPUs as possible and one running with a third of the available CPUs for both MPI and OMP. Generally there were small gains and losses across multiple kernels but often within the noise or within a few percent of each other.
On re-examining the case, this pattern matches. There are some corner cases for large machines that have low utilisation that are obvious. With the old behaviour, load balancing would even load evenly all available NUMA nodes while NUMA balancing would constantly adjust it for locality. The old load balancer does this even if a task starts with all of its memory local to one node.
The degree where it causes the most problems appears to be roughly for task counts lower than 2 * NR_NODES as per the small imbalance allowed by adjust_numa_imbalance but the actual distribution is variable. It's not always 2 per node, sometimes it can be a little higher depending on when idle balancing happens and other machine activity. This is not universal as other machine sizes and workloads are fine with the new behaviour and generally benefit.
The problem is particularly visible when the only active tasks in the system have set numa_preferred_nid because as far as the load balancer and NUMA balancer is concerned, there is no reason to force the SP workload to spread wide.
Quoted 6 lines
The largest machine I had available was 4 sockets.
The other curiousity is that you used C class. On bigger machines, that is very short lived to the point of being almost useless. Is D class similarly affected?
I expect D class to be similarly affected because the same pattern holds -- tasks say on CPUs local to their memory even though more memory bandwidth may be available on remote nodes.
Quoted 27 lines
Quoted 16 lines
5.6.0_rc3.tip_lb_numa+
sp.C.x_008_02 - CPU load average across the individual NUMA nodes (timestep is 5 seconds)# NUMA | AVR | Utilization over time in percentage 0 | 5 | 12 9 3 0 0 11 8 0 1 3 5 17 9 5 0 0 0 11 3 1 | 16 | 20 21 10 10 2 6 9 12 11 9 9 23 24 23 24 24 24 19 20 2 | 21 | 19 23 26 22 22 23 25 20 25 34 38 17 13 13 13 13 13 27 13 3 | 15 | 19 23 20 21 21 15 15 20 20 18 10 10 9 9 9 9 9 9 11 4 | 19 | 13 14 15 22 23 20 19 20 17 12 15 15 25 25 24 24 24 14 24 5 | 3 | 0 2 11 6 20 8 0 0 0 0 0 0 0 0 0 0 0 0 9 6 | 0 | 0 0 0 5 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 7 | 4 | 0 0 0 0 0 0 4 11 9 0 0 0 0 5 12 12 12 3 05.6.0-0.rc3.1.elrdy
sp.C.x_008_01 - CPU load average across the individual NUMA nodes (timestep is 5 seconds)Quoted 10 lines
# NUMA | AVR | Utilization over time in percentage 0 | 13 | 6 8 10 10 11 10 18 13 20 17 14 15 1 | 11 | 10 10 11 11 9 16 12 14 9 11 11 10 2 | 17 | 25 19 16 11 13 12 11 16 17 22 22 16 3 | 21 | 21 22 22 23 21 23 23 21 21 17 22 21 4 | 14 | 20 23 11 12 15 18 12 10 9 13 12 18 5 | 4 | 0 0 8 10 7 0 8 2 0 0 8 2 6 | 1 | 0 5 1 0 0 0 0 0 0 1 0 0 7 | 7 | 7 3 10 10 10 11 3 8 10 4 0 5
Quoted 8 lines
A critical difference with the series is that large imbalances shouldn't happen but prior to the series the NUMA balancing would keep trying to move tasks to a node with load balancing moving them back. That should not happen any more but there are cases where it's actually faster to have the fight between NUMA balancing and load balancing. Ideally a degree of imbalance would be allowed but I haven't found a way of doing that without side effects.
So this is what's happening -- at low utilisation, tasks are staying local to their memory. For a lot of cases, this is a good thing -- communicating tasks stay local for example and tasks that are not completely memory bound benefit. Machines that have sufficient local memory bandwidth also appear to benefit.
sp.C appears to be a significant corner case when the degree of parallelisation is lower than the number of NUMA nodes in the system and of the NAS workloads, bt is also mildly affected. In each cases, memory was almost completely local and there was low NUMA activity but performance suffered. This is the BT case;
5.6.0-rc3 5.6.0-rc3
vanilla schedcore-20200227
Min bt.C 176.05 ( 0.00%) 185.03 ( -5.10%)
Amean bt.C 178.62 ( 0.00%) 185.54 * -3.88%*
Stddev bt.C 4.26 ( 0.00%) 0.60 ( 85.95%)
CoeffVar bt.C 2.38 ( 0.00%) 0.32 ( 86.47%)
Max bt.C 186.09 ( 0.00%) 186.48 ( -0.21%)
BAmean-50 bt.C 176.18 ( 0.00%) 185.08 ( -5.06%)
BAmean-95 bt.C 176.75 ( 0.00%) 185.31 ( -4.84%)
BAmean-99 bt.C 176.75 ( 0.00%) 185.31 ( -4.84%)Note the spread in performance. tip/sched/core looks worse than average but its coefficient of variance was just 0.32% versus 2.38% with the vanilla kernel. The vanilla kernel is a lot less stable in terms of performance due to the fighting between CPU Load and NUMA Balancing.
A heatmap of the CPU usage per LLC showed 4 tasks running on 2 nodes with two nodes idle -- there was almost no other system activity that would allow the load balancer to balance on tasks that are unconcerned with locality. The vanilla case was interesting -- of the 5 iterations, 4 spread with one task on 4 nodes but one iteration stacked 4 tasks on 2 nodes so it's not even consistent. The NUMA activity looked like this for the overall workload.
Ops NUMA alloc hit 3450166.00 2406738.00
Ops NUMA alloc miss 0.00 0.00 Ops NUMA interleave hit 0.00 0.00 Ops NUMA alloc local 1047975.00 41131.00 Ops NUMA base-page range updates 15864254.00 16283456.00 Ops NUMA PTE updates 15148478.00 15563584.00 Ops NUMA PMD updates 1398.00 1406.00 Ops NUMA hint faults 15128332.00 15535357.00 Ops NUMA hint local faults % 12253847.00 14471269.00
Ops NUMA hint local percent 81.00 93.15
Ops NUMA pages migrated 993033.00 4.00
Ops AutoNUMA cost 75771.58 77790.77
PTE hinting was more or less the same but look at the locality. 81% local for the baseline vanilla kernel and 93.15% for what's in tip/sched/core. The baseline kernel migrates almost 1 million pages over 15 minutes (5 iterations) and tip/sched/core migrates ... 4 pages.
Looking at the faults over time, the baseline kernel initially faults with pages local, drops to 80% shortly after starting and then starts climbing back up again as pages get migrated. Initially the number of hints the baseline kernel traps is extremely high and drops as pages migrate
Most others were almost neutral with the impact of the series more obvious in some than others. is.C is really short-lived for example but locality of faults went from 43% to 95% local for example.
sp.C was by far the most obvious impact
5.6.0-rc3 5.6.0-rc3
vanilla schedcore-20200227
Min sp.C 141.52 ( 0.00%) 173.61 ( -22.68%)
Amean sp.C 141.87 ( 0.00%) 174.00 * -22.65%*
Stddev sp.C 0.26 ( 0.00%) 0.25 ( 5.06%)
CoeffVar sp.C 0.18 ( 0.00%) 0.14 ( 22.59%)
Max sp.C 142.10 ( 0.00%) 174.25 ( -22.62%)
BAmean-50 sp.C 141.59 ( 0.00%) 173.79 ( -22.74%)
BAmean-95 sp.C 141.81 ( 0.00%) 173.93 ( -22.65%)
BAmean-99 sp.C 141.81 ( 0.00%) 173.93 ( -22.65%)That's a big hit in terms of performance and it looks less variable. Looking at the NUMA stats
Ops NUMA alloc hit 3100836.00 2161667.00
Ops NUMA alloc miss 0.00 0.00 Ops NUMA interleave hit 0.00 0.00 Ops NUMA alloc local 915700.00 98531.00 Ops NUMA base-page range updates 12178032.00 13483382.00 Ops NUMA PTE updates 11809904.00 12792182.00
Ops NUMA PMD updates 719.00 1350.00
Ops NUMA hint faults 11791912.00 12782512.00
Ops NUMA hint local faults % 9345987.00 11467427.00
Ops NUMA hint local percent 79.26 89.71
Ops NUMA pages migrated 871805.00 21505.00
Ops AutoNUMA cost 59061.37 64007.35
Note the locality -- 79.26% to 89.71% but the vanilla kernel migrated 871K pages and the new kernel migrates 21K. Looking at migrations over time, I can see that the vanilla kernel migrates 180K pages in the first 10 seconds of each iteration while tip/sched/core migrated few enough that it's not even clear on the graph. The workload was long-lived enough that the initial disruption was less visible when running for long enough.
The problem is that there is nothing unique that the kernel measures that I can think of that uniquely identifies that SP should spread wide and migrate early to move its shared pages from other processes that are less memory bound or communicating heavily. The state is simply not maintained and it cannot be inferred from the runqueue or task state. From both a locality point of view and available CPUs, leaving SP alone makes sense but we do not detect that memory bandwidth is an issue. In other cases, the cost of migrations alone would damage performance and SP is an exception as it's long-lived enough to benefit once the first few seconds have passed.
I experimented with a few different approaches but without being able to detect the bandwidth, it was a case that SP can be improved but almost everything else suffers. For example, SP on 2-socket degrades when spread too quickly on machines with enough memory bandwidth so with tip/sched/core SP either benefits or suffers depending on the machine. Basic communicating tasks degrade 4-8% depending on the machine and exact workload when moving back to the vanilla kernel and that is fairly universal AFAIS.
So I think that the new behaviour generally is more sane -- do not excessively fight between memory and CPU balancing but if there are suggestions on how to distinguish between tasks that should spread wide and evenly regardless of initial memory locality then I'm all ears. I do not think migrating like crazy hoping it happens to work out and having CPU Load and NUMA Balancing using very different criteria for evaluation is a better approach.
Signature · 2 lines
Mel Gorman SUSE Labs
Hi Mel,
thanks a lot for analyzing it!
My big concern is that the performance drop for low threads counts (roughly up to 2x number of NUMA nodes) is not just a rare corner case, but it might be more common. We see the drop for the following benchmarks/tests, especially on 8 NUMA nodes servers. However, four and even 2 NUMA node servers are affected as well.
Numbers show average performance drop (median of runtime collected from 5 subsequential runs) compared to vanilla kernel.
2x AMD 7351 (EPYC Naples), 8 NUMA nodes
===================================
NAS: sp_C test: -50%, peak perf. drop with 8 threads
NAS: mg_D: -10% with 16 threads
SPECjvm2008: co_sunflow test: -20% (peak drop with 8 threads)
SPECjvm2008: compress and cr_signverify tests: -10%(peak drop with 8 threads)
SPECjbb2005: -10% for 16 threads
4x INTEL Xeon GOLD-6126 with Sub-NUMA clustering enabled, 8 NUMA nodes
=============================================================
NAS: sp_C test: -35%, peak perf. drop with 16 threads
SPECjvm2008: co_sunflow, compress and cr_signverify tests: -10%(peak drop with 8 threads)
SPECjbb2005: -10% for 24 threads
So far, I have run only a limited number of our tests. I can run our full testing suite next week when required. Please let me know.
Thanks!
Jirka
Quoted 221 lines · Mel Gorman
Quoted 13 lines · Mel Gorman
Quoted 4 lines
The actual data reports are on an intranet web page so they are harder to share. I can create PDFs or screenshots but I didn't want to just blast those to the list. I'd be happy to send some direclty if you are interested.
Send them to me privately please.
Quoted 5 lines
Some data in text format I can easily include shows imbalances across the numa nodes. This is for NAS sp.C.x benchmark because it was easiest to pull and see the data in text. The regressions can be seen in other tests as well.
Quoted 8 lines
What was the value for x?
I ask because I ran NAS across a variety of machines for C class in two configurations -- one using as many CPUs as possible and one running with a third of the available CPUs for both MPI and OMP. Generally there were small gains and losses across multiple kernels but often within the noise or within a few percent of each other.
On re-examining the case, this pattern matches. There are some corner cases for large machines that have low utilisation that are obvious. With the old behaviour, load balancing would even load evenly all available NUMA nodes while NUMA balancing would constantly adjust it for locality. The old load balancer does this even if a task starts with all of its memory local to one node.
The degree where it causes the most problems appears to be roughly for task counts lower than 2 *
NR_NODESas per the small imbalance allowed byadjust_numa_imbalancebut the actual distribution is variable. It's not always 2 per node, sometimes it can be a little higher depending on when idle balancing happens and other machine activity. This is not universal as other machine sizes and workloads are fine with the new behaviour and generally benefit.The problem is particularly visible when the only active tasks in the system have set
numa_preferred_nidbecause as far as the load balancer and NUMA balancer is concerned, there is no reason to force the SP workload to spread wide.Quoted 6 lines
The largest machine I had available was 4 sockets.
The other curiousity is that you used C class. On bigger machines, that is very short lived to the point of being almost useless. Is D class similarly affected?
I expect D class to be similarly affected because the same pattern holds -- tasks say on CPUs local to their memory even though more memory bandwidth may be available on remote nodes.
Quoted 27 lines
Quoted 16 lines
5.6.0_rc3.tip_lb_numa+
sp.C.x_008_02 - CPU load average across the individual NUMA nodes (timestep is 5 seconds)# NUMA | AVR | Utilization over time in percentage 0 | 5 | 12 9 3 0 0 11 8 0 1 3 5 17 9 5 0 0 0 11 3 1 | 16 | 20 21 10 10 2 6 9 12 11 9 9 23 24 23 24 24 24 19 20 2 | 21 | 19 23 26 22 22 23 25 20 25 34 38 17 13 13 13 13 13 27 13 3 | 15 | 19 23 20 21 21 15 15 20 20 18 10 10 9 9 9 9 9 9 11 4 | 19 | 13 14 15 22 23 20 19 20 17 12 15 15 25 25 24 24 24 14 24 5 | 3 | 0 2 11 6 20 8 0 0 0 0 0 0 0 0 0 0 0 0 9 6 | 0 | 0 0 0 5 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 7 | 4 | 0 0 0 0 0 0 4 11 9 0 0 0 0 5 12 12 12 3 05.6.0-0.rc3.1.elrdy
sp.C.x_008_01 - CPU load average across the individual NUMA nodes (timestep is 5 seconds)Quoted 10 lines
# NUMA | AVR | Utilization over time in percentage 0 | 13 | 6 8 10 10 11 10 18 13 20 17 14 15 1 | 11 | 10 10 11 11 9 16 12 14 9 11 11 10 2 | 17 | 25 19 16 11 13 12 11 16 17 22 22 16 3 | 21 | 21 22 22 23 21 23 23 21 21 17 22 21 4 | 14 | 20 23 11 12 15 18 12 10 9 13 12 18 5 | 4 | 0 0 8 10 7 0 8 2 0 0 8 2 6 | 1 | 0 5 1 0 0 0 0 0 0 1 0 0 7 | 7 | 7 3 10 10 10 11 3 8 10 4 0 5Quoted 8 lines
A critical difference with the series is that large imbalances shouldn't happen but prior to the series the NUMA balancing would keep trying to move tasks to a node with load balancing moving them back. That should not happen any more but there are cases where it's actually faster to have the fight between NUMA balancing and load balancing. Ideally a degree of imbalance would be allowed but I haven't found a way of doing that without side effects.
So this is what's happening -- at low utilisation, tasks are staying local to their memory. For a lot of cases, this is a good thing -- communicating tasks stay local for example and tasks that are not completely memory bound benefit. Machines that have sufficient local memory bandwidth also appear to benefit.
sp.C appears to be a significant corner case when the degree of parallelisation is lower than the number of NUMA nodes in the system and of the NAS workloads, bt is also mildly affected. In each cases, memory was almost completely local and there was low NUMA activity but performance suffered. This is the BT case;
5.6.0-rc3 5.6.0-rc3 vanilla schedcore-20200227 Min bt.C 176.05 ( 0.00%) 185.03 ( -5.10%) Amean bt.C 178.62 ( 0.00%) 185.54 * -3.88%* Stddev bt.C 4.26 ( 0.00%) 0.60 ( 85.95%) CoeffVar bt.C 2.38 ( 0.00%) 0.32 ( 86.47%) Max bt.C 186.09 ( 0.00%) 186.48 ( -0.21%) BAmean-50 bt.C 176.18 ( 0.00%) 185.08 ( -5.06%) BAmean-95 bt.C 176.75 ( 0.00%) 185.31 ( -4.84%) BAmean-99 bt.C 176.75 ( 0.00%) 185.31 ( -4.84%)Note the spread in performance. tip/sched/core looks worse than average but its coefficient of variance was just 0.32% versus 2.38% with the vanilla kernel. The vanilla kernel is a lot less stable in terms of performance due to the fighting between CPU Load and NUMA Balancing.
A heatmap of the CPU usage per LLC showed 4 tasks running on 2 nodes with two nodes idle -- there was almost no other system activity that would allow the load balancer to balance on tasks that are unconcerned with locality. The vanilla case was interesting -- of the 5 iterations, 4 spread with one task on 4 nodes but one iteration stacked 4 tasks on 2 nodes so it's not even consistent. The NUMA activity looked like this for the overall workload.
Ops NUMA alloc hit 3450166.00 2406738.00
Ops NUMA alloc miss 0.00 0.00 Ops NUMA interleave hit 0.00 0.00 Ops NUMA alloc local 1047975.00 41131.00 Ops NUMA base-page range updates 15864254.00 16283456.00 Ops NUMA PTE updates 15148478.00 15563584.00 Ops NUMA PMD updates 1398.00 1406.00 Ops NUMA hint faults 15128332.00 15535357.00 Ops NUMA hint local faults % 12253847.00 14471269.00Ops NUMA hint local percent 81.00 93.15
Ops NUMA pages migrated 993033.00 4.00
Ops AutoNUMA cost 75771.58 77790.77PTE hinting was more or less the same but look at the locality. 81% local for the baseline vanilla kernel and 93.15% for what's in tip/sched/core. The baseline kernel migrates almost 1 million pages over 15 minutes (5 iterations) and tip/sched/core migrates ... 4 pages.
Looking at the faults over time, the baseline kernel initially faults with pages local, drops to 80% shortly after starting and then starts climbing back up again as pages get migrated. Initially the number of hints the baseline kernel traps is extremely high and drops as pages migrate
Most others were almost neutral with the impact of the series more obvious in some than others. is.C is really short-lived for example but locality of faults went from 43% to 95% local for example.
sp.C was by far the most obvious impact
5.6.0-rc3 5.6.0-rc3 vanilla schedcore-20200227 Min sp.C 141.52 ( 0.00%) 173.61 ( -22.68%) Amean sp.C 141.87 ( 0.00%) 174.00 * -22.65%* Stddev sp.C 0.26 ( 0.00%) 0.25 ( 5.06%) CoeffVar sp.C 0.18 ( 0.00%) 0.14 ( 22.59%) Max sp.C 142.10 ( 0.00%) 174.25 ( -22.62%) BAmean-50 sp.C 141.59 ( 0.00%) 173.79 ( -22.74%) BAmean-95 sp.C 141.81 ( 0.00%) 173.93 ( -22.65%) BAmean-99 sp.C 141.81 ( 0.00%) 173.93 ( -22.65%)That's a big hit in terms of performance and it looks less variable. Looking at the NUMA stats
Ops NUMA alloc hit 3100836.00 2161667.00
Ops NUMA alloc miss 0.00 0.00 Ops NUMA interleave hit 0.00 0.00 Ops NUMA alloc local 915700.00 98531.00 Ops NUMA base-page range updates 12178032.00 13483382.00 Ops NUMA PTE updates 11809904.00 12792182.00Ops NUMA PMD updates 719.00 1350.00
Ops NUMA hint faults 11791912.00 12782512.00
Ops NUMA hint local faults % 9345987.00 11467427.00
Ops NUMA hint local percent 79.26 89.71
Ops NUMA pages migrated 871805.00 21505.00
Ops AutoNUMA cost 59061.37 64007.35Note the locality -- 79.26% to 89.71% but the vanilla kernel migrated 871K pages and the new kernel migrates 21K. Looking at migrations over time, I can see that the vanilla kernel migrates 180K pages in the first 10 seconds of each iteration while tip/sched/core migrated few enough that it's not even clear on the graph. The workload was long-lived enough that the initial disruption was less visible when running for long enough.
The problem is that there is nothing unique that the kernel measures that I can think of that uniquely identifies that SP should spread wide and migrate early to move its shared pages from other processes that are less memory bound or communicating heavily. The state is simply not maintained and it cannot be inferred from the runqueue or task state. From both a locality point of view and available CPUs, leaving SP alone makes sense but we do not detect that memory bandwidth is an issue. In other cases, the cost of migrations alone would damage performance and SP is an exception as it's long-lived enough to benefit once the first few seconds have passed.
I experimented with a few different approaches but without being able to detect the bandwidth, it was a case that SP can be improved but almost everything else suffers. For example, SP on 2-socket degrades when spread too quickly on machines with enough memory bandwidth so with tip/sched/core SP either benefits or suffers depending on the machine. Basic communicating tasks degrade 4-8% depending on the machine and exact workload when moving back to the vanilla kernel and that is fairly universal AFAIS.
So I think that the new behaviour generally is more sane -- do not excessively fight between memory and CPU balancing but if there are suggestions on how to distinguish between tasks that should spread wide and evenly regardless of initial memory locality then I'm all ears. I do not think migrating like crazy hoping it happens to work out and having CPU Load and NUMA Balancing using very different criteria for evaluation is a better approach.
--
Mel Gorman
SUSE Labs
Signature · 1 line
-Jirka
A message that is not in the archive (sent off-list or not yet received): CAE4VaGA4q4_qfC5qe3zaLRfiJhvMaSb2WADgOcQeTwmPvNat+A@mail.gmail.com
Quoted 7 lines · Jirka Hladky
Hi Mel,
thanks a lot for analyzing it!
My big concern is that the performance drop for low threads counts (roughly up to 2x number of NUMA nodes) is not just a rare corner case, but it might be more common.
That is hard to tell. In my case I was seeing the problem running a HPC workload, dedicated to that machine and using only 6% of available CPUs. I find it unlikely that is common because who acquires such a large machine and then uses a tiny percentage of it. Rember also the other points I made such as 1M migrations happening in the first few seconds just trying to deal with the load balancer and NUMA balancer fighting each other. While that might happen to be good for SP, it's not universally good behaviour and I've dealt with issues in the past whereby the NUMA balancer would get confused and just ramp up the frequency it samples and migrates trying to override the load balancer.
Quoted 10 lines
We see the drop for the following benchmarks/tests, especially on 8 NUMA nodes servers. However, four and even 2 NUMA node servers are affected as well.
Numbers show average performance drop (median of runtime collected from 5 subsequential runs) compared to vanilla kernel.
2x AMD 7351 (EPYC Naples), 8 NUMA nodes
===================================
NAS:sp_Ctest: -50%, peak perf. drop with 8 threads
I hadn't tested 8 threads specifically I think that works out as using 12.5% of the available machine. The allowed imbalance between nodes means that some SP instances will stack on the same node but not the same CPU.
Quoted 1 line
NAS:
mg_D: -10% with 16 threads
While I do not have the most up to date figures available, I found the opposite trend at 21 threads (the test case I used were 1/3rd available CPUs and as close to maximum CPUs as possible). There I found it was 10% faster for an 8 node machine.
For 4 nodes, using a single JVM was performance neutral *on average* but much less variable. With one JVM per node, there was a mix of small <2% regressions for some thread counts and up to 9% faster on others.
Quoted 3 lines
SPECjvm2008:
co_sunflowtest: -20% (peak drop with 8 threads)
SPECjvm2008: compress andcr_signverifytests: -10%(peak drop with 8 threads)
I didn't run SPECjvm for multiple thread sizes so I don't have data yet and may not have for another day at least.
Quoted 2 lines
SPECjbb2005: -10% for 16 threads
I found this to depend in the number of JVMs used and the thread count. Slower at low thread counts, faster at higher thread counts but with more stable results with the series applied and less NUMA balancer activity.
This is somewhat of a dilemma. Without the series, the load balancer and NUMA balancer use very different criteria on what should happen and results are not stable. In some cases acting randomly happens to work out and in others it does not and overall it depends on the workload and machine. It's very highly dependent on both the workload and the machine and it's a question if we want to continue dealing with two parts of the scheduler disagreeing on what criteria to use or try to improve the reconciled load and memory balancer sharing similar logic.
In *general*, I found that the series won a lot more than it lost across a spread of workloads and machines but unfortunately it's also an area where counter-examples can be found.
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 2 lines
find it unlikely that is common because who acquires such a large machine and then uses a tiny percentage of it.
I generally agree, but I also want to make a point that AMD made these large systems much more affordable with their EPYC CPUs. The 8 NUMA node server we are using costs under $8k.
Quoted 4 lines
This is somewhat of a dilemma. Without the series, the load balancer and NUMA balancer use very different criteria on what should happen and results are not stable.
Unfortunately, I see instabilities also for the series. This is again for the sp_C test with 8 threads executed on dual-socket AMD 7351 (EPYC Naples) server with 8 NUMA nodes. With the series applied, the runtime varies from 86 to 165 seconds! Could we do something about it? The runtime of 86 seconds would be acceptable. If we could stabilize this case and get consistent runtime around 80 seconds, the problem would be gone.
Do you experience the similar instability of results on your HW for sp_C with low thread counts?
Runtime with this series applied:
$ grep "Time in seconds" *log
sp.C.x.defaultRun.008threads.loop01.log: Time in seconds =
125.73
sp.C.x.defaultRun.008threads.loop02.log: Time in seconds =
87.54
sp.C.x.defaultRun.008threads.loop03.log: Time in seconds =
86.93
sp.C.x.defaultRun.008threads.loop04.log: Time in seconds =
165.98
sp.C.x.defaultRun.008threads.loop05.log: Time in seconds =
114.78For comparison, here are vanilla kernel results:
$ grep "Time in seconds" *log
sp.C.x.defaultRun.008threads.loop01.log: Time in seconds =
59.83
sp.C.x.defaultRun.008threads.loop02.log: Time in seconds =
67.72
sp.C.x.defaultRun.008threads.loop03.log: Time in seconds =
63.62
sp.C.x.defaultRun.008threads.loop04.log: Time in seconds =
55.01
sp.C.x.defaultRun.008threads.loop05.log: Time in seconds =
65.20Quoted 3 lines
In *general*, I found that the series won a lot more than it lost across a spread of workloads and machines but unfortunately it's also an area where counter-examples can be found.
OK, fair enough. I understand that there will always be trade-offs when making changes to scheduler like this. And I agree that cases with higher system load (where is series is helpful) outweigh the performance drops for low threads counts. I was hoping that it would be possible to improve the small threads results while keeping the gains for other scenarios:-) But let's be realistic - I would be happy to fix the extreme case mentioned above. The other issues where performance drop is about 20% are OK with me and are outweighed by the gains for different scenarios.
Thanks again for looking into it. I know that covering all cases is hard. I very much appreciate what you do!
Jirka
Quoted 80 lines · Mel Gorman
Quoted 7 lines · Jirka Hladky
Hi Mel,
thanks a lot for analyzing it!
My big concern is that the performance drop for low threads counts (roughly up to 2x number of NUMA nodes) is not just a rare corner case, but it might be more common.
That is hard to tell. In my case I was seeing the problem running a HPC workload, dedicated to that machine and using only 6% of available CPUs. I find it unlikely that is common because who acquires such a large machine and then uses a tiny percentage of it. Rember also the other points I made such as 1M migrations happening in the first few seconds just trying to deal with the load balancer and NUMA balancer fighting each other. While that might happen to be good for SP, it's not universally good behaviour and I've dealt with issues in the past whereby the NUMA balancer would get confused and just ramp up the frequency it samples and migrates trying to override the load balancer.
Quoted 10 lines
We see the drop for the following benchmarks/tests, especially on 8 NUMA nodes servers. However, four and even 2 NUMA node servers are affected as well.
Numbers show average performance drop (median of runtime collected from 5 subsequential runs) compared to vanilla kernel.
2x AMD 7351 (EPYC Naples), 8 NUMA nodes
===================================
NAS:sp_Ctest: -50%, peak perf. drop with 8 threadsI hadn't tested 8 threads specifically I think that works out as using 12.5% of the available machine. The allowed imbalance between nodes means that some SP instances will stack on the same node but not the same CPU.
Quoted 1 line
NAS:
mg_D: -10% with 16 threadsWhile I do not have the most up to date figures available, I found the opposite trend at 21 threads (the test case I used were 1/3rd available CPUs and as close to maximum CPUs as possible). There I found it was 10% faster for an 8 node machine.
For 4 nodes, using a single JVM was performance neutral *on average* but much less variable. With one JVM per node, there was a mix of small <2% regressions for some thread counts and up to 9% faster on others.
Quoted 3 lines
SPECjvm2008:
co_sunflowtest: -20% (peak drop with 8 threads)
SPECjvm2008: compress andcr_signverifytests: -10%(peak drop with 8 threads)I didn't run SPECjvm for multiple thread sizes so I don't have data yet and may not have for another day at least.
Quoted 2 lines
SPECjbb2005: -10% for 16 threads
I found this to depend in the number of JVMs used and the thread count. Slower at low thread counts, faster at higher thread counts but with more stable results with the series applied and less NUMA balancer activity.
This is somewhat of a dilemma. Without the series, the load balancer and NUMA balancer use very different criteria on what should happen and results are not stable. In some cases acting randomly happens to work out and in others it does not and overall it depends on the workload and machine. It's very highly dependent on both the workload and the machine and it's a question if we want to continue dealing with two parts of the scheduler disagreeing on what criteria to use or try to improve the reconciled load and memory balancer sharing similar logic.
In *general*, I found that the series won a lot more than it lost across a spread of workloads and machines but unfortunately it's also an area where counter-examples can be found.
--
Mel Gorman
SUSE Labs
Signature · 1 line
-Jirka
A message that is not in the archive (sent off-list or not yet received): CAE4VaGD8DUEi6JnKd8vrqUL_8HZXnNyHMoK2D+1-F5wo+5Z53Q@mail.gmail.com
Quoted 23 lines · Jirka Hladky
Quoted 3 lines
find it unlikely that is common because who acquires such a large machine and then uses a tiny percentage of it.
I generally agree, but I also want to make a point that AMD made these large systems much more affordable with their EPYC CPUs. The 8 NUMA node server we are using costs under $8k.
Quoted 3 lines
This is somewhat of a dilemma. Without the series, the load balancer and NUMA balancer use very different criteria on what should happen and results are not stable.
Unfortunately, I see instabilities also for the series. This is again for the
sp_Ctest with 8 threads executed on dual-socket AMD 7351 (EPYC Naples) server with 8 NUMA nodes. With the series applied, the runtime varies from 86 to 165 seconds! Could we do something about it? The runtime of 86 seconds would be acceptable. If we could stabilize this case and get consistent runtime around 80 seconds, the problem would be gone.
Quoted 3 lines
Do you experience the similar instability of results on your HW for
sp_Cwith low thread counts?
I saw something similar but observed that it depended on whether the worker tasks got spread wide or not which partially came down to luck. The question is if it's possible to pick a point where we spread wide and can recover quickly enough when tasks need to remain close without knowledge of the future. Putting a balancing limit on tasks that recently woke would be one option but that could also cause persistent improper balancing for tasks that wake frequently.
Quoted 33 lines
Runtime with this series applied:
$ grep "Time in seconds" *log sp.C.x.defaultRun.008threads.loop01.log: Time in seconds = 125.73 sp.C.x.defaultRun.008threads.loop02.log: Time in seconds = 87.54 sp.C.x.defaultRun.008threads.loop03.log: Time in seconds = 86.93 sp.C.x.defaultRun.008threads.loop04.log: Time in seconds = 165.98 sp.C.x.defaultRun.008threads.loop05.log: Time in seconds = 114.78For comparison, here are vanilla kernel results:
$ grep "Time in seconds" *log sp.C.x.defaultRun.008threads.loop01.log: Time in seconds = 59.83 sp.C.x.defaultRun.008threads.loop02.log: Time in seconds = 67.72 sp.C.x.defaultRun.008threads.loop03.log: Time in seconds = 63.62 sp.C.x.defaultRun.008threads.loop04.log: Time in seconds = 55.01 sp.C.x.defaultRun.008threads.loop05.log: Time in seconds = 65.20Quoted 3 lines
In *general*, I found that the series won a lot more than it lost across a spread of workloads and machines but unfortunately it's also an area where counter-examples can be found.
Quoted 9 lines
OK, fair enough. I understand that there will always be trade-offs when making changes to scheduler like this. And I agree that cases with higher system load (where is series is helpful) outweigh the performance drops for low threads counts. I was hoping that it would be possible to improve the small threads results while keeping the gains for other scenarios:-) But let's be realistic - I would be happy to fix the extreme case mentioned above. The other issues where performance drop is about 20% are OK with me and are outweighed by the gains for different scenarios.
I'll continue thinking about it but whatever chance there is of improving it while keeping CPU balancing, NUMA balancing and wake affine consistent with each other, I think there is no chance with the inconsistent logic used in the vanilla code :(
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 4 lines
I'll continue thinking about it but whatever chance there is of improving it while keeping CPU balancing, NUMA balancing and wake affine consistent with each other, I think there is no chance with the inconsistent logic used in the vanilla code :(
Thank you, Mel!
Quoted 89 lines · Mel Gorman
Quoted 23 lines · Jirka Hladky
Quoted 3 lines
find it unlikely that is common because who acquires such a large machine and then uses a tiny percentage of it.
I generally agree, but I also want to make a point that AMD made these large systems much more affordable with their EPYC CPUs. The 8 NUMA node server we are using costs under $8k.
Quoted 3 lines
This is somewhat of a dilemma. Without the series, the load balancer and NUMA balancer use very different criteria on what should happen and results are not stable.
Unfortunately, I see instabilities also for the series. This is again for the
sp_Ctest with 8 threads executed on dual-socket AMD 7351 (EPYC Naples) server with 8 NUMA nodes. With the series applied, the runtime varies from 86 to 165 seconds! Could we do something about it? The runtime of 86 seconds would be acceptable. If we could stabilize this case and get consistent runtime around 80 seconds, the problem would be gone.Quoted 3 lines
Do you experience the similar instability of results on your HW for
sp_Cwith low thread counts?I saw something similar but observed that it depended on whether the worker tasks got spread wide or not which partially came down to luck. The question is if it's possible to pick a point where we spread wide and can recover quickly enough when tasks need to remain close without knowledge of the future. Putting a balancing limit on tasks that recently woke would be one option but that could also cause persistent improper balancing for tasks that wake frequently.
Quoted 33 lines
Runtime with this series applied:
$ grep "Time in seconds" *log sp.C.x.defaultRun.008threads.loop01.log: Time in seconds = 125.73 sp.C.x.defaultRun.008threads.loop02.log: Time in seconds = 87.54 sp.C.x.defaultRun.008threads.loop03.log: Time in seconds = 86.93 sp.C.x.defaultRun.008threads.loop04.log: Time in seconds = 165.98 sp.C.x.defaultRun.008threads.loop05.log: Time in seconds = 114.78For comparison, here are vanilla kernel results:
$ grep "Time in seconds" *log sp.C.x.defaultRun.008threads.loop01.log: Time in seconds = 59.83 sp.C.x.defaultRun.008threads.loop02.log: Time in seconds = 67.72 sp.C.x.defaultRun.008threads.loop03.log: Time in seconds = 63.62 sp.C.x.defaultRun.008threads.loop04.log: Time in seconds = 55.01 sp.C.x.defaultRun.008threads.loop05.log: Time in seconds = 65.20Quoted 3 lines
In *general*, I found that the series won a lot more than it lost across a spread of workloads and machines but unfortunately it's also an area where counter-examples can be found.
Quoted 9 lines
OK, fair enough. I understand that there will always be trade-offs when making changes to scheduler like this. And I agree that cases with higher system load (where is series is helpful) outweigh the performance drops for low threads counts. I was hoping that it would be possible to improve the small threads results while keeping the gains for other scenarios:-) But let's be realistic - I would be happy to fix the extreme case mentioned above. The other issues where performance drop is about 20% are OK with me and are outweighed by the gains for different scenarios.
I'll continue thinking about it but whatever chance there is of improving it while keeping CPU balancing, NUMA balancing and wake affine consistent with each other, I think there is no chance with the inconsistent logic used in the vanilla code :(
--
Mel Gorman
SUSE Labs
Signature · 1 line
-Jirka
Hi Mel,
just a quick update. I have increased the testing coverage and other tests from the NAS shows a big performance drop for the low number of threads as well:
sp_C_x - show still the biggest drop upto 50%bt_C_x - performance drop upto 40%ua_C_x - performance drop upto 30%
My point is that the performance drop for the low number of threads is more common than we have initially thought.
Let me know what you need more data.
Thanks!
Jirka
Quoted 103 lines · Jirka Hladky
Quoted 5 lines
I'll continue thinking about it but whatever chance there is of improving it while keeping CPU balancing, NUMA balancing and wake affine consistent with each other, I think there is no chance with the inconsistent logic used in the vanilla code :(
Thank you, Mel!
Quoted 80 lines · Mel Gorman
Quoted 23 lines · Jirka Hladky
Quoted 3 lines
find it unlikely that is common because who acquires such a large machine and then uses a tiny percentage of it.
I generally agree, but I also want to make a point that AMD made these large systems much more affordable with their EPYC CPUs. The 8 NUMA node server we are using costs under $8k.
Quoted 3 lines
This is somewhat of a dilemma. Without the series, the load balancer and NUMA balancer use very different criteria on what should happen and results are not stable.
Unfortunately, I see instabilities also for the series. This is again for the
sp_Ctest with 8 threads executed on dual-socket AMD 7351 (EPYC Naples) server with 8 NUMA nodes. With the series applied, the runtime varies from 86 to 165 seconds! Could we do something about it? The runtime of 86 seconds would be acceptable. If we could stabilize this case and get consistent runtime around 80 seconds, the problem would be gone.Quoted 3 lines
Do you experience the similar instability of results on your HW for
sp_Cwith low thread counts?I saw something similar but observed that it depended on whether the worker tasks got spread wide or not which partially came down to luck. The question is if it's possible to pick a point where we spread wide and can recover quickly enough when tasks need to remain close without knowledge of the future. Putting a balancing limit on tasks that recently woke would be one option but that could also cause persistent improper balancing for tasks that wake frequently.
Quoted 33 lines
Runtime with this series applied:
$ grep "Time in seconds" *log sp.C.x.defaultRun.008threads.loop01.log: Time in seconds = 125.73 sp.C.x.defaultRun.008threads.loop02.log: Time in seconds = 87.54 sp.C.x.defaultRun.008threads.loop03.log: Time in seconds = 86.93 sp.C.x.defaultRun.008threads.loop04.log: Time in seconds = 165.98 sp.C.x.defaultRun.008threads.loop05.log: Time in seconds = 114.78For comparison, here are vanilla kernel results:
$ grep "Time in seconds" *log sp.C.x.defaultRun.008threads.loop01.log: Time in seconds = 59.83 sp.C.x.defaultRun.008threads.loop02.log: Time in seconds = 67.72 sp.C.x.defaultRun.008threads.loop03.log: Time in seconds = 63.62 sp.C.x.defaultRun.008threads.loop04.log: Time in seconds = 55.01 sp.C.x.defaultRun.008threads.loop05.log: Time in seconds = 65.20Quoted 3 lines
In *general*, I found that the series won a lot more than it lost across a spread of workloads and machines but unfortunately it's also an area where counter-examples can be found.
Quoted 9 lines
OK, fair enough. I understand that there will always be trade-offs when making changes to scheduler like this. And I agree that cases with higher system load (where is series is helpful) outweigh the performance drops for low threads counts. I was hoping that it would be possible to improve the small threads results while keeping the gains for other scenarios:-) But let's be realistic - I would be happy to fix the extreme case mentioned above. The other issues where performance drop is about 20% are OK with me and are outweighed by the gains for different scenarios.
Quoted 9 lines
I'll continue thinking about it but whatever chance there is of improving it while keeping CPU balancing, NUMA balancing and wake affine consistent with each other, I think there is no chance with the inconsistent logic used in the vanilla code :(
--
Mel Gorman
SUSE Labs--
-Jirka
Signature · 1 line
-Jirka
A message that is not in the archive (sent off-list or not yet received): CAE4VaGC09OfU2zXeq2yp_N0zXMbTku5ETz0KEocGi-RSiKXv-w@mail.gmail.com
Quoted 10 lines · Jirka Hladky
Hi Mel,
just a quick update. I have increased the testing coverage and other tests from the NAS shows a big performance drop for the low number of threads as well:
sp_C_x- show still the biggest drop upto 50%bt_C_x- performance drop upto 40%ua_C_x- performance drop upto 30%
MPI or OMP and what is a low thread count? For MPI at least, I saw a 0.4% gain on an 4-node machine for bt_C and a 3.88% regression on 8-nodes. I think it must be OMP you are using because I found I had to disable UA for MPI at some point in the past for reasons I no longer remember.
Quoted 5 lines
My point is that the performance drop for the low number of threads is more common than we have initially thought.
Let me know what you need more data.
I just a clarification on the thread count and a confirmation it's OMP. For MPI, I did note that some of the other NAS kernels shows a slight dip but it was nowhere near as severe as SP and the problem was the same as more -- two or more tasks stayed on the same node without spreading out because there was no pressure to do so. There was enough CPU and memory capacity with no obvious pattern that could be used to spread the load wide early.
One possibility would be to spread wide always at clone time and assume wake_affine will pull related tasks but it's fragile because it breaks if the cloned task execs and then allocates memory from a remote node only to migrate to a local node immediately.
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 4 lines
MPI or OMP and what is a low thread count? For MPI at least, I saw a 0.4% gain on an 4-node machine for
bt_Cand a 3.88% regression on 8-nodes. I think it must be OMP you are using because I found I had to disable UA for MPI at some point in the past for reasons I no longer remember.
Yes, it's indeed OMP. With low threads count, I mean up to 2x number of NUMA nodes (8 threads on 4 NUMA node servers, 16 threads on 8 NUMA node servers).
Quoted 4 lines
One possibility would be to spread wide always at clone time and assume
wake_affinewill pull related tasks but it's fragile because it breaks if the cloned task execs and then allocates memory from a remote node only to migrate to a local node immediately.
I think the only way to find out how it performs is to test it. If you could prepare a patch like that, I'm more than happy to give it a try!
Jirka
Quoted 40 lines · Mel Gorman
Quoted 10 lines · Jirka Hladky
Hi Mel,
just a quick update. I have increased the testing coverage and other tests from the NAS shows a big performance drop for the low number of threads as well:
sp_C_x- show still the biggest drop upto 50%bt_C_x- performance drop upto 40%ua_C_x- performance drop upto 30%MPI or OMP and what is a low thread count? For MPI at least, I saw a 0.4% gain on an 4-node machine for
bt_Cand a 3.88% regression on 8-nodes. I think it must be OMP you are using because I found I had to disable UA for MPI at some point in the past for reasons I no longer remember.Quoted 5 lines
My point is that the performance drop for the low number of threads is more common than we have initially thought.
Let me know what you need more data.
I just a clarification on the thread count and a confirmation it's OMP. For MPI, I did note that some of the other NAS kernels shows a slight dip but it was nowhere near as severe as SP and the problem was the same as more -- two or more tasks stayed on the same node without spreading out because there was no pressure to do so. There was enough CPU and memory capacity with no obvious pattern that could be used to spread the load wide early.
One possibility would be to spread wide always at clone time and assume
wake_affinewill pull related tasks but it's fragile because it breaks if the cloned task execs and then allocates memory from a remote node only to migrate to a local node immediately.--
Mel Gorman
SUSE Labs
Signature · 1 line
-Jirka
A message that is not in the archive (sent off-list or not yet received): CAE4VaGBGbTT8dqNyLWAwuiqL8E+3p1_SqP6XTTV71wNZMjc9Zg@mail.gmail.com
Quoted 11 lines · Jirka Hladky
Quoted 5 lines
MPI or OMP and what is a low thread count? For MPI at least, I saw a 0.4% gain on an 4-node machine for
bt_Cand a 3.88% regression on 8-nodes. I think it must be OMP you are using because I found I had to disable UA for MPI at some point in the past for reasons I no longer remember.Yes, it's indeed OMP. With low threads count, I mean up to 2x number of NUMA nodes (8 threads on 4 NUMA node servers, 16 threads on 8 NUMA node servers).
Ok, so we know it's within the imbalance threshold where a NUMA node can be left idle.
Quoted 9 lines
One possibility would be to spread wide always at clone time and assume
Quoted 3 lines
wake_affinewill pull related tasks but it's fragile because it breaks if the cloned task execs and then allocates memory from a remote node only to migrate to a local node immediately.I think the only way to find out how it performs is to test it. If you could prepare a patch like that, I'm more than happy to give it a try!
When the initial spreading was prevented, it was for pipelines mainly -- even basic shell scripts. In that case it was observed that a shell would fork/exec two tasks connected via pipe that started on separate nodes and had allocated remote data before being pulled close. The processes were typically too short lived for NUMA balancing to fix it up by exec time the information on where the fork happened was lost. See 2c83362734da ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on"). Now the logic has probably been partially broken since because of how SD_NUMA is now treated but the concern about spreading wide prematurely remains.
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 10 lines
When the initial spreading was prevented, it was for pipelines mainly -- even basic shell scripts. In that case it was observed that a shell would fork/exec two tasks connected via pipe that started on separate nodes and had allocated remote data before being pulled close. The processes were typically too short lived for NUMA balancing to fix it up by exec time the information on where the fork happened was lost. See
2c83362734da("sched/fair: ConsiderSD_NUMAwhen selecting the most idle group to schedule on"). Now the logic has probably been partially broken since because of howSD_NUMAis now treated but the concern about spreading wide prematurely remains.
I understand. It's a hard one - let's keep an eye on it. We will continue to test the upstream kernels and we will have some discussion internally. Let's see if anybody has some idea how to treat these special cases.
Enjoy the weekend!
Jirka
Quoted 42 lines · Mel Gorman
Quoted 11 lines · Jirka Hladky
Quoted 5 lines
MPI or OMP and what is a low thread count? For MPI at least, I saw a 0.4% gain on an 4-node machine for
bt_Cand a 3.88% regression on 8-nodes. I think it must be OMP you are using because I found I had to disable UA for MPI at some point in the past for reasons I no longer remember.Yes, it's indeed OMP. With low threads count, I mean up to 2x number of NUMA nodes (8 threads on 4 NUMA node servers, 16 threads on 8 NUMA node servers).
Ok, so we know it's within the imbalance threshold where a NUMA node can be left idle.
Quoted 9 lines
One possibility would be to spread wide always at clone time and assume
Quoted 3 lines
wake_affinewill pull related tasks but it's fragile because it breaks if the cloned task execs and then allocates memory from a remote node only to migrate to a local node immediately.I think the only way to find out how it performs is to test it. If you could prepare a patch like that, I'm more than happy to give it a try!
When the initial spreading was prevented, it was for pipelines mainly -- even basic shell scripts. In that case it was observed that a shell would fork/exec two tasks connected via pipe that started on separate nodes and had allocated remote data before being pulled close. The processes were typically too short lived for NUMA balancing to fix it up by exec time the information on where the fork happened was lost. See
2c83362734da("sched/fair: ConsiderSD_NUMAwhen selecting the most idle group to schedule on"). Now the logic has probably been partially broken since because of howSD_NUMAis now treated but the concern about spreading wide prematurely remains.--
Mel Gorman
SUSE Labs
Signature · 1 line
-Jirka
Hi Mel,
Quoted 6 lines
Quoted 3 lines
Yes, it's indeed OMP. With low threads count, I mean up to 2x number of NUMA nodes (8 threads on 4 NUMA node servers, 16 threads on 8 NUMA node servers).
Ok, so we know it's within the imbalance threshold where a NUMA node can be left idle.
we have discussed today with my colleagues the performance drop for some workloads for low threads counts (roughly up to 2x number of NUMA nodes). We are worried that it can be a severe issue for some use cases, which require a full memory bandwidth even when only part of CPUs is used.
We understand that scheduler cannot distinguish this type of workload from others automatically. However, there was an idea for a * new kernel tunable to control the imbalance threshold *. Based on the purpose of the server, users could set this tunable. See the tuned project, which allows creating performance profiles [1].
What do you think about this approach?
Thanks a lot!
Jirka
Quoted 42 lines · Mel Gorman
Quoted 11 lines · Jirka Hladky
Quoted 5 lines
MPI or OMP and what is a low thread count? For MPI at least, I saw a 0.4% gain on an 4-node machine for
bt_Cand a 3.88% regression on 8-nodes. I think it must be OMP you are using because I found I had to disable UA for MPI at some point in the past for reasons I no longer remember.Yes, it's indeed OMP. With low threads count, I mean up to 2x number of NUMA nodes (8 threads on 4 NUMA node servers, 16 threads on 8 NUMA node servers).
Ok, so we know it's within the imbalance threshold where a NUMA node can be left idle.
Quoted 9 lines
One possibility would be to spread wide always at clone time and assume
Quoted 3 lines
wake_affinewill pull related tasks but it's fragile because it breaks if the cloned task execs and then allocates memory from a remote node only to migrate to a local node immediately.I think the only way to find out how it performs is to test it. If you could prepare a patch like that, I'm more than happy to give it a try!
When the initial spreading was prevented, it was for pipelines mainly -- even basic shell scripts. In that case it was observed that a shell would fork/exec two tasks connected via pipe that started on separate nodes and had allocated remote data before being pulled close. The processes were typically too short lived for NUMA balancing to fix it up by exec time the information on where the fork happened was lost. See
2c83362734da("sched/fair: ConsiderSD_NUMAwhen selecting the most idle group to schedule on"). Now the logic has probably been partially broken since because of howSD_NUMAis now treated but the concern about spreading wide prematurely remains.--
Mel Gorman
SUSE Labs
Signature · 1 line
-Jirka
Quoted 15 lines · Jirka Hladky
Hi Mel,
Quoted 6 lines
Quoted 3 lines
Yes, it's indeed OMP. With low threads count, I mean up to 2x number of NUMA nodes (8 threads on 4 NUMA node servers, 16 threads on 8 NUMA node servers).
Ok, so we know it's within the imbalance threshold where a NUMA node can be left idle.
we have discussed today with my colleagues the performance drop for some workloads for low threads counts (roughly up to 2x number of NUMA nodes). We are worried that it can be a severe issue for some use cases, which require a full memory bandwidth even when only part of CPUs is used.
Quoted 6 lines
We understand that scheduler cannot distinguish this type of workload from others automatically. However, there was an idea for a * new kernel tunable to control the imbalance threshold *. Based on the purpose of the server, users could set this tunable. See the tuned project, which allows creating performance profiles [1].
I'm not completely opposed to it but given that the setting is global, I imagine it could have other consequences if two applications ran at different times have different requirements. Given that it's OMP, I would have imagined that an application that really cared about this would specify what was needed using OMP_PLACES. Why would someone prefer kernel tuning or a tuned profile over OMP_PLACES? After all, it requires specific knowledge of the application even to know that a particular tuned profile is needed.
Signature · 2 lines
Mel Gorman SUSE Labs
Hi Mel,
we are not targeting just OMP applications. We see the performance degradation also for other workloads, like SPECjbb2005 and SPECjvm2008. Even worse, it also affects a higher number of threads. For example, comparing 5.7.0-0.rc2 against 5.6 kernel, on 4 NUMA server with 2x AMD 7351 CPU, we see performance degradation 22% for 32 threads (the system has 64 CPUs in total). We observe this degradation only when we run a single SPECjbb binary. When running 4 SPECjbb binaries in parallel, there is no change in performance between 5.6 and 5.7.
That's why we are asking for the kernel tunable, which we would add to the tuned profile. We don't expect users to change this frequently but rather to set the performance profile once based on the purpose of the server.
If you could prepare a patch for us, we would be more than happy to test it extensively. Based on the results, we can then evaluate if it's the way to go. Thoughts?
Thanks a lot!
Jirka
Quoted 37 lines · Mel Gorman
Quoted 15 lines · Jirka Hladky
Hi Mel,
Quoted 6 lines
Quoted 3 lines
Yes, it's indeed OMP. With low threads count, I mean up to 2x number of NUMA nodes (8 threads on 4 NUMA node servers, 16 threads on 8 NUMA node servers).
Ok, so we know it's within the imbalance threshold where a NUMA node can be left idle.
we have discussed today with my colleagues the performance drop for some workloads for low threads counts (roughly up to 2x number of NUMA nodes). We are worried that it can be a severe issue for some use cases, which require a full memory bandwidth even when only part of CPUs is used.
Quoted 6 lines
We understand that scheduler cannot distinguish this type of workload from others automatically. However, there was an idea for a * new kernel tunable to control the imbalance threshold *. Based on the purpose of the server, users could set this tunable. See the tuned project, which allows creating performance profiles [1].
I'm not completely opposed to it but given that the setting is global, I imagine it could have other consequences if two applications ran at different times have different requirements. Given that it's OMP, I would have imagined that an application that really cared about this would specify what was needed using
OMP_PLACES. Why would someone prefer kernel tuning or a tuned profile overOMP_PLACES? After all, it requires specific knowledge of the application even to know that a particular tuned profile is needed.--
Mel Gorman
SUSE Labs
Signature · 1 line
-Jirka
Quoted 12 lines · On Thu, May 07, 2020 at 06:29:44PM +0200 Jirka Hladky wrote:
Hi Mel,
we are not targeting just OMP applications. We see the performance degradation also for other workloads, like SPECjbb2005 and SPECjvm2008. Even worse, it also affects a higher number of threads. For example, comparing 5.7.0-0.rc2 against 5.6 kernel, on 4 NUMA server with 2x AMD 7351 CPU, we see performance degradation 22% for 32 threads (the system has 64 CPUs in total). We observe this degradation only when we run a single SPECjbb binary. When running 4 SPECjbb binaries in parallel, there is no change in performance between 5.6 and 5.7.
Quoted 9 lines
That's why we are asking for the kernel tunable, which we would add to the tuned profile. We don't expect users to change this frequently but rather to set the performance profile once based on the purpose of the server.
If you could prepare a patch for us, we would be more than happy to test it extensively. Based on the results, we can then evaluate if it's the way to go. Thoughts?
I'm happy to spin up a patch once I'm sure what exactly the tuning would effect. At an initial glance I'm thinking it would be the imbalance_min which is currently hardcoded to 2. But there may be something else...
Cheers,
Phil
Quoted 46 lines
Thanks a lot!
JirkaQuoted 37 lines · Mel Gorman
Quoted 15 lines · Jirka Hladky
Hi Mel,
Quoted 6 lines
Quoted 3 lines
Yes, it's indeed OMP. With low threads count, I mean up to 2x number of NUMA nodes (8 threads on 4 NUMA node servers, 16 threads on 8 NUMA node servers).
Ok, so we know it's within the imbalance threshold where a NUMA node can be left idle.
we have discussed today with my colleagues the performance drop for some workloads for low threads counts (roughly up to 2x number of NUMA nodes). We are worried that it can be a severe issue for some use cases, which require a full memory bandwidth even when only part of CPUs is used.
Quoted 6 lines
We understand that scheduler cannot distinguish this type of workload from others automatically. However, there was an idea for a * new kernel tunable to control the imbalance threshold *. Based on the purpose of the server, users could set this tunable. See the tuned project, which allows creating performance profiles [1].
I'm not completely opposed to it but given that the setting is global, I imagine it could have other consequences if two applications ran at different times have different requirements. Given that it's OMP, I would have imagined that an application that really cared about this would specify what was needed using
OMP_PLACES. Why would someone prefer kernel tuning or a tuned profile overOMP_PLACES? After all, it requires specific knowledge of the application even to know that a particular tuned profile is needed.--
Mel Gorman
SUSE LabsSignature · 1 line
-Jirka
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Hillf,
thanks a lot for your patch!
Compared to 5.7 rc4 vanilla, we observe the following:
* Single-tenant jobs show improvement up to 15% for SPECjbb2005 and
up to 100% for NAS in low thread mode. In other words, it fixes all the problems we have reported in this thread.
* Multitenancy jobs show performance degradation up to 30% for SPECjbb2005
While it fixes problems with single-tenant jobs and with a performance at low system load, it breaks multi-tenant tasks.
We have compared it against kernel with adjust_numa_imbalance disabled [1], and both kernels perform at the same level for the single-tenant jobs, but the proposed patch is bad for the multitenancy mode. The kernel with adjust_numa_imbalance disabled is a clear winner here.
We would be very interested in what others think about disabling adjust_numa_imbalance function. The patch is bellow. It would be great to collect performance results for different scenarios to make sure the results are objective.
Thanks a lot!
Jirka
[1] Patch to test kernel with adjust_numa_imbalance disabled:
===============================================
kernel/sched/fair.c +0 -0
@@ -8907,14 +8907,6 @@ static inline long adjust_numa_imbalance(int
imbalance, int src_nr_running)
{
unsigned int imbalance_min;- /*
- * Allow a small imbalance based on a simple pair of communicating - * tasks that remain local when the source domain is almost idle. - */
- imbalance_min = 2;
- if (src_nr_running <= imbalance_min)
- return 0;
-
return imbalance; }
===============================================
Quoted 65 lines · Hillf Danton
Quoted 19 lines · On Thu, 7 May 2020 13:49:34 Phil Auld wrote:
Quoted 17 lines · On Thu, May 07, 2020 at 06:29:44PM +0200 Jirka Hladky wrote:
Hi Mel,
we are not targeting just OMP applications. We see the performance degradation also for other workloads, like SPECjbb2005 and SPECjvm2008. Even worse, it also affects a higher number of threads. For example, comparing 5.7.0-0.rc2 against 5.6 kernel, on 4 NUMA server with 2x AMD 7351 CPU, we see performance degradation 22% for 32 threads (the system has 64 CPUs in total). We observe this degradation only when we run a single SPECjbb binary. When running 4 SPECjbb binaries in parallel, there is no change in performance between 5.6 and 5.7.
That's why we are asking for the kernel tunable, which we would add to the tuned profile. We don't expect users to change this frequently but rather to set the performance profile once based on the purpose of the server.
Quoted 8 lines
Quoted 4 lines
If you could prepare a patch for us, we would be more than happy to test it extensively. Based on the results, we can then evaluate if it's the way to go. Thoughts?
I'm happy to spin up a patch once I'm sure what exactly the tuning would effect. At an initial glance I'm thinking it would be the
imbalance_minwhich is currently hardcoded to 2. But there may be something else...hrm... try to restore the old behavior by skipping task migrate in favor of the local node if there is no imbalance.
kernel/sched/fair.c +7
-9@@ -1928,18 +1928,16 @@ static void task_numa_find_cpu(struct ta src_running = env->src_stats.nr_running -1 ; dst_running = env->dst_stats.nr_running +1 ; imbalance =max (0 , dst_running - src_running); - imbalance =adjust_numa_imbalance (imbalance, src_running); + imbalance =adjust_numa_imbalance (imbalance, src_running +1 ); -/* Use idle CPU if there is no imbalance */ +/* No task migrate without imbalance */ if (!imbalance) { - maymove = true; -if (env->dst_stats.idle_cpu >=0 ) { - env->dst_cpu = env->dst_stats.idle_cpu; -task_numa_assign (env,NULL ,0 ); -return ; - } + env->best_cpu =-1 ; +return ; } - }else { + } + + do { long src_load, dst_load, load; /* * If the improvement from just moving env->p direction is better
kernel/sched/fair.c static void task_numa_find_cpu(struct ta · line 1953
Quoted 8 lines
kernel/sched/fair.c +1
-1@@ -1949,7 +1947,7 @@ static void task_numa_find_cpu(struct ta dst_load = env->dst_stats.load + load; src_load = env->src_stats.load - load; maymove = !load_too_imbalanced (src_load, dst_load, env); - } + }while (0 );for_each_cpu (cpu,cpumask_of_node (env->dst_nid)) {/* Skip this CPU if the source task cannot migrate */
--
-Jirka
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Hillf, Mel and all,
thanks for the patch! It has produced really GOOD results.
1) It has fixed performance problems with 5.7 vanilla kernel for single-tenant workload and low system load scenarios, without performance degradation for the multi-tenant tasks. It's producing the same results as the previous proof-of-concept patch where adjust_numa_imbalance function was modified to be a no-op (returning the same value of imbalance as it gets on the input).
2) We have also added Mel's netperf-cstate-small-cross-socket test to our test battery:
https://github.com/gormanm/mmtests/blob/master/configs/config-network-netperf-cstate-small-cross-socket
Mel told me that he had seen significant performance improvements with 5.7 over 5.6 for the netperf-cstate-small-cross-socket scenario.
Out of 6 different patches we have tested, your patch has performed the best for this scenario. Compared to vanilla, we see minimal performance degradation of 2.5% for the udp stream throughput and 0.6% for the tcp throughput. The testing was done on a dual-socket system with Gold 6132 CPU.
@Mel - could you please test Hillf's patch with your full testing suite? So far, it looks very promising, but I would like to check the patch thoroughly to make sure it does not hurt performance in other areas.
Thanks a lot!
Jirka
Quoted 93 lines · Hillf Danton
Hi Jirka
Quoted 5 lines · On Mon, 18 May 2020 16:52:52 +0200 Jirka Hladky wrote:
We have compared it against kernel with
adjust_numa_imbalancedisabled [1], and both kernels perform at the same level for the single-tenant jobs, but the proposed patch is bad for the multitenancy mode. The kernel withadjust_numa_imbalancedisabled is a clear winner here.Double thanks to you for the tests!
Quoted 2 lines
We would be very interested in what others think about disabling
adjust_numa_imbalancefunction. The patch is bellow. It would be greatA minute...
Quoted 2 lines
to collect performance results for different scenarios to make sure the results are objective.
I don't have another test case but a diff trying to confine the tool in question back to the hard-coded 2's field.
It's used in the first hunk below to detect imbalance before migrating a task, and a small churn of code is added at another call site when balancing idle CPUs.
Thanks
Hillfkernel/sched/fair.c +24
-18@@ -1916,20 +1916,26 @@ static void task_numa_find_cpu(struct ta* imbalance that would be overruled by the load balancer. */ if (env->dst_stats.node_type == node_has_spare) { -unsigned int imbalance; -int src_running, dst_running; +unsigned int imbalance =2 ; -/* -* Would movement cause an imbalance? Note that if src has -* more running tasks that the imbalance is ignored as the -* move improves the imbalance from the perspective of the -* CPU load balancer. -* */ - src_running = env->src_stats.nr_running -1 ; - dst_running = env->dst_stats.nr_running +1 ; - imbalance =max (0 , dst_running - src_running); - imbalance =adjust_numa_imbalance (imbalance, src_running); +//No imbalance computed without spare capacity +if (env->dst_stats.node_type != env->src_stats.node_type) + goto check_imb; + + imbalance =adjust_numa_imbalance (imbalance, + env->src_stats.nr_running); + +//Do nothing without imbalance +if (!imbalance) { + imbalance =2 ; + goto check_imb; + } + +//Migrate task if it's likely to grow balance +if (env->dst_stats.nr_running +1 < env->src_stats.nr_running) + imbalance =0 ; +check_imb:/* Use idle CPU if there is no imbalance */ if (!imbalance) { maymove = true;@@ -9011,12 +9017,13 @@ static inline void calculate_imbalance(s env->migration_type = migrate_task; env->imbalance =max_t (long,0 , (local->idle_cpus - busiest->idle_cpus) >>1 ); - } -/* Consider allowing a small imbalance between NUMA groups */ -if (env->sd->flags &SD_NUMA ) - env->imbalance =adjust_numa_imbalance (env->imbalance, - busiest->sum_nr_running); +/* Consider allowing a small imbalance between NUMA groups */ +if (env->sd->flags &SD_NUMA && + local->group_type == busiest->group_type) + env->imbalance =adjust_numa_imbalance (env->imbalance, + busiest->sum_nr_running); + }return ; } --
Signature · 1 line
-Jirka
I have an update on netperf-cstate-small-cross-socket results.
Reported performance degradation of 2.5% for the UDP stream throughput and 0.6% for the TCP throughput is for message size of 16kB. For smaller message sizes, the performance drop is higher - up to 5% for UDP throughput for a message size of 64B. See the numbers below [1]
We still think that it's acceptable given the gains in other situations (this is again compared to 5.7 vanilla) :
* solved the performance drop upto 20% with single instance SPECjbb2005 benchmark on 8 NUMA node servers (particularly on AMD EPYC Rome systems) => this performance drop was INCREASING with higher threads counts (10% for 16 threads and 20 % for 32 threads)
* solved the performance drop upto 50% for low load scenarios (SPECjvm2008 and NAS)
[1]
Hillf's patch compared to 5.7 (rc4) vanilla:
TCP throughput
Message size (B)
64 -2.6%
128 -2.3%
256 -2.6%
1024 -2.7% 2048 -2.2% 3312 -2.4% 4096 -1.1% 8192 -0.4%
16384 -0.6%
UDP throughput
64 -5.0%
128 -3.0%
256 -3.0%
1024 -3.1% 2048 -3.3% 3312 -3.5% 4096 -4.0% 8192 -3.3%
16384 -2.6%
Quoted 142 lines · Jirka Hladky
Hi Hillf, Mel and all,
thanks for the patch! It has produced really GOOD results.
1) It has fixed performance problems with 5.7 vanilla kernel for single-tenant workload and low system load scenarios, without performance degradation for the multi-tenant tasks. It's producing the same results as the previous proof-of-concept patch where
adjust_numa_imbalancefunction was modified to be a no-op (returning the same value of imbalance as it gets on the input).2) We have also added Mel's netperf-cstate-small-cross-socket test to our test battery:
https://github.com/gormanm/mmtests/blob/master/configs/config-network-netperf-cstate-small-cross-socketMel told me that he had seen significant performance improvements with 5.7 over 5.6 for the netperf-cstate-small-cross-socket scenario.
Out of 6 different patches we have tested, your patch has performed the best for this scenario. Compared to vanilla, we see minimal performance degradation of 2.5% for the udp stream throughput and 0.6% for the tcp throughput. The testing was done on a dual-socket system with Gold 6132 CPU.
@Mel - could you please test Hillf's patch with your full testing suite? So far, it looks very promising, but I would like to check the patch thoroughly to make sure it does not hurt performance in other areas.
Thanks a lot!
JirkaQuoted 83 lines · Hillf Danton
Hi Jirka
Quoted 5 lines · On Mon, 18 May 2020 16:52:52 +0200 Jirka Hladky wrote:
We have compared it against kernel with
adjust_numa_imbalancedisabled [1], and both kernels perform at the same level for the single-tenant jobs, but the proposed patch is bad for the multitenancy mode. The kernel withadjust_numa_imbalancedisabled is a clear winner here.Double thanks to you for the tests!
Quoted 2 lines
We would be very interested in what others think about disabling
adjust_numa_imbalancefunction. The patch is bellow. It would be greatA minute...
Quoted 2 lines
to collect performance results for different scenarios to make sure the results are objective.
I don't have another test case but a diff trying to confine the tool in question back to the hard-coded 2's field.
It's used in the first hunk below to detect imbalance before migrating a task, and a small churn of code is added at another call site when balancing idle CPUs.
Thanks
Hillfkernel/sched/fair.c +18
-17@@ -1916,20 +1916,26 @@ static void task_numa_find_cpu(struct ta* imbalance that would be overruled by the load balancer. */ if (env->dst_stats.node_type == node_has_spare) { -unsigned int imbalance; -int src_running, dst_running; +unsigned int imbalance =2 ; -/* -* Would movement cause an imbalance? Note that if src has -* more running tasks that the imbalance is ignored as the -* move improves the imbalance from the perspective of the -* CPU load balancer. -* */ - src_running = env->src_stats.nr_running -1 ; - dst_running = env->dst_stats.nr_running +1 ; - imbalance =max (0 , dst_running - src_running); - imbalance =adjust_numa_imbalance (imbalance, src_running); +//No imbalance computed without spare capacity +if (env->dst_stats.node_type != env->src_stats.node_type) + goto check_imb; + + imbalance =adjust_numa_imbalance (imbalance, + env->src_stats.nr_running); + +//Do nothing without imbalance +if (!imbalance) { + imbalance =2 ; + goto check_imb; + } + +//Migrate task if it's likely to grow balance +if (env->dst_stats.nr_running +1 < env->src_stats.nr_running) + imbalance =0 ; +check_imb:/* Use idle CPU if there is no imbalance */ if (!imbalance) { maymove = true;@@ -9011,12 +9017,13 @@ static inline void calculate_imbalance(s env->migration_type = migrate_task; env->imbalance =max_t (long,0 , (local->idle_cpus - busiest->idle_cpus) >>1 ); - } -/* Consider allowing a small imbalance between NUMA groups */ -if (env->sd->flags &SD_NUMA ) - env->imbalance =adjust_numa_imbalance (env->imbalance, - busiest->sum_nr_running);kernel/sched/fair.c static inline void calculate_imbalance(s
Quoted 10 lines
kernel/sched/fair.c +6
-1+/* Consider allowing a small imbalance between NUMA groups */ +if (env->sd->flags &SD_NUMA && + local->group_type == busiest->group_type) + env->imbalance =adjust_numa_imbalance (env->imbalance, + busiest->sum_nr_running); + }return ; } ----
-Jirka
Signature · 1 line
-Jirka
Quoted 11 lines · Jirka Hladky
Hi Hillf, Mel and all,
thanks for the patch! It has produced really GOOD results.
1) It has fixed performance problems with 5.7 vanilla kernel for single-tenant workload and low system load scenarios, without performance degradation for the multi-tenant tasks. It's producing the same results as the previous proof-of-concept patch where
adjust_numa_imbalancefunction was modified to be a no-op (returning the same value of imbalance as it gets on the input).
I worry that what it's doing is sort of reverts the patch but in a relatively obscure way. We already know that a side-effect of having a pair of tasks sharing cache is that the wakeup paths can be more expensive and the difference in the wakeup path exceeds the advantage of using local memory. However, I don't think the right approach long term is to keep tasks artificially apart so that a different wakeup path is used as a side-effect.
The patch also needs a changelog and better comments to follow exactly what the rationale is. Take this part
+ //No imbalance computed without spare capacity
+ if (env->dst_stats.node_type != env->src_stats.node_type)
+ goto check_imb;
I'm ignoring the coding style of c++ comments but minimally that should be fixed. More importantly, node_type can be one of node_overloaded, node_has_spare or node_fully busy and this is checking if there is a mismatch. However, it's not taking into account whether the dst_node is more or less loaded than the src and does not appear to be actually checking spare capacity like the comment suggests.
Then there is this part
+ imbalance = adjust_numa_imbalance(imbalance,
+ env->src_stats.nr_running);
+
+ //Do nothing without imbalance
+ if (!imbalance) {
+ imbalance = 2;
+ goto check_imb;
+ }
So... if there is no imbalance, assume there is an imbalance of 2, jump to a branch that will always be false and fall through to code that ignores the value of imbalance ...... it's hard to see exactly why that code flow is ideal.
Quoted 11 lines
2) We have also added Mel's netperf-cstate-small-cross-socket test to our test battery:
https://github.com/gormanm/mmtests/blob/master/configs/config-network-netperf-cstate-small-cross-socketMel told me that he had seen significant performance improvements with 5.7 over 5.6 for the netperf-cstate-small-cross-socket scenario.
Out of 6 different patches we have tested, your patch has performed the best for this scenario. Compared to vanilla, we see minimal performance degradation of 2.5% for the udp stream throughput and 0.6% for the tcp throughput.
Which implies that we are once again using remote memory for netperf and possibly getting some interference from NUMA balancing hinting faults.
Quoted 8 lines
The testing was done on a dual-socket system
with Gold 6132 CPU.@Mel - could you please test Hillf's patch with your full testing suite? So far, it looks very promising, but I would like to check the patch thoroughly to make sure it does not hurt performance in other areas.
I can't at the moment due to a backlog on my test grid which isn't helped that by the fact I lost two days development time due to a thrashed work machine. That aside, I'm finding it hard to see exactly why the patch is suitable. What I've seen so far is that there are two outstanding problems after the rewritten load balancer and the reconcilation between load balancer and NUMA balancer.
The first is that 57abff067a08 ("sched/fair: Rework find_idlest_group()") has shown up some problems with LKP reporting it here
https://lore.kernel.org/lkml/20200514141526.GA30976@xsang-OptiPlex-9020/
but I've also seen numerous workloads internally bisect to the same commit. This patch is meant to ensure that finding the busiest group uses similar logic to finding the idlest group but something is hiding in there.
The second is that waking two tasks sharing tasks can be more expensive than waking two remote tasks but it's preferable to fix the wakeup logic than keep related tasks on separate caches just because it happens to generate good numbers in some cases.
I feel that it makes sense to try and get both of those issues resolved before making adjust_numa_imbalance a tunable or reverting it but I really think it makes sense for communicating tasks to be sharing cache if possible unless a node is overloaded or limited by memory bandwidth.
Signature · 2 lines
Mel Gorman SUSE Labs
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 8 lines · Hillf Danton
Quoted 7 lines
I'm ignoring the coding style of c++ comments but minimally that should be fixed. More importantly,
node_typecan be one ofnode_overloaded,node_has_spareornode_fullybusy and this is checking if there is a mismatch. However, it's not taking into account whether thedst_nodeis more or less loaded than the src and does not appear to be actually checking spare capacity like the comment suggests.
Quoted 4 lines
Type other than
node_has_spareis not considered because node is not possible to be so idle that two communicating tasks would better "stay local."
You hardcode an imbalance of 2 at the start without computing any imbalance. Then if the source is fully_busy or overloaded while the dst is idle, a task can move but that is based on an imaginary hard-coded imbalance of 2. Finally, it appears that that the load balancer and NUMA balancer are again using separate logic again when one part of the objective of the series is that the load balancer and NUMA balancer would not override each other. As the imbalance is never computed, the patch can create one which the load balancer then overrides. What am I missing?
That said, it does make a certain amount of sense that if the dst has spare capacity and the src is fully busy or overloaded then allow the migration regardless of imbalance but that would be a different patch -- something like this but I didn't think too deeply or test it.
kernel/sched/fair.c +18 -10
@@ -1916,19 +1916,27 @@ static void task_numa_find_cpu(struct task_numa_env *env,* imbalance that would be overruled by the load balancer. */ if (env->dst_stats.node_type == node_has_spare) { -unsigned int imbalance; +unsigned int imbalance =0 ;int src_running, dst_running;/* -* Would movement cause an imbalance? Note that if src has -* more running tasks that the imbalance is ignored as the -* move improves the imbalance from the perspective of the -* CPU load balancer. -* */ - src_running = env->src_stats.nr_running -1 ; - dst_running = env->dst_stats.nr_running +1 ; - imbalance =max (0 , dst_running - src_running); - imbalance =adjust_numa_imbalance (imbalance, src_running); +* Check the imbalance if both src and dst have spare +* capacity. If src is fully_busy or overloaded then +* allow the task to move as it's both improving locality +* and reducing an imbalance. +*/ +if (env->src_stats.node_type == node_has_spare) { +/* +* Would movement cause an imbalance? Note that if src +* has more running tasks that the imbalance is +* ignored as the move improves the imbalance from the +* perspective of the CPU load balancer. +*/ + src_running = env->src_stats.nr_running -1 ; + dst_running = env->dst_stats.nr_running +1 ; + imbalance =max (0 , dst_running - src_running); + imbalance =adjust_numa_imbalance (imbalance, src_running); + }/* Use idle CPU if there is no imbalance */ if (!imbalance) {
Quoted 11 lines
Quoted 11 lines
Then there is this part
(quoted diff) +8
-0+ imbalance = adjust_numa_imbalance(imbalance, + env->src_stats.nr_running); + + //Do nothing without imbalance + if (!imbalance) { + imbalance = 2; + goto check_imb; + }
Quoted 6 lines
Quoted 5 lines
So... if there is no imbalance, assume there is an imbalance of 2, jump to a branch that will always be false and fall through to code that ignores the value of imbalance ...... it's hard to see exactly why that code flow is ideal.
With spare capacity be certain, then lets see how idle the node is.
But you no longer check how idle it is or what it's relative idleness of the destination is relative to the source. adjust_numa_imbalance does not calculate an imbalance, it only decides whether it should be ignored.
Quoted 5 lines
And I
can not do that without the tool you created, He bless you. Although not sure they're two tasks talking to each other. This is another topic in the coming days.
There is insufficient context in this path to determine if two tasks are communicating. In some cases it may be inferred from last_wakee but that only works for two tasks, it doesn't work for 1:n waker:wakees such as might happen with a producer/consumer pattern. I guess you could record the time that tasks migrated cross-node due to a wakeup and avoid migrating those tasks for a period of time by either NUMA or load balancer but that could cause transient overloads of a domain if there are a lot of wakeups (e.g. hackbench). It's a change that would need to be done on both the NUMA and load balancer to force the migration even if there are wakeup relationships if the domain is fully busy or overloaded.
Signature · 2 lines
Mel Gorman SUSE Labs
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 16 lines · Hillf Danton
Quoted 14 lines · On Thu, 21 May 2020 17:04:08 +0100 Mel Gorman wrote:
Quoted 12 lines · Hillf Danton
Quoted 7 lines
I'm ignoring the coding style of c++ comments but minimally that should be fixed. More importantly,
node_typecan be one ofnode_overloaded,node_has_spareornode_fullybusy and this is checking if there is a mismatch. However, it's not taking into account whether thedst_nodeis more or less loaded than the src and does not appear to be actually checking spare capacity like the comment suggests.Type other than
node_has_spareis not considered because node is not possible to be so idle that two communicating tasks would better "stay local."
Quoted 5 lines
Quoted 2 lines
You hardcode an imbalance of 2 at the start without computing any imbalance.
Same result comes up if it's a bool.
Then the magic number is simply confusing. The patch needs to be a lot clearer about what the intent is if my patch that adds a "if (env->src_stats.node_type == node_has_spare)" check is not what you were aiming for.
Quoted 8 lines
Quoted 3 lines
Then if the source is
fully_busyor overloaded while the
dst is idle, a task can move but that is based on an imaginary hard-coded imbalance of 2.This is the potpit I walked around by checking spare capacity first. As for overloaded, I see it as a signal that a glitch is not idle somewhere else, and I prefer to push it in to ICU before it's too late.
The domain could be overloaded simply due to CPU bindings. I cannot parse the ICU comment (Intensive Care Unit?!?) :(
Quoted 7 lines
Quoted 4 lines
Finally, it appears that that the load balancer and
NUMA balancer are again using separate logic again when one part of the objective of the series is that the load balancer and NUMA balancer would not override each other.This explains 80% of why it is a choppy road ahead.
And I don't think we should go back to the load balancer and NUMA balancer taking different actions. It ends up doing useless CPU migrations and can lead to higher NUMA scanning activity. It's partially why I changed task_numa_compare to prefer swapping with tasks that move to their preferred node when using an idle CPU would cause an imbalance.
Quoted 6 lines
Quoted 2 lines
As the imbalance is never computed, the patch
can create one which the load balancer then overrides. What am I missing?LB would give a green light if things move in the direction in favor of cutting imb.
Load balancing primarily cares about balancing the number of idle CPUs between domains when there is spare capacity. While it tries to avoid balancing by moving tasks from their preferred node, it will do so if there are no other candidates.
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
<SNIP>
Then there is this part
(quoted diff) +8
-0+ imbalance = adjust_numa_imbalance(imbalance, + env->src_stats.nr_running); + + //Do nothing without imbalance + if (!imbalance) { + imbalance = 2; + goto check_imb; + }So... if there is no imbalance, assume there is an imbalance of 2, jump to a branch that will always be false and fall through to code that ignores the value of imbalance ...... it's hard to see exactly why that code flow is ideal.
Quoted 9 lines
Quoted 6 lines
Quoted 1 line
With spare capacity be certain, then lets see how idle the node is.
But you no longer check how idle it is or what it's relative idleness of the destination is relative to the source.
adjust_numa_imbalancedoes not calculate an imbalance, it only decides whether it should be ignored.Then the idle CPU is no longer so tempting.
True. While it's prefectly possible to ignore imbalance and use an idle CPU if it exists, the load balancer will simply override it later and we go back to the NUMA balancer and load balancer fighting each other with the NUMA balancer retrying migrations based on p->numa_migrate_retry. Whatever logic is used to allow imbalances (be they based on communicating tasks or preferred locality), it needs to be the same in both balancers.
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 12 lines · Jirka Hladky
Hi Mel,
we are not targeting just OMP applications. We see the performance degradation also for other workloads, like SPECjbb2005 and SPECjvm2008. Even worse, it also affects a higher number of threads. For example, comparing 5.7.0-0.rc2 against 5.6 kernel, on 4 NUMA server with 2x AMD 7351 CPU, we see performance degradation 22% for 32 threads (the system has 64 CPUs in total). We observe this degradation only when we run a single SPECjbb binary. When running 4 SPECjbb binaries in parallel, there is no change in performance between 5.6 and 5.7.
Minimally I suggest confirming that it's really due to adjust_numa_imbalance() by making the function a no-op and retesting. I have found odd artifacts with it but I'm unsure how to proceed without causing problems elsehwere.
For example, netperf on localhost in some cases reported a regression when the client and server were running on the same node. The problem appears to be that netserver completes its work faster when running local and goes idle more regularly. The cost of going idle and waking up builds up and a lower throughput is reported but I'm not sure if gaming an artifact like that is a good idea.
Quoted 9 lines
That's why we are asking for the kernel tunable, which we would add to the tuned profile. We don't expect users to change this frequently but rather to set the performance profile once based on the purpose of the server.
If you could prepare a patch for us, we would be more than happy to test it extensively. Based on the results, we can then evaluate if it's the way to go. Thoughts?
I would suggest simply disabling that function first to ensure that is really what is causing problems for you.
Signature · 2 lines
Mel Gorman SUSE Labs
Hi Mel,
thanks for hints! We will try it.
@Phil - could you please prepare a kernel build for me to test?
Thank you!
Jirka
Quoted 44 lines · Mel Gorman
Quoted 12 lines · Jirka Hladky
Hi Mel,
we are not targeting just OMP applications. We see the performance degradation also for other workloads, like SPECjbb2005 and SPECjvm2008. Even worse, it also affects a higher number of threads. For example, comparing 5.7.0-0.rc2 against 5.6 kernel, on 4 NUMA server with 2x AMD 7351 CPU, we see performance degradation 22% for 32 threads (the system has 64 CPUs in total). We observe this degradation only when we run a single SPECjbb binary. When running 4 SPECjbb binaries in parallel, there is no change in performance between 5.6 and 5.7.
Minimally I suggest confirming that it's really due to
adjust_numa_imbalance()by making the function a no-op and retesting. I have found odd artifacts with it but I'm unsure how to proceed without causing problems elsehwere.For example, netperf on localhost in some cases reported a regression when the client and server were running on the same node. The problem appears to be that netserver completes its work faster when running local and goes idle more regularly. The cost of going idle and waking up builds up and a lower throughput is reported but I'm not sure if gaming an artifact like that is a good idea.
Quoted 9 lines
That's why we are asking for the kernel tunable, which we would add to the tuned profile. We don't expect users to change this frequently but rather to set the performance profile once based on the purpose of the server.
If you could prepare a patch for us, we would be more than happy to test it extensively. Based on the results, we can then evaluate if it's the way to go. Thoughts?
I would suggest simply disabling that function first to ensure that is really what is causing problems for you.
--
Mel Gorman
SUSE Labs
Signature · 1 line
-Jirka
A message that is not in the archive (sent off-list or not yet received): CAE4VaGC_v6On-YvqdTwAWu3Mq4ofiV0pLov-QpV+QHr_SJr+Rw@mail.gmail.com
Hi Mel,
we have tried the kernel with adjust_numa_imbalance() crippled to just return the imbalance it's given.
It has solved all the performance problems I have reported. Performance is the same as with 5.6 kernel (before the patch was applied).
* solved the performance drop upto 20% with single instance SPECjbb2005 benchmark on 8 NUMA node servers (particularly on AMD EPYC Rome systems) => this performance drop was INCREASING with higher threads counts (10% for 16 threads and 20 % for 32 threads)
* solved the performance drop for low load scenarios (SPECjvm2008 and NAS)
Any suggestions on how to proceed? One approach is to turn "imbalance_min" into the kernel tunable. Any other ideas?
Thanks a lot!
Jirka
Quoted 59 lines · Jirka Hladky
Hi Mel,
thanks for hints! We will try it.
@Phil - could you please prepare a kernel build for me to test?
Thank you!
JirkaQuoted 37 lines · Mel Gorman
Quoted 12 lines · Jirka Hladky
Hi Mel,
we are not targeting just OMP applications. We see the performance degradation also for other workloads, like SPECjbb2005 and SPECjvm2008. Even worse, it also affects a higher number of threads. For example, comparing 5.7.0-0.rc2 against 5.6 kernel, on 4 NUMA server with 2x AMD 7351 CPU, we see performance degradation 22% for 32 threads (the system has 64 CPUs in total). We observe this degradation only when we run a single SPECjbb binary. When running 4 SPECjbb binaries in parallel, there is no change in performance between 5.6 and 5.7.
Minimally I suggest confirming that it's really due to
adjust_numa_imbalance()by making the function a no-op and retesting. I have found odd artifacts with it but I'm unsure how to proceed without causing problems elsehwere.For example, netperf on localhost in some cases reported a regression when the client and server were running on the same node. The problem appears to be that netserver completes its work faster when running local and goes idle more regularly. The cost of going idle and waking up builds up and a lower throughput is reported but I'm not sure if gaming an artifact like that is a good idea.
Quoted 9 lines
That's why we are asking for the kernel tunable, which we would add to the tuned profile. We don't expect users to change this frequently but rather to set the performance profile once based on the purpose of the server.
If you could prepare a patch for us, we would be more than happy to test it extensively. Based on the results, we can then evaluate if it's the way to go. Thoughts?
Quoted 7 lines
I would suggest simply disabling that function first to ensure that is really what is causing problems for you.
--
Mel Gorman
SUSE Labs--
-Jirka
Signature · 1 line
-Jirka
Quoted 15 lines · Jirka Hladky
Hi Mel,
we have tried the kernel with
adjust_numa_imbalance()crippled to just return the imbalance it's given.It has solved all the performance problems I have reported. Performance is the same as with 5.6 kernel (before the patch was applied).
* solved the performance drop upto 20% with single instance SPECjbb2005 benchmark on 8 NUMA node servers (particularly on AMD EPYC Rome systems) => this performance drop was INCREASING with higher threads counts (10% for 16 threads and 20 % for 32 threads)
* solved the performance drop for low load scenarios (SPECjvm2008 and NAS)
Quoted 5 lines
Any suggestions on how to proceed? One approach is to turn "
imbalance_min" into the kernel tunable. Any other ideas?
Complete shot in the dark but restore adjust_numa_imbalance() and try this
kernel/sched/core.c +1 -1
@@ -2393,7 +2393,7 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)struct rq_flags rf;#if defined(CONFIG_SMP ) -if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { +if (sched_feat (TTWU_QUEUE )) {sched_clock_cpu (cpu);/* Sync clocks across CPUs */ ttwu_queue_remote (p, cpu, wake_flags);return ;
Signature · 2 lines
Mel Gorman SUSE Labs
Thank you, Mel!
I think I have to make sure we cover the scenario you have targeted when developing adjust_numa_imbalance:
=======================================================================
https://github.com/torvalds/linux/blob/4f8a3cc1183c442daee6cc65360e3385021131e4/kernel/sched/fair.c#L8910
/*
* Allow a small imbalance based on a simple pair of communicating
* tasks that remain local when the source domain is almost idle. */
=======================================================================
Could you point me to a benchmark for this scenario? I have checked
https://github.com/gormanm/mmtests
and we use lots of the same benchmarks but I'm not sure if we cover this particular scenario.
Jirka
Quoted 44 lines · Mel Gorman
Quoted 15 lines · Jirka Hladky
Hi Mel,
we have tried the kernel with
adjust_numa_imbalance()crippled to just return the imbalance it's given.It has solved all the performance problems I have reported. Performance is the same as with 5.6 kernel (before the patch was applied).
* solved the performance drop upto 20% with single instance SPECjbb2005 benchmark on 8 NUMA node servers (particularly on AMD EPYC Rome systems) => this performance drop was INCREASING with higher threads counts (10% for 16 threads and 20 % for 32 threads)
* solved the performance drop for low load scenarios (SPECjvm2008 and NAS)Quoted 5 lines
Any suggestions on how to proceed? One approach is to turn "
imbalance_min" into the kernel tunable. Any other ideas?Complete shot in the dark but restore
adjust_numa_imbalance()and try thiskernel/sched/core.c +1
-2@@ -2393,7 +2393,7 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)struct rq_flags rf;#if defined(CONFIG_SMP ) -if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { +if (sched_feat (TTWU_QUEUE )) {sched_clock_cpu (cpu);/* Sync clocks across CPUs */ ttwu_queue_remote (p, cpu, wake_flags);return ; --Mel Gorman
SUSE Labs
Signature · 1 line
-Jirka
Quoted 14 lines · Jirka Hladky
Thank you, Mel!
I think I have to make sure we cover the scenario you have targeted when developing
adjust_numa_imbalance:=======================================================================
https://github.com/torvalds/linux/blob/4f8a3cc1183c442daee6cc65360e3385021131e4/kernel/sched/fair.c#L8910/*
* Allow a small imbalance based on a simple pair of communicating
* tasks that remain local when the source domain is almost idle. */
=======================================================================
Quoted 5 lines
Could you point me to a benchmark for this scenario? I have checked
https://github.com/gormanm/mmtests
and we use lots of the same benchmarks but I'm not sure if we cover this particular scenario.
The NUMA imbalance part showed up as part of the general effort to reconcile NUMA balancing with Load balancing. It's been known for years that the two balancers disagreed to the extent that NUMA balancing retries migrations multiple times just to keep things local leading to excessive migrations. The full battery of tests that were used when I was trying to reconcile the balancers and later working on Vincent's version is as follows
scheduler-unbound
scheduler-forkintensive
scheduler-perfpipe
scheduler-perfpipe-cpufreq
scheduler-schbench
db-pgbench-timed-ro-small-xfs
hpc-nas-c-class-mpi-full-xfs
hpc-nas-c-class-mpi-half-xfs
hpc-nas-c-class-omp-full
hpc-nas-c-class-omp-half
hpc-nas-d-class-mpi-full-xfs
hpc-nas-d-class-mpi-half-xfs
hpc-nas-d-class-omp-full
hpc-nas-d-class-omp-half
io-dbench4-async-ext4
io-dbench4-async-xfs
jvm-specjbb2005-multi
jvm-specjbb2005-single
network-netperf-cstate
network-netperf-rr-cstate
network-netperf-rr-unbound
network-netperf-unbound
network-tbench
numa-autonumabench
workload-kerndevel-xfs
workload-shellscripts-xfs
Where there is -ext4 or -xfs, just remove the filesystem to get the base configuration. i.e. io-dbench4-async-ext4 basic configuration is io-dbench4-async. Both filesystems are sometimes tested because they interact differently with the scheduler due to ext4 using a journal thread and xfs using workqueues.
The imbalance one is most obvious with network-netperf-unbound running on localhost. If the client/server are on separate nodes, it's obvious from mpstat that two nodes are busy and it's migrating quite a bit. The second effect is that NUMA balancing is active, trapping hinting faults and migrating pages.
The biggest problem I have right now is that the wakeup path between tasks that are local is slower than doing a remote wakeup via wake_list that potentially sends an IPI which is ridiculous. The slower wakeup manifests as a loss of throughput for netperf even though all the accesses are local. At least that's what I'm looking at whenever I get the chance.
Signature · 2 lines
Mel Gorman SUSE Labs
A message that is not in the archive (sent off-list or not yet received): CAE4VaGCGUFOAZ+YHDnmeJ95o4W0j04Yb7EWnf8a43caUQs_WuQ@mail.gmail.com
Quoted 11 lines · Jirka Hladky
Thank you, Mel!
We are using netperf as well, but AFAIK it's running on two different hosts. I will check with colleagues, if they can
add network-netperf-unbound run on the localhost.Is this the right config?
https://github.com/gormanm/mmtests/blob/345f82bee77cbf09ba57f470a1cfc1ae413c97df/bin/generate-generic-configs
sed -e 's/NETPERF_BUFFER_SIZES=.*/NETPERF_BUFFER_SIZES=64/' config-network-netperf-unbound > config-network-netperf-unbound-small
That's one I was using at the moment to have a quick test after the reconciliation series was completed. It has since changed to config-network-netperf-cstate-small-cross-socket to limit cstates, bind the client and server to two local CPUs and using one buffer size. It was necessary to get an ftrace function graph of the wakeup path that was readable and not too noisy due to migrations, cpuidle exit costs etc.
Signature · 2 lines
Mel Gorman SUSE Labs
Thanks!
Do you have a link? I cannot find it on github
(https://github.com/gormanm/mmtests, searched for
config-network-netperf-cstate-small-cross-socket)
Quoted 25 lines · Mel Gorman
Quoted 11 lines · Jirka Hladky
Thank you, Mel!
We are using netperf as well, but AFAIK it's running on two different hosts. I will check with colleagues, if they can
add network-netperf-unbound run on the localhost.Is this the right config?
https://github.com/gormanm/mmtests/blob/345f82bee77cbf09ba57f470a1cfc1ae413c97df/bin/generate-generic-configs
sed -e 's/NETPERF_BUFFER_SIZES=.*/NETPERF_BUFFER_SIZES=64/' config-network-netperf-unbound > config-network-netperf-unbound-smallThat's one I was using at the moment to have a quick test after the reconciliation series was completed. It has since changed to config-network-netperf-cstate-small-cross-socket to limit cstates, bind the client and server to two local CPUs and using one buffer size. It was necessary to get an ftrace function graph of the wakeup path that was readable and not too noisy due to migrations, cpuidle exit costs etc.
--
Mel Gorman
SUSE Labs
Signature · 1 line
-Jirka
Quoted 6 lines · Jirka Hladky
Thanks!
Do you have a link? I cannot find it on github
(https://github.com/gormanm/mmtests, searched for
config-network-netperf-cstate-small-cross-socket)
Signature · 2 lines
Mel Gorman SUSE Labs
THANK YOU!
Quoted 15 lines · Mel Gorman
Quoted 6 lines · Jirka Hladky
Thanks!
Do you have a link? I cannot find it on github
(https://github.com/gormanm/mmtests, searched for
config-network-netperf-cstate-small-cross-socket)--
Mel Gorman
SUSE Labs
Signature · 1 line
-Jirka
Peter Zijlstra
maintainerkernel/sched/core.c static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags) · line 2396
Quoted 13 lines · Mel Gorman
Complete shot in the dark but restore
adjust_numa_imbalance()and try thiskernel/sched/core.c +1
-1@@ -2393,7 +2393,7 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)struct rq_flags rf;#if defined(CONFIG_SMP ) -if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { +if (sched_feat (TTWU_QUEUE )) {
just saying that this has the risk of regressing other workloads, see:
518cd6234178 ("sched: Only queue remote wakeups when crossing cache boundaries")
Quoted 7 lines
kernel/sched/core.c +0
-0sched_clock_cpu (cpu);/* Sync clocks across CPUs */ ttwu_queue_remote (p, cpu, wake_flags);return ;Signature · 3 lines
-- Mel Gorman SUSE Labs
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Mel Gorman
Complete shot in the dark but restore
adjust_numa_imbalance()and try thiskernel/sched/core.c +0
-0@@ -2393,7 +2393,7 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)struct rq_flags rf;
Quoted 8 lines
kernel/sched/core.c static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags) · line 2396
Quoted 3 lines
kernel/sched/core.c +1
-1#if defined(CONFIG_SMP ) -if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { +if (sched_feat (TTWU_QUEUE )) {just saying that this has the risk of regressing other workloads, see:
518cd6234178("sched: Only queue remote wakeups when crossing cache boundaries")
Sure, I didn't really think that this was appropriate but it was the best I had at the time. This can be a lot more targetted but it took me a while to put together a picture as it was difficult for me to analyse. I was going to post this as a seperate RFC but given this thread, what do you think of this?
---8<---
sched: Wake cache-local tasks via wake_list if wakee CPU is polling
There are two separate method for waking a task from idle, one for tasks running on CPUs that share a cache and one for when the caches are separate. The methods can loosely be called local and remote even though this is not directly related to NUMA and instead is due to the expected cost of accessing data that is cache-hot on another CPU. The "local" costs are expected to be relatively cheap as they share the LLC in comparison to a remote IPI that is potentially required when using the "remote" wakeup. The problem is that the local wakeup is not always cheaper and it appears to have degraded even further around the 4.19 mark.
There appears to be a couple of reasons why it can be slower.
The first is specific to pairs of tasks where one or both rapidly enters idle. For example, on netperf UDP_STREAM, the client sends a bunch of packets, wakes the server, the server processes some packets and goes back to sleep. There is a relationship between the tasks but it's not strictly synchronous. The timing is different if the client/server are on separate NUMA nodes and netserver is more likely to enter idle (measured as server entering idle 10% more times when tasks are local vs remote but machine-specific). However, the wakeups are so rapid that the wakeup happens while the server is descheduling. That forces the waker to spin on smp_cond_load_acquire for longer. In this case, it can be cheaper to add the task to the rq->wake_list even if that potentially requires an IPI.
The second is that the local wakeup path is simply not always that fast. Using ftrace, the cost of the locks, update_rq_clock and ttwu_do_activate was measured as roughly 4.5 microseconds. While it's a single instance, the cost of the "remote" wakeup for try_to_wake_up was roughly 2.5 microseconds versus 6.2 microseconds for the "fast" local wakeup. When there are tens of thousands of wakeups per second, these costs accumulate and manifest as a throughput regression in netperf UDP_STREAM.
The essential difference in costs comes down to whether the CPU is fully idle, a task is descheduling or polling in poll_idle(). This patch special cases ttwu_queue() to use the "remote" method if the CPUs task is polling as it's generally cheaper to use the wake_list in that case than the local method because an IPI should not be required. As it is race-prone, a reschedule IPI may still be sent but in that case the local wakeup would also have to send a reschedule IPI so it should be harmless.
The benefit is not universal as it requires the specific pattern of wakeups that happen when the wakee is likely to be descheduling. netperf UDP_STREAM is one of the more obvious examples and it is more obvious since the load balancer was reconciled and the client and server are more likely to be running on the same NUMA node which is why it was missed for so long.
netperf-udp
5.7.0-rc5 5.7.0-rc5
vanilla wakelist-v1r1
Hmean send-64 211.25 ( 0.00%) 228.61 * 8.22%*
Hmean send-128 413.49 ( 0.00%) 458.00 * 10.77%*
Hmean send-256 786.31 ( 0.00%) 879.30 * 11.83%*
Hmean send-1024 3055.75 ( 0.00%) 3429.45 * 12.23%*
Hmean send-2048 6052.79 ( 0.00%) 6878.99 * 13.65%*
Hmean send-3312 9208.09 ( 0.00%) 10832.19 * 17.64%*
Hmean send-4096 11268.45 ( 0.00%) 12968.39 * 15.09%*
Hmean send-8192 17943.12 ( 0.00%) 19886.07 * 10.83%*
Hmean send-16384 29732.94 ( 0.00%) 32297.44 * 8.63%*
Hmean recv-64 211.25 ( 0.00%) 228.61 * 8.22%*
Hmean recv-128 413.49 ( 0.00%) 458.00 * 10.77%*
Hmean recv-256 786.31 ( 0.00%) 879.30 * 11.83%*
Hmean recv-1024 3055.75 ( 0.00%) 3429.45 * 12.23%*
Hmean recv-2048 6052.79 ( 0.00%) 6878.99 * 13.65%*
Hmean recv-3312 9208.09 ( 0.00%) 10832.19 * 17.64%*
Hmean recv-4096 11268.45 ( 0.00%) 12968.39 * 15.09%*
Hmean recv-8192 17943.12 ( 0.00%) 19886.06 * 10.83%*
Hmean recv-16384 29731.75 ( 0.00%) 32296.08 * 8.62%*It's less obvious on something like TCP_STREAM as there is a stricter relationship between the client and server but with the patch, it's much less variable
netperf-tcp
5.7.0-rc5 5.7.0-rc5
vanilla wakelist-v1r1
Hmean 64 769.28 ( 0.00%) 779.96 * 1.39%*
Hmean 128 1497.54 ( 0.00%) 1509.33 * 0.79%*
Hmean 256 2806.00 ( 0.00%) 2787.59 ( -0.66%)
Hmean 1024 9700.80 ( 0.00%) 9720.02 ( 0.20%)
Hmean 2048 17071.28 ( 0.00%) 16968.92 ( -0.60%)
Hmean 3312 22976.20 ( 0.00%) 23012.80 ( 0.16%)
Hmean 4096 25885.74 ( 0.00%) 26152.49 * 1.03%*
Hmean 8192 34014.83 ( 0.00%) 33878.67 ( -0.40%)
Hmean 16384 39482.14 ( 0.00%) 40307.22 ( 2.09%)
Stddev 64 2.60 ( 0.00%) 0.17 ( 93.31%)
Stddev 128 3.69 ( 0.00%) 1.52 ( 58.97%)
Stddev 256 31.06 ( 0.00%) 14.76 ( 52.47%)
Stddev 1024 96.50 ( 0.00%) 46.43 ( 51.89%)
Stddev 2048 115.98 ( 0.00%) 62.59 ( 46.03%)
Stddev 3312 293.28 ( 0.00%) 41.92 ( 85.71%)
Stddev 4096 173.45 ( 0.00%) 123.19 ( 28.98%)
Stddev 8192 783.59 ( 0.00%) 129.62 ( 83.46%)
Stddev 16384 1012.37 ( 0.00%) 252.08 ( 75.10%)--- kernel/sched/core.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
kernel/sched/core.c +20 -1
@@ -2380,13 +2380,32 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)struct rq_flags rf;#if defined(CONFIG_SMP ) -if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { +if (sched_feat (TTWU_QUEUE )) { +/* +* A remote wakeup is often expensive as can require +* an IPI and the wakeup path is slow. However, in +* the specific case where the target CPU is idle +* and polling, the CPU is active and rapidly checking +* if a reschedule is needed. In this case, the idle +* task just needs to be marked for resched and p +* will rapidly be requeued which is less expensive +* than the direct wakeup path. +*/ +if (cpus_share_cache (smp_processor_id (), cpu)) { +struct thread_info *ti =task_thread_info (p); + typeof(ti->flags) val = READ_ONCE(ti->flags); + +if (val & _TIF_POLLING_NRFLAG) + goto activate; + } +sched_clock_cpu (cpu);/* Sync clocks across CPUs */ ttwu_queue_remote (p, cpu, wake_flags);return ; }#endif +activate:rq_lock (rq, &rf);update_rq_clock (rq);ttwu_do_activate (rq, p, wake_flags, &rf);
Peter Zijlstra
maintainerQuoted 4 lines · Mel Gorman
However, the wakeups are so rapid that the wakeup
happens while the server is descheduling. That forces the waker to spin onsmp_cond_load_acquirefor longer. In this case, it can be cheaper to add the task to therq->wake_listeven if that potentially requires an IPI.
Right, I think Rik ran into that as well at some point. He wanted to make ->on_cpu do a hand-off, but simply queueing the wakeup on the prev cpu (which is currently in the middle of schedule()) should be an easier proposition.
Maybe something like this untested thing... could explode most mighty, didn't thing too hard.
---
kernel/sched/core.c +24 -21
@@ -2312,7 +2312,7 @@ static void wake_csd_func(void *info)sched_ttwu_pending (); } -static void ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) +static void __ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) {struct rq *rq =cpu_rq (cpu);
@@ -2354,6 +2354,17 @@ bool cpus_share_cache(int this_cpu, int that_cpu) {return per_cpu (sd_llc_id, this_cpu) ==per_cpu (sd_llc_id, that_cpu); } + +static bool ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) +{ +if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { +sched_clock_cpu (cpu);/* Sync clocks across CPUs */ +__ttwu_queue_remote (p, cpu, wake_flags); +return true; + } + +return false; +}#endif /* CONFIG_SMP */ static void ttwu_queue (struct task_struct *p,int cpu,int wake_flags)
@@ -2362,11 +2373,8 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)struct rq_flags rf;#if defined(CONFIG_SMP ) -if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { -sched_clock_cpu (cpu);/* Sync clocks across CPUs */ -ttwu_queue_remote (p, cpu, wake_flags); +if (ttwu_queue_remote (p, cpu, wake_flags))return ; - }#endif rq_lock (rq, &rf);
@@ -2550,7 +2558,15 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)if (p->on_rq &&ttwu_remote (p, wake_flags)) goto unlock; +if (p->in_iowait) { +delayacct_blkio_end (p); +atomic_dec (&task_rq (p)->nr_iowait); + } +#ifdef CONFIG_SMP + p->sched_contributes_to_load = !!task_contributes_to_load (p); + p->state =TASK_WAKING ; +/* * Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be * possible to, falsely, observe p->on_cpu == 0.
@@ -2581,15 +2597,10 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)* This ensures that tasks getting woken will be fully ordered against * their previous state and preserve Program Order. */ -smp_cond_load_acquire (&p->on_cpu, !VAL ); - - p->sched_contributes_to_load = !!task_contributes_to_load (p); - p->state =TASK_WAKING ; +if (READ_ONCE (p->on_cpu) &&__ttwu_queue_remote (p, cpu, wake_flags)) + goto unlock; -if (p->in_iowait) { -delayacct_blkio_end (p); -atomic_dec (&task_rq (p)->nr_iowait); - } +smp_cond_load_acquire (&p->on_cpu, !VAL ); cpu =select_task_rq (p, p->wake_cpu,SD_BALANCE_WAKE , wake_flags);if (task_cpu (p) != cpu) {
@@ -2597,14 +2608,6 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)psi_ttwu_dequeue (p); set_task_cpu(p, cpu); } - -#else /* CONFIG_SMP */ - -if (p->in_iowait) { -delayacct_blkio_end (p); -atomic_dec (&task_rq (p)->nr_iowait); - } -#endif /* CONFIG_SMP */ ttwu_queue (p, cpu, wake_flags);
Quoted 85 lines · Peter Zijlstra
Quoted 4 lines · Mel Gorman
However, the wakeups are so rapid that the wakeup
happens while the server is descheduling. That forces the waker to spin onsmp_cond_load_acquirefor longer. In this case, it can be cheaper to add the task to therq->wake_listeven if that potentially requires an IPI.Right, I think Rik ran into that as well at some point. He wanted to make ->on_cpu do a hand-off, but simply queueing the wakeup on the prev cpu (which is currently in the middle of
schedule()) should be an easier proposition.Maybe something like this untested thing... could explode most mighty, didn't thing too hard.
---
kernel/sched/core.c +22
-9@@ -2312,7 +2312,7 @@ static void wake_csd_func(void *info)sched_ttwu_pending (); } -static void ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) +static void __ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) {struct rq *rq =cpu_rq (cpu);@@ -2354,6 +2354,17 @@ bool cpus_share_cache(int this_cpu, int that_cpu) {return per_cpu (sd_llc_id, this_cpu) ==per_cpu (sd_llc_id, that_cpu); } + +static bool ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) +{ +if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { +sched_clock_cpu (cpu);/* Sync clocks across CPUs */ +__ttwu_queue_remote (p, cpu, wake_flags); +return true; + } + +return false; +}#endif /* CONFIG_SMP */ static void ttwu_queue (struct task_struct *p,int cpu,int wake_flags)@@ -2362,11 +2373,8 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)struct rq_flags rf;#if defined(CONFIG_SMP ) -if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { -sched_clock_cpu (cpu);/* Sync clocks across CPUs */ -ttwu_queue_remote (p, cpu, wake_flags); +if (ttwu_queue_remote (p, cpu, wake_flags))return ; - }#endif rq_lock (rq, &rf);@@ -2550,7 +2558,15 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)if (p->on_rq &&ttwu_remote (p, wake_flags)) goto unlock; +if (p->in_iowait) { +delayacct_blkio_end (p); +atomic_dec (&task_rq (p)->nr_iowait); + } +#ifdef CONFIG_SMP + p->sched_contributes_to_load = !!task_contributes_to_load (p); + p->state =TASK_WAKING ; +/* * Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be * possible to, falsely, observe p->on_cpu == 0. @@ -2581,15 +2597,10 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)* This ensures that tasks getting woken will be fully ordered against * their previous state and preserve Program Order. */ -smp_cond_load_acquire (&p->on_cpu, !VAL ); - - p->sched_contributes_to_load = !!task_contributes_to_load (p); - p->state =TASK_WAKING ; +if (READ_ONCE (p->on_cpu) &&__ttwu_queue_remote (p, cpu, wake_flags))
kernel/sched/core.c try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) · line 2606
Quoted 10 lines
kernel/sched/core.c +2
-4+ goto unlock; -if (p->in_iowait) { -delayacct_blkio_end (p); -atomic_dec (&task_rq (p)->nr_iowait); - } +smp_cond_load_acquire (&p->on_cpu, !VAL ); cpu =select_task_rq (p, p->wake_cpu,SD_BALANCE_WAKE , wake_flags);if (task_cpu (p) != cpu) {
I don't see a problem with moving the updating of p->state to the other side of the barrier but I'm relying on the comment that the barrier is only related to on_rq and on_cpu.
However, I'm less sure about what exactly you intended to do. __ttwu_queue_remote is void so maybe you meant to use ttwu_queue_remote. In that case, we potentially avoid spinning on on_rq for wakeups between tasks that do not share CPU but it's not clear why it would be specific to remote tasks. If you meant to call __ttwu_queue_remote unconditionally, it's not clear why that's now safe when smp_cond_load_acquire() cared about on_rq being 0 before queueing a task for wakup or directly waking it up.
Also because __ttwu_queue_remote() now happens before select_task_rq(), is there not a risk that in some cases we end up stacking tasks unnecessarily?
Quoted 15 lines
kernel/sched/core.c +0
-8@@ -2597,14 +2608,6 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)psi_ttwu_dequeue (p); set_task_cpu(p, cpu); } - -#else /* CONFIG_SMP */ - -if (p->in_iowait) { -delayacct_blkio_end (p); -atomic_dec (&task_rq (p)->nr_iowait); - } -#endif /* CONFIG_SMP */ ttwu_queue (p, cpu, wake_flags);
Signature · 2 lines
Mel Gorman SUSE Labs
Peter Zijlstra
maintainerQuoted 13 lines · Mel Gorman
Quoted 11 lines · Peter Zijlstra
On Fri, May 15, 2020 at 09:47:40AM +0100, Mel Gorman wrote:
kernel/sched/core.c +10
-0+static bool ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) +{ +if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { +sched_clock_cpu (cpu);/* Sync clocks across CPUs */ +__ttwu_queue_remote (p, cpu, wake_flags); +return true; + } + +return false; +}
Quoted 6 lines
kernel/sched/core.c try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) · line 2601
Quoted 2 lines
kernel/sched/core.c +2
-0+if (READ_ONCE (p->on_cpu) &&__ttwu_queue_remote (p, cpu, wake_flags)) + goto unlock;I don't see a problem with moving the updating of
p->stateto the other side of the barrier but I'm relying on the comment that the barrier is only related toon_rqandon_cpu.
Yeah, I went with that too, like I said, didn't think too hard.
Quoted 2 lines
However, I'm less sure about what exactly you intended to do.
__ttwu_queue_remoteis void so maybe you meant to usettwu_queue_remote.
That!
Quoted 3 lines
In that case, we potentially avoid spinning on
on_rqfor wakeups between tasks that do not share CPU but it's not clear why it would be specific to remote tasks.
The thinking was that we can avoid spinning on ->on_cpu, and let the CPU get on with things. Rik had a workload where that spinning was significant, and I thought to have understood you saw the same.
By sticking the task on the wake_list of the CPU that's in charge of clearing ->on_cpu we ensure ->on_cpu is 0 by the time we get to doing the actual enqueue.
Peter Zijlstra
maintainerQuoted 5 lines
Quoted 4 lines
In that case, we potentially avoid spinning on
on_rqfor wakeups between tasks that do not share CPU but it's not clear why it would be specific to remote tasks.
Quoted 8 lines
The thinking was that we can avoid spinning on ->on_cpu, and let the CPU get on with things. Rik had a workload where that spinning was significant, and I thought to have understood you saw the same.
By sticking the task on the
wake_listof the CPU that's in charge of clearing ->on_cpu we ensure ->on_cpu is 0 by the time we get to doing the actual enqueue.
Of course, aside from sending an obviously broken patch, I also forgot to Cc Rik.
This one compiled, booted and built a kernel, so it must be perfect :-)
---
kernel/sched/core.c +24 -21
@@ -2330,7 +2330,7 @@ void scheduler_ipi(void)irq_exit (); } -static void ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) +static void __ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) { struct emote(p, cpu, wake_flags)) *rq = cpu_eturn;(cpu);
@@ -2372,6 +2372,17 @@ bool cpus_share_cache(int this_cpu, int that_cpu) { return per_cpu(sd_q_lock(rq, &rf);_id, this_cpu) == per_cpu(sd_y_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)_id, that_cpu); } + +static bool ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) +{ +if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { +sched_clock_cpu (cpu);/* Sync clocks across CPUs */ +__ttwu_queue_remote (p, cpu, wake_flags); +return true; + } + +return false; +}#endif /* q && ttwu_remote(p, wake_flags))_SMP */ static void ttwu_queue (struct task_struct *p,int cpu,int wake_flags)
@@ -2380,11 +2391,8 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)struct rq_flags rf;#if defined(CONFIG_SMP ) -if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { -sched_clock_cpu (cpu);/* Sync clocks across CPUs */ -ttwu_queue_remote (p, cpu, wake_flags); +if (ttwu_queue_remote (p, cpu, wake_flags))return ; - }#endif rq_lock (rq, &rf);
@@ -2568,7 +2576,15 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)if (p->on_rq &&ttwu_remote (p, wake_flags)) goto unlock; +if (p->in_iowait) { +delayacct_blkio_end (p); +atomic_dec (&task_rq (p)->nr_iowait); + } +#ifdef CONFIG_SMP + p->sched_contributes_to_load = !!task_contributes_to_load (p); + p->state =TASK_WAKING ; +/* * Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be * possible to, falsely, observe p->on_cpu == 0.
@@ -2599,15 +2615,10 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)* This ensures that tasks getting woken will be fully ordered against * their previous state and preserve Program Order. */ - smp_ibutes_to_load = !!task_contributes_to_load (p);_load_acquire (&p->on_cpu, !VAL ); - - p->sched_contributes_to_load = !!task_contributes_to_load (p); - p->state =TASK_WAKING ; +if (READ_ONCE (p->on_cpu) &&ttwu_queue_remote (p, cpu, wake_flags)) + goto unlock; -if (p->in_iowait) { -delayacct_blkio_end (p); -atomic_dec (&task_rq (p)->nr_iowait); - } +smp_cond_load_acquire (&p->on_cpu, !VAL ); cpu =select_task_rq (p, p->wake_cpu,SD_BALANCE_WAKE , wake_flags);if (task_cpu (p) != cpu) {
@@ -2615,14 +2626,6 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)psi_ttwu_dequeue (p); set_task_cpu(p, cpu); } - -#else /* CONFIG_SMP */ - -if (p->in_iowait) { -delayacct_blkio_end (p); -atomic_dec (&task_rq (p)->nr_iowait); - } -#endif /* CONFIG_SMP */ ttwu_queue (p, cpu, wake_flags);
Peter Zijlstra
maintainerQuoted 7 lines · Peter Zijlstra
Quoted 4 lines · Mel Gorman
However, the wakeups are so rapid that the wakeup
happens while the server is descheduling. That forces the waker to spin onsmp_cond_load_acquirefor longer. In this case, it can be cheaper to add the task to therq->wake_listeven if that potentially requires an IPI.
Quoted 8 lines
Right, I think Rik ran into that as well at some point. He wanted to make ->on_cpu do a hand-off, but simply queueing the wakeup on the prev cpu (which is currently in the middle of
schedule()) should be an easier proposition.Maybe something like this untested thing... could explode most mighty, didn't thing too hard.
Mel pointed out that that patch got mutilated somewhere (my own .Sent copy was fine), let me try again.
---
kernel/sched/core.c +24 -21
@@ -2330,7 +2330,7 @@ void scheduler_ipi(void)irq_exit (); } -static void ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) +static void __ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) {struct rq *rq =cpu_rq (cpu);
@@ -2372,6 +2372,17 @@ bool cpus_share_cache(int this_cpu, int that_cpu) {return per_cpu (sd_llc_id, this_cpu) ==per_cpu (sd_llc_id, that_cpu); } + +static bool ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) +{ +if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { +sched_clock_cpu (cpu);/* Sync clocks across CPUs */ +__ttwu_queue_remote (p, cpu, wake_flags); +return true; + } + +return false; +}#endif /* CONFIG_SMP */ static void ttwu_queue (struct task_struct *p,int cpu,int wake_flags)
@@ -2380,11 +2391,8 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)struct rq_flags rf;#if defined(CONFIG_SMP ) -if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { -sched_clock_cpu (cpu);/* Sync clocks across CPUs */ -ttwu_queue_remote (p, cpu, wake_flags); +if (ttwu_queue_remote (p, cpu, wake_flags))return ; - }#endif rq_lock (rq, &rf);
@@ -2568,7 +2576,15 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)if (p->on_rq &&ttwu_remote (p, wake_flags)) goto unlock; +if (p->in_iowait) { +delayacct_blkio_end (p); +atomic_dec (&task_rq (p)->nr_iowait); + } +#ifdef CONFIG_SMP + p->sched_contributes_to_load = !!task_contributes_to_load (p); + p->state =TASK_WAKING ; +/* * Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be * possible to, falsely, observe p->on_cpu == 0.
@@ -2599,15 +2615,10 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)* This ensures that tasks getting woken will be fully ordered against * their previous state and preserve Program Order. */ -smp_cond_load_acquire (&p->on_cpu, !VAL ); - - p->sched_contributes_to_load = !!task_contributes_to_load (p); - p->state =TASK_WAKING ; +if (READ_ONCE (p->on_cpu) &&ttwu_queue_remote (p, cpu, wake_flags)) + goto unlock; -if (p->in_iowait) { -delayacct_blkio_end (p); -atomic_dec (&task_rq (p)->nr_iowait); - } +smp_cond_load_acquire (&p->on_cpu, !VAL ); cpu =select_task_rq (p, p->wake_cpu,SD_BALANCE_WAKE , wake_flags);if (task_cpu (p) != cpu) {
@@ -2615,14 +2626,6 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)psi_ttwu_dequeue (p); set_task_cpu(p, cpu); } - -#else /* CONFIG_SMP */ - -if (p->in_iowait) { -delayacct_blkio_end (p); -atomic_dec (&task_rq (p)->nr_iowait); - } -#endif /* CONFIG_SMP */ ttwu_queue (p, cpu, wake_flags);
Quoted 13 lines · Peter Zijlstra
Quoted 12 lines · Peter Zijlstra
Quoted 4 lines · Mel Gorman
However, the wakeups are so rapid that the wakeup
happens while the server is descheduling. That forces the waker to spin onsmp_cond_load_acquirefor longer. In this case, it can be cheaper to add the task to therq->wake_listeven if that potentially requires an IPI.Right, I think Rik ran into that as well at some point. He wanted to make ->on_cpu do a hand-off, but simply queueing the wakeup on the prev cpu (which is currently in the middle of
schedule()) should be an easier proposition.
Quoted 7 lines
Quoted 3 lines
Maybe something like this untested thing... could explode most mighty, didn't thing too hard.
Mel pointed out that that patch got mutilated somewhere (my own .Sent copy was fine), let me try again.
Sorry for the slow response. My primary work machine suffered a catastrophic failure on Sunday night which is a fantastic way to start a Monday morning so I'm playing catchup.
IIUC, this patch front-loads as much work as possible before checking if the task is on_rq and then the waker/wakee shares a cache, queue task on the wake_list and otherwise do a direct wakeup.
The advantage is that spinning is avoided on p->on_rq when p does not share a cache. The disadvantage is that it may result in tasks being stacked but this should only happen when the domain is overloaded and select_task_eq() is unlikely to find an idle CPU. The load balancer would soon correct the situation anyway.
In terms of netperf for my testing, the benefit is marginal because the wakeups are primarily between tasks that share cache. It does trigger as perf indicates that some time is spent in ttwu_queue_remote with this patch, it's just that the overall time spent spinning on p->on_rq is very similar. I'm still waiting on other workloads to complete to see what the impact is.
However, intuitively at least, it makes sense to avoid spinning on p->on_rq when it's unnecessary and the other changes appear to be safe. Even if wake_list should be used in some cases for local wakeups, it would make sense to put that on top of this patch. Do you want to slap a changelog around this and update the comments or do you want me to do it? I should have more results in a few hours even if they are limited to one machine but ideally Rik would test his workload too.
Signature · 2 lines
Mel Gorman SUSE Labs
Peter Zijlstra
maintainerQuoted 10 lines · Mel Gorman
IIUC, this patch front-loads as much work as possible before checking if the task is
on_rqand then the waker/wakee shares a cache, queue task on thewake_listand otherwise do a direct wakeup.The advantage is that spinning is avoided on
p->on_rqwhen p does not share a cache. The disadvantage is that it may result in tasks being stacked but this should only happen when the domain is overloaded andselect_task_eq()is unlikely to find an idle CPU. The load balancer would soon correct the situation anyway.
Quoted 6 lines
In terms of netperf for my testing, the benefit is marginal because the wakeups are primarily between tasks that share cache. It does trigger as perf indicates that some time is spent in
ttwu_queue_remotewith this patch, it's just that the overall time spent spinning onp->on_rqis very similar. I'm still waiting on other workloads to complete to see what the impact is.
So it might make sense to play with the exact conditions under which we'll attempt this remote queue, if we see a large 'local' p->on_cpu spin time, it might make sense to attempt the queue even in this case.
We could for example change it to:
if (REAC_ONCE(p->on_cpu) && ttwu_queue_remote(p, cpu, wake_flags | WF_ON_CPU)) goto unlock;
and then use that in ttwu_queue_remote() to differentiate between these two cases.
Anyway, if it's a wash (atomic op vs spinning) then it's probably not worth it.
Another optimization might be to forgo the IPI entirely in this case and instead stick a sched_ttwu_pending() at the end of __schedule() or something like that.
Quoted 7 lines
However, intuitively at least, it makes sense to avoid spinning on
p->on_rqwhen it's unnecessary and the other changes appear to be safe. Even ifwake_listshould be used in some cases for local wakeups, it would make sense to put that on top of this patch. Do you want to slap a changelog around this and update the comments or do you want me to do it? I should have more results in a few hours even if they are limited to one machine but ideally Rik would test his workload too.
I've written you a Changelog, but please carry it in your set to evaluate if it's actually worth it.
---
Subject: sched: Optimize ttwu() spinning on p->on_cpu
From: Peter Zijlstra <[email protected]>
Date: Fri, 15 May 2020 16:24:44 +0200
Both Rik and Mel reported seeing ttwu() spend significant time on:
smp_cond_load_acquire(&p->on_cpu, !VAL);
Attempt to avoid this by queueing the wakeup on the CPU that own's the p->on_cpu value. This will then allow the ttwu() to complete without further waiting.
Since we run schedule() with interrupts disabled, the IPI is guaranteed to happen after p->on_cpu is cleared, this is what makes it safe to queue early.
--- kernel/sched/core.c | 45 ++++++++++++++++++++++++--------------------- 1 file changed, 24 insertions(+), 21 deletions(-)
kernel/sched/core.c +24 -21
@@ -2312,7 +2312,7 @@ static void wake_csd_func(void *info)sched_ttwu_pending (); } -static void ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) +static void __ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) {struct rq *rq =cpu_rq (cpu);
@@ -2354,6 +2354,17 @@ bool cpus_share_cache(int this_cpu, int {return per_cpu (sd_llc_id, this_cpu) ==per_cpu (sd_llc_id, that_cpu); } + +static bool ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) +{ +if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { +sched_clock_cpu (cpu);/* Sync clocks across CPUs */ +__ttwu_queue_remote (p, cpu, wake_flags); +return true; + } + +return false; +}#endif /* CONFIG_SMP */ static void ttwu_queue (struct task_struct *p,int cpu,int wake_flags)
@@ -2362,11 +2373,8 @@ static void ttwu_queue(struct task_strucstruct rq_flags rf;#if defined(CONFIG_SMP ) -if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { -sched_clock_cpu (cpu);/* Sync clocks across CPUs */ -ttwu_queue_remote (p, cpu, wake_flags); +if (ttwu_queue_remote (p, cpu, wake_flags))return ; - }#endif rq_lock (rq, &rf);
@@ -2550,7 +2558,15 @@ try_to_wake_up(struct task_struct *p, unif (p->on_rq &&ttwu_remote (p, wake_flags)) goto unlock; +if (p->in_iowait) { +delayacct_blkio_end (p); +atomic_dec (&task_rq (p)->nr_iowait); + } +#ifdef CONFIG_SMP + p->sched_contributes_to_load = !!task_contributes_to_load (p); + p->state =TASK_WAKING ; +/* * Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be * possible to, falsely, observe p->on_cpu == 0.
@@ -2581,15 +2597,10 @@ try_to_wake_up(struct task_struct *p, un* This ensures that tasks getting woken will be fully ordered against * their previous state and preserve Program Order. */ -smp_cond_load_acquire (&p->on_cpu, !VAL ); - - p->sched_contributes_to_load = !!task_contributes_to_load (p); - p->state =TASK_WAKING ; +if (READ_ONCE (p->on_cpu) &&ttwu_queue_remote (p, cpu, wake_flags)) + goto unlock; -if (p->in_iowait) { -delayacct_blkio_end (p); -atomic_dec (&task_rq (p)->nr_iowait); - } +smp_cond_load_acquire (&p->on_cpu, !VAL ); cpu =select_task_rq (p, p->wake_cpu,SD_BALANCE_WAKE , wake_flags);if (task_cpu (p) != cpu) {
@@ -2597,14 +2608,6 @@ try_to_wake_up(struct task_struct *p, unpsi_ttwu_dequeue (p); set_task_cpu(p, cpu); } - -#else /* CONFIG_SMP */ - -if (p->in_iowait) { -delayacct_blkio_end (p); -atomic_dec (&task_rq (p)->nr_iowait); - } -#endif /* CONFIG_SMP */ ttwu_queue (p, cpu, wake_flags);
Quoted 27 lines · Peter Zijlstra
Quoted 10 lines · Mel Gorman
IIUC, this patch front-loads as much work as possible before checking if the task is
on_rqand then the waker/wakee shares a cache, queue task on thewake_listand otherwise do a direct wakeup.The advantage is that spinning is avoided on
p->on_rqwhen p does not share a cache. The disadvantage is that it may result in tasks being stacked but this should only happen when the domain is overloaded andselect_task_eq()is unlikely to find an idle CPU. The load balancer would soon correct the situation anyway.Quoted 6 lines
In terms of netperf for my testing, the benefit is marginal because the wakeups are primarily between tasks that share cache. It does trigger as perf indicates that some time is spent in
ttwu_queue_remotewith this patch, it's just that the overall time spent spinning onp->on_rqis very similar. I'm still waiting on other workloads to complete to see what the impact is.So it might make sense to play with the exact conditions under which we'll attempt this remote queue, if we see a large 'local'
p->on_cpuspin time, it might make sense to attempt the queue even in this case.We could for example change it to:
if (REAC_ONCE(p->on_cpu) && ttwu_queue_remote(p, cpu, wake_flags | WF_ON_CPU)) goto unlock;
Quoted 7 lines
and then use that in
ttwu_queue_remote()to differentiate between these two cases.#endif /*
CONFIG_SMP*/
ttwu_queue(p, cpu, wake_flags);
Is something like this on top of your patch what you had in mind?
---8<---
--- kernel/sched/core.c | 35 ++++++++++++++++++++++++++--------- kernel/sched/sched.h | 3 ++- 2 files changed, 28 insertions(+), 10 deletions(-)
kernel/sched/core.c +26 -9
@@ -2330,13 +2330,19 @@ void scheduler_ipi(void)irq_exit (); } -static void __ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) +/* +* Queue a task on the target CPUs wake_list and wake the CPU via IPI if +* necessary. The wakee CPU on receipt of the IPI will queue the task +* via sched_ttwu_wakeup() for activation instead of the waking task +* activating and queueing the wakee. +*/ +static void __ttwu_queue_wakelist (struct task_struct *p,int cpu,int wake_flags) {struct rq *rq =cpu_rq (cpu); p->sched_remote_wakeup = !!(wake_flags &WF_MIGRATED ); -if (llist_add (&p->wake_entry, &cpu_rq (cpu)->wake_list)) { +if (llist_add (&p->wake_entry, &rq->wake_list)) {if (!set_nr_if_polling (rq->idle))smp_send_reschedule (cpu);else
@@ -2373,12 +2379,23 @@ bool cpus_share_cache(int this_cpu, int that_cpu)return per_cpu (sd_llc_id, this_cpu) ==per_cpu (sd_llc_id, that_cpu); } -static bool ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) +static bool ttwu_queue_wakelist (struct task_struct *p,int cpu,int wake_flags) { -if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { -sched_clock_cpu (cpu);/* Sync clocks across CPUs */ -__ttwu_queue_remote (p, cpu, wake_flags); -return true; +if (sched_feat (TTWU_QUEUE )) { +/* +* If CPU does not share cache then queue the task on the remote +* rqs wakelist to avoid accessing remote data. Alternatively, +* if the task is descheduling and the only running task on the +* CPU then use the wakelist to offload the task activation to +* the CPU that will soon be idle so the waker can continue. +* nr_running is checked to avoid unnecessary task stacking. +*/ +if (!cpus_share_cache (smp_processor_id (), cpu) || + ((wake_flags &WF_ON_RQ ) &&cpu_rq (cpu)->nr_running <=1 )) { +sched_clock_cpu (cpu);/* Sync clocks across CPUs */ +__ttwu_queue_wakelist (p, cpu, wake_flags); +return true; + } }return false;
@@ -2391,7 +2408,7 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)struct rq_flags rf;#if defined(CONFIG_SMP ) -if (ttwu_queue_remote (p, cpu, wake_flags)) +if (ttwu_queue_wakelist (p, cpu, wake_flags))return ;#endif
@@ -2611,7 +2628,7 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)* let the waker make forward progress. This is safe because IRQs are * disabled and the IPI will deliver after on_cpu is cleared. */ -if (READ_ONCE (p->on_cpu) &&ttwu_queue_remote (p, cpu, wake_flags)) +if (READ_ONCE (p->on_cpu) &&ttwu_queue_wakelist (p, cpu, wake_flags |WF_ON_RQ )) goto unlock;/*
kernel/sched/sched.h +2 -1
@@ -1688,7 +1688,8 @@ static inline int task_on_rq_migrating(struct task_struct *p)*/ #define WF_SYNC 0x01/* Waker goes to sleep after wakeup */ #define WF_FORK 0x02/* Child wakeup after fork */ -#define WF_MIGRATED 0x4/* Internal use, task got migrated */ +#define WF_MIGRATED 0x04/* Internal use, task got migrated */ +#define WF_ON_RQ 0x08/* Wakee is on_rq */ /* * To aid in avoiding the subversion of "niceness" due to uneven distribution
Peter Zijlstra
maintainerQuoted 1 line · Mel Gorman
Is something like this on top of your patch what you had in mind?
All under the assumption that is makes it go faster of course ;-)
Quoted 1 line
---8<---
static inline bool ttwu_queue_cond()
{
/*
* If the CPU does not share cache, then queue the task on the
* remote rqs wakelist to avoid accessing remote data.
*/
if (!cpus_share_cache(smp_processor_id(), cpu))
return true;
/*
* If the task is descheduling and the only running task on the
* CPU, ....
*/
if ((wake_flags & WF_ON_RQ) && cpu_rq(cpu)->nr_running <= 1)
return true;
return false;
}Quoted 12 lines
kernel/sched/core.c +6
-5-static bool ttwu_queue_remote (struct task_struct *p,int cpu,int wake_flags) +static bool ttwu_queue_wakelist (struct task_struct *p,int cpu,int wake_flags) { -if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { -sched_clock_cpu (cpu);/* Sync clocks across CPUs */ -__ttwu_queue_remote (p, cpu, wake_flags); -return true; +if (sched_feat (TTWU_QUEUE )) { +/* +* If CPU does not share cache then queue the task on the remote +* rqs wakelist to avoid accessing remote data. Alternatively, +* if the task is descheduling and the only running task on the
kernel/sched/core.c bool cpus_share_cache(int this_cpu, int that_cpu) · line 2398
Quoted 10 lines
kernel/sched/core.c +10
-0+* CPU then use the wakelist to offload the task activation to +* the CPU that will soon be idle so the waker can continue. +* nr_running is checked to avoid unnecessary task stacking. +*/ +if (!cpus_share_cache (smp_processor_id (), cpu) || + ((wake_flags &WF_ON_RQ ) &&cpu_rq (cpu)->nr_running <=1 )) { +sched_clock_cpu (cpu);/* Sync clocks across CPUs */ +__ttwu_queue_wakelist (p, cpu, wake_flags); +return true; + }
if (sched_feat(TTWU_QUEUE) && ttwu_queue_cond(cpu, wake_flags)) {
sched_clock_cpu(cpu); /* Sync clocks across CPUs */
__ttwu_queue_remote(p, cpu, wake_flags);
return true;kernel/sched/core.c bool cpus_share_cache(int this_cpu, int that_cpu) · line 2401
Quoted 3 lines
kernel/sched/core.c +0
-0}return false;
might be easier to read...
Peter Zijlstra
maintainerQuoted 34 lines · Mel Gorman
sched: Wake cache-local tasks via
wake_listif wakee CPU is pollingThere are two separate method for waking a task from idle, one for tasks running on CPUs that share a cache and one for when the caches are separate. The methods can loosely be called local and remote even though this is not directly related to NUMA and instead is due to the expected cost of accessing data that is cache-hot on another CPU. The "local" costs are expected to be relatively cheap as they share the LLC in comparison to a remote IPI that is potentially required when using the "remote" wakeup. The problem is that the local wakeup is not always cheaper and it appears to have degraded even further around the 4.19 mark.
There appears to be a couple of reasons why it can be slower.
The first is specific to pairs of tasks where one or both rapidly enters idle. For example, on netperf
UDP_STREAM, the client sends a bunch of packets, wakes the server, the server processes some packets and goes back to sleep. There is a relationship between the tasks but it's not strictly synchronous. The timing is different if the client/server are on separate NUMA nodes and netserver is more likely to enter idle (measured as server entering idle 10% more times when tasks are local vs remote but machine-specific). However, the wakeups are so rapid that the wakeup happens while the server is descheduling. That forces the waker to spin onsmp_cond_load_acquirefor longer. In this case, it can be cheaper to add the task to therq->wake_listeven if that potentially requires an IPI.The second is that the local wakeup path is simply not always that fast. Using ftrace, the cost of the locks,
update_rq_clockandttwu_do_activatewas measured as roughly 4.5 microseconds. While it's a single instance, the cost of the "remote" wakeup fortry_to_wake_upwas roughly 2.5 microseconds versus 6.2 microseconds for the "fast" local wakeup. When there are tens of thousands of wakeups per second, these costs accumulate and manifest as a throughput regression in netperfUDP_STREAM.
Quoted 7 lines
The essential difference in costs comes down to whether the CPU is fully idle, a task is descheduling or polling in
poll_idle(). This patch special casesttwu_queue()to use the "remote" method if the CPUs task is polling as it's generally cheaper to use thewake_listin that case than the local method because an IPI should not be required. As it is race-prone, a reschedule IPI may still be sent but in that case the local wakeup would also have to send a reschedule IPI so it should be harmless.
We don't in fact send a wakeup IPI when polling. So this might end up with an extra IPI.
Quoted 6 lines
kernel/sched/core.c +0
-0@@ -2380,13 +2380,32 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)struct rq_flags rf;
kernel/sched/core.c static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags) · line 2389
Quoted 10 lines
kernel/sched/core.c +7
-1#if defined(CONFIG_SMP ) -if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { +if (sched_feat (TTWU_QUEUE )) { +/* +* A remote wakeup is often expensive as can require +* an IPI and the wakeup path is slow. However, in +* the specific case where the target CPU is idle +* and polling, the CPU is active and rapidly checking +* if a reschedule is needed.
Not strictly true; MWAIT can be very deep idle, it's just that with POLLING we indicate we do not have to send an IPI to wake up. Just setting the TIF_NEED_RESCHED flag is sufficient to wake up -- the monitor part of monitor-wait.
kernel/sched/core.c static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags) · line 2399
Quoted 11 lines
kernel/sched/core.c +10
-0In this case, the idle +* task just needs to be marked for resched and p +* will rapidly be requeued which is less expensive +* than the direct wakeup path. +*/ +if (cpus_share_cache (smp_processor_id (), cpu)) { +struct thread_info *ti =task_thread_info (p); + typeof(ti->flags) val = READ_ONCE(ti->flags); + +if (val & _TIF_POLLING_NRFLAG) + goto activate;
I'm completely confused... the result here is that if you're polling you do _NOT_ queue on the wake_list, but instead immediately enqueue.
(which kinda makes sense, since if the remote CPU is idle, it doesn't have these lines in its cache anyway)
But the subject and comments all seem to suggest the opposite !?
Also, this will fail compilation when !defined(TIF_POLLING_NRFLAGG).
kernel/sched/core.c static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags) · line 2408
Quoted 9 lines
kernel/sched/core.c +3
-0+ } +sched_clock_cpu (cpu);/* Sync clocks across CPUs */ ttwu_queue_remote (p, cpu, wake_flags);return ; }#endif +activate:
The labels wants to go inside the ifdef, otherwise GCC will complain about unused labels etc..
Quoted 3 lines
kernel/sched/core.c +0
-0rq_lock (rq, &rf);update_rq_clock (rq);
ttwu_do_activate(rq, p, wake_flags, &rf);
Quoted 43 lines · Peter Zijlstra
Quoted 34 lines · Mel Gorman
sched: Wake cache-local tasks via
wake_listif wakee CPU is pollingThere are two separate method for waking a task from idle, one for tasks running on CPUs that share a cache and one for when the caches are separate. The methods can loosely be called local and remote even though this is not directly related to NUMA and instead is due to the expected cost of accessing data that is cache-hot on another CPU. The "local" costs are expected to be relatively cheap as they share the LLC in comparison to a remote IPI that is potentially required when using the "remote" wakeup. The problem is that the local wakeup is not always cheaper and it appears to have degraded even further around the 4.19 mark.
There appears to be a couple of reasons why it can be slower.
The first is specific to pairs of tasks where one or both rapidly enters idle. For example, on netperf
UDP_STREAM, the client sends a bunch of packets, wakes the server, the server processes some packets and goes back to sleep. There is a relationship between the tasks but it's not strictly synchronous. The timing is different if the client/server are on separate NUMA nodes and netserver is more likely to enter idle (measured as server entering idle 10% more times when tasks are local vs remote but machine-specific). However, the wakeups are so rapid that the wakeup happens while the server is descheduling. That forces the waker to spin onsmp_cond_load_acquirefor longer. In this case, it can be cheaper to add the task to therq->wake_listeven if that potentially requires an IPI.The second is that the local wakeup path is simply not always that fast. Using ftrace, the cost of the locks,
update_rq_clockandttwu_do_activatewas measured as roughly 4.5 microseconds. While it's a single instance, the cost of the "remote" wakeup fortry_to_wake_upwas roughly 2.5 microseconds versus 6.2 microseconds for the "fast" local wakeup. When there are tens of thousands of wakeups per second, these costs accumulate and manifest as a throughput regression in netperfUDP_STREAM.Quoted 7 lines
The essential difference in costs comes down to whether the CPU is fully idle, a task is descheduling or polling in
poll_idle(). This patch special casesttwu_queue()to use the "remote" method if the CPUs task is polling as it's generally cheaper to use thewake_listin that case than the local method because an IPI should not be required. As it is race-prone, a reschedule IPI may still be sent but in that case the local wakeup would also have to send a reschedule IPI so it should be harmless.
Quoted 1 line
We don't in fact send a wakeup IPI when polling.
I know when it's polling that an IPI can be skipped as set_nr_if_polling will succeed.
Quoted 3 lines
So this might end up
with an extra IPI.
Yes, it might. If between the check for polling and set_nr_and_not_polling, the polling may have stopped and the CPU entered a c-state and an IPI is sent but in that case, either path is likely to result in some sort of IPI.
Quoted 8 lines
Quoted 8 lines
kernel/sched/core.c +0
-0@@ -2380,13 +2380,32 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)struct rq_flags rf;
Quoted 15 lines
kernel/sched/core.c static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)
Quoted 9 lines
kernel/sched/core.c +7
-1#if defined(CONFIG_SMP ) -if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { +if (sched_feat (TTWU_QUEUE )) { +/* +* A remote wakeup is often expensive as can require +* an IPI and the wakeup path is slow. However, in +* the specific case where the target CPU is idle +* and polling, the CPU is active and rapidly checking +* if a reschedule is needed. Not strictly true; MWAIT can be very deep idle, it's just that with POLLING we indicate we do not have to send an IPI to wake up. Just setting the
TIF_NEED_RESCHEDflag is sufficient to wake up -- the monitor part of monitor-wait.
The terminology is unhelpful. In this case when I said "idle and polling", I was thinking of the idle task running poll_idle() and the CPU should not be in any c-state below C0 but it doesn't really matter due to your own comments.
Quoted 15 lines
kernel/sched/core.c static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags) · line 2399
Quoted 11 lines
kernel/sched/core.c +10
-0In this case, the idle +* task just needs to be marked for resched and p +* will rapidly be requeued which is less expensive +* than the direct wakeup path. +*/ +if (cpus_share_cache (smp_processor_id (), cpu)) { +struct thread_info *ti =task_thread_info (p); + typeof(ti->flags) val = READ_ONCE(ti->flags); + +if (val & _TIF_POLLING_NRFLAG) + goto activate;I'm completely confused... the result here is that if you're polling you do _NOT_ queue on the
wake_list, but instead immediately enqueue.
Quoted 3 lines
(which kinda makes sense, since if the remote CPU is idle, it doesn't have these lines in its cache anyway)
Crap, I rushed this and severely confused myself about what is going on. It is definitely the case that flipping this check does not give any benefit. The patch shows a benefit but I'm failing to understand exactly why. How I ended up here was perf indicating a lot of time spent on smp_cond_load_acquire() which made me look closely at ttwu_remote() and looking at function graphs to compare the different types of wakeups and their timings.
With this patch when netperf is waking the server, netperf is all that is active with the wakeup completing 3.489 microseconds. If this check is flipped, the idle thread is running at the same time and clearly interleaving with netperf in the ftrace function_graph and the wakeup takes 9.449 microseconds. The exact timings vary obviously but it's what led to the patch -- brute force starting at function_graphs until something fell out.
Quoted 2 lines
But the subject and comments all seem to suggest the opposite !?
Yeah, I need to think more about why exactly this patch makes a difference.
Quoted 2 lines
Also, this will fail compilation when !defined(TIF_POLLING_NRFLAGG).
I can fix that.
Quoted 8 lines
Quoted 8 lines
kernel/sched/core.c +2
-0+ } +sched_clock_cpu (cpu);/* Sync clocks across CPUs */ ttwu_queue_remote (p, cpu, wake_flags);return ; }#endif
Quoted 5 lines
kernel/sched/core.c static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags) · line 2408
Quoted 1 line
kernel/sched/core.c +1
-0+activate:The labels wants to go inside the ifdef, otherwise GCC will complain about unused labels etc..
And that too.
Thanks.
Signature · 2 lines
Mel Gorman SUSE Labs
Peter Zijlstra
maintainerQuoted 8 lines · Mel Gorman
Quoted 6 lines · Peter Zijlstra
kernel/sched/core.c static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags) · line 2399
Quoted 2 lines
kernel/sched/core.c +2
-0+if (val & _TIF_POLLING_NRFLAG) + goto activate;I'm completely confused... the result here is that if you're polling you do _NOT_ queue on the
wake_list, but instead immediately enqueue.
Quoted 5 lines
Quoted 3 lines
(which kinda makes sense, since if the remote CPU is idle, it doesn't have these lines in its cache anyway)
Crap, I rushed this and severely confused myself about what is going
Hehe, and here I though I was confused :-)
Quoted 6 lines
on. It is definitely the case that flipping this check does not give any benefit. The patch shows a benefit but I'm failing to understand exactly why. How I ended up here was perf indicating a lot of time spent on
smp_cond_load_acquire()which made me look closely atttwu_remote()and looking at function graphs to compare the different types of wakeups and their timings.
So the raisin we did this remote wakeup thing in the first place was that Oracle was having very heavy rq->lock cache-line contention. By farming off the enqueue to the CPU that was going to run the task anyway, the rq->lock (and the other runqueue structure lines) could stay in the CPU that was using them (hard). Less cacheline ping-pong, more win.
The observation here is that if a CPU is idle, it's rq will not be contended.
Quoted 7 lines
Complete shot in the dark but restore
adjust_numa_imbalance()and try thiskernel/sched/core.c +0
-0
kernel/sched/core.c static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags) · line 2398
Quoted 7 lines
kernel/sched/core.c +1
-1@@ -2393,7 +2393,7 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)struct rq_flags rf;#if defined(CONFIG_SMP ) -if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { +if (sched_feat (TTWU_QUEUE )) {sched_clock_cpu (cpu);/* Sync clocks across CPUs */ ttwu_queue_remote (p, cpu, wake_flags);return ;
Hi Mel,
we have performance results for the proposed patch above ^^^. Unfortunately, it hasn't helped the performance.
Jirka
Quoted 44 lines · Mel Gorman
Quoted 15 lines · Jirka Hladky
Hi Mel,
we have tried the kernel with
adjust_numa_imbalance()crippled to just return the imbalance it's given.It has solved all the performance problems I have reported. Performance is the same as with 5.6 kernel (before the patch was applied).
* solved the performance drop upto 20% with single instance SPECjbb2005 benchmark on 8 NUMA node servers (particularly on AMD EPYC Rome systems) => this performance drop was INCREASING with higher threads counts (10% for 16 threads and 20 % for 32 threads)
* solved the performance drop for low load scenarios (SPECjvm2008 and NAS)Quoted 5 lines
Any suggestions on how to proceed? One approach is to turn "
imbalance_min" into the kernel tunable. Any other ideas?Complete shot in the dark but restore
adjust_numa_imbalance()and try thiskernel/sched/core.c +1
-2@@ -2393,7 +2393,7 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)struct rq_flags rf;#if defined(CONFIG_SMP ) -if (sched_feat (TTWU_QUEUE ) && !cpus_share_cache (smp_processor_id (), cpu)) { +if (sched_feat (TTWU_QUEUE )) {sched_clock_cpu (cpu);/* Sync clocks across CPUs */ ttwu_queue_remote (p, cpu, wake_flags);return ; --Mel Gorman
SUSE Labs
Signature · 1 line
-Jirka