Skip to content

Revisit NUMA imbalance tolerance and fork balancing

Mel Gorman lkml 3 versions – Applied

Status from this message: The following commit has been merged into the sched/core branch of tip:

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest4 patches 6 repliesApplied
  2. v24 patches 0 repliesSuperseded
  3. v1 RFC3 patches 11 repliesSuperseded

Changes in v3 from the cover letter

o Build fix for !NUMA_BALANCING configurations

Patches in v3 Applied

#Subject and filesSizeRepliesApplied
1/4sched/numa: Rename nr_running and break out the magic numberkernel/sched/fair.c+6 -41 reply
2/4sched: Avoid unnecessary calculation of load imbalance at clone timekernel/sched/fair.c+5 -31 reply
3/4sched/numa: Allow a floating imbalance between NUMA nodeskernel/sched/fair.c+11 -101 reply
4/4sched: Limit the amount of NUMA imbalance that can exist at fork timekernel/sched/fair.c+15 -21 reply

Reviews trailers per patch

All 4 patches have a Reviewed-by or Acked-by.

PatchVincent Guittot
1/4R-b
2/4R-b
3/4R-b
4/4R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v3 thread

Cover letter [PATCH v3 0/4] Revisit NUMA imbalance tolerance and fork balancing

Changelog since v2
o Build fix for !NUMA_BALANCING configurations

Changelog since v1
o Split out patch that moves imbalance calculation
o Strongly connect fork imbalance considerations with adjust_numa_imbalance

When NUMA and CPU balancing were reconciled, there was an attempt to allow a degree of imbalance but it caused more problems than it solved. Instead, imbalance was only allowed with an almost idle NUMA domain. A lot of the problems have since been addressed so it's time for a revisit. There is also an issue with how fork is balanced across threads. It's mentioned in this context as patch 3 and 4 should share similar behaviour in terms of a nodes utilisation.

Patch 1 is just a cosmetic rename

Patch 2 moves an imbalance calculation. It is both a micro-optimisation
and avoids confusing what imbalance means for different group types.

Patch 3 allows a "floating" imbalance to exist so communicating tasks can
remain on the same domain until utilisation is higher. It aims to balance compute availability with memory bandwidth.

Patch 4 is the interesting one. Currently fork can allow a NUMA node
to be completely utilised as long as there are idle CPUs until the load balancer gets involved. This caused serious problems with a real workload that unfortunately I cannot share many details about but there is a proxy reproducer.

Signature · 13 lines
2.26.2

Mel Gorman (4):
  sched/numa: Rename nr_running and break out the magic number
  sched: Avoid unnecessary calculation of load imbalance at clone time
  sched/numa: Allow a floating imbalance between NUMA nodes
  sched: Limit the amount of NUMA imbalance that can exist at fork time

 kernel/sched/fair.c | 44 +++++++++++++++++++++++++++++++-------------
 1 file changed, 31 insertions(+), 13 deletions(-)

-- 
2.26.2