Skip to contentSkip to conversation

Basic scheduler support for automatic NUMA balancing V8

PATCH In review Mel Gorman 4 participants 69 messages Sep 27, 2013 – Oct 24, 2013

Subsystems (18)
lore
Share & tools
lore

Mel Gorman

author reviewer
#0 lore raw

cover Basic scheduler support for automatic NUMA balancing V8

Weighing in at 63 patches makes the term "basic" in the series title is a misnomer.

This series still has roughly the same goals as previous versions. It reduces overhead of automatic balancing through scan rate reduction and the avoidance of TLB flushes. It selects a preferred node and moves tasks towards their memory as well as moving memory toward their task. It handles shared pages and groups related tasks together. Some problems such as shared page interleaving and properly dealing with processes that are larger than a node are being deferred.

It is still based on 3.11 because that's what I was testing against. If we can agree this should be merged to -tip for testing I'll rebase to deal with any scheduler conflicts but for now I do not want to invalidate other peoples testing. The only obvious thing that is missing is hotplug handling. Peter is currently working on reducing [get|put]_online_cpus overhead so that it can be added to migrate_swap.

Peter, some of your patches are missing signed-offs-by -- 4-5, 43 and 55.

Changelog since V7
o THP migration race and pagetable insertion fixes
o Do no handle PMDs in batch
o Shared page migration throttling
o Various changes to how last nid/pid information is recorded
o False pid match sanity checks when joining NUMA task groups
o Adapt scan rate based on local/remote fault statistics
o Period retry of migration to preferred node
o Limit scope of system-wide search
o Schedule threads on the same node as process that created them
o Cleanup numa_group on exec

Changelog since V6
o Group tasks that share pages together
o More scan avoidance of VMAs mapping pages that are not likely to migrate
o cpunid conversion, system-wide searching of tasks to balance with

Changelog since V6
o Various TLB flush optimisations
o Comment updates
o Sanitise task_numa_fault callsites for consistent semantics
o Revert some of the scanning adaption stuff
o Revert patch that defers scanning until task schedules on another node
o Start delayed scanning properly
o Avoid the same task always performing the PTE scan
o Continue PTE scanning even if migration is rate limited

Changelog since V5
o Add __GFP_NOWARN for numa hinting fault count
o Use is_huge_zero_page
o Favour moving tasks towards nodes with higher faults
o Optionally resist moving tasks towards nodes with lower faults
o Scan shared THP pages

Changelog since V4
o Added code that avoids overloading preferred nodes
o Swap tasks if nodes are overloaded and the swap does not impair locality

Changelog since V3
o Correct detection of unset last nid/pid information
o Dropped nr_preferred_running and replaced it with Peter's load balancing
o Pass in correct node information for THP hinting faults
o Pressure tasks sharing a THP page to move towards same node
o Do not set pmd_numa if false sharing is detected

Changelog since V2
o Reshuffle to match Peter's implied preference for layout
o Reshuffle to move private/shared split towards end of series to make it easier to evaluate the impact
o Use PID information to identify private accesses
o Set the floor for PTE scanning based on virtual address space scan rates instead of time
o Some locking improvements
o Do not preempt pinned tasks unless they are kernel threads

Changelog since V1
o Scan pages with elevated map count (shared pages)
o Scale scan rates based on the vsz of the process so the sampling of the task is independant of its size
o Favour moving towards nodes with more faults even if it's not the preferred node
o Laughably basic accounting of a compute overloaded node when selecting the preferred node.
o Applied review comments

This series integrates basic scheduler support for automatic NUMA balancing. It was initially based on Peter Ziljstra's work in "sched, numa, mm:
Add adaptive NUMA affinity support" but deviates too much to preserve Signed-off-bys. As before, if the relevant authors are ok with it I'll add Signed-off-bys (or add them yourselves if you pick the patches up). There has been a tonne of additional work from both Peter and Rik van Riel.

Some reports indicate that the performance is getting close to manual bindings for some workloads but your mileage will vary. As before, the intention is not to complete the work but to incrementally improve mainline and preserve bisectability for any bug reports that crop up.

Patch 1 is a monolithic dump of patches thare are destined for upstream that
this series indirectly depends upon.

Patches 2-3 adds sysctl documentation and comment fixlets

Patch 4 avoids accounting for a hinting fault if another thread handled the
fault in parallel

Patches 5-6 avoid races with parallel THP migration and THP splits.

Patch 7 corrects a THP NUMA hint fault accounting bug

Patches 8-9 avoids TLB flushes during the PTE scan if no updates are made

Patch 10 sanitizes task_numa_fault callsites to have consist semantics and
always record the fault based on the correct location of the page.

Patch 11 closes races between THP migration and PMD clearing.

Patch 12 avoids trying to migrate the THP zero page

Patch 13 avoids the same task being selected to perform the PTE scan within
a shared address space.

Patch 14 continues PTE scanning even if migration rate limited

Patch 15 notes that delaying the PTE scan until a task is scheduled on an
alternative node misses the case where the task is only accessing shared memory on a partially loaded machine and reverts a patch.

Patch 16 initialises numa_next_scan properly so that PTE scanning is delayed
when a process starts.

Patch 17 sets the scan rate proportional to the size of the task being
scanned.

Patch 18 slows the scan rate if no hinting faults were trapped by an idle task.

Patch 19 tracks NUMA hinting faults per-task and per-node

Patches 20-24 selects a preferred node at the end of a PTE scan based on what
node incurred the highest number of NUMA faults. When the balancer is comparing two CPU it will prefer to locate tasks on their preferred node. When initially selected the task is rescheduled on the preferred node if it is not running on that node already. This avoids waiting for the scheduler to move the task slowly.

Patch 25 adds infrastructure to allow separate tracking of shared/private
pages but treats all faults as if they are private accesses. Laying it out this way reduces churn later in the series when private fault detection is introduced

Patch 26 avoids some unnecessary allocation

Patch 27-28 kicks away some training wheels and scans shared pages and
small VMAs.

Patch 29 introduces private fault detection based on the PID of the faulting
process and accounts for shared/private accesses differently.

Patch 30 avoids migrating memory immediately after the load balancer moves
a task to another node in case it's a transient migration.

Patch 31 avoids scanning VMAs that do not migrate-on-fault which addresses
a serious regression on a database performance test.

Patch 32 pick the least loaded CPU based on a preferred node based on
a scheduling domain common to both the source and destination NUMA node.

Patch 33 retries task migration if an earlier attempt failed

Patch 34 will begin task migration immediately if running on its preferred
node

Patch 35 will avoid trapping hinting faults for shared read-only library
pages as these never migrate anyway

Patch 36 avoids handling pmd hinting faults if none of the ptes below it were
marked pte numa

Patches 37-38 introduce a mechanism for swapping tasks

Patch 39 uses a system-wide search to find tasks that can be swapped
to improve the overall locality of the system.

Patch 40 notes that the system-wide search may ignore the preferred node and
will use the preferred node placement if it has spare compute capacity.

Patch 41 will perform a global search if a node that should have had capacity
cannot have a task migrated to it

Patches 42-43 use cpupid to track pages so potential sharing tasks can
be quickly found

Patch 44 reports the ID of the numa group a task belongs.

Patch 45 copies the cpupid on page migration

Patch 46 avoids grouping based on read-only pages

Patch 47 stops handling pages within a PMD in batch as it distorts fault
statistics and failed to flush TLBs correctly.

Patch 48 schedules new threads on the same node as the parent.

Patch 49 schedules tasks based on their numa group

Patch 50 cleans up tasks numa_group on exec

Patch 51 avoids parallel updates to group stats

Patch 52 adds some debugging aids

Patches 53-54 separately considers task and group weights when selecting the node to
schedule a task on

Patch 56 checks if PID truncation may have caused false matches before joining tasks
to a NUMA grou

Patch 57 uses the false shared detection information for scan rate adaption later

Patch 58 adapts the scan rate based on local/remote faults

Patch 59 removes the period scan rate reset

Patch 60-61 throttles shared page migrations

Patch 62 avoids the use of atomics protects the values with a spinlock

Patch 63 periodically retries migrating a task back to its preferred node

Kernel 3.11 is the testing baseline.

o account-v8 Patches 1-8
o falsedetect-v8 Patches 1-57
o skipshared-v8 Patches 1-61
o noatomics-v8 Patches 1-62
o periodretry-v8 PAtches 1-63

This is SpecJBB running on a 4-socket machine with THP enabled and one JVM running for the whole system.

specjbb
3.11.0 3.11.0 3.11.0 3.11.0 3.11.0
account-v8r20 falsedetect-v8r33 skipshared-v8r33 noatomics-v8r33 periodretry-v8r33

TPut 1      26137.00 (  0.00%)     26879.00 (  2.84%)     26296.00 (  0.61%)     26563.00 (  1.63%)     26043.00 ( -0.36%)
TPut 2      56027.00 (  0.00%)     55871.00 ( -0.28%)     55336.00 ( -1.23%)     56325.00 (  0.53%)     53021.00 ( -5.37%)
TPut 3      84375.00 (  0.00%)     86268.00 (  2.24%)     82832.00 ( -1.83%)     86164.00 (  2.12%)     82818.00 ( -1.85%)
TPut 4     114968.00 (  0.00%)    115229.00 (  0.23%)    108632.00 ( -5.51%)    110348.00 ( -4.02%)    106984.00 ( -6.94%)
TPut 5     139786.00 (  0.00%)    136641.00 ( -2.25%)    131027.00 ( -6.27%)    139978.00 (  0.14%)    139000.00 ( -0.56%)
TPut 6     168084.00 (  0.00%)    167331.00 ( -0.45%)    159409.00 ( -5.16%)    164444.00 ( -2.17%)    157117.00 ( -6.52%)
TPut 7     196709.00 (  0.00%)    194751.00 ( -1.00%)    182849.00 ( -7.05%)    192798.00 ( -1.99%)    184336.00 ( -6.29%)
TPut 8     217119.00 (  0.00%)    220303.00 (  1.47%)    211978.00 ( -2.37%)    215026.00 ( -0.96%)    213757.00 ( -1.55%)
TPut 9     240359.00 (  0.00%)    244058.00 (  1.54%)    237005.00 ( -1.40%)    240441.00 (  0.03%)    236827.00 ( -1.47%)
TPut 10    263615.00 (  0.00%)    265787.00 (  0.82%)    260203.00 ( -1.29%)    260737.00 ( -1.09%)    261361.00 ( -0.86%)
TPut 11    292052.00 (  0.00%)    293012.00 (  0.33%)    294855.00 (  0.96%)    295295.00 (  1.11%)    288431.00 ( -1.24%)
TPut 12    306763.00 (  0.00%)    318380.00 (  3.79%)    317420.00 (  3.47%)    317958.00 (  3.65%)    310027.00 (  1.06%)
TPut 13    336282.00 (  0.00%)    338831.00 (  0.76%)    342010.00 (  1.70%)    339773.00 (  1.04%)    334431.00 ( -0.55%)
TPut 14    359216.00 (  0.00%)    365707.00 (  1.81%)    366968.00 (  2.16%)    371908.00 (  3.53%)    354059.00 ( -1.44%)
TPut 15    394379.00 (  0.00%)    382206.00 ( -3.09%)    372864.00 ( -5.46%)    389394.00 ( -1.26%)    378397.00 ( -4.05%)
TPut 16    391503.00 (  0.00%)    394595.00 (  0.79%)    409349.00 (  4.56%)    410242.00 (  4.79%)    396774.00 (  1.35%)
TPut 17    418193.00 (  0.00%)    425690.00 (  1.79%)    427327.00 (  2.18%)    411347.00 ( -1.64%)    418920.00 (  0.17%)
TPut 18    445409.00 (  0.00%)    440833.00 ( -1.03%)    447030.00 (  0.36%)    451240.00 (  1.31%)    440823.00 ( -1.03%)
TPut 19    447209.00 (  0.00%)    455059.00 (  1.76%)    459879.00 (  2.83%)    464383.00 (  3.84%)    455354.00 (  1.82%)
TPut 20    462761.00 (  0.00%)    472095.00 (  2.02%)    479006.00 (  3.51%)    473824.00 (  2.39%)    471917.00 (  1.98%)
TPut 21    481207.00 (  0.00%)    485035.00 (  0.80%)    491456.00 (  2.13%)    487685.00 (  1.35%)    489372.00 (  1.70%)
TPut 22    495510.00 (  0.00%)    492799.00 ( -0.55%)    502316.00 (  1.37%)    500868.00 (  1.08%)    484667.00 ( -2.19%)
TPut 23    496669.00 (  0.00%)    486543.00 ( -2.04%)    518198.00 (  4.33%)    507320.00 (  2.14%)    507332.00 (  2.15%)
TPut 24    519722.00 (  0.00%)    513153.00 ( -1.26%)    521337.00 (  0.31%)    524108.00 (  0.84%)    516621.00 ( -0.60%)
TPut 25    516030.00 (  0.00%)    501461.00 ( -2.82%)    507013.00 ( -1.75%)    519228.00 (  0.62%)    522712.00 (  1.29%)
TPut 26    521824.00 (  0.00%)    499423.00 ( -4.29%)    522584.00 (  0.15%)    529076.00 (  1.39%)    512817.00 ( -1.73%)
TPut 27    514817.00 (  0.00%)    489511.00 ( -4.92%)    503722.00 ( -2.16%)    520135.00 (  1.03%)    524537.00 (  1.89%)
TPut 28    505838.00 (  0.00%)    485151.00 ( -4.09%)    499372.00 ( -1.28%)    514423.00 (  1.70%)    525086.00 (  3.81%)
TPut 29    503231.00 (  0.00%)    496580.00 ( -1.32%)    503838.00 (  0.12%)    520836.00 (  3.50%)    526196.00 (  4.56%)
TPut 30    487832.00 (  0.00%)    512245.00 (  5.00%)    506440.00 (  3.81%)    521409.00 (  6.88%)    526048.00 (  7.83%)
TPut 31    492896.00 (  0.00%)    520700.00 (  5.64%)    513648.00 (  4.21%)    514615.00 (  4.41%)    525456.00 (  6.61%)
TPut 32    499692.00 (  0.00%)    514756.00 (  3.01%)    487157.00 ( -2.51%)    510817.00 (  2.23%)    518296.00 (  3.72%)
TPut 33    494054.00 (  0.00%)    514193.00 (  4.08%)    499706.00 (  1.14%)    509734.00 (  3.17%)    508643.00 (  2.95%)
TPut 34    483213.00 (  0.00%)    512204.00 (  6.00%)    470473.00 ( -2.64%)    495556.00 (  2.55%)    510309.00 (  5.61%)
TPut 35    480068.00 (  0.00%)    506472.00 (  5.50%)    473011.00 ( -1.47%)    481812.00 (  0.36%)    513299.00 (  6.92%)
TPut 36    450114.00 (  0.00%)    506894.00 ( 12.61%)    477324.00 (  6.05%)    469524.00 (  4.31%)    486779.00 (  8.15%)
TPut 37    440154.00 (  0.00%)    506266.00 ( 15.02%)    459718.00 (  4.44%)    484438.00 ( 10.06%)    508613.00 ( 15.55%)
TPut 38    461536.00 (  0.00%)    499960.00 (  8.33%)    460417.00 ( -0.24%)    473332.00 (  2.56%)    514863.00 ( 11.55%)
TPut 39    460596.00 (  0.00%)    503353.00 (  9.28%)    461793.00 (  0.26%)    468966.00 (  1.82%)    498496.00 (  8.23%)
TPut 40    450746.00 (  0.00%)    497059.00 ( 10.27%)    440518.00 ( -2.27%)    446687.00 ( -0.90%)    497462.00 ( 10.36%)
TPut 41    456565.00 (  0.00%)    482549.00 (  5.69%)    451770.00 ( -1.05%)    444771.00 ( -2.58%)    491267.00 (  7.60%)
TPut 42    447258.00 (  0.00%)    490184.00 (  9.60%)    445414.00 ( -0.41%)    452870.00 (  1.25%)    469483.00 (  4.97%)
TPut 43    408979.00 (  0.00%)    467385.00 ( 14.28%)    416133.00 (  1.75%)    448137.00 (  9.57%)    486668.00 ( 19.00%)
TPut 44    415774.00 (  0.00%)    460710.00 ( 10.81%)    412956.00 ( -0.68%)    445221.00 (  7.08%)    479664.00 ( 15.37%)
TPut 45    419682.00 (  0.00%)    445293.00 (  6.10%)    411016.00 ( -2.06%)    426650.00 (  1.66%)    464358.00 ( 10.65%)
TPut 46    422698.00 (  0.00%)    425270.00 (  0.61%)    396870.00 ( -6.11%)    430955.00 (  1.95%)    460729.00 (  9.00%)
TPut 47    416622.00 (  0.00%)    395166.00 ( -5.15%)    403654.00 ( -3.11%)    397989.00 ( -4.47%)    463184.00 ( 11.18%)
TPut 48    409081.00 (  0.00%)    382490.00 ( -6.50%)    377057.00 ( -7.83%)    391214.00 ( -4.37%)    462186.00 ( 12.98%)

Mostly flat but it's interesting to note what a large impact the periodic migration of tasks to their preferred node is.

specjbb Peaks
3.11.0 3.11.0 3.11.0 3.11.0 3.11.0
account-v8r20 falsedetect-v8r33 skipshared-v8r33 noatomics-v8r33 periodretry-v8r33

 Expctd Warehouse          48.00 (  0.00%)          48.00 (  0.00%)          48.00 (  0.00%)          48.00 (  0.00%)          48.00 (  0.00%)
 Expctd Peak Bops      416622.00 (  0.00%)      395166.00 ( -5.15%)      403654.00 ( -3.11%)      397989.00 ( -4.47%)      463184.00 ( 11.18%)
 Actual Warehouse          27.00 (  0.00%)          32.00 ( 18.52%)          27.00 (  0.00%)          27.00 (  0.00%)          30.00 ( 11.11%)
 Actual Peak Bops      521824.00 (  0.00%)      520700.00 ( -0.22%)      522584.00 (  0.15%)      529076.00 (  1.39%)      526196.00 (  0.84%)
 SpecJBB Bops            8349.00 (  0.00%)        7806.00 ( -6.50%)        7695.00 ( -7.83%)        7984.00 ( -4.37%)        9432.00 ( 12.97%)
 SpecJBB Bops/JVM        8349.00 (  0.00%)        7806.00 ( -6.50%)        7695.00 ( -7.83%)        7984.00 ( -4.37%)        9432.00 ( 12.97%)

Same.

              3.11.0      3.11.0      3.11.0      3.11.0      3.11.0
        account-v8r20falsedetect-v8r33skipshared-v8r33noatomics-v8r33periodretry-v8r33

User 43797.04 44575.08 44571.62 44544.22 44591.39
System 863.95 135.20 151.93 137.42 165.06
Elapsed 1665.51 1665.06 1665.30 1665.22 1665.06

Big reduction in system CPU usage.

                                3.11.0      3.11.0      3.11.0      3.11.0      3.11.0
                          account-v8r20falsedetect-v8r33skipshared-v8r33noatomics-v8r33periodretry-v8r33
Compaction cost                 135535       13678       17594       15600       13820
NUMA PTE updates              19815264      697698     3495802     3195079     3668267
NUMA hint faults               4029907      694556     3471637     3167057     3638629
NUMA hint local faults         1109183      239859      985693      894926     1148882

NUMA hint local percent 27 34 28 28 31
NUMA pages migrated 130574018 13177453 16950408 15029583 13314876
AutoNUMA cost 22769 3728 17704 16143 18471

Big reduction in the number of PTE updates but similar-ish fault statistics. We're getting roughly the same benefit for much less work.

autonumabench
3.11.0 3.11.0 3.11.0 3.11.0 3.11.0
account-v8r20 falsedetect-v8r33 skipshared-v8r33 noatomics-v8r33 periodretry-v8r33

User    NUMA01               44031.89 (  0.00%)    60916.77 (-38.35%)    62988.76 (-43.05%)    62187.81 (-41.23%)    35403.01 ( 19.60%)
User    NUMA01_THEADLOCAL    16956.40 (  0.00%)    17665.21 ( -4.18%)    17809.92 ( -5.03%)    17605.32 ( -3.83%)    17238.74 ( -1.67%)
User    NUMA02                2044.83 (  0.00%)     2054.14 ( -0.46%)     2106.18 ( -3.00%)     2058.25 ( -0.66%)     2065.25 ( -1.00%)
User    NUMA02_SMT             982.52 (  0.00%)      978.55 (  0.40%)      985.73 ( -0.33%)      975.96 (  0.67%)     1001.48 ( -1.93%)
System  NUMA01                1038.02 (  0.00%)      320.44 ( 69.13%)      440.97 ( 57.52%)      179.55 ( 82.70%)      333.93 ( 67.83%)
System  NUMA01_THEADLOCAL      323.01 (  0.00%)      125.90 ( 61.02%)      132.59 ( 58.95%)      123.52 ( 61.76%)      171.04 ( 47.05%)
System  NUMA02                  10.26 (  0.00%)       11.34 (-10.53%)        9.97 (  2.83%)        9.07 ( 11.60%)        6.08 ( 40.74%)
System  NUMA02_SMT               3.34 (  0.00%)        3.43 ( -2.69%)        5.45 (-63.17%)        3.42 ( -2.40%)        3.40 ( -1.80%)
Elapsed NUMA01                 999.90 (  0.00%)     1363.26 (-36.34%)     1409.55 (-40.97%)     1386.34 (-38.65%)      792.81 ( 20.71%)
Elapsed NUMA01_THEADLOCAL      374.82 (  0.00%)      395.48 ( -5.51%)      399.18 ( -6.50%)      393.90 ( -5.09%)      388.22 ( -3.58%)
Elapsed NUMA02                  48.66 (  0.00%)       50.15 ( -3.06%)       52.75 ( -8.41%)       50.81 ( -4.42%)       48.48 (  0.37%)
Elapsed NUMA02_SMT              46.17 (  0.00%)       46.72 ( -1.19%)       46.51 ( -0.74%)       45.98 (  0.41%)       57.64 (-24.84%)
CPU     NUMA01                4507.00 (  0.00%)     4491.00 (  0.36%)     4499.00 (  0.18%)     4498.00 (  0.20%)     4507.00 (  0.00%)
CPU     NUMA01_THEADLOCAL     4609.00 (  0.00%)     4498.00 (  2.41%)     4494.00 (  2.50%)     4500.00 (  2.36%)     4484.00 (  2.71%)
CPU     NUMA02                4222.00 (  0.00%)     4117.00 (  2.49%)     4011.00 (  5.00%)     4068.00 (  3.65%)     4272.00 ( -1.18%)
CPU     NUMA02_SMT            2135.00 (  0.00%)     2101.00 (  1.59%)     2130.00 (  0.23%)     2129.00 (  0.28%)     1743.00 ( 18.36%)

In general, the numa01 (adverse workload) is hurt by the series until the periodic retry patch is merged. In general that patch helps a lot except in the case of numa02_SMT where it punished severely. It's odd because the amount of system CPU usage that workload incurs is tiny.

This is SpecJBB running on a 4-socket machine with THP enabled and one JVM running per node on the system.

specjbb
3.11.0 3.11.0 3.11.0 3.11.0 3.11.0
account-v8r20 falsedetect-v8r33 skipshared-v8r33 noatomics-v8r33 periodretry-v8r33

Mean   1      31915.75 (  0.00%)     30007.50 ( -5.98%)     29590.50 ( -7.29%)     30562.75 ( -4.24%)     28888.75 ( -9.48%)
Mean   2      61914.50 (  0.00%)     60854.00 ( -1.71%)     62163.25 (  0.40%)     60594.25 ( -2.13%)     62521.75 (  0.98%)
Mean   3      89693.25 (  0.00%)     91159.25 (  1.63%)     91939.00 (  2.50%)     93235.50 (  3.95%)     94956.75 (  5.87%)
Mean   4     115687.50 (  0.00%)    117438.00 (  1.51%)    119940.50 (  3.68%)    124273.75 (  7.42%)    124600.75 (  7.70%)
Mean   5     142789.25 (  0.00%)    141024.00 ( -1.24%)    146349.00 (  2.49%)    152133.25 (  6.54%)    150575.00 (  5.45%)
Mean   6     153345.25 (  0.00%)    160091.25 (  4.40%)    160991.00 (  4.99%)    174128.50 ( 13.55%)    176533.50 ( 15.12%)
Mean   7     153004.00 (  0.00%)    160972.00 (  5.21%)    180219.75 ( 17.79%)    176605.25 ( 15.43%)    179010.25 ( 17.00%)
Mean   8     151906.00 (  0.00%)    161832.50 (  6.53%)    164960.75 (  8.59%)    169345.25 ( 11.48%)    181374.75 ( 19.40%)
Mean   9     154198.50 (  0.00%)    148888.25 ( -3.44%)    167819.75 (  8.83%)    169039.75 (  9.62%)    181401.75 ( 17.64%)
Mean   10    153377.00 (  0.00%)    143134.75 ( -6.68%)    155716.00 (  1.53%)    161498.25 (  5.29%)    175450.50 ( 14.39%)
Mean   11    138278.50 (  0.00%)    139928.50 (  1.19%)    149627.25 (  8.21%)    155845.75 ( 12.70%)    170797.75 ( 23.52%)
Mean   12    128598.00 (  0.00%)    134543.50 (  4.62%)    142626.50 ( 10.91%)    141966.75 ( 10.40%)    163766.50 ( 27.35%)
Mean   13    130092.00 (  0.00%)    134991.25 (  3.77%)    137179.50 (  5.45%)    140382.75 (  7.91%)    159106.00 ( 22.30%)
Mean   14    121786.25 (  0.00%)    134701.25 ( 10.60%)    134942.75 ( 10.80%)    140522.50 ( 15.38%)    156676.25 ( 28.65%)
Mean   15    121842.25 (  0.00%)    134521.00 ( 10.41%)    134251.25 ( 10.18%)    136963.50 ( 12.41%)    154455.50 ( 26.77%)
Mean   16    120808.25 (  0.00%)    130800.50 (  8.27%)    136500.50 ( 12.99%)    134488.50 ( 11.32%)    150137.75 ( 24.28%)
Mean   17    118703.25 (  0.00%)    130281.00 (  9.75%)    134345.00 ( 13.18%)    132618.75 ( 11.72%)    147027.50 ( 23.86%)
Mean   18    116243.50 (  0.00%)    128010.75 ( 10.12%)    130742.75 ( 12.47%)    128031.25 ( 10.14%)    145852.50 ( 25.47%)
Mean   19    116508.75 (  0.00%)    127627.25 (  9.54%)    127561.50 (  9.49%)    123765.25 (  6.23%)    140879.25 ( 20.92%)
Mean   20    117386.50 (  0.00%)    133369.25 ( 13.62%)    126746.00 (  7.97%)    127327.75 (  8.47%)    141559.25 ( 20.59%)
Mean   21    118308.00 (  0.00%)    129000.25 (  9.04%)    127664.25 (  7.91%)    125308.50 (  5.92%)    138110.75 ( 16.74%)
Mean   22    116577.75 (  0.00%)    130031.50 ( 11.54%)    127755.75 (  9.59%)    129445.75 ( 11.04%)    136937.50 ( 17.46%)
Mean   23    117444.50 (  0.00%)    125593.50 (  6.94%)    125423.25 (  6.79%)    121178.50 (  3.18%)    134960.25 ( 14.91%)
Mean   24    108114.00 (  0.00%)    121164.25 ( 12.07%)    117293.75 (  8.49%)    119736.25 ( 10.75%)    130200.25 ( 20.43%)
Mean   25    114144.50 (  0.00%)    118106.50 (  3.47%)    118405.00 (  3.73%)    127544.75 ( 11.74%)    133198.50 ( 16.69%)
Mean   26    111531.00 (  0.00%)    120096.00 (  7.68%)    124074.50 ( 11.25%)    131079.50 ( 17.53%)    133463.50 ( 19.66%)
Mean   27    113381.50 (  0.00%)    121547.75 (  7.20%)    121492.75 (  7.15%)    127801.25 ( 12.72%)    135127.25 ( 19.18%)
Mean   28    111333.50 (  0.00%)    118054.25 (  6.04%)    121414.00 (  9.05%)    134754.00 ( 21.04%)    133270.00 ( 19.70%)
Mean   29    108411.25 (  0.00%)    119588.00 ( 10.31%)    123334.50 ( 13.77%)    132048.25 ( 21.80%)    131892.50 ( 21.66%)
Mean   30    110572.00 (  0.00%)    119673.50 (  8.23%)    118952.00 (  7.58%)    127541.50 ( 15.35%)    132982.00 ( 20.27%)
Mean   31    114267.00 (  0.00%)    117367.75 (  2.71%)    119463.75 (  4.55%)    126795.75 ( 10.96%)    133970.00 ( 17.24%)
Mean   32    112054.50 (  0.00%)    115357.00 (  2.95%)    111839.00 ( -0.19%)    126267.00 ( 12.68%)    133289.00 ( 18.95%)
Mean   33    113664.50 (  0.00%)    115657.50 (  1.75%)    119608.00 (  5.23%)    124996.50 (  9.97%)    135356.00 ( 19.08%)
Mean   34    110793.00 (  0.00%)    115023.00 (  3.82%)    122672.25 ( 10.72%)    124025.50 ( 11.94%)    135818.00 ( 22.59%)
Mean   35    116855.75 (  0.00%)    118724.00 (  1.60%)    115770.50 ( -0.93%)    129668.50 ( 10.96%)    136327.00 ( 16.66%)
Mean   36    112375.25 (  0.00%)    121457.50 (  8.08%)    116098.00 (  3.31%)    125627.25 ( 11.79%)    134052.25 ( 19.29%)
Mean   37    109171.25 (  0.00%)    119452.75 (  9.42%)    119359.75 (  9.33%)    119926.25 (  9.85%)    133200.75 ( 22.01%)
Mean   38    105759.50 (  0.00%)    124684.00 ( 17.89%)    117629.25 ( 11.22%)    119691.25 ( 13.17%)    132992.75 ( 25.75%)
Mean   39    110185.00 (  0.00%)    125151.75 ( 13.58%)    112839.00 (  2.41%)    117929.00 (  7.03%)    133400.75 ( 21.07%)
Mean   40    110683.50 (  0.00%)    122496.50 ( 10.67%)    118156.50 (  6.75%)    120739.00 (  9.08%)    132520.25 ( 19.73%)
Mean   41    102152.00 (  0.00%)    123401.50 ( 20.80%)    114519.00 ( 12.11%)    119549.00 ( 17.03%)    132827.00 ( 30.03%)
Mean   42    105378.50 (  0.00%)    115035.25 (  9.16%)    110092.25 (  4.47%)    119086.75 ( 13.01%)    130747.75 ( 24.07%)
Mean   43    107112.00 (  0.00%)    112384.75 (  4.92%)    110925.75 (  3.56%)    112715.75 (  5.23%)    133097.00 ( 24.26%)
Mean   44    105611.25 (  0.00%)    118682.75 ( 12.38%)    106846.50 (  1.17%)    114581.25 (  8.49%)    131913.25 ( 24.90%)
Mean   45    105092.25 (  0.00%)    116890.50 ( 11.23%)    110318.25 (  4.97%)    113010.00 (  7.53%)    130818.00 ( 24.48%)
Mean   46    111016.25 (  0.00%)    111954.00 (  0.84%)    106253.75 ( -4.29%)    111746.00 (  0.66%)    128427.25 ( 15.68%)
Mean   47    106660.75 (  0.00%)    116336.00 (  9.07%)    110024.00 (  3.15%)    109703.50 (  2.85%)    128223.50 ( 20.22%)
Mean   48    103423.75 (  0.00%)    115325.00 ( 11.51%)    108381.50 (  4.79%)    112662.50 (  8.93%)    128536.75 ( 24.28%)
Stddev 1       1309.96 (  0.00%)      1796.84 (-37.17%)      1529.84 (-16.78%)       725.83 ( 44.59%)      1207.28 (  7.84%)
Stddev 2        900.52 (  0.00%)       954.05 ( -5.94%)      2856.62 (-217.22%)      2730.12 (-203.17%)      2743.74 (-204.68%)
Stddev 3        845.76 (  0.00%)       778.56 (  7.95%)      3350.11 (-296.11%)      2794.68 (-230.43%)      1923.88 (-127.47%)
Stddev 4       4988.11 (  0.00%)      2321.83 ( 53.45%)      4635.26 (  7.07%)       867.46 ( 82.61%)      1288.21 ( 74.17%)
Stddev 5       5969.57 (  0.00%)      1605.41 ( 73.11%)      5415.52 (  9.28%)      2328.13 ( 61.00%)      1844.24 ( 69.11%)
Stddev 6      12478.42 (  0.00%)      3204.26 ( 74.32%)      4928.12 ( 60.51%)      1908.73 ( 84.70%)      1109.97 ( 91.10%)
Stddev 7       5975.15 (  0.00%)      5954.29 (  0.35%)      2558.42 ( 57.18%)      2245.10 ( 62.43%)       894.47 ( 85.03%)
Stddev 8       3404.47 (  0.00%)     16525.39 (-385.40%)      8166.52 (-139.88%)      6520.79 (-91.54%)      1233.67 ( 63.76%)
Stddev 9       8811.92 (  0.00%)     17373.77 (-97.16%)      8384.83 (  4.85%)      4511.21 ( 48.81%)      6092.53 ( 30.86%)
Stddev 10      4050.40 (  0.00%)     16781.65 (-314.32%)      9299.40 (-129.59%)      7901.83 (-95.09%)      7086.69 (-74.96%)
Stddev 11      9475.98 (  0.00%)     22925.45 (-141.93%)     16135.16 (-70.27%)     15935.55 (-68.17%)     12776.76 (-34.83%)
Stddev 12      1961.27 (  0.00%)     17661.49 (-800.51%)     12080.97 (-515.98%)     13293.01 (-577.77%)     16113.60 (-721.59%)
Stddev 13      2164.98 (  0.00%)     20902.15 (-865.47%)     13747.72 (-535.00%)     10093.27 (-366.21%)     17382.57 (-702.90%)
Stddev 14      4186.32 (  0.00%)     13552.26 (-223.73%)     15955.56 (-281.14%)      9782.02 (-133.67%)     17469.49 (-317.30%)
Stddev 15      1766.42 (  0.00%)     14148.85 (-700.99%)     10003.01 (-466.29%)      7777.42 (-340.29%)     17940.51 (-915.64%)
Stddev 16      4116.25 (  0.00%)     17131.59 (-316.19%)      9714.07 (-135.99%)      9721.54 (-136.17%)     16802.13 (-308.19%)
Stddev 17      4252.36 (  0.00%)     14060.30 (-230.65%)      7048.48 (-65.75%)      6797.22 (-59.85%)     14995.28 (-252.63%)
Stddev 18      4421.47 (  0.00%)     15675.15 (-254.52%)      6404.13 (-44.84%)      8467.89 (-91.52%)     14654.80 (-231.45%)
Stddev 19      1969.40 (  0.00%)     13932.78 (-607.46%)      8825.82 (-348.15%)      8576.52 (-335.49%)     14659.18 (-644.35%)
Stddev 20      4204.79 (  0.00%)      9151.72 (-117.65%)      6336.02 (-50.69%)      4104.31 (  2.39%)     13911.10 (-230.84%)
Stddev 21      2319.45 (  0.00%)      7818.64 (-237.09%)      5782.65 (-149.31%)      9329.94 (-302.25%)     15425.22 (-565.04%)
Stddev 22      1704.55 (  0.00%)      8563.76 (-402.40%)      4382.12 (-157.08%)      5264.66 (-208.86%)     14777.92 (-766.97%)
Stddev 23      4454.10 (  0.00%)      6454.89 (-44.92%)     10773.06 (-141.87%)      8137.56 (-82.70%)     12067.61 (-170.93%)
Stddev 24      3134.81 (  0.00%)      9448.45 (-201.40%)      5911.90 (-88.59%)      8289.99 (-164.45%)     19696.90 (-528.33%)
Stddev 25      1946.32 (  0.00%)      9054.46 (-365.21%)      5994.17 (-207.98%)      5082.87 (-161.15%)     15444.45 (-693.52%)
Stddev 26      6694.44 (  0.00%)      8657.01 (-29.32%)      6814.85 ( -1.80%)      3582.44 ( 46.49%)     15096.81 (-125.51%)
Stddev 27      5159.46 (  0.00%)      7280.85 (-41.12%)      4058.33 ( 21.34%)      6129.48 (-18.80%)     14158.65 (-174.42%)
Stddev 28      2665.79 (  0.00%)     10345.52 (-288.08%)      2327.74 ( 12.68%)      1748.34 ( 34.42%)     14139.33 (-430.40%)
Stddev 29      6353.05 (  0.00%)      7672.72 (-20.77%)      6876.07 ( -8.23%)      4385.79 ( 30.97%)     13514.71 (-112.73%)
Stddev 30      5378.33 (  0.00%)      8665.82 (-61.12%)      6681.09 (-24.22%)      7983.87 (-48.45%)     14576.47 (-171.02%)
Stddev 31     10359.52 (  0.00%)     10179.47 (  1.74%)      5075.29 ( 51.01%)      4198.25 ( 59.47%)     14020.78 (-35.34%)
Stddev 32      9090.09 (  0.00%)     17015.37 (-87.19%)      6124.63 ( 32.62%)      5635.11 ( 38.01%)     14899.79 (-63.91%)
Stddev 33      7143.20 (  0.00%)     15988.55 (-123.83%)      7464.48 ( -4.50%)      3019.97 ( 57.72%)     13049.06 (-82.68%)
Stddev 34      4954.01 (  0.00%)      8884.64 (-79.34%)      7827.59 (-58.01%)      4385.45 ( 11.48%)     13382.51 (-170.14%)
Stddev 35      6247.59 (  0.00%)      9136.41 (-46.24%)      4600.77 ( 26.36%)      3211.43 ( 48.60%)     11547.88 (-84.84%)
Stddev 36      7325.84 (  0.00%)      7969.69 ( -8.79%)      5746.70 ( 21.56%)      6279.05 ( 14.29%)     13622.23 (-85.95%)
Stddev 37      7952.43 (  0.00%)     11648.76 (-46.48%)      6065.84 ( 23.72%)      6666.17 ( 16.17%)     12860.81 (-61.72%)
Stddev 38      6473.08 (  0.00%)      6465.75 (  0.11%)      4958.70 ( 23.40%)      6539.37 ( -1.02%)     15220.62 (-135.14%)
Stddev 39      2568.69 (  0.00%)      5494.22 (-113.89%)      5649.24 (-119.93%)      5527.10 (-115.17%)     13685.29 (-432.77%)
Stddev 40      2823.65 (  0.00%)      7118.16 (-152.09%)      1970.80 ( 30.20%)      5372.43 (-90.27%)     13223.53 (-368.31%)
Stddev 41      2145.41 (  0.00%)      5532.78 (-157.89%)      3929.36 (-83.15%)      6514.51 (-203.65%)     13346.51 (-522.10%)
Stddev 42      2334.79 (  0.00%)      7695.64 (-229.61%)      5361.91 (-129.65%)      5845.25 (-150.35%)     13245.12 (-467.29%)
Stddev 43      8940.43 (  0.00%)     14286.16 (-59.79%)      9110.82 ( -1.91%)      8481.12 (  5.14%)     14413.92 (-61.22%)
Stddev 44      6738.12 (  0.00%)     10145.70 (-50.57%)      4789.94 ( 28.91%)      7463.67 (-10.77%)     15134.45 (-124.61%)
Stddev 45      3219.83 (  0.00%)      6743.00 (-109.42%)      7629.10 (-136.94%)      5441.27 (-68.99%)     13401.43 (-316.22%)
Stddev 46      7925.89 (  0.00%)      9956.81 (-25.62%)      9729.49 (-22.76%)     10130.88 (-27.82%)     14020.22 (-76.89%)
Stddev 47      7034.02 (  0.00%)     12456.90 (-77.09%)     12705.68 (-80.63%)      5117.46 ( 27.25%)     13224.66 (-88.01%)
Stddev 48     11774.16 (  0.00%)      8678.39 ( 26.29%)     11906.07 ( -1.12%)      5293.32 ( 55.04%)     13968.76 (-18.64%)
TPut   1     127663.00 (  0.00%)    120030.00 ( -5.98%)    118362.00 ( -7.29%)    122251.00 ( -4.24%)    115555.00 ( -9.48%)
TPut   2     247658.00 (  0.00%)    243416.00 ( -1.71%)    248653.00 (  0.40%)    242377.00 ( -2.13%)    250087.00 (  0.98%)
TPut   3     358773.00 (  0.00%)    364637.00 (  1.63%)    367756.00 (  2.50%)    372942.00 (  3.95%)    379827.00 (  5.87%)
TPut   4     462750.00 (  0.00%)    469752.00 (  1.51%)    479762.00 (  3.68%)    497095.00 (  7.42%)    498403.00 (  7.70%)
TPut   5     571157.00 (  0.00%)    564096.00 ( -1.24%)    585396.00 (  2.49%)    608533.00 (  6.54%)    602300.00 (  5.45%)
TPut   6     613381.00 (  0.00%)    640365.00 (  4.40%)    643964.00 (  4.99%)    696514.00 ( 13.55%)    706134.00 ( 15.12%)
TPut   7     612016.00 (  0.00%)    643888.00 (  5.21%)    720879.00 ( 17.79%)    706421.00 ( 15.43%)    716041.00 ( 17.00%)
TPut   8     607624.00 (  0.00%)    647330.00 (  6.53%)    659843.00 (  8.59%)    677381.00 ( 11.48%)    725499.00 ( 19.40%)
TPut   9     616794.00 (  0.00%)    595553.00 ( -3.44%)    671279.00 (  8.83%)    676159.00 (  9.62%)    725607.00 ( 17.64%)
TPut   10    613508.00 (  0.00%)    572539.00 ( -6.68%)    622864.00 (  1.53%)    645993.00 (  5.29%)    701802.00 ( 14.39%)
TPut   11    553114.00 (  0.00%)    559714.00 (  1.19%)    598509.00 (  8.21%)    623383.00 ( 12.70%)    683191.00 ( 23.52%)
TPut   12    514392.00 (  0.00%)    538174.00 (  4.62%)    570506.00 ( 10.91%)    567867.00 ( 10.40%)    655066.00 ( 27.35%)
TPut   13    520368.00 (  0.00%)    539965.00 (  3.77%)    548718.00 (  5.45%)    561531.00 (  7.91%)    636424.00 ( 22.30%)
TPut   14    487145.00 (  0.00%)    538805.00 ( 10.60%)    539771.00 ( 10.80%)    562090.00 ( 15.38%)    626705.00 ( 28.65%)
TPut   15    487369.00 (  0.00%)    538084.00 ( 10.41%)    537005.00 ( 10.18%)    547854.00 ( 12.41%)    617822.00 ( 26.77%)
TPut   16    483233.00 (  0.00%)    523202.00 (  8.27%)    546002.00 ( 12.99%)    537954.00 ( 11.32%)    600551.00 ( 24.28%)
TPut   17    474813.00 (  0.00%)    521124.00 (  9.75%)    537380.00 ( 13.18%)    530475.00 ( 11.72%)    588110.00 ( 23.86%)
TPut   18    464974.00 (  0.00%)    512043.00 ( 10.12%)    522971.00 ( 12.47%)    512125.00 ( 10.14%)    583410.00 ( 25.47%)
TPut   19    466035.00 (  0.00%)    510509.00 (  9.54%)    510246.00 (  9.49%)    495061.00 (  6.23%)    563517.00 ( 20.92%)
TPut   20    469546.00 (  0.00%)    533477.00 ( 13.62%)    506984.00 (  7.97%)    509311.00 (  8.47%)    566237.00 ( 20.59%)
TPut   21    473232.00 (  0.00%)    516001.00 (  9.04%)    510657.00 (  7.91%)    501234.00 (  5.92%)    552443.00 ( 16.74%)
TPut   22    466311.00 (  0.00%)    520126.00 ( 11.54%)    511023.00 (  9.59%)    517783.00 ( 11.04%)    547750.00 ( 17.46%)
TPut   23    469778.00 (  0.00%)    502374.00 (  6.94%)    501693.00 (  6.79%)    484714.00 (  3.18%)    539841.00 ( 14.91%)
TPut   24    432456.00 (  0.00%)    484657.00 ( 12.07%)    469175.00 (  8.49%)    478945.00 ( 10.75%)    520801.00 ( 20.43%)
TPut   25    456578.00 (  0.00%)    472426.00 (  3.47%)    473620.00 (  3.73%)    510179.00 ( 11.74%)    532794.00 ( 16.69%)
TPut   26    446124.00 (  0.00%)    480384.00 (  7.68%)    496298.00 ( 11.25%)    524318.00 ( 17.53%)    533854.00 ( 19.66%)
TPut   27    453526.00 (  0.00%)    486191.00 (  7.20%)    485971.00 (  7.15%)    511205.00 ( 12.72%)    540509.00 ( 19.18%)
TPut   28    445334.00 (  0.00%)    472217.00 (  6.04%)    485656.00 (  9.05%)    539016.00 ( 21.04%)    533080.00 ( 19.70%)
TPut   29    433645.00 (  0.00%)    478352.00 ( 10.31%)    493338.00 ( 13.77%)    528193.00 ( 21.80%)    527570.00 ( 21.66%)
TPut   30    442288.00 (  0.00%)    478694.00 (  8.23%)    475808.00 (  7.58%)    510166.00 ( 15.35%)    531928.00 ( 20.27%)
TPut   31    457068.00 (  0.00%)    469471.00 (  2.71%)    477855.00 (  4.55%)    507183.00 ( 10.96%)    535880.00 ( 17.24%)
TPut   32    448218.00 (  0.00%)    461428.00 (  2.95%)    447356.00 ( -0.19%)    505068.00 ( 12.68%)    533156.00 ( 18.95%)
TPut   33    454658.00 (  0.00%)    462630.00 (  1.75%)    478432.00 (  5.23%)    499986.00 (  9.97%)    541424.00 ( 19.08%)
TPut   34    443172.00 (  0.00%)    460092.00 (  3.82%)    490689.00 ( 10.72%)    496102.00 ( 11.94%)    543272.00 ( 22.59%)
TPut   35    467423.00 (  0.00%)    474896.00 (  1.60%)    463082.00 ( -0.93%)    518674.00 ( 10.96%)    545308.00 ( 16.66%)
TPut   36    449501.00 (  0.00%)    485830.00 (  8.08%)    464392.00 (  3.31%)    502509.00 ( 11.79%)    536209.00 ( 19.29%)
TPut   37    436685.00 (  0.00%)    477811.00 (  9.42%)    477439.00 (  9.33%)    479705.00 (  9.85%)    532803.00 ( 22.01%)
TPut   38    423038.00 (  0.00%)    498736.00 ( 17.89%)    470517.00 ( 11.22%)    478765.00 ( 13.17%)    531971.00 ( 25.75%)
TPut   39    440740.00 (  0.00%)    500607.00 ( 13.58%)    451356.00 (  2.41%)    471716.00 (  7.03%)    533603.00 ( 21.07%)
TPut   40    442734.00 (  0.00%)    489986.00 ( 10.67%)    472626.00 (  6.75%)    482956.00 (  9.08%)    530081.00 ( 19.73%)
TPut   41    408608.00 (  0.00%)    493606.00 ( 20.80%)    458076.00 ( 12.11%)    478196.00 ( 17.03%)    531308.00 ( 30.03%)
TPut   42    421514.00 (  0.00%)    460141.00 (  9.16%)    440369.00 (  4.47%)    476347.00 ( 13.01%)    522991.00 ( 24.07%)
TPut   43    428448.00 (  0.00%)    449539.00 (  4.92%)    443703.00 (  3.56%)    450863.00 (  5.23%)    532388.00 ( 24.26%)
TPut   44    422445.00 (  0.00%)    474731.00 ( 12.38%)    427386.00 (  1.17%)    458325.00 (  8.49%)    527653.00 ( 24.90%)
TPut   45    420369.00 (  0.00%)    467562.00 ( 11.23%)    441273.00 (  4.97%)    452040.00 (  7.53%)    523272.00 ( 24.48%)
TPut   46    444065.00 (  0.00%)    447816.00 (  0.84%)    425015.00 ( -4.29%)    446984.00 (  0.66%)    513709.00 ( 15.68%)
TPut   47    426643.00 (  0.00%)    465344.00 (  9.07%)    440096.00 (  3.15%)    438814.00 (  2.85%)    512894.00 ( 20.22%)
TPut   48    413695.00 (  0.00%)    461300.00 ( 11.51%)    433526.00 (  4.79%)    450650.00 (  8.93%)    514147.00 ( 24.28%)

This is looking much better overall and again the impact of the continual retries of migration makes a big difference. It indicates that it would be worth investigating why the retries are even necessary. It would be unfortunate if NUMA scheduling was constantly battling the load balancer.

specjbb Peaks
3.11.0 3.11.0 3.11.0 3.11.0 3.11.0
account-v8r20 falsedetect-v8r33 skipshared-v8r33 noatomics-v8r33 periodretry-v8r33

 Expctd Warehouse          12.00 (  0.00%)          12.00 (  0.00%)          12.00 (  0.00%)          12.00 (  0.00%)          12.00 (  0.00%)
 Expctd Peak Bops      553114.00 (  0.00%)      559714.00 (  1.19%)      598509.00 (  8.21%)      623383.00 ( 12.70%)      683191.00 ( 23.52%)
 Actual Warehouse          10.00 (  0.00%)           9.00 (-10.00%)           8.00 (-20.00%)           8.00 (-20.00%)          10.00 (  0.00%)
 Actual Peak Bops      616794.00 (  0.00%)      647330.00 (  4.95%)      720879.00 ( 16.88%)      706421.00 ( 14.53%)      725607.00 ( 17.64%)
 SpecJBB Bops          477666.00 (  0.00%)      521426.00 (  9.16%)      524010.00 (  9.70%)      523610.00 (  9.62%)      584513.00 ( 22.37%)
 SpecJBB Bops/JVM      119417.00 (  0.00%)      130357.00 (  9.16%)      131003.00 (  9.70%)      130903.00 (  9.62%)      146128.00 ( 22.37%)

Speaks for itself really, looks great.

              3.11.0      3.11.0      3.11.0      3.11.0      3.11.0
        account-v8r20falsedetect-v8r33skipshared-v8r33noatomics-v8r33periodretry-v8r33

User 466719.69 478006.22 476399.23 476733.11 476517.65
System 11049.55 913.00 1429.31 1317.87 994.02
Elapsed 10381.80 10415.18 10384.39 10383.33 10393.91

and system CPU usage is still way down so now we are seeing large improvements for less work.

                                3.11.0      3.11.0      3.11.0      3.11.0      3.11.0
                          account-v8r20falsedetect-v8r33skipshared-v8r33noatomics-v8r33periodretry-v8r33
Compaction cost                1432168       77123      127225      119169       66598
NUMA PTE updates             505320169     9237242    44679605    40306037    40633804
NUMA hint faults              67707706     9240325    44739890    40798082    40521421
NUMA hint local faults        22183991     3313857    14359951    13135180    25014367

NUMA hint local percent 32 35 32 32 61
NUMA pages migrated 1379738662 74299705 122568192 114806428 64160571
AutoNUMA cost 368290 47677 226341 206453 204110

That migrate retry thing does wonders for the local fault percentage. Really worth checking out why tasks are getting migrated off their preferred node.

The NAS parallel benchmarks had not run at the time of writing but I decided not to delay releasing this anyway.

Overall this series is now looking much better. The hotplug handling part is still missing but hopefully with Peter overhead reduction efforts it will not impair the results much. The primary curiousity for me is why the period retry makes such a difference because in my mind it implies the load balancer and numa scheduler are working against each other for some reason. Even without that answer I think this should get some time in tip when the hotplug part is filled in.

 Documentation/sysctl/kernel.txt   |   76 ++
 arch/x86/mm/numa.c                |    6 +-
 fs/exec.c                         |    1 +
 fs/proc/array.c                   |    2 +
 include/linux/mempolicy.h         |    1 +
 include/linux/migrate.h           |    7 +-
 include/linux/mm.h                |  118 ++-
 include/linux/mm_types.h          |   17 +-
 include/linux/page-flags-layout.h |   28 +-
 include/linux/sched.h             |   63 +-
 include/linux/sched/sysctl.h      |    1 -
 include/linux/stop_machine.h      |    1 +
 kernel/bounds.c                   |    4 +
 kernel/fork.c                     |    5 +-
 kernel/sched/core.c               |  197 ++++-
 kernel/sched/debug.c              |   60 +-
 kernel/sched/fair.c               | 1608 +++++++++++++++++++++++++++++--------
 kernel/sched/features.h           |   19 +-
 kernel/sched/idle_task.c          |    2 +-
 kernel/sched/rt.c                 |    5 +-
 kernel/sched/sched.h              |   27 +-
 kernel/sched/stop_task.c          |    2 +-
 kernel/stop_machine.c             |  272 ++++---
 kernel/sysctl.c                   |   21 +-
 lib/vsprintf.c                    |    5 +
 mm/huge_memory.c                  |  119 ++-
 mm/memory.c                       |  158 ++--
 mm/mempolicy.c                    |   82 +-
 mm/migrate.c                      |   49 +-
 mm/mm_init.c                      |   18 +-
 mm/mmzone.c                       |   14 +-
 mm/mprotect.c                     |   65 +-
 mm/page_alloc.c                   |    4 +-
 33 files changed, 2307 insertions(+), 750 deletions(-)
Signature · 1 line
1.8.1.4
author reviewer
#1 lore raw

PATCH 1/63 sched: monolithic code dump of what is being pushed upstream

From: Peter Zijlstra <[email protected]>

So I have the below patch in front of all your patches. It contains the 10 or so sched,fair patches I posted to lkml the other day.

I used these to poke at the group_imb crud, am now digging through traces of perf bench numa to see if there's anything else I need.

like said on IRC: I boot with ftrace=nop to ensure we allocate properly sized trace buffers. This can also be done at runtime by switching active tracer -- this allocates the default buffer size, or by explicitly setting a per-cpu buffer size in /debug/tracing/buffer_size_kb. By default the thing allocates a single page per cpu or something uselessly small like that.

I then run a benchmark and at an appropriate time (eg. when I see something 'weird' happen) I do something like:

  echo 0 > /debug/tracing/tracing_on  # disable writing into the buffers
  cat /debug/tracing/trace > ~/trace  # dump to file
  echo 0 > /debug/tracing/trace       # reset buffers
  echo 1 > /debug/tracing/tracing_on  # enable writing to the buffers

[ Note I mount debugfs at /debug, this is not the default location but I
think the rest of the world is wrong ;-) ]

Also, the brain seems to adapt once you're staring at them for longer than a day -- yay for human pattern recognition skillz.

Ingo tends to favour more verbose dumps, I tend to favour minimal dumps.. whatever works for you is something you'll learn with experience.

---
 arch/x86/mm/numa.c   |   6 +-
 kernel/sched/core.c  |  18 +-
 kernel/sched/fair.c  | 498 ++++++++++++++++++++++++++++-----------------------
 kernel/sched/sched.h |   1 +
 lib/vsprintf.c       |   5 +
 5 files changed, 288 insertions(+), 240 deletions(-)
arch/x86/mm/numa.c +2 -4
index 8bf93ba..4ed4612 100644
@@ -737,7 +737,6 @@ int early_cpu_to_node(int cpu)
 void debug_cpumask_set_cpu(int cpu, int node, bool enable)
 {
 	struct cpumask *mask;
-	char buf[64];
 
 	if (node == NUMA_NO_NODE) {
 		/* early_cpu_to_node() already emits a warning and trace */
@@ -755,10 +754,9 @@ void debug_cpumask_set_cpu(int cpu, int node, bool enable)
 	else
 		cpumask_clear_cpu(cpu, mask);
 
-	cpulist_scnprintf(buf, sizeof(buf), mask);
-	printk(KERN_DEBUG "%s cpu %d node %d: mask now %s\n",
+	printk(KERN_DEBUG "%s cpu %d node %d: mask now %pc\n",
 		enable ? "numa_add_cpu" : "numa_remove_cpu",
-		cpu, node, buf);
+		cpu, node, mask);
 	return;
 }
 
kernel/sched/core.c +11 -7
index 05c39f0..f307c2c 100644
@@ -4809,9 +4809,7 @@ static int sched_domain_debug_one(struct sched_domain *sd, int cpu, int level,
 				  struct cpumask *groupmask)
 {
 	struct sched_group *group = sd->groups;
-	char str[256];
 
-	cpulist_scnprintf(str, sizeof(str), sched_domain_span(sd));
 	cpumask_clear(groupmask);
 
 	printk(KERN_DEBUG "%*s domain %d: ", level, "", level);
@@ -4824,7 +4822,7 @@ static int sched_domain_debug_one(struct sched_domain *sd, int cpu, int level,
 		return -1;
 	}
 
-	printk(KERN_CONT "span %s level %s\n", str, sd->name);
+	printk(KERN_CONT "span %pc level %s\n", sched_domain_span(sd), sd->name);
 
 	if (!cpumask_test_cpu(cpu, sched_domain_span(sd))) {
 		printk(KERN_ERR "ERROR: domain->span does not contain "
@@ -4870,9 +4868,7 @@ static int sched_domain_debug_one(struct sched_domain *sd, int cpu, int level,
 
 		cpumask_or(groupmask, groupmask, sched_group_cpus(group));
 
-		cpulist_scnprintf(str, sizeof(str), sched_group_cpus(group));
-
-		printk(KERN_CONT " %s", str);
+		printk(KERN_CONT " %pc", sched_group_cpus(group));
 		if (group->sgp->power != SCHED_POWER_SCALE) {
 			printk(KERN_CONT " (cpu_power = %d)",
 				group->sgp->power);
@@ -4964,7 +4960,8 @@ sd_parent_degenerate(struct sched_domain *sd, struct sched_domain *parent)
 				SD_BALANCE_FORK |
 				SD_BALANCE_EXEC |
 				SD_SHARE_CPUPOWER |
-				SD_SHARE_PKG_RESOURCES);
+				SD_SHARE_PKG_RESOURCES |
+				SD_PREFER_SIBLING);
 		if (nr_node_ids == 1)
 			pflags &= ~SD_SERIALIZE;
 	}
@@ -5168,6 +5165,13 @@ cpu_attach_domain(struct sched_domain *sd, struct root_domain *rd, int cpu)
 			tmp->parent = parent->parent;
 			if (parent->parent)
 				parent->parent->child = tmp;
+			/*
+			 * Transfer SD_PREFER_SIBLING down in case of a
+			 * degenerate parent; the spans match for this
+			 * so the property transfers.
+			 */
+			if (parent->flags & SD_PREFER_SIBLING)
+				tmp->flags |= SD_PREFER_SIBLING;
 			destroy_sched_domain(parent, cpu);
 		} else
 			tmp = tmp->parent;
kernel/sched/fair.c +269 -229

840 lines not on this page · raw message

kernel/sched/sched.h +1 -0
index ef0a7b2..7c17661 100644
@@ -605,6 +605,7 @@ struct sched_group_power {
 	 */
 	unsigned int power, power_orig;
 	unsigned long next_update;
+	int imbalance; /* XXX unrelated to power but shared group state */
 	/*
 	 * Number of busy cpus in this group.
 	 */
lib/vsprintf.c +5 -0
index 739a363..5521015 100644
@@ -26,6 +26,7 @@
 #include <linux/math64.h>
 #include <linux/uaccess.h>
 #include <linux/ioport.h>
+#include <linux/cpumask.h>
 #include <net/addrconf.h>
 
 #include <asm/page.h>		/* for PAGE_SIZE */
@@ -1142,6 +1143,7 @@ int kptr_restrict __read_mostly;
  *            The maximum supported length is 64 bytes of the input. Consider
  *            to use print_hex_dump() for the larger input.
  * - 'a' For a phys_addr_t type and its derivative types (passed by reference)
+ * - 'c' For a cpumask list
  *
  * Note: The difference between 'S' and 'F' is that on ia64 and ppc64
  * function pointers are really function descriptors, which contain a
@@ -1253,6 +1255,8 @@ char *pointer(const char *fmt, char *buf, char *end, void *ptr,
 		spec.base = 16;
 		return number(buf, end,
 			      (unsigned long long) *((phys_addr_t *)ptr), spec);
+	case 'c':
+		return buf + cpulist_scnprintf(buf, end - buf, ptr);
 	}
 	spec.flags |= SMALL;
 	if (spec.field_width == -1) {
@@ -1494,6 +1498,7 @@ qualifier:
  *   case.
  * %*ph[CDN] a variable-length hex string with a separator (supports up to 64
  *           bytes of the input)
+ * %pc print a cpumask as comma-separated list
  * %n is ignored
  *
  * ** Please update Documentation/printk-formats.txt when making changes **
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#2 lore raw

PATCH 2/63 mm: numa: Document automatic NUMA balancing sysctls

In reply to Mel Gorman
---
 Documentation/sysctl/kernel.txt | 66 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)
Documentation/sysctl/kernel.txt +66 -0
index ab7d16e..ccadb52 100644
@@ -354,6 +354,72 @@ utilize.
 
 ==============================================================
 
+numa_balancing
+
+Enables/disables automatic page fault based NUMA memory
+balancing. Memory is moved automatically to nodes
+that access it often.
+
+Enables/disables automatic NUMA memory balancing. On NUMA machines, there
+is a performance penalty if remote memory is accessed by a CPU. When this
+feature is enabled the kernel samples what task thread is accessing memory
+by periodically unmapping pages and later trapping a page fault. At the
+time of the page fault, it is determined if the data being accessed should
+be migrated to a local memory node.
+
+The unmapping of pages and trapping faults incur additional overhead that
+ideally is offset by improved memory locality but there is no universal
+guarantee. If the target workload is already bound to NUMA nodes then this
+feature should be disabled. Otherwise, if the system overhead from the
+feature is too high then the rate the kernel samples for NUMA hinting
+faults may be controlled by the numa_balancing_scan_period_min_ms,
+numa_balancing_scan_delay_ms, numa_balancing_scan_period_reset,
+numa_balancing_scan_period_max_ms and numa_balancing_scan_size_mb sysctls.
+
+==============================================================
+
+numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms,
+numa_balancing_scan_period_max_ms, numa_balancing_scan_period_reset,
+numa_balancing_scan_size_mb
+
+Automatic NUMA balancing scans tasks address space and unmaps pages to
+detect if pages are properly placed or if the data should be migrated to a
+memory node local to where the task is running.  Every "scan delay" the task
+scans the next "scan size" number of pages in its address space. When the
+end of the address space is reached the scanner restarts from the beginning.
+
+In combination, the "scan delay" and "scan size" determine the scan rate.
+When "scan delay" decreases, the scan rate increases.  The scan delay and
+hence the scan rate of every task is adaptive and depends on historical
+behaviour. If pages are properly placed then the scan delay increases,
+otherwise the scan delay decreases.  The "scan size" is not adaptive but
+the higher the "scan size", the higher the scan rate.
+
+Higher scan rates incur higher system overhead as page faults must be
+trapped and potentially data must be migrated. However, the higher the scan
+rate, the more quickly a tasks memory is migrated to a local node if the
+workload pattern changes and minimises performance impact due to remote
+memory accesses. These sysctls control the thresholds for scan delays and
+the number of pages scanned.
+
+numa_balancing_scan_period_min_ms is the minimum delay in milliseconds
+between scans. It effectively controls the maximum scanning rate for
+each task.
+
+numa_balancing_scan_delay_ms is the starting "scan delay" used for a task
+when it initially forks.
+
+numa_balancing_scan_period_max_ms is the maximum delay between scans. It
+effectively controls the minimum scanning rate for each task.
+
+numa_balancing_scan_size_mb is how many megabytes worth of pages are
+scanned for a given scan.
+
+numa_balancing_scan_period_reset is a blunt instrument that controls how
+often a tasks scan delay is reset to detect sudden changes in task behaviour.
+
+==============================================================
+
 osrelease, ostype & version:
 
 # cat osrelease
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#3 lore raw

PATCH 3/63 sched, numa: Comment fixlets

In reply to Mel Gorman

From: Peter Zijlstra <[email protected]>

Fix a 80 column violation and a PTE vs PMD reference.

---
 kernel/sched/fair.c | 8 ++++----
 mm/huge_memory.c    | 2 +-
 2 files changed, 5 insertions(+), 5 deletions(-)
kernel/sched/fair.c +4 -4
index f238d49..a5d5b1e 100644
@@ -988,10 +988,10 @@ void task_numa_work(struct callback_head *work)
 
 out:
 	/*
-	 * It is possible to reach the end of the VMA list but the last few VMAs are
-	 * not guaranteed to the vma_migratable. If they are not, we would find the
-	 * !migratable VMA on the next scan but not reset the scanner to the start
-	 * so check it now.
+	 * It is possible to reach the end of the VMA list but the last few
+	 * VMAs are not guaranteed to the vma_migratable. If they are not, we
+	 * would find the !migratable VMA on the next scan but not reset the
+	 * scanner to the start so check it now.
 	 */
 	if (vma)
 		mm->numa_scan_offset = start;
mm/huge_memory.c +1 -1
index a92012a..860a368 100644
@@ -1317,7 +1317,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	spin_unlock(&mm->page_table_lock);
 	lock_page(page);
 
-	/* Confirm the PTE did not while locked */
+	/* Confirm the PMD did not change while page_table_lock was released */
 	spin_lock(&mm->page_table_lock);
 	if (unlikely(!pmd_same(pmd, *pmdp))) {
 		unlock_page(page);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#4 lore raw

PATCH 4/63 mm: numa: Do not account for a hinting fault if we raced

In reply to Mel Gorman

If another task handled a hinting fault in parallel then do not double account for it.

Not-signed-off-by: Peter Zijlstra

---
 mm/huge_memory.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
mm/huge_memory.c +4 -1
index 860a368..5c37cd2 100644
@@ -1337,8 +1337,11 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 
 check_same:
 	spin_lock(&mm->page_table_lock);
-	if (unlikely(!pmd_same(pmd, *pmdp)))
+	if (unlikely(!pmd_same(pmd, *pmdp))) {
+		/* Someone else took our fault */
+		current_nid = -1;
 		goto out_unlock;
+	}
 clear_pmdnuma:
 	pmd = pmd_mknonnuma(pmd);
 	set_pmd_at(mm, haddr, pmdp, pmd);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#5 lore raw

PATCH 5/63 mm: Wait for THP migrations to complete during NUMA hinting faults

In reply to Mel Gorman

The locking for migrating THP is unusual. While normal page migration prevents parallel accesses using a migration PTE, THP migration relies on a combination of the page_table_lock, the page lock and the existance of the NUMA hinting PTE to guarantee safety but there is a bug in the scheme.

If a THP page is currently being migrated and another thread traps a fault on the same page it checks if the page is misplaced. If it is not, then pmd_numa is cleared. The problem is that it checks if the page is misplaced without holding the page lock meaning that the racing thread can be migrating the THP when the second thread clears the NUMA bit and faults a stale page.

This patch checks if the page is potentially being migrated and stalls using the lock_page if it is potentially being migrated before checking if the page is misplaced or not.

Not-signed-off-by: Peter Zijlstra

---
 mm/huge_memory.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
mm/huge_memory.c +16 -7
index 5c37cd2..d0a3fce 100644
@@ -1307,13 +1307,14 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	if (current_nid == numa_node_id())
 		count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL);
 
-	target_nid = mpol_misplaced(page, vma, haddr);
-	if (target_nid == -1) {
-		put_page(page);
-		goto clear_pmdnuma;
-	}
+	/*
+	 * Acquire the page lock to serialise THP migrations but avoid dropping
+	 * page_table_lock if at all possible
+	 */
+	if (trylock_page(page))
+		goto got_lock;
 
-	/* Acquire the page lock to serialise THP migrations */
+	/* Serialise against migrationa and check placement check placement */
 	spin_unlock(&mm->page_table_lock);
 	lock_page(page);
 
@@ -1324,9 +1325,17 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		put_page(page);
 		goto out_unlock;
 	}
-	spin_unlock(&mm->page_table_lock);
+
+got_lock:
+	target_nid = mpol_misplaced(page, vma, haddr);
+	if (target_nid == -1) {
+		unlock_page(page);
+		put_page(page);
+		goto clear_pmdnuma;
+	}
 
 	/* Migrate the THP to the requested node */
+	spin_unlock(&mm->page_table_lock);
 	migrated = migrate_misplaced_transhuge_page(mm, vma,
 				pmdp, pmd, addr, page, target_nid);
 	if (!migrated)
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#6 lore raw

PATCH 6/63 mm: Prevent parallel splits during THP migration

In reply to Mel Gorman

THP migrations are serialised by the page lock but on its own that does not prevent THP splits. If the page is split during THP migration then the pmd_same checks will prevent page table corruption but the unlock page and other fix-ups potentially will cause corruption. This patch takes the anon_vma lock to prevent parallel splits during migration.

---
 mm/huge_memory.c | 44 ++++++++++++++++++++++++++++++--------------
 1 file changed, 30 insertions(+), 14 deletions(-)
mm/huge_memory.c +30 -14
index d0a3fce..cb34b7a 100644
@@ -1290,18 +1290,18 @@ out:
 int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 				unsigned long addr, pmd_t pmd, pmd_t *pmdp)
 {
+	struct anon_vma *anon_vma = NULL;
 	struct page *page;
 	unsigned long haddr = addr & HPAGE_PMD_MASK;
 	int target_nid;
 	int current_nid = -1;
-	bool migrated;
+	bool migrated, page_locked;
 
 	spin_lock(&mm->page_table_lock);
 	if (unlikely(!pmd_same(pmd, *pmdp)))
 		goto out_unlock;
 
 	page = pmd_page(pmd);
-	get_page(page);
 	current_nid = page_to_nid(page);
 	count_vm_numa_event(NUMA_HINT_FAULTS);
 	if (current_nid == numa_node_id())
@@ -1311,12 +1311,29 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	 * Acquire the page lock to serialise THP migrations but avoid dropping
 	 * page_table_lock if at all possible
 	 */
-	if (trylock_page(page))
-		goto got_lock;
+	page_locked = trylock_page(page);
+	target_nid = mpol_misplaced(page, vma, haddr);
+	if (target_nid == -1) {
+		/* If the page was locked, there are no parallel migrations */
+		if (page_locked) {
+			unlock_page(page);
+			goto clear_pmdnuma;
+		}
 
-	/* Serialise against migrationa and check placement check placement */
+		/* Otherwise wait for potential migrations and retry fault */
+		spin_unlock(&mm->page_table_lock);
+		wait_on_page_locked(page);
+		goto out;
+	}
+
+	/* Page is misplaced, serialise migrations and parallel THP splits */
+	get_page(page);
 	spin_unlock(&mm->page_table_lock);
-	lock_page(page);
+	if (!page_locked) {
+		lock_page(page);
+		page_locked = true;
+	}
+	anon_vma = page_lock_anon_vma_read(page);
 
 	/* Confirm the PMD did not change while page_table_lock was released */
 	spin_lock(&mm->page_table_lock);
@@ -1326,14 +1343,6 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		goto out_unlock;
 	}
 
-got_lock:
-	target_nid = mpol_misplaced(page, vma, haddr);
-	if (target_nid == -1) {
-		unlock_page(page);
-		put_page(page);
-		goto clear_pmdnuma;
-	}
-
 	/* Migrate the THP to the requested node */
 	spin_unlock(&mm->page_table_lock);
 	migrated = migrate_misplaced_transhuge_page(mm, vma,
@@ -1342,6 +1351,8 @@ got_lock:
 		goto check_same;
 
 	task_numa_fault(target_nid, HPAGE_PMD_NR, true);
+	if (anon_vma)
+		page_unlock_anon_vma_read(anon_vma);
 	return 0;
 
 check_same:
@@ -1358,6 +1369,11 @@ clear_pmdnuma:
 	update_mmu_cache_pmd(vma, addr, pmdp);
 out_unlock:
 	spin_unlock(&mm->page_table_lock);
+
+out:
+	if (anon_vma)
+		page_unlock_anon_vma_read(anon_vma);
+
 	if (current_nid != -1)
 		task_numa_fault(current_nid, HPAGE_PMD_NR, false);
 	return 0;
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#7 lore raw

PATCH 7/63 mm: Account for a THP NUMA hinting update as one PTE update

In reply to Mel Gorman

A THP PMD update is accounted for as 512 pages updated in vmstat. This is large difference when estimating the cost of automatic NUMA balancing and can be misleading when comparing results that had collapsed versus split THP. This patch addresses the accounting issue.

---
 mm/mprotect.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/mprotect.c +1 -1
index 94722a4..2bbb648 100644
@@ -145,7 +145,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,
 				split_huge_page_pmd(vma, addr, pmd);
 			else if (change_huge_pmd(vma, pmd, addr, newprot,
 						 prot_numa)) {
-				pages += HPAGE_PMD_NR;
+				pages++;
 				continue;
 			}
 			/* fall through */
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#8 lore raw

PATCH 8/63 mm: Do not flush TLB during protection change if !pte_present && !migration_entry

In reply to Mel Gorman

NUMA PTE scanning is expensive both in terms of the scanning itself and the TLB flush if there are any updates. Currently non-present PTEs are accounted for as an update and incurring a TLB flush where it is only necessary for anonymous migration entries. This patch addresses the problem and should reduce TLB flushes.

---
 mm/mprotect.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
mm/mprotect.c +2 -1
index 2bbb648..7bdbd4b 100644
@@ -101,8 +101,9 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
 				make_migration_entry_read(&entry);
 				set_pte_at(mm, addr, pte,
 					swp_entry_to_pte(entry));
+
+				pages++;
 			}
-			pages++;
 		}
 	} while (pte++, addr += PAGE_SIZE, addr != end);
 	arch_leave_lazy_mmu_mode();
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#9 lore raw

PATCH 9/63 mm: Only flush TLBs if a transhuge PMD is modified for NUMA pte scanning

In reply to Mel Gorman

NUMA PTE scanning is expensive both in terms of the scanning itself and the TLB flush if there are any updates. The TLB flush is avoided if no PTEs are updated but there is a bug where transhuge PMDs are considered to be updated even if they were already pmd_numa. This patch addresses the problem and TLB flushes should be reduced.

---
 mm/huge_memory.c | 19 ++++++++++++++++---
 mm/mprotect.c    | 14 ++++++++++----
 2 files changed, 26 insertions(+), 7 deletions(-)
mm/huge_memory.c +16 -3
index cb34b7a..656b341 100644
@@ -1472,6 +1472,12 @@ out:
 	return ret;
 }
 
+/*
+ * Returns
+ *  - 0 if PMD could not be locked
+ *  - 1 if PMD was locked but protections unchange and TLB flush unnecessary
+ *  - HPAGE_PMD_NR is protections changed and TLB flush necessary
+ */
 int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
 		unsigned long addr, pgprot_t newprot, int prot_numa)
 {
@@ -1480,9 +1486,11 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
 
 	if (__pmd_trans_huge_lock(pmd, vma) == 1) {
 		pmd_t entry;
-		entry = pmdp_get_and_clear(mm, addr, pmd);
+		ret = 1;
 		if (!prot_numa) {
+			entry = pmdp_get_and_clear(mm, addr, pmd);
 			entry = pmd_modify(entry, newprot);
+			ret = HPAGE_PMD_NR;
 			BUG_ON(pmd_write(entry));
 		} else {
 			struct page *page = pmd_page(*pmd);
@@ -1490,12 +1498,17 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
 			/* only check non-shared pages */
 			if (page_mapcount(page) == 1 &&
 			    !pmd_numa(*pmd)) {
+				entry = pmdp_get_and_clear(mm, addr, pmd);
 				entry = pmd_mknuma(entry);
+				ret = HPAGE_PMD_NR;
 			}
 		}
-		set_pmd_at(mm, addr, pmd, entry);
+
+		/* Set PMD if cleared earlier */
+		if (ret == HPAGE_PMD_NR)
+			set_pmd_at(mm, addr, pmd, entry);
+
 		spin_unlock(&vma->vm_mm->page_table_lock);
-		ret = 1;
 	}
 
 	return ret;
mm/mprotect.c +10 -4
index 7bdbd4b..2da33dc 100644
@@ -144,10 +144,16 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,
 		if (pmd_trans_huge(*pmd)) {
 			if (next - addr != HPAGE_PMD_SIZE)
 				split_huge_page_pmd(vma, addr, pmd);
-			else if (change_huge_pmd(vma, pmd, addr, newprot,
-						 prot_numa)) {
-				pages++;
-				continue;
+			else {
+				int nr_ptes = change_huge_pmd(vma, pmd, addr,
+						newprot, prot_numa);
+
+				if (nr_ptes) {
+					if (nr_ptes == HPAGE_PMD_NR)
+						pages++;
+
+					continue;
+				}
 			}
 			/* fall through */
 		}
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#10 lore raw

PATCH 10/63 mm: numa: Sanitize task_numa_fault() callsites

In reply to Mel Gorman

There are three callers of task_numa_fault():

- do_huge_pmd_numa_page():
Accounts against the current node, not the node where the page resides, unless we migrated, in which case it accounts against the node we migrated to.

- do_numa_page():
Accounts against the current node, not the node where the page resides, unless we migrated, in which case it accounts against the node we migrated to.

- do_pmd_numa_page():
Accounts not at all when the page isn't migrated, otherwise accounts against the node we migrated towards.

This seems wrong to me; all three sites should have the same sementaics, furthermore we should accounts against where the page really is, we already know where the task is.

So modify all three sites to always account; we did after all receive the fault; and always account to where the page is after migration, regardless of success.

They all still differ on when they clear the PTE/PMD; ideally that would get sorted too.

---
 mm/huge_memory.c | 25 +++++++++++++------------
 mm/memory.c      | 53 +++++++++++++++++++++--------------------------------
 2 files changed, 34 insertions(+), 44 deletions(-)
mm/huge_memory.c +13 -12
index 656b341..485b534 100644
@@ -1293,18 +1293,19 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	struct anon_vma *anon_vma = NULL;
 	struct page *page;
 	unsigned long haddr = addr & HPAGE_PMD_MASK;
+	int page_nid = -1, this_nid = numa_node_id();
 	int target_nid;
-	int current_nid = -1;
-	bool migrated, page_locked;
+	bool page_locked;
+	bool migrated = false;
 
 	spin_lock(&mm->page_table_lock);
 	if (unlikely(!pmd_same(pmd, *pmdp)))
 		goto out_unlock;
 
 	page = pmd_page(pmd);
-	current_nid = page_to_nid(page);
+	page_nid = page_to_nid(page);
 	count_vm_numa_event(NUMA_HINT_FAULTS);
-	if (current_nid == numa_node_id())
+	if (page_nid == this_nid)
 		count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL);
 
 	/*
@@ -1347,19 +1348,18 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	spin_unlock(&mm->page_table_lock);
 	migrated = migrate_misplaced_transhuge_page(mm, vma,
 				pmdp, pmd, addr, page, target_nid);
-	if (!migrated)
+	if (migrated)
+		page_nid = target_nid;
+	else
 		goto check_same;
 
-	task_numa_fault(target_nid, HPAGE_PMD_NR, true);
-	if (anon_vma)
-		page_unlock_anon_vma_read(anon_vma);
-	return 0;
+	goto out;
 
 check_same:
 	spin_lock(&mm->page_table_lock);
 	if (unlikely(!pmd_same(pmd, *pmdp))) {
 		/* Someone else took our fault */
-		current_nid = -1;
+		page_nid = -1;
 		goto out_unlock;
 	}
 clear_pmdnuma:
@@ -1374,8 +1374,9 @@ out:
 	if (anon_vma)
 		page_unlock_anon_vma_read(anon_vma);
 
-	if (current_nid != -1)
-		task_numa_fault(current_nid, HPAGE_PMD_NR, false);
+	if (page_nid != -1)
+		task_numa_fault(page_nid, HPAGE_PMD_NR, migrated);
+
 	return 0;
 }
 
mm/memory.c +21 -32

107 lines not on this page · raw message

Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#11 lore raw

PATCH 11/63 mm: Close races between THP migration and PMD numa clearing

In reply to Mel Gorman

THP migration uses the page lock to guard against parallel allocations but there are cases like this still open

Task A Task B
do_huge_pmd_numa_page do_huge_pmd_numa_page
lock_page
mpol_misplaced == -1
unlock_page
goto clear_pmdnuma

						lock_page
						mpol_misplaced == 2
						migrate_misplaced_transhuge

pmd = pmd_mknonnuma
set_pmd_at

During hours of testing, one crashed with weird errors and while I have no direct evidence, I suspect something like the race above happened. This patch extends the page lock to being held until the pmd_numa is cleared to prevent migration starting in parallel while the pmd_numa is being cleared. It also flushes the old pmd entry and orders pagetable insertion before rmap insertion.

---
 mm/huge_memory.c | 33 +++++++++++++++------------------
 mm/migrate.c     | 19 +++++++++++--------
 2 files changed, 26 insertions(+), 26 deletions(-)
mm/huge_memory.c +15 -18
index 485b534..ac5af18 100644
@@ -1316,24 +1316,25 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	target_nid = mpol_misplaced(page, vma, haddr);
 	if (target_nid == -1) {
 		/* If the page was locked, there are no parallel migrations */
-		if (page_locked) {
-			unlock_page(page);
+		if (page_locked)
 			goto clear_pmdnuma;
-		}
 
-		/* Otherwise wait for potential migrations and retry fault */
+		/*
+		 * Otherwise wait for potential migrations and retry. We do
+		 * relock and check_same as the page may no longer be mapped.
+		 * As the fault is being retried, do not account for it.
+		 */
 		spin_unlock(&mm->page_table_lock);
 		wait_on_page_locked(page);
+		page_nid = -1;
 		goto out;
 	}
 
 	/* Page is misplaced, serialise migrations and parallel THP splits */
 	get_page(page);
 	spin_unlock(&mm->page_table_lock);
-	if (!page_locked) {
+	if (!page_locked)
 		lock_page(page);
-		page_locked = true;
-	}
 	anon_vma = page_lock_anon_vma_read(page);
 
 	/* Confirm the PMD did not change while page_table_lock was released */
@@ -1341,32 +1342,28 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	if (unlikely(!pmd_same(pmd, *pmdp))) {
 		unlock_page(page);
 		put_page(page);
+		page_nid = -1;
 		goto out_unlock;
 	}
 
-	/* Migrate the THP to the requested node */
+	/*
+	 * Migrate the THP to the requested node, returns with page unlocked
+	 * and pmd_numa cleared.
+	 */
 	spin_unlock(&mm->page_table_lock);
 	migrated = migrate_misplaced_transhuge_page(mm, vma,
 				pmdp, pmd, addr, page, target_nid);
 	if (migrated)
 		page_nid = target_nid;
-	else
-		goto check_same;
 
 	goto out;
-
-check_same:
-	spin_lock(&mm->page_table_lock);
-	if (unlikely(!pmd_same(pmd, *pmdp))) {
-		/* Someone else took our fault */
-		page_nid = -1;
-		goto out_unlock;
-	}
 clear_pmdnuma:
+	BUG_ON(!PageLocked(page));
 	pmd = pmd_mknonnuma(pmd);
 	set_pmd_at(mm, haddr, pmdp, pmd);
 	VM_BUG_ON(pmd_numa(*pmdp));
 	update_mmu_cache_pmd(vma, addr, pmdp);
+	unlock_page(page);
 out_unlock:
 	spin_unlock(&mm->page_table_lock);
 
mm/migrate.c +11 -8
index 6f0c244..a7146a0 100644
@@ -1710,12 +1710,12 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
 		unlock_page(new_page);
 		put_page(new_page);		/* Free it */
 
-		unlock_page(page);
+		/* Retake the callers reference and putback on LRU */
+		get_page(page);
 		putback_lru_page(page);
-
-		count_vm_events(PGMIGRATE_FAIL, HPAGE_PMD_NR);
-		isolated = 0;
-		goto out;
+		mod_zone_page_state(page_zone(page),
+			 NR_ISOLATED_ANON + page_lru, -HPAGE_PMD_NR);
+		goto out_fail;
 	}
 
 	/*
@@ -1732,9 +1732,9 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
 	entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
 	entry = pmd_mkhuge(entry);
 
-	page_add_new_anon_rmap(new_page, vma, haddr);
-
+	pmdp_clear_flush(vma, address, pmd);
 	set_pmd_at(mm, haddr, pmd, entry);
+	page_add_new_anon_rmap(new_page, vma, haddr);
 	update_mmu_cache_pmd(vma, address, &entry);
 	page_remove_rmap(page);
 	/*
@@ -1753,7 +1753,6 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
 	count_vm_events(PGMIGRATE_SUCCESS, HPAGE_PMD_NR);
 	count_vm_numa_events(NUMA_PAGE_MIGRATE, HPAGE_PMD_NR);
 
-out:
 	mod_zone_page_state(page_zone(page),
 			NR_ISOLATED_ANON + page_lru,
 			-HPAGE_PMD_NR);
@@ -1762,6 +1761,10 @@ out:
 out_fail:
 	count_vm_events(PGMIGRATE_FAIL, HPAGE_PMD_NR);
 out_dropref:
+	entry = pmd_mknonnuma(entry);
+	set_pmd_at(mm, haddr, pmd, entry);
+	update_mmu_cache_pmd(vma, address, &entry);
+
 	unlock_page(page);
 	put_page(page);
 	return 0;
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#12 lore raw

Re: [PATCH 11/63] mm: Close races between THP migration and PMD numa clearing

mm/migrate.c int migrate_misplaced_transhuge_page(struct mm_struct *mm, · line 1740

Quoted 12 lines · Mel Gorman
mm/migrate.c +2 -2
@@ -1732,9 +1732,9 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
 	entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
 	entry = pmd_mkhuge(entry);
 
-	page_add_new_anon_rmap(new_page, vma, haddr);
-
+	pmdp_clear_flush(vma, address, pmd);
 	set_pmd_at(mm, haddr, pmd, entry);
+	page_add_new_anon_rmap(new_page, vma, haddr);
 	update_mmu_cache_pmd(vma, address, &entry);
 	page_remove_rmap(page);
 	/*

pmdp_clear_flush should have used haddr

Signature · 2 lines
Mel Gorman
SUSE Labs
Tested-by Jan Stancek
#13 lore raw

Re: [PATCH 11/63] mm: Close races between THP migration and PMD numa clearing

Quoted 5 lines · On Mon, 30 Sep 2013 09:52:59 +0100 Mel Gorman <[email protected]> wrote:
Quoted 4 lines · Mel Gorman
(quoted diff) +0 -0
@@ -1732,9 +1732,9 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
 	entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
 	entry = pmd_mkhuge(entry);
 
Quoted 10 lines

Review int migrate_misplaced_transhuge_page(struct mm_struct *mm, · line 1740

Quoted 8 lines
(quoted diff) +2 -2
-	page_add_new_anon_rmap(new_page, vma, haddr);
-
+	pmdp_clear_flush(vma, address, pmd);
 	set_pmd_at(mm, haddr, pmd, entry);
+	page_add_new_anon_rmap(new_page, vma, haddr);
 	update_mmu_cache_pmd(vma, address, &entry);
 	page_remove_rmap(page);
 	/*

pmdp_clear_flush should have used haddr

Dang, we both discovered this over the weekend? :)

In related news, it looks like update_mmu_cache_pmd should probably use haddr, too...

----

Subject: mm,numa: make THP migration mmu calls use haddr

The THP NUMA migration function migrate_misplaced_transhuge_page makes several calls into the architecture specific MMU code. Those calls all expect the virtual address of the huge page boundary, not the fault address from somewhere inside the huge page.

This fixes the below bug.

[   80.106362] kernel BUG at mm/pgtable-generic.c:103! 
...
[   80.333720] Call Trace: 
[   80.336450]  [<ffffffff811d5f8b>] migrate_misplaced_transhuge_page+0x1eb/0x500 
[   80.344505]  [<ffffffff811d8883>] do_huge_pmd_numa_page+0x1a3/0x330 
[   80.351497]  [<ffffffff811a3cc5>] handle_mm_fault+0x285/0x370 
[   80.357898]  [<ffffffff816d7df2>] __do_page_fault+0x172/0x5a0 
[   80.364307]  [<ffffffff8137a3dd>] ? trace_hardirqs_off_thunk+0x3a/0x3c 
[   80.371585]  [<ffffffff816d822e>] do_page_fault+0xe/0x10 
[   80.377510]  [<ffffffff816d41c8>] page_fault+0x28/0x30 
---
 mm/migrate.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/migrate.c +2 -2
index 1e1dbc9..5454151 100644
@@ -1736,10 +1736,10 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
 	entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
 	entry = pmd_mkhuge(entry);
 
-	pmdp_clear_flush(vma, address, pmd);
+	pmdp_clear_flush(vma, haddr, pmd);
 	set_pmd_at(mm, haddr, pmd, entry);
 	page_add_new_anon_rmap(new_page, vma, haddr);
-	update_mmu_cache_pmd(vma, address, &entry);
+	update_mmu_cache_pmd(vma, haddr, &entry);
 	page_remove_rmap(page);
 	/*
 	 * Finish the charge transaction under the page table lock to

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author reviewer
#14 lore raw

Re: [PATCH 11/63] mm: Close races between THP migration and PMD numa clearing

Quoted 19 lines · Rik van Riel
Quoted 5 lines · On Mon, 30 Sep 2013 09:52:59 +0100 Mel Gorman <[email protected]> wrote:
Quoted 4 lines · Mel Gorman
mm/migrate.c +0 -0
@@ -1732,9 +1732,9 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
 	entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
 	entry = pmd_mkhuge(entry);
 
Quoted 10 lines

Review int migrate_misplaced_transhuge_page(struct mm_struct *mm, · line 1740

Quoted 8 lines
(quoted diff) +2 -2
-	page_add_new_anon_rmap(new_page, vma, haddr);
-
+	pmdp_clear_flush(vma, address, pmd);
 	set_pmd_at(mm, haddr, pmd, entry);
+	page_add_new_anon_rmap(new_page, vma, haddr);
 	update_mmu_cache_pmd(vma, address, &entry);
 	page_remove_rmap(page);
 	/*

pmdp_clear_flush should have used haddr

Quoted 2 lines

Dang, we both discovered this over the weekend? :)

Saw it this morning running a debugging build.

Quoted 3 lines

In related news, it looks like update_mmu_cache_pmd should probably use haddr, too...

Does anything care? Other calls to update_mmu_cache_pmd are using address and not haddr so if this is a problem, it's a problem in a few places. Of the arches that support THP, only sparc appears to do anything useful and it shifts the address HPAGE_SHIFT so it does not matter if the address was aligned or not.

Signature · 8 lines
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#15 lore raw

PATCH 12/63 mm: numa: Do not migrate or account for hinting faults on the zero page

In reply to Mel Gorman

The zero page is not replicated between nodes and is often shared between processes. The data is read-only and likely to be cached in local CPUs if heavily accessed meaning that the remote memory access cost is less of a concern. This patch prevents trapping faults on the zero pages. For tasks using the zero page this will reduce the number of PTE updates, TLB flushes and hinting faults.

[[email protected]: Correct use of is_huge_zero_page]
---
 mm/huge_memory.c | 10 +++++++++-
 mm/memory.c      |  1 +
 2 files changed, 10 insertions(+), 1 deletion(-)
mm/huge_memory.c +9 -1
index ac5af18..dd94f4e 100644
@@ -1303,6 +1303,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		goto out_unlock;
 
 	page = pmd_page(pmd);
+	BUG_ON(is_huge_zero_page(page));
 	page_nid = page_to_nid(page);
 	count_vm_numa_event(NUMA_HINT_FAULTS);
 	if (page_nid == this_nid)
@@ -1493,8 +1494,15 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
 		} else {
 			struct page *page = pmd_page(*pmd);
 
-			/* only check non-shared pages */
+			/*
+			 * Only check non-shared pages. Do not trap faults
+			 * against the zero page. The read-only data is likely
+			 * to be read-cached on the local CPU cache and it is
+			 * less useful to know about local vs remote hits on
+			 * the zero page.
+			 */
 			if (page_mapcount(page) == 1 &&
+			    !is_huge_zero_page(page) &&
 			    !pmd_numa(*pmd)) {
 				entry = pmdp_get_and_clear(mm, addr, pmd);
 				entry = pmd_mknuma(entry);
mm/memory.c +1 -0
index c20f872..86c3caf 100644
@@ -3575,6 +3575,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		pte_unmap_unlock(ptep, ptl);
 		return 0;
 	}
+	BUG_ON(is_zero_pfn(page_to_pfn(page)));
 
 	page_nid = page_to_nid(page);
 	target_nid = numa_migrate_prep(page, vma, addr, page_nid);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#16 lore raw

PATCH 13/63 sched: numa: Mitigate chance that same task always updates PTEs

In reply to Mel Gorman

From: Peter Zijlstra <[email protected]>

With a trace_printk("working\n"); right after the cmpxchg in task_numa_work() we can see that of a 4 thread process, its always the same task winning the race and doing the protection change.

This is a problem since the task doing the protection change has a penalty for taking faults -- it is busy when marking the PTEs. If its always the same task the ->numa_faults[] get severely skewed.

Avoid this by delaying the task doing the protection change such that it is unlikely to win the privilege again.

Before:

root@interlagos:~# grep "thread 0/.*working" /debug/tracing/trace | tail -15
      thread 0/0-3232  [022] ....   212.787402: task_numa_work: working
      thread 0/0-3232  [022] ....   212.888473: task_numa_work: working
      thread 0/0-3232  [022] ....   212.989538: task_numa_work: working
      thread 0/0-3232  [022] ....   213.090602: task_numa_work: working
      thread 0/0-3232  [022] ....   213.191667: task_numa_work: working
      thread 0/0-3232  [022] ....   213.292734: task_numa_work: working
      thread 0/0-3232  [022] ....   213.393804: task_numa_work: working
      thread 0/0-3232  [022] ....   213.494869: task_numa_work: working
      thread 0/0-3232  [022] ....   213.596937: task_numa_work: working
      thread 0/0-3232  [022] ....   213.699000: task_numa_work: working
      thread 0/0-3232  [022] ....   213.801067: task_numa_work: working
      thread 0/0-3232  [022] ....   213.903155: task_numa_work: working
      thread 0/0-3232  [022] ....   214.005201: task_numa_work: working
      thread 0/0-3232  [022] ....   214.107266: task_numa_work: working
      thread 0/0-3232  [022] ....   214.209342: task_numa_work: working

After:

root@interlagos:~# grep "thread 0/.*working" /debug/tracing/trace | tail -15
      thread 0/0-3253  [005] ....   136.865051: task_numa_work: working
      thread 0/2-3255  [026] ....   136.965134: task_numa_work: working
      thread 0/3-3256  [024] ....   137.065217: task_numa_work: working
      thread 0/3-3256  [024] ....   137.165302: task_numa_work: working
      thread 0/3-3256  [024] ....   137.265382: task_numa_work: working
      thread 0/0-3253  [004] ....   137.366465: task_numa_work: working
      thread 0/2-3255  [026] ....   137.466549: task_numa_work: working
      thread 0/0-3253  [004] ....   137.566629: task_numa_work: working
      thread 0/0-3253  [004] ....   137.666711: task_numa_work: working
      thread 0/1-3254  [028] ....   137.766799: task_numa_work: working
      thread 0/0-3253  [004] ....   137.866876: task_numa_work: working
      thread 0/2-3255  [026] ....   137.966960: task_numa_work: working
      thread 0/1-3254  [028] ....   138.067041: task_numa_work: working
      thread 0/2-3255  [026] ....   138.167123: task_numa_work: working
      thread 0/3-3256  [024] ....   138.267207: task_numa_work: working
---
 kernel/sched/fair.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/sched/fair.c +7 -1
index a5d5b1e..db83eb1 100644
@@ -946,6 +946,12 @@ void task_numa_work(struct callback_head *work)
 		return;
 
 	/*
+	 * Delay this task enough that another task of this mm will likely win
+	 * the next time around.
+	 */
+	p->node_stamp += 2 * TICK_NSEC;
+
+	/*
 	 * Do not set pte_numa if the current running node is rate-limited.
 	 * This loses statistics on the fault but if we are unwilling to
 	 * migrate to this node, it is less likely we can do useful work
@@ -1026,7 +1032,7 @@ void task_tick_numa(struct rq *rq, struct task_struct *curr)
 	if (now - curr->node_stamp > period) {
 		if (!curr->node_stamp)
 			curr->numa_scan_period = sysctl_numa_balancing_scan_period_min;
-		curr->node_stamp = now;
+		curr->node_stamp += period;
 
 		if (!time_before(jiffies, curr->mm->numa_next_scan)) {
 			init_task_work(work, task_numa_work); /* TODO: move this into sched_fork() */
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#17 lore raw

PATCH 14/63 sched: numa: Continue PTE scanning even if migrate rate limited

In reply to Mel Gorman

From: Peter Zijlstra <[email protected]>

Avoiding marking PTEs pte_numa because a particular NUMA node is migrate rate limited sees like a bad idea. Even if this node can't migrate anymore other nodes might and we want up-to-date information to do balance decisions. We already rate limit the actual migrations, this should leave enough bandwidth to allow the non-migrating scanning. I think its important we keep up-to-date information if we're going to do placement based on it.

---
 kernel/sched/fair.c | 8 --------
 1 file changed, 8 deletions(-)
kernel/sched/fair.c +0 -8
index db83eb1..f2580ce 100644
@@ -951,14 +951,6 @@ void task_numa_work(struct callback_head *work)
 	 */
 	p->node_stamp += 2 * TICK_NSEC;
 
-	/*
-	 * Do not set pte_numa if the current running node is rate-limited.
-	 * This loses statistics on the fault but if we are unwilling to
-	 * migrate to this node, it is less likely we can do useful work
-	 */
-	if (migrate_ratelimited(numa_node_id()))
-		return;
-
 	start = mm->numa_scan_offset;
 	pages = sysctl_numa_balancing_scan_size;
 	pages <<= 20 - PAGE_SHIFT; /* MB in pages */
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#18 lore raw

PATCH 15/63 Revert "mm: sched: numa: Delay PTE scanning until a task is scheduled on a new node"

In reply to Mel Gorman

PTE scanning and NUMA hinting fault handling is expensive so commit 5bca2303 ("mm: sched: numa: Delay PTE scanning until a task is scheduled on a new node") deferred the PTE scan until a task had been scheduled on another node. The problem is that in the purely shared memory case that this may never happen and no NUMA hinting fault information will be captured. We are not ruling out the possibility that something better can be done here but for now, this patch needs to be reverted and depend entirely on the scan_delay to avoid punishing short-lived processes.

---
 include/linux/mm_types.h | 10 ----------
 kernel/fork.c            |  3 ---
 kernel/sched/fair.c      | 18 ------------------
 kernel/sched/features.h  |  4 +---
 4 files changed, 1 insertion(+), 34 deletions(-)
include/linux/mm_types.h +0 -10
index faf4b7c..4f12073 100644
@@ -427,20 +427,10 @@ struct mm_struct {
 
 	/* numa_scan_seq prevents two threads setting pte_numa */
 	int numa_scan_seq;
-
-	/*
-	 * The first node a task was scheduled on. If a task runs on
-	 * a different node than Make PTE Scan Go Now.
-	 */
-	int first_nid;
 #endif
 	struct uprobes_state uprobes_state;
 };
 
-/* first nid will either be a valid NID or one of these values */
-#define NUMA_PTE_SCAN_INIT	-1
-#define NUMA_PTE_SCAN_ACTIVE	-2
-
 static inline void mm_init_cpumask(struct mm_struct *mm)
 {
 #ifdef CONFIG_CPUMASK_OFFSTACK
kernel/fork.c +0 -3
index bf46287..3b1f6af 100644
@@ -820,9 +820,6 @@ struct mm_struct *dup_mm(struct task_struct *tsk)
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 	mm->pmd_huge_pte = NULL;
 #endif
-#ifdef CONFIG_NUMA_BALANCING
-	mm->first_nid = NUMA_PTE_SCAN_INIT;
-#endif
 	if (!mm_init(mm, tsk))
 		goto fail_nomem;
 
kernel/sched/fair.c +0 -18
index f2580ce..8510633 100644
@@ -901,24 +901,6 @@ void task_numa_work(struct callback_head *work)
 		return;
 
 	/*
-	 * We do not care about task placement until a task runs on a node
-	 * other than the first one used by the address space. This is
-	 * largely because migrations are driven by what CPU the task
-	 * is running on. If it's never scheduled on another node, it'll
-	 * not migrate so why bother trapping the fault.
-	 */
-	if (mm->first_nid == NUMA_PTE_SCAN_INIT)
-		mm->first_nid = numa_node_id();
-	if (mm->first_nid != NUMA_PTE_SCAN_ACTIVE) {
-		/* Are we running on a new node yet? */
-		if (numa_node_id() == mm->first_nid &&
-		    !sched_feat_numa(NUMA_FORCE))
-			return;
-
-		mm->first_nid = NUMA_PTE_SCAN_ACTIVE;
-	}
-
-	/*
 	 * Reset the scan period if enough time has gone by. Objective is that
 	 * scanning will be reduced if pages are properly placed. As tasks
 	 * can enter different phases this needs to be re-examined. Lacking
kernel/sched/features.h +1 -3
index 99399f8..cba5c61 100644
@@ -63,10 +63,8 @@ SCHED_FEAT(LB_MIN, false)
 /*
  * Apply the automatic NUMA scheduling policy. Enabled automatically
  * at runtime if running on a NUMA machine. Can be controlled via
- * numa_balancing=. Allow PTE scanning to be forced on UMA machines
- * for debugging the core machinery.
+ * numa_balancing=
  */
 #ifdef CONFIG_NUMA_BALANCING
 SCHED_FEAT(NUMA,	false)
-SCHED_FEAT(NUMA_FORCE,	false)
 #endif
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#19 lore raw

PATCH 16/63 sched: numa: Initialise numa_next_scan properly

In reply to Mel Gorman

Scan delay logic and resets are currently initialised to start scanning immediately instead of delaying properly. Initialise them properly at fork time and catch when a new mm has been allocated.

---
 kernel/sched/core.c | 4 ++--
 kernel/sched/fair.c | 7 +++++++
 2 files changed, 9 insertions(+), 2 deletions(-)
kernel/sched/core.c +2 -2
index f307c2c..9d7a33a 100644
@@ -1634,8 +1634,8 @@ static void __sched_fork(struct task_struct *p)
 
 #ifdef CONFIG_NUMA_BALANCING
 	if (p->mm && atomic_read(&p->mm->mm_users) == 1) {
-		p->mm->numa_next_scan = jiffies;
-		p->mm->numa_next_reset = jiffies;
+		p->mm->numa_next_scan = jiffies + msecs_to_jiffies(sysctl_numa_balancing_scan_delay);
+		p->mm->numa_next_reset = jiffies + msecs_to_jiffies(sysctl_numa_balancing_scan_period_reset);
 		p->mm->numa_scan_seq = 0;
 	}
 
kernel/sched/fair.c +7 -0
index 8510633..6edac70 100644
@@ -900,6 +900,13 @@ void task_numa_work(struct callback_head *work)
 	if (p->flags & PF_EXITING)
 		return;
 
+	if (!mm->numa_next_reset || !mm->numa_next_scan) {
+		mm->numa_next_scan = now +
+			msecs_to_jiffies(sysctl_numa_balancing_scan_delay);
+		mm->numa_next_reset = now +
+			msecs_to_jiffies(sysctl_numa_balancing_scan_period_reset);
+	}
+
 	/*
 	 * Reset the scan period if enough time has gone by. Objective is that
 	 * scanning will be reduced if pages are properly placed. As tasks
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#20 lore raw

PATCH 17/63 sched: Set the scan rate proportional to the memory usage of the task being scanned

In reply to Mel Gorman

The NUMA PTE scan rate is controlled with a combination of the numa_balancing_scan_period_min, numa_balancing_scan_period_max and numa_balancing_scan_size. This scan rate is independent of the size of the task and as an aside it is further complicated by the fact that numa_balancing_scan_size controls how many pages are marked pte_numa and not how much virtual memory is scanned.

In combination, it is almost impossible to meaningfully tune the min and max scan periods and reasoning about performance is complex when the time to complete a full scan is is partially a function of the tasks memory size. This patch alters the semantic of the min and max tunables to be about tuning the length time it takes to complete a scan of a tasks occupied virtual address space. Conceptually this is a lot easier to understand. There is a "sanity" check to ensure the scan rate is never extremely fast based on the amount of virtual memory that should be scanned in a second. The default of 2.5G seems arbitrary but it is to have the maximum scan rate after the patch roughly match the maximum scan rate before the patch was applied.

On a similar note, numa_scan_period is in milliseconds and not jiffies. Properly placed pages slow the scanning rate but adding 10 jiffies to numa_scan_period means that the rate scanning slows depends on HZ which is confusing. Get rid of the jiffies_to_msec conversion and treat it as ms.

---
 Documentation/sysctl/kernel.txt | 11 +++---
 include/linux/sched.h           |  1 +
 kernel/sched/fair.c             | 88 +++++++++++++++++++++++++++++++++++------
 3 files changed, 83 insertions(+), 17 deletions(-)
Documentation/sysctl/kernel.txt +6 -5
index ccadb52..ad8d4f5 100644
@@ -402,15 +402,16 @@ workload pattern changes and minimises performance impact due to remote
 memory accesses. These sysctls control the thresholds for scan delays and
 the number of pages scanned.
 
-numa_balancing_scan_period_min_ms is the minimum delay in milliseconds
-between scans. It effectively controls the maximum scanning rate for
-each task.
+numa_balancing_scan_period_min_ms is the minimum time in milliseconds to
+scan a tasks virtual memory. It effectively controls the maximum scanning
+rate for each task.
 
 numa_balancing_scan_delay_ms is the starting "scan delay" used for a task
 when it initially forks.
 
-numa_balancing_scan_period_max_ms is the maximum delay between scans. It
-effectively controls the minimum scanning rate for each task.
+numa_balancing_scan_period_max_ms is the maximum time in milliseconds to
+scan a tasks virtual memory. It effectively controls the minimum scanning
+rate for each task.
 
 numa_balancing_scan_size_mb is how many megabytes worth of pages are
 scanned for a given scan.
include/linux/sched.h +1 -0
index 078066d..49b426e 100644
@@ -1331,6 +1331,7 @@ struct task_struct {
 	int numa_scan_seq;
 	int numa_migrate_seq;
 	unsigned int numa_scan_period;
+	unsigned int numa_scan_period_max;
 	u64 node_stamp;			/* migration stamp  */
 	struct callback_head numa_work;
 #endif /* CONFIG_NUMA_BALANCING */
kernel/sched/fair.c +76 -12

143 lines not on this page · raw message

Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#21 lore raw

PATCH 18/63 sched: numa: Slow scan rate if no NUMA hinting faults are being recorded

In reply to Mel Gorman

NUMA PTE scanning slows if a NUMA hinting fault was trapped and no page was migrated. For long-lived but idle processes there may be no faults but the scan rate will be high and just waste CPU. This patch will slow the scan rate for processes that are not trapping faults.

---
 kernel/sched/fair.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
kernel/sched/fair.c +12 -0
index 8b71a5e..cdd656c 100644
@@ -1039,6 +1039,18 @@ void task_numa_work(struct callback_head *work)
 
 out:
 	/*
+	 * If the whole process was scanned without updates then no NUMA
+	 * hinting faults are being recorded and scan rate should be lower.
+	 */
+	if (mm->numa_scan_offset == 0 && !nr_pte_updates) {
+		p->numa_scan_period = min(p->numa_scan_period_max,
+			p->numa_scan_period << 1);
+
+		next_scan = now + msecs_to_jiffies(p->numa_scan_period);
+		mm->numa_next_scan = next_scan;
+	}
+
+	/*
 	 * It is possible to reach the end of the VMA list but the last few
 	 * VMAs are not guaranteed to the vma_migratable. If they are not, we
 	 * would find the !migratable VMA on the next scan but not reset the
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#22 lore raw

PATCH 19/63 sched: Track NUMA hinting faults on per-node basis

In reply to Mel Gorman

This patch tracks what nodes numa hinting faults were incurred on. This information is later used to schedule a task on the node storing the pages most frequently faulted by the task.

---
 include/linux/sched.h |  2 ++
 kernel/sched/core.c   |  3 +++
 kernel/sched/fair.c   | 11 ++++++++++-
 kernel/sched/sched.h  | 12 ++++++++++++
 4 files changed, 27 insertions(+), 1 deletion(-)
include/linux/sched.h +2 -0
index 49b426e..dfba435 100644
@@ -1334,6 +1334,8 @@ struct task_struct {
 	unsigned int numa_scan_period_max;
 	u64 node_stamp;			/* migration stamp  */
 	struct callback_head numa_work;
+
+	unsigned long *numa_faults;
 #endif /* CONFIG_NUMA_BALANCING */
 
 	struct rcu_head rcu;
kernel/sched/core.c +3 -0
index 9d7a33a..dbc2de6 100644
@@ -1644,6 +1644,7 @@ static void __sched_fork(struct task_struct *p)
 	p->numa_migrate_seq = p->mm ? p->mm->numa_scan_seq - 1 : 0;
 	p->numa_scan_period = sysctl_numa_balancing_scan_delay;
 	p->numa_work.next = &p->numa_work;
+	p->numa_faults = NULL;
 #endif /* CONFIG_NUMA_BALANCING */
 }
 
@@ -1905,6 +1906,8 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)
 	if (mm)
 		mmdrop(mm);
 	if (unlikely(prev_state == TASK_DEAD)) {
+		task_numa_free(prev);
+
 		/*
 		 * Remove function-return probe instances associated with this
 		 * task and put them back on the free list.
kernel/sched/fair.c +10 -1
index cdd656c..55ec7ad 100644
@@ -902,7 +902,14 @@ void task_numa_fault(int node, int pages, bool migrated)
 	if (!numabalancing_enabled)
 		return;
 
-	/* FIXME: Allocate task-specific structure for placement policy here */
+	/* Allocate buffer to track faults on a per-node basis */
+	if (unlikely(!p->numa_faults)) {
+		int size = sizeof(*p->numa_faults) * nr_node_ids;
+
+		p->numa_faults = kzalloc(size, GFP_KERNEL|__GFP_NOWARN);
+		if (!p->numa_faults)
+			return;
+	}
 
 	/*
 	 * If pages are properly placed (did not migrate) then scan slower.
@@ -918,6 +925,8 @@ void task_numa_fault(int node, int pages, bool migrated)
 	}
 
 	task_numa_placement(p);
+
+	p->numa_faults[node] += pages;
 }
 
 static void reset_ptenuma_scan(struct task_struct *p)
kernel/sched/sched.h +12 -0
index 7c17661..46c2068 100644
@@ -6,6 +6,7 @@
 #include <linux/spinlock.h>
 #include <linux/stop_machine.h>
 #include <linux/tick.h>
+#include <linux/slab.h>
 
 #include "cpupri.h"
 #include "cpuacct.h"
@@ -553,6 +554,17 @@ static inline u64 rq_clock_task(struct rq *rq)
 	return rq->clock_task;
 }
 
+#ifdef CONFIG_NUMA_BALANCING
+static inline void task_numa_free(struct task_struct *p)
+{
+	kfree(p->numa_faults);
+}
+#else /* CONFIG_NUMA_BALANCING */
+static inline void task_numa_free(struct task_struct *p)
+{
+}
+#endif /* CONFIG_NUMA_BALANCING */
+
 #ifdef CONFIG_SMP
 
 #define rcu_dereference_check_sched_domain(p) \
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#23 lore raw

PATCH 20/63 sched: Select a preferred node with the most numa hinting faults

In reply to Mel Gorman

This patch selects a preferred node for a task to run on based on the NUMA hinting faults. This information is later used to migrate tasks towards the node during balancing.

---
 include/linux/sched.h |  1 +
 kernel/sched/core.c   |  1 +
 kernel/sched/fair.c   | 17 +++++++++++++++--
 3 files changed, 17 insertions(+), 2 deletions(-)
include/linux/sched.h +1 -0
index dfba435..d6ec68a 100644
@@ -1336,6 +1336,7 @@ struct task_struct {
 	struct callback_head numa_work;
 
 	unsigned long *numa_faults;
+	int numa_preferred_nid;
 #endif /* CONFIG_NUMA_BALANCING */
 
 	struct rcu_head rcu;
kernel/sched/core.c +1 -0
index dbc2de6..0235ab8 100644
@@ -1643,6 +1643,7 @@ static void __sched_fork(struct task_struct *p)
 	p->numa_scan_seq = p->mm ? p->mm->numa_scan_seq : 0;
 	p->numa_migrate_seq = p->mm ? p->mm->numa_scan_seq - 1 : 0;
 	p->numa_scan_period = sysctl_numa_balancing_scan_delay;
+	p->numa_preferred_nid = -1;
 	p->numa_work.next = &p->numa_work;
 	p->numa_faults = NULL;
 #endif /* CONFIG_NUMA_BALANCING */
kernel/sched/fair.c +15 -2
index 55ec7ad..f9bc867 100644
@@ -879,7 +879,8 @@ static unsigned int task_scan_max(struct task_struct *p)
 
 static void task_numa_placement(struct task_struct *p)
 {
-	int seq;
+	int seq, nid, max_nid = -1;
+	unsigned long max_faults = 0;
 
 	if (!p->mm)	/* for example, ksmd faulting in a user's mm */
 		return;
@@ -889,7 +890,19 @@ static void task_numa_placement(struct task_struct *p)
 	p->numa_scan_seq = seq;
 	p->numa_scan_period_max = task_scan_max(p);
 
-	/* FIXME: Scheduling placement policy hints go here */
+	/* Find the node with the highest number of faults */
+	for_each_online_node(nid) {
+		unsigned long faults = p->numa_faults[nid];
+		p->numa_faults[nid] >>= 1;
+		if (faults > max_faults) {
+			max_faults = faults;
+			max_nid = nid;
+		}
+	}
+
+	/* Update the tasks preferred node if necessary */
+	if (max_faults && max_nid != p->numa_preferred_nid)
+		p->numa_preferred_nid = max_nid;
 }
 
 /*
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#24 lore raw

PATCH 21/63 sched: Update NUMA hinting faults once per scan

In reply to Mel Gorman

NUMA hinting fault counts and placement decisions are both recorded in the same array which distorts the samples in an unpredictable fashion. The values linearly accumulate during the scan and then decay creating a sawtooth-like pattern in the per-node counts. It also means that placement decisions are time sensitive. At best it means that it is very difficult to state that the buffer holds a decaying average of past faulting behaviour. At worst, it can confuse the load balancer if it sees one node with an artifically high count due to very recent faulting activity and may create a bouncing effect.

This patch adds a second array. numa_faults stores the historical data which is used for placement decisions. numa_faults_buffer holds the fault activity during the current scan window. When the scan completes, numa_faults decays and the values from numa_faults_buffer are copied across.

---
 include/linux/sched.h | 13 +++++++++++++
 kernel/sched/core.c   |  1 +
 kernel/sched/fair.c   | 16 +++++++++++++---
 3 files changed, 27 insertions(+), 3 deletions(-)
include/linux/sched.h +13 -0
index d6ec68a..84fb883 100644
@@ -1335,7 +1335,20 @@ struct task_struct {
 	u64 node_stamp;			/* migration stamp  */
 	struct callback_head numa_work;
 
+	/*
+	 * Exponential decaying average of faults on a per-node basis.
+	 * Scheduling placement decisions are made based on the these counts.
+	 * The values remain static for the duration of a PTE scan
+	 */
 	unsigned long *numa_faults;
+
+	/*
+	 * numa_faults_buffer records faults per node during the current
+	 * scan window. When the scan completes, the counts in numa_faults
+	 * decay and these values are copied.
+	 */
+	unsigned long *numa_faults_buffer;
+
 	int numa_preferred_nid;
 #endif /* CONFIG_NUMA_BALANCING */
 
kernel/sched/core.c +1 -0
index 0235ab8..2e8f3e2 100644
@@ -1646,6 +1646,7 @@ static void __sched_fork(struct task_struct *p)
 	p->numa_preferred_nid = -1;
 	p->numa_work.next = &p->numa_work;
 	p->numa_faults = NULL;
+	p->numa_faults_buffer = NULL;
 #endif /* CONFIG_NUMA_BALANCING */
 }
 
kernel/sched/fair.c +13 -3
index f9bc867..0fe4c4c 100644
@@ -892,8 +892,14 @@ static void task_numa_placement(struct task_struct *p)
 
 	/* Find the node with the highest number of faults */
 	for_each_online_node(nid) {
-		unsigned long faults = p->numa_faults[nid];
+		unsigned long faults;
+
+		/* Decay existing window and copy faults since last scan */
 		p->numa_faults[nid] >>= 1;
+		p->numa_faults[nid] += p->numa_faults_buffer[nid];
+		p->numa_faults_buffer[nid] = 0;
+
+		faults = p->numa_faults[nid];
 		if (faults > max_faults) {
 			max_faults = faults;
 			max_nid = nid;
@@ -919,9 +925,13 @@ void task_numa_fault(int node, int pages, bool migrated)
 	if (unlikely(!p->numa_faults)) {
 		int size = sizeof(*p->numa_faults) * nr_node_ids;
 
-		p->numa_faults = kzalloc(size, GFP_KERNEL|__GFP_NOWARN);
+		/* numa_faults and numa_faults_buffer share the allocation */
+		p->numa_faults = kzalloc(size * 2, GFP_KERNEL|__GFP_NOWARN);
 		if (!p->numa_faults)
 			return;
+
+		BUG_ON(p->numa_faults_buffer);
+		p->numa_faults_buffer = p->numa_faults + nr_node_ids;
 	}
 
 	/*
@@ -939,7 +949,7 @@ void task_numa_fault(int node, int pages, bool migrated)
 
 	task_numa_placement(p);
 
-	p->numa_faults[node] += pages;
+	p->numa_faults_buffer[node] += pages;
 }
 
 static void reset_ptenuma_scan(struct task_struct *p)
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#25 lore raw

PATCH 22/63 sched: Favour moving tasks towards the preferred node

In reply to Mel Gorman

This patch favours moving tasks towards NUMA node that recorded a higher number of NUMA faults during active load balancing. Ideally this is self-reinforcing as the longer the task runs on that node, the more faults it should incur causing task_numa_placement to keep the task running on that node. In reality a big weakness is that the nodes CPUs can be overloaded and it would be more efficient to queue tasks on an idle node and migrate to the new node. This would require additional smarts in the balancer so for now the balancer will simply prefer to place the task on the preferred node for a PTE scans which is controlled by the numa_balancing_settle_count sysctl. Once the settle_count number of scans has complete the schedule is free to place the task on an alternative node if the load is imbalanced.

[[email protected]: Fixed statistics]
[[email protected]: Tunable and use higher faults instead of preferred]
---
 Documentation/sysctl/kernel.txt |  8 +++++-
 include/linux/sched.h           |  1 +
 kernel/sched/core.c             |  3 +-
 kernel/sched/fair.c             | 63 ++++++++++++++++++++++++++++++++++++++---
 kernel/sched/features.h         |  7 +++++
 kernel/sysctl.c                 |  7 +++++
 6 files changed, 83 insertions(+), 6 deletions(-)
Documentation/sysctl/kernel.txt +7 -1
index ad8d4f5..23ff00a 100644
@@ -374,7 +374,8 @@ feature should be disabled. Otherwise, if the system overhead from the
 feature is too high then the rate the kernel samples for NUMA hinting
 faults may be controlled by the numa_balancing_scan_period_min_ms,
 numa_balancing_scan_delay_ms, numa_balancing_scan_period_reset,
-numa_balancing_scan_period_max_ms and numa_balancing_scan_size_mb sysctls.
+numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb and
+numa_balancing_settle_count sysctls.
 
 ==============================================================
 
@@ -419,6 +420,11 @@ scanned for a given scan.
 numa_balancing_scan_period_reset is a blunt instrument that controls how
 often a tasks scan delay is reset to detect sudden changes in task behaviour.
 
+numa_balancing_settle_count is how many scan periods must complete before
+the schedule balancer stops pushing the task towards a preferred node. This
+gives the scheduler a chance to place the task on an alternative node if the
+preferred node is overloaded.
+
 ==============================================================
 
 osrelease, ostype & version:
include/linux/sched.h +1 -0
index 84fb883..a2e661d 100644
@@ -776,6 +776,7 @@ enum cpu_idle_type {
 #define SD_ASYM_PACKING		0x0800  /* Place busy groups earlier in the domain */
 #define SD_PREFER_SIBLING	0x1000	/* Prefer to place tasks in a sibling domain */
 #define SD_OVERLAP		0x2000	/* sched_domains of this level overlap */
+#define SD_NUMA			0x4000	/* cross-node balancing */
 
 extern int __weak arch_sd_sibiling_asym_packing(void);
 
kernel/sched/core.c +2 -1
index 2e8f3e2..0dbd5cd 100644
@@ -1641,7 +1641,7 @@ static void __sched_fork(struct task_struct *p)
 
 	p->node_stamp = 0ULL;
 	p->numa_scan_seq = p->mm ? p->mm->numa_scan_seq : 0;
-	p->numa_migrate_seq = p->mm ? p->mm->numa_scan_seq - 1 : 0;
+	p->numa_migrate_seq = 0;
 	p->numa_scan_period = sysctl_numa_balancing_scan_delay;
 	p->numa_preferred_nid = -1;
 	p->numa_work.next = &p->numa_work;
@@ -5667,6 +5667,7 @@ sd_numa_init(struct sched_domain_topology_level *tl, int cpu)
 					| 0*SD_SHARE_PKG_RESOURCES
 					| 1*SD_SERIALIZE
 					| 0*SD_PREFER_SIBLING
+					| 1*SD_NUMA
 					| sd_local_flags(level)
 					,
 		.last_balance		= jiffies,
kernel/sched/fair.c +59 -4

96 lines not on this page · raw message

kernel/sched/features.h +7 -0
index cba5c61..d9278ce 100644
@@ -67,4 +67,11 @@ SCHED_FEAT(LB_MIN, false)
  */
 #ifdef CONFIG_NUMA_BALANCING
 SCHED_FEAT(NUMA,	false)
+
+/*
+ * NUMA_FAVOUR_HIGHER will favor moving tasks towards nodes where a
+ * higher number of hinting faults are recorded during active load
+ * balancing.
+ */
+SCHED_FEAT(NUMA_FAVOUR_HIGHER, true)
 #endif
kernel/sysctl.c +7 -0
index 07f6fc4..0015fb9 100644
@@ -391,6 +391,13 @@ static struct ctl_table kern_table[] = {
 		.mode		= 0644,
 		.proc_handler	= proc_dointvec,
 	},
+	{
+		.procname       = "numa_balancing_settle_count",
+		.data           = &sysctl_numa_balancing_settle_count,
+		.maxlen         = sizeof(unsigned int),
+		.mode           = 0644,
+		.proc_handler   = proc_dointvec,
+	},
 #endif /* CONFIG_NUMA_BALANCING */
 #endif /* CONFIG_SCHED_DEBUG */
 	{
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#26 lore raw

PATCH 23/63 sched: Resist moving tasks towards nodes with fewer hinting faults

In reply to Mel Gorman

Just as "sched: Favour moving tasks towards the preferred node" favours moving tasks towards nodes with a higher number of recorded NUMA hinting faults, this patch resists moving tasks towards nodes with lower faults.

---
 kernel/sched/fair.c     | 33 +++++++++++++++++++++++++++++++++
 kernel/sched/features.h |  8 ++++++++
 2 files changed, 41 insertions(+)
kernel/sched/fair.c +33 -0
index 7a9509c..051529e 100644
@@ -4060,12 +4060,43 @@ static bool migrate_improves_locality(struct task_struct *p, struct lb_env *env)
 
 	return false;
 }
+
+
+static bool migrate_degrades_locality(struct task_struct *p, struct lb_env *env)
+{
+	int src_nid, dst_nid;
+
+	if (!sched_feat(NUMA) || !sched_feat(NUMA_RESIST_LOWER))
+		return false;
+
+	if (!p->numa_faults || !(env->sd->flags & SD_NUMA))
+		return false;
+
+	src_nid = cpu_to_node(env->src_cpu);
+	dst_nid = cpu_to_node(env->dst_cpu);
+
+	if (src_nid == dst_nid ||
+	    p->numa_migrate_seq >= sysctl_numa_balancing_settle_count)
+		return false;
+
+	if (p->numa_faults[dst_nid] < p->numa_faults[src_nid])
+		return true;
+
+	return false;
+}
+
 #else
 static inline bool migrate_improves_locality(struct task_struct *p,
 					     struct lb_env *env)
 {
 	return false;
 }
+
+static inline bool migrate_degrades_locality(struct task_struct *p,
+					     struct lb_env *env)
+{
+	return false;
+}
 #endif
 
 /*
@@ -4130,6 +4161,8 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 	 * 3) too many balance attempts have failed.
 	 */
 	tsk_cache_hot = task_hot(p, rq_clock_task(env->src_rq), env->sd);
+	if (!tsk_cache_hot)
+		tsk_cache_hot = migrate_degrades_locality(p, env);
 
 	if (migrate_improves_locality(p, env)) {
 #ifdef CONFIG_SCHEDSTATS
kernel/sched/features.h +8 -0
index d9278ce..5716929 100644
@@ -74,4 +74,12 @@ SCHED_FEAT(NUMA,	false)
  * balancing.
  */
 SCHED_FEAT(NUMA_FAVOUR_HIGHER, true)
+
+/*
+ * NUMA_RESIST_LOWER will resist moving tasks towards nodes where a
+ * lower number of hinting faults have been recorded. As this has
+ * the potential to prevent a task ever migrating to a new node
+ * due to CPU overload it is disabled by default.
+ */
+SCHED_FEAT(NUMA_RESIST_LOWER, false)
 #endif
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#27 lore raw

PATCH 24/63 sched: Reschedule task on preferred NUMA node once selected

In reply to Mel Gorman

A preferred node is selected based on the node the most NUMA hinting faults was incurred on. There is no guarantee that the task is running on that node at the time so this patch rescheules the task to run on the most idle CPU of the selected node when selected. This avoids waiting for the balancer to make a decision.

---
 kernel/sched/core.c  | 19 +++++++++++++++++++
 kernel/sched/fair.c  | 46 +++++++++++++++++++++++++++++++++++++++++++++-
 kernel/sched/sched.h |  1 +
 3 files changed, 65 insertions(+), 1 deletion(-)
kernel/sched/core.c +19 -0
index 0dbd5cd..e94509d 100644
@@ -4368,6 +4368,25 @@ fail:
 	return ret;
 }
 
+#ifdef CONFIG_NUMA_BALANCING
+/* Migrate current task p to target_cpu */
+int migrate_task_to(struct task_struct *p, int target_cpu)
+{
+	struct migration_arg arg = { p, target_cpu };
+	int curr_cpu = task_cpu(p);
+
+	if (curr_cpu == target_cpu)
+		return 0;
+
+	if (!cpumask_test_cpu(target_cpu, tsk_cpus_allowed(p)))
+		return -EINVAL;
+
+	/* TODO: This is not properly updating schedstats */
+
+	return stop_one_cpu(curr_cpu, migration_cpu_stop, &arg);
+}
+#endif
+
 /*
  * migration_cpu_stop - this will be executed by a highprio stopper thread
  * and performs thread migration by bumping thread off CPU then
kernel/sched/fair.c +45 -1

61 lines not on this page · raw message

kernel/sched/sched.h +1 -0
index 46c2068..778f875 100644
@@ -555,6 +555,7 @@ static inline u64 rq_clock_task(struct rq *rq)
 }
 
 #ifdef CONFIG_NUMA_BALANCING
+extern int migrate_task_to(struct task_struct *p, int cpu);
 static inline void task_numa_free(struct task_struct *p)
 {
 	kfree(p->numa_faults);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#28 lore raw

PATCH 25/63 sched: Add infrastructure for split shared/private accounting of NUMA hinting faults

In reply to Mel Gorman

Ideally it would be possible to distinguish between NUMA hinting faults that are private to a task and those that are shared. This patch prepares infrastructure for separately accounting shared and private faults by allocating the necessary buffers and passing in relevant information. For now, all faults are treated as private and detection will be introduced later.

---
 include/linux/sched.h |  5 +++--
 kernel/sched/fair.c   | 46 +++++++++++++++++++++++++++++++++++-----------
 mm/huge_memory.c      |  5 +++--
 mm/memory.c           |  8 ++++++--
 4 files changed, 47 insertions(+), 17 deletions(-)
include/linux/sched.h +3 -2
index a2e661d..6eb8fa6 100644
@@ -1430,10 +1430,11 @@ struct task_struct {
 #define tsk_cpus_allowed(tsk) (&(tsk)->cpus_allowed)
 
 #ifdef CONFIG_NUMA_BALANCING
-extern void task_numa_fault(int node, int pages, bool migrated);
+extern void task_numa_fault(int last_node, int node, int pages, bool migrated);
 extern void set_numabalancing_state(bool enabled);
 #else
-static inline void task_numa_fault(int node, int pages, bool migrated)
+static inline void task_numa_fault(int last_node, int node, int pages,
+				   bool migrated)
 {
 }
 static inline void set_numabalancing_state(bool enabled)
kernel/sched/fair.c +35 -11

98 lines not on this page · raw message

mm/huge_memory.c +3 -2
index dd94f4e..9f3e008 100644
@@ -1294,7 +1294,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	struct page *page;
 	unsigned long haddr = addr & HPAGE_PMD_MASK;
 	int page_nid = -1, this_nid = numa_node_id();
-	int target_nid;
+	int target_nid, last_nid = -1;
 	bool page_locked;
 	bool migrated = false;
 
@@ -1305,6 +1305,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	page = pmd_page(pmd);
 	BUG_ON(is_huge_zero_page(page));
 	page_nid = page_to_nid(page);
+	last_nid = page_nid_last(page);
 	count_vm_numa_event(NUMA_HINT_FAULTS);
 	if (page_nid == this_nid)
 		count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL);
@@ -1373,7 +1374,7 @@ out:
 		page_unlock_anon_vma_read(anon_vma);
 
 	if (page_nid != -1)
-		task_numa_fault(page_nid, HPAGE_PMD_NR, migrated);
+		task_numa_fault(last_nid, page_nid, HPAGE_PMD_NR, migrated);
 
 	return 0;
 }
mm/memory.c +6 -2
index 86c3caf..bd016c2 100644
@@ -3547,6 +3547,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	struct page *page = NULL;
 	spinlock_t *ptl;
 	int page_nid = -1;
+	int last_nid;
 	int target_nid;
 	bool migrated = false;
 
@@ -3577,6 +3578,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	}
 	BUG_ON(is_zero_pfn(page_to_pfn(page)));
 
+	last_nid = page_nid_last(page);
 	page_nid = page_to_nid(page);
 	target_nid = numa_migrate_prep(page, vma, addr, page_nid);
 	pte_unmap_unlock(ptep, ptl);
@@ -3592,7 +3594,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 
 out:
 	if (page_nid != -1)
-		task_numa_fault(page_nid, 1, migrated);
+		task_numa_fault(last_nid, page_nid, 1, migrated);
 	return 0;
 }
 
@@ -3607,6 +3609,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	unsigned long offset;
 	spinlock_t *ptl;
 	bool numa = false;
+	int last_nid;
 
 	spin_lock(&mm->page_table_lock);
 	pmd = *pmdp;
@@ -3654,6 +3657,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		if (unlikely(page_mapcount(page) != 1))
 			continue;
 
+		last_nid = page_nid_last(page);
 		page_nid = page_to_nid(page);
 		target_nid = numa_migrate_prep(page, vma, addr, page_nid);
 		pte_unmap_unlock(pte, ptl);
@@ -3666,7 +3670,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		}
 
 		if (page_nid != -1)
-			task_numa_fault(page_nid, 1, migrated);
+			task_numa_fault(last_nid, page_nid, 1, migrated);
 
 		pte = pte_offset_map_lock(mm, pmdp, addr, &ptl);
 	}
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#29 lore raw

PATCH 26/63 sched: Check current->mm before allocating NUMA faults

In reply to Mel Gorman

task_numa_placement checks current->mm but after buffers for faults have already been uselessly allocated. Move the check earlier.

[[email protected]: Identified the problem]
---
 kernel/sched/fair.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/sched/fair.c +4 -2
index 96f6f08..7b7bc48 100644
@@ -930,8 +930,6 @@ static void task_numa_placement(struct task_struct *p)
 	int seq, nid, max_nid = -1;
 	unsigned long max_faults = 0;
 
-	if (!p->mm)	/* for example, ksmd faulting in a user's mm */
-		return;
 	seq = ACCESS_ONCE(p->mm->numa_scan_seq);
 	if (p->numa_scan_seq == seq)
 		return;
@@ -998,6 +996,10 @@ void task_numa_fault(int last_nid, int node, int pages, bool migrated)
 	if (!numabalancing_enabled)
 		return;
 
+	/* for example, ksmd faulting in a user's mm */
+	if (!p->mm)
+		return;
+
 	/* For now, do not attempt to detect private/shared accesses */
 	priv = 1;
 
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#30 lore raw

PATCH 27/63 mm: numa: Scan pages with elevated page_mapcount

In reply to Mel Gorman

Currently automatic NUMA balancing is unable to distinguish between false shared versus private pages except by ignoring pages with an elevated page_mapcount entirely. This avoids shared pages bouncing between the nodes whose task is using them but that is ignored quite a lot of data.

This patch kicks away the training wheels in preparation for adding support for identifying shared/private pages is now in place. The ordering is so that the impact of the shared/private detection can be easily measured. Note that the patch does not migrate shared, file-backed within vmas marked VM_EXEC as these are generally shared library pages. Migrating such pages is not beneficial as there is an expectation they are read-shared between caches and iTLB and iCache pressure is generally low.

---
 include/linux/migrate.h |  7 ++++---
 mm/huge_memory.c        | 12 +++++-------
 mm/memory.c             |  7 ++-----
 mm/migrate.c            | 17 ++++++-----------
 mm/mprotect.c           |  4 +---
 5 files changed, 18 insertions(+), 29 deletions(-)
include/linux/migrate.h +4 -3
index a405d3dc..e7e26af 100644
@@ -92,11 +92,12 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,
 #endif /* CONFIG_MIGRATION */
 
 #ifdef CONFIG_NUMA_BALANCING
-extern int migrate_misplaced_page(struct page *page, int node);
-extern int migrate_misplaced_page(struct page *page, int node);
+extern int migrate_misplaced_page(struct page *page,
+				  struct vm_area_struct *vma, int node);
 extern bool migrate_ratelimited(int node);
 #else
-static inline int migrate_misplaced_page(struct page *page, int node)
+static inline int migrate_misplaced_page(struct page *page,
+					 struct vm_area_struct *vma, int node)
 {
 	return -EAGAIN; /* can't migrate now */
 }
mm/huge_memory.c +5 -7
index 9f3e008..1f006d0 100644
@@ -1496,14 +1496,12 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
 			struct page *page = pmd_page(*pmd);
 
 			/*
-			 * Only check non-shared pages. Do not trap faults
-			 * against the zero page. The read-only data is likely
-			 * to be read-cached on the local CPU cache and it is
-			 * less useful to know about local vs remote hits on
-			 * the zero page.
+			 * Do not trap faults against the zero page. The
+			 * read-only data is likely to be read-cached on the
+			 * local CPU cache and it is less useful to know about
+			 * local vs remote hits on the zero page.
 			 */
-			if (page_mapcount(page) == 1 &&
-			    !is_huge_zero_page(page) &&
+			if (!is_huge_zero_page(page) &&
 			    !pmd_numa(*pmd)) {
 				entry = pmdp_get_and_clear(mm, addr, pmd);
 				entry = pmd_mknuma(entry);
mm/memory.c +2 -5
index bd016c2..e335ec0 100644
@@ -3588,7 +3588,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	}
 
 	/* Migrate to the requested node */
-	migrated = migrate_misplaced_page(page, target_nid);
+	migrated = migrate_misplaced_page(page, vma, target_nid);
 	if (migrated)
 		page_nid = target_nid;
 
@@ -3653,16 +3653,13 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		page = vm_normal_page(vma, addr, pteval);
 		if (unlikely(!page))
 			continue;
-		/* only check non-shared pages */
-		if (unlikely(page_mapcount(page) != 1))
-			continue;
 
 		last_nid = page_nid_last(page);
 		page_nid = page_to_nid(page);
 		target_nid = numa_migrate_prep(page, vma, addr, page_nid);
 		pte_unmap_unlock(pte, ptl);
 		if (target_nid != -1) {
-			migrated = migrate_misplaced_page(page, target_nid);
+			migrated = migrate_misplaced_page(page, vma, target_nid);
 			if (migrated)
 				page_nid = target_nid;
 		} else {
mm/migrate.c +6 -11
index a7146a0..e7191a3 100644
@@ -1596,7 +1596,8 @@ int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page)
  * node. Caller is expected to have an elevated reference count on
  * the page that will be dropped by this function before returning.
  */
-int migrate_misplaced_page(struct page *page, int node)
+int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,
+			   int node)
 {
 	pg_data_t *pgdat = NODE_DATA(node);
 	int isolated;
@@ -1604,10 +1605,11 @@ int migrate_misplaced_page(struct page *page, int node)
 	LIST_HEAD(migratepages);
 
 	/*
-	 * Don't migrate pages that are mapped in multiple processes.
-	 * TODO: Handle false sharing detection instead of this hammer
+	 * Don't migrate file pages that are mapped in multiple processes
+	 * with execute permissions as they are probably shared libraries.
 	 */
-	if (page_mapcount(page) != 1)
+	if (page_mapcount(page) != 1 && page_is_file_cache(page) &&
+	    (vma->vm_flags & VM_EXEC))
 		goto out;
 
 	/*
@@ -1658,13 +1660,6 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
 	int page_lru = page_is_file_cache(page);
 
 	/*
-	 * Don't migrate pages that are mapped in multiple processes.
-	 * TODO: Handle false sharing detection instead of this hammer
-	 */
-	if (page_mapcount(page) != 1)
-		goto out_dropref;
-
-	/*
 	 * Rate-limit the amount of data that is being migrated to a node.
 	 * Optimal placement is no good if the memory bus is saturated and
 	 * all the time is being spent migrating!
mm/mprotect.c +1 -3
index 2da33dc..41e0292 100644
@@ -69,9 +69,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
 					if (last_nid != this_nid)
 						all_same_node = false;
 
-					/* only check non-shared pages */
-					if (!pte_numa(oldpte) &&
-					    page_mapcount(page) == 1) {
+					if (!pte_numa(oldpte)) {
 						ptent = pte_mknuma(ptent);
 						updated = true;
 					}
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#31 lore raw

PATCH 28/63 sched: Remove check that skips small VMAs

In reply to Mel Gorman

task_numa_work skips small VMAs. At the time the logic was to reduce the scanning overhead which was considerable. It is a dubious hack at best. It would make much more sense to cache where faults have been observed and only rescan those regions during subsequent PTE scans. Remove this hack as motivation to do it properly in the future.

---
 kernel/sched/fair.c | 4 ----
 1 file changed, 4 deletions(-)
kernel/sched/fair.c +0 -4
index 7b7bc48..b7d197e 100644
@@ -1127,10 +1127,6 @@ void task_numa_work(struct callback_head *work)
 		if (!vma_migratable(vma))
 			continue;
 
-		/* Skip small VMAs. They are not likely to be of relevance */
-		if (vma->vm_end - vma->vm_start < HPAGE_SIZE)
-			continue;
-
 		do {
 			start = max(start, vma->vm_start);
 			end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#32 lore raw

PATCH 29/63 sched: Set preferred NUMA node based on number of private faults

In reply to Mel Gorman

Ideally it would be possible to distinguish between NUMA hinting faults that are private to a task and those that are shared. If treated identically there is a risk that shared pages bounce between nodes depending on the order they are referenced by tasks. Ultimately what is desirable is that task private pages remain local to the task while shared pages are interleaved between sharing tasks running on different nodes to give good average performance. This is further complicated by THP as even applications that partition their data may not be partitioning on a huge page boundary.

To start with, this patch assumes that multi-threaded or multi-process applications partition their data and that in general the private accesses are more important for cpu->memory locality in the general case. Also, no new infrastructure is required to treat private pages properly but interleaving for shared pages requires additional infrastructure.

To detect private accesses the pid of the last accessing task is required but the storage requirements are a high. This patch borrows heavily from Ingo Molnar's patch "numa, mm, sched: Implement last-CPU+PID hash tracking" to encode some bits from the last accessing task in the page flags as well as the node information. Collisions will occur but it is better than just depending on the node information. Node information is then used to determine if a page needs to migrate. The PID information is used to detect private/shared accesses. The preferred NUMA node is selected based on where the maximum number of approximately private faults were measured. Shared faults are not taken into consideration for a few reasons.

First, if there are many tasks sharing the page then they'll all move towards the same node. The node will be compute overloaded and then scheduled away later only to bounce back again. Alternatively the shared tasks would just bounce around nodes because the fault information is effectively noise. Either way accounting for shared faults the same as private faults can result in lower performance overall.

The second reason is based on a hypothetical workload that has a small number of very important, heavily accessed private pages but a large shared array. The shared array would dominate the number of faults and be selected as a preferred node even though it's the wrong decision.

The third reason is that multiple threads in a process will race each other to fault the shared page making the fault information unreliable.

[[email protected]: Fix complication error when !NUMA_BALANCING]
---
 include/linux/mm.h                | 89 +++++++++++++++++++++++++++++----------
 include/linux/mm_types.h          |  4 +-
 include/linux/page-flags-layout.h | 28 +++++++-----
 kernel/sched/fair.c               | 12 ++++--
 mm/huge_memory.c                  |  8 ++--
 mm/memory.c                       | 16 +++----
 mm/mempolicy.c                    |  8 ++--
 mm/migrate.c                      |  4 +-
 mm/mm_init.c                      | 18 ++++----
 mm/mmzone.c                       | 14 +++---
 mm/mprotect.c                     | 26 ++++++++----
 mm/page_alloc.c                   |  4 +-
 12 files changed, 149 insertions(+), 82 deletions(-)
include/linux/mm.h +67 -22
index f022460..0a0db6c 100644
@@ -588,11 +588,11 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
  * sets it, so none of the operations on it need to be atomic.
  */
 
-/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_NID] | ... | FLAGS | */
+/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_NIDPID] | ... | FLAGS | */
 #define SECTIONS_PGOFF		((sizeof(unsigned long)*8) - SECTIONS_WIDTH)
 #define NODES_PGOFF		(SECTIONS_PGOFF - NODES_WIDTH)
 #define ZONES_PGOFF		(NODES_PGOFF - ZONES_WIDTH)
-#define LAST_NID_PGOFF		(ZONES_PGOFF - LAST_NID_WIDTH)
+#define LAST_NIDPID_PGOFF	(ZONES_PGOFF - LAST_NIDPID_WIDTH)
 
 /*
  * Define the bit shifts to access each section.  For non-existent
@@ -602,7 +602,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
 #define SECTIONS_PGSHIFT	(SECTIONS_PGOFF * (SECTIONS_WIDTH != 0))
 #define NODES_PGSHIFT		(NODES_PGOFF * (NODES_WIDTH != 0))
 #define ZONES_PGSHIFT		(ZONES_PGOFF * (ZONES_WIDTH != 0))
-#define LAST_NID_PGSHIFT	(LAST_NID_PGOFF * (LAST_NID_WIDTH != 0))
+#define LAST_NIDPID_PGSHIFT	(LAST_NIDPID_PGOFF * (LAST_NIDPID_WIDTH != 0))
 
 /* NODE:ZONE or SECTION:ZONE is used to ID a zone for the buddy allocator */
 #ifdef NODE_NOT_IN_PAGE_FLAGS
@@ -624,7 +624,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
 #define ZONES_MASK		((1UL << ZONES_WIDTH) - 1)
 #define NODES_MASK		((1UL << NODES_WIDTH) - 1)
 #define SECTIONS_MASK		((1UL << SECTIONS_WIDTH) - 1)
-#define LAST_NID_MASK		((1UL << LAST_NID_WIDTH) - 1)
+#define LAST_NIDPID_MASK	((1UL << LAST_NIDPID_WIDTH) - 1)
 #define ZONEID_MASK		((1UL << ZONEID_SHIFT) - 1)
 
 static inline enum zone_type page_zonenum(const struct page *page)
@@ -668,48 +668,93 @@ static inline int page_to_nid(const struct page *page)
 #endif
 
 #ifdef CONFIG_NUMA_BALANCING
-#ifdef LAST_NID_NOT_IN_PAGE_FLAGS
-static inline int page_nid_xchg_last(struct page *page, int nid)
+static inline int nid_pid_to_nidpid(int nid, int pid)
 {
-	return xchg(&page->_last_nid, nid);
+	return ((nid & LAST__NID_MASK) << LAST__PID_SHIFT) | (pid & LAST__PID_MASK);
 }
 
-static inline int page_nid_last(struct page *page)
+static inline int nidpid_to_pid(int nidpid)
 {
-	return page->_last_nid;
+	return nidpid & LAST__PID_MASK;
 }
-static inline void page_nid_reset_last(struct page *page)
+
+static inline int nidpid_to_nid(int nidpid)
+{
+	return (nidpid >> LAST__PID_SHIFT) & LAST__NID_MASK;
+}
+
+static inline bool nidpid_pid_unset(int nidpid)
+{
+	return nidpid_to_pid(nidpid) == (-1 & LAST__PID_MASK);
+}
+
+static inline bool nidpid_nid_unset(int nidpid)
 {
-	page->_last_nid = -1;
+	return nidpid_to_nid(nidpid) == (-1 & LAST__NID_MASK);
+}
+
+#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS
+static inline int page_nidpid_xchg_last(struct page *page, int nid)
+{
+	return xchg(&page->_last_nidpid, nid);
+}
+
+static inline int page_nidpid_last(struct page *page)
+{
+	return page->_last_nidpid;
+}
+static inline void page_nidpid_reset_last(struct page *page)
+{
+	page->_last_nidpid = -1;
 }
 #else
-static inline int page_nid_last(struct page *page)
+static inline int page_nidpid_last(struct page *page)
 {
-	return (page->flags >> LAST_NID_PGSHIFT) & LAST_NID_MASK;
+	return (page->flags >> LAST_NIDPID_PGSHIFT) & LAST_NIDPID_MASK;
 }
 
-extern int page_nid_xchg_last(struct page *page, int nid);
+extern int page_nidpid_xchg_last(struct page *page, int nidpid);
 
-static inline void page_nid_reset_last(struct page *page)
+static inline void page_nidpid_reset_last(struct page *page)
 {
-	int nid = (1 << LAST_NID_SHIFT) - 1;
+	int nidpid = (1 << LAST_NIDPID_SHIFT) - 1;
 
-	page->flags &= ~(LAST_NID_MASK << LAST_NID_PGSHIFT);
-	page->flags |= (nid & LAST_NID_MASK) << LAST_NID_PGSHIFT;
+	page->flags &= ~(LAST_NIDPID_MASK << LAST_NIDPID_PGSHIFT);
+	page->flags |= (nidpid & LAST_NIDPID_MASK) << LAST_NIDPID_PGSHIFT;
 }
-#endif /* LAST_NID_NOT_IN_PAGE_FLAGS */
+#endif /* LAST_NIDPID_NOT_IN_PAGE_FLAGS */
 #else
-static inline int page_nid_xchg_last(struct page *page, int nid)
+static inline int page_nidpid_xchg_last(struct page *page, int nidpid)
 {
 	return page_to_nid(page);
 }
 
-static inline int page_nid_last(struct page *page)
+static inline int page_nidpid_last(struct page *page)
 {
 	return page_to_nid(page);
 }
 
-static inline void page_nid_reset_last(struct page *page)
+static inline int nidpid_to_nid(int nidpid)
+{
+	return -1;
+}
+
+static inline int nidpid_to_pid(int nidpid)
+{
+	return -1;
+}
+
+static inline int nid_pid_to_nidpid(int nid, int pid)
+{
+	return -1;
+}
+
+static inline bool nidpid_pid_unset(int nidpid)
+{
+	return 1;
+}
+
+static inline void page_nidpid_reset_last(struct page *page)
 {
 }
 #endif
include/linux/mm_types.h +2 -2
index 4f12073..f46378e 100644
@@ -174,8 +174,8 @@ struct page {
 	void *shadow;
 #endif
 
-#ifdef LAST_NID_NOT_IN_PAGE_FLAGS
-	int _last_nid;
+#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS
+	int _last_nidpid;
 #endif
 }
 /*
include/linux/page-flags-layout.h +17 -11
index 93506a1..02bc918 100644
@@ -38,10 +38,10 @@
  * The last is when there is insufficient space in page->flags and a separate
  * lookup is necessary.
  *
- * No sparsemem or sparsemem vmemmap: |       NODE     | ZONE |          ... | FLAGS |
- *         " plus space for last_nid: |       NODE     | ZONE | LAST_NID ... | FLAGS |
- * classic sparse with space for node:| SECTION | NODE | ZONE |          ... | FLAGS |
- *         " plus space for last_nid: | SECTION | NODE | ZONE | LAST_NID ... | FLAGS |
+ * No sparsemem or sparsemem vmemmap: |       NODE     | ZONE |             ... | FLAGS |
+ *      " plus space for last_nidpid: |       NODE     | ZONE | LAST_NIDPID ... | FLAGS |
+ * classic sparse with space for node:| SECTION | NODE | ZONE |             ... | FLAGS |
+ *      " plus space for last_nidpid: | SECTION | NODE | ZONE | LAST_NIDPID ... | FLAGS |
  * classic sparse no space for node:  | SECTION |     ZONE    | ... | FLAGS |
  */
 #if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP)
@@ -62,15 +62,21 @@
 #endif
 
 #ifdef CONFIG_NUMA_BALANCING
-#define LAST_NID_SHIFT NODES_SHIFT
+#define LAST__PID_SHIFT 8
+#define LAST__PID_MASK  ((1 << LAST__PID_SHIFT)-1)
+
+#define LAST__NID_SHIFT NODES_SHIFT
+#define LAST__NID_MASK  ((1 << LAST__NID_SHIFT)-1)
+
+#define LAST_NIDPID_SHIFT (LAST__PID_SHIFT+LAST__NID_SHIFT)
 #else
-#define LAST_NID_SHIFT 0
+#define LAST_NIDPID_SHIFT 0
 #endif
 
-#if SECTIONS_WIDTH+ZONES_WIDTH+NODES_SHIFT+LAST_NID_SHIFT <= BITS_PER_LONG - NR_PAGEFLAGS
-#define LAST_NID_WIDTH LAST_NID_SHIFT
+#if SECTIONS_WIDTH+ZONES_WIDTH+NODES_SHIFT+LAST_NIDPID_SHIFT <= BITS_PER_LONG - NR_PAGEFLAGS
+#define LAST_NIDPID_WIDTH LAST_NIDPID_SHIFT
 #else
-#define LAST_NID_WIDTH 0
+#define LAST_NIDPID_WIDTH 0
 #endif
 
 /*
@@ -81,8 +87,8 @@
 #define NODE_NOT_IN_PAGE_FLAGS
 #endif
 
-#if defined(CONFIG_NUMA_BALANCING) && LAST_NID_WIDTH == 0
-#define LAST_NID_NOT_IN_PAGE_FLAGS
+#if defined(CONFIG_NUMA_BALANCING) && LAST_NIDPID_WIDTH == 0
+#define LAST_NIDPID_NOT_IN_PAGE_FLAGS
 #endif
 
 #endif /* _LINUX_PAGE_FLAGS_LAYOUT */
kernel/sched/fair.c +9 -3
index b7d197e..891105c 100644
@@ -988,7 +988,7 @@ static void task_numa_placement(struct task_struct *p)
 /*
  * Got a PROT_NONE fault for a page on @node.
  */
-void task_numa_fault(int last_nid, int node, int pages, bool migrated)
+void task_numa_fault(int last_nidpid, int node, int pages, bool migrated)
 {
 	struct task_struct *p = current;
 	int priv;
@@ -1000,8 +1000,14 @@ void task_numa_fault(int last_nid, int node, int pages, bool migrated)
 	if (!p->mm)
 		return;
 
-	/* For now, do not attempt to detect private/shared accesses */
-	priv = 1;
+	/*
+	 * First accesses are treated as private, otherwise consider accesses
+	 * to be private if the accessing pid has not changed
+	 */
+	if (!nidpid_pid_unset(last_nidpid))
+		priv = ((p->pid & LAST__PID_MASK) == nidpid_to_pid(last_nidpid));
+	else
+		priv = 1;
 
 	/* Allocate buffer to track faults on a per-node basis */
 	if (unlikely(!p->numa_faults)) {
mm/huge_memory.c +4 -4
index 1f006d0..0e73685 100644
@@ -1294,7 +1294,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	struct page *page;
 	unsigned long haddr = addr & HPAGE_PMD_MASK;
 	int page_nid = -1, this_nid = numa_node_id();
-	int target_nid, last_nid = -1;
+	int target_nid, last_nidpid = -1;
 	bool page_locked;
 	bool migrated = false;
 
@@ -1305,7 +1305,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	page = pmd_page(pmd);
 	BUG_ON(is_huge_zero_page(page));
 	page_nid = page_to_nid(page);
-	last_nid = page_nid_last(page);
+	last_nidpid = page_nidpid_last(page);
 	count_vm_numa_event(NUMA_HINT_FAULTS);
 	if (page_nid == this_nid)
 		count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL);
@@ -1374,7 +1374,7 @@ out:
 		page_unlock_anon_vma_read(anon_vma);
 
 	if (page_nid != -1)
-		task_numa_fault(last_nid, page_nid, HPAGE_PMD_NR, migrated);
+		task_numa_fault(last_nidpid, page_nid, HPAGE_PMD_NR, migrated);
 
 	return 0;
 }
@@ -1694,7 +1694,7 @@ static void __split_huge_page_refcount(struct page *page,
 		page_tail->mapping = page->mapping;
 
 		page_tail->index = page->index + i;
-		page_nid_xchg_last(page_tail, page_nid_last(page));
+		page_nidpid_xchg_last(page_tail, page_nidpid_last(page));
 
 		BUG_ON(!PageAnon(page_tail));
 		BUG_ON(!PageUptodate(page_tail));
mm/memory.c +8 -8
index e335ec0..948ec32 100644
@@ -69,8 +69,8 @@
 
 #include "internal.h"
 
-#ifdef LAST_NID_NOT_IN_PAGE_FLAGS
-#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_nid.
+#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS
+#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_nidpid.
 #endif
 
 #ifndef CONFIG_NEED_MULTIPLE_NODES
@@ -3547,7 +3547,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	struct page *page = NULL;
 	spinlock_t *ptl;
 	int page_nid = -1;
-	int last_nid;
+	int last_nidpid;
 	int target_nid;
 	bool migrated = false;
 
@@ -3578,7 +3578,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	}
 	BUG_ON(is_zero_pfn(page_to_pfn(page)));
 
-	last_nid = page_nid_last(page);
+	last_nidpid = page_nidpid_last(page);
 	page_nid = page_to_nid(page);
 	target_nid = numa_migrate_prep(page, vma, addr, page_nid);
 	pte_unmap_unlock(ptep, ptl);
@@ -3594,7 +3594,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 
 out:
 	if (page_nid != -1)
-		task_numa_fault(last_nid, page_nid, 1, migrated);
+		task_numa_fault(last_nidpid, page_nid, 1, migrated);
 	return 0;
 }
 
@@ -3609,7 +3609,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	unsigned long offset;
 	spinlock_t *ptl;
 	bool numa = false;
-	int last_nid;
+	int last_nidpid;
 
 	spin_lock(&mm->page_table_lock);
 	pmd = *pmdp;
@@ -3654,7 +3654,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		if (unlikely(!page))
 			continue;
 
-		last_nid = page_nid_last(page);
+		last_nidpid = page_nidpid_last(page);
 		page_nid = page_to_nid(page);
 		target_nid = numa_migrate_prep(page, vma, addr, page_nid);
 		pte_unmap_unlock(pte, ptl);
@@ -3667,7 +3667,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		}
 
 		if (page_nid != -1)
-			task_numa_fault(last_nid, page_nid, 1, migrated);
+			task_numa_fault(last_nidpid, page_nid, 1, migrated);
 
 		pte = pte_offset_map_lock(mm, pmdp, addr, &ptl);
 	}
mm/mempolicy.c +5 -3
index 4baf12e..8e2a364 100644
@@ -2292,9 +2292,11 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long
 
 	/* Migrate the page towards the node whose CPU is referencing it */
 	if (pol->flags & MPOL_F_MORON) {
-		int last_nid;
+		int last_nidpid;
+		int this_nidpid;
 
 		polnid = numa_node_id();
+		this_nidpid = nid_pid_to_nidpid(polnid, current->pid);
 
 		/*
 		 * Multi-stage node selection is used in conjunction
@@ -2317,8 +2319,8 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long
 		 * it less likely we act on an unlikely task<->page
 		 * relation.
 		 */
-		last_nid = page_nid_xchg_last(page, polnid);
-		if (last_nid != polnid)
+		last_nidpid = page_nidpid_xchg_last(page, this_nidpid);
+		if (!nidpid_pid_unset(last_nidpid) && nidpid_to_nid(last_nidpid) != polnid)
 			goto out;
 	}
 
mm/migrate.c +2 -2
index e7191a3..cea719d 100644
@@ -1495,7 +1495,7 @@ static struct page *alloc_misplaced_dst_page(struct page *page,
 					  __GFP_NOWARN) &
 					 ~GFP_IOFS, 0);
 	if (newpage)
-		page_nid_xchg_last(newpage, page_nid_last(page));
+		page_nidpid_xchg_last(newpage, page_nidpid_last(page));
 
 	return newpage;
 }
@@ -1672,7 +1672,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
 	if (!new_page)
 		goto out_fail;
 
-	page_nid_xchg_last(new_page, page_nid_last(page));
+	page_nidpid_xchg_last(new_page, page_nidpid_last(page));
 
 	isolated = numamigrate_isolate_page(pgdat, page);
 	if (!isolated) {
mm/mm_init.c +9 -9
index 633c088..467de57 100644
@@ -71,26 +71,26 @@ void __init mminit_verify_pageflags_layout(void)
 	unsigned long or_mask, add_mask;
 
 	shift = 8 * sizeof(unsigned long);
-	width = shift - SECTIONS_WIDTH - NODES_WIDTH - ZONES_WIDTH - LAST_NID_SHIFT;
+	width = shift - SECTIONS_WIDTH - NODES_WIDTH - ZONES_WIDTH - LAST_NIDPID_SHIFT;
 	mminit_dprintk(MMINIT_TRACE, "pageflags_layout_widths",
-		"Section %d Node %d Zone %d Lastnid %d Flags %d\n",
+		"Section %d Node %d Zone %d Lastnidpid %d Flags %d\n",
 		SECTIONS_WIDTH,
 		NODES_WIDTH,
 		ZONES_WIDTH,
-		LAST_NID_WIDTH,
+		LAST_NIDPID_WIDTH,
 		NR_PAGEFLAGS);
 	mminit_dprintk(MMINIT_TRACE, "pageflags_layout_shifts",
-		"Section %d Node %d Zone %d Lastnid %d\n",
+		"Section %d Node %d Zone %d Lastnidpid %d\n",
 		SECTIONS_SHIFT,
 		NODES_SHIFT,
 		ZONES_SHIFT,
-		LAST_NID_SHIFT);
+		LAST_NIDPID_SHIFT);
 	mminit_dprintk(MMINIT_TRACE, "pageflags_layout_pgshifts",
-		"Section %lu Node %lu Zone %lu Lastnid %lu\n",
+		"Section %lu Node %lu Zone %lu Lastnidpid %lu\n",
 		(unsigned long)SECTIONS_PGSHIFT,
 		(unsigned long)NODES_PGSHIFT,
 		(unsigned long)ZONES_PGSHIFT,
-		(unsigned long)LAST_NID_PGSHIFT);
+		(unsigned long)LAST_NIDPID_PGSHIFT);
 	mminit_dprintk(MMINIT_TRACE, "pageflags_layout_nodezoneid",
 		"Node/Zone ID: %lu -> %lu\n",
 		(unsigned long)(ZONEID_PGOFF + ZONEID_SHIFT),
@@ -102,9 +102,9 @@ void __init mminit_verify_pageflags_layout(void)
 	mminit_dprintk(MMINIT_TRACE, "pageflags_layout_nodeflags",
 		"Node not in page flags");
 #endif
-#ifdef LAST_NID_NOT_IN_PAGE_FLAGS
+#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS
 	mminit_dprintk(MMINIT_TRACE, "pageflags_layout_nodeflags",
-		"Last nid not in page flags");
+		"Last nidpid not in page flags");
 #endif
 
 	if (SECTIONS_WIDTH) {
mm/mmzone.c +7 -7
index 2ac0afb..25bb477 100644
@@ -97,20 +97,20 @@ void lruvec_init(struct lruvec *lruvec)
 		INIT_LIST_HEAD(&lruvec->lists[lru]);
 }
 
-#if defined(CONFIG_NUMA_BALANCING) && !defined(LAST_NID_NOT_IN_PAGE_FLAGS)
-int page_nid_xchg_last(struct page *page, int nid)
+#if defined(CONFIG_NUMA_BALANCING) && !defined(LAST_NIDPID_NOT_IN_PAGE_FLAGS)
+int page_nidpid_xchg_last(struct page *page, int nidpid)
 {
 	unsigned long old_flags, flags;
-	int last_nid;
+	int last_nidpid;
 
 	do {
 		old_flags = flags = page->flags;
-		last_nid = page_nid_last(page);
+		last_nidpid = page_nidpid_last(page);
 
-		flags &= ~(LAST_NID_MASK << LAST_NID_PGSHIFT);
-		flags |= (nid & LAST_NID_MASK) << LAST_NID_PGSHIFT;
+		flags &= ~(LAST_NIDPID_MASK << LAST_NIDPID_PGSHIFT);
+		flags |= (nidpid & LAST_NIDPID_MASK) << LAST_NIDPID_PGSHIFT;
 	} while (unlikely(cmpxchg(&page->flags, old_flags, flags) != old_flags));
 
-	return last_nid;
+	return last_nidpid;
 }
 #endif
mm/mprotect.c +17 -9
index 41e0292..f0b087d 100644
@@ -37,14 +37,15 @@ static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot)
 
 static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
 		unsigned long addr, unsigned long end, pgprot_t newprot,
-		int dirty_accountable, int prot_numa, bool *ret_all_same_node)
+		int dirty_accountable, int prot_numa, bool *ret_all_same_nidpid)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	pte_t *pte, oldpte;
 	spinlock_t *ptl;
 	unsigned long pages = 0;
-	bool all_same_node = true;
+	bool all_same_nidpid = true;
 	int last_nid = -1;
+	int last_pid = -1;
 
 	pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
 	arch_enter_lazy_mmu_mode();
@@ -63,11 +64,18 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
 
 				page = vm_normal_page(vma, addr, oldpte);
 				if (page) {
-					int this_nid = page_to_nid(page);
+					int nidpid = page_nidpid_last(page);
+					int this_nid = nidpid_to_nid(nidpid);
+					int this_pid = nidpid_to_pid(nidpid);
+
 					if (last_nid == -1)
 						last_nid = this_nid;
-					if (last_nid != this_nid)
-						all_same_node = false;
+					if (last_pid == -1)
+						last_pid = this_pid;
+					if (last_nid != this_nid ||
+					    last_pid != this_pid) {
+						all_same_nidpid = false;
+					}
 
 					if (!pte_numa(oldpte)) {
 						ptent = pte_mknuma(ptent);
@@ -107,7 +115,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
 	arch_leave_lazy_mmu_mode();
 	pte_unmap_unlock(pte - 1, ptl);
 
-	*ret_all_same_node = all_same_node;
+	*ret_all_same_nidpid = all_same_nidpid;
 	return pages;
 }
 
@@ -134,7 +142,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,
 	pmd_t *pmd;
 	unsigned long next;
 	unsigned long pages = 0;
-	bool all_same_node;
+	bool all_same_nidpid;
 
 	pmd = pmd_offset(pud, addr);
 	do {
@@ -158,7 +166,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,
 		if (pmd_none_or_clear_bad(pmd))
 			continue;
 		pages += change_pte_range(vma, pmd, addr, next, newprot,
-				 dirty_accountable, prot_numa, &all_same_node);
+				 dirty_accountable, prot_numa, &all_same_nidpid);
 
 		/*
 		 * If we are changing protections for NUMA hinting faults then
@@ -166,7 +174,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,
 		 * node. This allows a regular PMD to be handled as one fault
 		 * and effectively batches the taking of the PTL
 		 */
-		if (prot_numa && all_same_node)
+		if (prot_numa && all_same_nidpid)
 			change_pmd_protnuma(vma->vm_mm, addr, pmd);
 	} while (pmd++, addr = next, addr != end);
 
mm/page_alloc.c +2 -2
index b100255..7bf960e 100644
@@ -622,7 +622,7 @@ static inline int free_pages_check(struct page *page)
 		bad_page(page);
 		return 1;
 	}
-	page_nid_reset_last(page);
+	page_nidpid_reset_last(page);
 	if (page->flags & PAGE_FLAGS_CHECK_AT_PREP)
 		page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP;
 	return 0;
@@ -3944,7 +3944,7 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,
 		mminit_verify_page_links(page, zone, nid, pfn);
 		init_page_count(page);
 		page_mapcount_reset(page);
-		page_nid_reset_last(page);
+		page_nidpid_reset_last(page);
 		SetPageReserved(page);
 		/*
 		 * Mark the block movable so that blocks are reserved for
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#33 lore raw

Re: [PATCH 29/63] sched: Set preferred NUMA node based on number of private faults

Quoted 10 lines
(quoted diff) +0 -0
@@ -2317,8 +2319,8 @@ int mpol_misplaced(struct page *page, struct

vm_area_struct *vma, unsigned long
* it less likely we act on an unlikely task<->page
* relation.
*/

mm/mempolicy.c +2 -2
-               last_nid = page_nid_xchg_last(page, polnid);
-               if (last_nid != polnid)
+               last_nidpid = page_nidpid_xchg_last(page, this_nidpid);
+               if (!nidpid_pid_unset(last_nidpid) &&

nidpid_to_nid(last_nidpid) != polnid)

Quoted 4 lines
                        goto out;
        }

Suppose that the first accessed page it will check the "if (curnid != polnid)" and maybe migrate the page later.
so maybe modify like that:

+ last_nidpid = page_nidpid_xchg_last(page, this_nidpid);
+ if (nidpid_pid_unset(last_nidpid))
+ goto out;
+ else if (nidpid_to_nid(last_nidpid) != polnid)

                        goto out;

Best,
Figo.zhang

author reviewer
#34 lore raw

PATCH 30/63 sched: Do not migrate memory immediately after switching node

In reply to Mel Gorman

From: Rik van Riel <[email protected]>

The load balancer can move tasks between nodes and does not take NUMA locality into account. With automatic NUMA balancing this may result in the tasks working set being migrated to the new node. However, as the fault buffer will still store faults from the old node the schduler may decide to reset the preferred node and migrate the task back resulting in more migrations.

The ideal would be that the scheduler did not migrate tasks with a heavy memory footprint but this may result nodes being overloaded. We could also discard the fault information on task migration but this would still cause all the tasks working set to be migrated. This patch simply avoids migrating the memory for a short time after a task is migrated.

---
 kernel/sched/core.c |  2 +-
 kernel/sched/fair.c | 18 ++++++++++++++++--
 mm/mempolicy.c      | 12 ++++++++++++
 3 files changed, 29 insertions(+), 3 deletions(-)
kernel/sched/core.c +1 -1
index e94509d..374da2b 100644
@@ -1641,7 +1641,7 @@ static void __sched_fork(struct task_struct *p)
 
 	p->node_stamp = 0ULL;
 	p->numa_scan_seq = p->mm ? p->mm->numa_scan_seq : 0;
-	p->numa_migrate_seq = 0;
+	p->numa_migrate_seq = 1;
 	p->numa_scan_period = sysctl_numa_balancing_scan_delay;
 	p->numa_preferred_nid = -1;
 	p->numa_work.next = &p->numa_work;
kernel/sched/fair.c +16 -2
index 891105c..f136057 100644
@@ -884,7 +884,7 @@ static unsigned int task_scan_max(struct task_struct *p)
  * the preferred node but still allow the scheduler to move the task again if
  * the nodes CPUs are overloaded.
  */
-unsigned int sysctl_numa_balancing_settle_count __read_mostly = 3;
+unsigned int sysctl_numa_balancing_settle_count __read_mostly = 4;
 
 static inline int task_faults_idx(int nid, int priv)
 {
@@ -980,7 +980,7 @@ static void task_numa_placement(struct task_struct *p)
 
 		/* Update the preferred nid and migrate task if possible */
 		p->numa_preferred_nid = max_nid;
-		p->numa_migrate_seq = 0;
+		p->numa_migrate_seq = 1;
 		migrate_task_to(p, preferred_cpu);
 	}
 }
@@ -4074,6 +4074,20 @@ static void move_task(struct task_struct *p, struct lb_env *env)
 	set_task_cpu(p, env->dst_cpu);
 	activate_task(env->dst_rq, p, 0);
 	check_preempt_curr(env->dst_rq, p, 0);
+#ifdef CONFIG_NUMA_BALANCING
+	if (p->numa_preferred_nid != -1) {
+		int src_nid = cpu_to_node(env->src_cpu);
+		int dst_nid = cpu_to_node(env->dst_cpu);
+
+		/*
+		 * If the load balancer has moved the task then limit
+		 * migrations from taking place in the short term in
+		 * case this is a short-lived migration.
+		 */
+		if (src_nid != dst_nid && dst_nid != p->numa_preferred_nid)
+			p->numa_migrate_seq = 0;
+	}
+#endif
 }
 
 /*
mm/mempolicy.c +12 -0
index 8e2a364..adc93b2 100644
@@ -2322,6 +2322,18 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long
 		last_nidpid = page_nidpid_xchg_last(page, this_nidpid);
 		if (!nidpid_pid_unset(last_nidpid) && nidpid_to_nid(last_nidpid) != polnid)
 			goto out;
+
+#ifdef CONFIG_NUMA_BALANCING
+		/*
+		 * If the scheduler has just moved us away from our
+		 * preferred node, do not bother migrating pages yet.
+		 * This way a short and temporary process migration will
+		 * not cause excessive memory migration.
+		 */
+		if (polnid != current->numa_preferred_nid &&
+				!current->numa_migrate_seq)
+			goto out;
+#endif
 	}
 
 	if (curnid != polnid)
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#35 lore raw

PATCH 31/63 mm: numa: only unmap migrate-on-fault VMAs

In reply to Mel Gorman

There is a 90% regression observed with a large Oracle performance test on a 4 node system. Profiles indicated that the overhead was due to contention on sp_lock when looking up shared memory policies. These policies do not have the appropriate flags to allow them to be automatically balanced so trapping faults on them is pointless. This patch skips VMAs that do not have MPOL_F_MOF set.

---
 include/linux/mempolicy.h |  1 +
 kernel/sched/fair.c       |  2 +-
 mm/mempolicy.c            | 24 ++++++++++++++++++++++++
 3 files changed, 26 insertions(+), 1 deletion(-)
include/linux/mempolicy.h +1 -0
index 0d7df39..f6b2c4d 100644
@@ -136,6 +136,7 @@ struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp,
 
 struct mempolicy *get_vma_policy(struct task_struct *tsk,
 		struct vm_area_struct *vma, unsigned long addr);
+bool vma_policy_mof(struct task_struct *task, struct vm_area_struct *vma);
 
 extern void numa_default_policy(void);
 extern void numa_policy_init(void);
kernel/sched/fair.c +1 -1
index f136057..f9329d5 100644
@@ -1130,7 +1130,7 @@ void task_numa_work(struct callback_head *work)
 		vma = mm->mmap;
 	}
 	for (; vma; vma = vma->vm_next) {
-		if (!vma_migratable(vma))
+		if (!vma_migratable(vma) || !vma_policy_mof(p, vma))
 			continue;
 
 		do {
mm/mempolicy.c +24 -0
index adc93b2..54004a8 100644
@@ -1633,6 +1633,30 @@ struct mempolicy *get_vma_policy(struct task_struct *task,
 	return pol;
 }
 
+bool vma_policy_mof(struct task_struct *task, struct vm_area_struct *vma)
+{
+	struct mempolicy *pol = get_task_policy(task);
+	if (vma) {
+		if (vma->vm_ops && vma->vm_ops->get_policy) {
+			bool ret = false;
+
+			pol = vma->vm_ops->get_policy(vma, vma->vm_start);
+			if (pol && (pol->flags & MPOL_F_MOF))
+				ret = true;
+			mpol_cond_put(pol);
+
+			return ret;
+		} else if (vma->vm_policy) {
+			pol = vma->vm_policy;
+		}
+	}
+
+	if (!pol)
+		return default_policy.flags & MPOL_F_MOF;
+
+	return pol->flags & MPOL_F_MOF;
+}
+
 static int apply_policy_zone(struct mempolicy *policy, enum zone_type zone)
 {
 	enum zone_type dynamic_policy_zone = policy_zone;
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#36 lore raw

PATCH 32/63 sched: Avoid overloading CPUs on a preferred NUMA node

In reply to Mel Gorman

This patch replaces find_idlest_cpu_node with task_numa_find_cpu. find_idlest_cpu_node has two critical limitations. It does not take the scheduling class into account when calculating the load and it is unsuitable for using when comparing loads between NUMA nodes.

task_numa_find_cpu uses similar load calculations to wake_affine() when selecting the least loaded CPU within a scheduling domain common to the source and destimation nodes. It avoids causing CPU load imbalances in the machine by refusing to migrate if the relative load on the target CPU is higher than the source CPU.

---
 kernel/sched/fair.c | 131 ++++++++++++++++++++++++++++++++++++++++------------
 1 file changed, 102 insertions(+), 29 deletions(-)
kernel/sched/fair.c +102 -29
index f9329d5..b130014 100644
@@ -901,28 +901,114 @@ static inline unsigned long task_faults(struct task_struct *p, int nid)
 }
 
 static unsigned long weighted_cpuload(const int cpu);
+static unsigned long source_load(int cpu, int type);
+static unsigned long target_load(int cpu, int type);
+static unsigned long power_of(int cpu);
+static long effective_load(struct task_group *tg, int cpu, long wl, long wg);
 
+struct numa_stats {
+	unsigned long load;
+	s64 eff_load;
+	unsigned long faults;
+};
 
-static int
-find_idlest_cpu_node(int this_cpu, int nid)
-{
-	unsigned long load, min_load = ULONG_MAX;
-	int i, idlest_cpu = this_cpu;
+struct task_numa_env {
+	struct task_struct *p;
 
-	BUG_ON(cpu_to_node(this_cpu) == nid);
+	int src_cpu, src_nid;
+	int dst_cpu, dst_nid;
 
-	rcu_read_lock();
-	for_each_cpu(i, cpumask_of_node(nid)) {
-		load = weighted_cpuload(i);
+	struct numa_stats src_stats, dst_stats;
 
-		if (load < min_load) {
-			min_load = load;
-			idlest_cpu = i;
+	unsigned long best_load;
+	int best_cpu;
+};
+
+static int task_numa_migrate(struct task_struct *p)
+{
+	int node_cpu = cpumask_first(cpumask_of_node(p->numa_preferred_nid));
+	struct task_numa_env env = {
+		.p = p,
+		.src_cpu = task_cpu(p),
+		.src_nid = cpu_to_node(task_cpu(p)),
+		.dst_cpu = node_cpu,
+		.dst_nid = p->numa_preferred_nid,
+		.best_load = ULONG_MAX,
+		.best_cpu = task_cpu(p),
+	};
+	struct sched_domain *sd;
+	int cpu;
+	struct task_group *tg = task_group(p);
+	unsigned long weight;
+	bool balanced;
+	int imbalance_pct, idx = -1;
+
+	/*
+	 * Find the lowest common scheduling domain covering the nodes of both
+	 * the CPU the task is currently running on and the target NUMA node.
+	 */
+	rcu_read_lock();
+	for_each_domain(env.src_cpu, sd) {
+		if (cpumask_test_cpu(node_cpu, sched_domain_span(sd))) {
+			/*
+			 * busy_idx is used for the load decision as it is the
+			 * same index used by the regular load balancer for an
+			 * active cpu.
+			 */
+			idx = sd->busy_idx;
+			imbalance_pct = sd->imbalance_pct;
+			break;
 		}
 	}
 	rcu_read_unlock();
 
-	return idlest_cpu;
+	if (WARN_ON_ONCE(idx == -1))
+		return 0;
+
+	/*
+	 * XXX the below is mostly nicked from wake_affine(); we should
+	 * see about sharing a bit if at all possible; also it might want
+	 * some per entity weight love.
+	 */
+	weight = p->se.load.weight;
+	env.src_stats.load = source_load(env.src_cpu, idx);
+	env.src_stats.eff_load = 100 + (imbalance_pct - 100) / 2;
+	env.src_stats.eff_load *= power_of(env.src_cpu);
+	env.src_stats.eff_load *= env.src_stats.load + effective_load(tg, env.src_cpu, -weight, -weight);
+
+	for_each_cpu(cpu, cpumask_of_node(env.dst_nid)) {
+		env.dst_cpu = cpu;
+		env.dst_stats.load = target_load(cpu, idx);
+
+		/* If the CPU is idle, use it */
+		if (!env.dst_stats.load) {
+			env.best_cpu = cpu;
+			goto migrate;
+		}
+
+		/* Otherwise check the target CPU load */
+		env.dst_stats.eff_load = 100;
+		env.dst_stats.eff_load *= power_of(cpu);
+		env.dst_stats.eff_load *= env.dst_stats.load + effective_load(tg, cpu, weight, weight);
+
+		/*
+		 * Destination is considered balanced if the destination CPU is
+		 * less loaded than the source CPU. Unfortunately there is a
+		 * risk that a task running on a lightly loaded CPU will not
+		 * migrate to its preferred node due to load imbalances.
+		 */
+		balanced = (env.dst_stats.eff_load <= env.src_stats.eff_load);
+		if (!balanced)
+			continue;
+
+		if (env.dst_stats.eff_load < env.best_load) {
+			env.best_load = env.dst_stats.eff_load;
+			env.best_cpu = cpu;
+		}
+	}
+
+migrate:
+	return migrate_task_to(p, env.best_cpu);
 }
 
 static void task_numa_placement(struct task_struct *p)
@@ -966,22 +1052,10 @@ static void task_numa_placement(struct task_struct *p)
 	 * the working set placement.
 	 */
 	if (max_faults && max_nid != p->numa_preferred_nid) {
-		int preferred_cpu;
-
-		/*
-		 * If the task is not on the preferred node then find the most
-		 * idle CPU to migrate to.
-		 */
-		preferred_cpu = task_cpu(p);
-		if (cpu_to_node(preferred_cpu) != max_nid) {
-			preferred_cpu = find_idlest_cpu_node(preferred_cpu,
-							     max_nid);
-		}
-
 		/* Update the preferred nid and migrate task if possible */
 		p->numa_preferred_nid = max_nid;
 		p->numa_migrate_seq = 1;
-		migrate_task_to(p, preferred_cpu);
+		task_numa_migrate(p);
 	}
 }
 
@@ -3274,7 +3348,7 @@ static long effective_load(struct task_group *tg, int cpu, long wl, long wg)
 {
 	struct sched_entity *se = tg->se[cpu];
 
-	if (!tg->parent)	/* the trivial, non-cgroup case */
+	if (!tg->parent || !wl)	/* the trivial, non-cgroup case */
 		return wl;
 
 	for_each_sched_entity(se) {
@@ -3327,8 +3401,7 @@ static long effective_load(struct task_group *tg, int cpu, long wl, long wg)
 }
 #else
 
-static inline unsigned long effective_load(struct task_group *tg, int cpu,
-		unsigned long wl, unsigned long wg)
+static long effective_load(struct task_group *tg, int cpu, long wl, long wg)
 {
 	return wl;
 }
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#37 lore raw

PATCH 33/63 sched: Retry migration of tasks to CPU on a preferred node

In reply to Mel Gorman

When a preferred node is selected for a tasks there is an attempt to migrate the task to a CPU there. This may fail in which case the task will only migrate if the active load balancer takes action. This may never happen if the conditions are not right. This patch will check at NUMA hinting fault time if another attempt should be made to migrate the task. It will only make an attempt once every five seconds.

---
 include/linux/sched.h |  1 +
 kernel/sched/fair.c   | 30 +++++++++++++++++++++++-------
 2 files changed, 24 insertions(+), 7 deletions(-)
include/linux/sched.h +1 -0
index 6eb8fa6..3418b0b 100644
@@ -1333,6 +1333,7 @@ struct task_struct {
 	int numa_migrate_seq;
 	unsigned int numa_scan_period;
 	unsigned int numa_scan_period_max;
+	unsigned long numa_migrate_retry;
 	u64 node_stamp;			/* migration stamp  */
 	struct callback_head numa_work;
 
kernel/sched/fair.c +23 -7
index b130014..00ccc68 100644
@@ -1011,6 +1011,23 @@ migrate:
 	return migrate_task_to(p, env.best_cpu);
 }
 
+/* Attempt to migrate a task to a CPU on the preferred node. */
+static void numa_migrate_preferred(struct task_struct *p)
+{
+	/* Success if task is already running on preferred CPU */
+	p->numa_migrate_retry = 0;
+	if (cpu_to_node(task_cpu(p)) == p->numa_preferred_nid)
+		return;
+
+	/* This task has no NUMA fault statistics yet */
+	if (unlikely(p->numa_preferred_nid == -1))
+		return;
+
+	/* Otherwise, try migrate to a CPU on the preferred node */
+	if (task_numa_migrate(p) != 0)
+		p->numa_migrate_retry = jiffies + HZ*5;
+}
+
 static void task_numa_placement(struct task_struct *p)
 {
 	int seq, nid, max_nid = -1;
@@ -1045,17 +1062,12 @@ static void task_numa_placement(struct task_struct *p)
 		}
 	}
 
-	/*
-	 * Record the preferred node as the node with the most faults,
-	 * requeue the task to be running on the idlest CPU on the
-	 * preferred node and reset the scanning rate to recheck
-	 * the working set placement.
-	 */
+	/* Preferred node as the node with the most faults */
 	if (max_faults && max_nid != p->numa_preferred_nid) {
 		/* Update the preferred nid and migrate task if possible */
 		p->numa_preferred_nid = max_nid;
 		p->numa_migrate_seq = 1;
-		task_numa_migrate(p);
+		numa_migrate_preferred(p);
 	}
 }
 
@@ -1111,6 +1123,10 @@ void task_numa_fault(int last_nidpid, int node, int pages, bool migrated)
 
 	task_numa_placement(p);
 
+	/* Retry task to preferred node migration if it previously failed */
+	if (p->numa_migrate_retry && time_after(jiffies, p->numa_migrate_retry))
+		numa_migrate_preferred(p);
+
 	p->numa_faults_buffer[task_faults_idx(node, priv)] += pages;
 }
 
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#38 lore raw

PATCH 34/63 sched: numa: increment numa_migrate_seq when task runs in correct location

In reply to Mel Gorman

From: Rik van Riel <[email protected]>

When a task is already running on its preferred node, increment numa_migrate_seq to indicate that the task is settled if migration is temporarily disabled, and memory should migrate towards it.

[[email protected]: Only increment migrate_seq if migration temporarily disabled]
---
 kernel/sched/fair.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
kernel/sched/fair.c +9 -1
index 00ccc68..1a17e5e 100644
@@ -1016,8 +1016,16 @@ static void numa_migrate_preferred(struct task_struct *p)
 {
 	/* Success if task is already running on preferred CPU */
 	p->numa_migrate_retry = 0;
-	if (cpu_to_node(task_cpu(p)) == p->numa_preferred_nid)
+	if (cpu_to_node(task_cpu(p)) == p->numa_preferred_nid) {
+		/*
+		 * If migration is temporarily disabled due to a task migration
+		 * then re-enable it now as the task is running on its
+		 * preferred node and memory should migrate locally
+		 */
+		if (!p->numa_migrate_seq)
+			p->numa_migrate_seq++;
 		return;
+	}
 
 	/* This task has no NUMA fault statistics yet */
 	if (unlikely(p->numa_preferred_nid == -1))
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#39 lore raw

PATCH 35/63 sched: numa: Do not trap hinting faults for shared libraries

In reply to Mel Gorman

NUMA hinting faults will not migrate a shared executable page mapped by multiple processes on the grounds that the data is probably in the CPU cache already and the page may just bounce between tasks running on multipl nodes. Even if the migration is avoided, there is still the overhead of trapping the fault, updating the statistics, making scheduler placement decisions based on the information etc. If we are never going to migrate the page, it is overhead for no gain and worse a process may be placed on a sub-optimal node for shared executable pages. This patch avoids trapping faults for shared libraries entirely.

---
 kernel/sched/fair.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
kernel/sched/fair.c +10 -0
index 1a17e5e..cd90e44 100644
@@ -1231,6 +1231,16 @@ void task_numa_work(struct callback_head *work)
 		if (!vma_migratable(vma) || !vma_policy_mof(p, vma))
 			continue;
 
+		/*
+		 * Shared library pages mapped by multiple processes are not
+		 * migrated as it is expected they are cache replicated. Avoid
+		 * hinting faults in read-only file-backed mappings or the vdso
+		 * as migrating the pages will be of marginal benefit.
+		 */
+		if (!vma->vm_mm ||
+		    (vma->vm_file && (vma->vm_flags & (VM_READ|VM_WRITE)) == (VM_READ)))
+			continue;
+
 		do {
 			start = max(start, vma->vm_start);
 			end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#40 lore raw

PATCH 36/63 mm: numa: Only trap pmd hinting faults if we would otherwise trap PTE faults

In reply to Mel Gorman

Base page PMD faulting is meant to batch handle NUMA hinting faults from PTEs. However, even is no PTE faults would ever be handled within a range the kernel still traps PMD hinting faults. This patch avoids the overhead.

---
 mm/mprotect.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
mm/mprotect.c +5 -2
index f0b087d..5aae390 100644
@@ -146,6 +146,8 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,
 
 	pmd = pmd_offset(pud, addr);
 	do {
+		unsigned long this_pages;
+
 		next = pmd_addr_end(addr, end);
 		if (pmd_trans_huge(*pmd)) {
 			if (next - addr != HPAGE_PMD_SIZE)
@@ -165,8 +167,9 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,
 		}
 		if (pmd_none_or_clear_bad(pmd))
 			continue;
-		pages += change_pte_range(vma, pmd, addr, next, newprot,
+		this_pages = change_pte_range(vma, pmd, addr, next, newprot,
 				 dirty_accountable, prot_numa, &all_same_nidpid);
+		pages += this_pages;
 
 		/*
 		 * If we are changing protections for NUMA hinting faults then
@@ -174,7 +177,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,
 		 * node. This allows a regular PMD to be handled as one fault
 		 * and effectively batches the taking of the PTL
 		 */
-		if (prot_numa && all_same_nidpid)
+		if (prot_numa && this_pages && all_same_nidpid)
 			change_pmd_protnuma(vma->vm_mm, addr, pmd);
 	} while (pmd++, addr = next, addr != end);
 
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#41 lore raw

PATCH 37/63 stop_machine: Introduce stop_two_cpus()

In reply to Mel Gorman

From: Peter Zijlstra <[email protected]>

Introduce stop_two_cpus() in order to allow controlled swapping of two tasks. It repurposes the stop_machine() state machine but only stops the two cpus which we can do with on-stack structures and avoid machine wide synchronization issues.

The ordering of CPUs is important to avoid deadlocks. If unordered then two cpus calling stop_two_cpus on each other simultaneously would attempt to queue in the opposite order on each CPU causing an AB-BA style deadlock. By always having the lowest number CPU doing the queueing of works, we can guarantee that works are always queued in the same order, and deadlocks are avoided.

---
 include/linux/stop_machine.h |   1 +
 kernel/stop_machine.c        | 272 +++++++++++++++++++++++++++----------------
 2 files changed, 175 insertions(+), 98 deletions(-)
include/linux/stop_machine.h +1 -0
index 3b5e910..d2abbdb 100644
@@ -28,6 +28,7 @@ struct cpu_stop_work {
 };
 
 int stop_one_cpu(unsigned int cpu, cpu_stop_fn_t fn, void *arg);
+int stop_two_cpus(unsigned int cpu1, unsigned int cpu2, cpu_stop_fn_t fn, void *arg);
 void stop_one_cpu_nowait(unsigned int cpu, cpu_stop_fn_t fn, void *arg,
 			 struct cpu_stop_work *work_buf);
 int stop_cpus(const struct cpumask *cpumask, cpu_stop_fn_t fn, void *arg);
kernel/stop_machine.c +174 -98

318 lines not on this page · raw message

Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#42 lore raw

PATCH 38/63 sched: Introduce migrate_swap()

In reply to Mel Gorman

From: Peter Zijlstra <[email protected]>

Use the new stop_two_cpus() to implement migrate_swap(), a function that flips two tasks between their respective cpus.

I'm fairly sure there's a less crude way than employing the stop_two_cpus() method, but everything I tried either got horribly fragile and/or complex. So keep it simple for now.

The notable detail is how we 'migrate' tasks that aren't runnable anymore. We'll make it appear like we migrated them before they went to sleep. The sole difference is the previous cpu in the wakeup path, so we override this.

TODO: I'm fairly sure we can get rid of the wake_cpu != -1 test by keeping wake_cpu to the actual task cpu; just couldn't be bothered to think through all the cases.

---
 include/linux/sched.h    |   1 +
 kernel/sched/core.c      | 103 ++++++++++++++++++++++++++++++++++++++++++++---
 kernel/sched/fair.c      |   3 +-
 kernel/sched/idle_task.c |   2 +-
 kernel/sched/rt.c        |   5 +--
 kernel/sched/sched.h     |   3 +-
 kernel/sched/stop_task.c |   2 +-
 7 files changed, 105 insertions(+), 14 deletions(-)
include/linux/sched.h +1 -0
index 3418b0b..3e8c547 100644
@@ -1035,6 +1035,7 @@ struct task_struct {
 #ifdef CONFIG_SMP
 	struct llist_node wake_entry;
 	int on_cpu;
+	int wake_cpu;
 #endif
 	int on_rq;
 
kernel/sched/core.c +98 -5

140 lines not on this page · raw message

kernel/sched/fair.c +1 -2
index cd90e44..9b59f5d 100644
@@ -3659,11 +3659,10 @@ done:
  * preempt must be disabled.
  */
 static int
-select_task_rq_fair(struct task_struct *p, int sd_flag, int wake_flags)
+select_task_rq_fair(struct task_struct *p, int prev_cpu, int sd_flag, int wake_flags)
 {
 	struct sched_domain *tmp, *affine_sd = NULL, *sd = NULL;
 	int cpu = smp_processor_id();
-	int prev_cpu = task_cpu(p);
 	int new_cpu = cpu;
 	int want_affine = 0;
 	int sync = wake_flags & WF_SYNC;
kernel/sched/idle_task.c +1 -1
index d8da010..516c3d9 100644
@@ -9,7 +9,7 @@
 
 #ifdef CONFIG_SMP
 static int
-select_task_rq_idle(struct task_struct *p, int sd_flag, int flags)
+select_task_rq_idle(struct task_struct *p, int cpu, int sd_flag, int flags)
 {
 	return task_cpu(p); /* IDLE tasks as never migrated */
 }
kernel/sched/rt.c +1 -4
index 01970c8..d81866d 100644
@@ -1169,13 +1169,10 @@ static void yield_task_rt(struct rq *rq)
 static int find_lowest_rq(struct task_struct *task);
 
 static int
-select_task_rq_rt(struct task_struct *p, int sd_flag, int flags)
+select_task_rq_rt(struct task_struct *p, int cpu, int sd_flag, int flags)
 {
 	struct task_struct *curr;
 	struct rq *rq;
-	int cpu;
-
-	cpu = task_cpu(p);
 
 	if (p->nr_cpus_allowed == 1)
 		goto out;
kernel/sched/sched.h +2 -1
index 778f875..99b1ecd 100644
@@ -556,6 +556,7 @@ static inline u64 rq_clock_task(struct rq *rq)
 
 #ifdef CONFIG_NUMA_BALANCING
 extern int migrate_task_to(struct task_struct *p, int cpu);
+extern int migrate_swap(struct task_struct *, struct task_struct *);
 static inline void task_numa_free(struct task_struct *p)
 {
 	kfree(p->numa_faults);
@@ -988,7 +989,7 @@ struct sched_class {
 	void (*put_prev_task) (struct rq *rq, struct task_struct *p);
 
 #ifdef CONFIG_SMP
-	int  (*select_task_rq)(struct task_struct *p, int sd_flag, int flags);
+	int  (*select_task_rq)(struct task_struct *p, int task_cpu, int sd_flag, int flags);
 	void (*migrate_task_rq)(struct task_struct *p, int next_cpu);
 
 	void (*pre_schedule) (struct rq *this_rq, struct task_struct *task);
kernel/sched/stop_task.c +1 -1
index e08fbee..47197de 100644
@@ -11,7 +11,7 @@
 
 #ifdef CONFIG_SMP
 static int
-select_task_rq_stop(struct task_struct *p, int sd_flag, int flags)
+select_task_rq_stop(struct task_struct *p, int cpu, int sd_flag, int flags)
 {
 	return task_cpu(p); /* stop tasks as never migrate */
 }
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#43 lore raw

PATCH 39/63 sched: numa: Use a system-wide search to find swap/migration candidates

In reply to Mel Gorman

This patch implements a system-wide search for swap/migration candidates based on total NUMA hinting faults. It has a balance limit, however it doesn't properly consider total node balance.

In the old scheme a task selected a preferred node based on the highest number of private faults recorded on the node. In this scheme, the preferred node is based on the total number of faults. If the preferred node for a task changes then task_numa_migrate will search the whole system looking for tasks to swap with that would improve both the overall compute balance and minimise the expected number of remote NUMA hinting faults.

Not there is no guarantee that the node the source task is placed on by task_numa_migrate() has any relationship to the newly selected task->numa_preferred_nid due to compute overloading.

[[email protected]: Do not swap with tasks that cannot run on source cpu]
---
 kernel/sched/core.c  |   4 +
 kernel/sched/fair.c  | 253 ++++++++++++++++++++++++++++++++++++---------------
 kernel/sched/sched.h |  13 +++
 3 files changed, 199 insertions(+), 71 deletions(-)
kernel/sched/core.c +4 -0
index 67f2b7b..f7a44dd 100644
@@ -5248,6 +5248,7 @@ static void destroy_sched_domains(struct sched_domain *sd, int cpu)
  */
 DEFINE_PER_CPU(struct sched_domain *, sd_llc);
 DEFINE_PER_CPU(int, sd_llc_id);
+DEFINE_PER_CPU(struct sched_domain *, sd_numa);
 
 static void update_top_cache_domain(int cpu)
 {
@@ -5260,6 +5261,9 @@ static void update_top_cache_domain(int cpu)
 
 	rcu_assign_pointer(per_cpu(sd_llc, cpu), sd);
 	per_cpu(sd_llc_id, cpu) = id;
+
+	sd = lowest_flag_domain(cpu, SD_NUMA);
+	rcu_assign_pointer(per_cpu(sd_numa, cpu), sd);
 }
 
 /*
kernel/sched/fair.c +182 -71

320 lines not on this page · raw message

kernel/sched/sched.h +13 -0
index 99b1ecd..82ff03a 100644
@@ -608,8 +608,21 @@ static inline struct sched_domain *highest_flag_domain(int cpu, int flag)
 	return hsd;
 }
 
+static inline struct sched_domain *lowest_flag_domain(int cpu, int flag)
+{
+	struct sched_domain *sd;
+
+	for_each_domain(cpu, sd) {
+		if (sd->flags & flag)
+			break;
+	}
+
+	return sd;
+}
+
 DECLARE_PER_CPU(struct sched_domain *, sd_llc);
 DECLARE_PER_CPU(int, sd_llc_id);
+DECLARE_PER_CPU(struct sched_domain *, sd_numa);
 
 struct sched_group_power {
 	atomic_t ref;
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#44 lore raw

PATCH 40/63 sched: numa: Favor placing a task on the preferred node

In reply to Mel Gorman

A tasks preferred node is selected based on the number of faults recorded for a node but the actual task_numa_migate() conducts a global search regardless of the preferred nid. This patch checks if the preferred nid has capacity and if so, searches for a CPU within that node. This avoids a global search when the preferred node is not overloaded.

---
 kernel/sched/fair.c | 54 ++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 35 insertions(+), 19 deletions(-)
kernel/sched/fair.c +35 -19
index b0e25f9..99b6711 100644
@@ -1052,6 +1052,20 @@ unlock:
 	rcu_read_unlock();
 }
 
+static void task_numa_find_cpu(struct task_numa_env *env, long imp)
+{
+	int cpu;
+
+	for_each_cpu(cpu, cpumask_of_node(env->dst_nid)) {
+		/* Skip this CPU if the source task cannot migrate */
+		if (!cpumask_test_cpu(cpu, tsk_cpus_allowed(env->p)))
+			continue;
+
+		env->dst_cpu = cpu;
+		task_numa_compare(env, imp);
+	}
+}
+
 static int task_numa_migrate(struct task_struct *p)
 {
 	struct task_numa_env env = {
@@ -1068,7 +1082,8 @@ static int task_numa_migrate(struct task_struct *p)
 	};
 	struct sched_domain *sd;
 	unsigned long faults;
-	int nid, cpu, ret;
+	int nid, ret;
+	long imp;
 
 	/*
 	 * Pick the lowest SD_NUMA domain, as that would have the smallest
@@ -1085,28 +1100,29 @@ static int task_numa_migrate(struct task_struct *p)
 
 	faults = task_faults(p, env.src_nid);
 	update_numa_stats(&env.src_stats, env.src_nid);
+	env.dst_nid = p->numa_preferred_nid;
+	imp = task_faults(env.p, env.dst_nid) - faults;
+	update_numa_stats(&env.dst_stats, env.dst_nid);
 
-	/* Find an alternative node with relatively better statistics */
-	for_each_online_node(nid) {
-		long imp;
-
-		if (nid == env.src_nid)
-			continue;
-
-		/* Only consider nodes that recorded more faults */
-		imp = task_faults(p, nid) - faults;
-		if (imp < 0)
-			continue;
+	/*
+	 * If the preferred nid has capacity then use it. Otherwise find an
+	 * alternative node with relatively better statistics.
+	 */
+	if (env.dst_stats.has_capacity) {
+		task_numa_find_cpu(&env, imp);
+	} else {
+		for_each_online_node(nid) {
+			if (nid == env.src_nid || nid == p->numa_preferred_nid)
+				continue;
 
-		env.dst_nid = nid;
-		update_numa_stats(&env.dst_stats, env.dst_nid);
-		for_each_cpu(cpu, cpumask_of_node(nid)) {
-			/* Skip this CPU if the source task cannot migrate */
-			if (!cpumask_test_cpu(cpu, tsk_cpus_allowed(p)))
+			/* Only consider nodes that recorded more faults */
+			imp = task_faults(env.p, nid) - faults;
+			if (imp < 0)
 				continue;
 
-			env.dst_cpu = cpu;
-			task_numa_compare(&env, imp);
+			env.dst_nid = nid;
+			update_numa_stats(&env.dst_stats, env.dst_nid);
+			task_numa_find_cpu(&env, imp);
 		}
 	}
 
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#45 lore raw

PATCH 41/63 sched: numa: fix placement of workloads spread across multiple nodes

In reply to Mel Gorman

From: Rik van Riel <[email protected]>

The load balancer will spread workloads across multiple NUMA nodes, in order to balance the load on the system. This means that sometimes a task's preferred node has available capacity, but moving the task there will not succeed, because that would create too large an imbalance.

In that case, other NUMA nodes need to be considered.

---
 kernel/sched/fair.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
kernel/sched/fair.c +5 -6
index 99b6711..8ebed0a 100644
@@ -1104,13 +1104,12 @@ static int task_numa_migrate(struct task_struct *p)
 	imp = task_faults(env.p, env.dst_nid) - faults;
 	update_numa_stats(&env.dst_stats, env.dst_nid);
 
-	/*
-	 * If the preferred nid has capacity then use it. Otherwise find an
-	 * alternative node with relatively better statistics.
-	 */
-	if (env.dst_stats.has_capacity) {
+	/* If the preferred nid has capacity, try to use it. */
+	if (env.dst_stats.has_capacity)
 		task_numa_find_cpu(&env, imp);
-	} else {
+
+	/* No space available on the preferred nid. Look elsewhere. */
+	if (env.best_cpu == -1) {
 		for_each_online_node(nid) {
 			if (nid == env.src_nid || nid == p->numa_preferred_nid)
 				continue;
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#46 lore raw

PATCH 42/63 mm: numa: Change page last {nid,pid} into {cpu,pid}

In reply to Mel Gorman

From: Peter Zijlstra <[email protected]>

Change the per page last fault tracking to use cpu,pid instead of nid,pid. This will allow us to try and lookup the alternate task more easily. Note that even though it is the cpu that is store in the page flags that the mpol_misplaced decision is still based on the node.

---
 include/linux/mm.h                | 90 ++++++++++++++++++++++-----------------
 include/linux/mm_types.h          |  4 +-
 include/linux/page-flags-layout.h | 22 +++++-----
 kernel/bounds.c                   |  4 ++
 kernel/sched/fair.c               |  6 +--
 mm/huge_memory.c                  |  8 ++--
 mm/memory.c                       | 16 +++----
 mm/mempolicy.c                    | 16 ++++---
 mm/migrate.c                      |  4 +-
 mm/mm_init.c                      | 18 ++++----
 mm/mmzone.c                       | 14 +++---
 mm/mprotect.c                     | 28 ++++++------
 mm/page_alloc.c                   |  4 +-
 13 files changed, 125 insertions(+), 109 deletions(-)
include/linux/mm.h +50 -40
index 0a0db6c..61dc023 100644
@@ -588,11 +588,11 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
  * sets it, so none of the operations on it need to be atomic.
  */
 
-/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_NIDPID] | ... | FLAGS | */
+/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_CPUPID] | ... | FLAGS | */
 #define SECTIONS_PGOFF		((sizeof(unsigned long)*8) - SECTIONS_WIDTH)
 #define NODES_PGOFF		(SECTIONS_PGOFF - NODES_WIDTH)
 #define ZONES_PGOFF		(NODES_PGOFF - ZONES_WIDTH)
-#define LAST_NIDPID_PGOFF	(ZONES_PGOFF - LAST_NIDPID_WIDTH)
+#define LAST_CPUPID_PGOFF	(ZONES_PGOFF - LAST_CPUPID_WIDTH)
 
 /*
  * Define the bit shifts to access each section.  For non-existent
@@ -602,7 +602,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
 #define SECTIONS_PGSHIFT	(SECTIONS_PGOFF * (SECTIONS_WIDTH != 0))
 #define NODES_PGSHIFT		(NODES_PGOFF * (NODES_WIDTH != 0))
 #define ZONES_PGSHIFT		(ZONES_PGOFF * (ZONES_WIDTH != 0))
-#define LAST_NIDPID_PGSHIFT	(LAST_NIDPID_PGOFF * (LAST_NIDPID_WIDTH != 0))
+#define LAST_CPUPID_PGSHIFT	(LAST_CPUPID_PGOFF * (LAST_CPUPID_WIDTH != 0))
 
 /* NODE:ZONE or SECTION:ZONE is used to ID a zone for the buddy allocator */
 #ifdef NODE_NOT_IN_PAGE_FLAGS
@@ -624,7 +624,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
 #define ZONES_MASK		((1UL << ZONES_WIDTH) - 1)
 #define NODES_MASK		((1UL << NODES_WIDTH) - 1)
 #define SECTIONS_MASK		((1UL << SECTIONS_WIDTH) - 1)
-#define LAST_NIDPID_MASK	((1UL << LAST_NIDPID_WIDTH) - 1)
+#define LAST_CPUPID_MASK	((1UL << LAST_CPUPID_WIDTH) - 1)
 #define ZONEID_MASK		((1UL << ZONEID_SHIFT) - 1)
 
 static inline enum zone_type page_zonenum(const struct page *page)
@@ -668,96 +668,106 @@ static inline int page_to_nid(const struct page *page)
 #endif
 
 #ifdef CONFIG_NUMA_BALANCING
-static inline int nid_pid_to_nidpid(int nid, int pid)
+static inline int cpu_pid_to_cpupid(int cpu, int pid)
 {
-	return ((nid & LAST__NID_MASK) << LAST__PID_SHIFT) | (pid & LAST__PID_MASK);
+	return ((cpu & LAST__CPU_MASK) << LAST__PID_SHIFT) | (pid & LAST__PID_MASK);
 }
 
-static inline int nidpid_to_pid(int nidpid)
+static inline int cpupid_to_pid(int cpupid)
 {
-	return nidpid & LAST__PID_MASK;
+	return cpupid & LAST__PID_MASK;
 }
 
-static inline int nidpid_to_nid(int nidpid)
+static inline int cpupid_to_cpu(int cpupid)
 {
-	return (nidpid >> LAST__PID_SHIFT) & LAST__NID_MASK;
+	return (cpupid >> LAST__PID_SHIFT) & LAST__CPU_MASK;
 }
 
-static inline bool nidpid_pid_unset(int nidpid)
+static inline int cpupid_to_nid(int cpupid)
 {
-	return nidpid_to_pid(nidpid) == (-1 & LAST__PID_MASK);
+	return cpu_to_node(cpupid_to_cpu(cpupid));
 }
 
-static inline bool nidpid_nid_unset(int nidpid)
+static inline bool cpupid_pid_unset(int cpupid)
 {
-	return nidpid_to_nid(nidpid) == (-1 & LAST__NID_MASK);
+	return cpupid_to_pid(cpupid) == (-1 & LAST__PID_MASK);
 }
 
-#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS
-static inline int page_nidpid_xchg_last(struct page *page, int nid)
+static inline bool cpupid_cpu_unset(int cpupid)
 {
-	return xchg(&page->_last_nidpid, nid);
+	return cpupid_to_cpu(cpupid) == (-1 & LAST__CPU_MASK);
 }
 
-static inline int page_nidpid_last(struct page *page)
+#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
+static inline int page_cpupid_xchg_last(struct page *page, int cpupid)
 {
-	return page->_last_nidpid;
+	return xchg(&page->_last_cpupid, cpupid);
 }
-static inline void page_nidpid_reset_last(struct page *page)
+
+static inline int page_cpupid_last(struct page *page)
+{
+	return page->_last_cpupid;
+}
+static inline void page_cpupid_reset_last(struct page *page)
 {
-	page->_last_nidpid = -1;
+	page->_last_cpupid = -1;
 }
 #else
-static inline int page_nidpid_last(struct page *page)
+static inline int page_cpupid_last(struct page *page)
 {
-	return (page->flags >> LAST_NIDPID_PGSHIFT) & LAST_NIDPID_MASK;
+	return (page->flags >> LAST_CPUPID_PGSHIFT) & LAST_CPUPID_MASK;
 }
 
-extern int page_nidpid_xchg_last(struct page *page, int nidpid);
+extern int page_cpupid_xchg_last(struct page *page, int cpupid);
 
-static inline void page_nidpid_reset_last(struct page *page)
+static inline void page_cpupid_reset_last(struct page *page)
 {
-	int nidpid = (1 << LAST_NIDPID_SHIFT) - 1;
+	int cpupid = (1 << LAST_CPUPID_SHIFT) - 1;
 
-	page->flags &= ~(LAST_NIDPID_MASK << LAST_NIDPID_PGSHIFT);
-	page->flags |= (nidpid & LAST_NIDPID_MASK) << LAST_NIDPID_PGSHIFT;
+	page->flags &= ~(LAST_CPUPID_MASK << LAST_CPUPID_PGSHIFT);
+	page->flags |= (cpupid & LAST_CPUPID_MASK) << LAST_CPUPID_PGSHIFT;
 }
-#endif /* LAST_NIDPID_NOT_IN_PAGE_FLAGS */
-#else
-static inline int page_nidpid_xchg_last(struct page *page, int nidpid)
+#endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */
+#else /* !CONFIG_NUMA_BALANCING */
+static inline int page_cpupid_xchg_last(struct page *page, int cpupid)
 {
-	return page_to_nid(page);
+	return page_to_nid(page); /* XXX */
 }
 
-static inline int page_nidpid_last(struct page *page)
+static inline int page_cpupid_last(struct page *page)
 {
-	return page_to_nid(page);
+	return page_to_nid(page); /* XXX */
 }
 
-static inline int nidpid_to_nid(int nidpid)
+static inline int cpupid_to_nid(int cpupid)
 {
 	return -1;
 }
 
-static inline int nidpid_to_pid(int nidpid)
+static inline int cpupid_to_pid(int cpupid)
 {
 	return -1;
 }
 
-static inline int nid_pid_to_nidpid(int nid, int pid)
+static inline int cpupid_to_cpu(int cpupid)
 {
 	return -1;
 }
 
-static inline bool nidpid_pid_unset(int nidpid)
+static inline int cpu_pid_to_cpupid(int nid, int pid)
+{
+	return -1;
+}
+
+static inline bool cpupid_pid_unset(int cpupid)
 {
 	return 1;
 }
 
-static inline void page_nidpid_reset_last(struct page *page)
+static inline void page_cpupid_reset_last(struct page *page)
 {
 }
-#endif
+#endif /* CONFIG_NUMA_BALANCING */
 
 static inline struct zone *page_zone(const struct page *page)
 {
include/linux/mm_types.h +2 -2
index f46378e..b0370cd 100644
@@ -174,8 +174,8 @@ struct page {
 	void *shadow;
 #endif
 
-#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS
-	int _last_nidpid;
+#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
+	int _last_cpupid;
 #endif
 }
 /*
include/linux/page-flags-layout.h +11 -11
index 02bc918..da52366 100644
@@ -39,9 +39,9 @@
  * lookup is necessary.
  *
  * No sparsemem or sparsemem vmemmap: |       NODE     | ZONE |             ... | FLAGS |
- *      " plus space for last_nidpid: |       NODE     | ZONE | LAST_NIDPID ... | FLAGS |
+ *      " plus space for last_cpupid: |       NODE     | ZONE | LAST_CPUPID ... | FLAGS |
  * classic sparse with space for node:| SECTION | NODE | ZONE |             ... | FLAGS |
- *      " plus space for last_nidpid: | SECTION | NODE | ZONE | LAST_NIDPID ... | FLAGS |
+ *      " plus space for last_cpupid: | SECTION | NODE | ZONE | LAST_CPUPID ... | FLAGS |
  * classic sparse no space for node:  | SECTION |     ZONE    | ... | FLAGS |
  */
 #if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP)
@@ -65,18 +65,18 @@
 #define LAST__PID_SHIFT 8
 #define LAST__PID_MASK  ((1 << LAST__PID_SHIFT)-1)
 
-#define LAST__NID_SHIFT NODES_SHIFT
-#define LAST__NID_MASK  ((1 << LAST__NID_SHIFT)-1)
+#define LAST__CPU_SHIFT NR_CPUS_BITS
+#define LAST__CPU_MASK  ((1 << LAST__CPU_SHIFT)-1)
 
-#define LAST_NIDPID_SHIFT (LAST__PID_SHIFT+LAST__NID_SHIFT)
+#define LAST_CPUPID_SHIFT (LAST__PID_SHIFT+LAST__CPU_SHIFT)
 #else
-#define LAST_NIDPID_SHIFT 0
+#define LAST_CPUPID_SHIFT 0
 #endif
 
-#if SECTIONS_WIDTH+ZONES_WIDTH+NODES_SHIFT+LAST_NIDPID_SHIFT <= BITS_PER_LONG - NR_PAGEFLAGS
-#define LAST_NIDPID_WIDTH LAST_NIDPID_SHIFT
+#if SECTIONS_WIDTH+ZONES_WIDTH+NODES_SHIFT+LAST_CPUPID_SHIFT <= BITS_PER_LONG - NR_PAGEFLAGS
+#define LAST_CPUPID_WIDTH LAST_CPUPID_SHIFT
 #else
-#define LAST_NIDPID_WIDTH 0
+#define LAST_CPUPID_WIDTH 0
 #endif
 
 /*
@@ -87,8 +87,8 @@
 #define NODE_NOT_IN_PAGE_FLAGS
 #endif
 
-#if defined(CONFIG_NUMA_BALANCING) && LAST_NIDPID_WIDTH == 0
-#define LAST_NIDPID_NOT_IN_PAGE_FLAGS
+#if defined(CONFIG_NUMA_BALANCING) && LAST_CPUPID_WIDTH == 0
+#define LAST_CPUPID_NOT_IN_PAGE_FLAGS
 #endif
 
 #endif /* _LINUX_PAGE_FLAGS_LAYOUT */
kernel/bounds.c +4 -0
index 0c9b862..e8ca97b 100644
@@ -10,6 +10,7 @@
 #include <linux/mmzone.h>
 #include <linux/kbuild.h>
 #include <linux/page_cgroup.h>
+#include <linux/log2.h>
 
 void foo(void)
 {
@@ -17,5 +18,8 @@ void foo(void)
 	DEFINE(NR_PAGEFLAGS, __NR_PAGEFLAGS);
 	DEFINE(MAX_NR_ZONES, __MAX_NR_ZONES);
 	DEFINE(NR_PCG_FLAGS, __NR_PCG_FLAGS);
+#ifdef CONFIG_SMP
+	DEFINE(NR_CPUS_BITS, ilog2(CONFIG_NR_CPUS));
+#endif
 	/* End of constants */
 }
kernel/sched/fair.c +3 -3
index 8ebed0a..7408951 100644
@@ -1210,7 +1210,7 @@ static void task_numa_placement(struct task_struct *p)
 /*
  * Got a PROT_NONE fault for a page on @node.
  */
-void task_numa_fault(int last_nidpid, int node, int pages, bool migrated)
+void task_numa_fault(int last_cpupid, int node, int pages, bool migrated)
 {
 	struct task_struct *p = current;
 	int priv;
@@ -1226,8 +1226,8 @@ void task_numa_fault(int last_nidpid, int node, int pages, bool migrated)
 	 * First accesses are treated as private, otherwise consider accesses
 	 * to be private if the accessing pid has not changed
 	 */
-	if (!nidpid_pid_unset(last_nidpid))
-		priv = ((p->pid & LAST__PID_MASK) == nidpid_to_pid(last_nidpid));
+	if (!cpupid_pid_unset(last_cpupid))
+		priv = ((p->pid & LAST__PID_MASK) == cpupid_to_pid(last_cpupid));
 	else
 		priv = 1;
 
mm/huge_memory.c +4 -4
index 0e73685..048d4b2 100644
@@ -1294,7 +1294,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	struct page *page;
 	unsigned long haddr = addr & HPAGE_PMD_MASK;
 	int page_nid = -1, this_nid = numa_node_id();
-	int target_nid, last_nidpid = -1;
+	int target_nid, last_cpupid = -1;
 	bool page_locked;
 	bool migrated = false;
 
@@ -1305,7 +1305,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	page = pmd_page(pmd);
 	BUG_ON(is_huge_zero_page(page));
 	page_nid = page_to_nid(page);
-	last_nidpid = page_nidpid_last(page);
+	last_cpupid = page_cpupid_last(page);
 	count_vm_numa_event(NUMA_HINT_FAULTS);
 	if (page_nid == this_nid)
 		count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL);
@@ -1374,7 +1374,7 @@ out:
 		page_unlock_anon_vma_read(anon_vma);
 
 	if (page_nid != -1)
-		task_numa_fault(last_nidpid, page_nid, HPAGE_PMD_NR, migrated);
+		task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR, migrated);
 
 	return 0;
 }
@@ -1694,7 +1694,7 @@ static void __split_huge_page_refcount(struct page *page,
 		page_tail->mapping = page->mapping;
 
 		page_tail->index = page->index + i;
-		page_nidpid_xchg_last(page_tail, page_nidpid_last(page));
+		page_cpupid_xchg_last(page_tail, page_cpupid_last(page));
 
 		BUG_ON(!PageAnon(page_tail));
 		BUG_ON(!PageUptodate(page_tail));
mm/memory.c +8 -8
index 948ec32..6b558a5 100644
@@ -69,8 +69,8 @@
 
 #include "internal.h"
 
-#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS
-#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_nidpid.
+#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
+#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_cpupid.
 #endif
 
 #ifndef CONFIG_NEED_MULTIPLE_NODES
@@ -3547,7 +3547,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	struct page *page = NULL;
 	spinlock_t *ptl;
 	int page_nid = -1;
-	int last_nidpid;
+	int last_cpupid;
 	int target_nid;
 	bool migrated = false;
 
@@ -3578,7 +3578,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	}
 	BUG_ON(is_zero_pfn(page_to_pfn(page)));
 
-	last_nidpid = page_nidpid_last(page);
+	last_cpupid = page_cpupid_last(page);
 	page_nid = page_to_nid(page);
 	target_nid = numa_migrate_prep(page, vma, addr, page_nid);
 	pte_unmap_unlock(ptep, ptl);
@@ -3594,7 +3594,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 
 out:
 	if (page_nid != -1)
-		task_numa_fault(last_nidpid, page_nid, 1, migrated);
+		task_numa_fault(last_cpupid, page_nid, 1, migrated);
 	return 0;
 }
 
@@ -3609,7 +3609,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	unsigned long offset;
 	spinlock_t *ptl;
 	bool numa = false;
-	int last_nidpid;
+	int last_cpupid;
 
 	spin_lock(&mm->page_table_lock);
 	pmd = *pmdp;
@@ -3654,7 +3654,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		if (unlikely(!page))
 			continue;
 
-		last_nidpid = page_nidpid_last(page);
+		last_cpupid = page_cpupid_last(page);
 		page_nid = page_to_nid(page);
 		target_nid = numa_migrate_prep(page, vma, addr, page_nid);
 		pte_unmap_unlock(pte, ptl);
@@ -3667,7 +3667,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		}
 
 		if (page_nid != -1)
-			task_numa_fault(last_nidpid, page_nid, 1, migrated);
+			task_numa_fault(last_cpupid, page_nid, 1, migrated);
 
 		pte = pte_offset_map_lock(mm, pmdp, addr, &ptl);
 	}
mm/mempolicy.c +9 -7
index 54004a8..07db309 100644
@@ -2268,6 +2268,8 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long
 	struct zone *zone;
 	int curnid = page_to_nid(page);
 	unsigned long pgoff;
+	int thiscpu = raw_smp_processor_id();
+	int thisnid = cpu_to_node(thiscpu);
 	int polnid = -1;
 	int ret = -1;
 
@@ -2316,11 +2318,11 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long
 
 	/* Migrate the page towards the node whose CPU is referencing it */
 	if (pol->flags & MPOL_F_MORON) {
-		int last_nidpid;
-		int this_nidpid;
+		int last_cpupid;
+		int this_cpupid;
 
-		polnid = numa_node_id();
-		this_nidpid = nid_pid_to_nidpid(polnid, current->pid);
+		polnid = thisnid;
+		this_cpupid = cpu_pid_to_cpupid(thiscpu, current->pid);
 
 		/*
 		 * Multi-stage node selection is used in conjunction
@@ -2343,8 +2345,8 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long
 		 * it less likely we act on an unlikely task<->page
 		 * relation.
 		 */
-		last_nidpid = page_nidpid_xchg_last(page, this_nidpid);
-		if (!nidpid_pid_unset(last_nidpid) && nidpid_to_nid(last_nidpid) != polnid)
+		last_cpupid = page_cpupid_xchg_last(page, this_cpupid);
+		if (!cpupid_pid_unset(last_cpupid) && cpupid_to_nid(last_cpupid) != thisnid)
 			goto out;
 
 #ifdef CONFIG_NUMA_BALANCING
@@ -2354,7 +2356,7 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long
 		 * This way a short and temporary process migration will
 		 * not cause excessive memory migration.
 		 */
-		if (polnid != current->numa_preferred_nid &&
+		if (thisnid != current->numa_preferred_nid &&
 				!current->numa_migrate_seq)
 			goto out;
 #endif
mm/migrate.c +2 -2
index cea719d..6ff845f 100644
@@ -1495,7 +1495,7 @@ static struct page *alloc_misplaced_dst_page(struct page *page,
 					  __GFP_NOWARN) &
 					 ~GFP_IOFS, 0);
 	if (newpage)
-		page_nidpid_xchg_last(newpage, page_nidpid_last(page));
+		page_cpupid_xchg_last(newpage, page_cpupid_last(page));
 
 	return newpage;
 }
@@ -1672,7 +1672,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
 	if (!new_page)
 		goto out_fail;
 
-	page_nidpid_xchg_last(new_page, page_nidpid_last(page));
+	page_cpupid_xchg_last(new_page, page_cpupid_last(page));
 
 	isolated = numamigrate_isolate_page(pgdat, page);
 	if (!isolated) {
mm/mm_init.c +9 -9
index 467de57..68562e9 100644
@@ -71,26 +71,26 @@ void __init mminit_verify_pageflags_layout(void)
 	unsigned long or_mask, add_mask;
 
 	shift = 8 * sizeof(unsigned long);
-	width = shift - SECTIONS_WIDTH - NODES_WIDTH - ZONES_WIDTH - LAST_NIDPID_SHIFT;
+	width = shift - SECTIONS_WIDTH - NODES_WIDTH - ZONES_WIDTH - LAST_CPUPID_SHIFT;
 	mminit_dprintk(MMINIT_TRACE, "pageflags_layout_widths",
-		"Section %d Node %d Zone %d Lastnidpid %d Flags %d\n",
+		"Section %d Node %d Zone %d Lastcpupid %d Flags %d\n",
 		SECTIONS_WIDTH,
 		NODES_WIDTH,
 		ZONES_WIDTH,
-		LAST_NIDPID_WIDTH,
+		LAST_CPUPID_WIDTH,
 		NR_PAGEFLAGS);
 	mminit_dprintk(MMINIT_TRACE, "pageflags_layout_shifts",
-		"Section %d Node %d Zone %d Lastnidpid %d\n",
+		"Section %d Node %d Zone %d Lastcpupid %d\n",
 		SECTIONS_SHIFT,
 		NODES_SHIFT,
 		ZONES_SHIFT,
-		LAST_NIDPID_SHIFT);
+		LAST_CPUPID_SHIFT);
 	mminit_dprintk(MMINIT_TRACE, "pageflags_layout_pgshifts",
-		"Section %lu Node %lu Zone %lu Lastnidpid %lu\n",
+		"Section %lu Node %lu Zone %lu Lastcpupid %lu\n",
 		(unsigned long)SECTIONS_PGSHIFT,
 		(unsigned long)NODES_PGSHIFT,
 		(unsigned long)ZONES_PGSHIFT,
-		(unsigned long)LAST_NIDPID_PGSHIFT);
+		(unsigned long)LAST_CPUPID_PGSHIFT);
 	mminit_dprintk(MMINIT_TRACE, "pageflags_layout_nodezoneid",
 		"Node/Zone ID: %lu -> %lu\n",
 		(unsigned long)(ZONEID_PGOFF + ZONEID_SHIFT),
@@ -102,9 +102,9 @@ void __init mminit_verify_pageflags_layout(void)
 	mminit_dprintk(MMINIT_TRACE, "pageflags_layout_nodeflags",
 		"Node not in page flags");
 #endif
-#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS
+#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
 	mminit_dprintk(MMINIT_TRACE, "pageflags_layout_nodeflags",
-		"Last nidpid not in page flags");
+		"Last cpupid not in page flags");
 #endif
 
 	if (SECTIONS_WIDTH) {
mm/mmzone.c +7 -7
index 25bb477..2c70c3a 100644
@@ -97,20 +97,20 @@ void lruvec_init(struct lruvec *lruvec)
 		INIT_LIST_HEAD(&lruvec->lists[lru]);
 }
 
-#if defined(CONFIG_NUMA_BALANCING) && !defined(LAST_NIDPID_NOT_IN_PAGE_FLAGS)
-int page_nidpid_xchg_last(struct page *page, int nidpid)
+#if defined(CONFIG_NUMA_BALANCING) && !defined(LAST_CPUPID_IN_PAGE_FLAGS)
+int page_cpupid_xchg_last(struct page *page, int cpupid)
 {
 	unsigned long old_flags, flags;
-	int last_nidpid;
+	int last_cpupid;
 
 	do {
 		old_flags = flags = page->flags;
-		last_nidpid = page_nidpid_last(page);
+		last_cpupid = page_cpupid_last(page);
 
-		flags &= ~(LAST_NIDPID_MASK << LAST_NIDPID_PGSHIFT);
-		flags |= (nidpid & LAST_NIDPID_MASK) << LAST_NIDPID_PGSHIFT;
+		flags &= ~(LAST_CPUPID_MASK << LAST_CPUPID_PGSHIFT);
+		flags |= (cpupid & LAST_CPUPID_MASK) << LAST_CPUPID_PGSHIFT;
 	} while (unlikely(cmpxchg(&page->flags, old_flags, flags) != old_flags));
 
-	return last_nidpid;
+	return last_cpupid;
 }
 #endif
mm/mprotect.c +14 -14
index 5aae390..9a74855 100644
@@ -37,14 +37,14 @@ static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot)
 
 static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
 		unsigned long addr, unsigned long end, pgprot_t newprot,
-		int dirty_accountable, int prot_numa, bool *ret_all_same_nidpid)
+		int dirty_accountable, int prot_numa, bool *ret_all_same_cpupid)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	pte_t *pte, oldpte;
 	spinlock_t *ptl;
 	unsigned long pages = 0;
-	bool all_same_nidpid = true;
-	int last_nid = -1;
+	bool all_same_cpupid = true;
+	int last_cpu = -1;
 	int last_pid = -1;
 
 	pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
@@ -64,17 +64,17 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
 
 				page = vm_normal_page(vma, addr, oldpte);
 				if (page) {
-					int nidpid = page_nidpid_last(page);
-					int this_nid = nidpid_to_nid(nidpid);
-					int this_pid = nidpid_to_pid(nidpid);
+					int cpupid = page_cpupid_last(page);
+					int this_cpu = cpupid_to_cpu(cpupid);
+					int this_pid = cpupid_to_pid(cpupid);
 
-					if (last_nid == -1)
-						last_nid = this_nid;
+					if (last_cpu == -1)
+						last_cpu = this_cpu;
 					if (last_pid == -1)
 						last_pid = this_pid;
-					if (last_nid != this_nid ||
+					if (last_cpu != this_cpu ||
 					    last_pid != this_pid) {
-						all_same_nidpid = false;
+						all_same_cpupid = false;
 					}
 
 					if (!pte_numa(oldpte)) {
@@ -115,7 +115,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
 	arch_leave_lazy_mmu_mode();
 	pte_unmap_unlock(pte - 1, ptl);
 
-	*ret_all_same_nidpid = all_same_nidpid;
+	*ret_all_same_cpupid = all_same_cpupid;
 	return pages;
 }
 
@@ -142,7 +142,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,
 	pmd_t *pmd;
 	unsigned long next;
 	unsigned long pages = 0;
-	bool all_same_nidpid;
+	bool all_same_cpupid;
 
 	pmd = pmd_offset(pud, addr);
 	do {
@@ -168,7 +168,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,
 		if (pmd_none_or_clear_bad(pmd))
 			continue;
 		this_pages = change_pte_range(vma, pmd, addr, next, newprot,
-				 dirty_accountable, prot_numa, &all_same_nidpid);
+				 dirty_accountable, prot_numa, &all_same_cpupid);
 		pages += this_pages;
 
 		/*
@@ -177,7 +177,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,
 		 * node. This allows a regular PMD to be handled as one fault
 		 * and effectively batches the taking of the PTL
 		 */
-		if (prot_numa && this_pages && all_same_nidpid)
+		if (prot_numa && this_pages && all_same_cpupid)
 			change_pmd_protnuma(vma->vm_mm, addr, pmd);
 	} while (pmd++, addr = next, addr != end);
 
mm/page_alloc.c +2 -2
index 7bf960e..4b6c4e8 100644
@@ -622,7 +622,7 @@ static inline int free_pages_check(struct page *page)
 		bad_page(page);
 		return 1;
 	}
-	page_nidpid_reset_last(page);
+	page_cpupid_reset_last(page);
 	if (page->flags & PAGE_FLAGS_CHECK_AT_PREP)
 		page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP;
 	return 0;
@@ -3944,7 +3944,7 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,
 		mminit_verify_page_links(page, zone, nid, pfn);
 		init_page_count(page);
 		page_mapcount_reset(page);
-		page_nidpid_reset_last(page);
+		page_cpupid_reset_last(page);
 		SetPageReserved(page);
 		/*
 		 * Mark the block movable so that blocks are reserved for
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#47 lore raw

PATCH 43/63 sched: numa: Use {cpu, pid} to create task groups for shared faults

In reply to Mel Gorman

From: Peter Zijlstra <[email protected]>

While parallel applications tend to align their data on the cache boundary, they tend not to align on the page or THP boundary. Consequently tasks that partition their data can still "false-share" pages presenting a problem for optimal NUMA placement.

This patch uses NUMA hinting faults to chain tasks together into numa_groups. As well as storing the NID a task was running on when accessing a page a truncated representation of the faulting PID is stored. If subsequent faults are from different PIDs it is reasonable to assume that those two tasks share a page and are candidates for being grouped together. Note that this patch makes no scheduling decisions based on the grouping information.

---
 include/linux/mm.h    |  11 ++++
 include/linux/sched.h |   3 +
 kernel/sched/core.c   |   3 +
 kernel/sched/fair.c   | 165 +++++++++++++++++++++++++++++++++++++++++++++++---
 kernel/sched/sched.h  |   5 +-
 mm/memory.c           |   8 +++
 6 files changed, 182 insertions(+), 13 deletions(-)
include/linux/mm.h +11 -0
index 61dc023..505c3fc 100644
@@ -698,6 +698,12 @@ static inline bool cpupid_cpu_unset(int cpupid)
 	return cpupid_to_cpu(cpupid) == (-1 & LAST__CPU_MASK);
 }
 
+static inline bool __cpupid_match_pid(pid_t task_pid, int cpupid)
+{
+	return (task_pid & LAST__PID_MASK) == cpupid_to_pid(cpupid);
+}
+
+#define cpupid_match_pid(task, cpupid) __cpupid_match_pid(task->pid, cpupid)
 #ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
 static inline int page_cpupid_xchg_last(struct page *page, int cpupid)
 {
@@ -767,6 +773,11 @@ static inline bool cpupid_pid_unset(int cpupid)
 static inline void page_cpupid_reset_last(struct page *page)
 {
 }
+
+static inline bool cpupid_match_pid(struct task_struct *task, int cpupid)
+{
+	return false;
+}
 #endif /* CONFIG_NUMA_BALANCING */
 
 static inline struct zone *page_zone(const struct page *page)
include/linux/sched.h +3 -0
index 3e8c547..ea057a2 100644
@@ -1338,6 +1338,9 @@ struct task_struct {
 	u64 node_stamp;			/* migration stamp  */
 	struct callback_head numa_work;
 
+	struct list_head numa_entry;
+	struct numa_group *numa_group;
+
 	/*
 	 * Exponential decaying average of faults on a per-node basis.
 	 * Scheduling placement decisions are made based on the these counts.
kernel/sched/core.c +3 -0
index f7a44dd..123ac92 100644
@@ -1740,6 +1740,9 @@ static void __sched_fork(struct task_struct *p)
 	p->numa_work.next = &p->numa_work;
 	p->numa_faults = NULL;
 	p->numa_faults_buffer = NULL;
+
+	INIT_LIST_HEAD(&p->numa_entry);
+	p->numa_group = NULL;
 #endif /* CONFIG_NUMA_BALANCING */
 }
 
kernel/sched/fair.c +156 -9

202 lines not on this page · raw message

kernel/sched/sched.h +1 -4
index 82ff03a..e0875dc 100644
@@ -557,10 +557,7 @@ static inline u64 rq_clock_task(struct rq *rq)
 #ifdef CONFIG_NUMA_BALANCING
 extern int migrate_task_to(struct task_struct *p, int cpu);
 extern int migrate_swap(struct task_struct *, struct task_struct *);
-static inline void task_numa_free(struct task_struct *p)
-{
-	kfree(p->numa_faults);
-}
+extern void task_numa_free(struct task_struct *p);
 #else /* CONFIG_NUMA_BALANCING */
 static inline void task_numa_free(struct task_struct *p)
 {
mm/memory.c +8 -0
index 6b558a5..f779403 100644
@@ -2730,6 +2730,14 @@ static int do_wp_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		get_page(dirty_page);
 
 reuse:
+		/*
+		 * Clear the pages cpupid information as the existing
+		 * information potentially belongs to a now completely
+		 * unrelated process.
+		 */
+		if (old_page)
+			page_cpupid_xchg_last(old_page, (1 << LAST_CPUPID_SHIFT) - 1);
+
 		flush_cache_page(vma, address, pte_pfn(orig_pte));
 		entry = pte_mkyoung(orig_pte);
 		entry = maybe_mkwrite(pte_mkdirty(entry), vma);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#48 lore raw

PATCH 44/63 sched: numa: Report a NUMA task group ID

In reply to Mel Gorman

It is desirable to model from userspace how the scheduler groups tasks over time. This patch adds an ID to the numa_group and reports it via /proc/PID/status.

---
 fs/proc/array.c       | 2 ++
 include/linux/sched.h | 5 +++++
 kernel/sched/fair.c   | 7 +++++++
 3 files changed, 14 insertions(+)
fs/proc/array.c +2 -0
index cbd0f1b..1bd2077 100644
@@ -183,6 +183,7 @@ static inline void task_state(struct seq_file *m, struct pid_namespace *ns,
 	seq_printf(m,
 		"State:\t%s\n"
 		"Tgid:\t%d\n"
+		"Ngid:\t%d\n"
 		"Pid:\t%d\n"
 		"PPid:\t%d\n"
 		"TracerPid:\t%d\n"
@@ -190,6 +191,7 @@ static inline void task_state(struct seq_file *m, struct pid_namespace *ns,
 		"Gid:\t%d\t%d\t%d\t%d\n",
 		get_task_state(p),
 		task_tgid_nr_ns(p, ns),
+		task_numa_group_id(p),
 		pid_nr_ns(pid, ns),
 		ppid, tpid,
 		from_kuid_munged(user_ns, cred->uid),
include/linux/sched.h +5 -0
index ea057a2..4fad1f17 100644
@@ -1436,12 +1436,17 @@ struct task_struct {
 
 #ifdef CONFIG_NUMA_BALANCING
 extern void task_numa_fault(int last_node, int node, int pages, bool migrated);
+extern pid_t task_numa_group_id(struct task_struct *p);
 extern void set_numabalancing_state(bool enabled);
 #else
 static inline void task_numa_fault(int last_node, int node, int pages,
 				   bool migrated)
 {
 }
+static inline pid_t task_numa_group_id(struct task_struct *p)
+{
+	return 0;
+}
 static inline void set_numabalancing_state(bool enabled)
 {
 }
kernel/sched/fair.c +7 -0
index a5673c6..abdbb7c 100644
@@ -893,12 +893,18 @@ struct numa_group {
 
 	spinlock_t lock; /* nr_tasks, tasks */
 	int nr_tasks;
+	pid_t gid;
 	struct list_head task_list;
 
 	struct rcu_head rcu;
 	atomic_long_t faults[0];
 };
 
+pid_t task_numa_group_id(struct task_struct *p)
+{
+	return p->numa_group ? p->numa_group->gid : 0;
+}
+
 static inline int task_faults_idx(int nid, int priv)
 {
 	return 2 * nid + priv;
@@ -1265,6 +1271,7 @@ static void task_numa_group(struct task_struct *p, int cpupid)
 		atomic_set(&grp->refcount, 1);
 		spin_lock_init(&grp->lock);
 		INIT_LIST_HEAD(&grp->task_list);
+		grp->gid = p->pid;
 
 		for (i = 0; i < 2*nr_node_ids; i++)
 			atomic_long_set(&grp->faults[i], p->numa_faults[i]);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#49 lore raw

PATCH 45/63 mm: numa: copy cpupid on page migration

In reply to Mel Gorman

From: Rik van Riel <[email protected]>

After page migration, the new page has the nidpid unset. This makes every fault on a recently migrated page look like a first numa fault, leading to another page migration.

Copying over the nidpid at page migration time should prevent erroneous migrations of recently migrated pages.

---
 mm/migrate.c | 9 +++++++++
 1 file changed, 9 insertions(+)
mm/migrate.c +9 -0
index 6ff845f..1e1dbc9 100644
@@ -439,6 +439,8 @@ int migrate_huge_page_move_mapping(struct address_space *mapping,
  */
 void migrate_page_copy(struct page *newpage, struct page *page)
 {
+	int cpupid;
+
 	if (PageHuge(page) || PageTransHuge(page))
 		copy_huge_page(newpage, page);
 	else
@@ -475,6 +477,13 @@ void migrate_page_copy(struct page *newpage, struct page *page)
 			__set_page_dirty_nobuffers(newpage);
  	}
 
+	/*
+	 * Copy NUMA information to the new page, to prevent over-eager
+	 * future migrations of this same page.
+	 */
+	cpupid = page_cpupid_xchg_last(page, -1);
+	page_cpupid_xchg_last(newpage, cpupid);
+
 	mlock_migrate_page(newpage, page);
 	ksm_migrate_page(newpage, page);
 	/*
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#50 lore raw

PATCH 46/63 mm: numa: Do not group on RO pages

In reply to Mel Gorman

From: Peter Zijlstra <[email protected]>

And here's a little something to make sure not the whole world ends up in a single group.

As while we don't migrate shared executable pages, we do scan/fault on them. And since everybody links to libc, everybody ends up in the same group.

---
 include/linux/sched.h |  7 +++++--
 kernel/sched/fair.c   |  5 +++--
 mm/huge_memory.c      | 15 +++++++++++++--
 mm/memory.c           | 30 ++++++++++++++++++++++++++----
 4 files changed, 47 insertions(+), 10 deletions(-)
include/linux/sched.h +5 -2
index 4fad1f17..15888f5 100644
@@ -1434,13 +1434,16 @@ struct task_struct {
 /* Future-safe accessor for struct task_struct's cpus_allowed. */
 #define tsk_cpus_allowed(tsk) (&(tsk)->cpus_allowed)
 
+#define TNF_MIGRATED	0x01
+#define TNF_NO_GROUP	0x02
+
 #ifdef CONFIG_NUMA_BALANCING
-extern void task_numa_fault(int last_node, int node, int pages, bool migrated);
+extern void task_numa_fault(int last_node, int node, int pages, int flags);
 extern pid_t task_numa_group_id(struct task_struct *p);
 extern void set_numabalancing_state(bool enabled);
 #else
 static inline void task_numa_fault(int last_node, int node, int pages,
-				   bool migrated)
+				   int flags)
 {
 }
 static inline pid_t task_numa_group_id(struct task_struct *p)
kernel/sched/fair.c +3 -2
index abdbb7c..896cf87 100644
@@ -1361,9 +1361,10 @@ void task_numa_free(struct task_struct *p)
 /*
  * Got a PROT_NONE fault for a page on @node.
  */
-void task_numa_fault(int last_cpupid, int node, int pages, bool migrated)
+void task_numa_fault(int last_cpupid, int node, int pages, int flags)
 {
 	struct task_struct *p = current;
+	bool migrated = flags & TNF_MIGRATED;
 	int priv;
 
 	if (!numabalancing_enabled)
@@ -1394,7 +1395,7 @@ void task_numa_fault(int last_cpupid, int node, int pages, bool migrated)
 		priv = 1;
 	} else {
 		priv = cpupid_match_pid(p, last_cpupid);
-		if (!priv)
+		if (!priv && !(flags & TNF_NO_GROUP))
 			task_numa_group(p, last_cpupid);
 	}
 
mm/huge_memory.c +13 -2
index 048d4b2..aaf46dc 100644
@@ -1297,6 +1297,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	int target_nid, last_cpupid = -1;
 	bool page_locked;
 	bool migrated = false;
+	int flags = 0;
 
 	spin_lock(&mm->page_table_lock);
 	if (unlikely(!pmd_same(pmd, *pmdp)))
@@ -1311,6 +1312,14 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL);
 
 	/*
+	 * Avoid grouping on DSO/COW pages in specific and RO pages
+	 * in general, RO pages shouldn't hurt as much anyway since
+	 * they can be in shared cache state.
+	 */
+	if (!pmd_write(pmd))
+		flags |= TNF_NO_GROUP;
+
+	/*
 	 * Acquire the page lock to serialise THP migrations but avoid dropping
 	 * page_table_lock if at all possible
 	 */
@@ -1355,8 +1364,10 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	spin_unlock(&mm->page_table_lock);
 	migrated = migrate_misplaced_transhuge_page(mm, vma,
 				pmdp, pmd, addr, page, target_nid);
-	if (migrated)
+	if (migrated) {
+		flags |= TNF_MIGRATED;
 		page_nid = target_nid;
+	}
 
 	goto out;
 clear_pmdnuma:
@@ -1374,7 +1385,7 @@ out:
 		page_unlock_anon_vma_read(anon_vma);
 
 	if (page_nid != -1)
-		task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR, migrated);
+		task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR, flags);
 
 	return 0;
 }
mm/memory.c +26 -4
index f779403..1aa4187 100644
@@ -3558,6 +3558,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	int last_cpupid;
 	int target_nid;
 	bool migrated = false;
+	int flags = 0;
 
 	/*
 	* The "pte" at this point cannot be used safely without
@@ -3586,6 +3587,14 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	}
 	BUG_ON(is_zero_pfn(page_to_pfn(page)));
 
+	/*
+	 * Avoid grouping on DSO/COW pages in specific and RO pages
+	 * in general, RO pages shouldn't hurt as much anyway since
+	 * they can be in shared cache state.
+	 */
+	if (!pte_write(pte))
+		flags |= TNF_NO_GROUP;
+
 	last_cpupid = page_cpupid_last(page);
 	page_nid = page_to_nid(page);
 	target_nid = numa_migrate_prep(page, vma, addr, page_nid);
@@ -3597,12 +3606,14 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 
 	/* Migrate to the requested node */
 	migrated = migrate_misplaced_page(page, vma, target_nid);
-	if (migrated)
+	if (migrated) {
 		page_nid = target_nid;
+		flags |= TNF_MIGRATED;
+	}
 
 out:
 	if (page_nid != -1)
-		task_numa_fault(last_cpupid, page_nid, 1, migrated);
+		task_numa_fault(last_cpupid, page_nid, 1, flags);
 	return 0;
 }
 
@@ -3643,6 +3654,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		int page_nid = -1;
 		int target_nid;
 		bool migrated = false;
+		int flags = 0;
 
 		if (!pte_present(pteval))
 			continue;
@@ -3662,20 +3674,30 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		if (unlikely(!page))
 			continue;
 
+		/*
+		 * Avoid grouping on DSO/COW pages in specific and RO pages
+		 * in general, RO pages shouldn't hurt as much anyway since
+		 * they can be in shared cache state.
+		 */
+		if (!pte_write(pteval))
+			flags |= TNF_NO_GROUP;
+
 		last_cpupid = page_cpupid_last(page);
 		page_nid = page_to_nid(page);
 		target_nid = numa_migrate_prep(page, vma, addr, page_nid);
 		pte_unmap_unlock(pte, ptl);
 		if (target_nid != -1) {
 			migrated = migrate_misplaced_page(page, vma, target_nid);
-			if (migrated)
+			if (migrated) {
 				page_nid = target_nid;
+				flags |= TNF_MIGRATED;
+			}
 		} else {
 			put_page(page);
 		}
 
 		if (page_nid != -1)
-			task_numa_fault(last_cpupid, page_nid, 1, migrated);
+			task_numa_fault(last_cpupid, page_nid, 1, flags);
 
 		pte = pte_offset_map_lock(mm, pmdp, addr, &ptl);
 	}
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#51 lore raw

PATCH 47/63 mm: numa: Do not batch handle PMD pages

In reply to Mel Gorman

With the THP migration races closed it is still possible to occasionally see corruption. The problem is related to handling PMD pages in batch. When a page fault is handled it can be assumed that the page being faulted will also be flushed from the TLB. The same flushing does not happen when handling PMD pages in batch. Fixing is straight forward but there are a number of reasons not to

1. Multiple TLB flushes may have to be sent depending on what pages get migrated
2. The handling of PMDs in batch means that faults get accounted to the task that is handling the fault. While care is taken to only mark PMDs where the last CPU and PID match it can still have problems due to PID truncation when matching PIDs.
3. Batching on the PMD level may reduce faults but setting pmd_numa requires taking a heavy lock that can contend with THP migration and handling the fault requires the release/acquisition of the PTL for every page migrated. It's still pretty heavy.

PMD batch handling is not something that people ever have been happy with. This patch removes it and later patches will deal with the additional fault overhead using more installigent migrate rate adaption.

---
 mm/memory.c   | 101 ++--------------------------------------------------------
 mm/mprotect.c |  47 ++-------------------------
 2 files changed, 4 insertions(+), 144 deletions(-)
mm/memory.c +2 -99
index 1aa4187..da65ad4 100644
@@ -3617,103 +3617,6 @@ out:
 	return 0;
 }
 
-/* NUMA hinting page fault entry point for regular pmds */
-#ifdef CONFIG_NUMA_BALANCING
-static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
-		     unsigned long addr, pmd_t *pmdp)
-{
-	pmd_t pmd;
-	pte_t *pte, *orig_pte;
-	unsigned long _addr = addr & PMD_MASK;
-	unsigned long offset;
-	spinlock_t *ptl;
-	bool numa = false;
-	int last_cpupid;
-
-	spin_lock(&mm->page_table_lock);
-	pmd = *pmdp;
-	if (pmd_numa(pmd)) {
-		set_pmd_at(mm, _addr, pmdp, pmd_mknonnuma(pmd));
-		numa = true;
-	}
-	spin_unlock(&mm->page_table_lock);
-
-	if (!numa)
-		return 0;
-
-	/* we're in a page fault so some vma must be in the range */
-	BUG_ON(!vma);
-	BUG_ON(vma->vm_start >= _addr + PMD_SIZE);
-	offset = max(_addr, vma->vm_start) & ~PMD_MASK;
-	VM_BUG_ON(offset >= PMD_SIZE);
-	orig_pte = pte = pte_offset_map_lock(mm, pmdp, _addr, &ptl);
-	pte += offset >> PAGE_SHIFT;
-	for (addr = _addr + offset; addr < _addr + PMD_SIZE; pte++, addr += PAGE_SIZE) {
-		pte_t pteval = *pte;
-		struct page *page;
-		int page_nid = -1;
-		int target_nid;
-		bool migrated = false;
-		int flags = 0;
-
-		if (!pte_present(pteval))
-			continue;
-		if (!pte_numa(pteval))
-			continue;
-		if (addr >= vma->vm_end) {
-			vma = find_vma(mm, addr);
-			/* there's a pte present so there must be a vma */
-			BUG_ON(!vma);
-			BUG_ON(addr < vma->vm_start);
-		}
-		if (pte_numa(pteval)) {
-			pteval = pte_mknonnuma(pteval);
-			set_pte_at(mm, addr, pte, pteval);
-		}
-		page = vm_normal_page(vma, addr, pteval);
-		if (unlikely(!page))
-			continue;
-
-		/*
-		 * Avoid grouping on DSO/COW pages in specific and RO pages
-		 * in general, RO pages shouldn't hurt as much anyway since
-		 * they can be in shared cache state.
-		 */
-		if (!pte_write(pteval))
-			flags |= TNF_NO_GROUP;
-
-		last_cpupid = page_cpupid_last(page);
-		page_nid = page_to_nid(page);
-		target_nid = numa_migrate_prep(page, vma, addr, page_nid);
-		pte_unmap_unlock(pte, ptl);
-		if (target_nid != -1) {
-			migrated = migrate_misplaced_page(page, vma, target_nid);
-			if (migrated) {
-				page_nid = target_nid;
-				flags |= TNF_MIGRATED;
-			}
-		} else {
-			put_page(page);
-		}
-
-		if (page_nid != -1)
-			task_numa_fault(last_cpupid, page_nid, 1, flags);
-
-		pte = pte_offset_map_lock(mm, pmdp, addr, &ptl);
-	}
-	pte_unmap_unlock(orig_pte, ptl);
-
-	return 0;
-}
-#else
-static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
-		     unsigned long addr, pmd_t *pmdp)
-{
-	BUG();
-	return 0;
-}
-#endif /* CONFIG_NUMA_BALANCING */
-
 /*
  * These routines also need to handle stuff like marking pages dirty
  * and/or accessed for architectures that don't do it in hardware (most
@@ -3857,8 +3760,8 @@ retry:
 		}
 	}
 
-	if (pmd_numa(*pmd))
-		return do_pmd_numa_page(mm, vma, address, pmd);
+	/* THP should already have been handled */
+	BUG_ON(pmd_numa(*pmd));
 
 	/*
 	 * Use __pte_alloc instead of pte_alloc_map, because we can't
mm/mprotect.c +2 -45

86 lines not on this page · raw message

Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#52 lore raw

PATCH 48/63 sched: numa: stay on the same node if CLONE_VM

In reply to Mel Gorman

From: Rik van Riel <[email protected]>

A newly spawned thread inside a process should stay on the same NUMA node as its parent. This prevents processes from being "torn" across multiple NUMA nodes every time they spawn a new thread.

---
 include/linux/sched.h |  2 +-
 kernel/fork.c         |  2 +-
 kernel/sched/core.c   | 14 +++++++++-----
 3 files changed, 11 insertions(+), 7 deletions(-)
include/linux/sched.h +1 -1
index 15888f5..4f51ceb 100644
@@ -2005,7 +2005,7 @@ extern void wake_up_new_task(struct task_struct *tsk);
 #else
  static inline void kick_process(struct task_struct *tsk) { }
 #endif
-extern void sched_fork(struct task_struct *p);
+extern void sched_fork(unsigned long clone_flags, struct task_struct *p);
 extern void sched_dead(struct task_struct *p);
 
 extern void proc_caches_init(void);
kernel/fork.c +1 -1
index 3b1f6af..51f6c4b 100644
@@ -1310,7 +1310,7 @@ static struct task_struct *copy_process(unsigned long clone_flags,
 #endif
 
 	/* Perform scheduler related setup. Assign this task to a CPU. */
-	sched_fork(p);
+	sched_fork(clone_flags, p);
 
 	retval = perf_event_init_task(p);
 	if (retval)
kernel/sched/core.c +9 -5
index 123ac92..336a8ba 100644
@@ -1699,7 +1699,7 @@ int wake_up_state(struct task_struct *p, unsigned int state)
  *
  * __sched_fork() is basic setup used by init_idle() too:
  */
-static void __sched_fork(struct task_struct *p)
+static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
 {
 	p->on_rq			= 0;
 
@@ -1732,11 +1732,15 @@ static void __sched_fork(struct task_struct *p)
 		p->mm->numa_scan_seq = 0;
 	}
 
+	if (clone_flags & CLONE_VM)
+		p->numa_preferred_nid = current->numa_preferred_nid;
+	else
+		p->numa_preferred_nid = -1;
+
 	p->node_stamp = 0ULL;
 	p->numa_scan_seq = p->mm ? p->mm->numa_scan_seq : 0;
 	p->numa_migrate_seq = 1;
 	p->numa_scan_period = sysctl_numa_balancing_scan_delay;
-	p->numa_preferred_nid = -1;
 	p->numa_work.next = &p->numa_work;
 	p->numa_faults = NULL;
 	p->numa_faults_buffer = NULL;
@@ -1768,12 +1772,12 @@ void set_numabalancing_state(bool enabled)
 /*
  * fork()/clone()-time setup:
  */
-void sched_fork(struct task_struct *p)
+void sched_fork(unsigned long clone_flags, struct task_struct *p)
 {
 	unsigned long flags;
 	int cpu = get_cpu();
 
-	__sched_fork(p);
+	__sched_fork(clone_flags, p);
 	/*
 	 * We mark the process as running here. This guarantees that
 	 * nobody will actually run it, and a signal or other external
@@ -4304,7 +4308,7 @@ void init_idle(struct task_struct *idle, int cpu)
 
 	raw_spin_lock_irqsave(&rq->lock, flags);
 
-	__sched_fork(idle);
+	__sched_fork(0, idle);
 	idle->state = TASK_RUNNING;
 	idle->se.exec_start = sched_clock();
 
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#53 lore raw

PATCH 49/63 sched: numa: use group fault statistics in numa placement

In reply to Mel Gorman

This patch uses the fraction of faults on a particular node for both task and group, to figure out the best node to place a task. If the task and group statistics disagree on what the preferred node should be then a full rescan will select the node with the best combined weight.

---
 include/linux/sched.h |   1 +
 kernel/sched/fair.c   | 124 +++++++++++++++++++++++++++++++++++++++++++-------
 2 files changed, 108 insertions(+), 17 deletions(-)
include/linux/sched.h +1 -0
index 4f51ceb..46fb36a 100644
@@ -1347,6 +1347,7 @@ struct task_struct {
 	 * The values remain static for the duration of a PTE scan
 	 */
 	unsigned long *numa_faults;
+	unsigned long total_numa_faults;
 
 	/*
 	 * numa_faults_buffer records faults per node during the current
kernel/sched/fair.c +107 -17

229 lines not on this page · raw message

Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#54 lore raw

PATCH 50/63 sched: numa: call task_numa_free from do_execve

In reply to Mel Gorman

From: Rik van Riel <[email protected]>

It is possible for a task in a numa group to call exec, and have the new (unrelated) executable inherit the numa group association from its former self.

This has the potential to break numa grouping, and is trivial to fix.

---
 fs/exec.c             | 1 +
 include/linux/sched.h | 4 ++++
 kernel/sched/fair.c   | 9 ++++++++-
 kernel/sched/sched.h  | 5 -----
 4 files changed, 13 insertions(+), 6 deletions(-)
fs/exec.c +1 -0
index fd774c7..e73ce23 100644
@@ -1549,6 +1549,7 @@ static int do_execve_common(const char *filename,
 	current->fs->in_exec = 0;
 	current->in_execve = 0;
 	acct_update_integrals(current);
+	task_numa_free(current);
 	free_bprm(bprm);
 	if (displaced)
 		put_files_struct(displaced);
include/linux/sched.h +4 -0
index 46fb36a..31d1740 100644
@@ -1442,6 +1442,7 @@ struct task_struct {
 extern void task_numa_fault(int last_node, int node, int pages, int flags);
 extern pid_t task_numa_group_id(struct task_struct *p);
 extern void set_numabalancing_state(bool enabled);
+extern void task_numa_free(struct task_struct *p);
 #else
 static inline void task_numa_fault(int last_node, int node, int pages,
 				   int flags)
@@ -1454,6 +1455,9 @@ static inline pid_t task_numa_group_id(struct task_struct *p)
 static inline void set_numabalancing_state(bool enabled)
 {
 }
+static inline void task_numa_free(struct task_struct *p)
+{
+}
 #endif
 
 static inline struct pid *task_pid(struct task_struct *task)
kernel/sched/fair.c +8 -1
index 801fc74..5237feb 100644
@@ -1418,6 +1418,7 @@ void task_numa_free(struct task_struct *p)
 {
 	struct numa_group *grp = p->numa_group;
 	int i;
+	void *numa_faults = p->numa_faults;
 
 	if (grp) {
 		for (i = 0; i < 2*nr_node_ids; i++)
@@ -1433,7 +1434,9 @@ void task_numa_free(struct task_struct *p)
 		put_numa_group(grp);
 	}
 
-	kfree(p->numa_faults);
+	p->numa_faults = NULL;
+	p->numa_faults_buffer = NULL;
+	kfree(numa_faults);
 }
 
 /*
@@ -1452,6 +1455,10 @@ void task_numa_fault(int last_cpupid, int node, int pages, int flags)
 	if (!p->mm)
 		return;
 
+	/* Do not worry about placement if exiting */
+	if (p->state == TASK_DEAD)
+		return;
+
 	/* Allocate buffer to track faults on a per-node basis */
 	if (unlikely(!p->numa_faults)) {
 		int size = sizeof(*p->numa_faults) * 2 * nr_node_ids;
kernel/sched/sched.h +0 -5
index e0875dc..657472d 100644
@@ -557,11 +557,6 @@ static inline u64 rq_clock_task(struct rq *rq)
 #ifdef CONFIG_NUMA_BALANCING
 extern int migrate_task_to(struct task_struct *p, int cpu);
 extern int migrate_swap(struct task_struct *, struct task_struct *);
-extern void task_numa_free(struct task_struct *p);
-#else /* CONFIG_NUMA_BALANCING */
-static inline void task_numa_free(struct task_struct *p)
-{
-}
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_SMP
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#55 lore raw

PATCH 51/63 sched: numa: Prevent parallel updates to group stats during placement

In reply to Mel Gorman

Having multiple tasks in a group go through task_numa_placement simultaneously can lead to a task picking a wrong node to run on, because the group stats may be in the middle of an update. This patch avoids parallel updates by holding the numa_group lock during placement decisions.

---
 kernel/sched/fair.c | 35 +++++++++++++++++++++++------------
 1 file changed, 23 insertions(+), 12 deletions(-)
kernel/sched/fair.c +23 -12
index 5237feb..86511a5 100644
@@ -1233,6 +1233,7 @@ static void task_numa_placement(struct task_struct *p)
 {
 	int seq, nid, max_nid = -1, max_group_nid = -1;
 	unsigned long max_faults = 0, max_group_faults = 0;
+	spinlock_t *group_lock = NULL;
 
 	seq = ACCESS_ONCE(p->mm->numa_scan_seq);
 	if (p->numa_scan_seq == seq)
@@ -1241,6 +1242,12 @@ static void task_numa_placement(struct task_struct *p)
 	p->numa_migrate_seq++;
 	p->numa_scan_period_max = task_scan_max(p);
 
+	/* If the task is part of a group prevent parallel updates to group stats */
+	if (p->numa_group) {
+		group_lock = &p->numa_group->lock;
+		spin_lock(group_lock);
+	}
+
 	/* Find the node with the highest number of faults */
 	for_each_online_node(nid) {
 		unsigned long faults = 0, group_faults = 0;
@@ -1279,20 +1286,24 @@ static void task_numa_placement(struct task_struct *p)
 		}
 	}
 
-	/*
-	 * If the preferred task and group nids are different,
-	 * iterate over the nodes again to find the best place.
-	 */
-	if (p->numa_group && max_nid != max_group_nid) {
-		unsigned long weight, max_weight = 0;
-
-		for_each_online_node(nid) {
-			weight = task_weight(p, nid) + group_weight(p, nid);
-			if (weight > max_weight) {
-				max_weight = weight;
-				max_nid = nid;
+	if (p->numa_group) {
+		/*
+		 * If the preferred task and group nids are different,
+		 * iterate over the nodes again to find the best place.
+		 */
+		if (max_nid != max_group_nid) {
+			unsigned long weight, max_weight = 0;
+
+			for_each_online_node(nid) {
+				weight = task_weight(p, nid) + group_weight(p, nid);
+				if (weight > max_weight) {
+					max_weight = weight;
+					max_nid = nid;
+				}
 			}
 		}
+
+		spin_unlock(group_lock);
 	}
 
 	/* Preferred node as the node with the most faults */
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#56 lore raw

PATCH 52/63 sched: numa: add debugging

In reply to Mel Gorman

From: Ingo Molnar <[email protected]>

---
 include/linux/sched.h |  6 ++++++
 kernel/sched/debug.c  | 60 +++++++++++++++++++++++++++++++++++++++++++++++++--
 kernel/sched/fair.c   |  5 ++++-
 3 files changed, 68 insertions(+), 3 deletions(-)
include/linux/sched.h +6 -0
index 31d1740..11b873f 100644
@@ -1357,6 +1357,7 @@ struct task_struct {
 	unsigned long *numa_faults_buffer;
 
 	int numa_preferred_nid;
+	unsigned long numa_pages_migrated;
 #endif /* CONFIG_NUMA_BALANCING */
 
 	struct rcu_head rcu;
@@ -2581,6 +2582,11 @@ static inline unsigned int task_cpu(const struct task_struct *p)
 	return task_thread_info(p)->cpu;
 }
 
+static inline int task_node(const struct task_struct *p)
+{
+	return cpu_to_node(task_cpu(p));
+}
+
 extern void set_task_cpu(struct task_struct *p, unsigned int cpu);
 
 #else
kernel/sched/debug.c +58 -2

96 lines not on this page · raw message

kernel/sched/fair.c +4 -1
index 86511a5..cd1b862 100644
@@ -1137,7 +1137,7 @@ static int task_numa_migrate(struct task_struct *p)
 		.p = p,
 
 		.src_cpu = task_cpu(p),
-		.src_nid = cpu_to_node(task_cpu(p)),
+		.src_nid = task_node(p),
 
 		.imbalance_pct = 112,
 
@@ -1515,6 +1515,9 @@ void task_numa_fault(int last_cpupid, int node, int pages, int flags)
 	if (p->numa_migrate_retry && time_after(jiffies, p->numa_migrate_retry))
 		numa_migrate_preferred(p);
 
+	if (migrated)
+		p->numa_pages_migrated += pages;
+
 	p->numa_faults_buffer[task_faults_idx(node, priv)] += pages;
 }
 
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#57 lore raw

PATCH 53/63 sched: numa: Decide whether to favour task or group weights based on swap candidate relationships

In reply to Mel Gorman

From: Rik van Riel <[email protected]>

This patch separately considers task and group affinities when searching for swap candidates during task NUMA placement. If tasks are not part of a group or the same group then the task weights are considered. Otherwise the group weights are compared.

---
 kernel/sched/fair.c | 59 ++++++++++++++++++++++++++++++++---------------------
 1 file changed, 36 insertions(+), 23 deletions(-)
kernel/sched/fair.c +36 -23
index cd1b862..60ca698 100644
@@ -1039,13 +1039,15 @@ static void task_numa_assign(struct task_numa_env *env,
  * into account that it might be best if task running on the dst_cpu should
  * be exchanged with the source task
  */
-static void task_numa_compare(struct task_numa_env *env, long imp)
+static void task_numa_compare(struct task_numa_env *env,
+			      long taskimp, long groupimp)
 {
 	struct rq *src_rq = cpu_rq(env->src_cpu);
 	struct rq *dst_rq = cpu_rq(env->dst_cpu);
 	struct task_struct *cur;
 	long dst_load, src_load;
 	long load;
+	long imp = (groupimp > 0) ? groupimp : taskimp;
 
 	rcu_read_lock();
 	cur = ACCESS_ONCE(dst_rq->curr);
@@ -1064,10 +1066,19 @@ static void task_numa_compare(struct task_numa_env *env, long imp)
 		if (!cpumask_test_cpu(env->src_cpu, tsk_cpus_allowed(cur)))
 			goto unlock;
 
-		imp += task_weight(cur, env->src_nid) +
-		       group_weight(cur, env->src_nid) -
-		       task_weight(cur, env->dst_nid) -
-		       group_weight(cur, env->dst_nid);
+		/*
+		 * If dst and source tasks are in the same NUMA group, or not
+		 * in any group then look only at task weights otherwise give
+		 * priority to the group weights.
+		 */
+		if (!cur->numa_group || !env->p->numa_group ||
+		    cur->numa_group == env->p->numa_group) {
+			imp = taskimp + task_weight(cur, env->src_nid) -
+			      task_weight(cur, env->dst_nid);
+		} else {
+			imp = groupimp + group_weight(cur, env->src_nid) -
+			       group_weight(cur, env->dst_nid);
+		}
 	}
 
 	if (imp < env->best_imp)
@@ -1117,7 +1128,8 @@ unlock:
 	rcu_read_unlock();
 }
 
-static void task_numa_find_cpu(struct task_numa_env *env, long imp)
+static void task_numa_find_cpu(struct task_numa_env *env,
+				long taskimp, long groupimp)
 {
 	int cpu;
 
@@ -1127,7 +1139,7 @@ static void task_numa_find_cpu(struct task_numa_env *env, long imp)
 			continue;
 
 		env->dst_cpu = cpu;
-		task_numa_compare(env, imp);
+		task_numa_compare(env, taskimp, groupimp);
 	}
 }
 
@@ -1146,9 +1158,9 @@ static int task_numa_migrate(struct task_struct *p)
 		.best_cpu = -1
 	};
 	struct sched_domain *sd;
-	unsigned long weight;
+	unsigned long taskweight, groupweight;
 	int nid, ret;
-	long imp;
+	long taskimp, groupimp;
 
 	/*
 	 * Pick the lowest SD_NUMA domain, as that would have the smallest
@@ -1163,15 +1175,17 @@ static int task_numa_migrate(struct task_struct *p)
 	env.imbalance_pct = 100 + (sd->imbalance_pct - 100) / 2;
 	rcu_read_unlock();
 
-	weight = task_weight(p, env.src_nid) + group_weight(p, env.src_nid);
+	taskweight = task_weight(p, env.src_nid);
+	groupweight = group_weight(p, env.src_nid);
 	update_numa_stats(&env.src_stats, env.src_nid);
 	env.dst_nid = p->numa_preferred_nid;
-	imp = task_weight(p, env.dst_nid) + group_weight(p, env.dst_nid) - weight;
+	taskimp = task_weight(p, env.dst_nid) - taskweight;
+	groupimp = group_weight(p, env.dst_nid) - groupweight;
 	update_numa_stats(&env.dst_stats, env.dst_nid);
 
 	/* If the preferred nid has capacity, try to use it. */
 	if (env.dst_stats.has_capacity)
-		task_numa_find_cpu(&env, imp);
+		task_numa_find_cpu(&env, taskimp, groupimp);
 
 	/* No space available on the preferred nid. Look elsewhere. */
 	if (env.best_cpu == -1) {
@@ -1180,13 +1194,14 @@ static int task_numa_migrate(struct task_struct *p)
 				continue;
 
 			/* Only consider nodes where both task and groups benefit */
-			imp = task_weight(p, nid) + group_weight(p, nid) - weight;
-			if (imp < 0)
+			taskimp = task_weight(p, nid) - taskweight;
+			groupimp = group_weight(p, nid) - groupweight;
+			if (taskimp < 0 && groupimp < 0)
 				continue;
 
 			env.dst_nid = nid;
 			update_numa_stats(&env.dst_stats, env.dst_nid);
-			task_numa_find_cpu(&env, imp);
+			task_numa_find_cpu(&env, taskimp, groupimp);
 		}
 	}
 
@@ -4632,10 +4647,9 @@ static bool migrate_improves_locality(struct task_struct *p, struct lb_env *env)
 	if (dst_nid == p->numa_preferred_nid)
 		return true;
 
-	/* After the task has settled, check if the new node is better. */
-	if (p->numa_migrate_seq >= sysctl_numa_balancing_settle_count &&
-			task_weight(p, dst_nid) + group_weight(p, dst_nid) >
-			task_weight(p, src_nid) + group_weight(p, src_nid))
+	/* If both task and group weight improve, this move is a winner. */
+	if (task_weight(p, dst_nid) > task_weight(p, src_nid) &&
+	    group_weight(p, dst_nid) > group_weight(p, src_nid))
 		return true;
 
 	return false;
@@ -4662,10 +4676,9 @@ static bool migrate_degrades_locality(struct task_struct *p, struct lb_env *env)
 	if (src_nid == p->numa_preferred_nid)
 		return true;
 
-	/* After the task has settled, check if the new node is worse. */
-	if (p->numa_migrate_seq >= sysctl_numa_balancing_settle_count &&
-			task_weight(p, dst_nid) + group_weight(p, dst_nid) <
-			task_weight(p, src_nid) + group_weight(p, src_nid))
+	/* If either task or group weight get worse, don't do it. */
+	if (task_weight(p, dst_nid) < task_weight(p, src_nid) ||
+	    group_weight(p, dst_nid) < group_weight(p, src_nid))
 		return true;
 
 	return false;
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#58 lore raw

PATCH 54/63 sched: numa: fix task or group comparison

In reply to Mel Gorman

From: Rik van Riel <[email protected]>

This patch separately considers task and group affinities when searching for swap candidates during NUMA placement. If tasks are part of the same group, or no group at all, the task weights are considered.

Some hysteresis is added to prevent tasks within one group from getting bounced between NUMA nodes due to tiny differences.

If tasks are part of different groups, the code compares group weights, in order to favor grouping task groups together.

The patch also changes the group weight multiplier to be the same as the task weight multiplier, since the two are no longer added up like before.

---
 kernel/sched/fair.c | 32 +++++++++++++++++++++++++-------
 1 file changed, 25 insertions(+), 7 deletions(-)
kernel/sched/fair.c +25 -7
index 60ca698..f232be6 100644
@@ -962,7 +962,7 @@ static inline unsigned long group_weight(struct task_struct *p, int nid)
 	if (!total_faults)
 		return 0;
 
-	return 1200 * group_faults(p, nid) / total_faults;
+	return 1000 * group_faults(p, nid) / total_faults;
 }
 
 static unsigned long weighted_cpuload(const int cpu);
@@ -1068,16 +1068,34 @@ static void task_numa_compare(struct task_numa_env *env,
 
 		/*
 		 * If dst and source tasks are in the same NUMA group, or not
-		 * in any group then look only at task weights otherwise give
-		 * priority to the group weights.
+		 * in any group then look only at task weights.
 		 */
-		if (!cur->numa_group || !env->p->numa_group ||
-		    cur->numa_group == env->p->numa_group) {
+		if (cur->numa_group == env->p->numa_group) {
 			imp = taskimp + task_weight(cur, env->src_nid) -
 			      task_weight(cur, env->dst_nid);
+			/*
+			 * Add some hysteresis to prevent swapping the
+			 * tasks within a group over tiny differences.
+			 */
+			if (cur->numa_group)
+				imp -= imp/16;
 		} else {
-			imp = groupimp + group_weight(cur, env->src_nid) -
-			       group_weight(cur, env->dst_nid);
+			/*
+			 * Compare the group weights. If a task is all by
+			 * itself (not part of a group), use the task weight
+			 * instead.
+			 */
+			if (env->p->numa_group)
+				imp = groupimp;
+			else
+				imp = taskimp;
+
+			if (cur->numa_group)
+				imp += group_weight(cur, env->src_nid) -
+				       group_weight(cur, env->dst_nid);
+			else
+				imp += task_weight(cur, env->src_nid) -
+				       task_weight(cur, env->dst_nid);
 		}
 	}
 
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#59 lore raw

PATCH 55/63 sched: numa: Avoid migrating tasks that are placed on their preferred node

In reply to Mel Gorman

From: Peter Zijlstra <[email protected]>

(This changelog needs more work, it's currently inaccurate and it's not clear at exactly what point rt > env->fbq_type is true for the logic to kick in)

This patch classifies scheduler domains and runqueues into FBQ (cannot guess what this expands to) types which are one of

regular: There are tasks running that do not care about their NUMA
placement

remote: There are tasks running that care about their placement but are
currently running on a node remote to their ideal placement

all: No distinction

To implement this the patch tracks the number of tasks that are optimally NUMA placed (rq->nr_preferred_running) and the number of tasks running that care about their placement (nr_numa_running). The load balancer uses this information to avoid migrating idea placed NUMA tasks as long as better options for load balancing exists.

Not-signed-off-by: Peter Zijlstra

---
 kernel/sched/core.c  |  29 ++++++++++++
 kernel/sched/fair.c  | 128 ++++++++++++++++++++++++++++++++++++++++++++++-----
 kernel/sched/sched.h |   5 ++
 3 files changed, 150 insertions(+), 12 deletions(-)
kernel/sched/core.c +29 -0
index 336a8ba..7abbae9 100644
@@ -4485,6 +4485,35 @@ int migrate_task_to(struct task_struct *p, int target_cpu)
 
 	return stop_one_cpu(curr_cpu, migration_cpu_stop, &arg);
 }
+
+/*
+ * Requeue a task on a given node and accurately track the number of NUMA
+ * tasks on the runqueues
+ */
+void sched_setnuma(struct task_struct *p, int nid)
+{
+	struct rq *rq;
+	unsigned long flags;
+	bool on_rq, running;
+
+	rq = task_rq_lock(p, &flags);
+	on_rq = p->on_rq;
+	running = task_current(rq, p);
+
+	if (on_rq)
+		dequeue_task(rq, p, 0);
+	if (running)
+		p->sched_class->put_prev_task(rq, p);
+
+	p->numa_preferred_nid = nid;
+	p->numa_migrate_seq = 1;
+
+	if (running)
+		p->sched_class->set_curr_task(rq);
+	if (on_rq)
+		enqueue_task(rq, p, 0);
+	task_rq_unlock(rq, p, &flags);
+}
 #endif
 
 /*
kernel/sched/fair.c +116 -12

223 lines not on this page · raw message

kernel/sched/sched.h +5 -0
index 657472d..22c470c 100644
@@ -407,6 +407,10 @@ struct rq {
 	 * remote CPUs use both these fields when doing load calculation.
 	 */
 	unsigned int nr_running;
+#ifdef CONFIG_NUMA_BALANCING
+	unsigned int nr_numa_running;
+	unsigned int nr_preferred_running;
+#endif
 	#define CPU_LOAD_IDX_MAX 5
 	unsigned long cpu_load[CPU_LOAD_IDX_MAX];
 	unsigned long last_load_update_tick;
@@ -555,6 +559,7 @@ static inline u64 rq_clock_task(struct rq *rq)
 }
 
 #ifdef CONFIG_NUMA_BALANCING
+extern void sched_setnuma(struct task_struct *p, int node);
 extern int migrate_task_to(struct task_struct *p, int cpu);
 extern int migrate_swap(struct task_struct *, struct task_struct *);
 #endif /* CONFIG_NUMA_BALANCING */
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#60 lore raw

PATCH 56/63 sched: numa: be more careful about joining numa groups

In reply to Mel Gorman

From: Rik van Riel <[email protected]>

Due to the way the pid is truncated, and tasks are moved between CPUs by the scheduler, it is possible for the current task_numa_fault to group together tasks that do not actually share memory together.

This patch adds a few easy sanity checks to task_numa_fault, joining tasks together if they share the same tsk->mm, or if the fault was on a page with an elevated mapcount, in a shared VMA.

---
 include/linux/sched.h |  1 +
 kernel/sched/fair.c   | 16 +++++++++++-----
 mm/memory.c           |  7 +++++++
 3 files changed, 19 insertions(+), 5 deletions(-)
include/linux/sched.h +1 -0
index 11b873f..718bb60 100644
@@ -1438,6 +1438,7 @@ struct task_struct {
 
 #define TNF_MIGRATED	0x01
 #define TNF_NO_GROUP	0x02
+#define TNF_SHARED	0x04
 
 #ifdef CONFIG_NUMA_BALANCING
 extern void task_numa_fault(int last_node, int node, int pages, int flags);
kernel/sched/fair.c +11 -5
index 83c6e69..1b0ff1d 100644
@@ -1381,7 +1381,7 @@ static void double_lock(spinlock_t *l1, spinlock_t *l2)
 	spin_lock_nested(l2, SINGLE_DEPTH_NESTING);
 }
 
-static void task_numa_group(struct task_struct *p, int cpupid)
+static void task_numa_group(struct task_struct *p, int cpupid, int flags)
 {
 	struct numa_group *grp, *my_grp;
 	struct task_struct *tsk;
@@ -1439,10 +1439,16 @@ static void task_numa_group(struct task_struct *p, int cpupid)
 	if (my_grp->nr_tasks == grp->nr_tasks && my_grp > grp)
 		goto unlock;
 
-	if (!get_numa_group(grp))
-		goto unlock;
+	/* Always join threads in the same process. */
+	if (tsk->mm == current->mm)
+		join = true;
+
+	/* Simple filter to avoid false positives due to PID collisions */
+	if (flags & TNF_SHARED)
+		join = true;
 
-	join = true;
+	if (join && !get_numa_group(grp))
+		join = false;
 
 unlock:
 	rcu_read_unlock();
@@ -1539,7 +1545,7 @@ void task_numa_fault(int last_cpupid, int node, int pages, int flags)
 	} else {
 		priv = cpupid_match_pid(p, last_cpupid);
 		if (!priv && !(flags & TNF_NO_GROUP))
-			task_numa_group(p, last_cpupid);
+			task_numa_group(p, last_cpupid, flags);
 	}
 
 	/*
mm/memory.c +7 -0
index da65ad4..7bef788 100644
@@ -3595,6 +3595,13 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	if (!pte_write(pte))
 		flags |= TNF_NO_GROUP;
 
+	/*
+	 * Flag if the page is shared between multiple address spaces. This
+	 * is later used when determining whether to group tasks together
+	 */
+	if (page_mapcount(page) > 1 && (vma->vm_flags & VM_SHARED))
+		flags |= TNF_SHARED;
+
 	last_cpupid = page_cpupid_last(page);
 	page_nid = page_to_nid(page);
 	target_nid = numa_migrate_prep(page, vma, addr, page_nid);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#61 lore raw

PATCH 57/63 sched: numa: Take false sharing into account when adapting scan rate

In reply to Mel Gorman

Scan rate is altered based on whether shared/private faults dominated. task_numa_group() may detect false sharing but that information is not taken into account when adapting the scan rate. Take it into account.

---
 kernel/sched/fair.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
kernel/sched/fair.c +6 -2
index 1b0ff1d..3486042 100644
@@ -1381,7 +1381,8 @@ static void double_lock(spinlock_t *l1, spinlock_t *l2)
 	spin_lock_nested(l2, SINGLE_DEPTH_NESTING);
 }
 
-static void task_numa_group(struct task_struct *p, int cpupid, int flags)
+static void task_numa_group(struct task_struct *p, int cpupid, int flags,
+			int *priv)
 {
 	struct numa_group *grp, *my_grp;
 	struct task_struct *tsk;
@@ -1447,6 +1448,9 @@ static void task_numa_group(struct task_struct *p, int cpupid, int flags)
 	if (flags & TNF_SHARED)
 		join = true;
 
+	/* Update priv based on whether false sharing was detected */
+	*priv = !join;
+
 	if (join && !get_numa_group(grp))
 		join = false;
 
@@ -1545,7 +1549,7 @@ void task_numa_fault(int last_cpupid, int node, int pages, int flags)
 	} else {
 		priv = cpupid_match_pid(p, last_cpupid);
 		if (!priv && !(flags & TNF_NO_GROUP))
-			task_numa_group(p, last_cpupid, flags);
+			task_numa_group(p, last_cpupid, flags, &priv);
 	}
 
 	/*
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#62 lore raw

PATCH 58/63 sched: numa: adjust scan rate in task_numa_placement

In reply to Mel Gorman

From: Rik van Riel <[email protected]>

Adjust numa_scan_period in task_numa_placement, depending on how much useful work the numa code can do. The more local faults there are in a given scan window the longer the period (and hence the slower the scan rate) during the next window. If there are excessive shared faults then the scan period will decrease with the amount of scaling depending on whether the ratio of shared/private faults. If the preferred node changes then the scan rate is reset to recheck if the task is properly placed.

---
 include/linux/sched.h |   9 ++++
 kernel/sched/fair.c   | 112 +++++++++++++++++++++++++++++++++++++++-----------
 mm/huge_memory.c      |   4 +-
 mm/memory.c           |   9 ++--
 4 files changed, 105 insertions(+), 29 deletions(-)
include/linux/sched.h +9 -0
index 718bb60..918baf3 100644
@@ -1356,6 +1356,14 @@ struct task_struct {
 	 */
 	unsigned long *numa_faults_buffer;
 
+	/*
+	 * numa_faults_locality tracks if faults recorded during the last
+	 * scan window were remote/local. The task scan period is adapted
+	 * based on the locality of the faults with different weights
+	 * depending on whether they were shared or private faults
+	 */
+	unsigned long numa_faults_locality[2];
+
 	int numa_preferred_nid;
 	unsigned long numa_pages_migrated;
 #endif /* CONFIG_NUMA_BALANCING */
@@ -1439,6 +1447,7 @@ struct task_struct {
 #define TNF_MIGRATED	0x01
 #define TNF_NO_GROUP	0x02
 #define TNF_SHARED	0x04
+#define TNF_FAULT_LOCAL	0x08
 
 #ifdef CONFIG_NUMA_BALANCING
 extern void task_numa_fault(int last_node, int node, int pages, int flags);
kernel/sched/fair.c +87 -25

164 lines not on this page · raw message

mm/huge_memory.c +3 -1
index aaf46dc..22096da 100644
@@ -1308,8 +1308,10 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	page_nid = page_to_nid(page);
 	last_cpupid = page_cpupid_last(page);
 	count_vm_numa_event(NUMA_HINT_FAULTS);
-	if (page_nid == this_nid)
+	if (page_nid == this_nid) {
 		count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL);
+		flags |= TNF_FAULT_LOCAL;
+	}
 
 	/*
 	 * Avoid grouping on DSO/COW pages in specific and RO pages
mm/memory.c +6 -3
index 7bef788..1865e4c 100644
@@ -3538,13 +3538,16 @@ static int do_nonlinear_fault(struct mm_struct *mm, struct vm_area_struct *vma,
 }
 
 int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
-				unsigned long addr, int page_nid)
+				unsigned long addr, int page_nid,
+				int *flags)
 {
 	get_page(page);
 
 	count_vm_numa_event(NUMA_HINT_FAULTS);
-	if (page_nid == numa_node_id())
+	if (page_nid == numa_node_id()) {
 		count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL);
+		*flags |= TNF_FAULT_LOCAL;
+	}
 
 	return mpol_misplaced(page, vma, addr);
 }
@@ -3604,7 +3607,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
 
 	last_cpupid = page_cpupid_last(page);
 	page_nid = page_to_nid(page);
-	target_nid = numa_migrate_prep(page, vma, addr, page_nid);
+	target_nid = numa_migrate_prep(page, vma, addr, page_nid, &flags);
 	pte_unmap_unlock(ptep, ptl);
 	if (target_nid == -1) {
 		put_page(page);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#63 lore raw

PATCH 59/63 sched: numa: Remove the numa_balancing_scan_period_reset sysctl

In reply to Mel Gorman

With scan rate adaptions based on whether the workload has properly converged or not there should be no need for the scan period reset hammer. Get rid of it.

---
 Documentation/sysctl/kernel.txt | 11 +++--------
 include/linux/mm_types.h        |  3 ---
 include/linux/sched/sysctl.h    |  1 -
 kernel/sched/core.c             |  1 -
 kernel/sched/fair.c             | 18 +-----------------
 kernel/sysctl.c                 |  7 -------
 6 files changed, 4 insertions(+), 37 deletions(-)
Documentation/sysctl/kernel.txt +3 -8
index 23ff00a..0d503df 100644
@@ -373,15 +373,13 @@ guarantee. If the target workload is already bound to NUMA nodes then this
 feature should be disabled. Otherwise, if the system overhead from the
 feature is too high then the rate the kernel samples for NUMA hinting
 faults may be controlled by the numa_balancing_scan_period_min_ms,
-numa_balancing_scan_delay_ms, numa_balancing_scan_period_reset,
-numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb and
-numa_balancing_settle_count sysctls.
+numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms,
+numa_balancing_scan_size_mb and numa_balancing_settle_count sysctls.
 
 ==============================================================
 
 numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms,
-numa_balancing_scan_period_max_ms, numa_balancing_scan_period_reset,
-numa_balancing_scan_size_mb
+numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb
 
 Automatic NUMA balancing scans tasks address space and unmaps pages to
 detect if pages are properly placed or if the data should be migrated to a
@@ -417,9 +415,6 @@ rate for each task.
 numa_balancing_scan_size_mb is how many megabytes worth of pages are
 scanned for a given scan.
 
-numa_balancing_scan_period_reset is a blunt instrument that controls how
-often a tasks scan delay is reset to detect sudden changes in task behaviour.
-
 numa_balancing_settle_count is how many scan periods must complete before
 the schedule balancer stops pushing the task towards a preferred node. This
 gives the scheduler a chance to place the task on an alternative node if the
include/linux/mm_types.h +0 -3
index b0370cd..7f70a00 100644
@@ -419,9 +419,6 @@ struct mm_struct {
 	 */
 	unsigned long numa_next_scan;
 
-	/* numa_next_reset is when the PTE scanner period will be reset */
-	unsigned long numa_next_reset;
-
 	/* Restart point for scanning and setting pte_numa */
 	unsigned long numa_scan_offset;
 
include/linux/sched/sysctl.h +0 -1
index bf8086b..10d16c4f 100644
@@ -47,7 +47,6 @@ extern enum sched_tunable_scaling sysctl_sched_tunable_scaling;
 extern unsigned int sysctl_numa_balancing_scan_delay;
 extern unsigned int sysctl_numa_balancing_scan_period_min;
 extern unsigned int sysctl_numa_balancing_scan_period_max;
-extern unsigned int sysctl_numa_balancing_scan_period_reset;
 extern unsigned int sysctl_numa_balancing_scan_size;
 extern unsigned int sysctl_numa_balancing_settle_count;
 
kernel/sched/core.c +0 -1
index 7abbae9..3d60433 100644
@@ -1728,7 +1728,6 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
 #ifdef CONFIG_NUMA_BALANCING
 	if (p->mm && atomic_read(&p->mm->mm_users) == 1) {
 		p->mm->numa_next_scan = jiffies + msecs_to_jiffies(sysctl_numa_balancing_scan_delay);
-		p->mm->numa_next_reset = jiffies + msecs_to_jiffies(sysctl_numa_balancing_scan_period_reset);
 		p->mm->numa_scan_seq = 0;
 	}
 
kernel/sched/fair.c +1 -17
index 4f99c09..7c362f9 100644
@@ -826,7 +826,6 @@ static unsigned long task_h_load(struct task_struct *p);
  */
 unsigned int sysctl_numa_balancing_scan_period_min = 1000;
 unsigned int sysctl_numa_balancing_scan_period_max = 60000;
-unsigned int sysctl_numa_balancing_scan_period_reset = 60000;
 
 /* Portion of address space to scan in MB */
 unsigned int sysctl_numa_balancing_scan_size = 256;
@@ -1685,24 +1684,9 @@ void task_numa_work(struct callback_head *work)
 	if (p->flags & PF_EXITING)
 		return;
 
-	if (!mm->numa_next_reset || !mm->numa_next_scan) {
+	if (!mm->numa_next_scan) {
 		mm->numa_next_scan = now +
 			msecs_to_jiffies(sysctl_numa_balancing_scan_delay);
-		mm->numa_next_reset = now +
-			msecs_to_jiffies(sysctl_numa_balancing_scan_period_reset);
-	}
-
-	/*
-	 * Reset the scan period if enough time has gone by. Objective is that
-	 * scanning will be reduced if pages are properly placed. As tasks
-	 * can enter different phases this needs to be re-examined. Lacking
-	 * proper tracking of reference behaviour, this blunt hammer is used.
-	 */
-	migrate = mm->numa_next_reset;
-	if (time_after(now, migrate)) {
-		p->numa_scan_period = task_scan_min(p);
-		next_scan = now + msecs_to_jiffies(sysctl_numa_balancing_scan_period_reset);
-		xchg(&mm->numa_next_reset, next_scan);
 	}
 
 	/*
kernel/sysctl.c +0 -7
index 0015fb9..4e080fe 100644
@@ -371,13 +371,6 @@ static struct ctl_table kern_table[] = {
 		.proc_handler	= proc_dointvec,
 	},
 	{
-		.procname	= "numa_balancing_scan_period_reset",
-		.data		= &sysctl_numa_balancing_scan_period_reset,
-		.maxlen		= sizeof(unsigned int),
-		.mode		= 0644,
-		.proc_handler	= proc_dointvec,
-	},
-	{
 		.procname	= "numa_balancing_scan_period_max_ms",
 		.data		= &sysctl_numa_balancing_scan_period_max,
 		.maxlen		= sizeof(unsigned int),
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#64 lore raw

PATCH 60/63 mm: numa: revert temporarily disabling of NUMA migration

In reply to Mel Gorman

From: Rik van Riel <[email protected]>

With the scan rate code working (at least for multi-instance specjbb), the large hammer that is "sched: Do not migrate memory immediately after switching node" can be replaced with something smarter. Revert temporarily migration disabling and all traces of numa_migrate_seq.

---
 include/linux/sched.h |  1 -
 kernel/sched/core.c   |  2 --
 kernel/sched/fair.c   | 25 +------------------------
 mm/mempolicy.c        | 12 ------------
 4 files changed, 1 insertion(+), 39 deletions(-)
include/linux/sched.h +0 -1
index 918baf3..33c53d6 100644
@@ -1331,7 +1331,6 @@ struct task_struct {
 #endif
 #ifdef CONFIG_NUMA_BALANCING
 	int numa_scan_seq;
-	int numa_migrate_seq;
 	unsigned int numa_scan_period;
 	unsigned int numa_scan_period_max;
 	unsigned long numa_migrate_retry;
kernel/sched/core.c +0 -2
index 3d60433..b7cf574 100644
@@ -1738,7 +1738,6 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
 
 	p->node_stamp = 0ULL;
 	p->numa_scan_seq = p->mm ? p->mm->numa_scan_seq : 0;
-	p->numa_migrate_seq = 1;
 	p->numa_scan_period = sysctl_numa_balancing_scan_delay;
 	p->numa_work.next = &p->numa_work;
 	p->numa_faults = NULL;
@@ -4505,7 +4504,6 @@ void sched_setnuma(struct task_struct *p, int nid)
 		p->sched_class->put_prev_task(rq, p);
 
 	p->numa_preferred_nid = nid;
-	p->numa_migrate_seq = 1;
 
 	if (running)
 		p->sched_class->set_curr_task(rq);
kernel/sched/fair.c +1 -24
index 7c362f9..9bca073 100644
@@ -1261,16 +1261,8 @@ static void numa_migrate_preferred(struct task_struct *p)
 {
 	/* Success if task is already running on preferred CPU */
 	p->numa_migrate_retry = 0;
-	if (cpu_to_node(task_cpu(p)) == p->numa_preferred_nid) {
-		/*
-		 * If migration is temporarily disabled due to a task migration
-		 * then re-enable it now as the task is running on its
-		 * preferred node and memory should migrate locally
-		 */
-		if (!p->numa_migrate_seq)
-			p->numa_migrate_seq++;
+	if (cpu_to_node(task_cpu(p)) == p->numa_preferred_nid)
 		return;
-	}
 
 	/* This task has no NUMA fault statistics yet */
 	if (unlikely(p->numa_preferred_nid == -1))
@@ -1367,7 +1359,6 @@ static void task_numa_placement(struct task_struct *p)
 	if (p->numa_scan_seq == seq)
 		return;
 	p->numa_scan_seq = seq;
-	p->numa_migrate_seq++;
 	p->numa_scan_period_max = task_scan_max(p);
 
 	/* If the task is part of a group prevent parallel updates to group stats */
@@ -4683,20 +4674,6 @@ static void move_task(struct task_struct *p, struct lb_env *env)
 	set_task_cpu(p, env->dst_cpu);
 	activate_task(env->dst_rq, p, 0);
 	check_preempt_curr(env->dst_rq, p, 0);
-#ifdef CONFIG_NUMA_BALANCING
-	if (p->numa_preferred_nid != -1) {
-		int src_nid = cpu_to_node(env->src_cpu);
-		int dst_nid = cpu_to_node(env->dst_cpu);
-
-		/*
-		 * If the load balancer has moved the task then limit
-		 * migrations from taking place in the short term in
-		 * case this is a short-lived migration.
-		 */
-		if (src_nid != dst_nid && dst_nid != p->numa_preferred_nid)
-			p->numa_migrate_seq = 0;
-	}
-#endif
 }
 
 /*
mm/mempolicy.c +0 -12
index 07db309..e554587 100644
@@ -2348,18 +2348,6 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long
 		last_cpupid = page_cpupid_xchg_last(page, this_cpupid);
 		if (!cpupid_pid_unset(last_cpupid) && cpupid_to_nid(last_cpupid) != thisnid)
 			goto out;
-
-#ifdef CONFIG_NUMA_BALANCING
-		/*
-		 * If the scheduler has just moved us away from our
-		 * preferred node, do not bother migrating pages yet.
-		 * This way a short and temporary process migration will
-		 * not cause excessive memory migration.
-		 */
-		if (thisnid != current->numa_preferred_nid &&
-				!current->numa_migrate_seq)
-			goto out;
-#endif
 	}
 
 	if (curnid != polnid)
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#65 lore raw

PATCH 61/63 sched: numa: skip some page migrations after a shared fault

In reply to Mel Gorman

From: Rik van Riel <[email protected]>

Shared faults can lead to lots of unnecessary page migrations, slowing down the system, and causing private faults to hit the per-pgdat migration ratelimit.

This patch adds sysctl numa_balancing_migrate_deferred, which specifies how many shared page migrations to skip unconditionally, after each page migration that is skipped because it is a shared fault.

This reduces the number of page migrations back and forth in shared fault situations. It also gives a strong preference to the tasks that are already running where most of the memory is, and to moving the other tasks to near the memory.

Testing this with a much higher scan rate than the default still seems to result in fewer page migrations than before.

Memory seems to be somewhat better consolidated than previously, with multi-instance specjbb runs on a 4 node system.

---
 Documentation/sysctl/kernel.txt | 10 ++++++++-
 include/linux/sched.h           |  5 ++++-
 kernel/sched/fair.c             |  8 +++++++
 kernel/sysctl.c                 |  7 ++++++
 mm/mempolicy.c                  | 48 ++++++++++++++++++++++++++++++++++++++++-
 5 files changed, 75 insertions(+), 3 deletions(-)
Documentation/sysctl/kernel.txt +9 -1
index 0d503df..a767ef2 100644
@@ -374,7 +374,8 @@ feature should be disabled. Otherwise, if the system overhead from the
 feature is too high then the rate the kernel samples for NUMA hinting
 faults may be controlled by the numa_balancing_scan_period_min_ms,
 numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms,
-numa_balancing_scan_size_mb and numa_balancing_settle_count sysctls.
+numa_balancing_scan_size_mb, numa_balancing_settle_count sysctls and
+numa_balancing_migrate_deferred.
 
 ==============================================================
 
@@ -420,6 +421,13 @@ the schedule balancer stops pushing the task towards a preferred node. This
 gives the scheduler a chance to place the task on an alternative node if the
 preferred node is overloaded.
 
+numa_balancing_migrate_deferred is how many page migrations get skipped
+unconditionally, after a page migration is skipped because a page is shared
+with other tasks. This reduces page migration overhead, and determines
+how much stronger the "move task near its memory" policy scheduler becomes,
+versus the "move memory near its task" memory management policy, for workloads
+with shared memory.
+
 ==============================================================
 
 osrelease, ostype & version:
include/linux/sched.h +4 -1
index 33c53d6..48c4947 100644
@@ -1333,6 +1333,8 @@ struct task_struct {
 	int numa_scan_seq;
 	unsigned int numa_scan_period;
 	unsigned int numa_scan_period_max;
+	int numa_preferred_nid;
+	int numa_migrate_deferred;
 	unsigned long numa_migrate_retry;
 	u64 node_stamp;			/* migration stamp  */
 	struct callback_head numa_work;
@@ -1363,7 +1365,6 @@ struct task_struct {
 	 */
 	unsigned long numa_faults_locality[2];
 
-	int numa_preferred_nid;
 	unsigned long numa_pages_migrated;
 #endif /* CONFIG_NUMA_BALANCING */
 
@@ -1453,6 +1454,8 @@ extern void task_numa_fault(int last_node, int node, int pages, int flags);
 extern pid_t task_numa_group_id(struct task_struct *p);
 extern void set_numabalancing_state(bool enabled);
 extern void task_numa_free(struct task_struct *p);
+
+extern unsigned int sysctl_numa_balancing_migrate_deferred;
 #else
 static inline void task_numa_fault(int last_node, int node, int pages,
 				   int flags)
kernel/sched/fair.c +8 -0
index 9bca073..c00611c 100644
@@ -833,6 +833,14 @@ unsigned int sysctl_numa_balancing_scan_size = 256;
 /* Scan @scan_size MB every @scan_period after an initial @scan_delay in ms */
 unsigned int sysctl_numa_balancing_scan_delay = 1000;
 
+/*
+ * After skipping a page migration on a shared page, skip N more numa page
+ * migrations unconditionally. This reduces the number of NUMA migrations
+ * in shared memory workloads, and has the effect of pulling tasks towards
+ * where their memory lives, over pulling the memory towards the task.
+ */
+unsigned int sysctl_numa_balancing_migrate_deferred = 16;
+
 static unsigned int task_nr_scan_windows(struct task_struct *p)
 {
 	unsigned long rss = 0;
kernel/sysctl.c +7 -0
index 4e080fe..92bb38a 100644
@@ -391,6 +391,13 @@ static struct ctl_table kern_table[] = {
 		.mode           = 0644,
 		.proc_handler   = proc_dointvec,
 	},
+	{
+		.procname       = "numa_balancing_migrate_deferred",
+		.data           = &sysctl_numa_balancing_migrate_deferred,
+		.maxlen         = sizeof(unsigned int),
+		.mode           = 0644,
+		.proc_handler   = proc_dointvec,
+	},
 #endif /* CONFIG_NUMA_BALANCING */
 #endif /* CONFIG_SCHED_DEBUG */
 	{
mm/mempolicy.c +47 -1

60 lines not on this page · raw message

Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#66 lore raw

PATCH 62/63 sched: numa: use unsigned longs for numa group fault stats

In reply to Mel Gorman

As Peter says "If you're going to hold locks you can also do away with all that atomic_long_*() nonsense". Lock aquisition moved slightly to protect the updates.

---
 kernel/sched/fair.c | 49 ++++++++++++++++++++-----------------------------
 1 file changed, 20 insertions(+), 29 deletions(-)
kernel/sched/fair.c +20 -29
index c00611c..fced601 100644
@@ -916,8 +916,8 @@ struct numa_group {
 	struct list_head task_list;
 
 	struct rcu_head rcu;
-	atomic_long_t total_faults;
-	atomic_long_t faults[0];
+	unsigned long total_faults;
+	unsigned long faults[0];
 };
 
 pid_t task_numa_group_id(struct task_struct *p)
@@ -944,8 +944,7 @@ static inline unsigned long group_faults(struct task_struct *p, int nid)
 	if (!p->numa_group)
 		return 0;
 
-	return atomic_long_read(&p->numa_group->faults[2*nid]) +
-	       atomic_long_read(&p->numa_group->faults[2*nid+1]);
+	return p->numa_group->faults[2*nid] + p->numa_group->faults[2*nid+1];
 }
 
 /*
@@ -971,17 +970,10 @@ static inline unsigned long task_weight(struct task_struct *p, int nid)
 
 static inline unsigned long group_weight(struct task_struct *p, int nid)
 {
-	unsigned long total_faults;
-
-	if (!p->numa_group)
-		return 0;
-
-	total_faults = atomic_long_read(&p->numa_group->total_faults);
-
-	if (!total_faults)
+	if (!p->numa_group || !p->numa_group->total_faults)
 		return 0;
 
-	return 1000 * group_faults(p, nid) / total_faults;
+	return 1000 * group_faults(p, nid) / p->numa_group->total_faults;
 }
 
 static unsigned long weighted_cpuload(const int cpu);
@@ -1397,9 +1389,9 @@ static void task_numa_placement(struct task_struct *p)
 			p->total_numa_faults += diff;
 			if (p->numa_group) {
 				/* safe because we can only change our own group */
-				atomic_long_add(diff, &p->numa_group->faults[i]);
-				atomic_long_add(diff, &p->numa_group->total_faults);
-				group_faults += atomic_long_read(&p->numa_group->faults[i]);
+				p->numa_group->faults[i] += diff;
+				p->numa_group->total_faults += diff;
+				group_faults += p->numa_group->faults[i];
 			}
 		}
 
@@ -1475,7 +1467,7 @@ static void task_numa_group(struct task_struct *p, int cpupid, int flags,
 
 	if (unlikely(!p->numa_group)) {
 		unsigned int size = sizeof(struct numa_group) +
-				    2*nr_node_ids*sizeof(atomic_long_t);
+				    2*nr_node_ids*sizeof(unsigned long);
 
 		grp = kzalloc(size, GFP_KERNEL | __GFP_NOWARN);
 		if (!grp)
@@ -1487,9 +1479,9 @@ static void task_numa_group(struct task_struct *p, int cpupid, int flags,
 		grp->gid = p->pid;
 
 		for (i = 0; i < 2*nr_node_ids; i++)
-			atomic_long_set(&grp->faults[i], p->numa_faults[i]);
+			grp->faults[i] = p->numa_faults[i];
 
-		atomic_long_set(&grp->total_faults, p->total_numa_faults);
+		grp->total_faults = p->total_numa_faults;
 
 		list_add(&p->numa_entry, &grp->task_list);
 		grp->nr_tasks++;
@@ -1543,14 +1535,14 @@ unlock:
 	if (!join)
 		return;
 
+	double_lock(&my_grp->lock, &grp->lock);
+
 	for (i = 0; i < 2*nr_node_ids; i++) {
-		atomic_long_sub(p->numa_faults[i], &my_grp->faults[i]);
-		atomic_long_add(p->numa_faults[i], &grp->faults[i]);
+		my_grp->faults[i] -= p->numa_faults[i];
+		grp->faults[i] += p->numa_faults[i];
 	}
-	atomic_long_sub(p->total_numa_faults, &my_grp->total_faults);
-	atomic_long_add(p->total_numa_faults, &grp->total_faults);
-
-	double_lock(&my_grp->lock, &grp->lock);
+	my_grp->total_faults -= p->total_numa_faults;
+	grp->total_faults += p->total_numa_faults;
 
 	list_move(&p->numa_entry, &grp->task_list);
 	my_grp->nr_tasks--;
@@ -1571,12 +1563,11 @@ void task_numa_free(struct task_struct *p)
 	void *numa_faults = p->numa_faults;
 
 	if (grp) {
+		spin_lock(&grp->lock);
 		for (i = 0; i < 2*nr_node_ids; i++)
-			atomic_long_sub(p->numa_faults[i], &grp->faults[i]);
-
-		atomic_long_sub(p->total_numa_faults, &grp->total_faults);
+			grp->faults[i] -= p->numa_faults[i];
+		grp->total_faults -= p->total_numa_faults;
 
-		spin_lock(&grp->lock);
 		list_del(&p->numa_entry);
 		grp->nr_tasks--;
 		spin_unlock(&grp->lock);
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author reviewer
#67 lore raw

PATCH 63/63 sched: numa: periodically retry task_numa_migrate

In reply to Mel Gorman

From: Rik van Riel <[email protected]>

Short spikes of CPU load can lead to a task being migrated away from its preferred node for temporary reasons.

It is important that the task is migrated back to where it belongs, in order to avoid migrating too much memory to its new location, and generally disturbing a task's NUMA location.

This patch fixes NUMA placement for 4 specjbb instances on a 4 node system. Without this patch, things take longer to converge, and processes are not always completely on their own node.

---
 kernel/sched/fair.c | 22 +++++++++++++---------
 1 file changed, 13 insertions(+), 9 deletions(-)
kernel/sched/fair.c +13 -9
index fced601..8fe5789 100644
@@ -1259,18 +1259,19 @@ static int task_numa_migrate(struct task_struct *p)
 /* Attempt to migrate a task to a CPU on the preferred node. */
 static void numa_migrate_preferred(struct task_struct *p)
 {
-	/* Success if task is already running on preferred CPU */
-	p->numa_migrate_retry = 0;
-	if (cpu_to_node(task_cpu(p)) == p->numa_preferred_nid)
+	/* This task has no NUMA fault statistics yet */
+	if (unlikely(p->numa_preferred_nid == -1 || !p->numa_faults))
 		return;
 
-	/* This task has no NUMA fault statistics yet */
-	if (unlikely(p->numa_preferred_nid == -1))
+	/* Periodically retry migrating the task to the preferred node */
+	p->numa_migrate_retry = jiffies + HZ;
+
+	/* Success if task is already running on preferred CPU */
+	if (cpu_to_node(task_cpu(p)) == p->numa_preferred_nid)
 		return;
 
 	/* Otherwise, try migrate to a CPU on the preferred node */
-	if (task_numa_migrate(p) != 0)
-		p->numa_migrate_retry = jiffies + HZ*5;
+	task_numa_migrate(p);
 }
 
 /*
@@ -1629,8 +1630,11 @@ void task_numa_fault(int last_cpupid, int node, int pages, int flags)
 
 	task_numa_placement(p);
 
-	/* Retry task to preferred node migration if it previously failed */
-	if (p->numa_migrate_retry && time_after(jiffies, p->numa_migrate_retry))
+	/*
+	 * Retry task to preferred node migration periodically, in case it
+	 * case it previously failed, or the scheduler moved us.
+	 */
+	if (time_after(jiffies, p->numa_migrate_retry))
 		numa_migrate_preferred(p);
 
 	if (migrated)
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#68 lore raw
In reply to Mel Gorman
Quoted 9 lines · Mel Gorman <mgorman <at> suse.de> writes:

Weighing in at 63 patches makes the term "basic" in the series title is a misnomer.

This series still has roughly the same goals as previous versions. It reduces overhead of automatic balancing through scan rate reduction and the avoidance of TLB flushes. It selects a preferred node and moves tasks towards their memory as well as moving memory toward their task. It handles shared pages and groups related tasks together. Some problems

such

Quoted 5 lines

as shared page interleaving and properly dealing with processes that are larger than a node are being deferred.

It is still based on 3.11 because that's what I was testing against. If we can agree this should be merged to -tip for testing I'll rebase to

deal

Quoted 1 line

with any scheduler conflicts but for now I do not want to invalidate

other

Quoted 1 line

peoples testing. The only obvious thing that is missing is hotplug

handling.

Quoted 17 lines

Peter is currently working on reducing [get|put]_online_cpus overhead so that it can be added to migrate_swap.

Peter, some of your patches are missing signed-offs-by -- 4-5, 43 and 55.

Changelog since V7
o THP migration race and pagetable insertion fixes
o Do no handle PMDs in batch
o Shared page migration throttling
o Various changes to how last nid/pid information is recorded
o False pid match sanity checks when joining NUMA task groups
o Adapt scan rate based on local/remote fault statistics
o Period retry of migration to preferred node
o Limit scope of system-wide search
o Schedule threads on the same node as process that created them
o Cleanup numa_group on exec

Quoted 3 lines

Changelog since V6
o Group tasks that share pages together
o More scan avoidance of VMAs mapping pages that are not likely to

migrate

Quoted 19 lines

o cpunid conversion, system-wide searching of tasks to balance with

Changelog since V6
o Various TLB flush optimisations
o Comment updates
o Sanitise task_numa_fault callsites for consistent semantics
o Revert some of the scanning adaption stuff
o Revert patch that defers scanning until task schedules on another node
o Start delayed scanning properly
o Avoid the same task always performing the PTE scan
o Continue PTE scanning even if migration is rate limited

Changelog since V5
o Add __GFP_NOWARN for numa hinting fault count
o Use is_huge_zero_page
o Favour moving tasks towards nodes with higher faults
o Optionally resist moving tasks towards nodes with lower faults
o Scan shared THP pages

Quoted 3 lines

Changelog since V4
o Added code that avoids overloading preferred nodes
o Swap tasks if nodes are overloaded and the swap does not impair

locality

Quoted 4 lines

Changelog since V3
o Correct detection of unset last nid/pid information
o Dropped nr_preferred_running and replaced it with Peter's load

balancing

Quoted 10 lines

o Pass in correct node information for THP hinting faults
o Pressure tasks sharing a THP page to move towards same node
o Do not set pmd_numa if false sharing is detected

Changelog since V2
o Reshuffle to match Peter's implied preference for layout
o Reshuffle to move private/shared split towards end of series to make it easier to evaluate the impact
o Use PID information to identify private accesses
o Set the floor for PTE scanning based on virtual address space scan

rates

Quoted 14 lines

instead of time
o Some locking improvements
o Do not preempt pinned tasks unless they are kernel threads

Changelog since V1
o Scan pages with elevated map count (shared pages)
o Scale scan rates based on the vsz of the process so the sampling of the task is independant of its size
o Favour moving towards nodes with more faults even if it's not the preferred node
o Laughably basic accounting of a compute overloaded node when selecting the preferred node.
o Applied review comments

Quoted 1 line

This series integrates basic scheduler support for automatic NUMA

balancing.

Quoted 5 lines

It was initially based on Peter Ziljstra's work in "sched, numa, mm:
Add adaptive NUMA affinity support" but deviates too much to preserve Signed-off-bys. As before, if the relevant authors are ok with it I'll add Signed-off-bys (or add them yourselves if you pick the patches up). There has been a tonne of additional work from both Peter and Rik van

Riel.

Quoted 4 lines

Some reports indicate that the performance is getting close to manual bindings for some workloads but your mileage will vary. As before, the intention is not to complete the work but to incrementally improve

mainline

Quoted 3 lines

and preserve bisectability for any bug reports that crop up.

Patch 1 is a monolithic dump of patches thare are destined for upstream

that

Quoted 5 lines

this series indirectly depends upon.

Patches 2-3 adds sysctl documentation and comment fixlets

Patch 4 avoids accounting for a hinting fault if another thread handled

the

Quoted 9 lines

fault in parallel

Patches 5-6 avoid races with parallel THP migration and THP splits.

Patch 7 corrects a THP NUMA hint fault accounting bug

Patches 8-9 avoids TLB flushes during the PTE scan if no updates are made

Patch 10 sanitizes task_numa_fault callsites to have consist semantics

and

Quoted 7 lines

always record the fault based on the correct location of the page.

Patch 11 closes races between THP migration and PMD clearing.

Patch 12 avoids trying to migrate the THP zero page

Patch 13 avoids the same task being selected to perform the PTE scan

within

Quoted 9 lines

a shared address space.

Patch 14 continues PTE scanning even if migration rate limited

Patch 15 notes that delaying the PTE scan until a task is scheduled on an
alternative node misses the case where the task is only accessing shared memory on a partially loaded machine and reverts a patch.

Patch 16 initialises numa_next_scan properly so that PTE scanning is

delayed

Quoted 6 lines

when a process starts.

Patch 17 sets the scan rate proportional to the size of the task being
scanned.

Patch 18 slows the scan rate if no hinting faults were trapped by an

idle task.

Quoted 4 lines

Patch 19 tracks NUMA hinting faults per-task and per-node

Patches 20-24 selects a preferred node at the end of a PTE scan based on

what

Quoted 11 lines

node incurred the highest number of NUMA faults. When the balancer is comparing two CPU it will prefer to locate tasks on their preferred node. When initially selected the task is rescheduled on the preferred node if it is not running on that node already. This avoids waiting for the scheduler to move the task slowly.

Patch 25 adds infrastructure to allow separate tracking of shared/private
pages but treats all faults as if they are private accesses. Laying it out this way reduces churn later in the series when private fault detection is introduced

Quoted 6 lines

Patch 26 avoids some unnecessary allocation

Patch 27-28 kicks away some training wheels and scans shared pages and
small VMAs.

Patch 29 introduces private fault detection based on the PID of the

faulting

Quoted 3 lines

process and accounts for shared/private accesses differently.

Patch 30 avoids migrating memory immediately after the load balancer

moves

Quoted 3 lines

a task to another node in case it's a transient migration.

Patch 31 avoids scanning VMAs that do not migrate-on-fault which

addresses

Quoted 9 lines

a serious regression on a database performance test.

Patch 32 pick the least loaded CPU based on a preferred node based on
a scheduling domain common to both the source and destination NUMA node.

Patch 33 retries task migration if an earlier attempt failed

Patch 34 will begin task migration immediately if running on its

preferred

Quoted 6 lines

node

Patch 35 will avoid trapping hinting faults for shared read-only library
pages as these never migrate anyway

Patch 36 avoids handling pmd hinting faults if none of the ptes below it

were

Quoted 8 lines

marked pte numa

Patches 37-38 introduce a mechanism for swapping tasks

Patch 39 uses a system-wide search to find tasks that can be swapped
to improve the overall locality of the system.

Patch 40 notes that the system-wide search may ignore the preferred node

and

Quoted 4 lines

will use the preferred node placement if it has spare compute capacity.

Patch 41 will perform a global search if a node that should have had

capacity

Quoted 18 lines

cannot have a task migrated to it

Patches 42-43 use cpupid to track pages so potential sharing tasks can
be quickly found

Patch 44 reports the ID of the numa group a task belongs.

Patch 45 copies the cpupid on page migration

Patch 46 avoids grouping based on read-only pages

Patch 47 stops handling pages within a PMD in batch as it distorts fault
statistics and failed to flush TLBs correctly.

Patch 48 schedules new threads on the same node as the parent.

Patch 49 schedules tasks based on their numa group

Quoted 7 lines

Patch 50 cleans up tasks numa_group on exec

Patch 51 avoids parallel updates to group stats

Patch 52 adds some debugging aids

Patches 53-54 separately considers task and group weights when selecting

the node to

Quoted 3 lines

schedule a task on

Patch 56 checks if PID truncation may have caused false matches before

joining tasks

Quoted 3 lines

to a NUMA grou

Patch 57 uses the false shared detection information for scan rate

adaption later

Quoted 13 lines

Patch 58 adapts the scan rate based on local/remote faults

Patch 59 removes the period scan rate reset

Patch 60-61 throttles shared page migrations

Patch 62 avoids the use of atomics protects the values with a spinlock

Patch 63 periodically retries migrating a task back to its preferred node

Kernel 3.11 is the testing baseline.

Quoted 7 lines

o account-v8 Patches 1-8
o falsedetect-v8 Patches 1-57
o skipshared-v8 Patches 1-61
o noatomics-v8 Patches 1-62
o periodretry-v8 PAtches 1-63

This is SpecJBB running on a 4-socket machine with THP enabled and one

JVM

Quoted 4 lines

running for the whole system.

specjbb

                       3.11.0                3.11.0                

3.11.0 3.11.0 3.11.0

Quoted 1 line

account-v8r20 falsedetect-v8r33 skipshared-

v8r33 noatomics-v8r33 periodretry-v8r33

Quoted 1 line

TPut 1 26137.00 ( 0.00%) 26879.00 ( 2.84%) 26296.00 (

0.61%) 26563.00 ( 1.63%) 26043.00 ( -0.36%)

Quoted 1 line

TPut 2 56027.00 ( 0.00%) 55871.00 ( -0.28%) 55336.00 ( -

1.23%) 56325.00 ( 0.53%) 53021.00 ( -5.37%)

Quoted 1 line

TPut 3 84375.00 ( 0.00%) 86268.00 ( 2.24%) 82832.00 ( -

1.83%) 86164.00 ( 2.12%) 82818.00 ( -1.85%)

Quoted 1 line

TPut 4 114968.00 ( 0.00%) 115229.00 ( 0.23%) 108632.00 ( -

5.51%) 110348.00 ( -4.02%) 106984.00 ( -6.94%)

Quoted 1 line

TPut 5 139786.00 ( 0.00%) 136641.00 ( -2.25%) 131027.00 ( -

6.27%) 139978.00 ( 0.14%) 139000.00 ( -0.56%)

Quoted 1 line

TPut 6 168084.00 ( 0.00%) 167331.00 ( -0.45%) 159409.00 ( -

5.16%) 164444.00 ( -2.17%) 157117.00 ( -6.52%)

Quoted 1 line

TPut 7 196709.00 ( 0.00%) 194751.00 ( -1.00%) 182849.00 ( -

7.05%) 192798.00 ( -1.99%) 184336.00 ( -6.29%)

Quoted 1 line

TPut 8 217119.00 ( 0.00%) 220303.00 ( 1.47%) 211978.00 ( -

2.37%) 215026.00 ( -0.96%) 213757.00 ( -1.55%)

Quoted 1 line

TPut 9 240359.00 ( 0.00%) 244058.00 ( 1.54%) 237005.00 ( -

1.40%) 240441.00 ( 0.03%) 236827.00 ( -1.47%)

Quoted 1 line

TPut 10 263615.00 ( 0.00%) 265787.00 ( 0.82%) 260203.00 ( -

1.29%) 260737.00 ( -1.09%) 261361.00 ( -0.86%)

Quoted 1 line

TPut 11 292052.00 ( 0.00%) 293012.00 ( 0.33%) 294855.00 (

0.96%) 295295.00 ( 1.11%) 288431.00 ( -1.24%)

Quoted 1 line

TPut 12 306763.00 ( 0.00%) 318380.00 ( 3.79%) 317420.00 (

3.47%) 317958.00 ( 3.65%) 310027.00 ( 1.06%)

Quoted 1 line

TPut 13 336282.00 ( 0.00%) 338831.00 ( 0.76%) 342010.00 (

1.70%) 339773.00 ( 1.04%) 334431.00 ( -0.55%)

Quoted 1 line

TPut 14 359216.00 ( 0.00%) 365707.00 ( 1.81%) 366968.00 (

2.16%) 371908.00 ( 3.53%) 354059.00 ( -1.44%)

Quoted 1 line

TPut 15 394379.00 ( 0.00%) 382206.00 ( -3.09%) 372864.00 ( -

5.46%) 389394.00 ( -1.26%) 378397.00 ( -4.05%)

Quoted 1 line

TPut 16 391503.00 ( 0.00%) 394595.00 ( 0.79%) 409349.00 (

4.56%) 410242.00 ( 4.79%) 396774.00 ( 1.35%)

Quoted 1 line

TPut 17 418193.00 ( 0.00%) 425690.00 ( 1.79%) 427327.00 (

2.18%) 411347.00 ( -1.64%) 418920.00 ( 0.17%)

Quoted 1 line

TPut 18 445409.00 ( 0.00%) 440833.00 ( -1.03%) 447030.00 (

0.36%) 451240.00 ( 1.31%) 440823.00 ( -1.03%)

Quoted 1 line

TPut 19 447209.00 ( 0.00%) 455059.00 ( 1.76%) 459879.00 (

2.83%) 464383.00 ( 3.84%) 455354.00 ( 1.82%)

Quoted 1 line

TPut 20 462761.00 ( 0.00%) 472095.00 ( 2.02%) 479006.00 (

3.51%) 473824.00 ( 2.39%) 471917.00 ( 1.98%)

Quoted 1 line

TPut 21 481207.00 ( 0.00%) 485035.00 ( 0.80%) 491456.00 (

2.13%) 487685.00 ( 1.35%) 489372.00 ( 1.70%)

Quoted 1 line

TPut 22 495510.00 ( 0.00%) 492799.00 ( -0.55%) 502316.00 (

1.37%) 500868.00 ( 1.08%) 484667.00 ( -2.19%)

Quoted 1 line

TPut 23 496669.00 ( 0.00%) 486543.00 ( -2.04%) 518198.00 (

4.33%) 507320.00 ( 2.14%) 507332.00 ( 2.15%)

Quoted 1 line

TPut 24 519722.00 ( 0.00%) 513153.00 ( -1.26%) 521337.00 (

0.31%) 524108.00 ( 0.84%) 516621.00 ( -0.60%)

Quoted 1 line

TPut 25 516030.00 ( 0.00%) 501461.00 ( -2.82%) 507013.00 ( -

1.75%) 519228.00 ( 0.62%) 522712.00 ( 1.29%)

Quoted 1 line

TPut 26 521824.00 ( 0.00%) 499423.00 ( -4.29%) 522584.00 (

0.15%) 529076.00 ( 1.39%) 512817.00 ( -1.73%)

Quoted 1 line

TPut 27 514817.00 ( 0.00%) 489511.00 ( -4.92%) 503722.00 ( -

2.16%) 520135.00 ( 1.03%) 524537.00 ( 1.89%)

Quoted 1 line

TPut 28 505838.00 ( 0.00%) 485151.00 ( -4.09%) 499372.00 ( -

1.28%) 514423.00 ( 1.70%) 525086.00 ( 3.81%)

Quoted 1 line

TPut 29 503231.00 ( 0.00%) 496580.00 ( -1.32%) 503838.00 (

0.12%) 520836.00 ( 3.50%) 526196.00 ( 4.56%)

Quoted 1 line

TPut 30 487832.00 ( 0.00%) 512245.00 ( 5.00%) 506440.00 (

3.81%) 521409.00 ( 6.88%) 526048.00 ( 7.83%)

Quoted 1 line

TPut 31 492896.00 ( 0.00%) 520700.00 ( 5.64%) 513648.00 (

4.21%) 514615.00 ( 4.41%) 525456.00 ( 6.61%)

Quoted 1 line

TPut 32 499692.00 ( 0.00%) 514756.00 ( 3.01%) 487157.00 ( -

2.51%) 510817.00 ( 2.23%) 518296.00 ( 3.72%)

Quoted 1 line

TPut 33 494054.00 ( 0.00%) 514193.00 ( 4.08%) 499706.00 (

1.14%) 509734.00 ( 3.17%) 508643.00 ( 2.95%)

Quoted 1 line

TPut 34 483213.00 ( 0.00%) 512204.00 ( 6.00%) 470473.00 ( -

2.64%) 495556.00 ( 2.55%) 510309.00 ( 5.61%)

Quoted 1 line

TPut 35 480068.00 ( 0.00%) 506472.00 ( 5.50%) 473011.00 ( -

1.47%) 481812.00 ( 0.36%) 513299.00 ( 6.92%)

Quoted 1 line

TPut 36 450114.00 ( 0.00%) 506894.00 ( 12.61%) 477324.00 (

6.05%) 469524.00 ( 4.31%) 486779.00 ( 8.15%)

Quoted 1 line

TPut 37 440154.00 ( 0.00%) 506266.00 ( 15.02%) 459718.00 (

4.44%) 484438.00 ( 10.06%) 508613.00 ( 15.55%)

Quoted 1 line

TPut 38 461536.00 ( 0.00%) 499960.00 ( 8.33%) 460417.00 ( -

0.24%) 473332.00 ( 2.56%) 514863.00 ( 11.55%)

Quoted 1 line

TPut 39 460596.00 ( 0.00%) 503353.00 ( 9.28%) 461793.00 (

0.26%) 468966.00 ( 1.82%) 498496.00 ( 8.23%)

Quoted 1 line

TPut 40 450746.00 ( 0.00%) 497059.00 ( 10.27%) 440518.00 ( -

2.27%) 446687.00 ( -0.90%) 497462.00 ( 10.36%)

Quoted 1 line

TPut 41 456565.00 ( 0.00%) 482549.00 ( 5.69%) 451770.00 ( -

1.05%) 444771.00 ( -2.58%) 491267.00 ( 7.60%)

Quoted 1 line

TPut 42 447258.00 ( 0.00%) 490184.00 ( 9.60%) 445414.00 ( -

0.41%) 452870.00 ( 1.25%) 469483.00 ( 4.97%)

Quoted 1 line

TPut 43 408979.00 ( 0.00%) 467385.00 ( 14.28%) 416133.00 (

1.75%) 448137.00 ( 9.57%) 486668.00 ( 19.00%)

Quoted 1 line

TPut 44 415774.00 ( 0.00%) 460710.00 ( 10.81%) 412956.00 ( -

0.68%) 445221.00 ( 7.08%) 479664.00 ( 15.37%)

Quoted 1 line

TPut 45 419682.00 ( 0.00%) 445293.00 ( 6.10%) 411016.00 ( -

2.06%) 426650.00 ( 1.66%) 464358.00 ( 10.65%)

Quoted 1 line

TPut 46 422698.00 ( 0.00%) 425270.00 ( 0.61%) 396870.00 ( -

6.11%) 430955.00 ( 1.95%) 460729.00 ( 9.00%)

Quoted 1 line

TPut 47 416622.00 ( 0.00%) 395166.00 ( -5.15%) 403654.00 ( -

3.11%) 397989.00 ( -4.47%) 463184.00 ( 11.18%)

Quoted 1 line

TPut 48 409081.00 ( 0.00%) 382490.00 ( -6.50%) 377057.00 ( -

7.83%) 391214.00 ( -4.37%) 462186.00 ( 12.98%)

Quoted 6 lines

Mostly flat but it's interesting to note what a large impact the periodic migration of tasks to their preferred node is.

specjbb Peaks

                                    3.11.0                   

3.11.0 3.11.0 3.11.0
3.11.0

Quoted 1 line

account-v8r20 falsedetect-

v8r33 skipshared-v8r33 noatomics-v8r33 periodretry-
v8r33

Quoted 1 line

Expctd Warehouse 48.00 ( 0.00%) 48.00 (

0.00%) 48.00 ( 0.00%) 48.00 ( 0.00%) 48.00 (
0.00%)

Quoted 1 line

Expctd Peak Bops 416622.00 ( 0.00%) 395166.00 ( -5.15%)

403654.00 ( -3.11%) 397989.00 ( -4.47%) 463184.00

Quoted 2 lines

( 11.18%)
Actual Warehouse 27.00 ( 0.00%) 32.00 (

18.52%) 27.00 ( 0.00%) 27.00 ( 0.00%) 30.00 (
11.11%)

Quoted 1 line

Actual Peak Bops 521824.00 ( 0.00%) 520700.00 ( -0.22%)

522584.00 ( 0.15%) 529076.00 ( 1.39%) 526196.00 ( 0.84%)

Quoted 1 line

SpecJBB Bops 8349.00 ( 0.00%) 7806.00 ( -

6.50%) 7695.00 ( -7.83%) 7984.00 ( -4.37%) 9432.00 (
12.97%)

Quoted 1 line

SpecJBB Bops/JVM 8349.00 ( 0.00%) 7806.00 ( -

6.50%) 7695.00 ( -7.83%) 7984.00 ( -4.37%) 9432.00 (
12.97%)

Quoted 5 lines

Same.

              3.11.0      3.11.0      3.11.0      3.11.0      3.11.0
        account-v8r20falsedetect-v8r33skipshared-v8r33noatomics-

v8r33periodretry-v8r33

Quoted 7 lines

User 43797.04 44575.08 44571.62 44544.22 44591.39
System 863.95 135.20 151.93 137.42 165.06
Elapsed 1665.51 1665.06 1665.30 1665.22 1665.06

Big reduction in system CPU usage.

                                3.11.0      3.11.0      3.11.0      

3.11.0 3.11.0

Quoted 1 line

account-v8r20falsedetect-v8r33skipshared-

v8r33noatomics-v8r33periodretry-v8r33

Quoted 1 line

Compaction cost 135535 13678 17594

15600 13820

Quoted 1 line

NUMA PTE updates 19815264 697698 3495802

3195079 3668267

Quoted 1 line

NUMA hint faults 4029907 694556 3471637

3167057 3638629

Quoted 1 line

NUMA hint local faults 1109183 239859 985693

894926 1148882

Quoted 1 line

NUMA hint local percent 27 34 28

28 31

Quoted 1 line

NUMA pages migrated 130574018 13177453 16950408

15029583 13314876

Quoted 1 line

AutoNUMA cost 22769 3728 17704

16143 18471

Quoted 2 lines

Big reduction in the number of PTE updates but similar-ish fault

statistics. We're getting roughly

Quoted 4 lines

the same benefit for much less work.

autonumabench

                                         3.11.0                

3.11.0 3.11.0 3.11.0 3.11.0

Quoted 1 line

account-v8r20 falsedetect-

v8r33 skipshared-v8r33 noatomics-v8r33 periodretry-v8r33

Quoted 1 line

User NUMA01 44031.89 ( 0.00%) 60916.77 (-38.35%)

62988.76 (-43.05%) 62187.81 (-41.23%) 35403.01 ( 19.60%)

Quoted 1 line

User NUMA01_THEADLOCAL 16956.40 ( 0.00%) 17665.21 ( -4.18%)

17809.92 ( -5.03%) 17605.32 ( -3.83%)

Quoted 2 lines

17238.74 ( -1.67%)
User NUMA02 2044.83 ( 0.00%) 2054.14 ( -

0.46%)     2106.18 ( -3.00%)     2058.25 ( -0.66%)     2065.25 ( -1.00%)
> User    NUMA02_SMT             982.52 (  0.00%)      978.55 (  
0.40%)      985.73 ( -0.33%)      975.96 (  0.67%)     1001.48 ( -1.93%)
> System  NUMA01                1038.02 (  0.00%)      320.44 ( 

69.13%) 440.97 ( 57.52%) 179.55 ( 82.70%) 333.93 ( 67.83%)

Quoted 1 line

System NUMA01_THEADLOCAL 323.01 ( 0.00%) 125.90 (

61.02%) 132.59 ( 58.95%) 123.52 ( 61.76%) 171.04 ( 47.05%)

Quoted 1 line

System NUMA02 10.26 ( 0.00%) 11.34 (-

10.53%) 9.97 ( 2.83%) 9.07 ( 11.60%) 6.08 ( 40.74%)

Quoted 1 line

System NUMA02_SMT 3.34 ( 0.00%) 3.43 ( -

2.69%) 5.45 (-63.17%) 3.42 ( -2.40%) 3.40 ( -1.80%)

Quoted 1 line

Elapsed NUMA01 999.90 ( 0.00%) 1363.26 (-

36.34%) 1409.55 (-40.97%) 1386.34 (-38.65%) 792.81 ( 20.71%)

Quoted 1 line

Elapsed NUMA01_THEADLOCAL 374.82 ( 0.00%) 395.48 ( -

5.51%)      399.18 ( -6.50%)      393.90 ( -5.09%)      388.22 ( -3.58%)
> Elapsed NUMA02                  48.66 (  0.00%)       50.15 ( -

3.06%) 52.75 ( -8.41%) 50.81 ( -4.42%) 48.48 ( 0.37%)

Quoted 1 line

Elapsed NUMA02_SMT 46.17 ( 0.00%) 46.72 ( -

1.19%) 46.51 ( -0.74%) 45.98 ( 0.41%) 57.64 (-24.84%)

Quoted 1 line

CPU NUMA01 4507.00 ( 0.00%) 4491.00 (

0.36%) 4499.00 ( 0.18%) 4498.00 ( 0.20%) 4507.00 ( 0.00%)

Quoted 1 line

CPU NUMA01_THEADLOCAL 4609.00 ( 0.00%) 4498.00 (

2.41%) 4494.00 ( 2.50%) 4500.00 ( 2.36%) 4484.00 ( 2.71%)

Quoted 1 line

CPU NUMA02 4222.00 ( 0.00%) 4117.00 (

2.49%) 4011.00 ( 5.00%) 4068.00 ( 3.65%) 4272.00 ( -1.18%)

Quoted 1 line

CPU NUMA02_SMT 2135.00 ( 0.00%) 2101.00 (

1.59%) 2130.00 ( 0.23%) 2129.00 ( 0.28%) 1743.00 ( 18.36%)

Quoted 4 lines

In general, the numa01 (adverse workload) is hurt by the series until the periodic retry patch is merged. In general that patch helps a lot except in the case of numa02_SMT where it punished severely. It's odd because

the

Quoted 3 lines

amount of system CPU usage that workload incurs is tiny.

This is SpecJBB running on a 4-socket machine with THP enabled and one

JVM

Quoted 4 lines

running per node on the system.

specjbb

                         3.11.0                3.11.0                

3.11.0 3.11.0 3.11.0

Quoted 1 line

account-v8r20 falsedetect-v8r33 skipshared-

v8r33 noatomics-v8r33 periodretry-v8r33

Quoted 1 line

Mean 1 31915.75 ( 0.00%) 30007.50 ( -5.98%) 29590.50 ( -

7.29%) 30562.75 ( -4.24%) 28888.75 ( -9.48%)

Quoted 1 line

Mean 2 61914.50 ( 0.00%) 60854.00 ( -1.71%) 62163.25 (

0.40%) 60594.25 ( -2.13%) 62521.75 ( 0.98%)

Quoted 1 line

Mean 3 89693.25 ( 0.00%) 91159.25 ( 1.63%) 91939.00 (

2.50%) 93235.50 ( 3.95%) 94956.75 ( 5.87%)

Quoted 1 line

Mean 4 115687.50 ( 0.00%) 117438.00 ( 1.51%) 119940.50 (

3.68%) 124273.75 ( 7.42%) 124600.75 ( 7.70%)

Quoted 1 line

Mean 5 142789.25 ( 0.00%) 141024.00 ( -1.24%) 146349.00 (

2.49%) 152133.25 ( 6.54%) 150575.00 ( 5.45%)

Quoted 1 line

Mean 6 153345.25 ( 0.00%) 160091.25 ( 4.40%) 160991.00 (

4.99%) 174128.50 ( 13.55%) 176533.50 ( 15.12%)

Quoted 1 line

Mean 7 153004.00 ( 0.00%) 160972.00 ( 5.21%) 180219.75 (

17.79%) 176605.25 ( 15.43%) 179010.25 ( 17.00%)

Quoted 1 line

Mean 8 151906.00 ( 0.00%) 161832.50 ( 6.53%) 164960.75 (

8.59%) 169345.25 ( 11.48%) 181374.75 ( 19.40%)

Quoted 1 line

Mean 9 154198.50 ( 0.00%) 148888.25 ( -3.44%) 167819.75 (

8.83%) 169039.75 ( 9.62%) 181401.75 ( 17.64%)

Quoted 1 line

Mean 10 153377.00 ( 0.00%) 143134.75 ( -6.68%) 155716.00 (

1.53%) 161498.25 ( 5.29%) 175450.50 ( 14.39%)

Quoted 1 line

Mean 11 138278.50 ( 0.00%) 139928.50 ( 1.19%) 149627.25 (

8.21%) 155845.75 ( 12.70%) 170797.75 ( 23.52%)

Quoted 1 line

Mean 12 128598.00 ( 0.00%) 134543.50 ( 4.62%) 142626.50 (

10.91%) 141966.75 ( 10.40%) 163766.50 ( 27.35%)

Quoted 1 line

Mean 13 130092.00 ( 0.00%) 134991.25 ( 3.77%) 137179.50 (

5.45%) 140382.75 ( 7.91%) 159106.00 ( 22.30%)

Quoted 1 line

Mean 14 121786.25 ( 0.00%) 134701.25 ( 10.60%) 134942.75 (

10.80%) 140522.50 ( 15.38%) 156676.25 ( 28.65%)

Quoted 1 line

Mean 15 121842.25 ( 0.00%) 134521.00 ( 10.41%) 134251.25 (

10.18%) 136963.50 ( 12.41%) 154455.50 ( 26.77%)

Quoted 1 line

Mean 16 120808.25 ( 0.00%) 130800.50 ( 8.27%) 136500.50 (

12.99%) 134488.50 ( 11.32%) 150137.75 ( 24.28%)

Quoted 1 line

Mean 17 118703.25 ( 0.00%) 130281.00 ( 9.75%) 134345.00 (

13.18%) 132618.75 ( 11.72%) 147027.50 ( 23.86%)

Quoted 1 line

Mean 18 116243.50 ( 0.00%) 128010.75 ( 10.12%) 130742.75 (

12.47%) 128031.25 ( 10.14%) 145852.50 ( 25.47%)

Quoted 1 line

Mean 19 116508.75 ( 0.00%) 127627.25 ( 9.54%) 127561.50 (

9.49%) 123765.25 ( 6.23%) 140879.25 ( 20.92%)

Quoted 1 line

Mean 20 117386.50 ( 0.00%) 133369.25 ( 13.62%) 126746.00 (

7.97%) 127327.75 ( 8.47%) 141559.25 ( 20.59%)

Quoted 1 line

Mean 21 118308.00 ( 0.00%) 129000.25 ( 9.04%) 127664.25 (

7.91%) 125308.50 ( 5.92%) 138110.75 ( 16.74%)

Quoted 1 line

Mean 22 116577.75 ( 0.00%) 130031.50 ( 11.54%) 127755.75 (

9.59%) 129445.75 ( 11.04%) 136937.50 ( 17.46%)

Quoted 1 line

Mean 23 117444.50 ( 0.00%) 125593.50 ( 6.94%) 125423.25 (

6.79%) 121178.50 ( 3.18%) 134960.25 ( 14.91%)

Quoted 1 line

Mean 24 108114.00 ( 0.00%) 121164.25 ( 12.07%) 117293.75 (

8.49%) 119736.25 ( 10.75%) 130200.25 ( 20.43%)

Quoted 1 line

Mean 25 114144.50 ( 0.00%) 118106.50 ( 3.47%) 118405.00 (

3.73%) 127544.75 ( 11.74%) 133198.50 ( 16.69%)

Quoted 1 line

Mean 26 111531.00 ( 0.00%) 120096.00 ( 7.68%) 124074.50 (

11.25%) 131079.50 ( 17.53%) 133463.50 ( 19.66%)

Quoted 1 line

Mean 27 113381.50 ( 0.00%) 121547.75 ( 7.20%) 121492.75 (

7.15%) 127801.25 ( 12.72%) 135127.25 ( 19.18%)

Quoted 1 line

Mean 28 111333.50 ( 0.00%) 118054.25 ( 6.04%) 121414.00 (

9.05%) 134754.00 ( 21.04%) 133270.00 ( 19.70%)

Quoted 1 line

Mean 29 108411.25 ( 0.00%) 119588.00 ( 10.31%) 123334.50 (

13.77%) 132048.25 ( 21.80%) 131892.50 ( 21.66%)

Quoted 1 line

Mean 30 110572.00 ( 0.00%) 119673.50 ( 8.23%) 118952.00 (

7.58%) 127541.50 ( 15.35%) 132982.00 ( 20.27%)

Quoted 1 line

Mean 31 114267.00 ( 0.00%) 117367.75 ( 2.71%) 119463.75 (

4.55%) 126795.75 ( 10.96%) 133970.00 ( 17.24%)

Quoted 1 line

Mean 32 112054.50 ( 0.00%) 115357.00 ( 2.95%) 111839.00 ( -

0.19%) 126267.00 ( 12.68%) 133289.00 ( 18.95%)

Quoted 1 line

Mean 33 113664.50 ( 0.00%) 115657.50 ( 1.75%) 119608.00 (

5.23%) 124996.50 ( 9.97%) 135356.00 ( 19.08%)

Quoted 1 line

Mean 34 110793.00 ( 0.00%) 115023.00 ( 3.82%) 122672.25 (

10.72%) 124025.50 ( 11.94%) 135818.00 ( 22.59%)

Quoted 1 line

Mean 35 116855.75 ( 0.00%) 118724.00 ( 1.60%) 115770.50 ( -

0.93%) 129668.50 ( 10.96%) 136327.00 ( 16.66%)

Quoted 1 line

Mean 36 112375.25 ( 0.00%) 121457.50 ( 8.08%) 116098.00 (

3.31%) 125627.25 ( 11.79%) 134052.25 ( 19.29%)

Quoted 1 line

Mean 37 109171.25 ( 0.00%) 119452.75 ( 9.42%) 119359.75 (

9.33%) 119926.25 ( 9.85%) 133200.75 ( 22.01%)

Quoted 1 line

Mean 38 105759.50 ( 0.00%) 124684.00 ( 17.89%) 117629.25 (

11.22%) 119691.25 ( 13.17%) 132992.75 ( 25.75%)

Quoted 1 line

Mean 39 110185.00 ( 0.00%) 125151.75 ( 13.58%) 112839.00 (

2.41%) 117929.00 ( 7.03%) 133400.75 ( 21.07%)

Quoted 1 line

Mean 40 110683.50 ( 0.00%) 122496.50 ( 10.67%) 118156.50 (

6.75%) 120739.00 ( 9.08%) 132520.25 ( 19.73%)

Quoted 1 line

Mean 41 102152.00 ( 0.00%) 123401.50 ( 20.80%) 114519.00 (

12.11%) 119549.00 ( 17.03%) 132827.00 ( 30.03%)

Quoted 1 line

Mean 42 105378.50 ( 0.00%) 115035.25 ( 9.16%) 110092.25 (

4.47%) 119086.75 ( 13.01%) 130747.75 ( 24.07%)

Quoted 1 line

Mean 43 107112.00 ( 0.00%) 112384.75 ( 4.92%) 110925.75 (

3.56%) 112715.75 ( 5.23%) 133097.00 ( 24.26%)

Quoted 1 line

Mean 44 105611.25 ( 0.00%) 118682.75 ( 12.38%) 106846.50 (

1.17%) 114581.25 ( 8.49%) 131913.25 ( 24.90%)

Quoted 1 line

Mean 45 105092.25 ( 0.00%) 116890.50 ( 11.23%) 110318.25 (

4.97%) 113010.00 ( 7.53%) 130818.00 ( 24.48%)

Quoted 1 line

Mean 46 111016.25 ( 0.00%) 111954.00 ( 0.84%) 106253.75 ( -

4.29%) 111746.00 ( 0.66%) 128427.25 ( 15.68%)

Quoted 1 line

Mean 47 106660.75 ( 0.00%) 116336.00 ( 9.07%) 110024.00 (

3.15%) 109703.50 ( 2.85%) 128223.50 ( 20.22%)

Quoted 1 line

Mean 48 103423.75 ( 0.00%) 115325.00 ( 11.51%) 108381.50 (

4.79%) 112662.50 ( 8.93%) 128536.75 ( 24.28%)

Quoted 1 line

Stddev 1 1309.96 ( 0.00%) 1796.84 (-37.17%) 1529.84 (-

16.78%) 725.83 ( 44.59%) 1207.28 ( 7.84%)

Quoted 1 line

Stddev 2 900.52 ( 0.00%) 954.05 ( -5.94%) 2856.62 (-

217.22%) 2730.12 (-203.17%) 2743.74 (-204.68%)

Quoted 1 line

Stddev 3 845.76 ( 0.00%) 778.56 ( 7.95%) 3350.11 (-

296.11%) 2794.68 (-230.43%) 1923.88 (-127.47%)

Quoted 1 line

Stddev 4 4988.11 ( 0.00%) 2321.83 ( 53.45%) 4635.26 (

7.07%) 867.46 ( 82.61%) 1288.21 ( 74.17%)

Quoted 1 line

Stddev 5 5969.57 ( 0.00%) 1605.41 ( 73.11%) 5415.52 (

9.28%) 2328.13 ( 61.00%) 1844.24 ( 69.11%)

Quoted 1 line

Stddev 6 12478.42 ( 0.00%) 3204.26 ( 74.32%) 4928.12 (

60.51%) 1908.73 ( 84.70%) 1109.97 ( 91.10%)

Quoted 1 line

Stddev 7 5975.15 ( 0.00%) 5954.29 ( 0.35%) 2558.42 (

57.18%) 2245.10 ( 62.43%) 894.47 ( 85.03%)

Quoted 1 line

Stddev 8 3404.47 ( 0.00%) 16525.39 (-385.40%) 8166.52 (-

139.88%) 6520.79 (-91.54%) 1233.67 ( 63.76%)

Quoted 1 line

Stddev 9 8811.92 ( 0.00%) 17373.77 (-97.16%) 8384.83 (

4.85%) 4511.21 ( 48.81%) 6092.53 ( 30.86%)

Quoted 1 line

Stddev 10 4050.40 ( 0.00%) 16781.65 (-314.32%) 9299.40 (-

129.59%) 7901.83 (-95.09%) 7086.69 (-74.96%)

Quoted 1 line

Stddev 11 9475.98 ( 0.00%) 22925.45 (-141.93%) 16135.16 (-

70.27%) 15935.55 (-68.17%) 12776.76 (-34.83%)

Quoted 1 line

Stddev 12 1961.27 ( 0.00%) 17661.49 (-800.51%) 12080.97 (-

515.98%) 13293.01 (-577.77%) 16113.60 (-721.59%)

Quoted 1 line

Stddev 13 2164.98 ( 0.00%) 20902.15 (-865.47%) 13747.72 (-

535.00%) 10093.27 (-366.21%) 17382.57 (-702.90%)

Quoted 1 line

Stddev 14 4186.32 ( 0.00%) 13552.26 (-223.73%) 15955.56 (-

281.14%) 9782.02 (-133.67%) 17469.49 (-317.30%)

Quoted 1 line

Stddev 15 1766.42 ( 0.00%) 14148.85 (-700.99%) 10003.01 (-

466.29%) 7777.42 (-340.29%) 17940.51 (-915.64%)

Quoted 1 line

Stddev 16 4116.25 ( 0.00%) 17131.59 (-316.19%) 9714.07 (-

135.99%) 9721.54 (-136.17%) 16802.13 (-308.19%)

Quoted 1 line

Stddev 17 4252.36 ( 0.00%) 14060.30 (-230.65%) 7048.48 (-

65.75%) 6797.22 (-59.85%) 14995.28 (-252.63%)

Quoted 1 line

Stddev 18 4421.47 ( 0.00%) 15675.15 (-254.52%) 6404.13 (-

44.84%) 8467.89 (-91.52%) 14654.80 (-231.45%)

Quoted 1 line

Stddev 19 1969.40 ( 0.00%) 13932.78 (-607.46%) 8825.82 (-

348.15%) 8576.52 (-335.49%) 14659.18 (-644.35%)

Quoted 1 line

Stddev 20 4204.79 ( 0.00%) 9151.72 (-117.65%) 6336.02 (-

50.69%) 4104.31 ( 2.39%) 13911.10 (-230.84%)

Quoted 1 line

Stddev 21 2319.45 ( 0.00%) 7818.64 (-237.09%) 5782.65 (-

149.31%) 9329.94 (-302.25%) 15425.22 (-565.04%)

Quoted 1 line

Stddev 22 1704.55 ( 0.00%) 8563.76 (-402.40%) 4382.12 (-

157.08%) 5264.66 (-208.86%) 14777.92 (-766.97%)

Quoted 1 line

Stddev 23 4454.10 ( 0.00%) 6454.89 (-44.92%) 10773.06 (-

141.87%) 8137.56 (-82.70%) 12067.61 (-170.93%)

Quoted 1 line

Stddev 24 3134.81 ( 0.00%) 9448.45 (-201.40%) 5911.90 (-

88.59%) 8289.99 (-164.45%) 19696.90 (-528.33%)

Quoted 1 line

Stddev 25 1946.32 ( 0.00%) 9054.46 (-365.21%) 5994.17 (-

207.98%) 5082.87 (-161.15%) 15444.45 (-693.52%)

Quoted 1 line

Stddev 26 6694.44 ( 0.00%) 8657.01 (-29.32%) 6814.85 ( -

1.80%) 3582.44 ( 46.49%) 15096.81 (-125.51%)

Quoted 1 line

Stddev 27 5159.46 ( 0.00%) 7280.85 (-41.12%) 4058.33 (

21.34%) 6129.48 (-18.80%) 14158.65 (-174.42%)

Quoted 1 line

Stddev 28 2665.79 ( 0.00%) 10345.52 (-288.08%) 2327.74 (

12.68%) 1748.34 ( 34.42%) 14139.33 (-430.40%)

Quoted 1 line

Stddev 29 6353.05 ( 0.00%) 7672.72 (-20.77%) 6876.07 ( -

8.23%) 4385.79 ( 30.97%) 13514.71 (-112.73%)

Quoted 1 line

Stddev 30 5378.33 ( 0.00%) 8665.82 (-61.12%) 6681.09 (-

24.22%) 7983.87 (-48.45%) 14576.47 (-171.02%)

Quoted 1 line

Stddev 31 10359.52 ( 0.00%) 10179.47 ( 1.74%) 5075.29 (

51.01%) 4198.25 ( 59.47%) 14020.78 (-35.34%)

Quoted 1 line

Stddev 32 9090.09 ( 0.00%) 17015.37 (-87.19%) 6124.63 (

32.62%) 5635.11 ( 38.01%) 14899.79 (-63.91%)

Quoted 1 line

Stddev 33 7143.20 ( 0.00%) 15988.55 (-123.83%) 7464.48 ( -

4.50%) 3019.97 ( 57.72%) 13049.06 (-82.68%)

Quoted 1 line

Stddev 34 4954.01 ( 0.00%) 8884.64 (-79.34%) 7827.59 (-

58.01%) 4385.45 ( 11.48%) 13382.51 (-170.14%)

Quoted 1 line

Stddev 35 6247.59 ( 0.00%) 9136.41 (-46.24%) 4600.77 (

26.36%) 3211.43 ( 48.60%) 11547.88 (-84.84%)

Quoted 1 line

Stddev 36 7325.84 ( 0.00%) 7969.69 ( -8.79%) 5746.70 (

21.56%) 6279.05 ( 14.29%) 13622.23 (-85.95%)

Quoted 1 line

Stddev 37 7952.43 ( 0.00%) 11648.76 (-46.48%) 6065.84 (

23.72%) 6666.17 ( 16.17%) 12860.81 (-61.72%)

Quoted 1 line

Stddev 38 6473.08 ( 0.00%) 6465.75 ( 0.11%) 4958.70 (

23.40%) 6539.37 ( -1.02%) 15220.62 (-135.14%)

Quoted 1 line

Stddev 39 2568.69 ( 0.00%) 5494.22 (-113.89%) 5649.24 (-

119.93%) 5527.10 (-115.17%) 13685.29 (-432.77%)

Quoted 1 line

Stddev 40 2823.65 ( 0.00%) 7118.16 (-152.09%) 1970.80 (

30.20%) 5372.43 (-90.27%) 13223.53 (-368.31%)

Quoted 1 line

Stddev 41 2145.41 ( 0.00%) 5532.78 (-157.89%) 3929.36 (-

83.15%) 6514.51 (-203.65%) 13346.51 (-522.10%)

Quoted 1 line

Stddev 42 2334.79 ( 0.00%) 7695.64 (-229.61%) 5361.91 (-

129.65%) 5845.25 (-150.35%) 13245.12 (-467.29%)

Quoted 1 line

Stddev 43 8940.43 ( 0.00%) 14286.16 (-59.79%) 9110.82 ( -

1.91%) 8481.12 ( 5.14%) 14413.92 (-61.22%)

Quoted 1 line

Stddev 44 6738.12 ( 0.00%) 10145.70 (-50.57%) 4789.94 (

28.91%) 7463.67 (-10.77%) 15134.45 (-124.61%)

Quoted 1 line

Stddev 45 3219.83 ( 0.00%) 6743.00 (-109.42%) 7629.10 (-

136.94%) 5441.27 (-68.99%) 13401.43 (-316.22%)

Quoted 1 line

Stddev 46 7925.89 ( 0.00%) 9956.81 (-25.62%) 9729.49 (-

22.76%) 10130.88 (-27.82%) 14020.22 (-76.89%)

Quoted 1 line

Stddev 47 7034.02 ( 0.00%) 12456.90 (-77.09%) 12705.68 (-

80.63%) 5117.46 ( 27.25%) 13224.66 (-88.01%)

Quoted 1 line

Stddev 48 11774.16 ( 0.00%) 8678.39 ( 26.29%) 11906.07 ( -

1.12%) 5293.32 ( 55.04%) 13968.76 (-18.64%)

Quoted 1 line

TPut 1 127663.00 ( 0.00%) 120030.00 ( -5.98%) 118362.00 ( -

7.29%) 122251.00 ( -4.24%) 115555.00 ( -9.48%)

Quoted 1 line

TPut 2 247658.00 ( 0.00%) 243416.00 ( -1.71%) 248653.00 (

0.40%) 242377.00 ( -2.13%) 250087.00 ( 0.98%)

Quoted 1 line

TPut 3 358773.00 ( 0.00%) 364637.00 ( 1.63%) 367756.00 (

2.50%) 372942.00 ( 3.95%) 379827.00 ( 5.87%)

Quoted 1 line

TPut 4 462750.00 ( 0.00%) 469752.00 ( 1.51%) 479762.00 (

3.68%) 497095.00 ( 7.42%) 498403.00 ( 7.70%)

Quoted 1 line

TPut 5 571157.00 ( 0.00%) 564096.00 ( -1.24%) 585396.00 (

2.49%) 608533.00 ( 6.54%) 602300.00 ( 5.45%)

Quoted 1 line

TPut 6 613381.00 ( 0.00%) 640365.00 ( 4.40%) 643964.00 (

4.99%) 696514.00 ( 13.55%) 706134.00 ( 15.12%)

Quoted 1 line

TPut 7 612016.00 ( 0.00%) 643888.00 ( 5.21%) 720879.00 (

17.79%) 706421.00 ( 15.43%) 716041.00 ( 17.00%)

Quoted 1 line

TPut 8 607624.00 ( 0.00%) 647330.00 ( 6.53%) 659843.00 (

8.59%) 677381.00 ( 11.48%) 725499.00 ( 19.40%)

Quoted 1 line

TPut 9 616794.00 ( 0.00%) 595553.00 ( -3.44%) 671279.00 (

8.83%) 676159.00 ( 9.62%) 725607.00 ( 17.64%)

Quoted 1 line

TPut 10 613508.00 ( 0.00%) 572539.00 ( -6.68%) 622864.00 (

1.53%) 645993.00 ( 5.29%) 701802.00 ( 14.39%)

Quoted 1 line

TPut 11 553114.00 ( 0.00%) 559714.00 ( 1.19%) 598509.00 (

8.21%) 623383.00 ( 12.70%) 683191.00 ( 23.52%)

Quoted 1 line

TPut 12 514392.00 ( 0.00%) 538174.00 ( 4.62%) 570506.00 (

10.91%) 567867.00 ( 10.40%) 655066.00 ( 27.35%)

Quoted 1 line

TPut 13 520368.00 ( 0.00%) 539965.00 ( 3.77%) 548718.00 (

5.45%) 561531.00 ( 7.91%) 636424.00 ( 22.30%)

Quoted 1 line

TPut 14 487145.00 ( 0.00%) 538805.00 ( 10.60%) 539771.00 (

10.80%) 562090.00 ( 15.38%) 626705.00 ( 28.65%)

Quoted 1 line

TPut 15 487369.00 ( 0.00%) 538084.00 ( 10.41%) 537005.00 (

10.18%) 547854.00 ( 12.41%) 617822.00 ( 26.77%)

Quoted 1 line

TPut 16 483233.00 ( 0.00%) 523202.00 ( 8.27%) 546002.00 (

12.99%) 537954.00 ( 11.32%) 600551.00 ( 24.28%)

Quoted 1 line

TPut 17 474813.00 ( 0.00%) 521124.00 ( 9.75%) 537380.00 (

13.18%) 530475.00 ( 11.72%) 588110.00 ( 23.86%)

Quoted 1 line

TPut 18 464974.00 ( 0.00%) 512043.00 ( 10.12%) 522971.00 (

12.47%) 512125.00 ( 10.14%) 583410.00 ( 25.47%)

Quoted 1 line

TPut 19 466035.00 ( 0.00%) 510509.00 ( 9.54%) 510246.00 (

9.49%) 495061.00 ( 6.23%) 563517.00 ( 20.92%)

Quoted 1 line

TPut 20 469546.00 ( 0.00%) 533477.00 ( 13.62%) 506984.00 (

7.97%) 509311.00 ( 8.47%) 566237.00 ( 20.59%)

Quoted 1 line

TPut 21 473232.00 ( 0.00%) 516001.00 ( 9.04%) 510657.00 (

7.91%) 501234.00 ( 5.92%) 552443.00 ( 16.74%)

Quoted 1 line

TPut 22 466311.00 ( 0.00%) 520126.00 ( 11.54%) 511023.00 (

9.59%) 517783.00 ( 11.04%) 547750.00 ( 17.46%)

Quoted 1 line

TPut 23 469778.00 ( 0.00%) 502374.00 ( 6.94%) 501693.00 (

6.79%) 484714.00 ( 3.18%) 539841.00 ( 14.91%)

Quoted 1 line

TPut 24 432456.00 ( 0.00%) 484657.00 ( 12.07%) 469175.00 (

8.49%) 478945.00 ( 10.75%) 520801.00 ( 20.43%)

Quoted 1 line

TPut 25 456578.00 ( 0.00%) 472426.00 ( 3.47%) 473620.00 (

3.73%) 510179.00 ( 11.74%) 532794.00 ( 16.69%)

Quoted 1 line

TPut 26 446124.00 ( 0.00%) 480384.00 ( 7.68%) 496298.00 (

11.25%) 524318.00 ( 17.53%) 533854.00 ( 19.66%)

Quoted 1 line

TPut 27 453526.00 ( 0.00%) 486191.00 ( 7.20%) 485971.00 (

7.15%) 511205.00 ( 12.72%) 540509.00 ( 19.18%)

Quoted 1 line

TPut 28 445334.00 ( 0.00%) 472217.00 ( 6.04%) 485656.00 (

9.05%) 539016.00 ( 21.04%) 533080.00 ( 19.70%)

Quoted 1 line

TPut 29 433645.00 ( 0.00%) 478352.00 ( 10.31%) 493338.00 (

13.77%) 528193.00 ( 21.80%) 527570.00 ( 21.66%)

Quoted 1 line

TPut 30 442288.00 ( 0.00%) 478694.00 ( 8.23%) 475808.00 (

7.58%) 510166.00 ( 15.35%) 531928.00 ( 20.27%)

Quoted 1 line

TPut 31 457068.00 ( 0.00%) 469471.00 ( 2.71%) 477855.00 (

4.55%) 507183.00 ( 10.96%) 535880.00 ( 17.24%)

Quoted 1 line

TPut 32 448218.00 ( 0.00%) 461428.00 ( 2.95%) 447356.00 ( -

0.19%) 505068.00 ( 12.68%) 533156.00 ( 18.95%)

Quoted 1 line

TPut 33 454658.00 ( 0.00%) 462630.00 ( 1.75%) 478432.00 (

5.23%) 499986.00 ( 9.97%) 541424.00 ( 19.08%)

Quoted 1 line

TPut 34 443172.00 ( 0.00%) 460092.00 ( 3.82%) 490689.00 (

10.72%) 496102.00 ( 11.94%) 543272.00 ( 22.59%)

Quoted 1 line

TPut 35 467423.00 ( 0.00%) 474896.00 ( 1.60%) 463082.00 ( -

0.93%) 518674.00 ( 10.96%) 545308.00 ( 16.66%)

Quoted 1 line

TPut 36 449501.00 ( 0.00%) 485830.00 ( 8.08%) 464392.00 (

3.31%) 502509.00 ( 11.79%) 536209.00 ( 19.29%)

Quoted 1 line

TPut 37 436685.00 ( 0.00%) 477811.00 ( 9.42%) 477439.00 (

9.33%) 479705.00 ( 9.85%) 532803.00 ( 22.01%)

Quoted 1 line

TPut 38 423038.00 ( 0.00%) 498736.00 ( 17.89%) 470517.00 (

11.22%) 478765.00 ( 13.17%) 531971.00 ( 25.75%)

Quoted 1 line

TPut 39 440740.00 ( 0.00%) 500607.00 ( 13.58%) 451356.00 (

2.41%) 471716.00 ( 7.03%) 533603.00 ( 21.07%)

Quoted 1 line

TPut 40 442734.00 ( 0.00%) 489986.00 ( 10.67%) 472626.00 (

6.75%) 482956.00 ( 9.08%) 530081.00 ( 19.73%)

Quoted 1 line

TPut 41 408608.00 ( 0.00%) 493606.00 ( 20.80%) 458076.00 (

12.11%) 478196.00 ( 17.03%) 531308.00 ( 30.03%)

Quoted 1 line

TPut 42 421514.00 ( 0.00%) 460141.00 ( 9.16%) 440369.00 (

4.47%) 476347.00 ( 13.01%) 522991.00 ( 24.07%)

Quoted 1 line

TPut 43 428448.00 ( 0.00%) 449539.00 ( 4.92%) 443703.00 (

3.56%) 450863.00 ( 5.23%) 532388.00 ( 24.26%)

Quoted 1 line

TPut 44 422445.00 ( 0.00%) 474731.00 ( 12.38%) 427386.00 (

1.17%) 458325.00 ( 8.49%) 527653.00 ( 24.90%)

Quoted 1 line

TPut 45 420369.00 ( 0.00%) 467562.00 ( 11.23%) 441273.00 (

4.97%) 452040.00 ( 7.53%) 523272.00 ( 24.48%)

Quoted 1 line

TPut 46 444065.00 ( 0.00%) 447816.00 ( 0.84%) 425015.00 ( -

4.29%) 446984.00 ( 0.66%) 513709.00 ( 15.68%)

Quoted 1 line

TPut 47 426643.00 ( 0.00%) 465344.00 ( 9.07%) 440096.00 (

3.15%) 438814.00 ( 2.85%) 512894.00 ( 20.22%)

Quoted 1 line

TPut 48 413695.00 ( 0.00%) 461300.00 ( 11.51%) 433526.00 (

4.79%) 450650.00 ( 8.93%) 514147.00 ( 24.28%)

Quoted 8 lines

This is looking much better overall and again the impact of the continual retries of migration makes a big difference. It indicates that it would be worth investigating why the retries are even necessary. It would be unfortunate if NUMA scheduling was constantly battling the load balancer.

specjbb Peaks

                                    3.11.0                   

3.11.0 3.11.0 3.11.0
3.11.0

Quoted 1 line

account-v8r20 falsedetect-

v8r33 skipshared-v8r33 noatomics-v8r33 periodretry-
v8r33

Quoted 1 line

Expctd Warehouse 12.00 ( 0.00%) 12.00 (

0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 (
0.00%)

Quoted 1 line

Expctd Peak Bops 553114.00 ( 0.00%) 559714.00 ( 1.19%)

598509.00 ( 8.21%) 623383.00 ( 12.70%) 683191.00 ( 23.52%)

Quoted 1 line

Actual Warehouse 10.00 ( 0.00%) 9.00 (-

10.00%) 8.00 (-20.00%) 8.00 (-20.00%) 10.00
( 0.00%)

Quoted 1 line

Actual Peak Bops 616794.00 ( 0.00%) 647330.00 ( 4.95%)

720879.00 ( 16.88%)      706421.00 ( 14.53%)      725607.00 ( 17.64%)
>  SpecJBB Bops          477666.00 (  0.00%)      521426.00 (  9.16%)      
524010.00 (  9.70%)      523610.00 (  9.62%)      584513.00 ( 22.37%)
>  SpecJBB Bops/JVM      119417.00 (  0.00%)      130357.00 (  9.16%)      
131003.00 (  9.70%)      130903.00 (  9.62%)      146128.00 ( 22.37%)
Quoted 5 lines

Speaks for itself really, looks great.

              3.11.0      3.11.0      3.11.0      3.11.0      3.11.0
        account-v8r20falsedetect-v8r33skipshared-v8r33noatomics-

v8r33periodretry-v8r33

Quoted 5 lines

User 466719.69 478006.22 476399.23 476733.11 476517.65
System 11049.55 913.00 1429.31 1317.87 994.02
Elapsed 10381.80 10415.18 10384.39 10383.33 10393.91

and system CPU usage is still way down so now we are seeing large

improvements for less work.

Quoted 2 lines
                                3.11.0      3.11.0      3.11.0      

3.11.0 3.11.0

Quoted 1 line

account-v8r20falsedetect-v8r33skipshared-

v8r33noatomics-v8r33periodretry-v8r33

Quoted 1 line

Compaction cost 1432168 77123 127225

119169 66598

Quoted 1 line

NUMA PTE updates 505320169 9237242 44679605

40306037 40633804

Quoted 1 line

NUMA hint faults 67707706 9240325 44739890

40798082 40521421

Quoted 1 line

NUMA hint local faults 22183991 3313857 14359951

13135180 25014367

Quoted 1 line

NUMA hint local percent 32 35 32

32 61

Quoted 1 line

NUMA pages migrated 1379738662 74299705 122568192

114806428 64160571

Quoted 1 line

AutoNUMA cost 368290 47677 226341

206453 204110

Quoted 2 lines

That migrate retry thing does wonders for the local fault percentage.

Really worth checking out

Quoted 3 lines

why tasks are getting migrated off their preferred node.

The NAS parallel benchmarks had not run at the time of writing but I

decided not to delay releasing

Quoted 19 lines

this anyway.

Overall this series is now looking much better. The hotplug handling part is still missing but hopefully with Peter overhead reduction efforts it will not impair the results much. The primary curiousity for me is why the period retry makes such a difference because in my mind it implies the load balancer and numa scheduler are working against each other for some reason. Even without that answer I think this should get some time in tip when the hotplug part is filled in.

 Documentation/sysctl/kernel.txt   |   76 ++
 arch/x86/mm/numa.c                |    6 +-
 fs/exec.c                         |    1 +
 fs/proc/array.c                   |    2 +
 include/linux/mempolicy.h         |    1 +
 include/linux/migrate.h           |    7 +-
 include/linux/mm.h                |  118 ++-
 include/linux/mm_types.h          |   17 +-
 include/linux/page-flags-layout.h |   28 +-
Quoted 8 lines
 include/linux/sched.h             |   63 +-
 include/linux/sched/sysctl.h      |    1 -
 include/linux/stop_machine.h      |    1 +
 kernel/bounds.c                   |    4 +
 kernel/fork.c                     |    5 +-
 kernel/sched/core.c               |  197 ++++-
 kernel/sched/debug.c              |   60 +-
 kernel/sched/fair.c               | 1608 +++++++++++++++++++++++++++++--

------

Quoted 9 lines
 kernel/sched/features.h           |   19 +-
 kernel/sched/idle_task.c          |    2 +-
 kernel/sched/rt.c                 |    5 +-
 kernel/sched/sched.h              |   27 +-
 kernel/sched/stop_task.c          |    2 +-
 kernel/stop_machine.c             |  272 ++++---
 kernel/sysctl.c                   |   21 +-
 lib/vsprintf.c                    |    5 +
 mm/huge_memory.c                  |  119 ++-
Quoted 9 lines
 mm/memory.c                       |  158 ++--
 mm/mempolicy.c                    |   82 +-
 mm/migrate.c                      |   49 +-
 mm/mm_init.c                      |   18 +-
 mm/mmzone.c                       |   14 +-
 mm/mprotect.c                     |   65 +-
 mm/page_alloc.c                   |    4 +-
 33 files changed, 2307 insertions(+), 750 deletions(-)

Hi Mel,

Attempts to boot large KVM guest with the host and the guest running V8 patch set (+ PeterZ's fix for race in migrate_swap_top) results in a guest OS panic (below or the details). Sometimes the guest boots up but dies within mins. (The problem doesn't happen if the guest is running the base 3.11 kernel).

Smaller sized guests (i tried upto 40vcpus) boot up and seem to run fine.

Thanks
Vinod

/usr/local/bin/qemu-system-x86_64 \
-enable-kvm \
-cpu host \
-name vm1 \
-smp sockets=8,cores=10,threads=1 \
-numa node,nodeid=0,cpus=0-9,mem=128g \
-numa node,nodeid=1,cpus=10-19,mem=128g \
-numa node,nodeid=2,cpus=20-29,mem=128g \
-numa node,nodeid=3,cpus=30-39,mem=128g \
-numa node,nodeid=4,cpus=40-49,mem=128g \
-numa node,nodeid=5,cpus=50-59,mem=128g \
-numa node,nodeid=6,cpus=60-69,mem=128g \
-numa node,nodeid=7,cpus=70-79,mem=128g \
-m 1048576 \
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm1.monitor,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no- shutdown \
-drive file=/dev/libvirt_lvm2/vm1,if=none,id=drive-virtio- disk0,format=raw,cache=none,aio=native \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio- disk0,id=virtio-disk0,bootindex=1 \
-serial stdio \
-monitor stdio \
-netdev tap,id=hostnet1,script=no,downscript=no,vhost=on,ifname=tap1 \
-device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:71:01:01 \
-vnc :1

guest_panic

BUG: unable to handle kernel paging request at ffff890010e563a0

IP: [<ffffffff81293644>] ioread32_rep+0x44/0x60

PGD 1edf067 PUD 1ee0067 PMD 8000010010e001e3

Oops: 000b [#1] SMP 

Modules linked in: ata_piix(F) dm_mirror(F) dm_region_hash(F) dm_log(F) 

dm_mod(F)

CPU: 71 PID: 447 Comm: kworker/71:1 Tainted: GF       W    3.11.0mg8+ #4

Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011

Workqueue: ata_sff ata_sff_pio_task

task: ffff890011ab3240 ti: ffff890011ab4000 task.ti: ffff890011ab4000

RIP: 0010:[<ffffffff81293644>]  [<ffffffff81293644>] ioread32_rep+0x44/0x60

RSP: 0018:ffff890011ab5cc8 EFLAGS: 00000202

RAX: 0000000000000170 RBX: 0000000000000200 RCX: 0000000000000080

RDX: 0000000000000170 RSI: ffff890010e563a0 RDI: ffff890010e563a0

RBP: ffff890011ab5cc8 R08: 6db6db6db6db6db7 R09: dbff456b0e0540b8

R10: 0000000000000000 R11: 0000000000000000 R12: ffff890010e563a0

R13: 0000000000010170 R14: 0000000000000004 R15: 0000000000000000

FS: 0000000000000000(0000) GS:ffff89001fc20000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b

CR2: ffff890010e563a0 CR3: 0000010012918000 CR4: 00000000000007e0

Stack:

ffff890011ab5d28 ffffffff813e040a ffff890011ab5ce8 ffffffff8105060f

ffff890011ab5cf8 ffff890011ab3290 ffff89001fc34040 ffff890010e55b80

0000000000000000 ffffea03803b22d0 0000000000000004 0000000000000058

Call Trace:

 [<ffffffff813e040a>] ata_sff_data_xfer32+0x7a/0x130

[<ffffffff8105060f>] ? kvm_clock_read+0x1f/0x30

 [<ffffffff813df43e>] ata_pio_sector+0xbe/0x130

 [<ffffffff813df52d>] ata_pio_sectors+0x7d/0xc0

 [<ffffffff813e1487>] ata_sff_hsm_move+0x207/0x5e0

 [<ffffffff813e1912>] ata_sff_pio_task+0xb2/0x190

 [<ffffffff8107cc59>] process_one_work+0x189/0x490

%G [<ffffffff8107def0>] worker_thread+0x120/0x3a0

[<ffffffff8107ddd0>] ? manage_workers+0x170/0x170

[<ffffffff81083f9e>] kthread+0xce/0xe0

[<ffffffff81083ed0>] ? kthread_freezable_should_stop+0x70/0x70

[<ffffffff815895ac>] ret_from_fork+0x7c/0xb0

[<ffffffff81083ed0>] ? kthread_freezable_should_stop+0x70/0x70

Code: 54 96 04 0f 1f 40 00 8b 07 89 06 48 83 c6 04 48 39 d6 75 f3 c9 c3 90 

48 81 ff 00 00 01 00 76 17 0f b7 c7 48 89 d1 48 89 f7 89 c2 <f3> 6d c9 c3 0f 1f 84 00 00 00 00 00 48 c7 c6 b5 46 80 81 e8 f4

RIP [<ffffffff81293644>] ioread32_rep+0x44/0x60

RSP <ffff890011ab5cc8>

CR2: ffff890010e563a0

---[ end trace e41683dcbc01154b ]---

BUG: unable to handle kernel paging request at ffffffffffffffd8

IP: [<ffffffff81083850>] kthread_data+0x10/0x20

PGD 1a0f067 PUD 1a11067 PMD 0

Oops: 0000 [#2] SMP 

Modules linked in: ata_piix(F) dm_mirror(F) dm_region_hash(F) dm_log(F) 

dm_mod(F)

CPU: 71 PID: 447 Comm: kworker/71:1 Tainted: GF     D W    3.11.0mg8+ #4

Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011

task: ffff890011ab3240 ti: ffff890011ab4000 task.ti: ffff890011ab4000

RIP: 0010:[<ffffffff81083850>]  [<ffffffff81083850>] kthread_data+0x10/0x20

RSP: 0018:ffff890011ab58f8 EFLAGS: 00010092

RAX: 0000000000000000 RBX: 0000000000000047 RCX: ffffffff81d52fe0

RDX: 0000000000000000 RSI: 0000000000000047 RDI: ffff890011ab3240

RBP: ffff890011ab58f8 R08: 0000000000000000 R09: dead000000200200

R10: 0000000000031b59 R11: 0000000000000000 R12: 0000000000000047

R13: ffff890011ab3820 R14: 0000000000000001 R15: 000000000000000b

FS: 0000000000000000(0000) GS:ffff89001fc20000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b

CR2: 0000000000000028 CR3: 0000010012918000 CR4: 00000000000007e0

Stack:

ffff890011ab5918 ffffffff8107a565 ffff890011ab5918 ffff89001fc34040

ffff890011ab59a8 ffffffff8157ea53 ffff890011ab5fd8 0000000000014040

ffff890011ab4010 0000000000014040 0000000000014040 0000000000014040

Call Trace:

 [<ffffffff8107a565>] wq_worker_sleeping+0x15/0xa0

 [<ffffffff8157ea53>] __schedule+0x5c3/0x710

[<ffffffff81063142>] ? release_task+0xc2/0x130

 [<ffffffff8157ed79>] schedule+0x29/0x70

 [<ffffffff81063f05>] do_exit+0x2d5/0x480

 [<ffffffff81581a61>] oops_end+0xb1/0x100

 [<ffffffff81055c3e>] no_context+0x11e/0x1f0

 [<ffffffff81055e3d>] __bad_area_nosemaphore+0x12d/0x230

 [<ffffffff81055f53>] bad_area_nosemaphore+0x13/0x20

 [<ffffffff815847c9>] __do_page_fault+0xd9/0x480

[<ffffffff8110ab62>] ? __buffer_unlock_commit+0x12/0x20

 [<ffffffff81584b7e>] do_page_fault+0xe/0x10

 [<ffffffff81584228>] do_async_page_fault+0x28/0xa0

 [<ffffffff81580ec8>] async_page_fault+0x28/0x30

[<ffffffff81293644>] ? ioread32_rep+0x44/0x60

[<ffffffff813e040a>] ata_sff_data_xfer32+0x7a/0x130

[<ffffffff8105060f>] ? kvm_clock_read+0x1f/0x30

 [<ffffffff813df43e>] ata_pio_sector+0xbe/0x130

 [<ffffffff813df52d>] ata_pio_sectors+0x7d/0xc0

 [<ffffffff813e1487>] ata_sff_hsm_move+0x207/0x5e0

 [<ffffffff813e1912>] ata_sff_pio_task+0xb2/0x190

 [<ffffffff8107cc59>] process_one_work+0x189/0x490

 [<ffffffff8107def0>] worker_thread+0x120/0x3a0

[<ffffffff8107ddd0>] ? manage_workers+0x170/0x170

[<ffffffff81083f9e>] kthread+0xce/0xe0

[<ffffffff81083ed0>] ? kthread_freezable_should_stop+0x70/0x70

[<ffffffff815895ac>] ret_from_fork+0x7c/0xb0

[<ffffffff81083ed0>] ? kthread_freezable_should_stop+0x70/0x70

Code: 88 05 00 00 48 8b 40 c8 c9 48 c1 e8 02 83 e0 01 c3 66 2e 0f 1f 84 00 

00 00 00 00 55 48 89 e5 66 66 66 66 90 48 8b 87 88 05 00 00 <48> 8b 40 d8 c9 c3 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 66

RIP [<ffffffff81083850>] kthread_data+0x10/0x20

RSP <ffff890011ab58f8>

CR2: ffffffffffffffd8

---[ end trace e41683dcbc01154c ]---

Fixing recursive fault but reboot is needed!