Basic scheduler support for automatic NUMA balancing V9
Subsystems (15)
Mel Gorman
reviewercover Basic scheduler support for automatic NUMA balancing V9
This series has roughly the same goals as previous versions despite the size. It reduces overhead of automatic balancing through scan rate reduction and the avoidance of TLB flushes. It selects a preferred node and moves tasks towards their memory as well as moving memory toward their task. It handles shared pages and groups related tasks together. Some problems such as shared page interleaving and properly dealing with processes that are larger than a node are being deferred. This version should be ready for wider testing in -tip.
Note that with kernel 3.12-rc3 that numa balancing will fail to boot if CONFIG_JUMP_LABEL is configured. This is a separate bug that is currently being dealt with.
Changelog since V8
o Rebased to v3.12-rc3
o Handle races against hotplug
Changelog since V7
o THP migration race and pagetable insertion fixes
o Do no handle PMDs in batch
o Shared page migration throttling
o Various changes to how last nid/pid information is recorded
o False pid match sanity checks when joining NUMA task groups
o Adapt scan rate based on local/remote fault statistics
o Period retry of migration to preferred node
o Limit scope of system-wide search
o Schedule threads on the same node as process that created them
o Cleanup numa_group on exec
Changelog since V6
o Group tasks that share pages together
o More scan avoidance of VMAs mapping pages that are not likely to migrate
o cpunid conversion, system-wide searching of tasks to balance with
Changelog since V6
o Various TLB flush optimisations
o Comment updates
o Sanitise task_numa_fault callsites for consistent semantics
o Revert some of the scanning adaption stuff
o Revert patch that defers scanning until task schedules on another node
o Start delayed scanning properly
o Avoid the same task always performing the PTE scan
o Continue PTE scanning even if migration is rate limited
Changelog since V5
o Add __GFP_NOWARN for numa hinting fault count
o Use is_huge_zero_page
o Favour moving tasks towards nodes with higher faults
o Optionally resist moving tasks towards nodes with lower faults
o Scan shared THP pages
Changelog since V4
o Added code that avoids overloading preferred nodes
o Swap tasks if nodes are overloaded and the swap does not impair locality
Changelog since V3
o Correct detection of unset last nid/pid information
o Dropped nr_preferred_running and replaced it with Peter's load balancing
o Pass in correct node information for THP hinting faults
o Pressure tasks sharing a THP page to move towards same node
o Do not set pmd_numa if false sharing is detected
Changelog since V2
o Reshuffle to match Peter's implied preference for layout
o Reshuffle to move private/shared split towards end of series to make it easier to evaluate the impact
o Use PID information to identify private accesses
o Set the floor for PTE scanning based on virtual address space scan rates instead of time
o Some locking improvements
o Do not preempt pinned tasks unless they are kernel threads
Changelog since V1
o Scan pages with elevated map count (shared pages)
o Scale scan rates based on the vsz of the process so the sampling of the task is independant of its size
o Favour moving towards nodes with more faults even if it's not the preferred node
o Laughably basic accounting of a compute overloaded node when selecting the preferred node.
o Applied review comments
This series integrates basic scheduler support for automatic NUMA balancing. It was initially based on Peter Ziljstra's work in "sched, numa, mm:
Add adaptive NUMA affinity support" but deviates too much to preserve Signed-off-bys. As before, if the relevant authors are ok with it I'll add Signed-off-bys (or add them yourselves if you pick the patches up). There has been a tonne of additional work from both Peter and Rik van Riel.
Some reports indicate that the performance is getting close to manual bindings for some workloads but your mileage will vary.
Patch 1 is a monolithic dump of patches thare are destined for upstream that
this series indirectly depends upon.
Patches 2-3 adds sysctl documentation and comment fixlets
Patch 4 avoids accounting for a hinting fault if another thread handled the
fault in parallel
Patches 5-6 avoid races with parallel THP migration and THP splits.
Patch 7 corrects a THP NUMA hint fault accounting bug
Patches 8-9 avoids TLB flushes during the PTE scan if no updates are made
Patch 10 sanitizes task_numa_fault callsites to have consist semantics and
always record the fault based on the correct location of the page.
Patch 11 closes races between THP migration and PMD clearing.
Patch 12 avoids trying to migrate the THP zero page
Patch 13 avoids the same task being selected to perform the PTE scan within
a shared address space.
Patch 14 continues PTE scanning even if migration rate limited
Patch 15 notes that delaying the PTE scan until a task is scheduled on an
alternative node misses the case where the task is only accessing shared memory on a partially loaded machine and reverts a patch.
Patch 16 initialises numa_next_scan properly so that PTE scanning is delayed
when a process starts.
Patch 17 sets the scan rate proportional to the size of the task being
scanned.
Patch 18 slows the scan rate if no hinting faults were trapped by an idle task.
Patch 19 tracks NUMA hinting faults per-task and per-node
Patches 20-24 selects a preferred node at the end of a PTE scan based on what
node incurred the highest number of NUMA faults. When the balancer is comparing two CPU it will prefer to locate tasks on their preferred node. When initially selected the task is rescheduled on the preferred node if it is not running on that node already. This avoids waiting for the scheduler to move the task slowly.
Patch 25 adds infrastructure to allow separate tracking of shared/private
pages but treats all faults as if they are private accesses. Laying it out this way reduces churn later in the series when private fault detection is introduced
Patch 26 avoids some unnecessary allocation
Patch 27-28 kicks away some training wheels and scans shared pages and
small VMAs.
Patch 29 introduces private fault detection based on the PID of the faulting
process and accounts for shared/private accesses differently.
Patch 30 avoids migrating memory immediately after the load balancer moves
a task to another node in case it's a transient migration.
Patch 31 avoids scanning VMAs that do not migrate-on-fault which addresses
a serious regression on a database performance test.
Patch 32 pick the least loaded CPU based on a preferred node based on
a scheduling domain common to both the source and destination NUMA node.
Patch 33 retries task migration if an earlier attempt failed
Patch 34 will begin task migration immediately if running on its preferred
node
Patch 35 will avoid trapping hinting faults for shared read-only library
pages as these never migrate anyway
Patch 36 avoids handling pmd hinting faults if none of the ptes below it were
marked pte numa
Patches 37-38 introduce a mechanism for swapping tasks
Patch 39 uses a system-wide search to find tasks that can be swapped
to improve the overall locality of the system.
Patch 40 notes that the system-wide search may ignore the preferred node and
will use the preferred node placement if it has spare compute capacity.
Patch 41 will perform a global search if a node that should have had capacity
cannot have a task migrated to it
Patches 42-43 use cpupid to track pages so potential sharing tasks can
be quickly found
Patch 44 reports the ID of the numa group a task belongs.
Patch 45 copies the cpupid on page migration
Patch 46 avoids grouping based on read-only pages
Patch 47 stops handling pages within a PMD in batch as it distorts fault
statistics and failed to flush TLBs correctly.
Patch 48 schedules new threads on the same node as the parent.
Patch 49 schedules tasks based on their numa group
Patch 50 cleans up tasks numa_group on exec
Patch 51 avoids parallel updates to group stats
Patch 52 adds some debugging aids
Patches 53-54 separately considers task and group weights when selecting the node to
schedule a task on
Patch 56 checks if PID truncation may have caused false matches before joining tasks
to a NUMA grou
Patch 57 uses the false shared detection information for scan rate adaption later
Patch 58 adapts the scan rate based on local/remote faults
Patch 59 removes the period scan rate reset
Patch 60-61 throttles shared page migrations
Patch 62 avoids the use of atomics protects the values with a spinlock
Patch 63 periodically retries migrating a task back to its preferred node
Kernel 3.12-rc3 is the testing baseline.
o account-v9 Patches 1-8
o periodretry-v8 Patches 1-63
This is SpecJBB running on a 4-socket machine with THP enabled and one JVM running for the whole system.
specjbb
3.12.0-rc3 3.12.0-rc3
account-v9 periodretry-v9
TPut 1 26187.00 ( 0.00%) 25922.00 ( -1.01%)
TPut 2 55752.00 ( 0.00%) 53928.00 ( -3.27%)
TPut 3 88878.00 ( 0.00%) 84689.00 ( -4.71%)
TPut 4 111226.00 ( 0.00%) 111843.00 ( 0.55%)
TPut 5 138700.00 ( 0.00%) 139712.00 ( 0.73%)
TPut 6 173467.00 ( 0.00%) 161226.00 ( -7.06%)
TPut 7 197609.00 ( 0.00%) 194035.00 ( -1.81%)
TPut 8 220501.00 ( 0.00%) 218853.00 ( -0.75%)
TPut 9 247997.00 ( 0.00%) 244480.00 ( -1.42%)
TPut 10 275616.00 ( 0.00%) 269962.00 ( -2.05%)
TPut 11 301610.00 ( 0.00%) 301051.00 ( -0.19%)
TPut 12 326151.00 ( 0.00%) 318040.00 ( -2.49%)
TPut 13 341671.00 ( 0.00%) 346890.00 ( 1.53%)
TPut 14 372805.00 ( 0.00%) 367204.00 ( -1.50%)
TPut 15 390175.00 ( 0.00%) 371538.00 ( -4.78%)
TPut 16 406716.00 ( 0.00%) 409835.00 ( 0.77%)
TPut 17 429094.00 ( 0.00%) 436172.00 ( 1.65%)
TPut 18 457167.00 ( 0.00%) 456528.00 ( -0.14%)
TPut 19 476963.00 ( 0.00%) 479680.00 ( 0.57%)
TPut 20 492751.00 ( 0.00%) 480019.00 ( -2.58%)
TPut 21 514952.00 ( 0.00%) 511950.00 ( -0.58%)
TPut 22 521962.00 ( 0.00%) 516450.00 ( -1.06%)
TPut 23 537268.00 ( 0.00%) 532825.00 ( -0.83%)
TPut 24 541231.00 ( 0.00%) 539425.00 ( -0.33%)
TPut 25 530459.00 ( 0.00%) 538714.00 ( 1.56%)
TPut 26 538837.00 ( 0.00%) 524894.00 ( -2.59%)
TPut 27 534132.00 ( 0.00%) 519628.00 ( -2.72%)
TPut 28 529470.00 ( 0.00%) 519044.00 ( -1.97%)
TPut 29 504426.00 ( 0.00%) 514158.00 ( 1.93%)
TPut 30 514785.00 ( 0.00%) 513080.00 ( -0.33%)
TPut 31 501018.00 ( 0.00%) 492377.00 ( -1.72%)
TPut 32 488377.00 ( 0.00%) 492108.00 ( 0.76%)
TPut 33 484809.00 ( 0.00%) 493612.00 ( 1.82%)
TPut 34 473015.00 ( 0.00%) 477716.00 ( 0.99%)
TPut 35 451833.00 ( 0.00%) 455368.00 ( 0.78%)
TPut 36 445787.00 ( 0.00%) 460138.00 ( 3.22%)
TPut 37 446034.00 ( 0.00%) 453011.00 ( 1.56%)
TPut 38 433305.00 ( 0.00%) 441966.00 ( 2.00%)
TPut 39 431202.00 ( 0.00%) 443747.00 ( 2.91%)
TPut 40 420040.00 ( 0.00%) 432818.00 ( 3.04%)
TPut 41 416519.00 ( 0.00%) 424105.00 ( 1.82%)
TPut 42 426047.00 ( 0.00%) 430164.00 ( 0.97%)
TPut 43 421725.00 ( 0.00%) 419106.00 ( -0.62%)
TPut 44 414340.00 ( 0.00%) 425471.00 ( 2.69%)
TPut 45 413836.00 ( 0.00%) 418506.00 ( 1.13%)
TPut 46 403636.00 ( 0.00%) 421177.00 ( 4.35%)
TPut 47 387726.00 ( 0.00%) 388190.00 ( 0.12%)
TPut 48 405375.00 ( 0.00%) 418321.00 ( 3.19%)Mostly flat. Profiles were interesting because they showed heavy contention on the mm->page_table_lock due to THP faults and migration. It is expected that Kirill's page table lock split lock work will help here. At the time of writing that series has been rebased on top for testing.
specjbb Peaks
3.12.0-rc3 3.12.0-rc3
account-v9 periodretry-v9
Expctd Warehouse 48.00 ( 0.00%) 48.00 ( 0.00%)
Expctd Peak Bops 387726.00 ( 0.00%) 388190.00 ( 0.12%)
Actual Warehouse 25.00 ( 0.00%) 25.00 ( 0.00%)
Actual Peak Bops 541231.00 ( 0.00%) 539425.00 ( -0.33%)
SpecJBB Bops 8273.00 ( 0.00%) 8537.00 ( 3.19%)
SpecJBB Bops/JVM 8273.00 ( 0.00%) 8537.00 ( 3.19%)Minor gain in the overal specjbb score but the peak performance is slightly lower.
3.12.0-rc3 3.12.0-rc3
account-v9 periodretry-v9 User 44731.08 44820.18
System 189.53 124.16
Elapsed 1665.71 1666.42
3.12.0-rc3 3.12.0-rc3
account-v9 periodretry-v9
Minor Faults 3815276 4471086
Major Faults 108 131
Compaction cost 12002 3214
NUMA PTE updates 17955537 3849428
NUMA hint faults 3950201 3822150
NUMA hint local faults 1032610 1029273NUMA hint local percent 26 26
NUMA pages migrated 11562658 3096443
AutoNUMA cost 20096 19196
As with previous releases system CPU usage is generally lower with fewer scans.
autonumabench
3.12.0-rc3 3.12.0-rc3
account-v9 periodretry-v9
User NUMA01 43871.21 ( 0.00%) 53162.55 (-21.18%)
User NUMA01_THEADLOCAL 25270.59 ( 0.00%) 28868.37 (-14.24%)
User NUMA02 2196.67 ( 0.00%) 2110.35 ( 3.93%)
User NUMA02_SMT 1039.18 ( 0.00%) 1035.41 ( 0.36%)
System NUMA01 187.11 ( 0.00%) 154.69 ( 17.33%)
System NUMA01_THEADLOCAL 216.47 ( 0.00%) 95.47 ( 55.90%)
System NUMA02 3.52 ( 0.00%) 3.26 ( 7.39%)
System NUMA02_SMT 2.42 ( 0.00%) 2.03 ( 16.12%)
Elapsed NUMA01 970.59 ( 0.00%) 1199.46 (-23.58%)
Elapsed NUMA01_THEADLOCAL 569.11 ( 0.00%) 643.37 (-13.05%)
Elapsed NUMA02 51.59 ( 0.00%) 49.94 ( 3.20%)
Elapsed NUMA02_SMT 49.73 ( 0.00%) 50.29 ( -1.13%)
CPU NUMA01 4539.00 ( 0.00%) 4445.00 ( 2.07%)
CPU NUMA01_THEADLOCAL 4478.00 ( 0.00%) 4501.00 ( -0.51%)
CPU NUMA02 4264.00 ( 0.00%) 4231.00 ( 0.77%)
CPU NUMA02_SMT 2094.00 ( 0.00%) 2062.00 ( 1.53%)The numa01 (adverse workload) is hit quite badly but it often is. The numa01-threadlocal regression is of greater concern and will be examined further. It is interesting to note that monitoring the workload affects the results quite severely. These results are based on no monitoring.
This is SpecJBB running on a 4-socket machine with THP enabled and one JVM running per node on the system.
specjbb
3.12.0-rc3 3.12.0-rc3
account-v9 periodretry-v9
Mean 1 30900.00 ( 0.00%) 29541.50 ( -4.40%)
Mean 2 62820.50 ( 0.00%) 63330.25 ( 0.81%)
Mean 3 92803.00 ( 0.00%) 92629.75 ( -0.19%)
Mean 4 119122.25 ( 0.00%) 121981.75 ( 2.40%)
Mean 5 142391.00 ( 0.00%) 148290.50 ( 4.14%)
Mean 6 151073.00 ( 0.00%) 169823.75 ( 12.41%)
Mean 7 152618.50 ( 0.00%) 166411.00 ( 9.04%)
Mean 8 141284.25 ( 0.00%) 153222.00 ( 8.45%)
Mean 9 136055.25 ( 0.00%) 139262.50 ( 2.36%)
Mean 10 124290.50 ( 0.00%) 133464.50 ( 7.38%)
Mean 11 139939.25 ( 0.00%) 159681.25 ( 14.11%)
Mean 12 137545.75 ( 0.00%) 159829.50 ( 16.20%)
Mean 13 133607.25 ( 0.00%) 157809.00 ( 18.11%)
Mean 14 135512.00 ( 0.00%) 153510.50 ( 13.28%)
Mean 15 132730.75 ( 0.00%) 151627.25 ( 14.24%)
Mean 16 129924.25 ( 0.00%) 148248.00 ( 14.10%)
Mean 17 130339.00 ( 0.00%) 149250.00 ( 14.51%)
Mean 18 124314.25 ( 0.00%) 146486.50 ( 17.84%)
Mean 19 120331.25 ( 0.00%) 143616.75 ( 19.35%)
Mean 20 118827.25 ( 0.00%) 141381.50 ( 18.98%)
Mean 21 120938.25 ( 0.00%) 138196.75 ( 14.27%)
Mean 22 118660.75 ( 0.00%) 136879.50 ( 15.35%)
Mean 23 117005.75 ( 0.00%) 134200.50 ( 14.70%)
Mean 24 112711.50 ( 0.00%) 131302.50 ( 16.49%)
Mean 25 115458.50 ( 0.00%) 129939.25 ( 12.54%)
Mean 26 114008.50 ( 0.00%) 128834.50 ( 13.00%)
Mean 27 115063.50 ( 0.00%) 128394.00 ( 11.59%)
Mean 28 114359.50 ( 0.00%) 124072.50 ( 8.49%)
Mean 29 113637.50 ( 0.00%) 124954.50 ( 9.96%)
Mean 30 113392.75 ( 0.00%) 123941.75 ( 9.30%)
Mean 31 115131.25 ( 0.00%) 121477.75 ( 5.51%)
Mean 32 112004.00 ( 0.00%) 122235.00 ( 9.13%)
Mean 33 111287.50 ( 0.00%) 120992.50 ( 8.72%)
Mean 34 111206.75 ( 0.00%) 118769.75 ( 6.80%)
Mean 35 108469.50 ( 0.00%) 120061.50 ( 10.69%)
Mean 36 105932.00 ( 0.00%) 118039.75 ( 11.43%)
Mean 37 107428.00 ( 0.00%) 118295.75 ( 10.12%)
Mean 38 102804.75 ( 0.00%) 120519.50 ( 17.23%)
Mean 39 104095.00 ( 0.00%) 121461.50 ( 16.68%)
Mean 40 103460.00 ( 0.00%) 122506.50 ( 18.41%)
Mean 41 100417.00 ( 0.00%) 118570.50 ( 18.08%)
Mean 42 101025.75 ( 0.00%) 120612.00 ( 19.39%)
Mean 43 100311.75 ( 0.00%) 120743.50 ( 20.37%)
Mean 44 101769.00 ( 0.00%) 120410.25 ( 18.32%)
Mean 45 99649.25 ( 0.00%) 121260.50 ( 21.69%)
Mean 46 101178.50 ( 0.00%) 121210.75 ( 19.80%)
Mean 47 101148.75 ( 0.00%) 119994.25 ( 18.63%)
Mean 48 103446.00 ( 0.00%) 120204.50 ( 16.20%)
Stddev 1 940.15 ( 0.00%) 1277.19 (-35.85%)
Stddev 2 292.47 ( 0.00%) 1851.80 (-533.15%)
Stddev 3 1750.78 ( 0.00%) 1808.61 ( -3.30%)
Stddev 4 859.01 ( 0.00%) 2790.10 (-224.80%)
Stddev 5 3236.13 ( 0.00%) 1892.19 ( 41.53%)
Stddev 6 2489.07 ( 0.00%) 2157.76 ( 13.31%)
Stddev 7 1981.85 ( 0.00%) 4299.27 (-116.93%)
Stddev 8 2586.24 ( 0.00%) 3090.27 (-19.49%)
Stddev 9 7250.82 ( 0.00%) 4762.66 ( 34.32%)
Stddev 10 1242.89 ( 0.00%) 1448.14 (-16.51%)
Stddev 11 1631.31 ( 0.00%) 9758.25 (-498.19%)
Stddev 12 1964.66 ( 0.00%) 17425.60 (-786.95%)
Stddev 13 2080.24 ( 0.00%) 17824.45 (-756.84%)
Stddev 14 1362.07 ( 0.00%) 18551.85 (-1262.03%)
Stddev 15 3142.86 ( 0.00%) 20410.21 (-549.42%)
Stddev 16 2026.28 ( 0.00%) 19767.72 (-875.57%)
Stddev 17 2059.98 ( 0.00%) 19358.07 (-839.72%)
Stddev 18 2832.80 ( 0.00%) 19434.41 (-586.05%)
Stddev 19 4248.17 ( 0.00%) 19590.94 (-361.16%)
Stddev 20 3163.70 ( 0.00%) 18608.43 (-488.19%)
Stddev 21 1046.22 ( 0.00%) 17766.10 (-1598.13%)
Stddev 22 1458.72 ( 0.00%) 16295.25 (-1017.09%)
Stddev 23 1453.80 ( 0.00%) 16933.28 (-1064.76%)
Stddev 24 3387.76 ( 0.00%) 17276.97 (-409.98%)
Stddev 25 467.26 ( 0.00%) 17228.85 (-3587.21%)
Stddev 26 269.10 ( 0.00%) 17614.19 (-6445.71%)
Stddev 27 1024.92 ( 0.00%) 16197.85 (-1480.40%)
Stddev 28 2547.19 ( 0.00%) 22532.91 (-784.62%)
Stddev 29 2496.51 ( 0.00%) 21734.79 (-770.61%)
Stddev 30 1777.21 ( 0.00%) 22407.22 (-1160.81%)
Stddev 31 2948.17 ( 0.00%) 22046.59 (-647.81%)
Stddev 32 3045.75 ( 0.00%) 21317.50 (-599.91%)
Stddev 33 3088.42 ( 0.00%) 24073.34 (-679.47%)
Stddev 34 1695.86 ( 0.00%) 25483.66 (-1402.69%)
Stddev 35 2392.89 ( 0.00%) 22319.81 (-832.76%)
Stddev 36 1002.99 ( 0.00%) 24788.30 (-2371.43%)
Stddev 37 1246.07 ( 0.00%) 22969.98 (-1743.39%)
Stddev 38 3340.47 ( 0.00%) 17764.75 (-431.80%)
Stddev 39 951.45 ( 0.00%) 17467.43 (-1735.88%)
Stddev 40 1861.87 ( 0.00%) 16746.88 (-799.47%)
Stddev 41 3019.63 ( 0.00%) 22203.85 (-635.32%)
Stddev 42 3305.80 ( 0.00%) 19226.07 (-481.59%)
Stddev 43 2149.96 ( 0.00%) 19788.85 (-820.43%)
Stddev 44 4743.81 ( 0.00%) 20232.47 (-326.50%)
Stddev 45 3701.87 ( 0.00%) 19876.40 (-436.93%)
Stddev 46 3742.49 ( 0.00%) 17963.46 (-379.99%)
Stddev 47 1637.98 ( 0.00%) 20138.13 (-1129.45%)
Stddev 48 2192.84 ( 0.00%) 16729.79 (-662.93%)
TPut 1 123600.00 ( 0.00%) 118166.00 ( -4.40%)
TPut 2 251282.00 ( 0.00%) 253321.00 ( 0.81%)
TPut 3 371212.00 ( 0.00%) 370519.00 ( -0.19%)
TPut 4 476489.00 ( 0.00%) 487927.00 ( 2.40%)
TPut 5 569564.00 ( 0.00%) 593162.00 ( 4.14%)
TPut 6 604292.00 ( 0.00%) 679295.00 ( 12.41%)
TPut 7 610474.00 ( 0.00%) 665644.00 ( 9.04%)
TPut 8 565137.00 ( 0.00%) 612888.00 ( 8.45%)
TPut 9 544221.00 ( 0.00%) 557050.00 ( 2.36%)
TPut 10 497162.00 ( 0.00%) 533858.00 ( 7.38%)
TPut 11 559757.00 ( 0.00%) 638725.00 ( 14.11%)
TPut 12 550183.00 ( 0.00%) 639318.00 ( 16.20%)
TPut 13 534429.00 ( 0.00%) 631236.00 ( 18.11%)
TPut 14 542048.00 ( 0.00%) 614042.00 ( 13.28%)
TPut 15 530923.00 ( 0.00%) 606509.00 ( 14.24%)
TPut 16 519697.00 ( 0.00%) 592992.00 ( 14.10%)
TPut 17 521356.00 ( 0.00%) 597000.00 ( 14.51%)
TPut 18 497257.00 ( 0.00%) 585946.00 ( 17.84%)
TPut 19 481325.00 ( 0.00%) 574467.00 ( 19.35%)
TPut 20 475309.00 ( 0.00%) 565526.00 ( 18.98%)
TPut 21 483753.00 ( 0.00%) 552787.00 ( 14.27%)
TPut 22 474643.00 ( 0.00%) 547518.00 ( 15.35%)
TPut 23 468023.00 ( 0.00%) 536802.00 ( 14.70%)
TPut 24 450846.00 ( 0.00%) 525210.00 ( 16.49%)
TPut 25 461834.00 ( 0.00%) 519757.00 ( 12.54%)
TPut 26 456034.00 ( 0.00%) 515338.00 ( 13.00%)
TPut 27 460254.00 ( 0.00%) 513576.00 ( 11.59%)
TPut 28 457438.00 ( 0.00%) 496290.00 ( 8.49%)
TPut 29 454550.00 ( 0.00%) 499818.00 ( 9.96%)
TPut 30 453571.00 ( 0.00%) 495767.00 ( 9.30%)
TPut 31 460525.00 ( 0.00%) 485911.00 ( 5.51%)
TPut 32 448016.00 ( 0.00%) 488940.00 ( 9.13%)
TPut 33 445150.00 ( 0.00%) 483970.00 ( 8.72%)
TPut 34 444827.00 ( 0.00%) 475079.00 ( 6.80%)
TPut 35 433878.00 ( 0.00%) 480246.00 ( 10.69%)
TPut 36 423728.00 ( 0.00%) 472159.00 ( 11.43%)
TPut 37 429712.00 ( 0.00%) 473183.00 ( 10.12%)
TPut 38 411219.00 ( 0.00%) 482078.00 ( 17.23%)
TPut 39 416380.00 ( 0.00%) 485846.00 ( 16.68%)
TPut 40 413840.00 ( 0.00%) 490026.00 ( 18.41%)
TPut 41 401668.00 ( 0.00%) 474282.00 ( 18.08%)
TPut 42 404103.00 ( 0.00%) 482448.00 ( 19.39%)
TPut 43 401247.00 ( 0.00%) 482974.00 ( 20.37%)
TPut 44 407076.00 ( 0.00%) 481641.00 ( 18.32%)
TPut 45 398597.00 ( 0.00%) 485042.00 ( 21.69%)
TPut 46 404714.00 ( 0.00%) 484843.00 ( 19.80%)
TPut 47 404595.00 ( 0.00%) 479977.00 ( 18.63%)
TPut 48 413784.00 ( 0.00%) 480818.00 ( 16.20%)This is looking much better overall although I am concerned about the increased variability between JVMs.
specjbb Peaks
3.12.0-rc3 3.12.0-rc3
account-v9 periodretry-v9
Expctd Warehouse 12.00 ( 0.00%) 12.00 ( 0.00%)
Expctd Peak Bops 559757.00 ( 0.00%) 638725.00 ( 14.11%)
Actual Warehouse 8.00 ( 0.00%) 7.00 (-12.50%)
Actual Peak Bops 610474.00 ( 0.00%) 679295.00 ( 11.27%)
SpecJBB Bops 502292.00 ( 0.00%) 582258.00 ( 15.92%)
SpecJBB Bops/JVM 125573.00 ( 0.00%) 145565.00 ( 15.92%)Looking fine.
3.12.0-rc3 3.12.0-rc3
account-v9 periodretry-v9 User 481412.08 481942.54
System 1301.91 578.20
Elapsed 10402.09 10404.47
3.12.0-rc3 3.12.0-rc3
account-v9 periodretry-v9
Compaction cost 105928 13748
NUMA PTE updates 457567880 45890118
NUMA hint faults 69831880 45725506
NUMA hint local faults 19303679 28637898NUMA hint local percent 27 62
NUMA pages migrated 102050548 13244738
AutoNUMA cost 354301 229200
and system CPU usage is still way down so now we are seeing large improvements for less work. Previous tests had indicated that period retrying of task migration was necessary for a good "local percent" of local/remote faults. It implies that the load balancer and NUMA scheduling may be making conflicting decisions.
While there is still plenty of future work it looks like this is ready for wider testing.
Documentation/sysctl/kernel.txt | 76 +++ fs/exec.c | 1 + fs/proc/array.c | 2 + include/linux/cpu.h | 67 ++- include/linux/mempolicy.h | 1 + include/linux/migrate.h | 7 +- include/linux/mm.h | 118 +++- include/linux/mm_types.h | 17 +- include/linux/page-flags-layout.h | 28 +- include/linux/sched.h | 67 ++- include/linux/sched/sysctl.h | 1 - include/linux/stop_machine.h | 1 + kernel/bounds.c | 4 + kernel/cpu.c | 227 ++++++-- kernel/fork.c | 5 +- kernel/sched/core.c | 184 ++++++- kernel/sched/debug.c | 60 +- kernel/sched/fair.c | 1092 ++++++++++++++++++++++++++++++++++--- kernel/sched/features.h | 19 +- kernel/sched/idle_task.c | 2 +- kernel/sched/rt.c | 5 +- kernel/sched/sched.h | 27 +- kernel/sched/stop_task.c | 2 +- kernel/stop_machine.c | 272 +++++---- kernel/sysctl.c | 21 +- mm/huge_memory.c | 119 +++- mm/memory.c | 158 ++---- mm/mempolicy.c | 82 ++- mm/migrate.c | 49 +- mm/mm_init.c | 18 +- mm/mmzone.c | 14 +- mm/mprotect.c | 65 +-- mm/page_alloc.c | 4 +- 33 files changed, 2248 insertions(+), 567 deletions(-)
Signature · 1 line
1.8.4
Mel Gorman
reviewerPATCH 1/63 hotplug: Optimize {get,put}_online_cpus()
From: Peter Zijlstra <[email protected]>
NOTE: This is a placeholder only. A more comprehensive series is in
progress but this patch on its own mitigates most of the overhead the migrate_swap patch is concerned with. It's expected that CPU hotplug locking series would go in before this series.
The current implementation of get_online_cpus() is global of nature and thus not suited for any kind of common usage.
Re-implement the current recursive r/w cpu hotplug lock such that the read side locks are as light as possible.
The current cpu hotplug lock is entirely reader biased; but since readers are expensive there aren't a lot of them about and writer starvation isn't a particular problem.
However by making the reader side more usable there is a fair chance it will get used more and thus the starvation issue becomes a real possibility.
Therefore this new implementation is fair, alternating readers and writers; this however requires per-task state to allow the reader recursion -- this new task_struct member is placed in a 4 byte hole on 64bit builds.
Many comments are contributed by Paul McKenney, and many previous attempts were shown to be inadequate by both Paul and Oleg; many thanks to them for persisting to poke holes in my attempts.
--- include/linux/cpu.h | 67 ++++++++++++++- include/linux/sched.h | 3 + kernel/cpu.c | 227 +++++++++++++++++++++++++++++++++++++------------- kernel/sched/core.c | 2 + 4 files changed, 237 insertions(+), 62 deletions(-)
include/linux/cpu.h +65 -2
@@ -16,6 +16,8 @@#include <linux/node.h> #include <linux/compiler.h> #include <linux/cpumask.h> +#include <linux/percpu.h> +#include <linux/sched.h> struct device ;
@@ -173,10 +175,69 @@ extern struct bus_type cpu_subsys;#ifdef CONFIG_HOTPLUG_CPU /* Stop CPUs going up and down. */ +extern void cpu_hotplug_init_task (struct task_struct *p); +extern void cpu_hotplug_begin (void );extern void cpu_hotplug_done (void ); -extern void get_online_cpus (void ); -extern void put_online_cpus (void ); + +extern int __cpuhp_state; +DECLARE_PER_CPU(unsigned int, __cpuhp_refcount); + +extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{ +might_sleep (); + +/* Support reader recursion */ +/* The value was >= 1 and remains so, reordering causes no harm. */ +if (current->cpuhp_ref++) +return ; + +preempt_disable (); +/* +* We are in an RCU-sched read-side critical section, so the writer +* cannot both change __cpuhp_state from readers_fast and start +* checking counters while we are here. So if we see !__cpuhp_state, +* we know that the writer won't be checking until we past the +* preempt_enable() and that once the synchronize_sched() is done, the +* writer will see anything we did within this RCU-sched read-side +* critical section. +*/ +if (likely (!__cpuhp_state)) +__this_cpu_inc (__cpuhp_refcount); +else +__get_online_cpus ();/* Unconditional memory barrier. */ +preempt_enable (); +/* +* The barrier() from preempt_enable() prevents the compiler from +* bleeding the critical section out. +*/ +} + +extern void __put_online_cpus (void ); + +static inline void put_online_cpus (void ) +{ +/* The value was >= 1 and remains so, reordering causes no harm. */ +if (--current->cpuhp_ref) +return ; + +/* +* The barrier() in preempt_disable() prevents the compiler from +* bleeding the critical section out. +*/ +preempt_disable (); +/* +* Same as in get_online_cpus(). +*/ +if (likely (!__cpuhp_state)) +__this_cpu_dec (__cpuhp_refcount); +else +__put_online_cpus ();/* Unconditional memory barrier. */ +preempt_enable (); +} +extern void cpu_hotplug_disable (void );extern void cpu_hotplug_enable (void );#define hotcpu_notifier (fn, pri) cpu_notifier(fn, pri)
@@ -200,6 +261,8 @@ static inline void cpu_hotplug_driver_unlock(void)#else /* CONFIG_HOTPLUG_CPU */ +static inline void cpu_hotplug_init_task (struct task_struct *p) {} +static inline void cpu_hotplug_begin (void ) {}static inline void cpu_hotplug_done (void ) {}#define get_online_cpus () do { } while (0)
include/linux/sched.h +3 -0
@@ -1026,6 +1026,9 @@ struct task_struct {#ifdef CONFIG_SMP struct llist_node wake_entry;int on_cpu; +#ifdef CONFIG_HOTPLUG_CPU +int cpuhp_ref; +#endif struct task_struct *last_wakee;unsigned long wakee_flips;unsigned long wakee_flip_decay_ts;
kernel/cpu.c +167 -60
255 lines not on this page · raw message
kernel/sched/core.c +2 -0
@@ -1630,6 +1630,8 @@ static void __sched_fork(struct task_struct *p) p->numa_scan_period = sysctl_numa_balancing_scan_delay; p->numa_work.next = &p->numa_work;#endif /* CONFIG_NUMA_BALANCING */ + +cpu_hotplug_init_task (p); }#ifdef CONFIG_NUMA_BALANCING
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 2/63 mm: numa: Document automatic NUMA balancing sysctls
In reply to Mel Gorman--- Documentation/sysctl/kernel.txt | 66 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+)
Documentation/sysctl/kernel.txt +66 -0
@@ -355,6 +355,72 @@ utilize. ============================================================== +numa_balancing + +Enables/disables automatic page fault based NUMA memory +balancing. Memory is moved automatically to nodes +that access it often. + +Enables/disables automatic NUMA memory balancing. On NUMA machines, there +is a performance penalty if remote memory is accessed by a CPU. When this +feature is enabled the kernel samples what task thread is accessing memory +by periodically unmapping pages and later trapping a page fault. At the +time of the page fault, it is determined if the data being accessed should +be migrated to a local memory node. + +The unmapping of pages and trapping faults incur additional overhead that +ideally is offset by improved memory locality but there is no universal +guarantee. If the target workload is already bound to NUMA nodes then this +feature should be disabled. Otherwise, if the system overhead from the +feature is too high then the rate the kernel samples for NUMA hinting +faults may be controlled by the numa_balancing_scan_period_min_ms, +numa_balancing_scan_delay_ms, numa_balancing_scan_period_reset, +numa_balancing_scan_period_max_ms and numa_balancing_scan_size_mb sysctls. + +============================================================== + +numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, +numa_balancing_scan_period_max_ms, numa_balancing_scan_period_reset, +numa_balancing_scan_size_mb + +Automatic NUMA balancing scans tasks address space and unmaps pages to +detect if pages are properly placed or if the data should be migrated to a +memory node local to where the task is running. Every "scan delay" the task +scans the next "scan size" number of pages in its address space. When the +end of the address space is reached the scanner restarts from the beginning. + +In combination, the "scan delay" and "scan size" determine the scan rate. +When "scan delay" decreases, the scan rate increases. The scan delay and +hence the scan rate of every task is adaptive and depends on historical +behaviour. If pages are properly placed then the scan delay increases, +otherwise the scan delay decreases. The "scan size" is not adaptive but +the higher the "scan size", the higher the scan rate. + +Higher scan rates incur higher system overhead as page faults must be +trapped and potentially data must be migrated. However, the higher the scan +rate, the more quickly a tasks memory is migrated to a local node if the +workload pattern changes and minimises performance impact due to remote +memory accesses. These sysctls control the thresholds for scan delays and +the number of pages scanned. + +numa_balancing_scan_period_min_ms is the minimum delay in milliseconds +between scans. It effectively controls the maximum scanning rate for +each task. + +numa_balancing_scan_delay_ms is the starting "scan delay" used for a task +when it initially forks. + +numa_balancing_scan_period_max_ms is the maximum delay between scans. It +effectively controls the minimum scanning rate for each task. + +numa_balancing_scan_size_mb is how many megabytes worth of pages are +scanned for a given scan. + +numa_balancing_scan_period_reset is a blunt instrument that controls how +often a tasks scan delay is reset to detect sudden changes in task behaviour. + +============================================================== + osrelease, ostype & version: # cat osrelease
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 02/63] mm: numa: Document automatic NUMA balancing sysctls
Quoted 2 lines · Mel Gorman
Cc stable <[email protected]>Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] mm: numa: Document automatic NUMA balancing sysctls
In reply to Mel Gormanmm: numa: Document automatic NUMA balancing sysctls
Commit-ID: 10fc05d0e551146ad6feb0ab8902d28a2d3c5624 Gitweb: http://git.kernel.org/tip/10fc05d0e551146ad6feb0ab8902d28a2d3c5624 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:40 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:39:20 +0200
mm: numa: Document automatic NUMA balancing sysctls
--- Documentation/sysctl/kernel.txt | 66 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+)
Documentation/sysctl/kernel.txt +66 -0
@@ -355,6 +355,72 @@ utilize. ============================================================== +numa_balancing + +Enables/disables automatic page fault based NUMA memory +balancing. Memory is moved automatically to nodes +that access it often. + +Enables/disables automatic NUMA memory balancing. On NUMA machines, there +is a performance penalty if remote memory is accessed by a CPU. When this +feature is enabled the kernel samples what task thread is accessing memory +by periodically unmapping pages and later trapping a page fault. At the +time of the page fault, it is determined if the data being accessed should +be migrated to a local memory node. + +The unmapping of pages and trapping faults incur additional overhead that +ideally is offset by improved memory locality but there is no universal +guarantee. If the target workload is already bound to NUMA nodes then this +feature should be disabled. Otherwise, if the system overhead from the +feature is too high then the rate the kernel samples for NUMA hinting +faults may be controlled by the numa_balancing_scan_period_min_ms, +numa_balancing_scan_delay_ms, numa_balancing_scan_period_reset, +numa_balancing_scan_period_max_ms and numa_balancing_scan_size_mb sysctls. + +============================================================== + +numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, +numa_balancing_scan_period_max_ms, numa_balancing_scan_period_reset, +numa_balancing_scan_size_mb + +Automatic NUMA balancing scans tasks address space and unmaps pages to +detect if pages are properly placed or if the data should be migrated to a +memory node local to where the task is running. Every "scan delay" the task +scans the next "scan size" number of pages in its address space. When the +end of the address space is reached the scanner restarts from the beginning. + +In combination, the "scan delay" and "scan size" determine the scan rate. +When "scan delay" decreases, the scan rate increases. The scan delay and +hence the scan rate of every task is adaptive and depends on historical +behaviour. If pages are properly placed then the scan delay increases, +otherwise the scan delay decreases. The "scan size" is not adaptive but +the higher the "scan size", the higher the scan rate. + +Higher scan rates incur higher system overhead as page faults must be +trapped and potentially data must be migrated. However, the higher the scan +rate, the more quickly a tasks memory is migrated to a local node if the +workload pattern changes and minimises performance impact due to remote +memory accesses. These sysctls control the thresholds for scan delays and +the number of pages scanned. + +numa_balancing_scan_period_min_ms is the minimum delay in milliseconds +between scans. It effectively controls the maximum scanning rate for +each task. + +numa_balancing_scan_delay_ms is the starting "scan delay" used for a task +when it initially forks. + +numa_balancing_scan_period_max_ms is the maximum delay between scans. It +effectively controls the minimum scanning rate for each task. + +numa_balancing_scan_size_mb is how many megabytes worth of pages are +scanned for a given scan. + +numa_balancing_scan_period_reset is a blunt instrument that controls how +often a tasks scan delay is reset to detect sudden changes in task behaviour. + +============================================================== + osrelease, ostype & version: # cat osrelease
Mel Gorman
reviewerPATCH 3/63 sched, numa: Comment fixlets
In reply to Mel GormanFrom: Peter Zijlstra <[email protected]>
Fix a 80 column violation and a PTE vs PMD reference.
--- kernel/sched/fair.c | 8 ++++---- mm/huge_memory.c | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-)
kernel/sched/fair.c +4 -4
@@ -988,10 +988,10 @@ void task_numa_work(struct callback_head *work) out:/* -* It is possible to reach the end of the VMA list but the last few VMAs are -* not guaranteed to the vma_migratable. If they are not, we would find the -* !migratable VMA on the next scan but not reset the scanner to the start -* so check it now. +* It is possible to reach the end of the VMA list but the last few +* VMAs are not guaranteed to the vma_migratable. If they are not, we +* would find the !migratable VMA on the next scan but not reset the +* scanner to the start so check it now. */ if (vma) mm->numa_scan_offset = start;
mm/huge_memory.c +1 -1
@@ -1305,7 +1305,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,spin_unlock (&mm->page_table_lock);lock_page (page); -/* Confirm the PTE did not while locked */ +/* Confirm the PMD did not change while page_table_lock was released */ spin_lock (&mm->page_table_lock);if (unlikely (!pmd_same (pmd, *pmdp))) {unlock_page (page);
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 03/63] sched, numa: Comment fixlets
Quoted 7 lines · Mel Gorman
From: Peter Zijlstra <[email protected]>
Fix a 80 column violation and a PTE vs PMD reference.
Cc stable <[email protected]>Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Peter Zijlstra
merge bot[tip:sched/core] sched/numa: Fix comments
In reply to Mel Gormansched/numa: Fix comments
Commit-ID: c69307d533d7aa7cc8894dbbb8a274599f8630d7 Gitweb: http://git.kernel.org/tip/c69307d533d7aa7cc8894dbbb8a274599f8630d7 Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:41 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:39:30 +0200
sched/numa: Fix comments
Fix a 80 column violation and a PTE vs PMD reference.
--- kernel/sched/fair.c | 8 ++++---- mm/huge_memory.c | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-)
kernel/sched/fair.c +4 -4
@@ -988,10 +988,10 @@ void task_numa_work(struct callback_head *work) out:/* -* It is possible to reach the end of the VMA list but the last few VMAs are -* not guaranteed to the vma_migratable. If they are not, we would find the -* !migratable VMA on the next scan but not reset the scanner to the start -* so check it now. +* It is possible to reach the end of the VMA list but the last few +* VMAs are not guaranteed to the vma_migratable. If they are not, we +* would find the !migratable VMA on the next scan but not reset the +* scanner to the start so check it now. */ if (vma) mm->numa_scan_offset = start;
mm/huge_memory.c +1 -1
@@ -1305,7 +1305,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,spin_unlock (&mm->page_table_lock);lock_page (page); -/* Confirm the PTE did not while locked */ +/* Confirm the PMD did not change while page_table_lock was released */ spin_lock (&mm->page_table_lock);if (unlikely (!pmd_same (pmd, *pmdp))) {unlock_page (page);
Mel Gorman
reviewerPATCH 4/63 mm: numa: Do not account for a hinting fault if we raced
In reply to Mel GormanIf another task handled a hinting fault in parallel then do not double account for it.
--- mm/huge_memory.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
mm/huge_memory.c +4 -1
@@ -1325,8 +1325,11 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, check_same:spin_lock (&mm->page_table_lock); -if (unlikely (!pmd_same (pmd, *pmdp))) +if (unlikely (!pmd_same (pmd, *pmdp))) { +/* Someone else took our fault */ + current_nid =-1 ; goto out_unlock; + } clear_pmdnuma: pmd =pmd_mknonnuma (pmd);set_pmd_at (mm, haddr, pmdp, pmd);
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 04/63] mm: numa: Do not account for a hinting fault if we raced
Quoted 6 lines · Mel Gorman
If another task handled a hinting fault in parallel then do not double account for it.
Cc stable <[email protected]>Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] mm: numa: Do not account for a hinting fault if we raced
In reply to Mel Gormanmm: numa: Do not account for a hinting fault if we raced
Commit-ID: 0c3a775e1e0b069bf765f8355b723ce0d18dcc6c Gitweb: http://git.kernel.org/tip/0c3a775e1e0b069bf765f8355b723ce0d18dcc6c Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:42 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:39:40 +0200
mm: numa: Do not account for a hinting fault if we raced
If another task handled a hinting fault in parallel then do not double account for it.
--- mm/huge_memory.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
mm/huge_memory.c +4 -1
@@ -1325,8 +1325,11 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, check_same:spin_lock (&mm->page_table_lock); -if (unlikely (!pmd_same (pmd, *pmdp))) +if (unlikely (!pmd_same (pmd, *pmdp))) { +/* Someone else took our fault */ + current_nid =-1 ; goto out_unlock; + } clear_pmdnuma: pmd =pmd_mknonnuma (pmd);set_pmd_at (mm, haddr, pmdp, pmd);
tip-bot for Mel Gorman
merge bot[tip:core/urgent] mm: numa: Do not account for a hinting fault if we raced
In reply to Mel Gormanmm: numa: Do not account for a hinting fault if we raced
Commit-ID: 1dd49bfa3465756b3ce72214b58a33e4afb67aa3 Gitweb: http://git.kernel.org/tip/1dd49bfa3465756b3ce72214b58a33e4afb67aa3 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:42 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 29 Oct 2013 11:37:05 +0100
mm: numa: Do not account for a hinting fault if we raced
If another task handled a hinting fault in parallel then do not double account for it.
--- mm/huge_memory.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
mm/huge_memory.c +4 -1
@@ -1325,8 +1325,11 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, check_same:spin_lock (&mm->page_table_lock); -if (unlikely (!pmd_same (pmd, *pmdp))) +if (unlikely (!pmd_same (pmd, *pmdp))) { +/* Someone else took our fault */ + current_nid =-1 ; goto out_unlock; + } clear_pmdnuma: pmd =pmd_mknonnuma (pmd);set_pmd_at (mm, haddr, pmdp, pmd);
Mel Gorman
reviewerPATCH 5/63 mm: Wait for THP migrations to complete during NUMA hinting faults
In reply to Mel GormanThe locking for migrating THP is unusual. While normal page migration prevents parallel accesses using a migration PTE, THP migration relies on a combination of the page_table_lock, the page lock and the existance of the NUMA hinting PTE to guarantee safety but there is a bug in the scheme.
If a THP page is currently being migrated and another thread traps a fault on the same page it checks if the page is misplaced. If it is not, then pmd_numa is cleared. The problem is that it checks if the page is misplaced without holding the page lock meaning that the racing thread can be migrating the THP when the second thread clears the NUMA bit and faults a stale page.
This patch checks if the page is potentially being migrated and stalls using the lock_page if it is potentially being migrated before checking if the page is misplaced or not.
--- mm/huge_memory.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-)
mm/huge_memory.c +16 -7
@@ -1295,13 +1295,14 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,if (current_nid ==numa_node_id ())count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL ); - target_nid =mpol_misplaced (page, vma, haddr); -if (target_nid ==-1 ) { -put_page (page); - goto clear_pmdnuma; - } +/* +* Acquire the page lock to serialise THP migrations but avoid dropping +* page_table_lock if at all possible +*/ +if (trylock_page (page)) + goto got_lock; -/* Acquire the page lock to serialise THP migrations */ +/* Serialise against migrationa and check placement check placement */ spin_unlock (&mm->page_table_lock);lock_page (page);
@@ -1312,9 +1313,17 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,put_page (page); goto out_unlock; } -spin_unlock (&mm->page_table_lock); + +got_lock: + target_nid =mpol_misplaced (page, vma, haddr); +if (target_nid ==-1 ) { +unlock_page (page); +put_page (page); + goto clear_pmdnuma; + }/* Migrate the THP to the requested node */ +spin_unlock (&mm->page_table_lock); migrated =migrate_misplaced_transhuge_page (mm, vma, pmdp, pmd, addr, page, target_nid);if (!migrated)
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 05/63] mm: Wait for THP migrations to complete during NUMA hinting faults
Quoted 12 lines · Mel Gorman
The locking for migrating THP is unusual. While normal page migration prevents parallel accesses using a migration PTE, THP migration relies on a combination of the
page_table_lock, the page lock and the existance of the NUMA hinting PTE to guarantee safety but there is a bug in the scheme.If a THP page is currently being migrated and another thread traps a fault on the same page it checks if the page is misplaced. If it is not, then
pmd_numais cleared. The problem is that it checks if the page is misplaced without holding the page lock meaning that the racing thread can be migrating the THP when the second thread clears the NUMA bit and faults a stale page.
Quoted 7 lines
This patch checks if the page is potentially being migrated and stalls using the
lock_pageif it is potentially being migrated before checking if the page is misplaced or not.Cc stable <[email protected]>Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] mm: Wait for THP migrations to complete during NUMA hinting faults
In reply to Mel Gormanmm: Wait for THP migrations to complete during NUMA hinting faults
Commit-ID: ff9042b11a71c81238c70af168cd36b98a6d5a3c Gitweb: http://git.kernel.org/tip/ff9042b11a71c81238c70af168cd36b98a6d5a3c Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:43 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:39:41 +0200
mm: Wait for THP migrations to complete during NUMA hinting faults
The locking for migrating THP is unusual. While normal page migration prevents parallel accesses using a migration PTE, THP migration relies on a combination of the page_table_lock, the page lock and the existance of the NUMA hinting PTE to guarantee safety but there is a bug in the scheme.
If a THP page is currently being migrated and another thread traps a fault on the same page it checks if the page is misplaced. If it is not, then pmd_numa is cleared. The problem is that it checks if the page is misplaced without holding the page lock meaning that the racing thread can be migrating the THP when the second thread clears the NUMA bit and faults a stale page.
This patch checks if the page is potentially being migrated and stalls using the lock_page if it is potentially being migrated before checking if the page is misplaced or not.
--- mm/huge_memory.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-)
mm/huge_memory.c +16 -7
@@ -1295,13 +1295,14 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,if (current_nid ==numa_node_id ())count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL ); - target_nid =mpol_misplaced (page, vma, haddr); -if (target_nid ==-1 ) { -put_page (page); - goto clear_pmdnuma; - } +/* +* Acquire the page lock to serialise THP migrations but avoid dropping +* page_table_lock if at all possible +*/ +if (trylock_page (page)) + goto got_lock; -/* Acquire the page lock to serialise THP migrations */ +/* Serialise against migrationa and check placement check placement */ spin_unlock (&mm->page_table_lock);lock_page (page);
@@ -1312,9 +1313,17 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,put_page (page); goto out_unlock; } -spin_unlock (&mm->page_table_lock); + +got_lock: + target_nid =mpol_misplaced (page, vma, haddr); +if (target_nid ==-1 ) { +unlock_page (page); +put_page (page); + goto clear_pmdnuma; + }/* Migrate the THP to the requested node */ +spin_unlock (&mm->page_table_lock); migrated =migrate_misplaced_transhuge_page (mm, vma, pmdp, pmd, addr, page, target_nid);if (!migrated)
tip-bot for Mel Gorman
merge bot[tip:core/urgent] mm: Wait for THP migrations to complete during NUMA hinting faults
In reply to Mel Gormanmm: Wait for THP migrations to complete during NUMA hinting faults
Commit-ID: 42836f5f8baa33085f547098b74aa98991ee9216 Gitweb: http://git.kernel.org/tip/42836f5f8baa33085f547098b74aa98991ee9216 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:43 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 29 Oct 2013 11:37:19 +0100
mm: Wait for THP migrations to complete during NUMA hinting faults
The locking for migrating THP is unusual. While normal page migration prevents parallel accesses using a migration PTE, THP migration relies on a combination of the page_table_lock, the page lock and the existance of the NUMA hinting PTE to guarantee safety but there is a bug in the scheme.
If a THP page is currently being migrated and another thread traps a fault on the same page it checks if the page is misplaced. If it is not, then pmd_numa is cleared. The problem is that it checks if the page is misplaced without holding the page lock meaning that the racing thread can be migrating the THP when the second thread clears the NUMA bit and faults a stale page.
This patch checks if the page is potentially being migrated and stalls using the lock_page if it is potentially being migrated before checking if the page is misplaced or not.
--- mm/huge_memory.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-)
mm/huge_memory.c +16 -7
@@ -1295,13 +1295,14 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,if (current_nid ==numa_node_id ())count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL ); - target_nid =mpol_misplaced (page, vma, haddr); -if (target_nid ==-1 ) { -put_page (page); - goto clear_pmdnuma; - } +/* +* Acquire the page lock to serialise THP migrations but avoid dropping +* page_table_lock if at all possible +*/ +if (trylock_page (page)) + goto got_lock; -/* Acquire the page lock to serialise THP migrations */ +/* Serialise against migrationa and check placement check placement */ spin_unlock (&mm->page_table_lock);lock_page (page);
@@ -1312,9 +1313,17 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,put_page (page); goto out_unlock; } -spin_unlock (&mm->page_table_lock); + +got_lock: + target_nid =mpol_misplaced (page, vma, haddr); +if (target_nid ==-1 ) { +unlock_page (page); +put_page (page); + goto clear_pmdnuma; + }/* Migrate the THP to the requested node */ +spin_unlock (&mm->page_table_lock); migrated =migrate_misplaced_transhuge_page (mm, vma, pmdp, pmd, addr, page, target_nid);if (!migrated)
Mel Gorman
reviewerPATCH 6/63 mm: Prevent parallel splits during THP migration
In reply to Mel GormanTHP migrations are serialised by the page lock but on its own that does not prevent THP splits. If the page is split during THP migration then the pmd_same checks will prevent page table corruption but the unlock page and other fix-ups potentially will cause corruption. This patch takes the anon_vma lock to prevent parallel splits during migration.
--- mm/huge_memory.c | 44 ++++++++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 14 deletions(-)
mm/huge_memory.c +30 -14
@@ -1278,18 +1278,18 @@ out:int do_huge_pmd_numa_page (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long addr,pmd_t pmd,pmd_t *pmdp) { +struct anon_vma *anon_vma =NULL ;struct page *page;unsigned long haddr = addr &HPAGE_PMD_MASK ;int target_nid;int current_nid =-1 ; -bool migrated; +bool migrated, page_locked;spin_lock (&mm->page_table_lock);if (unlikely (!pmd_same (pmd, *pmdp))) goto out_unlock; page =pmd_page (pmd); -get_page (page); current_nid =page_to_nid (page);count_vm_numa_event (NUMA_HINT_FAULTS );if (current_nid ==numa_node_id ())
@@ -1299,12 +1299,29 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,* Acquire the page lock to serialise THP migrations but avoid dropping * page_table_lock if at all possible */ -if (trylock_page (page)) - goto got_lock; + page_locked =trylock_page (page); + target_nid =mpol_misplaced (page, vma, haddr); +if (target_nid ==-1 ) { +/* If the page was locked, there are no parallel migrations */ +if (page_locked) { +unlock_page (page); + goto clear_pmdnuma; + } -/* Serialise against migrationa and check placement check placement */ +/* Otherwise wait for potential migrations and retry fault */ +spin_unlock (&mm->page_table_lock); +wait_on_page_locked (page); + goto out; + } + +/* Page is misplaced, serialise migrations and parallel THP splits */ +get_page (page);spin_unlock (&mm->page_table_lock); -lock_page (page); +if (!page_locked) { +lock_page (page); + page_locked = true; + } + anon_vma =page_lock_anon_vma_read (page);/* Confirm the PMD did not change while page_table_lock was released */ spin_lock (&mm->page_table_lock);
@@ -1314,14 +1331,6 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, goto out_unlock; } -got_lock: - target_nid =mpol_misplaced (page, vma, haddr); -if (target_nid ==-1 ) { -unlock_page (page); -put_page (page); - goto clear_pmdnuma; - } -/* Migrate the THP to the requested node */ spin_unlock (&mm->page_table_lock); migrated = migrate_misplaced_transhuge_page(mm, vma,
@@ -1330,6 +1339,8 @@ got_lock: goto check_same;task_numa_fault (target_nid,HPAGE_PMD_NR , true); +if (anon_vma) +page_unlock_anon_vma_read (anon_vma);return 0 ; check_same:
@@ -1346,6 +1357,11 @@ clear_pmdnuma:update_mmu_cache_pmd (vma, addr, pmdp); out_unlock:spin_unlock (&mm->page_table_lock); + +out: +if (anon_vma) +page_unlock_anon_vma_read (anon_vma); +if (current_nid !=-1 )task_numa_fault (current_nid,HPAGE_PMD_NR , false);return 0 ;
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 06/63] mm: Prevent parallel splits during THP migration
Quoted 8 lines · Mel Gorman
THP migrations are serialised by the page lock but on its own that does not prevent THP splits. If the page is split during THP migration then the
pmd_samechecks will prevent page table corruption but the unlock page and other fix-ups potentially will cause corruption. This patch takes theanon_vmalock to prevent parallel splits during migration.Cc stable <[email protected]>Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] mm: Prevent parallel splits during THP migration
In reply to Mel Gormanmm: Prevent parallel splits during THP migration
Commit-ID: b8916634b77bffb233d8f2f45703c80343457cc1 Gitweb: http://git.kernel.org/tip/b8916634b77bffb233d8f2f45703c80343457cc1 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:44 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:39:43 +0200
mm: Prevent parallel splits during THP migration
THP migrations are serialised by the page lock but on its own that does not prevent THP splits. If the page is split during THP migration then the pmd_same checks will prevent page table corruption but the unlock page and other fix-ups potentially will cause corruption. This patch takes the anon_vma lock to prevent parallel splits during migration.
--- mm/huge_memory.c | 44 ++++++++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 14 deletions(-)
mm/huge_memory.c +30 -14
@@ -1278,18 +1278,18 @@ out:int do_huge_pmd_numa_page (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long addr,pmd_t pmd,pmd_t *pmdp) { +struct anon_vma *anon_vma =NULL ;struct page *page;unsigned long haddr = addr &HPAGE_PMD_MASK ;int target_nid;int current_nid =-1 ; -bool migrated; +bool migrated, page_locked;spin_lock (&mm->page_table_lock);if (unlikely (!pmd_same (pmd, *pmdp))) goto out_unlock; page =pmd_page (pmd); -get_page (page); current_nid =page_to_nid (page);count_vm_numa_event (NUMA_HINT_FAULTS );if (current_nid ==numa_node_id ())
@@ -1299,12 +1299,29 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,* Acquire the page lock to serialise THP migrations but avoid dropping * page_table_lock if at all possible */ -if (trylock_page (page)) - goto got_lock; + page_locked =trylock_page (page); + target_nid =mpol_misplaced (page, vma, haddr); +if (target_nid ==-1 ) { +/* If the page was locked, there are no parallel migrations */ +if (page_locked) { +unlock_page (page); + goto clear_pmdnuma; + } -/* Serialise against migrationa and check placement check placement */ +/* Otherwise wait for potential migrations and retry fault */ +spin_unlock (&mm->page_table_lock); +wait_on_page_locked (page); + goto out; + } + +/* Page is misplaced, serialise migrations and parallel THP splits */ +get_page (page);spin_unlock (&mm->page_table_lock); -lock_page (page); +if (!page_locked) { +lock_page (page); + page_locked = true; + } + anon_vma =page_lock_anon_vma_read (page);/* Confirm the PMD did not change while page_table_lock was released */ spin_lock (&mm->page_table_lock);
@@ -1314,14 +1331,6 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, goto out_unlock; } -got_lock: - target_nid =mpol_misplaced (page, vma, haddr); -if (target_nid ==-1 ) { -unlock_page (page); -put_page (page); - goto clear_pmdnuma; - } -/* Migrate the THP to the requested node */ spin_unlock (&mm->page_table_lock); migrated = migrate_misplaced_transhuge_page(mm, vma,
@@ -1330,6 +1339,8 @@ got_lock: goto check_same;task_numa_fault (target_nid,HPAGE_PMD_NR , true); +if (anon_vma) +page_unlock_anon_vma_read (anon_vma);return 0 ; check_same:
@@ -1346,6 +1357,11 @@ clear_pmdnuma:update_mmu_cache_pmd (vma, addr, pmdp); out_unlock:spin_unlock (&mm->page_table_lock); + +out: +if (anon_vma) +page_unlock_anon_vma_read (anon_vma); +if (current_nid !=-1 )task_numa_fault (current_nid,HPAGE_PMD_NR , false);return 0 ;
tip-bot for Mel Gorman
merge bot[tip:core/urgent] mm: Prevent parallel splits during THP migration
In reply to Mel Gormanmm: Prevent parallel splits during THP migration
Commit-ID: 587fe586f44a48f9691001ba6c45b86c8e4ba21f Gitweb: http://git.kernel.org/tip/587fe586f44a48f9691001ba6c45b86c8e4ba21f Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:44 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 29 Oct 2013 11:37:39 +0100
mm: Prevent parallel splits during THP migration
THP migrations are serialised by the page lock but on its own that does not prevent THP splits. If the page is split during THP migration then the pmd_same checks will prevent page table corruption but the unlock page and other fix-ups potentially will cause corruption. This patch takes the anon_vma lock to prevent parallel splits during migration.
--- mm/huge_memory.c | 44 ++++++++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 14 deletions(-)
mm/huge_memory.c +30 -14
@@ -1278,18 +1278,18 @@ out:int do_huge_pmd_numa_page (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long addr,pmd_t pmd,pmd_t *pmdp) { +struct anon_vma *anon_vma =NULL ;struct page *page;unsigned long haddr = addr &HPAGE_PMD_MASK ;int target_nid;int current_nid =-1 ; -bool migrated; +bool migrated, page_locked;spin_lock (&mm->page_table_lock);if (unlikely (!pmd_same (pmd, *pmdp))) goto out_unlock; page =pmd_page (pmd); -get_page (page); current_nid =page_to_nid (page);count_vm_numa_event (NUMA_HINT_FAULTS );if (current_nid ==numa_node_id ())
@@ -1299,12 +1299,29 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,* Acquire the page lock to serialise THP migrations but avoid dropping * page_table_lock if at all possible */ -if (trylock_page (page)) - goto got_lock; + page_locked =trylock_page (page); + target_nid =mpol_misplaced (page, vma, haddr); +if (target_nid ==-1 ) { +/* If the page was locked, there are no parallel migrations */ +if (page_locked) { +unlock_page (page); + goto clear_pmdnuma; + } -/* Serialise against migrationa and check placement check placement */ +/* Otherwise wait for potential migrations and retry fault */ +spin_unlock (&mm->page_table_lock); +wait_on_page_locked (page); + goto out; + } + +/* Page is misplaced, serialise migrations and parallel THP splits */ +get_page (page);spin_unlock (&mm->page_table_lock); -lock_page (page); +if (!page_locked) { +lock_page (page); + page_locked = true; + } + anon_vma =page_lock_anon_vma_read (page);/* Confirm the PTE did not while locked */ spin_lock (&mm->page_table_lock);
@@ -1314,14 +1331,6 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, goto out_unlock; } -got_lock: - target_nid =mpol_misplaced (page, vma, haddr); -if (target_nid ==-1 ) { -unlock_page (page); -put_page (page); - goto clear_pmdnuma; - } -/* Migrate the THP to the requested node */ spin_unlock (&mm->page_table_lock); migrated = migrate_misplaced_transhuge_page(mm, vma,
@@ -1330,6 +1339,8 @@ got_lock: goto check_same;task_numa_fault (target_nid,HPAGE_PMD_NR , true); +if (anon_vma) +page_unlock_anon_vma_read (anon_vma);return 0 ; check_same:
@@ -1346,6 +1357,11 @@ clear_pmdnuma:update_mmu_cache_pmd (vma, addr, pmdp); out_unlock:spin_unlock (&mm->page_table_lock); + +out: +if (anon_vma) +page_unlock_anon_vma_read (anon_vma); +if (current_nid !=-1 )task_numa_fault (current_nid,HPAGE_PMD_NR , false);return 0 ;
Mel Gorman
reviewerPATCH 7/63 mm: numa: Sanitize task_numa_fault() callsites
In reply to Mel GormanThere are three callers of task_numa_fault():
- do_huge_pmd_numa_page():
Accounts against the current node, not the node where the page resides, unless we migrated, in which case it accounts against the node we migrated to.
- do_numa_page():
Accounts against the current node, not the node where the page resides, unless we migrated, in which case it accounts against the node we migrated to.
- do_pmd_numa_page():
Accounts not at all when the page isn't migrated, otherwise accounts against the node we migrated towards.
This seems wrong to me; all three sites should have the same sementaics, furthermore we should accounts against where the page really is, we already know where the task is.
So modify all three sites to always account; we did after all receive the fault; and always account to where the page is after migration, regardless of success.
They all still differ on when they clear the PTE/PMD; ideally that would get sorted too.
--- mm/huge_memory.c | 25 +++++++++++++------------ mm/memory.c | 53 +++++++++++++++++++++-------------------------------- 2 files changed, 34 insertions(+), 44 deletions(-)
mm/huge_memory.c +13 -12
@@ -1281,18 +1281,19 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct anon_vma *anon_vma =NULL ;struct page *page;unsigned long haddr = addr &HPAGE_PMD_MASK ; +int page_nid =-1 , this_nid =numa_node_id ();int target_nid; -int current_nid =-1 ; -bool migrated, page_locked; +bool page_locked; +bool migrated = false;spin_lock (&mm->page_table_lock);if (unlikely (!pmd_same (pmd, *pmdp))) goto out_unlock; page =pmd_page (pmd); - current_nid =page_to_nid (page); + page_nid =page_to_nid (page);count_vm_numa_event (NUMA_HINT_FAULTS ); -if (current_nid ==numa_node_id ()) +if (page_nid == this_nid)count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL );/*
@@ -1335,19 +1336,18 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,spin_unlock (&mm->page_table_lock); migrated =migrate_misplaced_transhuge_page (mm, vma, pmdp, pmd, addr, page, target_nid); -if (!migrated) +if (migrated) + page_nid = target_nid; +else goto check_same; -task_numa_fault (target_nid,HPAGE_PMD_NR , true); -if (anon_vma) -page_unlock_anon_vma_read (anon_vma); -return 0 ; + goto out; check_same:spin_lock (&mm->page_table_lock);if (unlikely (!pmd_same (pmd, *pmdp))) {/* Someone else took our fault */ - current_nid =-1 ; + page_nid =-1 ; goto out_unlock; } clear_pmdnuma:
@@ -1362,8 +1362,9 @@ out:if (anon_vma)page_unlock_anon_vma_read (anon_vma); -if (current_nid !=-1 ) -task_numa_fault (current_nid,HPAGE_PMD_NR , false); +if (page_nid !=-1 ) +task_numa_fault (page_nid,HPAGE_PMD_NR , migrated); +return 0 ; }
mm/memory.c +21 -32
107 lines not on this page · raw message
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 07/63] mm: numa: Sanitize task_numa_fault() callsites
Quoted 24 lines · Mel Gorman
There are three callers of
task_numa_fault():-
do_huge_pmd_numa_page():
Accounts against the current node, not the node where the page resides, unless we migrated, in which case it accounts against the node we migrated to.-
do_numa_page():
Accounts against the current node, not the node where the page resides, unless we migrated, in which case it accounts against the node we migrated to.-
do_pmd_numa_page():
Accounts not at all when the page isn't migrated, otherwise accounts against the node we migrated towards.This seems wrong to me; all three sites should have the same sementaics, furthermore we should accounts against where the page really is, we already know where the task is.
So modify all three sites to always account; we did after all receive the fault; and always account to where the page is after migration, regardless of success.
Quoted 6 lines
They all still differ on when they clear the PTE/PMD; ideally that would get sorted too.
Cc stable <[email protected]>Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] mm: numa: Sanitize task_numa_fault() callsites
In reply to Mel Gormanmm: numa: Sanitize task_numa_fault() callsites
Commit-ID: 8191acbd30c73e45c24ad16c372e0b42cc7ac8f8 Gitweb: http://git.kernel.org/tip/8191acbd30c73e45c24ad16c372e0b42cc7ac8f8 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:45 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:39:44 +0200
mm: numa: Sanitize task_numa_fault() callsites
There are three callers of task_numa_fault():
- do_huge_pmd_numa_page():
Accounts against the current node, not the node where the page resides, unless we migrated, in which case it accounts against the node we migrated to.
- do_numa_page():
Accounts against the current node, not the node where the page resides, unless we migrated, in which case it accounts against the node we migrated to.
- do_pmd_numa_page():
Accounts not at all when the page isn't migrated, otherwise accounts against the node we migrated towards.
This seems wrong to me; all three sites should have the same sementaics, furthermore we should accounts against where the page really is, we already know where the task is.
So modify all three sites to always account; we did after all receive the fault; and always account to where the page is after migration, regardless of success.
They all still differ on when they clear the PTE/PMD; ideally that would get sorted too.
--- mm/huge_memory.c | 25 +++++++++++++------------ mm/memory.c | 53 +++++++++++++++++++++-------------------------------- 2 files changed, 34 insertions(+), 44 deletions(-)
mm/huge_memory.c +13 -12
@@ -1281,18 +1281,19 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct anon_vma *anon_vma =NULL ;struct page *page;unsigned long haddr = addr &HPAGE_PMD_MASK ; +int page_nid =-1 , this_nid =numa_node_id ();int target_nid; -int current_nid =-1 ; -bool migrated, page_locked; +bool page_locked; +bool migrated = false;spin_lock (&mm->page_table_lock);if (unlikely (!pmd_same (pmd, *pmdp))) goto out_unlock; page =pmd_page (pmd); - current_nid =page_to_nid (page); + page_nid =page_to_nid (page);count_vm_numa_event (NUMA_HINT_FAULTS ); -if (current_nid ==numa_node_id ()) +if (page_nid == this_nid)count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL );/*
@@ -1335,19 +1336,18 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,spin_unlock (&mm->page_table_lock); migrated =migrate_misplaced_transhuge_page (mm, vma, pmdp, pmd, addr, page, target_nid); -if (!migrated) +if (migrated) + page_nid = target_nid; +else goto check_same; -task_numa_fault (target_nid,HPAGE_PMD_NR , true); -if (anon_vma) -page_unlock_anon_vma_read (anon_vma); -return 0 ; + goto out; check_same:spin_lock (&mm->page_table_lock);if (unlikely (!pmd_same (pmd, *pmdp))) {/* Someone else took our fault */ - current_nid =-1 ; + page_nid =-1 ; goto out_unlock; } clear_pmdnuma:
@@ -1362,8 +1362,9 @@ out:if (anon_vma)page_unlock_anon_vma_read (anon_vma); -if (current_nid !=-1 ) -task_numa_fault (current_nid,HPAGE_PMD_NR , false); +if (page_nid !=-1 ) +task_numa_fault (page_nid,HPAGE_PMD_NR , migrated); +return 0 ; }
mm/memory.c +21 -32
107 lines not on this page · raw message
tip-bot for Mel Gorman
merge bot[tip:core/urgent] mm: numa: Sanitize task_numa_fault() callsites
In reply to Mel Gormanmm: numa: Sanitize task_numa_fault() callsites
Commit-ID: c61109e34f60f6e85bb43c5a1cd51c0e3db40847 Gitweb: http://git.kernel.org/tip/c61109e34f60f6e85bb43c5a1cd51c0e3db40847 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:45 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 29 Oct 2013 11:37:52 +0100
mm: numa: Sanitize task_numa_fault() callsites
There are three callers of task_numa_fault():
- do_huge_pmd_numa_page():
Accounts against the current node, not the node where the page resides, unless we migrated, in which case it accounts against the node we migrated to.
- do_numa_page():
Accounts against the current node, not the node where the page resides, unless we migrated, in which case it accounts against the node we migrated to.
- do_pmd_numa_page():
Accounts not at all when the page isn't migrated, otherwise accounts against the node we migrated towards.
This seems wrong to me; all three sites should have the same sementaics, furthermore we should accounts against where the page really is, we already know where the task is.
So modify all three sites to always account; we did after all receive the fault; and always account to where the page is after migration, regardless of success.
They all still differ on when they clear the PTE/PMD; ideally that would get sorted too.
--- mm/huge_memory.c | 25 +++++++++++++------------ mm/memory.c | 53 +++++++++++++++++++++-------------------------------- 2 files changed, 34 insertions(+), 44 deletions(-)
mm/huge_memory.c +13 -12
@@ -1281,18 +1281,19 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct anon_vma *anon_vma =NULL ;struct page *page;unsigned long haddr = addr &HPAGE_PMD_MASK ; +int page_nid =-1 , this_nid =numa_node_id ();int target_nid; -int current_nid =-1 ; -bool migrated, page_locked; +bool page_locked; +bool migrated = false;spin_lock (&mm->page_table_lock);if (unlikely (!pmd_same (pmd, *pmdp))) goto out_unlock; page =pmd_page (pmd); - current_nid =page_to_nid (page); + page_nid =page_to_nid (page);count_vm_numa_event (NUMA_HINT_FAULTS ); -if (current_nid ==numa_node_id ()) +if (page_nid == this_nid)count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL );/*
@@ -1335,19 +1336,18 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,spin_unlock (&mm->page_table_lock); migrated =migrate_misplaced_transhuge_page (mm, vma, pmdp, pmd, addr, page, target_nid); -if (!migrated) +if (migrated) + page_nid = target_nid; +else goto check_same; -task_numa_fault (target_nid,HPAGE_PMD_NR , true); -if (anon_vma) -page_unlock_anon_vma_read (anon_vma); -return 0 ; + goto out; check_same:spin_lock (&mm->page_table_lock);if (unlikely (!pmd_same (pmd, *pmdp))) {/* Someone else took our fault */ - current_nid =-1 ; + page_nid =-1 ; goto out_unlock; } clear_pmdnuma:
@@ -1362,8 +1362,9 @@ out:if (anon_vma)page_unlock_anon_vma_read (anon_vma); -if (current_nid !=-1 ) -task_numa_fault (current_nid,HPAGE_PMD_NR , false); +if (page_nid !=-1 ) +task_numa_fault (page_nid,HPAGE_PMD_NR , migrated); +return 0 ; }
mm/memory.c +21 -32
107 lines not on this page · raw message
Mel Gorman
reviewerPATCH 8/63 mm: Close races between THP migration and PMD numa clearing
In reply to Mel GormanTHP migration uses the page lock to guard against parallel allocations but there are cases like this still open
Task A Task Bdo_huge_pmd_numa_page do_huge_pmd_numa_pagelock_pagempol_misplaced == -1unlock_page
goto clear_pmdnuma
lock_page mpol_misplaced == 2 migrate_misplaced_transhuge
pmd = pmd_mknonnumaset_pmd_at
During hours of testing, one crashed with weird errors and while I have no direct evidence, I suspect something like the race above happened. This patch extends the page lock to being held until the pmd_numa is cleared to prevent migration starting in parallel while the pmd_numa is being cleared. It also flushes the old pmd entry and orders pagetable insertion before rmap insertion.
--- mm/huge_memory.c | 33 +++++++++++++++------------------ mm/migrate.c | 19 +++++++++++-------- 2 files changed, 26 insertions(+), 26 deletions(-)
mm/huge_memory.c +15 -18
@@ -1304,24 +1304,25 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, target_nid =mpol_misplaced (page, vma, haddr);if (target_nid ==-1 ) {/* If the page was locked, there are no parallel migrations */ -if (page_locked) { -unlock_page (page); +if (page_locked) goto clear_pmdnuma; - } -/* Otherwise wait for potential migrations and retry fault */ +/* +* Otherwise wait for potential migrations and retry. We do +* relock and check_same as the page may no longer be mapped. +* As the fault is being retried, do not account for it. +*/ spin_unlock (&mm->page_table_lock);wait_on_page_locked (page); + page_nid =-1 ; goto out; }/* Page is misplaced, serialise migrations and parallel THP splits */ get_page (page);spin_unlock (&mm->page_table_lock); -if (!page_locked) { +if (!page_locked)lock_page (page); - page_locked = true; - } anon_vma =page_lock_anon_vma_read (page);/* Confirm the PMD did not change while page_table_lock was released */
@@ -1329,32 +1330,28 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,if (unlikely (!pmd_same (pmd, *pmdp))) {unlock_page (page);put_page (page); + page_nid =-1 ; goto out_unlock; } -/* Migrate the THP to the requested node */ +/* +* Migrate the THP to the requested node, returns with page unlocked +* and pmd_numa cleared. +*/ spin_unlock (&mm->page_table_lock); migrated =migrate_misplaced_transhuge_page (mm, vma, pmdp, pmd, addr, page, target_nid);if (migrated) page_nid = target_nid; -else - goto check_same; goto out; - -check_same: -spin_lock (&mm->page_table_lock); -if (unlikely (!pmd_same (pmd, *pmdp))) { -/* Someone else took our fault */ - page_nid =-1 ; - goto out_unlock; - } clear_pmdnuma: +BUG_ON (!PageLocked (page)); pmd =pmd_mknonnuma (pmd);set_pmd_at (mm, haddr, pmdp, pmd);VM_BUG_ON (pmd_numa (*pmdp));update_mmu_cache_pmd (vma, addr, pmdp); +unlock_page (page); out_unlock:spin_unlock (&mm->page_table_lock);
mm/migrate.c +11 -8
@@ -1713,12 +1713,12 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,unlock_page (new_page);put_page (new_page);/* Free it */ -unlock_page (page); +/* Retake the callers reference and putback on LRU */ +get_page (page);putback_lru_page (page); - -count_vm_events (PGMIGRATE_FAIL ,HPAGE_PMD_NR ); - isolated =0 ; - goto out; +mod_zone_page_state (page_zone (page), +NR_ISOLATED_ANON + page_lru, -HPAGE_PMD_NR ); + goto out_fail; }/*
@@ -1735,9 +1735,9 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); entry =pmd_mkhuge (entry); -page_add_new_anon_rmap (new_page, vma, haddr); - +pmdp_clear_flush (vma, haddr, pmd);set_pmd_at (mm, haddr, pmd, entry); +page_add_new_anon_rmap (new_page, vma, haddr);update_mmu_cache_pmd (vma, address, &entry);page_remove_rmap (page);/*
@@ -1756,7 +1756,6 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,count_vm_events (PGMIGRATE_SUCCESS ,HPAGE_PMD_NR );count_vm_numa_events (NUMA_PAGE_MIGRATE ,HPAGE_PMD_NR ); -out:mod_zone_page_state (page_zone (page),NR_ISOLATED_ANON + page_lru, -HPAGE_PMD_NR );
@@ -1765,6 +1764,10 @@ out: out_fail:count_vm_events (PGMIGRATE_FAIL ,HPAGE_PMD_NR ); out_dropref: + entry =pmd_mknonnuma (entry); +set_pmd_at (mm, haddr, pmd, entry); +update_mmu_cache_pmd (vma, address, &entry); +unlock_page (page);put_page (page);return 0 ;
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 08/63] mm: Close races between THP migration and PMD numa clearing
Quoted 22 lines · Mel Gorman
THP migration uses the page lock to guard against parallel allocations but there are cases like this still open
Task A Task B
do_huge_pmd_numa_pagedo_huge_pmd_numa_pagelock_pagempol_misplaced== -1unlock_page
gotoclear_pmdnumalock_page mpol_misplaced == 2 migrate_misplaced_transhugepmd =
pmd_mknonnumaset_pmd_atDuring hours of testing, one crashed with weird errors and while I have no direct evidence, I suspect something like the race above happened. This patch extends the page lock to being held until the
pmd_numais cleared to prevent migration starting in parallel while thepmd_numais being cleared. It also flushes the old pmd entry and orders pagetable insertion before rmap insertion.
Quoted 2 lines
Cc stable <[email protected]>Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] mm: Close races between THP migration and PMD numa clearing
In reply to Mel Gormanmm: Close races between THP migration and PMD numa clearing
Commit-ID: a54a407fbf7735fd8f7841375574f5d9b0375f93 Gitweb: http://git.kernel.org/tip/a54a407fbf7735fd8f7841375574f5d9b0375f93 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:46 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:39:45 +0200
mm: Close races between THP migration and PMD numa clearing
THP migration uses the page lock to guard against parallel allocations but there are cases like this still open
Task A Task B
--------------------- ---------------------
do_huge_pmd_numa_page do_huge_pmd_numa_page
lock_page
mpol_misplaced == -1
unlock_page
goto clear_pmdnuma
lock_page mpol_misplaced == 2 migrate_misplaced_transhuge
pmd = pmd_mknonnuma
set_pmd_at
During hours of testing, one crashed with weird errors and while I have no direct evidence, I suspect something like the race above happened. This patch extends the page lock to being held until the pmd_numa is cleared to prevent migration starting in parallel while the pmd_numa is being cleared. It also flushes the old pmd entry and orders pagetable insertion before rmap insertion.
--- mm/huge_memory.c | 33 +++++++++++++++------------------ mm/migrate.c | 19 +++++++++++-------- 2 files changed, 26 insertions(+), 26 deletions(-)
mm/huge_memory.c +15 -18
@@ -1304,24 +1304,25 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, target_nid =mpol_misplaced (page, vma, haddr);if (target_nid ==-1 ) {/* If the page was locked, there are no parallel migrations */ -if (page_locked) { -unlock_page (page); +if (page_locked) goto clear_pmdnuma; - } -/* Otherwise wait for potential migrations and retry fault */ +/* +* Otherwise wait for potential migrations and retry. We do +* relock and check_same as the page may no longer be mapped. +* As the fault is being retried, do not account for it. +*/ spin_unlock (&mm->page_table_lock);wait_on_page_locked (page); + page_nid =-1 ; goto out; }/* Page is misplaced, serialise migrations and parallel THP splits */ get_page (page);spin_unlock (&mm->page_table_lock); -if (!page_locked) { +if (!page_locked)lock_page (page); - page_locked = true; - } anon_vma =page_lock_anon_vma_read (page);/* Confirm the PMD did not change while page_table_lock was released */
@@ -1329,32 +1330,28 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,if (unlikely (!pmd_same (pmd, *pmdp))) {unlock_page (page);put_page (page); + page_nid =-1 ; goto out_unlock; } -/* Migrate the THP to the requested node */ +/* +* Migrate the THP to the requested node, returns with page unlocked +* and pmd_numa cleared. +*/ spin_unlock (&mm->page_table_lock); migrated =migrate_misplaced_transhuge_page (mm, vma, pmdp, pmd, addr, page, target_nid);if (migrated) page_nid = target_nid; -else - goto check_same; goto out; - -check_same: -spin_lock (&mm->page_table_lock); -if (unlikely (!pmd_same (pmd, *pmdp))) { -/* Someone else took our fault */ - page_nid =-1 ; - goto out_unlock; - } clear_pmdnuma: +BUG_ON (!PageLocked (page)); pmd =pmd_mknonnuma (pmd);set_pmd_at (mm, haddr, pmdp, pmd);VM_BUG_ON (pmd_numa (*pmdp));update_mmu_cache_pmd (vma, addr, pmdp); +unlock_page (page); out_unlock:spin_unlock (&mm->page_table_lock);
mm/migrate.c +11 -8
@@ -1713,12 +1713,12 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,unlock_page (new_page);put_page (new_page);/* Free it */ -unlock_page (page); +/* Retake the callers reference and putback on LRU */ +get_page (page);putback_lru_page (page); - -count_vm_events (PGMIGRATE_FAIL ,HPAGE_PMD_NR ); - isolated =0 ; - goto out; +mod_zone_page_state (page_zone (page), +NR_ISOLATED_ANON + page_lru, -HPAGE_PMD_NR ); + goto out_fail; }/*
@@ -1735,9 +1735,9 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); entry =pmd_mkhuge (entry); -page_add_new_anon_rmap (new_page, vma, haddr); - +pmdp_clear_flush (vma, haddr, pmd);set_pmd_at (mm, haddr, pmd, entry); +page_add_new_anon_rmap (new_page, vma, haddr);update_mmu_cache_pmd (vma, address, &entry);page_remove_rmap (page);/*
@@ -1756,7 +1756,6 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,count_vm_events (PGMIGRATE_SUCCESS ,HPAGE_PMD_NR );count_vm_numa_events (NUMA_PAGE_MIGRATE ,HPAGE_PMD_NR ); -out:mod_zone_page_state (page_zone (page),NR_ISOLATED_ANON + page_lru, -HPAGE_PMD_NR );
@@ -1765,6 +1764,10 @@ out: out_fail:count_vm_events (PGMIGRATE_FAIL ,HPAGE_PMD_NR ); out_dropref: + entry =pmd_mknonnuma (entry); +set_pmd_at (mm, haddr, pmd, entry); +update_mmu_cache_pmd (vma, address, &entry); +unlock_page (page);put_page (page);return 0 ;
tip-bot for Mel Gorman
merge bot[tip:core/urgent] mm: Close races between THP migration and PMD numa clearing
In reply to Mel Gormanmm: Close races between THP migration and PMD numa clearing
Commit-ID: 3f926ab945b60a5824369d21add7710622a2eac0 Gitweb: http://git.kernel.org/tip/3f926ab945b60a5824369d21add7710622a2eac0 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:46 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 29 Oct 2013 11:38:05 +0100
mm: Close races between THP migration and PMD numa clearing
THP migration uses the page lock to guard against parallel allocations but there are cases like this still open
Task A Task B
--------------------- ---------------------
do_huge_pmd_numa_page do_huge_pmd_numa_page
lock_page
mpol_misplaced == -1
unlock_page
goto clear_pmdnuma
lock_page mpol_misplaced == 2 migrate_misplaced_transhuge
pmd = pmd_mknonnuma
set_pmd_at
During hours of testing, one crashed with weird errors and while I have no direct evidence, I suspect something like the race above happened. This patch extends the page lock to being held until the pmd_numa is cleared to prevent migration starting in parallel while the pmd_numa is being cleared. It also flushes the old pmd entry and orders pagetable insertion before rmap insertion.
--- mm/huge_memory.c | 33 +++++++++++++++------------------ mm/migrate.c | 19 +++++++++++-------- 2 files changed, 26 insertions(+), 26 deletions(-)
mm/huge_memory.c +15 -18
@@ -1304,24 +1304,25 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, target_nid =mpol_misplaced (page, vma, haddr);if (target_nid ==-1 ) {/* If the page was locked, there are no parallel migrations */ -if (page_locked) { -unlock_page (page); +if (page_locked) goto clear_pmdnuma; - } -/* Otherwise wait for potential migrations and retry fault */ +/* +* Otherwise wait for potential migrations and retry. We do +* relock and check_same as the page may no longer be mapped. +* As the fault is being retried, do not account for it. +*/ spin_unlock (&mm->page_table_lock);wait_on_page_locked (page); + page_nid =-1 ; goto out; }/* Page is misplaced, serialise migrations and parallel THP splits */ get_page (page);spin_unlock (&mm->page_table_lock); -if (!page_locked) { +if (!page_locked)lock_page (page); - page_locked = true; - } anon_vma =page_lock_anon_vma_read (page);/* Confirm the PTE did not while locked */
@@ -1329,32 +1330,28 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,if (unlikely (!pmd_same (pmd, *pmdp))) {unlock_page (page);put_page (page); + page_nid =-1 ; goto out_unlock; } -/* Migrate the THP to the requested node */ +/* +* Migrate the THP to the requested node, returns with page unlocked +* and pmd_numa cleared. +*/ spin_unlock (&mm->page_table_lock); migrated =migrate_misplaced_transhuge_page (mm, vma, pmdp, pmd, addr, page, target_nid);if (migrated) page_nid = target_nid; -else - goto check_same; goto out; - -check_same: -spin_lock (&mm->page_table_lock); -if (unlikely (!pmd_same (pmd, *pmdp))) { -/* Someone else took our fault */ - page_nid =-1 ; - goto out_unlock; - } clear_pmdnuma: +BUG_ON (!PageLocked (page)); pmd =pmd_mknonnuma (pmd);set_pmd_at (mm, haddr, pmdp, pmd);VM_BUG_ON (pmd_numa (*pmdp));update_mmu_cache_pmd (vma, addr, pmdp); +unlock_page (page); out_unlock:spin_unlock (&mm->page_table_lock);
mm/migrate.c +11 -8
@@ -1715,12 +1715,12 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,unlock_page (new_page);put_page (new_page);/* Free it */ -unlock_page (page); +/* Retake the callers reference and putback on LRU */ +get_page (page);putback_lru_page (page); - -count_vm_events (PGMIGRATE_FAIL ,HPAGE_PMD_NR ); - isolated =0 ; - goto out; +mod_zone_page_state (page_zone (page), +NR_ISOLATED_ANON + page_lru, -HPAGE_PMD_NR ); + goto out_fail; }/*
@@ -1737,9 +1737,9 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); entry =pmd_mkhuge (entry); -page_add_new_anon_rmap (new_page, vma, haddr); - +pmdp_clear_flush (vma, haddr, pmd);set_pmd_at (mm, haddr, pmd, entry); +page_add_new_anon_rmap (new_page, vma, haddr);update_mmu_cache_pmd (vma, address, &entry);page_remove_rmap (page);/*
@@ -1758,7 +1758,6 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,count_vm_events (PGMIGRATE_SUCCESS ,HPAGE_PMD_NR );count_vm_numa_events (NUMA_PAGE_MIGRATE ,HPAGE_PMD_NR ); -out:mod_zone_page_state (page_zone (page),NR_ISOLATED_ANON + page_lru, -HPAGE_PMD_NR );
@@ -1767,6 +1766,10 @@ out: out_fail:count_vm_events (PGMIGRATE_FAIL ,HPAGE_PMD_NR ); out_dropref: + entry =pmd_mknonnuma (entry); +set_pmd_at (mm, haddr, pmd, entry); +update_mmu_cache_pmd (vma, address, &entry); +unlock_page (page);put_page (page);return 0 ;
Mel Gorman
reviewerPATCH 9/63 mm: Account for a THP NUMA hinting update as one PTE update
In reply to Mel GormanA THP PMD update is accounted for as 512 pages updated in vmstat. This is large difference when estimating the cost of automatic NUMA balancing and can be misleading when comparing results that had collapsed versus split THP. This patch addresses the accounting issue.
--- mm/mprotect.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/mprotect.c +1 -1
@@ -145,7 +145,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,split_huge_page_pmd (vma, addr, pmd); else if (change_huge_pmd(vma, pmd, addr, newprot,prot_numa )) { - pages +=HPAGE_PMD_NR ; + pages++;continue ; }/* fall through */
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 09/63] mm: Account for a THP NUMA hinting update as one PTE update
Quoted 7 lines · Mel Gorman
A THP PMD update is accounted for as 512 pages updated in vmstat. This is large difference when estimating the cost of automatic NUMA balancing and can be misleading when comparing results that had collapsed versus split THP. This patch addresses the accounting issue.
Cc stable <[email protected]>Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] mm: Account for a THP NUMA hinting update as one PTE update
In reply to Mel Gormanmm: Account for a THP NUMA hinting update as one PTE update
Commit-ID: afcae2655b0ab67e65f161b1bb214efcfa1db415 Gitweb: http://git.kernel.org/tip/afcae2655b0ab67e65f161b1bb214efcfa1db415 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:47 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:39:46 +0200
mm: Account for a THP NUMA hinting update as one PTE update
A THP PMD update is accounted for as 512 pages updated in vmstat. This is large difference when estimating the cost of automatic NUMA balancing and can be misleading when comparing results that had collapsed versus split THP. This patch addresses the accounting issue.
--- mm/mprotect.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/mprotect.c +1 -1
@@ -145,7 +145,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,split_huge_page_pmd (vma, addr, pmd); else if (change_huge_pmd(vma, pmd, addr, newprot,prot_numa )) { - pages +=HPAGE_PMD_NR ; + pages++;continue ; }/* fall through */
tip-bot for Mel Gorman
merge bot[tip:core/urgent] mm: Account for a THP NUMA hinting update as one PTE update
In reply to Mel Gormanmm: Account for a THP NUMA hinting update as one PTE update
Commit-ID: 0255d491848032f6c601b6410c3b8ebded3a37b1 Gitweb: http://git.kernel.org/tip/0255d491848032f6c601b6410c3b8ebded3a37b1 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:47 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 29 Oct 2013 11:38:17 +0100
mm: Account for a THP NUMA hinting update as one PTE update
A THP PMD update is accounted for as 512 pages updated in vmstat. This is large difference when estimating the cost of automatic NUMA balancing and can be misleading when comparing results that had collapsed versus split THP. This patch addresses the accounting issue.
--- mm/mprotect.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/mprotect.c +1 -1
@@ -148,7 +148,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,split_huge_page_pmd (vma, addr, pmd); else if (change_huge_pmd(vma, pmd, addr, newprot,prot_numa )) { - pages +=HPAGE_PMD_NR ; + pages++;continue ; }/* fall through */
Mel Gorman
reviewerPATCH 10/63 mm: Do not flush TLB during protection change if !pte_present && !migration_entry
In reply to Mel GormanNUMA PTE scanning is expensive both in terms of the scanning itself and the TLB flush if there are any updates. Currently non-present PTEs are accounted for as an update and incurring a TLB flush where it is only necessary for anonymous migration entries. This patch addresses the problem and should reduce TLB flushes.
--- mm/mprotect.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/mprotect.c +2 -1
@@ -101,8 +101,9 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,make_migration_entry_read (&entry);set_pte_at (mm, addr, pte,swp_entry_to_pte (entry)); + + pages++; } - pages++; } }while (pte++, addr +=PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode ();
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 10/63] mm: Do not flush TLB during protection change if !pte_present && !migration_entry
Quoted 7 lines · Mel Gorman
NUMA PTE scanning is expensive both in terms of the scanning itself and the TLB flush if there are any updates. Currently non-present PTEs are accounted for as an update and incurring a TLB flush where it is only necessary for anonymous migration entries. This patch addresses the problem and should reduce TLB flushes.
Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] mm: Do not flush TLB during protection change if !pte_present && !migration_entry
In reply to Mel Gormanmm: Do not flush TLB during protection change if !pte_present && !migration_entry
Commit-ID: e920e14ca29b0b2a981cfc90e4e20edd6f078d19 Gitweb: http://git.kernel.org/tip/e920e14ca29b0b2a981cfc90e4e20edd6f078d19 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:48 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:39:48 +0200
mm: Do not flush TLB during protection change if !pte_present && !migration_entry
NUMA PTE scanning is expensive both in terms of the scanning itself and the TLB flush if there are any updates. Currently non-present PTEs are accounted for as an update and incurring a TLB flush where it is only necessary for anonymous migration entries. This patch addresses the problem and should reduce TLB flushes.
--- mm/mprotect.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/mprotect.c +2 -1
@@ -101,8 +101,9 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,make_migration_entry_read (&entry);set_pte_at (mm, addr, pte,swp_entry_to_pte (entry)); + + pages++; } - pages++; } }while (pte++, addr +=PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode ();
Mel Gorman
reviewerPATCH 11/63 mm: Only flush TLBs if a transhuge PMD is modified for NUMA pte scanning
In reply to Mel GormanNUMA PTE scanning is expensive both in terms of the scanning itself and the TLB flush if there are any updates. The TLB flush is avoided if no PTEs are updated but there is a bug where transhuge PMDs are considered to be updated even if they were already pmd_numa. This patch addresses the problem and TLB flushes should be reduced.
--- mm/huge_memory.c | 19 ++++++++++++++++--- mm/mprotect.c | 14 ++++++++++---- 2 files changed, 26 insertions(+), 7 deletions(-)
mm/huge_memory.c +16 -3
@@ -1458,6 +1458,12 @@ out:return ret; } +/* +* Returns +* - 0 if PMD could not be locked +* - 1 if PMD was locked but protections unchange and TLB flush unnecessary +* - HPAGE_PMD_NR is protections changed and TLB flush necessary +*/ int change_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,pgprot_t newprot,int prot_numa) {
@@ -1466,9 +1472,11 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,if (__pmd_trans_huge_lock (pmd, vma) ==1 ) {pmd_t entry; - entry =pmdp_get_and_clear (mm, addr, pmd); + ret =1 ;if (!prot_numa) { + entry =pmdp_get_and_clear (mm, addr, pmd); entry =pmd_modify (entry, newprot); + ret =HPAGE_PMD_NR ;BUG_ON (pmd_write (entry)); }else {struct page *page =pmd_page (*pmd);
@@ -1476,12 +1484,17 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,/* only check non-shared pages */ if (page_mapcount (page) ==1 && !pmd_numa (*pmd)) { + entry =pmdp_get_and_clear (mm, addr, pmd); entry =pmd_mknuma (entry); + ret =HPAGE_PMD_NR ; } } -set_pmd_at (mm, addr, pmd, entry); + +/* Set PMD if cleared earlier */ +if (ret ==HPAGE_PMD_NR ) +set_pmd_at (mm, addr, pmd, entry); +spin_unlock (&vma->vm_mm->page_table_lock); - ret =1 ; }return ret;
mm/mprotect.c +10 -4
@@ -144,10 +144,16 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,if (pmd_trans_huge (*pmd)) {if (next - addr !=HPAGE_PMD_SIZE )split_huge_page_pmd (vma, addr, pmd); -else if (change_huge_pmd (vma, pmd, addr, newprot, - prot_numa)) { - pages++; -continue ; +else { +int nr_ptes =change_huge_pmd (vma, pmd, addr, + newprot, prot_numa); + +if (nr_ptes) { +if (nr_ptes ==HPAGE_PMD_NR ) + pages++; + +continue ; + } }/* fall through */ }
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] mm: Only flush TLBs if a transhuge PMD is modified for NUMA pte scanning
mm: Only flush TLBs if a transhuge PMD is modified for NUMA pte scanning
Commit-ID: f123d74abf91574837d14e5ea58f6a779a387bf5 Gitweb: http://git.kernel.org/tip/f123d74abf91574837d14e5ea58f6a779a387bf5 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:49 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:39:49 +0200
mm: Only flush TLBs if a transhuge PMD is modified for NUMA pte scanning
NUMA PTE scanning is expensive both in terms of the scanning itself and the TLB flush if there are any updates. The TLB flush is avoided if no PTEs are updated but there is a bug where transhuge PMDs are considered to be updated even if they were already pmd_numa. This patch addresses the problem and TLB flushes should be reduced.
--- mm/huge_memory.c | 19 ++++++++++++++++--- mm/mprotect.c | 14 ++++++++++---- 2 files changed, 26 insertions(+), 7 deletions(-)
mm/huge_memory.c +16 -3
@@ -1458,6 +1458,12 @@ out:return ret; } +/* +* Returns +* - 0 if PMD could not be locked +* - 1 if PMD was locked but protections unchange and TLB flush unnecessary +* - HPAGE_PMD_NR is protections changed and TLB flush necessary +*/ int change_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,pgprot_t newprot,int prot_numa) {
@@ -1466,9 +1472,11 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,if (__pmd_trans_huge_lock (pmd, vma) ==1 ) {pmd_t entry; - entry =pmdp_get_and_clear (mm, addr, pmd); + ret =1 ;if (!prot_numa) { + entry =pmdp_get_and_clear (mm, addr, pmd); entry =pmd_modify (entry, newprot); + ret =HPAGE_PMD_NR ;BUG_ON (pmd_write (entry)); }else {struct page *page =pmd_page (*pmd);
@@ -1476,12 +1484,17 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,/* only check non-shared pages */ if (page_mapcount (page) ==1 && !pmd_numa (*pmd)) { + entry =pmdp_get_and_clear (mm, addr, pmd); entry =pmd_mknuma (entry); + ret =HPAGE_PMD_NR ; } } -set_pmd_at (mm, addr, pmd, entry); + +/* Set PMD if cleared earlier */ +if (ret ==HPAGE_PMD_NR ) +set_pmd_at (mm, addr, pmd, entry); +spin_unlock (&vma->vm_mm->page_table_lock); - ret =1 ; }return ret;
mm/mprotect.c +10 -4
@@ -144,10 +144,16 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,if (pmd_trans_huge (*pmd)) {if (next - addr !=HPAGE_PMD_SIZE )split_huge_page_pmd (vma, addr, pmd); -else if (change_huge_pmd (vma, pmd, addr, newprot, - prot_numa)) { - pages++; -continue ; +else { +int nr_ptes =change_huge_pmd (vma, pmd, addr, + newprot, prot_numa); + +if (nr_ptes) { +if (nr_ptes ==HPAGE_PMD_NR ) + pages++; + +continue ; + } }/* fall through */ }
Mel Gorman
reviewerPATCH 12/63 mm: numa: Do not migrate or account for hinting faults on the zero page
In reply to Mel GormanThe zero page is not replicated between nodes and is often shared between processes. The data is read-only and likely to be cached in local CPUs if heavily accessed meaning that the remote memory access cost is less of a concern. This patch prevents trapping faults on the zero pages. For tasks using the zero page this will reduce the number of PTE updates, TLB flushes and hinting faults.
--- mm/huge_memory.c | 10 +++++++++- mm/memory.c | 1 + 2 files changed, 10 insertions(+), 1 deletion(-)
mm/huge_memory.c +9 -1
@@ -1291,6 +1291,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, goto out_unlock; page =pmd_page (pmd); +BUG_ON (is_huge_zero_page (page)); page_nid =page_to_nid (page);count_vm_numa_event (NUMA_HINT_FAULTS );if (page_nid == this_nid)
@@ -1481,8 +1482,15 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, }else {struct page *page =pmd_page (*pmd); -/* only check non-shared pages */ +/* +* Only check non-shared pages. Do not trap faults +* against the zero page. The read-only data is likely +* to be read-cached on the local CPU cache and it is +* less useful to know about local vs remote hits on +* the zero page. +*/ if (page_mapcount (page) ==1 && + !is_huge_zero_page (page) && !pmd_numa (*pmd)) { entry =pmdp_get_and_clear (mm, addr, pmd); entry =pmd_mknuma (entry);
mm/memory.c +1 -0
@@ -3564,6 +3564,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,pte_unmap_unlock (ptep, ptl);return 0 ; } +BUG_ON (is_zero_pfn (page_to_pfn (page))); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 12/63] mm: numa: Do not migrate or account for hinting faults on the zero page
Quoted 9 lines · Mel Gorman
The zero page is not replicated between nodes and is often shared between processes. The data is read-only and likely to be cached in local CPUs if heavily accessed meaning that the remote memory access cost is less of a concern. This patch prevents trapping faults on the zero pages. For tasks using the zero page this will reduce the number of PTE updates, TLB flushes and hinting faults.
[[email protected]: Correct use of is_huge_zero_page]Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] mm: numa: Do not migrate or account for hinting faults on the zero page
In reply to Mel Gormanmm: numa: Do not migrate or account for hinting faults on the zero page
Commit-ID: a1a46184e34cfd0764f06a54870defa052b0a094 Gitweb: http://git.kernel.org/tip/a1a46184e34cfd0764f06a54870defa052b0a094 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:50 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:39:50 +0200
mm: numa: Do not migrate or account for hinting faults on the zero page
The zero page is not replicated between nodes and is often shared between processes. The data is read-only and likely to be cached in local CPUs if heavily accessed meaning that the remote memory access cost is less of a concern. This patch prevents trapping faults on the zero pages. For tasks using the zero page this will reduce the number of PTE updates, TLB flushes and hinting faults.
--- mm/huge_memory.c | 10 +++++++++- mm/memory.c | 1 + 2 files changed, 10 insertions(+), 1 deletion(-)
mm/huge_memory.c +9 -1
@@ -1291,6 +1291,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, goto out_unlock; page =pmd_page (pmd); +BUG_ON (is_huge_zero_page (page)); page_nid =page_to_nid (page);count_vm_numa_event (NUMA_HINT_FAULTS );if (page_nid == this_nid)
@@ -1481,8 +1482,15 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, }else {struct page *page =pmd_page (*pmd); -/* only check non-shared pages */ +/* +* Only check non-shared pages. Do not trap faults +* against the zero page. The read-only data is likely +* to be read-cached on the local CPU cache and it is +* less useful to know about local vs remote hits on +* the zero page. +*/ if (page_mapcount (page) ==1 && + !is_huge_zero_page (page) && !pmd_numa (*pmd)) { entry =pmdp_get_and_clear (mm, addr, pmd); entry =pmd_mknuma (entry);
mm/memory.c +1 -0
@@ -3564,6 +3564,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,pte_unmap_unlock (ptep, ptl);return 0 ; } +BUG_ON (is_zero_pfn (page_to_pfn (page))); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);
Mel Gorman
reviewerPATCH 13/63 sched: numa: Mitigate chance that same task always updates PTEs
In reply to Mel GormanFrom: Peter Zijlstra <[email protected]>
With a trace_printk("working\n"); right after the cmpxchg in task_numa_work() we can see that of a 4 thread process, its always the same task winning the race and doing the protection change.
This is a problem since the task doing the protection change has a penalty for taking faults -- it is busy when marking the PTEs. If its always the same task the ->numa_faults[] get severely skewed.
Avoid this by delaying the task doing the protection change such that it is unlikely to win the privilege again.
Before:
root@interlagos:~# grep "thread 0/.*working" /debug/tracing/trace | tail -15
thread 0/0-3232 [022] .... 212.787402: task_numa_work: working
thread 0/0-3232 [022] .... 212.888473: task_numa_work: working
thread 0/0-3232 [022] .... 212.989538: task_numa_work: working
thread 0/0-3232 [022] .... 213.090602: task_numa_work: working
thread 0/0-3232 [022] .... 213.191667: task_numa_work: working
thread 0/0-3232 [022] .... 213.292734: task_numa_work: working
thread 0/0-3232 [022] .... 213.393804: task_numa_work: working
thread 0/0-3232 [022] .... 213.494869: task_numa_work: working
thread 0/0-3232 [022] .... 213.596937: task_numa_work: working
thread 0/0-3232 [022] .... 213.699000: task_numa_work: working
thread 0/0-3232 [022] .... 213.801067: task_numa_work: working
thread 0/0-3232 [022] .... 213.903155: task_numa_work: working
thread 0/0-3232 [022] .... 214.005201: task_numa_work: working
thread 0/0-3232 [022] .... 214.107266: task_numa_work: working
thread 0/0-3232 [022] .... 214.209342: task_numa_work: workingAfter:
root@interlagos:~# grep "thread 0/.*working" /debug/tracing/trace | tail -15
thread 0/0-3253 [005] .... 136.865051: task_numa_work: working
thread 0/2-3255 [026] .... 136.965134: task_numa_work: working
thread 0/3-3256 [024] .... 137.065217: task_numa_work: working
thread 0/3-3256 [024] .... 137.165302: task_numa_work: working
thread 0/3-3256 [024] .... 137.265382: task_numa_work: working
thread 0/0-3253 [004] .... 137.366465: task_numa_work: working
thread 0/2-3255 [026] .... 137.466549: task_numa_work: working
thread 0/0-3253 [004] .... 137.566629: task_numa_work: working
thread 0/0-3253 [004] .... 137.666711: task_numa_work: working
thread 0/1-3254 [028] .... 137.766799: task_numa_work: working
thread 0/0-3253 [004] .... 137.866876: task_numa_work: working
thread 0/2-3255 [026] .... 137.966960: task_numa_work: working
thread 0/1-3254 [028] .... 138.067041: task_numa_work: working
thread 0/2-3255 [026] .... 138.167123: task_numa_work: working
thread 0/3-3256 [024] .... 138.267207: task_numa_work: working--- kernel/sched/fair.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/sched/fair.c +7 -1
@@ -946,6 +946,12 @@ void task_numa_work(struct callback_head *work)return ;/* +* Delay this task enough that another task of this mm will likely win +* the next time around. +*/ + p->node_stamp +=2 *TICK_NSEC ; + +/* * Do not set pte_numa if the current running node is rate-limited. * This loses statistics on the fault but if we are unwilling to * migrate to this node, it is less likely we can do useful work
@@ -1026,7 +1032,7 @@ void task_tick_numa(struct rq *rq, struct task_struct *curr)if (now - curr->node_stamp > period) {if (!curr->node_stamp) curr->numa_scan_period = sysctl_numa_balancing_scan_period_min; - curr->node_stamp = now; + curr->node_stamp += period;if (!time_before (jiffies, curr->mm->numa_next_scan)) {init_task_work (work, task_numa_work);/* TODO: move this into sched_fork() */
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 13/63] sched: numa: Mitigate chance that same task always updates PTEs
Quoted 10 lines · Mel Gorman
From: Peter Zijlstra <[email protected]>
With a
trace_printk("working\n"); right after the cmpxchg intask_numa_work()we can see that of a 4 thread process, its always the same task winning the race and doing the protection change.This is a problem since the task doing the protection change has a penalty for taking faults -- it is busy when marking the PTEs. If its always the same task the ->numa_faults[] get severely skewed.
Quoted 5 lines
Avoid this by delaying the task doing the protection change such that it is unlikely to win the privilege again.
Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Peter Zijlstra
merge bot[tip:sched/core] sched/numa: Mitigate chance that same task always updates PTEs
In reply to Mel Gormansched/numa: Mitigate chance that same task always updates PTEs
Commit-ID: 19a78d110d7a8045aeb90d38ee8fe9743ce88c2d Gitweb: http://git.kernel.org/tip/19a78d110d7a8045aeb90d38ee8fe9743ce88c2d Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:51 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:39:56 +0200
sched/numa: Mitigate chance that same task always updates PTEs
With a trace_printk("working\n"); right after the cmpxchg in task_numa_work() we can see that of a 4 thread process, its always the same task winning the race and doing the protection change.
This is a problem since the task doing the protection change has a penalty for taking faults -- it is busy when marking the PTEs. If its always the same task the ->numa_faults[] get severely skewed.
Avoid this by delaying the task doing the protection change such that it is unlikely to win the privilege again.
Before:
root@interlagos:~# grep "thread 0/.*working" /debug/tracing/trace | tail -15
thread 0/0-3232 [022] .... 212.787402: task_numa_work: working
thread 0/0-3232 [022] .... 212.888473: task_numa_work: working
thread 0/0-3232 [022] .... 212.989538: task_numa_work: working
thread 0/0-3232 [022] .... 213.090602: task_numa_work: working
thread 0/0-3232 [022] .... 213.191667: task_numa_work: working
thread 0/0-3232 [022] .... 213.292734: task_numa_work: working
thread 0/0-3232 [022] .... 213.393804: task_numa_work: working
thread 0/0-3232 [022] .... 213.494869: task_numa_work: working
thread 0/0-3232 [022] .... 213.596937: task_numa_work: working
thread 0/0-3232 [022] .... 213.699000: task_numa_work: working
thread 0/0-3232 [022] .... 213.801067: task_numa_work: working
thread 0/0-3232 [022] .... 213.903155: task_numa_work: working
thread 0/0-3232 [022] .... 214.005201: task_numa_work: working
thread 0/0-3232 [022] .... 214.107266: task_numa_work: working
thread 0/0-3232 [022] .... 214.209342: task_numa_work: workingAfter:
root@interlagos:~# grep "thread 0/.*working" /debug/tracing/trace | tail -15
thread 0/0-3253 [005] .... 136.865051: task_numa_work: working
thread 0/2-3255 [026] .... 136.965134: task_numa_work: working
thread 0/3-3256 [024] .... 137.065217: task_numa_work: working
thread 0/3-3256 [024] .... 137.165302: task_numa_work: working
thread 0/3-3256 [024] .... 137.265382: task_numa_work: working
thread 0/0-3253 [004] .... 137.366465: task_numa_work: working
thread 0/2-3255 [026] .... 137.466549: task_numa_work: working
thread 0/0-3253 [004] .... 137.566629: task_numa_work: working
thread 0/0-3253 [004] .... 137.666711: task_numa_work: working
thread 0/1-3254 [028] .... 137.766799: task_numa_work: working
thread 0/0-3253 [004] .... 137.866876: task_numa_work: working
thread 0/2-3255 [026] .... 137.966960: task_numa_work: working
thread 0/1-3254 [028] .... 138.067041: task_numa_work: working
thread 0/2-3255 [026] .... 138.167123: task_numa_work: working
thread 0/3-3256 [024] .... 138.267207: task_numa_work: working--- kernel/sched/fair.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/sched/fair.c +7 -1
@@ -946,6 +946,12 @@ void task_numa_work(struct callback_head *work)return ;/* +* Delay this task enough that another task of this mm will likely win +* the next time around. +*/ + p->node_stamp +=2 *TICK_NSEC ; + +/* * Do not set pte_numa if the current running node is rate-limited. * This loses statistics on the fault but if we are unwilling to * migrate to this node, it is less likely we can do useful work
@@ -1026,7 +1032,7 @@ void task_tick_numa(struct rq *rq, struct task_struct *curr)if (now - curr->node_stamp > period) {if (!curr->node_stamp) curr->numa_scan_period = sysctl_numa_balancing_scan_period_min; - curr->node_stamp = now; + curr->node_stamp += period;if (!time_before (jiffies, curr->mm->numa_next_scan)) {init_task_work (work, task_numa_work);/* TODO: move this into sched_fork() */
Mel Gorman
reviewerPATCH 14/63 sched: numa: Continue PTE scanning even if migrate rate limited
In reply to Mel GormanFrom: Peter Zijlstra <[email protected]>
Avoiding marking PTEs pte_numa because a particular NUMA node is migrate rate limited sees like a bad idea. Even if this node can't migrate anymore other nodes might and we want up-to-date information to do balance decisions. We already rate limit the actual migrations, this should leave enough bandwidth to allow the non-migrating scanning. I think its important we keep up-to-date information if we're going to do placement based on it.
--- kernel/sched/fair.c | 8 -------- 1 file changed, 8 deletions(-)
kernel/sched/fair.c +0 -8
@@ -951,14 +951,6 @@ void task_numa_work(struct callback_head *work)*/ p->node_stamp +=2 *TICK_NSEC ; -/* -* Do not set pte_numa if the current running node is rate-limited. -* This loses statistics on the fault but if we are unwilling to -* migrate to this node, it is less likely we can do useful work -*/ -if (migrate_ratelimited (numa_node_id ())) -return ; - start = mm->numa_scan_offset; pages = sysctl_numa_balancing_scan_size; pages <<=20 -PAGE_SHIFT ;/* MB in pages */
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 14/63] sched: numa: Continue PTE scanning even if migrate rate limited
Quoted 11 lines · Mel Gorman
From: Peter Zijlstra <[email protected]>
Avoiding marking PTEs
pte_numabecause a particular NUMA node is migrate rate limited sees like a bad idea. Even if this node can't migrate anymore other nodes might and we want up-to-date information to do balance decisions. We already rate limit the actual migrations, this should leave enough bandwidth to allow the non-migrating scanning. I think its important we keep up-to-date information if we're going to do placement based on it.Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Peter Zijlstra
merge bot[tip:sched/core] sched/numa: Continue PTE scanning even if migrate rate limited
In reply to Mel Gormansched/numa: Continue PTE scanning even if migrate rate limited
Commit-ID: 9e645ab6d089f5822479a833c6977c785bcfffe3 Gitweb: http://git.kernel.org/tip/9e645ab6d089f5822479a833c6977c785bcfffe3 Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:52 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:09 +0200
sched/numa: Continue PTE scanning even if migrate rate limited
Avoiding marking PTEs pte_numa because a particular NUMA node is migrate rate limited sees like a bad idea. Even if this node can't migrate anymore other nodes might and we want up-to-date information to do balance decisions. We already rate limit the actual migrations, this should leave enough bandwidth to allow the non-migrating scanning. I think its important we keep up-to-date information if we're going to do placement based on it.
--- kernel/sched/fair.c | 8 -------- 1 file changed, 8 deletions(-)
kernel/sched/fair.c +0 -8
@@ -951,14 +951,6 @@ void task_numa_work(struct callback_head *work)*/ p->node_stamp +=2 *TICK_NSEC ; -/* -* Do not set pte_numa if the current running node is rate-limited. -* This loses statistics on the fault but if we are unwilling to -* migrate to this node, it is less likely we can do useful work -*/ -if (migrate_ratelimited (numa_node_id ())) -return ; - start = mm->numa_scan_offset; pages = sysctl_numa_balancing_scan_size; pages <<=20 -PAGE_SHIFT ;/* MB in pages */
Mel Gorman
reviewerPATCH 15/63 Revert "mm: sched: numa: Delay PTE scanning until a task is scheduled on a new node"
In reply to Mel GormanPTE scanning and NUMA hinting fault handling is expensive so commit 5bca2303 ("mm: sched: numa: Delay PTE scanning until a task is scheduled on a new node") deferred the PTE scan until a task had been scheduled on another node. The problem is that in the purely shared memory case that this may never happen and no NUMA hinting fault information will be captured. We are not ruling out the possibility that something better can be done here but for now, this patch needs to be reverted and depend entirely on the scan_delay to avoid punishing short-lived processes.
--- include/linux/mm_types.h | 10 ---------- kernel/fork.c | 3 --- kernel/sched/fair.c | 18 ------------------ kernel/sched/features.h | 4 +--- 4 files changed, 1 insertion(+), 34 deletions(-)
include/linux/mm_types.h +0 -10
@@ -428,20 +428,10 @@ struct mm_struct {/* numa_scan_seq prevents two threads setting pte_numa */ int numa_scan_seq; - -/* -* The first node a task was scheduled on. If a task runs on -* a different node than Make PTE Scan Go Now. -*/ -int first_nid;#endif struct uprobes_state uprobes_state; }; -/* first nid will either be a valid NID or one of these values */ -#define NUMA_PTE_SCAN_INIT -1 -#define NUMA_PTE_SCAN_ACTIVE -2 -static inline void mm_init_cpumask (struct mm_struct *mm) {#ifdef CONFIG_CPUMASK_OFFSTACK
kernel/fork.c +0 -3
@@ -817,9 +817,6 @@ struct mm_struct *dup_mm(struct task_struct *tsk)#ifdef CONFIG_TRANSPARENT_HUGEPAGE mm->pmd_huge_pte =NULL ;#endif -#ifdef CONFIG_NUMA_BALANCING - mm->first_nid =NUMA_PTE_SCAN_INIT ; -#endif if (!mm_init (mm, tsk)) goto fail_nomem;
kernel/sched/fair.c +0 -18
@@ -901,24 +901,6 @@ void task_numa_work(struct callback_head *work)return ;/* -* We do not care about task placement until a task runs on a node -* other than the first one used by the address space. This is -* largely because migrations are driven by what CPU the task -* is running on. If it's never scheduled on another node, it'll -* not migrate so why bother trapping the fault. -*/ -if (mm->first_nid ==NUMA_PTE_SCAN_INIT ) - mm->first_nid =numa_node_id (); -if (mm->first_nid !=NUMA_PTE_SCAN_ACTIVE ) { -/* Are we running on a new node yet? */ -if (numa_node_id () == mm->first_nid && - !sched_feat_numa (NUMA_FORCE )) -return ; - - mm->first_nid =NUMA_PTE_SCAN_ACTIVE ; - } - -/* * Reset the scan period if enough time has gone by. Objective is that * scanning will be reduced if pages are properly placed. As tasks * can enter different phases this needs to be re-examined. Lacking
kernel/sched/features.h +1 -3
@@ -63,10 +63,8 @@ SCHED_FEAT(LB_MIN, false)/* * Apply the automatic NUMA scheduling policy. Enabled automatically * at runtime if running on a NUMA machine. Can be controlled via -* numa_balancing=. Allow PTE scanning to be forced on UMA machines -* for debugging the core machinery. +* numa_balancing= */ #ifdef CONFIG_NUMA_BALANCING SCHED_FEAT(NUMA, false) -SCHED_FEAT(NUMA_FORCE, false)#endif
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 15/63] Revert "mm: sched: numa: Delay PTE scanning until a task is scheduled on a new node"
Quoted 10 lines · Mel Gorman
PTE scanning and NUMA hinting fault handling is expensive so commit
5bca2303("mm: sched: numa: Delay PTE scanning until a task is scheduled on a new node") deferred the PTE scan until a task had been scheduled on another node. The problem is that in the purely shared memory case that this may never happen and no NUMA hinting fault information will be captured. We are not ruling out the possibility that something better can be done here but for now, this patch needs to be reverted and depend entirely on thescan_delayto avoid punishing short-lived processes.Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] Revert "mm: sched: numa: Delay PTE scanning until a task is scheduled on a new node"
In reply to Mel GormanRevert "mm: sched: numa: Delay PTE scanning until a task is scheduled on a new node"
Commit-ID: b726b7dfb400c937546fa91cf8523dcb1aa2fc6e Gitweb: http://git.kernel.org/tip/b726b7dfb400c937546fa91cf8523dcb1aa2fc6e Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:53 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:17 +0200
Revert "mm: sched: numa: Delay PTE scanning until a task is scheduled on a new node"
PTE scanning and NUMA hinting fault handling is expensive so commit 5bca2303 ("mm: sched: numa: Delay PTE scanning until a task is scheduled on a new node") deferred the PTE scan until a task had been scheduled on another node. The problem is that in the purely shared memory case that this may never happen and no NUMA hinting fault information will be captured. We are not ruling out the possibility that something better can be done here but for now, this patch needs to be reverted and depend entirely on the scan_delay to avoid punishing short-lived processes.
--- include/linux/mm_types.h | 10 ---------- kernel/fork.c | 3 --- kernel/sched/fair.c | 18 ------------------ kernel/sched/features.h | 4 +--- 4 files changed, 1 insertion(+), 34 deletions(-)
include/linux/mm_types.h +0 -10
@@ -428,20 +428,10 @@ struct mm_struct {/* numa_scan_seq prevents two threads setting pte_numa */ int numa_scan_seq; - -/* -* The first node a task was scheduled on. If a task runs on -* a different node than Make PTE Scan Go Now. -*/ -int first_nid;#endif struct uprobes_state uprobes_state; }; -/* first nid will either be a valid NID or one of these values */ -#define NUMA_PTE_SCAN_INIT -1 -#define NUMA_PTE_SCAN_ACTIVE -2 -static inline void mm_init_cpumask (struct mm_struct *mm) {#ifdef CONFIG_CPUMASK_OFFSTACK
kernel/fork.c +0 -3
@@ -817,9 +817,6 @@ struct mm_struct *dup_mm(struct task_struct *tsk)#ifdef CONFIG_TRANSPARENT_HUGEPAGE mm->pmd_huge_pte =NULL ;#endif -#ifdef CONFIG_NUMA_BALANCING - mm->first_nid =NUMA_PTE_SCAN_INIT ; -#endif if (!mm_init (mm, tsk)) goto fail_nomem;
kernel/sched/fair.c +0 -18
@@ -901,24 +901,6 @@ void task_numa_work(struct callback_head *work)return ;/* -* We do not care about task placement until a task runs on a node -* other than the first one used by the address space. This is -* largely because migrations are driven by what CPU the task -* is running on. If it's never scheduled on another node, it'll -* not migrate so why bother trapping the fault. -*/ -if (mm->first_nid ==NUMA_PTE_SCAN_INIT ) - mm->first_nid =numa_node_id (); -if (mm->first_nid !=NUMA_PTE_SCAN_ACTIVE ) { -/* Are we running on a new node yet? */ -if (numa_node_id () == mm->first_nid && - !sched_feat_numa (NUMA_FORCE )) -return ; - - mm->first_nid =NUMA_PTE_SCAN_ACTIVE ; - } - -/* * Reset the scan period if enough time has gone by. Objective is that * scanning will be reduced if pages are properly placed. As tasks * can enter different phases this needs to be re-examined. Lacking
kernel/sched/features.h +1 -3
@@ -63,10 +63,8 @@ SCHED_FEAT(LB_MIN, false)/* * Apply the automatic NUMA scheduling policy. Enabled automatically * at runtime if running on a NUMA machine. Can be controlled via -* numa_balancing=. Allow PTE scanning to be forced on UMA machines -* for debugging the core machinery. +* numa_balancing= */ #ifdef CONFIG_NUMA_BALANCING SCHED_FEAT(NUMA, false) -SCHED_FEAT(NUMA_FORCE, false)#endif
Mel Gorman
reviewerPATCH 16/63 sched: numa: Initialise numa_next_scan properly
In reply to Mel GormanScan delay logic and resets are currently initialised to start scanning immediately instead of delaying properly. Initialise them properly at fork time and catch when a new mm has been allocated.
--- kernel/sched/core.c | 4 ++-- kernel/sched/fair.c | 7 +++++++ 2 files changed, 9 insertions(+), 2 deletions(-)
kernel/sched/core.c +2 -2
@@ -1619,8 +1619,8 @@ static void __sched_fork(struct task_struct *p)#ifdef CONFIG_NUMA_BALANCING if (p->mm &&atomic_read (&p->mm->mm_users) ==1 ) { - p->mm->numa_next_scan = jiffies; - p->mm->numa_next_reset = jiffies; + p->mm->numa_next_scan = jiffies +msecs_to_jiffies (sysctl_numa_balancing_scan_delay); + p->mm->numa_next_reset = jiffies +msecs_to_jiffies (sysctl_numa_balancing_scan_period_reset); p->mm->numa_scan_seq =0 ; }
kernel/sched/fair.c +7 -0
@@ -900,6 +900,13 @@ void task_numa_work(struct callback_head *work)if (p->flags &PF_EXITING )return ; +if (!mm->numa_next_reset || !mm->numa_next_scan) { + mm->numa_next_scan = now + +msecs_to_jiffies (sysctl_numa_balancing_scan_delay); + mm->numa_next_reset = now + +msecs_to_jiffies (sysctl_numa_balancing_scan_period_reset); + } +/* * Reset the scan period if enough time has gone by. Objective is that * scanning will be reduced if pages are properly placed. As tasks
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 16/63] sched: numa: Initialise numa_next_scan properly
Quoted 5 lines · Mel Gorman
Scan delay logic and resets are currently initialised to start scanning immediately instead of delaying properly. Initialise them properly at fork time and catch when a new mm has been allocated.
Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Initialise numa_next_scan properly
In reply to Mel Gormansched/numa: Initialise numa_next_scan properly
Commit-ID: 7e8d16b6cbccb2f5da579f5085479fb82ba851b8 Gitweb: http://git.kernel.org/tip/7e8d16b6cbccb2f5da579f5085479fb82ba851b8 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:54 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:19 +0200
sched/numa: Initialise numa_next_scan properly
Scan delay logic and resets are currently initialised to start scanning immediately instead of delaying properly. Initialise them properly at fork time and catch when a new mm has been allocated.
--- kernel/sched/core.c | 4 ++-- kernel/sched/fair.c | 7 +++++++ 2 files changed, 9 insertions(+), 2 deletions(-)
kernel/sched/core.c +2 -2
@@ -1624,8 +1624,8 @@ static void __sched_fork(struct task_struct *p)#ifdef CONFIG_NUMA_BALANCING if (p->mm &&atomic_read (&p->mm->mm_users) ==1 ) { - p->mm->numa_next_scan = jiffies; - p->mm->numa_next_reset = jiffies; + p->mm->numa_next_scan = jiffies +msecs_to_jiffies (sysctl_numa_balancing_scan_delay); + p->mm->numa_next_reset = jiffies +msecs_to_jiffies (sysctl_numa_balancing_scan_period_reset); p->mm->numa_scan_seq =0 ; }
kernel/sched/fair.c +7 -0
@@ -900,6 +900,13 @@ void task_numa_work(struct callback_head *work)if (p->flags &PF_EXITING )return ; +if (!mm->numa_next_reset || !mm->numa_next_scan) { + mm->numa_next_scan = now + +msecs_to_jiffies (sysctl_numa_balancing_scan_delay); + mm->numa_next_reset = now + +msecs_to_jiffies (sysctl_numa_balancing_scan_period_reset); + } +/* * Reset the scan period if enough time has gone by. Objective is that * scanning will be reduced if pages are properly placed. As tasks
Mel Gorman
reviewerPATCH 17/63 sched: Set the scan rate proportional to the memory usage of the task being scanned
In reply to Mel GormanThe NUMA PTE scan rate is controlled with a combination of the numa_balancing_scan_period_min, numa_balancing_scan_period_max and numa_balancing_scan_size. This scan rate is independent of the size of the task and as an aside it is further complicated by the fact that numa_balancing_scan_size controls how many pages are marked pte_numa and not how much virtual memory is scanned.
In combination, it is almost impossible to meaningfully tune the min and max scan periods and reasoning about performance is complex when the time to complete a full scan is is partially a function of the tasks memory size. This patch alters the semantic of the min and max tunables to be about tuning the length time it takes to complete a scan of a tasks occupied virtual address space. Conceptually this is a lot easier to understand. There is a "sanity" check to ensure the scan rate is never extremely fast based on the amount of virtual memory that should be scanned in a second. The default of 2.5G seems arbitrary but it is to have the maximum scan rate after the patch roughly match the maximum scan rate before the patch was applied.
On a similar note, numa_scan_period is in milliseconds and not jiffies. Properly placed pages slow the scanning rate but adding 10 jiffies to numa_scan_period means that the rate scanning slows depends on HZ which is confusing. Get rid of the jiffies_to_msec conversion and treat it as ms.
--- Documentation/sysctl/kernel.txt | 11 +++--- include/linux/sched.h | 1 + kernel/sched/fair.c | 88 +++++++++++++++++++++++++++++++++++------ 3 files changed, 83 insertions(+), 17 deletions(-)
Documentation/sysctl/kernel.txt +6 -5
@@ -403,15 +403,16 @@ workload pattern changes and minimises performance impact due to remote memory accesses. These sysctls control the thresholds for scan delays and the number of pages scanned. -numa_balancing_scan_period_min_ms is the minimum delay in milliseconds -between scans. It effectively controls the maximum scanning rate for -each task. +numa_balancing_scan_period_min_ms is the minimum time in milliseconds to +scan a tasks virtual memory. It effectively controls the maximum scanning +rate for each task. numa_balancing_scan_delay_ms is the starting "scan delay" used for a task when it initially forks. -numa_balancing_scan_period_max_ms is the maximum delay between scans. It -effectively controls the minimum scanning rate for each task. +numa_balancing_scan_period_max_ms is the maximum time in milliseconds to +scan a tasks virtual memory. It effectively controls the minimum scanning +rate for each task. numa_balancing_scan_size_mb is how many megabytes worth of pages are scanned for a given scan.
include/linux/sched.h +1 -0
@@ -1329,6 +1329,7 @@ struct task_struct {int numa_scan_seq;int numa_migrate_seq;unsigned int numa_scan_period; +unsigned int numa_scan_period_max;u64 node_stamp;/* migration stamp */ struct callback_head numa_work;#endif /* CONFIG_NUMA_BALANCING */
kernel/sched/fair.c +76 -12
143 lines not on this page · raw message
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 17/63] sched: Set the scan rate proportional to the memory usage of the task being scanned
Quoted 18 lines · Mel Gorman
The NUMA PTE scan rate is controlled with a combination of the
numa_balancing_scan_period_min,numa_balancing_scan_period_maxandnuma_balancing_scan_size. This scan rate is independent of the size of the task and as an aside it is further complicated by the fact thatnuma_balancing_scan_sizecontrols how many pages are markedpte_numaand not how much virtual memory is scanned.In combination, it is almost impossible to meaningfully tune the min and max scan periods and reasoning about performance is complex when the time to complete a full scan is is partially a function of the tasks memory size. This patch alters the semantic of the min and max tunables to be about tuning the length time it takes to complete a scan of a tasks occupied virtual address space. Conceptually this is a lot easier to understand. There is a "sanity" check to ensure the scan rate is never extremely fast based on the amount of virtual memory that should be scanned in a second. The default of 2.5G seems arbitrary but it is to have the maximum scan rate after the patch roughly match the maximum scan rate before the patch was applied.
Quoted 6 lines
On a similar note,
numa_scan_periodis in milliseconds and not jiffies. Properly placed pages slow the scanning rate but adding 10 jiffies tonuma_scan_periodmeans that the rate scanning slows depends on HZ which is confusing. Get rid of thejiffies_to_msecconversion and treat it as ms.Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Set the scan rate proportional to the memory usage of the task being scanned
In reply to Mel Gormansched/numa: Set the scan rate proportional to the memory usage of the task being scanned
Commit-ID: 598f0ec0bc996e90a806ee9564af919ea5aad401 Gitweb: http://git.kernel.org/tip/598f0ec0bc996e90a806ee9564af919ea5aad401 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:55 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:20 +0200
sched/numa: Set the scan rate proportional to the memory usage of the task being scanned
The NUMA PTE scan rate is controlled with a combination of the numa_balancing_scan_period_min, numa_balancing_scan_period_max and numa_balancing_scan_size. This scan rate is independent of the size of the task and as an aside it is further complicated by the fact that numa_balancing_scan_size controls how many pages are marked pte_numa and not how much virtual memory is scanned.
In combination, it is almost impossible to meaningfully tune the min and max scan periods and reasoning about performance is complex when the time to complete a full scan is is partially a function of the tasks memory size. This patch alters the semantic of the min and max tunables to be about tuning the length time it takes to complete a scan of a tasks occupied virtual address space. Conceptually this is a lot easier to understand. There is a "sanity" check to ensure the scan rate is never extremely fast based on the amount of virtual memory that should be scanned in a second. The default of 2.5G seems arbitrary but it is to have the maximum scan rate after the patch roughly match the maximum scan rate before the patch was applied.
On a similar note, numa_scan_period is in milliseconds and not jiffies. Properly placed pages slow the scanning rate but adding 10 jiffies to numa_scan_period means that the rate scanning slows depends on HZ which is confusing. Get rid of the jiffies_to_msec conversion and treat it as ms.
--- Documentation/sysctl/kernel.txt | 11 +++--- include/linux/sched.h | 1 + kernel/sched/fair.c | 88 +++++++++++++++++++++++++++++++++++------ 3 files changed, 83 insertions(+), 17 deletions(-)
Documentation/sysctl/kernel.txt +6 -5
@@ -403,15 +403,16 @@ workload pattern changes and minimises performance impact due to remote memory accesses. These sysctls control the thresholds for scan delays and the number of pages scanned. -numa_balancing_scan_period_min_ms is the minimum delay in milliseconds -between scans. It effectively controls the maximum scanning rate for -each task. +numa_balancing_scan_period_min_ms is the minimum time in milliseconds to +scan a tasks virtual memory. It effectively controls the maximum scanning +rate for each task. numa_balancing_scan_delay_ms is the starting "scan delay" used for a task when it initially forks. -numa_balancing_scan_period_max_ms is the maximum delay between scans. It -effectively controls the minimum scanning rate for each task. +numa_balancing_scan_period_max_ms is the maximum time in milliseconds to +scan a tasks virtual memory. It effectively controls the minimum scanning +rate for each task. numa_balancing_scan_size_mb is how many megabytes worth of pages are scanned for a given scan.
include/linux/sched.h +1 -0
@@ -1339,6 +1339,7 @@ struct task_struct {int numa_scan_seq;int numa_migrate_seq;unsigned int numa_scan_period; +unsigned int numa_scan_period_max;u64 node_stamp;/* migration stamp */ struct callback_head numa_work;#endif /* CONFIG_NUMA_BALANCING */
kernel/sched/fair.c +76 -12
143 lines not on this page · raw message
Mel Gorman
reviewerPATCH 18/63 sched: numa: Slow scan rate if no NUMA hinting faults are being recorded
In reply to Mel GormanNUMA PTE scanning slows if a NUMA hinting fault was trapped and no page was migrated. For long-lived but idle processes there may be no faults but the scan rate will be high and just waste CPU. This patch will slow the scan rate for processes that are not trapping faults.
--- kernel/sched/fair.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
kernel/sched/fair.c +12 -0
@@ -1039,6 +1039,18 @@ void task_numa_work(struct callback_head *work) out:/* +* If the whole process was scanned without updates then no NUMA +* hinting faults are being recorded and scan rate should be lower. +*/ +if (mm->numa_scan_offset ==0 && !nr_pte_updates) { + p->numa_scan_period =min (p->numa_scan_period_max, + p->numa_scan_period <<1 ); + + next_scan = now +msecs_to_jiffies (p->numa_scan_period); + mm->numa_next_scan = next_scan; + } + +/* * It is possible to reach the end of the VMA list but the last few * VMAs are not guaranteed to the vma_migratable. If they are not, we * would find the !migratable VMA on the next scan but not reset the
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 18/63] sched: numa: Slow scan rate if no NUMA hinting faults are being recorded
Quoted 6 lines · Mel Gorman
NUMA PTE scanning slows if a NUMA hinting fault was trapped and no page was migrated. For long-lived but idle processes there may be no faults but the scan rate will be high and just waste CPU. This patch will slow the scan rate for processes that are not trapping faults.
Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Slow scan rate if no NUMA hinting faults are being recorded
In reply to Mel Gormansched/numa: Slow scan rate if no NUMA hinting faults are being recorded
Commit-ID: f307cd1a32fab53012b01749a1f5ba10b0a7243f Gitweb: http://git.kernel.org/tip/f307cd1a32fab53012b01749a1f5ba10b0a7243f Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:56 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:21 +0200
sched/numa: Slow scan rate if no NUMA hinting faults are being recorded
NUMA PTE scanning slows if a NUMA hinting fault was trapped and no page was migrated. For long-lived but idle processes there may be no faults but the scan rate will be high and just waste CPU. This patch will slow the scan rate for processes that are not trapping faults.
--- kernel/sched/fair.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
kernel/sched/fair.c +12 -0
@@ -1039,6 +1039,18 @@ void task_numa_work(struct callback_head *work) out:/* +* If the whole process was scanned without updates then no NUMA +* hinting faults are being recorded and scan rate should be lower. +*/ +if (mm->numa_scan_offset ==0 && !nr_pte_updates) { + p->numa_scan_period =min (p->numa_scan_period_max, + p->numa_scan_period <<1 ); + + next_scan = now +msecs_to_jiffies (p->numa_scan_period); + mm->numa_next_scan = next_scan; + } + +/* * It is possible to reach the end of the VMA list but the last few * VMAs are not guaranteed to the vma_migratable. If they are not, we * would find the !migratable VMA on the next scan but not reset the
Mel Gorman
reviewerPATCH 19/63 sched: Track NUMA hinting faults on per-node basis
In reply to Mel GormanThis patch tracks what nodes numa hinting faults were incurred on. This information is later used to schedule a task on the node storing the pages most frequently faulted by the task.
--- include/linux/sched.h | 2 ++ kernel/sched/core.c | 3 +++ kernel/sched/fair.c | 11 ++++++++++- kernel/sched/sched.h | 12 ++++++++++++ 4 files changed, 27 insertions(+), 1 deletion(-)
include/linux/sched.h +2 -0
@@ -1332,6 +1332,8 @@ struct task_struct {unsigned int numa_scan_period_max;u64 node_stamp;/* migration stamp */ struct callback_head numa_work; + +unsigned long *numa_faults;#endif /* CONFIG_NUMA_BALANCING */ struct rcu_head rcu;
kernel/sched/core.c +3 -0
@@ -1629,6 +1629,7 @@ static void __sched_fork(struct task_struct *p) p->numa_migrate_seq = p->mm ? p->mm->numa_scan_seq -1 :0 ; p->numa_scan_period = sysctl_numa_balancing_scan_delay; p->numa_work.next = &p->numa_work; + p->numa_faults =NULL ;#endif /* CONFIG_NUMA_BALANCING */ cpu_hotplug_init_task (p);
@@ -1892,6 +1893,8 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)if (mm)mmdrop (mm);if (unlikely (prev_state ==TASK_DEAD )) { +task_numa_free (prev); +/* * Remove function-return probe instances associated with this * task and put them back on the free list.
kernel/sched/fair.c +10 -1
@@ -902,7 +902,14 @@ void task_numa_fault(int node, int pages, bool migrated)if (!numabalancing_enabled)return ; -/* FIXME: Allocate task-specific structure for placement policy here */ +/* Allocate buffer to track faults on a per-node basis */ +if (unlikely (!p->numa_faults)) { +int size =sizeof (*p->numa_faults) * nr_node_ids; + + p->numa_faults =kzalloc (size,GFP_KERNEL |__GFP_NOWARN); +if (!p->numa_faults) +return ; + }/* * If pages are properly placed (did not migrate) then scan slower.
@@ -918,6 +925,8 @@ void task_numa_fault(int node, int pages, bool migrated) }task_numa_placement (p); + + p->numa_faults[node] += pages; }static void reset_ptenuma_scan (struct task_struct *p)
kernel/sched/sched.h +12 -0
@@ -6,6 +6,7 @@#include <linux/spinlock.h> #include <linux/stop_machine.h> #include <linux/tick.h> +#include <linux/slab.h> #include "cpupri.h" #include "cpuacct.h"
@@ -552,6 +553,17 @@ static inline u64 rq_clock_task(struct rq *rq)return rq->clock_task; } +#ifdef CONFIG_NUMA_BALANCING +static inline void task_numa_free (struct task_struct *p) +{ +kfree (p->numa_faults); +} +#else /* CONFIG_NUMA_BALANCING */ +static inline void task_numa_free (struct task_struct *p) +{ +} +#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SMP #define rcu_dereference_check_sched_domain (p) \
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 19/63] sched: Track NUMA hinting faults on per-node basis
Quoted 5 lines · Mel Gorman
This patch tracks what nodes numa hinting faults were incurred on. This information is later used to schedule a task on the node storing the pages most frequently faulted by the task.
Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Track NUMA hinting faults on per-node basis
In reply to Mel Gormansched/numa: Track NUMA hinting faults on per-node basis
Commit-ID: f809ca9a554dda49fb264c79e31c722e0b063ff8 Gitweb: http://git.kernel.org/tip/f809ca9a554dda49fb264c79e31c722e0b063ff8 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:57 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:22 +0200
sched/numa: Track NUMA hinting faults on per-node basis
This patch tracks what nodes numa hinting faults were incurred on. This information is later used to schedule a task on the node storing the pages most frequently faulted by the task.
--- include/linux/sched.h | 2 ++ kernel/sched/core.c | 3 +++ kernel/sched/fair.c | 11 ++++++++++- kernel/sched/sched.h | 12 ++++++++++++ 4 files changed, 27 insertions(+), 1 deletion(-)
include/linux/sched.h +2 -0
@@ -1342,6 +1342,8 @@ struct task_struct {unsigned int numa_scan_period_max;u64 node_stamp;/* migration stamp */ struct callback_head numa_work; + +unsigned long *numa_faults;#endif /* CONFIG_NUMA_BALANCING */ struct rcu_head rcu;
kernel/sched/core.c +3 -0
@@ -1634,6 +1634,7 @@ static void __sched_fork(struct task_struct *p) p->numa_migrate_seq = p->mm ? p->mm->numa_scan_seq -1 :0 ; p->numa_scan_period = sysctl_numa_balancing_scan_delay; p->numa_work.next = &p->numa_work; + p->numa_faults =NULL ;#endif /* CONFIG_NUMA_BALANCING */ }
@@ -1892,6 +1893,8 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)if (mm)mmdrop (mm);if (unlikely (prev_state ==TASK_DEAD )) { +task_numa_free (prev); +/* * Remove function-return probe instances associated with this * task and put them back on the free list.
kernel/sched/fair.c +10 -1
@@ -902,7 +902,14 @@ void task_numa_fault(int node, int pages, bool migrated)if (!numabalancing_enabled)return ; -/* FIXME: Allocate task-specific structure for placement policy here */ +/* Allocate buffer to track faults on a per-node basis */ +if (unlikely (!p->numa_faults)) { +int size =sizeof (*p->numa_faults) * nr_node_ids; + + p->numa_faults =kzalloc (size,GFP_KERNEL |__GFP_NOWARN); +if (!p->numa_faults) +return ; + }/* * If pages are properly placed (did not migrate) then scan slower.
@@ -918,6 +925,8 @@ void task_numa_fault(int node, int pages, bool migrated) }task_numa_placement (p); + + p->numa_faults[node] += pages; }static void reset_ptenuma_scan (struct task_struct *p)
kernel/sched/sched.h +12 -0
@@ -6,6 +6,7 @@#include <linux/spinlock.h> #include <linux/stop_machine.h> #include <linux/tick.h> +#include <linux/slab.h> #include "cpupri.h" #include "cpuacct.h"
@@ -555,6 +556,17 @@ static inline u64 rq_clock_task(struct rq *rq)return rq->clock_task; } +#ifdef CONFIG_NUMA_BALANCING +static inline void task_numa_free (struct task_struct *p) +{ +kfree (p->numa_faults); +} +#else /* CONFIG_NUMA_BALANCING */ +static inline void task_numa_free (struct task_struct *p) +{ +} +#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SMP #define rcu_dereference_check_sched_domain (p) \
Re: [PATCH 19/63] sched: Track NUMA hinting faults on per-node basis
In reply to Mel GormanQuoted 38 lines · Mel Gorman
This patch tracks what nodes numa hinting faults were incurred on. This information is later used to schedule a task on the node storing the pages most frequently faulted by the task.
Signed-off-by Mel Gorman <[email protected]>---
include/linux/sched.h | 2 ++ kernel/sched/core.c | 3 +++ kernel/sched/fair.c | 11 ++++++++++- kernel/sched/sched.h | 12 ++++++++++++ 4 files changed, 27 insertions(+), 1 deletion(-)include/linux/sched.h +2
-0@@ -1332,6 +1332,8 @@ struct task_struct {unsigned int numa_scan_period_max;u64 node_stamp;/* migration stamp */ struct callback_head numa_work; + +unsigned long *numa_faults;#endif /* CONFIG_NUMA_BALANCING */ struct rcu_head rcu;kernel/sched/core.c +1
-0@@ -1629,6 +1629,7 @@ static void __sched_fork(struct task_struct *p) p->numa_migrate_seq = p->mm ? p->mm->numa_scan_seq -1 :0 ; p->numa_scan_period = sysctl_numa_balancing_scan_delay; p->numa_work.next = &p->numa_work; + p->numa_faults =NULL ;#endif /* CONFIG_NUMA_BALANCING */ cpu_hotplug_init_task (p);
kernel/sched/core.c static void finish_task_switch(struct rq *rq, struct task_struct *prev) · line 1896
Quoted 4 lines
kernel/sched/core.c +1
-0@@ -1892,6 +1893,8 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)if (mm)mmdrop (mm);if (unlikely (prev_state ==TASK_DEAD )) { +task_numa_free (prev);
Function task_numa_free() depends on patch 43/64.
Regards,
Wanpeng Li
Quoted 71 lines
kernel/sched/core.c +1
-0+/* * Remove function-return probe instances associated with this * task and put them back on the free list. kernel/sched/fair.c +10
-1@@ -902,7 +902,14 @@ void task_numa_fault(int node, int pages, bool migrated)if (!numabalancing_enabled)return ; -/* FIXME: Allocate task-specific structure for placement policy here */ +/* Allocate buffer to track faults on a per-node basis */ +if (unlikely (!p->numa_faults)) { +int size =sizeof (*p->numa_faults) * nr_node_ids; + + p->numa_faults =kzalloc (size,GFP_KERNEL |__GFP_NOWARN); +if (!p->numa_faults) +return ; + }/* * If pages are properly placed (did not migrate) then scan slower. @@ -918,6 +925,8 @@ void task_numa_fault(int node, int pages, bool migrated) }task_numa_placement (p); + + p->numa_faults[node] += pages; }static void reset_ptenuma_scan (struct task_struct *p)kernel/sched/sched.h +12
-0@@ -6,6 +6,7 @@#include <linux/spinlock.h> #include <linux/stop_machine.h> #include <linux/tick.h> +#include <linux/slab.h> #include "cpupri.h" #include "cpuacct.h" @@ -552,6 +553,17 @@ static inline u64 rq_clock_task(struct rq *rq)return rq->clock_task; } +#ifdef CONFIG_NUMA_BALANCING +static inline void task_numa_free (struct task_struct *p) +{ +kfree (p->numa_faults); +} +#else /* CONFIG_NUMA_BALANCING */ +static inline void task_numa_free (struct task_struct *p) +{ +} +#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SMP #define rcu_dereference_check_sched_domain (p) \Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 19/63] sched: Track NUMA hinting faults on per-node basis
Quoted 37 lines · Wanpeng Li
Quoted 36 lines · Mel Gorman
This patch tracks what nodes numa hinting faults were incurred on. This information is later used to schedule a task on the node storing the pages most frequently faulted by the task.
Signed-off-by Mel Gorman <[email protected]>---
include/linux/sched.h | 2 ++ kernel/sched/core.c | 3 +++ kernel/sched/fair.c | 11 ++++++++++- kernel/sched/sched.h | 12 ++++++++++++ 4 files changed, 27 insertions(+), 1 deletion(-)include/linux/sched.h +2
-0@@ -1332,6 +1332,8 @@ struct task_struct {unsigned int numa_scan_period_max;u64 node_stamp;/* migration stamp */ struct callback_head numa_work; + +unsigned long *numa_faults;#endif /* CONFIG_NUMA_BALANCING */ struct rcu_head rcu;kernel/sched/core.c +1
-0@@ -1629,6 +1629,7 @@ static void __sched_fork(struct task_struct *p) p->numa_migrate_seq = p->mm ? p->mm->numa_scan_seq -1 :0 ; p->numa_scan_period = sysctl_numa_balancing_scan_delay; p->numa_work.next = &p->numa_work; + p->numa_faults =NULL ;#endif /* CONFIG_NUMA_BALANCING */
Quoted 8 lines
kernel/sched/core.c static void finish_task_switch(struct rq *rq, struct task_struct *prev) · line 1896
Quoted 6 lines
kernel/sched/core.c +1
-0cpu_hotplug_init_task (p);@@ -1892,6 +1893,8 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)if (mm)mmdrop (mm);if (unlikely (prev_state ==TASK_DEAD )) { +task_numa_free (prev);Function
task_numa_free()depends on patch 43/64.
Sorry, I miss it.
Quoted 70 lines
Regards,
Wanpeng LiQuoted 66 lines
kernel/sched/core.c +1
-0+/* * Remove function-return probe instances associated with this * task and put them back on the free list. kernel/sched/fair.c +10
-1@@ -902,7 +902,14 @@ void task_numa_fault(int node, int pages, bool migrated)if (!numabalancing_enabled)return ; -/* FIXME: Allocate task-specific structure for placement policy here */ +/* Allocate buffer to track faults on a per-node basis */ +if (unlikely (!p->numa_faults)) { +int size =sizeof (*p->numa_faults) * nr_node_ids; + + p->numa_faults =kzalloc (size,GFP_KERNEL |__GFP_NOWARN); +if (!p->numa_faults) +return ; + }/* * If pages are properly placed (did not migrate) then scan slower. @@ -918,6 +925,8 @@ void task_numa_fault(int node, int pages, bool migrated) }task_numa_placement (p); + + p->numa_faults[node] += pages; }static void reset_ptenuma_scan (struct task_struct *p)kernel/sched/sched.h +12
-0@@ -6,6 +6,7 @@#include <linux/spinlock.h> #include <linux/stop_machine.h> #include <linux/tick.h> +#include <linux/slab.h> #include "cpupri.h" #include "cpuacct.h" @@ -552,6 +553,17 @@ static inline u64 rq_clock_task(struct rq *rq)return rq->clock_task; } +#ifdef CONFIG_NUMA_BALANCING +static inline void task_numa_free (struct task_struct *p) +{ +kfree (p->numa_faults); +} +#else /* CONFIG_NUMA_BALANCING */ +static inline void task_numa_free (struct task_struct *p) +{ +} +#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SMP #define rcu_dereference_check_sched_domain (p) \Signature · 1 line
1.8.4
Quoted 11 lines
Quoted 5 lines
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 20/63 sched: Select a preferred node with the most numa hinting faults
In reply to Mel GormanThis patch selects a preferred node for a task to run on based on the NUMA hinting faults. This information is later used to migrate tasks towards the node during balancing.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 1 + kernel/sched/fair.c | 17 +++++++++++++++-- 3 files changed, 17 insertions(+), 2 deletions(-)
include/linux/sched.h +1 -0
@@ -1334,6 +1334,7 @@ struct task_struct {struct callback_head numa_work;unsigned long *numa_faults; +int numa_preferred_nid;#endif /* CONFIG_NUMA_BALANCING */ struct rcu_head rcu;
kernel/sched/core.c +1 -0
@@ -1628,6 +1628,7 @@ static void __sched_fork(struct task_struct *p) p->numa_scan_seq = p->mm ? p->mm->numa_scan_seq :0 ; p->numa_migrate_seq = p->mm ? p->mm->numa_scan_seq -1 :0 ; p->numa_scan_period = sysctl_numa_balancing_scan_delay; + p->numa_preferred_nid =-1 ; p->numa_work.next = &p->numa_work; p->numa_faults =NULL ;#endif /* CONFIG_NUMA_BALANCING */
kernel/sched/fair.c +15 -2
@@ -879,7 +879,8 @@ static unsigned int task_scan_max(struct task_struct *p)static void task_numa_placement (struct task_struct *p) { -int seq; +int seq, nid, max_nid =-1 ; +unsigned long max_faults =0 ;if (!p->mm)/* for example, ksmd faulting in a user's mm */ return ;
@@ -889,7 +890,19 @@ static void task_numa_placement(struct task_struct *p) p->numa_scan_seq = seq; p->numa_scan_period_max =task_scan_max (p); -/* FIXME: Scheduling placement policy hints go here */ +/* Find the node with the highest number of faults */ +for_each_online_node (nid) { +unsigned long faults = p->numa_faults[nid]; + p->numa_faults[nid] >>=1 ; +if (faults > max_faults) { + max_faults = faults; + max_nid = nid; + } + } + +/* Update the tasks preferred node if necessary */ +if (max_faults && max_nid != p->numa_preferred_nid) + p->numa_preferred_nid = max_nid; }/*
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/63] sched: Select a preferred node with the most numa hinting faults
Quoted 5 lines · Mel Gorman
This patch selects a preferred node for a task to run on based on the NUMA hinting faults. This information is later used to migrate tasks towards the node during balancing.
Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Select a preferred node with the most numa hinting faults
In reply to Mel Gormansched/numa: Select a preferred node with the most numa hinting faults
Commit-ID: 688b7585d16ab57a17aa4422a3b290b3a55fa679 Gitweb: http://git.kernel.org/tip/688b7585d16ab57a17aa4422a3b290b3a55fa679 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:58 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:23 +0200
sched/numa: Select a preferred node with the most numa hinting faults
This patch selects a preferred node for a task to run on based on the NUMA hinting faults. This information is later used to migrate tasks towards the node during balancing.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 1 + kernel/sched/fair.c | 17 +++++++++++++++-- 3 files changed, 17 insertions(+), 2 deletions(-)
include/linux/sched.h +1 -0
@@ -1344,6 +1344,7 @@ struct task_struct {struct callback_head numa_work;unsigned long *numa_faults; +int numa_preferred_nid;#endif /* CONFIG_NUMA_BALANCING */ struct rcu_head rcu;
kernel/sched/core.c +1 -0
@@ -1633,6 +1633,7 @@ static void __sched_fork(struct task_struct *p) p->numa_scan_seq = p->mm ? p->mm->numa_scan_seq :0 ; p->numa_migrate_seq = p->mm ? p->mm->numa_scan_seq -1 :0 ; p->numa_scan_period = sysctl_numa_balancing_scan_delay; + p->numa_preferred_nid =-1 ; p->numa_work.next = &p->numa_work; p->numa_faults =NULL ;#endif /* CONFIG_NUMA_BALANCING */
kernel/sched/fair.c +15 -2
@@ -879,7 +879,8 @@ static unsigned int task_scan_max(struct task_struct *p)static void task_numa_placement (struct task_struct *p) { -int seq; +int seq, nid, max_nid =-1 ; +unsigned long max_faults =0 ;if (!p->mm)/* for example, ksmd faulting in a user's mm */ return ;
@@ -889,7 +890,19 @@ static void task_numa_placement(struct task_struct *p) p->numa_scan_seq = seq; p->numa_scan_period_max =task_scan_max (p); -/* FIXME: Scheduling placement policy hints go here */ +/* Find the node with the highest number of faults */ +for_each_online_node (nid) { +unsigned long faults = p->numa_faults[nid]; + p->numa_faults[nid] >>=1 ; +if (faults > max_faults) { + max_faults = faults; + max_nid = nid; + } + } + +/* Update the tasks preferred node if necessary */ +if (max_faults && max_nid != p->numa_preferred_nid) + p->numa_preferred_nid = max_nid; }/*
Mel Gorman
reviewerPATCH 21/63 sched: Update NUMA hinting faults once per scan
In reply to Mel GormanNUMA hinting fault counts and placement decisions are both recorded in the same array which distorts the samples in an unpredictable fashion. The values linearly accumulate during the scan and then decay creating a sawtooth-like pattern in the per-node counts. It also means that placement decisions are time sensitive. At best it means that it is very difficult to state that the buffer holds a decaying average of past faulting behaviour. At worst, it can confuse the load balancer if it sees one node with an artifically high count due to very recent faulting activity and may create a bouncing effect.
This patch adds a second array. numa_faults stores the historical data which is used for placement decisions. numa_faults_buffer holds the fault activity during the current scan window. When the scan completes, numa_faults decays and the values from numa_faults_buffer are copied across.
--- include/linux/sched.h | 13 +++++++++++++ kernel/sched/core.c | 1 + kernel/sched/fair.c | 16 +++++++++++++--- 3 files changed, 27 insertions(+), 3 deletions(-)
include/linux/sched.h +13 -0
@@ -1333,7 +1333,20 @@ struct task_struct {u64 node_stamp;/* migration stamp */ struct callback_head numa_work; +/* +* Exponential decaying average of faults on a per-node basis. +* Scheduling placement decisions are made based on the these counts. +* The values remain static for the duration of a PTE scan +*/ unsigned long *numa_faults; + +/* +* numa_faults_buffer records faults per node during the current +* scan window. When the scan completes, the counts in numa_faults +* decay and these values are copied. +*/ +unsigned long *numa_faults_buffer; +int numa_preferred_nid;#endif /* CONFIG_NUMA_BALANCING */
kernel/sched/core.c +1 -0
@@ -1631,6 +1631,7 @@ static void __sched_fork(struct task_struct *p) p->numa_preferred_nid =-1 ; p->numa_work.next = &p->numa_work; p->numa_faults =NULL ; + p->numa_faults_buffer =NULL ;#endif /* CONFIG_NUMA_BALANCING */ cpu_hotplug_init_task (p);
kernel/sched/fair.c +13 -3
@@ -892,8 +892,14 @@ static void task_numa_placement(struct task_struct *p)/* Find the node with the highest number of faults */ for_each_online_node (nid) { -unsigned long faults = p->numa_faults[nid]; +unsigned long faults; + +/* Decay existing window and copy faults since last scan */ p->numa_faults[nid] >>=1 ; + p->numa_faults[nid] += p->numa_faults_buffer[nid]; + p->numa_faults_buffer[nid] =0 ; + + faults = p->numa_faults[nid];if (faults > max_faults) { max_faults = faults; max_nid = nid;
@@ -919,9 +925,13 @@ void task_numa_fault(int node, int pages, bool migrated)if (unlikely (!p->numa_faults)) {int size =sizeof (*p->numa_faults) * nr_node_ids; - p->numa_faults =kzalloc (size,GFP_KERNEL |__GFP_NOWARN); +/* numa_faults and numa_faults_buffer share the allocation */ + p->numa_faults =kzalloc (size *2 ,GFP_KERNEL |__GFP_NOWARN);if (!p->numa_faults)return ; + +BUG_ON (p->numa_faults_buffer); + p->numa_faults_buffer = p->numa_faults + nr_node_ids; }/*
@@ -939,7 +949,7 @@ void task_numa_fault(int node, int pages, bool migrated)task_numa_placement (p); - p->numa_faults[node] += pages; + p->numa_faults_buffer[node] += pages; }static void reset_ptenuma_scan (struct task_struct *p)
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 21/63] sched: Update NUMA hinting faults once per scan
Quoted 9 lines · Mel Gorman
NUMA hinting fault counts and placement decisions are both recorded in the same array which distorts the samples in an unpredictable fashion. The values linearly accumulate during the scan and then decay creating a sawtooth-like pattern in the per-node counts. It also means that placement decisions are time sensitive. At best it means that it is very difficult to state that the buffer holds a decaying average of past faulting behaviour. At worst, it can confuse the load balancer if it sees one node with an artifically high count due to very recent faulting activity and may create a bouncing effect.
Quoted 7 lines
This patch adds a second array.
numa_faultsstores the historical data which is used for placement decisions.numa_faults_bufferholds the fault activity during the current scan window. When the scan completes,numa_faultsdecays and the values fromnuma_faults_bufferare copied across.Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Update NUMA hinting faults once per scan
In reply to Mel Gormansched/numa: Update NUMA hinting faults once per scan
Commit-ID: 745d61476ddb737aad3495fa6d9a8f8c2ee59f86 Gitweb: http://git.kernel.org/tip/745d61476ddb737aad3495fa6d9a8f8c2ee59f86 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:28:59 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:25 +0200
sched/numa: Update NUMA hinting faults once per scan
NUMA hinting fault counts and placement decisions are both recorded in the same array which distorts the samples in an unpredictable fashion. The values linearly accumulate during the scan and then decay creating a sawtooth-like pattern in the per-node counts. It also means that placement decisions are time sensitive. At best it means that it is very difficult to state that the buffer holds a decaying average of past faulting behaviour. At worst, it can confuse the load balancer if it sees one node with an artifically high count due to very recent faulting activity and may create a bouncing effect.
This patch adds a second array. numa_faults stores the historical data which is used for placement decisions. numa_faults_buffer holds the fault activity during the current scan window. When the scan completes, numa_faults decays and the values from numa_faults_buffer are copied across.
--- include/linux/sched.h | 13 +++++++++++++ kernel/sched/core.c | 1 + kernel/sched/fair.c | 16 +++++++++++++--- 3 files changed, 27 insertions(+), 3 deletions(-)
include/linux/sched.h +13 -0
@@ -1343,7 +1343,20 @@ struct task_struct {u64 node_stamp;/* migration stamp */ struct callback_head numa_work; +/* +* Exponential decaying average of faults on a per-node basis. +* Scheduling placement decisions are made based on the these counts. +* The values remain static for the duration of a PTE scan +*/ unsigned long *numa_faults; + +/* +* numa_faults_buffer records faults per node during the current +* scan window. When the scan completes, the counts in numa_faults +* decay and these values are copied. +*/ +unsigned long *numa_faults_buffer; +int numa_preferred_nid;#endif /* CONFIG_NUMA_BALANCING */
kernel/sched/core.c +1 -0
@@ -1636,6 +1636,7 @@ static void __sched_fork(struct task_struct *p) p->numa_preferred_nid =-1 ; p->numa_work.next = &p->numa_work; p->numa_faults =NULL ; + p->numa_faults_buffer =NULL ;#endif /* CONFIG_NUMA_BALANCING */ }
kernel/sched/fair.c +13 -3
@@ -892,8 +892,14 @@ static void task_numa_placement(struct task_struct *p)/* Find the node with the highest number of faults */ for_each_online_node (nid) { -unsigned long faults = p->numa_faults[nid]; +unsigned long faults; + +/* Decay existing window and copy faults since last scan */ p->numa_faults[nid] >>=1 ; + p->numa_faults[nid] += p->numa_faults_buffer[nid]; + p->numa_faults_buffer[nid] =0 ; + + faults = p->numa_faults[nid];if (faults > max_faults) { max_faults = faults; max_nid = nid;
@@ -919,9 +925,13 @@ void task_numa_fault(int node, int pages, bool migrated)if (unlikely (!p->numa_faults)) {int size =sizeof (*p->numa_faults) * nr_node_ids; - p->numa_faults =kzalloc (size,GFP_KERNEL |__GFP_NOWARN); +/* numa_faults and numa_faults_buffer share the allocation */ + p->numa_faults =kzalloc (size *2 ,GFP_KERNEL |__GFP_NOWARN);if (!p->numa_faults)return ; + +BUG_ON (p->numa_faults_buffer); + p->numa_faults_buffer = p->numa_faults + nr_node_ids; }/*
@@ -939,7 +949,7 @@ void task_numa_fault(int node, int pages, bool migrated)task_numa_placement (p); - p->numa_faults[node] += pages; + p->numa_faults_buffer[node] += pages; }static void reset_ptenuma_scan (struct task_struct *p)
Mel Gorman
reviewerPATCH 22/63 sched: Favour moving tasks towards the preferred node
In reply to Mel GormanThis patch favours moving tasks towards NUMA node that recorded a higher number of NUMA faults during active load balancing. Ideally this is self-reinforcing as the longer the task runs on that node, the more faults it should incur causing task_numa_placement to keep the task running on that node. In reality a big weakness is that the nodes CPUs can be overloaded and it would be more efficient to queue tasks on an idle node and migrate to the new node. This would require additional smarts in the balancer so for now the balancer will simply prefer to place the task on the preferred node for a PTE scans which is controlled by the numa_balancing_settle_count sysctl. Once the settle_count number of scans has complete the schedule is free to place the task on an alternative node if the load is imbalanced.
--- Documentation/sysctl/kernel.txt | 8 +++++- include/linux/sched.h | 1 + kernel/sched/core.c | 3 +- kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++++--- kernel/sched/features.h | 7 +++++ kernel/sysctl.c | 7 +++++ 6 files changed, 83 insertions(+), 6 deletions(-)
Documentation/sysctl/kernel.txt +7 -1
@@ -375,7 +375,8 @@ feature should be disabled. Otherwise, if the system overhead from the feature is too high then the rate the kernel samples for NUMA hinting faults may be controlled by the numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, numa_balancing_scan_period_reset, -numa_balancing_scan_period_max_ms and numa_balancing_scan_size_mb sysctls. +numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb and +numa_balancing_settle_count sysctls. ==============================================================
@@ -420,6 +421,11 @@ scanned for a given scan. numa_balancing_scan_period_reset is a blunt instrument that controls how often a tasks scan delay is reset to detect sudden changes in task behaviour. +numa_balancing_settle_count is how many scan periods must complete before +the schedule balancer stops pushing the task towards a preferred node. This +gives the scheduler a chance to place the task on an alternative node if the +preferred node is overloaded. + ============================================================== osrelease, ostype & version:
include/linux/sched.h +1 -0
@@ -768,6 +768,7 @@ enum cpu_idle_type {#define SD_ASYM_PACKING 0x0800/* Place busy groups earlier in the domain */ #define SD_PREFER_SIBLING 0x1000/* Prefer to place tasks in a sibling domain */ #define SD_OVERLAP 0x2000/* sched_domains of this level overlap */ +#define SD_NUMA 0x4000/* cross-node balancing */ extern int __weak arch_sd_sibiling_asym_packing (void );
kernel/sched/core.c +2 -1
@@ -1626,7 +1626,7 @@ static void __sched_fork(struct task_struct *p) p->node_stamp =0ULL ; p->numa_scan_seq = p->mm ? p->mm->numa_scan_seq :0 ; - p->numa_migrate_seq = p->mm ? p->mm->numa_scan_seq -1 :0 ; + p->numa_migrate_seq =0 ; p->numa_scan_period = sysctl_numa_balancing_scan_delay; p->numa_preferred_nid =-1 ; p->numa_work.next = &p->numa_work;
@@ -5661,6 +5661,7 @@ sd_numa_init(struct sched_domain_topology_level *tl, int cpu) | 0*SD_SHARE_PKG_RESOURCES |1 *SD_SERIALIZE |0 *SD_PREFER_SIBLING + |1 *SD_NUMA |sd_local_flags (level) , .last_balance = jiffies,
kernel/sched/fair.c +59 -4
96 lines not on this page · raw message
kernel/sched/features.h +7 -0
@@ -67,4 +67,11 @@ SCHED_FEAT(LB_MIN, false)*/ #ifdef CONFIG_NUMA_BALANCING SCHED_FEAT (NUMA , false) + +/* +* NUMA_FAVOUR_HIGHER will favor moving tasks towards nodes where a +* higher number of hinting faults are recorded during active load +* balancing. + */ +SCHED_FEAT(NUMA_FAVOUR_HIGHER, true)#endif
kernel/sysctl.c +7 -0
@@ -391,6 +391,13 @@ static struct ctl_table kern_table[] = { .mode = 0644, .proc_handler = proc_dointvec, }, + { + .procname = "numa_balancing_settle_count", + .data = &sysctl_numa_balancing_settle_count, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec, + },#endif /* CONFIG_NUMA_BALANCING */ #endif /* CONFIG_SCHED_DEBUG */ {
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 22/63] sched: Favour moving tasks towards the preferred node
Quoted 12 lines · Mel Gorman
This patch favours moving tasks towards NUMA node that recorded a higher number of NUMA faults during active load balancing. Ideally this is self-reinforcing as the longer the task runs on that node, the more faults it should incur causing
task_numa_placementto keep the task running on that node. In reality a big weakness is that the nodes CPUs can be overloaded and it would be more efficient to queue tasks on an idle node and migrate to the new node. This would require additional smarts in the balancer so for now the balancer will simply prefer to place the task on the preferred node for a PTE scans which is controlled by thenuma_balancing_settle_countsysctl. Once thesettle_countnumber of scans has complete the schedule is free to place the task on an alternative node if the load is imbalanced.
Quoted 4 lines
[[email protected]: Fixed statistics][[email protected]: Tunable and use higher faults instead of preferred]Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Favour moving tasks towards the preferred node
In reply to Mel Gormansched/numa: Favour moving tasks towards the preferred node
Commit-ID: 3a7053b3224f4a8b0e8184166190076593621617 Gitweb: http://git.kernel.org/tip/3a7053b3224f4a8b0e8184166190076593621617 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:00 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:26 +0200
sched/numa: Favour moving tasks towards the preferred node
This patch favours moving tasks towards NUMA node that recorded a higher number of NUMA faults during active load balancing. Ideally this is self-reinforcing as the longer the task runs on that node, the more faults it should incur causing task_numa_placement to keep the task running on that node. In reality a big weakness is that the nodes CPUs can be overloaded and it would be more efficient to queue tasks on an idle node and migrate to the new node. This would require additional smarts in the balancer so for now the balancer will simply prefer to place the task on the preferred node for a PTE scans which is controlled by the numa_balancing_settle_count sysctl. Once the settle_count number of scans has complete the schedule is free to place the task on an alternative node if the load is imbalanced.
--- Documentation/sysctl/kernel.txt | 8 +++++- include/linux/sched.h | 1 + kernel/sched/core.c | 3 +- kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++++--- kernel/sched/features.h | 7 +++++ kernel/sysctl.c | 7 +++++ 6 files changed, 83 insertions(+), 6 deletions(-)
Documentation/sysctl/kernel.txt +7 -1
@@ -375,7 +375,8 @@ feature should be disabled. Otherwise, if the system overhead from the feature is too high then the rate the kernel samples for NUMA hinting faults may be controlled by the numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, numa_balancing_scan_period_reset, -numa_balancing_scan_period_max_ms and numa_balancing_scan_size_mb sysctls. +numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb and +numa_balancing_settle_count sysctls. ==============================================================
@@ -420,6 +421,11 @@ scanned for a given scan. numa_balancing_scan_period_reset is a blunt instrument that controls how often a tasks scan delay is reset to detect sudden changes in task behaviour. +numa_balancing_settle_count is how many scan periods must complete before +the schedule balancer stops pushing the task towards a preferred node. This +gives the scheduler a chance to place the task on an alternative node if the +preferred node is overloaded. + ============================================================== osrelease, ostype & version:
include/linux/sched.h +1 -0
@@ -777,6 +777,7 @@ enum cpu_idle_type {#define SD_ASYM_PACKING 0x0800/* Place busy groups earlier in the domain */ #define SD_PREFER_SIBLING 0x1000/* Prefer to place tasks in a sibling domain */ #define SD_OVERLAP 0x2000/* sched_domains of this level overlap */ +#define SD_NUMA 0x4000/* cross-node balancing */ extern int __weak arch_sd_sibiling_asym_packing (void );
kernel/sched/core.c +2 -1
@@ -1631,7 +1631,7 @@ static void __sched_fork(struct task_struct *p) p->node_stamp =0ULL ; p->numa_scan_seq = p->mm ? p->mm->numa_scan_seq :0 ; - p->numa_migrate_seq = p->mm ? p->mm->numa_scan_seq -1 :0 ; + p->numa_migrate_seq =0 ; p->numa_scan_period = sysctl_numa_balancing_scan_delay; p->numa_preferred_nid =-1 ; p->numa_work.next = &p->numa_work;
@@ -5656,6 +5656,7 @@ sd_numa_init(struct sched_domain_topology_level *tl, int cpu) | 0*SD_SHARE_PKG_RESOURCES |1 *SD_SERIALIZE |0 *SD_PREFER_SIBLING + |1 *SD_NUMA |sd_local_flags (level) , .last_balance = jiffies,
kernel/sched/fair.c +59 -4
96 lines not on this page · raw message
kernel/sched/features.h +7 -0
@@ -67,4 +67,11 @@ SCHED_FEAT(LB_MIN, false)*/ #ifdef CONFIG_NUMA_BALANCING SCHED_FEAT (NUMA , false) + +/* +* NUMA_FAVOUR_HIGHER will favor moving tasks towards nodes where a +* higher number of hinting faults are recorded during active load +* balancing. + */ +SCHED_FEAT(NUMA_FAVOUR_HIGHER, true)#endif
kernel/sysctl.c +7 -0
@@ -391,6 +391,13 @@ static struct ctl_table kern_table[] = { .mode = 0644, .proc_handler = proc_dointvec, }, + { + .procname = "numa_balancing_settle_count", + .data = &sysctl_numa_balancing_settle_count, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec, + },#endif /* CONFIG_NUMA_BALANCING */ #endif /* CONFIG_SCHED_DEBUG */ {
Mel Gorman
reviewerPATCH 23/63 sched: Resist moving tasks towards nodes with fewer hinting faults
In reply to Mel GormanJust as "sched: Favour moving tasks towards the preferred node" favours moving tasks towards nodes with a higher number of recorded NUMA hinting faults, this patch resists moving tasks towards nodes with lower faults.
--- kernel/sched/fair.c | 33 +++++++++++++++++++++++++++++++++ kernel/sched/features.h | 8 ++++++++ 2 files changed, 41 insertions(+)
kernel/sched/fair.c +33 -0
@@ -4106,12 +4106,43 @@ static bool migrate_improves_locality(struct task_struct *p, struct lb_env *env)return false; } + + +static bool migrate_degrades_locality (struct task_struct *p,struct lb_env *env) +{ +int src_nid, dst_nid; + +if (!sched_feat (NUMA ) || !sched_feat (NUMA_RESIST_LOWER )) +return false; + +if (!p->numa_faults || !(env->sd->flags &SD_NUMA )) +return false; + + src_nid =cpu_to_node (env->src_cpu); + dst_nid =cpu_to_node (env->dst_cpu); + +if (src_nid == dst_nid || + p->numa_migrate_seq >= sysctl_numa_balancing_settle_count) +return false; + +if (p->numa_faults[dst_nid] < p->numa_faults[src_nid]) +return true; + +return false; +} +#else static inline bool migrate_improves_locality (struct task_struct *p,struct lb_env *env) {return false; } + +static inline bool migrate_degrades_locality (struct task_struct *p, +struct lb_env *env) +{ +return false; +}#endif /*
@@ -4174,6 +4205,8 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* 3) too many balance attempts have failed. */ tsk_cache_hot =task_hot (p,rq_clock_task (env->src_rq), env->sd); +if (!tsk_cache_hot) + tsk_cache_hot =migrate_degrades_locality (p, env);if (migrate_improves_locality (p, env)) {#ifdef CONFIG_SCHEDSTATS
kernel/sched/features.h +8 -0
@@ -74,4 +74,12 @@ SCHED_FEAT(NUMA, false)* balancing. */ SCHED_FEAT (NUMA_FAVOUR_HIGHER , true) + +/* +* NUMA_RESIST_LOWER will resist moving tasks towards nodes where a +* lower number of hinting faults have been recorded. As this has +* the potential to prevent a task ever migrating to a new node +* due to CPU overload it is disabled by default. + */ +SCHED_FEAT(NUMA_RESIST_LOWER, false)#endif
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 23/63] sched: Resist moving tasks towards nodes with fewer hinting faults
Quoted 7 lines · Mel Gorman
Just as "sched: Favour moving tasks towards the preferred node" favours moving tasks towards nodes with a higher number of recorded NUMA hinting faults, this patch resists moving tasks towards nodes with lower faults.
[[email protected]: changelog]Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Resist moving tasks towards nodes with fewer hinting faults
In reply to Mel Gormansched/numa: Resist moving tasks towards nodes with fewer hinting faults
Commit-ID: 7a0f308337d11fd5caa9f845c6d08cc5d6067988 Gitweb: http://git.kernel.org/tip/7a0f308337d11fd5caa9f845c6d08cc5d6067988 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:01 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:27 +0200
sched/numa: Resist moving tasks towards nodes with fewer hinting faults
Just as "sched: Favour moving tasks towards the preferred node" favours moving tasks towards nodes with a higher number of recorded NUMA hinting faults, this patch resists moving tasks towards nodes with lower faults.
--- kernel/sched/fair.c | 33 +++++++++++++++++++++++++++++++++ kernel/sched/features.h | 8 ++++++++ 2 files changed, 41 insertions(+)
kernel/sched/fair.c +33 -0
@@ -4107,12 +4107,43 @@ static bool migrate_improves_locality(struct task_struct *p, struct lb_env *env)return false; } + + +static bool migrate_degrades_locality (struct task_struct *p,struct lb_env *env) +{ +int src_nid, dst_nid; + +if (!sched_feat (NUMA ) || !sched_feat (NUMA_RESIST_LOWER )) +return false; + +if (!p->numa_faults || !(env->sd->flags &SD_NUMA )) +return false; + + src_nid =cpu_to_node (env->src_cpu); + dst_nid =cpu_to_node (env->dst_cpu); + +if (src_nid == dst_nid || + p->numa_migrate_seq >= sysctl_numa_balancing_settle_count) +return false; + +if (p->numa_faults[dst_nid] < p->numa_faults[src_nid]) +return true; + +return false; +} +#else static inline bool migrate_improves_locality (struct task_struct *p,struct lb_env *env) {return false; } + +static inline bool migrate_degrades_locality (struct task_struct *p, +struct lb_env *env) +{ +return false; +}#endif /*
@@ -4177,6 +4208,8 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* 3) too many balance attempts have failed. */ tsk_cache_hot =task_hot (p,rq_clock_task (env->src_rq), env->sd); +if (!tsk_cache_hot) + tsk_cache_hot =migrate_degrades_locality (p, env);if (migrate_improves_locality (p, env)) {#ifdef CONFIG_SCHEDSTATS
kernel/sched/features.h +8 -0
@@ -74,4 +74,12 @@ SCHED_FEAT(NUMA, false)* balancing. */ SCHED_FEAT (NUMA_FAVOUR_HIGHER , true) + +/* +* NUMA_RESIST_LOWER will resist moving tasks towards nodes where a +* lower number of hinting faults have been recorded. As this has +* the potential to prevent a task ever migrating to a new node +* due to CPU overload it is disabled by default. + */ +SCHED_FEAT(NUMA_RESIST_LOWER, false)#endif
Mel Gorman
reviewerPATCH 24/63 sched: Reschedule task on preferred NUMA node once selected
In reply to Mel GormanA preferred node is selected based on the node the most NUMA hinting faults was incurred on. There is no guarantee that the task is running on that node at the time so this patch rescheules the task to run on the most idle CPU of the selected node when selected. This avoids waiting for the balancer to make a decision.
--- kernel/sched/core.c | 19 +++++++++++++++++++ kernel/sched/fair.c | 46 +++++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 1 + 3 files changed, 65 insertions(+), 1 deletion(-)
kernel/sched/core.c +19 -0
@@ -4353,6 +4353,25 @@ fail:return ret; } +#ifdef CONFIG_NUMA_BALANCING +/* Migrate current task p to target_cpu */ +int migrate_task_to (struct task_struct *p,int target_cpu) +{ +struct migration_arg arg = { p, target_cpu }; +int curr_cpu =task_cpu (p); + +if (curr_cpu == target_cpu) +return 0 ; + +if (!cpumask_test_cpu (target_cpu,tsk_cpus_allowed (p))) +return -EINVAL ; + +/* TODO: This is not properly updating schedstats */ + +return stop_one_cpu (curr_cpu, migration_cpu_stop, &arg); +} +#endif +/* * migration_cpu_stop - this will be executed by a highprio stopper thread * and performs thread migration by bumping thread off CPU then
kernel/sched/fair.c +45 -1
61 lines not on this page · raw message
kernel/sched/sched.h +1 -0
@@ -554,6 +554,7 @@ static inline u64 rq_clock_task(struct rq *rq) }#ifdef CONFIG_NUMA_BALANCING +extern int migrate_task_to (struct task_struct *p,int cpu);static inline void task_numa_free (struct task_struct *p) {kfree (p->numa_faults);
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 24/63] sched: Reschedule task on preferred NUMA node once selected
Quoted 7 lines · Mel Gorman
A preferred node is selected based on the node the most NUMA hinting faults was incurred on. There is no guarantee that the task is running on that node at the time so this patch rescheules the task to run on the most idle CPU of the selected node when selected. This avoids waiting for the balancer to make a decision.
Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Reschedule task on preferred NUMA node once selected
In reply to Mel Gormansched/numa: Reschedule task on preferred NUMA node once selected
Commit-ID: e6628d5b0a2979f3e0ee6f7783ede5df50cb9ede Gitweb: http://git.kernel.org/tip/e6628d5b0a2979f3e0ee6f7783ede5df50cb9ede Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:02 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:28 +0200
sched/numa: Reschedule task on preferred NUMA node once selected
A preferred node is selected based on the node the most NUMA hinting faults was incurred on. There is no guarantee that the task is running on that node at the time so this patch rescheules the task to run on the most idle CPU of the selected node when selected. This avoids waiting for the balancer to make a decision.
--- kernel/sched/core.c | 19 +++++++++++++++++++ kernel/sched/fair.c | 46 +++++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 1 + 3 files changed, 65 insertions(+), 1 deletion(-)
kernel/sched/core.c +19 -0
@@ -4348,6 +4348,25 @@ fail:return ret; } +#ifdef CONFIG_NUMA_BALANCING +/* Migrate current task p to target_cpu */ +int migrate_task_to (struct task_struct *p,int target_cpu) +{ +struct migration_arg arg = { p, target_cpu }; +int curr_cpu =task_cpu (p); + +if (curr_cpu == target_cpu) +return 0 ; + +if (!cpumask_test_cpu (target_cpu,tsk_cpus_allowed (p))) +return -EINVAL ; + +/* TODO: This is not properly updating schedstats */ + +return stop_one_cpu (curr_cpu, migration_cpu_stop, &arg); +} +#endif +/* * migration_cpu_stop - this will be executed by a highprio stopper thread * and performs thread migration by bumping thread off CPU then
kernel/sched/fair.c +45 -1
61 lines not on this page · raw message
kernel/sched/sched.h +1 -0
@@ -557,6 +557,7 @@ static inline u64 rq_clock_task(struct rq *rq) }#ifdef CONFIG_NUMA_BALANCING +extern int migrate_task_to (struct task_struct *p,int cpu);static inline void task_numa_free (struct task_struct *p) {kfree (p->numa_faults);
Mel Gorman
reviewerPATCH 25/63 sched: Add infrastructure for split shared/private accounting of NUMA hinting faults
In reply to Mel GormanIdeally it would be possible to distinguish between NUMA hinting faults that are private to a task and those that are shared. This patch prepares infrastructure for separately accounting shared and private faults by allocating the necessary buffers and passing in relevant information. For now, all faults are treated as private and detection will be introduced later.
--- include/linux/sched.h | 5 +++-- kernel/sched/fair.c | 46 +++++++++++++++++++++++++++++++++++----------- mm/huge_memory.c | 5 +++-- mm/memory.c | 8 ++++++-- 4 files changed, 47 insertions(+), 17 deletions(-)
include/linux/sched.h +3 -2
@@ -1435,10 +1435,11 @@ struct task_struct {#define tsk_cpus_allowed (tsk) (&(tsk)->cpus_allowed)#ifdef CONFIG_NUMA_BALANCING -extern void task_numa_fault (int node,int pages,bool migrated); +extern void task_numa_fault (int last_node,int node,int pages,bool migrated);extern void set_numabalancing_state (bool enabled);#else -static inline void task_numa_fault (int node,int pages,bool migrated) +static inline void task_numa_fault (int last_node,int node,int pages, +bool migrated) { }static inline void set_numabalancing_state (bool enabled)
kernel/sched/fair.c +35 -11
98 lines not on this page · raw message
mm/huge_memory.c +3 -2
@@ -1282,7 +1282,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct page *page;unsigned long haddr = addr &HPAGE_PMD_MASK ;int page_nid =-1 , this_nid =numa_node_id (); -int target_nid; +int target_nid, last_nid =-1 ;bool page_locked;bool migrated = false;
@@ -1293,6 +1293,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, page =pmd_page (pmd);BUG_ON (is_huge_zero_page (page)); page_nid =page_to_nid (page); + last_nid =page_nid_last (page);count_vm_numa_event (NUMA_HINT_FAULTS );if (page_nid == this_nid)count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL );
@@ -1361,7 +1362,7 @@ out:page_unlock_anon_vma_read (anon_vma);if (page_nid !=-1 ) -task_numa_fault (page_nid,HPAGE_PMD_NR , migrated); +task_numa_fault (last_nid, page_nid,HPAGE_PMD_NR , migrated);return 0 ; }
mm/memory.c +6 -2
@@ -3536,6 +3536,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct page *page =NULL ;spinlock_t *ptl;int page_nid =-1 ; +int last_nid;int target_nid;bool migrated = false;
@@ -3566,6 +3567,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }BUG_ON (is_zero_pfn (page_to_pfn (page))); + last_nid =page_nid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (ptep, ptl);
@@ -3581,7 +3583,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, out:if (page_nid !=-1 ) -task_numa_fault (page_nid,1 , migrated); +task_numa_fault (last_nid, page_nid,1 , migrated);return 0 ; }
@@ -3596,6 +3598,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,unsigned long offset;spinlock_t *ptl;bool numa = false; +int last_nid;spin_lock (&mm->page_table_lock); pmd = *pmdp;
@@ -3643,6 +3646,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,if (unlikely (page_mapcount (page) !=1 ))continue ; + last_nid =page_nid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (pte, ptl);
@@ -3655,7 +3659,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }if (page_nid !=-1 ) -task_numa_fault (page_nid,1 , migrated); +task_numa_fault (last_nid, page_nid,1 , migrated); pte =pte_offset_map_lock (mm, pmdp, addr, &ptl); }
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 25/63] sched: Add infrastructure for split shared/private accounting of NUMA hinting faults
Quoted 8 lines · Mel Gorman
Ideally it would be possible to distinguish between NUMA hinting faults that are private to a task and those that are shared. This patch prepares infrastructure for separately accounting shared and private faults by allocating the necessary buffers and passing in relevant information. For now, all faults are treated as private and detection will be introduced later.
Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Add infrastructure for split shared/ private accounting of NUMA hinting faults
In reply to Mel Gormansched/numa: Add infrastructure for split shared/private accounting of NUMA hinting faults
Commit-ID: ac8e895bd260cb8bb19ade6a3abd44e7abe9a01d Gitweb: http://git.kernel.org/tip/ac8e895bd260cb8bb19ade6a3abd44e7abe9a01d Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:03 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:30 +0200
sched/numa: Add infrastructure for split shared/private accounting of NUMA hinting faults
Ideally it would be possible to distinguish between NUMA hinting faults that are private to a task and those that are shared. This patch prepares infrastructure for separately accounting shared and private faults by allocating the necessary buffers and passing in relevant information. For now, all faults are treated as private and detection will be introduced later.
--- include/linux/sched.h | 5 +++-- kernel/sched/fair.c | 46 +++++++++++++++++++++++++++++++++++----------- mm/huge_memory.c | 5 +++-- mm/memory.c | 8 ++++++-- 4 files changed, 47 insertions(+), 17 deletions(-)
include/linux/sched.h +3 -2
@@ -1445,10 +1445,11 @@ struct task_struct {#define tsk_cpus_allowed (tsk) (&(tsk)->cpus_allowed)#ifdef CONFIG_NUMA_BALANCING -extern void task_numa_fault (int node,int pages,bool migrated); +extern void task_numa_fault (int last_node,int node,int pages,bool migrated);extern void set_numabalancing_state (bool enabled);#else -static inline void task_numa_fault (int node,int pages,bool migrated) +static inline void task_numa_fault (int last_node,int node,int pages, +bool migrated) { }static inline void set_numabalancing_state (bool enabled)
kernel/sched/fair.c +35 -11
98 lines not on this page · raw message
mm/huge_memory.c +3 -2
@@ -1282,7 +1282,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct page *page;unsigned long haddr = addr &HPAGE_PMD_MASK ;int page_nid =-1 , this_nid =numa_node_id (); -int target_nid; +int target_nid, last_nid =-1 ;bool page_locked;bool migrated = false;
@@ -1293,6 +1293,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, page =pmd_page (pmd);BUG_ON (is_huge_zero_page (page)); page_nid =page_to_nid (page); + last_nid =page_nid_last (page);count_vm_numa_event (NUMA_HINT_FAULTS );if (page_nid == this_nid)count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL );
@@ -1361,7 +1362,7 @@ out:page_unlock_anon_vma_read (anon_vma);if (page_nid !=-1 ) -task_numa_fault (page_nid,HPAGE_PMD_NR , migrated); +task_numa_fault (last_nid, page_nid,HPAGE_PMD_NR , migrated);return 0 ; }
mm/memory.c +6 -2
@@ -3536,6 +3536,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct page *page =NULL ;spinlock_t *ptl;int page_nid =-1 ; +int last_nid;int target_nid;bool migrated = false;
@@ -3566,6 +3567,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }BUG_ON (is_zero_pfn (page_to_pfn (page))); + last_nid =page_nid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (ptep, ptl);
@@ -3581,7 +3583,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, out:if (page_nid !=-1 ) -task_numa_fault (page_nid,1 , migrated); +task_numa_fault (last_nid, page_nid,1 , migrated);return 0 ; }
@@ -3596,6 +3598,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,unsigned long offset;spinlock_t *ptl;bool numa = false; +int last_nid;spin_lock (&mm->page_table_lock); pmd = *pmdp;
@@ -3643,6 +3646,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,if (unlikely (page_mapcount (page) !=1 ))continue ; + last_nid =page_nid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (pte, ptl);
@@ -3655,7 +3659,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }if (page_nid !=-1 ) -task_numa_fault (page_nid,1 , migrated); +task_numa_fault (last_nid, page_nid,1 , migrated); pte =pte_offset_map_lock (mm, pmdp, addr, &ptl); }
Mel Gorman
reviewerPATCH 26/63 sched: Check current->mm before allocating NUMA faults
In reply to Mel Gormantask_numa_placement checks current->mm but after buffers for faults have already been uselessly allocated. Move the check earlier.
--- kernel/sched/fair.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/sched/fair.c +4 -2
@@ -930,8 +930,6 @@ static void task_numa_placement(struct task_struct *p)int seq, nid, max_nid =-1 ;unsigned long max_faults =0 ; -if (!p->mm)/* for example, ksmd faulting in a user's mm */ -return ; seq =ACCESS_ONCE (p->mm->numa_scan_seq);if (p->numa_scan_seq == seq)return ;
@@ -998,6 +996,10 @@ void task_numa_fault(int last_nid, int node, int pages, bool migrated)if (!numabalancing_enabled)return ; +/* for example, ksmd faulting in a user's mm */ +if (!p->mm) +return ; +/* For now, do not attempt to detect private/shared accesses */ priv =1 ;
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 26/63] sched: Check current->mm before allocating NUMA faults
Quoted 5 lines · Mel Gorman
task_numa_placementcheckscurrent->mmbut after buffers for faults have already been uselessly allocated. Move the check earlier.[[email protected]: Identified the problem]Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Check current-> mm before allocating NUMA faults
In reply to Mel Gormansched/numa: Check current->mm before allocating NUMA faults
Commit-ID: 9ff1d9ff3c2c8ab3feaeb2e8056a07ca293f7bde Gitweb: http://git.kernel.org/tip/9ff1d9ff3c2c8ab3feaeb2e8056a07ca293f7bde Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:04 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:31 +0200
sched/numa: Check current->mm before allocating NUMA faults
task_numa_placement checks current->mm but after buffers for faults have already been uselessly allocated. Move the check earlier.
[[email protected]: Identified the problem]
--- kernel/sched/fair.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/sched/fair.c +4 -2
@@ -930,8 +930,6 @@ static void task_numa_placement(struct task_struct *p)int seq, nid, max_nid =-1 ;unsigned long max_faults =0 ; -if (!p->mm)/* for example, ksmd faulting in a user's mm */ -return ; seq =ACCESS_ONCE (p->mm->numa_scan_seq);if (p->numa_scan_seq == seq)return ;
@@ -998,6 +996,10 @@ void task_numa_fault(int last_nid, int node, int pages, bool migrated)if (!numabalancing_enabled)return ; +/* for example, ksmd faulting in a user's mm */ +if (!p->mm) +return ; +/* For now, do not attempt to detect private/shared accesses */ priv =1 ;
Mel Gorman
reviewerPATCH 27/63 mm: numa: Scan pages with elevated page_mapcount
In reply to Mel GormanCurrently automatic NUMA balancing is unable to distinguish between false shared versus private pages except by ignoring pages with an elevated page_mapcount entirely. This avoids shared pages bouncing between the nodes whose task is using them but that is ignored quite a lot of data.
This patch kicks away the training wheels in preparation for adding support for identifying shared/private pages is now in place. The ordering is so that the impact of the shared/private detection can be easily measured. Note that the patch does not migrate shared, file-backed within vmas marked VM_EXEC as these are generally shared library pages. Migrating such pages is not beneficial as there is an expectation they are read-shared between caches and iTLB and iCache pressure is generally low.
--- include/linux/migrate.h | 7 ++++--- mm/huge_memory.c | 12 +++++------- mm/memory.c | 7 ++----- mm/migrate.c | 17 ++++++----------- mm/mprotect.c | 4 +--- 5 files changed, 18 insertions(+), 29 deletions(-)
include/linux/migrate.h +4 -3
@@ -90,11 +90,12 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,#endif /* CONFIG_MIGRATION */ #ifdef CONFIG_NUMA_BALANCING -extern int migrate_misplaced_page (struct page *page,int node); -extern int migrate_misplaced_page (struct page *page,int node); +extern int migrate_misplaced_page (struct page *page, +struct vm_area_struct *vma,int node);extern bool migrate_ratelimited (int node);#else -static inline int migrate_misplaced_page (struct page *page,int node) +static inline int migrate_misplaced_page (struct page *page, +struct vm_area_struct *vma,int node) {return -EAGAIN ;/* can't migrate now */ }
mm/huge_memory.c +5 -7
@@ -1484,14 +1484,12 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,struct page *page =pmd_page (*pmd);/* -* Only check non-shared pages. Do not trap faults -* against the zero page. The read-only data is likely -* to be read-cached on the local CPU cache and it is -* less useful to know about local vs remote hits on -* the zero page. +* Do not trap faults against the zero page. The +* read-only data is likely to be read-cached on the +* local CPU cache and it is less useful to know about +* local vs remote hits on the zero page. */ -if (page_mapcount (page) ==1 && - !is_huge_zero_page (page) && +if (!is_huge_zero_page (page) && !pmd_numa (*pmd)) { entry =pmdp_get_and_clear (mm, addr, pmd); entry =pmd_mknuma (entry);
mm/memory.c +2 -5
@@ -3577,7 +3577,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }/* Migrate to the requested node */ - migrated =migrate_misplaced_page (page, target_nid); + migrated =migrate_misplaced_page (page, vma, target_nid);if (migrated) page_nid = target_nid;
@@ -3642,16 +3642,13 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, page =vm_normal_page (vma, addr, pteval);if (unlikely (!page))continue ; -/* only check non-shared pages */ -if (unlikely (page_mapcount (page) !=1 )) -continue ; last_nid =page_nid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (pte, ptl);if (target_nid !=-1 ) { - migrated =migrate_misplaced_page (page, target_nid); + migrated =migrate_misplaced_page (page, vma, target_nid);if (migrated) page_nid = target_nid; }else {
mm/migrate.c +6 -11
@@ -1599,7 +1599,8 @@ int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page)* node. Caller is expected to have an elevated reference count on * the page that will be dropped by this function before returning. */ -int migrate_misplaced_page (struct page *page,int node) +int migrate_misplaced_page (struct page *page,struct vm_area_struct *vma, +int node) {pg_data_t *pgdat =NODE_DATA (node);int isolated;
@@ -1607,10 +1608,11 @@ int migrate_misplaced_page(struct page *page, int node)LIST_HEAD (migratepages);/* -* Don't migrate pages that are mapped in multiple processes. -* TODO: Handle false sharing detection instead of this hammer +* Don't migrate file pages that are mapped in multiple processes +* with execute permissions as they are probably shared libraries. */ -if (page_mapcount (page) !=1 ) +if (page_mapcount (page) !=1 &&page_is_file_cache (page) && + (vma->vm_flags &VM_EXEC )) goto out;/*
@@ -1661,13 +1663,6 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,int page_lru =page_is_file_cache (page);/* -* Don't migrate pages that are mapped in multiple processes. -* TODO: Handle false sharing detection instead of this hammer -*/ -if (page_mapcount (page) !=1 ) - goto out_dropref; - -/* * Rate-limit the amount of data that is being migrated to a node. * Optimal placement is no good if the memory bus is saturated and * all the time is being spent migrating!
mm/mprotect.c +1 -3
@@ -69,9 +69,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,if (last_nid != this_nid) all_same_node = false; -/* only check non-shared pages */ -if (!pte_numa (oldpte) && -page_mapcount (page) ==1 ) { +if (!pte_numa (oldpte)) { ptent =pte_mknuma (ptent); updated = true; }
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 27/63] mm: numa: Scan pages with elevated page_mapcount
Quoted 13 lines · Mel Gorman
Currently automatic NUMA balancing is unable to distinguish between false shared versus private pages except by ignoring pages with an elevated
page_mapcountentirely. This avoids shared pages bouncing between the nodes whose task is using them but that is ignored quite a lot of data.This patch kicks away the training wheels in preparation for adding support for identifying shared/private pages is now in place. The ordering is so that the impact of the shared/private detection can be easily measured. Note that the patch does not migrate shared, file-backed within vmas marked
VM_EXECas these are generally shared library pages. Migrating such pages is not beneficial as there is an expectation they are read-shared between caches and iTLB and iCache pressure is generally low.
Quoted 1 line
Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] mm: numa: Scan pages with elevated page_mapcount
In reply to Mel Gormanmm: numa: Scan pages with elevated page_mapcount
Commit-ID: 1bc115d87dffd1c43bdc3c9c9d1e3a51c195d18e Gitweb: http://git.kernel.org/tip/1bc115d87dffd1c43bdc3c9c9d1e3a51c195d18e Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:05 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:32 +0200
mm: numa: Scan pages with elevated page_mapcount
Currently automatic NUMA balancing is unable to distinguish between false shared versus private pages except by ignoring pages with an elevated page_mapcount entirely. This avoids shared pages bouncing between the nodes whose task is using them but that is ignored quite a lot of data.
This patch kicks away the training wheels in preparation for adding support for identifying shared/private pages is now in place. The ordering is so that the impact of the shared/private detection can be easily measured. Note that the patch does not migrate shared, file-backed within vmas marked VM_EXEC as these are generally shared library pages. Migrating such pages is not beneficial as there is an expectation they are read-shared between caches and iTLB and iCache pressure is generally low.
--- include/linux/migrate.h | 7 ++++--- mm/huge_memory.c | 12 +++++------- mm/memory.c | 7 ++----- mm/migrate.c | 17 ++++++----------- mm/mprotect.c | 4 +--- 5 files changed, 18 insertions(+), 29 deletions(-)
include/linux/migrate.h +4 -3
@@ -90,11 +90,12 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,#endif /* CONFIG_MIGRATION */ #ifdef CONFIG_NUMA_BALANCING -extern int migrate_misplaced_page (struct page *page,int node); -extern int migrate_misplaced_page (struct page *page,int node); +extern int migrate_misplaced_page (struct page *page, +struct vm_area_struct *vma,int node);extern bool migrate_ratelimited (int node);#else -static inline int migrate_misplaced_page (struct page *page,int node) +static inline int migrate_misplaced_page (struct page *page, +struct vm_area_struct *vma,int node) {return -EAGAIN ;/* can't migrate now */ }
mm/huge_memory.c +5 -7
@@ -1484,14 +1484,12 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,struct page *page =pmd_page (*pmd);/* -* Only check non-shared pages. Do not trap faults -* against the zero page. The read-only data is likely -* to be read-cached on the local CPU cache and it is -* less useful to know about local vs remote hits on -* the zero page. +* Do not trap faults against the zero page. The +* read-only data is likely to be read-cached on the +* local CPU cache and it is less useful to know about +* local vs remote hits on the zero page. */ -if (page_mapcount (page) ==1 && - !is_huge_zero_page (page) && +if (!is_huge_zero_page (page) && !pmd_numa (*pmd)) { entry =pmdp_get_and_clear (mm, addr, pmd); entry =pmd_mknuma (entry);
mm/memory.c +2 -5
@@ -3577,7 +3577,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }/* Migrate to the requested node */ - migrated =migrate_misplaced_page (page, target_nid); + migrated =migrate_misplaced_page (page, vma, target_nid);if (migrated) page_nid = target_nid;
@@ -3642,16 +3642,13 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, page =vm_normal_page (vma, addr, pteval);if (unlikely (!page))continue ; -/* only check non-shared pages */ -if (unlikely (page_mapcount (page) !=1 )) -continue ; last_nid =page_nid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (pte, ptl);if (target_nid !=-1 ) { - migrated =migrate_misplaced_page (page, target_nid); + migrated =migrate_misplaced_page (page, vma, target_nid);if (migrated) page_nid = target_nid; }else {
mm/migrate.c +6 -11
@@ -1599,7 +1599,8 @@ int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page)* node. Caller is expected to have an elevated reference count on * the page that will be dropped by this function before returning. */ -int migrate_misplaced_page (struct page *page,int node) +int migrate_misplaced_page (struct page *page,struct vm_area_struct *vma, +int node) {pg_data_t *pgdat =NODE_DATA (node);int isolated;
@@ -1607,10 +1608,11 @@ int migrate_misplaced_page(struct page *page, int node)LIST_HEAD (migratepages);/* -* Don't migrate pages that are mapped in multiple processes. -* TODO: Handle false sharing detection instead of this hammer +* Don't migrate file pages that are mapped in multiple processes +* with execute permissions as they are probably shared libraries. */ -if (page_mapcount (page) !=1 ) +if (page_mapcount (page) !=1 &&page_is_file_cache (page) && + (vma->vm_flags &VM_EXEC )) goto out;/*
@@ -1661,13 +1663,6 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,int page_lru =page_is_file_cache (page);/* -* Don't migrate pages that are mapped in multiple processes. -* TODO: Handle false sharing detection instead of this hammer -*/ -if (page_mapcount (page) !=1 ) - goto out_dropref; - -/* * Rate-limit the amount of data that is being migrated to a node. * Optimal placement is no good if the memory bus is saturated and * all the time is being spent migrating!
mm/mprotect.c +1 -3
@@ -69,9 +69,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,if (last_nid != this_nid) all_same_node = false; -/* only check non-shared pages */ -if (!pte_numa (oldpte) && -page_mapcount (page) ==1 ) { +if (!pte_numa (oldpte)) { ptent =pte_mknuma (ptent); updated = true; }
Mel Gorman
reviewerPATCH 28/63 sched: Remove check that skips small VMAs
In reply to Mel Gormantask_numa_work skips small VMAs. At the time the logic was to reduce the scanning overhead which was considerable. It is a dubious hack at best. It would make much more sense to cache where faults have been observed and only rescan those regions during subsequent PTE scans. Remove this hack as motivation to do it properly in the future.
--- kernel/sched/fair.c | 4 ---- 1 file changed, 4 deletions(-)
kernel/sched/fair.c +0 -4
@@ -1127,10 +1127,6 @@ void task_numa_work(struct callback_head *work)if (!vma_migratable (vma))continue ; -/* Skip small VMAs. They are not likely to be of relevance */ -if (vma->vm_end - vma->vm_start <HPAGE_SIZE ) -continue ; - do { start = max(start, vma->vm_start); end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 28/63] sched: Remove check that skips small VMAs
Quoted 7 lines · Mel Gorman
task_numa_workskips small VMAs. At the time the logic was to reduce the scanning overhead which was considerable. It is a dubious hack at best. It would make much more sense to cache where faults have been observed and only rescan those regions during subsequent PTE scans. Remove this hack as motivation to do it properly in the future.Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Remove check that skips small VMAs
In reply to Mel Gormansched/numa: Remove check that skips small VMAs
Commit-ID: 073b5beea735c7e1970686c94ff1f3aaac790a2a Gitweb: http://git.kernel.org/tip/073b5beea735c7e1970686c94ff1f3aaac790a2a Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:06 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:33 +0200
sched/numa: Remove check that skips small VMAs
task_numa_work skips small VMAs. At the time the logic was to reduce the scanning overhead which was considerable. It is a dubious hack at best. It would make much more sense to cache where faults have been observed and only rescan those regions during subsequent PTE scans. Remove this hack as motivation to do it properly in the future.
--- kernel/sched/fair.c | 4 ---- 1 file changed, 4 deletions(-)
kernel/sched/fair.c +0 -4
@@ -1127,10 +1127,6 @@ void task_numa_work(struct callback_head *work)if (!vma_migratable (vma))continue ; -/* Skip small VMAs. They are not likely to be of relevance */ -if (vma->vm_end - vma->vm_start <HPAGE_SIZE ) -continue ; - do { start = max(start, vma->vm_start); end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);
Mel Gorman
reviewerPATCH 29/63 sched: Set preferred NUMA node based on number of private faults
In reply to Mel GormanIdeally it would be possible to distinguish between NUMA hinting faults that are private to a task and those that are shared. If treated identically there is a risk that shared pages bounce between nodes depending on the order they are referenced by tasks. Ultimately what is desirable is that task private pages remain local to the task while shared pages are interleaved between sharing tasks running on different nodes to give good average performance. This is further complicated by THP as even applications that partition their data may not be partitioning on a huge page boundary.
To start with, this patch assumes that multi-threaded or multi-process applications partition their data and that in general the private accesses are more important for cpu->memory locality in the general case. Also, no new infrastructure is required to treat private pages properly but interleaving for shared pages requires additional infrastructure.
To detect private accesses the pid of the last accessing task is required but the storage requirements are a high. This patch borrows heavily from Ingo Molnar's patch "numa, mm, sched: Implement last-CPU+PID hash tracking" to encode some bits from the last accessing task in the page flags as well as the node information. Collisions will occur but it is better than just depending on the node information. Node information is then used to determine if a page needs to migrate. The PID information is used to detect private/shared accesses. The preferred NUMA node is selected based on where the maximum number of approximately private faults were measured. Shared faults are not taken into consideration for a few reasons.
First, if there are many tasks sharing the page then they'll all move towards the same node. The node will be compute overloaded and then scheduled away later only to bounce back again. Alternatively the shared tasks would just bounce around nodes because the fault information is effectively noise. Either way accounting for shared faults the same as private faults can result in lower performance overall.
The second reason is based on a hypothetical workload that has a small number of very important, heavily accessed private pages but a large shared array. The shared array would dominate the number of faults and be selected as a preferred node even though it's the wrong decision.
The third reason is that multiple threads in a process will race each other to fault the shared page making the fault information unreliable.
--- include/linux/mm.h | 89 +++++++++++++++++++++++++++++---------- include/linux/mm_types.h | 4 +- include/linux/page-flags-layout.h | 28 +++++++----- kernel/sched/fair.c | 12 ++++-- mm/huge_memory.c | 8 ++-- mm/memory.c | 16 +++---- mm/mempolicy.c | 8 ++-- mm/migrate.c | 4 +- mm/mm_init.c | 18 ++++---- mm/mmzone.c | 14 +++--- mm/mprotect.c | 26 ++++++++---- mm/page_alloc.c | 4 +- 12 files changed, 149 insertions(+), 82 deletions(-)
include/linux/mm.h +67 -22
@@ -581,11 +581,11 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)* sets it, so none of the operations on it need to be atomic. */ -/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_NID] | ... | FLAGS | */ +/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_NIDPID] | ... | FLAGS | */ #define SECTIONS_PGOFF ((sizeof(unsigned long)*8) - SECTIONS_WIDTH)#define NODES_PGOFF (SECTIONS_PGOFF - NODES_WIDTH)#define ZONES_PGOFF (NODES_PGOFF - ZONES_WIDTH) -#define LAST_NID_PGOFF (ZONES_PGOFF - LAST_NID_WIDTH) +#define LAST_NIDPID_PGOFF (ZONES_PGOFF - LAST_NIDPID_WIDTH)/* * Define the bit shifts to access each section. For non-existent
@@ -595,7 +595,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)#define SECTIONS_PGSHIFT (SECTIONS_PGOFF * (SECTIONS_WIDTH != 0))#define NODES_PGSHIFT (NODES_PGOFF * (NODES_WIDTH != 0))#define ZONES_PGSHIFT (ZONES_PGOFF * (ZONES_WIDTH != 0)) -#define LAST_NID_PGSHIFT (LAST_NID_PGOFF * (LAST_NID_WIDTH != 0)) +#define LAST_NIDPID_PGSHIFT (LAST_NIDPID_PGOFF * (LAST_NIDPID_WIDTH != 0))/* NODE:ZONE or SECTION:ZONE is used to ID a zone for the buddy allocator */ #ifdef NODE_NOT_IN_PAGE_FLAGS
@@ -617,7 +617,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)#define ZONES_MASK ((1UL << ZONES_WIDTH) - 1)#define NODES_MASK ((1UL << NODES_WIDTH) - 1)#define SECTIONS_MASK ((1UL << SECTIONS_WIDTH) - 1) -#define LAST_NID_MASK ((1UL << LAST_NID_WIDTH) - 1) +#define LAST_NIDPID_MASK ((1UL << LAST_NIDPID_WIDTH) - 1)#define ZONEID_MASK ((1UL << ZONEID_SHIFT) - 1)static inline enum zone_type page_zonenum (const struct page *page)
@@ -661,48 +661,93 @@ static inline int page_to_nid(const struct page *page)#endif #ifdef CONFIG_NUMA_BALANCING -#ifdef LAST_NID_NOT_IN_PAGE_FLAGS -static inline int page_nid_xchg_last (struct page *page,int nid) +static inline int nid_pid_to_nidpid (int nid,int pid) { -return xchg (&page->_last_nid, nid); +return ((nid &LAST__NID_MASK ) <<LAST__PID_SHIFT ) | (pid &LAST__PID_MASK ); } -static inline int page_nid_last (struct page *page) +static inline int nidpid_to_pid (int nidpid) { -return page->_last_nid; +return nidpid &LAST__PID_MASK ; } -static inline void page_nid_reset_last (struct page *page) + +static inline int nidpid_to_nid (int nidpid) +{ +return (nidpid >>LAST__PID_SHIFT ) &LAST__NID_MASK ; +} + +static inline bool nidpid_pid_unset (int nidpid) +{ +return nidpid_to_pid (nidpid) == (-1 &LAST__PID_MASK ); +} + +static inline bool nidpid_nid_unset (int nidpid) { - page->_last_nid =-1 ; +return nidpid_to_nid (nidpid) == (-1 &LAST__NID_MASK ); +} + +#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS +static inline int page_nidpid_xchg_last (struct page *page,int nid) +{ +return xchg (&page->_last_nidpid, nid); +} + +static inline int page_nidpid_last (struct page *page) +{ +return page->_last_nidpid; +} +static inline void page_nidpid_reset_last (struct page *page) +{ + page->_last_nidpid =-1 ; }#else -static inline int page_nid_last (struct page *page) +static inline int page_nidpid_last (struct page *page) { -return (page->flags >>LAST_NID_PGSHIFT ) &LAST_NID_MASK ; +return (page->flags >>LAST_NIDPID_PGSHIFT ) &LAST_NIDPID_MASK ; } -extern int page_nid_xchg_last (struct page *page,int nid); +extern int page_nidpid_xchg_last (struct page *page,int nidpid); -static inline void page_nid_reset_last (struct page *page) +static inline void page_nidpid_reset_last (struct page *page) { -int nid = (1 <<LAST_NID_SHIFT ) -1 ; +int nidpid = (1 <<LAST_NIDPID_SHIFT ) -1 ; - page->flags &= ~(LAST_NID_MASK <<LAST_NID_PGSHIFT ); - page->flags |= (nid &LAST_NID_MASK ) <<LAST_NID_PGSHIFT ; + page->flags &= ~(LAST_NIDPID_MASK <<LAST_NIDPID_PGSHIFT ); + page->flags |= (nidpid &LAST_NIDPID_MASK ) <<LAST_NIDPID_PGSHIFT ; } -#endif /* LAST_NID_NOT_IN_PAGE_FLAGS */ +#endif /* LAST_NIDPID_NOT_IN_PAGE_FLAGS */ #else -static inline int page_nid_xchg_last (struct page *page,int nid) +static inline int page_nidpid_xchg_last (struct page *page,int nidpid) {return page_to_nid (page); } -static inline int page_nid_last (struct page *page) +static inline int page_nidpid_last (struct page *page) {return page_to_nid (page); } -static inline void page_nid_reset_last (struct page *page) +static inline int nidpid_to_nid (int nidpid) +{ +return -1 ; +} + +static inline int nidpid_to_pid (int nidpid) +{ +return -1 ; +} + +static inline int nid_pid_to_nidpid (int nid,int pid) +{ +return -1 ; +} + +static inline bool nidpid_pid_unset (int nidpid) +{ +return 1 ; +} + +static inline void page_nidpid_reset_last (struct page *page) { }#endif
include/linux/mm_types.h +2 -2
@@ -174,8 +174,8 @@ struct page {void *shadow;#endif -#ifdef LAST_NID_NOT_IN_PAGE_FLAGS -int _last_nid; +#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS +int _last_nidpid;#endif }/*
include/linux/page-flags-layout.h +17 -11
@@ -38,10 +38,10 @@* The last is when there is insufficient space in page->flags and a separate * lookup is necessary. * -* No sparsemem or sparsemem vmemmap: | NODE | ZONE | ... | FLAGS | -* " plus space for last_nid: | NODE | ZONE | LAST_NID ... | FLAGS | -* classic sparse with space for node:| SECTION | NODE | ZONE | ... | FLAGS | -* " plus space for last_nid: | SECTION | NODE | ZONE | LAST_NID ... | FLAGS | +* No sparsemem or sparsemem vmemmap: | NODE | ZONE | ... | FLAGS | +* " plus space for last_nidpid: | NODE | ZONE | LAST_NIDPID ... | FLAGS | +* classic sparse with space for node:| SECTION | NODE | ZONE | ... | FLAGS | +* " plus space for last_nidpid: | SECTION | NODE | ZONE | LAST_NIDPID ... | FLAGS | * classic sparse no space for node: | SECTION | ZONE | ... | FLAGS | */ #if defined(CONFIG_SPARSEMEM ) && !defined(CONFIG_SPARSEMEM_VMEMMAP )
@@ -62,15 +62,21 @@#endif #ifdef CONFIG_NUMA_BALANCING -#define LAST_NID_SHIFT NODES_SHIFT +#define LAST__PID_SHIFT 8 +#define LAST__PID_MASK ((1 << LAST__PID_SHIFT)-1) + +#define LAST__NID_SHIFT NODES_SHIFT +#define LAST__NID_MASK ((1 << LAST__NID_SHIFT)-1) + +#define LAST_NIDPID_SHIFT (LAST__PID_SHIFT+LAST__NID_SHIFT)#else -#define LAST_NID_SHIFT 0 +#define LAST_NIDPID_SHIFT 0#endif -#if SECTIONS_WIDTH +ZONES_WIDTH +NODES_SHIFT +LAST_NID_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS -#define LAST_NID_WIDTH LAST_NID_SHIFT +#if SECTIONS_WIDTH +ZONES_WIDTH +NODES_SHIFT +LAST_NIDPID_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS +#define LAST_NIDPID_WIDTH LAST_NIDPID_SHIFT#else -#define LAST_NID_WIDTH 0 +#define LAST_NIDPID_WIDTH 0#endif /*
@@ -81,8 +87,8 @@#define NODE_NOT_IN_PAGE_FLAGS #endif -#if defined(CONFIG_NUMA_BALANCING ) &&LAST_NID_WIDTH ==0 -#define LAST_NID_NOT_IN_PAGE_FLAGS +#if defined(CONFIG_NUMA_BALANCING ) &&LAST_NIDPID_WIDTH ==0 +#define LAST_NIDPID_NOT_IN_PAGE_FLAGS #endif #endif /* _LINUX_PAGE_FLAGS_LAYOUT */
kernel/sched/fair.c +9 -3
@@ -988,7 +988,7 @@ static void task_numa_placement(struct task_struct *p)/* * Got a PROT_NONE fault for a page on @node. */ -void task_numa_fault (int last_nid,int node,int pages,bool migrated) +void task_numa_fault (int last_nidpid,int node,int pages,bool migrated) {struct task_struct *p = current;int priv;
@@ -1000,8 +1000,14 @@ void task_numa_fault(int last_nid, int node, int pages, bool migrated)if (!p->mm)return ; -/* For now, do not attempt to detect private/shared accesses */ - priv =1 ; +/* +* First accesses are treated as private, otherwise consider accesses +* to be private if the accessing pid has not changed +*/ +if (!nidpid_pid_unset (last_nidpid)) + priv = ((p->pid &LAST__PID_MASK ) ==nidpid_to_pid (last_nidpid)); +else + priv =1 ;/* Allocate buffer to track faults on a per-node basis */ if (unlikely (!p->numa_faults)) {
mm/huge_memory.c +4 -4
@@ -1282,7 +1282,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct page *page;unsigned long haddr = addr &HPAGE_PMD_MASK ;int page_nid =-1 , this_nid =numa_node_id (); -int target_nid, last_nid =-1 ; +int target_nid, last_nidpid =-1 ;bool page_locked;bool migrated = false;
@@ -1293,7 +1293,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, page =pmd_page (pmd);BUG_ON (is_huge_zero_page (page)); page_nid =page_to_nid (page); - last_nid =page_nid_last (page); + last_nidpid =page_nidpid_last (page);count_vm_numa_event (NUMA_HINT_FAULTS );if (page_nid == this_nid)count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL );
@@ -1362,7 +1362,7 @@ out:page_unlock_anon_vma_read (anon_vma);if (page_nid !=-1 ) -task_numa_fault (last_nid, page_nid,HPAGE_PMD_NR , migrated); +task_numa_fault (last_nidpid, page_nid,HPAGE_PMD_NR , migrated);return 0 ; }
@@ -1682,7 +1682,7 @@ static void __split_huge_page_refcount(struct page *page, page_tail->mapping = page->mapping; page_tail->index = page->index + i; -page_nid_xchg_last (page_tail,page_nid_last (page)); +page_nidpid_xchg_last (page_tail,page_nidpid_last (page));BUG_ON (!PageAnon (page_tail));BUG_ON (!PageUptodate (page_tail));
mm/memory.c +8 -8
@@ -69,8 +69,8 @@#include "internal.h" -#ifdef LAST_NID_NOT_IN_PAGE_FLAGS -#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_nid. +#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS +#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_nidpid.#endif #ifndef CONFIG_NEED_MULTIPLE_NODES
@@ -3536,7 +3536,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct page *page =NULL ;spinlock_t *ptl;int page_nid =-1 ; -int last_nid; +int last_nidpid;int target_nid;bool migrated = false;
@@ -3567,7 +3567,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }BUG_ON (is_zero_pfn (page_to_pfn (page))); - last_nid =page_nid_last (page); + last_nidpid =page_nidpid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (ptep, ptl);
@@ -3583,7 +3583,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, out:if (page_nid !=-1 ) -task_numa_fault (last_nid, page_nid,1 , migrated); +task_numa_fault (last_nidpid, page_nid,1 , migrated);return 0 ; }
@@ -3598,7 +3598,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,unsigned long offset;spinlock_t *ptl;bool numa = false; -int last_nid; +int last_nidpid;spin_lock (&mm->page_table_lock); pmd = *pmdp;
@@ -3643,7 +3643,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,if (unlikely (!page))continue ; - last_nid =page_nid_last (page); + last_nidpid =page_nidpid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (pte, ptl);
@@ -3656,7 +3656,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }if (page_nid !=-1 ) -task_numa_fault (last_nid, page_nid,1 , migrated); +task_numa_fault (last_nidpid, page_nid,1 , migrated); pte =pte_offset_map_lock (mm, pmdp, addr, &ptl); }
mm/mempolicy.c +5 -3
@@ -2348,9 +2348,11 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long/* Migrate the page towards the node whose CPU is referencing it */ if (pol->flags &MPOL_F_MORON ) { -int last_nid; +int last_nidpid; +int this_nidpid; polnid =numa_node_id (); + this_nidpid =nid_pid_to_nidpid (polnid, current->pid);/* * Multi-stage node selection is used in conjunction
@@ -2373,8 +2375,8 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long* it less likely we act on an unlikely task<->page * relation. */ - last_nid =page_nid_xchg_last (page, polnid); -if (last_nid != polnid) + last_nidpid =page_nidpid_xchg_last (page, this_nidpid); +if (!nidpid_pid_unset (last_nidpid) &&nidpid_to_nid (last_nidpid) != polnid) goto out; }
mm/migrate.c +2 -2
@@ -1498,7 +1498,7 @@ static struct page *alloc_misplaced_dst_page(struct page *page, __GFP_NOWARN) & ~GFP_IOFS, 0);if (newpage) -page_nid_xchg_last (newpage,page_nid_last (page)); +page_nidpid_xchg_last (newpage,page_nidpid_last (page));return newpage; }
@@ -1675,7 +1675,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,if (!new_page) goto out_fail; -page_nid_xchg_last (new_page,page_nid_last (page)); +page_nidpid_xchg_last (new_page,page_nidpid_last (page)); isolated =numamigrate_isolate_page (pgdat, page);if (!isolated) {
mm/mm_init.c +9 -9
@@ -71,26 +71,26 @@ void __init mminit_verify_pageflags_layout(void)unsigned long or_mask, add_mask; shift =8 *sizeof (unsigned long ); - width = shift -SECTIONS_WIDTH -NODES_WIDTH -ZONES_WIDTH -LAST_NID_SHIFT ; + width = shift -SECTIONS_WIDTH -NODES_WIDTH -ZONES_WIDTH -LAST_NIDPID_SHIFT ;mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_widths" , -"Section %d Node %d Zone %d Lastnid %d Flags %d\n" , +"Section %d Node %d Zone %d Lastnidpid %d Flags %d\n" ,SECTIONS_WIDTH ,NODES_WIDTH ,ZONES_WIDTH , -LAST_NID_WIDTH , +LAST_NIDPID_WIDTH ,NR_PAGEFLAGS );mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_shifts" , -"Section %d Node %d Zone %d Lastnid %d\n" , +"Section %d Node %d Zone %d Lastnidpid %d\n" ,SECTIONS_SHIFT ,NODES_SHIFT ,ZONES_SHIFT , -LAST_NID_SHIFT ); +LAST_NIDPID_SHIFT );mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_pgshifts" , -"Section %lu Node %lu Zone %lu Lastnid %lu\n" , +"Section %lu Node %lu Zone %lu Lastnidpid %lu\n" , (unsigned long )SECTIONS_PGSHIFT , (unsigned long )NODES_PGSHIFT , (unsigned long )ZONES_PGSHIFT , - (unsigned long )LAST_NID_PGSHIFT ); + (unsigned long )LAST_NIDPID_PGSHIFT ); mminit_dprintk(MMINIT_TRACE, "pageflags_layout_nodezoneid", "Node/Zone ID: %lu -> %lu\n", (unsigned long)(ZONEID_PGOFF + ZONEID_SHIFT),
@@ -102,9 +102,9 @@ void __init mminit_verify_pageflags_layout(void)mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_nodeflags" ,"Node not in page flags" );#endif -#ifdef LAST_NID_NOT_IN_PAGE_FLAGS +#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_nodeflags" , -"Last nid not in page flags" ); +"Last nidpid not in page flags" );#endif if (SECTIONS_WIDTH ) {
mm/mmzone.c +7 -7
@@ -97,20 +97,20 @@ void lruvec_init(struct lruvec *lruvec)INIT_LIST_HEAD (&lruvec->lists[lru]); } -#if defined(CONFIG_NUMA_BALANCING ) && !defined(LAST_NID_NOT_IN_PAGE_FLAGS ) -int page_nid_xchg_last (struct page *page,int nid) +#if defined(CONFIG_NUMA_BALANCING ) && !defined(LAST_NIDPID_NOT_IN_PAGE_FLAGS ) +int page_nidpid_xchg_last (struct page *page,int nidpid) {unsigned long old_flags, flags; -int last_nid; +int last_nidpid;do { old_flags = flags = page->flags; - last_nid =page_nid_last (page); + last_nidpid =page_nidpid_last (page); - flags &= ~(LAST_NID_MASK <<LAST_NID_PGSHIFT ); - flags |= (nid &LAST_NID_MASK ) <<LAST_NID_PGSHIFT ; + flags &= ~(LAST_NIDPID_MASK <<LAST_NIDPID_PGSHIFT ); + flags |= (nidpid &LAST_NIDPID_MASK ) <<LAST_NIDPID_PGSHIFT ; }while (unlikely (cmpxchg (&page->flags, old_flags, flags) != old_flags)); -return last_nid; +return last_nidpid; }#endif
mm/mprotect.c +17 -9
@@ -37,14 +37,15 @@ static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot)static unsigned long change_pte_range (struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot, -int dirty_accountable,int prot_numa,bool *ret_all_same_node) +int dirty_accountable,int prot_numa,bool *ret_all_same_nidpid) {struct mm_struct *mm = vma->vm_mm;pte_t *pte, oldpte;spinlock_t *ptl;unsigned long pages =0 ; -bool all_same_node = true; +bool all_same_nidpid = true;int last_nid =-1 ; +int last_pid =-1 ; pte =pte_offset_map_lock (mm, pmd, addr, &ptl);arch_enter_lazy_mmu_mode ();
@@ -63,11 +64,18 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, page =vm_normal_page (vma, addr, oldpte);if (page) { -int this_nid =page_to_nid (page); +int nidpid =page_nidpid_last (page); +int this_nid =nidpid_to_nid (nidpid); +int this_pid =nidpid_to_pid (nidpid); +if (last_nid ==-1 ) last_nid = this_nid; -if (last_nid != this_nid) - all_same_node = false; +if (last_pid ==-1 ) + last_pid = this_pid; +if (last_nid != this_nid || + last_pid != this_pid) { + all_same_nidpid = false; + }if (!pte_numa (oldpte)) { ptent =pte_mknuma (ptent);
@@ -107,7 +115,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,arch_leave_lazy_mmu_mode ();pte_unmap_unlock (pte -1 , ptl); - *ret_all_same_node = all_same_node; + *ret_all_same_nidpid = all_same_nidpid;return pages; }
@@ -134,7 +142,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,pmd_t *pmd;unsigned long next;unsigned long pages =0 ; -bool all_same_node; +bool all_same_nidpid; pmd =pmd_offset (pud, addr); do {
@@ -158,7 +166,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,if (pmd_none_or_clear_bad (pmd))continue ; pages +=change_pte_range (vma, pmd, addr, next, newprot, - dirty_accountable, prot_numa, &all_same_node); + dirty_accountable, prot_numa, &all_same_nidpid);/* * If we are changing protections for NUMA hinting faults then
@@ -166,7 +174,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,* node. This allows a regular PMD to be handled as one fault * and effectively batches the taking of the PTL */ -if (prot_numa && all_same_node) +if (prot_numa && all_same_nidpid)change_pmd_protnuma (vma->vm_mm, addr, pmd); }while (pmd++, addr = next, addr != end);
mm/page_alloc.c +2 -2
@@ -626,7 +626,7 @@ static inline int free_pages_check(struct page *page)bad_page (page);return 1 ; } -page_nid_reset_last (page); +page_nidpid_reset_last (page);if (page->flags &PAGE_FLAGS_CHECK_AT_PREP ) page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP ;return 0 ;
@@ -4015,7 +4015,7 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,mminit_verify_page_links (page, zone, nid, pfn);init_page_count (page);page_mapcount_reset (page); -page_nid_reset_last (page); +page_nidpid_reset_last (page);SetPageReserved (page);/* * Mark the block movable so that blocks are reserved for
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 29/63] sched: Set preferred NUMA node based on number of private faults
Quoted 39 lines · Mel Gorman
Ideally it would be possible to distinguish between NUMA hinting faults that are private to a task and those that are shared. If treated identically there is a risk that shared pages bounce between nodes depending on the order they are referenced by tasks. Ultimately what is desirable is that task private pages remain local to the task while shared pages are interleaved between sharing tasks running on different nodes to give good average performance. This is further complicated by THP as even applications that partition their data may not be partitioning on a huge page boundary.
To start with, this patch assumes that multi-threaded or multi-process applications partition their data and that in general the private accesses are more important for
cpu->memorylocality in the general case. Also, no new infrastructure is required to treat private pages properly but interleaving for shared pages requires additional infrastructure.To detect private accesses the pid of the last accessing task is required but the storage requirements are a high. This patch borrows heavily from Ingo Molnar's patch "numa, mm, sched: Implement last-CPU+PID hash tracking" to encode some bits from the last accessing task in the page flags as well as the node information. Collisions will occur but it is better than just depending on the node information. Node information is then used to determine if a page needs to migrate. The PID information is used to detect private/shared accesses. The preferred NUMA node is selected based on where the maximum number of approximately private faults were measured. Shared faults are not taken into consideration for a few reasons.
First, if there are many tasks sharing the page then they'll all move towards the same node. The node will be compute overloaded and then scheduled away later only to bounce back again. Alternatively the shared tasks would just bounce around nodes because the fault information is effectively noise. Either way accounting for shared faults the same as private faults can result in lower performance overall.
The second reason is based on a hypothetical workload that has a small number of very important, heavily accessed private pages but a large shared array. The shared array would dominate the number of faults and be selected as a preferred node even though it's the wrong decision.
Quoted 5 lines
The third reason is that multiple threads in a process will race each other to fault the shared page making the fault information unreliable.
[[email protected]: Fix complication error when !NUMA_BALANCING]Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Set preferred NUMA node based on number of private faults
In reply to Mel Gormansched/numa: Set preferred NUMA node based on number of private faults
Commit-ID: b795854b1fa70f6aee923ae5df74ff7afeaddcaa Gitweb: http://git.kernel.org/tip/b795854b1fa70f6aee923ae5df74ff7afeaddcaa Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:07 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:35 +0200
sched/numa: Set preferred NUMA node based on number of private faults
Ideally it would be possible to distinguish between NUMA hinting faults that are private to a task and those that are shared. If treated identically there is a risk that shared pages bounce between nodes depending on the order they are referenced by tasks. Ultimately what is desirable is that task private pages remain local to the task while shared pages are interleaved between sharing tasks running on different nodes to give good average performance. This is further complicated by THP as even applications that partition their data may not be partitioning on a huge page boundary.
To start with, this patch assumes that multi-threaded or multi-process applications partition their data and that in general the private accesses are more important for cpu->memory locality in the general case. Also, no new infrastructure is required to treat private pages properly but interleaving for shared pages requires additional infrastructure.
To detect private accesses the pid of the last accessing task is required but the storage requirements are a high. This patch borrows heavily from Ingo Molnar's patch "numa, mm, sched: Implement last-CPU+PID hash tracking" to encode some bits from the last accessing task in the page flags as well as the node information. Collisions will occur but it is better than just depending on the node information. Node information is then used to determine if a page needs to migrate. The PID information is used to detect private/shared accesses. The preferred NUMA node is selected based on where the maximum number of approximately private faults were measured. Shared faults are not taken into consideration for a few reasons.
First, if there are many tasks sharing the page then they'll all move towards the same node. The node will be compute overloaded and then scheduled away later only to bounce back again. Alternatively the shared tasks would just bounce around nodes because the fault information is effectively noise. Either way accounting for shared faults the same as private faults can result in lower performance overall.
The second reason is based on a hypothetical workload that has a small number of very important, heavily accessed private pages but a large shared array. The shared array would dominate the number of faults and be selected as a preferred node even though it's the wrong decision.
The third reason is that multiple threads in a process will race each other to fault the shared page making the fault information unreliable.
--- include/linux/mm.h | 89 +++++++++++++++++++++++++++++---------- include/linux/mm_types.h | 4 +- include/linux/page-flags-layout.h | 28 +++++++----- kernel/sched/fair.c | 12 ++++-- mm/huge_memory.c | 8 ++-- mm/memory.c | 16 +++---- mm/mempolicy.c | 8 ++-- mm/migrate.c | 4 +- mm/mm_init.c | 18 ++++---- mm/mmzone.c | 14 +++--- mm/mprotect.c | 26 ++++++++---- mm/page_alloc.c | 4 +- 12 files changed, 149 insertions(+), 82 deletions(-)
include/linux/mm.h +67 -22
@@ -581,11 +581,11 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)* sets it, so none of the operations on it need to be atomic. */ -/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_NID] | ... | FLAGS | */ +/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_NIDPID] | ... | FLAGS | */ #define SECTIONS_PGOFF ((sizeof(unsigned long)*8) - SECTIONS_WIDTH)#define NODES_PGOFF (SECTIONS_PGOFF - NODES_WIDTH)#define ZONES_PGOFF (NODES_PGOFF - ZONES_WIDTH) -#define LAST_NID_PGOFF (ZONES_PGOFF - LAST_NID_WIDTH) +#define LAST_NIDPID_PGOFF (ZONES_PGOFF - LAST_NIDPID_WIDTH)/* * Define the bit shifts to access each section. For non-existent
@@ -595,7 +595,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)#define SECTIONS_PGSHIFT (SECTIONS_PGOFF * (SECTIONS_WIDTH != 0))#define NODES_PGSHIFT (NODES_PGOFF * (NODES_WIDTH != 0))#define ZONES_PGSHIFT (ZONES_PGOFF * (ZONES_WIDTH != 0)) -#define LAST_NID_PGSHIFT (LAST_NID_PGOFF * (LAST_NID_WIDTH != 0)) +#define LAST_NIDPID_PGSHIFT (LAST_NIDPID_PGOFF * (LAST_NIDPID_WIDTH != 0))/* NODE:ZONE or SECTION:ZONE is used to ID a zone for the buddy allocator */ #ifdef NODE_NOT_IN_PAGE_FLAGS
@@ -617,7 +617,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)#define ZONES_MASK ((1UL << ZONES_WIDTH) - 1)#define NODES_MASK ((1UL << NODES_WIDTH) - 1)#define SECTIONS_MASK ((1UL << SECTIONS_WIDTH) - 1) -#define LAST_NID_MASK ((1UL << LAST_NID_WIDTH) - 1) +#define LAST_NIDPID_MASK ((1UL << LAST_NIDPID_WIDTH) - 1)#define ZONEID_MASK ((1UL << ZONEID_SHIFT) - 1)static inline enum zone_type page_zonenum (const struct page *page)
@@ -661,48 +661,93 @@ static inline int page_to_nid(const struct page *page)#endif #ifdef CONFIG_NUMA_BALANCING -#ifdef LAST_NID_NOT_IN_PAGE_FLAGS -static inline int page_nid_xchg_last (struct page *page,int nid) +static inline int nid_pid_to_nidpid (int nid,int pid) { -return xchg (&page->_last_nid, nid); +return ((nid &LAST__NID_MASK ) <<LAST__PID_SHIFT ) | (pid &LAST__PID_MASK ); } -static inline int page_nid_last (struct page *page) +static inline int nidpid_to_pid (int nidpid) { -return page->_last_nid; +return nidpid &LAST__PID_MASK ; } -static inline void page_nid_reset_last (struct page *page) + +static inline int nidpid_to_nid (int nidpid) +{ +return (nidpid >>LAST__PID_SHIFT ) &LAST__NID_MASK ; +} + +static inline bool nidpid_pid_unset (int nidpid) +{ +return nidpid_to_pid (nidpid) == (-1 &LAST__PID_MASK ); +} + +static inline bool nidpid_nid_unset (int nidpid) { - page->_last_nid =-1 ; +return nidpid_to_nid (nidpid) == (-1 &LAST__NID_MASK ); +} + +#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS +static inline int page_nidpid_xchg_last (struct page *page,int nid) +{ +return xchg (&page->_last_nidpid, nid); +} + +static inline int page_nidpid_last (struct page *page) +{ +return page->_last_nidpid; +} +static inline void page_nidpid_reset_last (struct page *page) +{ + page->_last_nidpid =-1 ; }#else -static inline int page_nid_last (struct page *page) +static inline int page_nidpid_last (struct page *page) { -return (page->flags >>LAST_NID_PGSHIFT ) &LAST_NID_MASK ; +return (page->flags >>LAST_NIDPID_PGSHIFT ) &LAST_NIDPID_MASK ; } -extern int page_nid_xchg_last (struct page *page,int nid); +extern int page_nidpid_xchg_last (struct page *page,int nidpid); -static inline void page_nid_reset_last (struct page *page) +static inline void page_nidpid_reset_last (struct page *page) { -int nid = (1 <<LAST_NID_SHIFT ) -1 ; +int nidpid = (1 <<LAST_NIDPID_SHIFT ) -1 ; - page->flags &= ~(LAST_NID_MASK <<LAST_NID_PGSHIFT ); - page->flags |= (nid &LAST_NID_MASK ) <<LAST_NID_PGSHIFT ; + page->flags &= ~(LAST_NIDPID_MASK <<LAST_NIDPID_PGSHIFT ); + page->flags |= (nidpid &LAST_NIDPID_MASK ) <<LAST_NIDPID_PGSHIFT ; } -#endif /* LAST_NID_NOT_IN_PAGE_FLAGS */ +#endif /* LAST_NIDPID_NOT_IN_PAGE_FLAGS */ #else -static inline int page_nid_xchg_last (struct page *page,int nid) +static inline int page_nidpid_xchg_last (struct page *page,int nidpid) {return page_to_nid (page); } -static inline int page_nid_last (struct page *page) +static inline int page_nidpid_last (struct page *page) {return page_to_nid (page); } -static inline void page_nid_reset_last (struct page *page) +static inline int nidpid_to_nid (int nidpid) +{ +return -1 ; +} + +static inline int nidpid_to_pid (int nidpid) +{ +return -1 ; +} + +static inline int nid_pid_to_nidpid (int nid,int pid) +{ +return -1 ; +} + +static inline bool nidpid_pid_unset (int nidpid) +{ +return 1 ; +} + +static inline void page_nidpid_reset_last (struct page *page) { }#endif
include/linux/mm_types.h +2 -2
@@ -174,8 +174,8 @@ struct page {void *shadow;#endif -#ifdef LAST_NID_NOT_IN_PAGE_FLAGS -int _last_nid; +#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS +int _last_nidpid;#endif }/*
include/linux/page-flags-layout.h +17 -11
@@ -38,10 +38,10 @@* The last is when there is insufficient space in page->flags and a separate * lookup is necessary. * -* No sparsemem or sparsemem vmemmap: | NODE | ZONE | ... | FLAGS | -* " plus space for last_nid: | NODE | ZONE | LAST_NID ... | FLAGS | -* classic sparse with space for node:| SECTION | NODE | ZONE | ... | FLAGS | -* " plus space for last_nid: | SECTION | NODE | ZONE | LAST_NID ... | FLAGS | +* No sparsemem or sparsemem vmemmap: | NODE | ZONE | ... | FLAGS | +* " plus space for last_nidpid: | NODE | ZONE | LAST_NIDPID ... | FLAGS | +* classic sparse with space for node:| SECTION | NODE | ZONE | ... | FLAGS | +* " plus space for last_nidpid: | SECTION | NODE | ZONE | LAST_NIDPID ... | FLAGS | * classic sparse no space for node: | SECTION | ZONE | ... | FLAGS | */ #if defined(CONFIG_SPARSEMEM ) && !defined(CONFIG_SPARSEMEM_VMEMMAP )
@@ -62,15 +62,21 @@#endif #ifdef CONFIG_NUMA_BALANCING -#define LAST_NID_SHIFT NODES_SHIFT +#define LAST__PID_SHIFT 8 +#define LAST__PID_MASK ((1 << LAST__PID_SHIFT)-1) + +#define LAST__NID_SHIFT NODES_SHIFT +#define LAST__NID_MASK ((1 << LAST__NID_SHIFT)-1) + +#define LAST_NIDPID_SHIFT (LAST__PID_SHIFT+LAST__NID_SHIFT)#else -#define LAST_NID_SHIFT 0 +#define LAST_NIDPID_SHIFT 0#endif -#if SECTIONS_WIDTH +ZONES_WIDTH +NODES_SHIFT +LAST_NID_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS -#define LAST_NID_WIDTH LAST_NID_SHIFT +#if SECTIONS_WIDTH +ZONES_WIDTH +NODES_SHIFT +LAST_NIDPID_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS +#define LAST_NIDPID_WIDTH LAST_NIDPID_SHIFT#else -#define LAST_NID_WIDTH 0 +#define LAST_NIDPID_WIDTH 0#endif /*
@@ -81,8 +87,8 @@#define NODE_NOT_IN_PAGE_FLAGS #endif -#if defined(CONFIG_NUMA_BALANCING ) &&LAST_NID_WIDTH ==0 -#define LAST_NID_NOT_IN_PAGE_FLAGS +#if defined(CONFIG_NUMA_BALANCING ) &&LAST_NIDPID_WIDTH ==0 +#define LAST_NIDPID_NOT_IN_PAGE_FLAGS #endif #endif /* _LINUX_PAGE_FLAGS_LAYOUT */
kernel/sched/fair.c +9 -3
@@ -988,7 +988,7 @@ static void task_numa_placement(struct task_struct *p)/* * Got a PROT_NONE fault for a page on @node. */ -void task_numa_fault (int last_nid,int node,int pages,bool migrated) +void task_numa_fault (int last_nidpid,int node,int pages,bool migrated) {struct task_struct *p = current;int priv;
@@ -1000,8 +1000,14 @@ void task_numa_fault(int last_nid, int node, int pages, bool migrated)if (!p->mm)return ; -/* For now, do not attempt to detect private/shared accesses */ - priv =1 ; +/* +* First accesses are treated as private, otherwise consider accesses +* to be private if the accessing pid has not changed +*/ +if (!nidpid_pid_unset (last_nidpid)) + priv = ((p->pid &LAST__PID_MASK ) ==nidpid_to_pid (last_nidpid)); +else + priv =1 ;/* Allocate buffer to track faults on a per-node basis */ if (unlikely (!p->numa_faults)) {
mm/huge_memory.c +4 -4
@@ -1282,7 +1282,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct page *page;unsigned long haddr = addr &HPAGE_PMD_MASK ;int page_nid =-1 , this_nid =numa_node_id (); -int target_nid, last_nid =-1 ; +int target_nid, last_nidpid =-1 ;bool page_locked;bool migrated = false;
@@ -1293,7 +1293,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, page =pmd_page (pmd);BUG_ON (is_huge_zero_page (page)); page_nid =page_to_nid (page); - last_nid =page_nid_last (page); + last_nidpid =page_nidpid_last (page);count_vm_numa_event (NUMA_HINT_FAULTS );if (page_nid == this_nid)count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL );
@@ -1362,7 +1362,7 @@ out:page_unlock_anon_vma_read (anon_vma);if (page_nid !=-1 ) -task_numa_fault (last_nid, page_nid,HPAGE_PMD_NR , migrated); +task_numa_fault (last_nidpid, page_nid,HPAGE_PMD_NR , migrated);return 0 ; }
@@ -1682,7 +1682,7 @@ static void __split_huge_page_refcount(struct page *page, page_tail->mapping = page->mapping; page_tail->index = page->index + i; -page_nid_xchg_last (page_tail,page_nid_last (page)); +page_nidpid_xchg_last (page_tail,page_nidpid_last (page));BUG_ON (!PageAnon (page_tail));BUG_ON (!PageUptodate (page_tail));
mm/memory.c +8 -8
@@ -69,8 +69,8 @@#include "internal.h" -#ifdef LAST_NID_NOT_IN_PAGE_FLAGS -#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_nid. +#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS +#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_nidpid.#endif #ifndef CONFIG_NEED_MULTIPLE_NODES
@@ -3536,7 +3536,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct page *page =NULL ;spinlock_t *ptl;int page_nid =-1 ; -int last_nid; +int last_nidpid;int target_nid;bool migrated = false;
@@ -3567,7 +3567,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }BUG_ON (is_zero_pfn (page_to_pfn (page))); - last_nid =page_nid_last (page); + last_nidpid =page_nidpid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (ptep, ptl);
@@ -3583,7 +3583,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, out:if (page_nid !=-1 ) -task_numa_fault (last_nid, page_nid,1 , migrated); +task_numa_fault (last_nidpid, page_nid,1 , migrated);return 0 ; }
@@ -3598,7 +3598,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,unsigned long offset;spinlock_t *ptl;bool numa = false; -int last_nid; +int last_nidpid;spin_lock (&mm->page_table_lock); pmd = *pmdp;
@@ -3643,7 +3643,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,if (unlikely (!page))continue ; - last_nid =page_nid_last (page); + last_nidpid =page_nidpid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (pte, ptl);
@@ -3656,7 +3656,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }if (page_nid !=-1 ) -task_numa_fault (last_nid, page_nid,1 , migrated); +task_numa_fault (last_nidpid, page_nid,1 , migrated); pte =pte_offset_map_lock (mm, pmdp, addr, &ptl); }
mm/mempolicy.c +5 -3
@@ -2348,9 +2348,11 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long/* Migrate the page towards the node whose CPU is referencing it */ if (pol->flags &MPOL_F_MORON ) { -int last_nid; +int last_nidpid; +int this_nidpid; polnid =numa_node_id (); + this_nidpid =nid_pid_to_nidpid (polnid, current->pid);/* * Multi-stage node selection is used in conjunction
@@ -2373,8 +2375,8 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long* it less likely we act on an unlikely task<->page * relation. */ - last_nid =page_nid_xchg_last (page, polnid); -if (last_nid != polnid) + last_nidpid =page_nidpid_xchg_last (page, this_nidpid); +if (!nidpid_pid_unset (last_nidpid) &&nidpid_to_nid (last_nidpid) != polnid) goto out; }
mm/migrate.c +2 -2
@@ -1498,7 +1498,7 @@ static struct page *alloc_misplaced_dst_page(struct page *page, __GFP_NOWARN) & ~GFP_IOFS, 0);if (newpage) -page_nid_xchg_last (newpage,page_nid_last (page)); +page_nidpid_xchg_last (newpage,page_nidpid_last (page));return newpage; }
@@ -1675,7 +1675,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,if (!new_page) goto out_fail; -page_nid_xchg_last (new_page,page_nid_last (page)); +page_nidpid_xchg_last (new_page,page_nidpid_last (page)); isolated =numamigrate_isolate_page (pgdat, page);if (!isolated) {
mm/mm_init.c +9 -9
@@ -71,26 +71,26 @@ void __init mminit_verify_pageflags_layout(void)unsigned long or_mask, add_mask; shift =8 *sizeof (unsigned long ); - width = shift -SECTIONS_WIDTH -NODES_WIDTH -ZONES_WIDTH -LAST_NID_SHIFT ; + width = shift -SECTIONS_WIDTH -NODES_WIDTH -ZONES_WIDTH -LAST_NIDPID_SHIFT ;mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_widths" , -"Section %d Node %d Zone %d Lastnid %d Flags %d\n" , +"Section %d Node %d Zone %d Lastnidpid %d Flags %d\n" ,SECTIONS_WIDTH ,NODES_WIDTH ,ZONES_WIDTH , -LAST_NID_WIDTH , +LAST_NIDPID_WIDTH ,NR_PAGEFLAGS );mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_shifts" , -"Section %d Node %d Zone %d Lastnid %d\n" , +"Section %d Node %d Zone %d Lastnidpid %d\n" ,SECTIONS_SHIFT ,NODES_SHIFT ,ZONES_SHIFT , -LAST_NID_SHIFT ); +LAST_NIDPID_SHIFT );mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_pgshifts" , -"Section %lu Node %lu Zone %lu Lastnid %lu\n" , +"Section %lu Node %lu Zone %lu Lastnidpid %lu\n" , (unsigned long )SECTIONS_PGSHIFT , (unsigned long )NODES_PGSHIFT , (unsigned long )ZONES_PGSHIFT , - (unsigned long )LAST_NID_PGSHIFT ); + (unsigned long )LAST_NIDPID_PGSHIFT ); mminit_dprintk(MMINIT_TRACE, "pageflags_layout_nodezoneid", "Node/Zone ID: %lu -> %lu\n", (unsigned long)(ZONEID_PGOFF + ZONEID_SHIFT),
@@ -102,9 +102,9 @@ void __init mminit_verify_pageflags_layout(void)mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_nodeflags" ,"Node not in page flags" );#endif -#ifdef LAST_NID_NOT_IN_PAGE_FLAGS +#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_nodeflags" , -"Last nid not in page flags" ); +"Last nidpid not in page flags" );#endif if (SECTIONS_WIDTH ) {
mm/mmzone.c +7 -7
@@ -97,20 +97,20 @@ void lruvec_init(struct lruvec *lruvec)INIT_LIST_HEAD (&lruvec->lists[lru]); } -#if defined(CONFIG_NUMA_BALANCING ) && !defined(LAST_NID_NOT_IN_PAGE_FLAGS ) -int page_nid_xchg_last (struct page *page,int nid) +#if defined(CONFIG_NUMA_BALANCING ) && !defined(LAST_NIDPID_NOT_IN_PAGE_FLAGS ) +int page_nidpid_xchg_last (struct page *page,int nidpid) {unsigned long old_flags, flags; -int last_nid; +int last_nidpid;do { old_flags = flags = page->flags; - last_nid =page_nid_last (page); + last_nidpid =page_nidpid_last (page); - flags &= ~(LAST_NID_MASK <<LAST_NID_PGSHIFT ); - flags |= (nid &LAST_NID_MASK ) <<LAST_NID_PGSHIFT ; + flags &= ~(LAST_NIDPID_MASK <<LAST_NIDPID_PGSHIFT ); + flags |= (nidpid &LAST_NIDPID_MASK ) <<LAST_NIDPID_PGSHIFT ; }while (unlikely (cmpxchg (&page->flags, old_flags, flags) != old_flags)); -return last_nid; +return last_nidpid; }#endif
mm/mprotect.c +17 -9
@@ -37,14 +37,15 @@ static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot)static unsigned long change_pte_range (struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot, -int dirty_accountable,int prot_numa,bool *ret_all_same_node) +int dirty_accountable,int prot_numa,bool *ret_all_same_nidpid) {struct mm_struct *mm = vma->vm_mm;pte_t *pte, oldpte;spinlock_t *ptl;unsigned long pages =0 ; -bool all_same_node = true; +bool all_same_nidpid = true;int last_nid =-1 ; +int last_pid =-1 ; pte =pte_offset_map_lock (mm, pmd, addr, &ptl);arch_enter_lazy_mmu_mode ();
@@ -63,11 +64,18 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, page =vm_normal_page (vma, addr, oldpte);if (page) { -int this_nid =page_to_nid (page); +int nidpid =page_nidpid_last (page); +int this_nid =nidpid_to_nid (nidpid); +int this_pid =nidpid_to_pid (nidpid); +if (last_nid ==-1 ) last_nid = this_nid; -if (last_nid != this_nid) - all_same_node = false; +if (last_pid ==-1 ) + last_pid = this_pid; +if (last_nid != this_nid || + last_pid != this_pid) { + all_same_nidpid = false; + }if (!pte_numa (oldpte)) { ptent =pte_mknuma (ptent);
@@ -107,7 +115,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,arch_leave_lazy_mmu_mode ();pte_unmap_unlock (pte -1 , ptl); - *ret_all_same_node = all_same_node; + *ret_all_same_nidpid = all_same_nidpid;return pages; }
@@ -134,7 +142,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,pmd_t *pmd;unsigned long next;unsigned long pages =0 ; -bool all_same_node; +bool all_same_nidpid; pmd =pmd_offset (pud, addr); do {
@@ -158,7 +166,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,if (pmd_none_or_clear_bad (pmd))continue ; pages +=change_pte_range (vma, pmd, addr, next, newprot, - dirty_accountable, prot_numa, &all_same_node); + dirty_accountable, prot_numa, &all_same_nidpid);/* * If we are changing protections for NUMA hinting faults then
@@ -166,7 +174,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,* node. This allows a regular PMD to be handled as one fault * and effectively batches the taking of the PTL */ -if (prot_numa && all_same_node) +if (prot_numa && all_same_nidpid)change_pmd_protnuma (vma->vm_mm, addr, pmd); }while (pmd++, addr = next, addr != end);
mm/page_alloc.c +2 -2
@@ -626,7 +626,7 @@ static inline int free_pages_check(struct page *page)bad_page (page);return 1 ; } -page_nid_reset_last (page); +page_nidpid_reset_last (page);if (page->flags &PAGE_FLAGS_CHECK_AT_PREP ) page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP ;return 0 ;
@@ -4015,7 +4015,7 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,mminit_verify_page_links (page, zone, nid, pfn);init_page_count (page);page_mapcount_reset (page); -page_nid_reset_last (page); +page_nidpid_reset_last (page);SetPageReserved (page);/* * Mark the block movable so that blocks are reserved for
Mel Gorman
reviewerPATCH 30/63 sched: Do not migrate memory immediately after switching node
In reply to Mel GormanFrom: Rik van Riel <[email protected]>
The load balancer can move tasks between nodes and does not take NUMA locality into account. With automatic NUMA balancing this may result in the tasks working set being migrated to the new node. However, as the fault buffer will still store faults from the old node the schduler may decide to reset the preferred node and migrate the task back resulting in more migrations.
The ideal would be that the scheduler did not migrate tasks with a heavy memory footprint but this may result nodes being overloaded. We could also discard the fault information on task migration but this would still cause all the tasks working set to be migrated. This patch simply avoids migrating the memory for a short time after a task is migrated.
--- kernel/sched/core.c | 2 +- kernel/sched/fair.c | 18 ++++++++++++++++-- mm/mempolicy.c | 12 ++++++++++++ 3 files changed, 29 insertions(+), 3 deletions(-)
kernel/sched/core.c +1 -1
@@ -1626,7 +1626,7 @@ static void __sched_fork(struct task_struct *p) p->node_stamp =0ULL ; p->numa_scan_seq = p->mm ? p->mm->numa_scan_seq :0 ; - p->numa_migrate_seq =0 ; + p->numa_migrate_seq =1 ; p->numa_scan_period = sysctl_numa_balancing_scan_delay; p->numa_preferred_nid =-1 ; p->numa_work.next = &p->numa_work;
kernel/sched/fair.c +16 -2
@@ -884,7 +884,7 @@ static unsigned int task_scan_max(struct task_struct *p)* the preferred node but still allow the scheduler to move the task again if * the nodes CPUs are overloaded. */ -unsigned int sysctl_numa_balancing_settle_count__read_mostly =3 ; +unsigned int sysctl_numa_balancing_settle_count__read_mostly =4 ;static inline int task_faults_idx (int nid,int priv) {
@@ -980,7 +980,7 @@ static void task_numa_placement(struct task_struct *p)/* Update the preferred nid and migrate task if possible */ p->numa_preferred_nid = max_nid; - p->numa_migrate_seq =0 ; + p->numa_migrate_seq =1 ;migrate_task_to (p, preferred_cpu); } }
@@ -4120,6 +4120,20 @@ static void move_task(struct task_struct *p, struct lb_env *env)set_task_cpu (p, env->dst_cpu);activate_task (env->dst_rq, p,0 );check_preempt_curr (env->dst_rq, p,0 ); +#ifdef CONFIG_NUMA_BALANCING +if (p->numa_preferred_nid !=-1 ) { +int src_nid =cpu_to_node (env->src_cpu); +int dst_nid =cpu_to_node (env->dst_cpu); + +/* +* If the load balancer has moved the task then limit +* migrations from taking place in the short term in +* case this is a short-lived migration. +*/ +if (src_nid != dst_nid && dst_nid != p->numa_preferred_nid) + p->numa_migrate_seq =0 ; + } +#endif }/*
mm/mempolicy.c +12 -0
@@ -2378,6 +2378,18 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long last_nidpid =page_nidpid_xchg_last (page, this_nidpid);if (!nidpid_pid_unset (last_nidpid) &&nidpid_to_nid (last_nidpid) != polnid) goto out; + +#ifdef CONFIG_NUMA_BALANCING +/* +* If the scheduler has just moved us away from our +* preferred node, do not bother migrating pages yet. +* This way a short and temporary process migration will +* not cause excessive memory migration. +*/ +if (polnid != current->numa_preferred_nid && + !current->numa_migrate_seq) + goto out; +#endif }if (curnid != polnid)
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Rik van Riel
merge bot[tip:sched/core] sched/numa: Do not migrate memory immediately after switching node
sched/numa: Do not migrate memory immediately after switching node
Commit-ID: 6fe6b2d6dabf392aceb3ad3a5e859b46a04465c6 Gitweb: http://git.kernel.org/tip/6fe6b2d6dabf392aceb3ad3a5e859b46a04465c6 Author: Rik van Riel <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:08 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:36 +0200
sched/numa: Do not migrate memory immediately after switching node
The load balancer can move tasks between nodes and does not take NUMA locality into account. With automatic NUMA balancing this may result in the tasks working set being migrated to the new node. However, as the fault buffer will still store faults from the old node the schduler may decide to reset the preferred node and migrate the task back resulting in more migrations.
The ideal would be that the scheduler did not migrate tasks with a heavy memory footprint but this may result nodes being overloaded. We could also discard the fault information on task migration but this would still cause all the tasks working set to be migrated. This patch simply avoids migrating the memory for a short time after a task is migrated.
--- kernel/sched/core.c | 2 +- kernel/sched/fair.c | 18 ++++++++++++++++-- mm/mempolicy.c | 12 ++++++++++++ 3 files changed, 29 insertions(+), 3 deletions(-)
kernel/sched/core.c +1 -1
@@ -1631,7 +1631,7 @@ static void __sched_fork(struct task_struct *p) p->node_stamp =0ULL ; p->numa_scan_seq = p->mm ? p->mm->numa_scan_seq :0 ; - p->numa_migrate_seq =0 ; + p->numa_migrate_seq =1 ; p->numa_scan_period = sysctl_numa_balancing_scan_delay; p->numa_preferred_nid =-1 ; p->numa_work.next = &p->numa_work;
kernel/sched/fair.c +16 -2
@@ -884,7 +884,7 @@ static unsigned int task_scan_max(struct task_struct *p)* the preferred node but still allow the scheduler to move the task again if * the nodes CPUs are overloaded. */ -unsigned int sysctl_numa_balancing_settle_count__read_mostly =3 ; +unsigned int sysctl_numa_balancing_settle_count__read_mostly =4 ;static inline int task_faults_idx (int nid,int priv) {
@@ -980,7 +980,7 @@ static void task_numa_placement(struct task_struct *p)/* Update the preferred nid and migrate task if possible */ p->numa_preferred_nid = max_nid; - p->numa_migrate_seq =0 ; + p->numa_migrate_seq =1 ;migrate_task_to (p, preferred_cpu); } }
@@ -4121,6 +4121,20 @@ static void move_task(struct task_struct *p, struct lb_env *env)set_task_cpu (p, env->dst_cpu);activate_task (env->dst_rq, p,0 );check_preempt_curr (env->dst_rq, p,0 ); +#ifdef CONFIG_NUMA_BALANCING +if (p->numa_preferred_nid !=-1 ) { +int src_nid =cpu_to_node (env->src_cpu); +int dst_nid =cpu_to_node (env->dst_cpu); + +/* +* If the load balancer has moved the task then limit +* migrations from taking place in the short term in +* case this is a short-lived migration. +*/ +if (src_nid != dst_nid && dst_nid != p->numa_preferred_nid) + p->numa_migrate_seq =0 ; + } +#endif }/*
mm/mempolicy.c +12 -0
@@ -2378,6 +2378,18 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long last_nidpid =page_nidpid_xchg_last (page, this_nidpid);if (!nidpid_pid_unset (last_nidpid) &&nidpid_to_nid (last_nidpid) != polnid) goto out; + +#ifdef CONFIG_NUMA_BALANCING +/* +* If the scheduler has just moved us away from our +* preferred node, do not bother migrating pages yet. +* This way a short and temporary process migration will +* not cause excessive memory migration. +*/ +if (polnid != current->numa_preferred_nid && + !current->numa_migrate_seq) + goto out; +#endif }if (curnid != polnid)
Mel Gorman
reviewerPATCH 31/63 mm: numa: only unmap migrate-on-fault VMAs
In reply to Mel GormanThere is a 90% regression observed with a large Oracle performance test on a 4 node system. Profiles indicated that the overhead was due to contention on sp_lock when looking up shared memory policies. These policies do not have the appropriate flags to allow them to be automatically balanced so trapping faults on them is pointless. This patch skips VMAs that do not have MPOL_F_MOF set.
--- include/linux/mempolicy.h | 1 + kernel/sched/fair.c | 2 +- mm/mempolicy.c | 24 ++++++++++++++++++++++++ 3 files changed, 26 insertions(+), 1 deletion(-)
include/linux/mempolicy.h +1 -0
@@ -136,6 +136,7 @@ struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp,struct mempolicy *get_vma_policy (struct task_struct *tsk,struct vm_area_struct *vma,unsigned long addr); +bool vma_policy_mof (struct task_struct *task,struct vm_area_struct *vma);extern void numa_default_policy (void );extern void numa_policy_init (void );
kernel/sched/fair.c +1 -1
@@ -1130,7 +1130,7 @@ void task_numa_work(struct callback_head *work) vma = mm->mmap; }for (; vma; vma = vma->vm_next) { -if (!vma_migratable (vma)) +if (!vma_migratable (vma) || !vma_policy_mof (p, vma))continue ; do {
mm/mempolicy.c +24 -0
@@ -1679,6 +1679,30 @@ struct mempolicy *get_vma_policy(struct task_struct *task,return pol; } +bool vma_policy_mof (struct task_struct *task,struct vm_area_struct *vma) +{ +struct mempolicy *pol =get_task_policy (task); +if (vma) { +if (vma->vm_ops && vma->vm_ops->get_policy) { +bool ret = false; + + pol = vma->vm_ops->get_policy (vma, vma->vm_start); +if (pol && (pol->flags &MPOL_F_MOF )) + ret = true; +mpol_cond_put (pol); + +return ret; + }else if (vma->vm_policy) { + pol = vma->vm_policy; + } + } + +if (!pol) +return default_policy.flags &MPOL_F_MOF ; + +return pol->flags &MPOL_F_MOF ; +} +static int apply_policy_zone (struct mempolicy *policy,enum zone_type zone) {enum zone_type dynamic_policy_zone = policy_zone;
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] mm: numa: Limit NUMA scanning to migrate-on-fault VMAs
mm: numa: Limit NUMA scanning to migrate-on-fault VMAs
Commit-ID: fc3147245d193bd0f57307859c698fa28a20b0fe Gitweb: http://git.kernel.org/tip/fc3147245d193bd0f57307859c698fa28a20b0fe Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:09 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:38 +0200
mm: numa: Limit NUMA scanning to migrate-on-fault VMAs
There is a 90% regression observed with a large Oracle performance test on a 4 node system. Profiles indicated that the overhead was due to contention on sp_lock when looking up shared memory policies. These policies do not have the appropriate flags to allow them to be automatically balanced so trapping faults on them is pointless. This patch skips VMAs that do not have MPOL_F_MOF set.
[[email protected]: Initial patch]
--- include/linux/mempolicy.h | 1 + kernel/sched/fair.c | 2 +- mm/mempolicy.c | 24 ++++++++++++++++++++++++ 3 files changed, 26 insertions(+), 1 deletion(-)
include/linux/mempolicy.h +1 -0
@@ -136,6 +136,7 @@ struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp,struct mempolicy *get_vma_policy (struct task_struct *tsk,struct vm_area_struct *vma,unsigned long addr); +bool vma_policy_mof (struct task_struct *task,struct vm_area_struct *vma);extern void numa_default_policy (void );extern void numa_policy_init (void );
kernel/sched/fair.c +1 -1
@@ -1130,7 +1130,7 @@ void task_numa_work(struct callback_head *work) vma = mm->mmap; }for (; vma; vma = vma->vm_next) { -if (!vma_migratable (vma)) +if (!vma_migratable (vma) || !vma_policy_mof (p, vma))continue ; do {
mm/mempolicy.c +24 -0
@@ -1679,6 +1679,30 @@ struct mempolicy *get_vma_policy(struct task_struct *task,return pol; } +bool vma_policy_mof (struct task_struct *task,struct vm_area_struct *vma) +{ +struct mempolicy *pol =get_task_policy (task); +if (vma) { +if (vma->vm_ops && vma->vm_ops->get_policy) { +bool ret = false; + + pol = vma->vm_ops->get_policy (vma, vma->vm_start); +if (pol && (pol->flags &MPOL_F_MOF )) + ret = true; +mpol_cond_put (pol); + +return ret; + }else if (vma->vm_policy) { + pol = vma->vm_policy; + } + } + +if (!pol) +return default_policy.flags &MPOL_F_MOF ; + +return pol->flags &MPOL_F_MOF ; +} +static int apply_policy_zone (struct mempolicy *policy,enum zone_type zone) {enum zone_type dynamic_policy_zone = policy_zone;
Mel Gorman
reviewerPATCH 32/63 sched: Avoid overloading CPUs on a preferred NUMA node
In reply to Mel GormanThis patch replaces find_idlest_cpu_node with task_numa_find_cpu. find_idlest_cpu_node has two critical limitations. It does not take the scheduling class into account when calculating the load and it is unsuitable for using when comparing loads between NUMA nodes.
task_numa_find_cpu uses similar load calculations to wake_affine() when selecting the least loaded CPU within a scheduling domain common to the source and destimation nodes. It avoids causing CPU load imbalances in the machine by refusing to migrate if the relative load on the target CPU is higher than the source CPU.
--- kernel/sched/fair.c | 131 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 102 insertions(+), 29 deletions(-)
kernel/sched/fair.c +102 -29
@@ -901,28 +901,114 @@ static inline unsigned long task_faults(struct task_struct *p, int nid) }static unsigned long weighted_cpuload (const int cpu); +static unsigned long source_load (int cpu,int type); +static unsigned long target_load (int cpu,int type); +static unsigned long power_of (int cpu); +static long effective_load (struct task_group *tg,int cpu,long wl,long wg); +struct numa_stats { +unsigned long load; +s64 eff_load; +unsigned long faults; +}; -static int -find_idlest_cpu_node (int this_cpu,int nid) -{ -unsigned long load, min_load =ULONG_MAX ; -int i, idlest_cpu = this_cpu; +struct task_numa_env { +struct task_struct *p; -BUG_ON (cpu_to_node (this_cpu) == nid); +int src_cpu, src_nid; +int dst_cpu, dst_nid; -rcu_read_lock (); -for_each_cpu (i,cpumask_of_node (nid)) { - load =weighted_cpuload (i); +struct numa_stats src_stats, dst_stats; -if (load < min_load) { - min_load = load; - idlest_cpu = i; +unsigned long best_load; +int best_cpu; +}; + +static int task_numa_migrate (struct task_struct *p) +{ +int node_cpu =cpumask_first (cpumask_of_node (p->numa_preferred_nid)); +struct task_numa_env env = { + .p = p, + .src_cpu =task_cpu (p), + .src_nid =cpu_to_node (task_cpu (p)), + .dst_cpu = node_cpu, + .dst_nid = p->numa_preferred_nid, + .best_load =ULONG_MAX , + .best_cpu =task_cpu (p), + }; +struct sched_domain *sd; +int cpu; +struct task_group *tg =task_group (p); +unsigned long weight; +bool balanced; +int imbalance_pct, idx =-1 ; + +/* +* Find the lowest common scheduling domain covering the nodes of both +* the CPU the task is currently running on and the target NUMA node. +*/ +rcu_read_lock (); +for_each_domain (env.src_cpu, sd) { +if (cpumask_test_cpu (node_cpu,sched_domain_span (sd))) { +/* +* busy_idx is used for the load decision as it is the +* same index used by the regular load balancer for an +* active cpu. +*/ + idx = sd->busy_idx; + imbalance_pct = sd->imbalance_pct; +break ; } }rcu_read_unlock (); -return idlest_cpu; +if (WARN_ON_ONCE (idx ==-1 )) +return 0 ; + +/* +* XXX the below is mostly nicked from wake_affine(); we should +* see about sharing a bit if at all possible; also it might want +* some per entity weight love. +*/ + weight = p->se.load.weight; + env.src_stats.load =source_load (env.src_cpu, idx); + env.src_stats.eff_load =100 + (imbalance_pct -100 ) /2 ; + env.src_stats.eff_load *=power_of (env.src_cpu); + env.src_stats.eff_load *= env.src_stats.load +effective_load (tg, env.src_cpu, -weight, -weight); + +for_each_cpu (cpu,cpumask_of_node (env.dst_nid)) { + env.dst_cpu = cpu; + env.dst_stats.load =target_load (cpu, idx); + +/* If the CPU is idle, use it */ +if (!env.dst_stats.load) { + env.best_cpu = cpu; + goto migrate; + } + +/* Otherwise check the target CPU load */ + env.dst_stats.eff_load =100 ; + env.dst_stats.eff_load *=power_of (cpu); + env.dst_stats.eff_load *= env.dst_stats.load +effective_load (tg, cpu, weight, weight); + +/* +* Destination is considered balanced if the destination CPU is +* less loaded than the source CPU. Unfortunately there is a +* risk that a task running on a lightly loaded CPU will not +* migrate to its preferred node due to load imbalances. +*/ + balanced = (env.dst_stats.eff_load <= env.src_stats.eff_load); +if (!balanced) +continue ; + +if (env.dst_stats.eff_load < env.best_load) { + env.best_load = env.dst_stats.eff_load; + env.best_cpu = cpu; + } + } + +migrate: +return migrate_task_to (p, env.best_cpu); }static void task_numa_placement (struct task_struct *p)
@@ -966,22 +1052,10 @@ static void task_numa_placement(struct task_struct *p)* the working set placement. */ if (max_faults && max_nid != p->numa_preferred_nid) { -int preferred_cpu; - -/* -* If the task is not on the preferred node then find the most -* idle CPU to migrate to. -*/ - preferred_cpu =task_cpu (p); -if (cpu_to_node (preferred_cpu) != max_nid) { - preferred_cpu =find_idlest_cpu_node (preferred_cpu, - max_nid); - } -/* Update the preferred nid and migrate task if possible */ p->numa_preferred_nid = max_nid; p->numa_migrate_seq =1 ; -migrate_task_to (p, preferred_cpu); +task_numa_migrate (p); } }
@@ -3292,7 +3366,7 @@ static long effective_load(struct task_group *tg, int cpu, long wl, long wg) {struct sched_entity *se = tg->se[cpu]; -if (!tg->parent)/* the trivial, non-cgroup case */ +if (!tg->parent || !wl)/* the trivial, non-cgroup case */ return wl;for_each_sched_entity (se) {
@@ -3345,8 +3419,7 @@ static long effective_load(struct task_group *tg, int cpu, long wl, long wg) }#else -static inline unsigned long effective_load (struct task_group *tg,int cpu, -unsigned long wl,unsigned long wg) +static long effective_load (struct task_group *tg,int cpu,long wl,long wg) {return wl; }
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 32/63] sched: Avoid overloading CPUs on a preferred NUMA node
Quoted 5 lines · Mel Gorman
This patch replaces
find_idlest_cpu_nodewithtask_numa_find_cpu.find_idlest_cpu_nodehas two critical limitations. It does not take the scheduling class into account when calculating the load and it is unsuitable for using when comparing loads between NUMA nodes.
Quoted 8 lines
task_numa_find_cpuuses similar load calculations towake_affine()when selecting the least loaded CPU within a scheduling domain common to the source and destimation nodes. It avoids causing CPU load imbalances in the machine by refusing to migrate if the relative load on the target CPU is higher than the source CPU.Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Avoid overloading CPUs on a preferred NUMA node
In reply to Mel Gormansched/numa: Avoid overloading CPUs on a preferred NUMA node
Commit-ID: 58d081b5082dd85e02ac9a1fb151d97395340a09 Gitweb: http://git.kernel.org/tip/58d081b5082dd85e02ac9a1fb151d97395340a09 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:10 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:39 +0200
sched/numa: Avoid overloading CPUs on a preferred NUMA node
This patch replaces find_idlest_cpu_node with task_numa_find_cpu. find_idlest_cpu_node has two critical limitations. It does not take the scheduling class into account when calculating the load and it is unsuitable for using when comparing loads between NUMA nodes.
task_numa_find_cpu uses similar load calculations to wake_affine() when selecting the least loaded CPU within a scheduling domain common to the source and destimation nodes. It avoids causing CPU load imbalances in the machine by refusing to migrate if the relative load on the target CPU is higher than the source CPU.
--- kernel/sched/fair.c | 131 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 102 insertions(+), 29 deletions(-)
kernel/sched/fair.c +102 -29
@@ -901,28 +901,114 @@ static inline unsigned long task_faults(struct task_struct *p, int nid) }static unsigned long weighted_cpuload (const int cpu); +static unsigned long source_load (int cpu,int type); +static unsigned long target_load (int cpu,int type); +static unsigned long power_of (int cpu); +static long effective_load (struct task_group *tg,int cpu,long wl,long wg); +struct numa_stats { +unsigned long load; +s64 eff_load; +unsigned long faults; +}; -static int -find_idlest_cpu_node (int this_cpu,int nid) -{ -unsigned long load, min_load =ULONG_MAX ; -int i, idlest_cpu = this_cpu; +struct task_numa_env { +struct task_struct *p; -BUG_ON (cpu_to_node (this_cpu) == nid); +int src_cpu, src_nid; +int dst_cpu, dst_nid; -rcu_read_lock (); -for_each_cpu (i,cpumask_of_node (nid)) { - load =weighted_cpuload (i); +struct numa_stats src_stats, dst_stats; -if (load < min_load) { - min_load = load; - idlest_cpu = i; +unsigned long best_load; +int best_cpu; +}; + +static int task_numa_migrate (struct task_struct *p) +{ +int node_cpu =cpumask_first (cpumask_of_node (p->numa_preferred_nid)); +struct task_numa_env env = { + .p = p, + .src_cpu =task_cpu (p), + .src_nid =cpu_to_node (task_cpu (p)), + .dst_cpu = node_cpu, + .dst_nid = p->numa_preferred_nid, + .best_load =ULONG_MAX , + .best_cpu =task_cpu (p), + }; +struct sched_domain *sd; +int cpu; +struct task_group *tg =task_group (p); +unsigned long weight; +bool balanced; +int imbalance_pct, idx =-1 ; + +/* +* Find the lowest common scheduling domain covering the nodes of both +* the CPU the task is currently running on and the target NUMA node. +*/ +rcu_read_lock (); +for_each_domain (env.src_cpu, sd) { +if (cpumask_test_cpu (node_cpu,sched_domain_span (sd))) { +/* +* busy_idx is used for the load decision as it is the +* same index used by the regular load balancer for an +* active cpu. +*/ + idx = sd->busy_idx; + imbalance_pct = sd->imbalance_pct; +break ; } }rcu_read_unlock (); -return idlest_cpu; +if (WARN_ON_ONCE (idx ==-1 )) +return 0 ; + +/* +* XXX the below is mostly nicked from wake_affine(); we should +* see about sharing a bit if at all possible; also it might want +* some per entity weight love. +*/ + weight = p->se.load.weight; + env.src_stats.load =source_load (env.src_cpu, idx); + env.src_stats.eff_load =100 + (imbalance_pct -100 ) /2 ; + env.src_stats.eff_load *=power_of (env.src_cpu); + env.src_stats.eff_load *= env.src_stats.load +effective_load (tg, env.src_cpu, -weight, -weight); + +for_each_cpu (cpu,cpumask_of_node (env.dst_nid)) { + env.dst_cpu = cpu; + env.dst_stats.load =target_load (cpu, idx); + +/* If the CPU is idle, use it */ +if (!env.dst_stats.load) { + env.best_cpu = cpu; + goto migrate; + } + +/* Otherwise check the target CPU load */ + env.dst_stats.eff_load =100 ; + env.dst_stats.eff_load *=power_of (cpu); + env.dst_stats.eff_load *= env.dst_stats.load +effective_load (tg, cpu, weight, weight); + +/* +* Destination is considered balanced if the destination CPU is +* less loaded than the source CPU. Unfortunately there is a +* risk that a task running on a lightly loaded CPU will not +* migrate to its preferred node due to load imbalances. +*/ + balanced = (env.dst_stats.eff_load <= env.src_stats.eff_load); +if (!balanced) +continue ; + +if (env.dst_stats.eff_load < env.best_load) { + env.best_load = env.dst_stats.eff_load; + env.best_cpu = cpu; + } + } + +migrate: +return migrate_task_to (p, env.best_cpu); }static void task_numa_placement (struct task_struct *p)
@@ -966,22 +1052,10 @@ static void task_numa_placement(struct task_struct *p)* the working set placement. */ if (max_faults && max_nid != p->numa_preferred_nid) { -int preferred_cpu; - -/* -* If the task is not on the preferred node then find the most -* idle CPU to migrate to. -*/ - preferred_cpu =task_cpu (p); -if (cpu_to_node (preferred_cpu) != max_nid) { - preferred_cpu =find_idlest_cpu_node (preferred_cpu, - max_nid); - } -/* Update the preferred nid and migrate task if possible */ p->numa_preferred_nid = max_nid; p->numa_migrate_seq =1 ; -migrate_task_to (p, preferred_cpu); +task_numa_migrate (p); } }
@@ -3292,7 +3366,7 @@ static long effective_load(struct task_group *tg, int cpu, long wl, long wg) {struct sched_entity *se = tg->se[cpu]; -if (!tg->parent)/* the trivial, non-cgroup case */ +if (!tg->parent || !wl)/* the trivial, non-cgroup case */ return wl;for_each_sched_entity (se) {
@@ -3345,8 +3419,7 @@ static long effective_load(struct task_group *tg, int cpu, long wl, long wg) }#else -static inline unsigned long effective_load (struct task_group *tg,int cpu, -unsigned long wl,unsigned long wg) +static long effective_load (struct task_group *tg,int cpu,long wl,long wg) {return wl; }
Mel Gorman
reviewerPATCH 33/63 sched: Retry migration of tasks to CPU on a preferred node
In reply to Mel GormanWhen a preferred node is selected for a tasks there is an attempt to migrate the task to a CPU there. This may fail in which case the task will only migrate if the active load balancer takes action. This may never happen if the conditions are not right. This patch will check at NUMA hinting fault time if another attempt should be made to migrate the task. It will only make an attempt once every five seconds.
--- include/linux/sched.h | 1 + kernel/sched/fair.c | 30 +++++++++++++++++++++++------- 2 files changed, 24 insertions(+), 7 deletions(-)
include/linux/sched.h +1 -0
@@ -1331,6 +1331,7 @@ struct task_struct {int numa_migrate_seq;unsigned int numa_scan_period;unsigned int numa_scan_period_max; +unsigned long numa_migrate_retry;u64 node_stamp;/* migration stamp */ struct callback_head numa_work;
kernel/sched/fair.c +23 -7
@@ -1011,6 +1011,23 @@ migrate:return migrate_task_to (p, env.best_cpu); } +/* Attempt to migrate a task to a CPU on the preferred node. */ +static void numa_migrate_preferred (struct task_struct *p) +{ +/* Success if task is already running on preferred CPU */ + p->numa_migrate_retry =0 ; +if (cpu_to_node (task_cpu (p)) == p->numa_preferred_nid) +return ; + +/* This task has no NUMA fault statistics yet */ +if (unlikely (p->numa_preferred_nid ==-1 )) +return ; + +/* Otherwise, try migrate to a CPU on the preferred node */ +if (task_numa_migrate (p) !=0 ) + p->numa_migrate_retry = jiffies +HZ *5 ; +} +static void task_numa_placement (struct task_struct *p) {int seq, nid, max_nid =-1 ;
@@ -1045,17 +1062,12 @@ static void task_numa_placement(struct task_struct *p) } } -/* -* Record the preferred node as the node with the most faults, -* requeue the task to be running on the idlest CPU on the -* preferred node and reset the scanning rate to recheck -* the working set placement. -*/ +/* Preferred node as the node with the most faults */ if (max_faults && max_nid != p->numa_preferred_nid) {/* Update the preferred nid and migrate task if possible */ p->numa_preferred_nid = max_nid; p->numa_migrate_seq =1 ; -task_numa_migrate (p); +numa_migrate_preferred (p); } }
@@ -1111,6 +1123,10 @@ void task_numa_fault(int last_nidpid, int node, int pages, bool migrated)task_numa_placement (p); +/* Retry task to preferred node migration if it previously failed */ +if (p->numa_migrate_retry &&time_after (jiffies, p->numa_migrate_retry)) +numa_migrate_preferred (p); + p->numa_faults_buffer[task_faults_idx (node, priv)] += pages; }
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 33/63] sched: Retry migration of tasks to CPU on a preferred node
Quoted 9 lines · Mel Gorman
When a preferred node is selected for a tasks there is an attempt to migrate the task to a CPU there. This may fail in which case the task will only migrate if the active load balancer takes action. This may never happen if the conditions are not right. This patch will check at NUMA hinting fault time if another attempt should be made to migrate the task. It will only make an attempt once every five seconds.
Signed-off-by Mel Gorman <[email protected]>Signed-off-by Peter Zijlstra <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Retry migration of tasks to CPU on a preferred node
In reply to Mel Gormansched/numa: Retry migration of tasks to CPU on a preferred node
Commit-ID: 6b9a7460b6baf6c77fc3d23d927ddfc3f3f05bf3 Gitweb: http://git.kernel.org/tip/6b9a7460b6baf6c77fc3d23d927ddfc3f3f05bf3 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:11 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:40 +0200
sched/numa: Retry migration of tasks to CPU on a preferred node
When a preferred node is selected for a tasks there is an attempt to migrate the task to a CPU there. This may fail in which case the task will only migrate if the active load balancer takes action. This may never happen if the conditions are not right. This patch will check at NUMA hinting fault time if another attempt should be made to migrate the task. It will only make an attempt once every five seconds.
--- include/linux/sched.h | 1 + kernel/sched/fair.c | 30 +++++++++++++++++++++++------- 2 files changed, 24 insertions(+), 7 deletions(-)
include/linux/sched.h +1 -0
@@ -1341,6 +1341,7 @@ struct task_struct {int numa_migrate_seq;unsigned int numa_scan_period;unsigned int numa_scan_period_max; +unsigned long numa_migrate_retry;u64 node_stamp;/* migration stamp */ struct callback_head numa_work;
kernel/sched/fair.c +23 -7
@@ -1011,6 +1011,23 @@ migrate:return migrate_task_to (p, env.best_cpu); } +/* Attempt to migrate a task to a CPU on the preferred node. */ +static void numa_migrate_preferred (struct task_struct *p) +{ +/* Success if task is already running on preferred CPU */ + p->numa_migrate_retry =0 ; +if (cpu_to_node (task_cpu (p)) == p->numa_preferred_nid) +return ; + +/* This task has no NUMA fault statistics yet */ +if (unlikely (p->numa_preferred_nid ==-1 )) +return ; + +/* Otherwise, try migrate to a CPU on the preferred node */ +if (task_numa_migrate (p) !=0 ) + p->numa_migrate_retry = jiffies +HZ *5 ; +} +static void task_numa_placement (struct task_struct *p) {int seq, nid, max_nid =-1 ;
@@ -1045,17 +1062,12 @@ static void task_numa_placement(struct task_struct *p) } } -/* -* Record the preferred node as the node with the most faults, -* requeue the task to be running on the idlest CPU on the -* preferred node and reset the scanning rate to recheck -* the working set placement. -*/ +/* Preferred node as the node with the most faults */ if (max_faults && max_nid != p->numa_preferred_nid) {/* Update the preferred nid and migrate task if possible */ p->numa_preferred_nid = max_nid; p->numa_migrate_seq =1 ; -task_numa_migrate (p); +numa_migrate_preferred (p); } }
@@ -1111,6 +1123,10 @@ void task_numa_fault(int last_nidpid, int node, int pages, bool migrated)task_numa_placement (p); +/* Retry task to preferred node migration if it previously failed */ +if (p->numa_migrate_retry &&time_after (jiffies, p->numa_migrate_retry)) +numa_migrate_preferred (p); + p->numa_faults_buffer[task_faults_idx (node, priv)] += pages; }
Mel Gorman
reviewerPATCH 34/63 sched: numa: increment numa_migrate_seq when task runs in correct location
In reply to Mel GormanFrom: Rik van Riel <[email protected]>
When a task is already running on its preferred node, increment numa_migrate_seq to indicate that the task is settled if migration is temporarily disabled, and memory should migrate towards it.
--- kernel/sched/fair.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
kernel/sched/fair.c +9 -1
@@ -1016,8 +1016,16 @@ static void numa_migrate_preferred(struct task_struct *p) {/* Success if task is already running on preferred CPU */ p->numa_migrate_retry =0 ; -if (cpu_to_node (task_cpu (p)) == p->numa_preferred_nid) +if (cpu_to_node (task_cpu (p)) == p->numa_preferred_nid) { +/* +* If migration is temporarily disabled due to a task migration +* then re-enable it now as the task is running on its +* preferred node and memory should migrate locally +*/ +if (!p->numa_migrate_seq) + p->numa_migrate_seq++;return ; + }/* This task has no NUMA fault statistics yet */ if (unlikely (p->numa_preferred_nid ==-1 ))
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Rik van Riel
merge bot[tip:sched/core] sched/numa: Increment numa_migrate_seq when task runs in correct location
sched/numa: Increment numa_migrate_seq when task runs in correct location
Commit-ID: 06ea5e035b4e66cc77790457a89fc7e368060c4b Gitweb: http://git.kernel.org/tip/06ea5e035b4e66cc77790457a89fc7e368060c4b Author: Rik van Riel <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:12 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:41 +0200
sched/numa: Increment numa_migrate_seq when task runs in correct location
When a task is already running on its preferred node, increment numa_migrate_seq to indicate that the task is settled if migration is temporarily disabled, and memory should migrate towards it.
--- kernel/sched/fair.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
kernel/sched/fair.c +9 -1
@@ -1016,8 +1016,16 @@ static void numa_migrate_preferred(struct task_struct *p) {/* Success if task is already running on preferred CPU */ p->numa_migrate_retry =0 ; -if (cpu_to_node (task_cpu (p)) == p->numa_preferred_nid) +if (cpu_to_node (task_cpu (p)) == p->numa_preferred_nid) { +/* +* If migration is temporarily disabled due to a task migration +* then re-enable it now as the task is running on its +* preferred node and memory should migrate locally +*/ +if (!p->numa_migrate_seq) + p->numa_migrate_seq++;return ; + }/* This task has no NUMA fault statistics yet */ if (unlikely (p->numa_preferred_nid ==-1 ))
Mel Gorman
reviewerPATCH 35/63 sched: numa: Do not trap hinting faults for shared libraries
In reply to Mel GormanNUMA hinting faults will not migrate a shared executable page mapped by multiple processes on the grounds that the data is probably in the CPU cache already and the page may just bounce between tasks running on multipl nodes. Even if the migration is avoided, there is still the overhead of trapping the fault, updating the statistics, making scheduler placement decisions based on the information etc. If we are never going to migrate the page, it is overhead for no gain and worse a process may be placed on a sub-optimal node for shared executable pages. This patch avoids trapping faults for shared libraries entirely.
--- kernel/sched/fair.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
kernel/sched/fair.c +10 -0
@@ -1231,6 +1231,16 @@ void task_numa_work(struct callback_head *work)if (!vma_migratable (vma) || !vma_policy_mof (p, vma))continue ; +/* +* Shared library pages mapped by multiple processes are not +* migrated as it is expected they are cache replicated. Avoid +* hinting faults in read-only file-backed mappings or the vdso +* as migrating the pages will be of marginal benefit. +*/ +if (!vma->vm_mm || + (vma->vm_file && (vma->vm_flags & (VM_READ |VM_WRITE )) == (VM_READ ))) +continue ; + do { start = max(start, vma->vm_start); end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 35/63] sched: numa: Do not trap hinting faults for shared libraries
Quoted 11 lines · Mel Gorman
NUMA hinting faults will not migrate a shared executable page mapped by multiple processes on the grounds that the data is probably in the CPU cache already and the page may just bounce between tasks running on multipl nodes. Even if the migration is avoided, there is still the overhead of trapping the fault, updating the statistics, making scheduler placement decisions based on the information etc. If we are never going to migrate the page, it is overhead for no gain and worse a process may be placed on a sub-optimal node for shared executable pages. This patch avoids trapping faults for shared libraries entirely.
Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Do not trap hinting faults for shared libraries
In reply to Mel Gormansched/numa: Do not trap hinting faults for shared libraries
Commit-ID: 4591ce4f2d22dc9de7a6719161ce409b5fd1caac Gitweb: http://git.kernel.org/tip/4591ce4f2d22dc9de7a6719161ce409b5fd1caac Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:13 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:42 +0200
sched/numa: Do not trap hinting faults for shared libraries
NUMA hinting faults will not migrate a shared executable page mapped by multiple processes on the grounds that the data is probably in the CPU cache already and the page may just bounce between tasks running on multipl nodes. Even if the migration is avoided, there is still the overhead of trapping the fault, updating the statistics, making scheduler placement decisions based on the information etc. If we are never going to migrate the page, it is overhead for no gain and worse a process may be placed on a sub-optimal node for shared executable pages. This patch avoids trapping faults for shared libraries entirely.
--- kernel/sched/fair.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
kernel/sched/fair.c +10 -0
@@ -1231,6 +1231,16 @@ void task_numa_work(struct callback_head *work)if (!vma_migratable (vma) || !vma_policy_mof (p, vma))continue ; +/* +* Shared library pages mapped by multiple processes are not +* migrated as it is expected they are cache replicated. Avoid +* hinting faults in read-only file-backed mappings or the vdso +* as migrating the pages will be of marginal benefit. +*/ +if (!vma->vm_mm || + (vma->vm_file && (vma->vm_flags & (VM_READ |VM_WRITE )) == (VM_READ ))) +continue ; + do { start = max(start, vma->vm_start); end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);
Mel Gorman
reviewerPATCH 36/63 mm: numa: Only trap pmd hinting faults if we would otherwise trap PTE faults
In reply to Mel GormanBase page PMD faulting is meant to batch handle NUMA hinting faults from PTEs. However, even is no PTE faults would ever be handled within a range the kernel still traps PMD hinting faults. This patch avoids the overhead.
--- mm/mprotect.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
mm/mprotect.c +5 -2
@@ -146,6 +146,8 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma, pmd =pmd_offset (pud, addr); do { + unsigned long this_pages; + next = pmd_addr_end(addr, end); if (pmd_trans_huge(*pmd)) { if (next - addr != HPAGE_PMD_SIZE)
@@ -165,8 +167,9 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma, }if (pmd_none_or_clear_bad (pmd))continue ; - pages +=change_pte_range (vma, pmd, addr, next, newprot, + this_pages =change_pte_range (vma, pmd, addr, next, newprot, dirty_accountable, prot_numa, &all_same_nidpid); + pages += this_pages;/* * If we are changing protections for NUMA hinting faults then
@@ -174,7 +177,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,* node. This allows a regular PMD to be handled as one fault * and effectively batches the taking of the PTL */ -if (prot_numa && all_same_nidpid) +if (prot_numa && this_pages && all_same_nidpid)change_pmd_protnuma (vma->vm_mm, addr, pmd); }while (pmd++, addr = next, addr != end);
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 36/63] mm: numa: Only trap pmd hinting faults if we would otherwise trap PTE faults
Quoted 6 lines · Mel Gorman
Base page PMD faulting is meant to batch handle NUMA hinting faults from PTEs. However, even is no PTE faults would ever be handled within a range the kernel still traps PMD hinting faults. This patch avoids the overhead.
Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] mm: numa: Trap pmd hinting faults only if we would otherwise trap PTE faults
In reply to Mel Gormanmm: numa: Trap pmd hinting faults only if we would otherwise trap PTE faults
Commit-ID: 25cbbef1924299249756bc4030fcb2436c019813 Gitweb: http://git.kernel.org/tip/25cbbef1924299249756bc4030fcb2436c019813 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:14 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:44 +0200
mm: numa: Trap pmd hinting faults only if we would otherwise trap PTE faults
Base page PMD faulting is meant to batch handle NUMA hinting faults from PTEs. However, even is no PTE faults would ever be handled within a range the kernel still traps PMD hinting faults. This patch avoids the overhead.
--- mm/mprotect.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
mm/mprotect.c +5 -2
@@ -146,6 +146,8 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma, pmd =pmd_offset (pud, addr); do { + unsigned long this_pages; + next = pmd_addr_end(addr, end); if (pmd_trans_huge(*pmd)) { if (next - addr != HPAGE_PMD_SIZE)
@@ -165,8 +167,9 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma, }if (pmd_none_or_clear_bad (pmd))continue ; - pages +=change_pte_range (vma, pmd, addr, next, newprot, + this_pages =change_pte_range (vma, pmd, addr, next, newprot, dirty_accountable, prot_numa, &all_same_nidpid); + pages += this_pages;/* * If we are changing protections for NUMA hinting faults then
@@ -174,7 +177,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,* node. This allows a regular PMD to be handled as one fault * and effectively batches the taking of the PTL */ -if (prot_numa && all_same_nidpid) +if (prot_numa && this_pages && all_same_nidpid)change_pmd_protnuma (vma->vm_mm, addr, pmd); }while (pmd++, addr = next, addr != end);
Mel Gorman
reviewerPATCH 37/63 stop_machine: Introduce stop_two_cpus()
In reply to Mel GormanFrom: Peter Zijlstra <[email protected]>
Introduce stop_two_cpus() in order to allow controlled swapping of two tasks. It repurposes the stop_machine() state machine but only stops the two cpus which we can do with on-stack structures and avoid machine wide synchronization issues.
The ordering of CPUs is important to avoid deadlocks. If unordered then two cpus calling stop_two_cpus on each other simultaneously would attempt to queue in the opposite order on each CPU causing an AB-BA style deadlock. By always having the lowest number CPU doing the queueing of works, we can guarantee that works are always queued in the same order, and deadlocks are avoided.
--- include/linux/stop_machine.h | 1 + kernel/stop_machine.c | 272 +++++++++++++++++++++++++++---------------- 2 files changed, 175 insertions(+), 98 deletions(-)
include/linux/stop_machine.h +1 -0
@@ -28,6 +28,7 @@ struct cpu_stop_work { };int stop_one_cpu (unsigned int cpu,cpu_stop_fn_t fn,void *arg); +int stop_two_cpus (unsigned int cpu1,unsigned int cpu2,cpu_stop_fn_t fn,void *arg);void stop_one_cpu_nowait (unsigned int cpu,cpu_stop_fn_t fn,void *arg,struct cpu_stop_work *work_buf);int stop_cpus (const struct cpumask *cpumask,cpu_stop_fn_t fn,void *arg);
kernel/stop_machine.c +174 -98
318 lines not on this page · raw message
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Peter Zijlstra
merge bot[tip:sched/core] stop_machine: Introduce stop_two_cpus()
stop_machine: Introduce stop_two_cpus()
Commit-ID: 1be0bd77c5dd7c903f46abf52f9a3650face3c1d Gitweb: http://git.kernel.org/tip/1be0bd77c5dd7c903f46abf52f9a3650face3c1d Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:15 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:45 +0200
stop_machine: Introduce stop_two_cpus()
Introduce stop_two_cpus() in order to allow controlled swapping of two tasks. It repurposes the stop_machine() state machine but only stops the two cpus which we can do with on-stack structures and avoid machine wide synchronization issues.
The ordering of CPUs is important to avoid deadlocks. If unordered then two cpus calling stop_two_cpus on each other simultaneously would attempt to queue in the opposite order on each CPU causing an AB-BA style deadlock. By always having the lowest number CPU doing the queueing of works, we can guarantee that works are always queued in the same order, and deadlocks are avoided.
--- include/linux/stop_machine.h | 1 + kernel/stop_machine.c | 272 +++++++++++++++++++++++++++---------------- 2 files changed, 175 insertions(+), 98 deletions(-)
include/linux/stop_machine.h +1 -0
@@ -28,6 +28,7 @@ struct cpu_stop_work { };int stop_one_cpu (unsigned int cpu,cpu_stop_fn_t fn,void *arg); +int stop_two_cpus (unsigned int cpu1,unsigned int cpu2,cpu_stop_fn_t fn,void *arg);void stop_one_cpu_nowait (unsigned int cpu,cpu_stop_fn_t fn,void *arg,struct cpu_stop_work *work_buf);int stop_cpus (const struct cpumask *cpumask,cpu_stop_fn_t fn,void *arg);
kernel/stop_machine.c +174 -98
318 lines not on this page · raw message
Mel Gorman
reviewerPATCH 38/63 sched: Introduce migrate_swap()
In reply to Mel GormanFrom: Peter Zijlstra <[email protected]>
Use the new stop_two_cpus() to implement migrate_swap(), a function that flips two tasks between their respective cpus.
I'm fairly sure there's a less crude way than employing the stop_two_cpus() method, but everything I tried either got horribly fragile and/or complex. So keep it simple for now.
The notable detail is how we 'migrate' tasks that aren't runnable anymore. We'll make it appear like we migrated them before they went to sleep. The sole difference is the previous cpu in the wakeup path, so we override this.
--- include/linux/sched.h | 2 + kernel/sched/core.c | 106 ++++++++++++++++++++++++++++++++++++++++++++--- kernel/sched/fair.c | 3 +- kernel/sched/idle_task.c | 2 +- kernel/sched/rt.c | 5 +-- kernel/sched/sched.h | 4 +- kernel/sched/stop_task.c | 2 +- 7 files changed, 110 insertions(+), 14 deletions(-)
include/linux/sched.h +2 -0
@@ -1033,6 +1033,8 @@ struct task_struct {struct task_struct *last_wakee;unsigned long wakee_flips;unsigned long wakee_flip_decay_ts; + +int wake_cpu;#endif int on_rq;
kernel/sched/core.c +101 -5
137 lines not on this page · raw message
kernel/sched/fair.c +1 -2
@@ -3706,11 +3706,10 @@ done:* preempt must be disabled. */ static int -select_task_rq_fair (struct task_struct *p,int sd_flag,int wake_flags) +select_task_rq_fair (struct task_struct *p,int prev_cpu,int sd_flag,int wake_flags) {struct sched_domain *tmp, *affine_sd =NULL , *sd =NULL ;int cpu =smp_processor_id (); -int prev_cpu =task_cpu (p);int new_cpu = cpu;int want_affine =0 ;int sync = wake_flags &WF_SYNC ;
kernel/sched/idle_task.c +1 -1
@@ -9,7 +9,7 @@#ifdef CONFIG_SMP static int -select_task_rq_idle (struct task_struct *p,int sd_flag,int flags) +select_task_rq_idle (struct task_struct *p,int cpu,int sd_flag,int flags) {return task_cpu (p);/* IDLE tasks as never migrated */ }
kernel/sched/rt.c +1 -4
@@ -1169,13 +1169,10 @@ static void yield_task_rt(struct rq *rq)static int find_lowest_rq (struct task_struct *task);static int -select_task_rq_rt (struct task_struct *p,int sd_flag,int flags) +select_task_rq_rt (struct task_struct *p,int cpu,int sd_flag,int flags) {struct task_struct *curr;struct rq *rq; -int cpu; - - cpu =task_cpu (p);if (p->nr_cpus_allowed ==1 ) goto out;
kernel/sched/sched.h +3 -1
@@ -555,6 +555,7 @@ static inline u64 rq_clock_task(struct rq *rq)#ifdef CONFIG_NUMA_BALANCING extern int migrate_task_to (struct task_struct *p,int cpu); +extern int migrate_swap (struct task_struct *,struct task_struct *);static inline void task_numa_free (struct task_struct *p) {kfree (p->numa_faults);
@@ -732,6 +733,7 @@ static inline void __set_task_cpu(struct task_struct *p, unsigned int cpu)*/ smp_wmb ();task_thread_info (p)->cpu = cpu; + p->wake_cpu = cpu;#endif }
@@ -987,7 +989,7 @@ struct sched_class {void (*put_prev_task) (struct rq *rq,struct task_struct *p);#ifdef CONFIG_SMP -int (*select_task_rq)(struct task_struct *p,int sd_flag,int flags); +int (*select_task_rq)(struct task_struct *p,int task_cpu,int sd_flag,int flags);void (*migrate_task_rq)(struct task_struct *p,int next_cpu);void (*pre_schedule) (struct rq *this_rq,struct task_struct *task);
kernel/sched/stop_task.c +1 -1
@@ -11,7 +11,7 @@#ifdef CONFIG_SMP static int -select_task_rq_stop (struct task_struct *p,int sd_flag,int flags) +select_task_rq_stop (struct task_struct *p,int cpu,int sd_flag,int flags) {return task_cpu (p);/* stop tasks as never migrate */ }
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 38/63] sched: Introduce migrate_swap()
Quoted 9 lines · Mel Gorman
From: Peter Zijlstra <[email protected]>
Use the new
stop_two_cpus()to implementmigrate_swap(), a function that flips two tasks between their respective cpus.I'm fairly sure there's a less crude way than employing the
stop_two_cpus()method, but everything I tried either got horribly fragile and/or complex. So keep it simple for now.
Quoted 7 lines
The notable detail is how we 'migrate' tasks that aren't runnable anymore. We'll make it appear like we migrated them before they went to sleep. The sole difference is the previous cpu in the wakeup path, so we override this.
Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Peter Zijlstra
merge bot[tip:sched/core] sched/numa: Introduce migrate_swap()
In reply to Mel Gormansched/numa: Introduce migrate_swap()
Commit-ID: ac66f5477239ebd3c4e2cbf2f591ef387aa09884 Gitweb: http://git.kernel.org/tip/ac66f5477239ebd3c4e2cbf2f591ef387aa09884 Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:16 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 12:40:46 +0200
sched/numa: Introduce migrate_swap()
Use the new stop_two_cpus() to implement migrate_swap(), a function that flips two tasks between their respective cpus.
I'm fairly sure there's a less crude way than employing the stop_two_cpus() method, but everything I tried either got horribly fragile and/or complex. So keep it simple for now.
The notable detail is how we 'migrate' tasks that aren't runnable anymore. We'll make it appear like we migrated them before they went to sleep. The sole difference is the previous cpu in the wakeup path, so we override this.
--- include/linux/sched.h | 2 + kernel/sched/core.c | 106 ++++++++++++++++++++++++++++++++++++++++++++--- kernel/sched/fair.c | 3 +- kernel/sched/idle_task.c | 2 +- kernel/sched/rt.c | 5 +-- kernel/sched/sched.h | 4 +- kernel/sched/stop_task.c | 2 +- 7 files changed, 110 insertions(+), 14 deletions(-)
include/linux/sched.h +2 -0
@@ -1043,6 +1043,8 @@ struct task_struct {struct task_struct *last_wakee;unsigned long wakee_flips;unsigned long wakee_flip_decay_ts; + +int wake_cpu;#endif int on_rq;
kernel/sched/core.c +101 -5
137 lines not on this page · raw message
kernel/sched/fair.c +1 -2
@@ -3706,11 +3706,10 @@ done:* preempt must be disabled. */ static int -select_task_rq_fair (struct task_struct *p,int sd_flag,int wake_flags) +select_task_rq_fair (struct task_struct *p,int prev_cpu,int sd_flag,int wake_flags) {struct sched_domain *tmp, *affine_sd =NULL , *sd =NULL ;int cpu =smp_processor_id (); -int prev_cpu =task_cpu (p);int new_cpu = cpu;int want_affine =0 ;int sync = wake_flags &WF_SYNC ;
kernel/sched/idle_task.c +1 -1
@@ -9,7 +9,7 @@#ifdef CONFIG_SMP static int -select_task_rq_idle (struct task_struct *p,int sd_flag,int flags) +select_task_rq_idle (struct task_struct *p,int cpu,int sd_flag,int flags) {return task_cpu (p);/* IDLE tasks as never migrated */ }
kernel/sched/rt.c +1 -4
@@ -1169,13 +1169,10 @@ static void yield_task_rt(struct rq *rq)static int find_lowest_rq (struct task_struct *task);static int -select_task_rq_rt (struct task_struct *p,int sd_flag,int flags) +select_task_rq_rt (struct task_struct *p,int cpu,int sd_flag,int flags) {struct task_struct *curr;struct rq *rq; -int cpu; - - cpu =task_cpu (p);if (p->nr_cpus_allowed ==1 ) goto out;
kernel/sched/sched.h +3 -1
@@ -558,6 +558,7 @@ static inline u64 rq_clock_task(struct rq *rq)#ifdef CONFIG_NUMA_BALANCING extern int migrate_task_to (struct task_struct *p,int cpu); +extern int migrate_swap (struct task_struct *,struct task_struct *);static inline void task_numa_free (struct task_struct *p) {kfree (p->numa_faults);
@@ -736,6 +737,7 @@ static inline void __set_task_cpu(struct task_struct *p, unsigned int cpu)*/ smp_wmb ();task_thread_info (p)->cpu = cpu; + p->wake_cpu = cpu;#endif }
@@ -991,7 +993,7 @@ struct sched_class {void (*put_prev_task) (struct rq *rq,struct task_struct *p);#ifdef CONFIG_SMP -int (*select_task_rq)(struct task_struct *p,int sd_flag,int flags); +int (*select_task_rq)(struct task_struct *p,int task_cpu,int sd_flag,int flags);void (*migrate_task_rq)(struct task_struct *p,int next_cpu);void (*pre_schedule) (struct rq *this_rq,struct task_struct *task);
kernel/sched/stop_task.c +1 -1
@@ -11,7 +11,7 @@#ifdef CONFIG_SMP static int -select_task_rq_stop (struct task_struct *p,int sd_flag,int flags) +select_task_rq_stop (struct task_struct *p,int cpu,int sd_flag,int flags) {return task_cpu (p);/* stop tasks as never migrate */ }
Re: [tip:sched/core] sched/numa: Introduce migrate_swap()
Quoted 1 line · tip-bot for Peter Zijlstra
sched/numa: Introduce
migrate_swap()
Thanks to Rik for writing the Changelog!
---
From: Peter Zijlstra <[email protected]>
Subject: sched: Fix race in migrate_swap_stop
There is a subtle race in migrate_swap, when task P, on CPU A, decides to swap places with task T, on CPU B.
Task P:
- call migrate_swap
Task T:
- go to sleep, removing itself from the runqueue
Task P:
- double lock the runqueues on CPU A & B
Task T:
- get woken up, place itself on the runqueue of CPU C
Task P:
- see that task T is on a runqueue, and pretend to remove it from the runqueue on CPU B
Now CPUs B & C both have corrupted scheduler data structures.
This patch fixes it, by holding the pi_lock for both of the tasks involved in the migrate swap. This prevents task T from waking up, and placing itself onto another runqueue, until after migrate_swap has released all locks.
This means that, when migrate_swap checks, task T will be either on the runqueue where it was originally seen, or not on any runqueue at all. Migrate_swap deals correctly with of those cases.
--- kernel/sched/core.c | 4 ++++ kernel/sched/fair.c | 9 --------- kernel/sched/sched.h | 18 ++++++++++++++++++ 3 files changed, 22 insertions(+), 9 deletions(-)
kernel/sched/core.c +4 -0
@@ -1049,6 +1049,8 @@ static int migrate_swap_stop(void *data) src_rq =cpu_rq (arg->src_cpu); dst_rq =cpu_rq (arg->dst_cpu); +double_raw_lock (&arg->src_task->pi_lock, + &arg->dst_task->pi_lock);double_rq_lock (src_rq, dst_rq);if (task_cpu (arg->dst_task) != arg->dst_cpu) goto unlock;
@@ -1069,6 +1071,8 @@ static int migrate_swap_stop(void *data) unlock:double_rq_unlock (src_rq, dst_rq); +raw_spin_unlock (&arg->dst_task->pi_lock); +raw_spin_unlock (&arg->src_task->pi_lock);return ret; }
kernel/sched/fair.c +0 -9
@@ -1448,15 +1448,6 @@ static inline void put_numa_group(struct numa_group *grp)kfree_rcu (grp, rcu); } -static void double_lock (spinlock_t *l1,spinlock_t *l2) -{ -if (l1 > l2) -swap (l1, l2); - -spin_lock (l1); -spin_lock_nested (l2,SINGLE_DEPTH_NESTING ); -} -static void task_numa_group (struct task_struct *p,int cpupid,int flags,int *priv) {
kernel/sched/sched.h +18 -0
@@ -1249,6 +1249,24 @@ static inline void double_unlock_balance(struct rq *this_rq, struct rq *busiest)lock_set_subclass (&this_rq->lock.dep_map,0 , _RET_IP_); } +static inline void double_lock (spinlock_t *l1,spinlock_t *l2) +{ +if (l1 > l2) +swap (l1, l2); + +spin_lock (l1); +spin_lock_nested (l2,SINGLE_DEPTH_NESTING ); +} + +static inline void double_raw_lock (raw_spinlock_t *l1,raw_spinlock_t *l2) +{ +if (l1 > l2) +swap (l1, l2); + +raw_spin_lock (l1); +raw_spin_lock_nested (l2,SINGLE_DEPTH_NESTING ); +} +/* * double_rq_lock - safely lock two runqueues *
Rik van Riel
Reviewed-byRe: [tip:sched/core] sched/numa: Introduce migrate_swap()
Quoted 31 lines · Peter Zijlstra
Quoted 1 line · tip-bot for Peter Zijlstra
sched/numa: Introduce
migrate_swap()Thanks to Rik for writing the Changelog!
---
From: Peter Zijlstra <[email protected]>
Subject: sched: Fix race inmigrate_swap_stopThere is a subtle race in
migrate_swap, when task P, on CPU A, decides to swap places with task T, on CPU B.Task P:
- callmigrate_swap
Task T:
- go to sleep, removing itself from the runqueue
Task P:
- double lock the runqueues on CPU A & B
Task T:
- get woken up, place itself on the runqueue of CPU C
Task P:
- see that task T is on a runqueue, and pretend to remove it from the runqueue on CPU BNow CPUs B & C both have corrupted scheduler data structures.
This patch fixes it, by holding the
pi_lockfor both of the tasks involved in the migrate swap. This prevents task T from waking up, and placing itself onto another runqueue, until aftermigrate_swaphas released all locks.
Quoted 6 lines
This means that, when
migrate_swapchecks, task T will be either on the runqueue where it was originally seen, or not on any runqueue at all.Migrate_swapdeals correctly with of those cases.Signed-off-by Peter Zijlstra <[email protected]>Tested-by Joe Mario <[email protected]>
Re: [tip:sched/core] sched/numa: Introduce migrate_swap()
In reply to Peter ZijlstraQuoted 31 lines · Peter Zijlstra
Quoted 1 line · tip-bot for Peter Zijlstra
sched/numa: Introduce
migrate_swap()Thanks to Rik for writing the Changelog!
---
From: Peter Zijlstra <[email protected]>
Subject: sched: Fix race inmigrate_swap_stopThere is a subtle race in
migrate_swap, when task P, on CPU A, decides to swap places with task T, on CPU B.Task P:
- callmigrate_swap
Task T:
- go to sleep, removing itself from the runqueue
Task P:
- double lock the runqueues on CPU A & B
Task T:
- get woken up, place itself on the runqueue of CPU C
Task P:
- see that task T is on a runqueue, and pretend to remove it from the runqueue on CPU BNow CPUs B & C both have corrupted scheduler data structures.
This patch fixes it, by holding the
pi_lockfor both of the tasks involved in the migrate swap. This prevents task T from waking up, and placing itself onto another runqueue, until aftermigrate_swaphas released all locks.
Quoted 6 lines
This means that, when
migrate_swapchecks, task T will be either on the runqueue where it was originally seen, or not on any runqueue at all.Migrate_swapdeals correctly with of those cases.Signed-off-by Peter Zijlstra <[email protected]>Tested-by Joe Mario <[email protected]>
Signature · 2 lines
Mel Gorman SUSE Labs
tip-bot for Peter Zijlstra
merge bot[tip:sched/core] sched: Fix race in migrate_swap_stop()
In reply to Peter Zijlstrasched: Fix race in migrate_swap_stop()
Commit-ID: 746023159c40c523b08a3bc3d213dac212385895 Gitweb: http://git.kernel.org/tip/746023159c40c523b08a3bc3d213dac212385895 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 10 Oct 2013 20:17:22 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 16 Oct 2013 14:22:14 +0200
sched: Fix race in migrate_swap_stop()
There is a subtle race in migrate_swap, when task P, on CPU A, decides to swap places with task T, on CPU B.
Task P:
- call migrate_swap
Task T:
- go to sleep, removing itself from the runqueue
Task P:
- double lock the runqueues on CPU A & B
Task T:
- get woken up, place itself on the runqueue of CPU C
Task P:
- see that task T is on a runqueue, and pretend to remove it from the runqueue on CPU B
Now CPUs B & C both have corrupted scheduler data structures.
This patch fixes it, by holding the pi_lock for both of the tasks involved in the migrate swap. This prevents task T from waking up, and placing itself onto another runqueue, until after migrate_swap has released all locks.
This means that, when migrate_swap checks, task T will be either on the runqueue where it was originally seen, or not on any runqueue at all. Migrate_swap deals correctly with of those cases.
--- kernel/sched/core.c | 4 ++++ kernel/sched/fair.c | 9 --------- kernel/sched/sched.h | 18 ++++++++++++++++++ 3 files changed, 22 insertions(+), 9 deletions(-)
kernel/sched/core.c +4 -0
@@ -1049,6 +1049,8 @@ static int migrate_swap_stop(void *data) src_rq =cpu_rq (arg->src_cpu); dst_rq =cpu_rq (arg->dst_cpu); +double_raw_lock (&arg->src_task->pi_lock, + &arg->dst_task->pi_lock);double_rq_lock (src_rq, dst_rq);if (task_cpu (arg->dst_task) != arg->dst_cpu) goto unlock;
@@ -1069,6 +1071,8 @@ static int migrate_swap_stop(void *data) unlock:double_rq_unlock (src_rq, dst_rq); +raw_spin_unlock (&arg->dst_task->pi_lock); +raw_spin_unlock (&arg->src_task->pi_lock);return ret; }
kernel/sched/fair.c +0 -9
@@ -1448,15 +1448,6 @@ static inline void put_numa_group(struct numa_group *grp)kfree_rcu (grp, rcu); } -static void double_lock (spinlock_t *l1,spinlock_t *l2) -{ -if (l1 > l2) -swap (l1, l2); - -spin_lock (l1); -spin_lock_nested (l2,SINGLE_DEPTH_NESTING ); -} -static void task_numa_group (struct task_struct *p,int cpupid,int flags,int *priv) {
kernel/sched/sched.h +18 -0
@@ -1249,6 +1249,24 @@ static inline void double_unlock_balance(struct rq *this_rq, struct rq *busiest)lock_set_subclass (&this_rq->lock.dep_map,0 , _RET_IP_); } +static inline void double_lock (spinlock_t *l1,spinlock_t *l2) +{ +if (l1 > l2) +swap (l1, l2); + +spin_lock (l1); +spin_lock_nested (l2,SINGLE_DEPTH_NESTING ); +} + +static inline void double_raw_lock (raw_spinlock_t *l1,raw_spinlock_t *l2) +{ +if (l1 > l2) +swap (l1, l2); + +raw_spin_lock (l1); +raw_spin_lock_nested (l2,SINGLE_DEPTH_NESTING ); +} +/* * double_rq_lock - safely lock two runqueues *
Mel Gorman
reviewerPATCH 39/63 sched: numa: Use a system-wide search to find swap/migration candidates
In reply to Mel GormanThis patch implements a system-wide search for swap/migration candidates based on total NUMA hinting faults. It has a balance limit, however it doesn't properly consider total node balance.
In the old scheme a task selected a preferred node based on the highest number of private faults recorded on the node. In this scheme, the preferred node is based on the total number of faults. If the preferred node for a task changes then task_numa_migrate will search the whole system looking for tasks to swap with that would improve both the overall compute balance and minimise the expected number of remote NUMA hinting faults.
Not there is no guarantee that the node the source task is placed on by task_numa_migrate() has any relationship to the newly selected task->numa_preferred_nid due to compute overloading.
--- kernel/sched/core.c | 4 + kernel/sched/fair.c | 253 ++++++++++++++++++++++++++++++++++++--------------- kernel/sched/sched.h | 13 +++ 3 files changed, 199 insertions(+), 71 deletions(-)
kernel/sched/core.c +4 -0
@@ -5241,6 +5241,7 @@ static void destroy_sched_domains(struct sched_domain *sd, int cpu) DEFINE_PER_CPU(struct sched_domain *, sd_llc);DEFINE_PER_CPU (int, sd_llc_size);DEFINE_PER_CPU (int, sd_llc_id); +DEFINE_PER_CPU(struct sched_domain *, sd_numa);static void update_top_cache_domain (int cpu) {
@@ -5257,6 +5258,9 @@ static void update_top_cache_domain(int cpu)rcu_assign_pointer (per_cpu (sd_llc, cpu), sd);per_cpu (sd_llc_size, cpu) = size;per_cpu (sd_llc_id, cpu) = id; + + sd =lowest_flag_domain (cpu,SD_NUMA ); +rcu_assign_pointer (per_cpu (sd_numa, cpu), sd); }/*
kernel/sched/fair.c +182 -71
320 lines not on this page · raw message
kernel/sched/sched.h +13 -0
@@ -607,9 +607,22 @@ static inline struct sched_domain *highest_flag_domain(int cpu, int flag)return hsd; } +static inline struct sched_domain *lowest_flag_domain (int cpu,int flag) +{ +struct sched_domain *sd; + +for_each_domain (cpu, sd) { +if (sd->flags & flag) +break ; + } + +return sd; +} + DECLARE_PER_CPU(struct sched_domain *, sd_llc);DECLARE_PER_CPU (int, sd_llc_size);DECLARE_PER_CPU (int, sd_llc_id); +DECLARE_PER_CPU(struct sched_domain *, sd_numa);struct sched_group_power {atomic_t ref;
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 39/63] sched: numa: Use a system-wide search to find swap/migration candidates
Quoted 11 lines · Mel Gorman
This patch implements a system-wide search for swap/migration candidates based on total NUMA hinting faults. It has a balance limit, however it doesn't properly consider total node balance.
In the old scheme a task selected a preferred node based on the highest number of private faults recorded on the node. In this scheme, the preferred node is based on the total number of faults. If the preferred node for a task changes then
task_numa_migratewill search the whole system looking for tasks to swap with that would improve both the overall compute balance and minimise the expected number of remote NUMA hinting faults.
Quoted 7 lines
Not there is no guarantee that the node the source task is placed on by
task_numa_migrate()has any relationship to the newly selectedtask->numa_preferred_niddue to compute overloading.[[email protected]: Do not swap with tasks that cannot run on source cpu]Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Use a system-wide search to find swap/migration candidates
In reply to Mel Gormansched/numa: Use a system-wide search to find swap/migration candidates
Commit-ID: fb13c7ee0ed387bd6bec4b4024a4d49b1bd504f1 Gitweb: http://git.kernel.org/tip/fb13c7ee0ed387bd6bec4b4024a4d49b1bd504f1 Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:17 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 14:47:25 +0200
sched/numa: Use a system-wide search to find swap/migration candidates
This patch implements a system-wide search for swap/migration candidates based on total NUMA hinting faults. It has a balance limit, however it doesn't properly consider total node balance.
In the old scheme a task selected a preferred node based on the highest number of private faults recorded on the node. In this scheme, the preferred node is based on the total number of faults. If the preferred node for a task changes then task_numa_migrate will search the whole system looking for tasks to swap with that would improve both the overall compute balance and minimise the expected number of remote NUMA hinting faults.
Not there is no guarantee that the node the source task is placed on by task_numa_migrate() has any relationship to the newly selected task->numa_preferred_nid due to compute overloading.
--- kernel/sched/core.c | 4 + kernel/sched/fair.c | 253 ++++++++++++++++++++++++++++++++++++--------------- kernel/sched/sched.h | 13 +++ 3 files changed, 199 insertions(+), 71 deletions(-)
kernel/sched/core.c +4 -0
@@ -5236,6 +5236,7 @@ static void destroy_sched_domains(struct sched_domain *sd, int cpu) DEFINE_PER_CPU(struct sched_domain *, sd_llc);DEFINE_PER_CPU (int, sd_llc_size);DEFINE_PER_CPU (int, sd_llc_id); +DEFINE_PER_CPU(struct sched_domain *, sd_numa);static void update_top_cache_domain (int cpu) {
@@ -5252,6 +5253,9 @@ static void update_top_cache_domain(int cpu)rcu_assign_pointer (per_cpu (sd_llc, cpu), sd);per_cpu (sd_llc_size, cpu) = size;per_cpu (sd_llc_id, cpu) = id; + + sd =lowest_flag_domain (cpu,SD_NUMA ); +rcu_assign_pointer (per_cpu (sd_numa, cpu), sd); }/*
kernel/sched/fair.c +182 -71
320 lines not on this page · raw message
kernel/sched/sched.h +13 -0
@@ -610,9 +610,22 @@ static inline struct sched_domain *highest_flag_domain(int cpu, int flag)return hsd; } +static inline struct sched_domain *lowest_flag_domain (int cpu,int flag) +{ +struct sched_domain *sd; + +for_each_domain (cpu, sd) { +if (sd->flags & flag) +break ; + } + +return sd; +} + DECLARE_PER_CPU(struct sched_domain *, sd_llc);DECLARE_PER_CPU (int, sd_llc_size);DECLARE_PER_CPU (int, sd_llc_id); +DECLARE_PER_CPU(struct sched_domain *, sd_numa);struct sched_group_power {atomic_t ref;
Mel Gorman
reviewerPATCH 40/63 sched: numa: Favor placing a task on the preferred node
In reply to Mel GormanA tasks preferred node is selected based on the number of faults recorded for a node but the actual task_numa_migate() conducts a global search regardless of the preferred nid. This patch checks if the preferred nid has capacity and if so, searches for a CPU within that node. This avoids a global search when the preferred node is not overloaded.
--- kernel/sched/fair.c | 54 ++++++++++++++++++++++++++++++++++------------------- 1 file changed, 35 insertions(+), 19 deletions(-)
kernel/sched/fair.c +35 -19
@@ -1052,6 +1052,20 @@ unlock:rcu_read_unlock (); } +static void task_numa_find_cpu (struct task_numa_env *env,long imp) +{ +int cpu; + +for_each_cpu (cpu,cpumask_of_node (env->dst_nid)) { +/* Skip this CPU if the source task cannot migrate */ +if (!cpumask_test_cpu (cpu,tsk_cpus_allowed (env->p))) +continue ; + + env->dst_cpu = cpu; +task_numa_compare (env, imp); + } +} +static int task_numa_migrate (struct task_struct *p) {struct task_numa_env env = {
@@ -1068,7 +1082,8 @@ static int task_numa_migrate(struct task_struct *p) };struct sched_domain *sd;unsigned long faults; -int nid, cpu, ret; +int nid, ret; +long imp;/* * Pick the lowest SD_NUMA domain, as that would have the smallest
@@ -1085,28 +1100,29 @@ static int task_numa_migrate(struct task_struct *p) faults =task_faults (p, env.src_nid);update_numa_stats (&env.src_stats, env.src_nid); + env.dst_nid = p->numa_preferred_nid; + imp =task_faults (env.p, env.dst_nid) - faults; +update_numa_stats (&env.dst_stats, env.dst_nid); -/* Find an alternative node with relatively better statistics */ -for_each_online_node (nid) { -long imp; - -if (nid == env.src_nid) -continue ; - -/* Only consider nodes that recorded more faults */ - imp =task_faults (p, nid) - faults; -if (imp <0 ) -continue ; +/* +* If the preferred nid has capacity then use it. Otherwise find an +* alternative node with relatively better statistics. +*/ +if (env.dst_stats.has_capacity) { +task_numa_find_cpu (&env, imp); + }else { +for_each_online_node (nid) { +if (nid == env.src_nid || nid == p->numa_preferred_nid) +continue ; - env.dst_nid = nid; -update_numa_stats (&env.dst_stats, env.dst_nid); -for_each_cpu (cpu,cpumask_of_node (nid)) { -/* Skip this CPU if the source task cannot migrate */ -if (!cpumask_test_cpu (cpu,tsk_cpus_allowed (p))) +/* Only consider nodes that recorded more faults */ + imp =task_faults (env.p, nid) - faults; +if (imp <0 )continue ; - env.dst_cpu = cpu; -task_numa_compare (&env, imp); + env.dst_nid = nid; +update_numa_stats (&env.dst_stats, env.dst_nid); +task_numa_find_cpu (&env, imp); } }
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 40/63] sched: numa: Favor placing a task on the preferred node
Quoted 8 lines · Mel Gorman
A tasks preferred node is selected based on the number of faults recorded for a node but the actual
task_numa_migate()conducts a global search regardless of the preferred nid. This patch checks if the preferred nid has capacity and if so, searches for a CPU within that node. This avoids a global search when the preferred node is not overloaded.Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Mel Gorman
merge bot[tip:sched/core] sched/numa: Favor placing a task on the preferred node
In reply to Mel Gormansched/numa: Favor placing a task on the preferred node
Commit-ID: 2c8a50aa873a7e1d6cc0913362051ff9912dc6ca Gitweb: http://git.kernel.org/tip/2c8a50aa873a7e1d6cc0913362051ff9912dc6ca Author: Mel Gorman <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:18 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 14:47:41 +0200
sched/numa: Favor placing a task on the preferred node
A tasks preferred node is selected based on the number of faults recorded for a node but the actual task_numa_migate() conducts a global search regardless of the preferred nid. This patch checks if the preferred nid has capacity and if so, searches for a CPU within that node. This avoids a global search when the preferred node is not overloaded.
--- kernel/sched/fair.c | 54 ++++++++++++++++++++++++++++++++++------------------- 1 file changed, 35 insertions(+), 19 deletions(-)
kernel/sched/fair.c +35 -19
@@ -1052,6 +1052,20 @@ unlock:rcu_read_unlock (); } +static void task_numa_find_cpu (struct task_numa_env *env,long imp) +{ +int cpu; + +for_each_cpu (cpu,cpumask_of_node (env->dst_nid)) { +/* Skip this CPU if the source task cannot migrate */ +if (!cpumask_test_cpu (cpu,tsk_cpus_allowed (env->p))) +continue ; + + env->dst_cpu = cpu; +task_numa_compare (env, imp); + } +} +static int task_numa_migrate (struct task_struct *p) {struct task_numa_env env = {
@@ -1068,7 +1082,8 @@ static int task_numa_migrate(struct task_struct *p) };struct sched_domain *sd;unsigned long faults; -int nid, cpu, ret; +int nid, ret; +long imp;/* * Pick the lowest SD_NUMA domain, as that would have the smallest
@@ -1085,28 +1100,29 @@ static int task_numa_migrate(struct task_struct *p) faults =task_faults (p, env.src_nid);update_numa_stats (&env.src_stats, env.src_nid); + env.dst_nid = p->numa_preferred_nid; + imp =task_faults (env.p, env.dst_nid) - faults; +update_numa_stats (&env.dst_stats, env.dst_nid); -/* Find an alternative node with relatively better statistics */ -for_each_online_node (nid) { -long imp; - -if (nid == env.src_nid) -continue ; - -/* Only consider nodes that recorded more faults */ - imp =task_faults (p, nid) - faults; -if (imp <0 ) -continue ; +/* +* If the preferred nid has capacity then use it. Otherwise find an +* alternative node with relatively better statistics. +*/ +if (env.dst_stats.has_capacity) { +task_numa_find_cpu (&env, imp); + }else { +for_each_online_node (nid) { +if (nid == env.src_nid || nid == p->numa_preferred_nid) +continue ; - env.dst_nid = nid; -update_numa_stats (&env.dst_stats, env.dst_nid); -for_each_cpu (cpu,cpumask_of_node (nid)) { -/* Skip this CPU if the source task cannot migrate */ -if (!cpumask_test_cpu (cpu,tsk_cpus_allowed (p))) +/* Only consider nodes that recorded more faults */ + imp =task_faults (env.p, nid) - faults; +if (imp <0 )continue ; - env.dst_cpu = cpu; -task_numa_compare (&env, imp); + env.dst_nid = nid; +update_numa_stats (&env.dst_stats, env.dst_nid); +task_numa_find_cpu (&env, imp); } }
Mel Gorman
reviewerPATCH 41/63 sched: numa: fix placement of workloads spread across multiple nodes
In reply to Mel GormanFrom: Rik van Riel <[email protected]>
The load balancer will spread workloads across multiple NUMA nodes, in order to balance the load on the system. This means that sometimes a task's preferred node has available capacity, but moving the task there will not succeed, because that would create too large an imbalance.
In that case, other NUMA nodes need to be considered.
--- kernel/sched/fair.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
kernel/sched/fair.c +5 -6
@@ -1104,13 +1104,12 @@ static int task_numa_migrate(struct task_struct *p) imp =task_faults (env.p, env.dst_nid) - faults;update_numa_stats (&env.dst_stats, env.dst_nid); -/* -* If the preferred nid has capacity then use it. Otherwise find an -* alternative node with relatively better statistics. -*/ -if (env.dst_stats.has_capacity) { +/* If the preferred nid has capacity, try to use it. */ +if (env.dst_stats.has_capacity)task_numa_find_cpu (&env, imp); - }else { + +/* No space available on the preferred nid. Look elsewhere. */ +if (env.best_cpu ==-1 ) {for_each_online_node (nid) {if (nid == env.src_nid || nid == p->numa_preferred_nid)continue ;
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Rik van Riel
merge bot[tip:sched/core] sched/numa: Fix placement of workloads spread across multiple nodes
sched/numa: Fix placement of workloads spread across multiple nodes
Commit-ID: e1dda8a797b59d7ec4b17e393152ec3273a552d5 Gitweb: http://git.kernel.org/tip/e1dda8a797b59d7ec4b17e393152ec3273a552d5 Author: Rik van Riel <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:19 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 14:47:43 +0200
sched/numa: Fix placement of workloads spread across multiple nodes
The load balancer will spread workloads across multiple NUMA nodes, in order to balance the load on the system. This means that sometimes a task's preferred node has available capacity, but moving the task there will not succeed, because that would create too large an imbalance.
In that case, other NUMA nodes need to be considered.
--- kernel/sched/fair.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
kernel/sched/fair.c +5 -6
@@ -1104,13 +1104,12 @@ static int task_numa_migrate(struct task_struct *p) imp =task_faults (env.p, env.dst_nid) - faults;update_numa_stats (&env.dst_stats, env.dst_nid); -/* -* If the preferred nid has capacity then use it. Otherwise find an -* alternative node with relatively better statistics. -*/ -if (env.dst_stats.has_capacity) { +/* If the preferred nid has capacity, try to use it. */ +if (env.dst_stats.has_capacity)task_numa_find_cpu (&env, imp); - }else { + +/* No space available on the preferred nid. Look elsewhere. */ +if (env.best_cpu ==-1 ) {for_each_online_node (nid) {if (nid == env.src_nid || nid == p->numa_preferred_nid)continue ;
Mel Gorman
reviewerPATCH 42/63 mm: numa: Change page last {nid,pid} into {cpu,pid}
In reply to Mel GormanFrom: Peter Zijlstra <[email protected]>
Change the per page last fault tracking to use cpu,pid instead of nid,pid. This will allow us to try and lookup the alternate task more easily. Note that even though it is the cpu that is store in the page flags that the mpol_misplaced decision is still based on the node.
--- include/linux/mm.h | 90 ++++++++++++++++++++++----------------- include/linux/mm_types.h | 4 +- include/linux/page-flags-layout.h | 22 +++++----- kernel/bounds.c | 4 ++ kernel/sched/fair.c | 6 +-- mm/huge_memory.c | 8 ++-- mm/memory.c | 16 +++---- mm/mempolicy.c | 16 ++++--- mm/migrate.c | 4 +- mm/mm_init.c | 18 ++++---- mm/mmzone.c | 14 +++--- mm/mprotect.c | 28 ++++++------ mm/page_alloc.c | 4 +- 13 files changed, 125 insertions(+), 109 deletions(-)
include/linux/mm.h +50 -40
@@ -581,11 +581,11 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)* sets it, so none of the operations on it need to be atomic. */ -/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_NIDPID] | ... | FLAGS | */ +/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_CPUPID] | ... | FLAGS | */ #define SECTIONS_PGOFF ((sizeof(unsigned long)*8) - SECTIONS_WIDTH)#define NODES_PGOFF (SECTIONS_PGOFF - NODES_WIDTH)#define ZONES_PGOFF (NODES_PGOFF - ZONES_WIDTH) -#define LAST_NIDPID_PGOFF (ZONES_PGOFF - LAST_NIDPID_WIDTH) +#define LAST_CPUPID_PGOFF (ZONES_PGOFF - LAST_CPUPID_WIDTH)/* * Define the bit shifts to access each section. For non-existent
@@ -595,7 +595,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)#define SECTIONS_PGSHIFT (SECTIONS_PGOFF * (SECTIONS_WIDTH != 0))#define NODES_PGSHIFT (NODES_PGOFF * (NODES_WIDTH != 0))#define ZONES_PGSHIFT (ZONES_PGOFF * (ZONES_WIDTH != 0)) -#define LAST_NIDPID_PGSHIFT (LAST_NIDPID_PGOFF * (LAST_NIDPID_WIDTH != 0)) +#define LAST_CPUPID_PGSHIFT (LAST_CPUPID_PGOFF * (LAST_CPUPID_WIDTH != 0))/* NODE:ZONE or SECTION:ZONE is used to ID a zone for the buddy allocator */ #ifdef NODE_NOT_IN_PAGE_FLAGS
@@ -617,7 +617,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)#define ZONES_MASK ((1UL << ZONES_WIDTH) - 1)#define NODES_MASK ((1UL << NODES_WIDTH) - 1)#define SECTIONS_MASK ((1UL << SECTIONS_WIDTH) - 1) -#define LAST_NIDPID_MASK ((1UL << LAST_NIDPID_WIDTH) - 1) +#define LAST_CPUPID_MASK ((1UL << LAST_CPUPID_WIDTH) - 1)#define ZONEID_MASK ((1UL << ZONEID_SHIFT) - 1)static inline enum zone_type page_zonenum (const struct page *page)
@@ -661,96 +661,106 @@ static inline int page_to_nid(const struct page *page)#endif #ifdef CONFIG_NUMA_BALANCING -static inline int nid_pid_to_nidpid (int nid,int pid) +static inline int cpu_pid_to_cpupid (int cpu,int pid) { -return ((nid &LAST__NID_MASK ) <<LAST__PID_SHIFT ) | (pid &LAST__PID_MASK ); +return ((cpu &LAST__CPU_MASK ) <<LAST__PID_SHIFT ) | (pid &LAST__PID_MASK ); } -static inline int nidpid_to_pid (int nidpid) +static inline int cpupid_to_pid (int cpupid) { -return nidpid &LAST__PID_MASK ; +return cpupid &LAST__PID_MASK ; } -static inline int nidpid_to_nid (int nidpid) +static inline int cpupid_to_cpu (int cpupid) { -return (nidpid >>LAST__PID_SHIFT ) &LAST__NID_MASK ; +return (cpupid >>LAST__PID_SHIFT ) &LAST__CPU_MASK ; } -static inline bool nidpid_pid_unset (int nidpid) +static inline int cpupid_to_nid (int cpupid) { -return nidpid_to_pid (nidpid) == (-1 &LAST__PID_MASK ); +return cpu_to_node (cpupid_to_cpu (cpupid)); } -static inline bool nidpid_nid_unset (int nidpid) +static inline bool cpupid_pid_unset (int cpupid) { -return nidpid_to_nid (nidpid) == (-1 &LAST__NID_MASK ); +return cpupid_to_pid (cpupid) == (-1 &LAST__PID_MASK ); } -#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS -static inline int page_nidpid_xchg_last (struct page *page,int nid) +static inline bool cpupid_cpu_unset (int cpupid) { -return xchg (&page->_last_nidpid, nid); +return cpupid_to_cpu (cpupid) == (-1 &LAST__CPU_MASK ); } -static inline int page_nidpid_last (struct page *page) +#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS +static inline int page_cpupid_xchg_last (struct page *page,int cpupid) { -return page->_last_nidpid; +return xchg (&page->_last_cpupid, cpupid); } -static inline void page_nidpid_reset_last (struct page *page) + +static inline int page_cpupid_last (struct page *page) +{ +return page->_last_cpupid; +} +static inline void page_cpupid_reset_last (struct page *page) { - page->_last_nidpid =-1 ; + page->_last_cpupid =-1 ; }#else -static inline int page_nidpid_last (struct page *page) +static inline int page_cpupid_last (struct page *page) { -return (page->flags >>LAST_NIDPID_PGSHIFT ) &LAST_NIDPID_MASK ; +return (page->flags >>LAST_CPUPID_PGSHIFT ) &LAST_CPUPID_MASK ; } -extern int page_nidpid_xchg_last (struct page *page,int nidpid); +extern int page_cpupid_xchg_last (struct page *page,int cpupid); -static inline void page_nidpid_reset_last (struct page *page) +static inline void page_cpupid_reset_last (struct page *page) { -int nidpid = (1 <<LAST_NIDPID_SHIFT ) -1 ; +int cpupid = (1 <<LAST_CPUPID_SHIFT ) -1 ; - page->flags &= ~(LAST_NIDPID_MASK <<LAST_NIDPID_PGSHIFT ); - page->flags |= (nidpid &LAST_NIDPID_MASK ) <<LAST_NIDPID_PGSHIFT ; + page->flags &= ~(LAST_CPUPID_MASK <<LAST_CPUPID_PGSHIFT ); + page->flags |= (cpupid &LAST_CPUPID_MASK ) <<LAST_CPUPID_PGSHIFT ; } -#endif /* LAST_NIDPID_NOT_IN_PAGE_FLAGS */ -#else -static inline int page_nidpid_xchg_last (struct page *page,int nidpid) +#endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */ +#else /* !CONFIG_NUMA_BALANCING */ +static inline int page_cpupid_xchg_last (struct page *page,int cpupid) { -return page_to_nid (page); +return page_to_nid (page);/* XXX */ } -static inline int page_nidpid_last (struct page *page) +static inline int page_cpupid_last (struct page *page) { -return page_to_nid (page); +return page_to_nid (page);/* XXX */ } -static inline int nidpid_to_nid (int nidpid) +static inline int cpupid_to_nid (int cpupid) {return -1 ; } -static inline int nidpid_to_pid (int nidpid) +static inline int cpupid_to_pid (int cpupid) {return -1 ; } -static inline int nid_pid_to_nidpid (int nid,int pid) +static inline int cpupid_to_cpu (int cpupid) {return -1 ; } -static inline bool nidpid_pid_unset (int nidpid) +static inline int cpu_pid_to_cpupid (int nid,int pid) +{ +return -1 ; +} + +static inline bool cpupid_pid_unset (int cpupid) {return 1 ; } -static inline void page_nidpid_reset_last (struct page *page) +static inline void page_cpupid_reset_last (struct page *page) { } -#endif +#endif /* CONFIG_NUMA_BALANCING */ static inline struct zone *page_zone (const struct page *page) {
include/linux/mm_types.h +2 -2
@@ -174,8 +174,8 @@ struct page {void *shadow;#endif -#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS -int _last_nidpid; +#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS +int _last_cpupid;#endif }/*
include/linux/page-flags-layout.h +11 -11
@@ -39,9 +39,9 @@* lookup is necessary. * * No sparsemem or sparsemem vmemmap: | NODE | ZONE | ... | FLAGS | -* " plus space for last_nidpid: | NODE | ZONE | LAST_NIDPID ... | FLAGS | +* " plus space for last_cpupid: | NODE | ZONE | LAST_CPUPID ... | FLAGS | * classic sparse with space for node:| SECTION | NODE | ZONE | ... | FLAGS | -* " plus space for last_nidpid: | SECTION | NODE | ZONE | LAST_NIDPID ... | FLAGS | +* " plus space for last_cpupid: | SECTION | NODE | ZONE | LAST_CPUPID ... | FLAGS | * classic sparse no space for node: | SECTION | ZONE | ... | FLAGS | */ #if defined(CONFIG_SPARSEMEM ) && !defined(CONFIG_SPARSEMEM_VMEMMAP )
@@ -65,18 +65,18 @@#define LAST__PID_SHIFT 8#define LAST__PID_MASK ((1 << LAST__PID_SHIFT)-1) -#define LAST__NID_SHIFT NODES_SHIFT -#define LAST__NID_MASK ((1 << LAST__NID_SHIFT)-1) +#define LAST__CPU_SHIFT NR_CPUS_BITS +#define LAST__CPU_MASK ((1 << LAST__CPU_SHIFT)-1) -#define LAST_NIDPID_SHIFT (LAST__PID_SHIFT+LAST__NID_SHIFT) +#define LAST_CPUPID_SHIFT (LAST__PID_SHIFT+LAST__CPU_SHIFT)#else -#define LAST_NIDPID_SHIFT 0 +#define LAST_CPUPID_SHIFT 0#endif -#if SECTIONS_WIDTH +ZONES_WIDTH +NODES_SHIFT +LAST_NIDPID_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS -#define LAST_NIDPID_WIDTH LAST_NIDPID_SHIFT +#if SECTIONS_WIDTH +ZONES_WIDTH +NODES_SHIFT +LAST_CPUPID_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS +#define LAST_CPUPID_WIDTH LAST_CPUPID_SHIFT#else -#define LAST_NIDPID_WIDTH 0 +#define LAST_CPUPID_WIDTH 0#endif /*
@@ -87,8 +87,8 @@#define NODE_NOT_IN_PAGE_FLAGS #endif -#if defined(CONFIG_NUMA_BALANCING ) &&LAST_NIDPID_WIDTH ==0 -#define LAST_NIDPID_NOT_IN_PAGE_FLAGS +#if defined(CONFIG_NUMA_BALANCING ) &&LAST_CPUPID_WIDTH ==0 +#define LAST_CPUPID_NOT_IN_PAGE_FLAGS #endif #endif /* _LINUX_PAGE_FLAGS_LAYOUT */
kernel/bounds.c +4 -0
@@ -10,6 +10,7 @@#include <linux/mmzone.h> #include <linux/kbuild.h> #include <linux/page_cgroup.h> +#include <linux/log2.h> void foo (void ) {
@@ -17,5 +18,8 @@ void foo(void)DEFINE (NR_PAGEFLAGS , __NR_PAGEFLAGS);DEFINE (MAX_NR_ZONES , __MAX_NR_ZONES);DEFINE (NR_PCG_FLAGS , __NR_PCG_FLAGS); +#ifdef CONFIG_SMP +DEFINE (NR_CPUS_BITS ,ilog2 (CONFIG_NR_CPUS )); +#endif /* End of constants */ }
kernel/sched/fair.c +3 -3
@@ -1210,7 +1210,7 @@ static void task_numa_placement(struct task_struct *p)/* * Got a PROT_NONE fault for a page on @node. */ -void task_numa_fault (int last_nidpid,int node,int pages,bool migrated) +void task_numa_fault (int last_cpupid,int node,int pages,bool migrated) {struct task_struct *p = current;int priv;
@@ -1226,8 +1226,8 @@ void task_numa_fault(int last_nidpid, int node, int pages, bool migrated)* First accesses are treated as private, otherwise consider accesses * to be private if the accessing pid has not changed */ -if (!nidpid_pid_unset (last_nidpid)) - priv = ((p->pid &LAST__PID_MASK ) ==nidpid_to_pid (last_nidpid)); +if (!cpupid_pid_unset (last_cpupid)) + priv = ((p->pid &LAST__PID_MASK ) ==cpupid_to_pid (last_cpupid));else priv =1 ;
mm/huge_memory.c +4 -4
@@ -1282,7 +1282,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct page *page;unsigned long haddr = addr &HPAGE_PMD_MASK ;int page_nid =-1 , this_nid =numa_node_id (); -int target_nid, last_nidpid =-1 ; +int target_nid, last_cpupid =-1 ;bool page_locked;bool migrated = false;
@@ -1293,7 +1293,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, page =pmd_page (pmd);BUG_ON (is_huge_zero_page (page)); page_nid =page_to_nid (page); - last_nidpid =page_nidpid_last (page); + last_cpupid =page_cpupid_last (page);count_vm_numa_event (NUMA_HINT_FAULTS );if (page_nid == this_nid)count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL );
@@ -1362,7 +1362,7 @@ out:page_unlock_anon_vma_read (anon_vma);if (page_nid !=-1 ) -task_numa_fault (last_nidpid, page_nid,HPAGE_PMD_NR , migrated); +task_numa_fault (last_cpupid, page_nid,HPAGE_PMD_NR , migrated);return 0 ; }
@@ -1682,7 +1682,7 @@ static void __split_huge_page_refcount(struct page *page, page_tail->mapping = page->mapping; page_tail->index = page->index + i; -page_nidpid_xchg_last (page_tail,page_nidpid_last (page)); +page_cpupid_xchg_last (page_tail,page_cpupid_last (page));BUG_ON (!PageAnon (page_tail));BUG_ON (!PageUptodate (page_tail));
mm/memory.c +8 -8
@@ -69,8 +69,8 @@#include "internal.h" -#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS -#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_nidpid. +#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS +#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_cpupid.#endif #ifndef CONFIG_NEED_MULTIPLE_NODES
@@ -3536,7 +3536,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct page *page =NULL ;spinlock_t *ptl;int page_nid =-1 ; -int last_nidpid; +int last_cpupid;int target_nid;bool migrated = false;
@@ -3567,7 +3567,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }BUG_ON (is_zero_pfn (page_to_pfn (page))); - last_nidpid =page_nidpid_last (page); + last_cpupid =page_cpupid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (ptep, ptl);
@@ -3583,7 +3583,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, out:if (page_nid !=-1 ) -task_numa_fault (last_nidpid, page_nid,1 , migrated); +task_numa_fault (last_cpupid, page_nid,1 , migrated);return 0 ; }
@@ -3598,7 +3598,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,unsigned long offset;spinlock_t *ptl;bool numa = false; -int last_nidpid; +int last_cpupid;spin_lock (&mm->page_table_lock); pmd = *pmdp;
@@ -3643,7 +3643,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,if (unlikely (!page))continue ; - last_nidpid =page_nidpid_last (page); + last_cpupid =page_cpupid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (pte, ptl);
@@ -3656,7 +3656,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }if (page_nid !=-1 ) -task_numa_fault (last_nidpid, page_nid,1 , migrated); +task_numa_fault (last_cpupid, page_nid,1 , migrated); pte =pte_offset_map_lock (mm, pmdp, addr, &ptl); }
mm/mempolicy.c +9 -7
@@ -2324,6 +2324,8 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned longstruct zone *zone;int curnid =page_to_nid (page);unsigned long pgoff; +int thiscpu =raw_smp_processor_id (); +int thisnid =cpu_to_node (thiscpu);int polnid =-1 ;int ret =-1 ;
@@ -2372,11 +2374,11 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long/* Migrate the page towards the node whose CPU is referencing it */ if (pol->flags &MPOL_F_MORON ) { -int last_nidpid; -int this_nidpid; +int last_cpupid; +int this_cpupid; - polnid =numa_node_id (); - this_nidpid =nid_pid_to_nidpid (polnid, current->pid); + polnid = thisnid; + this_cpupid =cpu_pid_to_cpupid (thiscpu, current->pid);/* * Multi-stage node selection is used in conjunction
@@ -2399,8 +2401,8 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long* it less likely we act on an unlikely task<->page * relation. */ - last_nidpid =page_nidpid_xchg_last (page, this_nidpid); -if (!nidpid_pid_unset (last_nidpid) &&nidpid_to_nid (last_nidpid) != polnid) + last_cpupid =page_cpupid_xchg_last (page, this_cpupid); +if (!cpupid_pid_unset (last_cpupid) &&cpupid_to_nid (last_cpupid) != thisnid) goto out;#ifdef CONFIG_NUMA_BALANCING
@@ -2410,7 +2412,7 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long* This way a short and temporary process migration will * not cause excessive memory migration. */ -if (polnid != current->numa_preferred_nid && +if (thisnid != current->numa_preferred_nid && !current->numa_migrate_seq) goto out;#endif
mm/migrate.c +2 -2
@@ -1498,7 +1498,7 @@ static struct page *alloc_misplaced_dst_page(struct page *page, __GFP_NOWARN) & ~GFP_IOFS, 0);if (newpage) -page_nidpid_xchg_last (newpage,page_nidpid_last (page)); +page_cpupid_xchg_last (newpage,page_cpupid_last (page));return newpage; }
@@ -1675,7 +1675,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,if (!new_page) goto out_fail; -page_nidpid_xchg_last (new_page,page_nidpid_last (page)); +page_cpupid_xchg_last (new_page,page_cpupid_last (page)); isolated =numamigrate_isolate_page (pgdat, page);if (!isolated) {
mm/mm_init.c +9 -9
@@ -71,26 +71,26 @@ void __init mminit_verify_pageflags_layout(void)unsigned long or_mask, add_mask; shift =8 *sizeof (unsigned long ); - width = shift -SECTIONS_WIDTH -NODES_WIDTH -ZONES_WIDTH -LAST_NIDPID_SHIFT ; + width = shift -SECTIONS_WIDTH -NODES_WIDTH -ZONES_WIDTH -LAST_CPUPID_SHIFT ;mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_widths" , -"Section %d Node %d Zone %d Lastnidpid %d Flags %d\n" , +"Section %d Node %d Zone %d Lastcpupid %d Flags %d\n" ,SECTIONS_WIDTH ,NODES_WIDTH ,ZONES_WIDTH , -LAST_NIDPID_WIDTH , +LAST_CPUPID_WIDTH ,NR_PAGEFLAGS );mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_shifts" , -"Section %d Node %d Zone %d Lastnidpid %d\n" , +"Section %d Node %d Zone %d Lastcpupid %d\n" ,SECTIONS_SHIFT ,NODES_SHIFT ,ZONES_SHIFT , -LAST_NIDPID_SHIFT ); +LAST_CPUPID_SHIFT );mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_pgshifts" , -"Section %lu Node %lu Zone %lu Lastnidpid %lu\n" , +"Section %lu Node %lu Zone %lu Lastcpupid %lu\n" , (unsigned long )SECTIONS_PGSHIFT , (unsigned long )NODES_PGSHIFT , (unsigned long )ZONES_PGSHIFT , - (unsigned long )LAST_NIDPID_PGSHIFT ); + (unsigned long )LAST_CPUPID_PGSHIFT ); mminit_dprintk(MMINIT_TRACE, "pageflags_layout_nodezoneid", "Node/Zone ID: %lu -> %lu\n", (unsigned long)(ZONEID_PGOFF + ZONEID_SHIFT),
@@ -102,9 +102,9 @@ void __init mminit_verify_pageflags_layout(void)mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_nodeflags" ,"Node not in page flags" );#endif -#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS +#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_nodeflags" , -"Last nidpid not in page flags" ); +"Last cpupid not in page flags" );#endif if (SECTIONS_WIDTH ) {
mm/mmzone.c +7 -7
@@ -97,20 +97,20 @@ void lruvec_init(struct lruvec *lruvec)INIT_LIST_HEAD (&lruvec->lists[lru]); } -#if defined(CONFIG_NUMA_BALANCING ) && !defined(LAST_NIDPID_NOT_IN_PAGE_FLAGS ) -int page_nidpid_xchg_last (struct page *page,int nidpid) +#if defined(CONFIG_NUMA_BALANCING ) && !defined(LAST_CPUPID_IN_PAGE_FLAGS ) +int page_cpupid_xchg_last (struct page *page,int cpupid) {unsigned long old_flags, flags; -int last_nidpid; +int last_cpupid;do { old_flags = flags = page->flags; - last_nidpid =page_nidpid_last (page); + last_cpupid =page_cpupid_last (page); - flags &= ~(LAST_NIDPID_MASK <<LAST_NIDPID_PGSHIFT ); - flags |= (nidpid &LAST_NIDPID_MASK ) <<LAST_NIDPID_PGSHIFT ; + flags &= ~(LAST_CPUPID_MASK <<LAST_CPUPID_PGSHIFT ); + flags |= (cpupid &LAST_CPUPID_MASK ) <<LAST_CPUPID_PGSHIFT ; }while (unlikely (cmpxchg (&page->flags, old_flags, flags) != old_flags)); -return last_nidpid; +return last_cpupid; }#endif
mm/mprotect.c +14 -14
@@ -37,14 +37,14 @@ static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot)static unsigned long change_pte_range (struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot, -int dirty_accountable,int prot_numa,bool *ret_all_same_nidpid) +int dirty_accountable,int prot_numa,bool *ret_all_same_cpupid) {struct mm_struct *mm = vma->vm_mm;pte_t *pte, oldpte;spinlock_t *ptl;unsigned long pages =0 ; -bool all_same_nidpid = true; -int last_nid =-1 ; +bool all_same_cpupid = true; +int last_cpu =-1 ;int last_pid =-1 ; pte =pte_offset_map_lock (mm, pmd, addr, &ptl);
@@ -64,17 +64,17 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, page =vm_normal_page (vma, addr, oldpte);if (page) { -int nidpid =page_nidpid_last (page); -int this_nid =nidpid_to_nid (nidpid); -int this_pid =nidpid_to_pid (nidpid); +int cpupid =page_cpupid_last (page); +int this_cpu =cpupid_to_cpu (cpupid); +int this_pid =cpupid_to_pid (cpupid); -if (last_nid ==-1 ) - last_nid = this_nid; +if (last_cpu ==-1 ) + last_cpu = this_cpu;if (last_pid ==-1 ) last_pid = this_pid; -if (last_nid != this_nid || +if (last_cpu != this_cpu || last_pid != this_pid) { - all_same_nidpid = false; + all_same_cpupid = false; }if (!pte_numa (oldpte)) {
@@ -115,7 +115,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,arch_leave_lazy_mmu_mode ();pte_unmap_unlock (pte -1 , ptl); - *ret_all_same_nidpid = all_same_nidpid; + *ret_all_same_cpupid = all_same_cpupid;return pages; }
@@ -142,7 +142,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,pmd_t *pmd;unsigned long next;unsigned long pages =0 ; -bool all_same_nidpid; +bool all_same_cpupid; pmd =pmd_offset (pud, addr); do {
@@ -168,7 +168,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,if (pmd_none_or_clear_bad (pmd))continue ; this_pages =change_pte_range (vma, pmd, addr, next, newprot, - dirty_accountable, prot_numa, &all_same_nidpid); + dirty_accountable, prot_numa, &all_same_cpupid); pages += this_pages;/*
@@ -177,7 +177,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,* node. This allows a regular PMD to be handled as one fault * and effectively batches the taking of the PTL */ -if (prot_numa && this_pages && all_same_nidpid) +if (prot_numa && this_pages && all_same_cpupid)change_pmd_protnuma (vma->vm_mm, addr, pmd); }while (pmd++, addr = next, addr != end);
mm/page_alloc.c +2 -2
@@ -626,7 +626,7 @@ static inline int free_pages_check(struct page *page)bad_page (page);return 1 ; } -page_nidpid_reset_last (page); +page_cpupid_reset_last (page);if (page->flags &PAGE_FLAGS_CHECK_AT_PREP ) page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP ;return 0 ;
@@ -4015,7 +4015,7 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,mminit_verify_page_links (page, zone, nid, pfn);init_page_count (page);page_mapcount_reset (page); -page_nidpid_reset_last (page); +page_cpupid_reset_last (page);SetPageReserved (page);/* * Mark the block movable so that blocks are reserved for
Signature · 7 lines
1.8.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 42/63] mm: numa: Change page last {nid,pid} into {cpu,pid}
Quoted 9 lines · Mel Gorman
From: Peter Zijlstra <[email protected]>
Change the per page last fault tracking to use cpu,pid instead of nid,pid. This will allow us to try and lookup the alternate task more easily. Note that even though it is the cpu that is store in the page flags that the
mpol_misplaceddecision is still based on the node.Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Mel Gorman <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Peter Zijlstra
merge bot[tip:sched/core] mm: numa: Change page last {nid,pid} into {cpu, pid}
In reply to Mel Gormanmm: numa: Change page last {nid,pid} into {cpu,pid}
Commit-ID: 90572890d202527c366aa9489b32404e88a7c020 Gitweb: http://git.kernel.org/tip/90572890d202527c366aa9489b32404e88a7c020 Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 7 Oct 2013 11:29:20 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 9 Oct 2013 14:47:45 +0200
mm: numa: Change page last {nid,pid} into {cpu,pid}
Change the per page last fault tracking to use cpu,pid instead of nid,pid. This will allow us to try and lookup the alternate task more easily. Note that even though it is the cpu that is store in the page flags that the mpol_misplaced decision is still based on the node.
--- include/linux/mm.h | 90 ++++++++++++++++++++++----------------- include/linux/mm_types.h | 4 +- include/linux/page-flags-layout.h | 22 +++++----- kernel/bounds.c | 4 ++ kernel/sched/fair.c | 6 +-- mm/huge_memory.c | 8 ++-- mm/memory.c | 16 +++---- mm/mempolicy.c | 16 ++++--- mm/migrate.c | 4 +- mm/mm_init.c | 18 ++++---- mm/mmzone.c | 14 +++--- mm/mprotect.c | 28 ++++++------ mm/page_alloc.c | 4 +- 13 files changed, 125 insertions(+), 109 deletions(-)
include/linux/mm.h +50 -40
@@ -581,11 +581,11 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)* sets it, so none of the operations on it need to be atomic. */ -/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_NIDPID] | ... | FLAGS | */ +/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_CPUPID] | ... | FLAGS | */ #define SECTIONS_PGOFF ((sizeof(unsigned long)*8) - SECTIONS_WIDTH)#define NODES_PGOFF (SECTIONS_PGOFF - NODES_WIDTH)#define ZONES_PGOFF (NODES_PGOFF - ZONES_WIDTH) -#define LAST_NIDPID_PGOFF (ZONES_PGOFF - LAST_NIDPID_WIDTH) +#define LAST_CPUPID_PGOFF (ZONES_PGOFF - LAST_CPUPID_WIDTH)/* * Define the bit shifts to access each section. For non-existent
@@ -595,7 +595,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)#define SECTIONS_PGSHIFT (SECTIONS_PGOFF * (SECTIONS_WIDTH != 0))#define NODES_PGSHIFT (NODES_PGOFF * (NODES_WIDTH != 0))#define ZONES_PGSHIFT (ZONES_PGOFF * (ZONES_WIDTH != 0)) -#define LAST_NIDPID_PGSHIFT (LAST_NIDPID_PGOFF * (LAST_NIDPID_WIDTH != 0)) +#define LAST_CPUPID_PGSHIFT (LAST_CPUPID_PGOFF * (LAST_CPUPID_WIDTH != 0))/* NODE:ZONE or SECTION:ZONE is used to ID a zone for the buddy allocator */ #ifdef NODE_NOT_IN_PAGE_FLAGS
@@ -617,7 +617,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)#define ZONES_MASK ((1UL << ZONES_WIDTH) - 1)#define NODES_MASK ((1UL << NODES_WIDTH) - 1)#define SECTIONS_MASK ((1UL << SECTIONS_WIDTH) - 1) -#define LAST_NIDPID_MASK ((1UL << LAST_NIDPID_WIDTH) - 1) +#define LAST_CPUPID_MASK ((1UL << LAST_CPUPID_WIDTH) - 1)#define ZONEID_MASK ((1UL << ZONEID_SHIFT) - 1)static inline enum zone_type page_zonenum (const struct page *page)
@@ -661,96 +661,106 @@ static inline int page_to_nid(const struct page *page)#endif #ifdef CONFIG_NUMA_BALANCING -static inline int nid_pid_to_nidpid (int nid,int pid) +static inline int cpu_pid_to_cpupid (int cpu,int pid) { -return ((nid &LAST__NID_MASK ) <<LAST__PID_SHIFT ) | (pid &LAST__PID_MASK ); +return ((cpu &LAST__CPU_MASK ) <<LAST__PID_SHIFT ) | (pid &LAST__PID_MASK ); } -static inline int nidpid_to_pid (int nidpid) +static inline int cpupid_to_pid (int cpupid) { -return nidpid &LAST__PID_MASK ; +return cpupid &LAST__PID_MASK ; } -static inline int nidpid_to_nid (int nidpid) +static inline int cpupid_to_cpu (int cpupid) { -return (nidpid >>LAST__PID_SHIFT ) &LAST__NID_MASK ; +return (cpupid >>LAST__PID_SHIFT ) &LAST__CPU_MASK ; } -static inline bool nidpid_pid_unset (int nidpid) +static inline int cpupid_to_nid (int cpupid) { -return nidpid_to_pid (nidpid) == (-1 &LAST__PID_MASK ); +return cpu_to_node (cpupid_to_cpu (cpupid)); } -static inline bool nidpid_nid_unset (int nidpid) +static inline bool cpupid_pid_unset (int cpupid) { -return nidpid_to_nid (nidpid) == (-1 &LAST__NID_MASK ); +return cpupid_to_pid (cpupid) == (-1 &LAST__PID_MASK ); } -#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS -static inline int page_nidpid_xchg_last (struct page *page,int nid) +static inline bool cpupid_cpu_unset (int cpupid) { -return xchg (&page->_last_nidpid, nid); +return cpupid_to_cpu (cpupid) == (-1 &LAST__CPU_MASK ); } -static inline int page_nidpid_last (struct page *page) +#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS +static inline int page_cpupid_xchg_last (struct page *page,int cpupid) { -return page->_last_nidpid; +return xchg (&page->_last_cpupid, cpupid); } -static inline void page_nidpid_reset_last (struct page *page) + +static inline int page_cpupid_last (struct page *page) +{ +return page->_last_cpupid; +} +static inline void page_cpupid_reset_last (struct page *page) { - page->_last_nidpid =-1 ; + page->_last_cpupid =-1 ; }#else -static inline int page_nidpid_last (struct page *page) +static inline int page_cpupid_last (struct page *page) { -return (page->flags >>LAST_NIDPID_PGSHIFT ) &LAST_NIDPID_MASK ; +return (page->flags >>LAST_CPUPID_PGSHIFT ) &LAST_CPUPID_MASK ; } -extern int page_nidpid_xchg_last (struct page *page,int nidpid); +extern int page_cpupid_xchg_last (struct page *page,int cpupid); -static inline void page_nidpid_reset_last (struct page *page) +static inline void page_cpupid_reset_last (struct page *page) { -int nidpid = (1 <<LAST_NIDPID_SHIFT ) -1 ; +int cpupid = (1 <<LAST_CPUPID_SHIFT ) -1 ; - page->flags &= ~(LAST_NIDPID_MASK <<LAST_NIDPID_PGSHIFT ); - page->flags |= (nidpid &LAST_NIDPID_MASK ) <<LAST_NIDPID_PGSHIFT ; + page->flags &= ~(LAST_CPUPID_MASK <<LAST_CPUPID_PGSHIFT ); + page->flags |= (cpupid &LAST_CPUPID_MASK ) <<LAST_CPUPID_PGSHIFT ; } -#endif /* LAST_NIDPID_NOT_IN_PAGE_FLAGS */ -#else -static inline int page_nidpid_xchg_last (struct page *page,int nidpid) +#endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */ +#else /* !CONFIG_NUMA_BALANCING */ +static inline int page_cpupid_xchg_last (struct page *page,int cpupid) { -return page_to_nid (page); +return page_to_nid (page);/* XXX */ } -static inline int page_nidpid_last (struct page *page) +static inline int page_cpupid_last (struct page *page) { -return page_to_nid (page); +return page_to_nid (page);/* XXX */ } -static inline int nidpid_to_nid (int nidpid) +static inline int cpupid_to_nid (int cpupid) {return -1 ; } -static inline int nidpid_to_pid (int nidpid) +static inline int cpupid_to_pid (int cpupid) {return -1 ; } -static inline int nid_pid_to_nidpid (int nid,int pid) +static inline int cpupid_to_cpu (int cpupid) {return -1 ; } -static inline bool nidpid_pid_unset (int nidpid) +static inline int cpu_pid_to_cpupid (int nid,int pid) +{ +return -1 ; +} + +static inline bool cpupid_pid_unset (int cpupid) {return 1 ; } -static inline void page_nidpid_reset_last (struct page *page) +static inline void page_cpupid_reset_last (struct page *page) { } -#endif +#endif /* CONFIG_NUMA_BALANCING */ static inline struct zone *page_zone (const struct page *page) {
include/linux/mm_types.h +2 -2
@@ -174,8 +174,8 @@ struct page {void *shadow;#endif -#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS -int _last_nidpid; +#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS +int _last_cpupid;#endif }/*
include/linux/page-flags-layout.h +11 -11
@@ -39,9 +39,9 @@* lookup is necessary. * * No sparsemem or sparsemem vmemmap: | NODE | ZONE | ... | FLAGS | -* " plus space for last_nidpid: | NODE | ZONE | LAST_NIDPID ... | FLAGS | +* " plus space for last_cpupid: | NODE | ZONE | LAST_CPUPID ... | FLAGS | * classic sparse with space for node:| SECTION | NODE | ZONE | ... | FLAGS | -* " plus space for last_nidpid: | SECTION | NODE | ZONE | LAST_NIDPID ... | FLAGS | +* " plus space for last_cpupid: | SECTION | NODE | ZONE | LAST_CPUPID ... | FLAGS | * classic sparse no space for node: | SECTION | ZONE | ... | FLAGS | */ #if defined(CONFIG_SPARSEMEM ) && !defined(CONFIG_SPARSEMEM_VMEMMAP )
@@ -65,18 +65,18 @@#define LAST__PID_SHIFT 8#define LAST__PID_MASK ((1 << LAST__PID_SHIFT)-1) -#define LAST__NID_SHIFT NODES_SHIFT -#define LAST__NID_MASK ((1 << LAST__NID_SHIFT)-1) +#define LAST__CPU_SHIFT NR_CPUS_BITS +#define LAST__CPU_MASK ((1 << LAST__CPU_SHIFT)-1) -#define LAST_NIDPID_SHIFT (LAST__PID_SHIFT+LAST__NID_SHIFT) +#define LAST_CPUPID_SHIFT (LAST__PID_SHIFT+LAST__CPU_SHIFT)#else -#define LAST_NIDPID_SHIFT 0 +#define LAST_CPUPID_SHIFT 0#endif -#if SECTIONS_WIDTH +ZONES_WIDTH +NODES_SHIFT +LAST_NIDPID_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS -#define LAST_NIDPID_WIDTH LAST_NIDPID_SHIFT +#if SECTIONS_WIDTH +ZONES_WIDTH +NODES_SHIFT +LAST_CPUPID_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS +#define LAST_CPUPID_WIDTH LAST_CPUPID_SHIFT#else -#define LAST_NIDPID_WIDTH 0 +#define LAST_CPUPID_WIDTH 0#endif /*
@@ -87,8 +87,8 @@#define NODE_NOT_IN_PAGE_FLAGS #endif -#if defined(CONFIG_NUMA_BALANCING ) &&LAST_NIDPID_WIDTH ==0 -#define LAST_NIDPID_NOT_IN_PAGE_FLAGS +#if defined(CONFIG_NUMA_BALANCING ) &&LAST_CPUPID_WIDTH ==0 +#define LAST_CPUPID_NOT_IN_PAGE_FLAGS #endif #endif /* _LINUX_PAGE_FLAGS_LAYOUT */
kernel/bounds.c +4 -0
@@ -10,6 +10,7 @@#include <linux/mmzone.h> #include <linux/kbuild.h> #include <linux/page_cgroup.h> +#include <linux/log2.h> void foo (void ) {
@@ -17,5 +18,8 @@ void foo(void)DEFINE (NR_PAGEFLAGS , __NR_PAGEFLAGS);DEFINE (MAX_NR_ZONES , __MAX_NR_ZONES);DEFINE (NR_PCG_FLAGS , __NR_PCG_FLAGS); +#ifdef CONFIG_SMP +DEFINE (NR_CPUS_BITS ,ilog2 (CONFIG_NR_CPUS )); +#endif /* End of constants */ }
kernel/sched/fair.c +3 -3
@@ -1210,7 +1210,7 @@ static void task_numa_placement(struct task_struct *p)/* * Got a PROT_NONE fault for a page on @node. */ -void task_numa_fault (int last_nidpid,int node,int pages,bool migrated) +void task_numa_fault (int last_cpupid,int node,int pages,bool migrated) {struct task_struct *p = current;int priv;
@@ -1226,8 +1226,8 @@ void task_numa_fault(int last_nidpid, int node, int pages, bool migrated)* First accesses are treated as private, otherwise consider accesses * to be private if the accessing pid has not changed */ -if (!nidpid_pid_unset (last_nidpid)) - priv = ((p->pid &LAST__PID_MASK ) ==nidpid_to_pid (last_nidpid)); +if (!cpupid_pid_unset (last_cpupid)) + priv = ((p->pid &LAST__PID_MASK ) ==cpupid_to_pid (last_cpupid));else priv =1 ;
mm/huge_memory.c +4 -4
@@ -1282,7 +1282,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct page *page;unsigned long haddr = addr &HPAGE_PMD_MASK ;int page_nid =-1 , this_nid =numa_node_id (); -int target_nid, last_nidpid =-1 ; +int target_nid, last_cpupid =-1 ;bool page_locked;bool migrated = false;
@@ -1293,7 +1293,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, page =pmd_page (pmd);BUG_ON (is_huge_zero_page (page)); page_nid =page_to_nid (page); - last_nidpid =page_nidpid_last (page); + last_cpupid =page_cpupid_last (page);count_vm_numa_event (NUMA_HINT_FAULTS );if (page_nid == this_nid)count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL );
@@ -1362,7 +1362,7 @@ out:page_unlock_anon_vma_read (anon_vma);if (page_nid !=-1 ) -task_numa_fault (last_nidpid, page_nid,HPAGE_PMD_NR , migrated); +task_numa_fault (last_cpupid, page_nid,HPAGE_PMD_NR , migrated);return 0 ; }
@@ -1682,7 +1682,7 @@ static void __split_huge_page_refcount(struct page *page, page_tail->mapping = page->mapping; page_tail->index = page->index + i; -page_nidpid_xchg_last (page_tail,page_nidpid_last (page)); +page_cpupid_xchg_last (page_tail,page_cpupid_last (page));BUG_ON (!PageAnon (page_tail));BUG_ON (!PageUptodate (page_tail));
mm/memory.c +8 -8
@@ -69,8 +69,8 @@#include "internal.h" -#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS -#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_nidpid. +#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS +#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_cpupid.#endif #ifndef CONFIG_NEED_MULTIPLE_NODES
@@ -3536,7 +3536,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct page *page =NULL ;spinlock_t *ptl;int page_nid =-1 ; -int last_nidpid; +int last_cpupid;int target_nid;bool migrated = false;
@@ -3567,7 +3567,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }BUG_ON (is_zero_pfn (page_to_pfn (page))); - last_nidpid =page_nidpid_last (page); + last_cpupid =page_cpupid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (ptep, ptl);
@@ -3583,7 +3583,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, out:if (page_nid !=-1 ) -task_numa_fault (last_nidpid, page_nid,1 , migrated); +task_numa_fault (last_cpupid, page_nid,1 , migrated);return 0 ; }
@@ -3598,7 +3598,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,unsigned long offset;spinlock_t *ptl;bool numa = false; -int last_nidpid; +int last_cpupid;spin_lock (&mm->page_table_lock); pmd = *pmdp;
@@ -3643,7 +3643,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,if (unlikely (!page))continue ; - last_nidpid =page_nidpid_last (page); + last_cpupid =page_cpupid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (pte, ptl);
@@ -3656,7 +3656,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }if (page_nid !=-1 ) -task_numa_fault (last_nidpid, page_nid,1 , migrated); +task_numa_fault (last_cpupid, page_nid,1 , migrated); pte =pte_offset_map_lock (mm, pmdp, addr, &ptl); }
mm/mempolicy.c +9 -7
@@ -2324,6 +2324,8 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned longstruct zone *zone;int curnid =page_to_nid (page);unsigned long pgoff; +int thiscpu =raw_smp_processor_id (); +int thisnid =cpu_to_node (thiscpu);int polnid =-1 ;int ret =-1 ;
@@ -2372,11 +2374,11 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long/* Migrate the page towards the node whose CPU is referencing it */ if (pol->flags &MPOL_F_MORON ) { -int last_nidpid; -int this_nidpid; +int last_cpupid; +int this_cpupid; - polnid =numa_node_id (); - this_nidpid =nid_pid_to_nidpid (polnid, current->pid); + polnid = thisnid; + this_cpupid =cpu_pid_to_cpupid (thiscpu, current->pid);/* * Multi-stage node selection is used in conjunction
@@ -2399,8 +2401,8 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long* it less likely we act on an unlikely task<->page * relation. */ - last_nidpid =page_nidpid_xchg_last (page, this_nidpid); -if (!nidpid_pid_unset (last_nidpid) &&nidpid_to_nid (last_nidpid) != polnid) + last_cpupid =page_cpupid_xchg_last (page, this_cpupid); +if (!cpupid_pid_unset (last_cpupid) &&cpupid_to_nid (last_cpupid) != thisnid) goto out;#ifdef CONFIG_NUMA_BALANCING
@@ -2410,7 +2412,7 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long* This way a short and temporary process migration will * not cause excessive memory migration. */ -if (polnid != current->numa_preferred_nid && +if (thisnid != current->numa_preferred_nid && !current->numa_migrate_seq) goto out;#endif
mm/migrate.c +2 -2
@@ -1498,7 +1498,7 @@ static struct page *alloc_misplaced_dst_page(struct page *page, __GFP_NOWARN) & ~GFP_IOFS, 0);if (newpage) -page_nidpid_xchg_last (newpage,page_nidpid_last (page)); +page_cpupid_xchg_last (newpage,page_cpupid_last (page));return newpage; }
@@ -1675,7 +1675,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,if (!new_page) goto out_fail; -page_nidpid_xchg_last (new_page,page_nidpid_last (page)); +page_cpupid_xchg_last (new_page,page_cpupid_last (page)); isolated =numamigrate_isolate_page (pgdat, page);if (!isolated) {
mm/mm_init.c +9 -9
@@ -71,26 +71,26 @@ void __init mminit_verify_pageflags_layout(void)unsigned long or_mask, add_mask; shift =8 *sizeof (unsigned long ); - width = shift -SECTIONS_WIDTH -NODES_WIDTH -ZONES_WIDTH -LAST_NIDPID_SHIFT ; + width = shift -SECTIONS_WIDTH -NODES_WIDTH -ZONES_WIDTH -LAST_CPUPID_SHIFT ;mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_widths" , -"Section %d Node %d Zone %d Lastnidpid %d Flags %d\n" , +"Section %d Node %d Zone %d Lastcpupid %d Flags %d\n" ,SECTIONS_WIDTH ,NODES_WIDTH ,ZONES_WIDTH , -LAST_NIDPID_WIDTH , +LAST_CPUPID_WIDTH ,NR_PAGEFLAGS );mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_shifts" , -"Section %d Node %d Zone %d Lastnidpid %d\n" , +"Section %d Node %d Zone %d Lastcpupid %d\n" ,SECTIONS_SHIFT ,NODES_SHIFT ,ZONES_SHIFT , -LAST_NIDPID_SHIFT ); +LAST_CPUPID_SHIFT );mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_pgshifts" , -"Section %lu Node %lu Zone %lu Lastnidpid %lu\n" , +"Section %lu Node %lu Zone %lu Lastcpupid %lu\n" , (unsigned long )SECTIONS_PGSHIFT , (unsigned long )NODES_PGSHIFT , (unsigned long )ZONES_PGSHIFT , - (unsigned long )LAST_NIDPID_PGSHIFT ); + (unsigned long )LAST_CPUPID_PGSHIFT ); mminit_dprintk(MMINIT_TRACE, "pageflags_layout_nodezoneid", "Node/Zone ID: %lu -> %lu\n", (unsigned long)(ZONEID_PGOFF + ZONEID_SHIFT),
@@ -102,9 +102,9 @@ void __init mminit_verify_pageflags_layout(void)mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_nodeflags" ,"Node not in page flags" );#endif -#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS +#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_nodeflags" , -"Last nidpid not in page flags" ); +"Last cpupid not in page flags" );#endif if (SECTIONS_WIDTH ) {
mm/mmzone.c +7 -7
@@ -97,20 +97,20 @@ void lruvec_init(struct lruvec *lruvec)INIT_LIST_HEAD (&lruvec->lists[lru]); } -#if defined(CONFIG_NUMA_BALANCING ) && !defined(LAST_NIDPID_NOT_IN_PAGE_FLAGS ) -int page_nidpid_xchg_last (struct page *page,int nidpid) +#if defined(CONFIG_NUMA_BALANCING ) && !defined(LAST_CPUPID_NOT_IN_PAGE_FLAGS ) +int page_cpupid_xchg_last (struct page *page,int cpupid) {unsigned long old_flags, flags; -int last_nidpid; +int last_cpupid;do { old_flags = flags = page->flags; - last_nidpid =page_nidpid_last (page); + last_cpupid =page_cpupid_last (page); - flags &= ~(LAST_NIDPID_MASK <<LAST_NIDPID_PGSHIFT ); - flags |= (nidpid &LAST_NIDPID_MASK ) <<LAST_NIDPID_PGSHIFT ; + flags &= ~(LAST_CPUPID_MASK <<LAST_CPUPID_PGSHIFT ); + flags |= (cpupid &LAST_CPUPID_MASK ) <<LAST_CPUPID_PGSHIFT ; }while (unlikely (cmpxchg (&page->flags, old_flags, flags) != old_flags)); -return last_nidpid; +return last_cpupid; }#endif
mm/mprotect.c +14 -14
@@ -37,14 +37,14 @@ static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot)static unsigned long change_pte_range (struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot, -int dirty_accountable,int prot_numa,bool *ret_all_same_nidpid) +int dirty_accountable,int prot_numa,bool *ret_all_same_cpupid) {struct mm_struct *mm = vma->vm_mm;pte_t *pte, oldpte;spinlock_t *ptl;unsigned long pages =0 ; -bool all_same_nidpid = true; -int last_nid =-1 ; +bool all_same_cpupid = true; +int last_cpu =-1 ;int last_pid =-1 ; pte =pte_offset_map_lock (mm, pmd, addr, &ptl);
@@ -64,17 +64,17 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, page =vm_normal_page (vma, addr, oldpte);if (page) { -int nidpid =page_nidpid_last (page); -int this_nid =nidpid_to_nid (nidpid); -int this_pid =nidpid_to_pid (nidpid); +int cpupid =page_cpupid_last (page); +int this_cpu =cpupid_to_cpu (cpupid); +int this_pid =cpupid_to_pid (cpupid); -if (last_nid ==-1 ) - last_nid = this_nid; +if (last_cpu ==-1 ) + last_cpu = this_cpu;if (last_pid ==-1 ) last_pid = this_pid; -if (last_nid != this_nid || +if (last_cpu != this_cpu || last_pid != this_pid) { - all_same_nidpid = false; + all_same_cpupid = false; }if (!pte_numa (oldpte)) {
@@ -115,7 +115,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,arch_leave_lazy_mmu_mode ();pte_unmap_unlock (pte -1 , ptl); - *ret_all_same_nidpid = all_same_nidpid; + *ret_all_same_cpupid = all_same_cpupid;return pages; }
@@ -142,7 +142,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,pmd_t *pmd;unsigned long next;unsigned long pages =0 ; -bool all_same_nidpid; +bool all_same_cpupid; pmd =pmd_offset (pud, addr); do {
@@ -168,7 +168,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,if (pmd_none_or_clear_bad (pmd))continue ; this_pages =change_pte_range (vma, pmd, addr, next, newprot, - dirty_accountable, prot_numa, &all_same_nidpid); + dirty_accountable, prot_numa, &all_same_cpupid); pages += this_pages;/*
@@ -177,7 +177,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,* node. This allows a regular PMD to be handled as one fault * and effectively batches the taking of the PTL */ -if (prot_numa && this_pages && all_same_nidpid) +if (prot_numa && this_pages && all_same_cpupid)change_pmd_protnuma (vma->vm_mm, addr, pmd); }while (pmd++, addr = next, addr != end);
mm/page_alloc.c +2 -2
@@ -626,7 +626,7 @@ static inline int free_pages_check(struct page *page)bad_page (page);return 1 ; } -page_nidpid_reset_last (page); +page_cpupid_reset_last (page);if (page->flags &PAGE_FLAGS_CHECK_AT_PREP ) page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP ;return 0 ;
@@ -4015,7 +4015,7 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,mminit_verify_page_links (page, zone, nid, pfn);init_page_count (page);page_mapcount_reset (page); -page_nidpid_reset_last (page); +page_cpupid_reset_last (page);SetPageReserved (page);/* * Mark the block movable so that blocks are reserved for