Basic scheduler support for automatic NUMA balancing V7
Subsystems (21)
Mel Gorman
reviewercover Basic scheduler support for automatic NUMA balancing V7
It has been a long time since V6 of this series and time for an update. Much of this is now stabilised with the most important addition being the inclusion of Peter and Rik's work on grouping tasks that share pages together.
This series has a number of goals. It reduces overhead of automatic balancing through scan rate reduction and the avoidance of TLB flushes. It selects a preferred node and moves tasks towards their memory as well as moving memory toward their task. It handles shared pages and groups related tasks together.
Changelog since V6
o Group tasks that share pages together
o More scan avoidance of VMAs mapping pages that are not likely to migrate
o cpunid conversion, system-wide searching of tasks to balance with
Changelog since V6
o Various TLB flush optimisations
o Comment updates
o Sanitise task_numa_fault callsites for consistent semantics
o Revert some of the scanning adaption stuff
o Revert patch that defers scanning until task schedules on another node
o Start delayed scanning properly
o Avoid the same task always performing the PTE scan
o Continue PTE scanning even if migration is rate limited
Changelog since V5
o Add __GFP_NOWARN for numa hinting fault count
o Use is_huge_zero_page
o Favour moving tasks towards nodes with higher faults
o Optionally resist moving tasks towards nodes with lower faults
o Scan shared THP pages
Changelog since V4
o Added code that avoids overloading preferred nodes
o Swap tasks if nodes are overloaded and the swap does not impair locality
Changelog since V3
o Correct detection of unset last nid/pid information
o Dropped nr_preferred_running and replaced it with Peter's load balancing
o Pass in correct node information for THP hinting faults
o Pressure tasks sharing a THP page to move towards same node
o Do not set pmd_numa if false sharing is detected
Changelog since V2
o Reshuffle to match Peter's implied preference for layout
o Reshuffle to move private/shared split towards end of series to make it easier to evaluate the impact
o Use PID information to identify private accesses
o Set the floor for PTE scanning based on virtual address space scan rates instead of time
o Some locking improvements
o Do not preempt pinned tasks unless they are kernel threads
Changelog since V1
o Scan pages with elevated map count (shared pages)
o Scale scan rates based on the vsz of the process so the sampling of the task is independant of its size
o Favour moving towards nodes with more faults even if it's not the preferred node
o Laughably basic accounting of a compute overloaded node when selecting the preferred node.
o Applied review comments
This series integrates basic scheduler support for automatic NUMA balancing. It borrows very heavily from Peter Ziljstra's work in "sched, numa, mm:
Add adaptive NUMA affinity support" but deviates too much to preserve Signed-off-bys. As before, if the relevant authors are ok with it I'll add Signed-off-bys (or add them yourselves if you pick the patches up).
There are still gaps between this series and manual binding but it's still an important series of steps in the right direction and the size of the series is getting unwieldly. As before, the intention is not to complete the work but to incrementally improve mainline and preserve bisectability for any bug reports that crop up.
Patch 1 is a monolothic dump of patches thare are destined for upstream that
this series indirectly depends upon.
Patches 2-3 adds sysctl documentation and comment fixlets
Patch 4 avoids accounting for a hinting fault if another thread handled the
fault in parallel
Patches 5-6 avoid races with parallel THP migration and THP splits.
Patch 7 corrects a THP NUMA hint fault accounting bug
Patch 8 sanitizes task_numa_fault callsites to have consist semantics and
always record the fault based on the correct location of the page.
Patch 9 avoids trying to migrate the THP zero page
Patch 10 avoids the same task being selected to perform the PTE scan within
a shared address space.
Patch 11 continues PTE scanning even if migration rate limited
Patch 12 notes that delaying the PTE scan until a task is scheduled on an
alternatie node misses the case where the task is only accessing shared memory on a partially loaded machine and reverts a patch.
Patches 13,15 initialses numa_next_scan properly so that PTE scanning is delayed
when a process starts.
Patch 14 sets the scan rate proportional to the size of the task being
scanned.
Patches 16-17 avoids TLB flushes during the PTE scan if no updates are made
Patch 18 slows the scan rate if no hinting faults were trapped by an idle task.
Patch 19 tracks NUMA hinting faults per-task and per-node
Patches 20-24 selects a preferred node at the end of a PTE scan based on what
node incurrent the highest number of NUMA faults. When the balancer is comparing two CPU it will prefer to locate tasks on their preferred node. When initially selected the task is rescheduled on the preferred node if it is not running on that node already. This avoids waiting for the scheduler to move the task slowly.
Patch 25 adds infrastructure to allow separate tracking of shared/private
pages but treats all faults as if they are private accesses. Laying it out this way reduces churn later in the series when private fault detection is introduced
Patch 26 avoids some unnecessary allocation
Patch 27-28 kicks away some training wheels and scans shared pages and
small VMAs.
Patch 29 introduces private fault detection based on the PID of the faulting
process and accounts for shared/private accesses differently.
Patch 30 avoids migrating memory immediately after the load balancer moves
a task to another node in case it's a transient migration.
Patch 31 pick the least loaded CPU based on a preferred node based on
a scheduling domain common to both the source and destination NUMA node.
Patch 32 retries task migration if an earlier attempt failed
Patch 33 will begin task migration immediately if running on its preferred
node
Patch 34 will avoid trapping hinting faults for shared read-only library
pages as these never migrate anyway
Patch 35 avoids handling pmd hinting faults if none of the ptes below it were
marked pte numa
Patches 36-37 introduce a mechanism for swapping tasks
Patch 38 uses a system-wide search to find tasks that can be swapped
to improve the overall locality of the system.
Patch 39 notes that the system-wide search may ignore the preferred node and
will use the preferred node placement if it has spare compute capacity.
Patches 40-42 use cpupid to track pages so potential sharing tasks can
be quickly found
Patches 43-44 avoids grouping based on read-only pages
Patches 45-46 schedules tasks based on their numa group
Patch 47 adds some debugging aids
Patches 48-49 separately considers task and group weights when selecting the node to
schedule a task on
Patch 50 avoids migrating tasks away from their preferred node.
Kernel 3.11-rc7 is the testing baseline.
o account-v7 Patches 1-7
o lesspmd-v7 Patches 1-35
o selectweight-v7 Patches 1-49
o avoidmove-v7 Patches 1-50
This is SpecJBB running on a 4-socket machine with THP enabled and one JVM running for the whole system.
specjbb
3.11.0-rc7 3.11.0-rc7 3.11.0-rc7 3.11.0-rc7
account-v7 lesspmd-v7 selectweight-v7 avoidmove-v7
TPut 1 26483.00 ( 0.00%) 26691.00 ( 0.79%) 26618.00 ( 0.51%) 25450.00 ( -3.90%) TPut 2 55009.00 ( 0.00%) 54744.00 ( -0.48%) 53200.00 ( -3.29%) 53998.00 ( -1.84%) TPut 3 86711.00 ( 0.00%) 85564.00 ( -1.32%) 86547.00 ( -0.19%) 85424.00 ( -1.48%) TPut 4 108073.00 ( 0.00%) 112757.00 ( 4.33%) 111408.00 ( 3.09%) 113522.00 ( 5.04%) TPut 5 138128.00 ( 0.00%) 137733.00 ( -0.29%) 140797.00 ( 1.93%) 140930.00 ( 2.03%) TPut 6 161949.00 ( 0.00%) 164499.00 ( 1.57%) 164759.00 ( 1.74%) 161916.00 ( -0.02%) TPut 7 185205.00 ( 0.00%) 190214.00 ( 2.70%) 189409.00 ( 2.27%) 191425.00 ( 3.36%) TPut 8 214152.00 ( 0.00%) 216550.00 ( 1.12%) 219510.00 ( 2.50%) 217374.00 ( 1.50%) TPut 9 245408.00 ( 0.00%) 242975.00 ( -0.99%) 241001.00 ( -1.80%) 243116.00 ( -0.93%) TPut 10 262786.00 ( 0.00%) 267812.00 ( 1.91%) 260897.00 ( -0.72%) 267728.00 ( 1.88%) TPut 11 293162.00 ( 0.00%) 299621.00 ( 2.20%) 291130.00 ( -0.69%) 300006.00 ( 2.33%) TPut 12 310423.00 ( 0.00%) 317867.00 ( 2.40%) 307821.00 ( -0.84%) 317531.00 ( 2.29%) TPut 13 328542.00 ( 0.00%) 347286.00 ( 5.71%) 327800.00 ( -0.23%) 344849.00 ( 4.96%) TPut 14 362081.00 ( 0.00%) 374173.00 ( 3.34%) 342014.00 ( -5.54%) 366256.00 ( 1.15%) TPut 15 374475.00 ( 0.00%) 393658.00 ( 5.12%) 348941.00 ( -6.82%) 376056.00 ( 0.42%) TPut 16 407367.00 ( 0.00%) 409212.00 ( 0.45%) 361272.00 (-11.32%) 409353.00 ( 0.49%) TPut 17 423282.00 ( 0.00%) 424424.00 ( 0.27%) 377808.00 (-10.74%) 410761.00 ( -2.96%) TPut 18 447960.00 ( 0.00%) 456736.00 ( 1.96%) 392421.00 (-12.40%) 437756.00 ( -2.28%) TPut 19 449296.00 ( 0.00%) 475797.00 ( 5.90%) 404142.00 (-10.05%) 446286.00 ( -0.67%) TPut 20 480073.00 ( 0.00%) 487883.00 ( 1.63%) 414085.00 (-13.75%) 453840.00 ( -5.46%) TPut 21 476891.00 ( 0.00%) 505589.00 ( 6.02%) 422953.00 (-11.31%) 458974.00 ( -3.76%) TPut 22 492092.00 ( 0.00%) 503878.00 ( 2.40%) 433232.00 (-11.96%) 461927.00 ( -6.13%) TPut 23 500602.00 ( 0.00%) 523202.00 ( 4.51%) 433320.00 (-13.44%) 454256.00 ( -9.26%) TPut 24 500408.00 ( 0.00%) 509350.00 ( 1.79%) 441878.00 (-11.70%) 460559.00 ( -7.96%) TPut 25 503390.00 ( 0.00%) 521126.00 ( 3.52%) 454313.00 ( -9.75%) 468970.00 ( -6.84%) TPut 26 514905.00 ( 0.00%) 523315.00 ( 1.63%) 453013.00 (-12.02%) 455508.00 (-11.54%) TPut 27 513125.00 ( 0.00%) 529317.00 ( 3.16%) 461561.00 (-10.05%) 463229.00 ( -9.72%) TPut 28 508313.00 ( 0.00%) 540357.00 ( 6.30%) 460727.00 ( -9.36%) 452718.00 (-10.94%) TPut 29 514726.00 ( 0.00%) 534836.00 ( 3.91%) 451867.00 (-12.21%) 449201.00 (-12.73%) TPut 30 509362.00 ( 0.00%) 526295.00 ( 3.32%) 453946.00 (-10.88%) 444615.00 (-12.71%) TPut 31 506812.00 ( 0.00%) 532603.00 ( 5.09%) 448303.00 (-11.54%) 450953.00 (-11.02%) TPut 32 500600.00 ( 0.00%) 524926.00 ( 4.86%) 452692.00 ( -9.57%) 432748.00 (-13.55%) TPut 33 491116.00 ( 0.00%) 525059.00 ( 6.91%) 436046.00 (-11.21%) 433109.00 (-11.81%) TPut 34 483206.00 ( 0.00%) 508843.00 ( 5.31%) 440762.00 ( -8.78%) 408980.00 (-15.36%) TPut 35 489281.00 ( 0.00%) 504354.00 ( 3.08%) 423368.00 (-13.47%) 408371.00 (-16.54%) TPut 36 480259.00 ( 0.00%) 489147.00 ( 1.85%) 415108.00 (-13.57%) 397698.00 (-17.19%) TPut 37 474611.00 ( 0.00%) 497076.00 ( 4.73%) 411894.00 (-13.21%) 396970.00 (-16.36%) TPut 38 470478.00 ( 0.00%) 487195.00 ( 3.55%) 407295.00 (-13.43%) 389028.00 (-17.31%) TPut 39 437255.00 ( 0.00%) 477739.00 ( 9.26%) 413837.00 ( -5.36%) 391655.00 (-10.43%) TPut 40 463513.00 ( 0.00%) 473658.00 ( 2.19%) 407789.00 (-12.02%) 383771.00 (-17.20%) TPut 41 426922.00 ( 0.00%) 446614.00 ( 4.61%) 384862.00 ( -9.85%) 376937.00 (-11.71%) TPut 42 423707.00 ( 0.00%) 442783.00 ( 4.50%) 393131.00 ( -7.22%) 389373.00 ( -8.10%) TPut 43 443489.00 ( 0.00%) 444903.00 ( 0.32%) 375795.00 (-15.26%) 377239.00 (-14.94%) TPut 44 415987.00 ( 0.00%) 432628.00 ( 4.00%) 367343.00 (-11.69%) 383026.00 ( -7.92%) TPut 45 409382.00 ( 0.00%) 424978.00 ( 3.81%) 364387.00 (-10.99%) 385429.00 ( -5.85%) TPut 46 402538.00 ( 0.00%) 393039.00 ( -2.36%) 359730.00 (-10.63%) 370411.00 ( -7.98%) TPut 47 373125.00 ( 0.00%) 406744.00 ( 9.01%) 342382.00 ( -8.24%) 375368.00 ( 0.60%) TPut 48 405485.00 ( 0.00%) 421600.00 ( 3.97%) 347063.00 (-14.41%) 400586.00 ( -1.21%)
So this is somewhat of a bad start. The initial bulk of the patches help but the grouping code did not work out as well. This tends to be a bit variable as a re-run sometimes behaves very differently. Modelling the task groupings show that threads in the same task group are still scheduled to run on CPUs from different nodes so more work is needed there.
specjbb Peaks
3.11.0-rc7 3.11.0-rc7 3.11.0-rc7 3.11.0-rc7
account-v7 lesspmd-v7 selectweight-v7 avoidmove-v7
Expctd Warehouse 48.00 ( 0.00%) 48.00 ( 0.00%) 48.00 ( 0.00%) 48.00 ( 0.00%) Expctd Peak Bops 373125.00 ( 0.00%) 406744.00 ( 9.01%) 342382.00 ( -8.24%) 375368.00 ( 0.60%) Actual Warehouse 27.00 ( 0.00%) 29.00 ( 7.41%) 28.00 ( 3.70%) 26.00 ( -3.70%) Actual Peak Bops 514905.00 ( 0.00%) 540357.00 ( 4.94%) 461561.00 (-10.36%) 468970.00 ( -8.92%) SpecJBB Bops 8275.00 ( 0.00%) 8604.00 ( 3.98%) 7083.00 (-14.40%) 8175.00 ( -1.21%) SpecJBB Bops/JVM 8275.00 ( 0.00%) 8604.00 ( 3.98%) 7083.00 (-14.40%) 8175.00 ( -1.21%)
The actual specjbb score for the overall series does not look as bad as the raw figures illustrate.
3.11.0-rc7 3.11.0-rc7 3.11.0-rc7 3.11.0-rc7
account-v7 lesspmd-v7 selectweight-v7 avoidmove-v7
User 43513.28 44403.17 44513.42 44406.55
System 871.01 122.46 107.05 116.15
Elapsed 1665.24 1664.94 1665.03 1665.06
A big positive at least is that system CPU overhead is slashed.
3.11.0-rc7 3.11.0-rc7 3.11.0-rc7 3.11.0-rc7
account-v7 lesspmd-v7 selectweight-v7 avoidmove-v7
Compaction stalls 0 0 0 0 Compaction success 0 0 0 0 Compaction failures 0 0 0 0 Page migrate success 133385393 14958732 9859116 12092458 Page migrate failure 0 0 0 0 Compaction pages isolated 0 0 0 0 Compaction migrate scanned 0 0 0 0 Compaction free scanned 0 0 0 0 Compaction cost 138454 15527 10233 12551 NUMA PTE updates 19952605 712634 674115 730464 NUMA hint faults 4113211 710022 668011 729294 NUMA hint local faults 1197939 274740 251230 273679
NUMA hint local percent 29 38 37 37
NUMA pages migrated 133385393 14958732 9859116 12092458
AutoNUMA cost 23240 3839 3532 3881
And the source of the reduction is obvious here from the much smaller number of PTE updates and hinting faults.
This is SpecJBB running on a 4-socket machine with THP enabled and one JVM running per node on the system.
specjbb
3.11.0-rc7 3.11.0-rc7 3.11.0-rc7 3.11.0-rc7
account-v7 lesspmd-v7 selectweight-v7 avoidmove-v7
Mean 1 29995.75 ( 0.00%) 30321.50 ( 1.09%) 29457.25 ( -1.80%) 30791.75 ( 2.65%) Mean 2 62699.25 ( 0.00%) 60564.75 ( -3.40%) 59721.00 ( -4.75%) 61050.00 ( -2.63%) Mean 3 88312.75 ( 0.00%) 89286.50 ( 1.10%) 88451.50 ( 0.16%) 90461.75 ( 2.43%) Mean 4 117827.00 ( 0.00%) 115583.00 ( -1.90%) 116043.00 ( -1.51%) 114945.75 ( -2.45%) Mean 5 139419.00 ( 0.00%) 137869.25 ( -1.11%) 137761.00 ( -1.19%) 136841.25 ( -1.85%) Mean 6 156185.25 ( 0.00%) 155811.50 ( -0.24%) 151628.50 ( -2.92%) 149850.25 ( -4.06%) Mean 7 162258.25 ( 0.00%) 160665.25 ( -0.98%) 154775.25 ( -4.61%) 154356.25 ( -4.87%) Mean 8 160665.00 ( 0.00%) 160376.75 ( -0.18%) 150849.00 ( -6.11%) 154266.50 ( -3.98%) Mean 9 156048.00 ( 0.00%) 159689.75 ( 2.33%) 150347.00 ( -3.65%) 150804.75 ( -3.36%) Mean 10 144640.75 ( 0.00%) 153683.50 ( 6.25%) 146165.50 ( 1.05%) 143256.00 ( -0.96%) Mean 11 136418.75 ( 0.00%) 146141.75 ( 7.13%) 139216.75 ( 2.05%) 137435.00 ( 0.74%) Mean 12 132808.00 ( 0.00%) 141567.75 ( 6.60%) 131523.25 ( -0.97%) 139129.50 ( 4.76%) Mean 13 126834.75 ( 0.00%) 140738.50 ( 10.96%) 124446.25 ( -1.88%) 138181.50 ( 8.95%) Mean 14 127837.25 ( 0.00%) 140882.00 ( 10.20%) 121495.50 ( -4.96%) 128275.75 ( 0.34%) Mean 15 122268.50 ( 0.00%) 139983.50 ( 14.49%) 115737.25 ( -5.34%) 119838.75 ( -1.99%) Mean 16 118739.25 ( 0.00%) 142654.25 ( 20.14%) 110902.25 ( -6.60%) 123369.50 ( 3.90%) Mean 17 117972.75 ( 0.00%) 136969.50 ( 16.10%) 108398.00 ( -8.12%) 115575.00 ( -2.03%) Mean 18 116308.50 ( 0.00%) 134009.50 ( 15.22%) 109094.25 ( -6.20%) 118385.75 ( 1.79%) Mean 19 114594.75 ( 0.00%) 125941.75 ( 9.90%) 108366.75 ( -5.43%) 117998.00 ( 2.97%) Mean 20 116338.50 ( 0.00%) 121586.50 ( 4.51%) 110267.25 ( -5.22%) 121703.00 ( 4.61%) Mean 21 114274.00 ( 0.00%) 118586.00 ( 3.77%) 105316.25 ( -7.84%) 112591.75 ( -1.47%) Mean 22 113135.00 ( 0.00%) 121886.75 ( 7.74%) 108124.25 ( -4.43%) 107672.00 ( -4.83%) Mean 23 109514.25 ( 0.00%) 117894.25 ( 7.65%) 111499.50 ( 1.81%) 108045.75 ( -1.34%) Mean 24 112897.00 ( 0.00%) 119902.75 ( 6.21%) 110615.50 ( -2.02%) 117146.00 ( 3.76%) Mean 25 107127.75 ( 0.00%) 125763.50 ( 17.40%) 107750.75 ( 0.58%) 116425.75 ( 8.68%) Mean 26 109338.75 ( 0.00%) 119034.00 ( 8.87%) 105875.25 ( -3.17%) 116591.00 ( 6.63%) Mean 27 110967.75 ( 0.00%) 122720.50 ( 10.59%) 99660.75 (-10.19%) 118399.25 ( 6.70%) Mean 28 116559.50 ( 0.00%) 121524.50 ( 4.26%) 98095.50 (-15.84%) 116433.50 ( -0.11%) Mean 29 113278.00 ( 0.00%) 115992.75 ( 2.40%) 101014.00 (-10.83%) 122954.25 ( 8.54%) Mean 30 110273.75 ( 0.00%) 112436.50 ( 1.96%) 103679.75 ( -5.98%) 127165.50 ( 15.32%) Mean 31 107409.50 ( 0.00%) 120160.00 ( 11.87%) 101122.75 ( -5.85%) 128566.25 ( 19.70%) Mean 32 105624.00 ( 0.00%) 122808.50 ( 16.27%) 100410.75 ( -4.94%) 126009.50 ( 19.30%) Mean 33 107521.75 ( 0.00%) 118049.50 ( 9.79%) 97788.25 ( -9.05%) 124172.00 ( 15.49%) Mean 34 108135.75 ( 0.00%) 118198.75 ( 9.31%) 99215.25 ( -8.25%) 129010.75 ( 19.30%) Mean 35 104407.75 ( 0.00%) 115090.50 ( 10.23%) 97804.00 ( -6.32%) 126019.75 ( 20.70%) Mean 36 101119.00 ( 0.00%) 118554.75 ( 17.24%) 101608.00 ( 0.48%) 126106.00 ( 24.71%) Mean 37 104228.25 ( 0.00%) 123893.25 ( 18.87%) 99277.75 ( -4.75%) 122410.25 ( 17.44%) Mean 38 104402.50 ( 0.00%) 118543.50 ( 13.54%) 97255.00 ( -6.85%) 118682.75 ( 13.68%) Mean 39 100158.50 ( 0.00%) 116866.00 ( 16.68%) 99918.00 ( -0.24%) 122019.75 ( 21.83%) Mean 40 101911.75 ( 0.00%) 117276.25 ( 15.08%) 98766.25 ( -3.09%) 121322.00 ( 19.05%) Mean 41 104757.50 ( 0.00%) 116656.75 ( 11.36%) 97970.25 ( -6.48%) 121403.00 ( 15.89%) Mean 42 104782.50 ( 0.00%) 116385.25 ( 11.07%) 96897.25 ( -7.53%) 118765.25 ( 13.34%) Mean 43 97073.00 ( 0.00%) 113745.50 ( 17.18%) 93433.00 ( -3.75%) 118571.25 ( 22.15%) Mean 44 99739.00 ( 0.00%) 116286.00 ( 16.59%) 96193.50 ( -3.55%) 116149.75 ( 16.45%) Mean 45 104422.25 ( 0.00%) 109978.25 ( 5.32%) 95737.50 ( -8.32%) 113604.75 ( 8.79%) Mean 46 103389.25 ( 0.00%) 110703.00 ( 7.07%) 93711.50 ( -9.36%) 110550.75 ( 6.93%) Mean 47 96092.25 ( 0.00%) 108942.50 ( 13.37%) 94220.50 ( -1.95%) 104079.00 ( 8.31%) Mean 48 97596.25 ( 0.00%) 109194.00 ( 11.88%) 101071.25 ( 3.56%) 101543.00 ( 4.04%) Stddev 1 1326.20 ( 0.00%) 1351.58 ( -1.91%) 1525.30 (-15.01%) 1048.89 ( 20.91%) Stddev 2 1837.05 ( 0.00%) 1538.27 ( 16.26%) 919.58 ( 49.94%) 1974.67 ( -7.49%) Stddev 3 1267.24 ( 0.00%) 2599.37 (-105.12%) 2323.12 (-83.32%) 2091.33 (-65.03%) Stddev 4 6125.28 ( 0.00%) 2980.50 ( 51.34%) 1706.84 ( 72.13%) 2497.81 ( 59.22%) Stddev 5 6161.12 ( 0.00%) 2495.59 ( 59.49%) 2466.47 ( 59.97%) 3077.78 ( 50.05%) Stddev 6 5784.16 ( 0.00%) 4799.20 ( 17.03%) 4580.83 ( 20.80%) 2889.81 ( 50.04%) Stddev 7 6607.07 ( 0.00%) 1167.21 ( 82.33%) 6196.26 ( 6.22%) 4385.20 ( 33.63%) Stddev 8 1671.12 ( 0.00%) 6631.06 (-296.80%) 6812.80 (-307.68%) 8598.19 (-414.52%) Stddev 9 6052.25 ( 0.00%) 6954.93 (-14.91%) 6382.84 ( -5.46%) 8987.78 (-48.50%) Stddev 10 11473.39 ( 0.00%) 4442.38 ( 61.28%) 6772.50 ( 40.97%) 16758.82 (-46.07%) Stddev 11 7093.02 ( 0.00%) 4526.31 ( 36.19%) 9026.86 (-27.26%) 13353.17 (-88.26%) Stddev 12 3865.06 ( 0.00%) 2743.41 ( 29.02%) 15584.41 (-303.21%) 14112.46 (-265.13%) Stddev 13 2777.36 ( 0.00%) 1050.96 ( 62.16%) 16286.28 (-486.39%) 8243.38 (-196.81%) Stddev 14 1795.89 ( 0.00%) 536.93 ( 70.10%) 13502.75 (-651.87%) 6328.98 (-252.42%) Stddev 15 2250.85 ( 0.00%) 1135.62 ( 49.55%) 9908.63 (-340.22%) 11274.74 (-400.91%) Stddev 16 1963.42 ( 0.00%) 379.50 ( 80.67%) 9645.69 (-391.27%) 2679.87 (-36.49%) Stddev 17 1592.42 ( 0.00%) 1388.57 ( 12.80%) 6322.29 (-297.02%) 3768.27 (-136.64%) Stddev 18 3317.92 ( 0.00%) 721.81 ( 78.25%) 3065.44 ( 7.61%) 6375.92 (-92.17%) Stddev 19 4525.33 ( 0.00%) 3273.36 ( 27.67%) 5565.31 (-22.98%) 3248.71 ( 28.21%) Stddev 20 4140.94 ( 0.00%) 2332.35 ( 43.68%) 8000.27 (-93.20%) 6237.91 (-50.64%) Stddev 21 1515.71 ( 0.00%) 3309.22 (-118.33%) 6587.02 (-334.58%) 10217.84 (-574.13%) Stddev 22 5498.36 ( 0.00%) 2437.41 ( 55.67%) 7920.50 (-44.05%) 8414.84 (-53.04%) Stddev 23 5637.68 ( 0.00%) 1832.68 ( 67.49%) 6543.07 (-16.06%) 5976.59 ( -6.01%) Stddev 24 4862.89 ( 0.00%) 6295.82 (-29.47%) 9229.15 (-89.79%) 9046.57 (-86.03%) Stddev 25 1725.07 ( 0.00%) 2986.87 (-73.15%) 13679.77 (-693.00%) 9521.44 (-451.95%) Stddev 26 4590.06 ( 0.00%) 1862.17 ( 59.43%) 10773.97 (-134.72%) 5417.65 (-18.03%) Stddev 27 6060.43 ( 0.00%) 1567.32 ( 74.14%) 10217.36 (-68.59%) 2934.56 ( 51.58%) Stddev 28 2742.94 ( 0.00%) 2533.06 ( 7.65%) 11375.97 (-314.74%) 3713.72 (-35.39%) Stddev 29 3878.01 ( 0.00%) 783.58 ( 79.79%) 8718.86 (-124.83%) 2870.90 ( 25.97%) Stddev 30 4446.49 ( 0.00%) 852.75 ( 80.82%) 5318.24 (-19.61%) 2174.56 ( 51.09%) Stddev 31 3825.27 ( 0.00%) 876.75 ( 77.08%) 7412.96 (-93.79%) 1517.78 ( 60.32%) Stddev 32 8118.60 ( 0.00%) 1367.48 ( 83.16%) 5757.34 ( 29.08%) 1025.48 ( 87.37%) Stddev 33 3237.05 ( 0.00%) 3807.47 (-17.62%) 7493.40 (-131.49%) 4600.54 (-42.12%) Stddev 34 7413.56 ( 0.00%) 3599.54 ( 51.45%) 8514.89 (-14.86%) 2999.21 ( 59.54%) Stddev 35 6061.77 ( 0.00%) 3756.88 ( 38.02%) 5594.20 ( 7.71%) 4241.61 ( 30.03%) Stddev 36 5836.80 ( 0.00%) 2944.03 ( 49.56%) 10641.97 (-82.33%) 1267.44 ( 78.29%) Stddev 37 2719.65 ( 0.00%) 3819.92 (-40.46%) 4075.76 (-49.86%) 2604.21 ( 4.24%) Stddev 38 3267.94 ( 0.00%) 2148.38 ( 34.26%) 5219.19 (-59.71%) 4865.10 (-48.87%) Stddev 39 3596.06 ( 0.00%) 1042.13 ( 71.02%) 5891.17 (-63.82%) 3067.42 ( 14.70%) Stddev 40 4303.03 ( 0.00%) 2518.02 ( 41.48%) 5279.70 (-22.70%) 1750.86 ( 59.31%) Stddev 41 10269.08 ( 0.00%) 3602.25 ( 64.92%) 5907.68 ( 42.47%) 3163.17 ( 69.20%) Stddev 42 3221.41 ( 0.00%) 3707.32 (-15.08%) 6926.80 (-115.02%) 2555.18 ( 20.68%) Stddev 43 7203.43 ( 0.00%) 3082.74 ( 57.20%) 6537.72 ( 9.24%) 3912.25 ( 45.69%) Stddev 44 6164.48 ( 0.00%) 2946.14 ( 52.21%) 4702.32 ( 23.72%) 3228.17 ( 47.63%) Stddev 45 7696.65 ( 0.00%) 2461.14 ( 68.02%) 4697.11 ( 38.97%) 4675.68 ( 39.25%) Stddev 46 6989.59 ( 0.00%) 3713.96 ( 46.86%) 5105.63 ( 26.95%) 5008.38 ( 28.35%) Stddev 47 5580.13 ( 0.00%) 4025.00 ( 27.87%) 4034.38 ( 27.70%) 5538.34 ( 0.75%) Stddev 48 5647.24 ( 0.00%) 1694.00 ( 70.00%) 2980.82 ( 47.22%) 8123.60 (-43.85%) TPut 1 119983.00 ( 0.00%) 121286.00 ( 1.09%) 117829.00 ( -1.80%) 123167.00 ( 2.65%) TPut 2 250797.00 ( 0.00%) 242259.00 ( -3.40%) 238884.00 ( -4.75%) 244200.00 ( -2.63%) TPut 3 353251.00 ( 0.00%) 357146.00 ( 1.10%) 353806.00 ( 0.16%) 361847.00 ( 2.43%) TPut 4 471308.00 ( 0.00%) 462332.00 ( -1.90%) 464172.00 ( -1.51%) 459783.00 ( -2.45%) TPut 5 557676.00 ( 0.00%) 551477.00 ( -1.11%) 551044.00 ( -1.19%) 547365.00 ( -1.85%) TPut 6 624741.00 ( 0.00%) 623246.00 ( -0.24%) 606514.00 ( -2.92%) 599401.00 ( -4.06%) TPut 7 649033.00 ( 0.00%) 642661.00 ( -0.98%) 619101.00 ( -4.61%) 617425.00 ( -4.87%) TPut 8 642660.00 ( 0.00%) 641507.00 ( -0.18%) 603396.00 ( -6.11%) 617066.00 ( -3.98%) TPut 9 624192.00 ( 0.00%) 638759.00 ( 2.33%) 601388.00 ( -3.65%) 603219.00 ( -3.36%) TPut 10 578563.00 ( 0.00%) 614734.00 ( 6.25%) 584662.00 ( 1.05%) 573024.00 ( -0.96%) TPut 11 545675.00 ( 0.00%) 584567.00 ( 7.13%) 556867.00 ( 2.05%) 549740.00 ( 0.74%) TPut 12 531232.00 ( 0.00%) 566271.00 ( 6.60%) 526093.00 ( -0.97%) 556518.00 ( 4.76%) TPut 13 507339.00 ( 0.00%) 562954.00 ( 10.96%) 497785.00 ( -1.88%) 552726.00 ( 8.95%) TPut 14 511349.00 ( 0.00%) 563528.00 ( 10.20%) 485982.00 ( -4.96%) 513103.00 ( 0.34%) TPut 15 489074.00 ( 0.00%) 559934.00 ( 14.49%) 462949.00 ( -5.34%) 479355.00 ( -1.99%) TPut 16 474957.00 ( 0.00%) 570617.00 ( 20.14%) 443609.00 ( -6.60%) 493478.00 ( 3.90%) TPut 17 471891.00 ( 0.00%) 547878.00 ( 16.10%) 433592.00 ( -8.12%) 462300.00 ( -2.03%) TPut 18 465234.00 ( 0.00%) 536038.00 ( 15.22%) 436377.00 ( -6.20%) 473543.00 ( 1.79%) TPut 19 458379.00 ( 0.00%) 503767.00 ( 9.90%) 433467.00 ( -5.43%) 471992.00 ( 2.97%) TPut 20 465354.00 ( 0.00%) 486346.00 ( 4.51%) 441069.00 ( -5.22%) 486812.00 ( 4.61%) TPut 21 457096.00 ( 0.00%) 474344.00 ( 3.77%) 421265.00 ( -7.84%) 450367.00 ( -1.47%) TPut 22 452540.00 ( 0.00%) 487547.00 ( 7.74%) 432497.00 ( -4.43%) 430688.00 ( -4.83%) TPut 23 438057.00 ( 0.00%) 471577.00 ( 7.65%) 445998.00 ( 1.81%) 432183.00 ( -1.34%) TPut 24 451588.00 ( 0.00%) 479611.00 ( 6.21%) 442462.00 ( -2.02%) 468584.00 ( 3.76%) TPut 25 428511.00 ( 0.00%) 503054.00 ( 17.40%) 431003.00 ( 0.58%) 465703.00 ( 8.68%) TPut 26 437355.00 ( 0.00%) 476136.00 ( 8.87%) 423501.00 ( -3.17%) 466364.00 ( 6.63%) TPut 27 443871.00 ( 0.00%) 490882.00 ( 10.59%) 398643.00 (-10.19%) 473597.00 ( 6.70%) TPut 28 466238.00 ( 0.00%) 486098.00 ( 4.26%) 392382.00 (-15.84%) 465734.00 ( -0.11%) TPut 29 453112.00 ( 0.00%) 463971.00 ( 2.40%) 404056.00 (-10.83%) 491817.00 ( 8.54%) TPut 30 441095.00 ( 0.00%) 449746.00 ( 1.96%) 414719.00 ( -5.98%) 508662.00 ( 15.32%) TPut 31 429638.00 ( 0.00%) 480640.00 ( 11.87%) 404491.00 ( -5.85%) 514265.00 ( 19.70%) TPut 32 422496.00 ( 0.00%) 491234.00 ( 16.27%) 401643.00 ( -4.94%) 504038.00 ( 19.30%) TPut 33 430087.00 ( 0.00%) 472198.00 ( 9.79%) 391153.00 ( -9.05%) 496688.00 ( 15.49%) TPut 34 432543.00 ( 0.00%) 472795.00 ( 9.31%) 396861.00 ( -8.25%) 516043.00 ( 19.30%) TPut 35 417631.00 ( 0.00%) 460362.00 ( 10.23%) 391216.00 ( -6.32%) 504079.00 ( 20.70%) TPut 36 404476.00 ( 0.00%) 474219.00 ( 17.24%) 406432.00 ( 0.48%) 504424.00 ( 24.71%) TPut 37 416913.00 ( 0.00%) 495573.00 ( 18.87%) 397111.00 ( -4.75%) 489641.00 ( 17.44%) TPut 38 417610.00 ( 0.00%) 474174.00 ( 13.54%) 389020.00 ( -6.85%) 474731.00 ( 13.68%) TPut 39 400634.00 ( 0.00%) 467464.00 ( 16.68%) 399672.00 ( -0.24%) 488079.00 ( 21.83%) TPut 40 407647.00 ( 0.00%) 469105.00 ( 15.08%) 395065.00 ( -3.09%) 485288.00 ( 19.05%) TPut 41 419030.00 ( 0.00%) 466627.00 ( 11.36%) 391881.00 ( -6.48%) 485612.00 ( 15.89%) TPut 42 419130.00 ( 0.00%) 465541.00 ( 11.07%) 387589.00 ( -7.53%) 475061.00 ( 13.34%) TPut 43 388292.00 ( 0.00%) 454982.00 ( 17.18%) 373732.00 ( -3.75%) 474285.00 ( 22.15%) TPut 44 398956.00 ( 0.00%) 465144.00 ( 16.59%) 384774.00 ( -3.55%) 464599.00 ( 16.45%) TPut 45 417689.00 ( 0.00%) 439913.00 ( 5.32%) 382950.00 ( -8.32%) 454419.00 ( 8.79%) TPut 46 413557.00 ( 0.00%) 442812.00 ( 7.07%) 374846.00 ( -9.36%) 442203.00 ( 6.93%) TPut 47 384369.00 ( 0.00%) 435770.00 ( 13.37%) 376882.00 ( -1.95%) 416316.00 ( 8.31%) TPut 48 390385.00 ( 0.00%) 436776.00 ( 11.88%) 404285.00 ( 3.56%) 406172.00 ( 4.04%)
This is looking a bit better overall. One would generally expect this JVM configuration to be handled better because there are far few problems dealing with shared pages.
specjbb Peaks
3.11.0-rc7 3.11.0-rc7 3.11.0-rc7 3.11.0-rc7
account-v7 lesspmd-v7 selectweight-v7 avoidmove-v7
Expctd Warehouse 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%) Expctd Peak Bops 545675.00 ( 0.00%) 584567.00 ( 7.13%) 556867.00 ( 2.05%) 549740.00 ( 0.74%) Actual Warehouse 8.00 ( 0.00%) 8.00 ( 0.00%) 8.00 ( 0.00%) 8.00 ( 0.00%) Actual Peak Bops 649033.00 ( 0.00%) 642661.00 ( -0.98%) 619101.00 ( -4.61%) 617425.00 ( -4.87%) SpecJBB Bops 474931.00 ( 0.00%) 523877.00 ( 10.31%) 454089.00 ( -4.39%) 482435.00 ( 1.58%) SpecJBB Bops/JVM 118733.00 ( 0.00%) 130969.00 ( 10.31%) 113522.00 ( -4.39%) 120609.00 ( 1.58%)
Because the specjvm score is based on lower number of clients this does not look as impressive but at least the overall series does not have a worse specjbb score.
3.11.0-rc7 3.11.0-rc7 3.11.0-rc7 3.11.0-rc7
account-v7 lesspmd-v7 selectweight-v7 avoidmove-v7
User 464762.73 474999.54 474756.33 475883.65
System 10593.13 725.15 752.36 689.00
Elapsed 10409.45 10414.85 10416.46 10441.17
On the other hand, look at the system CPU overhead. We are getting comparable or better performance at a small fraction of the cost.
3.11.0-rc7 3.11.0-rc7 3.11.0-rc7 3.11.0-rc7
account-v7 lesspmd-v7 selectweight-v7 avoidmove-v7
Compaction stalls 0 0 0 0 Compaction success 0 0 0 0 Compaction failures 0 0 0 0 Page migrate success 1339274585 55904453 50672239 48174428 Page migrate failure 0 0 0 0 Compaction pages isolated 0 0 0 0 Compaction migrate scanned 0 0 0 0 Compaction free scanned 0 0 0 0 Compaction cost 1390167 58028 52597 50005 NUMA PTE updates 501107230 9187590 8925627 9120756 NUMA hint faults 69895484 9184458 8917340 9096029 NUMA hint local faults 21848214 3778721 3832832 4025324
NUMA hint local percent 31 41 42 44
NUMA pages migrated 1339274585 55904453 50672239 48174428
AutoNUMA cost 378431 47048 45611 46459
And again the reduced cost is from massively reduced numbers of PTE updates and faults. This may mean some workloads may converge slower but the system will not get hammered constantly trying to converge either.
3.11.0-rc7 3.11.0-rc7 3.11.0-rc7 3.11.0-rc7
account-v7 lesspmd-v7 selectweight-v7 avoidmove-v7
User NUMA01 53586.49 ( 0.00%) 57956.00 ( -8.15%) 38838.20 ( 27.52%) 45977.10 ( 14.20%) User NUMA01_THEADLOCAL 16956.29 ( 0.00%) 17070.87 ( -0.68%) 16972.80 ( -0.10%) 17262.89 ( -1.81%) User NUMA02 2024.02 ( 0.00%) 2022.45 ( 0.08%) 2035.17 ( -0.55%) 2013.42 ( 0.52%) User NUMA02_SMT 968.96 ( 0.00%) 992.63 ( -2.44%) 979.86 ( -1.12%) 1379.96 (-42.42%) System NUMA01 1442.97 ( 0.00%) 542.69 ( 62.39%) 309.92 ( 78.52%) 405.48 ( 71.90%) System NUMA01_THEADLOCAL 117.16 ( 0.00%) 72.08 ( 38.48%) 75.60 ( 35.47%) 91.56 ( 21.85%) System NUMA02 7.12 ( 0.00%) 7.86 (-10.39%) 7.84 (-10.11%) 6.38 ( 10.39%) System NUMA02_SMT 8.49 ( 0.00%) 3.74 ( 55.95%) 3.53 ( 58.42%) 6.26 ( 26.27%) Elapsed NUMA01 1216.88 ( 0.00%) 1372.29 (-12.77%) 918.05 ( 24.56%) 1065.63 ( 12.43%) Elapsed NUMA01_THEADLOCAL 375.15 ( 0.00%) 388.68 ( -3.61%) 386.02 ( -2.90%) 382.63 ( -1.99%) Elapsed NUMA02 48.61 ( 0.00%) 52.19 ( -7.36%) 49.65 ( -2.14%) 51.85 ( -6.67%) Elapsed NUMA02_SMT 49.68 ( 0.00%) 51.23 ( -3.12%) 50.36 ( -1.37%) 80.91 (-62.86%) CPU NUMA01 4522.00 ( 0.00%) 4262.00 ( 5.75%) 4264.00 ( 5.71%) 4352.00 ( 3.76%) CPU NUMA01_THEADLOCAL 4551.00 ( 0.00%) 4410.00 ( 3.10%) 4416.00 ( 2.97%) 4535.00 ( 0.35%) CPU NUMA02 4178.00 ( 0.00%) 3890.00 ( 6.89%) 4114.00 ( 1.53%) 3895.00 ( 6.77%) CPU NUMA02_SMT 1967.00 ( 0.00%) 1944.00 ( 1.17%) 1952.00 ( 0.76%) 1713.00 ( 12.91%)
Elapsed figures here are poor. The numa01 test case saw an improvement but it's an adverse workload and not that interesting per-se. Its main benefit is from the reduction of system overhead. numa02_smt suffered badly due to the last patch in the series that needs addressing.
nas-omp
3.11.0-rc7 3.11.0-rc7 3.11.0-rc7 3.11.0-rc7
account-v7 lesspmd-v7 selectweight-v7 avoidmove-v7
Time bt.C 187.22 ( 0.00%) 188.02 ( -0.43%) 188.19 ( -0.52%) 197.49 ( -5.49%) Time cg.C 61.58 ( 0.00%) 49.64 ( 19.39%) 61.44 ( 0.23%) 56.84 ( 7.70%) Time ep.C 13.28 ( 0.00%) 13.28 ( 0.00%) 14.05 ( -5.80%) 13.34 ( -0.45%) Time ft.C 38.35 ( 0.00%) 37.39 ( 2.50%) 35.08 ( 8.53%) 37.05 ( 3.39%) Time is.C 2.12 ( 0.00%) 1.75 ( 17.45%) 2.20 ( -3.77%) 2.14 ( -0.94%) Time lu.C 180.71 ( 0.00%) 183.01 ( -1.27%) 186.64 ( -3.28%) 169.77 ( 6.05%) Time mg.C 32.02 ( 0.00%) 31.57 ( 1.41%) 29.45 ( 8.03%) 31.98 ( 0.12%) Time sp.C 413.92 ( 0.00%) 396.36 ( 4.24%) 400.92 ( 3.14%) 388.54 ( 6.13%) Time ua.C 200.27 ( 0.00%) 204.68 ( -2.20%) 211.46 ( -5.59%) 194.92 ( 2.67%)
This is Nasa Parallel Benchmark (npb) running with openmp. Some small improvements.
3.11.0-rc7 3.11.0-rc7 3.11.0-rc7 3.11.0-rc7
account-v7 lesspmd-v7 selectweight-v7 avoidmove-v7
User 47694.80 47262.68 47998.27 46282.27
System 421.02 136.12 129.91 131.36
Elapsed 1265.34 1242.74 1267.06 1229.70
With large reductions of system CPU usage.
So overall it is still a bit of a mixed bag. There is not a universal performance win but there are massive reductions in system CPU overhead which may of big benefit on larger machines meaning the series is still worth considering. The ratio of local/remote NUMA hinting faults is still very slow and the fact that there are tasks sharing a numa group running on different nodes should be examined closer.
Documentation/sysctl/kernel.txt | 73 ++ arch/x86/mm/numa.c | 6 +- fs/proc/array.c | 2 + include/linux/migrate.h | 7 +- include/linux/mm.h | 107 ++- include/linux/mm_types.h | 14 +- include/linux/page-flags-layout.h | 28 +- include/linux/sched.h | 45 +- include/linux/stop_machine.h | 1 + kernel/bounds.c | 4 + kernel/fork.c | 5 +- kernel/sched/core.c | 196 ++++- kernel/sched/debug.c | 60 +- kernel/sched/fair.c | 1523 ++++++++++++++++++++++++++++++------- kernel/sched/features.h | 19 +- kernel/sched/idle_task.c | 2 +- kernel/sched/rt.c | 5 +- kernel/sched/sched.h | 19 +- kernel/sched/stop_task.c | 2 +- kernel/stop_machine.c | 272 ++++--- kernel/sysctl.c | 7 + lib/vsprintf.c | 5 + mm/huge_memory.c | 103 ++- mm/memory.c | 95 ++- mm/mempolicy.c | 24 +- mm/migrate.c | 21 +- mm/mm_init.c | 18 +- mm/mmzone.c | 14 +- mm/mprotect.c | 70 +- mm/page_alloc.c | 4 +- 30 files changed, 2147 insertions(+), 604 deletions(-)
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 1/50 sched: monolithic code dump of what is being pushed upstream
From: Peter Zijlstra <[email protected]>
So I have the below patch in front of all your patches. It contains the 10 or so sched,fair patches I posted to lkml the other day.
I used these to poke at the group_imb crud, am now digging through traces of perf bench numa to see if there's anything else I need.
like said on IRC: I boot with ftrace=nop to ensure we allocate properly sized trace buffers. This can also be done at runtime by switching active tracer -- this allocates the default buffer size, or by explicitly setting a per-cpu buffer size in /debug/tracing/buffer_size_kb. By default the thing allocates a single page per cpu or something uselessly small like that.
I then run a benchmark and at an appropriate time (eg. when I see something 'weird' happen) I do something like:
echo 0 > /debug/tracing/tracing_on # disable writing into the buffers cat /debug/tracing/trace > ~/trace # dump to file echo 0 > /debug/tracing/trace # reset buffers echo 1 > /debug/tracing/tracing_on # enable writing to the buffers
[ Note I mount debugfs at /debug, this is not the default location but I
think the rest of the world is wrong ;-) ]
Also, the brain seems to adapt once you're staring at them for longer than a day -- yay for human pattern recognition skillz.
Ingo tends to favour more verbose dumps, I tend to favour minimal dumps.. whatever works for you is something you'll learn with experience.
--- arch/x86/mm/numa.c | 6 +- kernel/sched/core.c | 18 +- kernel/sched/fair.c | 498 ++++++++++++++++++++++++++++----------------------- kernel/sched/sched.h | 1 + lib/vsprintf.c | 5 + 5 files changed, 288 insertions(+), 240 deletions(-)
arch/x86/mm/numa.c +2 -4
@@ -737,7 +737,6 @@ int early_cpu_to_node(int cpu)void debug_cpumask_set_cpu (int cpu,int node,bool enable) {struct cpumask *mask; -char buf[64 ];if (node ==NUMA_NO_NODE ) {/* early_cpu_to_node() already emits a warning and trace */
@@ -755,10 +754,9 @@ void debug_cpumask_set_cpu(int cpu, int node, bool enable)else cpumask_clear_cpu (cpu, mask); -cpulist_scnprintf (buf,sizeof (buf), mask); -printk (KERN_DEBUG "%s cpu %d node %d: mask now %s\n" , +printk (KERN_DEBUG "%s cpu %d node %d: mask now %pc\n" , enable ?"numa_add_cpu" :"numa_remove_cpu" , - cpu, node, buf); + cpu, node, mask);return ; }
kernel/sched/core.c +11 -7
@@ -4809,9 +4809,7 @@ static int sched_domain_debug_one(struct sched_domain *sd, int cpu, int level, struct cpumask *groupmask) {struct sched_group *group = sd->groups; -char str[256 ]; -cpulist_scnprintf (str,sizeof (str),sched_domain_span (sd));cpumask_clear (groupmask);printk (KERN_DEBUG "%*s domain %d: " , level,"" , level);
@@ -4824,7 +4822,7 @@ static int sched_domain_debug_one(struct sched_domain *sd, int cpu, int level,return -1 ; } -printk (KERN_CONT "span %s level %s\n" , str, sd->name); +printk (KERN_CONT "span %pc level %s\n" ,sched_domain_span (sd), sd->name);if (!cpumask_test_cpu (cpu,sched_domain_span (sd))) { printk(KERN_ERR "ERROR: domain->span does not contain "
@@ -4870,9 +4868,7 @@ static int sched_domain_debug_one(struct sched_domain *sd, int cpu, int level,cpumask_or (groupmask, groupmask,sched_group_cpus (group)); -cpulist_scnprintf (str,sizeof (str),sched_group_cpus (group)); - -printk (KERN_CONT " %s" , str); +printk (KERN_CONT " %pc" ,sched_group_cpus (group));if (group->sgp->power !=SCHED_POWER_SCALE ) {printk (KERN_CONT " (cpu_power = %d)" , group->sgp->power);
@@ -4964,7 +4960,8 @@ sd_parent_degenerate(struct sched_domain *sd, struct sched_domain *parent)SD_BALANCE_FORK |SD_BALANCE_EXEC |SD_SHARE_CPUPOWER | - SD_SHARE_PKG_RESOURCES); +SD_SHARE_PKG_RESOURCES | + SD_PREFER_SIBLING);if (nr_node_ids ==1 ) pflags &= ~SD_SERIALIZE ; }
@@ -5168,6 +5165,13 @@ cpu_attach_domain(struct sched_domain *sd, struct root_domain *rd, int cpu) tmp->parent = parent->parent;if (parent->parent) parent->parent->child = tmp; +/* +* Transfer SD_PREFER_SIBLING down in case of a +* degenerate parent; the spans match for this +* so the property transfers. +*/ +if (parent->flags &SD_PREFER_SIBLING ) + tmp->flags |=SD_PREFER_SIBLING ;destroy_sched_domain (parent, cpu); }else tmp = tmp->parent;
kernel/sched/fair.c +269 -229
840 lines not on this page · raw message
kernel/sched/sched.h +1 -0
@@ -605,6 +605,7 @@ struct sched_group_power {*/ unsigned int power, power_orig;unsigned long next_update; +int imbalance;/* XXX unrelated to power but shared group state */ /* * Number of busy cpus in this group. */
lib/vsprintf.c +5 -0
@@ -26,6 +26,7 @@#include <linux/math64.h> #include <linux/uaccess.h> #include <linux/ioport.h> +#include <linux/cpumask.h> #include <net/addrconf.h> #include <asm/page.h> /* for PAGE_SIZE */
@@ -1142,6 +1143,7 @@ int kptr_restrict __read_mostly;* The maximum supported length is 64 bytes of the input. Consider * to use print_hex_dump() for the larger input. * - 'a' For a phys_addr_t type and its derivative types (passed by reference) +* - 'c' For a cpumask list * * Note: The difference between 'S' and 'F' is that on ia64 and ppc64 * function pointers are really function descriptors, which contain a
@@ -1253,6 +1255,8 @@ char *pointer(const char *fmt, char *buf, char *end, void *ptr, spec.base =16 ;return number (buf, end, (unsigned long long ) *((phys_addr_t *)ptr), spec); +case 'c' : +return buf +cpulist_scnprintf (buf, end - buf, ptr); } spec.flags |=SMALL ;if (spec.field_width ==-1 ) {
@@ -1494,6 +1498,7 @@ qualifier:* case. * %*ph[CDN] a variable-length hex string with a separator (supports up to 64 * bytes of the input) +* %pc print a cpumask as comma-separated list * %n is ignored * * ** Please update Documentation/printk-formats.txt when making changes **
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 01/50] sched: monolithic code dump of what is being pushed upstream
Quoted 28 lines · Mel Gorman
kernel/sched/fair.c +24
-0@@ -5045,15 +5038,50 @@ static int need_active_balance(struct lb_env *env)static int active_load_balance_cpu_stop (void *data); +static int should_we_balance (struct lb_env *env) +{ +struct sched_group *sg = env->sd->groups; +struct cpumask *sg_cpus, *sg_mask; +int cpu, balance_cpu =-1 ; + +/* +* In the newly idle case, we will allow all the cpu's +* to do the newly idle load balance. +*/ +if (env->idle ==CPU_NEWLY_IDLE ) +return 1 ; + + sg_cpus =sched_group_cpus (sg); + sg_mask =sched_group_mask (sg); +/* Try to find first idle cpu */ +for_each_cpu_and (cpu, sg_cpus, env->cpus) { +if (!cpumask_test_cpu (cpu, sg_mask) || !idle_cpu (cpu)) +continue ; + + balance_cpu = cpu; +break ; + } +
kernel/sched/fair.c static int need_active_balance(struct lb_env *env) · line 5074
Quoted 10 lines
kernel/sched/fair.c +10
-0+if (balance_cpu ==-1 ) + balance_cpu =group_balance_cpu (sg); + +/* +* First idle cpu or the first cpu(busiest) in this sched group +* is eligible for doing load balancing at this and above domains. +*/ +return balance_cpu != env->dst_cpu; +} +
Hello, Mel.
There is one mistake from me.
The last return statement in should_we_balance() should be 'return balance_cpu == env->dst_cpu'. The fix was submitted yesterday.
You can get more information on below thread.
https://lkml.org/lkml/2013/9/10/1
I think that this fix is somewhat important to scheduler's behavior, so it may be better to update your test result with this fix. Sorry for notifying this.
Thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 01/50] sched: monolithic code dump of what is being pushed upstream
In reply to Mel GormanQuoted 26 lines · Mel Gorman
kernel/sched/fair.c +22
-0@@ -5045,15 +5038,50 @@ static int need_active_balance(struct lb_env *env)static int active_load_balance_cpu_stop (void *data); +static int should_we_balance (struct lb_env *env) +{ +struct sched_group *sg = env->sd->groups; +struct cpumask *sg_cpus, *sg_mask; +int cpu, balance_cpu =-1 ; + +/* +* In the newly idle case, we will allow all the cpu's +* to do the newly idle load balance. +*/ +if (env->idle ==CPU_NEWLY_IDLE ) +return 1 ; + + sg_cpus =sched_group_cpus (sg); + sg_mask =sched_group_mask (sg); +/* Try to find first idle cpu */ +for_each_cpu_and (cpu, sg_cpus, env->cpus) { +if (!cpumask_test_cpu (cpu, sg_mask) || !idle_cpu (cpu)) +continue ; + + balance_cpu = cpu; +break ;
kernel/sched/fair.c static int need_active_balance(struct lb_env *env) · line 5072
Quoted 10 lines
kernel/sched/fair.c +10
-0+ } + +if (balance_cpu ==-1 ) + balance_cpu =group_balance_cpu (sg); + +/* +* First idle cpu or the first cpu(busiest) in this sched group +* is eligible for doing load balancing at this and above domains. +*/ +return balance_cpu != env->dst_cpu;
FYI: Here is a bug reported by Dave Chinner.
https://lkml.org/lkml/2013/9/10/1
And lets see if any changes in your SpecJBB results without it.
Hillf
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerRe: [PATCH 01/50] sched: monolithic code dump of what is being pushed upstream
Quoted 41 lines · Hillf Danton
Quoted 26 lines · Mel Gorman
kernel/sched/fair.c +22
-0@@ -5045,15 +5038,50 @@ static int need_active_balance(struct lb_env *env)static int active_load_balance_cpu_stop (void *data); +static int should_we_balance (struct lb_env *env) +{ +struct sched_group *sg = env->sd->groups; +struct cpumask *sg_cpus, *sg_mask; +int cpu, balance_cpu =-1 ; + +/* +* In the newly idle case, we will allow all the cpu's +* to do the newly idle load balance. +*/ +if (env->idle ==CPU_NEWLY_IDLE ) +return 1 ; + + sg_cpus =sched_group_cpus (sg); + sg_mask =sched_group_mask (sg); +/* Try to find first idle cpu */ +for_each_cpu_and (cpu, sg_cpus, env->cpus) { +if (!cpumask_test_cpu (cpu, sg_mask) || !idle_cpu (cpu)) +continue ; + + balance_cpu = cpu; +break ;kernel/sched/fair.c static int need_active_balance(struct lb_env *env) · line 5072
Quoted 10 lines
kernel/sched/fair.c +10
-0+ } + +if (balance_cpu ==-1 ) + balance_cpu =group_balance_cpu (sg); + +/* +* First idle cpu or the first cpu(busiest) in this sched group +* is eligible for doing load balancing at this and above domains. +*/ +return balance_cpu != env->dst_cpu;FYI: Here is a bug reported by Dave Chinner.
https://lkml.org/lkml/2013/9/10/1
Quoted 2 lines
And lets see if any changes in your SpecJBB results without it.
Thanks for pointing that out. I've picked up the one-liner fix.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 2/50 mm: numa: Document automatic NUMA balancing sysctls
In reply to Mel Gorman--- Documentation/sysctl/kernel.txt | 66 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+)
Documentation/sysctl/kernel.txt +66 -0
@@ -354,6 +354,72 @@ utilize. ============================================================== +numa_balancing + +Enables/disables automatic page fault based NUMA memory +balancing. Memory is moved automatically to nodes +that access it often. + +Enables/disables automatic NUMA memory balancing. On NUMA machines, there +is a performance penalty if remote memory is accessed by a CPU. When this +feature is enabled the kernel samples what task thread is accessing memory +by periodically unmapping pages and later trapping a page fault. At the +time of the page fault, it is determined if the data being accessed should +be migrated to a local memory node. + +The unmapping of pages and trapping faults incur additional overhead that +ideally is offset by improved memory locality but there is no universal +guarantee. If the target workload is already bound to NUMA nodes then this +feature should be disabled. Otherwise, if the system overhead from the +feature is too high then the rate the kernel samples for NUMA hinting +faults may be controlled by the numa_balancing_scan_period_min_ms, +numa_balancing_scan_delay_ms, numa_balancing_scan_period_reset, +numa_balancing_scan_period_max_ms and numa_balancing_scan_size_mb sysctls. + +============================================================== + +numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, +numa_balancing_scan_period_max_ms, numa_balancing_scan_period_reset, +numa_balancing_scan_size_mb + +Automatic NUMA balancing scans tasks address space and unmaps pages to +detect if pages are properly placed or if the data should be migrated to a +memory node local to where the task is running. Every "scan delay" the task +scans the next "scan size" number of pages in its address space. When the +end of the address space is reached the scanner restarts from the beginning. + +In combination, the "scan delay" and "scan size" determine the scan rate. +When "scan delay" decreases, the scan rate increases. The scan delay and +hence the scan rate of every task is adaptive and depends on historical +behaviour. If pages are properly placed then the scan delay increases, +otherwise the scan delay decreases. The "scan size" is not adaptive but +the higher the "scan size", the higher the scan rate. + +Higher scan rates incur higher system overhead as page faults must be +trapped and potentially data must be migrated. However, the higher the scan +rate, the more quickly a tasks memory is migrated to a local node if the +workload pattern changes and minimises performance impact due to remote +memory accesses. These sysctls control the thresholds for scan delays and +the number of pages scanned. + +numa_balancing_scan_period_min_ms is the minimum delay in milliseconds +between scans. It effectively controls the maximum scanning rate for +each task. + +numa_balancing_scan_delay_ms is the starting "scan delay" used for a task +when it initially forks. + +numa_balancing_scan_period_max_ms is the maximum delay between scans. It +effectively controls the minimum scanning rate for each task. + +numa_balancing_scan_size_mb is how many megabytes worth of pages are +scanned for a given scan. + +numa_balancing_scan_period_reset is a blunt instrument that controls how +often a tasks scan delay is reset to detect sudden changes in task behaviour. + +============================================================== + osrelease, ostype & version: # cat osrelease
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 3/50 sched, numa: Comment fixlets
In reply to Mel GormanFrom: Peter Zijlstra <[email protected]>
Fix a 80 column violation and a PTE vs PMD reference.
--- kernel/sched/fair.c | 8 ++++---- mm/huge_memory.c | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-)
kernel/sched/fair.c +4 -4
@@ -988,10 +988,10 @@ void task_numa_work(struct callback_head *work) out:/* -* It is possible to reach the end of the VMA list but the last few VMAs are -* not guaranteed to the vma_migratable. If they are not, we would find the -* !migratable VMA on the next scan but not reset the scanner to the start -* so check it now. +* It is possible to reach the end of the VMA list but the last few +* VMAs are not guaranteed to the vma_migratable. If they are not, we +* would find the !migratable VMA on the next scan but not reset the +* scanner to the start so check it now. */ if (vma) mm->numa_scan_offset = start;
mm/huge_memory.c +1 -1
@@ -1317,7 +1317,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,spin_unlock (&mm->page_table_lock);lock_page (page); -/* Confirm the PTE did not while locked */ +/* Confirm the PMD did not change while page_table_lock was released */ spin_lock (&mm->page_table_lock);if (unlikely (!pmd_same (pmd, *pmdp))) {unlock_page (page);
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 4/50 mm: numa: Do not account for a hinting fault if we raced
In reply to Mel GormanIf another task handled a hinting fault in parallel then do not double account for it.
Not-signed-off-by: Peter Zijlstra
--- mm/huge_memory.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
mm/huge_memory.c +4 -1
@@ -1337,8 +1337,11 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, check_same:spin_lock (&mm->page_table_lock); -if (unlikely (!pmd_same (pmd, *pmdp))) +if (unlikely (!pmd_same (pmd, *pmdp))) { +/* Someone else took our fault */ + current_nid =-1 ; goto out_unlock; + } clear_pmdnuma: pmd =pmd_mknonnuma (pmd);set_pmd_at (mm, haddr, pmdp, pmd);
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 5/50 mm: Wait for THP migrations to complete during NUMA hinting faults
In reply to Mel GormanThe locking for migrating THP is unusual. While normal page migration prevents parallel accesses using a migration PTE, THP migration relies on a combination of the page_table_lock, the page lock and the existance of the NUMA hinting PTE to guarantee safety but there is a bug in the scheme.
If a THP page is currently being migrated and another thread traps a fault on the same page it checks if the page is misplaced. If it is not, then pmd_numa is cleared. The problem is that it checks if the page is misplaced without holding the page lock meaning that the racing thread can be migrating the THP when the second thread clears the NUMA bit and faults a stale page.
This patch checks if the page is potentially being migrated and stalls using the lock_page if it is potentially being migrated before checking if the page is misplaced or not.
Not-signed-off-by: Peter Zijlstra
--- mm/huge_memory.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-)
mm/huge_memory.c +16 -7
@@ -1307,13 +1307,14 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,if (current_nid ==numa_node_id ())count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL ); - target_nid =mpol_misplaced (page, vma, haddr); -if (target_nid ==-1 ) { -put_page (page); - goto clear_pmdnuma; - } +/* +* Acquire the page lock to serialise THP migrations but avoid dropping +* page_table_lock if at all possible +*/ +if (trylock_page (page)) + goto got_lock; -/* Acquire the page lock to serialise THP migrations */ +/* Serialise against migrationa and check placement check placement */ spin_unlock (&mm->page_table_lock);lock_page (page);
@@ -1324,9 +1325,17 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,put_page (page); goto out_unlock; } -spin_unlock (&mm->page_table_lock); + +got_lock: + target_nid =mpol_misplaced (page, vma, haddr); +if (target_nid ==-1 ) { +unlock_page (page); +put_page (page); + goto clear_pmdnuma; + }/* Migrate the THP to the requested node */ +spin_unlock (&mm->page_table_lock); migrated =migrate_misplaced_transhuge_page (mm, vma, pmdp, pmd, addr, page, target_nid);if (!migrated)
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 6/50 mm: Prevent parallel splits during THP migration
In reply to Mel GormanTHP migrations are serialised by the page lock but on its own that does not prevent THP splits. If the page is split during THP migration then the pmd_same checks will prevent page table corruption but the unlock page and other fix-ups potentially will cause corruption. This patch takes the anon_vma lock to prevent parallel splits during migration.
--- mm/huge_memory.c | 43 +++++++++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 14 deletions(-)
mm/huge_memory.c +29 -14
@@ -1290,18 +1290,18 @@ out:int do_huge_pmd_numa_page (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long addr,pmd_t pmd,pmd_t *pmdp) { +struct anon_vma *anon_vma =NULL ;struct page *page;unsigned long haddr = addr &HPAGE_PMD_MASK ;int target_nid;int current_nid =-1 ; -bool migrated; +bool migrated, page_locked;spin_lock (&mm->page_table_lock);if (unlikely (!pmd_same (pmd, *pmdp))) goto out_unlock; page =pmd_page (pmd); -get_page (page); current_nid =page_to_nid (page);count_vm_numa_event (NUMA_HINT_FAULTS );if (current_nid ==numa_node_id ())
@@ -1311,12 +1311,29 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,* Acquire the page lock to serialise THP migrations but avoid dropping * page_table_lock if at all possible */ -if (trylock_page (page)) - goto got_lock; + page_locked =trylock_page (page); + target_nid =mpol_misplaced (page, vma, haddr); +if (target_nid ==-1 ) { +/* If the page was locked, there are no parallel migrations */ +if (page_locked) { +unlock_page (page); + goto clear_pmdnuma; + } -/* Serialise against migrationa and check placement check placement */ +/* Otherwise wait for potential migrations to complete */ +spin_unlock (&mm->page_table_lock); +wait_on_page_locked (page); + goto check_same; + } + +/* Page is misplaced, serialise migrations and parallel THP splits */ +get_page (page);spin_unlock (&mm->page_table_lock); -lock_page (page); +if (!page_locked) { +lock_page (page); + page_locked = true; + } + anon_vma =page_lock_anon_vma_read (page);/* Confirm the PMD did not change while page_table_lock was released */ spin_lock (&mm->page_table_lock);
@@ -1326,14 +1343,6 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, goto out_unlock; } -got_lock: - target_nid =mpol_misplaced (page, vma, haddr); -if (target_nid ==-1 ) { -unlock_page (page); -put_page (page); - goto clear_pmdnuma; - } -/* Migrate the THP to the requested node */ spin_unlock (&mm->page_table_lock); migrated = migrate_misplaced_transhuge_page(mm, vma,
@@ -1342,6 +1351,8 @@ got_lock: goto check_same;task_numa_fault (target_nid,HPAGE_PMD_NR , true); +if (anon_vma) +page_unlock_anon_vma_read (anon_vma);return 0 ; check_same:
@@ -1358,6 +1369,10 @@ clear_pmdnuma:update_mmu_cache_pmd (vma, addr, pmdp); out_unlock:spin_unlock (&mm->page_table_lock); + +if (anon_vma) +page_unlock_anon_vma_read (anon_vma); +if (current_nid !=-1 )task_numa_fault (current_nid,HPAGE_PMD_NR , false);return 0 ;
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 7/50 mm: Account for a THP NUMA hinting update as one PTE update
In reply to Mel GormanA THP PMD update is accounted for as 512 pages updated in vmstat. This is large difference when estimating the cost of automatic NUMA balancing and can be misleading when comparing results that had collapsed versus split THP. This patch addresses the accounting issue.
--- mm/mprotect.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/mprotect.c +1 -1
@@ -145,7 +145,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,split_huge_page_pmd (vma, addr, pmd); else if (change_huge_pmd(vma, pmd, addr, newprot,prot_numa )) { - pages +=HPAGE_PMD_NR ; + pages++;continue ; }/* fall through */
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH 07/50] mm: Account for a THP NUMA hinting update as one PTE update
Quoted 15 lines · Mel Gorman
A THP PMD update is accounted for as 512 pages updated in vmstat. This is large difference when estimating the cost of automatic NUMA balancing and can be misleading when comparing results that had collapsed versus split THP. This patch addresses the accounting issue.
Signed-off-by Mel Gorman <[email protected]>--- mm/mprotect.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/mprotect.c +0
-0
mm/mprotect.c static inline unsigned long change_pmd_range(struct vm_area_struct *vma, · line 148
Quoted 5 lines
mm/mprotect.c +1
-1@@ -145,7 +145,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,split_huge_page_pmd (vma, addr, pmd); else if (change_huge_pmd(vma, pmd, addr, newprot,prot_numa )) { - pages +=HPAGE_PMD_NR ; + pages++;
But now you're not counting pages anymore..
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 07/50] mm: Account for a THP NUMA hinting update as one PTE update
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Mel Gorman
A THP PMD update is accounted for as 512 pages updated in vmstat. This is large difference when estimating the cost of automatic NUMA balancing and can be misleading when comparing results that had collapsed versus split THP. This patch addresses the accounting issue.
Signed-off-by Mel Gorman <[email protected]>--- mm/mprotect.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 12 lines
mm/mprotect.c static inline unsigned long change_pmd_range(struct vm_area_struct *vma, · line 148
Quoted 10 lines
mm/mprotect.c +1
-1@@ -145,7 +145,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,split_huge_page_pmd (vma, addr, pmd); else if (change_huge_pmd(vma, pmd, addr, newprot,prot_numa )) { - pages +=HPAGE_PMD_NR ; + pages++;But now you're not counting pages anymore..
The migrate statistics still count pages. That makes sense, since the amount of work scales with the amount of memory moved.
It is just the "number of faults" counters that actually count the number of faults again, instead of the number of pages represented by each fault.
IMHO this change makes sense.
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH 07/50] mm: Account for a THP NUMA hinting update as one PTE update
Quoted 25 lines · Rik van Riel
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Mel Gorman
A THP PMD update is accounted for as 512 pages updated in vmstat. This is large difference when estimating the cost of automatic NUMA balancing and can be misleading when comparing results that had collapsed versus split THP. This patch addresses the accounting issue.
Signed-off-by Mel Gorman <[email protected]>--- mm/mprotect.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 12 lines
mm/mprotect.c static inline unsigned long change_pmd_range(struct vm_area_struct *vma, · line 148
Quoted 10 lines
mm/mprotect.c +1
-1@@ -145,7 +145,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,split_huge_page_pmd (vma, addr, pmd); else if (change_huge_pmd(vma, pmd, addr, newprot,prot_numa )) { - pages +=HPAGE_PMD_NR ; + pages++;But now you're not counting pages anymore..
Quoted 2 lines
The migrate statistics still count pages. That makes sense, since the amount of work scales with the amount of memory moved.
Right.
Quoted 3 lines
It is just the "number of faults" counters that actually count the number of faults again, instead of the number of pages represented by each fault.
So you're suggesting s/pages/faults/ or somesuch?
Quoted 1 line
IMHO this change makes sense.
I never said the change didn't make sense as such. Just that we're no longer counting pages in change_*_range().
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerRe: [PATCH 07/50] mm: Account for a THP NUMA hinting update as one PTE update
Quoted 29 lines · Peter Zijlstra
Quoted 25 lines · Rik van Riel
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Mel Gorman
A THP PMD update is accounted for as 512 pages updated in vmstat. This is large difference when estimating the cost of automatic NUMA balancing and can be misleading when comparing results that had collapsed versus split THP. This patch addresses the accounting issue.
Signed-off-by Mel Gorman <[email protected]>--- mm/mprotect.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 12 lines
mm/mprotect.c static inline unsigned long change_pmd_range(struct vm_area_struct *vma, · line 148
Quoted 10 lines
mm/mprotect.c +1
-1@@ -145,7 +145,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,split_huge_page_pmd (vma, addr, pmd); else if (change_huge_pmd(vma, pmd, addr, newprot,prot_numa )) { - pages +=HPAGE_PMD_NR ; + pages++;But now you're not counting pages anymore..
Quoted 2 lines
The migrate statistics still count pages. That makes sense, since the amount of work scales with the amount of memory moved.
Quoted 8 lines
Right.
Quoted 3 lines
It is just the "number of faults" counters that actually count the number of faults again, instead of the number of pages represented by each fault.
So you're suggesting s/pages/faults/ or somesuch?
It's really the number of ptes that are updated.
Quoted 4 lines
Quoted 1 line
IMHO this change makes sense.
I never said the change didn't make sense as such. Just that we're no longer counting pages in change_*_range().
well, it's still a THP page. Is it worth renaming?
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH 07/50] mm: Account for a THP NUMA hinting update as one PTE update
Quoted 4 lines · Mel Gorman
Quoted 2 lines
I never said the change didn't make sense as such. Just that we're no longer counting pages in change_*_range().
well, it's still a THP page. Is it worth renaming?
Dunno, the pedant in me needed to raise the issue :-)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 8/50 mm: numa: Sanitize task_numa_fault() callsites
In reply to Mel GormanThere are three callers of task_numa_fault():
- do_huge_pmd_numa_page():
Accounts against the current node, not the node where the page resides, unless we migrated, in which case it accounts against the node we migrated to.
- do_numa_page():
Accounts against the current node, not the node where the page resides, unless we migrated, in which case it accounts against the node we migrated to.
- do_pmd_numa_page():
Accounts not at all when the page isn't migrated, otherwise accounts against the node we migrated towards.
This seems wrong to me; all three sites should have the same sementaics, furthermore we should accounts against where the page really is, we already know where the task is.
So modify all three sites to always account; we did after all receive the fault; and always account to where the page is after migration, regardless of success.
They all still differ on when they clear the PTE/PMD; ideally that would get sorted too.
--- mm/huge_memory.c | 26 ++++++++++++++------------ mm/memory.c | 53 +++++++++++++++++++++-------------------------------- 2 files changed, 35 insertions(+), 44 deletions(-)
mm/huge_memory.c +14 -12
@@ -1293,18 +1293,19 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct anon_vma *anon_vma =NULL ;struct page *page;unsigned long haddr = addr &HPAGE_PMD_MASK ; +int page_nid =-1 , this_nid =numa_node_id ();int target_nid; -int current_nid =-1 ; -bool migrated, page_locked; +bool page_locked; +bool migrated = false;spin_lock (&mm->page_table_lock);if (unlikely (!pmd_same (pmd, *pmdp))) goto out_unlock; page =pmd_page (pmd); - current_nid =page_to_nid (page); + page_nid =page_to_nid (page);count_vm_numa_event (NUMA_HINT_FAULTS ); -if (current_nid ==numa_node_id ()) +if (page_nid == this_nid)count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL );/*
@@ -1347,19 +1348,18 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,spin_unlock (&mm->page_table_lock); migrated =migrate_misplaced_transhuge_page (mm, vma, pmdp, pmd, addr, page, target_nid); -if (!migrated) +if (migrated) + page_nid = target_nid; +else goto check_same; -task_numa_fault (target_nid,HPAGE_PMD_NR , true); -if (anon_vma) -page_unlock_anon_vma_read (anon_vma); -return 0 ; + goto out; check_same:spin_lock (&mm->page_table_lock);if (unlikely (!pmd_same (pmd, *pmdp))) {/* Someone else took our fault */ - current_nid =-1 ; + page_nid =-1 ; goto out_unlock; } clear_pmdnuma:
@@ -1370,11 +1370,13 @@ clear_pmdnuma: out_unlock:spin_unlock (&mm->page_table_lock); +out:if (anon_vma)page_unlock_anon_vma_read (anon_vma); -if (current_nid !=-1 ) -task_numa_fault (current_nid,HPAGE_PMD_NR , false); +if (page_nid !=-1 ) +task_numa_fault (page_nid,HPAGE_PMD_NR , migrated); +return 0 ; }
mm/memory.c +21 -32
107 lines not on this page · raw message
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 9/50 mm: numa: Do not migrate or account for hinting faults on the zero page
In reply to Mel GormanThe zero page is not replicated between nodes and is often shared between processes. The data is read-only and likely to be cached in local CPUs if heavily accessed meaning that the remote memory access cost is less of a concern. This patch prevents trapping faults on the zero pages. For tasks using the zero page this will reduce the number of PTE updates, TLB flushes and hinting faults.
--- mm/huge_memory.c | 7 ++++++- mm/memory.c | 1 + mm/mprotect.c | 10 +++++++++- 3 files changed, 16 insertions(+), 2 deletions(-)
mm/huge_memory.c +6 -1
@@ -1303,6 +1303,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, goto out_unlock; page =pmd_page (pmd); +BUG_ON (is_huge_zero_page (page)); page_nid =page_to_nid (page);count_vm_numa_event (NUMA_HINT_FAULTS );if (page_nid == this_nid)
@@ -1488,8 +1489,12 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, }else {struct page *page =pmd_page (*pmd); -/* only check non-shared pages */ +/* +* Only check non-shared pages. See change_pte_range +* for comment on why the zero page is not modified +*/ if (page_mapcount (page) ==1 && + !is_huge_zero_page (page) && !pmd_numa (*pmd)) { entry =pmd_mknuma (entry); }
mm/memory.c +1 -0
@@ -3575,6 +3575,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,pte_unmap_unlock (ptep, ptl);return 0 ; } +BUG_ON (is_zero_pfn (page_to_pfn (page))); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);
mm/mprotect.c +9 -1
@@ -62,7 +62,15 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,struct page *page; page =vm_normal_page (vma, addr, oldpte); -if (page) { + +/* +* Do not trap faults against the zero page. +* The read-only data is likely to be +* read-cached on the local CPU cache and it +* is less useful to know about local vs remote +* hits on the zero page +*/ +if (page && !is_zero_pfn (page_to_pfn (page))) {int this_nid =page_to_nid (page);if (last_nid ==-1 ) last_nid = this_nid;
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 10/50 sched: numa: Mitigate chance that same task always updates PTEs
In reply to Mel GormanFrom: Peter Zijlstra <[email protected]>
With a trace_printk("working\n"); right after the cmpxchg in task_numa_work() we can see that of a 4 thread process, its always the same task winning the race and doing the protection change.
This is a problem since the task doing the protection change has a penalty for taking faults -- it is busy when marking the PTEs. If its always the same task the ->numa_faults[] get severely skewed.
Avoid this by delaying the task doing the protection change such that it is unlikely to win the privilege again.
Before:
root@interlagos:~# grep "thread 0/.*working" /debug/tracing/trace | tail -15
thread 0/0-3232 [022] .... 212.787402: task_numa_work: working
thread 0/0-3232 [022] .... 212.888473: task_numa_work: working
thread 0/0-3232 [022] .... 212.989538: task_numa_work: working
thread 0/0-3232 [022] .... 213.090602: task_numa_work: working
thread 0/0-3232 [022] .... 213.191667: task_numa_work: working
thread 0/0-3232 [022] .... 213.292734: task_numa_work: working
thread 0/0-3232 [022] .... 213.393804: task_numa_work: working
thread 0/0-3232 [022] .... 213.494869: task_numa_work: working
thread 0/0-3232 [022] .... 213.596937: task_numa_work: working
thread 0/0-3232 [022] .... 213.699000: task_numa_work: working
thread 0/0-3232 [022] .... 213.801067: task_numa_work: working
thread 0/0-3232 [022] .... 213.903155: task_numa_work: working
thread 0/0-3232 [022] .... 214.005201: task_numa_work: working
thread 0/0-3232 [022] .... 214.107266: task_numa_work: working
thread 0/0-3232 [022] .... 214.209342: task_numa_work: workingAfter:
root@interlagos:~# grep "thread 0/.*working" /debug/tracing/trace | tail -15
thread 0/0-3253 [005] .... 136.865051: task_numa_work: working
thread 0/2-3255 [026] .... 136.965134: task_numa_work: working
thread 0/3-3256 [024] .... 137.065217: task_numa_work: working
thread 0/3-3256 [024] .... 137.165302: task_numa_work: working
thread 0/3-3256 [024] .... 137.265382: task_numa_work: working
thread 0/0-3253 [004] .... 137.366465: task_numa_work: working
thread 0/2-3255 [026] .... 137.466549: task_numa_work: working
thread 0/0-3253 [004] .... 137.566629: task_numa_work: working
thread 0/0-3253 [004] .... 137.666711: task_numa_work: working
thread 0/1-3254 [028] .... 137.766799: task_numa_work: working
thread 0/0-3253 [004] .... 137.866876: task_numa_work: working
thread 0/2-3255 [026] .... 137.966960: task_numa_work: working
thread 0/1-3254 [028] .... 138.067041: task_numa_work: working
thread 0/2-3255 [026] .... 138.167123: task_numa_work: working
thread 0/3-3256 [024] .... 138.267207: task_numa_work: working--- kernel/sched/fair.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/sched/fair.c +7 -1
@@ -946,6 +946,12 @@ void task_numa_work(struct callback_head *work)return ;/* +* Delay this task enough that another task of this mm will likely win +* the next time around. +*/ + p->node_stamp +=2 *TICK_NSEC ; + +/* * Do not set pte_numa if the current running node is rate-limited. * This loses statistics on the fault but if we are unwilling to * migrate to this node, it is less likely we can do useful work
@@ -1026,7 +1032,7 @@ void task_tick_numa(struct rq *rq, struct task_struct *curr)if (now - curr->node_stamp > period) {if (!curr->node_stamp) curr->numa_scan_period = sysctl_numa_balancing_scan_period_min; - curr->node_stamp = now; + curr->node_stamp += period;if (!time_before (jiffies, curr->mm->numa_next_scan)) {init_task_work (work, task_numa_work);/* TODO: move this into sched_fork() */
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 11/50 sched: numa: Continue PTE scanning even if migrate rate limited
In reply to Mel GormanFrom: Peter Zijlstra <[email protected]>
Avoiding marking PTEs pte_numa because a particular NUMA node is migrate rate limited sees like a bad idea. Even if this node can't migrate anymore other nodes might and we want up-to-date information to do balance decisions. We already rate limit the actual migrations, this should leave enough bandwidth to allow the non-migrating scanning. I think its important we keep up-to-date information if we're going to do placement based on it.
--- kernel/sched/fair.c | 8 -------- 1 file changed, 8 deletions(-)
kernel/sched/fair.c +0 -8
@@ -951,14 +951,6 @@ void task_numa_work(struct callback_head *work)*/ p->node_stamp +=2 *TICK_NSEC ; -/* -* Do not set pte_numa if the current running node is rate-limited. -* This loses statistics on the fault but if we are unwilling to -* migrate to this node, it is less likely we can do useful work -*/ -if (migrate_ratelimited (numa_node_id ())) -return ; - start = mm->numa_scan_offset; pages = sysctl_numa_balancing_scan_size; pages <<=20 -PAGE_SHIFT ;/* MB in pages */
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 12/50 Revert "mm: sched: numa: Delay PTE scanning until a task is scheduled on a new node"
In reply to Mel GormanPTE scanning and NUMA hinting fault handling is expensive so commit 5bca2303 ("mm: sched: numa: Delay PTE scanning until a task is scheduled on a new node") deferred the PTE scan until a task had been scheduled on another node. The problem is that in the purely shared memory case that this may never happen and no NUMA hinting fault information will be captured. We are not ruling out the possibility that something better can be done here but for now, this patch needs to be reverted and depend entirely on the scan_delay to avoid punishing short-lived processes.
--- include/linux/mm_types.h | 10 ---------- kernel/fork.c | 3 --- kernel/sched/fair.c | 18 ------------------ kernel/sched/features.h | 4 +--- 4 files changed, 1 insertion(+), 34 deletions(-)
include/linux/mm_types.h +0 -10
@@ -427,20 +427,10 @@ struct mm_struct {/* numa_scan_seq prevents two threads setting pte_numa */ int numa_scan_seq; - -/* -* The first node a task was scheduled on. If a task runs on -* a different node than Make PTE Scan Go Now. -*/ -int first_nid;#endif struct uprobes_state uprobes_state; }; -/* first nid will either be a valid NID or one of these values */ -#define NUMA_PTE_SCAN_INIT -1 -#define NUMA_PTE_SCAN_ACTIVE -2 -static inline void mm_init_cpumask (struct mm_struct *mm) {#ifdef CONFIG_CPUMASK_OFFSTACK
kernel/fork.c +0 -3
@@ -820,9 +820,6 @@ struct mm_struct *dup_mm(struct task_struct *tsk)#ifdef CONFIG_TRANSPARENT_HUGEPAGE mm->pmd_huge_pte =NULL ;#endif -#ifdef CONFIG_NUMA_BALANCING - mm->first_nid =NUMA_PTE_SCAN_INIT ; -#endif if (!mm_init (mm, tsk)) goto fail_nomem;
kernel/sched/fair.c +0 -18
@@ -901,24 +901,6 @@ void task_numa_work(struct callback_head *work)return ;/* -* We do not care about task placement until a task runs on a node -* other than the first one used by the address space. This is -* largely because migrations are driven by what CPU the task -* is running on. If it's never scheduled on another node, it'll -* not migrate so why bother trapping the fault. -*/ -if (mm->first_nid ==NUMA_PTE_SCAN_INIT ) - mm->first_nid =numa_node_id (); -if (mm->first_nid !=NUMA_PTE_SCAN_ACTIVE ) { -/* Are we running on a new node yet? */ -if (numa_node_id () == mm->first_nid && - !sched_feat_numa (NUMA_FORCE )) -return ; - - mm->first_nid =NUMA_PTE_SCAN_ACTIVE ; - } - -/* * Reset the scan period if enough time has gone by. Objective is that * scanning will be reduced if pages are properly placed. As tasks * can enter different phases this needs to be re-examined. Lacking
kernel/sched/features.h +1 -3
@@ -63,10 +63,8 @@ SCHED_FEAT(LB_MIN, false)/* * Apply the automatic NUMA scheduling policy. Enabled automatically * at runtime if running on a NUMA machine. Can be controlled via -* numa_balancing=. Allow PTE scanning to be forced on UMA machines -* for debugging the core machinery. +* numa_balancing= */ #ifdef CONFIG_NUMA_BALANCING SCHED_FEAT(NUMA, false) -SCHED_FEAT(NUMA_FORCE, false)#endif
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 13/50 sched: numa: Initialise numa_next_scan properly
In reply to Mel GormanScan delay logic and resets are currently initialised to start scanning immediately instead of delaying properly. Initialise them properly at fork time and catch when a new mm has been allocated.
--- kernel/sched/core.c | 4 ++-- kernel/sched/fair.c | 7 +++++++ 2 files changed, 9 insertions(+), 2 deletions(-)
kernel/sched/core.c +2 -2
@@ -1634,8 +1634,8 @@ static void __sched_fork(struct task_struct *p)#ifdef CONFIG_NUMA_BALANCING if (p->mm &&atomic_read (&p->mm->mm_users) ==1 ) { - p->mm->numa_next_scan = jiffies; - p->mm->numa_next_reset = jiffies; + p->mm->numa_next_scan = jiffies +msecs_to_jiffies (sysctl_numa_balancing_scan_delay); + p->mm->numa_next_reset = jiffies +msecs_to_jiffies (sysctl_numa_balancing_scan_period_reset); p->mm->numa_scan_seq =0 ; }
kernel/sched/fair.c +7 -0
@@ -900,6 +900,13 @@ void task_numa_work(struct callback_head *work)if (p->flags &PF_EXITING )return ; +if (!mm->numa_next_reset || !mm->numa_next_scan) { + mm->numa_next_scan = now + +msecs_to_jiffies (sysctl_numa_balancing_scan_delay); + mm->numa_next_reset = now + +msecs_to_jiffies (sysctl_numa_balancing_scan_period_reset); + } +/* * Reset the scan period if enough time has gone by. Objective is that * scanning will be reduced if pages are properly placed. As tasks
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 14/50 sched: Set the scan rate proportional to the memory usage of the task being scanned
In reply to Mel GormanThe NUMA PTE scan rate is controlled with a combination of the numa_balancing_scan_period_min, numa_balancing_scan_period_max and numa_balancing_scan_size. This scan rate is independent of the size of the task and as an aside it is further complicated by the fact that numa_balancing_scan_size controls how many pages are marked pte_numa and not how much virtual memory is scanned.
In combination, it is almost impossible to meaningfully tune the min and max scan periods and reasoning about performance is complex when the time to complete a full scan is is partially a function of the tasks memory size. This patch alters the semantic of the min and max tunables to be about tuning the length time it takes to complete a scan of a tasks occupied virtual address space. Conceptually this is a lot easier to understand. There is a "sanity" check to ensure the scan rate is never extremely fast based on the amount of virtual memory that should be scanned in a second. The default of 2.5G seems arbitrary but it is to have the maximum scan rate after the patch roughly match the maximum scan rate before the patch was applied.
--- Documentation/sysctl/kernel.txt | 11 +++--- include/linux/sched.h | 1 + kernel/sched/fair.c | 84 ++++++++++++++++++++++++++++++++++++----- 3 files changed, 81 insertions(+), 15 deletions(-)
Documentation/sysctl/kernel.txt +6 -5
@@ -402,15 +402,16 @@ workload pattern changes and minimises performance impact due to remote memory accesses. These sysctls control the thresholds for scan delays and the number of pages scanned. -numa_balancing_scan_period_min_ms is the minimum delay in milliseconds -between scans. It effectively controls the maximum scanning rate for -each task. +numa_balancing_scan_period_min_ms is the minimum time in milliseconds to +scan a tasks virtual memory. It effectively controls the maximum scanning +rate for each task. numa_balancing_scan_delay_ms is the starting "scan delay" used for a task when it initially forks. -numa_balancing_scan_period_max_ms is the maximum delay between scans. It -effectively controls the minimum scanning rate for each task. +numa_balancing_scan_period_max_ms is the maximum time in milliseconds to +scan a tasks virtual memory. It effectively controls the minimum scanning +rate for each task. numa_balancing_scan_size_mb is how many megabytes worth of pages are scanned for a given scan.
include/linux/sched.h +1 -0
@@ -1331,6 +1331,7 @@ struct task_struct {int numa_scan_seq;int numa_migrate_seq;unsigned int numa_scan_period; +unsigned int numa_scan_period_max;u64 node_stamp;/* migration stamp */ struct callback_head numa_work;#endif /* CONFIG_NUMA_BALANCING */
kernel/sched/fair.c +74 -10
140 lines not on this page · raw message
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH 14/50] sched: Set the scan rate proportional to the memory usage of the task being scanned
kernel/sched/fair.c void task_numa_fault(int node, int pages, bool migrated) · line 917
Quoted 13 lines · Mel Gorman
kernel/sched/fair.c +6
-2@@ -860,9 +908,14 @@ void task_numa_fault(int node, int pages, bool migrated)* If pages are properly placed (did not migrate) then scan slower. * This is reset periodically in case of phase changes */ -if (!migrated) - p->numa_scan_period =min (sysctl_numa_balancing_scan_period_max, +if (!migrated) { +/* Initialise if necessary */ +if (!p->numa_scan_period_max) + p->numa_scan_period_max =task_scan_max (p); + + p->numa_scan_period =min (p->numa_scan_period_max, p->numa_scan_period +jiffies_to_msecs (10 ));
So the next patch changes the jiffies_to_msec() thing.. is that really worth a whole separate patch?
Also, I really don't believe any of that is 'right', increasing the scan period by a fixed amount for every !migrated page is just wrong.
Firstly; there's the migration throttle which basically guarantees that most pages aren't migrated -- even when they ought to be, thus inflating the period.
Secondly; assume a _huge_ process, so large that even a small fraction of non-migrated pages will completely clip the scan period.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerRe: [PATCH 14/50] sched: Set the scan rate proportional to the memory usage of the task being scanned
Quoted 18 lines · Peter Zijlstra
kernel/sched/fair.c void task_numa_fault(int node, int pages, bool migrated) · line 917
Quoted 13 lines · Mel Gorman
kernel/sched/fair.c +6
-2@@ -860,9 +908,14 @@ void task_numa_fault(int node, int pages, bool migrated)* If pages are properly placed (did not migrate) then scan slower. * This is reset periodically in case of phase changes */ -if (!migrated) - p->numa_scan_period =min (sysctl_numa_balancing_scan_period_max, +if (!migrated) { +/* Initialise if necessary */ +if (!p->numa_scan_period_max) + p->numa_scan_period_max =task_scan_max (p); + + p->numa_scan_period =min (p->numa_scan_period_max, p->numa_scan_period +jiffies_to_msecs (10 ));So the next patch changes the
jiffies_to_msec()thing.. is that really worth a whole separate patch?
No, I can collapse them.
Quoted 3 lines
Also, I really don't believe any of that is 'right', increasing the scan period by a fixed amount for every !migrated page is just wrong.
At the moment Rik and I are both looking at adapting the scan rate based on whether the faults trapped since the last scan window were local or remote faults. It should be able to sensibly adapt the scan rate independently of the RSS of the process.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 15/50 sched: numa: Correct adjustment of numa_scan_period
In reply to Mel Gormannuma_scan_period is in milliseconds, not jiffies. Properly placed pages slow the scanning rate but adding 10 jiffies to numa_scan_period means that the rate scanning slows depends on HZ which is confusing. Get rid of the jiffies_to_msec conversion and treat it as ms.
--- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +1 -1
@@ -914,7 +914,7 @@ void task_numa_fault(int node, int pages, bool migrated) p->numa_scan_period_max =task_scan_max (p); p->numa_scan_period =min (p->numa_scan_period_max, - p->numa_scan_period +jiffies_to_msecs (10 )); + p->numa_scan_period +10 ); }task_numa_placement (p);
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 16/50 mm: Only flush TLBs if a transhuge PMD is modified for NUMA pte scanning
In reply to Mel GormanNUMA PTE scanning is expensive both in terms of the scanning itself and the TLB flush if there are any updates. The TLB flush is avoided if no PTEs are updated but there is a bug where transhuge PMDs are considered to be updated even if they were already pmd_numa. This patch addresses the problem and TLB flushes should be reduced.
--- mm/huge_memory.c | 19 ++++++++++++++++--- mm/mprotect.c | 14 ++++++++++---- 2 files changed, 26 insertions(+), 7 deletions(-)
mm/huge_memory.c +16 -3
@@ -1474,6 +1474,12 @@ out:return ret; } +/* +* Returns +* - 0 if PMD could not be locked +* - 1 if PMD was locked but protections unchange and TLB flush unnecessary +* - HPAGE_PMD_NR is protections changed and TLB flush necessary +*/ int change_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,pgprot_t newprot,int prot_numa) {
@@ -1482,9 +1488,11 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,if (__pmd_trans_huge_lock (pmd, vma) ==1 ) {pmd_t entry; - entry =pmdp_get_and_clear (mm, addr, pmd); + ret =1 ;if (!prot_numa) { + entry =pmdp_get_and_clear (mm, addr, pmd); entry =pmd_modify (entry, newprot); + ret =HPAGE_PMD_NR ;BUG_ON (pmd_write (entry)); }else {struct page *page =pmd_page (*pmd);
@@ -1496,12 +1504,17 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,if (page_mapcount (page) ==1 && !is_huge_zero_page (page) && !pmd_numa (*pmd)) { + entry =pmdp_get_and_clear (mm, addr, pmd); entry =pmd_mknuma (entry); + ret =HPAGE_PMD_NR ; } } -set_pmd_at (mm, addr, pmd, entry); + +/* Set PMD if cleared earlier */ +if (ret ==HPAGE_PMD_NR ) +set_pmd_at (mm, addr, pmd, entry); +spin_unlock (&vma->vm_mm->page_table_lock); - ret =1 ; }return ret;
mm/mprotect.c +10 -4
@@ -151,10 +151,16 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,if (pmd_trans_huge (*pmd)) {if (next - addr !=HPAGE_PMD_SIZE )split_huge_page_pmd (vma, addr, pmd); -else if (change_huge_pmd (vma, pmd, addr, newprot, - prot_numa)) { - pages++; -continue ; +else { +int nr_ptes =change_huge_pmd (vma, pmd, addr, + newprot, prot_numa); + +if (nr_ptes) { +if (nr_ptes ==HPAGE_PMD_NR ) + pages++; + +continue ; + } }/* fall through */ }
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 17/50 mm: Do not flush TLB during protection change if !pte_present && !migration_entry
In reply to Mel GormanNUMA PTE scanning is expensive both in terms of the scanning itself and the TLB flush if there are any updates. Currently non-present PTEs are accounted for as an update and incurring a TLB flush where it is only necessary for anonymous migration entries. This patch addresses the problem and should reduce TLB flushes.
--- mm/mprotect.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/mprotect.c +2 -1
@@ -109,8 +109,9 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,make_migration_entry_read (&entry);set_pte_at (mm, addr, pte,swp_entry_to_pte (entry)); + + pages++; } - pages++; } }while (pte++, addr +=PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode ();
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH 17/50] mm: Do not flush TLB during protection change if !pte_present && !migration_entry
Quoted 16 lines · Mel Gorman
NUMA PTE scanning is expensive both in terms of the scanning itself and the TLB flush if there are any updates. Currently non-present PTEs are accounted for as an update and incurring a TLB flush where it is only necessary for anonymous migration entries. This patch addresses the problem and should reduce TLB flushes.
Signed-off-by Mel Gorman <[email protected]>--- mm/mprotect.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/mprotect.c +0
-0
mm/mprotect.c static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, · line 117
Quoted 10 lines
mm/mprotect.c +2
-1@@ -109,8 +109,9 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,make_migration_entry_read (&entry);set_pte_at (mm, addr, pte,swp_entry_to_pte (entry)); + + pages++; } - pages++; } }while (pte++, addr +=PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode ();
Should we fold this into patch 7 ?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerRe: [PATCH 17/50] mm: Do not flush TLB during protection change if !pte_present && !migration_entry
Quoted 21 lines · Peter Zijlstra
Quoted 20 lines · Mel Gorman
NUMA PTE scanning is expensive both in terms of the scanning itself and the TLB flush if there are any updates. Currently non-present PTEs are accounted for as an update and incurring a TLB flush where it is only necessary for anonymous migration entries. This patch addresses the problem and should reduce TLB flushes.
Signed-off-by Mel Gorman <[email protected]>--- mm/mprotect.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/mprotect.c +1
-0@@ -109,8 +109,9 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,make_migration_entry_read (&entry);set_pte_at (mm, addr, pte,swp_entry_to_pte (entry)); +
Quoted 8 lines
mm/mprotect.c static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, · line 117
Quoted 6 lines
mm/mprotect.c +1
-1+ pages++; } - pages++; } }while (pte++, addr +=PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode ();Should we fold this into patch 7 ?
Looking closer at it, I think folding it into the patch would overload the purpose of patch 7 a little too much but I shuffled the series to keep the patches together.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 18/50 sched: numa: Slow scan rate if no NUMA hinting faults are being recorded
In reply to Mel GormanNUMA PTE scanning slows if a NUMA hinting fault was trapped and no page was migrated. For long-lived but idle processes there may be no faults but the scan rate will be high and just waste CPU. This patch will slow the scan rate for processes that are not trapping faults.
--- kernel/sched/fair.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
kernel/sched/fair.c +12 -0
@@ -1039,6 +1039,18 @@ void task_numa_work(struct callback_head *work) out:/* +* If the whole process was scanned without updates then no NUMA +* hinting faults are being recorded and scan rate should be lower. +*/ +if (mm->numa_scan_offset ==0 && !nr_pte_updates) { + p->numa_scan_period =min (p->numa_scan_period_max, + p->numa_scan_period <<1 ); + + next_scan = now +msecs_to_jiffies (p->numa_scan_period); + mm->numa_next_scan = next_scan; + } + +/* * It is possible to reach the end of the VMA list but the last few * VMAs are not guaranteed to the vma_migratable. If they are not, we * would find the !migratable VMA on the next scan but not reset the
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 19/50 sched: Track NUMA hinting faults on per-node basis
In reply to Mel GormanThis patch tracks what nodes numa hinting faults were incurred on. This information is later used to schedule a task on the node storing the pages most frequently faulted by the task.
--- include/linux/sched.h | 2 ++ kernel/sched/core.c | 3 +++ kernel/sched/fair.c | 11 ++++++++++- kernel/sched/sched.h | 12 ++++++++++++ 4 files changed, 27 insertions(+), 1 deletion(-)
include/linux/sched.h +2 -0
@@ -1334,6 +1334,8 @@ struct task_struct {unsigned int numa_scan_period_max;u64 node_stamp;/* migration stamp */ struct callback_head numa_work; + +unsigned long *numa_faults;#endif /* CONFIG_NUMA_BALANCING */ struct rcu_head rcu;
kernel/sched/core.c +3 -0
@@ -1644,6 +1644,7 @@ static void __sched_fork(struct task_struct *p) p->numa_migrate_seq = p->mm ? p->mm->numa_scan_seq -1 :0 ; p->numa_scan_period = sysctl_numa_balancing_scan_delay; p->numa_work.next = &p->numa_work; + p->numa_faults =NULL ;#endif /* CONFIG_NUMA_BALANCING */ }
@@ -1905,6 +1906,8 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)if (mm)mmdrop (mm);if (unlikely (prev_state ==TASK_DEAD )) { +task_numa_free (prev); +/* * Remove function-return probe instances associated with this * task and put them back on the free list.
kernel/sched/fair.c +10 -1
@@ -902,7 +902,14 @@ void task_numa_fault(int node, int pages, bool migrated)if (!numabalancing_enabled)return ; -/* FIXME: Allocate task-specific structure for placement policy here */ +/* Allocate buffer to track faults on a per-node basis */ +if (unlikely (!p->numa_faults)) { +int size =sizeof (*p->numa_faults) * nr_node_ids; + + p->numa_faults =kzalloc (size,GFP_KERNEL |__GFP_NOWARN); +if (!p->numa_faults) +return ; + }/* * If pages are properly placed (did not migrate) then scan slower.
@@ -918,6 +925,8 @@ void task_numa_fault(int node, int pages, bool migrated) }task_numa_placement (p); + + p->numa_faults[node] += pages; }static void reset_ptenuma_scan (struct task_struct *p)
kernel/sched/sched.h +12 -0
@@ -6,6 +6,7 @@#include <linux/spinlock.h> #include <linux/stop_machine.h> #include <linux/tick.h> +#include <linux/slab.h> #include "cpupri.h" #include "cpuacct.h"
@@ -553,6 +554,17 @@ static inline u64 rq_clock_task(struct rq *rq)return rq->clock_task; } +#ifdef CONFIG_NUMA_BALANCING +static inline void task_numa_free (struct task_struct *p) +{ +kfree (p->numa_faults); +} +#else /* CONFIG_NUMA_BALANCING */ +static inline void task_numa_free (struct task_struct *p) +{ +} +#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SMP #define rcu_dereference_check_sched_domain (p) \
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 20/50 sched: Select a preferred node with the most numa hinting faults
In reply to Mel GormanThis patch selects a preferred node for a task to run on based on the NUMA hinting faults. This information is later used to migrate tasks towards the node during balancing.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 1 + kernel/sched/fair.c | 17 +++++++++++++++-- 3 files changed, 17 insertions(+), 2 deletions(-)
include/linux/sched.h +1 -0
@@ -1336,6 +1336,7 @@ struct task_struct {struct callback_head numa_work;unsigned long *numa_faults; +int numa_preferred_nid;#endif /* CONFIG_NUMA_BALANCING */ struct rcu_head rcu;
kernel/sched/core.c +1 -0
@@ -1643,6 +1643,7 @@ static void __sched_fork(struct task_struct *p) p->numa_scan_seq = p->mm ? p->mm->numa_scan_seq :0 ; p->numa_migrate_seq = p->mm ? p->mm->numa_scan_seq -1 :0 ; p->numa_scan_period = sysctl_numa_balancing_scan_delay; + p->numa_preferred_nid =-1 ; p->numa_work.next = &p->numa_work; p->numa_faults =NULL ;#endif /* CONFIG_NUMA_BALANCING */
kernel/sched/fair.c +15 -2
@@ -879,7 +879,8 @@ static unsigned int task_scan_max(struct task_struct *p)static void task_numa_placement (struct task_struct *p) { -int seq; +int seq, nid, max_nid =-1 ; +unsigned long max_faults =0 ;if (!p->mm)/* for example, ksmd faulting in a user's mm */ return ;
@@ -889,7 +890,19 @@ static void task_numa_placement(struct task_struct *p) p->numa_scan_seq = seq; p->numa_scan_period_max =task_scan_max (p); -/* FIXME: Scheduling placement policy hints go here */ +/* Find the node with the highest number of faults */ +for_each_online_node (nid) { +unsigned long faults = p->numa_faults[nid]; + p->numa_faults[nid] >>=1 ; +if (faults > max_faults) { + max_faults = faults; + max_nid = nid; + } + } + +/* Update the tasks preferred node if necessary */ +if (max_faults && max_nid != p->numa_preferred_nid) + p->numa_preferred_nid = max_nid; }/*
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 21/50 sched: Update NUMA hinting faults once per scan
In reply to Mel GormanNUMA hinting fault counts and placement decisions are both recorded in the same array which distorts the samples in an unpredictable fashion. The values linearly accumulate during the scan and then decay creating a sawtooth-like pattern in the per-node counts. It also means that placement decisions are time sensitive. At best it means that it is very difficult to state that the buffer holds a decaying average of past faulting behaviour. At worst, it can confuse the load balancer if it sees one node with an artifically high count due to very recent faulting activity and may create a bouncing effect.
This patch adds a second array. numa_faults stores the historical data which is used for placement decisions. numa_faults_buffer holds the fault activity during the current scan window. When the scan completes, numa_faults decays and the values from numa_faults_buffer are copied across.
--- include/linux/sched.h | 13 +++++++++++++ kernel/sched/core.c | 1 + kernel/sched/fair.c | 16 +++++++++++++--- 3 files changed, 27 insertions(+), 3 deletions(-)
include/linux/sched.h +13 -0
@@ -1335,7 +1335,20 @@ struct task_struct {u64 node_stamp;/* migration stamp */ struct callback_head numa_work; +/* +* Exponential decaying average of faults on a per-node basis. +* Scheduling placement decisions are made based on the these counts. +* The values remain static for the duration of a PTE scan +*/ unsigned long *numa_faults; + +/* +* numa_faults_buffer records faults per node during the current +* scan window. When the scan completes, the counts in numa_faults +* decay and these values are copied. +*/ +unsigned long *numa_faults_buffer; +int numa_preferred_nid;#endif /* CONFIG_NUMA_BALANCING */
kernel/sched/core.c +1 -0
@@ -1646,6 +1646,7 @@ static void __sched_fork(struct task_struct *p) p->numa_preferred_nid =-1 ; p->numa_work.next = &p->numa_work; p->numa_faults =NULL ; + p->numa_faults_buffer =NULL ;#endif /* CONFIG_NUMA_BALANCING */ }
kernel/sched/fair.c +13 -3
@@ -892,8 +892,14 @@ static void task_numa_placement(struct task_struct *p)/* Find the node with the highest number of faults */ for_each_online_node (nid) { -unsigned long faults = p->numa_faults[nid]; +unsigned long faults; + +/* Decay existing window and copy faults since last scan */ p->numa_faults[nid] >>=1 ; + p->numa_faults[nid] += p->numa_faults_buffer[nid]; + p->numa_faults_buffer[nid] =0 ; + + faults = p->numa_faults[nid];if (faults > max_faults) { max_faults = faults; max_nid = nid;
@@ -919,9 +925,13 @@ void task_numa_fault(int node, int pages, bool migrated)if (unlikely (!p->numa_faults)) {int size =sizeof (*p->numa_faults) * nr_node_ids; - p->numa_faults =kzalloc (size,GFP_KERNEL |__GFP_NOWARN); +/* numa_faults and numa_faults_buffer share the allocation */ + p->numa_faults =kzalloc (size *2 ,GFP_KERNEL |__GFP_NOWARN);if (!p->numa_faults)return ; + +BUG_ON (p->numa_faults_buffer); + p->numa_faults_buffer = p->numa_faults + nr_node_ids; }/*
@@ -939,7 +949,7 @@ void task_numa_fault(int node, int pages, bool migrated)task_numa_placement (p); - p->numa_faults[node] += pages; + p->numa_faults_buffer[node] += pages; }static void reset_ptenuma_scan (struct task_struct *p)
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 22/50 sched: Favour moving tasks towards the preferred node
In reply to Mel GormanThis patch favours moving tasks towards NUMA node that recorded a higher number of NUMA faults during active load balancing. Ideally this is self-reinforcing as the longer the task runs on that node, the more faults it should incur causing task_numa_placement to keep the task running on that node. In reality a big weakness is that the nodes CPUs can be overloaded and it would be more efficient to queue tasks on an idle node and migrate to the new node. This would require additional smarts in the balancer so for now the balancer will simply prefer to place the task on the preferred node for a PTE scans which is controlled by the numa_balancing_settle_count sysctl. Once the settle_count number of scans has complete the schedule is free to place the task on an alternative node if the load is imbalanced.
--- Documentation/sysctl/kernel.txt | 8 +++++- include/linux/sched.h | 1 + kernel/sched/core.c | 3 +- kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++++--- kernel/sched/features.h | 7 +++++ kernel/sysctl.c | 7 +++++ 6 files changed, 83 insertions(+), 6 deletions(-)
Documentation/sysctl/kernel.txt +7 -1
@@ -374,7 +374,8 @@ feature should be disabled. Otherwise, if the system overhead from the feature is too high then the rate the kernel samples for NUMA hinting faults may be controlled by the numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, numa_balancing_scan_period_reset, -numa_balancing_scan_period_max_ms and numa_balancing_scan_size_mb sysctls. +numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb and +numa_balancing_settle_count sysctls. ==============================================================
@@ -419,6 +420,11 @@ scanned for a given scan. numa_balancing_scan_period_reset is a blunt instrument that controls how often a tasks scan delay is reset to detect sudden changes in task behaviour. +numa_balancing_settle_count is how many scan periods must complete before +the schedule balancer stops pushing the task towards a preferred node. This +gives the scheduler a chance to place the task on an alternative node if the +preferred node is overloaded. + ============================================================== osrelease, ostype & version:
include/linux/sched.h +1 -0
@@ -776,6 +776,7 @@ enum cpu_idle_type {#define SD_ASYM_PACKING 0x0800/* Place busy groups earlier in the domain */ #define SD_PREFER_SIBLING 0x1000/* Prefer to place tasks in a sibling domain */ #define SD_OVERLAP 0x2000/* sched_domains of this level overlap */ +#define SD_NUMA 0x4000/* cross-node balancing */ extern int __weak arch_sd_sibiling_asym_packing (void );
kernel/sched/core.c +2 -1
@@ -1641,7 +1641,7 @@ static void __sched_fork(struct task_struct *p) p->node_stamp =0ULL ; p->numa_scan_seq = p->mm ? p->mm->numa_scan_seq :0 ; - p->numa_migrate_seq = p->mm ? p->mm->numa_scan_seq -1 :0 ; + p->numa_migrate_seq =0 ; p->numa_scan_period = sysctl_numa_balancing_scan_delay; p->numa_preferred_nid =-1 ; p->numa_work.next = &p->numa_work;
@@ -5667,6 +5667,7 @@ sd_numa_init(struct sched_domain_topology_level *tl, int cpu) | 0*SD_SHARE_PKG_RESOURCES |1 *SD_SERIALIZE |0 *SD_PREFER_SIBLING + |1 *SD_NUMA |sd_local_flags (level) , .last_balance = jiffies,
kernel/sched/fair.c +59 -4
96 lines not on this page · raw message
kernel/sched/features.h +7 -0
@@ -67,4 +67,11 @@ SCHED_FEAT(LB_MIN, false)*/ #ifdef CONFIG_NUMA_BALANCING SCHED_FEAT (NUMA , false) + +/* +* NUMA_FAVOUR_HIGHER will favor moving tasks towards nodes where a +* higher number of hinting faults are recorded during active load +* balancing. + */ +SCHED_FEAT(NUMA_FAVOUR_HIGHER, true)#endif
kernel/sysctl.c +7 -0
@@ -391,6 +391,13 @@ static struct ctl_table kern_table[] = { .mode = 0644, .proc_handler = proc_dointvec, }, + { + .procname = "numa_balancing_settle_count", + .data = &sysctl_numa_balancing_settle_count, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec, + },#endif /* CONFIG_NUMA_BALANCING */ #endif /* CONFIG_SCHED_DEBUG */ {
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 23/50 sched: Resist moving tasks towards nodes with fewer hinting faults
In reply to Mel GormanJust as "sched: Favour moving tasks towards the preferred node" favours moving tasks towards nodes with a higher number of recorded NUMA hinting faults, this patch resists moving tasks towards nodes with lower faults.
--- kernel/sched/fair.c | 33 +++++++++++++++++++++++++++++++++ kernel/sched/features.h | 8 ++++++++ 2 files changed, 41 insertions(+)
kernel/sched/fair.c +33 -0
@@ -4060,12 +4060,43 @@ static bool migrate_improves_locality(struct task_struct *p, struct lb_env *env)return false; } + + +static bool migrate_degrades_locality (struct task_struct *p,struct lb_env *env) +{ +int src_nid, dst_nid; + +if (!sched_feat (NUMA ) || !sched_feat (NUMA_RESIST_LOWER )) +return false; + +if (!p->numa_faults || !(env->sd->flags &SD_NUMA )) +return false; + + src_nid =cpu_to_node (env->src_cpu); + dst_nid =cpu_to_node (env->dst_cpu); + +if (src_nid == dst_nid || + p->numa_migrate_seq >= sysctl_numa_balancing_settle_count) +return false; + +if (p->numa_faults[dst_nid] < p->numa_faults[src_nid]) +return true; + +return false; +} +#else static inline bool migrate_improves_locality (struct task_struct *p,struct lb_env *env) {return false; } + +static inline bool migrate_degrades_locality (struct task_struct *p, +struct lb_env *env) +{ +return false; +}#endif /*
@@ -4130,6 +4161,8 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* 3) too many balance attempts have failed. */ tsk_cache_hot =task_hot (p,rq_clock_task (env->src_rq), env->sd); +if (!tsk_cache_hot) + tsk_cache_hot =migrate_degrades_locality (p, env);if (migrate_improves_locality (p, env)) {#ifdef CONFIG_SCHEDSTATS
kernel/sched/features.h +8 -0
@@ -74,4 +74,12 @@ SCHED_FEAT(NUMA, false)* balancing. */ SCHED_FEAT (NUMA_FAVOUR_HIGHER , true) + +/* +* NUMA_RESIST_LOWER will resist moving tasks towards nodes where a +* lower number of hinting faults have been recorded. As this has +* the potential to prevent a task ever migrating to a new node +* due to CPU overload it is disabled by default. + */ +SCHED_FEAT(NUMA_RESIST_LOWER, false)#endif
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 24/50 sched: Reschedule task on preferred NUMA node once selected
In reply to Mel GormanA preferred node is selected based on the node the most NUMA hinting faults was incurred on. There is no guarantee that the task is running on that node at the time so this patch rescheules the task to run on the most idle CPU of the selected node when selected. This avoids waiting for the balancer to make a decision.
--- kernel/sched/core.c | 19 +++++++++++++++++++ kernel/sched/fair.c | 46 +++++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 1 + 3 files changed, 65 insertions(+), 1 deletion(-)
kernel/sched/core.c +19 -0
@@ -4368,6 +4368,25 @@ fail:return ret; } +#ifdef CONFIG_NUMA_BALANCING +/* Migrate current task p to target_cpu */ +int migrate_task_to (struct task_struct *p,int target_cpu) +{ +struct migration_arg arg = { p, target_cpu }; +int curr_cpu =task_cpu (p); + +if (curr_cpu == target_cpu) +return 0 ; + +if (!cpumask_test_cpu (target_cpu,tsk_cpus_allowed (p))) +return -EINVAL ; + +/* TODO: This is not properly updating schedstats */ + +return stop_one_cpu (curr_cpu, migration_cpu_stop, &arg); +} +#endif +/* * migration_cpu_stop - this will be executed by a highprio stopper thread * and performs thread migration by bumping thread off CPU then
kernel/sched/fair.c +45 -1
61 lines not on this page · raw message
kernel/sched/sched.h +1 -0
@@ -555,6 +555,7 @@ static inline u64 rq_clock_task(struct rq *rq) }#ifdef CONFIG_NUMA_BALANCING +extern int migrate_task_to (struct task_struct *p,int cpu);static inline void task_numa_free (struct task_struct *p) {kfree (p->numa_faults);
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 25/50 sched: Add infrastructure for split shared/private accounting of NUMA hinting faults
In reply to Mel GormanIdeally it would be possible to distinguish between NUMA hinting faults that are private to a task and those that are shared. This patch prepares infrastructure for separately accounting shared and private faults by allocating the necessary buffers and passing in relevant information. For now, all faults are treated as private and detection will be introduced later.
--- include/linux/sched.h | 5 +++-- kernel/sched/fair.c | 46 +++++++++++++++++++++++++++++++++++----------- mm/huge_memory.c | 5 +++-- mm/memory.c | 8 ++++++-- 4 files changed, 47 insertions(+), 17 deletions(-)
include/linux/sched.h +3 -2
@@ -1430,10 +1430,11 @@ struct task_struct {#define tsk_cpus_allowed (tsk) (&(tsk)->cpus_allowed)#ifdef CONFIG_NUMA_BALANCING -extern void task_numa_fault (int node,int pages,bool migrated); +extern void task_numa_fault (int last_node,int node,int pages,bool migrated);extern void set_numabalancing_state (bool enabled);#else -static inline void task_numa_fault (int node,int pages,bool migrated) +static inline void task_numa_fault (int last_node,int node,int pages, +bool migrated) { }static inline void set_numabalancing_state (bool enabled)
kernel/sched/fair.c +35 -11
98 lines not on this page · raw message
mm/huge_memory.c +3 -2
@@ -1294,7 +1294,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct page *page;unsigned long haddr = addr &HPAGE_PMD_MASK ;int page_nid =-1 , this_nid =numa_node_id (); -int target_nid; +int target_nid, last_nid =-1 ;bool page_locked;bool migrated = false;
@@ -1305,6 +1305,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, page =pmd_page (pmd);BUG_ON (is_huge_zero_page (page)); page_nid =page_to_nid (page); + last_nid =page_nid_last (page);count_vm_numa_event (NUMA_HINT_FAULTS );if (page_nid == this_nid)count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL );
@@ -1376,7 +1377,7 @@ out:page_unlock_anon_vma_read (anon_vma);if (page_nid !=-1 ) -task_numa_fault (page_nid,HPAGE_PMD_NR , migrated); +task_numa_fault (last_nid, page_nid,HPAGE_PMD_NR , migrated);return 0 ; }
mm/memory.c +6 -2
@@ -3547,6 +3547,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct page *page =NULL ;spinlock_t *ptl;int page_nid =-1 ; +int last_nid;int target_nid;bool migrated = false;
@@ -3577,6 +3578,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }BUG_ON (is_zero_pfn (page_to_pfn (page))); + last_nid =page_nid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (ptep, ptl);
@@ -3592,7 +3594,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, out:if (page_nid !=-1 ) -task_numa_fault (page_nid,1 , migrated); +task_numa_fault (last_nid, page_nid,1 , migrated);return 0 ; }
@@ -3607,6 +3609,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,unsigned long offset;spinlock_t *ptl;bool numa = false; +int last_nid;spin_lock (&mm->page_table_lock); pmd = *pmdp;
@@ -3654,6 +3657,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,if (unlikely (page_mapcount (page) !=1 ))continue ; + last_nid =page_nid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (pte, ptl);
@@ -3666,7 +3670,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }if (page_nid !=-1 ) -task_numa_fault (page_nid,1 , migrated); +task_numa_fault (last_nid, page_nid,1 , migrated); pte =pte_offset_map_lock (mm, pmdp, addr, &ptl); }
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 26/50 sched: Check current->mm before allocating NUMA faults
In reply to Mel Gormantask_numa_placement checks current->mm but after buffers for faults have already been uselessly allocated. Move the check earlier.
--- kernel/sched/fair.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/sched/fair.c +4 -2
@@ -930,8 +930,6 @@ static void task_numa_placement(struct task_struct *p)int seq, nid, max_nid =-1 ;unsigned long max_faults =0 ; -if (!p->mm)/* for example, ksmd faulting in a user's mm */ -return ; seq =ACCESS_ONCE (p->mm->numa_scan_seq);if (p->numa_scan_seq == seq)return ;
@@ -998,6 +996,10 @@ void task_numa_fault(int last_nid, int node, int pages, bool migrated)if (!numabalancing_enabled)return ; +/* for example, ksmd faulting in a user's mm */ +if (!p->mm) +return ; +/* For now, do not attempt to detect private/shared accesses */ priv =1 ;
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 27/50 mm: numa: Scan pages with elevated page_mapcount
In reply to Mel GormanCurrently automatic NUMA balancing is unable to distinguish between false shared versus private pages except by ignoring pages with an elevated page_mapcount entirely. This avoids shared pages bouncing between the nodes whose task is using them but that is ignored quite a lot of data.
This patch kicks away the training wheels in preparation for adding support for identifying shared/private pages is now in place. The ordering is so that the impact of the shared/private detection can be easily measured. Note that the patch does not migrate shared, file-backed within vmas marked VM_EXEC as these are generally shared library pages. Migrating such pages is not beneficial as there is an expectation they are read-shared between caches and iTLB and iCache pressure is generally low.
--- include/linux/migrate.h | 7 ++++--- mm/huge_memory.c | 8 ++------ mm/memory.c | 7 ++----- mm/migrate.c | 17 ++++++----------- mm/mprotect.c | 4 +--- 5 files changed, 15 insertions(+), 28 deletions(-)
include/linux/migrate.h +4 -3
@@ -92,11 +92,12 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,#endif /* CONFIG_MIGRATION */ #ifdef CONFIG_NUMA_BALANCING -extern int migrate_misplaced_page (struct page *page,int node); -extern int migrate_misplaced_page (struct page *page,int node); +extern int migrate_misplaced_page (struct page *page, +struct vm_area_struct *vma,int node);extern bool migrate_ratelimited (int node);#else -static inline int migrate_misplaced_page (struct page *page,int node) +static inline int migrate_misplaced_page (struct page *page, +struct vm_area_struct *vma,int node) {return -EAGAIN ;/* can't migrate now */ }
mm/huge_memory.c +2 -6
@@ -1498,12 +1498,8 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, }else {struct page *page =pmd_page (*pmd); -/* -* Only check non-shared pages. See change_pte_range -* for comment on why the zero page is not modified -*/ -if (page_mapcount (page) ==1 && - !is_huge_zero_page (page) && +/* See change_pte_range about the zero page */ +if (!is_huge_zero_page (page) && !pmd_numa (*pmd)) { entry =pmdp_get_and_clear (mm, addr, pmd); entry =pmd_mknuma (entry);
mm/memory.c +2 -5
@@ -3588,7 +3588,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }/* Migrate to the requested node */ - migrated =migrate_misplaced_page (page, target_nid); + migrated =migrate_misplaced_page (page, vma, target_nid);if (migrated) page_nid = target_nid;
@@ -3653,16 +3653,13 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, page =vm_normal_page (vma, addr, pteval);if (unlikely (!page))continue ; -/* only check non-shared pages */ -if (unlikely (page_mapcount (page) !=1 )) -continue ; last_nid =page_nid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (pte, ptl);if (target_nid !=-1 ) { - migrated =migrate_misplaced_page (page, target_nid); + migrated =migrate_misplaced_page (page, vma, target_nid);if (migrated) page_nid = target_nid; }else {
mm/migrate.c +6 -11
@@ -1596,7 +1596,8 @@ int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page)* node. Caller is expected to have an elevated reference count on * the page that will be dropped by this function before returning. */ -int migrate_misplaced_page (struct page *page,int node) +int migrate_misplaced_page (struct page *page,struct vm_area_struct *vma, +int node) {pg_data_t *pgdat =NODE_DATA (node);int isolated;
@@ -1604,10 +1605,11 @@ int migrate_misplaced_page(struct page *page, int node)LIST_HEAD (migratepages);/* -* Don't migrate pages that are mapped in multiple processes. -* TODO: Handle false sharing detection instead of this hammer +* Don't migrate file pages that are mapped in multiple processes +* with execute permissions as they are probably shared libraries. */ -if (page_mapcount (page) !=1 ) +if (page_mapcount (page) !=1 &&page_is_file_cache (page) && + (vma->vm_flags &VM_EXEC )) goto out;/*
@@ -1658,13 +1660,6 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,int page_lru =page_is_file_cache (page);/* -* Don't migrate pages that are mapped in multiple processes. -* TODO: Handle false sharing detection instead of this hammer -*/ -if (page_mapcount (page) !=1 ) - goto out_dropref; - -/* * Rate-limit the amount of data that is being migrated to a node. * Optimal placement is no good if the memory bus is saturated and * all the time is being spent migrating!
mm/mprotect.c +1 -3
@@ -77,9 +77,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,if (last_nid != this_nid) all_same_node = false; -/* only check non-shared pages */ -if (!pte_numa (oldpte) && -page_mapcount (page) ==1 ) { +if (!pte_numa (oldpte)) { ptent =pte_mknuma (ptent); updated = true; }
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 27/50] mm: numa: Scan pages with elevated page_mapcount
Hillo Mel
Quoted 15 lines · Mel Gorman
Currently automatic NUMA balancing is unable to distinguish between false shared versus private pages except by ignoring pages with an elevated
page_mapcountentirely. This avoids shared pages bouncing between the nodes whose task is using them but that is ignored quite a lot of data.This patch kicks away the training wheels in preparation for adding support for identifying shared/private pages is now in place. The ordering is so that the impact of the shared/private detection can be easily measured. Note that the patch does not migrate shared, file-backed within vmas marked
VM_EXECas these are generally shared library pages. Migrating such pages is not beneficial as there is an expectation they are read-shared between caches and iTLB and iCache pressure is generally low.Signed-off-by Mel Gorman <[email protected]>---
[...]
mm/migrate.c int migrate_misplaced_transhuge_page(struct mm_struct *mm, · line 1666
Quoted 10 lines
mm/migrate.c +0
-6@@ -1658,13 +1660,6 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,int page_lru =page_is_file_cache (page);/* -* Don't migrate pages that are mapped in multiple processes. -* TODO: Handle false sharing detection instead of this hammer -*/ -if (page_mapcount (page) !=1 ) - goto out_dropref; -
Is there rmap walk when migrating THP?
mm/migrate.c int migrate_misplaced_transhuge_page(struct mm_struct *mm, · line 1665
Quoted 4 lines
mm/migrate.c +0
-1-/* * Rate-limit the amount of data that is being migrated to a node. * Optimal placement is no good if the memory bus is saturated and * all the time is being spent migrating!
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerRe: [PATCH 27/50] mm: numa: Scan pages with elevated page_mapcount
Quoted 22 lines · Hillf Danton
Hillo Mel
Quoted 15 lines · Mel Gorman
Currently automatic NUMA balancing is unable to distinguish between false shared versus private pages except by ignoring pages with an elevated
page_mapcountentirely. This avoids shared pages bouncing between the nodes whose task is using them but that is ignored quite a lot of data.This patch kicks away the training wheels in preparation for adding support for identifying shared/private pages is now in place. The ordering is so that the impact of the shared/private detection can be easily measured. Note that the patch does not migrate shared, file-backed within vmas marked
VM_EXECas these are generally shared library pages. Migrating such pages is not beneficial as there is an expectation they are read-shared between caches and iTLB and iCache pressure is generally low.Signed-off-by Mel Gorman <[email protected]>---
[...]
Quoted 3 lines
mm/migrate.c +0
-0@@ -1658,13 +1660,6 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,int page_lru =page_is_file_cache (page);
Quoted 9 lines
mm/migrate.c int migrate_misplaced_transhuge_page(struct mm_struct *mm, · line 1666
Quoted 7 lines
mm/migrate.c +0
-6/* -* Don't migrate pages that are mapped in multiple processes. -* TODO: Handle false sharing detection instead of this hammer -*/ -if (page_mapcount (page) !=1 ) - goto out_dropref; -Is there rmap walk when migrating THP?
Should not be necessary for THP.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 28/50 sched: Remove check that skips small VMAs
In reply to Mel Gormantask_numa_work skips small VMAs. At the time the logic was to reduce the scanning overhead which was considerable. It is a dubious hack at best. It would make much more sense to cache where faults have been observed and only rescan those regions during subsequent PTE scans. Remove this hack as motivation to do it properly in the future.
--- kernel/sched/fair.c | 4 ---- 1 file changed, 4 deletions(-)
kernel/sched/fair.c +0 -4
@@ -1127,10 +1127,6 @@ void task_numa_work(struct callback_head *work)if (!vma_migratable (vma))continue ; -/* Skip small VMAs. They are not likely to be of relevance */ -if (vma->vm_end - vma->vm_start <HPAGE_SIZE ) -continue ; - do { start = max(start, vma->vm_start); end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 29/50 sched: Set preferred NUMA node based on number of private faults
In reply to Mel GormanIdeally it would be possible to distinguish between NUMA hinting faults that are private to a task and those that are shared. If treated identically there is a risk that shared pages bounce between nodes depending on the order they are referenced by tasks. Ultimately what is desirable is that task private pages remain local to the task while shared pages are interleaved between sharing tasks running on different nodes to give good average performance. This is further complicated by THP as even applications that partition their data may not be partitioning on a huge page boundary.
To start with, this patch assumes that multi-threaded or multi-process applications partition their data and that in general the private accesses are more important for cpu->memory locality in the general case. Also, no new infrastructure is required to treat private pages properly but interleaving for shared pages requires additional infrastructure.
To detect private accesses the pid of the last accessing task is required but the storage requirements are a high. This patch borrows heavily from Ingo Molnar's patch "numa, mm, sched: Implement last-CPU+PID hash tracking" to encode some bits from the last accessing task in the page flags as well as the node information. Collisions will occur but it is better than just depending on the node information. Node information is then used to determine if a page needs to migrate. The PID information is used to detect private/shared accesses. The preferred NUMA node is selected based on where the maximum number of approximately private faults were measured. Shared faults are not taken into consideration for a few reasons.
First, if there are many tasks sharing the page then they'll all move towards the same node. The node will be compute overloaded and then scheduled away later only to bounce back again. Alternatively the shared tasks would just bounce around nodes because the fault information is effectively noise. Either way accounting for shared faults the same as private faults can result in lower performance overall.
The second reason is based on a hypothetical workload that has a small number of very important, heavily accessed private pages but a large shared array. The shared array would dominate the number of faults and be selected as a preferred node even though it's the wrong decision.
The third reason is that multiple threads in a process will race each other to fault the shared page making the fault information unreliable.
--- include/linux/mm.h | 89 +++++++++++++++++++++++++++++---------- include/linux/mm_types.h | 4 +- include/linux/page-flags-layout.h | 28 +++++++----- kernel/sched/fair.c | 12 ++++-- mm/huge_memory.c | 8 ++-- mm/memory.c | 16 +++---- mm/mempolicy.c | 8 ++-- mm/migrate.c | 4 +- mm/mm_init.c | 18 ++++---- mm/mmzone.c | 14 +++--- mm/mprotect.c | 26 ++++++++---- mm/page_alloc.c | 4 +- 12 files changed, 149 insertions(+), 82 deletions(-)
include/linux/mm.h +67 -22
@@ -588,11 +588,11 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)* sets it, so none of the operations on it need to be atomic. */ -/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_NID] | ... | FLAGS | */ +/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_NIDPID] | ... | FLAGS | */ #define SECTIONS_PGOFF ((sizeof(unsigned long)*8) - SECTIONS_WIDTH)#define NODES_PGOFF (SECTIONS_PGOFF - NODES_WIDTH)#define ZONES_PGOFF (NODES_PGOFF - ZONES_WIDTH) -#define LAST_NID_PGOFF (ZONES_PGOFF - LAST_NID_WIDTH) +#define LAST_NIDPID_PGOFF (ZONES_PGOFF - LAST_NIDPID_WIDTH)/* * Define the bit shifts to access each section. For non-existent
@@ -602,7 +602,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)#define SECTIONS_PGSHIFT (SECTIONS_PGOFF * (SECTIONS_WIDTH != 0))#define NODES_PGSHIFT (NODES_PGOFF * (NODES_WIDTH != 0))#define ZONES_PGSHIFT (ZONES_PGOFF * (ZONES_WIDTH != 0)) -#define LAST_NID_PGSHIFT (LAST_NID_PGOFF * (LAST_NID_WIDTH != 0)) +#define LAST_NIDPID_PGSHIFT (LAST_NIDPID_PGOFF * (LAST_NIDPID_WIDTH != 0))/* NODE:ZONE or SECTION:ZONE is used to ID a zone for the buddy allocator */ #ifdef NODE_NOT_IN_PAGE_FLAGS
@@ -624,7 +624,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)#define ZONES_MASK ((1UL << ZONES_WIDTH) - 1)#define NODES_MASK ((1UL << NODES_WIDTH) - 1)#define SECTIONS_MASK ((1UL << SECTIONS_WIDTH) - 1) -#define LAST_NID_MASK ((1UL << LAST_NID_WIDTH) - 1) +#define LAST_NIDPID_MASK ((1UL << LAST_NIDPID_WIDTH) - 1)#define ZONEID_MASK ((1UL << ZONEID_SHIFT) - 1)static inline enum zone_type page_zonenum (const struct page *page)
@@ -668,48 +668,93 @@ static inline int page_to_nid(const struct page *page)#endif #ifdef CONFIG_NUMA_BALANCING -#ifdef LAST_NID_NOT_IN_PAGE_FLAGS -static inline int page_nid_xchg_last (struct page *page,int nid) +static inline int nid_pid_to_nidpid (int nid,int pid) { -return xchg (&page->_last_nid, nid); +return ((nid &LAST__NID_MASK ) <<LAST__PID_SHIFT ) | (pid &LAST__PID_MASK ); } -static inline int page_nid_last (struct page *page) +static inline int nidpid_to_pid (int nidpid) { -return page->_last_nid; +return nidpid &LAST__PID_MASK ; } -static inline void page_nid_reset_last (struct page *page) + +static inline int nidpid_to_nid (int nidpid) +{ +return (nidpid >>LAST__PID_SHIFT ) &LAST__NID_MASK ; +} + +static inline bool nidpid_pid_unset (int nidpid) +{ +return nidpid_to_pid (nidpid) == (-1 &LAST__PID_MASK ); +} + +static inline bool nidpid_nid_unset (int nidpid) { - page->_last_nid =-1 ; +return nidpid_to_nid (nidpid) == (-1 &LAST__NID_MASK ); +} + +#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS +static inline int page_nidpid_xchg_last (struct page *page,int nid) +{ +return xchg (&page->_last_nidpid, nid); +} + +static inline int page_nidpid_last (struct page *page) +{ +return page->_last_nidpid; +} +static inline void page_nidpid_reset_last (struct page *page) +{ + page->_last_nidpid =-1 ; }#else -static inline int page_nid_last (struct page *page) +static inline int page_nidpid_last (struct page *page) { -return (page->flags >>LAST_NID_PGSHIFT ) &LAST_NID_MASK ; +return (page->flags >>LAST_NIDPID_PGSHIFT ) &LAST_NIDPID_MASK ; } -extern int page_nid_xchg_last (struct page *page,int nid); +extern int page_nidpid_xchg_last (struct page *page,int nidpid); -static inline void page_nid_reset_last (struct page *page) +static inline void page_nidpid_reset_last (struct page *page) { -int nid = (1 <<LAST_NID_SHIFT ) -1 ; +int nidpid = (1 <<LAST_NIDPID_SHIFT ) -1 ; - page->flags &= ~(LAST_NID_MASK <<LAST_NID_PGSHIFT ); - page->flags |= (nid &LAST_NID_MASK ) <<LAST_NID_PGSHIFT ; + page->flags &= ~(LAST_NIDPID_MASK <<LAST_NIDPID_PGSHIFT ); + page->flags |= (nidpid &LAST_NIDPID_MASK ) <<LAST_NIDPID_PGSHIFT ; } -#endif /* LAST_NID_NOT_IN_PAGE_FLAGS */ +#endif /* LAST_NIDPID_NOT_IN_PAGE_FLAGS */ #else -static inline int page_nid_xchg_last (struct page *page,int nid) +static inline int page_nidpid_xchg_last (struct page *page,int nidpid) {return page_to_nid (page); } -static inline int page_nid_last (struct page *page) +static inline int page_nidpid_last (struct page *page) {return page_to_nid (page); } -static inline void page_nid_reset_last (struct page *page) +static inline int nidpid_to_nid (int nidpid) +{ +return -1 ; +} + +static inline int nidpid_to_pid (int nidpid) +{ +return -1 ; +} + +static inline int nid_pid_to_nidpid (int nid,int pid) +{ +return -1 ; +} + +static inline bool nidpid_pid_unset (int nidpid) +{ +return 1 ; +} + +static inline void page_nidpid_reset_last (struct page *page) { }#endif
include/linux/mm_types.h +2 -2
@@ -174,8 +174,8 @@ struct page {void *shadow;#endif -#ifdef LAST_NID_NOT_IN_PAGE_FLAGS -int _last_nid; +#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS +int _last_nidpid;#endif }/*
include/linux/page-flags-layout.h +17 -11
@@ -38,10 +38,10 @@* The last is when there is insufficient space in page->flags and a separate * lookup is necessary. * -* No sparsemem or sparsemem vmemmap: | NODE | ZONE | ... | FLAGS | -* " plus space for last_nid: | NODE | ZONE | LAST_NID ... | FLAGS | -* classic sparse with space for node:| SECTION | NODE | ZONE | ... | FLAGS | -* " plus space for last_nid: | SECTION | NODE | ZONE | LAST_NID ... | FLAGS | +* No sparsemem or sparsemem vmemmap: | NODE | ZONE | ... | FLAGS | +* " plus space for last_nidpid: | NODE | ZONE | LAST_NIDPID ... | FLAGS | +* classic sparse with space for node:| SECTION | NODE | ZONE | ... | FLAGS | +* " plus space for last_nidpid: | SECTION | NODE | ZONE | LAST_NIDPID ... | FLAGS | * classic sparse no space for node: | SECTION | ZONE | ... | FLAGS | */ #if defined(CONFIG_SPARSEMEM ) && !defined(CONFIG_SPARSEMEM_VMEMMAP )
@@ -62,15 +62,21 @@#endif #ifdef CONFIG_NUMA_BALANCING -#define LAST_NID_SHIFT NODES_SHIFT +#define LAST__PID_SHIFT 8 +#define LAST__PID_MASK ((1 << LAST__PID_SHIFT)-1) + +#define LAST__NID_SHIFT NODES_SHIFT +#define LAST__NID_MASK ((1 << LAST__NID_SHIFT)-1) + +#define LAST_NIDPID_SHIFT (LAST__PID_SHIFT+LAST__NID_SHIFT)#else -#define LAST_NID_SHIFT 0 +#define LAST_NIDPID_SHIFT 0#endif -#if SECTIONS_WIDTH +ZONES_WIDTH +NODES_SHIFT +LAST_NID_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS -#define LAST_NID_WIDTH LAST_NID_SHIFT +#if SECTIONS_WIDTH +ZONES_WIDTH +NODES_SHIFT +LAST_NIDPID_SHIFT <=BITS_PER_LONG -NR_PAGEFLAGS +#define LAST_NIDPID_WIDTH LAST_NIDPID_SHIFT#else -#define LAST_NID_WIDTH 0 +#define LAST_NIDPID_WIDTH 0#endif /*
@@ -81,8 +87,8 @@#define NODE_NOT_IN_PAGE_FLAGS #endif -#if defined(CONFIG_NUMA_BALANCING ) &&LAST_NID_WIDTH ==0 -#define LAST_NID_NOT_IN_PAGE_FLAGS +#if defined(CONFIG_NUMA_BALANCING ) &&LAST_NIDPID_WIDTH ==0 +#define LAST_NIDPID_NOT_IN_PAGE_FLAGS #endif #endif /* _LINUX_PAGE_FLAGS_LAYOUT */
kernel/sched/fair.c +9 -3
@@ -988,7 +988,7 @@ static void task_numa_placement(struct task_struct *p)/* * Got a PROT_NONE fault for a page on @node. */ -void task_numa_fault (int last_nid,int node,int pages,bool migrated) +void task_numa_fault (int last_nidpid,int node,int pages,bool migrated) {struct task_struct *p = current;int priv;
@@ -1000,8 +1000,14 @@ void task_numa_fault(int last_nid, int node, int pages, bool migrated)if (!p->mm)return ; -/* For now, do not attempt to detect private/shared accesses */ - priv =1 ; +/* +* First accesses are treated as private, otherwise consider accesses +* to be private if the accessing pid has not changed +*/ +if (!nidpid_pid_unset (last_nidpid)) + priv = ((p->pid &LAST__PID_MASK ) ==nidpid_to_pid (last_nidpid)); +else + priv =1 ;/* Allocate buffer to track faults on a per-node basis */ if (unlikely (!p->numa_faults)) {
mm/huge_memory.c +4 -4
@@ -1294,7 +1294,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct page *page;unsigned long haddr = addr &HPAGE_PMD_MASK ;int page_nid =-1 , this_nid =numa_node_id (); -int target_nid, last_nid =-1 ; +int target_nid, last_nidpid =-1 ;bool page_locked;bool migrated = false;
@@ -1305,7 +1305,7 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, page =pmd_page (pmd);BUG_ON (is_huge_zero_page (page)); page_nid =page_to_nid (page); - last_nid =page_nid_last (page); + last_nidpid =page_nidpid_last (page);count_vm_numa_event (NUMA_HINT_FAULTS );if (page_nid == this_nid)count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL );
@@ -1377,7 +1377,7 @@ out:page_unlock_anon_vma_read (anon_vma);if (page_nid !=-1 ) -task_numa_fault (last_nid, page_nid,HPAGE_PMD_NR , migrated); +task_numa_fault (last_nidpid, page_nid,HPAGE_PMD_NR , migrated);return 0 ; }
@@ -1692,7 +1692,7 @@ static void __split_huge_page_refcount(struct page *page, page_tail->mapping = page->mapping; page_tail->index = page->index + i; -page_nid_xchg_last (page_tail,page_nid_last (page)); +page_nidpid_xchg_last (page_tail,page_nidpid_last (page));BUG_ON (!PageAnon (page_tail));BUG_ON (!PageUptodate (page_tail));
mm/memory.c +8 -8
@@ -69,8 +69,8 @@#include "internal.h" -#ifdef LAST_NID_NOT_IN_PAGE_FLAGS -#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_nid. +#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS +#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_nidpid.#endif #ifndef CONFIG_NEED_MULTIPLE_NODES
@@ -3547,7 +3547,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,struct page *page =NULL ;spinlock_t *ptl;int page_nid =-1 ; -int last_nid; +int last_nidpid;int target_nid;bool migrated = false;
@@ -3578,7 +3578,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }BUG_ON (is_zero_pfn (page_to_pfn (page))); - last_nid =page_nid_last (page); + last_nidpid =page_nidpid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (ptep, ptl);
@@ -3594,7 +3594,7 @@ int do_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, out:if (page_nid !=-1 ) -task_numa_fault (last_nid, page_nid,1 , migrated); +task_numa_fault (last_nidpid, page_nid,1 , migrated);return 0 ; }
@@ -3609,7 +3609,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,unsigned long offset;spinlock_t *ptl;bool numa = false; -int last_nid; +int last_nidpid;spin_lock (&mm->page_table_lock); pmd = *pmdp;
@@ -3654,7 +3654,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,if (unlikely (!page))continue ; - last_nid =page_nid_last (page); + last_nidpid =page_nidpid_last (page); page_nid =page_to_nid (page); target_nid =numa_migrate_prep (page, vma, addr, page_nid);pte_unmap_unlock (pte, ptl);
@@ -3667,7 +3667,7 @@ static int do_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma, }if (page_nid !=-1 ) -task_numa_fault (last_nid, page_nid,1 , migrated); +task_numa_fault (last_nidpid, page_nid,1 , migrated); pte =pte_offset_map_lock (mm, pmdp, addr, &ptl); }
mm/mempolicy.c +5 -3
@@ -2292,9 +2292,11 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long/* Migrate the page towards the node whose CPU is referencing it */ if (pol->flags &MPOL_F_MORON ) { -int last_nid; +int last_nidpid; +int this_nidpid; polnid =numa_node_id (); + this_nidpid =nid_pid_to_nidpid (polnid, current->pid);/* * Multi-stage node selection is used in conjunction
@@ -2317,8 +2319,8 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long* it less likely we act on an unlikely task<->page * relation. */ - last_nid =page_nid_xchg_last (page, polnid); -if (last_nid != polnid) + last_nidpid =page_nidpid_xchg_last (page, this_nidpid); +if (!nidpid_pid_unset (last_nidpid) &&nidpid_to_nid (last_nidpid) != polnid) goto out; }
mm/migrate.c +2 -2
@@ -1495,7 +1495,7 @@ static struct page *alloc_misplaced_dst_page(struct page *page, __GFP_NOWARN) & ~GFP_IOFS, 0);if (newpage) -page_nid_xchg_last (newpage,page_nid_last (page)); +page_nidpid_xchg_last (newpage,page_nidpid_last (page));return newpage; }
@@ -1672,7 +1672,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,if (!new_page) goto out_fail; -page_nid_xchg_last (new_page,page_nid_last (page)); +page_nidpid_xchg_last (new_page,page_nidpid_last (page)); isolated =numamigrate_isolate_page (pgdat, page);if (!isolated) {
mm/mm_init.c +9 -9
@@ -71,26 +71,26 @@ void __init mminit_verify_pageflags_layout(void)unsigned long or_mask, add_mask; shift =8 *sizeof (unsigned long ); - width = shift -SECTIONS_WIDTH -NODES_WIDTH -ZONES_WIDTH -LAST_NID_SHIFT ; + width = shift -SECTIONS_WIDTH -NODES_WIDTH -ZONES_WIDTH -LAST_NIDPID_SHIFT ;mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_widths" , -"Section %d Node %d Zone %d Lastnid %d Flags %d\n" , +"Section %d Node %d Zone %d Lastnidpid %d Flags %d\n" ,SECTIONS_WIDTH ,NODES_WIDTH ,ZONES_WIDTH , -LAST_NID_WIDTH , +LAST_NIDPID_WIDTH ,NR_PAGEFLAGS );mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_shifts" , -"Section %d Node %d Zone %d Lastnid %d\n" , +"Section %d Node %d Zone %d Lastnidpid %d\n" ,SECTIONS_SHIFT ,NODES_SHIFT ,ZONES_SHIFT , -LAST_NID_SHIFT ); +LAST_NIDPID_SHIFT );mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_pgshifts" , -"Section %lu Node %lu Zone %lu Lastnid %lu\n" , +"Section %lu Node %lu Zone %lu Lastnidpid %lu\n" , (unsigned long )SECTIONS_PGSHIFT , (unsigned long )NODES_PGSHIFT , (unsigned long )ZONES_PGSHIFT , - (unsigned long )LAST_NID_PGSHIFT ); + (unsigned long )LAST_NIDPID_PGSHIFT ); mminit_dprintk(MMINIT_TRACE, "pageflags_layout_nodezoneid", "Node/Zone ID: %lu -> %lu\n", (unsigned long)(ZONEID_PGOFF + ZONEID_SHIFT),
@@ -102,9 +102,9 @@ void __init mminit_verify_pageflags_layout(void)mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_nodeflags" ,"Node not in page flags" );#endif -#ifdef LAST_NID_NOT_IN_PAGE_FLAGS +#ifdef LAST_NIDPID_NOT_IN_PAGE_FLAGS mminit_dprintk (MMINIT_TRACE ,"pageflags_layout_nodeflags" , -"Last nid not in page flags" ); +"Last nidpid not in page flags" );#endif if (SECTIONS_WIDTH ) {
mm/mmzone.c +7 -7
@@ -97,20 +97,20 @@ void lruvec_init(struct lruvec *lruvec)INIT_LIST_HEAD (&lruvec->lists[lru]); } -#if defined(CONFIG_NUMA_BALANCING ) && !defined(LAST_NID_NOT_IN_PAGE_FLAGS ) -int page_nid_xchg_last (struct page *page,int nid) +#if defined(CONFIG_NUMA_BALANCING ) && !defined(LAST_NIDPID_NOT_IN_PAGE_FLAGS ) +int page_nidpid_xchg_last (struct page *page,int nidpid) {unsigned long old_flags, flags; -int last_nid; +int last_nidpid;do { old_flags = flags = page->flags; - last_nid =page_nid_last (page); + last_nidpid =page_nidpid_last (page); - flags &= ~(LAST_NID_MASK <<LAST_NID_PGSHIFT ); - flags |= (nid &LAST_NID_MASK ) <<LAST_NID_PGSHIFT ; + flags &= ~(LAST_NIDPID_MASK <<LAST_NIDPID_PGSHIFT ); + flags |= (nidpid &LAST_NIDPID_MASK ) <<LAST_NIDPID_PGSHIFT ; }while (unlikely (cmpxchg (&page->flags, old_flags, flags) != old_flags)); -return last_nid; +return last_nidpid; }#endif
mm/mprotect.c +17 -9
@@ -37,14 +37,15 @@ static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot)static unsigned long change_pte_range (struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot, -int dirty_accountable,int prot_numa,bool *ret_all_same_node) +int dirty_accountable,int prot_numa,bool *ret_all_same_nidpid) {struct mm_struct *mm = vma->vm_mm;pte_t *pte, oldpte;spinlock_t *ptl;unsigned long pages =0 ; -bool all_same_node = true; +bool all_same_nidpid = true;int last_nid =-1 ; +int last_pid =-1 ; pte =pte_offset_map_lock (mm, pmd, addr, &ptl);arch_enter_lazy_mmu_mode ();
@@ -71,11 +72,18 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,* hits on the zero page */ if (page && !is_zero_pfn (page_to_pfn (page))) { -int this_nid =page_to_nid (page); +int nidpid =page_nidpid_last (page); +int this_nid =nidpid_to_nid (nidpid); +int this_pid =nidpid_to_pid (nidpid); +if (last_nid ==-1 ) last_nid = this_nid; -if (last_nid != this_nid) - all_same_node = false; +if (last_pid ==-1 ) + last_pid = this_pid; +if (last_nid != this_nid || + last_pid != this_pid) { + all_same_nidpid = false; + }if (!pte_numa (oldpte)) { ptent =pte_mknuma (ptent);
@@ -115,7 +123,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,arch_leave_lazy_mmu_mode ();pte_unmap_unlock (pte -1 , ptl); - *ret_all_same_node = all_same_node; + *ret_all_same_nidpid = all_same_nidpid;return pages; }
@@ -142,7 +150,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,pmd_t *pmd;unsigned long next;unsigned long pages =0 ; -bool all_same_node; +bool all_same_nidpid; pmd =pmd_offset (pud, addr); do {
@@ -166,7 +174,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,if (pmd_none_or_clear_bad (pmd))continue ; pages +=change_pte_range (vma, pmd, addr, next, newprot, - dirty_accountable, prot_numa, &all_same_node); + dirty_accountable, prot_numa, &all_same_nidpid);/* * If we are changing protections for NUMA hinting faults then
@@ -174,7 +182,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,* node. This allows a regular PMD to be handled as one fault * and effectively batches the taking of the PTL */ -if (prot_numa && all_same_node) +if (prot_numa && all_same_nidpid)change_pmd_protnuma (vma->vm_mm, addr, pmd); }while (pmd++, addr = next, addr != end);
mm/page_alloc.c +2 -2
@@ -622,7 +622,7 @@ static inline int free_pages_check(struct page *page)bad_page (page);return 1 ; } -page_nid_reset_last (page); +page_nidpid_reset_last (page);if (page->flags &PAGE_FLAGS_CHECK_AT_PREP ) page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP ;return 0 ;
@@ -3944,7 +3944,7 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,mminit_verify_page_links (page, zone, nid, pfn);init_page_count (page);page_mapcount_reset (page); -page_nid_reset_last (page); +page_nidpid_reset_last (page);SetPageReserved (page);/* * Mark the block movable so that blocks are reserved for
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 30/50 sched: Do not migrate memory immediately after switching node
In reply to Mel GormanFrom: Rik van Riel <[email protected]>
The load balancer can move tasks between nodes and does not take NUMA locality into account. With automatic NUMA balancing this may result in the tasks working set being migrated to the new node. However, as the fault buffer will still store faults from the old node the schduler may decide to reset the preferred node and migrate the task back resulting in more migrations.
The ideal would be that the scheduler did not migrate tasks with a heavy memory footprint but this may result nodes being overloaded. We could also discard the fault information on task migration but this would still cause all the tasks working set to be migrated. This patch simply avoids migrating the memory for a short time after a task is migrated.
Not-signed-off: Rik van Riel
--- kernel/sched/core.c | 2 +- kernel/sched/fair.c | 18 ++++++++++++++++-- mm/mempolicy.c | 12 ++++++++++++ 3 files changed, 29 insertions(+), 3 deletions(-)
kernel/sched/core.c +1 -1
@@ -1641,7 +1641,7 @@ static void __sched_fork(struct task_struct *p) p->node_stamp =0ULL ; p->numa_scan_seq = p->mm ? p->mm->numa_scan_seq :0 ; - p->numa_migrate_seq =0 ; + p->numa_migrate_seq =1 ; p->numa_scan_period = sysctl_numa_balancing_scan_delay; p->numa_preferred_nid =-1 ; p->numa_work.next = &p->numa_work;
kernel/sched/fair.c +16 -2
@@ -884,7 +884,7 @@ static unsigned int task_scan_max(struct task_struct *p)* the preferred node but still allow the scheduler to move the task again if * the nodes CPUs are overloaded. */ -unsigned int sysctl_numa_balancing_settle_count__read_mostly =3 ; +unsigned int sysctl_numa_balancing_settle_count__read_mostly =4 ;static inline int task_faults_idx (int nid,int priv) {
@@ -980,7 +980,7 @@ static void task_numa_placement(struct task_struct *p)/* Update the preferred nid and migrate task if possible */ p->numa_preferred_nid = max_nid; - p->numa_migrate_seq =0 ; + p->numa_migrate_seq =1 ;migrate_task_to (p, preferred_cpu); } }
@@ -4074,6 +4074,20 @@ static void move_task(struct task_struct *p, struct lb_env *env)set_task_cpu (p, env->dst_cpu);activate_task (env->dst_rq, p,0 );check_preempt_curr (env->dst_rq, p,0 ); +#ifdef CONFIG_NUMA_BALANCING +if (p->numa_preferred_nid !=-1 ) { +int src_nid =cpu_to_node (env->src_cpu); +int dst_nid =cpu_to_node (env->dst_cpu); + +/* +* If the load balancer has moved the task then limit +* migrations from taking place in the short term in +* case this is a short-lived migration. +*/ +if (src_nid != dst_nid && dst_nid != p->numa_preferred_nid) + p->numa_migrate_seq =0 ; + } +#endif }/*
mm/mempolicy.c +12 -0
@@ -2322,6 +2322,18 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long last_nidpid =page_nidpid_xchg_last (page, this_nidpid);if (!nidpid_pid_unset (last_nidpid) &&nidpid_to_nid (last_nidpid) != polnid) goto out; + +#ifdef CONFIG_NUMA_BALANCING +/* +* If the scheduler has just moved us away from our +* preferred node, do not bother migrating pages yet. +* This way a short and temporary process migration will +* not cause excessive memory migration. +*/ +if (polnid != current->numa_preferred_nid && + !current->numa_migrate_seq) + goto out; +#endif }if (curnid != polnid)
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 31/50 sched: Avoid overloading CPUs on a preferred NUMA node
In reply to Mel GormanThis patch replaces find_idlest_cpu_node with task_numa_find_cpu. find_idlest_cpu_node has two critical limitations. It does not take the scheduling class into account when calculating the load and it is unsuitable for using when comparing loads between NUMA nodes.
task_numa_find_cpu uses similar load calculations to wake_affine() when selecting the least loaded CPU within a scheduling domain common to the source and destimation nodes. It avoids causing CPU load imbalances in the machine by refusing to migrate if the relative load on the target CPU is higher than the source CPU.
--- kernel/sched/fair.c | 131 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 102 insertions(+), 29 deletions(-)
kernel/sched/fair.c +102 -29
@@ -901,28 +901,114 @@ static inline unsigned long task_faults(struct task_struct *p, int nid) }static unsigned long weighted_cpuload (const int cpu); +static unsigned long source_load (int cpu,int type); +static unsigned long target_load (int cpu,int type); +static unsigned long power_of (int cpu); +static long effective_load (struct task_group *tg,int cpu,long wl,long wg); +struct numa_stats { +unsigned long load; +s64 eff_load; +unsigned long faults; +}; -static int -find_idlest_cpu_node (int this_cpu,int nid) -{ -unsigned long load, min_load =ULONG_MAX ; -int i, idlest_cpu = this_cpu; +struct task_numa_env { +struct task_struct *p; -BUG_ON (cpu_to_node (this_cpu) == nid); +int src_cpu, src_nid; +int dst_cpu, dst_nid; -rcu_read_lock (); -for_each_cpu (i,cpumask_of_node (nid)) { - load =weighted_cpuload (i); +struct numa_stats src_stats, dst_stats; -if (load < min_load) { - min_load = load; - idlest_cpu = i; +unsigned long best_load; +int best_cpu; +}; + +static int task_numa_migrate (struct task_struct *p) +{ +int node_cpu =cpumask_first (cpumask_of_node (p->numa_preferred_nid)); +struct task_numa_env env = { + .p = p, + .src_cpu =task_cpu (p), + .src_nid =cpu_to_node (task_cpu (p)), + .dst_cpu = node_cpu, + .dst_nid = p->numa_preferred_nid, + .best_load =ULONG_MAX , + .best_cpu =task_cpu (p), + }; +struct sched_domain *sd; +int cpu; +struct task_group *tg =task_group (p); +unsigned long weight; +bool balanced; +int imbalance_pct, idx =-1 ; + +/* +* Find the lowest common scheduling domain covering the nodes of both +* the CPU the task is currently running on and the target NUMA node. +*/ +rcu_read_lock (); +for_each_domain (env.src_cpu, sd) { +if (cpumask_test_cpu (node_cpu,sched_domain_span (sd))) { +/* +* busy_idx is used for the load decision as it is the +* same index used by the regular load balancer for an +* active cpu. +*/ + idx = sd->busy_idx; + imbalance_pct = sd->imbalance_pct; +break ; } }rcu_read_unlock (); -return idlest_cpu; +if (WARN_ON_ONCE (idx ==-1 )) +return 0 ; + +/* +* XXX the below is mostly nicked from wake_affine(); we should +* see about sharing a bit if at all possible; also it might want +* some per entity weight love. +*/ + weight = p->se.load.weight; + env.src_stats.load =source_load (env.src_cpu, idx); + env.src_stats.eff_load =100 + (imbalance_pct -100 ) /2 ; + env.src_stats.eff_load *=power_of (env.src_cpu); + env.src_stats.eff_load *= env.src_stats.load +effective_load (tg, env.src_cpu, -weight, -weight); + +for_each_cpu (cpu,cpumask_of_node (env.dst_nid)) { + env.dst_cpu = cpu; + env.dst_stats.load =target_load (cpu, idx); + +/* If the CPU is idle, use it */ +if (!env.dst_stats.load) { + env.best_cpu = cpu; + goto migrate; + } + +/* Otherwise check the target CPU load */ + env.dst_stats.eff_load =100 ; + env.dst_stats.eff_load *=power_of (cpu); + env.dst_stats.eff_load *= env.dst_stats.load +effective_load (tg, cpu, weight, weight); + +/* +* Destination is considered balanced if the destination CPU is +* less loaded than the source CPU. Unfortunately there is a +* risk that a task running on a lightly loaded CPU will not +* migrate to its preferred node due to load imbalances. +*/ + balanced = (env.dst_stats.eff_load <= env.src_stats.eff_load); +if (!balanced) +continue ; + +if (env.dst_stats.eff_load < env.best_load) { + env.best_load = env.dst_stats.eff_load; + env.best_cpu = cpu; + } + } + +migrate: +return migrate_task_to (p, env.best_cpu); }static void task_numa_placement (struct task_struct *p)
@@ -966,22 +1052,10 @@ static void task_numa_placement(struct task_struct *p)* the working set placement. */ if (max_faults && max_nid != p->numa_preferred_nid) { -int preferred_cpu; - -/* -* If the task is not on the preferred node then find the most -* idle CPU to migrate to. -*/ - preferred_cpu =task_cpu (p); -if (cpu_to_node (preferred_cpu) != max_nid) { - preferred_cpu =find_idlest_cpu_node (preferred_cpu, - max_nid); - } -/* Update the preferred nid and migrate task if possible */ p->numa_preferred_nid = max_nid; p->numa_migrate_seq =1 ; -migrate_task_to (p, preferred_cpu); +task_numa_migrate (p); } }
@@ -3274,7 +3348,7 @@ static long effective_load(struct task_group *tg, int cpu, long wl, long wg) {struct sched_entity *se = tg->se[cpu]; -if (!tg->parent)/* the trivial, non-cgroup case */ +if (!tg->parent || !wl)/* the trivial, non-cgroup case */ return wl;for_each_sched_entity (se) {
@@ -3327,8 +3401,7 @@ static long effective_load(struct task_group *tg, int cpu, long wl, long wg) }#else -static inline unsigned long effective_load (struct task_group *tg,int cpu, -unsigned long wl,unsigned long wg) +static long effective_load (struct task_group *tg,int cpu,long wl,long wg) {return wl; }
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 32/50 sched: Retry migration of tasks to CPU on a preferred node
In reply to Mel GormanWhen a preferred node is selected for a tasks there is an attempt to migrate the task to a CPU there. This may fail in which case the task will only migrate if the active load balancer takes action. This may never happen if the conditions are not right. This patch will check at NUMA hinting fault time if another attempt should be made to migrate the task. It will only make an attempt once every five seconds.
--- include/linux/sched.h | 1 + kernel/sched/fair.c | 26 +++++++++++++++++++------- 2 files changed, 20 insertions(+), 7 deletions(-)
include/linux/sched.h +1 -0
@@ -1333,6 +1333,7 @@ struct task_struct {int numa_migrate_seq;unsigned int numa_scan_period;unsigned int numa_scan_period_max; +unsigned long numa_migrate_retry;u64 node_stamp;/* migration stamp */ struct callback_head numa_work;
kernel/sched/fair.c +19 -7
@@ -1011,6 +1011,19 @@ migrate:return migrate_task_to (p, env.best_cpu); } +/* Attempt to migrate a task to a CPU on the preferred node. */ +static void numa_migrate_preferred (struct task_struct *p) +{ +/* Success if task is already running on preferred CPU */ + p->numa_migrate_retry =0 ; +if (cpu_to_node (task_cpu (p)) == p->numa_preferred_nid) +return ; + +/* Otherwise, try migrate to a CPU on the preferred node */ +if (task_numa_migrate (p) !=0 ) + p->numa_migrate_retry = jiffies +HZ *5 ; +} +static void task_numa_placement (struct task_struct *p) {int seq, nid, max_nid =-1 ;
@@ -1045,17 +1058,12 @@ static void task_numa_placement(struct task_struct *p) } } -/* -* Record the preferred node as the node with the most faults, -* requeue the task to be running on the idlest CPU on the -* preferred node and reset the scanning rate to recheck -* the working set placement. -*/ +/* Preferred node as the node with the most faults */ if (max_faults && max_nid != p->numa_preferred_nid) {/* Update the preferred nid and migrate task if possible */ p->numa_preferred_nid = max_nid; p->numa_migrate_seq =1 ; -task_numa_migrate (p); +numa_migrate_preferred (p); } }
@@ -1111,6 +1119,10 @@ void task_numa_fault(int last_nidpid, int node, int pages, bool migrated)task_numa_placement (p); +/* Retry task to preferred node migration if it previously failed */ +if (p->numa_migrate_retry &&time_after (jiffies, p->numa_migrate_retry)) +numa_migrate_preferred (p); + p->numa_faults_buffer[task_faults_idx (node, priv)] += pages; }
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 33/50 sched: numa: increment numa_migrate_seq when task runs in correct location
In reply to Mel GormanFrom: Rik van Riel <[email protected]>
When a task is already running on its preferred node, increment numa_migrate_seq to indicate that the task is settled if migration is temporarily disabled, and memory should migrate towards it.
--- kernel/sched/fair.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
kernel/sched/fair.c +9 -1
@@ -1016,8 +1016,16 @@ static void numa_migrate_preferred(struct task_struct *p) {/* Success if task is already running on preferred CPU */ p->numa_migrate_retry =0 ; -if (cpu_to_node (task_cpu (p)) == p->numa_preferred_nid) +if (cpu_to_node (task_cpu (p)) == p->numa_preferred_nid) { +/* +* If migration is temporarily disabled due to a task migration +* then re-enable it now as the task is running on its +* preferred node and memory should migrate locally +*/ +if (!p->numa_migrate_seq) + p->numa_migrate_seq++;return ; + }/* Otherwise, try migrate to a CPU on the preferred node */ if (task_numa_migrate (p) !=0 )
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 34/50 sched: numa: Do not trap hinting faults for shared libraries
In reply to Mel GormanNUMA hinting faults will not migrate a shared executable page mapped by multiple processes on the grounds that the data is probably in the CPU cache already and the page may just bounce between tasks running on multipl nodes. Even if the migration is avoided, there is still the overhead of trapping the fault, updating the statistics, making scheduler placement decisions based on the information etc. If we are never going to migrate the page, it is overhead for no gain and worse a process may be placed on a sub-optimal node for shared executable pages. This patch avoids trapping faults for shared libraries entirely.
--- kernel/sched/fair.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
kernel/sched/fair.c +10 -0
@@ -1227,6 +1227,16 @@ void task_numa_work(struct callback_head *work)if (!vma_migratable (vma))continue ; +/* +* Shared library pages mapped by multiple processes are not +* migrated as it is expected they are cache replicated. Avoid +* hinting faults in read-only file-backed mappings or the vdso +* as migrating the pages will be of marginal benefit. +*/ +if (!vma->vm_mm || + (vma->vm_file && (vma->vm_flags & (VM_READ |VM_WRITE )) == (VM_READ ))) +continue ; + do { start = max(start, vma->vm_start); end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
答复: [PATCH 34/50] sched: numa: Do not trap hinting faults for shared libraries
index fd724bc..5d244d0 100644
kernel/sched/fair.c +10 -0
@@ -1227,6 +1227,16 @@ void task_numa_work(struct callback_head *work)if (!vma_migratable (vma))continue ; +/* +* Shared library pages mapped by multiple processes are not +* migrated as it is expected they are cache replicated. Avoid +* hinting faults in read-only file-backed mappings or the vdso +* as migrating the pages will be of marginal benefit. +*/ +if (!vma->vm_mm || + (vma->vm_file && (vma->vm_flags & (VM_READ |VM_WRITE )) == (VM_READ ))) +continue ; +
=》 May I ask a question, we should consider some VMAs canot be scaned for BalanceNuma? (VM_DONTEXPAND | VM_RESERVED | VM_INSERTPAGE |
VM_NONLINEAR | VM_MIXEDMAP | VM_SAO));
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerRe: ????: [PATCH 34/50] sched: numa: Do not trap hinting faults for shared libraries
Quoted 20 lines · ??????
index fd724bc..5d244d0 100644
kernel/sched/fair.c +10
-0@@ -1227,6 +1227,16 @@ void task_numa_work(struct callback_head *work)if (!vma_migratable (vma))continue ; +/* +* Shared library pages mapped by multiple processes are not +* migrated as it is expected they are cache replicated. Avoid +* hinting faults in read-only file-backed mappings or the vdso +* as migrating the pages will be of marginal benefit. +*/ +if (!vma->vm_mm || + (vma->vm_file && (vma->vm_flags & (VM_READ |VM_WRITE )) == (VM_READ ))) +continue ; +=?? May I ask a question, we should consider some VMAs canot be scaned for BalanceNuma? (
VM_DONTEXPAND|VM_RESERVED|VM_INSERTPAGE|
Quoted 1 line
VM_NONLINEAR | VM_MIXEDMAP | VM_SAO));
vma_migratable check covers most of the other VMAs we do not care about. I do not see the point of checking for some of the VMA flags you mention. Please state which of the additional flags that you think should be checked and why.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: ????: [PATCH 34/50] sched: numa: Do not trap hinting faults for shared libraries
2013/9/17 Mel Gorman <[email protected]>
Quoted 24 lines · ??????
kernel/sched/fair.c void task_numa_work(struct callback_head *work) · line 1231
Quoted 9 lines
index fd724bc..5d244d0 100644
kernel/sched/fair.c +2
-0@@ -1227,6 +1227,16 @@ void task_numa_work(struct callback_head *work)if (!vma_migratable (vma))continue ; +/* +* Shared library pages mapped by multiple processes are not
kernel/sched/fair.c void task_numa_work(struct callback_head *work) · line 1232
Quoted 1 line
kernel/sched/fair.c +1
-0+* migrated as it is expected they are cache replicated. Avoid
kernel/sched/fair.c void task_numa_work(struct callback_head *work) · line 1233
Quoted 1 line
kernel/sched/fair.c +1
-0+* hinting faults in read-only file-backed mappings or the vdso
kernel/sched/fair.c void task_numa_work(struct callback_head *work)
Quoted 4 lines
kernel/sched/fair.c +4
-0+* as migrating the pages will be of marginal benefit. +*/ + if (!vma->vm_mm || + (vma->vm_file && (vma->vm_flags & (VM_READ|VM_WRITE))== (
VM_READ)))Quoted 4 lines
kernel/sched/fair.c +2
-0+continue ; +=?? May I ask a question, we should consider some VMAs canot be scaned
Quoted 9 lines
for BalanceNuma?
Quoted 2 lines
(
VM_DONTEXPAND|VM_RESERVED|VM_INSERTPAGE|VM_NONLINEAR | VM_MIXEDMAP | VM_SAO));
vma_migratablecheck covers most of the other VMAs we do not care about. I do not see the point of checking for some of the VMA flags you mention. Please state which of the additional flags that you think should be checked and why.
=> we should filter out the VMAs of VM_MIXEDMAP, because of it just set pte_mknuma for normal mapping pages in change_pte_range.
Best,
Figo.zhang
Quoted 11 lines
--
Mel Gorman
SUSE Labs--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 35/50 mm: numa: Only trap pmd hinting faults if we would otherwise trap PTE faults
In reply to Mel GormanBase page PMD faulting is meant to batch handle NUMA hinting faults from PTEs. However, even is no PTE faults would ever be handled within a range the kernel still traps PMD hinting faults. This patch avoids the overhead.
--- mm/mprotect.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
mm/mprotect.c +5 -2
@@ -154,6 +154,8 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma, pmd =pmd_offset (pud, addr); do { + unsigned long this_pages; + next = pmd_addr_end(addr, end); if (pmd_trans_huge(*pmd)) { if (next - addr != HPAGE_PMD_SIZE)
@@ -173,8 +175,9 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma, }if (pmd_none_or_clear_bad (pmd))continue ; - pages +=change_pte_range (vma, pmd, addr, next, newprot, + this_pages =change_pte_range (vma, pmd, addr, next, newprot, dirty_accountable, prot_numa, &all_same_nidpid); + pages += this_pages;/* * If we are changing protections for NUMA hinting faults then
@@ -182,7 +185,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,* node. This allows a regular PMD to be handled as one fault * and effectively batches the taking of the PTL */ -if (prot_numa && all_same_nidpid) +if (prot_numa && this_pages && all_same_nidpid)change_pmd_protnuma (vma->vm_mm, addr, pmd); }while (pmd++, addr = next, addr != end);
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 36/50 stop_machine: Introduce stop_two_cpus()
In reply to Mel GormanFrom: Peter Zijlstra <[email protected]>
Introduce stop_two_cpus() in order to allow controlled swapping of two tasks. It repurposes the stop_machine() state machine but only stops the two cpus which we can do with on-stack structures and avoid machine wide synchronization issues.
The ordering of CPUs is important to avoid deadlocks. If unordered then two cpus calling stop_two_cpus on each other simultaneously would attempt to queue in the opposite order on each CPU causing an AB-BA style deadlock. By always having the lowest number CPU doing the queueing of works, we can guarantee that works are always queued in the same order, and deadlocks are avoided.
--- include/linux/stop_machine.h | 1 + kernel/stop_machine.c | 272 +++++++++++++++++++++++++++---------------- 2 files changed, 175 insertions(+), 98 deletions(-)
include/linux/stop_machine.h +1 -0
@@ -28,6 +28,7 @@ struct cpu_stop_work { };int stop_one_cpu (unsigned int cpu,cpu_stop_fn_t fn,void *arg); +int stop_two_cpus (unsigned int cpu1,unsigned int cpu2,cpu_stop_fn_t fn,void *arg);void stop_one_cpu_nowait (unsigned int cpu,cpu_stop_fn_t fn,void *arg,struct cpu_stop_work *work_buf);int stop_cpus (const struct cpumask *cpumask,cpu_stop_fn_t fn,void *arg);
kernel/stop_machine.c +174 -98
318 lines not on this page · raw message
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerPATCH 37/50 sched: Introduce migrate_swap()
In reply to Mel GormanFrom: Peter Zijlstra <[email protected]>
Use the new stop_two_cpus() to implement migrate_swap(), a function that flips two tasks between their respective cpus.
I'm fairly sure there's a less crude way than employing the stop_two_cpus() method, but everything I tried either got horribly fragile and/or complex. So keep it simple for now.
The notable detail is how we 'migrate' tasks that aren't runnable anymore. We'll make it appear like we migrated them before they went to sleep. The sole difference is the previous cpu in the wakeup path, so we override this.
TODO: I'm fairly sure we can get rid of the wake_cpu != -1 test by keeping wake_cpu to the actual task cpu; just couldn't be bothered to think through all the cases.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 103 ++++++++++++++++++++++++++++++++++++++++++++--- kernel/sched/fair.c | 3 +- kernel/sched/idle_task.c | 2 +- kernel/sched/rt.c | 5 +-- kernel/sched/sched.h | 3 +- kernel/sched/stop_task.c | 2 +- 7 files changed, 105 insertions(+), 14 deletions(-)
include/linux/sched.h +1 -0
@@ -1035,6 +1035,7 @@ struct task_struct {#ifdef CONFIG_SMP struct llist_node wake_entry;int on_cpu; +int wake_cpu;#endif int on_rq;
kernel/sched/core.c +98 -5
140 lines not on this page · raw message
kernel/sched/fair.c +1 -2
@@ -3655,11 +3655,10 @@ done:* preempt must be disabled. */ static int -select_task_rq_fair (struct task_struct *p,int sd_flag,int wake_flags) +select_task_rq_fair (struct task_struct *p,int prev_cpu,int sd_flag,int wake_flags) {struct sched_domain *tmp, *affine_sd =NULL , *sd =NULL ;int cpu =smp_processor_id (); -int prev_cpu =task_cpu (p);int new_cpu = cpu;int want_affine =0 ;int sync = wake_flags &WF_SYNC ;
kernel/sched/idle_task.c +1 -1
@@ -9,7 +9,7 @@#ifdef CONFIG_SMP static int -select_task_rq_idle (struct task_struct *p,int sd_flag,int flags) +select_task_rq_idle (struct task_struct *p,int cpu,int sd_flag,int flags) {return task_cpu (p);/* IDLE tasks as never migrated */ }
kernel/sched/rt.c +1 -4
@@ -1169,13 +1169,10 @@ static void yield_task_rt(struct rq *rq)static int find_lowest_rq (struct task_struct *task);static int -select_task_rq_rt (struct task_struct *p,int sd_flag,int flags) +select_task_rq_rt (struct task_struct *p,int cpu,int sd_flag,int flags) {struct task_struct *curr;struct rq *rq; -int cpu; - - cpu =task_cpu (p);if (p->nr_cpus_allowed ==1 ) goto out;
kernel/sched/sched.h +2 -1
@@ -556,6 +556,7 @@ static inline u64 rq_clock_task(struct rq *rq)#ifdef CONFIG_NUMA_BALANCING extern int migrate_task_to (struct task_struct *p,int cpu); +extern int migrate_swap (struct task_struct *,struct task_struct *);static inline void task_numa_free (struct task_struct *p) {kfree (p->numa_faults);
@@ -988,7 +989,7 @@ struct sched_class {void (*put_prev_task) (struct rq *rq,struct task_struct *p);#ifdef CONFIG_SMP -int (*select_task_rq)(struct task_struct *p,int sd_flag,int flags); +int (*select_task_rq)(struct task_struct *p,int task_cpu,int sd_flag,int flags);void (*migrate_task_rq)(struct task_struct *p,int next_cpu);void (*pre_schedule) (struct rq *this_rq,struct task_struct *task);
kernel/sched/stop_task.c +1 -1
@@ -11,7 +11,7 @@#ifdef CONFIG_SMP static int -select_task_rq_stop (struct task_struct *p,int sd_flag,int flags) +select_task_rq_stop (struct task_struct *p,int cpu,int sd_flag,int flags) {return task_cpu (p);/* stop tasks as never migrate */ }
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerPATCH 1/1 hotplug: Optimize {get,put}_online_cpus()
Subject: hotplug: Optimize {get,put}_online_cpus()
From: Peter Zijlstra <[email protected]>
Date: Tue Sep 17 16:17:11 CEST 2013
The cpu hotplug lock is a purely reader biased read-write lock.
The current implementation uses global state, change it so the reader side uses per-cpu state in the uncontended fast-path.
--- include/linux/cpu.h | 33 ++++++++++++++- kernel/cpu.c | 108 ++++++++++++++++++++++++++-------------------------- 2 files changed, 87 insertions(+), 54 deletions(-)
include/linux/cpu.h +31 -2
@@ -16,6 +16,7 @@#include <linux/node.h> #include <linux/compiler.h> #include <linux/cpumask.h> +#include <linux/percpu.h> struct device ;
@@ -175,8 +176,36 @@ extern struct bus_type cpu_subsys;extern void cpu_hotplug_begin (void );extern void cpu_hotplug_done (void ); -extern void get_online_cpus (void ); -extern void put_online_cpus (void ); + +extern struct task_struct *__cpuhp_writer; +DECLARE_PER_CPU(unsigned int, __cpuhp_refcount); + +extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{ +might_sleep (); + +this_cpu_inc (__cpuhp_refcount); +/* +* Order the refcount inc against the writer read; pairs with the full +* barrier in cpu_hotplug_begin(). +*/ +smp_mb (); +if (unlikely (__cpuhp_writer)) +__get_online_cpus (); +} + +extern void __put_online_cpus (void ); + +static inline void put_online_cpus (void ) +{ +barrier (); +this_cpu_dec (__cpuhp_refcount); +if (unlikely (__cpuhp_writer)) +__put_online_cpus (); +} +extern void cpu_hotplug_disable (void );extern void cpu_hotplug_enable (void );#define hotcpu_notifier (fn, pri) cpu_notifier(fn, pri)
kernel/cpu.c +56 -52
144 lines not on this page · raw message
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 45 lines · Peter Zijlstra
Subject: hotplug: Optimize {get,put}_online_cpus()
From: Peter Zijlstra <[email protected]>
Date: Tue Sep 17 16:17:11 CEST 2013The cpu hotplug lock is a purely reader biased read-write lock.
The current implementation uses global state, change it so the reader side uses per-cpu state in the uncontended fast-path.
Cc Oleg Nesterov <[email protected]>Cc Paul McKenney <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Steven Rostedt <[email protected]>Signed-off-by Peter Zijlstra <[email protected]>--- include/linux/cpu.h | 33 ++++++++++++++- kernel/cpu.c | 108 ++++++++++++++++++++++++++-------------------------- 2 files changed, 87 insertions(+), 54 deletions(-)include/linux/cpu.h +11
-2@@ -16,6 +16,7 @@#include <linux/node.h> #include <linux/compiler.h> #include <linux/cpumask.h> +#include <linux/percpu.h> struct device ;@@ -175,8 +176,36 @@ extern struct bus_type cpu_subsys;extern void cpu_hotplug_begin (void );extern void cpu_hotplug_done (void ); -extern void get_online_cpus (void ); -extern void put_online_cpus (void ); + +extern struct task_struct *__cpuhp_writer; +DECLARE_PER_CPU(unsigned int, __cpuhp_refcount); + +extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{ +might_sleep (); +
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 198
Quoted 10 lines
include/linux/cpu.h +10
-0+this_cpu_inc (__cpuhp_refcount); +/* +* Order the refcount inc against the writer read; pairs with the full +* barrier in cpu_hotplug_begin(). +*/ +smp_mb (); +if (unlikely (__cpuhp_writer)) +__get_online_cpus (); +} +
If the problem with get_online_cpus() is the shared global state then a full barrier in the fast path is still going to hurt. Granted, it will hurt a lot less and there should be no lock contention.
However, what barrier in cpu_hotplug_begin is the comment referring to? The other barrier is in the slowpath __get_online_cpus. Did you mean to do a rmb here and a wmb after __cpuhp_writer is set in cpu_hotplug_begin? I'm assuming you are currently using a full barrier to guarantee that an update if cpuhp_writer will be visible so get_online_cpus blocks but I'm not 100% sure because of the comments.
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 203
Quoted 5 lines
include/linux/cpu.h +5
-0+extern void __put_online_cpus (void ); + +static inline void put_online_cpus (void ) +{ +barrier ();
Why is this barrier necessary? I could not find anything that stated if an inline function is an implicit compiler barrier but whether it is or not, it's not clear why it's necessary at all.
Quoted 54 lines
include/linux/cpu.h +5
-0+this_cpu_dec (__cpuhp_refcount); +if (unlikely (__cpuhp_writer)) +__put_online_cpus (); +} +extern void cpu_hotplug_disable (void );extern void cpu_hotplug_enable (void );#define hotcpu_notifier (fn, pri) cpu_notifier(fn, pri)kernel/cpu.c +17
-19@@ -49,88 +49,92 @@ static int cpu_hotplug_disabled;#ifdef CONFIG_HOTPLUG_CPU -static struct { -struct task_struct *active_writer; -struct mutex lock;/* Synchronizes accesses to refcount, */ -/* -* Also blocks the new readers during -* an ongoing cpu hotplug operation. -*/ -int refcount; -} cpu_hotplug = { - .active_writer =NULL , - .lock =__MUTEX_INITIALIZER (cpu_hotplug.lock), - .refcount =0 , -}; +struct task_struct *__cpuhp_writer =NULL ; +EXPORT_SYMBOL_GPL (__cpuhp_writer); + +DEFINE_PER_CPU(unsigned int, __cpuhp_refcount); +EXPORT_PER_CPU_SYMBOL_GPL (__cpuhp_refcount); -void get_online_cpus (void ) +static DECLARE_WAIT_QUEUE_HEAD (cpuhp_wq); + +void __get_online_cpus (void ) { -might_sleep (); -if (cpu_hotplug.active_writer == current) +if (__cpuhp_writer == current)return ; -mutex_lock (&cpu_hotplug.lock); - cpu_hotplug.refcount++; -mutex_unlock (&cpu_hotplug.lock); +again: +/* +* Ensure a pending reading has a 0 refcount. +* +* Without this a new reader that comes in before cpu_hotplug_begin() +* reads the refcount will deadlock. +*/ +this_cpu_dec (__cpuhp_refcount);
kernel/cpu.c static int cpu_hotplug_disabled; · line 82
Quoted 10 lines
kernel/cpu.c +9
-0+wait_event (cpuhp_wq, !__cpuhp_writer); + +this_cpu_inc (__cpuhp_refcount); +/* +* See get_online_cpu(). +*/ +smp_mb (); +if (unlikely (__cpuhp_writer)) + goto again; }
If CPU hotplug operations are very frequent (or a stupid stress test) then it's possible for a new hotplug operation to start (updating __cpuhp_writer) before a caller to __get_online_cpus can update the refcount. Potentially a caller to __get_online_cpus gets starved although as it only affects a CPU hotplug stress test it may not be a serious issue.
Quoted 13 lines
kernel/cpu.c +4
-6-EXPORT_SYMBOL_GPL (get_online_cpus); +EXPORT_SYMBOL_GPL (__get_online_cpus); -void put_online_cpus (void ) +void __put_online_cpus (void ) { -if (cpu_hotplug.active_writer == current) -return ; -mutex_lock (&cpu_hotplug.lock); +unsigned int refcnt =0 ; +int cpu; -if (WARN_ON (!cpu_hotplug.refcount))
kernel/cpu.c static int cpu_hotplug_disabled; · line 95
Quoted 10 lines
kernel/cpu.c +4
-4- cpu_hotplug.refcount++;/* try to fix things up */ +if (__cpuhp_writer == current) +return ; -if (!--cpu_hotplug.refcount &&unlikely (cpu_hotplug.active_writer)) -wake_up_process (cpu_hotplug.active_writer); -mutex_unlock (&cpu_hotplug.lock); +for_each_possible_cpu (cpu) + refcnt +=per_cpu (__cpuhp_refcount, cpu);
This can result in spurious wakeups if CPU N calls get_online_cpus after its refcnt has been checked but I could not think of a case where it matters.
Quoted 39 lines
kernel/cpu.c +7
-19+if (!refcnt) +wake_up_process (__cpuhp_writer); } -EXPORT_SYMBOL_GPL (put_online_cpus); +EXPORT_SYMBOL_GPL (__put_online_cpus);/* * This ensures that the hotplug operation can begin only when the * refcount goes to zero. * -* Note that during a cpu-hotplug operation, the new readers, if any, -* will be blocked by the cpu_hotplug.lock -* * Since cpu_hotplug_begin() is always called after invoking * cpu_maps_update_begin(), we can be sure that only one writer is active. -* -* Note that theoretically, there is a possibility of a livelock: -* - Refcount goes to zero, last reader wakes up the sleeping -* writer. -* - Last reader unlocks the cpu_hotplug.lock. -* - A new reader arrives at this moment, bumps up the refcount. -* - The writer acquires the cpu_hotplug.lock finds the refcount -* non zero and goes to sleep again. -* -* However, this is very difficult to achieve in practice since -* get_online_cpus() not an api which is called all that often. -* */ void cpu_hotplug_begin (void ) { - cpu_hotplug.active_writer = current; + __cpuhp_writer = current;for (;;) { -mutex_lock (&cpu_hotplug.lock); -if (likely (!cpu_hotplug.refcount)) +unsigned int refcnt =0 ; +int cpu; +
kernel/cpu.c static int cpu_hotplug_disabled; · line 125
Quoted 10 lines
kernel/cpu.c +10
-0+/* +* Order the setting of writer against the reading of refcount; +* pairs with the full barrier in get_online_cpus(). +*/ + +set_current_state (TASK_UNINTERRUPTIBLE ); + +for_each_possible_cpu (cpu) + refcnt +=per_cpu (__cpuhp_refcount, cpu); +
CPU 0 CPU 1get_online_cpus
refcnt++
__cpuhp_writer = current refcnt > 0 schedule__get_online_cpus slowpath
refcnt--wait_event(!__cpuhp_writer)
What wakes up __cpuhp_writer to recheck the refcnts and see that they're all 0?
Quoted 19 lines
kernel/cpu.c +5
-4+if (!refcnt)break ; -__set_current_state (TASK_UNINTERRUPTIBLE ); -mutex_unlock (&cpu_hotplug.lock); +schedule (); } +__set_current_state (TASK_RUNNING ); }void cpu_hotplug_done (void ) { - cpu_hotplug.active_writer =NULL ; -mutex_unlock (&cpu_hotplug.lock); + __cpuhp_writer =NULL ; +wake_up_all (&cpuhp_wq); }/*
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 15 lines · Mel Gorman
Quoted 15 lines
include/linux/cpu.h +15
-0+extern struct task_struct *__cpuhp_writer; +DECLARE_PER_CPU(unsigned int, __cpuhp_refcount); + +extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{ +might_sleep (); + +this_cpu_inc (__cpuhp_refcount); +/* +* Order the refcount inc against the writer read; pairs with the full +* barrier in cpu_hotplug_begin(). +*/ +smp_mb ();
Quoted 8 lines
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 198
Quoted 4 lines
include/linux/cpu.h +4
-0+if (unlikely (__cpuhp_writer)) +__get_online_cpus (); +} +If the problem with
get_online_cpus()is the shared global state then a full barrier in the fast path is still going to hurt. Granted, it will hurt a lot less and there should be no lock contention.
I went for a lot less, I wasn't smart enough to get rid of it. Also, since its a lock op we should at least provide an ACQUIRE barrier.
Quoted 1 line
However, what barrier in
cpu_hotplug_beginis the comment referring to?
set_current_state() implies a full barrier and nicely separates the write to __cpuhp_writer and the read of __cpuph_refcount.
Quoted 3 lines
The
other barrier is in the slowpath__get_online_cpus. Did you mean to do a rmb here and a wmb after__cpuhp_writeris set incpu_hotplug_begin?
No, since we're ordering LOADs and STORES (see below) we must use full barriers.
Quoted 3 lines
I'm assuming you are currently using a full barrier to guarantee that an update if
cpuhp_writerwill be visible soget_online_cpusblocks but I'm not 100% sure because of the comments.
I'm ordering:
CPU0 -- get_online_cpus() CPU1 -- cpu_hotplug_begin()
STORE __cpuhp_refcount STORE __cpuhp_writer
MB MB
LOAD __cpuhp_writer LOAD __cpuhp_refcount
Such that neither can miss the state of the other and we get proper mutual exclusion.
Quoted 7 lines
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 203
Quoted 5 lines
include/linux/cpu.h +5
-0+extern void __put_online_cpus (void ); + +static inline void put_online_cpus (void ) +{ +barrier ();Why is this barrier necessary?
To ensure the compiler keeps all loads/stores done before the read-unlock before it.
Arguably it should be a complete RELEASE barrier. I should've put an XXX comment here but the brain gave out completely for the day.
Quoted 3 lines
I could not find anything that stated if an
inline function is an implicit compiler barrier but whether it is or not, it's not clear why it's necessary at all.
It is not, only actual function calls are an implied sync point for the compiler.
Quoted 35 lines
Quoted 35 lines
kernel/cpu.c +30
-0+this_cpu_dec (__cpuhp_refcount); +if (unlikely (__cpuhp_writer)) +__put_online_cpus (); +} + +struct task_struct *__cpuhp_writer =NULL ; +EXPORT_SYMBOL_GPL (__cpuhp_writer); + +DEFINE_PER_CPU(unsigned int, __cpuhp_refcount); +EXPORT_PER_CPU_SYMBOL_GPL (__cpuhp_refcount); +static DECLARE_WAIT_QUEUE_HEAD (cpuhp_wq); + +void __get_online_cpus (void ) { +if (__cpuhp_writer == current)return ; +again: +/* +* Ensure a pending reading has a 0 refcount. +* +* Without this a new reader that comes in before cpu_hotplug_begin() +* reads the refcount will deadlock. +*/ +this_cpu_dec (__cpuhp_refcount); +wait_event (cpuhp_wq, !__cpuhp_writer); + +this_cpu_inc (__cpuhp_refcount); +/* +* See get_online_cpu(). +*/ +smp_mb (); +if (unlikely (__cpuhp_writer))
Quoted 8 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 82
Quoted 2 lines
kernel/cpu.c +1
-0+ goto again; }If CPU hotplug operations are very frequent (or a stupid stress test) then it's possible for a new hotplug operation to start (updating
__cpuhp_writer) before a caller to__get_online_cpuscan update the refcount. Potentially a caller to__get_online_cpusgets starved although as it only affects a CPU hotplug stress test it may not be a serious issue.
Right.. If that ever becomes a problem we should fix it, but aside from stress tests hotplug should be extremely rare.
Initially I kept the reference over the wait_event() but realized (as per the comment) that that would deadlock cpu_hotplug_begin() for it would never observe !refcount.
One solution for this problem is having refcount as an array of 2 and flipping the index at the appropriate times.
Quoted 10 lines
Quoted 10 lines
kernel/cpu.c +6
-0+EXPORT_SYMBOL_GPL (__get_online_cpus); +void __put_online_cpus (void ) { +unsigned int refcnt =0 ; +int cpu; +if (__cpuhp_writer == current) +return ;
Quoted 7 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+ for_each_possible_cpu(cpu) + refcnt += per_cpu(__cpuhp_refcount, cpu);This can result in spurious wakeups if CPU N calls
get_online_cpusafter its refcnt has been checked but I could not think of a case where it matters.
Right and right.. too many wakeups aren't a correctness issue. One should try and minimize them for performance reasons though :-)
Quoted 41 lines
Quoted 20 lines
kernel/cpu.c +6
-0+if (!refcnt) +wake_up_process (__cpuhp_writer); }/* * This ensures that the hotplug operation can begin only when the * refcount goes to zero. * * Since cpu_hotplug_begin() is always called after invoking * cpu_maps_update_begin(), we can be sure that only one writer is active. */ void cpu_hotplug_begin (void ) { + __cpuhp_writer = current;for (;;) { +unsigned int refcnt =0 ; +int cpu; +kernel/cpu.c static int cpu_hotplug_disabled; · line 125
Quoted 10 lines
kernel/cpu.c +10
-0+/* +* Order the setting of writer against the reading of refcount; +* pairs with the full barrier in get_online_cpus(). +*/ + +set_current_state (TASK_UNINTERRUPTIBLE ); + +for_each_possible_cpu (cpu) + refcnt +=per_cpu (__cpuhp_refcount, cpu); +CPU 0 CPU 1
get_online_cpus
refcnt++
__cpuhp_writer= current refcnt > 0 schedule__get_online_cpusslowpath
refcnt--wait_event(!__cpuhp_writer)
Quoted 2 lines
What wakes up
__cpuhp_writerto recheck the refcnts and see that they're all 0?
The wakeup in __put_online_cpus() you just commented on? put_online_cpus() will drop into the slow path __put_online_cpus() if there's a writer and compute the refcount and perform the wakeup when !refcount.
Quoted 7 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 132
Quoted 7 lines
kernel/cpu.c +3
-0+if (!refcnt)break ; +schedule (); } +__set_current_state (TASK_RUNNING ); }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
New version, now with excessive comments.
I found a deadlock (where both reader and writer would go to sleep); identified below as case 1b.
The implementation without patch is reader biased, this implementation, as Mel pointed out, is writer biased. I should try and fix this but I'm stepping away from the computer now as I have the feeling I'll only wreck stuff from now on.
---
Subject: hotplug: Optimize {get,put}_online_cpus()
From: Peter Zijlstra <[email protected]>
Date: Tue Sep 17 16:17:11 CEST 2013
The current implementation uses global state, change it so the reader side uses per-cpu state in the contended fast path.
--- include/linux/cpu.h | 29 ++++++++- kernel/cpu.c | 159 ++++++++++++++++++++++++++++++++++------------------ 2 files changed, 134 insertions(+), 54 deletions(-)
include/linux/cpu.h +27 -2
@@ -16,6 +16,7 @@#include <linux/node.h> #include <linux/compiler.h> #include <linux/cpumask.h> +#include <linux/percpu.h> struct device ;
@@ -175,8 +176,32 @@ extern struct bus_type cpu_subsys;extern void cpu_hotplug_begin (void );extern void cpu_hotplug_done (void ); -extern void get_online_cpus (void ); -extern void put_online_cpus (void ); + +extern struct task_struct *__cpuhp_writer; +DECLARE_PER_CPU(unsigned int, __cpuhp_refcount); + +extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{ +might_sleep (); + +this_cpu_inc (__cpuhp_refcount); +smp_mb ();/* see comment near __get_online_cpus() */ +if (unlikely (__cpuhp_writer)) +__get_online_cpus (); +} + +extern void __put_online_cpus (void ); + +static inline void put_online_cpus (void ) +{ +this_cpu_dec (__cpuhp_refcount); +smp_mb ();/* see comment near __get_online_cpus() */ +if (unlikely (__cpuhp_writer)) +__put_online_cpus (); +} +extern void cpu_hotplug_disable (void );extern void cpu_hotplug_enable (void );#define hotcpu_notifier (fn, pri) cpu_notifier(fn, pri)
kernel/cpu.c +107 -52
195 lines not on this page · raw message
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Meh, I should stop poking at this..
This one lost all the comments again :/
It uses preempt_disable/preempt_enable vs synchronize_sched() to remove the barriers from the fast path.
After that it waits for !refcount before setting state, which stops new readers.
I used a per-cpu spinlock to keep the state check and refcount inc atomic vs the setting of state.
So the slow path is still per-cpu and mostly uncontended even in the pending writer case.
After setting state it again waits for !refcount -- someone could have sneaked in between the last !refcount and setting state. But this time we know refcount will stay 0.
The only thing I don't really like is the unconditional writer wake in the read-unlock slowpath, but I couldn't come up with anything better. Here at least we guarantee that there is a wakeup after the last dec -- although there might be far too many wakes.
---
Subject: hotplug: Optimize {get,put}_online_cpus()
From: Peter Zijlstra <[email protected]>
Date: Tue Sep 17 16:17:11 CEST 2013
--- include/linux/cpu.h | 32 ++++++++++- kernel/cpu.c | 151 +++++++++++++++++++++++++++++----------------------- 2 files changed, 116 insertions(+), 67 deletions(-)
include/linux/cpu.h +30 -2
@@ -16,6 +16,7 @@#include <linux/node.h> #include <linux/compiler.h> #include <linux/cpumask.h> +#include <linux/percpu.h> struct device ;
@@ -175,8 +176,35 @@ extern struct bus_type cpu_subsys;extern void cpu_hotplug_begin (void );extern void cpu_hotplug_done (void ); -extern void get_online_cpus (void ); -extern void put_online_cpus (void ); + +extern struct task_struct *__cpuhp_writer; +DECLARE_PER_CPU(unsigned int, __cpuhp_refcount); + +extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{ +might_sleep (); + +preempt_disable (); +if (likely (!__cpuhp_writer || __cpuhp_writer == current)) +this_cpu_inc (__cpuhp_refcount); +else +__get_online_cpus (); +preempt_enable (); +} + +extern void __put_online_cpus (void ); + +static inline void put_online_cpus (void ) +{ +preempt_disable (); +this_cpu_dec (__cpuhp_refcount); +if (unlikely (__cpuhp_writer && __cpuhp_writer != current)) +__put_online_cpus (); +preempt_enable (); +} +extern void cpu_hotplug_disable (void );extern void cpu_hotplug_enable (void );#define hotcpu_notifier (fn, pri) cpu_notifier(fn, pri)
kernel/cpu.c +86 -65
174 lines not on this page · raw message
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Sorry for delay, I was sick...
Quoted 3 lines · Peter Zijlstra
I used a per-cpu spinlock to keep the state check and refcount inc atomic vs the setting of state.
I think this could be simpler, see below.
Quoted 2 lines
So the slow path is still per-cpu and mostly uncontended even in the pending writer case.
Is it really important? I mean, per-cpu/uncontended even if the writer is pending?
Otherwise we could do
static DEFINE_PER_CPU(long, cpuhp_fast_ctr);
static struct task_struct *cpuhp_writer;
static DEFINE_MUTEX(cpuhp_slow_lock)
static long cpuhp_slow_ctr;
static bool update_fast_ctr(int inc)
{
bool success = true;
preempt_disable();
if (likely(!cpuhp_writer))
__get_cpu_var(cpuhp_fast_ctr) += inc;
else if (cpuhp_writer != current)
success = false;
preempt_enable();
return success;
}
void get_online_cpus(void)
{
if (likely(update_fast_ctr(+1));
return;
mutex_lock(&cpuhp_slow_lock);
cpuhp_slow_ctr++;
mutex_unlock(&cpuhp_slow_lock);
}
void put_online_cpus(void)
{
if (likely(update_fast_ctr(-1));
return;
mutex_lock(&cpuhp_slow_lock);
if (!--cpuhp_slow_ctr && cpuhp_writer)
wake_up_process(cpuhp_writer);
mutex_unlock(&cpuhp_slow_lock);
}
static void clear_fast_ctr(void)
{
long total = 0;
int cpu;
for_each_possible_cpu(cpu) {
total += per_cpu(cpuhp_fast_ctr, cpu);
per_cpu(cpuhp_fast_ctr, cpu) = 0;
}
return total;
}
static void cpu_hotplug_begin(void)
{
cpuhp_writer = current;
synchronize_sched();
/* Nobody except us can use can use cpuhp_fast_ctr */
mutex_lock(&cpuhp_slow_lock);
cpuhp_slow_ctr += clear_fast_ctr();
while (cpuhp_slow_ctr) {
__set_current_state(TASK_UNINTERRUPTIBLE);
mutex_unlock(&&cpuhp_slow_lock);
schedule();
mutex_lock(&cpuhp_slow_lock);
}
}
static void cpu_hotplug_done(void)
{
cpuhp_writer = NULL;
mutex_unlock(&cpuhp_slow_lock);
}I already sent this code in 2010, it needs some trivial updates.
But. We already have percpu_rw_semaphore, can't we reuse it? In fact I thought about this from the very beginning. Just we need percpu_down_write_recursive_readers() which does
bool xxx(brw)
{
if (down_trylock(&brw->rw_sem))
return false;
if (!atomic_read(&brw->slow_read_ctr))
return true;
up_write(&brw->rw_sem);
return false;
}
ait_event(brw->write_waitq, xxx(brw));instead of down_write() + wait_event(!atomic_read(&brw->slow_read_ctr)).
The only problem is the lockdep annotations in percpu_down_read(), but this looks simple, just we need down_read_no_lockdep() (like __up_read).
Note also that percpu_down_write/percpu_up_write can be improved wrt synchronize_sched(). We can turn the 2nd one into call_rcu(), and the 1nd one can be avoided if another percpu_down_write() comes "soon after" percpu_down_up().
As for the patch itself, I am not sure.
Quoted 8 lines
kernel/cpu.c +8
-0+static void cpuph_wait_refcount (void ) +{ +for (;;) { +unsigned int refcnt =0 ; +int cpu; + +set_current_state (TASK_UNINTERRUPTIBLE ); +
kernel/cpu.c static int cpu_hotplug_disabled; · line 106
Quoted 10 lines
kernel/cpu.c +10
-0+for_each_possible_cpu (cpu) + refcnt +=per_cpu (__cpuhp_refcount, cpu); + +if (!refcnt) +break ; + +schedule (); + } +__set_current_state (TASK_RUNNING ); +}
It seems, this can succeed while it should not, see below.
Quoted 7 lines
void
cpu_hotplug_begin(void)
{kernel/cpu.c +1
-3- cpu_hotplug.active_writer = current; +lockdep_assert_held (&cpu_add_remove_lock); -for (;;) { -mutex_lock (&cpu_hotplug.lock);
kernel/cpu.c static int cpu_hotplug_disabled; · line 135
Quoted 10 lines
kernel/cpu.c +4
-6-if (likely (!cpu_hotplug.refcount)) -break ; -__set_current_state (TASK_UNINTERRUPTIBLE ); -mutex_unlock (&cpu_hotplug.lock); -schedule (); - } + __cpuhp_writer = current; + +/* After this everybody will observe _writer and take the slow path. */ +synchronize_sched ();
Yes, the reader should see _writer, but:
Review
Quoted 2 lines
(quoted diff) +2
-0+ /* Wait for no readers -- reader preference */ + cpuhp_wait_refcount();
but how we can ensure the writer sees the results of the reader's updates?
Suppose that we have 2 CPU's, __cpuhp_refcount[0] = 0, __cpuhp_refcount[1] = 1. IOW, we have a single R reader which takes this lock on CPU_1 and sleeps.
Now,
- The writer calls cpuph_wait_refcount()
- cpuph_wait_refcount() does refcnt += __cpuhp_refcount[0]. refcnt == 0.
- another reader comes on CPU_0, increments __cpuhp_refcount[0].
- this reader migrates to CPU_1 and does put_online_cpus(), this decrements __cpuhp_refcount[1] which becomes zero.
- cpuph_wait_refcount() continues and reads __cpuhp_refcount[1] which is zero. refcnt == 0, return.
- The writer does cpuhp_set_state(1).
- The reader R (original reader) wakes up, calls get_online_cpus() recursively, and sleeps in wait_event(!__cpuhp_writer).
Btw, I think that __sb_start_write/etc is equally wrong. Perhaps it is another potential user of percpu_rw_sem.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 2 lines · Oleg Nesterov
As for the patch itself, I am not sure.
Forgot to mention... and with this patch cpu_hotplug_done() loses the "release" semantics, not sure this is fine.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Oleg NesterovQuoted 5 lines · Oleg Nesterov
Quoted 2 lines
So the slow path is still per-cpu and mostly uncontended even in the pending writer case.
Is it really important? I mean, per-cpu/uncontended even if the writer is pending?
I think so, once we make {get,put}_online_cpus() really cheap they'll get in more and more places, and the global count with pending writer will make things crawl on bigger machines.
Quoted 1 line
Otherwise we could do
<snip>
Quoted 1 line
I already sent this code in 2010, it needs some trivial updates.
Yeah, I found that a few days ago.. but per the above I didn't like the pending writer case.
Quoted 1 line
But. We already have
percpu_rw_semaphore,
Oh urgh, forgot about that one. /me goes read.
/me curses loudly.. that thing has an _expedited() call in it, those should die.
Also, it suffers the same problem. I think esp. for hotplug we should be 100% geared towards readers and pretty much damn writers.
I'd dread to think what would happen if a 4k cpu machine were to land in the slow path on that global mutex. Readers would never go-away and progress would make a glacier seem fast.
Quoted 4 lines
Note also that percpu_down_write/percpu_up_write can be improved wrt
synchronize_sched(). We can turn the 2nd one intocall_rcu(), and the 1nd one can be avoided if anotherpercpu_down_write()comes "soon after"percpu_down_up().
Write side be damned ;-)
It is anyway with a pure read bias and a large machine..
Quoted 37 lines
As for the patch itself, I am not sure.
Quoted 1 line
+static void
cpuph_wait_refcount(void)It seems, this can succeed while it should not, see below.
kernel/cpu.c static int cpu_hotplug_disabled; · line 135
Quoted 8 lines
void
cpu_hotplug_begin(void)
{kernel/cpu.c +5
-0+lockdep_assert_held (&cpu_add_remove_lock); + __cpuhp_writer = current; + +/* After this everybody will observe _writer and take the slow path. */ +synchronize_sched ();Yes, the reader should see
_writer, but:Review
Quoted 2 lines
(quoted diff) +2
-0+ /* Wait for no readers -- reader preference */ + cpuhp_wait_refcount();but how we can ensure the writer sees the results of the reader's updates?
Suppose that we have 2 CPU's, __cpuhp_refcount[0] = 0, __cpuhp_refcount[1] = 1. IOW, we have a single R reader which takes this lock on
CPU_1and sleeps.Now,
- The writer calls
cpuph_wait_refcount()-
cpuph_wait_refcount()does refcnt += __cpuhp_refcount[0]. refcnt == 0.- another reader comes on
CPU_0, increments __cpuhp_refcount[0].- this reader migrates to
CPU_1and doesput_online_cpus(), this decrements __cpuhp_refcount[1] which becomes zero.
Quoted 7 lines
-
cpuph_wait_refcount()continues and reads __cpuhp_refcount[1] which is zero. refcnt == 0, return.- The writer does
cpuhp_set_state(1).- The reader R (original reader) wakes up, calls
get_online_cpus()recursively, and sleeps inwait_event(!__cpuhp_writer).
Ah indeed..
The best I can come up with is something like:
static unsigned int cpuhp_refcount(void)
{
unsigned int refcount = 0;
int cpu;
for_each_possible_cpu(cpu)
refcount += per_cpu(__cpuhp_refcount, cpu);
}static void cpuhp_wait_refcount(void)
{
for (;;) {
unsigned int rc1, rc2;
rc1 = cpuhp_refcount();
set_current_state(TASK_UNINTERRUPTIBLE); /* MB */
rc2 = cpuhp_refcount();
if (rc1 == rc2 && !rc1)
break;
schedule();
}
__set_current_state(TASK_RUNNING);
}--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 11 lines · Peter Zijlstra
Quoted 5 lines · Oleg Nesterov
Quoted 2 lines
So the slow path is still per-cpu and mostly uncontended even in the pending writer case.
Is it really important? I mean, per-cpu/uncontended even if the writer is pending?
I think so, once we make {get,put}_online_cpus() really cheap they'll get in more and more places, and the global count with pending writer will make things crawl on bigger machines.
Hmm. But the writers should be rare.
Quoted 6 lines
Quoted 1 line
But. We already have
percpu_rw_semaphore,Oh urgh, forgot about that one. /me goes read.
/me curses loudly.. that thing has an
_expedited()call in it, those should die.
Probably yes, the original reason for _expedited() has gone away.
Quoted 3 lines
I'd dread to think what would happen if a 4k cpu machine were to land in the slow path on that global mutex. Readers would never go-away and progress would make a glacier seem fast.
Another problem is that write-lock can never succeed unless it prevents the new readers, but this needs the per-task counter.
Quoted 6 lines
Quoted 4 lines
Note also that percpu_down_write/percpu_up_write can be improved wrt
synchronize_sched(). We can turn the 2nd one intocall_rcu(), and the 1nd one can be avoided if anotherpercpu_down_write()comes "soon after"percpu_down_up().Write side be damned ;-)
Suppose that a 4k cpu machine does disable_nonboot_cpus(), every _cpu_down() does synchronize_sched()... OK, perhaps the locking can be changed so that cpu_hotplug_begin/end is called only once in this case.
Quoted 27 lines
Quoted 5 lines
- The writer calls
cpuph_wait_refcount()-
cpuph_wait_refcount()does refcnt += __cpuhp_refcount[0]. refcnt == 0.Quoted 7 lines
- another reader comes on
CPU_0, increments __cpuhp_refcount[0].- this reader migrates to
CPU_1and doesput_online_cpus(), this decrements __cpuhp_refcount[1] which becomes zero.-
cpuph_wait_refcount()continues and reads __cpuhp_refcount[1] which is zero. refcnt == 0, return.Ah indeed..
The best I can come up with is something like:
static unsigned int
cpuhp_refcount(void){ unsigned int refcount = 0; int cpu; for_each_possible_cpu(cpu) refcount += per_cpu(__cpuhp_refcount, cpu); }static void
cpuhp_wait_refcount(void)
Quoted 9 lines
{
for (;;) {
unsigned int rc1, rc2;
rc1 = cpuhp_refcount();
set_current_state(TASK_UNINTERRUPTIBLE); /* MB */
rc2 = cpuhp_refcount();
if (rc1 == rc2 && !rc1)But this only makes the race above "theoretical ** 2". Both cpuhp_refcount()'s can be equally fooled.
Looks like, cpuhp_refcount() should take all per-cpu cpuhp_lock's before it reads __cpuhp_refcount.
Oleg.
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 9 lines · Oleg Nesterov
Quoted 9 lines
static void
cpuhp_wait_refcount(void){ for (;;) { unsigned int rc1, rc2; rc1 = cpuhp_refcount(); set_current_state(TASK_UNINTERRUPTIBLE); /* MB */ rc2 = cpuhp_refcount();
Quoted 7 lines
Quoted 1 line
if (rc1 == rc2 && !rc1)But this only makes the race above "theoretical ** 2". Both cpuhp_refcount()'s can be equally fooled.
Looks like,
cpuhp_refcount()should take all per-cpu cpuhp_lock's before it reads__cpuhp_refcount.
Ah, so SRCU has a solution for this using a sequence count.
So now we drop from a no memory barriers fast path, into a memory barrier 'slow' path into blocking.
Only once we block do we hit global state..
include/linux/cpu.h +45 -2
@@ -16,6 +16,7 @@#include <linux/node.h> #include <linux/compiler.h> #include <linux/cpumask.h> +#include <linux/percpu.h> struct device ;
@@ -173,10 +174,50 @@ extern struct bus_type cpu_subsys;#ifdef CONFIG_HOTPLUG_CPU /* Stop CPUs going up and down. */ +extern void cpu_hotplug_init_task (struct task_struct *p); +extern void cpu_hotplug_begin (void );extern void cpu_hotplug_done (void ); -extern void get_online_cpus (void ); -extern void put_online_cpus (void ); + +extern int __cpuhp_writer; +DECLARE_PER_CPU(unsigned int, __cpuhp_refcount); + +extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{ +might_sleep (); + +/* Support reader-in-reader recursion */ +if (current->cpuhp_ref++) { +barrier (); +return ; + } + +preempt_disable (); +if (likely (!__cpuhp_writer)) +__this_cpu_inc (__cpuhp_refcount); +else +__get_online_cpus (); +preempt_enable (); +} + +extern void __put_online_cpus (void ); + +static inline void put_online_cpus (void ) +{ +barrier (); +if (--current->cpuhp_ref) +return ; + +preempt_disable (); +if (likely (!__cpuhp_writer)) +__this_cpu_dec (__cpuhp_refcount); +else +__put_online_cpus (); +preempt_enable (); +} +extern void cpu_hotplug_disable (void );extern void cpu_hotplug_enable (void );#define hotcpu_notifier (fn, pri) cpu_notifier(fn, pri)
@@ -200,6 +241,8 @@ static inline void cpu_hotplug_driver_un#else /* CONFIG_HOTPLUG_CPU */ +static inline void cpu_hotplug_init_task (struct task_struct *p) {} +static inline void cpu_hotplug_begin (void ) {}static inline void cpu_hotplug_done (void ) {}#define get_online_cpus () do { } while (0)
include/linux/sched.h +3 -0
@@ -1454,6 +1454,9 @@ struct task_struct {unsigned int sequential_io;unsigned int sequential_io_avg;#endif +#ifdef CONFIG_HOTPLUG_CPU +int cpuhp_ref; +#endif };/* Future-safe accessor for struct task_struct's cpus_allowed. */
kernel/cpu.c +115 -55
203 lines not on this page · raw message
kernel/sched/core.c +2 -0
@@ -1736,6 +1736,8 @@ static void __sched_fork(unsigned long cINIT_LIST_HEAD (&p->numa_entry); p->numa_group =NULL ;#endif /* CONFIG_NUMA_BALANCING */ + +cpu_hotplug_init_task (p); }#ifdef CONFIG_NUMA_BALANCING
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
kernel/cpu.c static int cpu_hotplug_disabled; · line 70
Quoted 3 lines · Peter Zijlstra
kernel/cpu.c +3
-0+void __get_online_cpus (void ) +{ +if (__cpuhp_writer ==1 ) {
take_ref:
Quoted 12 lines
kernel/cpu.c +11
-0+/* See __srcu_read_lock() */ +__this_cpu_inc (__cpuhp_refcount); +smp_mb (); +__this_cpu_inc (cpuhp_seq); +return ; + } + +atomic_inc (&cpuhp_waitcount); +/* +* We either call schedule() in the wait, or we'll fall through +* and reschedule on the preempt_enable() in get_online_cpus().
kernel/cpu.c static int cpu_hotplug_disabled; · line 92
Quoted 10 lines
kernel/cpu.c +8
-0*/ +preempt_enable_no_resched (); +wait_event (cpuhp_readers, !__cpuhp_writer); +preempt_disable (); +/* +* XXX list_empty_careful(&cpuhp_readers.task_list) ? +*/ +if (atomic_dec_and_test (&cpuhp_waitcount)) +wake_up_all (&cpuhp_writer);
goto take_ref;
Review
Quoted 2 lines
(quoted diff) +2
-0+} +EXPORT_SYMBOL_GPL(__get_online_cpus);
It would probably be a good idea to increment __cpuhp_refcount after the wait_event.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Peter ZijlstraQuoted 3 lines · Peter Zijlstra
So now we drop from a no memory barriers fast path, into a memory barrier 'slow' path into blocking.
Cough... can't understand the above ;) In fact I can't understand the patch... see below. But in any case, afaics the fast path needs mb() unless you add another synchronize_sched() into cpu_hotplug_done().
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 199
Quoted 13 lines
include/linux/cpu.h +13
-0+static inline void get_online_cpus (void ) +{ +might_sleep (); + +/* Support reader-in-reader recursion */ +if (current->cpuhp_ref++) { +barrier (); +return ; + } + +preempt_disable (); +if (likely (!__cpuhp_writer)) +__this_cpu_inc (__cpuhp_refcount);
mb() to ensure the reader can't miss, say, a STORE done inside the cpu_hotplug_begin/end section.
put_online_cpus() needs mb() as well.
kernel/cpu.c static int cpu_hotplug_disabled; · line 76
Quoted 9 lines
kernel/cpu.c +9
-0+void __get_online_cpus (void ) +{ +if (__cpuhp_writer ==1 ) { +/* See __srcu_read_lock() */ +__this_cpu_inc (__cpuhp_refcount); +smp_mb (); +__this_cpu_inc (cpuhp_seq); +return ; + }
OK, cpuhp_seq should guarantee cpuhp_readers_active_check() gets the "stable" numbers. Looks suspicious... but lets assume this works.
However, I do not see how "__cpuhp_writer == 1" can work, please see below.
kernel/cpu.c static int cpu_hotplug_disabled; · line 92
Quoted 5 lines
kernel/cpu.c +5
-0+/* +* XXX list_empty_careful(&cpuhp_readers.task_list) ? +*/ +if (atomic_dec_and_test (&cpuhp_waitcount)) +wake_up_all (&cpuhp_writer);
Same problem as in previous version. __get_online_cpus() succeeds without incrementing __cpuhp_refcount. "goto start" can't help afaics.
Quoted 14 lines
void
cpu_hotplug_begin(void)
{kernel/cpu.c +2
-9- cpu_hotplug.active_writer = current; +unsigned int count =0 ; +int cpu; -for (;;) { -mutex_lock (&cpu_hotplug.lock); -if (likely (!cpu_hotplug.refcount)) -break ; -__set_current_state (TASK_UNINTERRUPTIBLE ); -mutex_unlock (&cpu_hotplug.lock); -schedule (); - }
kernel/cpu.c static int cpu_hotplug_disabled; · line 167
Quoted 10 lines
kernel/cpu.c +10
-0+lockdep_assert_held (&cpu_add_remove_lock); + +/* allow reader-in-writer recursion */ + current->cpuhp_ref++; + +/* make readers take the slow path */ + __cpuhp_writer =1 ; + +/* See percpu_down_write() */ +synchronize_sched ();
Suppose there are no readers at this point,
kernel/cpu.c static int cpu_hotplug_disabled; · line 173
Quoted 6 lines
kernel/cpu.c +6
-0+ +/* make readers block */ + __cpuhp_writer =2 ; + +/* Wait for all readers to go away */ +wait_event (cpuhp_writer,cpuhp_readers_active_check ());
So wait_event() "quickly" returns.
Now. Why the new reader should see __cpuhp_writer = 2 ? It can still see it == 1, and take that "if (__cpuhp_writer == 1)" path above.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 9 lines · Oleg Nesterov
Quoted 3 lines · Peter Zijlstra
So now we drop from a no memory barriers fast path, into a memory barrier 'slow' path into blocking.
Cough... can't understand the above ;) In fact I can't understand the patch... see below. But in any case, afaics the fast path needs
mb()unless you add anothersynchronize_sched()intocpu_hotplug_done().
For whatever it is worth, I too don't see how it works without read-side memory barriers.
Thanx, Paul
Quoted 88 lines
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 199
Quoted 13 lines
include/linux/cpu.h +13
-0+static inline void get_online_cpus (void ) +{ +might_sleep (); + +/* Support reader-in-reader recursion */ +if (current->cpuhp_ref++) { +barrier (); +return ; + } + +preempt_disable (); +if (likely (!__cpuhp_writer)) +__this_cpu_inc (__cpuhp_refcount);
mb()to ensure the reader can't miss, say, a STORE done inside the cpu_hotplug_begin/end section.
put_online_cpus()needsmb()as well.kernel/cpu.c static int cpu_hotplug_disabled; · line 76
Quoted 9 lines
kernel/cpu.c +9
-0+void __get_online_cpus (void ) +{ +if (__cpuhp_writer ==1 ) { +/* See __srcu_read_lock() */ +__this_cpu_inc (__cpuhp_refcount); +smp_mb (); +__this_cpu_inc (cpuhp_seq); +return ; + }OK,
cpuhp_seqshould guaranteecpuhp_readers_active_check()gets the "stable" numbers. Looks suspicious... but lets assume this works.However, I do not see how "
__cpuhp_writer== 1" can work, please see below.kernel/cpu.c static int cpu_hotplug_disabled; · line 92
Quoted 5 lines
kernel/cpu.c +5
-0+/* +* XXX list_empty_careful(&cpuhp_readers.task_list) ? +*/ +if (atomic_dec_and_test (&cpuhp_waitcount)) +wake_up_all (&cpuhp_writer);Same problem as in previous version.
__get_online_cpus()succeeds without incrementing__cpuhp_refcount. "goto start" can't help afaics.Quoted 14 lines
void
cpu_hotplug_begin(void)
{kernel/cpu.c +2
-9- cpu_hotplug.active_writer = current; +unsigned int count =0 ; +int cpu; -for (;;) { -mutex_lock (&cpu_hotplug.lock); -if (likely (!cpu_hotplug.refcount)) -break ; -__set_current_state (TASK_UNINTERRUPTIBLE ); -mutex_unlock (&cpu_hotplug.lock); -schedule (); - }kernel/cpu.c static int cpu_hotplug_disabled; · line 167
Quoted 10 lines
kernel/cpu.c +10
-0+lockdep_assert_held (&cpu_add_remove_lock); + +/* allow reader-in-writer recursion */ + current->cpuhp_ref++; + +/* make readers take the slow path */ + __cpuhp_writer =1 ; + +/* See percpu_down_write() */ +synchronize_sched ();Suppose there are no readers at this point,
kernel/cpu.c static int cpu_hotplug_disabled; · line 173
Quoted 6 lines
kernel/cpu.c +6
-0+ +/* make readers block */ + __cpuhp_writer =2 ; + +/* Wait for all readers to go away */ +wait_event (cpuhp_writer,cpuhp_readers_active_check ());So
wait_event()"quickly" returns.Now. Why the new reader should see
__cpuhp_writer= 2 ? It can still see it == 1, and take that "if (__cpuhp_writer== 1)" path above.Oleg.
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Oleg NesterovQuoted 9 lines · Oleg Nesterov
Quoted 3 lines · Peter Zijlstra
So now we drop from a no memory barriers fast path, into a memory barrier 'slow' path into blocking.
Cough... can't understand the above ;) In fact I can't understand the patch... see below. But in any case, afaics the fast path needs
mb()unless you add anothersynchronize_sched()intocpu_hotplug_done().
Sure we can add more ;-) But I went with perpcu_up_write(), it too does the sync_sched() before clearing the fast path state.
Quoted 17 lines
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 199
Quoted 13 lines
include/linux/cpu.h +13
-0+static inline void get_online_cpus (void ) +{ +might_sleep (); + +/* Support reader-in-reader recursion */ +if (current->cpuhp_ref++) { +barrier (); +return ; + } + +preempt_disable (); +if (likely (!__cpuhp_writer)) +__this_cpu_inc (__cpuhp_refcount);
mb()to ensure the reader can't miss, say, a STORE done inside the cpu_hotplug_begin/end section.
Quoted 1 line
put_online_cpus()needsmb()as well.
OK, I'm not getting this; why isn't the sync_sched sufficient to get out of this fast path without barriers?
Quoted 5 lines
Quoted 5 lines
kernel/cpu.c +5
-0+void __get_online_cpus (void ) +{ +if (__cpuhp_writer ==1 ) { +/* See __srcu_read_lock() */ +__this_cpu_inc (__cpuhp_refcount);
Quoted 8 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 76
Quoted 4 lines
kernel/cpu.c +4
-0+smp_mb (); +__this_cpu_inc (cpuhp_seq); +return ; + }OK,
cpuhp_seqshould guaranteecpuhp_readers_active_check()gets the "stable" numbers. Looks suspicious... but lets assume this works.
I 'borrowed' it from SRCU, so if its broken here its broken there too I suppose.
Quoted 9 lines
However, I do not see how "
__cpuhp_writer== 1" can work, please see below.Review
Quoted 2 lines
(quoted diff) +2
-0+ if (atomic_dec_and_test(&cpuhp_waitcount)) + wake_up_all(&cpuhp_writer);Same problem as in previous version.
__get_online_cpus()succeeds without incrementing__cpuhp_refcount. "goto start" can't help afaics.
I added a goto into the cond-block, not before the cond; but see the version below.
Quoted 27 lines
Quoted 5 lines
void
cpu_hotplug_begin(void)
{kernel/cpu.c +2
-0+unsigned int count =0 ; +int cpu;kernel/cpu.c static int cpu_hotplug_disabled; · line 167
Quoted 10 lines
kernel/cpu.c +10
-0+lockdep_assert_held (&cpu_add_remove_lock); + +/* allow reader-in-writer recursion */ + current->cpuhp_ref++; + +/* make readers take the slow path */ + __cpuhp_writer =1 ; + +/* See percpu_down_write() */ +synchronize_sched ();Suppose there are no readers at this point,
kernel/cpu.c static int cpu_hotplug_disabled; · line 173
Quoted 6 lines
kernel/cpu.c +6
-0+ +/* make readers block */ + __cpuhp_writer =2 ; + +/* Wait for all readers to go away */ +wait_event (cpuhp_writer,cpuhp_readers_active_check ());So
wait_event()"quickly" returns.
Quoted 3 lines
Now. Why the new reader should see
__cpuhp_writer= 2 ? It can still see it == 1, and take that "if (__cpuhp_writer== 1)" path above.
OK, .. I see the hole, no immediate way to fix it -- too tired atm.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 20 lines · Peter Zijlstra
Quoted 17 lines · Oleg Nesterov
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 199
Quoted 13 lines
include/linux/cpu.h +13
-0+static inline void get_online_cpus (void ) +{ +might_sleep (); + +/* Support reader-in-reader recursion */ +if (current->cpuhp_ref++) { +barrier (); +return ; + } + +preempt_disable (); +if (likely (!__cpuhp_writer)) +__this_cpu_inc (__cpuhp_refcount);
mb()to ensure the reader can't miss, say, a STORE done inside the cpu_hotplug_begin/end section.
Quoted 4 lines
Quoted 1 line
put_online_cpus()needsmb()as well.OK, I'm not getting this; why isn't the
sync_schedsufficient to get out of this fast path without barriers?
Aah, sorry, I didn't notice this version has another synchronize_sched() in cpu_hotplug_done().
Then I need to recheck again...
No. Too tired too ;) damn LSB test failures...
Quoted 9 lines
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ if (atomic_dec_and_test(&cpuhp_waitcount)) + wake_up_all(&cpuhp_writer);Same problem as in previous version.
__get_online_cpus()succeeds without incrementing__cpuhp_refcount. "goto start" can't help afaics.I added a goto into the cond-block, not before the cond; but see the version below.
"into the cond-block" doesn't look right too, at first glance. This always succeeds, but by this time another writer can already hold the lock.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 1 line · Oleg Nesterov
No. Too tired too ;) damn LSB test failures...
ok; I cobbled this together.. I might think better of it tomorrow, but for now I think I closed the hole before wait_event(readers_active()) you pointed out -- of course I might have created new holes :/
For easy reading the + only version.
---
+++ b/include/linux/cpu.h
@@ -16,6 +16,7 @@
#include <linux/node.h>
#include <linux/compiler.h>
#include <linux/cpumask.h>
+#include <linux/percpu.h>
struct device;
@@ -173,10 +174,50 @@ extern struct bus_type cpu_subsys;
#ifdef CONFIG_HOTPLUG_CPU
/* Stop CPUs going up and down. */
+extern void cpu_hotplug_init_task(struct task_struct *p); +
extern void cpu_hotplug_begin(void);
extern void cpu_hotplug_done(void);
+
+extern int __cpuhp_writer; +DECLARE_PER_CPU(unsigned int, __cpuhp_refcount); +
+extern void __get_online_cpus(void);
+
+static inline void get_online_cpus(void)
+{
+ might_sleep();
+
+ /* Support reader-in-reader recursion */
+ if (current->cpuhp_ref++) {
+ barrier();
+ return;
+ }
+
+ preempt_disable();
+ if (likely(!__cpuhp_writer))
+ __this_cpu_inc(__cpuhp_refcount);
+ else
+ __get_online_cpus();
+ preempt_enable();
+}
+
+extern void __put_online_cpus(void);
+
+static inline void put_online_cpus(void)
+{
+ barrier();
+ if (--current->cpuhp_ref)
+ return;
+
+ preempt_disable();
+ if (likely(!__cpuhp_writer))
+ __this_cpu_dec(__cpuhp_refcount);
+ else
+ __put_online_cpus();
+ preempt_enable();
+}
+
extern void cpu_hotplug_disable(void);
extern void cpu_hotplug_enable(void);
#define hotcpu_notifier(fn, pri) cpu_notifier(fn, pri)
@@ -200,6 +241,8 @@ static inline void cpu_hotplug_driver_un
#else /* CONFIG_HOTPLUG_CPU */
+static inline void cpu_hotplug_init_task(struct task_struct *p) {} +
static inline void cpu_hotplug_begin(void) {}
static inline void cpu_hotplug_done(void) {}
#define get_online_cpus() do { } while (0)
+++ b/include/linux/sched.h
@@ -1454,6 +1454,9 @@ struct task_struct {
unsigned int sequential_io;
unsigned int sequential_io_avg;
#endif
+#ifdef CONFIG_HOTPLUG_CPU
+ int cpuhp_ref;
+#endif
};
/* Future-safe accessor for struct task_struct's cpus_allowed. */
+++ b/kernel/cpu.c
@@ -49,88 +49,140 @@ static int cpu_hotplug_disabled;
#ifdef CONFIG_HOTPLUG_CPU
+enum { readers_fast = 0, readers_slow, readers_block }; +
+int __cpuhp_writer; +EXPORT_SYMBOL_GPL(__cpuhp_writer);
+
+DEFINE_PER_CPU(unsigned int, __cpuhp_refcount); +EXPORT_PER_CPU_SYMBOL_GPL(__cpuhp_refcount);
+
+static DEFINE_PER_CPU(unsigned int, cpuhp_seq); +static atomic_t cpuhp_waitcount;
+static DECLARE_WAIT_QUEUE_HEAD(cpuhp_readers); +static DECLARE_WAIT_QUEUE_HEAD(cpuhp_writer); +
+void cpu_hotplug_init_task(struct task_struct *p) +{
+ p->cpuhp_ref = 0;
+}
+void __get_online_cpus(void)
{
+again:
+ /* See __srcu_read_lock() */
+ __this_cpu_inc(__cpuhp_refcount);
+ smp_mb(); /* A matches B, E */
+ __this_cpu_inc(cpuhp_seq);
+
+ if (unlikely(__cpuhp_writer == readers_block)) {
+ __put_online_cpus();
+
+ atomic_inc(&cpuhp_waitcount);
+
+ /*
+ * We either call schedule() in the wait, or we'll fall through
+ * and reschedule on the preempt_enable() in get_online_cpus().
+ */
+ preempt_enable_no_resched();
+ __wait_event(cpuhp_readers, __cpuhp_writer != readers_block);
+ preempt_disable();
+ if (atomic_dec_and_test(&cpuhp_waitcount))
+ wake_up_all(&cpuhp_writer);
+
+ goto again;
+ }
+}
+EXPORT_SYMBOL_GPL(__get_online_cpus);
+
+void __put_online_cpus(void)
+{
+ /* See __srcu_read_unlock() */
+ smp_mb(); /* C matches D */
+ this_cpu_dec(__cpuhp_refcount);
+
+ /* Prod writer to recheck readers_active */
+ wake_up_all(&cpuhp_writer);
}
+EXPORT_SYMBOL_GPL(__put_online_cpus);
+#define per_cpu_sum(var) \
+({ \
+ typeof(var) __sum = 0; \
+ int cpu; \
+ for_each_possible_cpu(cpu) \
+ __sum += per_cpu(var, cpu); \
+ __sum; \
+)}
+
+/*
+ * See srcu_readers_active_idx_check()
+ */
+static bool cpuhp_readers_active_check(void)
{
+ unsigned int seq = per_cpu_sum(cpuhp_seq);
+
+ smp_mb(); /* B matches A */
+ if (per_cpu_sum(__cpuhp_refcount) != 0)
+ return false;
+ smp_mb(); /* D matches C */
+ return per_cpu_sum(cpuhp_seq) == seq;
}
/*
* This ensures that the hotplug operation can begin only when the
* refcount goes to zero.
*
* Since cpu_hotplug_begin() is always called after invoking
* cpu_maps_update_begin(), we can be sure that only one writer is active. */
void cpu_hotplug_begin(void)
{
+ unsigned int count = 0;
+ int cpu;
+ lockdep_assert_held(&cpu_add_remove_lock);
+
+ /* allow reader-in-writer recursion */
+ current->cpuhp_ref++;
+
+ /* make readers take the slow path */
+ __cpuhp_writer = readers_slow;
+
+ /* See percpu_down_write() */
+ synchronize_sched();
+
+ /* make readers block */
+ __cpuhp_writer = readers_block;
+
+ smp_mb(); /* E matches A */
+
+ /* Wait for all readers to go away */
+ wait_event(cpuhp_writer, cpuhp_readers_active_check());
}
void cpu_hotplug_done(void)
{
+ /* Signal the writer is done, no fast path yet */
+ __cpuhp_writer = readers_slow;
+ wake_up_all(&cpuhp_readers);
+
+ /* See percpu_up_write() */
+ synchronize_sched();
+
+ /* Let 'em rip */
+ __cpuhp_writer = readers_fast;
+ current->cpuhp_ref--;
+
+ /*
+ * Wait for any pending readers to be running. This ensures readers
+ * after writer and avoids writers starving readers.
+ */
+ wait_event(cpuhp_writer, !atomic_read(&cpuhp_waitcount));
}
/*
+++ b/kernel/sched/core.c
@@ -1736,6 +1736,8 @@ static void __sched_fork(unsigned long c
INIT_LIST_HEAD(&p->numa_entry); p->numa_group = NULL;
#endif /* CONFIG_NUMA_BALANCING */
+
+ cpu_hotplug_init_task(p);
}
#ifdef CONFIG_NUMA_BALANCING
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 9 lines · Peter Zijlstra
Quoted 1 line · Oleg Nesterov
No. Too tired too ;) damn LSB test failures...
ok; I cobbled this together.. I might think better of it tomorrow, but for now I think I closed the hole before wait_event(readers_active()) you pointed out -- of course I might have created new holes :/
For easy reading the + only version.
A couple of nits and some commentary, but if there are races, they are quite subtle. ;-)
Thanx, Paul
Quoted 21 lines
---
+++b/include/linux/cpu.hinclude/linux/cpu.h +6
-0@@ -16,6 +16,7 @@#include <linux/node.h> #include <linux/compiler.h> #include <linux/cpumask.h> +#include <linux/percpu.h> struct device ;@@ -173,10 +174,50 @@ extern struct bus_type cpu_subsys;#ifdef CONFIG_HOTPLUG_CPU /* Stop CPUs going up and down. */ +extern void cpu_hotplug_init_task (struct task_struct *p); +extern void cpu_hotplug_begin (void );extern void cpu_hotplug_done (void ); + +extern int __cpuhp_writer; +DECLARE_PER_CPU(unsigned int, __cpuhp_refcount);
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 193
Quoted 10 lines
include/linux/cpu.h +10
-0+ +extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{ +might_sleep (); + +/* Support reader-in-reader recursion */ +if (current->cpuhp_ref++) { +barrier ();
Oleg was right, this barrier() can go. The value was >=1 and remains
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 199
Quoted 8 lines
include/linux/cpu.h +6
-0=1, so reordering causes no harm. (See below.) +return ; + } + +preempt_disable (); +if (likely (!__cpuhp_writer)) +__this_cpu_inc (__cpuhp_refcount);
The barrier required here is provided by synchronize_sched(), and all the code is contained by the barrier()s in preempt_disable() and preempt_enable().
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 201
Quoted 2 lines
include/linux/cpu.h +2
-0+else +__get_online_cpus ();
And a memory barrier is unconditionally executed by __get_online_cpus().
Review extern struct bus_type cpu_subsys; · line 204
Quoted 1 line
(quoted diff) +1
-0+ preempt_enable();
The barrier() in preempt_enable() prevents the compiler from bleeding the critical section out.
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 209
Quoted 7 lines
include/linux/cpu.h +7
-0+} + +extern void __put_online_cpus (void ); + +static inline void put_online_cpus (void ) +{ +barrier ();
This barrier() can also be dispensed with.
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 210
Quoted 1 line
include/linux/cpu.h +1
-0+if (--current->cpuhp_ref)
If we leave here, the value was >=1 and remains >=1, so reordering does no harm.
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 197
Quoted 3 lines
include/linux/cpu.h +3
-0+return ; + +preempt_disable ();
The barrier() in preempt_disable() prevents the compiler from bleeding the critical section out.
Review extern struct bus_type cpu_subsys; · line 217
Quoted 2 lines
(quoted diff) +2
-0+ if (likely(!__cpuhp_writer)) + __this_cpu_dec(__cpuhp_refcount);
The barrier here is supplied by synchronize_sched().
Review extern struct bus_type cpu_subsys; · line 219
Quoted 2 lines
(quoted diff) +2
-0+ else + __put_online_cpus();
And a memory barrier is unconditionally executed by __put_online_cpus().
Quoted 49 lines
include/linux/cpu.h +27
-0+preempt_enable (); +} +extern void cpu_hotplug_disable (void );extern void cpu_hotplug_enable (void );#define hotcpu_notifier (fn, pri) cpu_notifier(fn, pri)@@ -200,6 +241,8 @@ static inline void cpu_hotplug_driver_un#else /* CONFIG_HOTPLUG_CPU */ +static inline void cpu_hotplug_init_task (struct task_struct *p) {} +static inline void cpu_hotplug_begin (void ) {}static inline void cpu_hotplug_done (void ) {}#define get_online_cpus () do { } while (0) +++ b/include/linux/sched.h@@ -1454,6 +1454,9 @@ struct task_struct {unsigned int sequential_io;unsigned int sequential_io_avg;#endif +#ifdef CONFIG_HOTPLUG_CPU +int cpuhp_ref; +#endif };/* Future-safe accessor for struct task_struct's cpus_allowed. */ +++ b/kernel/cpu.c@@ -49,88 +49,140 @@ static int cpu_hotplug_disabled;#ifdef CONFIG_HOTPLUG_CPU +enum { readers_fast =0 , readers_slow, readers_block }; + +int __cpuhp_writer; +EXPORT_SYMBOL_GPL (__cpuhp_writer); + +DEFINE_PER_CPU(unsigned int, __cpuhp_refcount); +EXPORT_PER_CPU_SYMBOL_GPL (__cpuhp_refcount); + +static DEFINE_PER_CPU(unsigned int, cpuhp_seq); +static atomic_t cpuhp_waitcount; +static DECLARE_WAIT_QUEUE_HEAD (cpuhp_readers); +static DECLARE_WAIT_QUEUE_HEAD (cpuhp_writer); + +void cpu_hotplug_init_task (struct task_struct *p) +{ + p->cpuhp_ref =0 ; +}
include/linux/cpu.h static int cpu_hotplug_disabled; · line 79
Quoted 10 lines
include/linux/cpu.h +9
-0+void __get_online_cpus (void ) { +again: +/* See __srcu_read_lock() */ +__this_cpu_inc (__cpuhp_refcount); +smp_mb ();/* A matches B, E */ +__this_cpu_inc (cpuhp_seq); + +if (unlikely (__cpuhp_writer == readers_block)) { +__put_online_cpus ();
Suppose we got delayed here for some time. The writer might complete, and be awakened by the blocked readers (we have not incremented our counter yet). We would then drop through, do the atomic_dec_and_test() and deliver a spurious wake_up_all() at some random time in the future.
Which should be OK because __wait_event() looks to handle spurious wake_up()s.
kernel/cpu.c static int cpu_hotplug_disabled; · line 92
Quoted 12 lines
kernel/cpu.c +11
-0+atomic_inc (&cpuhp_waitcount); + +/* +* We either call schedule() in the wait, or we'll fall through +* and reschedule on the preempt_enable() in get_online_cpus(). +*/ +preempt_enable_no_resched (); +__wait_event (cpuhp_readers, __cpuhp_writer != readers_block); +preempt_disable (); +if (atomic_dec_and_test (&cpuhp_waitcount)) +wake_up_all (&cpuhp_writer);
There can be only one writer, so why the wake_up_all()?
kernel/cpu.c static int cpu_hotplug_disabled;
Quoted 10 lines
kernel/cpu.c +10
-0+ + goto again; + } +} +EXPORT_SYMBOL_GPL (__get_online_cpus); + +void __put_online_cpus (void ) +{ +/* See __srcu_read_unlock() */ +smp_mb ();/* C matches D */
In other words, if they see our decrement (presumably to aggregate zero, as that is the only time it matters) they will also see our critical section.
Quoted 14 lines
kernel/cpu.c +12
-0+this_cpu_dec (__cpuhp_refcount); + +/* Prod writer to recheck readers_active */ +wake_up_all (&cpuhp_writer); } +EXPORT_SYMBOL_GPL (__put_online_cpus); +#define per_cpu_sum (var) \ +({ \ + typeof(var) __sum = 0; \ + int cpu; \ + for_each_possible_cpu(cpu) \ + __sum += per_cpu(var, cpu); \ + __sum; \
kernel/cpu.c static int cpu_hotplug_disabled;
Quoted 10 lines
kernel/cpu.c +9
-0+)} + +/* +* See srcu_readers_active_idx_check() +*/ +static bool cpuhp_readers_active_check (void ) { +unsigned int seq =per_cpu_sum (cpuhp_seq); + +smp_mb ();/* B matches A */
In other words, if we see __get_online_cpus() cpuhp_seq increment, we are guaranteed to also see its __cpuhp_refcount increment.
kernel/cpu.c static int cpu_hotplug_disabled;
Quoted 6 lines
kernel/cpu.c +4
-0+if (per_cpu_sum (__cpuhp_refcount) !=0 ) +return false; +smp_mb ();/* D matches C */ +return per_cpu_sum (cpuhp_seq) == seq;
On equality, we know that there could not be any "sneak path" pairs where we see a decrement but not the corresponding increment for a given reader. If we saw its decrement, the memory barriers guarantee that we now see its cpuhp_seq increment.
Quoted 14 lines
kernel/cpu.c +2
-0}/* * This ensures that the hotplug operation can begin only when the * refcount goes to zero. * * Since cpu_hotplug_begin() is always called after invoking * cpu_maps_update_begin(), we can be sure that only one writer is active. */ void cpu_hotplug_begin (void ) { +unsigned int count =0 ; +int cpu;
kernel/cpu.c static int cpu_hotplug_disabled; · line 167
Quoted 10 lines
kernel/cpu.c +10
-0+lockdep_assert_held (&cpu_add_remove_lock); + +/* allow reader-in-writer recursion */ + current->cpuhp_ref++; + +/* make readers take the slow path */ + __cpuhp_writer = readers_slow; + +/* See percpu_down_write() */ +synchronize_sched ();
At this point, we know that all readers take the slow path.
kernel/cpu.c static int cpu_hotplug_disabled;
Quoted 4 lines
kernel/cpu.c +4
-0+/* make readers block */ + __cpuhp_writer = readers_block; + +smp_mb ();/* E matches A */
If they don't see our write of readers_block to __cpuhp_writer, then we are guaranteed to see their __cpuhp_refcount increment, and therefore will wait for them.
kernel/cpu.c static int cpu_hotplug_disabled; · line 180
Quoted 9 lines
kernel/cpu.c +5
-0+/* Wait for all readers to go away */ +wait_event (cpuhp_writer,cpuhp_readers_active_check ()); }void cpu_hotplug_done (void ) { +/* Signal the writer is done, no fast path yet */ + __cpuhp_writer = readers_slow; +wake_up_all (&cpuhp_readers);
OK, the wait_event()/wake_up_all() prevents the races where the readers are delayed between fetching __cpuhp_writer and blocking.
kernel/cpu.c static int cpu_hotplug_disabled; · line 183
Quoted 2 lines
kernel/cpu.c +2
-0+/* See percpu_up_write() */ +synchronize_sched ();
At this point, readers no longer attempt to block.
You avoid falling into the usual acquire-release-mismatch trap by using __cpuhp_refcount on both the fastpatch and the slowpath, so that it is OK to acquire on the fastpath and release on the slowpath (and vice versa).
Quoted 15 lines
kernel/cpu.c +10
-0+/* Let 'em rip */ + __cpuhp_writer = readers_fast; + current->cpuhp_ref--; + +/* +* Wait for any pending readers to be running. This ensures readers +* after writer and avoids writers starving readers. +*/ +wait_event (cpuhp_writer, !atomic_read (&cpuhp_waitcount)); }/* +++ b/kernel/sched/core.c
kernel/cpu.c static void __sched_fork(unsigned long c · line 1743
Quoted 8 lines
kernel/cpu.c +2
-0@@ -1736,6 +1736,8 @@ static void __sched_fork(unsigned long cINIT_LIST_HEAD (&p->numa_entry); p->numa_group =NULL ;#endif /* CONFIG_NUMA_BALANCING */ + +cpu_hotplug_init_task (p); }#ifdef CONFIG_NUMA_BALANCING
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 2 lines · Paul E. McKenney
A couple of nits and some commentary, but if there are races, they are quite subtle. ;-)
*whee*..
I made one little change in the logic; I moved the waitcount increment to before the __put_online_cpus() call, such that the writer will have to wait for us to wake up before trying again -- not for us to actually have acquired the read lock, for that we'd need to mess up __get_online_cpus() a bit more.
Complete patch below.
---
Subject: hotplug: Optimize {get,put}_online_cpus()
From: Peter Zijlstra <[email protected]>
Date: Tue Sep 17 16:17:11 CEST 2013
The current implementation of get_online_cpus() is global of nature and thus not suited for any kind of common usage.
Re-implement the current recursive r/w cpu hotplug lock such that the read side locks are as light as possible.
The current cpu hotplug lock is entirely reader biased; but since readers are expensive there aren't a lot of them about and writer starvation isn't a particular problem.
However by making the reader side more usable there is a fair chance it will get used more and thus the starvation issue becomes a real possibility.
Therefore this new implementation is fair, alternating readers and writers; this however requires per-task state to allow the reader recursion.
Many comments are contributed by Paul McKenney, and many previous attempts were shown to be inadequate by both Paul and Oleg; many thanks to them for persisting to poke holes in my attempts.
--- include/linux/cpu.h | 58 +++++++++++++ include/linux/sched.h | 3 kernel/cpu.c | 209 +++++++++++++++++++++++++++++++++++--------------- kernel/sched/core.c | 2 4 files changed, 208 insertions(+), 64 deletions(-)
include/linux/cpu.h +56 -2
@@ -16,6 +16,7 @@#include <linux/node.h> #include <linux/compiler.h> #include <linux/cpumask.h> +#include <linux/percpu.h> struct device ;
@@ -173,10 +174,61 @@ extern struct bus_type cpu_subsys;#ifdef CONFIG_HOTPLUG_CPU /* Stop CPUs going up and down. */ +extern void cpu_hotplug_init_task (struct task_struct *p); +extern void cpu_hotplug_begin (void );extern void cpu_hotplug_done (void ); -extern void get_online_cpus (void ); -extern void put_online_cpus (void ); + +extern int __cpuhp_state; +DECLARE_PER_CPU(unsigned int, __cpuhp_refcount); + +extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{ +might_sleep (); + +/* Support reader recursion */ +/* The value was >= 1 and remains so, reordering causes no harm. */ +if (current->cpuhp_ref++) +return ; + +preempt_disable (); +if (likely (!__cpuhp_state)) { +/* The barrier here is supplied by synchronize_sched(). */ +__this_cpu_inc (__cpuhp_refcount); + }else { +__get_online_cpus ();/* Unconditional memory barrier. */ + } +preempt_enable (); +/* +* The barrier() from preempt_enable() prevents the compiler from +* bleeding the critical section out. +*/ +} + +extern void __put_online_cpus (void ); + +static inline void put_online_cpus (void ) +{ +/* The value was >= 1 and remains so, reordering causes no harm. */ +if (--current->cpuhp_ref) +return ; + +/* +* The barrier() in preempt_disable() prevents the compiler from +* bleeding the critical section out. +*/ +preempt_disable (); +if (likely (!__cpuhp_state)) { +/* The barrier here is supplied by synchronize_sched(). */ +__this_cpu_dec (__cpuhp_refcount); + }else { +__put_online_cpus ();/* Unconditional memory barrier. */ + } +preempt_enable (); +} +extern void cpu_hotplug_disable (void );extern void cpu_hotplug_enable (void );#define hotcpu_notifier (fn, pri) cpu_notifier(fn, pri)
@@ -200,6 +252,8 @@ static inline void cpu_hotplug_driver_un#else /* CONFIG_HOTPLUG_CPU */ +static inline void cpu_hotplug_init_task (struct task_struct *p) {} +static inline void cpu_hotplug_begin (void ) {}static inline void cpu_hotplug_done (void ) {}#define get_online_cpus () do { } while (0)
include/linux/sched.h +3 -0
@@ -1454,6 +1454,9 @@ struct task_struct {unsigned int sequential_io;unsigned int sequential_io_avg;#endif +#ifdef CONFIG_HOTPLUG_CPU +int cpuhp_ref; +#endif };/* Future-safe accessor for struct task_struct's cpus_allowed. */
kernel/cpu.c +147 -62
235 lines not on this page · raw message
kernel/sched/core.c +2 -0
@@ -1736,6 +1736,8 @@ static void __sched_fork(unsigned long cINIT_LIST_HEAD (&p->numa_entry); p->numa_group =NULL ;#endif /* CONFIG_NUMA_BALANCING */ + +cpu_hotplug_init_task (p); }#ifdef CONFIG_NUMA_BALANCING
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
A message that is not in the archive (sent off-list or not yet received): [email protected]
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 12 lines · Oleg Nesterov
Quoted 11 lines · Peter Zijlstra
void
cpu_hotplug_done(void)
{kernel/cpu.c +7
-2- cpu_hotplug.active_writer =NULL ; -mutex_unlock (&cpu_hotplug.lock); +/* Signal the writer is done, no fast path yet. */ + __cpuhp_state = readers_slow; +wake_up_all (&cpuhp_readers); + +/* +* The wait_event()/wake_up_all() prevents the race where the readers +* are delayed between fetching __cpuhp_state and blocking.
Quoted 8 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 208
Quoted 4 lines
kernel/cpu.c +4
-0+*/ + +/* See percpu_up_write(); readers will no longer attempt to block. */ +synchronize_sched ();Shouldn't you move
wake_up_all(&cpuhp_readers)down aftersynchronize_sched()(or add another one) ? To ensure that a reader can't see state = BLOCK afterwakeup().
Well, if they are blocked, the wake_up_all() will do an actual try_to_wake_up() which issues a MB as per smp_mb__before_spinlock().
The woken task will get a MB from passing through the context switch to make it actually run. And therefore; like Paul's comment says; it cannot observe the previous BLOCK state but must indeed see the just issued SLOW state.
Right?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 23 lines · Peter Zijlstra
Quoted 12 lines · Oleg Nesterov
Quoted 11 lines · Peter Zijlstra
void
cpu_hotplug_done(void)
{kernel/cpu.c +7
-2- cpu_hotplug.active_writer =NULL ; -mutex_unlock (&cpu_hotplug.lock); +/* Signal the writer is done, no fast path yet. */ + __cpuhp_state = readers_slow; +wake_up_all (&cpuhp_readers); + +/* +* The wait_event()/wake_up_all() prevents the race where the readers +* are delayed between fetching __cpuhp_state and blocking. Quoted 8 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 208
Quoted 4 lines
kernel/cpu.c +4
-0+*/ + +/* See percpu_up_write(); readers will no longer attempt to block. */ +synchronize_sched ();Shouldn't you move
wake_up_all(&cpuhp_readers)down aftersynchronize_sched()(or add another one) ? To ensure that a reader can't see state = BLOCK afterwakeup().
Quoted 2 lines
Well, if they are blocked, the
wake_up_all()will do an actualtry_to_wake_up()which issues a MB as persmp_mb__before_spinlock().
Yes. Everything is fine with the already blocked readers.
I meant the new reader which still can see state = BLOCK after we do wakeup(), but I didn't notice it should do __wait_event() which takes the lock unconditionally, it must see the change after that.
Quoted 1 line
Right?
Yes, I was wrong, thanks.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 24 lines · Oleg Nesterov
Quoted 23 lines · Peter Zijlstra
Quoted 20 lines · Oleg Nesterov
Quoted 5 lines · Peter Zijlstra
void
cpu_hotplug_done(void)
{kernel/cpu.c +1
-2- cpu_hotplug.active_writer =NULL ; -mutex_unlock (&cpu_hotplug.lock); +/* Signal the writer is done, no fast path yet. */ kernel/cpu.c static int cpu_hotplug_disabled; · line 208
Quoted 10 lines
kernel/cpu.c +10
-0+ __cpuhp_state = readers_slow; +wake_up_all (&cpuhp_readers); + +/* +* The wait_event()/wake_up_all() prevents the race where the readers +* are delayed between fetching __cpuhp_state and blocking. +*/ + +/* See percpu_up_write(); readers will no longer attempt to block. */ +synchronize_sched ();Shouldn't you move
wake_up_all(&cpuhp_readers)down aftersynchronize_sched()(or add another one) ? To ensure that a reader can't see state = BLOCK afterwakeup().
Quoted 8 lines
Quoted 2 lines
Well, if they are blocked, the
wake_up_all()will do an actualtry_to_wake_up()which issues a MB as persmp_mb__before_spinlock().Yes. Everything is fine with the already blocked readers.
I meant the new reader which still can see state = BLOCK after we do
wakeup(), but I didn't notice it should do__wait_event()which takes the lock unconditionally, it must see the change after that.
Ah, because both __wake_up() and __wait_event()->prepare_to_wait() take q->lock. Thereby matching the __wake_up() RELEASE to the __wait_event() ACQUIRE, creating the full barrier.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Peter ZijlstraPeter,
Sorry. Unlikely I will be able to read this patch today. So let me ask another potentially wrong question without any thinking.
kernel/cpu.c static int cpu_hotplug_disabled; · line 78
Quoted 10 lines · Peter Zijlstra
kernel/cpu.c +9
-0+void __get_online_cpus (void ) +{ +again: +/* See __srcu_read_lock() */ +__this_cpu_inc (__cpuhp_refcount); +smp_mb ();/* A matches B, E */ +__this_cpu_inc (cpuhp_seq); + +if (unlikely (__cpuhp_state == readers_block)) {
OK. Either we should see state = BLOCK or the writer should notice the change in __cpuhp_refcount/seq. (altough I'd like to recheck this cpuhp_seq logic ;)
kernel/cpu.c static int cpu_hotplug_disabled; · line 84
Quoted 2 lines
kernel/cpu.c +2
-0+atomic_inc (&cpuhp_waitcount); +__put_online_cpus ();
OK, this does wake(cpuhp_writer).
Quoted 9 lines
void
cpu_hotplug_begin(void)
{
...kernel/cpu.c +6
-0+/* +* Notify new readers to block; up until now, and thus throughout the +* longish synchronize_sched() above, new readers could still come in. +*/ + __cpuhp_state = readers_block; +
kernel/cpu.c static int cpu_hotplug_disabled; · line 193
Quoted 10 lines
kernel/cpu.c +10
-0+smp_mb ();/* E matches A */ + +/* +* If they don't see our writer of readers_block to __cpuhp_state, +* then we are guaranteed to see their __cpuhp_refcount increment, and +* therefore will wait for them. +*/ + +/* Wait for all now active readers to complete. */ +wait_event (cpuhp_writer,cpuhp_readers_active_check ());
But. doesn't this mean that we need __wait_event() here as well?
Isn't it possible that the reader sees BLOCK but the writer does _not_ see the change in __cpuhp_refcount/cpuhp_seq? Those mb's guarantee "either", not "both".
Don't we need to ensure that we can't check cpuhp_readers_active_check() after wake(cpuhp_writer) was already called by the reader and before we take the same lock?
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 48 lines · Oleg Nesterov
Peter,
Sorry. Unlikely I will be able to read this patch today. So let me ask another potentially wrong question without any thinking.
kernel/cpu.c static int cpu_hotplug_disabled; · line 78
Quoted 10 lines · Peter Zijlstra
kernel/cpu.c +9
-0+void __get_online_cpus (void ) +{ +again: +/* See __srcu_read_lock() */ +__this_cpu_inc (__cpuhp_refcount); +smp_mb ();/* A matches B, E */ +__this_cpu_inc (cpuhp_seq); + +if (unlikely (__cpuhp_state == readers_block)) {OK. Either we should see state = BLOCK or the writer should notice the change in __cpuhp_refcount/seq. (altough I'd like to recheck this
cpuhp_seqlogic ;)kernel/cpu.c static int cpu_hotplug_disabled; · line 84
Quoted 2 lines
kernel/cpu.c +2
-0+atomic_inc (&cpuhp_waitcount); +__put_online_cpus ();OK, this does
wake(cpuhp_writer).Quoted 9 lines
void
cpu_hotplug_begin(void)
{
...kernel/cpu.c +6
-0+/* +* Notify new readers to block; up until now, and thus throughout the +* longish synchronize_sched() above, new readers could still come in. +*/ + __cpuhp_state = readers_block; +kernel/cpu.c static int cpu_hotplug_disabled; · line 193
Quoted 10 lines
kernel/cpu.c +10
-0+smp_mb ();/* E matches A */ + +/* +* If they don't see our writer of readers_block to __cpuhp_state, +* then we are guaranteed to see their __cpuhp_refcount increment, and +* therefore will wait for them. +*/ + +/* Wait for all now active readers to complete. */ +wait_event (cpuhp_writer,cpuhp_readers_active_check ());But. doesn't this mean that we need
__wait_event()here as well?
Quoted 3 lines
Isn't it possible that the reader sees BLOCK but the writer does _not_ see the change in __cpuhp_refcount/cpuhp_seq? Those mb's guarantee "either", not "both".
But if the readers does see BLOCK it will not be an active reader no more; and thus the writer doesn't need to observe and wait for it.
Quoted 3 lines
Don't we need to ensure that we can't check
cpuhp_readers_active_check()afterwake(cpuhp_writer)was already called by the reader and before we take the same lock?
I'm too tired to fully grasp what you're asking here; but given the previous answer I think not.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 3 lines · Peter Zijlstra
But if the readers does see BLOCK it will not be an active reader no more; and thus the writer doesn't need to observe and wait for it.
I meant they both can block, but please ignore. Today I simply can't understand what I was thinking about yesterday.
I tried hard to find any hole in this version but failed, I believe it is correct.
But, could you help me to understand some details?
kernel/cpu.c static int cpu_hotplug_disabled; · line 78
Quoted 9 lines
kernel/cpu.c +9
-0+void __get_online_cpus (void ) +{ +again: +/* See __srcu_read_lock() */ +__this_cpu_inc (__cpuhp_refcount); +smp_mb ();/* A matches B, E */ +__this_cpu_inc (cpuhp_seq); + +if (unlikely (__cpuhp_state == readers_block)) {
Note that there is no barrier() after inc(seq) and __cpuhp_state check, this inc() can be "postponed" till ...
kernel/cpu.c static int cpu_hotplug_disabled; · line 105
Quoted 10 lines
kernel/cpu.c +3
-6+void __put_online_cpus (void ) { -might_sleep (); -if (cpu_hotplug.active_writer == current) -return ; -mutex_lock (&cpu_hotplug.lock); - cpu_hotplug.refcount++; -mutex_unlock (&cpu_hotplug.lock); +/* See __srcu_read_unlock() */ +smp_mb ();/* C matches D */
... this mb() in __put_online_cpus().
And this is fine! The qustion is, perhaps it would be more "natural" and understandable to shift this_cpu_inc(cpuhp_seq) into __put_online_cpus().
We need to ensure 2 things:
1. The reader should notic state = BLOCK or the writer should see inc(__cpuhp_refcount). This is guaranteed by 2 mb's in __get_online_cpus() and in cpu_hotplug_begin().
We do not care if the writer misses some inc(__cpuhp_refcount) in per_cpu_sum(__cpuhp_refcount), that reader(s) should notice state = readers_block (and inc(cpuhp_seq) can't help anyway).
2. If the writer sees the result of this_cpu_dec(__cpuhp_refcount) from __put_online_cpus() (note that the writer can miss the corresponding inc() if it was done on another CPU, so this dec() can lead to sum() == 0), it should also notice the change in cpuhp_seq.
Fortunately, this can only happen if the reader migrates, in this case schedule() provides a barrier, the writer can't miss the change in cpuhp_seq.
IOW. Unless I missed something, cpuhp_seq is actually needed to serialize __put_online_cpus()->this_cpu_dec(__cpuhp_refcount) and and /* D matches C */ in cpuhp_readers_active_check(), and this is not immediately clear if you look at __get_online_cpus().
I do not suggest to change this code, but please tell me if my understanding is not correct.
Quoted 13 lines
kernel/cpu.c +9
-3+static bool cpuhp_readers_active_check (void ) { -if (cpu_hotplug.active_writer == current) -return ; -mutex_lock (&cpu_hotplug.lock); +unsigned int seq =per_cpu_sum (cpuhp_seq); + +smp_mb ();/* B matches A */ + +/* +* In other words, if we see __get_online_cpus() cpuhp_seq increment, +* we are guaranteed to also see its __cpuhp_refcount increment. +*/
kernel/cpu.c static int cpu_hotplug_disabled; · line 144
Quoted 10 lines
kernel/cpu.c +3
-5-if (WARN_ON (!cpu_hotplug.refcount)) - cpu_hotplug.refcount++;/* try to fix things up */ +if (per_cpu_sum (__cpuhp_refcount) !=0 ) +return false; -if (!--cpu_hotplug.refcount &&unlikely (cpu_hotplug.active_writer)) -wake_up_process (cpu_hotplug.active_writer); -mutex_unlock (&cpu_hotplug.lock); +smp_mb ();/* D matches C */
It seems that both barries could be smp_rmb() ? I am not sure the comments from srcu_readers_active_idx_check() can explain mb(), note that __srcu_read_lock() always succeeds unlike get_cpus_online().
Quoted 16 lines
void
cpu_hotplug_done(void)
{kernel/cpu.c +12
-2- cpu_hotplug.active_writer =NULL ; -mutex_unlock (&cpu_hotplug.lock); +/* Signal the writer is done, no fast path yet. */ + __cpuhp_state = readers_slow; +wake_up_all (&cpuhp_readers); + +/* +* The wait_event()/wake_up_all() prevents the race where the readers +* are delayed between fetching __cpuhp_state and blocking. +*/ + +/* See percpu_up_write(); readers will no longer attempt to block. */ +synchronize_sched (); +
kernel/cpu.c static int cpu_hotplug_disabled; · line 219
Quoted 10 lines
kernel/cpu.c +9
-0+/* Let 'em rip */ + __cpuhp_state = readers_fast; + current->cpuhp_ref--; + +/* +* Wait for any pending readers to be running. This ensures readers +* after writer and avoids writers starving readers. +*/ +wait_event (cpuhp_writer, !atomic_read (&cpuhp_waitcount)); }
OK, to some degree I can understand "avoids writers starving readers" part (although the next writer should do synchronize_sched() first), but could you explain "ensures readers after writer" ?
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 7 lines · Oleg Nesterov
Quoted 3 lines · Peter Zijlstra
But if the readers does see BLOCK it will not be an active reader no more; and thus the writer doesn't need to observe and wait for it.
I meant they both can block, but please ignore. Today I simply can't understand what I was thinking about yesterday.
I think we all know that state all too well ;-)
Quoted 2 lines
I tried hard to find any hole in this version but failed, I believe it is correct.
Yay!
Quoted 1 line
But, could you help me to understand some details?
I'll try, but I'm not too bright atm myself :-)
Quoted 20 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 78
Quoted 9 lines
kernel/cpu.c +9
-0+void __get_online_cpus (void ) +{ +again: +/* See __srcu_read_lock() */ +__this_cpu_inc (__cpuhp_refcount); +smp_mb ();/* A matches B, E */ +__this_cpu_inc (cpuhp_seq); + +if (unlikely (__cpuhp_state == readers_block)) {Note that there is no
barrier()after inc(seq) and__cpuhp_statecheck, thisinc()can be "postponed" till ...kernel/cpu.c static int cpu_hotplug_disabled; · line 105
Quoted 4 lines
kernel/cpu.c +3
-0+void __put_online_cpus (void ) { +/* See __srcu_read_unlock() */ +smp_mb ();/* C matches D */ ... this
mb()in__put_online_cpus().
Quoted 3 lines
And this is fine! The qustion is, perhaps it would be more "natural" and understandable to shift
this_cpu_inc(cpuhp_seq)into__put_online_cpus().
Possibly; I never got further than that the required order is:
ref++
MB
seq++
MB
ref--
It doesn't matter if the seq++ is in the lock or unlock primitive. I never considered one place more natural than the other.
Quoted 9 lines
We need to ensure 2 things:
1. The reader should notic state = BLOCK or the writer should see
inc(__cpuhp_refcount). This is guaranteed by 2 mb's in__get_online_cpus()and incpu_hotplug_begin().We do not care if the writer misses some
inc(__cpuhp_refcount)inper_cpu_sum(__cpuhp_refcount), that reader(s) should notice state =readers_block(andinc(cpuhp_seq)can't help anyway).
Agreed.
Quoted 8 lines
2. If the writer sees the result of
this_cpu_dec(__cpuhp_refcount)from__put_online_cpus()(note that the writer can miss the correspondinginc()if it was done on another CPU, so thisdec()can lead tosum()== 0), it should also notice the change incpuhp_seq.Fortunately, this can only happen if the reader migrates, in this case
schedule()provides a barrier, the writer can't miss the change incpuhp_seq.
Again, agreed; this is also the message of the second comment in cpuhp_readers_active_check() by Paul.
Quoted 7 lines
IOW. Unless I missed something,
cpuhp_seqis actually needed to serialize __put_online_cpus()->this_cpu_dec(__cpuhp_refcount) and and /* D matches C */ incpuhp_readers_active_check(), and this is not immediately clear if you look at__get_online_cpus().I do not suggest to change this code, but please tell me if my understanding is not correct.
I think you're entirely right.
Quoted 11 lines
Quoted 11 lines
kernel/cpu.c +9
-0+static bool cpuhp_readers_active_check (void ) { +unsigned int seq =per_cpu_sum (cpuhp_seq); + +smp_mb ();/* B matches A */ + +/* +* In other words, if we see __get_online_cpus() cpuhp_seq increment, +* we are guaranteed to also see its __cpuhp_refcount increment. +*/
Quoted 8 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 144
Quoted 4 lines
kernel/cpu.c +3
-0+if (per_cpu_sum (__cpuhp_refcount) !=0 ) +return false; +smp_mb ();/* D matches C */ It seems that both barries could be
smp_rmb()? I am not sure the comments fromsrcu_readers_active_idx_check()can explainmb(), note that__srcu_read_lock()always succeeds unlikeget_cpus_online().
I see what you mean; cpuhp_readers_active_check() is all purely reads; there are no writes to order.
Paul; is there any argument for the MB here as opposed to RMB; and if not should we change both these and SRCU?
Quoted 20 lines
Quoted 20 lines
void
cpu_hotplug_done(void)
{kernel/cpu.c +18
-0+/* Signal the writer is done, no fast path yet. */ + __cpuhp_state = readers_slow; +wake_up_all (&cpuhp_readers); + +/* +* The wait_event()/wake_up_all() prevents the race where the readers +* are delayed between fetching __cpuhp_state and blocking. +*/ + +/* See percpu_up_write(); readers will no longer attempt to block. */ +synchronize_sched (); + +/* Let 'em rip */ + __cpuhp_state = readers_fast; + current->cpuhp_ref--; + +/* +* Wait for any pending readers to be running. This ensures readers
Quoted 8 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 219
Quoted 4 lines
kernel/cpu.c +3
-0+* after writer and avoids writers starving readers. +*/ +wait_event (cpuhp_writer, !atomic_read (&cpuhp_waitcount)); }OK, to some degree I can understand "avoids writers starving readers" part (although the next writer should do
synchronize_sched()first), but could you explain "ensures readers after writer" ?
Suppose reader A sees state == BLOCK and goes to sleep; our writer B does cpu_hotplug_done() and wakes all pending readers. If for some reason A doesn't schedule to inc ref until B again executes cpu_hotplug_begin() and state is once again BLOCK, A will not have made any progress.
The waitcount increment before __put_online_cpus() ensures cpu_hotplug_done() sees the !0 waitcount and waits until out reader runs far enough to at least pass the dec_and_test().
And once past the dec_and_test() preemption is disabled and the sched_sync() in a new cpu_hotplug_begin() will suffice to guarantee we'll have acquired a reference and are an active reader.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 14 lines · Peter Zijlstra
Quoted 11 lines · Oleg Nesterov
Quoted 11 lines
kernel/cpu.c +9
-0+static bool cpuhp_readers_active_check (void ) { +unsigned int seq =per_cpu_sum (cpuhp_seq); + +smp_mb ();/* B matches A */ + +/* +* In other words, if we see __get_online_cpus() cpuhp_seq increment, +* we are guaranteed to also see its __cpuhp_refcount increment. +*/
Quoted 7 lines
Quoted 7 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 144
Quoted 4 lines
kernel/cpu.c +3
-0+if (per_cpu_sum (__cpuhp_refcount) !=0 ) +return false; +smp_mb ();/* D matches C */ It seems that both barries could be
smp_rmb()? I am not sure the comments fromsrcu_readers_active_idx_check()can explainmb(),
To avoid the confusion, I meant "those comments can't explain mb()s here, in cpuhp_readers_active_check()".
Quoted 2 lines
Quoted 2 lines
note that
__srcu_read_lock()always succeeds unlikeget_cpus_online().
And this cput_hotplug_ and synchronize_srcu() differ, see below.
Quoted 4 lines
I see what you mean;
cpuhp_readers_active_check()is all purely reads; there are no writes to order.Paul; is there any argument for the MB here as opposed to RMB;
Yes, Paul, please ;)
Quoted 2 lines
and if
not should we change both these and SRCU?
I guess that SRCU is more "complex" in this respect. IIUC, cpuhp_readers_active_check() needs "more" barriers because if synchronize_srcu() succeeds it needs to synchronize with the new readers which call srcu_read_lock/unlock() "right now". Again, unlike cpu-hotplug srcu never blocks the readers, srcu_read_*() always succeeds.
Hmm. I am wondering why __srcu_read_lock() needs ACCESS_ONCE() to increment ->c and ->seq. A plain this_cpu_inc() should be fine?
And since it disables preemption, why it can't use __this_cpu_inc() to inc ->c[idx]. OK, in general __this_cpu_inc() is not irq-safe (rmw) so we can't do __this_cpu_inc(seq[idx]), c[idx] should be fine? If irq does srcu_read_lock() it should also do _unlock.
But this is minor/offtopic.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
void
cpu_hotplug_done(void)
{
...
Quoted 11 lines
Quoted 10 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 219
Quoted 6 lines
kernel/cpu.c +5
-0+/* +* Wait for any pending readers to be running. This ensures readers +* after writer and avoids writers starving readers. +*/ +wait_event (cpuhp_writer, !atomic_read (&cpuhp_waitcount)); }OK, to some degree I can understand "avoids writers starving readers" part (although the next writer should do
synchronize_sched()first), but could you explain "ensures readers after writer" ?
Quoted 5 lines
Suppose reader A sees state == BLOCK and goes to sleep; our writer B does
cpu_hotplug_done()and wakes all pending readers. If for some reason A doesn't schedule to inc ref until B again executescpu_hotplug_begin()and state is once again BLOCK, A will not have made any progress.
Yes, yes, thanks, this is clear. But this explains "writers starving readers". And let me repeat, if B again executes cpu_hotplug_begin() it will do another synchronize_sched() before it sets BLOCK, so I am not sure we need this "in practice".
I was confused by "ensures readers after writer", I thought this means we need the additional synchronization with the readers which are going to increment cpuhp_waitcount, say, some sort of barries.
Please note that this wait_event() adds a problem... it doesn't allow to "offload" the final synchronize_sched(). Suppose a 4k cpu machine does disable_nonboot_cpus(), we do not want 2 * 4k * synchronize_sched's in this case. We can solve this, but this wait_event() complicates the problem.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 20 lines · Oleg Nesterov
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
void
cpu_hotplug_done(void)
{...
Quoted 11 lines
Quoted 10 lines
Review
Quoted 6 lines
(quoted diff) +5
-0+ /* + * Wait for any pending readers to be running. This ensures readers + * after writer and avoids writers starving readers. + */ + wait_event(cpuhp_writer, !atomic_read(&cpuhp_waitcount)); }OK, to some degree I can understand "avoids writers starving readers" part (although the next writer should do
synchronize_sched()first), but could you explain "ensures readers after writer" ?Quoted 5 lines
Suppose reader A sees state == BLOCK and goes to sleep; our writer B does
cpu_hotplug_done()and wakes all pending readers. If for some reason A doesn't schedule to inc ref until B again executescpu_hotplug_begin()and state is once again BLOCK, A will not have made any progress.
Quoted 8 lines
Yes, yes, thanks, this is clear. But this explains "writers starving readers". And let me repeat, if B again executes
cpu_hotplug_begin()it will do anothersynchronize_sched()before it sets BLOCK, so I am not sure we need this "in practice".I was confused by "ensures readers after writer", I thought this means we need the additional synchronization with the readers which are going to increment
cpuhp_waitcount, say, some sort of barries.
Ah no; I just wanted to guarantee that any pending readers did get a chance to run. And yes due to the two sync_sched() calls it seems somewhat unlikely in practise.
Quoted 5 lines
Please note that this
wait_event()adds a problem... it doesn't allow to "offload" the finalsynchronize_sched(). Suppose a 4k cpu machine doesdisable_nonboot_cpus(), we do not want 2 * 4k * synchronize_sched's in this case. We can solve this, but thiswait_event()complicates the problem.
That seems like a particularly easy fix; something like so?
--- include/linux/cpu.h | 1 kernel/cpu.c | 84 ++++++++++++++++++++++++++++++++++------------------ 2 files changed, 56 insertions(+), 29 deletions(-)
include/linux/cpu.h +1 -0
@@ -109,6 +109,7 @@ enum {#define CPU_DOWN_FAILED_FROZEN (CPU_DOWN_FAILED | CPU_TASKS_FROZEN)#define CPU_DEAD_FROZEN (CPU_DEAD | CPU_TASKS_FROZEN)#define CPU_DYING_FROZEN (CPU_DYING | CPU_TASKS_FROZEN) +#define CPU_POST_DEAD_FROZEN (CPU_POST_DEAD | CPU_TASKS_FROZEN)#define CPU_STARTING_FROZEN (CPU_STARTING | CPU_TASKS_FROZEN)
kernel/cpu.c +55 -29
171 lines not on this page · raw message
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Oleg Nesterov
Please note that this
wait_event()adds a problem... it doesn't allow to "offload" the finalsynchronize_sched(). Suppose a 4k cpu machine doesdisable_nonboot_cpus(), we do not want 2 * 4k * synchronize_sched's in this case. We can solve this, but thiswait_event()complicates the problem.That seems like a particularly easy fix; something like so?
Yes, but...
kernel/cpu.c int disable_nonboot_cpus(void) · line 607
Quoted 6 lines
kernel/cpu.c +4
-0@@ -586,6 +603,11 @@ int disable_nonboot_cpus(void) +cpu_hotplug_done (); + +for_each_cpu (cpu, frozen_cpus) +cpu_notify_nofail (CPU_POST_DEAD_FROZEN , (void *)(long )cpu);
This changes the protocol, I simply do not know if it is fine in general to do __cpu_down(another_cpu) without CPU_POST_DEAD(previous_cpu). Say, currently it is possible that CPU_DOWN_PREPARE takes some global lock released by CPU_DOWN_FAILED or CPU_POST_DEAD.
Hmm. Now that workqueues do not use CPU_POST_DEAD, it has only 2 users, mce_cpu_callback() and cpufreq_cpu_callback() and the 1st one even ignores this notification if FROZEN. So yes, probably this is fine, but needs an ack from cpufreq maintainers (cc'ed), for example to ensure that it is fine to call __cpufreq_remove_dev_prepare() twice without _finish().
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 26 lines · Oleg Nesterov
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Oleg Nesterov
Please note that this
wait_event()adds a problem... it doesn't allow to "offload" the finalsynchronize_sched(). Suppose a 4k cpu machine doesdisable_nonboot_cpus(), we do not want 2 * 4k * synchronize_sched's in this case. We can solve this, but thiswait_event()complicates the problem.That seems like a particularly easy fix; something like so?
Yes, but...
kernel/cpu.c int disable_nonboot_cpus(void) · line 607
Quoted 6 lines
kernel/cpu.c +4
-0@@ -586,6 +603,11 @@ int disable_nonboot_cpus(void) +cpu_hotplug_done (); + +for_each_cpu (cpu, frozen_cpus) +cpu_notify_nofail (CPU_POST_DEAD_FROZEN , (void *)(long )cpu);This changes the protocol, I simply do not know if it is fine in general to do
__cpu_down(another_cpu)withoutCPU_POST_DEAD(previous_cpu). Say, currently it is possible thatCPU_DOWN_PREPAREtakes some global lock released byCPU_DOWN_FAILEDorCPU_POST_DEAD.
Quoted 5 lines
Hmm. Now that workqueues do not use
CPU_POST_DEAD, it has only 2 users,mce_cpu_callback()andcpufreq_cpu_callback()and the 1st one even ignores this notification if FROZEN. So yes, probably this is fine, but needs an ack from cpufreq maintainers (cc'ed), for example to ensure that it is fine to call__cpufreq_remove_dev_prepare()twice without_finish().
To my eyes it will return -EBUSY when it tries to stop an already stopped governor, which will cause the entire chain to fail I guess.
Srivatsa has touched that code most recently, so he should know better, though.
Thanks,
Rafael
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 33 lines · Rafael J. Wysocki
Quoted 26 lines · Oleg Nesterov
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Oleg Nesterov
Please note that this
wait_event()adds a problem... it doesn't allow to "offload" the finalsynchronize_sched(). Suppose a 4k cpu machine doesdisable_nonboot_cpus(), we do not want 2 * 4k * synchronize_sched's in this case. We can solve this, but thiswait_event()complicates the problem.That seems like a particularly easy fix; something like so?
Yes, but...
kernel/cpu.c int disable_nonboot_cpus(void) · line 607
Quoted 6 lines
kernel/cpu.c +4
-0@@ -586,6 +603,11 @@ int disable_nonboot_cpus(void) +cpu_hotplug_done (); + +for_each_cpu (cpu, frozen_cpus) +cpu_notify_nofail (CPU_POST_DEAD_FROZEN , (void *)(long )cpu);This changes the protocol, I simply do not know if it is fine in general to do
__cpu_down(another_cpu)withoutCPU_POST_DEAD(previous_cpu). Say, currently it is possible thatCPU_DOWN_PREPAREtakes some global lock released byCPU_DOWN_FAILEDorCPU_POST_DEAD.Quoted 5 lines
Hmm. Now that workqueues do not use
CPU_POST_DEAD, it has only 2 users,mce_cpu_callback()andcpufreq_cpu_callback()and the 1st one even ignores this notification if FROZEN. So yes, probably this is fine, but needs an ack from cpufreq maintainers (cc'ed), for example to ensure that it is fine to call__cpufreq_remove_dev_prepare()twice without_finish().
Quoted 5 lines
To my eyes it will return
-EBUSYwhen it tries to stop an already stopped governor, which will cause the entire chain to fail I guess.Srivatsa has touched that code most recently, so he should know better, though.
Yes it will return -EBUSY, but unfortunately it gets scarier from that point onwards. When it gets an -EBUSY, __cpufreq_remove_dev_prepare() aborts its work mid-way and returns, but doesn't bubble up the error to the CPU-hotplug core. So the CPU hotplug code will continue to take that CPU down, with further notifications such as CPU_DEAD, and chaos will ensue.
And we can't exactly "fix" this by simply returning the error code to CPU-hotplug (since that would mean that suspend/resume would _always_ fail). Perhaps we can teach cpufreq to ignore the error in this particular case (since the governor has already been stopped and that's precisely what this function wanted to do as well), but the problems don't seem to end there.
The other issue is that the CPUs in the policy->cpus mask are removed in the _dev_finish() stage. So if that stage is post-poned like this, then _dev_prepare() will get thoroughly confused since it also depends on seeing an updated policy->cpus mask to decide when to nominate a new policy->cpu etc. (And the cpu nomination code itself might start ping-ponging between CPUs, since none of the CPUs would have been removed from the policy->cpus mask).
So, to summarize, this change to CPU hotplug code will break cpufreq (and suspend/resume) as things stand today, but I don't think these problems are insurmountable though..
However, as Oleg said, its definitely worth considering whether this proposed change in semantics is going to hurt us in the future. CPU_POST_DEAD has certainly proved to be very useful in certain challenging situations (commit 1aee40ac9c explains one such example), so IMHO we should be very careful not to undermine its utility.
Regards,
Srivatsa S. Bhat
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 5 lines · Srivatsa S. Bhat
However, as Oleg said, its definitely worth considering whether this proposed change in semantics is going to hurt us in the future.
CPU_POST_DEADhas certainly proved to be very useful in certain challenging situations (commit1aee40ac9cexplains one such example), so IMHO we should be very careful not to undermine its utility.
Urgh.. crazy things. I've always understood POST_DEAD to mean 'will be called at some time after the unplug' with no further guarantees. And my patch preserves that.
Its not at all clear to me why cpufreq needs more; 1aee40ac9c certainly doesn't explain it.
What's wrong with leaving a cleanup handle in percpu storage and effectively doing:
struct cpu_destroy {
void (*destroy)(void *);
void *args;
};DEFINE_PER_CPU(struct cpu_destroy, cpu_destroy);
POST_DEAD:
{
struct cpu_destroy x = per_cpu(cpu_destroy, cpu);
if (x.destroy)
x.destroy(x.arg);
}POST_DEAD cannot fail; so CPU_DEAD/CPU_DOWN_PREPARE can simply assume it will succeed; it has to.
The cpufreq situation simply doesn't make any kind of sense to me.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 11 lines · Peter Zijlstra
Quoted 5 lines · Srivatsa S. Bhat
However, as Oleg said, its definitely worth considering whether this proposed change in semantics is going to hurt us in the future.
CPU_POST_DEADhas certainly proved to be very useful in certain challenging situations (commit1aee40ac9cexplains one such example), so IMHO we should be very careful not to undermine its utility.Urgh.. crazy things. I've always understood
POST_DEADto mean 'will be called at some time after the unplug' with no further guarantees. And my patch preserves that.
I tend to agree with Srivatsa... Without a strong reason it would be better to preserve the current logic: "some time after" should not be after the next CPU_DOWN/UP*. But I won't argue too much.
But note that you do not strictly need this change. Just kill cpuhp_waitcount, then we can change cpu_hotplug_begin/end to use xxx_enter/exit we discuss in another thread, this should likely "join" all synchronize_sched's.
Or split cpu_hotplug_begin() into 2 helpers which handle FAST -> SLOW and SLOW -> BLOCK transitions, then move the first "FAST -> SLOW" handler outside of for_each_online_cpu().
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 9 lines · Oleg Nesterov
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Srivatsa S. Bhat
However, as Oleg said, its definitely worth considering whether this proposed change in semantics is going to hurt us in the future.
CPU_POST_DEADhas certainly proved to be very useful in certain challenging situations (commit1aee40ac9cexplains one such example), so IMHO we should be very careful not to undermine its utility.
Quoted 7 lines
Quoted 3 lines
Urgh.. crazy things. I've always understood
POST_DEADto mean 'will be called at some time after the unplug' with no further guarantees. And my patch preserves that.I tend to agree with Srivatsa... Without a strong reason it would be better to preserve the current logic: "some time after" should not be after the next CPU_DOWN/UP*. But I won't argue too much.
Nah, I think breaking it is the right thing :-)
Quoted 3 lines
But note that you do not strictly need this change. Just kill
cpuhp_waitcount, then we can change cpu_hotplug_begin/end to use xxx_enter/exit we discuss in another thread, this should likely "join" all synchronize_sched's.
That would still be 4k * sync_sched() == terribly long.
Quoted 3 lines
Or split
cpu_hotplug_begin()into 2 helpers which handle FAST -> SLOW and SLOW -> BLOCK transitions, then move the first "FAST -> SLOW" handler outside offor_each_online_cpu().
Right, that's more messy but would work if we cannot teach cpufreq (and possibly others) to not rely on state you shouldn't rely on anyway.
I tihnk the only guarnatee POST_DEAD should have is that it should be called before UP_PREPARE of the same cpu ;-) Nothing more, nothing less.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Oleg Nesterov
I tend to agree with Srivatsa... Without a strong reason it would be better to preserve the current logic: "some time after" should not be after the next CPU_DOWN/UP*. But I won't argue too much.
Nah, I think breaking it is the right thing :-)
I don't really agree but I won't argue ;)
Quoted 5 lines
Quoted 3 lines
But note that you do not strictly need this change. Just kill
cpuhp_waitcount, then we can change cpu_hotplug_begin/end to use xxx_enter/exit we discuss in another thread, this should likely "join" all synchronize_sched's.That would still be 4k *
sync_sched()== terribly long.
No? the next xxx_enter() avoids sync_sched() if rcu callback is still pending. Unless __cpufreq_remove_dev_finish() is "too slow" of course.
Quoted 6 lines
Quoted 3 lines
Or split
cpu_hotplug_begin()into 2 helpers which handle FAST -> SLOW and SLOW -> BLOCK transitions, then move the first "FAST -> SLOW" handler outside offor_each_online_cpu().Right, that's more messy but would work if we cannot teach cpufreq (and possibly others) to not rely on state you shouldn't rely on anyway.
Yes,
Quoted 2 lines
I tihnk the only guarnatee
POST_DEADshould have is that it should be called beforeUP_PREPAREof the same cpu ;-) Nothing more, nothing less.
See above... This makes POST_DEAD really "special" compared to other CPU_* events.
And again. Something like a global lock taken by CPU_DOWN_PREPARE and released by POST_DEAD or DOWN_FAILED does not look "too wrong" to me.
But I leave this to you and Srivatsa.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 11 lines · Oleg Nesterov
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Oleg Nesterov
I tend to agree with Srivatsa... Without a strong reason it would be better to preserve the current logic: "some time after" should not be after the next CPU_DOWN/UP*. But I won't argue too much.
Nah, I think breaking it is the right thing :-)
I don't really agree but I won't argue ;)
The authors of arch/x86/kernel/cpu/mcheck/mce.c would seem to be the guys who would need to complain, given that they seem to have the only use in 3.11.
Thanx, Paul
Quoted 24 lines
Quoted 5 lines
Quoted 3 lines
But note that you do not strictly need this change. Just kill
cpuhp_waitcount, then we can change cpu_hotplug_begin/end to use xxx_enter/exit we discuss in another thread, this should likely "join" all synchronize_sched's.That would still be 4k *
sync_sched()== terribly long.No? the next
xxx_enter()avoidssync_sched()if rcu callback is still pending. Unless__cpufreq_remove_dev_finish()is "too slow" of course.Quoted 6 lines
Quoted 3 lines
Or split
cpu_hotplug_begin()into 2 helpers which handle FAST -> SLOW and SLOW -> BLOCK transitions, then move the first "FAST -> SLOW" handler outside offor_each_online_cpu().Right, that's more messy but would work if we cannot teach cpufreq (and possibly others) to not rely on state you shouldn't rely on anyway.
Yes,
Quoted 2 lines
I tihnk the only guarnatee
POST_DEADshould have is that it should be called beforeUP_PREPAREof the same cpu ;-) Nothing more, nothing less.See above... This makes
POST_DEADreally "special" compared to other CPU_* events.
Quoted 7 lines
And again. Something like a global lock taken by
CPU_DOWN_PREPAREand released byPOST_DEADorDOWN_FAILEDdoes not look "too wrong" to me.But I leave this to you and Srivatsa.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 10 lines · Paul E. McKenney
Quoted 8 lines · Oleg Nesterov
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Oleg Nesterov
I tend to agree with Srivatsa... Without a strong reason it would be better to preserve the current logic: "some time after" should not be after the next CPU_DOWN/UP*. But I won't argue too much.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Nah, I think breaking it is the right thing :-)
I don't really agree but I won't argue ;)
The authors of
arch/x86/kernel/cpu/mcheck/mce.cwould seem to be the guys who would need to complain, given that they seem to have the only use in 3.11.
mce_cpu_callback() is fine, it ignores POST_DEAD if CPU_TASKS_FROZEN.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Oleg NesterovQuoted 8 lines · Oleg Nesterov
Quoted 5 lines
Quoted 3 lines
But note that you do not strictly need this change. Just kill
cpuhp_waitcount, then we can change cpu_hotplug_begin/end to use xxx_enter/exit we discuss in another thread, this should likely "join" all synchronize_sched's.That would still be 4k *
sync_sched()== terribly long.No? the next
xxx_enter()avoidssync_sched()if rcu callback is still pending. Unless__cpufreq_remove_dev_finish()is "too slow" of course.
Hmm,. not in the version you posted; there xxx_enter() would only not do the sync_sched if there's a concurrent 'writer', in which case it will wait for it.
You only avoid the sync_sched in xxx_exit() and potentially join in the sync_sched() of a next xxx_begin().
So with that scheme:
for (i= ; i<4096; i++) {
xxx_begin();
xxx_exit();
}Will get 4096 sync_sched() calls from the xxx_begin() and all but the last xxx_exit() will 'drop' the rcu callback.
And given the construct; I'm not entirely sure you can do away with the sync_sched() in between. While its clear to me you can merge the two into one; leaving it out entirely doesn't seem right.
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Oleg Nesterov
Quoted 4 lines
Quoted 3 lines
But note that you do not strictly need this change. Just kill
cpuhp_waitcount, then we can change cpu_hotplug_begin/end to use xxx_enter/exit we discuss in another thread, this should likely "join" all synchronize_sched's.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
That would still be 4k *
sync_sched()== terribly long.No? the next
xxx_enter()avoidssync_sched()if rcu callback is still pending. Unless__cpufreq_remove_dev_finish()is "too slow" of course.Hmm,. not in the version you posted; there
xxx_enter()would only not do thesync_schedif there's a concurrent 'writer', in which case it will wait for it.
No, please see below.
Quoted 5 lines
You only avoid the
sync_schedinxxx_exit()and potentially join in thesync_sched()of a nextxxx_begin().So with that scheme:
Quoted 7 lines
for (i= ; i<4096; i++) { xxx_begin(); xxx_exit(); }Will get 4096
sync_sched()calls from thexxx_begin()and all but the lastxxx_exit()will 'drop' the rcu callback.
No, the code above should call sync_sched() only once, no matter what this code does between _enter and _exit. This was one of the points.
To clarify, of course I mean the "likely" case. Say, a long preemption after _exit can lead to another sync_sched().
void xxx_enter(struct xxx_struct *xxx)
{
bool need_wait, need_sync;
spin_lock_irq(&xxx->xxx_lock);
need_wait = xxx->gp_count++;
need_sync = xxx->gp_state == GP_IDLE;
if (need_sync)
xxx->gp_state = GP_PENDING;
spin_unlock_irq(&xxx->xxx_lock);
BUG_ON(need_wait && need_sync);
if (need_sync) {
synchronize_sched();
xxx->gp_state = GP_PASSED;
wake_up_all(&xxx->gp_waitq);
} else if (need_wait) {
wait_event(&xxx->gp_waitq, xxx->gp_state == GP_PASSED);
} else {
BUG_ON(xxx->gp_state != GP_PASSED);
}
}The 1st iteration:
xxx_enter() does synchronize_sched() and sets gp_state = GP_PASSED.
xxx_exit() starts the rcu callback, but gp_state is still PASSED.
all other iterations in the "likely" case:
xxx_enter() should likely come before the pending callback fires and clears gp_state. In this case we only increment ->gp_count (this "disables" the rcu callback) and do nothing more, gp_state is still GP_PASSED.
xxx_exit() does another call_rcu_sched(), or does the CP_PENDING -> CB_REPLAY change. The latter is the same as "start another callback".
In short: unless a gp elapses between _exit() and _enter(), the next _enter() does nothing and avoids synchronize_sched().
Quoted 3 lines
And given the construct; I'm not entirely sure you can do away with the
sync_sched()in between. While its clear to me you can merge the two into one; leaving it out entirely doesn't seem right.
Could you explain?
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 6 lines · Oleg Nesterov
Quoted 3 lines · Peter Zijlstra
And given the construct; I'm not entirely sure you can do away with the
sync_sched()in between. While its clear to me you can merge the two into one; leaving it out entirely doesn't seem right.Could you explain?
Somehow I thought the fastpath got enabled; it doesn't since we never hit GP_IDLE, so we don't actually need that.
You're right.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Oleg NesterovQuoted 2 lines · Oleg Nesterov
In short: unless a gp elapses between
_exit()and_enter(), the next_enter()does nothing and avoidssynchronize_sched().
That does however make the entire scheme entirely writer biased; increasing the need for the waitcount thing I have. Otherwise we'll starve pending readers.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Oleg Nesterov
In short: unless a gp elapses between
_exit()and_enter(), the next_enter()does nothing and avoidssynchronize_sched().That does however make the entire scheme entirely writer biased;
Well, this makes the scheme "a bit more" writer biased, but this is exactly what we want in this case.
We do not block the readers after xxx_exit() entirely, but we do want to keep them in SLOW state and avoid the costly SLOW -> FAST -> SLOW transitions.
Lets even forget about disable_nonboot_cpus(), lets consider percpu_rwsem-like logic "in general".
Yes, it is heavily optimizied for readers. But if the writers come in a batch, or the same writer does down_write + up_write twice or more, I think state == FAST is pointless in between (if we can avoid it). This is the rare case (the writers should be rare), but if it happens it makes sense to optimize the writers too. And again, even
for (;;) {
percpu_down_write();
percpu_up_write();
}should not completely block the readers.
IOW. "turn sync_sched() into call_rcu_sched() in up_write()" is obviously a win. If the next down_write/xxx_enter "knows" that the readers are still in SLOW mode because gp was not completed yet, why should we add the artificial delay?
As for disable_nonboot_cpus(). You are going to move cpu_hotplug_begin() outside of the loop, this is the same thing.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 8 lines · Oleg Nesterov
And again, even
for (;;) { percpu_down_write(); percpu_up_write(); }should not completely block the readers.
Sure there's a tiny window, but don't forget that a reader will have to wait for the gp_state cacheline to transfer to shared state and the per-cpu refcount cachelines to be brought back into exclusive mode and the above can be aggressive enough that by that time we'll observe state == blocked again.
So I don't think that in practise a reader will get in.
Also, since the write side is exposed to userspace; you've got an effective DoS.
So I'll stick to waitcount -- as you can see in the patches I've just posted.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Oleg Nesterov
And again, even
for (;;) { percpu_down_write(); percpu_up_write(); }
Quoted 7 lines
Quoted 1 line
should not completely block the readers.
Sure there's a tiny window, but don't forget that a reader will have to wait for the
gp_statecacheline to transfer to shared state and the per-cpu refcount cachelines to be brought back into exclusive mode and the above can be aggressive enough that by that time we'll observe state == blocked again.
Sure, but don't forget that other callers of cpu_down() do a lot more work before/after they actually call cpu_hotplug_begin/end().
Quoted 2 lines
So I'll stick to waitcount -- as you can see in the patches I've just posted.
I still do not believe we need this waitcount "in practice" ;)
But even if I am right this is minor and we can reconsider this later, so please forget.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Oleg NesterovQuoted 6 lines · Oleg Nesterov
Quoted 5 lines · Peter Zijlstra
Quoted 2 lines · Oleg Nesterov
In short: unless a gp elapses between
_exit()and_enter(), the next_enter()does nothing and avoidssynchronize_sched().
Quoted 8 lines
Quoted 1 line
That does however make the entire scheme entirely writer biased;
Well, this makes the scheme "a bit more" writer biased, but this is exactly what we want in this case.
We do not block the readers after
xxx_exit()entirely, but we do want to keep them in SLOW state and avoid the costly SLOW -> FAST -> SLOW transitions.
Yes -- should help -a- -lot- for bulk write-side operations, such as onlining all CPUs at boot time. ;-)
Thanx, Paul
Quoted 21 lines
Lets even forget about
disable_nonboot_cpus(), lets consider percpu_rwsem-like logic "in general".Yes, it is heavily optimizied for readers. But if the writers come in a batch, or the same writer does
down_write+up_writetwice or more, I think state == FAST is pointless in between (if we can avoid it). This is the rare case (the writers should be rare), but if it happens it makes sense to optimize the writers too. And again, evenfor (;;) { percpu_down_write(); percpu_up_write(); }should not completely block the readers.
IOW. "turn
sync_sched()intocall_rcu_sched()inup_write()" is obviously a win. If the next down_write/xxx_enter "knows" that the readers are still in SLOW mode because gp was not completed yet, why should we add the artificial delay?
Quoted 5 lines
As for
disable_nonboot_cpus(). You are going to movecpu_hotplug_begin()outside of the loop, this is the same thing.Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Peter ZijlstraQuoted 30 lines · Peter Zijlstra
Quoted 9 lines · Oleg Nesterov
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Srivatsa S. Bhat
However, as Oleg said, its definitely worth considering whether this proposed change in semantics is going to hurt us in the future.
CPU_POST_DEADhas certainly proved to be very useful in certain challenging situations (commit1aee40ac9cexplains one such example), so IMHO we should be very careful not to undermine its utility.Quoted 7 lines
Quoted 3 lines
Urgh.. crazy things. I've always understood
POST_DEADto mean 'will be called at some time after the unplug' with no further guarantees. And my patch preserves that.I tend to agree with Srivatsa... Without a strong reason it would be better to preserve the current logic: "some time after" should not be after the next CPU_DOWN/UP*. But I won't argue too much.
Nah, I think breaking it is the right thing :-)
Quoted 3 lines
But note that you do not strictly need this change. Just kill
cpuhp_waitcount, then we can change cpu_hotplug_begin/end to use xxx_enter/exit we discuss in another thread, this should likely "join" all synchronize_sched's.That would still be 4k *
sync_sched()== terribly long.Quoted 3 lines
Or split
cpu_hotplug_begin()into 2 helpers which handle FAST -> SLOW and SLOW -> BLOCK transitions, then move the first "FAST -> SLOW" handler outside offor_each_online_cpu().
Quoted 6 lines
Right, that's more messy but would work if we cannot teach cpufreq (and possibly others) to not rely on state you shouldn't rely on anyway.
I tihnk the only guarnatee
POST_DEADshould have is that it should be called beforeUP_PREPAREof the same cpu ;-) Nothing more, nothing less.
Conceptually, that hints at a totally per-cpu implementation of CPU hotplug, in which what happens to one CPU doesn't affect the others in the hotplug path.. and yeah, that sounds very tempting! ;-) but I guess that will need to be preceded by a massive rework of many of the existing hotplug callbacks ;-)
Regards,
Srivatsa S. Bhat
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Peter ZijlstraQuoted 7 lines · Peter Zijlstra
Quoted 5 lines · Srivatsa S. Bhat
However, as Oleg said, its definitely worth considering whether this proposed change in semantics is going to hurt us in the future.
CPU_POST_DEADhas certainly proved to be very useful in certain challenging situations (commit1aee40ac9cexplains one such example), so IMHO we should be very careful not to undermine its utility.
Quoted 7 lines
Urgh.. crazy things. I've always understood
POST_DEADto mean 'will be called at some time after the unplug' with no further guarantees. And my patch preserves that.Its not at all clear to me why cpufreq needs more;
1aee40ac9ccertainly doesn't explain it.
Sorry if I was unclear - I didn't mean to say that cpufreq needs more guarantees than that. I was just saying that the cpufreq code would need certain additional changes/restructuring to accommodate the change in the semantics brought about by this patch. IOW, it won't work as it is, but it can certainly be fixed.
My other point (unrelated to cpufreq) was this: POST_DEAD of course means that it will be called after unplug, with hotplug lock dropped. But it also provides the guarantee (in the existing code), that a *new* hotplug operation won't start until the POST_DEAD stage is also completed. This patch doesn't seem to honor that part. The concern I have is in cases like those mentioned by Oleg - say you take a lock at DOWN_PREPARE and want to drop it at POST_DEAD; or some other requirement that makes it important to finish a full hotplug cycle before moving on to the next one. I don't really have such a requirement in mind at present, but I was just trying to think what we would be losing with this change...
But to reiterate, I believe cpufreq can be reworked so that it doesn't depend on things such as the above. But I wonder if dropping that latter guarantee is going to be OK, going forward.
Regards,
Srivatsa S. Bhat
Quoted 23 lines
What's wrong with leaving a cleanup handle in percpu storage and effectively doing:
struct cpu_destroy { void (*destroy)(void *); void *args; };
DEFINE_PER_CPU(struct cpu_destroy, cpu_destroy);POST_DEAD: { struct cpu_destroy x = per_cpu(cpu_destroy, cpu); if (x.destroy) x.destroy(x.arg); }
POST_DEADcannot fail; so CPU_DEAD/CPU_DOWN_PREPARE can simply assume it will succeed; it has to.The cpufreq situation simply doesn't make any kind of sense to me.
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 12 lines · Srivatsa S. Bhat
Quoted 11 lines · Peter Zijlstra
Quoted 5 lines · Srivatsa S. Bhat
However, as Oleg said, its definitely worth considering whether this proposed change in semantics is going to hurt us in the future.
CPU_POST_DEADhas certainly proved to be very useful in certain challenging situations (commit1aee40ac9cexplains one such example), so IMHO we should be very careful not to undermine its utility.Urgh.. crazy things. I've always understood
POST_DEADto mean 'will be called at some time after the unplug' with no further guarantees. And my patch preserves that.
Quoted 9 lines
Quoted 3 lines
Its not at all clear to me why cpufreq needs more;
1aee40ac9ccertainly doesn't explain it.Sorry if I was unclear - I didn't mean to say that cpufreq needs more guarantees than that. I was just saying that the cpufreq code would need certain additional changes/restructuring to accommodate the change in the semantics brought about by this patch. IOW, it won't work as it is, but it can certainly be fixed.
And an important reason why this change can be accommodated with not so much trouble is because you are changing it only in the suspend/resume path, where userspace has already been frozen, so all hotplug operations are initiated by the suspend path and that path *alone* (and so we enjoy certain "simplifiers" that we know before-hand, eg: all of them are CPU offline operations, happening one at a time, in sequence) and we don't expect any "interference" to this routine ;-). As a result the number and variety of races that we need to take care of tend to be far lesser. (For example, we don't have to worry about the deadlock caused by sysfs-writes that 1aee40ac9c was talking about).
On the other hand, if the proposal was to change the regular hotplug path as well on the same lines, then I guess it would have been a little more difficult to adjust to it. For example, in cpufreq, _dev_prepare() sends a STOP to the governor, whereas a part of _dev_finish() sends a START to it; so we might have races there, due to which we might proceed with CPU offline with a running governor, depending on the exact timing of the events. Of course, this problem doesn't occur in the suspend/resume case, and hence I didn't bring it up in my previous mail.
So this is another reason why I'm a little concerned about POST_DEAD: since this is a change in semantics, it might be worth asking ourselves whether we'd still want to go with that change, if we happened to be changing regular hotplug as well, rather than just the more controlled environment of suspend/resume. Yes, I know that's not what you proposed, but I feel it might be worth considering its implications while deciding how to solve the POST_DEAD issue.
Regards,
Srivatsa S. Bhat
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Srivatsa S. BhatQuoted 12 lines · Srivatsa S. Bhat
Quoted 11 lines · Peter Zijlstra
Quoted 5 lines · Srivatsa S. Bhat
However, as Oleg said, its definitely worth considering whether this proposed change in semantics is going to hurt us in the future.
CPU_POST_DEADhas certainly proved to be very useful in certain challenging situations (commit1aee40ac9cexplains one such example), so IMHO we should be very careful not to undermine its utility.Urgh.. crazy things. I've always understood
POST_DEADto mean 'will be called at some time after the unplug' with no further guarantees. And my patch preserves that.
Quoted 9 lines
Quoted 3 lines
Its not at all clear to me why cpufreq needs more;
1aee40ac9ccertainly doesn't explain it.Sorry if I was unclear - I didn't mean to say that cpufreq needs more guarantees than that. I was just saying that the cpufreq code would need certain additional changes/restructuring to accommodate the change in the semantics brought about by this patch. IOW, it won't work as it is, but it can certainly be fixed.
Ok, so I thought a bit more about the changes you are proposing, and I agree that they would be beneficial in the long run, especially given that it can eventually lead to a more stream-lined hotplug process where different CPUs can be hotplugged independently without waiting on each other, like you mentioned in your other mail. So I'm fine with the new POST_DEAD guarantees you are proposing - that they are run after unplug, and will be completed before UP_PREPARE of the same CPU. And its also very convenient that we need to fix only cpufreq to accommodate this change.
So below is a quick untested patch that modifies the cpufreq hotplug callbacks appropriately. With this, cpufreq should be able to handle the POST_DEAD changes, irrespective of whether we do that in the regular path or in the suspend/resume path. (Because, I've restructured it in such a way that the races that I had mentioned earlier are totally avoided. That is, the POST_DEAD handler now performs only the bare-minimal final cleanup, which doesn't race with or depend on anything else).
drivers/cpufreq/cpufreq.c +24 -29
@@ -1165,7 +1165,7 @@ static int __cpufreq_remove_dev_prepare(struct device *dev, bool frozen) {unsigned int cpu = dev->id, cpus; -int new_cpu, ret; +int new_cpu, ret =0 ;unsigned long flags;struct cpufreq_policy *policy;
@@ -1200,9 +1200,10 @@ static int __cpufreq_remove_dev_prepare(struct device *dev, policy->governor->name, CPUFREQ_NAME_LEN);#endif -lock_policy_rwsem_read (cpu); +lock_policy_rwsem_write (cpu); cpus =cpumask_weight (policy->cpus); -unlock_policy_rwsem_read (cpu); +cpumask_clear_cpu (cpu, policy->cpus); +unlock_policy_rwsem_write (cpu);if (cpu != policy->cpu) {if (!frozen)
@@ -1220,7 +1221,23 @@ static int __cpufreq_remove_dev_prepare(struct device *dev, } } -return 0 ; +/* If no target, nothing more to do */ +if (!cpufreq_driver->target) +return 0 ; + +/* If cpu is last user of policy, cleanup the policy governor */ +if (cpus ==1 ) { + ret =__cpufreq_governor (policy,CPUFREQ_GOV_POLICY_EXIT ); +if (ret) +pr_err ("%s: Failed to exit governor\n" , __func__); + }else { +if ((ret =__cpufreq_governor (policy,CPUFREQ_GOV_START )) || + (ret =__cpufreq_governor (policy,CPUFREQ_GOV_LIMITS ))) { +pr_err ("%s: Failed to start governor\n" , __func__); + } + } + +return ret; } static int __cpufreq_remove_dev_finish(struct device *dev,
@@ -1243,25 +1260,12 @@ static int __cpufreq_remove_dev_finish(struct device *dev,return -EINVAL ; } -WARN_ON (lock_policy_rwsem_write (cpu)); +WARN_ON (lock_policy_rwsem_read (cpu)); cpus =cpumask_weight (policy->cpus); - -if (cpus >1 ) -cpumask_clear_cpu (cpu, policy->cpus); -unlock_policy_rwsem_write (cpu); +unlock_policy_rwsem_read (cpu);/* If cpu is last user of policy, free policy */ -if (cpus ==1 ) { -if (cpufreq_driver->target) { - ret =__cpufreq_governor (policy, -CPUFREQ_GOV_POLICY_EXIT ); -if (ret) { -pr_err ("%s: Failed to exit governor\n" , - __func__); -return ret; - } - } - +if (cpus ==0 ) {if (!frozen) {lock_policy_rwsem_read (cpu); kobj = &policy->kobj;
@@ -1294,15 +1298,6 @@ static int __cpufreq_remove_dev_finish(struct device *dev,if (!frozen)cpufreq_policy_free (policy); - }else { -if (cpufreq_driver->target) { -if ((ret =__cpufreq_governor (policy,CPUFREQ_GOV_START )) || - (ret =__cpufreq_governor (policy,CPUFREQ_GOV_LIMITS ))) { -pr_err ("%s: Failed to start governor\n" , - __func__); -return ret; - } - } }per_cpu (cpufreq_cpu_data, cpu) =NULL ;
Regards,
Srivatsa S. Bhat
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Oleg NesterovQuoted 29 lines · Oleg Nesterov
Quoted 14 lines · Peter Zijlstra
Quoted 11 lines · Oleg Nesterov
Quoted 11 lines
kernel/cpu.c +9
-0+static bool cpuhp_readers_active_check (void ) { +unsigned int seq =per_cpu_sum (cpuhp_seq); + +smp_mb ();/* B matches A */ + +/* +* In other words, if we see __get_online_cpus() cpuhp_seq increment, +* we are guaranteed to also see its __cpuhp_refcount increment. +*/ Quoted 7 lines
Quoted 7 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 144
Quoted 4 lines
kernel/cpu.c +3
-0+if (per_cpu_sum (__cpuhp_refcount) !=0 ) +return false; +smp_mb ();/* D matches C */ It seems that both barries could be
smp_rmb()? I am not sure the comments fromsrcu_readers_active_idx_check()can explainmb(),To avoid the confusion, I meant "those comments can't explain mb()s here, in
cpuhp_readers_active_check()".Quoted 2 lines
Quoted 2 lines
note that
__srcu_read_lock()always succeeds unlikeget_cpus_online().
Quoted 8 lines
And this
cput_hotplug_andsynchronize_srcu()differ, see below.Quoted 4 lines
I see what you mean;
cpuhp_readers_active_check()is all purely reads; there are no writes to order.Paul; is there any argument for the MB here as opposed to RMB;
Yes, Paul, please ;)
Sorry to be slow -- I will reply by end of Monday Pacific time at the latest. I need to allow myself enough time so that it seems new...
Also I might try some mechanical proofs of parts of it.
Thanx, Paul
Quoted 50 lines
Quoted 2 lines
and if
not should we change both these and SRCU?I guess that SRCU is more "complex" in this respect. IIUC,
cpuhp_readers_active_check()needs "more" barriers because ifsynchronize_srcu()succeeds it needs to synchronize with the new readers which call srcu_read_lock/unlock() "right now". Again, unlike cpu-hotplug srcu never blocks the readers, srcu_read_*() always succeeds.Hmm. I am wondering why
__srcu_read_lock()needsACCESS_ONCE()to increment ->c and ->seq. A plainthis_cpu_inc()should be fine?And since it disables preemption, why it can't use
__this_cpu_inc()to inc ->c[idx]. OK, in general__this_cpu_inc()is not irq-safe (rmw) so we can't do__this_cpu_inc(seq[idx]), c[idx] should be fine? If irq doessrcu_read_lock()it should also do_unlock.But this is minor/offtopic.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
void
cpu_hotplug_done(void)
{...
Quoted 11 lines
Quoted 10 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 219
Quoted 6 lines
kernel/cpu.c +5
-0+/* +* Wait for any pending readers to be running. This ensures readers +* after writer and avoids writers starving readers. +*/ +wait_event (cpuhp_writer, !atomic_read (&cpuhp_waitcount)); }OK, to some degree I can understand "avoids writers starving readers" part (although the next writer should do
synchronize_sched()first), but could you explain "ensures readers after writer" ?Quoted 5 lines
Suppose reader A sees state == BLOCK and goes to sleep; our writer B does
cpu_hotplug_done()and wakes all pending readers. If for some reason A doesn't schedule to inc ref until B again executescpu_hotplug_begin()and state is once again BLOCK, A will not have made any progress.Yes, yes, thanks, this is clear. But this explains "writers starving readers". And let me repeat, if B again executes
cpu_hotplug_begin()it will do anothersynchronize_sched()before it sets BLOCK, so I am not sure we need this "in practice".I was confused by "ensures readers after writer", I thought this means we need the additional synchronization with the readers which are going to increment
cpuhp_waitcount, say, some sort of barries.
Quoted 8 lines
Please note that this
wait_event()adds a problem... it doesn't allow to "offload" the finalsynchronize_sched(). Suppose a 4k cpu machine doesdisable_nonboot_cpus(), we do not want 2 * 4k * synchronize_sched's in this case. We can solve this, but thiswait_event()complicates the problem.Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Peter ZijlstraQuoted 2 lines · Peter Zijlstra
Quoted 1 line · Oleg Nesterov
On 09/26, Peter Zijlstra wrote:
[ . . . ]
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
kernel/cpu.c +9
-0+static bool cpuhp_readers_active_check (void ) { +unsigned int seq =per_cpu_sum (cpuhp_seq); + +smp_mb ();/* B matches A */ + +/* +* In other words, if we see __get_online_cpus() cpuhp_seq increment, +* we are guaranteed to also see its __cpuhp_refcount increment. +*/ Quoted 8 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 144
Quoted 4 lines
kernel/cpu.c +3
-0+if (per_cpu_sum (__cpuhp_refcount) !=0 ) +return false; +smp_mb ();/* D matches C */ It seems that both barries could be
smp_rmb()? I am not sure the comments fromsrcu_readers_active_idx_check()can explainmb(), note that__srcu_read_lock()always succeeds unlikeget_cpus_online().
Quoted 5 lines
I see what you mean;
cpuhp_readers_active_check()is all purely reads; there are no writes to order.Paul; is there any argument for the MB here as opposed to RMB; and if not should we change both these and SRCU?
Given that these memory barriers execute only on the semi-slow path, why add the complexity of moving from smp_mb() to either smp_rmb() or smp_wmb()? Straight smp_mb() is easier to reason about and more robust against future changes.
Thanx, Paul
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 30 lines · Paul E. McKenney
Quoted 2 lines · Peter Zijlstra
Quoted 1 line · Oleg Nesterov
On 09/26, Peter Zijlstra wrote:
[ . . . ]
Quoted 23 lines
Quoted 11 lines
Quoted 11 lines
kernel/cpu.c +9
-0+static bool cpuhp_readers_active_check (void ) { +unsigned int seq =per_cpu_sum (cpuhp_seq); + +smp_mb ();/* B matches A */ + +/* +* In other words, if we see __get_online_cpus() cpuhp_seq increment, +* we are guaranteed to also see its __cpuhp_refcount increment. +*/ Quoted 8 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 144
Quoted 4 lines
kernel/cpu.c +3
-0+if (per_cpu_sum (__cpuhp_refcount) !=0 ) +return false; +smp_mb ();/* D matches C */ It seems that both barries could be
smp_rmb()? I am not sure the comments fromsrcu_readers_active_idx_check()can explainmb(), note that__srcu_read_lock()always succeeds unlikeget_cpus_online().I see what you mean;
cpuhp_readers_active_check()is all purely reads; there are no writes to order.
Quoted 7 lines
Quoted 2 lines
Paul; is there any argument for the MB here as opposed to RMB; and if not should we change both these and SRCU?
Given that these memory barriers execute only on the semi-slow path, why add the complexity of moving from
smp_mb()to eithersmp_rmb()orsmp_wmb()? Straightsmp_mb()is easier to reason about and more robust against future changes.
But otoh this looks misleading, and the comments add more confusion.
But please note another email, it seems to me we can simply kill cpuhp_seq and all the barriers in cpuhp_readers_active_check().
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 39 lines · Oleg Nesterov
Quoted 30 lines · Paul E. McKenney
Quoted 2 lines · Peter Zijlstra
Quoted 1 line · Oleg Nesterov
On 09/26, Peter Zijlstra wrote:
[ . . . ]
Quoted 23 lines
Quoted 11 lines
Quoted 11 lines
(quoted diff) +9
-0+static bool cpuhp_readers_active_check(void) { + unsigned int seq = per_cpu_sum(cpuhp_seq); + + smp_mb(); /* B matches A */ + + /* + * In other words, if we see __get_online_cpus() cpuhp_seq increment, + * we are guaranteed to also see its __cpuhp_refcount increment. + */Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +3
-0+ if (per_cpu_sum(__cpuhp_refcount) != 0) + return false; + smp_mb(); /* D matches C */It seems that both barries could be
smp_rmb()? I am not sure the comments fromsrcu_readers_active_idx_check()can explainmb(), note that__srcu_read_lock()always succeeds unlikeget_cpus_online().I see what you mean;
cpuhp_readers_active_check()is all purely reads; there are no writes to order.Quoted 7 lines
Quoted 2 lines
Paul; is there any argument for the MB here as opposed to RMB; and if not should we change both these and SRCU?
Given that these memory barriers execute only on the semi-slow path, why add the complexity of moving from
smp_mb()to eithersmp_rmb()orsmp_wmb()? Straightsmp_mb()is easier to reason about and more robust against future changes.
Quoted 4 lines
But otoh this looks misleading, and the comments add more confusion.
But please note another email, it seems to me we can simply kill
cpuhp_seqand all the barriers incpuhp_readers_active_check().
If you don't have cpuhp_seq, you need some other way to avoid counter overflow. Which might be provided by limited number of tasks, or, on 64-bit systems, 64-bit counters.
Thanx, Paul
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 3 lines · Paul E. McKenney
If you don't have
cpuhp_seq, you need some other way to avoid counter overflow. Which might be provided by limited number of tasks, or, on 64-bit systems, 64-bit counters.
How so? PID space is basically limited to 30 bits, so how could we overflow a 32bit reference counter?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Paul E. McKenney
If you don't have
cpuhp_seq, you need some other way to avoid counter overflow. Which might be provided by limited number of tasks, or, on 64-bit systems, 64-bit counters.How so? PID space is basically limited to 30 bits, so how could we overflow a 32bit reference counter?
Nesting.
Thanx, Paul
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 11 lines · Paul E. McKenney
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Paul E. McKenney
If you don't have
cpuhp_seq, you need some other way to avoid counter overflow. Which might be provided by limited number of tasks, or, on 64-bit systems, 64-bit counters.How so? PID space is basically limited to 30 bits, so how could we overflow a 32bit reference counter?
Nesting.
Still it seems that UINT_MAX / PID_MAX_LIMIT has enough room.
But again, OK lets make it ulong. The question is, how cpuhp_seq can help and why we can't kill it.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Paul E. McKenneyQuoted 8 lines · Paul E. McKenney
Quoted 3 lines · Oleg Nesterov
But please note another email, it seems to me we can simply kill
cpuhp_seqand all the barriers incpuhp_readers_active_check().If you don't have
cpuhp_seq, you need some other way to avoid counter overflow.
I don't think so. Overflows (espicially "unsigned") should be fine and in fact we can't avoid them.
Say, a task does get() on CPU_0 and put() on CPU_1, after that we have
CTR[0] == 1, CTR[1] = (unsigned)-1
iow, the counter was already overflowed (underflowed). But this is fine, all we care about is CTR[0] + CTR[1] == 0, and this is only true because of another overflow.
But probably you meant another thing,
Quoted 2 lines
Which might be provided by limited number of
tasks, or, on 64-bit systems, 64-bit counters.
perhaps you meant that max_threads * max_depth can overflow the counter? I don't think so... but OK, perhaps this counter should be u_long.
But how cpuhp_seq can help?
Oleg.
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Oleg NesterovQuoted 3 lines · Oleg Nesterov
I tried hard to find any hole in this version but failed, I believe it is correct.
And I still believe it is. But now I am starting to think that we don't need cpuhp_seq. (and imo cpuhp_waitcount, but this is minor).
Quoted 9 lines
We need to ensure 2 things:
1. The reader should notic state = BLOCK or the writer should see
inc(__cpuhp_refcount). This is guaranteed by 2 mb's in__get_online_cpus()and incpu_hotplug_begin().We do not care if the writer misses some
inc(__cpuhp_refcount)inper_cpu_sum(__cpuhp_refcount), that reader(s) should notice state =readers_block(andinc(cpuhp_seq)can't help anyway).
Yes!
Quoted 4 lines
2. If the writer sees the result of
this_cpu_dec(__cpuhp_refcount)from__put_online_cpus()(note that the writer can miss the correspondinginc()if it was done on another CPU, so thisdec()can lead tosum()== 0),
But this can't happen in this version? Somehow I forgot that __get_online_cpus() does inc/get under preempt_disable(), always on the same CPU. And thanks to mb's the writer should not miss the reader which has already passed the "state != BLOCK" check.
To simplify the discussion, lets ignore the "readers_fast" state, synchronize_sched() logic looks obviously correct. IOW, lets discuss only the SLOW -> BLOCK transition.
cput_hotplug_begin()
{
state = BLOCK;
mb();
wait_event(cpuhp_writer,
per_cpu_sum(__cpuhp_refcount) == 0);
}should work just fine? Ignoring all details, we have
get_online_cpus()
{
again:
preempt_disable();
__this_cpu_inc(__cpuhp_refcount);
mb();
if (state == BLOCK) {
mb();
__this_cpu_dec(__cpuhp_refcount);
wake_up_all(cpuhp_writer);
preempt_enable();
wait_event(state != BLOCK);
goto again;
}
preempt_enable();
}It seems to me that these mb's guarantee all we need, no?
It looks really simple. The reader can only succed if it doesn't see BLOCK, in this case per_cpu_sum() should see the change,
We have
WRITER READER on CPU X state = BLOCK; __cpuhp_refcount[X]++; mb(); mb(); ... count += __cpuhp_refcount[X]; if (state != BLOCK) ... return; mb(); __cpuhp_refcount[X]--;
Either reader or writer should notice the STORE we care about.
If a reader can decrement __cpuhp_refcount, we have 2 cases:
1. It is the reader holding this lock. In this case we can't miss the corresponding inc() done by this reader, because this reader didn't see BLOCK in the past.
It is just the
A == B == 0
CPU_0 CPU_1
----- -----
A = 1; B = 1;
mb(); mb();
b = B; a = A;
pattern, at least one CPU should see 1 in its a/b.
2. It is the reader which tries to take this lock and noticed state == BLOCK. We could miss the result of its inc(), but we do not care, this reader is going to block.
_If_ the reader could migrate between inc/dec, then yes, we have a problem. Because that dec() could make the result of per_cpu_sum() = 0. IOW, we could miss inc() but notice dec(). But given that it does this on the same CPU this is not possible.
So why do we need cpuhp_seq?
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 7 lines · Oleg Nesterov
Quoted 3 lines · Oleg Nesterov
I tried hard to find any hole in this version but failed, I believe it is correct.
And I still believe it is. But now I am starting to think that we don't need
cpuhp_seq. (and imocpuhp_waitcount, but this is minor).
Here is one scenario that I believe requires cpuhp_seq:
1. Task 0 on CPU 0 increments its counter on entry.
2. Task 1 on CPU 1 starts summing the counters and gets to
CPU 4. The sum thus far is 1 (Task 0).
3. Task 2 on CPU 2 increments its counter on entry.
Upon completing its entry code, it re-enables preemption.
4. Task 2 is preempted, and starts running on CPU 5.
5. Task 2 decrements its counter on exit.
6. Task 1 continues summing. Due to the fact that it saw Task 2's
exit but not its entry, the sum is zero.
One of cpuhp_seq's jobs is to prevent this scenario.
That said, bozo here still hasn't gotten to look at Peter's newest patch, so perhaps it prevents this scenario some other way, perhaps by your argument below.
Quoted 11 lines
Quoted 9 lines
We need to ensure 2 things:
1. The reader should notic state = BLOCK or the writer should see
inc(__cpuhp_refcount). This is guaranteed by 2 mb's in__get_online_cpus()and incpu_hotplug_begin().We do not care if the writer misses some
inc(__cpuhp_refcount)inper_cpu_sum(__cpuhp_refcount), that reader(s) should notice state =readers_block(andinc(cpuhp_seq)can't help anyway).Yes!
OK, I will look over the patch with this in mind.
Quoted 94 lines
Quoted 4 lines
2. If the writer sees the result of
this_cpu_dec(__cpuhp_refcount)from__put_online_cpus()(note that the writer can miss the correspondinginc()if it was done on another CPU, so thisdec()can lead tosum()== 0),But this can't happen in this version? Somehow I forgot that
__get_online_cpus()does inc/get underpreempt_disable(), always on the same CPU. And thanks to mb's the writer should not miss the reader which has already passed the "state != BLOCK" check.To simplify the discussion, lets ignore the "
readers_fast" state,synchronize_sched()logic looks obviously correct. IOW, lets discuss only the SLOW -> BLOCK transition.cput_hotplug_begin() { state = BLOCK; mb(); wait_event(cpuhp_writer, per_cpu_sum(__cpuhp_refcount) == 0); }should work just fine? Ignoring all details, we have
get_online_cpus() { again: preempt_disable(); __this_cpu_inc(__cpuhp_refcount); mb(); if (state == BLOCK) { mb(); __this_cpu_dec(__cpuhp_refcount); wake_up_all(cpuhp_writer); preempt_enable(); wait_event(state != BLOCK); goto again; } preempt_enable(); }It seems to me that these mb's guarantee all we need, no?
It looks really simple. The reader can only succed if it doesn't see BLOCK, in this case
per_cpu_sum()should see the change,We have
WRITER READER on CPU X state = BLOCK; __cpuhp_refcount[X]++; mb(); mb(); ... count += __cpuhp_refcount[X]; if (state != BLOCK) ... return; mb(); __cpuhp_refcount[X]--;Either reader or writer should notice the STORE we care about.
If a reader can decrement
__cpuhp_refcount, we have 2 cases:1. It is the reader holding this lock. In this case we can't miss the corresponding
inc()done by this reader, because this reader didn't see BLOCK in the past.It is just the
A == B == 0
CPU_0CPU_1
----- -----
A = 1; B = 1;
mb();mb();
b = B; a = A;pattern, at least one CPU should see 1 in its a/b.
2. It is the reader which tries to take this lock and noticed state == BLOCK. We could miss the result of its
inc(), but we do not care, this reader is going to block.
Quoted 7 lines
_If_ the reader could migrate between inc/dec, then yes, we have a problem. Because that
dec()could make the result ofper_cpu_sum()= 0. IOW, we could missinc()but noticedec(). But given that it does this on the same CPU this is not possible.So why do we need
cpuhp_seq?
Good question, I will look again.
Thanx, Paul
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 12 lines · Paul E. McKenney
Quoted 7 lines · Oleg Nesterov
Quoted 3 lines · Oleg Nesterov
I tried hard to find any hole in this version but failed, I believe it is correct.
And I still believe it is. But now I am starting to think that we don't need
cpuhp_seq. (and imocpuhp_waitcount, but this is minor).Here is one scenario that I believe requires
cpuhp_seq:
Quoted 7 lines
1. Task 0 on CPU 0 increments its counter on entry.
2. Task 1 on CPU 1 starts summing the counters and gets to
CPU 4. The sum thus far is 1 (Task 0).3. Task 2 on CPU 2 increments its counter on entry.
Upon completing its entry code, it re-enables preemption.
afaics at this stage it should notice state = BLOCK and decrement the same counter on the same CPU before it does preempt_enable().
Because:
Quoted 11 lines
Quoted 11 lines
2. It is the reader which tries to take this lock and noticed state == BLOCK. We could miss the result of its
inc(), but we do not care, this reader is going to block._If_ the reader could migrate between inc/dec, then yes, we have a problem. Because that
dec()could make the result ofper_cpu_sum()= 0. IOW, we could missinc()but noticedec(). But given that it does this on the same CPU this is not possible.
Quoted 3 lines
Quoted 1 line
So why do we need
cpuhp_seq?Good question, I will look again.
Thanks! much appreciated.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Peter ZijlstraQuoted 6 lines · Peter Zijlstra
Quoted 2 lines · Paul E. McKenney
A couple of nits and some commentary, but if there are races, they are quite subtle. ;-)
*whee*..
Quoted 7 lines
I made one little change in the logic; I moved the waitcount increment to before the
__put_online_cpus()call, such that the writer will have to wait for us to wake up before trying again -- not for us to actually have acquired the read lock, for that we'd need to mess up__get_online_cpus()a bit more.Complete patch below.
OK, looks like Oleg is correct, the cpuhp_seq can be dispensed with.
I still don't see anything wrong with it, so time for a serious stress test on a large system. ;-)
Additional commentary interspersed.
Thanx, Paul
Quoted 32 lines
---
Subject: hotplug: Optimize {get,put}_online_cpus()
From: Peter Zijlstra <[email protected]>
Date: Tue Sep 17 16:17:11 CEST 2013The current implementation of
get_online_cpus()is global of nature and thus not suited for any kind of common usage.Re-implement the current recursive r/w cpu hotplug lock such that the read side locks are as light as possible.
The current cpu hotplug lock is entirely reader biased; but since readers are expensive there aren't a lot of them about and writer starvation isn't a particular problem.
However by making the reader side more usable there is a fair chance it will get used more and thus the starvation issue becomes a real possibility.
Therefore this new implementation is fair, alternating readers and writers; this however requires per-task state to allow the reader recursion.
Many comments are contributed by Paul McKenney, and many previous attempts were shown to be inadequate by both Paul and Oleg; many thanks to them for persisting to poke holes in my attempts.
Cc Oleg Nesterov <[email protected]>Cc Paul McKenney <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Steven Rostedt <[email protected]>Signed-off-by Peter Zijlstra <[email protected]>
Quoted 6 lines
--- include/linux/cpu.h | 58 +++++++++++++ include/linux/sched.h | 3 kernel/cpu.c | 209 +++++++++++++++++++++++++++++++++++--------------- kernel/sched/core.c | 2 4 files changed, 208 insertions(+), 64 deletions(-)
I stripped the removed lines to keep my eyes from going buggy.
Quoted 26 lines
include/linux/cpu.h +11
-0@@ -16,6 +16,7 @@#include <linux/node.h> #include <linux/compiler.h> #include <linux/cpumask.h> +#include <linux/percpu.h> struct device ;@@ -173,10 +174,61 @@ extern struct bus_type cpu_subsys;#ifdef CONFIG_HOTPLUG_CPU /* Stop CPUs going up and down. */ +extern void cpu_hotplug_init_task (struct task_struct *p); +extern void cpu_hotplug_begin (void );extern void cpu_hotplug_done (void ); + +extern int __cpuhp_state; +DECLARE_PER_CPU(unsigned int, __cpuhp_refcount); + +extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 198
Quoted 10 lines
include/linux/cpu.h +10
-0+might_sleep (); + +/* Support reader recursion */ +/* The value was >= 1 and remains so, reordering causes no harm. */ +if (current->cpuhp_ref++) +return ; + +preempt_disable (); +if (likely (!__cpuhp_state)) { +/* The barrier here is supplied by synchronize_sched(). */
I guess I shouldn't complain about the comment given where it came from, but...
A more accurate comment would say that we are in an RCU-sched read-side critical section, so the writer cannot both change __cpuhp_state from readers_fast and start checking counters while we are here. So if we see !__cpuhp_state, we know that the writer won't be checking until we past the preempt_enable() and that once the synchronize_sched() is done, the writer will see anything we did within this RCU-sched read-side critical section.
(The writer -can- change __cpuhp_state from readers_slow to readers_block while we are in this read-side critical section and then start summing counters, but that corresponds to a different "if" statement.)
Quoted 16 lines
include/linux/cpu.h +16
-0+__this_cpu_inc (__cpuhp_refcount); + }else { +__get_online_cpus ();/* Unconditional memory barrier. */ + } +preempt_enable (); +/* +* The barrier() from preempt_enable() prevents the compiler from +* bleeding the critical section out. +*/ +} + +extern void __put_online_cpus (void ); + +static inline void put_online_cpus (void ) +{ +/* The value was >= 1 and remains so, reordering causes no harm. */
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 224
Quoted 10 lines
include/linux/cpu.h +10
-0+if (--current->cpuhp_ref) +return ; + +/* +* The barrier() in preempt_disable() prevents the compiler from +* bleeding the critical section out. +*/ +preempt_disable (); +if (likely (!__cpuhp_state)) { +/* The barrier here is supplied by synchronize_sched(). */
Same here, both for the implied self-criticism and the more complete story.
Due to the basic RCU guarantee, the writer cannot both change __cpuhp_state and start checking counters while we are in this RCU-sched read-side critical section. And again, if the synchronize_sched() had to wait on us (or if we were early enough that no waiting was needed), then once the synchronize_sched() completes, the writer will see anything that we did within this RCU-sched read-side critical section.
Quoted 52 lines
include/linux/cpu.h +9
-0+__this_cpu_dec (__cpuhp_refcount); + }else { +__put_online_cpus ();/* Unconditional memory barrier. */ + } +preempt_enable (); +} +extern void cpu_hotplug_disable (void );extern void cpu_hotplug_enable (void );#define hotcpu_notifier (fn, pri) cpu_notifier(fn, pri)@@ -200,6 +252,8 @@ static inline void cpu_hotplug_driver_un#else /* CONFIG_HOTPLUG_CPU */ +static inline void cpu_hotplug_init_task (struct task_struct *p) {} +static inline void cpu_hotplug_begin (void ) {}static inline void cpu_hotplug_done (void ) {}#define get_online_cpus () do { } while (0)include/linux/sched.h +3
-0@@ -1454,6 +1454,9 @@ struct task_struct {unsigned int sequential_io;unsigned int sequential_io_avg;#endif +#ifdef CONFIG_HOTPLUG_CPU +int cpuhp_ref; +#endif };/* Future-safe accessor for struct task_struct's cpus_allowed. */ kernel/cpu.c +15
-0@@ -49,88 +49,173 @@ static int cpu_hotplug_disabled;#ifdef CONFIG_HOTPLUG_CPU +enum { readers_fast =0 , readers_slow, readers_block }; + +int __cpuhp_state; +EXPORT_SYMBOL_GPL (__cpuhp_state); + +DEFINE_PER_CPU(unsigned int, __cpuhp_refcount); +EXPORT_PER_CPU_SYMBOL_GPL (__cpuhp_refcount); + +static DEFINE_PER_CPU(unsigned int, cpuhp_seq); +static atomic_t cpuhp_waitcount; +static DECLARE_WAIT_QUEUE_HEAD (cpuhp_readers); +static DECLARE_WAIT_QUEUE_HEAD (cpuhp_writer); + +void cpu_hotplug_init_task (struct task_struct *p) +{
kernel/cpu.c static int cpu_hotplug_disabled; · line 76
Quoted 10 lines
kernel/cpu.c +10
-0+ p->cpuhp_ref =0 ; +} + +void __get_online_cpus (void ) +{ +again: +/* See __srcu_read_lock() */ +__this_cpu_inc (__cpuhp_refcount); +smp_mb ();/* A matches B, E */ +// __this_cpu_inc(cpuhp_seq);
Deleting the above per Oleg's suggestion. We still need the preceding memory barrier.
kernel/cpu.c static int cpu_hotplug_disabled; · line 84
Quoted 8 lines
kernel/cpu.c +8
-0+ +if (unlikely (__cpuhp_state == readers_block)) { +/* +* Make sure an outgoing writer sees the waitcount to ensure +* we make progress. +*/ +atomic_inc (&cpuhp_waitcount); +__put_online_cpus ();
The decrement happens on the same CPU as the increment, avoiding the increment-on-one-CPU-and-decrement-on-another problem.
And yes, if the reader misses the writer's assignment of readers_block to __cpuhp_state, then the writer is guaranteed to see the reader's increment. Conversely, any readers that increment their __cpuhp_refcount after the writer looks are guaranteed to see the readers_block value, which in turn means that they are guaranteed to immediately decrement their __cpuhp_refcount, so that it doesn't matter that the writer missed them.
Unfortunately, this trick does not apply back to SRCU, at least not without adding a second memory barrier to the srcu_read_lock() path (one to separate reading the index from incrementing the counter and another to separate incrementing the counter from the critical section. Can't have everything, I guess!
kernel/cpu.c static int cpu_hotplug_disabled; · line 95
Quoted 11 lines
kernel/cpu.c +11
-0+ +/* +* We either call schedule() in the wait, or we'll fall through +* and reschedule on the preempt_enable() in get_online_cpus(). +*/ +preempt_enable_no_resched (); +__wait_event (cpuhp_readers, __cpuhp_state != readers_block); +preempt_disable (); + +if (atomic_dec_and_test (&cpuhp_waitcount)) +wake_up_all (&cpuhp_writer);
I still don't see why this is a wake_up_all() given that there can be only one writer. Not that it makes much difference, but...
Quoted 27 lines
kernel/cpu.c +23
-0+ + goto again; + } +} +EXPORT_SYMBOL_GPL (__get_online_cpus); +void __put_online_cpus (void ) { +/* See __srcu_read_unlock() */ +smp_mb ();/* C matches D */ +/* +* In other words, if they see our decrement (presumably to aggregate +* zero, as that is the only time it matters) they will also see our +* critical section. +*/ +this_cpu_dec (__cpuhp_refcount); +/* Prod writer to recheck readers_active */ +wake_up_all (&cpuhp_writer); } +EXPORT_SYMBOL_GPL (__put_online_cpus); + +#define per_cpu_sum (var) \ +({ \ + typeof(var) __sum = 0; \ + int cpu; \ + for_each_possible_cpu(cpu) \
kernel/cpu.c static int cpu_hotplug_disabled;
Quoted 10 lines
kernel/cpu.c +8
-0+ __sum +=per_cpu (var, cpu); \ + __sum; \ +)} +/* +* See srcu_readers_active_idx_check() for a rather more detailed explanation. +*/ +static bool cpuhp_readers_active_check (void ) { +// unsigned int seq = per_cpu_sum(cpuhp_seq);
Delete the above per Oleg's suggestion.
Quoted 11 lines
kernel/cpu.c +9
-0+ +smp_mb ();/* B matches A */ + +/* +* In other words, if we see __get_online_cpus() cpuhp_seq increment, +* we are guaranteed to also see its __cpuhp_refcount increment. +*/ +if (per_cpu_sum (__cpuhp_refcount) !=0 ) +return false;
kernel/cpu.c static int cpu_hotplug_disabled;
Quoted 10 lines
kernel/cpu.c +9
-0+smp_mb ();/* D matches C */ +/* +* On equality, we know that there could not be any "sneak path" pairs +* where we see a decrement but not the corresponding increment for a +* given reader. If we saw its decrement, the memory barriers guarantee +* that we now see its cpuhp_seq increment. +*/ + +// return per_cpu_sum(cpuhp_seq) == seq;
Delete the above per Oleg's suggestion, but actually need to replace with "return true;". We should be able to get rid of the first memory barrier (B matches A) because the smp_mb() in cpu_hotplug_begin() covers it, but we cannot git rid of the second memory barrier (D matches C).
Quoted 37 lines
kernel/cpu.c +30
-0}/* +* This will notify new readers to block and wait for all active readers to +* complete. */ void cpu_hotplug_begin (void ) { +/* +* Since cpu_hotplug_begin() is always called after invoking +* cpu_maps_update_begin(), we can be sure that only one writer is +* active. +*/ +lockdep_assert_held (&cpu_add_remove_lock); +/* Allow reader-in-writer recursion. */ + current->cpuhp_ref++; + +/* Notify readers to take the slow path. */ + __cpuhp_state = readers_slow; + +/* See percpu_down_write(); guarantees all readers take the slow path */ +synchronize_sched (); + +/* +* Notify new readers to block; up until now, and thus throughout the +* longish synchronize_sched() above, new readers could still come in. +*/ + __cpuhp_state = readers_block; + +smp_mb ();/* E matches A */ + +/* +* If they don't see our writer of readers_block to __cpuhp_state, +* then we are guaranteed to see their __cpuhp_refcount increment, and +* therefore will wait for them. +*/
kernel/cpu.c static int cpu_hotplug_disabled; · line 200
Quoted 10 lines
kernel/cpu.c +6
-0+ +/* Wait for all now active readers to complete. */ +wait_event (cpuhp_writer,cpuhp_readers_active_check ()); }void cpu_hotplug_done (void ) { +/* Signal the writer is done, no fast path yet. */ + __cpuhp_state = readers_slow; +wake_up_all (&cpuhp_readers);
And one reason that we cannot just immediately flip to readers_fast is that new readers might fail to see the results of this writer's critical section.
Quoted 33 lines
kernel/cpu.c +18
-0+ +/* +* The wait_event()/wake_up_all() prevents the race where the readers +* are delayed between fetching __cpuhp_state and blocking. +*/ + +/* See percpu_up_write(); readers will no longer attempt to block. */ +synchronize_sched (); + +/* Let 'em rip */ + __cpuhp_state = readers_fast; + current->cpuhp_ref--; + +/* +* Wait for any pending readers to be running. This ensures readers +* after writer and avoids writers starving readers. +*/ +wait_event (cpuhp_writer, !atomic_read (&cpuhp_waitcount)); }/* kernel/sched/core.c +2
-0@@ -1736,6 +1736,8 @@ static void __sched_fork(unsigned long cINIT_LIST_HEAD (&p->numa_entry); p->numa_group =NULL ;#endif /* CONFIG_NUMA_BALANCING */ + +cpu_hotplug_init_task (p); }#ifdef CONFIG_NUMA_BALANCING
Steven Rostedt
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Peter Zijlstrainclude/linux/cpu.h extern struct bus_type cpu_subsys; · line 195
Quoted 13 lines · On Thu, 19 Sep 2013 16:32:41 +0200 Peter Zijlstra <[email protected]> wrote:
include/linux/cpu.h +13
-0+extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{ +might_sleep (); + +preempt_disable (); +if (likely (!__cpuhp_writer || __cpuhp_writer == current)) +this_cpu_inc (__cpuhp_refcount); +else +__get_online_cpus (); +preempt_enable (); +}
This isn't much different than srcu_read_lock(). What about doing something like this:
static inline void get_online_cpus(void)
{
might_sleep();
srcu_read_lock(&cpuhp_srcu);
if (unlikely(__cpuhp_writer || __cpuhp_writer != current)) {
srcu_read_unlock(&cpuhp_srcu);
__get_online_cpus();
current->online_cpus_held++;
}
}static inline void put_online_cpus(void)
{
if (unlikely(current->online_cpus_held)) {
current->online_cpus_held--;
__put_online_cpus();
return;
}
srcu_read_unlock(&cpuhp_srcu);
}Then have the writer simply do:
__cpuhp_write = current; synchronize_srcu(&cpuhp_srcu); <grab the mutex here>
-- Steve
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 23 lines · Steven Rostedt
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 195
Quoted 13 lines · On Thu, 19 Sep 2013 16:32:41 +0200 Peter Zijlstra <[email protected]> wrote:
include/linux/cpu.h +13
-0+extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{ +might_sleep (); + +preempt_disable (); +if (likely (!__cpuhp_writer || __cpuhp_writer == current)) +this_cpu_inc (__cpuhp_refcount); +else +__get_online_cpus (); +preempt_enable (); +}This isn't much different than
srcu_read_lock(). What about doing something like this:static inline void
get_online_cpus(void)
Quoted 10 lines
{
might_sleep();
srcu_read_lock(&cpuhp_srcu);
if (unlikely(__cpuhp_writer || __cpuhp_writer != current)) {
srcu_read_unlock(&cpuhp_srcu);
__get_online_cpus();
current->online_cpus_held++;
}
}There's a full memory barrier in srcu_read_lock(), while there was no such thing in the previous fast path.
Also, why current->online_cpus_held()? That would make the write side O(nr_tasks) instead of O(nr_cpus).
Quoted 10 lines
static inline void
put_online_cpus(void){ if (unlikely(current->online_cpus_held)) { current->online_cpus_held--; __put_online_cpus(); return; } srcu_read_unlock(&cpuhp_srcu); }
Also, you might not have noticed but, srcu_read_{,un}lock() have an extra idx thing to pass about. That doesn't fit with the hotplug api.
Quoted 7 lines
Then have the writer simply do:
__cpuhp_write = current; synchronize_srcu(&cpuhp_srcu); <grab the mutex here>
How does that do reader preference?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Steven Rostedt
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 35 lines · On Mon, 23 Sep 2013 16:54:46 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 23 lines · Steven Rostedt
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 195
Quoted 13 lines · On Thu, 19 Sep 2013 16:32:41 +0200 Peter Zijlstra <[email protected]> wrote:
include/linux/cpu.h +13
-0+extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{ +might_sleep (); + +preempt_disable (); +if (likely (!__cpuhp_writer || __cpuhp_writer == current)) +this_cpu_inc (__cpuhp_refcount); +else +__get_online_cpus (); +preempt_enable (); +}This isn't much different than
srcu_read_lock(). What about doing something like this:static inline void
get_online_cpus(void)Quoted 10 lines
{ might_sleep(); srcu_read_lock(&cpuhp_srcu); if (unlikely(__cpuhp_writer || __cpuhp_writer != current)) { srcu_read_unlock(&cpuhp_srcu); __get_online_cpus(); current->online_cpus_held++; } }
Quoted 2 lines
There's a full memory barrier in
srcu_read_lock(), while there was no such thing in the previous fast path.
Yeah, I mentioned this to Paul, and we talked about making srcu_read_lock() work with no mb's. But currently, doesn't get_online_cpus() just take a mutex? What's wrong with a mb() as it still kicks ass over what is currently there today?
Quoted 3 lines
Also, why
current->online_cpus_held()? That would make the write sideO(nr_tasks)instead ofO(nr_cpus).
?? I'm not sure I understand this. The online_cpus_held++ was there for recursion. Can't get_online_cpus() nest? I was thinking it can. If so, once the "__cpuhp_writer" is set, we need to do __put_online_cpus() as many times as we did a __get_online_cpus(). I don't know where the O(nr_tasks) comes from. The ref here was just to account for doing the old "get_online_cpus" instead of a srcu_read_lock().
Quoted 9 lines
Quoted 9 lines
static inline void
put_online_cpus(void){ if (unlikely(current->online_cpus_held)) { current->online_cpus_held--; __put_online_cpus(); return; }
Quoted 5 lines
Quoted 2 lines
srcu_read_unlock(&cpuhp_srcu); }Also, you might not have noticed but, srcu_read_{,un}lock() have an extra idx thing to pass about. That doesn't fit with the hotplug api.
I'll have to look a that, as I'm not exactly sure about the idx thing.
Quoted 10 lines
Quoted 8 lines
Then have the writer simply do:
__cpuhp_write = current; synchronize_srcu(&cpuhp_srcu); <grab the mutex here>How does that do reader preference?
Well, the point I was trying to do was to let readers go very fast (well, with a mb instead of a mutex), and then when the CPU hotplug happens, it goes back to the current method.
That is, once we set __cpuhp_write, and then run synchronize_srcu(), the system will be in a state that does what it does today (grabbing mutexes, and upping refcounts).
I thought the whole point was to speed up the get_online_cpus() when no hotplug is happening. This does that, and is rather simple. It only gets slow when hotplug is in effect.
-- Steve
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 3 lines · Steven Rostedt
Well, the point I was trying to do was to let readers go very fast (well, with a mb instead of a mutex), and then when the CPU hotplug happens, it goes back to the current method.
Well, for that the thing Oleg proposed works just fine and the preempt_disable() section vs synchronize_sched() is hardly magic.
But I'd really like to get the writer pending case fast too.
Quoted 3 lines
That is, once we set
__cpuhp_write, and then runsynchronize_srcu(), the system will be in a state that does what it does today (grabbing mutexes, and upping refcounts).
Still no point in using srcu for this; preempt_disable + synchronize_sched() is similar and much faster -- its the rcu_sched equivalent of what you propose.
Quoted 3 lines
I thought the whole point was to speed up the
get_online_cpus()when no hotplug is happening. This does that, and is rather simple. It only gets slow when hotplug is in effect.
No, well, it also gets slow when a hotplug is pending, which can be quite a while if we go sprinkle get_online_cpus() all over the place and the machine is busy.
One we start a hotplug attempt we must wait for all readers to quiesce -- since the lock is full reader preference this can take an infinite amount of time -- while we're waiting for this all 4k+ CPUs will be bouncing the one mutex around on every get_online_cpus(); of which we'll have many since that's the entire point of making them cheap, to use more of them.
Steven Rostedt
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 3 lines · On Mon, 23 Sep 2013 17:22:23 +0200 Peter Zijlstra <[email protected]> wrote:
Still no point in using srcu for this;
preempt_disable+synchronize_sched()is similar and much faster -- its thercu_schedequivalent of what you propose.
To be honest, I sent this out last week and it somehow got trashed by my laptop and connecting to my smtp server. Where the last version of your patch still had the memory barrier ;-)
So yeah, a true synchronize_sched() is better.
-- Steve
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 10 lines · Steven Rostedt
Quoted 3 lines · On Mon, 23 Sep 2013 17:22:23 +0200 Peter Zijlstra <[email protected]> wrote:
Still no point in using srcu for this;
preempt_disable+synchronize_sched()is similar and much faster -- its thercu_schedequivalent of what you propose.To be honest, I sent this out last week and it somehow got trashed by my laptop and connecting to my smtp server. Where the last version of your patch still had the memory barrier ;-)
Ah, ok, yes in that case things start to make sense again.
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Steven RostedtQuoted 4 lines · Steven Rostedt
Quoted 1 line · On Mon, 23 Sep 2013 16:54:46 +0200 Peter Zijlstra <[email protected]> wrote:
On Mon, Sep 23, 2013 at 10:50:17AM -0400, Steven Rostedt wrote:
[ . . . ]
Quoted 16 lines
?? I'm not sure I understand this. The online_cpus_held++ was there for recursion. Can't
get_online_cpus()nest? I was thinking it can. If so, once the "__cpuhp_writer" is set, we need to do__put_online_cpus()as many times as we did a__get_online_cpus(). I don't know where theO(nr_tasks)comes from. The ref here was just to account for doing the old "get_online_cpus" instead of asrcu_read_lock().Quoted 9 lines
Quoted 9 lines
static inline void
put_online_cpus(void){ if (unlikely(current->online_cpus_held)) { current->online_cpus_held--; __put_online_cpus(); return; }
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
srcu_read_unlock(&cpuhp_srcu); }Also, you might not have noticed but, srcu_read_{,un}lock() have an extra idx thing to pass about. That doesn't fit with the hotplug api.
I'll have to look a that, as I'm not exactly sure about the idx thing.
Not a problem, just stuff the idx into some per-task thing. Either task_struct or taskinfo will work fine.
Quoted 14 lines
Quoted 9 lines
Quoted 7 lines
Then have the writer simply do:
__cpuhp_write = current; synchronize_srcu(&cpuhp_srcu); <grab the mutex here>How does that do reader preference?
Well, the point I was trying to do was to let readers go very fast (well, with a mb instead of a mutex), and then when the CPU hotplug happens, it goes back to the current method.
Quoted 7 lines
That is, once we set
__cpuhp_write, and then runsynchronize_srcu(), the system will be in a state that does what it does today (grabbing mutexes, and upping refcounts).I thought the whole point was to speed up the
get_online_cpus()when no hotplug is happening. This does that, and is rather simple. It only gets slow when hotplug is in effect.
Or to put it another way, if the underlying slow-path mutex is reader-preference, then the whole thing will be reader-preference.
Thanx, Paul
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 2 lines · Paul E. McKenney
Not a problem, just stuff the idx into some per-task thing. Either
task_structor taskinfo will work fine.
Still not seeing the point of using srcu though..
srcu_read_lock() vs synchronize_srcu() is the same but far more expensive than preempt_disable() vs synchronize_sched().
Quoted 2 lines
Or to put it another way, if the underlying slow-path mutex is reader-preference, then the whole thing will be reader-preference.
Right, so 1) we have no such mutex so we're going to have to open-code that anyway, and 2) like I just explained in the other email, I want the pending writer case to be _fast_ as well.
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines · Paul E. McKenney
Not a problem, just stuff the idx into some per-task thing. Either
task_structor taskinfo will work fine.Still not seeing the point of using srcu though..
srcu_read_lock()vssynchronize_srcu()is the same but far more expensive thanpreempt_disable()vssynchronize_sched().
Heh! You want the old-style SRCU. ;-)
Quoted 6 lines
Quoted 2 lines
Or to put it another way, if the underlying slow-path mutex is reader-preference, then the whole thing will be reader-preference.
Right, so 1) we have no such mutex so we're going to have to open-code that anyway, and 2) like I just explained in the other email, I want the pending writer case to be _fast_ as well.
At some point I suspect that we will want some form of fairness, but in the meantime, good point.
Thanx, Paul
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 2 lines · Paul E. McKenney
At some point I suspect that we will want some form of fairness, but in the meantime, good point.
I figured we could start a timer on hotplug to force quiesce the readers after about 10 minutes or so ;-)
Should be a proper discouragement from (ab)using this hotplug stuff...
Muwhahaha
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Peter ZijlstraAnd somehow I didn't notice that cpuhp_set_state() doesn't look right,
Quoted 13 lines · Peter Zijlstra
void
cpu_hotplug_begin(void)
{kernel/cpu.c +1
-9- cpu_hotplug.active_writer = current; +lockdep_assert_held (&cpu_add_remove_lock); -for (;;) { -mutex_lock (&cpu_hotplug.lock); -if (likely (!cpu_hotplug.refcount)) -break ; -__set_current_state (TASK_UNINTERRUPTIBLE ); -mutex_unlock (&cpu_hotplug.lock); -schedule (); - }
kernel/cpu.c static int cpu_hotplug_disabled; · line 141
Quoted 10 lines
kernel/cpu.c +10
-0+ __cpuhp_writer = current; + +/* After this everybody will observe _writer and take the slow path. */ +synchronize_sched (); + +/* Wait for no readers -- reader preference */ +cpuhp_wait_refcount (); + +/* Stop new readers. */ +cpuhp_set_state (1 );
But this stops all readers, not only new. Even if cpuhp_wait_refcount() was correct, a new reader can come right before cpuhp_set_state(1) and then it can call another recursive get_online_cpus() right after.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
OK, so another attempt.
This one is actually fair in that it immediately forces a reader quiescent state by explicitly implementing reader-reader recursion.
This does away with the potentially long pending writer case and can thus use the simpler global state.
I don't really like this lock being fair, but alas.
Also, please have a look at the atomic_dec_and_test(cpuhp_waitcount) and cpu_hotplug_done(). I think its ok, but I keep confusing myself.
---
include/linux/cpu.h +44 -2
@@ -16,6 +16,7 @@#include <linux/node.h> #include <linux/compiler.h> #include <linux/cpumask.h> +#include <linux/percpu.h> struct device ;
@@ -173,10 +174,49 @@ extern struct bus_type cpu_subsys;#ifdef CONFIG_HOTPLUG_CPU /* Stop CPUs going up and down. */ +extern void cpu_hotplug_init_task (struct task_struct *p); +extern void cpu_hotplug_begin (void );extern void cpu_hotplug_done (void ); -extern void get_online_cpus (void ); -extern void put_online_cpus (void ); + +extern int __cpuhp_writer; +DECLARE_PER_CPU(unsigned int, __cpuhp_refcount); + +extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{ +might_sleep (); + +if (current->cpuhp_ref++) { +barrier (); +return ; + } + +preempt_disable (); +if (likely (!__cpuhp_writer)) +__this_cpu_inc (__cpuhp_refcount); +else +__get_online_cpus (); +preempt_enable (); +} + +extern void __put_online_cpus (void ); + +static inline void put_online_cpus (void ) +{ +barrier (); +if (--current->cpuhp_ref) +return ; + +preempt_disable (); +if (likely (!__cpuhp_writer)) +__this_cpu_dec (__cpuhp_refcount); +else +__put_online_cpus (); +preempt_enable (); +} +extern void cpu_hotplug_disable (void );extern void cpu_hotplug_enable (void );#define hotcpu_notifier (fn, pri) cpu_notifier(fn, pri)
@@ -200,6 +240,8 @@ static inline void cpu_hotplug_driver_un#else /* CONFIG_HOTPLUG_CPU */ +static inline void cpu_hotplug_init_task (struct task_struct *p) {} +static inline void cpu_hotplug_begin (void ) {}static inline void cpu_hotplug_done (void ) {}#define get_online_cpus () do { } while (0)
include/linux/sched.h +3 -0
@@ -1454,6 +1454,9 @@ struct task_struct {unsigned int sequential_io;unsigned int sequential_io_avg;#endif +#ifdef CONFIG_HOTPLUG_CPU +int cpuhp_ref; +#endif };/* Future-safe accessor for struct task_struct's cpus_allowed. */
kernel/cpu.c +83 -56
171 lines not on this page · raw message
kernel/sched/core.c +2 -0
@@ -1736,6 +1736,8 @@ static void __sched_fork(unsigned long cINIT_LIST_HEAD (&p->numa_entry); p->numa_group =NULL ;#endif /* CONFIG_NUMA_BALANCING */ + +cpu_hotplug_init_task (p); }#ifdef CONFIG_NUMA_BALANCING
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 6 lines · Peter Zijlstra
OK, so another attempt.
This one is actually fair in that it immediately forces a reader quiescent state by explicitly implementing reader-reader recursion.
Quoted 7 lines
This does away with the potentially long pending writer case and can thus use the simpler global state.
I don't really like this lock being fair, but alas.
Also, please have a look at the
atomic_dec_and_test(cpuhp_waitcount)andcpu_hotplug_done(). I think its ok, but I keep confusing myself.
Cute!
Some commentary below. Also one question about how a race leading to a NULL-pointer dereference is avoided.
Thanx, Paul
Quoted 133 lines
---
include/linux/cpu.h +44
-2@@ -16,6 +16,7 @@#include <linux/node.h> #include <linux/compiler.h> #include <linux/cpumask.h> +#include <linux/percpu.h> struct device ;@@ -173,10 +174,49 @@ extern struct bus_type cpu_subsys;#ifdef CONFIG_HOTPLUG_CPU /* Stop CPUs going up and down. */ +extern void cpu_hotplug_init_task (struct task_struct *p); +extern void cpu_hotplug_begin (void );extern void cpu_hotplug_done (void ); -extern void get_online_cpus (void ); -extern void put_online_cpus (void ); + +extern int __cpuhp_writer; +DECLARE_PER_CPU(unsigned int, __cpuhp_refcount); + +extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{ +might_sleep (); + +if (current->cpuhp_ref++) { +barrier (); +return ; + } + +preempt_disable (); +if (likely (!__cpuhp_writer)) +__this_cpu_inc (__cpuhp_refcount); +else +__get_online_cpus (); +preempt_enable (); +} + +extern void __put_online_cpus (void ); + +static inline void put_online_cpus (void ) +{ +barrier (); +if (--current->cpuhp_ref) +return ; + +preempt_disable (); +if (likely (!__cpuhp_writer)) +__this_cpu_dec (__cpuhp_refcount); +else +__put_online_cpus (); +preempt_enable (); +} +extern void cpu_hotplug_disable (void );extern void cpu_hotplug_enable (void );#define hotcpu_notifier (fn, pri) cpu_notifier(fn, pri)@@ -200,6 +240,8 @@ static inline void cpu_hotplug_driver_un#else /* CONFIG_HOTPLUG_CPU */ +static inline void cpu_hotplug_init_task (struct task_struct *p) {} +static inline void cpu_hotplug_begin (void ) {}static inline void cpu_hotplug_done (void ) {}#define get_online_cpus () do { } while (0)include/linux/sched.h +3
-0@@ -1454,6 +1454,9 @@ struct task_struct {unsigned int sequential_io;unsigned int sequential_io_avg;#endif +#ifdef CONFIG_HOTPLUG_CPU +int cpuhp_ref; +#endif };/* Future-safe accessor for struct task_struct's cpus_allowed. */ kernel/cpu.c +16
-23@@ -49,88 +49,115 @@ static int cpu_hotplug_disabled;#ifdef CONFIG_HOTPLUG_CPU -static struct { -struct task_struct *active_writer; -struct mutex lock;/* Synchronizes accesses to refcount, */ -/* -* Also blocks the new readers during -* an ongoing cpu hotplug operation. -*/ -int refcount; -} cpu_hotplug = { - .active_writer =NULL , - .lock =__MUTEX_INITIALIZER (cpu_hotplug.lock), - .refcount =0 , -}; +static struct task_struct *cpuhp_writer_task =NULL ; -void get_online_cpus (void ) -{ -might_sleep (); -if (cpu_hotplug.active_writer == current) -return ; -mutex_lock (&cpu_hotplug.lock); - cpu_hotplug.refcount++; -mutex_unlock (&cpu_hotplug.lock); +int __cpuhp_writer; +EXPORT_SYMBOL_GPL (__cpuhp_writer); +DEFINE_PER_CPU(unsigned int, __cpuhp_refcount); +EXPORT_PER_CPU_SYMBOL_GPL (__cpuhp_refcount); + +static atomic_t cpuhp_waitcount; +static atomic_t cpuhp_slowcount; +static DECLARE_WAIT_QUEUE_HEAD (cpuhp_wq); + +void cpu_hotplug_init_task (struct task_struct *p) +{ + p->cpuhp_ref =0 ; } -EXPORT_SYMBOL_GPL (get_online_cpus); -void put_online_cpus (void ) +#define cpuhp_writer_wake () \ + wake_up_process(cpuhp_writer_task) +
kernel/cpu.c static int cpu_hotplug_disabled; · line 81
Quoted 10 lines
kernel/cpu.c +10
-0+#define cpuhp_writer_wait (cond) \ +do { \ + for (;;) { \ + set_current_state(TASK_UNINTERRUPTIBLE); \ + if (cond) \ + break; \ + schedule(); \ + } \ + __set_current_state(TASK_RUNNING); \ +} while (0)
Why not wait_event()? Presumably the above is a bit lighter weight, but is that even something that can be measured?
Quoted 7 lines
kernel/cpu.c +2
-2+void __get_online_cpus (void ) { -if (cpu_hotplug.active_writer == current) +if (cpuhp_writer_task == current)return ; -mutex_lock (&cpu_hotplug.lock);
kernel/cpu.c static int cpu_hotplug_disabled; · line 95
Quoted 10 lines
kernel/cpu.c +8
-2-if (WARN_ON (!cpu_hotplug.refcount)) - cpu_hotplug.refcount++;/* try to fix things up */ +atomic_inc (&cpuhp_waitcount); + +/* +* We either call schedule() in the wait, or we'll fall through +* and reschedule on the preempt_enable() in get_online_cpus(). +*/ +preempt_enable_no_resched (); +wait_event (cpuhp_wq, !__cpuhp_writer);
Finally! A good use for preempt_enable_no_resched(). ;-)
kernel/cpu.c static int cpu_hotplug_disabled; · line 106
Quoted 11 lines
kernel/cpu.c +11
-0+preempt_disable (); + +/* +* It would be possible for cpu_hotplug_done() to complete before +* the atomic_inc() above; in which case there is no writer waiting +* and doing a wakeup would be BAD (tm). +* +* If however we still observe cpuhp_writer_task here we know +* cpu_hotplug_done() is currently stuck waiting for cpuhp_waitcount. +*/ +if (atomic_dec_and_test (&cpuhp_waitcount) && cpuhp_writer_task)
OK, I'll bite... What sequence of events results in the atomic_dec_and_test() returning true but there being no cpuhp_writer_task?
Ah, I see it...
o Task A becomes the writer.
o Task B tries to read, but stalls for whatever reason before
the atomic_inc().
o Task A completes its write-side operation. It sees no readers
blocked, so goes on its merry way.
o Task B does its atomic_inc(), does its read, then sees
atomic_dec_and_test() return zero, but cpuhp_writer_task is NULL, so it doesn't do the wakeup.
But what prevents the following sequence of events?
o Task A becomes the writer.
o Task B tries to read, but stalls for whatever reason before
the atomic_inc().
o Task A completes its write-side operation. It sees no readers
blocked, so goes on its merry way, but is delayed before it NULLs cpuhp_writer_task.
o Task B does its atomic_inc(), does its read, then sees
atomic_dec_and_test() return zero. However, it sees cpuhp_writer_task as non-NULL.
o Then Task A NULLs cpuhp_writer_task.
o Task B's call to cpuhp_writer_wake() sees a NULL pointer.
Quoted 47 lines
kernel/cpu.c +14
-20+cpuhp_writer_wake (); +} +EXPORT_SYMBOL_GPL (__get_online_cpus); -if (!--cpu_hotplug.refcount &&unlikely (cpu_hotplug.active_writer)) -wake_up_process (cpu_hotplug.active_writer); -mutex_unlock (&cpu_hotplug.lock); +void __put_online_cpus (void ) +{ +if (cpuhp_writer_task == current) +return ; +if (atomic_dec_and_test (&cpuhp_slowcount)) +cpuhp_writer_wake (); } -EXPORT_SYMBOL_GPL (put_online_cpus); +EXPORT_SYMBOL_GPL (__put_online_cpus);/* * This ensures that the hotplug operation can begin only when the * refcount goes to zero. * -* Note that during a cpu-hotplug operation, the new readers, if any, -* will be blocked by the cpu_hotplug.lock -* * Since cpu_hotplug_begin() is always called after invoking * cpu_maps_update_begin(), we can be sure that only one writer is active. -* -* Note that theoretically, there is a possibility of a livelock: -* - Refcount goes to zero, last reader wakes up the sleeping -* writer. -* - Last reader unlocks the cpu_hotplug.lock. -* - A new reader arrives at this moment, bumps up the refcount. -* - The writer acquires the cpu_hotplug.lock finds the refcount -* non zero and goes to sleep again. -* -* However, this is very difficult to achieve in practice since -* get_online_cpus() not an api which is called all that often. -* */ void cpu_hotplug_begin (void ) { - cpu_hotplug.active_writer = current; +unsigned int count =0 ; +int cpu; + +lockdep_assert_held (&cpu_add_remove_lock);
kernel/cpu.c static int cpu_hotplug_disabled; · line 136
Quoted 10 lines
kernel/cpu.c +2
-7-for (;;) { -mutex_lock (&cpu_hotplug.lock); -if (likely (!cpu_hotplug.refcount)) -break ; -__set_current_state (TASK_UNINTERRUPTIBLE ); -mutex_unlock (&cpu_hotplug.lock); -schedule (); + __cpuhp_writer =1 ; + cpuhp_writer_task = current;
At this point, the value of cpuhp_slowcount can go negative. Can't see that this causes a problem, given the atomic_add() below.
kernel/cpu.c static int cpu_hotplug_disabled; · line 145
Quoted 9 lines
kernel/cpu.c +8
-0+ +/* After this everybody will observe writer and take the slow path. */ +synchronize_sched (); + +/* Collapse the per-cpu refcount into slowcount */ +for_each_possible_cpu (cpu) { + count +=per_cpu (__cpuhp_refcount, cpu); +per_cpu (__cpuhp_refcount, cpu) =0 ; }
The above is safe because the readers are no longer changing their __cpuhp_refcount values.
Quoted 32 lines
kernel/cpu.c +11
-2+atomic_add (count, &cpuhp_slowcount); + +/* Wait for all readers to go away */ +cpuhp_writer_wait (!atomic_read (&cpuhp_slowcount)); }void cpu_hotplug_done (void ) { - cpu_hotplug.active_writer =NULL ; -mutex_unlock (&cpu_hotplug.lock); +/* Signal the writer is done */ + cpuhp_writer =0 ; +wake_up_all (&cpuhp_wq); + +/* Wait for any pending readers to be running */ +cpuhp_writer_wait (!atomic_read (&cpuhp_waitcount)); + cpuhp_writer_task =NULL ; }/* kernel/sched/core.c +2
-0@@ -1736,6 +1736,8 @@ static void __sched_fork(unsigned long cINIT_LIST_HEAD (&p->numa_entry); p->numa_group =NULL ;#endif /* CONFIG_NUMA_BALANCING */ + +cpu_hotplug_init_task (p); }#ifdef CONFIG_NUMA_BALANCING
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 8 lines · Paul E. McKenney
Quoted 8 lines
(quoted diff) +8
-0+#define cpuhp_writer_wake() \ + wake_up_process(cpuhp_writer_task) + +#define cpuhp_writer_wait(cond) \ +do { \ + for (;;) { \ + set_current_state(TASK_UNINTERRUPTIBLE); \ + if (cond) \
Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ break; \ + schedule(); \ + } \ + __set_current_state(TASK_RUNNING); \ +} while (0)Why not
wait_event()? Presumably the above is a bit lighter weight, but is that even something that can be measured?
I didn't want to mix readers and writers on cpuhp_wq, and I suppose I could create a second waitqueue; that might also be a better solution for the NULL thing below.
Quoted 10 lines
Review
Quoted 8 lines
(quoted diff) +8
-0+ atomic_inc(&cpuhp_waitcount); + + /* + * We either call schedule() in the wait, or we'll fall through + * and reschedule on the preempt_enable() in get_online_cpus(). + */ + preempt_enable_no_resched(); + wait_event(cpuhp_wq, !__cpuhp_writer);Finally! A good use for
preempt_enable_no_resched(). ;-)
Hehe, there were a few others, but tglx removed most with the schedule_preempt_disabled() primitive.
In fact, I considered a wait_event_preempt_disabled() but was too lazy. That whole wait_event macro fest looks like it could use an iteration or two of collapse anyhow.
Quoted 16 lines
Review
Quoted 11 lines
(quoted diff) +11
-0+ preempt_disable(); + + /* + * It would be possible for cpu_hotplug_done() to complete before + * the atomic_inc() above; in which case there is no writer waiting + * and doing a wakeup would be BAD (tm). + * + * If however we still observe cpuhp_writer_task here we know + * cpu_hotplug_done() is currently stuck waiting for cpuhp_waitcount. + */ + if (atomic_dec_and_test(&cpuhp_waitcount) && cpuhp_writer_task)OK, I'll bite... What sequence of events results in the
atomic_dec_and_test()returning true but there being nocpuhp_writer_task?
Quoted 1 line
Ah, I see it...
<snip>
Indeed, and
Quoted 1 line
But what prevents the following sequence of events?
<snip>
Quoted 1 line
o Task B's call to
cpuhp_writer_wake()sees a NULL pointer.
Quite so.. nothing. See there was a reason I kept being confused about it.
Quoted 7 lines
Quoted 7 lines
void
cpu_hotplug_begin(void)
{(quoted diff) +4
-0+ unsigned int count = 0; + int cpu; + + lockdep_assert_held(&cpu_add_remove_lock);
Quoted 5 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ __cpuhp_writer = 1; + cpuhp_writer_task = current;At this point, the value of
cpuhp_slowcountcan go negative. Can't see that this causes a problem, given theatomic_add()below.
Agreed.
Quoted 4 lines
Quoted 4 lines
(quoted diff) +4
-0+ + /* After this everybody will observe writer and take the slow path. */ + synchronize_sched(); +
Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +4
-0+ /* Collapse the per-cpu refcount into slowcount */ + for_each_possible_cpu(cpu) { + count += per_cpu(__cpuhp_refcount, cpu); + per_cpu(__cpuhp_refcount, cpu) = 0; }The above is safe because the readers are no longer changing their
__cpuhp_refcountvalues.
Yes, I'll expand the comment.
So how about something like this?
---
include/linux/cpu.h +45 -2
@@ -16,6 +16,7 @@#include <linux/node.h> #include <linux/compiler.h> #include <linux/cpumask.h> +#include <linux/percpu.h> struct device ;
@@ -173,10 +174,50 @@ extern struct bus_type cpu_subsys;#ifdef CONFIG_HOTPLUG_CPU /* Stop CPUs going up and down. */ +extern void cpu_hotplug_init_task (struct task_struct *p); +extern void cpu_hotplug_begin (void );extern void cpu_hotplug_done (void ); -extern void get_online_cpus (void ); -extern void put_online_cpus (void ); + +extern struct task_struct *__cpuhp_writer; +DECLARE_PER_CPU(unsigned int, __cpuhp_refcount); + +extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{ +might_sleep (); + +/* Support reader-in-reader recursion */ +if (current->cpuhp_ref++) { +barrier (); +return ; + } + +preempt_disable (); +if (likely (!__cpuhp_writer)) +__this_cpu_inc (__cpuhp_refcount); +else +__get_online_cpus (); +preempt_enable (); +} + +extern void __put_online_cpus (void ); + +static inline void put_online_cpus (void ) +{ +barrier (); +if (--current->cpuhp_ref) +return ; + +preempt_disable (); +if (likely (!__cpuhp_writer)) +__this_cpu_dec (__cpuhp_refcount); +else +__put_online_cpus (); +preempt_enable (); +} +extern void cpu_hotplug_disable (void );extern void cpu_hotplug_enable (void );#define hotcpu_notifier (fn, pri) cpu_notifier(fn, pri)
@@ -200,6 +241,8 @@ static inline void cpu_hotplug_driver_un#else /* CONFIG_HOTPLUG_CPU */ +static inline void cpu_hotplug_init_task (struct task_struct *p) {} +static inline void cpu_hotplug_begin (void ) {}static inline void cpu_hotplug_done (void ) {}#define get_online_cpus () do { } while (0)
include/linux/sched.h +3 -0
@@ -1454,6 +1454,9 @@ struct task_struct {unsigned int sequential_io;unsigned int sequential_io_avg;#endif +#ifdef CONFIG_HOTPLUG_CPU +int cpuhp_ref; +#endif };/* Future-safe accessor for struct task_struct's cpus_allowed. */
kernel/cpu.c +68 -56
156 lines not on this page · raw message
kernel/sched/core.c +2 -0
@@ -1736,6 +1736,8 @@ static void __sched_fork(unsigned long cINIT_LIST_HEAD (&p->numa_entry); p->numa_group =NULL ;#endif /* CONFIG_NUMA_BALANCING */ + +cpu_hotplug_init_task (p); }#ifdef CONFIG_NUMA_BALANCING
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 16 lines · Peter Zijlstra
kernel/cpu.c +4
-7+void __get_online_cpus (void ) { -if (cpu_hotplug.active_writer == current) +/* Support reader-in-writer recursion */ +if (__cpuhp_writer == current)return ; -mutex_lock (&cpu_hotplug.lock); -if (WARN_ON (!cpu_hotplug.refcount)) - cpu_hotplug.refcount++;/* try to fix things up */ +atomic_inc (&cpuhp_waitcount); -if (!--cpu_hotplug.refcount &&unlikely (cpu_hotplug.active_writer)) -wake_up_process (cpu_hotplug.active_writer); -mutex_unlock (&cpu_hotplug.lock);
kernel/cpu.c static int cpu_hotplug_disabled; · line 85
Quoted 10 lines
kernel/cpu.c +10
-0+/* +* We either call schedule() in the wait, or we'll fall through +* and reschedule on the preempt_enable() in get_online_cpus(). +*/ +preempt_enable_no_resched (); +wait_event (cpuhp_readers, !__cpuhp_writer); +preempt_disable (); + +if (atomic_dec_and_test (&cpuhp_waitcount)) +wake_up_all (&cpuhp_writer);
Yes, this should fix the races with the exiting writer, but still this doesn't look right afaics.
In particular let me repeat,
Quoted 11 lines
void
cpu_hotplug_begin(void)
{kernel/cpu.c +4
-4- cpu_hotplug.active_writer = current; +unsigned int count =0 ; +int cpu; + +lockdep_assert_held (&cpu_add_remove_lock); -for (;;) { -mutex_lock (&cpu_hotplug.lock); -if (likely (!cpu_hotplug.refcount))
kernel/cpu.c static int cpu_hotplug_disabled; · line 118
Quoted 10 lines
kernel/cpu.c +6
-4-break ; -__set_current_state (TASK_UNINTERRUPTIBLE ); -mutex_unlock (&cpu_hotplug.lock); -schedule (); + __cpuhp_writer = current; + +/* +* After this everybody will observe writer and take the slow path. +*/ +synchronize_sched ();
synchronize_sched() is slow. The new readers will likely notice __cpuhp_writer != NULL much earlier and they will be blocked in __get_online_cpus() while the writer sleeps before it actually enters the critical section.
Or I completely misunderstood this all?
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Peter ZijlstraQuoted 18 lines · Peter Zijlstra
Quoted 8 lines · Paul E. McKenney
Quoted 8 lines
(quoted diff) +8
-0+#define cpuhp_writer_wake() \ + wake_up_process(cpuhp_writer_task) + +#define cpuhp_writer_wait(cond) \ +do { \ + for (;;) { \ + set_current_state(TASK_UNINTERRUPTIBLE); \ + if (cond) \Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ break; \ + schedule(); \ + } \ + __set_current_state(TASK_RUNNING); \ +} while (0)Why not
wait_event()? Presumably the above is a bit lighter weight, but is that even something that can be measured?
Quoted 3 lines
I didn't want to mix readers and writers on
cpuhp_wq, and I suppose I could create a second waitqueue; that might also be a better solution for the NULL thing below.
That would have the advantage of being a bit less racy.
Quoted 11 lines
Quoted 10 lines
kernel/cpu.c static int cpu_hotplug_disabled;
Quoted 8 lines
kernel/cpu.c +8
-0+atomic_inc (&cpuhp_waitcount); + +/* +* We either call schedule() in the wait, or we'll fall through +* and reschedule on the preempt_enable() in get_online_cpus(). +*/ +preempt_enable_no_resched (); +wait_event (cpuhp_wq, !__cpuhp_writer);Finally! A good use for
preempt_enable_no_resched(). ;-)
Quoted 2 lines
Hehe, there were a few others, but tglx removed most with the
schedule_preempt_disabled()primitive.
;-)
Quoted 3 lines
In fact, I considered a
wait_event_preempt_disabled()but was too lazy. That wholewait_eventmacro fest looks like it could use an iteration or two of collapse anyhow.
There are some serious layers there, aren't there?
Quoted 59 lines
Quoted 16 lines
Review
Quoted 11 lines
(quoted diff) +11
-0+ preempt_disable(); + + /* + * It would be possible for cpu_hotplug_done() to complete before + * the atomic_inc() above; in which case there is no writer waiting + * and doing a wakeup would be BAD (tm). + * + * If however we still observe cpuhp_writer_task here we know + * cpu_hotplug_done() is currently stuck waiting for cpuhp_waitcount. + */ + if (atomic_dec_and_test(&cpuhp_waitcount) && cpuhp_writer_task)OK, I'll bite... What sequence of events results in the
atomic_dec_and_test()returning true but there being nocpuhp_writer_task?Quoted 1 line
Ah, I see it...
<snip>
Indeed, and
Quoted 1 line
But what prevents the following sequence of events?
<snip>
Quoted 1 line
o Task B's call to
cpuhp_writer_wake()sees a NULL pointer.Quite so.. nothing. See there was a reason I kept being confused about it.
Quoted 7 lines
Quoted 7 lines
void
cpu_hotplug_begin(void)
{kernel/cpu.c +4
-0+unsigned int count =0 ; +int cpu; + +lockdep_assert_held (&cpu_add_remove_lock);Quoted 5 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ __cpuhp_writer = 1; + cpuhp_writer_task = current;At this point, the value of
cpuhp_slowcountcan go negative. Can't see that this causes a problem, given theatomic_add()below.Agreed.
Quoted 4 lines
Quoted 4 lines
kernel/cpu.c +4
-0+ +/* After this everybody will observe writer and take the slow path. */ +synchronize_sched (); +Quoted 8 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 128
Quoted 5 lines
kernel/cpu.c +4
-0+/* Collapse the per-cpu refcount into slowcount */ +for_each_possible_cpu (cpu) { + count +=per_cpu (__cpuhp_refcount, cpu); +per_cpu (__cpuhp_refcount, cpu) =0 ; }The above is safe because the readers are no longer changing their
__cpuhp_refcountvalues.
Quoted 3 lines
Yes, I'll expand the comment.
So how about something like this?
A few memory barriers required, if I am reading the code correctly. Some of them, perhaps all of them, called out by Oleg.
Thanx, Paul
Quoted 30 lines
---
include/linux/cpu.h +12
-2@@ -16,6 +16,7 @@#include <linux/node.h> #include <linux/compiler.h> #include <linux/cpumask.h> +#include <linux/percpu.h> struct device ;@@ -173,10 +174,50 @@ extern struct bus_type cpu_subsys;#ifdef CONFIG_HOTPLUG_CPU /* Stop CPUs going up and down. */ +extern void cpu_hotplug_init_task (struct task_struct *p); +extern void cpu_hotplug_begin (void );extern void cpu_hotplug_done (void ); -extern void get_online_cpus (void ); -extern void put_online_cpus (void ); + +extern struct task_struct *__cpuhp_writer; +DECLARE_PER_CPU(unsigned int, __cpuhp_refcount); + +extern void __get_online_cpus (void ); + +static inline void get_online_cpus (void ) +{ +might_sleep ();
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 199
Quoted 10 lines
include/linux/cpu.h +10
-0+ +/* Support reader-in-reader recursion */ +if (current->cpuhp_ref++) { +barrier (); +return ; + } + +preempt_disable (); +if (likely (!__cpuhp_writer)) +__this_cpu_inc (__cpuhp_refcount);
As Oleg noted, need a barrier here for when a new reader runs concurrently with a completing writer.
Quoted 6 lines
include/linux/cpu.h +6
-0+else +__get_online_cpus (); +preempt_enable (); +} + +extern void __put_online_cpus (void );
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 215
Quoted 10 lines
include/linux/cpu.h +10
-0+ +static inline void put_online_cpus (void ) +{ +barrier (); +if (--current->cpuhp_ref) +return ; + +preempt_disable (); +if (likely (!__cpuhp_writer)) +__this_cpu_dec (__cpuhp_refcount);
No barrier needed here because synchronize_sched() covers it.
Quoted 88 lines
include/linux/cpu.h +7
-0+else +__put_online_cpus (); +preempt_enable (); +} +extern void cpu_hotplug_disable (void );extern void cpu_hotplug_enable (void );#define hotcpu_notifier (fn, pri) cpu_notifier(fn, pri)@@ -200,6 +241,8 @@ static inline void cpu_hotplug_driver_un#else /* CONFIG_HOTPLUG_CPU */ +static inline void cpu_hotplug_init_task (struct task_struct *p) {} +static inline void cpu_hotplug_begin (void ) {}static inline void cpu_hotplug_done (void ) {}#define get_online_cpus () do { } while (0)include/linux/sched.h +3
-0@@ -1454,6 +1454,9 @@ struct task_struct {unsigned int sequential_io;unsigned int sequential_io_avg;#endif +#ifdef CONFIG_HOTPLUG_CPU +int cpuhp_ref; +#endif };/* Future-safe accessor for struct task_struct's cpus_allowed. */ kernel/cpu.c +16
-29@@ -49,88 +49,100 @@ static int cpu_hotplug_disabled;#ifdef CONFIG_HOTPLUG_CPU -static struct { -struct task_struct *active_writer; -struct mutex lock;/* Synchronizes accesses to refcount, */ -/* -* Also blocks the new readers during -* an ongoing cpu hotplug operation. -*/ -int refcount; -} cpu_hotplug = { - .active_writer =NULL , - .lock =__MUTEX_INITIALIZER (cpu_hotplug.lock), - .refcount =0 , -}; +struct task_struct *__cpuhp_writer; +EXPORT_SYMBOL_GPL (__cpuhp_writer); -void get_online_cpus (void ) -{ -might_sleep (); -if (cpu_hotplug.active_writer == current) -return ; -mutex_lock (&cpu_hotplug.lock); - cpu_hotplug.refcount++; -mutex_unlock (&cpu_hotplug.lock); +DEFINE_PER_CPU(unsigned int, __cpuhp_refcount); +EXPORT_PER_CPU_SYMBOL_GPL (__cpuhp_refcount); + +static atomic_t cpuhp_waitcount; +static atomic_t cpuhp_slowcount; +static DECLARE_WAIT_QUEUE_HEAD (cpuhp_readers); +static DECLARE_WAIT_QUEUE_HEAD (cpuhp_writer); +void cpu_hotplug_init_task (struct task_struct *p) +{ + p->cpuhp_ref =0 ; } -EXPORT_SYMBOL_GPL (get_online_cpus); -void put_online_cpus (void ) +void __get_online_cpus (void ) { -if (cpu_hotplug.active_writer == current) +/* Support reader-in-writer recursion */ +if (__cpuhp_writer == current)return ; -mutex_lock (&cpu_hotplug.lock); -if (WARN_ON (!cpu_hotplug.refcount)) - cpu_hotplug.refcount++;/* try to fix things up */ +atomic_inc (&cpuhp_waitcount); -if (!--cpu_hotplug.refcount &&unlikely (cpu_hotplug.active_writer)) -wake_up_process (cpu_hotplug.active_writer);
kernel/cpu.c static int cpu_hotplug_disabled; · line 84
Quoted 10 lines
kernel/cpu.c +9
-1-mutex_unlock (&cpu_hotplug.lock); +/* +* We either call schedule() in the wait, or we'll fall through +* and reschedule on the preempt_enable() in get_online_cpus(). +*/ +preempt_enable_no_resched (); +wait_event (cpuhp_readers, !__cpuhp_writer); +preempt_disable (); + +if (atomic_dec_and_test (&cpuhp_waitcount))
This provides the needed memory barrier for concurrent write releases.
kernel/cpu.c static int cpu_hotplug_disabled; · line 94
Quoted 10 lines
kernel/cpu.c +9
-0+wake_up_all (&cpuhp_writer); +} +EXPORT_SYMBOL_GPL (__get_online_cpus); + +void __put_online_cpus (void ) +{ +if (__cpuhp_writer == current) +return ; +if (atomic_dec_and_test (&cpuhp_slowcount))
This provides the needed memory barrier for concurrent write acquisitions.
Quoted 52 lines
kernel/cpu.c +16
-24+wake_up_all (&cpuhp_writer); } -EXPORT_SYMBOL_GPL (put_online_cpus); +EXPORT_SYMBOL_GPL (__put_online_cpus);/* * This ensures that the hotplug operation can begin only when the * refcount goes to zero. * -* Note that during a cpu-hotplug operation, the new readers, if any, -* will be blocked by the cpu_hotplug.lock -* * Since cpu_hotplug_begin() is always called after invoking * cpu_maps_update_begin(), we can be sure that only one writer is active. -* -* Note that theoretically, there is a possibility of a livelock: -* - Refcount goes to zero, last reader wakes up the sleeping -* writer. -* - Last reader unlocks the cpu_hotplug.lock. -* - A new reader arrives at this moment, bumps up the refcount. -* - The writer acquires the cpu_hotplug.lock finds the refcount -* non zero and goes to sleep again. -* -* However, this is very difficult to achieve in practice since -* get_online_cpus() not an api which is called all that often. -* */ void cpu_hotplug_begin (void ) { - cpu_hotplug.active_writer = current; +unsigned int count =0 ; +int cpu; + +lockdep_assert_held (&cpu_add_remove_lock); -for (;;) { -mutex_lock (&cpu_hotplug.lock); -if (likely (!cpu_hotplug.refcount)) -break ; -__set_current_state (TASK_UNINTERRUPTIBLE ); -mutex_unlock (&cpu_hotplug.lock); -schedule (); + __cpuhp_writer = current; + +/* +* After this everybody will observe writer and take the slow path. +*/ +synchronize_sched (); + +/* +* Collapse the per-cpu refcount into slowcount. This is safe because +* readers are now taking the slow path (per the above) which doesn't
kernel/cpu.c static int cpu_hotplug_disabled; · line 132
Quoted 10 lines
kernel/cpu.c +9
-0+* touch __cpuhp_refcount. +*/ +for_each_possible_cpu (cpu) { + count +=per_cpu (__cpuhp_refcount, cpu); +per_cpu (__cpuhp_refcount, cpu) =0 ; } +atomic_add (count, &cpuhp_slowcount); + +/* Wait for all readers to go away */ +wait_event (cpuhp_writer, !atomic_read (&cpuhp_slowcount));
Oddly enough, there appear to be cases where you need a memory barrier here. Suppose that all the readers finish after the atomic_add() above, but before the wait_event(). Then wait_event() just checks the condition without any memory barriers. So smp_mb() needed here.
/me runs off to check RCU's use of wait_event()...
Found one missing. And some places in need of comments. And a few places that could use an ACCESS_ONCE().
Back to the review...
kernel/cpu.c static int cpu_hotplug_disabled; · line 137
Quoted 7 lines
kernel/cpu.c +1
-2}void cpu_hotplug_done (void ) { - cpu_hotplug.active_writer =NULL ; -mutex_unlock (&cpu_hotplug.lock); +/* Signal the writer is done */
And I believe we need a memory barrier here to keep the write-side critical section confined from the viewpoint of a reader that starts just after the NULLing of cpuhp_writer.
Of course, being who I am, I cannot resist pointing out that you have the same number of memory barriers as would use of SRCU, and that synchronize_srcu() can be quite a bit faster than synchronize_sched() in the case where there are no readers. ;-)
Quoted 15 lines
kernel/cpu.c +8
-0+ cpuhp_writer =NULL ; +wake_up_all (&cpuhp_readers); + +/* +* Wait for any pending readers to be running. This ensures readers +* after writer and avoids writers starving readers. +*/ +wait_event (cpuhp_writer, !atomic_read (&cpuhp_waitcount)); }/*
kernel/sched/core.c static void __sched_fork(unsigned long c · line 1743
Quoted 8 lines
kernel/sched/core.c +2
-0@@ -1736,6 +1736,8 @@ static void __sched_fork(unsigned long cINIT_LIST_HEAD (&p->numa_entry); p->numa_group =NULL ;#endif /* CONFIG_NUMA_BALANCING */ + +cpu_hotplug_init_task (p); }#ifdef CONFIG_NUMA_BALANCING
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Peter Zijlstrainclude/linux/cpu.h extern struct bus_type cpu_subsys; · line 193
Quoted 8 lines · Peter Zijlstra
include/linux/cpu.h +7
-0+static inline void get_online_cpus (void ) +{ +might_sleep (); + +if (current->cpuhp_ref++) { +barrier (); +return ;
I don't undestand this barrier()... we are going to return if we already hold the lock, do we really need it?
The same for put_online_cpus().
kernel/cpu.c static int cpu_hotplug_disabled; · line 86
Quoted 5 lines
kernel/cpu.c +2
-1+void __get_online_cpus (void ) { -if (cpu_hotplug.active_writer == current) +if (cpuhp_writer_task == current)return ;
Probably it would be better to simply inc/dec ->cpuhp_ref in cpu_hotplug_begin/end and remove this check here and in __put_online_cpus().
This also means that the writer doing get/put_online_cpus() will always use the fast path, and __cpuhp_writer can go away, cpuhp_writer_task != NULL can be used instead.
Quoted 10 lines
kernel/cpu.c +10
-0+atomic_inc (&cpuhp_waitcount); + +/* +* We either call schedule() in the wait, or we'll fall through +* and reschedule on the preempt_enable() in get_online_cpus(). +*/ +preempt_enable_no_resched (); +wait_event (cpuhp_wq, !__cpuhp_writer); +preempt_disable (); +
kernel/cpu.c static int cpu_hotplug_disabled; · line 107
Quoted 10 lines
kernel/cpu.c +10
-0+/* +* It would be possible for cpu_hotplug_done() to complete before +* the atomic_inc() above; in which case there is no writer waiting +* and doing a wakeup would be BAD (tm). +* +* If however we still observe cpuhp_writer_task here we know +* cpu_hotplug_done() is currently stuck waiting for cpuhp_waitcount. +*/ +if (atomic_dec_and_test (&cpuhp_waitcount) && cpuhp_writer_task) +cpuhp_writer_wake ();
cpuhp_writer_wake() here and in __put_online_cpus() looks racy... Not only cpuhp_writer_wake() can hit cpuhp_writer_task == NULL (we need something like ACCESS_ONCE()), its task_struct can be already freed/reused if the writer exits.
And I don't really understand the logic... This slow path succeds without incrementing any counter (except current->cpuhp_ref)? How the next writer can notice the fact it should wait for this reader?
kernel/cpu.c static int cpu_hotplug_disabled; · line 160
Quoted 11 lines
void
cpu_hotplug_done(void)
{kernel/cpu.c +7
-2- cpu_hotplug.active_writer =NULL ; -mutex_unlock (&cpu_hotplug.lock); +/* Signal the writer is done */ + cpuhp_writer =0 ; +wake_up_all (&cpuhp_wq); + +/* Wait for any pending readers to be running */ +cpuhp_writer_wait (!atomic_read (&cpuhp_waitcount)); + cpuhp_writer_task =NULL ;
We also need to ensure that the next reader should see all changes done by the writer, iow this lacks "realease" semantics.
But, Peter, the main question is, why this is better than percpu_rw_semaphore performance-wise? (Assuming we add task_struct->cpuhp_ref).
If the writer is pending, percpu_down_read() does
down_read(&brw->rw_sem); atomic_inc(&brw->slow_read_ctr); __up_read(&brw->rw_sem);
is it really much worse than wait_event + atomic_dec_and_test?
And! please note that with your implementation the new readers will be likely blocked while the writer sleeps in synchronize_sched(). This doesn't happen with percpu_rw_semaphore.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Steven Rostedt
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 12 lines · On Tue, 24 Sep 2013 18:03:59 +0200 Oleg Nesterov <[email protected]> wrote:
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 193
Quoted 8 lines · Peter Zijlstra
include/linux/cpu.h +7
-0+static inline void get_online_cpus (void ) +{ +might_sleep (); + +if (current->cpuhp_ref++) { +barrier (); +return ;I don't undestand this
barrier()... we are going to return if we already hold the lock, do we really need it?
I'm confused too. Unless gcc moves this after the release, but the release uses preempt_disable() which is its own barrier.
If anything, it requires a comment.
-- Steve
Quoted 9 lines
The same for
put_online_cpus().Quoted 5 lines
kernel/cpu.c +2
-1+void __get_online_cpus (void ) { -if (cpu_hotplug.active_writer == current) +if (cpuhp_writer_task == current)return ;
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 17 lines · Steven Rostedt
Quoted 12 lines · On Tue, 24 Sep 2013 18:03:59 +0200 Oleg Nesterov <[email protected]> wrote:
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 193
Quoted 8 lines · Peter Zijlstra
include/linux/cpu.h +7
-0+static inline void get_online_cpus (void ) +{ +might_sleep (); + +if (current->cpuhp_ref++) { +barrier (); +return ;I don't undestand this
barrier()... we are going to return if we already hold the lock, do we really need it?
Quoted 4 lines
I'm confused too. Unless gcc moves this after the release, but the release uses
preempt_disable()which is its own barrier.If anything, it requires a comment.
And I am still confused even after emails from Paul and Peter...
If gcc can actually do something wrong, then I suspect this barrier() should be unconditional.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 21 lines · Oleg Nesterov
Quoted 20 lines · Steven Rostedt
Quoted 12 lines · On Tue, 24 Sep 2013 18:03:59 +0200 Oleg Nesterov <[email protected]> wrote:
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 193
Quoted 8 lines · Peter Zijlstra
include/linux/cpu.h +7
-0+static inline void get_online_cpus (void ) +{ +might_sleep (); + +if (current->cpuhp_ref++) { +barrier (); +return ;I don't undestand this
barrier()... we are going to return if we already hold the lock, do we really need it?I'm confused too. Unless gcc moves this after the release, but the release uses
preempt_disable()which is its own barrier.
Quoted 6 lines
Quoted 1 line
If anything, it requires a comment.
And I am still confused even after emails from Paul and Peter...
If gcc can actually do something wrong, then I suspect this
barrier()should be unconditional.
If you are saying that there should be a barrier() on all return paths from get_online_cpus(), I agree.
Thanx, Paul
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 8 lines · Paul E. McKenney
Quoted 3 lines · Oleg Nesterov
If gcc can actually do something wrong, then I suspect this
barrier()should be unconditional.If you are saying that there should be a
barrier()on all return paths fromget_online_cpus(), I agree.
Paul, Peter, could you provide any (even completely artificial) example to explain me why do we need this barrier() ? I am puzzled. And preempt_enable() already has barrier...
get_online_cpus(); do_something();
Yes, we need to ensure gcc doesn't reorder this code so that do_something() comes before get_online_cpus(). But it can't? At least it should check current->cpuhp_ref != 0 first? And if it is non-zero we do not really care, we are already in the critical section and this ->cpuhp_ref has only meaning in put_online_cpus().
Confused...
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 17 lines · Oleg Nesterov
Quoted 8 lines · Paul E. McKenney
Quoted 3 lines · Oleg Nesterov
If gcc can actually do something wrong, then I suspect this
barrier()should be unconditional.If you are saying that there should be a
barrier()on all return paths fromget_online_cpus(), I agree.Paul, Peter, could you provide any (even completely artificial) example to explain me why do we need this
barrier()? I am puzzled. Andpreempt_enable()already has barrier...get_online_cpus(); do_something();
Quoted 7 lines
Yes, we need to ensure gcc doesn't reorder this code so that
do_something()comes beforeget_online_cpus(). But it can't? At least it should checkcurrent->cpuhp_ref!= 0 first? And if it is non-zero we do not really care, we are already in the critical section and this ->cpuhp_ref has only meaning input_online_cpus().Confused...
So the reason I put it in was because of the inline; it could possibly make it do:
test 0, current->cpuhp_ref
je label1:
inc current->cpuhp_ref
label2:
do_something();
label1:
inc %gs:__preempt_count
test 0, __cpuhp_writer
jne label3
inc %gs:__cpuhp_refcount
label5
dec %gs:__preempt_count
je label4
jmp label2
label3:
call __get_online_cpus();
jmp label5
label4:
call ____preempt_schedule();
jmp label2
In which case the recursive fast path doesn't have a barrier() between taking the ref and starting do_something().
I wanted to make absolutely sure nothing of do_something leaked before the label2 thing. The other labels all have barrier() from the preempt_count ops.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 14 lines · Peter Zijlstra
Quoted 8 lines · Oleg Nesterov
Yes, we need to ensure gcc doesn't reorder this code so that
do_something()comes beforeget_online_cpus(). But it can't? At least it should checkcurrent->cpuhp_ref!= 0 first? And if it is non-zero we do not really care, we are already in the critical section and this ->cpuhp_ref has only meaning input_online_cpus().Confused...
So the reason I put it in was because of the inline; it could possibly make it do:
[...snip...]
Quoted 2 lines
In which case the recursive fast path doesn't have a
barrier()between taking the ref and startingdo_something().
Yes, but my point was, this can only happen in recursive fast path. And in this case (I think) we do not care, we are already in the critical section.
current->cpuhp_ref doesn't matter at all until we call put_online_cpus().
Suppose that gcc knows for sure that current->cpuhp_ref != 0. Then I think, for example,
get_online_cpus(); do_something(); put_online_cpus();
converted to
do_something(); current->cpuhp_ref++; current->cpuhp_ref--;
is fine. do_something() should not depend on ->cpuhp_ref.
OK, please forget. I guess I will never understand this ;)
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 1 line · Oleg Nesterov
Yes, but my point was, this can only happen in recursive fast path.
Right, I understood.
Quoted 2 lines
And in this case (I think) we do not care, we are already in the critical section.
I tend to agree, however paranoia..
Quoted 1 line
OK, please forget. I guess I will never understand this ;)
It might just be I'm less certain about there not being any avenue of mischief.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Oleg Nesterov
And in this case (I think) we do not care, we are already in the critical section.
I tend to agree, however paranoia..
Ah, in this case I tend to agree. better be paranoid ;)
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Oleg NesterovQuoted 13 lines · Oleg Nesterov
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 193
Quoted 8 lines · Peter Zijlstra
include/linux/cpu.h +7
-0+static inline void get_online_cpus (void ) +{ +might_sleep (); + +if (current->cpuhp_ref++) { +barrier (); +return ;I don't undestand this
barrier()... we are going to return if we already hold the lock, do we really need it?
Quoted 1 line
The same for
put_online_cpus().
The barrier() is needed because of the possibility of inlining, right?
Quoted 10 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 86
Quoted 5 lines
kernel/cpu.c +2
-1+void __get_online_cpus (void ) { -if (cpu_hotplug.active_writer == current) +if (cpuhp_writer_task == current)return ;Probably it would be better to simply inc/dec ->cpuhp_ref in cpu_hotplug_begin/end and remove this check here and in
__put_online_cpus().
Quoted 3 lines
This also means that the writer doing get/put_online_cpus() will always use the fast path, and
__cpuhp_writercan go away,cpuhp_writer_task!= NULL can be used instead.
I would need to see the code for this change to be sure. ;-)
Quoted 30 lines
Quoted 10 lines
kernel/cpu.c +10
-0+atomic_inc (&cpuhp_waitcount); + +/* +* We either call schedule() in the wait, or we'll fall through +* and reschedule on the preempt_enable() in get_online_cpus(). +*/ +preempt_enable_no_resched (); +wait_event (cpuhp_wq, !__cpuhp_writer); +preempt_disable (); +kernel/cpu.c static int cpu_hotplug_disabled; · line 107
Quoted 10 lines
kernel/cpu.c +10
-0+/* +* It would be possible for cpu_hotplug_done() to complete before +* the atomic_inc() above; in which case there is no writer waiting +* and doing a wakeup would be BAD (tm). +* +* If however we still observe cpuhp_writer_task here we know +* cpu_hotplug_done() is currently stuck waiting for cpuhp_waitcount. +*/ +if (atomic_dec_and_test (&cpuhp_waitcount) && cpuhp_writer_task) +cpuhp_writer_wake ();
cpuhp_writer_wake()here and in__put_online_cpus()looks racy... Not onlycpuhp_writer_wake()can hitcpuhp_writer_task== NULL (we need something likeACCESS_ONCE()), itstask_structcan be already freed/reused if the writer exits.And I don't really understand the logic... This slow path succeds without incrementing any counter (except
current->cpuhp_ref)? How the next writer can notice the fact it should wait for this reader?
Quoted 14 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 160
Quoted 11 lines
void
cpu_hotplug_done(void)
{kernel/cpu.c +7
-2- cpu_hotplug.active_writer =NULL ; -mutex_unlock (&cpu_hotplug.lock); +/* Signal the writer is done */ + cpuhp_writer =0 ; +wake_up_all (&cpuhp_wq); + +/* Wait for any pending readers to be running */ +cpuhp_writer_wait (!atomic_read (&cpuhp_waitcount)); + cpuhp_writer_task =NULL ;We also need to ensure that the next reader should see all changes done by the writer, iow this lacks "realease" semantics.
Good point -- I was expecting wake_up_all() to provide the release semantics, but code could be reordered into __wake_up()'s critical section, especially in the case where there was nothing to wake up, but where there were new readers starting concurrently with cpu_hotplug_done().
Quoted 10 lines
But, Peter, the main question is, why this is better than
percpu_rw_semaphoreperformance-wise? (Assuming we addtask_struct->cpuhp_ref).If the writer is pending,
percpu_down_read()doesdown_read(&brw->rw_sem); atomic_inc(&brw->slow_read_ctr); __up_read(&brw->rw_sem);
Quoted 5 lines
is it really much worse than
wait_event+atomic_dec_and_test?And! please note that with your implementation the new readers will be likely blocked while the writer sleeps in
synchronize_sched(). This doesn't happen withpercpu_rw_semaphore.
Thanx, Paul
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 13 lines · Paul E. McKenney
Quoted 4 lines
Quoted 4 lines
void
cpu_hotplug_done(void)
{kernel/cpu.c +2
-0+/* Signal the writer is done */ + cpuhp_writer =0 ;Quoted 8 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 160
Quoted 5 lines
kernel/cpu.c +5
-0+wake_up_all (&cpuhp_wq); + +/* Wait for any pending readers to be running */ +cpuhp_writer_wait (!atomic_read (&cpuhp_waitcount)); + cpuhp_writer_task =NULL ;We also need to ensure that the next reader should see all changes done by the writer, iow this lacks "realease" semantics.
Quoted 5 lines
Good point -- I was expecting
wake_up_all()to provide the release semantics, but code could be reordered into __wake_up()'s critical section, especially in the case where there was nothing to wake up, but where there were new readers starting concurrently withcpu_hotplug_done().
Doh, indeed. I missed this in Oleg's email, but yes I made that same assumption about wake_up_all().
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] hotplug: Optimize {get,put}_online_cpus()
Quoted 15 lines · Peter Zijlstra
Quoted 13 lines · Paul E. McKenney
Quoted 12 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 160
Quoted 9 lines
void
cpu_hotplug_done(void)
{kernel/cpu.c +7
-0+/* Signal the writer is done */ + cpuhp_writer =0 ; +wake_up_all (&cpuhp_wq); + +/* Wait for any pending readers to be running */ +cpuhp_writer_wait (!atomic_read (&cpuhp_waitcount)); + cpuhp_writer_task =NULL ;We also need to ensure that the next reader should see all changes done by the writer, iow this lacks "realease" semantics.
Quoted 8 lines
Quoted 5 lines
Good point -- I was expecting
wake_up_all()to provide the release semantics, but code could be reordered into __wake_up()'s critical section, especially in the case where there was nothing to wake up, but where there were new readers starting concurrently withcpu_hotplug_done().Doh, indeed. I missed this in Oleg's email, but yes I made that same assumption about
wake_up_all().
Well, I think this is even worse... No matter what the writer does, the new reader needs mb() after it checks !__cpuhp_writer. Or we need another synchronize_sched() in cpu_hotplug_done(). This is what percpu_rw_semaphore() does (to remind, this can be turned into call_rcu).
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Oleg NesterovQuoted 13 lines · Oleg Nesterov
include/linux/cpu.h extern struct bus_type cpu_subsys; · line 193
Quoted 8 lines · Peter Zijlstra
include/linux/cpu.h +7
-0+static inline void get_online_cpus (void ) +{ +might_sleep (); + +if (current->cpuhp_ref++) { +barrier (); +return ;I don't undestand this
barrier()... we are going to return if we already hold the lock, do we really need it?
Quoted 1 line
The same for
put_online_cpus().
to make {get,put}_online_cpus() always behave like per-cpu lock sections.
I don't think its ever 'correct' for loads/stores to escape the section, even if not strictly harmful.
Quoted 9 lines
kernel/cpu.c static int cpu_hotplug_disabled; · line 86
Quoted 5 lines
kernel/cpu.c +2
-1+void __get_online_cpus (void ) { -if (cpu_hotplug.active_writer == current) +if (cpuhp_writer_task == current)return ;Probably it would be better to simply inc/dec ->cpuhp_ref in cpu_hotplug_begin/end and remove this check here and in
__put_online_cpus().
Oh indeed!
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ if (atomic_dec_and_test(&cpuhp_waitcount) && cpuhp_writer_task) + cpuhp_writer_wake();
cpuhp_writer_wake()here and in__put_online_cpus()looks racy...
Yeah it is. Paul already said.
Quoted 10 lines
But, Peter, the main question is, why this is better than
percpu_rw_semaphoreperformance-wise? (Assuming we addtask_struct->cpuhp_ref).If the writer is pending,
percpu_down_read()doesdown_read(&brw->rw_sem); atomic_inc(&brw->slow_read_ctr); __up_read(&brw->rw_sem);
Quoted 5 lines
is it really much worse than
wait_event+atomic_dec_and_test?And! please note that with your implementation the new readers will be likely blocked while the writer sleeps in
synchronize_sched(). This doesn't happen withpercpu_rw_semaphore.
Good points both, no I don't think there's a significant performance gap there.
I'm still hoping we can come up with something better though :/ I don't particularly like either.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Steven Rostedt
maintainerRe: [PATCH] hotplug: Optimize {get,put}_online_cpus()
In reply to Peter Zijlstrakernel/cpu.c static int cpu_hotplug_disabled; · line 83
Quoted 12 lines · On Tue, 24 Sep 2013 14:38:21 +0200 Peter Zijlstra <[email protected]> wrote:
kernel/cpu.c +12
-0+#define cpuhp_writer_wait (cond) \ +do { \ + for (;;) { \ + set_current_state(TASK_UNINTERRUPTIBLE); \ + if (cond) \ + break; \ + schedule(); \ + } \ + __set_current_state(TASK_RUNNING); \ +} while (0) + +void __get_online_cpus (void )
The above really needs a comment about how it is used. Otherwise, I can envision someone calling this as "oh I can use this when I'm in a preempt disable section", and the comment below for the preempt_enable_no_resched() will no longer be true.
-- Steve
Quoted 21 lines
{
kernel/cpu.c +13
-4-if (cpu_hotplug.active_writer == current) +if (cpuhp_writer_task == current)return ; -mutex_lock (&cpu_hotplug.lock); -if (WARN_ON (!cpu_hotplug.refcount)) - cpu_hotplug.refcount++;/* try to fix things up */ +atomic_inc (&cpuhp_waitcount); + +/* +* We either call schedule() in the wait, or we'll fall through +* and reschedule on the preempt_enable() in get_online_cpus(). +*/ +preempt_enable_no_resched (); +wait_event (cpuhp_wq, !__cpuhp_writer); +preempt_disable (); + +/* +* It would be possible for cpu_hotplug_done() to complete before
kernel/cpu.c static int cpu_hotplug_disabled; · line 109
Quoted 10 lines
kernel/cpu.c +10
-0+* the atomic_inc() above; in which case there is no writer waiting +* and doing a wakeup would be BAD (tm). +* +* If however we still observe cpuhp_writer_task here we know +* cpu_hotplug_done() is currently stuck waiting for cpuhp_waitcount. +*/ +if (atomic_dec_and_test (&cpuhp_waitcount) && cpuhp_writer_task) +cpuhp_writer_wake (); +} +EXPORT_SYMBOL_GPL (__get_online_cpus);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
[RFC] introduce synchronize_sched_{enter,exit}()
In reply to Peter ZijlstraHello.
Paul, Peter, et al, could you review the code below?
I am not sending the patch, I think it is simpler to read the code inline (just in case, I didn't try to compile it yet).
It is functionally equivalent to
struct xxx_struct {
atomic_t counter;
};
static inline bool xxx_is_idle(struct xxx_struct *xxx)
{
return atomic_read(&xxx->counter) == 0;
}
static inline void xxx_enter(struct xxx_struct *xxx)
{
atomic_inc(&xxx->counter);
synchronize_sched();
}
static inline void xxx_enter(struct xxx_struct *xxx)
{
synchronize_sched();
atomic_dec(&xxx->counter);
}except: it records the state and synchronize_sched() is only called by xxx_enter() and only if necessary.
Why? Say, percpu_rw_semaphore, or upcoming changes in get_online_cpus(), (Peter, I think they should be unified anyway, but lets ignore this for now). Or freeze_super() (which currently looks buggy), perhaps something else. This pattern
writer: state = SLOW_MODE; synchronize_rcu/sched(); reader: preempt_disable(); // or rcu_read_lock(); if (state != SLOW_MODE) ...
is quite common.
Note:
- This implementation allows multiple writers, and sometimes this makes sense.
- But it's trivial to add "bool xxx->exclusive" set by xxx_init(). If it is true only one xxx_enter() is possible, other callers should block until xxx_exit(). This is what percpu_down_write() actually needs.
- Probably it makes sense to add xxx->rcu_domain = RCU/SCHED/ETC.
Do you think it is correct? Makes sense? (BUG_ON's are just comments).
Oleg.
// .h -----------------------------------------------------------------------
struct xxx_struct {
int gp_state;
int gp_count;
wait_queue_head_t gp_waitq;
int cb_state;
struct rcu_head cb_head;
};static inline bool xxx_is_idle(struct xxx_struct *xxx)
{
return !xxx->gp_state; /* GP_IDLE */
}extern void xxx_enter(struct xxx_struct *xxx);
extern void xxx_exit(struct xxx_struct *xxx);
// .c -----------------------------------------------------------------------
enum { GP_IDLE = 0, GP_PENDING, GP_PASSED };
enum { CB_IDLE = 0, CB_PENDING, CB_REPLAY };
#define xxx_lock gp_waitq.lock
void xxx_enter(struct xxx_struct *xxx)
{
bool need_wait, need_sync;
spin_lock_irq(&xxx->xxx_lock);
need_wait = xxx->gp_count++;
need_sync = xxx->gp_state == GP_IDLE;
if (need_sync)
xxx->gp_state = GP_PENDING;
spin_unlock_irq(&xxx->xxx_lock);
BUG_ON(need_wait && need_sync);
} if (need_sync) {
synchronize_sched();
xxx->gp_state = GP_PASSED;
wake_up_all(&xxx->gp_waitq);
} else if (need_wait) {
wait_event(&xxx->gp_waitq, xxx->gp_state == GP_PASSED);
} else {
BUG_ON(xxx->gp_state != GP_PASSED);
}
}static void cb_rcu_func(struct rcu_head *rcu)
{
struct xxx_struct *xxx = container_of(rcu, struct xxx_struct, cb_head);
long flags;
BUG_ON(xxx->gp_state != GP_PASSED);
BUG_ON(xxx->cb_state == CB_IDLE);
spin_lock_irqsave(&xxx->xxx_lock, flags);
if (xxx->gp_count) {
xxx->cb_state = CB_IDLE;
} else if (xxx->cb_state == CB_REPLAY) {
xxx->cb_state = CB_PENDING;
call_rcu_sched(&xxx->cb_head, cb_rcu_func);
} else {
xxx->cb_state = CB_IDLE;
xxx->gp_state = GP_IDLE;
}
spin_unlock_irqrestore(&xxx->xxx_lock, flags);
}void xxx_exit(struct xxx_struct *xxx)
{
spin_lock_irq(&xxx->xxx_lock);
if (!--xxx->gp_count) {
if (xxx->cb_state == CB_IDLE) {
xxx->cb_state = CB_PENDING;
call_rcu_sched(&xxx->cb_head, cb_rcu_func);
} else if (xxx->cb_state == CB_PENDING) {
xxx->cb_state = CB_REPLAY;
}
}
spin_unlock_irq(&xxx->xxx_lock);
}--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC] introduce synchronize_sched_{enter,exit}()
Quoted 21 lines · Oleg Nesterov
Hello.
Paul, Peter, et al, could you review the code below?
I am not sending the patch, I think it is simpler to read the code inline (just in case, I didn't try to compile it yet).
It is functionally equivalent to
struct xxx_struct { atomic_t counter; }; static inline bool xxx_is_idle(struct xxx_struct *xxx) { return atomic_read(&xxx->counter) == 0; } static inline void xxx_enter(struct xxx_struct *xxx) { atomic_inc(&xxx->counter);
Quoted 8 lines
synchronize_sched();
}
static inline void xxx_enter(struct xxx_struct *xxx)
{
synchronize_sched();
atomic_dec(&xxx->counter);
}But there is nothing for synchronize_sched() to wait for in the above. Presumably the caller of xxx_is_idle() is required to disable preemption or be under rcu_read_lock_sched()?
Quoted 17 lines
except: it records the state and
synchronize_sched()is only called byxxx_enter()and only if necessary.Why? Say,
percpu_rw_semaphore, or upcoming changes inget_online_cpus(), (Peter, I think they should be unified anyway, but lets ignore this for now). Orfreeze_super()(which currently looks buggy), perhaps something else. This patternwriter: state = SLOW_MODE; synchronize_rcu/sched(); reader: preempt_disable(); // or rcu_read_lock(); if (state != SLOW_MODE) ...
Quoted 1 line
is quite common.
And this does guarantee that by the time the writer's synchronize_whatever() exits, all readers will know that state==SLOW_MODE.
Quoted 3 lines
Note:
- This implementation allows multiple writers, and sometimes this makes sense.
If each writer atomically incremented SLOW_MODE, did its update, then atomically decremented it, sure. You could be more clever and avoid unneeded synchronize_whatever() calls, but I would have to see a good reason for doing so before recommending this.
OK, but you appear to be doing this below anyway. ;-)
Quoted 4 lines
- But it's trivial to add "bool
xxx->exclusive" set byxxx_init(). If it is true only onexxx_enter()is possible, other callers should block untilxxx_exit(). This is whatpercpu_down_write()actually needs.
Agreed.
Quoted 1 line
- Probably it makes sense to add
xxx->rcu_domain= RCU/SCHED/ETC.
Or just have pointers to the RCU functions in the xxx structure...
So you are trying to make something that abstracts the RCU-protected state-change pattern? Or perhaps more accurately, the RCU-protected state-change-and-back pattern?
Quoted 1 line
Do you think it is correct? Makes sense? (BUG_ON's are just comments).
... Maybe ... Please see below for commentary and a question.
Thanx, Paul
Quoted 4 lines
Oleg.
// .h -----------------------------------------------------------------------
Quoted 8 lines
struct xxx_struct {
int gp_state;
int gp_count;
wait_queue_head_t gp_waitq;
int cb_state;
struct rcu_head cb_head;spinlock_t xxx_lock; /* ? */
This spinlock might not make the big-system guys happy, but it appears to be needed below.
Quoted 18 lines
};
static inline bool
xxx_is_idle(struct xxx_struct *xxx){ return !xxx->gp_state; /* GP_IDLE */ }extern void
xxx_enter(struct xxx_struct *xxx);
extern voidxxx_exit(struct xxx_struct *xxx);// .c -----------------------------------------------------------------------
enum {
GP_IDLE= 0,GP_PENDING,GP_PASSED};enum {
CB_IDLE= 0,CB_PENDING,CB_REPLAY};#define
xxx_lockgp_waitq.lock
Quoted 7 lines
void
xxx_enter(struct xxx_struct *xxx){ bool need_wait, need_sync; spin_lock_irq(&xxx->xxx_lock); need_wait = xxx->gp_count++; need_sync = xxx->gp_state == GP_IDLE;
Suppose ->gp_state is GP_PASSED. It could transition to GP_IDLE at any time, right?
Quoted 4 lines
if (need_sync) xxx->gp_state = GP_PENDING; spin_unlock_irq(&xxx->xxx_lock);
Quoted 8 lines
BUG_ON(need_wait && need_sync);
} if (need_sync) {
synchronize_sched();
xxx->gp_state = GP_PASSED;
wake_up_all(&xxx->gp_waitq);
} else if (need_wait) {
wait_event(&xxx->gp_waitq, xxx->gp_state == GP_PASSED);Suppose the wakeup is delayed until after the state has been updated back to GP_IDLE? Ah, presumably the non-zero ->gp_count prevents this. Never mind!
Quoted 15 lines
} else { BUG_ON(xxx->gp_state != GP_PASSED); } }static void
cb_rcu_func(struct rcu_head *rcu){ struct xxx_struct *xxx = container_of(rcu, struct xxx_struct, cb_head); long flags; BUG_ON(xxx->gp_state != GP_PASSED); BUG_ON(xxx->cb_state == CB_IDLE); spin_lock_irqsave(&xxx->xxx_lock, flags); if (xxx->gp_count) {
Quoted 8 lines
xxx->cb_state = CB_IDLE;
} else if (xxx->cb_state == CB_REPLAY) {
xxx->cb_state = CB_PENDING;
call_rcu_sched(&xxx->cb_head, cb_rcu_func);
} else {
xxx->cb_state = CB_IDLE;
xxx->gp_state = GP_IDLE;
}It took me a bit to work out the above. It looks like the intent is to have the last xxx_exit() put the state back to GP_IDLE, which appears to be the state in which readers can use a fastpath.
This works because if ->gp_count is non-zero and ->cb_state is CB_IDLE, there must be an xxx_exit() in our future.
Quoted 4 lines
spin_unlock_irqrestore(&xxx->xxx_lock, flags); }void
xxx_exit(struct xxx_struct *xxx)
Quoted 12 lines
{
spin_lock_irq(&xxx->xxx_lock);
if (!--xxx->gp_count) {
if (xxx->cb_state == CB_IDLE) {
xxx->cb_state = CB_PENDING;
call_rcu_sched(&xxx->cb_head, cb_rcu_func);
} else if (xxx->cb_state == CB_PENDING) {
xxx->cb_state = CB_REPLAY;
}
}
spin_unlock_irq(&xxx->xxx_lock);
}Then we also have something like this?
bool xxx_readers_fastpath_ok(struct xxx_struct *xxx)
{
BUG_ON(!rcu_read_lock_sched_held());
return xxx->gp_state == GP_IDLE;
}--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC] introduce synchronize_sched_{enter,exit}()
Quoted 24 lines · Paul E. McKenney
Quoted 13 lines · Oleg Nesterov
struct xxx_struct { atomic_t counter; }; static inline bool xxx_is_idle(struct xxx_struct *xxx) { return atomic_read(&xxx->counter) == 0; } static inline void xxx_enter(struct xxx_struct *xxx) { atomic_inc(&xxx->counter);Quoted 8 lines
synchronize_sched(); } static inline void xxx_enter(struct xxx_struct *xxx) { synchronize_sched(); atomic_dec(&xxx->counter); }
Quoted 3 lines
But there is nothing for
synchronize_sched()to wait for in the above. Presumably the caller ofxxx_is_idle()is required to disable preemption or be underrcu_read_lock_sched()?
Yes, yes, sure, xxx_is_idle() should be called under preempt_disable(). (or rcu_read_lock() if xxx_enter() uses synchronize_rcu()).
Quoted 3 lines
So you are trying to make something that abstracts the RCU-protected state-change pattern? Or perhaps more accurately, the RCU-protected state-change-and-back pattern?
Yes, exactly.
Quoted 10 lines
Quoted 8 lines
struct xxx_struct { int gp_state; int gp_count; wait_queue_head_t gp_waitq; int cb_state; struct rcu_head cb_head;spinlock_t xxx_lock; /* ? */
See
#define xxx_lock gp_waitq.lock
in .c below, but we can add another spinlock.
Quoted 2 lines
This spinlock might not make the big-system guys happy, but it appears to be needed below.
Only the writers use this spinlock, and they should synchronize with each other anyway. I don't think this can really penalize, say, percpu_down_write or cpu_hotplug_begin.
Quoted 12 lines
Quoted 12 lines
// .c -----------------------------------------------------------------------
enum {
GP_IDLE= 0,GP_PENDING,GP_PASSED};enum {
CB_IDLE= 0,CB_PENDING,CB_REPLAY};#define
xxx_lockgp_waitq.lockvoid
xxx_enter(struct xxx_struct *xxx){ bool need_wait, need_sync;
Quoted 6 lines
Quoted 3 lines
spin_lock_irq(&xxx->xxx_lock); need_wait = xxx->gp_count++; need_sync = xxx->gp_state == GP_IDLE;Suppose ->gp_state is
GP_PASSED. It could transition toGP_IDLEat any time, right?
As you already pointed below - no.
Once we incremented ->nr_writers, nobody can set GP_IDLE. And if the caller is the "first" writer (need_sync == T) nobody else can change ->gp_state, so xxx_enter() sets GP_PASSED lockless.
Quoted 13 lines
Quoted 4 lines
if (need_sync) xxx->gp_state = GP_PENDING; spin_unlock_irq(&xxx->xxx_lock);Quoted 8 lines
BUG_ON(need_wait && need_sync); } if (need_sync) { synchronize_sched(); xxx->gp_state = GP_PASSED; wake_up_all(&xxx->gp_waitq); } else if (need_wait) { wait_event(&xxx->gp_waitq, xxx->gp_state == GP_PASSED);
Quoted 2 lines
Suppose the wakeup is delayed until after the state has been updated back to
GP_IDLE? Ah, presumably the non-zero ->gp_count prevents this.
Yes, exactly.
Quoted 19 lines
Quoted 10 lines
static void
cb_rcu_func(struct rcu_head *rcu){ struct xxx_struct *xxx = container_of(rcu, struct xxx_struct, cb_head); long flags; BUG_ON(xxx->gp_state != GP_PASSED); BUG_ON(xxx->cb_state == CB_IDLE); spin_lock_irqsave(&xxx->xxx_lock, flags); if (xxx->gp_count) {Quoted 8 lines
xxx->cb_state = CB_IDLE; } else if (xxx->cb_state == CB_REPLAY) { xxx->cb_state = CB_PENDING; call_rcu_sched(&xxx->cb_head, cb_rcu_func); } else { xxx->cb_state = CB_IDLE; xxx->gp_state = GP_IDLE; }
Quoted 3 lines
It took me a bit to work out the above. It looks like the intent is to have the last
xxx_exit()put the state back toGP_IDLE, which appears to be the state in which readers can use a fastpath.
Yes, and we we offload this work to rcu callback so xxx_exit() doesn't block.
The only complication is the next writer which does xxx_enter() after xxx_exit(). If there are no other writers, the next xxx_exit() should do
rcu_cancel(&xxx->cb_head); call_rcu_sched(&xxx->cb_head, cb_rcu_func);
to "extend" the gp, but since we do not have rcu_cancel() it simply sets CB_REPLAY to instruct cb_rcu_func() to reschedule itself.
Quoted 2 lines
This works because if ->gp_count is non-zero and ->cb_state is
CB_IDLE, there must be anxxx_exit()in our future.
Yes, but ->cb_state doesn't really matter if ->gp_count != 0 in xxx_exit() or cb_rcu_func() (except it can't be CB_IDLE in cb_rcu_func).
Quoted 14 lines
Quoted 13 lines
void
xxx_exit(struct xxx_struct *xxx){ spin_lock_irq(&xxx->xxx_lock); if (!--xxx->gp_count) { if (xxx->cb_state == CB_IDLE) { xxx->cb_state = CB_PENDING; call_rcu_sched(&xxx->cb_head, cb_rcu_func); } else if (xxx->cb_state == CB_PENDING) { xxx->cb_state = CB_REPLAY; } } spin_unlock_irq(&xxx->xxx_lock); }
Quoted 7 lines
Then we also have something like this?
bool
xxx_readers_fastpath_ok(struct xxx_struct *xxx){ BUG_ON(!rcu_read_lock_sched_held()); return xxx->gp_state == GP_IDLE; }
Yes, this is what xxx_is_idle() does (ignoring BUG_ON). It actually checks xxx->gp_state == 0, this is just to avoid the unnecessary export of GP_* enum.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Steven Rostedt
maintainerRe: [RFC] introduce synchronize_sched_{enter,exit}()
In reply to Oleg NesterovQuoted 5 lines · On Sun, 29 Sep 2013 20:36:34 +0200 Oleg Nesterov <[email protected]> wrote:
Why? Say,
percpu_rw_semaphore, or upcoming changes inget_online_cpus(), (Peter, I think they should be unified anyway, but lets ignore this for now). Orfreeze_super()(which currently looks buggy), perhaps something else. This pattern
Just so I'm clear to what you are trying to implement... This is to handle the case (as Paul said) to see changes to state by RCU and back again? That is, it isn't enough to see that the state changed to something (like SLOW MODE), but we also need a way to see it change back?
With get_online_cpus(), we need to see the state where it changed to "performing hotplug" where holders need to go into the slow path, and then also see the state change to "no longe performing hotplug" and the holders now go back to fast path. Is this the rational for this email?
Thanks,
-- Steve
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC] introduce synchronize_sched_{enter,exit}()
Quoted 11 lines · Steven Rostedt
Quoted 5 lines · On Sun, 29 Sep 2013 20:36:34 +0200 Oleg Nesterov <[email protected]> wrote:
Why? Say,
percpu_rw_semaphore, or upcoming changes inget_online_cpus(), (Peter, I think they should be unified anyway, but lets ignore this for now). Orfreeze_super()(which currently looks buggy), perhaps something else. This pattern
Quoted 5 lines
Just so I'm clear to what you are trying to implement... This is to handle the case (as Paul said) to see changes to state by RCU and back again? That is, it isn't enough to see that the state changed to something (like SLOW MODE), but we also need a way to see it change back?
Suppose this code was applied as is. Now we can change percpu_rwsem, see the "patch" below. (please ignore _expedited in the current code).
This immediately makes percpu_up_write() much faster, it no longer blocks. And the contending writers (or even the same writer which takes it again) can avoid synchronize_sched() in percpu_down_write().
And to remind, we can add xxx_struct->exclusive (or add the argument to xxx_enter/exit), and then (with some other changes) we can kill percpu_rw_semaphore->rw_sem.
Quoted 4 lines
With
get_online_cpus(), we need to see the state where it changed to "performing hotplug" where holders need to go into the slow path, and then also see the state change to "no longe performing hotplug" and the holders now go back to fast path. Is this the rational for this email?
The same. cpu_hotplug_begin/end (I mean the code written by Peter) can be changed to use xxx_enter/exit.
Oleg.
include/linux/percpu-rwsem.h +1 -1
@@ -8,8 +8,8 @@#include <linux/lockdep.h> struct percpu_rw_semaphore { +xxx_struct xxx;unsigned int __percpu *fast_read_ctr; -atomic_t write_ctr;struct rw_semaphore rw_sem;atomic_t slow_read_ctr;wait_queue_head_t write_waitq;
lib/percpu-rwsem.c +12 -19
@@ -17,7 +17,7 @@ int __percpu_init_rwsem(struct percpu_rw/* ->rw_sem represents the whole percpu_rw_semaphore for lockdep */ __init_rwsem (&brw->rw_sem, name, rwsem_key); -atomic_set (&brw->write_ctr,0 ); + xxx_init(&brw->xxx, ...);atomic_set (&brw->slow_read_ctr,0 );init_waitqueue_head (&brw->write_waitq);return 0 ;
@@ -25,6 +25,14 @@ int __percpu_init_rwsem(struct percpu_rwvoid percpu_free_rwsem (struct percpu_rw_semaphore *brw) { +might_sleep (); + +// pseudo code which needs another simple xxx_ helper +if (xxx->gp_state ==GP_REPLAY ) + xxx->gp_state ==GP_PENDING ; +if (xxx->gp_state) +synchronize_sched (); +free_percpu (brw->fast_read_ctr); brw->fast_read_ctr =NULL ;/* catch use after free bugs */ }
@@ -57,7 +65,7 @@ static bool update_fast_ctr(struct percpbool success = false;preempt_disable (); -if (likely (!atomic_read (&brw->write_ctr))) { +if (likely (xxx_is_idle (&brw->xxx))) {__this_cpu_add (*brw->fast_read_ctr, val); success = true; }
@@ -126,20 +134,7 @@ static int clear_fast_ctr(struct percpu_*/ void percpu_down_write (struct percpu_rw_semaphore *brw) { -/* tell update_fast_ctr() there is a pending writer */ -atomic_inc (&brw->write_ctr); -/* -* 1. Ensures that write_ctr != 0 is visible to any down_read/up_read -* so that update_fast_ctr() can't succeed. -* -* 2. Ensures we see the result of every previous this_cpu_add() in -* update_fast_ctr(). -* -* 3. Ensures that if any reader has exited its critical section via -* fast-path, it executes a full memory barrier before we return. -* See R_W case in the comment above update_fast_ctr(). -*/ -synchronize_sched_expedited (); +xxx_enter (&brw->xxx);/* exclude other writers, and block the new readers completely */ down_write (&brw->rw_sem);
@@ -159,7 +154,5 @@ void percpu_up_write(struct percpu_rw_se* Insert the barrier before the next fast-path in down_read, * see W_R case in the comment above update_fast_ctr(). */ -synchronize_sched_expedited (); -/* the last writer unblocks update_fast_ctr() */ -atomic_dec (&brw->write_ctr); +xxx_exit (); }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [RFC] introduce synchronize_sched_{enter,exit}()
In reply to Oleg NesterovQuoted 3 lines · Oleg Nesterov
Why? Say,
percpu_rw_semaphore, or upcoming changes inget_online_cpus(), (Peter, I think they should be unified anyway, but lets ignore this for now).
If you think the percpu_rwsem users can benefit sure.. So far its good I didn't go the percpu_rwsem route for it looks like we got something better at the end of it ;-)
Quoted 12 lines
Or
freeze_super()(which currently looks buggy), perhaps something else. This patternwriter: state = SLOW_MODE; synchronize_rcu/sched(); reader: preempt_disable(); // or rcu_read_lock(); if (state != SLOW_MODE) ...
Quoted 1 line
is quite common.
Well, if we make percpu_rwsem the defacto container of the pattern and use that throughout, we'd have only a single implementation and don't need the abstraction.
That said; we could still use the idea proposed; so let me take a look.
Quoted 53 lines
// .h -----------------------------------------------------------------------
struct xxx_struct { int gp_state; int gp_count; wait_queue_head_t gp_waitq; int cb_state; struct rcu_head cb_head; };static inline bool
xxx_is_idle(struct xxx_struct *xxx){ return !xxx->gp_state; /* GP_IDLE */ }extern void
xxx_enter(struct xxx_struct *xxx);
extern voidxxx_exit(struct xxx_struct *xxx);// .c -----------------------------------------------------------------------
enum {
GP_IDLE= 0,GP_PENDING,GP_PASSED};enum {
CB_IDLE= 0,CB_PENDING,CB_REPLAY};#define
xxx_lockgp_waitq.lockvoid
xxx_enter(struct xxx_struct *xxx){ bool need_wait, need_sync; spin_lock_irq(&xxx->xxx_lock); need_wait = xxx->gp_count++; need_sync = xxx->gp_state == GP_IDLE; if (need_sync) xxx->gp_state = GP_PENDING; spin_unlock_irq(&xxx->xxx_lock); BUG_ON(need_wait && need_sync); } if (need_sync) { synchronize_sched(); xxx->gp_state = GP_PASSED; wake_up_all(&xxx->gp_waitq); } else if (need_wait) { wait_event(&xxx->gp_waitq, xxx->gp_state == GP_PASSED); } else { BUG_ON(xxx->gp_state != GP_PASSED); } }static void
cb_rcu_func(struct rcu_head *rcu)
Quoted 10 lines
{
struct xxx_struct *xxx = container_of(rcu, struct xxx_struct, cb_head);
long flags;
BUG_ON(xxx->gp_state != GP_PASSED);
BUG_ON(xxx->cb_state == CB_IDLE);
spin_lock_irqsave(&xxx->xxx_lock, flags);
if (xxx->gp_count) {
xxx->cb_state = CB_IDLE;This seems to be when a new xxx_begin() has happened after our last xxx_end() and the sync_sched() from xxx_begin() merges with the xxx_end() one and we're done.
Quoted 3 lines
} else if (xxx->cb_state == CB_REPLAY) {
xxx->cb_state = CB_PENDING;
call_rcu_sched(&xxx->cb_head, cb_rcu_func);A later xxx_exit() has happened, and we need to requeue to catch a later GP.
Quoted 3 lines
} else {
xxx->cb_state = CB_IDLE;
xxx->gp_state = GP_IDLE;Nothing fancy happened and we're done.
Quoted 5 lines
} spin_unlock_irqrestore(&xxx->xxx_lock, flags); }void
xxx_exit(struct xxx_struct *xxx)
Quoted 12 lines
{
spin_lock_irq(&xxx->xxx_lock);
if (!--xxx->gp_count) {
if (xxx->cb_state == CB_IDLE) {
xxx->cb_state = CB_PENDING;
call_rcu_sched(&xxx->cb_head, cb_rcu_func);
} else if (xxx->cb_state == CB_PENDING) {
xxx->cb_state = CB_REPLAY;
}
}
spin_unlock_irq(&xxx->xxx_lock);
}So I don't immediately see the point of the concurrent write side; percpu_rwsem wouldn't allow this and afaict neither would freeze_super().
Other than that; yes this makes sense if you care about write side performance and I think its solid.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [RFC] introduce synchronize_sched_{enter,exit}()
Quoted 48 lines · Peter Zijlstra
Quoted 12 lines
static void
cb_rcu_func(struct rcu_head *rcu){ struct xxx_struct *xxx = container_of(rcu, struct xxx_struct, cb_head); long flags; BUG_ON(xxx->gp_state != GP_PASSED); BUG_ON(xxx->cb_state == CB_IDLE); spin_lock_irqsave(&xxx->xxx_lock, flags); if (xxx->gp_count) { xxx->cb_state = CB_IDLE;This seems to be when a new
xxx_begin()has happened after our lastxxx_end()and thesync_sched()fromxxx_begin()merges with thexxx_end()one and we're done.Quoted 3 lines
} else if (xxx->cb_state == CB_REPLAY) { xxx->cb_state = CB_PENDING; call_rcu_sched(&xxx->cb_head, cb_rcu_func);A later
xxx_exit()has happened, and we need to requeue to catch a later GP.Quoted 3 lines
} else { xxx->cb_state = CB_IDLE; xxx->gp_state = GP_IDLE;Nothing fancy happened and we're done.
Quoted 5 lines
} spin_unlock_irqrestore(&xxx->xxx_lock, flags); }void
xxx_exit(struct xxx_struct *xxx)Quoted 12 lines
{ spin_lock_irq(&xxx->xxx_lock); if (!--xxx->gp_count) { if (xxx->cb_state == CB_IDLE) { xxx->cb_state = CB_PENDING; call_rcu_sched(&xxx->cb_head, cb_rcu_func); } else if (xxx->cb_state == CB_PENDING) { xxx->cb_state = CB_REPLAY; } } spin_unlock_irq(&xxx->xxx_lock); }
Quoted 6 lines
So I don't immediately see the point of the concurrent write side;
percpu_rwsemwouldn't allow this and afaict neither wouldfreeze_super().Other than that; yes this makes sense if you care about write side performance and I think its solid.
Hmm, wait. I don't see how this is equivalent to:
xxx_end()
{
synchronize_sched();
atomic_dec(&xxx->counter);
}For that we'd have to decrement xxx->gp_count from cb_rcu_func(), wouldn't we?
Without that there's no guarantee the fast path readers will have a MB to observe the write critical section, unless I'm completely missing something obviuos here.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [RFC] introduce synchronize_sched_{enter,exit}()
Quoted 6 lines · Peter Zijlstra
For that we'd have to decrement
xxx->gp_countfromcb_rcu_func(), wouldn't we?Without that there's no guarantee the fast path readers will have a MB to observe the write critical section, unless I'm completely missing something obviuos here.
Duh.. we should be looking at gp_state like Paul said.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC] introduce synchronize_sched_{enter,exit}()
Quoted 10 lines · Peter Zijlstra
Quoted 6 lines · Peter Zijlstra
For that we'd have to decrement
xxx->gp_countfromcb_rcu_func(), wouldn't we?Without that there's no guarantee the fast path readers will have a MB to observe the write critical section, unless I'm completely missing something obviuos here.
Duh.. we should be looking at
gp_statelike Paul said.
Yes, yes, that is why we have xxx_is_idle(). Its name is confusing even ignoring "xxx".
OK, I'll try to invent the naming (but I'd like to hear suggestions ;) and send the patch. I am going to add "exclusive" and "rcu_domain/ops" later, currently percpu_rw_semaphore needs ->rw_sem anyway.
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC] introduce synchronize_sched_{enter,exit}()
In reply to Peter ZijlstraQuoted 9 lines · Peter Zijlstra
Quoted 3 lines · Oleg Nesterov
Why? Say,
percpu_rw_semaphore, or upcoming changes inget_online_cpus(), (Peter, I think they should be unified anyway, but lets ignore this for now).If you think the
percpu_rwsemusers can benefit sure.. So far its good I didn't go thepercpu_rwsemroute for it looks like we got something better at the end of it ;-)
I think you could simply improve percpu_rwsem instead. Once we add task_struct->cpuhp_ctr percpu_rwsem and get_online_cpus/hotplug_begin becomes absolutely congruent.
OTOH, it would be simpler to change hotplug first, then copy-and-paste the improvents into percpu_rwsem, then see if we can simply convert cpu_hotplug_begin/end into percpu_down/up_write.
Quoted 2 lines
Well, if we make
percpu_rwsemthe defacto container of the pattern and use that throughout, we'd have only a single implementation
Not sure. I think it can have other users. But even if not, please look at "struct sb_writers". Yes, I believe it makes sense to use percpu_rwsem here, but note that it is actually array of semaphores. I do not think each element needs its own xxx_struct.
Quoted 2 lines
and don't
need the abstraction.
And even if struct percpu_rw_semaphore will be the only container of xxx_struct, I think the code looks better and more understandable this way, exactly because it adds the new abstraction layer. Performance-wise this should be free.
Quoted 8 lines
Quoted 8 lines
static void
cb_rcu_func(struct rcu_head *rcu){ struct xxx_struct *xxx = container_of(rcu, struct xxx_struct, cb_head); long flags; BUG_ON(xxx->gp_state != GP_PASSED); BUG_ON(xxx->cb_state == CB_IDLE);
Quoted 7 lines
Quoted 3 lines
spin_lock_irqsave(&xxx->xxx_lock, flags); if (xxx->gp_count) { xxx->cb_state = CB_IDLE;This seems to be when a new
xxx_begin()has happened after our lastxxx_end()and thesync_sched()fromxxx_begin()merges with thexxx_end()one and we're done.
Yes,
Quoted 6 lines
Quoted 3 lines
} else if (xxx->cb_state == CB_REPLAY) { xxx->cb_state = CB_PENDING; call_rcu_sched(&xxx->cb_head, cb_rcu_func);A later
xxx_exit()has happened, and we need to requeue to catch a later GP.
Exactly.
Quoted 3 lines
So I don't immediately see the point of the concurrent write side;
percpu_rwsemwouldn't allow this and afaict neither wouldfreeze_super().
Oh I disagree. Even ignoring the fact I believe xxx_struct itself can have more users (I can be wrong of course), I do think that percpu_down_write_nonexclusive() makes sense (except "exclusive" should be the argument of percpu_init_rwsem). And in fact the initial implementation I sent didn't even has the "exclusive" mode.
Please look at uprobes (currently the only user). We do not really need the global write-lock, we can do the per-uprobe locking. However, every caller needs to block the percpu_down_read() callers (dup_mmap).
Quoted 2 lines
Other than that; yes this makes sense if you care about write side performance and I think its solid.
Great ;)
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [RFC] introduce synchronize_sched_{enter,exit}()
In reply to Oleg Nesterov--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC] introduce synchronize_sched_{enter,exit}()
Quoted 1 line · Peter Zijlstra
Fully agreed! :-)
Ingo
Peter Zijlstra
maintainerRe: [RFC] introduce synchronize_sched_{enter,exit}()
Quoted 6 lines · Ingo Molnar
Quoted 1 line · Peter Zijlstra
Fully agreed! :-)
haha.. never realized I send that email completely empty. It was supposed to contain the patch I later send as 2/3.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>