Drain remote per-cpu directly
Status from this message: [merged mm-stable] mm-page_alloc-replace-local_lock-with-normal-spinlock.patch removed from -mm tree
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v5 Latest7 patches 10 repliesApplied 7/7
- v47 patches 29 repliesSuperseded
- v16 patches 37 repliesSuperseded
Changes in v5 from the cover letter
o Fix lockdep issues in patch 7
o Refactor patch 5 to reduce churn in patches 6 and 7
o Rebase to 5.19-rc3
Patches in v5 All 7 applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/7 | mm/page_alloc: Add page->buddy_list and page->pcp_listinclude/ mm/ | +17 | 0 replies | Applied |
| 2/7 | mm/page_alloc: Use only one PCP list for THP-sized allocationsinclude/ mm/ | +9 | 0 replies | Applied |
| 3/7 | mm/page_alloc: Split out buddy removal code from rmqueue into separate helpermm/ | +47 | 0 replies | Applied |
| 4/7 | mm/page_alloc: Remove mistaken page == NULL check in rmqueuemm/ | +1 | 0 replies | Applied |
| 5/7 | mm/page_alloc: Protect PCP lists with a spinlockinclude/ mm/ | +98 | 3 replies | Applied |
| 6/7 | mm/page_alloc: Remotely drain per-cpu listsmm/ | +4 | 1 reply | Applied |
| 7/7 | mm/page_alloc: Replace local_lock with normal spinlockmm/ | +95 | 4 replies | Applied |
Reviews trailers per patch
All 7 patches have a Reviewed-by or Acked-by.
| Patch | Minchan Kim | Nicolas Saenz Julienne | Vlastimil Babka |
|---|---|---|---|
| 1/7 | T-b A-b | R-b | A-b |
| 2/7 | T-b A-b | A-b | |
| 3/7 | T-b A-b | R-b | A-b |
| 4/7 | A-b | ||
| 5/7 | R-b T-b | A-b | |
| 6/7 | A-b | ||
| 7/7 | R-b T-b | A-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.
Cover letter [PATCH v5 00/7] Drain remote per-cpu directly
This replaces the existing version on mm-unstable. While there are some fixes, this is mostly refactoring of patch 5 based on Vlastimil's feedback to reduce churn in later patches. The level of refactoring made -fix patches excessively complicated.
Changelog since v4
o Fix lockdep issues in patch 7
o Refactor patch 5 to reduce churn in patches 6 and 7
o Rebase to 5.19-rc3
Some setups, notably NOHZ_FULL CPUs, may be running realtime or latency-sensitive applications that cannot tolerate interference due to per-cpu drain work queued by __drain_all_pages(). Introduce a new mechanism to remotely drain the per-cpu lists. It is made possible by remotely locking 'struct per_cpu_pages' new per-cpu spinlocks. This has two advantages, the time to drain is more predictable and other unrelated tasks are not interrupted.
This series has the same intent as Nicolas' series "mm/page_alloc: Remote per-cpu lists drain support" -- avoid interference of a high priority task due to a workqueue item draining per-cpu page lists. While many workloads can tolerate a brief interruption, it may cause a real-time task running on a NOHZ_FULL CPU to miss a deadline and at minimum, the draining is non-deterministic.
Currently an IRQ-safe local_lock protects the page allocator per-cpu lists. The local_lock on its own prevents migration and the IRQ disabling protects from corruption due to an interrupt arriving while a page allocation is in progress.
This series adjusts the locking. A spinlock is added to struct per_cpu_pages to protect the list contents while local_lock_irq is ultimately replaced by just the spinlock in the final patch. This allows a remote CPU to safely. Follow-on work should allow the spin_lock_irqsave to be converted to spin_lock to avoid IRQs being disabled/enabled in most cases. The follow-on patch will be one kernel release later as it is relatively high risk and it'll make bisections more clear if there are any problems.
Patch 1 is a cosmetic patch to clarify when page->lru is storing buddy pages
and when it is storing per-cpu pages.
Patch 2 shrinks per_cpu_pages to make room for a spin lock. Strictly speaking
this is not necessary but it avoids per_cpu_pages consuming another cache line.
Patch 3 is a preparation patch to avoid code duplication.
Patch 4 is a minor correction.
Patch 5 uses a spin_lock to protect the per_cpu_pages contents while still
relying on local_lock to prevent migration, stabilise the pcp lookup and prevent IRQ reentrancy.
Patch 6 remote drains per-cpu pages directly instead of using a workqueue.
Patch 7 uses a normal spinlock instead of local_lock for remote draining
Nicolas Saenz Julienne (1):
mm/page_alloc: Remotely drain per-cpu lists
include/linux/mm_types.h | 5 + include/linux/mmzone.h | 12 +- mm/page_alloc.c | 386 ++++++++++++++++++++++++--------------- 3 files changed, 250 insertions(+), 153 deletions(-)
Signature · 1 line
2.35.3