RFC Fast noirq bulk page allocator v2r7
Get series with b4
b4 am [email protected]Versions newest first
- v1 RFC Latest4 patches 12 repliesIn review
Changelog since v1 from the cover letter
o Remove a scheduler point from the allocation path
o Finalise the bulk allocator and test it
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/4 | mm, page_alloc: Split buffered_rmqueuemm/ | +77 | 2 replies | |
| 2/4 | mm, page_alloc: Split alloc_pages_nodemaskmm/ | +49 | 2 replies | |
| 3/4 | mm, page_allocator: Only use per-cpu allocator for irq-safe requestsmm/ | +17 | 3 replies | |
| 4/4 | mm, page_alloc: Add a bulk page allocatorinclude/ mm/ | +139 | 4 replies |
Reviews trailers per patch
All 4 patches have a Reviewed-by or Acked-by.
| Patch | Hillf Danton | Jesper Dangaard Brouer |
|---|---|---|
| 1/4 | A-b | A-b |
| 2/4 | A-b | A-b |
| 3/4 | A-b | A-b |
| 4/4 | A-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [RFC PATCH 0/4] Fast noirq bulk page allocator v2r7
The biggest changes are in the final patch. In v1, it was a rough untested prototype. This version corrected a number of issues, tested it and includes a comparison between bulk allocating pages and allocating them one at a time. While there are still no in-kernel users, it is hoped that the bulk API would convince network drivers to avoid using high-order allocations. One slight caveat is that there still may be an advantage to doing the coherent setup on a high-order page instead of a list of order-0 pages. If that is the case, it would need to be covered by Jesper's generic page pool allocator.
Changelog since v1
o Remove a scheduler point from the allocation path
o Finalise the bulk allocator and test it
This series is motivated by a conversation led by Jesper Dangaard Brouer at the last LSF/MM proposing a generic page pool for DMA-coherent pages. Part of his motivation was due to the overhead of allocating multiple order-0 that led some drivers to use high-order allocations and splitting them which can be very slow if high-order pages are unavailable. This long-overdue series aims to show that raw bulk page allocation can be achieved relatively easily without introducing a completely new allocator. A new generic page pool allocator would then ideally focus on just the DMA-coherent part.
The first two patches in the series restructure the allocator such that it's relatively easy to build a bulk page allocator. The third patch alters the per-cpu alloctor to make it exclusive to !irq requests. This cuts allocation/free overhead by roughly 30% but it may not be noticable to anyone other than users of high-speed networks (I'm not one). The fourth patch introduces a bulk page allocator with no in-kernel users as an example for Jesper and others who want to build a page allocator for DMA-coherent pages. It hopefully is relatively easy to modify this API and the one core function to get the semantics they require. Note that Patch 3 is not required for patch 4 but it may be desirable if the bulk allocations happen from !IRQ context.
A comparison of costs of allocating one page at a time on the vanilla kernel vs the bulk allocator that forces the per-cpu allocator to be used from a !irq context is as follows
pagealloc
4.10.0-rc2 4.10.0-rc2
vanilla bulk-v2r7
Amean alloc-odr0-1 302.85 ( 0.00%) 106.62 ( 64.80%)
Amean alloc-odr0-2 227.85 ( 0.00%) 76.38 ( 66.48%)
Amean alloc-odr0-4 191.23 ( 0.00%) 57.23 ( 70.07%)
Amean alloc-odr0-8 167.54 ( 0.00%) 48.77 ( 70.89%)
Amean alloc-odr0-16 158.54 ( 0.00%) 45.38 ( 71.37%)
Amean alloc-odr0-32 150.46 ( 0.00%) 42.77 ( 71.57%)
Amean alloc-odr0-64 148.23 ( 0.00%) 41.00 ( 72.34%)
Amean alloc-odr0-128 145.00 ( 0.00%) 40.08 ( 72.36%)
Amean alloc-odr0-256 157.00 ( 0.00%) 56.00 ( 64.33%)
Amean alloc-odr0-512 170.00 ( 0.00%) 69.00 ( 59.41%)
Amean alloc-odr0-1024 181.00 ( 0.00%) 76.23 ( 57.88%)
Amean alloc-odr0-2048 186.00 ( 0.00%) 81.15 ( 56.37%)
Amean alloc-odr0-4096 192.92 ( 0.00%) 85.92 ( 55.46%)
Amean alloc-odr0-8192 194.00 ( 0.00%) 88.00 ( 54.64%)
Amean alloc-odr0-16384 202.15 ( 0.00%) 89.00 ( 55.97%)
Amean free-odr0-1 154.92 ( 0.00%) 55.69 ( 64.05%)
Amean free-odr0-2 115.31 ( 0.00%) 49.38 ( 57.17%)
Amean free-odr0-4 93.31 ( 0.00%) 45.38 ( 51.36%)
Amean free-odr0-8 82.62 ( 0.00%) 44.23 ( 46.46%)
Amean free-odr0-16 79.00 ( 0.00%) 45.00 ( 43.04%)
Amean free-odr0-32 75.15 ( 0.00%) 43.92 ( 41.56%)
Amean free-odr0-64 74.00 ( 0.00%) 43.00 ( 41.89%)
Amean free-odr0-128 73.00 ( 0.00%) 43.00 ( 41.10%)
Amean free-odr0-256 91.00 ( 0.00%) 60.46 ( 33.56%)
Amean free-odr0-512 108.00 ( 0.00%) 76.00 ( 29.63%)
Amean free-odr0-1024 119.00 ( 0.00%) 85.38 ( 28.25%)
Amean free-odr0-2048 125.08 ( 0.00%) 91.23 ( 27.06%)
Amean free-odr0-4096 130.00 ( 0.00%) 95.62 ( 26.45%)
Amean free-odr0-8192 130.00 ( 0.00%) 97.00 ( 25.38%)
Amean free-odr0-16384 134.46 ( 0.00%) 97.46 ( 27.52%)
Amean total-odr0-1 457.77 ( 0.00%) 162.31 ( 64.54%)
Amean total-odr0-2 343.15 ( 0.00%) 125.77 ( 63.35%)
Amean total-odr0-4 284.54 ( 0.00%) 102.62 ( 63.94%)
Amean total-odr0-8 250.15 ( 0.00%) 93.00 ( 62.82%)
Amean total-odr0-16 237.54 ( 0.00%) 90.38 ( 61.95%)
Amean total-odr0-32 225.62 ( 0.00%) 86.69 ( 61.58%)
Amean total-odr0-64 222.23 ( 0.00%) 84.00 ( 62.20%)
Amean total-odr0-128 218.00 ( 0.00%) 83.08 ( 61.89%)
Amean total-odr0-256 248.00 ( 0.00%) 116.46 ( 53.04%)
Amean total-odr0-512 278.00 ( 0.00%) 145.00 ( 47.84%)
Amean total-odr0-1024 300.00 ( 0.00%) 161.62 ( 46.13%)
Amean total-odr0-2048 311.08 ( 0.00%) 172.38 ( 44.58%)
Amean total-odr0-4096 322.92 ( 0.00%) 181.54 ( 43.78%)
Amean total-odr0-8192 324.00 ( 0.00%) 185.00 ( 42.90%)
Amean total-odr0-16384 336.62 ( 0.00%) 186.46 ( 44.61%)It's roughly a 50-70% reduction of allocation costs and roughly a halving of the overall cost of allocating/freeing batches of pages.
include/linux/gfp.h | 24 ++++ mm/page_alloc.c | 353 +++++++++++++++++++++++++++++++++++++--------------- 2 files changed, 278 insertions(+), 99 deletions(-)
Signature · 7 lines
2.11.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>