free order-0 pages through PCP in page_frag_free() and cleanup
Subsystems (4)
cover free order-0 pages through PCP in page_frag_free() and cleanup
This is a resend of the two patches.
Patch 1 is the same as:
[PATCH v2 1/2] mm/page_alloc: free order-0 pages through PCP in page_frag_free()
https://lkml.kernel.org/r/[email protected]
With one more ack from Tariq Toukan.
Patch 2 is the same as:
[PATCH v3 2/2] mm/page_alloc: use a single function to free page
https://lkml.kernel.org/r/[email protected]
With some changelog rewording.
Applies on top of v4.20-rc2-mmotm-2018-11-16-14-52.
Aaron Lu (2):
mm/page_alloc: free order-0 pages through PCP in page_frag_free() mm/page_alloc: use a single function to free page
mm/page_alloc.c | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-)
Signature · 1 line
2.17.2
PATCH RESEND v2 1/2 mm/page_alloc: free order-0 pages through PCP in page_frag_free()
page_frag_free() calls __free_pages_ok() to free the page back to Buddy. This is OK for high order page, but for order-0 pages, it misses the optimization opportunity of using Per-Cpu-Pages and can cause zone lock contention when called frequently.
Paweł Staszewski recently shared his result of 'how Linux kernel handles normal traffic'[1] and from perf data, Jesper Dangaard Brouer found the lock contention comes from page allocator:
mlx5e_poll_tx_cq
|
--16.34%--napi_consume_skb
|
|--12.65%--__free_pages_ok
| |
| --11.86%--free_one_page
| |
| |--10.10%--queued_spin_lock_slowpath
| |
| --0.65%--_raw_spin_lock
|
|--1.55%--page_frag_free
|
--1.44%--skb_release_dataJesper explained how it happened: mlx5 driver RX-page recycle mechanism is not effective in this workload and pages have to go through the page allocator. The lock contention happens during mlx5 DMA TX completion cycle. And the page allocator cannot keep up at these speeds.[2]
I thought that __free_pages_ok() are mostly freeing high order pages and thought this is an lock contention for high order pages but Jesper explained in detail that __free_pages_ok() here are actually freeing order-0 pages because mlx5 is using order-0 pages to satisfy its page pool allocation request.[3]
The free path as pointed out by Jesper is:skb_free_head()
-> skb_free_frag()
-> page_frag_free()
And the pages being freed on this path are order-0 pages.
Fix this by doing similar things as in __page_frag_cache_drain() - send the being freed page to PCP if it's an order-0 page, or directly to Buddy if it is a high order page.
With this change, Paweł hasn't noticed lock contention yet in his workload and Jesper has noticed a 7% performance improvement using a micro benchmark and lock contention is gone. Ilias' test on a 'low' speed 1Gbit interface on an cortex-a53 shows ~11% performance boost testing with 64byte packets and __free_pages_ok() disappeared from perf top.
[1]: https://www.spinics.net/lists/netdev/msg531362.html
[2]: https://www.spinics.net/lists/netdev/msg531421.html
[3]: https://www.spinics.net/lists/netdev/msg531556.html
--- mm/page_alloc.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
mm/page_alloc.c +8 -2
@@ -4677,8 +4677,14 @@ void page_frag_free(void *addr) {struct page *page =virt_to_head_page (addr); -if (unlikely (put_page_testzero (page))) -__free_pages_ok (page,compound_order (page)); +if (unlikely (put_page_testzero (page))) { +unsigned int order =compound_order (page); + +if (order ==0 ) +free_unref_page (page); +else +__free_pages_ok (page, order); + } }EXPORT_SYMBOL (page_frag_free);
Signature · 1 line
2.17.2
Re: [PATCH v2 RESEND 1/2] mm/page_alloc: free order-0 pages through PCP in page_frag_free()
Quoted 58 lines · Aaron Lu
page_frag_free()calls__free_pages_ok()to free the page back to Buddy. This is OK for high order page, but for order-0 pages, it misses the optimization opportunity of using Per-Cpu-Pages and can cause zone lock contention when called frequently.Paweł Staszewski recently shared his result of 'how Linux kernel handles normal traffic'[1] and from perf data, Jesper Dangaard Brouer found the lock contention comes from page allocator:
mlx5e_poll_tx_cq | --16.34%--napi_consume_skb | |--12.65%--__free_pages_ok | | | --11.86%--free_one_page | | | |--10.10%--queued_spin_lock_slowpath | | | --0.65%--_raw_spin_lock | |--1.55%--page_frag_free | --1.44%--skb_release_dataJesper explained how it happened: mlx5 driver RX-page recycle mechanism is not effective in this workload and pages have to go through the page allocator. The lock contention happens during mlx5 DMA TX completion cycle. And the page allocator cannot keep up at these speeds.[2]
I thought that
__free_pages_ok()are mostly freeing high order pages and thought this is an lock contention for high order pages but Jesper explained in detail that__free_pages_ok()here are actually freeing order-0 pages because mlx5 is using order-0 pages to satisfy its page pool allocation request.[3]The free path as pointed out by Jesper is:
skb_free_head()
->skb_free_frag()-> page_frag_free()And the pages being freed on this path are order-0 pages.
Fix this by doing similar things as in
__page_frag_cache_drain()- send the being freed page to PCP if it's an order-0 page, or directly to Buddy if it is a high order page.With this change, Paweł hasn't noticed lock contention yet in his workload and Jesper has noticed a 7% performance improvement using a micro benchmark and lock contention is gone. Ilias' test on a 'low' speed 1Gbit interface on an cortex-a53 shows ~11% performance boost testing with 64byte packets and
__free_pages_ok()disappeared from perf top.[1]: https://www.spinics.net/lists/netdev/msg531362.html
[2]: https://www.spinics.net/lists/netdev/msg531421.html
[3]: https://www.spinics.net/lists/netdev/msg531556.html
Quoted 9 lines
Reported-by Paweł Staszewski <[email protected]>Analysed-by Jesper Dangaard Brouer <[email protected]>Acked-by Vlastimil Babka <[email protected]>Acked-by Mel Gorman <[email protected]>Acked-by Ilias Apalodimas <[email protected]>Tested-by Ilias Apalodimas <[email protected]>Acked-by Alexander Duyck <[email protected]>Acked-by Tariq Toukan <[email protected]
missing '>' sign in my email tag.
Quoted 2 lines
Signed-off-by Aaron Lu <[email protected]>---
Re: [PATCH v2 RESEND 1/2] mm/page_alloc: free order-0 pages through PCP in page_frag_free()
Quoted 71 lines · Tariq Toukan
Quoted 58 lines · Aaron Lu
page_frag_free()calls__free_pages_ok()to free the page back to Buddy. This is OK for high order page, but for order-0 pages, it misses the optimization opportunity of using Per-Cpu-Pages and can cause zone lock contention when called frequently.Paweł Staszewski recently shared his result of 'how Linux kernel handles normal traffic'[1] and from perf data, Jesper Dangaard Brouer found the lock contention comes from page allocator:
mlx5e_poll_tx_cq | --16.34%--napi_consume_skb | |--12.65%--__free_pages_ok | | | --11.86%--free_one_page | | | |--10.10%--queued_spin_lock_slowpath | | | --0.65%--_raw_spin_lock | |--1.55%--page_frag_free | --1.44%--skb_release_dataJesper explained how it happened: mlx5 driver RX-page recycle mechanism is not effective in this workload and pages have to go through the page allocator. The lock contention happens during mlx5 DMA TX completion cycle. And the page allocator cannot keep up at these speeds.[2]
I thought that
__free_pages_ok()are mostly freeing high order pages and thought this is an lock contention for high order pages but Jesper explained in detail that__free_pages_ok()here are actually freeing order-0 pages because mlx5 is using order-0 pages to satisfy its page pool allocation request.[3]The free path as pointed out by Jesper is:
skb_free_head()
->skb_free_frag()-> page_frag_free()And the pages being freed on this path are order-0 pages.
Fix this by doing similar things as in
__page_frag_cache_drain()- send the being freed page to PCP if it's an order-0 page, or directly to Buddy if it is a high order page.With this change, Paweł hasn't noticed lock contention yet in his workload and Jesper has noticed a 7% performance improvement using a micro benchmark and lock contention is gone. Ilias' test on a 'low' speed 1Gbit interface on an cortex-a53 shows ~11% performance boost testing with 64byte packets and
__free_pages_ok()disappeared from perf top.[1]: https://www.spinics.net/lists/netdev/msg531362.html
[2]: https://www.spinics.net/lists/netdev/msg531421.html
[3]: https://www.spinics.net/lists/netdev/msg531556.htmlQuoted 9 lines
Reported-by Paweł Staszewski <[email protected]>Analysed-by Jesper Dangaard Brouer <[email protected]>Acked-by Vlastimil Babka <[email protected]>Acked-by Mel Gorman <[email protected]>Acked-by Ilias Apalodimas <[email protected]>Tested-by Ilias Apalodimas <[email protected]>Acked-by Alexander Duyck <[email protected]>Acked-by Tariq Toukan <[email protected]
Quoted 1 line
missing '>' sign in my email tag.
Sorry about that, will fix this and resend.
Quoted 2 lines
Quoted 2 lines
Signed-off-by Aaron Lu <[email protected]>---
PATCH RESEND v2 1/2 mm/page_alloc: free order-0 pages through PCP in page_frag_free()
In reply to Aaron Lupage_frag_free() calls __free_pages_ok() to free the page back to Buddy. This is OK for high order page, but for order-0 pages, it misses the optimization opportunity of using Per-Cpu-Pages and can cause zone lock contention when called frequently.
Paweł Staszewski recently shared his result of 'how Linux kernel handles normal traffic'[1] and from perf data, Jesper Dangaard Brouer found the lock contention comes from page allocator:
mlx5e_poll_tx_cq
|
--16.34%--napi_consume_skb
|
|--12.65%--__free_pages_ok
| |
| --11.86%--free_one_page
| |
| |--10.10%--queued_spin_lock_slowpath
| |
| --0.65%--_raw_spin_lock
|
|--1.55%--page_frag_free
|
--1.44%--skb_release_dataJesper explained how it happened: mlx5 driver RX-page recycle mechanism is not effective in this workload and pages have to go through the page allocator. The lock contention happens during mlx5 DMA TX completion cycle. And the page allocator cannot keep up at these speeds.[2]
I thought that __free_pages_ok() are mostly freeing high order pages and thought this is an lock contention for high order pages but Jesper explained in detail that __free_pages_ok() here are actually freeing order-0 pages because mlx5 is using order-0 pages to satisfy its page pool allocation request.[3]
The free path as pointed out by Jesper is:skb_free_head()
-> skb_free_frag()
-> page_frag_free()
And the pages being freed on this path are order-0 pages.
Fix this by doing similar things as in __page_frag_cache_drain() - send the being freed page to PCP if it's an order-0 page, or directly to Buddy if it is a high order page.
With this change, Paweł hasn't noticed lock contention yet in his workload and Jesper has noticed a 7% performance improvement using a micro benchmark and lock contention is gone. Ilias' test on a 'low' speed 1Gbit interface on an cortex-a53 shows ~11% performance boost testing with 64byte packets and __free_pages_ok() disappeared from perf top.
[1]: https://www.spinics.net/lists/netdev/msg531362.html
[2]: https://www.spinics.net/lists/netdev/msg531421.html
[3]: https://www.spinics.net/lists/netdev/msg531556.html
---
update: fix Tariq's email tag.
mm/page_alloc.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
mm/page_alloc.c +8 -2
@@ -4677,8 +4677,14 @@ void page_frag_free(void *addr) {struct page *page =virt_to_head_page (addr); -if (unlikely (put_page_testzero (page))) -__free_pages_ok (page,compound_order (page)); +if (unlikely (put_page_testzero (page))) { +unsigned int order =compound_order (page); + +if (order ==0 ) +free_unref_page (page); +else +__free_pages_ok (page, order); + } }EXPORT_SYMBOL (page_frag_free);
Signature · 1 line
2.17.2
Re: [PATCH v2 RESEND update 1/2] mm/page_alloc: free order-0 pages through PCP in page_frag_free()
Quoted 98 lines
page_frag_free()calls__free_pages_ok()to free the page back to Buddy. This is OK for high order page, but for order-0 pages, it misses the optimization opportunity of using Per-Cpu-Pages and can cause zone lock contention when called frequently.Paweł Staszewski recently shared his result of 'how Linux kernel handles normal traffic'[1] and from perf data, Jesper Dangaard Brouer found the lock contention comes from page allocator:
mlx5e_poll_tx_cq
|
--16.34%--napi_consume_skb| |--12.65%--__free_pages_ok | | | --11.86%--free_one_page | | | |--10.10%--queued_spin_lock_slowpath | | | --0.65%--_raw_spin_lock | |--1.55%--page_frag_free | --1.44%--skb_release_dataJesper explained how it happened: mlx5 driver RX-page recycle mechanism is not effective in this workload and pages have to go through the page allocator. The lock contention happens during mlx5 DMA TX completion cycle. And the page allocator cannot keep up at these speeds.[2]
I thought that
__free_pages_ok()are mostly freeing high order pages and thought this is an lock contention for high order pages but Jesper explained in detail that__free_pages_ok()here are actually freeing order-0 pages because mlx5 is using order-0 pages to satisfy its page pool allocation request.[3]The free path as pointed out by Jesper is:
skb_free_head()
->skb_free_frag()-> page_frag_free()And the pages being freed on this path are order-0 pages.
Fix this by doing similar things as in
__page_frag_cache_drain()- send the being freed page to PCP if it's an order-0 page, or directly to Buddy if it is a high order page.With this change, Paweł hasn't noticed lock contention yet in his workload and Jesper has noticed a 7% performance improvement using a micro benchmark and lock contention is gone. Ilias' test on a 'low' speed 1Gbit interface on an cortex-a53 shows ~11% performance boost testing with 64byte packets and
__free_pages_ok()disappeared from perf top.[1]: https://www.spinics.net/lists/netdev/msg531362.html
[2]: https://www.spinics.net/lists/netdev/msg531421.html
[3]: https://www.spinics.net/lists/netdev/msg531556.htmlReported-by Paweł Staszewski <[email protected]>Analysed-by Jesper Dangaard Brouer <[email protected]>Acked-by Vlastimil Babka <[email protected]>Acked-by Mel Gorman <[email protected]>Acked-by Ilias Apalodimas <[email protected]>Tested-by Ilias Apalodimas <[email protected]>Acked-by Alexander Duyck <[email protected]>Acked-by Tariq Toukan <[email protected]>Signed-off-by Aaron Lu <[email protected]>---
update: fix Tariq's email tag.mm/page_alloc.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)mm/page_alloc.c +8
-3@@ -4677,8 +4677,14 @@ void page_frag_free(void *addr) {struct page *page =virt_to_head_page (addr); -if (unlikely (put_page_testzero (page))) -__free_pages_ok (page,compound_order (page)); +if (unlikely (put_page_testzero (page))) { +unsigned int order =compound_order (page); + +if (order ==0 ) +free_unref_page (page); +else +__free_pages_ok (page, order); + } }EXPORT_SYMBOL (page_frag_free); --
Quoted 1 line
2.17.2
A good optimization for zero order allocations.
Thanks,
Pankaj
Andrew Morton
maintainerRe: [PATCH v2 RESEND update 1/2] mm/page_alloc: free order-0 pages through PCP in page_frag_free()
In reply to Aaron LuQuoted 5 lines · On Tue, 20 Nov 2018 09:45:44 +0800 Aaron Lu <[email protected]> wrote:
page_frag_free()calls__free_pages_ok()to free the page back to Buddy. This is OK for high order page, but for order-0 pages, it misses the optimization opportunity of using Per-Cpu-Pages and can cause zone lock contention when called frequently.
Looks nice to me. Let's tell our readers why we're doing this.
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-free-order-0-pages-through-pcp-in-page_frag_free-fix +1 -1
@@ -4684,7 +4684,7 @@ void page_frag_free(void *addr)if (unlikely (put_page_testzero (page))) {unsigned int order =compound_order (page); -if (order ==0 ) +if (order ==0 )/* Via pcp? */ free_unref_page (page);else __free_pages_ok (page, order);
_
PATCH RESEND v3 2/2 mm/page_alloc: use a single function to free page
In reply to Aaron LuThere are multiple places of freeing a page, they all do the same things so a common function can be used to reduce code duplicate.
It also avoids bug fixed in one function but left in another.
--- mm/page_alloc.c | 37 ++++++++++++++----------------------- 1 file changed, 14 insertions(+), 23 deletions(-)
mm/page_alloc.c +14 -23
@@ -4547,16 +4547,19 @@ unsigned long get_zeroed_page(gfp_t gfp_mask) }EXPORT_SYMBOL (get_zeroed_page); -void __free_pages (struct page *page,unsigned int order) +static inline void free_the_page (struct page *page,unsigned int order) { -if (put_page_testzero (page)) { -if (order ==0 ) -free_unref_page (page); -else -__free_pages_ok (page, order); - } +if (order ==0 ) +free_unref_page (page); +else +__free_pages_ok (page, order); } +void __free_pages (struct page *page,unsigned int order) +{ +if (put_page_testzero (page)) +free_the_page (page, order); +}EXPORT_SYMBOL (__free_pages);void free_pages (unsigned long addr,unsigned int order)
@@ -4605,14 +4608,8 @@ void __page_frag_cache_drain(struct page *page, unsigned int count) {VM_BUG_ON_PAGE (page_ref_count (page) ==0 , page); -if (page_ref_sub_and_test (page, count)) { -unsigned int order =compound_order (page); - -if (order ==0 ) -free_unref_page (page); -else -__free_pages_ok (page, order); - } +if (page_ref_sub_and_test (page, count)) +free_the_page (page,compound_order (page)); }EXPORT_SYMBOL (__page_frag_cache_drain);
@@ -4677,14 +4674,8 @@ void page_frag_free(void *addr) {struct page *page =virt_to_head_page (addr); -if (unlikely (put_page_testzero (page))) { -unsigned int order =compound_order (page); - -if (order ==0 ) -free_unref_page (page); -else -__free_pages_ok (page, order); - } +if (unlikely (put_page_testzero (page))) +free_the_page (page,compound_order (page)); }EXPORT_SYMBOL (page_frag_free);
Signature · 1 line
2.17.2