mvpp2: prefetch data early
Subsystems (3)
cover mvpp2: prefetch data early
From: Matteo Croce <[email protected]>
These two patches prefetch some data from RAM so to reduce stall and speedup the packet processing.
Matteo Croce (2):
mvpp2: prefetch right address
mvpp2: prefetch page
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
Signature · 1 line
2.31.1
PATCH 1/2 mvpp2: prefetch right address
From: Matteo Croce <[email protected]>
In the RX buffer, the received data starts after a headroom used to align the IP header and to allow prepending headers efficiently. The prefetch() should take this into account, and prefetch from the very start of the received data.
We can see that ether_addr_equal_64bits(), which is the first function to access the data, drops from the top of the perf top output.
prefetch(data):
Overhead Shared Object Symbol 11.64% [kernel] [k] eth_type_trans
prefetch(data + MVPP2_MH_SIZE + MVPP2_SKB_HEADROOM):
Overhead Shared Object Symbol 13.42% [kernel] [k] build_skb 10.35% [mvpp2] [k] mvpp2_rx 9.35% [kernel] [k] __netif_receive_skb_core 8.24% [kernel] [k] kmem_cache_free 7.97% [kernel] [k] dev_gro_receive 7.68% [kernel] [k] page_pool_put_page 7.32% [kernel] [k] kmem_cache_alloc 7.09% [mvpp2] [k] mvpp2_bm_pool_put 3.36% [kernel] [k] eth_type_trans
Also, move the eth_type_trans() call a bit down, to give the RAM more time to prefetch the data.
--- drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +2 -2
@@ -3938,7 +3938,7 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi, goto err_drop_frame;/* Prefetch header */ -prefetch (data); +prefetch (data +MVPP2_MH_SIZE +MVPP2_SKB_HEADROOM );if (bm_pool->frag_size >PAGE_SIZE ) frag_size =0 ;
@@ -4008,8 +4008,8 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,skb_reserve (skb,MVPP2_MH_SIZE +MVPP2_SKB_HEADROOM );skb_put (skb, rx_bytes); - skb->protocol =eth_type_trans (skb, dev);mvpp2_rx_csum (port, rx_status, skb); + skb->protocol =eth_type_trans (skb, dev);napi_gro_receive (napi, skb);continue ;
Signature · 1 line
2.31.1
PATCH 2/2 mvpp2: prefetch page
In reply to Matteo CroceFrom: Matteo Croce <[email protected]>
Most of the time during the RX is caused by the compound_head() call done at the end of the RX loop:
│ build_skb():
[...]
│ static inline struct page *compound_head(struct page *page)
│ {
│ unsigned long head = READ_ONCE(page->compound_head);65.23 │ ldr x2, [x1, #8]
Prefetch the page struct as soon as possible, to speedup the RX path noticeabily by a ~3-4% packet rate in a drop test.
│ build_skb():
[...]
│ static inline struct page *compound_head(struct page *page)
│ {
│ unsigned long head = READ_ONCE(page->compound_head);17.92 │ ldr x2, [x1, #8]
--- drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +7 -3
@@ -3900,15 +3900,19 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,phys_addr_t phys_addr;u32 rx_status, timestamp;int pool, rx_bytes, err, ret; +struct page *page;void *data; + phys_addr =mvpp2_rxdesc_cookie_get (port, rx_desc); + data = (void *)phys_to_virt (phys_addr); + page =virt_to_page (data); +prefetch (page); + rx_done++; rx_status =mvpp2_rxdesc_status_get (port, rx_desc); rx_bytes =mvpp2_rxdesc_size_get (port, rx_desc); rx_bytes -=MVPP2_MH_SIZE ; dma_addr =mvpp2_rxdesc_dma_addr_get (port, rx_desc); - phys_addr =mvpp2_rxdesc_cookie_get (port, rx_desc); - data = (void *)phys_to_virt (phys_addr); pool = (rx_status &MVPP2_RXD_BM_POOL_ID_MASK ) >>MVPP2_RXD_BM_POOL_ID_OFFS ;
@@ -3997,7 +4001,7 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi, }if (pp) -skb_mark_for_recycle (skb,virt_to_page (data), pp); +skb_mark_for_recycle (skb, page, pp);else dma_unmap_single_attrs(dev->dev.parent, dma_addr, bm_pool->buf_size, DMA_FROM_DEVICE,
Signature · 1 line
2.31.1
Hi Matteo,
śr., 9 cze 2021 o 15:47 Matteo Croce <[email protected]> napisał(a):
Quoted 6 lines
From: Matteo Croce <[email protected]>
These two patches prefetch some data from RAM so to reduce stall and speedup the packet processing.
Quoted 7 lines
Matteo Croce (2):
mvpp2: prefetch right address
mvpp2: prefetch pagedrivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
Thank you for the patches, they seem reasonable, however I'd like to stress it on the CN913x setup @10G - I should have some slot for that closer to EOW.
Best regards,
Marcin
This series was applied to netdev/net-next.git (refs/heads/master):
Hello:
This series was applied to netdev/net-next.git (refs/heads/master):
Quoted 10 lines · On Wed, 9 Jun 2021 15:47:12 +0200 you wrote:
From: Matteo Croce <[email protected]>
These two patches prefetch some data from RAM so to reduce stall and speedup the packet processing.
Matteo Croce (2):
mvpp2: prefetch right address
mvpp2: prefetch page[...]
Here is the summary with links:
- [net-next,1/2] mvpp2: prefetch right address
https://git.kernel.org/netdev/net-next/c/d8ea89fe8a49 - [net-next,2/2] mvpp2: prefetch page https://git.kernel.org/netdev/net-next/c/2f128eb3308a
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html