Skip to contentSkip to conversation

mvpp2: prefetch data early

PATCH Applied Matteo Croce 3 participants 5 messages Jun 9, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover mvpp2: prefetch data early

From: Matteo Croce <[email protected]>

These two patches prefetch some data from RAM so to reduce stall and speedup the packet processing.

Matteo Croce (2):
mvpp2: prefetch right address
mvpp2: prefetch page

 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

PATCH 1/2 mvpp2: prefetch right address

From: Matteo Croce <[email protected]>

In the RX buffer, the received data starts after a headroom used to align the IP header and to allow prepending headers efficiently. The prefetch() should take this into account, and prefetch from the very start of the received data.

We can see that ether_addr_equal_64bits(), which is the first function to access the data, drops from the top of the perf top output.

prefetch(data):

Overhead  Shared Object     Symbol
  11.64%  [kernel]          [k] eth_type_trans

prefetch(data + MVPP2_MH_SIZE + MVPP2_SKB_HEADROOM):

Overhead  Shared Object     Symbol
  13.42%  [kernel]          [k] build_skb
  10.35%  [mvpp2]           [k] mvpp2_rx
   9.35%  [kernel]          [k] __netif_receive_skb_core
   8.24%  [kernel]          [k] kmem_cache_free
   7.97%  [kernel]          [k] dev_gro_receive
   7.68%  [kernel]          [k] page_pool_put_page
   7.32%  [kernel]          [k] kmem_cache_alloc
   7.09%  [mvpp2]           [k] mvpp2_bm_pool_put
   3.36%  [kernel]          [k] eth_type_trans

Also, move the eth_type_trans() call a bit down, to give the RAM more time to prefetch the data.

---
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +2 -2
index 5663c1b21870..07d8f3e31b52 100644
@@ -3938,7 +3938,7 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
 			goto err_drop_frame;
 
 		/* Prefetch header */
-		prefetch(data);
+		prefetch(data + MVPP2_MH_SIZE + MVPP2_SKB_HEADROOM);
 
 		if (bm_pool->frag_size > PAGE_SIZE)
 			frag_size = 0;
@@ -4008,8 +4008,8 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
 
 		skb_reserve(skb, MVPP2_MH_SIZE + MVPP2_SKB_HEADROOM);
 		skb_put(skb, rx_bytes);
-		skb->protocol = eth_type_trans(skb, dev);
 		mvpp2_rx_csum(port, rx_status, skb);
+		skb->protocol = eth_type_trans(skb, dev);
 
 		napi_gro_receive(napi, skb);
 		continue;
Signature · 1 line
2.31.1
#2 lore raw

PATCH 2/2 mvpp2: prefetch page

In reply to Matteo Croce

From: Matteo Croce <[email protected]>

Most of the time during the RX is caused by the compound_head() call done at the end of the RX loop:

       │     build_skb():
       [...]
       │     static inline struct page *compound_head(struct page *page)
       │     {
       │     unsigned long head = READ_ONCE(page->compound_head);

65.23 │ ldr x2, [x1, #8]

Prefetch the page struct as soon as possible, to speedup the RX path noticeabily by a ~3-4% packet rate in a drop test.

       │     build_skb():
       [...]
       │     static inline struct page *compound_head(struct page *page)
       │     {
       │     unsigned long head = READ_ONCE(page->compound_head);

17.92 │ ldr x2, [x1, #8]

---
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +7 -3
index 07d8f3e31b52..9bca8c8f9f8d 100644
@@ -3900,15 +3900,19 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
 		phys_addr_t phys_addr;
 		u32 rx_status, timestamp;
 		int pool, rx_bytes, err, ret;
+		struct page *page;
 		void *data;
 
+		phys_addr = mvpp2_rxdesc_cookie_get(port, rx_desc);
+		data = (void *)phys_to_virt(phys_addr);
+		page = virt_to_page(data);
+		prefetch(page);
+
 		rx_done++;
 		rx_status = mvpp2_rxdesc_status_get(port, rx_desc);
 		rx_bytes = mvpp2_rxdesc_size_get(port, rx_desc);
 		rx_bytes -= MVPP2_MH_SIZE;
 		dma_addr = mvpp2_rxdesc_dma_addr_get(port, rx_desc);
-		phys_addr = mvpp2_rxdesc_cookie_get(port, rx_desc);
-		data = (void *)phys_to_virt(phys_addr);
 
 		pool = (rx_status & MVPP2_RXD_BM_POOL_ID_MASK) >>
 			MVPP2_RXD_BM_POOL_ID_OFFS;
@@ -3997,7 +4001,7 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
 		}
 
 		if (pp)
-			skb_mark_for_recycle(skb, virt_to_page(data), pp);
+			skb_mark_for_recycle(skb, page, pp);
 		else
 			dma_unmap_single_attrs(dev->dev.parent, dma_addr,
 					       bm_pool->buf_size, DMA_FROM_DEVICE,
Signature · 1 line
2.31.1
#3 lore raw
In reply to Matteo Croce

Hi Matteo,

śr., 9 cze 2021 o 15:47 Matteo Croce <[email protected]> napisał(a):

Quoted 6 lines

From: Matteo Croce <[email protected]>

These two patches prefetch some data from RAM so to reduce stall and speedup the packet processing.

Quoted 7 lines

Matteo Croce (2):
mvpp2: prefetch right address
mvpp2: prefetch page

 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

Thank you for the patches, they seem reasonable, however I'd like to stress it on the CN913x setup @10G - I should have some slot for that closer to EOW.

Best regards,
Marcin

#4 lore raw
In reply to Matteo Croce
This series was applied to netdev/net-next.git (refs/heads/master):

Hello:

This series was applied to netdev/net-next.git (refs/heads/master):

Quoted 10 lines · On Wed, 9 Jun 2021 15:47:12 +0200 you wrote:

From: Matteo Croce <[email protected]>

These two patches prefetch some data from RAM so to reduce stall and speedup the packet processing.

Matteo Croce (2):
mvpp2: prefetch right address
mvpp2: prefetch page

[...]

Here is the summary with links:
- [net-next,1/2] mvpp2: prefetch right address

    https://git.kernel.org/netdev/net-next/c/d8ea89fe8a49
  - [net-next,2/2] mvpp2: prefetch page
    https://git.kernel.org/netdev/net-next/c/2f128eb3308a

You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html