Skip to contentSkip to conversation

Regression in mvneta with XDP patches

DISCUSSION Andrew Lunn 3 participants 9 messages Nov 11, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi Lorenzo, Jesper, Ilias

I just found that the XDP patches to mvneta have caused a regression.

This one breaks networking:

commit 8dc9a0888f4c8e27b25e48ff1b4bc2b3a845cc2d (HEAD)

Author: Lorenzo Bianconi <[email protected]>
Date:   Sat Oct 19 10:13:23 2019 +0200

net: mvneta: rely on build_skb in mvneta_rx_swbm poll routine

Refactor mvneta_rx_swbm code introducing mvneta_swbm_rx_frame and mvneta_swbm_add_rx_fragment routines. Rely on build_skb in oreder to allocate skb since the previous patch introduced buffer recycling using the page_pool API.
This patch fixes even an issue in the original driver where dma buffers are accessed before dma sync.
mvneta driver can run on not cache coherent devices so it is necessary to sync DMA buffers before sending them to the device in order to avoid memory corruptions. Running perf analysis we can see a performance cost associated with this DMA-sync (anyway it is already there in the original driver code). In follow up patches we will add more logic to reduce DMA-sync as much as possible.

I'm using an Linksys WRT1900ac, which has an Armada XP SoC. Device tree is arch/arm/boot/dts/armada-xp-linksys-mamba.dts.

With this patch applied, transmit appears to work O.K. My dhcp server is seeing good looking BOOTP requests and replying. However what is being received by the WRT1900ac is bad.

11:36:20.038558 d8:f7:00:00:00:00 (oui Unknown) > 00:00:00:00:5a:45 (oui Ethernet) Null Informati4

        0x0000:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000                           ......
11:36:26.924914 d8:f7:00:00:00:00 (oui Unknown) > 00:00:00:00:5a:45 (oui Ethernet) Null Informati4
        0x0000:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000                           ......
11:36:27.636597 4c:69:6e:75:78:20 (oui Unknown) > 6e:20:47:4e:55:2f (oui Unknown), ethertype Unkn 
        0x0000:  2873 7472 6574 6368 2920 4c69 6e75 7820  (stretch).Linux.
        0x0010:  352e 342e 302d 7263 362d 3031 3438 312d  5.4.0-rc6-01481-
        0x0020:  6739 3264 3965 3038 3439 3662 382d 6469  g92d9e08496b8-di
        0x0030:  7274 7920 2333 2053 756e 204e 6f76 2031  rty.#3.Sun.Nov.1
        0x0040:  3020 3136 3a31 373a 3531 2043 5354 2032  0.16:17:51.CST.2
        0x0050:  3031 3920 6172 6d76 376c 0e04 009c 0080  019.armv7l......
        0x0060:  100c 0501 0a00 000e 0200 0000 0200 1018  ................
        0x0070:  1102 fe80 0000 0000 0000 eefa aaff fe01  ................
        0x0080:  12fe 0200 0000 0200 0804 6574 6830 fe09  ..........eth0..
        0x0090:  0012 0f03 0100 0000 00fe 0900 120f 0103  ................
        0x00a0:  ec00 0010 0000 e3ed 5509 0000 0000 0000  ........U.......
        0x00b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00e0:  0000 0000 0000

This actually looks like random kernel memory.

Andrew

#1 lore raw

Hi Andrew,

Quoted 5 lines · Andrew Lunn

Hi Lorenzo, Jesper, Ilias

I just found that the XDP patches to mvneta have caused a regression.

This one breaks networking:

Thaks for the report.
Looking at the DTS i can see 'buffer-manager' in it. The changes we made were for the driver path software buffer manager.
Can you confirm which one your hardware uses?

I tested the patches on an espressobin, but the DTS i was using back then did not register the dsa infrastructure and i only had an 'eth0'.

Quoted 20 lines

commit 8dc9a0888f4c8e27b25e48ff1b4bc2b3a845cc2d (HEAD)

Author: Lorenzo Bianconi <[email protected]>
Date:   Sat Oct 19 10:13:23 2019 +0200

net: mvneta: rely on build_skb in mvneta_rx_swbm poll routine

Refactor mvneta_rx_swbm code introducing mvneta_swbm_rx_frame and mvneta_swbm_add_rx_fragment routines. Rely on build_skb in oreder to allocate skb since the previous patch introduced buffer recycling using the page_pool API.
This patch fixes even an issue in the original driver where dma buffers are accessed before dma sync.
mvneta driver can run on not cache coherent devices so it is necessary to sync DMA buffers before sending them to the device in order to avoid memory corruptions. Running perf analysis we can see a performance cost associated with this DMA-sync (anyway it is already there in the original driver code). In follow up patches we will add more logic to reduce DMA-sync as much as possible.

Quoted 6 lines

I'm using an Linksys WRT1900ac, which has an Armada XP SoC. Device tree is arch/arm/boot/dts/armada-xp-linksys-mamba.dts.

With this patch applied, transmit appears to work O.K. My dhcp server is seeing good looking BOOTP requests and replying. However what is being received by the WRT1900ac is bad.

What if you assign a static ip address? Same effect?

Quoted 28 lines

11:36:20.038558 d8:f7:00:00:00:00 (oui Unknown) > 00:00:00:00:5a:45 (oui Ethernet) Null Informati4

        0x0000:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000                           ......
11:36:26.924914 d8:f7:00:00:00:00 (oui Unknown) > 00:00:00:00:5a:45 (oui Ethernet) Null Informati4
        0x0000:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000                           ......
11:36:27.636597 4c:69:6e:75:78:20 (oui Unknown) > 6e:20:47:4e:55:2f (oui Unknown), ethertype Unkn 
        0x0000:  2873 7472 6574 6368 2920 4c69 6e75 7820  (stretch).Linux.
        0x0010:  352e 342e 302d 7263 362d 3031 3438 312d  5.4.0-rc6-01481-
        0x0020:  6739 3264 3965 3038 3439 3662 382d 6469  g92d9e08496b8-di
        0x0030:  7274 7920 2333 2053 756e 204e 6f76 2031  rty.#3.Sun.Nov.1
        0x0040:  3020 3136 3a31 373a 3531 2043 5354 2032  0.16:17:51.CST.2
        0x0050:  3031 3920 6172 6d76 376c 0e04 009c 0080  019.armv7l......
        0x0060:  100c 0501 0a00 000e 0200 0000 0200 1018  ................
        0x0070:  1102 fe80 0000 0000 0000 eefa aaff fe01  ................
        0x0080:  12fe 0200 0000 0200 0804 6574 6830 fe09  ..........eth0..
        0x0090:  0012 0f03 0100 0000 00fe 0900 120f 0103  ................
        0x00a0:  ec00 0010 0000 e3ed 5509 0000 0000 0000  ........U.......
        0x00b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00e0:  0000 0000 0000
Quoted 3 lines

This actually looks like random kernel memory.

Andrew

Thanks
/Ilias

#2 lore raw
Quoted 5 lines · Ilias Apalodimas

Hi Andrew,

Quoted 2 lines · Andrew Lunn

Hi Lorenzo, Jesper, Ilias

Quoted 8 lines
Quoted 3 lines

I just found that the XDP patches to mvneta have caused a regression.

This one breaks networking:

Thaks for the report.
Looking at the DTS i can see 'buffer-manager' in it. The changes we made were for the driver path software buffer manager.
Can you confirm which one your hardware uses?

Hi Ilias

Ah, interesting.

# CONFIG_MVNETA_BM_ENABLE is not set

So in fact it is not being compiled in, so should be falling back to software buffer manager.

If i do enable it, then it works. So we are in a corner cases you probably never tested. Requested by DT, but not actually available.

Andrew

#3 lore raw
Quoted 21 lines · Andrew Lunn
Quoted 5 lines · Ilias Apalodimas

Hi Andrew,

Quoted 2 lines · Andrew Lunn

Hi Lorenzo, Jesper, Ilias

Quoted 8 lines
Quoted 3 lines

I just found that the XDP patches to mvneta have caused a regression.

This one breaks networking:

Thaks for the report.
Looking at the DTS i can see 'buffer-manager' in it. The changes we made were for the driver path software buffer manager.
Can you confirm which one your hardware uses?

Hi Ilias

Ah, interesting.

# CONFIG_MVNETA_BM_ENABLE is not set

Quoted 7 lines

So in fact it is not being compiled in, so should be falling back to software buffer manager.

If i do enable it, then it works. So we are in a corner cases you probably never tested. Requested by DT, but not actually available.

Andrew

Maybe I got the issue. If mvneta_bm_port_init fails (e.g. if CONFIG_MVNETA_BM_ENABLE is not set) rx_offset_correction will be set to a wrong value. To bisect the issue, could you please test the following patch?

Regards,
Lorenzo

drivers/net/ethernet/marvell/mvneta.c +3 -1
index 591d580c68b4..e476fb043379 100644
@@ -4944,7 +4944,6 @@ static int mvneta_probe(struct platform_device *pdev)
 	SET_NETDEV_DEV(dev, &pdev->dev);
 
 	pp->id = global_port_id++;
-	pp->rx_offset_correction = MVNETA_SKB_HEADROOM;
 
 	/* Obtain access to BM resources if enabled and already initialized */
 	bm_node = of_parse_phandle(dn, "buffer-manager", 0);
@@ -4969,6 +4968,9 @@ static int mvneta_probe(struct platform_device *pdev)
 	}
 	of_node_put(bm_node);
 
+	if (!pp->bm_priv)
+		pp->rx_offset_correction = MVNETA_SKB_HEADROOM;
+
 	err = mvneta_init(&pdev->dev, pp);
 	if (err < 0)
 		goto err_netdev;
#4 lore raw
Quoted 3 lines

Maybe I got the issue. If mvneta_bm_port_init fails (e.g. if CONFIG_MVNETA_BM_ENABLE is not set) rx_offset_correction will be set to a wrong value. To bisect the issue, could you please test the following patch?

Hi Lorenzo

Did not help. Received packets are still corrupt.

Andrew

#5 lore raw
In reply to Andrew Lunn

Hi Andrew,

Quoted 19 lines · Andrew Lunn
Quoted 5 lines · Ilias Apalodimas

Hi Andrew,

Quoted 2 lines · Andrew Lunn

Hi Lorenzo, Jesper, Ilias

Quoted 8 lines
Quoted 3 lines

I just found that the XDP patches to mvneta have caused a regression.

This one breaks networking:

Thaks for the report.
Looking at the DTS i can see 'buffer-manager' in it. The changes we made were for the driver path software buffer manager.
Can you confirm which one your hardware uses?

Hi Ilias

Ah, interesting.

Quoted 8 lines

# CONFIG_MVNETA_BM_ENABLE is not set

So in fact it is not being compiled in, so should be falling back to software buffer manager.

If i do enable it, then it works. So we are in a corner cases you probably never tested. Requested by DT, but not actually available.

Correct, i don't think any of us had access to a hardware with BM We did change some offsets of memory allocation in the swbm to make room for skb_shared_info and the xdp headlen we need.
Maybe we missed that on the 'fallback' scenario. I'll try to have a look on the driver

Quoted 1 line

Andrew

Thanks
/Ilias

#6 lore raw
In reply to Andrew Lunn
Quoted 2 lines

Hi Lorenzo, Jesper, Ilias

Hi Andrew,

Quoted 23 lines

I just found that the XDP patches to mvneta have caused a regression.

This one breaks networking:

commit 8dc9a0888f4c8e27b25e48ff1b4bc2b3a845cc2d (HEAD)

Author: Lorenzo Bianconi <[email protected]>
Date:   Sat Oct 19 10:13:23 2019 +0200

net: mvneta: rely on build_skb in mvneta_rx_swbm poll routine

Refactor mvneta_rx_swbm code introducing mvneta_swbm_rx_frame and mvneta_swbm_add_rx_fragment routines. Rely on build_skb in oreder to allocate skb since the previous patch introduced buffer recycling using the page_pool API.
This patch fixes even an issue in the original driver where dma buffers are accessed before dma sync.
mvneta driver can run on not cache coherent devices so it is necessary to sync DMA buffers before sending them to the device in order to avoid memory corruptions. Running perf analysis we can see a performance cost associated with this DMA-sync (anyway it is already there in the original driver code). In follow up patches we will add more logic to reduce DMA-sync as much as possible.

Quoted 2 lines

I'm using an Linksys WRT1900ac, which has an Armada XP SoC. Device tree is arch/arm/boot/dts/armada-xp-linksys-mamba.dts.

looking at the dts, could you please confirm mvneta is using hw or sw buffer manager on this board? Moreover are you using DSA as well?

Regards,
Lorenzo

Quoted 35 lines

With this patch applied, transmit appears to work O.K. My dhcp server is seeing good looking BOOTP requests and replying. However what is being received by the WRT1900ac is bad.

11:36:20.038558 d8:f7:00:00:00:00 (oui Unknown) > 00:00:00:00:5a:45 (oui Ethernet) Null Informati4

        0x0000:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000                           ......
11:36:26.924914 d8:f7:00:00:00:00 (oui Unknown) > 00:00:00:00:5a:45 (oui Ethernet) Null Informati4
        0x0000:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000                           ......
11:36:27.636597 4c:69:6e:75:78:20 (oui Unknown) > 6e:20:47:4e:55:2f (oui Unknown), ethertype Unkn 
        0x0000:  2873 7472 6574 6368 2920 4c69 6e75 7820  (stretch).Linux.
        0x0010:  352e 342e 302d 7263 362d 3031 3438 312d  5.4.0-rc6-01481-
        0x0020:  6739 3264 3965 3038 3439 3662 382d 6469  g92d9e08496b8-di
        0x0030:  7274 7920 2333 2053 756e 204e 6f76 2031  rty.#3.Sun.Nov.1
        0x0040:  3020 3136 3a31 373a 3531 2043 5354 2032  0.16:17:51.CST.2
        0x0050:  3031 3920 6172 6d76 376c 0e04 009c 0080  019.armv7l......
        0x0060:  100c 0501 0a00 000e 0200 0000 0200 1018  ................
        0x0070:  1102 fe80 0000 0000 0000 eefa aaff fe01  ................
        0x0080:  12fe 0200 0000 0200 0804 6574 6830 fe09  ..........eth0..
        0x0090:  0012 0f03 0100 0000 00fe 0900 120f 0103  ................
        0x00a0:  ec00 0010 0000 e3ed 5509 0000 0000 0000  ........U.......
        0x00b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00e0:  0000 0000 0000

This actually looks like random kernel memory.

Andrew

#7 lore raw
Quoted 2 lines

looking at the dts, could you please confirm mvneta is using hw or sw buffer manager on this board? Moreover are you using DSA as well?

So my reply to Ilias answered the first question. And yes, i'm using DSA. But that should not matter, mvneta is just receiving frames which happen to have an extra header after the two MAC addresses.

Andrew

#8 lore raw
Quoted 7 lines · Andrew Lunn
Quoted 2 lines

looking at the dts, could you please confirm mvneta is using hw or sw buffer manager on this board? Moreover are you using DSA as well?

So my reply to Ilias answered the first question. And yes, i'm using DSA. But that should not matter, mvneta is just receiving frames which happen to have an extra header after the two MAC addresses.

In theory (famous last words) DSA or not shouldn't be affected by this. The driver was already allocating a page per packet before our changes. We just allocate that page via page_pool.

Thanks
/Ilias