Regression in mvneta with XDP patches
Subsystems (3)
Hi Lorenzo, Jesper, Ilias
I just found that the XDP patches to mvneta have caused a regression.
This one breaks networking:
commit 8dc9a0888f4c8e27b25e48ff1b4bc2b3a845cc2d (HEAD)
Author: Lorenzo Bianconi <[email protected]> Date: Sat Oct 19 10:13:23 2019 +0200
net: mvneta: rely on build_skb in mvneta_rx_swbm poll routine
Refactor mvneta_rx_swbm code introducing mvneta_swbm_rx_frame and mvneta_swbm_add_rx_fragment routines. Rely on build_skb in oreder to allocate skb since the previous patch introduced buffer recycling using the page_pool API.
This patch fixes even an issue in the original driver where dma buffers are accessed before dma sync.
mvneta driver can run on not cache coherent devices so it is necessary to sync DMA buffers before sending them to the device in order to avoid memory corruptions. Running perf analysis we can see a performance cost associated with this DMA-sync (anyway it is already there in the original driver code). In follow up patches we will add more logic to reduce DMA-sync as much as possible.
I'm using an Linksys WRT1900ac, which has an Armada XP SoC. Device tree is arch/arm/boot/dts/armada-xp-linksys-mamba.dts.
With this patch applied, transmit appears to work O.K. My dhcp server is seeing good looking BOOTP requests and replying. However what is being received by the WRT1900ac is bad.
11:36:20.038558 d8:f7:00:00:00:00 (oui Unknown) > 00:00:00:00:5a:45 (oui Ethernet) Null Informati4
0x0000: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0030: 0000 0000 0000 ......
11:36:26.924914 d8:f7:00:00:00:00 (oui Unknown) > 00:00:00:00:5a:45 (oui Ethernet) Null Informati4
0x0000: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0030: 0000 0000 0000 ......
11:36:27.636597 4c:69:6e:75:78:20 (oui Unknown) > 6e:20:47:4e:55:2f (oui Unknown), ethertype Unkn
0x0000: 2873 7472 6574 6368 2920 4c69 6e75 7820 (stretch).Linux.
0x0010: 352e 342e 302d 7263 362d 3031 3438 312d 5.4.0-rc6-01481-
0x0020: 6739 3264 3965 3038 3439 3662 382d 6469 g92d9e08496b8-di
0x0030: 7274 7920 2333 2053 756e 204e 6f76 2031 rty.#3.Sun.Nov.1
0x0040: 3020 3136 3a31 373a 3531 2043 5354 2032 0.16:17:51.CST.2
0x0050: 3031 3920 6172 6d76 376c 0e04 009c 0080 019.armv7l......
0x0060: 100c 0501 0a00 000e 0200 0000 0200 1018 ................
0x0070: 1102 fe80 0000 0000 0000 eefa aaff fe01 ................
0x0080: 12fe 0200 0000 0200 0804 6574 6830 fe09 ..........eth0..
0x0090: 0012 0f03 0100 0000 00fe 0900 120f 0103 ................
0x00a0: ec00 0010 0000 e3ed 5509 0000 0000 0000 ........U.......
0x00b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x00c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x00d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x00e0: 0000 0000 0000This actually looks like random kernel memory.
Andrew
Hi Andrew,
Quoted 5 lines · Andrew Lunn
Hi Lorenzo, Jesper, Ilias
I just found that the XDP patches to mvneta have caused a regression.
This one breaks networking:
Thaks for the report.
Looking at the DTS i can see 'buffer-manager' in it. The changes we made were for the driver path software buffer manager.
Can you confirm which one your hardware uses?
I tested the patches on an espressobin, but the DTS i was using back then did not register the dsa infrastructure and i only had an 'eth0'.
Quoted 20 lines
commit
8dc9a0888f4c8e27b25e48ff1b4bc2b3a845cc2d(HEAD)Author: Lorenzo Bianconi <[email protected]> Date: Sat Oct 19 10:13:23 2019 +0200net: mvneta: rely on
build_skbinmvneta_rx_swbmpoll routineRefactor
mvneta_rx_swbmcode introducingmvneta_swbm_rx_frameandmvneta_swbm_add_rx_fragmentroutines. Rely onbuild_skbin oreder to allocate skb since the previous patch introduced buffer recycling using thepage_poolAPI.
This patch fixes even an issue in the original driver where dma buffers are accessed before dma sync.
mvneta driver can run on not cache coherent devices so it is necessary to sync DMA buffers before sending them to the device in order to avoid memory corruptions. Running perf analysis we can see a performance cost associated with this DMA-sync (anyway it is already there in the original driver code). In follow up patches we will add more logic to reduce DMA-sync as much as possible.
Quoted 6 lines
I'm using an Linksys WRT1900ac, which has an Armada XP SoC. Device tree is
arch/arm/boot/dts/armada-xp-linksys-mamba.dts.With this patch applied, transmit appears to work O.K. My dhcp server is seeing good looking BOOTP requests and replying. However what is being received by the WRT1900ac is bad.
What if you assign a static ip address? Same effect?
Quoted 28 lines
11:36:20.038558 d8:f7:00:00:00:00 (oui Unknown) > 00:00:00:00:5a:45 (oui Ethernet) Null Informati4
0x0000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0030: 0000 0000 0000 ...... 11:36:26.924914 d8:f7:00:00:00:00 (oui Unknown) > 00:00:00:00:5a:45 (oui Ethernet) Null Informati4 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0030: 0000 0000 0000 ...... 11:36:27.636597 4c:69:6e:75:78:20 (oui Unknown) > 6e:20:47:4e:55:2f (oui Unknown), ethertype Unkn 0x0000: 2873 7472 6574 6368 2920 4c69 6e75 7820 (stretch).Linux. 0x0010: 352e 342e 302d 7263 362d 3031 3438 312d 5.4.0-rc6-01481- 0x0020: 6739 3264 3965 3038 3439 3662 382d 6469 g92d9e08496b8-di 0x0030: 7274 7920 2333 2053 756e 204e 6f76 2031 rty.#3.Sun.Nov.1 0x0040: 3020 3136 3a31 373a 3531 2043 5354 2032 0.16:17:51.CST.2 0x0050: 3031 3920 6172 6d76 376c 0e04 009c 0080 019.armv7l...... 0x0060: 100c 0501 0a00 000e 0200 0000 0200 1018 ................ 0x0070: 1102 fe80 0000 0000 0000 eefa aaff fe01 ................ 0x0080: 12fe 0200 0000 0200 0804 6574 6830 fe09 ..........eth0.. 0x0090: 0012 0f03 0100 0000 00fe 0900 120f 0103 ................ 0x00a0: ec00 0010 0000 e3ed 5509 0000 0000 0000 ........U....... 0x00b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00e0: 0000 0000 0000
Quoted 3 lines
This actually looks like random kernel memory.
Andrew
Thanks
/Ilias
Quoted 5 lines · Ilias Apalodimas
Hi Andrew,
Quoted 2 lines · Andrew Lunn
Hi Lorenzo, Jesper, Ilias
Quoted 8 lines
Quoted 3 lines
I just found that the XDP patches to mvneta have caused a regression.
This one breaks networking:
Thaks for the report.
Looking at the DTS i can see 'buffer-manager' in it. The changes we made were for the driver path software buffer manager.
Can you confirm which one your hardware uses?
Hi Ilias
Ah, interesting.
# CONFIG_MVNETA_BM_ENABLE is not set
So in fact it is not being compiled in, so should be falling back to software buffer manager.
If i do enable it, then it works. So we are in a corner cases you probably never tested. Requested by DT, but not actually available.
Andrew
Quoted 21 lines · Andrew Lunn
Quoted 5 lines · Ilias Apalodimas
Hi Andrew,
Quoted 2 lines · Andrew Lunn
Hi Lorenzo, Jesper, Ilias
Quoted 8 lines
Quoted 3 lines
I just found that the XDP patches to mvneta have caused a regression.
This one breaks networking:
Thaks for the report.
Looking at the DTS i can see 'buffer-manager' in it. The changes we made were for the driver path software buffer manager.
Can you confirm which one your hardware uses?Hi Ilias
Ah, interesting.
#
CONFIG_MVNETA_BM_ENABLEis not set
Quoted 7 lines
So in fact it is not being compiled in, so should be falling back to software buffer manager.
If i do enable it, then it works. So we are in a corner cases you probably never tested. Requested by DT, but not actually available.
Andrew
Maybe I got the issue. If mvneta_bm_port_init fails (e.g. if CONFIG_MVNETA_BM_ENABLE is not set) rx_offset_correction will be set to a wrong value. To bisect the issue, could you please test the following patch?
Regards,
Lorenzo
drivers/net/ethernet/marvell/mvneta.c +3 -1
@@ -4944,7 +4944,6 @@ static int mvneta_probe(struct platform_device *pdev)SET_NETDEV_DEV (dev, &pdev->dev); pp->id = global_port_id++; - pp->rx_offset_correction =MVNETA_SKB_HEADROOM ;/* Obtain access to BM resources if enabled and already initialized */ bm_node =of_parse_phandle (dn,"buffer-manager" ,0 );
@@ -4969,6 +4968,9 @@ static int mvneta_probe(struct platform_device *pdev) }of_node_put (bm_node); +if (!pp->bm_priv) + pp->rx_offset_correction =MVNETA_SKB_HEADROOM ; + err =mvneta_init (&pdev->dev, pp);if (err <0 ) goto err_netdev;
Quoted 3 lines
Maybe I got the issue. If
mvneta_bm_port_initfails (e.g. ifCONFIG_MVNETA_BM_ENABLEis not set)rx_offset_correctionwill be set to a wrong value. To bisect the issue, could you please test the following patch?
Hi Lorenzo
Did not help. Received packets are still corrupt.
Andrew
Hi Andrew,
Quoted 19 lines · Andrew Lunn
Quoted 5 lines · Ilias Apalodimas
Hi Andrew,
Quoted 2 lines · Andrew Lunn
Hi Lorenzo, Jesper, Ilias
Quoted 8 lines
Quoted 3 lines
I just found that the XDP patches to mvneta have caused a regression.
This one breaks networking:
Thaks for the report.
Looking at the DTS i can see 'buffer-manager' in it. The changes we made were for the driver path software buffer manager.
Can you confirm which one your hardware uses?Hi Ilias
Ah, interesting.
Quoted 8 lines
#
CONFIG_MVNETA_BM_ENABLEis not setSo in fact it is not being compiled in, so should be falling back to software buffer manager.
If i do enable it, then it works. So we are in a corner cases you probably never tested. Requested by DT, but not actually available.
Correct, i don't think any of us had access to a hardware with BM We did change some offsets of memory allocation in the swbm to make room for skb_shared_info and the xdp headlen we need.
Maybe we missed that on the 'fallback' scenario. I'll try to have a look on the driver
Quoted 1 line
Andrew
Thanks
/Ilias
Quoted 2 lines
Hi Lorenzo, Jesper, Ilias
Hi Andrew,
Quoted 23 lines
I just found that the XDP patches to mvneta have caused a regression.
This one breaks networking:
commit
8dc9a0888f4c8e27b25e48ff1b4bc2b3a845cc2d(HEAD)Author: Lorenzo Bianconi <[email protected]> Date: Sat Oct 19 10:13:23 2019 +0200net: mvneta: rely on
build_skbinmvneta_rx_swbmpoll routineRefactor
mvneta_rx_swbmcode introducingmvneta_swbm_rx_frameandmvneta_swbm_add_rx_fragmentroutines. Rely onbuild_skbin oreder to allocate skb since the previous patch introduced buffer recycling using thepage_poolAPI.
This patch fixes even an issue in the original driver where dma buffers are accessed before dma sync.
mvneta driver can run on not cache coherent devices so it is necessary to sync DMA buffers before sending them to the device in order to avoid memory corruptions. Running perf analysis we can see a performance cost associated with this DMA-sync (anyway it is already there in the original driver code). In follow up patches we will add more logic to reduce DMA-sync as much as possible.
Quoted 2 lines
I'm using an Linksys WRT1900ac, which has an Armada XP SoC. Device tree is
arch/arm/boot/dts/armada-xp-linksys-mamba.dts.
looking at the dts, could you please confirm mvneta is using hw or sw buffer manager on this board? Moreover are you using DSA as well?
Regards,
Lorenzo
Quoted 35 lines
With this patch applied, transmit appears to work O.K. My dhcp server is seeing good looking BOOTP requests and replying. However what is being received by the WRT1900ac is bad.
11:36:20.038558 d8:f7:00:00:00:00 (oui Unknown) > 00:00:00:00:5a:45 (oui Ethernet) Null Informati4
0x0000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0030: 0000 0000 0000 ...... 11:36:26.924914 d8:f7:00:00:00:00 (oui Unknown) > 00:00:00:00:5a:45 (oui Ethernet) Null Informati4 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0030: 0000 0000 0000 ...... 11:36:27.636597 4c:69:6e:75:78:20 (oui Unknown) > 6e:20:47:4e:55:2f (oui Unknown), ethertype Unkn 0x0000: 2873 7472 6574 6368 2920 4c69 6e75 7820 (stretch).Linux. 0x0010: 352e 342e 302d 7263 362d 3031 3438 312d 5.4.0-rc6-01481- 0x0020: 6739 3264 3965 3038 3439 3662 382d 6469 g92d9e08496b8-di 0x0030: 7274 7920 2333 2053 756e 204e 6f76 2031 rty.#3.Sun.Nov.1 0x0040: 3020 3136 3a31 373a 3531 2043 5354 2032 0.16:17:51.CST.2 0x0050: 3031 3920 6172 6d76 376c 0e04 009c 0080 019.armv7l...... 0x0060: 100c 0501 0a00 000e 0200 0000 0200 1018 ................ 0x0070: 1102 fe80 0000 0000 0000 eefa aaff fe01 ................ 0x0080: 12fe 0200 0000 0200 0804 6574 6830 fe09 ..........eth0.. 0x0090: 0012 0f03 0100 0000 00fe 0900 120f 0103 ................ 0x00a0: ec00 0010 0000 e3ed 5509 0000 0000 0000 ........U....... 0x00b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00e0: 0000 0000 0000This actually looks like random kernel memory.
Andrew
Quoted 2 lines
looking at the dts, could you please confirm mvneta is using hw or sw buffer manager on this board? Moreover are you using DSA as well?
So my reply to Ilias answered the first question. And yes, i'm using DSA. But that should not matter, mvneta is just receiving frames which happen to have an extra header after the two MAC addresses.
Andrew
Quoted 7 lines · Andrew Lunn
Quoted 2 lines
looking at the dts, could you please confirm mvneta is using hw or sw buffer manager on this board? Moreover are you using DSA as well?
So my reply to Ilias answered the first question. And yes, i'm using DSA. But that should not matter, mvneta is just receiving frames which happen to have an extra header after the two MAC addresses.
In theory (famous last words) DSA or not shouldn't be affected by this. The driver was already allocating a page per packet before our changes. We just allocate that page via page_pool.
Thanks
/Ilias