Skip to contentSkip to conversation

riscv: dts: spacemit: k3-pico-itx: Fix non-functional ethernet TX timing

PATCH Applied Andrew Rembrandt 4 participants 13 messages Jun 8 – Jul 19

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 riscv: dts: spacemit: k3-pico-itx: Fix non-functional ethernet TX timing

The "rgmii-id" PHY mode applies a symmetric 2ns internal delay in the PHY. On the Pico-ITX board this leaves insufficient TX timing margin and no ethernet traffic passes.

Add a 400ps MAC-side TX internal delay on top of the PHY's delay to correct the TX clock/data skew. The value follows the vendor fork's asymmetric tuning (tx-phase 73 x 33.3ps - 2000ps ID ~= 400ps); the driver rounds tx-internal-delay-ps to the nearest supported step.

This builds on commit 3ea695eb111f ("dts: riscv: spacemit: k3: Fix I/O power settings"), which fixed the GMAC0 1.8V/3.3V I/O switch

Tested on Pico-ITX hardware with iperf3: 60s TCP each direction and 60s UDP, all at 1Gbps with no TX/RX errors on the interface.

Fixes 74657a376960 ("riscv: dts: spacemit: Add ethernet device for K3")
---
 arch/riscv/boot/dts/spacemit/k3-pico-itx.dts | 1 +
 1 file changed, 1 insertion(+)
arch/riscv/boot/dts/spacemit/k3-pico-itx.dts +1 -0
index b89c1521e664..59aecaa2d32a 100644
@@ -186,6 +186,7 @@ &eth0 {
 	pinctrl-names = "default";
 	pinctrl-0 = <&gmac0_rgmii_0_cfg>, <&gmac0_phy_0_cfg>;
 	phy-mode = "rgmii-id";
+	tx-internal-delay-ps = <400>;
 	phy-handle = <&phy0>;
 	status = "okay";
 
Signature · 1 line
2.54.0
#1 lore raw
Quoted 26 lines · Andrew Rembrandt

The "rgmii-id" PHY mode applies a symmetric 2ns internal delay in the PHY. On the Pico-ITX board this leaves insufficient TX timing margin and no ethernet traffic passes.

Add a 400ps MAC-side TX internal delay on top of the PHY's delay to correct the TX clock/data skew. The value follows the vendor fork's asymmetric tuning (tx-phase 73 x 33.3ps - 2000ps ID ~= 400ps); the driver rounds tx-internal-delay-ps to the nearest supported step.

This builds on commit 3ea695eb111f ("dts: riscv: spacemit: k3: Fix I/O power settings"), which fixed the GMAC0 1.8V/3.3V I/O switch

Tested on Pico-ITX hardware with iperf3: 60s TCP each direction and 60s UDP, all at 1Gbps with no TX/RX errors on the interface.

Fixes 74657a376960 ("riscv: dts: spacemit: Add ethernet device for K3")
---
 arch/riscv/boot/dts/spacemit/k3-pico-itx.dts | 1 +
 1 file changed, 1 insertion(+)
arch/riscv/boot/dts/spacemit/k3-pico-itx.dts +0 -0
index b89c1521e664..59aecaa2d32a 100644

arch/riscv/boot/dts/spacemit/k3-pico-itx.dts &eth0 { · line 192

Quoted 7 lines
arch/riscv/boot/dts/spacemit/k3-pico-itx.dts +1 -0
@@ -186,6 +186,7 @@ &eth0 {
 	pinctrl-names = "default";
 	pinctrl-0 = <&gmac0_rgmii_0_cfg>, <&gmac0_phy_0_cfg>;
 	phy-mode = "rgmii-id";
+	tx-internal-delay-ps = <400>;
 	phy-handle = <&phy0>;
 	status = "okay";
 

You should set tx-internal-delay-ps of the phy instead of the controller. This is a phy in "rgmii-id" mode.

Regards,
Inochi

#2 lore raw
Quoted 2 lines · Inochi Amaoto

You should set tx-internal-delay-ps of the phy instead of the controller. This is a phy in "rgmii-id" mode.

Thanks for the review. I don't think that works on this board, but please let me know if I'm missing something.

The PHY is a Realtek RTL8211F. Its driver only toggles the chip's fixed ~2ns delay on/off from phy-mode, and per the datasheet that's a hardware limit, not a driver one -- the RTL8211F has no tunable delay, so a ps value on the PHY node has nothing to act on.

The only configurable delay element on K3 is the SoC RGMII delay line, and dwmac-spacemit programs it from tx-/rx-internal-delay-ps read off the controller node (spacemit_dwmac_probe()), so that is where the property has to live to take effect. This is the MAC "fine tuning ... values expected here are small" case described in Documentation/devicetree/bindings/net/ethernet-controller.yaml: the RTL8211F supplies the fixed ~2ns via rgmii-id and the MAC adds a 400ps trim on top. Ideally the PHY would own the delay, but as Andrew Lunn noted reviewing dwmac-meson8b, that requires the PHY to implement a configurable delay, which not all do.

The already-merged k1-bananapi-f3 does the same thing: eth1 combines phy-mode = "rgmii-id" with tx-internal-delay-ps = <250> on the controller node (via the K1 EMAC rather than dwmac, but the same DT pattern).

Would you be OK keeping it on the controller given the above? For v2 I can add an explicit rx-internal-delay-ps = <0> to match the other SpacemiT boards (k1-bananapi-f3 included) and spell this reasoning out in the commit message.

Thanks,
Andrew

Reviewed-by
#3 lore raw
Quoted 22 lines · Andrew Rembrandt
Quoted 2 lines · Inochi Amaoto

You should set tx-internal-delay-ps of the phy instead of the controller. This is a phy in "rgmii-id" mode.

Thanks for the review. I don't think that works on this board, but please let me know if I'm missing something.

The PHY is a Realtek RTL8211F. Its driver only toggles the chip's fixed ~2ns delay on/off from phy-mode, and per the datasheet that's a hardware limit, not a driver one -- the RTL8211F has no tunable delay, so a ps value on the PHY node has nothing to act on.

The only configurable delay element on K3 is the SoC RGMII delay line, and dwmac-spacemit programs it from tx-/rx-internal-delay-ps read off the controller node (spacemit_dwmac_probe()), so that is where the property has to live to take effect. This is the MAC "fine tuning ... values expected here are small" case described in Documentation/devicetree/bindings/net/ethernet-controller.yaml: the RTL8211F supplies the fixed ~2ns via rgmii-id and the MAC adds a 400ps trim on top. Ideally the PHY would own the delay, but as Andrew Lunn noted reviewing dwmac-meson8b, that requires the PHY to implement a configurable delay, which not all do.

Quoted 6 lines

The already-merged k1-bananapi-f3 does the same thing: eth1 combines phy-mode = "rgmii-id" with tx-internal-delay-ps = <250> on the controller node (via the K1 EMAC rather than dwmac, but the same DT pattern).

Would you be OK keeping it on the controller given the above?

Yes, it is good for me.

Quoted 4 lines

For v2 I can add an explicit rx-internal-delay-ps = <0> to match the other SpacemiT boards (k1-bananapi-f3 included) and spell this reasoning out in the commit message.

Do not do this, just leave the rx-internal-delay-ps untouched.

For this patch

Regards,
Inochi.

#4 lore raw
Quoted 1 line · Inochi Amaoto

Do not do this, just leave the rx-internal-delay-ps untouched.

Sounds good, I'll leave v1 as-is. Many thanks for the review.

Regards,

Andrew

#5 lore raw

Hi Andrew,

Quoted 5 lines · Andrew Rembrandt
Quoted 1 line · Inochi Amaoto

Do not do this, just leave the rx-internal-delay-ps untouched.

Sounds good, I'll leave v1 as-is. Many thanks for the review.

Quoted 8 lines

Regards,

Andrew

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

I do not notice any difference with or without this property present, tested Linux -next with U-Boot modification to devicetree:

load usb 0:1 $fdt_addr_r /dtb/spacemit/k3-pico-itx.dtb fdt addr $fdt_addr_r
fdt resize
fdt header get filesize totalsize
fdt set /soc/ethernet@cac80000 tx-internal-delay-ps <400> load usb 0:1 $kernel_addr_r /EFI/BOOT/BOOTRISCV64.EFI

# throwaway use of ping is workaround for broken vendor U-Boot
ping 127.0.0.1; bootefi $kernel_addr_r $fdt_addr_r

Aside, there is still some problem with the driver which does cause it to fail. This can be reproduced by unloading the module and then loading the module; built from -next with 'olddefconfig' of debian Kconfig for kernel 7.1, no patches.

# modprobe -r dwmac_spacemit
# modprobe dwmac_spacemit
[   61.469745] spacemit-dwmac cac80000.ethernet end0: cannot attach to
PHY (error: -ENODEV)
[   91.544646] spacemit-dwmac cac80000.ethernet end0: cannot attach to

PHY (error: -ENODEV)

If not reloading the module then I don't see any difference, traffic passes (ping 1.1.1.1) with or without your suggested tx-internal-delay-ps property. Is there a specific test to run to observe expected difference?

Thanks,

-E

maintainer
#6 lore raw

Hi E Shattow,

Quoted 34 lines · E Shattow

Hi Andrew,

Quoted 5 lines · Andrew Rembrandt
Quoted 1 line · Inochi Amaoto

Do not do this, just leave the rx-internal-delay-ps untouched.

Sounds good, I'll leave v1 as-is. Many thanks for the review.

Quoted 8 lines

Regards,

Andrew

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

I do not notice any difference with or without this property present, tested Linux -next with U-Boot modification to devicetree:

load usb 0:1 $fdt_addr_r /dtb/spacemit/k3-pico-itx.dtb fdt addr $fdt_addr_r
fdt resize
fdt header get filesize totalsize
fdt set /soc/ethernet@cac80000 tx-internal-delay-ps <400> load usb 0:1 $kernel_addr_r /EFI/BOOT/BOOTRISCV64.EFI

# throwaway use of ping is workaround for broken vendor U-Boot
ping 127.0.0.1; bootefi $kernel_addr_r $fdt_addr_r

Aside, there is still some problem with the driver which does cause it to fail. This can be reproduced by unloading the module and then loading the module; built from -next with 'olddefconfig' of debian Kconfig for kernel 7.1, no patches.

Quoted 8 lines
# modprobe -r dwmac_spacemit
# modprobe dwmac_spacemit
[   61.469745] spacemit-dwmac cac80000.ethernet end0: cannot attach to
PHY (error: -ENODEV)
[   91.544646] spacemit-dwmac cac80000.ethernet end0: cannot attach to

PHY (error: -ENODEV)

The problem is unrelated to this patch itself, but I can actually confirm here

Quoted 5 lines

If not reloading the module then I don't see any difference, traffic passes (ping 1.1.1.1) with or without your suggested tx-internal-delay-ps property. Is there a specific test to run to observe expected difference?

Although I haven't narrowed down to the root cause, but it's kind of something wrong with the reset procedure, remove it from phy node will fix this issue..

+++ b/arch/riscv/boot/dts/spacemit/k3-pico-itx.dts

@@ -232,9 +232,6 @@ mdio {
                phy0: phy@1 {
                        compatible = "ethernet-phy-ieee802.3-c22";
                        reg = <1>;
-                       reset-gpios = <&gpio 0 15 GPIO_ACTIVE_LOW>;
-                       reset-assert-us = <10000>;
-                       reset-deassert-us = <10000>;
Signature · 1 line
Yixun Lan (dlan)
#7 lore raw

Hi Yixun,

Quoted 59 lines · Yixun Lan

Hi E Shattow,

Quoted 34 lines · E Shattow

Hi Andrew,

Quoted 5 lines · Andrew Rembrandt
Quoted 1 line · Inochi Amaoto

Do not do this, just leave the rx-internal-delay-ps untouched.

Sounds good, I'll leave v1 as-is. Many thanks for the review.

Quoted 8 lines

Regards,

Andrew

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

I do not notice any difference with or without this property present, tested Linux -next with U-Boot modification to devicetree:

load usb 0:1 $fdt_addr_r /dtb/spacemit/k3-pico-itx.dtb fdt addr $fdt_addr_r
fdt resize
fdt header get filesize totalsize
fdt set /soc/ethernet@cac80000 tx-internal-delay-ps <400> load usb 0:1 $kernel_addr_r /EFI/BOOT/BOOTRISCV64.EFI

# throwaway use of ping is workaround for broken vendor U-Boot
ping 127.0.0.1; bootefi $kernel_addr_r $fdt_addr_r

Aside, there is still some problem with the driver which does cause it to fail. This can be reproduced by unloading the module and then loading the module; built from -next with 'olddefconfig' of debian Kconfig for kernel 7.1, no patches.

Quoted 8 lines
# modprobe -r dwmac_spacemit
# modprobe dwmac_spacemit
[   61.469745] spacemit-dwmac cac80000.ethernet end0: cannot attach to
PHY (error: -ENODEV)
[   91.544646] spacemit-dwmac cac80000.ethernet end0: cannot attach to

PHY (error: -ENODEV)

The problem is unrelated to this patch itself, but I can actually confirm here

Quoted 5 lines

If not reloading the module then I don't see any difference, traffic passes (ping 1.1.1.1) with or without your suggested tx-internal-delay-ps property. Is there a specific test to run to observe expected difference?

Although I haven't narrowed down to the root cause, but it's kind of something wrong with the reset procedure, remove it from phy node will fix this issue..

+++ b/arch/riscv/boot/dts/spacemit/k3-pico-itx.dts

Review mdio { · line 236

Quoted 8 lines
(quoted diff) +0 -3
@@ -232,9 +232,6 @@ mdio {
                phy0: phy@1 {
                        compatible = "ethernet-phy-ieee802.3-c22";
                        reg = <1>;
-                       reset-gpios = <&gpio 0 15 GPIO_ACTIVE_LOW>;
-                       reset-assert-us = <10000>;
-                       reset-deassert-us = <10000>;
 
 

Yes, confirming that workaround (removing nodes as you suggested) is functional for debian-installer where that reset trouble is being observed. If there is a patch or discussion later I will follow up then, also. On the topic of this ethernet TX timing thread I don't know what this does? Is there a test situation I may try to know if it is doing anything? Thanks very much, -E

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#8 lore raw

Hi E, Yixun,

Quoted 3 lines · E Shattow

On the topic of this ethernet TX timing thread I don't know what this does? Is there a test situation I may try to know if it is doing anything? Thanks very much, -E

Apologies for the slow reply -- I've been swamped the past month.

Thank you for taking the time to test this; your results are what prompted me to go back and re-test properly, and it turns out my original premise was wrong.

The non-functional TX I originally saw was not fixed by this delay property at all. An unrelated commit that landed between v7.1-rc1 and v7.1-rc7 is what actually fixed the dead TX on my board. While developing this patch I'd inadvertently picked up that change, and I wrongly concluded the tx-internal-delay-ps addition was what got traffic flowing.

To confirm, I re-ran a series of iperf3 tests without the delay patch and saw no transmission problem whatsoever -- a full 1Gb/s in both directions with no errors. So this matches exactly what you and perhaps Yixun also observed: on a current base the property makes no measurable difference, because the TX path already works without it.

Given that, the patch isn't the fix as described in the changelog, and the property has no functional effect on this board.

Yixun -- since you've already applied it, could you drop it from your tree? A revert is fine too if it's landed somewhere non-rewindable, but I don't think it should go upstream as-is: the "Fixes:" tag and the "no ethernet traffic passes" rationale are both wrong. Sorry for the hassle, and thanks for queuing it in good faith.

Sorry again for the noise, and thanks E for the careful testing that caught this.

maintainer
#9 lore raw

Hi Andrew,

Quoted 28 lines · Andrew Rembrandt

Hi E, Yixun,

Quoted 3 lines · E Shattow

On the topic of this ethernet TX timing thread I don't know what this does? Is there a test situation I may try to know if it is doing anything? Thanks very much, -E

Apologies for the slow reply -- I've been swamped the past month.

Thank you for taking the time to test this; your results are what prompted me to go back and re-test properly, and it turns out my original premise was wrong.

The non-functional TX I originally saw was not fixed by this delay property at all. An unrelated commit that landed between v7.1-rc1 and v7.1-rc7 is what actually fixed the dead TX on my board. While developing this patch I'd inadvertently picked up that change, and I wrongly concluded the tx-internal-delay-ps addition was what got traffic flowing.

To confirm, I re-ran a series of iperf3 tests without the delay patch and saw no transmission problem whatsoever -- a full 1Gb/s in both directions with no errors. So this matches exactly what you and perhaps Yixun also observed: on a current base the property makes no measurable difference, because the TX path already works without it.

Given that, the patch isn't the fix as described in the changelog, and the property has no functional effect on this board.

..

Quoted 2 lines

Yixun -- since you've already applied it, could you drop it from your tree? A revert is fine too if it's landed somewhere non-rewindable, but

I can drop this patch

Quoted 4 lines

I don't think it should go upstream as-is: the "Fixes:" tag and the "no ethernet traffic passes" rationale are both wrong. Sorry for the hassle, and thanks for queuing it in good faith.

No worry

Quoted 3 lines

Sorry again for the noise, and thanks E for the careful testing that caught this.

Signature · 1 line
Yixun Lan (dlan)
#10 lore raw

Hi Andrew, Yixun,

Quoted 21 lines · Yixun Lan

Hi Andrew,

Quoted 18 lines · Andrew Rembrandt

Hi E, Yixun,

Quoted 3 lines · E Shattow

On the topic of this ethernet TX timing thread I don't know what this does? Is there a test situation I may try to know if it is doing anything? Thanks very much, -E

Apologies for the slow reply -- I've been swamped the past month.

Thank you for taking the time to test this; your results are what prompted me to go back and re-test properly, and it turns out my original premise was wrong.

The non-functional TX I originally saw was not fixed by this delay property at all. An unrelated commit that landed between v7.1-rc1 and v7.1-rc7 is what actually fixed the dead TX on my board. While developing this patch I'd inadvertently picked up that change, and I wrongly concluded the tx-internal-delay-ps addition was what got traffic flowing.

I guess this refers to commit 3ea695eb111f ("dts: riscv: spacemit: k3:
Fix I/O power settings") pulled in for 7.2 and with the fixes tag also applied for 7.1 prior to 7.1-rc6 tag. That was confusing for me until having the benefit of an explanation by the Fedora RISC-V special interest group discussion member(s) highlighting that patch, as I was experiencing three distinct problems for that network interface. My use of debian-installer just happens to be a reproducer for all three problems. The Fedora RISC-V SIG members did not have difficulty with their distro development processes on two of the three problems I experience which led me to identify there are remaining problems not fixed in 7.1

Quoted 6 lines
Quoted 6 lines

To confirm, I re-ran a series of iperf3 tests without the delay patch and saw no transmission problem whatsoever -- a full 1Gb/s in both directions with no errors. So this matches exactly what you and perhaps Yixun also observed: on a current base the property makes no measurable difference, because the TX path already works without it.

Andrew, there are remaining problems of de-assert delay [1] and rmmod/modprobe cycle failure [2]. I suggest you test and validate the proposed changes so providing another data point of [1] the minimum delay that is successful. The rmmod/modprobe cycling workaround [2] can also benefit from testing. In the case of Kconfig with k3 networking =m as modules, or =y compiled-in either way it would be helpful to know that your use of networking is not disrupted in some unexpected way with the expected fixes.

Yixun, are patches [1] [2] expected as fixes for 7.2 ?

1:
https://lore.kernel.org/lkml/[email protected]/
2: https://lore.kernel.org/lkml/[email protected]/

- E

maintainer Reviewed-by on v1 1/1
#11 lore raw
In reply to Andrew Rembrandt

Hi Andrew,

Quoted 4 lines · Andrew Rembrandt

The "rgmii-id" PHY mode applies a symmetric 2ns internal delay in the PHY. On the Pico-ITX board this leaves insufficient TX timing margin and no ethernet traffic passes.

So I think you are the only one who reporting this issue? My board works fine with or without this patch, or is there any particular case that can expose this? otherwise I'd say it's an individual board issue.

Quoted 5 lines

Add a 400ps MAC-side TX internal delay on top of the PHY's delay to correct the TX clock/data skew. The value follows the vendor fork's asymmetric tuning (tx-phase 73 x 33.3ps - 2000ps ID ~= 400ps); the driver rounds tx-internal-delay-ps to the nearest supported step.

But since the property copied from vendor, so I believe problem also noticed in vendor's kernell which imply it do exist

Quoted 8 lines

This builds on commit 3ea695eb111f ("dts: riscv: spacemit: k3: Fix I/O power settings"), which fixed the GMAC0 1.8V/3.3V I/O switch

Tested on Pico-ITX hardware with iperf3: 60s TCP each direction and 60s UDP, all at 1Gbps with no TX/RX errors on the interface.

Fixes 74657a376960 ("riscv: dts: spacemit: Add ethernet device for K3")

I'd plan to apply it for next tree (target v7.3), leave more time for people to test, let me know if you're not OK with this? and for this patch

Quoted 19 lines
---
 arch/riscv/boot/dts/spacemit/k3-pico-itx.dts | 1 +
 1 file changed, 1 insertion(+)
arch/riscv/boot/dts/spacemit/k3-pico-itx.dts +1 -0
index b89c1521e664..59aecaa2d32a 100644
@@ -186,6 +186,7 @@ &eth0 {
 	pinctrl-names = "default";
 	pinctrl-0 = <&gmac0_rgmii_0_cfg>, <&gmac0_phy_0_cfg>;
 	phy-mode = "rgmii-id";
+	tx-internal-delay-ps = <400>;
 	phy-handle = <&phy0>;
 	status = "okay";
 
Signature · 1 line
2.54.0
Signature · 1 line
Yixun Lan (dlan)
maintainer Applied
#12 lore raw
In reply to Andrew Rembrandt
Quoted 10 lines · Andrew Rembrandt

The "rgmii-id" PHY mode applies a symmetric 2ns internal delay in the PHY. On the Pico-ITX board this leaves insufficient TX timing margin and no ethernet traffic passes.

Add a 400ps MAC-side TX internal delay on top of the PHY's delay to correct the TX clock/data skew. The value follows the vendor fork's asymmetric tuning (tx-phase 73 x 33.3ps - 2000ps ID ~= 400ps); the driver rounds tx-internal-delay-ps to the nearest supported step.

[...]

Applied, thanks!

[1/1] riscv: dts: spacemit: k3-pico-itx: Fix non-functional ethernet TX timing

      https://git.kernel.org/pub/scm/linux/kernel/git/spacemit/linux.git/commit/?id=773a0731959424d7f82be207b64acfa5b99a4978

Best regards,

Signature · 1 line
Yixun Lan <[email protected]>