riscv: dts: spacemit: k3-pico-itx: Fix non-functional ethernet TX timing
Subsystems (3)
PATCH 1/1 riscv: dts: spacemit: k3-pico-itx: Fix non-functional ethernet TX timing
The "rgmii-id" PHY mode applies a symmetric 2ns internal delay in the PHY. On the Pico-ITX board this leaves insufficient TX timing margin and no ethernet traffic passes.
Add a 400ps MAC-side TX internal delay on top of the PHY's delay to correct the TX clock/data skew. The value follows the vendor fork's asymmetric tuning (tx-phase 73 x 33.3ps - 2000ps ID ~= 400ps); the driver rounds tx-internal-delay-ps to the nearest supported step.
This builds on commit 3ea695eb111f ("dts: riscv: spacemit: k3: Fix I/O power settings"), which fixed the GMAC0 1.8V/3.3V I/O switch
Tested on Pico-ITX hardware with iperf3: 60s TCP each direction and 60s UDP, all at 1Gbps with no TX/RX errors on the interface.
--- arch/riscv/boot/dts/spacemit/k3-pico-itx.dts | 1 + 1 file changed, 1 insertion(+)
arch/riscv/boot/dts/spacemit/k3-pico-itx.dts +1 -0
@@ -186,6 +186,7 @@ ð0 { pinctrl-names = "default"; pinctrl-0 = <&gmac0_rgmii_0_cfg>, <&gmac0_phy_0_cfg>; phy-mode = "rgmii-id"; + tx-internal-delay-ps = <400>; phy-handle = <&phy0>; status = "okay";
Signature · 1 line
2.54.0
Quoted 26 lines · Andrew Rembrandt
The "rgmii-id" PHY mode applies a symmetric 2ns internal delay in the PHY. On the Pico-ITX board this leaves insufficient TX timing margin and no ethernet traffic passes.
Add a 400ps MAC-side TX internal delay on top of the PHY's delay to correct the TX clock/data skew. The value follows the vendor fork's asymmetric tuning (tx-phase 73 x 33.3ps - 2000ps ID ~= 400ps); the driver rounds tx-internal-delay-ps to the nearest supported step.
This builds on commit
3ea695eb111f("dts: riscv: spacemit: k3: Fix I/O power settings"), which fixed the GMAC0 1.8V/3.3V I/O switchTested on Pico-ITX hardware with iperf3: 60s TCP each direction and 60s UDP, all at 1Gbps with no TX/RX errors on the interface.
Fixes 74657a376960 ("riscv: dts: spacemit: Add ethernet device for K3")Signed-off-by Andrew Rembrandt <[email protected]>--- arch/riscv/boot/dts/spacemit/k3-pico-itx.dts | 1 + 1 file changed, 1 insertion(+)arch/riscv/boot/dts/spacemit/k3-pico-itx.dts +0
-0
arch/riscv/boot/dts/spacemit/k3-pico-itx.dts ð0 { · line 192
Quoted 7 lines
arch/riscv/boot/dts/spacemit/k3-pico-itx.dts +1
-0@@ -186,6 +186,7 @@ ð0 { pinctrl-names = "default"; pinctrl-0 = <&gmac0_rgmii_0_cfg>, <&gmac0_phy_0_cfg>; phy-mode = "rgmii-id"; + tx-internal-delay-ps = <400>; phy-handle = <&phy0>; status = "okay";
You should set tx-internal-delay-ps of the phy instead of the controller. This is a phy in "rgmii-id" mode.
Regards,
Inochi
Quoted 2 lines · Inochi Amaoto
You should set tx-internal-delay-ps of the phy instead of the controller. This is a phy in "rgmii-id" mode.
Thanks for the review. I don't think that works on this board, but please let me know if I'm missing something.
The PHY is a Realtek RTL8211F. Its driver only toggles the chip's fixed ~2ns delay on/off from phy-mode, and per the datasheet that's a hardware limit, not a driver one -- the RTL8211F has no tunable delay, so a ps value on the PHY node has nothing to act on.
The only configurable delay element on K3 is the SoC RGMII delay line, and dwmac-spacemit programs it from tx-/rx-internal-delay-ps read off the controller node (spacemit_dwmac_probe()), so that is where the property has to live to take effect. This is the MAC "fine tuning ... values expected here are small" case described in Documentation/devicetree/bindings/net/ethernet-controller.yaml: the RTL8211F supplies the fixed ~2ns via rgmii-id and the MAC adds a 400ps trim on top. Ideally the PHY would own the delay, but as Andrew Lunn noted reviewing dwmac-meson8b, that requires the PHY to implement a configurable delay, which not all do.
The already-merged k1-bananapi-f3 does the same thing: eth1 combines phy-mode = "rgmii-id" with tx-internal-delay-ps = <250> on the controller node (via the K1 EMAC rather than dwmac, but the same DT pattern).
Would you be OK keeping it on the controller given the above? For v2 I can add an explicit rx-internal-delay-ps = <0> to match the other SpacemiT boards (k1-bananapi-f3 included) and spell this reasoning out in the commit message.
Thanks,
Andrew
Inochi Amaoto
Reviewed-byQuoted 22 lines · Andrew Rembrandt
Quoted 2 lines · Inochi Amaoto
You should set tx-internal-delay-ps of the phy instead of the controller. This is a phy in "rgmii-id" mode.
Thanks for the review. I don't think that works on this board, but please let me know if I'm missing something.
The PHY is a Realtek RTL8211F. Its driver only toggles the chip's fixed ~2ns delay on/off from phy-mode, and per the datasheet that's a hardware limit, not a driver one -- the RTL8211F has no tunable delay, so a ps value on the PHY node has nothing to act on.
The only configurable delay element on K3 is the SoC RGMII delay line, and dwmac-spacemit programs it from tx-/rx-internal-delay-ps read off the controller node (
spacemit_dwmac_probe()), so that is where the property has to live to take effect. This is the MAC "fine tuning ... values expected here are small" case described inDocumentation/devicetree/bindings/net/ethernet-controller.yaml: the RTL8211F supplies the fixed ~2ns via rgmii-id and the MAC adds a 400ps trim on top. Ideally the PHY would own the delay, but as Andrew Lunn noted reviewing dwmac-meson8b, that requires the PHY to implement a configurable delay, which not all do.
Quoted 6 lines
The already-merged k1-bananapi-f3 does the same thing: eth1 combines phy-mode = "rgmii-id" with tx-internal-delay-ps = <250> on the controller node (via the K1 EMAC rather than dwmac, but the same DT pattern).
Would you be OK keeping it on the controller given the above?
Yes, it is good for me.
Quoted 4 lines
For v2 I can add an explicit rx-internal-delay-ps = <0> to match the other SpacemiT boards (k1-bananapi-f3 included) and spell this reasoning out in the commit message.
Do not do this, just leave the rx-internal-delay-ps untouched.
For this patch
Regards,
Inochi.
Quoted 1 line · Inochi Amaoto
Do not do this, just leave the rx-internal-delay-ps untouched.
Sounds good, I'll leave v1 as-is. Many thanks for the review.
Regards,
Andrew
Hi Andrew,
Quoted 5 lines · Andrew Rembrandt
Quoted 1 line · Inochi Amaoto
Do not do this, just leave the rx-internal-delay-ps untouched.
Sounds good, I'll leave v1 as-is. Many thanks for the review.
Quoted 8 lines
Regards,
Andrew
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
I do not notice any difference with or without this property present, tested Linux -next with U-Boot modification to devicetree:
load usb 0:1 $fdt_addr_r /dtb/spacemit/k3-pico-itx.dtb fdt addr $fdt_addr_r
fdt resize
fdt header get filesize totalsize
fdt set /soc/ethernet@cac80000 tx-internal-delay-ps <400> load usb 0:1 $kernel_addr_r /EFI/BOOT/BOOTRISCV64.EFI
# throwaway use of ping is workaround for broken vendor U-Boot ping 127.0.0.1; bootefi $kernel_addr_r $fdt_addr_r
Aside, there is still some problem with the driver which does cause it to fail. This can be reproduced by unloading the module and then loading the module; built from -next with 'olddefconfig' of debian Kconfig for kernel 7.1, no patches.
# modprobe -r dwmac_spacemit # modprobe dwmac_spacemit [ 61.469745] spacemit-dwmac cac80000.ethernet end0: cannot attach to PHY (error: -ENODEV)
[ 91.544646] spacemit-dwmac cac80000.ethernet end0: cannot attach to
PHY (error: -ENODEV)
If not reloading the module then I don't see any difference, traffic passes (ping 1.1.1.1) with or without your suggested tx-internal-delay-ps property. Is there a specific test to run to observe expected difference?
Thanks,
-E
Hi Andrew,
Quoted 4 lines · Andrew Rembrandt
The "rgmii-id" PHY mode applies a symmetric 2ns internal delay in the PHY. On the Pico-ITX board this leaves insufficient TX timing margin and no ethernet traffic passes.
So I think you are the only one who reporting this issue? My board works fine with or without this patch, or is there any particular case that can expose this? otherwise I'd say it's an individual board issue.
Quoted 5 lines
Add a 400ps MAC-side TX internal delay on top of the PHY's delay to correct the TX clock/data skew. The value follows the vendor fork's asymmetric tuning (tx-phase 73 x 33.3ps - 2000ps ID ~= 400ps); the driver rounds tx-internal-delay-ps to the nearest supported step.
But since the property copied from vendor, so I believe problem also noticed in vendor's kernell which imply it do exist
Quoted 8 lines
This builds on commit
3ea695eb111f("dts: riscv: spacemit: k3: Fix I/O power settings"), which fixed the GMAC0 1.8V/3.3V I/O switchTested on Pico-ITX hardware with iperf3: 60s TCP each direction and 60s UDP, all at 1Gbps with no TX/RX errors on the interface.
Fixes 74657a376960 ("riscv: dts: spacemit: Add ethernet device for K3")Signed-off-by Andrew Rembrandt <[email protected]>
I'd plan to apply it for next tree (target v7.3), leave more time for people to test, let me know if you're not OK with this? and for this patch
Quoted 19 lines
--- arch/riscv/boot/dts/spacemit/k3-pico-itx.dts | 1 + 1 file changed, 1 insertion(+)arch/riscv/boot/dts/spacemit/k3-pico-itx.dts +1
-0@@ -186,6 +186,7 @@ ð0 { pinctrl-names = "default"; pinctrl-0 = <&gmac0_rgmii_0_cfg>, <&gmac0_phy_0_cfg>; phy-mode = "rgmii-id"; + tx-internal-delay-ps = <400>; phy-handle = <&phy0>; status = "okay";Signature · 1 line
2.54.0
Signature · 1 line
Yixun Lan (dlan)
Yixun Lan
maintainerHi E Shattow,
Quoted 34 lines · E Shattow
Hi Andrew,
Quoted 5 lines · Andrew Rembrandt
Quoted 1 line · Inochi Amaoto
Do not do this, just leave the rx-internal-delay-ps untouched.
Sounds good, I'll leave v1 as-is. Many thanks for the review.
Quoted 8 lines
Regards,
Andrew
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscvI do not notice any difference with or without this property present, tested Linux -next with U-Boot modification to devicetree:
load usb 0:1 $fdt_addr_r /dtb/spacemit/k3-pico-itx.dtb fdt addr $fdt_addr_r
fdt resize
fdt header get filesize totalsize
fdt set /soc/ethernet@cac80000 tx-internal-delay-ps <400> load usb 0:1 $kernel_addr_r /EFI/BOOT/BOOTRISCV64.EFI# throwaway use of ping is workaround for broken vendor U-Boot ping 127.0.0.1; bootefi $kernel_addr_r $fdt_addr_rAside, there is still some problem with the driver which does cause it to fail. This can be reproduced by unloading the module and then loading the module; built from -next with 'olddefconfig' of debian
Kconfigfor kernel 7.1, no patches.
Quoted 8 lines
# modprobe -r dwmac_spacemit # modprobe dwmac_spacemit [ 61.469745] spacemit-dwmac cac80000.ethernet end0: cannot attach to PHY (error: -ENODEV)[ 91.544646] spacemit-dwmac cac80000.ethernet end0: cannot attach toPHY (error:
-ENODEV)
The problem is unrelated to this patch itself, but I can actually confirm here
Quoted 5 lines
If not reloading the module then I don't see any difference, traffic passes (ping 1.1.1.1) with or without your suggested tx-internal-delay-ps property. Is there a specific test to run to observe expected difference?
Although I haven't narrowed down to the root cause, but it's kind of something wrong with the reset procedure, remove it from phy node will fix this issue..
+++ b/arch/riscv/boot/dts/spacemit/k3-pico-itx.dts
@@ -232,9 +232,6 @@ mdio {
phy0: phy@1 {
compatible = "ethernet-phy-ieee802.3-c22";
reg = <1>;
- reset-gpios = <&gpio 0 15 GPIO_ACTIVE_LOW>;
- reset-assert-us = <10000>;
- reset-deassert-us = <10000>;Signature · 1 line
Yixun Lan (dlan)
Hi Yixun,
Quoted 59 lines · Yixun Lan
Hi E Shattow,
Quoted 34 lines · E Shattow
Hi Andrew,
Quoted 5 lines · Andrew Rembrandt
Quoted 1 line · Inochi Amaoto
Do not do this, just leave the rx-internal-delay-ps untouched.
Sounds good, I'll leave v1 as-is. Many thanks for the review.
Quoted 8 lines
Regards,
Andrew
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscvI do not notice any difference with or without this property present, tested Linux -next with U-Boot modification to devicetree:
load usb 0:1 $fdt_addr_r /dtb/spacemit/k3-pico-itx.dtb fdt addr $fdt_addr_r
fdt resize
fdt header get filesize totalsize
fdt set /soc/ethernet@cac80000 tx-internal-delay-ps <400> load usb 0:1 $kernel_addr_r /EFI/BOOT/BOOTRISCV64.EFI# throwaway use of ping is workaround for broken vendor U-Boot ping 127.0.0.1; bootefi $kernel_addr_r $fdt_addr_rAside, there is still some problem with the driver which does cause it to fail. This can be reproduced by unloading the module and then loading the module; built from -next with 'olddefconfig' of debian
Kconfigfor kernel 7.1, no patches.Quoted 8 lines
# modprobe -r dwmac_spacemit # modprobe dwmac_spacemit [ 61.469745] spacemit-dwmac cac80000.ethernet end0: cannot attach to PHY (error: -ENODEV)[ 91.544646] spacemit-dwmac cac80000.ethernet end0: cannot attach toPHY (error:
-ENODEV)The problem is unrelated to this patch itself, but I can actually confirm here
Quoted 5 lines
If not reloading the module then I don't see any difference, traffic passes (ping 1.1.1.1) with or without your suggested tx-internal-delay-ps property. Is there a specific test to run to observe expected difference?
Although I haven't narrowed down to the root cause, but it's kind of something wrong with the reset procedure, remove it from phy node will fix this issue..
+++
b/arch/riscv/boot/dts/spacemit/k3-pico-itx.dts
Review mdio { · line 236
Quoted 8 lines
(quoted diff) +0
-3@@ -232,9 +232,6 @@ mdio { phy0: phy@1 { compatible = "ethernet-phy-ieee802.3-c22"; reg = <1>; - reset-gpios = <&gpio 0 15 GPIO_ACTIVE_LOW>; - reset-assert-us = <10000>; - reset-deassert-us = <10000>;
Yes, confirming that workaround (removing nodes as you suggested) is functional for debian-installer where that reset trouble is being observed. If there is a patch or discussion later I will follow up then, also. On the topic of this ethernet TX timing thread I don't know what this does? Is there a test situation I may try to know if it is doing anything? Thanks very much, -E
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 10 lines · Andrew Rembrandt
The "rgmii-id" PHY mode applies a symmetric 2ns internal delay in the PHY. On the Pico-ITX board this leaves insufficient TX timing margin and no ethernet traffic passes.
Add a 400ps MAC-side TX internal delay on top of the PHY's delay to correct the TX clock/data skew. The value follows the vendor fork's asymmetric tuning (tx-phase 73 x 33.3ps - 2000ps ID ~= 400ps); the driver rounds tx-internal-delay-ps to the nearest supported step.
[...]
Applied, thanks!
[1/1] riscv: dts: spacemit: k3-pico-itx: Fix non-functional ethernet TX timing
https://git.kernel.org/pub/scm/linux/kernel/git/spacemit/linux.git/commit/?id=773a0731959424d7f82be207b64acfa5b99a4978
Best regards,
Signature · 1 line
Yixun Lan <[email protected]>
Hi E, Yixun,
Quoted 3 lines · E Shattow
On the topic of this ethernet TX timing thread I don't know what this does? Is there a test situation I may try to know if it is doing anything? Thanks very much, -E
Apologies for the slow reply -- I've been swamped the past month.
Thank you for taking the time to test this; your results are what prompted me to go back and re-test properly, and it turns out my original premise was wrong.
The non-functional TX I originally saw was not fixed by this delay property at all. An unrelated commit that landed between v7.1-rc1 and v7.1-rc7 is what actually fixed the dead TX on my board. While developing this patch I'd inadvertently picked up that change, and I wrongly concluded the tx-internal-delay-ps addition was what got traffic flowing.
To confirm, I re-ran a series of iperf3 tests without the delay patch and saw no transmission problem whatsoever -- a full 1Gb/s in both directions with no errors. So this matches exactly what you and perhaps Yixun also observed: on a current base the property makes no measurable difference, because the TX path already works without it.
Given that, the patch isn't the fix as described in the changelog, and the property has no functional effect on this board.
Yixun -- since you've already applied it, could you drop it from your tree? A revert is fine too if it's landed somewhere non-rewindable, but I don't think it should go upstream as-is: the "Fixes:" tag and the "no ethernet traffic passes" rationale are both wrong. Sorry for the hassle, and thanks for queuing it in good faith.
Sorry again for the noise, and thanks E for the careful testing that caught this.
Yixun Lan
maintainerHi Andrew,
Quoted 28 lines · Andrew Rembrandt
Hi E, Yixun,
Quoted 3 lines · E Shattow
On the topic of this ethernet TX timing thread I don't know what this does? Is there a test situation I may try to know if it is doing anything? Thanks very much, -E
Apologies for the slow reply -- I've been swamped the past month.
Thank you for taking the time to test this; your results are what prompted me to go back and re-test properly, and it turns out my original premise was wrong.
The non-functional TX I originally saw was not fixed by this delay property at all. An unrelated commit that landed between v7.1-rc1 and v7.1-rc7 is what actually fixed the dead TX on my board. While developing this patch I'd inadvertently picked up that change, and I wrongly concluded the tx-internal-delay-ps addition was what got traffic flowing.
To confirm, I re-ran a series of iperf3 tests without the delay patch and saw no transmission problem whatsoever -- a full 1Gb/s in both directions with no errors. So this matches exactly what you and perhaps Yixun also observed: on a current base the property makes no measurable difference, because the TX path already works without it.
Given that, the patch isn't the fix as described in the changelog, and the property has no functional effect on this board.
..
Quoted 2 lines
Yixun -- since you've already applied it, could you drop it from your tree? A revert is fine too if it's landed somewhere non-rewindable, but
I can drop this patch
Quoted 4 lines
I don't think it should go upstream as-is: the "Fixes:" tag and the "no ethernet traffic passes" rationale are both wrong. Sorry for the hassle, and thanks for queuing it in good faith.
No worry
Quoted 3 lines
Sorry again for the noise, and thanks E for the careful testing that caught this.
Signature · 1 line
Yixun Lan (dlan)
Hi Andrew, Yixun,
Quoted 21 lines · Yixun Lan
Hi Andrew,
Quoted 18 lines · Andrew Rembrandt
Hi E, Yixun,
Quoted 3 lines · E Shattow
On the topic of this ethernet TX timing thread I don't know what this does? Is there a test situation I may try to know if it is doing anything? Thanks very much, -E
Apologies for the slow reply -- I've been swamped the past month.
Thank you for taking the time to test this; your results are what prompted me to go back and re-test properly, and it turns out my original premise was wrong.
The non-functional TX I originally saw was not fixed by this delay property at all. An unrelated commit that landed between v7.1-rc1 and v7.1-rc7 is what actually fixed the dead TX on my board. While developing this patch I'd inadvertently picked up that change, and I wrongly concluded the tx-internal-delay-ps addition was what got traffic flowing.
I guess this refers to commit 3ea695eb111f ("dts: riscv: spacemit: k3:
Fix I/O power settings") pulled in for 7.2 and with the fixes tag also applied for 7.1 prior to 7.1-rc6 tag. That was confusing for me until having the benefit of an explanation by the Fedora RISC-V special interest group discussion member(s) highlighting that patch, as I was experiencing three distinct problems for that network interface. My use of debian-installer just happens to be a reproducer for all three problems. The Fedora RISC-V SIG members did not have difficulty with their distro development processes on two of the three problems I experience which led me to identify there are remaining problems not fixed in 7.1
Quoted 6 lines
Quoted 6 lines
To confirm, I re-ran a series of iperf3 tests without the delay patch and saw no transmission problem whatsoever -- a full 1Gb/s in both directions with no errors. So this matches exactly what you and perhaps Yixun also observed: on a current base the property makes no measurable difference, because the TX path already works without it.
Andrew, there are remaining problems of de-assert delay [1] and rmmod/modprobe cycle failure [2]. I suggest you test and validate the proposed changes so providing another data point of [1] the minimum delay that is successful. The rmmod/modprobe cycling workaround [2] can also benefit from testing. In the case of Kconfig with k3 networking =m as modules, or =y compiled-in either way it would be helpful to know that your use of networking is not disrupted in some unexpected way with the expected fixes.
Yixun, are patches [1] [2] expected as fixes for 7.2 ?
1:
https://lore.kernel.org/lkml/[email protected]/
2: https://lore.kernel.org/lkml/[email protected]/
- E