mv88e6xxx: 2500base-x inband AN is broken on Amethyst? what to do?
Subsystems (2)
Hello,
it seems that inband AN is broken on Amethyst when on 2500base-x mode.
Even SERDES scripts for Amethyst from Marvell has autonegotiation disabled for 2500base-x mode. For all the other supported Serdes modes autonegotiation is enabled in these scripts.
The current implementation in mv88e6390_serdes_pcs_config() enables autonegotiation if phylink_autoneg_inband(mode) is true:
if (phylink_autoneg_inband(mode))
bmcr = val | BMCR_ANENABLE;
else
bmcr = val & ~BMCR_ANENABLE;But for PHY_INTERFACE_MODE_2500BASEX this is broken on Amethyst. The 2500base-x mode seems to work only with autoneg disabled.
The result is that when I connect via a passive SFP cable Amethyst serdes port with a Peridot serdes port, they will not link. If I disable autonegotiation on both sides, they will link, though.
What is strange is that if I don't use Peridot, but connect the SFP directly to Serdes on Armada 3720, where the mvneta driver also enables autonegotiation for 2500base-x mode, they will link even if Amethyst does not enable 2500base-x.
To summarize:
Amethyst <-> Peridot AN - AN - works AN - AN + does not work Amethyst <-> Armada 3720 serdes AN - AN + works
(It is possible that Marvell may find some workaround by touch some
undocumented registers, to solve this. I will try to open a bug report.)
Should we just print an error in the serdes_pcs_config method if inband autonegotiation is being requested?
phylink's code currently allows connecting SFPs in non MLO_AN_INBAND mode only for when there is Broadcom BCM84881 PHY inside the SFP (by method phylink_phy_no_inband() in phylink.c).
I wonder whether we can somehow in a sane way implement code to inform phylink from the mv88e6xxx driver that inband is not supported for the specific mode. Maybe the .mac_config/.pcs_config method could return an error indicating this? Or the mv88e6xxx driver can just print an error that the mode is not supported, and try to ask the user to disable AN? That would need implementing this in ethtool for SFP, though.
What do you guys think?
Marek
It seems this problem can manifest somehow with Peridot as well, at least when in combination with 88X3310 PHY.
I will do some more experiments and try to come up with a solution.
Marek
Quoted 47 lines · Marek Behún
Hello,
it seems that inband AN is broken on Amethyst when on 2500base-x mode.
Even SERDES scripts for Amethyst from Marvell has autonegotiation disabled for 2500base-x mode. For all the other supported Serdes modes autonegotiation is enabled in these scripts.
The current implementation in
mv88e6390_serdes_pcs_config()enables autonegotiation ifphylink_autoneg_inband(mode)is true:if (phylink_autoneg_inband(mode)) bmcr = val | BMCR_ANENABLE; else bmcr = val & ~BMCR_ANENABLE;But for
PHY_INTERFACE_MODE_2500BASEXthis is broken on Amethyst. The 2500base-x mode seems to work only with autoneg disabled.The result is that when I connect via a passive SFP cable Amethyst serdes port with a Peridot serdes port, they will not link. If I disable autonegotiation on both sides, they will link, though.
What is strange is that if I don't use Peridot, but connect the SFP directly to Serdes on Armada 3720, where the mvneta driver also enables autonegotiation for 2500base-x mode, they will link even if Amethyst does not enable 2500base-x.
To summarize:
Amethyst <-> Peridot AN - AN - works AN - AN + does not work Amethyst <-> Armada 3720 serdes AN - AN + works(It is possible that Marvell may find some workaround by touch some
undocumented registers, to solve this. I will try to open a bug report.)Should we just print an error in the
serdes_pcs_configmethod if inband autonegotiation is being requested?phylink's code currently allows connecting SFPs in non
MLO_AN_INBANDmode only for when there is Broadcom BCM84881 PHY inside the SFP (by methodphylink_phy_no_inband()inphylink.c).
Quoted 8 lines
I wonder whether we can somehow in a sane way implement code to inform phylink from the mv88e6xxx driver that inband is not supported for the specific mode. Maybe the .mac_config/.pcs_config method could return an error indicating this? Or the mv88e6xxx driver can just print an error that the mode is not supported, and try to ask the user to disable AN? That would need implementing this in ethtool for SFP, though.
What do you guys think?
It's regrettable that some have decided not to have working in-band AN for 2500base-X, since it prevents the automatic use of pause modes - which is essentially the only use of in-band negotiation with 1000base-X since many Gigabit MACs do not support half duplex.
I don't know whether the 88x3310 on the host side uses AN or not for 2500base-X - that detail isn't mentioned in the datasheet, and I don't have a setup that I could test that with.
I had assumed that most people would implement 2500base-X as merely an upclocked 1000base-X, as mvneta does - unfortunately, there is /no/ published standard for 2500base-X, so we're left guessing what this should be from the implementations available at the time.
As you have found, whether AN is supported at 2500base-X appears to be pretty random.
However, phylib has no way to report whether a PHY wants to use in-band AN to the MAC. Hence the hack in the phylink code for BCM84881. We really need a better way for phylib to communicate the capabilities of the PHY to its user (things like which interface modes are supported, which interface modes may be dynamically switched between, and whether they can use in-band AN, and whether in-band AN bypass is supported). Also similar properties for MAC/PCS drivers, then phylink can work out what should be done.
That still leaves an issue for SFP modules - when they're capable of supporting 2500base-X, we have no knowledge of the remote side. This is where knowing whether the MAC/PCS supports in-band AN bypass or not matters - if it doesn't then disabling in-band AN for 2500base-X may make sense, otherwise having in-and AN enabled but with bypass enabled would probably be sensible.
Right now, we just don't have the information to make the correct decisions.
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
Quoted 3 lines · On Wed, 13 Jan 2021 10:28:49 +0000 Russell King - ARM Linux admin…
I don't know whether the 88x3310 on the host side uses AN or not for 2500base-X - that detail isn't mentioned in the datasheet, and I don't have a setup that I could test that with.
It does not. I have been poking the registers on 88x3310, and also on Peridot and Amethyst SERDES PHYs when cmode on the switch is set to 2500base-x, and by default the inband AN is disabled.
Enabling it on the PHY in Amethyst does not work at all. On Peridot your code in mv88e6xxx enables it, but when 88x3310 is connected to Peridot SERDES, it does not work until the AN is disabled on Peridot.
Enabling the AN on 2500base-x mode on 88x3310 does not seem to work.
I will do some more tests by poking the registers and report this.
Marek
OK, so I did some tests with Peridot and 88X3310:
On 88E6390 switch, when CMODE in the port register for a port capable of SerDes is set to 1000base-x, the switch self-configures the SerDes PHY to inband AN:
register 4.2000 has value 0x1940
but when CMODE is set to 2500base-x, the switch self-configure the PHY without inband AN:
register 4.2000 has value 0x0940
Also the 88X3310 PHY, when configured with MACTYPE=4
(that is the mode that switches host interface between
10gbase-r/5gbase-r/2500base-x/sgmii, depending on copper speed)
and when copper speed is 2500, the PHY self-configures without inband AN:
register 4.2000 has value 0x0140
It seems to me that on these Marvell devices, they consider 2500base-x not capable of inband AN and disable it by default.
Moreover when the PHY has disabled inband AN and the Peridot switch has it enabled (by software), they won't link. I tried enabling the inband AN on the PHY, but it does not seem to work. Peridot can only communicate with the PHY in 2500base-x with inband AN disabled.
This means that the commit
a5a6858b793ff ("net: dsa: mv88e6xxx: extend phylink to Serdes PHYs")
causes a regression, since the code started enabling inband AN on 2500base-x mode on the mv88e6390 family, and they stopped working with the PHY.
Russell, could we, for now, just edit the code so that when
mv88e6390_serdes_pcs_config
is being configured with inband mode in 2500base-x, the inband mode won't be enabled and the function will print a warning? This could come with a Fixes tag so that it is backported to stable.
Afterwards we can work on refactoring the phylink code so that either the driver can inform phylink whether 2500base-x inband AN is supported, or maybe we can determine from some documentation or whatnot whether inband AN is supported on 2500base-x at all.
Marek
Quoted 9 lines · Marek Behún
Quoted 6 lines · On Wed, 13 Jan 2021 21:21:25 +0000 Russell King - ARM Linux admin…
One thing that I don't know is whether the GPON SFP ONT modules that use 2500base-x will still function with AN disabled - although I have the modules, it appeared that they both needed a connection to the ONU to switch from 1000base-x to 2500base-x on the host side - and as I don't have an ONU I can test with, I have no way to check their behaviour.
Quoted 3 lines
We have an ISP here in Prague who is willing to lend us some GPON ONU devices to get GPON SFP modules work on Turris. I will ask him if he has one capable of 2.5g and try to borrow it.
I know the Huawei MA5671A and the Nokia 3FE46541AA are - see the quirks table in sfp-bus.c ! They come from an ISP on the American continent who really really wanted these modules to link at 2.5Gbps.
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
Quoted 7 lines · On Wed, 13 Jan 2021 21:21:25 +0000 Russell King - ARM Linux admin…
Quoted 5 lines
Russell, could we, for now, just edit the code so that when
mv88e6390_serdes_pcs_config
is being configured with inband mode in 2500base-x, the inband mode won't be enabled and the function will print a warning? This could come with a Fixes tag so that it is backported to stable.I don't see any other easy option, so yes, please do that.
Will do.
Quoted 5 lines
Quoted 4 lines
Afterwards we can work on refactoring the phylink code so that either the driver can inform phylink whether 2500base-x inband AN is supported, or maybe we can determine from some documentation or whatnot whether inband AN is supported on 2500base-x at all.
Quoted 7 lines
I suspect there is no definitive documentation on exactly what 2500base-x actually is. I suspect it may just be easier to turn off AN for 2500base-x everywhere, so at least all Linux systems are compatible irrespective of the hardware.
Yes, it means losing pause negotiation, and people will have to manually set pause on each end.
This would need a little refactoring of the phylink_sfp_config function, but I don't think it will be that hard.
Quoted 6 lines
One thing that I don't know is whether the GPON SFP ONT modules that use 2500base-x will still function with AN disabled - although I have the modules, it appeared that they both needed a connection to the ONU to switch from 1000base-x to 2500base-x on the host side - and as I don't have an ONU I can test with, I have no way to check their behaviour.
We have an ISP here in Prague who is willing to lend us some GPON ONU devices to get GPON SFP modules work on Turris. I will ask him if he has one capable of 2.5g and try to borrow it.
Marek
Quoted 32 lines · Marek Behún
OK, so I did some tests with Peridot and 88X3310:
On 88E6390 switch, when CMODE in the port register for a port capable of SerDes is set to 1000base-x, the switch self-configures the SerDes PHY to inband AN:
register 4.2000 has value0x1940but when CMODE is set to 2500base-x, the switch self-configure the PHY without inband AN:
register 4.2000 has value0x0940Also the 88X3310 PHY, when configured with MACTYPE=4
(that is the mode that switches host interface between
10gbase-r/5gbase-r/2500base-x/sgmii, depending on copper speed)
and when copper speed is 2500, the PHY self-configures without inband AN:
register 4.2000 has value0x0140It seems to me that on these Marvell devices, they consider 2500base-x not capable of inband AN and disable it by default.
Moreover when the PHY has disabled inband AN and the Peridot switch has it enabled (by software), they won't link. I tried enabling the inband AN on the PHY, but it does not seem to work. Peridot can only communicate with the PHY in 2500base-x with inband AN disabled.
This means that the commit
a5a6858b793ff("net: dsa: mv88e6xxx: extend phylink to Serdes PHYs")
causes a regression, since the code started enabling inband AN on 2500base-x mode on the mv88e6390 family, and they stopped working with the PHY.
Quoted 5 lines
Russell, could we, for now, just edit the code so that when
mv88e6390_serdes_pcs_config
is being configured with inband mode in 2500base-x, the inband mode won't be enabled and the function will print a warning? This could come with a Fixes tag so that it is backported to stable.
I don't see any other easy option, so yes, please do that.
Quoted 4 lines
Afterwards we can work on refactoring the phylink code so that either the driver can inform phylink whether 2500base-x inband AN is supported, or maybe we can determine from some documentation or whatnot whether inband AN is supported on 2500base-x at all.
I suspect there is no definitive documentation on exactly what 2500base-x actually is. I suspect it may just be easier to turn off AN for 2500base-x everywhere, so at least all Linux systems are compatible irrespective of the hardware.
Yes, it means losing pause negotiation, and people will have to manually set pause on each end.
One thing that I don't know is whether the GPON SFP ONT modules that use 2500base-x will still function with AN disabled - although I have the modules, it appeared that they both needed a connection to the ONU to switch from 1000base-x to 2500base-x on the host side - and as I don't have an ONU I can test with, I have no way to check their behaviour.
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!