ocelot-pcs - qsgmii planning

DISCUSSION Colin Foster 2 participants 3 messages Jan 6, 2022

Subsystems (2)

Networking Network drivers

lore

Share & tools

lore

Colin Foster

author

2022-01-06 09:50 UTC #0 lore raw

Hi Alexandre, Vladimir, and those with interest of Ocelot chips,

I'm starting this thread to just touch base before I dive into any PCS changes for Ocelot. I've appreciated all your guidance, and the only time I felt I've been led astray is when Alexandre told me it should be easy :-)

I'm at the point where I'm starting to integrate the additional 4 copper ports of the VSC7512 reference board. They are 4 ports connected through a QSGMII bus, to a VSC8514 phy.

The 8514 driver seems to be getting invoked, and running just fine. Also, I was able to slightly modify (hack*) drivers/phy/mscc/phy-ocelot-serdes.c to work with my in-development ocelot-mfd. I believe that is what I need to configure the HSIO registers.

(*the device_is_mfd info I was using falls apart with the HSIO/syscon/mfd implementation here, sadly. A new probe function would easily clean that up, but it is more for me to think about... I digress)

I'm using these device tree settings:

    port@4 {
        reg = <4>;
        label = "swp4";
        status = "okay";
        phy-handle = <&sw_phy4>;
        phy-mode = "qsgmii";
        phys = <&serdes 4 SERDES6G(0)>;
    };
    port@5 {
        reg = <5>;
        label = "swp5";
        status = "okay";
        phy-handle = <&sw_phy5>;
        phy-mode = "qsgmii";
        phys = <&serdes 5 SERDES6G(0)>;
    };
...
    serdes: serdes {
        compatible = "mscc,vsc7514-serdes";
        #phy-cells = <2>;
    };
    mdio1: mdio1 {
        compatible = "mscc,ocelot-miim",
        pinctrl-names = "default";
        pinctrl-0 = <&miim1>;
        #address-cells = <1>;
        #size-cells = <0>;

        sw_phy4: ethernet-phy@4 {
            reg = <0x4>;
        };
    };

[    3.886787] libphy: ocelot_ext MDIO bus: probed
[    5.345891] ocelot-ext-switch ocelot-ext-switch: PHY [ocelot-ext-switch-mii:00] driver [Generic PHY] (irq=POLL)
[    5.357341] ocelot-ext-switch ocelot-ext-switch: configuring for phy/internal link mode
[    5.372525] ocelot-ext-switch ocelot-ext-switch swp1 (uninitialized): PHY [ocelot-ext-switch-mii:01] driver [Generic PHY] (irq=POLL)
[    5.388865] ocelot-ext-switch ocelot-ext-switch swp2 (uninitialized): PHY [ocelot-ext-switch-mii:02] driver [Generic PHY] (irq=POLL)
[    5.405086] ocelot-ext-switch ocelot-ext-switch swp3 (uninitialized): PHY [ocelot-ext-switch-mii:03] driver [Generic PHY] (irq=POLL)
[    6.291876] ocelot-ext-switch ocelot-ext-switch swp4 (uninitialized): PHY [ocelot-miim1-mii:04] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
[    6.471891] ocelot-ext-switch ocelot-ext-switch swp5 (uninitialized): PHY [ocelot-miim1-mii:05] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
[    6.651895] ocelot-ext-switch ocelot-ext-switch swp6 (uninitialized): PHY [ocelot-miim1-mii:06] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
[    6.831879] ocelot-ext-switch ocelot-ext-switch swp7 (uninitialized): PHY [ocelot-miim1-mii:07] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)

It seems like that, along with everything in vsc7514_phylink_mac_config, should be everything I need for operation of the four ports through the 8512. I've added OCELOT_QUIRK_SGMII_PORTS_MUST_BE_UP - but I'm not sure that's a quirk I need. Plus the only behavior it currently adds is once the port is up, it never comes back down.

The current behavior I'm seeing is links and rates get detected, packets appear to be getting transmitted (ethtool stats) but they aren't, and nothing is received on either end.

Is there something I'm missing with the device tree? Or is this the purpose of the PCS driver I'm looking into? I'm getting a feeling that my configuration is correct, and that I need to add SerDes support for these ports in phylink_mac_config... I noticed that there's the "SGMII only for now" comment, and when I look at the reference application for the 7512 there's a comment "external phy uses QSGMII interface" that appears to set the SGMII_MODE_ENA bit to 0.

Thank you as always for your time,

Colin Foster.

Vladimir Oltean

2022-01-06 10:48 UTC #1 lore raw

Hi Colin,

I'm sorry, but I have no direct experience with the SERDES and PCS of the Ocelot switches, since those blocks were completely replaced from the NXP instantiations I'm playing with.

Quoted 11 lines · Colin Foster

Hi Alexandre, Vladimir, and those with interest of Ocelot chips,
I'm starting this thread to just touch base before I dive into any PCS changes for Ocelot. I've appreciated all your guidance, and the only time I felt I've been led astray is when Alexandre told me it should be easy :-)
I'm at the point where I'm starting to integrate the additional 4 copper ports of the VSC7512 reference board. They are 4 ports connected through a QSGMII bus, to a VSC8514 phy.

Quoted 5 lines

The 8514 driver seems to be getting invoked, and running just fine. Also, I was able to slightly modify (hack*) drivers/phy/mscc/phy-ocelot-serdes.c to work with my in-development ocelot-mfd. I believe that is what I need to configure the HSIO registers.

And are you also getting a reference to it and calling it, like the ocelot driver does here?

	err = phy_set_mode_ext(serdes, PHY_MODE_ETHERNET, phy_mode);

Could we see that portion of the code in the felix driver?

Quoted 52 lines

I'm using these device tree settings:

    port@4 {
        reg = <4>;
        label = "swp4";
        status = "okay";
        phy-handle = <&sw_phy4>;
        phy-mode = "qsgmii";
        phys = <&serdes 4 SERDES6G(0)>;
    };
    port@5 {
        reg = <5>;
        label = "swp5";
        status = "okay";
        phy-handle = <&sw_phy5>;
        phy-mode = "qsgmii";
        phys = <&serdes 5 SERDES6G(0)>;
    };
...
    serdes: serdes {
        compatible = "mscc,vsc7514-serdes";
        #phy-cells = <2>;
    };
    mdio1: mdio1 {
        compatible = "mscc,ocelot-miim",
        pinctrl-names = "default";
        pinctrl-0 = <&miim1>;
        #address-cells = <1>;
        #size-cells = <0>;

        sw_phy4: ethernet-phy@4 {
            reg = <0x4>;
        };
    };

[    3.886787] libphy: ocelot_ext MDIO bus: probed
[    5.345891] ocelot-ext-switch ocelot-ext-switch: PHY [ocelot-ext-switch-mii:00] driver [Generic PHY] (irq=POLL)
[    5.357341] ocelot-ext-switch ocelot-ext-switch: configuring for phy/internal link mode
[    5.372525] ocelot-ext-switch ocelot-ext-switch swp1 (uninitialized): PHY [ocelot-ext-switch-mii:01] driver [Generic PHY] (irq=POLL)
[    5.388865] ocelot-ext-switch ocelot-ext-switch swp2 (uninitialized): PHY [ocelot-ext-switch-mii:02] driver [Generic PHY] (irq=POLL)
[    5.405086] ocelot-ext-switch ocelot-ext-switch swp3 (uninitialized): PHY [ocelot-ext-switch-mii:03] driver [Generic PHY] (irq=POLL)
[    6.291876] ocelot-ext-switch ocelot-ext-switch swp4 (uninitialized): PHY [ocelot-miim1-mii:04] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
[    6.471891] ocelot-ext-switch ocelot-ext-switch swp5 (uninitialized): PHY [ocelot-miim1-mii:05] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
[    6.651895] ocelot-ext-switch ocelot-ext-switch swp6 (uninitialized): PHY [ocelot-miim1-mii:06] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
[    6.831879] ocelot-ext-switch ocelot-ext-switch swp7 (uninitialized): PHY [ocelot-miim1-mii:07] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)

Quoted 5 lines

It seems like that, along with everything in vsc7514_phylink_mac_config, should be everything I need for operation of the four ports through the 8512. I've added OCELOT_QUIRK_SGMII_PORTS_MUST_BE_UP - but I'm not sure that's a quirk I need. Plus the only behavior it currently adds is once the port is up, it never comes back down.

Correct. I noticed that the code was structured to do this when I converted it to phylink, but I don't know if it's needed either. I just named that behavior as OCELOT_QUIRK_QSGMII_PORTS_MUST_BE_UP and made it optional.

Quoted 3 lines

The current behavior I'm seeing is links and rates get detected, packets appear to be getting transmitted (ethtool stats) but they aren't, and nothing is received on either end.

could you also check ethtool --phy-statistics swp4 on both ends?

Quoted 7 lines

Is there something I'm missing with the device tree? Or is this the purpose of the PCS driver I'm looking into? I'm getting a feeling that my configuration is correct, and that I need to add SerDes support for these ports in phylink_mac_config... I noticed that there's the "SGMII only for now" comment, and when I look at the reference application for the 7512 there's a comment "external phy uses QSGMII interface" that appears to set the SGMII_MODE_ENA bit to 0.

SGMII_MODE_ENA in PCS1G_MODE_CFG refers to the selection between Cisco SGMII/QSGMII vs IEEE 802.3 fiber modes. Different autonegotiation code words being transmitted. It's correct to leave it the way it is. It's curious that the reference application suggests to set it to 0 for QSGMII. I suppose that if in-band autoneg isn't enabled in PCS1G_ANEG_CFG, it doesn't really matter what kind of autoneg is used.

For SGMII vs QSGMII, the serdes_set_mode() function in phy-ocelot-serdes.c should set HSIO_HW_CFG_QSGMII_ENA where appropriate.

Quoted 3 lines

Thank you as always for your time,
Colin Foster.

What bootloader do you use? Do you have VSC8514 PHY initialization of any sort in the bootloader?

Colin Foster

author

2022-01-06 18:00 UTC #2 lore raw

Hi Vladimir,

Quoted 5 lines · Vladimir Oltean

Hi Colin,
I'm sorry, but I have no direct experience with the SERDES and PCS of the Ocelot switches, since those blocks were completely replaced from the NXP instantiations I'm playing with.

I realized that might be the case as I was writing this email. Thank you for still helping.

Quoted 19 lines · Colin Foster

Quoted 11 lines
Hi Alexandre, Vladimir, and those with interest of Ocelot chips,
I'm starting this thread to just touch base before I dive into any PCS changes for Ocelot. I've appreciated all your guidance, and the only time I felt I've been led astray is when Alexandre told me it should be easy :-)
I'm at the point where I'm starting to integrate the additional 4 copper ports of the VSC7512 reference board. They are 4 ports connected through a QSGMII bus, to a VSC8514 phy.
Quoted 5 lines
The 8514 driver seems to be getting invoked, and running just fine. Also, I was able to slightly modify (hack*) drivers/phy/mscc/phy-ocelot-serdes.c to work with my in-development ocelot-mfd. I believe that is what I need to configure the HSIO registers.

Quoted 6 lines

And are you also getting a reference to it and calling it, like the ocelot driver does here?
	err = phy_set_mode_ext(serdes, PHY_MODE_ETHERNET, phy_mode);
Could we see that portion of the code in the felix driver?

Oh, no I'm not. That's probably the hint that I need. I see that happens in VSC7514 probe before calling ocelot_port_init, unlike felix. Alright, there's some work for me to do! Thanks :)

Quoted 59 lines

Quoted 53 lines

I'm using these device tree settings:

    port@4 {
        reg = <4>;
        label = "swp4";
        status = "okay";
        phy-handle = <&sw_phy4>;
        phy-mode = "qsgmii";
        phys = <&serdes 4 SERDES6G(0)>;
    };
    port@5 {
        reg = <5>;
        label = "swp5";
        status = "okay";
        phy-handle = <&sw_phy5>;
        phy-mode = "qsgmii";
        phys = <&serdes 5 SERDES6G(0)>;
    };
...
    serdes: serdes {
        compatible = "mscc,vsc7514-serdes";
        #phy-cells = <2>;
    };
    mdio1: mdio1 {
        compatible = "mscc,ocelot-miim",
        pinctrl-names = "default";
        pinctrl-0 = <&miim1>;
        #address-cells = <1>;
        #size-cells = <0>;

        sw_phy4: ethernet-phy@4 {
            reg = <0x4>;
        };
    };

[    3.886787] libphy: ocelot_ext MDIO bus: probed
[    5.345891] ocelot-ext-switch ocelot-ext-switch: PHY [ocelot-ext-switch-mii:00] driver [Generic PHY] (irq=POLL)
[    5.357341] ocelot-ext-switch ocelot-ext-switch: configuring for phy/internal link mode
[    5.372525] ocelot-ext-switch ocelot-ext-switch swp1 (uninitialized): PHY [ocelot-ext-switch-mii:01] driver [Generic PHY] (irq=POLL)
[    5.388865] ocelot-ext-switch ocelot-ext-switch swp2 (uninitialized): PHY [ocelot-ext-switch-mii:02] driver [Generic PHY] (irq=POLL)
[    5.405086] ocelot-ext-switch ocelot-ext-switch swp3 (uninitialized): PHY [ocelot-ext-switch-mii:03] driver [Generic PHY] (irq=POLL)
[    6.291876] ocelot-ext-switch ocelot-ext-switch swp4 (uninitialized): PHY [ocelot-miim1-mii:04] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
[    6.471891] ocelot-ext-switch ocelot-ext-switch swp5 (uninitialized): PHY [ocelot-miim1-mii:05] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
[    6.651895] ocelot-ext-switch ocelot-ext-switch swp6 (uninitialized): PHY [ocelot-miim1-mii:06] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
[    6.831879] ocelot-ext-switch ocelot-ext-switch swp7 (uninitialized): PHY [ocelot-miim1-mii:07] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)

Quoted 5 lines

It seems like that, along with everything in vsc7514_phylink_mac_config, should be everything I need for operation of the four ports through the 8512. I've added OCELOT_QUIRK_SGMII_PORTS_MUST_BE_UP - but I'm not sure that's a quirk I need. Plus the only behavior it currently adds is once the port is up, it never comes back down.

Quoted 4 lines

Correct. I noticed that the code was structured to do this when I converted it to phylink, but I don't know if it's needed either. I just named that behavior as OCELOT_QUIRK_QSGMII_PORTS_MUST_BE_UP and made it optional.

I'll report my findings.

Quoted 6 lines

Quoted 4 lines
The current behavior I'm seeing is links and rates get detected, packets appear to be getting transmitted (ethtool stats) but they aren't, and nothing is received on either end.
could you also check ethtool --phy-statistics swp4 on both ends?

phy_cu_media_link_disconnect increments as expected. No activity on phy_cu_media_crc_good_count. I feel like it probably isn't worth digging too much into this until I fix the phy_set_mode_ext.

Quoted 16 lines

Quoted 8 lines
Is there something I'm missing with the device tree? Or is this the purpose of the PCS driver I'm looking into? I'm getting a feeling that my configuration is correct, and that I need to add SerDes support for these ports in phylink_mac_config... I noticed that there's the "SGMII only for now" comment, and when I look at the reference application for the 7512 there's a comment "external phy uses QSGMII interface" that appears to set the SGMII_MODE_ENA bit to 0.
SGMII_MODE_ENA in PCS1G_MODE_CFG refers to the selection between Cisco SGMII/QSGMII vs IEEE 802.3 fiber modes. Different autonegotiation code words being transmitted. It's correct to leave it the way it is. It's curious that the reference application suggests to set it to 0 for QSGMII. I suppose that if in-band autoneg isn't enabled in PCS1G_ANEG_CFG, it doesn't really matter what kind of autoneg is used.

Quoted 2 lines

For SGMII vs QSGMII, the serdes_set_mode() function in phy-ocelot-serdes.c should set HSIO_HW_CFG_QSGMII_ENA where appropriate.

I agree. I think you've answered my most important question, which is whether there's an obvious issue in my Device Tree / understanding. A blunder there can lead to significant loss of time making this work the wrong way. Shuffling datasheets, reference apps, register dumps...

Quoted 7 lines

Quoted 4 lines
Thank you as always for your time,
Colin Foster.
What bootloader do you use? Do you have VSC8514 PHY initialization of any sort in the bootloader?

U-Boot (now you know my secret next project...) that is basically the default configuration of BeagleBone. Definitely no VSC8514 knowledge by way of SPI > 7512 > MDIO(1) > 8514.