Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS,UNPARSEABLE_RELAY,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E439FC282C3 for ; Tue, 22 Jan 2019 21:39:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A070821726 for ; Tue, 22 Jan 2019 21:39:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726418AbfAVVjC (ORCPT ); Tue, 22 Jan 2019 16:39:02 -0500 Received: from belmont79srvr.owm.bell.net ([184.150.200.79]:58901 "EHLO mtlfep01.bell.net" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1725919AbfAVVjC (ORCPT ); Tue, 22 Jan 2019 16:39:02 -0500 Received: from bell.net mtlfep01 184.150.200.30 by mtlfep01.bell.net with ESMTP id <20190122213859.NJT28651.mtlfep01.bell.net@mtlspm01.bell.net> for ; Tue, 22 Jan 2019 16:38:59 -0500 Received: from [192.168.0.110] (really [64.231.92.241]) by mtlspm01.bell.net with ESMTP id <20190122213859.ZZHT17436.mtlspm01.bell.net@[192.168.0.110]>; Tue, 22 Jan 2019 16:38:59 -0500 Subject: Re: net: phylink: dsa: mv88e6xxx: flaky link detection on switch ports with internal PHYs To: Andrew Lunn Cc: Russell King , Vivien Didelot , Florian Fainelli , netdev@vger.kernel.org References: <49eec816-9238-c893-0860-602aa8965515@bell.net> <20190122202834.GB12052@lunn.ch> From: John David Anglin Message-ID: <09fce2e0-9dd2-ea30-7e7d-72f04eede68b@bell.net> Date: Tue, 22 Jan 2019 16:40:27 -0500 User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <20190122202834.GB12052@lunn.ch> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Cloudmark-Analysis: v=2.2 cv=QLaHfkDL c=1 sm=0 tr=0 a=jJmVOGxBrKxWvcjqaG+oVg==:17 a=IkcTkHD0fZMA:10 a=3JhidrIBZZsA:10 a=FBHGMhGWAAAA:8 a=gDQ0Tgp5ROyDzbRovbwA:9 a=bS5xHyseSSeTjlRM:21 a=QEXdDO2ut3YA:10 a=9gvnlMMaQFpL9xblJ6ne:22 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Hi Andrew, On 1/22/2019 3:28 PM, Andrew Lunn wrote: > Does it make a difference if you do it by hand? Bring up the master > interface, wan, lan0, lan1, add any bridge you need, etc. > >> From power on, none of the wan, lan0, lan1 or br0 achieve link >> (LOWER_UP). I can explore this but I don't know at the moment.  I believe that all interfaces are configured. In serdes.c, interrupts are used to monitor link changes.  However, phy.c doesn't do this and it doesn't call phylink_mac_change(). > Is the master interface up? What does ip link show give? The master interface (eth0 on mvneta) always comes up.  The link status of this interface appears to be checked by polling at 1 second intervals. After a power on boot, this is what I see: root@espressobin:~# ip link show 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: bond0: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000     link/ether 96:80:ba:01:e9:31 brd ff:ff:ff:ff:ff:ff 3: dummy0: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000     link/ether 46:42:f8:2b:d1:4e brd ff:ff:ff:ff:ff:ff 4: eth0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1024     link/ether 3e:c0:66:0a:db:ba brd ff:ff:ff:ff:ff:ff 5: wan@eth0: mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN mode DEFAULT group default qlen 1000     link/ether 3e:c0:66:0a:db:ba brd ff:ff:ff:ff:ff:ff 6: lan0@eth0: mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN mode DEFAULT group default qlen 1000     link/ether 3e:c0:66:0a:db:ba brd ff:ff:ff:ff:ff:ff 7: lan1@eth0: mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN mode DEFAULT group default qlen 1000     link/ether 3e:c0:66:0a:db:ba brd ff:ff:ff:ff:ff:ff 8: br0: mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000     link/ether 00:50:43:25:fb:84 brd ff:ff:ff:ff:ff:ff root@espressobin:~# networkctl IDX LINK             TYPE               OPERATIONAL SETUP   1 lo               loopback           carrier     unmanaged   2 bond0            bond               off         unmanaged   3 dummy0           ether              off         unmanaged   4 eth0             ether              degraded    configured   5 wan              dsa                no-carrier  configuring   6 lan0             dsa                no-carrier  configuring   7 lan1             dsa                no-carrier  configuring   8 br0              bridge             no-carrier  configuring 8 links listed. r0: flags=4099  mtu 1500         ether 00:50:43:25:fb:84  txqueuelen 1000  (Ethernet)         RX packets 0  bytes 0 (0.0 B)         RX errors 0  dropped 0  overruns 0  frame 0         TX packets 0  bytes 0 (0.0 B)         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 eth0: flags=4163  mtu 1500         inet6 fe80::3cc0:66ff:fe0a:dbba  prefixlen 64  scopeid 0x20         ether 3e:c0:66:0a:db:ba  txqueuelen 1024  (Ethernet)         RX packets 2229  bytes 160191 (156.4 KiB)         RX errors 0  dropped 0  overruns 0  frame 0         TX packets 14  bytes 1116 (1.0 KiB)         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0         device interrupt 11 lan0: flags=4099  mtu 1500         ether 3e:c0:66:0a:db:ba  txqueuelen 1000  (Ethernet)         RX packets 1075  bytes 55969 (54.6 KiB)         RX errors 0  dropped 551  overruns 0  frame 0         TX packets 0  bytes 0 (0.0 B)         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 lan1: flags=4099  mtu 1500         ether 3e:c0:66:0a:db:ba  txqueuelen 1000  (Ethernet)         RX packets 0  bytes 0 (0.0 B)         RX errors 0  dropped 0  overruns 0  frame 0         TX packets 0  bytes 0 (0.0 B)         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 lo: flags=73  mtu 65536         inet 127.0.0.1  netmask 255.0.0.0         inet6 ::1  prefixlen 128  scopeid 0x10         loop  txqueuelen 1000  (Local Loopback)         RX packets 0  bytes 0 (0.0 B)         RX errors 0  dropped 0  overruns 0  frame 0         TX packets 0  bytes 0 (0.0 B)         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 wan: flags=4099  mtu 1500         ether 3e:c0:66:0a:db:ba  txqueuelen 1000  (Ethernet)         RX packets 1154  bytes 55184 (53.8 KiB)         RX errors 0  dropped 14  overruns 0  frame 0         TX packets 0  bytes 0 (0.0 B)         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 Then, if I do a "shutdown -r +0", I see: root@espressobin:~# ip link show 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: bond0: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000     link/ether 26:dd:a4:7c:26:3f brd ff:ff:ff:ff:ff:ff 3: dummy0: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000     link/ether 0a:73:61:5a:ae:dc brd ff:ff:ff:ff:ff:ff 4: eth0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1024     link/ether f6:2d:a2:ab:32:4d brd ff:ff:ff:ff:ff:ff 5: wan@eth0: mtu 1500 qdisc noqueue master br0 state UP mode DEFAULT group default qlen 1000     link/ether f6:2d:a2:ab:32:4d brd ff:ff:ff:ff:ff:ff 6: lan0@eth0: mtu 1500 qdisc noqueue master br0 state UP mode DEFAULT group default qlen 1000     link/ether f6:2d:a2:ab:32:4d brd ff:ff:ff:ff:ff:ff 7: lan1@eth0: mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN mode DEFAULT group default qlen 1000     link/ether f6:2d:a2:ab:32:4d brd ff:ff:ff:ff:ff:ff 8: br0: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000     link/ether 00:50:43:25:fb:84 brd ff:ff:ff:ff:ff:ff root@espressobin:~# networkctl IDX LINK             TYPE               OPERATIONAL SETUP   1 lo               loopback           carrier     unmanaged   2 bond0            bond               off         unmanaged   3 dummy0           ether              off         unmanaged   4 eth0             ether              degraded    configured   5 wan              dsa                degraded    configured   6 lan0             dsa                degraded    configured   7 lan1             dsa                no-carrier  configuring   8 br0              bridge             routable    configured 8 links listed. root@espressobin:~# ifconfig br0: flags=4163  mtu 1500         inet 192.168.0.55  netmask 255.255.255.0  broadcast 192.168.0.255         inet6 fe80::250:43ff:fe25:fb84  prefixlen 64  scopeid 0x20         ether 00:50:43:25:fb:84  txqueuelen 1000  (Ethernet)         RX packets 1184  bytes 237936 (232.3 KiB)         RX errors 0  dropped 261  overruns 0  frame 0         TX packets 193  bytes 19444 (18.9 KiB)         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 eth0: flags=4163  mtu 1500         inet6 fe80::f42d:a2ff:feab:324d  prefixlen 64  scopeid 0x20         ether f6:2d:a2:ab:32:4d  txqueuelen 1024  (Ethernet)         RX packets 1331  bytes 276209 (269.7 KiB)         RX errors 0  dropped 0  overruns 0  frame 0         TX packets 233  bytes 24352 (23.7 KiB)         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0         device interrupt 11 lan0: flags=4163  mtu 1500         inet6 fe80::f42d:a2ff:feab:324d  prefixlen 64  scopeid 0x20         ether f6:2d:a2:ab:32:4d  txqueuelen 1000  (Ethernet)         RX packets 264  bytes 13773 (13.4 KiB)         RX errors 0  dropped 135  overruns 0  frame 0         TX packets 18  bytes 1368 (1.3 KiB)         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 lan1: flags=4099  mtu 1500         ether f6:2d:a2:ab:32:4d  txqueuelen 1000  (Ethernet)         RX packets 0  bytes 0 (0.0 B)         RX errors 0  dropped 0  overruns 0  frame 0         TX packets 0  bytes 0 (0.0 B)         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 lo: flags=73  mtu 65536         inet 127.0.0.1  netmask 255.0.0.0         inet6 ::1  prefixlen 128  scopeid 0x10         loop  txqueuelen 1000  (Local Loopback)         RX packets 0  bytes 0 (0.0 B)         RX errors 0  dropped 0  overruns 0  frame 0         TX packets 0  bytes 0 (0.0 B)         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 wan: flags=4163  mtu 1500         inet6 fe80::f42d:a2ff:feab:324d  prefixlen 64  scopeid 0x20         ether f6:2d:a2:ab:32:4d  txqueuelen 1000  (Ethernet)         RX packets 1067  bytes 233154 (227.6 KiB)         RX errors 0  dropped 4  overruns 0  frame 0         TX packets 203  bytes 20240 (19.7 KiB)         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 Everything seems normal.  The wan and lan0 ports are connected. From syslog for p2 (lan0): Jan 22 16:02:34 localhost kernel: [    5.249693] mv88e6085 d0032004.mdio-mii:01: p2: PortState set to Disabled Jan 22 16:02:34 localhost kernel: [    5.394717] mv88e6085 d0032004.mdio-mii:01: p2: Force link down Jan 22 16:02:34 localhost kernel: [    5.400554] mv88e6085 d0032004.mdio-mii:01: p2: Speed unforced Jan 22 16:02:34 localhost kernel: [    5.406558] mv88e6085 d0032004.mdio-mii:01: p2: Unforce half duplex Jan 22 16:02:34 localhost kernel: [    5.413007] mv88e6085 d0032004.mdio-mii:01: p2: Unforce link down Jan 22 16:02:34 localhost kernel: [    5.421850] mv88e6085 d0032004.mdio-mii:01: p2: 802.1QMode set to Disabled Jan 22 16:02:34 localhost kernel: [    5.430197] mv88e6085 d0032004.mdio-mii:01: p2: FID set to 0 Jan 22 16:02:34 localhost kernel: [    5.433696] mv88e6085 d0032004.mdio-mii:01: p2: VLANTable set to 001 Jan 22 16:02:34 localhost kernel: [    6.136188] mv88e6085 d0032004.mdio-mii:01 lan0 (uninitialized): PHY [!soc!internal-regs@d0000000!mdio@32004!switch0@1!mdio:12] driver [Marvell 88E6390] Jan 22 16:02:34 localhost kernel: [    8.890947] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this. Jan 22 16:02:34 localhost systemd-networkd[358]: lan0: Joined netdev Jan 22 16:02:34 localhost systemd-networkd[358]: lan0: Bringing link up Jan 22 16:02:34 localhost systemd-networkd[358]: lan0: Flags change: +UP Jan 22 16:02:34 localhost systemd-networkd[358]: lan0: Set link Jan 22 16:02:34 localhost kernel: [    9.017442] br0: port 2(lan0) entered blocking state Jan 22 16:02:34 localhost kernel: [    9.017478] br0: port 2(lan0) entered disabled state Jan 22 16:02:34 localhost kernel: [    9.018314] mv88e6085 d0032004.mdio-mii:01: p2: VLANTable set to 009 Jan 22 16:02:34 localhost kernel: [    9.021257] device lan0 entered promiscuous mode Jan 22 16:02:34 localhost kernel: [    9.033658] mv88e6085 d0032004.mdio-mii:01: p2: VLANTable set to 00b an 22 16:02:34 localhost kernel: [    9.138112] mv88e6085 d0032004.mdio-mii:01: p2: PortState set to Blocking/Listening Jan 22 16:02:34 localhost kernel: [    9.138149] mv88e6085 d0032004.mdio-mii:01 lan0: configuring for phy/ link mode Jan 22 16:02:34 localhost kernel: [    9.138158] mv88e6085 d0032004.mdio-mii:01: port=2, mode=0, link=0 Jan 22 16:02:34 localhost kernel: [    9.138267] mv88e6085 d0032004.mdio-mii:01: port=2, mode=0, link=0 Jan 22 16:02:34 localhost kernel: [    9.138274] mv88e6085 d0032004.mdio-mii:01 lan0: Link is Down - mac_link_dropped=0 Jan 22 16:02:34 localhost kernel: [    9.138885] IPv6: ADDRCONF(NETDEV_UP): lan0: link is not ready Jan 22 16:02:34 localhost kernel: [    9.213888] mv88e6085 d0032004.mdio-mii:01: port=2, mode=0, link=1 Jan 22 16:02:34 localhost kernel: [    9.213919] mv88e6085 d0032004.mdio-mii:01 lan0: Link is Up - 1Gbps/Full - flow control rx/tx Jan 22 16:02:34 localhost kernel: [    9.213962] IPv6: ADDRCONF(NETDEV_CHANGE): lan0: link becomes ready Jan 22 16:02:34 localhost kernel: [   11.356713] br0: port 2(lan0) entered blocking state Jan 22 16:02:34 localhost kernel: [   11.356719] br0: port 2(lan0) entered forwarding state Jan 22 16:02:34 localhost kernel: [   11.359228] mv88e6085 d0032004.mdio-mii:01: p2: PortState set to Blocking/Listening Jan 22 16:02:34 localhost kernel: [   11.360339] mv88e6085 d0032004.mdio-mii:01: p2: PortState set to Forwarding Jan 22 16:02:34 localhost systemd-networkd[358]: lan0: Flags change: +LOWER_UP +RUNNING Jan 22 16:02:34 localhost systemd-networkd[358]: lan0: Gained carrier Jan 22 16:02:34 localhost systemd-networkd[358]: lan0: Configured -- John David Anglin dave.anglin@bell.net