tg3 issues
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): [email protected]
CCing: netdev
Quoted 87 lines · patric
Hi,
To start with, i'm not sure if this should go to the dev or user list, but i'll start here..
I'm currently running a nfsroot via a Broadcom NetXtreme 1000-SX card (BCM5701) and i have a big problem with the tg3 drivers autonegotiation.
The issue seems to be that when the kernel comes so far as it's trying to mount the boot the autonegotiation has not yet completed and then causes a panic since it cannot mount the nfsroot.
From some debugging i have done here the issues seems to be related to
the flowcontrol configuration, and just to make it a bit more fun it does work some of the time.. (around once every 5-10 attempts.)On the console it looks something like this when failing. (written from memory since i don't have netconsole enabled)
tg3: eth0: Link is up at 1000 Mbps, full duplex.
tg3: eth0: Flow control is off for TX and off for RX.
IP-Config: Complete:device=eth0, addr=192.168.1.10, mask=255.255.255.0,gw=255.255.255.255,
host=amd, domain=, nis-domain=(none), bootserver=255.255.255.255, rootserver=192.168.1.1, rootpath=
Root-NFS: unknown option: nolocks
Looking up port of RPC 100003/3 on 192.168.1.1
rpcbind: server 192.168.1.1 not responding, timed outRoot-NFS: Unable to get nfsd port number from server, using default
Looking up port of RPC 100003/3 on 192.168.1.1
rpcbind: server 192.168.1.1 not responding, timed outRoot-NFS: Unable to get nfsd port number from server, using default
and so on until it panics...
And for a successful attempt:
tg3: eth0: Link is up at 1000 Mbps, full duplex.
tg3: eth0: Flow control is off for TX and off for RX.
IP-Config: Complete:device=eth0, addr=192.168.1.10, mask=255.255.255.0,gw=255.255.255.255,
host=amd, domain=, nis-domain=(none), bootserver=255.255.255.255, rootserver=192.168.1.1, rootpath=
Root-NFS: unknown option: nolocks
Looking up port of RPC 100003/3 on 192.168.1.1
tg3: eth0: Link is up at 1000 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.
rpcbind: server 192.168.1.1 not responding, timed out
Root-NFS: Unable to get nfsd port number from server, using default Looking up port of RPC 100005/3 on 192.168.1.1
VFS: Mounted root (nfs filesystem).Also, if i get the "flowcontrol is on" before it tries to mount the root it does not report any issues. And this is also not unique to NFS booting, but anytime i bring the link up (when not using it for booting).
Currently i'm running 2.6.22-git9 (for testing) and i experience the same issues in plain 2.6.22 and before.
Regards,
Patric
-
To unsubscribe from this list: send the line "unsubscribe linux-net" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 1 line
play the game
Quoted 47 lines · pradeep singh
CCing: netdev
Quoted 44 lines · patric
Hi,
To start with, i'm not sure if this should go to the dev or user list, but i'll start here..
I'm currently running a nfsroot via a Broadcom NetXtreme 1000-SX card (BCM5701) and i have a big problem with the tg3 drivers autonegotiation.
The issue seems to be that when the kernel comes so far as it's trying to mount the boot the autonegotiation has not yet completed and then causes a panic since it cannot mount the nfsroot.
From some debugging i have done here the issues seems to be related to the flowcontrol configuration, and just to make it a bit more fun it does work some of the time.. (around once every 5-10 attempts.)
On the console it looks something like this when failing. (written from memory since i don't have netconsole enabled)
tg3: eth0: Link is up at 1000 Mbps, full duplex.
tg3: eth0: Flow control is off for TX and off for RX.
IP-Config: Complete:device=eth0, addr=192.168.1.10, mask=255.255.255.0,gw=255.255.255.255,
host=amd, domain=, nis-domain=(none), bootserver=255.255.255.255, rootserver=192.168.1.1, rootpath=
Root-NFS: unknown option: nolocks
Looking up port of RPC 100003/3 on 192.168.1.1
rpcbind: server 192.168.1.1 not responding, timed outRoot-NFS: Unable to get nfsd port number from server, using default
Looking up port of RPC 100003/3 on 192.168.1.1
rpcbind: server 192.168.1.1 not responding, timed outRoot-NFS: Unable to get nfsd port number from server, using default
and so on until it panics...
IIRC, there are two main problems in this typ of situation
1) Spanning tree convergence
2) Firmware initalization latency
If you are running spanning tree on your network, it can take up to 2 minutes before your port will forward frames properly. if you have the options available, disable spanning tree on the switch port connected to your host system, or at least enable portfast if it is an option. That should fix any spanning tree issues you have
If the tg3 card is just taking a long time to initalize, there is not too much you can do about it. If your goal is to use nfs root, I would, instead of enabling nfs-root as a kernel config option, I would create an initramfs that:
A) Brings up your NIC
B) Mounts your nfs partition
C) executes a switch_root or pivot_root operation
That way you can calibrate a delay between steps (A) and (B) in your initramfs init script
Regards
Neil
Signature · 8 lines
/*************************************************** *Neil Horman *Software Engineer *Red Hat, Inc. *[email protected] *gpg keyid: 1024D / 0x92A74FA1 *http://pgp.mit.edu ***************************************************/
Quoted 70 lines · Neil Horman
Quoted 49 lines · pradeep singh
CCing: netdev
Quoted 45 lines · patric
Hi,
To start with, i'm not sure if this should go to the dev or user list, but i'll start here..
I'm currently running a nfsroot via a Broadcom NetXtreme 1000-SX card (BCM5701) and i have a big problem with the tg3 drivers autonegotiation.
The issue seems to be that when the kernel comes so far as it's trying to mount the boot the autonegotiation has not yet completed and then causes a panic since it cannot mount the nfsroot.
From some debugging i have done here the issues seems to be related to the flowcontrol configuration, and just to make it a bit more fun it does work some of the time.. (around once every 5-10 attempts.)
On the console it looks something like this when failing. (written from memory since i don't have netconsole enabled)
tg3: eth0: Link is up at 1000 Mbps, full duplex.
tg3: eth0: Flow control is off for TX and off for RX.
IP-Config: Complete:device=eth0, addr=192.168.1.10, mask=255.255.255.0,gw=255.255.255.255,
host=amd, domain=, nis-domain=(none), bootserver=255.255.255.255, rootserver=192.168.1.1, rootpath=
Root-NFS: unknown option: nolocks
Looking up port of RPC 100003/3 on 192.168.1.1
rpcbind: server 192.168.1.1 not responding, timed outRoot-NFS: Unable to get nfsd port number from server, using default
Looking up port of RPC 100003/3 on 192.168.1.1
rpcbind: server 192.168.1.1 not responding, timed outRoot-NFS: Unable to get nfsd port number from server, using default
and so on until it panics...
IIRC, there are two main problems in this typ of situation
1) Spanning tree convergence
2) Firmware initalization latencyIf you are running spanning tree on your network, it can take up to 2 minutes before your port will forward frames properly. if you have the options available, disable spanning tree on the switch port connected to your host system, or at least enable portfast if it is an option. That should fix any spanning tree issues you have
If the tg3 card is just taking a long time to initalize, there is not too much you can do about it. If your goal is to use nfs root, I would, instead of enabling nfs-root as a kernel config option, I would create an initramfs that:
A) Brings up your NIC
B) Mounts your nfs partition
C) executes aswitch_rootorpivot_rootoperation
Quoted 7 lines
That way you can calibrate a delay between steps (A) and (B) in your initramfs init script
Regards
Neil
Hi Neil and thanks for your quick reply, and thanks Pradeep for forwarding the question to the correct mailinglist.
Well, not using any switches and just a crossed cable between the machines. Did notice that it seems to get a 'good link' more often when cold-booting the client.
Have been thinking about using a initrd to get around the issue, but the problem is that you never know how long the init will be so there will always have to be a quite big delay before the system can boot. But don't really think the issue is that the card takes a long time to initialize since it does sometime work without delay during a warm-boot and the cards do report that they are up but they then are reporting different states of flow-control. Maybe set the flowcontrol static in the driver for a test, if i now can figure out how this driver works. :)
Just a hypothetical question. If the 2 network cards starts the autonegotiation would it be possible that they get into a loop where they are chasing each others state? Maybe a fix could be to add a sleep of a random length that would enable them to catch up? Maybe you know if any of the fiber-cards so support running without flowcontrol too since the cards don't seem to be able to get a link with flowcontrol turned off at least in this setup.
Regards,
Patric
Quoted 10 lines · patric
Just a hypothetical question. If the 2 network cards starts the autonegotiation would it be possible that they get into a loop where they are chasing each others state? Maybe a fix could be to add a sleep of a random length that would enable them to catch up? Maybe you know if any of the fiber-cards so support running without flowcontrol too since the cards don't seem to be able to get a link with flowcontrol turned off at least in this setup.
The old 5701 fiber NICs do not support autonegotiation in hardware so it is done "by hand" in the driver. It is not the most robust way of doing autoneg and what you described is totally possible. You might want to try disabling autoneg to see if it works any better. There is only one possible speed in fiber and autoneg is really only used to negotiate flow control. Some switch ports will not link up if the link partner does not do autoneg though.
You have to use ethtool in initrd to turn off autoneg or just modify the driver to disable autoneg.
Quoted 16 lines · Michael Chan
Quoted 4 lines · patric
Just a hypothetical question. If the 2 network cards starts the autonegotiation would it be possible that they get into a loop where they are chasing each others state? Maybe a fix could be to add a sleep of a random length that would enable them to catch up? Maybe you know if any of the fiber-cards so support running without flowcontrol too since the cards don't seem to be able to get a link with flowcontrol turned off at least in this setup.
The old 5701 fiber NICs do not support autonegotiation in hardware so it is done "by hand" in the driver. It is not the most robust way of doing autoneg and what you described is totally possible. You might want to try disabling autoneg to see if it works any better. There is only one possible speed in fiber and autoneg is really only used to negotiate flow control. Some switch ports will not link up if the link partner does not do autoneg though.
Quoted 5 lines
You have to use ethtool in initrd to turn off autoneg or just modify the driver to disable autoneg.
Thanks... That's a confirmation on what i suspected.. I think i'll dig into the code instead and try to figure out some way of making it work a bit better for my setup atleast... I'll post a patch later if i get something working... Btw, do you have any timings on how long it takes for the cards to get a lock on the link?
Thanks for your input!
/Patric
Quoted 6 lines · patric
Thanks... That's a confirmation on what i suspected.. I think i'll dig into the code instead and try to figure out some way of making it work a bit better for my setup atleast...
I'll post a patch later if i get something working... Btw, do you have any timings on how long it takes for the cards to get a lock on the link?
Serdes link comes up very fast. The clause 37 autoneg involves going through a few states and exchanging base pages and optional next pages. Each state completes in 10 msec, so the whole autoneg should complete in roughly 100 msec or so. But because it is done in software for these older cards, it can take much longer because it can get out-of-sync.
Quoted 14 lines · Michael Chan
Quoted 4 lines · patric
Thanks... That's a confirmation on what i suspected.. I think i'll dig into the code instead and try to figure out some way of making it work a bit better for my setup atleast... I'll post a patch later if i get something working... Btw, do you have any timings on how long it takes for the cards to get a lock on the link?
Serdes link comes up very fast. The clause 37 autoneg involves going through a few states and exchanging base pages and optional next pages. Each state completes in 10 msec, so the whole autoneg should complete in roughly 100 msec or so. But because it is done in software for these older cards, it can take much longer because it can get out-of-sync.
Quoted 6 lines
-
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi,
Think i got something working for me at least, and the fix is quite minimal and only downside that i could see from it was that you might get a small delay when bringing up the interface, but that's probably better than getting a non-functional interface that reports that it's up.
The fix seems to be quite simple with just a random sleep at the end of "tg3_setup_fiber_by_hand():"
tw32_f(MAC_MODE, tp->mac_mode);
udelay(40);
}out:
udelay( net_random() % 400 );
return current_link_up;
}Not sure that this is a good fix or if it might break on other systems, but maybe you could have a quick look at that?
Regards,
Patric
Quoted 17 lines · patric
Hi,
Think i got something working for me at least, and the fix is quite minimal and only downside that i could see from it was that you might get a small delay when bringing up the interface, but that's probably better than getting a non-functional interface that reports that it's up.
The fix seems to be quite simple with just a random sleep at the end of "
tg3_setup_fiber_by_hand():"tw32_f(MAC_MODE, tp->mac_mode); udelay(40); }out:
udelay( net_random() % 400 ); return current_link_up; }Not sure that this is a good fix or if it might break on other systems, but maybe you could have a quick look at that?
Quoted 8 lines
Regards,
Patric-
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Last update on this.
"udelay( 100 + net_random % 300 )" seems to work much better and i have not had a single problem getting the link up within 10 seconds of a cold or warm-boot, and most often the link comes up directly without any sort of delay instead like before when it could hang for 30 seconds before getting a link, if you even got a link.
/Patric
Quoted 32 lines · patric
Quoted 22 lines · patric
Hi,
Think i got something working for me at least, and the fix is quite minimal and only downside that i could see from it was that you might get a small delay when bringing up the interface, but that's probably better than getting a non-functional interface that reports that it's up.
The fix seems to be quite simple with just a random sleep at the end of "
tg3_setup_fiber_by_hand():"tw32_f(MAC_MODE, tp->mac_mode); udelay(40); }out:
udelay( net_random() % 400 ); return current_link_up; }Not sure that this is a good fix or if it might break on other systems, but maybe you could have a quick look at that?
Quoted 8 lines
Regards,
Patric-
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines
Last update on this.
"
udelay( 100 + net_random % 300 )" seems to work much better and i have not had a single problem getting the link up within 10 seconds of a cold or warm-boot, and most often the link comes up directly without any sort of delay instead like before when it could hang for 30 seconds before getting a link, if you even got a link.
We'll have to do some testing to see if we can find a better solution. Adding up to 400 usec of busy wait is not ideal. Are you connecting two 5701 fiber cards directly to each other in your setup?
Quoted 12 lines · Michael Chan
Quoted 5 lines
Last update on this.
"
udelay( 100 + net_random % 300 )" seems to work much better and i have not had a single problem getting the link up within 10 seconds of a cold or warm-boot, and most often the link comes up directly without any sort of delay instead like before when it could hang for 30 seconds before getting a link, if you even got a link.We'll have to do some testing to see if we can find a better solution. Adding up to 400 usec of busy wait is not ideal. Are you connecting two 5701 fiber cards directly to each other in your setup?
Hi,
Yea, it's and ugly hack, but it's a workaround that at least works for me.
Have done some additional tests and to me it seems that the driver just needs to wait a bit longer to detect the link + some random time to get around the issues it might have when chasing the other systems state.
Don't have the numbers in front of me, but i did some tests to get the 'optimum' delay to wait and it seems like the larger the wait time (even up to around 40-50ms!) causes the autoneg to go much smoother and faster. And remember that the link can be reported up, but no traffic can be passed via the link before the autoneg is complete and both cards reports back that flowcontrol is enabled so you need to verify the link by trying to send some data and not just look at the link-status.
Hope my conclusions might help, or atleast point you in the right direction.
I have 2
01:08.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15) ( 14e4:1645 )
connected via a single FC cable.
And the two systems are one single-core and one dual-core AMD system.
Regards,
Patric