RFC: ethernet links should remember routes the same as addresses
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 7 lines · Christopher Friesen
Suppose I have a fancy routing setup, dynamically configured by different binaries, scripts, etc, complete with multiple addresses per link, additional routing rules and tables specified using iproute2, etc.
An ethernet driver hangs. Could be a software bug, an intermittent hardware issue, whatever. It can be fixed up by setting the link down and up.
Quoted 6 lines
Currently, if I run "ip link set dev ethX down", all routes associated with that IP address in the additional routing tables are lost. This is somewhat understandable, as the addresses are not actually available anymore. However, the addresses are still visible associated with the link. Then I run "ip link set dev ethX up". The route in the main routing table comes back, but none of the other routes do. Somehow, all of those additional routes must be re-added.
ip route list dev device > BACKUP
...
while read i ; do ip route add $i ; done < BACKUP
Quoted 7 lines
Wouldn't it be nice if we could keep track of these additional routes? Then you could simply 'down' and 'up' the link and everything would be back the way it was before.
Does this sound like a good idea? How hard would this be to implement (not knowing what the current code looks like, I don't know how this would be done)?
In kernel very easy. The IP addresses and the driver init/cleanup are completely separated and can be easily done independent. You may need some way to prevent packets getting submitted to the driver (e.g. a netif_carrier_off but make sure to not confuse it with a real netif_carrier_off done by the driver, so you'll likely need a new flag)
I'm not sure it it worth it though given how easily it is to simulate in user space. If you really wanted it I guess best would be to add new ioctls for it. Coding should be easy.
-Andi
Quoted 9 lines · Andi Kleen
Quoted 6 lines · Christopher Friesen
Currently, if I run "ip link set dev ethX down", all routes associated with that IP address in the additional routing tables are lost. This is somewhat understandable, as the addresses are not actually available anymore. However, the addresses are still visible associated with the link. Then I run "ip link set dev ethX up". The route in the main routing table comes back, but none of the other routes do. Somehow, all of those additional routes must be re-added.
Quoted 5 lines
ip route list dev device > BACKUP
...
while read i ; do ip route add $i ; done < BACKUP
Unfortunately, this seems to only list the routes in the main routing table, and these routes are recreated automatically when I bring the link back up.
The problem is that I have routes in multiple other routing tables, and they don't show up in this command. I assume I'd have to run a similar command for each routing table, which is kind of a pain.
If the driver re-init is totally separate from the routing code, is there any real reason why shutting down the driver *should* remove all routes to that device? Maybe the simplest solution would be a new ioctl that would be a link *reset*...just down/up the link without affecting anything else....
Signature · 4 lines
Chris Friesen | MailStop: 043/33/F10 Nortel Networks | work: (613) 765-0557 3500 Carling Avenue | fax: (613) 765-2986 Nepean, ON K2H 8E9 Canada | email: [email protected]
Quoted 4 lines · Christopher Friesen
If the driver re-init is totally separate from the routing code, is there any real reason why shutting down the driver *should* remove all routes to that device? Maybe the simplest solution would be a new ioctl that would be a link *reset*...just down/up the link without affecting anything else....
What would want want the down/up to do? The reason I ask is that there is an MII-DIAG option to reset the transceiver, if that's all you want to do.
If you want to remove/re-install the driver, then you have to clean up all the links pointing to it (ie the routes associated with the device), or you will have all kinds of nasty dangling pointers (logical or otherwise) to clean up...
Ben
Quoted 3 lines
Signature · 3 lines
Ben Greear <[email protected]> <Ben_Greear AT excite.com> President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear
Quoted 8 lines · Ben Greear
Quoted 4 lines · Christopher Friesen
If the driver re-init is totally separate from the routing code, is there any real reason why shutting down the driver *should* remove all routes to that device? Maybe the simplest solution would be a new ioctl that would be a link *reset*...just down/up the link without affecting anything else....
Quoted 7 lines
What would want want the down/up to do? The reason I ask is that there is an MII-DIAG option to reset the transceiver, if that's all you want to do.
If you want to remove/re-install the driver, then you have to clean up all the links pointing to it (ie the routes associated with the device), or you will have all kinds of nasty dangling pointers (logical or otherwise) to clean up...
We've got what looks like a bug in the tulip driver (or maybe buggy hardware) that under certain circumstances can block with what looks like a problem in the ring buffer. Someone is looking at the driver, but in the meantime the only way to fix it is to do "ip link set dev ethX down" followed by "ip link set dev ethX up". Whatever code path this causes in the driver is enough to fix the issue. I assume its due to the re-initialization of the buffer descriptors.
On some consideration, maybe it would make more sense in this case to add something to the driver itself to re-initialize itself rather than involving the rest of the kernel. However, I do think that taking down a link and bringing it back up should be inverse procedures. If there were routes associated with a link before taking it down, then they should be associated with it after bringing it back up, whether or not they stick around when the link is actually down. This is how addresses behave, so I think routes should be similar.
Chris
Signature · 4 lines
Chris Friesen | MailStop: 043/33/F10 Nortel Networks | work: (613) 765-0557 3500 Carling Avenue | fax: (613) 765-2986 Nepean, ON K2H 8E9 Canada | email: [email protected]
Quoted 2 lines · Christopher Friesen
Unfortunately, this seems to only list the routes in the main routing table, and these routes are recreated automatically when I bring the link back up.
Well, it is not that hard to extend it.
Just iterate over ip rule ls output too.
(left as an exercise for the reader)
Quoted 4 lines
If the driver re-init is totally separate from the routing code, is there any real reason why shutting down the driver *should* remove all routes to that device? Maybe the simplest solution would be a new ioctl that would be a link *reset*...just down/up the link without affecting anything else....
If you just want to do a link reset you can use ethertool today with most drivers. What is different is a full driver reset to work around software/ hardware bugs. That would need to be a new ioctl.
-Andi