Skip to contentSkip to conversation

RFC: ethernet links should remember routes the same as addresses

DISCUSSION Andi Kleen 3 participants 5 messages Nov 29, 2001 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Andi Kleen

author
#1 lore raw
Quoted 7 lines · Christopher Friesen

Suppose I have a fancy routing setup, dynamically configured by different binaries, scripts, etc, complete with multiple addresses per link, additional routing rules and tables specified using iproute2, etc.

An ethernet driver hangs. Could be a software bug, an intermittent hardware issue, whatever. It can be fixed up by setting the link down and up.

Quoted 6 lines

Currently, if I run "ip link set dev ethX down", all routes associated with that IP address in the additional routing tables are lost. This is somewhat understandable, as the addresses are not actually available anymore. However, the addresses are still visible associated with the link. Then I run "ip link set dev ethX up". The route in the main routing table comes back, but none of the other routes do. Somehow, all of those additional routes must be re-added.

ip route list dev device > BACKUP

...

while read i ; do ip route add $i ; done < BACKUP

Quoted 7 lines

Wouldn't it be nice if we could keep track of these additional routes? Then you could simply 'down' and 'up' the link and everything would be back the way it was before.

Does this sound like a good idea? How hard would this be to implement (not knowing what the current code looks like, I don't know how this would be done)?

In kernel very easy. The IP addresses and the driver init/cleanup are completely separated and can be easily done independent. You may need some way to prevent packets getting submitted to the driver (e.g. a netif_carrier_off but make sure to not confuse it with a real netif_carrier_off done by the driver, so you'll likely need a new flag)

I'm not sure it it worth it though given how easily it is to simulate in user space. If you really wanted it I guess best would be to add new ioctls for it. Coding should be easy.

-Andi

#2 lore raw
In reply to a message not in the archive
Quoted 9 lines · Andi Kleen
Quoted 6 lines · Christopher Friesen

Currently, if I run "ip link set dev ethX down", all routes associated with that IP address in the additional routing tables are lost. This is somewhat understandable, as the addresses are not actually available anymore. However, the addresses are still visible associated with the link. Then I run "ip link set dev ethX up". The route in the main routing table comes back, but none of the other routes do. Somehow, all of those additional routes must be re-added.

Quoted 5 lines

ip route list dev device > BACKUP

...

while read i ; do ip route add $i ; done < BACKUP

Unfortunately, this seems to only list the routes in the main routing table, and these routes are recreated automatically when I bring the link back up.

The problem is that I have routes in multiple other routing tables, and they don't show up in this command. I assume I'd have to run a similar command for each routing table, which is kind of a pain.

If the driver re-init is totally separate from the routing code, is there any real reason why shutting down the driver *should* remove all routes to that device? Maybe the simplest solution would be a new ioctl that would be a link *reset*...just down/up the link without affecting anything else....

Signature · 4 lines
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]
#3 lore raw
In reply to a message not in the archive
Quoted 4 lines · Christopher Friesen

If the driver re-init is totally separate from the routing code, is there any real reason why shutting down the driver *should* remove all routes to that device? Maybe the simplest solution would be a new ioctl that would be a link *reset*...just down/up the link without affecting anything else....

What would want want the down/up to do? The reason I ask is that there is an MII-DIAG option to reset the transceiver, if that's all you want to do.

If you want to remove/re-install the driver, then you have to clean up all the links pointing to it (ie the routes associated with the device), or you will have all kinds of nasty dangling pointers (logical or otherwise) to clean up...

Ben

Quoted 3 lines
Signature · 3 lines
Ben Greear <[email protected]>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear
#4 lore raw
In reply to a message not in the archive
Quoted 8 lines · Ben Greear
Quoted 4 lines · Christopher Friesen

If the driver re-init is totally separate from the routing code, is there any real reason why shutting down the driver *should* remove all routes to that device? Maybe the simplest solution would be a new ioctl that would be a link *reset*...just down/up the link without affecting anything else....

Quoted 7 lines

What would want want the down/up to do? The reason I ask is that there is an MII-DIAG option to reset the transceiver, if that's all you want to do.

If you want to remove/re-install the driver, then you have to clean up all the links pointing to it (ie the routes associated with the device), or you will have all kinds of nasty dangling pointers (logical or otherwise) to clean up...

We've got what looks like a bug in the tulip driver (or maybe buggy hardware) that under certain circumstances can block with what looks like a problem in the ring buffer. Someone is looking at the driver, but in the meantime the only way to fix it is to do "ip link set dev ethX down" followed by "ip link set dev ethX up". Whatever code path this causes in the driver is enough to fix the issue. I assume its due to the re-initialization of the buffer descriptors.

On some consideration, maybe it would make more sense in this case to add something to the driver itself to re-initialize itself rather than involving the rest of the kernel. However, I do think that taking down a link and bringing it back up should be inverse procedures. If there were routes associated with a link before taking it down, then they should be associated with it after bringing it back up, whether or not they stick around when the link is actually down. This is how addresses behave, so I think routes should be similar.

Chris

Signature · 4 lines
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]
author
#5 lore raw
In reply to a message not in the archive
Quoted 2 lines · Christopher Friesen

Unfortunately, this seems to only list the routes in the main routing table, and these routes are recreated automatically when I bring the link back up.

Well, it is not that hard to extend it.

Just iterate over ip rule ls output too.

(left as an exercise for the reader)

Quoted 4 lines

If the driver re-init is totally separate from the routing code, is there any real reason why shutting down the driver *should* remove all routes to that device? Maybe the simplest solution would be a new ioctl that would be a link *reset*...just down/up the link without affecting anything else....

If you just want to do a link reset you can use ethertool today with most drivers. What is different is a full driver reset to work around software/ hardware bugs. That would need to be a new ioctl.

-Andi