Skip to contentSkip to conversation

Patch: Device operative state notification against 2.5.7

DISCUSSION jamal 5 participants 12 messages Apr 7, 2002 – Apr 12, 2002 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore

jamal

author
#1 lore raw
In reply to a message not in the archive

Stefan,

Quoted 10 lines · Stefan Rompf

Hi Jamal,

Quoted 4 lines

I think we should use SNMPs view of the world. RFC2863 still hasnt been obsoleted, AFAIK.
Look at sections 3.1.12-14. There is a correlation between admin/IFF_UP and oper/IFF_RUNNING.

Agreed. If we want to be that specific, I'd prefer to go even further than the SNMP specification:

Review

Quoted 8 lines
(quoted diff) +0 -4
-IFOP_UP
 Interface is operational up.
-IFOP_TESTING
 Interface is just being tested
-IFOP_UNKNOWN
 Driver is not capable of testing the interface status
-IFOP_DORMANT
 Interface is sleeping, lower layer status may be unknown.

If i understand you correctly, this would still be something along the lines of dial on demand PPP. It stays "dormant" until it receives outgoing packet which initiates connection establishment activity. The type of activity may vary ...
Cant remember what diald does, but if i am not mistaken it actually kills the device and creates a new one everytime it detects activity

Quoted 3 lines

-IFOP_DOWN_NOTPRESENT
Interface does not exist. This can be f.e. an USB device that has just
been unplugged

What do you plan to do with this? eg, the name and stats for example wont change. What do the USB people think?

Quoted 5 lines
(quoted diff) +0 -2
-IFOP_DOWN_LOWERLAYER
 Lower layer is down, f.e. ethernet interfaces below a VLAN
-IFOP_DOWN_NOCARRIER
 Interface link beat/framing is missing (current netif_carrier_o* maps

here)

Good naming convention for the above two.

Quoted 2 lines

-IFOP_DOWN_KEEPALIVE
Another form of keepalive is missing

Not clear on this one.

Quoted 2 lines

-IFOP_DOWN_TX_TIMEOUT
Transmit timeout signalled by scheduler

I take it you want to use this for informational purposes i.e someone has to solicit for this information (Note, it is legal to have transmit path timeout but receive path just fine)

Quoted 2 lines

Multiple down reasons can be active for an interface.

this could be tricky mostly because some of the DOWN values are actually different states (in the state machine)
eg IFOP_DOWN_NOTPRESENT --> IFOP_DOWN_NOCARRIER when the USB hardware is inserted
Also the only time IFOP_DOWN_LOWERLAYER makes sense is when the top stacked device is actually admin up (but none of the lower devices are operationally UP)

Maybe you mean there should be something generally refered to as IFOP_DOWN and the reason would be one of IFOP_DOWN_* ?

Quoted 6 lines

These states can be represented by atomic data types in the netdevice structure so that they can safely be set and queried from everywhere within the kernel. If IFOP_UP, IFOP_UNKNOWN or IFOP_DORMANT is set, IFF_RUNNING will be visible. If course it must be possible to get the detailed states from user mode and may be write to IFOP_DORMANT.

The only useful state transitions are to/from IFOP_DOWN_NOCARRIER in my opinion. What do you think?
In which case you only set IFF_RUNNING when you transition from IFOP_DOWN_NOCARRIER -> IFOP_UP and you clear it when IFOP_UP -> IFOP_DOWN_NOCARRIER
Things like IFOP_DORMANT can probably be set only by the driver and still retrieved via netlink or even ioctl when they are solicited for. By default you only announce via netlink transitions to/from IFOP_DOWN_NOCARRIER.

Quoted 5 lines

Userspace notification via netlink should remain optional. Does it make sense to queue up events or is it enough to report the last state after a bunch of changes has happened? For me, the second would be ok, especially as all drivers that can do link beat detection now utilise a timer for this purpose and will lose too rapid changes anyway.

I think the transitions to/from IFOP_DOWN_NOCARRIER would be useful for in kernel FIB or ARP updates as well; so maybe making these two optional would not be a wise decision. And if you send within the kernel then by default user space also gets it.

To avoid oscillations you only send the latest request _if it is still valid_, otherwise dont send any. Your code structuring should be built for this.
Also (this might be tricky to do), once you start flapping like that, it would also make sense to ratelimit (in exponential manner) the amount of netlink messages you send to announce carrier on/off.

One other thing you didnt talk about is the relation ship with IFF_UP; essentially, !IFF_UP would also mean !IFF_RUNNING and generally one of the DOWN states (NOCARRIER?).
when someone ifconfigs up, that should also invoke the device operational discovery activity (eg via device open()), so no need to worry there.

Apologies for making life difficult Stefan; we need to get it right and we have that opportunity ;->

cheers,
jamal

author
#2 lore raw
Quoted 2 lines · jamal

The only useful state transitions are to/from IFOP_DOWN_NOCARRIER in my opinion. What do you think?

I meant this view from a "listener" perspective; example from an SNMP NMS perspective or even a dynamic route daemon, this is the only really interesting state transition. So IMO, it only makes sense to send netlink messages for these.

cheers,
jamal

#4 lore raw
In reply to jamal

-----BEGIN PGP SIGNED MESSAGE-----

Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

"jamal" == jamal <[email protected]> writes:

-IFOP_DOWN_LOWERLAYER
Lower layer is down, f.e. ethernet interfaces below a VLAN -IFOP_DOWN_NOCARRIER
Interface link beat/framing is missing (current netif_carrier_o* maps here)

Are there docs on these interfaces somewhere?

jamal> Good naming convention for the above two.

>> -IFOP_DOWN_KEEPALIVE
>> Another form of keepalive is missing

jamal> Not clear on this one.

PPP, for instance, does keepalives.
An IPsec tunnel might also use such a mechanism (we have plans).

jamal> Also the only time IFOP_DOWN_LOWERLAYER makes sense is when the top jamal> stacked device is actually admin up (but none of the lower devices are jamal> operationally UP)

There are very good operational reasons why this is often done... It would be nice to see this kind of stuff in diagnostics.

]       ON HUMILITY: to err is human. To moo, bovine.           |  firewalls  [
]   Michael Richardson, Sandelman Software Works, Ottawa, ON    |net architect[
] [email protected] http://www.sandelman.ottawa.on.ca/ |device driver[

] panic("Just another NetBSD/notebook using, kernel hacking, security guy"); [

-----BEGIN PGP SIGNATURE-----
Version: 2.6.3ia
Charset: latin1
Comment: Finger me for keys

iQCVAwUBPLDD9YqHRg3pndX9AQGvhwP9HYlhJ1BJ5uJUDkRjcbLuaYLkwFrNWX+8 X4hvPb1mOgwQOPs3Fwz40Ng7FEHk4GECwSbRQlccEoSbPU1BLRmUESywN8xmAMyJ uGVqRt9UOzNzTtuvxOdpxk4LkNxZt06Flbs3DAvwv7G4d4W5Ciovu7I/JEjcnbCI qAcy0HuyDqI=
=EQ18
-----END PGP SIGNATURE-----

author
#5 lore raw
Quoted 12 lines · Michael Richardson

-----BEGIN PGP SIGNED MESSAGE-----

Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

"jamal" == jamal <[email protected]> writes:

-IFOP_DOWN_LOWERLAYER
Lower layer is down, f.e. ethernet interfaces below a VLAN -IFOP_DOWN_NOCARRIER
Interface link beat/framing is missing (current netif_carrier_o* maps here)

Are there docs on these interfaces somewhere?

I guess this discussion will result in some documentation...

Quoted 10 lines

jamal> Good naming convention for the above two.

>> -IFOP_DOWN_KEEPALIVE
>> Another form of keepalive is missing

jamal> Not clear on this one.

PPP, for instance, does keepalives.
An IPsec tunnel might also use such a mechanism (we have plans).

Well, in that case the device is actually IFOP_UP. And the keepalives could be looked at as "carrier/link level diagnostics". Should the heartbeats disappear, it would be fair to transition the device to IFOP_DOWN_NOCARRIER. Should the device underneath have its carrier dissapear (eg a cable removed on an ethernet which ipsec0 uses), then the transition is very quickly to IFOP_DOWN_LOWERLAYER.... I guess the device(ipsec0) may also transition to IFOP_DOWN_NOCARRIER since its carrier path is doewn etc.

Quoted 6 lines

jamal> Also the only time IFOP_DOWN_LOWERLAYER makes sense is when the top jamal> stacked device is actually admin up (but none of the lower devices are jamal> operationally UP)

There are very good operational reasons why this is often done... It would be nice to see this kind of stuff in diagnostics.

I think they should be accessible; whether you send netlink advertisements everytime there is some state transition is the question. In my opinion the only interesting things are the a transition from a case where a carrier (ethernet cable) is connected to where it is disconnected. The other direction is also important. For example, i think these are the only two route daemons need to know about.

cheers,
jamal

#3 lore raw
In reply to jamal
Quoted 7 lines
Quoted 2 lines

The only useful state transitions are to/from IFOP_DOWN_NOCARRIER in my opinion. What do you think?

I meant this view from a "listener" perspective; example from an SNMP NMS perspective or even a dynamic route daemon, this is the only really interesting state transition. So IMO, it only makes sense to send netlink messages for these.

But ... how about "carrier on"? What else would be able to trigger software to bring the interface up (so it could be routed or bridged) if there were no notification for "carrier on"? Not expecting some administrator to be initating it (or eyeballing some status display); one can ignore alerts, but one can't create them from nothing.

This is where my "asymmetry in design" alert triggers. It's usually been a good early warning system for trouble ... :)

- Dave

#6 lore raw
In reply to jamal

Hi Jamal,

Quoted 4 lines

[discussion whether IFOP_DOWN_LOWERLAYER should send netlink messages]

I think they should be accessible; whether you send netlink advertisements everytime there is some state transition is the question.

they should. For a routing daemon it is interesting to be notified when interfaces becomes capable or incapable of handling packets, independant of the reason. As long as we abstract virtual or subinterfaces as normal interfaces to the upper layers (which is IMHO the correct decision), IFOP_DOWN_LOWERLAYER must trigger a notification.

Stefan

#9 lore raw

QoS _put, _get and _delete class ops semantic

In reply to jamal

Hi Jamal,

please can you help me with this ? During testing of speed improved HTB (as we talked about) I have problem to understand rationale behing Subj and can't find any docs.
put and get seem to be simply reference counting on class object. I'd expect thet class creation will call get once to assure that it is locked in active state and "tc class del" would call last put to release this last reference (and class will destroy itself). Unfortunately there is delete class op which seems to do almost the same as put.
Do you know rationale behind it ? Why we have both put and delete and what tc framework expect from qdisc ? I know there is some description tcio-current but it doesn't go into great depth on this topic.

Thanks in advance,
devik

author
#7 lore raw
In reply to Stefan Rompf
Quoted 4 lines · Stefan Rompf

Hi Jamal,

Quoted 2 lines

[discussion whether IFOP_DOWN_LOWERLAYER should send netlink messages]

Quoted 8 lines
Quoted 2 lines

I think they should be accessible; whether you send netlink advertisements everytime there is some state transition is the question.

they should. For a routing daemon it is interesting to be notified when interfaces becomes capable or incapable of handling packets, independant of the reason. As long as we abstract virtual or subinterfaces as normal interfaces to the upper layers (which is IMHO the correct decision), IFOP_DOWN_LOWERLAYER must trigger a notification.

Well ... ok. Note, the user space program could maintain state and observe when all the interfaces are down to reach the same conclusion.

Ok, Stefan, i think you should be ready to roll some code now ;->

cheers,
jamal

author
#10 lore raw

Re: QoS _put, _get and _delete class ops semantic

In reply to Martin Devera

Hi Martin,

Quoted 4 lines · Martin Devera

Hi Jamal,

put and get seem to be simply reference counting on class object.

nod

Quoted 4 lines

I'd expect thet class creation will call
get once to assure that it is locked in active state and "tc class del" would call last put to release this last reference (and class will destroy itself).

actually sequence would be get();delete();put() whenever
"tc class del" is invoked (theres other activity that may happen here as well instead of delete(), example attaching filters to class etc).

I think what you do with the delete+put combo is specific to your needs though. So for example you could return 0 in get() and your put() code will never be invoked.

Quoted 4 lines

Unfortunately there is delete class op which seems to do almost the same as put.
Do you know rationale behind it ? Why we have both put and delete and what tc framework expect from qdisc ?

Alexey can give a better answer.
Typically put() is the backup for delete. Should delete fail to destroy because of reference counts being non-zero, put will catch it. CBQ is an easier scheduler to see this. For a fun one look at the ATM scheduler.

cheers,
jamal

#11 lore raw

Re: QoS _put, _get and _delete class ops semantic

Quoted 6 lines
Quoted 4 lines

Unfortunately there is delete class op which seems to do almost the same as put.
Do you know rationale behind it ? Why we have both put and delete and what tc framework expect from qdisc ?

Alexey can give a better answer.

probably he could step in here ;)

Quoted 2 lines

Typically put() is the backup for delete. Should delete fail to destroy because of reference counts being non-zero, put will catch it.

So that if I understand it correctly: delete can only assure that class is "invisible" from now to subsequent gets/walks and other uses and leave actual destroy to the last put. Do you think that it is reasonable ?

Quoted 2 lines

CBQ is an easier scheduler to see this. For a fun one look at the ATM scheduler.

I looked at them but I hate to blindly copy-n-paste code without in-depth knowledge of interface's expectation. Especially when yesterday (after year of flawless operation) someone made my code oops during class deletion under high load ;-\

thanks, Martin

author
#12 lore raw

Re: QoS _put, _get and _delete class ops semantic

Quoted 8 lines · Martin Devera
Quoted 2 lines

Typically put() is the backup for delete. Should delete fail to destroy because of reference counts being non-zero, put will catch it.

So that if I understand it correctly: delete can only assure that class is "invisible" from now to subsequent gets/walks and other uses and leave actual destroy to the last put. Do you think that it is reasonable ?

sort of. If you look at the ATM scheduler, you should see the refcount maybe incremented twice. In this case, delete will fail to destroy the class but put() will catch it.
If you only increment/decrement then delete should always catch it. In which case you dont need to destroy in put(). Your mileage may vary.

cheers,
jamal

#8 lore raw
In reply to jamal

Hi Jamal,

Quoted 1 line

Ok, Stefan, i think you should be ready to roll some code now ;->

indeed. It will take a couple of days as I'm not motivated to sit another four hours before the computer every evening ;-)

Cheers, Stefan