Patch: Device operative state notification against 2.5.7
Subsystems (2)
Stefan,
Quoted 10 lines · Stefan Rompf
Hi Jamal,
Quoted 4 lines
I think we should use SNMPs view of the world. RFC2863 still hasnt been obsoleted, AFAIK.
Look at sections 3.1.12-14. There is a correlation between admin/IFF_UP and oper/IFF_RUNNING.Agreed. If we want to be that specific, I'd prefer to go even further than the SNMP specification:
Review
Quoted 8 lines
(quoted diff) +0
-4-IFOP_UP Interface is operational up. -IFOP_TESTING Interface is just being tested -IFOP_UNKNOWN Driver is not capable of testing the interface status -IFOP_DORMANT Interface is sleeping, lower layer status may be unknown.
If i understand you correctly, this would still be something along the lines of dial on demand PPP. It stays "dormant" until it receives outgoing packet which initiates connection establishment activity. The type of activity may vary ...
Cant remember what diald does, but if i am not mistaken it actually kills the device and creates a new one everytime it detects activity
Quoted 3 lines
-IFOP_DOWN_NOTPRESENT
Interface does not exist. This can be f.e. an USB device that has just
been unplugged
What do you plan to do with this? eg, the name and stats for example wont change. What do the USB people think?
Quoted 5 lines
(quoted diff) +0
-2-IFOP_DOWN_LOWERLAYER Lower layer is down, f.e. ethernet interfaces below a VLAN -IFOP_DOWN_NOCARRIER Interface link beat/framing is missing (current netif_carrier_o* mapshere)
Good naming convention for the above two.
Quoted 2 lines
-IFOP_DOWN_KEEPALIVE
Another form of keepalive is missing
Not clear on this one.
Quoted 2 lines
-IFOP_DOWN_TX_TIMEOUT
Transmit timeout signalled by scheduler
I take it you want to use this for informational purposes i.e someone has to solicit for this information (Note, it is legal to have transmit path timeout but receive path just fine)
Quoted 2 lines
Multiple down reasons can be active for an interface.
this could be tricky mostly because some of the DOWN values are actually different states (in the state machine)
eg IFOP_DOWN_NOTPRESENT --> IFOP_DOWN_NOCARRIER when the USB hardware is inserted
Also the only time IFOP_DOWN_LOWERLAYER makes sense is when the top stacked device is actually admin up (but none of the lower devices are operationally UP)
Maybe you mean there should be something generally refered to as IFOP_DOWN and the reason would be one of IFOP_DOWN_* ?
Quoted 6 lines
These states can be represented by atomic data types in the netdevice structure so that they can safely be set and queried from everywhere within the kernel. If
IFOP_UP,IFOP_UNKNOWNorIFOP_DORMANTis set,IFF_RUNNINGwill be visible. If course it must be possible to get the detailed states from user mode and may be write toIFOP_DORMANT.
The only useful state transitions are to/from IFOP_DOWN_NOCARRIER in my opinion. What do you think?
In which case you only set IFF_RUNNING when you transition from IFOP_DOWN_NOCARRIER -> IFOP_UP and you clear it when IFOP_UP -> IFOP_DOWN_NOCARRIER
Things like IFOP_DORMANT can probably be set only by the driver and still retrieved via netlink or even ioctl when they are solicited for. By default you only announce via netlink transitions to/from IFOP_DOWN_NOCARRIER.
Quoted 5 lines
Userspace notification via netlink should remain optional. Does it make sense to queue up events or is it enough to report the last state after a bunch of changes has happened? For me, the second would be ok, especially as all drivers that can do link beat detection now utilise a timer for this purpose and will lose too rapid changes anyway.
I think the transitions to/from IFOP_DOWN_NOCARRIER would be useful for in kernel FIB or ARP updates as well; so maybe making these two optional would not be a wise decision. And if you send within the kernel then by default user space also gets it.
To avoid oscillations you only send the latest request _if it is still valid_, otherwise dont send any. Your code structuring should be built for this.
Also (this might be tricky to do), once you start flapping like that, it would also make sense to ratelimit (in exponential manner) the amount of netlink messages you send to announce carrier on/off.
One other thing you didnt talk about is the relation ship with IFF_UP; essentially, !IFF_UP would also mean !IFF_RUNNING and generally one of the DOWN states (NOCARRIER?).
when someone ifconfigs up, that should also invoke the device operational discovery activity (eg via device open()), so no need to worry there.
Apologies for making life difficult Stefan; we need to get it right and we have that opportunity ;->
cheers,
jamal
Quoted 2 lines · jamal
The only useful state transitions are to/from
IFOP_DOWN_NOCARRIERin my opinion. What do you think?
I meant this view from a "listener" perspective; example from an SNMP NMS perspective or even a dynamic route daemon, this is the only really interesting state transition. So IMO, it only makes sense to send netlink messages for these.
cheers,
jamal
-----BEGIN PGP SIGNED MESSAGE-----
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
"jamal" == jamal <[email protected]> writes:
-IFOP_DOWN_LOWERLAYER
Lower layer is down, f.e. ethernet interfaces below a VLAN -IFOP_DOWN_NOCARRIER
Interface link beat/framing is missing (current netif_carrier_o* maps here)
Are there docs on these interfaces somewhere?
jamal> Good naming convention for the above two.
>> -IFOP_DOWN_KEEPALIVE
>> Another form of keepalive is missing
jamal> Not clear on this one.
PPP, for instance, does keepalives.
An IPsec tunnel might also use such a mechanism (we have plans).
jamal> Also the only time IFOP_DOWN_LOWERLAYER makes sense is when the top jamal> stacked device is actually admin up (but none of the lower devices are jamal> operationally UP)
There are very good operational reasons why this is often done... It would be nice to see this kind of stuff in diagnostics.
] ON HUMILITY: to err is human. To moo, bovine. | firewalls [ ] Michael Richardson, Sandelman Software Works, Ottawa, ON |net architect[ ] [email protected] http://www.sandelman.ottawa.on.ca/ |device driver[
] panic("Just another NetBSD/notebook using, kernel hacking, security guy"); [
-----BEGIN PGP SIGNATURE-----
Version: 2.6.3ia
Charset: latin1
Comment: Finger me for keys
iQCVAwUBPLDD9YqHRg3pndX9AQGvhwP9HYlhJ1BJ5uJUDkRjcbLuaYLkwFrNWX+8 X4hvPb1mOgwQOPs3Fwz40Ng7FEHk4GECwSbRQlccEoSbPU1BLRmUESywN8xmAMyJ uGVqRt9UOzNzTtuvxOdpxk4LkNxZt06Flbs3DAvwv7G4d4W5Ciovu7I/JEjcnbCI qAcy0HuyDqI=
=EQ18
-----END PGP SIGNATURE-----
Quoted 12 lines · Michael Richardson
-----BEGIN PGP SIGNED MESSAGE-----
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
"jamal" == jamal <[email protected]> writes:
-IFOP_DOWN_LOWERLAYER
Lower layer is down, f.e. ethernet interfaces below a VLAN -IFOP_DOWN_NOCARRIER
Interface link beat/framing is missing (current netif_carrier_o* maps here)Are there docs on these interfaces somewhere?
I guess this discussion will result in some documentation...
Quoted 10 lines
jamal> Good naming convention for the above two.
>> -IFOP_DOWN_KEEPALIVE
>> Another form of keepalive is missingjamal> Not clear on this one.
PPP, for instance, does keepalives.
An IPsec tunnel might also use such a mechanism (we have plans).
Well, in that case the device is actually IFOP_UP. And the keepalives could be looked at as "carrier/link level diagnostics". Should the heartbeats disappear, it would be fair to transition the device to IFOP_DOWN_NOCARRIER. Should the device underneath have its carrier dissapear (eg a cable removed on an ethernet which ipsec0 uses), then the transition is very quickly to IFOP_DOWN_LOWERLAYER.... I guess the device(ipsec0) may also transition to IFOP_DOWN_NOCARRIER since its carrier path is doewn etc.
Quoted 6 lines
jamal> Also the only time
IFOP_DOWN_LOWERLAYERmakes sense is when the top jamal> stacked device is actually admin up (but none of the lower devices are jamal> operationally UP)There are very good operational reasons why this is often done... It would be nice to see this kind of stuff in diagnostics.
I think they should be accessible; whether you send netlink advertisements everytime there is some state transition is the question. In my opinion the only interesting things are the a transition from a case where a carrier (ethernet cable) is connected to where it is disconnected. The other direction is also important. For example, i think these are the only two route daemons need to know about.
cheers,
jamal
Quoted 7 lines
Quoted 2 lines
The only useful state transitions are to/from
IFOP_DOWN_NOCARRIERin my opinion. What do you think?I meant this view from a "listener" perspective; example from an SNMP NMS perspective or even a dynamic route daemon, this is the only really interesting state transition. So IMO, it only makes sense to send netlink messages for these.
But ... how about "carrier on"? What else would be able to trigger software to bring the interface up (so it could be routed or bridged) if there were no notification for "carrier on"? Not expecting some administrator to be initating it (or eyeballing some status display); one can ignore alerts, but one can't create them from nothing.
This is where my "asymmetry in design" alert triggers. It's usually been a good early warning system for trouble ... :)
- Dave
Hi Jamal,
Quoted 4 lines
[discussion whether
IFOP_DOWN_LOWERLAYERshould send netlink messages]I think they should be accessible; whether you send netlink advertisements everytime there is some state transition is the question.
they should. For a routing daemon it is interesting to be notified when interfaces becomes capable or incapable of handling packets, independant of the reason. As long as we abstract virtual or subinterfaces as normal interfaces to the upper layers (which is IMHO the correct decision), IFOP_DOWN_LOWERLAYER must trigger a notification.
Stefan
QoS _put, _get and _delete class ops semantic
In reply to jamalHi Jamal,
please can you help me with this ? During testing of speed improved HTB (as we talked about) I have problem to understand rationale behing Subj and can't find any docs.
put and get seem to be simply reference counting on class object. I'd expect thet class creation will call get once to assure that it is locked in active state and "tc class del" would call last put to release this last reference (and class will destroy itself). Unfortunately there is delete class op which seems to do almost the same as put.
Do you know rationale behind it ? Why we have both put and delete and what tc framework expect from qdisc ? I know there is some description tcio-current but it doesn't go into great depth on this topic.
Thanks in advance,
devik
Quoted 4 lines · Stefan Rompf
Hi Jamal,
Quoted 2 lines
[discussion whether
IFOP_DOWN_LOWERLAYERshould send netlink messages]
Quoted 8 lines
Quoted 2 lines
I think they should be accessible; whether you send netlink advertisements everytime there is some state transition is the question.
they should. For a routing daemon it is interesting to be notified when interfaces becomes capable or incapable of handling packets, independant of the reason. As long as we abstract virtual or subinterfaces as normal interfaces to the upper layers (which is IMHO the correct decision),
IFOP_DOWN_LOWERLAYERmust trigger a notification.
Well ... ok. Note, the user space program could maintain state and observe when all the interfaces are down to reach the same conclusion.
Ok, Stefan, i think you should be ready to roll some code now ;->
cheers,
jamal
Re: QoS _put, _get and _delete class ops semantic
In reply to Martin DeveraHi Martin,
Quoted 4 lines · Martin Devera
Hi Jamal,
put and get seem to be simply reference counting on class object.
nod
Quoted 4 lines
I'd expect thet class creation will call
get once to assure that it is locked in active state and "tc class del" would call last put to release this last reference (and class will destroy itself).
actually sequence would be get();delete();put() whenever
"tc class del" is invoked (theres other activity that may happen here as well instead of delete(), example attaching filters to class etc).
I think what you do with the delete+put combo is specific to your needs though. So for example you could return 0 in get() and your put() code will never be invoked.
Quoted 4 lines
Unfortunately there is delete class op which seems to do almost the same as put.
Do you know rationale behind it ? Why we have both put and delete and what tc framework expect from qdisc ?
Alexey can give a better answer.
Typically put() is the backup for delete. Should delete fail to destroy because of reference counts being non-zero, put will catch it. CBQ is an easier scheduler to see this. For a fun one look at the ATM scheduler.
cheers,
jamal
Re: QoS _put, _get and _delete class ops semantic
Quoted 6 lines
Quoted 4 lines
Unfortunately there is delete class op which seems to do almost the same as put.
Do you know rationale behind it ? Why we have both put and delete and what tc framework expect from qdisc ?Alexey can give a better answer.
probably he could step in here ;)
Quoted 2 lines
Typically
put()is the backup for delete. Should delete fail to destroy because of reference counts being non-zero, put will catch it.
So that if I understand it correctly: delete can only assure that class is "invisible" from now to subsequent gets/walks and other uses and leave actual destroy to the last put. Do you think that it is reasonable ?
Quoted 2 lines
CBQ is an easier scheduler to see this. For a fun one look at the ATM scheduler.
I looked at them but I hate to blindly copy-n-paste code without in-depth knowledge of interface's expectation. Especially when yesterday (after year of flawless operation) someone made my code oops during class deletion under high load ;-\
thanks, Martin
Re: QoS _put, _get and _delete class ops semantic
Quoted 8 lines · Martin Devera
Quoted 2 lines
Typically
put()is the backup for delete. Should delete fail to destroy because of reference counts being non-zero, put will catch it.So that if I understand it correctly: delete can only assure that class is "invisible" from now to subsequent gets/walks and other uses and leave actual destroy to the last put. Do you think that it is reasonable ?
sort of. If you look at the ATM scheduler, you should see the refcount maybe incremented twice. In this case, delete will fail to destroy the class but put() will catch it.
If you only increment/decrement then delete should always catch it. In which case you dont need to destroy in put(). Your mileage may vary.
cheers,
jamal
Hi Jamal,
Quoted 1 line
Ok, Stefan, i think you should be ready to roll some code now ;->
indeed. It will take a couple of days as I'm not motivated to sit another four hours before the computer every evening ;-)
Cheers, Stefan