support for Cobalt Networks (x86 only) systems (for real this time)

Nothing good, probably. Same story as /proc/apm, which only hits BIOS instead (and it's debateable what is better).

Perhaps it's mode 600 in their distro...

Yep... I meant to check for it but forgot.

-- Pete

Hmm, the MII related ioctl's in some net drivers (checked for 3c59x, tulip, eepro100) are also querying the hardware. And a user is allowed to ask for this info (but not able to modify it).

Sincerely,

Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: [email protected]

And a malicious user calling those at a high rate is bound to get ugly.

In both of these situations, calling the ioctls without priveleges is quite useful, so maybe rate-limiting for ioctls and proc files like this would be a good idea in general.

Jeff

Jeff Garzik      | Disbelief, that's why you fail.
Building 1024    |
MandrakeSoft     |

Many of them (like the MII and APM ones) the result can be cached

Only some of them can be cached... (some of the MIIs in some drivers
are already cached, in fact) you can't cache stuff like what your link
partner is advertising at the moment, or what your battery status is at the moment.

Jeff Garzik      | Disbelief, that's why you fail.
Building 1024    |
MandrakeSoft     |

I am sure that to an unpriviledged application reporting back the same result as we saw last time we asked the hardware unless it is over 30 seconds old will work fine. Maybe 10 for link partner ?

No way! If I implement a HA application which depends on link status, I want the info to be accurate, I don't want to know that 30 seconds ago I had good link.

IMHO, rate limiting is the only solution.

Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: [email protected]

To tangent a little bit, and add netdev to the CC...

The loss and regain of link status should be proactively signalled to userspace using netlink or something similar. Currently we have netif_carrier_{on,off,ok} but it is only passively checked. netif_carrier_{on,off} should probably schedule_task() to fire off a netlink message...

For your HA application specifically, right now, I would suggest making sure your net driver calls netif_carrier_xxx correctly, then checking for IFF_RUNNING interface flag. IFF_RUNNING will disappear if the interface is up, but there is no carrier [as according to netif_carrier_ok].

Jeff Garzik      | Disbelief, that's why you fail.
Building 1024    |
MandrakeSoft     |

Don't such HA apps need to run as root anyways?

Regardless, I agree that, long term, the way to do this is via netlink.

Later,
David S. Miller
[email protected]

Not necessarily, but eventually you can let root (CAP_NET_ADMIN, anyway) go through without any limitations, root can bring down the system at will in other ways.

In addition, the rate limiting solution allows a warning to be issued when the limit is exceeded, so that the poor sysadmin knows what hit him 8-)

Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: [email protected]

[ For the general discussion ]
I fully agree, but I just wanted to give an example of legit use from user space of _current_ values from hardware.

[ Link status details ]
Just that not all NICs have hardware support (and/or not all drivers use these facilities) for link status change notification using interrupts. Right now, most drivers _poll_ for media status and based on the poll rate, netif_carrier routines are (or should be) called. We can't make the poll rate very small for the general case, as MII access is time consuming (same discussion was some months ago when the bonding driver was updated). However, for users who know that they need this info to be more accurate (at the expense of CPU time), polling through ioctl's is the only solution.

[ Back to general discussion ]
So far, to the problem of too often access to hardware, 2 solutions were proposed:
1. cache the values. You can then let the user shoot him-/her-self in the foot by making too many ioctl calls. But this prevent any legit use of current hardware state.
2. rate limiting. You don't let the user access the hardware too often (to be defined), so he/she can't shoot his-/her-self in the foot. Legit use of current hardware state is possible.

IMHO, solution 2 is much better. Can you find situations when it's not ?

Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: [email protected]

Jeff, Thanks for copying netdev. Wish more people would do that.

Not really.

One idea i have been toying with is to maintain hysteris or threshold of some form in dev_watchdog;
example: if watchdog timer expires threshold times, you declare the link dead and send netif_carrier_off netlink message.
On recovery, you send netif_carrier_on

Assumption:
If the tx path is blocked, more than likely the link is down.

cheers,
jamal

Shame on me, I should have thought of that too... I joined lkml only about 2 weeks ago because netdev related topics are sometimes discussed only there...

AFAIK, dev_watchdog is right now used only for Tx (if I'm wrong, please correct me!). So how do you sense link loss if you expect only high Rx traffic ?

I assume that you mean "on recovery" as in "first succesful hard_start_xmit".

Yes, but is this a good approximation ? I'm not saying that it's not, I'm merely asking for counter-arguments.

Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: [email protected]

Good question. Makes me think. Thoughts further below.

right.

It is an indirect approximation. Note that if the system data is very asymetrical as in the case you pointed out, notification will take a long long time. You need a plan B. Still, the tx watchdogs are a good source of fault detection in the case of non-availabilty of MII detection and even with the presence of MII.

I hate making this more complex than it should be:

Since we already have a messaging system within the kernel and user<->kernel space aka "netlink" -- one could easily add a protocol in user space which "dynamically heartbeats" the devices. Control should come from user space; it would be a great idea to avoid ioctls.

"Dynamic" in the above sense means trying to totaly avoid making it a synchronous poll. The poll rate is a function of how many packets go out that device per average measurement time. Basically, the period that the user space app dumps "hello" netlink packets to the kernel is a variable.

cheers,
jamal

Agreed. But my question was a bit different: is there any legit situation where Tx timeouts can happen in a row _without_ having a link loss ? In this situation, we'd have false positives...

Sounds nice, but could this be implemented light enough ?

Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: [email protected]

yes

Jeff Garzik      | Echelon words of the day, from The Register:
Building 1024    | FRU Lebed HALO Spetznaz Al Amn al-Askari Glock 26 
MandrakeSoft     | Steak Knife Kill the President anarchy echelon
                 | nuclear assassinate Roswell Waco World Trade Center

Two places:
1) no MII indicators
2) shaky hardware and MII bounces. Is it on, is it off? What is going on? You could use them to "probe" to make sure that infact the MII indicators are not false positives.

Your mileage may vary.

Not as simple as synchronous polls.
Note, however, simple/light does not imply the best.

cheers,
jamal

Please re-read your comment. Then think about it. Then tell me how rate limiting differs from caching to the application.

Alan

[ OK, this time I cc'ed netdev 8-) ]

For caching, the kernel establishes the rate with which the info is updated. There's nothing wrong, but how is the application to know if the value is actual or cached (from when, until when) ? That means that a single application that needs data more often than the caching rate will get bogus data and not know about it.

With rate limiting, you always get new values, unless the limit is exceeded. When the limit is exceeded, you log and:
- block any request until some timer is expired. The application can detect that it's been blocked and react. You can detect if there are several calls waiting and return the same value to all.
- return error until some timer is expired. The application can again detect that.
In both cases, the application is also capable of guessing the value of the delay.

For one application which follows the rules (doesn't need data more often than the caching rate or doesn't exceed the rate limit) there is no difference, I agree. But when somebody is playing tricks while you need data, you have the chance of detecting this by using rate limits.

And yes, I agree that either of them (cache or rate limit) should be modifiable through proc entry/ioctl/whatever.

Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: [email protected]

Alan Cox <[email protected]> [01/06/01 10:32]:

I'd argue for rate limiting as the application only gets back new data, never a cached value n times in a row.

With caching, you'd have to let the application know when the cached value was last read and how long it will be cached for. With rate limiting, you'd just block the app and get the new data to the app once the timer has expired. Or, if the app doesn't read the data until after the timer has expired, it will not block at all.

Seems to me that you'd have less redundant application processing with rate limiting.

Which is worse. I cat the proc file a few times and your HA app is unlucky. It now gets *NO* data for five minutes. If we cache the values it gets approximate data

Rate limit: current values, sysadmin knows when something's wrong or
application knows when something's wrong (depending if you block or not when the limit is exceeded).
Caching: approximate values, sysadmin is not notified (unless something
like ioctl/second is available somewhere), application is not notified.

[ Whoa, I can't type at the speed of this thread 8-) ]

Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: [email protected]

Alan Cox <[email protected]> [01/06/01 10:39]:

Is it not possible to have several readers waiting for the same data and return it to all of them when the timer expires?

No the application gets back no data, ever, because a third party application keeps beating it. You don't even need maliciousness for this, synchronization effects and locking on the file will ensure it gets you in the end

fstat() mtime. That seems easy enough

Sure, but as I already wrote, you can detect that something is wrong. Then shoot the person!

This only answered the first part of the question: when. How do you pass the "how long" info ?
Does the same applies for the MII ioctl case ?

Now let's talk about implementation issues. For the MII case, you have several registers that have to be read. The way it generally done is for the ioctl to pass MII address and register number and receive back the value.

Caching means that the driver (I don't think that it can be done at higher levels) has to keep track of accesses to all MII interfaces (yes, there can be more than one on a NIC) and all of their registers. One solution is to read all registers at once and start the cache timer for each MII register access. Another solution is to have each register start its own cache timer.

OTOH, ioctl rate limiting can be done at higher level and you need only one timer per netdevice. So, it's done once and all net drivers benefit from it.

Guess which one I prefer...

Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: [email protected]

How does that solve the problem ?

The mtime tells you exactly that.

I disagree. A non priviledged app should not be able to poke around in MII registers anyway. So you only have to cache the generic state of the link.

You don't need timers.

You don't need any timers if you are caching. Zilch nada none. You know the last time a query came in. The mtime lets the app know the last time the value was modified.

Alan

[ As this is becoming more and more MII specific, I changed the subject ]

Alan, please consider this situation:

One application needs to poll link status with 1 second resolution. On a system where caching is done with an unknown cache expiring time, this application is sometimes fed incorrect data. So, you need a way to tell for how long this situation lasts. If you have a proc/ioctl interface for setting cache expiring time, this same interface can then be used for reading back this info. This application can then check that this value is lower than 1 second and if not, notify the user that it cannot run.

As this thread started as a general hardware access problem, would only _one_ value for all these cases be sufficient ? Or each case should have its own timeout ? Anyway, for MII, accessing the status at sub-second intervals might be a legit one, so what measuring units should be used?

At the beginning of this thread, Jeff said "calling the ioctls without priveleges is quite useful". Now if you say that there is no such case, the whole problem could simply be solved by checking for the appropriate priviledges.

I just realized another thing, important (IMHO) if a normal user is still allowed to access MII: the drivers (checked for 3c59x, eepro100, tulip) do not verify that the value passed for register number is within the allowed range and use it as:

        int read_cmd = (0xf6 << 10) | ((phy_id & 0x1f) << 5) | location;

(phy_id is the MII address and location is the register number).

There is also no check that the MII address specified is actually in use by the driver, but this is used with mii-diag to query a MII which was not correctly identified (maybe this should be allowed for CAP_NET_ADMIN only ?)

"MII transceivers have 32 management registers. The first 16 are reserved for standard-defined uses, and the remaining one are available for chip-specific features. Only the first seven registers are currently defined."

Usually, the transceivers return garbage if you read from locations you are not supposed to (overwritting phy_ad). But if you begin overwritting the READ command (0xf6 above)... Something like this should do:

        int read_cmd = (0xf6 << 10) | ((phy_id & 0x1f) << 5) | (location & 0x1f);

Too tired to think straight yesterday... You're right. And if you alloc 32*sizeof(int) (you want to keep jiffies, right ?) per netdevice, I think that it could even be done outside the driver. Hmm, most of my previous arguments are no longer valid 8-(

Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: [email protected]

Then it needs to be privileged

And if the approach is to block until the time for the next read occurs is done then the program get stuck for 30 seconds, misses its deadline and kills the cluster - how is this better ??

Some of them just hang.

Doing the MII monitoring somewhere centralised like the routing daemons would certainly let more inteillgent management and reporting get done

Fine. Can you think of a default value for expiring cache ?

Is not better. Well, when somebody is playing against you, you're in trouble either way:
- rate limit: - blocking - as above
- non-blocking - notify the user that you can't get the info and probably stop or aquire elevated priviledges and try to restart the network
- cache: get outdated info

But when a HA application runs, it's usually preferable to stop (and you notice it) than to continue with wrong data. Especially if you set the cache expiry to something like 30 seconds; think in terms of how many transactions/second today's hardware allows...

I don't argue over this point, already several people mentioned it. But I explained the present situation in a previous message: the MII info is normally read at a low rate and some applications need it more often. It doesn't matter that it's delivered through ioctl, netlink or any other way, you have to read it from the hardware and deliver to user-space at user request. So the "doing the MII monitoring" is the tough part.

Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: [email protected]

Yeah .. so long as its a default and tunable in /proc.

New day, new ideea. The original problem was that unpriviledged users can access it too often. How about exposing the MII registers as /dev entries? Then you can have normal access rights for them and no need to worry about frequency of access. Probably default would be 600 owned by root and for HA applications a user or a group can get read (or even write) access. It's up to the sysadmin to allow it, but has to be renewed after each boot. But I guess that this is not something to be applied to 2.2 and 2.4...

With clearer mind, I have to make some a correction to one of the previous
messages: the problem of not checking arguments range does not apply to 3c59x which has in the ioctl function '& 0x1f' for both transceiver number and register number. However, eepro100 and tulip don't do that. (I'm checking now with 2.4.3 from Mandrake 8, but I don't think that there were recent changes in these areas).

Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: [email protected]

half right -- tulip does this for the phy id but not the MII register number. I'll fix that up. Please bug Andrey about fixing up eepro100...

Jeff Garzik      | Echelon words of the day, from The Register:
Building 1024    | FRU Lebed HALO Spetznaz Al Amn al-Askari Glock 26 
MandrakeSoft     | Steak Knife Kill the President anarchy echelon
                 | nuclear assassinate Roswell Waco World Trade Center

OK, Andrey is now CC-ed. However, I only checked the 3 mentioned drivers, while MII ioctl's are used in many more... I was hoping that the mantainers would jump in!

Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: [email protected]

Linus,

Please apply the attached patch.
It fixes a security problem of user-controlled access to the card ports from a non-privileged ioctl which should have read-only semantics.

Best regards
Andrey

@@ -1913,7 +1913,7 @@
 		   timer routine.  2000/05/09 SAW */
 		saved_acpi = pci_set_power_state(sp->pdev, 0);
 		t = del_timer_sync(&sp->timer);
-		data[3] = mdio_read(ioaddr, data[0], data[1]);
+		data[3] = mdio_read(ioaddr, data[0] & 0x1f, data[1] & 0x1f);
 		if (t)
 			add_timer(&sp->timer); /* may be set to the past  --SAW */
 		pci_set_power_state(sp->pdev, saved_acpi);

Please no, 30 seconds is way too long for "your mains power is knackered, please fiddle with the power connector again" notification. When I jiggle the wire I need to know if it's making contact within 1 second, and if it's lost when I let it go, I need to know that within 1 second so I can jiggle it some more.

Probably should try some contact cleaner ;-)

Thanks...
-- Jamie