Skip to contentSkip to conversation

Rules on how to use sysfs in userspace programs

DISCUSSION Greg KH 8 participants 15 messages Jun 8, 2007 – Jun 25, 2007

lore
Share & tools
lore

Greg KH

author
#0 lore raw

Over time there have been a number of problems when sysfs has changed in "unexpected" ways. Here's a document that Kay wrote a while ago that I'd like to add to the kernel Documentation directory to help userspace programmers out.

Any comments or critique of this is greatly appreciated.

thanks,

greg k-h

------------------------------------

Rules to access device-information in the Linux kernel sysfs

The kernel exported sysfs exports internal kernel implementation-details and depends on internal kernel-structures and layout. It is agreed upon kernel developers, that the Linux kernel does not provide a stable internal API. As sysfs is a direct export of kernel internal structures, the sysfs interface can't provide a stable interface too, it may always change along with internal kernel changes.

To minimize the risk of breaking users of sysfs, which are in most cases low-level userspace applications, with a new kernel release, the users of sysfs must follow some rules to use an abstract-as-possible way to access this filesystem. The current udev and HAL programs already implement this and users are encouraged to plug, if possible, into the abstractions these both programs provide instead of accessing sysfs directly.

But if you really do want to access sysfs, please follow the following rules and then your programs should work for all future versions of sysfs.

- Do not use libsysfs
It makes assumptions about sysfs which are not true. Its API does not offer any abstraction, it exposes all the kernel driver-core implementation details in its own API. Therefore it is not better than reading directories and opening the files yourself.
Also, it is not actively maintained, in the sense of reflecting the current kernel-development. The goal of providing a stable interface to sysfs has failed, it causes more problems, than it solves. It violates many of the rules in this document.

- sysfs is always at /sys
Parsing /proc/mounts is a waste of time. Other mount points are a system configuration bug you should not try to solve. For test cases, possibly support SYSFS_PATH to overwrite the applications behavior, but never try to search for sysfs. Never try to mount it, if you are not an early boot script.

- devices are only "devices".
There is no such thing like class-, bus-, physical devices, interfaces, and such that you can rely on in userspace. Everything is just simply a "device" Class-, bus-, physical, ... types are just kernel implementation details, which should not be expected by applications that handle devices.

The properties of a device are:
o devpath (/devices/pci0000:00/0000:00:1d.1/usb2/2-2/2-2:1.0)
- identical to the DEVPATH value in the event sent from the kernel at device creation and removal
- the unique key to the device at that point in time
- the kernels path to the device-directory without the leading /sys, and always starting with with a slash
- all elements of a devpath must be real directories. Symlinks pointing to /sys/devices must always be resolved to their real target, and the target path must be used to access the device. That way the devpath to the device matches the devpath of the kernel used at event time.
- using or exposing symlink values as elements in a devpath string is a bug in the application

o kernel name (sda, tty, 0000:00:1f.2, ...)
- a directory name, identical to the last element of the devpath
- applications need to handle spaces and characters like '!' in the name

o subsystem (block, tty, pci, ...)
- simple string, never a path or a link
- retrieved by reading the "subsystem"-link and using only the last element of the target path
- the "bus"-link is identical to the "subsystem"-link, it is planned to remove the "bus"-link

o driver (tg3, ata_piix, uhci_hcd)
- a simple string, which may contain spaces, never a path or a link
- it is retrieved by reading the "driver"-link and using only the last element of the target path
- devices which don't have "driver"-link, just don't have a driver; copying the driver value in a child device context, is a bug in the application

o attributes
- the files in the device directory or files below a subdirectory of the same device directory
- exposing attributes reached by a symlink of another device, like the "device"-link, is a bug in the application

Everything else is just a kernel driver-core implementation detail, that should not be assumed to be stable across kernel releases.

- Properties of parent devices never belong into a child device. Always look at the parent devices themselves for determining device context properties. If the device 'eth0' or 'sda' does not have a "driver"-link, then it does not have a driver. It's value is empty. Never copy the value of the parent-device into a child-device. Parent device-properties may change dynamically without any notice to the child device.

- Hierarchy in a single device-tree
There is only one valid place in sysfs where hierarchy can be examined and this is below: /sys/devices.
It is planned, the all device directories will end up in the tree below this directory.

- Classification by subsystem
There are currently three places for classification of devices:
/sys/block, /sys/class and /sys/bus. It is planned, that these will not contain any device-directories themselves, but only flat lists of symlinks pointing to the unified /sys/devices tree.

All three places have completely different rules to access the information. It is planned to merge all three classification-directories into one place at /sys/subsystem/, following the current layout of the bus-directories. All buses and classes, including the converted block-subsystemm, will show up there.
The devices of a subsystem will create a symlink in the "devices" directory at /sys/subsystem/<name>/devices/.

If /sys/subsystem exists, /sys/bus, /sys/class and /sys/block can be ignored. If it doesn't exist, you have always to scan all three places, as the kernel is free to move a subsystem from one place to the other, as long as the devices are still reachable by the same subsystem name

Assuming /sys/class/<subsystem> and /sys/bus/<subsystem>, or /sys/block and /sys/class/block are not interchangeable, is a bug in the application.

- Block
The converted block-subsystem at /sys/class/block, or /sys/subsystem/block will contain the links for disks and partitions at the same level, never in a hierarchy. Assuming /sys/block to be there, is a bug in the application. Assuming the block-subsytem to contain only disks and not partition-devices in the same flat list is a bug in the application.

- "device"-link and <subsystem>:<kernel name>-links
Never depend on the "device"-link. The "device"-link is a workaround for the old layout, where class-devices are not created in /sys/devices/ like the bus-devices. If the link-resolving of a device-directory does not end in /sys/devices/, you can use the "device"-link to find the parent devices in
/sys/devices/. That's the single valid use of the "device"-link, it must never appear in any path as an element. Assuming the existence of the "device"-link for a device in /sys/devices/ is a bug in the application. Exposing /sys/class/net/eth0/device is a bug in the application. The "device"-link must be resolved to the real device-path it points to, if you look for the parent.

Never depend on the class specific links back to the /sys/class directory. These links are also a workaround for the design mistake, that class devices are not created in /sys/devices. If a device directory does not contain directories for child devices, these links may be used to find the child devices in /sys/class. That's the single valid use of these links, they must never appear in any path as an element. Assuming the existence of these links for devices which are real child device directories in the /sys/devices tree, is a bug in the application.

It is planned to remove all these links when when all class-device directories live in /sys/devices.

- Position of devices along device chain can change.
Never depend on a specific parent device in the devpath, or the chain of parent devices. The kernel is free to insert devices into the chain. You must always request the parent device you are looking for by its subsystem value. You need to walk up the chain until you find the device. Depending on a specific position of a parent device, or exposing relative paths, using "../" to access the chain of parents, is a bug in the application.

#1 lore raw
Quoted 5 lines · Greg KH

Any comments or critique of this is greatly appreciated.

------------------------------------

Quoted 8 lines

Rules to access device-information in the Linux kernel sysfs

The kernel exported sysfs exports internal kernel implementation-details and depends on internal kernel-structures and layout. It is agreed upon kernel developers, that the Linux kernel does not provide a stable internal API. As sysfs is a direct export of kernel internal structures, the sysfs interface can't provide a stable interface too, it may always change along with internal kernel changes.

Perhaps use the full writing form of "can't", "don't", etc. for, well, written text?

Quoted 4 lines

- Hierarchy in a single device-tree
There is only one valid place in sysfs where hierarchy can be examined and this is below: /sys/devices.
It is planned, the all device directories will end up in the tree

It is planned that all ...

Quoted 5 lines

below this directory.

- Classification by subsystem
There are currently three places for classification of devices:
/sys/block, /sys/class and /sys/bus. It is planned, that these will

No commas before "that".

Quoted 6 lines

not contain any device-directories themselves, but only flat lists of symlinks pointing to the unified /sys/devices tree.

All three places have completely different rules to access the information. It is planned to merge all three classification-directories into one place at /sys/subsystem/,

What about this future plan? If it is not going in soon or so, the paragraph should probably be left out.

Quoted 9 lines

following the current layout of the bus-directories. All buses and classes, including the converted block-subsystemm, will show up there.
The devices of a subsystem will create a symlink in the "devices" directory at /sys/subsystem/<name>/devices/.

It is planned to remove all these links when when all class-device directories live in /sys/devices.

Jan

#2 lore raw
In reply to Greg KH
Quoted 9 lines · Greg KH

- Do not use libsysfs
It makes assumptions about sysfs which are not true. Its API does not offer any abstraction, it exposes all the kernel driver-core implementation details in its own API. Therefore it is not better than reading directories and opening the files yourself.
Also, it is not actively maintained, in the sense of reflecting the current kernel-development. The goal of providing a stable interface to sysfs has failed, it causes more problems, than it solves. It violates many of the rules in this document.

The problem is that even on ftp.kernel.org there are packages like pcmciautils that still link to libsysfs and contain things like #define PATH_TO_SOCKET "/sys/class/pcmcia_socket/". Are they expected to be fixed?

Signature · 1 line
Alexander E. Patrakov
#3 lore raw
Quoted 11 lines · Alexander E. Patrakov
Quoted 9 lines · Greg KH

- Do not use libsysfs
It makes assumptions about sysfs which are not true. Its API does not offer any abstraction, it exposes all the kernel driver-core implementation details in its own API. Therefore it is not better than reading directories and opening the files yourself.
Also, it is not actively maintained, in the sense of reflecting the current kernel-development. The goal of providing a stable interface to sysfs has failed, it causes more problems, than it solves. It violates many of the rules in this document.

Quoted 3 lines

The problem is that even on ftp.kernel.org there are packages like pcmciautils that still link to libsysfs and contain things like #define PATH_TO_SOCKET "/sys/class/pcmcia_socket/". Are they expected to be fixed?

It will continue to work, because we keep the symlinks it expects, but libsysfs is so broken in its interface, that nothing should use it, especially nothing should start using it today.

Thanks,
Kay

#4 lore raw
In reply to Greg KH
Quoted 7 lines · Greg KH

Over time there have been a number of problems when sysfs has changed in "unexpected" ways. Here's a document that Kay wrote a while ago that I'd like to add to the kernel Documentation directory to help userspace programmers out.

Any comments or critique of this is greatly appreciated.

Don't forget to update Documentation/00-INDEX when you add the document :-)

More comments below.

Quoted 8 lines

thanks,

greg k-h

------------------------------------

Rules to access device-information in the Linux kernel sysfs

"Rules on how to access..." perhaps?

Quoted 3 lines

The kernel exported sysfs exports internal kernel implementation-details and depends on internal kernel-structures and layout. It is agreed upon kernel developers, that the Linux kernel does not provide a stable

You write "It is agreed upon kernel developers..." I would write "It is agreed upon by the kernel developers..."

Quoted 2 lines

internal API. As sysfs is a direct export of kernel internal structures, the sysfs interface can't provide a stable interface too, it

Shouldn't that be "can't provide a stable interface either" ?

Quoted 5 lines

may always change along with internal kernel changes.

To minimize the risk of breaking users of sysfs, which are in most cases low-level userspace applications, with a new kernel release, the users of sysfs must follow some rules to use an abstract-as-possible way to

You write "to use an abstract-as-possible way to" I would have written "to use an as abstract-as-possible way to"

Quoted 3 lines

access this filesystem. The current udev and HAL programs already implement this and users are encouraged to plug, if possible, into the abstractions these both programs provide instead of accessing sysfs

shouldn't that be "both these" not "these both" ?

Quoted 6 lines

directly.

But if you really do want to access sysfs, please follow the following rules and then your programs should work for all future versions of sysfs.

[snip]

Quoted 5 lines

- Properties of parent devices never belong into a child device. Always look at the parent devices themselves for determining device context properties. If the device 'eth0' or 'sda' does not have a "driver"-link, then it does not have a driver. It's value is empty.

"it's" is short for "it is", not the same as "its" which I believe is what you want here.

Quoted 8 lines

Never copy the value of the parent-device into a child-device. Parent device-properties may change dynamically without any notice to the child device.

- Hierarchy in a single device-tree
There is only one valid place in sysfs where hierarchy can be examined and this is below: /sys/devices.
It is planned, the all device directories will end up in the tree

Don't you mean "that all device..." ?

Quoted 9 lines

below this directory.

- Classification by subsystem
There are currently three places for classification of devices:
/sys/block, /sys/class and /sys/bus. It is planned, that these will not contain any device-directories themselves, but only flat lists of symlinks pointing to the unified /sys/devices tree.

All three places have completely different rules to access the

I believe that should be "rules on how to access the".

Quoted 8 lines

information. It is planned to merge all three classification-directories into one place at /sys/subsystem/, following the current layout of the bus-directories. All buses and classes, including the converted block-subsystemm, will show up there.
The devices of a subsystem will create a symlink in the "devices" directory at /sys/subsystem/<name>/devices/.

[snip]

Signature · 3 lines
Jesper Juhl <[email protected]>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html
#5 lore raw
In reply to Greg KH
Quoted 6 lines · Greg KH

The kernel exported sysfs exports internal kernel implementation-details and depends on internal kernel-structures and layout. It is agreed upon kernel developers, that the Linux kernel does not provide a stable internal API. As sysfs is a direct export of kernel internal structures, the sysfs interface can't provide a stable interface too, it may always change along with internal kernel changes.

I want to step back and ask a very fundamental philosophical question --- who are the intended users of sysfs? If it exports an interface, part of which is known not be stable, except for backwards compatibility issues, why and to whom are we exporting them?

If the answer is "no one", then maybe modulo backwards compatibility issues, we should only export via sysfs those things that are guaranteed to be a stable interface. Or maybe there should be a mount option that filters out anything which isn't guaranteed to be stable, so that user programs can easily determine if they are using anything "bad" --- and then a year or two from now, we make that mount option the default, and require a mount option if people want to see the unstable bits of sysfs.

Way back in 2003, Rusty's OLS keynote talked at great length about kernel design and interface similicity as a hallmark of good design:

1. Compiler/linker won't let you get it wrong.
2. Compiler will warn if you get it wrong.
3. The simplest use is the correct one.
4. The name tells you how to use it.
5. Do it right or it will break at runtime.
6. Follow common convention and you'll get it right.
7. Read the documentation and you'll get it right.
8. Read the implementation and you'll get it right.
9. Read the correct mailing list thread and you'll get it right.
10. Read the documentation and you'll get it wrong.
11. Follow common convention and you'll get it wrong.
12. Do it right and it will break at runtime.
13. The name tells you how not to use it.
14. The obvious use is wrong.
15. Compiler will warn if you get it right.
16. Compiler won't let you get it right.
17. It's impossible to get right.

The documentation is a good step, but wouldn't it be better if we can prohibit (or warn) applications who are getting it wrong in the first place? Providing interfaces that aren't stable, and then requiring people to read documentation so they know not to use them, seems to be providing an attractive nuisance.

- Ted

#6 lore raw
Quoted 8 lines · On Sun, 10 Jun 2007 10:02:00 -0400 Theodore Tso wrote:
Quoted 6 lines · Greg KH

The kernel exported sysfs exports internal kernel implementation-details and depends on internal kernel-structures and layout. It is agreed upon kernel developers, that the Linux kernel does not provide a stable internal API. As sysfs is a direct export of kernel internal structures, the sysfs interface can't provide a stable interface too, it may always change along with internal kernel changes.

Quoted 4 lines

I want to step back and ask a very fundamental philosophical question --- who are the intended users of sysfs? If it exports an interface, part of which is known not be stable, except for backwards compatibility issues, why and to whom are we exporting them?

I agree that stepping back and asking questions is the right thing to do here.

/proc also reports lots of kernel internals, yet it is required to maintain compatibility.

ISTM that sysfs should allow additions, but only in ways that don't break backwards compatibility. I.e., it should maintain backwards compatible interfaces when new ones are added. Yeah, that means leave some crud around. Yes, it may be as bad as /proc, but I've said that for quite awhile.

Quoted 30 lines

If the answer is "no one", then maybe modulo backwards compatibility issues, we should only export via sysfs those things that are guaranteed to be a stable interface. Or maybe there should be a mount option that filters out anything which isn't guaranteed to be stable, so that user programs can easily determine if they are using anything "bad" --- and then a year or two from now, we make that mount option the default, and require a mount option if people want to see the unstable bits of sysfs.

Way back in 2003, Rusty's OLS keynote talked at great length about kernel design and interface similicity as a hallmark of good design:

1. Compiler/linker won't let you get it wrong.
2. Compiler will warn if you get it wrong.
3. The simplest use is the correct one.
4. The name tells you how to use it.
5. Do it right or it will break at runtime.
6. Follow common convention and you'll get it right.
7. Read the documentation and you'll get it right.
8. Read the implementation and you'll get it right.
9. Read the correct mailing list thread and you'll get it right.
10. Read the documentation and you'll get it wrong.
11. Follow common convention and you'll get it wrong.
12. Do it right and it will break at runtime.
13. The name tells you how not to use it.
14. The obvious use is wrong.
15. Compiler will warn if you get it right.
16. Compiler won't let you get it right.
17. It's impossible to get right.

Quoted 5 lines

The documentation is a good step, but wouldn't it be better if we can prohibit (or warn) applications who are getting it wrong in the first place? Providing interfaces that aren't stable, and then requiring people to read documentation so they know not to use them, seems to be providing an attractive nuisance.

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

#7 lore raw
Quoted 18 lines · Randy Dunlap
Quoted 8 lines · On Sun, 10 Jun 2007 10:02:00 -0400 Theodore Tso wrote:
Quoted 6 lines · Greg KH

The kernel exported sysfs exports internal kernel implementation-details and depends on internal kernel-structures and layout. It is agreed upon kernel developers, that the Linux kernel does not provide a stable internal API. As sysfs is a direct export of kernel internal structures, the sysfs interface can't provide a stable interface too, it may always change along with internal kernel changes.

Quoted 4 lines

I want to step back and ask a very fundamental philosophical question --- who are the intended users of sysfs? If it exports an interface, part of which is known not be stable, except for backwards compatibility issues, why and to whom are we exporting them?

I agree that stepping back and asking questions is the right thing to do here.

Quoted 8 lines

/proc also reports lots of kernel internals, yet it is required to maintain compatibility.

ISTM that sysfs should allow additions, but only in ways that don't break backwards compatibility. I.e., it should maintain backwards compatible interfaces when new ones are added. Yeah, that means leave some crud around. Yes, it may be as bad as /proc, but I've said that for quite awhile.

We try hard to keep the properties of the exported devices and the classification directories stable, but sysfs is an one-to-one export of the kernels object-model and unfortunately not a list of artificially flat files like /proc. I don't see how a tree of directories, directly created by a kernel that change that insanely fast, can be kept stable.

There is also a big difference to /proc: the /sys layout itself can even change at runtime, some devices can be renamed, and others move around and change their devpath in the tree while they are active. In /sys, the _location_ of the information itself is not stable, only if you follow some rules how to extract information from the tree, you will be able to find it reliably.

Users need to follow some rules how to access information in the /sys tree, and sure, it is far more complicated than reading /proc. That's what this document tries to explain. It tries to document the rules, kind of similar to the (simple and intuitive) rule for /proc, that files need to be parsed for keys, instead of just assuming that the order of information is always the same, or the content of the files always contain the same keys.

Sysfs today exports all the completely useless implementation details of the kernel. The recent changes, and the work-in-progress to move everything into _one_ single device tree (/sys/devices) and _one_ symlinks-only based classification (/sys/subsystem) should make it much easier to keep the same information around, even when things in the kernel object-tree keep changing at the current rate.

This document just tries to describe the real world today. If you have any better idea, on how to export an internal device-tree from a kernel that changes every hour and subsystems come and go, get replaced and get changed all the time, we are more that happy to improve things here. :)

Thanks,
Kay

#8 lore raw
In reply to Greg KH
Quoted 6 lines · On Friday 08 June 2007 16:36:37 Greg KH wrote:

Over time there have been a number of problems when sysfs has changed in "unexpected" ways. Here's a document that Kay wrote a while ago that I'd like to add to the kernel Documentation directory to help userspace programmers out.

Any comments or critique of this is greatly appreciated.

Still catching up from my laptop dying.

I find the explanation of /sys/subsystem almost unintelligible. What will the new one actually look like?

If I want to find all block devices in the system, it looks like I should now look at /sys/subsystem/block. (And "subsystem" is not a variable here but the actual directory name? I presume it moved for Feng Shui reasons.)

If I want to find char devices, where do I look? /sys/subsystem... char? class? Is a char device now anything under /sys/subsystem that is _not_ in /sys/subsystem/block? (Minus bus devices?) Is there a specific directory for these?

This document is highly polluted with what NOT to do. I'm looking for a clear "what SHOULD I do", and it takes some wading to find it. (Historical cruft about what not to do is potentially a separate document, it's not useful for people learning this stuff now who weren't playing with the legacy mechanisms.) The description of /sys/subsystem spends so much time talking about old legacy issues it never gives a clear description of the new way of doing things, which is theoretically what this document is about...

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#9 lore raw
Quoted 6 lines · Rob Landley
Quoted 5 lines · On Friday 08 June 2007 16:36:37 Greg KH wrote:

Over time there have been a number of problems when sysfs has changed in "unexpected" ways. Here's a document that Kay wrote a while ago that I'd like to add to the kernel Documentation directory to help userspace programmers out.

Quoted 6 lines
Quoted 1 line

Any comments or critique of this is greatly appreciated.

Still catching up from my laptop dying.

I find the explanation of /sys/subsystem almost unintelligible. What will the new one actually look like?

"It is planned to merge all three classification-directories into one place at /sys/subsystem/, following the current layout of the bus-directories."

Means that /sys/subsystem/ will have a devices/ directory, full of symlinks to the devices, all in a flat list. Subsytem-global attribute files/directories are not mixed with the devices in the same directory like in /sys/class, it will also not contain any hierarchy like the layout of /sys/block.

Quoted 2 lines

If I want to find all block devices in the system, it looks like I should now look at /sys/subsystem/block.

Yes, in this order (if you want to use it, but /sys/block will still be there):
/sys/subsytem/block/devices/*
/sys/class/block/*
/sys/block/*/*

Quoted 2 lines

(And "subsystem" is not a variable here but
the actual directory name? I presume it moved for Feng Shui reasons.)

"one place at /sys/subsystem/"

Yes, it will be all pretty consistent, the event-environment contains $SUBSYSTEM, we will have /sys/subsystem/$SUBSYSTEM/devices/ directory and at every device a symlink named "subsystem" pointing back to the /sys/subsystem/$SUBSYSTEM/ directory.

Quoted 4 lines

If I want to find char devices, where do I look? /sys/subsystem... char? class? Is a char device now anything under /sys/subsystem that is _not_ in /sys/subsystem/block? (Minus bus devices?) Is there a specific directory for these?

If /sys/subsystem exists, just look at /sys/subsystem/*/devices/*, you will find every kernel device here, with exactly the logic to access it. Every device with a "dev" file, it is a char device, unless $SUBYSTEM=="block".

If /sys/subsystem/ doesn't exist, you have to search all through /sys/bus/, /sys/class/, /sys/block/, every directory with completely different access pattern to find your device. You may want to look at the udev code, it's all implemented there:
http://git.kernel.org/?p=linux/hotplug/udev.git;a=blob;f=udevtrigger.c;hb=HEAD#l498

Quoted 1 line

This document is highly polluted with what NOT to do.

Yeah, that's sysfs. It exports all the useless kernel implementation details, so that _what_not_to_do_ is the biggest problem we have. :) There is much too much internal stuff available here, that never can be kept stable in the usual sense, as long as we allow to change kernel/driver internals at the same time.

Quoted 7 lines

I'm looking for a
clear "what SHOULD I do", and it takes some wading to find it. (Historical cruft about what not to do is potentially a separate document, it's not useful for people learning this stuff now who weren't playing with the legacy mechanisms.) The description of /sys/subsystem spends so much time talking about old legacy issues it never gives a clear description of the new way of doing things, which is theoretically what this document is about...

It was a first cut, I did months ago, and sure, it needs some work.

Thanks,
Kay

#10 lore raw
Quoted 18 lines · On Saturday 23 June 2007 08:49:47 Kay Sievers wrote:
Quoted 6 lines · Rob Landley
Quoted 5 lines · On Friday 08 June 2007 16:36:37 Greg KH wrote:

Over time there have been a number of problems when sysfs has changed in "unexpected" ways. Here's a document that Kay wrote a while ago that I'd like to add to the kernel Documentation directory to help userspace programmers out.

Quoted 6 lines
Quoted 1 line

Any comments or critique of this is greatly appreciated.

Still catching up from my laptop dying.

I find the explanation of /sys/subsystem almost unintelligible. What will the new one actually look like?

"It is planned to merge all three classification-directories into one
place at /sys/subsystem/, following the current layout of the
bus-directories."

Quoted 5 lines

Means that /sys/subsystem/ will have a devices/ directory, full of symlinks to the devices, all in a flat list. Subsytem-global attribute files/directories are not mixed with the devices in the same directory like in /sys/class, it will also not contain any hierarchy like the layout of /sys/block.

But will it still be possible to distinguish block devices from character devices when teaching mdev to quickly scan for devices to populate /dev in embedded systems using the "new" locations for things?

Quoted 7 lines
Quoted 2 lines

If I want to find all block devices in the system, it looks like I should now look at /sys/subsystem/block.

Yes, in this order (if you want to use it, but /sys/block will still be there): /sys/subsytem/block/devices/*
/sys/class/block/*
/sys/block/*/*

If /sys/block will still be there, and this is reliable and just "not deprecated _yet_", then life is good. I got the impression from the document that /sys/block was going away at some point.

Quoted 9 lines
Quoted 2 lines

(And "subsystem" is not a variable here but
the actual directory name? I presume it moved for Feng Shui reasons.)

"one place at /sys/subsystem/"

Yes, it will be all pretty consistent, the event-environment contains $SUBSYSTEM, we will have /sys/subsystem/$SUBSYSTEM/devices/ directory and at every device a symlink named "subsystem" pointing back to the /sys/subsystem/$SUBSYSTEM/ directory.

Which is good to know, but only useful if called from event context rather than scanning for devices from init scripts so it can mknod in /dev and fire off subsidiary scripts.

Quoted 9 lines
Quoted 4 lines

If I want to find char devices, where do I look? /sys/subsystem... char? class? Is a char device now anything under /sys/subsystem that is _not_ in /sys/subsystem/block? (Minus bus devices?) Is there a specific directory for these?

If /sys/subsystem exists, just look at /sys/subsystem/*/devices/*, you will find every kernel device here, with exactly the logic to access it. Every device with a "dev" file, it is a char device, unless $SUBYSTEM=="block".

Oh good. That last sentence contains the heuristic I need.

Quoted 2 lines

If /sys/subsystem/ doesn't exist, you have to search all through /sys/bus/, /sys/class/, /sys/block/, every directory with completely

No, only /sys/class and /sys/block. Currently, /sys/class contains char devices and /sys/block contains block devices. You don't have to invoke mknod for a bus.

Quoted 9 lines

different access pattern to find your device. You may want to look at the udev code, it's all implemented there:
http://git.kernel.org/?p=linux/hotplug/udev.git;a=blob;f=udevtrigger.c;hb=H
EAD#l498

Quoted 1 line

This document is highly polluted with what NOT to do.

Yeah, that's sysfs. It exports all the useless kernel implementation details, so that _what_not_to_do_ is the biggest problem we have. :)

If you document a specific subset of the data it exports and say "you can rely on this being there, lots of the rest is implementation details", then people have less cause to complain when the bits you haven't documented change.

Right now, it's all undocumented and if you try to wade through the documentation it spends most of its time reminiscing about the bad old days of... nine months ago.

Quoted 3 lines

There is much too much internal stuff available here, that never can be kept stable in the usual sense, as long as we allow to change kernel/driver internals at the same time.

Then document what is stable, please.

Quoted 10 lines
Quoted 8 lines

I'm looking for a
clear "what SHOULD I do", and it takes some wading to find it. (Historical cruft about what not to do is potentially a separate document, it's not useful for people learning this stuff now who weren't playing with the legacy mechanisms.) The description of /sys/subsystem spends so much time talking about old legacy issues it never gives a clear description of the new way of doing things, which is theoretically what this document is about...

It was a first cut, I did months ago, and sure, it needs some work.

I'm very interested in helping out with it, and updating mdev based on the documentation rather than the source code, but not until after OLS I expect. :)

Quoted 2 lines

Thanks,
Kay

Thanks,

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#11 lore raw
Quoted 25 lines · Rob Landley
Quoted 18 lines · On Saturday 23 June 2007 08:49:47 Kay Sievers wrote:
Quoted 6 lines · Rob Landley
Quoted 5 lines · On Friday 08 June 2007 16:36:37 Greg KH wrote:

Over time there have been a number of problems when sysfs has changed in "unexpected" ways. Here's a document that Kay wrote a while ago that I'd like to add to the kernel Documentation directory to help userspace programmers out.

Quoted 6 lines
Quoted 1 line

Any comments or critique of this is greatly appreciated.

Still catching up from my laptop dying.

I find the explanation of /sys/subsystem almost unintelligible. What will the new one actually look like?

"It is planned to merge all three classification-directories into one
place at /sys/subsystem/, following the current layout of the
bus-directories."

Quoted 5 lines

Means that /sys/subsystem/ will have a devices/ directory, full of symlinks to the devices, all in a flat list. Subsytem-global attribute files/directories are not mixed with the devices in the same directory like in /sys/class, it will also not contain any hierarchy like the layout of /sys/block.

Quoted 3 lines

But will it still be possible to distinguish block devices from character devices when teaching mdev to quickly scan for devices to populate /dev in embedded systems using the "new" locations for things?

Sure, all devices have a "subsystem" link, you have to readlink() that, and if it ends in "block, you have a blockdev. But as mentioned in an earlier mail, you should stop scanning /sys/devices/ and always come from the subsystem directories, so you get "block" for free.

Quoted 36 lines
Quoted 7 lines
Quoted 2 lines

If I want to find all block devices in the system, it looks like I should now look at /sys/subsystem/block.

Yes, in this order (if you want to use it, but /sys/block will still be there): /sys/subsytem/block/devices/*
/sys/class/block/*
/sys/block/*/*

If /sys/block will still be there, and this is reliable and just "not deprecated _yet_", then life is good. I got the impression from the document that /sys/block was going away at some point.

Quoted 9 lines
Quoted 2 lines

(And "subsystem" is not a variable here but
the actual directory name? I presume it moved for Feng Shui reasons.)

"one place at /sys/subsystem/"

Yes, it will be all pretty consistent, the event-environment contains $SUBSYSTEM, we will have /sys/subsystem/$SUBSYSTEM/devices/ directory and at every device a symlink named "subsystem" pointing back to the /sys/subsystem/$SUBSYSTEM/ directory.

Which is good to know, but only useful if called from event context rather than scanning for devices from init scripts so it can mknod in /dev and fire off subsidiary scripts.

Quoted 9 lines
Quoted 4 lines

If I want to find char devices, where do I look? /sys/subsystem... char? class? Is a char device now anything under /sys/subsystem that is _not_ in /sys/subsystem/block? (Minus bus devices?) Is there a specific directory for these?

If /sys/subsystem exists, just look at /sys/subsystem/*/devices/*, you will find every kernel device here, with exactly the logic to access it. Every device with a "dev" file, it is a char device, unless $SUBYSTEM=="block".

Quoted 8 lines

Oh good. That last sentence contains the heuristic I need.

Quoted 2 lines

If /sys/subsystem/ doesn't exist, you have to search all through /sys/bus/, /sys/class/, /sys/block/, every directory with completely

No, only /sys/class and /sys/block. Currently, /sys/class contains char devices and /sys/block contains block devices. You don't have to invoke mknod for a bus.

Sure, you have! There are devices in /sys/bus which export device nodes, and the number will just grow.

That's why the document states: "There is no such thing like class-, bus-, physical devices, interfaces, and such that you can rely on in userspace. Everything is just simply a "device". Class-, bus-, physical, ... types are just kernel implementation details, which should not be expected by applications that handle devices."

Quoted 33 lines
Quoted 9 lines

different access pattern to find your device. You may want to look at the udev code, it's all implemented there:
http://git.kernel.org/?p=linux/hotplug/udev.git;a=blob;f=udevtrigger.c;hb=H
EAD#l498

Quoted 1 line

This document is highly polluted with what NOT to do.

Yeah, that's sysfs. It exports all the useless kernel implementation details, so that _what_not_to_do_ is the biggest problem we have. :)

If you document a specific subset of the data it exports and say "you can rely on this being there, lots of the rest is implementation details", then people have less cause to complain when the bits you haven't documented change.

Right now, it's all undocumented and if you try to wade through the documentation it spends most of its time reminiscing about the bad old days of... nine months ago.

Quoted 3 lines

There is much too much internal stuff available here, that never can be kept stable in the usual sense, as long as we allow to change kernel/driver internals at the same time.

Then document what is stable, please.

Quoted 9 lines
Quoted 8 lines

I'm looking for a
clear "what SHOULD I do", and it takes some wading to find it. (Historical cruft about what not to do is potentially a separate document, it's not useful for people learning this stuff now who weren't playing with the legacy mechanisms.) The description of /sys/subsystem spends so much time talking about old legacy issues it never gives a clear description of the new way of doing things, which is theoretically what this document is about...

Quoted 5 lines
Quoted 1 line

It was a first cut, I did months ago, and sure, it needs some work.

I'm very interested in helping out with it, and updating mdev based on the documentation rather than the source code, but not until after OLS I expect. :)

Sure, any help is welcome here.

Thanks,
Kay

#12 lore raw
Quoted 26 lines · On Sunday 24 June 2007 07:03:39 Kay Sievers wrote:
Quoted 25 lines · Rob Landley
Quoted 23 lines · On Saturday 23 June 2007 08:49:47 Kay Sievers wrote:
Quoted 6 lines · Rob Landley
Quoted 5 lines · On Friday 08 June 2007 16:36:37 Greg KH wrote:

Over time there have been a number of problems when sysfs has changed in "unexpected" ways. Here's a document that Kay wrote a while ago that I'd like to add to the kernel Documentation directory to help userspace programmers out.

Quoted 6 lines
Quoted 1 line

Any comments or critique of this is greatly appreciated.

Still catching up from my laptop dying.

I find the explanation of /sys/subsystem almost unintelligible. What will the new one actually look like?

"It is planned to merge all three classification-directories into one
place at /sys/subsystem/, following the current layout of the
bus-directories."

Means that /sys/subsystem/ will have a devices/ directory, full of symlinks to the devices, all in a flat list. Subsytem-global attribute files/directories are not mixed with the devices in the same directory like in /sys/class, it will also not contain any hierarchy like the layout of /sys/block.

Quoted 8 lines
Quoted 3 lines

But will it still be possible to distinguish block devices from character devices when teaching mdev to quickly scan for devices to populate /dev in embedded systems using the "new" locations for things?

Sure, all devices have a "subsystem" link, you have to readlink() that, and if it ends in "block, you have a blockdev. But as mentioned in an earlier mail, you should stop scanning /sys/devices/ and always come from the subsystem directories, so you get "block" for free.

I'm not scanning /sys/devices, I'm scanning /sys/block and /sys/class which this document implies are deprecated and going away. I'm trying to figure out what replaced them. Now instead of following a path to naturally get char devices in one pass and block devices in another, it seems I have to readlink a symlink and do string manipulation to identify the device type once I've found the device.

I can do this, I'd just like to confirm it's "the way" now, and by "the way" I mean Greg won't change his mind and yank it next month.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

If /sys/subsystem exists, just look at /sys/subsystem/*/devices/*, you will find every kernel device here, with exactly the logic to access it. Every device with a "dev" file, it is a char device, unless $SUBYSTEM=="block".

Oh good. That last sentence contains the heuristic I need.

Ok, hang on, looking back on this I'm confused again.

When you say /sys/subsystem are you referring to a literal path (which my sys directory currently doesn't have a subdirectory named "subsystem"), or do you mean /sys/$SUBSYSTEM where today I have /sys/class and /sys/block?

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

If /sys/subsystem/ doesn't exist, you have to search all through /sys/bus/, /sys/class/, /sys/block/, every directory with completely

No, only /sys/class and /sys/block. Currently, /sys/class contains char devices and /sys/block contains block devices. You don't have to invoke mknod for a bus.

Sure, you have! There are devices in /sys/bus which export device nodes, and the number will just grow.

I have yet to encounter any, could you give me some examples?

find /sys/bus -name dev

Nope, nothing...

Would these device nodes be char devices, or block devices, or... something else entirely? (The practical distinction between /sys/class and /sys/block used to be that one contained char devices and the other block devices. I could work with that. I'm sad to see it going away...)

Quoted 5 lines

That's why the document states: "There is no such thing like class-, bus-, physical devices, interfaces, and such that you can rely on in userspace. Everything is just simply a "device". Class-, bus-, physical, ... types are just kernel implementation details, which should not be expected by applications that handle devices."

Whether you feed "b" or "c" to mknod is a kernel implementation detail?

Are you unifying char and block devices so mknod doesn't have to distinguish between them anymore?

Quoted 5 lines
Quoted 3 lines

I'm very interested in helping out with it, and updating mdev based on the documentation rather than the source code, but not until after OLS I expect. :)

Sure, any help is welcome here.

What I've had to do each time was install a new kernel, find out what had changed by examination, and update programs to work with the new stuff.

I'd like to be able to do it from documentation.

Quoted 2 lines

Thanks,
Kay

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#13 lore raw
Quoted 31 lines · Rob Landley
Quoted 30 lines · On Sunday 24 June 2007 07:03:39 Kay Sievers wrote:
Quoted 28 lines · Rob Landley
Quoted 18 lines · On Saturday 23 June 2007 08:49:47 Kay Sievers wrote:
Quoted 6 lines · Rob Landley
Quoted 5 lines · On Friday 08 June 2007 16:36:37 Greg KH wrote:

Over time there have been a number of problems when sysfs has changed in "unexpected" ways. Here's a document that Kay wrote a while ago that I'd like to add to the kernel Documentation directory to help userspace programmers out.

Quoted 6 lines
Quoted 1 line

Any comments or critique of this is greatly appreciated.

Still catching up from my laptop dying.

I find the explanation of /sys/subsystem almost unintelligible. What will the new one actually look like?

"It is planned to merge all three classification-directories into one
place at /sys/subsystem/, following the current layout of the
bus-directories."

Quoted 5 lines

Means that /sys/subsystem/ will have a devices/ directory, full of symlinks to the devices, all in a flat list. Subsytem-global attribute files/directories are not mixed with the devices in the same directory like in /sys/class, it will also not contain any hierarchy like the layout of /sys/block.

But will it still be possible to distinguish block devices from character devices when teaching mdev to quickly scan for devices to populate /dev in embedded systems using the "new" locations for things?

Quoted 7 lines
Quoted 4 lines

Sure, all devices have a "subsystem" link, you have to readlink() that, and if it ends in "block, you have a blockdev. But as mentioned in an earlier mail, you should stop scanning /sys/devices/ and always come from the subsystem directories, so you get "block" for free.

I'm not scanning /sys/devices, I'm scanning /sys/block and /sys/class which this document implies are deprecated and going away.

No. "It is planned to merge all three classification-directories into one place at /sys/subsystem/", but the they will still be available for backwards compat, and continue to export silly details like differentiation between class- and bus-devices in the userspace export.

Quoted 5 lines

I'm trying to figure
out what replaced them. Now instead of following a path to naturally get char devices in one pass and block devices in another, it seems I have to readlink a symlink and do string manipulation to identify the device type once I've found the device.

You enter a well defined directory of symlinks, that directory is named "block". What's so difficult here?

Quoted 8 lines

I can do this, I'd just like to confirm it's "the way" now, and by "the way" I mean Greg won't change his mind and yank it next month.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

If /sys/subsystem exists, just look at /sys/subsystem/*/devices/*, you will find every kernel device here, with exactly the logic to access it. Every device with a "dev" file, it is a char device, unless $SUBYSTEM=="block".

Quoted 7 lines
Quoted 1 line
Quoted 1 line

Oh good. That last sentence contains the heuristic I need.

Ok, hang on, looking back on this I'm confused again.

When you say /sys/subsystem are you referring to a literal path (which my sys directory currently doesn't have a subdirectory named "subsystem"), or do you mean /sys/$SUBSYSTEM where today I have /sys/class and /sys/block?

"It is planned to merge ..." Today it exists only as a patch and will show up when the block-conversion eventually gets merged. It will be /sys/subsystem/$SUBSYSTEM/, just like /sys/bus looks today but containing _all_ devices from _all_ subsytems in _one_ place with _one_ access rule. Again, please look at the udev or HAL code, it's all pretty easy to understand by reading the code.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

If /sys/subsystem/ doesn't exist, you have to search all through /sys/bus/, /sys/class/, /sys/block/, every directory with completely

No, only /sys/class and /sys/block. Currently, /sys/class contains char devices and /sys/block contains block devices. You don't have to invoke mknod for a bus.

Quoted 6 lines
Quoted 2 lines

Sure, you have! There are devices in /sys/bus which export device nodes, and the number will just grow.

I have yet to encounter any, could you give me some examples?

find /sys/bus -name dev

There are only symlinks in /sys/bus, no attributes to "find" here.

Quoted 1 line

Nope, nothing...

Usb and firewire have device nodes at bus devices.

 $ ls -l /sys/bus/usb/devices/*/dev
 /sys/bus/usb/devices/1-2/dev
 /sys/bus/usb/devices/4-1/dev
 ...
Quoted 4 lines

Would these device nodes be char devices, or block devices, or... something else entirely? (The practical distinction between /sys/class and /sys/block used to be that one contained char devices and the other block devices. I could work with that. I'm sad to see it going away...)

SUBSTEM=="block" are block devices, everything else are char devices. Class- or bus-devices can be anything, you need to know the subsystem to distinguish them.

Unfortunately the "dev" file itself does not contain "c123:123", "b234:8" that would be the proper solution, but reading the subsytem works fine too for a long time.

Quoted 7 lines
Quoted 5 lines

That's why the document states: "There is no such thing like class-, bus-, physical devices, interfaces, and such that you can rely on in userspace. Everything is just simply a "device". Class-, bus-, physical, ... types are just kernel implementation details, which should not be expected by applications that handle devices."

Whether you feed "b" or "c" to mknod is a kernel implementation detail?

Please ...

Quoted 2 lines

Are you unifying char and block devices so mknod doesn't have to distinguish between them anymore?

No.

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

I'm very interested in helping out with it, and updating mdev based on the documentation rather than the source code, but not until after OLS I expect. :)

Sure, any help is welcome here.

What I've had to do each time was install a new kernel, find out what had changed by examination, and update programs to work with the new stuff.

I'd like to be able to do it from documentation.

Sure, that would be nice. any help documenting it in the right way, is appreciated.

Thanks,
Kay

#14 lore raw
Quoted 5 lines · On Monday 25 June 2007 10:57:35 Kay Sievers wrote:
Quoted 5 lines
Quoted 4 lines

Sure, all devices have a "subsystem" link, you have to readlink() that, and if it ends in "block, you have a blockdev. But as mentioned in an earlier mail, you should stop scanning /sys/devices/ and always come from the subsystem directories, so you get "block" for free.

Quoted 8 lines
Quoted 2 lines

I'm not scanning /sys/devices, I'm scanning /sys/block and /sys/class which this document implies are deprecated and going away.

No. "It is planned to merge all three classification-directories into one place at /sys/subsystem/", but the they will still be available for backwards compat, and continue to export silly details like differentiation between class- and bus-devices in the userspace export.

As long as "available for backwards compat" doesn't imply "will be deprecated and removed at some point in the future", then fine.

I'm not interested in the difference between class and bus. I don't even really know what it is. I'm interested in the difference between char and block, which is a fairly important difference if you want to actually mknod() anything.

Quoted 8 lines
Quoted 5 lines

I'm trying to figure
out what replaced them. Now instead of following a path to naturally get char devices in one pass and block devices in another, it seems I have to readlink a symlink and do string manipulation to identify the device type once I've found the device.

You enter a well defined directory of symlinks, that directory is named "block". What's so difficult here?

So anything that's not in /sys/subsystem/block is a char device? Is there one directory in which to look for char devices, or are there multiple directories? (Is there a /sys/subsystem/class? Or a /sys/subsystem/char? Or multiple directories I have to look at all of?)

You mentioned busses exporting device nodes. Would these be char or block device nodes?

Quoted 12 lines
Quoted 12 lines

I can do this, I'd just like to confirm it's "the way" now, and by "the way" I mean Greg won't change his mind and yank it next month.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

If /sys/subsystem exists, just look at /sys/subsystem/*/devices/*, you will find every kernel device here, with exactly the logic to access it. Every device with a "dev" file, it is a char device, unless $SUBYSTEM=="block".

Oh good. That last sentence contains the heuristic I need.

Ok, hang on, looking back on this I'm confused again.

Quoted 7 lines
Quoted 4 lines

When you say /sys/subsystem are you referring to a literal path (which my sys directory currently doesn't have a subdirectory named "subsystem"), or do you mean /sys/$SUBSYSTEM where today I have /sys/class and /sys/block?

"It is planned to merge ..." Today it exists only as a patch and will show up when the block-conversion eventually gets merged.

So code like mdev will need to keep using the old paths for this, or else not work on older kernels.

Quoted 4 lines

It will be
/sys/subsystem/$SUBSYSTEM/, just like /sys/bus looks today but containing _all_ devices from _all_ subsytems in _one_ place with _one_ access rule.

How is /sys/subsystem/{block,class} and improvement over /sys/{block,class}? I'm sure there _is_ a rationale for this, but I haven't actually been told what it is.

Quoted 2 lines

Again, please look at the udev or HAL code, it's all pretty easy to understand by reading the code.

I was trying to read the documentation Greg posted.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

If /sys/subsystem/ doesn't exist, you have to search all through /sys/bus/, /sys/class/, /sys/block/, every directory with completely

No, only /sys/class and /sys/block. Currently, /sys/class contains char devices and /sys/block contains block devices. You don't have to invoke mknod for a bus.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Sure, you have! There are devices in /sys/bus which export device nodes, and the number will just grow.

I have yet to encounter any, could you give me some examples?

find /sys/bus -name dev

There are only symlinks in /sys/bus, no attributes to "find" here.

There are devices in /sys/bus, but there are only symlinks in /sys/bus. That's very zen.

Would these devices be discoverable some other way? Do they currently show up under either /sys/class or /sys/block? If no, are they char device nodes or are they block device nodes?

Quoted 7 lines
Quoted 1 line

Nope, nothing...

Usb and firewire have device nodes at bus devices.

  $ ls -l /sys/bus/usb/devices/*/dev
  /sys/bus/usb/devices/1-2/dev
  /sys/bus/usb/devices/4-1/dev

Hmmm...

$ cd /sys/bus/usb/devices
$ ls */dev
$ ls: */dev: No such file or directory

Let me plug in my USB external hard drive...

$ ls */dev
ls: */dev: No such file or directory

Is this only in newer kernel than Ubuntu 7.04 comes with? Am I looking wrong?

Quoted 6 lines
Quoted 4 lines

Would these device nodes be char devices, or block devices, or... something else entirely? (The practical distinction between /sys/class and /sys/block used to be that one contained char devices and the other block devices. I could work with that. I'm sad to see it going away...)

SUBSTEM=="block" are block devices, everything else are char devices.

Ok. /sys/subsystem/block is where the block devices are, all other /sys/subsystem/* are char devices.

Quoted 2 lines

Class- or bus-devices can be anything, you need to know the subsystem to distinguish them.

So class and bus devices are NOT everything else, and therefore char devices?

I'M CONFUSED!

Quoted 3 lines

Unfortunately the "dev" file itself does not contain "c123:123", "b234:8" that would be the proper solution, but reading the subsytem works fine too for a long time.

Actually I've been doing:

  path[]="/sys/class"
  if (path[5]=='b') is_block();
  if (path[5]=='c') is_char();

Pretty straightforward, I thought. And it's had the added benefit of actually working, up until the config option to yank "old symlinks" went into the kernel...

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

I'm very interested in helping out with it, and updating mdev based on the documentation rather than the source code, but not until after OLS I expect. :)

Sure, any help is welcome here.

Quoted 7 lines
Quoted 4 lines

What I've had to do each time was install a new kernel, find out what had changed by examination, and update programs to work with the new stuff.

I'd like to be able to do it from documentation.

Sure, that would be nice. any help documenting it in the right way, is appreciated.

After OLS I'll take a whack at updating mdev to use non-legacy paths, and tell you what I needed to do.

Quoted 2 lines

Thanks,
Kay

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.