Device Registry (DevReg) Patch 0.2.0

The command
more foo/* foo/*/*
will display the values in the foo subtree nicely, I think.

Think of the /proc tree as the XML parse tree already exploded for you.

The only problem with /proc as it stands is that there is no formal syntax for its entries. Some of them are hard to parse.

Before we add a new /proc entry that generates XML which summarizes the rest of /proc, it might make sense to standardize /proc entries and write a regression test to verify they are formatted correctly. It would then be trivial to write a /proc to XML converter which ran solely in userspace.

See
http://www.uwsg.indiana.edu/hypermail/linux/kernel/0101.0/0506.html
and
http://marc.theaimsgroup.com/?l=linux-kernel&s=%2Fproc+xml

for prior discussion on the matter.

I don't want to dismiss the reasons you want to use XML for this, but tread carefully, lest you duplicate lots of code and introduce cruft. Better to factor the XML part out to a userspace library...

- Dan

Followup to:  <[email protected]>
By author:    Dan Kegel <[email protected]>
In newsgroup: linux.dev.kernel

/proc/sys is probably the method to follow. Every item is a datum of a simple datatype.

-hpa

<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Unfortunately it displays only the values. Dumping numbers and strings without knowing their meaning (and probably not even the order) is not very useful.

But the one-value per file approach is MORE work. It would be less work to create XML and factor out the directory structure in user-space :) Devreg collects its data from the drivers, each driver should contribute the information that it can provide about the device.
Printing a few values in XML format using the functions from xmlprocfs is as easy as writing
proc_printf(fragment, "<usb:topology port=\"%d\" portnum=\"%d\"/>\n",

                get_portnum(usbdev), usbdev->maxchild);

Extending the devreg output with driver-specific data means registering a callback function that prints the driver's data. The driver should use its own XML namespace, so whatever the driver adds will not break any (well-written) user-space applications. The data is created on-demand, so the values can be dynamic and do not waste any space when devreg is not used.

The code is easy to read and not larger than a solution that creates static /proc entries, and holding the data completely static would take much more memory. And it takes less code than a solution that would create the values in /proc dynamically because this would mean one callback per file or a complicated way to specify several values with a single callback.

bye...

--------- Received message begins Here ---------

Personally, I think

	proc_printf(fragment, "%d %d",get_portnum(usbdev), usbdev->maxchild);

(or the string "dddd ddd" with d representing a digit)

is shorter (and faster) to parse with

	fscanf(input,"%d %d",&usbdev,&maxchild);

Than it would be to try parsing

<usb:topology port="ddddd" portnum="dddd">

with an XML parser.

Sorry - XML is good for some things. It is not designed to be a interface language between a kernel and user space.

I am NOT in favor of "one file per value", but structured data needs to be written in a reasonable, concise manner. XML is intended for communication between disparate systems in an exreemly precise manner to allow some self documentation to be included when the communication fails.

Even Lisp S expressions are easier :-)

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]

Any opinions expressed are solely my own.

Agreed.

But one thing XML provides (potentially) is a DTD that defines meanings and formats. IMHO the kernel needs something like this for /proc (though not in DTD format!).

Has anyone ever tried to write a formal syntax for all the entries
in /proc? We have bits and pieces of /proc documentation in
/usr/src/linux/Documentation, but nothing you could feed directly into a parser generator. It'd be neat to have a good definition for /proc in the LSB, and have an LSB conformance test that could look in /proc and say "Yup, all the entries there conform to the spec and can be parsed properly."

(http://www.pathname.com/fhs/2.2-beta/fhs-2.2-beta.txt mentions /proc, but doesn't standardize any of it, except to suggest that /etc/mtab can be a symbolic link to /proc/mounts.)
- Dan

Right, but what happens if you need to extend the format? For example somebody adds support for USB 2.0 to the kernel and you need to some new values. Then you would have the choice between changing the format and breaking applications or keeping the format and dont provide the additional information.
With XML (or single-value-per-file) it is easy to tell application to ignore unknown tags (or files). When you just list values you will be damned sooner or later, unless you make up additional rules that say how apps should handle these cases. And then your approach is no longer simple, but possibly even more complicated

bye...

Tim Jansen <[email protected]>:

Not necessarily. If the "extended data" is put following the current data (since the data is currently record oriented) just making the output format longer will not/should not casue problems in reading the data. Just look at FORTRAN for an example of a extensible input :-) More data on the record will/should just be ignored. The only coding change might be to use a fgets to read a record, followed by a sscanf to get the known values.

Alternatively, you can always put one value per record:

	tag:value
	tag2:value2...

This is still simpler than XML to read, and to generate.

The problem with this and XML is the same - If the tag is no longer relevent (or changes its name), then the output must either continue to include it, or break applications that depend on that tag.

In all cases, atomic extraction of the structured data will be problematical since there may be buffering issues in output. XML is very verbose, and the tagged format better; but a series of values goes even farther...

Try them out - Just go through the /proc/net formats and stick in the XML... Just don't count on the regular utilities to decode them. It would give some actual results to compair with the current structure.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]

Any opinions expressed are solely my own.

Both solutions only work for simple data, they dont help for more complex things like adding a variable-sized list of structures. Actually the first devreg version used something like your second proposal and I gave it up because it wasnt flexible enough to add USB configuration data.

bye...

Just my two cents.

It looks clear that /proc is for programs, not for humans. So the best format for proc is just binary values. So programs can read it quickly, even in a chunk if they know the format. But sometimes it is usefull to do a cat on a /proc entry.

Question: it is possible to redirect the same fs call (say read) to different implementations, based on the open mode of the file descriptor ? So, if you open the entry in binary, you just get the number chunk, if you open it in ascii you get a pretty printed version, or a format description like
Bus: %d
Device: %h
..
to 'vprintf' the values.

J.A. Magallon                                          #  Let the source
mailto:[email protected]                              #  be with you, Luke... 

Linux werewolf 2.4.3-ac14 #1 SMP Wed Apr 25 02:07:45 CEST 2001 i686

There is no distinction between "text" and "binary" modes on a file descriptor. The distinction exists in the C stdio layer, but is a no-op on Unix systems.

-Doug

The rain man gave me two cures; he said jump right in,
The first was Texas medicine--the second was just railroad gin,
And like a fool I mixed them, and it strangled up my mind,
Now people just get uglier, and I got no sense of time...          --Dylan

of course. but we could trivially define O_PROC_BINARY, or an ioctl/fcntl, or even do something fancy like use lseek().

pardon my stream of consciousness here, but:

I think it's well-established that proc exists for humans, and that there's no real sympathy for the eternal whines of how terribly hard it is to parse. it's NOT hard to parse, but would be more trivial if it were more consistent.

the main goal at this point is to make kernel proc-related code more efficient, easy-to-use, etc. a purely secondary goal is to make user-space tools more robust, efficient, and simpler.

there are three things that need to be communicated through the proc interface, for each chunk of data: its type, it's name and its value. it's critical that data be tagged in some way, since that's the only way to permit back-compatibility. that is, a tool looking for a particular tag will naturally ignore new data with other tags.

/proc/sys is an attempt to provide tagged data; it works well, is easy to comprehend, but requires an open for each datum, and provides no hints about type.

/proc/cpuinfo is another attempt: "tag : data", with no attempt to provide types. the tags have also mutated somewhat over time.

/proc/partitions is an example of a record-oriented file:
one line per record, and tags for the record members at the top. still no typing information.

I have a sense that all of these could be collapsed into a single api where kernel systems would register hierarchies of tuples of <type,tag,callback>, where callback would be passed the tag, and proc code would take care of "rendering" the data into human readable text (default), binary, or even xml. the latter would require some signalling mechanism like O_PROC_XML or the like. further, programs could perform a meta-query, where they ask for the types and tags of a datum (or hierarchy), so that on subsequent queries, they'd now how to handle binary data.

if only one piece of code handled the rendering of /proc stuff, it could do more, without burdoning all the disparate /proc producers.

regards, mark hahn.

You also need to know the parent of the tuple to build a hierarchy. And it should be possible to create lists.

The callback prototypes for values could look like this:

int proc_value_cb_string(char *buf, void *context); // writes string to buf, // returns len of string or negative value for error
int proc_value_cb_int(int *value, void *context);

For parent/directory tuples you would provide two additional callbacks that set the context for their children and maybe take care of other things like locking (so they dont need to be done in every single value callback):

void *proc_value_cb_level_enter(void *old_context); // returns new context void proc_value_cb_level_leave(void *old_context, void *new_context);

For tuples with a list there would be two callbacks to get the list elements:

int proc_value_cb_list_num(void *context); // returns number of elements void *proc_value_cb_list_context(int index, void *context); // returns context // of the element at the given index or NULL

To register such a tuple you would have the following functions:
void proc_value_register_string(parent_handle_t parent,
const char *name,
proc_value_cb_string cb);
void proc_value_register_int(parent_handle_t parent,
const char *name,
proc_value_cb_int cb);
parent_handle_t proc_value_register_parent(parent_handle_t parent,
const char *name,
proc_value_cb_level_enter cb1,

		          proc_value_cb_level_leave cb2);
parent_handle_t proc_value_register_list(parent_handle_t h, 
	     	          proc_value_cb_list_num cbnum,
	                      proc_value_cb_list_context cbcon);

This is the simplest API that I can imagine for this. The only problem is that you need to write a callback for each value (file). Just printing XML still looks easier to me...

Then you can argue that once you have a single format implemented in the kernel you can convert it to whatever you like in user-space. And it seems like the decision for "one-value-per-file" in /proc has already been made (please correct me if not and we start all over again), so I will try to make a generic API like the one above for it.

That would undermine the only advantage of binary data: it's easy (and fast) to dump or read a C struct. Not that I would really care for binary data...

bye...

Yep, realized after the post, fopen() is a wrapper for open(). The idea is to (someway) set the proc entry in verbose vs fast-binary mode for reads. Perhaps an ioctl() or an fcntl() or something similar. So the verbose mode gives the field names, and the binary mode just gives the numbers. Applications that know what are reading can just read binary data, and fast.

J.A. Magallon                                          #  Let the source
mailto:[email protected]                              #  be with you, Luke... 

Linux werewolf 2.4.3-ac14 #1 SMP Wed Apr 25 02:07:45 CEST 2001 i686

OK, _what_ applications spend a considerable time (and considerable percentage of the total execution time) parsing stuff in /proc? ps(1)? top(1)? Fine. They touch how many files outside of /proc/<pid>/* ? Exactly.

_Please_, drop this idiotic "parsing ASCII is slow" strawman. Or show some valid examples.

Eh. Search in archives for "ascii is tough"...

marko

The meanings should be implied by the filenames, which are displayed (try it). The order is alphabetical by filename.

The corresponding one-value-per-file approach can probably be made to be a single call per value. IMHO that's more useful; it means that (once we agree on definitions) programs don't need to parse XML to access this data; they can go straight to the node in the document object model tree ( = /proc ). Think of /proc as a preparsed XML tree that hasn't been standardized yet.

... but XML parsing is something we don't want to force on people when we can provide the same data in a pre-parsed, much easier to access form, IMHO.

Have you bothered to go back and read the old discussions on this topic?

Are you trying to avoid writing a DTD? IMHO it would be better to have a single DTD for the entire tree, rather than a separate anything-goes namespace for each driver. Yes, this is more work, but all the Linux drivers are tightly integrated into the kernel source tree, we may as well have a tightly-integrated DTD documenting what each block, serial, synch, etc. driver must provide.

I think we both agree that there needs to be an easy, standardized way to access this data. IMHO there's a lot of standardizing that needs to happen before you can start writing code -- otherwise your new code won't help, and we'll be in the same mess we're in now.

The DTD can apply to both the existing /proc form and any proposed XML form of config info exported by the kernel; there should be an easy transformation between them. And it has to come first!

- Dan

Yes, the real problem is writing a callback-based filesystem (unless you want to hold everything in memory). After thinking about it for the last two hours I already find the one-value-per-file approach not as hard to do as I did before, but it's still a lot of work.

Yes. But in my case is different than, for example, the files in /proc/sys:
- the file names in /proc/sys are static. For devreg the filenames must be made dynamically (similar to the /proc process directories or usbdevfs)
- in /proc/sys there is just one piece for code responsible for every file or directory and no cooperation between different parts. If devreg creates, for example, a directory for a USB mouse it must be prepared to share this directory with the USB subsystem, the input subsystem and the USB hid driver. All four modules are responsible for their own files.
- files and their content should be created on demand, so there must be some callback to tell the USB subsystem something like "the user just opened the directory of device X, please tell me which directories or files you want to add".

It is certainly possible to convert devreg to the one-value-per-file approach and if this is all that it takes to get into some future (2.5) kernel I will do it. I just doubt that this is the easiest way to implement the functionality, because that's what I really want.

Yes, at least a have a complete DTD, because it would be a nightmare to maintain it. Each time somebody adds a new capability to a driver the DTD would have to be updated. And what about drivers that are not part of the official kernel?
I thought about using a separate XML Schema definition for each namespace though.

bye...