kernel support for non-english user messages

I'm one of those IBM guys who has been working on ideas for enhanced error/event logging in Linux. As part of this work, we've developed ideas for I18N of event-log messages. Here are some observations:

Most developers are comfortable with printk. Whatever approach you take, the closer you stay to printk in the programmatic interface, the more likely you are to gain acceptance.

Kernel messages need to be logged in English. By application of message catalogs or similar techniques, they can also be made available in other languages. Translation to other languages should be done when the log is viewed.

If you're serious about I18N, you must keep a message's format string distinct from the arg list. Your event-log viewer combines them all for you, using the desired language. For example, given the message

	printk(KERN_INFO "link up, %d Mbps, %s-duplex\n", speed, duplex);

you log the format string and the values of speed and duplex as separate attributes in the event log. If/when you compute a hash, it's on the format string (and possibly on the function name and/or source-file name, to provide more context).

Note that if you're particularly brave and/or stingy, you don't even have
to log the format string, just the hash. The event-log viewer can fetch the format string given the hash.

As the above suggests, a structured event log (see, for example, evlog.sourceforge.net) lends itself more readily to manipulation and analysis than does a simple text file.

We've come up with a couple of ways of generating message catalogs automatically. The method we prefer captures information about each printk* at compile time, and stores it in the .log section of the .o file. A post-processor then reads all the .o files and generates a sort of catalog entry for each printk call. The .log sections are stripped out when the bzImage is created. [*This technique requires some macro fussing. For reasons I won't go into, the macro is NOT called "printk", so this feature doesn't come for free.]

There are tens of thousands of printk calls in Linux. This has several implications:

- Inertia is a major factor.

- If we use 32-bit hash codes, there's a real chance of different messages
yielding the same hash code. We take the approach that each subsystem (e.g., each device driver) gets its own facility code (i.e., not KERN for everybody). If you key on hash code AND facility, the chance of collisions is very small.

- Although I believe it's worthwhile to categorize messages and tag them accordingly -- e.g., "hardware failure: <description of problem>" or "configuration note: <whatever>" -- most of the existing printk messages contain very context-specific info, and couldn't be entirely replaced with standard messages without loss of useful info.

Jim Keniston
IBM Linux Technology Center

Absolutely agreed. The only case you might want to argue otherwise is panic and very early boot messages. and even then it is questionable.

One of the problems is extracting the format string and other data.

Making the log hold
<6>%s: carrier dropped and smashed on the floor[U001]eth0

is in itself not hard and a big step forward.

There are less than 65536 files each of which is less than 65536 lines long, so it seems that a properly chosen automated index ought to be collision free ?

Hi Alan.

Some thoughts on that:

1. If the printk() messages are internationalised, we are going to see log extracts posted here in various languages, including some that the relevant maintainers don't understand. To stand any realistic chance of dealing with the resultant bug reports, we need to include the message code in the report so we can just feed the various reports through a tool that translates them into our preferred language.

2. For the above to work, we need the following guarantees:

a. A particular message code always refers to the same message.

b. A particular message is always referred to by the same
message code.

3. To obtain these guarantees, we need to ensure that the translation tool supplied with any particular kernel can handle all message codes from that kernel or from any earlier kernel in its direct ancestry. We thus can't reuse message codes once issued.

4. In some languages, the parameters will need to be specified in a different order to the English order.

5. We wish to keep the kernel size to a minimum.

The combination of the above points would lead me to suggest the following design:

1. The printk() function must NEVER be on the RHS of any #define statement. Many source files currently do this, and it kills any hope of an automated tool going through the kernel sources and allocating message numbers, irrespective of the numbering method chosen.

2. Given the above, it would be possible to change the compilation sequence such that the message indexing tool runs first and pre-processes each printk() call to replace the format string with an index into a table of message formats. This table would contain in each row first the message code allocated to that row, then the format string, and finally a key to the parameter order to be used. The table generated would thus be the English language file, and would be generated such that any existing messages therein were reused. This would have the benefit that where any particular message format occurs multiple times, they would be merged.

3. Given all of the above, a new printk() function would be written to index into the table and pick out the relevant row, then to produce a call to the current printk() function (renamed as printk2() or whatever) with its parameters sorted into the order specified by the final field in the table.

4. Where functions will be called prior to such internationalisations being available, they would call the printk2() function directly, and the message indexing tool would be designed to ignore such calls when doing its parsing.

5. The next step of the compilation would process the files produced by this tool rather than the original kernel sources.

This would then lead to the actual messages existing in a separate directory in the kernel source tree with the make *config process allowing one to select the appropriate language to be used, and auto-indexing the available languages (not hard to do). The compilation would then run a separate tool that created a *.h file with the relevant version of the table for that particular compilation.

One detail that would need to be handled is this: If the selected language file did not contain an entry for a particular message code, the entry for that message code would need to be extracted from the English language file. To help with translation, it should produce a report stating which message codes it had to do that for.

Also, the table would want to be sorted by message number to speed up access to the individual messages.

Best wishes from Riley.
---
* Nothing as pretty as a smile, nothing as ugly as a frown.

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.471 / Virus Database: 269 - Release Date: 10-Apr-2003

Providing the viewer is translating the originals always exist. Indeed you can do

	LANG=es view-logs
	LANG=ru view-logs
	...

You can have sysadmins with no common language("not a recommended configuration" ;))

You are right about needing to log parameters, but given a log line of the form

%s: went up in flames\n\0eth0\0\0

that can be handled by the log viewer

Wouldn't we be better off just more fully documenting the English error messages, though, and possibly translating that explaination document in to as many languages as possible? A lot of people search for error messages strings in the LKML archives, and variations of the same string will hinder this.

On a related note, if we are going to make a dmesg interpreter and translator, could we include the ability to generate a kernel .config from the boot time output? Note, I don't mean re-create the .config that the kernel was compiled with, but something like NetBSD, (at least), which lets you boot with a kernel that has drivers that you don't need, and generate a config file which will compile a kernel which contains only those that are actually needed.

John.

I don't think thats going to happen much. And to document them you have enumerate them so its still the same problem space strangely enough

I don't mean document them within the source, I mean a separate document, (distributed with the source), that people can refer to to find out what errors really mean. That solves the verbose/non versbose and translation problems. Users simply do a substring search of the manual for the error they're getting, and get an explaination in any language that manual has been translated in to. It doesn't have to be limited to errors, either, there are a lot of undocumented informational messages in the kernel, too.

I'd be happy to maintain such a document, but there doesn't seem much interest in it, and I can't see why.

John.