Skip to contentSkip to conversation

Documentation of kernel messages

RFC/PATCH In review holzheu 38 participants 133 messages Jun 13, 2007 – Aug 6, 2007

Subsystems (2)
lore
Share & tools
lore

holzheu

author
#0 lore raw

Greetings,

The operation of a Linux system sometimes requires to decode the meaning of a specific kernel message, e.g. an error message of a driver. Especially on our mainframe zSeries platform system administrators want to have descriptions for Linux kernel messages. They are used to that, because all other operating systems on that platform like z/OS, z/VM or z/VSE have message catalogs with detailed descriptions about the semantics of the messages.

In general we think, that also for Linux it is a good thing to have documentation for the most important kernel/driver messages. Even kernel hackers not always are aware of the meaning of kernel messages for components, which they don't know in detail. Most of the messages are self explaining but sometimes you get something like "Clocksource tsc unstable (delta = 7304132729 ns)" and you wonder if your system is going to explode.

Unfortunately currently there is no general infrastructure in the Linux kernel for the documentation of messages. I worked on a proposal, how that could be implemented in an easy way using the already existing kernel-doc infrastructure and using printk. The proposal is as follows

1. We use message identifiers in order to map messages to message descriptions. A message identifier consists out of a component name and within that component unique message number.

2. Messages and message descriptions are maintained together in the kernel sources in order to keep them up to date. Messages catalog are generated automatically for exactly one kernel level.

3. A special tool checks, if messages are not documented or if there are message descriptions without corresponding messages.

4. We use the already existing kernel-doc tool to generate an online message catalog or e.g. a pdf for offline documentation.

Current prototype implementation:
=================================

The structure of a kernel message is: <component>.<msg number>: <msg>

* component: Name of the kernel or driver component e.g. "pci", "ide", etc.
* msg number: Within the component unique number of a kernel message.
* msg: printk message

New macros KMSG_ERR(), KMSG_WARN(), etc. are defined, which have to be used in printk. These macros have as parameter the message number and are using a per c-file defined macro KMSG_COMPONENT.

Example: Define message 2 in component "kmsgtest":

#define KMSG_COMPONENT "kmsgtest"

void f(void)

{
        printk(KMSG_ERR(1) "device %x not online\n", devno);
}

The output of that kernel message would be:
"kmsgtest.1: device 4711 not online"

The messages have to be documented within the C source file in the following way:

/**
* message
* @0: device number of device.
*
* Description:
* An operation has been performed on the msgtest device, but the
* device has not been set online. Therefore the operation failed *
* User Response:
* Operator should set device online.
* Issue "chccwdev -e <device number>".
*
*/

KMSG_DOC(kmsgtest, 2, "Device %x not online");

I created a patch for the kernel-doc tool so it can be used to generate a catalog of all kernel messages:

Quoted 2 lines
Quoted 2 lines

kernel-doc -man kmsgtest.c > kmsgtest.2
man ./kmsgtest.2

# Kernel API(9) Linux Messages Kernel API(9)
#
# MESSAGE:
# kmsgtest.2: "device %x not online"
#
# Parameter:
# 1 Device number of device.
#
# Description:
# An operation has been performed on the msgtest device, but
# the device has not been set online. Therefore the operation failed.
#
# User Response:
# Operator should set device online.
# Issue "chccwdev -e <device number>".
#
# May 2007 kmsgtest.2 Kernel API(9)

A nice thing would be to include the online kernel message catalog in the kernel rpm. One possibility for that would be to have one man page per message. If an operator finds the message kmsgtest.2 in var/log/messages and wants to know what the message means, he simply issues "man kmsgtest.2" and gets the description.

To ensure that all messages are documented and there are no message descriptions without corresponding messages, a checker tool is provided. To enable message checking, in the toplevel Makefile the following has to be added:

CHECK = scripts/kmsg_check.pl check

To enable message checking during kernel build, the "C" option has to be used:

Quoted 1 line
Quoted 1 line

make modules C=1

  CHK     include/linux/version.h
  CHK     include/linux/utsrelease.h
  CHECK   drivers/kmsgtest/kmsgtest.c

drivers/kmsgtest/kmsgtest.c: Missing description for: kmsgtest.1 drivers/kmsgtest/kmsgtest.c: Description without message for: kmsgtest.3

Please note, that the patch for the kernel-doc and kmsg-doc tools is just a prototype and is neither complete nor perfect.

Michael

---

 Makefile                    |    5
 drivers/Makefile            |    1
 drivers/kmsgtest/Makefile   |    5
 drivers/kmsgtest/kmsgtest.c |   59 +++++++++++
 include/linux/kmsg.h        |   32 ++++++
 scripts/kernel-doc          |  116 +++++++++++++++++++++-
 scripts/kmsg-doc            |  231 ++++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 445 insertions(+), 4 deletions(-)
Makefile +2 -3
diff -Naur linux-2.6.21/Makefile linux-2.6.21-kmsg/Makefile
@@ -293,9 +293,8 @@
 DEPMOD		= /sbin/depmod
 KALLSYMS	= scripts/kallsyms
 PERL		= perl
-CHECK		= sparse
-
-CHECKFLAGS     := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ -Wbitwise $(CF)
+CHECK		= scripts/kmsg-doc check
+CHECKFLAGS	=
 MODFLAGS	= -DMODULE
 CFLAGS_MODULE   = $(MODFLAGS)
 AFLAGS_MODULE   = $(MODFLAGS)
drivers/Makefile +1 -0
diff -Naur linux-2.6.21/drivers/Makefile linux-2.6.21-kmsg/drivers/Makefile
@@ -8,6 +8,7 @@
 obj-$(CONFIG_PCI)		+= pci/
 obj-$(CONFIG_PARISC)		+= parisc/
 obj-$(CONFIG_RAPIDIO)		+= rapidio/
+obj-m				+= kmsgtest/
 obj-y				+= video/
 obj-$(CONFIG_ACPI)		+= acpi/
 # PnP must come after ACPI since it will eventually need to check if acpi
drivers/kmsgtest/Makefile +5 -0
diff -Naur linux-2.6.21/drivers/kmsgtest/Makefile linux-2.6.21-kmsg/drivers/kmsgtest/Makefile
@@ -0,0 +1,5 @@
+#
+# Makefile for kernel message test module
+#
+
+obj-m		+= kmsgtest.o
drivers/kmsgtest/kmsgtest.c +59 -0

59 lines not on this page · raw message

include/linux/kmsg.h +32 -0
diff -Naur linux-2.6.21/include/linux/kmsg.h linux-2.6.21-kmsg/include/linux/kmsg.h
@@ -0,0 +1,32 @@
+#ifndef _LINUX_KMSG_H
+#define _LINUX_KMSG_H
+
+#ifdef __KMSG_CHECKER
+
+#define KMSG_EMERG(num)   __KMSG_CHECK(EMERG, num)
+#define KMSG_ALERT(num)   __KMSG_CHECK(ALERT, num)
+#define KMSG_CRIT(num)    __KMSG_CHECK(CRIT, num)
+#define KMSG_ERR(num)     __KMSG_CHECK(ERR, num)
+#define KMSG_WARNING(num) __KMSG_CHECK(WARNING, num)
+#define KMSG_NOTICE(num)  __KMSG_CHECK(NOTICE, num)
+#define KMSG_INFO(num)    __KMSG_CHECK(INFO, num)
+#define KMSG_DEBUG(num)   __KMSG_CHECK(DEBUG, num)
+
+#define KMSG_DOC(comp, num, str) __KMSG_DOC(comp, num, str)
+
+#else
+
+#define KMSG_EMERG(num)   KERN_EMERG KMSG_COMPONENT "." #num ": "
+#define KMSG_ALERT(num)   KERN_ALERT KMSG_COMPONENT "." #num ": "
+#define KMSG_CRIT(num)    KERN_CRIT KMSG_COMPONENT "." #num ": "
+#define KMSG_ERR(num)     KERN_ERR KMSG_COMPONENT "." #num ": "
+#define KMSG_WARNING(num) KERN_WARNING KMSG_COMPONENT "." #num ": "
+#define KMSG_NOTICE(num)  KERN_NOTICE KMSG_COMPONENT "." #num ": "
+#define KMSG_INFO(num)    KERN_INFO KMSG_COMPONENT "." #num ": "
+#define KMSG_DEBUG(num)   KERN_DEBUG KMSG_COMPONENT "." #num ": "
+
+#define KMSG_DOC(comp, num, str)
+
+#endif /* __KMSG_CHECKER */
+
+#endif /* _LINUX_KMSG_H */
scripts/kernel-doc +115 -1

140 lines not on this page · raw message

scripts/kmsg-doc +231 -0

231 lines not on this page · raw message

#1 lore raw
Quoted 7 lines · holzheu

The operation of a Linux system sometimes requires to decode the meaning of a specific kernel message, e.g. an error message of a driver. Especially on our mainframe zSeries platform system administrators want to have descriptions for Linux kernel messages. They are used to that, because all other operating systems on that platform like z/OS, z/VM or z/VSE have message catalogs with detailed descriptions about the semantics of the messages.

I'm not sure we want to make Linux more like z/* in this regard. :)

The problem with your proposal is that every time a new message in the kernel is created or modified, you need somebody to go update that documentation. It's going to get out-of-sync very fast if this isn't done, and you either need to convince and teach each and every kernel contributor to follow your lead, or have a team of highly trained code monkeys to watch git-commits and resubmit documentation for every diff that touches a printk.

I think it's a great idea to go and update some of these obtuse kernel messages with more understandable wording. It might even be nice to update the Documentation/ about what kinds of messages are good or bad to have. But, I just don't think this is viable to convince everybody to do this. Do you have a list of printks that are causing your customers trouble?

Does every printk need one of these? What do we do with the existing printks? Who will convert them over? How do we enforce the rules if new printks must have documentation written for them? Who writes the documentation if the printk author does not?

If we have a catalog of kernel messages, it obviously needs to be versioned, but does that mean that we need to keep it in the kernel tree?

-- Dave

#2 lore raw
Quoted 11 lines · Dave Hansen
Quoted 7 lines · holzheu

The operation of a Linux system sometimes requires to decode the meaning of a specific kernel message, e.g. an error message of a driver. Especially on our mainframe zSeries platform system administrators want to have descriptions for Linux kernel messages. They are used to that, because all other operating systems on that platform like z/OS, z/VM or z/VSE have message catalogs with detailed descriptions about the semantics of the messages.

I'm not sure we want to make Linux more like z/* in this regard. :)

Quoted 7 lines

The problem with your proposal is that every time a new message in the kernel is created or modified, you need somebody to go update that documentation. It's going to get out-of-sync very fast if this isn't done, and you either need to convince and teach each and every kernel contributor to follow your lead, or have a team of highly trained code monkeys to watch git-commits and resubmit documentation for every diff that touches a printk.

This is no more and no less the same situation that we have with the kernel-doc documented functions/data in the kernel.

I like the concept that the description is kept close to the actual usage, the tool support and that in general looks like ordinary kernel-doc documentation.

And if people do not dare to update the kernel-doc documentation of a function then maybe they should not send patches in first place..

If we then really want to have the important printk's documented is another story.

Sam

author
#3 lore raw
In reply to Dave Hansen

Hi Dave,

On Wed, 2007-06-13 at 09:37 -0700, Dave Hansen wrote:

[snip]

Quoted 5 lines

I'm not sure we want to make Linux more like z/* in this regard. :)

The problem with your proposal is that every time a new message in the kernel is created or modified, you need somebody to go update that documentation.

You only have to document printks, which are using the KMSG macros. Since there are tons of self-explaining printks, most of them do not have to use them.

If you change the meaning of a KMSG printk and you don't update the documentation, it is the same thing like changing a kernel API function and forgetting to update the kernel-doc comment.

Quoted 5 lines

It's going to get out-of-sync very fast if this isn't
done, and you either need to convince and teach each and every kernel contributor to follow your lead, or have a team of highly trained code monkeys to watch git-commits and resubmit documentation for every diff that touches a printk.

It is a matter of discipline. If a device driver maintainer decides to document some messages using the KMSG macros, he has to take care that the documentation is up-to-date. Since the printks and the descriptions are maintained together in the kernel code, I think, that's doable.

And the checker tool can help to keep the descriptions up-to-date, since it produces warnings, if you use C=1 during the kernel build.

Quoted 5 lines

I think it's a great idea to go and update some of these obtuse kernel messages with more understandable wording. It might even be nice to update the Documentation/ about what kinds of messages are good or bad to have. But, I just don't think this is viable to convince everybody to do this.

Again, not everybody has to use KMSG printks.

Quoted 2 lines

Do you have a list of printks that are causing your customers trouble?

There are quite a lot of messages in our s390 device drivers which should be documented. Especially for character device drivers (e.g. tape), where the user has direct interaction with the device.

One Example from the tape device driver: "Another host has reserved the tape device"

We should document, how to find out the host, which reserved the tape and what has to be done on the other host in order to release the tape etc.

Quoted 2 lines

Does every printk need one of these? What do we do with the existing printks?

Again: Most printks don't need documentation. We can leave them as they are.

Quoted 3 lines

Who will convert them over? How do we enforce the rules if new printks must have documentation written for them? Who writes the documentation if the printk author does not?

The maintainer of the component (e.g. device driver) is responsible for the KMSG printks. If he accepts a patch, which modifies KMSG printks, he has to take care, that the patch also updates the KMSG_DOC part.

Quoted 3 lines

If we have a catalog of kernel messages, it obviously needs to be versioned, but does that mean that we need to keep it in the kernel tree?

For each kernel a separate message catalog can be automatically created. This message catalog fits exactly to one kernel and should be installed together with the kernel.

Michael

#4 lore raw
In reply to holzheu
Quoted 3 lines · holzheu

They are used to that, because all other operating systems on that platform like z/OS, z/VM or z/VSE have message catalogs with detailed descriptions about the semantics of the messages.

25 years ago, I did OS/MVT and OS/VS1 for a living, so I know *all* about the infamous "What does IEF507E mean again?"...

Quoted 7 lines

In general we think, that also for Linux it is a good thing to have documentation for the most important kernel/driver messages. Even kernel hackers not always are aware of the meaning of kernel messages for components, which they don't know in detail. Most of the messages are self explaining but sometimes you get something like "Clocksource tsc unstable (delta = 7304132729 ns)" and you wonder if your system is going to explode.

This is probably best addressed by cleaning up the actual messages so they're a bit more informative.

Quoted 3 lines

New macros KMSG_ERR(), KMSG_WARN(), etc. are defined, which have to be used in printk. These macros have as parameter the message number and are using a per c-file defined macro KMSG_COMPONENT.

Gaak. *NO*.

The *only* reason that the MVS and VM message catalogs worked at all is because each component had a message repository that went across *all* the source files - the instant you saw IEFnnns, you knew that IEF covered the job scheduler, nnn was a *unique* number, and s was a Severe/Warning/Info flag. IGG was always data management, and so on. This breaks horribly if you have 2 C files that define subtly different KMSG_COMPONENT values (or even worse, 2 or more duplicates).

[/usr/src/linux-2.6.22-rc4-mm2] find . -name '*.c' | wc -l 9959
[/usr/src/linux-2.6.22-rc4-mm2] find . -name '*.h' | wc -l 9933
[/usr/src/linux-2.6.22-rc4-mm2] find . -type d | wc -l 1736

You plan to maintain message uniqueness how?

[/usr/src/linux-2.6.22-rc4-mm2]1 find . -name '*.c' | sed -r 's?.*/([^/]*)?\1?' | sort | uniq -c | sort -nr | head

    105 setup.c
     90 irq.c
     66 time.c
     58 init.c
     50 inode.c
     39 io.c
     38 pci.c
     37 file.c
     32 signal.c
     32 ptrace.c

Looks like you're going to have to embed a lot of the path in that KMSG_COMPONENT to make it unique - and you want to keep that message under 80 or so chars total.

Quoted 7 lines

/**
* message
* @0: device number of device.
*
* Description:
* An operation has been performed on the msgtest device, but the
* device has not been set online. Therefore the operation failed

If you don't understand 'Device /dev/foo offline', this description doesn't help any. And that's true for *most* of the kernel messages already - if you don't understand the message already, a paragraph explanation isn't going to help much. Consider the average OOPS message, which contains stuff like 'EIP=0x..'. Telling the user that EIP means Execution Instruction Pointer isn't likely to help - if they knew what the pointer *did*, they'd probably already know EIP.

Quoted 4 lines

*
* User Response:
* Operator should set device online.
* Issue "chccwdev -e <device number>".

And this is where the weakness of this scheme *really* hits. I've actually run into cases where an operator followed the listed "Operator Response" for a "device offline", and issued a 'VARY 0C0,ONLINE'. And then we got a flood of I/O errors because the previous shift downed the device because it was having issues. The response the operator *should* have done is "assign a different tape drive, like, oh maybe the operational ones at 0C1 through 0C4"...

And it's the same here - if you get a message that /dev/sdb1 has no media present, there's a good chance that you typo'ed, and meant /dev/sda1 or /dev/sdc1 So following the directions for 'sdb1 offline' and putting in a blank DVD because sdb is the DVD burner won't fix things if what you were trying to do is mkfs something on another disk... ;)

And while we're at it, I'll point out that any attempt to "fix" the kernel messages on this scale had *better* solve all the I18N problems while we're there....

#5 lore raw
Quoted 7 lines · On Wednesday 13 June 2007 12:50:55 [email protected] wrote:
Quoted 7 lines

In general we think, that also for Linux it is a good thing to have documentation for the most important kernel/driver messages. Even kernel hackers not always are aware of the meaning of kernel messages for components, which they don't know in detail. Most of the messages are self explaining but sometimes you get something like "Clocksource tsc unstable (delta = 7304132729 ns)" and you wonder if your system is going to explode.

Isn't this what the severity levels are for? Not just filtering messages out but telling you whether or not it's important?

Quoted 2 lines

This is probably best addressed by cleaning up the actual messages so they're a bit more informative.

Seconded.

The point of a diagnostic message is to convey information. Deep in the bowels of chipsets there are diagnostic messages that get spit out as hexcodes that some program needs to interpret, but that's why we have tools like http://kernel.org/pub/linux/kernel/people/davej/tools/parsemce.c and such.

If the diagnostic messages need a talmud, I see this is as more of a problem with said diagnostic messages, rather than excuse to add another layer.

Quoted 5 lines
Quoted 3 lines

New macros KMSG_ERR(), KMSG_WARN(), etc. are defined, which have to be used in printk. These macros have as parameter the message number and are using a per c-file defined macro KMSG_COMPONENT.

Gaak. *NO*.

If you're going to tweak printk, please make the darn thing take an integer first argument rather than a string first argument, so we can use clever macros to remove them at compile time rather than having them compiled in but not displaying.

This is one of those things buried down on my todo list. I think it can be converted incrementally

Going back to the original patch here, there are exactly two examples of what this brave new infrastructure would be used for:

Quoted 10 lines
drivers/kmsgtest/kmsgtest.c +10 -0
+/**
+ * message
+ * @0: Device number of device
+ * @1: Status of device
+ *
+ * Description:
+ * Information message about the status of our virtual msgtest device. The
+ * following values for the status parameter are available.
+ *
+ *   0 - Device is offline

drivers/kmsgtest/kmsgtest.c line 45

Quoted 10 lines
drivers/kmsgtest/kmsgtest.c +10 -0
+ *
+ *   1 - Device is online
+ *
+ *   2 - Device is broken
+ *
+ * User Response:
+ * If device is broken, replace it or fix it.
+ */
+
+KMSG_DOC(kmsgtest, 1, "device %x has status %i");

This is just a thought: why not have the actual kprintf say "Device number %x is %s" where %s is "online", "offline", or "broken"?

I.E. have the kernel output a user readable message in the first place.

Review

Quoted 13 lines

+/**
+ * message
+ * @0: Device number of device.
+ *
+ * Description:
+ * An operation has been performed on the msgtest device, but the device has + * not been set online. Therefore the operation failed.
+ *
+ * User Response:
+ * Operator should set device online. Issue "chccwdev -e <device number>".
+ */

(quoted diff) +2 -0
+
+KMSG_DOC(kmsgtest, 2, "device %x not online");

kprintf(LEVEL "device %x not online, try 'chccwdev -e %x'", devnum, devnum)

Rob

author
#6 lore raw
In reply to [email protected]

Hi Valdis,

Quoted 7 lines · On Wed, 2007-06-13 at 12:50 -0400, [email protected] wrote:
Quoted 3 lines · holzheu

They are used to that, because all other operating systems on that platform like z/OS, z/VM or z/VSE have message catalogs with detailed descriptions about the semantics of the messages.

25 years ago, I did OS/MVT and OS/VS1 for a living, so I know *all* about the infamous "What does IEF507E mean again?"...

:-)

Quoted 10 lines
Quoted 7 lines

In general we think, that also for Linux it is a good thing to have documentation for the most important kernel/driver messages. Even kernel hackers not always are aware of the meaning of kernel messages for components, which they don't know in detail. Most of the messages are self explaining but sometimes you get something like "Clocksource tsc unstable (delta = 7304132729 ns)" and you wonder if your system is going to explode.

This is probably best addressed by cleaning up the actual messages so they're a bit more informative.

Of course that would be good, too.

But I think, that we sometimes have the dilemma, that we want to keep the printks short, but also want to provide as much information as possible.

If the information is to big for the printk itself, because you would need 10 lines to explain what happened, wouldn't it be good to have a place where to put that information?

Quoted 34 lines
Quoted 3 lines

New macros KMSG_ERR(), KMSG_WARN(), etc. are defined, which have to be used in printk. These macros have as parameter the message number and are using a per c-file defined macro KMSG_COMPONENT.

Gaak. *NO*.

The *only* reason that the MVS and VM message catalogs worked at all is because each component had a message repository that went across *all* the source files - the instant you saw IEFnnns, you knew that IEF covered the job scheduler, nnn was a *unique* number, and s was a Severe/Warning/Info flag. IGG was always data management, and so on. This breaks horribly if you have 2 C files that define subtly different KMSG_COMPONENT values (or even worse, 2 or more duplicates).

[/usr/src/linux-2.6.22-rc4-mm2] find . -name '*.c' | wc -l 9959
[/usr/src/linux-2.6.22-rc4-mm2] find . -name '*.h' | wc -l 9933
[/usr/src/linux-2.6.22-rc4-mm2] find . -type d | wc -l 1736

You plan to maintain message uniqueness how?
[/usr/src/linux-2.6.22-rc4-mm2]1 find . -name '*.c' | sed -r 's?.*/([^/]*)?\1?' | sort | uniq -c | sort -nr | head

    105 setup.c
     90 irq.c
     66 time.c
     58 init.c
     50 inode.c
     39 io.c
     38 pci.c
     37 file.c
     32 signal.c
     32 ptrace.c
Quoted 3 lines

Looks like you're going to have to embed a lot of the path in that KMSG_COMPONENT to make it unique - and you want to keep that message under 80 or so chars total.

For each kernel component, like a device driver, we could have one KMSG_COMPONENT (e.g. "acpi", "pci", etc). Within that component the message ids have to be unique. A tool could check, if messages are unique within the kernel sources.

We could use something like a Documentation/kmsg-components file with a list of all component names using KMSG printks.

Quoted 8 lines
Quoted 7 lines

/**
* message
* @0: device number of device.
*
* Description:
* An operation has been performed on the msgtest device, but the
* device has not been set online. Therefore the operation failed

Quoted 7 lines

If you don't understand 'Device /dev/foo offline', this description doesn't help any. And that's true for *most* of the kernel messages already - if you don't understand the message already, a paragraph explanation isn't going to help much. Consider the average OOPS message, which contains stuff like 'EIP=0x..'. Telling the user that EIP means Execution Instruction Pointer isn't likely to help - if they knew what the pointer *did*, they'd probably already know EIP.

I agree with you, that most of the kernel messages do not need further documentation. But I am convinced, that there are plenty of printks, where additional documentation would be helpful.

Quoted 11 lines
Quoted 4 lines

*
* User Response:
* Operator should set device online.
* Issue "chccwdev -e <device number>".

And this is where the weakness of this scheme *really* hits. I've actually run into cases where an operator followed the listed "Operator Response" for a "device offline", and issued a 'VARY 0C0,ONLINE'. And then we got a flood of I/O errors because the previous shift downed the device because it was having issues. The response the operator *should* have done is "assign a different tape drive, like, oh maybe the operational ones at 0C1 through 0C4"...

I can understand your frustration here. But that's a general problem with documentation. You never can foresee everything.

But should this mean, that we shouldn't document anything?

Michael

#7 lore raw

Hi,

Quoted 3 lines · holzheu

If the information is to big for the printk itself, because you would need 10 lines to explain what happened, wouldn't it be good to have a place where to put that information?

I think if a message really has to be 10 lines long to be clear then it should just be 10 lines long. The keyword is "really".

Example: oops.

Signature · 1 line
Krzysztof Halasa
#8 lore raw
In reply to [email protected]
Quoted 7 lines · On Jun 13 2007 12:50, [email protected] wrote:
Quoted 7 lines

In general we think, that also for Linux it is a good thing to have documentation for the most important kernel/driver messages. Even kernel hackers not always are aware of the meaning of kernel messages for components, which they don't know in detail. Most of the messages are self explaining but sometimes you get something like "Clocksource tsc unstable (delta = 7304132729 ns)" and you wonder if your system is going to explode.

I suppose no. If it is going to explode, it will tell you :-)

/* fs/super.c */
printk("VFS: Busy inodes after unmount of %s. "
"Self-destruct in 5 seconds. Have a nice day...\n", sb->s_id);

Quoted 2 lines

This is probably best addressed by cleaning up the actual messages so they're a bit more informative.

Jan

#9 lore raw
In reply to holzheu

Makefile line 297

Quoted 5 lines · holzheu
Makefile +2 -3
-CHECK		= sparse
-
-CHECKFLAGS     := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ -Wbitwise $(CF)
+CHECK		= scripts/kmsg-doc check
+CHECKFLAGS	=

Ick. Don't touch those checking kernel with sparse.

author
#10 lore raw
Quoted 8 lines · Alexey Dobriyan

Makefile line 297

Quoted 5 lines · holzheu
Makefile +2 -3
-CHECK		= sparse
-
-CHECKFLAGS     := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ -Wbitwise $(CF)
+CHECK		= scripts/kmsg-doc check
+CHECKFLAGS	=

Ick. Don't touch those checking kernel with sparse.

Of course we don't want to have that in the vanilla kernel. I just wanted to show, how the checker tool has to be used.

Michael

#11 lore raw
In reply to holzheu
Quoted 14 lines · holzheu

Current prototype implementation:
=================================

The structure of a kernel message is: <component>.<msg number>: <msg>

* component: Name of the kernel or driver component e.g. "pci", "ide", etc.
* msg number: Within the component unique number of a kernel message.
* msg: printk message

New macros KMSG_ERR(), KMSG_WARN(), etc. are defined, which have to be used in printk. These macros have as parameter the message number and are using a per c-file defined macro KMSG_COMPONENT.

Quoted 8 lines

Example: Define message 2 in component "kmsgtest":

#define KMSG_COMPONENT "kmsgtest"

void f(void)

{
        printk(KMSG_ERR(1) "device %x not online\n", devno);
}

Ick, why are you ignoring what we have already with dev_printk() and friends? We are just finally getting developers to use that, I think it will be almost impossible to get people to change to something else, especially one that isn't even as "correct" as what dev_printk() offers you today, will be quite hard.

So, why not use what we already have and work off of it?

thanks,

greg k-h

author
#12 lore raw

Hi Greg,

Quoted 7 lines

Ick, why are you ignoring what we have already with dev_printk() and friends? We are just finally getting developers to use that, I think it will be almost impossible to get people to change to something else, especially one that isn't even as "correct" as what dev_printk() offers you today, will be quite hard.

So, why not use what we already have and work off of it?

dev_printk() and friends are great, since they already define something like KMSG_COMPONENT: The driver name.

But they unfortunately do not solve our problem. We need an identifier for a documented message in order to find the right description for a message.

We could add new dev_printk macros for KMSG printks:

#define dev_printk_kmsg(kmsg, dev, format, arg...) \
        printk(kmsg "%s: " format , (dev)->bus_id , ## arg)

#define dev_err_kmsg(dev, nr, format, arg...) \
dev_printk_kmsg(KMSG_ERR(nr) , dev , format , ## arg)

E.g. hub.c:
===========

#define KMSG_COMPONENT "hub"

dev_err_kmsg(&udev->dev, 1,

       "can't read configurations, error %d\n", err);

Which would result in:

hub.1 5-0:1.0: can't read configuration error 4711

Michael

#13 lore raw
Quoted 8 lines · holzheu

Hi Greg,

Quoted 6 lines

Ick, why are you ignoring what we have already with dev_printk() and friends? We are just finally getting developers to use that, I think it will be almost impossible to get people to change to something else, especially one that isn't even as "correct" as what dev_printk() offers you today, will be quite hard.

Quoted 4 lines
Quoted 1 line

So, why not use what we already have and work off of it?

dev_printk() and friends are great, since they already define something like KMSG_COMPONENT: The driver name.

They provide way more than that, they also provide the explicit device that is being discussed, as well as other things depending on the device.

So if you are going to do this, please use the already-in-place macros to hook into, don't try to get the driver authors to pick up something new and different, as it's going to be _very_ difficult, trust me...

thanks,

greg k-h

#14 lore raw
Quoted 8 lines · Greg KH
Quoted 2 lines

dev_printk() and friends are great, since they already define

something

Quoted 1 line

like KMSG_COMPONENT: The driver name.

They provide way more than that, they also provide the explicit device that is being discussed, as well as other things depending on the device.

Instead of keeping a list of kernel message ids somewhere, can you base this off the dev_printk() format argument?

You could keep a list of those somewhere, extracted from the kernel source, and correlate documentation with the _format_ instead of an arbitrarily assigned message number.

-- Dave

#15 lore raw
In reply to Greg KH
Quoted 13 lines · Greg KH
Quoted 11 lines · holzheu

Hi Greg,

Quoted 7 lines

Ick, why are you ignoring what we have already with dev_printk() and friends? We are just finally getting developers to use that, I think it will be almost impossible to get people to change to something else, especially one that isn't even as "correct" as what dev_printk() offers you today, will be quite hard.

So, why not use what we already have and work off of it?

dev_printk() and friends are great, since they already define something like KMSG_COMPONENT: The driver name.

Quoted 7 lines

They provide way more than that, they also provide the explicit device that is being discussed, as well as other things depending on the device.

So if you are going to do this, please use the already-in-place macros to hook into, don't try to get the driver authors to pick up something new and different, as it's going to be _very_ difficult, trust me...

absolutely... I'm currently trying to convince everyone to get a generic netdevice-centric printk macro (dev_printk does not print the interface name, which the netdevice can't store locally since it can change) in and even that is a pain :)

Auke

#16 lore raw
In reply to Greg KH
Quoted 9 lines · Greg KH
Quoted 3 lines
Quoted 1 line

So, why not use what we already have and work off of it?

dev_printk() and friends are great, since they already define

something

Quoted 1 line

like KMSG_COMPONENT: The driver name.

They provide way more than that, they also provide the explicit device that is being discussed, as well as other things depending on the device.

dev_printk() and friends provide additional information for a printk that is related to a device. Not every printk is about a device, so I think Michaels proposal is orthorgonal to dev_printk. We definitly should have a dev_printk variant that uses the kmsg documentation tag AND we should have a normal printk variant as well that uses the kmsg tagging.

Quoted 3 lines

So if you are going to do this, please use the already-in-place macros to hook into, don't try to get the driver authors to pick up something new and different, as it's going to be _very_ difficult, trust me...

There is no doubt that it will be difficult to get a larger part of the developers use the kmsg scheme (e.g. see the reaction of Dave M.). We do not have the illusion that we can replace every single message in the kernel, nor do we think that it would make sense to do so. What we would do for s390 is to check each message in drivers/s390 and think hard if the message should be 1) removed, 2) replaced with a {dev_}printk_kmsg or 3) left as it is. Fortunatly for us there are not too many drivers for s390 ..

Signature · 4 lines
blue skies,
  Martin.

"Reality continues to ruin my life." - Calvin.
#17 lore raw
In reply to holzheu
Quoted 3 lines · holzheu

But they unfortunately do not solve our problem. We need an identifier for a documented message in order to find the right description for a message.

I believe it better to simply add __FILE__ & __LINE__ to the macro rather than some other externally specified unique identifier that adds developer overhead and easily gets stale.

If you go overboard, you could do something like:

#define dev_dbg(dev, format, arg...) \
dev_printk(KERN_DEBUG, dev, DBG_FMT format DBG_ARG, ## arg)

#define dev_err(dev, format, arg...) \
dev_printk(KERN_ERR, dev, PR_FMT format PR_ARG, ## arg)

etc...

and use config variables to control what additional info is printk'ed.

Go overboard without a life preserver and add __TIMESTAMP__ too.

/*
*/
#if defined DBG_FMT
#undef DBG_FMT
#endif
#if defined DBG_ARG
#undef DBG_ARG
#endif

#if defined DBG_FILE && defined DBG_FUNCTION && defined DBG_LINE
#define DBG_FMT "(%s:%s:%u) "
#define DBG_ARG , __FILE__ , __FUNCTION__ , __LINE__
#elif defined DBG_FILE && defined DBG_FUNCTION && !defined DBG_LINE #define DBG_FMT "(%s:%s) "
#define DBG_ARG , __FILE__ , __FUNCTION__
#elif defined DBG_FILE && !defined DBG_FUNCTION && defined DBG_LINE #define DBG_FMT "(%s:%u) "
#define DBG_ARG , __FILE__ , __LINE__
#elif defined DBG_FILE && !defined DBG_FUNCTION && !defined DBG_LINE #define DBG_FMT "(%s) "
#define DBG_ARG , __FILE__
#elif !defined DBG_FILE && defined DBG_FUNCTION && defined DBG_LINE #define DBG_FMT "(%s:%u) "
#define DBG_ARG , __FUNCTION__ , __LINE__
#elif !defined DBG_FILE && defined DBG_FUNCTION && !defined DBG_LINE #define DBG_FMT "(%s) "
#define DBG_ARG , __FUNCTION__
#elif !defined DBG_FILE && !defined DBG_FUNCTION && defined DBG_LINE #define DBG_FMT "(%u) "
#define DBG_ARG , __LINE__
#else
#define DBG_FMT
#define DBG_ARG
#endif

/*
*/

#if defined PR_FMT
#undef PR_FMT
#endif
#if defined PR_ARG
#undef PR_ARG
#endif

#if defined PR_FILE && defined PR_FUNCTION && defined PR_LINE
#define PR_FMT "(%s:%s:%u) "
#define PR_ARG , __FILE__ , __FUNCTION__ , __LINE__ #elif defined PR_FILE && defined PR_FUNCTION && !defined PR_LINE #define PR_FMT "(%s:%s) "
#define PR_ARG , __FILE__ , __FUNCTION__
#elif defined PR_FILE && !defined PR_FUNCTION && defined PR_LINE #define PR_FMT "(%s:%u) "
#define PR_ARG , __FILE__ , __LINE__
#elif defined PR_FILE && !defined PR_FUNCTION && !defined PR_LINE #define PR_FMT "(%s) "
#define PR_ARG , __FILE__
#elif !defined PR_FILE && defined PR_FUNCTION && defined PR_LINE #define PR_FMT "(%s:%u) "
#define PR_ARG , __FUNCTION__ , __LINE__
#elif !defined PR_FILE && defined PR_FUNCTION && !defined PR_LINE #define PR_FMT "(%s) "
#define PR_ARG , __FUNCTION__
#elif !defined PR_FILE && !defined PR_FUNCTION && defined PR_LINE #define PR_FMT "(%u) "
#define PR_ARG , __LINE__
#else
#define PR_FMT
#define PR_ARG
#endif

#18 lore raw
Quoted 3 lines · Joe Perches

I believe it better to simply add __FILE__ & __LINE__ to the macro rather than some other externally specified unique identifier that adds developer overhead and easily gets stale.

There's been plenty of times I've wished for that. Now if we just found a way to do something sane for drivers/net/wireless/mac80211/iwlwifi/base.c ;)

Of course, I'm probably atypical - for me, kernel messages come in two forms, the kind that are immediately obvious at first reading, and the kind that you end up having to look at the code and wonder WTF? it was doing inside that never-should-happen-on-this-hardware while loop in the first place... :)

#19 lore raw
Quoted 9 lines · [email protected] wrote:
Quoted 3 lines · Joe Perches

I believe it better to simply add __FILE__ & __LINE__ to the macro rather than some other externally specified unique identifier that adds developer overhead and easily gets stale.

There's been plenty of times I've wished for that. Now if we just found a way to do something sane for drivers/net/wireless/mac80211/iwlwifi/base.c ;)

Why don't you just redefine the KERN_* macros which is used by almost every printk in the system to contain said __FILE__ and __LINE__, then?

-hpa

#20 lore raw
In reply to Joe Perches
Quoted 6 lines · Joe Perches
Quoted 3 lines · holzheu

But they unfortunately do not solve our problem. We need an identifier for a documented message in order to find the right description for a message.

I believe it better to simply add __FILE__ & __LINE__ to the

Erk, no. That bloats the kernel to the max. Especially since kernel files are compiled with a rather longish path.

Jan

#21 lore raw
In reply to holzheu
Quoted 130 lines · On Wed, 13 Jun 2007 17:06:57 +0200 holzheu <[email protected]> wrote:

Greetings,

The operation of a Linux system sometimes requires to decode the meaning of a specific kernel message, e.g. an error message of a driver. Especially on our mainframe zSeries platform system administrators want to have descriptions for Linux kernel messages. They are used to that, because all other operating systems on that platform like z/OS, z/VM or z/VSE have message catalogs with detailed descriptions about the semantics of the messages.

In general we think, that also for Linux it is a good thing to have documentation for the most important kernel/driver messages. Even kernel hackers not always are aware of the meaning of kernel messages for components, which they don't know in detail. Most of the messages are self explaining but sometimes you get something like "Clocksource tsc unstable (delta = 7304132729 ns)" and you wonder if your system is going to explode.

Unfortunately currently there is no general infrastructure in the Linux kernel for the documentation of messages. I worked on a proposal, how that could be implemented in an easy way using the already existing kernel-doc infrastructure and using printk. The proposal is as follows

1. We use message identifiers in order to map messages to message descriptions. A message identifier consists out of a component name and within that component unique message number.

2. Messages and message descriptions are maintained together in the kernel sources in order to keep them up to date. Messages catalog are generated automatically for exactly one kernel level.

3. A special tool checks, if messages are not documented or if there are message descriptions without corresponding messages.

4. We use the already existing kernel-doc tool to generate an online message catalog or e.g. a pdf for offline documentation.

Current prototype implementation:
=================================

The structure of a kernel message is: <component>.<msg number>: <msg>

* component: Name of the kernel or driver component e.g. "pci", "ide", etc.
* msg number: Within the component unique number of a kernel message.
* msg: printk message

New macros KMSG_ERR(), KMSG_WARN(), etc. are defined, which have to be used in printk. These macros have as parameter the message number and are using a per c-file defined macro KMSG_COMPONENT.

Example: Define message 2 in component "kmsgtest":

#define KMSG_COMPONENT "kmsgtest"

void f(void)

{
        printk(KMSG_ERR(1) "device %x not online\n", devno);
}

The output of that kernel message would be:
"kmsgtest.1: device 4711 not online"

The messages have to be documented within the C source file in the following way:

/**
* message
* @0: device number of device.
*
* Description:
* An operation has been performed on the msgtest device, but the
* device has not been set online. Therefore the operation failed *
* User Response:
* Operator should set device online.
* Issue "chccwdev -e <device number>".
*
*/

KMSG_DOC(kmsgtest, 2, "Device %x not online");

I created a patch for the kernel-doc tool so it can be used to generate a catalog of all kernel messages:

Quoted 2 lines
Quoted 2 lines

kernel-doc -man kmsgtest.c > kmsgtest.2
man ./kmsgtest.2

# Kernel API(9) Linux Messages Kernel API(9)
#
# MESSAGE:
# kmsgtest.2: "device %x not online"
#
# Parameter:
# 1 Device number of device.
#
# Description:
# An operation has been performed on the msgtest device, but
# the device has not been set online. Therefore the operation failed.
#
# User Response:
# Operator should set device online.
# Issue "chccwdev -e <device number>".
#
# May 2007 kmsgtest.2 Kernel API(9)

A nice thing would be to include the online kernel message catalog in the kernel rpm. One possibility for that would be to have one man page per message. If an operator finds the message kmsgtest.2 in var/log/messages and wants to know what the message means, he simply issues "man kmsgtest.2" and gets the description.

To ensure that all messages are documented and there are no message descriptions without corresponding messages, a checker tool is provided. To enable message checking, in the toplevel Makefile the following has to be added:

CHECK = scripts/kmsg_check.pl check

To enable message checking during kernel build, the "C" option has to be used:

Quoted 1 line
Quoted 1 line

make modules C=1

  CHK     include/linux/version.h
  CHK     include/linux/utsrelease.h
  CHECK   drivers/kmsgtest/kmsgtest.c

drivers/kmsgtest/kmsgtest.c: Missing description for: kmsgtest.1 drivers/kmsgtest/kmsgtest.c: Description without message for: kmsgtest.3

Quoted 8 lines

Please note, that the patch for the kernel-doc and kmsg-doc tools is just a prototype and is neither complete nor perfect.

Michael

Your proposal is similar to one I made to some Japanese developers earlier this year. I was more modest, proposing that we

- add an enhanced printk

	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);

- An externally managed database will provide translations, diagnostics, remedial actions, etc, keyed off the msgid string

- Format and management of the msgid hasn't been clearly identified yet as far as I know. But all we really need is that each ID be unique, kernel-wide. We develop a simple tool which can check the whole tree for duplicated msgids.

- From a practical day-to-day POV what this means is that a person from the kernel-messages team will prepare a patch for your driver which adds the msgids and you the driver author should take care not to break the tagging.

This is deliberately designed to be minimum-impact upon general developers. We want a system in place where driver/fs/etc developers can concentrate on what they do, and kernel-message developers can as independently as possibe take care of what _they_ do.

- A project has started up and there is a mailing list (cc'ed here) and meetings are happening at the LF collaboration summit this week.

Please see https://lists.linux-foundation.org/mailman/listinfo/lf_kernel_messages
and don't miss the next conference call ;)

(argh. The lf_kernel_messages archives are subscriber-only and it could be that only members can post to it. Oh well. Please subscribe, and review the archives)

Quoted 527 lines
 Makefile                    |    5
 drivers/Makefile            |    1
 drivers/kmsgtest/Makefile   |    5
 drivers/kmsgtest/kmsgtest.c |   59 +++++++++++
 include/linux/kmsg.h        |   32 ++++++
 scripts/kernel-doc          |  116 +++++++++++++++++++++-
 scripts/kmsg-doc            |  231 ++++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 445 insertions(+), 4 deletions(-)

diff -Naur linux-2.6.21/Makefile linux-2.6.21-kmsg/Makefile

Makefile +2 -3
@@ -293,9 +293,8 @@
 DEPMOD		= /sbin/depmod
 KALLSYMS	= scripts/kallsyms
 PERL		= perl
-CHECK		= sparse
-
-CHECKFLAGS     := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ -Wbitwise $(CF)
+CHECK		= scripts/kmsg-doc check
+CHECKFLAGS	=
 MODFLAGS	= -DMODULE
 CFLAGS_MODULE   = $(MODFLAGS)
 AFLAGS_MODULE   = $(MODFLAGS)

diff -Naur linux-2.6.21/drivers/Makefile linux-2.6.21-kmsg/drivers/Makefile

drivers/Makefile +1 -0
@@ -8,6 +8,7 @@
 obj-$(CONFIG_PCI)		+= pci/
 obj-$(CONFIG_PARISC)		+= parisc/
 obj-$(CONFIG_RAPIDIO)		+= rapidio/
+obj-m				+= kmsgtest/
 obj-y				+= video/
 obj-$(CONFIG_ACPI)		+= acpi/
 # PnP must come after ACPI since it will eventually need to check if acpi

diff -Naur linux-2.6.21/drivers/kmsgtest/Makefile linux-2.6.21-kmsg/drivers/kmsgtest/Makefile

drivers/kmsgtest/Makefile +5 -0
@@ -0,0 +1,5 @@
+#
+# Makefile for kernel message test module
+#
+
+obj-m		+= kmsgtest.o

diff -Naur linux-2.6.21/drivers/kmsgtest/kmsgtest.c linux-2.6.21-kmsg/drivers/kmsgtest/kmsgtest.c

drivers/kmsgtest/kmsgtest.c +59 -0
@@ -0,0 +1,59 @@
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/kmsg.h>
+
+static int devno = 0x4711;
+static int status = 1;
+
+#define KMSG_COMPONENT "kmsgtest"
+
+static int __init kmsgtest_init(void)
+{
+	printk(KMSG_INFO(1) "device %x has status %i\n", devno, status);
+	printk(KMSG_ERR(2) "device %x not online\n", devno);
+
+	return 0;
+}
+
+static void __exit kmsgtest_exit(void)
+{
+	printk("kmsgtest module exit\n");
+}
+
+module_init(kmsgtest_init);
+module_exit(kmsgtest_exit);
+
+/**
+ * message
+ * @0: Device number of device
+ * @1: Status of device
+ *
+ * Description:
+ * Information message about the status of our virtual msgtest device. The
+ * following values for the status parameter are available.
+ *
+ *   0 - Device is offline
+ *
+ *   1 - Device is online
+ *
+ *   2 - Device is broken
+ *
+ * User Response:
+ * If device is broken, replace it or fix it.
+ */
+
+KMSG_DOC(kmsgtest, 1, "device %x has status %i");
+
+/**
+ * message
+ * @0: Device number of device.
+ *
+ * Description:
+ * An operation has been performed on the msgtest device, but the device has
+ * not been set online. Therefore the operation failed.
+ *
+ * User Response:
+ * Operator should set device online. Issue "chccwdev -e <device number>".
+ */
+
+KMSG_DOC(kmsgtest, 2, "device %x not online");

diff -Naur linux-2.6.21/include/linux/kmsg.h linux-2.6.21-kmsg/include/linux/kmsg.h

include/linux/kmsg.h +32 -0
@@ -0,0 +1,32 @@
+#ifndef _LINUX_KMSG_H
+#define _LINUX_KMSG_H
+
+#ifdef __KMSG_CHECKER
+
+#define KMSG_EMERG(num)   __KMSG_CHECK(EMERG, num)
+#define KMSG_ALERT(num)   __KMSG_CHECK(ALERT, num)
+#define KMSG_CRIT(num)    __KMSG_CHECK(CRIT, num)
+#define KMSG_ERR(num)     __KMSG_CHECK(ERR, num)
+#define KMSG_WARNING(num) __KMSG_CHECK(WARNING, num)
+#define KMSG_NOTICE(num)  __KMSG_CHECK(NOTICE, num)
+#define KMSG_INFO(num)    __KMSG_CHECK(INFO, num)
+#define KMSG_DEBUG(num)   __KMSG_CHECK(DEBUG, num)
+
+#define KMSG_DOC(comp, num, str) __KMSG_DOC(comp, num, str)
+
+#else
+
+#define KMSG_EMERG(num)   KERN_EMERG KMSG_COMPONENT "." #num ": "
+#define KMSG_ALERT(num)   KERN_ALERT KMSG_COMPONENT "." #num ": "
+#define KMSG_CRIT(num)    KERN_CRIT KMSG_COMPONENT "." #num ": "
+#define KMSG_ERR(num)     KERN_ERR KMSG_COMPONENT "." #num ": "
+#define KMSG_WARNING(num) KERN_WARNING KMSG_COMPONENT "." #num ": "
+#define KMSG_NOTICE(num)  KERN_NOTICE KMSG_COMPONENT "." #num ": "
+#define KMSG_INFO(num)    KERN_INFO KMSG_COMPONENT "." #num ": "
+#define KMSG_DEBUG(num)   KERN_DEBUG KMSG_COMPONENT "." #num ": "
+
+#define KMSG_DOC(comp, num, str)
+
+#endif /* __KMSG_CHECKER */
+
+#endif /* _LINUX_KMSG_H */

diff -Naur linux-2.6.21/scripts/kernel-doc linux-2.6.21-kmsg/scripts/kernel-doc

scripts/kernel-doc +115 -1
@@ -256,7 +256,7 @@
 my $in_doc_sect;
 
 #declaration types: can be
-# 'function', 'struct', 'union', 'enum', 'typedef'
+# 'function', 'struct', 'union', 'enum', 'typedef', 'message'
 my $decl_type;
 
 my $doc_special = "\@\%\$\&";
@@ -1163,6 +1163,55 @@
     output_section_text(@_);
 }
 
+##
+# output message in text
+sub output_message_text(%) {
+	my %args = %{$_[0]};
+	my ($parameter);
+
+	print $args{'id'}.": \"".$args{'message'}."\"\n\n";
+
+	print "Parameters:\n\n";
+	foreach $parameter (@{$args{'parameterlist'}}) {
+		($parameter =~ /^#/) && next;
+		my $parameter_name = $parameter;
+		$parameter_name =~ s/\[.*//;
+		($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
+		print "$parameter\n\t";
+		print $args{'parameterdescs'}{$parameter_name}."\n";
+	}
+	output_section_text(@_);
+}
+
+##
+# output message in man
+sub output_message_man(%) {
+	my %args = %{$_[0]};
+	my ($parameter, $section);
+
+	print ".TH \"$args{'module'}\" 9 \"".$args{'id'}."\" \"$man_date\" \"Linux Messages\" LINUX\n";
+
+	print ".SH MESSAGE\n";
+	print $args{'id'}.": "."\"".$args{'message'}."\"\n";
+
+	print ".SH Parameters\n";
+	foreach $parameter (@{$args{'parameterlist'}}) {
+		($parameter =~ /^#/) && next;
+
+		my $parameter_name = $parameter;
+		$parameter_name =~ s/\[.*//;
+
+		($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
+		print ".IP \"".$parameter."\" 12\n";
+		output_highlight($args{'parameterdescs'}{$parameter_name});
+	}
+	foreach $section (@{$args{'sectionlist'}}) {
+		print ".SH \"$section\"\n";
+		output_highlight($args{'sections'}{$section});
+		print "\n";
+	}
+}
+
 #output sections in text
 sub output_section_text(%) {
     my %args = %{$_[0]};
@@ -1607,6 +1656,69 @@
 		       });
 }
 
+sub create_parameterlist_msg($$) {
+	my $args = shift;
+	my $file = shift;
+	my $splitter = "%";
+	my $type;
+	my $param = 0;
+	my $first = 1;
+
+	# temporarily replace commas
+	while ($args =~ /(\([^\),]+),/) {
+		$args =~ s/(\([^\),]+),/$1#/g;
+	}
+
+	foreach my $arg (split($splitter, $args)) {
+		if ($first) {
+			$first = 0;
+			next;
+		}
+		$type = substr($arg,0,1);
+
+		# XXX introduce better type checking
+
+		push_parameter($param, $type, $file);
+		$param += 1;
+	}
+}
+
+##
+# takes a message prototype and the name of the current file being
+# processed and spits out all the details stored in the global
+# arrays/hashes.
+sub dump_message($$) {
+	my $x = shift;
+	my $file = shift;
+
+	if ($x =~/(KMSG_DOC)\(\s*(\w+)\s*\,\s*(\w+)\s*\,\s*\"(.*)\"\)/) {
+	$declaration_name = "$2.$3";
+	my $members = $4;
+	# strip comments:
+	$members =~ s/\/\*.*?\*\///gos;
+
+	create_parameterlist_msg($members, $file);
+
+	output_declaration($declaration_name,
+			   'message',
+			   {'message' => $members,
+			    'id' => $declaration_name,
+			    'module' => $modulename,
+			    'parameterlist' => \@parameterlist,
+			    'parameterdescs' => \%parameterdescs,
+			    'parametertypes' => \%parametertypes,
+			    'sectionlist' => \@sectionlist,
+			    'sections' => \%sections,
+			    'purpose' => $declaration_purpose,
+			    'type' => $decl_type
+			   });
+	}
+	else {
+		print STDERR "Error(${file}:$.): Cannot parse message!\n";
+		++$errors;
+	}
+}
+
 sub process_file($);
 
 # Read the file that maps relative names to absolute names for
@@ -1782,6 +1894,8 @@
 		    $decl_type = 'enum';
 		} elsif ($identifier =~ m/^typedef/) {
 		    $decl_type = 'typedef';
+		} elsif ($identifier =~ m/^message/) {
+		    $decl_type = 'message';
 		} else {
 		    $decl_type = 'function';
 		}

diff -Naur linux-2.6.21/scripts/kmsg-doc linux-2.6.21-kmsg/scripts/kmsg-doc

scripts/kmsg-doc +231 -0
@@ -0,0 +1,231 @@
+#!/usr/bin/perl
+#
+# Tool to check kernel messages
+#
+# Can be used in toplevel Linux Makefile in the following way:
+#
+# CHECK = scripts/kmsg-doc check
+# CHECKFLAGS =
+#
+# Note: This is just a prototype and neither perfect nor complete!
+#
+# Copyright (C) IBM Corp. 2007
+# Author(s): Michael Holzheu <[email protected]>
+#
+
+sub create_message($$$$$)
+{
+	my ($sev, $component, $number, $text, $params) = @_;
+
+	$text =~ s/\\n//; # remove trailing newline character
+	$message_id = "$component.$number";
+	$messages{$message_count}->{'ID'} = $message_id;
+	$messages{$message_count}->{'COMP'} = $component;
+	$messages{$message_count}->{'NR'} = $number;
+	$messages{$message_count}->{'MSG'} = $text;
+	$messages{$message_count}->{'SEV'} = $sev;
+
+	@parms = split(/[\s]*,[\s]*/,$params);
+	$parm_count = 0;
+	foreach $parm (@parms) {
+		if (!($parm eq "")) {
+			$messages{$message_count}->{'PARM_NAME'}->{$parm_count} = $parm;
+			$parm_count += 1;
+		}
+	}
+	$messages{$message_count}->{'PARM_COUNT'} = $parm_count;
+	$message_count += 1;
+}
+
+sub get_msgs($)
+{
+	my ($filename)=@_;
+
+	$message_count = 0;
+	open(FD, $filename);
+	my @lines=<FD>;
+	foreach $line (@lines) {
+		if ($line =~ /\s*printk\([\s]*__KMSG_CHECK\((.*)\,[\s]*(.*)\)[\s]*"(.*)"[\s]*(.*)[\s]*\);/) {
+			create_message($1, $component, $2, $3, $4);
+		}
+	}
+}
+
+sub get_descriptions($)
+{
+	my ($filename)=@_;
+	my $desc_start;
+
+	$description_count = 0;
+	$desc_start = 0;
+	open(FD, $filename);
+	my @lines=<FD>;
+	foreach $line (@lines) {
+		if ($line =~ /#define [\s]*KMSG_COMPONENT [\s]*"(.*)"/) {
+			$component = $1;
+		}
+		if ($line =~ /\s*\/\*\*$/) {
+			$msg_start = 1;
+			$parm_count = 0;
+			next;
+		}
+		if (($msg_start == 1) && ($line =~ / \* message/)) {
+			$desc_start = 1;
+			next;
+		}
+		if ($line =~ / \*\//) {
+			$desc_start = 0;
+			next;
+		}
+		if ($desc_start == 1) {
+			$descriptions{$description_count}->{'DESC'} .= "$line";
+			next;
+		}
+		if ($line =~
+			/\s*KMSG_DOC\(\s*(.*)\s*\,\s*(.*)\s*\,\s*\"(.*)\"\s*\);/) {
+			my $param_count = 0;
+			my $first = 1;
+			my $type;
+
+			$descriptions{$description_count}->{'ID'} = "$1\.$2";
+			$descriptions{$description_count}->{'COMP'} = "$1";
+			$descriptions{$description_count}->{'NR'} = "$2";
+			$descriptions{$description_count}->{'MSG'} = "$3";
+			foreach my $arg (split("%", $3)) {
+				if ($first) {
+					$first = 0;
+					next;
+				}
+				$type = substr($arg, 0, 1);
+				$descriptions{$description_count}->{'PARM_TYPE'}->{$param_count} = $type;
+				$param_count += 1;
+			}
+			$descriptions{$description_count}->{'PARM_COUNT'} = $param_count;
+			$description_count += 1;
+		}
+	}
+}
+
+sub print_messages()
+{
+	for ($i = 0; $i < $message_count; $i++) {
+		print "MESSAGE: $messages{$i}->{'ID'}\n";
+	}
+}
+
+sub print_descriptions($)
+{
+	my ($message_id)=@_;
+
+	for ($i = 0; $i < $description_count; $i++) {
+		if (($descriptions{$i}->{'ID'} eq $message_id) || $message_id eq "all") {
+			print "==============================================================================\n";
+			print "\[$descriptions{$i}->{'COMP'}\.$descriptions{$i}->{'NR'}\] $descriptions{$i}->{'MSG'}\n";
+			print "\n";
+			print "Parameters:\n";
+			for ($j = 0; $j < $descriptions{$i}->{'PARM_COUNT'}; $j++) {
+				print " $descriptions{$i}->{'PARM_TYPE'}->{$j}: $descriptions{$i}->{'PARM_DESC'}->{$j}\n";
+			}
+			print "\n";
+			print "Description:\n";
+			print "$descriptions{$i}->{'DESC'}\n";
+			print "==============================================================================\n";
+		}
+	}
+}
+
+sub check_messages($)
+{
+	my ($filename)=@_;
+
+	for ($i = 0; $i < $message_count; $i++) {
+		$found = 0;
+		for ($j = 0; $j < $description_count; $j++) {
+			if ($messages{$i}->{'ID'} eq $descriptions{$j}->{'ID'}) {
+				$found = 1;
+				last;
+			}
+		}
+		if (!$found) {
+			print STDERR "$filename: Missing description for: $messages{$i}->{'ID'}\n";
+		}
+	}
+	for ($i = 0; $i < $description_count; $i++) {
+		$found = 0;
+		for ($j = 0; $j < $message_count; $j++) {
+			if ($messages{$j}->{'ID'} eq $descriptions{$i}->{'ID'}) {
+				$found = 1;
+				last;
+			}
+		}
+		if (!$found) {
+			print STDERR "$filename: Description without message for: $descriptions{$i}->{'ID'}\n";
+		}
+	}
+}
+
+sub print_templates()
+{
+
+	for ($i = 0; $i < $message_count; $i++) {
+		$found = 0;
+		for ($j = 0; $j < $description_count; $j++) {
+			if ($messages{$i}->{'ID'} eq $descriptions{$j}->{'ID'}) {
+				$found = 1;
+			}
+		}
+		if (!$found) {
+			print "/**\n";
+			print " * message\n";
+			for ($k = 0; $k < $messages{$i}->{'PARM_COUNT'}; $k++) {
+				print " * \@$k: $messages{$i}->{'PARM_NAME'}->{$k}\n";
+			}
+			print " *\n";
+			print " * Description:\n";
+			print " *\n";
+			print " * User Response:\n";
+			print " */\n";
+			print "\n";
+			print "KMSG_DOC($messages{$i}->{'COMP'}, $messages{$i}->{'NR'}, \"$messages{$i}->{'MSG'}\");\n"
+		}
+	}
+}
+
+sub usage()
+{
+	print "USAGE: kmsg_tool print | check <file>\n";
+	exit 1;
+}
+
+$option = shift;
+
+if ($option eq "check") {
+	$gcc_options = "-E -D __KMSG_CHECKER ";
+	do {
+		$filename = $tmp;
+		$tmp = shift;
+		$tmp =~ s/\(/\\\(/;
+		$tmp =~ s/\)/\\\)/;
+		$gcc_options .= " $tmp";
+	} while (!($tmp eq ""));
+
+	$gcc_options =~ s/-Wbitwise//; # XXX hack to remove -Wbitwise CHECKFLAG
+	$gcc_options =~ s/-D__STDC__//; # XXX hack to remove -D__STDC__
+	$tmp_file = "$filename.msg";
+	system("gcc $gcc_options > $tmp_file");
+	get_descriptions($filename);
+	get_msgs($tmp_file);
+	check_messages($filename);
+	print_templates();
+	system("rm $tmp_file");
+} elsif ($option eq "print") {
+	$filename = shift;
+	do {
+		print STDERR "Processing: $filename\n";
+		get_descriptions($filename);
+		print_descriptions("all");
+		$filename = shift;
+	} while (!($filename eq ""));
+} else {
+	usage();
+}
#22 lore raw
Quoted 2 lines · Andrew Morton

Your proposal is similar to one I made to some Japanese developers earlier this year.

Interesting. Our requirement comes from Japan as well.

Quoted 5 lines

I was more modest, proposing that we

- add an enhanced printk

        xxprintk(msgid, KERN_ERR "some text %d\n", some_number);

Some kind of id is always required to be able to identify the message. The task is to make the tagging as simple as possible.

Quoted 2 lines

- An externally managed database will provide translations, diagnostics, remedial actions, etc, keyed off the msgid string

There I have my doubts if this can work. If you keep an external database with the additional information about the messages the kernel code and the database will get out of sync. Thats why we advocate the kernel-doc style approach: the message catalogue will always be in sync because it is delivered with the kernel.

Quoted 4 lines

- Format and management of the msgid hasn't been clearly identified yet as far as I know. But all we really need is that each ID be unique, kernel-wide. We develop a simple tool which can check the whole tree for duplicated msgids.

This is a paint point with Michaels approach as well. The KMSG_COMPONENT defines are rather ugly. A clever way how to generate kernel unique IDs would be nice. Intervention required..

Quoted 9 lines

- From a practical day-to-day POV what this means is that a person from the kernel-messages team will prepare a patch for your driver which adds the msgids and you the driver author should take care not to break the tagging.

This is deliberately designed to be minimum-impact upon general developers. We want a system in place where driver/fs/etc developers can concentrate on what they do, and kernel-message developers can as independently as possibe take care of what _they_ do.

With the KMSG approach all that needs to be done is to replace the KERN_xxx with KMSG_xxx(<number) and to add a comment at the end of the file.

Quoted 8 lines

- A project has started up and there is a mailing list (cc'ed here) and meetings are happening at the LF collaboration summit this week.

Please see https://lists.linux-foundation.org/mailman/listinfo/lf_kernel_messages
and don't miss the next conference call ;)

(argh. The lf_kernel_messages archives are subscriber-only and it could be that only members can post to it. Oh well. Please subscribe, and review the archives)

Very interesting. We did not know about this project. Thanks for the hint.

Signature · 4 lines
blue skies,
  Martin.

"Reality continues to ruin my life." - Calvin.
#23 lore raw
In reply to Andrew Morton

<snip>

Quoted 6 lines

Your proposal is similar to one I made to some Japanese developers earlier this year. I was more modest, proposing that we

- add an enhanced printk

	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);

Maybe a stupid idea but why do we want to assign these numbers by hand?
I can imagine it could introduce collisions when merging tons of patches with new messages... Wouldn't it be better to compute say, 8-byte hash from the message and use it as it's identifier? We could do this automagically at compile time. I know it also has it's problems - you fix a spelling and the message gets a different id and you have to update translation/documentation catalogue but maybe that could be solved too...

Honza

author
#24 lore raw
Quoted 5 lines · Jan Kara

<snip>

Quoted 3 lines

Your proposal is similar to one I made to some Japanese developers earlier this year. I was more modest, proposing that we

Quoted 8 lines
Quoted 3 lines

- add an enhanced printk

	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);

Maybe a stupid idea but why do we want to assign these numbers by hand?
I can imagine it could introduce collisions when merging tons of patches with new messages... Wouldn't it be better to compute say, 8-byte hash from the message and use it as it's identifier? We could do this automagically at compile time.

Of course automatically generated message numbers would be great and something like:

hub.4a5bcd77: Detected some problem.

looks acceptable for me.

We could generate the hash using the format string of the printk. Since we specify the format string also in KMSG_DOC, the hash for the KMSG_DOC and the printk should match and we have the required link between printk and description.

So technically that's probably doable.

Problems are:

* hashes are not unique
* We need an additional preprocessor step
* The might be people, who find 8 character hash values ugly in printks

The big advantage is, that we do not need to maintain message numbers.

Quoted 4 lines

I know it also has it's problems - you
fix a spelling and the message gets a different id and you have to update translation/documentation catalogue but maybe that could be solved too...

Since in our approach the message catalog is created automatically for exactly one kernel and the message catalog belongs therefore to exactly one kernel, I think the problem of changing error numbers is not too big.

Michael

author
#25 lore raw
Quoted 15 lines · holzheu
Quoted 5 lines · Jan Kara

<snip>

Quoted 3 lines

Your proposal is similar to one I made to some Japanese developers earlier this year. I was more modest, proposing that we

Quoted 8 lines
Quoted 3 lines

- add an enhanced printk

	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);

Maybe a stupid idea but why do we want to assign these numbers by hand?
I can imagine it could introduce collisions when merging tons of patches with new messages... Wouldn't it be better to compute say, 8-byte hash from the message and use it as it's identifier? We could do this automagically at compile time.

Quoted 4 lines

Of course automatically generated message numbers would be great and something like:

hub.4a5bcd77: Detected some problem.

Sorry, I first read: 8 characters not 8 bytes.

Indeed, "hub.d41d8cd98f00b204: Detected some problem." ... does not look like very beautiful.

But maybe also 4 bytes would be enough, since the hash only has to be unique within one component e.g. "hub".

Michael

#26 lore raw
Quoted 21 lines · holzheu
Quoted 15 lines · holzheu
Quoted 5 lines · Jan Kara

<snip>

Quoted 3 lines

Your proposal is similar to one I made to some Japanese developers earlier this year. I was more modest, proposing that we

Quoted 8 lines
Quoted 3 lines

- add an enhanced printk

	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);

Maybe a stupid idea but why do we want to assign these numbers by hand?
I can imagine it could introduce collisions when merging tons of patches with new messages... Wouldn't it be better to compute say, 8-byte hash from the message and use it as it's identifier? We could do this automagically at compile time.

Quoted 4 lines

Of course automatically generated message numbers would be great and something like:

hub.4a5bcd77: Detected some problem.

Quoted 7 lines

Sorry, I first read: 8 characters not 8 bytes.

Indeed, "hub.d41d8cd98f00b204: Detected some problem." ... does not look like very beautiful.

But maybe also 4 bytes would be enough, since the hash only has to be unique within one component e.g. "hub".

It depends how large components you expect. For example for 10000
messages there is already 1% probability of collision so it means sooner or later we are going to hit it... For 1000 messages the probability is roughly 0.1% which is still not so small that I'd be comfortable with it. On the other hand we could use something like gperf to generate perfect hashing function and then we don't have to worry about colisions.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SuSE CR Labs
author
#27 lore raw

On Thu, 2007-06-14 at 14:26 +0200, Jan Kara wrote:

<snip>

Quoted 6 lines
Quoted 2 lines

But maybe also 4 bytes would be enough, since the hash only has to be unique within one component e.g. "hub".

It depends how large components you expect. For example for 10000
messages there is already 1% probability of collision so it means sooner or later we are going to hit it... For 1000 messages the probability is roughly 0.1% which is still not so small that I'd be comfortable with it.

I wouldn't expect, that a "normal" component like a device driver will have more than 100 documented messages.

Collisions are ugly, but will not hurt much. The operator will see for a message two possible descriptions. Normally it should be possible to figure out manually which description is the right one.

Michael

#28 lore raw
In reply to holzheu
Quoted 42 lines · holzheu
Quoted 5 lines · Jan Kara

<snip>

Quoted 3 lines

Your proposal is similar to one I made to some Japanese developers earlier this year. I was more modest, proposing that we

Quoted 8 lines
Quoted 3 lines

- add an enhanced printk

	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);

Maybe a stupid idea but why do we want to assign these numbers by hand?
I can imagine it could introduce collisions when merging tons of patches with new messages... Wouldn't it be better to compute say, 8-byte hash from the message and use it as it's identifier? We could do this automagically at compile time.

Of course automatically generated message numbers would be great and something like:

hub.4a5bcd77: Detected some problem.

looks acceptable for me.

We could generate the hash using the format string of the printk. Since we specify the format string also in KMSG_DOC, the hash for the KMSG_DOC and the printk should match and we have the required link between printk and description.

So technically that's probably doable.

Problems are:

* hashes are not unique
* We need an additional preprocessor step
* The might be people, who find 8 character hash values ugly in printks

The big advantage is, that we do not need to maintain message numbers.

Quoted 4 lines

I know it also has it's problems - you
fix a spelling and the message gets a different id and you have to update translation/documentation catalogue but maybe that could be solved too...

Quoted 4 lines

Since in our approach the message catalog is created automatically for exactly one kernel and the message catalog belongs therefore to exactly one kernel, I think the problem of changing error numbers is not too big.

We just had a meeting with the Japanese and several other participants from the vendor and community side and came up with a potential proposal that is similar to many things discussed here. It has the benefit that it seems implementable and low/no overhead on most kernel developers.

The basic proposal is to use a tool, run by the kernel Makefile to extract kernel messages from either the kernel source or the .i files (both have advantages, although I prefer the source to the .i file since it 1) gets all messages and 2) is probably a little quicker with less impact to the standard kernel make.

These messages would be stored in a file in the source tree, e.g. usr/src/linux/Translations/English. As each message is added to that file, we calculate, say, an MD5 sum of the printk (dev_printk, sdev_printk, etc.) string, and the text file ultimately contains:

MD5 Checksum of text; the printk text itself, the File name, the line number.

The checksum is run over just the printk. We definitely would not include the line number since the line number is too volatile. Including the file name in the hash *might* help disambiguate the hash a bit better in the case of duplicates, but there was some debate that duplicates might be better handled in other ways.

Andrew mentioned a mechanism for adding a subsystem tag or other tag which helps disambiguate the message, either in the message file or in the end user documentation (e.g. the Message Pedia/mPedia that the Japanese have already created with ~350 messages, and a total of ~700 targetted by the end of the year).

That tag could be appended to the beginning of the printk, to the end of the printk, or even in a formatted comment at the end of the printk that the tool could extract.

Then, the translations could be managed by anyone outside of the normal/ core kernel community, by simply creating a translation file, e.g. usr/src/linux/Translations/Japanese, which contained the MD5 sum, the translated message, the file name and line number (the last two redundent perhaps but informational, and automatically generated if possible).

The files in the Translations directory could be uesd as the unique keys for an external database (such as the Message Pedia, vendors or distributions help pages, etc.) to help look up and explain root cause of a problem. The key property here is that the MD5 sum becomes the key to all database entries to look up that key.

Further, yet another kernel config option could allow distros to output the calculated MD5 sum to be printed, much like we do with timestamps today.

End result is that these in-kernel message catalogs for translation are updated automatically (mostly no kernel developer changes needed) and the translations can be maintained by anyone who is interested.

On the topic of MD5 collisions, using a disambiguating tag would be a simple addition for the few cases where that happens, the tool could be educated to use that tag in the calculation of the MD5 sum, and we have a 98% solution which impacts <1% of the kernel developers.

Folks present for this discussion included Ted T'so, James Bottomley, several of the key Japenese folks interested in using this for debugging, and reps from several vendors who would find this sort of info useful for their folks supporting Linux in the field.

Comments?

gerrit

#29 lore raw
Quoted 105 lines · Gerrit Huizenga
Quoted 36 lines · holzheu
Quoted 5 lines · Jan Kara

<snip>

Quoted 3 lines

Your proposal is similar to one I made to some Japanese developers earlier this year. I was more modest, proposing that we

Quoted 8 lines
Quoted 3 lines

- add an enhanced printk

	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);

Maybe a stupid idea but why do we want to assign these numbers by hand?
I can imagine it could introduce collisions when merging tons of patches with new messages... Wouldn't it be better to compute say, 8-byte hash from the message and use it as it's identifier? We could do this automagically at compile time.

Of course automatically generated message numbers would be great and something like:

hub.4a5bcd77: Detected some problem.

looks acceptable for me.

We could generate the hash using the format string of the printk. Since we specify the format string also in KMSG_DOC, the hash for the KMSG_DOC and the printk should match and we have the required link between printk and description.

So technically that's probably doable.

Problems are:

* hashes are not unique
* We need an additional preprocessor step
* The might be people, who find 8 character hash values ugly in printks

The big advantage is, that we do not need to maintain message numbers.

Quoted 8 lines
Quoted 4 lines

I know it also has it's problems - you
fix a spelling and the message gets a different id and you have to update translation/documentation catalogue but maybe that could be solved too...

Since in our approach the message catalog is created automatically for exactly one kernel and the message catalog belongs therefore to exactly one kernel, I think the problem of changing error numbers is not too big.

We just had a meeting with the Japanese and several other participants from the vendor and community side and came up with a potential proposal that is similar to many things discussed here. It has the benefit that it seems implementable and low/no overhead on most kernel developers.

The basic proposal is to use a tool, run by the kernel Makefile to extract kernel messages from either the kernel source or the .i files (both have advantages, although I prefer the source to the .i file since it 1) gets all messages and 2) is probably a little quicker with less impact to the standard kernel make.

These messages would be stored in a file in the source tree, e.g. usr/src/linux/Translations/English. As each message is added to that file, we calculate, say, an MD5 sum of the printk (dev_printk, sdev_printk, etc.) string, and the text file ultimately contains:

MD5 Checksum of text; the printk text itself, the File name, the line number.

The checksum is run over just the printk. We definitely would not include the line number since the line number is too volatile. Including the file name in the hash *might* help disambiguate the hash a bit better in the case of duplicates, but there was some debate that duplicates might be better handled in other ways.

Andrew mentioned a mechanism for adding a subsystem tag or other tag which helps disambiguate the message, either in the message file or in the end user documentation (e.g. the Message Pedia/mPedia that the Japanese have already created with ~350 messages, and a total of ~700 targetted by the end of the year).

That tag could be appended to the beginning of the printk, to the end of the printk, or even in a formatted comment at the end of the printk that the tool could extract.

Then, the translations could be managed by anyone outside of the normal/ core kernel community, by simply creating a translation file, e.g. usr/src/linux/Translations/Japanese, which contained the MD5 sum, the translated message, the file name and line number (the last two redundent perhaps but informational, and automatically generated if possible).

The files in the Translations directory could be uesd as the unique keys for an external database (such as the Message Pedia, vendors or distributions help pages, etc.) to help look up and explain root cause of a problem. The key property here is that the MD5 sum becomes the key to all database entries to look up that key.

Further, yet another kernel config option could allow distros to output the calculated MD5 sum to be printed, much like we do with timestamps today.

End result is that these in-kernel message catalogs for translation are updated automatically (mostly no kernel developer changes needed) and the translations can be maintained by anyone who is interested.

On the topic of MD5 collisions, using a disambiguating tag would be a simple addition for the few cases where that happens, the tool could be educated to use that tag in the calculation of the MD5 sum, and we have a 98% solution which impacts <1% of the kernel developers.

Quoted 6 lines

Folks present for this discussion included Ted T'so, James Bottomley, several of the key Japenese folks interested in using this for debugging, and reps from several vendors who would find this sort of info useful for their folks supporting Linux in the field.

Comments?

For those of us who have not been in on these meetings, I think that some serious justifications are needed. The last paragraph began to go in that direction, but it needs to be more detailed and convincing. And "for debugging" doesn't cut it IMO.

Signature · 2 lines
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***
#30 lore raw
Quoted 100 lines · Randy Dunlap
Quoted 90 lines · Gerrit Huizenga
Quoted 36 lines · holzheu
Quoted 5 lines · Jan Kara

<snip>

Quoted 3 lines

Your proposal is similar to one I made to some Japanese developers earlier this year. I was more modest, proposing that we

Quoted 8 lines
Quoted 3 lines

- add an enhanced printk

	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);

Maybe a stupid idea but why do we want to assign these numbers by hand?
I can imagine it could introduce collisions when merging tons of patches with new messages... Wouldn't it be better to compute say, 8-byte hash from the message and use it as it's identifier? We could do this automagically at compile time.

Of course automatically generated message numbers would be great and something like:

hub.4a5bcd77: Detected some problem.

looks acceptable for me.

We could generate the hash using the format string of the printk. Since we specify the format string also in KMSG_DOC, the hash for the KMSG_DOC and the printk should match and we have the required link between printk and description.

So technically that's probably doable.

Problems are:

* hashes are not unique
* We need an additional preprocessor step
* The might be people, who find 8 character hash values ugly in printks

The big advantage is, that we do not need to maintain message numbers.

Quoted 8 lines
Quoted 4 lines

I know it also has it's problems - you
fix a spelling and the message gets a different id and you have to update translation/documentation catalogue but maybe that could be solved too...

Since in our approach the message catalog is created automatically for exactly one kernel and the message catalog belongs therefore to exactly one kernel, I think the problem of changing error numbers is not too big.

We just had a meeting with the Japanese and several other participants from the vendor and community side and came up with a potential proposal that is similar to many things discussed here. It has the benefit that it seems implementable and low/no overhead on most kernel developers. The basic proposal is to use a tool, run by the kernel Makefile to extract kernel messages from either the kernel source or the .i files (both have advantages, although I prefer the source to the .i file since it 1) gets all messages and 2) is probably a little quicker with less impact to the standard kernel make.
These messages would be stored in a file in the source tree, e.g. usr/src/linux/Translations/English. As each message is added to that file, we calculate, say, an MD5 sum of the printk (dev_printk, sdev_printk, etc.) string, and the text file ultimately contains:
MD5 Checksum of text; the printk text itself, the File name, the line number.
The checksum is run over just the printk. We definitely would not include the line number since the line number is too volatile. Including the file name in the hash *might* help disambiguate the hash a bit better in the case of duplicates, but there was some debate that duplicates might be better handled in other ways.
Andrew mentioned a mechanism for adding a subsystem tag or other tag which helps disambiguate the message, either in the message file or in the end user documentation (e.g. the Message Pedia/mPedia that the Japanese have already created with ~350 messages, and a total of ~700 targetted by the end of the year).
That tag could be appended to the beginning of the printk, to the end of the printk, or even in a formatted comment at the end of the printk that the tool could extract.
Then, the translations could be managed by anyone outside of the normal/ core kernel community, by simply creating a translation file, e.g. usr/src/linux/Translations/Japanese, which contained the MD5 sum, the translated message, the file name and line number (the last two redundent perhaps but informational, and automatically generated if possible). The files in the Translations directory could be uesd as the unique keys for an external database (such as the Message Pedia, vendors or distributions help pages, etc.) to help look up and explain root cause of a problem. The key property here is that the MD5 sum becomes the key to all database entries to look up that key.
Further, yet another kernel config option could allow distros to output the calculated MD5 sum to be printed, much like we do with timestamps today.
End result is that these in-kernel message catalogs for translation are updated automatically (mostly no kernel developer changes needed) and the translations can be maintained by anyone who is interested. On the topic of MD5 collisions, using a disambiguating tag would be a

Quoted 8 lines

simple addition for the few cases where that happens, the tool could be educated to use that tag in the calculation of the MD5 sum, and we have a 98% solution which impacts <1% of the kernel developers. Folks present for this discussion included Ted T'so, James Bottomley, several of the key Japenese folks interested in using this for debugging, and reps from several vendors who would find this sort of info useful for their folks supporting Linux in the field.
Comments?

Quoted 4 lines

For those of us who have not been in on these meetings, I think that some serious justifications are needed. The last paragraph began to go in that direction, but it needs to be more detailed and convincing. And "for debugging" doesn't cut it IMO.

Full "justification" has never really been clear outside of the need to try to duplicate what other Unixes have had for their support departments in the past. And yes, I've seen and sat through the presentation by the Japanese group many different times, and talked to the individual people who want this.

And you know, that's fine. If anyone wants to go off and do such a project like this, in a way that is almost entirely self-contained with no affect on the development process or any kernel developer, I have no objection to it at all.

And I think the proposal that Andrew made meets that requirement, so let's wait and see what actual code comes out of this and judge it on that merit at that time.

thanks,

greg k-h

#31 lore raw
In reply to Randy Dunlap
Quoted 8 lines · Randy Dunlap

For those of us who have not been in on these meetings, I think that some serious justifications are needed. The last paragraph began to go in that direction, but it needs to be more detailed and convincing. And "for debugging" doesn't cut it IMO.

Signature · 1 line
~Randy

Fair point, Randy.

This came up this time because people in Japan supporting Linux customers are not always familiar with Linux kernel internals and thus debugging problems based on a terse, English language message is just painful. Also, any given message doesn't really provide any input about what *really* failed, what should be done about it, how to correct this issue, or what parts to pull out of your machine and replace with properly working parts.

So, the Message Pedia is basically a Japanese language wiki today where people doing support can annotate the messages with real life experience on what to *do* about the error.

Part of the problem is that the error messages when extracted don't stand alone and don't easily allow tracing back to the real source. When the problem is handed from the second line support folks deeper into the support or development organization, the message has to be found in the source tree in some non-ambiguous fashion.

The other problem that comes up in other, non-English locales is that debugging your Linux box when you are a non-native English speaker is just plain impossible. Providing a means for getting localized kernel messages allows more end users to at least reasonably debug their own machine. Most every user level project today allows localization directly but the kernel has never had any reasonable mechanism for localisation.

Now, the proposal here doesn't make the kernel "as good as" user level application, but it would at least provide a searchable key to help people get localized guidance on problems.

I'm sure there are other benefits that most can see based on just simple debuggability. But the Japanese goal is really to construct a database of debugging info & context based on their experiences.

And, I'm sure that others can speak for themselves. I'm mostly playing scribe on this one - we abandoned our error and event subsystem work years ago, even though this was one of the biggest weaknesses we saw in customer support 3-5 years ago, simply because our changes were still viewed as too invasive for mainline kernel. These proposed changes help a lot, while remaining almost completely invisible to most developers.

gerrit

author
#32 lore raw
In reply to Gerrit Huizenga

Hi Gerrit,

The common thing of your and our approach is, that we need an ID to identify a message either by:

* Using hashes + component name (maybe)
* Or using hand-made message numbers + component name. Numbers have to be unique within the kernel component

I think the main difference of your approach is, that documentation will be maintained outside the kernel. This has advantages, but also disadvantages:

(+) Developers have less work to do :-)
(+) Kernel sources and patches have less lines of code, since message
descriptions are not contained in the kernel sources.
(+) You can support multiple languages

(-) Hard to keep printks and descriptions up-to-date. If using hashes,
each typo fix in a printk breaks the link to your documentation.
(-) Since the developer knows the meaning of a printk best, he probably
would be the right person to write the description.
(-) For every Distribution or vanilla kernel you probably need separate
external message databases.

I like the fact, that every printk gets an ID in your approach and you do not need additional printk macros.

Maybe both approaches are not mutually exclusive. If developers would like to document their messages within the kernel sources, they still could use kernel-doc and our KMSG_DOC macro do do that.

I admit, that I am not sure about the the best solution here.

Michael

#33 lore raw
Quoted 4 lines · holzheu

Hi Gerrit,

The common thing of your and our approach is, that we need an ID to identify a message either by:

Maybe I am missing something big, but why is an ID needed? The message IS the ID right? That's the only thing that is robust against code moving about....

(And yes I am aware that very occasionally there might be a duplicate message that REALLY is different, but I suspect that that is so incredibly rare that it's just simpler to reword one of the two in such case)

#34 lore raw
Quoted 10 lines · Arjan van de Ven
Quoted 4 lines · holzheu

Hi Gerrit,

The common thing of your and our approach is, that we need an ID to identify a message either by:

Maybe I am missing something big, but why is an ID needed? The message IS the ID right? That's the only thing that is robust against code moving about....

I think the problem is messages contain device numbers, and all similar
kind of data. So what you'd really like is to use >format string< as ID. But that is not easily accessible for the user, so you'd have to print it with each message. And that's ... ugly IMHO.

Quoted 4 lines

(And yes I am aware that very occasionally there might be a duplicate message that REALLY is different, but I suspect that that is so incredibly rare that it's just simpler to reword one of the two in such case)

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SuSE CR Labs
author
#35 lore raw
In reply to Arjan van de Ven
Quoted 10 lines · Arjan van de Ven
Quoted 4 lines · holzheu

Hi Gerrit,

The common thing of your and our approach is, that we need an ID to identify a message either by:

Maybe I am missing something big, but why is an ID needed? The message IS the ID right? That's the only thing that is robust against code moving about....

Yes. As already discussed with Pavel, it is one option to use the format string of the message as message ID. The disadvantage compared to message IDs like hashes is, that format strings might be even less unique than hashes and it's probably less convenient for searching by operators.

An operator has to issue either:

Quoted 1 line
Quoted 1 line

info lp.4711

or

Quoted 1 line
Quoted 1 line

info "lp0: on fire"

Michael

#36 lore raw
Quoted 8 lines · holzheu
Quoted 7 lines · Arjan van de Ven
Quoted 4 lines · holzheu

Hi Gerrit,

The common thing of your and our approach is, that we need an ID to identify a message either by:

Quoted 8 lines
Quoted 3 lines

Maybe I am missing something big, but why is an ID needed? The message IS the ID right? That's the only thing that is robust against code moving about....

Yes. As already discussed with Pavel, it is one option to use the format string of the message as message ID. The disadvantage compared to message IDs like hashes is, that format strings might be even less unique than hashes

if the hash comes from the string in the first place I have a hard time believing that.

Quoted 2 lines

and it's probably less convenient for searching by operators.

I'm not convinced of that...

Quoted 6 lines

An operator has to issue either:

Quoted 1 line
Quoted 1 line

info lp.4711

or

Quoted 1 line
Quoted 1 line

info "lp0: on fire"

well.... surely the messages are caught by some userspace program, right? (like syslog).. that can do the lookup and make it all conveniently lookup-able and cross-referencable etc etc....

Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
author
#37 lore raw
Quoted 13 lines · Arjan van de Ven
Quoted 12 lines · holzheu
Quoted 10 lines · Arjan van de Ven
Quoted 4 lines · holzheu

Hi Gerrit,

The common thing of your and our approach is, that we need an ID to identify a message either by:

Maybe I am missing something big, but why is an ID needed? The message IS the ID right? That's the only thing that is robust against code moving about....

Quoted 8 lines
Quoted 4 lines

Yes. As already discussed with Pavel, it is one option to use the format string of the message as message ID. The disadvantage compared to message IDs like hashes is, that format strings might be even less unique than hashes

if the hash comes from the string in the first place I have a hard time believing that.

Just think of all messages containing %s, which can expand to every possible string.

Michael

#38 lore raw
Quoted 19 lines · holzheu
Quoted 18 lines · Arjan van de Ven
Quoted 16 lines · holzheu
Quoted 10 lines · Arjan van de Ven
Quoted 4 lines · holzheu

Hi Gerrit,

The common thing of your and our approach is, that we need an ID to identify a message either by:

Maybe I am missing something big, but why is an ID needed? The message IS the ID right? That's the only thing that is robust against code moving about....

Yes. As already discussed with Pavel, it is one option to use the format string of the message as message ID. The disadvantage compared to message IDs like hashes is, that format strings might be even less unique than hashes

Quoted 6 lines
Quoted 3 lines

if the hash comes from the string in the first place I have a hard time believing that.

Just think of all messages containing %s, which can expand to every possible string.

of course you hash on the string before you do the format stuff....

Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
author
#39 lore raw
In reply to Arjan van de Ven

On Mon, 2007-06-18 at 18:36 -0700, Arjan van de Ven wrote:

<snip>

Quoted 3 lines

well.... surely the messages are caught by some userspace program, right? (like syslog).. that can do the lookup and make it all conveniently lookup-able and cross-referencable etc etc....

Ok, I agree. Maybe that's really a good idea!

So what about the following proposal:

* Use printk format strings as message IDs
* Use kernel-doc (or similar mechanism) for message documentation
* Use Subsystem tag/component ID for each printk. This might be optional, but would make sense to get messages more unique.
* Implement a new tool (kmsg) to allow operators
to access kernel message descriptions in a convenient way.

The kmsg tool will read the dmsg buffer together with the list of documented printks, which has been created in the kernel build using the format strings defined in the KMSG_DOC macros. It matches the known format strings to the dmesg messages and informs the operator, for which message he can get a description.

Something like:

Quoted 1 line
Quoted 1 line

kmsg

Linux version 2.6.9-42.0.3.ELsmp...
....
[<ID>] lp0 on fire

Quoted 1 line
Quoted 1 line

kmsg <ID>

Desciption: your printer failed basic tests, you should check cabling

The <ID> is generated by the kmsg tool and could be e.g. the hashes for the matching format strings.

Michael

#40 lore raw
In reply to holzheu

Hi Michael,

I initiated this discussion in The Linux Foundation Summit@Google Campus last week from the viewpoint of requirements by Japanese vendors/users.

As for a file to store message descriptions, I perfectly agree with your analysis and we also noticed some (+) and (-) you pointed out in the meeting last week. Our conclusion was to have a file outside of the kernel source codes to lessen developers burdens and to encourage better documentations of descriptions (as Andrew mentioned in his initial message).

Takashi Kunai,
The Linux Foundation Japan

Quoted 43 lines · holzheu

Hi Gerrit,

The common thing of your and our approach is, that we need an ID to identify a message either by:

* Using hashes + component name (maybe)
* Or using hand-made message numbers + component name. Numbers have to be unique within the kernel component

I think the main difference of your approach is, that documentation will be maintained outside the kernel. This has advantages, but also disadvantages:

(+) Developers have less work to do :-)
(+) Kernel sources and patches have less lines of code, since message
descriptions are not contained in the kernel sources.
(+) You can support multiple languages

(-) Hard to keep printks and descriptions up-to-date. If using hashes,
each typo fix in a printk breaks the link to your documentation.
(-) Since the developer knows the meaning of a printk best, he probably
would be the right person to write the description.
(-) For every Distribution or vanilla kernel you probably need separate
external message databases.

I like the fact, that every printk gets an ID in your approach and you do not need additional printk macros.

Maybe both approaches are not mutually exclusive. If developers would like to document their messages within the kernel sources, they still could use kernel-doc and our KMSG_DOC macro do do that.

I admit, that I am not sure about the the best solution here.

Michael

_______________________________________________
Lf_kernel_messages mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/lf_kernel_messages

Signature · 5 lines
Kunai, Takashi
The Linux Foundation Japan (New since '07/2/5)
Shibuya Mark City West 22nd Floor
1-12-1 Dogenzaka,Shibuya-ku,Tokyo,Japan
zip150-0043 tel+81-3-4360-5493
#41 lore raw
In reply to Gerrit Huizenga
Quoted 10 lines · Gerrit Huizenga

Andrew mentioned a mechanism for adding a subsystem tag or other tag which helps disambiguate the message, either in the message file or in the end user documentation (e.g. the Message Pedia/mPedia that the Japanese have already created with ~350 messages, and a total of ~700 targetted by the end of the year).

That tag could be appended to the beginning of the printk, to the end of the printk, or even in a formatted comment at the end of the printk that the tool could extract.

The best way to maintain such a subsystem tag most be supported by kbuild. So say we have the subsystem "scsi" then we can set this subsystem tag in drivers/scsi/Makefile and let drivers/scsi/arm/Makefile inherit this vaule doing nothing.

Addign the subsystem tag to the printk could than be done using the C preprocessor and we are all fine.

If we go by the kernel-doc way to document kernel messages (the few that actually needs additional explanation) then we need to teach kernel-doc to collect the same subsystem tag but that should be doable.

Sam

#42 lore raw
Quoted 17 lines · On Mon, 18 Jun 2007 15:11:57 +0200 Sam Ravnborg wrote:
Quoted 10 lines · Gerrit Huizenga

Andrew mentioned a mechanism for adding a subsystem tag or other tag which helps disambiguate the message, either in the message file or in the end user documentation (e.g. the Message Pedia/mPedia that the Japanese have already created with ~350 messages, and a total of ~700 targetted by the end of the year).

That tag could be appended to the beginning of the printk, to the end of the printk, or even in a formatted comment at the end of the printk that the tool could extract.

The best way to maintain such a subsystem tag most be supported by kbuild. So say we have the subsystem "scsi" then we can set this subsystem tag in drivers/scsi/Makefile and let drivers/scsi/arm/Makefile inherit this vaule doing nothing.

Quoted 6 lines

Addign the subsystem tag to the printk could than be done using the C preprocessor and we are all fine.

If we go by the kernel-doc way to document kernel messages (the few that actually needs additional explanation) then we need to teach kernel-doc to collect the same subsystem tag but that should be doable.

Actually I would prefer that such a tool be a clone of scripts/kernel-doc and use a (slightly?) modified tag instead of the kernel-doc "/**" tag if this route is used at all.

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

#43 lore raw
In reply to Gerrit Huizenga
Quoted 5 lines · Gerrit Huizenga

Further, yet another kernel config option could allow distros to output the calculated MD5 sum to be printed, much like we do with timestamps today.

Comments?

Would the compiled-in text then also become replaceable? Or is the MD5 sum output expected to be in addition to the regular English message?

If message replacement at compile-time is supported, this could allow for creating "short" versions of the messages, which could have a beneficial impact on kernel size.

Right now, it is possible to completely disable printks and re-claim about 100K of memory. However, in some embedded configurations, even if you are space-constrained it's desirable to retain some of the printks, for in-field debugging. Thus not very many embedded developers disable printks completely, even though the option has been supported for a while. (That, and many aren't caught up to the kernel version where it was introduced (2.6.10) :-)

But compressed messages (shortened text through abbreviations, or just outputting the ID alone, etc.) could save SOME of the space, in trade for less readability. Heck, just removing all vowels would probably save 10k, and not hurt readability that much.

Finally, for testing, it's handy to also
have automatic translation generators.
At a former company I worked for, they had translators that would output:
* all messages shortened by 20%
* all messages lengthened by 20%
* every message converted to pig-latin

These were mostly used for testing if the strings broke screen real-estate constraints (which don't apply to kernel messages). But the automatic translators would sometimes catch messages with weird attributes.

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================

#44 lore raw
Quoted 10 lines · Tim Bird
Quoted 5 lines · Gerrit Huizenga

Further, yet another kernel config option could allow distros to output the calculated MD5 sum to be printed, much like we do with timestamps today.

Comments?

Would the compiled-in text then also become replaceable? Or is the MD5 sum output expected to be in addition to the regular English message?

The MD5 sum would be calculated at print time, no proposal in here to replace the in-kernel text. And, I'm not sure that replacing it with an MD5 sum would every be significantly shorter, ergo I don't think this helps your problem.

The methods of post-processing to "shrink" the kernel text here seem too ugly to ponder. And I just pondered a few that made my head hurt (sort the MD5 sums, re-insert the number of the MD5 sum as an integer instead of the original text, and, beyond being gross, what do you do with the formatting info about the args?).

Quoted 4 lines

If message replacement at compile-time is supported, this could allow for creating "short" versions of the messages, which could have a beneficial impact on kernel size.

Quoted 8 lines

Right now, it is possible to completely disable printks and re-claim about 100K of memory. However, in some embedded configurations, even if you are space-constrained it's desirable to retain some of the printks, for in-field debugging. Thus not very many embedded developers disable printks completely, even though the option has been supported for a while. (That, and many aren't caught up to the kernel version where it was introduced (2.6.10) :-)

Which ones matter? Odds are you could use the KERNEL_LOGLEVEL or such to mark those, then compile out the rest.

Quoted 5 lines

But compressed messages (shortened text through abbreviations, or just outputting the ID alone, etc.) could save SOME of the space, in trade for less readability. Heck, just removing all vowels would probably save 10k, and not hurt readability that much.

Ick. Are you serious? How about running them through a valley girl filter and then converting to high school text messaging?

Quoted 7 lines

Finally, for testing, it's handy to also
have automatic translation generators.
At a former company I worked for, they had translators that would output:
* all messages shortened by 20%
* all messages lengthened by 20%
* every message converted to pig-latin

Double yucko.

Quoted 4 lines

These were mostly used for testing if the strings broke screen real-estate constraints (which don't apply to kernel messages). But the automatic translators would sometimes catch messages with weird attributes.

I don't think people are worried about the correctness of the messages and message formats - much of that can actually be detected by simple tools. The goal here was to at least allow people that support operating systems and for whom English (and English collation sequences) is not a primary language do some initial system diagosis.

I think compiling out the messages of something below some LOGLEVEL is a lot more practical.

gerrit

#45 lore raw
In reply to Jan Kara

Hi!

Quoted 10 lines
Quoted 6 lines

Your proposal is similar to one I made to some Japanese developers earlier this year. I was more modest, proposing that we

- add an enhanced printk

	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);

Maybe a stupid idea but why do we want to assign these numbers by hand?
I can imagine it could introduce collisions when merging tons of patches with new messages... Wouldn't it be better to compute say, 8-byte hash from the message and use it as it's identifier? We could do this

Why hashes? Just key it off full message, and treat it like normal translation problem.

'lp%d: on fire' -> 'your printer failed basic tests, you should check cabling'.

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
author
#46 lore raw

Hi Pavel,

Quoted 13 lines · Pavel Machek

Hi!

Quoted 10 lines
Quoted 6 lines

Your proposal is similar to one I made to some Japanese developers earlier this year. I was more modest, proposing that we

- add an enhanced printk

	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);

Maybe a stupid idea but why do we want to assign these numbers by hand?
I can imagine it could introduce collisions when merging tons of patches with new messages... Wouldn't it be better to compute say, 8-byte hash from the message and use it as it's identifier? We could do this

Quoted 5 lines

Why hashes? Just key it off full message, and treat it like normal translation problem.

'lp%d: on fire' -> 'your printer failed basic tests, you should check cabling'.

Hmmm... What about extracting all format strings from the KMSG_DOC macros and storing them in a file. E.g.:

  printk("lp%d: on fire");

/**
* message
*
* Description:
* Your printer failed basic tests, you should check cabling
...

  KMSG_DOC(lp, "lp%d: on fire");

Some build mechanism creates a file with all format strings of the KMSG_DOC macros:

E.g.: cat kmsgs.txt

# MSG: "lp%d: on fire"
# COMPONENT: lp
# DESC: Your printer failed basic tests, you should check cabling

Then we provide a tool e.g. "kmsg", which gets as input the actual kernel message and searches the message database for matching format strings:

Quoted 2 lines
Quoted 2 lines

kmsg "lp0: on fire"
Your printer ...

Of course, we can get duplicate hits, too.

The advantage of that approach is, that we don't need additional message identifiers like hashes.

Michael

#47 lore raw

Hi!

Quoted 8 lines
Quoted 2 lines

'lp%d: on fire' -> 'your printer failed basic tests, you should check cabling'.

Hmmm... What about extracting all format strings from the KMSG_DOC macros and storing them in a file. E.g.:

  printk("lp%d: on fire");
Quoted 8 lines

/**
* message
*
* Description:
* Your printer failed basic tests, you should check cabling
...

  KMSG_DOC(lp, "lp%d: on fire");

Yes, that was the idea. But we have already tools and standard format for this (gettext), we should just use that.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#48 lore raw

Documentation of kernel messages (Summary)

In reply to Andrew Morton

Hi all,

Any idea, how to proceed with this topic? Do you think that any of the suggested solutions for documentation / translation of kernel messages will have a chance to be included in the kernel?

I tried to summarize the outcome of this discussion...

There are two main issues:

* Translate messages
* Document messages

For both, we have to give messages identifiers in order to match them to the corresponding documentation / translation. Three possible solutions have been suggested:

1. Use message numbers maintained by driver owners
2. Automatically create hashes for printks
3. Use printk format string as message identifier

Regarding the question where to put documentation and translations we have two options:

* In the kernel tree
* Somewhere outside the kernel tree

Another issue of the discussion was the usage of the component name for printks.

Useful comments:
================
Andrew Morton:
Suggested an alternative approach using a new printk function with message ID parameter. Documentation / Translation should be done outside the kernel.

Gerrit Huizenga:
Suggested an alternative approach using hashes as message IDs.

Greg Kroah-Hartman:
He pointed out, that we need a solution which integrates dev_xxx() macros.

Randy Dunlop:
He doesn't want to have message documentation included in the kernel-doc tool. We should create new tool for kernel messages.

Pavel Machek:
Suggested to use gettext for message documentation / translation.

Useful links:
=============
There already exists a mailing list for the kernel messages topic:
https://lists.linux-foundation.org/mailman/listinfo/lf_kernel_messages

There was an LWN article summarizing some aspects of the discussion:
http://lwn.net/Articles/238948/

Pros and Cons of the different Message ID methods:
==================================================
Message numbers maintained by driver owners:
+ Unique IDs
+ Can be kept stable between different kernel versions
+ Efficient way of matching printk to Documentation or translation
- Difficult to maintain

printk hashes:
+ Easy to maintain
+ Efficient way of matching printk to Documentation or translation
- Additional preprocessor step needed
- No unique IDs
- Ugly in messages, since hashes can have many characters
- Volatile, since typo fixes change hash values

Format strings:
+ No change needed for printks
+ Matching of printk to corresponding documentation or translation is not as efficient as using hashes
- No unique IDs

Pros and Cons of external documentation (vs. internal documentation):
====================================================================
+ Developers have less work to do
+ Kernel sources and patches have less lines of code, since message descriptions are not contained in the kernel sources.
+ You can support multiple languages

- Hard to keep printks and descriptions up-to-date. If using hashes, each typo fix in a printk breaks the link to your documentation.
- Since the developer knows the meaning of a printk best, he probably would be the right person to write the description.
- For every Distribution or vanilla kernel you probably need separate external message databases.

Pros and Cons of using component names in printks:
==================================================
+ Makes it easier to identify source of message
+ Makes message more unique (especially, when using Hashes or format strings as message IDs)
- Changes of printks necessary

Implementation issues (depending on what solution we implement):
===============================================================
* Preprocessor tool for hashes
* Check tool for messages/message description
* Tool to extract kernel documentation from kernel tree
* Sysadmin tool to query message descriptions
* New kernel macros for Messages to support Component names / message numbers

#49 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 5 lines · On Monday 25 June 2007 09:48:41 Michael Holzheu wrote:

Hi all,

Any idea, how to proceed with this topic? Do you think that any of the suggested solutions for documentation / translation of kernel messages will have a chance to be included in the kernel?

Personally? No to the second question, which renders the first "do it yourself outside of the tree".

Just a guess, and I don't speak for anyone else here, but I think most of us are waiting to see how long it takes you to lose interest.

Quoted 6 lines

I tried to summarize the outcome of this discussion...

There are two main issues:

* Translate messages
* Document messages

I'm somewhat sympathetic to translation (although I tend to think of it as a distro's job, and still wonder why IBM is pushing this directly rather than trying to bounce it off Red Hat or somebody). But you could put a sed filter around dmesg to give you swahili output entirely in userspace. Have you shown the simple approach to be insufficient yet?

Is "document messages" the same problem as "translate messages", or are these two different problems conflated together?

I find "document messages" to be a horrible idea conceptually, because I think the messages should _be_ documentation. If the message is unclear it should be clarified. "There's too much garbage on the floor, we should laminate it so we can't smell it anymore." Er, no...

Quoted 5 lines

Three possible solutions have been suggested:

1. Use message numbers maintained by driver owners
2. Automatically create hashes for printks
3. Use printk format string as message identifier

#2 has the advantage that you can do it without buy-in from the community, and without touching the code. (Of course it doesn't necessarily help you if the printk string is "%s:%s".)

But I'm still convinced you could tackle over 90% of the problem from userspace, without touching the kernel.

Quoted 7 lines

Pros and Cons of the different Message ID methods:
==================================================
Message numbers maintained by driver owners:
+ Unique IDs
+ Can be kept stable between different kernel versions
+ Efficient way of matching printk to Documentation or translation
- Difficult to maintain

Will at best only ever be done for some drivers, not all. But that's probably inescapable no matter what your approach is.

Quoted 2 lines

printk hashes:
- Additional preprocessor step needed

Only for you guys. Those of us building our own kernels and NOT translating them or producing "the big book of explanations of kernel messages for IBM customers who don't understand them" shouldn't have to run this preprocessor step.

Quoted 11 lines

Pros and Cons of external documentation (vs. internal documentation):
====================================================================
+ Developers have less work to do
+ Kernel sources and patches have less lines of code, since message descriptions are not contained in the kernel sources.
+ You can support multiple languages

- Hard to keep printks and descriptions up-to-date. If using hashes, each typo fix in a printk breaks the link to your documentation.
- Since the developer knows the meaning of a printk best, he probably would be the right person to write the description.

Imposing additional requirements on developers isn't going to work unless you reject patches from developers who don't follow your procedures and submit the appropriately filled-out forms with each patch for processing by the central bureaucracy.

Signed-off-by works because A) Linus enforces it, B) The overhead's very low,
C) attribution is already highly good thing within our community values, D) it annoys SCO.

The javadoc function comments in the source don't have complete coverage and probably never will (although they're getting better). The premise of doing something similar for messages is that a translating dmesg in userspace can't necessarily get complete coverage, so you need a solution modeled on something that doesn't have complete covereage years after its introduction...

Quoted 2 lines

- For every Distribution or vanilla kernel you probably need separate external message databases.

"If you're resigned to doing that anyway you don't actually need him to lie there all the time, do you?" - Ford Prefect

If you need an external message database what's wrong with doing it entirely in userspace via a sed-based translating dmesg?

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#50 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 4 lines

I find "document messages" to be a horrible idea conceptually, because I think the messages should _be_ documentation. If the message is unclear it should be clarified. "There's too much garbage on the floor, we should laminate it so we can't smell it anymore." Er, no...

"Document message" is for the cases where the message itself does not convey enough info.
No need to add more text when the initial message is good enough.

Sam

#51 lore raw

Re: Documentation of kernel messages (Summary)

In reply to Rob Landley
Quoted 9 lines · Rob Landley
Quoted 5 lines · On Monday 25 June 2007 09:48:41 Michael Holzheu wrote:

Hi all,

Any idea, how to proceed with this topic? Do you think that any of the suggested solutions for documentation / translation of kernel messages will have a chance to be included in the kernel?

Personally? No to the second question, which renders the first "do it yourself outside of the tree".

If that is the opinion of the majority here, fine. If there is no hard rule on how to define printk macros, one option for us would be to define some new s390 specific printk macros for our device drivers. Similar to hundreds of other driver specific printk macros in the kernel.

Quoted 2 lines

Just a guess, and I don't speak for anyone else here, but I think most of us are waiting to see how long it takes you to lose interest.

:-) Work is not always fun. Sometimes it is just a duty.

Michael

#52 lore raw

Re: Documentation of kernel messages (Summary)

Hi Micheal,

This is a late response to your summary below.

First of all, there is a lot of interest in this issue among Japanese vendor community. Recently, I have built a Japanese mailing list to discuss this issue. I'm posting this comment resulting from such discussion.

(1) Your kernel development proposal will be greatly supported by Japanese vendor community. At the same time, it needs support from the kernel communities, as well.
(2) As for message ID's, you proposed 3 ways; (a)adding message numbers, (b)printk hashes and (c)Format strings. (a) seems difficult to get supports from kernel developers and (c) lacks uniqueness of each message. Though (b) also lacks uniqueness, adding component id and/or file name (files name could be hashed, as well), it could achieve some practical level of uniqueness. To avoid ugly message format, it could be possible to introduce a system parameter to suppress this hash ids. Thus, our preference is (b).
(3) As for a documentation file, outside of kernel source tree would be fine.
(4) When this kernel message scheme become feasible, it will be needed to talk about the standardized contents of each description (meanings/actions/etc).

Takashi Kunai

Signature · 5 lines
Kunai, Takashi
The Linux Foundation Japan (New since '07/2/5)
Shibuya Mark City West 22nd Floor
1-12-1 Dogenzaka,Shibuya-ku,Tokyo,Japan
zip150-0043 tel+81-3-4360-5493
Quoted 26 lines · Michael Holzheu
Quoted 8 lines · Rob Landley
Quoted 5 lines · On Monday 25 June 2007 09:48:41 Michael Holzheu wrote:

Hi all,

Any idea, how to proceed with this topic? Do you think that any of the suggested solutions for documentation / translation of kernel messages will have a chance to be included in the kernel?

Personally? No to the second question, which renders the first "do it yourself outside of the tree".

If that is the opinion of the majority here, fine. If there is no hard rule on how to define printk macros, one option for us would be to define some new s390 specific printk macros for our device drivers. Similar to hundreds of other driver specific printk macros in the kernel.

Quoted 2 lines

Just a guess, and I don't speak for anyone else here, but I think most of us are waiting to see how long it takes you to lose interest.

:-) Work is not always fun. Sometimes it is just a duty.

Michael

#53 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 3 lines · Takashi

(1) Your kernel development proposal will be greatly supported by Japanese vendor community. At the same time, it needs support from the kernel communities, as well.

There is a very strong reason for the kernel community to NOT support
this: it makes it much harder to deal with bug reports.

Like it or not, English is a lingua franca[1] of the technology industry. When I deal with the rest of the kernel community, I use English, instead of my native language (Swedish).

For things that are targetted toward end users, that's a different matter, of course, but kernel messages are inherently directed toward developers and technically sophisticated users.

-hpa

#54 lore raw

Re: Documentation of kernel messages (Summary)

In regard to translating kernel messages:

Quoted 7 lines · On Monday 09 July 2007 01:36:31 H. Peter Anvin wrote:
Quoted 3 lines · Takashi

(1) Your kernel development proposal will be greatly supported by Japanese vendor community. At the same time, it needs support from the kernel communities, as well.

There is a very strong reason for the kernel community to NOT support
this: it makes it much harder to deal with bug reports.

Agreed.

There's a proposal in play for translations of documentation, and to recruit "language maintainers" (which is an idea I see that Matt Mackall came up with before I did).

A language maintainer is somebody who can accept a patch in language X, translate its description to post on the appropriate english-only list with a
cc: to the appropriate maintainer(s), and translate questions and reviews back and forth for the original author until the patch can be merged. (This means that documentation translations don't result in a stream of unmergeable patches.)

As for the documentation translations themselves, I note that Eric Raymond dissuaded me from actually hosting translations of kernel documentation on
http://kernel.org/doc due to his experience with translations of his own writings. If he hosts the translations on his website, they never get updated again. But if he makes the contributor host them on their own website, then they sometimes get updated.

For my part, I can't _tell_ when a given translation is out of date because I can't read it, and I certainly can't update it. So I agree with Eric and I'm linking to sites hosting kernel documentation in other languages rather than hosting them myself. I've got a good link to a Japanese site, need to ping the contributors of the chinese documentation and see if they have a site for it...

I await hearing back from Satoru Ueda who might have a lead on a potential Japanese maintainer, probably after some event in Japan (a "Jamboree") in a week or two...

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#55 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 12 lines · Rob Landley

In regard to translating kernel messages:

Quoted 7 lines · On Monday 09 July 2007 01:36:31 H. Peter Anvin wrote:
Quoted 3 lines · Takashi

(1) Your kernel development proposal will be greatly supported by Japanese vendor community. At the same time, it needs support from the kernel communities, as well.

There is a very strong reason for the kernel community to NOT support
this: it makes it much harder to deal with bug reports.

Agreed.

[...]

Quoted 8 lines

As for the documentation translations themselves, I note that Eric Raymond dissuaded me from actually hosting translations of kernel documentation on
http://kernel.org/doc due to his experience with translations of his own writings. If he hosts the translations on his website, they never get updated again. But if he makes the contributor host them on their own website, then they sometimes get updated.

Quoted 6 lines

For my part, I can't _tell_ when a given translation is out of date because I can't read it, and I certainly can't update it. So I agree with Eric and I'm linking to sites hosting kernel documentation in other languages rather than hosting them myself. I've got a good link to a Japanese site, need to ping the contributors of the chinese documentation and see if they have a site for it...

Yeah, but it seems like having a translations directory in the kernel avoids that problem - anyone can update, it is a single source, no digging for sites that aren't tied to the kernel, available in the distros directly, etc.

I'm not sure that the web site hosting argument is a good one. Arch maintainers and Language Maintainers have some similarities. Also, being able to clearly mark which version of a document was last translated would help a bit with the fact that most of us aren't proficient in all of the world's languages. But, knowing who the translation maintainer is, and when the translation was last updated allows both the original doc maintainer and the translation document maintainer to see when a document likely needs to be updated. And that is probably good enough.

gerrit

#56 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 30 lines · On Monday 09 July 2007 13:18:57 Gerrit Huizenga wrote:
Quoted 12 lines · Rob Landley

In regard to translating kernel messages:

Quoted 7 lines · On Monday 09 July 2007 01:36:31 H. Peter Anvin wrote:
Quoted 3 lines · Takashi

(1) Your kernel development proposal will be greatly supported by Japanese vendor community. At the same time, it needs support from the kernel communities, as well.

There is a very strong reason for the kernel community to NOT support
this: it makes it much harder to deal with bug reports.

Agreed.

[...]

Quoted 7 lines

As for the documentation translations themselves, I note that Eric Raymond dissuaded me from actually hosting translations of kernel documentation on http://kernel.org/doc due to his experience with translations of his own writings. If he hosts the translations on his website, they never get updated again. But if he makes the contributor host them on their own website, then they sometimes get updated.

Quoted 6 lines

For my part, I can't _tell_ when a given translation is out of date because I can't read it, and I certainly can't update it. So I agree with Eric and I'm linking to sites hosting kernel documentation in other languages rather than hosting them myself. I've got a good link to a Japanese site, need to ping the contributors of the chinese documentation and see if they have a site for it...

Quoted 4 lines

Yeah, but it seems like having a translations directory in the kernel avoids that problem - anyone can update, it is a single source, no digging for sites that aren't tied to the kernel, available in the distros directly, etc.

No. It doesn't help.

99% of the kernel directory is C. That means any random passerby can review code. Everyone who has the kernel tarball should be able to review code that's in there, plus when you compile it breaks. So merging _code_ into the kernel helps keep it up to date.

Merging documentation into the kernel doesn't help keep it up to date, because documentation being out of date doesn't break the build. It may get the documentation more review, but the existing state of Documentation/* argues against that. It's a struggle to keep the english versions on the same continent as "up to date" or "complete", and most of the _good_ documentation is out in OLS papers and such (which I'm off indexing as we speak).

Now add in the non-english nature of the new Documentation/stale subdirectories you're proposing. Almost nobody in the existing kernel community can even _read_ this documentation, let alone update it. People who update Documentation/* _can't_ update the translations, and that will _never_change_.

Merging into the kernel is a great way to keep CODE up to date. Don't think it's magic pixie dust for documentation. It never has been before.

Quoted 2 lines

I'm not sure that the web site hosting argument is a good one. Arch maintainers and Language Maintainers have some similarities.

Ah, but It's not a language maintainer's job to update documentation. It's their job to ACCEPT PATCHES. Translate patches, translate questions back and forth. This has NOTHING to do with documentation unless they're converting documentation accompanying the patch one way; into english.

Quoted 4 lines

Also, being
able to clearly mark which version of a document was last translated would help a bit with the fact that most of us aren't proficient in all of the world's languages.

Keeping translations up to date is not something I can do, and thus not my problem.

Quoted 4 lines

But, knowing who the translation maintainer is,
and when the translation was last updated allows both the original doc maintainer and the translation document maintainer to see when a document likely needs to be updated. And that is probably good enough.

Thank you. You've just scared away all potential language maintainers by making them think we want them to translate all the world's existing documentation. Do you know how long it takes just to read through the existing Documentation directory? (Have you done it?)

When I'm talking about language maintainers I am NOT looking for them to translate existing documentation or keep Documentation/* up to date for some other langauge. That's a more than full-time job. I've been looking around for weeks for someone to just accept Japanese patches, and everybody I've talked to says it's too much time commitment without any extra work piled on top of it in by armchair commentators who won't be _doing_ that work.

Quoted 1 line

gerrit

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#57 lore raw

Re: Documentation of kernel messages (Summary)

Hi Rob,

On Tue, 2007-07-10 at 12:12 -0400, Rob Landley wrote:

[snip]

Quoted 12 lines
Quoted 4 lines

Yeah, but it seems like having a translations directory in the kernel avoids that problem - anyone can update, it is a single source, no digging for sites that aren't tied to the kernel, available in the distros directly, etc.

No. It doesn't help.

99% of the kernel directory is C. That means any random passerby can review code. Everyone who has the kernel tarball should be able to review code that's in there, plus when you compile it breaks. So merging _code_ into the kernel helps keep it up to date.

Quoted 6 lines

Merging documentation into the kernel doesn't help keep it up to date, because documentation being out of date doesn't break the build. It may get the documentation more review, but the existing state of Documentation/* argues against that. It's a struggle to keep the english versions on the same continent as "up to date" or "complete", and most of the _good_ documentation is out in OLS papers and such (which I'm off indexing as we speak).

With the checker tool, which we suggested in the initial proposal, it is possible to verify

* that every marked message has a description
* that there are no descriptions without corresponding messages
* that format strings of message and description match

So when compiling the kernel using C=1, you will at least see warnings, when a message has changed or a message disappeared:

Quoted 1 line
Quoted 1 line

make modules C=1

  CHK     include/linux/version.h
  CHK     include/linux/utsrelease.h
  CHECK   drivers/kmsgtest/kmsgtest.c

drivers/kmsgtest/kmsgtest.c: Missing description for: kmsgtest.1 drivers/kmsgtest/kmsgtest.c: Description without message for: kmsgtest.3

Michael

#58 lore raw

Re: Documentation of kernel messages (Summary)

In reply to Rob Landley
Quoted 71 lines · Rob Landley
Quoted 6 lines · On Monday 09 July 2007 13:18:57 Gerrit Huizenga wrote:
Quoted 5 lines · Rob Landley

In regard to translating kernel messages:

Quoted 2 lines · On Monday 09 July 2007 01:36:31 H. Peter Anvin wrote:
Quoted 1 line · Takashi

(1) Your kernel development proposal will be greatly supported

by

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Japanese vendor community. At the same time, it needs support

from

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

the kernel communities, as well.

There is a very strong reason for the kernel community to NOT

support

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

this: it makes it much harder to deal with bug reports.

Agreed.

[...]

Quoted 1 line

As for the documentation translations themselves, I note that

Eric

Quoted 3 lines
Quoted 3 lines

Raymond dissuaded me from actually hosting translations of kernel documentation on http://kernel.org/doc due to his experience with translations of his own writings. If he hosts the translations on

his

Quoted 1 line
Quoted 1 line

website, they never get updated again. But if he makes the

contributor

Quoted 3 lines
Quoted 3 lines

host them on their own website, then they sometimes get updated.

For my part, I can't _tell_ when a given translation is out of

date

Quoted 1 line
Quoted 1 line

because I can't read it, and I certainly can't update it. So I

agree

Quoted 1 line
Quoted 1 line

with Eric and I'm linking to sites hosting kernel documentation in

other

Quoted 1 line
Quoted 1 line

languages rather than hosting them myself. I've got a good link

to a

Quoted 1 line
Quoted 1 line

Japanese site, need to ping the contributors of the chinese

documentation

Quoted 3 lines
Quoted 1 line

and see if they have a site for it...

Yeah, but it seems like having a translations directory in the

kernel

Quoted 1 line

avoids that problem - anyone can update, it is a single source, no

digging

Quoted 2 lines

for sites that aren't tied to the kernel, available in the distros directly, etc.

No. It doesn't help.

99% of the kernel directory is C. That means any random passerby can review
code. Everyone who has the kernel tarball should be able to review code
that's in there, plus when you compile it breaks. So merging _code_ into the
kernel helps keep it up to date.

Merging documentation into the kernel doesn't help keep it up to date, because
documentation being out of date doesn't break the build. It may get the
documentation more review, but the existing state of Documentation/* argues
against that. It's a struggle to keep the english versions on the same
continent as "up to date" or "complete", and most of the _good_ documentation
is out in OLS papers and such (which I'm off indexing as we speak).

Quoted 8 lines

Now add in the non-english nature of the new Documentation/stale subdirectories you're proposing. Almost nobody in the existing kernel
community can even _read_ this documentation, let alone update it. People
who update Documentation/* _can't_ update the translations, and that will
_never_change_.

There are quite a lot kernel developers for each of the popular language, AFAIK. For non-popular languages, there shouldn't be translation available in the first place. It was really a surprise when I post my Chinese translation on the LKML so many Chinese speakers have commented on the translation and encouraged the translation work. They are not visible as they usually don't talk too much. :) So I set up the Chinese kernel development maillist([email protected]), there will be more and more experienced kernel developer who can read and update the Chinese documents after they read the translated documentation and become kernel hacker.

Quoted 4 lines

Merging into the kernel is a great way to keep CODE up to date. Don't think
it's magic pixie dust for documentation. It never has been before.

IMHO, having contribution merged into the kernel has the MAGIC to attract people to work for recognition. When more and more people volunteer to work on it, the documentation will be up to date magically.

Quoted 5 lines
Quoted 2 lines

I'm not sure that the web site hosting argument is a good one.

Arch

Quoted 1 line

maintainers and Language Maintainers have some similarities.

Quoted 7 lines

Ah, but It's not a language maintainer's job to update documentation. It's
their job to ACCEPT PATCHES. Translate patches, translate questions back and
forth. This has NOTHING to do with documentation unless they're converting
documentation accompanying the patch one way; into english.

Language maintainers can integrate updates to the documentation just like integrating any updates to the code. Working on the documentation is ,IMHO, a perfect task for novice kernel hacker to get familiar with the process.

Quoted 26 lines
Quoted 3 lines

Also, being
able to clearly mark which version of a document was last

translated

Quoted 1 line

would help a bit with the fact that most of us aren't proficient in

all

Quoted 1 line

of the world's languages.

Keeping translations up to date is not something I can do, and thus not my
problem.

Quoted 2 lines

But, knowing who the translation maintainer is,
and when the translation was last updated allows both the original

doc

Quoted 1 line

maintainer and the translation document maintainer to see when a

document

Quoted 1 line

likely needs to be updated. And that is probably good enough.

Thank you. You've just scared away all potential language maintainers by
making them think we want them to translate all the world's existing documentation. Do you know how long it takes just to read through the
existing Documentation directory? (Have you done it?)

Quoted 11 lines

When I'm talking about language maintainers I am NOT looking for them to
translate existing documentation or keep Documentation/* up to date for some
other langauge. That's a more than full-time job. I've been looking around
for weeks for someone to just accept Japanese patches, and everybody I've
talked to says it's too much time commitment without any extra work piled on
top of it in by armchair commentators who won't be _doing_ that work.

It won't be too hard if the work is shared by a community. Like the one I'm trying to establish.

- Leo

Advertisement time:
Chinese kernel development community http://zh-kernel.org :)

#59 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 3 lines · Li Yang

There are quite a lot kernel developers for each of the popular language, AFAIK. For non-popular languages, there shouldn't be translation available in the first place.

I don't distinguish between "popular" and "non-popular" languages. If somebody's willing to do the translation work, it's popular. If nobody's willing to do the work, then even a language 1/3 of the planet's population speaks isn't "popular" for kernel development.

I wouldn't discourage a translator into Klingon if they were willing to keep their translation up to date and/or it actually resulted in patches.

Quoted 8 lines

It was really a surprise when
I post my Chinese translation on the LKML so many Chinese speakers have commented on the translation and encouraged the translation work. They are not visible as they usually don't talk too much. :) So I set up the Chinese kernel development maillist([email protected]), there will be more and more experienced kernel developer who can read and update the Chinese documents after they read the translated documentation and become kernel hacker.

The chinese mailing list is highly cool, and my first instinct was to note it on kernel.org/doc, but it would be better if the chinese website I already link to notes it instead. (That way I don't have to worry about how to spam-guard your email address. :)

This also highlights the need for language maintainers to help the patches go upstream to the english list. (If we can avoid armchair commentators saddling them with the task of translating the entire Documentation directory and keeping such a translation up to date, we might actually get one too. Fielding patches and questions sounds like plenty to me...)

Could you ask on said list if anyone is likely to volunteer for the position? (JUST translating patches and questions, as part of pushing code upstream.)

Quoted 7 lines
Quoted 3 lines

Merging into the kernel is a great way to keep CODE up to date. Don't think
it's magic pixie dust for documentation. It never has been before.

IMHO, having contribution merged into the kernel has the MAGIC to attract people to work for recognition. When more and more people volunteer to work on it, the documentation will be up to date magically.

Obvious counter-arguments include the floppy driver, the floppy tape driver, the tty layer, and most of the existing english Documentation directory...

I'll happily stay out of the way of people who actually want to merge translations of documentation into the kernel. I reserve the right to say "I told you so" in about five years.

Quoted 8 lines
Quoted 7 lines

Ah, but It's not a language maintainer's job to update documentation. It's
their job to ACCEPT PATCHES. Translate patches, translate questions back and
forth. This has NOTHING to do with documentation unless they're converting
documentation accompanying the patch one way; into english.

Quoted 6 lines

Language maintainers can integrate updates to the documentation just like integrating any updates to the code. Working on the documentation is ,IMHO, a perfect task for novice kernel hacker to get familiar with the process.

From a language maintainer's perspective documentation patches wouldn't be any

different than any other patches. Translate the description and questions going back and forth. The patch itself doesn't get translated when it's C and applies to scripts/kconfig/, why would it when it's UTF-8 and applies to Documentation/?

Of course this brings up the question "what kind of function and variable names do chinese people pick?" (I honestly don't know, but I note that attempts to use names that don't fit in 7-bit ascii would probably be frowned upon in a big way...)

Quoted 2 lines

It won't be too hard if the work is shared by a community. Like the one I'm trying to establish.

A list works fine as a point of contact. I note that in general, maintainers are individuals (who delegate like mad, of course), because otherwise agenda items languish with everyone thinking it's someone else's responsibility.

http://www.trumanlibrary.org/buckstop.htm

Quoted 4 lines

- Leo

Advertisement time:
Chinese kernel development community http://zh-kernel.org :)

Hmmm... Now I'm wondering if I should link directly to the docs pages of the chinese and japanese sites, or to the top level community pages?

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#60 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 10 lines · Rob Landley
Quoted 3 lines

IMHO, having contribution merged into the kernel has the MAGIC to attract people to work for recognition. When more and more people volunteer to work on it, the documentation will be up to date magically.

Obvious counter-arguments include the floppy driver, the floppy tape driver, the tty layer, and most of the existing english Documentation directory...

I'll happily stay out of the way of people who actually want to merge translations of documentation into the kernel. I reserve the right to say "I told you so" in about five years.

One thing that would be a VERY good idea is to make sure that each translated document is tagged with when it was last translated, and against which kernel version (using either a GIT commit id or a 2.6.x.y revision, I don't care), and who is responsible for keeping that particular document translated. That way, it becomes a lot easier to recognize when something is stale, and if has gotten stale, we can just delete it.

If someone keeping some files translated into Klingon up to date for 3 months, and then disappears, we can just delete it, and it's no big deal. You can even say "I told you so" when you submit the patch to delete the stale translation. :-)

Quoted 3 lines

A list works fine as a point of contact. I note that in general, maintainers are individuals (who delegate like mad, of course), because otherwise agenda items languish with everyone thinking it's someone else's responsibility.

I agree; if we're going to have a language translated in Documentation/<lang>, there ought to be one or two overall language maintainer, plus a maintainer for each file that is being translated, who is responsible for updating the translated file when the base English Documentation file is updated.

- Ted

#61 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 3 lines

One thing that would be a VERY good idea is to make sure that each translated document is tagged with when it was last translated, and against which kernel version (using either a GIT commit id or a

The English versions need a last updated too, that way we would know when they are past their best before date (as most of them are)

#62 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 6 lines · Alan Cox
Quoted 3 lines

One thing that would be a VERY good idea is to make sure that each translated document is tagged with when it was last translated, and against which kernel version (using either a GIT commit id or a

The English versions need a last updated too, that way we would know when they are past their best before date (as most of them are)

Good idea. So a "last updated" date field will be required for both the English and translated documents. Date will be more straight forward than using commit id, IMHO.

- Leo

#63 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 11 lines · Li Yang
Quoted 6 lines · Alan Cox
Quoted 3 lines

One thing that would be a VERY good idea is to make sure that each translated document is tagged with when it was last translated, and against which kernel version (using either a GIT commit id or a

The English versions need a last updated too, that way we would know when they are past their best before date (as most of them are)

Good idea. So a "last updated" date field will be required for both the English and translated documents. Date will be more straight forward than using commit id, IMHO.

Well, date and kernel version id, then. Otherwise they may be confusion if the translator is using 2.6.16.35 as opposed to 2.6.22-git3, etc. I would hope that the translator would always be using only the latest development branch, but I wouldn't want to always bet on that. One advantage of using git commit id is that there's no question about what the source document version was for the translation. It also makes it easier for the translator since they can just diff between the last git commit id, and HEAD, and then update their translation accordingly.

- Ted

#64 lore raw

Re: Documentation of kernel messages (Summary)

In reply to Alan Cox
Quoted 2 lines · Alan Cox

The English versions need a last updated too, that way we would know when they are past their best before date (as most of them are)

I've got the impression that whenever I saw a "last updated:" note in a documentation, this note was out of date.

Signature · 3 lines
Stefan Richter
-=====-=-=== -=== -==-=
http://arcgraph.de/sr/
#65 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 6 lines · Stefan Richter
Quoted 2 lines · Alan Cox

The English versions need a last updated too, that way we would know when they are past their best before date (as most of them are)

I've got the impression that whenever I saw a "last updated:" note in a documentation, this note was out of date.

RCS and SCCS let you stick stuff like $DATE and $VERSION into the source, so it would be automagically replaced at checkin/checkout time. Not sure how well that would work across a mesh of 'git pull' and 'git push' across trees. :)

#66 lore raw

Re: Documentation of kernel messages (Summary)

In reply to Alan Cox
Quoted 6 lines · Alan Cox
Quoted 3 lines

One thing that would be a VERY good idea is to make sure that each translated document is tagged with when it was last translated, and against which kernel version (using either a GIT commit id or a

The English versions need a last updated too, that way we would know when they are past their best before date (as most of them are)

You can get this out of git annotate. (Or in my case, hg annotate from
http://kernel.org/hg/linux-2.6 .)

Cleaning up Documentation is on my todo list, but for this month I'm trying to integrate the existing Documentation files with things like the OLS papers, linux journal articles, lwn.net kernel articles, man-pages, and so on into one big index. (Indexing it requires reading large chunks of it. My brain hurts.)

If there's interest, I can push some patches to clean up Documentation by moving files into subdirectories, but Documentation's not well-suited to link out to the web. (You need html for that, and it's text.)

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#67 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 9 lines · On Fri, 13 Jul 2007 14:05:49 -0400 Rob Landley wrote:

Cleaning up Documentation is on my todo list, but for this month I'm trying to integrate the existing Documentation files with things like the OLS papers, linux journal articles, lwn.net kernel articles, man-pages, and so on into one big index. (Indexing it requires reading large chunks of it. My brain hurts.)

If there's interest, I can push some patches to clean up Documentation by moving files into subdirectories, but Documentation's not well-suited to link out to the web. (You need html for that, and it's text.)

I think that you should start moving Documentation/ files around and adding subdirectories -- if you are pretty sure of where they should go (i.e., they won't likely be moved again later on).

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

#68 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 7 lines · Randy Dunlap
Quoted 3 lines

If there's interest, I can push some patches to clean up Documentation by moving files into subdirectories, but Documentation's not well-suited to link out to the web. (You need html for that, and it's text.)

I think that you should start moving Documentation/ files around and adding subdirectories -- if you are pretty sure of where they should go (i.e., they won't likely be moved again later on).

You mean like these?
http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#69 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 5 lines · On Sat, 14 Jul 2007 21:56:15 -0400 Rob Landley wrote:
Quoted 4 lines · Randy Dunlap
Quoted 3 lines

If there's interest, I can push some patches to clean up Documentation by moving files into subdirectories, but Documentation's not well-suited to link out to the web. (You need html for that, and it's text.)

Quoted 7 lines
Quoted 3 lines

I think that you should start moving Documentation/ files around and adding subdirectories -- if you are pretty sure of where they should go (i.e., they won't likely be moved again later on).

You mean like these?
http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html

Yes. Have any of these been merged?

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

#70 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 11 lines · Randy Dunlap
Quoted 10 lines · On Sat, 14 Jul 2007 21:56:15 -0400 Rob Landley wrote:
Quoted 8 lines · Randy Dunlap
Quoted 4 lines

If there's interest, I can push some patches to clean up Documentation by moving files into subdirectories, but Documentation's not well-suited to link out to the web. (You need html for that, and it's text.)

I think that you should start moving Documentation/ files around and adding subdirectories -- if you are pretty sure of where they should go (i.e., they won't likely be moved again later on).

Quoted 5 lines
Quoted 3 lines

You mean like these?
http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html

Yes. Have any of these been merged?

Not that I know of. They seemed to meet with resounding indifference, so I went on to other things...

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#71 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 12 lines · On Mon, 16 Jul 2007 15:53:06 -0400 Rob Landley wrote:
Quoted 11 lines · Randy Dunlap
Quoted 10 lines · On Sat, 14 Jul 2007 21:56:15 -0400 Rob Landley wrote:
Quoted 8 lines · Randy Dunlap
Quoted 4 lines

If there's interest, I can push some patches to clean up Documentation by moving files into subdirectories, but Documentation's not well-suited to link out to the web. (You need html for that, and it's text.)

I think that you should start moving Documentation/ files around and adding subdirectories -- if you are pretty sure of where they should go (i.e., they won't likely be moved again later on).

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

You mean like these?
http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html

Yes. Have any of these been merged?

Not that I know of. They seemed to meet with resounding indifference, so I went on to other things...

You need to be persistent. Please re-submit this.

and since it's just file & dir moves and done via git, it should go to Linus. However, he may not want it right now since the primary merge window for 2.6.23 has closed (although he has been merging some other doc updates recently).

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

#72 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 17 lines · Randy Dunlap
Quoted 16 lines · On Mon, 16 Jul 2007 15:53:06 -0400 Rob Landley wrote:
Quoted 15 lines · Randy Dunlap
Quoted 13 lines · On Sat, 14 Jul 2007 21:56:15 -0400 Rob Landley wrote:
Quoted 8 lines · Randy Dunlap
Quoted 4 lines

If there's interest, I can push some patches to clean up Documentation by moving files into subdirectories, but Documentation's not well-suited to link out to the web. (You need html for that, and it's text.)

I think that you should start moving Documentation/ files around and adding subdirectories -- if you are pretty sure of where they should go (i.e., they won't likely be moved again later on).

You mean like these?
http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Yes. Have any of these been merged?

Not that I know of. They seemed to meet with resounding indifference, so I went on to other things...

You need to be persistent. Please re-submit this.

Greg KH thinks it's a good idea to add language directories to the top level of Documentation, and there are slightly more than two of those:
http://www.translatorscafe.com/cafe/Help.asp?HelpID=59

Since you think it's worth doing, I'll resubmit the work I've already done here, but I no longer think trying to shovel Documentation into some semblance of order against the will of people like Greg is a good use of time.

These days I'm trying to create an html index that links into Documentation in a coherent order (with categories and everything), and using automated tools to detect files that aren't linked to, or links that point to a file that isn't there anymore. This is obviously still a work in progress, but I think it's a better approach.

Quoted 4 lines

and since it's just file & dir moves and done via git, it should go to Linus. However, he may not want it right now since the primary merge window for 2.6.23 has closed (although he has been merging some other doc updates recently).

If moving text files from one location to another in the Documentation directory breaks the build or causes bugs in the kernel, we have bigger problems. :)

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#73 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 25 lines · Rob Landley
Quoted 17 lines · Randy Dunlap
Quoted 16 lines · On Mon, 16 Jul 2007 15:53:06 -0400 Rob Landley wrote:
Quoted 15 lines · Randy Dunlap
Quoted 13 lines · On Sat, 14 Jul 2007 21:56:15 -0400 Rob Landley wrote:
Quoted 8 lines · Randy Dunlap
Quoted 4 lines

If there's interest, I can push some patches to clean up Documentation by moving files into subdirectories, but Documentation's not well-suited to link out to the web. (You need html for that, and it's text.)

I think that you should start moving Documentation/ files around and adding subdirectories -- if you are pretty sure of where they should go (i.e., they won't likely be moved again later on).

You mean like these?
http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Yes. Have any of these been merged?

Not that I know of. They seemed to meet with resounding indifference, so I went on to other things...

You need to be persistent. Please re-submit this.

Quoted 8 lines

Greg KH thinks it's a good idea to add language directories to the top level of Documentation, and there are slightly more than two of those:
http://www.translatorscafe.com/cafe/Help.asp?HelpID=59

Since you think it's worth doing, I'll resubmit the work I've already done here, but I no longer think trying to shovel Documentation into some semblance of order against the will of people like Greg is a good use of time.

My "will" is only that some of these documents be translated _and_ put in the main kernel source tree. I really have no opinion on where they need to go within that directory. If you have a better place for them, please let me know.

thanks,

greg k-h

#74 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 36 lines · Greg KH
Quoted 26 lines · Rob Landley
Quoted 18 lines · Randy Dunlap
Quoted 17 lines · On Mon, 16 Jul 2007 15:53:06 -0400 Rob Landley wrote:
Quoted 16 lines · Randy Dunlap
Quoted 14 lines · On Sat, 14 Jul 2007 21:56:15 -0400 Rob Landley wrote:
Quoted 9 lines · Randy Dunlap
Quoted 4 lines

If there's interest, I can push some patches to clean up Documentation by moving files into subdirectories, but Documentation's not well-suited to link out to the web. (You need html for that, and it's text.)

I think that you should start moving Documentation/ files around and adding subdirectories -- if you are pretty sure of where they should go (i.e., they won't likely be moved again later on).

You mean like these?
http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Yes. Have any of these been merged?

Not that I know of. They seemed to meet with resounding indifference, so I went on to other things...

You need to be persistent. Please re-submit this.

Quoted 8 lines

Greg KH thinks it's a good idea to add language directories to the top level of Documentation, and there are slightly more than two of those:
http://www.translatorscafe.com/cafe/Help.asp?HelpID=59

Since you think it's worth doing, I'll resubmit the work I've already done here, but I no longer think trying to shovel Documentation into some semblance of order against the will of people like Greg is a good use of time.

Quoted 4 lines

My "will" is only that some of these documents be translated _and_ put in the main kernel source tree. I really have no opinion on where they need to go within that directory. If you have a better place for them, please let me know.

A common subdirectory. I'd probably put it in "Documentation/translated" with a README at the top level of that explaining what it's for.

I can push a patch to move them, but reorganizing Documentation doesn't turn it into an index that can link out resources outside itself. If reorganizing it to be less of a compost heap is a separate goal in and of itself, I can do that...

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#75 lore raw

Re: Documentation of kernel messages (Summary)

In reply to Rob Landley
Quoted 6 lines · Rob Landley

These days I'm trying to create an html index that links into Documentation in a coherent order (with categories and everything), and using automated tools to detect files that aren't linked to, or links that point to a file that isn't there anymore. This is obviously still a work in progress, but I think it's a better approach.

Better than cleaning up what we have in the kernel source tree? Why not work on that first, then the "automated" type stuff would be much easier to do later, right?

thanks,

greg k-h

#76 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 8 lines · Greg KH
Quoted 5 lines · Rob Landley

These days I'm trying to create an html index that links into Documentation in a coherent order (with categories and everything), and using automated tools to detect files that aren't linked to, or links that point to a file that isn't there anymore. This is obviously still a work in progress, but I think it's a better approach.

Better than cleaning up what we have in the kernel source tree?

Yes, I just said that.

Quoted 2 lines

Why not work on that first, then the "automated" type stuff would be much easier to do later, right?

How does an automated 404 checker that identifies files nothing's linking to get easier if the source files are in a different order? They could be alphabetical in one big directory and that would work the same.

Moving Documentation around is pointless churn that does nothing to prevent you from adding language directories to the top level on a whim, as if there were only two instead of (as I pointed out last message), several dozen. (While Randy Dunlap's poking me to resubmit my patch to group the architecture documentation into an architecture subdirectory, you're adding language directories to the top level instead of in their own subdirectory.)

Documentation doesn't even cross-link to the output of make htmldocs (which has its own structure imposed on it due to being extracted from the kernel sources). The kernel tarball has _two_ documentation sources that don't significantly cross-reference each other, and Rusty just submitted "make Preparation!" for lguest that's totally unrelated to either of them (and starts from a README file buried in the source code). None of this links to the menuconfig help entries, and the only reference in Documentation/ to "make help" is in Documentation/kbuild/makefiles.txt which explains that its purpose is to list the available architecture targets (something it does not, in my experience, actually do).

The idea that the kernel Documentation directory is the master repository of kernel documentation is an unworkable fantasy. The Documentation directory cannot index for all the kernel documentation resources out on the web, because it's in text not html. Documentation was created on the assumption that it would contain all interesting resources, as text files, but that doesn't match reality. Documentation is merely one resource among many, and to link _out_ you need HTML.

Some of the resources out there are organized chronologically, such as the Linux Journal archives http://www.linuxjournal.com/xstatic/magazine/archives,
the Ottawa Linux Symposium papers (http://kernel.org/doc/ols), or the kernel-traffic archives
http://www.kernel-traffic.org/kernel-traffic/archives.html. Some have their own indexes, such as the Linux Weekly News kernel articles
http://lwn.net/Kernel/Index/ and the Linux Device Drivers book
http://lwn.net/Kernel/LDD3/. Some are random bits picked out of developer blogs, found with google, reasonably coherent articles on wikipedia. Some are huge self-contained lumps on specific topics such as Mel Gorman's mm documentation or lots of the embedded stuff.

So no matter what reorganization I do to Documentation, its structure (or lack thereof) is incidental to coherently indexing most of the kernel documentation that's out there. (Right now the best index is "google" but that's only useful if you know what questions to ask. But getting up-to-date versions of Documentation and the output of make htmldocs on the web lets google find it. (Last year, back when I was still working on BusyBox, I did a google trawl for ext2 documentation to replace the horrible mke2fs busybox used to have, and the first three pages of hits did _NOT_ include a copy of Documentation/filesystems/ext2.txt. Ironically, if you google for "ext2 documentation" today the sixth hit is the unfinished ext2 documentation I'd just started to write before I found Documentation/filesystems/ext2.txt.)

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#77 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 2 lines · Rob Landley

Documentation is merely one resource among many, and to link _out_ you need HTML.

Do you suggest HTML files in linux/Documentation? I think we can follow URIs and other references in plaintext files just fine.

(People who want clickable links in plaintext files can use plaintext viewers which implement this, or utilities or browser plugins which follow URLs in the clipboard.)

Signature · 3 lines
Stefan Richter
-=====-=-=== =--- --=--
http://arcgraph.de/sr/
#78 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 5 lines · Stefan Richter
Quoted 2 lines · Rob Landley

Documentation is merely one resource among many, and to link _out_ you need HTML.

Do you suggest HTML files in linux/Documentation?

I've thought about it, but right now Documentation is text. Converting all the existing text files to html is a fairly intrusive change, and having multiple file formats in there seems even more untidy. (What next, pdf?).

I'm aware there are some docbook files in there, plus an ancient graphic file, plus the bloody stains of various chicken sacrificies, but the point would be how best to clean it up...

Quoted 2 lines

I think we can follow
URIs and other references in plaintext files just fine.

Go for it. Have fun. I'll be over here.

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#79 lore raw

Re: Documentation of kernel messages (Summary)

In reply to Rob Landley
Quoted 2 lines · Rob Landley

I wouldn't discourage a translator into Klingon if they were willing to keep their translation up to date and/or it actually resulted in patches.

The guys at the Klingon Language Institute are going to have a fit - what's the Klingon word for 'freezer' as used in our suspend code? :)

#80 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 6 lines · On Wednesday 11 July 2007 5:53:54 pm [email protected] wrote:
Quoted 2 lines · Rob Landley

I wouldn't discourage a translator into Klingon if they were willing to keep their translation up to date and/or it actually resulted in patches.

The guys at the Klingon Language Institute are going to have a fit - what's the Klingon word for 'freezer' as used in our suspend code? :)

Considering that Klingon didn't have "to be" before Star Trek 6 needed hamlet, I suspect they'll manage somehow.

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#81 lore raw

RE: Documentation of kernel messages (Summary)

In reply to Rob Landley
Quoted 7 lines · Rob Landley
Quoted 3 lines · Li Yang

There are quite a lot kernel developers for each of the popular language, AFAIK. For non-popular languages, there shouldn't be translation available in the first place.

I don't distinguish between "popular" and "non-popular" languages. If somebody's willing to do the translation work, it's popular. If

nobody's

Quoted 1 line

willing to do the work, then even a language 1/3 of the planet's

population

Quoted 3 lines

speaks isn't "popular" for kernel development.

I wouldn't discourage a translator into Klingon if they were willing

to keep

Quoted 4 lines

their translation up to date and/or it actually resulted in patches.

Quoted 2 lines

It was really a surprise when
I post my Chinese translation on the LKML so many Chinese speakers

have

Quoted 1 line
Quoted 1 line

commented on the translation and encouraged the translation work.

They

Quoted 1 line
Quoted 1 line

are not visible as they usually don't talk too much. :) So I set up

the

Quoted 1 line
Quoted 1 line

Chinese kernel development maillist([email protected]),

there

Quoted 5 lines
Quoted 3 lines

will be more and more experienced kernel developer who can read and update the Chinese documents after they read the translated documentation and become kernel hacker.

The chinese mailing list is highly cool, and my first instinct was to

note it

Quoted 1 line

on kernel.org/doc, but it would be better if the chinese website I

already

Quoted 1 line

link to notes it instead. (That way I don't have to worry about how

to

Quoted 1 line

spam-guard your email address. :)

The benefit of a localized maillist is that patches can be reviewed and commented in native language for common problems (like style, API).

Quoted 2 lines

This also highlights the need for language maintainers to help the

patches go

Quoted 2 lines

upstream to the english list. (If we can avoid armchair commentators saddling them with the task of translating the entire Documentation

directory

Quoted 1 line

and keeping such a translation up to date, we might actually get one

too.

Quoted 1 line

Fielding patches and questions sounds like plenty to me...)

I do think the documentation translation is very necessary even when there is a language maintainer, especially for the policy documents as HOWTO, codestyle , and etc. The contributors should go through these policies and check their code for compliance before going to the language maintainer for help, or there will be too much for the language maintainer to translate. The language maintainer doesn't need to translate all the documents himself, but he can help to coordinate the translation effort and help to make it update to date.

Quoted 2 lines

Could you ask on said list if anyone is likely to volunteer for the

position?

Quoted 1 line

(JUST translating patches and questions, as part of pushing code

upstream.)

If we do need a contact person, I can do it. However I don't think there will be much translation work to do here. As I stated before, most Chinese programmers are more or less capable of read/write technical English. The difficult part is to let them know the benefit of merging code in kernel and teach them how to do it. That's why the policy documents in native language will be very useful.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Merging into the kernel is a great way to keep CODE up to date.

Don't

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

think
it's magic pixie dust for documentation. It never has been

before.

Quoted 4 lines
Quoted 4 lines

IMHO, having contribution merged into the kernel has the MAGIC to attract people to work for recognition. When more and more people volunteer to work on it, the documentation will be up to date

magically.

Quoted 2 lines

Obvious counter-arguments include the floppy driver, the floppy tape

driver,

Quoted 1 line

the tty layer, and most of the existing english Documentation

directory...

Quoted 3 lines

I'll happily stay out of the way of people who actually want to merge translations of documentation into the kernel. I reserve the right to

say "I

Quoted 3 lines

told you so" in about five years.

Quoted 1 line
Quoted 1 line

Ah, but It's not a language maintainer's job to update

documentation.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

It's
their job to ACCEPT PATCHES. Translate patches, translate

questions

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

back and
forth. This has NOTHING to do with documentation unless they're converting
documentation accompanying the patch one way; into english.

Language maintainers can integrate updates to the documentation just like integrating any updates to the code. Working on the

documentation

Quoted 1 line
Quoted 1 line

is ,IMHO, a perfect task for novice kernel hacker to get familiar

with

Quoted 3 lines
Quoted 1 line

the process.

From a language maintainer's perspective documentation patches

wouldn't be any

Quoted 1 line

different than any other patches. Translate the description and

questions

Quoted 1 line

going back and forth. The patch itself doesn't get translated when

it's C

Quoted 1 line

and applies to scripts/kconfig/, why would it when it's UTF-8 and

applies to

Quoted 3 lines

Documentation/?

Of course this brings up the question "what kind of function and

variable

Quoted 1 line

names do chinese people pick?" (I honestly don't know, but I note

that

Quoted 1 line

attempts to use names that don't fit in 7-bit ascii would probably be

frowned

Quoted 3 lines

upon in a big way...)

Quoted 1 line

It won't be too hard if the work is shared by a community. Like the

one

Quoted 3 lines
Quoted 1 line

I'm trying to establish.

A list works fine as a point of contact. I note that in general,

maintainers

Quoted 1 line

are individuals (who delegate like mad, of course), because otherwise

agenda

Quoted 1 line

items languish with everyone thinking it's someone else's

responsibility.

Got it. I can do it as long as it doesn't consume too much time. :)

Quoted 9 lines

http://www.trumanlibrary.org/buckstop.htm

Quoted 4 lines

- Leo

Advertisement time:
Chinese kernel development community http://zh-kernel.org :)

Hmmm... Now I'm wondering if I should link directly to the docs pages

of the

Quoted 1 line

chinese and japanese sites, or to the top level community pages?

The documents and maillist are also linked in the front page. It's better to link the top level page.

- Leo

#82 lore raw

PATCH 1/1 Chinese Language Maintainer

Proposed patch adding Li Yang to MAINTAINERS, and Documentation describing what a language maintainer is.

--

Quoted 10 lines · On Thursday 12 July 2007 9:53:54 am Li Yang-r58472 wrote:
Quoted 3 lines
Quoted 1 line

I'm trying to establish.

A list works fine as a point of contact. I note that in general,

maintainers

Quoted 1 line

are individuals (who delegate like mad, of course), because otherwise

agenda

Quoted 1 line

items languish with everyone thinking it's someone else's

responsibility.

Got it. I can do it as long as it doesn't consume too much time. :)

Woot! See attached patch and sign-off-by if you want the job. :)

The bit about "the maintainer forwards, not the list" is a suggestion to prevent duplicates. It doesn't mean you can't delegate, just that if it's open season on the list translating stuff _and_ forwarding it, then linux-kernel will get dupes. (Plus other maintainers are more likely to pay attention to somebody they've heard from before.)

Greg KH had entire an OLS presentation about how the developer -> maintainer -> subsystem -> anderew+linus forwarding is more an ideal than a reality, but at least it gives us a frame of reference to diverge from:
http://lwn.net/Articles/240402/

MAINTAINERS +7 -0
diff -r edfd2d6f670d MAINTAINERS
@@ -2146,6 +2146,13 @@ W:	http://auxdisplay.googlepages.com/
 W:	http://auxdisplay.googlepages.com/
 S:	Maintained
 
+LANGUAGE (CHINESE)
+P:	Li Yang
+M:	[email protected]
+L:	[email protected]
+W:	http://zh-kernel.org
+S:	Maintained
+
 LAPB module
 L:	[email protected]
 S:	Orphan
Documentation/language-maintainers.txt new file +20 -0
@@ -0,0 +1,20 @@
+The langauges in which the Linux kernel is developed are C and English.
+(Note that Linus Torvalds' native language is Swedish, yet he chose
+to hold technical discussions in English.)
+
+The job of a language maintainer is to provide a point of contact for
+non-english speaking developers who wish to merge their patches into the
+Linux kernel.  Each language needs a specific language maintainer, who
+accepts non-english patch submissions on behalf of the Linux kernel.
+
+A language maintainer accepts patches to the Linux kernel, written in C, from
+authors who do not also speak English.  The language maintainer translates the
+description of each patch into English, forwards the patches to linux-kernel
+and to the appropriate maintainers for inclusion in the Linux kernel, and
+translates questions and replies about such patches as part of the
+patch review process.
+
+Some language maintainers provide a mailing list as a point of contact, to
+distribute the translation work, but the maintainer is still the person who
+ultimately forwards the results (to prevent duplicates), and the one to contact
+if patches and questions don't get translated and forwarded in a timely fashion.
Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#83 lore raw

Re: [PATCH] Chinese Language Maintainer

Documentation/language-maintainers.txt line 1

Quoted 4 lines · Rob Landley
Documentation/language-maintainers.txt new file +1 -0
@@ -0,0 +1,20 @@
+The langauges in which the Linux kernel is developed are C and English.
       ^^^^^^^^^
       languages

Documentation/language-maintainers.txt line 8

Quoted 7 lines
Documentation/language-maintainers.txt new file +7 -0
+(Note that Linus Torvalds' native language is Swedish, yet he chose
+to hold technical discussions in English.)
+
+The job of a language maintainer is to provide a point of contact for
+non-english speaking developers who wish to merge their patches into the
+Linux kernel.  Each language needs a specific language maintainer, who
+accepts non-english patch submissions on behalf of the Linux kernel.
           ^^^^^^^^^^^
	   non-English

Documentation/language-maintainers.txt line 11

Quoted 3 lines
Documentation/language-maintainers.txt new file +3 -0
+
+A language maintainer accepts patches to the Linux kernel, written in C, from
+authors who do not also speak English.  The language maintainer translates the

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#84 lore raw

Re: [PATCH] Chinese Language Maintainer

In reply to Rob Landley

Hi!

Documentation/language-maintainers.txt line 2

Quoted 3 lines
Documentation/language-maintainers.txt new file +2 -0
@@ -0,0 +1,20 @@
+The langauges in which the Linux kernel is developed are C and English.
+(Note that Linus Torvalds' native language is Swedish, yet he chose

choose?

Documentation/language-maintainers.txt line 15

Quoted 6 lines
Documentation/language-maintainers.txt new file +6 -0
+A language maintainer accepts patches to the Linux kernel, written in C, from
+authors who do not also speak English.  The language maintainer translates the
+description of each patch into English, forwards the patches to linux-kernel
+and to the appropriate maintainers for inclusion in the Linux kernel, and
+translates questions and replies about such patches as part of the
+patch review process.

Poor person... I'm not sure this can work for non-trivial patches.

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#85 lore raw

Re: [PATCH] Chinese Language Maintainer

Quoted 7 lines · Pavel Machek

Documentation/language-maintainers.txt line 2

Quoted 3 lines
Documentation/language-maintainers.txt new file +2 -0
@@ -0,0 +1,20 @@
+The langauges in which the Linux kernel is developed are C and English.
+(Note that Linus Torvalds' native language is Swedish, yet he chose

choose?

From `dict choose':

Choose \Choose\, v. t. [imp. {Chose}; p. p. {Chosen}, {Chose}
(Obs.); p. pr. & vb. n. {Choosing}.] [OE. chesen, cheosen, AS. ce['o]san; akin to OS. kiosan, D. kiezen, G. kiesen, Icel. kj[=o]sa, Goth. kiusan, L. gustare to taste, Gr. ?, Skr. jush to enjoy. [root]46. Cf. {Choice}, 2d {Gust}.]

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#86 lore raw

Re: [PATCH] Chinese Language Maintainer

In reply to Pavel Machek
Quoted 7 lines · Pavel Machek

Hi!

Documentation/language-maintainers.txt line 2

Quoted 3 lines
Documentation/language-maintainers.txt new file +2 -0
@@ -0,0 +1,20 @@
+The langauges in which the Linux kernel is developed are C and English.
+(Note that Linus Torvalds' native language is Swedish, yet he chose

choose?

I was trying for past tense.

Quoted 8 lines
Quoted 6 lines
(quoted diff) +1 -0
+A language maintainer accepts patches to the Linux kernel, written in C,

from +authors who do not also speak English. The language maintainer translates the +description of each patch into English, forwards the patches to linux-kernel +and to the appropriate maintainers for inclusion in the Linux kernel, and +translates questions and replies about such patches as part of the +patch review process.

Poor person... I'm not sure this can work for non-trivial patches.

We'll see. Personally, I'm more worried about how people who don't speak english are supposed to choose function and variable names. But translating _just_ the patch is not the end of it, and I wanted to make that clear.

I suppose a translator can always push back and ask for a _concise_ review, for the sake of translation. But let's not invent problems before they actually occur...

Quoted 1 line

Pavel

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#87 lore raw

Re: [PATCH] Chinese Language Maintainer

In reply to Rob Landley
Quoted 4 lines · Rob Landley

Proposed patch adding Li Yang to MAINTAINERS, and Documentation describing what a language maintainer is.

Quoted 55 lines

--

Quoted 10 lines · On Thursday 12 July 2007 9:53:54 am Li Yang-r58472 wrote:
Quoted 3 lines
Quoted 1 line

I'm trying to establish.

A list works fine as a point of contact. I note that in general,

maintainers

Quoted 1 line

are individuals (who delegate like mad, of course), because otherwise

agenda

Quoted 1 line

items languish with everyone thinking it's someone else's

responsibility.

Got it. I can do it as long as it doesn't consume too much time. :)

Woot! See attached patch and sign-off-by if you want the job. :)

The bit about "the maintainer forwards, not the list" is a suggestion to prevent duplicates. It doesn't mean you can't delegate, just that if it's open season on the list translating stuff _and_ forwarding it, then linux-kernel will get dupes. (Plus other maintainers are more likely to pay attention to somebody they've heard from before.)

Greg KH had entire an OLS presentation about how the developer -> maintainer -> subsystem -> anderew+linus forwarding is more an ideal than a reality, but at least it gives us a frame of reference to diverge from:
http://lwn.net/Articles/240402/

diff -r edfd2d6f670d MAINTAINERS

MAINTAINERS +7 -0
@@ -2146,6 +2146,13 @@ W:	http://auxdisplay.googlepages.com/
 W:	http://auxdisplay.googlepages.com/
 S:	Maintained
 
+LANGUAGE (CHINESE)
+P:	Li Yang
+M:	[email protected]
+L:	[email protected]
+W:	http://zh-kernel.org
+S:	Maintained
+
 LAPB module
 L:	[email protected]
 S:	Orphan
Documentation/language-maintainers.txt new file +5 -0
@@ -0,0 +1,20 @@
+The langauges in which the Linux kernel is developed are C and English.
+(Note that Linus Torvalds' native language is Swedish, yet he chose
+to hold technical discussions in English.)
+
+The job of a language maintainer is to provide a point of contact for

Documentation/language-maintainers.txt line 15

Quoted 10 lines
Documentation/language-maintainers.txt new file +10 -0
+non-english speaking developers who wish to merge their patches into the
+Linux kernel.  Each language needs a specific language maintainer, who
+accepts non-english patch submissions on behalf of the Linux kernel.
+
+A language maintainer accepts patches to the Linux kernel, written in C, from
+authors who do not also speak English.  The language maintainer translates the
+description of each patch into English, forwards the patches to linux-kernel
+and to the appropriate maintainers for inclusion in the Linux kernel, and
+translates questions and replies about such patches as part of the
+patch review process.

In addiction to this responsibility, I would like to add two more which, in my opinion, are more important. And these are what I'm trying to do. :)
First, promoting contribution to Linux kernel in local language. Second, coordinate the translation effort of key kernel documents.

Quoted 6 lines
Documentation/language-maintainers.txt new file +5 -0
+
+Some language maintainers provide a mailing list as a point of contact, to
+distribute the translation work, but the maintainer is still the person who
+ultimately forwards the results (to prevent duplicates), and the one to contact
+if patches and questions don't get translated and forwarded in a timely fashion.
#88 lore raw

Re: [PATCH] Chinese Language Maintainer

Quoted 7 lines · Li Yang
Quoted 4 lines · Rob Landley

Proposed patch adding Li Yang to MAINTAINERS, and Documentation describing what a language maintainer is.

Yay!

Quoted 7 lines
Quoted 6 lines

+A language maintainer accepts patches to the Linux kernel, written in C, from +authors who do not also speak English. The language maintainer translates the +description of each patch into English, forwards the patches to linux-kernel +and to the appropriate maintainers for inclusion in the Linux kernel, and +translates questions and replies about such patches as part of the +patch review process.

Quoted 5 lines

In addiction to this responsibility, I would like to add two more which, in my opinion, are more important. And these are what I'm trying to do. :)
First, promoting contribution to Linux kernel in local language. Second, coordinate the translation effort of key kernel documents.

Cool. It's good to do that, but not the problem I'm worried about solving.

I was trying to describe the minimum requirements for being a language maintainer, I.E. what non-english users need in order to be able to merge their patches. Because without someone to contribute patches to (I.E a language maintainer), documentation in non-english languages promotes the creation of patches that can't be merged. That's the problem I'm trying to solve.

To me, finding language maintainers is the flip side of the coin of translating documentation.

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#89 lore raw

Re: [PATCH] Chinese Language Maintainer

Quoted 24 lines · Rob Landley
Quoted 8 lines · Li Yang
Quoted 6 lines · Rob Landley

+A language maintainer accepts patches to the Linux kernel, written in C, from +authors who do not also speak English. The language maintainer translates the +description of each patch into English, forwards the patches to linux-kernel +and to the appropriate maintainers for inclusion in the Linux kernel, and +translates questions and replies about such patches as part of the +patch review process.

Quoted 5 lines

In addiction to this responsibility, I would like to add two more which, in my opinion, are more important. And these are what I'm trying to do. :)
First, promoting contribution to Linux kernel in local language. Second, coordinate the translation effort of key kernel documents.

Cool. It's good to do that, but not the problem I'm worried about solving.

I was trying to describe the minimum requirements for being a language maintainer, I.E. what non-english users need in order to be able to merge their patches. Because without someone to contribute patches to (I.E a language maintainer), documentation in non-english languages promotes the creation of patches that can't be merged. That's the problem I'm trying to solve.

Quoted 2 lines

To me, finding language maintainers is the flip side of the coin of translating documentation.

I think you worried too much about this problem. :) Let me explain the situation here in China more clearly. Actually, English is mandatory in most schools and universities. Only very few people learn other language as a second language. Therefore software developers who are almost educated should have the basic English skill. However, that doesn't mean that they can read English or communicate with native English speaker very easily. Consider your second language learn in school for analogy. Read in English will be much slower and more likely to cause misunderstanding. This will reduce the likelihood greatly of English documentation being read. If we are promoting contribution to the Linux community, we should maximum the possibility that these key documents being read. Translation will serve this purpose very well.

So the possibility is very little that a translator is needed between the Linux maintainer and a Chinese developer. Although sometimes help is needed when there is misunderstanding.

After a brief talk with the Japanese translator, I think the case is similar for Japanese too.

Therefore, in my opinion, language maintainer should be more a helper and promoter rather than a gatekeeper. I will give a proposed process later about how this helper mechanism works.

- Leo

#90 lore raw

Re: [PATCH] Chinese Language Maintainer

Quoted 9 lines · Li Yang

I think you worried too much about this problem. :) Let me explain the situation here in China more clearly. Actually, English is mandatory in most schools and universities. Only very few people learn other language as a second language. Therefore software developers who are almost educated should have the basic English skill. However, that doesn't mean that they can read English or communicate with native English speaker very easily. Consider your second language learn in school for analogy.

Actually, I disagree. English *is* the second language learned in school for most European developers (except, obviously, the ones from the British isles), and we don't have that problem.

Quoted 6 lines

Read in English will be much slower and more likely to cause misunderstanding. This will
reduce the likelihood greatly of English documentation being read. If we are promoting contribution to the Linux community, we should maximum the possibility that these key documents being read. Translation will serve this purpose very well.

What we have found in Europe, is that that it has limited value, and that the closer to the core you are, the less value it is, because at that stage you should be communicating more with other developers. Putting yourself behind a wall of translation is unfortunately a detriment in that way.

-hpa

#91 lore raw

Re: [PATCH] Chinese Language Maintainer

O> Actually, I disagree. English *is* the second language learned in

Quoted 2 lines

school for most European developers (except, obviously, the ones from the British isles), and we don't have that problem.

Not all those from the British Isles. A first language an English as school language is not that uncommon for segments of the population, and in Wales schooling may also be such that English is learned as a first foreign language.

That aside, please remember that Europe as a whole is a small place on the bigger world stage. The total volume of potential developers in the huge and rapidly modernising nations like India and China is vast, and there are large highly skilled technical nations that don't teach English to everyone technical by default.

Key documents in other languages is a big help, especially those about values, culture and standards because they are things that are not easy to understand if your use of English is technically oriented.

Alan

#92 lore raw

Re: [PATCH] Chinese Language Maintainer

Quoted 15 lines · Alan Cox

O> Actually, I disagree. English *is* the second language learned in

Quoted 2 lines

school for most European developers (except, obviously, the ones from the British isles), and we don't have that problem.

Not all those from the British Isles. A first language an English as school language is not that uncommon for segments of the population, and in Wales schooling may also be such that English is learned as a first foreign language.

That aside, please remember that Europe as a whole is a small place on the bigger world stage. The total volume of potential developers in the huge and rapidly modernising nations like India and China is vast, and there are large highly skilled technical nations that don't teach English to everyone technical by default.

Quoted 3 lines

Key documents in other languages is a big help, especially those about values, culture and standards because they are things that are not easy to understand if your use of English is technically oriented.

No doubt. My point was merely that the closer to the core of development, the less translated documents help as the emphasis on interaction *has* to increase.

In that sense, a translation of "Linux Kernel Internals" and the other books written on the Linux kernel is more useful. That doesn't mean, of course, that there isn't documentation distributed with the kernel that is intended for users and therefore more useful in translation. However, I do feel that trying to keep up-to-date translations of design documentation is at least to some degree a fool's errand, which ends up having people rely on incomplete and outdated documentation, and cut them off from the overall community.

-hpa

#93 lore raw

Re: [PATCH] Chinese Language Maintainer

Quoted 8 lines · H. Peter Anvin
Quoted 3 lines · Alan Cox

Key documents in other languages is a big help, especially those about values, culture and standards because they are things that are not easy to understand if your use of English is technically oriented.

In that sense, a translation of "Linux Kernel Internals" and the other books written on the Linux kernel is more useful.

Several groups of people are translating the "important bits" (a different selection for each culture) of the kernelnewbies documentation to different languages already:

http://kernelnewbies.org/RegionalNewbies

I can easily set up additional wikis for anybody who wants to set up such a kernel documentation site in their language.

Signature · 3 lines
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.
#94 lore raw

Re: [PATCH] Chinese Language Maintainer

In reply to Alan Cox
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Alan Cox <[email protected]> writes:

That aside, please remember that Europe as a whole is a small place on the bigger world stage. The total volume of potential developers in the huge and rapidly modernising nations like India and China is vast, and there are large highly skilled technical nations that don't teach English to everyone technical by default.

Speaking for India only, while the medium of instruction in primary and high school is available in native languages, English is almost universally preferred. Moreover, technical education is completely in English. I don't see much value in translating developer documentation into Indian languages. User documentation is another story.

Ganesan

Signature · 1 line
Ganesan Rajagopal
#95 lore raw

Re: [PATCH] Chinese Language Maintainer

Quoted 3 lines

preferred. Moreover, technical education is completely in English. I don't see much value in translating developer documentation into Indian languages. User documentation is another story.

I am not convinced that users do not become developers, nor that we should have language barriers that make this unneccessarily hard.

Alan

#96 lore raw

Re: [PATCH] Chinese Language Maintainer

Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

Alan Cox <[email protected]> writes:

preferred. Moreover, technical education is completely in English. I don't see much value in translating developer documentation into Indian languages. User documentation is another story.

I am not convinced that users do not become developers, nor that we should have language barriers that make this unneccessarily hard.

Again speaking only from the Indian context, we're somewhat unique when compared to China and Japan ;-). It's extremely unlikely that a non-english speaking user will become a developer. As I mentioned in my previous mail technical education is almost completely in English.

Most Indian IT work places communicate almost exclusively in English, including verbal communication because of employees from all over India with different native languages. English is not a language barrier for developers or even technical users from India.

Ganesan

Signature · 1 line
Ganesan Rajagopal
#97 lore raw

Re: [PATCH] Chinese Language Maintainer

In reply to H. Peter Anvin
Quoted 11 lines · H. Peter Anvin
Quoted 9 lines · Li Yang

I think you worried too much about this problem. :) Let me explain the situation here in China more clearly. Actually, English is mandatory in most schools and universities. Only very few people learn other language as a second language. Therefore software developers who are almost educated should have the basic English skill. However, that doesn't mean that they can read English or communicate with native English speaker very easily. Consider your second language learn in school for analogy.

Quoted 3 lines

Actually, I disagree. English *is* the second language learned in school for most European developers (except, obviously, the ones from the British isles), and we don't have that problem.

Then this could be the main reason for the disagreement we have on this topic. Many people here don't grasp English very well even after the English classes.

- Leo

#98 lore raw

Re: [PATCH] Chinese Language Maintainer

In reply to H. Peter Anvin
Quoted 11 lines · H. Peter Anvin
Quoted 8 lines · Li Yang

I think you worried too much about this problem. :) Let me explain the situation here in China more clearly. Actually, English is mandatory in most schools and universities. Only very few people learn other language as a second language. Therefore software developers who are almost educated should have the basic English skill. However, that doesn't mean that they can read English or communicate with native English speaker very easily. Consider your second language learn in school for analogy

Quoted 3 lines

Actually, I disagree. English *is* the second language learned in school for most European developers (except, obviously, the ones from the British isles), and we don't have that problem.

There is somewhat of a difference though: English and both our native languages are very much related. English, Dutch (my native language) and Swedish (yours) are all Germanic languages -- English and Dutch both West Germanic and Swedish only slightly farther removed as North Germanic. Word for word translations not _too_ infrequently actually make okay or at least understandable Dutch...

Other than Germanic, the Slavic (Polish, Czech, Russian, ...) and Latin (Italian, Spanish, French, ...) language families are two other direct Indo-European descendants that are fairly well represented in the kernel community but I believe it's a largely unsurprising observation that members of both these families have a somewhat harder time adopting English. And Chinese is not even Indo-European...

However -- in Europe I notice culture might be even more important than school _or_ language family is and as such I believe the above argument isn't all that important anyway. English is the second language we learn in school here in the Netherlands but it's much more popular than say German, a language even closer to Dutch, due to us having heard English basically from the time we're old enough to hear music and watch TV. Most _Dutch_ bands sing in English and the ones that don't are for a part targetting the elderly and/or mentally handicapped...

German is close enough to Dutch (and English itself) to rule out most differences related to native language, but in Germany a significantly smaller percentage of people speaks English well enough to be comfortable with it than in the Netherlands simply due to them producing more of their own culture. It's a larger language zone to market to and they dub the remaining English-language content on TV. Over here in the Netherlands we sub-title.

France is another good example. While a bit farther removed from English family-wise, English has had lots of influences from French as well and in any case, learning English shouldn't be harder for a Frenchman than learning French is for a Dutchman (we are, or were when I was there, taught French in school as well) which is to say not very hard. Yet, mastery of English is extremely poor in France. Not as a coincedence, most _all_ of the French culture is in the French language including dubbed originally English songs for example.

Both the German and, slower, French examples get less true with every new generation, but both still hold...

Popular culture in the sense of music, tv and these days very much games is something you start to experience at a very young age, years younger than the Chinese will be taught English in school and age is extremely important in mastering a language -- the human brain is by far best at it at the time you start mastering your native language (in fact, this is what defines "native") and every single year after that makes it harder.

That is -- let's just solve our Chinese translation problem by overthrowing the Chinese government and forcing the sub-titled Harry Potter film-series down the throat of the population... |-/

Seriously, I only wanted to say that both language family and (import of) culture are very important and as such, concentrating on schooling alone might not be all that sensible. I did not want to say that I feel that all these translations make a great deal of sense. Some of us have an easier time learning English than others do, the Chinese probably don't have an easy time at all, but a single common language is still the thing to aim for. The subset and semblance of English spoken on this list seems like something that should serve well as that common language, especially given the _help_ English language education gives in it. Also, I hear China is in fact fairly rapidly opening up to Western (American) popular culture which might be an argument if you have a generation or two to spare...

Apologies for ranting...

Cheers,
Rene.

#99 lore raw

Re: [PATCH] Chinese Language Maintainer

In reply to H. Peter Anvin

What I would like to see is the Americans learning English as a FIRST language.

Keith (England)

----- Original Message -----
From: "H. Peter Anvin" <[email protected]>
To: "Li Yang" <[email protected]>
Cc: "Rob Landley" <[email protected]>; "Gerrit Huizenga" <[email protected]>; "Kunai, Takashi" <[email protected]>; <[email protected]>; "Andrew Morton" <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>; <[email protected]>
Sent: Sunday, July 15, 2007 7:12 PM
Subject: Re: [PATCH] Chinese Language Maintainer

Quoted 33 lines
Quoted 9 lines · Li Yang

I think you worried too much about this problem. :) Let me explain the situation here in China more clearly. Actually, English is mandatory in most schools and universities. Only very few people learn other language as a second language. Therefore software developers who are almost educated should have the basic English skill. However, that doesn't mean that they can read English or communicate with native English speaker very easily. Consider your second language learn in school for analogy.

Actually, I disagree. English *is* the second language learned in school for most European developers (except, obviously, the ones from the British isles), and we don't have that problem.

Quoted 6 lines

Read in English will be much slower and more likely to cause misunderstanding. This will
reduce the likelihood greatly of English documentation being read. If we are promoting contribution to the Linux community, we should maximum the possibility that these key documents being read. Translation will serve this purpose very well.

What we have found in Europe, is that that it has limited value, and that the closer to the core you are, the less value it is, because at that stage you should be communicating more with other developers. Putting yourself behind a wall of translation is unfortunately a detriment in that way.

(quoted diff) +0 -2
-hpa
-

To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#100 lore raw

Re: [PATCH] Chinese Language Maintainer

In reply to Li Yang
Quoted 42 lines · leo
Quoted 24 lines · Rob Landley
Quoted 8 lines · Li Yang
Quoted 6 lines · Rob Landley

+A language maintainer accepts patches to the Linux kernel, written in C, from +authors who do not also speak English. The language maintainer translates the +description of each patch into English, forwards the patches to linux-kernel +and to the appropriate maintainers for inclusion in the Linux kernel, and +translates questions and replies about such patches as part of the +patch review process.

Quoted 5 lines

In addiction to this responsibility, I would like to add two more which, in my opinion, are more important. And these are what I'm trying to do. :)
First, promoting contribution to Linux kernel in local language. Second, coordinate the translation effort of key kernel documents.

Cool. It's good to do that, but not the problem I'm worried about solving.

I was trying to describe the minimum requirements for being a language maintainer, I.E. what non-english users need in order to be able to merge their patches. Because without someone to contribute patches to (I.E a language maintainer), documentation in non-english languages promotes the creation of patches that can't be merged. That's the problem I'm trying to solve.

Quoted 2 lines

To me, finding language maintainers is the flip side of the coin of translating documentation.

I think you worried too much about this problem. :) Let me explain the situation here in China more clearly. Actually, English is mandatory in most schools and universities. Only very few people learn other language as a second language. Therefore software developers who are almost educated should have the basic English skill. However, that doesn't mean that they can read English or communicate with native English speaker very easily. Consider your second language learn in school for analogy. Read in English will be much slower and more likely to cause misunderstanding. This will reduce the likelihood greatly of English documentation being read. If we are promoting contribution to the Linux community, we should maximum the possibility that these key documents being read. Translation will serve this purpose very well.

Quoted 6 lines

So the possibility is very little that a translator is needed between the Linux maintainer and a Chinese developer. Although sometimes help is needed when there is misunderstanding.

After a brief talk with the Japanese translator, I think the case is similar for Japanese too.

Yes, In Japan, situation is mostly the same.
We are trying to increase number of Linux community developer with Linux Foundation Japan or CELF people in Japan.
In our discussion, the problem is not only Language.

In case of some developer, once he step forward (he try to send patch or comment on LKML), he got some comment and he can work with community even if it's slow (because of he was non-native). So, I thought if some key document are available in Japanese like HOWTO, that will help such early stage of developers.

Quoted 3 lines

Therefore, in my opinion, language maintainer should be more a helper and promoter rather than a gatekeeper. I will give a proposed process later about how this helper mechanism works.

I will be able to help this as a stand point of Japanese situation.

Quoted 8 lines

- Leo
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#101 lore raw

Re: [PATCH] Chinese Language Maintainer

Quoted 6 lines · Tsugikazu Shibata

Yes, In Japan, situation is mostly the same.
We are trying to increase number of Linux community developer with Linux Foundation Japan or CELF people in Japan.
In our discussion, the problem is not only Language.

Quoted 6 lines

In case of some developer, once he step forward (he try to send patch or comment on LKML), he got some comment and he can work with community even if it's slow (because of he was non-native). So, I thought if some key document are available in Japanese like HOWTO, that will help such early stage of developers.

I think you're absolutely right about that. The stuff that will help most of have translated is (kernel)newbie type documentation, the stuff one wants when being introduced to the community.

-hpa

#102 lore raw

Re: [PATCH] Chinese Language Maintainer

In reply to Tsugikazu Shibata
Quoted 58 lines · Tsugikazu Shibata
Quoted 39 lines · leo
Quoted 22 lines · Rob Landley
Quoted 7 lines · Li Yang
Quoted 6 lines · Rob Landley

+A language maintainer accepts patches to the Linux kernel, written in C, from +authors who do not also speak English. The language maintainer translates the +description of each patch into English, forwards the patches to linux-kernel +and to the appropriate maintainers for inclusion in the Linux kernel, and +translates questions and replies about such patches as part of the +patch review process.

Quoted 5 lines

In addiction to this responsibility, I would like to add two more which, in my opinion, are more important. And these are what I'm trying to do. :)
First, promoting contribution to Linux kernel in local language. Second, coordinate the translation effort of key kernel documents.

Cool. It's good to do that, but not the problem I'm worried about solving.

I was trying to describe the minimum requirements for being a language maintainer, I.E. what non-english users need in order to be able to merge their patches. Because without someone to contribute patches to (I.E a language maintainer), documentation in non-english languages promotes the creation of patches that can't be merged. That's the problem I'm trying to solve.

Quoted 2 lines

To me, finding language maintainers is the flip side of the coin of translating documentation.

I think you worried too much about this problem. :) Let me explain the situation here in China more clearly. Actually, English is mandatory in most schools and universities. Only very few people learn other language as a second language. Therefore software developers who are almost educated should have the basic English skill. However, that doesn't mean that they can read English or communicate with native English speaker very easily. Consider your second language learn in school for analogy. Read in English will be much slower and more likely to cause misunderstanding. This will reduce the likelihood greatly of English documentation being read. If we are promoting contribution to the Linux community, we should maximum the possibility that these key documents being read. Translation will serve this purpose very well.

Quoted 6 lines

So the possibility is very little that a translator is needed between the Linux maintainer and a Chinese developer. Although sometimes help is needed when there is misunderstanding.

After a brief talk with the Japanese translator, I think the case is similar for Japanese too.

Yes, In Japan, situation is mostly the same.
We are trying to increase number of Linux community developer with Linux Foundation Japan or CELF people in Japan.
In our discussion, the problem is not only Language.

In case of some developer, once he step forward (he try to send patch or comment on LKML), he got some comment and he can work with community even if it's slow (because of he was non-native). So, I thought if some key document are available in Japanese like HOWTO, that will help such early stage of developers.

Quoted 5 lines
Quoted 3 lines

Therefore, in my opinion, language maintainer should be more a helper and promoter rather than a gatekeeper. I will give a proposed process later about how this helper mechanism works.

I will be able to help this as a stand point of Japanese situation.

Here is the helper process I propose to help more people to participate.
Suggestions and comments are welcomed.

1) Developer who can't speak English or has a problem going through the submission process sends patches to the Language maintainer and cc the local mail list provided.
2) Language maintainer and experienced Linux developer on the local mail list can review the patch and give comments in native language.
3) When the language maintainer thinks the patch is good in general, he will pass the patch onto corresponding subsystem maintainer and mail list for further review, cc the author and local list.
4) The author communicates directly with the community.
5) If the author has a problem understanding the comments or expressing himself in English, he can ask for help in native language on the local list. The language maintainer and the developers on the list are responsible to help out.
6) When the developer gets familiar with the submission process and can go through it with no problem, he can then submit patch directly without the helper process and probably join the mail list to help other new developer.

- Leo

#103 lore raw

Re: [PATCH] Chinese Language Maintainer

Quoted 12 lines · Li Yang

Here is the helper process I propose to help more people to participate.
Suggestions and comments are welcomed.

1) Developer who can't speak English or has a problem going through the submission process sends patches to the Language maintainer and cc the local mail list provided.
2) Language maintainer and experienced Linux developer on the local mail list can review the patch and give comments in native language.
3) When the language maintainer thinks the patch is good in general, he will pass the patch onto corresponding subsystem maintainer and mail list for further review, cc the author and local list.
4) The author communicates directly with the community.

Quoted 8 lines

5) If the author has a problem understanding the comments or expressing himself in English, he can ask for help in native language on the local list. The language maintainer and the developers on the list are responsible to help out.
6) When the developer gets familiar with the submission process and can go through it with no problem, he can then submit patch directly without the helper process and probably join the mail list to help other new developer.

You know your area better than I do. If you think that will work, I certainly have no objections...

Quoted 1 line

- Leo

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#104 lore raw

Re: Documentation of kernel messages (Summary)

In reply to Li Yang-r58472
Quoted 51 lines · LeoLi
Quoted 7 lines · Rob Landley
Quoted 3 lines · Li Yang

There are quite a lot kernel developers for each of the popular language, AFAIK. For non-popular languages, there shouldn't be translation available in the first place.

I don't distinguish between "popular" and "non-popular" languages. If somebody's willing to do the translation work, it's popular. If

nobody's

Quoted 1 line

willing to do the work, then even a language 1/3 of the planet's

population

Quoted 3 lines

speaks isn't "popular" for kernel development.

I wouldn't discourage a translator into Klingon if they were willing

to keep

Quoted 4 lines

their translation up to date and/or it actually resulted in patches.

Quoted 2 lines

It was really a surprise when
I post my Chinese translation on the LKML so many Chinese speakers

have

Quoted 1 line
Quoted 1 line

commented on the translation and encouraged the translation work.

They

Quoted 1 line
Quoted 1 line

are not visible as they usually don't talk too much. :) So I set up

the

Quoted 1 line
Quoted 1 line

Chinese kernel development maillist([email protected]),

there

Quoted 5 lines
Quoted 3 lines

will be more and more experienced kernel developer who can read and update the Chinese documents after they read the translated documentation and become kernel hacker.

The chinese mailing list is highly cool, and my first instinct was to

note it

Quoted 1 line

on kernel.org/doc, but it would be better if the chinese website I

already

Quoted 1 line

link to notes it instead. (That way I don't have to worry about how

to

Quoted 1 line

spam-guard your email address. :)

The benefit of a localized maillist is that patches can be reviewed and commented in native language for common problems (like style, API).

Quoted 2 lines

This also highlights the need for language maintainers to help the

patches go

Quoted 2 lines

upstream to the english list. (If we can avoid armchair commentators saddling them with the task of translating the entire Documentation

directory

Quoted 1 line

and keeping such a translation up to date, we might actually get one

too.

Quoted 1 line

Fielding patches and questions sounds like plenty to me...)

Quoted 8 lines

I do think the documentation translation is very necessary even when there is a language maintainer, especially for the policy documents as HOWTO, codestyle , and etc. The contributors should go through these policies and check their code for compliance before going to the language maintainer for help, or there will be too much for the language maintainer to translate. The language maintainer doesn't need to translate all the documents himself, but he can help to coordinate the translation effort and help to make it update to date.

I would like to comment from Japanese situation.
I do also think the documentation translation is necessary even if there are language maintainer are not there. HOWTO like very fundamental documents will great help for early stage of developers. The contributors/early stage of developer will be able to follow coding style and so on if the documents are easy to read.

So, maintain this kind of document in local language are necessary even if there are no language maintainer.

I believe the language maintainer should mostly take care about the patch.

Quoted 11 lines
Quoted 1 line

Could you ask on said list if anyone is likely to volunteer for the

position?

Quoted 1 line

(JUST translating patches and questions, as part of pushing code

upstream.)

If we do need a contact person, I can do it. However I don't think there will be much translation work to do here. As I stated before, most Chinese programmers are more or less capable of read/write technical English. The difficult part is to let them know the benefit of merging code in kernel and teach them how to do it. That's why the policy documents in native language will be very useful.

I also think that most Japanese developers can read/write technical English at some level.

Quoted 85 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

Merging into the kernel is a great way to keep CODE up to date.

Don't

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

think
it's magic pixie dust for documentation. It never has been

before.

Quoted 4 lines
Quoted 4 lines

IMHO, having contribution merged into the kernel has the MAGIC to attract people to work for recognition. When more and more people volunteer to work on it, the documentation will be up to date

magically.

Quoted 2 lines

Obvious counter-arguments include the floppy driver, the floppy tape

driver,

Quoted 1 line

the tty layer, and most of the existing english Documentation

directory...

Quoted 3 lines

I'll happily stay out of the way of people who actually want to merge translations of documentation into the kernel. I reserve the right to

say "I

Quoted 3 lines

told you so" in about five years.

Quoted 1 line
Quoted 1 line

Ah, but It's not a language maintainer's job to update

documentation.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

It's
their job to ACCEPT PATCHES. Translate patches, translate

questions

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

back and
forth. This has NOTHING to do with documentation unless they're converting
documentation accompanying the patch one way; into english.

Language maintainers can integrate updates to the documentation just like integrating any updates to the code. Working on the

documentation

Quoted 1 line
Quoted 1 line

is ,IMHO, a perfect task for novice kernel hacker to get familiar

with

Quoted 3 lines
Quoted 1 line

the process.

From a language maintainer's perspective documentation patches

wouldn't be any

Quoted 1 line

different than any other patches. Translate the description and

questions

Quoted 1 line

going back and forth. The patch itself doesn't get translated when

it's C

Quoted 1 line

and applies to scripts/kconfig/, why would it when it's UTF-8 and

applies to

Quoted 3 lines

Documentation/?

Of course this brings up the question "what kind of function and

variable

Quoted 1 line

names do chinese people pick?" (I honestly don't know, but I note

that

Quoted 1 line

attempts to use names that don't fit in 7-bit ascii would probably be

frowned

Quoted 3 lines

upon in a big way...)

Quoted 1 line

It won't be too hard if the work is shared by a community. Like the

one

Quoted 3 lines
Quoted 1 line

I'm trying to establish.

A list works fine as a point of contact. I note that in general,

maintainers

Quoted 1 line

are individuals (who delegate like mad, of course), because otherwise

agenda

Quoted 1 line

items languish with everyone thinking it's someone else's

responsibility.

Got it. I can do it as long as it doesn't consume too much time. :)

Quoted 9 lines

http://www.trumanlibrary.org/buckstop.htm

Quoted 4 lines

- Leo

Advertisement time:
Chinese kernel development community http://zh-kernel.org :)

Hmmm... Now I'm wondering if I should link directly to the docs pages

of the

Quoted 1 line

chinese and japanese sites, or to the top level community pages?

The documents and maillist are also linked in the front page. It's better to link the top level page.

- Leo

#105 lore raw

Re: Documentation of kernel messages (Summary)

In reply to Li Yang-r58472
Quoted 10 lines · On Thursday 12 July 2007 9:53:54 am Li Yang-r58472 wrote:
Quoted 1 line

Fielding patches and questions sounds like plenty to me...)

I do think the documentation translation is very necessary even when there is a language maintainer, especially for the policy documents as HOWTO, codestyle , and etc. The contributors should go through these policies and check their code for compliance before going to the language maintainer for help, or there will be too much for the language maintainer to translate. The language maintainer doesn't need to translate all the documents himself, but he can help to coordinate the translation effort and help to make it update to date.

It would help if all the policy documents got grouped into a single Documentation/development directory so we could separate "policy documents in each language would be nice" from "that document about the amiga zorro bus really needs to be kept up-to-date in Navajo and that should be in the kernel tarball please".

Lemme see, which ones are we talking about? The candidates are:
applying-patches.txt
BUG-HUNTING
Changes
CodingStyle
debugging-modules.txt
feature-removal-schedule.txt
HOWTO
kernel-docs.txt
language-maintainers.txt
ManagementStyle
oops-tracing.txt
SecurityBugs
sparse.txt
stable_api_nonsense.txt
stable_kernel_rules.txt
SubmitChecklist
SubmittingDrivers
SubmittingPatches
volatile-considered-harmful.txt

That's everything I noticed in the top level directory that's a good candidate to be grouped into a "development" subdirectory. Did I miss anything?

I note that Changes is a bit stale in places (16 bit assembly?), feature-removal-schedule.txt changes often but is good to know, kernel-docs.txt might be useless to translate considering it's mostly links to english documentation, language-maintainers.txt is assuming my patch from earlier today gets accepted...

I can submit a patch grouping all that stuff together into a subdirectory if it would help...

Quoted 6 lines

If we do need a contact person, I can do it. However I don't think there will be much translation work to do here. As I stated before, most Chinese programmers are more or less capable of read/write technical English. The difficult part is to let them know the benefit of merging code in kernel and teach them how to do it. That's why the policy documents in native language will be very useful.

Does the above look like a good list? There are more that need to be written, but that's what I saw in Documentation...

Rob

P.S. Dear kmail/postfix developers: having a transient DNS lookup failure on one address in a long cc: list is _NOT_ a reason to have the message stay in the kmail outbox. It should go into the postfix send queue and be retried from there. Right now, if I tell it to resend, how do I know who it has and hasn't been successfully distributed to on the first attempt? I've gotten dinged for trimming the cc: list before, but now I'm about to send out duplicates. Wheee...

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#106 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 18 lines · rob
Quoted 10 lines · On Thursday 12 July 2007 9:53:54 am Li Yang-r58472 wrote:
Quoted 1 line

Fielding patches and questions sounds like plenty to me...)

I do think the documentation translation is very necessary even when there is a language maintainer, especially for the policy documents as HOWTO, codestyle , and etc. The contributors should go through these policies and check their code for compliance before going to the language maintainer for help, or there will be too much for the language maintainer to translate. The language maintainer doesn't need to translate all the documents himself, but he can help to coordinate the translation effort and help to make it update to date.

It would help if all the policy documents got grouped into a single Documentation/development directory so we could separate "policy documents in each language would be nice" from "that document about the amiga zorro bus really needs to be kept up-to-date in Navajo and that should be in the kernel tarball please".

Quoted 6 lines

Lemme see, which ones are we talking about? The candidates are:
applying-patches.txt
BUG-HUNTING
Changes
CodingStyle
debugging-modules.txt

This file seems mostly technical. may not policy related document.

Quoted 6 lines

feature-removal-schedule.txt
HOWTO
kernel-docs.txt
language-maintainers.txt
ManagementStyle
oops-tracing.txt

Quoted 8 lines

SecurityBugs
sparse.txt
stable_api_nonsense.txt
stable_kernel_rules.txt
SubmitChecklist
SubmittingDrivers
SubmittingPatches
volatile-considered-harmful.txt

How about adding;
kernel-doc-nano-HOWTO.txt

These three slightly include some policy in the documents but purpose are mostly technical. (So, it's just comment)
devices.txt
kernel-parameters.txt
unicode.txt

Quoted 31 lines

That's everything I noticed in the top level directory that's a good candidate to be grouped into a "development" subdirectory. Did I miss anything?

I note that Changes is a bit stale in places (16 bit assembly?), feature-removal-schedule.txt changes often but is good to know, kernel-docs.txt might be useless to translate considering it's mostly links to english documentation, language-maintainers.txt is assuming my patch from earlier today gets accepted...

I can submit a patch grouping all that stuff together into a subdirectory if it would help...

Quoted 6 lines

If we do need a contact person, I can do it. However I don't think there will be much translation work to do here. As I stated before, most Chinese programmers are more or less capable of read/write technical English. The difficult part is to let them know the benefit of merging code in kernel and teach them how to do it. That's why the policy documents in native language will be very useful.

Does the above look like a good list? There are more that need to be written, but that's what I saw in Documentation...

Rob

P.S. Dear kmail/postfix developers: having a transient DNS lookup failure on one address in a long cc: list is _NOT_ a reason to have the message stay in the kmail outbox. It should go into the postfix send queue and be retried from there. Right now, if I tell it to resend, how do I know who it has and hasn't been successfully distributed to on the first attempt? I've gotten dinged for trimming the cc: list before, but now I'm about to send out duplicates. Wheee...

#107 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 19 lines · Tsugikazu Shibata
Quoted 18 lines · rob
Quoted 10 lines · On Thursday 12 July 2007 9:53:54 am Li Yang-r58472 wrote:
Quoted 1 line

Fielding patches and questions sounds like plenty to me...)

I do think the documentation translation is very necessary even when there is a language maintainer, especially for the policy documents as HOWTO, codestyle , and etc. The contributors should go through these policies and check their code for compliance before going to the language maintainer for help, or there will be too much for the language maintainer to translate. The language maintainer doesn't need to translate all the documents himself, but he can help to coordinate the translation effort and help to make it update to date.

It would help if all the policy documents got grouped into a single Documentation/development directory so we could separate "policy documents in each language would be nice" from "that document about the amiga zorro bus really needs to be kept up-to-date in Navajo and that should be in the kernel tarball please".

Quoted 8 lines
Quoted 6 lines

Lemme see, which ones are we talking about? The candidates are:
applying-patches.txt
BUG-HUNTING
Changes
CodingStyle
debugging-modules.txt

This file seems mostly technical. may not policy related document.

Yeah, I guess so. My mental filter was actually more like "how to do linux development", and it seemed both short and easy to translate, and also generally relevant to the majority of developers no matter what subsystem they're working on.

But I agree, it's not policy.

Quoted 8 lines
Quoted 8 lines

feature-removal-schedule.txt
HOWTO
kernel-docs.txt
language-maintainers.txt
ManagementStyle
oops-tracing.txt
SecurityBugs
sparse.txt

That one's not policy either, but it is general non-subsystem-specific development. Probably "applying-patches.txt" above goes in the same bucket.

Whether or not to include that bucket in the new directory depends on whether the directory is labeled "development" or "policy" or something else...

Quoted 9 lines
Quoted 6 lines

stable_api_nonsense.txt
stable_kernel_rules.txt
SubmitChecklist
SubmittingDrivers
SubmittingPatches
volatile-considered-harmful.txt

How about adding;
kernel-doc-nano-HOWTO.txt

The problem is, the generated htmdocs are in english. This file is about how to generate (and author) English documentation that won't be translated. What's the point of translating the instructions if the result won't be translated?

On the other hand, if the authors of patches _do_ put javadoc comments into the source code, the language maintainer should be aware of them and should have some easy way to translate them for the list...

Quoted 5 lines

These three slightly include some policy in the documents but purpose are mostly technical. (So, it's just comment)
devices.txt
kernel-parameters.txt
unicode.txt

Extracting the policy out into separate documents I could see, but I agree those are primarily technical.

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#108 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 45 lines · rob
Quoted 19 lines · Tsugikazu Shibata
Quoted 18 lines · rob
Quoted 10 lines · On Thursday 12 July 2007 9:53:54 am Li Yang-r58472 wrote:
Quoted 1 line

Fielding patches and questions sounds like plenty to me...)

I do think the documentation translation is very necessary even when there is a language maintainer, especially for the policy documents as HOWTO, codestyle , and etc. The contributors should go through these policies and check their code for compliance before going to the language maintainer for help, or there will be too much for the language maintainer to translate. The language maintainer doesn't need to translate all the documents himself, but he can help to coordinate the translation effort and help to make it update to date.

It would help if all the policy documents got grouped into a single Documentation/development directory so we could separate "policy documents in each language would be nice" from "that document about the amiga zorro bus really needs to be kept up-to-date in Navajo and that should be in the kernel tarball please".

Quoted 8 lines
Quoted 6 lines

Lemme see, which ones are we talking about? The candidates are:
applying-patches.txt
BUG-HUNTING
Changes
CodingStyle
debugging-modules.txt

This file seems mostly technical. may not policy related document.

Yeah, I guess so. My mental filter was actually more like "how to do linux development", and it seemed both short and easy to translate, and also generally relevant to the majority of developers no matter what subsystem they're working on.

But I agree, it's not policy.

Quoted 8 lines
Quoted 8 lines

feature-removal-schedule.txt
HOWTO
kernel-docs.txt
language-maintainers.txt
ManagementStyle
oops-tracing.txt
SecurityBugs
sparse.txt

Quoted 5 lines

That one's not policy either, but it is general non-subsystem-specific development. Probably "applying-patches.txt" above goes in the same bucket.

Whether or not to include that bucket in the new directory depends on whether the directory is labeled "development" or "policy" or something else...

Agree.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

stable_api_nonsense.txt
stable_kernel_rules.txt
SubmitChecklist
SubmittingDrivers
SubmittingPatches
volatile-considered-harmful.txt

Quoted 7 lines
Quoted 2 lines

How about adding;
kernel-doc-nano-HOWTO.txt

The problem is, the generated htmdocs are in english. This file is about how to generate (and author) English documentation that won't be translated. What's the point of translating the instructions if the result won't be translated?

People (who even in non-native) would better to know and read it such htmldocs because there are good and important documents even in English. I thought that translation of this file may help.

Quoted 3 lines

On the other hand, if the authors of patches _do_ put javadoc comments into the source code, the language maintainer should be aware of them and should have some easy way to translate them for the list...

maybe.

Quoted 9 lines
Quoted 6 lines

These three slightly include some policy in the documents but purpose are mostly technical. (So, it's just comment)
devices.txt
kernel-parameters.txt
unicode.txt

Extracting the policy out into separate documents I could see, but I agree those are primarily technical.

Further comment on this is that we may need to have another 00-INDEX file, for this new directory.

Quoted 6 lines

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#109 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 11 lines · Tsugikazu Shibata
Quoted 7 lines
Quoted 2 lines

How about adding;
kernel-doc-nano-HOWTO.txt

The problem is, the generated htmdocs are in english. This file is about how to generate (and author) English documentation that won't be translated. What's the point of translating the instructions if the result won't be translated?

People (who even in non-native) would better to know and read it such htmldocs because there are good and important documents even in English. I thought that translation of this file may help.

The english version of htmldocs is generated from the source code. If
translating remotely current versions of htmldocs into other languages was feasible, then the english version wouldn't need to be in the source code in the first place.

Translating a document _about_ htmldocs boils down to "Dear non-english
speaker: here's something you can't read, nor can you update it either except though your language maintainer."

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#110 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 13 lines · rob
Quoted 11 lines · Tsugikazu Shibata
Quoted 7 lines
Quoted 2 lines

How about adding;
kernel-doc-nano-HOWTO.txt

The problem is, the generated htmdocs are in english. This file is about how to generate (and author) English documentation that won't be translated. What's the point of translating the instructions if the result won't be translated?

People (who even in non-native) would better to know and read it such htmldocs because there are good and important documents even in English. I thought that translation of this file may help.

Quoted 8 lines

The english version of htmldocs is generated from the source code. If
translating remotely current versions of htmldocs into other languages was feasible, then the english version wouldn't need to be in the source code in the first place.

Translating a document _about_ htmldocs boils down to "Dear non-english
speaker: here's something you can't read, nor can you update it either except though your language maintainer."

No, kernel-doc-nano-HOWTO includes explanation of tools to extract the document, format of extractable document in source files and so on. I thought this file would help non-native people, how to extract or add in-kernel extractable documents.
I understand that these tools will generate English documents but that's OK. Such large volume of documents are not easy to translate into other languages.

Quoted 6 lines

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#111 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 19 lines · Tsugikazu Shibata
Quoted 18 lines · rob
Quoted 11 lines · Tsugikazu Shibata
Quoted 7 lines
Quoted 2 lines

How about adding;
kernel-doc-nano-HOWTO.txt

The problem is, the generated htmdocs are in english. This file is about how to generate (and author) English documentation that won't be translated. What's the point of translating the instructions if the result won't be translated?

People (who even in non-native) would better to know and read it such htmldocs because there are good and important documents even in English. I thought that translation of this file may help.

The english version of htmldocs is generated from the source code. If
translating remotely current versions of htmldocs into other languages was feasible, then the english version wouldn't need to be in the source code in the first place.

Quoted 8 lines
Quoted 3 lines

Translating a document _about_ htmldocs boils down to "Dear non-english
speaker: here's something you can't read, nor can you update it either except though your language maintainer."

No, kernel-doc-nano-HOWTO includes explanation of tools to extract the document, format of extractable document in source files and so on. I thought this file would help non-native people, how to extract or add in-kernel extractable documents.

Which will be in english, so why can't the instructions on how to do it be in english too?

Quoted 2 lines

I understand that these tools will generate English documents but that's OK.

Then the instructions how to do it in english should be ok too. If they can't read the instructions, they won't be able to read the result.

Quoted 2 lines

Such large volume of documents are not easy to translate into other languages.

"Here's how to get a result that will be useless to you."

Why? What's the point? I'm not seeing it.

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#112 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 29 lines · rob
Quoted 19 lines · Tsugikazu Shibata
Quoted 18 lines · rob
Quoted 11 lines · Tsugikazu Shibata
Quoted 7 lines
Quoted 2 lines

How about adding;
kernel-doc-nano-HOWTO.txt

The problem is, the generated htmdocs are in english. This file is about how to generate (and author) English documentation that won't be translated. What's the point of translating the instructions if the result won't be translated?

People (who even in non-native) would better to know and read it such htmldocs because there are good and important documents even in English. I thought that translation of this file may help.

The english version of htmldocs is generated from the source code. If
translating remotely current versions of htmldocs into other languages was feasible, then the english version wouldn't need to be in the source code in the first place.

Quoted 8 lines
Quoted 3 lines

Translating a document _about_ htmldocs boils down to "Dear non-english
speaker: here's something you can't read, nor can you update it either except though your language maintainer."

No, kernel-doc-nano-HOWTO includes explanation of tools to extract the document, format of extractable document in source files and so on. I thought this file would help non-native people, how to extract or add in-kernel extractable documents.

Quoted 8 lines

Which will be in english, so why can't the instructions on how to do it be in english too?

Quoted 2 lines

I understand that these tools will generate English documents but that's OK.

Then the instructions how to do it in english should be ok too. If they can't read the instructions, they won't be able to read the result.

It seems my explanation was not enough.
We (like me, usually not using English) can read English but it takes more time than you (and any natives). So, reading large document takes huge amount of time. In case of htmldocs, it is actually large and developer may give up before start.
I thought that if some part (like how to do it) are translated into his/her familiar language, he/she may not give it up.
Yes, it may not be.

Another viewpoint is that if he/she need to modify some interface and it may need to change some part of extractable document, this file includes, "How to add extractable documentation to your source files". It will be some help as well.
Yes, He/she should write English and it may take long time but hopefully the translation result would push his/her back.

Quoted 12 lines
Quoted 2 lines

Such large volume of documents are not easy to translate into other languages.

"Here's how to get a result that will be useless to you."

Why? What's the point? I'm not seeing it.

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#113 lore raw

Re: Documentation of kernel messages (Summary)

In reply to Rob Landley
Quoted 58 lines · Rob Landley
Quoted 10 lines · On Thursday 12 July 2007 9:53:54 am Li Yang-r58472 wrote:
Quoted 1 line

Fielding patches and questions sounds like plenty to me...)

I do think the documentation translation is very necessary even when there is a language maintainer, especially for the policy documents as HOWTO, codestyle , and etc. The contributors should go through these policies and check their code for compliance before going to the language maintainer for help, or there will be too much for the language maintainer to translate. The language maintainer doesn't need to translate all the documents himself, but he can help to coordinate the translation effort and help to make it update to date.

It would help if all the policy documents got grouped into a single Documentation/development directory so we could separate "policy documents in each language would be nice" from "that document about the amiga zorro bus really needs to be kept up-to-date in Navajo and that should be in the kernel tarball please".

Lemme see, which ones are we talking about? The candidates are:
applying-patches.txt
BUG-HUNTING
Changes
CodingStyle
debugging-modules.txt
feature-removal-schedule.txt
HOWTO
kernel-docs.txt
language-maintainers.txt
ManagementStyle
oops-tracing.txt
SecurityBugs
sparse.txt
stable_api_nonsense.txt
stable_kernel_rules.txt
SubmitChecklist
SubmittingDrivers
SubmittingPatches
volatile-considered-harmful.txt

That's everything I noticed in the top level directory that's a good candidate to be grouped into a "development" subdirectory. Did I miss anything?

I note that Changes is a bit stale in places (16 bit assembly?), feature-removal-schedule.txt changes often but is good to know, kernel-docs.txt might be useless to translate considering it's mostly links to english documentation, language-maintainers.txt is assuming my patch from earlier today gets accepted...

I can submit a patch grouping all that stuff together into a subdirectory if it would help...

Quoted 6 lines

If we do need a contact person, I can do it. However I don't think there will be much translation work to do here. As I stated before, most Chinese programmers are more or less capable of read/write technical English. The difficult part is to let them know the benefit of merging code in kernel and teach them how to do it. That's why the policy documents in native language will be very useful.

Quoted 2 lines

Does the above look like a good list? There are more that need to be written, but that's what I saw in Documentation...

Yes, I do think this is a good list for translation.

Greg,
Do you have anything to add?

- Leo

#114 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 4 lines · Li Yang

Yes, I do think this is a good list for translation.

Greg,
Do you have anything to add?

There are a few documents not in the kernel tree that you may want to translate too, if your goal is to increase the number of kernel developers:

http://kernelnewbies.org/KernelGlossary

http://kernelnewbies.org/FAQ

http://kernelnewbies.org/UpstreamMerge

Signature · 3 lines
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.
#115 lore raw

Re: Documentation of kernel messages (Summary)

In reply to Rob Landley
Quoted 10 lines · Rob Landley

...
[regarding translated documentation]

For my part, I can't _tell_ when a given translation is out of date because I can't read it, and I certainly can't update it. So I agree with Eric and I'm linking to sites hosting kernel documentation in other languages rather than hosting them myself. I've got a good link to a Japanese site, need to ping the contributors of the chinese documentation and see if they have a site for it...
...

Include the last git commit on this file in the translated document?

This would also make it easier for a translator to update a document because a diff on the original document since the last translation would be trivial.

Quoted 1 line

Rob

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#116 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 12 lines · On Monday 09 July 2007 14:33:09 Adrian Bunk wrote:
Quoted 10 lines · Rob Landley

...
[regarding translated documentation]

For my part, I can't _tell_ when a given translation is out of date because I can't read it, and I certainly can't update it. So I agree with Eric and I'm linking to sites hosting kernel documentation in other languages rather than hosting them myself. I've got a good link to a Japanese site, need to ping the contributors of the chinese documentation and see if they have a site for it...
...

Quoted 5 lines

Include the last git commit on this file in the translated document?

This would also make it easier for a translator to update a document because a diff on the original document since the last translation would be trivial.

Ok, back up.

Let's look at _english_ documentation.

If you go to http://kernel.org/doc/ols you should find, nicely split up, all the OLS papers from 2002-2007. By my count, there are 337 of them, totaling 61.8 megabytes when they're split up like that. Being PDFs, they don't compress all that well.

Which subset of these should be copied into the Documentation directory? How do you update them? This is the documentation from just ONE SOURCE.

And PDF isn't close to the worst of it: Linuxconf.au has lots of video, as does google video. (I don't even have _english_ transcripts for those videos, which would be nice.) Do you want to merge all of this into the kernel tarball as well?

How does this help keep anything up to date?

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#117 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 9 lines · Rob Landley

...
Let's look at _english_ documentation.

If you go to http://kernel.org/doc/ols you should find, nicely split up, all the OLS papers from 2002-2007. By my count, there are 337 of them, totaling 61.8 megabytes when they're split up like that. Being PDFs, they don't compress all that well.

Which subset of these should be copied into the Documentation directory?

You don't have to copy everything.

And you anyway have to handle cases like e.g. most books where you are not permitted to copy them, but might want to point people to them.

Quoted 2 lines

How
do you update them? This is the documentation from just ONE SOURCE.

You aren't legally permitted to update any OLS paper unless you've gotten explicit permission by the author. And when he gives legal permission, he can as well give you the LaTeX source. I'd expect legal and technical update possibilities to usually be related this way.

And you shouldn't include non-free documentation into the kernel sources.

Quoted 4 lines

And PDF isn't close to the worst of it: Linuxconf.au has lots of video, as does google video. (I don't even have _english_ transcripts for those videos, which would be nice.) Do you want to merge all of this into the kernel tarball as well?

Not at all.

Quoted 1 line

How does this help keep anything up to date?

You can only update things you are legally and technically able to update. For the rest, you can only ship pointers like kernel-docs.txt.

Quoted 1 line

Rob

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#118 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 9 lines · On Tuesday 10 July 2007 14:54:15 Adrian Bunk wrote:
Quoted 8 lines · Rob Landley

...
Let's look at _english_ documentation.

If you go to http://kernel.org/doc/ols you should find, nicely split up, all the OLS papers from 2002-2007. By my count, there are 337 of them, totaling 61.8 megabytes when they're split up like that. Being PDFs, they don't compress all that well.

Quoted 6 lines
Quoted 1 line

Which subset of these should be copied into the Documentation directory?

You don't have to copy everything.

And you anyway have to handle cases like e.g. most books where you are not permitted to copy them, but might want to point people to them.

This is exactly what I'm saying. I don't want to copy the translated documentation, I want to point people to it. Out on the web. Where it's maintained by the people who translated it, which history says they're noticeably less likely to do with a copy.

Quoted 5 lines
Quoted 2 lines

How
do you update them? This is the documentation from just ONE SOURCE.

You aren't legally permitted to update any OLS paper unless you've gotten explicit permission by the author.

I wasn't planning to. I'm pointing out that it's hard.

Quoted 1 line

And you shouldn't include non-free documentation into the kernel sources.

A) http://www.linuxsymposium.org/2007/cfp.php under "Publication Rights":

Further, as stated in the official templates, and on the Credits page from the Proceedings of this year and prior years: "Authors retain copyright to all submitted papers, but have granted unlimited redistribution rights to all as a condition of submission."

So it's free as in speech but not beer.

B) I'm using these as examples of things to NOT merge into the kernel sources. Thank you for making my point. There's a LOT of this stuff out there.

Quoted 4 lines
Quoted 1 line

How does this help keep anything up to date?

You can only update things you are legally and technically able to update. For the rest, you can only ship pointers like kernel-docs.txt.

As an english speaker, I am not technically able to update non-english documentation. Since the only common language required of linux kernel developers has ever been English, only the english documentation makes sense to merge into the kernel tarball. Merging anything else guarantees that the vast majority of the contributors will never be able to review or update it, so merging it into the kernel tarball does not result in extra review or maintenance.

Extrapolating the review benefits of C code that breaks the build for everybody if you do it wrong to chinese documentation you can't even _read_... Apples and oranges, it's a false comparison. I don't see how merging translations of documentation is ever likely to be a win.

Quoted 4 lines
Quoted 1 line

Rob

cu
Adrian

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#119 lore raw

Re: Documentation of kernel messages (Summary)

In reply to Rob Landley
Quoted 2 lines · Rob Landley

If you go to http://kernel.org/doc/ols you should find, nicely split up, all the OLS papers from 2002-2007.

Oooh! That's nice! I didn't notice the "nicely split up" part earlier. Any chance we can get the original docbook inputs that OLS uses for paper submissions? Have you asked Andrew or Craig about this?

Also, it would be nice to correlate the talk names with the individual PDFs. Do you want me to solicit inside CELF for a volunteer for this? We'd probably produce a wiki page of links, based on your list.txt file and the proceedings index.
-- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================

#120 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 5 lines · Tim Bird

Oooh! That's nice! I didn't notice the "nicely split up" part earlier. Any chance we can get the original docbook inputs that OLS uses for paper submissions? Have you asked Andrew or Craig about this?

OLS uses LaTeX, not DocBook, for submissions AFAIK.

-hpa

#121 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 6 lines · H. Peter Anvin
Quoted 3 lines · Tim Bird

Oooh! That's nice! I didn't notice the "nicely split up" part earlier. Any chance we can get the original docbook inputs that OLS uses for paper submissions? Have you asked Andrew or Craig about this?

OLS uses LaTeX, not DocBook, for submissions AFAIK.

Yeah, but their retention of the original latex has been... Iffy. And their indexing was a bit rushed. http://www.linuxsymposium.org/proceedings.php
goes up to 2005. If you go to
http://www.linuxsymposium.org/2006/proceedings.php you'll see a mirror of the other page with 2006 added (why not update the top level one? No idea.) Then 2007 is off on a Red Hat site (mis-numbered) and not even hosted on linuxsymposium.org.

If you look at http://kernel.org/doc/ols/2006/slides/ you'll notice I mirrored all the presentation slides (not the same as the papers). Two of them had already gone down before I got there, but I managed to get a fresh copy of Steven Hill's uClibc NPTL slides when I bumped into him at OLS. (My mirror has it, the original site it links to is 404.) I still haven't tracked down Kristen Accardi's slides, and that's one that was theoretically mirrored on the OLS website itself, yet the link is 404...

I'm not complaining, by the way: busy people get the data out and then move on, it's up to people like me to go back and clean it up if you want any kind of data retention. (And THEN you have to periodically figure out what's still relevant and what's only of historical interest. Wheee...)

Right now, I'd be happy to just get the PDFs of 2001 and earlier. There's a apparently one year they only have on paper, and one of the others is on a CD that Andrew thinks he knows where it is, but hadn't had time to dig up when last I poked him...

If latex shows up, I'm all for it. But for now, I'm happy with the PDFs. (Hey Tim, you want something for a volunteer to do? Index the 2006 papers, then match up the 2006 slides with the corresponding papers. I can go through and add in summaries afterwards... And note that the index.html should have _relative_ links, not absolute ones, because I want anyone to be able to tarball up the whole site and locally mirror it. I need to do a big script that fetches the external things like the ols papers and breaks them up so you have reproducible local copies without blindly copying mine. It's on my todo list...)

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#122 lore raw

Re: Documentation of kernel messages (Summary)

In reply to Tim Bird
Quoted 7 lines · Tim Bird
Quoted 2 lines · Rob Landley

If you go to http://kernel.org/doc/ols you should find, nicely split up, all the OLS papers from 2002-2007.

Oooh! That's nice! I didn't notice the "nicely split up" part earlier. Any chance we can get the original docbook inputs that OLS uses for paper submissions? Have you asked Andrew or Craig about this?

I've asked, but OLS was in the process of happening and they were kind of swamped...

Quoted 2 lines

Also, it would be nice to correlate the talk names with the individual PDFs.

I'm working on that: http://kernel.org/doc/ols/2002

I keep getting distracted by new material coming in. Need to shut it out for a bit and focus on processing the existing pile...

Quoted 1 line

Do you want me to solicit inside CELF for a volunteer for this?

Yes please. I'm happy to have volunteers help out with any of this, if I can figure out how to sanely partition it.

I'm not just collating names with papers, I'm also reading the paper and trying to come up with a one page summary. If you look at
http://kernel.org/doc you'll see a (now stale) version of an index skeleton. I'd like to link to all these papers from one big index. And also link the Documentation files into the same index. And http://lwn.net/Kernel/Index/
and Linux Journal's archives and...

That's the hard part. Indexing it. Piling up a big heap is easy.

You'll notice that Documentation has its own 00-index file, which doesn't even refer to the make htmldocs output. The lwn kernel material has an excellent index. The Linux Journal articles have a chronological archive. Every volume of OLS papers starts with a brief index.

And the problem is, every time somebody notices the problem they start a _new_ index that doesn't use any of the others. (And they keep wanting to do it as a wiki, which dooms the project. In my experience wikis aren't stable and only locally versioned. They're not designed to be snapshotted as a coherent whole, which is the _point_ of an index. If you deep-link into a wiki you get 404 errors after a while. They're a good tool for the "piling up information" part, but a bad tool for editorial jobs like organizing and indexing existing information. Wikis are designed to be decentralized, so the left hand doesn't have to know what the right hand is doing, but editorial functions is all about collating, coordinating, and correlating into a coherent whole. Which is nontrivial.)

Quoted 2 lines

We'd probably produce a wiki page of links, based on your list.txt file and the proceedings index.

I have a list.txt file? (Rummages.) Oh, that's actually extracted from Red Hat's 2007 OLS web page (see the link at top, "mirrored from here".)

I was using Red hat's broken-up pages (and pre-broken-up ones I found for 2004) for a while, but I went back and I re-broke up the pages with my scripts (which you'll notice I link to; I want people other than me to be able to reproduce this.) I did it because all the others I broke up have the two OLS boilerplate pages at the _end_ rather than the beginning, and I wanted to be consistent.

And that list.txt page doesn't have the paragraph per article summary (which I can sometimes take from the abstract, but even when there is an abstract it's often not what I need and I read the darn paper anyway to see what interesting stuff's buried in it). I need that in order to index the articles, title isn't enough. Some articles need to get linked from more than one place. (James Bottomley's 2002 scsi paper has good "history of hotplug" material, for example.)

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#123 lore raw

RE: Documentation of kernel messages (Summary)

In reply to Rob Landley
Quoted 8 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Rob Landley
Sent: Tuesday, July 10, 2007 12:48 AM
To: H. Peter Anvin
Cc: Kunai, Takashi; [email protected]; Andrew Morton; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 1 line

[email protected]; [email protected]; [email protected];

[email protected];

Quoted 6 lines

[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: Documentation of kernel messages (Summary)

Quoted 6 lines

In regard to translating kernel messages:

Quoted 3 lines · On Monday 09 July 2007 01:36:31 H. Peter Anvin wrote:
Quoted 2 lines · Takashi

(1) Your kernel development proposal will be greatly supported by Japanese vendor community. At the same time, it needs support from

the

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

kernel communities, as well.

There is a very strong reason for the kernel community to NOT

support

Quoted 6 lines
Quoted 1 line

this: it makes it much harder to deal with bug reports.

Agreed.

There's a proposal in play for translations of documentation, and to recruit "language maintainers" (which is an idea I see that Matt

Mackall came

Quoted 3 lines

up with before I did).

A language maintainer is somebody who can accept a patch in language

X,

Quoted 1 line

translate its description to post on the appropriate english-only list

with a

Quoted 1 line
cc to the appropriate maintainer(s), and translate questions and

reviews

Quoted 1 line

back and forth for the original author until the patch can be merged.

(This

Quoted 1 line

means that documentation translations don't result in a stream of

unmergeable

Quoted 3 lines

patches.)

As for the documentation translations themselves, I note that Eric

Raymond

Quoted 1 line

dissuaded me from actually hosting translations of kernel

documentation on

Quoted 1 line

http://kernel.org/doc due to his experience with translations of his

own

Quoted 3 lines

writings. If he hosts the translations on his website, they never get updated again. But if he makes the contributor host them on their own website, then they sometimes get updated.

If the documents are already maintained together on one site, it won't be very hard to synchronize with kernel.org. Eric is right for the case that separate documents on separate sites.

Quoted 2 lines

For my part, I can't _tell_ when a given translation is out of date

because I

Quoted 1 line

can't read it, and I certainly can't update it. So I agree with Eric

and I'm

Quoted 1 line

linking to sites hosting kernel documentation in other languages

rather than

Quoted 1 line

hosting them myself. I've got a good link to a Japanese site, need to

ping

Quoted 1 line

the contributors of the chinese documentation and see if they have a

site for

Quoted 1 line

it...

Please add a link to http://zh-kernel.org/docs for Chinese translated kernel documentation.

- Leo

#124 lore raw

Re: Documentation of kernel messages (Summary)

In reply to H. Peter Anvin

Hi,

Quoted 8 lines · H. Peter Anvin
Quoted 3 lines · Takashi

(1) Your kernel development proposal will be greatly supported by Japanese vendor community. At the same time, it needs support from the kernel communities, as well.

There is a very strong reason for the kernel community to NOT support
this: it makes it much harder to deal with bug reports.

Quoted 8 lines

Like it or not, English is a lingua franca[1] of the technology industry. When I deal with the rest of the kernel community, I use English, instead of my native language (Swedish).

For things that are targetted toward end users, that's a different matter, of course, but kernel messages are inherently directed toward developers and technically sophisticated users.

I strongly agree with you.

Please keep kernel clean.

#125 lore raw

Re: Documentation of kernel messages (Summary)

In reply to Kunai, Takashi

Am Montag, 9. Juli 2007 schrieb Kunai, Takashi:

Quoted 3 lines

(b)printk hashes and (c)Format strings. (a) seems difficult to get supports from kernel developers and (c) lacks uniqueness of each message. Though (b) also lacks uniqueness, adding component id and/or

We have generators for hash functions that guarantee unique results for a set of inputs. They are even GPLed. As you are operating against a known target, that should work.

Regards
Oliver

#126 lore raw

Re: Documentation of kernel messages (Summary)

Quoted 7 lines · Oliver Neukum

Am Montag, 9. Juli 2007 schrieb Kunai, Takashi:

Quoted 3 lines

(b)printk hashes and (c)Format strings. (a) seems difficult to get supports from kernel developers and (c) lacks uniqueness of each message. Though (b) also lacks uniqueness, adding component id and/or

We have generators for hash functions that guarantee unique results for a set of inputs.

Right, perfect hash generators require the input set to be known *a priori*.

Quoted 2 lines

They are even GPLed. As you are operating against
a known target, that should work.

But, I'm not sure they'd be operating against a known target -- I don't really know what exactly would be hashed, but if it's kernel printk() messages (the format string, obviously), then please remember that new messages would get added all the time, and unless we're also willing to change the hash function every time a printk() gets added to the kernel (whew!) it's going to be a problem -- especially because we don't really want to generate new hash functions for every kernel version / or for every kernel build, because we'd obviously like the same error message to hash to the same value across versions.

If we really care about uniqueness, the only solution I see is to use a strong/cryptographic hash function (say SHA-1, which would never change in the future) that generates the hashes for all the printk() messages that are getting compiled-in at build-time ... but it would be slow, of course.

Satyam

#127 lore raw

Re: Documentation of kernel messages (Summary)

Am Dienstag, 10. Juli 2007 schrieb Satyam Sharma:

Quoted 9 lines

But, I'm not sure they'd be operating against a known target -- I don't really know what exactly would be hashed, but if it's kernel printk() messages (the format string, obviously), then please remember that new messages would get added all the time, and unless we're also willing to change the hash function every time a printk() gets added to the kernel (whew!) it's going to be a problem -- especially because we don't really want to generate new hash functions for every kernel version / or for every kernel build, because we'd obviously like the same error message to hash to the same value across versions.

I suggest exactly that you build a new hash function for every build. Nothing but your translating demon need ever see it. We build a new System.map every time and it doesn't hurt.

Regards
Oliver

#128 lore raw

Re: [Lf_kernel_messages] Re: Documentation of kernel messages (Summary)

In reply to Rob Landley
Quoted 5 lines · Rob Landley

Personally? No to the second question, which renders the first "do it yourself outside of the tree".

Just a guess, and I don't speak for anyone else here, but I think most of us are waiting to see how long it takes you to lose interest.

Actually, at a brainstorming session at the recent Linux Foundation collaboration summit, there were a number of kernel developers, including James Bottomley, Greg K-H, Cristoph Hellwig (I think), and others, who were actually generally symapthetic to the idea, if done right.

The idea that was tossed about was essentially printk hashes (hashed on message plus filename, with optional reporting of filename+line number), with the messaging catalog being maintained externally. Yes, that will be a major pain for whoever wants to do the translation --- but it puts the onus on the people who would be getting the value for this to do the work.

The other idea that was tossed about was that for subsystems that are using structured reporting (i.e., dev_printk, and sdev_printk in the SCSI subsystem --- guess who suggested that :-), that there is an opportunity to push a lot more information out to userspace for people who want to use this facility to more easily determine when something has bad happened to a particular disk device, for example. This goes a little a beyond the original stated desire for translation support, but it was another idea to explore.

Quoted 4 lines

I find "document messages" to be a horrible idea conceptually, because I think the messages should _be_ documentation. If the message is unclear it should be clarified. "There's too much garbage on the floor, we should laminate it so we can't smell it anymore." Er, no...

There are two separable desires being addressed here. The first is "cute" messages such as "lp0: on fire". Yes, maybe those should be fixed, but sometimes kernel developers *like* cute messages. (Sniff, I'm pretty sure there used to be a "eaten by a grue" printk, but it doesn't seem to be there any more. :-) Fixing these is largely a political problem: "of course printer on fire makes sense".

The second desire is where people want entire paragraphs discsusing possible causes of the messages and recommended responses to be carried out by the data center's 3rd shift operations guy (aka "tape monkey"). That's not something you *ever* want to put in the kernel, and who ever compiles such a long document will probably sell it for $$$ as significant value add. Which is fine, because it is going to cost a lot of $$$ to keep it up to date. :-)

Quoted 7 lines
Quoted 2 lines

printk hashes:
- Additional preprocessor step needed

Only for you guys. Those of us building our own kernels and NOT translating them or producing "the big book of explanations of kernel messages for IBM customers who don't understand them" shouldn't have to run this preprocessor step.

Yep, I suspect it will only be used by distro kernels. I probably wouldn't turn it on myself. :-)

Quoted 9 lines
Quoted 4 lines

- Hard to keep printks and descriptions up-to-date. If using hashes, each typo fix in a printk breaks the link to your documentation.
- Since the developer knows the meaning of a printk best, he probably would be the right person to write the description.

Imposing additional requirements on developers isn't going to work unless you reject patches from developers who don't follow your procedures and submit the appropriately filled-out forms with each patch for processing by the central bureaucracy.

Yep, it's going to be up to the translators to keep the message catalogs up to date. The gettext folks have figured out ways of doing fuzzy string matches, and so if you keep the external database of filename, line #, string, hash, and translation(s), when you go to new version of the kernel, it's not that hard to make the fuzzy translations work.

- Ted

#129 lore raw

Re: [Lf_kernel_messages] Re: Documentation of kernel messages (Summary)

O> Yep, it's going to be up to the translators to keep the message

Quoted 5 lines

catalogs up to date. The gettext folks have figured out ways of doing fuzzy string matches, and so if you keep the external database of filename, line #, string, hash, and translation(s), when you go to new version of the kernel, it's not that hard to make the fuzzy translations work.

The tools can extract the strings if marked _() which is easy work to do and you usually only want to hit a subset of strings anyway [the ones that occur usually].

Having implemented an encheferizer in a much older kernel its worth it even for the humour value of a valspeak kernel (plus some people will never be happy until it says things like

SYS-E-FOO: Incorrect operating system in use

#130 lore raw

Re: [Lf_kernel_messages] Re: Documentation of kernel messages (Summary)

In reply to Theodore Tso
Quoted 6 lines · On Monday 09 July 2007 14:10:07 Theodore Tso wrote:

The idea that was tossed about was essentially printk hashes (hashed on message plus filename, with optional reporting of filename+line number), with the messaging catalog being maintained externally. Yes, that will be a major pain for whoever wants to do the translation --- but it puts the onus on the people who would be getting the value for this to do the work.

Sounds good to me.

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#131 lore raw
In reply to holzheu

I think my general response to something like this, if it goes in, would be to stop emitting useful kernel log messages in the code I write because having to document it too on top of that is just too much extra work to be worthwhile.

author
#132 lore raw
Quoted 4 lines · David Miller

I think my general response to something like this, if it goes in, would be to stop emitting useful kernel log messages in the code I write because having to document it too on top of that is just too much extra work to be worthwhile.

Nobody will force you to document your messages. As I said, there is no need to document self-explaining printks.

Michael