Skip to contentSkip to conversation

Linux-tiny project revival

Announce Tim Bird 25 participants 63 messages Sep 19, 2007 – Sep 30, 2007

lore
Share & tools
lore

Tim Bird

author
#0 lore raw

Recently, the CE Linux forum has been working to revive the Linux-tiny project. At OLS, I asked for interested parties to volunteer to become the new maintainer for the Linux-tiny patchset.

A few candidates came forward, but eventually Michael Opdenacker was selected as the new primary maintainer. A few other people, including John Cooper of Wind River and myself are working to support this effort.

Recently, many of the Linux-tiny patches have been brought up-to-date and are now available for use with a 2.6.22 kernel. The intent is to test these, and begin mainlining the most effective sub-patches, in the next few months.

Some automated testing has already been set up, with some preliminary results published at a CELF conference in Japan. (See the linux-tiny page below for a link to the presentation.) Hopefully, results publishing will also be automated soon.

We encourage anyone with interest in this project to get involved. If you have ideas how to reduce the static or dynamic memory footprint of Linux, or, even better, patches for this, please let us know about them.

Please see http://elinux.org/Linux_Tiny

A related document: http://elinux.org/Kernel_Size_Tuning_Guide
is undergoing an update this week.

Thanks,
-- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================

#1 lore raw
Quoted 24 lines · Tim Bird

Recently, the CE Linux forum has been working to revive the Linux-tiny project. At OLS, I asked for interested parties to volunteer to become the new maintainer for the Linux-tiny patchset.

A few candidates came forward, but eventually Michael Opdenacker was selected as the new primary maintainer. A few other people, including John Cooper of Wind River and myself are working to support this effort.

Recently, many of the Linux-tiny patches have been brought up-to-date and are now available for use with a 2.6.22 kernel. The intent is to test these, and begin mainlining the most effective sub-patches, in the next few months.

Some automated testing has already been set up, with some preliminary results published at a CELF conference in Japan. (See the linux-tiny page below for a link to the presentation.) Hopefully, results publishing will also be automated soon.

We encourage anyone with interest in this project to get involved. If you have ideas how to reduce the static or dynamic memory footprint of Linux, or, even better, patches for this, please let us know about them.

Quoted 4 lines

Please see http://elinux.org/Linux_Tiny

A related document: http://elinux.org/Kernel_Size_Tuning_Guide
is undergoing an update this week.

Will there be a separate git for testing? Is the idea to still keep moving patches upstream?

Luis

author
#2 lore raw
Quoted 1 line · Luis R. Rodriguez

Will there be a separate git for testing?

Right now we're publishing a quilt-able tarball. Instructions for applying this are on the
http://elinux.org/Linux_Tiny page.

Note that some broken patches are kept around in the patches/tiny directory, but commented out in the series file, in case someone wants to un-bitrot them.

Quoted 2 lines

Is the idea to still keep
moving patches upstream?

Yes, very much so. These patches have been showing up in CE products, and this is part of an overall effort to dust them off, test them, and move them upstream.
-- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================

#3 lore raw
In reply to Tim Bird
Quoted 9 lines · Tim Bird

Recently, the CE Linux forum has been working to revive the Linux-tiny project. At OLS, I asked for interested parties to volunteer to become the new maintainer for the Linux-tiny patchset.

A few candidates came forward, but eventually Michael Opdenacker was selected as the new primary maintainer. A few other people, including John Cooper of Wind River and myself are working to support this effort.

congratulations to the new maintainer!

nice to see this subset of patches being revived.

;-)

Signature · 3 lines
Christian
--
http://detaolb.sourceforge.net/, a linux distribution for Qemu with Git inside !
#4 lore raw
In reply to Tim Bird
Quoted 3 lines · Tim Bird

Recently, the CE Linux forum has been working to revive the Linux-tiny project. At OLS, I asked for interested parties to volunteer to become the new maintainer for the Linux-tiny patchset.

Not sure what the point is of a separate patchkit. If it's a reasonable patch it should just be put into mainline. And hopefully all the patches will be reasonable.

-Andi

author
#5 lore raw
Quoted 10 lines · Andi Kleen
Quoted 3 lines · Tim Bird

Recently, the CE Linux forum has been working to revive the Linux-tiny project. At OLS, I asked for interested parties to volunteer to become the new maintainer for the Linux-tiny patchset.

Not sure what the point is of a separate patchkit. If it's a reasonable patch it should just be put into mainline. And hopefully all the patches will be reasonable.

I don't know the detailed history, but my understanding is that for some of these, mainline attempts were made in the past.

The patchkit gives a place for things to live while they are out of mainline, and still have multiple people use and work on them. Optimally the duration of being out-of-mainline would be short, but my experience is that sometimes what an embedded developer considers reasonable to hack off the kernel is not considered so reasonable by other developers (even with config options). Also, sometimes it takes a while for a patch to mature to the point where it is acceptable for general use, while still being usable for special-purpose projects.
-- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================

#6 lore raw
Quoted 2 lines · Tim Bird

The patchkit gives a place for things to live while they are out of mainline, and still have multiple people use and work on them.

Is anybody working on testing that the patchkit "does no harm" for bigger boxes (laptops, desktops, servers)? That would probably be helpful info when pieces are pushed towards mainline....

(Was just looking at the patch details, there's at least a few that look interesting for my purposes even though I'm not an embedded developer)...

author
#7 lore raw
Quoted 6 lines · [email protected] wrote:
Quoted 2 lines · Tim Bird

The patchkit gives a place for things to live while they are out of mainline, and still have multiple people use and work on them.

Is anybody working on testing that the patchkit "does no harm" for bigger boxes (laptops, desktops, servers)?

Not to my knowledge. Most of the things it provides are only activated by config options. So my sense is that just applying the patches should be harmless. But we'll need some runtime testing to see what affect some of these have on big iron.

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================

#8 lore raw
Quoted 11 lines · Tim Bird
Quoted 3 lines · [email protected] wrote:

Is anybody working on testing that the patchkit "does no harm" for bigger boxes (laptops, desktops, servers)?

Not to my knowledge. Most of the things it provides are only activated by config options. So my sense is that just applying the patches should be harmless. But we'll need some runtime testing to see what affect some of these have on big iron.

Tim is right, Linux-Tiny features are only activated by configuration options.

However, that doesn't prevent us from proposing size reduction changes impacting all kernel users, in cases a configuration option doesn't make sense. Extensive testing in the -mm tree will then be required.

Cheers,

Michael.

Signature · 3 lines
Michael Opdenacker
http://free-electrons.com
+33 621 604 642
#9 lore raw
In reply to Tim Bird
Quoted 3 lines · On Wed, 19 Sep 2007 11:03:09 -0700 Tim Bird <[email protected]> wrote:

Recently, the CE Linux forum has been working to revive the Linux-tiny project. At OLS, I asked for interested parties to volunteer to become the new maintainer for the Linux-tiny patchset.

I volunteer! Send patches to me, cc linux-kernel and celinuv-dev.

Seriously, putting this stuff into some private patch collection should be a complete last resort - you should only do this with patches which you (and the rest of us) agree have no hope of ever getting into mainline.

author
#10 lore raw
Quoted 7 lines · Andrew Morton
Quoted 3 lines · On Wed, 19 Sep 2007 11:03:09 -0700 Tim Bird <[email protected]> wrote:

Recently, the CE Linux forum has been working to revive the Linux-tiny project. At OLS, I asked for interested parties to volunteer to become the new maintainer for the Linux-tiny patchset.

Quoted 5 lines

I volunteer! Send patches to me, cc linux-kernel and celinuv-dev.

Seriously, putting this stuff into some private patch collection should be a complete last resort - you should only do this with patches which you (and the rest of us) agree have no hope of ever getting into mainline.

OK, I'll try to accelerate the effort to send these to you. We'll still need some kind of bucket for the patches that don't apply to recent kernels, but which no one has yet had time to bring up-to-date (or evaluate for permanent dismissal). And dribbling them out, fixing them up, responding to issues - all take time that I can't commit to personally for the next week or so.
I'll let Michael respond whether he can get to this sooner rather than later, as planned.

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================

#11 lore raw
Quoted 17 lines · Tim Bird
Quoted 8 lines · Andrew Morton
Quoted 4 lines · On Wed, 19 Sep 2007 11:03:09 -0700 Tim Bird <[email protected]> wrote:

Recently, the CE Linux forum has been working to revive the Linux-tiny project. At OLS, I asked for interested parties to volunteer to become the new maintainer for the Linux-tiny patchset.

Quoted 6 lines

I volunteer! Send patches to me, cc linux-kernel and celinuv-dev.

Seriously, putting this stuff into some private patch collection should be a complete last resort - you should only do this with patches which you (and the rest of us) agree have no hope of ever getting into mainline.

Quoted 10 lines

OK, I'll try to accelerate the effort to send these to you. We'll still need some kind of bucket for the patches that don't apply to recent kernels, but which no one has yet had time to bring up-to-date (or evaluate for permanent dismissal). And dribbling them out, fixing them up, responding to issues - all take time that I can't commit to personally for the next week or so.
I'll let Michael respond whether he can get to this sooner rather than later, as planned.

Andrew, you're completely right... The patches should all aim at being included into mainline or die.

I'm finishing a sequence of crazy weeks and I will have time to send you patches one by one next week, starting with the easiest ones.

Thanks for your support.

Cheers,

Michael.

Signature · 3 lines
Michael Opdenacker
http://free-electrons.com
+33 621 604 642
#12 lore raw
Quoted 5 lines · Michael Opdenacker

Andrew, you're completely right... The patches should all aim at being included into mainline or die.

I'm finishing a sequence of crazy weeks and I will have time to send you patches one by one next week, starting with the easiest ones.

Well thats good news. In response to the comments made about testing the impact of these patches on big-iron I was going to suggest we ask Andrew to include your patch set in -mm so that it firstly gets at least compiled on big-iron, and secondly so we could think about how to test with some of the options enabled on big-iron.

Knowing nothing about these options, from a test perspective it would be nice if we were able to simply enable "the lot" so we can do "normal" -mm runs and "tiny" -mm runs without any manual intervention?

-apw

author
#13 lore raw

Monster switch for small size (was Linux-tiny revival)

Quoted 3 lines · Andy Whitcroft

Knowing nothing about these options, from a test perspective it would be nice if we were able to simply enable "the lot" so we can do "normal" -mm runs and "tiny" -mm runs without any manual intervention?

I agree completely.

I have been thinking for a while about how to make a "monster switch" (the kind they always seem to have in Frankenstein movies) that switches a whole bunch of settings at once. We currently have methods in the kernel for:
* default (or recommended) config for a particular platform
* all yes - to build as much as possible
* all no - to build as little as possible

The problem with "allno" is that it rarely produces a usable kernel.

There are three possible approaches that I can think of:
1) use a tool to start from default and turn off options to make a small (but still usable) config
* I have a tool to do this now as part of my automated test I haven't published it, but I can if anyone's interested.
2) use the kconfig dependency system to disable stuff automatically if someone chooses the "make_it_small" option.
3) create defconfig_small files for the platforms that care about size

3) is easiest to implement at first. It's trivial to make a new defconfig, and we could easily come up with a convention for them. However, they would be a pain to maintain (this would essentially double the defconfig maintenance), and you'd have to convince people that it's worth carrying these in the mainline tree.

I haven't looked at 2), so I'm not sure exactly what would be involved. I'm not sure if you can centralize all the dependency information in the "make_it_small" option, or if you'd have to spread it out to the related configs. I'm not even sure which arrangement of the info would be the easiest to maintain. Would it be best to have a single size-conscious person maintain the dependencies, or better for config authors to maintain this info in parallel?

Anyway, those are just some thoughts on the subject. Feedback on an acceptable solution would be welcome.
-- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================

#14 lore raw

Re: Monster switch for small size (was Linux-tiny revival)

Quoted 15 lines · Tim Bird
Quoted 3 lines · Andy Whitcroft

Knowing nothing about these options, from a test perspective it would be nice if we were able to simply enable "the lot" so we can do "normal" -mm runs and "tiny" -mm runs without any manual intervention?

I agree completely.

I have been thinking for a while about how to make a "monster switch" (the kind they always seem to have in Frankenstein movies) that switches a whole bunch of settings at once. We currently have methods in the kernel for:
* default (or recommended) config for a particular platform
* all yes - to build as much as possible
* all no - to build as little as possible

Quoted 2 lines

The problem with "allno" is that it rarely produces a usable kernel.

Beyond that, allno doesn't come close to switching everything off.

1) You have to _enable_ CONFIG_EMBEDDED in order to go into that menu and switch _off_ the stuff in there.

2) The stuff CONFIG_EMBEDDED reveals isn't all in that menu. CONFIG_BLOCK is at the top level menu. CONFIG_VT and CONFIG_UNIX98_PTYS are buried down under device drivers->character devices, and there's more sprinkled all over. You have to track it all down and switch it off to get an _actual_ allnoconfig kernel.

(I cut the bit where you reinvent miniconfig. People keep doing this. I dig it up and resubmit it every year or so, so Roman Zippel can shoot it down again. Meanwhile, not only is Firmware Linux happily using it, but I even wrote more documentation at
http://landley.net/code/firmware/new_platform.html although you have to scroll down a bit to get to the stuff about miniconfig...)

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#15 lore raw

Re: Monster switch for small size (was Linux-tiny revival)

Quoted 24 lines · On Thu, 20 Sep 2007 16:41:22 -0500 Rob Landley wrote:
Quoted 15 lines · Tim Bird
Quoted 3 lines · Andy Whitcroft

Knowing nothing about these options, from a test perspective it would be nice if we were able to simply enable "the lot" so we can do "normal" -mm runs and "tiny" -mm runs without any manual intervention?

I agree completely.

I have been thinking for a while about how to make a "monster switch" (the kind they always seem to have in Frankenstein movies) that switches a whole bunch of settings at once. We currently have methods in the kernel for:
* default (or recommended) config for a particular platform
* all yes - to build as much as possible
* all no - to build as little as possible

Quoted 2 lines

The problem with "allno" is that it rarely produces a usable kernel.

Beyond that, allno doesn't come close to switching everything off.

1) You have to _enable_ CONFIG_EMBEDDED in order to go into that menu and switch _off_ the stuff in there.

Quoted 7 lines

2) The stuff CONFIG_EMBEDDED reveals isn't all in that menu. CONFIG_BLOCK is at the top level menu. CONFIG_VT and CONFIG_UNIX98_PTYS are buried down under device drivers->character devices, and there's more sprinkled all over. You have to track it all down and switch it off to get an _actual_ allnoconfig kernel.

(I cut the bit where you reinvent miniconfig. People keep doing this. I dig

I noticed that too.

Quoted 5 lines

it up and resubmit it every year or so, so Roman Zippel can shoot it down again. Meanwhile, not only is Firmware Linux happily using it, but I even wrote more documentation at
http://landley.net/code/firmware/new_platform.html although you have to scroll down a bit to get to the stuff about miniconfig...)

I use it for daily build/boot/run-some-number-like-30-tests kernel testing.

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

#16 lore raw

Re: Monster switch for small size (was Linux-tiny revival)

In reply to Rob Landley
Quoted 6 lines · Rob Landley

(I cut the bit where you reinvent miniconfig. People keep doing this. I dig it up and resubmit it every year or so, so Roman Zippel can shoot it down again. Meanwhile, not only is Firmware Linux happily using it, but I even wrote more documentation at
http://landley.net/code/firmware/new_platform.html although you have to scroll down a bit to get to the stuff about miniconfig...)

DetaolB (see signature) is also using miniconfig, which is good existing code. It can actually be pushed further by storing a /proc/miniconfig.gz rather than the full /proc/config.gz

You can find patches on how to perform this inside the iso of DetaolB v0.6

Signature · 3 lines
Christian
--
http://detaolb.sourceforge.net/, a linux distribution for Qemu with Git inside !
#17 lore raw
In reply to Andrew Morton
Quoted 7 lines · Andrew Morton

On Wed, 19 Sep 2007 11:03:09 -0700

Quoted 3 lines · Tim Bird

Recently, the CE Linux forum has been working to revive the Linux-tiny project. At OLS, I asked for interested parties to volunteer to become the new maintainer for the Linux-tiny patchset.

Quoted 5 lines

I volunteer! Send patches to me, cc linux-kernel and celinuv-dev.

Seriously, putting this stuff into some private patch collection should be a complete last resort - you should only do this with patches which you (and the rest of us) agree have no hope of ever getting into mainline.

History!

<computer historian hat on>

The -tiny tree started out as a separate patch kit of Matt Mackall's, which he stopped updating circa 2.6.14 because he didn't think keeping them out of tree was helping attract other developers, nor was it helping to get them inline. He decided to focus on pushing the existing patches into mainline, and stop maintaining the out of tree patcheset for new releases. His last post on the subject (to the linux-tiny mailing list) was a year ago:
http://selenic.com/pipermail/linux-tiny/2006-March/000314.html

But what happened is that most of the abandoned patches stopped applying to new kernels yet still weren't available in mainline a year later, so Tim and Michael have stepped in to revive the -tiny tree. (Tim talked about this a bit at the CELF BOF at OLS, which is more acronyms than should really show up immediately after one another in any confersation, FYI.)

So yay new tree. Tried without it, didn't work. Broken up to make merging easier, but mainline will probably never _fully_ catch up, any more than it'll catch up with any of the other special-interest development trees. Making -tiny an .hg tree would be really really nice, though... :)

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#18 lore raw
In reply to Tim Bird
Quoted 9 lines · Tim Bird

Recently, the CE Linux forum has been working to revive the Linux-tiny project. At OLS, I asked for interested parties to volunteer to become the new maintainer for the Linux-tiny patchset.

A few candidates came forward, but eventually Michael Opdenacker was selected as the new primary maintainer. A few other people, including John Cooper of Wind River and myself are working to support this effort.

Quoted 4 lines

Recently, many of the Linux-tiny patches have been brought up-to-date and are now available for use with a 2.6.22 kernel. The intent is to test these, and begin mainlining the most effective sub-patches, in the next few months.

Cool!

Could you update http://www.selenic.com/linux-tiny/ to mention the new maintainer and new URLs?

Quoted 9 lines

Some automated testing has already been set up, with some preliminary results published at a CELF conference in Japan. (See the linux-tiny page below for a link to the presentation.) Hopefully, results publishing will also be automated soon.

We encourage anyone with interest in this project to get involved. If you have ideas how to reduce the static or dynamic memory footprint of Linux, or, even better, patches for this, please let us know about them.

I've been playing with an idea for a while to improve the printk() situation, but it's a more intrusive change than I've had time to bang on.

Right now, the first argument to printk() is a loglevel, but it's handled via string concatenation. I'd like to change that to be an integer, and make it an actual comma-separated first argument. (Mandatory, not optional.)

So instead of:
printk(KERN_NOTICE "Fruit=%d\n", banana);
It would now be:
printk(KERN_NOTICE, "Fruit=%d\n", banana);

Change the header from:
#define KERN_NOTICE "<5>"
to:
#define KERN_NOTICE 5

Then you can change the printk guts to do something vaguely like (untested):
#define printk(arg1, arg2, ...) actual_printk("<" #arg1 ">" arg2, __VA_ARGS__)

And so far no behavior has changed. But now the _fun_ part is, you can add a config symbol for "what is the minimum loglevel I care about?" Set that as a number from 0-9. And then you can define the printk to do:

#define printk(level, str, ...) \

  do { \
    if (level < CONFIG_PRINTK_DOICARE) \
      actual_printk("<" #level ">" str, __VA_ARGS__); \

} while(0);

And viola (however you spell that, I think I'm using the stringed instrument but it's french and I'm not trying to type a diacritical mark anyway), the compiler's dead code eliminator zaps the printks you don't care about so they don't bloat the kernel image. But this doesn't _completely_ eliminate printks, so you can still get the panic() calls and such. You tweak precisly how much bloat you want, using the granularity information that's already there in the source code...

Opinions?

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#19 lore raw
Quoted 17 lines · Rob Landley

I've been playing with an idea for a while to improve the printk() situation, but it's a more intrusive change than I've had time to bang on.

Right now, the first argument to printk() is a loglevel, but it's handled via string concatenation. I'd like to change that to be an integer, and make it an actual comma-separated first argument. (Mandatory, not optional.)

So instead of:
printk(KERN_NOTICE "Fruit=%d\n", banana);
It would now be:
printk(KERN_NOTICE, "Fruit=%d\n", banana);

Change the header from:
#define KERN_NOTICE "<5>"
to:
#define KERN_NOTICE 5

Quoted 5 lines

Then you can change the printk guts to do something vaguely like (untested):
#define printk(arg1, arg2, ...) actual_printk("<" #arg1 ">" arg2, __VA_ARGS__)

And so far no behavior has changed. But now the _fun_ part is, you can add a config symbol for "what is the minimum loglevel I care about?"

Given that
a) there're plenty of printks without any KERN_* bloat,
b) there're printks that SHOULD NOT have KERN_* bloat,
c) debugging-by-printk method is widely used and this will force additional typing, head-scratching and swear words
d) time wasted on pointless discussions whether some particular printk ALERT or CRIT
e) flag day for printk,

I think that this idea is not worth it.

Quoted 7 lines

#define printk(level, str, ...) \

  do { \
    if (level < CONFIG_PRINTK_DOICARE) \
      actual_printk("<" #level ">" str, __VA_ARGS__); \

} while(0);

Opinions?

Ick.

Alexey "ignore_loglevel" Dobriyan

include/linux/kernel.h +10 -0
@@ -52,6 +52,7 @@ extern const char linux_proc_banner[];
  */
 #define upper_32_bits(n) ((u32)(((n) >> 16) >> 16))
 
+#ifdef CONFIG_FOO
 #define	KERN_EMERG	"<0>"	/* system is unusable			*/
 #define	KERN_ALERT	"<1>"	/* action must be taken immediately	*/
 #define	KERN_CRIT	"<2>"	/* critical conditions			*/
@@ -59,6 +60,15 @@ extern const char linux_proc_banner[];
 #define	KERN_WARNING	"<4>"	/* warning conditions			*/
 #define	KERN_NOTICE	"<5>"	/* normal but significant condition	*/
 #define	KERN_INFO	"<6>"	/* informational			*/
+#else
+#define	KERN_EMERG	""
+#define	KERN_ALERT	""
+#define	KERN_CRIT	""
+#define	KERN_ERR	""
+#define	KERN_WARNING	""
+#define	KERN_NOTICE	""
+#define	KERN_INFO	""
+#endif
 #define	KERN_DEBUG	"<7>"	/* debug-level messages			*/
 
 extern int console_printk[];
author
#20 lore raw

printk proposal - (was Linux-tiny project revival)

Quoted 3 lines · Alexey Dobriyan

Given that
a) there're plenty of printks without any KERN_* bloat,
b) there're printks that SHOULD NOT have KERN_* bloat,

Just to clarify, which bloat are you concerned about? I presume source code bloat (but maybe you mean message size bloat, or object code bloat)?

-- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================

#21 lore raw

Re: printk proposal - (was Linux-tiny project revival)

Quoted 8 lines · Tim Bird
Quoted 3 lines · Alexey Dobriyan

Given that
a) there're plenty of printks without any KERN_* bloat,
b) there're printks that SHOULD NOT have KERN_* bloat,

Just to clarify, which bloat are you concerned about? I presume source code bloat (but maybe you mean message size bloat, or object code bloat)?

Users of ignore_loglevel still has "<[0-7]>" prefixes in kernel image, yes. On source level, too -- as someone who never saw value in them, KERN_* are just needless characters, making code harder to read.

#22 lore raw

Re: printk proposal - (was Linux-tiny project revival)

Quoted 6 lines · Alexey Dobriyan
Quoted 5 lines · Tim Bird
Quoted 3 lines · Alexey Dobriyan

Given that
a) there're plenty of printks without any KERN_* bloat,
b) there're printks that SHOULD NOT have KERN_* bloat,

Quoted 7 lines
Quoted 3 lines

Just to clarify, which bloat are you concerned about? I presume source code bloat (but maybe you mean message size bloat, or object code bloat)?

Users of ignore_loglevel still has "<[0-7]>" prefixes in kernel image, yes. On source level, too -- as someone who never saw value in them, KERN_* are just needless characters, making code harder to read.

Yeah, I bumped into this when I implemented dmesg for busybox. (Had to parse and filter it out.) That said, userspace kind of expects it now...

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#23 lore raw
In reply to Alexey Dobriyan
Quoted 24 lines · Alexey Dobriyan
Quoted 23 lines · Rob Landley

I've been playing with an idea for a while to improve the printk() situation, but it's a more intrusive change than I've had time to bang on.

Right now, the first argument to printk() is a loglevel, but it's handled via string concatenation. I'd like to change that to be an integer, and make it an actual comma-separated first argument. (Mandatory, not optional.)

So instead of:
printk(KERN_NOTICE "Fruit=%d\n", banana);
It would now be:
printk(KERN_NOTICE, "Fruit=%d\n", banana);

Change the header from:
#define KERN_NOTICE "<5>"
to:
#define KERN_NOTICE 5

Then you can change the printk guts to do something vaguely like (untested): #define printk(arg1, arg2, ...) actual_printk("<" #arg1 ">" arg2, __VA_ARGS__)

Quoted 7 lines
Quoted 2 lines

And so far no behavior has changed. But now the _fun_ part is, you can add a config symbol for "what is the minimum loglevel I care about?"

Given that
a) there're plenty of printks without any KERN_* bloat,

b) there're printks that SHOULD NOT have KERN_* bloat,

So define a level 0 that doesn't prepend any level to the string, and have the macro filter that out at the same default level it counts as now. (KERN_INFO, I think?) The tests are all on contants which should resolve at compile time and the dead code eliminator should zap it, even if the macro gets more complicated it shouldn't result in a bigger binary.

Quoted 2 lines

c) debugging-by-printk method is widely used and this will force additional typing, head-scratching and swear words

Because we never change kernel internal APIs. Oh yeah. Never happens.

Quoted 2 lines

d) time wasted on pointless discussions whether some particular printk ALERT or CRIT

Let me get this straight: you're objecting to actually making the printk levels useful enough that developers start to care what they're set to, because then they might be motivated to want some of them changed?

Make it useful, people might care, thus they might talk about it...

Sorry, I'm still missing the downside here.

Quoted 1 line

e) flag day for printk,

That's the main reason I haven't played with it so far, although it would be easy to define a new symbol (dprintk or some such, although I note several drivers are already using that) and transition gradually.

Quoted 1 line

I think that this idea is not worth it.

*Shrug*.

My problem is that switching off printk is the single biggest bloat cutter in the kernel, yet it makes the resulting system very hard to support. It combines a big upside with a big downside, and I'd like something in between.

Quoted 11 lines
Quoted 7 lines

#define printk(level, str, ...) \

  do { \
    if (level < CONFIG_PRINTK_DOICARE) \
      actual_printk("<" #level ">" str, __VA_ARGS__); \

} while(0);

Opinions?

Ick.

Alexey "ignore_loglevel" Dobriyan

But ignore_loglevel doesn't decrease the size of the _binary_. That's what we're talking about here with the -tiny tree. Embedded developers want to squeeze more code onto smaller flash/rom chips. Setting ignore_loglevel does prevent these messages from ever being emitted, but they're still in the kernel image as dead weight. It saves noise but doesn't save _space_.

I'm proposing allowing an ignore_loglevel to remove the unused messages at compile time so they don't take up space. Doing that requires the levels to be integers so they can be compared with < or >, and the remaining changes follow logically. (To me, anyway...)

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#24 lore raw
Quoted 1 line
Quoted 1 line

I think that this idea is not worth it.

Don't use the config option then....

Quoted 3 lines

My problem is that switching off printk is the single biggest bloat cutter in the kernel, yet it makes the resulting system very hard to support. It combines a big upside with a big downside, and I'd like something in between.

It's not such a big downside IMHO. You can support a kernel without printk. Need to debug the kernel without printk? Use a JTAG debugger...

If you have a system that actually configures out printk's, chances are you don't have storage and output mechanisms to do much with the messages anyway. Think embedded _products_ here. Sure the development boards have serial, ethernet, and all that jazz but tens of millions of ARM based gadgets don't.

#25 lore raw
Quoted 4 lines · Jared Hulbert
Quoted 1 line
Quoted 1 line

I think that this idea is not worth it.

Don't use the config option then....

Quoted 8 lines
Quoted 4 lines

My problem is that switching off printk is the single biggest bloat cutter in the kernel, yet it makes the resulting system very hard to support. It combines a big upside with a big downside, and I'd like something in between.

It's not such a big downside IMHO. You can support a kernel without printk. Need to debug the kernel without printk? Use a JTAG debugger...

I don't actually own a jtag. (I do use qemu's gdb support to debug the target kernel, but it's darn awkward and has limited hardware support.)

Quoted 3 lines

If you have a system that actually configures out printk's, chances are you don't have storage and output mechanisms to do much with the messages anyway. Think embedded _products_ here.

I plead the fifth.

Quoted 3 lines

Sure the
development boards have serial, ethernet, and all that jazz but tens of millions of ARM based gadgets don't.

I wonder if that "Monsoon" gadget does? (Sorry, just on my mind today...)

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#26 lore raw
In reply to Rob Landley
Quoted 8 lines

-----Original Message-----
From: [email protected] [mailto:celinux-dev- [email protected]] On Behalf Of Rob Landley
Sent: Thursday, September 20, 2007 3:02 PM
To: Alexey Dobriyan
Cc: Michael Opdenacker; [email protected]; CE Linux Developers

List;

Quoted 7 lines

linux kernel
Subject: [Celinux-dev] Re: [Announce] Linux-tiny project revival

Quoted 3 lines · Alexey Dobriyan
Quoted 2 lines · Rob Landley

I've been playing with an idea for a while to improve the printk() situation, but it's a more intrusive change than I've had time to

bang

Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

on.

Right now, the first argument to printk() is a loglevel, but it's

handled

Quoted 1 line
Quoted 1 line

via string concatenation. I'd like to change that to be an

integer,

Quoted 16 lines

and

Quoted 15 lines
Quoted 15 lines

make it an actual comma-separated first argument. (Mandatory, not optional.)

So instead of:
printk(KERN_NOTICE "Fruit=%d\n", banana);
It would now be:
printk(KERN_NOTICE, "Fruit=%d\n", banana);

Change the header from:
#define KERN_NOTICE "<5>"
to:
#define KERN_NOTICE 5

Then you can change the printk guts to do something vaguely like (untested): #define printk(arg1, arg2, ...) actual_printk("<" #arg1

">"

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

arg2, __VA_ARGS__)

And so far no behavior has changed. But now the _fun_ part is, you

can

Quoted 1 line
Quoted 1 line
Quoted 1 line

add a config symbol for "what is the minimum loglevel I care

about?"

Quoted 7 lines
Quoted 5 lines

Given that
a) there're plenty of printks without any KERN_* bloat,

b) there're printks that SHOULD NOT have KERN_* bloat,

So define a level 0 that doesn't prepend any level to the string, and

have

Quoted 3 lines

the
macro filter that out at the same default level it counts as now. (KERN_INFO, I think?) The tests are all on contants which should

resolve

Quoted 2 lines

at
compile time and the dead code eliminator should zap it, even if the

macro

Quoted 11 lines

gets more complicated it shouldn't result in a bigger binary.

Quoted 2 lines

c) debugging-by-printk method is widely used and this will force additional typing, head-scratching and swear words

Because we never change kernel internal APIs. Oh yeah. Never happens.

Quoted 2 lines

d) time wasted on pointless discussions whether some particular printk ALERT or CRIT

Let me get this straight: you're objecting to actually making the

printk

Quoted 10 lines

levels useful enough that developers start to care what they're set to, because then they might be motivated to want some of them changed?

Make it useful, people might care, thus they might talk about it...

Sorry, I'm still missing the downside here.

Quoted 1 line

e) flag day for printk,

That's the main reason I haven't played with it so far, although it

would

Quoted 2 lines

be
easy to define a new symbol (dprintk or some such, although I note

several

Quoted 7 lines

drivers are already using that) and transition gradually.

Quoted 1 line

I think that this idea is not worth it.

*Shrug*.

My problem is that switching off printk is the single biggest bloat

cutter

Quoted 4 lines

in
the kernel, yet it makes the resulting system very hard to support. It combines a big upside with a big downside, and I'd like something in between.

What about getting even more hard core?

Use compiler tricks to remove ALL the static printk string from the kernel and replace the printk with something that outputs an decimal index followed by tuples, of zero to N, hex-strings on a single line. Then have the syslogd or some other utility take this cryptic output and convolve it with a table (created at compile time) to re-create what would have been dumped to the sys-log ring buffer. This way you strip out most of the static text from the kernel and yet can still re-create the kernlog output.

At least as a post processing operation....

Is this an old idea? I'm guessing this has been at least proposed before....

--mgross

the

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

#define printk(level, str, ...) \

  do { \
    if (level < CONFIG_PRINTK_DOICARE) \
      actual_printk("<" #level ">" str, __VA_ARGS__); \

} while(0);

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Opinions?

Ick.

Alexey "ignore_loglevel" Dobriyan

But ignore_loglevel doesn't decrease the size of the _binary_. That's

what

Quoted 1 line

we're talking about here with the -tiny tree. Embedded developers want

to

Quoted 1 line

squeeze more code onto smaller flash/rom chips. Setting

ignore_loglevel

Quoted 2 lines

does
prevent these messages from ever being emitted, but they're still in

the

Quoted 3 lines

kernel image as dead weight. It saves noise but doesn't save _space_.

I'm proposing allowing an ignore_loglevel to remove the unused messages

at

Quoted 1 line

compile time so they don't take up space. Doing that requires the

levels

Quoted 2 lines

to
be integers so they can be compared with < or >, and the remaining

changes

Quoted 10 lines

follow logically. (To me, anyway...)

Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
_______________________________________________
Celinux-dev mailing list
[email protected]
http://tree.celinuxforum.org/mailman/listinfo/celinux-dev

#27 lore raw

Message codes (Re: [Announce] Linux-tiny project revival)

* Thu, 20 Sep 2007 15:15:47 -0700
* X-MimeOLE: Produced By Microsoft Exchange V6.5
[]

Quoted 9 lines
Quoted 3 lines

*Shrug*.

My problem is that switching off printk is the single biggest bloat

cutter

Quoted 4 lines

in
the kernel, yet it makes the resulting system very hard to support. It combines a big upside with a big downside, and I'd like something in between.

Quoted 5 lines

What about getting even more hard core?

Use compiler tricks to remove ALL the static printk string from the kernel and replace the printk with something that outputs an decimal index followed by tuples, of zero to N, hex-strings on a single line.

Not all, but critical info, that must exist in human-readable form of course.

Quoted 7 lines

Then have the syslogd or some other utility take this cryptic output and convolve it with a table (created at compile time) to re-create what would have been dumped to the sys-log ring buffer. This way you strip out most of the static text from the kernel and yet can still re-create the kernlog output.

At least as a post processing operation....

Sure, but a little problem is, that many kernel developers do C (mostly) and Perl (occasionally), i.e. very few do non-trivial userspace (even userspace do too much C and Perl sometimes [:
<http://thread.gmane.org/gmane.comp.lib.glibc.alpha/12739>)

Quoted 2 lines

Is this an old idea? I'm guessing this has been at least proposed before....

Seriously. When in the Windows there are only messages like:

    "Error (Code:0x00002012)".

In the Linux... well, embedded targets, this can be turned in a very efficient thing by means of userspace. On other setups this can be nice and pleasant one, with yet another L10N project, recently promoted by README translations. But,,, see problem above.
____

#28 lore raw

RE: Message codes (Re: [Announce] Linux-tiny project revival)

Quoted 10 lines

-----Original Message-----
From: Oleg Verych [mailto:[email protected]]
Sent: Thursday, September 20, 2007 5:58 PM
To: Gross, Mark
Cc: Rob Landley; Alexey Dobriyan; Michael Opdenacker; linux- [email protected]; CE Linux Developers List; linux kernel
Subject: Message codes (Re: [Announce] Linux-tiny project revival)

Quoted 9 lines

* Thu, 20 Sep 2007 15:15:47 -0700
* X-MimeOLE: Produced By Microsoft Exchange V6.5
[]

Quoted 6 lines
Quoted 3 lines

*Shrug*.

My problem is that switching off printk is the single biggest bloat

cutter

Quoted 2 lines

in
the kernel, yet it makes the resulting system very hard to support.

It

Quoted 11 lines
Quoted 8 lines
Quoted 2 lines

combines a big upside with a big downside, and I'd like something in between.

What about getting even more hard core?

Use compiler tricks to remove ALL the static printk string from the kernel and replace the printk with something that outputs an decimal index followed by tuples, of zero to N, hex-strings on a single line.

Not all, but critical info, that must exist in human-readable form of course.

I disagree. For a production product the you want minimal information to reduce the communication bandwidth required between the remote customer and the support organization.

In fact there is a good argument that you don't what the remote customer to know enough to start guessing. In the support stage of a products life cycle you really don't want guessing or fudging of information based on guessing. Keeping the output cryptic and short will avoid these things. (That really does happen and it cost a lot in support engineering time)

Quoted 2 lines
Quoted 2 lines

Then have the syslogd or some other utility take this cryptic output

and

Quoted 2 lines
Quoted 2 lines

convolve it with a table (created at compile time) to re-create what would have been dumped to the sys-log ring buffer. This way you

strip

Quoted 1 line
Quoted 1 line

out most of the static text from the kernel and yet can still

re-create

Quoted 5 lines
Quoted 3 lines

the kernlog output.

At least as a post processing operation....

Sure, but a little problem is, that many kernel developers do C

(mostly)

Quoted 4 lines

and Perl (occasionally), i.e. very few do non-trivial userspace (even userspace do too much C and Perl sometimes [:
<http://thread.gmane.org/gmane.comp.lib.glibc.alpha/12739>)

Oh poo. Kernel programmers can use user mode tools and write them too, and I *know* even I could write such a post processing program to take a somewhat compressed and cryptic output and generate what would have been created in the syslog used. (in python)

Quoted 6 lines
Quoted 2 lines

Is this an old idea? I'm guessing this has been at least proposed before....

Seriously. When in the Windows there are only messages like:

   "Error (Code:0x00002012)".

Now it's been ~8 years since I did any serious windows work, but if I recall correctly ALL THE FRICKING TIME!!! When was the last time you've seen a bug check on windows? This is about all you get.

Quoted 8 lines

In the Linux... well, embedded targets, this can be turned in a very efficient thing by means of userspace. On other setups this can be nice and pleasant one, with yet another L10N project, recently promoted by README translations. But,,, see problem above.
____

From you comments I'm not sure you are for or against this idea.

--mgross

#29 lore raw

Re: Message codes (Re: [Announce] Linux-tiny project revival)

Quoted 35 lines · Mark
Quoted 8 lines

-----Original Message-----
From: Oleg Verych [mailto:[email protected]]
Sent: Thursday, September 20, 2007 5:58 PM
To: Gross, Mark
Cc: Rob Landley; Alexey Dobriyan; Michael Opdenacker; linux- [email protected]; CE Linux Developers List; linux kernel
Subject: Message codes (Re: [Announce] Linux-tiny project revival)

Quoted 12 lines

* Thu, 20 Sep 2007 15:15:47 -0700
* X-MimeOLE: Produced By Microsoft Exchange V6.5 []

Quoted 8 lines
Quoted 3 lines

*Shrug*.

My problem is that switching off printk is the single biggest bloat

cutter

Quoted 2 lines

in
the kernel, yet it makes the resulting system very hard to support.

It

Quoted 11 lines
Quoted 8 lines
Quoted 2 lines

combines a big upside with a big downside, and I'd like something in between.

What about getting even more hard core?

Use compiler tricks to remove ALL the static printk string from the kernel and replace the printk with something that outputs an decimal index followed by tuples, of zero to N, hex-strings on a single line.

Not all, but critical info, that must exist in human-readable form of course.

Quoted 6 lines

I disagree. For a production product the you want minimal information to reduce the communication bandwidth required between the remote customer and the support organization.

In fact there is a good argument that you don't what the remote customer to know enough to start guessing.

Don't use Linux then. Open source is a horrible fit for the way you think.

I'm sympathetic to "shrink the binary size" arguments. I'm not really sympathic to "keep the customer in the dark intentionally" arguments, whether the justification is "because they're stupid", "to increase dependency on our support staff", or any other reason.

Quoted 7 lines
Quoted 3 lines

Seriously. When in the Windows there are only messages like:

   "Error (Code:0x00002012)".

Now it's been ~8 years since I did any serious windows work, but if I recall correctly ALL THE FRICKING TIME!!! When was the last time you've seen a bug check on windows? This is about all you get.

I believe he was holding it up as a bad example, and definitely not something we want to emulate.

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#30 lore raw

RE: Message codes (Re: [Announce] Linux-tiny project revival)

Quoted 10 lines

-----Original Message-----
From: Rob Landley [mailto:[email protected]]
Sent: Friday, September 21, 2007 2:16 PM
To: Gross, Mark
Cc: Oleg Verych; Alexey Dobriyan; Michael Opdenacker; linux- [email protected]; CE Linux Developers List; linux kernel
Subject: Re: Message codes (Re: [Announce] Linux-tiny project revival)

Quoted 16 lines
Quoted 15 lines · Mark
Quoted 15 lines

-----Original Message-----
From: Oleg Verych [mailto:[email protected]]
Sent: Thursday, September 20, 2007 5:58 PM
To: Gross, Mark
Cc: Rob Landley; Alexey Dobriyan; Michael Opdenacker; linux- [email protected]; CE Linux Developers List; linux kernel
Subject: Message codes (Re: [Announce] Linux-tiny project revival)

* Thu, 20 Sep 2007 15:15:47 -0700
* X-MimeOLE: Produced By Microsoft Exchange V6.5 []

Quoted 3 lines
Quoted 3 lines

*Shrug*.

My problem is that switching off printk is the single biggest

bloat

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

cutter

Quoted 2 lines

in
the kernel, yet it makes the resulting system very hard to

support.

Quoted 4 lines
Quoted 4 lines

It

Quoted 1 line
Quoted 1 line
Quoted 1 line

combines a big upside with a big downside, and I'd like something

in

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

between.

What about getting even more hard core?

Use compiler tricks to remove ALL the static printk string from

the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

kernel and replace the printk with something that outputs an

decimal

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

index followed by tuples, of zero to N, hex-strings on a single

line.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Not all, but critical info, that must exist in human-readable form

of

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

course.

I disagree. For a production product the you want minimal

information

Quoted 4 lines
Quoted 4 lines

to reduce the communication bandwidth required between the remote customer and the support organization.

In fact there is a good argument that you don't what the remote

customer

Quoted 3 lines
Quoted 1 line

to know enough to start guessing.

Don't use Linux then. Open source is a horrible fit for the way you

think.

I'll do what I like, thank you. I'll continue to use Linux, I think it's a fine fit for the way *I* think.

Quoted 5 lines

I'm sympathetic to "shrink the binary size" arguments. I'm not really sympathic to "keep the customer in the dark intentionally" arguments, whether
the justification is "because they're stupid", "to increase dependency

on

Quoted 2 lines

our
support staff", or any other reason.

You are now talking religion at this point. Do you have a technical or even experience based point to make?

I have experience that has shown that providing too much information in a production product can is confusing, harmful and costly when dealing with consumers. Your opinions won't change that.

I proposed a mechanism for keeping all the printk data and saving space buy doing some table based compressions that has the side effect of making the syslog not human readable. You proposed a mechanism for no-oping out complete log-levels.

Which way hides more from the user? No-oping the log-levels is the easier to implement.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Seriously. When in the Windows there are only messages like:

   "Error (Code:0x00002012)".

Now it's been ~8 years since I did any serious windows work, but if I recall correctly ALL THE FRICKING TIME!!! When was the last time

you've

Quoted 5 lines
Quoted 1 line

seen a bug check on windows? This is about all you get.

I believe he was holding it up as a bad example, and definitely not something
we want to emulate.

There is a time and place for many things. Even error codes.

--mgross

#31 lore raw

RE: Message codes (Re: [Announce] Linux-tiny project revival)

Quoted 8 lines · Mark

Use compiler tricks to remove ALL the static printk string from the kernel and replace the printk with something that outputs a decimal index followed by tuples, of zero to N, hex-strings on

I proposed a mechanism for keeping all the printk data and saving space buy doing some table based compressions that has the side effect of making the syslog not human readable. You proposed a mechanism for no-oping out complete log-levels.

How about compiler tricks to compress the static printk strings? These could be expanded at runtime to use as the format.

Timothy Miller suggested something similar awhile ago.

#32 lore raw

RE: Message codes (Re: [Announce] Linux-tiny project revival)

Quoted 7 lines

-----Original Message-----
From: Joe Perches [mailto:[email protected]]
Sent: Friday, September 21, 2007 3:33 PM
To: Gross, Mark
Cc: Rob Landley; Oleg Verych; Alexey Dobriyan; Michael Opdenacker;

linux-

Quoted 9 lines

[email protected]; CE Linux Developers List; linux kernel
Subject: RE: Message codes (Re: [Announce] Linux-tiny project revival)

Quoted 5 lines · Mark

Use compiler tricks to remove ALL the static printk string from the kernel and replace the printk with something that outputs a decimal index followed by tuples, of zero to N, hex-strings on

I proposed a mechanism for keeping all the printk data and saving

space

Quoted 6 lines
Quoted 3 lines

buy doing some table based compressions that has the side effect of making the syslog not human readable. You proposed a mechanism for no-oping out complete log-levels.

How about compiler tricks to compress the static printk strings? These could be expanded at runtime to use as the format.

You would have to hold the text table (compressed) in memory to do this at run time. That would still be pretty large hunk of memory.

Quoted 2 lines

Timothy Miller suggested something similar awhile ago.

#33 lore raw

Re: Message codes (Re: [Announce] Linux-tiny project revival)

In reply to Rob Landley

On Fri, Sep 21, 2007 at 04:15:39PM -0500, Rob Landley wrote:
[]

Quoted 10 lines
Quoted 9 lines
Quoted 2 lines

Not all, but critical info, that must exist in human-readable form of course.

I disagree. For a production product the you want minimal information to reduce the communication bandwidth required between the remote customer and the support organization.

In fact there is a good argument that you don't what the remote customer to know enough to start guessing.

Quoted 6 lines

Don't use Linux then. Open source is a horrible fit for the way you think.

I'm sympathetic to "shrink the binary size" arguments. I'm not really sympathic to "keep the customer in the dark intentionally" arguments, whether the justification is "because they're stupid", "to increase dependency on our support staff", or any other reason.

{1}

Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

Seriously. When in the Windows there are only messages like:

   "Error (Code:0x00002012)".

Now it's been ~8 years since I did any serious windows work, but if I recall correctly ALL THE FRICKING TIME!!! When was the last time you've seen a bug check on windows? This is about all you get.

I believe he was holding it up as a bad example, and definitely not something we want to emulate.

I tried to show, that keeping users in compete information vacuum is a bad thing. Even without sources, _configuration_ makes another area of mis-working and bugs, usually addressed by reinstalling.

That may be bad example, because here talk is about developers and testers, who are not just ordinary users. And by applying Torvalds's Law, all users are such in some degree. That's why {1} in your reply, Rob, makes perfect sense.

If Mark have a bad experience with lusers only, then i just can say: what a pity! AFAIK nobody can read somebody's plain-bin OOP output.

Anyway, anything must be opted by config options, even schedulers. But maintenance and flame wars rule otherwise :).

What i can propose is form of binary-only "printk", where all info:
diagnostic, error, bug, statistics messages (in not debugging environment, of course), is just fed right to output buffer (see, pa, no kmallocs). Info itself must have structured content, that makes it easy to extract and locate human-readable representation of both message and data.

This doesn't address loglevels, though.

Implementation (seems) as easy as feeding output to od to have unambiguous form of various troublesome bytes, like "0x00" and "0x0A",

Structuring, who is printing, i.e. arch code, fs, driver whatever, must be agreed:

* Profiles[0]: originator's ID of a message is a byte (or word, or double word) 0x01 - arch, 0x02 - fs, 0x03 - net, 0x04 - hw drivers, etc.

* Data itself can be sent in form of [0]

[0] Banana -- extendable protocol for sending and receiving s-expressions
http://twistedmatrix.com/projects/core/documentation/specifications/banana.html

and having shell script with functions, that have names that correspond to actual structured content:
_*_

olecom@flower:/tmp$ sh banana.sh < banana.c >bb
olecom@flower:/tmp$ sh -c '. ./bb ; _07080'
start
olecom@flower:/tmp$ sh -c '. ./bb ; _07081'
ti_startup - product 0000, num configurations 0, configuration value 0
olecom@flower:/tmp$ sh -c '. ./bb ; _07082'
not reached
olecom@flower:/tmp$
olecom@flower:/tmp$ sh -c '. ./bb ; _07081 777 7 8'
ti_startup - product 0x0309, num configurations 7, configuration value 8
olecom@flower:/tmp$

_(banana.c and banana.sh can be found in the ftp /upload on my server)_

Quoted 1 line

From file linux/drivers/usb/serial/ti_usb*c with

[...]

        dbg("%s - product 0x%4X, num configurations %d, configuration value %d",
            __FUNCTION__, le16_to_cpu(dev->descriptor.idProduct),
            dev->descriptor.bNumConfigurations,
            dev->actconfig->desc.bConfigurationValue);

[...]

lets tacke one particular function (transformed a little bit):
_*_

#include <stdio.h>

#define dbg printf
#define ti_startup(foo) main (int argc, char **argv)
#define dev_descriptor_idProduct 3
#define dev_descriptor_bNumConfigurations 4
#define dev_actconfig_desc_bConfigurationValue 5

/* declaration */
int ti_startup(void);

/* implementation */
int ti_startup(void)

{
	dbg("start\n");

	return dbg("%s - product %#.4x, num configurations %d, "
			"configuration value %d\n",
	__FUNCTION__, dev_descriptor_idProduct,
	dev_descriptor_bNumConfigurations,
	dev_actconfig_desc_bConfigurationValue);

	/* bla bla */
	dbg("not reached\n");
}

_*_

* Process this file with this script: *

_*_

# just as an example
USB_SERIAL_ID=07
TI_USB_ID=08
__FILE__="ti_usb_3410_5052.c" # possible
i=0

sed -n '

# finding function body
/^[[:alpha:]]/{
	# found, print it for __FUNCTION__ keyword
	s_[^ ]* *\([^ (]*\).*[^;]$_\1_p;
	t_func ; b ;
	# walking inside of a function
	:_func;
	# load next string from the input
	n;
	# while not the end of a function
	/^}/{
		# insert delimiter, i.e \n\n is the end of a function
		s_.*_\n_p;
		# go out
		b;
	};
	# find "dbg" (or printk, whatever) inside
	/dbg *(/{
		:_dbg;
		# while not the end of argumet list, i.e ";\n", add other line
		/;$/!{N ; b_dbg}
		# change C to sh
		s_dbg *(_@printf _;
		# clear beginnig of the string
		s_^[^@]*@__;
		# if there is a __FUNCTION__ or __func__ keyword
		/__[Ff]/{
			# NOTE: then first %s in format is replaced with $FUNC
			s_%s_$FUNC_
		}
		# strip "\n\tabs"
		s_\n[[:blank:]]*__g;
		# clear the end, i.e |..."format"/,.*//| or |.."format"/).*//|
		s_"[,)].*_"_;
		# print result
		p;
	}
	b_func;
}'| while read FUNC
do
    printf "_$USB_SERIAL_ID$TI_USB_ID(){
FUNC=$FUNC
}
"
    FUNC=_$USB_SERIAL_ID$TI_USB_ID
    while read -r PRINT
       do test "$PRINT" && printf "$FUNC$i() {
$FUNC
%s "'"$@"'"
}
" "$PRINT"
	  i=$(($i+1))
     done
done
_*_

Now switch on imagination and see how printk just puts '\007\010\params'

and userspace after some banana oddity just calls shell functions.

Yet another thing to export to the userspace. But surely only for those who whats that.
--
-o--=O`C
#oo'L O
<___=E M

#34 lore raw
In reply to Rob Landley

Rob Landley <[email protected]> wrote:

| I'm proposing allowing an ignore_loglevel to remove the unused messages at 
| compile time so they don't take up space.  Doing that requires the levels to 
| be integers so they can be compared with < or >, and the remaining changes 
| follow logically.  (To me, anyway...)

Gcc seems to be smart enough to do contant folding on string subscripts, so you don't need to change any of the printk()s. Here is what I mean:

#include <stdio.h>

#define actual_printk printf

#define KERN_ERR        "<3>"   /* error conditions                     */
#define KERN_WARNING    "<4>"   /* warning conditions                   */
#define KERN_NOTICE     "<5>"   /* normal but significant condition     */

#define printk(str, ...) \

  do { \
    if ((str)[0] != '<' || (str)[1] < '5') \
      actual_printk((str), __VA_ARGS__); \

} while(0);

int main(void)

{
	printk(KERN_ERR "error %d\n", 1);
	printk(KERN_WARNING "warning %d\n", 2);
	printk(KERN_NOTICE "notice %d\n", 3);
	printk("no level %d\n", 4);
	return 0;
}
Signature · 3 lines
Dick Streefland                      ////                      Altium BV
[email protected]           (@ @)          http://www.altium.com
--------------------------------oOO--(_)--OOo---------------------------
#35 lore raw
In reply to Rob Landley
Quoted 10 lines · Rob Landley

And so far no behavior has changed. But now the _fun_ part is, you can add a config symbol for "what is the minimum loglevel I care about?" Set that as a number from 0-9. And then you can define the printk to do:

#define printk(level, str, ...) \

  do { \
    if (level < CONFIG_PRINTK_DOICARE) \
      actual_printk("<" #level ">" str, __VA_ARGS__); \

} while(0);

Quoted 7 lines

And viola (however you spell that, I think I'm using the stringed instrument

But this doesn't _completely_ eliminate
printks, so you can still get the panic() calls and such. You tweak precisly how much bloat you want, using the granularity information that's already there in the source code...
Opinions?

I'd rather take the opportunity to convert all the printks to use pr_<level>. That way, you can pick'n'choose if you want arbitrary combinations of KERN_<level> compiled in or not.

#36 lore raw
Quoted 19 lines · Joe Perches
Quoted 10 lines · Rob Landley

And so far no behavior has changed. But now the _fun_ part is, you can add a config symbol for "what is the minimum loglevel I care about?" Set that as a number from 0-9. And then you can define the printk to do:

#define printk(level, str, ...) \

  do { \
    if (level < CONFIG_PRINTK_DOICARE) \
      actual_printk("<" #level ">" str, __VA_ARGS__); \

} while(0);

Quoted 7 lines

And viola (however you spell that, I think I'm using the stringed instrument

But this doesn't _completely_ eliminate
printks, so you can still get the panic() calls and such. You tweak precisly how much bloat you want, using the granularity information that's already there in the source code...
Opinions?

Quoted 3 lines

I'd rather take the opportunity to convert all the printks to use pr_<level>. That way, you can pick'n'choose if you want arbitrary combinations of KERN_<level> compiled in or not.

Or:

    if ((1 << level) & CONFIG_PRINTK_MASK)
	actual_printk("<" #level ">" str, __VA_ARGS__);

But it would indeed be nice to have all of pr_<level> (and not only pr_info() and pr_debug()) anyway.

With kind regards,

Geert Uytterhoeven
Software Architect

Sony Network and Software Technology Center Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium

Phone:    +32 (0)2 700 8453	
Fax:      +32 (0)2 700 8622	
E-mail:   [email protected]	
Internet: http://www.sony-europe.com/

Sony Network and Software Technology Center Europe
A division of Sony Service Centre (Europe) N.V. Registered office: Technologielaan 7 · B-1840 Londerzeel · Belgium VAT BE 0413.825.160 · RPR Brussels
Fortis Bank Zaventem · Swift GEBABEBB08A · IBAN BE39001382358619

#37 lore raw
In reply to Rob Landley
Quoted 12 lines · Rob Landley

I've been playing with an idea for a while to improve the printk() situation, but it's a more intrusive change than I've had time to bang on.

Right now, the first argument to printk() is a loglevel, but it's handled via string concatenation. I'd like to change that to be an integer, and make it an actual comma-separated first argument. (Mandatory, not optional.)

So instead of:
printk(KERN_NOTICE "Fruit=%d\n", banana);
It would now be:
printk(KERN_NOTICE, "Fruit=%d\n", banana);

Quoted 4 lines

Change the header from:
#define KERN_NOTICE "<5>"
to:
#define KERN_NOTICE 5

You have to jump through less hoops if you do:

#define KERN_NOTICE 5,

But the problem remains that there are printk's which don't have a KERN_* as the first argument. Those are also impossible to get rid off in this way, as the loglevel is unknown (and you don't want partially printed messages).

So adding the comma is really needed and in addition all printk's without a loglevel should get one. Which clutters the code and may increase codesize.

A quick scroll through a vmlinux binary shows that there are quite a lot areas consisting only of some repeated pattern. Mostly 0x00, but also 0x90 and ".GCC: (GNU) 4.2.1.". Getting rid of those would save something between 50 and 100KB.

Greetings,

Indan

#38 lore raw
Quoted 15 lines · Indan Zupancic
Quoted 14 lines · Rob Landley

I've been playing with an idea for a while to improve the printk() situation, but it's a more intrusive change than I've had time to bang on.

Right now, the first argument to printk() is a loglevel, but it's handled via string concatenation. I'd like to change that to be an integer, and make it an actual comma-separated first argument. (Mandatory, not optional.)

So instead of:
printk(KERN_NOTICE "Fruit=%d\n", banana);
It would now be:
printk(KERN_NOTICE, "Fruit=%d\n", banana);

Quoted 8 lines
Quoted 4 lines

Change the header from:
#define KERN_NOTICE "<5>"
to:
#define KERN_NOTICE 5

You have to jump through less hoops if you do:

#define KERN_NOTICE 5,

Less change to the source, but the result is less obvious about what it's doing. I'd personally rather have the churn than wind up with magic syntax...

Quoted 8 lines

But the problem remains that there are printk's which don't have a KERN_* as the first argument. Those are also impossible to get rid off in this way, as the loglevel is unknown (and you don't want partially printed messages).

So adding the comma is really needed and in addition all printk's without a loglevel should get one. Which clutters the code and may increase codesize.

It's ok to _explicitly_ not have a loglevel, and thus take a known default. The problem is printing out less than a full line, continuing it later, and not making obvious at compile time what the level of this chunk is.

Quoted 4 lines

A quick scroll through a vmlinux binary shows that there are quite a lot areas consisting only of some repeated pattern. Mostly 0x00, but also 0x90 and ".GCC: (GNU) 4.2.1.". Getting rid of those would save something between 50 and 100KB.

Worse, if you feed an absolute path to O= when you build the kernel out of tree, then it uses absolute paths for all the __FILE__ strings and that makes kernel BIIIIIG. (Did that by accident a while back.) Too bad there's no way to keep the __FILE__ strings compressed at runtime and gunzip them as needed like busybox does with help messages... :)

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#39 lore raw
Quoted 11 lines · Rob Landley
Quoted 4 lines · Indan Zupancic

A quick scroll through a vmlinux binary shows that there are quite a lot areas consisting only of some repeated pattern. Mostly 0x00, but also 0x90 and ".GCC: (GNU) 4.2.1.". Getting rid of those would save something between 50 and 100KB.

Worse, if you feed an absolute path to O= when you build the kernel out of tree, then it uses absolute paths for all the __FILE__ strings and that makes kernel BIIIIIG. (Did that by accident a while back.) Too bad there's no way to keep the __FILE__ strings compressed at runtime and gunzip them as needed like busybox does with help messages... :)

I suspect that can be fixed by changing the built system. How can using O= change the source file path anyway? That seems unnecessary.

It seems to be worse, full pathnames are also used when giving a relative path. (I'm using O=../obj/).

On the other hand, it doesn't seem to cause that much bloat here:

$ strings vmlinux | grep /home/ |wc
    119     181    6400

CC'ing Sam Ravnborg, perhaps he has some ideas.

Greetings,

Indan

#40 lore raw
Quoted 7 lines · Indan Zupancic
Quoted 6 lines · Rob Landley
Quoted 4 lines · Indan Zupancic

A quick scroll through a vmlinux binary shows that there are quite a lot areas consisting only of some repeated pattern. Mostly 0x00, but also 0x90 and ".GCC: (GNU) 4.2.1.". Getting rid of those would save something between 50 and 100KB.

Quoted 8 lines
Quoted 5 lines

Worse, if you feed an absolute path to O= when you build the kernel out of tree, then it uses absolute paths for all the __FILE__ strings and that makes kernel BIIIIIG. (Did that by accident a while back.) Too bad there's no way to keep the __FILE__ strings compressed at runtime and gunzip them as needed like busybox does with help messages... :)

I suspect that can be fixed by changing the built system. How can using O= change the source file path anyway? That seems unnecessary.

Thats a bit unavoidable as the build system works. __FILE__ is passed the filename supplied as argument to gcc.

Try:
echo "char *s = __FILE__;" > sam.c

gcc -E sam.c
This gives you:
# 1 "sam.c"
# 1 "<built-in>"
# 1 "<command line>"
# 1 "sam.c"
char *s="sam.c";

gcc -E ~/sam.c
This gives you:
# 1 "/home/sam/sam.c"
# 1 "<built-in>"
# 1 "<command line>"
# 1 "/home/sam/sam.c"
char *s="/home/sam/sam.c";

So __FILE__ expand differently depending on the path on
the gcc command line.
I once posted a patch to fix up on this, especialy for BUG_ON and friends. The solution was to let kbuild generate the filename and to use this define in the source code.
But a quick grep for __FILE__ in the kernel source made me chicken out. Simply too much chrunch at that time to justify it.

Googeling a bit I found it here: http://lkml.org/lkml/2006/7/8/22
The better approach would be to use at least the patch inside the kernel.
This patch should be easy to update to latest kernel if anyone is up to play with it.

I recall that there was some problems with the path used. But I cannot remember the details.
Andrew had some inputs from his testing IIRC and google should be able to tell the full story.

Sam

Quoted 16 lines

It seems to be worse, full pathnames are also used when giving a relative path. (I'm using O=../obj/).

On the other hand, it doesn't seem to cause that much bloat here:

$ strings vmlinux | grep /home/ |wc
    119     181    6400

CC'ing Sam Ravnborg, perhaps he has some ideas.

Greetings,

Indan

#41 lore raw
Quoted 9 lines · Sam Ravnborg

...
So __FILE__ expand differently depending on the path on
the gcc command line.
I once posted a patch to fix up on this, especialy for BUG_ON and friends. The solution was to let kbuild generate the filename and to use this define in the source code.
But a quick grep for __FILE__ in the kernel source made me chicken out. Simply too much chrunch at that time to justify it.
...

Although it's not a big problem, we have the same issue with gcc warnings and errors displaying the full path with O= builds making the error messages quite long.

The root is that for O= builds we are feeding gcc the full path We might perhaps attack it at this point?

The simplest solution that comes into my mind would be to create links for the source file in the output dir before calling gcc and then give gcc the link as input file.

Quoted 1 line

Sam

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#42 lore raw
Quoted 3 lines · Adrian Bunk

The simplest solution that comes into my mind would be to create links for the source file in the output dir before calling gcc and then give gcc the link as input file.

The way I've been building various packages out-of-tree (including the Linux kernel) is:

cp -sR /path/to/actual/source/tree newdir
cd newdir
configure
make
install
cd ..
rm -rf newdir

The cp -sR creates a new tree of symlinks to the original source, and everything I've tried so far happily builds in such a directory.

(Yes, this breaks on cygwin. Ask me if I care.)

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#43 lore raw
In reply to Rob Landley
Quoted 5 lines · Rob Landley

Worse, if you feed an absolute path to O= when you build the kernel out of tree, then it uses absolute paths for all the __FILE__ strings and that makes kernel BIIIIIG. (Did that by accident a while back.) Too bad there's no way to keep the __FILE__ strings compressed at runtime and gunzip them as needed like busybox does with help messages... :)

What about something *really* hardcore ugly like:

#ifdef __FILE__
#undef __FILE__
#define __FILE__ ""
#endif

(or similar preprocessor blecherousness) if you want to *really* shrink that binary down?

#44 lore raw
Quoted 7 lines · On Fri, 2007-09-21 at 13:16 -0400, [email protected] wrote:

What about something *really* hardcore ugly like:
#ifdef __FILE__
#undef __FILE__
#define __FILE__ ""
#endif
(or similar preprocessor blecherousness) if you want to *really* shrink that binary down?

I prefer removing all __FILE__, __FUNCTION__, __LINE__ uses from printks and defining something that modifies pr_<level>.

Something like:

#define PR_FILE
#define PR_FUNCTION
#define PR_LINE

#if defined PR_FILE && defined PR_FUNCTION && defined PR_LINE
#define PR_FMT "(%s:%s:%u) "
#define PR_ARG , __FILE__ , __FUNCTION__ , __LINE__ #elif defined PR_FILE && defined PR_FUNCTION && !defined PR_LINE #define PR_FMT "(%s:%s) "
#define PR_ARG , __FILE__ , __FUNCTION__
#elif defined PR_FILE && !defined PR_FUNCTION && defined PR_LINE #define PR_FMT "(%s:%u) "
#define PR_ARG , __FILE__ , __LINE__
#elif defined PR_FILE && !defined PR_FUNCTION && !defined PR_LINE #define PR_FMT "(%s) "
#define PR_ARG , __FILE__
#elif !defined PR_FILE && defined PR_FUNCTION && defined PR_LINE #define PR_FMT "(%s:%u) "
#define PR_ARG , __FUNCTION__ , __LINE__
#elif !defined PR_FILE && defined PR_FUNCTION && !defined PR_LINE #define PR_FMT "(%s) "
#define PR_ARG , __FUNCTION__
#elif !defined PR_FILE && !defined PR_FUNCTION && defined PR_LINE #define PR_FMT "(%u) "
#define PR_ARG , __LINE__
#else
#define PR_FMT
#define PR_ARG
#endif

#define pr_info(fmt, arg) printk(KERN_INFO PR_FMT fmt PR_ARG, ##arg)

#45 lore raw
Quoted 11 lines · Joe Perches
Quoted 7 lines · On Fri, 2007-09-21 at 13:16 -0400, [email protected] wrote:

What about something *really* hardcore ugly like:
#ifdef __FILE__
#undef __FILE__
#define __FILE__ ""
#endif
(or similar preprocessor blecherousness) if you want to *really* shrink that binary down?

I prefer removing all __FILE__, __FUNCTION__, __LINE__ uses from printks and defining something that modifies pr_<level>.

pr_level doesn't exist in mainline.

Quoted 1 line

#define pr_info(fmt, arg) printk(KERN_INFO PR_FMT fmt PR_ARG, ##arg)

Do we really need another layer of indirection?

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#46 lore raw
Quoted 2 lines · Rob Landley
Quoted 1 line

from printks and defining something that modifies pr_<level>.

pr_level doesn't exist in mainline.

pr_info and pr_debug do.

pr_alert, pr_emerg, pr_crit, pr_err, and pr_warn could be added.

Quoted 2 lines
Quoted 1 line

#define pr_info(fmt, arg) printk(KERN_INFO PR_FMT fmt PR_ARG, ##arg)

Do we really need another layer of indirection?

It'd make file/function/line cost free for embedded use.

#47 lore raw
In reply to Rob Landley
Quoted 6 lines · Rob Landley
Quoted 3 lines · Indan Zupancic

But the problem remains that there are printk's which don't have a KERN_* as the first argument. Those are also impossible to get rid off in this way, as the loglevel is unknown (and you don't want partially printed messages).

So adding the comma is really needed and in addition all printk's without a loglevel should get one. Which clutters the code and may increase codesize.

It's ok to _explicitly_ not have a loglevel, and thus take a known default. The problem is printing out less than a full line, continuing it later, and not making obvious at compile time what the level of this chunk is.

That's actually a fairly straightforward problem to solve. Really we ought to be able to guarantee that lines from different CPUs are not intermixed. This can end up in a log like this:

info: Current device state: <1>netdev watchdog timeout: eth0 reg1=0xABCD reg2=0xDEADBEEF

Clearly unpleasant, no? Really any logging which wants to print out a bunch of stuff with multiple printk()s should use an API something like this:

Option 1: Buffer allocated with kmalloc(). Sleep-ish context, can handle failures easily

Quoted 7 lines

struct printk_queue qpk;
if (qprintk_kmalloc(&qpk, level, GFP_KERNEL))

	return -ENOMEM;

qprintk(&qpk, "Some string here:");
qprintk(&qpk, " %d, %s, %d", 4, some_string, 42);
qprintk_finish(&qpk);

Option 2: Preallocated per-cpu buffer. preempt_disable()d

Quoted 4 lines

struct printk_queue qpk;
qprintk_percpu(&qpk, level);
[...]
qprintk_finish(&qpk);

Option 3: Preallocated per-cpu interrupts-only buffer. Only from code which may interrupt a preempt_disable() section

Quoted 4 lines

struct printk_queue qpk;
qprintk_irq(&pqk, level)
[...]
qprintk_finish(&qpk);

For all of the above, the final "qprintk_finish()" call would essentially take the printk spinlock and write the contents of the qpk->buffer into the dmesg ringbuffer, inserting ("<%d>", level) at the beginning and after each newline. The buffers would all be some useful fixed size (a page?); if you need more than that then you are probably trying to queue too much and excess data would be truncated.

Since the level would be a constant passed to qprintk_{kmalloc,percpu,irq} in almost every case, you could easily do something like this:

static inline int qprintk_kmalloc(struct printk_queue *qpk,
unsigned int level, gfp_t gfp)

{
	if (level > CONFIG_MAX_LOG_LEVEL) {
		qpk->type = QPRINTK_TYPE_NONE; /* also 0 */
		qpk->buffer = NULL;
		return 0;
	}

	_qprintk_kmalloc(qpk, level, gfp);
}
#define qprintk(QPK, FMT...) do { \
		if ((QPK)->type)
			_qprintk(QPK, FMT);
	} while(0)

With a bit more glue that would cause GCC to notice that for a given qprintk_kmalloc the "qpk->type" is always zero because the level is too high, and therefore it would optimize out *ALL* of the _qprintk_kmalloc(), _qprintk(), and _qprintk_finish() calls.

Cheers,
Kyle Moffett

#48 lore raw
Quoted 4 lines · Kyle Moffett

With a bit more glue that would cause GCC to notice that for a given qprintk_kmalloc the "qpk->type" is always zero because the level is too high, and therefore it would optimize out *ALL* of the _qprintk_kmalloc(), _qprintk(), and _qprintk_finish() calls.

A negative is that lockup conditions swallow partial messages.

Another approach that doesn't require any new buffering is:

id = printk_block_start();
printk_block(id, fmt, ...)
printk_block_end(id)

and have print_block output the id when multiple IDs are concurrently issued.

This requires a trivial tool to post-process the log when messages are interleaved.

#49 lore raw
Quoted 4 lines · Joe Perches
Quoted 1 line · Kyle Moffett

With a bit more glue that would cause GCC to notice that for a given qprintk_kmalloc the "qpk->type" is always zero because the level is too high, and therefore it would optimize out *ALL* of the _qprintk_kmalloc(), _qprintk(), and _qprintk_finish() calls.

A negative is that lockup conditions swallow partial messages.

But typically you don't care if a "partial line" gets swallowed regardless. The only reason people really use partial lines is when they're accumulating a variable number of things into a single line and so a single printk() won't do, and in that case it's really not a problem to "lose" the first half of the line in event of a crash. And hell, if it matters that much you could just make the qprintk_{kmalloc,percpu,irq} functions chain the qpk variables on a little linked list and stuff an smp_wmb() in the _gprint() function after writing the text and before writing the size. That way any panic could very carefully look at the messages being queued during the crash and attempt to write out partial buffers.

It's a technique which in combination with looking at the first 3 characters of the arguments to printk() would let you elide 99% of the non-critical printks pretty easily while only needing to change the much smaller proportion of the printk()s which are partial lines. Furthermore it's pretty easy to grep for the partial-line printk()s and you can even have it emit warnings when you hit a partial-line printk() (it doesn't start with "<"[0-9]">") in -mm to help fix up the last few users and keep people from adding new ones.

Cheers,
Kyle Moffett

author
#50 lore raw
In reply to Rob Landley
Quoted 12 lines · Rob Landley

So instead of:
printk(KERN_NOTICE "Fruit=%d\n", banana);
It would now be:
printk(KERN_NOTICE, "Fruit=%d\n", banana);

Change the header from:
#define KERN_NOTICE "<5>"
to:
#define KERN_NOTICE 5

Then you can change the printk guts to do something vaguely like (untested):
#define printk(arg1, arg2, ...) actual_printk("<" #arg1 ">" arg2, __VA_ARGS__)

...

Quoted 3 lines

[then] the
compiler's dead code eliminator zaps the printks you don't care about so they don't bloat the kernel image.

I agree in principal with the idea, but there are some major practical wrinkles that would have to be worked through.

First, not all printks that are missing a log level should have one. People do stuff like this:

printk(KERN_INFO "interesting info follows:");
...
printk("var5: %d\n", var5);

Or even things that evaluate to:
printk("");

The code inside printk currently has to examine the strings, looking for line feeds and inserting log levels.

Given that there are about 60,000 printks in the kernel (and that's not counting wrappers like dprintk() and other locally-defined functions and macros) it would be a huge task to examine the code and differentiate strings that really start a new log message (and thus should have an attached log level) and strings that don't.
-- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================

#51 lore raw
Quoted 37 lines · Tim Bird
Quoted 13 lines · Rob Landley

So instead of:
printk(KERN_NOTICE "Fruit=%d\n", banana);
It would now be:
printk(KERN_NOTICE, "Fruit=%d\n", banana);

Change the header from:
#define KERN_NOTICE "<5>"
to:
#define KERN_NOTICE 5

Then you can change the printk guts to do something vaguely like (untested): #define printk(arg1, arg2, ...) actual_printk("<" #arg1 ">" arg2, __VA_ARGS__)

...

Quoted 3 lines

[then] the
compiler's dead code eliminator zaps the printks you don't care about so they don't bloat the kernel image.

I agree in principal with the idea, but there are some major practical wrinkles that would have to be worked through.

First, not all printks that are missing a log level should have one. People do stuff like this:

printk(KERN_INFO "interesting info follows:");
...
printk("var5: %d\n", var5);

Or even things that evaluate to:
printk("");

The code inside printk currently has to examine the strings, looking for line feeds and inserting log levels.

Quoted 6 lines

Given that there are about 60,000 printks in the kernel (and that's not counting wrappers like dprintk() and other locally-defined functions and macros) it would be a huge task to examine the code and differentiate strings that really start a new log message (and thus should have an attached log level) and strings that don't.

Hmmm. The hard part isn't making printk(0,blah) mean the same as not having a log level message now, because the current logic already handles it. The problem is that filtering continuations of previous messages involves knowing what log level the previous message was so you know whether or not to filter it.

Yeah, that would take some doing to untangle. An incremental switchever (easy printks first, I.E. the ones that currently specify a loglevel) seems more strongly indicated...

That said, I started this by noting I haven't personally had time to bang on this since I thought of it. You did ask for ideas. :)

Quoted 1 line

-- Tim

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#52 lore raw
In reply to Tim Bird
Quoted 6 lines · Tim Bird

Given that there are about 60,000 printks in the kernel (and that's not counting wrappers like dprintk() and other locally-defined functions and macros) it would be a huge task to examine the code and differentiate strings that really start a new log message (and thus should have an attached log level) and strings that don't.

I've converted most all of that treewide.

printk(KERN_<level> to pr_<level>(

It's pretty automated.

$ cat pr_alert.sh
#!/bin/sh
egrep -r -w --include=*.[ch] -l "printk[[:space:]]*\([[:space:]]*KERN_ALERT" * | \
 xargs perl ../cvt_pr.pl KERN_ALERT pr_alert
$ cat cvt_pr.pl
if ($#ARGV < 3) {
    print "usage: KERN_<level> pr_<level> files...\n";
    exit;
}
for ($i=2; $i<$#ARGV; $i++) {
    PrintkSearchReplace($ARGV[$i], $ARGV[0], $ARGV[1]);
}
sub PrintkSearchReplace{
    my($file, $search, $replace) = @_;

    my $content = "";
    local( $/ );
    open( my $fh, $file ) or die "File not found '$file'\n";
    $content = <$fh>;
    close(my $fh);
    my $orig = $content;

    $content =~ s/\bprintk[[:space:]]*\([[:space:]]*${search}[[:space:]]*([^\"]*)\"([^\\]*)\\n\"/${replace}\(\1 \"\2\"/mgs;
    $content =~ s/\b${replace}\( /${replace}\(/mgs;
    
    if ($orig ne $content)
    {
        open(my $fh, ">${file}") or die "Could not open '$file'\n";
        print $fh $content;
        close(my $fh);
    } 
}
#53 lore raw
Quoted 8 lines · Joe Perches
Quoted 6 lines · Tim Bird

Given that there are about 60,000 printks in the kernel (and that's not counting wrappers like dprintk() and other locally-defined functions and macros) it would be a huge task to examine the code and differentiate strings that really start a new log message (and thus should have an attached log level) and strings that don't.

Quoted 5 lines

I've converted most all of that treewide.

printk(KERN_<level> to pr_<level>(

It's pretty automated.

Perl, being a write-only language, does not help my poor little brain understand what's going on. You convert printk(KERN_INFO, blah) to pr_INFO(blah)? I'm not finding pr_INFO with a grep on the files in 2.6.23-rc7. Is this something you added?

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#54 lore raw
Quoted 1 line · Rob Landley

You convert printk(KERN_INFO, blah) to pr_INFO(blah)?

more or less.
printk(KERN_INFO foo) to pr_info(foo)
printk(KERN_EMERG foo) to pr_emerge(foo)
etc.

Quoted 2 lines

I'm not finding pr_INFO with a grep on the files in 2.6.23-rc7.

I haven't submitted them.

There's a lot of sensible resistance to what appears to be churn. Some of the resistance is historical, merging large changes used to be much more painful pre git, some is just simple resistance to change.

I started with submitting an add of pr_err to kernel.h which Andrew Morton picked up for awhile, but dropped.

I've got a local tree with those changes.

for example:

KERN_EMERG  -> pr_emerg  is  ~100KB
KERN_ALERT  -> pr_alert  is   ~80KB
KERN_CRIT   -> pr_crit   is  ~200KB

KERN_NOTICE -> pr_notice is ~400KB
KERN_INFO -> pr_info is ~2500KB

I intended to strip the "\n" trailer from the messages.

Back to the scripts:

In this case, there are multiple files.

A script that finds all the files that contain a search string, and a perl script that effectively s/search/replace/g on those files.

sed didn't work as well as perl here because I wanted to play with perl a bit and many printk(KERN_<level> foo) function calls are split across multiple lines.

I've still got to show a real use to this change set.

I believe controlling the interleaving of log messages by having multiple statement printks have a start and end, choosing specific message levels in compiled code, and choosing to print file/function/line per compiled code block might do, but more utility to the changes is probably necessary before it could be applied.

cheers, Joe

#55 lore raw
In reply to Rob Landley
Quoted 49 lines · Rob Landley
Quoted 9 lines · Tim Bird

Recently, the CE Linux forum has been working to revive the Linux-tiny project. At OLS, I asked for interested parties to volunteer to become the new maintainer for the Linux-tiny patchset.

A few candidates came forward, but eventually Michael Opdenacker was selected as the new primary maintainer. A few other people, including John Cooper of Wind River and myself are working to support this effort.

Quoted 4 lines

Recently, many of the Linux-tiny patches have been brought up-to-date and are now available for use with a 2.6.22 kernel. The intent is to test these, and begin mainlining the most effective sub-patches, in the next few months.

Cool!

Could you update http://www.selenic.com/linux-tiny/ to mention the new maintainer and new URLs?

Quoted 9 lines

Some automated testing has already been set up, with some preliminary results published at a CELF conference in Japan. (See the linux-tiny page below for a link to the presentation.) Hopefully, results publishing will also be automated soon.

We encourage anyone with interest in this project to get involved. If you have ideas how to reduce the static or dynamic memory footprint of Linux, or, even better, patches for this, please let us know about them.

I've been playing with an idea for a while to improve the printk() situation, but it's a more intrusive change than I've had time to bang on.

Right now, the first argument to printk() is a loglevel, but it's handled via string concatenation. I'd like to change that to be an integer, and make it an actual comma-separated first argument. (Mandatory, not optional.)

So instead of:
printk(KERN_NOTICE "Fruit=%d\n", banana);
It would now be:
printk(KERN_NOTICE, "Fruit=%d\n", banana);

Change the header from:
#define KERN_NOTICE "<5>"
to:
#define KERN_NOTICE 5

Then you can change the printk guts to do something vaguely like (untested):
#define printk(arg1, arg2, ...) actual_printk("<" #arg1 ">" arg2, __VA_ARGS__)

And so far no behavior has changed. But now the _fun_ part is, you can add a config symbol for "what is the minimum loglevel I care about?" Set that as a number from 0-9. And then you can define the printk to do:

#define printk(level, str, ...) \

Quoted 9 lines
  do { \
    if (level < CONFIG_PRINTK_DOICARE) \
      actual_printk("<" #level ">" str, __VA_ARGS__); \

} while(0);

And viola (however you spell that, I think I'm using the stringed instrument but it's french and I'm not trying to type a diacritical mark anyway), the compiler's dead code eliminator zaps the printks you don't care about so they don't bloat the kernel image. But this doesn't _completely_ eliminate printks, so you can still get the panic() calls and such. You tweak precisly how much bloat you want, using the granularity information that's already there in the source code...

Opinions?

All the people who don't have the need will come up with reasons not to do it. Like "saves too little" and "makes debugging hard" (these are the people who don't realize that having no output device and human in the loop makes it hard, too). How about "too complex, would confuse users," that's popular.

I could go into a list of thing people want to take out or keep out for reasons which boil down to "I don't need it" or "leaving it out only bothers lazy users who don't want to convert to {flavor_of_the_month}."

People with really small systems care about every few bytes, people with big systems don't understand (in general) about people with small systems. The best developers do, fortunately, and will probably approve of kernel liposuction.

Signature · 3 lines
Bill Davidsen <[email protected]>
  "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot
#56 lore raw
In reply to Rob Landley
Quoted 17 lines · you

So instead of:
  printk(KERN_NOTICE "Fruit=%d\n", banana);
It would now be:
  printk(KERN_NOTICE, "Fruit=%d\n", banana);

Change the header from:
  #define KERN_NOTICE "<5>"
to:
  #define KERN_NOTICE 5

Then you can change the printk guts to do something vaguely like (untested):
#define printk(arg1, arg2, ...) actual_printk("<" #arg1 ">" arg2, __VA_ARGS__)

And so far no behavior has changed.  But now the _fun_ part is, you can add a config symbol for "what is the minimum loglevel I care about?"  Set that as a number from 0-9.  And then you can define the printk to do:

Quoted 6 lines

#define printk(level, str, ...) \
  do { \
    if (level < CONFIG_PRINTK_DOICARE) \
      actual_printk("<" #level ">" str, __VA_ARGS__); \
  } while(0);

Assuming that we want to go down that road, I think you can do better with more evil macro magic, by using something along the lines of

#define KERN_NOTICE "<5>",

#define PRINTK_CONTINUED "",

#define printk(level, str, ...) \

   do { \
     if (sizeof(level) == 1) /* continued printk */\
	actual_printk(str, __VA_ARGS__); \
     else if ((level[1] - '0') < CONFIG_PRINTK_DOICARE) \
       actual_printk(level str, __VA_ARGS__); \

} while(0);

Then you don't have to change every single printk in the kernel, but only those that don't currently come with a log level. More importantly, you can do the conversion without a flag day, by spreading (an empty) PRINTK_CONTINUED in places that do need a printk without a log level.

	Arnd <><
#57 lore raw
Quoted 15 lines · Arnd Bergmann

Assuming that we want to go down that road, I think you can do better with more evil macro magic, by using something along the lines of

#define KERN_NOTICE "<5>",

#define PRINTK_CONTINUED "",

 #define printk(level, str, ...) \
   do { \
     if (sizeof(level) == 1) /* continued printk */\
	actual_printk(str, __VA_ARGS__); \
     else if ((level[1] - '0') < CONFIG_PRINTK_DOICARE) \
       actual_printk(level str, __VA_ARGS__); \
   } while(0);
Quoted 4 lines

Then you don't have to change every single printk in the kernel, but only those that don't currently come with a log level. More importantly, you can do the conversion without a flag day, by spreading (an empty) PRINTK_CONTINUED in places that do need a printk without a log level.

The problem is, how do you know whether to print a continued printk or not? It depends on the loglevel of the first printk.

So besides compile-time parsing of the source code, replacing printk with loglevel specific alternatives (one way or the other) seems the only option.

Greetings,

Indan

#58 lore raw
Quoted 7 lines · you
Quoted 4 lines

Then you don't have to change every single printk in the kernel, but only those that don't currently come with a log level. More importantly, you can do the conversion without a flag day, by spreading (an empty) PRINTK_CONTINUED in places that do need a printk without a log level.

The problem is, how do you know whether to print a continued printk or not? It depends on the loglevel of the first printk.

Those need to be looked at individually. You can normally see easily from the context whether the missing log level was an accident, or the author actually has multiple printk statements for a single line. In one case, you would add a log level, in the other case, you can add PRINTK_CONTINUED, or something similar. An alternative to PRINTK_CONTINUED might be a new function, e.g. printk_continued() or similar that does not expect a log level.

Quoted 2 lines

So besides compile-time parsing of the source code, replacing printk with loglevel specific alternatives (one way or the other) seems the only option.

That would mean replacing all of them, not just those that currently lack a loglevel.

	Arnd <><
#59 lore raw
Quoted 10 lines · Arnd Bergmann
Quoted 7 lines · you
Quoted 4 lines

Then you don't have to change every single printk in the kernel, but only those that don't currently come with a log level. More importantly, you can do the conversion without a flag day, by spreading (an empty) PRINTK_CONTINUED in places that do need a printk without a log level.

The problem is, how do you know whether to print a continued printk or not? It depends on the loglevel of the first printk.

Those need to be looked at individually. You can normally see easily from

If think you misunderstood:
Say, you compile out everything of DEBUG level.
Say, you have a continued printk() after each and every pr_debug().

Then how is the macro supposed to know (at compile-time) that the continued printk() actually belongs to a DEBUG printk (and not another one)?

Bernd

Signature · 3 lines
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services
#60 lore raw
Quoted 8 lines · Bernd Petrovitsch

If think you misunderstood:
Say, you compile out everything of DEBUG level.
Say, you have a continued printk() after each and every pr_debug().

Then how is the macro supposed to know (at compile-time) that the continued printk() actually belongs to a DEBUG printk (and not another one)?

#if CONFIG_PRINTK_DOICARE >= 3
#define PRINTK_DEBUG_CONTINUED "",
#else
#define PRINTK_DEBUG_CONTINUED PRINTK_DEBUG
#endif
...

#define printk(level, str, ...) \

   do { \
     if (sizeof(level) == 1) /* continued printk */\
        actual_printk(str, __VA_ARGS__); \
     else if ((level[1] - '0') < CONFIG_PRINTK_DOICARE) \
       actual_printk(level str, __VA_ARGS__); \

} while(0);

Now you can mark a continued KERN_ERR call as such. And depending on CONFIG_PRINTK_DOICARE it will either be printed as always or turned into a stand-alone printk() that gets optimized away. Do the same for all the other levels.

Open question remains: do we want to go down that road? I have no idea.

Jörn

Signature · 4 lines
I've never met a human being who would want to read 17,000 pages of
documentation, and if there was, I'd kill him to get him out of the
gene pool.
-- Joseph Costello
#61 lore raw
In reply to Arnd Bergmann
Quoted 12 lines · Arnd Bergmann

#define KERN_NOTICE "<5>",

#define PRINTK_CONTINUED "",

 #define printk(level, str, ...) \
   do { \
     if (sizeof(level) == 1) /* continued printk */\
	actual_printk(str, __VA_ARGS__); \
     else if ((level[1] - '0') < CONFIG_PRINTK_DOICARE) \
       actual_printk(level str, __VA_ARGS__); \
   } while(0);
Quoted 4 lines

Then you don't have to change every single printk in the kernel, but only those that don't currently come with a log level. More importantly, you can do the conversion without a flag day, by spreading (an empty) PRINTK_CONTINUED in places that do need a printk without a log level.

The "change every printk in the kernel" suggestion came from me trying to figure out how to get the printk() calls below a certain log level to optimize out and not take up space in the binary.

The above doesn't address the original cause of the thread, as far as I can tell.

Quoted 1 line
	Arnd <><

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#62 lore raw

Rob Landley <[email protected]> wrote:

| The "change every printk in the kernel" suggestion came from me trying to 
| figure out how to get the printk() calls below a certain log level to 
| optimize out and not take up space in the binary.
| 
| The above doesn't address the original cause of the thread, as far as I can 
| tell.

It does, provided you change the macro definition to:

#define _printk(level, str, ...) \

   do { \
     if (sizeof(level) == 1) /* continued printk */\
      actual_printk(str, __VA_ARGS__); \
     else if ((level[1] - '0') < CONFIG_PRINTK_DOICARE) \
       actual_printk(level str, __VA_ARGS__); \

} while(0);
#define printk(level_str, ...) _printk(level_str, __VA_ARGS__)

Gcc will do constant folding on the string subscripts, and remove the code and the string constants for calls above the desired level.

This is basically the same as I proposed in my earlier message [*], but with the disadvantage that you need to modify all printk() calls without an explicit level and add the ugly comma to the KERN_* macros. Just compile the code in my message with -O and -S and you will see that the KERN_NOTICE call and the corresponding string literal are optimized away.

[*] http://lkml.org/lkml/2007/9/21/151

Signature · 3 lines
Dick Streefland                      ////                      Altium BV
[email protected]           (@ @)          http://www.altium.com
--------------------------------oOO--(_)--OOo---------------------------