Perl in the toolchain
Kai, what's your opinion? I suspect you missed my posing to l-k
http://www.uwsg.indiana.edu/hypermail/linux/kernel/0301.3/1203.html
-- Pete
Date: Thu, 30 Jan 2003 14:53:59 +0100
From: Konrad Eisele <[email protected]>
To: "PeteZaitcev" <[email protected]>
Subject: Adding sparc-leon linux to sourcetree
.....
There is also one change I have made on the buildsystem. Because I'm using some perl inline scripts in the $cmd_xxx the >'< and >$< signs in the inline perl scripts cause trouble (perl -e '...$x=....'), the >'< because of the echo command, the >$< when rereading from the xxx..cmd files. Could this be applied to the original file?
scripts/Makefile.lib:
+-------------------------------+
# =========================================================================== # Generic stuff
# ===========================================================================
# function to only execute the passed command if necessary # the ' -> '\'' and $ to $$ substitution are done if $(cmd_$(1)) includes a inline perlscript
if_changed = $(if $(strip $? \ $(filter-out $(cmd_$(1)),$(cmd_$@))\ $(filter-out $(cmd_$@),$(cmd_$(1)))),\ @set -e; \ $(if $($(quiet)cmd_$(1)),echo ' $(subst ','\'',$($(quiet)cmd_$(1)))';) \ $(cmd_$(1)); \ echo 'cmd_$@ := $(subst $$,$$$$,$(subst ','\'',$(cmd_$(1))))' > $(@D)/.$(@F).cmd)
----- End forwarded message -----
Quoted 7 lines · Pete Zaitcev
Kai, what's your opinion? I suspect you missed my posing to l-k
Date: Thu, 30 Jan 2003 14:53:59 +0100
From: Konrad Eisele <[email protected]>
To: "PeteZaitcev" <[email protected]>
Subject: Adding sparc-leon linux to sourcetree
Quoted 5 lines
.....
There is also one change I have made on the buildsystem. Because I'm using some perl inline scripts in the $cmd_xxx the >'< and >$< signs in the inline perl scripts cause trouble (perl -e '...$x=....'), the >'< because of the echo command, the >$< when rereading from the xxx..cmd files. Could this be applied to the original file?
Unfortunately, I cannot find the original posting quoted above, since that would probably reveal where the actual usage of perl is.
Generally, we've been trying to not make perl a prequisite for the kernel build, and I'd like to keep it that way. Except for some arch specific stuff I don't really care about, the uses of perl are for the optional "make checkconfig" etc. (which btw look mostly obsolete and should probably be killed), and for generating some firmware, though by default a shipped version of the generated files is used.
That said, I'm not necessarily opposed to allowing for use of ',$ in the command variables as the change above does (getting that as a patch would be nice, though).
--Kai
Quoted 17 lines
scripts/Makefile.lib:
+-------------------------------+# =========================================================================== # Generic stuff
# ===========================================================================# function to only execute the passed command if necessary # the ' -> '\'' and $ to $$ substitution are done if $(cmd_$(1)) includes a inline perlscriptif_changed = $(if $(strip $? \ $(filter-out $(cmd_$(1)),$(cmd_$@))\ $(filter-out $(cmd_$@),$(cmd_$(1)))),\ @set -e; \ $(if $($(quiet)cmd_$(1)),echo ' $(subst ','\'',$($(quiet)cmd_$(1)))';) \ $(cmd_$(1)); \ echo 'cmd_$@ := $(subst $$,$$$$,$(subst ','\'',$(cmd_$(1))))' > $(@D)/.$(@F).cmd)
Quoted 2 lines · Kai Germaschewski
Generally, we've been trying to not make perl a prequisite for the kernel build, and I'd like to keep it that way. Except for some arch specific
That's pretty much out the window when klibc gets merged, so perl will indeed be a build requirement for all platforms...
Jeff
Hello All, And no one wanted python ? I find the necessity for
extraneous packages silly . Perl for gods sake . JimL my .02
Quoted 6 lines · Jeff Garzik
Quoted 2 lines · Kai Germaschewski
Generally, we've been trying to not make perl a prequisite for the kernel build, and I'd like to keep it that way. Except for some arch specific
That's pretty much out the window when klibc gets merged, so perl will indeed be a build requirement for all platforms...
Jeff
Signature · 5 lines
+------------------------------------------------------------------+
| James W. Laferriere | System Techniques | Give me VMS |
| Network Engineer | P.O. Box 854 | Give me Linux |
| [email protected] | Coudersport PA 16915 | only on AXP |
+------------------------------------------------------------------+Quoted 6 lines · Jeff Garzik
Quoted 2 lines · Kai Germaschewski
Generally, we've been trying to not make perl a prequisite for the kernel build, and I'd like to keep it that way. Except for some arch specific
That's pretty much out the window when klibc gets merged, so perl will indeed be a build requirement for all platforms...
None of the perl scripts looks complicated.
Obivious question is if the same functionality could be achived by a simple c program.
In the tool chain we use small C utilities in favour of for example perl scripts in several places.
I would like to see perl kept out of the short-list of required programs for the main stream kernels.
If the prize to do this is to write one or two small c tools to klibc then I'm willing to pay that.
Sam
Quoted 10 lines · Sam Ravnborg
Quoted 6 lines · Jeff Garzik
Quoted 1 line · Kai Germaschewski
Generally, we've been trying to not make perl a prequisite for the kernel build, and I'd like to keep it that way. Except for some arch specific
That's pretty much out the window when klibc gets merged, so perl will indeed be a build requirement for all platforms...
Quoted 5 lines
None of the perl scripts looks complicated.
Obivious question is if the same functionality could be achived by a simple c program.
In the tool chain we use small C utilities in favour of for example perl scripts in several places.
I've seen a lot of machines, and I've yet to see one that has the gcc toolchain but not perl. Why do you want to keep perl out?
Ben
Signature · 3 lines
Ben Greear <[email protected]> <Ben_Greear AT excite.com> President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear
Quoted 7 lines · On 2003.01.31 Jeff Garzik wrote:
Quoted 2 lines · Kai Germaschewski
Generally, we've been trying to not make perl a prequisite for the kernel build, and I'd like to keep it that way. Except for some arch specific
That's pretty much out the window when klibc gets merged, so perl will indeed be a build requirement for all platforms...
So in short, kernel people:
- do not want perl in the kernel build
- allow qt to pollute the kernel to have a decent gui config tool
- have to rewrite half perl features in C
- but perl will be needed anyways
instead of
- do all parsing in perl, that is what perl is for and what is mainly done in kconfig scripts
- do the config backend in perl, and...
- do the gui in perl-XXX, so you can have perl-GTK, perl-GTK2, perl-QT or perl-Tk, even perl-Xaw (so you get rid of tcl/tk)
I really do not understand...
Signature · 4 lines
J.A. Magallon <[email protected]> \ Software is like sex: werewolf.able.es \ It's better when it's free Mandrake Linux release 9.1 (Cooker) for i586 Linux 2.4.21-pre4-jam1 (gcc 3.2.1 (Mandrake Linux 9.1 3.2.1-5mdk))
Quoted 2 lines · J.A. Magallon
So in short, kernel people:
- do not want perl in the kernel build
Correct, at least for mainstream architectures.
The rationale here is that we already put a lot of constraints on what tools people need to build a kernel. If we can avoid an extra _mandatory_ tool then this will make life easier for a lot of people.
For optional features additional requirements are OK, for example to geneate docbook documentation a lot of extra stuff is needed.
Quoted 1 line
- allow qt to pollute the kernel to have a decent gui config tool
It would be good to get a replacement, but until that shows up. Then yes an optional frontend that uses qt is OK.
kconfig is prepared for and one gtk frontend is on the way. If someone comes up with a decent perl based frontend ten that could be considered.
Quoted 1 line
- but perl will be needed anyways
No.
Quoted 3 lines
instead of
- do all parsing in perl, that is what perl is for and what is mainly done in kconfig scripts
flex and bison is better for this job.
Sam
(sigh, I did not wish to spark a religious debate, though I suppose I should have expected it... :))
Quoted 9 lines · Sam Ravnborg
Quoted 2 lines · J.A. Magallon
So in short, kernel people:
- do not want perl in the kernel buildCorrect, at least for mainstream architectures.
The rationale here is that we already put a lot of constraints on what tools people need to build a kernel. If we can avoid an extra _mandatory_ tool then this will make life easier for a lot of people.
This is a logically correct argument, but also one that ignores basic numbers.
The fact of the matter is, the area of build tools matters most to people who cross-compile their kernels, because every tool is generally hand-built rather than automatically installed on their Linux system. For this audience, as well as the typical non-cross-compiling kernel developer, Perl is on their system.
However, that fact is less significant than the more basic and core argument:
klibc uses perl for text munging. i.e. one of Perl's acknowledged strengths. This is not a case of choosing a favorite script language, but instead a case of choosing "the right tool for the job." Regardless of whether you think Perl is line noise :) or not, from a technical basis Perl is clearly superior to sed+awk in this case.
Therefore, any rewrite of _this_ _particular_ script in C or shell script would be willfully choosing a sub-optimal implementation language for this task. If you take into account the fact that the overwhelming majority of the target audience does indeed have Perl on their system, then that only serves to make it more clear that any such perl-to-C rewrite would not be on any technical nor practical basis at all.
Adding some final thoughts, perl is already used in nooks and crannies in the build system. Instead of being motivated to stomp those out, please [respectfully!] consider that the Perl scripts might be there because an evaluation of the best tool for the job took place. script_asm.pl in drivers/scsi is a favorite example here.
Thanks for raising this subject! I wanted to give your answer some consideration, because it is worth mentioning, and discussing.
Jeff
Followup to: <[email protected]> By author: Jeff Garzik <[email protected]> In newsgroup: linux.dev.kernel
Quoted 10 lines
This is a logically correct argument, but also one that ignores basic numbers.
The fact of the matter is, the area of build tools matters most to people who cross-compile their kernels, because every tool is generally hand-built rather than automatically installed on their Linux system. For this audience, as well as the typical non-cross-compiling kernel developer, Perl is on their system.
Quoted 9 lines
However, that fact is less significant than the more basic and core argument:
klibc uses perl for text munging. i.e. one of Perl's acknowledged strengths. This is not a case of choosing a favorite script language, but instead a case of choosing "the right tool for the job." Regardless of whether you think Perl is line noise :) or not, from a technical basis Perl is clearly superior to sed+awk in this case.
Thanks Jeff :)
To emphasize things a bit further, Perl is:
a) good at munging text;
b) available on basically all development systems;
c) not host- or target-specific.
Thus, I cannot see it as being an issue, and I challenge anyone to find a machine on which they regularly build kernels which doesn't have Perl. Like it or not, today it's as much a part of a general-purpose Unix platform as sed or awk.
Yes, you can write complete shit code in Perl. You can write shit code in any language (Perl does, however, make it easier, so if you're programming in Perl you need to watch out for this.) Yes, you can require 47 different obscure interdependent modules which were just released last week on CPAN, but you can require an equivalent number of obscure libraries in C. Doing that, or require features only available in very recent versions of Perl, would be wholly inappropriate for the kernel build. My personal rule of thumb is that it should work at least as far back as Perl 5.004.
There is one klibc script which possibly ought to be rewritten, and that is the one that uses Digest::MD5 which may not be available on some very old platforms. If so, I'd probably just include the MD5 digest code in the script itself, rather than having to deal with C code that is compiled for the host in the klibc tree.
-hpa
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." Architectures needed: cris ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64
Quoted 4 lines · Jeff Garzik
klibc uses perl for text munging. i.e. one of Perl's acknowledged strengths. This is not a case of choosing a favorite script language, but instead a case of choosing "the right tool for the job."
Choosing the right tool for the right job is an argument that matters in this context.
Two major type of arguments pop's up when selecting diverse tools:
1) "the right tool for the right job"
2) "the most preferred tool"
My main objective is that argument 2) should _not_ be the rationale behind adding a new tool to the mandatory tool chain.
Quoted 3 lines
Therefore, any rewrite of _this_ _particular_ script in C or shell script would be willfully choosing a sub-optimal implementation language for this task.
The perl scripts used in klibc looks straightforward, expect one that uses some md5 CPAN module.
Like with C you can in perl pull in a lot of uncommon stuff.
My main objective is to avoid a lot of noise due to one added tool that may not be required.
Part of the noise will be because people do not understand the syntax.
Quoted 4 lines
If you take into account the fact that the overwhelming majority of the target audience does indeed have Perl on their system, then that only serves to make it more clear that any such perl-to-C rewrite would not be on any technical nor practical basis at all.
One example could be fixdep:
fixdep could have been done in perl as well. Implementing that one in c does not count as an "perl-to-C" translation.
To repeat my point.
There is only one good reason to add additional tools to the mandatory set in the tool-chain. At that reason is that the tool is the right choice, and not just someones favourite.
If that imply a small C utility then fine.
Quoted 2 lines
Thanks for raising this subject! I wanted to give your answer some consideration, because it is worth mentioning, and discussing.
Thanks self. I just want to make sure the technically right choice is taken.
Sam
Quoted 1 line · On 2003.01.31 Sam Ravnborg wrote:
On Fri, Jan 31, 2003 at 10:38:27PM +0100, J.A. Magallon wrote:
[...]
Quoted 4 lines
Quoted 3 lines
instead of
- do all parsing in perl, that is what perl is for and what is mainly done in kconfig scriptsflex and bison is better for this job.
Sure ? KConfig syntax is now context-dependent or the like ?
Signature · 4 lines
J.A. Magallon <[email protected]> \ Software is like sex: werewolf.able.es \ It's better when it's free Mandrake Linux release 9.1 (Cooker) for i586 Linux 2.4.21-pre4-jam1 (gcc 3.2.1 (Mandrake Linux 9.1 3.2.1-5mdk))
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines
The fact of the matter is, the area of build tools matters most to people who cross-compile their kernels, because every tool is generally hand-built rather than automatically installed on their Linux system. For this audience, as well as the typical non-cross-compiling kernel developer, Perl is on their system.
Sure, but I want to self-host, too. I can cross-compile kernels, but cross-compiling modutils is an insane pain, for example, so to this day I never ever used Rusty's new shiny module loader, and thus I never got around to fix dots in export symbols.
For the record, the userland which I posess does have a somewhat working Perl RPM, which originates from Red Hat 5.2, I believe. So, I cannot invoke "missing perl" argument in good conscience. However, I shudder to think what happens if I need to rebuild it for some reason.
Quoted 8 lines
However, that fact is less significant than the more basic and core argument:
klibc uses perl for text munging. i.e. one of Perl's acknowledged strengths. This is not a case of choosing a favorite script language, but instead a case of choosing "the right tool for the job." Regardless of whether you think Perl is line noise :) or not, from a technical basis Perl is clearly superior to sed+awk in this case.
I hear you. Actually, your argument comes in force only after a non-trivial amount of mungling, but to slay your argument I need to see your klibc stuff first.
Quoted 5 lines
Adding some final thoughts, perl is already used in nooks and crannies in the build system. Instead of being motivated to stomp those out, please [respectfully!] consider that the Perl scripts might be there because an evaluation of the best tool for the job took place. script_asm.pl in
drivers/scsiis a favorite example here.
There's also a subject of a skillset. I know nil about Perl. (ok, I hacked sirc long ago. I don't think it counts.) This means I cannot debug and fix your text mungling. Perl not only has a reputation for being perfect for text mungling, but also a reputation for being unreadable by anyone but the author (due to the fact that the same thing can be expressed in a half a dozen different ways - so says Larry Wall himself in my Perl book (ok, he actually makes strict subsets of knowledge, which helps very little in my example ("Programming Perl", p33 - What You Don't Know...))).
-- Pete
Quoted 6 lines · Pete Zaitcev
For the record, the userland which I posess does have a somewhat working Perl RPM, which originates from Red Hat 5.2, I believe. So, I cannot invoke "missing perl" argument in good conscience. However, I shudder to think what happens if I need to rebuild it for some reason.
Well, do "perl -v" and see if it's at least 5.004 (5.004.003 I think would be better.) If so, hpa's requirement that everything work on 5.004 is fine. (And really, I don't believe that requirement is going to be hard to follow, except that it does constrain the modules in use a little bit - I'm pretty sure the Digest::MD5 was missing from the 5.004 default set of modules.)
Quoted 2 lines
There's also a subject of a skillset. I know nil about Perl. (ok, I hacked sirc long ago. I don't think it counts.)
If you hacked sirc... Well, that's an example of incredibly bad Perl, so, you should be able to follow well-written and documented Perl, imo.
(I did an amazing amount of sirc hacking in my time, and it was an exercise in pain, to be honest.)
Signature · 2 lines
Ryan Anderson sometimes Pug Majere
Hi,
Quoted 4 lines · J.A. Magallon
instead of
- do all parsing in perl, that is what perl is for and what is mainly done in kconfig scripts
- do the config backend in perl, and...
Parsing is only the very first step to generate the dependency tree, which is used generate a consistent configuration, so most of the work are dependency calculations.
BTW if we use a script language I prefer ruby. :)
bye, Roman
Quoted 17 lines · J.A. Magallon
Quoted 7 lines · On 2003.01.31 Jeff Garzik wrote:
Quoted 1 line · Kai Germaschewski
Generally, we've been trying to not make perl a prequisite for the kernel build, and I'd like to keep it that way. Except for some arch specific
That's pretty much out the window when klibc gets merged, so perl will indeed be a build requirement for all platforms...
So in short, kernel people:
- do not want perl in the kernel build
- allow qt to pollute the kernel to have a decent gui config tool
- have to rewrite half perl features in C
- but perl will be needed anyways
Quoted 8 lines
instead of
- do all parsing in perl, that is what perl is for and what is mainly done in kconfig scripts
- do the config backend in perl, and...
- do the gui in perl-XXX, so you can have perl-GTK, perl-GTK2, perl-QT or perl-Tk, even perl-Xaw (so you get rid of tcl/tk)I really do not understand...
Well, you appear to be taking the superset of opinions, which is guaranteed to generate a paradox, I should think ;-)
Specifically regarding kconf, it does not require Qt; Qt is merely an optional piece.
For Perl, yes I personally think it is needed anyway. And re-creating Perl's features in C, just to avoid Perl, is not the best technical solution when developers already have Perl installed.
Jeff
Quoted 32 lines · On 2003.02.01 Jeff Garzik wrote:
Quoted 18 lines · J.A. Magallon
Quoted 8 lines · On 2003.01.31 Jeff Garzik wrote:
Quoted 2 lines · Kai Germaschewski
Generally, we've been trying to not make perl a prequisite for the kernel build, and I'd like to keep it that way. Except for some arch specific
That's pretty much out the window when klibc gets merged, so perl will indeed be a build requirement for all platforms...
So in short, kernel people:
- do not want perl in the kernel build
- allow qt to pollute the kernel to have a decent gui config tool
- have to rewrite half perl features in C
- but perl will be needed anywaysQuoted 8 lines
instead of
- do all parsing in perl, that is what perl is for and what is mainly done in kconfig scripts
- do the config backend in perl, and...
- do the gui in perl-XXX, so you can have perl-GTK, perl-GTK2, perl-QT or perl-Tk, even perl-Xaw (so you get rid of tcl/tk)I really do not understand...
Well, you appear to be taking the superset of opinions, which is guaranteed to generate a paradox, I should think ;-)
Quoted 7 lines
Specifically regarding kconf, it does not require Qt; Qt is merely an optional piece.
For Perl, yes I personally think it is needed anyway. And re-creating Perl's features in C, just to avoid Perl, is not the best technical solution when developers already have Perl installed.
As someone said, parsing and string processing is one of the jobs in kernel config. Kernel gurus decided to rewrite the thing in C to avoid the lacks in the current bash-ish kconfig, because writing logic for dependencies, checks and so on is a pain...so let's rewrite the logic handling, _and_ the string processing, btw.
The easies way (from my point of view): write Perl::KConfig in C to do the logic hard work and build the big thing in perl. That will be putting a perl interface on top of klibc ?
Signature · 4 lines
J.A. Magallon <[email protected]> \ Software is like sex: werewolf.able.es \ It's better when it's free Mandrake Linux release 9.1 (Cooker) for i586 Linux 2.4.21-pre4-jam1 (gcc 3.2.1 (Mandrake Linux 9.1 3.2.1-5mdk))
Quoted 8 lines · J.A. Magallon
As someone said, parsing and string processing is one of the jobs in kernel config. Kernel gurus decided to rewrite the thing in C to avoid the lacks in the current bash-ish kconfig, because writing logic for dependencies, checks and so on is a pain...so let's rewrite the logic handling, _and_ the string processing, btw.
The easies way (from my point of view): writePerl::KConfigin C to do the logic hard work and build the big thing in perl. That will be putting a perl interface on top of klibc ?
_Any_ rewrite is pure pedantry and useless work.
Hi,
Quoted 3 lines · J.A. Magallon
The easies way (from my point of view): write
Perl::KConfigin C to do the logic hard work and build the big thing in perl. That will be putting a perl interface on top of klibc ?
You gain _nothing_ by rewritting it in perl. The backend is already a library and a swig interface file exists, so it's already trivial to generate Perl::KConfig. There is absolutely no reason to force people to use perl.
bye, Roman
Quoted 4 lines · On 2003.02.01 Roman Zippel wrote:
Hi,
On Sat, 1 Feb 2003, J.A. Magallon wrote:
Quoted 9 lines
Quoted 3 lines
The easies way (from my point of view): write
Perl::KConfigin C to do the logic hard work and build the big thing in perl. That will be putting a perl interface on top of klibc ?You gain _nothing_ by rewritting it in perl. The backend is already a library and a swig interface file exists, so it's already trivial to generate
Perl::KConfig. There is absolutely no reason to force people to use perl.
No, that was exactly what I tried to say, take nowadays C library, and build a loadable module for perl (it has not to be written in perl).
Signature · 4 lines
J.A. Magallon <[email protected]> \ Software is like sex: werewolf.able.es \ It's better when it's free Mandrake Linux release 9.1 (Cooker) for i586 Linux 2.4.21-pre4-jam1 (gcc 3.2.1 (Mandrake Linux 9.1 3.2.1-5mdk))