is it desirable to improve the build system?
Dear Linux-Kernel Community,
I am a consultant specializing in builds, and I recently worked for a large client company, a world-wide leader in its field, where I overhauled their build system: sped it up by more of an order of magnitude, and improved maintainability, for example making comment-to-code ratio approach 1:1.
A part of their build is a modified Linux kernel. I rebuilt it countless times in various configurations, but made only a few further changes, because those improvements would have a small effect on the whole system, and because they want to stay close to your current release for ease of porting.
Quoted 1 line
From that limited experience, it nevertheless seemed to me, that the Linux kernel build, while correct, is somewhat slow, and the sources could be more readable.
Does the Linux-Kernel Community perceive that is the case?
If so, do you think it is possible to improve?
If so, would such an attempt be welcome, including and especially by, the current maintainer(s) of the build? Of course it would have to be completely backwards-compatible, including to the text output interface and requirements for modules makefiles.
I do apologize if my impressions are simply the result of unfamilliarity and naivete, and that I don't understand the deep reasons why "it has to be this way", and that I am unaware that such attempts were already made by some very skilled people.
Thank you,
Mark
Quoted 14 lines · Mark Galeck
Dear Linux-Kernel Community,
I am a consultant specializing in builds, and I recently worked for a large client company, a world-wide leader in its field, where I overhauled their build system: sped it up by more of an order of magnitude, and improved maintainability, for example making comment-to-code ratio approach 1:1.
A part of their build is a modified Linux kernel. I rebuilt it countless times in various configurations, but made only a few further changes, because those improvements would have a small effect on the whole system, and because they want to stay close to your current release for ease of porting.
Quoted 3 lines
From that limited experience, it nevertheless seemed to me, that the Linux kernel build, while correct, is somewhat slow, and the sources could be more readable.
How is it "slow"? And it has to be correct, fast and non-correct doesn't work well, does it :)
What "sources" are you referring to as being not readable?
Quoted 8 lines
Does the Linux-Kernel Community perceive that is the case?
If so, do you think it is possible to improve?
If so, would such an attempt be welcome, including and especially by, the current maintainer(s) of the build? Of course it would have to be completely backwards-compatible, including to the text output interface and requirements for modules makefiles.
Patches are always gladly accepted, if they work well, please feel free to submit them.
Quoted 4 lines
I do apologize if my impressions are simply the result of unfamilliarity and naivete, and that I don't understand the deep reasons why "it has to be this way", and that I am unaware that such attempts were already made by some very skilled people.
What do you not understand that you think could be changed?
Have you looked at the history of the build code to help understand why things were changed to be they way they are? git should help you out here.
thanks,
greg k-h
Quoted 3 lines
Quoted 3 lines
Linux kernel build, while correct, is somewhat slow, and the sources
could be more readable.
Quoted 1 line · Greg
How is it "slow"?
Well, the proportion of time spent by the CPU cores on activities other than compiling seemed high.
Quoted 1 line
What "sources" are you referring to as being not readable?
Not "not readable at all", just "could be made more readable" - the main Makefile and other main Make files; it seemed that one reason, IMHO, was not enough comments.
Quoted 1 line
What do you not understand that you think could be changed?
I did not make careful notes in that area (details would come back to me if I look at this carefully), but right now I remember two things.
As every child in kindergarten knows, recursive make is bad (except when it is good, which you learn in primary school). One reason is that all that re-parsing costs time. Linux kernel build is very heavy recursive.
Frequent use of FORCE phony prerequisites to circumvent the normal GNU Make recipe avoidance mechanism, and then using a custom recipe mechanism to decide what to execute, seems to go against the philosophy of the tool being used (GNU Make) and as such seems, IMHO, to also waste time.
Of course if one were to attempt a change, the first thing would be to do look carefully at the amount of time spent on such activities, rather than using words such as "seems".
What I don't understand of course is the reasons behind these choices have been made.
Quoted 1 line
Have you looked at the history of the build code to help understand why
things were changed to be they way they are? git should help you out here.
No. IMHO, looking at source, and especially past history thereof, to understand it, is a very inefficient use of one's time, only to be undertaken if all else fails. Much better is looking at comments and documentation, if available, and also, asking well-informed persons such as yourself.
Mark
Quoted 9 lines · Mark Galeck
Quoted 3 lines
Quoted 3 lines
Linux kernel build, while correct, is somewhat slow, and the sources
could be more readable.
Quoted 1 line · Greg
How is it "slow"?
Well, the proportion of time spent by the CPU cores on activities other than compiling seemed high.
Really? What is that proportion?
Quoted 5 lines
Quoted 1 line
What "sources" are you referring to as being not readable?
Not "not readable at all", just "could be made more readable" - the main
Makefileand other main Make files; it seemed that one reason, IMHO, was not enough comments.
Everyone always wants more comments :)
Have you read the documentation about the build system that we have in the kernel Documentation/ directory?
There is also a mailing list just for the build system as well, see the MAINTAINERS file for details.
greg k-h
Hi!
Quoted 7 lines
Does the Linux-Kernel Community perceive that is the case?
If so, do you think it is possible to improve?
If so, would such an attempt be welcome, including and especially by, the current maintainer(s) of the build? Of course it would have to be completely backwards-compatible, including to the text output interface and requirements for modules makefiles.
Yes, speeding up build would be nice. Especially nice would be speeding up "empty" build, when only one or two files changed...
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 11 lines · Pavel Machek
Hi!
Quoted 7 lines
Does the Linux-Kernel Community perceive that is the case?
If so, do you think it is possible to improve?
If so, would such an attempt be welcome, including and especially by, the current maintainer(s) of the build? Of course it would have to be completely backwards-compatible, including to the text output interface and requirements for modules makefiles.
Yes, speeding up build would be nice. Especially nice would be speeding up "empty" build, when only one or two files changed...
The answer to "is it desirable to improve X?" is always "yes." But the only way to make progress in Linux is to actually post patches that "improve X." This is unlike many corporate environments, where you might need to get somebody's approval before you even start trying. In Linux, there's no such thing as prior approval. Even if you could get somebody to say "this looks like a good thing to try," that would be no guarantee that subsequent patches would be accepted.
It's hard enough to get people to comment on actual Linux code changes, i.e., patches. It's almost impossible to get useful feedback on proposals without code.
If you posted patches that made significant improvements in build time, I think people would be very interested.
Bjorn
Quoted 1 line
The answer to "is it desirable to improve X?" is always "yes." But
the only way to make progress in Linux is to actually post patches that "improve X." This is unlike many corporate environments, where you might need to get somebody's approval
Precisely. Please excuse me coming from a corporate background.
In corporate, even if you "get somebody's approval", there may be "other stakeholders" who may "feel ownership" etc etc
Thank you
Mark
Quoted 9 lines · Mark Galeck
Quoted 1 line
The answer to "is it desirable to improve X?" is always "yes." But
the only way to make progress in Linux is to actually post patches that "improve X." This is unlike many corporate environments, where you might need to get somebody's approval
Precisely. Please excuse me coming from a corporate background.
In corporate, even if you "get somebody's approval", there may be "other stakeholders" who may "feel ownership" etc etc
Yes, your approved project may get canned later. But the point was, that you aren't allowed to even get started without someone's approval.
Thanks,
Richard