packaging issues: some serious, some not so serious
there are a few bugs in the packaging targets, both with local
packaging and remote packaging if you've used "O=" for your kernel build. first, let's consider local builds.
consider the packaging makefile scripts/package/Makefile. these
early lines look entirely superfluous (nothing uses the variable TAR_IGNORE):
...
# Ignore the following files/directories during tar operation TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --exclude CVS
...
next, related to my earlier email, if your kernel source happens to
be in its own filesystem so that there's a "lost+found" directory, any "tar" operation is going to fail due to lack of permissions for that directory:
$ make rpm-pkg ... find: ./lost+found: Permission denied ... tar: kernel-2.6.25rc200015g1309d4e/./lost+found: Cannot open: Permission denied tar: Error exit delayed from previous errors make[1]: *** [rpm-pkg] Error 2 make: *** [rpm-pkg] Error 2 $
it seems to be sufficient to apply the following patch:
scripts/package/Makefile +1 -1
@@ -39,7 +39,7 @@ $(objtree)/kernel.spec: $(MKSPEC) $(srctree)/Makefile rpm-pkg rpm: $(objtree)/kernel.spec FORCE $(MAKE) clean $(PREV) ln -sf $(srctree) $(KERNELPATH) - $(PREV) tar -cz $(RCS_TAR_IGNORE) -f $(KERNELPATH).tar.gz $(KERNELPATH)/. + $(PREV) tar -cz --exclude lost+found $(RCS_TAR_IGNORE) -f $(KERNELPATH).tar.gz $(KERNELPATH)/. $(PREV) rm $(KERNELPATH) set -e; \
but it's probable that a number of other operations need to take a "lost+found" directory into account; that is, anything involving "tar".
observations on remote packaging to come.
rday
--
========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
Have classroom, will lecture.
http://crashcourse.ca Waterloo, Ontario, CANADA
========================================================================
Quoted 37 lines · Robert P. J. Day
there are a few bugs in the packaging targets, both with local
packaging and remote packaging if you've used "O=" for your kernel build. first, let's consider local builds.consider the packaging makefile
scripts/package/Makefile. these
early lines look entirely superfluous (nothing uses the variableTAR_IGNORE):...
# Ignore the following files/directories during tar operationTAR_IGNORE:= --exclude SCCS --exclude BitKeeper --exclude .svn --exclude CVS
...next, related to my earlier email, if your kernel source happens to
be in its own filesystem so that there's a "lost+found" directory, any "tar" operation is going to fail due to lack of permissions for that directory:$ make rpm-pkg ... find: ./lost+found: Permission denied ... tar: kernel-2.6.25rc200015g1309d4e/./lost+found: Cannot open: Permission denied tar: Error exit delayed from previous errors make[1]: *** [rpm-pkg] Error 2 make: *** [rpm-pkg] Error 2 $it seems to be sufficient to apply the following patch:
scripts/package/Makefile +0
-0
scripts/package/Makefile $(objtree)/kernel.spec: $(MKSPEC) $(srctree)/Makefile · line 44
Quoted 7 lines
scripts/package/Makefile +1
-1@@ -39,7 +39,7 @@ $(objtree)/kernel.spec: $(MKSPEC) $(srctree)/Makefile rpm-pkg rpm: $(objtree)/kernel.spec FORCE $(MAKE) clean $(PREV) ln -sf $(srctree) $(KERNELPATH) - $(PREV) tar -cz $(RCS_TAR_IGNORE) -f $(KERNELPATH).tar.gz $(KERNELPATH)/. + $(PREV) tar -cz --exclude lost+found $(RCS_TAR_IGNORE) -f $(KERNELPATH).tar.gz $(KERNELPATH)/. $(PREV) rm $(KERNELPATH)
I would much rather you persuaded your earlier idea list all the directories to visit. This list is known and easy to maintain whereas a blacklist like we have now only will grow bigger over time.
We could add something like:
Makefile +5 -0
@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex +# all dirs +KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init +KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt +export KBUILD_ALL_DIRS + # =========================================================================== # Rules shared between *config targets and build targets
And then use this definition in all find / tar operations. We do not need to change everything in one go as the old RCS_TAR_IGNORE and RCS_FIND_IGNORE can continue as is for a while.
Can you please try out this approach?
Sam
Quoted 23 lines · Sam Ravnborg
I would much rather you persuaded your earlier idea list all the directories to visit. This list is known and easy to maintain whereas a blacklist like we have now only will grow bigger over time.
We could add something like:
Makefile +5
-0@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex +# all dirs +KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init +KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt +export KBUILD_ALL_DIRS + # =========================================================================== # Rules shared between *config targets and build targets
Quoted 5 lines
And then use this definition in all find / tar operations. We do not need to change everything in one go as the old
RCS_TAR_IGNOREandRCS_FIND_IGNOREcan continue as is for a while.Can you please try out this approach?
sure, i prefer that strategy anyway. let me work on it.
rday
--
========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
Have classroom, will lecture.
http://crashcourse.ca Waterloo, Ontario, CANADA
========================================================================
Quoted 6 lines · Sam Ravnborg
We could add something like:
Makefile +0
-0
Makefile export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU · line 365
Quoted 8 lines
Makefile +5
-0@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex +# all dirs +KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init +KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt +export KBUILD_ALL_DIRS +
would you not also need to include the regular files at the top of the source tree as well? for some of the packaging targets, those files should be included as well.
rday
--
========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
Have classroom, will lecture.
http://crashcourse.ca Waterloo, Ontario, CANADA
========================================================================
Quoted 11 lines · Robert P. J. Day
Quoted 9 lines · Sam Ravnborg
We could add something like:
Makefile +0
-0@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex
Quoted 9 lines
Makefile export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU · line 365
Quoted 5 lines
Makefile +5
-0+# all dirs +KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init +KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt +export KBUILD_ALL_DIRS +would you not also need to include the regular files at the top of the source tree as well? for some of the packaging targets, those files should be included as well.
Yes.
'find --max-depth=1 *' or maybe we should provide a seperate list for all the relevant filenmaes?
Sam
Quoted 21 lines · Sam Ravnborg
Quoted 11 lines · Robert P. J. Day
Quoted 9 lines · Sam Ravnborg
We could add something like:
Makefile +0
-0@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --exQuoted 9 lines
Makefile export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU · line 365
Quoted 5 lines
Makefile +5
-0+# all dirs +KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init +KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt +export KBUILD_ALL_DIRS +would you not also need to include the regular files at the top of the source tree as well? for some of the packaging targets, those files should be included as well.
Quoted 3 lines
Yes.
'find --max-depth=1 *' or maybe we should provide a seperate list for all the relevant filenmaes?
oh, dang ... it just occurred to me that it's still a good idea to add dirs like lost+found to the RCS_FIND_IGNORE and RCS_TAR_IGNORE variables since someone might, weirdly, decide to mount a subdirectory of the kernel source tree on a separate partition. unlikely, yes, but it can happen.
it may be that we want a more general variable for those blacklists as well -- ALL_FIND_IGNORE, ALL_TAR_IGNORE, that sort of thing.
thoughts?
rday
--
========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
Have classroom, will lecture.
http://crashcourse.ca Waterloo, Ontario, CANADA
========================================================================
Quoted 27 lines · Robert P. J. Day
Quoted 21 lines · Sam Ravnborg
Quoted 11 lines · Robert P. J. Day
Quoted 9 lines · Sam Ravnborg
We could add something like:
Makefile +0
-0@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --exQuoted 9 lines
Makefile export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU · line 365
Quoted 5 lines
Makefile +5
-0+# all dirs +KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init +KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt +export KBUILD_ALL_DIRS +would you not also need to include the regular files at the top of the source tree as well? for some of the packaging targets, those files should be included as well.
Quoted 3 lines
Yes.
'find --max-depth=1 *' or maybe we should provide a seperate list for all the relevant filenmaes?
Quoted 5 lines
oh, dang ... it just occurred to me that it's still a good idea to add dirs like lost+found to the
RCS_FIND_IGNOREandRCS_TAR_IGNOREvariables since someone might, weirdly, decide to mount a subdirectory of the kernel source tree on a separate partition. unlikely, yes, but it can happen.
If they do I would not care at all.
We should strive to make the normal cases behave as expected and try to come up with general solutions that does not prevent the more exotic cases. But trying to imagine all the weird cases and handle them will only result in an un-maintainable mess.
Sam
Quoted 34 lines · Sam Ravnborg
Quoted 27 lines · Robert P. J. Day
Quoted 21 lines · Sam Ravnborg
Quoted 11 lines · Robert P. J. Day
Quoted 9 lines · Sam Ravnborg
We could add something like:
Makefile +0
-0@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --exQuoted 9 lines
Makefile export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU · line 365
Quoted 5 lines
Makefile +5
-0+# all dirs +KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init +KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt +export KBUILD_ALL_DIRS +would you not also need to include the regular files at the top of the source tree as well? for some of the packaging targets, those files should be included as well.
Quoted 3 lines
Yes.
'find --max-depth=1 *' or maybe we should provide a seperate list for all the relevant filenmaes?Quoted 5 lines
oh, dang ... it just occurred to me that it's still a good idea to add dirs like lost+found to the
RCS_FIND_IGNOREandRCS_TAR_IGNOREvariables since someone might, weirdly, decide to mount a subdirectory of the kernel source tree on a separate partition. unlikely, yes, but it can happen.
Quoted 5 lines
If they do I would not care at all. We should strive to make the normal cases behave as expected and try to come up with general solutions that does not prevent the more exotic cases. But trying to imagine all the weird cases and handle them will only result in an un-maintainable mess.
i agree. from what seemed like a simple fix, this has gotten a bit
out of hand. i think the easiest solution would be to just come up with a single whitelist, and a number of command-specific blacklists, and leave it at that. that's guaranteed to work, and it's trivially extensible.
so, as an example, define the global whitelist:
KBUILD_ALL_FILES := COPYING CREDITS Kbuild MAINTAINERS Makefile README REPORTING-BUGS KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init ipc KBUILD_ALL_DIRS += kernel lib mm net samples scripts security sound usr virt KBUILD_ALL SRCS := $(KBUILD_ALL_FILES) $(KBUILD_ALL_DIRS)
export KBUILD_ALL_FILES KBUILD_ALL_DIRS KBUILD_ALL_SRCS
those variables above are valid for everything and are easily augmented.
in terms of what are currently called RCS_FIND_IGNORE andRCS_TAR_IGNORE, scrap those entirely as they clearly reflect only "rcs"-related objects, and just invent new variables:
FIND_OBJS_TO_IGNORE := ... both files and dirs ... TAR_OBJS_TO_IGNORE := ... both files and dirs ...
and that's it. problem solved. don't try to partition off the RCS-related stuff into its own list of objects, it's not worth the trouble. and note how that entirely solves the lost+found issue, and any other issues that might ever arise.
thoughts?
rday
--
========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
Have classroom, will lecture.
http://crashcourse.ca Waterloo, Ontario, CANADA
========================================================================
Quoted 66 lines · Robert P. J. Day
Quoted 34 lines · Sam Ravnborg
Quoted 27 lines · Robert P. J. Day
Quoted 21 lines · Sam Ravnborg
Quoted 11 lines · Robert P. J. Day
Quoted 9 lines · Sam Ravnborg
We could add something like:
Makefile +0
-0@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --exQuoted 9 lines
Makefile export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU · line 365
Quoted 5 lines
Makefile +5
-0+# all dirs +KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init +KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt +export KBUILD_ALL_DIRS +would you not also need to include the regular files at the top of the source tree as well? for some of the packaging targets, those files should be included as well.
Quoted 3 lines
Yes.
'find --max-depth=1 *' or maybe we should provide a seperate list for all the relevant filenmaes?Quoted 5 lines
oh, dang ... it just occurred to me that it's still a good idea to add dirs like lost+found to the
RCS_FIND_IGNOREandRCS_TAR_IGNOREvariables since someone might, weirdly, decide to mount a subdirectory of the kernel source tree on a separate partition. unlikely, yes, but it can happen.Quoted 5 lines
If they do I would not care at all. We should strive to make the normal cases behave as expected and try to come up with general solutions that does not prevent the more exotic cases. But trying to imagine all the weird cases and handle them will only result in an un-maintainable mess.
i agree. from what seemed like a simple fix, this has gotten a bit
out of hand. i think the easiest solution would be to just come up with a single whitelist, and a number of command-specific blacklists, and leave it at that. that's guaranteed to work, and it's trivially extensible.so, as an example, define the global whitelist:
KBUILD_ALL_FILES:= COPYING CREDITSKbuildMAINTAINERSMakefileREADME REPORTING-BUGSKBUILD_ALL_DIRS:= arch block crypto Documentation drivers fs include init ipcKBUILD_ALL_DIRS+= kernel lib mm net samples scripts security sound usr virtKBUILD_ALLSRCS := $(KBUILD_ALL_FILES) $(KBUILD_ALL_DIRS)
exportKBUILD_ALL_FILESKBUILD_ALL_DIRSKBUILD_ALL_SRCSthose variables above are valid for everything and are easily augmented.
in terms of what are currently called
RCS_FIND_IGNOREandRCS_TAR_IGNORE, scrap those entirely as they clearly reflect only "rcs"-related objects, and just invent new variables:
FIND_OBJS_TO_IGNORE:= ... both files and dirs ...TAR_OBJS_TO_IGNORE:= ... both files and dirs ...
Quoted 6 lines
and that's it. problem solved. don't try to partition off the RCS-related stuff into its own list of objects, it's not worth the trouble. and note how that entirely solves the lost+found issue, and any other issues that might ever arise.
thoughts?
Looks good - more or less in line with how I think we should do it. I assume you cook up a patch for this. Try one of the simple cases first.
Sam