Skip to contentSkip to conversation

packaging issues: some serious, some not so serious

PATCH Robert P. J. Day 2 participants 9 messages Feb 18, 2008 – Feb 19, 2008 Kbuild

lore
Share & tools
lore
#0 lore raw

there are a few bugs in the packaging targets, both with local
packaging and remote packaging if you've used "O=" for your kernel build. first, let's consider local builds.

consider the packaging makefile scripts/package/Makefile. these
early lines look entirely superfluous (nothing uses the variable TAR_IGNORE):

...
# Ignore the following files/directories during tar operation TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --exclude CVS
...

next, related to my earlier email, if your kernel source happens to
be in its own filesystem so that there's a "lost+found" directory, any "tar" operation is going to fail due to lack of permissions for that directory:

$ make rpm-pkg
...
find: ./lost+found: Permission denied
...
tar: kernel-2.6.25rc200015g1309d4e/./lost+found: Cannot open:
Permission denied
tar: Error exit delayed from previous errors
make[1]: *** [rpm-pkg] Error 2
make: *** [rpm-pkg] Error 2
$

it seems to be sufficient to apply the following patch:

scripts/package/Makefile +1 -1
index 5e32607..d3307f1 100644
@@ -39,7 +39,7 @@ $(objtree)/kernel.spec: $(MKSPEC) $(srctree)/Makefile
 rpm-pkg rpm: $(objtree)/kernel.spec FORCE
        $(MAKE) clean
        $(PREV) ln -sf $(srctree) $(KERNELPATH)
-       $(PREV) tar -cz $(RCS_TAR_IGNORE) -f $(KERNELPATH).tar.gz $(KERNELPATH)/.
+       $(PREV) tar -cz --exclude lost+found $(RCS_TAR_IGNORE) -f $(KERNELPATH).tar.gz $(KERNELPATH)/.
        $(PREV) rm $(KERNELPATH)
 
        set -e; \

but it's probable that a number of other operations need to take a "lost+found" directory into account; that is, anything involving "tar".

observations on remote packaging to come.

rday
--

========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
Have classroom, will lecture.

http://crashcourse.ca Waterloo, Ontario, CANADA
========================================================================

#1 lore raw
Quoted 37 lines · Robert P. J. Day

there are a few bugs in the packaging targets, both with local
packaging and remote packaging if you've used "O=" for your kernel build. first, let's consider local builds.

consider the packaging makefile scripts/package/Makefile. these
early lines look entirely superfluous (nothing uses the variable TAR_IGNORE):

...
# Ignore the following files/directories during tar operation TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --exclude CVS
...

next, related to my earlier email, if your kernel source happens to
be in its own filesystem so that there's a "lost+found" directory, any "tar" operation is going to fail due to lack of permissions for that directory:

$ make rpm-pkg
...
find: ./lost+found: Permission denied
...
tar: kernel-2.6.25rc200015g1309d4e/./lost+found: Cannot open:
Permission denied
tar: Error exit delayed from previous errors
make[1]: *** [rpm-pkg] Error 2
make: *** [rpm-pkg] Error 2
$

it seems to be sufficient to apply the following patch:

scripts/package/Makefile +0 -0
index 5e32607..d3307f1 100644

scripts/package/Makefile $(objtree)/kernel.spec: $(MKSPEC) $(srctree)/Makefile · line 44

Quoted 7 lines
scripts/package/Makefile +1 -1
@@ -39,7 +39,7 @@ $(objtree)/kernel.spec: $(MKSPEC) $(srctree)/Makefile
 rpm-pkg rpm: $(objtree)/kernel.spec FORCE
        $(MAKE) clean
        $(PREV) ln -sf $(srctree) $(KERNELPATH)
-       $(PREV) tar -cz $(RCS_TAR_IGNORE) -f $(KERNELPATH).tar.gz $(KERNELPATH)/.
+       $(PREV) tar -cz --exclude lost+found $(RCS_TAR_IGNORE) -f $(KERNELPATH).tar.gz $(KERNELPATH)/.
        $(PREV) rm $(KERNELPATH)
 

I would much rather you persuaded your earlier idea list all the directories to visit. This list is known and easy to maintain whereas a blacklist like we have now only will grow bigger over time.

We could add something like:

Makefile +5 -0
index 0d585c0..fb5bbc3 100644
@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU
 RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS
 export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex
 
+# all dirs
+KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init
+KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt
+export KBUILD_ALL_DIRS
+
 # ===========================================================================
 # Rules shared between *config targets and build targets
 

And then use this definition in all find / tar operations. We do not need to change everything in one go as the old RCS_TAR_IGNORE and RCS_FIND_IGNORE can continue as is for a while.

Can you please try out this approach?

Sam

#2 lore raw
Quoted 23 lines · Sam Ravnborg

I would much rather you persuaded your earlier idea list all the directories to visit. This list is known and easy to maintain whereas a blacklist like we have now only will grow bigger over time.

We could add something like:

Makefile +5 -0
index 0d585c0..fb5bbc3 100644
@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU
 RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS
 export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex
 
+# all dirs
+KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init
+KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt
+export KBUILD_ALL_DIRS
+
 # ===========================================================================
 # Rules shared between *config targets and build targets
 
Quoted 5 lines

And then use this definition in all find / tar operations. We do not need to change everything in one go as the old RCS_TAR_IGNORE and RCS_FIND_IGNORE can continue as is for a while.

Can you please try out this approach?

sure, i prefer that strategy anyway. let me work on it.

rday
--

========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
Have classroom, will lecture.

http://crashcourse.ca Waterloo, Ontario, CANADA
========================================================================

#3 lore raw
In reply to Sam Ravnborg
Quoted 6 lines · Sam Ravnborg

We could add something like:

Makefile +0 -0
index 0d585c0..fb5bbc3 100644

Makefile export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU · line 365

Quoted 8 lines
Makefile +5 -0
@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU
 RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS
 export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex
 
+# all dirs
+KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init
+KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt
+export KBUILD_ALL_DIRS
+

would you not also need to include the regular files at the top of the source tree as well? for some of the packaging targets, those files should be included as well.

rday
--

========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
Have classroom, will lecture.

http://crashcourse.ca Waterloo, Ontario, CANADA
========================================================================

#4 lore raw
Quoted 11 lines · Robert P. J. Day
Quoted 9 lines · Sam Ravnborg

We could add something like:

Makefile +0 -0
index 0d585c0..fb5bbc3 100644
@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU
 RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS
 export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex
 
Quoted 9 lines

Makefile export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU · line 365

Quoted 5 lines
Makefile +5 -0
+# all dirs
+KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init
+KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt
+export KBUILD_ALL_DIRS
+

would you not also need to include the regular files at the top of the source tree as well? for some of the packaging targets, those files should be included as well.

Yes.
'find --max-depth=1 *' or maybe we should provide a seperate list for all the relevant filenmaes?

Sam

#5 lore raw
Quoted 21 lines · Sam Ravnborg
Quoted 11 lines · Robert P. J. Day
Quoted 9 lines · Sam Ravnborg

We could add something like:

Makefile +0 -0
index 0d585c0..fb5bbc3 100644
@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU
 RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS
 export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex
 
Quoted 9 lines

Makefile export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU · line 365

Quoted 5 lines
Makefile +5 -0
+# all dirs
+KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init
+KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt
+export KBUILD_ALL_DIRS
+

would you not also need to include the regular files at the top of the source tree as well? for some of the packaging targets, those files should be included as well.

Quoted 3 lines

Yes.
'find --max-depth=1 *' or maybe we should provide a seperate list for all the relevant filenmaes?

oh, dang ... it just occurred to me that it's still a good idea to add dirs like lost+found to the RCS_FIND_IGNORE and RCS_TAR_IGNORE variables since someone might, weirdly, decide to mount a subdirectory of the kernel source tree on a separate partition. unlikely, yes, but it can happen.

it may be that we want a more general variable for those blacklists as well -- ALL_FIND_IGNORE, ALL_TAR_IGNORE, that sort of thing.

thoughts?

rday
--

========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
Have classroom, will lecture.

http://crashcourse.ca Waterloo, Ontario, CANADA
========================================================================

#6 lore raw
Quoted 27 lines · Robert P. J. Day
Quoted 21 lines · Sam Ravnborg
Quoted 11 lines · Robert P. J. Day
Quoted 9 lines · Sam Ravnborg

We could add something like:

Makefile +0 -0
index 0d585c0..fb5bbc3 100644
@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU
 RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS
 export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex
 
Quoted 9 lines

Makefile export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU · line 365

Quoted 5 lines
Makefile +5 -0
+# all dirs
+KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init
+KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt
+export KBUILD_ALL_DIRS
+

would you not also need to include the regular files at the top of the source tree as well? for some of the packaging targets, those files should be included as well.

Quoted 3 lines

Yes.
'find --max-depth=1 *' or maybe we should provide a seperate list for all the relevant filenmaes?

Quoted 5 lines

oh, dang ... it just occurred to me that it's still a good idea to add dirs like lost+found to the RCS_FIND_IGNORE and RCS_TAR_IGNORE variables since someone might, weirdly, decide to mount a subdirectory of the kernel source tree on a separate partition. unlikely, yes, but it can happen.

If they do I would not care at all.
We should strive to make the normal cases behave as expected and try to come up with general solutions that does not prevent the more exotic cases. But trying to imagine all the weird cases and handle them will only result in an un-maintainable mess.

Sam

#7 lore raw
Quoted 34 lines · Sam Ravnborg
Quoted 27 lines · Robert P. J. Day
Quoted 21 lines · Sam Ravnborg
Quoted 11 lines · Robert P. J. Day
Quoted 9 lines · Sam Ravnborg

We could add something like:

Makefile +0 -0
index 0d585c0..fb5bbc3 100644
@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU
 RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS
 export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex
 
Quoted 9 lines

Makefile export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU · line 365

Quoted 5 lines
Makefile +5 -0
+# all dirs
+KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init
+KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt
+export KBUILD_ALL_DIRS
+

would you not also need to include the regular files at the top of the source tree as well? for some of the packaging targets, those files should be included as well.

Quoted 3 lines

Yes.
'find --max-depth=1 *' or maybe we should provide a seperate list for all the relevant filenmaes?

Quoted 5 lines

oh, dang ... it just occurred to me that it's still a good idea to add dirs like lost+found to the RCS_FIND_IGNORE and RCS_TAR_IGNORE variables since someone might, weirdly, decide to mount a subdirectory of the kernel source tree on a separate partition. unlikely, yes, but it can happen.

Quoted 5 lines

If they do I would not care at all. We should strive to make the normal cases behave as expected and try to come up with general solutions that does not prevent the more exotic cases. But trying to imagine all the weird cases and handle them will only result in an un-maintainable mess.

i agree. from what seemed like a simple fix, this has gotten a bit
out of hand. i think the easiest solution would be to just come up with a single whitelist, and a number of command-specific blacklists, and leave it at that. that's guaranteed to work, and it's trivially extensible.

so, as an example, define the global whitelist:

KBUILD_ALL_FILES := COPYING CREDITS Kbuild MAINTAINERS Makefile README REPORTING-BUGS KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init ipc KBUILD_ALL_DIRS += kernel lib mm net samples scripts security sound usr virt KBUILD_ALL SRCS := $(KBUILD_ALL_FILES) $(KBUILD_ALL_DIRS)
export KBUILD_ALL_FILES KBUILD_ALL_DIRS KBUILD_ALL_SRCS

those variables above are valid for everything and are easily augmented.

in terms of what are currently called RCS_FIND_IGNORE and
RCS_TAR_IGNORE, scrap those entirely as they clearly reflect only "rcs"-related objects, and just invent new variables:

FIND_OBJS_TO_IGNORE := ... both files and dirs ... TAR_OBJS_TO_IGNORE := ... both files and dirs ...

and that's it. problem solved. don't try to partition off the RCS-related stuff into its own list of objects, it's not worth the trouble. and note how that entirely solves the lost+found issue, and any other issues that might ever arise.

thoughts?

rday
--

========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
Have classroom, will lecture.

http://crashcourse.ca Waterloo, Ontario, CANADA
========================================================================

#8 lore raw
Quoted 66 lines · Robert P. J. Day
Quoted 34 lines · Sam Ravnborg
Quoted 27 lines · Robert P. J. Day
Quoted 21 lines · Sam Ravnborg
Quoted 11 lines · Robert P. J. Day
Quoted 9 lines · Sam Ravnborg

We could add something like:

Makefile +0 -0
index 0d585c0..fb5bbc3 100644
@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU
 RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS
 export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex
 
Quoted 9 lines

Makefile export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU · line 365

Quoted 5 lines
Makefile +5 -0
+# all dirs
+KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init
+KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt
+export KBUILD_ALL_DIRS
+

would you not also need to include the regular files at the top of the source tree as well? for some of the packaging targets, those files should be included as well.

Quoted 3 lines

Yes.
'find --max-depth=1 *' or maybe we should provide a seperate list for all the relevant filenmaes?

Quoted 5 lines

oh, dang ... it just occurred to me that it's still a good idea to add dirs like lost+found to the RCS_FIND_IGNORE and RCS_TAR_IGNORE variables since someone might, weirdly, decide to mount a subdirectory of the kernel source tree on a separate partition. unlikely, yes, but it can happen.

Quoted 5 lines

If they do I would not care at all. We should strive to make the normal cases behave as expected and try to come up with general solutions that does not prevent the more exotic cases. But trying to imagine all the weird cases and handle them will only result in an un-maintainable mess.

i agree. from what seemed like a simple fix, this has gotten a bit
out of hand. i think the easiest solution would be to just come up with a single whitelist, and a number of command-specific blacklists, and leave it at that. that's guaranteed to work, and it's trivially extensible.

so, as an example, define the global whitelist:

KBUILD_ALL_FILES := COPYING CREDITS Kbuild MAINTAINERS Makefile README REPORTING-BUGS KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init ipc KBUILD_ALL_DIRS += kernel lib mm net samples scripts security sound usr virt KBUILD_ALL SRCS := $(KBUILD_ALL_FILES) $(KBUILD_ALL_DIRS)
export KBUILD_ALL_FILES KBUILD_ALL_DIRS KBUILD_ALL_SRCS

those variables above are valid for everything and are easily augmented.

in terms of what are currently called RCS_FIND_IGNORE and
RCS_TAR_IGNORE, scrap those entirely as they clearly reflect only "rcs"-related objects, and just invent new variables:

FIND_OBJS_TO_IGNORE := ... both files and dirs ... TAR_OBJS_TO_IGNORE := ... both files and dirs ...

Quoted 6 lines

and that's it. problem solved. don't try to partition off the RCS-related stuff into its own list of objects, it's not worth the trouble. and note how that entirely solves the lost+found issue, and any other issues that might ever arise.

thoughts?

Looks good - more or less in line with how I think we should do it. I assume you cook up a patch for this. Try one of the simple cases first.

Sam