*clean targets

Hi,

Currently the clean target (as well as dirclean) does nothing useful.

I propose the following change:

# wipe target
clean:
# remove generated files, retain configs
realclean:
# remove generated files, including configs (but not DL_DIR!)
distclean:
# roll a release tarball, in gz, bz2, xz format
release:

hm, something like this as a base (probably incomplete, but better than what we have now)
-------------- next part --------------

index bbd49d4..ddd58ec 100644
@@ -444,19 +444,20 @@ external-deps:
 # Cleanup and misc junk
 #
 #############################################################
+# wipe target
 clean: $(TARGETS_CLEAN)
-	rm -rf $(STAGING_DIR) $(TARGET_DIR) $(IMAGE) $(BUILD_DIR)/.root $(STAMP_DIR)
+	$(Q)rm -rf $(TARGET_DIR) $(IMAGE) $(BUILD_DIR)/.root $(STAMP_DIR)
 
-dirclean: $(TARGETS_DIRCLEAN)
-	rm -rf $(STAGING_DIR) $(TARGET_DIR) $(IMAGE) $(BUILD_DIR)/.root $(STAMP_DIR)
+# remove generated files, retain configs
+dirclean: clean $(TARGETS_DIRCLEAN)
+	$(Q)rm -rf $(STAGING_DIR) $(HOST_DIR) $(TOOLCHAIN_DIR)
 
-distclean:
-ifeq ($(DL_DIR),$(TOPDIR)/dl)
-	rm -rf $(DL_DIR)
-endif
-	rm -rf $(TOOLCHAIN_DIR) $(BUILD_DIR) $(BINARIES_DIR) \
-	.config.cmd
-	$(MAKE) -C $(CONFIG) clean
+# remove generated files, including configs (but not DL_DIR!)
+distclean: clean dirclean
+	$(Q)$(MAKE) -C $(CONFIG) clean
+	$(Q)rm -rf $(TOOLCHAIN_DIR) $(BUILD_DIR) $(BINARIES_DIR) \
+		.config .config.cmd .config.old .auto.deps \
+		$(BR2_DEPENDS_DIR)
 
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 

@@ -573,18 +574,7 @@ defconfig: $(CONFIG)/conf
 source-check: allyesconfig
 	$(MAKE) _source-check
 
-#############################################################
-#
-# Cleanup and misc junk
-#
-#############################################################
-clean:
-	rm -f .config .config.old .config.cmd .tmpconfig.h .lognr.*
-	-$(MAKE) -C $(CONFIG) clean
-
-distclean: clean
-	rm -rf sources/*
-
+clean dirclean distclean:
 
 endif # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 

@@ -635,10 +625,11 @@ help:
 	@echo
 
 release: distclean
-	OUT=buildroot-$$(grep -A2 BR2_VERSION $(CONFIG_CONFIG_IN)|grep default|cut -f2 -d\"); \
-	rm -rf ../$$OUT*; cp -al . ../$$OUT; cd ..; \
-	tar cfz $$OUT.tar.gz --exclude .svn --exclude .git --exclude \*~ $$OUT; \
-	rm -rf $$OUT
+	$(Q)BR2_VERSION=$$(grep -A2 BR2_VERSION $(CONFIG_CONFIG_IN)|grep default|cut -f2 -d\"); \
+	git archive HEAD --format=tar --prefix="buildroot-$$BR2_VERSION/" > ../buildroot-$$BR2_VERSION.tar; \
+	cat ../buildroot-$$BR2_VERSION.tar | bzip2 -9 > ../buildroot-$$BR2_VERSION.tar.bz2 || rm -f ../buildroot-$$BR2_VERSION.tar.bz2; \
+	cat ../buildroot-$$BR2_VERSION.tar | gzip -9 > ../buildroot-$$BR2_VERSION.tar.gz || rm -f ../buildroot-$$BR2_VERSION.tar.gz; \
+	cat ../buildroot-$$BR2_VERSION.tar | xz -9 > ../buildroot-$$BR2_VERSION.tar.xz || rm -f ../buildroot-$$BR2_VERSION.tar.xz
 
 .PHONY: $(noconfig_targets)
 

That target is usually called 'dist'. At least that's what automake puts in your Makefile...

Sven

Sven> That target is usually called 'dist'. At least that's what automake puts Sven> in your Makefile...

Yes, but we've had a release target for a long time (and I use it).

I'm not strongly against renaming it though.

Bye, Peter Korsgaard

we can just
dist release:
but let's keep release, too.

Bernhard> Hi,
Bernhard> Currently the clean target (as well as dirclean) does nothing useful.

Bernhard> I propose the following change:

Bernhard> # wipe target
Bernhard> clean:
Bernhard> # remove generated files, retain configs Bernhard> realclean:
Bernhard> # remove generated files, including configs (but not DL_DIR!) Bernhard> distclean:
Bernhard> # roll a release tarball, in gz, bz2, xz format Bernhard> release:

Sounds sensible. I would prefer to not touch the release target right now, as I don't have xz installed, noone else needs it and I already have my own release script signing the tarballs and uploading them.

Bernhard> hm, something like this as a base (probably incomplete, but better than Bernhard> what we have now)

Bernhard> diff --git a/Makefile b/Makefile
Bernhard> index bbd49d4..ddd58ec 100644
Bernhard> --- a/Makefile
Bernhard> +++ b/Makefile
Bernhard> @@ -444,19 +444,20 @@ external-deps:

 Bernhard>  # Cleanup and misc junk
 Bernhard>  #
 Bernhard>  #############################################################

Bernhard> +# wipe target
Bernhard> clean: $(TARGETS_CLEAN)
Bernhard> - rm -rf $(STAGING_DIR) $(TARGET_DIR) $(IMAGE) $(BUILD_DIR)/.root $(STAMP_DIR)
Bernhard> + $(Q)rm -rf $(TARGET_DIR) $(IMAGE) $(BUILD_DIR)/.root $(STAMP_DIR)

Makes sense. Why would we want to run the -clean targets of all the individual packages first? That just tends to take ages (and fails on buggy packages), and we're nuking the TARGET_DIR afterwards anyway.

Bernhard> -dirclean: $(TARGETS_DIRCLEAN)
Bernhard> - rm -rf $(STAGING_DIR) $(TARGET_DIR) $(IMAGE) $(BUILD_DIR)/.root $(STAMP_DIR)
Bernhard> +# remove generated files, retain configs Bernhard> +dirclean: clean $(TARGETS_DIRCLEAN)
Bernhard> + $(Q)rm -rf $(STAGING_DIR) $(HOST_DIR) $(TOOLCHAIN_DIR)

Same comment about TARGETS_DIRCLEAN.

Bernhard> -distclean:
Bernhard> -ifeq ($(DL_DIR),$(TOPDIR)/dl)
Bernhard> - rm -rf $(DL_DIR)
Bernhard> -endif
Bernhard> - rm -rf $(TOOLCHAIN_DIR) $(BUILD_DIR) $(BINARIES_DIR) \
Bernhard> - .config.cmd
Bernhard> - $(MAKE) -C $(CONFIG) clean
Bernhard> +# remove generated files, including configs (but not DL_DIR!) Bernhard> +distclean: clean dirclean
Bernhard> + $(Q)$(MAKE) -C $(CONFIG) clean
Bernhard> + $(Q)rm -rf $(TOOLCHAIN_DIR) $(BUILD_DIR) $(BINARIES_DIR) \

dirclean already gets rid of those dirs.

Bernhard> + .config .config.cmd .config.old .auto.deps \
Bernhard> + $(BR2_DEPENDS_DIR)
Bernhard> release: distclean
Bernhard> - OUT=buildroot-$$(grep -A2 BR2_VERSION $(CONFIG_CONFIG_IN)|grep default|cut -f2 -d\"); \
Bernhard> - rm -rf ../$$OUT*; cp -al . ../$$OUT; cd ..; \
Bernhard> - tar cfz $$OUT.tar.gz --exclude .svn --exclude .git --exclude \*~ $$OUT; \
Bernhard> - rm -rf $$OUT
Bernhard> + $(Q)BR2_VERSION=$$(grep -A2 BR2_VERSION $(CONFIG_CONFIG_IN)|grep default|cut -f2 -d\"); \
Bernhard> + git archive HEAD --format=tar --prefix="buildroot-$$BR2_VERSION/" > ../buildroot-$$BR2_VERSION.tar; \

The git archive part is fine by me, but my script just expects the .tar.gz, so I would like to keep it like that (for now atleast).

Bye, Peter Korsgaard

There is no need for xz, if it doesn't exists then no .tar.xz will be created.

calling the individual targets can be dropped, yes. I just didn't remove it (yet).

ack.

s/those dirs/TOOLCHAIN_DIR/
ack.

It would be nice if you could still pick it up and also sign/upload the .bz2

Hi,

Bernhard> There is no need for xz, if it doesn't exists then no .tar.xz Bernhard> will be created.

Ahh yes, missed the || at the end of the line.

Bernhard> calling the individual targets can be dropped, yes. I just Bernhard> didn't remove Bernhard> it (yet).

OK. I notice now that your have rm -rf $(IMAGE) instead of $(BINARIES_DIR). I guess we want the latter, as $(IMAGE) is just a prefix, not a file name.

Bernhard> -dirclean: $(TARGETS_DIRCLEAN)
Bernhard> - rm -rf $(STAGING_DIR) $(TARGET_DIR) $(IMAGE) $(BUILD_DIR)/.root $(STAMP_DIR)
Bernhard> +# remove generated files, retain configs Bernhard> +dirclean: clean $(TARGETS_DIRCLEAN)
Bernhard> + $(Q)rm -rf $(STAGING_DIR) $(HOST_DIR) $(TOOLCHAIN_DIR)

Bernhard> ack.

OK.

What about $(BUILD_DIR)? I guess we would want to delete that as well?

Bernhard> It would be nice if you could still pick it up and also sign/upload the .bz2

I do already:

ls *.bz2*

buildroot-2009.02-rc1.tar.bz2       buildroot-2009.05-rc3.tar.bz2
buildroot-2009.02-rc1.tar.bz2.sign  buildroot-2009.05-rc3.tar.bz2.sign
buildroot-2009.02-rc2.tar.bz2       buildroot-2009.05.tar.bz2
buildroot-2009.02-rc2.tar.bz2.sign  buildroot-2009.05.tar.bz2.sign
buildroot-2009.02-rc3.tar.bz2       buildroot-2009.08-rc1.tar.bz2
buildroot-2009.02-rc3.tar.bz2.sign  buildroot-2009.08-rc1.tar.bz2.sign
buildroot-2009.02-rc4.tar.bz2       buildroot-2009.08-rc2.tar.bz2
buildroot-2009.02-rc4.tar.bz2.sign  buildroot-2009.08-rc2.tar.bz2.sign
buildroot-2009.02.tar.bz2           buildroot-2009.08-rc3.tar.bz2
buildroot-2009.02.tar.bz2.sign      buildroot-2009.08-rc3.tar.bz2.sign
buildroot-2009.05-rc1.tar.bz2       buildroot-2009.08.tar.bz2
buildroot-2009.05-rc1.tar.bz2.sign  buildroot-2009.08.tar.bz2.sign
buildroot-2009.05-rc2.tar.bz2       buildroot-snapshot.tar.bz2

buildroot-2009.05-rc2.tar.bz2.sign

But the script generates it all from a single .tar.gz - Anyway, that's a minor detail.

Bye, Peter Korsgaard

Le Wed, 7 Oct 2009 20:31:57 +0200,
Bernhard Reutner-Fischer <[email protected]> a ?crit :

Agreed. The <pkg>-clean targets of the packages are fragile anyway.

and BUILD_DIR.

Thomas

Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

Bernhard> Hi,
Bernhard> Currently the clean target (as well as dirclean) does nothing useful.

Bernhard> I propose the following change:

Bernhard> # wipe target
Bernhard> clean:
Bernhard> # remove generated files, retain configs Bernhard> realclean:
Bernhard> # remove generated files, including configs (but not DL_DIR!)

We btw need to update the 'make help' output to match.

Bye, Peter Korsgaard

Hello,

Good to get a discussion on clean targets. They are currently a mess. Thanks for raising the topic and starting a discussion on this topic.

Le Wed, 7 Oct 2009 18:24:22 +0200,
Bernhard Reutner-Fischer <[email protected]> a ?crit :

I'm not a fan of a make target that only cleans the target. People might think that by doing so, there are able to remove packages from their system. However, previously selected packages will still be present in the staging/ directory, which is not clean. Consider the following scenario :

* We have package A that has an optional dependency on package B. I.e, A does not depend on B, but when A ./configure detects that B is present, then it uses B features, otherwise it just disables this feature flawlessly.

* The user select A and B, and compiles the system. In the staging/ and target/ directories, we have both A and B

* The user removes B, and does "make clean" with the semantic of your proposal. So target/ is empty, but staging/ still contains A and B

* The user runs "make" to regenerate the target/. This will reinstall a version of A that thinks that B will be present on the target, which is not longer the case.

Moreover, just removing the target/ directory doesn't trigger the reinstallation of packages using the Makefile.autotools.in machinery, since this machinery uses stamp files outside of the target/ directory.

Until we have a proven working way of cleanly removing packages, I'd prefer not to let the user think that he can remove packages safely.

Could you specify "remove generated files" ? And the semantic when O= is used ?

Agreed on realclean vs. distclean concerning the removal of .config, and the fact that DL_DIR must be kept.

But still need to define what are the ? generated files ? and how you plan to remove them.

Sincerly,

Thomas

Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

Hello,

Le Fri, 9 Oct 2009 09:51:52 +0200,
Thomas Petazzoni <[email protected]> a ?crit :

Ignore some of my comments, I didn't see the attached patch (it seems it was not inline), making it harder to comment on.

* On the clean target

I see that all the stamps are removed. So this will trigger basically a reconfigure + rebuild of all packages. Is this what we want ?

Same question as before: is removing the target/ directory without removing the staging/ directory really safe ?

   Removing $(IMAGE) will not do anything, since $(IMAGE) is the prefix
   for the filesystem images. I.e, the tar image is $(IMAGE).tar.
   Shouldn't we just remove $(BINARIES_DIR) ?

* Your description mention realclean, but the patch implements dirclean

* Detail: in dirclean (realclean ?), shouldn't we erase $(BASE_DIR) simply, instead of erasing all individual directories it contains ?

* On distclean: removing the BUILD_DIR and TOOLCHAIN_DIR is already done by dirclean, on which distclean depends. If clean removes BINARIES_DIR as suggested above, we can also drop this clean here.

* Documentation (both 'make help' and the HTML documentation)

Again, thanks a lot for working on this issue.

Thomas

Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

yes, this is what i as a user would expect.
Something broke, i have to rebuild stuff, so i
make clean ; make

the IMAGE was there already. Ask Ulf what it was ment to be.

works for me, but they have to be regenerated anyway, so not much point, i'd say.

That was not a patch, it was the start of an explanation of what i wanted to say.

make O=.
pathological case ;)
I don't mind if we just erase BASE_DIR or the directories.

that was already mentioned, yes.

not a patch.

Please incooperate the comments and provide a real patch that we can discuss.

Hi,

Bernhard> yes, this is what i as a user would expect. Bernhard> Something broke, i have to rebuild stuff, so i Bernhard> make clean ; make

So in other words, we should just have clean and distclean, where the only difference between those two is that clean doesn't delete your .config?

Bernhard> the IMAGE was there already. Ask Ulf what it was ment to be.

Ok, but lets really fix up stuff now that we're changing these things.

Bernhard> works for me, but they have to be regenerated anyway, so not Bernhard> much point, i'd say.

They should imho get removed so the user doesn't get confused if the build fails, but they still see (old) binaries in BINARIES_DIR.

But ok, moot point if we make clean remove everything.

Bye, Peter Korsgaard

Peter> Hi,

Bernhard> yes, this is what i as a user would expect. Bernhard> Something broke, i have to rebuild stuff, so i Bernhard> make clean ; make

Peter> So in other words, we should just have clean and distclean, where the Peter> only difference between those two is that clean doesn't delete your Peter> .config?

No objections? Ok, I'll implement it like that then.

Bye, Peter Korsgaard

I'd keep conf,mconf and related .o on clean

Peter> So in other words, we should just have clean and distclean, where the Peter> only difference between those two is that clean doesn't delete your Peter> .config?

Bernhard> I'd keep conf,mconf and related .o on clean

Me too, clean now only deletes everything in output/.

Bye, Peter Korsgaard

.. and of course keep DL_DIR

so, to come back to my initial proposal

# wipe target and staging dirs (think %-uninstall):
clean:
# remove generated files, retain: configs and DL_DIR, {m,}conf{,.o}, host-*:
realclean:
# remove generated files, including configs {m,}conf host-* (but not DL_DIR!):
# think git clean -d -x
distclean:
# make a release tarball; see how existing DL_DIR does not matter:
dist release: FOO=buildroot VERS=1.2.3
	git archive --prefix $(FOO)-$(VERS) HEAD | tee >(bzip2>$(FOO).tar.bz2) >(gzip>$(FOO).tar.gz) | xv>$(FOO).tar.xv

Hi,

Bernhard> .. and of course keep DL_DIR

Bernhard> so, to come back to my initial proposal

Bernhard> # wipe target and staging dirs (think %-uninstall):
Bernhard> clean:
Bernhard> # remove generated files, retain: configs and DL_DIR, {m,}conf{,.o}, host-*:
Bernhard> realclean:
Bernhard> # remove generated files, including configs {m,}conf host-* (but not DL_DIR!):
Bernhard> # think git clean -d -x
Bernhard> distclean:

Check the tree. I believe the commit does that (except we don't have a realclean).

Bernhard> # make a release tarball; see how existing DL_DIR does not matter:
Bernhard> dist release: FOO=buildroot VERS=1.2.3
Bernhard> git archive --prefix $(FOO)-$(VERS) HEAD | tee >(bzip2>$(FOO).tar.bz2) >(gzip>$(FOO).tar.gz) | xv>$(FOO).tar.xv

I prefer to not touch make release right now - Not a high priority as I'm the only one using it.

Bye, Peter Korsgaard