Skip to contentSkip to conversation

xfsprogs: Refactor release scripts to conform to using git archive

PATCH Ben Myers 3 participants 21 messages Feb 1, 2013 – Feb 14, 2013 5 messages not in the archive Kbuild

lore
Share & tools
lore

Ben Myers

author Reviewed-by
#2 lore raw
In reply to a message not in the archive

Andrew,

Quoted 9 lines · Andrew Dahl

Refactored release scripts to conform to using git archive

When generating a release, there is a risk of missing necessary source files. This is fixed by using git archive, which also fixes the lack of conformity between the xfs utilities. As well, some files may be stale during packaging. This is fixed with a clean at the beginning of release generation.

Compared the old 'make dist' with the new one:

# diff old.list new.list
129a130
Quoted 1 line

xfsprogs-3.1.10/.gitignore

The only difference in the file list is the addition of .gitignore.

Quoted 19 lines
---
 Makefile   |   11 ++++++++---
 release.sh |    3 +++
 2 files changed, 11 insertions(+), 3 deletions(-)
Makefile +2 -0
index 0bdc5e8..e73af88 100644
@@ -29,6 +29,8 @@ SRCTAR = $(PKG_NAME)-$(PKG_VERSION).tar.gz
 
 CONFIGURE = aclocal.m4 configure config.guess config.sub install-sh ltmain.sh
 LSRCFILES = configure.ac release.sh README VERSION $(CONFIGURE)
+SRCTARINC = m4/libtool.m4 m4/lt~obsolete.m4 m4/ltoptions.m4 m4/ltsugar.m4 \
+           m4/ltversion.m4 po/xfsprogs.pot $(CONFIGURE)
 
 LDIRT = config.log .ltdep .dep config.status config.cache confdefs.h \
 	conftest* built .census install.* install-dev.* *.gz \

Makefile $(SRCDIR) : $(_FORCE) · line 151

Quoted 9 lines
Makefile +3 -3
@@ -144,7 +146,10 @@ $(SRCDIR) : $(_FORCE)
 	rm -fr $@
 	mkdir -p $@
 
-$(SRCTAR) : default $(SRCDIR)
-	$(Q)$(MAKE) $(MAKEOPTS) source-link
-	unset TAPE; $(TAR) -cf - $(SRCDIR) | $(ZIP) --best > $@ && \
+$(SRCTAR) : default
+	$(Q)git archive --prefix=$(SRCDIR)/ --format=tar v$(PKG_VERSION) \
+	  -o $(SRCDIR).tar

Older versions of git do not support the -o option. I changed this to > $(SRCDIR).tar and it works fine.

Makefile $(SRCDIR) : $(_FORCE) · line 152

Quoted 1 line
Makefile +1 -0
+	$(Q)$(TAR) --transform "s,^,$(SRCDIR)/," -rf $(SRCARC).tar \

SRCDIR

With those changes this looks and works fine.

-Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author Reviewed-by
#4 lore raw

Re: [PATCH 2/4] xfsdump: Refactor release scripts to conform to using git archive

In reply to a message not in the archive

Andrew,

Quoted 10 lines · Andrew Dahl

Refactored release scripts to conform to using git archive

When generating a release, there is a risk of some files being stale, such as configure and the m4/autotools temp files. This is fixed with a clean at the beginning of the release generation. In addition, there is no uniformity in the current method of source tar generation between xfs utilities. Using git archive solves this issue accross all utilities.

Here is the difference between the old 'make dist' and the new one:

# diff old.list new.list
99a100
Quoted 1 line

xfsdump-3.1.2/.gitignore

190d190
< xfsdump-3.1.2/po/xfsdump.pot

Adding .gitignore is fine, and xfsdump.pot needn't be in the archive, it is an output file from gettext.

Quoted 14 lines
Makefile +2 -0
index 39881af..23b6a9e 100644
@@ -29,6 +29,8 @@ SRCTAR = $(PKG_NAME)-$(PKG_VERSION).tar.gz
 
 CONFIGURE = aclocal.m4 configure config.guess config.sub install-sh ltmain.sh
 LSRCFILES = configure.ac release.sh README VERSION $(CONFIGURE)
+SRCTARINC = m4/libtool.m4 m4/lt~obsolete.m4 m4/ltoptions.m4 m4/ltsugar.m4 \
+            m4/ltversion.m4 $(CONFIGURE)
 
 LDIRT = config.log .ltdep .dep config.status config.cache confdefs.h \
 	conftest* built .census install.* install-dev.* *.gz \

Makefile $(SRCDIR) : $(_FORCE) · line 131

Quoted 9 lines
Makefile +3 -3
@@ -124,7 +126,10 @@ $(SRCDIR) : $(_FORCE)
 	rm -fr $@
 	mkdir -p $@
 
-$(SRCTAR) : default $(SRCDIR)
-	$(Q)$(MAKE) $(MAKEOPTS) source-link
-	unset TAPE; $(TAR) -cf - $(SRCDIR) | $(ZIP) --best > $@ && \
+$(SRCTAR) : default
+	$(Q)git archive --prefix=$(SRCDIR)/ --format=tar v$(PKG_VERSION) \
+           -o $(SRCDIR).tar

Changed this to

Quoted 1 line

$(SRCDIR).tar

Regards,
Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author Reviewed-by
#6 lore raw

Re: [PATCH 3/4] xfstests: Refactor release scripts to conform to using git archive

In reply to a message not in the archive

Andrew,

Quoted 43 lines · Andrew Dahl

Refactored release scripts to conform to using git archive

When generating a release, there is a risk of some files being stale, such as configure and the m4/autotools temp files. This is fixed with a clean at the beginning of release generation. In addition, there is no uniformity in the current method of source tar generation between xfs utilities. Using git archive solves this issue across all utilities.

---
 Makefile   |   20 ++++++++++++++++++++
 release.sh |   11 +++++++++--
 2 files changed, 29 insertions(+), 2 deletions(-)
 mode change 100644 => 100755 release.sh
Makefile +6 -0
index b68ec11..a0aef85 100644
@@ -39,12 +39,17 @@ ifeq ($(HAVE_BUILDDEFS), yes)
 include $(TOPDIR)/include/builddefs
 endif
 
+SRCTAR = $(PKG_NAME)-$(PKG_VERSION).tar.gz
+
 TESTS = $(shell sed -n -e '/^[0-9][0-9][0-9]*/s/ .*//p' group)
 CONFIGURE = configure include/builddefs include/config.h
 LSRCFILES = configure configure.ac aclocal.m4 README VERSION
 LDIRT = config.log .ltdep .dep config.status config.cache confdefs.h \
 	conftest* check.log check.time
 
+ifeq ($(HAVE_BUILDDEFS), yes)
+LDIRT += $(SRCTAR)
+endif
 
 LIB_SUBDIRS = include lib
 TOOL_SUBDIRS = ltp src m4
@@ -102,3 +107,18 @@ install-dev install-lib:
 realclean distclean: clean
 	$(Q)rm -f $(LDIRT) $(CONFIGURE)
 	$(Q)rm -rf autom4te.cache Logs
+

Makefile install-dev install-lib: · line 120

Quoted 10 lines
Makefile +10 -0
+dist: include/builddefs include/config.h default
+ifeq ($(HAVE_BUILDDEFS), no)
+	$(Q)$(MAKE) $(MAKEOPTS) -C . $@
+else
+	$(Q)$(MAKE) $(MAKEOPTS) $(SRCTAR)
+endif
+
+$(SRCTAR) : default
+	$(Q)git archive --prefix=$(PKG_NAME)-$(PKG_VERSION)/ --format=tar \
+	  v$(PKG_VERSION) -o $(PKG_NAME)-$(PKG_VERSION).tar

Again, changed this to

Quoted 1 line

$(PKG_NAME)-$(PKG_VERSION).tar

And it worked fine.

Regards,
Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author Reviewed-by
#8 lore raw

Re: [PATCH 4/4] dmapi: Refactor release scripts to conform to using git archive

In reply to a message not in the archive

Andrew,

Quoted 36 lines · Andrew Dahl

Refactored release scripts to conform to using git archive

When generating a release, there is a risk of some files being stale, such as configure and the m4/autotools temp files. This is fixed with a clean at the beginning of release generation. In addition, there is no uniformity in the current method of source tar generation between xfs utilities. Using git archive solves this issue across all utilities.

---
 Makefile       |   19 +++++++++++++++++++
 Makepkgs       |    4 ++--
 build/Makefile |   22 +---------------------
 release.sh     |    5 ++++-
 4 files changed, 26 insertions(+), 24 deletions(-)
Makefile +4 -0
index 0bddb07..58e5caf 100644
@@ -9,6 +9,8 @@ ifeq ($(HAVE_BUILDDEFS), yes)
 include $(TOPDIR)/include/builddefs
 endif
 
+SRCTAR=$(PKG_NAME)-$(PKG_VERSION).tar.gz
+
 CONFIGURE = aclocal.m4 configure config.guess config.sub configure install-sh \
 	    ltmain.sh m4/libtool.m4 m4/ltoptions.m4 m4/ltsugar.m4 \
 	    m4/ltversion.m4 m4/lt~obsolete.m4
@@ -84,3 +86,19 @@ distclean: clean
 
 realclean: distclean
 	rm -f $(CONFIGURE)
+
+dist: default

Makefile distclean: clean · line 100

Quoted 10 lines
Makefile +10 -0
+ifeq ($(HAVE_BUILDDEFS), no)
+	$(MAKE) $(MAKEOPTS) -C . $@
+else
+	$(MAKE) $(MAKEOPTS) $(SRCTAR)
+	$(MAKE) $(MAKEOPTS) -C build dist
+endif
+
+$(SRCTAR) : default
+	git archive --prefix=$(PKG_NAME)-$(PKG_VERSION)/ --format=tar \
+	  v$(PKG_VERSION) -o $(PKG_NAME)-$(PKG_VERSION).tar
		   	  > 

Other than that it looks fine.

# diff old.list new.list 
0a1
Quoted 1 line

dmapi-2.2.12/

1a3

Quoted 1 line

dmapi-2.2.12/build/

2a5

Quoted 1 line

dmapi-2.2.12/build/rpm/

6a10

Quoted 1 line

dmapi-2.2.12/build/tar/

12a17

Quoted 1 line

dmapi-2.2.12/debian/

17a23

Quoted 1 line

dmapi-2.2.12/doc/

22a29,30

Quoted 2 lines

dmapi-2.2.12/.gitignore
dmapi-2.2.12/include/

30a39

Quoted 1 line

dmapi-2.2.12/libdm/

49a59

Quoted 1 line

dmapi-2.2.12/m4/

62a73

Quoted 1 line

dmapi-2.2.12/man/

63a75

Quoted 1 line

dmapi-2.2.12/man/man3/

The old source tarball didn't have some directories in it. That's ok.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#9 lore raw

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

In reply to a message not in the archive
Quoted 5 lines · Andrew Dahl

The following four patches fix an issue seen during the last user space release where not all source files were included in xfsprogs. They each change xfsprogs, xfstests, xfsdump, and dmapi to all use git archive for source tarball generation and add 'make realclean' in the release scripts to ensure the configure and m4/autotools temp files are refreshed.

Applied this series.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#10 lore raw

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

Quoted 8 lines · Ben Myers
Quoted 5 lines · Andrew Dahl

The following four patches fix an issue seen during the last user space release where not all source files were included in xfsprogs. They each change xfsprogs, xfstests, xfsdump, and dmapi to all use git archive for source tarball generation and add 'make realclean' in the release scripts to ensure the configure and m4/autotools temp files are refreshed.

Applied this series.

It breaks the 'make deb' command for all the trees.

The debian package build creates a link tree so that it builds outside the main source tree (e.g. in ./xfsprogs-3.1.10) and as part of the build it runs 'make dist' to build a tarball from that link tree. That now fails with:

fatal: current working directory is untracked

coming from the git archive command that is now used to build the tarball.

Nathan, can the debian package build use git-archive? I'm under the impression that you have to be able to build debian packages from a source tarball or a source package, and so the build can't rely on being inside a VCS controlled tree. That rules out using git archive for tarball creation...

Also, given that the previous tarball creation used the same link tree build as the debian package build (i.e. make source-link), that implies the debian pacakges are going to be missing files using this technique. Doesn't that mean we have to fix the file list being fed into the source-link target regardless of whether we use git archive to build the tarball?

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#11 lore raw

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

Quoted 3 lines · Dave Chinner <david <at> fromorbit.com> writes:

It breaks the 'make deb' command for all the trees.

(thanks for checking!)

Quoted 7 lines

The debian package build creates a link tree so that it builds outside the main source tree (e.g. in ./xfsprogs-3.1.10) and as part of the build it runs 'make dist' to build a tarball from that link tree. That now fails with:

fatal: current working directory is untracked

Quoted 9 lines

coming from the git archive command that is now used to build the tarball.

Nathan, can the debian package build use git-archive? I'm under the impression that you have to be able to build debian packages from a source tarball or a source package, and so the build can't rely on being inside a VCS controlled tree. That rules out using git archive for tarball creation...

IIRC, the requirement is to be building in a tree that had a basename of for <package>-<version> - hence the link'd tree build previously. If that can be maintained somehow, the build should be fine. Symlinks did not work very well (IIRC) - the build tools might've used basename or realpath or something like that, but I do remember trying that.

Quoted 6 lines

Also, given that the previous tarball creation used the same link tree build as the debian package build (i.e. make source-link), that implies the debian pacakges are going to be missing files using this technique. Doesn't that mean we have to fix the file list being fed into the source-link target regardless of whether we use git archive to build the tarball?

The source tarball definitely needs to have the package-version prefix for all the files it contains - has a mechanism to acheive that been found?

cheers.

--
Nathan

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#12 lore raw

Re: [PATCH 0/4] Refactor release scripts to conform to using git?archive

Quoted 38 lines · Nathan Scott
Quoted 3 lines · Dave Chinner <david <at> fromorbit.com> writes:

It breaks the 'make deb' command for all the trees.

(thanks for checking!)

Quoted 7 lines

The debian package build creates a link tree so that it builds outside the main source tree (e.g. in ./xfsprogs-3.1.10) and as part of the build it runs 'make dist' to build a tarball from that link tree. That now fails with:

fatal: current working directory is untracked

Quoted 9 lines

coming from the git archive command that is now used to build the tarball.

Nathan, can the debian package build use git-archive? I'm under the impression that you have to be able to build debian packages from a source tarball or a source package, and so the build can't rely on being inside a VCS controlled tree. That rules out using git archive for tarball creation...

IIRC, the requirement is to be building in a tree that had a basename of for <package>-<version> - hence the link'd tree build previously. If that can be maintained somehow, the build should be fine. Symlinks did not work very well (IIRC) - the build tools might've used basename or realpath or something like that, but I do remember trying that.

Quoted 6 lines

Also, given that the previous tarball creation used the same link tree build as the debian package build (i.e. make source-link), that implies the debian pacakges are going to be missing files using this technique. Doesn't that mean we have to fix the file list being fed into the source-link target regardless of whether we use git archive to build the tarball?

Quoted 3 lines

The source tarball definitely needs to have the package-version prefix for all the files it contains - has a mechanism to acheive that been found?

It has... with a combo of git archive --prefix and tar --transform. I'll see what I can do to get 'make deb' going again.

Thanks,
Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#13 lore raw

Re: [PATCH 0/4] Refactor release scripts to conform to using git?archive

In reply to Nathan Scott
Quoted 7 lines · Nathan Scott
Quoted 3 lines · Dave Chinner <david <at> fromorbit.com> writes:

It breaks the 'make deb' command for all the trees.

(thanks for checking!)

I didn't - my CI system caught it immediately once I switched everything back on after being away for a week at LCA. It builds debs and ships them to VMs, installs them and then runs various tests after every upstream repository check in that occurs.

It was when I checked the RSS feeds coming back from the CI system that I noticed that it was busted and started looking into the problem....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#14 lore raw

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

In reply to Dave Chinner
Quoted 11 lines · Dave Chinner
Quoted 8 lines · Ben Myers
Quoted 5 lines · Andrew Dahl

The following four patches fix an issue seen during the last user space release where not all source files were included in xfsprogs. They each change xfsprogs, xfstests, xfsdump, and dmapi to all use git archive for source tarball generation and add 'make realclean' in the release scripts to ensure the configure and m4/autotools temp files are refreshed.

Applied this series.

It breaks the 'make deb' command for all the trees.

xfsprogs: update 'make deb' to use tarball

This patch changes the build process so that 'make deb' uses the same process of creating a source tree as the release script.

* Add a list of files which go in the release tarball in .git_census This is needed so that you can create a tarball in a bare release tree, when .git is not available.

* Modify the SRCTAR target to include files from .git_census and use tar instead of git archive.

* Modify the SRCTARINC files to include .git_census, and include .git_census in the 'make realclean' target.

* 'make deb' now creates unsigned packages by default, 'make debsign' creates signed packages.

* remove the 'make source-link' target.

---
 Makefile           |   29 ++++++++++++++++-------------
 include/buildrules |   11 -----------
 2 files changed, 16 insertions(+), 24 deletions(-)
Makefile +16 -13
Index: b/Makefile
===================================================================
@@ -30,8 +30,7 @@ SRCTAR = $(PKG_NAME)-$(PKG_VERSION).tar.
 CONFIGURE = aclocal.m4 configure config.guess config.sub install-sh ltmain.sh
 LSRCFILES = configure.ac release.sh README VERSION $(CONFIGURE)
 SRCTARINC = m4/libtool.m4 m4/lt~obsolete.m4 m4/ltoptions.m4 m4/ltsugar.m4 \
-           m4/ltversion.m4 po/xfsprogs.pot $(CONFIGURE)
-
+           m4/ltversion.m4 po/xfsprogs.pot .git_census $(CONFIGURE)
 LDIRT = config.log .ltdep .dep config.status config.cache confdefs.h \
 	conftest* built .census install.* install-dev.* *.gz \
 	autom4te.cache/* libtool include/builddefs include/platform_defs.h
@@ -120,7 +119,7 @@ distclean: clean
 	$(Q)rm -f $(LDIRT)
 
 realclean: distclean
-	$(Q)rm -f $(CONFIGURE)
+	$(Q)rm -f $(CONFIGURE) .git_census
 
 #
 # All this gunk is to allow for a make dist on an unconfigured tree
@@ -137,18 +136,22 @@ ifeq ($(HAVE_BUILDDEFS), no)
 	$(Q)$(MAKE) $(MAKEOPTS) -C . $@
 else
 	$(Q)$(MAKE) $(MAKEOPTS) $(SRCDIR)
-	$(Q)$(MAKE) $(MAKEOPTS) -C po
-	$(Q)$(MAKE) $(MAKEOPTS) source-link
-	$(Q)cd $(SRCDIR) && dpkg-buildpackage
+	$(Q)cd $(SRCDIR) && dpkg-buildpackage -us -uc
 endif
 
-$(SRCDIR) : $(_FORCE)
+debsign: deb
+	debsign
+
+$(SRCDIR) : $(_FORCE) $(SRCTAR)
 	rm -fr $@
-	mkdir -p $@
+	$(Q)$(TAR) -zxvf $(SRCTAR)
 
-$(SRCTAR) : default
-	$(Q)git archive --prefix=$(SRCDIR)/ --format=tar v$(PKG_VERSION) > $(SRCDIR).tar 
-	$(Q)$(TAR) --transform "s,^,$(SRCDIR)/," -rf $(SRCDIR).tar \
-	   $(SRCTARINC) 
-	$(Q)$(ZIP) $(SRCDIR).tar
+$(SRCTAR) : default $(SRCTARINC) .git_census
+	$(Q)$(TAR) --transform "s,^,$(SRCDIR)/," -zcf $(SRCDIR).tar.gz  \
+	   `cat .git_census` $(SRCTARINC)
 	echo Wrote: $@
+
+.git_census: $(_FORCE)
+	$(Q)if test -d .git; then \
+	  git ls-files > .git_census && echo "new .git_census"; \
+	fi
include/buildrules +0 -11
Index: b/include/buildrules
===================================================================
@@ -23,17 +23,6 @@ $(SUBDIRS):
 	$(Q)$(MAKE) $(MAKEOPTS) -q -C $@ || $(MAKE) $(MAKEOPTS) -C $@
 endif
 
-source-link: 
-	@test -z "$$DIR" && DIR="."; \
-	for f in `echo $(SRCFILES) $(SUBDIRS) $(POTHEAD)`; do \
-	    if test -d $$f ; then \
-		mkdir $(TOPDIR)/$(PKG_NAME)-$(PKG_VERSION)/$$DIR/$$f || exit $$?; \
-		$(MAKEF) DIR=$$DIR/$$f -C $$f $@ || exit $$?; \
-	    else \
-		ln $$f $(TOPDIR)/$(PKG_NAME)-$(PKG_VERSION)/$$DIR/$$f || exit $$?; \
-	    fi; \
-	done
-
 #
 # Standard targets
 #

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#15 lore raw

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

Hi Ben,

----- Original Message -----

Quoted 13 lines · Dave Chinner
Quoted 2 lines

...
It breaks the 'make deb' command for all the trees.

xfsprogs: update 'make deb' to use tarball

This patch changes the build process so that 'make deb' uses the same process of creating a source tree as the release script.

* Add a list of files which go in the release tarball in .git_census This is needed so that you can create a tarball in a bare release tree, when .git is not available.

I think you'll need .git_census in .gitignore (ala configure). Possibly remove the underscore for naming consistency - *shrug*.

Quoted 4 lines

* 'make deb' now creates unsigned packages by default, 'make debsign' creates signed packages.

Ehrm - why? Everything else in your patch worked, but this part broke (which suggests a larger problem, in that this build path is no longer checked on every build) ...

debsign: Can't find or can't read changes file !

Makefile ifeq ($(HAVE_BUILDDEFS), no) · line 144

Quoted 3 lines
Makefile +3 -0
+debsign: deb
+	debsign
+

(That's not valid usage, FWIW).

I would recommend just removing that change in behaviour, "make deb" was fine as it was (for me anyway, and evidently for Dave too). Best to go secure-by-default and not change this.

thanks!

--
Nathan

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#16 lore raw

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

Hey Nathan,

Quoted 10 lines · Nathan Scott

----- Original Message -----

Quoted 9 lines · Dave Chinner
Quoted 2 lines

...
It breaks the 'make deb' command for all the trees.

xfsprogs: update 'make deb' to use tarball

This patch changes the build process so that 'make deb' uses the same process of creating a source tree as the release script.

Quoted 7 lines
Quoted 4 lines

* Add a list of files which go in the release tarball in .git_census This is needed so that you can create a tarball in a bare release tree, when .git is not available.

I think you'll need .git_census in .gitignore (ala configure). Possibly remove the underscore for naming consistency - *shrug*.

Will do. Thanks.

Quoted 18 lines
Quoted 5 lines

* 'make deb' now creates unsigned packages by default, 'make debsign' creates signed packages.

Ehrm - why? Everything else in your patch worked, but this part broke (which suggests a larger problem, in that this build path is no longer checked on every build) ...

debsign: Can't find or can't read changes file !

Makefile ifeq ($(HAVE_BUILDDEFS), no) · line 144

Quoted 3 lines
Makefile +3 -0
+debsign: deb
+	debsign
+

(That's not valid usage, FWIW).

Quoted 3 lines

I would recommend just removing that change in behaviour, "make deb" was fine as it was (for me anyway, and evidently for Dave too). Best to go secure-by-default and not change this.

'make deb' failed for me like this:

signfile xfsprogs_3.1.9.dsc
gpg: skipped "Nathan Scott <[email protected]>": secret key not available
gpg: [stdin]: clearsign failed: secret key not available

dpkg-genchanges >../xfsprogs_3.1.9_amd64.changes
dpkg-genchanges: including full source code in upload
dpkg-buildpackage: full upload; Debian-native package (full source is included)
dpkg-buildpackage: warning: Failed to sign .dsc and .changes file
make: *** [deb] Error 1

Secure-by-default is a great idea but it seems that in this case it also requires that anyone who wants to do a build for debian has to do it on a machine with a keyring containing your secret key. ;)

I think it is also reasonable for devs to be able to build test debs on machines that have empty keyrings. An unsigned 'make deb' does return 0. I'd like this to suit everyone's needs if it can. I'll split this part out into a separate patch and try to figure something out later.

Thanks,
Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#17 lore raw

xfsprogs: update 'make deb' to use tarball

This patch changes the build process so that 'make deb' uses the same process of creating a source tree as the release script.

* Add a list of files which go in the release tarball in .gitcensus This is needed so that you can create a tarball in a bare release tree, when .git is not available.

* Modify the SRCTAR target to include files from .git_census and use tar instead of git archive.

* Modify the SRCTARINC files to include .git_census, and include .git_census in the 'make realclean' target.

* remove the 'make source-link' target.

---

v2: don't muck with signing the deb in this patch, rename .git_census to .gitcensus, add .gitcensus to .gitignore.

 .gitignore         |    1 +
 Makefile           |   24 ++++++++++++------------
 include/buildrules |   11 -----------
 3 files changed, 13 insertions(+), 23 deletions(-)
.gitignore +1 -0
Index: b/.gitignore
===================================================================
@@ -5,6 +5,7 @@
 
 # build system
 .census
+.gitcensus
 /include/platform_defs.h
 /include/builddefs
 /install-sh
Makefile +12 -12
Index: b/Makefile
===================================================================
@@ -30,8 +30,7 @@ SRCTAR = $(PKG_NAME)-$(PKG_VERSION).tar.
 CONFIGURE = aclocal.m4 configure config.guess config.sub install-sh ltmain.sh
 LSRCFILES = configure.ac release.sh README VERSION $(CONFIGURE)
 SRCTARINC = m4/libtool.m4 m4/lt~obsolete.m4 m4/ltoptions.m4 m4/ltsugar.m4 \
-           m4/ltversion.m4 po/xfsprogs.pot $(CONFIGURE)
-
+           m4/ltversion.m4 po/xfsprogs.pot .gitcensus $(CONFIGURE)
 LDIRT = config.log .ltdep .dep config.status config.cache confdefs.h \
 	conftest* built .census install.* install-dev.* *.gz \
 	autom4te.cache/* libtool include/builddefs include/platform_defs.h
@@ -120,7 +119,7 @@ distclean: clean
 	$(Q)rm -f $(LDIRT)
 
 realclean: distclean
-	$(Q)rm -f $(CONFIGURE)
+	$(Q)rm -f $(CONFIGURE) .gitcensus
 
 #
 # All this gunk is to allow for a make dist on an unconfigured tree
@@ -137,18 +136,19 @@ ifeq ($(HAVE_BUILDDEFS), no)
 	$(Q)$(MAKE) $(MAKEOPTS) -C . $@
 else
 	$(Q)$(MAKE) $(MAKEOPTS) $(SRCDIR)
-	$(Q)$(MAKE) $(MAKEOPTS) -C po
-	$(Q)$(MAKE) $(MAKEOPTS) source-link
 	$(Q)cd $(SRCDIR) && dpkg-buildpackage
 endif
 
-$(SRCDIR) : $(_FORCE)
+$(SRCDIR) : $(_FORCE) $(SRCTAR)
 	rm -fr $@
-	mkdir -p $@
+	$(Q)$(TAR) -zxvf $(SRCTAR)
 
-$(SRCTAR) : default
-	$(Q)git archive --prefix=$(SRCDIR)/ --format=tar v$(PKG_VERSION) > $(SRCDIR).tar 
-	$(Q)$(TAR) --transform "s,^,$(SRCDIR)/," -rf $(SRCDIR).tar \
-	   $(SRCTARINC) 
-	$(Q)$(ZIP) $(SRCDIR).tar
+$(SRCTAR) : default $(SRCTARINC) .gitcensus
+	$(Q)$(TAR) --transform "s,^,$(SRCDIR)/," -zcf $(SRCDIR).tar.gz  \
+	   `cat .gitcensus` $(SRCTARINC)
 	echo Wrote: $@
+
+.gitcensus: $(_FORCE)
+	$(Q)if test -d .git; then \
+	  git ls-files > .gitcensus && echo "new .gitcensus"; \
+	fi
include/buildrules +0 -11
Index: b/include/buildrules
===================================================================
@@ -23,17 +23,6 @@ $(SUBDIRS):
 	$(Q)$(MAKE) $(MAKEOPTS) -q -C $@ || $(MAKE) $(MAKEOPTS) -C $@
 endif
 
-source-link: 
-	@test -z "$$DIR" && DIR="."; \
-	for f in `echo $(SRCFILES) $(SUBDIRS) $(POTHEAD)`; do \
-	    if test -d $$f ; then \
-		mkdir $(TOPDIR)/$(PKG_NAME)-$(PKG_VERSION)/$$DIR/$$f || exit $$?; \
-		$(MAKEF) DIR=$$DIR/$$f -C $$f $@ || exit $$?; \
-	    else \
-		ln $$f $(TOPDIR)/$(PKG_NAME)-$(PKG_VERSION)/$$DIR/$$f || exit $$?; \
-	    fi; \
-	done
-
 #
 # Standard targets
 #

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#20 lore raw

xfsdump: update 'make deb' to use tarball archive

In reply to Ben Myers

This patch changes the build process so that 'make deb' uses the same process of creating a source tree as the release script.

* Add a list of files which go in the release tarball in .gitcensus This is needed so that you can create a tarball in a bare release tree, when .git is not available.

* Modify the SRCTAR target to include files from .gitcensus and use tar instead of git archive.

* Modify the SRCTARINC files to include .gitcensus, and include .gitcensus in the 'make realclean' target.

* remove the 'make source-link' target.

---
 .gitignore         |    1 +
 Makefile           |   23 ++++++++++++-----------
 include/buildrules |   11 -----------
 3 files changed, 13 insertions(+), 22 deletions(-)
.gitignore +1 -0
Index: b/.gitignore
===================================================================
@@ -4,6 +4,7 @@
 
 # build system
 .census
+.gitcensus
 /include/config.h
 /include/builddefs
 /install-sh
Makefile +12 -11
Index: b/Makefile
===================================================================
@@ -30,7 +30,7 @@ SRCTAR = $(PKG_NAME)-$(PKG_VERSION).tar.
 CONFIGURE = aclocal.m4 configure config.guess config.sub install-sh ltmain.sh
 LSRCFILES = configure.ac release.sh README VERSION $(CONFIGURE)
 SRCTARINC = m4/libtool.m4 m4/lt~obsolete.m4 m4/ltoptions.m4 m4/ltsugar.m4 \
-            m4/ltversion.m4 $(CONFIGURE)
+            m4/ltversion.m4 .gitcensus $(CONFIGURE)
 
 LDIRT = config.log .ltdep .dep config.status config.cache confdefs.h \
 	conftest* built .census install.* install-dev.* *.gz \
@@ -100,7 +100,7 @@ distclean: clean
 	rm -f $(LDIRT)
 
 realclean: distclean
-	rm -f $(CONFIGURE)
+	rm -f $(CONFIGURE) .gitcensus
 
 #
 # All this gunk is to allow for a make dist on an unconfigured tree
@@ -117,18 +117,19 @@ ifeq ($(HAVE_BUILDDEFS), no)
 	$(MAKE) $(MAKEOPTS) -C . $@
 else
 	$(Q)$(MAKE) $(MAKEOPTS) $(SRCDIR)
-	$(Q)$(MAKE) $(MAKEOPTS) -C po
-	$(Q)$(MAKE) $(MAKEOPTS) source-link
 	$(Q)cd $(SRCDIR) && dpkg-buildpackage
 endif
 
-$(SRCDIR) : $(_FORCE)
+$(SRCDIR) : $(_FORCE) $(SRCTAR)
 	rm -fr $@
-	mkdir -p $@
+	$(Q)$(TAR) -zxvf $(SRCTAR)
 
-$(SRCTAR) : default
-	$(Q)git archive --prefix=$(SRCDIR)/ --format=tar v$(PKG_VERSION) > $(SRCDIR).tar
-	$(Q)$(TAR) --transform "s,^,$(SRCDIR)/," -rf $(SRCDIR).tar \
-	  $(SRCTARINC)
-	$(Q)$(ZIP) $(SRCDIR).tar
+$(SRCTAR) : default $(SRCTARINC) .gitcensus
+	$(Q)$(TAR) --transform "s,^,$(SRCDIR)/," -zcf $(SRCDIR).tar.gz \
+	  `cat .gitcensus` $(SRCTARINC)
 	echo Wrote: $@
+
+.gitcensus: $(_FORCE)
+	$(Q) if test -d .git; then \
+	  git ls-files > .gitcensus && echo "new .gitcensus"; \
+	fi
include/buildrules +0 -11
Index: b/include/buildrules
===================================================================
@@ -23,17 +23,6 @@ $(SUBDIRS):
 	$(Q)$(MAKE) $(MAKEOPTS) -q -C $@ || $(MAKE) $(MAKEOPTS) -C $@
 endif
 
-source-link: 
-	@test -z "$$DIR" && DIR="."; \
-	for f in `echo $(SRCFILES) $(SUBDIRS) $(POTHEAD)`; do \
-	    if test -d $$f ; then \
-		mkdir $(TOPDIR)/$(PKG_NAME)-$(PKG_VERSION)/$$DIR/$$f || exit $$?; \
-		$(MAKEF) DIR=$$DIR/$$f -C $$f $@ || exit $$?; \
-	    else \
-		ln $$f $(TOPDIR)/$(PKG_NAME)-$(PKG_VERSION)/$$DIR/$$f || exit $$?; \
-	    fi; \
-	done
-
 #
 # Standard targets
 #

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#23 lore raw

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

In reply to Ben Myers

----- Original Message -----

Quoted 15 lines

...
'make deb' failed for me like this:

signfile xfsprogs_3.1.9.dsc
gpg: skipped "Nathan Scott <[email protected]>": secret key not available
gpg: [stdin]: clearsign failed: secret key not available

dpkg-genchanges >../xfsprogs_3.1.9_amd64.changes
dpkg-genchanges: including full source code in upload
dpkg-buildpackage: full upload; Debian-native package (full source is included)
dpkg-buildpackage: warning: Failed to sign .dsc and .changes file
make: *** [deb] Error 1

Quoted 6 lines

Secure-by-default is a great idea but it seems that in this case it also
requires that anyone who wants to do a build for debian has to do it on a
machine with a keyring containing your secret key. ;)

Awkward. :)

Quoted 1 line

I think it is also reasonable for devs to be able to build test debs

*nod* ... not sure what trickery Dave had used. But, perhaps a better way to tackle this would be similar to the way the build allows extra configure options to be passed in.

include/builddefs: configure
./configure $$LOCAL_CONFIGURE_OPTIONS

so, perhaps something like this...

# LOCAL_BUILDPACKAGE_OPTIONS="-us -uc" for unsigned debs
deb: include/builddefs include/platform_defs.h
...

        $(Q)cd $(SRCDIR) && dpkg-buildpackage $$LOCAL_BUILDPACKAGE_OPTIONS

cheers.

--
Nathan

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Reviewed-by
#18 lore raw

Re: xfsprogs: update 'make deb' to use tarball

In reply to Ben Myers

----- Original Message -----

Quoted 11 lines

This patch changes the build process so that 'make deb' uses the same process of creating a source tree as the release script.

* Add a list of files which go in the release tarball in .gitcensus This is needed so that you can create a tarball in a bare release tree, when .git is not available.

* Modify the SRCTAR target to include files from .git_census and use tar
instead of git archive.

Quoted 7 lines

* Modify the SRCTARINC files to include .git_census, and include .git_census in the 'make realclean' target.

* remove the 'make source-link' target.

thanks!

--
Nathan

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Reviewed-by
#21 lore raw

Re: xfsdump: update 'make deb' to use tarball archive

In reply to Ben Myers

----- Original Message -----

Quoted 11 lines

This patch changes the build process so that 'make deb' uses the same process of creating a source tree as the release script.

* Add a list of files which go in the release tarball in .gitcensus This is needed so that you can create a tarball in a bare release tree, when .git is not available.

* Modify the SRCTAR target to include files from .gitcensus and use tar
instead of git archive.

Quoted 7 lines

* Modify the SRCTARINC files to include .gitcensus, and include .gitcensus in the 'make realclean' target.

* remove the 'make source-link' target.

thanks!

--
Nathan

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#25 lore raw

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

In reply to Ben Myers
Quoted 51 lines · Ben Myers

Hey Nathan,

Quoted 10 lines · Nathan Scott

----- Original Message -----

Quoted 9 lines · Dave Chinner
Quoted 2 lines

...
It breaks the 'make deb' command for all the trees.

xfsprogs: update 'make deb' to use tarball

This patch changes the build process so that 'make deb' uses the same process of creating a source tree as the release script.

Quoted 7 lines
Quoted 4 lines

* Add a list of files which go in the release tarball in .git_census This is needed so that you can create a tarball in a bare release tree, when .git is not available.

I think you'll need .git_census in .gitignore (ala configure). Possibly remove the underscore for naming consistency - *shrug*.

Will do. Thanks.

Quoted 18 lines
Quoted 5 lines

* 'make deb' now creates unsigned packages by default, 'make debsign' creates signed packages.

Ehrm - why? Everything else in your patch worked, but this part broke (which suggests a larger problem, in that this build path is no longer checked on every build) ...

debsign: Can't find or can't read changes file !

Makefile ifeq ($(HAVE_BUILDDEFS), no) · line 144

Quoted 3 lines
Makefile +3 -0
+debsign: deb
+	debsign
+

(That's not valid usage, FWIW).

Quoted 3 lines

I would recommend just removing that change in behaviour, "make deb" was fine as it was (for me anyway, and evidently for Dave too). Best to go secure-by-default and not change this.

'make deb' failed for me like this:

signfile xfsprogs_3.1.9.dsc
gpg: skipped "Nathan Scott <[email protected]>": secret key not available
gpg: [stdin]: clearsign failed: secret key not available

Quoted 5 lines

dpkg-genchanges >../xfsprogs_3.1.9_amd64.changes
dpkg-genchanges: including full source code in upload
dpkg-buildpackage: full upload; Debian-native package (full source is included)
dpkg-buildpackage: warning: Failed to sign .dsc and .changes file
make: *** [deb] Error 1

Don't know what you are doing differently to me, but make deb is not even trying to build signed packages on my build machine. It never has, and I don't ever recall seeing make deb fail because it was trying to build signed packages in the last 5 years on any of my machines...

I'm not passing -uc -us to dpkg-buildpackage, so somewhere aling the line it is working out that it shouldn't be signing the packages itself...

Quoted 2 lines

I think it is also reasonable for devs to be able to build test debs on machines that have empty keyrings.

That's exactly what I have been doing for a long time.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#24 lore raw

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

In reply to Nathan Scott
Quoted 16 lines · Nathan Scott

----- Original Message -----

Quoted 15 lines

...
'make deb' failed for me like this:

signfile xfsprogs_3.1.9.dsc
gpg: skipped "Nathan Scott <[email protected]>": secret key not available
gpg: [stdin]: clearsign failed: secret key not available

dpkg-genchanges >../xfsprogs_3.1.9_amd64.changes
dpkg-genchanges: including full source code in upload
dpkg-buildpackage: full upload; Debian-native package (full source is included)
dpkg-buildpackage: warning: Failed to sign .dsc and .changes file
make: *** [deb] Error 1

Quoted 8 lines
Quoted 6 lines

Secure-by-default is a great idea but it seems that in this case it also
requires that anyone who wants to do a build for debian has to do it on a
machine with a keyring containing your secret key. ;)

Awkward. :)

Really I don't mind. What was your passphrase again?

Quoted 3 lines
Quoted 1 line

I think it is also reasonable for devs to be able to build test debs

*nod* ... not sure what trickery Dave had used.

I tried setting DEB_BUILD_OPTIONS="-us -uc" but no luck.

Quoted 8 lines

But, perhaps a better
way to tackle this would be similar to the way the build allows extra configure options to be passed in.

include/builddefs: configure
./configure $$LOCAL_CONFIGURE_OPTIONS

Quoted 6 lines

so, perhaps something like this...

# LOCAL_BUILDPACKAGE_OPTIONS="-us -uc" for unsigned debs
deb: include/builddefs include/platform_defs.h
...

        $(Q)cd $(SRCDIR) && dpkg-buildpackage $$LOCAL_BUILDPACKAGE_OPTIONS

Gotcha. I'll take a look. Thanks for the reviews!

-Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Reviewed-by
#19 lore raw

Re: xfsprogs: update 'make deb' to use tarball

In reply to Ben Myers
Quoted 10 lines · Ben Myers

This patch changes the build process so that 'make deb' uses the same process of creating a source tree as the release script.

* Add a list of files which go in the release tarball in .gitcensus This is needed so that you can create a tarball in a bare release tree, when .git is not available.

* Modify the SRCTAR target to include files from .git_census and use tar instead of git archive.

Quoted 6 lines

* Modify the SRCTARINC files to include .git_census, and include .git_census in the 'make realclean' target.

* remove the 'make source-link' target.

Works fine here.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Reviewed-by
#22 lore raw

Re: xfsdump: update 'make deb' to use tarball archive

In reply to Ben Myers
Quoted 10 lines · Ben Myers

This patch changes the build process so that 'make deb' uses the same process of creating a source tree as the release script.

* Add a list of files which go in the release tarball in .gitcensus This is needed so that you can create a tarball in a bare release tree, when .git is not available.

* Modify the SRCTAR target to include files from .gitcensus and use tar instead of git archive.

Quoted 6 lines

* Modify the SRCTARINC files to include .gitcensus, and include .gitcensus in the 'make realclean' target.

* remove the 'make source-link' target.

Works fine.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs