infra: Add a way to apply patches using git

PATCH New Jérôme Pouiller 3 participants 6 messages Jul 2, 2014 – Nov 3, 2014 Kbuild

Share & tools

Jérôme Pouiller

author

2014-07-02 17:20 UTC #0 lore raw

PATCH 1/1 infra: Add a way to apply patches using git

This option is mainly used for package developpers. When activated, Buildroot will automaticaly create a git repository for each package it extract. Initial version is tagged as `upstream'. Next, during patch step, git is used instead of patch. BTW, patches are easier to modify/fix/rebase/cherry-pick/upstream. Once satisfied with changes, developper may use
git format-patch -o $TOPDIR/package/$PKG upstream
to save its patches to Buildroot.

Notice this option should be compatible with guilt' (a clone of quilt')

Notice you can also use this feature for a specific package using
make BR2_USE_GIT_TO_PATCH=y $PKG-dirclean $PKG-patch

Current known limitations:
- If a patch cannot be handled by git am', we use git apply`. In this case, patch description is not reported in commit message.
- When patch is handled with git apply, subject used may not be perfect (it is extracted from patch filename)
- `git format-patch' does not prefix patch files with name of package.
- If upstream was downloaded with git, we may also run

    `git remote add $($(PKG)_SITE)' during git repository creation

Signed-off-by J?r?me Pouiller <[email protected]>

---
 Config.in                        | 18 ++++++++++++++++++
 Makefile                         |  4 ++++
 package/pkg-generic.mk           |  7 +++++++
 support/scripts/apply-patches.sh | 29 +++++++++++++++++++++++++++--
 4 files changed, 56 insertions(+), 2 deletions(-)

Config.in +18 -0

index 50968fb..3645fe4 100644
@@ -541,6 +541,24 @@ config BR2_GLOBAL_PATCH_DIR
 	  Otherwise, if the directory <global-patch-dir>/<packagename> exists,
 	  then all *.patch files in the directory will be applied.
 
+config BR2_USE_GIT_TO_PATCH
+	bool "use git repositories to handle patches"
+	help
+	  This option is mainly used for package developpers. When activated,
+	  Buildroot will automaticaly create a git repository for each package
+	  it extract. Initial version is tagged as `upstream'. Next, during
+	  patch step, git is used instead of patch. BTW, patches are easier
+	  to modify/fix/rebase/cherry-pick/upstream. Once satisfied with 
+	  changes, developper may use
+	      git format-patch -o $TOPDIR/package/$PKG upstream
+	  to save its patches to Buildroot.
+
+	  Notice this option should be compatible with `guilt' (a clone of
+	  `quilt')
+
+	  Notice you can also use this feature for a specific package using
+	      make BR2_USE_GIT_TO_PATCH=y $PKG-dirclean $PKG-patch
+
 endmenu
 
 source "toolchain/Config.in"

Makefile +4 -0

index b3bd91e..b35e5a6 100644
@@ -177,6 +177,10 @@ ifeq ($(filter $(noconfig_targets),$(MAKECMDGOALS)),)
 -include $(BR2_CONFIG)
 endif
 
+# apply-patches.sh rely on this varaible to know git should be used instead 
+# of patch
+export BR2_USE_GIT_TO_PATCH
+
 # To put more focus on warnings, be less verbose as default
 # Use 'make V=1' to see the full commands
 ifdef V

package/pkg-generic.mk +7 -0

index ab1f8d2..86368cc 100644
@@ -108,6 +108,13 @@ $(BUILD_DIR)/%/.stamp_extracted:
 # some packages have messed up permissions inside
 	$(Q)chmod -R +rw $(@D)
 	$(foreach hook,$($(PKG)_POST_EXTRACT_HOOKS),$(call $(hook))$(sep))
+ifeq ($(BR2_USE_GIT_TO_PATCH),y)
+	$(Q)cd $(@D) ; \
+	git init ; \
+	git add . > /dev/null ; \
+	git commit -m "$($(PKG)_RAWNAME) $($(PKG)_VERSION)" > /dev/null ; \
+	git tag upstream
+endif
 	$(Q)touch $@
 	@$(call step_end,extract)

support/scripts/apply-patches.sh +27 -2

index 37f2d81..0524b9a 100755
@@ -84,8 +84,28 @@ function apply_patch {
 	exit 1
     fi
     echo $patch >> ${builddir}/.applied_patches_list
-    ${uncomp} "${path}/$patch" | patch -g0 -p1 -E -d "${builddir}" -t -N
-    if [ $? != 0 ] ; then
+    if [[ -n "$BR2_USE_GIT_TO_PATCH" && -e "${builddir}/.git" ]]; then
+        abspatch=$(readlink -f "${path}/$patch")
+        pushd ${builddir} > /dev/null
+        ${uncomp} "$abspatch" | git am --reject > /dev/null
+        RET=$?
+        if [[ $RET != 0 ]]; then
+            ${uncomp} "$abspatch" | git apply --index --reject - > /dev/null
+            RET=$?
+            if [[ $RET == 0 ]]; then
+                # This regexp is far to be perfect
+                pkgname=$(echo ${builddir} | sed 's/^.*\///; s/-[0-9\.]\+$//;')
+                msg="$(echo ${patch} | sed "s/^.*\///; s/^$pkgname-//;  s/^[0-9\.]\+-//; s/\.[a-zA-Z]*\$//; y/-/ /;")"
+                git commit -m "$msg" > /dev/null
+                RET=$?
+            fi
+        fi
+        popd > /dev/null
+    else
+        ${uncomp} "${path}/$patch" | patch -g0 -p1 -E -d "${builddir}" -t -N
+        RET=$?
+    fi
+    if [ $RET != 0 ] ; then
         echo "Patch failed!  Please fix ${patch}!"
 	exit 1
     fi

@@ -121,6 +141,11 @@ function scan_patchdir {
 
 scan_patchdir "$patchdir" "$patchpattern"
 
+if [[ -n "$BR2_USE_GIT_TO_PATCH" && ! -e "${builddir}/.git" ]]; then
+    echo "BR2_USE_GIT_TO_PATCH supplied but git repository not found."
+    echo "Falling back to classical patch method."
+fi
+
 # Check for rejects...
 if [ "`find $builddir/ '(' -name '*.rej' -o -name '.*.rej' ')' -print`" ] ; then
     echo "Aborting.  Reject files found."

Signature · 1 line

2.0.0

Thomas Petazzoni

2014-11-01 21:47 UTC #1 lore raw

PATCH 1/1 infra: Add a way to apply patches using git

Dear J?r?me Pouiller,

Quoted 22 lines · J?r?me Pouiller

This option is mainly used for package developpers. When activated, Buildroot will automaticaly create a git repository for each package it extract. Initial version is tagged as `upstream'. Next, during patch step, git is used instead of patch. BTW, patches are easier to modify/fix/rebase/cherry-pick/upstream. Once satisfied with changes, developper may use
git format-patch -o $TOPDIR/package/$PKG upstream
to save its patches to Buildroot.
Notice this option should be compatible with guilt' (a clone of quilt')
Notice you can also use this feature for a specific package using
make BR2_USE_GIT_TO_PATCH=y $PKG-dirclean $PKG-patch
Current known limitations:
- If a patch cannot be handled by git am', we use git apply`. In this case, patch description is not reported in commit message.
- When patch is handled with git apply, subject used may not be perfect (it is extracted from patch filename)
- `git format-patch' does not prefix patch files with name of package.
- If upstream was downloaded with git, we may also run

Quoted 3 lines

    `git remote add $($(PKG)_SITE)' during git repository creation
Signed-off-by J?r?me Pouiller <[email protected]>

We discussed this proposal at the last Buildroot Developers meeting. Even though we understand the idea and some of us found it potentially interesting, we didn't really like the implementation, but nobody really found a way of implementing something like this in a nice way.

One of the problem is that not all our patches are Git patches, which you had to work-around in your patch in a way that isn't really nice.

Maybe Arnout and Yann (in Cc) remember more of the issues that we discussed at the meeting regarding this proposal.

Thanks,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Arnout Vandecappelle

2014-11-01 22:06 UTC #2 lore raw

PATCH 1/1 infra: Add a way to apply patches using git

Quoted 34 lines · Thomas Petazzoni

Dear J?r?me Pouiller,
Quoted 22 lines · J?r?me Pouiller
This option is mainly used for package developpers. When activated, Buildroot will automaticaly create a git repository for each package it extract. Initial version is tagged as `upstream'. Next, during patch step, git is used instead of patch. BTW, patches are easier to modify/fix/rebase/cherry-pick/upstream. Once satisfied with changes, developper may use
git format-patch -o $TOPDIR/package/$PKG upstream
to save its patches to Buildroot.
Notice this option should be compatible with guilt' (a clone of quilt')
Notice you can also use this feature for a specific package using
make BR2_USE_GIT_TO_PATCH=y $PKG-dirclean $PKG-patch
Current known limitations:
- If a patch cannot be handled by git am', we use git apply`. In this case, patch description is not reported in commit message.
- When patch is handled with git apply, subject used may not be perfect (it is extracted from patch filename)
- `git format-patch' does not prefix patch files with name of package.
- If upstream was downloaded with git, we may also run
Quoted 3 lines
    `git remote add $($(PKG)_SITE)' during git repository creation
Signed-off-by J?r?me Pouiller <[email protected]>
We discussed this proposal at the last Buildroot Developers meeting. Even though we understand the idea and some of us found it potentially interesting, we didn't really like the implementation, but nobody really found a way of implementing something like this in a nice way.

Quoted 5 lines

One of the problem is that not all our patches are Git patches, which you had to work-around in your patch in a way that isn't really nice.
Maybe Arnout and Yann (in Cc) remember more of the issues that we discussed at the meeting regarding this proposal.

For simple patches, a simple diff is normally sufficient. For more complicated
stuff, it is really worthwhile to clone the upstream repository and work from there - you'll anyway need to do that to upstream the patches.

So we felt that the need for this feature did not outweigh the complexity of
its implementation.

Regards,
Arnout

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Thomas Petazzoni

2014-11-01 22:11 UTC #3 lore raw

PATCH 1/1 infra: Add a way to apply patches using git

Dear Arnout Vandecappelle,

Quoted 11 lines · Arnout Vandecappelle

Quoted 10 lines
We discussed this proposal at the last Buildroot Developers meeting. Even though we understand the idea and some of us found it potentially interesting, we didn't really like the implementation, but nobody really found a way of implementing something like this in a nice way.
One of the problem is that not all our patches are Git patches, which you had to work-around in your patch in a way that isn't really nice.
Maybe Arnout and Yann (in Cc) remember more of the issues that we discussed at the meeting regarding this proposal.

Quoted 6 lines

For simple patches, a simple diff is normally sufficient. For more complicated
stuff, it is really worthwhile to clone the upstream repository and work from there - you'll anyway need to do that to upstream the patches.
So we felt that the need for this feature did not outweigh the complexity of
its implementation.

Ah yes, right: we thought that having a Git repository that isn't the one from upstream made it unpractical to submit patches back to upstream, and was making the entire feature a bit useless.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Jérôme Pouiller

author

2014-11-03 15:01 UTC #4 lore raw

PATCH 1/1 infra: Add a way to apply patches using git

Quoted 19 lines · On Saturday 01 November 2014 23:11:25 Thomas Petazzoni wrote:

Dear Arnout Vandecappelle,
Quoted 16 lines · Arnout Vandecappelle
Quoted 10 lines
We discussed this proposal at the last Buildroot Developers meeting. Even though we understand the idea and some of us found it potentially interesting, we didn't really like the implementation, but nobody really found a way of implementing something like this in a nice way.
One of the problem is that not all our patches are Git patches, which you had to work-around in your patch in a way that isn't really nice.
Maybe Arnout and Yann (in Cc) remember more of the issues that we discussed at the meeting regarding this proposal.
For simple patches, a simple diff is normally sufficient. For more complicated>
stuff, it is really worthwhile to clone the upstream repository and work from there - you'll anyway need to do that to upstream the patches.

Quoted 7 lines

Quoted 3 lines
So we felt that the need for this feature did not outweigh the complexity of>
its implementation.
Ah yes, right: we thought that having a Git repository that isn't the one from upstream made it unpractical to submit patches back to upstream, and was making the entire feature a bit useless.

I also worried about this restriction. My first idea was to recreate the link between source directory and upstream repository with something like:

   cd $PKG_BUILD_DIR
   git clone $PKG_SITE --bare .git
   git config --bool core.bare false
   git rev-parse $PKG_VERSION > .git/HEAD
   git reset

However, my patch began to smell black magic.

Ideally, I wanted to create an external tool to do this. However, it is not easy to integrate an external tool with Makefile system (For example, I did not found any handy way to get variables associated with package).

If anyone have ideas about this subject, I would be happy to work on and resubmit a patch.

Signature · 1 line

J?r?me Pouiller, Sysmic

Arnout Vandecappelle

2014-11-03 20:21 UTC #5 lore raw

PATCH 1/1 infra: Add a way to apply patches using git

Quoted 24 lines · J?r?me Pouiller

Quoted 23 lines · On Saturday 01 November 2014 23:11:25 Thomas Petazzoni wrote:
Dear Arnout Vandecappelle,
Quoted 19 lines · Arnout Vandecappelle
Quoted 10 lines
We discussed this proposal at the last Buildroot Developers meeting. Even though we understand the idea and some of us found it potentially interesting, we didn't really like the implementation, but nobody really found a way of implementing something like this in a nice way.
One of the problem is that not all our patches are Git patches, which you had to work-around in your patch in a way that isn't really nice.
Maybe Arnout and Yann (in Cc) remember more of the issues that we discussed at the meeting regarding this proposal.
For simple patches, a simple diff is normally sufficient. For more complicated>
stuff, it is really worthwhile to clone the upstream repository and work from there - you'll anyway need to do that to upstream the patches.
So we felt that the need for this feature did not outweigh the complexity of>
its implementation.

Quoted 8 lines

Quoted 3 lines
Ah yes, right: we thought that having a Git repository that isn't the one from upstream made it unpractical to submit patches back to upstream, and was making the entire feature a bit useless.
I also worried about this restriction. My first idea was to recreate the link between source directory and upstream repository with something like:
   cd $PKG_BUILD_DIR
   git clone $PKG_SITE --bare .git

This would only work if $PKG_SITE happened to be a git repository... But since
we prefer to use release tarballs, the usual situation should be that the repository is somewhere else.

Quoted 10 lines

   git config --bool core.bare false
   git rev-parse $PKG_VERSION > .git/HEAD
   git reset
However, my patch began to smell black magic.
Ideally, I wanted to create an external tool to do this. However, it is not easy to integrate an external tool with Makefile system (For example, I did not found any handy way to get variables associated with package).

At the BR developer meeting, we had the feeling that the number of situations
where such a tool is useful is very limited, so that it is really not worth the effort to invest in it.

Regards,
Arnout

Quoted 6 lines

If anyone have ideas about this subject, I would be happy to work on and resubmit a patch.

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F