Skip to contentSkip to conversation

support/download/git: log checked out sha1

PATCH New Ricardo Martincoski 9 participants 108 messages Nov 1, 2016 – Feb 26, 2019 Page 2 of 2

lore
Share & tools
lore
#89 lore raw

PATCH v5 7/10 testing/tests/download: test git submodules

In reply to Ricardo Martincoski

From: Ricardo Martincoski <[email protected]>

Add two submodules as static repos, add a branch to the main static repo and check on the git refs test case the download of a git package:
- repo with submodule but without support in the package;
- repo with recursive submodules with support in the package.

Cc Arnout Vandecappelle <[email protected]>

---
Layout of static repos with the series applied until this patch:

 $ git -C support/testing/tests/download/git-remote/repo.git \
   log --all --decorate --graph --oneline --decorate
 * 2fa37f6 (submodule) sub0-2
 * a9dbc1e sub0-1
 | * 68c197d (mybranch) branch2
 | * 3174063 branch1
 |/
 * a238b1d (HEAD -> master) initial
 $ git -C support/testing/tests/download/git-remote/refs-sub1.git \
   log --all --decorate --graph --oneline --decorate
 * 1df823c (submodule) sub1-2
 * f8001e5 sub1-1
 * cb545fa (HEAD) initial
 $ git -C support/testing/tests/download/git-remote/refs-sub2.git \
   log --all --decorate --graph --oneline --decorate
 * e83f6f8 (submodule) sub2-2
 * cd4d62f sub2-1
 * 32d61ba (HEAD) initial

In this layout:
- sub0-1 adds refs-sub1.git as submodule to refs.git pointing to sub1-1 commit;
- sub1-1 adds refs-sub2.git as submodule to refs-sub1.git pointing to sub2-1 commit.

Changes v4 -> v5:
- no changes

Changes v3 -> v4:
- add hash for submodule enabled tarball;

Changes v2 -> v3:
- complete rewrite using a static repo;

Changes v1 -> v2:
- rewrite using new git test infra;
- this patch is part of series 1/3 of a new version of

    http://patchwork.ozlabs.org/patch/690097/
---
 .../git-submodule-disabled.hash               |   2 ++
 .../git-submodule-disabled.mk                 |  11 +++++++
 .../git-submodule-enabled.hash                |   4 +++
 .../git-submodule-enabled.mk                  |  28 ++++++++++++++++++
 .../git-remote/refs-sub1.git/.gitattributes   |   1 +
 .../download/git-remote/refs-sub1.git/HEAD    |   1 +
 .../download/git-remote/refs-sub1.git/config  |   4 +++
 .../07/cabc655213bdf7087d8dd50fda95124e935570 | Bin 0 -> 22 bytes
 .../1d/f823cb8a6d1866148ae50a8009762a9c4c777f | Bin 0 -> 180 bytes
 .../7d/52f458bdb0f9f5a4beb46fa82824421b8b988e | Bin 0 -> 65 bytes
 .../99/f2e3e1cb15f9b52fa29f66d380dda061d917ab | Bin 0 -> 49 bytes
 .../a1/30af1626bbadd64841b2cbcb6ac4ed0638ba1a | Bin 0 -> 122 bytes
 .../bc/c81ba6e5bd8bf52c95688ae8d3e697e131fa23 | Bin 0 -> 22 bytes
 .../cb/545facf77bbc5f24f95b6d503c338d10b7b717 | Bin 0 -> 151 bytes
 .../dd/130f6f4629514adaf2e03407f3ed9344eb6cd6 | Bin 0 -> 122 bytes
 .../e7/9c5e8f964493290a409888d5413a737e8e5dd5 | Bin 0 -> 23 bytes
 .../f8/001e5780100962a5e30a25cbc4c9609cfc7bf3 | Bin 0 -> 184 bytes
 .../refs-sub1.git/refs/heads/submodule        |   1 +
 .../git-remote/refs-sub2.git/.gitattributes   |   1 +
 .../download/git-remote/refs-sub2.git/HEAD    |   1 +
 .../download/git-remote/refs-sub2.git/config  |   4 +++
 .../0a/846af45c3e455789435f49f80d70e86b65b9d7 | Bin 0 -> 22 bytes
 .../0b/32ebd8fc52cec991f18c94be980e85a8341585 | Bin 0 -> 22 bytes
 .../32/d61bae693af7879da63b89a60d3ae67f851e56 | Bin 0 -> 151 bytes
 .../3e/9b0a5198c64cea9c00f820433411e3b4d50c1c | Bin 0 -> 48 bytes
 .../85/8f666af75b7c0dfba6b8be7eac5f196e7a1221 | Bin 0 -> 49 bytes
 .../99/f2e3e1cb15f9b52fa29f66d380dda061d917ab | Bin 0 -> 49 bytes
 .../cd/4d62ff218ab7b4a04f5bfdf800ace087af3ceb | Bin 0 -> 184 bytes
 .../e7/9c5e8f964493290a409888d5413a737e8e5dd5 | Bin 0 -> 23 bytes
 .../e8/3f6f805bd016b90acafc8702c52d778eb57310 | Bin 0 -> 182 bytes
 .../refs-sub2.git/refs/heads/submodule        |   1 +
 .../0b/d8ceb961c3b2b210f64a67d57f4b5cd669d343 | Bin 0 -> 22 bytes
 .../25/59d83bfe937fc0412d96ed664663c9e8a99055 | Bin 0 -> 22 bytes
 .../2f/a37f6885d7eb746df75eccaddbacf3ac82799d | Bin 0 -> 180 bytes
 .../8b/8a7e885a041da72b1ee9a47c5b9300b172a9e7 | Bin 0 -> 65 bytes
 .../a9/dbc1e23c45e8e1b88c0448763f54d714eb6f8f | Bin 0 -> 183 bytes
 .../d4/e2941d18a63535400476324ddeb7f40164be41 | Bin 0 -> 121 bytes
 .../fe/74231105841041d5f441e70399d37f0e600aa5 | Bin 0 -> 121 bytes
 .../git-remote/repo.git/refs/heads/submodule  |   1 +
 support/testing/tests/download/test_git.py    |   2 ++
 40 files changed, 62 insertions(+)
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-submodule-disabled/git-submodule-disabled.hash
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-submodule-disabled/git-submodule-disabled.mk
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.hash
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk
 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/.gitattributes
 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/HEAD
 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/config
 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/07/cabc655213bdf7087d8dd50fda95124e935570
 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/1d/f823cb8a6d1866148ae50a8009762a9c4c777f
 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/7d/52f458bdb0f9f5a4beb46fa82824421b8b988e
 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/99/f2e3e1cb15f9b52fa29f66d380dda061d917ab
 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/a1/30af1626bbadd64841b2cbcb6ac4ed0638ba1a
 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/bc/c81ba6e5bd8bf52c95688ae8d3e697e131fa23
 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/cb/545facf77bbc5f24f95b6d503c338d10b7b717
 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/dd/130f6f4629514adaf2e03407f3ed9344eb6cd6
 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/e7/9c5e8f964493290a409888d5413a737e8e5dd5
 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/f8/001e5780100962a5e30a25cbc4c9609cfc7bf3
 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/refs/heads/submodule
 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/.gitattributes
 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/HEAD
 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/config
 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/0a/846af45c3e455789435f49f80d70e86b65b9d7
 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/0b/32ebd8fc52cec991f18c94be980e85a8341585
 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/32/d61bae693af7879da63b89a60d3ae67f851e56
 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/3e/9b0a5198c64cea9c00f820433411e3b4d50c1c
 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/85/8f666af75b7c0dfba6b8be7eac5f196e7a1221
 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/99/f2e3e1cb15f9b52fa29f66d380dda061d917ab
 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/cd/4d62ff218ab7b4a04f5bfdf800ace087af3ceb
 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/e7/9c5e8f964493290a409888d5413a737e8e5dd5
 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/e8/3f6f805bd016b90acafc8702c52d778eb57310
 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/refs/heads/submodule
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/0b/d8ceb961c3b2b210f64a67d57f4b5cd669d343
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/25/59d83bfe937fc0412d96ed664663c9e8a99055
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/2f/a37f6885d7eb746df75eccaddbacf3ac82799d
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/8b/8a7e885a041da72b1ee9a47c5b9300b172a9e7
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/a9/dbc1e23c45e8e1b88c0448763f54d714eb6f8f
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/d4/e2941d18a63535400476324ddeb7f40164be41
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/fe/74231105841041d5f441e70399d37f0e600aa5
 create mode 100644 support/testing/tests/download/git-remote/repo.git/refs/heads/submodule
support/testing/tests/download/br2-external/git-refs/package/git-submodule-disabled/git-submodule-disabled.hash new file +2 -0
new file mode 100644
index 0000000000..3bd0a44693
@@ -0,0 +1,2 @@
+sha256  f9d46ff8a1a344c6c31fa4211220f3085c446abd31626232540703158276f22c  git-submodule-disabled-a9dbc1e23c45e8e1b88c0448763f54d714eb6f8f.tar.gz
+sha256  ba8b6ddc4726bfb6a05045ebfd8c43263c968ad1bc601bd46a25bc055008eddc  file
support/testing/tests/download/br2-external/git-refs/package/git-submodule-disabled/git-submodule-disabled.mk new file +11 -0
new file mode 100644
index 0000000000..7a35b3b51a
@@ -0,0 +1,11 @@
+################################################################################
+#
+# git-submodule-disabled
+#
+################################################################################
+
+GIT_SUBMODULE_DISABLED_VERSION = a9dbc1e23c45e8e1b88c0448763f54d714eb6f8f
+GIT_SUBMODULE_DISABLED_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+GIT_SUBMODULE_DISABLED_LICENSE_FILES = file
+
+$(eval $(generic-package))
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.hash new file +4 -0
new file mode 100644
index 0000000000..089eed2365
@@ -0,0 +1,4 @@
+sha256  139a34c3c844c844dee74b6746418cfa75fbcc4205794ac8c0b3cd7d55a76792  git-submodule-enabled-a9dbc1e23c45e8e1b88c0448763f54d714eb6f8f.tar.gz
+sha256  ba8b6ddc4726bfb6a05045ebfd8c43263c968ad1bc601bd46a25bc055008eddc  file
+sha256  5219dcb50c7753bcdd72bc68a7b48af33ac2e42c5c61de78a9dd81589c4d50b6  refs-sub1/file
+sha256  b93c1fa9014d25a17fee36771d26ae023f043da656315ffe8947c30ad1ba141f  refs-sub1/refs-sub2/file
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +28 -0
new file mode 100644
index 0000000000..019dd076e3
@@ -0,0 +1,28 @@
+################################################################################
+#
+# git-submodule-enabled
+#
+################################################################################
+
+GIT_SUBMODULE_ENABLED_VERSION = a9dbc1e23c45e8e1b88c0448763f54d714eb6f8f
+GIT_SUBMODULE_ENABLED_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+GIT_SUBMODULE_ENABLED_GIT_SUBMODULES = YES
+GIT_SUBMODULE_ENABLED_LICENSE_FILES = \
+	file \
+	refs-sub1/file \
+	refs-sub1/refs-sub2/file
+
+# Some versions of git client fill the .git file for the second level submodule
+# using the absolute path to the main .git directory, which in the case of the
+# buildroot download helper is always different since it uses a temporary
+# directory. This version of git have the issue:
+# - 2.7.4 included in Ubuntu 16.04;
+# The same does not occur using newer versions of git:
+# - 2.11.0 included in Debian 9;
+# - 2.14.2 latest at the time of writing;
+# In order to allow running this test case locally in many systems, do not check
+# for the hash of the tarball, but instead check the hash of each meaningful
+# file included in the tarball.
+BR_NO_CHECK_HASH_FOR += $(GIT_SUBMODULE_ENABLED_SOURCE)
+
+$(eval $(generic-package))
support/testing/tests/download/git-remote/refs-sub1.git/.gitattributes new file +1 -0
new file mode 100644
index 0000000000..eb50c64a21
@@ -0,0 +1 @@
+objects/*/* binary
support/testing/tests/download/git-remote/refs-sub1.git/HEAD new file +1 -0
new file mode 100644
index 0000000000..4e92c1f200
@@ -0,0 +1 @@
+cb545facf77bbc5f24f95b6d503c338d10b7b717
support/testing/tests/download/git-remote/refs-sub1.git/config new file +4 -0
new file mode 100644
index 0000000000..07d359d07c
@@ -0,0 +1,4 @@
+[core]
+	repositoryformatversion = 0
+	filemode = true
+	bare = true
support/testing/tests/download/git-remote/refs-sub1.git/objects/07/cabc655213bdf7087d8dd50fda95124e935570 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..cbf5085af8cdca19c43017c92369ddc8002f1a3d
GIT binary patch
support/testing/tests/download/git-remote/refs-sub1.git/objects/1d/f823cb8a6d1866148ae50a8009762a9c4c777f new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..04055b899ecbdc3bceb2d7936757f360e90544a5
GIT binary patch

iLYL_l{*)8*lT at dLIH$Ld{<)Z=(YXjRlEVZfTHP_gG8-

literal 0
HcmV?d00001

support/testing/tests/download/git-remote/refs-sub1.git/objects/7d/52f458bdb0f9f5a4beb46fa82824421b8b988e new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..2b78a01965068cc6f7c89402eea4e7f9a03ba807
GIT binary patch
support/testing/tests/download/git-remote/refs-sub1.git/objects/99/f2e3e1cb15f9b52fa29f66d380dda061d917ab new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..9db72668cf9374b0b85a25a19f30084fd460072d
GIT binary patch
support/testing/tests/download/git-remote/refs-sub1.git/objects/a1/30af1626bbadd64841b2cbcb6ac4ed0638ba1a new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..f49d53326e0b22b95fd16670f0a25e3cd84718ae
GIT binary patch

zq2eUnJ);k*EG;u9m4W@#p41 at Wz27-%d$01}nkwWsIkdpg%m4@!ic-^xb&E at rj2O=P cCjD3J+P-ChfArrU3~L^=ueW&(03qxzkxyhgmjD0&

literal 0
HcmV?d00001

support/testing/tests/download/git-remote/refs-sub1.git/objects/bc/c81ba6e5bd8bf52c95688ae8d3e697e131fa23 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..7d2a9d55c1a92dfa2122308cc05a4a5b96d70d8f
GIT binary patch
support/testing/tests/download/git-remote/refs-sub1.git/objects/cb/545facf77bbc5f24f95b6d503c338d10b7b717 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..dc7de29459f4156b999f14fdeb6cb58b58c21a19
GIT binary patch
support/testing/tests/download/git-remote/refs-sub1.git/objects/dd/130f6f4629514adaf2e03407f3ed9344eb6cd6 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..37037d3920820ae2dab7e40d0c1df612af6f7a94
GIT binary patch

c-=zPFUE8-T at Q?ocgJI2s_VqTe0l at My>J95W4gdfE

literal 0
HcmV?d00001

support/testing/tests/download/git-remote/refs-sub1.git/objects/e7/9c5e8f964493290a409888d5413a737e8e5dd5 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..df2037a2d2d30afb20e31cd558c92c2edabe7cf6
GIT binary patch
support/testing/tests/download/git-remote/refs-sub1.git/objects/f8/001e5780100962a5e30a25cbc4c9609cfc7bf3 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..d824d3c17c1cc1c13a18eb1d2c0a7e59f66ba08b
GIT binary patch

m3s?Yk)7$hc|CCdgE1AV8b*ksR at X|#V`g#7#tNI184{X$IS6W&C

literal 0
HcmV?d00001

support/testing/tests/download/git-remote/refs-sub1.git/refs/heads/submodule new file +1 -0
new file mode 100644
index 0000000000..bf97dc4185
@@ -0,0 +1 @@
+1df823cb8a6d1866148ae50a8009762a9c4c777f
support/testing/tests/download/git-remote/refs-sub2.git/.gitattributes new file +1 -0
new file mode 100644
index 0000000000..eb50c64a21
@@ -0,0 +1 @@
+objects/*/* binary
support/testing/tests/download/git-remote/refs-sub2.git/HEAD new file +1 -0
new file mode 100644
index 0000000000..bbf19a4bb4
@@ -0,0 +1 @@
+32d61bae693af7879da63b89a60d3ae67f851e56
support/testing/tests/download/git-remote/refs-sub2.git/config new file +4 -0
new file mode 100644
index 0000000000..07d359d07c
@@ -0,0 +1,4 @@
+[core]
+	repositoryformatversion = 0
+	filemode = true
+	bare = true
support/testing/tests/download/git-remote/refs-sub2.git/objects/0a/846af45c3e455789435f49f80d70e86b65b9d7 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..aaa29dbd8a5a74a24f51a8522b695a7474349f3c
GIT binary patch
support/testing/tests/download/git-remote/refs-sub2.git/objects/0b/32ebd8fc52cec991f18c94be980e85a8341585 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..50caf8b6282d2287878d1b3481c14431c2a557af
GIT binary patch
support/testing/tests/download/git-remote/refs-sub2.git/objects/32/d61bae693af7879da63b89a60d3ae67f851e56 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..a8887b1e611d8f76d7cfe6ab0bb693fdebc2ef19
GIT binary patch

zL;YZ%zj&H6 at zVQ}k03QpsEr&gbf^%;rrSh|f6577xiKw5TzOt&=t@eRd+mRj6>nh7 FU at NX0KJNek

literal 0
HcmV?d00001

support/testing/tests/download/git-remote/refs-sub2.git/objects/3e/9b0a5198c64cea9c00f820433411e3b4d50c1c new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..b1fb722c53d827847c6d13029e601a5434b84ac4
GIT binary patch
support/testing/tests/download/git-remote/refs-sub2.git/objects/85/8f666af75b7c0dfba6b8be7eac5f196e7a1221 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..10f24a087ff08320b8fc5922db728472bd276e9a
GIT binary patch
support/testing/tests/download/git-remote/refs-sub2.git/objects/99/f2e3e1cb15f9b52fa29f66d380dda061d917ab new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..9db72668cf9374b0b85a25a19f30084fd460072d
GIT binary patch
support/testing/tests/download/git-remote/refs-sub2.git/objects/cd/4d62ff218ab7b4a04f5bfdf800ace087af3ceb new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..35d253f0d15b31ea0bc79ed1a287819deed9987d
GIT binary patch

zeR?YT87lKJ&evQHHQdjRq*5tF$Mz2eg>npQHpgm$pR at 75A3{t50g_7*#(Vx at k>bP zbo2ooB)#*Jy;W?Tz~{KNF5e1aPffmJA7!Ps-HSZ+A3WAyyj&}mk0Z5umdb($>r7bW m1T%oxb&nqWPj_N_#(7dgyqD|BX=*u7@$=;t>O#gcx)*r3s*J(

literal 0
HcmV?d00001

support/testing/tests/download/git-remote/refs-sub2.git/objects/e7/9c5e8f964493290a409888d5413a737e8e5dd5 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..df2037a2d2d30afb20e31cd558c92c2edabe7cf6
GIT binary patch
support/testing/tests/download/git-remote/refs-sub2.git/objects/e8/3f6f805bd016b90acafc8702c52d778eb57310 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..05c04d2887e9354ec43ad69cef20ed4e7166013d
GIT binary patch

k2QXK?PEUDI&YUmo7jMj0J#12zuDbB=%RR5=4+?N at DHiEdA^-pY

literal 0
HcmV?d00001

support/testing/tests/download/git-remote/refs-sub2.git/refs/heads/submodule new file +1 -0
new file mode 100644
index 0000000000..39dc53f019
@@ -0,0 +1 @@
+e83f6f805bd016b90acafc8702c52d778eb57310
support/testing/tests/download/git-remote/repo.git/objects/0b/d8ceb961c3b2b210f64a67d57f4b5cd669d343 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..ef7fbd4154aea755d8ee2684e420700467e9245c
GIT binary patch
support/testing/tests/download/git-remote/repo.git/objects/25/59d83bfe937fc0412d96ed664663c9e8a99055 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..cbf60fc7298ec12b3d7cd388fecac606ca31536e
GIT binary patch
support/testing/tests/download/git-remote/repo.git/objects/2f/a37f6885d7eb746df75eccaddbacf3ac82799d new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..2dd14580e666d35483727598f006148f3eeb03be
GIT binary patch

zoEleBZ(1ed0ZDFA1Yzu;w9*KG<DevlKyHi(Vn9bI*wa>;8n at fvydbGOYBD68F&+ at j z>O_rU6i9A}CNjIUTdDk<yse?|ldUaHzRdT;_w at t&{DsGBw&~IPk`G*HFrddZ%R7(& iY^CdTqd(=?yg2A2W0!QDD0C&p&b@ZOys|fCz-zg3O;?Bj

literal 0
HcmV?d00001

support/testing/tests/download/git-remote/repo.git/objects/8b/8a7e885a041da72b1ee9a47c5b9300b172a9e7 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..70cf1795f3210f81951d0995823bb958867ba710
GIT binary patch
support/testing/tests/download/git-remote/repo.git/objects/a9/dbc1e23c45e8e1b88c0448763f54d714eb6f8f new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..2cbdce1f95493b284df1f8ca3ca953a5fb813072
GIT binary patch

z*+yrKCG>Htpv)kOZ{w$R{aqq=Y5Kr at lr^^Nx#*Anz+HXf*Qs*3>|(2*8XW|19*ox$ l7yz~DHa*fm<<#}Iz=TR2%V|wAwVbErzJKLa{Q<P{YY&#BUEBZw

literal 0
HcmV?d00001

support/testing/tests/download/git-remote/repo.git/objects/d4/e2941d18a63535400476324ddeb7f40164be41 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..4d82169300d0d8a2e31ced481501bff6647ef6b7
GIT binary patch
support/testing/tests/download/git-remote/repo.git/objects/fe/74231105841041d5f441e70399d37f0e600aa5 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..6d0ae484c3333d6b18cf726d5bb468669f9d16c1
GIT binary patch
support/testing/tests/download/git-remote/repo.git/refs/heads/submodule new file +1 -0
new file mode 100644
index 0000000000..e9f4f32c30
@@ -0,0 +1 @@
+2fa37f6885d7eb746df75eccaddbacf3ac82799d
support/testing/tests/download/test_git.py +2 -0
index 279d0ed8eb..803a50d054 100644
@@ -69,3 +69,5 @@ class TestGitRefs(GitTestBase):
         self.check_download("git-partial-sha1-reachable-by-branch")
         self.check_download("git-sha1-branch-head")
         self.check_download("git-sha1-reachable-by-branch")
+        self.check_download("git-submodule-disabled")
+        self.check_download("git-submodule-enabled")
Signature · 1 line
2.17.0
#90 lore raw

PATCH v5 7/10 testing/tests/download: test git submodules

Quoted 15 lines · Ricardo Martincoski
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +15 -0
+################################################################################
+#
+# git-submodule-enabled
+#
+################################################################################
+
+GIT_SUBMODULE_ENABLED_VERSION = a9dbc1e23c45e8e1b88c0448763f54d714eb6f8f
+GIT_SUBMODULE_ENABLED_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+GIT_SUBMODULE_ENABLED_GIT_SUBMODULES = YES
+GIT_SUBMODULE_ENABLED_LICENSE_FILES = \
+	file \
+	refs-sub1/file \
+	refs-sub1/refs-sub2/file
+
+# Some versions of git client fill the .git file for the second level submodule

support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk line 25

Quoted 10 lines
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +10 -0
+# using the absolute path to the main .git directory, which in the case of the
+# buildroot download helper is always different since it uses a temporary
+# directory. This version of git have the issue:
+# - 2.7.4 included in Ubuntu 16.04;
+# The same does not occur using newer versions of git:
+# - 2.11.0 included in Debian 9;
+# - 2.14.2 latest at the time of writing;
+# In order to allow running this test case locally in many systems, do not check
+# for the hash of the tarball, but instead check the hash of each meaningful
+# file included in the tarball.

This is a bug! Funny that we never hit it in the autobuilders, don't we have an
Ubuntu 16.04 autobuilder and don't we have packages with submodules and a hash?

Anyway, I think maybe we should fix the bug first... Ideally there should be a
test case for it as well, but that's difficult since it depends on the installed git version.

Yann, how about if we change the find command in the git helper to

find . -name .git -prune -o -print

?

Regards,
Arnout

Quoted 3 lines
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +3 -0
+BR_NO_CHECK_HASH_FOR += $(GIT_SUBMODULE_ENABLED_SOURCE)
+
+$(eval $(generic-package))
#91 lore raw

PATCH v5 7/10 testing/tests/download: test git submodules

Quoted 22 lines · Arnout Vandecappelle
Quoted 19 lines · Ricardo Martincoski
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +19 -0
+################################################################################
+#
+# git-submodule-enabled
+#
+################################################################################
+
+GIT_SUBMODULE_ENABLED_VERSION = a9dbc1e23c45e8e1b88c0448763f54d714eb6f8f
+GIT_SUBMODULE_ENABLED_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+GIT_SUBMODULE_ENABLED_GIT_SUBMODULES = YES
+GIT_SUBMODULE_ENABLED_LICENSE_FILES = \
+	file \
+	refs-sub1/file \
+	refs-sub1/refs-sub2/file
+
+# Some versions of git client fill the .git file for the second level submodule
+# using the absolute path to the main .git directory, which in the case of the
+# buildroot download helper is always different since it uses a temporary
+# directory. This version of git have the issue:
+# - 2.7.4 included in Ubuntu 16.04;
Quoted 8 lines

support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk line 25

Quoted 6 lines
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +6 -0
+# The same does not occur using newer versions of git:
+# - 2.11.0 included in Debian 9;
+# - 2.14.2 latest at the time of writing;
+# In order to allow running this test case locally in many systems, do not check
+# for the hash of the tarball, but instead check the hash of each meaningful
+# file included in the tarball.

This is a bug!

So, I've removed this workaround from the test and pushed. It will most likely
fail in gitlab because our container is based on Ubuntu 16.04. But thats exactly what we want, because it is a bug.

Regards,
Arnout

Quoted 19 lines

Funny that we never hit it in the autobuilders, don't we have an Ubuntu 16.04 autobuilder and don't we have packages with submodules and a hash?

Anyway, I think maybe we should fix the bug first... Ideally there should be a
test case for it as well, but that's difficult since it depends on the installed git version.

Yann, how about if we change the find command in the git helper to

find . -name .git -prune -o -print

?

Regards,
Arnout

Quoted 3 lines
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +3 -0
+BR_NO_CHECK_HASH_FOR += $(GIT_SUBMODULE_ENABLED_SOURCE)
+
+$(eval $(generic-package))
#92 lore raw

PATCH v5 7/10 testing/tests/download: test git submodules

In reply to Arnout Vandecappelle

Arnout, All,

On 2019-02-05 11:03 +0100, Arnout Vandecappelle spake thusly:

Quoted 30 lines · Ricardo Martincoski
Quoted 15 lines
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +15 -0
+################################################################################
+#
+# git-submodule-enabled
+#
+################################################################################
+
+GIT_SUBMODULE_ENABLED_VERSION = a9dbc1e23c45e8e1b88c0448763f54d714eb6f8f
+GIT_SUBMODULE_ENABLED_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+GIT_SUBMODULE_ENABLED_GIT_SUBMODULES = YES
+GIT_SUBMODULE_ENABLED_LICENSE_FILES = \
+	file \
+	refs-sub1/file \
+	refs-sub1/refs-sub2/file
+
+# Some versions of git client fill the .git file for the second level submodule

support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk line 25

Quoted 10 lines
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +10 -0
+# using the absolute path to the main .git directory, which in the case of the
+# buildroot download helper is always different since it uses a temporary
+# directory. This version of git have the issue:
+# - 2.7.4 included in Ubuntu 16.04;
+# The same does not occur using newer versions of git:
+# - 2.11.0 included in Debian 9;
+# - 2.14.2 latest at the time of writing;
+# In order to allow running this test case locally in many systems, do not check
+# for the hash of the tarball, but instead check the hash of each meaningful
+# file included in the tarball.

This is a bug! Funny that we never hit it in the autobuilders, don't we have an
Ubuntu 16.04 autobuilder and don't we have packages with submodules and a hash?

Quoted 7 lines

Anyway, I think maybe we should fix the bug first... Ideally there should be a
test case for it as well, but that's difficult since it depends on the installed git version.

Yann, how about if we change the find command in the git helper to
find . -name .git -prune -o -print
?

Yeah, I too consider it a bug, and this should be fixed. I'll tackle this as part of my git-backend rework in the upcoming days.

Sorry I did not reply to the mail earlier, with all the live discussion we had yesterday it just slipped my mind...

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#93 lore raw

PATCH v5 8/10 testing/tests/download: test git tag

In reply to Ricardo Martincoski

From: Ricardo Martincoski <[email protected]>

Add a tag to the static repo and check on the git refs test case the download of a git package:
- with the name of a tag as version;
- with the sha1 of a tag itself as version;
- with the partial sha1 of a tag itself as version;
- with the sha1 of a commit pointed by a tag as version;
- with the partial sha1 of a commit pointed by a tag as version;
- with the sha1 of a commit reachable only by a tag as version;
- with the partial sha1 of a commit reachable only by a tag as version.

Cc Arnout Vandecappelle <[email protected]>

---
Layout of static repo with the series applied until this patch:

 $ git -C support/testing/tests/download/git-remote/repo.git \
   log --all --decorate --graph --oneline --decorate
 * 2fa37f6 (submodule) sub0-2
 * a9dbc1e sub0-1
 | * 68c197d (mybranch) branch2
 | * 3174063 branch1
 |/
 | * 516c9c5 (tag: mytag) tag2
 | * 46bae5b tag1
 |/
 * a238b1d (HEAD -> master) initial

Changes v4 -> v5:
- no changes

Changes v1 -> v4:
- not included on v2 or v3;
- rewrite from v1 using new git test infra with static repo;

---
 .../git-partial-sha1-reachable-by-tag.hash        |   2 ++
 .../git-partial-sha1-reachable-by-tag.mk          |  11 +++++++++++
 .../git-partial-sha1-tag-itself.hash              |   2 ++
 .../git-partial-sha1-tag-itself.mk                |  11 +++++++++++
 .../git-partial-sha1-tag-points-to.hash           |   2 ++
 .../git-partial-sha1-tag-points-to.mk             |  11 +++++++++++
 .../git-sha1-reachable-by-tag.hash                |   2 ++
 .../git-sha1-reachable-by-tag.mk                  |  11 +++++++++++
 .../git-sha1-tag-itself/git-sha1-tag-itself.hash  |   2 ++
 .../git-sha1-tag-itself/git-sha1-tag-itself.mk    |  11 +++++++++++
 .../git-sha1-tag-points-to.hash                   |   2 ++
 .../git-sha1-tag-points-to.mk                     |  11 +++++++++++
 .../git-refs/package/git-tag/git-tag.hash         |   2 ++
 .../git-refs/package/git-tag/git-tag.mk           |  11 +++++++++++
 .../2b/0e0d98a49c97da6a618ab36337e2058eb733a2     | Bin 0 -> 137 bytes
 .../46/bae5b639e5a18e2cc4dc508f080d566baeff59     | Bin 0 -> 182 bytes
 .../51/6c9c5f64ec66534d4d069c2e408d9ae4dce023     | Bin 0 -> 182 bytes
 .../92/ef85be57d627f280d8ce3724452ac21c9a6452     | Bin 0 -> 20 bytes
 .../cf/0f4f85d7a1237e377a2d25b996518a877ea001     | Bin 0 -> 49 bytes
 .../e2/2695cbf976fed1f543ad7486a531c0af473482     | Bin 0 -> 49 bytes
 .../f6/476b879f65e956d7dedd5b08736369e9a24acc     | Bin 0 -> 20 bytes
 .../download/git-remote/repo.git/refs/tags/mytag  |   1 +
 support/testing/tests/download/test_git.py        |   7 +++++++
 23 files changed, 99 insertions(+)
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-reachable-by-tag/git-partial-sha1-reachable-by-tag.hash
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-reachable-by-tag/git-partial-sha1-reachable-by-tag.mk
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-tag-itself/git-partial-sha1-tag-itself.hash
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-tag-itself/git-partial-sha1-tag-itself.mk
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-tag-points-to/git-partial-sha1-tag-points-to.hash
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-tag-points-to/git-partial-sha1-tag-points-to.mk
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-sha1-reachable-by-tag/git-sha1-reachable-by-tag.hash
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-sha1-reachable-by-tag/git-sha1-reachable-by-tag.mk
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-sha1-tag-itself/git-sha1-tag-itself.hash
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-sha1-tag-itself/git-sha1-tag-itself.mk
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-sha1-tag-points-to/git-sha1-tag-points-to.hash
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-sha1-tag-points-to/git-sha1-tag-points-to.mk
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-tag/git-tag.hash
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-tag/git-tag.mk
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/2b/0e0d98a49c97da6a618ab36337e2058eb733a2
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/46/bae5b639e5a18e2cc4dc508f080d566baeff59
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/51/6c9c5f64ec66534d4d069c2e408d9ae4dce023
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/92/ef85be57d627f280d8ce3724452ac21c9a6452
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/cf/0f4f85d7a1237e377a2d25b996518a877ea001
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/e2/2695cbf976fed1f543ad7486a531c0af473482
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/f6/476b879f65e956d7dedd5b08736369e9a24acc
 create mode 100644 support/testing/tests/download/git-remote/repo.git/refs/tags/mytag
support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-reachable-by-tag/git-partial-sha1-reachable-by-tag.hash new file +2 -0
new file mode 100644
index 0000000000..7696086690
@@ -0,0 +1,2 @@
+sha256  f2ef9772f13a9ef9a2c7cde0795e179defb12320d1747fc8d2408748ef5844c2  git-partial-sha1-reachable-by-tag-46bae5b639e5a18e2cc4.tar.gz
+sha256  2de87d77a2f226813f2d9bda906e970e4195605cdba6680443c0c04d89c532b6  file
support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-reachable-by-tag/git-partial-sha1-reachable-by-tag.mk new file +11 -0
new file mode 100644
index 0000000000..05aa659dd2
@@ -0,0 +1,11 @@
+################################################################################
+#
+# git-partial-sha1-reachable-by-tag
+#
+################################################################################
+
+GIT_PARTIAL_SHA1_REACHABLE_BY_TAG_VERSION = 46bae5b639e5a18e2cc4
+GIT_PARTIAL_SHA1_REACHABLE_BY_TAG_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+GIT_PARTIAL_SHA1_REACHABLE_BY_TAG_LICENSE_FILES = file
+
+$(eval $(generic-package))
support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-tag-itself/git-partial-sha1-tag-itself.hash new file +2 -0
new file mode 100644
index 0000000000..e627caf91e
@@ -0,0 +1,2 @@
+sha256  721143b41b8e56cfd9025833f1602e900a490627db2504e5b4907baa23e0019e  git-partial-sha1-tag-itself-2b0e0d98a49c97da6a61.tar.gz
+sha256  6de8772a0a58fa62e2b8c58d4dae55c9db7534ad3b3918ecc849a9008d58f081  file
support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-tag-itself/git-partial-sha1-tag-itself.mk new file +11 -0
new file mode 100644
index 0000000000..515492397e
@@ -0,0 +1,11 @@
+################################################################################
+#
+# git-partial-sha1-tag-itself
+#
+################################################################################
+
+GIT_PARTIAL_SHA1_TAG_ITSELF_VERSION = 2b0e0d98a49c97da6a61
+GIT_PARTIAL_SHA1_TAG_ITSELF_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+GIT_PARTIAL_SHA1_TAG_ITSELF_LICENSE_FILES = file
+
+$(eval $(generic-package))
support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-tag-points-to/git-partial-sha1-tag-points-to.hash new file +2 -0
new file mode 100644
index 0000000000..f957a0e23c
@@ -0,0 +1,2 @@
+sha256  0fbf7fe935f962ceaafcf8e0ffd25dd2a83752c3f0fd055a942a53f8c9235fa7  git-partial-sha1-tag-points-to-516c9c5f64ec66534d4d.tar.gz
+sha256  6de8772a0a58fa62e2b8c58d4dae55c9db7534ad3b3918ecc849a9008d58f081  file
support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-tag-points-to/git-partial-sha1-tag-points-to.mk new file +11 -0
new file mode 100644
index 0000000000..c810e81175
@@ -0,0 +1,11 @@
+################################################################################
+#
+# git-partial-sha1-tag-points-to
+#
+################################################################################
+
+GIT_PARTIAL_SHA1_TAG_POINTS_TO_VERSION = 516c9c5f64ec66534d4d
+GIT_PARTIAL_SHA1_TAG_POINTS_TO_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+GIT_PARTIAL_SHA1_TAG_POINTS_TO_LICENSE_FILES = file
+
+$(eval $(generic-package))
support/testing/tests/download/br2-external/git-refs/package/git-sha1-reachable-by-tag/git-sha1-reachable-by-tag.hash new file +2 -0
new file mode 100644
index 0000000000..0eb0ca0917
@@ -0,0 +1,2 @@
+sha256  9b20256a3058221a8e91031f11700d9945ea84e8f328fa8e42e1cb9f7a30e3b2  git-sha1-reachable-by-tag-46bae5b639e5a18e2cc4dc508f080d566baeff59.tar.gz
+sha256  2de87d77a2f226813f2d9bda906e970e4195605cdba6680443c0c04d89c532b6  file
support/testing/tests/download/br2-external/git-refs/package/git-sha1-reachable-by-tag/git-sha1-reachable-by-tag.mk new file +11 -0
new file mode 100644
index 0000000000..b5fde7b586
@@ -0,0 +1,11 @@
+################################################################################
+#
+# git-sha1-reachable-by-tag
+#
+################################################################################
+
+GIT_SHA1_REACHABLE_BY_TAG_VERSION = 46bae5b639e5a18e2cc4dc508f080d566baeff59
+GIT_SHA1_REACHABLE_BY_TAG_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+GIT_SHA1_REACHABLE_BY_TAG_LICENSE_FILES = file
+
+$(eval $(generic-package))
support/testing/tests/download/br2-external/git-refs/package/git-sha1-tag-itself/git-sha1-tag-itself.hash new file +2 -0
new file mode 100644
index 0000000000..48c1348538
@@ -0,0 +1,2 @@
+sha256  7d301c1a1054d6aee49193ca9e938f4da561ff73fb01719662865aa38bdc4361  git-sha1-tag-itself-2b0e0d98a49c97da6a618ab36337e2058eb733a2.tar.gz
+sha256  6de8772a0a58fa62e2b8c58d4dae55c9db7534ad3b3918ecc849a9008d58f081  file
support/testing/tests/download/br2-external/git-refs/package/git-sha1-tag-itself/git-sha1-tag-itself.mk new file +11 -0
new file mode 100644
index 0000000000..8914496653
@@ -0,0 +1,11 @@
+################################################################################
+#
+# git-sha1-tag-itself
+#
+################################################################################
+
+GIT_SHA1_TAG_ITSELF_VERSION = 2b0e0d98a49c97da6a618ab36337e2058eb733a2
+GIT_SHA1_TAG_ITSELF_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+GIT_SHA1_TAG_ITSELF_LICENSE_FILES = file
+
+$(eval $(generic-package))
support/testing/tests/download/br2-external/git-refs/package/git-sha1-tag-points-to/git-sha1-tag-points-to.hash new file +2 -0
new file mode 100644
index 0000000000..3dcafc2094
@@ -0,0 +1,2 @@
+sha256  c1f9f5734529a31140a71c031534460811f001b4db37d26833f386358150ab47  git-sha1-tag-points-to-516c9c5f64ec66534d4d069c2e408d9ae4dce023.tar.gz
+sha256  6de8772a0a58fa62e2b8c58d4dae55c9db7534ad3b3918ecc849a9008d58f081  file
support/testing/tests/download/br2-external/git-refs/package/git-sha1-tag-points-to/git-sha1-tag-points-to.mk new file +11 -0
new file mode 100644
index 0000000000..adaae7329c
@@ -0,0 +1,11 @@
+################################################################################
+#
+# git-sha1-tag-points-to
+#
+################################################################################
+
+GIT_SHA1_TAG_POINTS_TO_VERSION = 516c9c5f64ec66534d4d069c2e408d9ae4dce023
+GIT_SHA1_TAG_POINTS_TO_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+GIT_SHA1_TAG_POINTS_TO_LICENSE_FILES = file
+
+$(eval $(generic-package))
support/testing/tests/download/br2-external/git-refs/package/git-tag/git-tag.hash new file +2 -0
new file mode 100644
index 0000000000..1cd0b15f27
@@ -0,0 +1,2 @@
+sha256  85dcb5bcf9bed496688d0eb01c7a3ce05c5b46b984cc1e9e76a6dbefd976e6b3  git-tag-mytag.tar.gz
+sha256  6de8772a0a58fa62e2b8c58d4dae55c9db7534ad3b3918ecc849a9008d58f081  file
support/testing/tests/download/br2-external/git-refs/package/git-tag/git-tag.mk new file +11 -0
new file mode 100644
index 0000000000..6960ceb2cb
@@ -0,0 +1,11 @@
+################################################################################
+#
+# git-tag
+#
+################################################################################
+
+GIT_TAG_VERSION = mytag
+GIT_TAG_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+GIT_TAG_LICENSE_FILES = file
+
+$(eval $(generic-package))
support/testing/tests/download/git-remote/repo.git/objects/2b/0e0d98a49c97da6a618ab36337e2058eb733a2 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..39c913094a3f2d8d9d84cda13a89306bc577f982
GIT binary patch

zcmV;40CxX)0Y#2W4gw(%1zmHBzJQY+enMi57jWYZuxSY at gF!ZPdoi<c^HQlr<@JzY zHfmRI52g-gw&cm&t!A>;C>?b~>q)Xk7kywIiBT%i*Ea*zQgRo4G|D>mM~NF;Gldo_ rTte%4q564dIJQkr<uA{vg#4OD<}^;=!5b&h!5$T&5L at yK5lk}^QNuc<

literal 0
HcmV?d00001

support/testing/tests/download/git-remote/repo.git/objects/46/bae5b639e5a18e2cc4dc508f080d566baeff59 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..3bf8865ea67386bd0fd88ab7222392d5b66b3607
GIT binary patch

zcmV;n07?IN0j<t4ZUZ3 at KvCD6f(xw0FyLXMDDnl;#3z_#V2u)%2$RCMx1|*BEw``u zmxnohDM9K(Noefo9uG%LJSruArV#ewpWkhrod-u!^2xu<Ptwip(W93@yUjwGPt zjL}<*4k!i(AhQw0f8*S{wk5XH2ukalr6T)chT?tANTcv$F*|#*~eBtG<ZX^4{-2$ kGH8st>wS8p7v<F86_h%abxSg}oTugb_ROie0nJ8hV&q3%KL7v#

literal 0
HcmV?d00001

support/testing/tests/download/git-remote/repo.git/objects/51/6c9c5f64ec66534d4d069c2e408d9ae4dce023 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..0bda23ab4d6487524fc254c54b673e53bfe69435
GIT binary patch
support/testing/tests/download/git-remote/repo.git/objects/92/ef85be57d627f280d8ce3724452ac21c9a6452 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..1f2474ba3209802c0f297f390eb517b4419621f9
GIT binary patch
support/testing/tests/download/git-remote/repo.git/objects/cf/0f4f85d7a1237e377a2d25b996518a877ea001 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..fd224552518d57b0b8b0a1d618eef507f798555f
GIT binary patch

zcmV-10M7q-0V^p=O;s>9VlXr?Ff%bxNXyJgWtjB7bzk^3^-m2q&Y7#YY8{f9l at bI1 H7n%=ExH=M#

literal 0
HcmV?d00001

support/testing/tests/download/git-remote/repo.git/objects/e2/2695cbf976fed1f543ad7486a531c0af473482 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..15d80c78fac8655d4dfe4b8b543076c8bd26408e
GIT binary patch
support/testing/tests/download/git-remote/repo.git/objects/f6/476b879f65e956d7dedd5b08736369e9a24acc new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..ee01041e0abcefa3c783ef3fc91e31c33edfb7cf
GIT binary patch
support/testing/tests/download/git-remote/repo.git/refs/tags/mytag new file +1 -0
new file mode 100644
index 0000000000..8c09448ad2
@@ -0,0 +1 @@
+2b0e0d98a49c97da6a618ab36337e2058eb733a2
support/testing/tests/download/test_git.py +7 -0
index 803a50d054..ce3655121c 100644
@@ -67,7 +67,14 @@ class TestGitRefs(GitTestBase):
             self.check_download("git-wrong-sha1")
         self.check_download("git-partial-sha1-branch-head")
         self.check_download("git-partial-sha1-reachable-by-branch")
+        self.check_download("git-partial-sha1-reachable-by-tag")
+        self.check_download("git-partial-sha1-tag-itself")
+        self.check_download("git-partial-sha1-tag-points-to")
         self.check_download("git-sha1-branch-head")
         self.check_download("git-sha1-reachable-by-branch")
+        self.check_download("git-sha1-reachable-by-tag")
+        self.check_download("git-sha1-tag-itself")
+        self.check_download("git-sha1-tag-points-to")
         self.check_download("git-submodule-disabled")
         self.check_download("git-submodule-enabled")
+        self.check_download("git-tag")
Signature · 1 line
2.17.0
#94 lore raw

PATCH v5 8/10 testing/tests/download: test git tag

Quoted 12 lines · Ricardo Martincoski

From: Ricardo Martincoski <[email protected]>

Add a tag to the static repo and check on the git refs test case the download of a git package:
- with the name of a tag as version;
- with the sha1 of a tag itself as version;
- with the partial sha1 of a tag itself as version;
- with the sha1 of a commit pointed by a tag as version;
- with the partial sha1 of a commit pointed by a tag as version;
- with the sha1 of a commit reachable only by a tag as version;
- with the partial sha1 of a commit reachable only by a tag as version.

Quoted 2 lines
Cc Arnout Vandecappelle <[email protected]>

Applied to master, thanks.

Regards,
Arnout

#95 lore raw

PATCH v5 9/10 testing/tests/download: test git special ref

In reply to Ricardo Martincoski

From: Ricardo Martincoski <[email protected]>

Add a special ref to the static repo and check on the git refs test case the download of a git package:
- with the name of a special ref as version;

Cc Arnout Vandecappelle <[email protected]>

---
Layout of static repo with the series applied until this patch:

 $ git -C support/testing/tests/download/git-remote/repo.git \
   log --all --decorate --graph --oneline --decorate
 * 2fa37f6 (submodule) sub0-2
 * a9dbc1e sub0-1
 | * b72ff60 (refs/changes/01/1/2) specialref
 |/
 | * 68c197d (mybranch) branch2
 | * 3174063 branch1
 |/
 | * 516c9c5 (tag: mytag) tag2
 | * 46bae5b tag1
 |/
 * a238b1d (HEAD -> master) initial

Changes v4 -> v5:
- no changes

Changes v1 -> v4:
- not included on v2 or v3;
- rewrite from v1 using new git test infra with static repo;

---
 .../package/git-special-ref/git-special-ref.hash  |   2 ++
 .../package/git-special-ref/git-special-ref.mk    |  11 +++++++++++
 .../17/c409592968d17600ff9912e96fdd461bb72e74     | Bin 0 -> 48 bytes
 .../53/122a7b0454b33c6d9e159c10657173db77899e     | Bin 0 -> 27 bytes
 .../b7/2ff6078f62522a87f5cae5e9f34dedf5ec3885     | Bin 0 -> 186 bytes
 .../git-remote/repo.git/refs/changes/01/1/2       |   1 +
 support/testing/tests/download/test_git.py        |   1 +
 7 files changed, 15 insertions(+)
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-special-ref/git-special-ref.hash
 create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-special-ref/git-special-ref.mk
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/17/c409592968d17600ff9912e96fdd461bb72e74
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/53/122a7b0454b33c6d9e159c10657173db77899e
 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/b7/2ff6078f62522a87f5cae5e9f34dedf5ec3885
 create mode 100644 support/testing/tests/download/git-remote/repo.git/refs/changes/01/1/2
support/testing/tests/download/br2-external/git-refs/package/git-special-ref/git-special-ref.hash new file +2 -0
new file mode 100644
index 0000000000..13e21aaeca
@@ -0,0 +1,2 @@
+sha256  c639b15260d098d94b8802cae812453c9f878ffd5f80cca6d87c58c01eed5c5a  git-special-ref-refs_changes_01_1_2.tar.gz
+sha256  b252b594be4679764e2e06cfea77a3e50638dd445af9b472b0f298c43b134518  file
support/testing/tests/download/br2-external/git-refs/package/git-special-ref/git-special-ref.mk new file +11 -0
new file mode 100644
index 0000000000..e7561bafca
@@ -0,0 +1,11 @@
+################################################################################
+#
+# git-special-ref
+#
+################################################################################
+
+GIT_SPECIAL_REF_VERSION = refs/changes/01/1/2
+GIT_SPECIAL_REF_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+GIT_SPECIAL_REF_LICENSE_FILES = file
+
+$(eval $(generic-package))
support/testing/tests/download/git-remote/repo.git/objects/17/c409592968d17600ff9912e96fdd461bb72e74 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..8c19de6455168ac65c81d70219e01eb2431c069e
GIT binary patch
support/testing/tests/download/git-remote/repo.git/objects/53/122a7b0454b33c6d9e159c10657173db77899e new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..d72d49eed56298f84aaf85967dbf051da94f3871
GIT binary patch
support/testing/tests/download/git-remote/repo.git/objects/b7/2ff6078f62522a87f5cae5e9f34dedf5ec3885 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..6727b2e8083f2a31c48e849caa4cd772a6778e07
GIT binary patch

o0SllGJ*KDpQ%>D)l54sSDN4Q9^DeyfQkL!O at h`vX8x-PgzB#pAi~s-t

literal 0
HcmV?d00001

support/testing/tests/download/git-remote/repo.git/refs/changes/01/1/2 new file +1 -0
new file mode 100644
index 0000000000..7066394dd0
@@ -0,0 +1 @@
+b72ff6078f62522a87f5cae5e9f34dedf5ec3885
support/testing/tests/download/test_git.py +1 -0
index ce3655121c..162c03623b 100644
@@ -75,6 +75,7 @@ class TestGitRefs(GitTestBase):
         self.check_download("git-sha1-reachable-by-tag")
         self.check_download("git-sha1-tag-itself")
         self.check_download("git-sha1-tag-points-to")
+        self.check_download("git-special-ref")
         self.check_download("git-submodule-disabled")
         self.check_download("git-submodule-enabled")
         self.check_download("git-tag")
Signature · 1 line
2.17.0
#96 lore raw

PATCH v5 9/10 testing/tests/download: test git special ref

Quoted 5 lines · Ricardo Martincoski

From: Ricardo Martincoski <[email protected]>

Add a special ref to the static repo and check on the git refs test case the download of a git package:
- with the name of a special ref as version;

We might remove support for getting special refs by name, but keep on
supporting getting special refs by sha1. So I've changed the test into getting it by sha1 instead.

Ideally there should also be a check for getting the parent of the special ref
by sha1, but I think that actually does *not* work now.

Regards,
Arnout

#97 lore raw

PATCH v5 9/10 testing/tests/download: test git special ref

Hello,

Quoted 10 lines · Arnout Vandecappelle
Quoted 5 lines · Ricardo Martincoski

From: Ricardo Martincoski <[email protected]>

Add a special ref to the static repo and check on the git refs test case the download of a git package:
- with the name of a special ref as version;

We might remove support for getting special refs by name, but keep on
supporting getting special refs by sha1. So I've changed the test into getting it by sha1 instead.

OK... but it depends on a recent git version to work. The same does not occur when getting special refs by name.

When using recent versions of git the download of the sha1 of a special ref works because of this line in support/download/git:
_git fetch origin "'${cset}:${cset}'"
I tested using:
git version 2.17.1

The git version installed in our docker image does not support it.

I am not sure what is the best fix for the TestGitRefs failure in GitLab CI. I would change the test to use special ref by name, and *when* we change the behavior, we change the test in the same commit to expect it to fail.

Quoted 3 lines

Ideally there should also be a check for getting the parent of the special ref
by sha1, but I think that actually does *not* work now.

I expect it to not work. But I didn't tested recently.

Regards,
Ricardo

#98 lore raw

PATCH v5 9/10 testing/tests/download: test git special ref

Quoted 11 lines · Ricardo Martincoski

Hello,

Quoted 7 lines · Arnout Vandecappelle
Quoted 5 lines · Ricardo Martincoski

From: Ricardo Martincoski <[email protected]>

Add a special ref to the static repo and check on the git refs test case the download of a git package:
- with the name of a special ref as version;

Quoted 5 lines
Quoted 3 lines

We might remove support for getting special refs by name, but keep on
supporting getting special refs by sha1. So I've changed the test into getting it by sha1 instead.

OK... but it depends on a recent git version to work.

Ah, darn, I forgot that...

Quoted 5 lines

The same does not occur when getting special refs by name.

When using recent versions of git the download of the sha1 of a special ref works because of this line in support/download/git:
_git fetch origin "'${cset}:${cset}'"

So, IIRC, Yann had an idea for how to rewrite this in a way that we would (1)
no longer support branch names and (2) we would support hashes that refer to special refs or ancestors of special refs. But I don't remember what it was. Maybe it was

_git fetch origin '+refs/*:refs/remotes/origin/*' '+refs/tags/*:refs/tags/*'

That way (I think) a branch name would no longer be usable (because when doing the checkout the origin/ bit would be missing), but a tag name still would be usable.

Quoted 8 lines

I tested using:
git version 2.17.1

The git version installed in our docker image does not support it.

I am not sure what is the best fix for the TestGitRefs failure in GitLab CI. I would change the test to use special ref by name, and *when* we change the behavior, we change the test in the same commit to expect it to fail.

Or maybe just remove the test for now.

Regards,
Arnout

Quoted 10 lines
Quoted 4 lines

Ideally there should also be a check for getting the parent of the special ref
by sha1, but I think that actually does *not* work now.

I expect it to not work. But I didn't tested recently.

Regards,
Ricardo

#99 lore raw

PATCH v5 9/10 testing/tests/download: test git special ref

Hello,

Quoted 30 lines · Arnout Vandecappelle
Quoted 10 lines · Ricardo Martincoski
Quoted 7 lines · Arnout Vandecappelle
Quoted 5 lines · Ricardo Martincoski

From: Ricardo Martincoski <[email protected]>

Add a special ref to the static repo and check on the git refs test case the download of a git package:
- with the name of a special ref as version;

Quoted 5 lines
Quoted 3 lines

We might remove support for getting special refs by name, but keep on
supporting getting special refs by sha1. So I've changed the test into getting it by sha1 instead.

OK... but it depends on a recent git version to work.

Ah, darn, I forgot that...

Quoted 5 lines

The same does not occur when getting special refs by name.

When using recent versions of git the download of the sha1 of a special ref works because of this line in support/download/git:
_git fetch origin "'${cset}:${cset}'"

So, IIRC, Yann had an idea for how to rewrite this in a way that we would (1)
no longer support branch names and (2) we would support hashes that refer to special refs or ancestors of special refs. But I don't remember what it was. Maybe it was

Quoted 5 lines

_git fetch origin '+refs/*:refs/remotes/origin/*' '+refs/tags/*:refs/tags/*'

That way (I think) a branch name would no longer be usable (because when doing the checkout the origin/ bit would be missing), but a tag name still would be usable.

I didn't tested it but I like the idea.

Quoted 11 lines
Quoted 9 lines

I tested using:
git version 2.17.1

The git version installed in our docker image does not support it.

I am not sure what is the best fix for the TestGitRefs failure in GitLab CI. I would change the test to use special ref by name, and *when* we change the behavior, we change the test in the same commit to expect it to fail.

Or maybe just remove the test for now.

OK. When new code is added to support hashes that refer to special refs the test can be added again.

Regards,
Ricardo

#100 lore raw

PATCH v5 10/10 support/testing: test extra download with site method git

In reply to Ricardo Martincoski

Add one test case to ensure the use of extra download works for git packages:
- an extra download is correctly downloaded using wget after the main download finished using the git backend;
- when the main download using the git backend fails, the build fails;
- when the extra download using wget fails, the build fails.

Reuse the commit in the static repo.
Add required infra and fixture:
- a HttpServer class, that starts a http server in the host machine to emulate a remote http server under the control of the test;
- a file to be served by the http server;
- a br2-external (git-extra-download) with one package for each part of the test case.
Since the HttpServer is similar to GitRemote, extract the commonalities to a new base class called Server.

Cc Luca Ceresoli <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Yann E. MORIN <[email protected]>

---
Changes v4 -> v5:
- no changes

Changes v3 -> v4:
- new patch

I use docker only with the Buildroot image for the test infra. Sorry, I give up for now on trying to run apache on the docker image.

As a midground solution between:
- adding more files to http://autobuild.buildroot.net/artefacts/ that won't scale well if we need different files for different test cases;
- installing a full web server to the docker image auto-loading as root when the container is started as user and allowing the test infra to copy files to the served path as user;
and inspired by this article:
https://www.pcsuggest.com/best-lightweight-web-server-linux/
I created this patch using SimpleHTTPServer being started by the test infra as a user.
I know it is weird to use pexpect to fork a server written in Python, but it has these advantages:
- theoretically replacing SimpleHTTPServer with a similar solution (say the busybox httpd) in the future would require only small changes to the test infra code;
- SimpleHTTPServer is already on the docker image, busybox isn't, so no need to change the Dockerfile, generate the image, publish it and update the docker image tag on .gitlab-ci.yml;
- the log file generated by the server is handled in the same way as other log files generated by the test infra;

---
 .gitlab-ci.yml                                |  1 +
 support/testing/infra/server.py               | 39 +++++++++++++++++
 .../br2-external/git-extra-download/Config.in |  0
 .../git-extra-download/external.desc          |  1 +
 .../git-extra-download/external.mk            |  6 +++
 .../package/extra-fails/extra-fails.hash      |  3 ++
 .../package/extra-fails/extra-fails.mk        | 12 ++++++
 .../package/main-fails/main-fails.hash        |  3 ++
 .../package/main-fails/main-fails.mk          | 12 ++++++
 .../git-extra-download/package/ok/ok.hash     |  3 ++
 .../git-extra-download/package/ok/ok.mk       | 12 ++++++
 support/testing/tests/download/gitremote.py   | 43 +++----------------
 .../testing/tests/download/http-server/extra  |  1 +
 support/testing/tests/download/httpserver.py  | 14 ++++++
 support/testing/tests/download/test_git.py    | 21 +++++++++
 15 files changed, 135 insertions(+), 36 deletions(-)
 create mode 100644 support/testing/infra/server.py
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/Config.in
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/external.desc
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/external.mk
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/extra-fails/extra-fails.hash
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/extra-fails/extra-fails.mk
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/main-fails/main-fails.hash
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/main-fails/main-fails.mk
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/ok/ok.hash
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/ok/ok.mk
 create mode 100644 support/testing/tests/download/http-server/extra
 create mode 100644 support/testing/tests/download/httpserver.py
.gitlab-ci.yml +1 -0
index 61ccfdfd2d..53f7c72ac3 100644
@@ -263,6 +263,7 @@ tests.core.test_rootfs_overlay.TestRootfsOverlay: *runtime_test
 tests.core.test_timezone.TestGlibcAllTimezone: *runtime_test
 tests.core.test_timezone.TestGlibcNonDefaultLimitedTimezone: *runtime_test
 tests.core.test_timezone.TestNoTimezone: *runtime_test
+tests.download.test_git.TestGitExtraDownload: *runtime_test
 tests.download.test_git.TestGitHash: *runtime_test
 tests.download.test_git.TestGitRefs: *runtime_test
 tests.fs.test_ext.TestExt2: *runtime_test
support/testing/infra/server.py new file +39 -0
new file mode 100644
index 0000000000..0fbffdf846
@@ -0,0 +1,39 @@
+# subprocess does not kill the child daemon when a test case fails by raising
+# an exception. So use pexpect instead.
+import pexpect
+
+import infra
+
+
+class Server(object):
+    def __init__(self, builddir, serveddir, logtofile, daemon_cmd, port_arg, port_initial, port_last, good_msg, bad_msg):
+        """
+        Start a local server.
+
+        In order to support test cases in parallel, select the port the
+        server will listen to in runtime. Since there is no reliable way
+        to allocate the port prior to starting the server (another
+        process in the host machine can use the port between it is
+        selected from a list and it is really allocated to the server)
+        try to start the server in a port and in the case it is already
+        in use, try the next one in the allowed range.
+        """
+        self.name = self.__class__.__name__.lower()
+        self.daemon = None
+        self.port = None
+        self.logfile = infra.open_log_file(builddir, self.name, logtofile)
+
+        for port in range(port_initial, port_last + 1):
+            cmd = daemon_cmd + [port_arg.format(port=port)]
+            self.logfile.write("> starting {} with 'cd {} && {}'\n".format(self.name, serveddir, " ".join(cmd)))
+            self.daemon = pexpect.spawn(cmd[0], cmd[1:], logfile=self.logfile, cwd=serveddir)
+            ret = self.daemon.expect(good_msg + bad_msg)
+            if ret < len(good_msg):
+                self.port = port
+                return
+        raise SystemError("Could not find a free port to run {}".format(self.name))
+
+    def stop(self):
+        if self.daemon is None:
+            return
+        self.daemon.terminate(force=True)
support/testing/tests/download/br2-external/git-extra-download/Config.in new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
support/testing/tests/download/br2-external/git-extra-download/external.desc new file +1 -0
new file mode 100644
index 0000000000..6ebd5a534d
@@ -0,0 +1 @@
+name: GIT_EXTRA_DOWNLOAD
support/testing/tests/download/br2-external/git-extra-download/external.mk new file +6 -0
new file mode 100644
index 0000000000..c6080f571b
@@ -0,0 +1,6 @@
+include $(sort $(wildcard $(BR2_EXTERNAL_GIT_EXTRA_DOWNLOAD_PATH)/package/*/*.mk))
+
+# Get the git server port number from the test infra
+GITREMOTE_PORT_NUMBER ?= 9418
+# Get the http server port number from the test infra
+HTTP_SERVER_PORT_NUMBER ?= 8000
support/testing/tests/download/br2-external/git-extra-download/package/extra-fails/extra-fails.hash new file +3 -0
new file mode 100644
index 0000000000..89b1b1f682
@@ -0,0 +1,3 @@
+sha256  1e6bc73fabdcce8857361e36e3c812c4ee42d8ffa30d56492bc56f8fcad7eb90  extra-fails-a238b1dfcd825d47d834af3c5223417c8411d90d.tar.gz
+sha256  b9e68e1bea3e5b19ca6b2f98b73a54b73daafaa250484902e09982e07a12e733  notfound
+sha256  da68f54607d5f5644954096ce1597c006c5bb9f2497e07441bf064b81003ef8a  file
support/testing/tests/download/br2-external/git-extra-download/package/extra-fails/extra-fails.mk new file +12 -0
new file mode 100644
index 0000000000..dcf3ebe7a6
@@ -0,0 +1,12 @@
+################################################################################
+#
+# extra-fails
+#
+################################################################################
+
+EXTRA_FAILS_VERSION = a238b1dfcd825d47d834af3c5223417c8411d90d
+EXTRA_FAILS_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+EXTRA_FAILS_EXTRA_DOWNLOADS = http://localhost:$(HTTP_SERVER_PORT_NUMBER)/notfound
+EXTRA_FAILS_LICENSE_FILES = file
+
+$(eval $(generic-package))
support/testing/tests/download/br2-external/git-extra-download/package/main-fails/main-fails.hash new file +3 -0
new file mode 100644
index 0000000000..ad81243751
@@ -0,0 +1,3 @@
+sha256  cd6851ef519a83345e4547f376b33d6bbd622d4ccbb234af9997c43854c602de  main-fails-a238b1dfcd825d47d834af3c5223417c8411d90d.tar.gz
+sha256  b9e68e1bea3e5b19ca6b2f98b73a54b73daafaa250484902e09982e07a12e733  extra
+sha256  da68f54607d5f5644954096ce1597c006c5bb9f2497e07441bf064b81003ef8a  file
support/testing/tests/download/br2-external/git-extra-download/package/main-fails/main-fails.mk new file +12 -0
new file mode 100644
index 0000000000..022bb37cbb
@@ -0,0 +1,12 @@
+################################################################################
+#
+# main-fails
+#
+################################################################################
+
+MAIN_FAILS_VERSION = a238b1dfcd825d47d834af3c5223417c8411d90d
+MAIN_FAILS_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/notfound.git
+MAIN_FAILS_EXTRA_DOWNLOADS = http://localhost:$(HTTP_SERVER_PORT_NUMBER)/extra
+MAIN_FAILS_LICENSE_FILES = file
+
+$(eval $(generic-package))
support/testing/tests/download/br2-external/git-extra-download/package/ok/ok.hash new file +3 -0
new file mode 100644
index 0000000000..366940754b
@@ -0,0 +1,3 @@
+sha256  737b4fd21506dbaa34cedc93c53c1ebb1f950db2c7644572bb006ae9297961da  ok-a238b1dfcd825d47d834af3c5223417c8411d90d.tar.gz
+sha256  b9e68e1bea3e5b19ca6b2f98b73a54b73daafaa250484902e09982e07a12e733  extra
+sha256  da68f54607d5f5644954096ce1597c006c5bb9f2497e07441bf064b81003ef8a  file
support/testing/tests/download/br2-external/git-extra-download/package/ok/ok.mk new file +12 -0
new file mode 100644
index 0000000000..ce102cb0de
@@ -0,0 +1,12 @@
+################################################################################
+#
+# ok
+#
+################################################################################
+
+OK_VERSION = a238b1dfcd825d47d834af3c5223417c8411d90d
+OK_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+OK_EXTRA_DOWNLOADS = http://localhost:$(HTTP_SERVER_PORT_NUMBER)/extra
+OK_LICENSE_FILES = file
+
+$(eval $(generic-package))
support/testing/tests/download/gitremote.py +7 -36

51 lines not on this page · raw message

support/testing/tests/download/http-server/extra new file +1 -0
new file mode 100644
index 0000000000..8e27be7d61
@@ -0,0 +1 @@
+text
support/testing/tests/download/httpserver.py new file +14 -0
new file mode 100644
index 0000000000..9f4f947304
@@ -0,0 +1,14 @@
+from infra.server import Server
+
+HTTP_SERVER_PORT_INITIAL = 8000
+HTTP_SERVER_PORT_LAST = HTTP_SERVER_PORT_INITIAL + 99
+
+
+class HttpServer(Server):
+    def __init__(self, builddir, serveddir, logtofile):
+        daemon_cmd = ["python", "-m", "SimpleHTTPServer"]
+        super(HttpServer, self).__init__(
+            builddir, serveddir, logtofile,
+            daemon_cmd, "{port}", HTTP_SERVER_PORT_INITIAL, HTTP_SERVER_PORT_LAST,
+            ["Serving HTTP"],
+            ["Address already in use"])
support/testing/tests/download/test_git.py +21 -0
index 162c03623b..e0abc3fa65 100644
@@ -3,6 +3,7 @@ import shutil
 
 import infra
 from gitremote import GitRemote
+from httpserver import HttpServer
 
 
 class GitTestBase(infra.basetest.BRTest):
@@ -12,13 +13,19 @@ class GitTestBase(infra.basetest.BRTest):
         """
     gitremotedir = infra.filepath("tests/download/git-remote")
     gitremote = None
+    httpserverdir = None
+    httpserver = None
 
     def setUp(self):
         super(GitTestBase, self).setUp()
         self.gitremote = GitRemote(self.builddir, self.gitremotedir, self.logtofile)
+        if self.httpserverdir:
+            self.httpserver = HttpServer(self.builddir, self.httpserverdir, self.logtofile)
 
     def tearDown(self):
         self.show_msg("Cleaning up")
+        if self.httpserver:
+            self.httpserver.stop()
         if self.gitremote:
             self.gitremote.stop()
         if self.b and not self.keepbuilds:
@@ -42,11 +49,25 @@ class GitTestBase(infra.basetest.BRTest):
             shutil.rmtree(dl_dir)
         env = {"BR2_DL_DIR": dl_dir,
                "GITREMOTE_PORT_NUMBER": str(self.gitremote.port)}
+        if self.httpserver:
+            env["HTTP_SERVER_PORT_NUMBER"] = str(self.httpserver.port)
         self.b.build(["{}-dirclean".format(package),
                       "{}-legal-info".format(package)],
                      env)
 
 
+class TestGitExtraDownload(GitTestBase):
+    br2_external = [infra.filepath("tests/download/br2-external/git-extra-download")]
+    httpserverdir = infra.filepath("tests/download/http-server")
+
+    def test_run(self):
+        with self.assertRaises(SystemError):
+            self.check_download("extra-fails")
+        with self.assertRaises(SystemError):
+            self.check_download("main-fails")
+        self.check_download("ok")
+
+
 class TestGitHash(GitTestBase):
     br2_external = [infra.filepath("tests/download/br2-external/git-hash")]
 
Signature · 1 line
2.17.0
#101 lore raw

PATCH v5 10/10 support/testing: test extra download with site method git

Quoted 7 lines · Ricardo Martincoski

Add one test case to ensure the use of extra download works for git packages:
- an extra download is correctly downloaded using wget after the main download finished using the git backend;
- when the main download using the git backend fails, the build fails;
- when the extra download using wget fails, the build fails.

Quoted 9 lines

Reuse the commit in the static repo.
Add required infra and fixture:
- a HttpServer class, that starts a http server in the host machine to emulate a remote http server under the control of the test;
- a file to be served by the http server;
- a br2-external (git-extra-download) with one package for each part of the test case.
Since the HttpServer is similar to GitRemote, extract the commonalities to a new base class called Server.

This one is a little too complicated to just apply, so I'm going to leave it
here for the time being...

It should for sure be split into two patches, one adding the common Server
class to the infra, and the next one adding the http test.

Also I'd probably prefer first just adding an http download test case and only
later adding the extra downloads test.

However, I also don't really like the approach of launching our own http
server. I think it's much easier to rely on an existing one, perhaps using sources.buildroot.org. You could say that it makes it impossible to run the tests offline, but that's already the case for a most of them, because they download sources and kernel binaries etc.

So I've marked this one as Changes Requested in patchwork.

Regards,
Arnout

Quoted 367 lines
Cc Luca Ceresoli <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Yann E. MORIN <[email protected]>

---
Changes v4 -> v5:
- no changes

Changes v3 -> v4:
- new patch

I use docker only with the Buildroot image for the test infra. Sorry, I give up for now on trying to run apache on the docker image.

As a midground solution between:
- adding more files to http://autobuild.buildroot.net/artefacts/ that won't scale well if we need different files for different test cases;
- installing a full web server to the docker image auto-loading as root when the container is started as user and allowing the test infra to copy files to the served path as user;
and inspired by this article:
https://www.pcsuggest.com/best-lightweight-web-server-linux/
I created this patch using SimpleHTTPServer being started by the test infra as a user.
I know it is weird to use pexpect to fork a server written in Python, but it has these advantages:
- theoretically replacing SimpleHTTPServer with a similar solution (say the busybox httpd) in the future would require only small changes to the test infra code;
- SimpleHTTPServer is already on the docker image, busybox isn't, so no need to change the Dockerfile, generate the image, publish it and update the docker image tag on .gitlab-ci.yml;
- the log file generated by the server is handled in the same way as other log files generated by the test infra;

---
 .gitlab-ci.yml                                |  1 +
 support/testing/infra/server.py               | 39 +++++++++++++++++
 .../br2-external/git-extra-download/Config.in |  0
 .../git-extra-download/external.desc          |  1 +
 .../git-extra-download/external.mk            |  6 +++
 .../package/extra-fails/extra-fails.hash      |  3 ++
 .../package/extra-fails/extra-fails.mk        | 12 ++++++
 .../package/main-fails/main-fails.hash        |  3 ++
 .../package/main-fails/main-fails.mk          | 12 ++++++
 .../git-extra-download/package/ok/ok.hash     |  3 ++
 .../git-extra-download/package/ok/ok.mk       | 12 ++++++
 support/testing/tests/download/gitremote.py   | 43 +++----------------
 .../testing/tests/download/http-server/extra  |  1 +
 support/testing/tests/download/httpserver.py  | 14 ++++++
 support/testing/tests/download/test_git.py    | 21 +++++++++
 15 files changed, 135 insertions(+), 36 deletions(-)
 create mode 100644 support/testing/infra/server.py
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/Config.in
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/external.desc
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/external.mk
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/extra-fails/extra-fails.hash
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/extra-fails/extra-fails.mk
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/main-fails/main-fails.hash
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/main-fails/main-fails.mk
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/ok/ok.hash
 create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/ok/ok.mk
 create mode 100644 support/testing/tests/download/http-server/extra
 create mode 100644 support/testing/tests/download/httpserver.py
.gitlab-ci.yml +1 -0
index 61ccfdfd2d..53f7c72ac3 100644
@@ -263,6 +263,7 @@ tests.core.test_rootfs_overlay.TestRootfsOverlay: *runtime_test
 tests.core.test_timezone.TestGlibcAllTimezone: *runtime_test
 tests.core.test_timezone.TestGlibcNonDefaultLimitedTimezone: *runtime_test
 tests.core.test_timezone.TestNoTimezone: *runtime_test
+tests.download.test_git.TestGitExtraDownload: *runtime_test
 tests.download.test_git.TestGitHash: *runtime_test
 tests.download.test_git.TestGitRefs: *runtime_test
 tests.fs.test_ext.TestExt2: *runtime_test
support/testing/infra/server.py new file +39 -0
new file mode 100644
index 0000000000..0fbffdf846
@@ -0,0 +1,39 @@
+# subprocess does not kill the child daemon when a test case fails by raising
+# an exception. So use pexpect instead.
+import pexpect
+
+import infra
+
+
+class Server(object):
+    def __init__(self, builddir, serveddir, logtofile, daemon_cmd, port_arg, port_initial, port_last, good_msg, bad_msg):
+        """
+        Start a local server.
+
+        In order to support test cases in parallel, select the port the
+        server will listen to in runtime. Since there is no reliable way
+        to allocate the port prior to starting the server (another
+        process in the host machine can use the port between it is
+        selected from a list and it is really allocated to the server)
+        try to start the server in a port and in the case it is already
+        in use, try the next one in the allowed range.
+        """
+        self.name = self.__class__.__name__.lower()
+        self.daemon = None
+        self.port = None
+        self.logfile = infra.open_log_file(builddir, self.name, logtofile)
+
+        for port in range(port_initial, port_last + 1):
+            cmd = daemon_cmd + [port_arg.format(port=port)]
+            self.logfile.write("> starting {} with 'cd {} && {}'\n".format(self.name, serveddir, " ".join(cmd)))
+            self.daemon = pexpect.spawn(cmd[0], cmd[1:], logfile=self.logfile, cwd=serveddir)
+            ret = self.daemon.expect(good_msg + bad_msg)
+            if ret < len(good_msg):
+                self.port = port
+                return
+        raise SystemError("Could not find a free port to run {}".format(self.name))
+
+    def stop(self):
+        if self.daemon is None:
+            return
+        self.daemon.terminate(force=True)
support/testing/tests/download/br2-external/git-extra-download/Config.in new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
support/testing/tests/download/br2-external/git-extra-download/external.desc new file +1 -0
new file mode 100644
index 0000000000..6ebd5a534d
@@ -0,0 +1 @@
+name: GIT_EXTRA_DOWNLOAD
support/testing/tests/download/br2-external/git-extra-download/external.mk new file +6 -0
new file mode 100644
index 0000000000..c6080f571b
@@ -0,0 +1,6 @@
+include $(sort $(wildcard $(BR2_EXTERNAL_GIT_EXTRA_DOWNLOAD_PATH)/package/*/*.mk))
+
+# Get the git server port number from the test infra
+GITREMOTE_PORT_NUMBER ?= 9418
+# Get the http server port number from the test infra
+HTTP_SERVER_PORT_NUMBER ?= 8000
support/testing/tests/download/br2-external/git-extra-download/package/extra-fails/extra-fails.hash new file +3 -0
new file mode 100644
index 0000000000..89b1b1f682
@@ -0,0 +1,3 @@
+sha256  1e6bc73fabdcce8857361e36e3c812c4ee42d8ffa30d56492bc56f8fcad7eb90  extra-fails-a238b1dfcd825d47d834af3c5223417c8411d90d.tar.gz
+sha256  b9e68e1bea3e5b19ca6b2f98b73a54b73daafaa250484902e09982e07a12e733  notfound
+sha256  da68f54607d5f5644954096ce1597c006c5bb9f2497e07441bf064b81003ef8a  file
support/testing/tests/download/br2-external/git-extra-download/package/extra-fails/extra-fails.mk new file +12 -0
new file mode 100644
index 0000000000..dcf3ebe7a6
@@ -0,0 +1,12 @@
+################################################################################
+#
+# extra-fails
+#
+################################################################################
+
+EXTRA_FAILS_VERSION = a238b1dfcd825d47d834af3c5223417c8411d90d
+EXTRA_FAILS_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+EXTRA_FAILS_EXTRA_DOWNLOADS = http://localhost:$(HTTP_SERVER_PORT_NUMBER)/notfound
+EXTRA_FAILS_LICENSE_FILES = file
+
+$(eval $(generic-package))
support/testing/tests/download/br2-external/git-extra-download/package/main-fails/main-fails.hash new file +3 -0
new file mode 100644
index 0000000000..ad81243751
@@ -0,0 +1,3 @@
+sha256  cd6851ef519a83345e4547f376b33d6bbd622d4ccbb234af9997c43854c602de  main-fails-a238b1dfcd825d47d834af3c5223417c8411d90d.tar.gz
+sha256  b9e68e1bea3e5b19ca6b2f98b73a54b73daafaa250484902e09982e07a12e733  extra
+sha256  da68f54607d5f5644954096ce1597c006c5bb9f2497e07441bf064b81003ef8a  file
support/testing/tests/download/br2-external/git-extra-download/package/main-fails/main-fails.mk new file +12 -0
new file mode 100644
index 0000000000..022bb37cbb
@@ -0,0 +1,12 @@
+################################################################################
+#
+# main-fails
+#
+################################################################################
+
+MAIN_FAILS_VERSION = a238b1dfcd825d47d834af3c5223417c8411d90d
+MAIN_FAILS_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/notfound.git
+MAIN_FAILS_EXTRA_DOWNLOADS = http://localhost:$(HTTP_SERVER_PORT_NUMBER)/extra
+MAIN_FAILS_LICENSE_FILES = file
+
+$(eval $(generic-package))
support/testing/tests/download/br2-external/git-extra-download/package/ok/ok.hash new file +3 -0
new file mode 100644
index 0000000000..366940754b
@@ -0,0 +1,3 @@
+sha256  737b4fd21506dbaa34cedc93c53c1ebb1f950db2c7644572bb006ae9297961da  ok-a238b1dfcd825d47d834af3c5223417c8411d90d.tar.gz
+sha256  b9e68e1bea3e5b19ca6b2f98b73a54b73daafaa250484902e09982e07a12e733  extra
+sha256  da68f54607d5f5644954096ce1597c006c5bb9f2497e07441bf064b81003ef8a  file
support/testing/tests/download/br2-external/git-extra-download/package/ok/ok.mk new file +12 -0
new file mode 100644
index 0000000000..ce102cb0de
@@ -0,0 +1,12 @@
+################################################################################
+#
+# ok
+#
+################################################################################
+
+OK_VERSION = a238b1dfcd825d47d834af3c5223417c8411d90d
+OK_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git
+OK_EXTRA_DOWNLOADS = http://localhost:$(HTTP_SERVER_PORT_NUMBER)/extra
+OK_LICENSE_FILES = file
+
+$(eval $(generic-package))
support/testing/tests/download/gitremote.py +7 -36
index 60bc49fbf8..9766b19ce0 100644
@@ -1,44 +1,15 @@
-# subprocess does not kill the child daemon when a test case fails by raising
-# an exception. So use pexpect instead.
-import pexpect
-
-import infra
+from infra.server import Server
 
 GIT_REMOTE_PORT_INITIAL = 9418
 GIT_REMOTE_PORT_LAST = GIT_REMOTE_PORT_INITIAL + 99
 
 
-class GitRemote(object):
+class GitRemote(Server):
     def __init__(self, builddir, serveddir, logtofile):
-        """
-        Start a local git server.
-
-        In order to support test cases in parallel, select the port the
-        server will listen to in runtime. Since there is no reliable way
-        to allocate the port prior to starting the server (another
-        process in the host machine can use the port between it is
-        selected from a list and it is really allocated to the server)
-        try to start the server in a port and in the case it is already
-        in use, try the next one in the allowed range.
-        """
-        self.daemon = None
-        self.port = None
-        self.logfile = infra.open_log_file(builddir, "gitremote", logtofile)
-
         daemon_cmd = ["git", "daemon", "--reuseaddr", "--verbose", "--listen=localhost", "--export-all",
                       "--base-path={}".format(serveddir)]
-        for port in range(GIT_REMOTE_PORT_INITIAL, GIT_REMOTE_PORT_LAST + 1):
-            cmd = daemon_cmd + ["--port={port}".format(port=port)]
-            self.logfile.write("> starting git remote with '{}'\n".format(" ".join(cmd)))
-            self.daemon = pexpect.spawn(cmd[0], cmd[1:], logfile=self.logfile)
-            ret = self.daemon.expect(["Ready to rumble",
-                                      "Address already in use"])
-            if ret == 0:
-                self.port = port
-                return
-        raise SystemError("Could not find a free port to run git remote")
-
-    def stop(self):
-        if self.daemon is None:
-            return
-        self.daemon.terminate(force=True)
+        super(GitRemote, self).__init__(
+            builddir, serveddir, logtofile,
+            daemon_cmd, "--port={port}", GIT_REMOTE_PORT_INITIAL, GIT_REMOTE_PORT_LAST,
+            ["Ready to rumble"],
+            ["Address already in use"])
support/testing/tests/download/http-server/extra new file +1 -0
new file mode 100644
index 0000000000..8e27be7d61
@@ -0,0 +1 @@
+text
support/testing/tests/download/httpserver.py new file +14 -0
new file mode 100644
index 0000000000..9f4f947304
@@ -0,0 +1,14 @@
+from infra.server import Server
+
+HTTP_SERVER_PORT_INITIAL = 8000
+HTTP_SERVER_PORT_LAST = HTTP_SERVER_PORT_INITIAL + 99
+
+
+class HttpServer(Server):
+    def __init__(self, builddir, serveddir, logtofile):
+        daemon_cmd = ["python", "-m", "SimpleHTTPServer"]
+        super(HttpServer, self).__init__(
+            builddir, serveddir, logtofile,
+            daemon_cmd, "{port}", HTTP_SERVER_PORT_INITIAL, HTTP_SERVER_PORT_LAST,
+            ["Serving HTTP"],
+            ["Address already in use"])
support/testing/tests/download/test_git.py +21 -0
index 162c03623b..e0abc3fa65 100644
@@ -3,6 +3,7 @@ import shutil
 
 import infra
 from gitremote import GitRemote
+from httpserver import HttpServer
 
 
 class GitTestBase(infra.basetest.BRTest):
@@ -12,13 +13,19 @@ class GitTestBase(infra.basetest.BRTest):
         """
     gitremotedir = infra.filepath("tests/download/git-remote")
     gitremote = None
+    httpserverdir = None
+    httpserver = None
 
     def setUp(self):
         super(GitTestBase, self).setUp()
         self.gitremote = GitRemote(self.builddir, self.gitremotedir, self.logtofile)
+        if self.httpserverdir:
+            self.httpserver = HttpServer(self.builddir, self.httpserverdir, self.logtofile)
 
     def tearDown(self):
         self.show_msg("Cleaning up")
+        if self.httpserver:
+            self.httpserver.stop()
         if self.gitremote:
             self.gitremote.stop()
         if self.b and not self.keepbuilds:
@@ -42,11 +49,25 @@ class GitTestBase(infra.basetest.BRTest):
             shutil.rmtree(dl_dir)
         env = {"BR2_DL_DIR": dl_dir,
                "GITREMOTE_PORT_NUMBER": str(self.gitremote.port)}
+        if self.httpserver:
+            env["HTTP_SERVER_PORT_NUMBER"] = str(self.httpserver.port)
         self.b.build(["{}-dirclean".format(package),
                       "{}-legal-info".format(package)],
                      env)
 
 
+class TestGitExtraDownload(GitTestBase):
+    br2_external = [infra.filepath("tests/download/br2-external/git-extra-download")]
+    httpserverdir = infra.filepath("tests/download/http-server")
+
+    def test_run(self):
+        with self.assertRaises(SystemError):
+            self.check_download("extra-fails")
+        with self.assertRaises(SystemError):
+            self.check_download("main-fails")
+        self.check_download("ok")
+
+
 class TestGitHash(GitTestBase):
     br2_external = [infra.filepath("tests/download/br2-external/git-hash")]
 
#102 lore raw

PATCH 3/3 support/download/git: do not use git clone

In reply to Ricardo Martincoski

The logic of the script was completely rewritten based in some ideas discussed during the review of [1].

Always using git init + git fetch has these advantages:
- git fetch works with all kinds of refs, while git clone supports only branches and tags without the ref/heads/ and ref/tags/ prefixes;
- a shallow fetch can be done for the head of those refs.

The remote is first asked for its references using git ls-remote saving the output to a file.
This file is later on compared to the desired change set, determining if it is a branch, tag, special ref or sha1.
A concern that arrives from this method is that the remote can change between the git ls-remote and the git fetch, but in this case, once the script creates a shallow fetch, the checkout does what it is expected:
- for a named reference (branch, tag or special ref), the fetch and checkout are successful using the "new" sha1 from the remote;
- for a sha1 of a branch, the fetch fails (because the sha1 it is not anymore at branch head), falling back to a successful full fetch;
- for a sha1 of a commit pointed by a tag (that should not be changed but it can be changed) the same behavior of a sha1 of a branch takes place;
- for a sha1 only accessible by a special refs, the checkout fails, falling back to an unsuccessful checkout after full fetch. This can be caused only be a error from a developer.

An analytical solution is used instead of a single awk line. It makes each line of code simple: grep to check the entry is in the ls-remote output and awk to actually get the reference to use.

[1] http://patchwork.ozlabs.org/patch/681841/

---
 support/download/git | 91 ++++++++++++++++++++++++++++++++++------------------
 1 file changed, 60 insertions(+), 31 deletions(-)
support/download/git +60 -31
index f7eef15..1411d94 100755
@@ -39,43 +39,72 @@ _git() {
     eval ${GIT} "${@}"
 }
 
-# Try a shallow clone, since it is faster than a full clone - but that only
-# works if the version is a ref (tag or branch). Before trying to do a shallow
-# clone we check if ${cset} is in the list provided by git ls-remote. If not
-# we fall back on a full clone.
-#
-# Messages for the type of clone used are provided to ease debugging in case of
-# problems
-git_done=0
-if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then
-    printf "Doing shallow clone\n"
-    if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${repo}'" "'${basename}'"; then
-        git_done=1
-    else
-        printf "Shallow clone failed, falling back to doing a full clone\n"
+_git init ${verbose} "'${basename}'"
+
+pushd "${basename}" >/dev/null
+
+# Save the temporary file inside the .git directory that will be deleted after the checkout is done.
+tmpf=".git/ls-remote"
+_git ls-remote "'${repo}'" > "${tmpf}"
+
+do_a_shallow_fetch=0
+if grep "\<\(\|\(\|refs/\)\(heads\|tags\)/\)${cset}$" "${tmpf}" >/dev/null 2>&1; then
+    printf "The desired version is a named reference\n"
+    # Support branches and tags in the simplified form.
+    # Support branches and tags and special refs in the complete form refs/heads/branch.
+    # When the name is ambiguous, the branch will be selected (by git fetch or git clone).
+    ref="${cset}"
+    do_a_shallow_fetch=1
+elif grep "^${cset}" "${tmpf}" >/dev/null 2>&1; then
+    printf "The desired version is a sha1\n"
+    if [ ${#cset} -lt 40 -a "1" != "$(awk "/^${cset}/{print \$1|\"sort -u | wc -l\"}" "${tmpf}")" ]; then
+        printf "Ambiguous partial sha1\n"
+        awk "/^${cset}/{print \$1|\"sort -u | wc -l\"}" "${tmpf}"
+        exit 1
+    fi
+    # Accept full or unambiguous partial sha1. A sha1 can be referenced by many names.
+    # Prefer sha1 of commit pointed by a tag or sha1 of the tag itself,
+    # then sha1 pointed by any ref/*, and only then sha1 pointed by *HEAD.
+    ref="$(awk -F'[\t^]' "/^${cset}.*\trefs\/tags\//{ print \$2; exit }" "${tmpf}")"
+    if [ -z "${ref}" ]; then
+        ref="$(awk -F'[\t^]' "/^${cset}.*\trefs\//{ print \$2; exit }" "${tmpf}")"
+    fi
+    if [ -z "${ref}" ]; then
+        # sha1 is referenced by HEAD
+        ref="$(awk -F'[\t^]' "/^${cset}/{ print \$2; exit }" "${tmpf}")"
+    fi
+    if [ -n "${ref}" ]; then
+        printf "Fetching '%s' to get '%s'\n" "${ref}" "${cset}"
+        do_a_shallow_fetch=1
     fi
-fi
-if [ ${git_done} -eq 0 ]; then
-    printf "Doing full clone\n"
-    _git clone ${verbose} "${@}" "'${repo}'" "'${basename}'"
 fi
 
-pushd "${basename}" >/dev/null
+git_fetch_done=0
+if [ ${do_a_shallow_fetch} -eq 1 ]; then
+    printf "Doing shallow fetch\n"
+    if _git fetch ${verbose} "${@}" --depth 1 "'${repo}'" "'${ref}:refs/buildroot/${cset}'" 2>&1; then
+        git_fetch_done=1
+    else
+        # It catches the case the remote changed after ls-remote.
+        printf "Shallow fetch failed, falling back to doing a full fetch\n"
+    fi
+fi
 
-# Try to get the special refs exposed by some forges (pull-requests for
-# github, changes for gerrit...). There is no easy way to know whether
-# the cset the user passed us is such a special ref or a tag or a sha1
-# or whatever else. We'll eventually fail at checking out that cset,
-# below, if there is an issue anyway. Since most of the cset we're gonna
-# have to clone are not such special refs, consign the output to oblivion
-# so as not to alarm unsuspecting users, but still trace it as a warning.
-if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then
-    printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}"
+git_checkout_done=0
+if [ ${git_fetch_done} -eq 1 ]; then
+    if _git checkout -q "'refs/buildroot/${cset}'" 2>&1; then
+        git_checkout_done=1
+    else
+        printf "Checkout failed, falling back to doing a full fetch\n"
+    fi
 fi
 
-# Checkout the required changeset, so that we can update the required
-# submodules.
-_git checkout -q "'${cset}'"
+if [ ${git_checkout_done} -eq 0 ]; then
+    printf "Doing full fetch\n"
+    _git remote add origin "'${repo}'"
+    _git fetch ${verbose} "${@}" origin
+    _git checkout -q "'${cset}'"
+fi
 
 # Get date of commit to generate a reproducible archive.
 # %cD is RFC2822, so it's fully qualified, with TZ and all.
Signature · 1 line
2.9.3
#103 lore raw

PATCH 3/3 support/download/git: do not use git clone

Quoted 26 lines · Ricardo Martincoski

The logic of the script was completely rewritten based in some ideas discussed during the review of [1].

Always using git init + git fetch has these advantages:
- git fetch works with all kinds of refs, while git clone supports only branches and tags without the ref/heads/ and ref/tags/ prefixes;
- a shallow fetch can be done for the head of those refs.

The remote is first asked for its references using git ls-remote saving the output to a file.
This file is later on compared to the desired change set, determining if it is a branch, tag, special ref or sha1.
A concern that arrives from this method is that the remote can change between the git ls-remote and the git fetch, but in this case, once the script creates a shallow fetch, the checkout does what it is expected:
- for a named reference (branch, tag or special ref), the fetch and checkout are successful using the "new" sha1 from the remote;
- for a sha1 of a branch, the fetch fails (because the sha1 it is not anymore at branch head), falling back to a successful full fetch;
- for a sha1 of a commit pointed by a tag (that should not be changed but it can be changed) the same behavior of a sha1 of a branch takes place;
- for a sha1 only accessible by a special refs, the checkout fails, falling back to an unsuccessful checkout after full fetch. This can be caused only be a error from a developer.

Quoted 3 lines

An analytical solution is used instead of a single awk line. It makes each line of code simple: grep to check the entry is in the ls-remote output and awk to actually get the reference to use.

Excellent commit message!

Quoted 4 lines

[1] http://patchwork.ozlabs.org/patch/681841/

I have a bunch of remarks below, but nothing that should stop the acceptance of
this patch. So:

This is for next, obviously.

Quoted 3 lines
---
 support/download/git | 91 ++++++++++++++++++++++++++++++++++------------------
 1 file changed, 60 insertions(+), 31 deletions(-)

Note that, although lines are added here, it really is a simplification and
explicitation of the logic.

Quoted 18 lines
support/download/git +0 -9
index f7eef15..1411d94 100755
@@ -39,43 +39,72 @@ _git() {
     eval ${GIT} "${@}"
 }
 
-# Try a shallow clone, since it is faster than a full clone - but that only
-# works if the version is a ref (tag or branch). Before trying to do a shallow
-# clone we check if ${cset} is in the list provided by git ls-remote. If not
-# we fall back on a full clone.
-#
-# Messages for the type of clone used are provided to ease debugging in case of
-# problems
-git_done=0
-if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then

support/download/git _git() { · line 46

Quoted 10 lines
support/download/git +5 -5
-    printf "Doing shallow clone\n"
-    if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${repo}'" "'${basename}'"; then
-        git_done=1
-    else
-        printf "Shallow clone failed, falling back to doing a full clone\n"
+_git init ${verbose} "'${basename}'"
+
+pushd "${basename}" >/dev/null
+
+# Save the temporary file inside the .git directory that will be deleted after the checkout is done.

Don't forget to wrap at 80 columns.

support/download/git _git() { · line 51

Quoted 5 lines
support/download/git +5 -0
+tmpf=".git/ls-remote"
+_git ls-remote "'${repo}'" > "${tmpf}"
+
+do_a_shallow_fetch=0
+if grep "\<\(\|\(\|refs/\)\(heads\|tags\)/\)${cset}$" "${tmpf}" >/dev/null 2>&1; then

This is where grep -E becomes useful :-)

support/download/git _git() { · line 60

Quoted 9 lines
support/download/git +9 -0
+    printf "The desired version is a named reference\n"
+    # Support branches and tags in the simplified form.
+    # Support branches and tags and special refs in the complete form refs/heads/branch.
+    # When the name is ambiguous, the branch will be selected (by git fetch or git clone).
+    ref="${cset}"
+    do_a_shallow_fetch=1
+elif grep "^${cset}" "${tmpf}" >/dev/null 2>&1; then
+    printf "The desired version is a sha1\n"
+    if [ ${#cset} -lt 40 -a "1" != "$(awk "/^${cset}/{print \$1|\"sort -u | wc -l\"}" "${tmpf}")" ]; then

I think the check for -lt 40 is redundant.

I think all the logic here could be made a bit simpler by saving to a temporary
file again:

elif grep "^${cset}" "${tmpf}" > .git/matching_refs 2>/dev/null; then

    printf ...
    if "$(cut -f 1 .git/matching_refs | sort -u | wc -l) != 2; then
        ...

support/download/git _git() { · line 62

Quoted 2 lines
support/download/git +2 -0
+        printf "Ambiguous partial sha1\n"
+        awk "/^${cset}/{print \$1|\"sort -u | wc -l\"}" "${tmpf}"

Here we should print the full lines, so without sort -u and wc. Sort is still
useful though.

support/download/git _git() { · line 67

Quoted 5 lines
support/download/git +5 -0
+        exit 1
+    fi
+    # Accept full or unambiguous partial sha1. A sha1 can be referenced by many names.
+    # Prefer sha1 of commit pointed by a tag or sha1 of the tag itself,
+    # then sha1 pointed by any ref/*, and only then sha1 pointed by *HEAD.

I don't understand why this bit is needed. Any of the refs will do, right? It's
true that there is a risk that the ref gets updated between the ls-remote and the fetch, and that for tags the chance that it gets updated is smaller than for e.g. HEAD. But still, this is very much a corner case, and if it _does_ happen, we still fall back to full clone. So the only important thing is to avoid HEAD. Since ls-remotes already seems to sort the refs, which typically puts tags near the end, just take the last one and we're done:

    ref="$(cut -f 2 .git/matching_refs | tail -1)"
Quoted 11 lines
support/download/git +11 -0
+    ref="$(awk -F'[\t^]' "/^${cset}.*\trefs\/tags\//{ print \$2; exit }" "${tmpf}")"
+    if [ -z "${ref}" ]; then
+        ref="$(awk -F'[\t^]' "/^${cset}.*\trefs\//{ print \$2; exit }" "${tmpf}")"
+    fi
+    if [ -z "${ref}" ]; then
+        # sha1 is referenced by HEAD
+        ref="$(awk -F'[\t^]' "/^${cset}/{ print \$2; exit }" "${tmpf}")"
+    fi
+    if [ -n "${ref}" ]; then
+        printf "Fetching '%s' to get '%s'\n" "${ref}" "${cset}"
+        do_a_shallow_fetch=1

support/download/git _git() { · line 83

Quoted 10 lines
support/download/git +2 -5
     fi
-fi
-if [ ${git_done} -eq 0 ]; then
-    printf "Doing full clone\n"
-    _git clone ${verbose} "${@}" "'${repo}'" "'${basename}'"
 fi
 
-pushd "${basename}" >/dev/null
+git_fetch_done=0
+if [ ${do_a_shallow_fetch} -eq 1 ]; then

Actually, if [ "$ref" ] would be sufficient.

Quoted 6 lines
support/download/git +6 -0
+    printf "Doing shallow fetch\n"
+    if _git fetch ${verbose} "${@}" --depth 1 "'${repo}'" "'${ref}:refs/buildroot/${cset}'" 2>&1; then
+        git_fetch_done=1
+    else
+        # It catches the case the remote changed after ls-remote.
+        printf "Shallow fetch failed, falling back to doing a full fetch\n"

support/download/git _git() { · line 71

Quoted 10 lines
support/download/git +2 -7
+    fi
+fi
 
-# Try to get the special refs exposed by some forges (pull-requests for
-# github, changes for gerrit...). There is no easy way to know whether
-# the cset the user passed us is such a special ref or a tag or a sha1
-# or whatever else. We'll eventually fail at checking out that cset,
-# below, if there is an issue anyway. Since most of the cset we're gonna
-# have to clone are not such special refs, consign the output to oblivion
-# so as not to alarm unsuspecting users, but still trace it as a warning.

Yann will be happy that this gets removed :-)

support/download/git _git() { · line 95

Quoted 5 lines
support/download/git +3 -2
-if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then
-    printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}"
+git_checkout_done=0
+if [ ${git_fetch_done} -eq 1 ]; then
+    if _git checkout -q "'refs/buildroot/${cset}'" 2>&1; then

I'm not sure if I like this split. git_fetch_done gets set in only one place,
so we could just do the checkout directly there. That would also make it more explicit that this is a checkout specifically for shallow fetches, while the checkout below is specifically for full fetches.

support/download/git _git() { · line 105

Quoted 13 lines
support/download/git +8 -3
+        git_checkout_done=1
+    else
+        printf "Checkout failed, falling back to doing a full fetch\n"
+    fi
 fi
 
-# Checkout the required changeset, so that we can update the required
-# submodules.
-_git checkout -q "'${cset}'"
+if [ ${git_checkout_done} -eq 0 ]; then
+    printf "Doing full fetch\n"
+    _git remote add origin "'${repo}'"
+    _git fetch ${verbose} "${@}" origin

Why do we need to create a remote here?

support/download/git _git() { · line 107

Quoted 2 lines
support/download/git +2 -0
+    _git checkout -q "'${cset}'"
+fi

At a later stage, we could update the fetch with --submodules=yes/no depending
on -r option.

Another interesting feature would be to first try a shallow fetch of all tags
and branches with a depth of, say, 100. That is still way less than a full clone for large repositories, and has a good chance to still capture a sha1 that is not at a branch or tag head. So it would provide a nice middle ground for the cases for which we fail to do a shallow clone now. And if it does fail, nothing is wasted because the full clone can start from the part that was already downloaded.

Regards,
Arnout

Quoted 4 lines
support/download/git +0 -0
 # Get date of commit to generate a reproducible archive.
 # %cD is RFC2822, so it's fully qualified, with TZ and all.
 
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#104 lore raw

PATCH 3/3 support/download/git: do not use git clone

Arnout,

Sorry the delay.

On Sat, Nov 05, 2016 at 11:19 PM, Arnout Vandecappelle wrote:

[snip]

Your suggestions will simplify the script.

I will answer all questions in each review and resend the series in the next days. So I marked 1/3 and 3/3 as Changes Requested.

Regards,
Ricardo

#105 lore raw

PATCH 3/3 support/download/git: do not use git clone

In reply to Arnout Vandecappelle

Arnout,

On Sat, Nov 05, 2016 at 11:19 PM, Arnout Vandecappelle wrote:

[snip]

Quoted 1 line

Excellent commit message!

Thanks.

[snip]

Quoted 5 lines

I have a bunch of remarks below, but nothing that should stop the acceptance of
this patch. So:

I won't carry these for v2 because there will be a bunch of improvements based on your review ;)

Quoted 3 lines

This is for next, obviously.

[snip]

Quoted 3 lines
Quoted 1 line

+# Save the temporary file inside the .git directory that will be deleted after the checkout is done.

Don't forget to wrap at 80 columns.

Sorry. This line is way too long.

But just to be clear:
I think you mean to wrap to 80 as a rule of thumb (e.g. comments are easy to do) and not as a hard requirement.
Right?

I ask this because I think breaking some lines with commands would make it
harder to read, e.g.
if _git fetch ${verbose} "${@}" --depth 1 "'${repo}'" "'${ref}:refs/buildroot/${cset}'" 2>&1; then

[snip]

Quoted 3 lines
Quoted 1 line

+if grep "\<\(\|\(\|refs/\)\(heads\|tags\)/\)${cset}$" "${tmpf}" >/dev/null 2>&1; then

This is where grep -E becomes useful :-)

Thanks. Much nicer:
if grep -E "\<(|(|refs/)(heads|tags)/)${cset}$" "${tmpf}" >/dev/null 2>&1; then

[snip]

Quoted 5 lines

support/download/git _git() { · line 60

Quoted 3 lines
support/download/git +3 -0
+elif grep "^${cset}" "${tmpf}" >/dev/null 2>&1; then
+    printf "The desired version is a sha1\n"
+    if [ ${#cset} -lt 40 -a "1" != "$(awk "/^${cset}/{print \$1|\"sort -u | wc -l\"}" "${tmpf}")" ]; then

I think the check for -lt 40 is redundant.

You are right.
But this line will be removed in v2 because it is not needed and not wanted. It is not needed because unambiguous partial sha1 (of any commit) can be checked out after the full fetch (but without optimized download, of course). It is not wanted because, as I realized after your comment, the output of ls-remote is a subset of all sha1s and can't be used to test a sha1 is not ambiguous. We could end up checking out a wrong sha1 when we really need to let git checkout fail.
BTW, an example of partial sha1 is in the current master, see commit 956fcc2.

Quoted 8 lines

I think all the logic here could be made a bit simpler by saving to a temporary
file again:

elif grep "^${cset}" "${tmpf}" > .git/matching_refs 2>/dev/null; then

    printf ...
    if "$(cut -f 1 .git/matching_refs | sort -u | wc -l) != 2; then
        ...

Yes. Much simpler.
Actually this 'if' is removed in v2 (see above) but I will use temporary file and cut instead of awk in v2 (see below).

Quoted 7 lines

support/download/git _git() { · line 62

Quoted 4 lines
support/download/git +2 -0
+        printf "Ambiguous partial sha1\n"
+        awk "/^${cset}/{print \$1|\"sort -u | wc -l\"}" "${tmpf}"

Here we should print the full lines, so without sort -u and wc. Sort is still
useful though.

Indeed. But will be removed in v2.
If the partial sha1 is ambiguous, let git checkout fail. Git checkout fails even when the sha1 of the commit is ambiguous to a sha1 of a tree.
BTW, the best real example I have so far between 2 refs is b3766 from
https://github.com/vim/vim.git .

Quoted 15 lines

support/download/git _git() { · line 67

Quoted 6 lines
support/download/git +5 -0
+        exit 1
+    fi
+    # Accept full or unambiguous partial sha1. A sha1 can be referenced by many names.
+    # Prefer sha1 of commit pointed by a tag or sha1 of the tag itself,
+    # then sha1 pointed by any ref/*, and only then sha1 pointed by *HEAD.

I don't understand why this bit is needed. Any of the refs will do, right? It's
true that there is a risk that the ref gets updated between the ls-remote and the fetch, and that for tags the chance that it gets updated is smaller than for e.g. HEAD. But still, this is very much a corner case, and if it _does_ happen, we still fall back to full clone. So the only important thing is to avoid HEAD. Since ls-remotes already seems to sort the refs, which typically puts tags near the end, just take the last one and we're done:

Quoted 1 line
    ref="$(cut -f 2 .git/matching_refs | tail -1)"

Yes. It is simpler.

I just noticed I missed to explain the awk -F'[\t^]' in the comments (or in the commit log) :/ .
Its is needed because annotated tags have their own sha1 and the commit pointed by a tag has ^{} at the end of the ref name in the ls-remote output.
9395452b4aab7bc2475ef8935b4a4fb99d778d70 refs/tags/v4.8-rc6^{}

But after 'cut' I can use bash's 'Pattern substitution' ref=${ref/%"^{}"}
and, of course, add a nice comment for it.

[snip]

Quoted 3 lines
Quoted 1 line

+if [ ${do_a_shallow_fetch} -eq 1 ]; then

Actually, if [ "$ref" ] would be sufficient.

OK.

[snip]

Quoted 10 lines

support/download/git _git() { · line 95

Quoted 5 lines
support/download/git +3 -2
-if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then
-    printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}"
+git_checkout_done=0
+if [ ${git_fetch_done} -eq 1 ]; then
+    if _git checkout -q "'refs/buildroot/${cset}'" 2>&1; then

I'm not sure if I like this split. git_fetch_done gets set in only one place,
so we could just do the checkout directly there. That would also make it more explicit that this is a checkout specifically for shallow fetches, while the checkout below is specifically for full fetches.

OK.

[snip]

Quoted 4 lines

support/download/git _git() { · line 105

Quoted 2 lines
support/download/git +2 -0
+    _git remote add origin "'${repo}'"
+    _git fetch ${verbose} "${@}" origin

Why do we need to create a remote here?

Actually, we don't.
I missed your comment in http://patchwork.ozlabs.org/patch/681841/
"just use $repo everywhere"
I will do this in v2.

I think I can do this too:
"Do it in two patches: first convert the current situation to fetches, then add the shallow download of tag SHAs."

Quoted 6 lines

support/download/git _git() { · line 107

Quoted 3 lines
support/download/git +2 -0
+    _git checkout -q "'${cset}'"
+fi

At a later stage, we could update the fetch with --submodules=yes/no depending
on -r option.

In theory, OK. But because of a bug in git maybe it's better to keep the current 'git submodule update --init --recursive'.

git fetch --help # git 2.10.2
BUGS
Using --recurse-submodules can only fetch new commits in already checked out submodules right now. When e.g. upstream added a new submodule in the just fetched commits of the superproject the submodule itself can not be fetched, making it impossible to check out that submodule later without having to do a fetch again. This is expected to be fixed in a future Git version.

Quoted 8 lines

Another interesting feature would be to first try a shallow fetch of all tags
and branches with a depth of, say, 100. That is still way less than a full clone for large repositories, and has a good chance to still capture a sha1 that is not at a branch or tag head. So it would provide a nice middle ground for the cases for which we fail to do a shallow clone now. And if it does fail, nothing is wasted because the full clone can start from the part that was already downloaded.

Yes. It should be useful especially for hardware-specific kernel repos when using sha1 as custom repo version.
The buildroot policy is to use preferably a tag, and if a tag cannot be used, use the sha1 in a branch, right?
So I think we will have the best gain by fetching only from branches in this case.

A quick test shows good results: make warp7_defconfig ; make linux-source
without this feature: 1.69 GiB

                            5148191
with this feature (100):    945.59 MiB + 73.89 KiB
                            1997387 + 138 = 1997525
with this feature (10+100): 402.04 MiB + 470.10 MiB
                            347225 + 32 + 1650245 = 1997502

(I didn't dig to understand why the sum is not exactly the same)

Notice there is some extra processing in the server side ('Compressing objects') that takes some more seconds but I guess the bandwidth saved worth this.

I can add this patch (depth 100) at the end of the series as a RFC. This way people can comment/test/argue a different depth to use.

1) without this feature
Doing full clone
Cloning into 'linux-efaf36531fe7b1fc15a48033e5972825c91f9fc6'...
remote: Counting objects: 5148191, done.
remote: Compressing objects: 100% (395/395), done.
remote: Total 5148191 (delta 377), reused 183 (delta 183), pack-reused 5147613 Receiving objects: 100% (5148191/5148191), 1.69 GiB | 1.73 MiB/s, done.

2) with this feature, only for branches, depth 100
Doing shallow fetch of all branches
remote: Counting objects: 1997387, done.
remote: Compressing objects: 100% (453860/453860), done.
remote: Total 1997387 (delta 1605759), reused 1902796 (delta 1529294), pack-reused 0 Receiving objects: 100% (1997387/1997387), 945.59 MiB | 1.74 MiB/s, done.
...
remote: Counting objects: 138, done.
remote: Compressing objects: 100% (138/138), done.
remote: Total 138 (delta 0), reused 138 (delta 0), pack-reused 0 Receiving objects: 100% (138/138), 73.89 KiB | 0 bytes/s, done.

3) with this feature, only for branches, 1st step depth 10, 2nd step depth 100 Doing shallow fetch of all branches (10)
remote: Counting objects: 347225, done.
remote: Compressing objects: 100% (177196/177196), done.
remote: Total 347225 (delta 239120), reused 251422 (delta 167173), pack-reused 0 Receiving objects: 100% (347225/347225), 402.04 MiB | 1.56 MiB/s, done.
...
remote: Counting objects: 32, done.
remote: Compressing objects: 100% (32/32), done.
remote: Total 32 (delta 0), reused 32 (delta 0), pack-reused 0 Unpacking objects: 100% (32/32), done.
...
fatal: reference is not a tree: efaf36531fe7b1fc15a48033e5972825c91f9fc6 Doing shallow fetch of all branches (100)
remote: Counting objects: 1650245, done.
remote: Compressing objects: 100% (459302/459302), done.
remote: Total 1650245 (delta 1364443), reused 1452442 (delta 1179566), pack-reused 0 Receiving objects: 100% (1650245/1650245), 470.10 MiB | 1.68 MiB/s, done. Resolving deltas: 100% (1364443/1364443), completed with 38986 local objects. Checked out 'efaf36531fe7b1fc15a48033e5972825c91f9fc6'.

Regards,
Ricardo

#106 lore raw

PATCH 1/3 support/download/git: log checked out sha1

In reply to Ricardo Martincoski
Quoted 4 lines · Ricardo Martincoski

In preparation to have automated tests for this script, display the checked out sha1 in the build log.

Quoted 12 lines

---
Another solution would be to use an environment variable set by the test to print that info, e.g. if [ "1" == "${GIT_DOWNLOAD_AUTOTEST}" ]; then

---
 support/download/git | 6 ++++++
 1 file changed, 6 insertions(+)
support/download/git +0 -0
index 7921411..7c44c65 100755

support/download/git if [ ${recurse} -eq 1 ]; then · line 89

Quoted 5 lines
support/download/git +2 -0
@@ -85,6 +85,12 @@ if [ ${recurse} -eq 1 ]; then
     _git submodule update --init --recursive
 fi
 
+# Log the sha1. It can be used for automated tests and debug.
+if [ -z ${verbose} ]; then

It's quite weird to check that verbose is empty, but that's how the variable is
defined... Perhaps s/verbose/quiet/ would be a good idea...

Regards,
Arnout

Quoted 8 lines
support/download/git +4 -0
+    sha1=$(_git rev-parse HEAD)
+    printf "Checked out '%s'.\n" "${sha1}"
+fi
+
 # We do not need the .git dir; we keep other .git files, in case they
 # are the only files in their directory.
 rm -rf .git
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#107 lore raw

PATCH 1/3 support/download/git: log checked out sha1

Arnout,

Thank you for reviewing this series.

On Sat, Nov 05, 2016 at 07:50 PM, Arnout Vandecappelle wrote:

[snip]

Quoted 4 lines
Quoted 1 line

+if [ -z ${verbose} ]; then

It's quite weird to check that verbose is empty, but that's how the variable is
defined... Perhaps s/verbose/quiet/ would be a good idea...

OK, makes sense. I can send a followup patch.

Regards,
Ricardo