support/download/git: log checked out sha1
PATCH v5 7/10 testing/tests/download: test git submodules
In reply to Ricardo MartincoskiFrom: Ricardo Martincoski <[email protected]>
Add two submodules as static repos, add a branch to the main static repo and check on the git refs test case the download of a git package:
- repo with submodule but without support in the package;
- repo with recursive submodules with support in the package.
---
Layout of static repos with the series applied until this patch:
$ git -C support/testing/tests/download/git-remote/repo.git \ log --all --decorate --graph --oneline --decorate * 2fa37f6 (submodule) sub0-2 * a9dbc1e sub0-1 | * 68c197d (mybranch) branch2 | * 3174063 branch1 |/ * a238b1d (HEAD -> master) initial
$ git -C support/testing/tests/download/git-remote/refs-sub1.git \ log --all --decorate --graph --oneline --decorate * 1df823c (submodule) sub1-2 * f8001e5 sub1-1 * cb545fa (HEAD) initial
$ git -C support/testing/tests/download/git-remote/refs-sub2.git \ log --all --decorate --graph --oneline --decorate * e83f6f8 (submodule) sub2-2 * cd4d62f sub2-1 * 32d61ba (HEAD) initial
In this layout:
- sub0-1 adds refs-sub1.git as submodule to refs.git pointing to sub1-1 commit;
- sub1-1 adds refs-sub2.git as submodule to refs-sub1.git pointing to sub2-1 commit.
Changes v4 -> v5:
- no changes
Changes v3 -> v4:
- add hash for submodule enabled tarball;
Changes v2 -> v3:
- complete rewrite using a static repo;
Changes v1 -> v2:
- rewrite using new git test infra;
- this patch is part of series 1/3 of a new version of
http://patchwork.ozlabs.org/patch/690097/ --- .../git-submodule-disabled.hash | 2 ++ .../git-submodule-disabled.mk | 11 +++++++ .../git-submodule-enabled.hash | 4 +++ .../git-submodule-enabled.mk | 28 ++++++++++++++++++ .../git-remote/refs-sub1.git/.gitattributes | 1 + .../download/git-remote/refs-sub1.git/HEAD | 1 + .../download/git-remote/refs-sub1.git/config | 4 +++ .../07/cabc655213bdf7087d8dd50fda95124e935570 | Bin 0 -> 22 bytes .../1d/f823cb8a6d1866148ae50a8009762a9c4c777f | Bin 0 -> 180 bytes .../7d/52f458bdb0f9f5a4beb46fa82824421b8b988e | Bin 0 -> 65 bytes .../99/f2e3e1cb15f9b52fa29f66d380dda061d917ab | Bin 0 -> 49 bytes .../a1/30af1626bbadd64841b2cbcb6ac4ed0638ba1a | Bin 0 -> 122 bytes .../bc/c81ba6e5bd8bf52c95688ae8d3e697e131fa23 | Bin 0 -> 22 bytes .../cb/545facf77bbc5f24f95b6d503c338d10b7b717 | Bin 0 -> 151 bytes .../dd/130f6f4629514adaf2e03407f3ed9344eb6cd6 | Bin 0 -> 122 bytes .../e7/9c5e8f964493290a409888d5413a737e8e5dd5 | Bin 0 -> 23 bytes .../f8/001e5780100962a5e30a25cbc4c9609cfc7bf3 | Bin 0 -> 184 bytes .../refs-sub1.git/refs/heads/submodule | 1 + .../git-remote/refs-sub2.git/.gitattributes | 1 + .../download/git-remote/refs-sub2.git/HEAD | 1 + .../download/git-remote/refs-sub2.git/config | 4 +++ .../0a/846af45c3e455789435f49f80d70e86b65b9d7 | Bin 0 -> 22 bytes .../0b/32ebd8fc52cec991f18c94be980e85a8341585 | Bin 0 -> 22 bytes .../32/d61bae693af7879da63b89a60d3ae67f851e56 | Bin 0 -> 151 bytes .../3e/9b0a5198c64cea9c00f820433411e3b4d50c1c | Bin 0 -> 48 bytes .../85/8f666af75b7c0dfba6b8be7eac5f196e7a1221 | Bin 0 -> 49 bytes .../99/f2e3e1cb15f9b52fa29f66d380dda061d917ab | Bin 0 -> 49 bytes .../cd/4d62ff218ab7b4a04f5bfdf800ace087af3ceb | Bin 0 -> 184 bytes .../e7/9c5e8f964493290a409888d5413a737e8e5dd5 | Bin 0 -> 23 bytes .../e8/3f6f805bd016b90acafc8702c52d778eb57310 | Bin 0 -> 182 bytes .../refs-sub2.git/refs/heads/submodule | 1 + .../0b/d8ceb961c3b2b210f64a67d57f4b5cd669d343 | Bin 0 -> 22 bytes .../25/59d83bfe937fc0412d96ed664663c9e8a99055 | Bin 0 -> 22 bytes .../2f/a37f6885d7eb746df75eccaddbacf3ac82799d | Bin 0 -> 180 bytes .../8b/8a7e885a041da72b1ee9a47c5b9300b172a9e7 | Bin 0 -> 65 bytes .../a9/dbc1e23c45e8e1b88c0448763f54d714eb6f8f | Bin 0 -> 183 bytes .../d4/e2941d18a63535400476324ddeb7f40164be41 | Bin 0 -> 121 bytes .../fe/74231105841041d5f441e70399d37f0e600aa5 | Bin 0 -> 121 bytes .../git-remote/repo.git/refs/heads/submodule | 1 + support/testing/tests/download/test_git.py | 2 ++ 40 files changed, 62 insertions(+) create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-submodule-disabled/git-submodule-disabled.hash create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-submodule-disabled/git-submodule-disabled.mk create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.hash create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/.gitattributes create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/HEAD create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/config create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/07/cabc655213bdf7087d8dd50fda95124e935570 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/1d/f823cb8a6d1866148ae50a8009762a9c4c777f create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/7d/52f458bdb0f9f5a4beb46fa82824421b8b988e create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/99/f2e3e1cb15f9b52fa29f66d380dda061d917ab create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/a1/30af1626bbadd64841b2cbcb6ac4ed0638ba1a create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/bc/c81ba6e5bd8bf52c95688ae8d3e697e131fa23 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/cb/545facf77bbc5f24f95b6d503c338d10b7b717 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/dd/130f6f4629514adaf2e03407f3ed9344eb6cd6 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/e7/9c5e8f964493290a409888d5413a737e8e5dd5 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/objects/f8/001e5780100962a5e30a25cbc4c9609cfc7bf3 create mode 100644 support/testing/tests/download/git-remote/refs-sub1.git/refs/heads/submodule create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/.gitattributes create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/HEAD create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/config create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/0a/846af45c3e455789435f49f80d70e86b65b9d7 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/0b/32ebd8fc52cec991f18c94be980e85a8341585 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/32/d61bae693af7879da63b89a60d3ae67f851e56 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/3e/9b0a5198c64cea9c00f820433411e3b4d50c1c create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/85/8f666af75b7c0dfba6b8be7eac5f196e7a1221 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/99/f2e3e1cb15f9b52fa29f66d380dda061d917ab create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/cd/4d62ff218ab7b4a04f5bfdf800ace087af3ceb create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/e7/9c5e8f964493290a409888d5413a737e8e5dd5 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/objects/e8/3f6f805bd016b90acafc8702c52d778eb57310 create mode 100644 support/testing/tests/download/git-remote/refs-sub2.git/refs/heads/submodule create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/0b/d8ceb961c3b2b210f64a67d57f4b5cd669d343 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/25/59d83bfe937fc0412d96ed664663c9e8a99055 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/2f/a37f6885d7eb746df75eccaddbacf3ac82799d create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/8b/8a7e885a041da72b1ee9a47c5b9300b172a9e7 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/a9/dbc1e23c45e8e1b88c0448763f54d714eb6f8f create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/d4/e2941d18a63535400476324ddeb7f40164be41 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/fe/74231105841041d5f441e70399d37f0e600aa5 create mode 100644 support/testing/tests/download/git-remote/repo.git/refs/heads/submodule
support/testing/tests/download/br2-external/git-refs/package/git-submodule-disabled/git-submodule-disabled.hash new file +2 -0
@@ -0,0 +1,2 @@ +sha256 f9d46ff8a1a344c6c31fa4211220f3085c446abd31626232540703158276f22c git-submodule-disabled-a9dbc1e23c45e8e1b88c0448763f54d714eb6f8f.tar.gz +sha256 ba8b6ddc4726bfb6a05045ebfd8c43263c968ad1bc601bd46a25bc055008eddc file
support/testing/tests/download/br2-external/git-refs/package/git-submodule-disabled/git-submodule-disabled.mk new file +11 -0
@@ -0,0 +1,11 @@ +################################################################################ +# +# git-submodule-disabled +# +################################################################################ + +GIT_SUBMODULE_DISABLED_VERSION = a9dbc1e23c45e8e1b88c0448763f54d714eb6f8f +GIT_SUBMODULE_DISABLED_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +GIT_SUBMODULE_DISABLED_LICENSE_FILES = file + +$(eval $(generic-package))
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.hash new file +4 -0
@@ -0,0 +1,4 @@ +sha256 139a34c3c844c844dee74b6746418cfa75fbcc4205794ac8c0b3cd7d55a76792 git-submodule-enabled-a9dbc1e23c45e8e1b88c0448763f54d714eb6f8f.tar.gz +sha256 ba8b6ddc4726bfb6a05045ebfd8c43263c968ad1bc601bd46a25bc055008eddc file +sha256 5219dcb50c7753bcdd72bc68a7b48af33ac2e42c5c61de78a9dd81589c4d50b6 refs-sub1/file +sha256 b93c1fa9014d25a17fee36771d26ae023f043da656315ffe8947c30ad1ba141f refs-sub1/refs-sub2/file
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +28 -0
@@ -0,0 +1,28 @@ +################################################################################ +# +# git-submodule-enabled +# +################################################################################ + +GIT_SUBMODULE_ENABLED_VERSION = a9dbc1e23c45e8e1b88c0448763f54d714eb6f8f +GIT_SUBMODULE_ENABLED_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +GIT_SUBMODULE_ENABLED_GIT_SUBMODULES = YES +GIT_SUBMODULE_ENABLED_LICENSE_FILES = \ + file \ + refs-sub1/file \ + refs-sub1/refs-sub2/file + +# Some versions of git client fill the .git file for the second level submodule +# using the absolute path to the main .git directory, which in the case of the +# buildroot download helper is always different since it uses a temporary +# directory. This version of git have the issue: +# - 2.7.4 included in Ubuntu 16.04; +# The same does not occur using newer versions of git: +# - 2.11.0 included in Debian 9; +# - 2.14.2 latest at the time of writing; +# In order to allow running this test case locally in many systems, do not check +# for the hash of the tarball, but instead check the hash of each meaningful +# file included in the tarball. +BR_NO_CHECK_HASH_FOR += $(GIT_SUBMODULE_ENABLED_SOURCE) + +$(eval $(generic-package))
support/testing/tests/download/git-remote/refs-sub1.git/.gitattributes new file +1 -0
@@ -0,0 +1 @@ +objects/*/* binary
support/testing/tests/download/git-remote/refs-sub1.git/HEAD new file +1 -0
@@ -0,0 +1 @@ +cb545facf77bbc5f24f95b6d503c338d10b7b717
support/testing/tests/download/git-remote/refs-sub1.git/config new file +4 -0
@@ -0,0 +1,4 @@ +[core] + repositoryformatversion = 0 + filemode = true + bare = true
support/testing/tests/download/git-remote/refs-sub1.git/objects/07/cabc655213bdf7087d8dd50fda95124e935570 new file binary +0 -0
support/testing/tests/download/git-remote/refs-sub1.git/objects/1d/f823cb8a6d1866148ae50a8009762a9c4c777f new file binary +0 -0
iLYL_l{*)8*lT at dLIH$Ld{<)Z=(YXjRlEVZfTHP_gG8-
literal 0
HcmV?d00001
support/testing/tests/download/git-remote/refs-sub1.git/objects/7d/52f458bdb0f9f5a4beb46fa82824421b8b988e new file binary +0 -0
support/testing/tests/download/git-remote/refs-sub1.git/objects/99/f2e3e1cb15f9b52fa29f66d380dda061d917ab new file binary +0 -0
support/testing/tests/download/git-remote/refs-sub1.git/objects/a1/30af1626bbadd64841b2cbcb6ac4ed0638ba1a new file binary +0 -0
zq2eUnJ);k*EG;u9m4W@#p41 at Wz27-%d$01}nkwWsIkdpg%m4@!ic-^xb&E at rj2O=P cCjD3J+P-ChfArrU3~L^=ueW&(03qxzkxyhgmjD0&
literal 0
HcmV?d00001
support/testing/tests/download/git-remote/refs-sub1.git/objects/bc/c81ba6e5bd8bf52c95688ae8d3e697e131fa23 new file binary +0 -0
support/testing/tests/download/git-remote/refs-sub1.git/objects/cb/545facf77bbc5f24f95b6d503c338d10b7b717 new file binary +0 -0
support/testing/tests/download/git-remote/refs-sub1.git/objects/dd/130f6f4629514adaf2e03407f3ed9344eb6cd6 new file binary +0 -0
c-=zPFUE8-T at Q?ocgJI2s_VqTe0l at My>J95W4gdfE
literal 0
HcmV?d00001
support/testing/tests/download/git-remote/refs-sub1.git/objects/e7/9c5e8f964493290a409888d5413a737e8e5dd5 new file binary +0 -0
support/testing/tests/download/git-remote/refs-sub1.git/objects/f8/001e5780100962a5e30a25cbc4c9609cfc7bf3 new file binary +0 -0
m3s?Yk)7$hc|CCdgE1AV8b*ksR at X|#V`g#7#tNI184{X$IS6W&C
literal 0
HcmV?d00001
support/testing/tests/download/git-remote/refs-sub1.git/refs/heads/submodule new file +1 -0
@@ -0,0 +1 @@ +1df823cb8a6d1866148ae50a8009762a9c4c777f
support/testing/tests/download/git-remote/refs-sub2.git/.gitattributes new file +1 -0
@@ -0,0 +1 @@ +objects/*/* binary
support/testing/tests/download/git-remote/refs-sub2.git/HEAD new file +1 -0
@@ -0,0 +1 @@ +32d61bae693af7879da63b89a60d3ae67f851e56
support/testing/tests/download/git-remote/refs-sub2.git/config new file +4 -0
@@ -0,0 +1,4 @@ +[core] + repositoryformatversion = 0 + filemode = true + bare = true
support/testing/tests/download/git-remote/refs-sub2.git/objects/0a/846af45c3e455789435f49f80d70e86b65b9d7 new file binary +0 -0
support/testing/tests/download/git-remote/refs-sub2.git/objects/0b/32ebd8fc52cec991f18c94be980e85a8341585 new file binary +0 -0
support/testing/tests/download/git-remote/refs-sub2.git/objects/32/d61bae693af7879da63b89a60d3ae67f851e56 new file binary +0 -0
zL;YZ%zj&H6 at zVQ}k03QpsEr&gbf^%;rrSh|f6577xiKw5TzOt&=t@eRd+mRj6>nh7 FU at NX0KJNek
literal 0
HcmV?d00001
support/testing/tests/download/git-remote/refs-sub2.git/objects/3e/9b0a5198c64cea9c00f820433411e3b4d50c1c new file binary +0 -0
support/testing/tests/download/git-remote/refs-sub2.git/objects/85/8f666af75b7c0dfba6b8be7eac5f196e7a1221 new file binary +0 -0
support/testing/tests/download/git-remote/refs-sub2.git/objects/99/f2e3e1cb15f9b52fa29f66d380dda061d917ab new file binary +0 -0
support/testing/tests/download/git-remote/refs-sub2.git/objects/cd/4d62ff218ab7b4a04f5bfdf800ace087af3ceb new file binary +0 -0
zeR?YT87lKJ&evQHHQdjRq*5tF$Mz2eg>npQHpgm$pR at 75A3{t50g_7*#(Vx at k>bP zbo2ooB)#*Jy;W?Tz~{KNF5e1aPffmJA7!Ps-HSZ+A3WAyyj&}mk0Z5umdb($>r7bW m1T%oxb&nqWPj_N_#(7dgyqD|BX=*u7@$=;t>O#gcx)*r3s*J(
literal 0
HcmV?d00001
support/testing/tests/download/git-remote/refs-sub2.git/objects/e7/9c5e8f964493290a409888d5413a737e8e5dd5 new file binary +0 -0
support/testing/tests/download/git-remote/refs-sub2.git/objects/e8/3f6f805bd016b90acafc8702c52d778eb57310 new file binary +0 -0
k2QXK?PEUDI&YUmo7jMj0J#12zuDbB=%RR5=4+?N at DHiEdA^-pY
literal 0
HcmV?d00001
support/testing/tests/download/git-remote/refs-sub2.git/refs/heads/submodule new file +1 -0
@@ -0,0 +1 @@ +e83f6f805bd016b90acafc8702c52d778eb57310
support/testing/tests/download/git-remote/repo.git/objects/0b/d8ceb961c3b2b210f64a67d57f4b5cd669d343 new file binary +0 -0
support/testing/tests/download/git-remote/repo.git/objects/25/59d83bfe937fc0412d96ed664663c9e8a99055 new file binary +0 -0
support/testing/tests/download/git-remote/repo.git/objects/2f/a37f6885d7eb746df75eccaddbacf3ac82799d new file binary +0 -0
zoEleBZ(1ed0ZDFA1Yzu;w9*KG<DevlKyHi(Vn9bI*wa>;8n at fvydbGOYBD68F&+ at j z>O_rU6i9A}CNjIUTdDk<yse?|ldUaHzRdT;_w at t&{DsGBw&~IPk`G*HFrddZ%R7(& iY^CdTqd(=?yg2A2W0!QDD0C&p&b@ZOys|fCz-zg3O;?Bj
literal 0
HcmV?d00001
support/testing/tests/download/git-remote/repo.git/objects/8b/8a7e885a041da72b1ee9a47c5b9300b172a9e7 new file binary +0 -0
support/testing/tests/download/git-remote/repo.git/objects/a9/dbc1e23c45e8e1b88c0448763f54d714eb6f8f new file binary +0 -0
z*+yrKCG>Htpv)kOZ{w$R{aqq=Y5Kr at lr^^Nx#*Anz+HXf*Qs*3>|(2*8XW|19*ox$ l7yz~DHa*fm<<#}Iz=TR2%V|wAwVbErzJKLa{Q<P{YY&#BUEBZw
literal 0
HcmV?d00001
support/testing/tests/download/git-remote/repo.git/objects/d4/e2941d18a63535400476324ddeb7f40164be41 new file binary +0 -0
support/testing/tests/download/git-remote/repo.git/objects/fe/74231105841041d5f441e70399d37f0e600aa5 new file binary +0 -0
support/testing/tests/download/git-remote/repo.git/refs/heads/submodule new file +1 -0
@@ -0,0 +1 @@ +2fa37f6885d7eb746df75eccaddbacf3ac82799d
support/testing/tests/download/test_git.py +2 -0
@@ -69,3 +69,5 @@ class TestGitRefs(GitTestBase): self.check_download("git-partial-sha1-reachable-by-branch") self.check_download("git-sha1-branch-head") self.check_download("git-sha1-reachable-by-branch") + self.check_download("git-submodule-disabled") + self.check_download("git-submodule-enabled")
Signature · 1 line
2.17.0
PATCH v5 7/10 testing/tests/download: test git submodules
Quoted 15 lines · Ricardo Martincoski
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +15
-0+################################################################################ +# +# git-submodule-enabled +# +################################################################################ + +GIT_SUBMODULE_ENABLED_VERSION = a9dbc1e23c45e8e1b88c0448763f54d714eb6f8f +GIT_SUBMODULE_ENABLED_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +GIT_SUBMODULE_ENABLED_GIT_SUBMODULES = YES +GIT_SUBMODULE_ENABLED_LICENSE_FILES = \ + file \ + refs-sub1/file \ + refs-sub1/refs-sub2/file + +# Some versions of git client fill the .git file for the second level submodule
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk line 25
Quoted 10 lines
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +10
-0+# using the absolute path to the main .git directory, which in the case of the +# buildroot download helper is always different since it uses a temporary +# directory. This version of git have the issue: +# - 2.7.4 included in Ubuntu 16.04; +# The same does not occur using newer versions of git: +# - 2.11.0 included in Debian 9; +# - 2.14.2 latest at the time of writing; +# In order to allow running this test case locally in many systems, do not check +# for the hash of the tarball, but instead check the hash of each meaningful +# file included in the tarball.
This is a bug! Funny that we never hit it in the autobuilders, don't we have an
Ubuntu 16.04 autobuilder and don't we have packages with submodules and a hash?
Anyway, I think maybe we should fix the bug first... Ideally there should be a
test case for it as well, but that's difficult since it depends on the installed git version.
Yann, how about if we change the find command in the git helper to
find . -name .git -prune -o -print
?
Regards,
Arnout
Quoted 3 lines
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +3
-0+BR_NO_CHECK_HASH_FOR += $(GIT_SUBMODULE_ENABLED_SOURCE) + +$(eval $(generic-package))
PATCH v5 7/10 testing/tests/download: test git submodules
Quoted 22 lines · Arnout Vandecappelle
Quoted 19 lines · Ricardo Martincoski
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +19
-0+################################################################################ +# +# git-submodule-enabled +# +################################################################################ + +GIT_SUBMODULE_ENABLED_VERSION = a9dbc1e23c45e8e1b88c0448763f54d714eb6f8f +GIT_SUBMODULE_ENABLED_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +GIT_SUBMODULE_ENABLED_GIT_SUBMODULES = YES +GIT_SUBMODULE_ENABLED_LICENSE_FILES = \ + file \ + refs-sub1/file \ + refs-sub1/refs-sub2/file + +# Some versions of git client fill the .git file for the second level submodule +# using the absolute path to the main .git directory, which in the case of the +# buildroot download helper is always different since it uses a temporary +# directory. This version of git have the issue: +# - 2.7.4 included in Ubuntu 16.04;
Quoted 8 lines
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk line 25
Quoted 6 lines
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +6
-0+# The same does not occur using newer versions of git: +# - 2.11.0 included in Debian 9; +# - 2.14.2 latest at the time of writing; +# In order to allow running this test case locally in many systems, do not check +# for the hash of the tarball, but instead check the hash of each meaningful +# file included in the tarball.This is a bug!
So, I've removed this workaround from the test and pushed. It will most likely
fail in gitlab because our container is based on Ubuntu 16.04. But thats exactly what we want, because it is a bug.
Regards,
Arnout
Quoted 19 lines
Funny that we never hit it in the autobuilders, don't we have an Ubuntu 16.04 autobuilder and don't we have packages with submodules and a hash?
Anyway, I think maybe we should fix the bug first... Ideally there should be a
test case for it as well, but that's difficult since it depends on the installed git version.Yann, how about if we change the find command in the git helper to
find . -name .git -prune -o -print
?
Regards,
ArnoutQuoted 3 lines
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +3
-0+BR_NO_CHECK_HASH_FOR += $(GIT_SUBMODULE_ENABLED_SOURCE) + +$(eval $(generic-package))
PATCH v5 7/10 testing/tests/download: test git submodules
In reply to Arnout VandecappelleArnout, All,
On 2019-02-05 11:03 +0100, Arnout Vandecappelle spake thusly:
Quoted 30 lines · Ricardo Martincoski
Quoted 15 lines
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +15
-0+################################################################################ +# +# git-submodule-enabled +# +################################################################################ + +GIT_SUBMODULE_ENABLED_VERSION = a9dbc1e23c45e8e1b88c0448763f54d714eb6f8f +GIT_SUBMODULE_ENABLED_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +GIT_SUBMODULE_ENABLED_GIT_SUBMODULES = YES +GIT_SUBMODULE_ENABLED_LICENSE_FILES = \ + file \ + refs-sub1/file \ + refs-sub1/refs-sub2/file + +# Some versions of git client fill the .git file for the second level submodulesupport/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk line 25
Quoted 10 lines
support/testing/tests/download/br2-external/git-refs/package/git-submodule-enabled/git-submodule-enabled.mk new file +10
-0+# using the absolute path to the main .git directory, which in the case of the +# buildroot download helper is always different since it uses a temporary +# directory. This version of git have the issue: +# - 2.7.4 included in Ubuntu 16.04; +# The same does not occur using newer versions of git: +# - 2.11.0 included in Debian 9; +# - 2.14.2 latest at the time of writing; +# In order to allow running this test case locally in many systems, do not check +# for the hash of the tarball, but instead check the hash of each meaningful +# file included in the tarball.This is a bug! Funny that we never hit it in the autobuilders, don't we have an
Ubuntu 16.04 autobuilder and don't we have packages with submodules and a hash?
Quoted 7 lines
Anyway, I think maybe we should fix the bug first... Ideally there should be a
test case for it as well, but that's difficult since it depends on the installed git version.Yann, how about if we change the find command in the git helper to
find . -name .git -prune -o -print
?
Yeah, I too consider it a bug, and this should be fixed. I'll tackle this as part of my git-backend rework in the upcoming days.
Sorry I did not reply to the mail earlier, with all the live discussion we had yesterday it just slipped my mind...
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v5 8/10 testing/tests/download: test git tag
In reply to Ricardo MartincoskiFrom: Ricardo Martincoski <[email protected]>
Add a tag to the static repo and check on the git refs test case the download of a git package:
- with the name of a tag as version;
- with the sha1 of a tag itself as version;
- with the partial sha1 of a tag itself as version;
- with the sha1 of a commit pointed by a tag as version;
- with the partial sha1 of a commit pointed by a tag as version;
- with the sha1 of a commit reachable only by a tag as version;
- with the partial sha1 of a commit reachable only by a tag as version.
---
Layout of static repo with the series applied until this patch:
$ git -C support/testing/tests/download/git-remote/repo.git \ log --all --decorate --graph --oneline --decorate * 2fa37f6 (submodule) sub0-2 * a9dbc1e sub0-1 | * 68c197d (mybranch) branch2 | * 3174063 branch1 |/ | * 516c9c5 (tag: mytag) tag2 | * 46bae5b tag1 |/ * a238b1d (HEAD -> master) initial
Changes v4 -> v5:
- no changes
Changes v1 -> v4:
- not included on v2 or v3;
- rewrite from v1 using new git test infra with static repo;
--- .../git-partial-sha1-reachable-by-tag.hash | 2 ++ .../git-partial-sha1-reachable-by-tag.mk | 11 +++++++++++ .../git-partial-sha1-tag-itself.hash | 2 ++ .../git-partial-sha1-tag-itself.mk | 11 +++++++++++ .../git-partial-sha1-tag-points-to.hash | 2 ++ .../git-partial-sha1-tag-points-to.mk | 11 +++++++++++ .../git-sha1-reachable-by-tag.hash | 2 ++ .../git-sha1-reachable-by-tag.mk | 11 +++++++++++ .../git-sha1-tag-itself/git-sha1-tag-itself.hash | 2 ++ .../git-sha1-tag-itself/git-sha1-tag-itself.mk | 11 +++++++++++ .../git-sha1-tag-points-to.hash | 2 ++ .../git-sha1-tag-points-to.mk | 11 +++++++++++ .../git-refs/package/git-tag/git-tag.hash | 2 ++ .../git-refs/package/git-tag/git-tag.mk | 11 +++++++++++ .../2b/0e0d98a49c97da6a618ab36337e2058eb733a2 | Bin 0 -> 137 bytes .../46/bae5b639e5a18e2cc4dc508f080d566baeff59 | Bin 0 -> 182 bytes .../51/6c9c5f64ec66534d4d069c2e408d9ae4dce023 | Bin 0 -> 182 bytes .../92/ef85be57d627f280d8ce3724452ac21c9a6452 | Bin 0 -> 20 bytes .../cf/0f4f85d7a1237e377a2d25b996518a877ea001 | Bin 0 -> 49 bytes .../e2/2695cbf976fed1f543ad7486a531c0af473482 | Bin 0 -> 49 bytes .../f6/476b879f65e956d7dedd5b08736369e9a24acc | Bin 0 -> 20 bytes .../download/git-remote/repo.git/refs/tags/mytag | 1 + support/testing/tests/download/test_git.py | 7 +++++++ 23 files changed, 99 insertions(+) create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-reachable-by-tag/git-partial-sha1-reachable-by-tag.hash create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-reachable-by-tag/git-partial-sha1-reachable-by-tag.mk create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-tag-itself/git-partial-sha1-tag-itself.hash create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-tag-itself/git-partial-sha1-tag-itself.mk create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-tag-points-to/git-partial-sha1-tag-points-to.hash create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-tag-points-to/git-partial-sha1-tag-points-to.mk create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-sha1-reachable-by-tag/git-sha1-reachable-by-tag.hash create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-sha1-reachable-by-tag/git-sha1-reachable-by-tag.mk create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-sha1-tag-itself/git-sha1-tag-itself.hash create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-sha1-tag-itself/git-sha1-tag-itself.mk create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-sha1-tag-points-to/git-sha1-tag-points-to.hash create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-sha1-tag-points-to/git-sha1-tag-points-to.mk create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-tag/git-tag.hash create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-tag/git-tag.mk create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/2b/0e0d98a49c97da6a618ab36337e2058eb733a2 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/46/bae5b639e5a18e2cc4dc508f080d566baeff59 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/51/6c9c5f64ec66534d4d069c2e408d9ae4dce023 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/92/ef85be57d627f280d8ce3724452ac21c9a6452 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/cf/0f4f85d7a1237e377a2d25b996518a877ea001 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/e2/2695cbf976fed1f543ad7486a531c0af473482 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/f6/476b879f65e956d7dedd5b08736369e9a24acc create mode 100644 support/testing/tests/download/git-remote/repo.git/refs/tags/mytag
support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-reachable-by-tag/git-partial-sha1-reachable-by-tag.hash new file +2 -0
@@ -0,0 +1,2 @@ +sha256 f2ef9772f13a9ef9a2c7cde0795e179defb12320d1747fc8d2408748ef5844c2 git-partial-sha1-reachable-by-tag-46bae5b639e5a18e2cc4.tar.gz +sha256 2de87d77a2f226813f2d9bda906e970e4195605cdba6680443c0c04d89c532b6 file
support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-reachable-by-tag/git-partial-sha1-reachable-by-tag.mk new file +11 -0
@@ -0,0 +1,11 @@ +################################################################################ +# +# git-partial-sha1-reachable-by-tag +# +################################################################################ + +GIT_PARTIAL_SHA1_REACHABLE_BY_TAG_VERSION = 46bae5b639e5a18e2cc4 +GIT_PARTIAL_SHA1_REACHABLE_BY_TAG_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +GIT_PARTIAL_SHA1_REACHABLE_BY_TAG_LICENSE_FILES = file + +$(eval $(generic-package))
support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-tag-itself/git-partial-sha1-tag-itself.hash new file +2 -0
@@ -0,0 +1,2 @@ +sha256 721143b41b8e56cfd9025833f1602e900a490627db2504e5b4907baa23e0019e git-partial-sha1-tag-itself-2b0e0d98a49c97da6a61.tar.gz +sha256 6de8772a0a58fa62e2b8c58d4dae55c9db7534ad3b3918ecc849a9008d58f081 file
support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-tag-itself/git-partial-sha1-tag-itself.mk new file +11 -0
@@ -0,0 +1,11 @@ +################################################################################ +# +# git-partial-sha1-tag-itself +# +################################################################################ + +GIT_PARTIAL_SHA1_TAG_ITSELF_VERSION = 2b0e0d98a49c97da6a61 +GIT_PARTIAL_SHA1_TAG_ITSELF_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +GIT_PARTIAL_SHA1_TAG_ITSELF_LICENSE_FILES = file + +$(eval $(generic-package))
support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-tag-points-to/git-partial-sha1-tag-points-to.hash new file +2 -0
@@ -0,0 +1,2 @@ +sha256 0fbf7fe935f962ceaafcf8e0ffd25dd2a83752c3f0fd055a942a53f8c9235fa7 git-partial-sha1-tag-points-to-516c9c5f64ec66534d4d.tar.gz +sha256 6de8772a0a58fa62e2b8c58d4dae55c9db7534ad3b3918ecc849a9008d58f081 file
support/testing/tests/download/br2-external/git-refs/package/git-partial-sha1-tag-points-to/git-partial-sha1-tag-points-to.mk new file +11 -0
@@ -0,0 +1,11 @@ +################################################################################ +# +# git-partial-sha1-tag-points-to +# +################################################################################ + +GIT_PARTIAL_SHA1_TAG_POINTS_TO_VERSION = 516c9c5f64ec66534d4d +GIT_PARTIAL_SHA1_TAG_POINTS_TO_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +GIT_PARTIAL_SHA1_TAG_POINTS_TO_LICENSE_FILES = file + +$(eval $(generic-package))
support/testing/tests/download/br2-external/git-refs/package/git-sha1-reachable-by-tag/git-sha1-reachable-by-tag.hash new file +2 -0
@@ -0,0 +1,2 @@ +sha256 9b20256a3058221a8e91031f11700d9945ea84e8f328fa8e42e1cb9f7a30e3b2 git-sha1-reachable-by-tag-46bae5b639e5a18e2cc4dc508f080d566baeff59.tar.gz +sha256 2de87d77a2f226813f2d9bda906e970e4195605cdba6680443c0c04d89c532b6 file
support/testing/tests/download/br2-external/git-refs/package/git-sha1-reachable-by-tag/git-sha1-reachable-by-tag.mk new file +11 -0
@@ -0,0 +1,11 @@ +################################################################################ +# +# git-sha1-reachable-by-tag +# +################################################################################ + +GIT_SHA1_REACHABLE_BY_TAG_VERSION = 46bae5b639e5a18e2cc4dc508f080d566baeff59 +GIT_SHA1_REACHABLE_BY_TAG_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +GIT_SHA1_REACHABLE_BY_TAG_LICENSE_FILES = file + +$(eval $(generic-package))
support/testing/tests/download/br2-external/git-refs/package/git-sha1-tag-itself/git-sha1-tag-itself.hash new file +2 -0
@@ -0,0 +1,2 @@ +sha256 7d301c1a1054d6aee49193ca9e938f4da561ff73fb01719662865aa38bdc4361 git-sha1-tag-itself-2b0e0d98a49c97da6a618ab36337e2058eb733a2.tar.gz +sha256 6de8772a0a58fa62e2b8c58d4dae55c9db7534ad3b3918ecc849a9008d58f081 file
support/testing/tests/download/br2-external/git-refs/package/git-sha1-tag-itself/git-sha1-tag-itself.mk new file +11 -0
@@ -0,0 +1,11 @@ +################################################################################ +# +# git-sha1-tag-itself +# +################################################################################ + +GIT_SHA1_TAG_ITSELF_VERSION = 2b0e0d98a49c97da6a618ab36337e2058eb733a2 +GIT_SHA1_TAG_ITSELF_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +GIT_SHA1_TAG_ITSELF_LICENSE_FILES = file + +$(eval $(generic-package))
support/testing/tests/download/br2-external/git-refs/package/git-sha1-tag-points-to/git-sha1-tag-points-to.hash new file +2 -0
@@ -0,0 +1,2 @@ +sha256 c1f9f5734529a31140a71c031534460811f001b4db37d26833f386358150ab47 git-sha1-tag-points-to-516c9c5f64ec66534d4d069c2e408d9ae4dce023.tar.gz +sha256 6de8772a0a58fa62e2b8c58d4dae55c9db7534ad3b3918ecc849a9008d58f081 file
support/testing/tests/download/br2-external/git-refs/package/git-sha1-tag-points-to/git-sha1-tag-points-to.mk new file +11 -0
@@ -0,0 +1,11 @@ +################################################################################ +# +# git-sha1-tag-points-to +# +################################################################################ + +GIT_SHA1_TAG_POINTS_TO_VERSION = 516c9c5f64ec66534d4d069c2e408d9ae4dce023 +GIT_SHA1_TAG_POINTS_TO_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +GIT_SHA1_TAG_POINTS_TO_LICENSE_FILES = file + +$(eval $(generic-package))
support/testing/tests/download/br2-external/git-refs/package/git-tag/git-tag.hash new file +2 -0
@@ -0,0 +1,2 @@ +sha256 85dcb5bcf9bed496688d0eb01c7a3ce05c5b46b984cc1e9e76a6dbefd976e6b3 git-tag-mytag.tar.gz +sha256 6de8772a0a58fa62e2b8c58d4dae55c9db7534ad3b3918ecc849a9008d58f081 file
support/testing/tests/download/br2-external/git-refs/package/git-tag/git-tag.mk new file +11 -0
@@ -0,0 +1,11 @@ +################################################################################ +# +# git-tag +# +################################################################################ + +GIT_TAG_VERSION = mytag +GIT_TAG_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +GIT_TAG_LICENSE_FILES = file + +$(eval $(generic-package))
support/testing/tests/download/git-remote/repo.git/objects/2b/0e0d98a49c97da6a618ab36337e2058eb733a2 new file binary +0 -0
zcmV;40CxX)0Y#2W4gw(%1zmHBzJQY+enMi57jWYZuxSY at gF!ZPdoi<c^HQlr<@JzY zHfmRI52g-gw&cm&t!A>;C>?b~>q)Xk7kywIiBT%i*Ea*zQgRo4G|D>mM~NF;Gldo_ rTte%4q564dIJQkr<uA{vg#4OD<}^;=!5b&h!5$T&5L at yK5lk}^QNuc<
literal 0
HcmV?d00001
support/testing/tests/download/git-remote/repo.git/objects/46/bae5b639e5a18e2cc4dc508f080d566baeff59 new file binary +0 -0
zcmV;n07?IN0j<t4ZUZ3 at KvCD6f(xw0FyLXMDDnl;#3z_#V2u)%2$RCMx1|*BEw``u zmxnohDM9K(Noefo9uG%LJSruArV#ewpWkhrod-u!^2xu<Ptwip(W93@yUjwGPt zjL}<*4k!i(AhQw0f8*S{wk5XH2ukalr6T)chT?tANTcv$F*|#*~eBtG<ZX^4{-2$ kGH8st>wS8p7v<F86_h%abxSg}oTugb_ROie0nJ8hV&q3%KL7v#
literal 0
HcmV?d00001
support/testing/tests/download/git-remote/repo.git/objects/51/6c9c5f64ec66534d4d069c2e408d9ae4dce023 new file binary +0 -0
support/testing/tests/download/git-remote/repo.git/objects/92/ef85be57d627f280d8ce3724452ac21c9a6452 new file binary +0 -0
support/testing/tests/download/git-remote/repo.git/objects/cf/0f4f85d7a1237e377a2d25b996518a877ea001 new file binary +0 -0
zcmV-10M7q-0V^p=O;s>9VlXr?Ff%bxNXyJgWtjB7bzk^3^-m2q&Y7#YY8{f9l at bI1 H7n%=ExH=M#
literal 0
HcmV?d00001
support/testing/tests/download/git-remote/repo.git/objects/e2/2695cbf976fed1f543ad7486a531c0af473482 new file binary +0 -0
support/testing/tests/download/git-remote/repo.git/objects/f6/476b879f65e956d7dedd5b08736369e9a24acc new file binary +0 -0
support/testing/tests/download/git-remote/repo.git/refs/tags/mytag new file +1 -0
@@ -0,0 +1 @@ +2b0e0d98a49c97da6a618ab36337e2058eb733a2
support/testing/tests/download/test_git.py +7 -0
@@ -67,7 +67,14 @@ class TestGitRefs(GitTestBase): self.check_download("git-wrong-sha1") self.check_download("git-partial-sha1-branch-head") self.check_download("git-partial-sha1-reachable-by-branch") + self.check_download("git-partial-sha1-reachable-by-tag") + self.check_download("git-partial-sha1-tag-itself") + self.check_download("git-partial-sha1-tag-points-to") self.check_download("git-sha1-branch-head") self.check_download("git-sha1-reachable-by-branch") + self.check_download("git-sha1-reachable-by-tag") + self.check_download("git-sha1-tag-itself") + self.check_download("git-sha1-tag-points-to") self.check_download("git-submodule-disabled") self.check_download("git-submodule-enabled") + self.check_download("git-tag")
Signature · 1 line
2.17.0
Arnout Vandecappelle
AppliedPATCH v5 8/10 testing/tests/download: test git tag
Quoted 12 lines · Ricardo Martincoski
From: Ricardo Martincoski <[email protected]>
Add a tag to the static repo and check on the git refs test case the download of a git package:
- with the name of a tag as version;
- with the sha1 of a tag itself as version;
- with the partial sha1 of a tag itself as version;
- with the sha1 of a commit pointed by a tag as version;
- with the partial sha1 of a commit pointed by a tag as version;
- with the sha1 of a commit reachable only by a tag as version;
- with the partial sha1 of a commit reachable only by a tag as version.
Quoted 2 lines
Signed-off-by Ricardo Martincoski <[email protected]>Cc Arnout Vandecappelle <[email protected]>
Applied to master, thanks.
Regards,
Arnout
PATCH v5 9/10 testing/tests/download: test git special ref
In reply to Ricardo MartincoskiFrom: Ricardo Martincoski <[email protected]>
Add a special ref to the static repo and check on the git refs test case the download of a git package:
- with the name of a special ref as version;
---
Layout of static repo with the series applied until this patch:
$ git -C support/testing/tests/download/git-remote/repo.git \ log --all --decorate --graph --oneline --decorate * 2fa37f6 (submodule) sub0-2 * a9dbc1e sub0-1 | * b72ff60 (refs/changes/01/1/2) specialref |/ | * 68c197d (mybranch) branch2 | * 3174063 branch1 |/ | * 516c9c5 (tag: mytag) tag2 | * 46bae5b tag1 |/ * a238b1d (HEAD -> master) initial
Changes v4 -> v5:
- no changes
Changes v1 -> v4:
- not included on v2 or v3;
- rewrite from v1 using new git test infra with static repo;
--- .../package/git-special-ref/git-special-ref.hash | 2 ++ .../package/git-special-ref/git-special-ref.mk | 11 +++++++++++ .../17/c409592968d17600ff9912e96fdd461bb72e74 | Bin 0 -> 48 bytes .../53/122a7b0454b33c6d9e159c10657173db77899e | Bin 0 -> 27 bytes .../b7/2ff6078f62522a87f5cae5e9f34dedf5ec3885 | Bin 0 -> 186 bytes .../git-remote/repo.git/refs/changes/01/1/2 | 1 + support/testing/tests/download/test_git.py | 1 + 7 files changed, 15 insertions(+) create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-special-ref/git-special-ref.hash create mode 100644 support/testing/tests/download/br2-external/git-refs/package/git-special-ref/git-special-ref.mk create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/17/c409592968d17600ff9912e96fdd461bb72e74 create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/53/122a7b0454b33c6d9e159c10657173db77899e create mode 100644 support/testing/tests/download/git-remote/repo.git/objects/b7/2ff6078f62522a87f5cae5e9f34dedf5ec3885 create mode 100644 support/testing/tests/download/git-remote/repo.git/refs/changes/01/1/2
support/testing/tests/download/br2-external/git-refs/package/git-special-ref/git-special-ref.hash new file +2 -0
@@ -0,0 +1,2 @@ +sha256 c639b15260d098d94b8802cae812453c9f878ffd5f80cca6d87c58c01eed5c5a git-special-ref-refs_changes_01_1_2.tar.gz +sha256 b252b594be4679764e2e06cfea77a3e50638dd445af9b472b0f298c43b134518 file
support/testing/tests/download/br2-external/git-refs/package/git-special-ref/git-special-ref.mk new file +11 -0
@@ -0,0 +1,11 @@ +################################################################################ +# +# git-special-ref +# +################################################################################ + +GIT_SPECIAL_REF_VERSION = refs/changes/01/1/2 +GIT_SPECIAL_REF_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +GIT_SPECIAL_REF_LICENSE_FILES = file + +$(eval $(generic-package))
support/testing/tests/download/git-remote/repo.git/objects/17/c409592968d17600ff9912e96fdd461bb72e74 new file binary +0 -0
support/testing/tests/download/git-remote/repo.git/objects/53/122a7b0454b33c6d9e159c10657173db77899e new file binary +0 -0
support/testing/tests/download/git-remote/repo.git/objects/b7/2ff6078f62522a87f5cae5e9f34dedf5ec3885 new file binary +0 -0
o0SllGJ*KDpQ%>D)l54sSDN4Q9^DeyfQkL!O at h`vX8x-PgzB#pAi~s-t
literal 0
HcmV?d00001
support/testing/tests/download/git-remote/repo.git/refs/changes/01/1/2 new file +1 -0
@@ -0,0 +1 @@ +b72ff6078f62522a87f5cae5e9f34dedf5ec3885
support/testing/tests/download/test_git.py +1 -0
@@ -75,6 +75,7 @@ class TestGitRefs(GitTestBase): self.check_download("git-sha1-reachable-by-tag") self.check_download("git-sha1-tag-itself") self.check_download("git-sha1-tag-points-to") + self.check_download("git-special-ref") self.check_download("git-submodule-disabled") self.check_download("git-submodule-enabled") self.check_download("git-tag")
Signature · 1 line
2.17.0
PATCH v5 9/10 testing/tests/download: test git special ref
Quoted 5 lines · Ricardo Martincoski
From: Ricardo Martincoski <[email protected]>
Add a special ref to the static repo and check on the git refs test case the download of a git package:
- with the name of a special ref as version;
We might remove support for getting special refs by name, but keep on
supporting getting special refs by sha1. So I've changed the test into getting it by sha1 instead.
Ideally there should also be a check for getting the parent of the special ref
by sha1, but I think that actually does *not* work now.
Regards,
Arnout
PATCH v5 9/10 testing/tests/download: test git special ref
Hello,
Quoted 10 lines · Arnout Vandecappelle
Quoted 5 lines · Ricardo Martincoski
From: Ricardo Martincoski <[email protected]>
Add a special ref to the static repo and check on the git refs test case the download of a git package:
- with the name of a special ref as version;We might remove support for getting special refs by name, but keep on
supporting getting special refs by sha1. So I've changed the test into getting it by sha1 instead.
OK... but it depends on a recent git version to work. The same does not occur when getting special refs by name.
When using recent versions of git the download of the sha1 of a special ref works because of this line in support/download/git:
_git fetch origin "'${cset}:${cset}'"
I tested using:
git version 2.17.1
The git version installed in our docker image does not support it.
I am not sure what is the best fix for the TestGitRefs failure in GitLab CI. I would change the test to use special ref by name, and *when* we change the behavior, we change the test in the same commit to expect it to fail.
Quoted 3 lines
Ideally there should also be a check for getting the parent of the special ref
by sha1, but I think that actually does *not* work now.
I expect it to not work. But I didn't tested recently.
Regards,
Ricardo
PATCH v5 9/10 testing/tests/download: test git special ref
Quoted 11 lines · Ricardo Martincoski
Hello,
Quoted 7 lines · Arnout Vandecappelle
Quoted 5 lines · Ricardo Martincoski
From: Ricardo Martincoski <[email protected]>
Add a special ref to the static repo and check on the git refs test case the download of a git package:
- with the name of a special ref as version;
Quoted 5 lines
Quoted 3 lines
We might remove support for getting special refs by name, but keep on
supporting getting special refs by sha1. So I've changed the test into getting it by sha1 instead.OK... but it depends on a recent git version to work.
Ah, darn, I forgot that...
Quoted 5 lines
The same does not occur when getting special refs by name.
When using recent versions of git the download of the sha1 of a special ref works because of this line in support/download/git:
_gitfetch origin "'${cset}:${cset}'"
So, IIRC, Yann had an idea for how to rewrite this in a way that we would (1)
no longer support branch names and (2) we would support hashes that refer to special refs or ancestors of special refs. But I don't remember what it was. Maybe it was
_git fetch origin '+refs/*:refs/remotes/origin/*' '+refs/tags/*:refs/tags/*'
That way (I think) a branch name would no longer be usable (because when doing the checkout the origin/ bit would be missing), but a tag name still would be usable.
Quoted 8 lines
I tested using:
git version 2.17.1The git version installed in our docker image does not support it.
I am not sure what is the best fix for the TestGitRefs failure in GitLab CI. I would change the test to use special ref by name, and *when* we change the behavior, we change the test in the same commit to expect it to fail.
Or maybe just remove the test for now.
Regards,
Arnout
Quoted 10 lines
Quoted 4 lines
Ideally there should also be a check for getting the parent of the special ref
by sha1, but I think that actually does *not* work now.I expect it to not work. But I didn't tested recently.
Regards,
Ricardo
PATCH v5 9/10 testing/tests/download: test git special ref
Hello,
Quoted 30 lines · Arnout Vandecappelle
Quoted 10 lines · Ricardo Martincoski
Quoted 7 lines · Arnout Vandecappelle
Quoted 5 lines · Ricardo Martincoski
From: Ricardo Martincoski <[email protected]>
Add a special ref to the static repo and check on the git refs test case the download of a git package:
- with the name of a special ref as version;Quoted 5 lines
Quoted 3 lines
We might remove support for getting special refs by name, but keep on
supporting getting special refs by sha1. So I've changed the test into getting it by sha1 instead.OK... but it depends on a recent git version to work.
Ah, darn, I forgot that...
Quoted 5 lines
The same does not occur when getting special refs by name.
When using recent versions of git the download of the sha1 of a special ref works because of this line in support/download/git:
_gitfetch origin "'${cset}:${cset}'"So, IIRC, Yann had an idea for how to rewrite this in a way that we would (1)
no longer support branch names and (2) we would support hashes that refer to special refs or ancestors of special refs. But I don't remember what it was. Maybe it was
Quoted 5 lines
_gitfetch origin '+refs/*:refs/remotes/origin/*' '+refs/tags/*:refs/tags/*'That way (I think) a branch name would no longer be usable (because when doing the checkout the origin/ bit would be missing), but a tag name still would be usable.
I didn't tested it but I like the idea.
Quoted 11 lines
Quoted 9 lines
I tested using:
git version 2.17.1The git version installed in our docker image does not support it.
I am not sure what is the best fix for the TestGitRefs failure in GitLab CI. I would change the test to use special ref by name, and *when* we change the behavior, we change the test in the same commit to expect it to fail.
Or maybe just remove the test for now.
OK. When new code is added to support hashes that refer to special refs the test can be added again.
Regards,
Ricardo
PATCH v5 10/10 support/testing: test extra download with site method git
In reply to Ricardo MartincoskiAdd one test case to ensure the use of extra download works for git packages:
- an extra download is correctly downloaded using wget after the main download finished using the git backend;
- when the main download using the git backend fails, the build fails;
- when the extra download using wget fails, the build fails.
Reuse the commit in the static repo.
Add required infra and fixture:
- a HttpServer class, that starts a http server in the host machine to emulate a remote http server under the control of the test;
- a file to be served by the http server;
- a br2-external (git-extra-download) with one package for each part of the test case.
Since the HttpServer is similar to GitRemote, extract the commonalities to a new base class called Server.
---
Changes v4 -> v5:
- no changes
Changes v3 -> v4:
- new patch
I use docker only with the Buildroot image for the test infra. Sorry, I give up for now on trying to run apache on the docker image.
As a midground solution between:
- adding more files to http://autobuild.buildroot.net/artefacts/ that won't scale well if we need different files for different test cases;
- installing a full web server to the docker image auto-loading as root when the container is started as user and allowing the test infra to copy files to the served path as user;
and inspired by this article:
https://www.pcsuggest.com/best-lightweight-web-server-linux/
I created this patch using SimpleHTTPServer being started by the test infra as a user.
I know it is weird to use pexpect to fork a server written in Python, but it has these advantages:
- theoretically replacing SimpleHTTPServer with a similar solution (say the busybox httpd) in the future would require only small changes to the test infra code;
- SimpleHTTPServer is already on the docker image, busybox isn't, so no need to change the Dockerfile, generate the image, publish it and update the docker image tag on .gitlab-ci.yml;
- the log file generated by the server is handled in the same way as other log files generated by the test infra;
--- .gitlab-ci.yml | 1 + support/testing/infra/server.py | 39 +++++++++++++++++ .../br2-external/git-extra-download/Config.in | 0 .../git-extra-download/external.desc | 1 + .../git-extra-download/external.mk | 6 +++ .../package/extra-fails/extra-fails.hash | 3 ++ .../package/extra-fails/extra-fails.mk | 12 ++++++ .../package/main-fails/main-fails.hash | 3 ++ .../package/main-fails/main-fails.mk | 12 ++++++ .../git-extra-download/package/ok/ok.hash | 3 ++ .../git-extra-download/package/ok/ok.mk | 12 ++++++ support/testing/tests/download/gitremote.py | 43 +++---------------- .../testing/tests/download/http-server/extra | 1 + support/testing/tests/download/httpserver.py | 14 ++++++ support/testing/tests/download/test_git.py | 21 +++++++++ 15 files changed, 135 insertions(+), 36 deletions(-) create mode 100644 support/testing/infra/server.py create mode 100644 support/testing/tests/download/br2-external/git-extra-download/Config.in create mode 100644 support/testing/tests/download/br2-external/git-extra-download/external.desc create mode 100644 support/testing/tests/download/br2-external/git-extra-download/external.mk create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/extra-fails/extra-fails.hash create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/extra-fails/extra-fails.mk create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/main-fails/main-fails.hash create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/main-fails/main-fails.mk create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/ok/ok.hash create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/ok/ok.mk create mode 100644 support/testing/tests/download/http-server/extra create mode 100644 support/testing/tests/download/httpserver.py
.gitlab-ci.yml +1 -0
@@ -263,6 +263,7 @@ tests.core.test_rootfs_overlay.TestRootfsOverlay: *runtime_test tests.core.test_timezone.TestGlibcAllTimezone: *runtime_test tests.core.test_timezone.TestGlibcNonDefaultLimitedTimezone: *runtime_test tests.core.test_timezone.TestNoTimezone: *runtime_test +tests.download.test_git.TestGitExtraDownload: *runtime_test tests.download.test_git.TestGitHash: *runtime_test tests.download.test_git.TestGitRefs: *runtime_test tests.fs.test_ext.TestExt2: *runtime_test
support/testing/infra/server.py new file +39 -0
@@ -0,0 +1,39 @@ +# subprocess does not kill the child daemon when a test case fails by raising +# an exception. So use pexpect instead. +import pexpect + +import infra + + +class Server(object): + def __init__(self, builddir, serveddir, logtofile, daemon_cmd, port_arg, port_initial, port_last, good_msg, bad_msg): + """ + Start a local server. + + In order to support test cases in parallel, select the port the + server will listen to in runtime. Since there is no reliable way + to allocate the port prior to starting the server (another + process in the host machine can use the port between it is + selected from a list and it is really allocated to the server) + try to start the server in a port and in the case it is already + in use, try the next one in the allowed range. + """ + self.name = self.__class__.__name__.lower() + self.daemon = None + self.port = None + self.logfile = infra.open_log_file(builddir, self.name, logtofile) + + for port in range(port_initial, port_last + 1): + cmd = daemon_cmd + [port_arg.format(port=port)] + self.logfile.write("> starting {} with 'cd {} && {}'\n".format(self.name, serveddir, " ".join(cmd))) + self.daemon = pexpect.spawn(cmd[0], cmd[1:], logfile=self.logfile, cwd=serveddir) + ret = self.daemon.expect(good_msg + bad_msg) + if ret < len(good_msg): + self.port = port + return + raise SystemError("Could not find a free port to run {}".format(self.name)) + + def stop(self): + if self.daemon is None: + return + self.daemon.terminate(force=True)
support/testing/tests/download/br2-external/git-extra-download/Config.in new file +0 -0
support/testing/tests/download/br2-external/git-extra-download/external.desc new file +1 -0
@@ -0,0 +1 @@ +name: GIT_EXTRA_DOWNLOAD
support/testing/tests/download/br2-external/git-extra-download/external.mk new file +6 -0
@@ -0,0 +1,6 @@ +include $(sort $(wildcard $(BR2_EXTERNAL_GIT_EXTRA_DOWNLOAD_PATH)/package/*/*.mk)) + +# Get the git server port number from the test infra +GITREMOTE_PORT_NUMBER ?= 9418 +# Get the http server port number from the test infra +HTTP_SERVER_PORT_NUMBER ?= 8000
support/testing/tests/download/br2-external/git-extra-download/package/extra-fails/extra-fails.hash new file +3 -0
@@ -0,0 +1,3 @@ +sha256 1e6bc73fabdcce8857361e36e3c812c4ee42d8ffa30d56492bc56f8fcad7eb90 extra-fails-a238b1dfcd825d47d834af3c5223417c8411d90d.tar.gz +sha256 b9e68e1bea3e5b19ca6b2f98b73a54b73daafaa250484902e09982e07a12e733 notfound +sha256 da68f54607d5f5644954096ce1597c006c5bb9f2497e07441bf064b81003ef8a file
support/testing/tests/download/br2-external/git-extra-download/package/extra-fails/extra-fails.mk new file +12 -0
@@ -0,0 +1,12 @@ +################################################################################ +# +# extra-fails +# +################################################################################ + +EXTRA_FAILS_VERSION = a238b1dfcd825d47d834af3c5223417c8411d90d +EXTRA_FAILS_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +EXTRA_FAILS_EXTRA_DOWNLOADS = http://localhost:$(HTTP_SERVER_PORT_NUMBER)/notfound +EXTRA_FAILS_LICENSE_FILES = file + +$(eval $(generic-package))
support/testing/tests/download/br2-external/git-extra-download/package/main-fails/main-fails.hash new file +3 -0
@@ -0,0 +1,3 @@ +sha256 cd6851ef519a83345e4547f376b33d6bbd622d4ccbb234af9997c43854c602de main-fails-a238b1dfcd825d47d834af3c5223417c8411d90d.tar.gz +sha256 b9e68e1bea3e5b19ca6b2f98b73a54b73daafaa250484902e09982e07a12e733 extra +sha256 da68f54607d5f5644954096ce1597c006c5bb9f2497e07441bf064b81003ef8a file
support/testing/tests/download/br2-external/git-extra-download/package/main-fails/main-fails.mk new file +12 -0
@@ -0,0 +1,12 @@ +################################################################################ +# +# main-fails +# +################################################################################ + +MAIN_FAILS_VERSION = a238b1dfcd825d47d834af3c5223417c8411d90d +MAIN_FAILS_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/notfound.git +MAIN_FAILS_EXTRA_DOWNLOADS = http://localhost:$(HTTP_SERVER_PORT_NUMBER)/extra +MAIN_FAILS_LICENSE_FILES = file + +$(eval $(generic-package))
support/testing/tests/download/br2-external/git-extra-download/package/ok/ok.hash new file +3 -0
@@ -0,0 +1,3 @@ +sha256 737b4fd21506dbaa34cedc93c53c1ebb1f950db2c7644572bb006ae9297961da ok-a238b1dfcd825d47d834af3c5223417c8411d90d.tar.gz +sha256 b9e68e1bea3e5b19ca6b2f98b73a54b73daafaa250484902e09982e07a12e733 extra +sha256 da68f54607d5f5644954096ce1597c006c5bb9f2497e07441bf064b81003ef8a file
support/testing/tests/download/br2-external/git-extra-download/package/ok/ok.mk new file +12 -0
@@ -0,0 +1,12 @@ +################################################################################ +# +# ok +# +################################################################################ + +OK_VERSION = a238b1dfcd825d47d834af3c5223417c8411d90d +OK_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +OK_EXTRA_DOWNLOADS = http://localhost:$(HTTP_SERVER_PORT_NUMBER)/extra +OK_LICENSE_FILES = file + +$(eval $(generic-package))
support/testing/tests/download/gitremote.py +7 -36
51 lines not on this page · raw message
support/testing/tests/download/http-server/extra new file +1 -0
@@ -0,0 +1 @@ +text
support/testing/tests/download/httpserver.py new file +14 -0
@@ -0,0 +1,14 @@ +from infra.server import Server + +HTTP_SERVER_PORT_INITIAL = 8000 +HTTP_SERVER_PORT_LAST = HTTP_SERVER_PORT_INITIAL + 99 + + +class HttpServer(Server): + def __init__(self, builddir, serveddir, logtofile): + daemon_cmd = ["python", "-m", "SimpleHTTPServer"] + super(HttpServer, self).__init__( + builddir, serveddir, logtofile, + daemon_cmd, "{port}", HTTP_SERVER_PORT_INITIAL, HTTP_SERVER_PORT_LAST, + ["Serving HTTP"], + ["Address already in use"])
support/testing/tests/download/test_git.py +21 -0
@@ -3,6 +3,7 @@ import shutil import infra from gitremote import GitRemote +from httpserver import HttpServer class GitTestBase(infra.basetest.BRTest):
@@ -12,13 +13,19 @@ class GitTestBase(infra.basetest.BRTest): """ gitremotedir = infra.filepath("tests/download/git-remote") gitremote = None + httpserverdir = None + httpserver = None def setUp(self): super(GitTestBase, self).setUp() self.gitremote = GitRemote(self.builddir, self.gitremotedir, self.logtofile) + if self.httpserverdir: + self.httpserver = HttpServer(self.builddir, self.httpserverdir, self.logtofile) def tearDown(self): self.show_msg("Cleaning up") + if self.httpserver: + self.httpserver.stop() if self.gitremote: self.gitremote.stop() if self.b and not self.keepbuilds:
@@ -42,11 +49,25 @@ class GitTestBase(infra.basetest.BRTest): shutil.rmtree(dl_dir) env = {"BR2_DL_DIR": dl_dir, "GITREMOTE_PORT_NUMBER": str(self.gitremote.port)} + if self.httpserver: + env["HTTP_SERVER_PORT_NUMBER"] = str(self.httpserver.port) self.b.build(["{}-dirclean".format(package), "{}-legal-info".format(package)], env) +class TestGitExtraDownload(GitTestBase): + br2_external = [infra.filepath("tests/download/br2-external/git-extra-download")] + httpserverdir = infra.filepath("tests/download/http-server") + + def test_run(self): + with self.assertRaises(SystemError): + self.check_download("extra-fails") + with self.assertRaises(SystemError): + self.check_download("main-fails") + self.check_download("ok") + + class TestGitHash(GitTestBase): br2_external = [infra.filepath("tests/download/br2-external/git-hash")]
Signature · 1 line
2.17.0
PATCH v5 10/10 support/testing: test extra download with site method git
Quoted 7 lines · Ricardo Martincoski
Add one test case to ensure the use of extra download works for git packages:
- an extra download is correctly downloaded using wget after the main download finished using the git backend;
- when the main download using the git backend fails, the build fails;
- when the extra download using wget fails, the build fails.
Quoted 9 lines
Reuse the commit in the static repo.
Add required infra and fixture:
- a HttpServer class, that starts a http server in the host machine to emulate a remote http server under the control of the test;
- a file to be served by the http server;
- a br2-external (git-extra-download) with one package for each part of the test case.
Since the HttpServer is similar to GitRemote, extract the commonalities to a new base class called Server.
This one is a little too complicated to just apply, so I'm going to leave it
here for the time being...
It should for sure be split into two patches, one adding the common Server
class to the infra, and the next one adding the http test.
Also I'd probably prefer first just adding an http download test case and only
later adding the extra downloads test.
However, I also don't really like the approach of launching our own http
server. I think it's much easier to rely on an existing one, perhaps using sources.buildroot.org. You could say that it makes it impossible to run the tests offline, but that's already the case for a most of them, because they download sources and kernel binaries etc.
So I've marked this one as Changes Requested in patchwork.
Regards,
Arnout
Quoted 367 lines
Signed-off-by Ricardo Martincoski <[email protected]>Cc Luca Ceresoli <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Yann E. MORIN <[email protected]>---
Changes v4 -> v5:
- no changesChanges v3 -> v4:
- new patchI use docker only with the Buildroot image for the test infra. Sorry, I give up for now on trying to run apache on the docker image.
As a midground solution between:
- adding more files to http://autobuild.buildroot.net/artefacts/ that won't scale well if we need different files for different test cases;
- installing a full web server to the docker image auto-loading as root when the container is started as user and allowing the test infra to copy files to the served path as user;
and inspired by this article:
https://www.pcsuggest.com/best-lightweight-web-server-linux/
I created this patch using SimpleHTTPServer being started by the test infra as a user.
I know it is weird to use pexpect to fork a server written in Python, but it has these advantages:
- theoretically replacing SimpleHTTPServer with a similar solution (say the busybox httpd) in the future would require only small changes to the test infra code;
- SimpleHTTPServer is already on the docker image, busybox isn't, so no need to change the Dockerfile, generate the image, publish it and update the docker image tag on .gitlab-ci.yml;
- the log file generated by the server is handled in the same way as other log files generated by the test infra;--- .gitlab-ci.yml | 1 + support/testing/infra/server.py | 39 +++++++++++++++++ .../br2-external/git-extra-download/Config.in | 0 .../git-extra-download/external.desc | 1 + .../git-extra-download/external.mk | 6 +++ .../package/extra-fails/extra-fails.hash | 3 ++ .../package/extra-fails/extra-fails.mk | 12 ++++++ .../package/main-fails/main-fails.hash | 3 ++ .../package/main-fails/main-fails.mk | 12 ++++++ .../git-extra-download/package/ok/ok.hash | 3 ++ .../git-extra-download/package/ok/ok.mk | 12 ++++++ support/testing/tests/download/gitremote.py | 43 +++---------------- .../testing/tests/download/http-server/extra | 1 + support/testing/tests/download/httpserver.py | 14 ++++++ support/testing/tests/download/test_git.py | 21 +++++++++ 15 files changed, 135 insertions(+), 36 deletions(-) create mode 100644 support/testing/infra/server.py create mode 100644 support/testing/tests/download/br2-external/git-extra-download/Config.in create mode 100644 support/testing/tests/download/br2-external/git-extra-download/external.desc create mode 100644 support/testing/tests/download/br2-external/git-extra-download/external.mk create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/extra-fails/extra-fails.hash create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/extra-fails/extra-fails.mk create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/main-fails/main-fails.hash create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/main-fails/main-fails.mk create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/ok/ok.hash create mode 100644 support/testing/tests/download/br2-external/git-extra-download/package/ok/ok.mk create mode 100644 support/testing/tests/download/http-server/extra create mode 100644 support/testing/tests/download/httpserver.py.gitlab-ci.yml +1
-0@@ -263,6 +263,7 @@ tests.core.test_rootfs_overlay.TestRootfsOverlay: *runtime_test tests.core.test_timezone.TestGlibcAllTimezone: *runtime_test tests.core.test_timezone.TestGlibcNonDefaultLimitedTimezone: *runtime_test tests.core.test_timezone.TestNoTimezone: *runtime_test +tests.download.test_git.TestGitExtraDownload: *runtime_test tests.download.test_git.TestGitHash: *runtime_test tests.download.test_git.TestGitRefs: *runtime_test tests.fs.test_ext.TestExt2: *runtime_testsupport/testing/infra/server.py new file +39
-0@@ -0,0 +1,39 @@ +# subprocess does not kill the child daemon when a test case fails by raising +# an exception. So use pexpect instead. +import pexpect + +import infra + + +class Server(object): + def __init__(self, builddir, serveddir, logtofile, daemon_cmd, port_arg, port_initial, port_last, good_msg, bad_msg): + """ + Start a local server. + + In order to support test cases in parallel, select the port the + server will listen to in runtime. Since there is no reliable way + to allocate the port prior to starting the server (another + process in the host machine can use the port between it is + selected from a list and it is really allocated to the server) + try to start the server in a port and in the case it is already + in use, try the next one in the allowed range. + """ + self.name = self.__class__.__name__.lower() + self.daemon = None + self.port = None + self.logfile = infra.open_log_file(builddir, self.name, logtofile) + + for port in range(port_initial, port_last + 1): + cmd = daemon_cmd + [port_arg.format(port=port)] + self.logfile.write("> starting {} with 'cd {} && {}'\n".format(self.name, serveddir, " ".join(cmd))) + self.daemon = pexpect.spawn(cmd[0], cmd[1:], logfile=self.logfile, cwd=serveddir) + ret = self.daemon.expect(good_msg + bad_msg) + if ret < len(good_msg): + self.port = port + return + raise SystemError("Could not find a free port to run {}".format(self.name)) + + def stop(self): + if self.daemon is None: + return + self.daemon.terminate(force=True)support/testing/tests/download/br2-external/git-extra-download/Config.in new file +0
-0support/testing/tests/download/br2-external/git-extra-download/external.desc new file +1
-0@@ -0,0 +1 @@ +name: GIT_EXTRA_DOWNLOADsupport/testing/tests/download/br2-external/git-extra-download/external.mk new file +6
-0@@ -0,0 +1,6 @@ +include $(sort $(wildcard $(BR2_EXTERNAL_GIT_EXTRA_DOWNLOAD_PATH)/package/*/*.mk)) + +# Get the git server port number from the test infra +GITREMOTE_PORT_NUMBER ?= 9418 +# Get the http server port number from the test infra +HTTP_SERVER_PORT_NUMBER ?= 8000support/testing/tests/download/br2-external/git-extra-download/package/extra-fails/extra-fails.hash new file +3
-0@@ -0,0 +1,3 @@ +sha256 1e6bc73fabdcce8857361e36e3c812c4ee42d8ffa30d56492bc56f8fcad7eb90 extra-fails-a238b1dfcd825d47d834af3c5223417c8411d90d.tar.gz +sha256 b9e68e1bea3e5b19ca6b2f98b73a54b73daafaa250484902e09982e07a12e733 notfound +sha256 da68f54607d5f5644954096ce1597c006c5bb9f2497e07441bf064b81003ef8a filesupport/testing/tests/download/br2-external/git-extra-download/package/extra-fails/extra-fails.mk new file +12
-0@@ -0,0 +1,12 @@ +################################################################################ +# +# extra-fails +# +################################################################################ + +EXTRA_FAILS_VERSION = a238b1dfcd825d47d834af3c5223417c8411d90d +EXTRA_FAILS_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +EXTRA_FAILS_EXTRA_DOWNLOADS = http://localhost:$(HTTP_SERVER_PORT_NUMBER)/notfound +EXTRA_FAILS_LICENSE_FILES = file + +$(eval $(generic-package))support/testing/tests/download/br2-external/git-extra-download/package/main-fails/main-fails.hash new file +3
-0@@ -0,0 +1,3 @@ +sha256 cd6851ef519a83345e4547f376b33d6bbd622d4ccbb234af9997c43854c602de main-fails-a238b1dfcd825d47d834af3c5223417c8411d90d.tar.gz +sha256 b9e68e1bea3e5b19ca6b2f98b73a54b73daafaa250484902e09982e07a12e733 extra +sha256 da68f54607d5f5644954096ce1597c006c5bb9f2497e07441bf064b81003ef8a filesupport/testing/tests/download/br2-external/git-extra-download/package/main-fails/main-fails.mk new file +12
-0@@ -0,0 +1,12 @@ +################################################################################ +# +# main-fails +# +################################################################################ + +MAIN_FAILS_VERSION = a238b1dfcd825d47d834af3c5223417c8411d90d +MAIN_FAILS_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/notfound.git +MAIN_FAILS_EXTRA_DOWNLOADS = http://localhost:$(HTTP_SERVER_PORT_NUMBER)/extra +MAIN_FAILS_LICENSE_FILES = file + +$(eval $(generic-package))support/testing/tests/download/br2-external/git-extra-download/package/ok/ok.hash new file +3
-0@@ -0,0 +1,3 @@ +sha256 737b4fd21506dbaa34cedc93c53c1ebb1f950db2c7644572bb006ae9297961da ok-a238b1dfcd825d47d834af3c5223417c8411d90d.tar.gz +sha256 b9e68e1bea3e5b19ca6b2f98b73a54b73daafaa250484902e09982e07a12e733 extra +sha256 da68f54607d5f5644954096ce1597c006c5bb9f2497e07441bf064b81003ef8a filesupport/testing/tests/download/br2-external/git-extra-download/package/ok/ok.mk new file +12
-0@@ -0,0 +1,12 @@ +################################################################################ +# +# ok +# +################################################################################ + +OK_VERSION = a238b1dfcd825d47d834af3c5223417c8411d90d +OK_SITE = git://localhost:$(GITREMOTE_PORT_NUMBER)/repo.git +OK_EXTRA_DOWNLOADS = http://localhost:$(HTTP_SERVER_PORT_NUMBER)/extra +OK_LICENSE_FILES = file + +$(eval $(generic-package))support/testing/tests/download/gitremote.py +7
-36@@ -1,44 +1,15 @@ -# subprocess does not kill the child daemon when a test case fails by raising -# an exception. So use pexpect instead. -import pexpect - -import infra +from infra.server import Server GIT_REMOTE_PORT_INITIAL = 9418 GIT_REMOTE_PORT_LAST = GIT_REMOTE_PORT_INITIAL + 99 -class GitRemote(object): +class GitRemote(Server): def __init__(self, builddir, serveddir, logtofile): - """ - Start a local git server. - - In order to support test cases in parallel, select the port the - server will listen to in runtime. Since there is no reliable way - to allocate the port prior to starting the server (another - process in the host machine can use the port between it is - selected from a list and it is really allocated to the server) - try to start the server in a port and in the case it is already - in use, try the next one in the allowed range. - """ - self.daemon = None - self.port = None - self.logfile = infra.open_log_file(builddir, "gitremote", logtofile) - daemon_cmd = ["git", "daemon", "--reuseaddr", "--verbose", "--listen=localhost", "--export-all", "--base-path={}".format(serveddir)] - for port in range(GIT_REMOTE_PORT_INITIAL, GIT_REMOTE_PORT_LAST + 1): - cmd = daemon_cmd + ["--port={port}".format(port=port)] - self.logfile.write("> starting git remote with '{}'\n".format(" ".join(cmd))) - self.daemon = pexpect.spawn(cmd[0], cmd[1:], logfile=self.logfile) - ret = self.daemon.expect(["Ready to rumble", - "Address already in use"]) - if ret == 0: - self.port = port - return - raise SystemError("Could not find a free port to run git remote") - - def stop(self): - if self.daemon is None: - return - self.daemon.terminate(force=True) + super(GitRemote, self).__init__( + builddir, serveddir, logtofile, + daemon_cmd, "--port={port}", GIT_REMOTE_PORT_INITIAL, GIT_REMOTE_PORT_LAST, + ["Ready to rumble"], + ["Address already in use"])support/testing/tests/download/http-server/extra new file +1
-0@@ -0,0 +1 @@ +textsupport/testing/tests/download/httpserver.py new file +14
-0@@ -0,0 +1,14 @@ +from infra.server import Server + +HTTP_SERVER_PORT_INITIAL = 8000 +HTTP_SERVER_PORT_LAST = HTTP_SERVER_PORT_INITIAL + 99 + + +class HttpServer(Server): + def __init__(self, builddir, serveddir, logtofile): + daemon_cmd = ["python", "-m", "SimpleHTTPServer"] + super(HttpServer, self).__init__( + builddir, serveddir, logtofile, + daemon_cmd, "{port}", HTTP_SERVER_PORT_INITIAL, HTTP_SERVER_PORT_LAST, + ["Serving HTTP"], + ["Address already in use"])support/testing/tests/download/test_git.py +21
-0@@ -3,6 +3,7 @@ import shutil import infra from gitremote import GitRemote +from httpserver import HttpServer class GitTestBase(infra.basetest.BRTest):@@ -12,13 +13,19 @@ class GitTestBase(infra.basetest.BRTest): """ gitremotedir = infra.filepath("tests/download/git-remote") gitremote = None + httpserverdir = None + httpserver = None def setUp(self): super(GitTestBase, self).setUp() self.gitremote = GitRemote(self.builddir, self.gitremotedir, self.logtofile) + if self.httpserverdir: + self.httpserver = HttpServer(self.builddir, self.httpserverdir, self.logtofile) def tearDown(self): self.show_msg("Cleaning up") + if self.httpserver: + self.httpserver.stop() if self.gitremote: self.gitremote.stop() if self.b and not self.keepbuilds:@@ -42,11 +49,25 @@ class GitTestBase(infra.basetest.BRTest): shutil.rmtree(dl_dir) env = {"BR2_DL_DIR": dl_dir, "GITREMOTE_PORT_NUMBER": str(self.gitremote.port)} + if self.httpserver: + env["HTTP_SERVER_PORT_NUMBER"] = str(self.httpserver.port) self.b.build(["{}-dirclean".format(package), "{}-legal-info".format(package)], env) +class TestGitExtraDownload(GitTestBase): + br2_external = [infra.filepath("tests/download/br2-external/git-extra-download")] + httpserverdir = infra.filepath("tests/download/http-server") + + def test_run(self): + with self.assertRaises(SystemError): + self.check_download("extra-fails") + with self.assertRaises(SystemError): + self.check_download("main-fails") + self.check_download("ok") + + class TestGitHash(GitTestBase): br2_external = [infra.filepath("tests/download/br2-external/git-hash")]
PATCH 3/3 support/download/git: do not use git clone
In reply to Ricardo MartincoskiThe logic of the script was completely rewritten based in some ideas discussed during the review of [1].
Always using git init + git fetch has these advantages:
- git fetch works with all kinds of refs, while git clone supports only branches and tags without the ref/heads/ and ref/tags/ prefixes;
- a shallow fetch can be done for the head of those refs.
The remote is first asked for its references using git ls-remote saving the output to a file.
This file is later on compared to the desired change set, determining if it is a branch, tag, special ref or sha1.
A concern that arrives from this method is that the remote can change between the git ls-remote and the git fetch, but in this case, once the script creates a shallow fetch, the checkout does what it is expected:
- for a named reference (branch, tag or special ref), the fetch and checkout are successful using the "new" sha1 from the remote;
- for a sha1 of a branch, the fetch fails (because the sha1 it is not anymore at branch head), falling back to a successful full fetch;
- for a sha1 of a commit pointed by a tag (that should not be changed but it can be changed) the same behavior of a sha1 of a branch takes place;
- for a sha1 only accessible by a special refs, the checkout fails, falling back to an unsuccessful checkout after full fetch. This can be caused only be a error from a developer.
An analytical solution is used instead of a single awk line. It makes each line of code simple: grep to check the entry is in the ls-remote output and awk to actually get the reference to use.
[1] http://patchwork.ozlabs.org/patch/681841/
--- support/download/git | 91 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 60 insertions(+), 31 deletions(-)
support/download/git +60 -31
@@ -39,43 +39,72 @@ _git() { eval ${GIT} "${@}" } -# Try a shallow clone, since it is faster than a full clone - but that only -# works if the version is a ref (tag or branch). Before trying to do a shallow -# clone we check if ${cset} is in the list provided by git ls-remote. If not -# we fall back on a full clone. -# -# Messages for the type of clone used are provided to ease debugging in case of -# problems -git_done=0 -if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then - printf "Doing shallow clone\n" - if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${repo}'" "'${basename}'"; then - git_done=1 - else - printf "Shallow clone failed, falling back to doing a full clone\n" +_git init ${verbose} "'${basename}'" + +pushd "${basename}" >/dev/null + +# Save the temporary file inside the .git directory that will be deleted after the checkout is done. +tmpf=".git/ls-remote" +_git ls-remote "'${repo}'" > "${tmpf}" + +do_a_shallow_fetch=0 +if grep "\<\(\|\(\|refs/\)\(heads\|tags\)/\)${cset}$" "${tmpf}" >/dev/null 2>&1; then + printf "The desired version is a named reference\n" + # Support branches and tags in the simplified form. + # Support branches and tags and special refs in the complete form refs/heads/branch. + # When the name is ambiguous, the branch will be selected (by git fetch or git clone). + ref="${cset}" + do_a_shallow_fetch=1 +elif grep "^${cset}" "${tmpf}" >/dev/null 2>&1; then + printf "The desired version is a sha1\n" + if [ ${#cset} -lt 40 -a "1" != "$(awk "/^${cset}/{print \$1|\"sort -u | wc -l\"}" "${tmpf}")" ]; then + printf "Ambiguous partial sha1\n" + awk "/^${cset}/{print \$1|\"sort -u | wc -l\"}" "${tmpf}" + exit 1 + fi + # Accept full or unambiguous partial sha1. A sha1 can be referenced by many names. + # Prefer sha1 of commit pointed by a tag or sha1 of the tag itself, + # then sha1 pointed by any ref/*, and only then sha1 pointed by *HEAD. + ref="$(awk -F'[\t^]' "/^${cset}.*\trefs\/tags\//{ print \$2; exit }" "${tmpf}")" + if [ -z "${ref}" ]; then + ref="$(awk -F'[\t^]' "/^${cset}.*\trefs\//{ print \$2; exit }" "${tmpf}")" + fi + if [ -z "${ref}" ]; then + # sha1 is referenced by HEAD + ref="$(awk -F'[\t^]' "/^${cset}/{ print \$2; exit }" "${tmpf}")" + fi + if [ -n "${ref}" ]; then + printf "Fetching '%s' to get '%s'\n" "${ref}" "${cset}" + do_a_shallow_fetch=1 fi -fi -if [ ${git_done} -eq 0 ]; then - printf "Doing full clone\n" - _git clone ${verbose} "${@}" "'${repo}'" "'${basename}'" fi -pushd "${basename}" >/dev/null +git_fetch_done=0 +if [ ${do_a_shallow_fetch} -eq 1 ]; then + printf "Doing shallow fetch\n" + if _git fetch ${verbose} "${@}" --depth 1 "'${repo}'" "'${ref}:refs/buildroot/${cset}'" 2>&1; then + git_fetch_done=1 + else + # It catches the case the remote changed after ls-remote. + printf "Shallow fetch failed, falling back to doing a full fetch\n" + fi +fi -# Try to get the special refs exposed by some forges (pull-requests for -# github, changes for gerrit...). There is no easy way to know whether -# the cset the user passed us is such a special ref or a tag or a sha1 -# or whatever else. We'll eventually fail at checking out that cset, -# below, if there is an issue anyway. Since most of the cset we're gonna -# have to clone are not such special refs, consign the output to oblivion -# so as not to alarm unsuspecting users, but still trace it as a warning. -if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then - printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}" +git_checkout_done=0 +if [ ${git_fetch_done} -eq 1 ]; then + if _git checkout -q "'refs/buildroot/${cset}'" 2>&1; then + git_checkout_done=1 + else + printf "Checkout failed, falling back to doing a full fetch\n" + fi fi -# Checkout the required changeset, so that we can update the required -# submodules. -_git checkout -q "'${cset}'" +if [ ${git_checkout_done} -eq 0 ]; then + printf "Doing full fetch\n" + _git remote add origin "'${repo}'" + _git fetch ${verbose} "${@}" origin + _git checkout -q "'${cset}'" +fi # Get date of commit to generate a reproducible archive. # %cD is RFC2822, so it's fully qualified, with TZ and all.
Signature · 1 line
2.9.3
PATCH 3/3 support/download/git: do not use git clone
Quoted 26 lines · Ricardo Martincoski
The logic of the script was completely rewritten based in some ideas discussed during the review of [1].
Always using git init + git fetch has these advantages:
- git fetch works with all kinds of refs, while git clone supports only branches and tags without the ref/heads/ and ref/tags/ prefixes;
- a shallow fetch can be done for the head of those refs.The remote is first asked for its references using git ls-remote saving the output to a file.
This file is later on compared to the desired change set, determining if it is a branch, tag, special ref or sha1.
A concern that arrives from this method is that the remote can change between the git ls-remote and the git fetch, but in this case, once the script creates a shallow fetch, the checkout does what it is expected:
- for a named reference (branch, tag or special ref), the fetch and checkout are successful using the "new" sha1 from the remote;
- for a sha1 of a branch, the fetch fails (because the sha1 it is not anymore at branch head), falling back to a successful full fetch;
- for a sha1 of a commit pointed by a tag (that should not be changed but it can be changed) the same behavior of a sha1 of a branch takes place;
- for a sha1 only accessible by a special refs, the checkout fails, falling back to an unsuccessful checkout after full fetch. This can be caused only be a error from a developer.
Quoted 3 lines
An analytical solution is used instead of a single awk line. It makes each line of code simple: grep to check the entry is in the ls-remote output and awk to actually get the reference to use.
Excellent commit message!
Quoted 4 lines
[1] http://patchwork.ozlabs.org/patch/681841/
Signed-off-by Ricardo Martincoski <[email protected]>
I have a bunch of remarks below, but nothing that should stop the acceptance of
this patch. So:
This is for next, obviously.
Quoted 3 lines
--- support/download/git | 91 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 60 insertions(+), 31 deletions(-)
Note that, although lines are added here, it really is a simplification and
explicitation of the logic.
Quoted 18 lines
support/download/git +0
-9@@ -39,43 +39,72 @@ _git() { eval ${GIT} "${@}" } -# Try a shallow clone, since it is faster than a full clone - but that only -# works if the version is a ref (tag or branch). Before trying to do a shallow -# clone we check if ${cset} is in the list provided by git ls-remote. If not -# we fall back on a full clone. -# -# Messages for the type of clone used are provided to ease debugging in case of -# problems -git_done=0 -if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then
support/download/git _git() { · line 46
Quoted 10 lines
support/download/git +5
-5- printf "Doing shallow clone\n" - if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${repo}'" "'${basename}'"; then - git_done=1 - else - printf "Shallow clone failed, falling back to doing a full clone\n" +_git init ${verbose} "'${basename}'" + +pushd "${basename}" >/dev/null + +# Save the temporary file inside the .git directory that will be deleted after the checkout is done.
Don't forget to wrap at 80 columns.
support/download/git _git() { · line 51
Quoted 5 lines
support/download/git +5
-0+tmpf=".git/ls-remote" +_git ls-remote "'${repo}'" > "${tmpf}" + +do_a_shallow_fetch=0 +if grep "\<\(\|\(\|refs/\)\(heads\|tags\)/\)${cset}$" "${tmpf}" >/dev/null 2>&1; then
This is where grep -E becomes useful :-)
support/download/git _git() { · line 60
Quoted 9 lines
support/download/git +9
-0+ printf "The desired version is a named reference\n" + # Support branches and tags in the simplified form. + # Support branches and tags and special refs in the complete form refs/heads/branch. + # When the name is ambiguous, the branch will be selected (by git fetch or git clone). + ref="${cset}" + do_a_shallow_fetch=1 +elif grep "^${cset}" "${tmpf}" >/dev/null 2>&1; then + printf "The desired version is a sha1\n" + if [ ${#cset} -lt 40 -a "1" != "$(awk "/^${cset}/{print \$1|\"sort -u | wc -l\"}" "${tmpf}")" ]; then
I think the check for -lt 40 is redundant.
I think all the logic here could be made a bit simpler by saving to a temporary
file again:
elif grep "^${cset}" "${tmpf}" > .git/matching_refs 2>/dev/null; then
printf ...
if "$(cut -f 1 .git/matching_refs | sort -u | wc -l) != 2; then
...support/download/git _git() { · line 62
Quoted 2 lines
support/download/git +2
-0+ printf "Ambiguous partial sha1\n" + awk "/^${cset}/{print \$1|\"sort -u | wc -l\"}" "${tmpf}"
Here we should print the full lines, so without sort -u and wc. Sort is still
useful though.
support/download/git _git() { · line 67
Quoted 5 lines
support/download/git +5
-0+ exit 1 + fi + # Accept full or unambiguous partial sha1. A sha1 can be referenced by many names. + # Prefer sha1 of commit pointed by a tag or sha1 of the tag itself, + # then sha1 pointed by any ref/*, and only then sha1 pointed by *HEAD.
I don't understand why this bit is needed. Any of the refs will do, right? It's
true that there is a risk that the ref gets updated between the ls-remote and the fetch, and that for tags the chance that it gets updated is smaller than for e.g. HEAD. But still, this is very much a corner case, and if it _does_ happen, we still fall back to full clone. So the only important thing is to avoid HEAD. Since ls-remotes already seems to sort the refs, which typically puts tags near the end, just take the last one and we're done:
ref="$(cut -f 2 .git/matching_refs | tail -1)"
Quoted 11 lines
support/download/git +11
-0+ ref="$(awk -F'[\t^]' "/^${cset}.*\trefs\/tags\//{ print \$2; exit }" "${tmpf}")" + if [ -z "${ref}" ]; then + ref="$(awk -F'[\t^]' "/^${cset}.*\trefs\//{ print \$2; exit }" "${tmpf}")" + fi + if [ -z "${ref}" ]; then + # sha1 is referenced by HEAD + ref="$(awk -F'[\t^]' "/^${cset}/{ print \$2; exit }" "${tmpf}")" + fi + if [ -n "${ref}" ]; then + printf "Fetching '%s' to get '%s'\n" "${ref}" "${cset}" + do_a_shallow_fetch=1
support/download/git _git() { · line 83
Quoted 10 lines
support/download/git +2
-5fi -fi -if [ ${git_done} -eq 0 ]; then - printf "Doing full clone\n" - _git clone ${verbose} "${@}" "'${repo}'" "'${basename}'" fi -pushd "${basename}" >/dev/null +git_fetch_done=0 +if [ ${do_a_shallow_fetch} -eq 1 ]; then
Actually, if [ "$ref" ] would be sufficient.
Quoted 6 lines
support/download/git +6
-0+ printf "Doing shallow fetch\n" + if _git fetch ${verbose} "${@}" --depth 1 "'${repo}'" "'${ref}:refs/buildroot/${cset}'" 2>&1; then + git_fetch_done=1 + else + # It catches the case the remote changed after ls-remote. + printf "Shallow fetch failed, falling back to doing a full fetch\n"
support/download/git _git() { · line 71
Quoted 10 lines
support/download/git +2
-7+ fi +fi -# Try to get the special refs exposed by some forges (pull-requests for -# github, changes for gerrit...). There is no easy way to know whether -# the cset the user passed us is such a special ref or a tag or a sha1 -# or whatever else. We'll eventually fail at checking out that cset, -# below, if there is an issue anyway. Since most of the cset we're gonna -# have to clone are not such special refs, consign the output to oblivion -# so as not to alarm unsuspecting users, but still trace it as a warning.
Yann will be happy that this gets removed :-)
support/download/git _git() { · line 95
Quoted 5 lines
support/download/git +3
-2-if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then - printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}" +git_checkout_done=0 +if [ ${git_fetch_done} -eq 1 ]; then + if _git checkout -q "'refs/buildroot/${cset}'" 2>&1; then
I'm not sure if I like this split. git_fetch_done gets set in only one place,
so we could just do the checkout directly there. That would also make it more explicit that this is a checkout specifically for shallow fetches, while the checkout below is specifically for full fetches.
support/download/git _git() { · line 105
Quoted 13 lines
support/download/git +8
-3+ git_checkout_done=1 + else + printf "Checkout failed, falling back to doing a full fetch\n" + fi fi -# Checkout the required changeset, so that we can update the required -# submodules. -_git checkout -q "'${cset}'" +if [ ${git_checkout_done} -eq 0 ]; then + printf "Doing full fetch\n" + _git remote add origin "'${repo}'" + _git fetch ${verbose} "${@}" origin
Why do we need to create a remote here?
support/download/git _git() { · line 107
Quoted 2 lines
support/download/git +2
-0+ _git checkout -q "'${cset}'" +fi
At a later stage, we could update the fetch with --submodules=yes/no depending
on -r option.
Another interesting feature would be to first try a shallow fetch of all tags
and branches with a depth of, say, 100. That is still way less than a full clone for large repositories, and has a good chance to still capture a sha1 that is not at a branch or tag head. So it would provide a nice middle ground for the cases for which we fail to do a shallow clone now. And if it does fail, nothing is wasted because the full clone can start from the part that was already downloaded.
Regards,
Arnout
Quoted 4 lines
support/download/git +0
-0# Get date of commit to generate a reproducible archive. # %cD is RFC2822, so it's fully qualified, with TZ and all.
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH 3/3 support/download/git: do not use git clone
Arnout,
Sorry the delay.
On Sat, Nov 05, 2016 at 11:19 PM, Arnout Vandecappelle wrote:
[snip]
Your suggestions will simplify the script.
I will answer all questions in each review and resend the series in the next days. So I marked 1/3 and 3/3 as Changes Requested.
Regards,
Ricardo
PATCH 3/3 support/download/git: do not use git clone
In reply to Arnout VandecappelleArnout,
On Sat, Nov 05, 2016 at 11:19 PM, Arnout Vandecappelle wrote:
[snip]
Quoted 1 line
Excellent commit message!
Thanks.
[snip]
Quoted 5 lines
I have a bunch of remarks below, but nothing that should stop the acceptance of
this patch. So:
I won't carry these for v2 because there will be a bunch of improvements based on your review ;)
Quoted 3 lines
This is for next, obviously.
[snip]
Quoted 3 lines
Quoted 1 line
+# Save the temporary file inside the .git directory that will be deleted after the checkout is done.
Don't forget to wrap at 80 columns.
Sorry. This line is way too long.
But just to be clear:
I think you mean to wrap to 80 as a rule of thumb (e.g. comments are easy to do) and not as a hard requirement.
Right?
I ask this because I think breaking some lines with commands would make it
harder to read, e.g.
if _git fetch ${verbose} "${@}" --depth 1 "'${repo}'" "'${ref}:refs/buildroot/${cset}'" 2>&1; then
[snip]
Quoted 3 lines
Quoted 1 line
+if grep "\<\(\|\(\|refs/\)\(heads\|tags\)/\)${cset}$" "${tmpf}" >/dev/null 2>&1; then
This is where grep -E becomes useful :-)
Thanks. Much nicer:
if grep -E "\<(|(|refs/)(heads|tags)/)${cset}$" "${tmpf}" >/dev/null 2>&1; then
[snip]
Quoted 5 lines
support/download/git _git() { · line 60
Quoted 3 lines
support/download/git +3
-0+elif grep "^${cset}" "${tmpf}" >/dev/null 2>&1; then + printf "The desired version is a sha1\n" + if [ ${#cset} -lt 40 -a "1" != "$(awk "/^${cset}/{print \$1|\"sort -u | wc -l\"}" "${tmpf}")" ]; thenI think the check for -lt 40 is redundant.
You are right.
But this line will be removed in v2 because it is not needed and not wanted. It is not needed because unambiguous partial sha1 (of any commit) can be checked out after the full fetch (but without optimized download, of course). It is not wanted because, as I realized after your comment, the output of ls-remote is a subset of all sha1s and can't be used to test a sha1 is not ambiguous. We could end up checking out a wrong sha1 when we really need to let git checkout fail.
BTW, an example of partial sha1 is in the current master, see commit 956fcc2.
Quoted 8 lines
I think all the logic here could be made a bit simpler by saving to a temporary
file again:elif grep "^${cset}" "${tmpf}" > .git/matching_refs 2>/dev/null; then
printf ... if "$(cut -f 1 .git/matching_refs | sort -u | wc -l) != 2; then ...
Yes. Much simpler.
Actually this 'if' is removed in v2 (see above) but I will use temporary file and cut instead of awk in v2 (see below).
Quoted 7 lines
support/download/git _git() { · line 62
Quoted 4 lines
support/download/git +2
-0+ printf "Ambiguous partial sha1\n" + awk "/^${cset}/{print \$1|\"sort -u | wc -l\"}" "${tmpf}"Here we should print the full lines, so without sort -u and wc. Sort is still
useful though.
Indeed. But will be removed in v2.
If the partial sha1 is ambiguous, let git checkout fail. Git checkout fails even when the sha1 of the commit is ambiguous to a sha1 of a tree.
BTW, the best real example I have so far between 2 refs is b3766 from
https://github.com/vim/vim.git .
Quoted 15 lines
support/download/git _git() { · line 67
Quoted 6 lines
support/download/git +5
-0+ exit 1 + fi + # Accept full or unambiguous partial sha1. A sha1 can be referenced by many names. + # Prefer sha1 of commit pointed by a tag or sha1 of the tag itself, + # then sha1 pointed by any ref/*, and only then sha1 pointed by *HEAD.I don't understand why this bit is needed. Any of the refs will do, right? It's
true that there is a risk that the ref gets updated between the ls-remote and the fetch, and that for tags the chance that it gets updated is smaller than for e.g. HEAD. But still, this is very much a corner case, and if it _does_ happen, we still fall back to full clone. So the only important thing is to avoid HEAD. Since ls-remotes already seems to sort the refs, which typically puts tags near the end, just take the last one and we're done:
Quoted 1 line
ref="$(cut -f 2 .git/matching_refs | tail -1)"
Yes. It is simpler.
I just noticed I missed to explain the awk -F'[\t^]' in the comments (or in the commit log) :/ .
Its is needed because annotated tags have their own sha1 and the commit pointed by a tag has ^{} at the end of the ref name in the ls-remote output.9395452b4aab7bc2475ef8935b4a4fb99d778d70 refs/tags/v4.8-rc6^{}
But after 'cut' I can use bash's 'Pattern substitution' ref=${ref/%"^{}"}
and, of course, add a nice comment for it.
[snip]
Quoted 3 lines
Quoted 1 line
+if [ ${do_a_shallow_fetch} -eq 1 ]; then
Actually, if [ "$ref" ] would be sufficient.
OK.
[snip]
Quoted 10 lines
support/download/git _git() { · line 95
Quoted 5 lines
support/download/git +3
-2-if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then - printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}" +git_checkout_done=0 +if [ ${git_fetch_done} -eq 1 ]; then + if _git checkout -q "'refs/buildroot/${cset}'" 2>&1; thenI'm not sure if I like this split.
git_fetch_donegets set in only one place,
so we could just do the checkout directly there. That would also make it more explicit that this is a checkout specifically for shallow fetches, while the checkout below is specifically for full fetches.
OK.
[snip]
Quoted 4 lines
support/download/git _git() { · line 105
Quoted 2 lines
support/download/git +2
-0+ _git remote add origin "'${repo}'" + _git fetch ${verbose} "${@}" originWhy do we need to create a remote here?
Actually, we don't.
I missed your comment in http://patchwork.ozlabs.org/patch/681841/
"just use $repo everywhere"
I will do this in v2.
I think I can do this too:
"Do it in two patches: first convert the current situation to fetches, then add the shallow download of tag SHAs."
Quoted 6 lines
support/download/git _git() { · line 107
Quoted 3 lines
support/download/git +2
-0+ _git checkout -q "'${cset}'" +fiAt a later stage, we could update the fetch with --submodules=yes/no depending
on -r option.
In theory, OK. But because of a bug in git maybe it's better to keep the current 'git submodule update --init --recursive'.
git fetch --help # git 2.10.2
BUGS
Using --recurse-submodules can only fetch new commits in already checked out submodules right now. When e.g. upstream added a new submodule in the just fetched commits of the superproject the submodule itself can not be fetched, making it impossible to check out that submodule later without having to do a fetch again. This is expected to be fixed in a future Git version.
Quoted 8 lines
Another interesting feature would be to first try a shallow fetch of all tags
and branches with a depth of, say, 100. That is still way less than a full clone for large repositories, and has a good chance to still capture a sha1 that is not at a branch or tag head. So it would provide a nice middle ground for the cases for which we fail to do a shallow clone now. And if it does fail, nothing is wasted because the full clone can start from the part that was already downloaded.
Yes. It should be useful especially for hardware-specific kernel repos when using sha1 as custom repo version.
The buildroot policy is to use preferably a tag, and if a tag cannot be used, use the sha1 in a branch, right?
So I think we will have the best gain by fetching only from branches in this case.
A quick test shows good results: make warp7_defconfig ; make linux-source
without this feature: 1.69 GiB
5148191
with this feature (100): 945.59 MiB + 73.89 KiB
1997387 + 138 = 1997525
with this feature (10+100): 402.04 MiB + 470.10 MiB
347225 + 32 + 1650245 = 1997502(I didn't dig to understand why the sum is not exactly the same)
Notice there is some extra processing in the server side ('Compressing objects') that takes some more seconds but I guess the bandwidth saved worth this.
I can add this patch (depth 100) at the end of the series as a RFC. This way people can comment/test/argue a different depth to use.
1) without this feature
Doing full clone
Cloning into 'linux-efaf36531fe7b1fc15a48033e5972825c91f9fc6'...
remote: Counting objects: 5148191, done.
remote: Compressing objects: 100% (395/395), done.
remote: Total 5148191 (delta 377), reused 183 (delta 183), pack-reused 5147613 Receiving objects: 100% (5148191/5148191), 1.69 GiB | 1.73 MiB/s, done.
2) with this feature, only for branches, depth 100
Doing shallow fetch of all branches
remote: Counting objects: 1997387, done.
remote: Compressing objects: 100% (453860/453860), done.
remote: Total 1997387 (delta 1605759), reused 1902796 (delta 1529294), pack-reused 0 Receiving objects: 100% (1997387/1997387), 945.59 MiB | 1.74 MiB/s, done.
...
remote: Counting objects: 138, done.
remote: Compressing objects: 100% (138/138), done.
remote: Total 138 (delta 0), reused 138 (delta 0), pack-reused 0 Receiving objects: 100% (138/138), 73.89 KiB | 0 bytes/s, done.
3) with this feature, only for branches, 1st step depth 10, 2nd step depth 100 Doing shallow fetch of all branches (10)
remote: Counting objects: 347225, done.
remote: Compressing objects: 100% (177196/177196), done.
remote: Total 347225 (delta 239120), reused 251422 (delta 167173), pack-reused 0 Receiving objects: 100% (347225/347225), 402.04 MiB | 1.56 MiB/s, done.
...
remote: Counting objects: 32, done.
remote: Compressing objects: 100% (32/32), done.
remote: Total 32 (delta 0), reused 32 (delta 0), pack-reused 0 Unpacking objects: 100% (32/32), done.
...
fatal: reference is not a tree: efaf36531fe7b1fc15a48033e5972825c91f9fc6 Doing shallow fetch of all branches (100)
remote: Counting objects: 1650245, done.
remote: Compressing objects: 100% (459302/459302), done.
remote: Total 1650245 (delta 1364443), reused 1452442 (delta 1179566), pack-reused 0 Receiving objects: 100% (1650245/1650245), 470.10 MiB | 1.68 MiB/s, done. Resolving deltas: 100% (1364443/1364443), completed with 38986 local objects. Checked out 'efaf36531fe7b1fc15a48033e5972825c91f9fc6'.
Regards,
Ricardo
PATCH 1/3 support/download/git: log checked out sha1
In reply to Ricardo MartincoskiQuoted 4 lines · Ricardo Martincoski
In preparation to have automated tests for this script, display the checked out sha1 in the build log.
Signed-off-by Ricardo Martincoski <[email protected]>
Quoted 12 lines
---
Another solution would be to use an environment variable set by the test to print that info, e.g. if [ "1" == "${GIT_DOWNLOAD_AUTOTEST}" ]; then--- support/download/git | 6 ++++++ 1 file changed, 6 insertions(+)support/download/git +0
-0
support/download/git if [ ${recurse} -eq 1 ]; then · line 89
Quoted 5 lines
support/download/git +2
-0@@ -85,6 +85,12 @@ if [ ${recurse} -eq 1 ]; then _git submodule update --init --recursive fi +# Log the sha1. It can be used for automated tests and debug. +if [ -z ${verbose} ]; then
It's quite weird to check that verbose is empty, but that's how the variable is
defined... Perhaps s/verbose/quiet/ would be a good idea...
Regards,
Arnout
Quoted 8 lines
support/download/git +4
-0+ sha1=$(_git rev-parse HEAD) + printf "Checked out '%s'.\n" "${sha1}" +fi + # We do not need the .git dir; we keep other .git files, in case they # are the only files in their directory. rm -rf .git
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH 1/3 support/download/git: log checked out sha1
Arnout,
Thank you for reviewing this series.
On Sat, Nov 05, 2016 at 07:50 PM, Arnout Vandecappelle wrote:
[snip]
Quoted 4 lines
Quoted 1 line
+if [ -z ${verbose} ]; then
It's quite weird to check that verbose is empty, but that's how the variable is
defined... Perhaps s/verbose/quiet/ would be a good idea...
OK, makes sense. I can send a followup patch.
Regards,
Ricardo