travis-ci: remove bogus 'pyenv' in the Linux jobs
RFC PATCH 1/1 travis-ci: remove bogus 'pyenv' in the Linux jobs
In our test suite, when 'git p4' invokes a Git command as a subprocesses, then it should run the 'git' binary we are testing. Unfortunately, this is not the case in the 'linux-clang' and 'linux-gcc' jobs on Travis CI, where 'git p4' runs the system '/usr/bin/git' instead.
Travis CI's default Linux image includes 'pyenv', and all Python invocations that involve PATH lookup go through 'pyenv', e.g. our 'PYTHON_PATH=$(which python3)' sets '/opt/pyenv/shims/python3' as PYTHON_PATH, which in turn will invoke '/usr/bin/python3'. Alas, the 'pyenv' version included in this image is buggy, and prepends the directory containing the Python binary to PATH even if that is a system directory already in PATH near the end. Consequently, 'git p4' in those jobs ends up with its PATH starting with '/usr/bin', and then runs '/usr/bin/git'.
So remove 'pyenv' in Travis CI's Linux jobs to prevet it from interfering with our 'git p4' tests.
---
Notes:
This issue triggered test failures in all 'git p4' test scripts in the recent GIT_TEST_DEFAULT_HASH=sha256 test runs, because the system Git doesn't understand the 'objectformat' extension, e.g.:
https://travis-ci.org/github/git/git/jobs/710159470#L3267
This is not an issue in Travis CI's macOS jobs or on Azure Pipelines, because they don't use 'pyenv'.
However, perhaps removing 'pyenv' is not the best solution here.
We set PYTHON_PATH in 'ci/lib.sh', which is sourced at the beginning of (almost) all of our CI scripts. Consequently, in these jobs we first run 'ci/install-dependencies.sh', which sources 'ci/lib.sh', assigns PYTHON_PATH=/opt/pyenv/shims/python3 (which is never used), and removes 'pyenv', and then run 'ci/run-build-and-test.sh', which sources 'ci/lib.sh' and assigns PYTHON_PATH=/usr/bin/python3 (which is then used in the build process to set 'pit p4's shebang line). Not really nice, is it.
Alternatively, we could avoid the PATH lookup and thus the bogus 'pyenv' by explicitly using '/usr/bin/python{2,3}'. The Linux images in both Travis CI and Azure Pipelines are standard Ubuntu images, so I think we can safely rely on these Python paths.
ci/install-dependencies.sh | 8 ++++++++ 1 file changed, 8 insertions(+)
ci/install-dependencies.sh +8 -0
@@ -36,6 +36,14 @@ linux-clang|linux-gcc) tar --extract --gunzip --file "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz" cp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs . popd + + if test true = "$TRAVIS" && + pyenv_root=$(pyenv root) + then + # pyenv in Travis CI's current default (xenial) Linux + # image messes up PATH for 'git p4'. + sudo rm -rf "$pyenv_root" + fi ;; osx-clang|osx-gcc) export HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1
Signature · 1 line
2.28.0.rc1.319.g12ef029c8a
Quoted 16 lines · SZEDER Gábor
In our test suite, when 'git p4' invokes a Git command as a subprocesses, then it should run the 'git' binary we are testing. Unfortunately, this is not the case in the 'linux-clang' and 'linux-gcc' jobs on Travis CI, where 'git p4' runs the system '
/usr/bin/git' instead.Travis CI's default Linux image includes 'pyenv', and all Python invocations that involve PATH lookup go through 'pyenv', e.g. our 'PYTHON_PATH=$(which python3)' sets '
/opt/pyenv/shims/python3' asPYTHON_PATH, which in turn will invoke '/usr/bin/python3'. Alas, the 'pyenv' version included in this image is buggy, and prepends the directory containing the Python binary to PATH even if that is a system directory already in PATH near the end. Consequently, 'git p4' in those jobs ends up with its PATH starting with '/usr/bin', and then runs '/usr/bin/git'.
Quoted 1 line
So remove 'pyenv' in Travis CI's Linux jobs to prevet it from
s/prevet/prevent/
ci/install-dependencies.sh linux-clang|linux-gcc) · line 41
Quoted 10 lines
interfering with our 'git p4' tests.
ci/install-dependencies.sh +6
-0@@ -36,6 +36,14 @@ linux-clang|linux-gcc) + + if test true = "$TRAVIS" && + pyenv_root=$(pyenv root) + then + # pyenv in Travis CI's current default (xenial) Linux + # image messes up PATH for 'git p4'.
I wonder if this comment should give more precise detail about exactly how it "messes up" PATH.
Quoted 2 lines
ci/install-dependencies.sh +2
-0+ sudo rm -rf "$pyenv_root" + fi
Quoted 4 lines · SZEDER Gábor
Alternatively, we could avoid the PATH lookup and thus the bogus 'pyenv' by explicitly using '
/usr/bin/python{2,3}'. The Linux images in both Travis CI and Azure Pipelines are standard Ubuntu images, so I think we can safely rely on these Python paths.
That sounds quite sensible, actually. The CI environment being "special" in the sense that we have certain control over it makes this rather straight-forward.
PATCH v2 1/1 ci: use absolute PYTHON_PATH in the Linux jobs
In our test suite, when 'git p4' invokes a Git command as a subprocesses, then it should run the 'git' binary we are testing. Unfortunately, this is not the case in the 'linux-clang' and 'linux-gcc' jobs on Travis CI, where 'git p4' runs the system '/usr/bin/git' instead.
Travis CI's default Linux image includes 'pyenv', and all Python invocations that involve PATH lookup go through 'pyenv', e.g. our 'PYTHON_PATH=$(which python3)' sets '/opt/pyenv/shims/python3' as PYTHON_PATH, which in turn will invoke '/usr/bin/python3'. Alas, the 'pyenv' version included in this image is buggy, and prepends the directory containing the Python binary to PATH even if that is a system directory already in PATH near the end. Consequently, 'git p4' in those jobs ends up with its PATH starting with '/usr/bin', and then runs '/usr/bin/git'.
So use the absolute paths '/usr/bin/python{2,3}' explicitly when setting PYTHON_PATH in those Linux jobs to avoid the PATH lookup and thus the bogus 'pyenv' from interfering with our 'git p4' tests. Don't bother with special-casing Travis CI: while this issue doesn't affect the corresponding Linux jobs on GitHub Actions, both CI systems use Ubuntu LTS-based images, so we can safely rely on these Python paths.
---
Junio, I see that you picked up the first/RFC version and applied it on top of v2.26.2. This patch won't work on v2.26.2, because its 'git p4' is not compatible with python3 yet.
ci/lib.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
ci/lib.sh +2 -2
@@ -184,9 +184,9 @@ linux-clang|linux-gcc) if [ "$jobname" = linux-gcc ] then export CC=gcc-8 - MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python3)" + MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3" else - MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python2)" + MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python2" fi export GIT_TEST_HTTPD=true
Signature · 1 line
2.28.0.rc2.319.g2e16345e2b
Re: [PATCH v2] ci: use absolute PYTHON_PATH in the Linux jobs
Quoted 16 lines · SZEDER Gábor
In our test suite, when 'git p4' invokes a Git command as a subprocesses, then it should run the 'git' binary we are testing. Unfortunately, this is not the case in the 'linux-clang' and 'linux-gcc' jobs on Travis CI, where 'git p4' runs the system '
/usr/bin/git' instead.Travis CI's default Linux image includes 'pyenv', and all Python invocations that involve PATH lookup go through 'pyenv', e.g. our 'PYTHON_PATH=$(which python3)' sets '
/opt/pyenv/shims/python3' asPYTHON_PATH, which in turn will invoke '/usr/bin/python3'. Alas, the 'pyenv' version included in this image is buggy, and prepends the directory containing the Python binary to PATH even if that is a system directory already in PATH near the end. Consequently, 'git p4' in those jobs ends up with its PATH starting with '/usr/bin', and then runs '/usr/bin/git'.
Quoted 7 lines
So use the absolute paths '
/usr/bin/python{2,3}' explicitly when settingPYTHON_PATHin those Linux jobs to avoid the PATH lookup and thus the bogus 'pyenv' from interfering with our 'git p4' tests. Don't bother with special-casing Travis CI: while this issue doesn't affect the corresponding Linux jobs on GitHub Actions, both CI systems use Ubuntu LTS-based images, so we can safely rely on these Python paths.
Good. This does not need to touch pyenv but take advantage of the fact that we know where Python 2 and 3 are. Nice.
Quoted 7 lines
Signed-off-by SZEDER Gábor <[email protected]>---
Junio, I see that you picked up the first/RFC version and applied it on top of v2.26.2. This patch won't work on v2.26.2, because its 'git p4' is not compatible with python3 yet.
Thanks. Which means that we do not need to touch maint track (not that we'd be merging the SHA-256 topic down to the maint track anyway).
Will queue.
Re: [PATCH v2] ci: use absolute PYTHON_PATH in the Linux jobs
Quoted 6 lines · Junio C Hamano
Quoted 3 lines · SZEDER Gábor
Junio, I see that you picked up the first/RFC version and applied it on top of v2.26.2. This patch won't work on v2.26.2, because its 'git p4' is not compatible with python3 yet.
Thanks. Which means that we do not need to touch maint track
Yeah, that system Git is v2.21.x, as far as I remember, but apparently 'git p4' doesn't use any shiny new features since then, because all its test succeeded even with that old version.
Quoted 3 lines
(not
that we'd be merging the SHA-256 topic down to the maint track anyway).
It might be worth applying the final SHA-256 topic on top of this, though, so Travis CI builds of that branch on its own can succeed.
Quoted 6 lines · SZEDER Gábor
In our test suite, when 'git p4' invokes a Git command as a subprocesses, then it should run the 'git' binary we are testing. Unfortunately, this is not the case in the 'linux-clang' and 'linux-gcc' jobs on Travis CI, where 'git p4' runs the system '
/usr/bin/git' instead.
Quoted 9 lines
Travis CI's default Linux image includes 'pyenv', and all Python invocations that involve PATH lookup go through 'pyenv', e.g. our 'PYTHON_PATH=$(which python3)' sets '
/opt/pyenv/shims/python3' asPYTHON_PATH, which in turn will invoke '/usr/bin/python3'. Alas, the 'pyenv' version included in this image is buggy, and prepends the directory containing the Python binary to PATH even if that is a system directory already in PATH near the end. Consequently, 'git p4' in those jobs ends up with its PATH starting with '/usr/bin', and then runs '/usr/bin/git'.
Ouch. So we're testing git-p4, but not the git binary underlying it. Fixing this definitely seems like a good idea.
Signature · 1 line
brian m. carlson: Houston, Texas, US