fetch2: git: Check if directory is a bare git repo
PATCH 1/1 fetch2: git: Check if directory is a bare git repo
If the clone target directory exists, but isn't the valid top level of a bare git repo, it needs to be erased and re-cloned. One example of how this can happen is if a clone creates the directory, but then fails to actual clone and make it a git repository. This left-over directory can be particularly problematic if the download directory is a descent of some top layer git repo (e.g. the default with poky), as the commands that operate on the remote later will then mangle the layers git repository instead of the download git repo.
--- bitbake/lib/bb/fetch2/git.py | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +23 -1
@@ -65,6 +65,7 @@ import fnmatch import os import re import shlex +import shutil import subprocess import tempfile import bb
@@ -365,8 +366,29 @@ class Git(FetchMethod): runfetchcmd(fetch_cmd, d, workdir=ud.clonedir) repourl = self._get_repo_url(ud) + needs_clone = False + if os.path.exists(ud.clonedir): + # The directory may exist, but not be the top level of a bare git + # repository in which case it needs to be deleted and re-cloned. + try: + # Since clones are bare, use --absolute-git-dir instead of --show-toplevel + output = runfetchcmd("LANG=C %s rev-parse --absolute-git-dir" % ud.basecmd, d, workdir=ud.clonedir) + except bb.fetch2.FetchError as e: + logger.warning("Unable to get top level for %s (not a git directory?): %s", ud.clonedir, e) + needs_clone = True + else: + toplevel = output.rstrip() + if os.path.abspath(toplevel) != os.path.abspath(ud.clonedir): + logger.warning("Top level directory '%s' doesn't match expected '%s'. Re-cloning", toplevel, ud.clonedir) + needs_clone = True + + if needs_clone: + shutil.rmtree(ud.clonedir) + else: + needs_clone = True + # If the repo still doesn't exist, fallback to cloning it - if not os.path.exists(ud.clonedir): + if needs_clone: # We do this since git will use a "-l" option automatically for local urls where possible, # but it doesn't work when git/objects is a symlink, only works when it is a directory. if repourl.startswith("file://"):
Signature · 1 line
2.33.0
Hello Joshua,
I'm fairly sure this causes the following bitbake-selftest failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/2307/steps/11/logs/stdio
Quoted 70 lines · Joshua Watt
If the clone target directory exists, but isn't the valid top level of a bare git repo, it needs to be erased and re-cloned. One example of how this can happen is if a clone creates the directory, but then fails to actual clone and make it a git repository. This left-over directory can be particularly problematic if the download directory is a descent of some top layer git repo (e.g. the default with poky), as the commands that operate on the remote later will then mangle the layers git repository instead of the download git repo.
Signed-off-by Joshua Watt <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +23
-1@@ -65,6 +65,7 @@ import fnmatch import os import re import shlex +import shutil import subprocess import tempfile import bb@@ -365,8 +366,29 @@ class Git(FetchMethod): runfetchcmd(fetch_cmd, d, workdir=ud.clonedir) repourl = self._get_repo_url(ud) + needs_clone = False + if os.path.exists(ud.clonedir): + # The directory may exist, but not be the top level of a bare git + # repository in which case it needs to be deleted and re-cloned. + try: + # Since clones are bare, use --absolute-git-dir instead of --show-toplevel + output = runfetchcmd("LANG=C %s rev-parse --absolute-git-dir" % ud.basecmd, d, workdir=ud.clonedir) + except bb.fetch2.FetchError as e: + logger.warning("Unable to get top level for %s (not a git directory?): %s", ud.clonedir, e) + needs_clone = True + else: + toplevel = output.rstrip() + if os.path.abspath(toplevel) != os.path.abspath(ud.clonedir): + logger.warning("Top level directory '%s' doesn't match expected '%s'. Re-cloning", toplevel, ud.clonedir) + needs_clone = True + + if needs_clone: + shutil.rmtree(ud.clonedir) + else: + needs_clone = True + # If the repo still doesn't exist, fallback to cloning it - if not os.path.exists(ud.clonedir): + if needs_clone: # We do this since git will use a "-l" option automatically for local urls where possible, # but it doesn't work when git/objects is a symlink, only works when it is a directory. if repourl.startswith("file://"):Signature · 11 lines
2.33.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14915): https://lists.openembedded.org/g/bitbake-devel/message/14915 Mute This Topic: https://lists.openembedded.org/mt/100548945/3617179 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Alexandre,
I'm not sure that's quite correct; I was able to run the test fine locally, and that failure looks like a 502 error from the server, which really shouldn't have anything to do with this patch?
Quoted 84 lines · Alexandre Belloni
Hello Joshua,
I'm fairly sure this causes the following bitbake-selftest failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/2307/steps/11/logs/stdio
Quoted 62 lines · Joshua Watt
If the clone target directory exists, but isn't the valid top level of a bare git repo, it needs to be erased and re-cloned. One example of how this can happen is if a clone creates the directory, but then fails to actual clone and make it a git repository. This left-over directory can be particularly problematic if the download directory is a descent of some top layer git repo (e.g. the default with poky), as the commands that operate on the remote later will then mangle the layers git repository instead of the download git repo.
Signed-off-by Joshua Watt <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +23
-2@@ -65,6 +65,7 @@ import fnmatch import os import re import shlex +import shutil import subprocess import tempfile import bb@@ -365,8 +366,29 @@ class Git(FetchMethod): runfetchcmd(fetch_cmd, d, workdir=ud.clonedir) repourl = self._get_repo_url(ud) + needs_clone = False + if os.path.exists(ud.clonedir): + # The directory may exist, but not be the top level of a bare git + # repository in which case it needs to be deleted and re-cloned. + try: + # Since clones are bare, use --absolute-git-dir instead of --show-toplevel + output = runfetchcmd("LANG=C %s rev-parse --absolute-git-dir" % ud.basecmd, d, workdir=ud.clonedir) + except bb.fetch2.FetchError as e: + logger.warning("Unable to get top level for %s (not a git directory?): %s", ud.clonedir, e) + needs_clone = True + else: + toplevel = output.rstrip() + if os.path.abspath(toplevel) != os.path.abspath(ud.clonedir): + logger.warning("Top level directory '%s' doesn't match expected '%s'. Re-cloning", toplevel, ud.clonedir) + needs_clone = True + + if needs_clone: + shutil.rmtree(ud.clonedir) + else: + needs_clone = True + # If the repo still doesn't exist, fallback to cloning it - if not os.path.exists(ud.clonedir): + if needs_clone: # We do this since git will use a "-l" option automatically for local urls where possible, # but it doesn't work when git/objects is a symlink, only works when it is a directory. if repourl.startswith("file://"): --2.33.0
Quoted 8 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14915): https://lists.openembedded.org/g/bitbake-devel/message/14915
Mute This Topic: https://lists.openembedded.org/mt/100548945/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=---
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Quoted 6 lines · Joshua Watt
Alexandre,
I'm not sure that's quite correct; I was able to run the test fine locally, and that failure looks like a 502 error from the server, which really shouldn't have anything to do with this patch?
I'm pretty sure you are right...
Quoted 86 lines
Quoted 84 lines · Alexandre Belloni
Hello Joshua,
I'm fairly sure this causes the following bitbake-selftest failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/2307/steps/11/logs/stdio
Quoted 62 lines · Joshua Watt
If the clone target directory exists, but isn't the valid top level of a bare git repo, it needs to be erased and re-cloned. One example of how this can happen is if a clone creates the directory, but then fails to actual clone and make it a git repository. This left-over directory can be particularly problematic if the download directory is a descent of some top layer git repo (e.g. the default with poky), as the commands that operate on the remote later will then mangle the layers git repository instead of the download git repo.
Signed-off-by Joshua Watt <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +23
-2@@ -65,6 +65,7 @@ import fnmatch import os import re import shlex +import shutil import subprocess import tempfile import bb@@ -365,8 +366,29 @@ class Git(FetchMethod): runfetchcmd(fetch_cmd, d, workdir=ud.clonedir) repourl = self._get_repo_url(ud) + needs_clone = False + if os.path.exists(ud.clonedir): + # The directory may exist, but not be the top level of a bare git + # repository in which case it needs to be deleted and re-cloned. + try: + # Since clones are bare, use --absolute-git-dir instead of --show-toplevel + output = runfetchcmd("LANG=C %s rev-parse --absolute-git-dir" % ud.basecmd, d, workdir=ud.clonedir) + except bb.fetch2.FetchError as e: + logger.warning("Unable to get top level for %s (not a git directory?): %s", ud.clonedir, e) + needs_clone = True + else: + toplevel = output.rstrip() + if os.path.abspath(toplevel) != os.path.abspath(ud.clonedir): + logger.warning("Top level directory '%s' doesn't match expected '%s'. Re-cloning", toplevel, ud.clonedir) + needs_clone = True + + if needs_clone: + shutil.rmtree(ud.clonedir) + else: + needs_clone = True + # If the repo still doesn't exist, fallback to cloning it - if not os.path.exists(ud.clonedir): + if needs_clone: # We do this since git will use a "-l" option automatically for local urls where possible, # but it doesn't work when git/objects is a symlink, only works when it is a directory. if repourl.startswith("file://"): --2.33.0
Quoted 8 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14915): https://lists.openembedded.org/g/bitbake-devel/message/14915
Mute This Topic: https://lists.openembedded.org/mt/100548945/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=---
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
This was because I didn't point to the correct build error:
======================================================================
ERROR: test_mirror_commit_exists (bb.tests.fetch.FetchPremirroronlyLocalTest.test_mirror_commit_exists)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 3075, in test_mirror_commit_exists
fetcher.download()
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/fetch2/__init__.py", line 1746, in download
done = m.try_mirrors(self, ud, self.d, mirrors)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/fetch2/__init__.py", line 1612, in try_mirrors
return bool(try_mirrors(fetch, d, urldata, mirrors, check))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/fetch2/__init__.py", line 1136, in try_mirrors
ret = try_mirror_url(fetch, origud, uds[index], ld, check)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/fetch2/__init__.py", line 1065, in try_mirror_url
origud.method.download(origud, ld)
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/fetch2/git.py", line 401, in download
bb.fetch2.check_network_access(d, clone_cmd, ud.url)
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/fetch2/__init__.py", line 944, in check_network_access
raise NetworkAccess(url, info)
bb.fetch2.NetworkAccess: Network access disabled through BB_NO_NETWORK (or set indirectly due to use of BB_FETCH_PREMIRRORONLY) but access requested with command LANG=C git -c gc.autoDetach=false -c core.pager=cat clone --bare --mirror https://git.fake.repo/bitbake /tmp/bitbake-fetch-xeon98ff/download/git2/git.fake.repo.bitbake --progress (for url git://git.fake.repo/bitbake;branch=master;protocol=https)
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5619/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5577/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/1935/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5653/steps/11/logs/stdio
Quoted 105 lines · Alexandre Belloni via lists.openembedded.org
Quoted 6 lines · Joshua Watt
Alexandre,
I'm not sure that's quite correct; I was able to run the test fine locally, and that failure looks like a 502 error from the server, which really shouldn't have anything to do with this patch?
I'm pretty sure you are right...
Quoted 80 lines
Quoted 78 lines · Alexandre Belloni
Hello Joshua,
I'm fairly sure this causes the following bitbake-selftest failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/2307/steps/11/logs/stdio
Quoted 58 lines · Joshua Watt
If the clone target directory exists, but isn't the valid top level of a bare git repo, it needs to be erased and re-cloned. One example of how this can happen is if a clone creates the directory, but then fails to actual clone and make it a git repository. This left-over directory can be particularly problematic if the download directory is a descent of some top layer git repo (e.g. the default with poky), as the commands that operate on the remote later will then mangle the layers git repository instead of the download git repo.
Signed-off-by Joshua Watt <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +23
-2@@ -65,6 +65,7 @@ import fnmatch import os import re import shlex +import shutil import subprocess import tempfile import bb@@ -365,8 +366,29 @@ class Git(FetchMethod): runfetchcmd(fetch_cmd, d, workdir=ud.clonedir) repourl = self._get_repo_url(ud) + needs_clone = False + if os.path.exists(ud.clonedir): + # The directory may exist, but not be the top level of a bare git + # repository in which case it needs to be deleted and re-cloned. + try: + # Since clones are bare, use --absolute-git-dir instead of --show-toplevel + output = runfetchcmd("LANG=C %s rev-parse --absolute-git-dir" % ud.basecmd, d, workdir=ud.clonedir) + except bb.fetch2.FetchError as e: + logger.warning("Unable to get top level for %s (not a git directory?): %s", ud.clonedir, e) + needs_clone = True + else: + toplevel = output.rstrip() + if os.path.abspath(toplevel) != os.path.abspath(ud.clonedir): + logger.warning("Top level directory '%s' doesn't match expected '%s'. Re-cloning", toplevel, ud.clonedir) + needs_clone = True + + if needs_clone: + shutil.rmtree(ud.clonedir) + else: + needs_clone = True + # If the repo still doesn't exist, fallback to cloning it - if not os.path.exists(ud.clonedir): + if needs_clone: # We do this since git will use a "-l" option automatically for local urls where possible, # but it doesn't work when git/objects is a symlink, only works when it is a directory. if repourl.startswith("file://"): --Quoted 6 lines
2.33.0
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.comSignature · 12 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14952): https://lists.openembedded.org/g/bitbake-devel/message/14952 Mute This Topic: https://lists.openembedded.org/mt/100548945/3617179 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 2 lines · Alexandre Belloni
This was because I didn't point to the correct build error:
Yep, fixed in V2 (and I validated that the tests pass locally also)
Quoted 143 lines
======================================================================
ERROR:test_mirror_commit_exists(bb.tests.fetch.FetchPremirroronlyLocalTest.test_mirror_commit_exists)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 3075, intest_mirror_commit_exists
fetcher.download()
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/fetch2/__init__.py", line 1746, in download
done =m.try_mirrors(self, ud, self.d, mirrors)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/fetch2/__init__.py", line 1612, intry_mirrors
returnbool(try_mirrors(fetch, d, urldata, mirrors, check))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/fetch2/__init__.py", line 1136, intry_mirrors
ret =try_mirror_url(fetch, origud, uds[index], ld, check)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/fetch2/__init__.py", line 1065, intry_mirror_url
origud.method.download(origud, ld)
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/fetch2/git.py", line 401, in download
bb.fetch2.check_network_access(d, clone_cmd, ud.url)
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/fetch2/__init__.py", line 944, incheck_network_access
raiseNetworkAccess(url, info)
bb.fetch2.NetworkAccess: Network access disabled throughBB_NO_NETWORK(or set indirectly due to use ofBB_FETCH_PREMIRRORONLY) but access requested with command LANG=C git -c gc.autoDetach=false -c core.pager=cat clone --bare --mirror https://git.fake.repo/bitbake/tmp/bitbake-fetch-xeon98ff/download/git2/git.fake.repo.bitbake--progress (for urlgit://git.fake.repo/bitbake;branch=master;protocol=https)https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5619/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5577/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/1935/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5653/steps/11/logs/stdioQuoted 97 lines · Alexandre Belloni via lists.openembedded.org
Quoted 6 lines · Joshua Watt
Alexandre,
I'm not sure that's quite correct; I was able to run the test fine locally, and that failure looks like a 502 error from the server, which really shouldn't have anything to do with this patch?
I'm pretty sure you are right...
Quoted 76 lines
Quoted 74 lines · Alexandre Belloni
Hello Joshua,
I'm fairly sure this causes the following bitbake-selftest failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/2307/steps/11/logs/stdio
Quoted 64 lines · Joshua Watt
If the clone target directory exists, but isn't the valid top level of a bare git repo, it needs to be erased and re-cloned. One example of how this can happen is if a clone creates the directory, but then fails to actual clone and make it a git repository. This left-over directory can be particularly problematic if the download directory is a descent of some top layer git repo (e.g. the default with poky), as the commands that operate on the remote later will then mangle the layers git repository instead of the download git repo.
Signed-off-by Joshua Watt <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +23
-2@@ -65,6 +65,7 @@ import fnmatch import os import re import shlex +import shutil import subprocess import tempfile import bb@@ -365,8 +366,29 @@ class Git(FetchMethod): runfetchcmd(fetch_cmd, d, workdir=ud.clonedir) repourl = self._get_repo_url(ud) + needs_clone = False + if os.path.exists(ud.clonedir): + # The directory may exist, but not be the top level of a bare git + # repository in which case it needs to be deleted and re-cloned. + try: + # Since clones are bare, use --absolute-git-dir instead of --show-toplevel + output = runfetchcmd("LANG=C %s rev-parse --absolute-git-dir" % ud.basecmd, d, workdir=ud.clonedir) + except bb.fetch2.FetchError as e: + logger.warning("Unable to get top level for %s (not a git directory?): %s", ud.clonedir, e) + needs_clone = True + else: + toplevel = output.rstrip() + if os.path.abspath(toplevel) != os.path.abspath(ud.clonedir): + logger.warning("Top level directory '%s' doesn't match expected '%s'. Re-cloning", toplevel, ud.clonedir) + needs_clone = True + + if needs_clone: + shutil.rmtree(ud.clonedir) + else: + needs_clone = True + # If the repo still doesn't exist, fallback to cloning it - if not os.path.exists(ud.clonedir): + if needs_clone: # We do this since git will use a "-l" option automatically for local urls where possible, # but it doesn't work when git/objects is a symlink, only works when it is a directory. if repourl.startswith("file://"): --2.33.0
Quoted 4 lines
Quoted 4 lines
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.comQuoted 8 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14952): https://lists.openembedded.org/g/bitbake-devel/message/14952
Mute This Topic: https://lists.openembedded.org/mt/100548945/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=---
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
PATCH v2 1/1 fetch2: git: Check if clone directory is a git repo
In reply to Joshua WattIf the clone target directory exists and is a valid git repo, but the git directory is not a child, it needs to be erased and re-cloned. One example of how this can happen is if a clone creates the directory, but then fails to actual clone and make it a git repository. This left-over directory can be particularly problematic if the download directory is a descent of some top layer git repo (e.g. the default with poky), as the commands that operate on the remote later will then mangle the layers git repository instead of the download git repo.
--- bitbake/lib/bb/fetch2/git.py | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +29 -1
@@ -65,6 +65,7 @@ import fnmatch import os import re import shlex +import shutil import subprocess import tempfile import bb
@@ -365,8 +366,35 @@ class Git(FetchMethod): runfetchcmd(fetch_cmd, d, workdir=ud.clonedir) repourl = self._get_repo_url(ud) + needs_clone = False + if os.path.exists(ud.clonedir): + # The directory may exist, but not be the top level of a bare git + # repository in which case it needs to be deleted and re-cloned. + try: + # Since clones can be bare, use --absolute-git-dir instead of --show-toplevel + output = runfetchcmd("LANG=C %s rev-parse --absolute-git-dir" % ud.basecmd, d, workdir=ud.clonedir) + except bb.fetch2.FetchError as e: + logger.warning("Unable to get top level for %s (not a git directory?): %s", ud.clonedir, e) + needs_clone = True + else: + toplevel = os.path.abspath(output.rstrip()) + abs_clonedir = os.path.abspath(ud.clonedir).rstrip('/') + # The top level Git directory must either be the clone directory + # or a child of the clone directory. Any ancestor directory of + # the clone directory is not valid as the Git directory (and + # probably belongs to some other unrelated repository), so a + # clone is required + if os.path.commonprefix([abs_clonedir, toplevel]) != abs_clonedir: + logger.warning("Top level directory '%s' doesn't match expected '%s'. Re-cloning", toplevel, ud.clonedir) + needs_clone = True + + if needs_clone: + shutil.rmtree(ud.clonedir) + else: + needs_clone = True + # If the repo still doesn't exist, fallback to cloning it - if not os.path.exists(ud.clonedir): + if needs_clone: # We do this since git will use a "-l" option automatically for local urls where possible, # but it doesn't work when git/objects is a symlink, only works when it is a directory. if repourl.startswith("file://"):
Signature · 1 line
2.33.0
Re: [bitbake-devel][PATCH v2] fetch2: git: Check if clone directory is a git repo
Quoted 67 lines · Joshua Watt
If the clone target directory exists and is a valid git repo, but the git directory is not a child, it needs to be erased and re-cloned. One example of how this can happen is if a clone creates the directory, but then fails to actual clone and make it a git repository. This left-over directory can be particularly problematic if the download directory is a descent of some top layer git repo (e.g. the default with poky), as the commands that operate on the remote later will then mangle the layers git repository instead of the download git repo.
Signed-off-by Joshua Watt <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +29
-2@@ -65,6 +65,7 @@ import fnmatch import os import re import shlex +import shutil import subprocess import tempfile import bb@@ -365,8 +366,35 @@ class Git(FetchMethod): runfetchcmd(fetch_cmd, d, workdir=ud.clonedir) repourl = self._get_repo_url(ud) + needs_clone = False + if os.path.exists(ud.clonedir): + # The directory may exist, but not be the top level of a bare git + # repository in which case it needs to be deleted and re-cloned. + try: + # Since clones can be bare, use --absolute-git-dir instead of --show-toplevel + output = runfetchcmd("LANG=C %s rev-parse --absolute-git-dir" % ud.basecmd, d, workdir=ud.clonedir) + except bb.fetch2.FetchError as e: + logger.warning("Unable to get top level for %s (not a git directory?): %s", ud.clonedir, e) + needs_clone = True + else: + toplevel = os.path.abspath(output.rstrip()) + abs_clonedir = os.path.abspath(ud.clonedir).rstrip('/') + # The top level Git directory must either be the clone directory + # or a child of the clone directory. Any ancestor directory of + # the clone directory is not valid as the Git directory (and + # probably belongs to some other unrelated repository), so a + # clone is required + if os.path.commonprefix([abs_clonedir, toplevel]) != abs_clonedir: + logger.warning("Top level directory '%s' doesn't match expected '%s'. Re-cloning", toplevel, ud.clonedir) + needs_clone = True + + if needs_clone: + shutil.rmtree(ud.clonedir) + else: + needs_clone = True + # If the repo still doesn't exist, fallback to cloning it - if not os.path.exists(ud.clonedir): + if needs_clone: # We do this since git will use a "-l" option automatically for local urls where possible, # but it doesn't work when git/objects is a symlink, only works when it is a directory. if repourl.startswith("file://"): --2.33.0
Quoted 8 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14979): https://lists.openembedded.org/g/bitbake-devel/message/14979
Mute This Topic: https://lists.openembedded.org/mt/100942876/4454782
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Is it possible to factor the try/except else to a function? Then we would not need the else, and would just return immediately and have a self contained code, which in my opinion is easier to read. I also think f-string would be more readable for runfetchcmd. f-string are available since python 3.6.
Can you share the rationale for the else clause? I do not see anything inside the else clause that might raise a FetchError, thus the else clause is redundant.
Another change I suggest is to move the rmtree inside the usual need_clone and ignore the error if the path does not exist. Removes another conditional.
I also think using needs_clone to assert whether the rmtree should be done is confusing A different variable should be used or use a function's return value. All the code inside if os.path.exists(ud.clonedir): deals with the need for removal and as a side effect will trigger the clone in another part of the code.
Re: [bitbake-devel][PATCH v2] fetch2: git: Check if clone directory is a git repo
Quoted 83 lines · Paulo Neves
Quoted 67 lines · Joshua Watt
If the clone target directory exists and is a valid git repo, but the git directory is not a child, it needs to be erased and re-cloned. One example of how this can happen is if a clone creates the directory, but then fails to actual clone and make it a git repository. This left-over directory can be particularly problematic if the download directory is a descent of some top layer git repo (e.g. the default with poky), as the commands that operate on the remote later will then mangle the layers git repository instead of the download git repo.
Signed-off-by Joshua Watt <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +29
-2@@ -65,6 +65,7 @@ import fnmatch import os import re import shlex +import shutil import subprocess import tempfile import bb@@ -365,8 +366,35 @@ class Git(FetchMethod): runfetchcmd(fetch_cmd, d, workdir=ud.clonedir) repourl = self._get_repo_url(ud) + needs_clone = False + if os.path.exists(ud.clonedir): + # The directory may exist, but not be the top level of a bare git + # repository in which case it needs to be deleted and re-cloned. + try: + # Since clones can be bare, use --absolute-git-dir instead of --show-toplevel + output = runfetchcmd("LANG=C %s rev-parse --absolute-git-dir" % ud.basecmd, d, workdir=ud.clonedir) + except bb.fetch2.FetchError as e: + logger.warning("Unable to get top level for %s (not a git directory?): %s", ud.clonedir, e) + needs_clone = True + else: + toplevel = os.path.abspath(output.rstrip()) + abs_clonedir = os.path.abspath(ud.clonedir).rstrip('/') + # The top level Git directory must either be the clone directory + # or a child of the clone directory. Any ancestor directory of + # the clone directory is not valid as the Git directory (and + # probably belongs to some other unrelated repository), so a + # clone is required + if os.path.commonprefix([abs_clonedir, toplevel]) != abs_clonedir: + logger.warning("Top level directory '%s' doesn't match expected '%s'. Re-cloning", toplevel, ud.clonedir) + needs_clone = True + + if needs_clone: + shutil.rmtree(ud.clonedir) + else: + needs_clone = True + # If the repo still doesn't exist, fallback to cloning it - if not os.path.exists(ud.clonedir): + if needs_clone: # We do this since git will use a "-l" option automatically for local urls where possible, # but it doesn't work when git/objects is a symlink, only works when it is a directory. if repourl.startswith("file://"): --2.33.0
Quoted 8 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14979): https://lists.openembedded.org/g/bitbake-devel/message/14979
Mute This Topic: https://lists.openembedded.org/mt/100942876/4454782
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-Is it possible to factor the try/except else to a function? Then we would not need the else, and would just return immediately and have a self contained code, which in my opinion is easier to read. I also think f-string would be more readable for runfetchcmd. f-string are available since python 3.6.
Quoted 3 lines
Can you share the rationale for the else clause? I do not see anything inside the else clause that might raise a FetchError, thus the else clause is redundant.
The else clause is pretty much the entire reason for this patch in the first place :)
The problem is that the fetcher assumes that the simple existence of the download directory means it can can add a remote and clone there, but that is not true. If your clone directory is a subdirectory of another git repository, but not a git repository itself, bitbake will (incorrectly) add the remote the higher level git repos and fetch there, leaving it a mess.
More visually, if your layout is like this:
poky <-- This is a git repository
build
downloads
git2
foo <-- This is NOT a git repository
The old code would see that the foo directory existed, see that it was part of a git repo (poky!) add the remotes and start cloning objects into the poky git repo, which is not at all what we want.
The else clause corrects this as it makes the fetcher check if "foo" is not only a git repo, but that the it is the toplevel of a git repo (or at least that it's --git-dir is not a parent of that directory). If it's not, it deletes it because it's not a valid place to clone things into anymore.
Quoted 13 lines
Another change I suggest is to move the rmtree inside the usual
need_cloneand ignore the error if the path does not exist. Removes another conditional.I also think using
needs_cloneto assert whether the rmtree should be done is confusing A different variable should be used or use a function's return value. All the code inside ifos.path.exists(ud.clonedir): deals with the need for removal and as a side effect will trigger the clone in another part of the code.
Re: [bitbake-devel][PATCH v2] fetch2: git: Check if clone directory is a git repo
Quoted 102 lines · Joshua Watt
Quoted 76 lines · Paulo Neves
Quoted 64 lines · Joshua Watt
If the clone target directory exists and is a valid git repo, but the git directory is not a child, it needs to be erased and re-cloned. One example of how this can happen is if a clone creates the directory, but then fails to actual clone and make it a git repository. This left-over directory can be particularly problematic if the download directory is a descent of some top layer git repo (e.g. the default with poky), as the commands that operate on the remote later will then mangle the layers git repository instead of the download git repo.
Signed-off-by Joshua Watt <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +29
-2@@ -65,6 +65,7 @@ import fnmatch import os import re import shlex +import shutil import subprocess import tempfile import bb@@ -365,8 +366,35 @@ class Git(FetchMethod): runfetchcmd(fetch_cmd, d, workdir=ud.clonedir) repourl = self._get_repo_url(ud) + needs_clone = False + if os.path.exists(ud.clonedir): + # The directory may exist, but not be the top level of a bare git + # repository in which case it needs to be deleted and re-cloned. + try: + # Since clones can be bare, use --absolute-git-dir instead of --show-toplevel + output = runfetchcmd("LANG=C %s rev-parse --absolute-git-dir" % ud.basecmd, d, workdir=ud.clonedir) + except bb.fetch2.FetchError as e: + logger.warning("Unable to get top level for %s (not a git directory?): %s", ud.clonedir, e) + needs_clone = True + else: + toplevel = os.path.abspath(output.rstrip()) + abs_clonedir = os.path.abspath(ud.clonedir).rstrip('/') + # The top level Git directory must either be the clone directory + # or a child of the clone directory. Any ancestor directory of + # the clone directory is not valid as the Git directory (and + # probably belongs to some other unrelated repository), so a + # clone is required + if os.path.commonprefix([abs_clonedir, toplevel]) != abs_clonedir: + logger.warning("Top level directory '%s' doesn't match expected '%s'. Re-cloning", toplevel, ud.clonedir) + needs_clone = True + + if needs_clone: + shutil.rmtree(ud.clonedir) + else: + needs_clone = True + # If the repo still doesn't exist, fallback to cloning it - if not os.path.exists(ud.clonedir): + if needs_clone: # We do this since git will use a "-l" option automatically for local urls where possible, # but it doesn't work when git/objects is a symlink, only works when it is a directory. if repourl.startswith("file://"): --Quoted 5 lines
2.33.0
Is it possible to factor the try/except else to a function? Then we would not need the else, and would just return immediately and have a self contained code, which in my opinion is easier to read. I also think f-string would be more readable for runfetchcmd. f-string are available since python 3.6.
Quoted 3 lines
Can you share the rationale for the else clause? I do not see anything inside the else clause that might raise a FetchError, thus the else clause is redundant.
The else clause is pretty much the entire reason for this patch in the first place :)
The problem is that the fetcher assumes that the simple existence of the download directory means it can can add a remote and clone there, but that is not true. If your clone directory is a subdirectory of another git repository, but not a git repository itself, bitbake will (incorrectly) add the remote the higher level git repos and fetch there, leaving it a mess.
More visually, if your layout is like this:
poky <-- This is a git repository
build
downloads
git2
foo <-- This is NOT a git repositoryThe old code would see that the foo directory existed, see that it was part of a git repo (poky!) add the remotes and start cloning objects into the poky git repo, which is not at all what we want.
Quoted 5 lines
The else clause corrects this as it makes the fetcher check if "foo" is not only a git repo, but that the it is the toplevel of a git repo (or at least that it's --git-dir is not a parent of that directory). If it's not, it deletes it because it's not a valid place to clone things into anymore.
I understand the conceptual point of the patch and I think it is valid. What i mean is that you do not need the else clause. The code under the else clause can continue inside the try clause and not change the logic. The try-else idiom is very unusual and represents a meaning which is not needed here: That the code inside the else should not trigger the except clause.
Quoted 20 lines
Quoted 4 lines
Another change I suggest is to move the rmtree inside the usual
need_cloneand ignore the error if the path does not exist. Removes another conditional.Quoted 8 lines
I also think using
needs_cloneto assert whether the rmtree should be done is confusing A different variable should be used or use a function's return value. All the code inside ifos.path.exists(ud.clonedir): deals with the need for removal and as a side effect will trigger the clone in another part of the code.-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14984): https://lists.openembedded.org/g/bitbake-devel/message/14984
Mute This Topic: https://lists.openembedded.org/mt/100942876/4454782
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [bitbake-devel][PATCH v2] fetch2: git: Check if clone directory is a git repo
Quoted 110 lines · Paulo Neves
Quoted 102 lines · Joshua Watt
Quoted 76 lines · Paulo Neves
Quoted 64 lines · Joshua Watt
If the clone target directory exists and is a valid git repo, but the git directory is not a child, it needs to be erased and re-cloned. One example of how this can happen is if a clone creates the directory, but then fails to actual clone and make it a git repository. This left-over directory can be particularly problematic if the download directory is a descent of some top layer git repo (e.g. the default with poky), as the commands that operate on the remote later will then mangle the layers git repository instead of the download git repo.
Signed-off-by Joshua Watt <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +29
-2@@ -65,6 +65,7 @@ import fnmatch import os import re import shlex +import shutil import subprocess import tempfile import bb@@ -365,8 +366,35 @@ class Git(FetchMethod): runfetchcmd(fetch_cmd, d, workdir=ud.clonedir) repourl = self._get_repo_url(ud) + needs_clone = False + if os.path.exists(ud.clonedir): + # The directory may exist, but not be the top level of a bare git + # repository in which case it needs to be deleted and re-cloned. + try: + # Since clones can be bare, use --absolute-git-dir instead of --show-toplevel + output = runfetchcmd("LANG=C %s rev-parse --absolute-git-dir" % ud.basecmd, d, workdir=ud.clonedir) + except bb.fetch2.FetchError as e: + logger.warning("Unable to get top level for %s (not a git directory?): %s", ud.clonedir, e) + needs_clone = True + else: + toplevel = os.path.abspath(output.rstrip()) + abs_clonedir = os.path.abspath(ud.clonedir).rstrip('/') + # The top level Git directory must either be the clone directory + # or a child of the clone directory. Any ancestor directory of + # the clone directory is not valid as the Git directory (and + # probably belongs to some other unrelated repository), so a + # clone is required + if os.path.commonprefix([abs_clonedir, toplevel]) != abs_clonedir: + logger.warning("Top level directory '%s' doesn't match expected '%s'. Re-cloning", toplevel, ud.clonedir) + needs_clone = True + + if needs_clone: + shutil.rmtree(ud.clonedir) + else: + needs_clone = True + # If the repo still doesn't exist, fallback to cloning it - if not os.path.exists(ud.clonedir): + if needs_clone: # We do this since git will use a "-l" option automatically for local urls where possible, # but it doesn't work when git/objects is a symlink, only works when it is a directory. if repourl.startswith("file://"): --Quoted 5 lines
2.33.0
Is it possible to factor the try/except else to a function? Then we would not need the else, and would just return immediately and have a self contained code, which in my opinion is easier to read. I also think f-string would be more readable for runfetchcmd. f-string are available since python 3.6.
Quoted 3 lines
Can you share the rationale for the else clause? I do not see anything inside the else clause that might raise a FetchError, thus the else clause is redundant.
The else clause is pretty much the entire reason for this patch in the first place :)
The problem is that the fetcher assumes that the simple existence of the download directory means it can can add a remote and clone there, but that is not true. If your clone directory is a subdirectory of another git repository, but not a git repository itself, bitbake will (incorrectly) add the remote the higher level git repos and fetch there, leaving it a mess.
More visually, if your layout is like this:
poky <-- This is a git repository
build
downloads
git2
foo <-- This is NOT a git repositoryThe old code would see that the foo directory existed, see that it was part of a git repo (poky!) add the remotes and start cloning objects into the poky git repo, which is not at all what we want.
Quoted 5 lines
The else clause corrects this as it makes the fetcher check if "foo" is not only a git repo, but that the it is the toplevel of a git repo (or at least that it's --git-dir is not a parent of that directory). If it's not, it deletes it because it's not a valid place to clone things into anymore.
Quoted 6 lines
I understand the conceptual point of the patch and I think it is valid. What i mean is that you do not need the else clause. The code under the else clause can continue inside the try clause and not change the logic. The try-else idiom is very unusual and represents a meaning which is not needed here: That the code inside the else should not trigger the except clause.
Ah, yes. I see what you are saying now
Quoted 23 lines
Quoted 21 lines
Quoted 5 lines
Another change I suggest is to move the rmtree inside the usual
need_cloneand ignore the error if the path does not exist. Removes another conditional.Quoted 8 lines
I also think using
needs_cloneto assert whether the rmtree should be done is confusing A different variable should be used or use a function's return value. All the code inside ifos.path.exists(ud.clonedir): deals with the need for removal and as a side effect will trigger the clone in another part of the code.-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14984): https://lists.openembedded.org/g/bitbake-devel/message/14984
Mute This Topic: https://lists.openembedded.org/mt/100942876/4454782
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [bitbake-devel][PATCH v2] fetch2: git: Check if clone directory is a git repo
Quoted 106 lines · Paulo Neves
Quoted 64 lines · Joshua Watt
If the clone target directory exists and is a valid git repo, but the git directory is not a child, it needs to be erased and re-cloned. One example of how this can happen is if a clone creates the directory, but then fails to actual clone and make it a git repository. This left-over directory can be particularly problematic if the download directory is a descent of some top layer git repo (e.g. the default with poky), as the commands that operate on the remote later will then mangle the layers git repository instead of the download git repo.
Signed-off-by: Joshua Watt [<[email protected]>](mailto:[email protected])
--- bitbake/lib/bb/fetch2/git.py | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +29
-1@@ -65,6 +65,7 @@ import fnmatch import os import re import shlex +import shutil import subprocess import tempfile import bb@@ -365,8 +366,35 @@ class Git(FetchMethod): runfetchcmd(fetch_cmd, d, workdir=ud.clonedir) repourl = self._get_repo_url(ud) + needs_clone = False + if os.path.exists(ud.clonedir): + # The directory may exist, but not be the top level of a bare git + # repository in which case it needs to be deleted and re-cloned. + try: + # Since clones can be bare, use --absolute-git-dir instead of --show-toplevel + output = runfetchcmd("LANG=C %s rev-parse --absolute-git-dir" % ud.basecmd, d, workdir=ud.clonedir) + except bb.fetch2.FetchError as e: + logger.warning("Unable to get top level for %s (not a git directory?): %s", ud.clonedir, e) + needs_clone = True + else: + toplevel = os.path.abspath(output.rstrip()) + abs_clonedir = os.path.abspath(ud.clonedir).rstrip('/') + # The top level Git directory must either be the clone directory + # or a child of the clone directory. Any ancestor directory of + # the clone directory is not valid as the Git directory (and + # probably belongs to some other unrelated repository), so a + # clone is required + if os.path.commonprefix([abs_clonedir, toplevel]) != abs_clonedir: + logger.warning("Top level directory '%s' doesn't match expected '%s'. Re-cloning", toplevel, ud.clonedir) + needs_clone = True + + if needs_clone: + shutil.rmtree(ud.clonedir) + else: + needs_clone = True + # If the repo still doesn't exist, fallback to cloning it - if not os.path.exists(ud.clonedir): + if needs_clone: # We do this since git will use a "-l" option automatically for local urls where possible, # but it doesn't work when git/objects is a symlink, only works when it is a directory. if repourl.startswith(Quoted 4 lines
["file://"](file://)
):
--
2.33.0Is it possible to factor the try/except else to a function? Then we would not need the else, and would just return immediately and have a self contained code, which in my opinion is easier to read. I also think f-string would be more readable for runfetchcmd. f-string are available since python 3.6.
Can you share the rationale for the else clause? I do not see anything inside the else clause that might raise a FetchError, thus the else clause is redundant.
Another change I suggest is to move the rmtree inside the usual
need_cloneand ignore the error if the path does not exist. Removes another conditional.I also think using
needs_cloneto assert whether the rmtree should be done is confusing A different variable should be used or use a function's return value. All the code inside ifos.path.exists(ud.clonedir): deals with the need for removal and as a side effect will trigger the clone in another part of the code.-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#14982):
https://lists.openembedded.org/g/bitbake-devel/message/14982
Mute This Topic:
https://lists.openembedded.org/mt/100942876/4454782
Group Owner:
[email protected] Unsubscribe:
https://lists.openembedded.org/g/bitbake-devel/unsub
[
[email protected]
]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 1 line
Another change I suggest is to move the rmtree inside the usual
need_clone and ignore the error if the path does not exist. Removes another conditional.
Disregard this comment. It is incorrect.