gitsm: When switching SRCREVs verify submodules
cover gitsm: When switching SRCREVs verify submodules
I'm not going to claim that this is a 'v2' version. But the work is in response to the patch submittted by William A. Kennington III to the list.
I've used his test case to determine what the issue is, and developed a different solution to the problem.
The primary difference in overall behavior is that I wanted to avoid any sort of iteration into the submodules during processing. Otherwise we run the risk of race conditions and locking issues. (The original work from William did address locking concerns, but prior versions of the gitsm fetcher proved this was still problematic.)
The test case from William's patch is included in this, unmodified.
Mark Hatle (1):
gitsm: Add need_update method to determine when we are going to a new
SRCREV
lib/bb/fetch2/gitsm.py | 20 ++++++++++++++++++++ lib/bb/tests/fetch.py | 19 +++++++++++++++++++ 2 files changed, 39 insertions(+)
Signature · 1 line
2.16.0.rc2
PATCH 1/1 gitsm: Add need_update method to determine when we are going to a new SRCREV
If the system had previously fetched a source repository for use by gitsm, and then the SRCREV was updated and the new commit already existed, the system would not re-evaluate the submodules and update them accordingly.
The cause of this issue was that need_update was being used, unmodified, from the base git fetcher. It did not have any knowledge, nor did it care if we were moving commits and needed to re-evaluate what was happening due to this switch.
To fix the issue, during the download process we add all processed (by gitsm) srcrevs to the git config file, as bitbake.srcrev. This allows us to use a new need_update function that not only checks if the git commit is present, but if we have previously processed this commit to ensure all of the submodule components are also present.
This approach is used, instead of iterating over the submodules in need_update to avoid a potential race condition that has affected us in the past. The need_update is called only with the parent locking. Any time we need to dive into the submodules, we need to lock, and unlock them, at each stage. This opens the possibility of errors in either the code, or unintended race conditions with rm_work.
This issue was discovered by William A. Kennington III <[email protected]>. The included test case was also written by him, and included unmodified.
--- lib/bb/fetch2/gitsm.py | 20 ++++++++++++++++++++ lib/bb/tests/fetch.py | 19 +++++++++++++++++++ 2 files changed, 39 insertions(+)
lib/bb/fetch2/gitsm.py +20 -0
@@ -147,6 +147,23 @@ class GitSM(Git): return submodules != [] + def need_update(self, ud, d): + if Git.need_update(self, ud, d): + return True + + try: + # Check for the nugget dropped by the download operation + known_srcrevs = runfetchcmd("%s config --get-all bitbake.srcrev" % \ + (ud.basecmd), d, workdir=ud.clonedir) + + if ud.revisions[ud.names[0]] not in known_srcrevs.split(): + return True + except bb.fetch2.FetchError: + # No srcrev nuggets, so this is new and needs to be updated + return True + + return False + def download(self, ud, d): def download_submodule(ud, url, module, modpath, d): url += ";bareclone=1;nobranch=1"
@@ -157,6 +174,9 @@ class GitSM(Git): try: newfetch = Fetch([url], d, cache=False) newfetch.download() + # Drop a nugget to add each of the srcrevs we've fetched (used by need_update) + runfetchcmd("%s config --add bitbake.srcrev %s" % \ + (ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=ud.clonedir) except Exception as e: logger.error('gitsm: submodule download failed: %s %s' % (type(e).__name__, str(e))) raise
lib/bb/tests/fetch.py +19 -0
@@ -942,6 +942,25 @@ class FetcherNetworkTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"') self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"') + def test_git_submodule_update_CLI11(self): + """ Prevent regression on update detection not finding missing submodule, or modules without needed commits """ + url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714" + fetcher = bb.fetch.Fetch([url], self.d) + fetcher.download() + + # CLI11 that pulls in a newer nlohmann-json + url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca" + fetcher = bb.fetch.Fetch([url], self.d) + fetcher.download() + # Previous cwd has been deleted + os.chdir(os.path.dirname(self.unpackdir)) + fetcher.unpack(self.unpackdir) + + repo_path = os.path.join(self.tempdir, 'unpacked', 'git') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/googletest/config')), msg='Missing submodule config "extern/googletest"') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"') + def test_git_submodule_aktualizr(self): url = "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44" fetcher = bb.fetch.Fetch([url], self.d)
Signature · 1 line
2.16.0.rc2
Re: [PATCH 1/1] gitsm: Add need_update method to determine when we are going to a new SRCREV
I think this looks good in terms of fixing the issue. It might be overly pessimistic in triggering need_update since it doesn't re-use git metadata, so it isn't just looking at the tree for each download.
Out of curiosity, since git does its own locking, is there any harm in not taking the download locks but reading the metadata of the bare git repos? No downloader should ever be purging commits from the tree, so needs_update would never return a false negative.
Quoted 103 lines · Mark Hatle
If the system had previously fetched a source repository for use by gitsm, and then the SRCREV was updated and the new commit already existed, the system would not re-evaluate the submodules and update them accordingly.
The cause of this issue was that
need_updatewas being used, unmodified, from the base git fetcher. It did not have any knowledge, nor did it care if we were moving commits and needed to re-evaluate what was happening due to this switch.To fix the issue, during the download process we add all processed (by gitsm) srcrevs to the git config file, as bitbake.srcrev. This allows us to use a new
need_updatefunction that not only checks if the git commit is present, but if we have previously processed this commit to ensure all of the submodule components are also present.This approach is used, instead of iterating over the submodules in
need_updateto avoid a potential race condition that has affected us in the past. Theneed_updateis called only with the parent locking. Any time we need to dive into the submodules, we need to lock, and unlock them, at each stage. This opens the possibility of errors in either the code, or unintended race conditions withrm_work.This issue was discovered by William A. Kennington III <[email protected]>. The included test case was also written by him, and included unmodified.
Signed-off-by Mark Hatle <[email protected]>--- lib/bb/fetch2/gitsm.py | 20 ++++++++++++++++++++ lib/bb/tests/fetch.py | 19 +++++++++++++++++++ 2 files changed, 39 insertions(+)lib/bb/fetch2/gitsm.py +20
-0@@ -147,6 +147,23 @@ class GitSM(Git): return submodules != [] + def need_update(self, ud, d): + if Git.need_update(self, ud, d): + return True + + try: + # Check for the nugget dropped by the download operation + known_srcrevs = runfetchcmd("%s config --get-all bitbake.srcrev" % \ + (ud.basecmd), d, workdir=ud.clonedir) + + if ud.revisions[ud.names[0]] not in known_srcrevs.split(): + return True + except bb.fetch2.FetchError: + # No srcrev nuggets, so this is new and needs to be updated + return True + + return False + def download(self, ud, d): def download_submodule(ud, url, module, modpath, d): url += ";bareclone=1;nobranch=1"@@ -157,6 +174,9 @@ class GitSM(Git): try: newfetch = Fetch([url], d, cache=False) newfetch.download() + # Drop a nugget to add each of the srcrevs we've fetched (used by need_update) + runfetchcmd("%s config --add bitbake.srcrev %s" % \ + (ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=ud.clonedir) except Exception as e: logger.error('gitsm: submodule download failed: %s %s' % (type(e).__name__, str(e))) raiselib/bb/tests/fetch.py +19
-1@@ -942,6 +942,25 @@ class FetcherNetworkTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"') self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"') + def test_git_submodule_update_CLI11(self): + """ Prevent regression on update detection not finding missing submodule, or modules without needed commits """ + url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714" + fetcher = bb.fetch.Fetch([url], self.d) + fetcher.download() + + # CLI11 that pulls in a newer nlohmann-json + url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca" + fetcher = bb.fetch.Fetch([url], self.d) + fetcher.download() + # Previous cwd has been deleted + os.chdir(os.path.dirname(self.unpackdir)) + fetcher.unpack(self.unpackdir) + + repo_path = os.path.join(self.tempdir, 'unpacked', 'git') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/googletest/config')), msg='Missing submodule config "extern/googletest"') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"') + def test_git_submodule_aktualizr(self): url = "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44" fetcher = bb.fetch.Fetch([url], self.d) --2.16.0.rc2
Re: [PATCH 1/1] gitsm: Add need_update method to determine when we are going to a new SRCREV
A follow up for the case I am talking about, your commit will try to download the second fetch even if it doesn't have to. I realize this is still better than breaking the unpack stage, but we should be able to support this type of thing since the data is already there.
lib/bb/tests/fetch.py +0 -0
@@ -961,6 +961,26 @@ class FetcherNetworkTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(repo_path,
'.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"')
self.assertTrue(os.path.exists(os.path.join(repo_path,
'.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"')
+ def test_git_submodule_no_update_CLI11(self):
+ """ Prevent regression on update detection not finding
already downloaded commits """
+ url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=da901cca542612a133efcb04e8e78080186991e4"
+ fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher.download()
+
+ self.d.setVar("BB_NO_NETWORK", 1)
+ # CLI11 that pulls in a newer nlohmann-json which we already fetched
+ url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=899100c1619ea1282620604ac9199010e65f907e"
+ fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher.download()
+ # Previous cwd has been deleted
+ os.chdir(os.path.dirname(self.unpackdir))
+ fetcher.unpack(self.unpackdir)+
+ repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
+ self.assertTrue(os.path.exists(os.path.join(repo_path,
'.git/modules/extern/googletest/config')), msg='Missing submodule
config "extern/googletest"')
+ self.assertTrue(os.path.exists(os.path.join(repo_path,
'.git/modules/extern/json/config')), msg='Missing submodule config
"extern/json"')
+ self.assertTrue(os.path.exists(os.path.join(repo_path,
'.git/modules/extern/sanitizers/config')), msg='Missing submodule
config "extern/sanitizers"')
+
def test_git_submodule_aktualizr(self):
url = "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44" fetcher = bb.fetch.Fetch([url], self.d)
Quoted 114 lines · William Kennington
I think this looks good in terms of fixing the issue. It might be overly pessimistic in triggering
need_updatesince it doesn't re-use git metadata, so it isn't just looking at the tree for each download.Out of curiosity, since git does its own locking, is there any harm in not taking the download locks but reading the metadata of the bare git repos? No downloader should ever be purging commits from the tree, so
needs_updatewould never return a false negative.Quoted 103 lines · Mark Hatle
If the system had previously fetched a source repository for use by gitsm, and then the SRCREV was updated and the new commit already existed, the system would not re-evaluate the submodules and update them accordingly.
The cause of this issue was that
need_updatewas being used, unmodified, from the base git fetcher. It did not have any knowledge, nor did it care if we were moving commits and needed to re-evaluate what was happening due to this switch.To fix the issue, during the download process we add all processed (by gitsm) srcrevs to the git config file, as bitbake.srcrev. This allows us to use a new
need_updatefunction that not only checks if the git commit is present, but if we have previously processed this commit to ensure all of the submodule components are also present.This approach is used, instead of iterating over the submodules in
need_updateto avoid a potential race condition that has affected us in the past. Theneed_updateis called only with the parent locking. Any time we need to dive into the submodules, we need to lock, and unlock them, at each stage. This opens the possibility of errors in either the code, or unintended race conditions withrm_work.This issue was discovered by William A. Kennington III <[email protected]>. The included test case was also written by him, and included unmodified.
Signed-off-by Mark Hatle <[email protected]>--- lib/bb/fetch2/gitsm.py | 20 ++++++++++++++++++++ lib/bb/tests/fetch.py | 19 +++++++++++++++++++ 2 files changed, 39 insertions(+)lib/bb/fetch2/gitsm.py +20
-0@@ -147,6 +147,23 @@ class GitSM(Git): return submodules != [] + def need_update(self, ud, d): + if Git.need_update(self, ud, d): + return True + + try: + # Check for the nugget dropped by the download operation + known_srcrevs = runfetchcmd("%s config --get-all bitbake.srcrev" % \ + (ud.basecmd), d, workdir=ud.clonedir) + + if ud.revisions[ud.names[0]] not in known_srcrevs.split(): + return True + except bb.fetch2.FetchError: + # No srcrev nuggets, so this is new and needs to be updated + return True + + return False + def download(self, ud, d): def download_submodule(ud, url, module, modpath, d): url += ";bareclone=1;nobranch=1"@@ -157,6 +174,9 @@ class GitSM(Git): try: newfetch = Fetch([url], d, cache=False) newfetch.download() + # Drop a nugget to add each of the srcrevs we've fetched (used by need_update) + runfetchcmd("%s config --add bitbake.srcrev %s" % \ + (ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=ud.clonedir) except Exception as e: logger.error('gitsm: submodule download failed: %s %s' % (type(e).__name__, str(e))) raiselib/bb/tests/fetch.py +19
-1@@ -942,6 +942,25 @@ class FetcherNetworkTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"') self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"') + def test_git_submodule_update_CLI11(self): + """ Prevent regression on update detection not finding missing submodule, or modules without needed commits """ + url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714" + fetcher = bb.fetch.Fetch([url], self.d) + fetcher.download() + + # CLI11 that pulls in a newer nlohmann-json + url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca" + fetcher = bb.fetch.Fetch([url], self.d) + fetcher.download() + # Previous cwd has been deleted + os.chdir(os.path.dirname(self.unpackdir)) + fetcher.unpack(self.unpackdir) + + repo_path = os.path.join(self.tempdir, 'unpacked', 'git') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/googletest/config')), msg='Missing submodule config "extern/googletest"') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"') + def test_git_submodule_aktualizr(self): url = "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44" fetcher = bb.fetch.Fetch([url], self.d) --2.16.0.rc2
Re: [PATCH 1/1] gitsm: Add need_update method to determine when we are going to a new SRCREV
Sorry, bad send from gmail which seems to want to wrap and deduplicate my text. You can apply the following patch to test against over pessimism.
lib/bb/tests/fetch.py +0 -0
@@ -961,6 +961,26 @@ class FetcherNetworkTest(FetcherTest):
self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"') self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"')
+ def test_git_submodule_no_update_CLI11(self):
+ """ Prevent regression on update detection not finding already downloaded commits """
+ url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=da901cca542612a133efcb04e8e78080186991e4"
+ fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher.download()
+
+ self.d.setVar("BB_NO_NETWORK", 1)
+ # CLI11 that pulls in a newer nlohmann-json which we already fetched
+ url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=899100c1619ea1282620604ac9199010e65f907e"
+ fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher.download()
+ # Previous cwd has been deleted
+ os.chdir(os.path.dirname(self.unpackdir))
+ fetcher.unpack(self.unpackdir)
+
+ repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/googletest/config')), msg='Missing submodule config "extern/googletest"') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"') +
def test_git_submodule_aktualizr(self):
url = "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44" fetcher = bb.fetch.Fetch([url], self.d)
Re: [PATCH 1/1] gitsm: Add need_update method to determine when we are going to a new SRCREV
One last time, sorrry for the spam. I usually just use git send-email with wrapping options disabled passed to our special sendmail when doing patches and didn't have a proper mail client set up. I tested this against myself and it worked last time so hopefully formatting is fine now.
lib/bb/tests/fetch.py +0 -0
@@ -961,6 +961,26 @@ class FetcherNetworkTest(FetcherTest):
self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"') self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"')
+ def test_git_submodule_no_update_CLI11(self):
+ """ Prevent regression on update detection not finding already downloaded commits """
+ url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=da901cca542612a133efcb04e8e78080186991e4"
+ fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher.download()
+
+ self.d.setVar("BB_NO_NETWORK", 1)
+ # CLI11 that pulls in a newer nlohmann-json which we already fetched
+ url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=899100c1619ea1282620604ac9199010e65f907e"
+ fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher.download()
+ # Previous cwd has been deleted
+ os.chdir(os.path.dirname(self.unpackdir))
+ fetcher.unpack(self.unpackdir)
+
+ repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/googletest/config')), msg='Missing submodule config "extern/googletest"') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"') +
def test_git_submodule_aktualizr(self):
url = "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44" fetcher = bb.fetch.Fetch([url], self.d)
Re: [PATCH 1/1] gitsm: Add need_update method to determine when we are going to a new SRCREV
In reply to William KenningtonQuoted 4 lines · William Kennington
A follow up for the case I am talking about, your commit will try to download the second fetch even if it doesn't have to. I realize this is still better than breaking the unpack stage, but we should be able to support this type of thing since the data is already there.
There is no direct mechanism that I'm aware of to be able to avoid the secondary fetch. So the original patch would certainly permit and expect that behavior.
The 'download' sets up the lock, and then calls needs_update and proceeds from there. (with a true/false answer).
It may be possible to call the -git- needs_update from within the gitsm 'download' prior to trying to directly call the 'git.Download()'... Since the lock is held, this is a safe operation to go..
So something like:
- Git.download(self, ud, d)
+ if Git.need_update(ud, d):
+ Git.download(self, ud, d)
--Mark
Quoted 158 lines
lib/bb/tests/fetch.py +0
-0@@ -961,6 +961,26 @@ class FetcherNetworkTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(repo_path,'.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"')
self.assertTrue(os.path.exists(os.path.join(repo_path,
'.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"')lib/bb/tests/fetch.py +2
-0+ def test_git_submodule_no_update_CLI11(self): + """ Prevent regression on update detection not findingalready downloaded commits """
lib/bb/tests/fetch.py +15
-0+ url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=da901cca542612a133efcb04e8e78080186991e4" + fetcher = bb.fetch.Fetch([url], self.d) + fetcher.download() + + self.d.setVar("BB_NO_NETWORK", 1) + # CLI11 that pulls in a newer nlohmann-json which we already fetched + url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=899100c1619ea1282620604ac9199010e65f907e" + fetcher = bb.fetch.Fetch([url], self.d) + fetcher.download() + # Previous cwd has been deleted + os.chdir(os.path.dirname(self.unpackdir)) + fetcher.unpack(self.unpackdir) + + repo_path = os.path.join(self.tempdir, 'unpacked', 'git') + self.assertTrue(os.path.exists(os.path.join(repo_path,'.git/modules/extern/googletest/config')), msg='Missing submodule
config "extern/googletest"')
+ self.assertTrue(os.path.exists(os.path.join(repo_path,
'.git/modules/extern/json/config')), msg='Missing submodule config
"extern/json"')
+ self.assertTrue(os.path.exists(os.path.join(repo_path,
'.git/modules/extern/sanitizers/config')), msg='Missing submodule
config "extern/sanitizers"')
+
deftest_git_submodule_aktualizr(self):
url = "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44" fetcher =bb.fetch.Fetch([url], self.d)Quoted 114 lines · William Kennington
I think this looks good in terms of fixing the issue. It might be overly pessimistic in triggering
need_updatesince it doesn't re-use git metadata, so it isn't just looking at the tree for each download.Out of curiosity, since git does its own locking, is there any harm in not taking the download locks but reading the metadata of the bare git repos? No downloader should ever be purging commits from the tree, so
needs_updatewould never return a false negative.Quoted 103 lines · Mark Hatle
If the system had previously fetched a source repository for use by gitsm, and then the SRCREV was updated and the new commit already existed, the system would not re-evaluate the submodules and update them accordingly.
The cause of this issue was that
need_updatewas being used, unmodified, from the base git fetcher. It did not have any knowledge, nor did it care if we were moving commits and needed to re-evaluate what was happening due to this switch.To fix the issue, during the download process we add all processed (by gitsm) srcrevs to the git config file, as bitbake.srcrev. This allows us to use a new
need_updatefunction that not only checks if the git commit is present, but if we have previously processed this commit to ensure all of the submodule components are also present.This approach is used, instead of iterating over the submodules in
need_updateto avoid a potential race condition that has affected us in the past. Theneed_updateis called only with the parent locking. Any time we need to dive into the submodules, we need to lock, and unlock them, at each stage. This opens the possibility of errors in either the code, or unintended race conditions withrm_work.This issue was discovered by William A. Kennington III <[email protected]>. The included test case was also written by him, and included unmodified.
Signed-off-by Mark Hatle <[email protected]>--- lib/bb/fetch2/gitsm.py | 20 ++++++++++++++++++++ lib/bb/tests/fetch.py | 19 +++++++++++++++++++ 2 files changed, 39 insertions(+)lib/bb/fetch2/gitsm.py +20
-0@@ -147,6 +147,23 @@ class GitSM(Git): return submodules != [] + def need_update(self, ud, d): + if Git.need_update(self, ud, d): + return True + + try: + # Check for the nugget dropped by the download operation + known_srcrevs = runfetchcmd("%s config --get-all bitbake.srcrev" % \ + (ud.basecmd), d, workdir=ud.clonedir) + + if ud.revisions[ud.names[0]] not in known_srcrevs.split(): + return True + except bb.fetch2.FetchError: + # No srcrev nuggets, so this is new and needs to be updated + return True + + return False + def download(self, ud, d): def download_submodule(ud, url, module, modpath, d): url += ";bareclone=1;nobranch=1"@@ -157,6 +174,9 @@ class GitSM(Git): try: newfetch = Fetch([url], d, cache=False) newfetch.download() + # Drop a nugget to add each of the srcrevs we've fetched (used by need_update) + runfetchcmd("%s config --add bitbake.srcrev %s" % \ + (ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=ud.clonedir) except Exception as e: logger.error('gitsm: submodule download failed: %s %s' % (type(e).__name__, str(e))) raiselib/bb/tests/fetch.py +19
-1@@ -942,6 +942,25 @@ class FetcherNetworkTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"') self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"') + def test_git_submodule_update_CLI11(self): + """ Prevent regression on update detection not finding missing submodule, or modules without needed commits """ + url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714" + fetcher = bb.fetch.Fetch([url], self.d) + fetcher.download() + + # CLI11 that pulls in a newer nlohmann-json + url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca" + fetcher = bb.fetch.Fetch([url], self.d) + fetcher.download() + # Previous cwd has been deleted + os.chdir(os.path.dirname(self.unpackdir)) + fetcher.unpack(self.unpackdir) + + repo_path = os.path.join(self.tempdir, 'unpacked', 'git') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/googletest/config')), msg='Missing submodule config "extern/googletest"') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"') + def test_git_submodule_aktualizr(self): url = "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44" fetcher = bb.fetch.Fetch([url], self.d) --2.16.0.rc2
Re: [PATCH 1/1] gitsm: Add need_update method to determine when we are going to a new SRCREV
In reply to William KenningtonQuoted 7 lines · William Kennington
I think this looks good in terms of fixing the issue. It might be overly pessimistic in triggering
need_updatesince it doesn't re-use git metadata, so it isn't just looking at the tree for each download.Out of curiosity, since git does its own locking, is there any harm in not taking the download locks but reading the metadata of the bare git repos? No downloader should ever be purging commits from the tree, so
Unfortunately there is a case here where one recipe (or build with a shared download dir) could be doing a clean while you are doing a download. The read of the git information in the middle of this can cause a race. We restructed the code a while back to never do a fetch (or other operation) on a submodule (directly) which is why we call the fetcher and it's operations recursively the way we do.
The fetch2/__init__.py is what actually takes the locks in most cases BTW. need_update is a case that is called by other places, so it doesn't usually take any locks..
Quoted 1 line
needs_updatewould never return a false negative.
It's allowed actually, while not really desired behavior -- adding 'rm_work' to the system where multiple shared recipes may use the same component can trigger this. It's more likely when you introduce submodules, since the recipes themselves don't know this thing is actually in use -- and it has caused problems in the past.
--Mark
Quoted 104 lines · Mark Hatle
Quoted 103 lines
If the system had previously fetched a source repository for use by gitsm, and then the SRCREV was updated and the new commit already existed, the system would not re-evaluate the submodules and update them accordingly.
The cause of this issue was that
need_updatewas being used, unmodified, from the base git fetcher. It did not have any knowledge, nor did it care if we were moving commits and needed to re-evaluate what was happening due to this switch.To fix the issue, during the download process we add all processed (by gitsm) srcrevs to the git config file, as bitbake.srcrev. This allows us to use a new
need_updatefunction that not only checks if the git commit is present, but if we have previously processed this commit to ensure all of the submodule components are also present.This approach is used, instead of iterating over the submodules in
need_updateto avoid a potential race condition that has affected us in the past. Theneed_updateis called only with the parent locking. Any time we need to dive into the submodules, we need to lock, and unlock them, at each stage. This opens the possibility of errors in either the code, or unintended race conditions withrm_work.This issue was discovered by William A. Kennington III <[email protected]>. The included test case was also written by him, and included unmodified.
Signed-off-by Mark Hatle <[email protected]>--- lib/bb/fetch2/gitsm.py | 20 ++++++++++++++++++++ lib/bb/tests/fetch.py | 19 +++++++++++++++++++ 2 files changed, 39 insertions(+)lib/bb/fetch2/gitsm.py +20
-0@@ -147,6 +147,23 @@ class GitSM(Git): return submodules != [] + def need_update(self, ud, d): + if Git.need_update(self, ud, d): + return True + + try: + # Check for the nugget dropped by the download operation + known_srcrevs = runfetchcmd("%s config --get-all bitbake.srcrev" % \ + (ud.basecmd), d, workdir=ud.clonedir) + + if ud.revisions[ud.names[0]] not in known_srcrevs.split(): + return True + except bb.fetch2.FetchError: + # No srcrev nuggets, so this is new and needs to be updated + return True + + return False + def download(self, ud, d): def download_submodule(ud, url, module, modpath, d): url += ";bareclone=1;nobranch=1"@@ -157,6 +174,9 @@ class GitSM(Git): try: newfetch = Fetch([url], d, cache=False) newfetch.download() + # Drop a nugget to add each of the srcrevs we've fetched (used by need_update) + runfetchcmd("%s config --add bitbake.srcrev %s" % \ + (ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=ud.clonedir) except Exception as e: logger.error('gitsm: submodule download failed: %s %s' % (type(e).__name__, str(e))) raiselib/bb/tests/fetch.py +19
-1@@ -942,6 +942,25 @@ class FetcherNetworkTest(FetcherTest): self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"') self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"') + def test_git_submodule_update_CLI11(self): + """ Prevent regression on update detection not finding missing submodule, or modules without needed commits """ + url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714" + fetcher = bb.fetch.Fetch([url], self.d) + fetcher.download() + + # CLI11 that pulls in a newer nlohmann-json + url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca" + fetcher = bb.fetch.Fetch([url], self.d) + fetcher.download() + # Previous cwd has been deleted + os.chdir(os.path.dirname(self.unpackdir)) + fetcher.unpack(self.unpackdir) + + repo_path = os.path.join(self.tempdir, 'unpacked', 'git') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/googletest/config')), msg='Missing submodule config "extern/googletest"') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"') + self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"') + def test_git_submodule_aktualizr(self): url = "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44" fetcher = bb.fetch.Fetch([url], self.d) --2.16.0.rc2