Skip to contentSkip to conversation

fetch2: Handle missing donestamp file when content is valid

PATCH New Nathan Rossi 1 participant 1 message Feb 2, 2018

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 fetch2: Handle missing donestamp file when content is valid

In order to allow users to manually populate the download directory with valid content change the assumption that missing the donestamp file means unfetched content.

This allows users to populate the download dir without needing to create dummy .done files such that a user does not need a PREMIRROR when using BB_NO_NETWORK to provide valid content files in the download directory.

To ensure the correct result this change also fails first if the localpath does not exist. This prevents further parts of the function attempting to calculating the checksum on non-existent files. This also fixes some edge conditions around where if the donestamp exists but the localpath does not it returns, and did not remove the donestamp.

Also added test cases to cover this use case and additional use cases where for example the fetcher does not support checksums.

---
Changes in v2:
* Added test cases for the 4 common donestamp cases
Changes in v3:
* Add more test cases around edge conditions
* Fix regression with previous versions where the verify would return true when there was no localpath and no donestamp (added tests for this too)
* Check if the local file exists first, and fail first but also clean up the donestamp

---
 lib/bb/fetch2/__init__.py |  21 +++++-----
 lib/bb/tests/fetch.py     | 104 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 114 insertions(+), 11 deletions(-)
lib/bb/fetch2/__init__.py +10 -11
index 6bd040493e..72d6092deb 100644
@@ -643,26 +643,25 @@ def verify_donestamp(ud, d, origud=None):
     if not ud.needdonestamp or (origud and not origud.needdonestamp):
         return True
 
-    if not os.path.exists(ud.donestamp):
+    if not os.path.exists(ud.localpath):
+        # local path does not exist
+        if os.path.exists(ud.donestamp):
+            # done stamp exists, but the downloaded file does not; the done stamp
+            # must be incorrect, re-trigger the download
+            bb.utils.remove(ud.donestamp)
         return False
 
     if (not ud.method.supports_checksum(ud) or
         (origud and not origud.method.supports_checksum(origud))):
-        # done stamp exists, checksums not supported; assume the local file is
-        # current
-        return True
-
-    if not os.path.exists(ud.localpath):
-        # done stamp exists, but the downloaded file does not; the done stamp
-        # must be incorrect, re-trigger the download
-        bb.utils.remove(ud.donestamp)
-        return False
+        # if done stamp exists and checksums not supported; assume the local
+        # file is current
+        return os.path.exists(ud.donestamp)
 
     precomputed_checksums = {}
     # Only re-use the precomputed checksums if the donestamp is newer than the
     # file. Do not rely on the mtime of directories, though. If ud.localpath is
     # a directory, there will probably not be any checksums anyway.
-    if (os.path.isdir(ud.localpath) or
+    if os.path.exists(ud.donestamp) and (os.path.isdir(ud.localpath) or
             os.path.getmtime(ud.localpath) < os.path.getmtime(ud.donestamp)):
         try:
             with open(ud.donestamp, "rb") as cachefile:
lib/bb/tests/fetch.py +104 -0

116 lines not on this page · raw message

Signature · 1 line
2.15.1