Skip to contentSkip to conversation

SRC_URI checksum support v3

PATCH In review Yu Ke 3 participants 8 messages Dec 17, 2010 – Dec 21, 2010

lore
Share & tools
lore

Yu Ke

author
#0 lore raw

cover SRC_URI checksum support v3

This patch add SRC_URI checksum support. With this patch,fetcher can verify the MD5 and SHA256 checksum of download src with the value defined in recipes SRC_URI.

This is the v3 patch with following changes compared with v2:
- add configurable variable BB_STRICT_CHECKSUM to handle checksum missing case if checksum is missing and BB_STRICT_CHECKSUM = "1", bitbake will fatal
- add check to only verify checksum for protocol http/https/ftp/ftps, not verify checksum for local file and other SCM

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/srcuri-v3
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri-v3

Thanks,
Yu Ke <[email protected]>
---

Yu Ke (1):
FetchData: add SRC_URI checksum

 bitbake/lib/bb/fetch/__init__.py |   49 ++++++++++++++++++++++++++++++++++++++
 1 files changed, 49 insertions(+), 0 deletions(-)
author
#1 lore raw

PATCH 1/1 FetchData: add SRC_URI checksum

This patch add the per-recipe SRC_URI checksum verification.

- SRC_URI format
The format of SRC_URI checksum follow OE definition:

1. SRC_URI has single src
SRC_URI = "http://some.domain/file.tar.gz"
SRC_URI[md5sum] = "xxxxxxxxxxxxxxx"
SRC_URI[sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"

2. SRC_URI has multiple src, every src need specify name

SRC_URI = "http://some.domain/file1.tar.gz;name=name1 \
           http://some.domain/file2.tar.gz;name=name2 "

SRC_URI[name1.md5sum] = "xxxxxxxxxxxxxxx" SRC_URI[name1.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx" SRC_URI[name2.md5sum] = "xxxxxxxxxxxxxxx" SRC_URI[name2.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"

- SRC_URI checking invocation:
the checksum checking is invoked in do_fetch phase, so it can be invoked manually by

# bitbake -f -c fetch <recipe_name>

if recipes has no SRC_URI checksum item, bitbake will show warning:
"
WARNING: Missing SRC_URI checksum for xxxx.tar.gz, consider to add SRC_URI[md5sum] = "5c69f16d452b0bb3d44bc3c10556c072"
SRC_URI[sha256sum] = "f4e0ada8d4d516bbb8600a3ee7d9046c9c79e38cd781df9ffc46d8f16acd1768" "
thus recipe author can add it to recpie file after SRC_URI

- control variable BB_STRICT_CHECKSUM
when SRC_URI checksum is missing, this variable decide pass or not
if BB_STRICT_CHECKSUM = "1", bitbake should fatal in this case, otherwise bitbake just pass

---
 bitbake/lib/bb/fetch/__init__.py |   49 ++++++++++++++++++++++++++++++++++++++
 1 files changed, 49 insertions(+), 0 deletions(-)
bitbake/lib/bb/fetch/__init__.py +49 -0
index 50955f1..387de66 100644
@@ -231,6 +231,42 @@ def removefile(f):
     except:
         pass
 
+def verify_checksum(d, ud):
+    """
+    verify the MD5 and SHA256 checksum for downloaded src
+
+    return value:
+        - True: checksum matched
+        - False: checksum unmatched
+
+    if checksum is missing in recipes file, "BB_STRICT_CHECKSUM" decide the return value.
+    if BB_STRICT_CHECKSUM = "1" then return false as unmatched, otherwise return true as
+    matched
+    """
+
+    if not ud.type in ["http", "https", "ftp", "ftps"]:
+        return True
+
+    md5data = bb.utils.md5_file(ud.localpath)
+    sha256data = bb.utils.sha256_file(ud.localpath)
+
+    if (ud.md5_expected == None or ud.sha256_expected == None):
+        bb.warn("Missing SRC_URI checksum for %s, consider to add\n" \
+                "SRC_URI[%s] = \"%s\"\nSRC_URI[%s] = \"%s\"" \
+                % (ud.localpath, ud.md5_name, md5data, ud.sha256_name, sha256data))
+        if bb.data.getVar("BB_STRICT_CHECKSUM", d, True) == "1":
+            return False
+        else:
+            return True
+
+    if (ud.md5_expected != md5data or ud.sha256_expected != sha256data):
+        bb.error("The checksums for '%s' did not match." % ud.localpath)
+        bb.error("Expected MD5: '%s' and Got: '%s'" % (ud.md5_expected, md5data))
+        bb.error("Expected SHA256: '%s' and Got: '%s'" % (ud.sha256_expected, sha256data))
+        return False
+
+    return True
+
 def go(d, urls = None):
     """
     Fetch all urls
@@ -283,6 +319,9 @@ def go(d, urls = None):
         else:
             Fetch.write_md5sum(u, ud, d)
 
+        if not verify_checksum(d, ud):
+            raise FetchError("%s checksum mismatch." % u)
+
         bb.utils.unlockfile(lf)
 
 def checkstatus(d, urls = None):
@@ -502,6 +541,16 @@ class FetchData(object):
         if not self.pswd and "pswd" in self.parm:
             self.pswd = self.parm["pswd"]
         self.setup = False
+
+        if "name" in self.parm:
+            self.md5_name = "%s.md5sum" % self.parm["name"]
+            self.sha256_name = "%s.sha256sum" % self.parm["name"]
+        else:
+            self.md5_name = "md5sum"
+            self.sha256_name = "sha256sum"
+        self.md5_expected = bb.data.getVarFlag("SRC_URI", self.md5_name, d)
+        self.sha256_expected = bb.data.getVarFlag("SRC_URI", self.sha256_name, d)
+
         for m in methods:
             if m.supports(url, self, d):
                 self.method = m
Signature · 1 line
1.7.0.4
#2 lore raw
In reply to Yu Ke
Quoted 10 lines · Yu Ke

This patch add SRC_URI checksum support. With this patch,fetcher can verify the MD5 and SHA256 checksum of download src with the value defined in recipes SRC_URI.

This is the v3 patch with following changes compared with v2:
- add configurable variable BB_STRICT_CHECKSUM to handle checksum missing case if checksum is missing and BB_STRICT_CHECKSUM = "1", bitbake will fatal
- add check to only verify checksum for protocol http/https/ftp/ftps, not verify checksum for local file and other SCM

Quoted 3 lines

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/srcuri-v3
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri-v3

I merged this, then realised there was a problem with subsequent builds proceeding with corrupt files. I've pushed a fix for at least part of that.

I'm wondering if we should rename the file to something like the original name + ".corrupt" to make sure we don't use a known broken file?

Cheers,

Richard

author
#3 lore raw
Quoted 11 lines · Richard Purdie
Quoted 10 lines · Yu Ke

This patch add SRC_URI checksum support. With this patch,fetcher can verify the MD5 and SHA256 checksum of download src with the value defined in recipes SRC_URI.

This is the v3 patch with following changes compared with v2:
- add configurable variable BB_STRICT_CHECKSUM to handle checksum missing case if checksum is missing and BB_STRICT_CHECKSUM = "1", bitbake will fatal
- add check to only verify checksum for protocol http/https/ftp/ftps, not verify checksum for local file and other SCM

Quoted 7 lines
Quoted 3 lines

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/srcuri-v3
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri-v3

I merged this, then realised there was a problem with subsequent builds proceeding with corrupt files. I've pushed a fix for at least part of that.

Thanks, it is indeed necessary fix.

BTW, I am also thinking if we could merge Fetch.write_md5sum and Fetch.verify_md5sum into verify_checksum, since they are all checksum related, puting them together would be more clean logically.

Quoted 4 lines

I'm wondering if we should rename the file to something like the original name + ".corrupt" to make sure we don't use a known broken file?

Yes, I prefer to do that. It is more robust, and when meet corrupt file, if bitbake did not clean the downloaded file, user also need to do that. so it is better for bitbake to do that automatically for user.

so combined with the above points, the verify_checksum can be:
"
if ud.md5 file exist
touch ud.md5 and return

check the checksum, if mismatch, rename download file to *.corrupt and raise exception

write the ud.md5 file
"

Comment?

Regards
Ke

Quoted 9 lines

Cheers,

Richard

_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

#4 lore raw
Quoted 20 lines · Yu Ke
Quoted 11 lines · Richard Purdie
Quoted 10 lines · Yu Ke

This patch add SRC_URI checksum support. With this patch,fetcher can verify the MD5 and SHA256 checksum of download src with the value defined in recipes SRC_URI.

This is the v3 patch with following changes compared with v2:
- add configurable variable BB_STRICT_CHECKSUM to handle checksum missing case if checksum is missing and BB_STRICT_CHECKSUM = "1", bitbake will fatal
- add check to only verify checksum for protocol http/https/ftp/ftps, not verify checksum for local file and other SCM

Quoted 7 lines
Quoted 3 lines

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/srcuri-v3
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri-v3

I merged this, then realised there was a problem with subsequent builds proceeding with corrupt files. I've pushed a fix for at least part of that.

Quoted 5 lines

Thanks, it is indeed necessary fix.

BTW, I am also thinking if we could merge Fetch.write_md5sum and Fetch.verify_md5sum into verify_checksum, since they are all checksum related, puting them together would be more clean logically.

That makes the assumption that you only want to verify a checksum when fetching one, which I don't think is correct?

Cheers,
Joshua

Signature · 2 lines
Joshua Lock
        Intel Open Source Technology Centre
#5 lore raw
Quoted 7 lines · Joshua Lock
Quoted 3 lines · Yu Ke

BTW, I am also thinking if we could merge Fetch.write_md5sum and Fetch.verify_md5sum into verify_checksum, since they are all checksum related, puting them together would be more clean logically.

That makes the assumption that you only want to verify a checksum when fetching one, which I don't think is correct?

We really only want to take the time to compute the checksums once, then mark something to say we've checked it and move on.

I think the use of the ".md5" file is misleading and we should probably find a more logical approach to this but in principle checking after download should be ok. We can always have something to enable a recheck if the user really wanted it I guess but most of the time it will be a waste of time.

Cheers,

Richard

author
#6 lore raw
In reply to Joshua Lock
Quoted 21 lines · Joshua Lock
Quoted 20 lines · Yu Ke
Quoted 18 lines · Richard Purdie
Quoted 10 lines · Yu Ke

This patch add SRC_URI checksum support. With this patch,fetcher can verify the MD5 and SHA256 checksum of download src with the value defined in recipes SRC_URI.

This is the v3 patch with following changes compared with v2:
- add configurable variable BB_STRICT_CHECKSUM to handle checksum missing case if checksum is missing and BB_STRICT_CHECKSUM = "1", bitbake will fatal
- add check to only verify checksum for protocol http/https/ftp/ftps, not verify checksum for local file and other SCM

Quoted 3 lines

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/srcuri-v3
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri-v3

I merged this, then realised there was a problem with subsequent builds proceeding with corrupt files. I've pushed a fix for at least part of that.

Quoted 8 lines
Quoted 5 lines

Thanks, it is indeed necessary fix.

BTW, I am also thinking if we could merge Fetch.write_md5sum and Fetch.verify_md5sum into verify_checksum, since they are all checksum related, puting them together would be more clean logically.

That makes the assumption that you only want to verify a checksum when fetching one, which I don't think is correct?

Do you mean other place may also need verify a checksum? If that is the case, it still can call the verify_checksum() to perform verifcation, right?

Regards
Ke

Quoted 11 lines

Cheers,
Joshua

Signature · 7 lines
Joshua Lock
        Intel Open Source Technology Centre

_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky
author
#7 lore raw
Quoted 24 lines · Yu Ke
Quoted 23 lines · Joshua Lock
Quoted 22 lines · Yu Ke
Quoted 11 lines · Richard Purdie
Quoted 10 lines · Yu Ke

This patch add SRC_URI checksum support. With this patch,fetcher can verify the MD5 and SHA256 checksum of download src with the value defined in recipes SRC_URI.

This is the v3 patch with following changes compared with v2:
- add configurable variable BB_STRICT_CHECKSUM to handle checksum missing case if checksum is missing and BB_STRICT_CHECKSUM = "1", bitbake will fatal
- add check to only verify checksum for protocol http/https/ftp/ftps, not verify checksum for local file and other SCM

Quoted 7 lines
Quoted 3 lines

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/srcuri-v3
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri-v3

I merged this, then realised there was a problem with subsequent builds proceeding with corrupt files. I've pushed a fix for at least part of that.

Thanks, it is indeed necessary fix.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

BTW, I am also thinking if we could merge Fetch.write_md5sum and Fetch.verify_md5sum into verify_checksum, since they are all checksum related, puting them together would be more clean logically.

That makes the assumption that you only want to verify a checksum when fetching one, which I don't think is correct?

Do you mean other place may also need verify a checksum? If that is the case, it still can call the verify_checksum() to perform verifcation, right?

Oh, just see Richard has replied the email, it is pretty clear, so no question from me now.

Regards
Ke

Quoted 12 lines
Quoted 12 lines

Cheers,
Joshua

Signature · 7 lines
Joshua Lock
        Intel Open Source Technology Centre

_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky