Skip to contentSkip to conversation

SRC_URI checksum support v2

PATCH In review Yu Ke 3 participants 5 messages Dec 14, 2010 – Dec 16, 2010

lore
Share & tools
lore

Yu Ke

author
#0 lore raw

cover SRC_URI checksum support v2

This patch add SRC_URI checksum support. With this patch,fetcher can verify the MD5 and SHA256 checksum of download src with the value defined in recipes SRC_URI.

This is the V2 patch. The major change compared to V1 is:
moving the logic from poky bbclass to bitbake fetcher, per Richard's comment

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/srcuri-v2
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri-v2

Thanks,
Yu Ke <[email protected]>
---

Yu Ke (1):
FetchData: add SRC_URI checksum

 bitbake/lib/bb/fetch/__init__.py |   36 ++++++++++++++++++++++++++++++++++++
 1 files changed, 36 insertions(+), 0 deletions(-)
author
#1 lore raw

PATCH 1/1 FetchData: add SRC_URI checksum

This patch add the per-recipe SRC_URI checksum verification.

- SRC_URI format
The format of SRC_URI checksum follow OE definition:

1. SRC_URI has single src
SRC_URI = "http://some.domain/file.tar.gz"
SRC_URI[md5sum] = "xxxxxxxxxxxxxxx"
SRC_URI[sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"

2. SRC_URI has multiple src, every src need specify name

SRC_URI = "http://some.domain/file1.tar.gz;name=name1 \
           http://some.domain/file2.tar.gz;name=name2 "

SRC_URI[name1.md5sum] = "xxxxxxxxxxxxxxx" SRC_URI[name1.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx" SRC_URI[name2.md5sum] = "xxxxxxxxxxxxxxx" SRC_URI[name2.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"

- SRC_URI checking invocation:
the checksum checking is invoked in do_fetch phase, so it can be invoked manually by

# bitbake -f -c fetch <recipe_name>

if recipes has no SRC_URI checksum item, bitbake will show warning:
"
WARNING: Missing SRC_URI checksum for xxxx.tar.gz, consider to add SRC_URI[md5sum] = "5c69f16d452b0bb3d44bc3c10556c072"
SRC_URI[sha256sum] = "f4e0ada8d4d516bbb8600a3ee7d9046c9c79e38cd781df9ffc46d8f16acd1768" "
thus recipe author can add it to recpie file after SRC_URI

---
 bitbake/lib/bb/fetch/__init__.py |   36 ++++++++++++++++++++++++++++++++++++
 1 files changed, 36 insertions(+), 0 deletions(-)
bitbake/lib/bb/fetch/__init__.py +36 -0
index 50955f1..d3360b1 100644
@@ -231,6 +231,29 @@ def removefile(f):
     except:
         pass
 
+def verify_checksum(d, ud):
+    """
+    verify the MD5 and SHA256 checksum for downloaded src
+    return True if matched, False if not
+    """
+    md5data = bb.utils.md5_file(ud.localpath)
+    sha256data = bb.utils.sha256_file(ud.localpath)
+
+    if (ud.md5_expected == None or ud.sha256_expected == None):
+        bb.warn("Missing SRC_URI checksum for %s, consider to add\n" \
+                "SRC_URI[%s] = \"%s\"\nSRC_URI[%s] = \"%s\"" \
+                % (ud.localpath, ud.md5_name, md5data, ud.sha256_name, sha256data))
+        # TODO: change to "return False" once all recpies has checksum entry
+        return True
+
+    if (ud.md5_expected != md5data or ud.sha256_expected != sha256data):
+        bb.error("The checksums for '%s' did not match." % ud.localpath)
+        bb.error("Expected MD5: '%s' and Got: '%s'" % (ud.md5_expected, md5data))
+        bb.error("Expected SHA256: '%s' and Got: '%s'" % (ud.sha256_expected, sha256data))
+        return False
+
+    return True
+
 def go(d, urls = None):
     """
     Fetch all urls
@@ -283,6 +306,9 @@ def go(d, urls = None):
         else:
             Fetch.write_md5sum(u, ud, d)
 
+        if not verify_checksum(d, ud):
+            raise FetchError("%s checksum mismatch." % u)
+
         bb.utils.unlockfile(lf)
 
 def checkstatus(d, urls = None):
@@ -502,6 +528,16 @@ class FetchData(object):
         if not self.pswd and "pswd" in self.parm:
             self.pswd = self.parm["pswd"]
         self.setup = False
+
+        if "name" in self.parm:
+            self.md5_name = "%s.md5sum" % self.parm["name"]
+            self.sha256_name = "%s.sha256sum" % self.parm["name"]
+        else:
+            self.md5_name = "md5sum"
+            self.sha256_name = "sha256sum"
+        self.md5_expected = bb.data.getVarFlag("SRC_URI", self.md5_name, d)
+        self.sha256_expected = bb.data.getVarFlag("SRC_URI", self.sha256_name, d)
+
         for m in methods:
             if m.supports(url, self, d):
                 self.method = m
Signature · 1 line
1.7.0.4
#2 lore raw

Re: [PATCH 1/1] FetchData: add SRC_URI checksum

Hi Ke,

This patch looks good in general, just a couple of minor things:

bitbake/lib/bb/fetch/__init__.py def removefile(f): · line 238

Quoted 6 lines · Yu Ke
bitbake/lib/bb/fetch/__init__.py +5 -0
+def verify_checksum(d, ud):
+    """
+    verify the MD5 and SHA256 checksum for downloaded src
+    return True if matched, False if not
+    """

This should be more explicit about the missing checksums case.

bitbake/lib/bb/fetch/__init__.py def removefile(f): · line 247

Quoted 9 lines
bitbake/lib/bb/fetch/__init__.py +9 -0
+    md5data = bb.utils.md5_file(ud.localpath)
+    sha256data = bb.utils.sha256_file(ud.localpath)
+
+    if (ud.md5_expected == None or ud.sha256_expected == None):
+        bb.warn("Missing SRC_URI checksum for %s, consider to add\n" \
+                "SRC_URI[%s] = \"%s\"\nSRC_URI[%s] = \"%s\"" \
+                % (ud.localpath, ud.md5_name, md5data, ud.sha256_name, sha256data))
+        # TODO: change to "return False" once all recpies has checksum entry
+        return True

We can't just redefine "policy" like this in bitbake. I think missing checksums will always be warnings and we might just make warnings optionally fatal for bitbake.

What does this do for file:// urls or SCM urls?

Otherwise I'm good with the patch and if you tweak these things I'll merge it.

Cheers,

Richard

author
#3 lore raw

Re: [PATCH 1/1] FetchData: add SRC_URI checksum

Quoted 6 lines · Richard Purdie

Hi Ke,

This patch looks good in general, just a couple of minor things:

Quoted 1 line · Yu Ke
Quoted 7 lines

bitbake/lib/bb/fetch/__init__.py def removefile(f): · line 238

Quoted 5 lines
bitbake/lib/bb/fetch/__init__.py +5 -0
+def verify_checksum(d, ud):
+    """
+    verify the MD5 and SHA256 checksum for downloaded src
+    return True if matched, False if not
+    """

This should be more explicit about the missing checksums case.

thanks for the review. and yes, will do that.

Quoted 11 lines

bitbake/lib/bb/fetch/__init__.py def removefile(f): · line 247

Quoted 10 lines
bitbake/lib/bb/fetch/__init__.py +9 -0
+    md5data = bb.utils.md5_file(ud.localpath)
+    sha256data = bb.utils.sha256_file(ud.localpath)
+
+    if (ud.md5_expected == None or ud.sha256_expected == None):
+        bb.warn("Missing SRC_URI checksum for %s, consider to add\n" \
+                "SRC_URI[%s] = \"%s\"\nSRC_URI[%s] = \"%s\"" \
+                % (ud.localpath, ud.md5_name, md5data, ud.sha256_name, sha256data))
+        # TODO: change to "return False" once all recpies has checksum entry
+        return True
Quoted 3 lines

We can't just redefine "policy" like this in bitbake. I think missing checksums will always be warnings and we might just make warnings optionally fatal for bitbake.

for the "make warnings optionally fatal in bitbake", do you mean add a variable like "BB_STRICT_CHECKSUM" in config file, and if for example "BB_STRICT_CHECKSUMS" = "1", then fatal, otherwise, just warning?

Quoted 2 lines

What does this do for file:// urls or SCM urls?

Good catch. It should only verify for "http/https/ftp/ftps" protocol, other protocol like local file and SCM url should just return.

Regards
Ke

Quoted 13 lines

Otherwise I'm good with the patch and if you tweak these things I'll merge it.

Cheers,

Richard

_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

#4 lore raw
In reply to Yu Ke

Op 14 dec 2010, om 11:01 heeft Yu Ke het volgende geschreven:

Quoted 3 lines

This patch add SRC_URI checksum support. With this patch,fetcher can verify the MD5 and SHA256 checksum of download src with the value defined in recipes SRC_URI.

Awesome, I have been waiting for that!

Quoted 24 lines

This is the V2 patch. The major change compared to V1 is:
moving the logic from poky bbclass to bitbake fetcher, per Richard's comment

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/srcuri-v2
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri-v2

Thanks,
Yu Ke <[email protected]>
---

Yu Ke (1):
FetchData: add SRC_URI checksum

bitbake/lib/bb/fetch/__init__.py | 36 ++++++++++++++++++++++++++++++++++++

1 files changed, 36 insertions(+), 0 deletions(-)

_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky