Improve usability of checksums
cover Improve usability of checksums
I'd heard some complaints around the usability of checksums and, having done some recipe work recently, must agree with them.
This series ams to make using checksums less intrusive by:
a) Removing the requirement that if one type of checksum is defined the other must be too.
b) Reporting all sum mismatches at once
c) Enabling sums to be defined as a parameter of a SRC_URI
Regards,
Joshua
The following changes since commit dca46cc2e1c75b6add2c4801e2994a4812745f5b:
fetch2: Export additional variables to the fetchers (2011-10-05 14:23:16 +0100)
are available in the git repository at:
git://github.com/incandescant/bitbake checksum
https://github.com/incandescant/bitbake/tree/checksum
Joshua Lock (2):
fetch2: improve usability of checksums
fetch2: enable checksum definition as SRC_URI parameter
lib/bb/fetch2/__init__.py | 82 +++++++++++++++++++++++++-------------------- 1 files changed, 46 insertions(+), 36 deletions(-)
Signature · 1 line
1.7.6.4
PATCH 1/2 fetch2: improve usability of checksums
This patch improves the usability of checksums by only requiring one checksum be defined.
Further, checksum verification will provide as much information as possible at, rather than a bit at a time. No longer will you need to run fetch, see an md5sum mismatch, fix it, run fetch, seen an sha256sum mismatch, fix it and fetch again. If neither checksum is defined we now report both missing sums at once - rather than one after the other.
Finally, if both sums are incorrect, we'll report both incorrect sums at the same time.
Fixes part of [YOCTO #1399]
--- lib/bb/fetch2/__init__.py | 72 +++++++++++++++++++++++--------------------- 1 files changed, 38 insertions(+), 34 deletions(-)
lib/bb/fetch2/__init__.py +38 -34
@@ -93,28 +93,6 @@ class ParameterError(BBFetchException): BBFetchException.__init__(self, msg) self.args = (message, url) -class MD5SumError(BBFetchException): - """Exception raised when a MD5 checksum of a file does not match for a downloaded file""" - def __init__(self, path, wanted, got, url): - msg = "File: '%s' has md5 checksum %s when %s was expected (from URL: '%s')" % (path, got, wanted, url) - self.url = url - self.path = path - self.wanted = wanted - self.got = got - BBFetchException.__init__(self, msg) - self.args = (path, wanted, got, url) - -class SHA256SumError(MD5SumError): - """Exception raised when a SHA256 checksum of a file does not match for a downloaded file""" - def __init__(self, path, wanted, got, url): - msg = "File: '%s' has sha256 checksum %s when %s was expected (from URL: '%s')" % (path, got, wanted, url) - self.url = url - self.path = path - self.wanted = wanted - self.got = got - BBFetchException.__init__(self, msg) - self.args = (path, wanted, got, url) - class NetworkAccess(BBFetchException): """Exception raised when network access is disabled but it is required.""" def __init__(self, url, cmd):
@@ -271,8 +249,8 @@ def verify_checksum(u, ud, d): verify the MD5 and SHA256 checksum for downloaded src return value: - - True: checksum matched - - False: checksum unmatched + - True: a checksum matched + - False: neither checksum matched if checksum is missing in recipes file, "BB_STRICT_CHECKSUM" decide the return value. if BB_STRICT_CHECKSUM = "1" then return false as unmatched, otherwise return true as
@@ -285,20 +263,46 @@ def verify_checksum(u, ud, d): md5data = bb.utils.md5_file(ud.localpath) sha256data = bb.utils.sha256_file(ud.localpath) - if (ud.md5_expected == None or ud.sha256_expected == None): - logger.warn('Missing SRC_URI checksum for %s, consider adding to the recipe:\n' - 'SRC_URI[%s] = "%s"\nSRC_URI[%s] = "%s"', - ud.localpath, ud.md5_name, md5data, - ud.sha256_name, sha256data) - if bb.data.getVar("BB_STRICT_CHECKSUM", d, True) == "1": - raise FetchError("No checksum specified for %s." % u, u) - return + # If strict checking enabled and neither sum defined, raise error + strict = bb.data.getVar("BB_STRICT_CHECKSUM", d, True) or None + if (strict and ud.md5_expected == None and ud.sha256_expected == None): + raise FetchError('No checksum specified for %s, please add at least one to the recipe:\n' + 'SRC_URI[%s] = "%s"\nSRC_URI[%s] = "%s"', u, + ud.localpath, ud.md5_name, md5data, + ud.sha256_name, sha256data) + + # Log missing sums so user can more easily add them + if ud.md5_expected == None: + logger.warn('Missing md5 SRC_URI checksum for %s, consider adding to the recipe:\n' + 'SRC_URI[%s] = "%s"', + ud.localpath, ud.md5_name, md5data) + + if ud.sha256_expected == None: + logger.warn('Missing sha256 SRC_URI checksum for %s, consider adding to the recipe:\n' + 'SRC_URI[%s] = "%s"', + ud.localpath, ud.sha256_name, sha256data) + + md5mismatch = False + sha256mismatch = False if ud.md5_expected != md5data: - raise MD5SumError(ud.localpath, ud.md5_expected, md5data, u) + md5mismatch = True if ud.sha256_expected != sha256data: - raise SHA256SumError(ud.localpath, ud.sha256_expected, sha256data, u) + sha256mismatch = True + + # We want to alert the user if a checksum is defined in the recipe but + # it does not match. + msg = "" + if md5mismatch and ud.md5_expected: + msg = msg + "\nFile: '%s' has %s checksum %s when %s was expected (from URL: '%s')" % (ud.localpath, 'md5', ud.md5_expected, md5data, u) + + if sha256mismatch and ud.sha256_expected: + msg = msg + "\nFile: '%s' has %s checksum %s when %s was expected (from URL: '%s')" % (ud.localpath, 'sha256', ud.sha256_expected, sha256data, u) + + if len(msg): + raise FetchError('Checksum mismatch!%s' % msg, u) + def update_stamp(u, ud, d): """
Signature · 1 line
1.7.6.4
Re: [PATCH 1/2] fetch2: improve usability of checksums
Quoted 11 lines · Joshua Lock
This patch improves the usability of checksums by only requiring one checksum be defined.
Further, checksum verification will provide as much information as possible at, rather than a bit at a time. No longer will you need to run fetch, see an md5sum mismatch, fix it, run fetch, seen an sha256sum mismatch, fix it and fetch again. If neither checksum is defined we now report both missing sums at once - rather than one after the other.
Finally, if both sums are incorrect, we'll report both incorrect sums at the same time.
Quoted 1 line
Fixes part of [YOCTO #1399]
Nicely done, the fetch/mismatch cycle was quite an annoying one :)
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Re: [PATCH 1/2] fetch2: improve usability of checksums
Quoted 12 lines · Chris Larson
Quoted 11 lines · Joshua Lock
This patch improves the usability of checksums by only requiring one checksum be defined.
Further, checksum verification will provide as much information as possible at, rather than a bit at a time. No longer will you need to run fetch, see an md5sum mismatch, fix it, run fetch, seen an sha256sum mismatch, fix it and fetch again. If neither checksum is defined we now report both missing sums at once - rather than one after the other.
Finally, if both sums are incorrect, we'll report both incorrect sums at the same time.
Quoted 3 lines
Quoted 1 line
Fixes part of [YOCTO #1399]
Nicely done, the fetch/mismatch cycle was quite an annoying one :)
yeah, great, thanks!
finally it looks like
http://git.openembedded.org/openembedded/commit?id=353b4dc7223f8dd50fcf21d7c031b9dd91a724a2
again :)
Regards,
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Re: [PATCH 1/2] fetch2: improve usability of checksums
In reply to Joshua Locklib/bb/fetch2/__init__.py def verify_checksum(u, ud, d): · line 302
Quoted 13 lines · Joshua Lock
if ud.sha256_expected != sha256data:lib/bb/fetch2/__init__.py +11
-1- raise SHA256SumError(ud.localpath, ud.sha256_expected, sha256data, u) + sha256mismatch = True + + # We want to alert the user if a checksum is defined in the recipe but + # it does not match. + msg = "" + if md5mismatch and ud.md5_expected: + msg = msg + "\nFile: '%s' has %s checksum %s when %s was expected (from URL: '%s')" % (ud.localpath, 'md5', ud.md5_expected, md5data, u) + + if sha256mismatch and ud.sha256_expected: + msg = msg + "\nFile: '%s' has %s checksum %s when %s was expected (from URL: '%s')" % (ud.localpath, 'sha256', ud.sha256_expected, sha256data, u) +
Just realised I was passing the expected and data parameters in the wrong order - I've pushed a fix to the same branch.
Cheers,
Joshua
Signature · 3 lines
Joshua Lock
Yocto Project "Johannes factotum"
Intel Open Source Technology CentrePATCH 2/2 fetch2: enable checksum definition as SRC_URI parameter
In reply to Joshua LockURI parameters should be able to be defined as a parameter of the SRC_URI, this patch enables thus for checksums.
An example;
Addresses the remainder of [YOCTO #1399]
--- lib/bb/fetch2/__init__.py | 10 ++++++++-- 1 files changed, 8 insertions(+), 2 deletions(-)
lib/bb/fetch2/__init__.py +8 -2
@@ -562,8 +562,14 @@ class FetchData(object): else: self.md5_name = "md5sum" self.sha256_name = "sha256sum" - self.md5_expected = bb.data.getVarFlag("SRC_URI", self.md5_name, d) - self.sha256_expected = bb.data.getVarFlag("SRC_URI", self.sha256_name, d) + if self.md5_name in self.parm: + self.md5_expected = self.parm[self.md5_name] + else: + self.md5_expected = bb.data.getVarFlag("SRC_URI", self.md5_name, d) + if self.sha256_name in self.parm: + self.sha256_expected = self.parm[self.sha256_name] + else: + self.sha256_expected = bb.data.getVarFlag("SRC_URI", self.sha256_name, d) self.names = self.parm.get("name",'default').split(',')
Signature · 1 line
1.7.6.4
Re: [PATCH 2/2] fetch2: enable checksum definition as SRC_URI parameter
Quoted 6 lines · Joshua Lock
URI parameters should be able to be defined as a parameter of the
SRC_URI, this patch enables thus for checksums.An example;
this should go into manual/documentation as well.
Quoted 38 lines
Addresses the remainder of [YOCTO #1399]
Signed-off-by Joshua Lock <[email protected]>---
lib/bb/fetch2/__init__.py | 10 ++++++++--1 files changed, 8 insertions(+), 2 deletions(-)lib/bb/fetch2/__init__.py +0
-0@@ -562,8 +562,14 @@ class FetchData(object):else:
self.md5_name = "md5sum"
self.sha256_name = "sha256sum"
- self.md5_expected =bb.data.getVarFlag("SRC_URI", self.md5_name, d)
- self.sha256_expected =bb.data.getVarFlag("SRC_URI", self.sha256_name, d)
+ if self.md5_name in self.parm:
+ self.md5_expected = self.parm[self.md5_name]
+ else:
+ self.md5_expected =bb.data.getVarFlag("SRC_URI", self.md5_name, d)
+ if self.sha256_name in self.parm:
+ self.sha256_expected = self.parm[self.sha256_name]
+ else:
+ self.sha256_expected =bb.data.getVarFlag("SRC_URI", self.sha256_name, d)self.names = self.parm.get("name",'default').split(',')
--
1.7.6.4_______________________________________________
bitbake-devel mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
Re: [PATCH 2/2] fetch2: enable checksum definition as SRC_URI parameter
Quoted 9 lines · Khem Raj
Quoted 6 lines · Joshua Lock
URI parameters should be able to be defined as a parameter of the
SRC_URI, this patch enables thus for checksums.An example;
this should go into manual/documentation as well.
Good point, I pushed an extra patch onto the branch for the manual:
https://github.com/incandescant/bitbake/commit/fcf17a39ce89b5911ffd9f89ff13505a7a7633fd
Regards,
Joshua
Signature · 3 lines
Joshua Lock
Yocto Project "Johannes factotum"
Intel Open Source Technology CentreQuoted 8 lines · Joshua Lock
I'd heard some complaints around the usability of checksums and, having done some recipe work recently, must agree with them.
This series ams to make using checksums less intrusive by:a) Removing the requirement that if one type of checksum is defined the other must be too.
b) Reporting all sum mismatches at once
c) Enabling sums to be defined as a parameter of aSRC_URI
Very nice, thank you!
Signature · 3 lines
Scott Garman Embedded Linux Engineer - Yocto Project Intel Open Source Technology Center
Richard Purdie
AppliedQuoted 16 lines · Joshua Lock
I'd heard some complaints around the usability of checksums and, having done some recipe work recently, must agree with them.
This series ams to make using checksums less intrusive by:a) Removing the requirement that if one type of checksum is defined the other must be too.
b) Reporting all sum mismatches at once
c) Enabling sums to be defined as a parameter of aSRC_URIRegards,
JoshuaThe following changes since commit
dca46cc2e1c75b6add2c4801e2994a4812745f5b:fetch2: Export additional variables to the fetchers (2011-10-05 14:23:16 +0100)
Quoted 7 lines
are available in the git repository at:
git://github.com/incandescant/bitbakechecksum
https://github.com/incandescant/bitbake/tree/checksumJoshua Lock (2):
fetch2: improve usability of checksums
fetch2: enable checksum definition asSRC_URIparameter
Merged to master (or will be when I can actually push the changes).
Cheers,
Richard