Skip to contentSkip to conversation

Add SRC_URI checksum support

RFC PATCH In review Yu Ke 3 participants 5 messages Dec 2, 2010 – Dec 7, 2010

lore
Share & tools
lore

Yu Ke

author
#0 lore raw

cover Add SRC_URI checksum support

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/srcuri
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri

Thanks,
Yu Ke <[email protected]>
---

Yu Ke (2):
[RFC] Add SRC_URI checksum
[RFC] Automatically patch recipes to add the SRC_URI checksum

 meta/classes/base.bbclass  |    4 +-
 meta/classes/utils.bbclass |  101 ++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 92 insertions(+), 13 deletions(-)
author
#1 lore raw

RFC PATCH 1/2 Add SRC_URI checksum

This patch add the per-recipe SRC_URI checksum. The code is ported from OE.

- SRC_URI format
The format of SRC_URI checksum follow OE definition:

1. SRC_URI has single src
SRC_URI = "http://some.domain/file.tar.gz" SRC_URI[md5sum] = "xxxxxxxxxxxxxxx" SRC_URI[sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"

2. SRC_URI has multiple src, every src need specify name

SRC_URI = "http://some.domain/file1.tar.gz;name=name1 \
           http://some.domain/file2.tar.gz;name=name2 "

SRC_URI[name1.md5sum] = "xxxxxxxxxxxxxxx" SRC_URI[name1.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx" SRC_URI[name2.md5sum] = "xxxxxxxxxxxxxxx" SRC_URI[name2.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"

- SRC_URI checking invocation:
the checksum checking is invoked in do_fetch phase, so it can be invoked manually by

# bitbake -f -c fetch <recipe_name>

if recipes has no SRC_URI checksum item, bitbake will show warning:
"
WARNING: Missing SRC_URI checksum for xxxx.tar.gz, recommend to add SRC_URI[md5sum] = "5c69f16d452b0bb3d44bc3c10556c072"
SRC_URI[sha256sum] = "f4e0ada8d4d516bbb8600a3ee7d9046c9c79e38cd781df9ffc46d8f16acd1768" "

- Control Varialbe
One variable OE_STRICT_CHECKSUMS is also defined to toggle strict checking. if OE_STRICT_CHECKSUMS = "1"
then there will be fatal error if SRC_URI is not defined during do_fetch

---
 meta/classes/base.bbclass  |    4 ++-
 meta/classes/utils.bbclass |   78 +++++++++++++++++++++++++++++++++++++-------
 2 files changed, 69 insertions(+), 13 deletions(-)
meta/classes/base.bbclass +3 -1
index 384e723..f9955bc 100644
@@ -181,10 +181,12 @@ python base_do_fetch() {
 	# Check each URI
 	for url in src_uri.split():
 		localpath = bb.data.expand(bb.fetch.localpath(url, localdata), localdata)
-		(type,host,path,_,_,_) = bb.decodeurl(url)
+		(type,host,path,_,_,params) = bb.decodeurl(url)
 		uri = "%s://%s%s" % (type,host,path)
 		try:
 			if type == "http" or type == "https" or type == "ftp" or type == "ftps":
+				if not base_chk_src_uri_checksum(localpath, params, d):
+					bb.fatal("%s-%s: %s cannot check archive integrity" % (pn,pv,uri))
 				if not base_chk_file(parser, pn, pv,uri, localpath, d):
 					bb.note("%s-%s: %s has no entry in conf/checksums.ini, not checking URI" % (pn,pv,uri))
 		except Exception:
meta/classes/utils.bbclass +66 -12

92 lines not on this page · raw message

Signature · 1 line
1.7.0.4
author
#2 lore raw

RFC PATCH 2/2 Automatically patch recipes to add the SRC_URI checksum

In reply to Yu Ke

Initially, most recpies don't have SRC_URI[] checksum, so we need one mechanism to add SRC_URI[] checksum for all recpies.

This patch add the mechanism, the usage is:

1. add the following variable to conf/local.conf:
OE_CHECKSUMS_PATCH = "1"

2. # bitbake -f -c fetch <recipe-name>

it will automatically patch the recipe file to append checksum, like SRC_URI[md5sum] = "6e497f861c77bbba2f7da4e10270995b"
SRC_URI[sha256sum] = "f3f6ce41b8e0f327abd05c95990f113ddafbae131e10f79a99728ed46458494b"

Note: this patch is only for one time usage, so not sure if we need to checked into upstream

---
 meta/classes/utils.bbclass |   23 +++++++++++++++++++++++
 1 files changed, 23 insertions(+), 0 deletions(-)
meta/classes/utils.bbclass +23 -0
index 766cfc5..2729fe4 100644
@@ -146,6 +146,8 @@ def base_chk_src_uri_checksum(localpath, params, data):
         bb.warn("Missing SRC_URI checksum for %s, recommend to add\n" \
                 "SRC_URI[%s] = \"%s\"\nSRC_URI[%s] = \"%s\"" \
 				% (localpath, md5flag, md5data, sha256flag, sha256data))
+        base_patch_src_uri_checksum(expected_md5sum, expected_sha256sum, \
+                md5data, sha256data, md5flag, sha256flag, data)
         return not strict_checking
 
     if (expected_md5sum != md5data or expected_sha256sum != sha256data):
@@ -208,6 +210,27 @@ def base_chk_file(parser, pn, pv, src_uri, localpath, data):
 
     return True
 
+def base_patch_src_uri_checksum(expected_md5sum, expected_sha256sum, md5data, sha256data, md5flag, sha256flag, data):
+    #
+    # Purpose: patch the recpies file to append SRC_URI[] checksum automatically
+    #          this behavor is only enabled when OE_CHECKSUMS_PATCH = "1"
+    #
+    if (bb.data.getVar("OE_CHECKSUMS_PATCH", data, True) != "1"):
+        return
+
+    buf = []
+    if (expected_md5sum == None):
+        buf.append('SRC_URI[%s] = \"%s\"' % (md5flag, md5data))
+    if (expected_sha256sum == None):
+        buf.append('SRC_URI[%s] = \"%s\"' % (sha256flag, sha256data))
+    if ( len(buf) > 0):
+        bbfile = bb.data.getVar("FILE", data, True)
+        bb.note("bbfile=%s" % bbfile)
+        f = open(bbfile, 'a')
+        f.write('\n'.join(buf))
+        f.close()
+        bb.note("Add SRC_URI checksum for %s succeeded" % bbfile)
+
 oe_soinstall() {
 	# Purpose: Install shared library file and
 	#          create the necessary links
Signature · 1 line
1.7.0.4
#3 lore raw
In reply to Yu Ke

Ke,
This automatic SRC_URI checksum is what I was looking for. Thanks for doing it.

Nitin

Quoted 30 lines

-----Original Message-----
From: [email protected] [mailto:poky- [email protected]] On Behalf Of Yu Ke
Sent: Thursday, December 02, 2010 4:04 AM
To: [email protected]
Subject: [poky] [RFC PATCH 0/2] Add SRC_URI checksum support

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/srcuri
Browse: http://git.pokylinux.org/cgit.cgi/poky-
contrib/log/?h=kyu3/srcuri

Thanks,
Yu Ke <[email protected]>
---

Yu Ke (2):
[RFC] Add SRC_URI checksum
[RFC] Automatically patch recipes to add the SRC_URI checksum

 meta/classes/base.bbclass  |    4 +-
 meta/classes/utils.bbclass |  101
++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 92 insertions(+), 13 deletions(-)

_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

#4 lore raw
Quoted 3 lines · Nitin A

Ke,
This automatic SRC_URI checksum is what I was looking for. Thanks for doing it.

Nitin,

I will be running this over the complete metadata set this week, please do not use this on your recipes since it adds the checksums to the end and not inline.

Thanks

Sau!

Quoted 38 lines

Nitin

Quoted 26 lines

-----Original Message-----
From: [email protected] [mailto:poky- [email protected]] On Behalf Of Yu Ke
Sent: Thursday, December 02, 2010 4:04 AM
To: [email protected]
Subject: [poky] [RFC PATCH 0/2] Add SRC_URI checksum support

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/srcuri
Browse: http://git.pokylinux.org/cgit.cgi/poky-
contrib/log/?h=kyu3/srcuri

Thanks,
Yu Ke<[email protected]>
---

Yu Ke (2):
[RFC] Add SRC_URI checksum
[RFC] Automatically patch recipes to add the SRC_URI checksum

  meta/classes/base.bbclass  |    4 +-
  meta/classes/utils.bbclass |  101
++++++++++++++++++++++++++++++++++++++-----
  2 files changed, 92 insertions(+), 13 deletions(-)
Quoted 4 lines

_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky