Skip to contentSkip to conversation

new PRService features in meta

PATCH In review Lianhao Lu 2 participants 5 messages Jan 10, 2012 – Jan 12, 2012

lore
Share & tools
lore

Lianhao Lu

author
#0 lore raw

v3 cover new PRService features in meta

V3: Using event handler instead of new tasks to interfacing to bitbake.
V2: Using bb.fatal instead of bb.FuncFailed.

NOTE: This set of patches added supporting for new features in meta layer. It should work together with another branch of patches in bitbake.
git://git.pokylinux.org/poky-contrib llu/PRService_bitbake

It supports the export/import AUTOPR values from/to PRservice to lockdown the PR values for release. The typical usage scenario is:

1. UserA enabled the PRService and run buildings. The PRService will record the AUTOPR values(which will be included in the finale PKGR) for different packages the userA builds.

2. UserA runs "bitbake-prserv-tool export <file>" to export the AUTOPR values from the current PRService into an exported inc file.

3. UserB gets the exported inc file and runs the command
"bitbake -R <exported file> target" to lockdown and reproduce the same AUTOPR when generating packages.

4. UserB may run "bitbake-prserv-tool import <file>" to import the exported AUTOPR values back into his own PRService. The AUTOPR values will be incremented from there.

This set of changes also supports the per-recipe based PRSERV_HOST and PRSERV_PORT.

This set of changes makes the basichash instead of the basic as the default signature generator.

The following changes since commit fff18970706913e7fd7f4a119d798dddb44b388a:
Gary Thomas (1):
base-files: filesystems: fix mount order

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib llu/PRService_oecore
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=llu/PRService_oecore

Lianhao Lu (3):
meta/PRService: Added export/import fuctions. package.bbclass: per recipe PRSERV_HOST support conf/bitbake.conf: basichash as default signature.

 meta/classes/package.bbclass  |   21 +++++++-
 meta/classes/prexport.bbclass |   45 ++++++++++++++++
 meta/classes/primport.bbclass |   17 ++++++
 meta/classes/prserv.bbclass   |   36 +++++--------
 meta/conf/bitbake.conf        |    6 +-
 meta/conf/prexport.conf       |    1 +
 meta/conf/primport.conf       |    1 +
 meta/lib/oe/prservice.py      |  113 +++++++++++++++++++++++++++++++++++++++++
 scripts/bitbake-prserv-tool   |   57 +++++++++++++++++++++
 9 files changed, 269 insertions(+), 28 deletions(-)
 create mode 100644 meta/classes/prexport.bbclass
 create mode 100644 meta/classes/primport.bbclass
 create mode 100644 meta/conf/prexport.conf
 create mode 100644 meta/conf/primport.conf
 create mode 100644 meta/lib/oe/prservice.py
 create mode 100755 scripts/bitbake-prserv-tool
author
#1 lore raw

PATCH v3 1/3 meta/PRService: Added export/import fuctions.

[YOCTO #1556]
- Modified meta/class/package.bbclass and prserv.bbclass according to the change in PR service by adding PACKAGE_ARCH into the query tuple.

- Added prexport.bbclass, primport.bbclass to export/import AUTOPR values from/to PRService.

- Move PR service related common code to lib/oe/prservice.py.

- Supported reading the AUTOPR values from the exported .inc file instead of reading it from remote PR service.

- Created a new script bitbake-prserv-tool to export/import the AUTOPR values from/to the PR service.

Typical usage scenario of the export/import is:
1. bitbake-prserv-tool export <file> to export the AUTOPR values from the current PR service into an exported .inc file.

2. Others may use that exported .inc file(to be included in the local.conf) to lockdown and reproduce the same AUTOPR when generating package feeds.

3. Others may "bitbake-prserv-tool import <file>" to import the AUTOPR values into their own PR service and the AUTOPR values will be incremented from there.

---
 meta/classes/package.bbclass  |   13 ++++-
 meta/classes/prexport.bbclass |   45 ++++++++++++++++
 meta/classes/primport.bbclass |   17 ++++++
 meta/classes/prserv.bbclass   |   36 +++++--------
 meta/conf/bitbake.conf        |    4 +-
 meta/conf/prexport.conf       |    1 +
 meta/conf/primport.conf       |    1 +
 meta/lib/oe/prservice.py      |  113 +++++++++++++++++++++++++++++++++++++++++
 scripts/bitbake-prserv-tool   |   57 +++++++++++++++++++++
 9 files changed, 260 insertions(+), 27 deletions(-)
 create mode 100644 meta/classes/prexport.bbclass
 create mode 100644 meta/classes/primport.bbclass
 create mode 100644 meta/conf/prexport.conf
 create mode 100644 meta/conf/primport.conf
 create mode 100644 meta/lib/oe/prservice.py
 create mode 100755 scripts/bitbake-prserv-tool
meta/classes/package.bbclass +10 -3
index 9040eb4..65e6571 100644
@@ -351,10 +351,17 @@ def runtime_mapping_rename (varname, d):
 
 python package_get_auto_pr() {
 	if d.getVar('USE_PR_SERV', True) != "0":
-		auto_pr=prserv_get_pr_auto(d)
-		if auto_pr is None:
-			bb.fatal("Can NOT get auto PR revision from remote PR service")
+		try:
+			auto_pr=prserv_get_pr_auto(d)
+		except Exception as e:
+			bb.fatal("Can NOT get PRAUTO, exception %s" %  str(e))
 			return
+		if auto_pr is None:
+			if d.getVar('PRSERV_LOCKDOWN', True):
+				bb.fatal("Can NOT get PRAUTO from lockdown exported file")
+			else:
+				bb.fatal("Can NOT get PRAUTO from remote PR service")
+			return 
 		d.setVar('PRAUTO',str(auto_pr))
 }
 
meta/classes/prexport.bbclass new file +45 -0

45 lines not on this page · raw message

meta/classes/primport.bbclass new file +17 -0
new file mode 100644
index 0000000..08e5a8f
@@ -0,0 +1,17 @@
+python primport_handler () {
+    import bb.event
+    if not e.data:
+        return
+
+    if isinstance(e, bb.event.ParseCompleted):
+        import oe.prservice
+        #import all exported AUTOPR values
+        imported = oe.prservice.prserv_import_db(e.data)
+        if imported is None:
+            bb.fatal("import failed!")
+
+        for (version, pkgarch, checksum, value) in imported:
+            bb.note("imported (%s,%s,%s,%d)" % (version, pkgarch, checksum, value))
+}
+
+addhandler primport_handler
meta/classes/prserv.bbclass +14 -22
index 18b8589..0825306 100644
@@ -1,29 +1,21 @@
-def prserv_make_conn(d):
-    import prserv.serv
-    host=d.getVar("PRSERV_HOST",True)
-    port=d.getVar("PRSERV_PORT",True)
-    try:
-        conn=None
-        conn=prserv.serv.PRServerConnection(host,int(port))
-        d.setVar("__PRSERV_CONN",conn)
-    except Exception, exc:
-        bb.fatal("Connecting to PR service %s:%s failed: %s" % (host, port, str(exc)))
-
-    return conn
-
 def prserv_get_pr_auto(d):
-    if d.getVar('USE_PR_SERV', True) != "0":
+    import oe.prservice
+    if d.getVar('USE_PR_SERV', True) != "1":
         bb.warn("Not using network based PR service")
         return None
 
-    conn=d.getVar("__PRSERV_CONN", True)
-    if conn is None:
-        conn=prserv_make_conn(d)
+    version = d.getVar("PRAUTOINX", True)
+    pkgarch = d.getVar("PACKAGE_ARCH", True)
+    checksum = d.getVar("BB_TASKHASH", True)
+
+    if d.getVar('PRSERV_LOCKDOWN', True):
+        auto_rev = d.getVar('PRAUTO_' + version + '_' + pkgarch, True) or d.getVar('PRAUTO_' + version, True) or None
+    else:
+        conn = d.getVar("__PRSERV_CONN", True)
         if conn is None:
-            return None
+            conn = oe.prservice.prserv_make_conn(d)
+            if conn is None:
+                return None
+        auto_rev = conn.getPR(version, pkgarch, checksum)
 
-    version=d.getVar("PF", True)
-    checksum=d.getVar("BB_TASKHASH", True)
-    auto_rev=conn.getPR(version,checksum)
-    bb.debug(1,"prserv_get_pr_auto: version: %s checksum: %s result %d" % (version, checksum, auto_rev))
     return auto_rev
meta/conf/bitbake.conf +2 -2
index b858364..5deedec 100644
@@ -190,7 +190,7 @@ BP = "${BPN}-${PV}"
 #
 # network based PR service
 #
-USE_PR_SERV = "${@[1,0][(d.getVar('PRSERV_HOST',1) is None) or (d.getVar('PRSERV_PORT',1) is None)]}"
+USE_PR_SERV = "${@[1,0][((not d.getVar('PRSERV_HOST', True)) or (not d.getVar('PRSERV_PORT', True))) and (not d.getVar('PRSERV_LOCKDOWN', True))]}"
 
 # Package info.
 
@@ -733,7 +733,7 @@ BB_CONSOLELOG = "${TMPDIR}/cooker.log.${DATETIME}"
 # Setup our default hash policy
 BB_SIGNATURE_HANDLER ?= "basic"
 BB_HASHTASK_WHITELIST ?= "(.*-cross$|.*-native$|.*-cross-initial$|.*-cross-intermediate$|^virtual:native:.*|^virtual:nativesdk:.*)"
-BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE"
+BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST PRSERV_PORT PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN"
 
 MLPREFIX ??= ""
 MULTILIB_VARIANTS ??= ""
meta/conf/prexport.conf new file +1 -0
new file mode 100644
index 0000000..12f3acb
@@ -0,0 +1 @@
+INHERIT += "prexport"
meta/conf/primport.conf new file +1 -0
new file mode 100644
index 0000000..d94ea1b
@@ -0,0 +1 @@
+INHERIT += "primport"
meta/lib/oe/prservice.py new file +113 -0

113 lines not on this page · raw message

scripts/bitbake-prserv-tool new file +57 -0

57 lines not on this page · raw message

Signature · 1 line
1.7.0.4
author
#2 lore raw

PATCH v3 2/3 package.bbclass: per recipe PRSERV_HOST support

[YOCTO #1126]
Added per recipe PRSERV_HOST PRSERV_PORT support.

---
 meta/classes/package.bbclass |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)
meta/classes/package.bbclass +8 -0
index 65e6571..f8f0a12 100644
@@ -350,6 +350,14 @@ def runtime_mapping_rename (varname, d):
 #
 
 python package_get_auto_pr() {
+	# per recipe PRSERV_HOST PRSERV_PORT
+	pn = d.getVar('PN', True)
+	host = d.getVar("PRSERV_HOST_" + pn, True)
+	port = d.getVar("PRSERV_PORT_" + pn, True)
+	if not (host is None):
+		d.setVar("PRSERV_HOST", host)
+	if not (port is None):
+		d.setVar("PRSERV_PORT", port)
 	if d.getVar('USE_PR_SERV', True) != "0":
 		try:
 			auto_pr=prserv_get_pr_auto(d)
Signature · 1 line
1.7.0.4
author
#3 lore raw

PATCH v3 3/3 conf/bitbake.conf: basichash as default signature.

In reply to Lianhao Lu

[YOCTO #1555]
Make basichash as default signature handler

---
 meta/conf/bitbake.conf |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
meta/conf/bitbake.conf +1 -1
index 5deedec..a5d695c 100644
@@ -731,7 +731,7 @@ TRANSLATED_TARGET_ARCH ??= "${@d.getVar('TARGET_ARCH', True).replace("_", "-")}"
 BB_CONSOLELOG = "${TMPDIR}/cooker.log.${DATETIME}"
 
 # Setup our default hash policy
-BB_SIGNATURE_HANDLER ?= "basic"
+BB_SIGNATURE_HANDLER ?= "basichash"
 BB_HASHTASK_WHITELIST ?= "(.*-cross$|.*-native$|.*-cross-initial$|.*-cross-intermediate$|^virtual:native:.*|^virtual:nativesdk:.*)"
 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST PRSERV_PORT PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN"
 
Signature · 1 line
1.7.0.4
#4 lore raw
In reply to Lianhao Lu
Quoted 35 lines · Lianhao Lu

V3: Using event handler instead of new tasks to interfacing to bitbake.
V2: Using bb.fatal instead of bb.FuncFailed.

NOTE: This set of patches added supporting for new features in meta layer. It should work together with another branch of patches in bitbake.
git://git.pokylinux.org/poky-contrib llu/PRService_bitbake

It supports the export/import AUTOPR values from/to PRservice to lockdown the PR values for release. The typical usage scenario is:

1. UserA enabled the PRService and run buildings. The PRService will record the AUTOPR values(which will be included in the finale PKGR) for different packages the userA builds.

2. UserA runs "bitbake-prserv-tool export<file>" to export the AUTOPR values from the current PRService into an exported inc file.

3. UserB gets the exported inc file and runs the command
"bitbake -R<exported file> target" to lockdown and reproduce the same AUTOPR when generating packages.

4. UserB may run "bitbake-prserv-tool import<file>" to import the exported AUTOPR values back into his own PRService. The AUTOPR values will be incremented from there.

This set of changes also supports the per-recipe based PRSERV_HOST and PRSERV_PORT.

This set of changes makes the basichash instead of the basic as the default signature generator.

The following changes since commit fff18970706913e7fd7f4a119d798dddb44b388a:
Gary Thomas (1):
base-files: filesystems: fix mount order

Quoted 8 lines

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib llu/PRService_oecore
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=llu/PRService_oecore

Lianhao Lu (3):
meta/PRService: Added export/import fuctions. package.bbclass: per recipe PRSERV_HOST support

Merged these first 2 as infrastructure

Quoted 2 lines

conf/bitbake.conf: basichash as default signature.

Held off on this, please enable it in poky.conf first for more testing in poky first.

Thanks
Sau!

Quoted 23 lines
  meta/classes/package.bbclass  |   21 +++++++-
  meta/classes/prexport.bbclass |   45 ++++++++++++++++
  meta/classes/primport.bbclass |   17 ++++++
  meta/classes/prserv.bbclass   |   36 +++++--------
  meta/conf/bitbake.conf        |    6 +-
  meta/conf/prexport.conf       |    1 +
  meta/conf/primport.conf       |    1 +
  meta/lib/oe/prservice.py      |  113 +++++++++++++++++++++++++++++++++++++++++
  scripts/bitbake-prserv-tool   |   57 +++++++++++++++++++++
  9 files changed, 269 insertions(+), 28 deletions(-)

create mode 100644 meta/classes/prexport.bbclass create mode 100644 meta/classes/primport.bbclass create mode 100644 meta/conf/prexport.conf create mode 100644 meta/conf/primport.conf create mode 100644 meta/lib/oe/prservice.py create mode 100755 scripts/bitbake-prserv-tool

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core