new PRService features in bitbake

PATCH New Lianhao Lu 1 participant 3 messages Dec 30, 2011

Share & tools

Lianhao Lu

author

2011-12-30 03:58 UTC #0 lore raw

cover new PRService features in bitbake

NOTE: This set of patches added supporting for new features in bitbake. It should work together with another branch of patches in meta layers.
git://git.pokylinux.org/poky-contrib llu/PRService_oecore

It added PRService's export/import support in bitbake and also added a no history mode as the default mode for PRSerivce. In this mode, the AUTOPR value(which will be included in the finale PKGR) returned by the PRService will never decrement. It also added PACKAGE_ARCH into the index to the DB table, so the packages for different architcture will have different AUTOPR values.

The PRService can be started in 2 ways. If the user wants share the PRService for multiple buildings, it can start the PRService in the old way, by running "bitbake-prserv --start [options]", and set PRSERV_HOST and PRSERV_PORT in local.conf accordingly.

If the user only wants the PRService to be used by current build, a local service may be started automatically by setting PRSERV_HOST to "localhost" and PRSERV_PORT to "0". This automatically started local service will only be effective to the builds running in the same build directory.

The following changes since commit f5aa3bbda623c8fae3a761d72fddc95631ad0706:
Saul Wold (1):
coreutils: ensure --color works so DEPEND on libcap

are available in the git repository at:

git://git.pokylinux.org/poky-contrib llu/PRService_bitbake
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=llu/PRService_bitbake

Lianhao Lu (2):
bitbake/PRservice: Added no_hist mode and export/import.
bitbake: Automatically start local PR service.

 bitbake/bin/bitbake-prserv     |   26 +++---
 bitbake/lib/bb/cooker.py       |    3 +
 bitbake/lib/prserv/__init__.py |    5 +-
 bitbake/lib/prserv/db.py       |  213 +++++++++++++++++++++++++++++++++------
 bitbake/lib/prserv/serv.py     |  208 +++++++++++++++++++++++++++++----------
 5 files changed, 358 insertions(+), 97 deletions(-)
 mode change 100644 => 100755 bitbake/bin/bitbake-prserv

Lianhao Lu

author

2011-12-30 03:58 UTC #1 lore raw

PATCH 1/2 bitbake/PRservice: Added no_hist mode and export/import.

[YOCTO #1556]
1. Added the package_arch into the index to the DB table. Because the change in PACKAGE_ARCH will results in different checksum, and it is better to have seperate PR value domains for differnt PACKAGE_ARCH of the same pakcage.

2. Changed the PR service to operate in no history mode. In this mode, the for a given query tuple (version, pkgarch, checksum), the returned value will be the largest among all the values of the same (version, pkgarch). This means the PR value returned can NOT be decremented.

3. Added export function. For each (version, pkgarch) tuple, only the record with the maximum value will be exported.

4. Added import function. The record will only be imported if the imported value is larger than the value stored in the DB with the same (version, pkgarch, checksum) tuple.

Signed-off-by Lianhao Lu <[email protected]>

---
 bitbake/bin/bitbake-prserv     |   23 +++--
 bitbake/lib/prserv/__init__.py |    5 +-
 bitbake/lib/prserv/db.py       |  209 ++++++++++++++++++++++++++++++++++------
 bitbake/lib/prserv/serv.py     |  106 +++++++++++++-------
 4 files changed, 263 insertions(+), 80 deletions(-)
 mode change 100644 => 100755 bitbake/bin/bitbake-prserv

bitbake/bin/bitbake-prserv +13 ~~-10~~

old mode 100644
new mode 100755
index 14073ca..a7ab55f
@@ -16,31 +16,34 @@ PRPORT_DEFAULT=8585
 def main():
     parser = optparse.OptionParser(
         version="Bitbake PR Service Core version %s, %%prog version %s" % (prserv.__version__, __version__),
-        usage = "%prog [options]")
+        usage = "%prog < --start | --stop > [options]")
 
-    parser.add_option("-f", "--file", help="database filename(default prserv.db)", action="store",
+    parser.add_option("-f", "--file", help="database filename(default: prserv.db)", action="store",
                       dest="dbfile", type="string", default="prserv.db")
-    parser.add_option("-l", "--log", help="log filename(default prserv.log)", action="store",
+    parser.add_option("-l", "--log", help="log filename(default: prserv.log)", action="store",
                       dest="logfile", type="string", default="prserv.log")
     parser.add_option("--loglevel", help="logging level, i.e. CRITICAL, ERROR, WARNING, INFO, DEBUG",
-                      action = "store", type="string", dest="loglevel", default = "WARNING")
+                      action = "store", type="string", dest="loglevel", default = "INFO")
     parser.add_option("--start", help="start daemon",
-                      action="store_true", dest="start", default="True")
+                      action="store_true", dest="start")
     parser.add_option("--stop", help="stop daemon",
-                      action="store_false", dest="start")
+                      action="store_true", dest="stop")
     parser.add_option("--host", help="ip address to bind", action="store",
                       dest="host", type="string", default=PRHOST_DEFAULT)
-    parser.add_option("--port", help="port number(default 8585)", action="store",
+    parser.add_option("--port", help="port number(default: 8585)", action="store",
                       dest="port", type="int", default=PRPORT_DEFAULT)
 
     options, args = parser.parse_args(sys.argv)
-
     prserv.init_logger(os.path.abspath(options.logfile),options.loglevel)
 
     if options.start:
-        prserv.serv.start_daemon(options)
+        ret=prserv.serv.start_daemon(dbfile=options.dbfile, interface=(options.host, options.port),
+                      logfile=os.path.abspath(options.logfile))
+    elif options.stop:
+        ret=prserv.serv.stop_daemon(options.host, options.port)
     else:
-        prserv.serv.stop_daemon()
+        ret=parser.print_help()
+    return ret
 
 if __name__ == "__main__":
     try:

bitbake/lib/prserv/__init__.py +4 -1

index 2837e13..c27fffe 100644
@@ -7,5 +7,8 @@ def init_logger(logfile, loglevel):
     numeric_level = getattr(logging, loglevel.upper(), None)
     if not isinstance(numeric_level, int):
         raise ValueError('Invalid log level: %s' % loglevel)
-    logging.basicConfig(level=numeric_level, filename=logfile)
+    FORMAT = '%(asctime)-15s %(message)s'
+    logging.basicConfig(level=numeric_level, filename=logfile, format=FORMAT)
 
+class NotFoundError(StandardError):
+    pass
\ No newline at end of file

bitbake/lib/prserv/db.py +177 ~~-32~~

270 lines not on this page · raw message

bitbake/lib/prserv/serv.py +69 ~~-37~~

201 lines not on this page · raw message

Signature · 1 line

1.7.0.4

Lianhao Lu

author

2011-12-30 03:58 UTC #2 lore raw

PATCH 2/2 bitbake: Automatically start local PR service.

[YOCTO #1126]
A local PR service will be started and stopped automatically along with the bitbake invocation/ternimation.

This local PR service will be started only and if only when the PRSERV_HOST is set to 'localhost' and PRSERV_PORT is set to '0'.

When started, the sqlite3 database is stored at "${PERSISTEN_DIR}/prserv.sqlite3" or "${CACHE}/prserv.sqlite3".

Signed-off-by Lianhao Lu <[email protected]>

---
 bitbake/bin/bitbake-prserv |    9 +--
 bitbake/lib/bb/cooker.py   |    3 +
 bitbake/lib/prserv/db.py   |   12 ++--
 bitbake/lib/prserv/serv.py |  144 +++++++++++++++++++++++++++++++++-----------
 4 files changed, 123 insertions(+), 45 deletions(-)

bitbake/bin/bitbake-prserv +4 -5

index a7ab55f..a8d7acb 100755
@@ -10,7 +10,7 @@ import prserv.serv
 
 __version__="1.0.0"
 
-PRHOST_DEFAULT=''
+PRHOST_DEFAULT='0.0.0.0'
 PRPORT_DEFAULT=8585
 
 def main():

@@ -18,8 +18,8 @@ def main():
         version="Bitbake PR Service Core version %s, %%prog version %s" % (prserv.__version__, __version__),
         usage = "%prog < --start | --stop > [options]")
 
-    parser.add_option("-f", "--file", help="database filename(default: prserv.db)", action="store",
-                      dest="dbfile", type="string", default="prserv.db")
+    parser.add_option("-f", "--file", help="database filename(default: prserv.sqlite3)", action="store",
+                      dest="dbfile", type="string", default="prserv.sqlite3")
     parser.add_option("-l", "--log", help="log filename(default: prserv.log)", action="store",
                       dest="logfile", type="string", default="prserv.log")
     parser.add_option("--loglevel", help="logging level, i.e. CRITICAL, ERROR, WARNING, INFO, DEBUG",

@@ -37,8 +37,7 @@ def main():
     prserv.init_logger(os.path.abspath(options.logfile),options.loglevel)
 
     if options.start:
-        ret=prserv.serv.start_daemon(dbfile=options.dbfile, interface=(options.host, options.port),
-                      logfile=os.path.abspath(options.logfile))
+        ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile))
     elif options.stop:
         ret=prserv.serv.stop_daemon(options.host, options.port)
     else:

bitbake/lib/bb/cooker.py +3 -0

index 15243f2..a421f5f 100644
@@ -36,6 +36,7 @@ from functools import wraps
 from collections import defaultdict
 import bb, bb.exceptions
 from bb import utils, data, parse, event, cache, providers, taskdata, command, runqueue
+import prserv.serv
 
 logger      = logging.getLogger("BitBake")
 collectlog  = logging.getLogger("BitBake.Collection")

@@ -1290,9 +1291,11 @@ class BBCooker:
         # Empty the environment. The environment will be populated as
         # necessary from the data store.
         #bb.utils.empty_environment()
+        prserv.serv.auto_start(self.configuration.data)
         return
 
     def post_serve(self):
+        prserv.serv.auto_shutdown(self.configuration.data)
         bb.event.fire(CookerExit(), self.configuration.event_data)
 
     def shutdown(self):

bitbake/lib/prserv/db.py +7 -5

index f267dae..9d8e9db 100644
@@ -8,6 +8,8 @@ try:
 except ImportError:
     from pysqlite2 import dbapi2 as sqlite3
 
+logger = logging.getLogger("BitBake.PRserv")
+
 sqlversion = sqlite3.sqlite_version_info
 if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
     raise Exception("sqlite3 version 3.3.0 or later is required.")

@@ -55,7 +57,7 @@ class PRTable():
                            (version,pkgarch, checksum,version, pkgarch))
                 self.conn.commit()
             except sqlite3.IntegrityError as exc:
-                logging.error(str(exc))
+                logger.error(str(exc))
 
             data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,
                                (version, pkgarch, checksum))

@@ -83,7 +85,7 @@ class PRTable():
                                (version, pkgarch, checksum, version, pkgarch))
                 self.conn.commit()
             except sqlite3.IntegrityError as exc:
-                logging.error(str(exc))
+                logger.error(str(exc))
                 self.conn.rollback()
 
             data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,

@@ -115,7 +117,7 @@ class PRTable():
                            (version, pkgarch, checksum, value))
                 self.conn.commit()
             except sqlite3.IntegrityError as exc:
-                logging.error(str(exc))
+                logger.error(str(exc))
 
             data = self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,
                            (version, pkgarch, checksum))

@@ -140,7 +142,7 @@ class PRTable():
                                (value,version,pkgarch,checksum,value))
                 self.conn.commit()
             except sqlite3.IntegrityError as exc:
-                logging.error(str(exc))
+                logger.error(str(exc))
 
         data = self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=? AND value>=?;" % self.table,
                             (version,pkgarch,checksum,value))

@@ -241,5 +243,5 @@ class PRData(object):
     def __delitem__(self, tblname):
         if tblname in self._tables:
             del self._tables[tblname]
-        logging.info("drop table %s" % (tblname))
+        logger.info("drop table %s" % (tblname))
         self.connection.execute("DROP TABLE IF EXISTS %s;" % tblname)

bitbake/lib/prserv/serv.py +109 ~~-35~~

261 lines not on this page · raw message

Signature · 1 line

1.7.0.4