new PRService features in bitbake

PATCH Applied Lianhao Lu 2 participants 4 messages Jan 10, 2012 – Jan 11, 2012

lore

Share & tools

lore

Lianhao Lu

author

2012-01-10 06:24 UTC #0 lore raw

v2 cover new PRService features in bitbake

V2: rebase to latest code

NOTE: This set of patches added supporting for new features in bitbake. It should work together with another branch of patches in meta layers.
git://git.pokylinux.org/poky-contrib llu/PRService_oecore

It added PRService's export/import support in bitbake and also added a no history mode as the default mode for PRSerivce. In this mode, the AUTOPR value(which will be included in the finale PKGR) returned by the PRService will never decrement. It also added PACKAGE_ARCH into the index to the DB table, so the packages for different architcture will have different AUTOPR values.

The PRService can be started in 2 ways. If the user wants share the PRService for multiple buildings, it can start the PRService in the old way, by running "bitbake-prserv --start [options]", and set PRSERV_HOST and PRSERV_PORT in local.conf accordingly.

If the user only wants the PRService to be used by current build, a local service may be started automatically by setting PRSERV_HOST to "localhost" and PRSERV_PORT to "0". This automatically started local service will only be effective to the builds running in the same build directory.

The following changes since commit 468998cddbe1a803096c9b357e1b5daa3b7e8c2e:
Dongxiao Xu (1):
command.py: add parseConfigurationFiles API

are available in the git repository at:

git://git.pokylinux.org/poky-contrib llu/PRService_bitbake
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=llu/PRService_bitbake

Lianhao Lu (2):
bitbake/PRservice: Added no_hist mode and export/import.
bitbake: Automatically start local PR service.

 bitbake/bin/bitbake-prserv     |   26 +++---
 bitbake/lib/bb/cooker.py       |    3 +
 bitbake/lib/prserv/__init__.py |    5 +-
 bitbake/lib/prserv/db.py       |  213 +++++++++++++++++++++++++++++++++------
 bitbake/lib/prserv/serv.py     |  208 +++++++++++++++++++++++++++++----------
 5 files changed, 358 insertions(+), 97 deletions(-)
 mode change 100644 => 100755 bitbake/bin/bitbake-prserv

Lianhao Lu

author

2012-01-10 06:23 UTC #1 lore raw

PATCH v2 1/2 bitbake/PRservice: Added no_hist mode and export/import.

[YOCTO #1556]
1. Added the package_arch into the index to the DB table. Because the change in PACKAGE_ARCH will results in different checksum, and it is better to have seperate PR value domains for differnt PACKAGE_ARCH of the same pakcage.

2. Changed the PR service to operate in no history mode. In this mode, the for a given query tuple (version, pkgarch, checksum), the returned value will be the largest among all the values of the same (version, pkgarch). This means the PR value returned can NOT be decremented.

3. Added export function. For each (version, pkgarch) tuple, only the record with the maximum value will be exported.

4. Added import function. The record will only be imported if the imported value is larger than the value stored in the DB with the same (version, pkgarch, checksum) tuple.

Signed-off-by Lianhao Lu <[email protected]>

---
 bitbake/bin/bitbake-prserv     |   23 +++--
 bitbake/lib/prserv/__init__.py |    5 +-
 bitbake/lib/prserv/db.py       |  209 ++++++++++++++++++++++++++++++++++------
 bitbake/lib/prserv/serv.py     |  106 +++++++++++++-------
 4 files changed, 263 insertions(+), 80 deletions(-)
 mode change 100644 => 100755 bitbake/bin/bitbake-prserv

bitbake/bin/bitbake-prserv +13 ~~-10~~

old mode 100644
new mode 100755
index 14073ca..a7ab55f
@@ -16,31 +16,34 @@ PRPORT_DEFAULT=8585
 def main():
     parser = optparse.OptionParser(
         version="Bitbake PR Service Core version %s, %%prog version %s" % (prserv.__version__, __version__),
-        usage = "%prog [options]")
+        usage = "%prog < --start | --stop > [options]")
 
-    parser.add_option("-f", "--file", help="database filename(default prserv.db)", action="store",
+    parser.add_option("-f", "--file", help="database filename(default: prserv.db)", action="store",
                       dest="dbfile", type="string", default="prserv.db")
-    parser.add_option("-l", "--log", help="log filename(default prserv.log)", action="store",
+    parser.add_option("-l", "--log", help="log filename(default: prserv.log)", action="store",
                       dest="logfile", type="string", default="prserv.log")
     parser.add_option("--loglevel", help="logging level, i.e. CRITICAL, ERROR, WARNING, INFO, DEBUG",
-                      action = "store", type="string", dest="loglevel", default = "WARNING")
+                      action = "store", type="string", dest="loglevel", default = "INFO")
     parser.add_option("--start", help="start daemon",
-                      action="store_true", dest="start", default="True")
+                      action="store_true", dest="start")
     parser.add_option("--stop", help="stop daemon",
-                      action="store_false", dest="start")
+                      action="store_true", dest="stop")
     parser.add_option("--host", help="ip address to bind", action="store",
                       dest="host", type="string", default=PRHOST_DEFAULT)
-    parser.add_option("--port", help="port number(default 8585)", action="store",
+    parser.add_option("--port", help="port number(default: 8585)", action="store",
                       dest="port", type="int", default=PRPORT_DEFAULT)
 
     options, args = parser.parse_args(sys.argv)
-
     prserv.init_logger(os.path.abspath(options.logfile),options.loglevel)
 
     if options.start:
-        prserv.serv.start_daemon(options)
+        ret=prserv.serv.start_daemon(dbfile=options.dbfile, interface=(options.host, options.port),
+                      logfile=os.path.abspath(options.logfile))
+    elif options.stop:
+        ret=prserv.serv.stop_daemon(options.host, options.port)
     else:
-        prserv.serv.stop_daemon()
+        ret=parser.print_help()
+    return ret
 
 if __name__ == "__main__":
     try:

bitbake/lib/prserv/__init__.py +4 -1

index 2837e13..c27fffe 100644
@@ -7,5 +7,8 @@ def init_logger(logfile, loglevel):
     numeric_level = getattr(logging, loglevel.upper(), None)
     if not isinstance(numeric_level, int):
         raise ValueError('Invalid log level: %s' % loglevel)
-    logging.basicConfig(level=numeric_level, filename=logfile)
+    FORMAT = '%(asctime)-15s %(message)s'
+    logging.basicConfig(level=numeric_level, filename=logfile, format=FORMAT)
 
+class NotFoundError(StandardError):
+    pass
\ No newline at end of file

bitbake/lib/prserv/db.py +177 ~~-32~~

270 lines not on this page · raw message

bitbake/lib/prserv/serv.py +69 ~~-37~~

201 lines not on this page · raw message

Signature · 1 line

1.7.0.4

Lianhao Lu

author

2012-01-10 06:24 UTC #2 lore raw

PATCH v2 2/2 bitbake: Automatically start local PR service.

[YOCTO #1126]
A local PR service will be started and stopped automatically along with the bitbake invocation/ternimation.

This local PR service will be started only and if only when the PRSERV_HOST is set to 'localhost' and PRSERV_PORT is set to '0'.

When started, the sqlite3 database is stored at "${PERSISTEN_DIR}/prserv.sqlite3" or "${CACHE}/prserv.sqlite3".

Signed-off-by Lianhao Lu <[email protected]>

---
 bitbake/bin/bitbake-prserv |    9 +--
 bitbake/lib/bb/cooker.py   |    3 +
 bitbake/lib/prserv/db.py   |   12 ++--
 bitbake/lib/prserv/serv.py |  144 +++++++++++++++++++++++++++++++++-----------
 4 files changed, 123 insertions(+), 45 deletions(-)

bitbake/bin/bitbake-prserv +4 -5

index a7ab55f..a8d7acb 100755
@@ -10,7 +10,7 @@ import prserv.serv
 
 __version__="1.0.0"
 
-PRHOST_DEFAULT=''
+PRHOST_DEFAULT='0.0.0.0'
 PRPORT_DEFAULT=8585
 
 def main():

@@ -18,8 +18,8 @@ def main():
         version="Bitbake PR Service Core version %s, %%prog version %s" % (prserv.__version__, __version__),
         usage = "%prog < --start | --stop > [options]")
 
-    parser.add_option("-f", "--file", help="database filename(default: prserv.db)", action="store",
-                      dest="dbfile", type="string", default="prserv.db")
+    parser.add_option("-f", "--file", help="database filename(default: prserv.sqlite3)", action="store",
+                      dest="dbfile", type="string", default="prserv.sqlite3")
     parser.add_option("-l", "--log", help="log filename(default: prserv.log)", action="store",
                       dest="logfile", type="string", default="prserv.log")
     parser.add_option("--loglevel", help="logging level, i.e. CRITICAL, ERROR, WARNING, INFO, DEBUG",

@@ -37,8 +37,7 @@ def main():
     prserv.init_logger(os.path.abspath(options.logfile),options.loglevel)
 
     if options.start:
-        ret=prserv.serv.start_daemon(dbfile=options.dbfile, interface=(options.host, options.port),
-                      logfile=os.path.abspath(options.logfile))
+        ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile))
     elif options.stop:
         ret=prserv.serv.stop_daemon(options.host, options.port)
     else:

bitbake/lib/bb/cooker.py +3 -0

index 2032718..194046e 100644
@@ -36,6 +36,7 @@ from functools import wraps
 from collections import defaultdict
 import bb, bb.exceptions, bb.command
 from bb import utils, data, parse, event, cache, providers, taskdata, runqueue
+import prserv.serv
 
 logger      = logging.getLogger("BitBake")
 collectlog  = logging.getLogger("BitBake.Collection")

@@ -1311,9 +1312,11 @@ class BBCooker:
         # Empty the environment. The environment will be populated as
         # necessary from the data store.
         #bb.utils.empty_environment()
+        prserv.serv.auto_start(self.configuration.data)
         return
 
     def post_serve(self):
+        prserv.serv.auto_shutdown(self.configuration.data)
         bb.event.fire(CookerExit(), self.configuration.event_data)
 
     def shutdown(self):

bitbake/lib/prserv/db.py +7 -5

index f267dae..9d8e9db 100644
@@ -8,6 +8,8 @@ try:
 except ImportError:
     from pysqlite2 import dbapi2 as sqlite3
 
+logger = logging.getLogger("BitBake.PRserv")
+
 sqlversion = sqlite3.sqlite_version_info
 if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
     raise Exception("sqlite3 version 3.3.0 or later is required.")

@@ -55,7 +57,7 @@ class PRTable():
                            (version,pkgarch, checksum,version, pkgarch))
                 self.conn.commit()
             except sqlite3.IntegrityError as exc:
-                logging.error(str(exc))
+                logger.error(str(exc))
 
             data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,
                                (version, pkgarch, checksum))

@@ -83,7 +85,7 @@ class PRTable():
                                (version, pkgarch, checksum, version, pkgarch))
                 self.conn.commit()
             except sqlite3.IntegrityError as exc:
-                logging.error(str(exc))
+                logger.error(str(exc))
                 self.conn.rollback()
 
             data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,

@@ -115,7 +117,7 @@ class PRTable():
                            (version, pkgarch, checksum, value))
                 self.conn.commit()
             except sqlite3.IntegrityError as exc:
-                logging.error(str(exc))
+                logger.error(str(exc))
 
             data = self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,
                            (version, pkgarch, checksum))

@@ -140,7 +142,7 @@ class PRTable():
                                (value,version,pkgarch,checksum,value))
                 self.conn.commit()
             except sqlite3.IntegrityError as exc:
-                logging.error(str(exc))
+                logger.error(str(exc))
 
         data = self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=? AND value>=?;" % self.table,
                             (version,pkgarch,checksum,value))

@@ -241,5 +243,5 @@ class PRData(object):
     def __delitem__(self, tblname):
         if tblname in self._tables:
             del self._tables[tblname]
-        logging.info("drop table %s" % (tblname))
+        logger.info("drop table %s" % (tblname))
         self.connection.execute("DROP TABLE IF EXISTS %s;" % tblname)

bitbake/lib/prserv/serv.py +109 ~~-35~~

261 lines not on this page · raw message

Signature · 1 line

1.7.0.4

Richard Purdie

Applied

2012-01-11 10:45 UTC #3 lore raw

In reply to Lianhao Lu

Quoted 27 lines · Lianhao Lu

V2: rebase to latest code
NOTE: This set of patches added supporting for new features in bitbake. It should work together with another branch of patches in meta layers.
git://git.pokylinux.org/poky-contrib llu/PRService_oecore
It added PRService's export/import support in bitbake and also added a no history mode as the default mode for PRSerivce. In this mode, the AUTOPR value(which will be included in the finale PKGR) returned by the PRService will never decrement. It also added PACKAGE_ARCH into the index to the DB table, so the packages for different architcture will have different AUTOPR values.
The PRService can be started in 2 ways. If the user wants share the PRService for multiple buildings, it can start the PRService in the old way, by running "bitbake-prserv --start [options]", and set PRSERV_HOST and PRSERV_PORT in local.conf accordingly.
If the user only wants the PRService to be used by current build, a local service may be started automatically by setting PRSERV_HOST to "localhost" and PRSERV_PORT to "0". This automatically started local service will only be effective to the builds running in the same build directory.
The following changes since commit 468998cddbe1a803096c9b357e1b5daa3b7e8c2e:
Dongxiao Xu (1):
command.py: add parseConfigurationFiles API

Quoted 8 lines

are available in the git repository at:
git://git.pokylinux.org/poky-contrib llu/PRService_bitbake
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=llu/PRService_bitbake
Lianhao Lu (2):
bitbake/PRservice: Added no_hist mode and export/import.
bitbake: Automatically start local PR service.

Merged to master, thanks.

Richard