fixed toaster-eventreplay (#9585)
cover fixed toaster-eventreplay (#9585)
Hi,
This patchet contains changes made to toaster-eventreplay script to make it working with the recent codebase.
Most of the changes were caused by moving to Python 3. Old event serializing code in cooker and toaster-eventreplay doesn't work in Python 3.
There were also changes made to toasterui and buildinfohelper that made the script outdated, e.g. setEventMask was used in toasterui, but was not implemented in toaster-eventreplay.
Test instructions:
- generate event file using bitbake -u knotty -w events core-image-minimal
- start toaster: . ../bitbake/bin/toaster start
- check the the latest build:
../bitbake/lib/toaster/manage.py shell
In [1]: from orm.models import Build
In [2]: Build.objects.last()
Out[2]: <Build: 58 Project object core-image-minimal>
- replay event file: toaster-eventreplay events
- make sure latest build number is incremented:
../bitbake/lib/toaster/manage.py shell
In [1]: from orm.models import Build
In [2]: Build.objects.last()
Out[2]: <Build: 59 Project object core-image-minimal>
Note: This patchset depends on recent changes to cooker code:
http://lists.openembedded.org/pipermail/bitbake-devel/2016-June/007602.html
The following changes since commit 94dd70f042cd750a44ddfcef879a9547b072788f:
toaster-tests: define capabilities for latest Firefox driver (2016-06-21 13:48:30 +0300)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib ed/toaster/eventplay-9585
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/eventplay-9585
Ed Bartosh (6):
eventreplay: add MockConnection.getEventHandle method
eventprelay: implement setEventMask command
eventreplay: fix event loading code
eventreplay: replace MockConfigParameters with namedtuple
eventreplay: reorganize imports
eventreplay: rewrite the script
bitbake/bin/toaster-eventreplay | 220 ++++++++++++++++------------------------ 1 file changed, 86 insertions(+), 134 deletions(-)
--
Regards,
Ed
PATCH 1/6 eventreplay: add MockConnection.getEventHandle method
Fixed AttributeError: 'MockConnection' object has no attribute 'getEventHandle'
[YOCTO #9585]
--- bitbake/bin/toaster-eventreplay | 2 ++ 1 file changed, 2 insertions(+)
bitbake/bin/toaster-eventreplay +2 -0
@@ -86,6 +86,8 @@ class FileReadEventsServerConnection(): """ do not do anything """ pass + def getEventHandle(self): + pass class EventReader():
Signature · 1 line
2.1.4
PATCH 2/6 eventprelay: implement setEventMask command
In reply to Ed BartoshStored event mask list as self.eventmask for future use. Fixed Exception: Command setEventMask not implemented.
[YOCTO #9585]
--- bitbake/bin/toaster-eventreplay | 6 ++++++ 1 file changed, 6 insertions(+)
bitbake/bin/toaster-eventreplay +6 -0
@@ -53,6 +53,7 @@ class FileReadEventsServerConnection(): """ def __init__(self, sc): self._sc = sc + self.eventmask = [] def runCommand(self, commandArray): """ emulates running a command on the server; only read-only commands are accepted """
@@ -79,6 +80,11 @@ class FileReadEventsServerConnection(): except Exception as e: print(e) return (dump, None) + + elif command_name == 'setEventMask': + self.eventmask = commandArray[-1] + return True, None + else: raise Exception("Command %s not implemented" % commandArray[0])
Signature · 1 line
2.1.4
PATCH 3/6 eventreplay: fix event loading code
In reply to Ed BartoshEvent objects are represented by base64-encoded strings in the event file and can't be loaded by existing eventreplay code.
Fixed the code of loading events from file by decoding base64 strings into the binary form and loading them with pickle.load.
[YOCTO #9585]
--- bitbake/bin/toaster-eventreplay | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
bitbake/bin/toaster-eventreplay +3 -1
@@ -29,6 +29,7 @@ from __future__ import print_function import os import sys, logging +import codecs # mangle syspath to allow easy import of modules sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
@@ -117,7 +118,8 @@ class FileReadEventsServerConnection(): try: event_data = json.loads(line.strip()) event_class = _import_class(event_data['class']) - event_object = pickle.loads(json.loads(event_data['vars'])) + event_str = event_data['vars'].encode('utf-8') + event_object = pickle.loads(codecs.decode(event_str, 'base64')) except ValueError as e: print("Failed loading ", line) raise e
Signature · 1 line
2.1.4
PATCH 4/6 eventreplay: replace MockConfigParameters with namedtuple
In reply to Ed Bartoshclass MockConfigParameters has only one attribute and only __init__ method. Replacing it with namedtuple makes code less nested and more readable.
[YOCTO #9585]
--- bitbake/bin/toaster-eventreplay | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-)
bitbake/bin/toaster-eventreplay +2 -12
@@ -30,6 +30,7 @@ from __future__ import print_function import os import sys, logging import codecs +from collections import namedtuple # mangle syspath to allow easy import of modules sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
@@ -159,17 +160,6 @@ class FileReadEventsServerConnection(): self.events = FileReadEventsServerConnection.EventReader(self) - - - - -class MockConfigParameters(): - """ stand-in for cookerdata.ConfigParameters; as we don't really config a cooker, this - serves just to supply needed interfaces for the toaster ui to work """ - def __init__(self): - self.observe_only = True # we can only read files - - # run toaster ui on our mock bitbake class if __name__ == "__main__": if len(sys.argv) < 2:
@@ -178,7 +168,7 @@ if __name__ == "__main__": file_name = sys.argv[-1] mock_connection = FileReadEventsServerConnection(file_name) - configParams = MockConfigParameters() + configParams = namedtuple('ConfigParams', ['observe_only'])(True) # run the main program and set exit code to the returned value sys.exit(toasterui.main(mock_connection.connection, mock_connection.events, configParams))
Signature · 1 line
2.1.4
PATCH 5/6 eventreplay: reorganize imports
In reply to Ed BartoshCleaned up module imports:
- Removed unused imports
- Removed import of print_function
- Removed duplicated imports
- Splitted importing bb.lib to 2 lines
--- bitbake/bin/toaster-eventreplay | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
bitbake/bin/toaster-eventreplay +6 -11
@@ -26,24 +26,19 @@ # as a build eventlog, and the ToasterUI is used to process events in the file # and log data in the database -from __future__ import print_function -import os -import sys, logging +import sys +import json +import pickle import codecs + from collections import namedtuple # mangle syspath to allow easy import of modules -sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), - 'lib')) - +from os.path import join, dirname, abspath +sys.path.insert(0, join(dirname(dirname(abspath(__file__))), 'lib')) import bb.cooker from bb.ui import toasterui -import sys -import logging - -import json, pickle - class FileReadEventsServerConnection(): """ Emulates a connection to a bitbake server that feeds
Signature · 1 line
2.1.4
PATCH 6/6 eventreplay: rewrite the script
In reply to Ed BartoshRewritten toaster-eventreplay to make code working as expected, more compact and readable.
[YOCTO #9585]
--- bitbake/bin/toaster-eventreplay | 207 ++++++++++++++++------------------------ 1 file changed, 82 insertions(+), 125 deletions(-)
bitbake/bin/toaster-eventreplay +82 -125
@@ -21,11 +21,13 @@ # with this program; if not, write to the Free Software Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +""" +This command takes a filename as a single parameter. The filename is read +as a build eventlog, and the ToasterUI is used to process events in the file +and log data in the database +""" -# This command takes a filename as a single parameter. The filename is read -# as a build eventlog, and the ToasterUI is used to process events in the file -# and log data in the database - +import os import sys import json import pickle
@@ -40,130 +42,85 @@ sys.path.insert(0, join(dirname(dirname(abspath(__file__))), 'lib')) import bb.cooker from bb.ui import toasterui -class FileReadEventsServerConnection(): - """ Emulates a connection to a bitbake server that feeds - events coming actually read from a saved log file. - """ - - class MockConnection(): - """ fill-in for the proxy to the server. we just return generic data +class EventPlayer: + """Emulate a connection to a bitbake server.""" + + def __init__(self, eventfile, variables): + self.eventfile = eventfile + self.variables = variables + self.eventmask = [] + + def waitEvent(self, _timeout): + """Read event from the file.""" + line = self.eventfile.readline().strip() + if not line: + return + try: + event_str = json.loads(line)['vars'].encode('utf-8') + event = pickle.loads(codecs.decode(event_str, 'base64')) + event_name = "%s.%s" % (event.__module__, event.__class__.__name__) + if event_name not in self.eventmask: + return + return event + except ValueError as err: + print("Failed loading ", line) + raise err + + def runCommand(self, command_line): + """Emulate running a command on the server.""" + name = command_line[0] + + if name == "getVariable": + var_name = command_line[1] + variable = self.variables.get(var_name) + if variable: + return variable['v'], None + return None, "Missing variable %s" % var_name + + elif name == "getAllKeysWithFlags": + dump = {} + flaglist = command_line[1] + for key, val in self.variables.items(): + try: + if not key.startswith("__"): + dump[key] = { + 'v': val['v'], + 'history' : val['history'], + } + for flag in flaglist: + dump[key][flag] = val[flag] + except Exception as err: + print(err) + return (dump, None) + + elif name == 'setEventMask': + self.eventmask = command_line[-1] + return True, None + + else: + raise Exception("Command %s not implemented" % command_line[0]) + + def getEventHandle(self): """ - def __init__(self, sc): - self._sc = sc - self.eventmask = [] - - def runCommand(self, commandArray): - """ emulates running a command on the server; only read-only commands are accepted """ - command_name = commandArray[0] - - if command_name == "getVariable": - if commandArray[1] in self._sc._variables: - return (self._sc._variables[commandArray[1]]['v'], None) - return (None, "Missing variable") - - elif command_name == "getAllKeysWithFlags": - dump = {} - flaglist = commandArray[1] - for k in self._sc._variables.keys(): - try: - if not k.startswith("__"): - v = self._sc._variables[k]['v'] - dump[k] = { - 'v' : v , - 'history' : self._sc._variables[k]['history'], - } - for d in flaglist: - dump[k][d] = self._sc._variables[k][d] - except Exception as e: - print(e) - return (dump, None) - - elif command_name == 'setEventMask': - self.eventmask = commandArray[-1] - return True, None - - else: - raise Exception("Command %s not implemented" % commandArray[0]) - - def terminateServer(self): - """ do not do anything """ - pass - - def getEventHandle(self): - pass - - - class EventReader(): - def __init__(self, sc): - self._sc = sc - self.firstraise = 0 - - def _create_event(self, line): - def _import_class(name): - assert len(name) > 0 - assert "." in name, name - - components = name.strip().split(".") - modulename = ".".join(components[:-1]) - moduleklass = components[-1] - - module = __import__(modulename, fromlist=[str(moduleklass)]) - return getattr(module, moduleklass) - - # we build a toaster event out of current event log line - try: - event_data = json.loads(line.strip()) - event_class = _import_class(event_data['class']) - event_str = event_data['vars'].encode('utf-8') - event_object = pickle.loads(codecs.decode(event_str, 'base64')) - except ValueError as e: - print("Failed loading ", line) - raise e - - if not isinstance(event_object, event_class): - raise Exception("Error loading objects %s class %s ", event_object, event_class) - - return event_object - - def waitEvent(self, timeout): - - nextline = self._sc._eventfile.readline() - if len(nextline) == 0: - # the build data ended, while toasterui still waits for events. - # this happens when the server was abruptly stopped, so we simulate this - self.firstraise += 1 - if self.firstraise == 1: - raise KeyboardInterrupt() - else: - return None - else: - self._sc.lineno += 1 - return self._create_event(nextline) - - - def _readVariables(self, variableline): - self._variables = json.loads(variableline.strip())['allvariables'] - - - def __init__(self, file_name): - self.connection = FileReadEventsServerConnection.MockConnection(self) - self._eventfile = open(file_name, "r") - - # we expect to have the variable dump at the start of the file - self.lineno = 1 - self._readVariables(self._eventfile.readline()) - - self.events = FileReadEventsServerConnection.EventReader(self) + This method is called by toasterui. + The return value is passed to self.runCommand but not used there. + """ + pass + +def main(argv): + with open(argv[-1]) as eventfile: + # load variables from the first line + variables = json.loads(eventfile.readline().strip())['allvariables'] + + params = namedtuple('ConfigParams', ['observe_only'])(True) + player = EventPlayer(eventfile, variables) + + return toasterui.main(player, player, params) # run toaster ui on our mock bitbake class if __name__ == "__main__": - if len(sys.argv) < 2: - print("Usage: %s event.log " % sys.argv[0]) + if len(sys.argv) != 2: + print("Usage: %s <event file>" % os.path.basename(sys.argv[0])) sys.exit(1) - file_name = sys.argv[-1] - mock_connection = FileReadEventsServerConnection(file_name) - configParams = namedtuple('ConfigParams', ['observe_only'])(True) - - # run the main program and set exit code to the returned value - sys.exit(toasterui.main(mock_connection.connection, mock_connection.events, configParams)) + sys.exit(main(sys.argv))
Signature · 1 line
2.1.4
Submitted to bitbake-devel and added to toaster-next.
Thanks.
Elliot
Quoted 67 lines · Ed Bartosh
Hi,
This patchet contains changes made to toaster-eventreplay script to make it working
with the recent codebase.Most of the changes were caused by moving to Python 3. Old event serializing code
in cooker and toaster-eventreplay doesn't work in Python 3.There were also changes made to toasterui and buildinfohelper that made the script
outdated, e.g. setEventMask was used in toasterui, but was not implemented in toaster-eventreplay.Test instructions:
- generate event file using bitbake -u knotty -w events core-image-minimal
- start toaster: . ../bitbake/bin/toaster start
- check the the latest build:
../bitbake/lib/toaster/manage.pyshell
In [1]: from orm.models import Build
In [2]:Build.objects.last()
Out[2]: <Build: 58 Project object core-image-minimal>
- replay event file: toaster-eventreplay events
- make sure latest build number is incremented:
../bitbake/lib/toaster/manage.pyshell
In [1]: from orm.models import Build
In [2]:Build.objects.last()
Out[2]: <Build: 59 Project object core-image-minimal>Note: This patchset depends on recent changes to cooker code:
http://lists.openembedded.org/pipermail/bitbake-devel/2016-June/007602.htmlThe following changes since commit
94dd70f042cd750a44ddfcef879a9547b072788f:toaster-tests: define capabilities for latest Firefox driver (2016-06-21
13:48:30 +0300)are available in the git repository at:
git://git.yoctoproject.org/poky-contribed/toaster/eventplay-9585http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/eventplay-9585
Ed Bartosh (6):
eventreplay: add MockConnection.getEventHandle method
eventprelay: implement setEventMask command
eventreplay: fix event loading code
eventreplay: replace MockConfigParameters with namedtuple
eventreplay: reorganize imports
eventreplay: rewrite the scriptbitbake/bin/toaster-eventreplay | 220 ++++++++++++++++------------------------ 1 file changed, 86 insertions(+), 134 deletions(-)--
Regards,
Ed--
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster
Signature · 3 lines
Elliot Smith Software Engineer Intel Open Source Technology Centre