buildstats.bbclass: add functionality to collect build system stats
PATCH v2 1/1 buildstats.bbclass: add functionality to collect build system stats
There are a number of timeout and hang defects where it would be useful to collect statistics about what is running on a build host when that condition occurs.
This adds functionality to collect build system stats on a regular interval and/or on task failure. Both features are disabled by default.
To enable logging on a regular interval, set:BB_HEARTBEAT_EVENT = "<interval>" BB_LOG_HOST_STAT_ON_INTERVAL = <boolean>
Logs are stored in ${BUILDSTATS_BASE}/<build_name>/host_stats
To enable logging on a task failure, set:BB_LOG_HOST_STAT_ON_FAILURE = "<boolean>"
Logs are stored in ${BUILDSTATS_BASE}/<build_name>/build_stats
The list of commands, along with the desired options, need to be specified in the BB_LOG_HOST_STAT_CMDS variable delimited by ; as such:BB_LOG_HOST_STAT_CMDS = "command1 ; command2 ;... ;"
--- meta/classes/buildstats.bbclass | 40 ++++++++++++++++++++++++++++++--- 1 file changed, 37 insertions(+), 3 deletions(-)
meta/classes/buildstats.bbclass +37 -3
@@ -104,14 +104,46 @@ def write_task_data(status, logfile, e, d): f.write("Status: FAILED \n") f.write("Ended: %0.2f \n" % e.time) +def write_host_data(logfile, e, d): + import subprocess, os, datetime + cmds = d.getVar('BB_LOG_HOST_STAT_CMDS') + if cmds is None: + d.setVar("BB_LOG_HOST_STAT_ON_INTERVAL", "0") + d.setVar("BB_LOG_HOST_STAT_ON_FAILURE", "0") + bb.warn("buildstats: Collecting host data failed. Set BB_LOG_HOST_STAT_CMDS=\"command1 ; command2 ; ... \" in conf\/local.conf\n") + return + path = d.getVar("PATH") + opath = d.getVar("BB_ORIGENV", False).getVar("PATH") + ospath = os.environ['PATH'] + os.environ['PATH'] = path + ":" + opath + ":" + ospath + with open(logfile, "a") as f: + f.write("Event Time: %f\nDate: %s\n" % (e.time, datetime.datetime.now())) + for cmd in cmds.split(";"): + if len(cmd) == 0: + continue + try: + output = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT, timeout=1).decode('utf-8') + except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError) as err: + output = "Error running command: %s\n%s\n" % (cmd, err) + f.write("%s\n%s\n" % (cmd, output)) + os.environ['PATH'] = ospath + python run_buildstats () { import bb.build import bb.event import time, subprocess, platform bn = d.getVar('BUILDNAME') - bsdir = os.path.join(d.getVar('BUILDSTATS_BASE'), bn) - taskdir = os.path.join(bsdir, d.getVar('PF')) + ######################################################################## + # bitbake fires HeartbeatEvent even before a build has been + # triggered, causing BUILDNAME to be None + ######################################################################## + if bn is not None: + bsdir = os.path.join(d.getVar('BUILDSTATS_BASE'), bn) + taskdir = os.path.join(bsdir, d.getVar('PF')) + if isinstance(e, bb.event.HeartbeatEvent) and bb.utils.to_boolean(d.getVar("BB_LOG_HOST_STAT_ON_INTERVAL")): + bb.utils.mkdirhier(bsdir) + write_host_data(os.path.join(bsdir, "host_stats"), e, d) if isinstance(e, bb.event.BuildStarted): ########################################################################
@@ -186,10 +218,12 @@ python run_buildstats () { build_status = os.path.join(bsdir, "build_stats") with open(build_status, "a") as f: f.write(d.expand("Failed at: ${PF} at task: %s \n" % e.task)) + if bb.utils.to_boolean(d.getVar("BB_LOG_HOST_STAT_ON_FAILURE")): + write_host_data(build_status, e, d) } addhandler run_buildstats -run_buildstats[eventmask] = "bb.event.BuildStarted bb.event.BuildCompleted bb.build.TaskStarted bb.build.TaskSucceeded bb.build.TaskFailed" +run_buildstats[eventmask] = "bb.event.BuildStarted bb.event.BuildCompleted bb.event.HeartbeatEvent bb.build.TaskStarted bb.build.TaskSucceeded bb.build.TaskFailed" python runqueue_stats () { import buildstats
Signature · 1 line
2.29.2
Hi,
Quoted 13 lines · Sakib Sajal
There are a number of timeout and hang defects where it would be useful to collect statistics about what is running on a build host when that condition occurs.
This adds functionality to collect build system stats on a regular interval and/or on task failure. Both features are disabled by default.
To enable logging on a regular interval, set:
BB_HEARTBEAT_EVENT= "<interval>"BB_LOG_HOST_STAT_ON_INTERVAL= <boolean>
Logs are stored in ${BUILDSTATS_BASE}/<build_name>/host_stats
Quoted 8 lines
To enable logging on a task failure, set:
BB_LOG_HOST_STAT_ON_FAILURE= "<boolean>"
Logs are stored in ${BUILDSTATS_BASE}/<build_name>/build_statsThe list of commands, along with the desired options, need to be specified in the
BB_LOG_HOST_STAT_CMDSvariable delimited by ; as such:BB_LOG_HOST_STAT_CMDS= "command1 ; command2 ;... ;"
I can understand why and have been debugging crashing and hanging build machines, but I would not have found this change useful. Do you have more concrete examples how this could be used?
Instead, I found that normal Linux server admin practices were best:
* collect build machine kernel, journald and syslogs to remote host, e.g. rsyslog
* monitor CPU, memory, IO, network etc performance, also to a remote host, e.g. pcp.io tooling or collectd
* collect bitbake build logs with system timestamps to remote host, e.g. don't trust jenkins and its timestamps
With those, I have been able to find problems in Linux kernels, bugs
in VMWare cloud storage stack triggering IO hangs, stalls and eventually kernel crashes, broken HW like memory. And of course basic things like full disks, full /tmp, kernel oom killer kicking in when build slaves ran out of RAM during bitbake build which results in either build machine changes or tuning of parallel build flags to account also physical RAM.
Wihtout full remote logging infrastructure, I could not have solved anything. Running individual commands is not enough when only full kernel dmesg of affected machine can tell that IO stack has a hang or an Oops or that disk had been mounted read-only due to errors.
Cheers,
-Mikko