host_dmesg_logger: dmesg diff on BuildStarted and BuildCompleted
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH 1/1 host_dmesg_logger: dmesg diff on BuildStarted and BuildCompleted
This class is used to capture the host dmesg output when BuildStarted and BuildCompleted. Then computes a diff and prints the result.
Fixes [YOCTO #15557]
--- meta/classes-global/host_dmesg_logger.bbclass | 86 +++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 meta/classes-global/host_dmesg_logger.bbclass
meta/classes-global/host_dmesg_logger.bbclass new file +86 -0
@@ -0,0 +1,86 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +# This class captures the host's dmesg output at the start and completion of a build. +# It stores these outputs in a variable and upon build completion, computes a diff between +# the two logs to detect any changes that occurred during the build process. It prints the diff +# using bb.warn(). The user needs to have privileges to run dmesg. + +def append_log(msg, d): + """ + Appends a log message to variable '_dmesg_log' + """ + current_log = d.getVar('_dmesg_log') or '' + current_log += msg + '\n' + d.setVar('_dmesg_log', current_log) + +def capture_dmesg(): + """ + Returns the current output of dmesg command + """ + import subprocess + + try: + result = subprocess.run(['dmesg'], capture_output=True, text=True, check=False) + if result.returncode != 0: + return f"Error running dmesg: {result.stderr.strip()}" + return result.stdout + except Exception as e: + return f"Exception running dmesg: {str(e)}" + + +addhandler hostdmesglogger_eventhandler +python hostdmesglogger_eventhandler() { + import difflib + + start_marker = "=== BuildStarted dmesg ===" + end_marker = "=== BuildCompleted dmesg ===" + diff_marker = "=== dmesg diff ===" + + # execute dmesg when BuildStarted event is fired + if isinstance(e, bb.event.BuildStarted): + dmesg_output = capture_dmesg() + if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"): + bb.warn(dmesg_output) + else: + append_log(start_marker, d) + append_log(dmesg_output, d) + + # execute dmesg when BuildCompleted event is fired + if isinstance(e, bb.event.BuildCompleted): + dmesg_output = capture_dmesg() + if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"): + bb.warn(dmesg_output) + else: + append_log(end_marker, d) + append_log(dmesg_output, d) + + content = d.getVar('_dmesg_log') or '' + if start_marker in content and end_marker in content: + start_dmesg = content.split(start_marker)[1].split(end_marker)[0] + end_dmesg = content.split(end_marker)[1] + + start_lines = start_dmesg.strip().splitlines() + end_lines = end_dmesg.strip().splitlines() + + # generating diff between BuildStarted and BuildCompleted dmesg outputs + diff = list(difflib.unified_diff( + start_lines, end_lines, + fromfile='dmesg_start', + tofile='dmesg_end', + lineterm='' + )) + + append_log(diff_marker, d) + if diff: + for line in diff: + bb.warn(line) + else: + bb.warn("No differences in dmesg output.") + else: + bb.warn("Could not find both dmesg sections for diff.") +} +hostdmesglogger_eventhandler[eventmask] = "bb.event.BuildStarted bb.event.BuildCompleted"
Signature · 1 line
2.34.1
Quoted 3 lines · Christos Gavros
This class is used to capture the host dmesg output when BuildStarted and BuildCompleted. Then computes a diff and prints the result.
I'd like to see a few words about when this is expected to be useful and what the drawbacks are.
In the bug Alex said:
For debugging purposes, it would be nice to record the host dmesg at the start of bitbake and the end of the build to be able to do a diff. This would probably uncover interesting failures that are currently hidden/ignored by the underlying build system. We know there are gcc/ld segfaults and sometime qemu segfaults.
The obvious downside (to most people) is that it will capture unrelated dmesg output should the build be running on a shared machine and that some distro limit dmesg usage to root/sudo.
Also please mention usage:
To enable dmesg logging add the following to your local.conf:
INHERIT += "host_dmesg_logger"
Now that I type that, it's really a host_dmesg_diff_logger but maybe we don't care.
I didn't actually try it yet ... but LGTM other than the comments above and some whitespace issue mentioned below.
Thanks for the patch Christos!
../Randy
Quoted 81 lines
Fixes [YOCTO #15557]
CC Yoann Congal<[email protected]>CC Randy MacLeod<[email protected]>CC Alexandre Belloni<[email protected]>Signed-off-by Christos Gavros<[email protected]>--- meta/classes-global/host_dmesg_logger.bbclass | 86 +++++++++++++++++++ 1 file changed, 86 insertions(+)create mode 100644 meta/classes-global/host_dmesg_logger.bbclass
meta/classes-global/host_dmesg_logger.bbclass new file +64
-0@@ -0,0 +1,86 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +# This class captures the host's dmesg output at the start and completion of a build. +# It stores these outputs in a variable and upon build completion, computes a diff between +# the two logs to detect any changes that occurred during the build process. It prints the diff +# using bb.warn(). The user needs to have privileges to run dmesg. + +def append_log(msg, d): + """ + Appends a log message to variable '_dmesg_log' + """ + current_log = d.getVar('_dmesg_log') or '' + current_log += msg + '\n' + d.setVar('_dmesg_log', current_log) + +def capture_dmesg(): + """ + Returns the current output of dmesg command + """ + import subprocess + + try: + result = subprocess.run(['dmesg'], capture_output=True, text=True, check=False) + if result.returncode != 0: + return f"Error running dmesg: {result.stderr.strip()}" + return result.stdout + except Exception as e: + return f"Exception running dmesg: {str(e)}" + + +addhandler hostdmesglogger_eventhandler +python hostdmesglogger_eventhandler() { + import difflib + + start_marker = "=== BuildStarted dmesg ===" + end_marker = "=== BuildCompleted dmesg ===" + diff_marker = "=== dmesg diff ===" + + # execute dmesg when BuildStarted event is fired + if isinstance(e, bb.event.BuildStarted): + dmesg_output = capture_dmesg() + if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"): + bb.warn(dmesg_output) + else: + append_log(start_marker, d) + append_log(dmesg_output, d) + + # execute dmesg when BuildCompleted event is fired + if isinstance(e, bb.event.BuildCompleted): + dmesg_output = capture_dmesg() + if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"): + bb.warn(dmesg_output) + else: + append_log(end_marker, d) + append_log(dmesg_output, d) + + content = d.getVar('_dmesg_log') or '' + if start_marker in content and end_marker in content: + start_dmesg = content.split(start_marker)[1].split(end_marker)[0] + end_dmesg = content.split(end_marker)[1]
meta/classes-global/host_dmesg_logger.bbclass line 74
Quoted 10 lines
meta/classes-global/host_dmesg_logger.bbclass new file +10
-0+ + start_lines = start_dmesg.strip().splitlines() + end_lines = end_dmesg.strip().splitlines() + + # generating diff between BuildStarted and BuildCompleted dmesg outputs + diff = list(difflib.unified_diff( + start_lines, end_lines, + fromfile='dmesg_start', + tofile='dmesg_end', + lineterm=''
Add spaces around "=" ?
Quoted 12 lines
meta/classes-global/host_dmesg_logger.bbclass new file +12
-0+ )) + + append_log(diff_marker, d) + if diff: + for line in diff: + bb.warn(line) + else: + bb.warn("No differences in dmesg output.") + else: + bb.warn("Could not find both dmesg sections for diff.") +} +hostdmesglogger_eventhandler[eventmask] = "bb.event.BuildStarted bb.event.BuildCompleted"
Signature · 2 lines
# Randy MacLeod # Wind River Linux
Quoted 8 lines · Randy MacLeod
Quoted 3 lines · Christos Gavros
This class is used to capture the host dmesg output when BuildStarted and BuildCompleted. Then computes a diff and prints the result.
I'd like to see a few words about when this is expected to be useful and what the drawbacks are.
Quoted 9 lines
In the bug Alex said:
For debugging purposes, it would be nice to record the host dmesg at the start of bitbake and the end of the build to be able to do a diff. This would probably uncover interesting failures that are currently hidden/ignored by the underlying build system. We know there are gcc/ld segfaults and sometime qemu segfaults.
The obvious downside (to most people) is that it will capture unrelated dmesg output should the build be running on a shared machine and that some distro limit dmesg usage to root/sudo.
i will extend the description of the class and send v2.
Quoted 8 lines
Also please mention usage:
To enable dmesg logging add the following to your local.conf:
INHERIT += "
host_dmesg_logger"
i will extend the description of the class and send v2.
Quoted 3 lines
Now that I type that, it's really a
host_dmesg_diff_loggerbut maybe we don't care.
I kept the name that Richard suggested in the mail.
Quoted 6 lines
I didn't actually try it yet ... but LGTM other than the comments above and some whitespace issue mentioned below.
Thanks for the patch Christos!
I will fix them in v2
Quoted 113 lines
../Randy
Quoted 81 lines
Fixes [YOCTO #15557]
CC Yoann Congal<[email protected]>CC Randy MacLeod<[email protected]>CC Alexandre Belloni<[email protected]>Signed-off-by Christos Gavros<[email protected]>--- meta/classes-global/host_dmesg_logger.bbclass | 86 +++++++++++++++++++ 1 file changed, 86 insertions(+)create mode 100644 meta/classes-global/host_dmesg_logger.bbclass
meta/classes-global/host_dmesg_logger.bbclass new file +64
-0@@ -0,0 +1,86 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +# This class captures the host's dmesg output at the start and completion of a build. +# It stores these outputs in a variable and upon build completion, computes a diff between +# the two logs to detect any changes that occurred during the build process. It prints the diff +# using bb.warn(). The user needs to have privileges to run dmesg. + +def append_log(msg, d): + """ + Appends a log message to variable '_dmesg_log' + """ + current_log = d.getVar('_dmesg_log') or '' + current_log += msg + '\n' + d.setVar('_dmesg_log', current_log) + +def capture_dmesg(): + """ + Returns the current output of dmesg command + """ + import subprocess + + try: + result = subprocess.run(['dmesg'], capture_output=True, text=True, check=False) + if result.returncode != 0: + return f"Error running dmesg: {result.stderr.strip()}" + return result.stdout + except Exception as e: + return f"Exception running dmesg: {str(e)}" + + +addhandler hostdmesglogger_eventhandler +python hostdmesglogger_eventhandler() { + import difflib + + start_marker = "=== BuildStarted dmesg ===" + end_marker = "=== BuildCompleted dmesg ===" + diff_marker = "=== dmesg diff ===" + + # execute dmesg when BuildStarted event is fired + if isinstance(e, bb.event.BuildStarted): + dmesg_output = capture_dmesg() + if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"): + bb.warn(dmesg_output) + else: + append_log(start_marker, d) + append_log(dmesg_output, d) + + # execute dmesg when BuildCompleted event is fired + if isinstance(e, bb.event.BuildCompleted): + dmesg_output = capture_dmesg() + if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"): + bb.warn(dmesg_output) + else: + append_log(end_marker, d) + append_log(dmesg_output, d) + + content = d.getVar('_dmesg_log') or '' + if start_marker in content and end_marker in content: + start_dmesg = content.split(start_marker)[1].split(end_marker)[0] + end_dmesg = content.split(end_marker)[1]meta/classes-global/host_dmesg_logger.bbclass line 74
Quoted 10 lines
meta/classes-global/host_dmesg_logger.bbclass new file +10
-0+ + start_lines = start_dmesg.strip().splitlines() + end_lines = end_dmesg.strip().splitlines() + + # generating diff between BuildStarted and BuildCompleted dmesg outputs + diff = list(difflib.unified_diff( + start_lines, end_lines, + fromfile='dmesg_start', + tofile='dmesg_end', + lineterm=''Add spaces around "=" ?
Quoted 12 lines
meta/classes-global/host_dmesg_logger.bbclass new file +12
-0+ )) + + append_log(diff_marker, d) + if diff: + for line in diff: + bb.warn(line) + else: + bb.warn("No differences in dmesg output.") + else: + bb.warn("Could not find both dmesg sections for diff.") +} +hostdmesglogger_eventhandler[eventmask] = "bb.event.BuildStarted bb.event.BuildCompleted"Signature · 2 lines
# Randy MacLeod # Wind River Linux