Skip to contentSkip to conversation

host_dmesg_logger: dmesg diff on BuildStarted and BuildCompleted

PATCH Superseded Christos Gavros 2 participants 3 messages Jun 10, 2025 – Jun 23, 2025 1 message not in the archive

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

PATCH 1/1 host_dmesg_logger: dmesg diff on BuildStarted and BuildCompleted

This class is used to capture the host dmesg output when BuildStarted and BuildCompleted. Then computes a diff and prints the result.
Fixes [YOCTO #15557]

CC Yoann Congal <[email protected]>
CC Randy MacLeod <[email protected]>
CC Alexandre Belloni <[email protected]>
---
 meta/classes-global/host_dmesg_logger.bbclass | 86 +++++++++++++++++++
 1 file changed, 86 insertions(+)
 create mode 100644 meta/classes-global/host_dmesg_logger.bbclass
meta/classes-global/host_dmesg_logger.bbclass new file +86 -0
new file mode 100644
index 0000000000..0caf5f0612
@@ -0,0 +1,86 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+# This class captures the host's dmesg output at the start and completion of a build.
+# It stores these outputs in a variable and upon build completion, computes a diff between
+# the two logs to detect any changes that occurred during the build process. It prints the diff
+# using bb.warn(). The user needs to have privileges to run dmesg.
+
+def append_log(msg, d):
+    """
+    Appends a log message to variable '_dmesg_log'
+    """
+    current_log = d.getVar('_dmesg_log') or ''
+    current_log += msg + '\n'
+    d.setVar('_dmesg_log', current_log)
+
+def capture_dmesg():
+    """
+    Returns the current output of dmesg command
+    """
+    import subprocess
+
+    try:
+        result = subprocess.run(['dmesg'], capture_output=True, text=True, check=False)
+        if result.returncode != 0:
+            return f"Error running dmesg: {result.stderr.strip()}"
+        return result.stdout
+    except Exception as e:
+        return f"Exception running dmesg: {str(e)}"
+
+
+addhandler hostdmesglogger_eventhandler
+python hostdmesglogger_eventhandler() {
+    import difflib
+
+    start_marker = "=== BuildStarted dmesg ==="
+    end_marker = "=== BuildCompleted dmesg ==="
+    diff_marker = "=== dmesg diff ==="
+
+    # execute dmesg when BuildStarted event is fired
+    if isinstance(e, bb.event.BuildStarted):
+        dmesg_output = capture_dmesg()
+        if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"):
+            bb.warn(dmesg_output)
+        else:
+            append_log(start_marker, d)
+            append_log(dmesg_output, d)
+
+    # execute dmesg when BuildCompleted event is fired
+    if isinstance(e, bb.event.BuildCompleted):
+        dmesg_output = capture_dmesg()
+        if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"):
+            bb.warn(dmesg_output)
+        else:
+            append_log(end_marker, d)
+            append_log(dmesg_output, d)
+
+        content = d.getVar('_dmesg_log') or ''
+        if start_marker in content and end_marker in content:
+            start_dmesg = content.split(start_marker)[1].split(end_marker)[0]
+            end_dmesg = content.split(end_marker)[1]
+
+            start_lines = start_dmesg.strip().splitlines()
+            end_lines = end_dmesg.strip().splitlines()
+
+            # generating diff between BuildStarted and BuildCompleted dmesg outputs
+            diff = list(difflib.unified_diff(
+                start_lines, end_lines,
+                fromfile='dmesg_start',
+                tofile='dmesg_end',
+                lineterm=''
+            ))
+
+            append_log(diff_marker, d)
+            if diff:
+                for line in diff:
+                    bb.warn(line)
+            else:
+                bb.warn("No differences in dmesg output.")
+        else:
+            bb.warn("Could not find both dmesg sections for diff.")
+}
+hostdmesglogger_eventhandler[eventmask] = "bb.event.BuildStarted bb.event.BuildCompleted"
Signature · 1 line
2.34.1
#2 lore raw
Quoted 3 lines · Christos Gavros

This class is used to capture the host dmesg output when BuildStarted and BuildCompleted. Then computes a diff and prints the result.

I'd like to see a few words about when this is expected to be useful and what the drawbacks are.

In the bug Alex said:

   For debugging purposes, it would be nice to record the host dmesg at the start of bitbake and the end of the build to be able to do a diff. This would probably uncover interesting failures that are currently hidden/ignored by the underlying build system. We know there are gcc/ld segfaults and sometime qemu segfaults.

The obvious downside (to most people) is that it will capture unrelated dmesg output should the build be running on a shared machine and that some distro limit dmesg usage to root/sudo.

Also please mention usage:

To enable dmesg logging add the following to your local.conf:

INHERIT += "host_dmesg_logger"

Now that I type that, it's really a host_dmesg_diff_logger but maybe we don't care.

I didn't actually try it yet ... but LGTM other than the comments above and some whitespace issue mentioned below.

Thanks for the patch Christos!

../Randy

Quoted 81 lines

Fixes [YOCTO #15557]

CC Yoann Congal<[email protected]>
CC Randy MacLeod<[email protected]>
CC Alexandre Belloni<[email protected]>
---
  meta/classes-global/host_dmesg_logger.bbclass | 86 +++++++++++++++++++
  1 file changed, 86 insertions(+)

create mode 100644 meta/classes-global/host_dmesg_logger.bbclass

meta/classes-global/host_dmesg_logger.bbclass new file +64 -0
new file mode 100644
index 0000000000..0caf5f0612
@@ -0,0 +1,86 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+# This class captures the host's dmesg output at the start and completion of a build.
+# It stores these outputs in a variable and upon build completion, computes a diff between
+# the two logs to detect any changes that occurred during the build process. It prints the diff
+# using bb.warn(). The user needs to have privileges to run dmesg.
+
+def append_log(msg, d):
+    """
+    Appends a log message to variable '_dmesg_log'
+    """
+    current_log = d.getVar('_dmesg_log') or ''
+    current_log += msg + '\n'
+    d.setVar('_dmesg_log', current_log)
+
+def capture_dmesg():
+    """
+    Returns the current output of dmesg command
+    """
+    import subprocess
+
+    try:
+        result = subprocess.run(['dmesg'], capture_output=True, text=True, check=False)
+        if result.returncode != 0:
+            return f"Error running dmesg: {result.stderr.strip()}"
+        return result.stdout
+    except Exception as e:
+        return f"Exception running dmesg: {str(e)}"
+
+
+addhandler hostdmesglogger_eventhandler
+python hostdmesglogger_eventhandler() {
+    import difflib
+
+    start_marker = "=== BuildStarted dmesg ==="
+    end_marker = "=== BuildCompleted dmesg ==="
+    diff_marker = "=== dmesg diff ==="
+
+    # execute dmesg when BuildStarted event is fired
+    if isinstance(e, bb.event.BuildStarted):
+        dmesg_output = capture_dmesg()
+        if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"):
+            bb.warn(dmesg_output)
+        else:
+            append_log(start_marker, d)
+            append_log(dmesg_output, d)
+
+    # execute dmesg when BuildCompleted event is fired
+    if isinstance(e, bb.event.BuildCompleted):
+        dmesg_output = capture_dmesg()
+        if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"):
+            bb.warn(dmesg_output)
+        else:
+            append_log(end_marker, d)
+            append_log(dmesg_output, d)
+
+        content = d.getVar('_dmesg_log') or ''
+        if start_marker in content and end_marker in content:
+            start_dmesg = content.split(start_marker)[1].split(end_marker)[0]
+            end_dmesg = content.split(end_marker)[1]

meta/classes-global/host_dmesg_logger.bbclass line 74

Quoted 10 lines
meta/classes-global/host_dmesg_logger.bbclass new file +10 -0
+
+            start_lines = start_dmesg.strip().splitlines()
+            end_lines = end_dmesg.strip().splitlines()
+
+            # generating diff between BuildStarted and BuildCompleted dmesg outputs
+            diff = list(difflib.unified_diff(
+                start_lines, end_lines,
+                fromfile='dmesg_start',
+                tofile='dmesg_end',
+                lineterm=''

Add spaces around "=" ?

Quoted 12 lines
meta/classes-global/host_dmesg_logger.bbclass new file +12 -0
+            ))
+
+            append_log(diff_marker, d)
+            if diff:
+                for line in diff:
+                    bb.warn(line)
+            else:
+                bb.warn("No differences in dmesg output.")
+        else:
+            bb.warn("Could not find both dmesg sections for diff.")
+}
+hostdmesglogger_eventhandler[eventmask] = "bb.event.BuildStarted bb.event.BuildCompleted"
Signature · 2 lines
# Randy MacLeod
# Wind River Linux
#3 lore raw
Quoted 8 lines · Randy MacLeod
Quoted 3 lines · Christos Gavros

This class is used to capture the host dmesg output when BuildStarted and BuildCompleted. Then computes a diff and prints the result.

I'd like to see a few words about when this is expected to be useful and what the drawbacks are.

Quoted 9 lines

In the bug Alex said:

   For debugging purposes, it would be nice to record the host dmesg at the start of bitbake and the end of the build to be able to do a diff. This would probably uncover interesting failures that are currently hidden/ignored by the underlying build system. We know there are gcc/ld segfaults and sometime qemu segfaults.

The obvious downside (to most people) is that it will capture unrelated dmesg output should the build be running on a shared machine and that some distro limit dmesg usage to root/sudo.

i will extend the description of the class and send v2.

Quoted 8 lines

Also please mention usage:

To enable dmesg logging add the following to your local.conf:

INHERIT += "host_dmesg_logger"

i will extend the description of the class and send v2.

Quoted 3 lines

Now that I type that, it's really a host_dmesg_diff_logger but maybe we don't care.

I kept the name that Richard suggested in the mail.

Quoted 6 lines

I didn't actually try it yet ... but LGTM other than the comments above and some whitespace issue mentioned below.

Thanks for the patch Christos!

I will fix them in v2

Quoted 113 lines

../Randy

Quoted 81 lines

Fixes [YOCTO #15557]

CC Yoann Congal<[email protected]>
CC Randy MacLeod<[email protected]>
CC Alexandre Belloni<[email protected]>
---
  meta/classes-global/host_dmesg_logger.bbclass | 86 +++++++++++++++++++
  1 file changed, 86 insertions(+)

create mode 100644 meta/classes-global/host_dmesg_logger.bbclass

meta/classes-global/host_dmesg_logger.bbclass new file +64 -0
new file mode 100644
index 0000000000..0caf5f0612
@@ -0,0 +1,86 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+# This class captures the host's dmesg output at the start and completion of a build.
+# It stores these outputs in a variable and upon build completion, computes a diff between
+# the two logs to detect any changes that occurred during the build process. It prints the diff
+# using bb.warn(). The user needs to have privileges to run dmesg.
+
+def append_log(msg, d):
+    """
+    Appends a log message to variable '_dmesg_log'
+    """
+    current_log = d.getVar('_dmesg_log') or ''
+    current_log += msg + '\n'
+    d.setVar('_dmesg_log', current_log)
+
+def capture_dmesg():
+    """
+    Returns the current output of dmesg command
+    """
+    import subprocess
+
+    try:
+        result = subprocess.run(['dmesg'], capture_output=True, text=True, check=False)
+        if result.returncode != 0:
+            return f"Error running dmesg: {result.stderr.strip()}"
+        return result.stdout
+    except Exception as e:
+        return f"Exception running dmesg: {str(e)}"
+
+
+addhandler hostdmesglogger_eventhandler
+python hostdmesglogger_eventhandler() {
+    import difflib
+
+    start_marker = "=== BuildStarted dmesg ==="
+    end_marker = "=== BuildCompleted dmesg ==="
+    diff_marker = "=== dmesg diff ==="
+
+    # execute dmesg when BuildStarted event is fired
+    if isinstance(e, bb.event.BuildStarted):
+        dmesg_output = capture_dmesg()
+        if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"):
+            bb.warn(dmesg_output)
+        else:
+            append_log(start_marker, d)
+            append_log(dmesg_output, d)
+
+    # execute dmesg when BuildCompleted event is fired
+    if isinstance(e, bb.event.BuildCompleted):
+        dmesg_output = capture_dmesg()
+        if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"):
+            bb.warn(dmesg_output)
+        else:
+            append_log(end_marker, d)
+            append_log(dmesg_output, d)
+
+        content = d.getVar('_dmesg_log') or ''
+        if start_marker in content and end_marker in content:
+            start_dmesg = content.split(start_marker)[1].split(end_marker)[0]
+            end_dmesg = content.split(end_marker)[1]

meta/classes-global/host_dmesg_logger.bbclass line 74

Quoted 10 lines
meta/classes-global/host_dmesg_logger.bbclass new file +10 -0
+
+            start_lines = start_dmesg.strip().splitlines()
+            end_lines = end_dmesg.strip().splitlines()
+
+            # generating diff between BuildStarted and BuildCompleted dmesg outputs
+            diff = list(difflib.unified_diff(
+                start_lines, end_lines,
+                fromfile='dmesg_start',
+                tofile='dmesg_end',
+                lineterm=''

Add spaces around "=" ?

Quoted 12 lines
meta/classes-global/host_dmesg_logger.bbclass new file +12 -0
+            ))
+
+            append_log(diff_marker, d)
+            if diff:
+                for line in diff:
+                    bb.warn(line)
+            else:
+                bb.warn("No differences in dmesg output.")
+        else:
+            bb.warn("Could not find both dmesg sections for diff.")
+}
+hostdmesglogger_eventhandler[eventmask] = "bb.event.BuildStarted bb.event.BuildCompleted"
Signature · 2 lines
# Randy MacLeod
# Wind River Linux