Skip to contentSkip to conversation

event: Inject empty lines to make code match lineno in filename

PATCH In review [email protected] 2 participants 5 messages Jan 9, 2024 – Jan 10, 2024

lore
Share & tools
lore
#0 lore raw

v4 cover event: Inject empty lines to make code match lineno in filename

From: Robert Yang <[email protected]>

* V4:
- Check lineno is not None
- Add a testcase for it

// Robert

The following changes since commit 070aa227057e4913cf77c2465393a5a8398d5641:

bitbake: toaster/tests: Delay driver first action on create new project page (2024-01-07 22:59:55 +0000)

are available in the Git repository at:

https://github.com/robertlinux/yocto rbt/event
https://github.com/robertlinux/yocto/tree/rbt/event

Robert Yang (2):
bitbake: event: Inject empty lines to make code match lineno in
filename
bitbake: tests/event: Add test_lineno_in_eventhandler

 bitbake/lib/bb/event.py       |  5 +++--
 bitbake/lib/bb/tests/event.py | 24 ++++++++++++++++++++++++
 2 files changed, 27 insertions(+), 2 deletions(-)
Signature · 1 line
2.35.5
#1 lore raw

PATCH v4 1/2 bitbake: event: Inject empty lines to make code match lineno in filename

From: Robert Yang <[email protected]>

So that we can get the correct error messages.

* In python 3.10.9, the error message was:
ERROR: Unable to register event handler 'defaultbase_eventhandler':
File "/path/to/poky/meta/classes-global/base.bbclass", line 4
# SPDX-License-Identifier: MIT
^^^^^
SyntaxError: invalid syntax

This is hard to debug since the error line number 4 is incorrect, but nothing is wrong with the code in line 4.

* Now the error message and lineno is correct:
ERROR: Unable to register event handler 'defaultbase_eventhandler':
File "/path/to/poky/meta/classes-global/base.bbclass", line 256
an error line
^^^^^
SyntaxError: invalid syntax

And no impact on parsing time:
* Before:

$ rm -fr cache tmp; time bitbake -p
real    0m27.254s

* Now:

$ rm -fr cache tmp; time bitbake -p
real    0m27.200s
---
 bitbake/lib/bb/event.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
bitbake/lib/bb/event.py +3 -2
index f8acacd80d1..4761c868800 100644
@@ -257,14 +257,15 @@ def register(name, handler, mask=None, filename=None, lineno=None, data=None):
         # handle string containing python code
         if isinstance(handler, str):
             tmp = "def %s(e, d):\n%s" % (name, handler)
+            # Inject empty lines to make code match lineno in filename
+            if lineno is not None:
+                tmp = "\n" * (lineno-1) + tmp
             try:
                 code = bb.methodpool.compile_cache(tmp)
                 if not code:
                     if filename is None:
                         filename = "%s(e, d)" % name
                     code = compile(tmp, filename, "exec", ast.PyCF_ONLY_AST)
-                    if lineno is not None:
-                        ast.increment_lineno(code, lineno-1)
                     code = compile(code, filename, "exec")
                     bb.methodpool.compile_cache_add(tmp, code)
             except SyntaxError:
Signature · 1 line
2.35.5
#2 lore raw

PATCH v4 2/2 bitbake: tests/event: Add test_lineno_in_eventhandler

In reply to [email protected]

From: Robert Yang <[email protected]>

Add test_lineno_in_eventhandler to test lineno in eventhandler.

---
 bitbake/lib/bb/tests/event.py | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
bitbake/lib/bb/tests/event.py +24 -0
index d959f2d95db..ef61891d302 100644
@@ -13,6 +13,7 @@ import pickle
 import threading
 import time
 import unittest
+import tempfile
 from unittest.mock import Mock
 from unittest.mock import call
 
@@ -468,6 +469,8 @@ class EventClassesTest(unittest.TestCase):
 
     def setUp(self):
         bb.event.worker_pid = EventClassesTest._worker_pid
+        self.d = bb.data.init()
+        bb.parse.siggen = bb.siggen.init(self.d)
 
     def test_Event(self):
         """ Test the Event base class """
@@ -950,3 +953,24 @@ class EventClassesTest(unittest.TestCase):
         event = bb.event.FindSigInfoResult(result)
         self.assertEqual(event.result, result)
         self.assertEqual(event.pid, EventClassesTest._worker_pid)
+
+    def test_lineno_in_eventhandler(self):
+        # The error lineno is 5, not 4 since the first line is '\n'
+        error_line = """
+# Comment line1
+# Comment line2
+python test_lineno_in_eventhandler() {
+    This is an error line
+}
+addhandler test_lineno_in_eventhandler
+test_lineno_in_eventhandler[eventmask] = "bb.event.ConfigParsed"
+"""
+
+        with self.assertLogs() as logs:
+            f = tempfile.NamedTemporaryFile(suffix = '.bb')
+            f.write(bytes(error_line, "utf-8"))
+            f.flush()
+            d = bb.parse.handle(f.name, self.d)['']
+
+        output = "".join(logs.output)
+        self.assertTrue(" line 5\n" in output)
Signature · 1 line
2.35.5
#3 lore raw

Re: [bitbake-devel] [PATCH v4 2/2] bitbake: tests/event: Add test_lineno_in_eventhandler

This does look like a good idea, though I wonder if injecting newlines is truly the best approach. SyntaxError is a bit of a special case, since it occurs before we have the ast to adjust the lineno, but it can be corrected without string manipulation if you catch and re-raise SyntaxError. Ex:

def _syntaxerror_offset(value, lineoffset):
"""Adjust the line number in a SyntaxError exception"""
if lineoffset:
msg, (efname, elineno, eoffset, badline) = value.args value.args = (msg, (efname, elineno + lineoffset, eoffset, badline)) value.lineno = elineno + lineoffset

—
Christopher Larson
[email protected], [email protected], [email protected] Principal Software Engineer, Embedded Linux Solutions, Siemens Digital Industries Software

Quoted 69 lines · Robert Yang via lists.openembedded.org

From: Robert Yang <[email protected]>

Add test_lineno_in_eventhandler to test lineno in eventhandler.

---
bitbake/lib/bb/tests/event.py | 24 ++++++++++++++++++++++++

1 file changed, 24 insertions(+)
bitbake/lib/bb/tests/event.py +0 -0
index d959f2d95db..ef61891d302 100644
@@ -13,6 +13,7 @@ import pickle

import threading
import time
import unittest
+import tempfile
from unittest.mock import Mock
from unittest.mock import call

bitbake/lib/bb/tests/event.py +23 -0
@@ -468,6 +469,8 @@ class EventClassesTest(unittest.TestCase):
 
    def setUp(self):
        bb.event.worker_pid = EventClassesTest._worker_pid
+        self.d = bb.data.init()
+        bb.parse.siggen = bb.siggen.init(self.d)
 
    def test_Event(self):
        """ Test the Event base class """
@@ -950,3 +953,24 @@ class EventClassesTest(unittest.TestCase):
        event = bb.event.FindSigInfoResult(result)
        self.assertEqual(event.result, result)
        self.assertEqual(event.pid, EventClassesTest._worker_pid)
+
+    def test_lineno_in_eventhandler(self):
+        # The error lineno is 5, not 4 since the first line is '\n'
+        error_line = """
+# Comment line1
+# Comment line2
+python test_lineno_in_eventhandler() {
+    This is an error line
+}
+addhandler test_lineno_in_eventhandler
+test_lineno_in_eventhandler[eventmask] = "bb.event.ConfigParsed"
+"""
+
+        with self.assertLogs() as logs:
+            f = tempfile.NamedTemporaryFile(suffix = '.bb')
+            f.write(bytes(error_line, "utf-8"))
+            f.flush()
+            d = bb.parse.handle(f.name, self.d)['']
+
+        output = "".join(logs.output)
+        self.assertTrue(" line 5\n" in output)
Signature · 10 lines
2.35.5


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#15756): https://lists.openembedded.org/g/bitbake-devel/message/15756
Mute This Topic: https://lists.openembedded.org/mt/103616263/3617123
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
#4 lore raw

Re: [bitbake-devel] [PATCH v4 2/2] bitbake: tests/event: Add test_lineno_in_eventhandler

Hi Christop,

Quoted 1 line · Christop her Larson

This does look like a good idea, though I wonder if injecting newlines is truly the best approach. SyntaxError is a bit of a special case, since it occurs

Injecting newlines are much simple, and it also can catch more errors such as IndentationError, now the IndentationError can be reported correctly:

ERROR: Unable to register event handler 'defaultbase_eventhandler':
File "/path/to/poky/meta/classes-global/base.bbclass", line 259
d.setVar("ORIGNATIVELSBSTRING", d.getVar("NATIVELSBSTRING", False))
^
IndentationError: unindent does not match any outer indentation level

Such errors were very hard to debug. The _syntaxerror_offset may work, but it is much more complicated than inject newlines.

// Robert

Quoted 87 lines

before we have the ast to adjust the lineno, but it can be corrected without string manipulation if you catch and re-raise SyntaxError. Ex:

def _syntaxerror_offset(value, lineoffset):
    """Adjust the line number in a SyntaxError exception"""
    if lineoffset:
        msg, (efname, elineno, eoffset, badline) = value.args value.args = (msg, (efname, elineno + lineoffset, eoffset, badline)) value.lineno = elineno + lineoffset

—
Christopher Larson
[email protected], [email protected], [email protected] Principal Software Engineer, Embedded Linux Solutions, Siemens Digital Industries Software

Quoted 72 lines · On Jan 9, 2024, at 12:45 AM, Robert Yang via lists.openembedded.org…

From: Robert Yang <[email protected]>

Add test_lineno_in_eventhandler to test lineno in eventhandler.

---
bitbake/lib/bb/tests/event.py <https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$> | 24 ++++++++++++++++++++++++

1 file changed, 24 insertions(+)
bitbake/lib/bb/tests/event.py <https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$> +0 -1
index d959f2d95db..ef61891d302 100644
--- a/bitbake/lib/bb/tests/event.py 

<https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$>
+++ b/bitbake/lib/bb/tests/event.py
<https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$>

bitbake/lib/bb/tests/event.py <https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$> +0 -0
@@ -13,6 +13,7 @@ import pickle

import threading
import time
import unittest
+import tempfile
from unittest.mock import Mock
from unittest.mock import call

bitbake/lib/bb/tests/event.py <https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$> +0 -0
@@ -468,6 +469,8 @@ class EventClassesTest(unittest.TestCase):
 

    def setUp(self):
        bb.event.worker_pid = EventClassesTest._worker_pid
+        self.d = bb.data.init()
+        bb.parse.siggen = bb.siggen.init(self.d)

    def test_Event(self):
        """ Test the Event base class """

bitbake/lib/bb/tests/event.py <https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$> +18 -0
@@ -950,3 +953,24 @@ class EventClassesTest(unittest.TestCase):
         event = bb.event.FindSigInfoResult(result)
         self.assertEqual(event.result, result)
         self.assertEqual(event.pid <https://urldefense.com/v3/__http://event.pid__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENhxad__$>, EventClassesTest._worker_pid)
+
+    def test_lineno_in_eventhandler(self):
+        # The error lineno is 5, not 4 since the first line is '\n'
+        error_line = """
+# Comment line1
+# Comment line2
+python test_lineno_in_eventhandler() {
+    This is an error line
+}
+addhandler test_lineno_in_eventhandler
+test_lineno_in_eventhandler[eventmask] = "bb.event.ConfigParsed"
+"""
+
+        with self.assertLogs() as logs:
+            f = tempfile.NamedTemporaryFile(suffix = '.bb')
+            f.write(bytes(error_line, "utf-8"))
+            f.flush()
+            d = bb.parse.handle(f.name 

<https://urldefense.com/v3/__http://f.name__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNEDC-ZnzD$>, self.d)[''] +
+        output = "".join(logs.output)
+        self.assertTrue(" line 5\n" in output)

Signature · 10 lines
2.35.5


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#15756): https://lists.openembedded.org/g/bitbake-devel/message/15756 <https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/message/15756__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNEHCVD821$>
Mute This Topic: https://lists.openembedded.org/mt/103616263/3617123 <https://urldefense.com/v3/__https://lists.openembedded.org/mt/103616263/3617123__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNEIbsb3Sa$>
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub <https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/unsub__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNEG8cswfl$> [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-