event: Inject empty lines to make code match lineno in filename
v4 cover event: Inject empty lines to make code match lineno in filename
From: Robert Yang <[email protected]>
* V4:
- Check lineno is not None
- Add a testcase for it
// Robert
The following changes since commit 070aa227057e4913cf77c2465393a5a8398d5641:
bitbake: toaster/tests: Delay driver first action on create new project page (2024-01-07 22:59:55 +0000)
are available in the Git repository at:
https://github.com/robertlinux/yocto rbt/event
https://github.com/robertlinux/yocto/tree/rbt/event
Robert Yang (2):
bitbake: event: Inject empty lines to make code match lineno in
filename
bitbake: tests/event: Add test_lineno_in_eventhandler
bitbake/lib/bb/event.py | 5 +++-- bitbake/lib/bb/tests/event.py | 24 ++++++++++++++++++++++++ 2 files changed, 27 insertions(+), 2 deletions(-)
Signature · 1 line
2.35.5
PATCH v4 1/2 bitbake: event: Inject empty lines to make code match lineno in filename
From: Robert Yang <[email protected]>
So that we can get the correct error messages.
* In python 3.10.9, the error message was:
ERROR: Unable to register event handler 'defaultbase_eventhandler':
File "/path/to/poky/meta/classes-global/base.bbclass", line 4
# SPDX-License-Identifier: MIT
^^^^^
SyntaxError: invalid syntax
This is hard to debug since the error line number 4 is incorrect, but nothing is wrong with the code in line 4.
* Now the error message and lineno is correct:
ERROR: Unable to register event handler 'defaultbase_eventhandler':
File "/path/to/poky/meta/classes-global/base.bbclass", line 256
an error line
^^^^^
SyntaxError: invalid syntax
And no impact on parsing time:
* Before:
$ rm -fr cache tmp; time bitbake -p real 0m27.254s
* Now:
$ rm -fr cache tmp; time bitbake -p real 0m27.200s
--- bitbake/lib/bb/event.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
bitbake/lib/bb/event.py +3 -2
@@ -257,14 +257,15 @@ def register(name, handler, mask=None, filename=None, lineno=None, data=None): # handle string containing python code if isinstance(handler, str): tmp = "def %s(e, d):\n%s" % (name, handler) + # Inject empty lines to make code match lineno in filename + if lineno is not None: + tmp = "\n" * (lineno-1) + tmp try: code = bb.methodpool.compile_cache(tmp) if not code: if filename is None: filename = "%s(e, d)" % name code = compile(tmp, filename, "exec", ast.PyCF_ONLY_AST) - if lineno is not None: - ast.increment_lineno(code, lineno-1) code = compile(code, filename, "exec") bb.methodpool.compile_cache_add(tmp, code) except SyntaxError:
Signature · 1 line
2.35.5
PATCH v4 2/2 bitbake: tests/event: Add test_lineno_in_eventhandler
In reply to [email protected]From: Robert Yang <[email protected]>
Add test_lineno_in_eventhandler to test lineno in eventhandler.
--- bitbake/lib/bb/tests/event.py | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
bitbake/lib/bb/tests/event.py +24 -0
@@ -13,6 +13,7 @@ import pickle import threading import time import unittest +import tempfile from unittest.mock import Mock from unittest.mock import call
@@ -468,6 +469,8 @@ class EventClassesTest(unittest.TestCase): def setUp(self): bb.event.worker_pid = EventClassesTest._worker_pid + self.d = bb.data.init() + bb.parse.siggen = bb.siggen.init(self.d) def test_Event(self): """ Test the Event base class """
@@ -950,3 +953,24 @@ class EventClassesTest(unittest.TestCase): event = bb.event.FindSigInfoResult(result) self.assertEqual(event.result, result) self.assertEqual(event.pid, EventClassesTest._worker_pid) + + def test_lineno_in_eventhandler(self): + # The error lineno is 5, not 4 since the first line is '\n' + error_line = """ +# Comment line1 +# Comment line2 +python test_lineno_in_eventhandler() { + This is an error line +} +addhandler test_lineno_in_eventhandler +test_lineno_in_eventhandler[eventmask] = "bb.event.ConfigParsed" +""" + + with self.assertLogs() as logs: + f = tempfile.NamedTemporaryFile(suffix = '.bb') + f.write(bytes(error_line, "utf-8")) + f.flush() + d = bb.parse.handle(f.name, self.d)[''] + + output = "".join(logs.output) + self.assertTrue(" line 5\n" in output)
Signature · 1 line
2.35.5
Re: [bitbake-devel] [PATCH v4 2/2] bitbake: tests/event: Add test_lineno_in_eventhandler
This does look like a good idea, though I wonder if injecting newlines is truly the best approach. SyntaxError is a bit of a special case, since it occurs before we have the ast to adjust the lineno, but it can be corrected without string manipulation if you catch and re-raise SyntaxError. Ex:
def _syntaxerror_offset(value, lineoffset):
"""Adjust the line number in a SyntaxError exception"""
if lineoffset:
msg, (efname, elineno, eoffset, badline) = value.args value.args = (msg, (efname, elineno + lineoffset, eoffset, badline)) value.lineno = elineno + lineoffset
—
Christopher Larson
[email protected], [email protected], [email protected] Principal Software Engineer, Embedded Linux Solutions, Siemens Digital Industries Software
Quoted 69 lines · Robert Yang via lists.openembedded.org
From: Robert Yang <[email protected]>
Add
test_lineno_in_eventhandlerto test lineno in eventhandler.Signed-off-by Robert Yang <[email protected]>---
bitbake/lib/bb/tests/event.py| 24 ++++++++++++++++++++++++1 file changed, 24 insertions(+)bitbake/lib/bb/tests/event.py +0
-0@@ -13,6 +13,7 @@ import pickleimport threading
import time
import unittest
+import tempfile
from unittest.mock import Mock
from unittest.mock import callbitbake/lib/bb/tests/event.py +23
-0@@ -468,6 +469,8 @@ class EventClassesTest(unittest.TestCase): def setUp(self): bb.event.worker_pid = EventClassesTest._worker_pid + self.d = bb.data.init() + bb.parse.siggen = bb.siggen.init(self.d) def test_Event(self): """ Test the Event base class """@@ -950,3 +953,24 @@ class EventClassesTest(unittest.TestCase): event = bb.event.FindSigInfoResult(result) self.assertEqual(event.result, result) self.assertEqual(event.pid, EventClassesTest._worker_pid) + + def test_lineno_in_eventhandler(self): + # The error lineno is 5, not 4 since the first line is '\n' + error_line = """ +# Comment line1 +# Comment line2 +python test_lineno_in_eventhandler() { + This is an error line +} +addhandler test_lineno_in_eventhandler +test_lineno_in_eventhandler[eventmask] = "bb.event.ConfigParsed" +""" + + with self.assertLogs() as logs: + f = tempfile.NamedTemporaryFile(suffix = '.bb') + f.write(bytes(error_line, "utf-8")) + f.flush() + d = bb.parse.handle(f.name, self.d)[''] + + output = "".join(logs.output) + self.assertTrue(" line 5\n" in output)Signature · 10 lines
2.35.5 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#15756): https://lists.openembedded.org/g/bitbake-devel/message/15756 Mute This Topic: https://lists.openembedded.org/mt/103616263/3617123 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
Re: [bitbake-devel] [PATCH v4 2/2] bitbake: tests/event: Add test_lineno_in_eventhandler
Hi Christop,
Quoted 1 line · Christop her Larson
This does look like a good idea, though I wonder if injecting newlines is truly the best approach. SyntaxError is a bit of a special case, since it occurs
Injecting newlines are much simple, and it also can catch more errors such as IndentationError, now the IndentationError can be reported correctly:
ERROR: Unable to register event handler 'defaultbase_eventhandler':
File "/path/to/poky/meta/classes-global/base.bbclass", line 259
d.setVar("ORIGNATIVELSBSTRING", d.getVar("NATIVELSBSTRING", False))
^
IndentationError: unindent does not match any outer indentation level
Such errors were very hard to debug. The _syntaxerror_offset may work, but it is much more complicated than inject newlines.
// Robert
Quoted 87 lines
before we have the ast to adjust the lineno, but it can be corrected without string manipulation if you catch and re-raise SyntaxError. Ex:
def
_syntaxerror_offset(value, lineoffset):
"""Adjust the line number in a SyntaxError exception"""
if lineoffset:
msg, (efname, elineno, eoffset, badline) = value.args value.args = (msg, (efname, elineno + lineoffset, eoffset, badline)) value.lineno = elineno + lineoffset—
Christopher Larson
[email protected], [email protected], [email protected] Principal Software Engineer, Embedded Linux Solutions, Siemens Digital Industries SoftwareQuoted 72 lines · On Jan 9, 2024, at 12:45 AM, Robert Yang via lists.openembedded.org…
From: Robert Yang <[email protected]>
Add
test_lineno_in_eventhandlerto test lineno in eventhandler.Signed-off-by Robert Yang <[email protected]>---
bitbake/lib/bb/tests/event.py<https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$> | 24 ++++++++++++++++++++++++1 file changed, 24 insertions(+)bitbake/lib/bb/tests/event.py <https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$> +0
-1--- a/bitbake/lib/bb/tests/event.py<https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$>
+++b/bitbake/lib/bb/tests/event.py
<https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$>bitbake/lib/bb/tests/event.py <https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$> +0
-0@@ -13,6 +13,7 @@ import pickleimport threading
import time
import unittest
+import tempfile
from unittest.mock import Mock
from unittest.mock import callbitbake/lib/bb/tests/event.py <https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$> +0
-0@@ -468,6 +469,8 @@ class EventClassesTest(unittest.TestCase):def setUp(self):
bb.event.worker_pid = EventClassesTest._worker_pid
+ self.d =bb.data.init()
+ bb.parse.siggen =bb.siggen.init(self.d)def
test_Event(self):
""" Test the Event base class """bitbake/lib/bb/tests/event.py <https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$> +18
-0@@ -950,3 +953,24 @@ class EventClassesTest(unittest.TestCase): event = bb.event.FindSigInfoResult(result) self.assertEqual(event.result, result) self.assertEqual(event.pid <https://urldefense.com/v3/__http://event.pid__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENhxad__$>, EventClassesTest._worker_pid) + + def test_lineno_in_eventhandler(self): + # The error lineno is 5, not 4 since the first line is '\n' + error_line = """ +# Comment line1 +# Comment line2 +python test_lineno_in_eventhandler() { + This is an error line +} +addhandler test_lineno_in_eventhandler +test_lineno_in_eventhandler[eventmask] = "bb.event.ConfigParsed" +""" + + with self.assertLogs() as logs: + f = tempfile.NamedTemporaryFile(suffix = '.bb') + f.write(bytes(error_line, "utf-8")) + f.flush() + d = bb.parse.handle(f.name<https://urldefense.com/v3/__http://f.name__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNEDC-ZnzD$>, self.d)[''] +
+ output = "".join(logs.output)
+ self.assertTrue(" line 5\n" in output)Signature · 10 lines
2.35.5 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#15756): https://lists.openembedded.org/g/bitbake-devel/message/15756 <https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/message/15756__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNEHCVD821$> Mute This Topic: https://lists.openembedded.org/mt/103616263/3617123 <https://urldefense.com/v3/__https://lists.openembedded.org/mt/103616263/3617123__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNEIbsb3Sa$> Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub <https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/unsub__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNEG8cswfl$> [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-