Skip to contentSkip to conversation

bitbake: event: Inject empty lines to make code match lineno in filename

PATCH Superseded [email protected] 3 participants 5 messages Dec 29, 2023 – Jan 8, 2024

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover bitbake: event: Inject empty lines to make code match lineno in filename

From: Robert Yang <[email protected]>

* V3: Inject empty lines to make code match lineno in filename as suggested by RP.

// Robert

The following changes since commit 6ce42d73a31bf9b2fc362fa4acad11de97ac96ec:

scripts: Drop shell sstate-cache-management (2023-12-28 10:57:49 +0000)

are available in the Git repository at:

https://github.com/robertlinux/yocto rbt/event
https://github.com/robertlinux/yocto/tree/rbt/event

Robert Yang (1):
bitbake: event: Inject empty lines to make code match lineno in
filename

 bitbake/lib/bb/event.py | 2 ++
 1 file changed, 2 insertions(+)
Signature · 1 line
2.42.0
#1 lore raw

PATCH v3 1/1 bitbake: event: Inject empty lines to make code match lineno in filename

From: Robert Yang <[email protected]>

So that we can get the correct error messages.

* In python 3.10.9, the error message was:
ERROR: Unable to register event handler 'defaultbase_eventhandler':
File "/path/to/poky/meta/classes-global/base.bbclass", line 4
# SPDX-License-Identifier: MIT
^^^^^
SyntaxError: invalid syntax

This is hard to debug since the error line number 4 is incorrect, but nothing is wrong with the code in line 4.

* Now the error message and lineno is correct:
ERROR: Unable to register event handler 'defaultbase_eventhandler':
File "/path/to/poky/meta/classes-global/base.bbclass", line 256
an error line
^^^^^
SyntaxError: invalid syntax

And no impact on parsing time:
* Before:

$ rm -fr cache tmp; time bitbake -p
real    0m27.254s

* Now:

$ rm -fr cache tmp; time bitbake -p
real    0m27.200s
---
 bitbake/lib/bb/event.py | 2 ++
 1 file changed, 2 insertions(+)
bitbake/lib/bb/event.py +2 -0
index f8acacd80d1..7722258b7e8 100644
@@ -257,6 +257,8 @@ def register(name, handler, mask=None, filename=None, lineno=None, data=None):
         # handle string containing python code
         if isinstance(handler, str):
             tmp = "def %s(e, d):\n%s" % (name, handler)
+            # Inject empty lines to make code match lineno in filename
+            tmp = "\n" * (lineno-1) + tmp
             try:
                 code = bb.methodpool.compile_cache(tmp)
                 if not code:
Signature · 1 line
2.42.0
#2 lore raw
Quoted 40 lines · Robert Yang via lists.openembedded.org

From: Robert Yang <[email protected]>

So that we can get the correct error messages.

* In python 3.10.9, the error message was:
ERROR: Unable to register event handler 'defaultbase_eventhandler':
File "/path/to/poky/meta/classes-global/base.bbclass", line 4
# SPDX-License-Identifier: MIT
^^^^^
SyntaxError: invalid syntax

This is hard to debug since the error line number 4 is incorrect, but nothing is wrong with the code in line 4.

* Now the error message and lineno is correct:
ERROR: Unable to register event handler 'defaultbase_eventhandler':
File "/path/to/poky/meta/classes-global/base.bbclass", line 256
an error line
^^^^^
SyntaxError: invalid syntax

And no impact on parsing time:
* Before:

$ rm -fr cache tmp; time bitbake -p
real    0m27.254s

* Now:

$ rm -fr cache tmp; time bitbake -p
real    0m27.200s
---
 bitbake/lib/bb/event.py | 2 ++
 1 file changed, 2 insertions(+)
bitbake/lib/bb/event.py +0 -0
index f8acacd80d1..7722258b7e8 100644

bitbake/lib/bb/event.py def register(name, handler, mask=None, filename=None, lineno=None, data=None): · line 264

Quoted 8 lines
bitbake/lib/bb/event.py +2 -0
@@ -257,6 +257,8 @@ def register(name, handler, mask=None, filename=None, lineno=None, data=None):
         # handle string containing python code
         if isinstance(handler, str):
             tmp = "def %s(e, d):\n%s" % (name, handler)
+            # Inject empty lines to make code match lineno in filename
+            tmp = "\n" * (lineno-1) + tmp
             try:
                 code = bb.methodpool.compile_cache(tmp)
                 if not code:

Looks good but don't you need remove the ast line number adjustment a few lines later as well?

Should we put some tests into bitbake-selftest for this?

Cheers,

Richard

#4 lore raw
In reply to [email protected]

This causes bitbake-selftest failures:

======================================================================
ERROR: test_register_from_string (bb.tests.event.EventHandlingTest)
Test register method receiving code in string
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/bitbake/lib/bb/tests/event.py", line 121, in test_register_from_string
result = bb.event.register("string_handler", " return True")
File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/bitbake/lib/bb/event.py", line 261, in register

    tmp = "\n" * (lineno-1) + tmp

TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6202/steps/11/logs/stdio

Quoted 61 lines · Robert Yang via lists.openembedded.org

From: Robert Yang <[email protected]>

So that we can get the correct error messages.

* In python 3.10.9, the error message was:
ERROR: Unable to register event handler 'defaultbase_eventhandler':
File "/path/to/poky/meta/classes-global/base.bbclass", line 4
# SPDX-License-Identifier: MIT
^^^^^
SyntaxError: invalid syntax

This is hard to debug since the error line number 4 is incorrect, but nothing is wrong with the code in line 4.

* Now the error message and lineno is correct:
ERROR: Unable to register event handler 'defaultbase_eventhandler':
File "/path/to/poky/meta/classes-global/base.bbclass", line 256
an error line
^^^^^
SyntaxError: invalid syntax

And no impact on parsing time:
* Before:

$ rm -fr cache tmp; time bitbake -p
real    0m27.254s

* Now:

$ rm -fr cache tmp; time bitbake -p
real    0m27.200s
---
 bitbake/lib/bb/event.py | 2 ++
 1 file changed, 2 insertions(+)
bitbake/lib/bb/event.py +2 -0
index f8acacd80d1..7722258b7e8 100644
@@ -257,6 +257,8 @@ def register(name, handler, mask=None, filename=None, lineno=None, data=None):
         # handle string containing python code
         if isinstance(handler, str):
             tmp = "def %s(e, d):\n%s" % (name, handler)
+            # Inject empty lines to make code match lineno in filename
+            tmp = "\n" * (lineno-1) + tmp
             try:
                 code = bb.methodpool.compile_cache(tmp)
                 if not code:
Signature · 11 lines
2.42.0



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#15717): https://lists.openembedded.org/g/bitbake-devel/message/15717
Mute This Topic: https://lists.openembedded.org/mt/103413642/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#3 lore raw
In reply to Richard Purdie
Quoted 37 lines · Richard Purdie
Quoted 35 lines · Robert Yang via lists.openembedded.org

From: Robert Yang <[email protected]>

So that we can get the correct error messages.

* In python 3.10.9, the error message was:
ERROR: Unable to register event handler 'defaultbase_eventhandler':
File "/path/to/poky/meta/classes-global/base.bbclass", line 4
# SPDX-License-Identifier: MIT
^^^^^
SyntaxError: invalid syntax

This is hard to debug since the error line number 4 is incorrect, but nothing is wrong with the code in line 4.

* Now the error message and lineno is correct:
ERROR: Unable to register event handler 'defaultbase_eventhandler':
File "/path/to/poky/meta/classes-global/base.bbclass", line 256
an error line
^^^^^
SyntaxError: invalid syntax

And no impact on parsing time:
* Before:

$ rm -fr cache tmp; time bitbake -p
real    0m27.254s

* Now:

$ rm -fr cache tmp; time bitbake -p
real    0m27.200s
---
  bitbake/lib/bb/event.py | 2 ++
  1 file changed, 2 insertions(+)
Quoted 16 lines

bitbake/lib/bb/event.py def register(name, handler, mask=None, filename=None, lineno=None, data=None): · line 264

Quoted 13 lines
bitbake/lib/bb/event.py +2 -0
index f8acacd80d1..7722258b7e8 100644
@@ -257,6 +257,8 @@ def register(name, handler, mask=None, filename=None, lineno=None, data=None):
          # handle string containing python code
          if isinstance(handler, str):
              tmp = "def %s(e, d):\n%s" % (name, handler)
+            # Inject empty lines to make code match lineno in filename
+            tmp = "\n" * (lineno-1) + tmp
              try:
                  code = bb.methodpool.compile_cache(tmp)
                  if not code:

Looks good but don't you need remove the ast line number adjustment a few lines later as well?

Yes, you're right, the code has incremented the lineno by '\n', so the following line isn't needed:

ast.increment_lineno(code, lineno-1)

Now the patch is: (Check lineno is not None to fix bitbake-selftest error)

bitbake/lib/bb/event.py +3 -2
@@ -257,14 +257,15 @@ def register(name, handler, mask=None, filename=None, lineno=None, data=None):
         # handle string containing python code
         if isinstance(handler, str):
             tmp = "def %s(e, d):\n%s" % (name, handler)
+            # Inject empty lines to make code match lineno in filename
+            if lineno is not None:
+                tmp = "\n" * (lineno-1) + tmp
             try:
                 code = bb.methodpool.compile_cache(tmp)
                 if not code:
                     if filename is None:
                         filename = "%s(e, d)" % name
                     code = compile(tmp, filename, "exec", ast.PyCF_ONLY_AST)
-                    if lineno is not None:
-                        ast.increment_lineno(code, lineno-1)
                     code = compile(code, filename, "exec")
                     bb.methodpool.compile_cache_add(tmp, code)
             except SyntaxError:

I will send a V4 add with a testcase for it.

// Robert

Quoted 6 lines

Should we put some tests into bitbake-selftest for this?

Cheers,

Richard