Skip to contentSkip to conversation

perf: Correct the substitution of python shebangs

PATCH Superseded [email protected] 6 participants 8 messages Feb 14, 2020 – Feb 21, 2020

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 perf: Correct the substitution of python shebangs

From: He Zhe <[email protected]>

To make the native python3 is always used,

- Use sed one-liner instead
- Add substitution for ${S}/scripts/bpf_helpers_doc.py to fix the following warning.

File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata'

This issue is first reported by Joel Stanley <[email protected]> The sed one-liner is credited to Anuj Mittal <[email protected]>

---
 meta/recipes-kernel/perf/perf.bb | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
meta/recipes-kernel/perf/perf.bb +2 -5
index 6d1b06693d..a6fb51d3db 100644
@@ -239,11 +239,8 @@ do_configure_prepend () {
     fi
 
     # use /usr/bin/env instead of version specific python
-    for s in `find ${S}/tools/perf/ -name '*.py'`; do
-        sed -i 's,/usr/bin/python,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/python2,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/env python2,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/python3,/usr/bin/env python3,' "${s}"
+    for s in `find ${S}/tools/perf/ -name '*.py'` ${S}/scripts/bpf_helpers_doc.py; do
+        sed -i -e "s,#!.*python.*,#!${USRBINPATH}/env python3," ${s}
     done
 
     # unistd.h can be out of sync between libc-headers and the captured version in the perf source
Signature · 1 line
2.24.1
#1 lore raw

TA

________________________________________
From: [email protected] [[email protected]] on behalf of [email protected] [[email protected]]
Sent: Friday, February 14, 2020 11:57 AM
To: [email protected]
Subject: [OE-core] [PATCH v3] perf: Correct the substitution of python shebangs

From: He Zhe <[email protected]>

To make the native python3 is always used,

- Use sed one-liner instead
- Add substitution for ${S}/scripts/bpf_helpers_doc.py to fix the following warning.

File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata'

This issue is first reported by Joel Stanley <[email protected]> The sed one-liner is credited to Anuj Mittal <[email protected]>

---
 meta/recipes-kernel/perf/perf.bb | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
meta/recipes-kernel/perf/perf.bb +2 -5
index 6d1b06693d..a6fb51d3db 100644
@@ -239,11 +239,8 @@ do_configure_prepend () {
     fi
 
     # use /usr/bin/env instead of version specific python
-    for s in `find ${S}/tools/perf/ -name '*.py'`; do
-        sed -i 's,/usr/bin/python,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/python2,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/env python2,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/python3,/usr/bin/env python3,' "${s}"
+    for s in `find ${S}/tools/perf/ -name '*.py'` ${S}/scripts/bpf_helpers_doc.py; do
+        sed -i -e "s,#!.*python.*,#!${USRBINPATH}/env python3," ${s}
     done
 
     # unistd.h can be out of sync between libc-headers and the captured version in the perf source

--
2.24.1

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#2 lore raw
In reply to [email protected]
Quoted 9 lines · On Thu, Feb 13, 2020 at 10:58 PM <[email protected]> wrote:

From: He Zhe <[email protected]>

To make the native python3 is always used,

- Use sed one-liner instead
- Add substitution for ${S}/scripts/bpf_helpers_doc.py to fix the following warning.

Quoted 6 lines

File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata'

This issue is first reported by Joel Stanley <[email protected]> The sed one-liner is credited to Anuj Mittal <[email protected]>

Thanks!

This looks good to me.

Cheers,

Bruce

Quoted 32 lines
---
 meta/recipes-kernel/perf/perf.bb | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
meta/recipes-kernel/perf/perf.bb +2 -6
index 6d1b06693d..a6fb51d3db 100644
@@ -239,11 +239,8 @@ do_configure_prepend () {
     fi
 
     # use /usr/bin/env instead of version specific python
-    for s in `find ${S}/tools/perf/ -name '*.py'`; do
-        sed -i 's,/usr/bin/python,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/python2,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/env python2,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/python3,/usr/bin/env python3,' "${s}"
+    for s in `find ${S}/tools/perf/ -name '*.py'` ${S}/scripts/bpf_helpers_doc.py; do
+        sed -i -e "s,#!.*python.*,#!${USRBINPATH}/env python3," ${s}
     done
 
     # unistd.h can be out of sync between libc-headers and the captured version in the perf source
--

2.24.1

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#3 lore raw
Quoted 14 lines · Bruce Ashfield
Quoted 13 lines · On Thu, Feb 13, 2020 at 10:58 PM <[email protected]> wrote:

From: He Zhe <[email protected]>

To make the native python3 is always used,

- Use sed one-liner instead
- Add substitution for ${S}/scripts/bpf_helpers_doc.py to fix the following warning.

File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix _temp = __import__(name, globals(), locals(), ['build_time_vars'],
0)
ModuleNotFoundError: No module named '_sysconfigdata'

Quoted 7 lines
Quoted 3 lines

This issue is first reported by Joel Stanley <[email protected]> The sed one-liner is credited to Anuj Mittal <[email protected]

Quoted 1 line

Thanks!

This looks good to me.

I'm not sure how but with this applied we see breaking kernel module tests:

https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/1555
https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/275
https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/1559
(mips/sysvinit, mips/systemd and mips64)

I'll remove this patch from -next and retest see if I can confirm it is this change somehow.

Cheers,

Richard

#4 lore raw
In reply to [email protected]
Quoted 26 lines · On Thu, Feb 13, 2020 at 9:58 PM <[email protected]> wrote:

From: He Zhe <[email protected]>

To make the native python3 is always used,

- Use sed one-liner instead
- Add substitution for ${S}/scripts/bpf_helpers_doc.py to fix the following warning.

File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata'

This issue is first reported by Joel Stanley <[email protected]> The sed one-liner is credited to Anuj Mittal <[email protected]>

---
 meta/recipes-kernel/perf/perf.bb | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
meta/recipes-kernel/perf/perf.bb +0 -0
index 6d1b06693d..a6fb51d3db 100644

meta/recipes-kernel/perf/perf.bb do_configure_prepend () { · line 242

Quoted 9 lines
meta/recipes-kernel/perf/perf.bb +1 -5
@@ -239,11 +239,8 @@ do_configure_prepend () {
     fi
 
     # use /usr/bin/env instead of version specific python
-    for s in `find ${S}/tools/perf/ -name '*.py'`; do
-        sed -i 's,/usr/bin/python,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/python2,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/env python2,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/python3,/usr/bin/env python3,' "${s}"
+    for s in `find ${S}/tools/perf/ -name '*.py'` ${S}/scripts/bpf_helpers_doc.py; do

Do we have to hard code bpf_helpers_doc.py? It doesn't exist in older versions (e.g. 4.14) and causes do_configure to fail.

Quoted 12 lines
meta/recipes-kernel/perf/perf.bb +1 -1
+        sed -i -e "s,#!.*python.*,#!${USRBINPATH}/env python3," ${s}
     done
 
     # unistd.h can be out of sync between libc-headers and the captured version in the perf source
--

2.24.1

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#5 lore raw
Quoted 29 lines · Joshua Watt
Quoted 28 lines · On Thu, Feb 13, 2020 at 9:58 PM <[email protected]> wrote:

From: He Zhe <[email protected]>

To make the native python3 is always used,

- Use sed one-liner instead
- Add substitution for ${S}/scripts/bpf_helpers_doc.py to fix the following warning.

File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata'

This issue is first reported by Joel Stanley <[email protected]> The sed one-liner is credited to Anuj Mittal <[email protected]>

---
  meta/recipes-kernel/perf/perf.bb | 7 ++-----
  1 file changed, 2 insertions(+), 5 deletions(-)
meta/recipes-kernel/perf/perf.bb +0 -0
index 6d1b06693d..a6fb51d3db 100644
@@ -239,11 +239,8 @@ do_configure_prepend () {
      fi
 
Quoted 10 lines

meta/recipes-kernel/perf/perf.bb do_configure_prepend () { · line 242

Quoted 7 lines
meta/recipes-kernel/perf/perf.bb +1 -5
      # use /usr/bin/env instead of version specific python
-    for s in `find ${S}/tools/perf/ -name '*.py'`; do
-        sed -i 's,/usr/bin/python,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/python2,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/env python2,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/python3,/usr/bin/env python3,' "${s}"
+    for s in `find ${S}/tools/perf/ -name '*.py'` ${S}/scripts/bpf_helpers_doc.py; do

Do we have to hard code bpf_helpers_doc.py? It doesn't exist in older versions (e.g. 4.14) and causes do_configure to fail.

Probably not, I also stumbled into this problem.

Quoted 13 lines
Quoted 13 lines
meta/recipes-kernel/perf/perf.bb +1 -1
+        sed -i -e "s,#!.*python.*,#!${USRBINPATH}/env python3," ${s}
      done
 
      # unistd.h can be out of sync between libc-headers and the captured version in the perf source
--

2.24.1

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#6 lore raw
Quoted 44 lines · Khem Raj
Quoted 29 lines · Joshua Watt
Quoted 28 lines · On Thu, Feb 13, 2020 at 9:58 PM <[email protected]> wrote:

From: He Zhe <[email protected]>

To make the native python3 is always used,

- Use sed one-liner instead
- Add substitution for ${S}/scripts/bpf_helpers_doc.py to fix the following warning.

File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata'

This issue is first reported by Joel Stanley <[email protected]> The sed one-liner is credited to Anuj Mittal <[email protected]>

---
  meta/recipes-kernel/perf/perf.bb | 7 ++-----
  1 file changed, 2 insertions(+), 5 deletions(-)
meta/recipes-kernel/perf/perf.bb +0 -0
index 6d1b06693d..a6fb51d3db 100644
@@ -239,11 +239,8 @@ do_configure_prepend () {
      fi
 
Quoted 10 lines

meta/recipes-kernel/perf/perf.bb do_configure_prepend () { · line 242

Quoted 7 lines
meta/recipes-kernel/perf/perf.bb +1 -5
      # use /usr/bin/env instead of version specific python
-    for s in `find ${S}/tools/perf/ -name '*.py'`; do
-        sed -i 's,/usr/bin/python,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/python2,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/env python2,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/python3,/usr/bin/env python3,' "${s}"
+    for s in `find ${S}/tools/perf/ -name '*.py'` ${S}/scripts/bpf_helpers_doc.py; do

Do we have to hard code bpf_helpers_doc.py? It doesn't exist in older versions (e.g. 4.14) and causes do_configure to fail.

Quoted 1 line

Probably not, I also stumbled into this problem.

Yah,

We can either just do the entire directory (but during the review, we were talking about the fact that since we aren't using them, we don't actually know if they are py3 safe, so blindly converting the shebang isnt' a great idea either).

But the solution we use in the devsrc recipe would work easily enough here, just do $(find . -name ${S}/scripts/bpf_helpers_doc.py) and let it return nothing for those older kernels.

Bruce

Quoted 19 lines
Quoted 9 lines
Quoted 9 lines
meta/recipes-kernel/perf/perf.bb +1 -1
+        sed -i -e "s,#!.*python.*,#!${USRBINPATH}/env python3," ${s}
      done
 
      # unistd.h can be out of sync between libc-headers and the captured version in the perf source
--

2.24.1

Quoted 5 lines
Quoted 5 lines

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#7 lore raw
Quoted 49 lines · Bruce Ashfield
Quoted 44 lines · Khem Raj
Quoted 29 lines · Joshua Watt
Quoted 28 lines · On Thu, Feb 13, 2020 at 9:58 PM <[email protected]> wrote:

From: He Zhe <[email protected]>

To make the native python3 is always used,

- Use sed one-liner instead
- Add substitution for ${S}/scripts/bpf_helpers_doc.py to fix the following warning.

File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata'

This issue is first reported by Joel Stanley <[email protected]> The sed one-liner is credited to Anuj Mittal <[email protected]>

---
   meta/recipes-kernel/perf/perf.bb | 7 ++-----
   1 file changed, 2 insertions(+), 5 deletions(-)
meta/recipes-kernel/perf/perf.bb +0 -0
index 6d1b06693d..a6fb51d3db 100644
@@ -239,11 +239,8 @@ do_configure_prepend () {
       fi
 
Quoted 10 lines

meta/recipes-kernel/perf/perf.bb do_configure_prepend () { · line 242

Quoted 7 lines
meta/recipes-kernel/perf/perf.bb +1 -5
       # use /usr/bin/env instead of version specific python
-    for s in `find ${S}/tools/perf/ -name '*.py'`; do
-        sed -i 's,/usr/bin/python,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/python2,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/env python2,/usr/bin/env python3,' "${s}"
-        sed -i 's,/usr/bin/python3,/usr/bin/env python3,' "${s}"
+    for s in `find ${S}/tools/perf/ -name '*.py'` ${S}/scripts/bpf_helpers_doc.py; do

Do we have to hard code bpf_helpers_doc.py? It doesn't exist in older versions (e.g. 4.14) and causes do_configure to fail.

Quoted 1 line

Probably not, I also stumbled into this problem.

Yah,

Quoted 8 lines

We can either just do the entire directory (but during the review, we were talking about the fact that since we aren't using them, we don't actually know if they are py3 safe, so blindly converting the shebang isnt' a great idea either).

But the solution we use in the devsrc recipe would work easily enough here, just do $(find . -name ${S}/scripts/bpf_helpers_doc.py) and let it return nothing for those older kernels.

then perhaps its better to callout each script by name and check if that file exists, so older kernels can ignore that

Quoted 25 lines

Bruce

Quoted 19 lines
Quoted 9 lines
Quoted 9 lines
meta/recipes-kernel/perf/perf.bb +1 -1
+        sed -i -e "s,#!.*python.*,#!${USRBINPATH}/env python3," ${s}
       done
 
       # unistd.h can be out of sync between libc-headers and the captured version in the perf source
--

2.24.1

Quoted 5 lines
Quoted 5 lines

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core