Skip to contentSkip to conversation

devshell: introduce intercepts as per native / nativesdk

PATCH In review Paul Gortmaker 4 participants 4 messages Apr 21, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 devshell: introduce intercepts as per native / nativesdk

In a devshell, recent versions of git will complain if the repo is owned by someone other than the current UID - consider this example:

------
bitbake -c devshell linux-yocto

[...]

kernel-source#git branch
fatal: unsafe repository ('/home/paul/poky/build-qemuarm64/tmp/work-shared/qemuarm64/kernel-source' is owned by someone else) To add an exception for this directory, call:

        git config --global --add safe.directory /home/paul/poky/build-qemuarm64/tmp/work-shared/qemuarm64/kernel-source

kernel-source#
------

Of course the devshell has UID zero and the "real" UID is for "paul" in this case. And so recent git versions complain.

As the whole purpose of the devshell is to invoke a shell where development can take place, having a non-functional git is clearly unacceptable.

Richard suggested we could use PSEUDO_UNLOAD=1 to evade this issue, and I suggested we probably will see other similar instances like this and should make use of PATH to intercept via devshell wrappers - conveniently we already have examples of this.

Here, we copy the existing "ar" example and tune it to the needs of git to combine Richard's suggestion and mine.

As such we now also can store commit logs and use send-email with our user specific settings, instead of "root", so in additon to fixing basic commands like "git branch" it should also increase general usefulness.

meta/classes/devshell.bbclass +2 -0
index 62dc958d9a4a..7ac134e0950f 100644
@@ -2,6 +2,8 @@ inherit terminal
 
 DEVSHELL = "${SHELL}"
 
+PATH:prepend = "${COREBASE}/scripts/devshell-intercept:"
+
 python do_devshell () {
     if d.getVarFlag("do_devshell", "manualfakeroot"):
        d.prependVar("DEVSHELL", "pseudo ")
scripts/devshell-intercept/git new file +19 -0
new file mode 100755
index 000000000000..8adf5c9ecb71
@@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+#
+# Wrapper around 'git' that doesn't think we are root
+
+import os
+import shutil
+import sys
+
+os.environ['PSEUDO_UNLOAD'] = '1'
+
+# calculate path to the real 'git'
+path = os.environ['PATH']
+path = path.replace(os.path.dirname(sys.argv[0]), '')
+real_git = shutil.which('git', path=path)
+
+if len(sys.argv) == 1:
+    os.execl(real_git, 'git')
+
+os.execv(real_git, sys.argv)
Signature · 1 line
2.17.1
#1 lore raw
Quoted 28 lines · Paul Gortmaker

In a devshell, recent versions of git will complain if the repo is owned by someone other than the current UID - consider this example:

------
bitbake -c devshell linux-yocto

[...]

kernel-source#git branch
fatal: unsafe repository ('/home/paul/poky/build-qemuarm64/tmp/work-shared/qemuarm64/kernel-source' is owned by someone else) To add an exception for this directory, call:

        git config --global --add safe.directory /home/paul/poky/build-qemuarm64/tmp/work-shared/qemuarm64/kernel-source

kernel-source#
------

Of course the devshell has UID zero and the "real" UID is for "paul" in this case. And so recent git versions complain.

As the whole purpose of the devshell is to invoke a shell where development can take place, having a non-functional git is clearly unacceptable.

Richard suggested we could use PSEUDO_UNLOAD=1 to evade this issue, and I suggested we probably will see other similar instances like this and should make use of PATH to intercept via devshell wrappers - conveniently we already have examples of this.

Quoted 6 lines

Here, we copy the existing "ar" example and tune it to the needs of git to combine Richard's suggestion and mine.

As such we now also can store commit logs and use send-email with our user specific settings, instead of "root", so in additon to fixing basic commands like "git branch" it should also increase general usefulness.

It looks like this will address the performance issues of using git from a devshell too. Trying to run git log, git blame, etc for largeish repos in a devshell is annoyingly slow due to the overhead of pseudo.

I always wondered what people do in a devshell which actually _does_ need to run under pseudo...

Quoted 53 lines
meta/classes/devshell.bbclass +2 -0
index 62dc958d9a4a..7ac134e0950f 100644
@@ -2,6 +2,8 @@ inherit terminal
 
 DEVSHELL = "${SHELL}"
 
+PATH:prepend = "${COREBASE}/scripts/devshell-intercept:"
+
 python do_devshell () {
     if d.getVarFlag("do_devshell", "manualfakeroot"):
        d.prependVar("DEVSHELL", "pseudo ")
scripts/devshell-intercept/git new file +19 -1
new file mode 100755
index 000000000000..8adf5c9ecb71
@@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+#
+# Wrapper around 'git' that doesn't think we are root
+
+import os
+import shutil
+import sys
+
+os.environ['PSEUDO_UNLOAD'] = '1'
+
+# calculate path to the real 'git'
+path = os.environ['PATH']
+path = path.replace(os.path.dirname(sys.argv[0]), '')
+real_git = shutil.which('git', path=path)
+
+if len(sys.argv) == 1:
+    os.execl(real_git, 'git')
+
+os.execv(real_git, sys.argv)
--

2.17.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#164701): https://lists.openembedded.org/g/openembedded-core/message/164701
Mute This Topic: https://lists.openembedded.org/mt/90585970/3619030
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#2 lore raw
Quoted 55 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Andre McCurdy
Sent: den 21 april 2022 01:25
To: Paul Gortmaker <[email protected]>
Cc: OE Core mailing list <[email protected]>; Richard Purdie <[email protected]>
Subject: Re: [OE-core] [PATCH] devshell: introduce intercepts as per native / nativesdk

Quoted 11 lines · Paul Gortmaker

In a devshell, recent versions of git will complain if the repo is owned by someone other than the current UID - consider this example:

------
bitbake -c devshell linux-yocto

[...]

kernel-source#git branch
fatal: unsafe repository ('/home/paul/poky/build-qemuarm64/tmp/work-

shared/qemuarm64/kernel-source' is owned by someone else)

Quoted 3 lines

To add an exception for this directory, call:

        git config --global --add safe.directory /home/paul/poky/build-

qemuarm64/tmp/work-shared/qemuarm64/kernel-source

Quoted 7 lines

kernel-source#
------

Of course the devshell has UID zero and the "real" UID is for "paul" in this case. And so recent git versions complain.

As the whole purpose of the devshell is to invoke a shell where

development

Quoted 3 lines

can take place, having a non-functional git is clearly unacceptable.

Richard suggested we could use PSEUDO_UNLOAD=1 to evade this issue, and

I

Quoted 1 line

suggested we probably will see other similar instances like this and

should

Quoted 1 line

make use of PATH to intercept via devshell wrappers - conveniently we

already

Quoted 3 lines

have examples of this.

Here, we copy the existing "ar" example and tune it to the needs of git

to

Quoted 3 lines

combine Richard's suggestion and mine.

As such we now also can store commit logs and use send-email with our

user

Quoted 2 lines

specific settings, instead of "root", so in additon to fixing basic commands like "git branch" it should also increase general usefulness.

Quoted 7 lines

It looks like this will address the performance issues of using git from a devshell too. Trying to run git log, git blame, etc for largeish repos in a devshell is annoyingly slow due to the overhead of pseudo.

I always wondered what people do in a devshell which actually _does_ need to run under pseudo...

To look at the output from, e.g., do_install to see it the way bitbakes sees it when it comes to users/groups of the files.

//Peter

Quoted 50 lines

Review

Quoted 5 lines
(quoted diff) +0 -0

b/meta/classes/devshell.bbclass

scripts/devshell-

Quoted 13 lines

index 62dc958d9a4a..7ac134e0950f 100644

meta/classes/devshell.bbclass +2 -0
@@ -2,6 +2,8 @@ inherit terminal
 
 DEVSHELL = "${SHELL}"
 
+PATH:prepend = "${COREBASE}/scripts/devshell-intercept:"
+
 python do_devshell () {
     if d.getVarFlag("do_devshell", "manualfakeroot"):
        d.prependVar("DEVSHELL", "pseudo ")
scripts/devshell- renamed from scripts/devshell-intercept/git +0 -0

intercept/git

Quoted 30 lines

new file mode 100755
index 000000000000..8adf5c9ecb71

scripts/devshell-intercept/git new file +19 -1
@@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+#
+# Wrapper around 'git' that doesn't think we are root
+
+import os
+import shutil
+import sys
+
+os.environ['PSEUDO_UNLOAD'] = '1'
+
+# calculate path to the real 'git'
+path = os.environ['PATH']
+path = path.replace(os.path.dirname(sys.argv[0]), '')
+real_git = shutil.which('git', path=path)
+
+if len(sys.argv) == 1:
+    os.execl(real_git, 'git')
+
+os.execv(real_git, sys.argv)
--

2.17.1

#3 lore raw
In reply to Paul Gortmaker
Quoted 42 lines · Paul Gortmaker

In a devshell, recent versions of git will complain if the repo is owned by someone other than the current UID - consider this example:

------
bitbake -c devshell linux-yocto

[...]

kernel-source#git branch
fatal: unsafe repository ('/home/paul/poky/build-qemuarm64/tmp/work-shared/qemuarm64/kernel-source' is owned by someone else) To add an exception for this directory, call:

        git config --global --add safe.directory /home/paul/poky/build-qemuarm64/tmp/work-shared/qemuarm64/kernel-source

kernel-source#
------

Of course the devshell has UID zero and the "real" UID is for "paul" in this case. And so recent git versions complain.

As the whole purpose of the devshell is to invoke a shell where development can take place, having a non-functional git is clearly unacceptable.

Richard suggested we could use PSEUDO_UNLOAD=1 to evade this issue, and I suggested we probably will see other similar instances like this and should make use of PATH to intercept via devshell wrappers - conveniently we already have examples of this.

Here, we copy the existing "ar" example and tune it to the needs of git to combine Richard's suggestion and mine.

As such we now also can store commit logs and use send-email with our user specific settings, instead of "root", so in additon to fixing basic commands like "git branch" it should also increase general usefulness.

meta/classes/devshell.bbclass +0 -0
index 62dc958d9a4a..7ac134e0950f 100644

meta/classes/devshell.bbclass inherit terminal · line 6

Quoted 5 lines
meta/classes/devshell.bbclass +2 -0
@@ -2,6 +2,8 @@ inherit terminal
 
 DEVSHELL = "${SHELL}"
 
+PATH:prepend = "${COREBASE}/scripts/devshell-intercept:"
+

I just realised, we can't do that! It would apply globally and mean builds change behaviour depending on whether devshell is inherited or not. It also means all git calls run through the wrapper, not just devshell ones.

I've also realised we have a bigger problem since some software is running git to get the version during do_install.

My proposal is to change scripts/devshell-intercept to scripts/git-intercept, make the PATH change in the do_devshell function below but also add something like:

PATH:prepend:task-do_install = "${COREBASE}/scripts/git-intercept:"

in common code.

Since this hasn't merged yet, I can tweak the patch if that works for you (or you can)?

Cheers,

Richard