devshell: introduce intercepts as per native / nativesdk
PATCH 1/1 devshell: introduce intercepts as per native / nativesdk
In a devshell, recent versions of git will complain if the repo is owned by someone other than the current UID - consider this example:
------
bitbake -c devshell linux-yocto
[...]
kernel-source#git branch
fatal: unsafe repository ('/home/paul/poky/build-qemuarm64/tmp/work-shared/qemuarm64/kernel-source' is owned by someone else) To add an exception for this directory, call:
git config --global --add safe.directory /home/paul/poky/build-qemuarm64/tmp/work-shared/qemuarm64/kernel-source
kernel-source#
------
Of course the devshell has UID zero and the "real" UID is for "paul" in this case. And so recent git versions complain.
As the whole purpose of the devshell is to invoke a shell where development can take place, having a non-functional git is clearly unacceptable.
Richard suggested we could use PSEUDO_UNLOAD=1 to evade this issue, and I suggested we probably will see other similar instances like this and should make use of PATH to intercept via devshell wrappers - conveniently we already have examples of this.
Here, we copy the existing "ar" example and tune it to the needs of git to combine Richard's suggestion and mine.
As such we now also can store commit logs and use send-email with our user specific settings, instead of "root", so in additon to fixing basic commands like "git branch" it should also increase general usefulness.
meta/classes/devshell.bbclass +2 -0
@@ -2,6 +2,8 @@ inherit terminal DEVSHELL = "${SHELL}" +PATH:prepend = "${COREBASE}/scripts/devshell-intercept:" + python do_devshell () { if d.getVarFlag("do_devshell", "manualfakeroot"): d.prependVar("DEVSHELL", "pseudo ")
scripts/devshell-intercept/git new file +19 -0
@@ -0,0 +1,19 @@ +#!/usr/bin/env python3 +# +# Wrapper around 'git' that doesn't think we are root + +import os +import shutil +import sys + +os.environ['PSEUDO_UNLOAD'] = '1' + +# calculate path to the real 'git' +path = os.environ['PATH'] +path = path.replace(os.path.dirname(sys.argv[0]), '') +real_git = shutil.which('git', path=path) + +if len(sys.argv) == 1: + os.execl(real_git, 'git') + +os.execv(real_git, sys.argv)
Signature · 1 line
2.17.1
Quoted 28 lines · Paul Gortmaker
In a devshell, recent versions of git will complain if the repo is owned by someone other than the current UID - consider this example:
------
bitbake -c devshell linux-yocto[...]
kernel-source#git branch
fatal: unsafe repository ('/home/paul/poky/build-qemuarm64/tmp/work-shared/qemuarm64/kernel-source' is owned by someone else) To add an exception for this directory, call:git config --global --add safe.directory /home/paul/poky/build-qemuarm64/tmp/work-shared/qemuarm64/kernel-sourcekernel-source#
------Of course the devshell has UID zero and the "real" UID is for "paul" in this case. And so recent git versions complain.
As the whole purpose of the devshell is to invoke a shell where development can take place, having a non-functional git is clearly unacceptable.
Richard suggested we could use PSEUDO_UNLOAD=1 to evade this issue, and I suggested we probably will see other similar instances like this and should make use of PATH to intercept via devshell wrappers - conveniently we already have examples of this.
Quoted 6 lines
Here, we copy the existing "ar" example and tune it to the needs of git to combine Richard's suggestion and mine.
As such we now also can store commit logs and use send-email with our user specific settings, instead of "root", so in additon to fixing basic commands like "git branch" it should also increase general usefulness.
It looks like this will address the performance issues of using git from a devshell too. Trying to run git log, git blame, etc for largeish repos in a devshell is annoyingly slow due to the overhead of pseudo.
I always wondered what people do in a devshell which actually _does_ need to run under pseudo...
Quoted 53 lines
Cc Richard Purdie <[email protected]>Signed-off-by Paul Gortmaker <[email protected]>meta/classes/devshell.bbclass +2
-0@@ -2,6 +2,8 @@ inherit terminal DEVSHELL = "${SHELL}" +PATH:prepend = "${COREBASE}/scripts/devshell-intercept:" + python do_devshell () { if d.getVarFlag("do_devshell", "manualfakeroot"): d.prependVar("DEVSHELL", "pseudo ")scripts/devshell-intercept/git new file +19
-1@@ -0,0 +1,19 @@ +#!/usr/bin/env python3 +# +# Wrapper around 'git' that doesn't think we are root + +import os +import shutil +import sys + +os.environ['PSEUDO_UNLOAD'] = '1' + +# calculate path to the real 'git' +path = os.environ['PATH'] +path = path.replace(os.path.dirname(sys.argv[0]), '') +real_git = shutil.which('git', path=path) + +if len(sys.argv) == 1: + os.execl(real_git, 'git') + +os.execv(real_git, sys.argv) --2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#164701): https://lists.openembedded.org/g/openembedded-core/message/164701
Mute This Topic: https://lists.openembedded.org/mt/90585970/3619030
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 55 lines
-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Andre McCurdy
Sent: den 21 april 2022 01:25
To: Paul Gortmaker <[email protected]>
Cc: OE Core mailing list <[email protected]>; Richard Purdie <[email protected]>
Subject: Re: [OE-core] [PATCH] devshell: introduce intercepts as per native / nativesdkQuoted 11 lines · Paul Gortmaker
In a devshell, recent versions of git will complain if the repo is owned by someone other than the current UID - consider this example:
------
bitbake -c devshell linux-yocto[...]
kernel-source#git branch
fatal: unsafe repository ('/home/paul/poky/build-qemuarm64/tmp/work-shared/qemuarm64/kernel-source' is owned by someone else)
Quoted 3 lines
To add an exception for this directory, call:
git config --global --add safe.directory /home/paul/poky/build-qemuarm64/tmp/work-shared/qemuarm64/kernel-source
Quoted 7 lines
kernel-source#
------Of course the devshell has UID zero and the "real" UID is for "paul" in this case. And so recent git versions complain.
As the whole purpose of the devshell is to invoke a shell where
development
Quoted 3 lines
can take place, having a non-functional git is clearly unacceptable.
Richard suggested we could use PSEUDO_UNLOAD=1 to evade this issue, and
I
Quoted 1 line
suggested we probably will see other similar instances like this and
should
Quoted 1 line
make use of PATH to intercept via devshell wrappers - conveniently we
already
Quoted 3 lines
have examples of this.
Here, we copy the existing "ar" example and tune it to the needs of git
to
Quoted 3 lines
combine Richard's suggestion and mine.
As such we now also can store commit logs and use send-email with our
user
Quoted 2 lines
specific settings, instead of "root", so in additon to fixing basic commands like "git branch" it should also increase general usefulness.
Quoted 7 lines
It looks like this will address the performance issues of using git from a devshell too. Trying to run git log, git blame, etc for largeish repos in a devshell is annoyingly slow due to the overhead of pseudo.
I always wondered what people do in a devshell which actually _does_ need to run under pseudo...
To look at the output from, e.g., do_install to see it the way bitbakes sees it when it comes to users/groups of the files.
//Peter
Quoted 50 lines
Review
Quoted 5 lines
Cc Richard Purdie <[email protected]>Signed-off-by Paul Gortmaker <[email protected]>(quoted diff) +0
-0b/meta/classes/devshell.bbclass
scripts/devshell-
Quoted 13 lines
index
62dc958d9a4a..7ac134e0950f100644meta/classes/devshell.bbclass +2
-0@@ -2,6 +2,8 @@ inherit terminal DEVSHELL = "${SHELL}" +PATH:prepend = "${COREBASE}/scripts/devshell-intercept:" + python do_devshell () { if d.getVarFlag("do_devshell", "manualfakeroot"): d.prependVar("DEVSHELL", "pseudo ")scripts/devshell- renamed from scripts/devshell-intercept/git +0
-0intercept/git
Quoted 30 lines
new file mode 100755
index 000000000000..8adf5c9ecb71scripts/devshell-intercept/git new file +19
-1@@ -0,0 +1,19 @@ +#!/usr/bin/env python3 +# +# Wrapper around 'git' that doesn't think we are root + +import os +import shutil +import sys + +os.environ['PSEUDO_UNLOAD'] = '1' + +# calculate path to the real 'git' +path = os.environ['PATH'] +path = path.replace(os.path.dirname(sys.argv[0]), '') +real_git = shutil.which('git', path=path) + +if len(sys.argv) == 1: + os.execl(real_git, 'git') + +os.execv(real_git, sys.argv) --2.17.1
Quoted 42 lines · Paul Gortmaker
In a devshell, recent versions of git will complain if the repo is owned by someone other than the current UID - consider this example:
------
bitbake -c devshell linux-yocto[...]
kernel-source#git branch
fatal: unsafe repository ('/home/paul/poky/build-qemuarm64/tmp/work-shared/qemuarm64/kernel-source' is owned by someone else) To add an exception for this directory, call:git config --global --add safe.directory /home/paul/poky/build-qemuarm64/tmp/work-shared/qemuarm64/kernel-sourcekernel-source#
------Of course the devshell has UID zero and the "real" UID is for "paul" in this case. And so recent git versions complain.
As the whole purpose of the devshell is to invoke a shell where development can take place, having a non-functional git is clearly unacceptable.
Richard suggested we could use PSEUDO_UNLOAD=1 to evade this issue, and I suggested we probably will see other similar instances like this and should make use of PATH to intercept via devshell wrappers - conveniently we already have examples of this.
Here, we copy the existing "ar" example and tune it to the needs of git to combine Richard's suggestion and mine.
As such we now also can store commit logs and use send-email with our user specific settings, instead of "root", so in additon to fixing basic commands like "git branch" it should also increase general usefulness.
Cc Richard Purdie <[email protected]>Signed-off-by Paul Gortmaker <[email protected]>meta/classes/devshell.bbclass +0
-0
meta/classes/devshell.bbclass inherit terminal · line 6
Quoted 5 lines
meta/classes/devshell.bbclass +2
-0@@ -2,6 +2,8 @@ inherit terminal DEVSHELL = "${SHELL}" +PATH:prepend = "${COREBASE}/scripts/devshell-intercept:" +
I just realised, we can't do that! It would apply globally and mean builds change behaviour depending on whether devshell is inherited or not. It also means all git calls run through the wrapper, not just devshell ones.
I've also realised we have a bigger problem since some software is running git to get the version during do_install.
My proposal is to change scripts/devshell-intercept to scripts/git-intercept, make the PATH change in the do_devshell function below but also add something like:
PATH:prepend:task-do_install = "${COREBASE}/scripts/git-intercept:"
in common code.
Since this hasn't merged yet, I can tweak the patch if that works for you (or you can)?
Cheers,
Richard