Fix some bitbake authentication issues
cover Fix some bitbake authentication issues
Hi,
while using bitbake in our corporate environment I stumbled over some authentication/stability issues.
This is my first patchset to this list. So please correct me if I missed anything.
Regards Stefan
Stefan Klug (3):
fetch2: Fix fetching of git repositories with kerberos authentication
fetch2: Gracefully handle corrupt download-cache tarballs fetch2/wget: Fix authentication in checkstatus() of the wget fetcher
lib/bb/fetch2/__init__.py | 1 + lib/bb/fetch2/git.py | 7 +++++-- lib/bb/fetch2/wget.py | 4 ++-- 3 files changed, 8 insertions(+), 4 deletions(-)
PATCH 1/3 fetch2: Fix fetching of git repositories with kerberos authentication
When using pam_krb to login to a system KRB5CCNAME is set to the corresponding kerberos auth cache file. The bitbake fetcher removes this variable from the environment leading to a git authentication failure. Also the fetcher ignores the normally used BB_ENV_[EXTRA_]WHITE variables and relies on a hardcoded list.
Therefore it is impossible to fix this issue outside of bitbake.
--- lib/bb/fetch2/__init__.py | 1 + 1 file changed, 1 insertion(+)
lib/bb/fetch2/__init__.py +1 -0
@@ -830,6 +830,7 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None): 'GIT_SSH', 'GIT_SSL_CAINFO', 'GIT_SMART_HTTP', + 'KRB5CCNAME', 'SSH_AUTH_SOCK', 'SSH_AGENT_PID', 'SOCKS5_USER', 'SOCKS5_PASSWD', 'DBUS_SESSION_BUS_ADDRESS',
Re: [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication
Quoted 6 lines · Stefan Klug
When using
pam_krbto login to a system KRB5CCNAME is set to the corresponding kerberos auth cache file. The bitbake fetcher removes this variable from the environment leading to a git authentication failure.
Also the fetcher ignores the normally used BB_ENV_[EXTRA_]WHITE variables and relies on a hardcoded list.
This code should probably include the BB_ENV_[EXTRA_]WHITE list variables...
That doesn't invalidate this patch but does suggest we have a bigger problem.
Cheers,
Richard
Quoted 23 lines
Therefore it is impossible to fix this issue outside of bitbake.
Signed-off-by Stefan Klug <[email protected]>--- lib/bb/fetch2/__init__.py | 1 + 1 file changed, 1 insertion(+)lib/bb/fetch2/__init__.py +0
-0@@ -830,6 +830,7 @@ def runfetchcmd(cmd, d, quiet=False,cleanup=None, log=None, workdir=None):
'GIT_SSH', 'GIT_SSL_CAINFO', 'GIT_SMART_HTTP', + 'KRB5CCNAME', 'SSH_AUTH_SOCK', 'SSH_AGENT_PID', 'SOCKS5_USER', 'SOCKS5_PASSWD', 'DBUS_SESSION_BUS_ADDRESS',
PATCH 2/3 fetch2: Gracefully handle corrupt download-cache tarballs
In reply to Stefan KlugIf the fullmirror tarball is corrupt for whatever reason (IMHO there are no checksums on the download cache) a series of nasty events was triggered:
- tar left a partially extracted bare git repo there
- on the next yocto build, the corrupt bare repo is found and bitbake starts to update that bare repo using git
- git fails to detect it as bare repo. Therefore all following git commands ripple up the directory tree, in our case modifying a top level git repo.
--- lib/bb/fetch2/git.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
lib/bb/fetch2/git.py +5 -2
@@ -334,8 +334,11 @@ class Git(FetchMethod): ud.localpath = ud.fullshallow return elif os.path.exists(ud.fullmirror) and not os.path.exists(ud.clonedir): - bb.utils.mkdirhier(ud.clonedir) - runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir) + try: + bb.utils.mkdirhier(ud.clonedir) + runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir, cleanup=[ud.clonedir]) + except: + logger.info("Extracting tarball of git repository failed, falling back to clone.") repourl = self._get_repo_url(ud)
Re: [PATCH 2/3] fetch2: Gracefully handle corrupt download-cache tarballs
Quoted 20 lines · Stefan Klug
If the fullmirror tarball is corrupt for whatever reason (IMHO there are no checksums on the download cache) a series of nasty events was triggered:
- tar left a partially extracted bare git repo there
- on the next yocto build, the corrupt bare repo is found and bitbake starts to update that bare repo using git
- git fails to detect it as bare repo. Therefore all following git commands ripple up the directory tree, in our case modifying a top level git repo.Signed-off-by Stefan Klug <[email protected]>--- lib/bb/fetch2/git.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)lib/bb/fetch2/git.py +0
-0
lib/bb/fetch2/git.py class Git(FetchMethod): · line 341
Quoted 10 lines
lib/bb/fetch2/git.py +5
-2@@ -334,8 +334,11 @@ class Git(FetchMethod): ud.localpath = ud.fullshallow return elif os.path.exists(ud.fullmirror) and not os.path.exists(ud.clonedir): - bb.utils.mkdirhier(ud.clonedir) - runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir) + try: + bb.utils.mkdirhier(ud.clonedir) + runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir, cleanup=[ud.clonedir]) + except: + logger.info("Extracting tarball of git repository failed, falling back to clone.")
General "except:" clauses are a world of pain. Can we be more specific here?
For an example of what I mean, put a syntax error in the command...
Cheers,
Richard
PATCH 3/3 fetch2/wget: Fix authentication in checkstatus() of the wget fetcher
In reply to Stefan KlugI wonder how this used to work for anybody.
--- lib/bb/fetch2/wget.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
lib/bb/fetch2/wget.py +2 -2
@@ -322,8 +322,8 @@ class Wget(FetchMethod): authheader = "Basic %s" % encodeuser r.add_header("Authorization", authheader) - if ud.user: - add_basic_auth(ud.user, r) + if ud.user and ud.pswd: + add_basic_auth(ud.user + ':' + ud.pswd, r) try: import netrc, urllib.parse
Re: [PATCH 3/3] fetch2/wget: Fix authentication in checkstatus() of the wget fetcher
Quoted 2 lines · Stefan Klug
I wonder how this used to work for anybody.
No need to wonder when you have access to the git history... :-)
http://git.openembedded.org/bitbake/commit/?id=cea8113d14da9e12db80a5b6b5811a47a7dfdeef
It looks like ud.user used to contain both the username and password... and when that changed, download() was updated but checkstatus() was not:
http://git.openembedded.org/bitbake/commit/?id=6a917ec99d659e684b15fa8af94c325172676062
Quoted 28 lines
Signed-off-by Stefan Klug <[email protected]>--- lib/bb/fetch2/wget.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)lib/bb/fetch2/wget.py +2
-4@@ -322,8 +322,8 @@ class Wget(FetchMethod): authheader = "Basic %s" % encodeuser r.add_header("Authorization", authheader) - if ud.user: - add_basic_auth(ud.user, r) + if ud.user and ud.pswd: + add_basic_auth(ud.user + ':' + ud.pswd, r) try: import netrc, urllib.parse -- --_______________________________________________
bitbake-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/bitbake-devel