Skip to contentSkip to conversation

Fix some bitbake authentication issues

PATCH In review Stefan Klug 3 participants 7 messages Mar 18, 2019 – Mar 19, 2019

lore
Share & tools
lore
#0 lore raw

cover Fix some bitbake authentication issues

Hi,

while using bitbake in our corporate environment I stumbled over some authentication/stability issues.
This is my first patchset to this list. So please correct me if I missed anything.

Regards Stefan

Stefan Klug (3):
fetch2: Fix fetching of git repositories with kerberos authentication
fetch2: Gracefully handle corrupt download-cache tarballs fetch2/wget: Fix authentication in checkstatus() of the wget fetcher

 lib/bb/fetch2/__init__.py | 1 +
 lib/bb/fetch2/git.py      | 7 +++++--
 lib/bb/fetch2/wget.py     | 4 ++--
 3 files changed, 8 insertions(+), 4 deletions(-)
#1 lore raw

PATCH 1/3 fetch2: Fix fetching of git repositories with kerberos authentication

When using pam_krb to login to a system KRB5CCNAME is set to the corresponding kerberos auth cache file. The bitbake fetcher removes this variable from the environment leading to a git authentication failure. Also the fetcher ignores the normally used BB_ENV_[EXTRA_]WHITE variables and relies on a hardcoded list.
Therefore it is impossible to fix this issue outside of bitbake.

---
 lib/bb/fetch2/__init__.py | 1 +
 1 file changed, 1 insertion(+)
lib/bb/fetch2/__init__.py +1 -0
index f112067d..df8e83e7 100644
@@ -830,6 +830,7 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None):
                   'GIT_SSH',
                   'GIT_SSL_CAINFO',
                   'GIT_SMART_HTTP',
+                  'KRB5CCNAME',
                   'SSH_AUTH_SOCK', 'SSH_AGENT_PID',
                   'SOCKS5_USER', 'SOCKS5_PASSWD',
                   'DBUS_SESSION_BUS_ADDRESS',
#2 lore raw

Re: [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication

Quoted 6 lines · Stefan Klug

When using pam_krb to login to a system KRB5CCNAME is set to the corresponding kerberos auth cache file. The bitbake fetcher removes this variable from the environment leading to a git authentication failure.
Also the fetcher ignores the normally used BB_ENV_[EXTRA_]WHITE variables and relies on a hardcoded list.

This code should probably include the BB_ENV_[EXTRA_]WHITE list variables...

That doesn't invalidate this patch but does suggest we have a bigger problem.

Cheers,

Richard

Quoted 23 lines

Therefore it is impossible to fix this issue outside of bitbake.

---
 lib/bb/fetch2/__init__.py | 1 +
 1 file changed, 1 insertion(+)
lib/bb/fetch2/__init__.py +0 -0
index f112067d..df8e83e7 100644
@@ -830,6 +830,7 @@ def runfetchcmd(cmd, d, quiet=False,

cleanup=None, log=None, workdir=None):

                   'GIT_SSH',
                   'GIT_SSL_CAINFO',
                   'GIT_SMART_HTTP',
+                  'KRB5CCNAME',
                   'SSH_AUTH_SOCK', 'SSH_AGENT_PID',
                   'SOCKS5_USER', 'SOCKS5_PASSWD',
                   'DBUS_SESSION_BUS_ADDRESS',
#3 lore raw

PATCH 2/3 fetch2: Gracefully handle corrupt download-cache tarballs

In reply to Stefan Klug

If the fullmirror tarball is corrupt for whatever reason (IMHO there are no checksums on the download cache) a series of nasty events was triggered:
- tar left a partially extracted bare git repo there
- on the next yocto build, the corrupt bare repo is found and bitbake starts to update that bare repo using git
- git fails to detect it as bare repo. Therefore all following git commands ripple up the directory tree, in our case modifying a top level git repo.

---
 lib/bb/fetch2/git.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
lib/bb/fetch2/git.py +5 -2
index 1a8ebe3d..d5f9bbcd 100644
@@ -334,8 +334,11 @@ class Git(FetchMethod):
             ud.localpath = ud.fullshallow
             return
         elif os.path.exists(ud.fullmirror) and not os.path.exists(ud.clonedir):
-            bb.utils.mkdirhier(ud.clonedir)
-            runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir)
+            try:
+                bb.utils.mkdirhier(ud.clonedir)
+                runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir, cleanup=[ud.clonedir])
+            except:
+                logger.info("Extracting tarball of git repository failed, falling back to clone.")
 
         repourl = self._get_repo_url(ud)
 
#4 lore raw

Re: [PATCH 2/3] fetch2: Gracefully handle corrupt download-cache tarballs

Quoted 20 lines · Stefan Klug

If the fullmirror tarball is corrupt for whatever reason (IMHO there are no checksums on the download cache) a series of nasty events was triggered:
- tar left a partially extracted bare git repo there
- on the next yocto build, the corrupt bare repo is found and bitbake starts to update that bare repo using git
- git fails to detect it as bare repo. Therefore all following git commands ripple up the directory tree, in our case modifying a top level git repo.

---
 lib/bb/fetch2/git.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
lib/bb/fetch2/git.py +0 -0
index 1a8ebe3d..d5f9bbcd 100644

lib/bb/fetch2/git.py class Git(FetchMethod): · line 341

Quoted 10 lines
lib/bb/fetch2/git.py +5 -2
@@ -334,8 +334,11 @@ class Git(FetchMethod):
             ud.localpath = ud.fullshallow
             return
         elif os.path.exists(ud.fullmirror) and not os.path.exists(ud.clonedir):
-            bb.utils.mkdirhier(ud.clonedir)
-            runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir)
+            try:
+                bb.utils.mkdirhier(ud.clonedir)
+                runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir, cleanup=[ud.clonedir])
+            except:
+                logger.info("Extracting tarball of git repository failed, falling back to clone.")

General "except:" clauses are a world of pain. Can we be more specific here?

For an example of what I mean, put a syntax error in the command...

Cheers,

Richard

#5 lore raw

PATCH 3/3 fetch2/wget: Fix authentication in checkstatus() of the wget fetcher

In reply to Stefan Klug

I wonder how this used to work for anybody.

---
 lib/bb/fetch2/wget.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
lib/bb/fetch2/wget.py +2 -2
index e2037511..3addb219 100644
@@ -322,8 +322,8 @@ class Wget(FetchMethod):
                 authheader =  "Basic %s" % encodeuser
                 r.add_header("Authorization", authheader)
 
-            if ud.user:
-                add_basic_auth(ud.user, r)
+            if ud.user and ud.pswd:
+                add_basic_auth(ud.user + ':' + ud.pswd, r)
 
             try:
                 import netrc, urllib.parse
#6 lore raw

Re: [PATCH 3/3] fetch2/wget: Fix authentication in checkstatus() of the wget fetcher

Quoted 2 lines · Stefan Klug

I wonder how this used to work for anybody.

No need to wonder when you have access to the git history... :-)

http://git.openembedded.org/bitbake/commit/?id=cea8113d14da9e12db80a5b6b5811a47a7dfdeef

It looks like ud.user used to contain both the username and password... and when that changed, download() was updated but checkstatus() was not:

http://git.openembedded.org/bitbake/commit/?id=6a917ec99d659e684b15fa8af94c325172676062

Quoted 28 lines
---
 lib/bb/fetch2/wget.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
lib/bb/fetch2/wget.py +2 -4
index e2037511..3addb219 100644
@@ -322,8 +322,8 @@ class Wget(FetchMethod):
                 authheader =  "Basic %s" % encodeuser
                 r.add_header("Authorization", authheader)
 
-            if ud.user:
-                add_basic_auth(ud.user, r)
+            if ud.user and ud.pswd:
+                add_basic_auth(ud.user + ':' + ud.pswd, r)
 
             try:
                 import netrc, urllib.parse
--
 
 
--

_______________________________________________
bitbake-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/bitbake-devel