Skip to contentSkip to conversation

Add a new variable:'FETCH_ENV_WHITELIST'

PATCH In review Mingrui Ren 2 participants 3 messages Jul 9, 2021 – Jul 10, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Add a new variable:'FETCH_ENV_WHITELIST'

From d228100b7551b2327d66072b03f2a809744e8f52 Mon Sep 17 00:00:00 2001
From: Mingrui Ren <[email protected]>
Date: Thu, 8 Jul 2021 23:44:53 +0800
Subject: [PATCH] Add a new variable:'FETCH_ENV_WHITELIST'

The enviroment variables used by Fetcher are hard-coded, and are obtained from HOST env instead of bitbake datastore.
This patch adds a new variable 'FETCH_ENV_WHITELIST' similar to 'BB_ENV_EXTRAWHITE', trying to fix the problems above.

---
The situation I encountered was that I tried to use a custom git-proxy by setting GIT_PROXY_COMMAND and SOCK_PROXY enviroment variables. However, I found that even if I set BB_ENV_EXTRAWHITE, the SOCK_PROXY variable does not work.

 bitbake/lib/bb/fetch2/__init__.py | 16 +---------------
 bitbake/lib/bb/utils.py           |  5 +++++
 scripts/oe-buildenv-internal      |  7 +++++++
 3 files changed, 13 insertions(+), 15 deletions(-)
bitbake/lib/bb/fetch2/__init__.py +1 -15
index c8e91262a9..7cdc6c5ee8 100644
@@ -820,21 +820,7 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None):
     # rather than host provided
     # Also include some other variables.
     # FIXME: Should really include all export varaiables?
-    exportvars = ['HOME', 'PATH',
-                  'HTTP_PROXY', 'http_proxy',
-                  'HTTPS_PROXY', 'https_proxy',
-                  'FTP_PROXY', 'ftp_proxy',
-                  'FTPS_PROXY', 'ftps_proxy',
-                  'NO_PROXY', 'no_proxy',
-                  'ALL_PROXY', 'all_proxy',
-                  'GIT_PROXY_COMMAND',
-                  'GIT_SSH',
-                  'GIT_SSL_CAINFO',
-                  'GIT_SMART_HTTP',
-                  'SSH_AUTH_SOCK', 'SSH_AGENT_PID',
-                  'SOCKS5_USER', 'SOCKS5_PASSWD',
-                  'DBUS_SESSION_BUS_ADDRESS',
-                  'P4CONFIG']
+    exportvars = (d.getVar("FETCH_ENV_WHITELIST") or "").split()
 
     if not cleanup:
         cleanup = []
bitbake/lib/bb/utils.py +5 -0
index 6ba1d2a376..6676584845 100644
@@ -585,6 +585,7 @@ def preserved_envvars():
         'BB_PRESERVE_ENV',
         'BB_ENV_WHITELIST',
         'BB_ENV_EXTRAWHITE',
+        'FETCH_ENV_WHITELIST',
     ]
     return v + preserved_envvars_exported()
 
@@ -630,6 +631,10 @@ def approved_variables():
         approved.extend(os.environ['BB_ENV_EXTRAWHITE'].split())
         if 'BB_ENV_EXTRAWHITE' not in approved:
             approved.extend(['BB_ENV_EXTRAWHITE'])
+    if 'FETCH_ENV_WHITELIST' in os.environ:
+        approved.extend(os.environ['FETCH_ENV_WHITELIST'].split())
+        if 'FETCH_ENV_WHITELIST' not in approved:
+            approved.extend(['FETCH_ENV_WHITELIST'])
     return approved
 
 def clean_environment():
scripts/oe-buildenv-internal +7 -0
index e0d920f2fc..39aa2f2444 100755
@@ -106,6 +106,13 @@ unset BITBAKEDIR newpath
 export BUILDDIR
 export PATH
 
+FETCH_ENV_WHITELIST="HOME PATH HTTP_PROXY http_proxy HTTPS_PROXY https_proxy \
+FTP_PROXY ftp_proxy FTPS_PROXY ftps_proxy NO_PROXY no_proxy ALL_PROXY all_proxy \
+GIT_PROXY_COMMAND GIT_SSH GIT_SSL_CAINFO GIT_SMART_HTTP SSH_AUTH_SOCK SSH_AGENT_PID \
+SOCKS5_USER SOCKS5_PASSWD DBUS_SESSION_BUS_ADDRESS P4CONFIG"
+
+export FETCH_ENV_WHITELIST
+
 BB_ENV_EXTRAWHITE_OE="MACHINE DISTRO TCMODE TCLIBC HTTP_PROXY http_proxy \
 HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy FTPS_PROXY ftps_proxy ALL_PROXY \
 all_proxy NO_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY \

--
2.20.1

#1 lore raw
Quoted 17 lines · On Fri, 2021-07-09 at 09:44 +0800, [email protected] wrote:

From d228100b7551b2327d66072b03f2a809744e8f52 Mon Sep 17 00:00:00 2001
From: Mingrui Ren <[email protected]>
Date: Thu, 8 Jul 2021 23:44:53 +0800
Subject: [PATCH] Add a new variable:'FETCH_ENV_WHITELIST'

The enviroment variables used by Fetcher are hard-coded, and are obtained from HOST env instead of bitbake datastore.
This patch adds a new variable 'FETCH_ENV_WHITELIST' similar to 'BB_ENV_EXTRAWHITE', trying to fix the problems above.

---
The situation I encountered was that I tried to use a custom git-proxy by setting GIT_PROXY_COMMAND and SOCK_PROXY enviroment variables. However, I found that even if I set BB_ENV_EXTRAWHITE, the SOCK_PROXY variable does not work.

Quoted 4 lines
 bitbake/lib/bb/fetch2/__init__.py | 16 +---------------
 bitbake/lib/bb/utils.py           |  5 +++++
 scripts/oe-buildenv-internal      |  7 +++++++
 3 files changed, 13 insertions(+), 15 deletions(-)

The patch looks like it was generated against poky since that script is in openembedded-core, not bitbake.

For the patch itself, I'd much rather add something to extend the list rather than replace it entirely which would make backwards compatibility easier.

There is also an issue with the variable naming. We have adopted a policy around inclusive langauage so new variables need to be mindful of that. We're in the process of trying to work out what to do with some of our older variables names.

https://wiki.yoctoproject.org/wiki/Inclusive_language

Whatever we do to address this issue really needs to match how we fix the existing BB_ENV_EXTRAWHITE.

Also, since fetching happens much later, we should be able to set this variable from config such as bitbake.conf or local.conf and there shouldn't be any need to export it to the environment.

Cheers,

Richard

#2 lore raw

Thanks for you reply,

Quoted 9 lines · Richard Purdie

On Fri, 2021-07-09 at 09:44 +0800, [email protected] wrote:
The patch looks like it was generated against poky since that script is in openembedded-core, not bitbake.
Whatever we do to address this issue really needs to match how we fix the existing BB_ENV_EXTRAWHITE.

Also, since fetching happens much later, we should be able to set this variable from config such as bitbake.conf or local.conf and there shouldn't be any need to export it to the environment.

Yes, BB_ENV_EXTRAWHITE must be modified to solve this problem gracefully, it seems that needs patching both of poky and bitbake.
Should I send an e-mail to both poky and bitbake mailling list?

Quoted 3 lines

For the patch itself, I'd much rather add something to extend the list rather than replace it entirely which would make backwards compatibility easier.

About backwards compatibility, I have considered it, But I'm not sure if you prefer cleaner code or backward compatibility.

For example:

FETCH_ENV_WHITELIST = "HOME PATH HTTP_PROXY http_proxy ..." exportvars = (d.getVar("FETCH_ENV_WHITELIST") or "").split()

This code is cleaner, and better for forwards compatibility.

FETCH_ENV_WHITELIST = ""
exportvars = (d.getVar("FETCH_ENV_WHITELIST") or "").split().extend "HOME PATH HTTP_PROXY http_proxy ..."

This code is better for backwards compatibility.

Quoted 6 lines

There is also an issue with the variable naming. We have adopted a policy around inclusive langauage so new variables need to be mindful of that. We're in the process of trying to work out what to do with some of our older variables names.

https://wiki.yoctoproject.org/wiki/Inclusive_language

I'll fix it.

thanks,
Mingrui Ren