Add a new variable:'FETCH_ENV_WHITELIST'
PATCH 1/1 Add a new variable:'FETCH_ENV_WHITELIST'
From d228100b7551b2327d66072b03f2a809744e8f52 Mon Sep 17 00:00:00 2001
From: Mingrui Ren <[email protected]>
Date: Thu, 8 Jul 2021 23:44:53 +0800
Subject: [PATCH] Add a new variable:'FETCH_ENV_WHITELIST'
The enviroment variables used by Fetcher are hard-coded, and are obtained from HOST env instead of bitbake datastore.
This patch adds a new variable 'FETCH_ENV_WHITELIST' similar to 'BB_ENV_EXTRAWHITE', trying to fix the problems above.
---
The situation I encountered was that I tried to use a custom git-proxy by setting GIT_PROXY_COMMAND and SOCK_PROXY enviroment variables. However, I found that even if I set BB_ENV_EXTRAWHITE, the SOCK_PROXY variable does not work.
bitbake/lib/bb/fetch2/__init__.py | 16 +--------------- bitbake/lib/bb/utils.py | 5 +++++ scripts/oe-buildenv-internal | 7 +++++++ 3 files changed, 13 insertions(+), 15 deletions(-)
bitbake/lib/bb/fetch2/__init__.py +1 -15
@@ -820,21 +820,7 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None): # rather than host provided # Also include some other variables. # FIXME: Should really include all export varaiables? - exportvars = ['HOME', 'PATH', - 'HTTP_PROXY', 'http_proxy', - 'HTTPS_PROXY', 'https_proxy', - 'FTP_PROXY', 'ftp_proxy', - 'FTPS_PROXY', 'ftps_proxy', - 'NO_PROXY', 'no_proxy', - 'ALL_PROXY', 'all_proxy', - 'GIT_PROXY_COMMAND', - 'GIT_SSH', - 'GIT_SSL_CAINFO', - 'GIT_SMART_HTTP', - 'SSH_AUTH_SOCK', 'SSH_AGENT_PID', - 'SOCKS5_USER', 'SOCKS5_PASSWD', - 'DBUS_SESSION_BUS_ADDRESS', - 'P4CONFIG'] + exportvars = (d.getVar("FETCH_ENV_WHITELIST") or "").split() if not cleanup: cleanup = []
bitbake/lib/bb/utils.py +5 -0
@@ -585,6 +585,7 @@ def preserved_envvars(): 'BB_PRESERVE_ENV', 'BB_ENV_WHITELIST', 'BB_ENV_EXTRAWHITE', + 'FETCH_ENV_WHITELIST', ] return v + preserved_envvars_exported()
@@ -630,6 +631,10 @@ def approved_variables(): approved.extend(os.environ['BB_ENV_EXTRAWHITE'].split()) if 'BB_ENV_EXTRAWHITE' not in approved: approved.extend(['BB_ENV_EXTRAWHITE']) + if 'FETCH_ENV_WHITELIST' in os.environ: + approved.extend(os.environ['FETCH_ENV_WHITELIST'].split()) + if 'FETCH_ENV_WHITELIST' not in approved: + approved.extend(['FETCH_ENV_WHITELIST']) return approved def clean_environment():
scripts/oe-buildenv-internal +7 -0
@@ -106,6 +106,13 @@ unset BITBAKEDIR newpath export BUILDDIR export PATH +FETCH_ENV_WHITELIST="HOME PATH HTTP_PROXY http_proxy HTTPS_PROXY https_proxy \ +FTP_PROXY ftp_proxy FTPS_PROXY ftps_proxy NO_PROXY no_proxy ALL_PROXY all_proxy \ +GIT_PROXY_COMMAND GIT_SSH GIT_SSL_CAINFO GIT_SMART_HTTP SSH_AUTH_SOCK SSH_AGENT_PID \ +SOCKS5_USER SOCKS5_PASSWD DBUS_SESSION_BUS_ADDRESS P4CONFIG" + +export FETCH_ENV_WHITELIST + BB_ENV_EXTRAWHITE_OE="MACHINE DISTRO TCMODE TCLIBC HTTP_PROXY http_proxy \ HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy FTPS_PROXY ftps_proxy ALL_PROXY \ all_proxy NO_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY \
--
2.20.1
Quoted 17 lines · On Fri, 2021-07-09 at 09:44 +0800, [email protected] wrote:
From
d228100b7551b2327d66072b03f2a809744e8f52Mon Sep 17 00:00:00 2001
From: Mingrui Ren <[email protected]>
Date: Thu, 8 Jul 2021 23:44:53 +0800
Subject: [PATCH] Add a new variable:'FETCH_ENV_WHITELIST'The enviroment variables used by Fetcher are hard-coded, and are obtained from HOST env instead of bitbake datastore.
This patch adds a new variable 'FETCH_ENV_WHITELIST' similar to 'BB_ENV_EXTRAWHITE', trying to fix the problems above.Signed-off-by Mingrui Ren <[email protected]>---
The situation I encountered was that I tried to use a custom git-proxy by settingGIT_PROXY_COMMANDandSOCK_PROXYenviroment variables. However, I found that even if I setBB_ENV_EXTRAWHITE, theSOCK_PROXYvariable does not work.
Quoted 4 lines
bitbake/lib/bb/fetch2/__init__.py | 16 +--------------- bitbake/lib/bb/utils.py | 5 +++++ scripts/oe-buildenv-internal | 7 +++++++ 3 files changed, 13 insertions(+), 15 deletions(-)
The patch looks like it was generated against poky since that script is in openembedded-core, not bitbake.
For the patch itself, I'd much rather add something to extend the list rather than replace it entirely which would make backwards compatibility easier.
There is also an issue with the variable naming. We have adopted a policy around inclusive langauage so new variables need to be mindful of that. We're in the process of trying to work out what to do with some of our older variables names.
https://wiki.yoctoproject.org/wiki/Inclusive_language
Whatever we do to address this issue really needs to match how we fix the existing BB_ENV_EXTRAWHITE.
Also, since fetching happens much later, we should be able to set this variable from config such as bitbake.conf or local.conf and there shouldn't be any need to export it to the environment.
Cheers,
Richard
Thanks for you reply,
Quoted 9 lines · Richard Purdie
On Fri, 2021-07-09 at 09:44 +0800, [email protected] wrote:
The patch looks like it was generated against poky since that script is in openembedded-core, not bitbake.
Whatever we do to address this issue really needs to match how we fix the existingBB_ENV_EXTRAWHITE.Also, since fetching happens much later, we should be able to set this variable from config such as bitbake.conf or local.conf and there shouldn't be any need to export it to the environment.
Yes, BB_ENV_EXTRAWHITE must be modified to solve this problem gracefully, it seems that needs patching both of poky and bitbake.
Should I send an e-mail to both poky and bitbake mailling list?
Quoted 3 lines
For the patch itself, I'd much rather add something to extend the list rather than replace it entirely which would make backwards compatibility easier.
About backwards compatibility, I have considered it, But I'm not sure if you prefer cleaner code or backward compatibility.
For example:
FETCH_ENV_WHITELIST = "HOME PATH HTTP_PROXY http_proxy ..." exportvars = (d.getVar("FETCH_ENV_WHITELIST") or "").split()
This code is cleaner, and better for forwards compatibility.
FETCH_ENV_WHITELIST = ""
exportvars = (d.getVar("FETCH_ENV_WHITELIST") or "").split().extend "HOME PATH HTTP_PROXY http_proxy ..."
This code is better for backwards compatibility.
Quoted 6 lines
There is also an issue with the variable naming. We have adopted a policy around inclusive langauage so new variables need to be mindful of that. We're in the process of trying to work out what to do with some of our older variables names.
I'll fix it.
thanks,
Mingrui Ren