Git fetcher and proxy handling updates
cover Git fetcher and proxy handling updates
The current git fetcher environment and proxy handling between bitbake, base.bbclass, and oe-buildenv-internal has become stale and is largely non-functional. The crux of the problem is that GIT_CONFIG is no longer honored by anything other than the git-config command, rendering the GIT_CORE_CONFIG and the assocaited proxy handling routines obsolete.
The GIT_PROXY_COMMAND, ALL_PROXY, and NO_PROXY environment variables are more than adequate for storing the required information to manage proxies with a git helper script.
These patches series aggressively removes the old stuff, fixes up some missing environment settings, and adds a new git proxy handling script.
Note: These patches DO NOT address the bitbake gui builder.py references to git
proxy settings!
I fully expect this series to be a bit controversial and that a few iterations will be required to get it acceptable to everyone.
Finally, I felt this series was best viewed as a whole, so I'm sending it to poky AND oe-core and CC'ing bitbake-devel as appropriate. Anything else seemed too likely to result in confusion, missing context, and lots and lots of redundant discussion.
The following changes since commit 7e9cb437b1b720e656f4a33f73c07c5dc7356a2c:
dropbear: fix RPROVIDES (2013-02-04 23:31:02 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib dvhart/gitproxy
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=dvhart/gitproxy
Darren Hart (8):
bitbake: fetch2: Print the complete SRCREV variable name when INVALID
bitbake: fetch2: Export upper and lower case environment variables
bitbake: fetch2: Remove broken git variables from the environment
oe-buildenv-internal: Remove GIT variables from BB_ENV_EXTRAWHITE
oe-buildenv-internal: Add upper and lower case proxy vars to
BB_ENV_EXTRAWHITE
base.bbclass: Remove generate_git_config() oe-git-proxy.sh: Add a new comprehensive git proxy script git proxy: Document usage of oe-git-proxy.sh and remove alternatives
bitbake/lib/bb/fetch2/__init__.py | 22 +- meta-yocto/conf/site.conf.sample | 35 +- meta/classes/base.bbclass | 21 - scripts/oe-buildenv-internal | 2 +- scripts/oe-git-proxy-command | 10 - scripts/oe-git-proxy-socks-command | 23 - scripts/oe-git-proxy-socks.c | 2982 ------------------------------------ scripts/oe-git-proxy.sh | 125 ++ 8 files changed, 150 insertions(+), 3070 deletions(-) delete mode 100755 scripts/oe-git-proxy-command delete mode 100755 scripts/oe-git-proxy-socks-command delete mode 100755 scripts/oe-git-proxy-socks.c create mode 100755 scripts/oe-git-proxy.sh
Signature · 1 line
1.7.5.4
PATCH 1/8 bitbake: fetch2: Print the complete SRCREV variable name when INVALID
If a particular SRCREV (say for a particular branch) is missing, the fetcher will currently just report an obtuse error about the "SRCREV" being invalid. If there is more information is to be had (say from name, i.e. branch, and pn) then display that as well.
The new error looks something like this:
ERROR: ExpansionError during parsing /home/dvhart/source/poky/meta/recipes-kernel/linux/linux-yocto_3.4.bb: Failure expanding variable do_patch: ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure for URL: 'git://otcgit.jf.intel.com/dvhart/linux-yocto-minnow-3.4.git;protocol=git;nocheckout=1;branch=standard/minnow,meta,emgd-1.14;name=machine,meta,emgd'. Please set SRCREV_emgd_pn-linux-yocto to a valid value
Note the variable listed as invalid is "SRCREV_emgd_pn-linux-yocto", making it explicit what is wrong.
--- bitbake/lib/bb/fetch2/__init__.py | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-)
bitbake/lib/bb/fetch2/__init__.py +5 -2
@@ -650,11 +650,14 @@ def srcrev_internal_helper(ud, d, name): if not rev: rev = d.getVar("SRCREV_%s" % name, True) if not rev: - rev = d.getVar("SRCREV_pn-%s" % pn, True) + rev = d.getVar("SRCREV_pn-%s" % pn, True) if not rev: rev = d.getVar("SRCREV", True) if rev == "INVALID": - raise FetchError("Please set SRCREV to a valid value", ud.url) + var = "SRCREV_pn-%s" % pn + if name != '': + var = "SRCREV_%s_pn-%s" % (name, pn) + raise FetchError("Please set %s to a valid value" % var, ud.url) if rev == "AUTOINC": rev = ud.method.latest_revision(ud.url, ud, d, name)
Signature · 1 line
1.7.5.4
PATCH 2/8 bitbake: fetch2: Export upper and lower case environment variables
In reply to Darren HartApplications are inconsistent in their use of upper and lower case proxy variables. Curl, for example, specifies NO_PROXY (not no_proxy) in the man page (changed in 2009 [1]). Avoid proxy issues by ensuring both the upper and lower case versions of each proxy variable is available in the environment for the fetcher commands.
Add FTPS_PROXY and ftps_proxy to the list as well.
1. http://curl.haxx.se/mail/tracker-2009-04/0012.html
--- bitbake/lib/bb/fetch2/__init__.py | 9 +++++++-- 1 files changed, 7 insertions(+), 2 deletions(-)
bitbake/lib/bb/fetch2/__init__.py +7 -2
@@ -459,8 +459,13 @@ def runfetchcmd(cmd, d, quiet = False, cleanup = []): # Also include some other variables. # FIXME: Should really include all export varaiables? exportvars = ['PATH', 'GIT_PROXY_COMMAND', 'GIT_PROXY_HOST', - 'GIT_PROXY_PORT', 'GIT_CONFIG', 'http_proxy', 'ftp_proxy', - 'https_proxy', 'no_proxy', 'ALL_PROXY', 'all_proxy', + 'GIT_PROXY_PORT', 'GIT_CONFIG', + 'HTTP_PROXY', 'http_proxy', + 'HTTPS_PROXY', 'https_proxy', + 'FTP_PROXY', 'ftp_proxy', + 'FTPS_PROXY', 'ftps_proxy', + 'NO_PROXY', 'no_proxy', + 'ALL_PROXY', 'all_proxy', 'SSH_AUTH_SOCK', 'SSH_AGENT_PID', 'HOME', 'GIT_PROXY_IGNORE', 'SOCKS5_USER', 'SOCKS5_PASSWD']
Signature · 1 line
1.7.5.4
PATCH 3/8 bitbake: fetch2: Remove broken git variables from the environment
In reply to Darren HartThe following variables perform no function outside of bitbake:
GIT_CONFIG
GIT_PROXY_HOST
GIT_PROXY_PORT
GIT_PROXY_IGNORE
GIT_CONFIG only affects the git-config command which is not relevant to the fetcher. This was previously used with the OE GIT_CORE_CONFIG variable which would provide a basic git config to use instead of the user's config. This usage was deprecated by git for over a year now:
http://git.661346.n2.nabble.com/Overriding-gitconfig-using-GIT-CONFIG-td6680977.html
GIT_PROXY_HOST and GIT_PROXY_PORT are not used by git.
GIT_PROXY_IGNORE was an OE construct used to create the custom git config and had no meaning outside of the OE environment. It is not used by git.
Remove these variables from the fetcher environment.
Users wishing to configure git to work with a proxy should define the GIT_PROXY_COMMAND environment variable to use an external script. NO_PROXY can be used within this script to skip the proxy for certain hosts.
--- bitbake/lib/bb/fetch2/__init__.py | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-)
bitbake/lib/bb/fetch2/__init__.py +4 -4
@@ -458,16 +458,16 @@ def runfetchcmd(cmd, d, quiet = False, cleanup = []): # rather than host provided # Also include some other variables. # FIXME: Should really include all export varaiables? - exportvars = ['PATH', 'GIT_PROXY_COMMAND', 'GIT_PROXY_HOST', - 'GIT_PROXY_PORT', 'GIT_CONFIG', + exportvars = ['HOME', 'PATH', 'HTTP_PROXY', 'http_proxy', 'HTTPS_PROXY', 'https_proxy', 'FTP_PROXY', 'ftp_proxy', 'FTPS_PROXY', 'ftps_proxy', 'NO_PROXY', 'no_proxy', 'ALL_PROXY', 'all_proxy', - 'SSH_AUTH_SOCK', 'SSH_AGENT_PID', 'HOME', - 'GIT_PROXY_IGNORE', 'SOCKS5_USER', 'SOCKS5_PASSWD'] + 'GIT_PROXY_COMMAND', + 'SSH_AUTH_SOCK', 'SSH_AGENT_PID', + 'SOCKS5_USER', 'SOCKS5_PASSWD'] for var in exportvars: val = d.getVar(var, True)
Signature · 1 line
1.7.5.4
PATCH 4/8 oe-buildenv-internal: Remove GIT variables from BB_ENV_EXTRAWHITE
In reply to Darren HartThe following variables perform no function outside of bitbake:
GIT_CONFIG
GIT_PROXY_HOST
GIT_PROXY_PORT
GIT_PROXY_IGNORE
GIT_CONFIG only affects the git-config command which is not relevant to the fetcher. This was previously used with the OE GIT_CORE_CONFIG variable which would provide a basic git config to use instead of the user's config. This usage was deprecated by git for over a year now:
http://git.661346.n2.nabble.com/Overriding-gitconfig-using-GIT-CONFIG-td6680977
GIT_PROXY_HOST and GIT_PROXY_PORT are not used by git.
GIT_PROXY_IGNORE was an OE construct used to create the custom git config and had no meaning outside of the OE environment. It is not used by git.
Remove these variables from the fetcher environment.
Users wishing to configure git to work with a proxy should define the GIT_PROXY_COMMAND environment variable to use an external script. NO_PROXY can be used within this script to skip the proxy for certain hosts.
--- scripts/oe-buildenv-internal | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
scripts/oe-buildenv-internal +1 -1
@@ -80,4 +80,4 @@ unset BITBAKEDIR # Used by the runqemu script export BUILDDIR export PATH -export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER SCREENDIR" +export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy NO_PROXY SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND SOCKS5_PASSWD SOCKS5_USER SCREENDIR"
Signature · 1 line
1.7.5.4
PATCH 5/8 oe-buildenv-internal: Add upper and lower case proxy vars to BB_ENV_EXTRAWHITE
In reply to Darren HartApplications are inconsistent in their use of upper and lower case proxy variables. Curl, for example, specifies NO_PROXY (not no_proxy) in the man page (changed in 2009 [1]). Avoid proxy issues by ensuring both the upper and lower case versions of each proxy variable are available in the environment for the fetcher commands.
Add FTPS_PROXY and ftps_proxy to the list as well.
1. http://curl.haxx.se/mail/tracker-2009-04/0012.html
--- scripts/oe-buildenv-internal | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
scripts/oe-buildenv-internal +1 -1
@@ -80,4 +80,4 @@ unset BITBAKEDIR # Used by the runqemu script export BUILDDIR export PATH -export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy NO_PROXY SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND SOCKS5_PASSWD SOCKS5_USER SCREENDIR" +export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC HTTP_PROXY http_proxy HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy FTPS_PROXY ftps_proxy ALL_PROXY all_proxy NO_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND SOCKS5_PASSWD SOCKS5_USER SCREENDIR"
Signature · 1 line
1.7.5.4
PATCH 6/8 base.bbclass: Remove generate_git_config()
In reply to Darren Hartgit no longer supports the use of GIT_CONFIG which defeats the purpose of GIT_CORE_CONFIG and the generate_git_config() function. Remove it.
--- meta/classes/base.bbclass | 21 --------------------- 1 files changed, 0 insertions(+), 21 deletions(-)
meta/classes/base.bbclass +0 -21
@@ -133,26 +133,6 @@ python base_do_unpack() { raise bb.build.FuncFailed(e) } -GIT_CONFIG_PATH = "${STAGING_DIR_NATIVE}/etc" -GIT_CONFIG = "${GIT_CONFIG_PATH}/gitconfig" - -def generate_git_config(e): - if e.data.getVar('GIT_CORE_CONFIG', True): - gitconfig_path = e.data.getVar('GIT_CONFIG', True) - proxy_command = " gitProxy = %s\n" % e.data.getVar('OE_GIT_PROXY_COMMAND', True) - - bb.mkdirhier(e.data.expand("${GIT_CONFIG_PATH}")) - if (os.path.exists(gitconfig_path)): - os.remove(gitconfig_path) - - f = open(gitconfig_path, 'w') - f.write("[core]\n") - ignore_hosts = e.data.getVar('GIT_PROXY_IGNORE', True).split() - for ignore_host in ignore_hosts: - f.write(" gitProxy = none for %s\n" % ignore_host) - f.write(proxy_command) - f.close - def pkgarch_mapping(d): # Compatibility mappings of TUNE_PKGARCH (opt in) if d.getVar("PKGARCHCOMPAT_ARMV7A", True):
@@ -311,7 +291,6 @@ addhandler base_eventhandler python base_eventhandler() { if isinstance(e, bb.event.ConfigParsed): e.data.setVar('BB_VERSION', bb.__version__) - generate_git_config(e) pkgarch_mapping(e.data) preferred_ml_updates(e.data) e.data.appendVar('DISTRO_FEATURES', oe.utils.features_backfill("DISTRO_FEATURES", e.data))
Signature · 1 line
1.7.5.4
PATCH 7/8 oe-git-proxy.sh: Add a new comprehensive git proxy script
In reply to Darren Hartoe-git-proxy.sh is a simple tool to be used via GIT_PROXY_COMMAND. It uses BSD netcat to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It is known to work with both bash and dash shells.
--- scripts/oe-git-proxy.sh | 125 +++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 125 insertions(+), 0 deletions(-) create mode 100755 scripts/oe-git-proxy.sh
scripts/oe-git-proxy.sh new file +125 -0
@@ -0,0 +1,125 @@ +#!/bin/sh + +# oe-git-proxy.sh is a simple tool to be via GIT_PROXY_COMMAND. It uses BSD netcat +# to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the +# proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for +# a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks +# (192.168.1.0/24). It is known to work with both bash and dash shells. +# +# BSD netcat is provided by netcat-openbsd on Ubuntu and nc on Fedora. +# +# Example ALL_PROXY values: +# ALL_PROXY=socks://socks.example.com:1080 +# ALL_PROXY=https://proxy.example.com:8080 +# +# Copyright (c) 2013, Intel Corporation. +# All rights reserved. +# +# AUTHORS +# Darren Hart <[email protected]> + +# Locate the netcat binary +NC=$(which nc 2>/dev/null) +if [ $? -ne 0 ]; then + echo "ERROR: nc binary not in PATH" + exit 1 +fi +METHOD="" + +# Test for a valid IPV4 quad with optional bitmask +valid_ipv4() { + echo $1 | egrep -q "^([1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])){3}(/(3[0-2]|[1-2]?[0-9]))?$" + return $? +} + +# Convert an IPV4 address into a 32bit integer +ipv4_val() { + SHIFT=24 + VAL=0 + for B in $(echo "$1" | sed -e 's/\./ /g'); do + VAL=$(($VAL+$(($B<<$SHIFT)))) + SHIFT=$(($SHIFT-8)) + done + echo "$VAL" +} + +# Determine if two IPs are equivalent, or if the CIDR contains the IP +match_ipv4() { + CIDR=$1 + IP=$2 + + if [ -z "${IP%%$CIDR}" ]; then + return 0 + fi + + # Determine the mask bitlength + BITS=${CIDR##*/} + if [ -z "$BITS" ]; then + return 1 + fi + + IPVAL=$(ipv4_val $IP) + IP2VAL=$(ipv4_val ${CIDR%%/*}) + + # OR in the unmasked bits + for i in $(seq 0 $((32-$BITS))); do + IP2VAL=$(($IP2VAL|$((1<<$i)))) + IPVAL=$(($IPVAL|$((1<<$i)))) + done + + echo "$IPVAL vs. $IP2VAL" + if [ $IPVAL -eq $IP2VAL ]; then + return 0 + fi + return 1 +} + +# Test to see if GLOB matches HOST +match_host() { + HOST=$1 + GLOB=$2 + + if [ -z "${HOST%%$GLOB}" ]; then + return 0 + fi + + # Match by netmask + if valid_ipv4 $GLOB; then + HOST_IP=$(gethostip -d $HOST) + if valid_ipv4 $HOST_IP; then + match_ipv4 $GLOB $HOST_IP + if [ $? -eq 0 ]; then + return 0 + fi + fi + fi + + return 1 +} + +# If no proxy is set, just connect directly +if [ -z "$ALL_PROXY" ]; then + $NC -X connect $* + exit +fi + +# Connect directly to hosts in NO_PROXY +for H in $(echo "$NO_PROXY" | sed -e 's/,/ /g'); do + if match_host $1 $H; then + METHOD="-X connect" + break + fi +done + +if [ -z "$METHOD" ]; then + # strip the protocol and the trailing slash + PROTO=$(echo $ALL_PROXY | sed -e 's/\([^:]*\):\/\/.*/\1/') + PROXY=$(echo $ALL_PROXY | sed -e 's/.*:\/\/\([^:]*:[0-9]*\).*/\1/') + if [ "$PROTO" = "socks" ]; then + METHOD="-X 5 -x $PROXY" + elif [ "$PROTO" = "https" ]; then + METHOD="-X connect -x $PROXY" + fi +fi + +$NC $METHOD $*
Signature · 1 line
1.7.5.4
PATCH 8/8 git proxy: Document usage of oe-git-proxy.sh and remove alternatives
In reply to Darren HartThe new oe-git-proxy.sh should address all git proxying needs. Document it's usage in meta-yocto/conf/site.conf.sample and remove the other implementations.
--- meta-yocto/conf/site.conf.sample | 35 +- scripts/oe-git-proxy-command | 10 - scripts/oe-git-proxy-socks-command | 23 - scripts/oe-git-proxy-socks.c | 2982 ------------------------------------ 4 files changed, 9 insertions(+), 3041 deletions(-) delete mode 100755 scripts/oe-git-proxy-command delete mode 100755 scripts/oe-git-proxy-socks-command delete mode 100755 scripts/oe-git-proxy-socks.c
meta-yocto/conf/site.conf.sample +9 -26
@@ -16,32 +16,15 @@ SCONF_VERSION = "1" #http-proxy-port = 81 # -# Uncomment to cause git to use the proxy host specificed -# although this only works for http -#GIT_PROXY_HOST = "proxy.example.com" -#GIT_PROXY_PORT = "81" -#export GIT_PROXY_COMMAND = "${COREBASE}/scripts/oe-git-proxy-command" - -# Set to yes to have a gitconfig generated for handling proxies; you -# might not want this if you have all that set in your global git -# configuration. If you don't enable it, the rest of the entries -# (_PROXY_IGNORE, etc) don't really work that well -#GIT_CORE_CONFIG = "Yes" - -# Space separate list of hosts to ignore for GIT proxy -#GIT_PROXY_IGNORE = "host.server.com another.server.com" - -# If SOCKS is available run the following command to comple a simple transport -# gcc scripts/oe-git-proxy-socks.c -o oe-git-proxy-socks -# and then share that binary somewhere in PATH, then use the following settings -#GIT_PROXY_HOST = "proxy.example.com" -#GIT_PROXY_PORT = "81" - -# GIT_PROXY_COMMAND is used by git to override all proxy settings from -# configuration files, so we prefix OE_ to avoid breaking havoc on the -# generated (or local) gitconfig's. -#OE_GIT_PROXY_COMMAND = "${COREBASE}/scripts/oe-git-proxy-socks-command" - +# To use git with a proxy, you must use an external git proxy command, such as +# the one provided by scripts/oe-git-proxy.sh. To use this script, copy it to +# your PATH and uncomment the following: +#GIT_PROXY_COMMAND="oe-git-proxy.sh" +#ALL_PROXY="socks://socks.example.com:1080" +#or +#ALL_PROXY="https://proxy.example.com:8080" +# If you wish to use certain hosts without the proxy, specify them in NO_PROXY. +# See the script for details on syntax. # Uncomment this to use a shared download directory #DL_DIR = "/some/shared/download/directory/"
scripts/oe-git-proxy-command deleted +0 -10
@@ -1,10 +0,0 @@ -#! /bin/bash - -(echo "CONNECT $1:$2 HTTP/1.0"; - echo; - cat ) | nc $GIT_PROXY_HOST $GIT_PROXY_PORT | - -(read a; - read a; - read a; - cat )
scripts/oe-git-proxy-socks-command deleted +0 -23
@@ -1,23 +0,0 @@ -#! /bin/bash -SCRIPTDIR=`dirname $0` -# Check oe-git-proxy-socks exists -PROXYSOCKS=`which oe-git-proxy-socks 2> /dev/null` -if [ -z "$PROXYSOCKS" -a -e "$SCRIPTDIR/oe-git-proxy-socks.c" ]; then - # If not try and build it - gcc $SCRIPTDIR/oe-git-proxy-socks.c -o $SCRIPTDIR/oe-git-proxy-socks -fi -PROXYSOCKS=`which oe-git-proxy-socks 2> /dev/null` -if [ ! -x "$PROXYSOCKS" ]; then - # If that fails, we can see if netcat (nc) is available - NETCAT=`which nc 2> /dev/null` - if [ ! -x "$NETCAT" ]; then - # If that fails, explain to the user - echo "Unable to find oe-git-proxy-socks. This is usually created with the command" - echo "'gcc scripts/oe-git-proxy-socks.c -o scripts/oe-git-proxy-socks' which we tried" - echo "but it doesn't seem to have worked. Please compile the binary manually." - echo "Alternativly, install nc (netcat) on this machine." - exit 1 - fi - exec $NETCAT -x $GIT_PROXY_HOST:$GIT_PROXY_PORT "$@" -fi -oe-git-proxy-socks -S $GIT_PROXY_HOST:$GIT_PROXY_PORT $@
scripts/oe-git-proxy-socks.c deleted +0 -2982
2,925 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
1.7.5.4
Part of the diff is not shown here. View the raw message.
Re: [PATCH 7/8] oe-git-proxy.sh: Add a new comprehensive git proxy script
In reply to Darren Hartscripts/oe-git-proxy.sh line 102
Quoted 1 line · Darren Hart
scripts/oe-git-proxy.sh new file +1
-0+ $NC -X connect $*
why '$*' but not '"$@*"'?
Quoted 1 line
+$NC $METHOD $*
ditto
Enrico
Re: [PATCH 7/8] oe-git-proxy.sh: Add a new comprehensive git proxy script
Quoted 4 lines · Enrico Scholz
Darren Hart <dvhart-VuQAYsv1563Yd54FQh9/[email protected]> writes:
Quoted 8 lines
scripts/oe-git-proxy.sh line 102
Quoted 1 line
scripts/oe-git-proxy.sh new file +1
-0+ $NC -X connect $*why '$*' but not '"$@*"'?
Quoted 1 line
+$NC $METHOD $*
ditto
I'm not familiar with $@*
As for $* versus $@, the issue is how the arguments are presented. $* as a single word, $@ each argument is quoted separately. I believe I ran into issues with $@. I haven't had any trouble with $*. Is there a particular use case where you can see this failing as is?
Signature · 3 lines
Darren Hart Intel Open Source Technology Center Yocto Project - Technical Lead - Linux Kernel
Re: [PATCH 7/8] oe-git-proxy.sh: Add a new comprehensive git proxy script
Quoted 5 lines · Darren Hart
Quoted 4 lines
scripts/oe-git-proxy.sh line 102
Quoted 1 line
scripts/oe-git-proxy.sh new file +1
-0+ $NC -X connect $*why '$*' but not '"$@*"'?
I'm not familiar with $@*
sorry... I meant "$@"
Quoted 3 lines
As for $* versus $@, the issue is how the arguments are presented. $* as a single word, $@ each argument is quoted separately. I believe I ran into issues with $@. I haven't had any trouble with $*.
$* is causing trouble all the time because it does not retain whitespaces or empty parameters. There are only very few cases, where $* makes sense.
Quoted 1 line
Is there a particular use case where you can see this failing as is?
"$@" is just the right thing to do in this situation. E.g. when your script is called as
| oe-git-proxy.sh "${HOST}" "${PORT}"
and HOST is undefined due to some reason, you will try to connect to "${PORT}" with $*. The "$@" will cause nc to complain about the broken HOST parameter.
Btw...
| exec $NC $METHOD "$@"
would be the school book implementation for the thing you want to do...
Enrico
Re: [PATCH 7/8] oe-git-proxy.sh: Add a new comprehensive git proxy script
Quoted 33 lines · Enrico Scholz
Quoted 5 lines · Darren Hart
Quoted 4 lines
scripts/oe-git-proxy.sh line 102
Quoted 1 line
scripts/oe-git-proxy.sh new file +1
-0+ $NC -X connect $*why '$*' but not '"$@*"'?
I'm not familiar with $@*
sorry... I meant "$@"
Quoted 3 lines
As for $* versus $@, the issue is how the arguments are presented. $* as a single word, $@ each argument is quoted separately. I believe I ran into issues with $@. I haven't had any trouble with $*.
$* is causing trouble all the time because it does not retain whitespaces or empty parameters. There are only very few cases, where $* makes sense.
Quoted 1 line
Is there a particular use case where you can see this failing as is?
"$@" is just the right thing to do in this situation. E.g. when your script is called as
|
oe-git-proxy.sh"${HOST}" "${PORT}"and HOST is undefined due to some reason, you will try to connect to "${PORT}" with $*. The "$@" will cause nc to complain about the broken HOST parameter.
Btw...
Quoted 7 lines
| exec $NC $METHOD "$@"
would be the school book implementation for the thing you want to do...
Enrico
That all makes sense. When I read up the difference again in the bash documentation I was surprised I had used $*, but thought I had done that dance already. I'll update with "$@" and do some tests.
Thank you for the review and catching that.
Signature · 3 lines
Darren Hart Intel Open Source Technology Center Yocto Project - Technical Lead - Linux Kernel
Re: [PATCH 7/8] oe-git-proxy.sh: Add a new comprehensive git proxy script
Quoted 42 lines · Darren Hart
Quoted 39 lines · Enrico Scholz
Quoted 5 lines · Darren Hart
Quoted 4 lines
Quoted 1 line
+ $NC -X connect $*
why '$*' but not '"$@*"'?
I'm not familiar with $@*
sorry... I meant "$@"
Quoted 3 lines
As for $* versus $@, the issue is how the arguments are presented. $* as a single word, $@ each argument is quoted separately. I believe I ran into issues with $@. I haven't had any trouble with $*.
$* is causing trouble all the time because it does not retain whitespaces or empty parameters. There are only very few cases, where $* makes sense.
Quoted 1 line
Is there a particular use case where you can see this failing as is?
"$@" is just the right thing to do in this situation. E.g. when your script is called as
|
oe-git-proxy.sh"${HOST}" "${PORT}"and HOST is undefined due to some reason, you will try to connect to "${PORT}" with $*. The "$@" will cause nc to complain about the broken HOST parameter.
Btw...
| exec $NC $METHOD "$@"
would be the school book implementation for the thing you want to do...
Quoted 7 lines
Quoted 1 line
Enrico
That all makes sense. When I read up the difference again in the bash documentation I was surprised I had used $*, but thought I had done that dance already. I'll update with "$@" and do some tests.
Thank you for the review and catching that.
Please give it a try in dash as well.
--
Otavio Salvador O.S. Systems
E-mail: [email protected] http://www.ossystems.com.br
Mobile: +55 53 9981-7854 http://projetos.ossystems.com.br
Re: [PATCH 7/8] oe-git-proxy.sh: Add a new comprehensive git proxy script
Quoted 45 lines · Otavio Salvador
Quoted 44 lines · Darren Hart
Quoted 40 lines · Enrico Scholz
Quoted 5 lines · Darren Hart
Quoted 4 lines
Quoted 1 line
+ $NC -X connect $*
why '$*' but not '"$@*"'?
I'm not familiar with $@*
sorry... I meant "$@"
Quoted 3 lines
As for $* versus $@, the issue is how the arguments are presented. $* as a single word, $@ each argument is quoted separately. I believe I ran into issues with $@. I haven't had any trouble with $*.
$* is causing trouble all the time because it does not retain whitespaces or empty parameters. There are only very few cases, where $* makes sense.
Quoted 1 line
Is there a particular use case where you can see this failing as is?
"$@" is just the right thing to do in this situation. E.g. when your script is called as
|
oe-git-proxy.sh"${HOST}" "${PORT}"and HOST is undefined due to some reason, you will try to connect to "${PORT}" with $*. The "$@" will cause nc to complain about the broken HOST parameter.
Btw...
| exec $NC $METHOD "$@"
would be the school book implementation for the thing you want to do...
Enrico
Quoted 7 lines
Quoted 5 lines
That all makes sense. When I read up the difference again in the bash documentation I was surprised I had used $*, but thought I had done that dance already. I'll update with "$@" and do some tests.
Thank you for the review and catching that.
Please give it a try in dash as well.
$@ also works with dash. I have been testing in bash and dash throughout development as well.
Signature · 3 lines
Darren Hart Intel Open Source Technology Center Yocto Project - Technical Lead - Linux Kernel
Re: [PATCH 7/8] oe-git-proxy.sh: Add a new comprehensive git proxy script
In reply to Otavio SalvadorQuoted 1 line · Otavio Salvador
Please give it a try in dash as well.
fwiw, what's the point in writing such scripts plain sh? I guess. every machine where this script is running has /bin/bash and performance is not critical for it. Using '#! /bin/bash' shebang fixes the problem where sh is non bash.
Some constructs in the scripts can be expressed mor efficiently in bash (e.g. the '... | sed' statements, or using arrays for the arguments).
Enrico
Re: [PATCH 7/8] oe-git-proxy.sh: Add a new comprehensive git proxy script
Quoted 11 lines · Enrico Scholz
Quoted 1 line · Otavio Salvador
Please give it a try in dash as well.
fwiw, what's the point in writing such scripts plain sh? I guess. every machine where this script is running has
/bin/bashand performance is not critical for it. Using '#!/bin/bash' shebang fixes the problem where sh is non bash.Some constructs in the scripts can be expressed mor efficiently in bash (e.g. the '... | sed' statements, or using arrays for the arguments).
Indeed, in fact I had to remove the bash parameter expansions (substrings, etc) I had in my original script in order to make it dash-able. I hate dash and personally find it to be completely pointless, but for better or worse, people like it and there is an expectation that scripts run in both dash and bash in OE-core.
If RP tells me a #!/bin/bash is acceptable, I'll restore the bashisms in a heartbeat.
Signature · 3 lines
Darren Hart Intel Open Source Technology Center Yocto Project - Technical Lead - Linux Kernel
Re: [PATCH 7/8] oe-git-proxy.sh: Add a new comprehensive git proxy script
Quoted 15 lines · Darren Hart
Quoted 11 lines · Enrico Scholz
Quoted 1 line · Otavio Salvador
Please give it a try in dash as well.
fwiw, what's the point in writing such scripts plain sh? I guess. every machine where this script is running has
/bin/bashand performance is not critical for it. Using '#!/bin/bash' shebang fixes the problem where sh is non bash.Some constructs in the scripts can be expressed mor efficiently in bash (e.g. the '... | sed' statements, or using arrays for the arguments).
Quoted 8 lines
Indeed, in fact I had to remove the bash parameter expansions (substrings, etc) I had in my original script in order to make it dash-able. I hate dash and personally find it to be completely pointless, but for better or worse, people like it and there is an expectation that scripts run in both dash and bash in OE-core.
If RP tells me a #!/bin/bash is acceptable, I'll restore the bashisms in a heartbeat.
Please drop the .sh than; I'd promptly call it as:
sh oe-git-proxy.sh
And in my system /bin/sh is dash.
--
Otavio Salvador O.S. Systems
E-mail: [email protected] http://www.ossystems.com.br
Mobile: +55 53 9981-7854 http://projetos.ossystems.com.br
Re: [PATCH 7/8] oe-git-proxy.sh: Add a new comprehensive git proxy script
In reply to Darren HartQuoted 14 lines · Darren Hart
Quoted 11 lines · Enrico Scholz
Quoted 1 line · Otavio Salvador
Please give it a try in dash as well.
fwiw, what's the point in writing such scripts plain sh? I guess. every machine where this script is running has
/bin/bashand performance is not critical for it. Using '#!/bin/bash' shebang fixes the problem where sh is non bash.Some constructs in the scripts can be expressed mor efficiently in bash (e.g. the '... | sed' statements, or using arrays for the arguments).
Quoted 8 lines
Indeed, in fact I had to remove the bash parameter expansions (substrings, etc) I had in my original script in order to make it dash-able. I hate dash and personally find it to be completely pointless, but for better or worse, people like it and there is an expectation that scripts run in both dash and bash in OE-core.
If RP tells me a #!/bin/bash is acceptable, I'll restore the bashisms in a heartbeat.
General shell code in metadata should run under /bin/sh. Its fine for an external scripts to use bashisms as long as it says so at the top of the file :)
Cheers,
Richard