Git fetcher and proxy handling updates
v2 cover Git fetcher and proxy handling updates
The current git fetcher environment and proxy handling between bitbake, base.bbclass, and oe-buildenv-internal has become stale and is largely non-functional. The crux of the problem is that GIT_CONFIG is no longer honored by anything other than the git-config command, rendering the GIT_CORE_CONFIG and the assocaited proxy handling routines obsolete.
The GIT_PROXY_COMMAND, ALL_PROXY, and NO_PROXY environment variables are more than adequate for storing the required information to manage proxies with a git helper script.
These patches series aggressively removes the old stuff, fixes up some missing environment settings, and adds a new git proxy handling script.
Note: These patches DO NOT address the bitbake gui builder.py references to git
proxy settings!
I felt this series was best viewed as a whole, so I'm sending it to poky AND oe-core and CC'ing bitbake-devel as appropriate. Anything else seemed too likely to result in confusion, missing context, and lots and lots of redundant discussion.
V2: Updates to the new oe-git-proxy script from feedback received
Split out meta-yocto changes
Split out removal of old scripts
Minor commit log corrections
The following changes since commit 7e9cb437b1b720e656f4a33f73c07c5dc7356a2c:
dropbear: fix RPROVIDES (2013-02-04 23:31:02 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib dvhart/gitproxy
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=dvhart/gitproxy
Darren Hart (9):
bitbake: fetch2: Print the complete SRCREV variable name when INVALID
bitbake: fetch2: Export upper and lower case environment variables
bitbake: fetch2: Remove broken git variables from the environment
oe-buildenv-internal: Remove GIT variables from BB_ENV_EXTRAWHITE
oe-buildenv-internal: Add upper and lower case proxy vars to
BB_ENV_EXTRAWHITE
base.bbclass: Remove generate_git_config()
oe-git-proxy: Add a new comprehensive git proxy script
meta-yocto: Document new oe-git-proxy in site.conf.sample oe-git-proxy*: Remove previous git proxy solutions
bitbake/lib/bb/fetch2/__init__.py | 22 +- meta-yocto/conf/site.conf.sample | 35 +- meta/classes/base.bbclass | 21 - scripts/oe-buildenv-internal | 2 +- scripts/oe-git-proxy | 124 ++ scripts/oe-git-proxy-command | 10 - scripts/oe-git-proxy-socks-command | 23 - scripts/oe-git-proxy-socks.c | 2982 ------------------------------------ 8 files changed, 149 insertions(+), 3070 deletions(-) create mode 100755 scripts/oe-git-proxy delete mode 100755 scripts/oe-git-proxy-command delete mode 100755 scripts/oe-git-proxy-socks-command delete mode 100755 scripts/oe-git-proxy-socks.c
Signature · 1 line
1.7.5.4
PATCH v2 1/9 bitbake: fetch2: Print the complete SRCREV variable name when INVALID
If a particular SRCREV (say for a particular branch) is missing, the fetcher will currently just report an obtuse error about the "SRCREV" being invalid. If there is more information is to be had (say from name, i.e. branch, and pn) then display that as well.
The new error looks something like this:
ERROR: ExpansionError during parsing /home/dvhart/source/poky/meta/recipes-kernel/linux/linux-yocto_3.4.bb: Failure expanding variable do_patch: ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure for URL: 'git://otcgit.jf.intel.com/dvhart/linux-yocto-minnow-3.4.git;protocol=git;nocheckout=1;branch=standard/minnow,meta,emgd-1.14;name=machine,meta,emgd'. Please set SRCREV_emgd_pn-linux-yocto to a valid value
Note the variable listed as invalid is "SRCREV_emgd_pn-linux-yocto", making it explicit what is wrong.
--- bitbake/lib/bb/fetch2/__init__.py | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-)
bitbake/lib/bb/fetch2/__init__.py +5 -2
@@ -650,11 +650,14 @@ def srcrev_internal_helper(ud, d, name): if not rev: rev = d.getVar("SRCREV_%s" % name, True) if not rev: - rev = d.getVar("SRCREV_pn-%s" % pn, True) + rev = d.getVar("SRCREV_pn-%s" % pn, True) if not rev: rev = d.getVar("SRCREV", True) if rev == "INVALID": - raise FetchError("Please set SRCREV to a valid value", ud.url) + var = "SRCREV_pn-%s" % pn + if name != '': + var = "SRCREV_%s_pn-%s" % (name, pn) + raise FetchError("Please set %s to a valid value" % var, ud.url) if rev == "AUTOINC": rev = ud.method.latest_revision(ud.url, ud, d, name)
Signature · 1 line
1.7.5.4
PATCH v2 2/9 bitbake: fetch2: Export upper and lower case environment variables
In reply to Darren HartApplications are inconsistent in their use of upper and lower case proxy variables. Curl, for example, specifies NO_PROXY (not no_proxy) in the man page (changed in 2009 [1]). Avoid proxy issues by ensuring both the upper and lower case versions of each proxy variable is available in the environment for the fetcher commands.
Add FTPS_PROXY and ftps_proxy to the list as well.
1. http://curl.haxx.se/mail/tracker-2009-04/0012.html
--- bitbake/lib/bb/fetch2/__init__.py | 9 +++++++-- 1 files changed, 7 insertions(+), 2 deletions(-)
bitbake/lib/bb/fetch2/__init__.py +7 -2
@@ -459,8 +459,13 @@ def runfetchcmd(cmd, d, quiet = False, cleanup = []): # Also include some other variables. # FIXME: Should really include all export varaiables? exportvars = ['PATH', 'GIT_PROXY_COMMAND', 'GIT_PROXY_HOST', - 'GIT_PROXY_PORT', 'GIT_CONFIG', 'http_proxy', 'ftp_proxy', - 'https_proxy', 'no_proxy', 'ALL_PROXY', 'all_proxy', + 'GIT_PROXY_PORT', 'GIT_CONFIG', + 'HTTP_PROXY', 'http_proxy', + 'HTTPS_PROXY', 'https_proxy', + 'FTP_PROXY', 'ftp_proxy', + 'FTPS_PROXY', 'ftps_proxy', + 'NO_PROXY', 'no_proxy', + 'ALL_PROXY', 'all_proxy', 'SSH_AUTH_SOCK', 'SSH_AGENT_PID', 'HOME', 'GIT_PROXY_IGNORE', 'SOCKS5_USER', 'SOCKS5_PASSWD']
Signature · 1 line
1.7.5.4
PATCH v2 3/9 bitbake: fetch2: Remove broken git variables from the environment
In reply to Darren HartThe following variables perform no function outside of bitbake:
GIT_CONFIG
GIT_PROXY_HOST
GIT_PROXY_PORT
GIT_PROXY_IGNORE
GIT_CONFIG only affects the git-config command which is not relevant to the fetcher. This was previously used with the OE GIT_CORE_CONFIG variable which would provide a basic git config to use instead of the user's config. This usage was deprecated by git for over a year now:
http://git.661346.n2.nabble.com/Overriding-gitconfig-using-GIT-CONFIG-td6680977.html
GIT_PROXY_HOST and GIT_PROXY_PORT are not used by git.
GIT_PROXY_IGNORE was an OE construct used to create the custom git config and had no meaning outside of the OE environment. It is not used by git.
Remove these variables from the fetcher environment.
Users wishing to configure git to work with a proxy should define the GIT_PROXY_COMMAND environment variable to use an external script. NO_PROXY can be used within this script to skip the proxy for certain hosts.
--- bitbake/lib/bb/fetch2/__init__.py | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-)
bitbake/lib/bb/fetch2/__init__.py +4 -4
@@ -458,16 +458,16 @@ def runfetchcmd(cmd, d, quiet = False, cleanup = []): # rather than host provided # Also include some other variables. # FIXME: Should really include all export varaiables? - exportvars = ['PATH', 'GIT_PROXY_COMMAND', 'GIT_PROXY_HOST', - 'GIT_PROXY_PORT', 'GIT_CONFIG', + exportvars = ['HOME', 'PATH', 'HTTP_PROXY', 'http_proxy', 'HTTPS_PROXY', 'https_proxy', 'FTP_PROXY', 'ftp_proxy', 'FTPS_PROXY', 'ftps_proxy', 'NO_PROXY', 'no_proxy', 'ALL_PROXY', 'all_proxy', - 'SSH_AUTH_SOCK', 'SSH_AGENT_PID', 'HOME', - 'GIT_PROXY_IGNORE', 'SOCKS5_USER', 'SOCKS5_PASSWD'] + 'GIT_PROXY_COMMAND', + 'SSH_AUTH_SOCK', 'SSH_AGENT_PID', + 'SOCKS5_USER', 'SOCKS5_PASSWD'] for var in exportvars: val = d.getVar(var, True)
Signature · 1 line
1.7.5.4
PATCH v2 4/9 oe-buildenv-internal: Remove GIT variables from BB_ENV_EXTRAWHITE
In reply to Darren HartThe following variables perform no function outside of bitbake:
GIT_CONFIG
GIT_PROXY_HOST
GIT_PROXY_PORT
GIT_PROXY_IGNORE
GIT_CONFIG only affects the git-config command which is not relevant to the fetcher. This was previously used with the OE GIT_CORE_CONFIG variable which would provide a basic git config to use instead of the user's config. This usage was deprecated by git for over a year now:
http://git.661346.n2.nabble.com/Overriding-gitconfig-using-GIT-CONFIG-td6680977
GIT_PROXY_HOST and GIT_PROXY_PORT are not used by git.
GIT_PROXY_IGNORE was an OE construct used to create the custom git config and had no meaning outside of the OE environment. It is not used by git.
Remove these variables from the fetcher environment.
Users wishing to configure git to work with a proxy should define the GIT_PROXY_COMMAND environment variable to use an external script. NO_PROXY can be used within this script to skip the proxy for certain hosts.
--- scripts/oe-buildenv-internal | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
scripts/oe-buildenv-internal +1 -1
@@ -80,4 +80,4 @@ unset BITBAKEDIR # Used by the runqemu script export BUILDDIR export PATH -export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER SCREENDIR" +export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy NO_PROXY SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND SOCKS5_PASSWD SOCKS5_USER SCREENDIR"
Signature · 1 line
1.7.5.4
PATCH v2 5/9 oe-buildenv-internal: Add upper and lower case proxy vars to BB_ENV_EXTRAWHITE
In reply to Darren HartApplications are inconsistent in their use of upper and lower case proxy variables. Curl, for example, specifies NO_PROXY (not no_proxy) in the man page (changed in 2009 [1]). Avoid proxy issues by ensuring both the upper and lower case versions of each proxy variable are available in the environment for the fetcher commands.
Add FTPS_PROXY and ftps_proxy to the list as well.
1. http://curl.haxx.se/mail/tracker-2009-04/0012.html
--- scripts/oe-buildenv-internal | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
scripts/oe-buildenv-internal +1 -1
@@ -80,4 +80,4 @@ unset BITBAKEDIR # Used by the runqemu script export BUILDDIR export PATH -export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy NO_PROXY SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND SOCKS5_PASSWD SOCKS5_USER SCREENDIR" +export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC HTTP_PROXY http_proxy HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy FTPS_PROXY ftps_proxy ALL_PROXY all_proxy NO_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND SOCKS5_PASSWD SOCKS5_USER SCREENDIR"
Signature · 1 line
1.7.5.4
PATCH v2 6/9 base.bbclass: Remove generate_git_config()
In reply to Darren Hartgit no longer supports the use of GIT_CONFIG which defeats the purpose of GIT_CORE_CONFIG and the generate_git_config() function. Remove it.
--- meta/classes/base.bbclass | 21 --------------------- 1 files changed, 0 insertions(+), 21 deletions(-)
meta/classes/base.bbclass +0 -21
@@ -133,26 +133,6 @@ python base_do_unpack() { raise bb.build.FuncFailed(e) } -GIT_CONFIG_PATH = "${STAGING_DIR_NATIVE}/etc" -GIT_CONFIG = "${GIT_CONFIG_PATH}/gitconfig" - -def generate_git_config(e): - if e.data.getVar('GIT_CORE_CONFIG', True): - gitconfig_path = e.data.getVar('GIT_CONFIG', True) - proxy_command = " gitProxy = %s\n" % e.data.getVar('OE_GIT_PROXY_COMMAND', True) - - bb.mkdirhier(e.data.expand("${GIT_CONFIG_PATH}")) - if (os.path.exists(gitconfig_path)): - os.remove(gitconfig_path) - - f = open(gitconfig_path, 'w') - f.write("[core]\n") - ignore_hosts = e.data.getVar('GIT_PROXY_IGNORE', True).split() - for ignore_host in ignore_hosts: - f.write(" gitProxy = none for %s\n" % ignore_host) - f.write(proxy_command) - f.close - def pkgarch_mapping(d): # Compatibility mappings of TUNE_PKGARCH (opt in) if d.getVar("PKGARCHCOMPAT_ARMV7A", True):
@@ -311,7 +291,6 @@ addhandler base_eventhandler python base_eventhandler() { if isinstance(e, bb.event.ConfigParsed): e.data.setVar('BB_VERSION', bb.__version__) - generate_git_config(e) pkgarch_mapping(e.data) preferred_ml_updates(e.data) e.data.appendVar('DISTRO_FEATURES', oe.utils.features_backfill("DISTRO_FEATURES", e.data))
Signature · 1 line
1.7.5.4
PATCH v2 7/9 oe-git-proxy: Add a new comprehensive git proxy script
In reply to Darren Hartoe-git-proxy.sh is a simple tool to be used via GIT_PROXY_COMMAND. It uses BSD netcat to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It is known to work with both bash and dash shells.
V2: Implement recommendations by Enrico Scholz:
o Use exec for the nc calls
o Use "$@" instead of $* to avoid quoting issues inherent with $*
o Use bash explicitly and simplify some of the string manipulations Also:
o Drop the .sh in the name per Otavio Salvador
o Remove a stray debug statement
--- scripts/oe-git-proxy | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 124 insertions(+), 0 deletions(-) create mode 100755 scripts/oe-git-proxy
scripts/oe-git-proxy new file +124 -0
@@ -0,0 +1,124 @@ +#!/bin/bash + +# oe-git-proxy.sh is a simple tool to be via GIT_PROXY_COMMAND. It uses BSD netcat +# to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the +# proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for +# a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks +# (192.168.1.0/24). It is known to work with both bash and dash shells. +# +# BSD netcat is provided by netcat-openbsd on Ubuntu and nc on Fedora. +# +# Example ALL_PROXY values: +# ALL_PROXY=socks://socks.example.com:1080 +# ALL_PROXY=https://proxy.example.com:8080 +# +# Copyright (c) 2013, Intel Corporation. +# All rights reserved. +# +# AUTHORS +# Darren Hart <[email protected]> + +# Locate the netcat binary +NC=$(which nc 2>/dev/null) +if [ $? -ne 0 ]; then + echo "ERROR: nc binary not in PATH" + exit 1 +fi +METHOD="" + +# Test for a valid IPV4 quad with optional bitmask +valid_ipv4() { + echo $1 | egrep -q "^([1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])){3}(/(3[0-2]|[1-2]?[0-9]))?$" + return $? +} + +# Convert an IPV4 address into a 32bit integer +ipv4_val() { + IP="$1" + SHIFT=24 + VAL=0 + for B in ${IP//./ }; do + VAL=$(($VAL+$(($B<<$SHIFT)))) + SHIFT=$(($SHIFT-8)) + done + echo "$VAL" +} + +# Determine if two IPs are equivalent, or if the CIDR contains the IP +match_ipv4() { + CIDR=$1 + IP=$2 + + if [ -z "${IP%%$CIDR}" ]; then + return 0 + fi + + # Determine the mask bitlength + BITS=${CIDR##*/} + if [ -z "$BITS" ]; then + return 1 + fi + + IPVAL=$(ipv4_val $IP) + IP2VAL=$(ipv4_val ${CIDR%%/*}) + + # OR in the unmasked bits + for i in $(seq 0 $((32-$BITS))); do + IP2VAL=$(($IP2VAL|$((1<<$i)))) + IPVAL=$(($IPVAL|$((1<<$i)))) + done + + if [ $IPVAL -eq $IP2VAL ]; then + return 0 + fi + return 1 +} + +# Test to see if GLOB matches HOST +match_host() { + HOST=$1 + GLOB=$2 + + if [ -z "${HOST%%$GLOB}" ]; then + return 0 + fi + + # Match by netmask + if valid_ipv4 $GLOB; then + HOST_IP=$(gethostip -d $HOST) + if valid_ipv4 $HOST_IP; then + match_ipv4 $GLOB $HOST_IP + if [ $? -eq 0 ]; then + return 0 + fi + fi + fi + + return 1 +} + +# If no proxy is set, just connect directly +if [ -z "$ALL_PROXY" ]; then + exec $NC -X connect "$@" +fi + +# Connect directly to hosts in NO_PROXY +for H in ${NO_PROXY//,/ }; do + if match_host $1 $H; then + METHOD="-X connect" + break + fi +done + +if [ -z "$METHOD" ]; then + # strip the protocol and the trailing slash + PROTO=$(echo $ALL_PROXY | sed -e 's/\([^:]*\):\/\/.*/\1/') + PROXY=$(echo $ALL_PROXY | sed -e 's/.*:\/\/\([^:]*:[0-9]*\).*/\1/') + if [ "$PROTO" = "socks" ]; then + METHOD="-X 5 -x $PROXY" + elif [ "$PROTO" = "https" ]; then + METHOD="-X connect -x $PROXY" + fi +fi + +exec $NC $METHOD "$@"
Signature · 1 line
1.7.5.4
PATCH v2 8/9 meta-yocto: Document new oe-git-proxy in site.conf.sample
In reply to Darren HartThe new oe-git-proxy.sh should address all git proxying needs. Document its usage in meta-yocto/conf/site.conf.sample and remove references to the old mechanisms.
V2: Separate out the meta-yocto changes from the oe-core changes
--- meta-yocto/conf/site.conf.sample | 35 +++++++++-------------------------- 1 files changed, 9 insertions(+), 26 deletions(-)
meta-yocto/conf/site.conf.sample +9 -26
@@ -16,32 +16,15 @@ SCONF_VERSION = "1" #http-proxy-port = 81 # -# Uncomment to cause git to use the proxy host specificed -# although this only works for http -#GIT_PROXY_HOST = "proxy.example.com" -#GIT_PROXY_PORT = "81" -#export GIT_PROXY_COMMAND = "${COREBASE}/scripts/oe-git-proxy-command" - -# Set to yes to have a gitconfig generated for handling proxies; you -# might not want this if you have all that set in your global git -# configuration. If you don't enable it, the rest of the entries -# (_PROXY_IGNORE, etc) don't really work that well -#GIT_CORE_CONFIG = "Yes" - -# Space separate list of hosts to ignore for GIT proxy -#GIT_PROXY_IGNORE = "host.server.com another.server.com" - -# If SOCKS is available run the following command to comple a simple transport -# gcc scripts/oe-git-proxy-socks.c -o oe-git-proxy-socks -# and then share that binary somewhere in PATH, then use the following settings -#GIT_PROXY_HOST = "proxy.example.com" -#GIT_PROXY_PORT = "81" - -# GIT_PROXY_COMMAND is used by git to override all proxy settings from -# configuration files, so we prefix OE_ to avoid breaking havoc on the -# generated (or local) gitconfig's. -#OE_GIT_PROXY_COMMAND = "${COREBASE}/scripts/oe-git-proxy-socks-command" - +# To use git with a proxy, you must use an external git proxy command, such as +# the one provided by scripts/oe-git-proxy.sh. To use this script, copy it to +# your PATH and uncomment the following: +#GIT_PROXY_COMMAND="oe-git-proxy.sh" +#ALL_PROXY="socks://socks.example.com:1080" +#or +#ALL_PROXY="https://proxy.example.com:8080" +# If you wish to use certain hosts without the proxy, specify them in NO_PROXY. +# See the script for details on syntax. # Uncomment this to use a shared download directory #DL_DIR = "/some/shared/download/directory/"
Signature · 1 line
1.7.5.4
PATCH v2 9/9 oe-git-proxy*: Remove previous git proxy solutions
In reply to Darren HartThe new oe-git-proxy.sh should address all git proxying needs, remove the previous scripts.
V2: Separate the removal of the old scripts into their own patch
--- scripts/oe-git-proxy-command | 10 - scripts/oe-git-proxy-socks-command | 23 - scripts/oe-git-proxy-socks.c | 2982 ------------------------------------ 3 files changed, 0 insertions(+), 3015 deletions(-) delete mode 100755 scripts/oe-git-proxy-command delete mode 100755 scripts/oe-git-proxy-socks-command delete mode 100755 scripts/oe-git-proxy-socks.c
scripts/oe-git-proxy-command deleted +0 -10
@@ -1,10 +0,0 @@ -#! /bin/bash - -(echo "CONNECT $1:$2 HTTP/1.0"; - echo; - cat ) | nc $GIT_PROXY_HOST $GIT_PROXY_PORT | - -(read a; - read a; - read a; - cat )
scripts/oe-git-proxy-socks-command deleted +0 -23
@@ -1,23 +0,0 @@ -#! /bin/bash -SCRIPTDIR=`dirname $0` -# Check oe-git-proxy-socks exists -PROXYSOCKS=`which oe-git-proxy-socks 2> /dev/null` -if [ -z "$PROXYSOCKS" -a -e "$SCRIPTDIR/oe-git-proxy-socks.c" ]; then - # If not try and build it - gcc $SCRIPTDIR/oe-git-proxy-socks.c -o $SCRIPTDIR/oe-git-proxy-socks -fi -PROXYSOCKS=`which oe-git-proxy-socks 2> /dev/null` -if [ ! -x "$PROXYSOCKS" ]; then - # If that fails, we can see if netcat (nc) is available - NETCAT=`which nc 2> /dev/null` - if [ ! -x "$NETCAT" ]; then - # If that fails, explain to the user - echo "Unable to find oe-git-proxy-socks. This is usually created with the command" - echo "'gcc scripts/oe-git-proxy-socks.c -o scripts/oe-git-proxy-socks' which we tried" - echo "but it doesn't seem to have worked. Please compile the binary manually." - echo "Alternativly, install nc (netcat) on this machine." - exit 1 - fi - exec $NETCAT -x $GIT_PROXY_HOST:$GIT_PROXY_PORT "$@" -fi -oe-git-proxy-socks -S $GIT_PROXY_HOST:$GIT_PROXY_PORT $@
scripts/oe-git-proxy-socks.c deleted +0 -2982
2,966 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
1.7.5.4
Part of the diff is not shown here. View the raw message.
Re: [PATCH 7/9] oe-git-proxy: Add a new comprehensive git proxy script
In reply to Otavio SalvadorQuoted 36 lines · Otavio Salvador
Quoted 29 lines · Darren Hart
oe-git-proxy.shis a simple tool to be used viaGIT_PROXY_COMMAND. It uses BSD netcat to make SOCKS5 or HTTPS proxy connections. It usesALL_PROXYto determine the proxy server, protocol, and port. It usesNO_PROXYto skip using the proxy for a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It is known to work with both bash and dash shells.V2: Implement recommendations by Enrico Scholz:
o Use exec for the nc calls
o Use "$@" instead of $* to avoid quoting issues inherent with $*
o Use bash explicitly and simplify some of the string manipulations Also:
o Drop the .sh in the name per Otavio Salvador
o Remove a stray debug statementSigned-off-by Darren Hart <[email protected]>Cc Enrico Scholz <[email protected]>Cc Otavio Salvador <[email protected]>--- scripts/oe-git-proxy | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 124 insertions(+), 0 deletions(-) create mode 100755 scripts/oe-git-proxyscripts/oe-git-proxy new file +0
-0scripts/oe-git-proxy line 3
Quoted 3 lines
scripts/oe-git-proxy new file +3
-0@@ -0,0 +1,124 @@ +#!/bin/bash + +# oe-git-proxy.sh is a simple tool to be via GIT_PROXY_COMMAND. It uses BSD netcatPlease fix the script name.
Quoted 15 lines
scripts/oe-git-proxy line 16
Quoted 13 lines
scripts/oe-git-proxy new file +13
-0+# to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the +# proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for +# a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks +# (192.168.1.0/24). It is known to work with both bash and dash shells. +# +# BSD netcat is provided by netcat-openbsd on Ubuntu and nc on Fedora. +# +# Example ALL_PROXY values: +# ALL_PROXY=socks://socks.example.com:1080 +# ALL_PROXY=https://proxy.example.com:8080 +# +# Copyright (c) 2013, Intel Corporation. +# All rights reserved.And please add a clear license here.
Both fixed in the contrib branch. GPL license blurb added.
Have you happened to have tested the script?
Thanks Otavio.
Signature · 3 lines
Darren Hart Intel Open Source Technology Center Yocto Project - Technical Lead - Linux Kernel
Re: [PATCH 7/9] oe-git-proxy: Add a new comprehensive git proxy script
In reply to Darren HartQuoted 29 lines · Darren Hart
oe-git-proxy.shis a simple tool to be used viaGIT_PROXY_COMMAND. It uses BSD netcat to make SOCKS5 or HTTPS proxy connections. It usesALL_PROXYto determine the proxy server, protocol, and port. It usesNO_PROXYto skip using the proxy for a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It is known to work with both bash and dash shells.V2: Implement recommendations by Enrico Scholz:
o Use exec for the nc calls
o Use "$@" instead of $* to avoid quoting issues inherent with $*
o Use bash explicitly and simplify some of the string manipulations Also:
o Drop the .sh in the name per Otavio Salvador
o Remove a stray debug statementSigned-off-by Darren Hart <[email protected]>Cc Enrico Scholz <[email protected]>Cc Otavio Salvador <[email protected]>--- scripts/oe-git-proxy | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 124 insertions(+), 0 deletions(-) create mode 100755 scripts/oe-git-proxyscripts/oe-git-proxy new file +0
-0
scripts/oe-git-proxy line 3
Quoted 3 lines
scripts/oe-git-proxy new file +3
-0@@ -0,0 +1,124 @@ +#!/bin/bash + +# oe-git-proxy.sh is a simple tool to be via GIT_PROXY_COMMAND. It uses BSD netcat
Please fix the script name.
scripts/oe-git-proxy line 16
Quoted 13 lines
scripts/oe-git-proxy new file +13
-0+# to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the +# proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for +# a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks +# (192.168.1.0/24). It is known to work with both bash and dash shells. +# +# BSD netcat is provided by netcat-openbsd on Ubuntu and nc on Fedora. +# +# Example ALL_PROXY values: +# ALL_PROXY=socks://socks.example.com:1080 +# ALL_PROXY=https://proxy.example.com:8080 +# +# Copyright (c) 2013, Intel Corporation. +# All rights reserved.
And please add a clear license here.
Quoted 108 lines
scripts/oe-git-proxy new file +107
-1+# AUTHORS +# Darren Hart <[email protected]> + +# Locate the netcat binary +NC=$(which nc 2>/dev/null) +if [ $? -ne 0 ]; then + echo "ERROR: nc binary not in PATH" + exit 1 +fi +METHOD="" + +# Test for a valid IPV4 quad with optional bitmask +valid_ipv4() { + echo $1 | egrep -q "^([1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])){3}(/(3[0-2]|[1-2]?[0-9]))?$" + return $? +} + +# Convert an IPV4 address into a 32bit integer +ipv4_val() { + IP="$1" + SHIFT=24 + VAL=0 + for B in ${IP//./ }; do + VAL=$(($VAL+$(($B<<$SHIFT)))) + SHIFT=$(($SHIFT-8)) + done + echo "$VAL" +} + +# Determine if two IPs are equivalent, or if the CIDR contains the IP +match_ipv4() { + CIDR=$1 + IP=$2 + + if [ -z "${IP%%$CIDR}" ]; then + return 0 + fi + + # Determine the mask bitlength + BITS=${CIDR##*/} + if [ -z "$BITS" ]; then + return 1 + fi + + IPVAL=$(ipv4_val $IP) + IP2VAL=$(ipv4_val ${CIDR%%/*}) + + # OR in the unmasked bits + for i in $(seq 0 $((32-$BITS))); do + IP2VAL=$(($IP2VAL|$((1<<$i)))) + IPVAL=$(($IPVAL|$((1<<$i)))) + done + + if [ $IPVAL -eq $IP2VAL ]; then + return 0 + fi + return 1 +} + +# Test to see if GLOB matches HOST +match_host() { + HOST=$1 + GLOB=$2 + + if [ -z "${HOST%%$GLOB}" ]; then + return 0 + fi + + # Match by netmask + if valid_ipv4 $GLOB; then + HOST_IP=$(gethostip -d $HOST) + if valid_ipv4 $HOST_IP; then + match_ipv4 $GLOB $HOST_IP + if [ $? -eq 0 ]; then + return 0 + fi + fi + fi + + return 1 +} + +# If no proxy is set, just connect directly +if [ -z "$ALL_PROXY" ]; then + exec $NC -X connect "$@" +fi + +# Connect directly to hosts in NO_PROXY +for H in ${NO_PROXY//,/ }; do + if match_host $1 $H; then + METHOD="-X connect" + break + fi +done + +if [ -z "$METHOD" ]; then + # strip the protocol and the trailing slash + PROTO=$(echo $ALL_PROXY | sed -e 's/\([^:]*\):\/\/.*/\1/') + PROXY=$(echo $ALL_PROXY | sed -e 's/.*:\/\/\([^:]*:[0-9]*\).*/\1/') + if [ "$PROTO" = "socks" ]; then + METHOD="-X 5 -x $PROXY" + elif [ "$PROTO" = "https" ]; then + METHOD="-X connect -x $PROXY" + fi +fi + +exec $NC $METHOD "$@" --
Quoted 2 lines
1.7.5.4
--
Otavio Salvador O.S. Systems
E-mail: [email protected] http://www.ossystems.com.br
Mobile: +55 53 9981-7854 http://projetos.ossystems.com.br
Re: [PATCH 7/9] oe-git-proxy: Add a new comprehensive git proxy script
In reply to Otavio SalvadorQuoted 59 lines · Otavio Salvador
Quoted 58 lines · Darren Hart
Quoted 36 lines · Otavio Salvador
Quoted 29 lines · Darren Hart
oe-git-proxy.shis a simple tool to be used viaGIT_PROXY_COMMAND. It uses BSD netcat to make SOCKS5 or HTTPS proxy connections. It usesALL_PROXYto determine the proxy server, protocol, and port. It usesNO_PROXYto skip using the proxy for a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It is known to work with both bash and dash shells.V2: Implement recommendations by Enrico Scholz:
o Use exec for the nc calls
o Use "$@" instead of $* to avoid quoting issues inherent with $*
o Use bash explicitly and simplify some of the string manipulations Also:
o Drop the .sh in the name per Otavio Salvador
o Remove a stray debug statementSigned-off-by Darren Hart <[email protected]>Cc Enrico Scholz <[email protected]>Cc Otavio Salvador <[email protected]>--- scripts/oe-git-proxy | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 124 insertions(+), 0 deletions(-) create mode 100755 scripts/oe-git-proxyscripts/oe-git-proxy new file +0
-0scripts/oe-git-proxy line 3
Quoted 3 lines
scripts/oe-git-proxy new file +3
-0@@ -0,0 +1,124 @@ +#!/bin/bash + +# oe-git-proxy.sh is a simple tool to be via GIT_PROXY_COMMAND. It uses BSD netcatPlease fix the script name.
Quoted 15 lines
scripts/oe-git-proxy line 16
Quoted 13 lines
scripts/oe-git-proxy new file +13
-0+# to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the +# proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for +# a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks +# (192.168.1.0/24). It is known to work with both bash and dash shells. +# +# BSD netcat is provided by netcat-openbsd on Ubuntu and nc on Fedora. +# +# Example ALL_PROXY values: +# ALL_PROXY=socks://socks.example.com:1080 +# ALL_PROXY=https://proxy.example.com:8080 +# +# Copyright (c) 2013, Intel Corporation. +# All rights reserved.And please add a clear license here.
Both fixed in the contrib branch. GPL license blurb added.
Have you happened to have tested the script?
Quoted 8 lines
Quoted 1 line
Thanks Otavio.
No; not yet.
I am wondering if the code wouldn't be easier to read if done in Python ... the ipv4 stuff is UGLY and hard to follow ... I understand it is very simple script but maybe Python would be the right tool for it. What do you think?
I had originally written the IP matching stuff in C, but converted to shell to
keep this script self contained and easily deployable. The proper way to do this
with python would require the use of external modules, which may or may not be
installed on the target system. Although I suppose even just replacing this script and all the math it does in python might be an improvement. However, For
the purposes of a one-stop-shop example git proxy script for oe-core, I think
bash still serves us best here. Otherwise, yes, I completely agree it's horrible
to look at :-)
Signature · 3 lines
Darren Hart Intel Open Source Technology Center Yocto Project - Technical Lead - Linux Kernel
Re: [PATCH 7/9] oe-git-proxy: Add a new comprehensive git proxy script
In reply to Darren HartQuoted 54 lines · Darren Hart
Quoted 36 lines · Otavio Salvador
Quoted 29 lines · Darren Hart
oe-git-proxy.shis a simple tool to be used viaGIT_PROXY_COMMAND. It uses BSD netcat to make SOCKS5 or HTTPS proxy connections. It usesALL_PROXYto determine the proxy server, protocol, and port. It usesNO_PROXYto skip using the proxy for a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It is known to work with both bash and dash shells.V2: Implement recommendations by Enrico Scholz:
o Use exec for the nc calls
o Use "$@" instead of $* to avoid quoting issues inherent with $*
o Use bash explicitly and simplify some of the string manipulations Also:
o Drop the .sh in the name per Otavio Salvador
o Remove a stray debug statementSigned-off-by Darren Hart <[email protected]>Cc Enrico Scholz <[email protected]>Cc Otavio Salvador <[email protected]>--- scripts/oe-git-proxy | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 124 insertions(+), 0 deletions(-) create mode 100755 scripts/oe-git-proxyscripts/oe-git-proxy new file +0
-0scripts/oe-git-proxy line 3
Quoted 3 lines
scripts/oe-git-proxy new file +3
-0@@ -0,0 +1,124 @@ +#!/bin/bash + +# oe-git-proxy.sh is a simple tool to be via GIT_PROXY_COMMAND. It uses BSD netcatPlease fix the script name.
Quoted 15 lines
scripts/oe-git-proxy line 16
Quoted 13 lines
scripts/oe-git-proxy new file +13
-0+# to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the +# proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for +# a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks +# (192.168.1.0/24). It is known to work with both bash and dash shells. +# +# BSD netcat is provided by netcat-openbsd on Ubuntu and nc on Fedora. +# +# Example ALL_PROXY values: +# ALL_PROXY=socks://socks.example.com:1080 +# ALL_PROXY=https://proxy.example.com:8080 +# +# Copyright (c) 2013, Intel Corporation. +# All rights reserved.And please add a clear license here.
Quoted 5 lines
Both fixed in the contrib branch. GPL license blurb added.
Have you happened to have tested the script?
Thanks Otavio.
No; not yet.
I am wondering if the code wouldn't be easier to read if done in Python ... the ipv4 stuff is UGLY and hard to follow ... I understand it is very simple script but maybe Python would be the right tool for it. What do you think?
--
Otavio Salvador O.S. Systems
E-mail: [email protected] http://www.ossystems.com.br
Mobile: +55 53 9981-7854 http://projetos.ossystems.com.br
Re: [PATCH 8/9] meta-yocto: Document new oe-git-proxy in site.conf.sample
In reply to Otavio SalvadorQuoted 44 lines · Otavio Salvador
Quoted 43 lines · Darren Hart
The new
oe-git-proxy.shshould address all git proxying needs. Document its usage in meta-yocto/conf/site.conf.sample and remove references to the old mechanisms.V2: Separate out the meta-yocto changes from the oe-core changes
Signed-off-by Darren Hart <[email protected]>--- meta-yocto/conf/site.conf.sample | 35 +++++++++-------------------------- 1 files changed, 9 insertions(+), 26 deletions(-)meta-yocto/conf/site.conf.sample +0
-24@@ -16,32 +16,15 @@ SCONF_VERSION = "1" #http-proxy-port = 81 # -# Uncomment to cause git to use the proxy host specificed -# although this only works for http -#GIT_PROXY_HOST = "proxy.example.com" -#GIT_PROXY_PORT = "81" -#export GIT_PROXY_COMMAND = "${COREBASE}/scripts/oe-git-proxy-command" - -# Set to yes to have a gitconfig generated for handling proxies; you -# might not want this if you have all that set in your global git -# configuration. If you don't enable it, the rest of the entries -# (_PROXY_IGNORE, etc) don't really work that well -#GIT_CORE_CONFIG = "Yes" - -# Space separate list of hosts to ignore for GIT proxy -#GIT_PROXY_IGNORE = "host.server.com another.server.com" - -# If SOCKS is available run the following command to comple a simple transport -# gcc scripts/oe-git-proxy-socks.c -o oe-git-proxy-socks -# and then share that binary somewhere in PATH, then use the following settings -#GIT_PROXY_HOST = "proxy.example.com" -#GIT_PROXY_PORT = "81" - -# GIT_PROXY_COMMAND is used by git to override all proxy settings from -# configuration files, so we prefix OE_ to avoid breaking havoc on the -# generated (or local) gitconfig's.
Quoted 8 lines
meta-yocto/conf/site.conf.sample SCONF_VERSION = "1" · line 22
Quoted 6 lines
meta-yocto/conf/site.conf.sample +4
-2-#OE_GIT_PROXY_COMMAND = "${COREBASE}/scripts/oe-git-proxy-socks-command" - +# To use git with a proxy, you must use an external git proxy command, such as +# the one provided by scripts/oe-git-proxy.sh. To use this script, copy it to +# your PATH and uncomment the following: +#GIT_PROXY_COMMAND="oe-git-proxy.sh".sh
Fixed
Quoted 1 line
A similar change ought to be done for oe-core .sample, no?
I didn't see any git proxy documentation there and no conf.sample files in oe-core at all. Perhaps someone would like to add a site.conf.sample for oe-core?
Signature · 3 lines
Darren Hart Intel Open Source Technology Center Yocto Project - Technical Lead - Linux Kernel
Re: [PATCH 8/9] meta-yocto: Document new oe-git-proxy in site.conf.sample
In reply to Darren HartQuoted 39 lines · Darren Hart
The new
oe-git-proxy.shshould address all git proxying needs. Document its usage in meta-yocto/conf/site.conf.sample and remove references to the old mechanisms.V2: Separate out the meta-yocto changes from the oe-core changes
Signed-off-by Darren Hart <[email protected]>--- meta-yocto/conf/site.conf.sample | 35 +++++++++-------------------------- 1 files changed, 9 insertions(+), 26 deletions(-)meta-yocto/conf/site.conf.sample +0
-20@@ -16,32 +16,15 @@ SCONF_VERSION = "1" #http-proxy-port = 81 # -# Uncomment to cause git to use the proxy host specificed -# although this only works for http -#GIT_PROXY_HOST = "proxy.example.com" -#GIT_PROXY_PORT = "81" -#export GIT_PROXY_COMMAND = "${COREBASE}/scripts/oe-git-proxy-command" - -# Set to yes to have a gitconfig generated for handling proxies; you -# might not want this if you have all that set in your global git -# configuration. If you don't enable it, the rest of the entries -# (_PROXY_IGNORE, etc) don't really work that well -#GIT_CORE_CONFIG = "Yes" - -# Space separate list of hosts to ignore for GIT proxy -#GIT_PROXY_IGNORE = "host.server.com another.server.com" - -# If SOCKS is available run the following command to comple a simple transport -# gcc scripts/oe-git-proxy-socks.c -o oe-git-proxy-socks -# and then share that binary somewhere in PATH, then use the following settings -#GIT_PROXY_HOST = "proxy.example.com" -#GIT_PROXY_PORT = "81"
meta-yocto/conf/site.conf.sample SCONF_VERSION = "1" · line 22
Quoted 10 lines
meta-yocto/conf/site.conf.sample +4
-6- -# GIT_PROXY_COMMAND is used by git to override all proxy settings from -# configuration files, so we prefix OE_ to avoid breaking havoc on the -# generated (or local) gitconfig's. -#OE_GIT_PROXY_COMMAND = "${COREBASE}/scripts/oe-git-proxy-socks-command" - +# To use git with a proxy, you must use an external git proxy command, such as +# the one provided by scripts/oe-git-proxy.sh. To use this script, copy it to +# your PATH and uncomment the following: +#GIT_PROXY_COMMAND="oe-git-proxy.sh"
.sh
A similar change ought to be done for oe-core .sample, no?
Quoted 12 lines
meta-yocto/conf/site.conf.sample +5
-1+#ALL_PROXY="socks://socks.example.com:1080" +#or +#ALL_PROXY="https://proxy.example.com:8080" +# If you wish to use certain hosts without the proxy, specify them in NO_PROXY. +# See the script for details on syntax. # Uncomment this to use a shared download directory #DL_DIR = "/some/shared/download/directory/" --1.7.5.4
Quoted 4 lines
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
--
Otavio Salvador O.S. Systems
E-mail: [email protected] http://www.ossystems.com.br
Mobile: +55 53 9981-7854 http://projetos.ossystems.com.br