Git proxy handling updates (now with socat)
v3 cover Git proxy handling updates (now with socat)
The current git fetcher environment and proxy handling between bitbake, base.bbclass, and oe-buildenv-internal has become stale and is largely non-functional. The crux of the problem is that GIT_CONFIG is no longer honored by anything other than the git-config command, rendering the GIT_CORE_CONFIG and the assocaited proxy handling routines obsolete.
(the bitbake patches have been merged and dropped from this series)
The GIT_PROXY_COMMAND, ALL_PROXY, and NO_PROXY environment variables are more than adequate for storing the required information to manage proxies with a git helper script.
These patches series aggressively removes the old stuff, fixes up some missing environment settings, and adds a new git proxy handling script.
Note: These patches DO NOT address the bitbake gui builder.py references to git
proxy settings!
I felt this series was best viewed as a whole, so I'm sending it to poky AND oe-core. Anything else seemed too likely to result in confusion, missing context, and lots and lots of redundant discussion.
V2: Updates to the new oe-git-proxy script from feedback received
Split out meta-yocto changes
Split out removal of old scripts
Minor commit log corrections
V3: Fedora 18 dropped BSD nc support, so add a patch converting oe-git-proxy
to use socat which is broadly available. HTTP proxy still remains untested.
The following changes since commit 20b4dcace222d6b1d659a8813b4affad075ec0c6:
dbus-native: Don't install dbus-launch (2013-02-08 14:50:38 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib dvhart/gitproxy
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=dvhart/gitproxy
Darren Hart (7):
oe-buildenv-internal: Remove GIT variables from BB_ENV_EXTRAWHITE
oe-buildenv-internal: Add upper and lower case proxy vars to
BB_ENV_EXTRAWHITE
base.bbclass: Remove generate_git_config()
oe-git-proxy: Add a new comprehensive git proxy script
meta-yocto: Document new oe-git-proxy in site.conf.sample oe-git-proxy*: Remove previous git proxy solutions
oe-git-proxy: Use socat instead of BSD nc
meta-yocto/conf/site.conf.sample | 35 +- meta/classes/base.bbclass | 21 - scripts/oe-buildenv-internal | 2 +- scripts/oe-git-proxy | 133 ++ scripts/oe-git-proxy-command | 10 - scripts/oe-git-proxy-socks-command | 23 - scripts/oe-git-proxy-socks.c | 2982 ------------------------------------ 7 files changed, 143 insertions(+), 3063 deletions(-) create mode 100755 scripts/oe-git-proxy delete mode 100755 scripts/oe-git-proxy-command delete mode 100755 scripts/oe-git-proxy-socks-command delete mode 100755 scripts/oe-git-proxy-socks.c
Signature · 1 line
1.7.5.4
PATCH v3 1/7 oe-buildenv-internal: Remove GIT variables from BB_ENV_EXTRAWHITE
The following variables perform no function outside of bitbake:
GIT_CONFIG
GIT_PROXY_HOST
GIT_PROXY_PORT
GIT_PROXY_IGNORE
GIT_CONFIG only affects the git-config command which is not relevant to the fetcher. This was previously used with the OE GIT_CORE_CONFIG variable which would provide a basic git config to use instead of the user's config. This usage was deprecated by git for over a year now:
http://git.661346.n2.nabble.com/Overriding-gitconfig-using-GIT-CONFIG-td6680977
GIT_PROXY_HOST and GIT_PROXY_PORT are not used by git.
GIT_PROXY_IGNORE was an OE construct used to create the custom git config and had no meaning outside of the OE environment. It is not used by git.
Remove these variables from the fetcher environment.
Users wishing to configure git to work with a proxy should define the GIT_PROXY_COMMAND environment variable to use an external script. NO_PROXY can be used within this script to skip the proxy for certain hosts.
--- scripts/oe-buildenv-internal | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
scripts/oe-buildenv-internal +1 -1
@@ -80,4 +80,4 @@ unset BITBAKEDIR # Used by the runqemu script export BUILDDIR export PATH -export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER SCREENDIR" +export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy NO_PROXY SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND SOCKS5_PASSWD SOCKS5_USER SCREENDIR"
Signature · 1 line
1.7.5.4
PATCH v3 2/7 oe-buildenv-internal: Add upper and lower case proxy vars to BB_ENV_EXTRAWHITE
In reply to Darren HartApplications are inconsistent in their use of upper and lower case proxy variables. Curl, for example, specifies NO_PROXY (not no_proxy) in the man page (changed in 2009 [1]). Avoid proxy issues by ensuring both the upper and lower case versions of each proxy variable are available in the environment for the fetcher commands.
Add FTPS_PROXY and ftps_proxy to the list as well.
1. http://curl.haxx.se/mail/tracker-2009-04/0012.html
--- scripts/oe-buildenv-internal | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
scripts/oe-buildenv-internal +1 -1
@@ -80,4 +80,4 @@ unset BITBAKEDIR # Used by the runqemu script export BUILDDIR export PATH -export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy NO_PROXY SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND SOCKS5_PASSWD SOCKS5_USER SCREENDIR" +export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC HTTP_PROXY http_proxy HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy FTPS_PROXY ftps_proxy ALL_PROXY all_proxy NO_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND SOCKS5_PASSWD SOCKS5_USER SCREENDIR"
Signature · 1 line
1.7.5.4
PATCH v3 3/7 base.bbclass: Remove generate_git_config()
In reply to Darren Hartgit no longer supports the use of GIT_CONFIG which defeats the purpose of GIT_CORE_CONFIG and the generate_git_config() function. Remove it.
--- meta/classes/base.bbclass | 21 --------------------- 1 files changed, 0 insertions(+), 21 deletions(-)
meta/classes/base.bbclass +0 -21
@@ -132,26 +132,6 @@ python base_do_unpack() { raise bb.build.FuncFailed(e) } -GIT_CONFIG_PATH = "${STAGING_DIR_NATIVE}/etc" -GIT_CONFIG = "${GIT_CONFIG_PATH}/gitconfig" - -def generate_git_config(e): - if e.data.getVar('GIT_CORE_CONFIG', True): - gitconfig_path = e.data.getVar('GIT_CONFIG', True) - proxy_command = " gitProxy = %s\n" % e.data.getVar('OE_GIT_PROXY_COMMAND', True) - - bb.mkdirhier(e.data.expand("${GIT_CONFIG_PATH}")) - if (os.path.exists(gitconfig_path)): - os.remove(gitconfig_path) - - f = open(gitconfig_path, 'w') - f.write("[core]\n") - ignore_hosts = e.data.getVar('GIT_PROXY_IGNORE', True).split() - for ignore_host in ignore_hosts: - f.write(" gitProxy = none for %s\n" % ignore_host) - f.write(proxy_command) - f.close - def pkgarch_mapping(d): # Compatibility mappings of TUNE_PKGARCH (opt in) if d.getVar("PKGARCHCOMPAT_ARMV7A", True):
@@ -310,7 +290,6 @@ addhandler base_eventhandler python base_eventhandler() { if isinstance(e, bb.event.ConfigParsed): e.data.setVar('BB_VERSION', bb.__version__) - generate_git_config(e) pkgarch_mapping(e.data) preferred_ml_updates(e.data) e.data.appendVar('DISTRO_FEATURES', oe.utils.features_backfill("DISTRO_FEATURES", e.data))
Signature · 1 line
1.7.5.4
PATCH v3 4/7 oe-git-proxy: Add a new comprehensive git proxy script
In reply to Darren Hartoe-git-proxy.sh is a simple tool to be used via GIT_PROXY_COMMAND. It uses BSD netcat to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It is known to work with both bash and dash shells.
V2: Implement recommendations by Enrico Scholz:
o Use exec for the nc calls
o Use "$@" instead of $* to avoid quoting issues inherent with $*
o Use bash explicitly and simplify some of the string manipulations Also:
o Drop the .sh in the name per Otavio Salvador
o Remove a stray debug statement
V3: Implement recommendations by Otavio Salvador
o GPL license blurb
o Fix minor typo in comment block
git-proxy cleanup
--- scripts/oe-git-proxy | 138 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 138 insertions(+), 0 deletions(-) create mode 100755 scripts/oe-git-proxy
scripts/oe-git-proxy new file +138 -0
@@ -0,0 +1,138 @@ +#!/bin/bash + +# oe-git-proxy is a simple tool to be via GIT_PROXY_COMMAND. It uses BSD netcat +# to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the +# proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for +# a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR +# masks (192.168.1.0/24). It is known to work with both bash and dash shells. +# +# BSD netcat is provided by netcat-openbsd on Ubuntu and nc on Fedora. +# +# Example ALL_PROXY values: +# ALL_PROXY=socks://socks.example.com:1080 +# ALL_PROXY=https://proxy.example.com:8080 +# +# Copyright (c) 2013, Intel Corporation. +# All rights reserved. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. +# +# AUTHORS +# Darren Hart <[email protected]> + +# Locate the netcat binary +NC=$(which nc 2>/dev/null) +if [ $? -ne 0 ]; then + echo "ERROR: nc binary not in PATH" + exit 1 +fi +METHOD="" + +# Test for a valid IPV4 quad with optional bitmask +valid_ipv4() { + echo $1 | egrep -q "^([1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])){3}(/(3[0-2]|[1-2]?[0-9]))?$" + return $? +} + +# Convert an IPV4 address into a 32bit integer +ipv4_val() { + IP="$1" + SHIFT=24 + VAL=0 + for B in ${IP//./ }; do + VAL=$(($VAL+$(($B<<$SHIFT)))) + SHIFT=$(($SHIFT-8)) + done + echo "$VAL" +} + +# Determine if two IPs are equivalent, or if the CIDR contains the IP +match_ipv4() { + CIDR=$1 + IP=$2 + + if [ -z "${IP%%$CIDR}" ]; then + return 0 + fi + + # Determine the mask bitlength + BITS=${CIDR##*/} + if [ -z "$BITS" ]; then + return 1 + fi + + IPVAL=$(ipv4_val $IP) + IP2VAL=$(ipv4_val ${CIDR%%/*}) + + # OR in the unmasked bits + for i in $(seq 0 $((32-$BITS))); do + IP2VAL=$(($IP2VAL|$((1<<$i)))) + IPVAL=$(($IPVAL|$((1<<$i)))) + done + + if [ $IPVAL -eq $IP2VAL ]; then + return 0 + fi + return 1 +} + +# Test to see if GLOB matches HOST +match_host() { + HOST=$1 + GLOB=$2 + + if [ -z "${HOST%%$GLOB}" ]; then + return 0 + fi + + # Match by netmask + if valid_ipv4 $GLOB; then + HOST_IP=$(gethostip -d $HOST) + if valid_ipv4 $HOST_IP; then + match_ipv4 $GLOB $HOST_IP + if [ $? -eq 0 ]; then + return 0 + fi + fi + fi + + return 1 +} + +# If no proxy is set, just connect directly +if [ -z "$ALL_PROXY" ]; then + exec $NC -X connect "$@" +fi + +# Connect directly to hosts in NO_PROXY +for H in ${NO_PROXY//,/ }; do + if match_host $1 $H; then + METHOD="-X connect" + break + fi +done + +if [ -z "$METHOD" ]; then + # strip the protocol and the trailing slash + PROTO=$(echo $ALL_PROXY | sed -e 's/\([^:]*\):\/\/.*/\1/') + PROXY=$(echo $ALL_PROXY | sed -e 's/.*:\/\/\([^:]*:[0-9]*\).*/\1/') + if [ "$PROTO" = "socks" ]; then + METHOD="-X 5 -x $PROXY" + elif [ "$PROTO" = "https" ]; then + METHOD="-X connect -x $PROXY" + fi +fi + +exec $NC $METHOD "$@"
Signature · 1 line
1.7.5.4
Re: [PATCH 4/7] oe-git-proxy: Add a new comprehensive git proxy script
Quoted 19 lines · Darren Hart
oe-git-proxy.shis a simple tool to be used viaGIT_PROXY_COMMAND. It uses BSD netcat to make SOCKS5 or HTTPS proxy connections. It usesALL_PROXYto determine the proxy server, protocol, and port. It usesNO_PROXYto skip using the proxy for a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It is known to work with both bash and dash shells.V2: Implement recommendations by Enrico Scholz:
o Use exec for the nc calls
o Use "$@" instead of $* to avoid quoting issues inherent with $*
o Use bash explicitly and simplify some of the string manipulations Also:
o Drop the .sh in the name per Otavio Salvador
o Remove a stray debug statementV3: Implement recommendations by Otavio Salvador
o GPL license blurb
o Fix minor typo in comment block
Quoted 5 lines
Signed-off-by Darren Hart <[email protected]>Cc Enrico Scholz <[email protected]>Cc Otavio Salvador <[email protected]>git-proxy cleanup
All those comments ought to be bellow --- or those will be included in commit log.
Quoted 153 lines
Signed-off-by Darren Hart <[email protected]>--- scripts/oe-git-proxy | 138 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 138 insertions(+), 0 deletions(-) create mode 100755 scripts/oe-git-proxyscripts/oe-git-proxy new file +138
-1@@ -0,0 +1,138 @@ +#!/bin/bash + +# oe-git-proxy is a simple tool to be via GIT_PROXY_COMMAND. It uses BSD netcat +# to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the +# proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for +# a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR +# masks (192.168.1.0/24). It is known to work with both bash and dash shells. +# +# BSD netcat is provided by netcat-openbsd on Ubuntu and nc on Fedora. +# +# Example ALL_PROXY values: +# ALL_PROXY=socks://socks.example.com:1080 +# ALL_PROXY=https://proxy.example.com:8080 +# +# Copyright (c) 2013, Intel Corporation. +# All rights reserved. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. +# +# AUTHORS +# Darren Hart <[email protected]> + +# Locate the netcat binary +NC=$(which nc 2>/dev/null) +if [ $? -ne 0 ]; then + echo "ERROR: nc binary not in PATH" + exit 1 +fi +METHOD="" + +# Test for a valid IPV4 quad with optional bitmask +valid_ipv4() { + echo $1 | egrep -q "^([1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])){3}(/(3[0-2]|[1-2]?[0-9]))?$" + return $? +} + +# Convert an IPV4 address into a 32bit integer +ipv4_val() { + IP="$1" + SHIFT=24 + VAL=0 + for B in ${IP//./ }; do + VAL=$(($VAL+$(($B<<$SHIFT)))) + SHIFT=$(($SHIFT-8)) + done + echo "$VAL" +} + +# Determine if two IPs are equivalent, or if the CIDR contains the IP +match_ipv4() { + CIDR=$1 + IP=$2 + + if [ -z "${IP%%$CIDR}" ]; then + return 0 + fi + + # Determine the mask bitlength + BITS=${CIDR##*/} + if [ -z "$BITS" ]; then + return 1 + fi + + IPVAL=$(ipv4_val $IP) + IP2VAL=$(ipv4_val ${CIDR%%/*}) + + # OR in the unmasked bits + for i in $(seq 0 $((32-$BITS))); do + IP2VAL=$(($IP2VAL|$((1<<$i)))) + IPVAL=$(($IPVAL|$((1<<$i)))) + done + + if [ $IPVAL -eq $IP2VAL ]; then + return 0 + fi + return 1 +} + +# Test to see if GLOB matches HOST +match_host() { + HOST=$1 + GLOB=$2 + + if [ -z "${HOST%%$GLOB}" ]; then + return 0 + fi + + # Match by netmask + if valid_ipv4 $GLOB; then + HOST_IP=$(gethostip -d $HOST) + if valid_ipv4 $HOST_IP; then + match_ipv4 $GLOB $HOST_IP + if [ $? -eq 0 ]; then + return 0 + fi + fi + fi + + return 1 +} + +# If no proxy is set, just connect directly +if [ -z "$ALL_PROXY" ]; then + exec $NC -X connect "$@" +fi + +# Connect directly to hosts in NO_PROXY +for H in ${NO_PROXY//,/ }; do + if match_host $1 $H; then + METHOD="-X connect" + break + fi +done + +if [ -z "$METHOD" ]; then + # strip the protocol and the trailing slash + PROTO=$(echo $ALL_PROXY | sed -e 's/\([^:]*\):\/\/.*/\1/') + PROXY=$(echo $ALL_PROXY | sed -e 's/.*:\/\/\([^:]*:[0-9]*\).*/\1/') + if [ "$PROTO" = "socks" ]; then + METHOD="-X 5 -x $PROXY" + elif [ "$PROTO" = "https" ]; then + METHOD="-X connect -x $PROXY" + fi +fi + +exec $NC $METHOD "$@" --1.7.5.4
Signature · 3 lines
Otavio Salvador O.S. Systems E-mail: [email protected] http://www.ossystems.com.br Mobile: +55 53 9981-7854 http://projetos.ossystems.com.br
Re: [PATCH 4/7] oe-git-proxy: Add a new comprehensive git proxy script
Quoted 20 lines · Otavio Salvador
Quoted 19 lines · Darren Hart
oe-git-proxy.shis a simple tool to be used viaGIT_PROXY_COMMAND. It uses BSD netcat to make SOCKS5 or HTTPS proxy connections. It usesALL_PROXYto determine the proxy server, protocol, and port. It usesNO_PROXYto skip using the proxy for a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It is known to work with both bash and dash shells.V2: Implement recommendations by Enrico Scholz:
o Use exec for the nc calls
o Use "$@" instead of $* to avoid quoting issues inherent with $*
o Use bash explicitly and simplify some of the string manipulations Also:
o Drop the .sh in the name per Otavio Salvador
o Remove a stray debug statementV3: Implement recommendations by Otavio Salvador
o GPL license blurb
o Fix minor typo in comment block
Quoted 9 lines
Quoted 5 lines
Signed-off-by Darren Hart <[email protected]>Cc Enrico Scholz <[email protected]>Cc Otavio Salvador <[email protected]>git-proxy cleanup
All those comments ought to be bellow --- or those will be included in commit log.
Which is fine.
Quoted 156 lines
Quoted 153 lines
Signed-off-by Darren Hart <[email protected]>--- scripts/oe-git-proxy | 138 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 138 insertions(+), 0 deletions(-) create mode 100755 scripts/oe-git-proxyscripts/oe-git-proxy new file +138
-1@@ -0,0 +1,138 @@ +#!/bin/bash + +# oe-git-proxy is a simple tool to be via GIT_PROXY_COMMAND. It uses BSD netcat +# to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the +# proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for +# a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR +# masks (192.168.1.0/24). It is known to work with both bash and dash shells. +# +# BSD netcat is provided by netcat-openbsd on Ubuntu and nc on Fedora. +# +# Example ALL_PROXY values: +# ALL_PROXY=socks://socks.example.com:1080 +# ALL_PROXY=https://proxy.example.com:8080 +# +# Copyright (c) 2013, Intel Corporation. +# All rights reserved. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. +# +# AUTHORS +# Darren Hart <[email protected]> + +# Locate the netcat binary +NC=$(which nc 2>/dev/null) +if [ $? -ne 0 ]; then + echo "ERROR: nc binary not in PATH" + exit 1 +fi +METHOD="" + +# Test for a valid IPV4 quad with optional bitmask +valid_ipv4() { + echo $1 | egrep -q "^([1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])){3}(/(3[0-2]|[1-2]?[0-9]))?$" + return $? +} + +# Convert an IPV4 address into a 32bit integer +ipv4_val() { + IP="$1" + SHIFT=24 + VAL=0 + for B in ${IP//./ }; do + VAL=$(($VAL+$(($B<<$SHIFT)))) + SHIFT=$(($SHIFT-8)) + done + echo "$VAL" +} + +# Determine if two IPs are equivalent, or if the CIDR contains the IP +match_ipv4() { + CIDR=$1 + IP=$2 + + if [ -z "${IP%%$CIDR}" ]; then + return 0 + fi + + # Determine the mask bitlength + BITS=${CIDR##*/} + if [ -z "$BITS" ]; then + return 1 + fi + + IPVAL=$(ipv4_val $IP) + IP2VAL=$(ipv4_val ${CIDR%%/*}) + + # OR in the unmasked bits + for i in $(seq 0 $((32-$BITS))); do + IP2VAL=$(($IP2VAL|$((1<<$i)))) + IPVAL=$(($IPVAL|$((1<<$i)))) + done + + if [ $IPVAL -eq $IP2VAL ]; then + return 0 + fi + return 1 +} + +# Test to see if GLOB matches HOST +match_host() { + HOST=$1 + GLOB=$2 + + if [ -z "${HOST%%$GLOB}" ]; then + return 0 + fi + + # Match by netmask + if valid_ipv4 $GLOB; then + HOST_IP=$(gethostip -d $HOST) + if valid_ipv4 $HOST_IP; then + match_ipv4 $GLOB $HOST_IP + if [ $? -eq 0 ]; then + return 0 + fi + fi + fi + + return 1 +} + +# If no proxy is set, just connect directly +if [ -z "$ALL_PROXY" ]; then + exec $NC -X connect "$@" +fi + +# Connect directly to hosts in NO_PROXY +for H in ${NO_PROXY//,/ }; do + if match_host $1 $H; then + METHOD="-X connect" + break + fi +done + +if [ -z "$METHOD" ]; then + # strip the protocol and the trailing slash + PROTO=$(echo $ALL_PROXY | sed -e 's/\([^:]*\):\/\/.*/\1/') + PROXY=$(echo $ALL_PROXY | sed -e 's/.*:\/\/\([^:]*:[0-9]*\).*/\1/') + if [ "$PROTO" = "socks" ]; then + METHOD="-X 5 -x $PROXY" + elif [ "$PROTO" = "https" ]; then + METHOD="-X connect -x $PROXY" + fi +fi + +exec $NC $METHOD "$@" --1.7.5.4
Signature · 3 lines
Darren Hart Intel Open Source Technology Center Yocto Project - Technical Lead - Linux Kernel
Re: [PATCH 4/7] oe-git-proxy: Add a new comprehensive git proxy script
Quoted 27 lines · Darren Hart
Quoted 24 lines · Otavio Salvador
Quoted 23 lines · Darren Hart
oe-git-proxy.shis a simple tool to be used viaGIT_PROXY_COMMAND. It uses BSD netcat to make SOCKS5 or HTTPS proxy connections. It usesALL_PROXYto determine the proxy server, protocol, and port. It usesNO_PROXYto skip using the proxy for a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It is known to work with both bash and dash shells.V2: Implement recommendations by Enrico Scholz:
o Use exec for the nc calls
o Use "$@" instead of $* to avoid quoting issues inherent with $*
o Use bash explicitly and simplify some of the string manipulations Also:
o Drop the .sh in the name per Otavio Salvador
o Remove a stray debug statementV3: Implement recommendations by Otavio Salvador
o GPL license blurb
o Fix minor typo in comment blockSigned-off-by Darren Hart <[email protected]>Cc Enrico Scholz <[email protected]>Cc Otavio Salvador <[email protected]>
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
git-proxy cleanup
All those comments ought to be bellow --- or those will be included in commit log.
Which is fine.
Well, usually the log of the revisions are not included in the commit logs; any special reason for it to be done different?
Signature · 3 lines
Otavio Salvador O.S. Systems E-mail: [email protected] http://www.ossystems.com.br Mobile: +55 53 9981-7854 http://projetos.ossystems.com.br
Re: [PATCH 4/7] oe-git-proxy: Add a new comprehensive git proxy script
Quoted 32 lines · Otavio Salvador
Quoted 4 lines · Darren Hart
Quoted 1 line · Otavio Salvador
On Fri, Feb 8, 2013 at 8:27 PM, Darren Hart <[email protected]>
Quoted 25 lines · wrote:
Quoted 25 lines
Quoted 24 lines
oe-git-proxy.shis a simple tool to be used viaGIT_PROXY_COMMAND. It uses BSD netcat to make SOCKS5 or HTTPS proxy connections. It usesALL_PROXYto determine the proxy server, protocol, and port. It usesNO_PROXYto skip using the proxy for a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It is known to work with both bash and dash shells.V2: Implement recommendations by Enrico Scholz:
o Use exec for the nc calls
o Use "$@" instead of $* to avoid quoting issues inherent with $*
o Use bash explicitly and simplify some of the string manipulations Also:
o Drop the .sh in the name per Otavio Salvador
o Remove a stray debug statementV3: Implement recommendations by Otavio Salvador
o GPL license blurb
o Fix minor typo in comment blockSigned-off-by Darren Hart <[email protected]>Cc Enrico Scholz <[email protected]>Cc Otavio Salvador <[email protected]>git-proxy cleanup
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
All those comments ought to be bellow --- or those will be included in commit log.
Which is fine.
Well, usually the log of the revisions are not included in the commit logs; any special reason for it to be done different?
Randomly jumping in,
It's not all that uncommon for kernel developers to keep the revision history
in long logs. It's used as a way to track the intermediate history, give review credit .. or just "because".
So while not always done, it isn't unheard of. I have no strong opinion one way or the other, just thought I'd throw that out!
Cheers,
Bruce
Quoted 11 lines
--
Otavio SalvadorO.S. Systems
E-mail: [email protected] http://www.ossystems.com.br
Mobile: +55 53 9981-7854 http://projetos.ossystems.com.br_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Re: [PATCH 4/7] oe-git-proxy: Add a new comprehensive git proxy script
Quoted 12 lines · Bruce Ashfield
Quoted 5 lines
Quoted 3 lines
All those comments ought to be bellow --- or those will be included in commit log.
Which is fine.
Well, usually the log of the revisions are not included in the commit logs; any special reason for it to be done different?
Randomly jumping in,
Quoted 7 lines
It's not all that uncommon for kernel developers to keep the revision history
in long logs. It's used as a way to track the intermediate history, give review credit .. or just "because".So while not always done, it isn't unheard of. I have no strong opinion one way or the other, just thought I'd throw that out!
I'll just leave that to the maintainer. If whoever pushes it objects, they can drop it, or ask me to resend.
Signature · 3 lines
Darren Hart Intel Open Source Technology Center Yocto Project - Technical Lead - Linux Kernel
PATCH v3 5/7 meta-yocto: Document new oe-git-proxy in site.conf.sample
In reply to Darren HartThe new oe-git-proxy.sh should address all git proxying needs. Document its usage in meta-yocto/conf/site.conf.sample and remove references to the old mechanisms.
V2: Separate out the meta-yocto changes from the oe-core changes
--- meta-yocto/conf/site.conf.sample | 35 +++++++++-------------------------- 1 files changed, 9 insertions(+), 26 deletions(-)
meta-yocto/conf/site.conf.sample +9 -26
@@ -16,32 +16,15 @@ SCONF_VERSION = "1" #http-proxy-port = 81 # -# Uncomment to cause git to use the proxy host specificed -# although this only works for http -#GIT_PROXY_HOST = "proxy.example.com" -#GIT_PROXY_PORT = "81" -#export GIT_PROXY_COMMAND = "${COREBASE}/scripts/oe-git-proxy-command" - -# Set to yes to have a gitconfig generated for handling proxies; you -# might not want this if you have all that set in your global git -# configuration. If you don't enable it, the rest of the entries -# (_PROXY_IGNORE, etc) don't really work that well -#GIT_CORE_CONFIG = "Yes" - -# Space separate list of hosts to ignore for GIT proxy -#GIT_PROXY_IGNORE = "host.server.com another.server.com" - -# If SOCKS is available run the following command to comple a simple transport -# gcc scripts/oe-git-proxy-socks.c -o oe-git-proxy-socks -# and then share that binary somewhere in PATH, then use the following settings -#GIT_PROXY_HOST = "proxy.example.com" -#GIT_PROXY_PORT = "81" - -# GIT_PROXY_COMMAND is used by git to override all proxy settings from -# configuration files, so we prefix OE_ to avoid breaking havoc on the -# generated (or local) gitconfig's. -#OE_GIT_PROXY_COMMAND = "${COREBASE}/scripts/oe-git-proxy-socks-command" - +# To use git with a proxy, you must use an external git proxy command, such as +# the one provided by scripts/oe-git-proxy.sh. To use this script, copy it to +# your PATH and uncomment the following: +#GIT_PROXY_COMMAND="oe-git-proxy" +#ALL_PROXY="socks://socks.example.com:1080" +#or +#ALL_PROXY="https://proxy.example.com:8080" +# If you wish to use certain hosts without the proxy, specify them in NO_PROXY. +# See the script for details on syntax. # Uncomment this to use a shared download directory #DL_DIR = "/some/shared/download/directory/"
Signature · 1 line
1.7.5.4
PATCH v3 6/7 oe-git-proxy*: Remove previous git proxy solutions
In reply to Darren HartThe new oe-git-proxy should address all git proxying needs, remove the previous scripts.
V2: Separate the removal of the old scripts into their own patch
--- scripts/oe-git-proxy-command | 10 - scripts/oe-git-proxy-socks-command | 23 - scripts/oe-git-proxy-socks.c | 2982 ------------------------------------ 3 files changed, 0 insertions(+), 3015 deletions(-) delete mode 100755 scripts/oe-git-proxy-command delete mode 100755 scripts/oe-git-proxy-socks-command delete mode 100755 scripts/oe-git-proxy-socks.c
scripts/oe-git-proxy-command deleted +0 -10
@@ -1,10 +0,0 @@ -#! /bin/bash - -(echo "CONNECT $1:$2 HTTP/1.0"; - echo; - cat ) | nc $GIT_PROXY_HOST $GIT_PROXY_PORT | - -(read a; - read a; - read a; - cat )
scripts/oe-git-proxy-socks-command deleted +0 -23
@@ -1,23 +0,0 @@ -#! /bin/bash -SCRIPTDIR=`dirname $0` -# Check oe-git-proxy-socks exists -PROXYSOCKS=`which oe-git-proxy-socks 2> /dev/null` -if [ -z "$PROXYSOCKS" -a -e "$SCRIPTDIR/oe-git-proxy-socks.c" ]; then - # If not try and build it - gcc $SCRIPTDIR/oe-git-proxy-socks.c -o $SCRIPTDIR/oe-git-proxy-socks -fi -PROXYSOCKS=`which oe-git-proxy-socks 2> /dev/null` -if [ ! -x "$PROXYSOCKS" ]; then - # If that fails, we can see if netcat (nc) is available - NETCAT=`which nc 2> /dev/null` - if [ ! -x "$NETCAT" ]; then - # If that fails, explain to the user - echo "Unable to find oe-git-proxy-socks. This is usually created with the command" - echo "'gcc scripts/oe-git-proxy-socks.c -o scripts/oe-git-proxy-socks' which we tried" - echo "but it doesn't seem to have worked. Please compile the binary manually." - echo "Alternativly, install nc (netcat) on this machine." - exit 1 - fi - exec $NETCAT -x $GIT_PROXY_HOST:$GIT_PROXY_PORT "$@" -fi -oe-git-proxy-socks -S $GIT_PROXY_HOST:$GIT_PROXY_PORT $@
scripts/oe-git-proxy-socks.c deleted +0 -2982
2,966 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
1.7.5.4
Part of the diff is not shown here. View the raw message.
PATCH v3 7/7 oe-git-proxy: Use socat instead of BSD nc
In reply to Darren HartBSD nc was commonly available on the current distros until Fedora 18 appears to have dropped it. socat appears to be a reasonable replacement with availability on Fedora and Ubuntu and going back some time as well.
Update the script to use the socat syntax.
Simplify the logic a bit by using exec for the no-proxy-needed cases.
--- scripts/oe-git-proxy | 59 ++++++++++++++++++++++--------------------------- 1 files changed, 27 insertions(+), 32 deletions(-)
scripts/oe-git-proxy +27 -32
@@ -1,13 +1,11 @@ #!/bin/bash -# oe-git-proxy is a simple tool to be via GIT_PROXY_COMMAND. It uses BSD netcat +# oe-git-proxy is a simple tool to be via GIT_PROXY_COMMAND. It uses socat # to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the # proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for # a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR # masks (192.168.1.0/24). It is known to work with both bash and dash shells. # -# BSD netcat is provided by netcat-openbsd on Ubuntu and nc on Fedora. -# # Example ALL_PROXY values: # ALL_PROXY=socks://socks.example.com:1080 # ALL_PROXY=https://proxy.example.com:8080
@@ -15,27 +13,13 @@ # Copyright (c) 2013, Intel Corporation. # All rights reserved. # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. -# # AUTHORS # Darren Hart <[email protected]> # Locate the netcat binary -NC=$(which nc 2>/dev/null) +SOCAT=$(which socat 2>/dev/null) if [ $? -ne 0 ]; then - echo "ERROR: nc binary not in PATH" + echo "ERROR: socat binary not in PATH" exit 1 fi METHOD=""
@@ -111,28 +95,39 @@ match_host() { return 1 } -# If no proxy is set, just connect directly +# If no proxy is set or needed, just connect directly +METHOD="TCP:$1:$2" + if [ -z "$ALL_PROXY" ]; then - exec $NC -X connect "$@" + exec $SOCAT STDIO $METHOD fi # Connect directly to hosts in NO_PROXY for H in ${NO_PROXY//,/ }; do if match_host $1 $H; then - METHOD="-X connect" - break + exec $SOCAT STDIO $METHOD fi done -if [ -z "$METHOD" ]; then - # strip the protocol and the trailing slash - PROTO=$(echo $ALL_PROXY | sed -e 's/\([^:]*\):\/\/.*/\1/') - PROXY=$(echo $ALL_PROXY | sed -e 's/.*:\/\/\([^:]*:[0-9]*\).*/\1/') - if [ "$PROTO" = "socks" ]; then - METHOD="-X 5 -x $PROXY" - elif [ "$PROTO" = "https" ]; then - METHOD="-X connect -x $PROXY" +# Proxy is necessary, determine protocol, server, and port +PROTO=$(echo $ALL_PROXY | sed -e 's/\([^:]*\):\/\/.*/\1/') +PROXY=$(echo $ALL_PROXY | sed -e 's/.*:\/\/\([^:]*\).*/\1/') +PORT=$(echo $ALL_PROXY | sed -e 's/.*:\([0-9]*\)\/?$/\1/') +if [ "$PORT" = "$ALL_PROXY" ]; then + PORT="" +fi + +if [ "$PROTO" = "socks" ]; then + if [ -z "$PORT" ]; then + PORT="1080" + fi + METHOD="SOCKS4:$PROXY:$1:$2,socksport=$PORT" +else + # Assume PROXY (http, https, etc) + if [ -z "$PORT" ]; then + PORT="8080" fi + METHOD="PROXY:$PROXY:$1:$2,proxyport=$PORT" fi -exec $NC $METHOD "$@" +exec $SOCAT STDIO $METHOD
Signature · 1 line
1.7.5.4