Skip to contentSkip to conversation

Git proxy handling updates (now with socat)

PATCH In review Darren Hart 3 participants 13 messages Feb 8, 2013 – Feb 10, 2013

lore
Share & tools
lore
#0 lore raw

v3 cover Git proxy handling updates (now with socat)

The current git fetcher environment and proxy handling between bitbake, base.bbclass, and oe-buildenv-internal has become stale and is largely non-functional. The crux of the problem is that GIT_CONFIG is no longer honored by anything other than the git-config command, rendering the GIT_CORE_CONFIG and the assocaited proxy handling routines obsolete.

(the bitbake patches have been merged and dropped from this series)

The GIT_PROXY_COMMAND, ALL_PROXY, and NO_PROXY environment variables are more than adequate for storing the required information to manage proxies with a git helper script.

These patches series aggressively removes the old stuff, fixes up some missing environment settings, and adds a new git proxy handling script.

Note: These patches DO NOT address the bitbake gui builder.py references to git
proxy settings!

I felt this series was best viewed as a whole, so I'm sending it to poky AND oe-core. Anything else seemed too likely to result in confusion, missing context, and lots and lots of redundant discussion.

V2: Updates to the new oe-git-proxy script from feedback received
Split out meta-yocto changes
Split out removal of old scripts
Minor commit log corrections

V3: Fedora 18 dropped BSD nc support, so add a patch converting oe-git-proxy
to use socat which is broadly available. HTTP proxy still remains untested.

The following changes since commit 20b4dcace222d6b1d659a8813b4affad075ec0c6:

dbus-native: Don't install dbus-launch (2013-02-08 14:50:38 +0000)

are available in the git repository at:
git://git.yoctoproject.org/poky-contrib dvhart/gitproxy
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=dvhart/gitproxy

Darren Hart (7):
oe-buildenv-internal: Remove GIT variables from BB_ENV_EXTRAWHITE
oe-buildenv-internal: Add upper and lower case proxy vars to
BB_ENV_EXTRAWHITE
base.bbclass: Remove generate_git_config()
oe-git-proxy: Add a new comprehensive git proxy script
meta-yocto: Document new oe-git-proxy in site.conf.sample oe-git-proxy*: Remove previous git proxy solutions
oe-git-proxy: Use socat instead of BSD nc

 meta-yocto/conf/site.conf.sample   |   35 +-
 meta/classes/base.bbclass          |   21 -
 scripts/oe-buildenv-internal       |    2 +-
 scripts/oe-git-proxy               |  133 ++
 scripts/oe-git-proxy-command       |   10 -
 scripts/oe-git-proxy-socks-command |   23 -
 scripts/oe-git-proxy-socks.c       | 2982 ------------------------------------
 7 files changed, 143 insertions(+), 3063 deletions(-)
 create mode 100755 scripts/oe-git-proxy
 delete mode 100755 scripts/oe-git-proxy-command
 delete mode 100755 scripts/oe-git-proxy-socks-command
 delete mode 100755 scripts/oe-git-proxy-socks.c
Signature · 1 line
1.7.5.4
#1 lore raw

PATCH v3 1/7 oe-buildenv-internal: Remove GIT variables from BB_ENV_EXTRAWHITE

The following variables perform no function outside of bitbake:

GIT_CONFIG
GIT_PROXY_HOST
GIT_PROXY_PORT
GIT_PROXY_IGNORE

GIT_CONFIG only affects the git-config command which is not relevant to the fetcher. This was previously used with the OE GIT_CORE_CONFIG variable which would provide a basic git config to use instead of the user's config. This usage was deprecated by git for over a year now:

http://git.661346.n2.nabble.com/Overriding-gitconfig-using-GIT-CONFIG-td6680977

GIT_PROXY_HOST and GIT_PROXY_PORT are not used by git.

GIT_PROXY_IGNORE was an OE construct used to create the custom git config and had no meaning outside of the OE environment. It is not used by git.

Remove these variables from the fetcher environment.

Users wishing to configure git to work with a proxy should define the GIT_PROXY_COMMAND environment variable to use an external script. NO_PROXY can be used within this script to skip the proxy for certain hosts.

---
 scripts/oe-buildenv-internal |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
scripts/oe-buildenv-internal +1 -1
index c3a30fb..044b6f9 100755
@@ -80,4 +80,4 @@ unset BITBAKEDIR
 # Used by the runqemu script
 export BUILDDIR
 export PATH
-export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER SCREENDIR"
+export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy NO_PROXY SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND SOCKS5_PASSWD SOCKS5_USER SCREENDIR"
Signature · 1 line
1.7.5.4
#2 lore raw

PATCH v3 2/7 oe-buildenv-internal: Add upper and lower case proxy vars to BB_ENV_EXTRAWHITE

In reply to Darren Hart

Applications are inconsistent in their use of upper and lower case proxy variables. Curl, for example, specifies NO_PROXY (not no_proxy) in the man page (changed in 2009 [1]). Avoid proxy issues by ensuring both the upper and lower case versions of each proxy variable are available in the environment for the fetcher commands.

Add FTPS_PROXY and ftps_proxy to the list as well.

1. http://curl.haxx.se/mail/tracker-2009-04/0012.html

---
 scripts/oe-buildenv-internal |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
scripts/oe-buildenv-internal +1 -1
index 044b6f9..7783574 100755
@@ -80,4 +80,4 @@ unset BITBAKEDIR
 # Used by the runqemu script
 export BUILDDIR
 export PATH
-export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy NO_PROXY SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND SOCKS5_PASSWD SOCKS5_USER SCREENDIR"
+export BB_ENV_EXTRAWHITE="MACHINE DISTRO TCMODE TCLIBC HTTP_PROXY http_proxy HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy FTPS_PROXY ftps_proxy ALL_PROXY all_proxy NO_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND SOCKS5_PASSWD SOCKS5_USER SCREENDIR"
Signature · 1 line
1.7.5.4
#3 lore raw

PATCH v3 3/7 base.bbclass: Remove generate_git_config()

In reply to Darren Hart

git no longer supports the use of GIT_CONFIG which defeats the purpose of GIT_CORE_CONFIG and the generate_git_config() function. Remove it.

---
 meta/classes/base.bbclass |   21 ---------------------
 1 files changed, 0 insertions(+), 21 deletions(-)
meta/classes/base.bbclass +0 -21
index 1c284b7..4d69caf 100644
@@ -132,26 +132,6 @@ python base_do_unpack() {
         raise bb.build.FuncFailed(e)
 }
 
-GIT_CONFIG_PATH = "${STAGING_DIR_NATIVE}/etc"
-GIT_CONFIG = "${GIT_CONFIG_PATH}/gitconfig"
-
-def generate_git_config(e):
-    if e.data.getVar('GIT_CORE_CONFIG', True):
-        gitconfig_path = e.data.getVar('GIT_CONFIG', True)
-        proxy_command = "    gitProxy = %s\n" % e.data.getVar('OE_GIT_PROXY_COMMAND', True)
-
-        bb.mkdirhier(e.data.expand("${GIT_CONFIG_PATH}"))
-        if (os.path.exists(gitconfig_path)):
-            os.remove(gitconfig_path)
-
-        f = open(gitconfig_path, 'w')
-        f.write("[core]\n")
-        ignore_hosts = e.data.getVar('GIT_PROXY_IGNORE', True).split()
-        for ignore_host in ignore_hosts:
-            f.write("    gitProxy = none for %s\n" % ignore_host)
-        f.write(proxy_command)
-        f.close
-
 def pkgarch_mapping(d):
     # Compatibility mappings of TUNE_PKGARCH (opt in)
     if d.getVar("PKGARCHCOMPAT_ARMV7A", True):
@@ -310,7 +290,6 @@ addhandler base_eventhandler
 python base_eventhandler() {
     if isinstance(e, bb.event.ConfigParsed):
         e.data.setVar('BB_VERSION', bb.__version__)
-        generate_git_config(e)
         pkgarch_mapping(e.data)
         preferred_ml_updates(e.data)
         e.data.appendVar('DISTRO_FEATURES', oe.utils.features_backfill("DISTRO_FEATURES", e.data))
Signature · 1 line
1.7.5.4
#4 lore raw

PATCH v3 4/7 oe-git-proxy: Add a new comprehensive git proxy script

In reply to Darren Hart

oe-git-proxy.sh is a simple tool to be used via GIT_PROXY_COMMAND. It uses BSD netcat to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It is known to work with both bash and dash shells.

V2: Implement recommendations by Enrico Scholz:
o Use exec for the nc calls
o Use "$@" instead of $* to avoid quoting issues inherent with $*
o Use bash explicitly and simplify some of the string manipulations Also:
o Drop the .sh in the name per Otavio Salvador
o Remove a stray debug statement

V3: Implement recommendations by Otavio Salvador
o GPL license blurb
o Fix minor typo in comment block

Cc Enrico Scholz <[email protected]>
Cc Otavio Salvador <[email protected]>

git-proxy cleanup

---
 scripts/oe-git-proxy |  138 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 138 insertions(+), 0 deletions(-)
 create mode 100755 scripts/oe-git-proxy
scripts/oe-git-proxy new file +138 -0
new file mode 100755
index 0000000..4c2f179
@@ -0,0 +1,138 @@
+#!/bin/bash
+
+# oe-git-proxy is a simple tool to be via GIT_PROXY_COMMAND. It uses BSD netcat
+# to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the
+# proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for
+# a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR
+# masks (192.168.1.0/24). It is known to work with both bash and dash shells.
+#
+# BSD netcat is provided by netcat-openbsd on Ubuntu and nc on Fedora.
+#
+# Example ALL_PROXY values:
+# ALL_PROXY=socks://socks.example.com:1080
+# ALL_PROXY=https://proxy.example.com:8080
+#
+# Copyright (c) 2013, Intel Corporation.
+# All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+#
+# AUTHORS
+# Darren Hart <[email protected]>
+
+# Locate the netcat binary
+NC=$(which nc 2>/dev/null)
+if [ $? -ne 0 ]; then
+	echo "ERROR: nc binary not in PATH"
+	exit 1
+fi
+METHOD=""
+
+# Test for a valid IPV4 quad with optional bitmask
+valid_ipv4() {
+	echo $1 | egrep -q "^([1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])){3}(/(3[0-2]|[1-2]?[0-9]))?$"
+	return $?
+}
+
+# Convert an IPV4 address into a 32bit integer
+ipv4_val() {
+	IP="$1"
+	SHIFT=24
+	VAL=0
+	for B in ${IP//./ }; do
+		VAL=$(($VAL+$(($B<<$SHIFT))))
+		SHIFT=$(($SHIFT-8))
+	done
+	echo "$VAL"
+}
+
+# Determine if two IPs are equivalent, or if the CIDR contains the IP
+match_ipv4() {
+	CIDR=$1
+	IP=$2
+
+	if [ -z "${IP%%$CIDR}" ]; then
+		return 0
+	fi
+
+	# Determine the mask bitlength
+	BITS=${CIDR##*/}
+	if [ -z "$BITS" ]; then
+		return 1
+	fi
+
+	IPVAL=$(ipv4_val $IP)
+	IP2VAL=$(ipv4_val ${CIDR%%/*})
+
+	# OR in the unmasked bits
+	for i in $(seq 0 $((32-$BITS))); do
+		IP2VAL=$(($IP2VAL|$((1<<$i))))
+		IPVAL=$(($IPVAL|$((1<<$i))))
+	done
+
+	if [ $IPVAL -eq $IP2VAL ]; then
+		return 0
+	fi
+	return 1
+}
+
+# Test to see if GLOB matches HOST
+match_host() {
+	HOST=$1
+	GLOB=$2
+
+	if [ -z "${HOST%%$GLOB}" ]; then
+		return 0
+	fi
+
+	# Match by netmask
+	if valid_ipv4 $GLOB; then
+		HOST_IP=$(gethostip -d $HOST)
+		if valid_ipv4 $HOST_IP; then
+			match_ipv4 $GLOB $HOST_IP
+			if [ $? -eq 0 ]; then
+				return 0
+			fi
+		fi
+	fi
+
+	return 1
+}
+
+# If no proxy is set, just connect directly
+if [ -z "$ALL_PROXY" ]; then
+	exec $NC -X connect "$@"
+fi
+
+# Connect directly to hosts in NO_PROXY
+for H in ${NO_PROXY//,/ }; do
+	if match_host $1 $H; then
+		METHOD="-X connect"
+		break
+	fi
+done
+
+if [ -z "$METHOD" ]; then
+	# strip the protocol and the trailing slash
+	PROTO=$(echo $ALL_PROXY | sed -e 's/\([^:]*\):\/\/.*/\1/')
+	PROXY=$(echo $ALL_PROXY | sed -e 's/.*:\/\/\([^:]*:[0-9]*\).*/\1/')
+	if [ "$PROTO" = "socks" ]; then
+		METHOD="-X 5 -x $PROXY"
+	elif [ "$PROTO" = "https" ]; then
+		METHOD="-X connect -x $PROXY"
+	fi
+fi
+
+exec $NC $METHOD "$@"
Signature · 1 line
1.7.5.4
#5 lore raw

Re: [PATCH 4/7] oe-git-proxy: Add a new comprehensive git proxy script

Quoted 19 lines · Darren Hart

oe-git-proxy.sh is a simple tool to be used via GIT_PROXY_COMMAND. It uses BSD netcat to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It is known to work with both bash and dash shells.

V2: Implement recommendations by Enrico Scholz:
o Use exec for the nc calls
o Use "$@" instead of $* to avoid quoting issues inherent with $*
o Use bash explicitly and simplify some of the string manipulations Also:
o Drop the .sh in the name per Otavio Salvador
o Remove a stray debug statement

V3: Implement recommendations by Otavio Salvador
o GPL license blurb
o Fix minor typo in comment block

Quoted 5 lines
Cc Enrico Scholz <[email protected]>
Cc Otavio Salvador <[email protected]>

git-proxy cleanup

All those comments ought to be bellow --- or those will be included in commit log.

Quoted 153 lines
---
 scripts/oe-git-proxy |  138 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 138 insertions(+), 0 deletions(-)
 create mode 100755 scripts/oe-git-proxy
scripts/oe-git-proxy new file +138 -1
new file mode 100755
index 0000000..4c2f179
@@ -0,0 +1,138 @@
+#!/bin/bash
+
+# oe-git-proxy is a simple tool to be via GIT_PROXY_COMMAND. It uses BSD netcat
+# to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the
+# proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for
+# a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR
+# masks (192.168.1.0/24). It is known to work with both bash and dash shells.
+#
+# BSD netcat is provided by netcat-openbsd on Ubuntu and nc on Fedora.
+#
+# Example ALL_PROXY values:
+# ALL_PROXY=socks://socks.example.com:1080
+# ALL_PROXY=https://proxy.example.com:8080
+#
+# Copyright (c) 2013, Intel Corporation.
+# All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+#
+# AUTHORS
+# Darren Hart <[email protected]>
+
+# Locate the netcat binary
+NC=$(which nc 2>/dev/null)
+if [ $? -ne 0 ]; then
+       echo "ERROR: nc binary not in PATH"
+       exit 1
+fi
+METHOD=""
+
+# Test for a valid IPV4 quad with optional bitmask
+valid_ipv4() {
+       echo $1 | egrep -q "^([1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])){3}(/(3[0-2]|[1-2]?[0-9]))?$"
+       return $?
+}
+
+# Convert an IPV4 address into a 32bit integer
+ipv4_val() {
+       IP="$1"
+       SHIFT=24
+       VAL=0
+       for B in ${IP//./ }; do
+               VAL=$(($VAL+$(($B<<$SHIFT))))
+               SHIFT=$(($SHIFT-8))
+       done
+       echo "$VAL"
+}
+
+# Determine if two IPs are equivalent, or if the CIDR contains the IP
+match_ipv4() {
+       CIDR=$1
+       IP=$2
+
+       if [ -z "${IP%%$CIDR}" ]; then
+               return 0
+       fi
+
+       # Determine the mask bitlength
+       BITS=${CIDR##*/}
+       if [ -z "$BITS" ]; then
+               return 1
+       fi
+
+       IPVAL=$(ipv4_val $IP)
+       IP2VAL=$(ipv4_val ${CIDR%%/*})
+
+       # OR in the unmasked bits
+       for i in $(seq 0 $((32-$BITS))); do
+               IP2VAL=$(($IP2VAL|$((1<<$i))))
+               IPVAL=$(($IPVAL|$((1<<$i))))
+       done
+
+       if [ $IPVAL -eq $IP2VAL ]; then
+               return 0
+       fi
+       return 1
+}
+
+# Test to see if GLOB matches HOST
+match_host() {
+       HOST=$1
+       GLOB=$2
+
+       if [ -z "${HOST%%$GLOB}" ]; then
+               return 0
+       fi
+
+       # Match by netmask
+       if valid_ipv4 $GLOB; then
+               HOST_IP=$(gethostip -d $HOST)
+               if valid_ipv4 $HOST_IP; then
+                       match_ipv4 $GLOB $HOST_IP
+                       if [ $? -eq 0 ]; then
+                               return 0
+                       fi
+               fi
+       fi
+
+       return 1
+}
+
+# If no proxy is set, just connect directly
+if [ -z "$ALL_PROXY" ]; then
+       exec $NC -X connect "$@"
+fi
+
+# Connect directly to hosts in NO_PROXY
+for H in ${NO_PROXY//,/ }; do
+       if match_host $1 $H; then
+               METHOD="-X connect"
+               break
+       fi
+done
+
+if [ -z "$METHOD" ]; then
+       # strip the protocol and the trailing slash
+       PROTO=$(echo $ALL_PROXY | sed -e 's/\([^:]*\):\/\/.*/\1/')
+       PROXY=$(echo $ALL_PROXY | sed -e 's/.*:\/\/\([^:]*:[0-9]*\).*/\1/')
+       if [ "$PROTO" = "socks" ]; then
+               METHOD="-X 5 -x $PROXY"
+       elif [ "$PROTO" = "https" ]; then
+               METHOD="-X connect -x $PROXY"
+       fi
+fi
+
+exec $NC $METHOD "$@"
--

1.7.5.4

Signature · 3 lines
Otavio Salvador                             O.S. Systems
E-mail: [email protected]  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br
#6 lore raw

Re: [PATCH 4/7] oe-git-proxy: Add a new comprehensive git proxy script

Quoted 20 lines · Otavio Salvador
Quoted 19 lines · Darren Hart

oe-git-proxy.sh is a simple tool to be used via GIT_PROXY_COMMAND. It uses BSD netcat to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It is known to work with both bash and dash shells.

V2: Implement recommendations by Enrico Scholz:
o Use exec for the nc calls
o Use "$@" instead of $* to avoid quoting issues inherent with $*
o Use bash explicitly and simplify some of the string manipulations Also:
o Drop the .sh in the name per Otavio Salvador
o Remove a stray debug statement

V3: Implement recommendations by Otavio Salvador
o GPL license blurb
o Fix minor typo in comment block

Quoted 9 lines
Quoted 5 lines
Cc Enrico Scholz <[email protected]>
Cc Otavio Salvador <[email protected]>

git-proxy cleanup

All those comments ought to be bellow --- or those will be included in commit log.

Which is fine.

Quoted 156 lines
Quoted 153 lines
---
 scripts/oe-git-proxy |  138 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 138 insertions(+), 0 deletions(-)
 create mode 100755 scripts/oe-git-proxy
scripts/oe-git-proxy new file +138 -1
new file mode 100755
index 0000000..4c2f179
@@ -0,0 +1,138 @@
+#!/bin/bash
+
+# oe-git-proxy is a simple tool to be via GIT_PROXY_COMMAND. It uses BSD netcat
+# to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the
+# proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for
+# a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR
+# masks (192.168.1.0/24). It is known to work with both bash and dash shells.
+#
+# BSD netcat is provided by netcat-openbsd on Ubuntu and nc on Fedora.
+#
+# Example ALL_PROXY values:
+# ALL_PROXY=socks://socks.example.com:1080
+# ALL_PROXY=https://proxy.example.com:8080
+#
+# Copyright (c) 2013, Intel Corporation.
+# All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+#
+# AUTHORS
+# Darren Hart <[email protected]>
+
+# Locate the netcat binary
+NC=$(which nc 2>/dev/null)
+if [ $? -ne 0 ]; then
+       echo "ERROR: nc binary not in PATH"
+       exit 1
+fi
+METHOD=""
+
+# Test for a valid IPV4 quad with optional bitmask
+valid_ipv4() {
+       echo $1 | egrep -q "^([1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])){3}(/(3[0-2]|[1-2]?[0-9]))?$"
+       return $?
+}
+
+# Convert an IPV4 address into a 32bit integer
+ipv4_val() {
+       IP="$1"
+       SHIFT=24
+       VAL=0
+       for B in ${IP//./ }; do
+               VAL=$(($VAL+$(($B<<$SHIFT))))
+               SHIFT=$(($SHIFT-8))
+       done
+       echo "$VAL"
+}
+
+# Determine if two IPs are equivalent, or if the CIDR contains the IP
+match_ipv4() {
+       CIDR=$1
+       IP=$2
+
+       if [ -z "${IP%%$CIDR}" ]; then
+               return 0
+       fi
+
+       # Determine the mask bitlength
+       BITS=${CIDR##*/}
+       if [ -z "$BITS" ]; then
+               return 1
+       fi
+
+       IPVAL=$(ipv4_val $IP)
+       IP2VAL=$(ipv4_val ${CIDR%%/*})
+
+       # OR in the unmasked bits
+       for i in $(seq 0 $((32-$BITS))); do
+               IP2VAL=$(($IP2VAL|$((1<<$i))))
+               IPVAL=$(($IPVAL|$((1<<$i))))
+       done
+
+       if [ $IPVAL -eq $IP2VAL ]; then
+               return 0
+       fi
+       return 1
+}
+
+# Test to see if GLOB matches HOST
+match_host() {
+       HOST=$1
+       GLOB=$2
+
+       if [ -z "${HOST%%$GLOB}" ]; then
+               return 0
+       fi
+
+       # Match by netmask
+       if valid_ipv4 $GLOB; then
+               HOST_IP=$(gethostip -d $HOST)
+               if valid_ipv4 $HOST_IP; then
+                       match_ipv4 $GLOB $HOST_IP
+                       if [ $? -eq 0 ]; then
+                               return 0
+                       fi
+               fi
+       fi
+
+       return 1
+}
+
+# If no proxy is set, just connect directly
+if [ -z "$ALL_PROXY" ]; then
+       exec $NC -X connect "$@"
+fi
+
+# Connect directly to hosts in NO_PROXY
+for H in ${NO_PROXY//,/ }; do
+       if match_host $1 $H; then
+               METHOD="-X connect"
+               break
+       fi
+done
+
+if [ -z "$METHOD" ]; then
+       # strip the protocol and the trailing slash
+       PROTO=$(echo $ALL_PROXY | sed -e 's/\([^:]*\):\/\/.*/\1/')
+       PROXY=$(echo $ALL_PROXY | sed -e 's/.*:\/\/\([^:]*:[0-9]*\).*/\1/')
+       if [ "$PROTO" = "socks" ]; then
+               METHOD="-X 5 -x $PROXY"
+       elif [ "$PROTO" = "https" ]; then
+               METHOD="-X connect -x $PROXY"
+       fi
+fi
+
+exec $NC $METHOD "$@"
--

1.7.5.4

Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Technical Lead - Linux Kernel
#7 lore raw

Re: [PATCH 4/7] oe-git-proxy: Add a new comprehensive git proxy script

Quoted 27 lines · Darren Hart
Quoted 24 lines · Otavio Salvador
Quoted 23 lines · Darren Hart

oe-git-proxy.sh is a simple tool to be used via GIT_PROXY_COMMAND. It uses BSD netcat to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It is known to work with both bash and dash shells.

V2: Implement recommendations by Enrico Scholz:
o Use exec for the nc calls
o Use "$@" instead of $* to avoid quoting issues inherent with $*
o Use bash explicitly and simplify some of the string manipulations Also:
o Drop the .sh in the name per Otavio Salvador
o Remove a stray debug statement

V3: Implement recommendations by Otavio Salvador
o GPL license blurb
o Fix minor typo in comment block

Cc Enrico Scholz <[email protected]>
Cc Otavio Salvador <[email protected]>
Quoted 7 lines
Quoted 5 lines
Quoted 1 line

git-proxy cleanup

All those comments ought to be bellow --- or those will be included in commit log.

Which is fine.

Well, usually the log of the revisions are not included in the commit logs; any special reason for it to be done different?

Signature · 3 lines
Otavio Salvador                             O.S. Systems
E-mail: [email protected]  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br
#8 lore raw

Re: [PATCH 4/7] oe-git-proxy: Add a new comprehensive git proxy script

Quoted 32 lines · Otavio Salvador
Quoted 4 lines · Darren Hart
Quoted 1 line · Otavio Salvador

On Fri, Feb 8, 2013 at 8:27 PM, Darren Hart <[email protected]>

Quoted 25 lines · wrote:
Quoted 25 lines
Quoted 24 lines

oe-git-proxy.sh is a simple tool to be used via GIT_PROXY_COMMAND. It uses BSD netcat to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It is known to work with both bash and dash shells.

V2: Implement recommendations by Enrico Scholz:
o Use exec for the nc calls
o Use "$@" instead of $* to avoid quoting issues inherent with $*
o Use bash explicitly and simplify some of the string manipulations Also:
o Drop the .sh in the name per Otavio Salvador
o Remove a stray debug statement

V3: Implement recommendations by Otavio Salvador
o GPL license blurb
o Fix minor typo in comment block

Cc Enrico Scholz <[email protected]>
Cc Otavio Salvador <[email protected]>

git-proxy cleanup

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

All those comments ought to be bellow --- or those will be included in commit log.

Which is fine.

Well, usually the log of the revisions are not included in the commit logs; any special reason for it to be done different?

Randomly jumping in,

It's not all that uncommon for kernel developers to keep the revision history
in long logs. It's used as a way to track the intermediate history, give review credit .. or just "because".

So while not always done, it isn't unheard of. I have no strong opinion one way or the other, just thought I'd throw that out!

Cheers,

Bruce

Quoted 11 lines

--
Otavio Salvador O.S. Systems
E-mail: [email protected] http://www.ossystems.com.br
Mobile: +55 53 9981-7854 http://projetos.ossystems.com.br

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end"
#9 lore raw

Re: [PATCH 4/7] oe-git-proxy: Add a new comprehensive git proxy script

Quoted 12 lines · Bruce Ashfield
Quoted 5 lines
Quoted 3 lines

All those comments ought to be bellow --- or those will be included in commit log.

Which is fine.

Well, usually the log of the revisions are not included in the commit logs; any special reason for it to be done different?

Randomly jumping in,

Quoted 7 lines

It's not all that uncommon for kernel developers to keep the revision history
in long logs. It's used as a way to track the intermediate history, give review credit .. or just "because".

So while not always done, it isn't unheard of. I have no strong opinion one way or the other, just thought I'd throw that out!

I'll just leave that to the maintainer. If whoever pushes it objects, they can drop it, or ask me to resend.

Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Technical Lead - Linux Kernel
#10 lore raw

PATCH v3 5/7 meta-yocto: Document new oe-git-proxy in site.conf.sample

In reply to Darren Hart

The new oe-git-proxy.sh should address all git proxying needs. Document its usage in meta-yocto/conf/site.conf.sample and remove references to the old mechanisms.

V2: Separate out the meta-yocto changes from the oe-core changes

---
 meta-yocto/conf/site.conf.sample |   35 +++++++++--------------------------
 1 files changed, 9 insertions(+), 26 deletions(-)
meta-yocto/conf/site.conf.sample +9 -26
index 68d1da9..b2d01d0 100644
@@ -16,32 +16,15 @@ SCONF_VERSION = "1"
 #http-proxy-port = 81
 #
 
-# Uncomment to cause git to use the proxy host specificed 
-# although this only works for http
-#GIT_PROXY_HOST = "proxy.example.com"
-#GIT_PROXY_PORT = "81"
-#export GIT_PROXY_COMMAND = "${COREBASE}/scripts/oe-git-proxy-command"
-
-# Set to yes to have a gitconfig generated for handling proxies; you
-# might not want this if you have all that set in your global git
-# configuration. If you don't enable it, the rest of the entries
-# (_PROXY_IGNORE, etc) don't really work that well
-#GIT_CORE_CONFIG = "Yes"
-
-# Space separate list of hosts to ignore for GIT proxy
-#GIT_PROXY_IGNORE = "host.server.com another.server.com"
-
-# If SOCKS is available run the following command to comple a simple transport
-# gcc scripts/oe-git-proxy-socks.c -o oe-git-proxy-socks
-# and then share that binary somewhere in PATH, then use the following settings
-#GIT_PROXY_HOST = "proxy.example.com"
-#GIT_PROXY_PORT = "81"
-
-# GIT_PROXY_COMMAND is used by git to override all proxy settings from
-# configuration files, so we prefix OE_ to avoid breaking havoc on the
-# generated (or local) gitconfig's.
-#OE_GIT_PROXY_COMMAND = "${COREBASE}/scripts/oe-git-proxy-socks-command"
-
+# To use git with a proxy, you must use an external git proxy command, such as
+# the one provided by scripts/oe-git-proxy.sh. To use this script, copy it to
+# your PATH and uncomment the following:
+#GIT_PROXY_COMMAND="oe-git-proxy"
+#ALL_PROXY="socks://socks.example.com:1080"
+#or
+#ALL_PROXY="https://proxy.example.com:8080"
+# If you wish to use certain hosts without the proxy, specify them in NO_PROXY.
+# See the script for details on syntax.
 
 # Uncomment this to use a shared download directory
 #DL_DIR = "/some/shared/download/directory/"
Signature · 1 line
1.7.5.4
#11 lore raw

PATCH v3 6/7 oe-git-proxy*: Remove previous git proxy solutions

In reply to Darren Hart

The new oe-git-proxy should address all git proxying needs, remove the previous scripts.

V2: Separate the removal of the old scripts into their own patch

---
 scripts/oe-git-proxy-command       |   10 -
 scripts/oe-git-proxy-socks-command |   23 -
 scripts/oe-git-proxy-socks.c       | 2982 ------------------------------------
 3 files changed, 0 insertions(+), 3015 deletions(-)
 delete mode 100755 scripts/oe-git-proxy-command
 delete mode 100755 scripts/oe-git-proxy-socks-command
 delete mode 100755 scripts/oe-git-proxy-socks.c
scripts/oe-git-proxy-command deleted +0 -10
deleted file mode 100755
index d31f85a..0000000
@@ -1,10 +0,0 @@
-#! /bin/bash
-
-(echo "CONNECT $1:$2 HTTP/1.0"; 
- echo; 
- cat ) | nc $GIT_PROXY_HOST $GIT_PROXY_PORT | 
-  
-(read a; 
- read a; 
- read a; 
- cat )
\ No newline at end of file
scripts/oe-git-proxy-socks-command deleted +0 -23
deleted file mode 100755
index 8acffb5..0000000
@@ -1,23 +0,0 @@
-#! /bin/bash
-SCRIPTDIR=`dirname $0`
-# Check oe-git-proxy-socks exists
-PROXYSOCKS=`which oe-git-proxy-socks 2> /dev/null`
-if [ -z "$PROXYSOCKS" -a -e "$SCRIPTDIR/oe-git-proxy-socks.c" ]; then
-	# If not try and build it
-	gcc $SCRIPTDIR/oe-git-proxy-socks.c -o $SCRIPTDIR/oe-git-proxy-socks
-fi
-PROXYSOCKS=`which oe-git-proxy-socks 2> /dev/null`
-if [ ! -x "$PROXYSOCKS" ]; then
-	# If that fails, we can see if netcat (nc) is available
-	NETCAT=`which nc 2> /dev/null`
-	if [ ! -x "$NETCAT" ]; then
-		# If that fails, explain to the user
-		echo "Unable to find oe-git-proxy-socks. This is usually created with the command"
-		echo "'gcc scripts/oe-git-proxy-socks.c -o scripts/oe-git-proxy-socks' which we tried"
-		echo "but it doesn't seem to have worked. Please compile the binary manually."
-		echo "Alternativly, install nc (netcat) on this machine."
-		exit 1
-	fi
-	exec $NETCAT -x $GIT_PROXY_HOST:$GIT_PROXY_PORT "$@"
-fi
-oe-git-proxy-socks -S $GIT_PROXY_HOST:$GIT_PROXY_PORT $@
scripts/oe-git-proxy-socks.c deleted +0 -2982

2,966 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
1.7.5.4

Part of the diff is not shown here. View the raw message.

#12 lore raw

PATCH v3 7/7 oe-git-proxy: Use socat instead of BSD nc

In reply to Darren Hart

BSD nc was commonly available on the current distros until Fedora 18 appears to have dropped it. socat appears to be a reasonable replacement with availability on Fedora and Ubuntu and going back some time as well.

Update the script to use the socat syntax.
Simplify the logic a bit by using exec for the no-proxy-needed cases.

---
 scripts/oe-git-proxy |   59 ++++++++++++++++++++++---------------------------
 1 files changed, 27 insertions(+), 32 deletions(-)
scripts/oe-git-proxy +27 -32
index 4c2f179..98191fa 100755
@@ -1,13 +1,11 @@
 #!/bin/bash
 
-# oe-git-proxy is a simple tool to be via GIT_PROXY_COMMAND. It uses BSD netcat
+# oe-git-proxy is a simple tool to be via GIT_PROXY_COMMAND. It uses socat
 # to make SOCKS5 or HTTPS proxy connections. It uses ALL_PROXY to determine the
 # proxy server, protocol, and port. It uses NO_PROXY to skip using the proxy for
 # a comma delimited list of hosts, host globs (*.example.com), IPs, or CIDR
 # masks (192.168.1.0/24). It is known to work with both bash and dash shells.
 #
-# BSD netcat is provided by netcat-openbsd on Ubuntu and nc on Fedora.
-#
 # Example ALL_PROXY values:
 # ALL_PROXY=socks://socks.example.com:1080
 # ALL_PROXY=https://proxy.example.com:8080
@@ -15,27 +13,13 @@
 # Copyright (c) 2013, Intel Corporation.
 # All rights reserved.
 #
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
-#
 # AUTHORS
 # Darren Hart <[email protected]>
 
 # Locate the netcat binary
-NC=$(which nc 2>/dev/null)
+SOCAT=$(which socat 2>/dev/null)
 if [ $? -ne 0 ]; then
-	echo "ERROR: nc binary not in PATH"
+	echo "ERROR: socat binary not in PATH"
 	exit 1
 fi
 METHOD=""
@@ -111,28 +95,39 @@ match_host() {
 	return 1
 }
 
-# If no proxy is set, just connect directly
+# If no proxy is set or needed, just connect directly
+METHOD="TCP:$1:$2"
+
 if [ -z "$ALL_PROXY" ]; then
-	exec $NC -X connect "$@"
+	exec $SOCAT STDIO $METHOD
 fi
 
 # Connect directly to hosts in NO_PROXY
 for H in ${NO_PROXY//,/ }; do
 	if match_host $1 $H; then
-		METHOD="-X connect"
-		break
+		exec $SOCAT STDIO $METHOD
 	fi
 done
 
-if [ -z "$METHOD" ]; then
-	# strip the protocol and the trailing slash
-	PROTO=$(echo $ALL_PROXY | sed -e 's/\([^:]*\):\/\/.*/\1/')
-	PROXY=$(echo $ALL_PROXY | sed -e 's/.*:\/\/\([^:]*:[0-9]*\).*/\1/')
-	if [ "$PROTO" = "socks" ]; then
-		METHOD="-X 5 -x $PROXY"
-	elif [ "$PROTO" = "https" ]; then
-		METHOD="-X connect -x $PROXY"
+# Proxy is necessary, determine protocol, server, and port
+PROTO=$(echo $ALL_PROXY | sed -e 's/\([^:]*\):\/\/.*/\1/')
+PROXY=$(echo $ALL_PROXY | sed -e 's/.*:\/\/\([^:]*\).*/\1/')
+PORT=$(echo $ALL_PROXY | sed -e 's/.*:\([0-9]*\)\/?$/\1/')
+if [ "$PORT" = "$ALL_PROXY" ]; then
+	PORT=""
+fi
+
+if [ "$PROTO" = "socks" ]; then
+	if [ -z "$PORT" ]; then
+		PORT="1080"
+	fi
+	METHOD="SOCKS4:$PROXY:$1:$2,socksport=$PORT"
+else
+	# Assume PROXY (http, https, etc)
+	if [ -z "$PORT" ]; then
+		PORT="8080"
 	fi
+	METHOD="PROXY:$PROXY:$1:$2,proxyport=$PORT"
 fi
 
-exec $NC $METHOD "$@"
+exec $SOCAT STDIO $METHOD
Signature · 1 line
1.7.5.4