Add proxy download support
PATCH 1/1 Add proxy download support
This patch adds support for downloading sources over a http proxy. Supports wget with http, https & ftp targets, bzr, hg, svn. Git does work with forcing https instead of git protocol. Cvs is not supported, but Buildroot does not currently have any packages relying on cvs repos (plus this would fall back to the Buildroot mirror over http).
--- Config.in | 25 +++++++++++++++++++++++++ Makefile | 9 +++++++++ support/download/wrapper | 26 ++++++++++++++++++++++++++ 3 files changed, 60 insertions(+)
Config.in +25 -0
@@ -253,6 +253,31 @@ config BR2_CPAN_MIRROR endmenu +config BR2_PROXY + bool "Use http proxy" + help + Use a http proxy when downloading sources with various protocols. + +if BR2_PROXY + +config BR2_PROXY_URL + string "Http proxy URL" + help + Http proxy URL in the form of <IP or FQDN>:<port>. + +config BR2_PROXY_USER + string "Http proxy username" + help + Http proxy username (if needed). + +config BR2_PROXY_PASS + string "Http proxy password" + help + Http proxy password (if needed). + PLEASE NOTE! This password will be saved in the .config file! + +endif + config BR2_JLEVEL int "Number of jobs to run simultaneously (0 for auto)" default "0"
Makefile +9 -0
@@ -338,6 +338,15 @@ TARGET_SKELETON = $(TOPDIR)/system/skeleton # should not be used as the root filesystem. TARGET_DIR_WARNING_FILE = $(TARGET_DIR)/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM +ifeq ($(BR2_PROXY),y) +BR_PROXY_URL = $(call qstrip,$(BR2_PROXY_URL)) +export BR_PROXY_URL +BR_PROXY_USER = $(call qstrip,$(BR2_PROXY_USER)) +export BR_PROXY_USER +BR_PROXY_PASS = $(call qstrip,$(BR2_PROXY_PASS)) +export BR_PROXY_PASS +endif + ifeq ($(BR2_CCACHE),y) CCACHE := $(HOST_DIR)/usr/bin/ccache BR_CACHE_DIR = $(call qstrip,$(BR2_CCACHE_DIR))
support/download/wrapper +26 -0
@@ -44,6 +44,32 @@ tmpf="${tmpd}/output" # Doing the 'cd' here rather than in all helpers is easier. cd "${tmpd}" +# Set the proxy environment variables. +# cvs doesn't support proxies OOB, but there is currently no cvs +# based package in Buildroot. Also, in that case after the download +# fails, the wget method would be used from sources.buildroot.net. +if [ -n "${BR_PROXY_URL}" ]; then +# SVN does not support http_proxy environment variables, so we use +# it's commandline arguments +proxy_host=`echo $BR_PROXY_URL | cut -d":" -f1` +proxy_port=`echo $BR_PROXY_URL | cut -d":" -f2` +SVN="${SVN} --config-option servers:global:http-proxy-host=${proxy_host} \ +--config-option servers:global:http-proxy-port=${proxy_port}" +if [ -n "${BR_PROXY_USER}" ] && [ -n "${BR_PROXY_PASS}" ]; then +SVN="${SVN} --config-option servers:global:http-proxy-username=${BR_PROXY_USER} \ +--config-option servers:global:http-proxy-password=${BR_PROXY_PASS}" +proxy="http://${BR_PROXY_USER}:${BR_PROXY_PASS}@${BR_PROXY_URL}" +else +proxy="http://${BR_PROXY_URL}" +fi +export http_proxy=${proxy} +export https_proxy=${proxy} +export ftp_proxy=${proxy} +# While git does support http_proxy environment variables, we need to use +# the config option below to force proxying over https instead of git +GIT="${GIT} -c url.https://.insteadOf=git://" +fi + # If the helper fails, we can just remove the temporary directory to # remove all the cruft it may have left behind. Then we just exit in # error too.
Signature · 1 line
1.7.10.4
Yann E. MORIN
NAKPATCH 1/1 Add proxy download support
Karoly, All,
On 2014-10-21 16:53 +0200, Karoly Kasza spake thusly:
Quoted 8 lines
This patch adds support for downloading sources over a http proxy. Supports wget with http, https & ftp targets, bzr, hg, svn. Git does work with forcing https instead of git protocol. Cvs is not supported, but Buildroot does not currently have any packages relying on cvs repos (plus this would fall back to the Buildroot mirror over http).
Signed-off-by Karoly Kasza <[email protected]>
NAK.
This should belong to the user's envirnment.
I once tried to handle proxy support in crosstool-NG, and after trying to handle the exponentially growing cases (http proxy, with or without authentication, socks proxy...)
Also, this solution is not bullet proof, because it leaves alone non-http protocols: cvs, git, svn, scp...
Actually, it does take care of git, but some proxies do not allow the CONNECT to anything else than 443, so we're back to square one for the git protocol anyway.
If we were to add a proxy setting, then users would expect it to work for all kind of downloads, which is absolutely impossible to achieve.
So, we do not have a 100% coverage, and quite some is left out.
So, I prefer we not have any of this proxy settings in Buildroot at all, and just rely on the user's environment to be properly set up.
Regards,
Yann E. MORIN.
Quoted 102 lines
--- Config.in | 25 +++++++++++++++++++++++++ Makefile | 9 +++++++++ support/download/wrapper | 26 ++++++++++++++++++++++++++ 3 files changed, 60 insertions(+)Config.in +25
-0@@ -253,6 +253,31 @@ config BR2_CPAN_MIRROR endmenu +config BR2_PROXY + bool "Use http proxy" + help + Use a http proxy when downloading sources with various protocols. + +if BR2_PROXY + +config BR2_PROXY_URL + string "Http proxy URL" + help + Http proxy URL in the form of <IP or FQDN>:<port>. + +config BR2_PROXY_USER + string "Http proxy username" + help + Http proxy username (if needed). + +config BR2_PROXY_PASS + string "Http proxy password" + help + Http proxy password (if needed). + PLEASE NOTE! This password will be saved in the .config file! + +endif + config BR2_JLEVEL int "Number of jobs to run simultaneously (0 for auto)" default "0"Makefile +9
-0@@ -338,6 +338,15 @@ TARGET_SKELETON = $(TOPDIR)/system/skeleton # should not be used as the root filesystem. TARGET_DIR_WARNING_FILE = $(TARGET_DIR)/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM +ifeq ($(BR2_PROXY),y) +BR_PROXY_URL = $(call qstrip,$(BR2_PROXY_URL)) +export BR_PROXY_URL +BR_PROXY_USER = $(call qstrip,$(BR2_PROXY_USER)) +export BR_PROXY_USER +BR_PROXY_PASS = $(call qstrip,$(BR2_PROXY_PASS)) +export BR_PROXY_PASS +endif + ifeq ($(BR2_CCACHE),y) CCACHE := $(HOST_DIR)/usr/bin/ccache BR_CACHE_DIR = $(call qstrip,$(BR2_CCACHE_DIR))support/download/wrapper +26
-0@@ -44,6 +44,32 @@ tmpf="${tmpd}/output" # Doing the 'cd' here rather than in all helpers is easier. cd "${tmpd}" +# Set the proxy environment variables. +# cvs doesn't support proxies OOB, but there is currently no cvs +# based package in Buildroot. Also, in that case after the download +# fails, the wget method would be used from sources.buildroot.net. +if [ -n "${BR_PROXY_URL}" ]; then +# SVN does not support http_proxy environment variables, so we use +# it's commandline arguments +proxy_host=`echo $BR_PROXY_URL | cut -d":" -f1` +proxy_port=`echo $BR_PROXY_URL | cut -d":" -f2` +SVN="${SVN} --config-option servers:global:http-proxy-host=${proxy_host} \ +--config-option servers:global:http-proxy-port=${proxy_port}" +if [ -n "${BR_PROXY_USER}" ] && [ -n "${BR_PROXY_PASS}" ]; then +SVN="${SVN} --config-option servers:global:http-proxy-username=${BR_PROXY_USER} \ +--config-option servers:global:http-proxy-password=${BR_PROXY_PASS}" +proxy="http://${BR_PROXY_USER}:${BR_PROXY_PASS}@${BR_PROXY_URL}" +else +proxy="http://${BR_PROXY_URL}" +fi +export http_proxy=${proxy} +export https_proxy=${proxy} +export ftp_proxy=${proxy} +# While git does support http_proxy environment variables, we need to use +# the config option below to force proxying over https instead of git +GIT="${GIT} -c url.https://.insteadOf=git://" +fi + # If the helper fails, we can just remove the temporary directory to # remove all the cruft it may have left behind. Then we just exit in # error too.Signature · 1 line
1.7.10.4
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 Add proxy download support
Quoted 4 lines
NAK.
Quoted 9 lines
This should belong to the user's envirnment.
OK, but then I think this should be stated in the user manual.
I once tried to handle proxy support in crosstool-NG, and after trying to handle the exponentially growing cases (http proxy, with or without authentication, socks proxy...)
This handles standard http proxies (hence the name) with or without auth. Most proxies (and even corporate ones) fall into this category (like Squid, MS Forefront, etc).
Quoted 4 lines
Also, this solution is not bullet proof, because it leaves alone non-http protocols: cvs, git, svn, scp...
Not true. Only CVS won't work, but:
A, There is currently no CVS based package in Buildroot
B, If there would, it would fall back the sources.buildroot.com, which is http
C, I may have mislooked and CVS can be proxied - if You allow me to send a V2, I will introduce that too
All other download methods work with the distributed packages. Should at least, I only downloaded a few packages from every possible type.
AFAIK SCP & SSH are not standard download methods in the packaging system, they are
added functionality for custom packages.
Quoted 4 lines
Actually, it does take care of git, but some proxies do not allow the CONNECT to anything else than 443, so we're back to square one for the git protocol anyway.
Git always falls back back to https (the "instead of" directive) and the standard 443,
because git:// protocol simply cannot be proxied.
Quoted 3 lines
If we were to add a proxy setting, then users would expect it to work for all kind of downloads, which is absolutely impossible to achieve.
It does work with all existing packages (again, I found no CVS, SCP or SSH based packages in the distribution).
I can proxy scp or ssh over http, but that relies on the user's proxy server - there are proxies which won't allow a
CONNECT method for ssh.
Quoted 3 lines
So, we do not have a 100% coverage, and quite some is left out.
Please name, I will try my best, to send a V2 covering those methods. Currently it support the 5 types that
are used in the packages.
Quoted 4 lines
So, I prefer we not have any of this proxy settings in Buildroot at all, and just rely on the user's environment to be properly set up.
I think setting up all methods in the user environment is possible, but sometimes can be quite complicated.
Thank you for your opinion, could you state in which case would you accept the patch?
I think I can add CVS support, and ssh/scp support - which means ALL 8 download helpers would be able to
use http proxies.
And again, this is a http proxy patch, I didn't aim to support all proxy types in the world, as that would be
probably pointless.
Best regards,
Karoly
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20141021/32f00315/attachment.html>
PATCH 1/1 Add proxy download support
K?roly, All,
On 2014-10-21 20:23 +0200, K?roly Kasza spake thusly:
Quoted 4 lines
Quoted 3 lines
NAK.
This should belong to the user's envirnment.OK, but then I think this should be stated in the user manual.
This is by far not specific to Buildroot at all, but we can add a blurb in the manual, telling the user that Buildroot uses the user's settings.
Quoted 7 lines
Quoted 4 lines
I once tried to handle proxy support in crosstool-NG, and after trying to handle the exponentially growing cases (http proxy, with or without authentication, socks proxy...)
This handles standard http proxies (hence the name) with or without auth. Most proxies (and even corporate ones) fall into this category (like Squid, MS Forefront, etc).
That's just am incorrect assumption. I know first-hand of network setups that use much complex proxy setups...
Quoted 7 lines
Quoted 3 lines
Also, this solution is not bullet proof, because it leaves alone non-http protocols: cvs, git, svn, scp...
Not true. Only CVS won't work, but:
A, There is currently no CVS based package in Buildroot
B, If there would, it would fall back the sources.buildroot.com, which is http
The initial reason for sources.buildroot.com is to be a fallback location in case an upstream disapears after we cut a release, so we ensure that releases continue to work for the foreseeable future.
Quoted 2 lines
C, I may have mislooked and CVS can be proxied - if You allow me to send a V2, I will introduce that too
Well, it seems it is possible to proxy cvs through an http proxy, but that proxy must allow the CONNECT to the CVS port, which a lot of enterprise proxies just disallow.
For example, my company's proxy only allow CONNECT to the port 443, and blocks anything else. Which means I can not use it to connect to:
- a git server using the git protocol
- a SVN server using the svn protocol
- a CVS server
- an ssh server
Quoted 5 lines
All other download methods work with the distributed packages. Should at least, I only downloaded a few packages from every possible type.
AFAIK SCP & SSH are not standard download methods in the packaging system, they are added functionality for custom packages.
True, but nothing would prevent a user to add a custom package that fetches via scp from a remote server outside the enterprise network (e.g. to retrieve a package from a subsidiary network, for example).
Quoted 6 lines
Quoted 3 lines
Actually, it does take care of git, but some proxies do not allow the CONNECT to anything else than 443, so we're back to square one for the git protocol anyway.
Git always falls back back to https (the "instead of" directive) and the standard 443, because
git://protocol simply cannot be proxied.
Both assumptions are wrong. There are git servers that serves only via the git protocol, so there is no possible fallback to http.
And even if there were, the relative paths of the two URLs may be different between the two schemes, so just using http:// instead ofgit:// is not sufficient.
And third, the git protocol *can* be proxified, even through an http proxy, but it requires that proxy to be configured to allow the CONNECT method to any port, not only 443.
Quoted 5 lines
Quoted 3 lines
If we were to add a proxy setting, then users would expect it to work for all kind of downloads, which is absolutely impossible to achieve.
It does work with all existing packages (again, I found no CVS, SCP or SSH based packages in the distribution).
Yes, they are not used in Buildroot itself. But if we do changes to the download infra, all our download helpers should behave the same.
Quoted 2 lines
I can proxy scp or ssh over http, but that relies on the user's proxy server - there are proxies which won't allow a CONNECT method for ssh.
Exactly, so goes for CVS and GIT.
Quoted 4 lines
Quoted 2 lines
So, we do not have a 100% coverage, and quite some is left out.
Please name, I will try my best, to send a V2 covering those methods. Currently it support the 5 types that are used in the packages.
Well, I agree that it does cover all the methods Buildroot uses, but not all the methods Buildroot uses.
What are we gonna explain our users that the proxy settings he put in the menuconfig are not being used, because a package in its BR2_EXTERNAL tree uses a download method that we do not know how to proxify?
Quoted 5 lines
Quoted 3 lines
So, I prefer we not have any of this proxy settings in Buildroot at all, and just rely on the user's environment to be properly set up.
I think setting up all methods in the user environment is possible, but sometimes can be quite complicated.
Well, it is really easy:
export http-proxy='http://user:pass at proxy:port/' export https_proxy="${http_proxy}"
export ftp_proxy="${http_proxy}"
. tsocks -on
Et voil?, all things are proxified (one just has to edit /etc/tsocks.conf but that is something we really can do ourselves anyway.... Yes, I know we can use TSOCKS_CONF_FILE=/path/to/my/tsocks.conf)
Quoted 8 lines
Thank you for your opinion, could you state in which case would you accept the patch?
I think I can add CVS support, and ssh/scp support - which means ALL 8 download helpers would be able to
use http proxies.
And again, this is a http proxy patch, I didn't aim to support all proxy types in the world, as that would be
probably pointless.
I still believe we should not have this in Buildroot at all. That is my opinion, others may disagree with me. Let them speak up! ;-)
I know this can be a bit frustrating, but I can assure you I have already gone down this road in the past, and it is not a road I would like to go down again...
To me, the best would be to just document that Buildroot uses whatever settings are set in the user's system.
BTW, I should know about proxies: absolutely everything I do on my home machine *is* proxified via either an http or a socks proxy. Even this very mail will leave my home network via a proxy... ;-]
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 Add proxy download support
Quoted 2 lines · Yann E. MORIN
I still believe we should not have this in Buildroot at all. That is my opinion, others may disagree with me. Let them speak up! ;-)
Sorry, Yann, I won't speak up because I completely agree with you :-)
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
PATCH 1/1 Add proxy download support
In reply to Yann E. MORINHi Yann,
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
I once tried to handle proxy support in crosstool-NG, and after trying to handle the exponentially growing cases (http proxy, with or without authentication, socks proxy...)
Quoted 8 lines
Quoted 2 lines
This handles standard http proxies (hence the name) with or without auth. Most proxies (and even corporate ones) fall into this category (like
Squid,
Quoted 1 line
MS Forefront, etc).
That's just am incorrect assumption. I know first-hand of network setups that use much complex proxy setups...
I know some enterprises with tens of thousands computers who use http proxies
from the users point of view (US & German multinationals). Using anything else than http proxies are overcomplicated for them and can not be authorized
reliably (even using SSO). They of course use strict security settings, IPS/IDS chains,
antivirus and antispam filters, but from the end user perspective they are all just http proxies...
NAT with L7 filter or transparent proxying are rare, using socks in a corporate is
even more rare (in my experience at least) - even if it would be available. I would still say that http proxies are the most common solution, and that's not just an
assumption. See
https://www.websense.com/assets/reports/report-gartner-magic-quadrant-for-security-web-gateway-2011-en.pdf
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Also, this solution is not bullet proof, because it leaves alone non-http protocols: cvs, git, svn, scp...
Not true. Only CVS won't work, but:
A, There is currently no CVS based package in Buildroot
B, If there would, it would fall back the sources.buildroot.com, which
Quoted 7 lines
is
Quoted 1 line
http
The initial reason for sources.buildroot.com is to be a fallback location in case an upstream disapears after we cut a release, so we ensure that releases continue to work for the foreseeable future.
Aye, but this does not make it unusable if the user falls back there from another protocol.
Quoted 9 lines
Quoted 2 lines
C, I may have mislooked and CVS can be proxied - if You allow me to send
a
Quoted 1 line
V2, I will introduce that too
Well, it seems it is possible to proxy cvs through an http proxy, but that proxy must allow the CONNECT to the CVS port, which a lot of enterprise proxies just disallow.
Very true. In that case you simply cannot use a proxy, you won't be able to use one.
The proxies' security settings are of course outside of the scope of Buildroot.
Quoted 4 lines
For example, my company's proxy only allow CONNECT to the port 443, and blocks anything else. Which means I can not use it to connect to:
- a git server using the git protocol
I did not find any information about git allowed over proxy without 3rd parties (socat, etc).
The insteadof method is the standard for git.
Quoted 4 lines
- a SVN server using the svn protocol
- a CVS server
- an ssh server
All true. In these cases you simply won't be able to use a proxy. Not from Buildroot, nor
from the user's environment. Again, security settings can prevent anything.
Quoted 5 lines
Quoted 3 lines
All other download methods work with the distributed packages. Should at least, I only downloaded a few packages from every possible
type.
Quoted 1 line
Quoted 8 lines
Quoted 1 line
AFAIK SCP & SSH are not standard download methods in the packaging
system,
Quoted 1 line
they are added functionality for custom packages.
True, but nothing would prevent a user to add a custom package that fetches via scp from a remote server outside the enterprise network (e.g. to retrieve a package from a subsidiary network, for example).
Aye, in that case I could still add some logic to the wrapper, like using netcat or corkscrew
(which are both 3rd party ofc).
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
Actually, it does take care of git, but some proxies do not allow the CONNECT to anything else than 443, so we're back to square one for the git protocol anyway.
Git always falls back back to https (the "instead of" directive) and the standard 443, because
git://protocol simply cannot be proxied.Both assumptions are wrong. There are git servers that serves only via the git protocol, so there is no possible fallback to http.
I'm not saying that it's falling back by standard. It falls back because of the directive.
Ofc servers who don't serve over https are can not be proxied this way - only with socat.
Quoted 4 lines
And even if there were, the relative paths of the two URLs may be different between the two schemes, so just using http:// instead of
git://is not sufficient.
In that case this is a server based problem.
Quoted 4 lines
And third, the git protocol *can* be proxified, even through an http proxy, but it requires that proxy to be configured to allow the CONNECT method to any port, not only 443.
But only with a 3rd party software, not by default. The default recommended method is
falling back.
Quoted 11 lines
Quoted 5 lines
Quoted 4 lines
If we were to add a proxy setting, then users would expect it to work for all kind of downloads, which is absolutely impossible to achieve.
It does work with all existing packages (again, I found no CVS, SCP or
SSH
Quoted 1 line
based packages in the distribution).
Yes, they are not used in Buildroot itself. But if we do changes to the download infra, all our download helpers should behave the same.
You are completely right.
Quoted 6 lines
Quoted 3 lines
I can proxy scp or ssh over http, but that relies on the user's proxy server - there are proxies which won't allow a CONNECT method for ssh.
Exactly, so goes for CVS and GIT.
Aye, in this case the users won't be able to proxy from their environments.
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
So, we do not have a 100% coverage, and quite some is left out.
Please name, I will try my best, to send a V2 covering those methods. Currently it support the 5 types that are used in the packages.
Well, I agree that it does cover all the methods Buildroot uses, but not all the methods Buildroot uses.
You mean Buildroot could use?
Quoted 5 lines
What are we gonna explain our users that the proxy settings he put in the menuconfig are not being used, because a package in its
BR2_EXTERNALtree uses a download method that we do not know how to proxify?
Exactly. Buildroot would support using http proxies (if allowed by the proxy administrator)
with download methods that are pre-packaged in Buildroot. I still think that's fair, more than
telling to solve it for yourself.
Quoted 16 lines
Quoted 2 lines
Quoted 2 lines
So, I prefer we not have any of this proxy settings in Buildroot at
all,
Quoted 4 lines
Quoted 2 lines
and just rely on the user's environment to be properly set up.
I think setting up all methods in the user environment is possible, but sometimes can be quite complicated.
Well, it is really easy:
export http-proxy='http://user:pass at proxy:port/' export https_proxy="${http_proxy}"
export ftp_proxy="${http_proxy}"
. tsocks -on
Quoted 4 lines
Et voil?, all things are proxified (one just has to edit
/etc/tsocks.confbut that is something we really can do ourselves anyway.... Yes, I know we can use TSOCKS_CONF_FILE=/path/to/my/tsocks.conf)
And nope - just as you said, this would work with wget, bzr, hg, and would not with
cvs, git://, scp & ssh. For those the user would have to setup individual startup configs.
Also, while I personally like Socks, I find it a very rarely used protocol outside of socks
through ssh.
Quoted 11 lines
Quoted 2 lines
Thank you for your opinion, could you state in which case would you
accept
Quoted 7 lines
the patch?
I think I can add CVS support, and ssh/scp support - which means ALL 8 download helpers would be able to
use http proxies.
And again, this is a http proxy patch, I didn't aim to support all proxy types in the world, as that would be
probably pointless.
Quoted 3 lines
I still believe we should not have this in Buildroot at all. That is my opinion, others may disagree with me. Let them speak up! ;-)
I can accept it (even if I don't agree with it).
Than again, from another point of view, if somebody can not solve him/herself the
using of a proxy, why would he/she want to compile a complete embedded linux system from sources...
Quoted 5 lines
I know this can be a bit frustrating, but I can assure you I have already gone down this road in the past, and it is not a road I would like to go down again...
OK.
Quoted 8 lines
To me, the best would be to just document that Buildroot uses whatever settings are set in the user's system.
BTW, I should know about proxies: absolutely everything I do on my home machine *is* proxified via either an http or a socks proxy. Even this very mail will leave my home network via a proxy... ;-]
:D
Best regards,
Karoly
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20141022/eb10635b/attachment.html>
PATCH 1/1 Add proxy download support
In reply to Karoly KaszaKaroly, All,
On 2014-10-21 16:53 +0200, Karoly Kasza spake thusly:
Quoted 8 lines
This patch adds support for downloading sources over a http proxy. Supports wget with http, https & ftp targets, bzr, hg, svn. Git does work with forcing https instead of git protocol. Cvs is not supported, but Buildroot does not currently have any packages relying on cvs repos (plus this would fall back to the Buildroot mirror over http).
Signed-off-by Karoly Kasza <[email protected]>
After speaking with Peter, and given the feedback from both Arnout and I, we believe this belongs to the user's settings, and has no place in Buildroot.
So, I've marked this patch as Rejected in patchwork.
Regards,
Yann E. MORIN.
Quoted 102 lines
--- Config.in | 25 +++++++++++++++++++++++++ Makefile | 9 +++++++++ support/download/wrapper | 26 ++++++++++++++++++++++++++ 3 files changed, 60 insertions(+)Config.in +25
-0@@ -253,6 +253,31 @@ config BR2_CPAN_MIRROR endmenu +config BR2_PROXY + bool "Use http proxy" + help + Use a http proxy when downloading sources with various protocols. + +if BR2_PROXY + +config BR2_PROXY_URL + string "Http proxy URL" + help + Http proxy URL in the form of <IP or FQDN>:<port>. + +config BR2_PROXY_USER + string "Http proxy username" + help + Http proxy username (if needed). + +config BR2_PROXY_PASS + string "Http proxy password" + help + Http proxy password (if needed). + PLEASE NOTE! This password will be saved in the .config file! + +endif + config BR2_JLEVEL int "Number of jobs to run simultaneously (0 for auto)" default "0"Makefile +9
-0@@ -338,6 +338,15 @@ TARGET_SKELETON = $(TOPDIR)/system/skeleton # should not be used as the root filesystem. TARGET_DIR_WARNING_FILE = $(TARGET_DIR)/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM +ifeq ($(BR2_PROXY),y) +BR_PROXY_URL = $(call qstrip,$(BR2_PROXY_URL)) +export BR_PROXY_URL +BR_PROXY_USER = $(call qstrip,$(BR2_PROXY_USER)) +export BR_PROXY_USER +BR_PROXY_PASS = $(call qstrip,$(BR2_PROXY_PASS)) +export BR_PROXY_PASS +endif + ifeq ($(BR2_CCACHE),y) CCACHE := $(HOST_DIR)/usr/bin/ccache BR_CACHE_DIR = $(call qstrip,$(BR2_CCACHE_DIR))support/download/wrapper +26
-0@@ -44,6 +44,32 @@ tmpf="${tmpd}/output" # Doing the 'cd' here rather than in all helpers is easier. cd "${tmpd}" +# Set the proxy environment variables. +# cvs doesn't support proxies OOB, but there is currently no cvs +# based package in Buildroot. Also, in that case after the download +# fails, the wget method would be used from sources.buildroot.net. +if [ -n "${BR_PROXY_URL}" ]; then +# SVN does not support http_proxy environment variables, so we use +# it's commandline arguments +proxy_host=`echo $BR_PROXY_URL | cut -d":" -f1` +proxy_port=`echo $BR_PROXY_URL | cut -d":" -f2` +SVN="${SVN} --config-option servers:global:http-proxy-host=${proxy_host} \ +--config-option servers:global:http-proxy-port=${proxy_port}" +if [ -n "${BR_PROXY_USER}" ] && [ -n "${BR_PROXY_PASS}" ]; then +SVN="${SVN} --config-option servers:global:http-proxy-username=${BR_PROXY_USER} \ +--config-option servers:global:http-proxy-password=${BR_PROXY_PASS}" +proxy="http://${BR_PROXY_USER}:${BR_PROXY_PASS}@${BR_PROXY_URL}" +else +proxy="http://${BR_PROXY_URL}" +fi +export http_proxy=${proxy} +export https_proxy=${proxy} +export ftp_proxy=${proxy} +# While git does support http_proxy environment variables, we need to use +# the config option below to force proxying over https instead of git +GIT="${GIT} -c url.https://.insteadOf=git://" +fi + # If the helper fails, we can just remove the temporary directory to # remove all the cruft it may have left behind. Then we just exit in # error too.Signature · 1 line
1.7.10.4
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 Add proxy download support
Ok, I understand.
Best regards,
Karoly
Quoted 139 lines · Yann E. MORIN
Karoly, All,
On 2014-10-21 16:53 +0200, Karoly Kasza spake thusly:
Quoted 8 lines
This patch adds support for downloading sources over a http proxy. Supports wget with http, https & ftp targets, bzr, hg, svn. Git does work with forcing https instead of git protocol. Cvs is not supported, but Buildroot does not currently have any packages relying on cvs repos (plus this would fall back to the Buildroot mirror over http).
Signed-off-by Karoly Kasza <[email protected]>After speaking with Peter, and given the feedback from both Arnout and I, we believe this belongs to the user's settings, and has no place in Buildroot.
So, I've marked this patch as Rejected in patchwork.
Regards,
Yann E. MORIN.Quoted 70 lines
--- Config.in | 25 +++++++++++++++++++++++++ Makefile | 9 +++++++++ support/download/wrapper | 26 ++++++++++++++++++++++++++ 3 files changed, 60 insertions(+)Config.in +25
-0@@ -253,6 +253,31 @@ config BR2_CPAN_MIRROR endmenu +config BR2_PROXY + bool "Use http proxy" + help + Use a http proxy when downloading sources with various protocols. + +if BR2_PROXY + +config BR2_PROXY_URL + string "Http proxy URL" + help + Http proxy URL in the form of <IP or FQDN>:<port>. + +config BR2_PROXY_USER + string "Http proxy username" + help + Http proxy username (if needed). + +config BR2_PROXY_PASS + string "Http proxy password" + help + Http proxy password (if needed). + PLEASE NOTE! This password will be saved in the .config file! + +endif + config BR2_JLEVEL int "Number of jobs to run simultaneously (0 for auto)" default "0"Makefile +9
-0@@ -338,6 +338,15 @@ TARGET_SKELETON = $(TOPDIR)/system/skeleton # should not be used as the root filesystem. TARGET_DIR_WARNING_FILE = $(TARGET_DIR)/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM +ifeq ($(BR2_PROXY),y) +BR_PROXY_URL = $(call qstrip,$(BR2_PROXY_URL)) +export BR_PROXY_URL +BR_PROXY_USER = $(call qstrip,$(BR2_PROXY_USER)) +export BR_PROXY_USER +BR_PROXY_PASS = $(call qstrip,$(BR2_PROXY_PASS)) +export BR_PROXY_PASS +endif + ifeq ($(BR2_CCACHE),y) CCACHE := $(HOST_DIR)/usr/bin/ccache BR_CACHE_DIR = $(call qstrip,$(BR2_CCACHE_DIR))support/download/wrapper +0
-0@@ -44,6 +44,32 @@ tmpf="${tmpd}/output" # Doing the 'cd' here rather than in all helpers is easier. cd "${tmpd}"support/download/wrapper tmpf="${tmpd}/output" · line 56
Quoted 10 lines
support/download/wrapper +10
-0+# Set the proxy environment variables. +# cvs doesn't support proxies OOB, but there is currently no cvs +# based package in Buildroot. Also, in that case after the download +# fails, the wget method would be used from sources.buildroot.net. +if [ -n "${BR_PROXY_URL}" ]; then +# SVN does not support http_proxy environment variables, so we use +# it's commandline arguments +proxy_host=`echo $BR_PROXY_URL | cut -d":" -f1` +proxy_port=`echo $BR_PROXY_URL | cut -d":" -f2` +SVN="${SVN} --config-optionservers:global:http-proxy-host=${proxy_host} \
support/download/wrapper tmpf="${tmpd}/output"
Quoted 3 lines
support/download/wrapper +3
-0+--config-option servers:global:http-proxy-port=${proxy_port}" +if [ -n "${BR_PROXY_USER}" ] && [ -n "${BR_PROXY_PASS}" ]; then +SVN="${SVN} --config-optionservers:global:http-proxy-username=${BR_PROXY_USER} \
support/download/wrapper tmpf="${tmpd}/output"
Quoted 9 lines
support/download/wrapper +9
-0+--config-option servers:global:http-proxy-password=${BR_PROXY_PASS}" +proxy="http://${BR_PROXY_USER}:${BR_PROXY_PASS}@${BR_PROXY_URL}" +else +proxy="http://${BR_PROXY_URL}" +fi +export http_proxy=${proxy} +export https_proxy=${proxy} +export ftp_proxy=${proxy} +# While git does support http_proxy environment variables, we need touse
Quoted 10 lines
support/download/wrapper +4
-1+# the config option below to force proxying over https instead of git +GIT="${GIT} -c url.https://.insteadOf=git://" +fi + # If the helper fails, we can just remove the temporary directory to # remove all the cruft it may have left behind. Then we just exit in # error too. --1.7.10.4
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
Quoted 2 lines
'------------------------------^-------^------------------^--------------------'
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20141025/5018238a/attachment.html>
PATCH 1/1 Add proxy download support
In reply to Yann E. MORINDear Yann E. MORIN,
Quoted 8 lines · Yann E. MORIN
On 2014-10-21 16:53 +0200, Karoly Kasza spake thusly:
Quoted 7 lines
This patch adds support for downloading sources over a http proxy. Supports wget with http, https & ftp targets, bzr, hg, svn. Git does work with forcing https instead of git protocol. Cvs is not supported, but Buildroot does not currently have any packages relying on cvs repos (plus this would fall back to the Buildroot mirror over http).
Quoted 5 lines
Quoted 1 line
Signed-off-by Karoly Kasza <[email protected]>After speaking with Peter, and given the feedback from both Arnout and I, we believe this belongs to the user's settings, and has no place in Buildroot.
I agree with this decision, but I certainly wouldn't be opposed to an addition to the manual that explains what to use Buildroot through proxies. At least the relevant pointers to the appropriate environment variables, some hints, etc.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com