Skip to contentSkip to conversation

bitbake-worker: export_proxies if network flag = 1

PATCH In review [email protected] 3 participants 10 messages Aug 23, 2023 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bitbake-worker: export_proxies if network flag = 1

From: Randolph Sapp <[email protected]>

Call the export_proxies() function if the network flag is set. Not sure why anyone would want to enable access without passing proxy variables if they are set.

---
 bin/bitbake-worker | 4 ++++
 1 file changed, 4 insertions(+)
bin/bitbake-worker +4 -0
index 451e6926..ba199ef9 100755
@@ -285,6 +285,10 @@ def fork_off_task(cfg, data, databuilder, workerdata, extraconfigdata, runtask):
                 for e, v in exports:
                     os.environ[e] = v
 
+                # if network is true we should assume proxy variables are to be passed through
+                if the_data.getVarFlag(taskname, 'network', False):
+                    bb.utils.export_proxies(the_data)
+
                 for e in fakeenv:
                     os.environ[e] = fakeenv[e]
                     the_data.setVar(e, fakeenv[e])
Signature · 1 line
2.41.0
#1 lore raw

Something doesn’t compute here. How did it work all this time in tasks like fetch?

Alex

Quoted 43 lines · Randolph Sapp via lists.openembedded.org

From: Randolph Sapp <[email protected]>

Call the export_proxies() function if the network flag is set. Not sure why anyone would want to enable access without passing proxy variables if they are set.

---
 bin/bitbake-worker | 4 ++++
 1 file changed, 4 insertions(+)
bin/bitbake-worker +0 -0
index 451e6926..ba199ef9 100755
@@ -285,6 +285,10 @@ def fork_off_task(cfg, data, databuilder, workerdata,

extraconfigdata, runtask):
for e, v in exports:
os.environ[e] = v

+ # if network is true we should assume proxy variables are
to be passed through

bin/bitbake-worker +3 -1
+                if the_data.getVarFlag(taskname, 'network', False):
+                    bb.utils.export_proxies(the_data)
+
                 for e in fakeenv:
                     os.environ[e] = fakeenv[e]
                     the_data.setVar(e, fakeenv[e])
--

2.41.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#14959):
https://lists.openembedded.org/g/bitbake-devel/message/14959
Mute This Topic: https://lists.openembedded.org/mt/100924096/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#2 lore raw

Because fetchers explicitly sets proxies through a different mechanism, partially because different fetch mechanisms may have different ways to set proxies. The variables set by export_proxies are the most common, widely accepted ones though so I figured they should be set by default for tasks that need network access.

Quoted 52 lines · Alexander Kanavin

Something doesn’t compute here. How did it work all this time in tasks like fetch?

Alex

On Wed 23. Aug 2023 at 23.04, Randolph Sapp via lists.openembedded.org <http://lists.openembedded.org> <[email protected] <mailto:[email protected]>> wrote:

From: Randolph Sapp <[email protected] <mailto:[email protected]>>

Call the export_proxies() function if the network flag is set. Not sure why anyone would want to enable access without passing proxy variables if they are set.

Signed-off-by Randolph Sapp <[email protected] <mailto:[email protected]>>

---
 bin/bitbake-worker | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/bin/bitbake-worker b/bin/bitbake-worker index 451e6926..ba199ef9 100755
--- a/bin/bitbake-worker
+++ b/bin/bitbake-worker
@@ -285,6 +285,10 @@ def fork_off_task(cfg, data, databuilder, workerdata, extraconfigdata, runtask):
                 for e, v in exports:
                     os.environ[e] = v

+                # if network is true we should assume proxy variables are to be passed through
+                if the_data.getVarFlag(taskname, 'network', False):
+                    bb.utils.export_proxies(the_data)
+
                 for e in fakeenv:
                     os.environ[e] = fakeenv[e] the_data.setVar(e, fakeenv[e])
--
2.41.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#14959):
https://lists.openembedded.org/g/bitbake-devel/message/14959
<https://lists.openembedded.org/g/bitbake-devel/message/14959>
Mute This Topic: https://lists.openembedded.org/mt/100924096/1686489
<https://lists.openembedded.org/mt/100924096/1686489>
Group Owner: [email protected] <mailto:bitbake-devel%[email protected]>
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub
<https://lists.openembedded.org/g/bitbake-devel/unsub>
[[email protected] <mailto:[email protected]>]
-=-=-=-=-=-=-=-=-=-=-=-

#3 lore raw
In reply to Alexander Kanavin
Quoted 2 lines · Alexander Kanavin

Something doesn’t compute here. How did it work all this time in tasks like fetch?

Before we had network isolation, we did ensure we only set the network proxy information before network access code. This meant the things would break for "unknown" network access outside of the points in the code where we expected it on systems that used a proxy.

I've mixed feelings on "simplifying" things like this as we'd lose that markup of the code and a second level of checking, even if that check does need a proxy setup.

Was there a specific place where the proxy information was needed but not present?

Cheers,

Richard

#4 lore raw
Quoted 9 lines · Richard Purdie
Quoted 2 lines · Alexander Kanavin

Something doesn’t compute here. How did it work all this time in tasks like fetch?

Before we had network isolation, we did ensure we only set the network proxy information before network access code. This meant the things would break for "unknown" network access outside of the points in the code where we expected it on systems that used a proxy.

Quoted 6 lines

I've mixed feelings on "simplifying" things like this as we'd lose that markup of the code and a second level of checking, even if that check does need a proxy setup.

Was there a specific place where the proxy information was needed but not present?

Rust, Go, Flutter, Matter, and all of the other new frameworks and languages that decided to be special and make their own build systems that reach out during the do_compile phase.

Though, it could be argued that this is just a cover-up for the real issue, which is lack of proper integration of these build systems into bitbake. I still believe this is useful for compilations behind a corporate proxy for those few recipes still *cheating*.

#5 lore raw
Quoted 17 lines · Randolph Sapp
Quoted 9 lines · Richard Purdie
Quoted 2 lines · Alexander Kanavin

Something doesn’t compute here. How did it work all this time in tasks like fetch?

Before we had network isolation, we did ensure we only set the network proxy information before network access code. This meant the things would break for "unknown" network access outside of the points in the code where we expected it on systems that used a proxy.

Quoted 6 lines

I've mixed feelings on "simplifying" things like this as we'd lose that markup of the code and a second level of checking, even if that check does need a proxy setup.

Was there a specific place where the proxy information was needed but not present?

Quoted 8 lines

Rust, Go, Flutter, Matter, and all of the other new frameworks and languages that decided to be special and make their own build systems that reach out during the do_compile phase.

Though, it could be argued that this is just a cover-up for the real issue, which is lack of proper integration of these build systems into bitbake. I still believe this is useful for compilations behind a corporate proxy for those few recipes still *cheating*.

Do we really want to make it easier for people to ignore that set of problems? :/

People seem very very reluctant to try and fix any of these issues properly and with the kinds of regulations coming in, this is a serious issue whether people like it or not...

Cheers,

Richard

#6 lore raw
Quoted 22 lines · Richard Purdie
Quoted 21 lines · Randolph Sapp
Quoted 9 lines · Richard Purdie
Quoted 2 lines · Alexander Kanavin

Something doesn’t compute here. How did it work all this time in tasks like fetch?

Before we had network isolation, we did ensure we only set the network proxy information before network access code. This meant the things would break for "unknown" network access outside of the points in the code where we expected it on systems that used a proxy.

Quoted 6 lines

I've mixed feelings on "simplifying" things like this as we'd lose that markup of the code and a second level of checking, even if that check does need a proxy setup.

Was there a specific place where the proxy information was needed but not present?

Rust, Go, Flutter, Matter, and all of the other new frameworks and languages that decided to be special and make their own build systems that reach out during the do_compile phase.

Quoted 7 lines
Quoted 4 lines

Though, it could be argued that this is just a cover-up for the real issue, which is lack of proper integration of these build systems into bitbake. I still believe this is useful for compilations behind a corporate proxy for those few recipes still *cheating*.

Do we really want to make it easier for people to ignore that set of problems? :/

I suppose not. Though it is a little rough to add a fetcher and class for every one-off build system that Google introduces for their shinny new product :P

Feel free to drop this patch. Just figured I would test the waters.

#7 lore raw
In reply to Richard Purdie
Quoted 27 lines · Richard Purdie
Quoted 17 lines · Randolph Sapp
Quoted 9 lines · Richard Purdie
Quoted 2 lines · Alexander Kanavin

Something doesn’t compute here. How did it work all this time in tasks like fetch?

Before we had network isolation, we did ensure we only set the network proxy information before network access code. This meant the things would break for "unknown" network access outside of the points in the code where we expected it on systems that used a proxy.

Quoted 6 lines

I've mixed feelings on "simplifying" things like this as we'd lose that markup of the code and a second level of checking, even if that check does need a proxy setup.

Was there a specific place where the proxy information was needed but not present?

Quoted 8 lines

Rust, Go, Flutter, Matter, and all of the other new frameworks and languages that decided to be special and make their own build systems that reach out during the do_compile phase.

Though, it could be argued that this is just a cover-up for the real issue, which is lack of proper integration of these build systems into bitbake. I still believe this is useful for compilations behind a corporate proxy for those few recipes still *cheating*.

Quoted 6 lines

Do we really want to make it easier for people to ignore that set of problems? :/

People seem very very reluctant to try and fix any of these issues properly and with the kinds of regulations coming in, this is a serious issue whether people like it or not...

I would argue that as long as the 'network' flag exists people will just continue to ignore the real issue.

The current proxy problem just makes it slightly more annoying for a small percentage of the people currently circumventing the issue.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw
Quoted 14 lines · Randolph Sapp via lists.openembedded.org
Quoted 13 lines · Richard Purdie
Quoted 2 lines · Alexander Kanavin

Something doesn’t compute here. How did it work all this time in tasks like fetch?

Before we had network isolation, we did ensure we only set the network proxy information before network access code. This meant the things would break for "unknown" network access outside of the points in the code where we expected it on systems that used a proxy.

I've mixed feelings on "simplifying" things like this as we'd lose that markup of the code and a second level of checking, even if that check does need a proxy setup.

Quoted 6 lines
Quoted 2 lines

Was there a specific place where the proxy information was needed but not present?

Rust, Go, Flutter, Matter, and all of the other new frameworks and languages that decided to be special and make their own build systems that reach out during the do_compile phase.

Though, it could be argued that this is just a cover-up for the real issue, which is lack of proper integration of these build systems into bitbake. I still believe this is useful for compilations behind a corporate proxy for those few recipes still *cheating*.

Also, any custom tasks people add that have network dependencies. I just think it's a little rough having to play whack-a-mole with task and proxy oddities.

#10 lore raw
Quoted 22 lines · Randolph Sapp
Quoted 21 lines · Randolph Sapp via lists.openembedded.org
Quoted 9 lines · Richard Purdie
Quoted 2 lines · Alexander Kanavin

Something doesn’t compute here. How did it work all this time in tasks like fetch?

Before we had network isolation, we did ensure we only set the network proxy information before network access code. This meant the things would break for "unknown" network access outside of the points in the code where we expected it on systems that used a proxy.

Quoted 6 lines

I've mixed feelings on "simplifying" things like this as we'd lose that markup of the code and a second level of checking, even if that check does need a proxy setup.

Was there a specific place where the proxy information was needed but not present?

Rust, Go, Flutter, Matter, and all of the other new frameworks and languages that decided to be special and make their own build systems that reach out during the do_compile phase.

Quoted 8 lines
Quoted 4 lines

Though, it could be argued that this is just a cover-up for the real issue, which is lack of proper integration of these build systems into bitbake. I still believe this is useful for compilations behind a corporate proxy for those few recipes still *cheating*.

Also, any custom tasks people add that have network dependencies. I just think it's a little rough having to play whack-a-mole with task and proxy oddities.

The idea is that you realise something is not right and then fix the issue properly, which would be proper SRC_URI support which then means we can cover SPDX manifests, license auditing, source mirroring and reproducibility amongst other things.

Sadly most people just want to ignore those things and go for a quick/easy fix.

I'm therefore torn on the patch. How easily do we want people to ignore these issues?

Cheers,

Richard