Skip to contentSkip to conversation

local.conf.sample.extended: Document how to limit parallel fetch tasks

PATCH In review Richard Purdie 2 participants 4 messages Feb 19, 2020 – Feb 20, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 local.conf.sample.extended: Document how to limit parallel fetch tasks

[YOCTO #13235]

---
 meta/conf/local.conf.sample.extended | 8 ++++++++
 1 file changed, 8 insertions(+)
meta/conf/local.conf.sample.extended +8 -0
index fcf4a9b7c1a..443b4e83f9a 100644
@@ -23,6 +23,14 @@
 #
 # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
 # be appropriate for example.
+#
+# Some users are behind firewalls or use servers where the number of parallel connections
+# is limited. In such cases you can limit the number of fetch tasks which run in parallel by
+# setting the option below, in this case limiting to a maximum of 4 fetch tasks in parallel:
+#
+#do_fetch[number_threads] = "4"
+#
+
 
 #DISTRO_FEATURES = "alsa bluetooth ext2 irda ipv4 ipv6 pcmcia usbgadget usbhost wifi nfs zeroconf pci"
 
Signature · 1 line
2.25.0
#1 lore raw
Quoted 13 lines · Richard Purdie

[YOCTO #13235]

---
 meta/conf/local.conf.sample.extended | 8 ++++++++
 1 file changed, 8 insertions(+)
meta/conf/local.conf.sample.extended +0 -0
index fcf4a9b7c1a..443b4e83f9a 100644

meta/conf/local.conf.sample.extended line 31

Quoted 9 lines
meta/conf/local.conf.sample.extended +6 -0
@@ -23,6 +23,14 @@
 #
 # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
 # be appropriate for example.
+#
+# Some users are behind firewalls or use servers where the number of parallel connections
+# is limited. In such cases you can limit the number of fetch tasks which run in parallel by
+# setting the option below, in this case limiting to a maximum of 4 fetch tasks in parallel:
+#
+#do_fetch[number_threads] = "4"

How does this interact with BB_NUMBER_THREADS? If, for example, both are set to 4 does it mean there could be 4 fetch threads and 4 compile threads running in parallel?

A related issue when the number of parallel connections is limited is parsing a large number of AUTOREV recipes. From what I remember the workaround for that was to set BB_NUMBER_PARSE_THREADS. Maybe that should be documented in local.conf.sample.extended too? Perhaps even better would be to define a single new variable which limits the number of parallel connections in both cases?

Quoted 13 lines
meta/conf/local.conf.sample.extended +2 -1
+#
+
 
 #DISTRO_FEATURES = "alsa bluetooth ext2 irda ipv4 ipv6 pcmcia usbgadget usbhost wifi nfs zeroconf pci"
 
--

2.25.0

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#2 lore raw
Quoted 24 lines · Andre McCurdy
Quoted 22 lines · Richard Purdie

[YOCTO #13235]

---
 meta/conf/local.conf.sample.extended | 8 ++++++++
 1 file changed, 8 insertions(+)
meta/conf/local.conf.sample.extended +0 -0
index fcf4a9b7c1a..443b4e83f9a 100644
@@ -23,6 +23,14 @@
 #
 # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE

= "-j 4" would

 # be appropriate for example.
meta/conf/local.conf.sample.extended +2 -0
+#
+# Some users are behind firewalls or use servers where the number

of parallel connections
+# is limited. In such cases you can limit the number of fetch tasks which run in parallel by

Quoted 8 lines

meta/conf/local.conf.sample.extended

Quoted 4 lines

+# setting the option below, in this case limiting to a maximum of 4 fetch tasks in parallel:

meta/conf/local.conf.sample.extended +2 -0
+#
+#do_fetch[number_threads] = "4"

How does this interact with BB_NUMBER_THREADS? If, for example, both are set to 4 does it mean there could be 4 fetch threads and 4 compile threads running in parallel?

BB_NUMBER_THREADS is an upper limit which includes fetch tasks.

Quoted 6 lines

A related issue when the number of parallel connections is limited is parsing a large number of AUTOREV recipes. From what I remember the workaround for that was to set BB_NUMBER_PARSE_THREADS. Maybe that should be documented in local.conf.sample.extended too? Perhaps even better would be to define a single new variable which limits the number of parallel connections in both cases?

I don't know how you'd implement a total number of connections limit from a code perspective :/.

You'd probably need something like a make job server where you obtained tokens before making connections and I can see this becoming "fun" since you'd have to markup every code section where a network connection was made.

We have a conflict between code complexity, maintainability, performance and functionality and I'm not sure the use case here warrants the complexity...

Cheers,

Richard

#3 lore raw
Quoted 44 lines · Richard Purdie
Quoted 24 lines · Andre McCurdy
Quoted 22 lines · Richard Purdie

[YOCTO #13235]

---
 meta/conf/local.conf.sample.extended | 8 ++++++++
 1 file changed, 8 insertions(+)
meta/conf/local.conf.sample.extended +0 -0
index fcf4a9b7c1a..443b4e83f9a 100644
@@ -23,6 +23,14 @@
 #
 # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE

= "-j 4" would

 # be appropriate for example.
meta/conf/local.conf.sample.extended +2 -0
+#
+# Some users are behind firewalls or use servers where the number

of parallel connections
+# is limited. In such cases you can limit the number of fetch tasks which run in parallel by

Quoted 8 lines

meta/conf/local.conf.sample.extended

Quoted 4 lines

+# setting the option below, in this case limiting to a maximum of 4 fetch tasks in parallel:

meta/conf/local.conf.sample.extended +2 -0
+#
+#do_fetch[number_threads] = "4"

How does this interact with BB_NUMBER_THREADS? If, for example, both are set to 4 does it mean there could be 4 fetch threads and 4 compile threads running in parallel?

BB_NUMBER_THREADS is an upper limit which includes fetch tasks.

Quoted 6 lines

A related issue when the number of parallel connections is limited is parsing a large number of AUTOREV recipes. From what I remember the workaround for that was to set BB_NUMBER_PARSE_THREADS. Maybe that should be documented in local.conf.sample.extended too? Perhaps even better would be to define a single new variable which limits the number of parallel connections in both cases?

Quoted 2 lines

I don't know how you'd implement a total number of connections limit from a code perspective :/.

A single variable such as BB_MAX_CONNECTION_THREADS which, if defined, would set the limit for both do_fetch[number_threads] and BB_NUMBER_PARSE_THREADS ?

Quoted 8 lines

You'd probably need something like a make job server where you obtained tokens before making connections and I can see this becoming "fun" since you'd have to markup every code section where a network connection was made.

We have a conflict between code complexity, maintainability, performance and functionality and I'm not sure the use case here warrants the complexity...

Of course. I just wanted to share it since it is a real world problem (I've seen a development team stumped for a few days before finding BB_NUMBER_PARSE_THREADS, which isn't nearly as well documented as BB_NUMBER_THREADS and PARALLEL_MAKE). A comment for BB_NUMBER_PARSE_THREADS in local.conf.sample.extended might have sped up that debugging process.