local.conf.sample.extended: Document how to limit parallel fetch tasks
PATCH 1/1 local.conf.sample.extended: Document how to limit parallel fetch tasks
[YOCTO #13235]
--- meta/conf/local.conf.sample.extended | 8 ++++++++ 1 file changed, 8 insertions(+)
meta/conf/local.conf.sample.extended +8 -0
@@ -23,6 +23,14 @@ # # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would # be appropriate for example. +# +# Some users are behind firewalls or use servers where the number of parallel connections +# is limited. In such cases you can limit the number of fetch tasks which run in parallel by +# setting the option below, in this case limiting to a maximum of 4 fetch tasks in parallel: +# +#do_fetch[number_threads] = "4" +# + #DISTRO_FEATURES = "alsa bluetooth ext2 irda ipv4 ipv6 pcmcia usbgadget usbhost wifi nfs zeroconf pci"
Signature · 1 line
2.25.0
Quoted 13 lines · Richard Purdie
[YOCTO #13235]
Signed-off-by Richard Purdie <[email protected]>--- meta/conf/local.conf.sample.extended | 8 ++++++++ 1 file changed, 8 insertions(+)meta/conf/local.conf.sample.extended +0
-0
meta/conf/local.conf.sample.extended line 31
Quoted 9 lines
meta/conf/local.conf.sample.extended +6
-0@@ -23,6 +23,14 @@ # # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would # be appropriate for example. +# +# Some users are behind firewalls or use servers where the number of parallel connections +# is limited. In such cases you can limit the number of fetch tasks which run in parallel by +# setting the option below, in this case limiting to a maximum of 4 fetch tasks in parallel: +# +#do_fetch[number_threads] = "4"
How does this interact with BB_NUMBER_THREADS? If, for example, both are set to 4 does it mean there could be 4 fetch threads and 4 compile threads running in parallel?
A related issue when the number of parallel connections is limited is parsing a large number of AUTOREV recipes. From what I remember the workaround for that was to set BB_NUMBER_PARSE_THREADS. Maybe that should be documented in local.conf.sample.extended too? Perhaps even better would be to define a single new variable which limits the number of parallel connections in both cases?
Quoted 13 lines
meta/conf/local.conf.sample.extended +2
-1+# + #DISTRO_FEATURES = "alsa bluetooth ext2 irda ipv4 ipv6 pcmcia usbgadget usbhost wifi nfs zeroconf pci" --2.25.0
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 24 lines · Andre McCurdy
Quoted 22 lines · Richard Purdie
[YOCTO #13235]
Signed-off-by Richard Purdie <[email protected]>--- meta/conf/local.conf.sample.extended | 8 ++++++++ 1 file changed, 8 insertions(+)meta/conf/local.conf.sample.extended +0
-0@@ -23,6 +23,14 @@ # # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE= "-j 4" would
# be appropriate for example.meta/conf/local.conf.sample.extended +2
-0+# +# Some users are behind firewalls or use servers where the numberof parallel connections
+# is limited. In such cases you can limit the number of fetch tasks which run in parallel by
Quoted 8 lines
meta/conf/local.conf.sample.extended
Quoted 4 lines
+# setting the option below, in this case limiting to a maximum of 4 fetch tasks in parallel:
meta/conf/local.conf.sample.extended +2
-0+# +#do_fetch[number_threads] = "4"How does this interact with
BB_NUMBER_THREADS? If, for example, both are set to 4 does it mean there could be 4 fetch threads and 4 compile threads running in parallel?
BB_NUMBER_THREADS is an upper limit which includes fetch tasks.
Quoted 6 lines
A related issue when the number of parallel connections is limited is parsing a large number of AUTOREV recipes. From what I remember the workaround for that was to set
BB_NUMBER_PARSE_THREADS. Maybe that should be documented in local.conf.sample.extended too? Perhaps even better would be to define a single new variable which limits the number of parallel connections in both cases?
I don't know how you'd implement a total number of connections limit from a code perspective :/.
You'd probably need something like a make job server where you obtained tokens before making connections and I can see this becoming "fun" since you'd have to markup every code section where a network connection was made.
We have a conflict between code complexity, maintainability, performance and functionality and I'm not sure the use case here warrants the complexity...
Cheers,
Richard
Quoted 44 lines · Richard Purdie
Quoted 24 lines · Andre McCurdy
Quoted 22 lines · Richard Purdie
[YOCTO #13235]
Signed-off-by Richard Purdie <[email protected]>--- meta/conf/local.conf.sample.extended | 8 ++++++++ 1 file changed, 8 insertions(+)meta/conf/local.conf.sample.extended +0
-0@@ -23,6 +23,14 @@ # # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE= "-j 4" would
# be appropriate for example.meta/conf/local.conf.sample.extended +2
-0+# +# Some users are behind firewalls or use servers where the numberof parallel connections
+# is limited. In such cases you can limit the number of fetch tasks which run in parallel byQuoted 8 lines
meta/conf/local.conf.sample.extended
Quoted 4 lines
+# setting the option below, in this case limiting to a maximum of 4 fetch tasks in parallel:
meta/conf/local.conf.sample.extended +2
-0+# +#do_fetch[number_threads] = "4"How does this interact with
BB_NUMBER_THREADS? If, for example, both are set to 4 does it mean there could be 4 fetch threads and 4 compile threads running in parallel?
BB_NUMBER_THREADSis an upper limit which includes fetch tasks.Quoted 6 lines
A related issue when the number of parallel connections is limited is parsing a large number of AUTOREV recipes. From what I remember the workaround for that was to set
BB_NUMBER_PARSE_THREADS. Maybe that should be documented in local.conf.sample.extended too? Perhaps even better would be to define a single new variable which limits the number of parallel connections in both cases?
Quoted 2 lines
I don't know how you'd implement a total number of connections limit from a code perspective :/.
A single variable such as BB_MAX_CONNECTION_THREADS which, if defined, would set the limit for both do_fetch[number_threads] and BB_NUMBER_PARSE_THREADS ?
Quoted 8 lines
You'd probably need something like a make job server where you obtained tokens before making connections and I can see this becoming "fun" since you'd have to markup every code section where a network connection was made.
We have a conflict between code complexity, maintainability, performance and functionality and I'm not sure the use case here warrants the complexity...
Of course. I just wanted to share it since it is a real world problem (I've seen a development team stumped for a few days before finding BB_NUMBER_PARSE_THREADS, which isn't nearly as well documented as BB_NUMBER_THREADS and PARALLEL_MAKE). A comment for BB_NUMBER_PARSE_THREADS in local.conf.sample.extended might have sped up that debugging process.