package/python3: add patch to fix highly parallel builds
PATCH 1/1 package/python3: add patch to fix highly parallel builds
Highly parallel host-python3 builds sometimes fail with:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/tmp/instance-3/output-1/host/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 317, in run
result_item, is_broken, cause = self.wait_result_broken_or_wakeup()
File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 376, in wait_result_broken_or_wakeup
worker_sentinels = [p.sentinel for p in self.processes.values()]
File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 376, in <listcomp>
worker_sentinels = [p.sentinel for p in self.processes.values()]
RuntimeError: dictionary changed size during iteration
During the compile_all.py step of host-python3. This issue is reported upstream at https://bugs.python.org/issue43498, and while not yet fixed upstream, a PR was proposed with a possible fix for it. Seems the PR seems reasonable, let's give it a chance and see if it improves the situation.
Hopefully Fixes:
http://autobuild.buildroot.net/results/ae6c4ab292589a4e4442dfb0a1286349a9bf4d29/
--- ...teration-error-in-_ExecutorManagerTh.patch | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 package/python3/0036-Fix-dictionary-iteration-error-in-_ExecutorManagerTh.patch
package/python3/0036-Fix-dictionary-iteration-error-in-_ExecutorManagerTh.patch new file +29 -0
@@ -0,0 +1,29 @@ +From 331966be70c371b268a4fcce9e97280cd869f137 Mon Sep 17 00:00:00 2001 +From: Jakub Kulik <[email protected]> +Date: Mon, 15 Mar 2021 08:49:28 +0100 +Subject: [PATCH] Fix dictionary iteration error in _ExecutorManagerThread + +[Thomas: Taken from upstream pull request +https://github.com/python/cpython/pull/24868, which is aimed at fixing +https://bugs.python.org/issue43498] +Signed-off-by: Thomas Petazzoni <[email protected]> +--- + Lib/concurrent/futures/process.py | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py +index 90bc98bf2ec..e3b36dff572 100644 +--- a/Lib/concurrent/futures/process.py ++++ b/Lib/concurrent/futures/process.py +@@ -373,7 +373,7 @@ class _ExecutorManagerThread(threading.Thread): + assert not self.thread_wakeup._closed + wakeup_reader = self.thread_wakeup._reader + readers = [result_reader, wakeup_reader] +- worker_sentinels = [p.sentinel for p in self.processes.values()] ++ worker_sentinels = [p.sentinel for p in self.processes.copy().values()] + ready = mp.connection.wait(readers + worker_sentinels) + + cause = None +-- +2.31.1 +
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Yann E. MORIN
AppliedThomas, All,
On 2021-08-02 23:10 +0200, Thomas Petazzoni spake thusly:
Quoted 20 lines
Highly parallel host-python3 builds sometimes fail with:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/tmp/instance-3/output-1/host/lib/python3.9/threading.py", line 973, in_bootstrap_innerself.run()File "
/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 317, in run
result_item,is_broken, cause =self.wait_result_broken_or_wakeup()
File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 376, inwait_result_broken_or_wakeup
worker_sentinels= [p.sentinel for p inself.processes.values()]
File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 376, in <listcomp>
worker_sentinels= [p.sentinel for p inself.processes.values()]
RuntimeError: dictionary changed size during iterationDuring the
compile_all.pystep of host-python3. This issue is reported upstream at https://bugs.python.org/issue43498, and while not yet fixed upstream, a PR was proposed with a possible fix for it. Seems the PR seems reasonable, let's give it a chance and see if it improves the situation.
Quoted 5 lines
Hopefully Fixes:
http://autobuild.buildroot.net/results/ae6c4ab292589a4e4442dfb0a1286349a9bf4d29/
Signed-off-by Thomas Petazzoni <[email protected]>
Applied to master, thanks.
It seems reasonable enough that we can have a shot at it in the autobuilders.
Note that it has been reported that this is not necessarily 100% bullet-proof, so we may still see some failures, but the situation should be muh, much better now.
Thanks!
Regards,
Yann E. MORIN.
Quoted 47 lines
--- ...teration-error-in-_ExecutorManagerTh.patch | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 package/python3/0036-Fix-dictionary-iteration-error-in-_ExecutorManagerTh.patchpackage/python3/0036-Fix-dictionary-iteration-error-in-_ExecutorManagerTh.patch new file +29
-0@@ -0,0 +1,29 @@ +From 331966be70c371b268a4fcce9e97280cd869f137 Mon Sep 17 00:00:00 2001 +From: Jakub Kulik <[email protected]> +Date: Mon, 15 Mar 2021 08:49:28 +0100 +Subject: [PATCH] Fix dictionary iteration error in _ExecutorManagerThread + +[Thomas: Taken from upstream pull request +https://github.com/python/cpython/pull/24868, which is aimed at fixing +https://bugs.python.org/issue43498] +Signed-off-by: Thomas Petazzoni <[email protected]> +--- + Lib/concurrent/futures/process.py | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py +index 90bc98bf2ec..e3b36dff572 100644 +--- a/Lib/concurrent/futures/process.py ++++ b/Lib/concurrent/futures/process.py +@@ -373,7 +373,7 @@ class _ExecutorManagerThread(threading.Thread): + assert not self.thread_wakeup._closed + wakeup_reader = self.thread_wakeup._reader + readers = [result_reader, wakeup_reader] +- worker_sentinels = [p.sentinel for p in self.processes.values()] ++ worker_sentinels = [p.sentinel for p in self.processes.copy().values()] + ready = mp.connection.wait(readers + worker_sentinels) + + cause = None +-- +2.31.1 +Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Quoted 21 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
Highly parallel host-python3 builds sometimes fail with:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/tmp/instance-3/output-1/host/lib/python3.9/threading.py", line 973, in_bootstrap_innerself.run()File "
/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 317, in run
result_item,is_broken, cause =self.wait_result_broken_or_wakeup()
File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 376, inwait_result_broken_or_wakeup
worker_sentinels= [p.sentinel for p inself.processes.values()]
File "/tmp/instance-3/output-1/host/lib/python3.9/concurrent/futures/process.py", line 376, in <listcomp>
worker_sentinels= [p.sentinel for p inself.processes.values()]
RuntimeError: dictionary changed size during iterationDuring the
compile_all.pystep of host-python3. This issue is reported upstream at https://bugs.python.org/issue43498, and while not yet fixed upstream, a PR was proposed with a possible fix for it. Seems the PR seems reasonable, let's give it a chance and see if it improves the situation.
Quoted 5 lines
Hopefully Fixes:
http://autobuild.buildroot.net/results/ae6c4ab292589a4e4442dfb0a1286349a9bf4d29/
Signed-off-by Thomas Petazzoni <[email protected]>
Committed to 2021.02.x and 2021.05.x, thanks.
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot