support/testing/run-tests: specify multiprocessing method
A message that is not in the archive (sent off-list or not yet received): 20260119144010.1646792-1
PATCH 1/1 support/testing/run-tests: specify multiprocessing method
Since Python 3.14, the multiprocessing library API has changed:
Quoted 3 lines
On POSIX platforms the default start method was changed from fork to forkserver to retain the performance but avoid common multithreaded process incompatibilities.
https://docs.python.org/3/library/multiprocessing.html
Since this is expected to be under "if __name__ == '__main__'", we need to have this in support/testing/run-tests regardless of how this will be fixed in nose2.
--- support/testing/run-tests | 1 + 1 file changed, 1 insertion(+)
support/testing/run-tests +1 -0
@@ -142,4 +142,5 @@ def main(): if __name__ == "__main__": + multiprocessing.set_start_method("fork") sys.exit(main())
Signature · 6 lines
2.43.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Julien Olivain via buildroot
AppliedHi Jimmy,
Quoted 11 lines · Jimmy Durand Wesolowski via buildroot
Since Python 3.14, the multiprocessing library API has changed:
Quoted 3 lines
On POSIX platforms the default start method was changed from fork to forkserver to retain the performance but avoid common multithreaded process incompatibilities.
https://docs.python.org/3/library/multiprocessing.html
Since this is expected to be under "if
__name__== '__main__'", we need to have this in support/testing/run-tests regardless of how this will be fixed in nose2.Signed-off-by Jimmy Durand Wesolowski <[email protected]>
Applied to master, thanks.
For info, I slightly reworded the commit log. See:
https://gitlab.com/buildroot.org/buildroot/-/commit/3d2141bceefda32bef06ddda594eaf8665913276
Best regards,
Julien.
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Hi Jimmy, Julien, all!
Am 02.02.26 um 17:45 schrieb Julien Olivain via buildroot:
Quoted 13 lines
Hi Jimmy,
Quoted 10 lines · Jimmy Durand Wesolowski via buildroot
Since Python 3.14, the multiprocessing library API has changed:
Quoted 3 lines
On POSIX platforms the default start method was changed from fork to forkserver to retain the performance but avoid common multithreaded process incompatibilities.
https://docs.python.org/3/library/multiprocessing.html
Since this is expected to be under "if
__name__== '__main__'", we need to have this in support/testing/run-tests regardless of how this will be fixed in nose2.
Quoted 6 lines
Quoted 1 line
Signed-off-by Jimmy Durand Wesolowski <[email protected]>Applied to master, thanks.
For info, I slightly reworded the commit log. See:
https://gitlab.com/buildroot.org/buildroot/-/ commit/3d2141bceefda32bef06ddda594eaf8665913276
Unfortunately this patch makes run-tests fail on Debian Sid, because a similar patch [1] was accepted into the Debian nose2 package to fix build errors there [2]. The result is that multiprocessing.set_start_method() is called twice, once in run-tests and once in nose2, and that second call causes a crash with:
RuntimeError: context has already been set
Commenting the multiprocessing.set_start_method() call in run-tests fixes the problem. I'm not sure if there is a good fix we can apply in Buildroot (if we revert the patch it'll probably fail again on systems with unpatched nose2), but I wanted to point it out and mention the workaround.
I assume a proper fix would have to be in nose2 itself, either not relying on a specific start method, or using a multiprocessing context to select the needed one independent of global settings/defaults.
Best regards,
Fiona
[1] https://sources.debian.org/src/nose2/0.15.1-3/debian/patches/0004-plugins-mp-set-context-to-fork-for-Python-3.14-mp-AP.patch
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1123154
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Fiona" == Fiona Klute via buildroot <[email protected]> writes:
Hi,
Quoted 19 lines
Quoted 4 lines
Applied to master, thanks.
For info, I slightly reworded the commit log. See:
https://gitlab.com/buildroot.org/buildroot/-/
commit/3d2141bceefda32bef06ddda594eaf8665913276Unfortunately this patch makes run-tests fail on Debian Sid, because a similar patch [1] was accepted into the Debian nose2 package to fix build errors there [2]. The result is that
multiprocessing.set_start_method()is called twice, once in run-tests and once in nose2, and that second call causes a crash with:RuntimeError: context has already been set
Commenting the
multiprocessing.set_start_method()call in run-tests fixes the problem. I'm not sure if there is a good fix we can apply in Buildroot (if we revert the patch it'll probably fail again on systems with unpatched nose2), but I wanted to point it out and mention the workaround.
Quoted 3 lines
I assume a proper fix would have to be in nose2 itself, either not relying on a specific start method, or using a multiprocessing context to select the needed one independent of global settings/defaults.
Crap :/
How about just putting the set_start_method() call inside a try/catch and skip it if already set?
https://github.com/python/cpython/blob/3.14/Lib/multiprocessing/context.py#L245-L251
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Am 15.02.26 um 10:29 schrieb Peter Korsgaard:
Quoted 27 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Fiona" == Fiona Klute via buildroot <[email protected]> writes:
Hi,
Quoted 19 lines
Quoted 4 lines
Applied to master, thanks.
For info, I slightly reworded the commit log. See:
https://gitlab.com/buildroot.org/buildroot/-/
commit/3d2141bceefda32bef06ddda594eaf8665913276Unfortunately this patch makes run-tests fail on Debian Sid, because a similar patch [1] was accepted into the Debian nose2 package to fix build errors there [2]. The result is that
multiprocessing.set_start_method()is called twice, once in run-tests and once in nose2, and that second call causes a crash with:RuntimeError: context has already been set
Commenting the
multiprocessing.set_start_method()call in run-tests fixes the problem. I'm not sure if there is a good fix we can apply in Buildroot (if we revert the patch it'll probably fail again on systems with unpatched nose2), but I wanted to point it out and mention the workaround.Quoted 3 lines
I assume a proper fix would have to be in nose2 itself, either not relying on a specific start method, or using a multiprocessing context to select the needed one independent of global settings/defaults.
Quoted 6 lines
Crap :/
How about just putting the
set_start_method()call inside a try/catch and skip it if already set?https://github.com/python/cpython/blob/3.14/Lib/multiprocessing/context.py#L245-L251
I've been digging a little more, and it looks less bleak. Key results:
* Changing the multiprocessing start method in or for nose is *not* the correct approach, or needed (at least not in general – there are some issues using coverage I didn't dig into, but we don't need that in Buildroot because we're not actually testing Python code). See [1] for details.
* The only reason we have trouble without "fork" in Buildroot is that support/testing/run-tests sets class variables in BRConfigTest, and those are lost if the infra.basetest module is loaded again in a new process (instead of carried over during fork).
Conclusion: The proper fix for Buildroot is to avoid passing config in class variables.
I've tested this with environment variables and it worked fine, though I'd prefer something that allows type checking, and doesn't require setting a bunch of environment variables. How about adding a little dataclass (or serialize/deserialize functions in BRConfigTest) and using an environment variable to pass the path of a file with serialized config to be loaded in BRConfigTest.__init__()? I'd probably use JSON because it's easy to handle with the Python standard library.
Another option would be to switch to a test runner that has built-in support for passing config (like Pytest), but that'd be a major change, not a simple fix.
Best regards,
Fiona
[1] https://github.com/nose-devs/nose2/pull/644#issuecomment-3905000508
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Fiona" == Fiona Klute <[email protected]> writes:
Hi,
Quoted 7 lines
I've been digging a little more, and it looks less bleak. Key results:
* Changing the multiprocessing start method in or for nose is *not* the correct approach, or needed (at least not in general – there are some issues using coverage I didn't dig into, but we don't need that in Buildroot because we're not actually testing Python code). See
[1] for details.
OK, thanks.
Quoted 8 lines
* The only reason we have trouble without "fork" in Buildroot is that support/testing/run-tests sets class variables in BRConfigTest, and those are lost if the infra.basetest module is loaded again in a new process (instead of carried over during fork).
Conclusion: The proper fix for Buildroot is to avoid passing config in class variables.
Quoted 8 lines
I've tested this with environment variables and it worked fine, though I'd prefer something that allows type checking, and doesn't require setting a bunch of environment variables. How about adding a little dataclass (or serialize/deserialize functions in BRConfigTest) and using an environment variable to pass the path of a file with serialized config to be loaded in
BRConfigTest.__init__()? I'd probably use JSON because it's easy to handle with the Python standard library.
Yes, that sounds good - Care to send a patch?
Quoted 3 lines
Another option would be to switch to a test runner that has built-in support for passing config (like Pytest), but that'd be a major change, not a simple fix.
Indeed, that is probably what we should do longer term, but not a quick fix.
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Hi Fiona, hi Peter,
Thank you for checking this change.
I am sorry this broke Debian Sid. I should have tested on more distributions.
Quoted 10 lines
Quoted 8 lines
I've tested this with environment variables and it worked fine, though I'd prefer something that allows type checking, and doesn't require setting a bunch of environment variables. How about adding a little dataclass (or serialize/deserialize functions in BRConfigTest) and using an environment variable to pass the path of a file with serialized config to be loaded in
BRConfigTest.__init__()? I'd probably use JSON because it's easy to handle with the Python standard library.Yes, that sounds good - Care to send a patch?
I recall that we are planning to move away from Nose2. Do we still want to invest in fixing this long term?
Quoted 6 lines
Quoted 3 lines
Another option would be to switch to a test runner that has built-in support for passing config (like Pytest), but that'd be a major change, not a simple fix.
Indeed, that is probably what we should do longer term, but not a quick fix.
I can try to support in that effort, but as you could observe, it could take a while for me to find the time.
Best regards,
Signature · 5 lines
Jimmy Durand Wesolowski _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Am 16.02.26 um 10:00 schrieb Jimmy Wesolowski via buildroot:
Quoted 17 lines
Hi Fiona, hi Peter,
Thank you for checking this change.
I am sorry this broke Debian Sid. I should have tested on more distributions.Quoted 10 lines
Quoted 8 lines
I've tested this with environment variables and it worked fine, though I'd prefer something that allows type checking, and doesn't require setting a bunch of environment variables. How about adding a little dataclass (or serialize/deserialize functions in BRConfigTest) and using an environment variable to pass the path of a file with serialized config to be loaded in
BRConfigTest.__init__()? I'd probably use JSON because it's easy to handle with the Python standard library.Yes, that sounds good - Care to send a patch?
I intend to, some time in the next days.
Quoted 2 lines
I recall that we are planning to move away from Nose2. Do we still want to invest in fixing this long term?
This is about fixing the current problem, not a larger rework. :-)
That said, so far I see few places in the testing code that rely on nose2 specifically (run-tests, and the config that enables the "mp" plugin). The test code itself seems to be mostly just using unittest classes, as far as I've seen. If there isn't anything else hiding in parts I haven't looked at switching to another runner that can handle unittest classes shouldn't be too hard (but I'd still want to fix the immediate issue first).
Best regards,
Fiona
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Another option would be to switch to a test runner that has built-in support for passing config (like Pytest), but that'd be a major change, not a simple fix.
Quoted 8 lines
Quoted 2 lines
Indeed, that is probably what we should do longer term, but not a quick fix.
I can try to support in that effort, but as you could observe, it could take a while for me to find the time.
Best regards,
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Thomas Perale via buildroot
AppliedIn reply of:
Quoted 11 lines
Since Python 3.14, the multiprocessing library API has changed:
Quoted 3 lines
On POSIX platforms the default start method was changed from fork to forkserver to retain the performance but avoid common multithreaded process incompatibilities.
https://docs.python.org/3/library/multiprocessing.html
Since this is expected to be under "if
__name__== '__main__'", we need to have this in support/testing/run-tests regardless of how this will be fixed in nose2.Signed-off-by Jimmy Durand Wesolowski <[email protected]>
Applied to 2025.02.x & 2025.11.x. Thanks
Quoted 21 lines
--- support/testing/run-tests | 1 + 1 file changed, 1 insertion(+)support/testing/run-tests +1
-0@@ -142,4 +142,5 @@ def main(): if __name__ == "__main__": + multiprocessing.set_start_method("fork") sys.exit(main())Signature · 6 lines
2.43.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot