python: misc cleanups for python code
cover python: misc cleanups for python code
This spun out of understanding why we didn't get prior warning about our use of the deprecated 'asyncio.get_event_loop' feature. The answer is that python doesn't print any deprecation messages by default unless directly triggered by the __main__ module.
So here I wanted to get all python warning messages enabled for the iotests and functional tests, so we get a heads up of any other failures coming out way.
In doing this it uncovered numerous places where we failed to cleanup with dangling file handles.
I further uncovered lots of old compat code for py3.6 which we stopped needing a long while ago.
I'm in two minds over whether the very last commit is suitable or not though. The iotests do strict comparison of stdout / stderr, so if we allow python warnings to appear that will break the iotests. At the same time though we do really want to see these warnings, especially when developers run the iotests manually, so just turning them on in CI is not very desirable. So wondering how to get the warnings for devs but not for users running from released tarballs?
Also note that I ran 'check', 'check -qcow' and 'check -nbd' but this doesn't cover all possible iotest usage. So there might still be some resource leaks hiding in there which will cause failures for devs running tests with unusual config scenarios.
Daniel P. Berrangé (13):
python: avoid deprecation warning with get_event_loop
python: drop 'is_closing' back compat helper
python: drop 'wait_closed' back compat helper
python: drop 'asyncio_run' back compat helper
python: drop 'create_task' back compat helper
iotests: drop compat for old version context manager
functional: ensure log handlers are closed
functional: ensure sockets and files are closed
functional: always enable all python warnings
python: ensure QEMUQtestProtocol closes its socket iotests/147: ensure temporary sockets are closed before exiting iotests/151: ensure subprocesses are cleaned up iotests/check: always enable all python warnings
Richard W.M. Jones (1):
python: Replace asyncio.get_event_loop for Python 3.14
python/qemu/machine/qtest.py | 2 + python/qemu/qmp/legacy.py | 10 ++- python/qemu/qmp/protocol.py | 13 ++-- python/qemu/qmp/qmp_tui.py | 10 +-- python/qemu/qmp/util.py | 97 +------------------------- python/tests/protocol.py | 9 ++- tests/functional/qemu_test/testcase.py | 6 ++ tests/functional/test_multiprocess.py | 3 + tests/functional/test_virtio_gpu.py | 2 + tests/qemu-iotests/147 | 1 + tests/qemu-iotests/151 | 5 ++ tests/qemu-iotests/check | 4 ++ tests/qemu-iotests/testenv.py | 7 +- tests/qemu-iotests/testrunner.py | 9 +-- 14 files changed, 51 insertions(+), 127 deletions(-)
Signature · 1 line
2.49.0
PATCH 1/14 python: Replace asyncio.get_event_loop for Python 3.14
From: Richard W.M. Jones <[email protected]>
In Python 3.14, no asyncio event loop gets generated automatically. Instead create one when we need it. This should work with Python 3.13 as well. This change was suggested here:
https://bugzilla.redhat.com/show_bug.cgi?id=2375004#c4
See-also: https://docs.python.org/3.14/whatsnew/3.14.html#id7
Thanks: Miro Hrončok, Daniel P. Berrangé
--- python/qemu/qmp/legacy.py | 5 ++++- python/qemu/qmp/qmp_tui.py | 2 +- python/tests/protocol.py | 2 +- 3 files changed, 6 insertions(+), 3 deletions(-)
python/qemu/qmp/legacy.py +4 -1
@@ -86,7 +86,10 @@ def __init__(self, "server argument should be False when passing a socket") self._qmp = QMPClient(nickname) - self._aloop = asyncio.get_event_loop() + try: + self._aloop = asyncio.get_event_loop() + except RuntimeError: + self._aloop = asyncio.new_event_loop() self._address = address self._timeout: Optional[float] = None
python/qemu/qmp/qmp_tui.py +1 -1
@@ -377,7 +377,7 @@ def run(self, debug: bool = False) -> None: screen = urwid.raw_display.Screen() screen.set_terminal_properties(256) - self.aloop = asyncio.get_event_loop() + self.aloop = asyncio.new_event_loop() self.aloop.set_debug(debug) # Gracefully handle SIGTERM and SIGINT signals
python/tests/protocol.py +1 -1
@@ -228,7 +228,7 @@ def async_test(async_test_method): Decorator; adds SetUp and TearDown to async tests. """ async def _wrapper(self, *args, **kwargs): - loop = asyncio.get_event_loop() + loop = asyncio.new_event_loop() loop.set_debug(True) await self._asyncSetUp()
Signature · 1 line
2.49.0
Re: [PATCH 01/14] python: Replace asyncio.get_event_loop for Python 3.14
Quoted 7 lines · Daniel P. Berrangé
From: Richard W.M. Jones <[email protected]>
In Python 3.14, no asyncio event loop gets generated automatically. Instead create one when we need it. This should work with Python 3.13 as well. This change was suggested here:
Quoted 5 lines
https://bugzilla.redhat.com/show_bug.cgi?id=2375004#c4
See-also: https://docs.python.org/3.14/whatsnew/3.14.html#id7
Thanks: Miro Hrončok, Daniel P. BerrangéSigned-off-by Richard W.M. Jones <[email protected]>
We fixed this in the standalone tree, but I hadn't synchronized the changes over yet. Shall we take your patches first, and then I'll worry about synchronizing later?
(I desperately need to drop this code from the qemu.git tree, urgh.)
Quoted 51 lines
--- python/qemu/qmp/legacy.py | 5 ++++- python/qemu/qmp/qmp_tui.py | 2 +- python/tests/protocol.py | 2 +- 3 files changed, 6 insertions(+), 3 deletions(-)python/qemu/qmp/legacy.py +4
-1@@ -86,7 +86,10 @@ def __init__(self, "server argument should be False when passing a socket") self._qmp = QMPClient(nickname) - self._aloop = asyncio.get_event_loop() + try: + self._aloop = asyncio.get_event_loop() + except RuntimeError: + self._aloop = asyncio.new_event_loop() self._address = address self._timeout: Optional[float] = Nonepython/qemu/qmp/qmp_tui.py +1
-1@@ -377,7 +377,7 @@ def run(self, debug: bool = False) -> None: screen = urwid.raw_display.Screen() screen.set_terminal_properties(256) - self.aloop = asyncio.get_event_loop() + self.aloop = asyncio.new_event_loop() self.aloop.set_debug(debug) # Gracefully handle SIGTERM and SIGINT signalspython/tests/protocol.py +1
-2@@ -228,7 +228,7 @@ def async_test(async_test_method): Decorator; adds SetUp and TearDown to async tests. """ async def _wrapper(self, *args, **kwargs): - loop = asyncio.get_event_loop() + loop = asyncio.new_event_loop() loop.set_debug(True) await self._asyncSetUp() --2.49.0
Re: [PATCH 01/14] python: Replace asyncio.get_event_loop for Python 3.14
Quoted 14 lines · John Snow
Quoted 7 lines · Daniel P. Berrangé
From: Richard W.M. Jones <[email protected]>
In Python 3.14, no asyncio event loop gets generated automatically. Instead create one when we need it. This should work with Python 3.13 as well. This change was suggested here:
Quoted 5 lines
https://bugzilla.redhat.com/show_bug.cgi?id=2375004#c4
See-also: https://docs.python.org/3.14/whatsnew/3.14.html#id7
Thanks: Miro Hrončok, Daniel P. BerrangéSigned-off-by Richard W.M. Jones <[email protected]>
Quoted 5 lines
Reviewed-by John Snow <[email protected]>We fixed this in the standalone tree, but I hadn't synchronized the changes over yet. Shall we take your patches first, and then I'll worry about synchronizing later?
This really needs to be in 10.1.0, because tests crash and burn under python 3.14 without this - we're applying this in Fedora builds locally to deal with the brokeness.
Quoted 1 line
(I desperately need to drop this code from the qemu.git tree, urgh.)
IMHO qemu.git should be considered authoritative right now because that's what we're actually shipping & thus take priority for merge.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 01/14] python: Replace asyncio.get_event_loop for Python 3.14
Quoted 18 lines · Daniel P. Berrangé
Quoted 16 lines · John Snow
Quoted 7 lines · Daniel P. Berrangé
From: Richard W.M. Jones <[email protected]>
In Python 3.14, no asyncio event loop gets generated automatically. Instead create one when we need it. This should work with Python 3.13 as well. This change was suggested here:
Quoted 5 lines
https://bugzilla.redhat.com/show_bug.cgi?id=2375004#c4
See-also: https://docs.python.org/3.14/whatsnew/3.14.html#id7
Thanks: Miro Hrončok, Daniel P. BerrangéSigned-off-by Richard W.M. Jones <[email protected]>Reviewed-by John Snow <[email protected]>
Quoted 7 lines
Quoted 3 lines
We fixed this in the standalone tree, but I hadn't synchronized the changes over yet. Shall we take your patches first, and then I'll worry about synchronizing later?
This really needs to be in 10.1.0, because tests crash and burn under python 3.14 without this - we're applying this in Fedora builds locally to deal with the brokeness.
Urk. I didn't realize we were waiting on this; please include it with my blessings and apologies.
Quoted 5 lines
Quoted 2 lines
(I desperately need to drop this code from the qemu.git tree, urgh.)
IMHO qemu.git should be considered authoritative right now because that's what we're actually shipping & thus take priority for merge.
Yup, you're right. I just mean I need to stop the duplication, because stuff like this happens.
Quoted 8 lines
With regards,
Daniel
--|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 2/14 python: avoid deprecation warning with get_event_loop
In reply to Daniel P. BerrangéWe need to call get_event_loop but have no way of knowing ahead of time whether the current thread has an event loop of not. We can handle a missing event loop, but we need to hide the warning python will emit to avoid tripping up iotests expected output.
--- python/qemu/qmp/legacy.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
python/qemu/qmp/legacy.py +6 -1
@@ -34,6 +34,7 @@ TypeVar, Union, ) +import warnings from .error import QMPError from .protocol import Runstate, SocketAddrT
@@ -87,7 +88,11 @@ def __init__(self, self._qmp = QMPClient(nickname) try: - self._aloop = asyncio.get_event_loop() + with warnings.catch_warnings(): + # Python <= 3.13 will trigger deprecation warnings + # if no event loop is set + warnings.simplefilter("ignore") + self._aloop = asyncio.get_event_loop() except RuntimeError: self._aloop = asyncio.new_event_loop() self._address = address
Signature · 1 line
2.49.0
Re: [PATCH 02/14] python: avoid deprecation warning with get_event_loop
Quoted 2 lines · Daniel P. Berrangé
We need to call
get_event_loopbut have no way of knowing ahead of time whether the current thread has an event loop of not. We
s/of not /or not /
Quoted 8 lines
can handle a missing event loop, but we need to hide the warning python will emit to avoid tripping up iotests expected output.
Signed-off-by Daniel P. Berrangé <[email protected]>--- python/qemu/qmp/legacy.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. Virtualization: qemu.org | libguestfs.org
John Snow
Reviewed-byRe: [PATCH 02/14] python: avoid deprecation warning with get_event_loop
Quoted 11 lines · Eric Blake
Quoted 2 lines · Daniel P. Berrangé
We need to call
get_event_loopbut have no way of knowing ahead of time whether the current thread has an event loop of not. Wes/of not /or not /
Quoted 4 lines
can handle a missing event loop, but we need to hide the warning python will emit to avoid tripping up iotests expected output.
Signed-off-by Daniel P. Berrangé <[email protected]>
Do you mind if I collect this patch and apply it against the standalone tree?
--js
Quoted 12 lines
Quoted 4 lines
--- python/qemu/qmp/legacy.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)Reviewed-by Eric Blake <[email protected]>--
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization: qemu.org | libguestfs.org
Re: [PATCH 02/14] python: avoid deprecation warning with get_event_loop
Quoted 8 lines · John Snow
Quoted 7 lines · Eric Blake
Quoted 2 lines · Daniel P. Berrangé
We need to call
get_event_loopbut have no way of knowing ahead of time whether the current thread has an event loop of not. Wes/of not /or not /
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
can handle a missing event loop, but we need to hide the warning python will emit to avoid tripping up iotests expected output.
Signed-off-by Daniel P. Berrangé <[email protected]>Reviewed-by John Snow <[email protected]>Do you mind if I collect this patch and apply it against the standalone tree?
If it applies cleanly, then go ahead, and likewise for anything else following in this series.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 3/14 python: drop 'is_closing' back compat helper
In reply to Daniel P. BerrangéOur minimum python is now 3.9, so back compat with python 3.6 is no longer required.
--- python/qemu/qmp/protocol.py | 3 +-- python/qemu/qmp/util.py | 16 ---------------- 2 files changed, 1 insertion(+), 18 deletions(-)
python/qemu/qmp/protocol.py +1 -2
@@ -39,7 +39,6 @@ create_task, exception_summary, flush, - is_closing, pretty_traceback, upper_half, wait_closed,
@@ -825,7 +824,7 @@ async def _bh_close_stream(self, error_pathway: bool = False) -> None: if not self._writer: return - if not is_closing(self._writer): + if not self._writer.is_closing(): self.logger.debug("Closing StreamWriter.") self._writer.close()
python/qemu/qmp/util.py +0 -16
@@ -104,22 +104,6 @@ def create_task(coro: Coroutine[Any, Any, T], return asyncio.ensure_future(coro, loop=loop) -def is_closing(writer: asyncio.StreamWriter) -> bool: - """ - Python 3.6-compatible `asyncio.StreamWriter.is_closing` wrapper. - - :param writer: The `asyncio.StreamWriter` object. - :return: `True` if the writer is closing, or closed. - """ - if sys.version_info >= (3, 7): - return writer.is_closing() - - # Python 3.6: - transport = writer.transport - assert isinstance(transport, asyncio.WriteTransport) - return transport.is_closing() - - async def wait_closed(writer: asyncio.StreamWriter) -> None: """ Python 3.6-compatible `asyncio.StreamWriter.wait_closed` wrapper.
Signature · 1 line
2.49.0
Re: [PATCH 03/14] python: drop 'is_closing' back compat helper
Quoted 4 lines · Daniel P. Berrangé
Our minimum python is now 3.9, so back compat with python 3.6 is no longer required.
Signed-off-by Daniel P. Berrangé <[email protected]>
Signature · 2 lines
Alex Bennée Virtualisation Tech Lead @ Linaro
Re: [PATCH 03/14] python: drop 'is_closing' back compat helper
In reply to Daniel P. BerrangéQuoted 5 lines · Daniel P. Berrangé
Our minimum python is now 3.9, so back compat with python 3.6 is no longer required.
Signed-off-by Daniel P. Berrangé <[email protected]>
Another thing that happened in the standalone tree, but not in qemu.git. Ditto patches 4, 5, 6.
Quoted 56 lines
--- python/qemu/qmp/protocol.py | 3 +-- python/qemu/qmp/util.py | 16 ---------------- 2 files changed, 1 insertion(+), 18 deletions(-)python/qemu/qmp/protocol.py +1
-2@@ -39,7 +39,6 @@ create_task, exception_summary, flush, - is_closing, pretty_traceback, upper_half, wait_closed,@@ -825,7 +824,7 @@ async def _bh_close_stream(self, error_pathway: bool = False) -> None: if not self._writer: return - if not is_closing(self._writer): + if not self._writer.is_closing(): self.logger.debug("Closing StreamWriter.") self._writer.close()python/qemu/qmp/util.py +0
-17@@ -104,22 +104,6 @@ def create_task(coro: Coroutine[Any, Any, T], return asyncio.ensure_future(coro, loop=loop) -def is_closing(writer: asyncio.StreamWriter) -> bool: - """ - Python 3.6-compatible `asyncio.StreamWriter.is_closing` wrapper. - - :param writer: The `asyncio.StreamWriter` object. - :return: `True` if the writer is closing, or closed. - """ - if sys.version_info >= (3, 7): - return writer.is_closing() - - # Python 3.6: - transport = writer.transport - assert isinstance(transport, asyncio.WriteTransport) - return transport.is_closing() - - async def wait_closed(writer: asyncio.StreamWriter) -> None: """ Python 3.6-compatible `asyncio.StreamWriter.wait_closed` wrapper. --2.49.0
PATCH 4/14 python: drop 'wait_closed' back compat helper
In reply to Daniel P. BerrangéOur minimum python is now 3.9, so back compat with python 3.6 is no longer required.
--- python/qemu/qmp/protocol.py | 3 +-- python/qemu/qmp/util.py | 29 ----------------------------- 2 files changed, 1 insertion(+), 31 deletions(-)
python/qemu/qmp/protocol.py +1 -2
@@ -41,7 +41,6 @@ flush, pretty_traceback, upper_half, - wait_closed, )
@@ -830,7 +829,7 @@ async def _bh_close_stream(self, error_pathway: bool = False) -> None: self.logger.debug("Waiting for StreamWriter to close ...") try: - await wait_closed(self._writer) + await self._writer.wait_closed() except Exception: # pylint: disable=broad-except # It's hard to tell if the Stream is already closed or # not. Even if one of the tasks has failed, it may have
python/qemu/qmp/util.py +0 -29
@@ -104,35 +104,6 @@ def create_task(coro: Coroutine[Any, Any, T], return asyncio.ensure_future(coro, loop=loop) -async def wait_closed(writer: asyncio.StreamWriter) -> None: - """ - Python 3.6-compatible `asyncio.StreamWriter.wait_closed` wrapper. - - :param writer: The `asyncio.StreamWriter` to wait on. - """ - if sys.version_info >= (3, 7): - await writer.wait_closed() - return - - # Python 3.6 - transport = writer.transport - assert isinstance(transport, asyncio.WriteTransport) - - while not transport.is_closing(): - await asyncio.sleep(0) - - # This is an ugly workaround, but it's the best I can come up with. - sock = transport.get_extra_info('socket') - - if sock is None: - # Our transport doesn't have a socket? ... - # Nothing we can reasonably do. - return - - while sock.fileno() != -1: - await asyncio.sleep(0) - - def asyncio_run(coro: Coroutine[Any, Any, T], *, debug: bool = False) -> T: """ Python 3.6-compatible `asyncio.run` wrapper.
Signature · 1 line
2.49.0
Re: [PATCH 04/14] python: drop 'wait_closed' back compat helper
Quoted 5 lines · Daniel P. Berrangé
Our minimum python is now 3.9, so back compat with python 3.6 is no longer required.
Signed-off-by Daniel P. Berrangé <[email protected]>
Quoted 69 lines
--- python/qemu/qmp/protocol.py | 3 +-- python/qemu/qmp/util.py | 29 ----------------------------- 2 files changed, 1 insertion(+), 31 deletions(-)python/qemu/qmp/protocol.py +1
-2@@ -41,7 +41,6 @@ flush, pretty_traceback, upper_half, - wait_closed, )@@ -830,7 +829,7 @@ async def _bh_close_stream(self, error_pathway: bool = False) -> None: self.logger.debug("Waiting for StreamWriter to close ...") try: - await wait_closed(self._writer) + await self._writer.wait_closed() except Exception: # pylint: disable=broad-except # It's hard to tell if the Stream is already closed or # not. Even if one of the tasks has failed, it may havepython/qemu/qmp/util.py +0
-30@@ -104,35 +104,6 @@ def create_task(coro: Coroutine[Any, Any, T], return asyncio.ensure_future(coro, loop=loop) -async def wait_closed(writer: asyncio.StreamWriter) -> None: - """ - Python 3.6-compatible `asyncio.StreamWriter.wait_closed` wrapper. - - :param writer: The `asyncio.StreamWriter` to wait on. - """ - if sys.version_info >= (3, 7): - await writer.wait_closed() - return - - # Python 3.6 - transport = writer.transport - assert isinstance(transport, asyncio.WriteTransport) - - while not transport.is_closing(): - await asyncio.sleep(0) - - # This is an ugly workaround, but it's the best I can come up with. - sock = transport.get_extra_info('socket') - - if sock is None: - # Our transport doesn't have a socket? ... - # Nothing we can reasonably do. - return - - while sock.fileno() != -1: - await asyncio.sleep(0) - - def asyncio_run(coro: Coroutine[Any, Any, T], *, debug: bool = False) -> T: """ Python 3.6-compatible `asyncio.run` wrapper. --2.49.0
PATCH 5/14 python: drop 'asyncio_run' back compat helper
In reply to Daniel P. BerrangéOur minimum python is now 3.9, so back compat with python 3.6 is no longer required.
--- python/qemu/qmp/util.py | 19 ------------------- python/tests/protocol.py | 2 +- 2 files changed, 1 insertion(+), 20 deletions(-)
python/qemu/qmp/util.py +0 -19
@@ -104,25 +104,6 @@ def create_task(coro: Coroutine[Any, Any, T], return asyncio.ensure_future(coro, loop=loop) -def asyncio_run(coro: Coroutine[Any, Any, T], *, debug: bool = False) -> T: - """ - Python 3.6-compatible `asyncio.run` wrapper. - - :param coro: A coroutine to execute now. - :return: The return value from the coroutine. - """ - if sys.version_info >= (3, 7): - return asyncio.run(coro, debug=debug) - - # Python 3.6 - loop = asyncio.get_event_loop() - loop.set_debug(debug) - ret = loop.run_until_complete(coro) - loop.close() - - return ret - - # ---------------------------- # Section: Logging & Debugging # ----------------------------
python/tests/protocol.py +1 -1
@@ -8,7 +8,7 @@ from qemu.qmp import ConnectError, Runstate from qemu.qmp.protocol import AsyncProtocol, StateError -from qemu.qmp.util import asyncio_run, create_task +from qemu.qmp.util import create_task class NullProtocol(AsyncProtocol[None]):
Signature · 1 line
2.49.0
Re: [PATCH 05/14] python: drop 'asyncio_run' back compat helper
Quoted 5 lines · Daniel P. Berrangé
Our minimum python is now 3.9, so back compat with python 3.6 is no longer required.
Signed-off-by Daniel P. Berrangé <[email protected]>
Quoted 51 lines
--- python/qemu/qmp/util.py | 19 ------------------- python/tests/protocol.py | 2 +- 2 files changed, 1 insertion(+), 20 deletions(-)python/qemu/qmp/util.py +0
-19@@ -104,25 +104,6 @@ def create_task(coro: Coroutine[Any, Any, T], return asyncio.ensure_future(coro, loop=loop) -def asyncio_run(coro: Coroutine[Any, Any, T], *, debug: bool = False) -> T: - """ - Python 3.6-compatible `asyncio.run` wrapper. - - :param coro: A coroutine to execute now. - :return: The return value from the coroutine. - """ - if sys.version_info >= (3, 7): - return asyncio.run(coro, debug=debug) - - # Python 3.6 - loop = asyncio.get_event_loop() - loop.set_debug(debug) - ret = loop.run_until_complete(coro) - loop.close() - - return ret - - # ---------------------------- # Section: Logging & Debugging # ----------------------------python/tests/protocol.py +1
-2@@ -8,7 +8,7 @@ from qemu.qmp import ConnectError, Runstate from qemu.qmp.protocol import AsyncProtocol, StateError -from qemu.qmp.util import asyncio_run, create_task +from qemu.qmp.util import create_task class NullProtocol(AsyncProtocol[None]): --2.49.0
PATCH 6/14 python: drop 'create_task' back compat helper
In reply to Daniel P. BerrangéOur minimum python is now 3.9, so back compat with python 3.6 is no longer required.
--- python/qemu/qmp/protocol.py | 7 +++---- python/qemu/qmp/qmp_tui.py | 8 ++++---- python/qemu/qmp/util.py | 33 +-------------------------------- python/tests/protocol.py | 7 +++---- 4 files changed, 11 insertions(+), 44 deletions(-)
python/qemu/qmp/protocol.py +3 -4
@@ -36,7 +36,6 @@ from .error import QMPError from .util import ( bottom_half, - create_task, exception_summary, flush, pretty_traceback,
@@ -661,8 +660,8 @@ async def _establish_session(self) -> None: reader_coro = self._bh_loop_forever(self._bh_recv_message, 'Reader') writer_coro = self._bh_loop_forever(self._bh_send_message, 'Writer') - self._reader_task = create_task(reader_coro) - self._writer_task = create_task(writer_coro) + self._reader_task = asyncio.create_task(reader_coro) + self._writer_task = asyncio.create_task(writer_coro) self._bh_tasks = asyncio.gather( self._reader_task,
@@ -687,7 +686,7 @@ def _schedule_disconnect(self) -> None: if not self._dc_task: self._set_state(Runstate.DISCONNECTING) self.logger.debug("Scheduling disconnect.") - self._dc_task = create_task(self._bh_disconnect()) + self._dc_task = asyncio.create_task(self._bh_disconnect()) @upper_half async def _wait_disconnect(self) -> None:
python/qemu/qmp/qmp_tui.py +4 -4
@@ -40,7 +40,7 @@ from .message import DeserializationError, Message, UnexpectedTypeError from .protocol import ConnectError, Runstate from .qmp_client import ExecInterruptedError, QMPClient -from .util import create_task, pretty_traceback +from .util import pretty_traceback # The name of the signal that is used to update the history list
@@ -225,7 +225,7 @@ def cb_send_to_server(self, raw_msg: str) -> None: """ try: msg = Message(bytes(raw_msg, encoding='utf-8')) - create_task(self._send_to_server(msg)) + asyncio.create_task(self._send_to_server(msg)) except (DeserializationError, UnexpectedTypeError) as err: raw_msg = format_json(raw_msg) logging.info('Invalid message: %s', err.error_message)
@@ -246,7 +246,7 @@ def kill_app(self) -> None: Initiates killing of app. A bridge between asynchronous and synchronous code. """ - create_task(self._kill_app()) + asyncio.create_task(self._kill_app()) async def _kill_app(self) -> None: """
@@ -393,7 +393,7 @@ def run(self, debug: bool = False) -> None: handle_mouse=True, event_loop=event_loop) - create_task(self.manage_connection(), self.aloop) + self.aloop.create_task(self.manage_connection()) try: main_loop.run() except Exception as err:
python/qemu/qmp/util.py +1 -32
@@ -13,13 +13,7 @@ import asyncio import sys import traceback -from typing import ( - Any, - Coroutine, - Optional, - TypeVar, - cast, -) +from typing import TypeVar, cast T = TypeVar('T')
@@ -79,31 +73,6 @@ def bottom_half(func: T) -> T: return func -# ------------------------------- -# Section: Compatibility Wrappers -# ------------------------------- - - -def create_task(coro: Coroutine[Any, Any, T], - loop: Optional[asyncio.AbstractEventLoop] = None - ) -> 'asyncio.Future[T]': - """ - Python 3.6-compatible `asyncio.create_task` wrapper. - - :param coro: The coroutine to execute in a task. - :param loop: Optionally, the loop to create the task in. - - :return: An `asyncio.Future` object. - """ - if sys.version_info >= (3, 7): - if loop is not None: - return loop.create_task(coro) - return asyncio.create_task(coro) # pylint: disable=no-member - - # Python 3.6: - return asyncio.ensure_future(coro, loop=loop) - - # ---------------------------- # Section: Logging & Debugging # ----------------------------
python/tests/protocol.py +3 -4
@@ -8,7 +8,6 @@ from qemu.qmp import ConnectError, Runstate from qemu.qmp.protocol import AsyncProtocol, StateError -from qemu.qmp.util import create_task class NullProtocol(AsyncProtocol[None]):
@@ -124,7 +123,7 @@ async def _runner(): if allow_cancellation: return raise - return create_task(_runner()) + return asyncio.create_task(_runner()) @contextmanager
@@ -271,7 +270,7 @@ async def _watcher(): msg=f"Expected state '{state.name}'", ) - self.runstate_watcher = create_task(_watcher()) + self.runstate_watcher = asyncio.create_task(_watcher()) # Kick the loop and force the task to block on the event. await asyncio.sleep(0)
@@ -589,7 +588,7 @@ async def _asyncTearDown(self): async def testSmoke(self): with TemporaryDirectory(suffix='.qmp') as tmpdir: sock = os.path.join(tmpdir, type(self.proto).__name__ + ".sock") - server_task = create_task(self.server.start_server_and_accept(sock)) + server_task = asyncio.create_task(self.server.start_server_and_accept(sock)) # give the server a chance to start listening [...] await asyncio.sleep(0)
Signature · 1 line
2.49.0
Re: [PATCH 06/14] python: drop 'create_task' back compat helper
Quoted 5 lines · Daniel P. Berrangé
Our minimum python is now 3.9, so back compat with python 3.6 is no longer required.
Signed-off-by Daniel P. Berrangé <[email protected]>
Quoted 172 lines
--- python/qemu/qmp/protocol.py | 7 +++---- python/qemu/qmp/qmp_tui.py | 8 ++++---- python/qemu/qmp/util.py | 33 +-------------------------------- python/tests/protocol.py | 7 +++---- 4 files changed, 11 insertions(+), 44 deletions(-)python/qemu/qmp/protocol.py +3
-4@@ -36,7 +36,6 @@ from .error import QMPError from .util import ( bottom_half, - create_task, exception_summary, flush, pretty_traceback,@@ -661,8 +660,8 @@ async def _establish_session(self) -> None: reader_coro = self._bh_loop_forever(self._bh_recv_message, 'Reader') writer_coro = self._bh_loop_forever(self._bh_send_message, 'Writer') - self._reader_task = create_task(reader_coro) - self._writer_task = create_task(writer_coro) + self._reader_task = asyncio.create_task(reader_coro) + self._writer_task = asyncio.create_task(writer_coro) self._bh_tasks = asyncio.gather( self._reader_task,@@ -687,7 +686,7 @@ def _schedule_disconnect(self) -> None: if not self._dc_task: self._set_state(Runstate.DISCONNECTING) self.logger.debug("Scheduling disconnect.") - self._dc_task = create_task(self._bh_disconnect()) + self._dc_task = asyncio.create_task(self._bh_disconnect()) @upper_half async def _wait_disconnect(self) -> None:python/qemu/qmp/qmp_tui.py +4
-4@@ -40,7 +40,7 @@ from .message import DeserializationError, Message, UnexpectedTypeError from .protocol import ConnectError, Runstate from .qmp_client import ExecInterruptedError, QMPClient -from .util import create_task, pretty_traceback +from .util import pretty_traceback # The name of the signal that is used to update the history list@@ -225,7 +225,7 @@ def cb_send_to_server(self, raw_msg: str) -> None: """ try: msg = Message(bytes(raw_msg, encoding='utf-8')) - create_task(self._send_to_server(msg)) + asyncio.create_task(self._send_to_server(msg)) except (DeserializationError, UnexpectedTypeError) as err: raw_msg = format_json(raw_msg) logging.info('Invalid message: %s', err.error_message)@@ -246,7 +246,7 @@ def kill_app(self) -> None: Initiates killing of app. A bridge between asynchronous and synchronous code. """ - create_task(self._kill_app()) + asyncio.create_task(self._kill_app()) async def _kill_app(self) -> None: """@@ -393,7 +393,7 @@ def run(self, debug: bool = False) -> None: handle_mouse=True, event_loop=event_loop) - create_task(self.manage_connection(), self.aloop) + self.aloop.create_task(self.manage_connection()) try: main_loop.run() except Exception as err:python/qemu/qmp/util.py +1
-32@@ -13,13 +13,7 @@ import asyncio import sys import traceback -from typing import ( - Any, - Coroutine, - Optional, - TypeVar, - cast, -) +from typing import TypeVar, cast T = TypeVar('T')@@ -79,31 +73,6 @@ def bottom_half(func: T) -> T: return func -# ------------------------------- -# Section: Compatibility Wrappers -# ------------------------------- - - -def create_task(coro: Coroutine[Any, Any, T], - loop: Optional[asyncio.AbstractEventLoop] = None - ) -> 'asyncio.Future[T]': - """ - Python 3.6-compatible `asyncio.create_task` wrapper. - - :param coro: The coroutine to execute in a task. - :param loop: Optionally, the loop to create the task in. - - :return: An `asyncio.Future` object. - """ - if sys.version_info >= (3, 7): - if loop is not None: - return loop.create_task(coro) - return asyncio.create_task(coro) # pylint: disable=no-member - - # Python 3.6: - return asyncio.ensure_future(coro, loop=loop) - - # ---------------------------- # Section: Logging & Debugging # ----------------------------python/tests/protocol.py +3
-5@@ -8,7 +8,6 @@ from qemu.qmp import ConnectError, Runstate from qemu.qmp.protocol import AsyncProtocol, StateError -from qemu.qmp.util import create_task class NullProtocol(AsyncProtocol[None]):@@ -124,7 +123,7 @@ async def _runner(): if allow_cancellation: return raise - return create_task(_runner()) + return asyncio.create_task(_runner()) @contextmanager@@ -271,7 +270,7 @@ async def _watcher(): msg=f"Expected state '{state.name}'", ) - self.runstate_watcher = create_task(_watcher()) + self.runstate_watcher = asyncio.create_task(_watcher()) # Kick the loop and force the task to block on the event. await asyncio.sleep(0)@@ -589,7 +588,7 @@ async def _asyncTearDown(self): async def testSmoke(self): with TemporaryDirectory(suffix='.qmp') as tmpdir: sock = os.path.join(tmpdir, type(self.proto).__name__ + ".sock") - server_task = create_task(self.server.start_server_and_accept(sock)) + server_task = asyncio.create_task(self.server.start_server_and_accept(sock)) # give the server a chance to start listening [...] await asyncio.sleep(0) --2.49.0
PATCH 7/14 iotests: drop compat for old version context manager
In reply to Daniel P. BerrangéOur minimum python is now 3.9, so back compat with prior python versions is no longer required.
--- tests/qemu-iotests/testenv.py | 7 ++----- tests/qemu-iotests/testrunner.py | 9 ++------- 2 files changed, 4 insertions(+), 12 deletions(-)
tests/qemu-iotests/testenv.py +2 -5
@@ -22,15 +22,12 @@ from pathlib import Path import shutil import collections +import contextlib import random import subprocess import glob from typing import List, Dict, Any, Optional -if sys.version_info >= (3, 9): - from contextlib import AbstractContextManager as ContextManager -else: - from typing import ContextManager DEF_GDB_OPTIONS = 'localhost:12345'
@@ -58,7 +55,7 @@ def get_default_machine(qemu_prog: str) -> str: return default_machine -class TestEnv(ContextManager['TestEnv']): +class TestEnv(contextlib.AbstractContextManager['TestEnv']): """ Manage system environment for running tests
tests/qemu-iotests/testrunner.py +2 -7
@@ -30,11 +30,6 @@ from typing import List, Optional, Any, Sequence, Dict from testenv import TestEnv -if sys.version_info >= (3, 9): - from contextlib import AbstractContextManager as ContextManager -else: - from typing import ContextManager - def silent_unlink(path: Path) -> None: try:
@@ -57,7 +52,7 @@ def file_diff(file1: str, file2: str) -> List[str]: return res -class LastElapsedTime(ContextManager['LastElapsedTime']): +class LastElapsedTime(contextlib.AbstractContextManager['LastElapsedTime']): """ Cache for elapsed time for tests, to show it during new test run It is safe to use get() at any time. To use update(), you must either
@@ -112,7 +107,7 @@ def __init__(self, status: str, description: str = '', self.interrupted = interrupted -class TestRunner(ContextManager['TestRunner']): +class TestRunner(contextlib.AbstractContextManager['TestRunner']): shared_self = None @staticmethod
Signature · 1 line
2.49.0
Re: [PATCH 07/14] iotests: drop compat for old version context manager
Quoted 5 lines · Daniel P. Berrangé
Our minimum python is now 3.9, so back compat with prior python versions is no longer required.
Signed-off-by Daniel P. Berrangé <[email protected]>
PATCH 8/14 functional: ensure log handlers are closed
In reply to Daniel P. BerrangéThis avoids a resource leak warning from python when the log handler is garbage collected.
--- tests/functional/qemu_test/testcase.py | 2 ++ 1 file changed, 2 insertions(+)
tests/functional/qemu_test/testcase.py +2 -0
@@ -232,6 +232,7 @@ def tearDown(self): self.socketdir = None self.machinelog.removeHandler(self._log_fh) self.log.removeHandler(self._log_fh) + self._log_fh.close() def main(): path = os.path.basename(sys.argv[0])[:-3]
@@ -399,4 +400,5 @@ def tearDown(self): for vm in self._vms.values(): vm.shutdown() logging.getLogger('console').removeHandler(self._console_log_fh) + self._console_log_fh.close() super().tearDown()
Signature · 1 line
2.49.0
Re: [PATCH 08/14] functional: ensure log handlers are closed
Quoted 21 lines · Daniel P. Berrangé
This avoids a resource leak warning from python when the log handler is garbage collected.
Signed-off-by Daniel P. Berrangé <[email protected]>--- tests/functional/qemu_test/testcase.py | 2 ++ 1 file changed, 2 insertions(+)tests/functional/qemu_test/testcase.py +1
-0@@ -232,6 +232,7 @@ def tearDown(self): self.socketdir = None self.machinelog.removeHandler(self._log_fh) self.log.removeHandler(self._log_fh) + self._log_fh.close() def main(): path = os.path.basename(sys.argv[0])[:-3]
tests/functional/qemu_test/testcase.py def tearDown(self): · line 404
Quoted 5 lines
tests/functional/qemu_test/testcase.py +1
-0@@ -399,4 +400,5 @@ def tearDown(self): for vm in self._vms.values(): vm.shutdown() logging.getLogger('console').removeHandler(self._console_log_fh) + self._console_log_fh.close() super().tearDown()
PATCH 9/14 functional: ensure sockets and files are closed
In reply to Daniel P. BerrangéThe multiprocess and virtio_gpu tests open sockets but then forget to close them, which triggers resource leak warnings
The virtio_gpu test also fails to close a log file it opens.
--- tests/functional/test_multiprocess.py | 3 +++ tests/functional/test_virtio_gpu.py | 2 ++ 2 files changed, 5 insertions(+)
tests/functional/test_multiprocess.py +3 -0
@@ -83,6 +83,9 @@ def do_test(self, kernel_asset, initrd_asset, 'cat /sys/bus/pci/devices/*/uevent', 'PCI_ID=1000:0012') + proxy_sock.close() + remote_sock.close() + def test_multiprocess(self): kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE if self.arch == 'x86_64':
tests/functional/test_virtio_gpu.py +2 -0
@@ -108,6 +108,7 @@ def test_vhost_user_vga_virgl(self): shell=False, close_fds=False, ) + self._vug_log_file.close() self.vm.set_console() self.vm.add_args("-cpu", "host")
@@ -135,6 +136,7 @@ def test_vhost_user_vga_virgl(self): "features: +virgl +edid") self.vm.shutdown() qemu_sock.close() + vug_sock.close() vugp.terminate() vugp.wait()
Signature · 1 line
2.49.0
Re: [PATCH 09/14] functional: ensure sockets and files are closed
Quoted 30 lines · Daniel P. Berrangé
The multiprocess and
virtio_gputests open sockets but then forget to close them, which triggers resource leak warningsThe
virtio_gputest also fails to close a log file it opens.Signed-off-by Daniel P. Berrangé <[email protected]>--- tests/functional/test_multiprocess.py | 3 +++ tests/functional/test_virtio_gpu.py | 2 ++ 2 files changed, 5 insertions(+)tests/functional/test_multiprocess.py +3
-0@@ -83,6 +83,9 @@ def do_test(self, kernel_asset, initrd_asset, 'cat /sys/bus/pci/devices/*/uevent', 'PCI_ID=1000:0012') + proxy_sock.close() + remote_sock.close() + def test_multiprocess(self): kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE if self.arch == 'x86_64':tests/functional/test_virtio_gpu.py +0
-0
tests/functional/test_virtio_gpu.py def test_vhost_user_vga_virgl(self): · line 111
Quoted 4 lines
tests/functional/test_virtio_gpu.py +1
-0@@ -108,6 +108,7 @@ def test_vhost_user_vga_virgl(self): shell=False, close_fds=False, ) + self._vug_log_file.close()
Maybe cleaner to close it at the end of the function?
Thomas
Quoted 10 lines
tests/functional/test_virtio_gpu.py +1
-0self.vm.set_console() self.vm.add_args("-cpu", "host")@@ -135,6 +136,7 @@ def test_vhost_user_vga_virgl(self): "features: +virgl +edid") self.vm.shutdown() qemu_sock.close() + vug_sock.close() vugp.terminate() vugp.wait()
Re: [PATCH 09/14] functional: ensure sockets and files are closed
Quoted 28 lines · Thomas Huth
Quoted 27 lines · Daniel P. Berrangé
The multiprocess and
virtio_gputests open sockets but then forget to close them, which triggers resource leak warningsThe
virtio_gputest also fails to close a log file it opens.Signed-off-by Daniel P. Berrangé <[email protected]>--- tests/functional/test_multiprocess.py | 3 +++ tests/functional/test_virtio_gpu.py | 2 ++ 2 files changed, 5 insertions(+)tests/functional/test_multiprocess.py +3
-0@@ -83,6 +83,9 @@ def do_test(self, kernel_asset, initrd_asset, 'cat /sys/bus/pci/devices/*/uevent', 'PCI_ID=1000:0012') + proxy_sock.close() + remote_sock.close() + def test_multiprocess(self): kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE if self.arch == 'x86_64':tests/functional/test_virtio_gpu.py +0
-1--- a/tests/functional/test_virtio_gpu.py
Quoted 8 lines
tests/functional/test_virtio_gpu.py def test_vhost_user_vga_virgl(self): · line 111
Quoted 6 lines
+++
b/tests/functional/test_virtio_gpu.pytests/functional/test_virtio_gpu.py +1
-0@@ -108,6 +108,7 @@ def test_vhost_user_vga_virgl(self): shell=False, close_fds=False, ) + self._vug_log_file.close()Maybe cleaner to close it at the end of the function?
This was the last place it was needed - just out of sight in the context of the diff the file is passed to Popen() which connects it to stderr. So nothing in this test may access the file after this point to avoid concurent overlapping writes to the same file.
Quoted 4 lines
Thomas
Quoted 10 lines
tests/functional/test_virtio_gpu.py def test_vhost_user_vga_virgl(self): · line 141
Quoted 9 lines
tests/functional/test_virtio_gpu.py +1
-0self.vm.set_console() self.vm.add_args("-cpu", "host")@@ -135,6 +136,7 @@ def test_vhost_user_vga_virgl(self): "features: +virgl +edid") self.vm.shutdown() qemu_sock.close() + vug_sock.close() vugp.terminate() vugp.wait()
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Thomas Huth
Reviewed-byRe: [PATCH 09/14] functional: ensure sockets and files are closed
Quoted 38 lines · Daniel P. Berrangé
Quoted 28 lines · Thomas Huth
Quoted 27 lines · Daniel P. Berrangé
The multiprocess and
virtio_gputests open sockets but then forget to close them, which triggers resource leak warningsThe
virtio_gputest also fails to close a log file it opens.Signed-off-by Daniel P. Berrangé <[email protected]>--- tests/functional/test_multiprocess.py | 3 +++ tests/functional/test_virtio_gpu.py | 2 ++ 2 files changed, 5 insertions(+)tests/functional/test_multiprocess.py +3
-0@@ -83,6 +83,9 @@ def do_test(self, kernel_asset, initrd_asset, 'cat /sys/bus/pci/devices/*/uevent', 'PCI_ID=1000:0012') + proxy_sock.close() + remote_sock.close() + def test_multiprocess(self): kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE if self.arch == 'x86_64':tests/functional/test_virtio_gpu.py +0
-1--- a/tests/functional/test_virtio_gpu.pyQuoted 8 lines
tests/functional/test_virtio_gpu.py def test_vhost_user_vga_virgl(self): · line 111
Quoted 6 lines
+++
b/tests/functional/test_virtio_gpu.pytests/functional/test_virtio_gpu.py +1
-0@@ -108,6 +108,7 @@ def test_vhost_user_vga_virgl(self): shell=False, close_fds=False, ) + self._vug_log_file.close()Maybe cleaner to close it at the end of the function?
Quoted 4 lines
This was the last place it was needed - just out of sight in the context of the diff the file is passed to
Popen()which connects it to stderr. So nothing in this test may access the file after this point to avoid concurent overlapping writes to the same file.
Ok, makes sense now.
PATCH 10/14 functional: always enable all python warnings
In reply to Daniel P. BerrangéOf most importance is that this gives us a heads-up if anything we rely on has been deprecated. The default python behaviour only emits a warning if triggered from __main__ which is very limited.
Setting the env variable further ensures that any python child processes will also display warnings.
--- tests/functional/qemu_test/testcase.py | 4 ++++ 1 file changed, 4 insertions(+)
tests/functional/qemu_test/testcase.py +4 -0
@@ -19,6 +19,7 @@ from subprocess import run import sys import tempfile +import warnings import unittest import uuid
@@ -235,6 +236,9 @@ def tearDown(self): self._log_fh.close() def main(): + warnings.simplefilter("default") + os.environ["PYTHONWARNINGS"] = "default" + path = os.path.basename(sys.argv[0])[:-3] cache = os.environ.get("QEMU_TEST_PRECACHE", None)
Signature · 1 line
2.49.0
Re: [PATCH 10/14] functional: always enable all python warnings
Quoted 26 lines · Daniel P. Berrangé
Of most importance is that this gives us a heads-up if anything we rely on has been deprecated. The default python behaviour only emits a warning if triggered from
__main__which is very limited.Setting the env variable further ensures that any python child processes will also display warnings.
Signed-off-by Daniel P. Berrangé <[email protected]>--- tests/functional/qemu_test/testcase.py | 4 ++++ 1 file changed, 4 insertions(+)tests/functional/qemu_test/testcase.py +1
-0@@ -19,6 +19,7 @@ from subprocess import run import sys import tempfile +import warnings import unittest import uuid
tests/functional/qemu_test/testcase.py def tearDown(self): · line 244
Quoted 9 lines
tests/functional/qemu_test/testcase.py +3
-0@@ -235,6 +236,9 @@ def tearDown(self): self._log_fh.close() def main(): + warnings.simplefilter("default") + os.environ["PYTHONWARNINGS"] = "default" + path = os.path.basename(sys.argv[0])[:-3] cache = os.environ.get("QEMU_TEST_PRECACHE", None)
Re: [PATCH 10/14] functional: always enable all python warnings
Quoted 11 lines · Thomas Huth
Quoted 9 lines · Daniel P. Berrangé
Of most importance is that this gives us a heads-up if anything we rely on has been deprecated. The default python behaviour only emits a warning if triggered from
__main__which is very limited.Setting the env variable further ensures that any python child processes will also display warnings.
Signed-off-by Daniel P. Berrangé <[email protected]>
Looks like 8, 9, 10 already got merged; skipping.
Quoted 29 lines
Quoted 17 lines
--- tests/functional/qemu_test/testcase.py | 4 ++++ 1 file changed, 4 insertions(+)tests/functional/qemu_test/testcase.py +1
-0@@ -19,6 +19,7 @@ from subprocess import run import sys import tempfile +import warnings import unittest import uuidtests/functional/qemu_test/testcase.py def tearDown(self): · line 244
Quoted 9 lines
tests/functional/qemu_test/testcase.py +3
-0@@ -235,6 +236,9 @@ def tearDown(self): self._log_fh.close() def main(): + warnings.simplefilter("default") + os.environ["PYTHONWARNINGS"] = "default" + path = os.path.basename(sys.argv[0])[:-3] cache = os.environ.get("QEMU_TEST_PRECACHE", None)Acked-by Thomas Huth <[email protected]>
PATCH 11/14 python: ensure QEMUQtestProtocol closes its socket
In reply to Daniel P. BerrangéWhile QEMUQtestMachine closes the socket that was passed to QEMUQtestProtocol, the python resource leak manager still believes that the copy QEMUQtestProtocol holds is open. We must explicitly call close to avoid this leak warnnig.
--- python/qemu/machine/qtest.py | 2 ++ 1 file changed, 2 insertions(+)
python/qemu/machine/qtest.py +2 -0
@@ -177,6 +177,8 @@ def _post_shutdown(self) -> None: self._qtest_sock_pair[0].close() self._qtest_sock_pair[1].close() self._qtest_sock_pair = None + if self._qtest is not None: + self._qtest.close() super()._post_shutdown() def qtest(self, cmd: str) -> str:
Signature · 1 line
2.49.0
Re: [PATCH 11/14] python: ensure QEMUQtestProtocol closes its socket
Quoted 5 lines · Daniel P. Berrangé
While QEMUQtestMachine closes the socket that was passed to QEMUQtestProtocol, the python resource leak manager still believes that the copy QEMUQtestProtocol holds is open. We must explicitly call close to avoid this leak warnnig.
Huh. I wonder if that's a false positive, or if it's actually true?
Quoted 2 lines
Signed-off-by Daniel P. Berrangé <[email protected]>
Well, either way...
Quoted 20 lines
--- python/qemu/machine/qtest.py | 2 ++ 1 file changed, 2 insertions(+)python/qemu/machine/qtest.py +2
-1@@ -177,6 +177,8 @@ def _post_shutdown(self) -> None: self._qtest_sock_pair[0].close() self._qtest_sock_pair[1].close() self._qtest_sock_pair = None + if self._qtest is not None: + self._qtest.close() super()._post_shutdown() def qtest(self, cmd: str) -> str: --2.49.0
Re: [PATCH 11/14] python: ensure QEMUQtestProtocol closes its socket
Quoted 8 lines · John Snow
Quoted 5 lines · Daniel P. Berrangé
While QEMUQtestMachine closes the socket that was passed to QEMUQtestProtocol, the python resource leak manager still believes that the copy QEMUQtestProtocol holds is open. We must explicitly call close to avoid this leak warnnig.
Huh. I wonder if that's a false positive, or if it's actually true?
I'm not sure to be honest, but the extra close doesn't seem to cause any ill effects.
Quoted 24 lines
Quoted 3 lines
Signed-off-by Daniel P. Berrangé <[email protected]>Well, either way...
Reviewed-by John Snow <[email protected]>Quoted 16 lines
--- python/qemu/machine/qtest.py | 2 ++ 1 file changed, 2 insertions(+)python/qemu/machine/qtest.py +2
-0@@ -177,6 +177,8 @@ def _post_shutdown(self) -> None: self._qtest_sock_pair[0].close() self._qtest_sock_pair[1].close() self._qtest_sock_pair = None + if self._qtest is not None: + self._qtest.close() super()._post_shutdown()
Quoted 5 lines
Quoted 4 lines
python/qemu/machine/qtest.py +0
-1def qtest(self, cmd: str) -> str: --2.49.0
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 12/14 iotests/147: ensure temporary sockets are closed before exiting
In reply to Daniel P. BerrangéThis avoids the python resource leak detector from issuing warnings in the iotests.
--- tests/qemu-iotests/147 | 1 + 1 file changed, 1 insertion(+)
tests/qemu-iotests/147 +1 -0
@@ -277,6 +277,7 @@ class BuiltinNBD(NBDBlockdevAddBase): } } self.client_test(filename, flatten_sock_addr(address), 'nbd-export') + sockfd.close() self._server_down()
Signature · 1 line
2.49.0
Re: [PATCH 12/14] iotests/147: ensure temporary sockets are closed before exiting
Quoted 5 lines · Daniel P. Berrangé
This avoids the python resource leak detector from issuing warnings in the iotests.
Signed-off-by Daniel P. Berrangé <[email protected]>
Quoted 19 lines
--- tests/qemu-iotests/147 | 1 + 1 file changed, 1 insertion(+)tests/qemu-iotests/147 +1
-1@@ -277,6 +277,7 @@ class BuiltinNBD(NBDBlockdevAddBase): } } self.client_test(filename, flatten_sock_addr(address), 'nbd-export') + sockfd.close() self._server_down() --2.49.0
PATCH 13/14 iotests/151: ensure subprocesses are cleaned up
In reply to Daniel P. BerrangéThe iotest 151 creates a bunch of subprocesses, with their stdout connected to a pipe but never reads any data from them and does not gurantee the processes are killed on cleanup.
This triggers resource leak warnings from python when the subprocess.Popen object is garbage collected.
--- tests/qemu-iotests/151 | 5 +++++ 1 file changed, 5 insertions(+)
tests/qemu-iotests/151 +5 -0
@@ -263,6 +263,11 @@ class TestThrottledWithNbdExportBase(iotests.QMPTestCase): break except subprocess.TimeoutExpired: self.vm.qtest(f'clock_step {1 * 1000 * 1000 * 1000}') + try: + p.kill() + p.stdout.close() + except: + pass except IndexError: pass
Signature · 1 line
2.49.0
Re: [PATCH 13/14] iotests/151: ensure subprocesses are cleaned up
Quoted 18 lines · Daniel P. Berrangé
The iotest 151 creates a bunch of subprocesses, with their stdout connected to a pipe but never reads any data from them and does not gurantee the processes are killed on cleanup.
This triggers resource leak warnings from python when the subprocess.Popen object is garbage collected.
Signed-off-by Daniel P. Berrangé <[email protected]>--- tests/qemu-iotests/151 | 5 +++++ 1 file changed, 5 insertions(+)tests/qemu-iotests/151 +0
-0
tests/qemu-iotests/151 class TestThrottledWithNbdExportBase(iotests.QMPTestCase): · line 270
Quoted 8 lines
tests/qemu-iotests/151 +5
-0@@ -263,6 +263,11 @@ class TestThrottledWithNbdExportBase(iotests.QMPTestCase): break except subprocess.TimeoutExpired: self.vm.qtest(f'clock_step {1 * 1000 * 1000 * 1000}') + try: + p.kill() + p.stdout.close() + except: + pass
You can decide how much you care here, but using "except Exception:" will allow critical exceptions to still crash the program, like KeyboardInterrupt and SystemExit et al. It's good hygiene for things that may or may not hang so you can still use Ctrl+C.
Quoted 6 lines
tests/qemu-iotests/151 +0
-1except IndexError: pass --2.49.0
Re: [PATCH 13/14] iotests/151: ensure subprocesses are cleaned up
Quoted 24 lines · John Snow
Quoted 23 lines · Daniel P. Berrangé
The iotest 151 creates a bunch of subprocesses, with their stdout connected to a pipe but never reads any data from them and does not gurantee the processes are killed on cleanup.
This triggers resource leak warnings from python when the subprocess.Popen object is garbage collected.
Signed-off-by Daniel P. Berrangé <[email protected]>--- tests/qemu-iotests/151 | 5 +++++ 1 file changed, 5 insertions(+)tests/qemu-iotests/151 +2
-0@@ -263,6 +263,11 @@ class TestThrottledWithNbdExportBase(iotests.QMPTestCase): break except subprocess.TimeoutExpired: self.vm.qtest(f'clock_step {1 * 1000 * 1000 * 1000}') + try: + p.kill()
Quoted 8 lines
tests/qemu-iotests/151 class TestThrottledWithNbdExportBase(iotests.QMPTestCase): · line 270
Quoted 3 lines
tests/qemu-iotests/151 +3
-0+ p.stdout.close() + except: + passYou can decide how much you care here, but using "except Exception:" will allow critical exceptions to still crash the program, like KeyboardInterrupt and SystemExit et al. It's good hygiene for things that may or may not hang so you can still use Ctrl+C.
Yes, that should be changed as you describe.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 14/14 iotests/check: always enable all python warnings
In reply to Daniel P. BerrangéOf most importance is that this gives us a heads-up if anything we rely on has been deprecated. The default python behaviour only emits a warning if triggered from __main__ which is very limited.
Setting the env variable further ensures that any python child processes will also display warnings.
--- tests/qemu-iotests/check | 4 ++++ 1 file changed, 4 insertions(+)
tests/qemu-iotests/check +4 -0
@@ -21,6 +21,7 @@ import sys import argparse import shutil from pathlib import Path +import warnings from findtests import TestFinder from testenv import TestEnv
@@ -137,6 +138,9 @@ def make_argparser() -> argparse.ArgumentParser: if __name__ == '__main__': + warnings.simplefilter("default") + os.environ["PYTHONWARNINGS"] = "default" + args = make_argparser().parse_args() env = TestEnv(source_dir=args.source_dir,
Signature · 1 line
2.49.0
Re: [PATCH 14/14] iotests/check: always enable all python warnings
Quoted 10 lines · Daniel P. Berrangé
Of most importance is that this gives us a heads-up if anything we rely on has been deprecated. The default python behaviour only emits a warning if triggered from
__main__which is very limited.Setting the env variable further ensures that any python child processes will also display warnings.
Signed-off-by Daniel P. Berrangé <[email protected]>
Thanks for this one. I sent a PR for the first two patches, I will stage everything else.
Quoted 29 lines
--- tests/qemu-iotests/check | 4 ++++ 1 file changed, 4 insertions(+)tests/qemu-iotests/check +4
-1@@ -21,6 +21,7 @@ import sys import argparse import shutil from pathlib import Path +import warnings from findtests import TestFinder from testenv import TestEnv@@ -137,6 +138,9 @@ def make_argparser() -> argparse.ArgumentParser: if __name__ == '__main__': + warnings.simplefilter("default") + os.environ["PYTHONWARNINGS"] = "default" + args = make_argparser().parse_args() env = TestEnv(source_dir=args.source_dir, --2.49.0