Switch iotests to pyvenv
cover Switch iotests to pyvenv
Hi, this is ... a fairly incomplete series about trying to get iotests to run out of the configure-time venv. I'm looking for some feedback, so out to the list it goes.
Primarily, I'm having doubts about these points:
1) I think I need something like "mkvenv install" in the first patch, but mkvenv.py is getting pretty long...
2) Is there a way to optimize the speed for patch #2? Maybe installing this package can be skipped until it's needed, but that means that things like iotest's ./check might get complicated to support that.
3) I cheated quite a bit in patch 4 to use the correct Python to launch iotests, but I'm wondering if there's a nicer way to solve this more *completely*.
John Snow (6):
experiment: add mkvenv install
build, tests: Add qemu in-tree packages to pyvenv at configure time.
iotests: get rid of '..' in path environment output
iotests: use the correct python to run linters
iotests: use pyvenv/bin/python3 to launch child test processes
iotests: don't add qemu.git/python to PYTHONPATH
configure | 31 +++++++++++++++++++++++++++ python/scripts/mkvenv.py | 40 +++++++++++++++++++++++++++++++++++ tests/qemu-iotests/linters.py | 2 +- tests/qemu-iotests/testenv.py | 21 ++++++++++++------ 4 files changed, 87 insertions(+), 7 deletions(-)
Signature · 1 line
2.40.1
RFC PATCH 1/6 experiment: add mkvenv install
This is just so I can do "mkvenv install './python'" or "mkvenv install file:python" to install the in-tree packages to pyvenv.
It probably isn't quite appropriate to bypass do_ensure in its entirety like this because we miss out on a lot of error handling, but as a quick proof of concept it works just fine.
--- python/scripts/mkvenv.py | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)
python/scripts/mkvenv.py +40 -0
@@ -940,6 +940,35 @@ def _add_ensure_subcommand(subparsers: Any) -> None: ) +def _add_install_subcommand(subparsers: Any) -> None: + subparser = subparsers.add_parser( + "install", help="Install the specified package." + ) + subparser.add_argument( + "--online", + action="store_true", + help="Install packages from PyPI, if necessary.", + ) + subparser.add_argument( + "--dir", + type=str, + action="store", + help="Path to vendored packages where we may install from.", + ) + subparser.add_argument( + '--editable', + action="store_true", + help="Should package(s) be installed in editable mode?" + ) + subparser.add_argument( + "dep_specs", + type=str, + action="store", + help="PEP 508 Dependency specification, e.g. 'meson>=0.61.5'", + nargs="+", + ) + + def main() -> int: """CLI interface to make_qemu_venv. See module docstring.""" if os.environ.get("DEBUG") or os.environ.get("GITLAB_CI"):
@@ -964,6 +993,7 @@ def main() -> int: _add_create_subcommand(subparsers) _add_post_init_subcommand(subparsers) _add_ensure_subcommand(subparsers) + _add_install_subcommand(subparsers) args = parser.parse_args() try:
@@ -982,6 +1012,16 @@ def main() -> int: wheels_dir=args.dir, prog=args.diagnose, ) + if args.command == "install": + print(f"mkvenv: installing {', '.join(args.dep_specs)}", file=sys.stderr) + pip_args = list(args.dep_specs) + if args.editable: + pip_args.insert(0, "--editable") + pip_install( + args=pip_args, + online=args.online, + wheels_dir=args.dir + ) logger.debug("mkvenv.py %s: exiting", args.command) except Ouch as exc: print("\n*** Ouch! ***\n", file=sys.stderr)
Signature · 1 line
2.40.1
RFC PATCH 2/6 build, tests: Add qemu in-tree packages to pyvenv at configure time.
In reply to John Snowthough, ouch: on my machine this takes 3-4 entire seconds to do. I wish it wasn't so slow, but we can't rely on these packages not having any dependencies any more.
We could theoretically use a .pth hack when creating the venv to automatically include this directory as an "installed packages" location, but when we go to drop qemu.qmp in the future, that will break
- I think we need to *install* this package.
--- configure | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
configure +31 -0
@@ -250,6 +250,7 @@ git_submodules_action="update" git="git" debug_tcg="no" docs="auto" +tests="enabled" EXESUF="" prefix="/usr/local" qemu_suffix="qemu"
@@ -639,6 +640,10 @@ for opt do ;; --disable-docs) docs=disabled ;; + --enable-tests) tests=enabled + ;; + --disable-tests) tests=disabled + ;; --cpu=*) ;; --target-list=*) target_list="$optarg"
@@ -985,6 +990,32 @@ if test "$docs" != "disabled" ; then fi fi +# Optionally pre-load the testing pre-requisites. This is for iotests, +# vmtests, and anything else that uses Python qemu.* packages. Note that +# our in-tree qemu packages are currently pure python with zero external +# dependencies. For this reason, it excludes the Avocado dependencies +# which are installed on-demand at time of use instead. + +mkvenv_flags="" +if test "$pypi" = "enabled" ; then + mkvenv_flags="--online" +fi + +if test "$tests" = "enabled" ; then + if ! $mkvenv install \ + $mkvenv_flags \ + --editable \ + --dir "${source_path}/python/wheels" \ + "${source_path}/python/"; + then + echo "There was a problem installing the in-tree python packages for testing." + exit 1 + fi + touch pyvenv/tests.group +fi + +echo "mkvenv: done for now, ciao!" + # Probe for ninja if test -z "$ninja"; then
Signature · 1 line
2.40.1
RFC PATCH 3/6 iotests: get rid of '..' in path environment output
In reply to John SnowResolve the build_root before we append more items onto it so that the environment output is more concise with less parent directory confetti in it.
--- tests/qemu-iotests/testenv.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/testenv.py +1 -1
@@ -216,7 +216,7 @@ def __init__(self, source_dir: str, build_dir: str, self.source_iotests = source_dir self.build_iotests = build_dir - self.build_root = os.path.join(self.build_iotests, '..', '..') + self.build_root = Path(self.build_iotests).parent.parent self.init_directories()
Signature · 1 line
2.40.1
RFC PATCH 4/6 iotests: use the correct python to run linters
In reply to John SnowWhichever python is used to run iotest 297 should be the one used to actually run the linters.
--- tests/qemu-iotests/linters.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/linters.py +1 -1
@@ -68,7 +68,7 @@ def run_linter( :raise CalledProcessError: If the linter process exits with failure. """ subprocess.run( - ('python3', '-m', tool, *args), + (sys.executable, '-m', tool, *args), env=env, check=True, stdout=subprocess.PIPE if suppress_output else None,
Signature · 1 line
2.40.1
RFC PATCH 5/6 iotests: use pyvenv/bin/python3 to launch child test processes
In reply to John SnowNow that there's a fancy venv set up for us by configure, we should take care to use it even when check is invoked directly.
./check will now use the pyvenv environment when launching python tests, which allows those tests to find and access the 'qemu.*' packages without PYTHONPATH modifications.
RFC: This patch now means that ./check may launch test subprocesses using a different Python than the one used to launch it. If that isn't acceptable, we might need a launcher shim whose job it is to sit above "check" and just chooses the correct Python.
...Or maybe it's fine the way it is.
Comments welcome, sorry for my indecision.
--- tests/qemu-iotests/testenv.py | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
tests/qemu-iotests/testenv.py +14 -1
@@ -138,7 +138,20 @@ def init_binaries(self) -> None: PYTHON (for bash tests) QEMU_PROG, QEMU_IMG_PROG, QEMU_IO_PROG, QEMU_NBD_PROG, QSD_PROG """ - self.python = sys.executable + # The python we want to use to launch tests. + self.python: str = str( + Path(self.build_root).joinpath('pyvenv', 'bin', 'python3') + ) + # RFC: Do I need to amend '.exe' for windows, or nah? + + if self.python != sys.executable: + print( + "Note: " + f"check was launched with a Python ({sys.executable}) " + f"that doesn't match QEMU's configured Python ({self.python})." + " QEMU's Python will be used for individual test processes.", + file=sys.stderr + ) def root(*names: str) -> str: return os.path.join(self.build_root, *names)
Signature · 1 line
2.40.1
RFC PATCH 6/6 iotests: don't add qemu.git/python to PYTHONPATH
In reply to John Snowqemu.* should be provided by the configure-time venv, now.
--- tests/qemu-iotests/testenv.py | 4 ---- 1 file changed, 4 deletions(-)
tests/qemu-iotests/testenv.py +0 -4
@@ -108,12 +108,8 @@ def init_directories(self) -> None: SAMPLE_IMG_DIR """ - # Path where qemu goodies live in this source tree. - qemu_srctree_path = Path(__file__, '../../../python').resolve() - self.pythonpath = os.pathsep.join(filter(None, ( self.source_iotests, - str(qemu_srctree_path), os.getenv('PYTHONPATH'), )))
Signature · 1 line
2.40.1
Il mer 21 giu 2023, 02:21 John Snow <[email protected]> ha scritto:
Quoted 9 lines
Hi, this is ... a fairly incomplete series about trying to get iotests to run out of the configure-time venv. I'm looking for some feedback, so out to the list it goes.
Primarily, I'm having doubts about these points:
1) I think I need something like "mkvenv install" in the first patch, but
mkvenv.pyis getting pretty long...
It's not a lot of code, but using pip install from configure might also be good enough, I don't know.
2) Is there a way to optimize the speed for patch #2? Maybe installing
this package can be skipped until it's needed, but that means that
Quoted 6 lines
things like iotest's ./check might get complicated to support that.
3) I cheated quite a bit in patch 4 to use the correct Python to launch iotests, but I'm wondering if there's a nicer way to solve this more *completely*.
Maybe patch 4 can use distlib.scripts as well to create the check script in the build directory? (Yes that's another mkvenv functionality...) On a phone and don't have the docs at hand, so I am not sure. If not, your solution is good enough.
Apart from this the only issue is the speed. IIRC having a prebuilt .whl would fix it, I think for Meson we observed that the slow part was building the wheel. Possibilities:
1) using --no-pep517 if that also speeds it up?
2) already removing the sources to qemu.qmp since that's the plan anyway; and then, if you want editability you can install the package with --user --editable, i.e. outside the venv
Paolo
Quoted 19 lines
John Snow (6):
experiment: add mkvenv install
build, tests: Add qemu in-tree packages to pyvenv at configure time.
iotests: get rid of '..' in path environment output
iotests: use the correct python to run linters
iotests: use pyvenv/bin/python3 to launch child test processes
iotests: don't add qemu.git/python to PYTHONPATHconfigure | 31 +++++++++++++++++++++++++++ python/scripts/mkvenv.py | 40 +++++++++++++++++++++++++++++++++++ tests/qemu-iotests/linters.py | 2 +- tests/qemu-iotests/testenv.py | 21 ++++++++++++------ 4 files changed, 87 insertions(+), 7 deletions(-)--
2.40.1
Quoted 7 lines · Paolo Bonzini
Maybe patch 4 can use distlib.scripts as well to create the check script in the build directory? (Yes that's another mkvenv functionality...) On a phone and don't have the docs at hand, so I am not sure. If not, your solution is good enough.
Apart from this the only issue is the speed. IIRC having a prebuilt .whl would fix it, I think for Meson we observed that the slow part was building the wheel. Possibilities:
1) using --no-pep517 if that also speeds it up?
2) already removing the sources to qemu.qmp since that's the plan anyway; and then, if you want editability you can install the package with --user --editable, i.e. outside the venv
Nope, it's 3 second always and 1.5 even with the wheel.
Maybe replace qemu.qmp with a wheel and leaving PYTHONPATH for the rest?
Paolo
Quoted 21 lines
Paolo
Quoted 19 lines
John Snow (6):
experiment: add mkvenv install
build, tests: Add qemu in-tree packages to pyvenv at configure time.
iotests: get rid of '..' in path environment output
iotests: use the correct python to run linters
iotests: use pyvenv/bin/python3 to launch child test processes
iotests: don't add qemu.git/python to PYTHONPATHconfigure | 31 +++++++++++++++++++++++++++ python/scripts/mkvenv.py | 40 +++++++++++++++++++++++++++++++++++ tests/qemu-iotests/linters.py | 2 +- tests/qemu-iotests/testenv.py | 21 ++++++++++++------ 4 files changed, 87 insertions(+), 7 deletions(-)--
2.40.1
Quoted 4 lines · Paolo Bonzini
Quoted 2 lines · Paolo Bonzini
Maybe patch 4 can use distlib.scripts as well to create the check script in the build directory? (Yes that's another mkvenv functionality...) On a phone and don't have the docs at hand, so I am not sure. If not, your solution is good enough.
Yeah, that's a possibility... we could "install" the iotests script. That might keep things simple. I'll investigate it.
Quoted 4 lines
Quoted 4 lines
Apart from this the only issue is the speed. IIRC having a prebuilt .whl would fix it, I think for Meson we observed that the slow part was building the wheel. Possibilities:
1) using --no-pep517 if that also speeds it up?
Quoted 8 lines
Quoted 1 line
2) already removing the sources to qemu.qmp since that's the plan anyway; and then, if you want editability you can install the package with --user --editable, i.e. outside the venv
Nope, it's 3 second always and 1.5 even with the wheel.
Maybe replace qemu.qmp with a wheel and leaving PYTHONPATH for the rest?
Paolo
Hm, I guess so. It's just disappointing because I was really hoping to be able to use "pip install" to handle dependencies like a normal package instead of trying to shoulder that burden with an increasing amount of custom logic that's hard for anyone but me (or you, now) to maintain.
It kind of defeats the point of having formatted it as a package to begin with.
Maybe there's a sane way to amortize the cost of installation by not re-creating it after every call to configure instead -- the rest of the script is fast enough, perhaps we could default clear to *False* from now on and use the _get_version() bits to detect if the local internal package is already installed or not -- and if it is, just leave it alone.
If we always install it in editable mode, and the path where it is "installed" is what we expect it to be, it shouldn't have any problems with being out of date.... I think. We could conceivably use the "faux" package version the internal package has to signal when the script needs to re-install it.
Something like that?
--js
Quoted 5 lines · John Snow
If we always install it in editable mode, and the path where it is "installed" is what we expect it to be, it shouldn't have any problems with being out of date.... I think. We could conceivably use the "faux" package version the internal package has to signal when the script needs to re-install it.
Stupid question, why not treat it just like avocado?
Quoted 10 lines · Paolo Bonzini
Quoted 5 lines · John Snow
If we always install it in editable mode, and the path where it is "installed" is what we expect it to be, it shouldn't have any problems with being out of date.... I think. We could conceivably use the "faux" package version the internal package has to signal when the script needs to re-install it.
Stupid question, why not treat it just like avocado?
How do you mean? (i.e. installing it on-demand in reaction to "make check-avocado"?)
Quoted 10 lines · John Snow
Quoted 8 lines · Paolo Bonzini
Quoted 5 lines · John Snow
If we always install it in editable mode, and the path where it is "installed" is what we expect it to be, it shouldn't have any problems with being out of date.... I think. We could conceivably use the "faux" package version the internal package has to signal when the script needs to re-install it.
Quoted 5 lines
Quoted 2 lines
Stupid question, why not treat it just like avocado?
How do you mean? (i.e. installing it on-demand in reaction to "make check-avocado"?)
Yes, installing it on-demand the first time "make check-iotests" is run, using a "depend:" keyword argument in tests/qemu-iotests/meson.build.
BTW,
from distlib.scripts import ScriptMakerScriptMaker('..', '.').make('foo.py')
Seems to do the right thing as long as foo.py includes a shebang (I tested it inside a virtual environment).
Paolo
Quoted 24 lines · Paolo Bonzini
Quoted 10 lines · John Snow
Quoted 8 lines · Paolo Bonzini
Quoted 5 lines · John Snow
If we always install it in editable mode, and the path where it is "installed" is what we expect it to be, it shouldn't have any problems with being out of date.... I think. We could conceivably use the "faux" package version the internal package has to signal when the script needs to re-install it.
Quoted 5 lines
Quoted 2 lines
Stupid question, why not treat it just like avocado?
How do you mean? (i.e. installing it on-demand in reaction to "make check-avocado"?)
Yes, installing it on-demand the first time "make check-iotests" is run, using a "depend:" keyword argument in tests/qemu-iotests/meson.build.
BTW,
Quoted 7 lines
from distlib.scripts import ScriptMaker
ScriptMaker('..', '.').make('foo.py')Seems to do the right thing as long as
foo.pyincludes a shebang (I tested it inside a virtual environment).Paolo
That's possible, but it means that it will break if you run configure and then immediately go to invoke iotests, unless we have a way to have iotests bootstrap itself. Which I think can't be done through the makefile, because we don't know which "make" to run in order to get that to happen. (Or at least, I don't!)
Possibly I could teach mkvenv a new trick, like "mkvenv init iotests" and have the mkvenv script DTRT at that point, whatever that is -- ideally exiting very quickly without doing anything.
Il gio 22 giu 2023, 23:18 John Snow <[email protected]> ha scritto:
Quoted 4 lines
Possibly I could teach mkvenv a new trick, like "mkvenv init iotests" and have the mkvenv script DTRT at that point, whatever that is -- ideally exiting very quickly without doing anything.
Or maybe check itself should do the bootstrap if it's invoked from the venv?!?
Paolo