configure: create a python venv and ensure meson, sphinx
v3 cover configure: create a python venv and ensure meson, sphinx
GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
(All green, except Python self-tests, see below)
This patch series creates a mandatory python virtual environment ("venv") during configure time and uses it to ensure the availability of meson and sphinx.
See https://www.qemu.org/2023/03/24/python/ for details. The summary is that the goal of this series is to ensure that the python used to run meson is the same python used to run Sphinx, tests, and any build-time python scripting we have. As it stands, meson and sphinx (and their extensions) *may* run in a different python environment than the one configured and chosen by the user at configure/build time.
The effective change of this series is that QEMU will now unconditionally create a venv at configure-time and will ensure that meson (and sphinx, if docs are enabled) are available through that venv.
Some important points as a pre-emptive "FAQ":
- This venv is unconditionally created and lives at {build_dir}/pyvenv.
- The python interpreter used by this venv is always the one identified by configure. (Which in turn is always the one specified by --python or $PYTHON)
- *almost* all python scripts in qemu.git executed as part of the build system, meson, sphinx, avocado tests, vm tests or CI are always executed within this venv.
(iotests are not yet integrated; I plan to tackle this separately as a follow-up in order to have a more tightly focused scope on that series.)
- It remains possible to build and test fully offline.
(In most cases, you just need meson and sphinx from your distro's repo.)
- Distribution packaged 'meson' and 'sphinx' are still utilized whenever possible as the highest preference.
- Vendored versions of e.g. 'meson' are always preferred to PyPI versions for speed, repeatability and ensuring tarball builds work as-is offline.
(Sphinx will not be vendored, just like it already isn't.)
- Missing dependencies, when possible, are fetched and installed on-demand automatically to make developer environments "just work".
- Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
- No new dependencies (...for most platforms. Debian and NetBSD get an asterisk.)
- The meson git submodule is unused after this series and can be removed.
For reviewers, here's how the series is broken up:
Patch 1 is a testing pre-req. Note that even with this patch, 'check-python-minreqs' and 'check-python-tox' CI jobs will both still fail on origin/master because this series requires 3.7+, but origin/master is currently still 3.6+.
- python: update pylint configuration
Patches 2-8 add the mkvenv script. The first patch checks in the barest essentials, and each subsequent patch adds a workaround or feature one at a time.
- python: add mkvenv.py
- mkvenv: add console script entry point generation
- mkvenv: Add better error message for missing pyexapt module
- mkvenv: generate console entry shims from inside the venv
- mkvenv: work around broken pip installations on Debian 10
- mkvenv: add nested venv workaround
- mkvenv: add ensure subcommand
Patches 9-11 modify our testing configuration to add new dependencies as needed.
- tests/docker: add python3-venv dependency
- tests/vm: Configure netbsd to use Python 3.10
- tests/vm: add py310-expat to NetBSD
Patch 12 changes how we package release tarballs.
- scripts/make-release: download meson==0.61.5 .whl
Patches 13-16 wire mkvenv into configure and tests.
- configure: create a python venv unconditionally
- configure: use 'mkvenv ensure meson' to bootstrap meson
- configure: add --enable-pypi and --disable-pypi
- tests: Use configure-provided pyvenv for tests
Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these changes could be folded earlier in the series (like the diagnose() patch), but I'm keeping it separate for review for now.
- configure: move --enable-docs and --disable-docs back to configure
- mkvenv: add diagnose() method for ensure() failures
- configure: use --diagnose option with meson ensure
- configure: bootstrap sphinx with mkvenv
That's all for now, seeya!
--js
John Snow (20):
python: update pylint configuration
python: add mkvenv.py
mkvenv: add console script entry point generation
mkvenv: Add better error message for missing pyexpat module
mkvenv: generate console entry shims from inside the venv
mkvenv: work around broken pip installations on Debian 10
mkvenv: add nested venv workaround
mkvenv: add ensure subcommand
tests/docker: add python3-venv dependency
tests/vm: Configure netbsd to use Python 3.10
tests/vm: add py310-expat to NetBSD scripts/make-release: download meson==0.61.5 .whl
configure: create a python venv unconditionally
configure: use 'mkvenv ensure meson' to bootstrap meson
configure: add --enable-pypi and --disable-pypi
tests: Use configure-provided pyvenv for tests
configure: move --enable-docs and --disable-docs back to configure
mkvenv: add diagnose() method for ensure() failures
configure: use --diagnose option with meson ensure
configure: bootstrap sphinx with mkvenv
docs/devel/acpi-bits.rst | 6 +- docs/devel/testing.rst | 14 +- configure | 139 +-- .gitlab-ci.d/buildtest-template.yml | 4 +- .gitlab-ci.d/buildtest.yml | 6 +- python/scripts/mkvenv.py | 871 ++++++++++++++++++ python/setup.cfg | 10 + python/tests/flake8.sh | 1 + python/tests/isort.sh | 1 + python/tests/mypy.sh | 1 + python/tests/pylint.sh | 1 + .../org.centos/stream/8/x86_64/test-avocado | 4 +- scripts/device-crash-test | 2 +- scripts/make-release | 11 + tests/Makefile.include | 10 +- .../dockerfiles/debian-all-test-cross.docker | 3 +- .../dockerfiles/debian-hexagon-cross.docker | 3 +- .../dockerfiles/debian-riscv64-cross.docker | 3 +- .../dockerfiles/debian-tricore-cross.docker | 3 +- tests/requirements.txt | 7 +- tests/vm/netbsd | 2 + 21 files changed, 1016 insertions(+), 86 deletions(-) create mode 100644 python/scripts/mkvenv.py
Signature · 1 line
2.39.2
RFC PATCH v3 1/20 python: update pylint configuration
Pylint 2.17.x decided that SocketAddrT was a bad name for a Type Alias for some reason. Sure, fine, whatever.
--- python/setup.cfg | 1 + 1 file changed, 1 insertion(+)
python/setup.cfg +1 -0
@@ -132,6 +132,7 @@ good-names=i, fd, # fd = os.open(...) c, # for c in string: ... T, # for TypeVars. See pylint#3401 + SocketAddrT, # Not sure why this is invalid. [pylint.similarities] # Ignore imports when computing similarities.
Signature · 1 line
2.39.2
Re: [RFC PATCH v3 01/20] python: update pylint configuration
Quoted 7 lines · John Snow
Pylint 2.17.x decided that SocketAddrT was a bad name for a Type Alias for some reason. Sure, fine, whatever.
Signed-off-by John Snow <[email protected]>--- python/setup.cfg | 1 + 1 file changed, 1 insertion(+)
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
RFC PATCH v3 2/20 python: add mkvenv.py
In reply to John SnowThis script will be responsible for building a lightweight Python virtual environment at configure time. It works with Python 3.7 or newer.
It has been designed to:
- work *offline*, no PyPI required.
- work *quickly*, The fast path is only ~65ms on my machine.
- work *robustly*, with multiple fallbacks to keep things working.
- work *cooperatively*, using system packages where possible. (You can use your distro's meson, no problem.)
Due to its unique position in the build chain, it exists outside of the installable python packages in-tree and *must* be runnable without any third party dependencies.
Under normal circumstances, the only dependency required to execute this script is Python 3.7+ itself. The script is *faster* by several seconds when setuptools and pip are installed in the host environment, which is probably the case for a typical multi-purpose developer workstation.
In the event that pip/setuptools are missing or not usable, additional dependencies may be required on some distributions which remove certain Python stdlib modules to package them separately:
- Debian may require python3-venv to provide "ensurepip"
- NetBSD may require py310-expat to provide "pyexpat" * (* Or whichever version is current for NetBSD.)
--- python/scripts/mkvenv.py | 277 +++++++++++++++++++++++++++++++++++++++ python/setup.cfg | 9 ++ python/tests/flake8.sh | 1 + python/tests/isort.sh | 1 + python/tests/mypy.sh | 1 + python/tests/pylint.sh | 1 + 6 files changed, 290 insertions(+) create mode 100644 python/scripts/mkvenv.py
python/scripts/mkvenv.py new file +277 -0
@@ -0,0 +1,277 @@ +""" +mkvenv - QEMU pyvenv bootstrapping utility + +usage: mkvenv [-h] command ... + +QEMU pyvenv bootstrapping utility + +options: + -h, --help show this help message and exit + +Commands: + command Description + create create a venv + +-------------------------------------------------- + +usage: mkvenv create [-h] target + +positional arguments: + target Target directory to install virtual environment into. + +options: + -h, --help show this help message and exit + +""" + +# Copyright (C) 2022-2023 Red Hat, Inc. +# +# Authors: +# John Snow <[email protected]> +# Paolo Bonzini <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +import argparse +from importlib.util import find_spec +import logging +import os +from pathlib import Path +import subprocess +import sys +import traceback +from types import SimpleNamespace +from typing import Any, Optional, Union +import venv + + +# Do not add any mandatory dependencies from outside the stdlib: +# This script *must* be usable standalone! + +logger = logging.getLogger("mkvenv") + + +class Ouch(RuntimeError): + """An Exception class we can't confuse with a builtin.""" + + +class QemuEnvBuilder(venv.EnvBuilder): + """ + An extension of venv.EnvBuilder for building QEMU's configure-time venv. + + As of this commit, it does not yet do anything particularly + different than the standard venv-creation utility. The next several + commits will gradually change that in small commits that highlight + each feature individually. + + Parameters for base class init: + - system_site_packages: bool = False + - clear: bool = False + - symlinks: bool = False + - upgrade: bool = False + - with_pip: bool = False + - prompt: Optional[str] = None + - upgrade_deps: bool = False (Since 3.9) + """ + + def __init__(self, *args: Any, **kwargs: Any) -> None: + logger.debug("QemuEnvBuilder.__init__(...)") + super().__init__(*args, **kwargs) + + # The EnvBuilder class is cute and toggles this setting off + # before post_setup, but we need it to decide if we want to + # generate shims or not. + self._system_site_packages = self.system_site_packages + + def post_setup(self, context: SimpleNamespace) -> None: + logger.debug("post_setup(...)") + + # print the python executable to stdout for configure. + print(context.env_exe) + + +def need_ensurepip() -> bool: + """ + Tests for the presence of setuptools and pip. + + :return: `True` if we do not detect both packages. + """ + # Don't try to actually import them, it's fraught with danger: + # https://github.com/pypa/setuptools/issues/2993 + if find_spec("setuptools") and find_spec("pip"): + return False + return True + + +def check_ensurepip(with_pip: bool) -> None: + """ + Check that we have ensurepip. + + Raise a fatal exception with a helpful hint if it isn't available. + """ + if with_pip and not find_spec("ensurepip"): + msg = ( + "Python's ensurepip module is not found.\n" + "It's normally part of the Python standard library, " + "maybe your distribution packages it separately?\n" + "Either install ensurepip, or alleviate the need for it in the " + "first place by installing pip and setuptools for " + f"'{sys.executable}'.\n" + "(Hint: Debian puts ensurepip in its python3-venv package.)" + ) + raise Ouch(msg) + + +def make_venv( # pylint: disable=too-many-arguments + env_dir: Union[str, Path], + system_site_packages: bool = False, + clear: bool = True, + symlinks: Optional[bool] = None, + with_pip: Optional[bool] = None, +) -> None: + """ + Create a venv using `QemuEnvBuilder`. + + This is analogous to the `venv.create` module-level convenience + function that is part of the Python stdblib, except it uses + `QemuEnvBuilder` instead. + + :param env_dir: The directory to create/install to. + :param system_site_packages: + Allow inheriting packages from the system installation. + :param clear: When True, fully remove any prior venv and files. + :param symlinks: + Whether to use symlinks to the target interpreter or not. If + left unspecified, it will use symlinks except on Windows to + match behavior with the "venv" CLI tool. + :param with_pip: + Whether to run "ensurepip" or not. If unspecified, this will + default to False if system_site_packages is True and a usable + version of pip is found. + """ + logging.debug( + "%s: make_venv(env_dir=%s, system_site_packages=%s, " + "clear=%s, symlinks=%s, with_pip=%s)", + __file__, + str(env_dir), + system_site_packages, + clear, + symlinks, + with_pip, + ) + + print(f"MKVENV {str(env_dir)}", file=sys.stderr) + + # ensurepip is slow: venv creation can be very fast for cases where + # we allow the use of system_site_packages. Toggle ensure_pip on only + # in the cases where we really need it. + if with_pip is None: + with_pip = True if not system_site_packages else need_ensurepip() + logger.debug("with_pip unset, choosing %s", with_pip) + + check_ensurepip(with_pip) + + if symlinks is None: + # Default behavior of standard venv CLI + symlinks = os.name != "nt" + + builder = QemuEnvBuilder( + system_site_packages=system_site_packages, + clear=clear, + symlinks=symlinks, + with_pip=with_pip, + ) + try: + logger.debug("Invoking builder.create()") + try: + builder.create(str(env_dir)) + except SystemExit as exc: + # Some versions of the venv module raise SystemExit; *nasty*! + # We want the exception that prompted it. It might be a subprocess + # error that has output we *really* want to see. + logger.debug("Intercepted SystemExit from EnvBuilder.create()") + raise exc.__cause__ or exc.__context__ or exc + logger.debug("builder.create() finished") + except subprocess.CalledProcessError as exc: + print(f"cmd: {exc.cmd}", file=sys.stderr) + print(f"returncode: {exc.returncode}", file=sys.stderr) + + def _stringify(data: Optional[Union[str, bytes]]) -> Optional[str]: + if isinstance(data, bytes): + return data.decode() + return data + + if exc.stdout: + print( + "========== stdout ==========", + _stringify(exc.stdout), + "============================", + sep="\n", + file=sys.stderr, + ) + if exc.stderr: + print( + "========== stderr ==========", + _stringify(exc.stderr), + "============================", + sep="\n", + file=sys.stderr, + ) + raise Ouch("VENV creation subprocess failed.") from exc + + +def _add_create_subcommand(subparsers: Any) -> None: + subparser = subparsers.add_parser("create", help="create a venv") + subparser.add_argument( + "target", + type=str, + action="store", + help="Target directory to install virtual environment into.", + ) + + +def main() -> int: + """CLI interface to make_qemu_venv. See module docstring.""" + if os.environ.get("DEBUG") or os.environ.get("GITLAB_CI"): + # You're welcome. + logging.basicConfig(level=logging.DEBUG) + elif os.environ.get("V"): + logging.basicConfig(level=logging.INFO) + + parser = argparse.ArgumentParser( + prog="mkvenv", + description="QEMU pyvenv bootstrapping utility", + ) + subparsers = parser.add_subparsers( + title="Commands", + dest="command", + required=True, + metavar="command", + help="Description", + ) + + _add_create_subcommand(subparsers) + + args = parser.parse_args() + try: + if args.command == "create": + make_venv( + args.target, + system_site_packages=True, + clear=True, + ) + logger.debug("mkvenv.py %s: exiting", args.command) + except Ouch as exc: + print("\n*** Ouch! ***\n", file=sys.stderr) + print(str(exc), "\n\n", file=sys.stderr) + return 1 + except: # pylint: disable=bare-except + print("mkvenv did not complete successfully:", file=sys.stderr) + traceback.print_exc() + return 2 + return 0 + + +if __name__ == "__main__": + sys.exit(main())
python/setup.cfg +9 -0
@@ -103,6 +103,15 @@ ignore_missing_imports = True [mypy-pygments] ignore_missing_imports = True +[mypy-importlib.metadata] +ignore_missing_imports = True + +[mypy-importlib_metadata] +ignore_missing_imports = True + +[mypy-pkg_resources] +ignore_missing_imports = True + [pylint.messages control] # Disable the message, report, category or checker with the given id(s). You # can either give multiple identifiers separated by comma (,) or put this
python/tests/flake8.sh +1 -0
@@ -1,2 +1,3 @@ #!/bin/sh -e python3 -m flake8 qemu/ +python3 -m flake8 scripts/
python/tests/isort.sh +1 -0
@@ -1,2 +1,3 @@ #!/bin/sh -e python3 -m isort -c qemu/ +python3 -m isort -c scripts/
python/tests/mypy.sh +1 -0
@@ -1,2 +1,3 @@ #!/bin/sh -e python3 -m mypy -p qemu +python3 -m mypy scripts/
python/tests/pylint.sh +1 -0
@@ -1,3 +1,4 @@ #!/bin/sh -e # See commit message for environment variable explainer. SETUPTOOLS_USE_DISTUTILS=stdlib python3 -m pylint qemu/ +SETUPTOOLS_USE_DISTUTILS=stdlib python3 -m pylint scripts/
Signature · 1 line
2.39.2
RFC PATCH v3 3/20 mkvenv: add console script entry point generation
In reply to John SnowWhen creating a virtual environment that inherits system packages, script entry points (like "meson", "sphinx-build", etc) are not re-generated with the correct shebang. When you are *inside* of the venv, this is not a problem, but if you are *outside* of it, you will not have a script that engages the virtual environment appropriately.
Add a mechanism that generates new entry points for pre-existing packages so that we can use these scripts to run "meson", "sphinx-build", "pip", unambiguously inside the venv.
--- python/scripts/mkvenv.py | 179 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 172 insertions(+), 7 deletions(-)
python/scripts/mkvenv.py +172 -7
@@ -14,13 +14,14 @@ -------------------------------------------------- -usage: mkvenv create [-h] target +usage: mkvenv create [-h] [--gen GEN] target positional arguments: target Target directory to install virtual environment into. options: -h, --help show this help message and exit + --gen GEN Regenerate console_scripts for given packages, if found. """
@@ -38,11 +39,20 @@ import logging import os from pathlib import Path +import re +import stat import subprocess import sys import traceback from types import SimpleNamespace -from typing import Any, Optional, Union +from typing import ( + Any, + Dict, + Iterator, + Optional, + Sequence, + Union, +) import venv
@@ -60,10 +70,9 @@ class QemuEnvBuilder(venv.EnvBuilder): """ An extension of venv.EnvBuilder for building QEMU's configure-time venv. - As of this commit, it does not yet do anything particularly - different than the standard venv-creation utility. The next several - commits will gradually change that in small commits that highlight - each feature individually. + The only functional change is that it adds the ability to regenerate + console_script shims for packages available via system_site + packages. Parameters for base class init: - system_site_packages: bool = False
@@ -77,6 +86,7 @@ class QemuEnvBuilder(venv.EnvBuilder): def __init__(self, *args: Any, **kwargs: Any) -> None: logger.debug("QemuEnvBuilder.__init__(...)") + self.script_packages = kwargs.pop("script_packages", ()) super().__init__(*args, **kwargs) # The EnvBuilder class is cute and toggles this setting off
@@ -87,6 +97,12 @@ def __init__(self, *args: Any, **kwargs: Any) -> None: def post_setup(self, context: SimpleNamespace) -> None: logger.debug("post_setup(...)") + # Generate console_script entry points for system packages: + if self._system_site_packages: + generate_console_scripts( + context.env_exe, context.bin_path, self.script_packages + ) + # print the python executable to stdout for configure. print(context.env_exe)
@@ -129,6 +145,7 @@ def make_venv( # pylint: disable=too-many-arguments clear: bool = True, symlinks: Optional[bool] = None, with_pip: Optional[bool] = None, + script_packages: Sequence[str] = (), ) -> None: """ Create a venv using `QemuEnvBuilder`.
@@ -149,16 +166,20 @@ def make_venv( # pylint: disable=too-many-arguments Whether to run "ensurepip" or not. If unspecified, this will default to False if system_site_packages is True and a usable version of pip is found. + :param script_packages: + A sequence of package names to generate console entry point + shims for, when system_site_packages is True. """ logging.debug( "%s: make_venv(env_dir=%s, system_site_packages=%s, " - "clear=%s, symlinks=%s, with_pip=%s)", + "clear=%s, symlinks=%s, with_pip=%s, script_packages=%s)", __file__, str(env_dir), system_site_packages, clear, symlinks, with_pip, + script_packages, ) print(f"MKVENV {str(env_dir)}", file=sys.stderr)
@@ -181,6 +202,7 @@ def make_venv( # pylint: disable=too-many-arguments clear=clear, symlinks=symlinks, with_pip=with_pip, + script_packages=script_packages, ) try: logger.debug("Invoking builder.create()")
@@ -221,8 +243,147 @@ def _stringify(data: Optional[Union[str, bytes]]) -> Optional[str]: raise Ouch("VENV creation subprocess failed.") from exc +def _gen_importlib(packages: Sequence[str]) -> Iterator[Dict[str, str]]: + # pylint: disable=import-outside-toplevel + try: + # First preference: Python 3.8+ stdlib + from importlib.metadata import ( + PackageNotFoundError, + distribution, + ) + except ImportError as exc: + logger.debug("%s", str(exc)) + # Second preference: Commonly available PyPI backport + from importlib_metadata import ( + PackageNotFoundError, + distribution, + ) + + # Borrowed from CPython (Lib/importlib/metadata/__init__.py) + pattern = re.compile( + r"(?P<module>[\w.]+)\s*" + r"(:\s*(?P<attr>[\w.]+)\s*)?" + r"((?P<extras>\[.*\])\s*)?$" + ) + + def _generator() -> Iterator[Dict[str, str]]: + for package in packages: + try: + entry_points = distribution(package).entry_points + except PackageNotFoundError: + continue + + # The EntryPoints type is only available in 3.10+, + # treat this as a vanilla list and filter it ourselves. + entry_points = filter( + lambda ep: ep.group == "console_scripts", entry_points + ) + + for entry_point in entry_points: + # Python 3.8 doesn't have 'module' or 'attr' attributes + if not ( + hasattr(entry_point, "module") + and hasattr(entry_point, "attr") + ): + match = pattern.match(entry_point.value) + assert match is not None + module = match.group("module") + attr = match.group("attr") + else: + module = entry_point.module + attr = entry_point.attr + yield { + "name": entry_point.name, + "module": module, + "import_name": attr, + "func": attr, + } + + return _generator() + + +def _gen_pkg_resources(packages: Sequence[str]) -> Iterator[Dict[str, str]]: + # pylint: disable=import-outside-toplevel + # Bundled with setuptools; has a good chance of being available. + import pkg_resources + + def _generator() -> Iterator[Dict[str, str]]: + for package in packages: + try: + eps = pkg_resources.get_entry_map(package, "console_scripts") + except pkg_resources.DistributionNotFound: + continue + + for entry_point in eps.values(): + yield { + "name": entry_point.name, + "module": entry_point.module_name, + "import_name": ".".join(entry_point.attrs), + "func": ".".join(entry_point.attrs), + } + + return _generator() + + +# Borrowed/adapted from pip's vendored version of distutils: +SCRIPT_TEMPLATE = r"""#!{python_path:s} +# -*- coding: utf-8 -*- +import re +import sys +from {module:s} import {import_name:s} +if __name__ == '__main__': + sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) + sys.exit({func:s}()) +""" + + +def generate_console_scripts( + python_path: str, bin_path: str, packages: Sequence[str] +) -> None: + """ + Generate script shims for console_script entry points in @packages. + """ + if not packages: + return + + def _get_entry_points() -> Iterator[Dict[str, str]]: + """Python 3.7 compatibility shim for iterating entry points.""" + # Python 3.8+, or Python 3.7 with importlib_metadata installed. + try: + return _gen_importlib(packages) + except ImportError as exc: + logger.debug("%s", str(exc)) + + # Python 3.7 with setuptools installed. + try: + return _gen_pkg_resources(packages) + except ImportError as exc: + logger.debug("%s", str(exc)) + raise Ouch( + "Neither importlib.metadata nor pkg_resources found, " + "can't generate console script shims.\n" + "Use Python 3.8+, or install importlib-metadata or setuptools." + ) from exc + + for entry_point in _get_entry_points(): + script_path = os.path.join(bin_path, entry_point["name"]) + script = SCRIPT_TEMPLATE.format(python_path=python_path, **entry_point) + with open(script_path, "w", encoding="UTF-8") as file: + file.write(script) + mode = os.stat(script_path).st_mode | stat.S_IEXEC + os.chmod(script_path, mode) + + logger.debug("wrote '%s'", script_path) + + def _add_create_subcommand(subparsers: Any) -> None: subparser = subparsers.add_parser("create", help="create a venv") + subparser.add_argument( + "--gen", + type=str, + action="append", + help="Regenerate console_scripts for given packages, if found.", + ) subparser.add_argument( "target", type=str,
@@ -256,10 +417,14 @@ def main() -> int: args = parser.parse_args() try: if args.command == "create": + script_packages = [] + for element in args.gen or (): + script_packages.extend(element.split(",")) make_venv( args.target, system_site_packages=True, clear=True, + script_packages=script_packages, ) logger.debug("mkvenv.py %s: exiting", args.command) except Ouch as exc:
Signature · 1 line
2.39.2
RFC PATCH v3 4/20 mkvenv: Add better error message for missing pyexpat module
In reply to John SnowNetBSD debundles pyexpat from python, but ensurepip needs pyexpat. Try our best to offer a helpful error message instead of just failing catastrophically.
--- python/scripts/mkvenv.py | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)
python/scripts/mkvenv.py +17 -1
@@ -126,7 +126,10 @@ def check_ensurepip(with_pip: bool) -> None: Raise a fatal exception with a helpful hint if it isn't available. """ - if with_pip and not find_spec("ensurepip"): + if not with_pip: + return + + if not find_spec("ensurepip"): msg = ( "Python's ensurepip module is not found.\n" "It's normally part of the Python standard library, "
@@ -138,6 +141,19 @@ def check_ensurepip(with_pip: bool) -> None: ) raise Ouch(msg) + # ensurepip uses pyexpat, which can also go missing on us: + if not find_spec("pyexpat"): + msg = ( + "Python's pyexpat module is not found.\n" + "It's normally part of the Python standard library, " + "maybe your distribution packages it separately?\n" + "Either install pyexpat, or alleviate the need for it in the " + "first place by installing pip and setuptools for " + f"'{sys.executable}'.\n\n" + "(Hint: NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)" + ) + raise Ouch(msg) + def make_venv( # pylint: disable=too-many-arguments env_dir: Union[str, Path],
Signature · 1 line
2.39.2
RFC PATCH v3 5/20 mkvenv: generate console entry shims from inside the venv
In reply to John SnowThis patch is meant to ensure that console entry point scripts will always generate on Python 3.7 installations where we may not have access to importlib.metadata. By running it from a separate process *inside* the venv, we can be assured to have access to setuptools and by extension pkg_resources as a fallback.
It isn't strictly necessary to do this for Python 3.8+, which will always have importlib.metadata available.
--- python/scripts/mkvenv.py | 99 ++++++++++++++++++++++++++++++++++------ 1 file changed, 85 insertions(+), 14 deletions(-)
python/scripts/mkvenv.py +85 -14
@@ -11,6 +11,8 @@ Commands: command Description create create a venv + post_init + post-venv initialization --------------------------------------------------
@@ -23,6 +25,15 @@ -h, --help show this help message and exit --gen GEN Regenerate console_scripts for given packages, if found. +-------------------------------------------------- + +usage: mkvenv post_init [-h] [--gen GEN] [--binpath BINPATH] + +options: + -h, --help show this help message and exit + --gen GEN Regenerate console_scripts for given packages, if found. + --binpath BINPATH Path where console script shims should be generated + """ # Copyright (C) 2022-2023 Red Hat, Inc.
@@ -59,6 +70,7 @@ # Do not add any mandatory dependencies from outside the stdlib: # This script *must* be usable standalone! +DirType = Union[str, bytes, "os.PathLike[str]", "os.PathLike[bytes]"] logger = logging.getLogger("mkvenv")
@@ -89,23 +101,42 @@ def __init__(self, *args: Any, **kwargs: Any) -> None: self.script_packages = kwargs.pop("script_packages", ()) super().__init__(*args, **kwargs) - # The EnvBuilder class is cute and toggles this setting off - # before post_setup, but we need it to decide if we want to - # generate shims or not. - self._system_site_packages = self.system_site_packages + # Make the context available post-creation: + self._context: Optional[SimpleNamespace] = None + + def ensure_directories(self, env_dir: DirType) -> SimpleNamespace: + logger.debug("ensure_directories(env_dir=%s)", env_dir) + self._context = super().ensure_directories(env_dir) + return self._context + + def create(self, env_dir: DirType) -> None: + logger.debug("create(env_dir=%s)", env_dir) + super().create(env_dir) + assert self._context is not None + self.post_post_setup(self._context) def post_setup(self, context: SimpleNamespace) -> None: logger.debug("post_setup(...)") - - # Generate console_script entry points for system packages: - if self._system_site_packages: - generate_console_scripts( - context.env_exe, context.bin_path, self.script_packages - ) - # print the python executable to stdout for configure. print(context.env_exe) + def post_post_setup(self, context: SimpleNamespace) -> None: + """ + The final, final hook. Enter the venv and run commands inside of it. + """ + args = [ + context.env_exe, + __file__, + "post_init", + "--binpath", + context.bin_path, + ] + if self.system_site_packages: + scripts = ",".join(self.script_packages) + if scripts: + args += ["--gen", scripts] + subprocess.run(args, check=True) + def need_ensurepip() -> bool: """
@@ -359,6 +390,13 @@ def generate_console_scripts( """ Generate script shims for console_script entry points in @packages. """ + logger.debug( + "generate_console_scripts(python_path=%s, bin_path=%s, packages=%s)", + python_path, + bin_path, + packages, + ) + if not packages: return
@@ -392,6 +430,17 @@ def _get_entry_points() -> Iterator[Dict[str, str]]: logger.debug("wrote '%s'", script_path) +def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None: + """ + This is intended to be run *inside the venv* after it is created. + """ + python_path = sys.executable + logger.debug( + "post_venv_setup(bin_path=%s, packages=%s)", bin_path, packages + ) + generate_console_scripts(python_path, bin_path, packages) + + def _add_create_subcommand(subparsers: Any) -> None: subparser = subparsers.add_parser("create", help="create a venv") subparser.add_argument(
@@ -408,6 +457,24 @@ def _add_create_subcommand(subparsers: Any) -> None: ) +def _add_post_init_subcommand(subparsers: Any) -> None: + subparser = subparsers.add_parser( + "post_init", help="post-venv initialization" + ) + subparser.add_argument( + "--gen", + type=str, + action="append", + help="Regenerate console_scripts for given packages, if found.", + ) + subparser.add_argument( + "--binpath", + type=str, + action="store", + help="Path where console script shims should be generated", + ) + + def main() -> int: """CLI interface to make_qemu_venv. See module docstring.""" if os.environ.get("DEBUG") or os.environ.get("GITLAB_CI"):
@@ -429,19 +496,23 @@ def main() -> int: ) _add_create_subcommand(subparsers) + _add_post_init_subcommand(subparsers) args = parser.parse_args() + script_packages = [] + for element in args.gen or (): + script_packages.extend(element.split(",")) + try: if args.command == "create": - script_packages = [] - for element in args.gen or (): - script_packages.extend(element.split(",")) make_venv( args.target, system_site_packages=True, clear=True, script_packages=script_packages, ) + if args.command == "post_init": + post_venv_setup(args.binpath, script_packages) logger.debug("mkvenv.py %s: exiting", args.command) except Ouch as exc: print("\n*** Ouch! ***\n", file=sys.stderr)
Signature · 1 line
2.39.2
RFC PATCH v3 6/20 mkvenv: work around broken pip installations on Debian 10
In reply to John SnowThis is a workaround intended for Debian 10, where the debian-patched pip does not function correctly if accessed from within a virtual environment.
We don't support Debian 10 any longer, but it's possible that this bug might appear on other derivative platforms and this workaround may prove useful.
RFC, a note from Paolo:
"BTW, another way to repair Debian 10's pip is to create a symbolic link to sys.base_prefix + '/share/python-wheels' in sys.prefix + '/share/python-wheels'. Since this is much faster, perhaps it can be done unconditionally [...] ?"
I was slightly apprehensive about this as it felt "hackier", but it is indeed a lot less code and much faster. It's probably low-risk. Should we do that instead, or should we just scrap any fix at all under the premise that Debian 10 support is dropped anyway?
--- python/scripts/mkvenv.py | 67 +++++++++++++++++++++++++++++++--------- 1 file changed, 52 insertions(+), 15 deletions(-)
python/scripts/mkvenv.py +52 -15
@@ -151,26 +151,26 @@ def need_ensurepip() -> bool: return True -def check_ensurepip(with_pip: bool) -> None: +def check_ensurepip(prefix: str = "", suggest_remedy: bool = False) -> None: """ Check that we have ensurepip. Raise a fatal exception with a helpful hint if it isn't available. """ - if not with_pip: - return - if not find_spec("ensurepip"): msg = ( "Python's ensurepip module is not found.\n" "It's normally part of the Python standard library, " "maybe your distribution packages it separately?\n" - "Either install ensurepip, or alleviate the need for it in the " - "first place by installing pip and setuptools for " - f"'{sys.executable}'.\n" - "(Hint: Debian puts ensurepip in its python3-venv package.)" + "(Debian puts ensurepip in its python3-venv package.)\n" ) - raise Ouch(msg) + if suggest_remedy: + msg += ( + "Either install ensurepip, or alleviate the need for it in the" + " first place by installing pip and setuptools for " + f"'{sys.executable}'.\n" + ) + raise Ouch(prefix + msg) # ensurepip uses pyexpat, which can also go missing on us: if not find_spec("pyexpat"):
@@ -178,12 +178,15 @@ def check_ensurepip(with_pip: bool) -> None: "Python's pyexpat module is not found.\n" "It's normally part of the Python standard library, " "maybe your distribution packages it separately?\n" - "Either install pyexpat, or alleviate the need for it in the " - "first place by installing pip and setuptools for " - f"'{sys.executable}'.\n\n" - "(Hint: NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)" + "(NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)\n" ) - raise Ouch(msg) + if suggest_remedy: + msg += ( + "Either install pyexpat, or alleviate the need for it in the " + "first place by installing pip and setuptools for " + f"'{sys.executable}'.\n" + ) + raise Ouch(prefix + msg) def make_venv( # pylint: disable=too-many-arguments
@@ -238,7 +241,8 @@ def make_venv( # pylint: disable=too-many-arguments with_pip = True if not system_site_packages else need_ensurepip() logger.debug("with_pip unset, choosing %s", with_pip) - check_ensurepip(with_pip) + if with_pip: + check_ensurepip(suggest_remedy=True) if symlinks is None: # Default behavior of standard venv CLI
@@ -430,6 +434,36 @@ def _get_entry_points() -> Iterator[Dict[str, str]]: logger.debug("wrote '%s'", script_path) +def checkpip() -> None: + """ + Debian10 has a pip that's broken when used inside of a virtual environment. + + We try to detect and correct that case here. + """ + try: + # pylint: disable=import-outside-toplevel, unused-import + import pip._internal # noqa: F401 + + logger.debug("pip appears to be working correctly.") + return + except ModuleNotFoundError as exc: + if exc.name == "pip._internal": + # Uh, fair enough. They did say "internal". + # Let's just assume it's fine. + return + logger.warning("pip appears to be malfunctioning: %s", str(exc)) + + check_ensurepip("pip appears to be non-functional, and ") + + logging.debug("Attempting to repair pip ...") + subprocess.run( + (sys.executable, "-m", "ensurepip"), + stdout=subprocess.DEVNULL, + check=True, + ) + logging.debug("Pip is now (hopefully) repaired!") + + def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None: """ This is intended to be run *inside the venv* after it is created.
@@ -440,6 +474,9 @@ def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None: ) generate_console_scripts(python_path, bin_path, packages) + # Test for a broken pip (Debian 10 or derivative?) and fix it if needed + checkpip() + def _add_create_subcommand(subparsers: Any) -> None: subparser = subparsers.add_parser("create", help="create a venv")
Signature · 1 line
2.39.2
RFC PATCH v3 7/20 mkvenv: add nested venv workaround
In reply to John SnowPython virtual environments do not typically nest; they may inherit from the top-level system packages or not at all.
For our purposes, it would be convenient to emulate "nested" virtual environments to allow callers of the configure script to install specific versions of python utilities in order to test build system features, utility version compatibility, etc.
While it is possible to install packages into the system environment (say, by using the --user flag), it's nicer to install test packages into a totally isolated environment instead.
As detailed in https://www.qemu.org/2023/03/24/python/, Emulate a nested venv environment by using .pth files installed into the site-packages folder that points to the parent environment when appropriate.
--- python/scripts/mkvenv.py | 72 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 69 insertions(+), 3 deletions(-)
python/scripts/mkvenv.py +69 -3
@@ -51,9 +51,11 @@ import os from pathlib import Path import re +import site import stat import subprocess import sys +import sysconfig import traceback from types import SimpleNamespace from typing import (
@@ -74,6 +76,11 @@ logger = logging.getLogger("mkvenv") +def inside_a_venv() -> bool: + """Returns True if it is executed inside of a virtual environment.""" + return sys.prefix != sys.base_prefix + + class Ouch(RuntimeError): """An Exception class we can't confuse with a builtin."""
@@ -82,9 +89,15 @@ class QemuEnvBuilder(venv.EnvBuilder): """ An extension of venv.EnvBuilder for building QEMU's configure-time venv. - The only functional change is that it adds the ability to regenerate - console_script shims for packages available via system_site - packages. + The primary differences are: + + (1) It adds the ability to regenerate console_script shims for + packages available via system_site_packages for any packages + specified by the 'script_packages' argument + + (2) It emulates a "nested" virtual environment when invoked from + inside of an existing virtual environment by including packages from + the parent. Parameters for base class init: - system_site_packages: bool = False
@@ -99,11 +112,51 @@ class QemuEnvBuilder(venv.EnvBuilder): def __init__(self, *args: Any, **kwargs: Any) -> None: logger.debug("QemuEnvBuilder.__init__(...)") self.script_packages = kwargs.pop("script_packages", ()) + + # For nested venv emulation: + self.use_parent_packages = False + if inside_a_venv(): + # Include parent packages only if we're in a venv and + # system_site_packages was True. + self.use_parent_packages = kwargs.pop( + "system_site_packages", False + ) + # Include system_site_packages only when the parent, + # The venv we are currently in, also does so. + kwargs["system_site_packages"] = sys.base_prefix in site.PREFIXES + super().__init__(*args, **kwargs) # Make the context available post-creation: self._context: Optional[SimpleNamespace] = None + def compute_venv_libpath(self, context: SimpleNamespace) -> str: + """ + Compatibility wrapper for context.lib_path for Python < 3.12 + """ + # Python 3.12+, not strictly necessary because it's documented + # to be the same as 3.10 code below: + if sys.version_info >= (3, 12): + return context.lib_path + + # Python 3.10+ + if "venv" in sysconfig.get_scheme_names(): + return sysconfig.get_path( + "purelib", scheme="venv", vars={"base": context.env_dir} + ) + + # For Python <= 3.9 we need to hardcode this. Fortunately the + # code below was the same in Python 3.6-3.10, so there is only + # one case. + if sys.platform == "win32": + return os.path.join(context.env_dir, "Lib", "site-packages") + return os.path.join( + context.env_dir, + "lib", + "python%d.%d" % sys.version_info[:2], + "site-packages", + ) + def ensure_directories(self, env_dir: DirType) -> SimpleNamespace: logger.debug("ensure_directories(env_dir=%s)", env_dir) self._context = super().ensure_directories(env_dir)
@@ -124,6 +177,19 @@ def post_post_setup(self, context: SimpleNamespace) -> None: """ The final, final hook. Enter the venv and run commands inside of it. """ + if self.use_parent_packages: + # We're inside of a venv and we want to include the parent + # venv's packages. + parent_libpath = sysconfig.get_path("purelib") + logger.debug("parent_libpath: %s", parent_libpath) + + our_libpath = self.compute_venv_libpath(context) + logger.debug("our_libpath: %s", our_libpath) + + pth_file = os.path.join(our_libpath, "nested.pth") + with open(pth_file, "w", encoding="UTF-8") as file: + file.write(parent_libpath + os.linesep) + args = [ context.env_exe, __file__,
Signature · 1 line
2.39.2
RFC PATCH v3 8/20 mkvenv: add ensure subcommand
In reply to John SnowThis command is to be used to add various packages (or ensure they're already present) into the configure-provided venv in a modular fashion.
Examples:
mkvenv ensure --online --dir "${source_dir}/python/wheels/" "meson>=0.61.5" mkvenv ensure --online "sphinx>=1.6.0"
mkvenv ensure "qemu.qmp==0.0.2"
It's designed to look for packages in three places, in order:
(1) In system packages, if the version installed is already good enough. This way your distribution-provided meson, sphinx, etc are always used as first preference.
(2) In a vendored packages directory. Here I am suggesting qemu.git/python/wheels/ as that directory. This is intended to serve as a replacement for vendoring the meson source for QEMU tarballs. It is also highly likely to be extremely useful for packaging the "qemu.qmp" package in source distributions for platforms that do not yet package qemu.qmp separately.
(3) Online, via PyPI, ***only when "--online" is passed***. This is only ever used as a fallback if the first two sources do not have an appropriate package that meets the requirement. The ability to build QEMU and run tests *completely offline* is not impinged.
--- python/scripts/mkvenv.py | 116 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 114 insertions(+), 2 deletions(-)
python/scripts/mkvenv.py +114 -2
@@ -13,6 +13,7 @@ create create a venv post_init post-venv initialization + ensure Ensure that the specified package is installed. --------------------------------------------------
@@ -34,6 +35,18 @@ --gen GEN Regenerate console_scripts for given packages, if found. --binpath BINPATH Path where console script shims should be generated +-------------------------------------------------- + +usage: mkvenv ensure [-h] [--online] [--dir DIR] dep_spec + +positional arguments: + dep_spec PEP 508 Dependency specification, e.g. 'meson>=0.61.5' + +options: + -h, --help show this help message and exit + --online Install packages from PyPI, if necessary. + --dir DIR Path to vendored packages where we may install from. + """ # Copyright (C) 2022-2023 Red Hat, Inc.
@@ -530,6 +543,71 @@ def checkpip() -> None: logging.debug("Pip is now (hopefully) repaired!") +def pip_install( + args: Sequence[str], + online: bool = False, + wheels_dir: Optional[Union[str, Path]] = None, +) -> None: + """ + Use pip to install a package or package(s) as specified in @args. + """ + loud = bool( + os.environ.get("DEBUG") + or os.environ.get("GITLAB_CI") + or os.environ.get("V") + ) + + full_args = [ + sys.executable, + "-m", + "pip", + "install", + "--disable-pip-version-check", + "-v" if loud else "-q", + ] + if not online: + full_args += ["--no-index"] + if wheels_dir: + full_args += ["--find-links", f"file://{str(wheels_dir)}"] + full_args += list(args) + subprocess.run(full_args, check=True) + + +def ensure( + dep_spec: str, + online: bool = False, + wheels_dir: Optional[Union[str, Path]] = None, +) -> None: + """ + Use pip to ensure we have the package specified by @dep_spec. + + If the package is already installed, do nothing. If online and + wheels_dir are both provided, prefer packages found in wheels_dir + first before connecting to PyPI. + + :param dep_spec: + A PEP 508 dependency specification. e.g. 'meson>=0.61.5'. + :param online: If True, fall back to PyPI. + :param wheels_dir: If specified, search this path for packages. + """ + # This first install command will: + # (A) Do nothing, if we already have a suitable package. + # (B) Install the package from vendored source, if possible. + # (C) Fail if neither A nor B. + try: + pip_install([dep_spec], online=False, wheels_dir=wheels_dir) + # (A) or (B) happened. Success. + return + except subprocess.CalledProcessError: + # (C) Happened. + # The package is missing or isn't a suitable version, + # and we weren't able to install a suitable vendored package. + if online: + pip_install([dep_spec], online=True) + else: + raise + + def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None: """ This is intended to be run *inside the venv* after it is created.
@@ -578,6 +656,29 @@ def _add_post_init_subcommand(subparsers: Any) -> None: ) +def _add_ensure_subcommand(subparsers: Any) -> None: + subparser = subparsers.add_parser( + "ensure", help="Ensure that the specified package is installed." + ) + subparser.add_argument( + "--online", + action="store_true", + help="Install packages from PyPI, if necessary.", + ) + subparser.add_argument( + "--dir", + type=str, + action="store", + help="Path to vendored packages where we may install from.", + ) + subparser.add_argument( + "dep_spec", + type=str, + action="store", + help="PEP 508 Dependency specification, e.g. 'meson>=0.61.5'", + ) + + def main() -> int: """CLI interface to make_qemu_venv. See module docstring.""" if os.environ.get("DEBUG") or os.environ.get("GITLAB_CI"):
@@ -600,14 +701,18 @@ def main() -> int: _add_create_subcommand(subparsers) _add_post_init_subcommand(subparsers) + _add_ensure_subcommand(subparsers) args = parser.parse_args() script_packages = [] - for element in args.gen or (): - script_packages.extend(element.split(",")) + + def _normalize_gen() -> None: + for element in args.gen or (): + script_packages.extend(element.split(",")) try: if args.command == "create": + _normalize_gen() make_venv( args.target, system_site_packages=True,
@@ -615,7 +720,14 @@ def main() -> int: script_packages=script_packages, ) if args.command == "post_init": + _normalize_gen() post_venv_setup(args.binpath, script_packages) + if args.command == "ensure": + ensure( + dep_spec=args.dep_spec, + online=args.online, + wheels_dir=args.dir, + ) logger.debug("mkvenv.py %s: exiting", args.command) except Ouch as exc: print("\n*** Ouch! ***\n", file=sys.stderr)
Signature · 1 line
2.39.2
RFC PATCH v3 9/20 tests/docker: add python3-venv dependency
In reply to John SnowSeveral debian-based tests need the python3-venv dependency as a consequence of Debian debundling the "ensurepip" module normally included with Python.
As mkvenv.py stands as of this commit, Debian requires EITHER:
(A) setuptools and pip, or
(B) ensurepip
mkvenv is a few seconds faster if you have setuptools and pip, so developers should prefer the first requirement. For the purposes of CI, the time-save is a wash; it's only a matter of who is responsible for installing pip and when; the timing is about the same.
Arbitrarily, I chose adding ensurepip to the test configuration because it is normally part of the Python stdlib, and always having it allows us a more consistent cross-platform environment.
--- tests/docker/dockerfiles/debian-all-test-cross.docker | 3 ++- tests/docker/dockerfiles/debian-hexagon-cross.docker | 3 ++- tests/docker/dockerfiles/debian-riscv64-cross.docker | 3 ++- tests/docker/dockerfiles/debian-tricore-cross.docker | 3 ++- 4 files changed, 8 insertions(+), 4 deletions(-)
tests/docker/dockerfiles/debian-all-test-cross.docker +2 -1
@@ -57,7 +57,8 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \ gcc-sh4-linux-gnu \ libc6-dev-sh4-cross \ gcc-sparc64-linux-gnu \ - libc6-dev-sparc64-cross + libc6-dev-sparc64-cross \ + python3-venv ENV QEMU_CONFIGURE_OPTS --disable-system --disable-docs --disable-tools ENV DEF_TARGET_LIST aarch64-linux-user,alpha-linux-user,arm-linux-user,hppa-linux-user,i386-linux-user,m68k-linux-user,mips-linux-user,mips64-linux-user,mips64el-linux-user,mipsel-linux-user,ppc-linux-user,ppc64-linux-user,ppc64le-linux-user,riscv64-linux-user,s390x-linux-user,sh4-linux-user,sparc64-linux-user
tests/docker/dockerfiles/debian-hexagon-cross.docker +2 -1
@@ -20,7 +20,8 @@ RUN apt-get update && \ bison \ flex \ git \ - ninja-build && \ + ninja-build \ + python3-venv && \ # Install QEMU build deps for use in CI DEBIAN_FRONTEND=noninteractive eatmydata \ apt build-dep -yy --arch-only qemu
tests/docker/dockerfiles/debian-riscv64-cross.docker +2 -1
@@ -28,7 +28,8 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata apt install -yy \ libglib2.0-dev \ ninja-build \ pkg-config \ - python3 + python3 \ + python3-venv # Add ports and riscv64 architecture RUN echo "deb http://ftp.ports.debian.org/debian-ports/ sid main" >> /etc/apt/sources.list
tests/docker/dockerfiles/debian-tricore-cross.docker +2 -1
@@ -33,7 +33,8 @@ RUN apt update && \ pkgconf \ python3-pip \ python3-setuptools \ - python3-wheel + python3-wheel \ + python3-venv RUN curl -#SL https://github.com/bkoppelmann/package_940/releases/download/tricore-toolchain-9.40/tricore-toolchain-9.4.0.tar.gz \ | tar -xzC /usr/local/
Signature · 1 line
2.39.2
Re: [RFC PATCH v3 09/20] tests/docker: add python3-venv dependency
Quoted 9 lines · John Snow
Several debian-based tests need the python3-venv dependency as a consequence of Debian debundling the "ensurepip" module normally included with Python.
As
mkvenv.pystands as of this commit, Debian requires EITHER:(A) setuptools and pip, or
(B) ensurepip
Quoted 8 lines
mkvenv is a few seconds faster if you have setuptools and pip, so developers should prefer the first requirement. For the purposes of CI, the time-save is a wash; it's only a matter of who is responsible for installing pip and when; the timing is about the same.
Arbitrarily, I chose adding ensurepip to the test configuration because it is normally part of the Python stdlib, and always having it allows us a more consistent cross-platform environment.
python3-venv also matches the package that we already list in the lcitool auto generated dockerfiles, so that consistency is good.
Quoted 8 lines
Signed-off-by John Snow <[email protected]>--- tests/docker/dockerfiles/debian-all-test-cross.docker | 3 ++- tests/docker/dockerfiles/debian-hexagon-cross.docker | 3 ++- tests/docker/dockerfiles/debian-riscv64-cross.docker | 3 ++- tests/docker/dockerfiles/debian-tricore-cross.docker | 3 ++- 4 files changed, 8 insertions(+), 4 deletions(-)
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
RFC PATCH v3 10/20 tests/vm: Configure netbsd to use Python 3.10
In reply to John SnowNetBSD removes some packages from the Python stdlib, but only re-packages them for Python 3.10. Switch to using Python 3.10.
--- tests/vm/netbsd | 1 + 1 file changed, 1 insertion(+)
tests/vm/netbsd +1 -0
@@ -30,6 +30,7 @@ class NetBSDVM(basevm.BaseVM): "git-base", "pkgconf", "xz", + "python310", "ninja-build", # gnu tools
Signature · 1 line
2.39.2
Re: [RFC PATCH v3 10/20] tests/vm: Configure netbsd to use Python 3.10
Quoted 7 lines · John Snow
NetBSD removes some packages from the Python stdlib, but only re-packages them for Python 3.10. Switch to using Python 3.10.
Signed-off-by John Snow <[email protected]>--- tests/vm/netbsd | 1 + 1 file changed, 1 insertion(+)
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
RFC PATCH v3 11/20 tests/vm: add py310-expat to NetBSD
In reply to John SnowNetBSD cannot successfully run "ensurepip" without access to the pyexpat module, which NetBSD debundles. Like the Debian patch, it would be strictly faster long term to install pip/setuptools, and I recommend developers at their workstations take that approach instead.
For the purposes of a throwaway VM, there's not really a speed difference for who is responsible for installing pip; us (needs py310-pip) or Python (needs py310-expat).
--- tests/vm/netbsd | 1 + 1 file changed, 1 insertion(+)
tests/vm/netbsd +1 -0
@@ -31,6 +31,7 @@ class NetBSDVM(basevm.BaseVM): "pkgconf", "xz", "python310", + "py310-expat", "ninja-build", # gnu tools
Signature · 1 line
2.39.2
Re: [RFC PATCH v3 11/20] tests/vm: add py310-expat to NetBSD
Quoted 4 lines · John Snow
NetBSD cannot successfully run "ensurepip" without access to the pyexpat module, which NetBSD debundles. Like the Debian patch, it would be strictly faster long term to install pip/setuptools, and I recommend developers at their workstations take that approach instead.
This sounds a bit like a NetBSD packaging bug surely ? If we've got ensurepip installed and it doesn't work due to missing expat I would have though NetBSD should add a dependancy.
None the less, we'll need a fix in QEMU until that's addressed.
Quoted 9 lines
For the purposes of a throwaway VM, there's not really a speed difference for who is responsible for installing pip; us (needs py310-pip) or Python (needs py310-expat).
Signed-off-by John Snow <[email protected]>--- tests/vm/netbsd | 1 + 1 file changed, 1 insertion(+)
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [RFC PATCH v3 11/20] tests/vm: add py310-expat to NetBSD
Quoted 10 lines · Daniel P. Berrangé
Quoted 4 lines · John Snow
NetBSD cannot successfully run "ensurepip" without access to the pyexpat module, which NetBSD debundles. Like the Debian patch, it would be strictly faster long term to install pip/setuptools, and I recommend developers at their workstations take that approach instead.
This sounds a bit like a NetBSD packaging bug surely ? If we've got ensurepip installed and it doesn't work due to missing expat I would have though NetBSD should add a dependancy.
Yes, I agree - though I think NetBSD doesn't unbundle ensurepip, so there might be nothing to add a dependency against ... and I assume they split out pyexpat for a reason.
I can report it, anyway. It should at least fail more gracefully than it does.
Quoted 25 lines
None the less, we'll need a fix in QEMU until that's addressed.
Quoted 9 lines
For the purposes of a throwaway VM, there's not really a speed difference for who is responsible for installing pip; us (needs py310-pip) or Python (needs py310-expat).
Signed-off-by John Snow <[email protected]>--- tests/vm/netbsd | 1 + 1 file changed, 1 insertion(+)Reviewed-by Daniel P. Berrangé <[email protected]>With regards,
Daniel
--
|: https://berrange.com -o-
https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-
https://fstop138.berrange.com :|
|: https://entangle-photo.org -o-
https://www.instagram.com/dberrange :|
RFC PATCH v3 12/20 scripts/make-release: download meson==0.61.5 .whl
In reply to John SnowIn preference to vendoring meson source, vendor a built distributable. This has two benefits:
(1) We can get rid of a git submodule,
(2) Installing built meson into a venv doesn't require any extra dependencies.
RFC:
The alternative approach is to just check in the .whl file into the git tree directly, and have it available for both git and tarball installations. That approach slightly changes the necessity of some subsequent patches, but otherwise either way will work.
Owing to how "mkvenv ensure" will prefer vendored files prior to connecting to PyPI, checking in a vendored meson file in this manner means we will generally never use PyPI to install meson ever.
("Vote now on your phones.")
--- scripts/make-release | 11 +++++++++++ 1 file changed, 11 insertions(+)
scripts/make-release +11 -0
@@ -41,6 +41,17 @@ git submodule update --init --single-branch BaseTools/Source/C/BrotliCompress/brotli \ CryptoPkg/Library/OpensslLib/openssl \ MdeModulePkg/Library/BrotliCustomDecompressLib/brotli) + +# Handle vendoring Python dependencies: +mkdir python/wheels +pushd python/wheels +pip download meson==0.61.5 +sha256sum -c <<EOF +58c2ddb5f885da0e929f15d89f38d8a7d97f981f56815bcba008414f8511f59a meson-0.61.5-py3-none-any.whl +EOF popd + +popd + tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination} rm -rf ${destination}
Signature · 1 line
2.39.2
RFC PATCH v3 13/20 configure: create a python venv unconditionally
In reply to John SnowThis patch changes the configure script so that it always creates and uses a python virtual environment unconditionally.
Meson bootstrapping is temporarily altered to force the use of meson from git or vendored source. The next commit restores the use of distribution-vendored Meson.
--- configure | 36 +++++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-)
configure +31 -5
@@ -625,7 +625,6 @@ check_py_version() { python= first_python= if test -z "${PYTHON}"; then - explicit_python=no # A bare 'python' is traditionally python 2.x, but some distros # have it as python 3.x, so check in both places. for binary in python3 python python3.11 python3.10 python3.9 python3.8 python3.7 python3.6; do
@@ -644,7 +643,6 @@ else # Same as above, but only check the environment variable. has "${PYTHON}" || error_exit "The PYTHON environment variable does not point to an executable" python=$(command -v "$PYTHON") - explicit_python=yes if check_py_version "$python"; then # This one is good. first_python=
@@ -729,7 +727,7 @@ for opt do ;; --install=*) ;; - --python=*) python="$optarg" ; explicit_python=yes + --python=*) python="$optarg" ;; --skip-meson) skip_meson=yes ;;
@@ -1089,8 +1087,34 @@ if ! check_py_version "$python"; then "Use --python=/path/to/python to specify a supported Python." fi -# Resolve PATH + suppress writing compiled files -python="$(command -v "$python") -B" +# Resolve PATH +python="$(command -v "$python")" +explicit_python=yes + +# Create a Python virtual environment using our configured python. +# The stdout of this script will be the location of a symlink that +# points to the configured Python. +# Entry point scripts for pip, meson, and sphinx are generated if those +# packages are present. + +# Defaults assumed for now: +# - venv is cleared if it exists already; +# - venv is allowed to use system packages; +# - all setup is performed **offline**; +# - No packages are installed by default; +# - pip is not installed into the venv when possible, +# but ensurepip is called as a fallback when necessary. + +echo "python determined to be '$python'" +echo "python version: $($python --version)" + +python="$($python -B "${source_path}/python/scripts/mkvenv.py" create --gen pip,meson,sphinx pyvenv)" +if test "$?" -ne 0 ; then + error_exit "python venv creation failed" +fi + +# Suppress writing compiled files +python="$python -B" has_meson() { local python_dir=$(dirname "$python")
@@ -1145,6 +1169,8 @@ case "$meson" in *) meson=$(command -v "$meson") ;; esac +echo "MKVENV ok!" + # Probe for ninja if test -z "$ninja"; then
Signature · 1 line
2.39.2
RFC PATCH v3 14/20 configure: use 'mkvenv ensure meson' to bootstrap meson
In reply to John SnowThis commit changes how we detect and install meson. It notably removes '--meson='.
The previous patch creates a lightweight Python virtual environment unconditionally using the user's configured $python that inherits system packages. If Meson is installed there and meets our minimum version requirements, we will use that Meson.
In the event that Meson is installed but *not for the chosen Python interpreter*, not found, or of insufficient version, we will attempt to install Meson from vendored source into the newly created Python virtual environment. This vendored installation is considered to replace the mechanism from prior tarball distributions.
This commit restores the ability to use a system meson, but in turn temporarily removes the ability to use a meson as obtained from the internet at configure-time (git submodules, as it stood prior to this patch); that ability will be restored in the next commit.
As a result of this patch, the Python interpreter we use for both our own build scripts *and* Meson extensions are always known to be the exact same Python. As a further benefit, there will also be a symlink available in the build directory that points to the correct, configured python and can be used by e.g. manual tests to invoke the correct, configured Python unambiguously.
--- configure | 72 ++++++++--------------------- .gitlab-ci.d/buildtest-template.yml | 4 +- 2 files changed, 21 insertions(+), 55 deletions(-)
configure +19 -53
@@ -731,8 +731,6 @@ for opt do ;; --skip-meson) skip_meson=yes ;; - --meson=*) meson="$optarg" - ;; --ninja=*) ninja="$optarg" ;; --smbd=*) smbd="$optarg"
@@ -1016,7 +1014,6 @@ Advanced options (experts only): --cross-prefix-ARCH=PREFIX cross compiler prefix when building ARCH guest test cases --make=MAKE use specified make [$make] --python=PYTHON use specified python [$python] - --meson=MESON use specified meson [$meson] --ninja=NINJA use specified ninja [$ninja] --smbd=SMBD use specified smbd [$smbd] --with-git=GIT use specified git [$git]
@@ -1089,7 +1086,8 @@ fi # Resolve PATH python="$(command -v "$python")" -explicit_python=yes +# This variable is intended to be used only for error messages: +target_python=$python # Create a Python virtual environment using our configured python. # The stdout of this script will be the location of a symlink that
@@ -1101,7 +1099,6 @@ explicit_python=yes # - venv is cleared if it exists already; # - venv is allowed to use system packages; # - all setup is performed **offline**; -# - No packages are installed by default; # - pip is not installed into the venv when possible, # but ensurepip is called as a fallback when necessary.
@@ -1116,58 +1113,27 @@ fi # Suppress writing compiled files python="$python -B" -has_meson() { - local python_dir=$(dirname "$python") - # PEP405: pyvenv.cfg is either adjacent to the Python executable - # or one directory above - if test -f $python_dir/pyvenv.cfg || test -f $python_dir/../pyvenv.cfg; then - # Ensure that Meson and Python come from the same virtual environment - test -x "$python_dir/meson" && - test "$(command -v meson)" -ef "$python_dir/meson" - else - has meson - fi -} -if test -z "$meson"; then - if test "$explicit_python" = no && has_meson && version_ge "$(meson --version)" 0.61.5; then - meson=meson - elif test "$git_submodules_action" != 'ignore' ; then - meson=git - elif test -e "${source_path}/meson/meson.py" ; then - meson=internal - else - if test "$explicit_python" = yes; then - error_exit "--python requires using QEMU's embedded Meson distribution, but it was not found." - else - error_exit "Meson not found. Use --meson=/path/to/meson" - fi +if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \ + --dir "${source_path}/python/wheels" \ + "meson>=0.61.5" ; +then + # We're very out of luck. Try to give a good diagnostic. + if test -e pyvenv/bin/meson; then + echo "Meson is too old: $(pyvenv/bin/meson --version)" + elif has meson ; then + echo "Meson was found installed on your system," \ + "but not for the configured Python interpreter ($target_python)." + echo "(Hint: check '$(which meson)' to see which interpreter its shebang uses.)" fi -else - # Meson uses its own Python interpreter to invoke other Python scripts, - # but the user wants to use the one they specified with --python. - # - # We do not want to override the distro Python interpreter (and sometimes - # cannot: for example in Homebrew /usr/bin/meson is a bash script), so - # just require --meson=git|internal together with --python. - if test "$explicit_python" = yes; then - case "$meson" in - git | internal) ;; - *) error_exit "--python requires using QEMU's embedded Meson distribution." ;; - esac - fi -fi - -if test "$meson" = git; then - git_submodules="${git_submodules} meson" + exit 1 fi -case "$meson" in - git | internal) - meson="$python ${source_path}/meson/meson.py" - ;; - *) meson=$(command -v "$meson") ;; -esac +# At this point, we expect Meson to be installed and available. +# We expect mkvenv or pip to have created pyvenv/bin/meson for us. +# We ignore PATH completely here: we want to use the venv's Meson +# *exclusively*. +meson="$(cd pyvenv/bin; pwd)/meson" echo "MKVENV ok!"
.gitlab-ci.d/buildtest-template.yml +2 -2
@@ -12,12 +12,12 @@ - mkdir build - cd build - ../configure --enable-werror --disable-docs --enable-fdt=system - ${LD_JOBS:+--meson=git} ${TARGETS:+--target-list="$TARGETS"} + ${TARGETS:+--target-list="$TARGETS"} $CONFIGURE_ARGS || { cat config.log meson-logs/meson-log.txt && exit 1; } - if test -n "$LD_JOBS"; then - ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ; + pyvenv/bin/meson configure . -Dbackend_max_links="$LD_JOBS" ; fi || exit 1; - make -j"$JOBS" - if test -n "$MAKE_CHECK_ARGS";
Signature · 1 line
2.39.2
Re: [RFC PATCH v3 14/20] configure: use 'mkvenv ensure meson' to bootstrap meson
Quoted 102 lines · John Snow
This commit changes how we detect and install meson. It notably removes '--meson='.
The previous patch creates a lightweight Python virtual environment unconditionally using the user's configured $python that inherits system packages. If Meson is installed there and meets our minimum version requirements, we will use that Meson.
In the event that Meson is installed but *not for the chosen Python interpreter*, not found, or of insufficient version, we will attempt to install Meson from vendored source into the newly created Python virtual environment. This vendored installation is considered to replace the mechanism from prior tarball distributions.
This commit restores the ability to use a system meson, but in turn temporarily removes the ability to use a meson as obtained from the internet at configure-time (git submodules, as it stood prior to this patch); that ability will be restored in the next commit.
As a result of this patch, the Python interpreter we use for both our own build scripts *and* Meson extensions are always known to be the exact same Python. As a further benefit, there will also be a symlink available in the build directory that points to the correct, configured python and can be used by e.g. manual tests to invoke the correct, configured Python unambiguously.
Signed-off-by John Snow <[email protected]>--- configure | 72 ++++++++--------------------- .gitlab-ci.d/buildtest-template.yml | 4 +- 2 files changed, 21 insertions(+), 55 deletions(-)configure +0
-2@@ -731,8 +731,6 @@ for opt do ;; --skip-meson) skip_meson=yes ;; - --meson=*) meson="$optarg" - ;; --ninja=*) ninja="$optarg" ;; --smbd=*) smbd="$optarg"@@ -1016,7 +1014,6 @@ Advanced options (experts only): --cross-prefix-ARCH=PREFIX cross compiler prefix when building ARCHguest test cases
--make=MAKE use specified make [$make]
--python=PYTHON use specified python [$python]configure +2
-8- --meson=MESON use specified meson [$meson] --ninja=NINJA use specified ninja [$ninja] --smbd=SMBD use specified smbd [$smbd] --with-git=GIT use specified git [$git]@@ -1089,7 +1086,8 @@ fi # Resolve PATH python="$(command -v "$python")" -explicit_python=yes +# This variable is intended to be used only for error messages: +target_python=$python # Create a Python virtual environment using our configured python. # The stdout of this script will be the location of a symlink that@@ -1101,7 +1099,6 @@ explicit_python=yes # - venv is cleared if it exists already; # - venv is allowed to use system packages; # - all setup is performed **offline**; -# - No packages are installed by default; # - pip is not installed into the venv when possible, # but ensurepip is called as a fallback when necessary.@@ -1116,58 +1113,27 @@ fi # Suppress writing compiled files python="$python -B" -has_meson() { - local python_dir=$(dirname "$python") - # PEP405: pyvenv.cfg is either adjacent to the Python executable - # or one directory above - if test -f $python_dir/pyvenv.cfg || test -f $python_dir/../pyvenv.cfg;then
configure +0
-18- # Ensure that Meson and Python come from the same virtual environment - test -x "$python_dir/meson" && - test "$(command -v meson)" -ef "$python_dir/meson" - else - has meson - fi -} -if test -z "$meson"; then - if test "$explicit_python" = no && has_meson && version_ge "$(meson --version)" 0.61.5; then - meson=meson - elif test "$git_submodules_action" != 'ignore' ; then - meson=git - elif test -e "${source_path}/meson/meson.py" ; then - meson=internal - else - if test "$explicit_python" = yes; then - error_exit "--python requires using QEMU's embedded Mesondistribution, but it was not found."
configure fi
Quoted 10 lines
configure +7
-3- else - error_exit "Meson not found. Use --meson=/path/to/meson" - fi +if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \ + --dir "${source_path}/python/wheels" \ + "meson>=0.61.5" ; +then + # We're very out of luck. Try to give a good diagnostic. + if test -e pyvenv/bin/meson; then + echo "Meson is too old:
Does a minimum version still get printed? I've needed to know that in the past when I got the error...
Warner
$(pyvenv/bin/meson --version)"
Quoted 68 lines
configure +3
-0+ elif has meson ; then + echo "Meson was found installed on your system," \ + "but not for the configured Python interpreter($target_python)."
+ echo "(Hint: check '$(which meson)' to see which interpreter its
shebang uses.)"
ficonfigure +0
-2-else - # Meson uses its own Python interpreter to invoke other Pythonscripts,
configure +0
-3- # but the user wants to use the one they specified with --python. - # - # We do not want to override the distro Python interpreter (andsometimes
configure +0
-6- # cannot: for example in Homebrew /usr/bin/meson is a bash script), so - # just require --meson=git|internal together with --python. - if test "$explicit_python" = yes; then - case "$meson" in - git | internal) ;; - *) error_exit "--python requires using QEMU's embedded Mesondistribution." ;;
configure +6
-12- esac - fi -fi - -if test "$meson" = git; then - git_submodules="${git_submodules} meson" + exit 1 fi -case "$meson" in - git | internal) - meson="$python ${source_path}/meson/meson.py" - ;; - *) meson=$(command -v "$meson") ;; -esac +# At this point, we expect Meson to be installed and available. +# We expect mkvenv or pip to have created pyvenv/bin/meson for us. +# We ignore PATH completely here: we want to use the venv's Meson +# *exclusively*. +meson="$(cd pyvenv/bin; pwd)/meson" echo "MKVENV ok!".gitlab-ci.d/buildtest-template.yml +2
-3@@ -12,12 +12,12 @@ - mkdir build - cd build - ../configure --enable-werror --disable-docs --enable-fdt=system - ${LD_JOBS:+--meson=git} ${TARGETS:+--target-list="$TARGETS"} + ${TARGETS:+--target-list="$TARGETS"} $CONFIGURE_ARGS || { cat config.log meson-logs/meson-log.txt && exit 1; } - if test -n "$LD_JOBS"; then - ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ; + pyvenv/bin/meson configure . -Dbackend_max_links="$LD_JOBS" ; fi || exit 1; - make -j"$JOBS" - if test -n "$MAKE_CHECK_ARGS"; --2.39.2
Re: [RFC PATCH v3 14/20] configure: use 'mkvenv ensure meson' to bootstrap meson
Quoted 117 lines · Warner Losh
Quoted 99 lines · John Snow
This commit changes how we detect and install meson. It notably removes '--meson='.
The previous patch creates a lightweight Python virtual environment unconditionally using the user's configured $python that inherits system packages. If Meson is installed there and meets our minimum version requirements, we will use that Meson.
In the event that Meson is installed but *not for the chosen Python interpreter*, not found, or of insufficient version, we will attempt to install Meson from vendored source into the newly created Python virtual environment. This vendored installation is considered to replace the mechanism from prior tarball distributions.
This commit restores the ability to use a system meson, but in turn temporarily removes the ability to use a meson as obtained from the internet at configure-time (git submodules, as it stood prior to this patch); that ability will be restored in the next commit.
As a result of this patch, the Python interpreter we use for both our own build scripts *and* Meson extensions are always known to be the exact same Python. As a further benefit, there will also be a symlink available in the build directory that points to the correct, configured python and can be used by e.g. manual tests to invoke the correct, configured Python unambiguously.
Signed-off-by John Snow <[email protected]>--- configure | 72 ++++++++--------------------- .gitlab-ci.d/buildtest-template.yml | 4 +- 2 files changed, 21 insertions(+), 55 deletions(-)configure +2
-27@@ -731,8 +731,6 @@ for opt do ;; --skip-meson) skip_meson=yes ;; - --meson=*) meson="$optarg" - ;; --ninja=*) ninja="$optarg" ;; --smbd=*) smbd="$optarg"@@ -1016,7 +1014,6 @@ Advanced options (experts only): --cross-prefix-ARCH=PREFIX cross compiler prefix when building ARCH guest test cases --make=MAKE use specified make [$make] --python=PYTHON use specified python [$python] - --meson=MESON use specified meson [$meson] --ninja=NINJA use specified ninja [$ninja] --smbd=SMBD use specified smbd [$smbd] --with-git=GIT use specified git [$git]@@ -1089,7 +1086,8 @@ fi # Resolve PATH python="$(command -v "$python")" -explicit_python=yes +# This variable is intended to be used only for error messages: +target_python=$python # Create a Python virtual environment using our configured python. # The stdout of this script will be the location of a symlink that@@ -1101,7 +1099,6 @@ explicit_python=yes # - venv is cleared if it exists already; # - venv is allowed to use system packages; # - all setup is performed **offline**; -# - No packages are installed by default; # - pip is not installed into the venv when possible, # but ensurepip is called as a fallback when necessary.@@ -1116,58 +1113,27 @@ fi # Suppress writing compiled files python="$python -B" -has_meson() { - local python_dir=$(dirname "$python") - # PEP405: pyvenv.cfg is either adjacent to the Python executable - # or one directory above - if test -f $python_dir/pyvenv.cfg || test -f $python_dir/../pyvenv.cfg; then - # Ensure that Meson and Python come from the same virtual environment - test -x "$python_dir/meson" && - test "$(command -v meson)" -ef "$python_dir/meson" - else - has meson - fi -} -if test -z "$meson"; then - if test "$explicit_python" = no && has_meson && version_ge "$(meson --version)" 0.61.5; then - meson=meson - elif test "$git_submodules_action" != 'ignore' ; then - meson=git - elif test -e "${source_path}/meson/meson.py" ; then - meson=internal - else - if test "$explicit_python" = yes; then - error_exit "--python requires using QEMU's embedded Meson distribution, but it was not found."configure fi · line 1123
Quoted 10 lines
configure +7
-3- else - error_exit "Meson not found. Use --meson=/path/to/meson" - fi +if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \ + --dir "${source_path}/python/wheels" \ + "meson>=0.61.5" ; +then + # We're very out of luck. Try to give a good diagnostic. + if test -e pyvenv/bin/meson; then + echo "Meson is too old:Does a minimum version still get printed? I've needed to know that in the past when I got the error...
Quoted 2 lines
Warner
$(pyvenv/bin/meson --version)"
At the end of the series, here's what happens if i change the meson requirement to a fictionally too-high version that would be impossible to fulfill:
--------------------------------------------------------------------------------
jsnow@scv ~/s/q/b/git (python-configure-venv)> ../../configure --enable-debug --disable-pypi;
python determined to be '/usr/bin/python3'
python version: Python 3.11.2
MKVENV pyvenv
MKVENV ensure meson>=2.61.5
ERROR: Could not find a version that satisfies the requirement meson>=2.61.5 (from versions: none)
ERROR: No matching distribution found for meson>=2.61.5
*** Ouch! ***
Could not ensure availability of 'meson>=2.61.5':
• Python package 'meson' version '1.0.1' was found, but isn't suitable.
• No suitable version found in, or failed to install from
'/home/jsnow/src/qemu/python/wheels'.
• mkvenv was configured to operate offline and did not check PyPI.
--------------------------------------------------------------------------------
Re: [RFC PATCH v3 14/20] configure: use 'mkvenv ensure meson' to bootstrap meson
Quoted 143 lines · John Snow
Quoted 37 lines · Warner Losh
Quoted 33 lines · John Snow
This commit changes how we detect and install meson. It notably removes '--meson='.
The previous patch creates a lightweight Python virtual environment unconditionally using the user's configured $python that inherits system packages. If Meson is installed there and meets our minimum version requirements, we will use that Meson.
In the event that Meson is installed but *not for the chosen Python interpreter*, not found, or of insufficient version, we will attempt to install Meson from vendored source into the newly created Python virtual environment. This vendored installation is considered to replace the mechanism from prior tarball distributions.
This commit restores the ability to use a system meson, but in turn temporarily removes the ability to use a meson as obtained from the internet at configure-time (git submodules, as it stood prior to this patch); that ability will be restored in the next commit.
As a result of this patch, the Python interpreter we use for both our own build scripts *and* Meson extensions are always known to be the exact same Python. As a further benefit, there will also be a symlink available in the build directory that points to the correct, configured python and can be used by e.g. manual tests to invoke the correct, configured Python unambiguously.
Signed-off-by John Snow <[email protected]>--- configure | 72 ++++++++--------------------- .gitlab-ci.d/buildtest-template.yml | 4 +- 2 files changed, 21 insertions(+), 55 deletions(-)Quoted 15 lines
Quoted 14 lines
configure +0
-2@@ -731,8 +731,6 @@ for opt do ;; --skip-meson) skip_meson=yes ;; - --meson=*) meson="$optarg" - ;; --ninja=*) ninja="$optarg" ;; --smbd=*) smbd="$optarg"configure Advanced options (experts only): · line 1014
Quoted 1 line
configure +0
-0@@ -1016,7 +1014,6 @@ Advanced options (experts only): --cross-prefix-ARCH=PREFIX cross compiler prefix when building ARCHguest test cases
Quoted 28 lines
Quoted 28 lines
configure +2
-3--make=MAKE use specified make [$make] --python=PYTHON use specified python [$python] - --meson=MESON use specified meson [$meson] --ninja=NINJA use specified ninja [$ninja] --smbd=SMBD use specified smbd [$smbd] --with-git=GIT use specified git [$git]@@ -1089,7 +1086,8 @@ fi # Resolve PATH python="$(command -v "$python")" -explicit_python=yes +# This variable is intended to be used only for error messages: +target_python=$python # Create a Python virtual environment using our configured python. # The stdout of this script will be the location of a symlink that@@ -1101,7 +1099,6 @@ explicit_python=yes # - venv is cleared if it exists already; # - venv is allowed to use system packages; # - all setup is performed **offline**; -# - No packages are installed by default; # - pip is not installed into the venv when possible, # but ensurepip is called as a fallback when necessary.@@ -1116,58 +1113,27 @@ fi # Suppress writing compiled files python="$python -B"Quoted 5 lines
configure fi
Quoted 5 lines
configure +0
-5-has_meson() { - local python_dir=$(dirname "$python") - # PEP405: pyvenv.cfg is either adjacent to the Python executable - # or one directory above - if test -f $python_dir/pyvenv.cfg || test -f$python_dir/../pyvenv.cfg; then
Quoted 1 line
configure fi · line 1124
Quoted 1 line
configure +0
-1- # Ensure that Meson and Python come from the same virtualenvironment
Quoted 9 lines
configure fi
Quoted 9 lines
configure +0
-8- test -x "$python_dir/meson" && - test "$(command -v meson)" -ef "$python_dir/meson" - else - has meson - fi -} -if test -z "$meson"; then - if test "$explicit_python" = no && has_meson && version_ge"$(meson --version)" 0.61.5; then
Quoted 8 lines
configure fi
Quoted 8 lines
configure +0
-8- meson=meson - elif test "$git_submodules_action" != 'ignore' ; then - meson=git - elif test -e "${source_path}/meson/meson.py" ; then - meson=internal - else - if test "$explicit_python" = yes; then - error_exit "--python requires using QEMU's embedded Mesondistribution, but it was not found."
Quoted 5 lines
Quoted 5 lines
configure +2
-3- else - error_exit "Meson not found. Use --meson=/path/to/meson" - fi +if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \ + --dir "${source_path}/python/wheels" \Quoted 8 lines
configure fi
Quoted 5 lines
configure +5
-0+ "meson>=0.61.5" ; +then + # We're very out of luck. Try to give a good diagnostic. + if test -e pyvenv/bin/meson; then + echo "Meson is too old:Does a minimum version still get printed? I've needed to know that in
the past when I got the error...
Quoted 3 lines
Warner
$(pyvenv/bin/meson --version)"At the end of the series, here's what happens if i change the meson requirement to a fictionally too-high version that would be impossible to fulfill:
--------------------------------------------------------------------------------
jsnow@scv ~/s/q/b/git (python-configure-venv)> ../../configure --enable-debug --disable-pypi;
python determined to be '/usr/bin/python3'
python version: Python 3.11.2
MKVENV pyvenv
MKVENV ensure meson>=2.61.5
ERROR: Could not find a version that satisfies the requirement meson>=2.61.5 (from versions: none)
ERROR: No matching distribution found for meson>=2.61.5
Quoted 2 lines
*** Ouch! ***
Oh, that's enough :) :) thanks! I'd be happy with less...
Warner
Quoted 9 lines
Could not ensure availability of 'meson>=2.61.5':
• Python package 'meson' version '1.0.1' was found, but isn't suitable.
• No suitable version found in, or failed to install from
'/home/jsnow/src/qemu/python/wheels'.
• mkvenv was configured to operate offline and did not check PyPI.--------------------------------------------------------------------------------
RFC PATCH v3 15/20 configure: add --enable-pypi and --disable-pypi
In reply to John SnowIn the event that there's no vendored source present and no sufficient version of meson is found, we will attempt to connect to PyPI to install the package ... only if '--disable-pypi' was not passed. This mechanism is intended to replace the online functionality of the meson git submodule.
While --enable-pypi is the default, vendored source will always be preferred when found, making PyPI a fallback. This should ensure that configure-time venv building "just works" for almost everyone in almost every circumstance.
--- configure | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
configure +14 -1
@@ -623,6 +623,7 @@ check_py_version() { } python= +pypi="enabled" first_python= if test -z "${PYTHON}"; then # A bare 'python' is traditionally python 2.x, but some distros
@@ -883,6 +884,10 @@ for opt do --with-git-submodules=*) git_submodules_action="$optarg" ;; + --disable-pypi) pypi="disabled" + ;; + --enable-pypi) pypi="enabled" + ;; --enable-plugins) if test "$mingw32" = "yes"; then error_exit "TCG plugins not currently supported on Windows platforms" else
@@ -1098,7 +1103,9 @@ target_python=$python # Defaults assumed for now: # - venv is cleared if it exists already; # - venv is allowed to use system packages; -# - all setup is performed **offline**; +# - all setup can be performed offline; +# - missing packages may be fetched from PyPI, +# unless --disable-pypi is passed. # - pip is not installed into the venv when possible, # but ensurepip is called as a fallback when necessary.
@@ -1114,7 +1121,13 @@ fi python="$python -B" +mkvenv_flags="" +if test "$pypi" = "enabled" ; then + mkvenv_flags="--online" +fi + if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \ + $mkvenv_flags \ --dir "${source_path}/python/wheels" \ "meson>=0.61.5" ; then
Signature · 1 line
2.39.2
RFC PATCH v3 16/20 tests: Use configure-provided pyvenv for tests
In reply to John SnowThis patch changes how the avocado tests are provided, ever so slightly. Instead of creating a new testing venv, use the configure-provided 'pyvenv' instead, and install optional packages into that.
Note 1: This doesn't respect the "disable-pypi" setting; the avocado tests remain an unconditional PyPI hit when you run "make check-avocado". Future series may endeavor to offer tighter integration, for now, this is a cheap win that doesn't regress anything.
Note 2: At the time of writing, avocado tests require avocado-framework<90 whereas the qemu.qmp self-tests rely on avocado-framework>=90. This collision is avoided for now because the qemu.git/python/qemu/ code does not need avocado at *runtime*; it does not install avocado-framework as a necessary dependency and is skipped in this circumstance.
Nevertheless, we do want to address that discrepancy in the future so that it will be possible to re-use the same venv for qemu.git/python/qemu self-tests to introduce them to make check as "make check-python".
--- docs/devel/acpi-bits.rst | 6 +++--- docs/devel/testing.rst | 14 +++++++------- .gitlab-ci.d/buildtest.yml | 6 +++--- scripts/ci/org.centos/stream/8/x86_64/test-avocado | 4 ++-- scripts/device-crash-test | 2 +- tests/Makefile.include | 10 +++++----- tests/requirements.txt | 7 +++++-- 7 files changed, 26 insertions(+), 23 deletions(-)
docs/devel/acpi-bits.rst +3 -3
@@ -61,19 +61,19 @@ Under ``tests/avocado/`` as the root we have: :: $ make check-venv (needed only the first time to create the venv) - $ ./tests/venv/bin/avocado run -t acpi tests/avocado + $ ./pyvenv/bin/avocado run -t acpi tests/avocado The above will run all acpi avocado tests including this one. In order to run the individual tests, perform the following: :: - $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py --tap - + $ ./pyvenv/bin/avocado run tests/avocado/acpi-bits.py --tap - The above will produce output in tap format. You can omit "--tap -" in the end and it will produce output like the following: :: - $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py + $ ./pyvenv/bin/avocado run tests/avocado/acpi-bits.py Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits JOB ID : eab225724da7b64c012c65705dc2fa14ab1defef JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-10T17.58-eab2257/job.log
docs/devel/testing.rst +7 -7
@@ -882,9 +882,9 @@ You can run the avocado tests simply by executing: make check-avocado -This involves the automatic creation of Python virtual environment -within the build tree (at ``tests/venv``) which will have all the -right dependencies, and will save tests results also within the +This involves the automatic installation, from PyPI, of all the +necessary avocado-framework dependencies into the QEMU venv within the +build tree (at ``./pyvenv``). Test results are also saved within the build tree (at ``tests/results``). Note: the build environment must be using a Python 3 stack, and have
@@ -941,7 +941,7 @@ may be invoked by running: .. code:: - tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/ + pyvenv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/ Note that if ``make check-avocado`` was not executed before, it is possible to create the Python virtual environment with the dependencies
@@ -956,20 +956,20 @@ a test file. To run tests from a single file within the build tree, use: .. code:: - tests/venv/bin/avocado run tests/avocado/$TESTFILE + pyvenv/bin/avocado run tests/avocado/$TESTFILE To run a single test within a test file, use: .. code:: - tests/venv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME + pyvenv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME Valid test names are visible in the output from any previous execution of Avocado or ``make check-avocado``, and can also be queried using: .. code:: - tests/venv/bin/avocado list tests/avocado + pyvenv/bin/avocado list tests/avocado Manual Installation ~~~~~~~~~~~~~~~~~~~
.gitlab-ci.d/buildtest.yml +3 -3
@@ -103,7 +103,7 @@ crash-test-debian: script: - cd build - make check-venv - - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-i386 + - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-i386 build-system-fedora: extends:
@@ -146,8 +146,8 @@ crash-test-fedora: script: - cd build - make check-venv - - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-ppc - - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-riscv32 + - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-ppc + - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-riscv32 build-system-centos: extends:
scripts/ci/org.centos/stream/8/x86_64/test-avocado +2 -2
@@ -4,7 +4,7 @@ # KVM and x86_64, or tests that are generic enough to be valid for all # targets. Such a test list can be generated with: # -# ./tests/venv/bin/avocado list --filter-by-tags-include-empty \ +# ./pyvenv/bin/avocado list --filter-by-tags-include-empty \ # --filter-by-tags-include-empty-key -t accel:kvm,arch:x86_64 \ # tests/avocado/ #
@@ -22,7 +22,7 @@ # - tests/avocado/virtio_check_params.py:VirtioMaxSegSettingsCheck.test_machine_types # make get-vm-images -./tests/venv/bin/avocado run \ +./pyvenv/bin/avocado run \ --job-results-dir=tests/results/ \ tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_i440fx_kvm \ tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_q35_kvm \
scripts/device-crash-test +1 -1
@@ -43,7 +43,7 @@ except ModuleNotFoundError as exc: print(f"Module '{exc.name}' not found.") print(" Try 'make check-venv' from your build directory,") print(" and then one way to run this script is like so:") - print(f' > $builddir/tests/venv/bin/python3 "{path}"') + print(f' > $builddir/pyvenv/bin/python3 "{path}"') sys.exit(1) logger = logging.getLogger('device-crash-test')
tests/Makefile.include +5 -5
@@ -89,7 +89,8 @@ distclean-tcg: $(DISTCLEAN_TCG_TARGET_RULES) # Build up our target list from the filtered list of ninja targets TARGETS=$(patsubst libqemu-%.fa, %, $(filter libqemu-%.fa, $(ninja-targets))) -TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv +TESTS_VENV_DIR=$(BUILD_DIR)/pyvenv +TESTS_VENV_TOKEN=$(BUILD_DIR)/pyvenv/tests.group TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results TESTS_PYTHON=$(TESTS_VENV_DIR)/bin/python3
@@ -111,8 +112,7 @@ quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \ $(TESTS_PYTHON) -m pip -q --disable-pip-version-check $1, \ "VENVPIP","$1") -$(TESTS_VENV_DIR): $(TESTS_VENV_REQ) - $(call quiet-command, $(PYTHON) -m venv $@, VENV, $@) +$(TESTS_VENV_TOKEN): $(TESTS_VENV_REQ) $(call quiet-venv-pip,install -e "$(SRC_PATH)/python/") $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ)) $(call quiet-command, touch $@)
@@ -121,7 +121,7 @@ $(TESTS_RESULTS_DIR): $(call quiet-command, mkdir -p $@, \ MKDIR, $@) -check-venv: $(TESTS_VENV_DIR) +check-venv: $(TESTS_VENV_TOKEN) FEDORA_31_ARCHES_TARGETS=$(patsubst %-softmmu,%, $(filter %-softmmu,$(TARGETS))) FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(FEDORA_31_ARCHES_TARGETS))
@@ -163,7 +163,7 @@ check: check-build: run-ninja check-clean: - rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR) + rm -rf $(TESTS_RESULTS_DIR) clean: check-clean clean-tcg distclean: distclean-tcg
tests/requirements.txt +5 -2
@@ -1,6 +1,9 @@ # Add Python module requirements, one per line, to be installed -# in the tests/venv Python virtual environment. For more info, +# in the qemu build_dir/pyvenv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 -# Note that qemu.git/python/ is always implicitly installed. +# +# Note that qemu.git/python/ is implicitly installed to this venv when +# 'make check-venv' is run, and will persist until configure is run +# again. avocado-framework==88.1 pycdlib==1.11.0
Signature · 1 line
2.39.2
RFC PATCH v3 17/20 configure: move --enable-docs and --disable-docs back to configure
In reply to John SnowMove this option back from meson into configure for the purposes of using the configuration value to bootstrap Sphinx in different ways based on this value.
--- configure | 6 ++++++ 1 file changed, 6 insertions(+)
configure +6 -0
@@ -279,6 +279,7 @@ debug_tcg="no" sanitizers="no" tsan="no" fortify_source="yes" +docs="auto" EXESUF="" modules="no" prefix="/usr/local"
@@ -752,6 +753,10 @@ for opt do ;; --disable-debug-info) meson_option_add -Ddebug=false ;; + --enable-docs) docs=enabled + ;; + --disable-docs) docs=disabled + ;; --enable-modules) modules="yes" ;;
@@ -2638,6 +2643,7 @@ if test "$skip_meson" = no; then # QEMU options test "$cfi" != false && meson_option_add "-Dcfi=$cfi" + test "$docs" != auto && meson_option_add "-Ddocs=$docs" test "$fdt" != auto && meson_option_add "-Dfdt=$fdt" test -n "${LIB_FUZZING_ENGINE+xxx}" && meson_option_add "-Dfuzzing_engine=$LIB_FUZZING_ENGINE" test "$qemu_suffix" != qemu && meson_option_add "-Dqemu_suffix=$qemu_suffix"
Signature · 1 line
2.39.2
RFC PATCH v3 18/20 mkvenv: add diagnose() method for ensure() failures
In reply to John SnowThis is a routine that is designed to print some usable info for human beings back out to the terminal if/when "mkvenv ensure" fails to locate or install a package during configure time, such as meson or sphinx.
Since we are requiring that "meson" and "sphinx" are installed to the same Python environment as QEMU is configured to build with, this can produce some surprising failures when things are mismatched. This method is here to try and ease that sting by offering some actionable diagnosis.
--- python/scripts/mkvenv.py | 153 +++++++++++++++++++++++++++++++++++---- 1 file changed, 140 insertions(+), 13 deletions(-)
python/scripts/mkvenv.py +140 -13
@@ -64,6 +64,7 @@ import os from pathlib import Path import re +import shutil import site import stat import subprocess
@@ -543,6 +544,103 @@ def checkpip() -> None: logging.debug("Pip is now (hopefully) repaired!") +def diagnose( + dep_spec: str, + online: bool, + wheels_dir: Optional[Union[str, Path]], + prog: Optional[str], +) -> str: + """ + Offer a summary to the user as to why a package failed to be installed. + + :param dep_spec: The package we tried to ensure, e.g. 'meson>=0.61.5' + :param online: Did we allow PyPI access? + :param prog: + Optionally, a shell program name that can be used as a + bellwether to detect if this program is installed elsewhere on + the system. This is used to offer advice when a program is + detected for a different python version. + :param wheels_dir: + Optionally, a directory that was searched for vendored packages. + """ + # pylint: disable=too-many-branches + + # Parse name out of PEP-508 depspec. + # See https://peps.python.org/pep-0508/#names + match = re.match( + r"^([A-Z0-9]([A-Z0-9._-]*[A-Z0-9])?)", dep_spec, re.IGNORECASE + ) + if not match: + raise ValueError( + f"dep_spec '{dep_spec}'" + " does not appear to contain a valid package name" + ) + pkg_name = match.group(0) + pkg_version = None + + has_importlib = False + try: + # Python 3.8+ stdlib + # pylint: disable=import-outside-toplevel + from importlib.metadata import PackageNotFoundError, version + + has_importlib = True + try: + pkg_version = version(pkg_name) + except PackageNotFoundError: + pass + except ModuleNotFoundError: + pass + + lines = [] + + if pkg_version: + lines.append( + f"Python package '{pkg_name}' version '{pkg_version}' was found," + " but isn't suitable." + ) + elif has_importlib: + lines.append( + f"Python package '{pkg_name}' was not found nor installed." + ) + else: + lines.append( + f"Python package '{pkg_name}' is either not found or" + " not a suitable version." + ) + + if wheels_dir: + lines.append( + "No suitable version found in, or failed to install from" + f" '{wheels_dir}'." + ) + else: + lines.append("No local package directory was searched.") + + if online: + lines.append("A suitable version could not be obtained from PyPI.") + else: + lines.append( + "mkvenv was configured to operate offline and did not check PyPI." + ) + + if prog and not pkg_version: + which = shutil.which(prog) + if which: + pypath = Path(sys.executable).resolve() + lines.append( + f"'{prog}' was detected on your system at '{which}', " + f"but the Python package '{pkg_name}' was not found by this " + f"Python interpreter ('{pypath}'). " + f"Typically this means that '{prog}' has been installed " + "against a different Python interpreter on your system." + ) + + lines = [f" • {line}" for line in lines] + lines.insert(0, f"Could not ensure availability of '{dep_spec}':") + return os.linesep.join(lines) + + def pip_install( args: Sequence[str], online: bool = False,
@@ -573,23 +671,11 @@ def pip_install( subprocess.run(full_args, check=True) -def ensure( +def _do_ensure( dep_spec: str, online: bool = False, wheels_dir: Optional[Union[str, Path]] = None, ) -> None: - """ - Use pip to ensure we have the package specified by @dep_spec. - - If the package is already installed, do nothing. If online and - wheels_dir are both provided, prefer packages found in wheels_dir - first before connecting to PyPI. - - :param dep_spec: - A PEP 508 dependency specification. e.g. 'meson>=0.61.5'. - :param online: If True, fall back to PyPI. - :param wheels_dir: If specified, search this path for packages. - """ # This first install command will: # (A) Do nothing, if we already have a suitable package. # (B) Install the package from vendored source, if possible.
@@ -603,11 +689,42 @@ def ensure( # The package is missing or isn't a suitable version, # and we weren't able to install a suitable vendored package. if online: + logger.info("offline ensure failed, trying PyPI ...") pip_install([dep_spec], online=True) else: raise +def ensure( + dep_spec: str, + online: bool = False, + wheels_dir: Optional[Union[str, Path]] = None, + prog: Optional[str] = None, +) -> None: + """ + Use pip to ensure we have the package specified by @dep_spec. + + If the package is already installed, do nothing. If online and + wheels_dir are both provided, prefer packages found in wheels_dir + first before connecting to PyPI. + + :param dep_spec: + A PEP 508 dependency specification. e.g. 'meson>=0.61.5'. + :param online: If True, fall back to PyPI. + :param wheels_dir: If specified, search this path for packages. + :param prog: + If specified, use this program name for error diagnostics that will + be presented to the user. e.g., 'sphinx-build' can be used as a + bellwether for the presence of 'sphinx'. + """ + print(f"MKVENV ensure {dep_spec}", file=sys.stderr) + try: + _do_ensure(dep_spec, online, wheels_dir) + except subprocess.CalledProcessError as exc: + # Well, that's not good. + raise Ouch(diagnose(dep_spec, online, wheels_dir, prog)) from exc + + def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None: """ This is intended to be run *inside the venv* after it is created.
@@ -671,6 +788,15 @@ def _add_ensure_subcommand(subparsers: Any) -> None: action="store", help="Path to vendored packages where we may install from.", ) + subparser.add_argument( + "--diagnose", + type=str, + action="store", + help=( + "Name of a shell utility to use for " + "diagnostics if this command fails." + ), + ) subparser.add_argument( "dep_spec", type=str,
@@ -727,6 +853,7 @@ def _normalize_gen() -> None: dep_spec=args.dep_spec, online=args.online, wheels_dir=args.dir, + prog=args.diagnose, ) logger.debug("mkvenv.py %s: exiting", args.command) except Ouch as exc:
Signature · 1 line
2.39.2
RFC PATCH v3 19/20 configure: use --diagnose option with meson ensure
In reply to John SnowRely on the diagnosis information from mkvenv so it can be removed from the configure script.
--- configure | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-)
configure +1 -10
@@ -1096,8 +1096,6 @@ fi # Resolve PATH python="$(command -v "$python")" -# This variable is intended to be used only for error messages: -target_python=$python # Create a Python virtual environment using our configured python. # The stdout of this script will be the location of a symlink that
@@ -1134,16 +1132,9 @@ fi if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \ $mkvenv_flags \ --dir "${source_path}/python/wheels" \ + --diagnose "meson" \ "meson>=0.61.5" ; then - # We're very out of luck. Try to give a good diagnostic. - if test -e pyvenv/bin/meson; then - echo "Meson is too old: $(pyvenv/bin/meson --version)" - elif has meson ; then - echo "Meson was found installed on your system," \ - "but not for the configured Python interpreter ($target_python)." - echo "(Hint: check '$(which meson)' to see which interpreter its shebang uses.)" - fi exit 1 fi
Signature · 1 line
2.39.2
RFC PATCH v3 20/20 configure: bootstrap sphinx with mkvenv
In reply to John SnowWhen docs are explicitly requested, require Sphinx>=1.6.0. When docs are explicitly disabled, don't bother to check for Sphinx at all. If docs are set to "auto", attempt to locate Sphinx, but continue onward if it wasn't located.
For the case that --enable-pypi is set to 'enabled' (the default) but docs are set to 'auto' (also the default), do not actually consult PyPI to install Sphinx. Only perform this action when docs are requested explicitly.
--- configure | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-)
configure +25 -2
@@ -1122,14 +1122,14 @@ fi # Suppress writing compiled files python="$python -B" - +mkvenv="$python ${source_path}/python/scripts/mkvenv.py" mkvenv_flags="" if test "$pypi" = "enabled" ; then mkvenv_flags="--online" fi -if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \ +if ! $mkvenv ensure \ $mkvenv_flags \ --dir "${source_path}/python/wheels" \ --diagnose "meson" \
@@ -1144,6 +1144,29 @@ fi # *exclusively*. meson="$(cd pyvenv/bin; pwd)/meson" +# Conditionally ensure Sphinx is installed. + +mkvenv_flags="" +if test "$pypi" = "enabled" -a "$docs" = "enabled" ; then + mkvenv_flags="--online" +fi + +if test "$docs" != "disabled" ; then + if ! $mkvenv ensure \ + $mkvenv_flags \ + --diagnose "sphinx-build" \ + "sphinx>=1.6.0" ; + then + if test "$docs" = "enabled" ; then + exit 1 + fi + echo "Sphinx not found/usable, disabling docs." + docs=disabled + else + docs=enabled + fi +fi + echo "MKVENV ok!" # Probe for ninja
Signature · 1 line
2.39.2
Quoted 96 lines · John Snow
GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
(All green, except Python self-tests, see below)This patch series creates a mandatory python virtual environment ("venv") during configure time and uses it to ensure the availability of meson and sphinx.
See https://www.qemu.org/2023/03/24/python/ for details. The summary is that the goal of this series is to ensure that the
pythonused to run meson is the samepythonused to run Sphinx, tests, and any build-time python scripting we have. As it stands, meson and sphinx (and their extensions) *may* run in a different python environment than the one configured and chosen by the user at configure/build time.The effective change of this series is that QEMU will now unconditionally create a venv at configure-time and will ensure that meson (and sphinx, if docs are enabled) are available through that venv.
Some important points as a pre-emptive "FAQ":
- This venv is unconditionally created and lives at {build_dir}/pyvenv.
- The python interpreter used by this venv is always the one identified by configure. (Which in turn is always the one specified by --python or $PYTHON)
- *almost* all python scripts in qemu.git executed as part of the build system, meson, sphinx, avocado tests, vm tests or CI are always executed within this venv.
(iotests are not yet integrated; I plan to tackle this separately as a follow-up in order to have a more tightly focused scope on that series.)
- It remains possible to build and test fully offline.
(In most cases, you just need meson and sphinx from your distro's repo.)- Distribution packaged 'meson' and 'sphinx' are still utilized whenever possible as the highest preference.
- Vendored versions of e.g. 'meson' are always preferred to PyPI versions for speed, repeatability and ensuring tarball builds work as-is offline.
(Sphinx will not be vendored, just like it already isn't.)
- Missing dependencies, when possible, are fetched and installed on-demand automatically to make developer environments "just work".
- Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
- No new dependencies (...for most platforms. Debian and NetBSD get an asterisk.)
- The meson git submodule is unused after this series and can be removed.
For reviewers, here's how the series is broken up:
Patch 1 is a testing pre-req. Note that even with this patch, 'check-python-minreqs' and 'check-python-tox' CI jobs will both still fail on origin/master because this series requires 3.7+, but origin/master is currently still 3.6+.
- python: update pylint configuration
Patches 2-8 add the mkvenv script. The first patch checks in the barest essentials, and each subsequent patch adds a workaround or feature one at a time.
- python: add
mkvenv.py
- mkvenv: add console script entry point generation
- mkvenv: Add better error message for missing pyexapt module
- mkvenv: generate console entry shims from inside the venv
- mkvenv: work around broken pip installations on Debian 10
- mkvenv: add nested venv workaround
- mkvenv: add ensure subcommandPatches 9-11 modify our testing configuration to add new dependencies as needed.
- tests/docker: add python3-venv dependency
- tests/vm: Configure netbsd to use Python 3.10
- tests/vm: add py310-expat to NetBSDPatch 12 changes how we package release tarballs.
-
scripts/make-release: download meson==0.61.5 .whlPatches 13-16 wire mkvenv into configure and tests.
- configure: create a python venv unconditionally
- configure: use 'mkvenv ensure meson' to bootstrap meson
- configure: add --enable-pypi and --disable-pypi
- tests: Use configure-provided pyvenv for tests
Quoted 8 lines
Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these changes could be folded earlier in the series (like the
diagnose()patch), but I'm keeping it separate for review for now.- configure: move --enable-docs and --disable-docs back to configure
- mkvenv: adddiagnose()method forensure()failures
- configure: use --diagnose option with meson ensure
- configure: bootstrap sphinx with mkvenv
I'm not sure this last bit is working.
I uninstalled meson and python3-sphinx from my F38 host and ran configure --target-list=x86_64-softmmu and got this:
$ ./configure --target-list=x86_64-softmmu Using './build' as the directory for build output python determined to be '/usr/bin/python3' python version: Python 3.11.3 MKVENV pyvenv Configured python as '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B' MKVENV ensure meson>=0.61.5 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is ignored: it is neither a file nor a directory. ERROR: Could not find a version that satisfies the requirement meson>=0.61.5 (from versions: none) ERROR: No matching distribution found for meson>=0.61.5 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' MKVENV ensure sphinx>=1.6.0 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' ERROR: Could not find a version that satisfies the requirement sphinx>=1.6.0 (from versions: none) ERROR: No matching distribution found for sphinx>=1.6.0
*** Ouch! ***
Could not ensure availability of 'sphinx>=1.6.0':
• Python package 'sphinx' was not found nor installed.
• No local package directory was searched.
• mkvenv was configured to operate offline and did not check PyPI.
Sphinx not found/usable, disabling docs.
MKVENV ok!
It says mkvenv was configured to run offline, but I didn't do anything. I thought the intention was for developers it would live download from PyPI ?
On a system where i already have meson/sphinx installed, it all just worked fine AFAICT.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 159 lines · Daniel P. Berrangé
Quoted 30 lines · John Snow
GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
(All green, except Python self-tests, see below)This patch series creates a mandatory python virtual environment ("venv") during configure time and uses it to ensure the availability of meson and sphinx.
See https://www.qemu.org/2023/03/24/python/ for details. The summary is that the goal of this series is to ensure that the
pythonused to run meson is the samepythonused to run Sphinx, tests, and any build-time python scripting we have. As it stands, meson and sphinx (and their extensions) *may* run in a different python environment than the one configured and chosen by the user at configure/build time.The effective change of this series is that QEMU will now unconditionally create a venv at configure-time and will ensure that meson (and sphinx, if docs are enabled) are available through that venv.
Some important points as a pre-emptive "FAQ":
- This venv is unconditionally created and lives at {build_dir}/pyvenv.
- The python interpreter used by this venv is always the one identified by configure. (Which in turn is always the one specified by --python or $PYTHON)
- *almost* all python scripts in qemu.git executed as part of the build system, meson, sphinx, avocado tests, vm tests or CI are always executed within this venv.
Quoted 6 lines
(iotests are not yet integrated; I plan to tackle this separately as a follow-up in order to have a more tightly focused scope on that series.)
- It remains possible to build and test fully offline. (In most cases, you just need meson and sphinx from your distro's
repo.)
Quoted 60 lines
- Distribution packaged 'meson' and 'sphinx' are still utilized whenever possible as the highest preference.
- Vendored versions of e.g. 'meson' are always preferred to PyPI versions for speed, repeatability and ensuring tarball builds work as-is offline.
(Sphinx will not be vendored, just like it already isn't.)
- Missing dependencies, when possible, are fetched and installed on-demand automatically to make developer environments "just work".
- Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
- No new dependencies (...for most platforms. Debian and NetBSD get an asterisk.)
- The meson git submodule is unused after this series and can be removed.
For reviewers, here's how the series is broken up:
Patch 1 is a testing pre-req. Note that even with this patch, 'check-python-minreqs' and 'check-python-tox' CI jobs will both still fail on origin/master because this series requires 3.7+, but origin/master is currently still 3.6+.
- python: update pylint configuration
Patches 2-8 add the mkvenv script. The first patch checks in the barest essentials, and each subsequent patch adds a workaround or feature one at a time.
- python: add
mkvenv.py
- mkvenv: add console script entry point generation
- mkvenv: Add better error message for missing pyexapt module
- mkvenv: generate console entry shims from inside the venv
- mkvenv: work around broken pip installations on Debian 10
- mkvenv: add nested venv workaround
- mkvenv: add ensure subcommandPatches 9-11 modify our testing configuration to add new dependencies as needed.
- tests/docker: add python3-venv dependency
- tests/vm: Configure netbsd to use Python 3.10
- tests/vm: add py310-expat to NetBSDPatch 12 changes how we package release tarballs.
-
scripts/make-release: download meson==0.61.5 .whlPatches 13-16 wire mkvenv into configure and tests.
- configure: create a python venv unconditionally
- configure: use 'mkvenv ensure meson' to bootstrap meson
- configure: add --enable-pypi and --disable-pypi
- tests: Use configure-provided pyvenv for testsQuoted 8 lines
Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these changes could be folded earlier in the series (like the
diagnose()patch), but I'm keeping it separate for review for now.- configure: move --enable-docs and --disable-docs back to configure
- mkvenv: adddiagnose()method forensure()failures
- configure: use --diagnose option with meson ensure
- configure: bootstrap sphinx with mkvenvI'm not sure this last bit is working.
I uninstalled meson and python3-sphinx from my F38 host and ran configure --target-list=x86_64-softmmu and got this:
$ ./configure --target-list=x86_64-softmmu Using './build' as the directory for build output python determined to be '/usr/bin/python3' python version: Python 3.11.3 MKVENV pyvenv Configured python as '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B' MKVENV ensure meson>=0.61.5 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is ignored: it is neither a file nor a directory. ERROR: Could not find a version that satisfies the requirement meson>=0.61.5 (from versions: none) ERROR: No matching distribution found for meson>=0.61.5 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' MKVENV ensure sphinx>=1.6.0 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' ERROR: Could not find a version that satisfies the requirement sphinx>=1.6.0 (from versions: none) ERROR: No matching distribution found for sphinx>=1.6.0*** Ouch! ***
Could not ensure availability of 'sphinx>=1.6.0':
• Python package 'sphinx' was not found nor installed.
• No local package directory was searched.
• mkvenv was configured to operate offline and did not check PyPI.
Quoted 9 lines
Sphinx not found/usable, disabling docs.
MKVENV ok!It says mkvenv was configured to run offline, but I didn't do anything. I thought the intention was for developers it would live download from PyPI ?
Ah. So... with enable pypi being the default and with docs set to "auto", I actually fall back to not installing sphinx from pypi *by default*. It tries to locate it on your system and will enable docs if it can, but it doesn't try too hard and doesn't get upset if it fails.
Try passing --enable-docs to convince the build system you'd really definitely like docs, and it'll force the pypi access.
Quoted 15 lines
On a system where i already have meson/sphinx installed, it all just worked fine AFAICT.
With regards,
Daniel
--
|: https://berrange.com -o-
https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-
https://fstop138.berrange.com :|
|: https://entangle-photo.org -o-
https://www.instagram.com/dberrange :|
Quoted 175 lines · John Snow
Quoted 160 lines · Daniel P. Berrangé
Quoted 30 lines · John Snow
GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
(All green, except Python self-tests, see below)This patch series creates a mandatory python virtual environment ("venv") during configure time and uses it to ensure the availability of meson and sphinx.
See https://www.qemu.org/2023/03/24/python/ for details. The summary is that the goal of this series is to ensure that the
pythonused to run meson is the samepythonused to run Sphinx, tests, and any build-time python scripting we have. As it stands, meson and sphinx (and their extensions) *may* run in a different python environment than the one configured and chosen by the user at configure/build time.The effective change of this series is that QEMU will now unconditionally create a venv at configure-time and will ensure that meson (and sphinx, if docs are enabled) are available through that venv.
Some important points as a pre-emptive "FAQ":
- This venv is unconditionally created and lives at {build_dir}/pyvenv.
- The python interpreter used by this venv is always the one identified by configure. (Which in turn is always the one specified by --python or $PYTHON)
- *almost* all python scripts in qemu.git executed as part of the build system, meson, sphinx, avocado tests, vm tests or CI are always executed within this venv.
Quoted 6 lines
(iotests are not yet integrated; I plan to tackle this separately as a follow-up in order to have a more tightly focused scope on that series.)
- It remains possible to build and test fully offline. (In most cases, you just need meson and sphinx from your distro's
repo.)
Quoted 13 lines
- Distribution packaged 'meson' and 'sphinx' are still utilized whenever possible as the highest preference.
- Vendored versions of e.g. 'meson' are always preferred to PyPI versions for speed, repeatability and ensuring tarball builds work as-is offline.
(Sphinx will not be vendored, just like it already isn't.)
- Missing dependencies, when possible, are fetched and installed on-demand automatically to make developer environments "just work".
Quoted 7 lines
- Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
- No new dependencies (...for most platforms. Debian and NetBSD get an asterisk.)
- The meson git submodule is unused after this series and can be
removed.
Quoted 40 lines
For reviewers, here's how the series is broken up:
Patch 1 is a testing pre-req. Note that even with this patch, 'check-python-minreqs' and 'check-python-tox' CI jobs will both still fail on origin/master because this series requires 3.7+, but origin/master is currently still 3.6+.
- python: update pylint configuration
Patches 2-8 add the mkvenv script. The first patch checks in the barest essentials, and each subsequent patch adds a workaround or feature one at a time.
- python: add
mkvenv.py
- mkvenv: add console script entry point generation
- mkvenv: Add better error message for missing pyexapt module
- mkvenv: generate console entry shims from inside the venv
- mkvenv: work around broken pip installations on Debian 10
- mkvenv: add nested venv workaround
- mkvenv: add ensure subcommandPatches 9-11 modify our testing configuration to add new dependencies as needed.
- tests/docker: add python3-venv dependency
- tests/vm: Configure netbsd to use Python 3.10
- tests/vm: add py310-expat to NetBSDPatch 12 changes how we package release tarballs.
-
scripts/make-release: download meson==0.61.5 .whlPatches 13-16 wire mkvenv into configure and tests.
- configure: create a python venv unconditionally
- configure: use 'mkvenv ensure meson' to bootstrap meson
- configure: add --enable-pypi and --disable-pypi
- tests: Use configure-provided pyvenv for testsQuoted 8 lines
Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these changes could be folded earlier in the series (like the
diagnose()patch), but I'm keeping it separate for review for now.- configure: move --enable-docs and --disable-docs back to configure
- mkvenv: adddiagnose()method forensure()failures
- configure: use --diagnose option with meson ensure
- configure: bootstrap sphinx with mkvenvI'm not sure this last bit is working.
I uninstalled meson and python3-sphinx from my F38 host and ran configure --target-list=x86_64-softmmu and got this:
$ ./configure --target-list=x86_64-softmmu Using './build' as the directory for build output python determined to be '/usr/bin/python3' python version: Python 3.11.3 MKVENV pyvenv Configured python as '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B' MKVENV ensure meson>=0.61.5 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is ignored: it is neither a file nor a directory. ERROR: Could not find a version that satisfies the requirement meson>=0.61.5 (from versions: none) ERROR: No matching distribution found for meson>=0.61.5 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' MKVENV ensure sphinx>=1.6.0 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' ERROR: Could not find a version that satisfies the requirement sphinx>=1.6.0 (from versions: none) ERROR: No matching distribution found for sphinx>=1.6.0*** Ouch! ***
Could not ensure availability of 'sphinx>=1.6.0':
• Python package 'sphinx' was not found nor installed.
• No local package directory was searched.
• mkvenv was configured to operate offline and did not check PyPI.Quoted 9 lines
Sphinx not found/usable, disabling docs.
MKVENV ok!It says mkvenv was configured to run offline, but I didn't do anything. I thought the intention was for developers it would live download from PyPI ?
Quoted 5 lines
Ah. So... with enable pypi being the default and with docs set to "auto", I actually fall back to not installing sphinx from pypi *by default*. It tries to locate it on your system and will enable docs if it can, but it doesn't try too hard and doesn't get upset if it fails.
(Though all of those errors and warnings sure are noisy for meaning "we couldn't find an optional package". mkvenv just doesn't distinguish between optional and required right now so it just leaves it to the caller to interpret. Any suggestions for improving this?)
Quoted 3 lines
Try passing --enable-docs to convince the build system you'd really definitely like docs, and it'll force the pypi access.
I can make it try PyPI in this double-default case too, I was just being very conservative about when we tried PyPI - this solution is fairly reluctant by design to do it.
If we all agree that it shouldn't be so reluctant, and anyone who doesnt want it at all should just pass --disable-pypi, I can make that change easily.
I wonder if I should make an "auto" setting for the pypi access which represents this behavior ("only if I have to"), and make "enable" more aggressive (consult pypi even for optional features.)
I was just being conservative on the first pass. Trying to strike a balance between convenience, speed, and least surprise.
Quoted 16 lines
Quoted 16 lines
On a system where i already have meson/sphinx installed, it all just worked fine AFAICT.
With regards,
Daniel
--
|: https://berrange.com -o-
https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-
https://fstop138.berrange.com :|
|: https://entangle-photo.org -o-
https://www.instagram.com/dberrange :|
Quoted 201 lines · John Snow
Quoted 175 lines · John Snow
Quoted 160 lines · Daniel P. Berrangé
Quoted 30 lines · John Snow
GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
(All green, except Python self-tests, see below)This patch series creates a mandatory python virtual environment ("venv") during configure time and uses it to ensure the availability of meson and sphinx.
See https://www.qemu.org/2023/03/24/python/ for details. The summary is that the goal of this series is to ensure that the
pythonused to run meson is the samepythonused to run Sphinx, tests, and any build-time python scripting we have. As it stands, meson and sphinx (and their extensions) *may* run in a different python environment than the one configured and chosen by the user at configure/build time.The effective change of this series is that QEMU will now unconditionally create a venv at configure-time and will ensure that meson (and sphinx, if docs are enabled) are available through that venv.
Some important points as a pre-emptive "FAQ":
- This venv is unconditionally created and lives at {build_dir}/pyvenv.
- The python interpreter used by this venv is always the one identified by configure. (Which in turn is always the one specified by --python or $PYTHON)
- *almost* all python scripts in qemu.git executed as part of the build system, meson, sphinx, avocado tests, vm tests or CI are always executed within this venv.
Quoted 6 lines
(iotests are not yet integrated; I plan to tackle this separately as a follow-up in order to have a more tightly focused scope on that series.)
- It remains possible to build and test fully offline. (In most cases, you just need meson and sphinx from your distro's
repo.)
Quoted 13 lines
- Distribution packaged 'meson' and 'sphinx' are still utilized whenever possible as the highest preference.
- Vendored versions of e.g. 'meson' are always preferred to PyPI versions for speed, repeatability and ensuring tarball builds work as-is offline.
(Sphinx will not be vendored, just like it already isn't.)
- Missing dependencies, when possible, are fetched and installed on-demand automatically to make developer environments "just work".
Quoted 7 lines
- Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
- No new dependencies (...for most platforms. Debian and NetBSD get an asterisk.)
- The meson git submodule is unused after this series and can be
removed.
Quoted 40 lines
For reviewers, here's how the series is broken up:
Patch 1 is a testing pre-req. Note that even with this patch, 'check-python-minreqs' and 'check-python-tox' CI jobs will both still fail on origin/master because this series requires 3.7+, but origin/master is currently still 3.6+.
- python: update pylint configuration
Patches 2-8 add the mkvenv script. The first patch checks in the barest essentials, and each subsequent patch adds a workaround or feature one at a time.
- python: add
mkvenv.py
- mkvenv: add console script entry point generation
- mkvenv: Add better error message for missing pyexapt module
- mkvenv: generate console entry shims from inside the venv
- mkvenv: work around broken pip installations on Debian 10
- mkvenv: add nested venv workaround
- mkvenv: add ensure subcommandPatches 9-11 modify our testing configuration to add new dependencies as needed.
- tests/docker: add python3-venv dependency
- tests/vm: Configure netbsd to use Python 3.10
- tests/vm: add py310-expat to NetBSDPatch 12 changes how we package release tarballs.
-
scripts/make-release: download meson==0.61.5 .whlPatches 13-16 wire mkvenv into configure and tests.
- configure: create a python venv unconditionally
- configure: use 'mkvenv ensure meson' to bootstrap meson
- configure: add --enable-pypi and --disable-pypi
- tests: Use configure-provided pyvenv for testsQuoted 8 lines
Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these changes could be folded earlier in the series (like the
diagnose()patch), but I'm keeping it separate for review for now.- configure: move --enable-docs and --disable-docs back to configure
- mkvenv: adddiagnose()method forensure()failures
- configure: use --diagnose option with meson ensure
- configure: bootstrap sphinx with mkvenvI'm not sure this last bit is working.
I uninstalled meson and python3-sphinx from my F38 host and ran configure --target-list=x86_64-softmmu and got this:
$ ./configure --target-list=x86_64-softmmu Using './build' as the directory for build output python determined to be '/usr/bin/python3' python version: Python 3.11.3 MKVENV pyvenv Configured python as '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B' MKVENV ensure meson>=0.61.5 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is ignored: it is neither a file nor a directory. ERROR: Could not find a version that satisfies the requirement meson>=0.61.5 (from versions: none) ERROR: No matching distribution found for meson>=0.61.5 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' MKVENV ensure sphinx>=1.6.0 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' ERROR: Could not find a version that satisfies the requirement sphinx>=1.6.0 (from versions: none) ERROR: No matching distribution found for sphinx>=1.6.0*** Ouch! ***
Could not ensure availability of 'sphinx>=1.6.0':
• Python package 'sphinx' was not found nor installed.
• No local package directory was searched.
• mkvenv was configured to operate offline and did not check PyPI.Quoted 9 lines
Sphinx not found/usable, disabling docs.
MKVENV ok!It says mkvenv was configured to run offline, but I didn't do anything. I thought the intention was for developers it would live download from PyPI ?
Quoted 5 lines
Ah. So... with enable pypi being the default and with docs set to "auto", I actually fall back to not installing sphinx from pypi *by default*. It tries to locate it on your system and will enable docs if it can, but it doesn't try too hard and doesn't get upset if it fails.
(Though all of those errors and warnings sure are noisy for meaning "we couldn't find an optional package". mkvenv just doesn't distinguish between optional and required right now so it just leaves it to the caller to interpret. Any suggestions for improving this?)
Quoted 3 lines
Try passing --enable-docs to convince the build system you'd really definitely like docs, and it'll force the pypi access.
I can make it try PyPI in this double-default case too, I was just being very conservative about when we tried PyPI - this solution is fairly reluctant by design to do it.
If we all agree that it shouldn't be so reluctant, and anyone who doesnt want it at all should just pass --disable-pypi, I can make that change easily.
Quoted 6 lines
I wonder if I should make an "auto" setting for the pypi access which represents this behavior ("only if I have to"), and make "enable" more aggressive (consult pypi even for optional features.)
I was just being conservative on the first pass. Trying to strike a balance between convenience, speed, and least surprise.
How about having --enable-pypi never|auto|force with the following semantics for --enable-docs + --enable-pypi
* docs=no - pypi never used
* docs=auto + pypi=never => docs only enable if sphinx is already installed locally, otherwise disabled
* docs=auto + pypi=auto => docs enable if sphinx is already installed locally, or can download from pypi as fallback
* docs=auto + pypi=force => always download sphinx from pypi
* docs=yes + pypi=never => ERROR if sphinx is not already installed locally
* docs=yes + pypi=auto => docs enable if sphinx is already installed locally, or can download from pypi as fallback
* docs=yes + pypi=force => always download sphinx from pypi
So eg distros could use pypi=never, devs would use pypi=auto mostly, while CI might use pypi=force to test specific versions indepenant of distros ?
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 214 lines · Daniel P. Berrangé
Quoted 201 lines · John Snow
Quoted 175 lines · John Snow
Quoted 160 lines · Daniel P. Berrangé
Quoted 30 lines · John Snow
GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
(All green, except Python self-tests, see below)This patch series creates a mandatory python virtual environment ("venv") during configure time and uses it to ensure the availability of meson and sphinx.
See https://www.qemu.org/2023/03/24/python/ for details. The summary is that the goal of this series is to ensure that the
pythonused to run meson is the samepythonused to run Sphinx, tests, and any build-time python scripting we have. As it stands, meson and sphinx (and their extensions) *may* run in a different python environment than the one configured and chosen by the user at configure/build time.The effective change of this series is that QEMU will now unconditionally create a venv at configure-time and will ensure that meson (and sphinx, if docs are enabled) are available through that venv.
Some important points as a pre-emptive "FAQ":
- This venv is unconditionally created and lives at {build_dir}/pyvenv.
- The python interpreter used by this venv is always the one identified by configure. (Which in turn is always the one specified by --python or $PYTHON)
- *almost* all python scripts in qemu.git executed as part of the build system, meson, sphinx, avocado tests, vm tests or CI are always executed within this venv.
Quoted 6 lines
(iotests are not yet integrated; I plan to tackle this separately as a follow-up in order to have a more tightly focused scope on that series.)
- It remains possible to build and test fully offline. (In most cases, you just need meson and sphinx from your distro's
repo.)
Quoted 13 lines
- Distribution packaged 'meson' and 'sphinx' are still utilized whenever possible as the highest preference.
- Vendored versions of e.g. 'meson' are always preferred to PyPI versions for speed, repeatability and ensuring tarball builds work as-is offline.
(Sphinx will not be vendored, just like it already isn't.)
- Missing dependencies, when possible, are fetched and installed on-demand automatically to make developer environments "just work".
Quoted 7 lines
- Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
- No new dependencies (...for most platforms. Debian and NetBSD get an asterisk.)
- The meson git submodule is unused after this series and can be
removed.
Quoted 40 lines
For reviewers, here's how the series is broken up:
Patch 1 is a testing pre-req. Note that even with this patch, 'check-python-minreqs' and 'check-python-tox' CI jobs will both still fail on origin/master because this series requires 3.7+, but origin/master is currently still 3.6+.
- python: update pylint configuration
Patches 2-8 add the mkvenv script. The first patch checks in the barest essentials, and each subsequent patch adds a workaround or feature one at a time.
- python: add
mkvenv.py
- mkvenv: add console script entry point generation
- mkvenv: Add better error message for missing pyexapt module
- mkvenv: generate console entry shims from inside the venv
- mkvenv: work around broken pip installations on Debian 10
- mkvenv: add nested venv workaround
- mkvenv: add ensure subcommandPatches 9-11 modify our testing configuration to add new dependencies as needed.
- tests/docker: add python3-venv dependency
- tests/vm: Configure netbsd to use Python 3.10
- tests/vm: add py310-expat to NetBSDPatch 12 changes how we package release tarballs.
-
scripts/make-release: download meson==0.61.5 .whlPatches 13-16 wire mkvenv into configure and tests.
- configure: create a python venv unconditionally
- configure: use 'mkvenv ensure meson' to bootstrap meson
- configure: add --enable-pypi and --disable-pypi
- tests: Use configure-provided pyvenv for testsQuoted 8 lines
Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these changes could be folded earlier in the series (like the
diagnose()patch), but I'm keeping it separate for review for now.- configure: move --enable-docs and --disable-docs back to configure
- mkvenv: adddiagnose()method forensure()failures
- configure: use --diagnose option with meson ensure
- configure: bootstrap sphinx with mkvenvI'm not sure this last bit is working.
I uninstalled meson and python3-sphinx from my F38 host and ran configure --target-list=x86_64-softmmu and got this:
$ ./configure --target-list=x86_64-softmmu Using './build' as the directory for build output python determined to be '/usr/bin/python3' python version: Python 3.11.3 MKVENV pyvenv Configured python as '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B' MKVENV ensure meson>=0.61.5 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is ignored: it is neither a file nor a directory. ERROR: Could not find a version that satisfies the requirement meson>=0.61.5 (from versions: none) ERROR: No matching distribution found for meson>=0.61.5 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' MKVENV ensure sphinx>=1.6.0 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' ERROR: Could not find a version that satisfies the requirement sphinx>=1.6.0 (from versions: none) ERROR: No matching distribution found for sphinx>=1.6.0*** Ouch! ***
Could not ensure availability of 'sphinx>=1.6.0':
• Python package 'sphinx' was not found nor installed.
• No local package directory was searched.
• mkvenv was configured to operate offline and did not check PyPI.Quoted 9 lines
Sphinx not found/usable, disabling docs.
MKVENV ok!It says mkvenv was configured to run offline, but I didn't do anything. I thought the intention was for developers it would live download from PyPI ?
Quoted 5 lines
Ah. So... with enable pypi being the default and with docs set to "auto", I actually fall back to not installing sphinx from pypi *by default*. It tries to locate it on your system and will enable docs if it can, but it doesn't try too hard and doesn't get upset if it fails.
(Though all of those errors and warnings sure are noisy for meaning "we couldn't find an optional package". mkvenv just doesn't distinguish between optional and required right now so it just leaves it to the caller to interpret. Any suggestions for improving this?)
Quoted 3 lines
Try passing --enable-docs to convince the build system you'd really definitely like docs, and it'll force the pypi access.
I can make it try PyPI in this double-default case too, I was just being very conservative about when we tried PyPI - this solution is fairly reluctant by design to do it.
If we all agree that it shouldn't be so reluctant, and anyone who doesnt want it at all should just pass --disable-pypi, I can make that change easily.
Quoted 6 lines
I wonder if I should make an "auto" setting for the pypi access which represents this behavior ("only if I have to"), and make "enable" more aggressive (consult pypi even for optional features.)
I was just being conservative on the first pass. Trying to strike a balance between convenience, speed, and least surprise.
How about having --enable-pypi never|auto|force with the following semantics for --enable-docs + --enable-pypi
Quoted 9 lines
* docs=no - pypi never used
* docs=auto + pypi=never => docs only enable if sphinx is already installed locally, otherwise disabled
* docs=auto + pypi=auto => docs enable if sphinx is already installed locally, or can download from pypi as fallback
As long as this doesn't cause needless trouble for existing configure invocations baked into scripts and the like. It's a bit more aggressive about enabling docs than we have been in the past, but maybe that's not a bad thing.
Quoted 2 lines
* docs=auto + pypi=force => always download sphinx from pypi
So if you already have Sphinx, this should perform an upgrade to the latest version?
Quoted 9 lines
* docs=yes + pypi=never => ERROR if sphinx is not already installed locally
* docs=yes + pypi=auto => docs enable if sphinx is already installed locally, or can download from pypi as fallback
* docs=yes + pypi=force => always download sphinx from pypi
Yeah, there's room for settings like these, and the above mostly makes sense to me.
Paolo and I had written up a more elaborate set of flags at one point under the premise of integrating testing, too -- for instance, pre-loading the testing dependencies for iotests (local qemu package, qemu.qmp, etc) or for avocado tests (avocado-framework and other deps needed for those tests). I wanted to try and check in a "minimum viable" version of this patch set first close to the 8.1 tree opening to iron out any bugs in the core feature and then I'd work to expand the flags and capability of the system as differential patches.
So, I think changing --enable-pypi too much is beyond the scope of the current patchset, but it's on my mind for what will follow almost immediately after it. With that in mind, we can brainstorm a little here and throw some darts at the wall:
The version of flags we originally came up with was this:
--python=... # runtime used to create venv --enable-pip-groups=testing,devel,avocado,meson,sphinx --enable-pip=now # install all python deps now --enable-pip=on-demand # install qemu.git/meson/sphinx, delay the rest
--enable-pip=no # offline
--{enable,disable}-isolated-venv # let venv use system/distro if disable
I may have pulled us in a slightly different direction with the version of the patches I actually sent here today, but the key ideas here were:
- The ability to opt into or out of different package groups at configure-time. meson is always required, docs/sphinx is optional, and the other three groups are testing related. (testing: iotests and qemu.qmp, devel: python self-tests, avocado: avocado tests.)
- The ability to decide when packages get installed to the venv; either at configure time ("now") or as-needed ("on-demand") or functionally never ("no")
- The ability to enable or disable venv isolation (turning on or off system packages).
I think I abandoned the idea of configuring the venv isolation and have it set to "always off". We may revisit this later, but I think for this series it's fine to have glossed over it.
I also skipped over the idea of having package installation being "now", "on-demand" or "no" -- this patch-set more or less is using a hybrid of "now" and "on-demand" where meson and sphinx are "now" but avocado is "on-demand". Unifying that discrepancy can occur in subsequent patches as it closely resembles what we had in 8.0 and earlier.
I also didn't implement configurable package groups at all yet. meson and sphinx are just implicitly selected; everything else is not yet thoroughly integrated.
The testing portions of this -- avocado, qemu.qmp, and python self-tests -- are not as integrated as the build portions, sphinx and meson. That will likely be the focus of my next patchset when I move to integrate iotests and actually set about the work of finally dropping the qemu.qmp package from the qemu.git source tree.
I guess before I spend *too* much time writing that series (for the fourth time...!) I wanted to try and solidify the basics here to give me a nice solid foundation.
--js
Quoted 7 lines
So eg distros could use pypi=never, devs would use pypi=auto mostly, while CI might use pypi=force to test specific versions indepenant of distros ?
Sounds reasonable in general, I think, but I have some questions about what 'force' does, exactly.
Quoted 7 lines
With regards,
Daniel
--|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 20 lines · John Snow
Quoted 4 lines · Daniel P. Berrangé
How about having --enable-pypi never|auto|force with the following semantics for --enable-docs + --enable-pypi
Quoted 9 lines
* docs=no - pypi never used
* docs=auto + pypi=never => docs only enable if sphinx is already installed locally, otherwise disabled
* docs=auto + pypi=auto => docs enable if sphinx is already installed locally, or can download from pypi as fallback
As long as this doesn't cause needless trouble for existing configure invocations baked into scripts and the like. It's a bit more aggressive about enabling docs than we have been in the past, but maybe that's not a bad thing.
Quoted 5 lines
Quoted 2 lines
* docs=auto + pypi=force => always download sphinx from pypi
So if you already have Sphinx, this should perform an upgrade to the latest version?
Essentially I meant 'force' to mean *never* use the host python installation packages. Always install all the deps in the venv, even if they exist in the host with sufficient version met.
Quoted 21 lines
Quoted 8 lines
* docs=yes + pypi=never => ERROR if sphinx is not already installed locally
* docs=yes + pypi=auto => docs enable if sphinx is already installed locally, or can download from pypi as fallback
* docs=yes + pypi=force => always download sphinx from pypi
Yeah, there's room for settings like these, and the above mostly makes sense to me.
Paolo and I had written up a more elaborate set of flags at one point under the premise of integrating testing, too -- for instance, pre-loading the testing dependencies for iotests (local qemu package, qemu.qmp, etc) or for avocado tests (avocado-framework and other deps needed for those tests). I wanted to try and check in a "minimum viable" version of this patch set first close to the 8.1 tree opening to iron out any bugs in the core feature and then I'd work to expand the flags and capability of the system as differential patches.
Quoted 4 lines
So, I think changing --enable-pypi too much is beyond the scope of the current patchset, but it's on my mind for what will follow almost immediately after it. With that in mind, we can brainstorm a little here and throw some darts at the wall:
Yep, I don't consider it a pre-requisite for this series.
Quoted 9 lines
The version of flags we originally came up with was this:
--python=... # runtime used to create venv --enable-pip-groups=testing,devel,avocado,meson,sphinx --enable-pip=now # install all python deps now --enable-pip=on-demand # install qemu.git/meson/sphinx, delay the rest
--enable-pip=no # offline
--{enable,disable}-isolated-venv # let venv use system/distro if disable
This feels like a bit of overkill to me, and would create a hell of a lot of combinations to test if you expand the matrix of options.
Quoted 8 lines
I may have pulled us in a slightly different direction with the version of the patches I actually sent here today, but the key ideas here were:
- The ability to opt into or out of different package groups at configure-time. meson is always required, docs/sphinx is optional, and the other three groups are testing related. (testing: iotests and qemu.qmp, devel: python self-tests, avocado: avocado tests.)
I think this is especially overkill. While you can probably come up with some scenarios where you could use this fine grained selection, I don't think it would be very compelling.
Quoted 3 lines
- The ability to decide when packages get installed to the venv; either at configure time ("now") or as-needed ("on-demand") or functionally never ("no")
The distinction between now vs on-demand is really just about avoiding the overhead of downloading bits that you might not end up using. eg not downloading avocado unless you will be running 'make check-avocado'. That's saving a few 10s or 100s of KB of download, but doesn't feel like it'd make a noticable difference in the overall QEMU build time.
THe 'now' and 'no' options look sufficient (corresponding to the 'auto' and 'never' options I suggested above)
Quoted 2 lines
- The ability to enable or disable venv isolation (turning on or off system packages).
Corresponds to the 'force' option I suggested above.
Quoted 10 lines
I think I abandoned the idea of configuring the venv isolation and have it set to "always off". We may revisit this later, but I think for this series it's fine to have glossed over it.
I also skipped over the idea of having package installation being "now", "on-demand" or "no" -- this patch-set more or less is using a hybrid of "now" and "on-demand" where meson and sphinx are "now" but avocado is "on-demand". Unifying that discrepancy can occur in subsequent patches as it closely resembles what we had in 8.0 and earlier.
I guess the hybrid of 'now' and 'on-demand' could equally well map to what I suggested by 'auto'.
Quoted 7 lines
Quoted 4 lines
So eg distros could use pypi=never, devs would use pypi=auto mostly, while CI might use pypi=force to test specific versions indepenant of distros ?
Sounds reasonable in general, I think, but I have some questions about what 'force' does, exactly.
Turning of system packages in the venv, and always installing everything from pip.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 6 lines · Daniel P. Berrangé
Quoted 2 lines
So if you already have Sphinx, this should perform an upgrade to the latest version?
Essentially I meant 'force' to mean*never* use the host python installation packages. Always install all the deps in the venv, even if they exist in the host with sufficient version met.
I think this is essentially --enable-isolated-venv. I don't think there is a usecase for "let the venv use system packages, but override them with pip right away".
Quoted 11 lines
Quoted 7 lines
--python=... # runtime used to create venv --enable-pip-groups=testing,devel,avocado,meson,sphinx --enable-pip=now # install all python deps now --enable-pip=on-demand # install qemu.git/meson/sphinx, delay the rest
--enable-pip=no # offline
--{enable,disable}-isolated-venv # let venv use system/distro if disableThis feels like a bit of overkill to me, and would create a hell of a lot of combinations to test if you expand the matrix of options.
Yeah, this is a bit overkill. I think we can reduce it to three cases, corresponding to:
- --enable-pypi --enable-isolated-venv - use pip to install everything, including for options in "auto" state (e.g. would install sphinx without --enable-docs)
- --enable-pypi --disable-isolated-venv - use pip to install missing packages. TBD whether to do so for options in "auto" state or only for "enabled" (i.e., TBD whether to install sphinx without --enable-docs).
- --disable-pypi (only meaningful for --disable-isolated-venv) - apart from vendored wheels, just use system site packages (same as QEMU <= 8.0)
I think we want to hash out this detail first, and thus we should leave online mode out of the non-RFC version. It can be implemented together with isolated mode.
Paolo
Quoted 172 lines · John Snow
Quoted 159 lines · Daniel P. Berrangé
Quoted 30 lines · John Snow
GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
(All green, except Python self-tests, see below)This patch series creates a mandatory python virtual environment ("venv") during configure time and uses it to ensure the availability of meson and sphinx.
See https://www.qemu.org/2023/03/24/python/ for details. The summary is that the goal of this series is to ensure that the
pythonused to run meson is the samepythonused to run Sphinx, tests, and any build-time python scripting we have. As it stands, meson and sphinx (and their extensions) *may* run in a different python environment than the one configured and chosen by the user at configure/build time.The effective change of this series is that QEMU will now unconditionally create a venv at configure-time and will ensure that meson (and sphinx, if docs are enabled) are available through that venv.
Some important points as a pre-emptive "FAQ":
- This venv is unconditionally created and lives at {build_dir}/pyvenv.
- The python interpreter used by this venv is always the one identified by configure. (Which in turn is always the one specified by --python or $PYTHON)
- *almost* all python scripts in qemu.git executed as part of the build system, meson, sphinx, avocado tests, vm tests or CI are always executed within this venv.
Quoted 6 lines
(iotests are not yet integrated; I plan to tackle this separately as a follow-up in order to have a more tightly focused scope on that series.)
- It remains possible to build and test fully offline. (In most cases, you just need meson and sphinx from your distro's
repo.)
Quoted 60 lines
- Distribution packaged 'meson' and 'sphinx' are still utilized whenever possible as the highest preference.
- Vendored versions of e.g. 'meson' are always preferred to PyPI versions for speed, repeatability and ensuring tarball builds work as-is offline.
(Sphinx will not be vendored, just like it already isn't.)
- Missing dependencies, when possible, are fetched and installed on-demand automatically to make developer environments "just work".
- Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
- No new dependencies (...for most platforms. Debian and NetBSD get an asterisk.)
- The meson git submodule is unused after this series and can be removed.
For reviewers, here's how the series is broken up:
Patch 1 is a testing pre-req. Note that even with this patch, 'check-python-minreqs' and 'check-python-tox' CI jobs will both still fail on origin/master because this series requires 3.7+, but origin/master is currently still 3.6+.
- python: update pylint configuration
Patches 2-8 add the mkvenv script. The first patch checks in the barest essentials, and each subsequent patch adds a workaround or feature one at a time.
- python: add
mkvenv.py
- mkvenv: add console script entry point generation
- mkvenv: Add better error message for missing pyexapt module
- mkvenv: generate console entry shims from inside the venv
- mkvenv: work around broken pip installations on Debian 10
- mkvenv: add nested venv workaround
- mkvenv: add ensure subcommandPatches 9-11 modify our testing configuration to add new dependencies as needed.
- tests/docker: add python3-venv dependency
- tests/vm: Configure netbsd to use Python 3.10
- tests/vm: add py310-expat to NetBSDPatch 12 changes how we package release tarballs.
-
scripts/make-release: download meson==0.61.5 .whlPatches 13-16 wire mkvenv into configure and tests.
- configure: create a python venv unconditionally
- configure: use 'mkvenv ensure meson' to bootstrap meson
- configure: add --enable-pypi and --disable-pypi
- tests: Use configure-provided pyvenv for testsQuoted 8 lines
Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these changes could be folded earlier in the series (like the
diagnose()patch), but I'm keeping it separate for review for now.- configure: move --enable-docs and --disable-docs back to configure
- mkvenv: adddiagnose()method forensure()failures
- configure: use --diagnose option with meson ensure
- configure: bootstrap sphinx with mkvenvI'm not sure this last bit is working.
I uninstalled meson and python3-sphinx from my F38 host and ran configure --target-list=x86_64-softmmu and got this:
$ ./configure --target-list=x86_64-softmmu Using './build' as the directory for build output python determined to be '/usr/bin/python3' python version: Python 3.11.3 MKVENV pyvenv Configured python as '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B' MKVENV ensure meson>=0.61.5 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is ignored: it is neither a file nor a directory. ERROR: Could not find a version that satisfies the requirement meson>=0.61.5 (from versions: none) ERROR: No matching distribution found for meson>=0.61.5 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' MKVENV ensure sphinx>=1.6.0 WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name' ERROR: Could not find a version that satisfies the requirement sphinx>=1.6.0 (from versions: none) ERROR: No matching distribution found for sphinx>=1.6.0*** Ouch! ***
Could not ensure availability of 'sphinx>=1.6.0':
• Python package 'sphinx' was not found nor installed.
• No local package directory was searched.
• mkvenv was configured to operate offline and did not check PyPI.Quoted 9 lines
Sphinx not found/usable, disabling docs.
MKVENV ok!It says mkvenv was configured to run offline, but I didn't do anything. I thought the intention was for developers it would live download from PyPI ?
Quoted 4 lines
Ah. So... with enable pypi being the default and with docs set to "auto", I actually fall back to not installing sphinx from pypi *by default*. It tries to locate it on your system and will enable docs if it can, but it doesn't try too hard and doesn't get upset if it fails.
Ah right, yes, I think that behaviour makes sense.
Quoted 2 lines
Try passing --enable-docs to convince the build system you'd really definitely like docs, and it'll force the pypi access.
Ok will try.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 31 lines · John Snow
Some important points as a pre-emptive "FAQ":
- This venv is unconditionally created and lives at {build_dir}/pyvenv.
- The python interpreter used by this venv is always the one identified by configure. (Which in turn is always the one specified by --python or $PYTHON)
-*almost* all python scripts in qemu.git executed as part of the build
system, meson, sphinx, avocado tests, vm tests or CI are always executed within this venv.(iotests are not yet integrated; I plan to tackle this separately as a follow-up in order to have a more tightly focused scope on that series.)
- It remains possible to build and test fully offline.
(In most cases, you just need meson and sphinx from your distro's repo.)- Distribution packaged 'meson' and 'sphinx' are still utilized whenever possible as the highest preference.
- Vendored versions of e.g. 'meson' are always preferred to PyPI versions for speed, repeatability and ensuring tarball builds work as-is offline.
(Sphinx will not be vendored, just like it already isn't.)
- Missing dependencies, when possible, are fetched and installed on-demand automatically to make developer environments "just work".
Quoted 7 lines
- Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
- No new dependencies (...for most platforms. Debian and NetBSD get an asterisk.)
- The meson git submodule is unused after this series and can be removed.
Thanks, this looks pretty good. Some changes I'd make for the non-RFC version:
- I think we should just check in the meson wheel (which also removes the need for patch 12, so it can be done in its stead) and remove the submodule
- The verbosity of mkvenv.py can be tuned down and most prints replaced with logger.info() or logger.debug()
- While I agree with keeping patch 18 separate, I would move it earlier so that patch 19 can be squashed into patch 14
- I am ambivalent about keeping --enable/--disable-pypi in the first committed patchset, but in any case I would move patches 16 and 20 before patch 15
Paolo
Quoted 7 lines · Paolo Bonzini
Quoted 1 line
Thanks, this looks pretty good. Some changes I'd make for the non-RFC version:
- I think we should just check in the meson wheel (which also removes the need for patch 12, so it can be done in its stead) and remove the submodule
Quoted 8 lines
- The verbosity of
mkvenv.pycan be tuned down and most prints replaced withlogger.info()orlogger.debug()- While I agree with keeping patch 18 separate, I would move it earlier so that patch 19 can be squashed into patch 14
- I am ambivalent about keeping --enable/--disable-pypi in the first committed patchset, but in any case I would move patches 16 and 20 before patch 15
Just one extra thing, since we're changing so much of Python handling and since the code is written, I would keep the Debian 10 workarounds for now, and only drop them after we drop support for 3.6.
Paolo
Quoted 10 lines · Paolo Bonzini
Quoted 8 lines · Paolo Bonzini
Quoted 1 line
Thanks, this looks pretty good. Some changes I'd make for the non-RFC version:
- I think we should just check in the meson wheel (which also removes the need for patch 12, so it can be done in its stead) and remove the submodule
OK, if there's no objections.
As a consequence, meson will likely never be downloaded from PyPI with the patch set written as-is.
No real problem with that, just a difference.
Quoted 4 lines
Quoted 3 lines
- The verbosity of
mkvenv.pycan be tuned down and most prints replaced withlogger.info()orlogger.debug()
😅
Quoted 4 lines
Quoted 4 lines
- While I agree with keeping patch 18 separate, I would move it earlier so that patch 19 can be squashed into patch 14
Agree. It was newer / more experimental so I kept it unsquash'd. If you like it enough, asquashing I shall go.
Quoted 4 lines
Quoted 3 lines
- I am ambivalent about keeping --enable/--disable-pypi in the first committed patchset, but in any case I would move patches 16 and 20 before patch 15
I might be stubborn but I think I want to keep it in for now. If it needs redesigned to fit with the other flags you want to add, I think that's OK.
if we vendor the whl directly in qemu.git we won't need PyPI for meson, but it's still useful for Sphinx so I think I'm still leaning towards keeping it.
I'll try to refactor to keep it at the tail end of the series.
Quoted 4 lines
Just one extra thing, since we're changing so much of Python handling and since the code is written, I would keep the Debian 10 workarounds for now, and only drop them after we drop support for 3.6.
This series was written assuming we get to drop 3.6 as a prereq. Is that not the case?
Or did you mean to write 3.7 there?
Quoted 3 lines
Paolo
Quoted 8 lines · John Snow
> - I am ambivalent about keeping --enable/--disable-pypi in the first > committed patchset, but in any case I would move patches 16 and 20 > before patch 15
I might be stubborn but I think I want to keep it in for now. If it needs redesigned to fit with the other flags you want to add, I think that's OK.
if we vendor the whl directly in qemu.git we won't need PyPI for meson, but it's still useful for Sphinx so I think I'm still leaning towards keeping it.
Yes, it's definitely useful. It's just that unifying $python with sphinx-build's interpreter is as a separate (and earlier) step than introducing PyPI.
Quoted 1 line
I'll try to refactor to keep it at the tail end of the series.
Cool, thanks.
Quoted 8 lines
Just one extra thing, since we're changing so much of Python handling and since the code is written, I would keep the Debian 10 workarounds for now, and only drop them after we drop support for 3.6.
This series was written assuming we get to drop 3.6 as a prereq. Is that not the case?
Or did you mean to write 3.7 there?
Yes, 3.7.
Paolo
Quoted 7 lines · John Snow
GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
(All green, except Python self-tests, see below)This patch series creates a mandatory python virtual environment ("venv") during configure time and uses it to ensure the availability of meson and sphinx.
Quoted 6 lines
See https://www.qemu.org/2023/03/24/python/ for details. The summary is that the goal of this series is to ensure that the
pythonused to run meson is the samepythonused to run Sphinx, tests, and any build-time python scripting we have. As it stands, meson and sphinx (and their extensions) *may* run in a different python environment than the one configured and chosen by the user at configure/build time.
I mentioned this when we were chatting on IRC, but to repeat for the general audience..
I think it'd be useful for support purposes if the configure script summary printed details of how we setup the venv. eg perhaps a summary for each python module of whether we added it to the venv, or relied on te site packages:
python venv: meson (site), sphinx (venv), avocado (venv)
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 6 lines · Daniel P. Berrangé
I think it'd be useful for support purposes if the configure script summary printed details of how we setup the venv. eg perhaps a summary for each python module of whether we added it to the venv, or relied on te site packages:
python venv: meson (site), sphinx (venv), avocado (venv)
Yes, I agree John did a great job with the errors but a little more work is needed to cleanup of mkvenv.py's logging of the "good" case. The installation case is already covered by "pip install"'s output, but we need to have something like:
mkvenv: Creating {isolated|non-isolated} virtual environment [based on /home/pbonzini/myvenv]
and when creating the console-scripts shims:
mkvenv: Found avocado v90.0
Paolo
Quoted 8 lines · Paolo Bonzini
Quoted 6 lines · Daniel P. Berrangé
I think it'd be useful for support purposes if the configure script summary printed details of how we setup the venv. eg perhaps a summary for each python module of whether we added it to the venv, or relied on te site packages:
python venv: meson (site), sphinx (venv), avocado (venv)
OK, I'll try to work this out.
At the moment "already had the package" or "installed it from vendored source" is the same pip call, so it's hard to differentiate post-hoc without some further analysis.
One reason this is hard is because I don't have a good method for quickly determining "Do we already have $package that fulfills $depspec" except by invoking pip, and that's costly, 500ms at the absolute fastest I have yet managed. 1.2s in more typical cases. Owowowow.
So, for speed, I combine "do we already have it" and "can we install it from vendored source" into a single call to amortize pip startup cost.
With Python 3.8+ I might be able to inspect the package post-hoc to find where it's installed to and determine if it came from system packages or not.
It's easy to tell the user if we used PyPI or not, though.
I'll experiment.
Quoted 2 lines
Yes, I agree John did a great job with the errors but a little more work is needed to cleanup of mkvenv.py's logging of the "good" case.
In the good case, it worked! you should be happy! :D
(kidding.)
The
Quoted 6 lines
installation case is already covered by "pip install"'s output, but we need to have something like:
mkvenv: Creating {isolated|non-isolated} virtual environment [based on
/home/pbonzini/myvenv]
"based on ..." for nested venv case only?
Quoted 4 lines
and when creating the console-scripts shims:
mkvenv: Found avocado v90.0
Sure. Up to the user at that point to figure out if we used that package or not.
e.g. "found meson 0.5.0" might occur even if we require >=1.0.
Simple to implement, though.
Quoted 3 lines
Paolo
Quoted 5 lines · John Snow
mkvenv: Creating {isolated|non-isolated} virtual environment [based on
/home/pbonzini/myvenv]"based on ..." for nested venv case only?
Yes.
Quoted 8 lines
and when creating the console-scripts shims:
mkvenv: Found avocado v90.0
Sure. Up to the user at that point to figure out if we used that package or not.
e.g. "found meson 0.5.0" might occur even if we require >=1.0.
John and I discussed offlist and he'll try to remove --gen so that all system packages handling is done at "mkvenv ensure" time. The "Found" message can then be printed at "mkvenv ensure" time too.
Thanks,
Paolo
Quoted 2 lines
Simple to implement, though.
Quoted 154 lines · John Snow
GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
(All green, except Python self-tests, see below)This patch series creates a mandatory python virtual environment ("venv") during configure time and uses it to ensure the availability of meson and sphinx.
See https://www.qemu.org/2023/03/24/python/ for details. The summary is that the goal of this series is to ensure that the
pythonused to run meson is the samepythonused to run Sphinx, tests, and any build-time python scripting we have. As it stands, meson and sphinx (and their extensions) *may* run in a different python environment than the one configured and chosen by the user at configure/build time.The effective change of this series is that QEMU will now unconditionally create a venv at configure-time and will ensure that meson (and sphinx, if docs are enabled) are available through that venv.
Some important points as a pre-emptive "FAQ":
- This venv is unconditionally created and lives at {build_dir}/pyvenv.
- The python interpreter used by this venv is always the one identified by configure. (Which in turn is always the one specified by --python or $PYTHON)
- *almost* all python scripts in qemu.git executed as part of the build system, meson, sphinx, avocado tests, vm tests or CI are always executed within this venv.
(iotests are not yet integrated; I plan to tackle this separately as a follow-up in order to have a more tightly focused scope on that series.)
- It remains possible to build and test fully offline.
(In most cases, you just need meson and sphinx from your distro's repo.)- Distribution packaged 'meson' and 'sphinx' are still utilized whenever possible as the highest preference.
- Vendored versions of e.g. 'meson' are always preferred to PyPI versions for speed, repeatability and ensuring tarball builds work as-is offline.
(Sphinx will not be vendored, just like it already isn't.)
- Missing dependencies, when possible, are fetched and installed on-demand automatically to make developer environments "just work".
- Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
- No new dependencies (...for most platforms. Debian and NetBSD get an asterisk.)
- The meson git submodule is unused after this series and can be removed.
For reviewers, here's how the series is broken up:
Patch 1 is a testing pre-req. Note that even with this patch, 'check-python-minreqs' and 'check-python-tox' CI jobs will both still fail on origin/master because this series requires 3.7+, but origin/master is currently still 3.6+.
- python: update pylint configuration
Patches 2-8 add the mkvenv script. The first patch checks in the barest essentials, and each subsequent patch adds a workaround or feature one at a time.
- python: add
mkvenv.py
- mkvenv: add console script entry point generation
- mkvenv: Add better error message for missing pyexapt module
- mkvenv: generate console entry shims from inside the venv
- mkvenv: work around broken pip installations on Debian 10
- mkvenv: add nested venv workaround
- mkvenv: add ensure subcommandPatches 9-11 modify our testing configuration to add new dependencies as needed.
- tests/docker: add python3-venv dependency
- tests/vm: Configure netbsd to use Python 3.10
- tests/vm: add py310-expat to NetBSDPatch 12 changes how we package release tarballs.
-
scripts/make-release: download meson==0.61.5 .whlPatches 13-16 wire mkvenv into configure and tests.
- configure: create a python venv unconditionally
- configure: use 'mkvenv ensure meson' to bootstrap meson
- configure: add --enable-pypi and --disable-pypi
- tests: Use configure-provided pyvenv for testsPatches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these changes could be folded earlier in the series (like the
diagnose()patch), but I'm keeping it separate for review for now.- configure: move --enable-docs and --disable-docs back to configure
- mkvenv: adddiagnose()method forensure()failures
- configure: use --diagnose option with meson ensure
- configure: bootstrap sphinx with mkvenvThat's all for now, seeya!
--jsJohn Snow (20):
python: update pylint configuration
python: addmkvenv.py
mkvenv: add console script entry point generation
mkvenv: Add better error message for missing pyexpat module
mkvenv: generate console entry shims from inside the venv
mkvenv: work around broken pip installations on Debian 10
mkvenv: add nested venv workaround
mkvenv: add ensure subcommand
tests/docker: add python3-venv dependency
tests/vm: Configure netbsd to use Python 3.10
tests/vm: add py310-expat to NetBSDscripts/make-release: download meson==0.61.5 .whl
configure: create a python venv unconditionally
configure: use 'mkvenv ensure meson' to bootstrap meson
configure: add --enable-pypi and --disable-pypi
tests: Use configure-provided pyvenv for tests
configure: move --enable-docs and --disable-docs back to configure
mkvenv: adddiagnose()method forensure()failures
configure: use --diagnose option with meson ensure
configure: bootstrap sphinx with mkvenvdocs/devel/acpi-bits.rst | 6 +- docs/devel/testing.rst | 14 +- configure | 139 +-- .gitlab-ci.d/buildtest-template.yml | 4 +- .gitlab-ci.d/buildtest.yml | 6 +- python/scripts/mkvenv.py | 871 ++++++++++++++++++ python/setup.cfg | 10 + python/tests/flake8.sh | 1 + python/tests/isort.sh | 1 + python/tests/mypy.sh | 1 + python/tests/pylint.sh | 1 + .../org.centos/stream/8/x86_64/test-avocado | 4 +- scripts/device-crash-test | 2 +- scripts/make-release | 11 + tests/Makefile.include | 10 +- .../dockerfiles/debian-all-test-cross.docker | 3 +- .../dockerfiles/debian-hexagon-cross.docker | 3 +- .../dockerfiles/debian-riscv64-cross.docker | 3 +- .../dockerfiles/debian-tricore-cross.docker | 3 +- tests/requirements.txt | 7 +- tests/vm/netbsd | 2 + 21 files changed, 1016 insertions(+), 86 deletions(-) create mode 100644 python/scripts/mkvenv.py
Quoted 3 lines
--
2.39.2
Just a note to say I'm working on the respin here - Paolo suggested I remove the "--gen" arguments to mkvenv and generate the entry point scripts conditionally at ensure time instead.
Good idea!
I ran into some problems with msys2, though, where the entry point scripts native to the system are binary .exe files I don't know how to recreate.
I have some research to do on windows perhaps to understand what's expected on that platform and how to achieve it.
I'm hoping the fix is as easy as "You can still use console scripts, but you need to use the .py or .pyw suffix". I'll know shortly!
If I don't smooth over all the issues before Wednesday, though, I'm going to be away from my desk for the rest of the week and may need to ask Paolo for help finishing up the polish so we can land this in a timely fashion so we have a lot of runway to fix build issues before 8.1 rc0.
--js