Skip to contentSkip to conversation

configure: create a python venv and install meson

RFC PATCH Superseded John Snow 3 participants 18 messages Mar 28, 2023 – Apr 13, 2023

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

John Snow

author
#0 lore raw

cover configure: create a python venv and install meson

This patch series creates a mandatory venv during configure time and uses it to install meson.

The eventual point of this is to ensure that the Python used to run meson is the same Python used to run Sphinx, tests, and any build-time scripting we have. As it stands, meson and sphinx (and their extensions) may run in a different Python environment than the one configured and chosen by the user at configure/build time.

My goals for this series and future work are:

- Unconditionally create a venv to be used for both building and testing at configure time
- The python interpreter used by this venv is unconditionally the one specified by configure
- *all* python scripts in qemu.git executed as part of the build system, tests, or CI are *always* executed in the context of this venv.
- It is possible to build and test fully offline with sufficient preparation by installing appropriate system packages.
- Missing depdencies, when possible, are fetched and installed automatically to make developer environments "just work".
- Minimize cost during configure time whenever possible.

This series is in RFC state; some notable things:
- So far, only meson is included in this setup.
- There's a ton of debugging prints everywhere. It's extremely chatty right now.
- Sphinx and qemu.qmp are not yet included in this venv.
(qemu.qmp isn't able to be removed from the tree yet.)
- Testing isn't yet *fully* switched over.
- There is no online functionality yet, this series is *100% offline* - it's the harder option, so I tackled it first.

Some known bugs as of now:
- venv-in-venv setups are not yet handled.
- python3.7 setups without setuptools/pip in the host environment may be unable to generate script file shims; a workaround is in development but wasn't ready today. I decided to exclude it.

The good news:
- Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS, Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
- Startup cost is only milliseconds on the fast path
- No new dependencies (...for most platforms; the asterisk is Debian.)
- No new configure flags (...yet.)

John Snow (3):
python: add mkvenv.py
tests: add python3-venv dependency
configure: install meson to a python virtual environment

 configure                                     | 155 ++++--
 .gitlab-ci.d/buildtest-template.yml           |   2 +-
 python/scripts/mkvenv.py                      | 445 ++++++++++++++++++
 .../dockerfiles/debian-all-test-cross.docker  |   3 +-
 .../dockerfiles/debian-hexagon-cross.docker   |   3 +-
 .../dockerfiles/debian-riscv64-cross.docker   |   3 +-
 .../dockerfiles/debian-tricore-cross.docker   |   3 +-
 7 files changed, 561 insertions(+), 53 deletions(-)
 create mode 100644 python/scripts/mkvenv.py
Signature · 1 line
2.39.2
author
#1 lore raw

RFC PATCH 1/3 python: add mkvenv.py

This script will be responsible for building a Python virtual environment at configure time. As such, it exists outside of the installable python packages and *must* be runnable with minimal dependencies.

---
 python/scripts/mkvenv.py | 445 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 445 insertions(+)
 create mode 100644 python/scripts/mkvenv.py
python/scripts/mkvenv.py new file +445 -0
new file mode 100644
index 00000000000..d48880c4205
@@ -0,0 +1,445 @@
+"""
+mkvenv - QEMU venv bootstrapping utility
+
+usage: TODO/FIXME
+"""
+
+# Copyright (C) 2022-2023 Red Hat, Inc.
+#
+# Authors:
+#  John Snow <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+import argparse
+from importlib.util import find_spec
+import logging
+import os
+from os import PathLike
+from pathlib import Path
+import re
+import stat
+import subprocess
+import sys
+import traceback
+from types import SimpleNamespace
+from typing import Generator, Dict, Sequence, Optional, Union, Iterator
+import venv
+
+
+# Do not add any mandatory dependencies from outside the stdlib:
+# This script *must* be usable standalone!
+
+DirType = Union[str, bytes, 'PathLike[str]', 'PathLike[bytes]']
+logger = logging.getLogger('mkvenv')
+
+
+class Ouch(RuntimeError):
+    """An Exception class we can't confuse with a builtin."""
+
+
+class QemuEnvBuilder(venv.EnvBuilder):
+    """
+    An extension of venv.EnvBuilder for building QEMU's configure-time venv.
+
+    The only functional change is that it adds the ability to regenerate
+    console_script shims for packages available via system_site
+    packages.
+
+    (And a metric ton of debugging printfs)
+
+    Parameters for base class init:
+      - system_site_packages: bool = False
+      - clear: bool = False
+      - symlinks: bool = False
+      - upgrade: bool = False
+      - with_pip: bool = False
+      - prompt: Optional[str] = None
+      - upgrade_deps: bool = False             (Since 3.9)
+    """
+    def __init__(self, *args, **kwargs) -> None:
+        self.script_packages = kwargs.pop('script_packages', ())
+        super().__init__(*args, **kwargs)
+        # venv class is cute and toggles this off before post_setup,
+        # but we need it to decide if we want to generate shims or not.
+        self._system_site_packages = self.system_site_packages
+        # Make the context available post-creation:
+        self._context: Optional[SimpleNamespace] = None
+
+    def ensure_directories(self, env_dir: DirType) -> SimpleNamespace:
+        logger.debug("ensure_directories(env_dir=%s)", env_dir)
+        context = super().ensure_directories(env_dir)
+        # Here's what's in the context blob:
+        logger.debug("env_dir      %s", context.env_dir)
+        logger.debug("env_name     %s", context.env_name)
+        logger.debug("prompt       %s", context.prompt)
+        logger.debug("executable   %s", context.executable)
+        logger.debug("inc_path     %s", context.inc_path)
+        if 'lib_path' in context.__dict__:
+            # 3.12+
+            logger.debug("lib_path     %s", context.lib_path)
+        logger.debug("bin_path     %s", context.bin_path)
+        logger.debug("bin_name     %s", context.bin_name)
+        logger.debug("env_exe      %s", context.env_exe)
+        if 'env_exec_cmd' in context.__dict__:
+            # 3.9+
+            logger.debug("env_exec_cmd %s", context.env_exec_cmd)
+        self._context = context
+        return context
+
+    def create(self, env_dir: DirType) -> None:
+        logger.debug("create(env_dir=%s)", env_dir)
+        super().create(env_dir)
+        self.post_post_setup(self._context)
+
+    def create_configuration(self, context: SimpleNamespace) -> None:
+        logger.debug("create_configuration(...)")
+        super().create_configuration(context)
+
+    def setup_python(self, context: SimpleNamespace) -> None:
+        logger.debug("setup_python(...)")
+        super().setup_python(context)
+
+    def setup_scripts(self, context: SimpleNamespace) -> None:
+        logger.debug("setup_scripts(...)")
+        super().setup_scripts(context)
+
+    # def upgrade_dependencies(self, context): ...  # only Since 3.9
+
+    def post_setup(self, context: SimpleNamespace) -> None:
+        # Generate console_script entry points for system packages
+        # e.g. meson, sphinx, pip, etc.
+        logger.debug("post_setup(...)")
+        if self._system_site_packages:
+            generate_console_scripts(
+                context.env_exe, context.bin_path, self.script_packages)
+        #
+        # print the python executable to stdout for configure.
+        print(context.env_exe)
+
+    def post_post_setup(self, context: SimpleNamespace) -> None:
+        # This is the very final hook that occurs *after* enabling
+        # system-site-packages.
+        subprocess.run((context.env_exe, __file__, 'checkpip'), check=True)
+
+
+def need_ensurepip() -> bool:
+    """
+    Tests for the presence of setuptools and pip.
+
+    :return: `True` if we do not detect both packages.
+    """
+    # Don't try to actually import them, it's fraught with danger:
+    # https://github.com/pypa/setuptools/issues/2993
+    if find_spec("setuptools") and find_spec("pip"):
+        return False
+    return True
+
+
+def make_venv(  # pylint: disable=too-many-arguments
+        venv_path: Union[str, Path],
+        system_site_packages: bool = False,
+        clear: bool = True,
+        symlinks: Optional[bool] = None,
+        upgrade: bool = False,
+        with_pip: Optional[bool] = None,
+        script_packages: Sequence[str]=(),
+) -> None:
+    """
+    Create a venv using the QemuEnvBuilder class.
+
+    TODO: write docs O:-)
+    """
+    logging.debug("%s: make_venv(venv_path=%s, system_site_packages=%s, "
+                  "clear=%s, upgrade=%s, with_pip=%s, script_packages=%s)",
+                  __file__, str(venv_path), system_site_packages,
+                  clear, upgrade, with_pip, script_packages)
+
+    print(f"MKVENV {str(venv_path)}", file=sys.stderr)
+
+    # ensurepip is slow: venv creation can be very fast for cases where
+    # we allow the use of system_site_packages. Toggle ensure_pip on only
+    # in the cases where we really need it.
+    if with_pip is None:
+        with_pip = True if not system_site_packages else need_ensurepip()
+        logger.debug("with_pip unset, choosing %s", with_pip)
+
+    if symlinks is None:
+        # Default behavior of standard venv CLI
+        symlinks = os.name != "nt"
+
+    if with_pip and not find_spec("ensurepip"):
+        msg = ("Python's ensurepip module is not found.\n"
+
+               "It's normally part of the Python standard library, "
+               "maybe your distribution packages it separately?\n"
+
+               "Either install ensurepip, or alleviate the need for it in the "
+               "first place by installing pip and setuptools for "
+               f"'{sys.executable}'.\n"
+
+               "(Hint: Debian puts ensurepip in its python3-venv package.)")
+        raise Ouch(msg)
+
+    builder = QemuEnvBuilder(
+        system_site_packages=system_site_packages,
+        clear=clear,
+        symlinks=symlinks,
+        upgrade=upgrade,
+        with_pip=with_pip,
+        script_packages=script_packages,
+    )
+    try:
+        logger.debug("Invoking builder.create()")
+        try:
+            builder.create(str(venv_path))
+        except SystemExit as exc:
+            # Some versions of the venv module raise SystemExit; *nasty*!
+            # We want the exception that prompted it. It might be a subprocess
+            # error that has output we *really* want to see.
+            logger.debug("Intercepted SystemExit from EnvBuilder.create()")
+            raise exc.__cause__ or exc.__context__ or exc
+        logger.debug("builder.create() finished")
+    except subprocess.CalledProcessError as exc:
+        print(f"cmd: {exc.cmd}", file=sys.stderr)
+        print(f"returncode: {exc.returncode}", file=sys.stderr)
+        if exc.stdout:
+            print("========== stdout ==========", file=sys.stderr)
+            print(exc.stdout, file=sys.stderr)
+            print("============================", file=sys.stderr)
+        if exc.stderr:
+            print("========== stderr ==========", file=sys.stderr)
+            print(exc.stderr, file=sys.stderr)
+            print("============================", file=sys.stderr)
+        raise Ouch("VENV creation subprocess failed.") from exc
+
+
+def _gen_importlib(packages: Sequence[str]) -> Iterator[Dict[str, str]]:
+    # pylint: disable=import-outside-toplevel
+    try:
+        # First preference: Python 3.8+ stdlib
+        from importlib.metadata import (
+            PackageNotFoundError,
+            distribution,
+        )
+    except ImportError as exc:
+        logger.debug("%s", str(exc))
+        # Second preference: Commonly available PyPI backport
+        from importlib_metadata import (
+            PackageNotFoundError,
+            distribution,
+        )
+
+    # Borrowed from CPython (Lib/importlib/metadata/__init__.py)
+    pattern = re.compile(
+        r'(?P<module>[\w.]+)\s*'
+        r'(:\s*(?P<attr>[\w.]+)\s*)?'
+        r'((?P<extras>\[.*\])\s*)?$'
+    )
+
+    def _generator() -> Iterator[Dict[str, str]]:
+        for package in packages:
+            try:
+                entry_points = distribution(package).entry_points
+            except PackageNotFoundError:
+                continue
+
+            # The EntryPoints type is only available in 3.10+,
+            # treat this as a vanilla list and filter it ourselves.
+            entry_points = filter(
+                lambda ep: ep.group == 'console_scripts', entry_points)
+
+            for entry_point in entry_points:
+                # Python 3.8 doesn't have 'module' or 'attr' attributes
+                if not (hasattr(entry_point, 'module') and
+                        hasattr(entry_point, 'attr')):
+                    match = pattern.match(entry_point.value)
+                    assert match is not None
+                    module = match.group('module')
+                    attr = match.group('attr')
+                else:
+                    module = entry_point.module
+                    attr = entry_point.attr
+                yield {
+                    'name': entry_point.name,
+                    'module': module,
+                    'import_name': attr,
+                    'func': attr,
+                }
+
+    return _generator()
+
+
+def _gen_pkg_resources(packages: Sequence[str]) -> Iterator[Dict[str, str]]:
+    # pylint: disable=import-outside-toplevel
+    # Bundled with setuptools; has a good chance of being available.
+    import pkg_resources
+
+    def _generator() -> Iterator[Dict[str, str]]:
+        for package in packages:
+            try:
+                eps = pkg_resources.get_entry_map(package, 'console_scripts')
+            except pkg_resources.DistributionNotFound:
+                continue
+
+            for entry_point in eps.values():
+                yield {
+                    'name': entry_point.name,
+                    'module': entry_point.module_name,
+                    'import_name': ".".join(entry_point.attrs),
+                    'func': ".".join(entry_point.attrs),
+                }
+
+    return _generator()
+
+
+# Borrowed/adapted from pip's vendored version of distutils:
+SCRIPT_TEMPLATE = r'''#!{python_path:s}
+# -*- coding: utf-8 -*-
+import re
+import sys
+from {module:s} import {import_name:s}
+if __name__ == '__main__':
+    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
+    sys.exit({func:s}())
+'''
+
+
+def generate_console_scripts(python_path: str, bin_path: str,
+                             packages: Sequence[str]) -> None:
+    """
+    Generate script shims for console_script entry points in @packages.
+    """
+    if not packages:
+        return
+
+    def _get_entry_points() -> Iterator[Dict[str, str]]:
+        try:
+            return _gen_importlib(packages)
+        except ImportError as exc:
+            logger.debug("%s", str(exc))
+        return _gen_pkg_resources(packages)
+
+    try:
+        entry_points = _get_entry_points()
+    except ImportError as exc:
+        logger.debug("%s", str(exc))
+        raise Ouch(
+            "Neither importlib.metadata nor pkg_resources found, "
+            "can't generate console script shims.\n"
+            "Use Python 3.8+, or install importlib-metadata, or setuptools."
+        ) from exc
+
+    for entry_point in entry_points:
+        script_path = os.path.join(bin_path, entry_point['name'])
+        script = SCRIPT_TEMPLATE.format(python_path=python_path, **entry_point)
+        with open(script_path, "w", encoding='UTF-8') as file:
+            file.write(script)
+            fd = file.fileno()
+            os.chmod(fd, os.stat(fd).st_mode | stat.S_IEXEC)
+        logger.debug("wrote '%s'", script_path)
+
+
+def checkpip():
+    """
+    Debian10 has a pip that's broken when used inside of a virtual environment.
+
+    We try to detect and correct that case here.
+    """
+    try:
+        import pip._internal
+        logger.debug("pip appears to be working correctly.")
+        return
+    except ModuleNotFoundError as exc:
+        if exc.name == 'pip._internal':
+            # Uh, fair enough. They did say "internal".
+            # Let's just assume it's fine.
+            return
+        logger.warning("pip appears to be malfunctioning: %s", str(exc))
+
+    # Test for ensurepip:
+    try:
+        import ensurepip
+    except ImportError as exc:
+        raise Ouch(
+            "pip appears to be non-functional, "
+            "and Python's ensurepip module is not found.\n"
+            "It's normally part of the Python standard library, "
+            "maybe your distribution packages it separately?\n"
+            "(Hint: Debian puts ensurepip in its python3-venv package.)"
+        ) from exc
+
+    logging.debug("Attempting to repair pip ...")
+    subprocess.run((sys.executable, '-m', 'ensurepip'),
+                   stdout=subprocess.DEVNULL, check=True)
+    logging.debug("Pip is now (hopefully) repaired!")
+
+
+def main() -> int:
+    """CLI interface to make_qemu_venv. See module docstring."""
+    if os.environ.get('DEBUG') or os.environ.get('GITLAB_CI'):
+        # You're welcome.
+        logging.basicConfig(level=logging.DEBUG)
+
+    parser = argparse.ArgumentParser(description="Bootstrap QEMU venv.")
+    subparsers = parser.add_subparsers(
+        title="Commands",
+        description="Various actions this utility can perform",
+        prog="prog",
+        dest="command",
+        required=True,
+        metavar="command",
+        help='Description')
+
+    #
+    subparser = subparsers.add_parser('create', help='create a venv')
+    subparser.add_argument(
+        '--gen',
+        type=str,
+        action='append',
+        help="Regenerate console_scripts for given packages, if found.",
+    )
+    subparser.add_argument(
+        'target',
+        type=str,
+        action='store',
+        help="Target directory to install virtual environment into.",
+    )
+
+    #
+    subparser = subparsers.add_parser(
+        'checkpip', help='test pip and fix if necessary')
+
+    args = parser.parse_args()
+
+    try:
+        if args.command == 'create':
+            script_packages = []
+            for element in args.gen or ():
+                script_packages.extend(element.split(","))
+            make_venv(
+                args.target,
+                system_site_packages=True,
+                clear=True,
+                upgrade=False,
+                with_pip=None,  # Autodetermine
+                script_packages=script_packages,
+            )
+            logger.debug("mkvenv.py create - exiting")
+        if args.command == 'checkpip':
+            checkpip()
+            logger.debug("mkvenv.py checkpip - exiting")
+    except Ouch as exc:
+        print("\n*** Ouch! ***\n", file=sys.stderr)
+        print(str(exc), "\n\n", file=sys.stderr)
+        return 1
+    except:  # pylint: disable=bare-except
+        print("mkvenv did not complete successfully:", file=sys.stderr)
+        traceback.print_exc()
+        return 2
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main())
Signature · 1 line
2.39.2
#2 lore raw

Re: [RFC PATCH 1/3] python: add mkvenv.py

python/scripts/mkvenv.py line 65

Quoted 2 lines · John Snow
python/scripts/mkvenv.py new file +2 -0
+        # venv class is cute and toggles this off before post_setup,
+        # but we need it to decide if we want to generate shims or not.

Ha, yeah that's a bug in the venv package. post_setup() can already run with system_site_packages reverted to True.

Quoted 6 lines
python/scripts/mkvenv.py new file +6 -0
+            for entry_point in entry_points:
+                # Python 3.8 doesn't have 'module' or 'attr' attributes
+                if not (hasattr(entry_point, 'module') and
+                        hasattr(entry_point, 'attr')):
+                    match = pattern.match(entry_point.value)
+                    assert match is not None

python/scripts/mkvenv.py line 268

Quoted 10 lines
python/scripts/mkvenv.py new file +10 -0
+                    module = match.group('module')
+                    attr = match.group('attr')
+                else:
+                    module = entry_point.module
+                    attr = entry_point.attr
+                yield {
+                    'name': entry_point.name,
+                    'module': module,
+                    'import_name': attr,
+                    'func': attr,

What about using a dataclass or namedtuple instead of a dictionary?

python/scripts/mkvenv.py line 332

Quoted 11 lines
python/scripts/mkvenv.py new file +10 -0
+
+    try:
+        entry_points = _get_entry_points()
+    except ImportError as exc:
+        logger.debug("%s", str(exc))
+        raise Ouch(
+            "Neither importlib.metadata nor pkg_resources found, "
+            "can't generate console script shims.\n"
+            "Use Python 3.8+, or install importlib-metadata, or setuptools."
+        ) from exc

Why not put this extra try/except inside _get_entry_points()?

python/scripts/mkvenv.py line 363

Quoted 4 lines
python/scripts/mkvenv.py new file +4 -0
+
+    # Test for ensurepip:
+    try:
+        import ensurepip

Use find_spec()?

BTW, another way to repair Debian 10's pip is to create a symbolic link to sys.base_prefix + '/share/python-wheels' in sys.prefix + '/share/python-wheels'. Since this is much faster, perhaps it can be done unconditionally and checkpip mode can go away together with self._context?

Paolo

author
#3 lore raw

Re: [RFC PATCH 1/3] python: add mkvenv.py

Quoted 24 lines · Paolo Bonzini

python/scripts/mkvenv.py line 65

Quoted 2 lines · John Snow
python/scripts/mkvenv.py new file +2 -0
+        # venv class is cute and toggles this off before post_setup,
+        # but we need it to decide if we want to generate shims or not.

Ha, yeah that's a bug in the venv package. post_setup() can already run with system_site_packages reverted to True.

Quoted 6 lines
python/scripts/mkvenv.py new file +6 -0
+            for entry_point in entry_points:
+                # Python 3.8 doesn't have 'module' or 'attr' attributes
+                if not (hasattr(entry_point, 'module') and
+                        hasattr(entry_point, 'attr')):
+                    match = pattern.match(entry_point.value)
+                    assert match is not None

python/scripts/mkvenv.py line 268

Quoted 10 lines
python/scripts/mkvenv.py new file +10 -0
+                    module = match.group('module')
+                    attr = match.group('attr')
+                else:
+                    module = entry_point.module
+                    attr = entry_point.attr
+                yield {
+                    'name': entry_point.name,
+                    'module': module,
+                    'import_name': attr,
+                    'func': attr,
Quoted 2 lines

What about using a dataclass or namedtuple instead of a dictionary?

Sure. Once 3.8 is our minimum there's no point, though.

Quoted 10 lines

python/scripts/mkvenv.py

Quoted 10 lines
python/scripts/mkvenv.py new file +9 -0
+
+    try:
+        entry_points = _get_entry_points()
+    except ImportError as exc:
+        logger.debug("%s", str(exc))
+        raise Ouch(
+            "Neither importlib.metadata nor pkg_resources found, "
+            "can't generate console script shims.\n"
+            "Use Python 3.8+, or install importlib-metadata, or
Quoted 5 lines

setuptools."

Quoted 1 line

+ ) from exc

Why not put this extra try/except inside _get_entry_points()?

I don't remember. I'll look! I know it looks goofy. The ultimate answer is "So I can log all import failures without nesting eight layers deep".

Quoted 7 lines

python/scripts/mkvenv.py line 363

Quoted 4 lines
python/scripts/mkvenv.py new file +4 -0
+
+    # Test for ensurepip:
+    try:
+        import ensurepip

Use find_spec()?

That might be better. Originally I tried to use ensurepip directly, but found it didn't work right if you had already imported pip. This survived from the earlier draft.

Quoted 6 lines

BTW, another way to repair Debian 10's pip is to create a symbolic link to sys.base_prefix + '/share/python-wheels' in sys.prefix + '/share/python-wheels'. Since this is much faster, perhaps it can be done unconditionally and checkpip mode can go away together with self._context?

I guess I like it less because it's way more Debian-specific at that point. I think I'd sooner say "Sorry, Debian 10 isn't supported!"

(Or encourage users to upgrade their pip/setuptools/ensurepip to something that doesn't trigger the bug.)

Or, IOW, I feel like it's normal to expect ensurepip to work but mussing around with symlinks to special directories created by a distribution just feels way more fiddly.

Quoted 3 lines

Paolo

#4 lore raw

Re: [RFC PATCH 1/3] python: add mkvenv.py

Quoted 11 lines · John Snow

> +                yield {
> +                    'name': entry_point.name
<http://entry_point.name>,
> +                    'module': module, > +                    'import_name': attr, > +                    'func': attr,

What about using a dataclass or namedtuple instead of a dictionary?

Sure. Once 3.8 is our minimum there's no point, though.

Well, that's why I also mentioned namedtuples. But no big deal.

Quoted 7 lines

BTW, another way to repair Debian 10's pip is to create a symbolic link to sys.base_prefix + '/share/python-wheels' in sys.prefix + '/share/python-wheels'.  Since this is much faster, perhaps it can be done unconditionally and checkpip mode can go away together with self._context?

Quoted 5 lines

I guess I like it less because it's way more Debian-specific at that point. I think I'd sooner say "Sorry, Debian 10 isn't supported!"

(Or encourage users to upgrade their pip/setuptools/ensurepip to something that doesn't trigger the bug.)

Or, IOW, I feel like it's normal to expect ensurepip to work but mussing around with symlinks to special directories created by a distribution just feels way more fiddly.

No doubt about that. It's just the balance between simple fiddly code and more robust code that is also longer.

Anyhow later on we will split mkvenv.py in multiple patches so it will be easy to revert checkpip when time comes. For example, when Python 3.7 is dropped for good rather than being just "untested but should work", this Debian 10 hack and the importlib_metadata/pkg_resources fallbacks go away at the same time.

Paolo

#5 lore raw

Re: [RFC PATCH 1/3] python: add mkvenv.py

Quoted 7 lines · Paolo Bonzini
Quoted 5 lines

    What about using a dataclass or namedtuple instead of a dictionary?

Sure. Once 3.8 is our minimum there's no point, though.

Well, that's why I also mentioned namedtuples.  But no big deal.

Sorry, I misunderstood this (I read "until 3.8 is our minimum" and interpreted that as "dataclasses are not in 3.6").

I agree, not much need to future-proof the <=3.7 parts of the code.

Paolo

author
#6 lore raw

Re: [RFC PATCH 1/3] python: add mkvenv.py

In reply to Paolo Bonzini
Quoted 20 lines · Paolo Bonzini
Quoted 6 lines · John Snow
python/scripts/mkvenv.py new file +6 -0
+            for entry_point in entry_points:
+                # Python 3.8 doesn't have 'module' or 'attr' attributes
+                if not (hasattr(entry_point, 'module') and
+                        hasattr(entry_point, 'attr')):
+                    match = pattern.match(entry_point.value)
+                    assert match is not None

python/scripts/mkvenv.py line 268

Quoted 10 lines
python/scripts/mkvenv.py new file +10 -0
+                    module = match.group('module')
+                    attr = match.group('attr')
+                else:
+                    module = entry_point.module
+                    attr = entry_point.attr
+                yield {
+                    'name': entry_point.name,
+                    'module': module,
+                    'import_name': attr,
+                    'func': attr,
Quoted 2 lines

What about using a dataclass or namedtuple instead of a dictionary?

I suppose what I meant was: Once 3.8 is our minimum, we can delete most of this compat code anyway, so there may not be a point in creating a new type-safe structure to house it. I can definitely add that in if you'd like, but I suppose I felt like a dict was "good enough" for now, since 3.7 will also get dropped off the face of the earth soon, too.

Before I send a non-RFC patch I'll get everything scrubbed down with the usual pylint/mypy/isort/flake8 combo, and if I wind up needing to for type safety I will add something.

Or if you are requesting it specifically. :~)

Quoted 12 lines

python/scripts/mkvenv.py line 332

Quoted 11 lines
python/scripts/mkvenv.py new file +10 -0
+
+    try:
+        entry_points = _get_entry_points()
+    except ImportError as exc:
+        logger.debug("%s", str(exc))
+        raise Ouch(
+            "Neither importlib.metadata nor pkg_resources found, "
+            "can't generate console script shims.\n"
+            "Use Python 3.8+, or install importlib-metadata, or setuptools."
+        ) from exc
Quoted 1 line

Why not put this extra try/except inside _get_entry_points()?

Hm, no good reason, apparently. O:-) I've fixed this one up.

Unrelated question I'm going to tuck in here:

For the script generation, I am making another call to mkvenv.py using the venv'ified python to do final initializations. As part of that, I pass the binpath to the script again because I wasn't sure it was safe to compute it again myself. CPython seems to assume it's always going to be env_path/Scripts/ or env_path/bin/, but I wasn't 1000% sure that this wasn't patched by e.g. Debian or had some complications with the adjustments to site configuration in recent times. I'll circle back around to investigating this, but for now I've left it with the dumber approach of always passing the bindir.

author
#7 lore raw

Re: [RFC PATCH 1/3] python: add mkvenv.py

In reply to Paolo Bonzini
Quoted 9 lines · Paolo Bonzini

BTW, another way to repair Debian 10's pip is to create a symbolic link to sys.base_prefix + '/share/python-wheels' in sys.prefix + '/share/python-wheels'. Since this is much faster, perhaps it can be done unconditionally and checkpip mode can go away together with self._context?

Paolo

I'm coming around on this one a bit; it's definitely going to be a lot faster. As you say, my version is more robust, but more complex and with more lines. We may decide to drop any workarounds for Debian 10 entirely and we can live without either fix. I'll mention this in the commit message for the Debian 10 workaround.

I do not know right now if other distros suffer from the same problem; we could attempt to omit the fix and just see if anyone barks. Not very nice, but impossible to enumerate all of the bugs that exist in various downstream distros...

--js

author
#8 lore raw

RFC PATCH 2/3 tests: add python3-venv dependency

In reply to John Snow

Several debian-based tests need the python3-venv dependency as a consequence of Debian debundling the "ensurepip" module normally included with Python.

As this series stands right now, in most cases, Debian requires EITHER:

(A) setuptools and pip, or
(B) ensurepip

It's quite likely most developer workstations running Debian or a Debian-based distro already have all of these packages. venv creation will actually be quite a bit faster by installing pip and setuptools, but for the purposes of this patch I am only adding the strict minimum necessary.

(For that reason, I chose ensurepip to make sure that as we continue tweaking and developing the mkvenv script, we can count on Debian having a fully functional Python installation; but for the sake of this series it is an arbitrary choice here.)

In one case, even though we already have pip and setuptools, we still need ensurepip because the version of these tools that shipped with Debian are known to be broken and do not function correctly inside of a venv. Using ensurepip bypasses the bug. (This is only for Debian 10, which we technically no longer support. Even so, I wanted to prove it's possible to support our oldest platforms.)

---
 tests/docker/dockerfiles/debian-all-test-cross.docker | 3 ++-
 tests/docker/dockerfiles/debian-hexagon-cross.docker  | 3 ++-
 tests/docker/dockerfiles/debian-riscv64-cross.docker  | 3 ++-
 tests/docker/dockerfiles/debian-tricore-cross.docker  | 3 ++-
 4 files changed, 8 insertions(+), 4 deletions(-)
tests/docker/dockerfiles/debian-all-test-cross.docker +2 -1
index 981e9bdc7b0..f9f401544a0 100644
@@ -57,7 +57,8 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
         gcc-sh4-linux-gnu \
         libc6-dev-sh4-cross \
         gcc-sparc64-linux-gnu \
-        libc6-dev-sparc64-cross
+        libc6-dev-sparc64-cross \
+        python3-venv
 
 ENV QEMU_CONFIGURE_OPTS --disable-system --disable-docs --disable-tools
 ENV DEF_TARGET_LIST aarch64-linux-user,alpha-linux-user,arm-linux-user,hppa-linux-user,i386-linux-user,m68k-linux-user,mips-linux-user,mips64-linux-user,mips64el-linux-user,mipsel-linux-user,ppc-linux-user,ppc64-linux-user,ppc64le-linux-user,riscv64-linux-user,s390x-linux-user,sh4-linux-user,sparc64-linux-user
tests/docker/dockerfiles/debian-hexagon-cross.docker +2 -1
index 5308ccb8fe9..344eec900b0 100644
@@ -20,7 +20,8 @@ RUN apt-get update && \
         bison \
         flex \
         git \
-        ninja-build && \
+        ninja-build \
+        python3-venv && \
 # Install QEMU build deps for use in CI
     DEBIAN_FRONTEND=noninteractive eatmydata \
     apt build-dep -yy --arch-only qemu
tests/docker/dockerfiles/debian-riscv64-cross.docker +2 -1
index 803afb95734..081404e0141 100644
@@ -28,7 +28,8 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata apt install -yy \
     libglib2.0-dev \
     ninja-build \
     pkg-config \
-    python3
+    python3 \
+    python3-venv
 
 # Add ports and riscv64 architecture
 RUN echo "deb http://ftp.ports.debian.org/debian-ports/ sid main" >> /etc/apt/sources.list
tests/docker/dockerfiles/debian-tricore-cross.docker +2 -1
index cfd2faf9a80..269bfa8d423 100644
@@ -33,7 +33,8 @@ RUN apt update && \
        pkgconf \
        python3-pip \
        python3-setuptools \
-       python3-wheel
+       python3-wheel \
+       python3-venv
 
 RUN curl -#SL https://github.com/bkoppelmann/package_940/releases/download/tricore-toolchain-9.40/tricore-toolchain-9.4.0.tar.gz \
     | tar -xzC /usr/local/
Signature · 1 line
2.39.2
author
#9 lore raw

RFC PATCH 3/3 configure: install meson to a python virtual environment

In reply to John Snow

This patch changes how we detect and install meson.

This patch creates a lightweight Python virtual environment using the user's configured $python that inherits system packages. If Meson is installed there and meets our minimum version requirements, we will use that Meson.

In the event that Meson is installed but *not for the chosen Python interpreter*, not found, or of insufficient version, we will attempt to install Meson from source into the newly created Python virtual environment.

At present, the source is obtained in the same manner as it has been; preferring git submodules first and relying on vendored source as a backup.

This patch does *not* connect to PyPI and will work offline for all supported build platforms. Future improvements to this patch may add optional PyPI support as a replacement for git submodules.

As a result of this patch, the Python interpreter we use for both our own build scripts *and* Meson extensions are always known to be the exact same Python. As a further benefit, there will also be a symlink available in the build directory that points to the correct, configured python and can be used by e.g. manual tests to invoke the correct, configured Python unambiguously.

---
 configure                           | 155 +++++++++++++++++++---------
 .gitlab-ci.d/buildtest-template.yml |   2 +-
 2 files changed, 108 insertions(+), 49 deletions(-)
configure +107 -48
index 05bed4f4a11..1ec054775f6 100755
@@ -607,7 +607,6 @@ check_py_version() {
 python=
 first_python=
 if test -z "${PYTHON}"; then
-    explicit_python=no
     # A bare 'python' is traditionally python 2.x, but some distros
     # have it as python 3.x, so check in both places.
     for binary in python3 python python3.11 python3.10 python3.9 python3.8 python3.7 python3.6; do
@@ -626,7 +625,6 @@ else
     # Same as above, but only check the environment variable.
     has "${PYTHON}" || error_exit "The PYTHON environment variable does not point to an executable"
     python=$(command -v "$PYTHON")
-    explicit_python=yes
     if check_py_version "$python"; then
         # This one is good.
         first_python=
@@ -718,7 +716,7 @@ for opt do
   ;;
   --install=*)
   ;;
-  --python=*) python="$optarg" ; explicit_python=yes
+  --python=*) python="$optarg"
   ;;
   --skip-meson) skip_meson=yes
   ;;
@@ -1078,61 +1076,122 @@ if ! check_py_version "$python"; then
       "Use --python=/path/to/python to specify a supported Python."
 fi
 
-# Resolve PATH + suppress writing compiled files
-python="$(command -v "$python") -B"
-
-has_meson() {
-  local python_dir=$(dirname "$python")
-  # PEP405: pyvenv.cfg is either adjacent to the Python executable
-  # or one directory above
-  if test -f $python_dir/pyvenv.cfg || test -f $python_dir/../pyvenv.cfg; then
-    # Ensure that Meson and Python come from the same virtual environment
-    test -x "$python_dir/meson" &&
-      test "$(command -v meson)" -ef "$python_dir/meson"
-  else
-    has meson
-  fi
+# Resolve PATH
+python="$(command -v "$python")"
+# This variable is intended to be used only for error messages:
+target_python=$python
+
+# Create a Python virtual environment using our configured python.
+# The stdout of this script will be the location of a symlink that
+# points to the configured Python.
+# Entry point scripts for pip, meson, and sphinx are generated if those
+# packages are present.
+
+# Defaults assumed for now:
+# - venv is cleared if it exists already;
+# - venv is allowed to use system packages;
+# - all setup is performed **offline**;
+# - No packages are installed by default;
+# - pip is not installed into the venv when possible,
+#   but ensurepip is called as a fallback when necessary.
+
+echo "python determined to be '$python'"
+echo "python version: $($python --version)"
+
+echo " === pip list (host) === "
+$python -m pip list --disable-pip-version-check
+echo " ======================= "
+echo ""
+
+python="$($python -B "${source_path}/python/scripts/mkvenv.py" create --gen pip,meson,sphinx pyvenv)"
+if test "$?" -ne 0 ; then
+    error_exit "python venv creation failed"
+fi
+
+# Suppress writing compiled files
+python="$python -B"
+echo "Configured python as '$python'"
+
+echo " === pip list (venv) === "
+$python -m pip list --disable-pip-version-check
+echo " ======================= "
+echo ""
+
+pip_install() {
+    $python -m pip install -v \
+            --disable-pip-version-check \
+            --no-index \
+            "$@"
 }
 
-if test -z "$meson"; then
-    if test "$explicit_python" = no && has_meson && version_ge "$(meson --version)" 0.61.5; then
-        meson=meson
-    elif test "$git_submodules_action" != 'ignore' ; then
-        meson=git
-    elif test -e "${source_path}/meson/meson.py" ; then
-        meson=internal
+# OK, let's have some fun!
+
+# This install command is intended to either fail or be a NOP;
+# because we're offline, it's just a convenient version check.
+if ! pip_install 'meson>=0.61.5'; then
+    # Either we don't have Meson, or our Meson is too old.
+    # (Future revisions of this patchset can be less chatty.)
+    if test -e pyvenv/bin/meson; then
+        echo "Meson in pyvenv is too old: $(pyvenv/bin/meson --version)"
+    elif has meson ; then
+        echo "Meson was found installed on your system," \
+             "but not for the configured Python interpreter ($target_python)."
+        echo "(Hint: check '$(which meson)' to see which interpreter its shebang uses.)"
     else
-        if test "$explicit_python" = yes; then
-            error_exit "--python requires using QEMU's embedded Meson distribution, but it was not found."
-        else
-            error_exit "Meson not found.  Use --meson=/path/to/meson"
+        echo "Meson was not found."
+    fi
+
+    # OK, but can we fix it, though? :~)
+    if test "$git_submodules_action" != 'ignore' ; then
+        git_submodules="${git_submodules} meson"
+        echo "Attempting to install meson from git submodule ..."
+        # Stolen from later in the configure file.
+        # Is it a problem if we front-load this now and run it again later?
+        if ! (GIT="$git" "$source_path/scripts/git-submodule.sh" "$git_submodules_action" "$git_submodules"); then
+            exit 1
         fi
+    elif test -e "${source_path}/meson/setup.cfg" ; then
+        echo "Attempting to install meson from vendored source ..."
+    else
+        # In the future, we could use PyPI as a source if the user allows it.
+        # For now, you're outta luck!
+        error_exit "A suitable version of Meson was not found."
     fi
-else
-    # Meson uses its own Python interpreter to invoke other Python scripts,
-    # but the user wants to use the one they specified with --python.
+
+    # If we're here, we have the meson source and we can attempt to
+    # install it into our venv.
+
+    # We want to install meson with --no-use-pep517 if possible,
+    # because it avoids needing a 'wheel' dependency. Old versions
+    # of pip do this by default, so test for the behavior.
     #
-    # We do not want to override the distro Python interpreter (and sometimes
-    # cannot: for example in Homebrew /usr/bin/meson is a bash script), so
-    # just require --meson=git|internal together with --python.
-    if test "$explicit_python" = yes; then
-        case "$meson" in
-            git | internal) ;;
-            *) error_exit "--python requires using QEMU's embedded Meson distribution." ;;
-        esac
+    # --no-build-isolation was added to pip 10.0.
+    # --no-use-pep517 was added ... sometime after 18.1?
+    pip_flags='--no-build-isolation'
+    if $python -m pip install --help | grep 'no-use-pep517' > /dev/null 2>&1 ; then
+        pip_flags="${pip_flags} --no-use-pep517"
+    fi
+    if ! pip_install $pip_flags "${source_path}/meson" ; then
+        exit 1
     fi
 fi
 
-if test "$meson" = git; then
-    git_submodules="${git_submodules} meson"
-fi
+# At this point, we expect Meson to be installed and available.
+# We expect mkvenv or pip to have created pyvenv/bin/meson for us.
+# We ignore PATH completely here: we want to use the venv's Meson
+# *exclusively*.
+meson="$(cd pyvenv/bin; pwd)/meson"
 
-case "$meson" in
-    git | internal)
-        meson="$python ${source_path}/meson/meson.py"
-        ;;
-    *) meson=$(command -v "$meson") ;;
-esac
+# Smoke tests: these shouldn't fail, but I have little faith in esoteric
+# platforms I've never used to not do something Deep Strange when I am
+# not looking.
+if ! test -e "$meson" ; then
+    error_exit "??? pyvenv/bin/meson not found, somehow ???"
+fi
+echo "Meson $($meson --version) found: $meson"
+if ! version_ge "$($meson --version)" 0.61.5; then
+    error_exit "??? Meson version is too old ???"
+fi
 
 # Probe for ninja
 
.gitlab-ci.d/buildtest-template.yml +1 -1
index a6cfe9be970..6cfe644d8fe 100644
@@ -17,7 +17,7 @@
       { cat config.log meson-logs/meson-log.txt && exit 1; }
     - if test -n "$LD_JOBS";
       then
-        ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ;
+        pyvenv/bin/meson configure . -Dbackend_max_links="$LD_JOBS" ;
       fi || exit 1;
     - make -j"$JOBS"
     - if test -n "$MAKE_CHECK_ARGS";
Signature · 1 line
2.39.2
#10 lore raw

Re: [RFC PATCH 3/3] configure: install meson to a python virtual environment

Quoted 1 line · John Snow

+ # Is it a problem if we front-load this now and run it again later?

I would just move all the "pip install" bits after configure has called out to git-submodule.sh.

configure if ! check_py_version "$python"; then · line 1177

Quoted 10 lines
configure +8 -0
+    # --no-build-isolation was added to pip 10.0.
+    # --no-use-pep517 was added ... sometime after 18.1?
+    pip_flags='--no-build-isolation'
+    if $python -m pip install --help | grep 'no-use-pep517' > /dev/null 2>&1 ; then
+        pip_flags="${pip_flags} --no-use-pep517"
+    fi
+    if ! pip_install $pip_flags "${source_path}/meson" ; then
+        exit 1
      fi
  fi

Hmm, I'm not a huge fan of using this flag to cover (essentially) an implementation detail of pip. I would really keep on using either meson.py here, though I appreciate that this patch is effective in showing off the venv concept, without getting into downloading packages and adding flags to the configure script.

configure if ! check_py_version "$python"; then · line 1194

Quoted 11 lines
configure +10 -0
+# Smoke tests: these shouldn't fail, but I have little faith in esoteric
+# platforms I've never used to not do something Deep Strange when I am
+# not looking.
+if ! test -e "$meson" ; then
+    error_exit "??? pyvenv/bin/meson not found, somehow ???"
+fi
+echo "Meson $($meson --version) found: $meson"
+if ! version_ge "$($meson --version)" 0.61.5; then
+    error_exit "??? Meson version is too old ???"
+fi

I think this can go away in the final version, since the version is checked against the project() invocation of meson.build.

Paolo

#11 lore raw

Re: [RFC PATCH 3/3] configure: install meson to a python virtual environment

In reply to John Snow
Quoted 27 lines · John Snow

This patch changes how we detect and install meson.

This patch creates a lightweight Python virtual environment using the user's configured $python that inherits system packages. If Meson is installed there and meets our minimum version requirements, we will use that Meson.

In the event that Meson is installed but *not for the chosen Python interpreter*, not found, or of insufficient version, we will attempt to install Meson from source into the newly created Python virtual environment.

At present, the source is obtained in the same manner as it has been; preferring git submodules first and relying on vendored source as a backup.

This patch does *not* connect to PyPI and will work offline for all supported build platforms. Future improvements to this patch may add optional PyPI support as a replacement for git submodules.

As a result of this patch, the Python interpreter we use for both our own build scripts *and* Meson extensions are always known to be the exact same Python. As a further benefit, there will also be a symlink available in the build directory that points to the correct, configured python and can be used by e.g. manual tests to invoke the correct, configured Python unambiguously.

Quoted 3 lines
---
 configure                           | 155 +++++++++++++++++++---------

I would love to see a day where configure isn't involved in the build process. Obviously we can't put this new logic into meson as its a chicken & egg problem. Could we potentially have your new python/scripts/mkvenv.py script be responsible for setting up meson in the venv though, so we can avoid adding more shell code to configure ?

Quoted 217 lines
 .gitlab-ci.d/buildtest-template.yml |   2 +-
 2 files changed, 108 insertions(+), 49 deletions(-)
configure +107 -48
index 05bed4f4a11..1ec054775f6 100755
@@ -607,7 +607,6 @@ check_py_version() {
 python=
 first_python=
 if test -z "${PYTHON}"; then
-    explicit_python=no
     # A bare 'python' is traditionally python 2.x, but some distros
     # have it as python 3.x, so check in both places.
     for binary in python3 python python3.11 python3.10 python3.9 python3.8 python3.7 python3.6; do
@@ -626,7 +625,6 @@ else
     # Same as above, but only check the environment variable.
     has "${PYTHON}" || error_exit "The PYTHON environment variable does not point to an executable"
     python=$(command -v "$PYTHON")
-    explicit_python=yes
     if check_py_version "$python"; then
         # This one is good.
         first_python=
@@ -718,7 +716,7 @@ for opt do
   ;;
   --install=*)
   ;;
-  --python=*) python="$optarg" ; explicit_python=yes
+  --python=*) python="$optarg"
   ;;
   --skip-meson) skip_meson=yes
   ;;
@@ -1078,61 +1076,122 @@ if ! check_py_version "$python"; then
       "Use --python=/path/to/python to specify a supported Python."
 fi
 
-# Resolve PATH + suppress writing compiled files
-python="$(command -v "$python") -B"
-
-has_meson() {
-  local python_dir=$(dirname "$python")
-  # PEP405: pyvenv.cfg is either adjacent to the Python executable
-  # or one directory above
-  if test -f $python_dir/pyvenv.cfg || test -f $python_dir/../pyvenv.cfg; then
-    # Ensure that Meson and Python come from the same virtual environment
-    test -x "$python_dir/meson" &&
-      test "$(command -v meson)" -ef "$python_dir/meson"
-  else
-    has meson
-  fi
+# Resolve PATH
+python="$(command -v "$python")"
+# This variable is intended to be used only for error messages:
+target_python=$python
+
+# Create a Python virtual environment using our configured python.
+# The stdout of this script will be the location of a symlink that
+# points to the configured Python.
+# Entry point scripts for pip, meson, and sphinx are generated if those
+# packages are present.
+
+# Defaults assumed for now:
+# - venv is cleared if it exists already;
+# - venv is allowed to use system packages;
+# - all setup is performed **offline**;
+# - No packages are installed by default;
+# - pip is not installed into the venv when possible,
+#   but ensurepip is called as a fallback when necessary.
+
+echo "python determined to be '$python'"
+echo "python version: $($python --version)"
+
+echo " === pip list (host) === "
+$python -m pip list --disable-pip-version-check
+echo " ======================= "
+echo ""
+
+python="$($python -B "${source_path}/python/scripts/mkvenv.py" create --gen pip,meson,sphinx pyvenv)"
+if test "$?" -ne 0 ; then
+    error_exit "python venv creation failed"
+fi
+
+# Suppress writing compiled files
+python="$python -B"
+echo "Configured python as '$python'"
+
+echo " === pip list (venv) === "
+$python -m pip list --disable-pip-version-check
+echo " ======================= "
+echo ""
+
+pip_install() {
+    $python -m pip install -v \
+            --disable-pip-version-check \
+            --no-index \
+            "$@"
 }
 
-if test -z "$meson"; then
-    if test "$explicit_python" = no && has_meson && version_ge "$(meson --version)" 0.61.5; then
-        meson=meson
-    elif test "$git_submodules_action" != 'ignore' ; then
-        meson=git
-    elif test -e "${source_path}/meson/meson.py" ; then
-        meson=internal
+# OK, let's have some fun!
+
+# This install command is intended to either fail or be a NOP;
+# because we're offline, it's just a convenient version check.
+if ! pip_install 'meson>=0.61.5'; then
+    # Either we don't have Meson, or our Meson is too old.
+    # (Future revisions of this patchset can be less chatty.)
+    if test -e pyvenv/bin/meson; then
+        echo "Meson in pyvenv is too old: $(pyvenv/bin/meson --version)"
+    elif has meson ; then
+        echo "Meson was found installed on your system," \
+             "but not for the configured Python interpreter ($target_python)."
+        echo "(Hint: check '$(which meson)' to see which interpreter its shebang uses.)"
     else
-        if test "$explicit_python" = yes; then
-            error_exit "--python requires using QEMU's embedded Meson distribution, but it was not found."
-        else
-            error_exit "Meson not found.  Use --meson=/path/to/meson"
+        echo "Meson was not found."
+    fi
+
+    # OK, but can we fix it, though? :~)
+    if test "$git_submodules_action" != 'ignore' ; then
+        git_submodules="${git_submodules} meson"
+        echo "Attempting to install meson from git submodule ..."
+        # Stolen from later in the configure file.
+        # Is it a problem if we front-load this now and run it again later?
+        if ! (GIT="$git" "$source_path/scripts/git-submodule.sh" "$git_submodules_action" "$git_submodules"); then
+            exit 1
         fi
+    elif test -e "${source_path}/meson/setup.cfg" ; then
+        echo "Attempting to install meson from vendored source ..."
+    else
+        # In the future, we could use PyPI as a source if the user allows it.
+        # For now, you're outta luck!
+        error_exit "A suitable version of Meson was not found."
     fi
-else
-    # Meson uses its own Python interpreter to invoke other Python scripts,
-    # but the user wants to use the one they specified with --python.
+
+    # If we're here, we have the meson source and we can attempt to
+    # install it into our venv.
+
+    # We want to install meson with --no-use-pep517 if possible,
+    # because it avoids needing a 'wheel' dependency. Old versions
+    # of pip do this by default, so test for the behavior.
     #
-    # We do not want to override the distro Python interpreter (and sometimes
-    # cannot: for example in Homebrew /usr/bin/meson is a bash script), so
-    # just require --meson=git|internal together with --python.
-    if test "$explicit_python" = yes; then
-        case "$meson" in
-            git | internal) ;;
-            *) error_exit "--python requires using QEMU's embedded Meson distribution." ;;
-        esac
+    # --no-build-isolation was added to pip 10.0.
+    # --no-use-pep517 was added ... sometime after 18.1?
+    pip_flags='--no-build-isolation'
+    if $python -m pip install --help | grep 'no-use-pep517' > /dev/null 2>&1 ; then
+        pip_flags="${pip_flags} --no-use-pep517"
+    fi
+    if ! pip_install $pip_flags "${source_path}/meson" ; then
+        exit 1
     fi
 fi
 
-if test "$meson" = git; then
-    git_submodules="${git_submodules} meson"
-fi
+# At this point, we expect Meson to be installed and available.
+# We expect mkvenv or pip to have created pyvenv/bin/meson for us.
+# We ignore PATH completely here: we want to use the venv's Meson
+# *exclusively*.
+meson="$(cd pyvenv/bin; pwd)/meson"
 
-case "$meson" in
-    git | internal)
-        meson="$python ${source_path}/meson/meson.py"
-        ;;
-    *) meson=$(command -v "$meson") ;;
-esac
+# Smoke tests: these shouldn't fail, but I have little faith in esoteric
+# platforms I've never used to not do something Deep Strange when I am
+# not looking.
+if ! test -e "$meson" ; then
+    error_exit "??? pyvenv/bin/meson not found, somehow ???"
+fi
+echo "Meson $($meson --version) found: $meson"
+if ! version_ge "$($meson --version)" 0.61.5; then
+    error_exit "??? Meson version is too old ???"
+fi
 
 # Probe for ninja
 
.gitlab-ci.d/buildtest-template.yml +1 -1
index a6cfe9be970..6cfe644d8fe 100644
@@ -17,7 +17,7 @@
       { cat config.log meson-logs/meson-log.txt && exit 1; }
     - if test -n "$LD_JOBS";
       then
-        ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ;
+        pyvenv/bin/meson configure . -Dbackend_max_links="$LD_JOBS" ;
       fi || exit 1;
     - make -j"$JOBS"
     - if test -n "$MAKE_CHECK_ARGS";
Signature · 1 line
2.39.2

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#12 lore raw

Re: [RFC PATCH 3/3] configure: install meson to a python virtual environment

Quoted 6 lines · Daniel P. Berrangé

I would love to see a day where configure isn't involved in the build process. Obviously we can't put this new logic into meson as its a chicken & egg problem. Could we potentially have your new python/scripts/mkvenv.py script be responsible for setting up meson in the venv though, so we can avoid adding more shell code to configure ?

Not sure this is the same thing a what you want, but I do have a pipedream of rewriting configure in Python. At this point it has a lot more logic than it has command invocations.

John, do you think it makes sense to do the "pip install" steps already in mkvenv's post_post_setup phase? Maybe by passing a version spec like meson>=0.61.5 in place of the simpler "--gen pip,meson,sphinx" that you have now?

Not for this series; just dumping some thoughts prompted by Daniel's question.

Paolo

#13 lore raw

Re: [RFC PATCH 3/3] configure: install meson to a python virtual environment

Quoted 11 lines · Paolo Bonzini
Quoted 6 lines · Daniel P. Berrangé

I would love to see a day where configure isn't involved in the build process. Obviously we can't put this new logic into meson as its a chicken & egg problem. Could we potentially have your new python/scripts/mkvenv.py script be responsible for setting up meson in the venv though, so we can avoid adding more shell code to configure ?

Not sure this is the same thing a what you want, but I do have a pipedream of rewriting configure in Python. At this point it has a lot more logic than it has command invocations.

In a choice between 'shell configure' and 'python configure', I'll take the python version, as it is a fundamentally better language to be writing anything non-trivial in. My desire is to see the elimination of as much shell code as possible. As a broad goal, python should be the only scripting language we aim to use, in preference to shell or perl or any equivalent.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
author
#14 lore raw

Re: [RFC PATCH 3/3] configure: install meson to a python virtual environment

Quoted 14 lines · Daniel P. Berrangé
Quoted 9 lines · Paolo Bonzini
Quoted 6 lines · Daniel P. Berrangé

I would love to see a day where configure isn't involved in the build process. Obviously we can't put this new logic into meson as its a chicken & egg problem. Could we potentially have your new python/scripts/mkvenv.py script be responsible for setting up meson in the venv though, so we can avoid adding more shell code to configure ?

Not sure this is the same thing a what you want, but I do have a

pipedream

Quoted 2 lines

of rewriting configure in Python. At this point it has a lot more logic than it has command invocations.

Quoted 7 lines

In a choice between 'shell configure' and 'python configure', I'll take the python version, as it is a fundamentally better language to be writing anything non-trivial in. My desire is to see the elimination of as much shell code as possible. As a broad goal, python should be the only scripting language we aim to use, in preference to shell or perl or any equivalent.

I certainly prefer python to shell; I can try to move as much logic as I can into mkvenv.

For this RFC I tried to "disrupt" as little as I could so it was easier to see the through-line and gain a sense for what exactly was changing. If there's an appetite for being bolder and more aggressive, we can certainly do that. I'm going to work out the kinks for the baseline functionality first, then we can apply the lipstick.

Thanks,
(And sorry for the hiatus again. I haven't been feeling well, please forgive my spotty attention.)

--js

Quoted 11 lines

With regards,
Daniel
--
|: https://berrange.com -o-
https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-
https://fstop138.berrange.com :|
|: https://entangle-photo.org -o-
https://www.instagram.com/dberrange :|

#15 lore raw
In reply to John Snow

Thanks for posting this!

Quoted 9 lines · John Snow

This series is in RFC state; some notable things:
- So far, only meson is included in this setup.
- There's a ton of debugging prints everywhere. It's extremely chatty right now.
- Sphinx and qemu.qmp are not yet included in this venv.
(qemu.qmp isn't able to be removed from the tree yet.)
- Testing isn't yet*fully* switched over.
- There is no online functionality yet, this series is *100% offline* - it's the harder option, so I tackled it first.

Quoted 5 lines

Some known bugs as of now:
- venv-in-venv setups are not yet handled.
- python3.7 setups without setuptools/pip in the host environment may be unable to generate script file shims; a workaround is in development but wasn't ready today. I decided to exclude it.

I don't think that's needed at all---requiring setuptools on those older Python versions is good enough and can be mentioned in either an error message or the release notes. But out of curiosity what was the workaround?

I think the steps for the first working version should be:

* fix venv-in-venv using the .pth trick

* undo the meson parts from PATCH 3; make patch 3 create the venv + subsume the MKVENV parts of the Makefiles + always set explicit_python=yes (so that at this point the in-tree meson is always used).

* add a patch that starts rejecting --meson=/path/to/meson and drops explicit_python (instead using pyvenv/bin/meson to check whether a system meson is usable)

* make Meson use a sphinx-build binary from the virtual environment (i.e. pass -Dsphinx_build=$PWD/pyvenv/bin/sphinx-build)

These are the bare minimum needed to drop Python 3.6 support in QEMU 8.1.

Paolo

author
#16 lore raw
Quoted 19 lines · Paolo Bonzini

Thanks for posting this!

Quoted 3 lines · John Snow

This series is in RFC state; some notable things:
- So far, only meson is included in this setup.
- There's a ton of debugging prints everywhere. It's extremely chatty

right now.

Quoted 11 lines

- Sphinx and qemu.qmp are not yet included in this venv. (qemu.qmp isn't able to be removed from the tree yet.)
- Testing isn't yet*fully* switched over.
- There is no online functionality yet, this series is *100% offline* - it's the harder option, so I tackled it first.

Some known bugs as of now:
- venv-in-venv setups are not yet handled.
- python3.7 setups without setuptools/pip in the host environment may be unable to generate script file shims; a workaround is in development but wasn't ready today. I decided to exclude it.

Quoted 5 lines

I don't think that's needed at all---requiring setuptools on those older Python versions is good enough and can be mentioned in either an error message or the release notes. But out of curiosity what was the workaround?

Run the shim generation inside the venv, alongside checkpip.

That way we know we have setuptools.

(It broke for some other reason I didn't diagnose in time to send.)

Quoted 4 lines

I think the steps for the first working version should be:

* fix venv-in-venv using the .pth trick

Yep.

Quoted 5 lines

* undo the meson parts from PATCH 3; make patch 3 create the venv + subsume the MKVENV parts of the Makefiles + always set explicit_python=yes (so that at this point the in-tree meson is always used).

Quoted 7 lines

* add a patch that starts rejecting --meson=/path/to/meson and drops explicit_python (instead using pyvenv/bin/meson to check whether a system meson is usable)

* make Meson use a sphinx-build binary from the virtual environment (i.e. pass -Dsphinx_build=$PWD/pyvenv/bin/sphinx-build)

Yep, let's talk about this part in particular.

Quoted 5 lines

These are the bare minimum needed to drop Python 3.6 support in QEMU 8.1.

Paolo

#17 lore raw
Quoted 9 lines · John Snow

* undo the meson parts from PATCH 3; make patch 3 create the venv + subsume the MKVENV parts of the Makefiles + always set explicit_python=yes (so that at this point the in-tree meson is always used).

* add a patch that starts rejecting --meson=/path/to/meson and drops explicit_python (instead using pyvenv/bin/meson to check whether a system meson is usable)

Quoted 5 lines

* make Meson use a sphinx-build binary from the virtual environment (i.e. pass -Dsphinx_build=$PWD/pyvenv/bin/sphinx-build)

Yep, let's talk about this part in particular.

Oh, wait, for this one I already have a patch from my experiment that used importlib.metadata to look up the entry point dynamically[1] (and that's where the shim idea developed from). All I need to do is change the path passed to find_program() and rewrite the commit message.

Paolo

[1] https://lore.kernel.org/qemu-devel/[email protected]/

--------------------------- 8< --------------------------------
From: Paolo Bonzini <[email protected]>
Subject: [PATCH] meson: pick sphinx-build from virtual environment

configure is now creating a virtual environment and populating it with shim binaries that always refer to the correct Python runtime. docs/meson.build can rely on this, and stop using a sphinx_build option that may or may not refer to the same version of Python that is used for the rest of the build.

In the long term, it may actually make sense for Meson's Python module to include the logic to build such shims, so that other programs can do the same without needing a full-blown virtual environment. However, in the context of QEMU there is no need to wait for that; QEMU's meson.build already relies on config-host.mak and on the target list that configure prepares, i.e. it is not standalone.

docs/conf.py +0 -9
index 7e215aa9a5c6..c687ff266301 100644
@@ -32,15 +32,6 @@
 from distutils.version import LooseVersion
 from sphinx.errors import ConfigError
 
-# Make Sphinx fail cleanly if using an old Python, rather than obscurely
-# failing because some code in one of our extensions doesn't work there.
-# In newer versions of Sphinx this will display nicely; in older versions
-# Sphinx will also produce a Python backtrace but at least the information
-# gets printed...
-if sys.version_info < (3,7):
-    raise ConfigError(
-        "QEMU requires a Sphinx that uses Python 3.7 or better\n")
-
 # The per-manual conf.py will set qemu_docdir for a single-manual build;
 # otherwise set it here if this is an entire-manual-set build.
 # This is always the absolute path of the docs/ directory in the source tree.
docs/meson.build +0 -1
index f220800e3e59..1c5fd66bfa7f 100644
@@ -1,5 +1,6 @@
-sphinx_build = find_program(get_option('sphinx_build'),

# This assumes that Python is inside the venv that configure prepares +sphinx_build = find_program(fs.parent(python.full_path()) / 'sphinx-build',
required: get_option('docs'))

# Check if tools are available to build documentation.

meson_options.txt +0 -2
index b541ab2851dd..8dedec0cf91a 100644
@@ -12,8 +12,6 @@ option('pkgversion', type : 'string', value : '',
        description: 'use specified string as sub-version of the package')
 option('smbd', type : 'string', value : '',
        description: 'Path to smbd for slirp networking')
-option('sphinx_build', type : 'string', value : 'sphinx-build',
-       description: 'Use specified sphinx-build for building document')
 option('iasl', type : 'string', value : '',
        description: 'Path to ACPI disassembler')
 option('tls_priority', type : 'string', value : 'NORMAL',
scripts/meson-buildoptions.sh +0 -3
index bf852f4b957e..6a71c3bad296 100644
@@ -58,8 +58,6 @@ meson_options_help() {
   printf "%s\n" '  --localedir=VALUE        Locale data directory [share/locale]'
   printf "%s\n" '  --localstatedir=VALUE    Localstate data directory [/var/local]'
   printf "%s\n" '  --mandir=VALUE           Manual page directory [share/man]'
-  printf "%s\n" '  --sphinx-build=VALUE     Use specified sphinx-build for building document'
-  printf "%s\n" '                           [sphinx-build]'
   printf "%s\n" '  --sysconfdir=VALUE       Sysconf data directory [etc]'
   printf "%s\n" '  --tls-priority=VALUE     Default TLS protocol/cipher priority string'
   printf "%s\n" '                           [NORMAL]'
@@ -429,7 +427,6 @@ _meson_option_parse() {
     --disable-sndio) printf "%s" -Dsndio=disabled ;;
     --enable-sparse) printf "%s" -Dsparse=enabled ;;
     --disable-sparse) printf "%s" -Dsparse=disabled ;;
-    --sphinx-build=*) quote_sh "-Dsphinx_build=$2" ;;
     --enable-spice) printf "%s" -Dspice=enabled ;;
     --disable-spice) printf "%s" -Dspice=disabled ;;
     --enable-spice-protocol) printf "%s" -Dspice_protocol=enabled ;;
Signature · 1 line
2.39.2