Functional/integration testing

new file mode 100644
index 000000000..e23b1eeb9
@@ -0,0 +1,331 @@
+===============
+test-functional
+===============
+
+**test-functional** [*OPTIONS*]
+
+DESCRIPTION
+===========
+
+**test-functional(1)** is used for functional testing of BlueZ and
+kernel using multiple virtual machine environments, connected by real
+or virtual controllers.
+
+It uses https://pypi.org/project/pytest-bluezenv as VM-based test
+framework. For details, see its documentation.
+
+QUICK EXAMPLE
+=============
+
+Install `qemu-system-x86_64` first. Then,
+
+.. code-block::
+
+   $ python3 -mpip install -r test/functional/requirements.txt
+   $ ./configure --enable-functional-testing --enable-testing --enable-tools
+   $ make -j8
+   $ test/test-functional --kernel-build -v
+
+Or, if you already have a kernel image:
+
+.. code-block::
+
+   $ test/test-functional --kernel /pathto/bzImage -v
+
+OPTIONS
+=======
+
+The `test-functional` script simply runs `Pytest
+<https://pytest.org>`__ which can take the following options:
+https://docs.pytest.org/en/stable/reference/reference.html#command-line-flags
+
+The following additional options apply:
+
+``--list``
+	Output brief lists of existing tests.
+
+``--kernel=<image>``
+        Kernel image (or built Linux source tree root) to
+	use.  See **test-runner(1)** and `tester.config` for required
+	kernel config.
+
+	If not provided, value from `FUNCTIONAL_TESTING_KERNEL`
+	environment variable is used. If none, no image is used.
+
+``--usb=hci0,hci1``
+        USB controllers to use in tests that require use of
+	real controllers.
+
+	If not provided, value from `FUNCTIONAL_TESTING_CONTROLLERS`
+	environment variable is used. If none, all USB controllers
+	with suitable permissions are considered.
+
+``--force-usb``
+        Force tests to use USB controllers instead of `btvirt`.
+
+``--bluez-build-dir=<path>``
+        Path to build directory where to search for BlueZ
+        executables.
+
+``--bluez-src-dir=<path>``
+        Path to build BlueZ source directory.
+
+``--log-filter=[+-]<pattern>,[+-]<pattern>,...``
+        Allow/deny lists
+	for filtering logging output. The pattern is a shell glob matching
+	to the logger names.
+
+``--no-log-reorder``
+	Don't reorder logs to timestamp order.
+
+``--vm-timeout=<seconds>``
+        Specify timeout for communication with VM hosts.
+
+``--btmon``
+        Launch btmon on all hosts to log events, and dump traffic to
+	test-bluezenv-\*.btsnoop
+
+``--kernel-build=no/use/auto/force``
+        Build a suitable kernel image from source.
+
+``--kernel-upstream=<GIT_URL>``
+        URL for Git clone of kernel sources.
+
+``--kernel-branch=<GIT_BRANCH>``
+        Git branch to build from.
+
+
+Tests that require kernel image or USB controllers are skipped if none
+are available. Normally, tests use `btvirt`.
+
+VM instances share a directory ``/run/shared`` with host machine,
+located on host usually in ``/tmp/bluez-func-test-*/shared-*``.  Core
+dumps etc. are copied out from it before test instance is shut down.
+
+REQUIREMENTS
+============
+
+General
+-------
+
+The following are needed:
+
+- QEmu (x86_64)
+- ``dbus-daemon`` available
+
+Recommended:
+
+- KVM-enabled x86_64 host system
+- Preferably built BlueZ source tree
+- ``chronyd`` available
+- ``util-linux`` tools available
+- ``agetty`` available
+
+Python
+------
+
+The following Python packages are required:
+
+.. code-block::
+
+	pytest>=8
+	pytest-bluezenv==0.1.6
+
+To install them via pip::
+
+	python3 -m pip install -r test/functional/requirements.txt
+
+On Fedora / RHEL, the dependencies aside from `pytest-bluezenv` can be
+installed via::
+
+	sudo dnf install python3-pytest python3-pexpect python3-dbus
+
+Kernel
+------
+
+The **test-functional(1)** tool requires a kernel image with similar
+config as **test-runner(1)**.  If given `--kernel-build` option, a
+suitable image is built from sources downloaded under
+`test/.pytest_cache`.
+
+Simplest setup is
+
+.. code-block::
+
+	cp ../bluez/doc/tester.config .config
+	make olddefconfig
+	make -j8
+
+To get log timestamps right, the kernel should have the following
+configuration enabled:
+
+.. code-block::
+
+	CONFIG_HYPERVISOR_GUEST=y
+	CONFIG_PARAVIRT=y
+	CONFIG_KVM_GUEST=y
+
+	CONFIG_PTP_1588_CLOCK=y
+	CONFIG_PTP_1588_CLOCK_KVM=y
+	CONFIG_PTP_1588_CLOCK_VMCLOCK=y
+
+USB
+---
+
+Some tests may require a hardware controller instead of the virtual `btvirt` one.
+
+EXAMPLES
+========
+
+Run all tests
+-------------
+
+.. code-block::
+
+	$ test/test-functional --kernel=/pathto/bzImage
+
+	$ export FUNCTIONAL_TESTING_KERNEL=/pathto/bzImage
+	$ test/test-functional
+
+Test output is logged in ``test-functional.log``.
+
+Show output during run
+----------------------
+
+.. code-block::
+
+	$ test/test-functional --log-cli-level=0
+
+Show only specific loggers:
+
+.. code-block::
+
+	$ test/test-functional --log-cli-level=0 --log-filter=rpc,host
+
+	$ test/test-functional --log-cli-level=0 --log-filter=*.bluetoothctl
+
+Filter out loggers:
+
+.. code-block::
+
+	$ test/test-functional --log-cli-level=0 --log-filter=-host
+
+	$ test/test-functional --log-cli-level=0 --log-filter=host,-host.*.1
+
+Run selected tests
+------------------
+
+.. code-block::
+
+	$ test/test-functional test/functional/test_cli_simple.py::test_bluetoothctl_script_show
+
+	$ test/test-functional -k test_bluetoothctl_script_show
+
+	$ test/test-functional -k 'test_btmgmt or test_bluetoothctl'
+
+	$ test/test-functional -k 'not btmgmt'
+
+Don't run tests with a given marker (see `test/pytest.ini` for defined markers):
+
+.. code-block::
+
+	$ test/test-functional -m "not tester"
+
+To exclude kernel testers from the run.
+E.g. run only security advisory regression tests:
+
+.. code-block::
+
+	$ test/test-functional -m sa
+
+Don't run known-failing tests:
+
+.. code-block::
+
+	$ test/test-functional -m "not xfail"
+
+Note that otherwise known-failing tests would be run, but with
+failures suppressed.
+
+Run previously failed and stop on failure
+-----------------------------------------
+
+.. code-block::
+
+	$ test/test-functional -x --ff
+
+List all tests
+--------------
+
+.. code-block::
+
+	$ test/test-functional --list
+
+This can be combined with the test selection options above.
+
+Show errors from know-failing test
+----------------------------------
+
+.. code-block::
+
+	$ test/test-functional --runxfail -k test_btmgmt_info
+
+Redirect USB devices
+--------------------
+
+.. code-block::
+
+	$ test/test-functional --usb=hci0,hci1
+
+	$ export FUNCTIONAL_TESTING_CONTROLLERS=hci0,hci1
+	$ test/test-functional -vv
+
+This does not require running as root. Changing device permissions is
+sufficient. In verbose mode (``-vv``) some instructions are printed.
+
+Tests will still prefer using emulated btvirt controller when
+possible.  To force all tests use the USB controllers:
+
+.. code-block::
+
+	$ test/test-functional --usb=hci0,hci1 --force-usb
+
+Run tests in parallel
+---------------------
+
+pytest-xdist is required for parallel execution. To run:
+
+.. code-block::
+
+	$ test/test-functional -n auto --dist loadgroup
+
+Logging in to a test VM instance
+--------------------------------
+
+While test is running:
+
+.. code-block::
+
+	$ test/test-functional-attach
+
+For this to be useful, usually, you need to pause the test
+e.g. by running with ``--trace`` option.
+
+To do it manually, when starting the tester will log a line like::
+
+	TTY: socat /tmp/bluez-func-test-q658swgi/bluez-func-test-tty-0 STDIO,rawer
+
+with the location of the socket where the serial is connected to.
+
+WRITING TESTS
+=============
+
+The functional tests are written in files (test modules) names
+`test/functional/test_*.py`.  They are written using standard Pytest
+style.  See https://docs.pytest.org/en/stable/getting-started.html
+
+See https://pypi.org/project/pytest-bluezenv/ for documentation of
+how to write VM-using tests.
+
+Use `Black <https://black.readthedocs.io/en/stable/>`__ to autoformat
+Python test code.

new file mode 100644
index 000000000..fe1c85178
@@ -0,0 +1,2 @@
+# -*- coding: utf-8; mode: python; eval: (blacken-mode); -*-
+# SPDX-License-Identifier: GPL-2.0-or-later

58 lines not on this page · raw message

new file mode 100644
index 000000000..e0e3bfee6
@@ -0,0 +1,2 @@
+pytest>=8
+pytest-bluezenv==0.1.9

161 lines not on this page · raw message

new file mode 100644
index 000000000..00cf3c18c
@@ -0,0 +1,34 @@
+# -*- coding: utf-8; mode: python; eval: (blacken-mode); -*-
+# SPDX-License-Identifier: GPL-2.0-or-later
+"""
+Tests for btmgmt using VM instances
+"""
+
+import sys
+import pytest
+import subprocess
+import tempfile
+
+from pytest_bluezenv import host_config, find_exe, run
+
+pytestmark = [pytest.mark.vm]
+
+
+@host_config([])
+def test_btmgmt_info(hosts):
+    (host,) = hosts
+
+    try:
+        btmgmt = find_exe("tools", "btmgmt")
+    except FileNotFoundError:
+        pytest.skip("btmgmt not found")
+
+    result = host.call(
+        run,
+        [btmgmt, "--index", "0", "info"],
+        stdout=subprocess.PIPE,
+        stdin=subprocess.DEVNULL,
+        encoding="utf-8",
+    )
+    assert result.returncode == 0
+    assert f"addr {host.bdaddr.upper()}" in result.stdout

new file mode 100644
index 000000000..92820e819
@@ -0,0 +1,23 @@
+[pytest]
+log_format = %(asctime)s %(levelname)-6s %(name)-20s:  %(message)s
+log_date_format = %Y-%m-%d %H:%M:%S.%f
+log_level = 0
+log_file = test-functional.log
+markers =
+    vm: tests requiring VM image
+
+addopts =
+    -p pytest_bluezenv
+
+# Default timeout
+vm_timeout = 45
+vm_mem = 400M
+
+# Default sources for kernel-build when requested
+kernel_upstream = https://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth-next.git/
+kernel_branch = master
+
+filterwarnings =
+    error::pytest_bluezenv.KernelBugWarning
+    error::pytest_bluezenv.SanitizerWarning
+    error::pytest_bluezenv.CoredumpWarning

new file mode 100755
index 000000000..95f5c57e9
@@ -0,0 +1,21 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# See doc/test-functional.rst
+#
+TESTDIR="$(dirname "$0")"
+SRCDIR="$TESTDIR/.."
+
+BUILDDIR=
+for d in "$SRCDIR" "$SRCDIR/build" "$SRCDIR/builddir"; do
+    if [ -f "$d/src/bluetoothd" ]; then
+        BUILDDIR="$d"
+        break
+    fi
+done
+
+if [ -n "$BUILDDIR" ] && [ -d "$BUILDDIR" ]; then
+    exec python3 -m pytest "$TESTDIR/functional" --bluez-src-dir "$SRCDIR" --bluez-build-dir "$BUILDDIR" "$@"
+else
+    exec python3 -m pytest "$TESTDIR/functional" --bluez-src-dir "$SRCDIR" "$@"
+fi

new file mode 100755
index 000000000..6e65464f7
@@ -0,0 +1,7 @@
+#!/bin/sh
+#
+# test-functional-attach
+#
+# Start Tmux and connect to active test-functional VM hosts.
+#
+exec python3 -mpytest_bluezenv attach "$@"

index 723210f93..e152ead0a 100644
@@ -857,6 +857,16 @@ endif
 TESTS = $(unit_tests)
 AM_TESTS_ENVIRONMENT = MALLOC_CHECK_=3 MALLOC_PERTURB_=69
 
+check-functional: all
+	python3 -m pytest "$(srcdir)/test/functional" -v \
+		--kernel="$(FUNCTIONAL_TESTING_KERNEL)" \
+		--bluez-build-dir="$(top_builddir)" \
+		--bluez-src-dir="$(srcdir)"
+
+if FUNCTIONAL_TESTING
+check: check-functional
+endif
+
 if DBUS_RUN_SESSION
 AM_TESTS_ENVIRONMENT += dbus-run-session --
 endif

index b011e9b0a..b1354d7f8 100644
@@ -407,6 +407,28 @@ if (test "${enable_testing}" = "yes"); then
 		#include <linux/net_tstamp.h>]])
 fi
 
+AC_ARG_ENABLE(functional-testing, AS_HELP_STRING([--enable-functional-testing],
+			[enable functional testing tools]),
+			[enable_functional_testing=yes; functional_testing_kernel=${enableval}],
+                        [enable_functional_testing=no])
+AM_CONDITIONAL(FUNCTIONAL_TESTING, test "${enable_functional_testing}" = "yes")
+AC_ARG_VAR(FUNCTIONAL_TESTING_KERNEL, [vmlinux image to use for functional testing])
+FUNCTIONAL_TESTING_KERNEL=${functional_testing_kernel}
+
+if (test "${enable_functional_testing}" = "yes"); then
+  if (test "${enable_client}" = "no" || \
+      test "${enable_tools}" != "yes" || \
+      test "${enable_testing}" != "yes"); then
+    AC_MSG_ERROR([--enable-functional-testing requires --enable-client --enable-tools --enable-testing])
+  fi
+  AC_MSG_CHECKING([pytest and dependencies])
+  python3 -m pip install --dry-run --no-index -r "${srcdir}/test/functional/requirements.txt" >/dev/null
+  if (test "$?" != "0"); then
+    AC_MSG_ERROR([pytest or dependencies missing])
+  fi
+  AC_MSG_RESULT([ok])
+fi
+
 AC_ARG_ENABLE(experimental, AS_HELP_STRING([--enable-experimental],
 			[enable experimental tools]),
 					[enable_experimental=${enableval}])

index e0e3bfee6..b8a1d99ee 100644
@@ -1,2 +1,3 @@
 pytest>=8
 pytest-bluezenv==0.1.9
+black

new file mode 100644
index 000000000..d196cb1d7
@@ -0,0 +1,24 @@
+# -*- coding: utf-8; mode: python; eval: (blacken-mode); -*-
+# SPDX-License-Identifier: GPL-2.0-or-later
+"""
+Tests for the test suite itself
+"""
+
+import sys
+import subprocess
+import warnings
+from pathlib import Path
+
+import pytest
+
+
+def test_formatting():
+    pytest.importorskip("black")
+
+    result = subprocess.run(
+        [sys.executable, "-mblack", "--check", "--diff", "-q", Path(__file__).parent],
+        stdout=subprocess.PIPE,
+        encoding="utf-8",
+    )
+    if result.returncode != 0:
+        warnings.warn(f"Formatting incorrect:\n{result.stdout}")

new file mode 100644
index 000000000..f2ce34576
@@ -0,0 +1,47 @@
+# -*- coding: utf-8; mode: python; eval: (blacken-mode); -*-
+# SPDX-License-Identifier: GPL-2.0-or-later
+"""
+Tests for bluetoothctl using VM instances
+"""
+
+import sys
+import re
+import pytest
+import subprocess
+import tempfile
+
+import time
+import logging
+
+
+from pytest_bluezenv import host_config, Agent, wait_until
+
+pytestmark = [pytest.mark.vm]
+
+
+@host_config([Agent()], [Agent()])
[email protected]("success", [True, False], ids=["accept", "reject"])
+def test_agent_pair_bredr(hosts, success):
+    host0, host1 = hosts
+
+    host0.agent.adapter_method("StartDiscovery")
+    host0.agent.expect("org.bluez.Adapter1.StartDiscovery:reply")
+
+    host1.agent.adapter_set("Pairable", True)
+    host1.agent.adapter_set("Discoverable", True)
+
+    wait_until(host0.agent.has_device, host1.bdaddr)
+
+    host0.agent.device_method(host1.bdaddr, "Pair")
+
+    confirm_0 = host0.agent.expect("org.bluez.Agent1.RequestConfirmation")
+    confirm_1 = host1.agent.expect("org.bluez.Agent1.RequestConfirmation")
+    assert confirm_0.passkey == confirm_1.passkey
+    host0.agent.reply()
+
+    if success:
+        host1.agent.reply()
+        host0.agent.expect("org.bluez.Device1.Pair:reply")
+    else:
+        host1.agent.reply_error()
+        host0.agent.expect("org.bluez.Device1.Pair:error")

new file mode 100644
index 000000000..345b61326
@@ -0,0 +1,286 @@
+# -*- coding: utf-8; mode: python; eval: (blacken-mode); -*-
+# SPDX-License-Identifier: LGPL-2.1-or-later
+"""
+Tests for Obex
+"""
+
+import sys
+import os
+import re
+import pytest
+import subprocess
+import tempfile
+import time
+import logging
+import json
+import dbus
+import threading
+from pathlib import Path
+
+import pytest
+
+from pytest_bluezenv import (
+    HostPlugin,
+    Agent,
+    host_config,
+    find_exe,
+    Bluetoothd,
+    Bluetoothctl,
+    Obexd,
+    LogStream,
+    wait_until,
+    mainloop_wrap,
+    mainloop_assert,
+    Event,
+    EventPluginMixin,
+    dbus_service_event_method,
+    Pexpect,
+    utils,
+)
+
+pytestmark = [pytest.mark.vm]
+
+log = logging.getLogger(__name__)
+
+
+BUS_NAME = "org.bluez.obex"
+PATH = "/org/bluez/obex"
+AGENT_MANAGER_INTERFACE = "org.bluez.obex.AgentManager1"
+AGENT_INTERFACE = "org.bluez.obex.Agent1"
+CLIENT_INTERFACE = "org.bluez.obex.Client1"
+SESSION_INTERFACE = "org.bluez.obex.Session1"
+FILE_TRANSFER_INTERFACE = "org.bluez.obex.FileTransfer1"
+TRANSFER_INTERFACE = "org.bluez.obex.Transfer1"
+
+FTP_UUID = "00001106-0000-1000-8000-00805f9b34fb"
+
+
+class ObexAgent(HostPlugin, EventPluginMixin):
+    depends = [Bluetoothd()]
+    name = "obex_agent"
+
+    def __init__(self, path="/obexagent"):
+        self.path = path
+
+    @mainloop_wrap
+    def setup(self, impl):
+        EventPluginMixin.setup(self, impl)
+
+        self.bus = dbus.SessionBus()
+        self.bus.set_exit_on_disconnect(False)
+
+        self.agent = ObexAgentObject(self.bus, self.path, self.events)
+
+        bluez = self.bus.get_object(BUS_NAME, PATH)
+        self.manager = dbus.Interface(bluez, AGENT_MANAGER_INTERFACE)
+        self.manager.RegisterAgent(self.path)
+
+        log.info("Obex agent registered")
+
+    def cleanup(self):
+        path = Path("/run/obex")
+        for f in path.iterdir():
+            f.unlink()
+
+
+def agent_method(*a, **kw):
+    return dbus_service_event_method(AGENT_INTERFACE, *a, **kw)
+
+
+class ObexAgentObject(dbus.service.Object):
+    @mainloop_assert
+    def __init__(self, bus, path, events):
+        self.events = events
+        super().__init__(bus, path)
+
+    AuthorizePush = agent_method("AuthorizePush", ("path",), "o", "s", sync=False)
+    Cancel = agent_method("Cancel")
+
+
+def write_obex_file(name, content):
+    with open(f"/run/obex/{name}", "w") as f:
+        f.write(content)
+
+
+def read_file(name):
+    with open(name, "r") as f:
+        return f.read()
+
+
+#
+# Direct Obex Python client API tests
+#
+
+
+class ObexClient(HostPlugin, EventPluginMixin):
+    name = "obex"
+
+    @mainloop_wrap
+    def setup(self, impl):
+        EventPluginMixin.setup(self, impl)
+
+        self.transferred = 0
+        self.transfer_path = None
+        self.transfer_size = 0
+
+        self.bus = dbus.SessionBus()
+        self.bus.set_exit_on_disconnect(False)
+        self.log = logging.getLogger(self.name)
+        self.client = dbus.Interface(
+            self.bus.get_object(BUS_NAME, PATH), CLIENT_INTERFACE
+        )
+
+        self.bus.add_signal_receiver(
+            self.properties_changed,
+            dbus_interface="org.freedesktop.DBus.Properties",
+            signal_name="PropertiesChanged",
+            path_keyword="path",
+        )
+
+    @mainloop_wrap
+    def connect(self, bdaddr):
+        def reply(path):
+            obj = self.bus.get_object(BUS_NAME, path)
+            self.session = dbus.Interface(obj, SESSION_INTERFACE)
+            self.ftp = dbus.Interface(obj, FILE_TRANSFER_INTERFACE)
+
+        self._object_method(
+            self.client, "CreateSession", bdaddr, {"Target": "ftp"}, reply_handler=reply
+        )
+
+    @mainloop_assert
+    def properties_changed(self, interface, properties, invalidated, path):
+        if path != self.transfer_path:
+            return
+
+        if "Status" in properties and (
+            properties["Status"] == "complete" or properties["Status"] == "error"
+        ):
+            self.events.put(
+                Event(
+                    f"{FILE_TRANSFER_INTERFACE}:{properties['Status']}",
+                    properties=properties,
+                )
+            )
+            self.log.debug(f"Transfer {properties['Status']}")
+
+        if "Transferred" not in properties:
+            return
+
+        value = properties["Transferred"]
+        speed = (value - self.transferred) / 1000
+        self.log.debug(
+            f"Transfer progress {value}/{self.transfer_size} at {speed} kBps"
+        )
+        self.transferred = value
+
+    @mainloop_wrap
+    def ftp_list_folder(self):
+        return self.ftp.ListFolder()
+
+    @mainloop_wrap
+    def ftp_get_file(self, dst, src):
+        path, properties = self.ftp.GetFile(dst, src)
+        self.transfer_path = path
+        self.transfer_size = properties["Size"]
+        return properties["Filename"]
+
+
[email protected]
+def paired_hosts(hosts):
+    from .test_agent import test_agent_pair_bredr
+
+    if hosts[0].agent.has_device(hosts[1].bdaddr):
+        return hosts
+
+    test_agent_pair_bredr(hosts, True)
+    return hosts
+
+
+obex_host_config = host_config(
+    [Agent(), Obexd(), ObexClient(), Pexpect()],
+    [Agent(), Obexd(), ObexAgent()],
+    reuse=True,
+)
+
+
[email protected]
+def obex_hosts(paired_hosts):
+    host0, host1 = paired_hosts
+
+    if hasattr(host0, "session"):
+        return paired_hosts
+
+    host0.obex.connect(host1.bdaddr)
+
+    service = host1.agent.expect("org.bluez.Agent1.AuthorizeService")
+    assert service.uuid == FTP_UUID
+    host1.agent.reply()
+
+    host0.obex.expect("org.bluez.obex.Client1.CreateSession:reply")
+
+    yield paired_hosts
+
+    host1.obex_agent.cleanup()
+
+
+@obex_host_config
+def test_obex_ftp_list(obex_hosts):
+    host0, host1 = obex_hosts
+
+    host1.call(write_obex_file, "test", "1234")
+
+    (item,) = host0.obex.ftp_list_folder()
+    assert item["Type"] == "file"
+    assert item["Name"] == "test"
+    assert item["Size"] == 4
+
+
+@obex_host_config
+def test_obex_ftp_get(obex_hosts):
+    host0, host1 = obex_hosts
+
+    host1.call(write_obex_file, "test", "1234")
+
+    filename = host0.obex.ftp_get_file("", "test")
+    host0.obex.expect("org.bluez.obex.FileTransfer1:complete")
+    assert host0.call(read_file, filename) == "1234"
+
+
+#
+# obexctl tests
+#
+
+
[email protected]
+def obexctl(obex_hosts):
+    host0, host1 = obex_hosts
+
+    exe = find_exe("tools", "obexctl")
+    obexctl = host0.pexpect.spawn([exe])
+
+    obexctl.expect("Client /org/bluez/obex")
+    obexctl.send(f"connect {host1.bdaddr} {FTP_UUID}\n")
+
+    service = host1.agent.expect("org.bluez.Agent1.AuthorizeService")
+    assert service.uuid == FTP_UUID
+    host1.agent.reply()
+
+    obexctl.expect("Connection successful")
+    obexctl.send(f"select /org/bluez/obex/client/session1\n")
+
+    yield obexctl
+
+    obexctl.close()
+
+
+@obex_host_config
+def test_obexctl_list(obex_hosts, obexctl):
+    host0, host1 = obex_hosts
+
+    host1.call(write_obex_file, "test", "1234")
+
+    obexctl.send(f"ls\n")
+    obexctl.expect(f"Type: file")
+    obexctl.expect(f"Name: test")
+    obexctl.expect(f"Size: 4")

new file mode 100644
index 000000000..bfdfb03e7
@@ -0,0 +1,178 @@
+# -*- coding: utf-8; mode: python; eval: (blacken-mode); -*-
+# SPDX-License-Identifier: GPL-2.0-or-later
+"""
+Tests for the test suite itself
+"""
+
+import re
+import sys
+import subprocess
+import warnings
+from pathlib import Path
+
+import pytest
+
+from pytest_bluezenv import find_exe, host_config, run
+from pytest_bluezenv.utils import DEFAULT_TIMEOUT
+
+# Testers that can reuse the VM instance
+TESTERS = [
+    "mgmt-tester",
+    "smp-tester",
+    "l2cap-tester",
+    "rfcomm-tester",
+    "sco-tester",
+    "iso-tester",
+    "mesh-tester",
+    "ioctl-tester",
+    "bnep-tester",
+    "userchan-tester",
+    "6lowpan-tester",
+]
+
+XFAIL = {
+    "mesh-tester": {
+        # Always fail:
+        "Mesh - Send cancel - 1",
+        "Mesh - Send cancel - 2",
+    },
+    "mgmt-tester": {
+        # Always fail:
+        "Read Exp Feature - Success",
+        # Flaky, randomly fail:
+        "LL Privacy - Add Device 1 (Add to AL)",
+        "LL Privacy - Add Device 2 (2 Devices to AL)",
+        "LL Privacy - Add Device 3 (AL is full)",
+        "LL Privacy - Set Device Flag 1 (Device Privacy)",
+        "LL Privacy - Set Flags 1 (Add to RL)",
+        "LL Privacy - Set Flags 2 (Enable RL)",
+        "LL Privacy - Set Flags 3 (2 Devices to RL)",
+        "LL Privacy - Set Flags 4 (RL is full)",
+        "LL Privacy - Set Flags 5 (Multi Adv)",
+        "LL Privacy - Set Flags 6 (Multi Dev and Multi Adv)",
+        "LL Privacy - Start Discovery 2 (Disable RL)",
+    },
+}
+
+host_setup = host_config([], controller=False, reuse=True)
+
+
+def run_tester(hosts, tester):
+    (host,) = hosts
+    xfails = set(XFAIL.get(tester, ()))
+
+    tester = find_exe("tools", tester)
+    res = host.call(
+        run,
+        [tester],
+        stdout=subprocess.PIPE,
+        timeout=2 * DEFAULT_TIMEOUT + 60,
+        encoding="utf-8",
+        errors="surrogateescape",
+    )
+
+    passed, failed, not_run = _parse_tester_output(res.stdout)
+    xfailed = []
+
+    for name in list(failed):
+        if name in xfails:
+            failed.remove(name)
+            xfailed.append(name)
+            xfails.remove(name)
+
+    assert failed == [], failed
+
+    if xfails:
+        warnings.warn("XPASS: " + ", ".join(xfails))
+
+    if xfailed:
+        pytest.xfail(reason=", ".join(xfailed))
+
+
[email protected]
[email protected]("tester", TESTERS)
+@host_setup
+def test_kernel_tester(hosts, tester):
+    run_tester(hosts, tester)
+
+
+@host_setup
+def test_kernel_selftest(hosts):
+    (host,) = hosts
+
+    tester = find_exe("tools", "check-selftest")
+    res = host.call(
+        run,
+        [tester],
+        stdout=subprocess.PIPE,
+        timeout=2 * DEFAULT_TIMEOUT + 60,
+        encoding="utf-8",
+        errors="surrogateescape",
+    )
+    assert res.returncode == 0
+
+    if not res.stdout.strip():
+        pytest.skip("CONFIG_BT_SELFTEST not enabled")
+
+    assert "PASS" in res.stdout, res.stdout
+    assert "FAIL" not in res.stdout, res.stdout
+
+
+def _parse_tester_output(output):
+    total_count = 0
+    passed_count = 0
+    failed_count = 0
+    not_run_count = 0
+
+    passed = []
+    failed = []
+    not_run = []
+
+    in_summary = False
+    for line in output.splitlines():
+        line = line.strip()
+
+        if "Test Summary" in line:
+            in_summary = True
+            continue
+        if not in_summary:
+            continue
+        if "----" in line or "Overall execution time" in line or not line:
+            continue
+
+        m = re.search(
+            r"Total: (\d+).*Passed: (\d+).*Failed: (\d+).*Not Run: (\d+)", line
+        )
+        if m:
+            total_count += int(m.group(1))
+            passed_count += int(m.group(2))
+            failed_count += int(m.group(3))
+            not_run_count += int(m.group(4))
+            continue
+
+        m = re.match(r"^([\x20-\xff]+)\s+.*(Passed|Failed|Not Run|Timed out)", line)
+        assert m, line
+
+        name = m.group(1).strip()
+        verdict = m.group(2).strip().lower()
+
+        if verdict == "passed":
+            passed.append(name)
+        elif verdict in ("failed", "timed out"):
+            failed.append(name)
+        elif verdict == "not run":
+            not_run.append(name)
+        else:
+            raise ValueError("Unknown {verdict=!r}")
+
+    # Check we parsed everything
+    if len(passed) != passed_count:
+        raise ValueError(f"Invalid {passed_count=}, {passed=}")
+    if len(failed) != failed_count:
+        raise ValueError(f"Invalid {failed_count=}, {failed=}")
+    if len(not_run) != not_run_count:
+        raise ValueError(f"Invalid {not_run_count=}, {not_run=}")
+    if total_count == 0:
+        raise RuntimeError(f"No tests ran")
+
+    return passed, failed, not_run

index 92820e819..6b97c6525 100644
@@ -5,6 +5,7 @@ log_level = 0
 log_file = test-functional.log
 markers =
     vm: tests requiring VM image
+    tester: kernel testers
 
 addopts =
     -p pytest_bluezenv

index bd26b7e4f..9e3f6c50d 100644
@@ -50,9 +50,15 @@ def pytest_collection_finish(session):
 
 @pytest.hookimpl()
 def pytest_collection_modifyitems(session, config, items):
+    sa_re = re.compile(r"_GHSA_...._...._....")
+
     for item in items:
         callspec = getattr(item, "callspec", None)
 
         # Add vm mark to VM-using tests
         if callspec is not None and callspec.params.get("vm_setup", None) is not None:
             item.add_marker(pytest.mark.vm)
+
+        # Add security advisory marks based on test name
+        if sa_re.search(item.name):
+            item.add_marker(pytest.mark.sa)

123 lines not on this page · raw message

index 6b97c6525..8e4c109bb 100644
@@ -5,6 +5,7 @@ log_level = 0
 log_file = test-functional.log
 markers =
     vm: tests requiring VM image
+    sa: security advisory regression tests
     tester: kernel testers
 
 addopts =

index 9e3f6c50d..6f50efe0e 100644
@@ -3,6 +3,7 @@
 import os
 import re
 from pathlib import Path
+import pytest
 
 
 def pytest_addoption(parser):

@@ -62,3 +63,14 @@ def pytest_collection_modifyitems(session, config, items):
         # Add security advisory marks based on test name
         if sa_re.search(item.name):
             item.add_marker(pytest.mark.sa)
+
+
[email protected]
+def paired_hosts_bredr(hosts):
+    from .test_agent import test_agent_pair_bredr
+
+    if hosts[0].agent.has_device(hosts[1].bdaddr):
+        return hosts
+
+    test_agent_pair_bredr(hosts, True)
+    return hosts

281 lines not on this page · raw message

index 345b61326..79a217eae 100644
@@ -186,17 +186,6 @@ class ObexClient(HostPlugin, EventPluginMixin):
         return properties["Filename"]
 
 
[email protected]
-def paired_hosts(hosts):
-    from .test_agent import test_agent_pair_bredr
-
-    if hosts[0].agent.has_device(hosts[1].bdaddr):
-        return hosts
-
-    test_agent_pair_bredr(hosts, True)
-    return hosts
-
-
 obex_host_config = host_config(
     [Agent(), Obexd(), ObexClient(), Pexpect()],
     [Agent(), Obexd(), ObexAgent()],

@@ -205,11 +194,11 @@ obex_host_config = host_config(
 
 
 @pytest.fixture
-def obex_hosts(paired_hosts):
-    host0, host1 = paired_hosts
+def obex_hosts(paired_hosts_bredr):
+    host0, host1 = paired_hosts_bredr
 
     if hasattr(host0, "session"):
-        return paired_hosts
+        return paired_hosts_bredr
 
     host0.obex.connect(host1.bdaddr)
 

@@ -219,7 +208,7 @@ def obex_hosts(paired_hosts):
 
     host0.obex.expect("org.bluez.obex.Client1.CreateSession:reply")
 
-    yield paired_hosts
+    yield paired_hosts_bredr
 
     host1.obex_agent.cleanup()