python/machine: Fix AF_UNIX path too long
Quoted 3 lines · Peter Delevoryas
$ make check-avocadoAvocado crashed: TypeError: cannot pickle '_thread.RLock' object
Yeah... you have to pick Cleber's patches from
https://gitlab.com/cleber.gnu/qemu/-/commits/WIP/tests_fixes
and run as:
$ TMPDIR=/tmp avocado run --test-runner=nrunner ...
This is tracked on the avocado project:
https://github.com/avocado-framework/avocado/issues/5138#issuecomment-1314551148
v5 cover python/machine: Fix AF_UNIX path too long
v1: https://lore.kernel.org/qemu-devel/[email protected]/
v2: https://lore.kernel.org/qemu-devel/[email protected]/
v3:
- Changed QEMUMachine._name to f"{id(self):x}". Suggestion was to do f"{id(self):02x}", but the id's look like they are probably just the object address (8-byte pointer), so the "02" had no effect.
- Changed QMP socket name suffix from "-monitor.sock" to ".qmp".
- Changed console socket name suffix from "-console.sock" to ".con".
v4:
- Just resending v3 after rebasing from a long time ago
v5:
- I forgot to disable the VPN on my laptop before sending the v4 series, and I always have very mysterious errors and/or delays when I send email through the VPN because our security team doesn't allow split tunneling.
- I also forgot to include the reviewed-by tags. Daniel reviewed v3, Phil reviewed v4.
I tried to run make check-avocado before sending again, but it looks like there is some other issue. Probably related to the python version I have (I have like 5 different Python versions installed on my work laptop).
Thanks,
Peter
$ make check-avocado changing dir to build for /Library/Developer/CommandLineTools/usr/bin/make "check-avocado"... GIT ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc VENV /Users/pdel/qemu/build/tests/venv VENVPIP install -e /Users/pdel/qemu/python/ VENVPIP install -r /Users/pdel/qemu/tests/requirements.txt MKDIR /Users/pdel/qemu/build/tests/results AVOCADO Downloading avocado tests VM image for aarch64 The image was downloaded: Provider Version Architecture File fedora 31 aarch64 /Users/pdel/avocado/data/cache/by_location/4f156e531446a679cbfe13caef8b7c9f9f79aafa/Fedora-C loud-Base-31-1.9.aarch64.qcow2 AVOCADO tests/avocado Fetching asset from tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf Fetching asset from tests/avocado/boot_xen.py:BootXen.test_arm64_xen_411_and_dom0 Fetching asset from tests/avocado/boot_xen.py:BootXen.test_arm64_xen_414_and_dom0 Fetching asset from tests/avocado/boot_xen.py:BootXen.test_arm64_xen_415_and_dom0 Fetching asset from tests/avocado/machine_aarch64_virt.py:Aarch64VirtMachine.test_alpine_virt_tcg_gic_max Fetching asset from tests/avocado/machine_aarch64_virt.py:Aarch64VirtMachine.test_aarch64_virt Fetching asset from tests/avocado/replay_kernel.py:ReplayKernelNormal.test_aarch64_virt Fetching asset from tests/avocado/reverse_debugging.py:ReverseDebugging_AArch64.test_aarch64_virt JOB ID : 18a949ed9150e22d6ecea69b99ede1ded17233f4 JOB LOG : /Users/pdel/qemu/build/tests/results/job-2023-01-10T00.03-18a949e/job.log
Avocado crashed: TypeError: cannot pickle '_thread.RLock' object
Peter Delevoryas (1):
python/machine: Fix AF_UNIX path too long on macOS
python/qemu/machine/machine.py | 6 +++--- tests/avocado/avocado_qemu/__init__.py | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-)
Signature · 1 line
2.39.0
PATCH v5 1/1 python/machine: Fix AF_UNIX path too long on macOS
On macOS, private $TMPDIR's are the default. These $TMPDIR's are generated from a user's unix UID and UUID [1], which can create a relatively long path:
/var/folders/d7/rz20f6hd709c1ty8f6_6y_z40000gn/T/
QEMU's avocado tests create a temporary directory prefixed by "avo_qemu_sock_", and create QMP sockets within _that_ as well. The QMP socket is unnecessarily long, because a temporary directory is created for every QEMUMachine object.
/avo_qemu_sock_uh3w_dgc/qemu-37331-10bacf110-monitor.sock
The path limit for unix sockets on macOS is 104: [2]
/*
* [XSI] Definitions for UNIX IPC domain.
*/
struct sockaddr_un {
unsigned char sun_len; /* sockaddr len including null */
sa_family_t sun_family; /* [XSI] AF_UNIX */
char sun_path[104]; /* [XSI] path name (gag) */
};This results in avocado tests failing on macOS because the QMP unix socket can't be created, because the path is too long:
ERROR| Failed to establish connection: OSError: AF_UNIX path too long
This change resolves by reducing the size of the socket directory prefix and the suffix on the QMP and console socket names.
The result is paths like this:
pdel@pdel-mbp:/var/folders/d7/rz20f6hd709c1ty8f6_6y_z40000gn/T
$ tree qemu*
qemu_df4evjeq
qemu_jbxel3gy
qemu_ml9s_gg7
qemu_oc7h7f3u
qemu_oqb1yf97
├── 10a004050.con
└── 10a004050.qmp[1] https://apple.stackexchange.com/questions/353832/why-is-mac-osx-temp-directory-in-weird-path
[2] /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/sys/un.h
--- python/qemu/machine/machine.py | 6 +++--- tests/avocado/avocado_qemu/__init__.py | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-)
python/qemu/machine/machine.py +3 -3
@@ -157,7 +157,7 @@ def __init__(self, self._wrapper = wrapper self._qmp_timer = qmp_timer - self._name = name or f"qemu-{os.getpid()}-{id(self):02x}" + self._name = name or f"{id(self):x}" self._temp_dir: Optional[str] = None self._base_temp_dir = base_temp_dir self._sock_dir = sock_dir
@@ -167,7 +167,7 @@ def __init__(self, self._monitor_address = monitor_address else: self._monitor_address = os.path.join( - self.sock_dir, f"{self._name}-monitor.sock" + self.sock_dir, f"{self._name}.qmp" ) self._console_log_path = console_log
@@ -192,7 +192,7 @@ def __init__(self, self._console_set = False self._console_device_type: Optional[str] = None self._console_address = os.path.join( - self.sock_dir, f"{self._name}-console.sock" + self.sock_dir, f"{self._name}.con" ) self._console_socket: Optional[socket.socket] = None self._remove_files: List[str] = []
tests/avocado/avocado_qemu/__init__.py +1 -1
@@ -306,7 +306,7 @@ def require_netdev(self, netdevname): self.cancel('no support for user networking') def _new_vm(self, name, *args): - self._sd = tempfile.TemporaryDirectory(prefix="avo_qemu_sock_") + self._sd = tempfile.TemporaryDirectory(prefix="qemu_") vm = QEMUMachine(self.qemu_bin, base_temp_dir=self.workdir, sock_dir=self._sd.name, log_dir=self.logdir) self.log.debug('QEMUMachine "%s" created', name)
Signature · 1 line
2.39.0
Quoted 11 lines · Philippe Mathieu-Daudé
Quoted 3 lines · Peter Delevoryas
$ make check-avocadoAvocado crashed: TypeError: cannot pickle '_thread.RLock' object
Yeah... you have to pick Cleber's patches from
https://gitlab.com/cleber.gnu/qemu/-/commits/WIP/tests_fixes
and run as:$ TMPDIR=/tmp avocado run --test-runner=nrunner ...
I guess I still get this error from make check-avocado on Aarch64 tests:
(17/35) tests/avocado/replay_linux.py:ReplayLinuxAarch64.test_virt_gicv2: ERROR: Failed to fetch Fedora-Cloud-Base-31-1.9
.aarch64.qcow2 (Can't pickle local object 'url_download.<locals>.download'). (0.89 s)
But ./build/tests/venv/bin/avocado run tests/avocado worked ok.
Quoted 3 lines
This is tracked on the avocado project:
https://github.com/avocado-framework/avocado/issues/5138#issuecomment-1314551148
Thanks!
- Peter