test/py: Use loop mounts if guestmount fails in filesystem tests
PATCH 1/2 test/py: Use loop mounts if guestmount fails in filesystem tests
If guestmount isn't available on the system, filesystem test setup falls back to using loop mounts to prepare its disk images. If guestmount is available but fails to work, the tests are immediately skipped. Instead of giving up on a guestmount failure, try using loop mounts as an attempt to keep tests running.
Also stop checking if guestmount is in PATH, as trying to run a missing guestmount can now follow the same failure codepath and fall back to loop mounts anyway.
---
test/py/tests/test_fs/conftest.py | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-)
test/py/tests/test_fs/conftest.py +15 -16
@@ -205,24 +205,23 @@ def mount_fs(fs_type, device, mount_point): """ global fuse_mounted - fuse_mounted = False try: - if tool_is_in_path('guestmount'): - fuse_mounted = True - check_call('guestmount -a %s -m /dev/sda %s' - % (device, mount_point), shell=True) - else: - mount_opt = 'loop,rw' - if re.match('fat', fs_type): - mount_opt += ',umask=0000' - - check_call('sudo mount -o %s %s %s' - % (mount_opt, device, mount_point), shell=True) - - # may not be effective for some file systems - check_call('sudo chmod a+rw %s' % mount_point, shell=True) + check_call('guestmount -a %s -m /dev/sda %s' + % (device, mount_point), shell=True) + fuse_mounted = True + return except CalledProcessError: - raise + fuse_mounted = False + + mount_opt = 'loop,rw' + if re.match('fat', fs_type): + mount_opt += ',umask=0000' + + check_call('sudo mount -o %s %s %s' + % (mount_opt, device, mount_point), shell=True) + + # may not be effective for some file systems + check_call('sudo chmod a+rw %s' % mount_point, shell=True) def umount_fs(mount_point): """Unmount a volume.
Signature · 1 line
2.32.0.rc2
PATCH 2/2 test/py: Wait for guestmount worker to exit after running guestunmount
Some filesystem tests are failing when their image is prepared with guestmount, but succeeding if loop mounts are used instead. The reason seems to be a race condition the guestmount(1) manual page explains:
When guestunmount(1)/fusermount(1) exits, guestmount may still be running and cleaning up the mountpoint. The disk image will not be fully finalized.
This means that scripts like the following have a nasty race condition:
guestmount -a disk.img -i /mnt
# copy things into /mnt
guestunmount /mnt
# immediately try to use 'disk.img' ** UNSAFE **
The solution is to use the --pid-file option to write the guestmount PID to a file, then after guestunmount spin waiting for this PID to exit.
The Python standard library has an os.waitpid() function for waiting a child to terminate, but it cannot wait on non-child processes. Implement a utility function that can do this by polling the process repeatedly for a given duration, optionally killing the process if it won't terminate on its own. Apply the suggested solution with this utility function, which makes the failing tests succeed again.
---
test/py/tests/test_fs/conftest.py | 13 ++++++++++- test/py/u_boot_utils.py | 36 +++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 1 deletion(-)
test/py/tests/test_fs/conftest.py +12 -1
@@ -8,6 +8,7 @@ import re from subprocess import call, check_call, check_output, CalledProcessError from fstest_defs import * +import u_boot_utils as util supported_fs_basic = ['fat16', 'fat32', 'ext4'] supported_fs_ext = ['fat16', 'fat32']
@@ -206,7 +207,7 @@ def mount_fs(fs_type, device, mount_point): global fuse_mounted try: - check_call('guestmount -a %s -m /dev/sda %s' + check_call('guestmount --pid-file guestmount.pid -a %s -m /dev/sda %s' % (device, mount_point), shell=True) fuse_mounted = True return
@@ -235,6 +236,16 @@ def umount_fs(mount_point): if fuse_mounted: call('sync') call('guestunmount %s' % mount_point, shell=True) + + try: + with open("guestmount.pid", "r") as pidfile: + pid = int(pidfile.read()) + util.waitpid(pid, kill=True) + os.remove("guestmount.pid") + + except FileNotFoundError: + pass + else: call('sudo umount %s' % mount_point, shell=True)
test/py/u_boot_utils.py +36 -0
@@ -8,6 +8,7 @@ import os import os.path import pytest +import signal import sys import time import re
@@ -339,3 +340,38 @@ def crc32(u_boot_console, address, count): assert m, 'CRC32 operation failed.' return m.group(1) + +def waitpid(pid, timeout=60, kill=False): + """Wait a process to terminate by its PID + + This is an alternative to a os.waitpid(pid, 0) call that works on + processes that aren't children of the python process. + + Args: + pid: PID of a running process. + timeout: Time in seconds to wait. + kill: Whether to forcibly kill the process after timeout. + + Returns: + True, if the process ended on its own. + False, if the process was killed by this function. + + Raises: + TimeoutError, if the process is still running after timeout. + """ + try: + for _ in range(timeout): + os.kill(pid, 0) + time.sleep(1) + + if kill: + os.kill(pid, signal.SIGKILL) + return False + + except ProcessLookupError: + return True + + raise TimeoutError( + "Process with PID {} did not terminate after {} seconds." + .format(pid, timeout) + )
Signature · 1 line
2.32.0.rc2
Re: [PATCH 2/2] test/py: Wait for guestmount worker to exit after running guestunmount
Quoted 27 lines · Alper Nebi Yasak
Some filesystem tests are failing when their image is prepared with guestmount, but succeeding if loop mounts are used instead. The reason seems to be a race condition the guestmount(1) manual page explains:
When guestunmount(1)/fusermount(1) exits, guestmount may still be running and cleaning up the mountpoint. The disk image will not be fully finalized.
This means that scripts like the following have a nasty race condition:
guestmount -a disk.img -i /mnt
# copy things into /mnt
guestunmount /mnt
# immediately try to use 'disk.img' ** UNSAFE **The solution is to use the --pid-file option to write the guestmount PID to a file, then after guestunmount spin waiting for this PID to exit.
The Python standard library has an
os.waitpid()function for waiting a child to terminate, but it cannot wait on non-child processes. Implement a utility function that can do this by polling the process repeatedly for a given duration, optionally killing the process if it won't terminate on its own. Apply the suggested solution with this utility function, which makes the failing tests succeed again.
Quoted 7 lines
Signed-off-by Alper Nebi Yasak <[email protected]>---
test/py/tests/test_fs/conftest.py | 13 ++++++++++- test/py/u_boot_utils.py | 36 +++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 1 deletion(-)
Quoted 22 lines
test/py/tests/test_fs/conftest.py +2
-1@@ -8,6 +8,7 @@ import re from subprocess import call, check_call, check_output, CalledProcessError from fstest_defs import * +import u_boot_utils as util supported_fs_basic = ['fat16', 'fat32', 'ext4'] supported_fs_ext = ['fat16', 'fat32']@@ -206,7 +207,7 @@ def mount_fs(fs_type, device, mount_point): global fuse_mounted try: - check_call('guestmount -a %s -m /dev/sda %s' + check_call('guestmount --pid-file guestmount.pid -a %s -m /dev/sda %s' % (device, mount_point), shell=True) fuse_mounted = True return
test/py/tests/test_fs/conftest.py def umount_fs(mount_point): · line 237
Quoted 2 lines
test/py/tests/test_fs/conftest.py +0
-0@@ -235,6 +236,16 @@ def umount_fs(mount_point): if fuse_mounted: call('sync')
should you remove guestmount.pid first in case it is there from a previous crash?
Quoted 68 lines
test/py/tests/test_fs/conftest.py +10
-0call('guestunmount %s' % mount_point, shell=True) + + try: + with open("guestmount.pid", "r") as pidfile: + pid = int(pidfile.read()) + util.waitpid(pid, kill=True) + os.remove("guestmount.pid") + + except FileNotFoundError: + pass + else: call('sudo umount %s' % mount_point, shell=True)test/py/u_boot_utils.py +36
-1@@ -8,6 +8,7 @@ import os import os.path import pytest +import signal import sys import time import re@@ -339,3 +340,38 @@ def crc32(u_boot_console, address, count): assert m, 'CRC32 operation failed.' return m.group(1) + +def waitpid(pid, timeout=60, kill=False): + """Wait a process to terminate by its PID + + This is an alternative to a os.waitpid(pid, 0) call that works on + processes that aren't children of the python process. + + Args: + pid: PID of a running process. + timeout: Time in seconds to wait. + kill: Whether to forcibly kill the process after timeout. + + Returns: + True, if the process ended on its own. + False, if the process was killed by this function. + + Raises: + TimeoutError, if the process is still running after timeout. + """ + try: + for _ in range(timeout): + os.kill(pid, 0) + time.sleep(1) + + if kill: + os.kill(pid, signal.SIGKILL) + return False + + except ProcessLookupError: + return True + + raise TimeoutError( + "Process with PID {} did not terminate after {} seconds." + .format(pid, timeout) + ) --2.32.0.rc2
Re: [PATCH 2/2] test/py: Wait for guestmount worker to exit after running guestunmount
Quoted 53 lines · Simon Glass
Quoted 27 lines · Alper Nebi Yasak
Some filesystem tests are failing when their image is prepared with guestmount, but succeeding if loop mounts are used instead. The reason seems to be a race condition the guestmount(1) manual page explains:
When guestunmount(1)/fusermount(1) exits, guestmount may still be running and cleaning up the mountpoint. The disk image will not be fully finalized.
This means that scripts like the following have a nasty race condition:
guestmount -a disk.img -i /mnt
# copy things into /mnt
guestunmount /mnt
# immediately try to use 'disk.img' ** UNSAFE **The solution is to use the --pid-file option to write the guestmount PID to a file, then after guestunmount spin waiting for this PID to exit.
The Python standard library has an
os.waitpid()function for waiting a child to terminate, but it cannot wait on non-child processes. Implement a utility function that can do this by polling the process repeatedly for a given duration, optionally killing the process if it won't terminate on its own. Apply the suggested solution with this utility function, which makes the failing tests succeed again.Quoted 7 lines
Signed-off-by Alper Nebi Yasak <[email protected]>---
test/py/tests/test_fs/conftest.py | 13 ++++++++++- test/py/u_boot_utils.py | 36 +++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 1 deletion(-)Reviewed-by Simon Glass <[email protected]>Quoted 15 lines
test/py/tests/test_fs/conftest.py +1
-0@@ -8,6 +8,7 @@ import re from subprocess import call, check_call, check_output, CalledProcessError from fstest_defs import * +import u_boot_utils as util supported_fs_basic = ['fat16', 'fat32', 'ext4'] supported_fs_ext = ['fat16', 'fat32']@@ -206,7 +207,7 @@ def mount_fs(fs_type, device, mount_point): global fuse_mounted
Quoted 12 lines
test/py/tests/test_fs/conftest.py def umount_fs(mount_point): · line 237
Quoted 9 lines
test/py/tests/test_fs/conftest.py +1
-1try: - check_call('guestmount -a %s -m /dev/sda %s' + check_call('guestmount --pid-file guestmount.pid -a %s -m /dev/sda %s' % (device, mount_point), shell=True) fuse_mounted = True return@@ -235,6 +236,16 @@ def umount_fs(mount_point): if fuse_mounted: call('sync')should you remove guestmount.pid first in case it is there from a previous crash?
Guestmount overwrites the pid file without complaining, and the worker deletes the file on its own when it terminates after the guestunmount call.
I think it's OK to ignore an existing pid file if the mount point is clean. If not clean, that's the actual problem and both guestmount and mount will fail, causing the tests to be skipped. We could clean mount points automatically before attempting to mount.
Quoted 68 lines
Quoted 68 lines
test/py/tests/test_fs/conftest.py +10
-0call('guestunmount %s' % mount_point, shell=True) + + try: + with open("guestmount.pid", "r") as pidfile: + pid = int(pidfile.read()) + util.waitpid(pid, kill=True) + os.remove("guestmount.pid") + + except FileNotFoundError: + pass + else: call('sudo umount %s' % mount_point, shell=True)test/py/u_boot_utils.py +36
-1@@ -8,6 +8,7 @@ import os import os.path import pytest +import signal import sys import time import re@@ -339,3 +340,38 @@ def crc32(u_boot_console, address, count): assert m, 'CRC32 operation failed.' return m.group(1) + +def waitpid(pid, timeout=60, kill=False): + """Wait a process to terminate by its PID + + This is an alternative to a os.waitpid(pid, 0) call that works on + processes that aren't children of the python process. + + Args: + pid: PID of a running process. + timeout: Time in seconds to wait. + kill: Whether to forcibly kill the process after timeout. + + Returns: + True, if the process ended on its own. + False, if the process was killed by this function. + + Raises: + TimeoutError, if the process is still running after timeout. + """ + try: + for _ in range(timeout): + os.kill(pid, 0) + time.sleep(1) + + if kill: + os.kill(pid, signal.SIGKILL) + return False + + except ProcessLookupError: + return True + + raise TimeoutError( + "Process with PID {} did not terminate after {} seconds." + .format(pid, timeout) + ) --2.32.0.rc2
Re: [PATCH 2/2] test/py: Wait for guestmount worker to exit after running guestunmount
In reply to Alper Nebi YasakQuoted 26 lines · Alper Nebi Yasak
Some filesystem tests are failing when their image is prepared with guestmount, but succeeding if loop mounts are used instead. The reason seems to be a race condition the guestmount(1) manual page explains:
When guestunmount(1)/fusermount(1) exits, guestmount may still be running and cleaning up the mountpoint. The disk image will not be fully finalized.
This means that scripts like the following have a nasty race condition:
guestmount -a disk.img -i /mnt
# copy things into /mnt
guestunmount /mnt
# immediately try to use 'disk.img' ** UNSAFE **The solution is to use the --pid-file option to write the guestmount PID to a file, then after guestunmount spin waiting for this PID to exit.
The Python standard library has an
os.waitpid()function for waiting a child to terminate, but it cannot wait on non-child processes. Implement a utility function that can do this by polling the process repeatedly for a given duration, optionally killing the process if it won't terminate on its own. Apply the suggested solution with this utility function, which makes the failing tests succeed again.
Quoted 2 lines
Signed-off-by Alper Nebi Yasak <[email protected]>Reviewed-by Simon Glass <[email protected]>
Applied to u-boot/master, thanks!
Signature · 1 line
Tom
Quoted 11 lines · Alper Nebi Yasak
If guestmount isn't available on the system, filesystem test setup falls back to using loop mounts to prepare its disk images. If guestmount is available but fails to work, the tests are immediately skipped. Instead of giving up on a guestmount failure, try using loop mounts as an attempt to keep tests running.
Also stop checking if guestmount is in PATH, as trying to run a missing guestmount can now follow the same failure codepath and fall back to loop mounts anyway.
Quoted 5 lines
Signed-off-by Alper Nebi Yasak <[email protected]>---
test/py/tests/test_fs/conftest.py | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-)
https://source.denx.de/u-boot/custodians/u-boot-dm/-/jobs/282788
thoughts below
Quoted 25 lines
test/py/tests/test_fs/conftest.py +0
-15@@ -205,24 +205,23 @@ def mount_fs(fs_type, device, mount_point): """ global fuse_mounted - fuse_mounted = False try: - if tool_is_in_path('guestmount'): - fuse_mounted = True - check_call('guestmount -a %s -m /dev/sda %s' - % (device, mount_point), shell=True) - else: - mount_opt = 'loop,rw' - if re.match('fat', fs_type): - mount_opt += ',umask=0000' - - check_call('sudo mount -o %s %s %s' - % (mount_opt, device, mount_point), shell=True) - - # may not be effective for some file systems - check_call('sudo chmod a+rw %s' % mount_point, shell=True)
test/py/tests/test_fs/conftest.py def mount_fs(fs_type, device, mount_point): · line 217
Quoted 10 lines
test/py/tests/test_fs/conftest.py +8
-1+ check_call('guestmount -a %s -m /dev/sda %s' + % (device, mount_point), shell=True) + fuse_mounted = True + return except CalledProcessError: - raise + fuse_mounted = False + + mount_opt = 'loop,rw' + if re.match('fat', fs_type):
This seems ambiguous. I think you should use:
if 'fat' in fs_type:
or
if fs_type.startswith('fat'):
or
if fs_type == 'fat':
so it is clear what you mean.
Quoted 11 lines
test/py/tests/test_fs/conftest.py +7
-1+ mount_opt += ',umask=0000' + + check_call('sudo mount -o %s %s %s' + % (mount_opt, device, mount_point), shell=True) + + # may not be effective for some file systems + check_call('sudo chmod a+rw %s' % mount_point, shell=True) def umount_fs(mount_point): """Unmount a volume. --
Quoted 2 lines
2.32.0.rc2
Regards,
Simon
Quoted 15 lines · Simon Glass
Quoted 14 lines · Alper Nebi Yasak
If guestmount isn't available on the system, filesystem test setup falls back to using loop mounts to prepare its disk images. If guestmount is available but fails to work, the tests are immediately skipped. Instead of giving up on a guestmount failure, try using loop mounts as an attempt to keep tests running.
Also stop checking if guestmount is in PATH, as trying to run a missing guestmount can now follow the same failure codepath and fall back to loop mounts anyway.
Signed-off-by Alper Nebi Yasak <[email protected]>---
Quoted 6 lines
Quoted 2 lines
test/py/tests/test_fs/conftest.py | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-)https://source.denx.de/u-boot/custodians/u-boot-dm/-/jobs/282788
thoughts below
That CI run doesn't have this patch/series applied.
Quoted 42 lines
Quoted 25 lines
test/py/tests/test_fs/conftest.py +0
-15@@ -205,24 +205,23 @@ def mount_fs(fs_type, device, mount_point): """ global fuse_mounted - fuse_mounted = False try: - if tool_is_in_path('guestmount'): - fuse_mounted = True - check_call('guestmount -a %s -m /dev/sda %s' - % (device, mount_point), shell=True) - else: - mount_opt = 'loop,rw' - if re.match('fat', fs_type): - mount_opt += ',umask=0000' - - check_call('sudo mount -o %s %s %s' - % (mount_opt, device, mount_point), shell=True) - - # may not be effective for some file systems - check_call('sudo chmod a+rw %s' % mount_point, shell=True)test/py/tests/test_fs/conftest.py def mount_fs(fs_type, device, mount_point): · line 217
Quoted 10 lines
test/py/tests/test_fs/conftest.py +8
-1+ check_call('guestmount -a %s -m /dev/sda %s' + % (device, mount_point), shell=True) + fuse_mounted = True + return except CalledProcessError: - raise + fuse_mounted = False + + mount_opt = 'loop,rw' + if re.match('fat', fs_type):This seems ambiguous. I think you should use:
if 'fat' in
fs_type:or
Quoted 7 lines
if
fs_type.startswith('fat'):or
if
fs_type== 'fat':so it is clear what you mean.
This just unindents existing code. I can send v2 with startswith('fat') though.
(I think I should rewrite the tests with guestfish and remove these mount functions instead; but I'm busy with other things for a while.)
Quoted 17 lines
Quoted 11 lines
test/py/tests/test_fs/conftest.py +7
-1+ mount_opt += ',umask=0000' + + check_call('sudo mount -o %s %s %s' + % (mount_opt, device, mount_point), shell=True) + + # may not be effective for some file systems + check_call('sudo chmod a+rw %s' % mount_point, shell=True) def umount_fs(mount_point): """Unmount a volume. --Quoted 2 lines
2.32.0.rc2
Regards,
Simon
Quoted 11 lines · Alper Nebi Yasak
If guestmount isn't available on the system, filesystem test setup falls back to using loop mounts to prepare its disk images. If guestmount is available but fails to work, the tests are immediately skipped. Instead of giving up on a guestmount failure, try using loop mounts as an attempt to keep tests running.
Also stop checking if guestmount is in PATH, as trying to run a missing guestmount can now follow the same failure codepath and fall back to loop mounts anyway.
Signed-off-by Alper Nebi Yasak <[email protected]>
Applied to u-boot/master, thanks!
Signature · 1 line
Tom