lib: tst_device.c - /dev/loop0 in use
Hello Cyril,
Apparently, when having "all_filesystems" flag set for one test, there is a room of opportunity for the subsequent filesystems tests to find the device, acquired by "tst_acquire_device()", busy.
I suppose, in the case bellow, fsync() caused async work in some kernel thread to cause the "/dev/loop0 is apparently in use by the system" error when trying a new mkfs.XXXX call - for the subsequent filesystem - using the same acquired device.
With the "all_filesystems" logic, the acquired device is not detached, nor find_free_loopdev() is called again, which *could* mitigate this.
The error:
tst_test.c:1015: INFO: Timeout per run is 0h 15m 00s fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
tst_test.c:1076: INFO: Testing on ext3
tst_mkfs.c:83: INFO: Formatting /dev/loop0 with ext3 opts='' extra opts='' mke2fs 1.43.5 (04-Aug-2017)
[ 48.457646] EXT4-fs (loop0): mounting ext3 file system using the ext4 subsystem [ 48.502872] EXT4-fs (loop0): mounted filesystem with ordered data
mode. Opts: (null)
tst_test.c:1015: INFO: Timeout per run is 0h 15m 00s fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
tst_test.c:1076: INFO: Testing on ext4
tst_mkfs.c:83: INFO: Formatting /dev/loop0 with ext4 opts='' extra opts='' mke2fs 1.43.5 (04-Aug-2017)/dev/loop0 is apparently in use by the system; will not make a filesystem here!
tst_mkfs.c:94: BROK: mkfs.ext4:1: tst_test.c failed with 729
Thoughts ?
Cheers o/
Signature · 2 lines
Rafael D. Tinoco Linaro - Kernel Validation
Hi!
Quoted 11 lines
Apparently, when having "
all_filesystems" flag set for one test, there is a room of opportunity for the subsequent filesystems tests to find the device, acquired by "tst_acquire_device()", busy.I suppose, in the case bellow,
fsync()caused async work in some kernel thread to cause the "/dev/loop0is apparently in use by the system" error when trying a new mkfs.XXXX call - for the subsequent filesystem - using the same acquired device.With the "
all_filesystems" logic, the acquired device is not detached, norfind_free_loopdev()is called again, which *could* mitigate this.
Right, we expect that the device is free to be reused once the umount() returns to the userspace, which apparently is not true in your case.
Looking into the mkfs code it tries to open() the device RDONLY and we get this error if the attempt has failed with EBUSY, which likely means that the umount() is still running in background in kernel.
I guess that we may as well patch the tst_umount() function in lib/tst_device.c to loop until open() fails with EBUSY in the similar manner we attempt to retry umount() there.
Signature · 2 lines
Cyril Hrubis [email protected]
PATCH 1/1 tst_umount: Retry open() after umount to handle delayed device release
Currently, tst_umount() in lib/tst_device.c tries to repeatedly umount() a mount point, with a retry loop if it gets EBUSY. However, after umount() reports success, devices (especially loop devices) can still be held open by another process or kernel, delaying their actual release. This can lead to race conditions when the next operation tries to reuse the device like mkfs.ext3 error:
==== setxattr01 ====
command: setxattr01
tst_test.c:1953: TINFO: LTP version: 20250530
...
tst_supported_fs_types.c:48: TINFO: mkfs is not needed for tmpfs
tst_test.c:1888: TINFO: === Testing on ext2 ===
tst_test.c:1217: TINFO: Formatting /dev/loop0 with ext2 opts='' extra opts=''
mke2fs 1.47.1 (20-May-2024)
tst_test.c:1229: TINFO: Mounting /dev/loop0 to /tmp/LTP_setd24dAf/mntpoint fstyp=ext2 flags=0
...
setxattr01.c:174: TPASS: setxattr(2) failed: ERANGE (34)
setxattr01.c:174: TPASS: setxattr(2) failed: EFAULT (14)
tst_test.c:1888: TINFO: === Testing on ext3 ===
tst_test.c:1217: TINFO: Formatting /dev/loop0 with ext3 opts='' extra opts='' mke2fs 1.47.1 (20-May-2024)
/dev/loop0 is apparently in use by the system; will not make a filesystem here! tst_test.c:1217: TBROK: mkfs.ext3 failed with exit code 1
==== close_range01 ====
command: close_range01
tst_test.c:1953: TINFO: LTP version: 20250530
...
tst_test.c:1888: TINFO: === Testing on ext2 ===
tst_test.c:1217: TINFO: Formatting /dev/loop0 with ext2 opts='' extra opts=''
mke2fs 1.47.1 (20-May-2024)
tst_test.c:1229: TINFO: Mounting /dev/loop0 to /tmp/LTP_cloXeXI39/mnt fstyp=ext2 flags=0
...
close_range01.c:188: TPASS: No kernel taints
tst_test.c:1888: TINFO: === Testing on ext3 ===
tst_test.c:1217: TINFO: Formatting /dev/loop0 with ext3 opts='' extra opts='' mke2fs 1.47.1 (20-May-2024)
/dev/loop0 is apparently in use by the system; will not make a filesystem here! tst_test.c:1217: TBROK: mkfs.ext3 failed with exit code 1
This patch improves robustness by, after a successful umount(), attempting to open() the device in O_RDWR mode and retrying if open() fails with EBUSY, similar to how umount() is retried. This ensures that tst_umount() only returns success after the device is truly available for reuse, reducing race conditions in rapid test cycles.
--- lib/tst_device.c | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-)
lib/tst_device.c +25 -2
@@ -397,14 +397,37 @@ int tst_clear_device(const char *dev)int tst_umount (const char *path) { -int err, ret, i; +int err, ret, i, j;for (i =0 ; i <50 ; i++) { ret =umount (path); err = errno; -if (!ret) +if (!ret) { +/* After successful umount, check device open busy */ +for (j =0 ; j <50 ; j++) { +int fd =open (path,O_RDWR ); +if (fd >=0 ) { +close (fd); +break ; + } +if (errno ==EBUSY ) { +tst_resm (TINFO ,"open('%s') got EBUSY after umount, retry %d..." , + path, j+1 ); +usleep (100000 ); +continue ; + } +break ; + } +if (j ==50 ) { +tst_resm (TWARN ,"Device '%s' is still busy after %d open retries" , + path,10 ); + errno =EBUSY ; +return -1 ; + } +return 0 ; + }if (err !=EBUSY ) { tst_resm(TWARN, "umount('%s') failed with %s",
Signature · 5 lines
2.50.1 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH] tst_umount: Retry open() after umount to handle delayed device release
Hi!
Quoted 6 lines
Currently,
tst_umount()inlib/tst_device.ctries to repeatedlyumount()a mount point, with a retry loop if it gets EBUSY. However, afterumount()reports success, devices (especially loop devices) can still be held open by another process or kernel, delaying their actual release. This can lead to race conditions when the next operation tries to reuse the device like mkfs.ext3 error:
This looks like working around a kernel bug to me. The umount() without MNT_DETACH should make the device reusable once the syscall returns back to the userspace. At least I wouldn't add this code to LTP unless kernel devs tells us that this broken behavior is to be expected.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH] tst_umount: Retry open() after umount to handle delayed device release
Hi Cyril,
Quoted 6 lines · Cyril Hrubis
Hi!
Quoted 2 lines
Currently,
tst_umount()inlib/tst_device.ctries to repeatedlyumount()a mount point, with a retry loop if it gets EBUSY. However, after
umount()Quoted 2 lines
reports success, devices (especially loop devices) can still be held open by another process or kernel, delaying their actual release. This can
Quoted 8 lines
lead
Quoted 2 lines
to race conditions when the next operation tries to reuse the device like mkfs.ext3 error:
This looks like working around a kernel bug to me. The
umount()withoutMNT_DETACHshould make the device reusable once the syscall returns back to the userspace. At least I wouldn't add this code to LTP unless kernel devs tells us that this broken behavior is to be expected.
Perhaps not yet fall into the kernel issue.
Looking into the details for many days, there seems to be something in userspace briefly reopens the block device right after unmounting it, I suspect that “something” is likely systemd-udevd running its blkid builtin.
To prove this, I did some debugging work:
1. Confirm the tested system enables rule:
# cat /usr/lib/udev/rules.d/60-persistent-storage.rules | grep blkid
KERNEL!="sr*|mmcblk[0-9]boot[0-9]", IMPORT{builtin}="blkid"2. Check the core functions of udev-builtin-blkid:
https://github.com/systemd/systemd/blob/main/src/udev/udev-builtin-blkid.c#L353
3. Run udev monitor in a separate terminal, and get output with running ./close_range01 test simultaneously:
# udevadm monitor --kernel --udev monitor will print the received events for: UDEV - the event which udev sends out after rule processing KERNEL - the kernel uevent
KERNEL[283886.216245] change /devices/virtual/block/loop0 (block) KERNEL[283886.271087] change /devices/virtual/block/loop0 (block) UDEV [283886.277479] change /devices/virtual/block/loop0 (block) UDEV [283886.280656] change /devices/virtual/block/loop0 (block) KERNEL[283886.299308] change /devices/virtual/block/loop0 (block) UDEV [283886.303885] change /devices/virtual/block/loop0 (block) KERNEL[283886.307296] change /devices/virtual/block/loop0 (block) UDEV [283886.313293] change /devices/virtual/block/loop0 (block) KERNEL[283886.317716] change /devices/virtual/block/loop0 (block) UDEV [283886.321284] change /devices/virtual/block/loop0 (block) KERNEL[283886.358952] change /devices/virtual/block/loop0 (block) UDEV [283886.364841] change /devices/virtual/block/loop0 (block) KERNEL[283886.369347] change /devices/virtual/block/loop0 (block) UDEV [283886.376379] change /devices/virtual/block/loop0 (block) KERNEL[283886.378823] change /devices/virtual/block/loop0 (block) UDEV [283886.382704] change /devices/virtual/block/loop0 (block) KERNEL[283886.463900] change /devices/virtual/block/loop0 (block) UDEV [283886.467966] change /devices/virtual/block/loop0 (block) KERNEL[283886.471508] change /devices/virtual/block/loop0 (block) UDEV [283886.475901] change /devices/virtual/block/loop0 (block) KERNEL[283886.491129] change /devices/virtual/block/loop0 (block) UDEV [283886.495081] change /devices/virtual/block/loop0 (block) KERNEL[283886.498525] change /devices/virtual/block/loop0 (block) UDEV [283886.502380] change /devices/virtual/block/loop0 (block) KERNEL[283886.504077] change /devices/virtual/block/loop0 (block) UDEV [283886.507968] change /devices/virtual/block/loop0 (block) KERNEL[283886.558605] change /devices/virtual/block/loop0 (block) KERNEL[283886.559145] change /devices/virtual/block/loop0 (block) UDEV [283886.560932] change /devices/virtual/block/loop0 (block) UDEV [283886.562050] change /devices/virtual/block/loop0 (block)
Looks like udev-builtin-blkid indeed probes via libblkid to populate ID_FS_* properties.
Then, I use strace tool to trace the openat on systemd-udevd, but weirdly I got all opens with 'O_PATH', which opens do not conflict with mkfs.ext*’s O_EXCL and should not cause “apparently in use by the system”.
4. #sudo strace -f -e trace=openat,close -p $(pidof systemd-udevd) -o log
# cat log | grep '/dev/loop0' 275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 17 702 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH <unfinished ...> 275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 17 275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_CLOEXEC|O_PATH) = 17 275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 17 275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 17 275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_CLOEXEC|O_PATH) = 17 702 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 13 275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 17 ...
However, switch to another bpftrace tool gives more suggestive clues, it catches not only the udev-worker but also udeskd operations on the /dev/loop0, after checking the flags I doubt that race probably comes from udiskd operation.
5. Run bpftrace in a separate terminal and get output with running ./close_range01 test simultaneously:
# sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat
/str(args->filename)=="/dev/loop0"/ { printf("%s pid=%d flags=0x%x %s\n",
comm, pid, args->flags, str(args->filename)); }'
Attaching 1 probe...
close_range01 pid=298948 flags=0x2 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
close_range01 pid=298948 flags=0x0 /dev/loop0
systemd-udevd pid=702 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x280000 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x280000 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0
close_range01 pid=298948 flags=0x241 /dev/loop0
systemd-udevd pid=702 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
mkfs.ext2 pid=298969 flags=0x80 /dev/loop0
mkfs.ext2 pid=298969 flags=0x0 /dev/loop0
mkfs.ext2 pid=298969 flags=0x0 /dev/loop0
mkfs.ext2 pid=298969 flags=0x0 /dev/loop0
mkfs.ext2 pid=298969 flags=0x80800 /dev/loop0
mkfs.ext2 pid=298969 flags=0x0 /dev/loop0
mkfs.ext2 pid=298969 flags=0x82 /dev/loop0
(udev-worker) pid=298949 flags=0x280000 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0
systemd-udevd pid=702 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0
(udev-worker) pid=298949 flags=0x280000 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0 <--------------
close_range01 pid=298948 flags=0x241 /dev/loop0
systemd-udevd pid=702 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
mkfs.ext3 pid=298977 flags=0x80 /dev/loop0
mkfs.ext3 pid=298977 flags=0x0 /dev/loop0
mkfs.ext3 pid=298977 flags=0x0 /dev/loop0
mkfs.ext3 pid=298977 flags=0x0 /dev/loop0
mkfs.ext3 pid=298977 flags=0x80800 /dev/loop0
mkfs.ext3 pid=298977 flags=0x0 /dev/loop0
mkfs.ext3 pid=298977 flags=0x82 /dev/loop0
...From the bpfstrace logs, there seems udisksd listening to the same udevevents and probes the device via libblockdev/blkid, performing real O_RDONLY opens (flags=0x0). These O_RDONLY opens from udisksd race with mkfs.ext3’s O_EXCL open, intermittently causing “apparently in use by the system” (EBUSY).
Does this analysis sound reasonable to you?
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH] tst_umount: Retry open() after umount to handle delayed device release
Hi!
Quoted 7 lines
From the bpfstrace logs, there seems udisksd listening to the same udevevents and probes the device via libblockdev/blkid, performing real
O_RDONLYopens (flags=0x0). TheseO_RDONLYopens from udisksd race with mkfs.ext3’sO_EXCLopen, intermittently causing “apparently in use by the system” (EBUSY).Does this analysis sound reasonable to you?
Yes. So we have another daemon that probes block devices on each change.
I suppose that turning off udisksd fixes the problem right? If that is so I would be inclined just to print a message to turn off udisksd if mkfs on a loop device fails...
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH] tst_umount: Retry open() after umount to handle delayed device release
Quoted 7 lines · Cyril Hrubis
Hi!
Quoted 6 lines
From the bpfstrace logs, there seems udisksd listening to the same udevevents and probes the device via libblockdev/blkid, performing real
O_RDONLYopens (flags=0x0). TheseO_RDONLYopens from udisksd race with mkfs.ext3’sO_EXCLopen, intermittently causing “apparently in use by the system” (EBUSY).
Quoted 8 lines
Quoted 1 line
Does this analysis sound reasonable to you?
Yes. So we have another daemon that probes block devices on each change.
I suppose that turning off udisksd fixes the problem right? If that is so I would be inclined just to print a message to turn off udisksd if mkfs on a loop device fails...
That's right, this intermittent EBUSY is not related to LTP (and not be a big deal), a hint print should be useful enough, we can leave stopping udisks2 operation to the tester env setup.
What about this amendment:
lib/tst_mkfs.c +0 -0
@@ -107,6 +107,13 @@ void tst_mkfs_(const char *file, const int lineno,
void (cleanup_fn)(void),
"%s not found in $PATH", mkfs);
break;
default:+ tst_resm_(file, lineno, TINFO,
+ "Check if mkfs failed with the (loop) device busy. "
+ "Background probing (e.g., udisks2) can cause this.
\n"
+ "Consider temporarily stopping udisks2 during the
test:\n"
+ " systemctl stop udisks2.service\n"
+ "and restart afterward:\n"
+ " systemctl start udisks2.service");
tst_brkm_(file, lineno, TBROK, cleanup_fn,
"%s failed with exit code %i", mkfs, ret);
}Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH] tst_umount: Retry open() after umount to handle delayed device release
Hi!
Quoted 13 lines
lib/tst_mkfs.c +3
-0@@ -107,6 +107,13 @@ void tst_mkfs_(const char *file, const int lineno, void (cleanup_fn)(void), "%s not found in $PATH", mkfs);break ;default : +tst_resm_ (file, lineno,TINFO , +"Check if mkfs failed with the (loop) device busy. " + "Background probing (e.g., udisks2) can cause this.\n"
+ "Consider temporarily stopping udisks2 during the
test:\n"
lib/tst_mkfs.c
Quoted 3 lines
lib/tst_mkfs.c +3
-0+" systemctl stop udisks2.service\n" +"and restart afterward:\n" + " systemctl start udisks2.service");
I wouldn't include the systemctl commands in the message, anybody who is using LTP to test the system should know how to start and stop system daemons. Let's keep the message short and to the point.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp