loop: Make explicit loop device destruction lazy
Subsystems (2)
PATCH 1/1 loop: Make explicit loop device destruction lazy
From: Dave Chinner <[email protected]>
xfstests has always had random failures of tests due to loop devices failing to be torn down and hence leaving filesytems that cannot be unmounted. This causes test runs to immediately stop.
Over the past 6 or 7 years we've added hacks like explicit unmount -d commands for loop mounts, losetup -d after unmount -d fails, etc, but still the problems persist. Recently, the frequency of loop related failures increased again to the point that xfstests 259 will reliably fail with a stray loop device that was not torn down.
That is despite the fact the test is above as simple as it gets - loop 5 or 6 times running mkfs.xfs with different paramters:
lofile=$(losetup -f)
losetup $lofile "$testfile"
"$MKFS_XFS_PROG" -b size=512 $lofile >/dev/null || echo "mkfs failed!" sync
losetup -d $lofile
And losteup -d $lofile is failing with EBUSY on 1-3 of these loops every time the test is run.
Turns out that blkid is running simultaneously with losetup -d, and so it sees an elevated reference count and returns EBUSY. But why is blkid running? It's obvious, isn't it? udev has decided to try and find out what is on the block device as a result of a creation notification. And it is racing with mkfs, so might still be scanning the device when mkfs finishes and we try to tear it down.
So, make losetup -d force autoremove behaviour. That is, when the last reference goes away, tear down the device. xfstests wants it *gone*, not causing random teardown failures when we know that all the operations the tests have specifically run on the device have completed and are no longer referencing the loop device.
--- drivers/block/loop.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/block/loop.c +15 -2
@@ -976,8 +976,21 @@ static int loop_clr_fd(struct loop_device *lo)if (lo->lo_state != Lo_bound)return -ENXIO ; -if (lo->lo_refcnt >1 )/* we needed one fd for the ioctl */ -return -EBUSY ; +/* +* If we've explicitly asked to tear down the loop device, +* and it has an elevated reference count, set it for auto-teardown when +* the last reference goes away. This stops $!~#$@ udev from +* preventing teardown because it decided that it needs to run blkid on +* the loopback device whenever they appear. xfstests is notorious for +* failing tests because blkid via udev races with a losetup +* <dev>/do something like mkfs/losetup -d <dev> causing the losetup -d +* command to fail with EBUSY. +*/ +if (lo->lo_refcnt >1 ) { + lo->lo_flags |=LO_FLAGS_AUTOCLEAR ; +mutex_unlock (&lo->lo_ctl_mutex); +return 0 ; + }if (filp ==NULL )return -EINVAL ;
Signature · 1 line
1.7.10
Jens Axboe
maintainerQuoted 31 lines · Dave Chinner
From: Dave Chinner <[email protected]>
xfstests has always had random failures of tests due to loop devices failing to be torn down and hence leaving filesytems that cannot be unmounted. This causes test runs to immediately stop.
Over the past 6 or 7 years we've added hacks like explicit unmount -d commands for loop mounts, losetup -d after unmount -d fails, etc, but still the problems persist. Recently, the frequency of loop related failures increased again to the point that xfstests 259 will reliably fail with a stray loop device that was not torn down.
That is despite the fact the test is above as simple as it gets - loop 5 or 6 times running mkfs.xfs with different paramters:
lofile=$(losetup -f)
losetup $lofile "$testfile"
"$MKFS_XFS_PROG" -b size=512 $lofile >/dev/null || echo "mkfs failed!" sync
losetup -d $lofileAnd losteup -d $lofile is failing with EBUSY on 1-3 of these loops every time the test is run.
Turns out that blkid is running simultaneously with losetup -d, and so it sees an elevated reference count and returns EBUSY. But why is blkid running? It's obvious, isn't it? udev has decided to try and find out what is on the block device as a result of a creation notification. And it is racing with mkfs, so might still be scanning the device when mkfs finishes and we try to tear it down.
Quoted 5 lines
So, make losetup -d force autoremove behaviour. That is, when the last reference goes away, tear down the device. xfstests wants it *gone*, not causing random teardown failures when we know that all the operations the tests have specifically run on the device have completed and are no longer referencing the loop device.
I hear that %^#@#! blkid behavior, it is such a pain in the neck. I don't know how many times I've had to explain that behaviour to people who run write testing with tracing, wonder wtf there are reads in the trace.
Patch looks fine, seems like the sane thing to do (lazy-remove on last drop) for this case.
Signature · 1 line
Jens Axboe
Quoted 11 lines · Jens Axboe
Quoted 7 lines
Turns out that blkid is running simultaneously with losetup -d, and so it sees an elevated reference count and returns EBUSY. But why is blkid running? It's obvious, isn't it? udev has decided to try and find out what is on the block device as a result of a creation notification. And it is racing with mkfs, so might still be scanning the device when mkfs finishes and we try to tear it down.
I hear that %^#@#! blkid behavior, it is such a pain in the neck. I don't know how many times I've had to explain that behaviour to people who run write testing with tracing, wonder wtf there are reads in the trace.
Could this problem explain this bug too ? https://bugzilla.redhat.com/show_bug.cgi?id=853674
Dave
Jens Axboe
maintainerQuoted 9 lines · Dave Jones
Quoted 7 lines · Jens Axboe
Quoted 7 lines
Turns out that blkid is running simultaneously with losetup -d, and so it sees an elevated reference count and returns EBUSY. But why is blkid running? It's obvious, isn't it? udev has decided to try and find out what is on the block device as a result of a creation notification. And it is racing with mkfs, so might still be scanning the device when mkfs finishes and we try to tear it down.
Quoted 6 lines
Quoted 4 lines
I hear that %^#@#! blkid behavior, it is such a pain in the neck. I don't know how many times I've had to explain that behaviour to people who run write testing with tracing, wonder wtf there are reads in the trace.
Could this problem explain this bug too ? https://bugzilla.redhat.com/show_bug.cgi?id=853674
It certainly looks like it!
Signature · 1 line
Jens Axboe
Quoted 38 lines · Jens Axboe
Quoted 31 lines · Dave Chinner
From: Dave Chinner <[email protected]>
xfstests has always had random failures of tests due to loop devices failing to be torn down and hence leaving filesytems that cannot be unmounted. This causes test runs to immediately stop.
Over the past 6 or 7 years we've added hacks like explicit unmount -d commands for loop mounts, losetup -d after unmount -d fails, etc, but still the problems persist. Recently, the frequency of loop related failures increased again to the point that xfstests 259 will reliably fail with a stray loop device that was not torn down.
That is despite the fact the test is above as simple as it gets - loop 5 or 6 times running mkfs.xfs with different paramters:
lofile=$(losetup -f)
losetup $lofile "$testfile"
"$MKFS_XFS_PROG" -b size=512 $lofile >/dev/null || echo "mkfs failed!" sync
losetup -d $lofileAnd losteup -d $lofile is failing with EBUSY on 1-3 of these loops every time the test is run.
Turns out that blkid is running simultaneously with losetup -d, and so it sees an elevated reference count and returns EBUSY. But why is blkid running? It's obvious, isn't it? udev has decided to try and find out what is on the block device as a result of a creation notification. And it is racing with mkfs, so might still be scanning the device when mkfs finishes and we try to tear it down.
Quoted 5 lines
So, make losetup -d force autoremove behaviour. That is, when the last reference goes away, tear down the device. xfstests wants it *gone*, not causing random teardown failures when we know that all the operations the tests have specifically run on the device have completed and are no longer referencing the loop device.
Quoted 7 lines
I hear that %^#@#! blkid behavior, it is such a pain in the neck. I don't know how many times I've had to explain that behaviour to people who run write testing with tracing, wonder wtf there are reads in the trace.
Patch looks fine, seems like the sane thing to do (lazy-remove on last drop) for this case.
Do we also want to prevent further opens?
Jens Axboe
maintainerQuoted 45 lines · Jeff Moyer
Quoted 43 lines · Jens Axboe
Quoted 31 lines · Dave Chinner
From: Dave Chinner <[email protected]>
xfstests has always had random failures of tests due to loop devices failing to be torn down and hence leaving filesytems that cannot be unmounted. This causes test runs to immediately stop.
Over the past 6 or 7 years we've added hacks like explicit unmount -d commands for loop mounts, losetup -d after unmount -d fails, etc, but still the problems persist. Recently, the frequency of loop related failures increased again to the point that xfstests 259 will reliably fail with a stray loop device that was not torn down.
That is despite the fact the test is above as simple as it gets - loop 5 or 6 times running mkfs.xfs with different paramters:
lofile=$(losetup -f)
losetup $lofile "$testfile"
"$MKFS_XFS_PROG" -b size=512 $lofile >/dev/null || echo "mkfs failed!" sync
losetup -d $lofileAnd losteup -d $lofile is failing with EBUSY on 1-3 of these loops every time the test is run.
Turns out that blkid is running simultaneously with losetup -d, and so it sees an elevated reference count and returns EBUSY. But why is blkid running? It's obvious, isn't it? udev has decided to try and find out what is on the block device as a result of a creation notification. And it is racing with mkfs, so might still be scanning the device when mkfs finishes and we try to tear it down.
Quoted 5 lines
So, make losetup -d force autoremove behaviour. That is, when the last reference goes away, tear down the device. xfstests wants it *gone*, not causing random teardown failures when we know that all the operations the tests have specifically run on the device have completed and are no longer referencing the loop device.
I hear that %^#@#! blkid behavior, it is such a pain in the neck. I don't know how many times I've had to explain that behaviour to people who run write testing with tracing, wonder wtf there are reads in the trace.
Quoted 4 lines
Quoted 2 lines
Patch looks fine, seems like the sane thing to do (lazy-remove on last drop) for this case.
Do we also want to prevent further opens?
That's not a bad idea, at least it would be the logical thing to do. But it does get into the realm of potentially breaking existing behaviour.
Signature · 1 line
Jens Axboe
Quoted 46 lines · Jens Axboe
Quoted 45 lines · Jeff Moyer
Quoted 43 lines · Jens Axboe
Quoted 31 lines · Dave Chinner
From: Dave Chinner <[email protected]>
xfstests has always had random failures of tests due to loop devices failing to be torn down and hence leaving filesytems that cannot be unmounted. This causes test runs to immediately stop.
Over the past 6 or 7 years we've added hacks like explicit unmount -d commands for loop mounts, losetup -d after unmount -d fails, etc, but still the problems persist. Recently, the frequency of loop related failures increased again to the point that xfstests 259 will reliably fail with a stray loop device that was not torn down.
That is despite the fact the test is above as simple as it gets - loop 5 or 6 times running mkfs.xfs with different paramters:
lofile=$(losetup -f)
losetup $lofile "$testfile"
"$MKFS_XFS_PROG" -b size=512 $lofile >/dev/null || echo "mkfs failed!" sync
losetup -d $lofileAnd losteup -d $lofile is failing with EBUSY on 1-3 of these loops every time the test is run.
Turns out that blkid is running simultaneously with losetup -d, and so it sees an elevated reference count and returns EBUSY. But why is blkid running? It's obvious, isn't it? udev has decided to try and find out what is on the block device as a result of a creation notification. And it is racing with mkfs, so might still be scanning the device when mkfs finishes and we try to tear it down.
Quoted 5 lines
So, make losetup -d force autoremove behaviour. That is, when the last reference goes away, tear down the device. xfstests wants it *gone*, not causing random teardown failures when we know that all the operations the tests have specifically run on the device have completed and are no longer referencing the loop device.
I hear that %^#@#! blkid behavior, it is such a pain in the neck. I don't know how many times I've had to explain that behaviour to people who run write testing with tracing, wonder wtf there are reads in the trace.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Patch looks fine, seems like the sane thing to do (lazy-remove on last drop) for this case.
Do we also want to prevent further opens?
That's not a bad idea, at least it would be the logical thing to do. But it does get into the realm of potentially breaking existing behaviour.
What do you think could rely on the existing behaviour (that isn't broken by design)?
-Jeff