Fix API breakage in libblkid
cover Fix API breakage in libblkid
From: Carlos Maiolino <[email protected]>
Patch d05a84b22e54 ("libblkid: check for private DM device before open") broke blkid_new_probe_from_filename() API.
Before the patch users were able, via the low-level API, to open and create blkid probes via the device's filename even from private device-mapper devices.
This change broke Stratis project and xfsprogs.
xfsprogs uses blkid_new_probe_from_filename() to gather topology information from the device, and the above mentioned change now prevents it to be done on device-mapper private devices, as Stratis does by attempting to initialize a XFS filesystem on it.
Quoting the patch description:
"
blkid_new_probe_from_filename() opens the device before calling blkid_probe_set_device(), which checks sysfs_devno_is_dm_private() and sets BLKID_FL_NOSCAN_DEV. But the open() itself bumps the kernel open count, so a concurrent DM_DEVICE_REMOVE ioctl sees EBUSY even though blkid never actually probes the device.
"
I don't think the last statement here is correct. blkid_probe_set_device() marks the probe as BLKID_FL_NOSCAN_DEV, but it does not error out, so, for low-level API calls, we simply ended up with a probe with BLKID_FL_NOSCAN_DEV set. But the call succeeded and we ended up with a probe to use and query device's information.
As far as I understood it, the patch aimed to close a possible race window when issuing a DM_DEVICE_REMOVE ioctl() to the same device being probed by blkid_new_probe_from_filename().
Regarding the race window which this patch, at least for the low-level API users, this seems to be something that should be dealt in the user's side, not within the library. But I didn't dig into the details of the aforementioned race.
Please take the above with a grain of salt though, we've been using libblkid for ages, but I never actually looked into the implementation until today, so, hopefully I did get the details right.
As for xfsprogs, I have a patch which 'fixes' it replacing blkid_new_probe_from_filename() by blkid_probe_set_device(), but this is just taping over the root cause, which is the API breakage.
Carlos Maiolino (2):
Revert "libblkid: add debug message for private DM device skip" Revert "libblkid: check for private DM device before open"
libblkid/src/probe.c | 15 +-------------- 1 file changed, 1 insertion(+), 14 deletions(-)
Signature · 1 line
2.53.0
RFC PATCH 1/2 Revert "libblkid: add debug message for private DM device skip"
From: Carlos Maiolino <[email protected]>
This reverts commit b85d46a2edf3f871504ca86080102cde93e3725c.
--- libblkid/src/probe.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
libblkid/src/probe.c +1 -4
@@ -224,11 +224,8 @@ blkid_probe blkid_new_probe_from_filename(const char *filename)* DM_DEVICE_REMOVE would otherwise see EBUSY. */ if (stat (filename, &sb) ==0 &&S_ISBLK (sb.st_mode) && -sysfs_devno_is_dm_private (sb.st_rdev,NULL )) { -DBG (LOWPROBE ,ul_debug ("ignore private device mapper device" )); - errno =EINVAL ; +sysfs_devno_is_dm_private (sb.st_rdev,NULL ))return NULL ; - } fd =open (filename,O_RDONLY |O_CLOEXEC |O_NONBLOCK );if (fd <0 )
Signature · 1 line
2.53.0
RFC PATCH 2/2 Revert "libblkid: check for private DM device before open"
In reply to [email protected]From: Carlos Maiolino <[email protected]>
This reverts commit d05a84b22e549527cbcbcc7d5efc6bad06668170.
--- libblkid/src/probe.c | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-)
libblkid/src/probe.c +1 -11
@@ -216,18 +216,8 @@ blkid_probe blkid_new_probe_from_filename(const char *filename) {int fd;blkid_probe pr =NULL ; -struct stat sb; - -/* -* Check for private device-mapper devices (LVM internals, etc.) -* before open() to avoid bumping the kernel open count. A racing -* DM_DEVICE_REMOVE would otherwise see EBUSY. -*/ -if (stat (filename, &sb) ==0 &&S_ISBLK (sb.st_mode) && -sysfs_devno_is_dm_private (sb.st_rdev,NULL )) -return NULL ; - fd =open (filename,O_RDONLY |O_CLOEXEC |O_NONBLOCK ); + fd =open (filename,O_RDONLY |O_CLOEXEC |O_NONBLOCK );if (fd <0 )return NULL ;
Signature · 1 line
2.53.0
Dne 08. 04. 26 v 12:35 [email protected] napsal(a):
Quoted 9 lines
From: Carlos Maiolino <[email protected]>
Patch
d05a84b22e54("libblkid: check for private DM device before open") brokeblkid_new_probe_from_filename()API.Before the patch users were able, via the low-level API, to open and create blkid probes via the device's filename even from private device-mapper devices.
Hi
So I'm not sure how it happened that Stratis kind of 'reuses' libdm private logic we have embedded into util-linux - but something wrong is likely happening on Stratis side here.
For libdm & util-linux - there is basic principle - when udev should be scanning a 'DM' device that is supposed to be private - in our case it's
uuid with "LVM-" prefix and "-anysuffix: - such device is supposed to be NOT touched by any udev rule logic - it's private - so nothing should be 'randomly' opening a device at unpredictable moments..
Somehow it seems Stratis selected to use similar notion (at least when I look there in sysfs_devno_is_dm_private() - there is check for uuid:
"stratis-1-private"
Quoted 3 lines
This change broke Stratis project and xfsprogs.
xfsprogs usesblkid_new_probe_from_filename()to gather topology information from the device, and the above mentioned change now prevents
If the private is not meant to be private - just drop:
} else if (strncmp(id, "stratis-1-private", 17) == 0) {
rc = 1;
}from sysfs_devno_is_dm_private()
Quoted 8 lines
it to be done on device-mapper private devices, as Stratis does by attempting to initialize a XFS filesystem on it.
Quoted 1 line
I don't think the last statement here is correct.
blkid_probe_set_device()marks the probe asBLKID_FL_NOSCAN_DEV, but it does not error out, so, for low-level API calls, we simply ended up with a probe withBLKID_FL_NOSCAN_DEVset. But the call succeeded and we ended up with a probe to use and query device's information.
Please don't try to break original LVM logic which was the main reason why sysfs_devno_is_dm_private() even exists in the first place.
Quoted 4 lines
As far as I understood it, the patch aimed to close a possible race window when issuing a
DM_DEVICE_REMOVEioctl()to the same device being probed byblkid_new_probe_from_filename().
Yes - but there is higher logic behind - private 'device' is private and should not be randomly opened by any tool.
Quoted 3 lines
As for xfsprogs, I have a patch which 'fixes' it replacing
blkid_new_probe_from_filename()byblkid_probe_set_device(), but this is just taping over the root cause, which is the API breakage.
I think main problem is that Stratis mangled 'private' meaning in some way.
So let's fix Stratis instead of breaking correct logic in util-linux made for lvm2.
Regards
Zdenek
Quoted 23 lines · Zdenek Kabelac
Dne 08. 04. 26 v 12:35 [email protected] napsal(a):
Quoted 9 lines
From: Carlos Maiolino <[email protected]>
Patch
d05a84b22e54("libblkid: check for private DM device before open") brokeblkid_new_probe_from_filename()API.Before the patch users were able, via the low-level API, to open and create blkid probes via the device's filename even from private device-mapper devices.
Hi
So I'm not sure how it happened that Stratis kind of 'reuses' libdm private logic we have embedded into util-linux - but something wrong is likely happening on Stratis side here.
For libdm & util-linux - there is basic principle - when udev should be scanning a 'DM' device that is supposed to be private - in our case it's uuid with "LVM-" prefix and "-anysuffix: - such device is supposed to be NOT touched by any udev rule logic - it's private - so nothing should be 'randomly' opening a device at unpredictable moments..
Quoted 3 lines
Somehow it seems Stratis selected to use similar notion (at least when I look there in
sysfs_devno_is_dm_private()- there is check for uuid:
"stratis-1-private"
Thanks for the details. Indeed I noticed sysfs_devno_is_dm_private() skipping 'stratis-1-private' suffixed devices, but to be honest, their functionality is opaque to me, I don't really know what they mean or how they are used.
I got a report that mkfs.xfs stopped working for them, and my investigation led to this subtle behavior change in blkid_new_probe_from_filename().
So, IIUC what you're saying, stratis shouldn't be using devices they flag as private the way they are using, so I agree, and perhaps their understanding of what a private device means is wrong and not using the API properly.
On the other hand though -- playing The Devil's Advocate a bit --, this is still a subtle API break and I don't think your patch is the right way to fix the race you're seeing.
I honestly couldn't find any information about it, whether blkid_new_probe_from_filename() could be used or not with private devices, the only thing I really see (from a user perspective) was the library call not working as before.
Also, reading what you explained above regarding the usage, I think I was right when I said this is not the right way to fix the race you mentioned. Because the race is still there, it is now just hidden for LVM's specific use case.
Quoting libblkid documentation for blkid_new_probe_from_filename():
"
This function is same as call open(filename), blkid_new_probe() and blkid_probe_set_device(pr, fd, 0, 0).
"
This was exactly how I "fixed" xfsprogs to allow Stratis behavior to continue the way it is. Which by your explanation above is incorrect, as is the same sequence call that you mentioned that should not be allowed... open() the device then probe_set_device()... Which tells me that any libblkid user following this same sequence is fundamentally broken.
So, although I agree with you that Stratis is using libblkid incorrectly, I still think this patch is wrong, as it changes the library's behavior to fix a problem in LVM (if there is where the race was), and not some inherent library problem. If, no user can use libblkid to query into a private device, then the above lib call sequence wasn't supposed to work either, and perhaps then, blkid_probe_set_device() should actually fail in a private device.
Again, I just looked into the libblkid low-level implementation today, but it doesn't seem to be the library's job to be responsible for a race between open() and ioctl(DM_DEVICE_REMOVE).
Cheers,
Carlos
Quoted 43 lines
Quoted 5 lines
This change broke Stratis project and xfsprogs.
xfsprogs usesblkid_new_probe_from_filename()to gather topology information from the device, and the above mentioned change now preventsIf the private is not meant to be private - just drop:
} else if (strncmp(id, "stratis-1-private", 17) == 0) { rc = 1; }from
sysfs_devno_is_dm_private()Quoted 8 lines
it to be done on device-mapper private devices, as Stratis does by attempting to initialize a XFS filesystem on it.
Quoted 1 line
I don't think the last statement here is correct.
blkid_probe_set_device()marks the probe asBLKID_FL_NOSCAN_DEV, but it does not error out, so, for low-level API calls, we simply ended up with a probe withBLKID_FL_NOSCAN_DEVset. But the call succeeded and we ended up with a probe to use and query device's information.Please don't try to break original LVM logic which was the main reason why
sysfs_devno_is_dm_private()even exists in the first place.Quoted 4 lines
As far as I understood it, the patch aimed to close a possible race window when issuing a
DM_DEVICE_REMOVEioctl()to the same device being probed byblkid_new_probe_from_filename().Yes - but there is higher logic behind - private 'device' is private and should not be randomly opened by any tool.
Quoted 3 lines
As for xfsprogs, I have a patch which 'fixes' it replacing
blkid_new_probe_from_filename()byblkid_probe_set_device(), but this is just taping over the root cause, which is the API breakage.I think main problem is that Stratis mangled 'private' meaning in some way.
So let's fix Stratis instead of breaking correct logic in util-linux made for lvm2.
Dne 08. 04. 26 v 15:19 Carlos Maiolino napsal(a):
Quoted 12 lines · Zdenek Kabelac
Quoted 3 lines
Dne 08. 04. 26 v 12:35 [email protected] napsal(a):
Quoted 2 lines
From: Carlos Maiolino <[email protected]>
So, although I agree with you that Stratis is using libblkid incorrectly, I still think this patch is wrong, as it changes the library's behavior to fix a problem in LVM (if there is where the race was), and not some inherent library problem. If, no user can use libblkid to query into a private device, then the above lib call sequence wasn't supposed to work either, and perhaps then,
blkid_probe_set_device()should actually fail in a private device.
Quoted 4 lines
Again, I just looked into the libblkid low-level implementation today, but it doesn't seem to be the library's job to be responsible for a race between
open()andioctl(DM_DEVICE_REMOVE).
Hi
The main missed concept here is - this is not breaking any API.
The concept of private DM device is to be a device that is not being touched by libblkid. So this was a pure bug in the libblkid code when there was a code path skipping check (and actually doing this check after opening private device).
And actually IMHO there is a bigger issue - xfs tools are actually relying too much on content of udev DB - this DB is by design unreliable - so it's likely good for some quick checking - but i.e. in lvm2 - we always do a real disk read - udevDB is only advisory.
Regards
Zdenek
Quoted 13 lines · Zdenek Kabelac
Dne 08. 04. 26 v 15:19 Carlos Maiolino napsal(a):
Quoted 12 lines · Zdenek Kabelac
Quoted 3 lines
Dne 08. 04. 26 v 12:35 [email protected] napsal(a):
Quoted 2 lines
From: Carlos Maiolino <[email protected]>
So, although I agree with you that Stratis is using libblkid incorrectly, I still think this patch is wrong, as it changes the library's behavior to fix a problem in LVM (if there is where the race was), and not some inherent library problem. If, no user can use libblkid to query into a private device, then the above lib call sequence wasn't supposed to work either, and perhaps then,
blkid_probe_set_device()should actually fail in a private device.
Quoted 8 lines
Quoted 4 lines
Again, I just looked into the libblkid low-level implementation today, but it doesn't seem to be the library's job to be responsible for a race between
open()andioctl(DM_DEVICE_REMOVE).Hi
The main missed concept here is - this is not breaking any API.
I disagree... Same lib call, same device, works before your patch, does not after... You may call it a bug that shouldn't be relied on, but that's for sure an API breakage.
Quoted 5 lines
The concept of private DM device is to be a device that is not being touched by libblkid. So this was a pure bug in the libblkid code when there was a code path skipping check (and actually doing this check after opening private device).
Ok, you told me from where it comes from.
How different is it of any other tool that uses libblkid doing open(); blkid_new_probe(); blkid_probe_set_device();
Instead of calling blkid_new_probe_from_filename() directly?
Because this still can easily access a private device.
Quoted 5 lines
And actually IMHO there is a bigger issue - xfs tools are actually relying too much on content of udev DB - this DB is by design unreliable - so it's likely good for some quick checking - but i.e. in lvm2 - we always do a real disk read - udevDB is only advisory.
We don't do anything with udev. We use libblkid to query device's geometry to properly configure the filesystem to be initialized there. If blkid_new_probe_from_filename() fails, mkfs abort. It's up to the user to pass the right device.
Quoted 6 lines
Regards
Zdenek
Dne 08. 04. 26 v 16:49 Carlos Maiolino napsal(a):
Quoted 19 lines · Zdenek Kabelac
Quoted 18 lines
Dne 08. 04. 26 v 15:19 Carlos Maiolino napsal(a):
Quoted 16 lines · Zdenek Kabelac
Quoted 3 lines
Dne 08. 04. 26 v 12:35 [email protected] napsal(a):
Quoted 2 lines
From: Carlos Maiolino <[email protected]>
So, although I agree with you that Stratis is using libblkid incorrectly, I still think this patch is wrong, as it changes the library's behavior to fix a problem in LVM (if there is where the race was), and not some inherent library problem. If, no user can use libblkid to query into a private device, then the above lib call sequence wasn't supposed to work either, and perhaps then,
blkid_probe_set_device()should actually fail in a private device.Again, I just looked into the libblkid low-level implementation today, but it doesn't seem to be the library's job to be responsible for a race between
open()andioctl(DM_DEVICE_REMOVE).
Quoted 8 lines
Quoted 3 lines
Hi
The main missed concept here is - this is not breaking any API.
I disagree... Same lib call, same device, works before your patch, does not after... You may call it a bug that shouldn't be relied on, but that's for sure an API breakage.
Well - DM with private suffix is private - that's the API and defined long time in history - to avoid any interaction with private device and libblkid.
The fact here was a bug for a long while that such device could have been still explored by some libblkid calls does not make it a legit API.
Stratis has 2 easy solutions/fixes:
a) dropping 'private' check for their device in util-linux.
b) use non-private UUID for a device that should be managed by user-space tools.
For lvm2/dm - it's clear we never want libblkid to access internal raid legs or other types of private DM devs.
Also it should be noted - Stratis should not have been using 'internal' logic without fully understanding concept of 'private' device in the 1st. place....
So eventually as a 'middle' ground solution we may possibly have new:
sysfs_devno_is_dm_really_private()
that would be called in my patch in this function that would be a replica of sysfs_devno_is_dm_private() just with LVM suffix check. But still this kind of defeats the original 'intention' of this check and makes the code harder to follow.
So I'd still prefer a fix on Stratis to correct notation of the private device.
Regads
Zdenek