Assumption on fixed device numbers in Plasma's desktop search Baloo
Subsystems (2)
Hi!
I found repeatedly that Baloo indexes the same files twice or even more often after a while.
I reported this upstream in:
Bug 438434 - Baloo appears to be indexing twice the number of files than are actually in my home directory
https://bugs.kde.org/show_bug.cgi?id=438434
And got back that if the device number changes, Baloo will think it has new files even tough the path is still the same. And found over time that the device number for the single BTRFS filesystem on a NVMe SSD in a ThinkPad T14 Gen1 AMD can change. It is not (maybe yet) RAID 1. I do have BTRFS RAID 1 in another laptop and there I also had this issue already.
I argued that a desktop application has no business to rely on a device number and got back that search/indexing is in the middle between an application and system software. And that Baloo needs an "invariant" for a file. See comment #11 of that bug report:
https://bugs.kde.org/show_bug.cgi?id=438434#c11
I got the suggestion to try to find a way to tell the kernel to use a fixed device number.
I still think, an application or an infrastructure service for a desktop environment or even anything else in user space should not rely on a device number to be fixed and never change upon reboots.
But maybe you have a different idea about that and it is okay for an userspace component to do that. I would like to hear your idea about that.
Another question would be whether I could somehow make sure that the device number does not change, even if just as a work-around. I know for NFS there is a fsid= mount option, but it does not appear to be something generic, at least the mount man page seems to have nothing related to fsid.
Best,
Signature · 1 line
Martin
Quoted 10 lines · Martin Steigerwald
Hi!
I found repeatedly that Baloo indexes the same files twice or even more often after a while.
I reported this upstream in:
Bug 438434 - Baloo appears to be indexing twice the number of files than are actually in my home directory
Quoted 8 lines
https://bugs.kde.org/show_bug.cgi?id=438434
And got back that if the device number changes, Baloo will think it has new files even tough the path is still the same. And found over time that the device number for the single BTRFS filesystem on a NVMe SSD in a ThinkPad T14 Gen1 AMD can change. It is not (maybe yet) RAID 1. I do have BTRFS RAID 1 in another laptop and there I also had this issue already.
Since btrfs has multi-device support by default, it reports anonymous device number, just as if you use a filesystem over LVM.
The problem is why the anonymous device number change.
If the fs is always mounted at a fixed sequence with fixed snapshots/subvolume mount, it should not get a new anonymous device number.
But if snapshots or new subvolumes are involved, or just mounting/reading subvolumes in different order, then the device number for each subvolume will change.
Quoted 7 lines
I argued that a desktop application has no business to rely on a device number and got back that search/indexing is in the middle between an application and system software. And that Baloo needs an "invariant" for a file. See comment #11 of that bug report:
Well, a lot of tools relies on device number to distinguish filesystem boundary, like find.
Thus it's a little hard to argue.
But on the other hand, it also means baloo can't handle regular fs over LVM cases well neither.
Quoted 3 lines
I got the suggestion to try to find a way to tell the kernel to use a fixed device number.
I don't think it's possible for btrfs, as each subvolume get its anonymous device number assigned when it gets first read.
Thus it's really hard to make it fixed, as the reason for anonymous device number is to avoid conflicts.
Quoted 4 lines
I still think, an application or an infrastructure service for a desktop environment or even anything else in user space should not rely on a device number to be fixed and never change upon reboots.
Well, LVM/device mapper is doing the same thing, a lot of behavior change is never a good idea for the kernel.
Thus for use cases where we really need a proper mapping, we use hashes, not just device number, like what we did in dupremover.
Quoted 7 lines
But maybe you have a different idea about that and it is okay for an userspace component to do that. I would like to hear your idea about that.
Another question would be whether I could somehow make sure that the device number does not change, even if just as a work-around.
If you really just want a fixed device number, you can ensure that by:
- Make sure all users of anonymous devices get fixed sequence Things like device mapper/LVM, btrfs should get loaded/initialized in a fixed order.
- Make sure the subvolume you care always get mounted/read before any other subvolumes
So that the target subvolume always get the first device number in the pool.
But this also means, all later subvolumes not in the fixed mount/read sequence can not get a fixed number.
Thanks,
Qu
Quoted 8 lines
I know for
NFS there is a fsid= mount option, but it does not appear to be something generic, at least the mount man page seems to have nothing related to fsid.Best,
Qu Wenruo - 26.06.21, 02:27:54 CEST:
Quoted 20 lines · Martin Steigerwald
Quoted 10 lines
Hi!
I found repeatedly that Baloo indexes the same files twice or even more often after a while.
I reported this upstream in:
Bug 438434 - Baloo appears to be indexing twice the number of files than are actually in my home directory
Quoted 8 lines
https://bugs.kde.org/show_bug.cgi?id=438434
And got back that if the device number changes, Baloo will think it has new files even tough the path is still the same. And found over time that the device number for the single BTRFS filesystem on a NVMe SSD in a ThinkPad T14 Gen1 AMD can change. It is not (maybe yet) RAID 1. I do have BTRFS RAID 1 in another laptop and there I also had this issue already.
Quoted 2 lines
Since btrfs has multi-device support by default, it reports anonymous device number, just as if you use a filesystem over LVM.
Ah, this!
I forgot to mention that: I use BTRFS on top of LVM on top of LUKS based dm-crypt on a partition on the NVMe SSD. Sorry, somehow I forgot to mention that here. I mentioned it in the bug report. I'd use a different approach if there would be one that give me full disk encryption. I am not willing to use ecryptfs on top of BTRFS and as far as I know BTRFS cannot yet encrypt by itself.
I still think this could give a fixed order of loading:
1. Unlock LUKS.
2. Activate LVM logical volumes. No idea whether that happens in a fixed order though or whether it can have a different order on each boot.
3. Mount BTRFS. /home is always on the same subvolume. So that should not change.
Quoted 1 line
The problem is why the anonymous device number change.
Good question. Maybe I have an idea about that. See below.
Quoted 5 lines
Quoted 5 lines
I argued that a desktop application has no business to rely on a device number and got back that search/indexing is in the middle between an application and system software. And that Baloo needs an "invariant" for a file. See comment #11 of that bug report:
Quoted 8 lines
Quoted 1 line
Well, a lot of tools relies on device number to distinguish filesystem boundary, like find.
Thus it's a little hard to argue.But on the other hand, it also means baloo can't handle regular fs over LVM cases well neither.
Yes. Also it could not handle the case of a driver loading race condition with two or more different controllers in a desktop machine.
Quoted 8 lines
Quoted 2 lines
I got the suggestion to try to find a way to tell the kernel to use a fixed device number.
I don't think it's possible for btrfs, as each subvolume get its anonymous device number assigned when it gets first read.
Thus it's really hard to make it fixed, as the reason for anonymous device number is to avoid conflicts.
Fair enough.
Quoted 9 lines
Quoted 3 lines
I still think, an application or an infrastructure service for a desktop environment or even anything else in user space should not rely on a device number to be fixed and never change upon reboots.
Well, LVM/device mapper is doing the same thing, a lot of behavior change is never a good idea for the kernel.
Thus for use cases where we really need a proper mapping, we use hashes, not just device number, like what we did in dupremover.
I think I suggested that some time ago.
Quoted 8 lines
Quoted 2 lines
Another question would be whether I could somehow make sure that the device number does not change, even if just as a work-around.
If you really just want a fixed device number, you can ensure that by:
- Make sure all users of anonymous devices get fixed sequence Things like device mapper/LVM, btrfs should get loaded/initialized in a fixed order.
Ah, I see.
Quoted 4 lines
- Make sure the subvolume you care always get mounted/read before any other subvolumes
So that the target subvolume always get the first device number in
the pool.
Hmm, that may be a pointer. This is what I currently have in fstab:
/dev/nvme/home /home btrfs lazytime,compress=zstd 0 0 /dev/nvme/home /zeit/home btrfs subvol=zeit 0 0
In the first line the default subvolume is used which I changed accordingly after creating this BTRFS. I use the approach to keep (temporary) snapshots separated from the directory tree in /home.
Could it be that this order between these two mounts is not the same on every boot? I use Devuan with Runit, so the mounting would happen by some init scripts (instead of Systemd).
I am not aware of an option for fstab to mount this one first and then the other second, but I could set the second mount to noauto and mount it when I need it.
Quoted 2 lines
But this also means, all later subvolumes not in the fixed
mount/read sequence can not get a fixed number.
I somehow thought this would get complicated.
Best,
Signature · 1 line
Martin
Quoted 33 lines · Martin Steigerwald
Qu Wenruo - 26.06.21, 02:27:54 CEST:
Quoted 20 lines · Martin Steigerwald
Quoted 10 lines
Hi!
I found repeatedly that Baloo indexes the same files twice or even more often after a while.
I reported this upstream in:
Bug 438434 - Baloo appears to be indexing twice the number of files than are actually in my home directory
Quoted 8 lines
https://bugs.kde.org/show_bug.cgi?id=438434
And got back that if the device number changes, Baloo will think it has new files even tough the path is still the same. And found over time that the device number for the single BTRFS filesystem on a NVMe SSD in a ThinkPad T14 Gen1 AMD can change. It is not (maybe yet) RAID 1. I do have BTRFS RAID 1 in another laptop and there I also had this issue already.
Quoted 2 lines
Since btrfs has multi-device support by default, it reports anonymous device number, just as if you use a filesystem over LVM.
Ah, this!
I forgot to mention that: I use BTRFS on top of LVM on top of LUKS based dm-crypt on a partition on the NVMe SSD. Sorry, somehow I forgot to mention that here. I mentioned it in the bug report. I'd use a different approach if there would be one that give me full disk encryption. I am not willing to use ecryptfs on top of BTRFS and as far as I know BTRFS cannot yet encrypt by itself.
Quoted 6 lines
I still think this could give a fixed order of loading:
1. Unlock LUKS.
2. Activate LVM logical volumes. No idea whether that happens in a fixed order though or whether it can have a different order on each boot.
LVM/LUKS normally isn't a big deal, as most of them are initialized before btrfs, and have a pretty fixed initialization sequence.
Unless you change the LVM setup, then at least all your LVs should have a fixed device number.
(But there are still cases where kernel update may change them)
Quoted 3 lines
3. Mount BTRFS. /home is always on the same subvolume. So that should not change.
Normally it won't change.
But it's more dependent on the btrfs behavior.
Thus I'm not that confident it won't change forever.
But at this point I guess you already get the point, under normal cases, no config change then device number won't change.
However any change in kernel/storage stack/config can lead to different device number.
Quoted 16 lines
Quoted 2 lines
The problem is why the anonymous device number change.
Good question. Maybe I have an idea about that. See below.
Quoted 11 lines
Quoted 6 lines
I argued that a desktop application has no business to rely on a device number and got back that search/indexing is in the middle between an application and system software. And that Baloo needs an "invariant" for a file. See comment #11 of that bug report:
Well, a lot of tools relies on device number to distinguish filesystem boundary, like find.
Thus it's a little hard to argue.
Quoted 5 lines
Quoted 2 lines
But on the other hand, it also means baloo can't handle regular fs over LVM cases well neither.
Yes. Also it could not handle the case of a driver loading race condition with two or more different controllers in a desktop machine.
Thus the idea from Neil should help, instead of using device number, using f_fsid from statfs() should provide a way more stable result.
And f_fsid can also handle btrfs subvolumes pretty well.
But this also means, if one day you change your default/mounted subvolume, baloo will again rebuild the cache using the new f_fsid.
Quoted 45 lines
Quoted 9 lines
Quoted 3 lines
I got the suggestion to try to find a way to tell the kernel to use a fixed device number.
I don't think it's possible for btrfs, as each subvolume get its anonymous device number assigned when it gets first read.
Thus it's really hard to make it fixed, as the reason for anonymous device number is to avoid conflicts.
Fair enough.
Quoted 9 lines
Quoted 3 lines
I still think, an application or an infrastructure service for a desktop environment or even anything else in user space should not rely on a device number to be fixed and never change upon reboots.
Well, LVM/device mapper is doing the same thing, a lot of behavior change is never a good idea for the kernel.
Thus for use cases where we really need a proper mapping, we use hashes, not just device number, like what we did in dupremover.
I think I suggested that some time ago.
Quoted 8 lines
Quoted 2 lines
Another question would be whether I could somehow make sure that the device number does not change, even if just as a work-around.
If you really just want a fixed device number, you can ensure that by:
- Make sure all users of anonymous devices get fixed sequence Things like device mapper/LVM, btrfs should get loaded/initialized in a fixed order.
Ah, I see.
Quoted 4 lines
- Make sure the subvolume you care always get mounted/read before any other subvolumes
So that the target subvolume always get the first device number in
the pool.Hmm, that may be a pointer. This is what I currently have in fstab:
/dev/nvme/home/home btrfs lazytime,compress=zstd 0 0/dev/nvme/home/zeit/home btrfs subvol=zeit 0 0
Quoted 8 lines
In the first line the default subvolume is used which I changed accordingly after creating this BTRFS. I use the approach to keep (temporary) snapshots separated from the directory tree in /home.
Could it be that this order between these two mounts is not the same on every boot?
I use Devuan with Runit, so the mounting would happen by some init scripts (instead of Systemd).
Then it's out of the scope of btrfs.
I was just wondering if systemd is involved, but you just ruled it out. But still if the init tool choose to shuffle the mount sequence to do more parallel mounts, then device number will be even more unreliable.
Quoted 9 lines
I am not aware of an option for fstab to mount this one first and then the other second, but I could set the second mount to noauto and mount it when I need it.
Quoted 2 lines
But this also means, all later subvolumes not in the fixed
mount/read sequence can not get a fixed number.I somehow thought this would get complicated.
It's already complicated.
So this just proves Neil is right, device number is only reliable at the lifespan of the fs, nothing else.
Thanks,
Qu
Quoted 3 lines
Best,
Qu Wenruo - 26.06.21, 11:33:17 CEST:
Quoted 7 lines
Quoted 7 lines
I am not aware of an option for fstab to mount this one first and then the other second, but I could set the second mount to noauto and mount it when I need it.
Quoted 2 lines
But this also means, all later subvolumes not in the fixed mount/read sequence can not get a fixed number.
Quoted 6 lines
Quoted 1 line
I somehow thought this would get complicated.
It's already complicated.
So this just proves Neil is right, device number is only reliable at the lifespan of the fs, nothing else.
Thank you again.
I informed upstream about the conclusions from this thread.
Let's see what they come up with.
They have an energy efficiency goal, for that it would be desirable to stop indexing files twice or thrice or even more times. :)
Best,
Signature · 1 line
Martin
Quoted 19 lines · Martin Steigerwald
Hi!
I found repeatedly that Baloo indexes the same files twice or even more often after a while.
I reported this upstream in:
Bug 438434 - Baloo appears to be indexing twice the number of files than are actually in my home directory
https://bugs.kde.org/show_bug.cgi?id=438434
And got back that if the device number changes, Baloo will think it has new files even tough the path is still the same. And found over time that the device number for the single BTRFS filesystem on a NVMe SSD in a ThinkPad T14 Gen1 AMD can change. It is not (maybe yet) RAID 1. I do have BTRFS RAID 1 in another laptop and there I also had this issue already.
Quoted 3 lines
I argued that a desktop application has no business to rely on a device number and got back that search/indexing is in the middle between an application and system software.
NO SOFTWARE can rely on device numbers being stable in Linux. Not desktop, not system, not anything. They are stable while the device is in use (e.g. while the filesystem is mounted) but can definitely change on reboot. This has been the case since about Linux 2.4.
Quoted 2 lines
And that Baloo needs an "invariant" for
a file. See comment #11 of that bug report:
That is really hard to provide in general. Possibly the best approach is to use the statfs() systemcall to get the "f_fsid" field. This is 64bits. It is not supported uniformly well by all filesystems, but I think it is at least not worse than using the device number. For a lot of older filesystems it is just an encoding of the device number.
For btrfs, xfs, ext4 it is much much better.
NeilBrown
Quoted 27 lines
https://bugs.kde.org/show_bug.cgi?id=438434#c11
I got the suggestion to try to find a way to tell the kernel to use a fixed device number.
I still think, an application or an infrastructure service for a desktop environment or even anything else in user space should not rely on a device number to be fixed and never change upon reboots.
But maybe you have a different idea about that and it is okay for an userspace component to do that. I would like to hear your idea about that.
Another question would be whether I could somehow make sure that the device number does not change, even if just as a work-around. I know for NFS there is a fsid= mount option, but it does not appear to be something generic, at least the mount man page seems to have nothing related to fsid.
Best,
Signature · 1 line
Martin
Quoted 11 lines · NeilBrown
Quoted 2 lines · Martin Steigerwald
And that Baloo needs an "invariant" for
a file. See comment #11 of that bug report:That is really hard to provide in general. Possibly the best approach is to use the
statfs()systemcall to get the "f_fsid" field. This is 64bits. It is not supported uniformly well by all filesystems, but I think it is at least not worse than using the device number. For a lot of older filesystems it is just an encoding of the device number.For btrfs, xfs, ext4 it is much much better.
How about combining the UUID of the partition with the file path? An example from one of the VMs on my workstation:
$ df .
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda1 25670972 12730276 11613648 53% /
$ lsblk -O | grep vda1
└─vda1 vda1 /dev/vda1 252:1 11.1G 24.5G ext4 12.1G 50% 1.0
/ 84cebea8-7e6f-4c2a-8a1b-8bc0c9744751 ae2151de
dos 0x83 Linux ae2151de-01
0x80 128 0 0 0
25G root disk brw-rw---- 0 512
0 512 512 1 mq-deadline 256 part 0 512B
2G 0 0B 0 vda block:virtio:pci
none 0In other words, UUID 84cebea8-7e6f-4c2a-8a1b-8bc0c9744751 has been associated with the block device under the filesystem that owns the directory from which the 'df' command has been run.
Bart.
Quoted 11 lines · Bart Van Assche
Quoted 10 lines · NeilBrown
Quoted 2 lines · Martin Steigerwald
And that Baloo needs an "invariant" for
a file. See comment #11 of that bug report:That is really hard to provide in general. Possibly the best approach is to use the
statfs()systemcall to get the "f_fsid" field. This is 64bits. It is not supported uniformly well by all filesystems, but I think it is at least not worse than using the device number. For a lot of older filesystems it is just an encoding of the device number.
Quoted 4 lines
Quoted 1 line
For btrfs, xfs, ext4 it is much much better.
How about combining the UUID of the partition with the file path? An example from one of the VMs on my workstation:
A btrfs filesystem can span multiple partitions, and those partitions can be added and removed dynamically. So you could migrated from one to another.
f_fsid really is best for any modern filesystem.
NeilBrown
Quoted 12 lines · NeilBrown
Quoted 11 lines · Bart Van Assche
Quoted 10 lines · NeilBrown
Quoted 2 lines · Martin Steigerwald
And that Baloo needs an "invariant" for
a file. See comment #11 of that bug report:That is really hard to provide in general. Possibly the best approach is to use the
statfs()systemcall to get the "f_fsid" field. This is 64bits. It is not supported uniformly well by all filesystems, but I think it is at least not worse than using the device number. For a lot of older filesystems it is just an encoding of the device number.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
For btrfs, xfs, ext4 it is much much better.
How about combining the UUID of the partition with the file path? An example from one of the VMs on my workstation:
A btrfs filesystem can span multiple partitions, and those partitions can be added and removed dynamically. So you could migrated from one to another.
I suspect it was intended to be "filesytemm UUID". At least that is the field in lsblk output that was referenced.
Quoted 4 lines
f_fsidreally is best for any modern filesystem.NeilBrown
Quoted 18 lines · Andrei Borzenkov
Quoted 17 lines · NeilBrown
Quoted 15 lines · Bart Van Assche
Quoted 11 lines · NeilBrown
Quoted 2 lines · Martin Steigerwald
And that Baloo needs an "invariant" for
a file. See comment #11 of that bug report:That is really hard to provide in general. Possibly the best approach is to use the
statfs()systemcall to get the "f_fsid" field. This is 64bits. It is not supported uniformly well by all filesystems, but I think it is at least not worse than using the device number. For a lot of older filesystems it is just an encoding of the device number.For btrfs, xfs, ext4 it is much much better.
How about combining the UUID of the partition with the file path? An example from one of the VMs on my workstation:
Quoted 7 lines
Quoted 4 lines
A btrfs filesystem can span multiple partitions, and those partitions can be added and removed dynamically. So you could migrated from one to another.
I suspect it was intended to be "filesytemm UUID". At least that is the field in lsblk output that was referenced.
Filesystem UUID is not enough.
In btrfs, all subvolumes share the same fsid.
While for statfs() call, we do extra XOR with subvolume id to generate unique f_fsid for each subvolume.
Thanks,
Qu
Quoted 6 lines
Quoted 5 lines
f_fsidreally is best for any modern filesystem.NeilBrown
NeilBrown - 26.06.21, 02:54:09 CEST:
Quoted 10 lines
Quoted 3 lines
And that Baloo needs an "invariant" for
a file. See comment #11 of that bug report:
That is really hard to provide in general. Possibly the best approach is to use the
statfs()systemcall to get the "f_fsid" field. This is 64bits. It is not supported uniformly well by all filesystems, but I think it is at least not worse than using the device number. For a lot of older filesystems it is just an encoding of the device number.For btrfs, xfs, ext4 it is much much better.
Thank you for the clear statement and for your alternative suggestion. I will forward this to Baloo upstream.
I think the main focus of Baloo would be to work on currently mostly in use Linux filesystem which should be BTRFS, XFS, EXT4 and probably F2FS.
Signature · 1 line
Martin