btrfs loses 32-bit application compatibility after a while
As far as I can tell, btrfs assigns inode numbers sequentially using this function:
int btrfs_get_free_objectid(struct btrfs_root *root, u64 *objectid)
{
int ret;
mutex_lock(&root->objectid_mutex);
if (unlikely(root->free_objectid >= BTRFS_LAST_FREE_OBJECTID)) {
btrfs_warn(root->fs_info,
"the objectid of root %llu reaches its highest value",
root->root_key.objectid);
ret = -ENOSPC;
goto out;
}
*objectid = root->free_objectid++;
ret = 0;
out:
mutex_unlock(&root->objectid_mutex);
return ret;
}Even after deletion of the object, inode numbers are never reused.
This is a problem because after creating and deleting two billion files, the 31-bit inode number space is exhausted. (Not sure if negative inode numbers are a thing, so there could be one extra bit.) From that point onwards, future files will end up with an inode number that cannot be represented with the non-LFS interfaces (stat, getdents, etc.), causing system calls to fail with EOVERFLOW.
For ABI reasons, we can't switch everything to 64-bit ino_t and LFS interfaces. (If we could recompile, we wouldn't still be using 32-bit software.)
So far, we have seen this on our 32-bit Koji builders, which create and delete many, many files. But I suspect end users will encounter it eventually, too.
It seems to me that the on-disk format already allows range searches on inode numbers (see btrfs_init_root_free_objectid), so maybe this could be used to find a sufficiently large unused range to allocate from.
Thanks,
Florian
Quoted 23 lines · Florian Weimer
As far as I can tell, btrfs assigns inode numbers sequentially using this function:
int
btrfs_get_free_objectid(struct btrfs_root *root, u64 *objectid){ int ret; mutex_lock(&root->objectid_mutex); if (unlikely(root->free_objectid >= BTRFS_LAST_FREE_OBJECTID)) { btrfs_warn(root->fs_info, "the objectid of root %llu reaches its highest value", root->root_key.objectid); ret = -ENOSPC; goto out; } *objectid = root->free_objectid++; ret = 0; out: mutex_unlock(&root->objectid_mutex); return ret; }
Quoted 9 lines
Even after deletion of the object, inode numbers are never reused.
This is a problem because after creating and deleting two billion files, the 31-bit inode number space is exhausted. (Not sure if negative inode numbers are a thing, so there could be one extra bit.) From that point onwards, future files will end up with an inode number that cannot be represented with the non-LFS interfaces (stat, getdents, etc.), causing system calls to fail with EOVERFLOW.
It is a know btrfs issue. On 32 bit some workloads may cause the exhaustion of the inode number.
There was the inode_cache feature that it was supposed to permit to reuse of the inode. It was removed, below some more details
https://patchwork.kernel.org/project/linux-btrfs/patch/[email protected]/
It was deprecated in 5.9 and removed in 5.11 (see commit f5297199a8bca12b8b96afcbf2341605efb6005de)
Despite the technical details, I am curious about how many 32 bit systems still uses btrfs.
Quoted 15 lines
For ABI reasons, we can't switch everything to 64-bit
ino_tand LFS interfaces. (If we could recompile, we wouldn't still be using 32-bit software.)So far, we have seen this on our 32-bit Koji builders, which create and delete many, many files. But I suspect end users will encounter it eventually, too.
It seems to me that the on-disk format already allows range searches on inode numbers (see
btrfs_init_root_free_objectid), so maybe this could be used to find a sufficiently large unused range to allocate from.Thanks,
Florian
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
Quoted 40 lines · Goffredo Baroncelli
Quoted 23 lines · Florian Weimer
As far as I can tell, btrfs assigns inode numbers sequentially using this function:
int
btrfs_get_free_objectid(struct btrfs_root *root, u64 *objectid){ int ret; mutex_lock(&root->objectid_mutex); if (unlikely(root->free_objectid >= BTRFS_LAST_FREE_OBJECTID)) { btrfs_warn(root->fs_info, "the objectid of root %llu reaches its highest value", root->root_key.objectid); ret = -ENOSPC; goto out; } *objectid = root->free_objectid++; ret = 0; out: mutex_unlock(&root->objectid_mutex); return ret; }Quoted 9 lines
Even after deletion of the object, inode numbers are never reused.
This is a problem because after creating and deleting two billion files, the 31-bit inode number space is exhausted. (Not sure if negative inode numbers are a thing, so there could be one extra bit.) From that point onwards, future files will end up with an inode number that cannot be represented with the non-LFS interfaces (stat, getdents, etc.), causing system calls to fail with EOVERFLOW.
It is a know btrfs issue. On 32 bit some workloads may cause the exhaustion of the inode number.
There was the
inode_cachefeature that it was supposed to permit to reuse of the inode. It was removed, below some more details
Quoted 8 lines
https://patchwork.kernel.org/project/linux-btrfs/patch/[email protected]/
It was deprecated in 5.9 and removed in 5.11 (see commit
f5297199a8bca12b8b96afcbf2341605efb6005de)Despite the technical details, I am curious about how many 32 bit systems still uses btrfs.
It was somewhat common for Fedora users. A number of Fedora 32-bit ARM variants used Btrfs until 32-bit ARM was discontinued in Fedora 37[1]. openSUSE still has 32-bit ARM and x86 support in Tumbleweed.
This issue is also possible on 64-bit x86 systems where 32-bit x86 applications run on it. That's what this report is about. We're hitting it in Fedora because our 32-bit x86 builds in Fedora infrastructure run on 64-bit x86 environments and triggered this[2].
[1]: https://fedoraproject.org/wiki/Changes/RetireARMv7
[2]: https://pagure.io/releng/issue/11531
Signature · 1 line
真実はいつも一つ!/ Always, there's only one truth!
Quoted 2 lines · Neal Gompa
Quoted 1 line · Goffredo Baroncelli
[..]
Quoted 10 lines
It was somewhat common for Fedora users. A number of Fedora 32-bit ARM variants used Btrfs until 32-bit ARM was discontinued in Fedora 37[1]. openSUSE still has 32-bit ARM and x86 support in Tumbleweed.
This issue is also possible on 64-bit x86 systems where 32-bit x86 applications run on it. That's what this report is about. We're hitting it in Fedora because our 32-bit x86 builds in Fedora infrastructure run on 64-bit x86 environments and triggered this[2].
From what you wrote, it is seems more "it is technically supported" but not big users. Otherwise I expected that a lot of bugs or complaints happened when it was deprecated from 5.9 and removed in 5.11.
Despite that, I am curious about what could happen when a 32 bit application tries to access a 64 bit inode: does the kernel return only the lower part of the inode number ? How this is handled in other FS: what happens when an fs hosts more than 2^32 files ? Unlikely but this may happen. BTRFS makes this more easy to happen.
Quoted 4 lines
[1]: https://fedoraproject.org/wiki/Changes/RetireARMv7
[2]: https://pagure.io/releng/issue/11531
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
* Goffredo Baroncelli:
Quoted 14 lines · Neal Gompa
Quoted 2 lines · Goffredo Baroncelli
Quoted 1 line
[..]
Quoted 9 lines
It was somewhat common for Fedora users. A number of Fedora 32-bit ARM
variants used Btrfs until 32-bit ARM was discontinued in Fedora 37[1]. openSUSE still has 32-bit ARM and x86 support in Tumbleweed. This issue is also possible on 64-bit x86 systems where 32-bit x86 applications run on it. That's what this report is about. We're hitting it in Fedora because our 32-bit x86 builds in Fedora infrastructure run on 64-bit x86 environments and triggered this[2].
Quoted 3 lines
From what you wrote, it is seems more "it is technically supported" but not big users. Otherwise I expected that a lot of bugs or complaints happened when it was deprecated from 5.9 and removed in 5.11.
I think that for most users, it will take some time (years?) until they hit this issue. The builders are a bit of a special case.
This is something where telemetry really could help.
Quoted 5 lines
Despite that, I am curious about what could happen when a 32 bit application tries to access a 64 bit inode: does the kernel return only the lower part of the inode number ? How this is handled in other FS: what happens when an fs hosts more than 2^32 files ? Unlikely but this may happen. BTRFS makes this more easy to happen.
The expectation is that the kernel interfaces return EOVERFLOW, the same error code that is used for file sizes that cannot be represented using the 32-bit interfaces.
Thanks,
Florian