"Too many links (31)" issue
Hi,
I have a host with an xfs filesystem on it, with about 25 million files. It contains an rsnapshot backup that has been aggressively deduplicated by means of hardlinks and there's probably only about 7 million unique files on there.
I'm trying to rsync it to a different host into a btrfs filesystem, but part way through the rsync I get a "Too many links (31)" error:
rsync: [generator] link "/data/backup/rsnapshot/daily.0/chacha/var/lib/dpkg/info/.apt-utils.postrm.0" => daily.0/backup1/var/lib/dpkg/info/libpango1.0-0.postrm failed: Too many links (31) Hlink node data for 219191 already has path=daily.0/backup1/var/lib/dpkg/info/libpango1.0-0.postrm (daily.0/chacha/var/lib/dpkg/info/apt-utils.postrm)
rsync error: errors with program diagnostics (code 13) at hlink.c(539) [generator=3.2.3]
I searched around on this topic and found hits from 10 years ago about maximum hardlinks per directory and being dependent upon length of file path. Is that still relevant today?
This particular file does indeed have a huge number of hardlinks:
$ stat daily.0/chacha/var/lib/dpkg/info/apt-utils.postrm File: daily.0/chacha/var/lib/dpkg/info/apt-utils.postrm Size: 132 Blocks: 8 IO Block: 4096 regular file Device: fd05h/64773d Inode: 1342355538 Links: 7565 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2022-02-03 04:51:29.978915067 +0000 Modify: 2011-03-01 21:59:29.000000000 +0000 Change: 2022-01-01 17:35:06.598921506 +0000 Birth: - $ sudo find . -mount -samefile daily.0/chacha/var/lib/dpkg/info/apt-utils.postrm | wc -l 7565
I guess it is some sort of template file that is littered all over Debian systems.
Is there anything I can do to get this working?
The receiving host with the btrfs filesystem is Debian 11 (bullseye), stock Debian kernel 5.10.0-11-amd64. The btrfs filesystem is mounted with options:
/dev/mapper/backupenc on /data/backup type btrfs (rw,noatime,compress=zstd:15,space_cache,subvolid=5,subvol=/)
As an aside, when the file is as small as 132 bytes is there actually any advantage in hardlinking copies of it together rather than just having multiple copies of it? Is there some minimum file size where it's just not worth it?
(Yes I am aware of offline deuplication which works in XFS as well, it's just that where the files are known to be entirely identical I've found that simply hardlinking them together was faster and easier.)
Cheers,
Andy
Quoted 14 lines · On Thu, 3 Feb 2022 16:31:08 +0000 Andy Smith <[email protected]> wrote:
Hi,
I have a host with an xfs filesystem on it, with about 25 million files. It contains an rsnapshot backup that has been aggressively deduplicated by means of hardlinks and there's probably only about 7 million unique files on there.
I'm trying to rsync it to a different host into a btrfs filesystem, but part way through the rsync I get a "Too many links (31)" error:
rsync: [generator] link "/data/backup/rsnapshot/daily.0/chacha/var/lib/dpkg/info/.apt-utils.postrm.0" => daily.0/backup1/var/lib/dpkg/info/libpango1.0-0.postrm failed: Too many links (31) Hlink node data for 219191 already has path=daily.0/backup1/var/lib/dpkg/info/libpango1.0-0.postrm (daily.0/chacha/var/lib/dpkg/info/apt-utils.postrm)
rsync error: errors with program diagnostics (code 13) athlink.c(539)[generator=3.2.3]
Quoted 7 lines
I searched around on this topic and found hits from 10 years ago about maximum hardlinks per directory and being dependent upon length of file path. Is that still relevant today?
[...]
Is there anything I can do to get this working?
Hello, Have you tried the "extended_iref" mount option?
Hi Lukas,
Quoted 11 lines · Lukas Straub
Quoted 7 lines · On Thu, 3 Feb 2022 16:31:08 +0000 Andy Smith <[email protected]> wrote:
I searched around on this topic and found hits from 10 years ago about maximum hardlinks per directory and being dependent upon length of file path. Is that still relevant today?
[...]
Is there anything I can do to get this working?
Hello, Have you tried the "
extended_iref" mount option?
During my searching I did see mention of it, but as far as I could see this is a mkfs feature not a mount option, and it's default on now anyway. So I already have it don't I? (how to check enabled filesystem features?)
Anyway, it is not recognised as a mount option.
Cheers,
Andy
Quoted 11 lines · Andy Smith
Hi Lukas,
Quoted 8 lines · Lukas Straub
Quoted 6 lines · On Thu, 3 Feb 2022 16:31:08 +0000 Andy Smith <[email protected]> wrote:
I searched around on this topic and found hits from 10 years ago about maximum hardlinks per directory and being dependent upon length of file path. Is that still relevant today?
[...]
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Is there anything I can do to get this working?
Hello, Have you tried the "
extended_iref" mount option?During my searching I did see mention of it, but as far as I could see this is a mkfs feature not a mount option, and it's default on now anyway. So I already have it don't I? (how to check enabled filesystem features?)
Yes, it's a feature enabled by mkfs, and it has been enabled by default for many years now.
You can confirm if it's enabled like this:
$ cat /sys/fs/btrfs/80353f52-8425-45c2-a9a2-103c3e25ebb9/features/extended_iref 1
Where the 8035(...) is the UUID of the mounted filesystem. You can get the UUID using blkid:
$ blkid (...) /dev/sdj: UUID="80353f52-8425-45c2-a9a2-103c3e25ebb9" UUID_SUB="0d1e7dfe-6ec8-494c-8467-bc4eabca56f2" BLOCK_SIZE="4096" TYPE="btrfs" (...)
With extent irefs, the hard link limit is 65535 (64K - 1), I just confirmed with a quick test that it is indeed being respected:
$ mkfs.btrfs -f /dev/sdj $ mount /dev/sdj /mnt/sdj $ cd /mnt/sdj $ touch foo $ for ((i = 1; i < 65536; i++)); do echo "Iteration $i"; ln foo foo_$i; done (...) Iteration 65534 Iteration 65535 ln: failed to create hard link to 'foo': Too many links
$ stat foo File: foo Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 2ch/44d Inode: 257 Links: 65535 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2022-02-04 14:18:49.788113075 +0000 Modify: 2022-02-04 14:18:49.788113075 +0000 Change: 2022-02-04 14:21:58.432166650 +0000 Birth: 2022-02-04 14:18:49.788113075 +0000
$ ls -1 | wc -l 65535
Quoted 5 lines
Anyway, it is not recognised as a mount option.
Cheers,
Andy