Skip to contentSkip to conversation

"Too many links (31)" issue

DISCUSSION Andy Smith 3 participants 4 messages Feb 3, 2022 – Feb 4, 2022 Btrfs file system

lore
Share & tools
lore

Andy Smith

author
#0 lore raw

Hi,

I have a host with an xfs filesystem on it, with about 25 million files. It contains an rsnapshot backup that has been aggressively deduplicated by means of hardlinks and there's probably only about 7 million unique files on there.

I'm trying to rsync it to a different host into a btrfs filesystem, but part way through the rsync I get a "Too many links (31)" error:

rsync: [generator] link "/data/backup/rsnapshot/daily.0/chacha/var/lib/dpkg/info/.apt-utils.postrm.0" => daily.0/backup1/var/lib/dpkg/info/libpango1.0-0.postrm failed: Too many links (31) Hlink node data for 219191 already has path=daily.0/backup1/var/lib/dpkg/info/libpango1.0-0.postrm (daily.0/chacha/var/lib/dpkg/info/apt-utils.postrm)
rsync error: errors with program diagnostics (code 13) at hlink.c(539) [generator=3.2.3]

I searched around on this topic and found hits from 10 years ago about maximum hardlinks per directory and being dependent upon length of file path. Is that still relevant today?

This particular file does indeed have a huge number of hardlinks:

$ stat daily.0/chacha/var/lib/dpkg/info/apt-utils.postrm
  File: daily.0/chacha/var/lib/dpkg/info/apt-utils.postrm
  Size: 132             Blocks: 8          IO Block: 4096   regular file
Device: fd05h/64773d    Inode: 1342355538  Links: 7565
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-02-03 04:51:29.978915067 +0000
Modify: 2011-03-01 21:59:29.000000000 +0000
Change: 2022-01-01 17:35:06.598921506 +0000
 Birth: -
$ sudo find . -mount -samefile daily.0/chacha/var/lib/dpkg/info/apt-utils.postrm | wc -l
7565

I guess it is some sort of template file that is littered all over Debian systems.

Is there anything I can do to get this working?

The receiving host with the btrfs filesystem is Debian 11 (bullseye), stock Debian kernel 5.10.0-11-amd64. The btrfs filesystem is mounted with options:

/dev/mapper/backupenc on /data/backup type btrfs (rw,noatime,compress=zstd:15,space_cache,subvolid=5,subvol=/)

As an aside, when the file is as small as 132 bytes is there actually any advantage in hardlinking copies of it together rather than just having multiple copies of it? Is there some minimum file size where it's just not worth it?

(Yes I am aware of offline deuplication which works in XFS as well, it's just that where the files are known to be entirely identical I've found that simply hardlinking them together was faster and easier.)

Cheers,
Andy

#1 lore raw
Quoted 14 lines · On Thu, 3 Feb 2022 16:31:08 +0000 Andy Smith <[email protected]> wrote:

Hi,

I have a host with an xfs filesystem on it, with about 25 million files. It contains an rsnapshot backup that has been aggressively deduplicated by means of hardlinks and there's probably only about 7 million unique files on there.

I'm trying to rsync it to a different host into a btrfs filesystem, but part way through the rsync I get a "Too many links (31)" error:

rsync: [generator] link "/data/backup/rsnapshot/daily.0/chacha/var/lib/dpkg/info/.apt-utils.postrm.0" => daily.0/backup1/var/lib/dpkg/info/libpango1.0-0.postrm failed: Too many links (31) Hlink node data for 219191 already has path=daily.0/backup1/var/lib/dpkg/info/libpango1.0-0.postrm (daily.0/chacha/var/lib/dpkg/info/apt-utils.postrm)
rsync error: errors with program diagnostics (code 13) at hlink.c(539) [generator=3.2.3]

Quoted 7 lines

I searched around on this topic and found hits from 10 years ago about maximum hardlinks per directory and being dependent upon length of file path. Is that still relevant today?

[...]

Is there anything I can do to get this working?

Hello, Have you tried the "extended_iref" mount option?

author
#2 lore raw

Hi Lukas,

Quoted 11 lines · Lukas Straub
Quoted 7 lines · On Thu, 3 Feb 2022 16:31:08 +0000 Andy Smith <[email protected]> wrote:

I searched around on this topic and found hits from 10 years ago about maximum hardlinks per directory and being dependent upon length of file path. Is that still relevant today?

[...]

Is there anything I can do to get this working?

Hello, Have you tried the "extended_iref" mount option?

During my searching I did see mention of it, but as far as I could see this is a mkfs feature not a mount option, and it's default on now anyway. So I already have it don't I? (how to check enabled filesystem features?)

Anyway, it is not recognised as a mount option.

Cheers,
Andy

#3 lore raw
Quoted 11 lines · Andy Smith

Hi Lukas,

Quoted 8 lines · Lukas Straub
Quoted 6 lines · On Thu, 3 Feb 2022 16:31:08 +0000 Andy Smith <[email protected]> wrote:

I searched around on this topic and found hits from 10 years ago about maximum hardlinks per directory and being dependent upon length of file path. Is that still relevant today?

[...]

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Is there anything I can do to get this working?

Hello, Have you tried the "extended_iref" mount option?

During my searching I did see mention of it, but as far as I could see this is a mkfs feature not a mount option, and it's default on now anyway. So I already have it don't I? (how to check enabled filesystem features?)

Yes, it's a feature enabled by mkfs, and it has been enabled by default for many years now.

You can confirm if it's enabled like this:

$ cat /sys/fs/btrfs/80353f52-8425-45c2-a9a2-103c3e25ebb9/features/extended_iref 
1

Where the 8035(...) is the UUID of the mounted filesystem. You can get the UUID using blkid:

$ blkid
(...)
/dev/sdj: UUID="80353f52-8425-45c2-a9a2-103c3e25ebb9" UUID_SUB="0d1e7dfe-6ec8-494c-8467-bc4eabca56f2" BLOCK_SIZE="4096" TYPE="btrfs"
(...)

With extent irefs, the hard link limit is 65535 (64K - 1), I just confirmed with a quick test that it is indeed being respected:

$ mkfs.btrfs -f /dev/sdj
$ mount /dev/sdj /mnt/sdj
$ cd /mnt/sdj
$ touch foo
$ for ((i = 1; i < 65536; i++)); do echo "Iteration $i"; ln foo foo_$i; done
(...)
Iteration 65534
Iteration 65535
ln: failed to create hard link to 'foo': Too many links
$ stat foo
  File: foo
  Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: 2ch/44d	Inode: 257         Links: 65535
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-02-04 14:18:49.788113075 +0000
Modify: 2022-02-04 14:18:49.788113075 +0000
Change: 2022-02-04 14:21:58.432166650 +0000
 Birth: 2022-02-04 14:18:49.788113075 +0000
$ ls -1 | wc -l
65535
Quoted 5 lines

Anyway, it is not recognised as a mount option.

Cheers,
Andy