Skip to contentSkip to conversation

2 PB filesystem ok?

DISCUSSION Ulli Horlacher 5 participants 8 messages Apr 28, 2024 – Jun 28, 2025 Btrfs file system

lore
Share & tools
lore
#0 lore raw

We are planing to use a Seagate Exos Corvault System with 2 PB as the storage backend for restic backup. It has some hundreds hard disks and exports them as a single virtual disk (via SAS). Linux sees only a big SCSI disk. This special RAID is all in hardware, no need for BTRFS software RAID.

So far my biggest btrfs filesystems have only about 100 TB.

Is 2 PB ok for one btrfs filesystem?

Signature · 6 lines
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart         E-Mail: [email protected]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
REF:<[email protected]>
#1 lore raw

Il giorno lun 29 apr 2024 alle ore 01:40 Ulli Horlacher <[email protected]> ha scritto:

Quoted 1 line

Is 2 PB ok for one btrfs filesystem?

Well, it works (you can play by yourself with loopback just for fun with a test file!!!).

But, I dunno about RAM request if you are in trouble and need to run fsck or repair. Also about the RAM requested to have all extents mapped.
You could be in sad situation to not be able to mount it. Also, better if you can update the kernel whenever you want, and you don't have third party constraints.

In my experience (max 200TB volume), I use BTRFS for big volumes, but they need more extra care than the usual EXT4/XFS.

#2 lore raw
In reply to Ulli Horlacher
Quoted 7 lines · Ulli Horlacher

We are planing to use a Seagate Exos Corvault System with 2 PB as the storage backend for restic backup. It has some hundreds hard disks and exports them as a single virtual disk (via SAS). Linux sees only a big SCSI disk. This special RAID is all in hardware, no need for BTRFS software RAID.

Quoted 5 lines

So far my biggest btrfs filesystems have only about 100 TB.

Is 2 PB ok for one btrfs filesystem?

I have used 2PB filesystems in my RAID Stripe Tree test environment, but for practical uses, I suggest you to enable the block-group tree feature during mkfs time.

This stores block-group items in a separate tree instead of the extent tree, drastically improving mount times on huge FSes.

#3 lore raw
Quoted 3 lines · Johannes Thumshirn

I have used 2PB filesystems in my RAID Stripe Tree test environment, but for practical uses, I suggest you to enable the block-group tree feature during mkfs time.

I cannot find such an option in man-page for mkfs.btrfs

Signature · 6 lines
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart         E-Mail: [email protected]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
REF:<[email protected]>
#4 lore raw
Quoted 7 lines · Ulli Horlacher
Quoted 3 lines · Johannes Thumshirn

I have used 2PB filesystems in my RAID Stripe Tree test environment, but for practical uses, I suggest you to enable the block-group tree feature during mkfs time.

I cannot find such an option in man-page for mkfs.btrfs

    $ mkfs.btrfs  -O list-all
    Filesystem features available:
    mixed-bg            - mixed data and metadata block groups (compat=2.6.37, safe=2.6.37)
    quota               - quota support (qgroups) (compat=3.4)
    extref              - increased hardlink limit per file to 65536 (compat=3.7, safe=3.12, default=3.12)
    raid56              - raid56 extended format (compat=3.9)
    skinny-metadata     - reduced-size metadata extent refs (compat=3.10, safe=3.18, default=3.18)
    no-holes            - no explicit hole extents for files (compat=3.14, safe=4.0, default=5.15)
    free-space-tree     - free space tree (space_cache=v2) (compat=4.5, safe=4.9, default=5.15)
    raid1c34            - RAID1 with 3 or 4 copies (compat=5.5)
    zoned               - support zoned devices (compat=5.12)
    block-group-tree    - block group tree to reduce mount time (compat=6.1)

So I believe it would be `mkfs.btrfs -O block-group-tree ...'.

Quoted 10 lines

--
Ullrich Horlacher Server und Virtualisierung
Rechenzentrum TIK

Universitaet Stuttgart         E-Mail: [email protected]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/

REF:<[email protected]>

--
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

#5 lore raw
Quoted 23 lines · Tomas Volf
Quoted 7 lines · Ulli Horlacher
Quoted 3 lines · Johannes Thumshirn

I have used 2PB filesystems in my RAID Stripe Tree test environment, but for practical uses, I suggest you to enable the block-group tree feature during mkfs time.

I cannot find such an option in man-page for mkfs.btrfs

    $ mkfs.btrfs  -O list-all
    Filesystem features available:
    mixed-bg            - mixed data and metadata block groups (compat=2.6.37, safe=2.6.37)
    quota               - quota support (qgroups) (compat=3.4)
    extref              - increased hardlink limit per file to 65536 (compat=3.7, safe=3.12, default=3.12)
    raid56              - raid56 extended format (compat=3.9)
    skinny-metadata     - reduced-size metadata extent refs (compat=3.10, safe=3.18, default=3.18)
    no-holes            - no explicit hole extents for files (compat=3.14, safe=4.0, default=5.15)
    free-space-tree     - free space tree (space_cache=v2) (compat=4.5, safe=4.9, default=5.15)
    raid1c34            - RAID1 with 3 or 4 copies (compat=5.5)
    zoned               - support zoned devices (compat=5.12)
    block-group-tree    - block group tree to reduce mount time (compat=6.1)
Quoted 1 line

So I believe it would be `mkfs.btrfs -O block-group-tree ...'.

root@fex:~# mkfs.btrfs  -O list-all
Filesystem features available:
mixed-bg            - mixed data and metadata block groups (0x4, compat=2.6.37, safe=2.6.37)
extref              - increased hardlink limit per file to 65536 (0x40, compat=3.7, safe=3.12, default=3.12)
raid56              - raid56 extended format (0x80, compat=3.9)
skinny-metadata     - reduced-size metadata extent refs (0x100, compat=3.10, safe=3.18, default=3.18)
no-holes            - no explicit hole extents for files (0x200, compat=3.14, safe=4.0, default=5.15)
raid1c34            - RAID1 with 3 or 4 copies (0x800, compat=5.5)
zoned               - support zoned devices (0x1000, compat=5.12)
root@fex:~# btrfs --version
btrfs-progs v5.16.2

This is Ubuntu 22.04

Signature · 6 lines
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart         E-Mail: [email protected]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
REF:<ZjJOuiu0o1PqV3jA@ws>
#6 lore raw

Ulli Horlacher - 01.05.24, 20:19:03 CEST:

Quoted 10 lines · Tomas Volf
Quoted 9 lines · Ulli Horlacher
Quoted 7 lines · Johannes Thumshirn
Quoted 3 lines

I have used 2PB filesystems in my RAID Stripe Tree test environment, but for practical uses, I suggest you to enable the block-group tree feature during mkfs time.

I cannot find such an option in man-page for mkfs.btrfs

    $ mkfs.btrfs  -O list-all

[…]

Quoted 3 lines
Quoted 1 line

So I believe it would be `mkfs.btrfs -O block-group-tree ...'.

root@fex:~# mkfs.btrfs  -O list-all

[…]

Quoted 5 lines
root@fex:~# btrfs --version
btrfs-progs v5.16.2

This is Ubuntu 22.04

Too old.

See here for minimum requirements:

https://btrfs.readthedocs.io/en/latest/Feature-by-version.html

Kernel (and I bet) btrfs progs 6.1.

I bet there is a PPA for newer kernels for Ubuntu. Whether there is a PPA for never btrfs progs, I don't know.

Best,

Signature · 1 line
Martin
#7 lore raw
Quoted 11 lines · Martin Steigerwald
Quoted 3 lines
Quoted 1 line

So I believe it would be `mkfs.btrfs -O block-group-tree ...'.

root@fex:~# mkfs.btrfs  -O list-all

[...]

Quoted 5 lines
root@fex:~# btrfs --version
btrfs-progs v5.16.2

This is Ubuntu 22.04

Too old.

root@zentux:~# mkfs.btrfs  -O list-all 2>&1 | grep block-group-tree
bgt                 - block-group-tree alias
block-group-tree    - block group tree to reduce mount time (compat=6.1)
root@zentux:~# sysinfo
System:        Linux zentux 6.8.0-62-generic x86_64
Distribution:  Ubuntu 24.04.2 LTS /sw
root@zentux:~# btrfs --version
btrfs-progs v6.6.3

Got it! :-)

Signature · 6 lines
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart         E-Mail: [email protected]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
REF:<[email protected]>