2 PB filesystem ok?
We are planing to use a Seagate Exos Corvault System with 2 PB as the storage backend for restic backup. It has some hundreds hard disks and exports them as a single virtual disk (via SAS). Linux sees only a big SCSI disk. This special RAID is all in hardware, no need for BTRFS software RAID.
So far my biggest btrfs filesystems have only about 100 TB.
Is 2 PB ok for one btrfs filesystem?
Signature · 6 lines
Ullrich Horlacher Server und Virtualisierung Rechenzentrum TIK Universitaet Stuttgart E-Mail: [email protected] Allmandring 30a Tel: ++49-711-68565868 70569 Stuttgart (Germany) WWW: https://www.tik.uni-stuttgart.de/ REF:<[email protected]>
Il giorno lun 29 apr 2024 alle ore 01:40 Ulli Horlacher <[email protected]> ha scritto:
Quoted 1 line
Is 2 PB ok for one btrfs filesystem?
Well, it works (you can play by yourself with loopback just for fun with a test file!!!).
But, I dunno about RAM request if you are in trouble and need to run fsck or repair. Also about the RAM requested to have all extents mapped.
You could be in sad situation to not be able to mount it. Also, better if you can update the kernel whenever you want, and you don't have third party constraints.
In my experience (max 200TB volume), I use BTRFS for big volumes, but they need more extra care than the usual EXT4/XFS.
Quoted 7 lines · Ulli Horlacher
We are planing to use a Seagate Exos Corvault System with 2 PB as the storage backend for restic backup. It has some hundreds hard disks and exports them as a single virtual disk (via SAS). Linux sees only a big SCSI disk. This special RAID is all in hardware, no need for BTRFS software RAID.
Quoted 5 lines
So far my biggest btrfs filesystems have only about 100 TB.
Is 2 PB ok for one btrfs filesystem?
I have used 2PB filesystems in my RAID Stripe Tree test environment, but for practical uses, I suggest you to enable the block-group tree feature during mkfs time.
This stores block-group items in a separate tree instead of the extent tree, drastically improving mount times on huge FSes.
Quoted 3 lines · Johannes Thumshirn
I have used 2PB filesystems in my RAID Stripe Tree test environment, but for practical uses, I suggest you to enable the block-group tree feature during mkfs time.
I cannot find such an option in man-page for mkfs.btrfs
Signature · 6 lines
Ullrich Horlacher Server und Virtualisierung Rechenzentrum TIK Universitaet Stuttgart E-Mail: [email protected] Allmandring 30a Tel: ++49-711-68565868 70569 Stuttgart (Germany) WWW: https://www.tik.uni-stuttgart.de/ REF:<[email protected]>
Quoted 7 lines · Ulli Horlacher
Quoted 3 lines · Johannes Thumshirn
I have used 2PB filesystems in my RAID Stripe Tree test environment, but for practical uses, I suggest you to enable the block-group tree feature during mkfs time.
I cannot find such an option in man-page for mkfs.btrfs
$ mkfs.btrfs -O list-all
Filesystem features available:
mixed-bg - mixed data and metadata block groups (compat=2.6.37, safe=2.6.37)
quota - quota support (qgroups) (compat=3.4)
extref - increased hardlink limit per file to 65536 (compat=3.7, safe=3.12, default=3.12)
raid56 - raid56 extended format (compat=3.9)
skinny-metadata - reduced-size metadata extent refs (compat=3.10, safe=3.18, default=3.18)
no-holes - no explicit hole extents for files (compat=3.14, safe=4.0, default=5.15)
free-space-tree - free space tree (space_cache=v2) (compat=4.5, safe=4.9, default=5.15)
raid1c34 - RAID1 with 3 or 4 copies (compat=5.5)
zoned - support zoned devices (compat=5.12)
block-group-tree - block group tree to reduce mount time (compat=6.1)So I believe it would be `mkfs.btrfs -O block-group-tree ...'.
Quoted 10 lines
--
Ullrich Horlacher Server und Virtualisierung
Rechenzentrum TIKUniversitaet Stuttgart E-Mail: [email protected] Allmandring 30a Tel: ++49-711-68565868 70569 Stuttgart (Germany) WWW: https://www.tik.uni-stuttgart.de/REF:<[email protected]>
--
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.
Quoted 23 lines · Tomas Volf
Quoted 7 lines · Ulli Horlacher
Quoted 3 lines · Johannes Thumshirn
I have used 2PB filesystems in my RAID Stripe Tree test environment, but for practical uses, I suggest you to enable the block-group tree feature during mkfs time.
I cannot find such an option in man-page for mkfs.btrfs
$ mkfs.btrfs -O list-all Filesystem features available: mixed-bg - mixed data and metadata block groups (compat=2.6.37, safe=2.6.37) quota - quota support (qgroups) (compat=3.4) extref - increased hardlink limit per file to 65536 (compat=3.7, safe=3.12, default=3.12) raid56 - raid56 extended format (compat=3.9) skinny-metadata - reduced-size metadata extent refs (compat=3.10, safe=3.18, default=3.18) no-holes - no explicit hole extents for files (compat=3.14, safe=4.0, default=5.15) free-space-tree - free space tree (space_cache=v2) (compat=4.5, safe=4.9, default=5.15) raid1c34 - RAID1 with 3 or 4 copies (compat=5.5) zoned - support zoned devices (compat=5.12) block-group-tree - block group tree to reduce mount time (compat=6.1)
Quoted 1 line
So I believe it would be `mkfs.btrfs -O block-group-tree ...'.
root@fex:~# mkfs.btrfs -O list-all Filesystem features available: mixed-bg - mixed data and metadata block groups (0x4, compat=2.6.37, safe=2.6.37) extref - increased hardlink limit per file to 65536 (0x40, compat=3.7, safe=3.12, default=3.12) raid56 - raid56 extended format (0x80, compat=3.9) skinny-metadata - reduced-size metadata extent refs (0x100, compat=3.10, safe=3.18, default=3.18) no-holes - no explicit hole extents for files (0x200, compat=3.14, safe=4.0, default=5.15) raid1c34 - RAID1 with 3 or 4 copies (0x800, compat=5.5) zoned - support zoned devices (0x1000, compat=5.12)
root@fex:~# btrfs --version btrfs-progs v5.16.2
This is Ubuntu 22.04
Signature · 6 lines
Ullrich Horlacher Server und Virtualisierung Rechenzentrum TIK Universitaet Stuttgart E-Mail: [email protected] Allmandring 30a Tel: ++49-711-68565868 70569 Stuttgart (Germany) WWW: https://www.tik.uni-stuttgart.de/ REF:<ZjJOuiu0o1PqV3jA@ws>
Ulli Horlacher - 01.05.24, 20:19:03 CEST:
Quoted 10 lines · Tomas Volf
Quoted 9 lines · Ulli Horlacher
Quoted 7 lines · Johannes Thumshirn
Quoted 3 lines
I have used 2PB filesystems in my RAID Stripe Tree test environment, but for practical uses, I suggest you to enable the block-group tree feature during mkfs time.
I cannot find such an option in man-page for mkfs.btrfs
$ mkfs.btrfs -O list-all
[…]
Quoted 3 lines
Quoted 1 line
So I believe it would be `mkfs.btrfs -O block-group-tree ...'.
root@fex:~# mkfs.btrfs -O list-all
[…]
Quoted 5 lines
root@fex:~# btrfs --version btrfs-progs v5.16.2This is Ubuntu 22.04
Too old.
See here for minimum requirements:
https://btrfs.readthedocs.io/en/latest/Feature-by-version.html
Kernel (and I bet) btrfs progs 6.1.
I bet there is a PPA for newer kernels for Ubuntu. Whether there is a PPA for never btrfs progs, I don't know.
Best,
Signature · 1 line
Martin
Quoted 11 lines · Martin Steigerwald
Quoted 3 lines
Quoted 1 line
So I believe it would be `mkfs.btrfs -O block-group-tree ...'.
root@fex:~# mkfs.btrfs -O list-all[...]
Quoted 5 lines
root@fex:~# btrfs --version btrfs-progs v5.16.2This is Ubuntu 22.04
Too old.
root@zentux:~# mkfs.btrfs -O list-all 2>&1 | grep block-group-tree bgt - block-group-tree alias block-group-tree - block group tree to reduce mount time (compat=6.1)
root@zentux:~# sysinfo System: Linux zentux 6.8.0-62-generic x86_64 Distribution: Ubuntu 24.04.2 LTS /sw
root@zentux:~# btrfs --version btrfs-progs v6.6.3
Got it! :-)
Signature · 6 lines
Ullrich Horlacher Server und Virtualisierung Rechenzentrum TIK Universitaet Stuttgart E-Mail: [email protected] Allmandring 30a Tel: ++49-711-68565868 70569 Stuttgart (Germany) WWW: https://www.tik.uni-stuttgart.de/ REF:<[email protected]>