experience with very large filesystems
Hi everyone,
I have a large filesystem (1.2 PiB) that's been working flawlessly for 4 years (an LV aggregating 4 300~ish TB RAID-6 arrays). However it's getting really full (more than 1.1 PiB full right now), and I'm considering expanding it to 1.8 PiB or more.
Any thoughts, caveats or else?
Signature · 6 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------Quoted 8 lines · Emmanuel Florac
Hi everyone,
I have a large filesystem (1.2 PiB) that's been working flawlessly for 4 years (an LV aggregating 4 300~ish TB RAID-6 arrays). However it's getting really full (more than 1.1 PiB full right now), and I'm considering expanding it to 1.8 PiB or more.
Any thoughts, caveats or else?
From the filesystem side, that should not be an issue at all.
From the storage side, you really want to expand the storage with chunks that have the same geometry (stripe unit and stripe width) so that it doesn't screw up the alignment of the filesystem to the new storage.
And that's where the difficultly may lie. If the existing storage volume the filesystem sits on doesn't end exactly on a stripe width boundary, you're going to have to offset the start of the new storage volumes part way into the first stripe width in the volumes to ensure that when the filesystem expands, then end of the first stripe width in the new volume is exactly where the filesystem expects it to be.
Other than that, there shouldn't be any filesystem level concerns about doubling the size of the filesystem capacity via growfs.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Le Thu, 24 Feb 2022 08:10:06 +1100
Dave Chinner <[email protected]> écrivait:
Quoted 4 lines
From the storage side, you really want to expand the storage with chunks that have the same geometry (stripe unit and stripe width) so that it doesn't screw up the alignment of the filesystem to the new storage.
That part should be easy, I plan to use the same type of hardware anyway (12 TB Ultrastar disks).
Quoted 7 lines
And that's where the difficultly may lie. If the existing storage volume the filesystem sits on doesn't end exactly on a stripe width boundary, you're going to have to offset the start of the new storage volumes part way into the first stripe width in the volumes to ensure that when the filesystem expands, then end of the first stripe width in the new volume is exactly where the filesystem expects it to be.
That should be somewhat more manageable given that I've setup everything as an LVM volume. I may use partitions on the new devices to precisely align the PVs before expanding the LV.
Signature · 6 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------Quoted 19 lines · Emmanuel Florac
Le Thu, 24 Feb 2022 08:10:06 +1100
Dave Chinner <[email protected]> écrivait:Quoted 4 lines
From the storage side, you really want to expand the storage with chunks that have the same geometry (stripe unit and stripe width) so that it doesn't screw up the alignment of the filesystem to the new storage.
That part should be easy, I plan to use the same type of hardware anyway (12 TB Ultrastar disks).
Quoted 7 lines
And that's where the difficultly may lie. If the existing storage volume the filesystem sits on doesn't end exactly on a stripe width boundary, you're going to have to offset the start of the new storage volumes part way into the first stripe width in the volumes to ensure that when the filesystem expands, then end of the first stripe width in the new volume is exactly where the filesystem expects it to be.
Quoted 3 lines
That should be somewhat more manageable given that I've setup everything as an LVM volume. I may use partitions on the new devices to precisely align the PVs before expanding the LV.
You can use the PV/LV header size options to align the contents of the LV.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]