Skip to contentSkip to conversation

How to resize to an bigger then 16TB ext4 filesysteem on Debian

DISCUSSION Jelle de Jong 2 participants 4 messages Apr 10, 2014 – Apr 11, 2014

lore
Share & tools
lore
#0 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello everyone,

https://packages.debian.org/wheezy/e2fsprogs

# resize2fs /dev/lvm0-vol/storage
resize2fs 1.42.5 (29-Jul-2012)
resize2fs: New size too large to be expressed in 32 bits

(will the command work with e2fsprogs 1.42.9-3?)

# tune2fs -l /dev/lvm0-vol/storage
http://paste.debian.net/92889/

Suggestions has to be stable enough for production systems.

I now did a lvreduce -L 16TB /dev/lvm0-vol/storage and resize2fs /dev/lvm0-vol/storage (its running now)

# resize2fs /dev/lvm0-vol/storage
resize2fs 1.42.5 (29-Jul-2012)
Filesystem at /dev/lvm0-vol/storage is mounted on /srv/storage;
on-line resizing required
old_desc_blocks = 464, new_desc_blocks = 1024
Performing an on-line resize of /dev/lvm0-vol/storage to 4294967295
(4k) blocks.

Kind regards,

Jelle de Jong
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iJwEAQECAAYFAlNGgFAACgkQ1WclBW9j5HnDbQP/YRQvZq0HPJ4brpBas+95xeP3 lOnN0OzEOEpjC9pY7J9ppWfkvOkBx7ayjt8MqmJgluwVBNQIRUU3ctftaI7cUPTH abgj0JHLICCjSIs3nVzQo9pq8URnhTbDXDcXLBemglT5wa+Potlz4YYFA+9Cd/ty G2jfKBiaNCN17Nmikjo=
=bwbX
-----END PGP SIGNATURE-----

#1 lore raw
Quoted 9 lines · Jelle de Jong

Hello everyone,

https://packages.debian.org/wheezy/e2fsprogs

# resize2fs /dev/lvm0-vol/storage
resize2fs 1.42.5 (29-Jul-2012)
resize2fs: New size too large to be expressed in 32 bits
Quoted 6 lines

(will the command work with e2fsprogs 1.42.9-3?)

# tune2fs -l /dev/lvm0-vol/storage
http://paste.debian.net/92889/

Suggestions has to be stable enough for production systems.

I'm going to assume the file system was created without having the 64-bit file system feature set. (Running dumpe2fs -h /dev/lvm0-vol/storage can verify this).

If that's true, and you require a production-stable way of doing this, there's really only one thing I can suggest:

1) Backup your file system completely
2) Get the development version of e2fsprogs, and apply a set of in-development patches that haven't yet been merged
3) Use this in-development resize2fs to convert to a 64-bit file system, realizing that it will have completely rewrite your inode tables. Cross your fingers.
4) If it works, you're done. If not, you can restore from backups. :-)

So basically steps 2 and 3 are ways of optimizing needing to do a full restore of your file system.....

Cheers,

- Ted

#2 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 30 lines · Theodore Ts'o
Quoted 8 lines · Jelle de Jong

Hello everyone,

https://packages.debian.org/wheezy/e2fsprogs

# resize2fs /dev/lvm0-vol/storage resize2fs 1.42.5 (29-Jul-2012) 
resize2fs: New size too large to be expressed in 32 bits
Quoted 6 lines

(will the command work with e2fsprogs 1.42.9-3?)

# tune2fs -l /dev/lvm0-vol/storage 
http://paste.debian.net/92889/

Suggestions has to be stable enough for production systems.

I'm going to assume the file system was created without having the 64-bit file system feature set. (Running dumpe2fs -h /dev/lvm0-vol/storage can verify this).

If that's true, and you require a production-stable way of doing this, there's really only one thing I can suggest:

1) Backup your file system completely 2) Get the development version of e2fsprogs, and apply a set of in-development patches that haven't yet been merged 3) Use this in-development resize2fs to convert to a 64-bit file system, realizing that it will have completely rewrite your inode tables. Cross your fingers. 4) If it works, you're done. If not, you can restore from backups. :-)

Quoted 2 lines

So basically steps 2 and 3 are ways of optimizing needing to do a full restore of your file system.....

Thank you Ted for your quick reply.

In october 2012 I created the ext4 file system with the command:
mkfs.ext4 -m 1 /dev/lvm0-vol/storage

I assumed by using ext4 with lvm2 I would have had a solid basis for some future storage expeditions. Apparently I made a mistake somewhere but I don't know where yet or did I hit a bug?

# tune2fs -l /dev/lvm0-vol/storage
# dumpe2fs -h /dev/lvm0-vol/storage
http://paste.debian.net/plain/92953

Where can I see that my ext4 file system is limed to 16TB?

Your suggestion to use a development version of e2fsprogs doesn't really suit my needs for a stable solution that is maintainable.

1. I can temporary transfer all the data to an other system and recreate the ext4 file system, but will this be stable with version 1.42.5 and how much TB can the file-system handle?

2. I can put current the 16TB ext4 file system in production and wait for e2fsprogs support in Debian stable or backports to arrive and resize the to 22TB later. When do you expect this functionality to arrive in as stable release of e2fsprogs?

Thank you for your help.

Kind regards,

Jelle de Jong
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iJwEAQECAAYFAlNGs7IACgkQ1WclBW9j5Hl5NgQAjwMLRPu3KQuBesD7BSHkUrf3 JcaFMD90vvAYmbKMbMumGhR86BA1NXVBfeEA6DnrZUrPzly1VBglCOwCJzw52GCc TZHp3h5RD/Ce1c8/T+VD4DrVxhjIMUAyUT+/eGN008mHtrOfRppnFlcutNxZ5RZJ oGksAsJ5i7hWmeilqcc=
=ECQP
-----END PGP SIGNATURE-----

#3 lore raw
Quoted 8 lines · Jelle de Jong

In october 2012 I created the ext4 file system with the command:
mkfs.ext4 -m 1 /dev/lvm0-vol/storage

I assumed by using ext4 with lvm2 I would have had a solid basis for some future storage expeditions. Apparently I made a mistake somewhere but I don't know where yet or did I hit a bug?

Quoted 5 lines
# tune2fs -l /dev/lvm0-vol/storage
# dumpe2fs -h /dev/lvm0-vol/storage
http://paste.debian.net/plain/92953

Where can I see that my ext4 file system is limed to 16TB?

So here's the problem. The original ext2/3 block group descriptors only had room for 32-bit block numbers. So in order to resize the file system past 16TB, it was necessary to change the layout of the block group descriptors, and change them from needing 32 bytes to 64 bytes per block group.

So in order to make this change there was a new "64-bit" feature flag that changes the size of the block group descriptors. We didn't enable this feature at first, because we wanted to make sure the 64-bit support was well shaken down and "production ready" before it was enabled.

(In particular there were some online resizing bugs relating to 64-bit file systems that wasn't ironed out until relatively recently, so you should be glad we didn't enable this feature by default. :-)

Quoted 2 lines

Your suggestion to use a development version of e2fsprogs doesn't really suit my needs for a stable solution that is maintainable.

Well, unfortunately, there is no way to do what you request using the stable version of e2fsprogs. Sorry.

Quoted 3 lines

1. I can temporary transfer all the data to an other system and recreate the ext4 file system, but will this be stable with version 1.42.5 and how much TB can the file-system handle?

You should definitely go to the 1.42.9 version of e2fsprogs to make sure you have the latest fixes.

As far as the kernel is concerned, are you still using the Debian stable 3.2 kernel? The problem is that not all of the bug fixes get backported to a stable kernel which is that old. One of the value adds that a company like Red Hat provides is that they have someone paid to try to support ext4 on older kernels. But there have been some online resize fixes on the kernel side since 3.2, and I don't think all of them have been backported because they are fairly complex and involved patches.

If you're not willing to use the kernel from Debian testing, it may very well be that you will be better off waiting for Debian testing to ship before you even try to use ext4 file systems > that 16TB. I really can't guarantee that all of the necessary fixes have been backported to the 3.2 stable kernel.

Is there some way you could live with two large file systems instead of trying to create one very large file system?

Regards,

- Ted