How to disable/revoke 'compression'?
Hi,
I used the mount option 'compression' on some mounted sub volumes. How can I revoke the compression? Means to delete the option and get all data uncompressed on this volume.
Is it enough to remount the sub volume without this option? Or is it necessary to do some addional step (balancing?) to get all stored data uncompressed. Beside of it, is it possible to find out what the real and compressed size of a file, for example or the ratio? Bye
Frank
Quoted 7 lines · Cloud Admin
Hi,
I used the mount option 'compression' on some mounted sub volumes. How can I revoke the compression? Means to delete the option and get all data uncompressed on this volume.
Is it enough to remount the sub volume without this option? Or is it necessary to do some addional step (balancing?) to get all stored data uncompressed.
If you set it via mount option, removing the option is enough to disable compression for _new_ files. Other ways are chattr +c and btrfs-property, but if you haven't heard about those you almost surely don't have such attributes set.
After remounting, you may uncompress existing files. Balancing won't do this as it moves extents around without looking inside; defrag on the other hand rewrites extents thus as a side effect it applies new [non]compression settings. Thus: ｢btrfs fi defrag -r /path/to/filesystem｣.
Quoted 2 lines
Beside of it, is it possible to find out what the real and compressed size of a file, for example or the ratio?
Currently not.
I've once written a tool which does this, but 1. it's extremely slow, 2. insane, 3. so insane a certain member of this list would kill me had I distributed the tool. Thus, I'd need to rewrite it first...
Meow!
Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢰⠒⠀⣿⡁ Vat kind uf sufficiently advanced technology iz dis!? ⢿⡄⠘⠷⠚⠋⠀ -- Genghis Ht'rok'din ⠈⠳⣄⠀⠀⠀⠀
Quoted 19 lines · Adam Borowski
Quoted 7 lines · Cloud Admin
Hi,
I used the mount option 'compression' on some mounted sub volumes. How can I revoke the compression? Means to delete the option and get all data uncompressed on this volume.
Is it enough to remount the sub volume without this option? Or is it necessary to do some addional step (balancing?) to get all stored data uncompressed.If you set it via mount option, removing the option is enough to disable compression for _new_ files. Other ways are chattr +c and btrfs-property, but if you haven't heard about those you almost surely don't have such attributes set.
After remounting, you may uncompress existing files. Balancing won't do this as it moves extents around without looking inside; defrag on the other hand rewrites extents thus as a side effect it applies new [non]compression settings. Thus: ｢btrfs fi defrag -r /path/to/filesystem｣.
Quoted 8 lines
Quoted 2 lines
Beside of it, is it possible to find out what the real and compressed size of a file, for example or the ratio?
Currently not.
I've once written a tool which does this, but 1. it's extremely slow, 2. insane, 3. so insane a certain member of this list would kill me had I distributed the tool. Thus, I'd need to rewrite it first...
Heh, I wouldn't do that, since I need you to do my debian uploads. :D
But it would certainly help to be a bit less stubborn only wanting to code in the language that matches your country code. :O
Or maybe I can help a bit, since it sounds like a nice one for the coding examples in the lib. ;] Days are getting shorter again, so the amount of indoor coding activity will hopefully increase a bit again soon.
Signature · 1 line
Hans van Kranenburg
Quoted 7 lines · Hans van Kranenburg
Quoted 6 lines · Adam Borowski
Quoted 2 lines · Cloud Admin
Beside of it, is it possible to find out what the real and compressed size of a file, for example or the ratio?
Currently not.
Quoted 8 lines
Quoted 3 lines
I've once written a tool which does this, but 1. it's extremely slow, 2. insane, 3. so insane a certain member of this list would kill me had I distributed the tool. Thus, I'd need to rewrite it first...
Heh, I wouldn't do that, since I need you to do my debian uploads. :D
But it would certainly help to be a bit less stubborn only wanting to code in the language that matches your country code. :O
It's even funnier that I prefer the C.UTF-8 locale over pl_PL.UTF-8, and the C code works several orders of magnitude faster than the Perl+Python version.
Ok, ok, in this case it's C++ but only because I couldn't be arsed to implement a <set> the hard way.
Quoted 2 lines
Or maybe I can help a bit, since it sounds like a nice one for the coding examples in the lib. ;]
The lib is a great help -- the ioctls are totally undocumented, your code is a lot easier than RTFKing.
Meow!
Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢰⠒⠀⣿⡁ Vat kind uf sufficiently advanced technology iz dis!? ⢿⡄⠘⠷⠚⠋⠀ -- Genghis Ht'rok'din ⠈⠳⣄⠀⠀⠀⠀
Quoted 19 lines · Adam Borowski
Quoted 7 lines · Cloud Admin
Hi,
I used the mount option 'compression' on some mounted sub volumes. How can I revoke the compression? Means to delete the option and get all data uncompressed on this volume.
Is it enough to remount the sub volume without this option? Or is it necessary to do some addional step (balancing?) to get all stored data uncompressed.If you set it via mount option, removing the option is enough to disable compression for _new_ files. Other ways are chattr +c and btrfs-property, but if you haven't heard about those you almost surely don't have such attributes set.
After remounting, you may uncompress existing files. Balancing won't do this as it moves extents around without looking inside; defrag on the other hand rewrites extents thus as a side effect it applies new [non]compression settings. Thus: ｢btrfs fi defrag -r /path/to/filesystem｣.
Quoted 8 lines
Quoted 2 lines
Beside of it, is it possible to find out what the real and compressed size of a file, for example or the ratio?
Currently not.
I've once written a tool which does this, but 1. it's extremely slow, 2. insane, 3. so insane a certain member of this list would kill me had I distributed the tool. Thus, I'd need to rewrite it first...
AFAIK the only method to determine the compression ratio is to check the EXTENT_DATA key and its corresponding file_extent_item structure. (Which I assume Adam is doing this way)
In that structure is records its on-disk data size and in-memory data size. (All rounded up to sectorsize, which is 4K in most case) So in theory it's possible to determine the compression ratio.
The only method I can think of (maybe I forgot some methods?) is to use offline tool (btrfs-debug-tree) to check that. FS APIs like fiemap doesn't even support to report on-disk data size so we can't use it.
But the problem is more complicated, especially when compressed CoW is involved.
For example, there is an extent (A) which represents the data for inode 258, range [0,128k). On disk size its just 4K.
And when we write the range [32K, 64K), which get CoWed and compressed, resulting a new file extent (B) for inode 258, range [32K, 64K), and on disk size is 4K as an example.
Then file extent layout for 258 will be:
[0,32k): range [0,32K) of uncompressed Extent A [32k, 64k): range [0,32k) of uncompressed Extent B [64k, 128k): range [64k, 128K) of uncompressed Extent A.
And on disk extent size is 4K (compressed Extent A) + 4K (compressed Extent B) = 8K.
Before the write, the compresstion ratio is 4K/128K = 3.125% While after write, the compression ratio is 8K/128K = 6.25%
Not to mention that it's possible to have uncompressed file extent.
So it's complicated even we're just using offline tool to determine the compression ratio of btrfs compressed file.
Thanks,
Qu
Quoted 4 lines
Meow!
Quoted 9 lines · Qu Wenruo
Quoted 3 lines · Adam Borowski
I've once written a tool which does this, but 1. it's extremely slow, 2. insane, 3. so insane a certain member of this list would kill me had I distributed the tool. Thus, I'd need to rewrite it first...
AFAIK the only method to determine the compression ratio is to check the
EXTENT_DATAkey and its correspondingfile_extent_itemstructure. (Which I assume Adam is doing this way)
Quoted 8 lines
In that structure is records its on-disk data size and in-memory data size. (All rounded up to sectorsize, which is 4K in most case)
So in theory it's possible to determine the compression ratio.The only method I can think of (maybe I forgot some methods?) is to use offline tool (btrfs-debug-tree) to check that.
FS APIs like fiemap doesn't even support to report on-disk data size so we can't use it.
BTRFS_IOC_TREE_SEARCH_V2 returns all we want to know; its only downside is being root only.
Quoted 16 lines
But the problem is more complicated, especially when compressed CoW is involved.
For example, there is an extent (A) which represents the data for inode 258, range [0,128k).
On disk size its just 4K.And when we write the range [32K, 64K), which get CoWed and compressed, resulting a new file extent (B) for inode 258, range [32K, 64K), and on disk size is 4K as an example.
Then file extent layout for 258 will be:
[0,32k): range [0,32K) of uncompressed Extent A [32k, 64k): range [0,32k) of uncompressed Extent B [64k, 128k): range [64k, 128K) of uncompressed Extent A.
Quoted 5 lines
And on disk extent size is 4K (compressed Extent A) + 4K (compressed Extent
B) = 8K.Before the write, the compresstion ratio is 4K/128K = 3.125% While after write, the compression ratio is 8K/128K = 6.25%
There's no real meaningful way to speak about compression ratio of a partial extent. Thus, I decided to, for every extent, take compressed:uncompressed sizes of the whole extent, no matter whether the file uses only a few bytes of that extent or references it a thousand times.
Quoted 1 line
Not to mention that it's possible to have uncompressed file extent.
Yeah, the tool gives a report like:
all 74% 9.2M/ 13M lzo 68% 7.1M/ 11M none 100% 2.1M/ 2.1M
as you typically have a mix of compressible and uncompressible data.
喵!
Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢰⠒⠀⣿⡁ Vat kind uf sufficiently advanced technology iz dis!? ⢿⡄⠘⠷⠚⠋⠀ -- Genghis Ht'rok'din ⠈⠳⣄⠀⠀⠀⠀
Quoted 14 lines · Adam Borowski
Quoted 13 lines · Qu Wenruo
Quoted 3 lines · Adam Borowski
I've once written a tool which does this, but 1. it's extremely slow, 2. insane, 3. so insane a certain member of this list would kill me had I distributed the tool. Thus, I'd need to rewrite it first...
AFAIK the only method to determine the compression ratio is to check the
EXTENT_DATAkey and its correspondingfile_extent_itemstructure. (Which I assume Adam is doing this way)In that structure is records its on-disk data size and in-memory data size. (All rounded up to sectorsize, which is 4K in most case)
So in theory it's possible to determine the compression ratio.
Quoted 7 lines
Quoted 4 lines
The only method I can think of (maybe I forgot some methods?) is to use offline tool (btrfs-debug-tree) to check that.
FS APIs like fiemap doesn't even support to report on-disk data size so we can't use it.
BTRFS_IOC_TREE_SEARCH_V2returns all we want to know; its only downside is being root only.
Just forgot that.
Quoted 20 lines
Quoted 20 lines
But the problem is more complicated, especially when compressed CoW is involved.
For example, there is an extent (A) which represents the data for inode 258, range [0,128k).
On disk size its just 4K.And when we write the range [32K, 64K), which get CoWed and compressed, resulting a new file extent (B) for inode 258, range [32K, 64K), and on disk size is 4K as an example.
Then file extent layout for 258 will be:
[0,32k): range [0,32K) of uncompressed Extent A [32k, 64k): range [0,32k) of uncompressed Extent B [64k, 128k): range [64k, 128K) of uncompressed Extent A.And on disk extent size is 4K (compressed Extent A) + 4K (compressed Extent
B) = 8K.
Quoted 7 lines
Quoted 2 lines
Before the write, the compresstion ratio is 4K/128K = 3.125% While after write, the compression ratio is 8K/128K = 6.25%
There's no real meaningful way to speak about compression ratio of a partial extent. Thus, I decided to, for every extent, take compressed:uncompressed sizes of the whole extent, no matter whether the file uses only a few bytes of that extent or references it a thousand times.
Very clever move.
Quoted 8 lines
Quoted 2 lines
Not to mention that it's possible to have uncompressed file extent.
Yeah, the tool gives a report like:
all 74% 9.2M/ 13M lzo 68% 7.1M/ 11M none 100% 2.1M/ 2.1Mas you typically have a mix of compressible and uncompressible data.
Looks quite nice!
Thanks,
Qu
Quoted 4 lines
喵!
Quoted 53 lines · Qu Wenruo
Quoted 19 lines · Adam Borowski
Quoted 7 lines · Cloud Admin
Hi,
I used the mount option 'compression' on some mounted sub volumes. How can I revoke the compression? Means to delete the option and get all data uncompressed on this volume.
Is it enough to remount the sub volume without this option? Or is it necessary to do some addional step (balancing?) to get all stored data uncompressed.If you set it via mount option, removing the option is enough to disable compression for _new_ files. Other ways are chattr +c and btrfs-property, but if you haven't heard about those you almost surely don't have such attributes set.
After remounting, you may uncompress existing files. Balancing won't do this as it moves extents around without looking inside; defrag on the other hand rewrites extents thus as a side effect it applies new [non]compression settings. Thus: ｢btrfs fi defrag -r /path/to/filesystem｣.
Quoted 8 lines
Quoted 2 lines
Beside of it, is it possible to find out what the real and compressed size of a file, for example or the ratio?
Currently not.
I've once written a tool which does this, but 1. it's extremely slow, 2. insane, 3. so insane a certain member of this list would kill me had I distributed the tool. Thus, I'd need to rewrite it first...
AFAIK the only method to determine the compression ratio is to check the
EXTENT_DATAkey and its correspondingfile_extent_itemstructure. (Which I assume Adam is doing this way)In that structure is records its on-disk data size and in-memory data size. (All rounded up to sectorsize, which is 4K in most case) So in theory it's possible to determine the compression ratio.
The only method I can think of (maybe I forgot some methods?) is to use offline tool (btrfs-debug-tree) to check that. FS APIs like fiemap doesn't even support to report on-disk data size so we can't use it.
But the problem is more complicated, especially when compressed CoW is involved.
For example, there is an extent (A) which represents the data for inode 258, range [0,128k). On disk size its just 4K.
And when we write the range [32K, 64K), which get CoWed and compressed, resulting a new file extent (B) for inode 258, range [32K, 64K), and on disk size is 4K as an example.
Then file extent layout for 258 will be:
[0,32k): range [0,32K) of uncompressed Extent A [32k, 64k): range [0,32k) of uncompressed Extent B [64k, 128k): range [64k, 128K) of uncompressed Extent A.
Quoted 8 lines
And on disk extent size is 4K (compressed Extent A) + 4K (compressed Extent B) = 8K.
Before the write, the compresstion ratio is 4K/128K = 3.125% While after write, the compression ratio is 8K/128K = 6.25%
Not to mention that it's possible to have uncompressed file extent.
So it's complicated even we're just using offline tool to determine the compression ratio of btrfs compressed file.
Out of curiosity, is there any easier method if you just want an aggregate ratio for the whole filesystem? The intuitive option of comparing du -sh output to how much space is actually used in chunks is obviously out because that will count sparse ranges as 'compressed', and there should actually be a significant difference in values there for an uncompressed filesystem (the chunk usage should be higher).
Quoted 56 lines · Austin S. Hemmelgarn
Quoted 55 lines · Qu Wenruo
Quoted 19 lines · Adam Borowski
Quoted 7 lines · Cloud Admin
Hi,
I used the mount option 'compression' on some mounted sub volumes. How can I revoke the compression? Means to delete the option and get all data uncompressed on this volume.
Is it enough to remount the sub volume without this option? Or is it necessary to do some addional step (balancing?) to get all stored data uncompressed.If you set it via mount option, removing the option is enough to disable compression for _new_ files. Other ways are chattr +c and btrfs-property, but if you haven't heard about those you almost surely don't have such attributes set.
After remounting, you may uncompress existing files. Balancing won't do this as it moves extents around without looking inside; defrag on the other hand rewrites extents thus as a side effect it applies new [non]compression settings. Thus: ｢btrfs fi defrag -r /path/to/filesystem｣.
Quoted 8 lines
Quoted 2 lines
Beside of it, is it possible to find out what the real and compressed size of a file, for example or the ratio?
Currently not.
I've once written a tool which does this, but 1. it's extremely slow, 2. insane, 3. so insane a certain member of this list would kill me had I distributed the tool. Thus, I'd need to rewrite it first...
AFAIK the only method to determine the compression ratio is to check the
EXTENT_DATAkey and its correspondingfile_extent_itemstructure. (Which I assume Adam is doing this way)In that structure is records its on-disk data size and in-memory data size. (All rounded up to sectorsize, which is 4K in most case) So in theory it's possible to determine the compression ratio.
The only method I can think of (maybe I forgot some methods?) is to use offline tool (btrfs-debug-tree) to check that. FS APIs like fiemap doesn't even support to report on-disk data size so we can't use it.
But the problem is more complicated, especially when compressed CoW is involved.
For example, there is an extent (A) which represents the data for inode 258, range [0,128k). On disk size its just 4K.
And when we write the range [32K, 64K), which get CoWed and compressed, resulting a new file extent (B) for inode 258, range [32K, 64K), and on disk size is 4K as an example.
Then file extent layout for 258 will be:
[0,32k): range [0,32K) of uncompressed Extent A [32k, 64k): range [0,32k) of uncompressed Extent B [64k, 128k): range [64k, 128K) of uncompressed Extent A.And on disk extent size is 4K (compressed Extent A) + 4K (compressed Extent B) = 8K.
Quoted 7 lines
Quoted 6 lines
Before the write, the compresstion ratio is 4K/128K = 3.125% While after write, the compression ratio is 8K/128K = 6.25%
Not to mention that it's possible to have uncompressed file extent.
So it's complicated even we're just using offline tool to determine the compression ratio of btrfs compressed file.
Out of curiosity, is there any easier method if you just want an aggregate ratio for the whole filesystem? The intuitive option of comparing
du -shoutput to how much space is actually used in chunks is obviously out because that will count sparse ranges as 'compressed', and there should actually be a significant difference in values there for an uncompressed filesystem (the chunk usage should be higher).
I can be totally wrong (since I just forgot the quite obvious SEARCH_TREE ioctl), but according to btrfs on-disk format, only EXTENT_DATA contains the compression ratio (ram size and on-disk size).
So to get ratio for the whole fs, one needs to iterate through the whole extent tree, and follows the (any is enough) backref to locate the EXTENT_DATA and get the compression ratio.
That's to say, that will be slow anyway.
Thanks,
Qu
Quoted 4 lines
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html