CEPH to BTRFS over NFS results in no compression
Hi list,
I have noticed an interesting and surprising behaviour of my BTRFS with regards to compression of the files and NFS.
I have BTRFS RAID10 with 8 disks , that is mounted with the " nofail,noatime,space_cache=v2,compress-force=zstd:9,subvol=@cloudstack-secondary" flags and is exported via NFS.
When I create a snapshot of a disk in Cloudstack from CEPH and save it to a secondary storage to this BTRFS RAID10 over NFS, the file does not compress, despite the compress-force mount option being set on FS So in the below example, the file eeceaf0e-9780-408b-a748-1495d517a9b6 was copied over NFS and is not compressed. When I copy the same file directly on a host, it does get compressed pretty well, as per example below.
root@backup1 : /mnt/backup1/cloudstack-secondary/snapshots/49/5355: compsize eeceaf0e-9780-408b-a748-1495d517a9b6 Type Perc Disk Usage Uncompressed Referenced
TOTAL 99% 9.8G 9.8G 9.8G
none 100% 9.8G 9.8G 9.8G
zstd 3% 4.0K 128K 124K
root@backup1 : /mnt/backup1/cloudstack-secondary/snapshots/49/5355: cp eeceaf0e-9780-408b-a748-1495d517a9b6 testfile 'eeceaf0e-9780-408b-a748-1495d517a9b6' -> 'testfile'
root@backup1 : /mnt/backup1/cloudstack-secondary/snapshots/49/5355: compsize testfile Type Perc Disk Usage Uncompressed Referenced
TOTAL 38% 3.7G 9.8G 9.8G
none 100% 1.1G 1.1G 1.1G
zstd 30% 2.6G 8.6G 8.6G
root@backup1: /mnt/backup1/cloudstack-secondary/snapshots/49/5355 btrfs filesystem defrag -v -f -czstd ./eeceaf0e-9780-408b-a748-1495d517a9b6 ./eeceaf0e-9780-408b-a748-1495d517a9b6
root@backup1: /mnt/backup1/cloudstack-secondary/snapshots/49/5355 compsize eeceaf0e-9780-408b-a748-1495d517a9b6 Type Perc Disk Usage Uncompressed Referenced
TOTAL 38% 3.7G 9.8G 9.8G
none 100% 1.1G 1.1G 1.1G
zstd 30% 2.6G 8.6G 8.6G
So what I have checked so far what works
- after the original files is copied over NFS, the copy of the file using #cp gets compressed.
- after the original files is copied over NFS, the original file can be compressed using #btrfs defrag -czstd option
- If I copy the original file to some other host, and copy it back via NFS using cp, it does get compressed.
So the problem seems to appear only when the file is exported from Ceph and copied to NFS.
Any hints what could be causing such a behaviour?
Yours sincerely,
Kos
Quoted 17 lines · On Sun, 2 Jan 2022 11:32:53 +0000 (UTC) "Konstantin V. Gavrilenko"…
Hi list,
I have noticed an interesting and surprising behaviour of my BTRFS with regards to compression of the files and NFS.
I have BTRFS RAID10 with 8 disks , that is mounted with the " nofail,noatime,space_cache=v2,compress-force=zstd:9,subvol=@cloudstack-secondary" flags and is exported via NFS.
When I create a snapshot of a disk in Cloudstack from CEPH and save it to a secondary storage to this BTRFS RAID10 over NFS, the file does not compress, despite the compress-force mount option being set on FS So in the below example, the file eeceaf0e-9780-408b-a748-1495d517a9b6 was copied over NFS and is not compressed. When I copy the same file directly on a host, it does get compressed pretty well, as per example below.
[...]
So what I have checked so far what works
- after the original files is copied over NFS, the copy of the file using #cp gets compressed.
- after the original files is copied over NFS, the original file can be compressed using #btrfs defrag -czstd option
- If I copy the original file to some other host, and copy it back via NFS using cp, it does get compressed.
Quoted 9 lines
So the problem seems to appear only when the file is exported from Ceph and copied to NFS.
Any hints what could be causing such a behaviour?
Yours sincerely,
Kos
Btrfs doesn't compress direct-io writes. This suggests that whatever you use for "I create a snapshot of a disk in Cloudstack from CEPH and save it to a secondary storage to this BTRFS RAID10 over NFS", it writes with O_DIRECT.
Regards,
Lukas Straub
Thanks for clarification Lukas. I didn't know that.
I guess the easiest option for me is to recompress snapshots on a daily basis after they get uploaded.
yours,
kos
----- Original Message -----
Quoted 55 lines
From: "Lukas Straub" <[email protected]>
To: "Konstantin V Gavrilenko" <[email protected]>
Cc: "linux-btrfs" <[email protected]>
Sent: Sunday, 2 January, 2022 15:08:33
Subject: Re: CEPH to BTRFS over NFS results in no compressionQuoted 30 lines · On Sun, 2 Jan 2022 11:32:53 +0000 (UTC) "Konstantin V. Gavrilenko"…
Hi list,
I have noticed an interesting and surprising behaviour of my BTRFS with regards to compression of the files and NFS.
I have BTRFS RAID10 with 8 disks , that is mounted with the " nofail,noatime,space_cache=v2,compress-force=zstd:9,subvol=@cloudstack-secondary" flags and is exported via NFS.
When I create a snapshot of a disk in Cloudstack from CEPH and save it to a secondary storage to this BTRFS RAID10 over NFS, the file does not compress, despite the compress-force mount option being set on FS
So in the below example, the file eeceaf0e-9780-408b-a748-1495d517a9b6 was copied over NFS and is not compressed. When I copy the same file directly on a host, it does get compressed pretty well, as per example below.[...]
So what I have checked so far what works
- after the original files is copied over NFS, the copy of the file using #cp gets compressed.
- after the original files is copied over NFS, the original file can be compressed using #btrfs defrag -czstd option
- If I copy the original file to some other host, and copy it back via NFS using cp, it does get compressed.So the problem seems to appear only when the file is exported from Ceph and copied to NFS.
Quoted 7 lines
Any hints what could be causing such a behaviour?
Yours sincerely,
KosBtrfs doesn't compress direct-io writes. This suggests that whatever you use for "I create a snapshot of a disk in Cloudstack from CEPH and save it to a secondary storage to this BTRFS RAID10 over NFS", it writes with
O_DIRECT.Regards,
Lukas Straub--