Skip to contentSkip to conversation

CEPH to BTRFS over NFS results in no compression

DISCUSSION Konstantin V. Gavrilenko 2 participants 3 messages Jan 2, 2022 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi list,

I have noticed an interesting and surprising behaviour of my BTRFS with regards to compression of the files and NFS.

I have BTRFS RAID10 with 8 disks , that is mounted with the " nofail,noatime,space_cache=v2,compress-force=zstd:9,subvol=@cloudstack-secondary" flags and is exported via NFS.

When I create a snapshot of a disk in Cloudstack from CEPH and save it to a secondary storage to this BTRFS RAID10 over NFS, the file does not compress, despite the compress-force mount option being set on FS So in the below example, the file eeceaf0e-9780-408b-a748-1495d517a9b6 was copied over NFS and is not compressed. When I copy the same file directly on a host, it does get compressed pretty well, as per example below.

root@backup1 : /mnt/backup1/cloudstack-secondary/snapshots/49/5355: compsize eeceaf0e-9780-408b-a748-1495d517a9b6 Type Perc Disk Usage Uncompressed Referenced
TOTAL 99% 9.8G 9.8G 9.8G
none 100% 9.8G 9.8G 9.8G
zstd 3% 4.0K 128K 124K

root@backup1 : /mnt/backup1/cloudstack-secondary/snapshots/49/5355: cp eeceaf0e-9780-408b-a748-1495d517a9b6 testfile 'eeceaf0e-9780-408b-a748-1495d517a9b6' -> 'testfile'

root@backup1 : /mnt/backup1/cloudstack-secondary/snapshots/49/5355: compsize testfile Type Perc Disk Usage Uncompressed Referenced
TOTAL 38% 3.7G 9.8G 9.8G
none 100% 1.1G 1.1G 1.1G
zstd 30% 2.6G 8.6G 8.6G

root@backup1: /mnt/backup1/cloudstack-secondary/snapshots/49/5355  btrfs filesystem defrag -v -f -czstd ./eeceaf0e-9780-408b-a748-1495d517a9b6 ./eeceaf0e-9780-408b-a748-1495d517a9b6

root@backup1: /mnt/backup1/cloudstack-secondary/snapshots/49/5355  compsize eeceaf0e-9780-408b-a748-1495d517a9b6 Type Perc Disk Usage Uncompressed Referenced
TOTAL 38% 3.7G 9.8G 9.8G
none 100% 1.1G 1.1G 1.1G
zstd 30% 2.6G 8.6G 8.6G

So what I have checked so far what works
- after the original files is copied over NFS, the copy of the file using #cp gets compressed.
- after the original files is copied over NFS, the original file can be compressed using #btrfs defrag -czstd option
- If I copy the original file to some other host, and copy it back via NFS using cp, it does get compressed.

So the problem seems to appear only when the file is exported from Ceph and copied to NFS.

Any hints what could be causing such a behaviour?

Yours sincerely,
Kos

#1 lore raw
Quoted 17 lines · On Sun, 2 Jan 2022 11:32:53 +0000 (UTC) "Konstantin V. Gavrilenko"…

Hi list,

I have noticed an interesting and surprising behaviour of my BTRFS with regards to compression of the files and NFS.

I have BTRFS RAID10 with 8 disks , that is mounted with the " nofail,noatime,space_cache=v2,compress-force=zstd:9,subvol=@cloudstack-secondary" flags and is exported via NFS.

When I create a snapshot of a disk in Cloudstack from CEPH and save it to a secondary storage to this BTRFS RAID10 over NFS, the file does not compress, despite the compress-force mount option being set on FS So in the below example, the file eeceaf0e-9780-408b-a748-1495d517a9b6 was copied over NFS and is not compressed. When I copy the same file directly on a host, it does get compressed pretty well, as per example below.

[...]

So what I have checked so far what works
- after the original files is copied over NFS, the copy of the file using #cp gets compressed.
- after the original files is copied over NFS, the original file can be compressed using #btrfs defrag -czstd option
- If I copy the original file to some other host, and copy it back via NFS using cp, it does get compressed.

Quoted 9 lines

So the problem seems to appear only when the file is exported from Ceph and copied to NFS.

Any hints what could be causing such a behaviour?

Yours sincerely,
Kos

Btrfs doesn't compress direct-io writes. This suggests that whatever you use for "I create a snapshot of a disk in Cloudstack from CEPH and save it to a secondary storage to this BTRFS RAID10 over NFS", it writes with O_DIRECT.

Regards,
Lukas Straub

#2 lore raw

Thanks for clarification Lukas. I didn't know that.

I guess the easiest option for me is to recompress snapshots on a daily basis after they get uploaded.

yours,
kos

----- Original Message -----

Quoted 55 lines

From: "Lukas Straub" <[email protected]>
To: "Konstantin V Gavrilenko" <[email protected]>
Cc: "linux-btrfs" <[email protected]>
Sent: Sunday, 2 January, 2022 15:08:33
Subject: Re: CEPH to BTRFS over NFS results in no compression

Quoted 30 lines · On Sun, 2 Jan 2022 11:32:53 +0000 (UTC) "Konstantin V. Gavrilenko"…

Hi list,

I have noticed an interesting and surprising behaviour of my BTRFS with regards to compression of the files and NFS.

I have BTRFS RAID10 with 8 disks , that is mounted with the " nofail,noatime,space_cache=v2,compress-force=zstd:9,subvol=@cloudstack-secondary" flags and is exported via NFS.

When I create a snapshot of a disk in Cloudstack from CEPH and save it to a secondary storage to this BTRFS RAID10 over NFS, the file does not compress, despite the compress-force mount option being set on FS
So in the below example, the file eeceaf0e-9780-408b-a748-1495d517a9b6 was copied over NFS and is not compressed. When I copy the same file directly on a host, it does get compressed pretty well, as per example below.

[...]

So what I have checked so far what works
- after the original files is copied over NFS, the copy of the file using #cp gets compressed.
- after the original files is copied over NFS, the original file can be compressed using #btrfs defrag -czstd option
- If I copy the original file to some other host, and copy it back via NFS using cp, it does get compressed.

So the problem seems to appear only when the file is exported from Ceph and copied to NFS.

Quoted 7 lines

Any hints what could be causing such a behaviour?

Yours sincerely,
Kos

Btrfs doesn't compress direct-io writes. This suggests that whatever you use for "I create a snapshot of a disk in Cloudstack from CEPH and save it to a secondary storage to this BTRFS RAID10 over NFS", it writes with O_DIRECT.

Regards,
Lukas Straub

--