Skip to contentSkip to conversation

Sharing my faster compsize implementation

DISCUSSION Sun YangKai 1 participant 1 message Apr 24, 2025 Btrfs file system

lore
Share & tools
lore
#0 lore raw

I love compsize a lot. However, It is a little slow for me. So I'm trying to write a faster compsize implementation, just for my use case, which gets a great speedup by both reducing syscalls and multi-threading.

I'm sharing this in case it might be of interest to others in the community. All feedback and questions are welcome. The project is on
https://github.com/SaltyKitkat/xsz

And followings are the performance comparison on my machine.

On a SATA SSD device, with some snapshots for backing up, and some large media files.
Cache is cleared before each run.

```
$ sudo time ./xsz -j1 /mnt/1
Processed 5663444 files, 1097233 regular extents (3859780 refs), 3459956 
inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       95%      827G         865G         1.3T
none       100%      811G         811G         1.1T
zstd        29%       15G          54G         192G
4.27user 37.46system 1:14.13elapsed 56%CPU (0avgtext+0avgdata 
31420maxresident)k
5060832inputs+0outputs (0major+46434minor)pagefaults 0swaps

$ sudo time ./xsz -j4 /mnt/1
Processed 5663444 files, 1097233 regular extents (3859780 refs), 3459956 
inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       95%      827G         865G         1.3T
none       100%      811G         811G         1.1T
zstd        29%       15G          54G         192G
4.89user 37.88system 0:20.42elapsed 209%CPU (0avgtext+0avgdata 
35820maxresident)k
5060832inputs+0outputs (0major+79430minor)pagefaults 0swaps

$ sudo time compsize /mnt/1
Processed 5663444 files, 1097233 regular extents (3859780 refs), 3459956 
inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       95%      827G         865G         1.3T
none       100%      811G         811G         1.1T
zstd        29%       15G          54G         192G
3.72user 72.98system 1:50.07elapsed 69%CPU (0avgtext+0avgdata 
80132maxresident)k
5254008inputs+0outputs (1major+24967minor)pagefaults 0swaps
```

On a HDD device, with a lot of small files, and some program files.

Cache is cleared before each run.

```
$ sudo time ./xsz -j1 /mnt/guest
Processed 393486 files, 215207 regular extents (215207 refs), 178308 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%       29G          29G          29G
none       100%       29G          29G          29G
0.37user 3.47system 1:07.54elapsed 5%CPU (0avgtext+0avgdata 5436maxresident)k
830912inputs+0outputs (0major+7825minor)pagefaults 0swaps

$ sudo time ./xsz -j4 /mnt/guest
Processed 393486 files, 215207 regular extents (215207 refs), 178308 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%       29G          29G          29G
none       100%       29G          29G          29G
0.40user 3.97system 0:57.72elapsed 7%CPU (0avgtext+0avgdata 8064maxresident)k
830912inputs+0outputs (0major+5093minor)pagefaults 0swaps

$ sudo time compsize /mnt/guest
Processed 393486 files, 215207 regular extents (215207 refs), 178308 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%       29G          29G          29G
none       100%       29G          29G          29G
0.42user 9.45system 2:49.78elapsed 5%CPU (0avgtext+0avgdata 12648maxresident)k
954080inputs+0outputs (0major+2990minor)pagefaults 0swaps
```