Sharing my faster compsize implementation
I love compsize a lot. However, It is a little slow for me. So I'm trying to write a faster compsize implementation, just for my use case, which gets a great speedup by both reducing syscalls and multi-threading.
I'm sharing this in case it might be of interest to others in the community. All feedback and questions are welcome. The project is on
https://github.com/SaltyKitkat/xsz
And followings are the performance comparison on my machine.
On a SATA SSD device, with some snapshots for backing up, and some large media files.
Cache is cleared before each run.
``` $ sudo time ./xsz -j1 /mnt/1 Processed 5663444 files, 1097233 regular extents (3859780 refs), 3459956 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 95% 827G 865G 1.3T none 100% 811G 811G 1.1T zstd 29% 15G 54G 192G 4.27user 37.46system 1:14.13elapsed 56%CPU (0avgtext+0avgdata 31420maxresident)k 5060832inputs+0outputs (0major+46434minor)pagefaults 0swaps $ sudo time ./xsz -j4 /mnt/1 Processed 5663444 files, 1097233 regular extents (3859780 refs), 3459956 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 95% 827G 865G 1.3T none 100% 811G 811G 1.1T zstd 29% 15G 54G 192G 4.89user 37.88system 0:20.42elapsed 209%CPU (0avgtext+0avgdata 35820maxresident)k 5060832inputs+0outputs (0major+79430minor)pagefaults 0swaps $ sudo time compsize /mnt/1 Processed 5663444 files, 1097233 regular extents (3859780 refs), 3459956 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 95% 827G 865G 1.3T none 100% 811G 811G 1.1T zstd 29% 15G 54G 192G 3.72user 72.98system 1:50.07elapsed 69%CPU (0avgtext+0avgdata 80132maxresident)k 5254008inputs+0outputs (1major+24967minor)pagefaults 0swaps ```
On a HDD device, with a lot of small files, and some program files.
Cache is cleared before each run.
``` $ sudo time ./xsz -j1 /mnt/guest Processed 393486 files, 215207 regular extents (215207 refs), 178308 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 100% 29G 29G 29G none 100% 29G 29G 29G 0.37user 3.47system 1:07.54elapsed 5%CPU (0avgtext+0avgdata 5436maxresident)k 830912inputs+0outputs (0major+7825minor)pagefaults 0swaps $ sudo time ./xsz -j4 /mnt/guest Processed 393486 files, 215207 regular extents (215207 refs), 178308 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 100% 29G 29G 29G none 100% 29G 29G 29G 0.40user 3.97system 0:57.72elapsed 7%CPU (0avgtext+0avgdata 8064maxresident)k 830912inputs+0outputs (0major+5093minor)pagefaults 0swaps $ sudo time compsize /mnt/guest Processed 393486 files, 215207 regular extents (215207 refs), 178308 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 100% 29G 29G 29G none 100% 29G 29G 29G 0.42user 9.45system 2:49.78elapsed 5%CPU (0avgtext+0avgdata 12648maxresident)k 954080inputs+0outputs (0major+2990minor)pagefaults 0swaps ```