New tools and changes

Hi,

I have committed a few new tools to the fio repo, and enabled the building of these tools (and the older ones) by default now. Previously, you'd have to do

$ make t/<toolname>

manually to build them, that is no longer the case. Anyway, the new tools are:

t/dedupe: this is a dedupe checker. You run this on a device or a file, and it will tell you how well the blocks on that device will dedupe. The block size defaults to 4KB, but can be set to anything you wish. By default this will use a bloom filter, which (for my testing) is within 1% or better of the actually fully tracked check. You can do a full check with -B0 to turn off the bloom, however that can consume large amounts of memory on larger devices. If you are truly paranoid, there's a -c option as well to do full checking in case there are checksum collisions. The checksum used is md5. The output looks like this:

axboe@nelson:/home/axboe/git/fio $ sudo t/dedupe /dev/sda5 Will check </dev/sda5>, size <49999249408>, using 8 threads Threads(8): 12206848 items processed Extents=12206848, Unique extents=1625328
De-dupe ratio: 1:6.51
Fio setting: dedupe_percentage=87

dedupe will default to using as many threads as there are CPUs in the system, to speed things up. This is also settable. dedupe will also give you the fio setting to use to create a write workload with similar dedupability. The dedupe_percentage is a new option that allows you to set that.

t/btrace2fio: this is a tool that takes a blkparse binary dump and generates a fio job file matching that dump. This tools needs a bit more development effort, and I'd welcome people joining in and helping out. Right now it figures out the queue depth, read vs write mix, block size distribution, etc. It makes no attempt to collapse identical threads into one job with numjobs=x for readability, however.

I should write up some documentation for these tools, and will do so in the not-so-distant future. The dedupe tool probably needs a bit of error checking and handling and option range checking, but apart from that, it's mostly done. The btrace2fio tool is a bit more rough around the edges. I'd welcome feedback, suggestions, improvements for both, as usual.

Jens Axboe

Hi,

I forgot to mention another handy change that went in recently - fio now supports including files from job files. This should make it easier to manage a test harness of fio files, since you can have more manageable files with a global (or per-host, whatever) include fio that has options for that particular instance. It's pretty simple to use, here's the section that was added to the HOWTO to explain it:

-----------

When fio is utilized as a basis of any reasonably large test suite, it might be desirable to share a set of standardized settings across multiple job files. Instead of copy/pasting such settings, any section may pull in an external .fio file with 'include filename' directive, as in the following example:

; -- start job file including.fio --
[global]
filename=/tmp/test
filesize=1m
include glob-include.fio

[test]
rw=randread
bs=4k
time_based=1
runtime=10
include test-include.fio
; -- end job file including.fio --

; -- start job file glob-include.fio --
thread=1
group_reporting=1
; -- end job file glob-include.fio --

; -- start job file test-include.fio --
ioengine=libaio
iodepth=4
; -- end job file test-include.fio --

Settings pulled into a section apply to that section only (except global section). Include directives may be nested in that any included file may contain further include directive(s). Include files may not contain [] sections.

--------

This feature courtesy of Andrey Kuzmin <[email protected]> who did all the work to make this feasible.

Jens Axboe