Have we changed number of fields in fio --minimal output

Hi,

I was running latest fio and noticed that number of fields in fio --minimal output have gone up from 69 to 77. A increase of 8 fields. Don't see any update in --minimal documentation. Is it regarding total latency thing?

Quoted 6 lines · Vivek Goyal

Hi,

I was running latest fio and noticed that number of fields in fio --minimal output have gone up from 69 to 77. A increase of 8 fields. Don't see any update in --minimal documentation. Is it regarding total latency thing?

Woops yes, there's a total latency in there as well now. Should just be 4 extra fields, though. It gets logged after completion latency, but before bandwidth stats. I'll update the documentation.

Should we perhaps put a versioning field in there? Now would seem to be a good time, since the output has changed anyway. I'm open to suggestions from you or other terse output users.

Quoted 8 lines · Jens Axboe

Quoted 6 lines · Vivek Goyal

Hi,

I was running latest fio and noticed that number of fields in fio --minimal output have gone up from 69 to 77. A increase of 8 fields. Don't see any update in --minimal documentation. Is it regarding total latency thing?

Quoted 8 lines

Woops yes, there's a total latency in there as well now. Should just be 4 extra fields, though. It gets logged after completion latency, but before bandwidth stats. I'll update the documentation.

Should we perhaps put a versioning field in there? Now would seem to be a good time, since the output has changed anyway. I'm open to suggestions from you or other terse output users.

How about redesigning it a bit to make it more bullet proof... We could prefix series of fields with the value they are logging. So for instance, the 4 completion latency fields would include a clat prefix first:

clat[%lu;%lu;%f;%f],foo[%lu;%lu],etc

Would that not be more resilient to future changes? New fields would not bother you, and reordering should also be fine.

Any other ideas?

Quoted 23 lines · Jens Axboe

Quoted 8 lines · Jens Axboe

Quoted 6 lines · Vivek Goyal

Hi,

I was running latest fio and noticed that number of fields in fio --minimal output have gone up from 69 to 77. A increase of 8 fields. Don't see any update in --minimal documentation. Is it regarding total latency thing?

Quoted 8 lines

Woops yes, there's a total latency in there as well now. Should just be 4 extra fields, though. It gets logged after completion latency, but before bandwidth stats. I'll update the documentation.

Should we perhaps put a versioning field in there? Now would seem to be a good time, since the output has changed anyway. I'm open to suggestions from you or other terse output users.

How about redesigning it a bit to make it more bullet proof... We could prefix series of fields with the value they are logging. So for instance, the 4 completion latency fields would include a clat prefix first:

Quoted 6 lines

clat[%lu;%lu;%f;%f],foo[%lu;%lu],etc

Would that not be more resilient to future changes? New fields would not bother you, and reordering should also be fine.

Any other ideas?

With that change, the output would be modified from:

file;0;0;131072;356015;377;0;0;0.000000;0.000000;0;0;0.000000;0.000000;0;0;0.000000;0.000000;0;0;0.000000%;0.000000;0.000000;131072;303660;442;0;0;0.000000;0.000000;0;0;0.000000;0.000000;0;0;0.000000;0.000000;0;0;0.000000%;0.000000;0.000000;99.265606%;0.367197%;95;0;343;100.0%;0.0%;0.0%;0.0%;0.0%;0.0%;0.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%

to

id[file;0;0];overview[131072;357913;375];slat[0;0;0.000000;0.000000];clat[0;0;0.000000;0.000000];lat[0;0;0.000000;0.000000];bw[0;0;0.000000%;0.000000;0.000000];overview[131072;304348;441];slat[0;0;0.000000;0.000000];clat[0;0;0.000000;0.000000];lat[0;0;0.000000;0.000000];bw[0;0;0.000000%;0.000000;0.000000];sys[99.754601%;0.000000%;116;0;342];iodepth[100.0%;0.0%;0.0%;0.0%;0.0%;0.0%;0.0%];iolat[0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%];

The upside is that it should be easier to parse, and it's even humanly readable to a much greater extent than the current format. But let me know what you think.

Quoted 35 lines · Jens Axboe

Quoted 23 lines · Jens Axboe

Quoted 8 lines · Jens Axboe

Quoted 6 lines · Vivek Goyal

Hi,

I was running latest fio and noticed that number of fields in fio --minimal output have gone up from 69 to 77. A increase of 8 fields. Don't see any update in --minimal documentation. Is it regarding total latency thing?

Quoted 8 lines

Woops yes, there's a total latency in there as well now. Should just be 4 extra fields, though. It gets logged after completion latency, but before bandwidth stats. I'll update the documentation.

Should we perhaps put a versioning field in there? Now would seem to be a good time, since the output has changed anyway. I'm open to suggestions from you or other terse output users.

How about redesigning it a bit to make it more bullet proof... We could prefix series of fields with the value they are logging. So for instance, the 4 completion latency fields would include a clat prefix first:

Quoted 6 lines

clat[%lu;%lu;%f;%f],foo[%lu;%lu],etc

Would that not be more resilient to future changes? New fields would not bother you, and reordering should also be fine.

Any other ideas?

With that change, the output would be modified from:

file;0;0;131072;356015;377;0;0;0.000000;0.000000;0;0;0.000000;0.000000;0;0;0.000000;0.000000;0;0;0.000000%;0.000000;0.000000;131072;303660;442;0;0;0.000000;0.000000;0;0;0.000000;0.000000;0;0;0.000000;0.000000;0;0;0.000000%;0.000000;0.000000;99.265606%;0.367197%;95;0;343;100.0%;0.0%;0.0%;0.0%;0.0%;0.0%;0.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%

Quoted 7 lines

to

id[file;0;0];overview[131072;357913;375];slat[0;0;0.000000;0.000000];clat[0;0;0.000000;0.000000];lat[0;0;0.000000;0.000000];bw[0;0;0.000000%;0.000000;0.000000];overview[131072;304348;441];slat[0;0;0.000000;0.000000];clat[0;0;0.000000;0.000000];lat[0;0;0.000000;0.000000];bw[0;0;0.000000%;0.000000;0.000000];sys[99.754601%;0.000000%;116;0;342];iodepth[100.0%;0.0%;0.0%;0.0%;0.0%;0.0%;0.0%];iolat[0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%];

The upside is that it should be easier to parse, and it's even humanly readable to a much greater extent than the current format. But let me know what you think.

Hi Jens,

I have a very simple awk script which looks for bw and max clat fields. I can definitely enhance it to parse this new format.

Above will be broken if you decide the change the name of existing field or try to introduce more stats in the existing field. Say some thing additional in "overview" field.

Personally I would prefer to version the fio and change the version whenever something significant like this happen. Then I can change my parsing method based on version.

I think irrespective of the format of the string, versioning fio is probably a good idea.

May be we can also provide this new format of output with a new fio option say, "fio --terse".

At the end of the day, I will just adjust my scripts based on whatever format you decide to keep. :-)

Quoted 57 lines · Vivek Goyal

Quoted 35 lines · Jens Axboe

Quoted 23 lines · Jens Axboe

Quoted 8 lines · Jens Axboe

Quoted 6 lines · Vivek Goyal

Hi,

I was running latest fio and noticed that number of fields in fio --minimal output have gone up from 69 to 77. A increase of 8 fields. Don't see any update in --minimal documentation. Is it regarding total latency thing?

Quoted 8 lines

Woops yes, there's a total latency in there as well now. Should just be 4 extra fields, though. It gets logged after completion latency, but before bandwidth stats. I'll update the documentation.

Should we perhaps put a versioning field in there? Now would seem to be a good time, since the output has changed anyway. I'm open to suggestions from you or other terse output users.

How about redesigning it a bit to make it more bullet proof... We could prefix series of fields with the value they are logging. So for instance, the 4 completion latency fields would include a clat prefix first:

Quoted 6 lines

clat[%lu;%lu;%f;%f],foo[%lu;%lu],etc

Would that not be more resilient to future changes? New fields would not bother you, and reordering should also be fine.

Any other ideas?

With that change, the output would be modified from:

file;0;0;131072;356015;377;0;0;0.000000;0.000000;0;0;0.000000;0.000000;0;0;0.000000;0.000000;0;0;0.000000%;0.000000;0.000000;131072;303660;442;0;0;0.000000;0.000000;0;0;0.000000;0.000000;0;0;0.000000;0.000000;0;0;0.000000%;0.000000;0.000000;99.265606%;0.367197%;95;0;343;100.0%;0.0%;0.0%;0.0%;0.0%;0.0%;0.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%

Quoted 7 lines

to

id[file;0;0];overview[131072;357913;375];slat[0;0;0.000000;0.000000];clat[0;0;0.000000;0.000000];lat[0;0;0.000000;0.000000];bw[0;0;0.000000%;0.000000;0.000000];overview[131072;304348;441];slat[0;0;0.000000;0.000000];clat[0;0;0.000000;0.000000];lat[0;0;0.000000;0.000000];bw[0;0;0.000000%;0.000000;0.000000];sys[99.754601%;0.000000%;116;0;342];iodepth[100.0%;0.0%;0.0%;0.0%;0.0%;0.0%;0.0%];iolat[0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%];

The upside is that it should be easier to parse, and it's even humanly readable to a much greater extent than the current format. But let me know what you think.

Hi Jens,

I have a very simple awk script which looks for bw and max clat fields. I can definitely enhance it to parse this new format.

Above will be broken if you decide the change the name of existing field or try to introduce more stats in the existing field. Say some thing additional in "overview" field.

Personally I would prefer to version the fio and change the version whenever something significant like this happen. Then I can change my parsing method based on version.

Quoted 5 lines

I think irrespective of the format of the string, versioning fio is probably a good idea.

May be we can also provide this new format of output with a new fio option say, "fio --terse".

I don't think adding a new command line parameter for that will make a lot of sense, only if I revert the offending commit and then add the parameter when readding it.

So lets break it and add the version number up front, if that is what you prefer. If that is easy for you to handle, then I'm guessing it will be similar for others that use it.

Quoted 2 lines

At the end of the day, I will just adjust my scripts based on whatever format you decide to keep. :-)

Thanks :-)

http://git.kernel.dk/?p=fio.git;a=commit;h=525c2bfabdb7e0093a8775a09ad3e772d962760e

It's new prefixed with version 2, and I updated the documentation and man page as well on both the new format and the version field.

Oh, and you were right about the 8 extra fields of course, it's 4 added fields but it's per data direction.