suggestion for fio --client to remove garbled output
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 30 lines
This patch makes fio --client JSON output work better when many server threads are used by removing interleaved output from fio servers with --output-format=json .
Background: Red Hat is testing large numbers of OpenStack guests with Cinder block devices backed by Ceph storage using fio. Originally we were using ssh to start independent fio processes on the guests, but we ran into problems with threads not starting and stopping at the same time, despite
using options to use fixed elapsed time for all threads. But when we
started using --client, we noticed that thread time skew is greatly reduced, because fio is *already running* on the server and we only have to connect
to it and start the test. This is great, but for large numbers of fio
servers, it becomes impossible to parse the output.For example, here's a sample of per-VM JSON output:
Disk stats (read/write):
<vm4> {<vm4> "fio version" : "fio-2.2.4", <vm4> "timestamp" : 1421677173, <vm4> "time" : "Mon Jan 19 09:19:33 2015", <vm4> "jobs" : [<vm4>
<vm4> ],
<vm4> "disk_util" : [
...
<vm4> ]
<vm4> }
Quoted 6 lines
https://s3.amazonaws.com/ben.england/client.c.diff
https://s3.amazonaws.com/ben.england/fio-results.2015-01-20-11-50.json.gzThe above output works fine for a few threads and is not hard to filter out, but the various guests' output sections get mixed together for large thread counts such as 256 client threads.
I've had the same issue when testing lots of VMs. At about 32 VMs (32 fio servers), the <tag> no longer adequately differentiate one server's output from another.
Quoted 17 lines
if you look at the previous URL
fio-results*json.gz containing garbled output, the cause of this output is thehandle_text()routine, which is called by fio whenever aFIO_NET_CMD_TEXTPDU is received by the fio client (infio_handle_client()). If you use --debug=all you can see that these PDUs are processed in whatever order they are received from the fio server processes, essentially in random order. But almost all of this info is in the final json object output at the end of the file (except for the timestamp field which I wish was there!) so there is no reason to output twice, except as a debugging tool perhaps. The patch is to V2.2.4.[ben@localhost fio-master]$ diff -u client.{c.sav2,c}
client.c renamed from client.c.sav2 +0
-0@@ -968,12 +968,6 @@const char *name; int fio_unused ret;
client.c line 973
Quoted 10 lines
client.c renamed from client.c.sav2 +0
-6- name = client->name ? client->name : client->hostname; - -if (!client->skip_newline) -fprintf (f_out,"<%s> " , name); - ret =fwrite (buf, pdu->buf_len,1 , f_out); -fflush (f_out); client->skip_newline =strchr (buf,'\n' ) ==NULL ; }
I have tested this with 60 fio servers, and it works well for me.
-Andrew