using fio client on 64-bit machine with fio server on 32-bit machine

DISCUSSION Frank D'Arcy 2 participants 7 messages Jun 27, 2014 – Jul 1, 2014

Share & tools

Frank D'Arcy

author

2014-06-27 07:00 UTC #0 lore raw

Hi,
I've been playing around with fio (which is really great thanks)
with a view to testing hard disks on set-top boxes (which I don't have yet) and as prep for this I created a test environment consisting of
2 x virtual machines running 32-bit ubuntu (using "hashicorp/precise32" from vagrant cloud) running on a 64-bit machine with ubuntu 12.04.

I started fio (v2.1.10) running in server mode on both virtual machines as follows

$ fio --server=,8880

and launched 2 fio (v2.1.10) instances in client mode on the host 64-bit machine,

$ fio --client=ip:127.0.0.1,8882 test.fio
$ fio --client=ip:127.0.0.1,8881 test.fio
where ports 8881 and 8882 are forwarded to port 8880 on each VM

The sample test.fio (below) runs for 25 seconds but the values printed by the clients look garbled. eg. runtime is a huge number ("runt= 107374182400000msec")

Same issue is seen when I switched to using latest fio from "git clone" and the same issue is seen with all output formats (json, terse, normal)

I changed one of the virtual machines to 64-bit ubuntu (using "hashicorp/precise64" from vagrant cloud) and the client talking to the fio server on that VM gave the correct results (eg. "runt= 25001msec") while the client talking to the 32-bit ubuntu VM continued to show garbled values.

So it looks like support is not there for client-server setups on non-matching architectures ? eg. 32-bit client to 64-bit server in this case,
and perhaps endianess differences would be a problem too? ( but I haven't tried that )

Apologies if I've missed this issue raised elsewhere, I did a quick search and found no mention of it. It's not critical for us, as the simple workaround is to run fio standalone on each set-top box outputting results to a file and pulling the file back to a central machine for parsing. Just wanted to raise it if its unknown to you.

Thanks very much,
Frank

my test ....

[global]
;run for 25 seconds
runtime=25
time_based=1
;directory in which to create files
directory=.
size=1024k
log_avg_msec=500

[job1]
name=job1_stb1
description=Job1 Description
rw=read
blocksize=2
rate=1k
write_bw_log=job1_stb1
write_lat_log=job1_stb1
write_iops_log=job1_stb1

[job2]
name=job2_stb1
description=Job2 Description
rw=write
blocksize=2
rate=500
write_bw_log=job2_stb1
write_lat_log=job2_stb1
write_iops_log=job2_stb1

Signature · 6 lines

Frank D'Arcy
S3 Group Ltd.   South County Business Park, Leopardstown, Dublin 18, Ireland
Visit S3 website: www.s3group.com
-----

The information contained in this e-mail and in any attachments is confidential and is designated solely for the attention of the intended recipient(s). If you are not an intended recipient, you must not use, disclose, copy, distribute or retain this e-mail or any part thereof. If you have received this e-mail in error, please notify the sender by return e-mail and delete all copies of this e-mail from your computer system(s). Please direct any additional queries to: [email protected]. Thank You. Silicon and Software Systems Limited (S3 Group). Registered in Ireland no. 378073. Registered Office: South County Business Park, Leopardstown, Dublin 18.

Jens Axboe

2014-06-27 16:47 UTC #1 lore raw

Quoted 38 lines · Frank D'Arcy

Hi,
I've been playing around with fio (which is really great thanks) with a view to testing hard disks on set-top boxes (which I don't have yet) and as prep for this I created a test environment consisting of 2 x virtual machines running 32-bit ubuntu (using "hashicorp/precise32" from vagrant cloud)
running on a 64-bit machine with ubuntu 12.04.
I started fio (v2.1.10) running in server mode on both virtual machines as follows
$ fio --server=,8880
and launched 2 fio (v2.1.10) instances in client mode on the host 64-bit machine,
$ fio --client=ip:127.0.0.1,8882 test.fio
$ fio --client=ip:127.0.0.1,8881 test.fio
where ports 8881 and 8882 are forwarded to port 8880 on each VM
The sample test.fio (below) runs for 25 seconds but the values printed by the clients look garbled. eg. runtime is a huge number ("runt= 107374182400000msec")
Same issue is seen when I switched to using latest fio from "git clone" and the same issue is seen with all output formats (json, terse, normal)
I changed one of the virtual machines to 64-bit ubuntu (using "hashicorp/precise64" from vagrant cloud)
and the client talking to the fio server on that VM gave the correct results (eg. "runt= 25001msec")
while the client talking to the 32-bit ubuntu VM continued to show garbled values.
So it looks like support is not there for client-server setups on non-matching architectures ?
eg. 32-bit client to 64-bit server in this case,
and perhaps endianess differences would be a problem too? ( but I haven't tried that )

Quoted 7 lines

Apologies if I've missed this issue raised elsewhere, I did a quick search and found no mention of it.
It's not critical for us, as the simple workaround is to run fio standalone on each set-top box
outputting results to a file and pulling the file back to a central machine for parsing.
Just wanted to raise it if its unknown to you.

Any mix of 32 vs 64-bit and big vs little endian should work, all the data structures are sanitized across the wire. So if it doesn't work, then that is surely a bug! I'll take a look at this.

Signature · 1 line

Jens Axboe

Jens Axboe

2014-06-27 21:02 UTC #2 lore raw

Quoted 48 lines · Jens Axboe

Quoted 39 lines · Frank D'Arcy
Hi,
I've been playing around with fio (which is really great thanks) with a view to testing hard disks on set-top boxes (which I don't have yet)
and as prep for this I created a test environment consisting of 2 x virtual machines running 32-bit ubuntu (using "hashicorp/precise32" from vagrant cloud)
running on a 64-bit machine with ubuntu 12.04.
I started fio (v2.1.10) running in server mode on both virtual machines as follows
$ fio --server=,8880
and launched 2 fio (v2.1.10) instances in client mode on the host 64-bit machine,
$ fio --client=ip:127.0.0.1,8882 test.fio
$ fio --client=ip:127.0.0.1,8881 test.fio
where ports 8881 and 8882 are forwarded to port 8880 on each VM
The sample test.fio (below) runs for 25 seconds but the values printed by the clients look garbled. eg. runtime is a huge number ("runt= 107374182400000msec")
Same issue is seen when I switched to using latest fio from "git clone" and the same issue is seen with all output formats (json, terse, normal)
I changed one of the virtual machines to 64-bit ubuntu (using "hashicorp/precise64" from vagrant cloud)
and the client talking to the fio server on that VM gave the correct results (eg. "runt= 25001msec")
while the client talking to the 32-bit ubuntu VM continued to show garbled values.
So it looks like support is not there for client-server setups on non-matching architectures ?
eg. 32-bit client to 64-bit server in this case,
and perhaps endianess differences would be a problem too? ( but I haven't tried that )
Quoted 7 lines
Apologies if I've missed this issue raised elsewhere, I did a quick search and found no mention of it.
It's not critical for us, as the simple workaround is to run fio standalone on each set-top box
outputting results to a file and pulling the file back to a central machine for parsing.
Just wanted to raise it if its unknown to you.

Quoted 3 lines

Any mix of 32 vs 64-bit and big vs little endian should work, all the data structures are sanitized across the wire. So if it doesn't work, then that is surely a bug! I'll take a look at this.

http://git.kernel.dk/?p=fio.git;a=commit;h=819db5f5009742975cfdd036440522bd45b86fe7

this should make it behave better. Let me know if there are still issues with 32 vs 64 bit client/server.

Signature · 1 line

Jens Axboe

Frank D'Arcy

author

2014-06-30 11:09 UTC #3 lore raw

Hi Jens,
thanks for getting back to me.
I've rebuilt with the latest but I still see problems with the numbers.

I do see one improvement,
the "% done" printed while the test is in progress now increases nicely from 0% to 100% whereas before it would jump around to seemingly random values between 0% to 100%

I wonder is it picking the wrong set of numbers to print in some cases.
eg. from a test with 2 jobs that runs for 23 seconds I faound the following in normal output ...

when run standalone on a 32-bit VM I see this for a "rw=randread" job

Quoted 1 line

read : io=47106B, bw=2047B/s, iops=1023, runt= 23001msec

whereas when the same test is run from 64-bit client to 32-bit server I see this for the same job

Quoted 1 line

trim: io=98789GB, bw=1000/s, iops=0, runt=98784247808000msec

so it says "trim" where it should say "read"

and for the 2nd job which is "rw=write"
when run standalone on a 32-bit VM I see this

Quoted 1 line

write: io=34502B, bw=1500B/s, iops=750, runt= 23001msec

whereas when the same test is run from 64-bit client to 32-bit server I see this for the same job

Quoted 1 line

read : io=148185GB, bw=1500/s, iops=750, runt=98788542775296msec

so it says "read" where it should say "write"

In the same test run, the following results are identical when run both ways ...

Quoted 6 lines

Run status group 0 (all jobs):
   READ: io=46KB, aggrb=1KB/s, minb=1KB/s, maxb=1KB/s, mint=23001msec, 
maxt=23001msec
WRITE: io=33KB, aggrb=1KB/s, minb=1KB/s, maxb=1KB/s, mint=23001msec,
maxt=23001msec

so these results look correct.

All the above problems are only seen when run from 64-bit client to 32-bit server, no problems seen when running standalone,
and no problems seen when running 64-bit client to 64-bit server.

Thanks,
Frank

Quoted 56 lines · Jens Axboe

Quoted 47 lines · Jens Axboe
Quoted 39 lines · Frank D'Arcy
Hi,
I've been playing around with fio (which is really great thanks) with a view to testing hard disks on set-top boxes (which I don't have yet)
and as prep for this I created a test environment consisting of 2 x virtual machines running 32-bit ubuntu (using "hashicorp/precise32" from vagrant cloud)
running on a 64-bit machine with ubuntu 12.04.
I started fio (v2.1.10) running in server mode on both virtual machines as follows
$ fio --server=,8880
and launched 2 fio (v2.1.10) instances in client mode on the host 64-bit machine,
$ fio --client=ip:127.0.0.1,8882 test.fio
$ fio --client=ip:127.0.0.1,8881 test.fio
where ports 8881 and 8882 are forwarded to port 8880 on each VM
The sample test.fio (below) runs for 25 seconds but the values printed by the clients look garbled. eg. runtime is a huge number ("runt= 107374182400000msec")
Same issue is seen when I switched to using latest fio from "git clone" and the same issue is seen with all output formats (json, terse, normal)
I changed one of the virtual machines to 64-bit ubuntu (using "hashicorp/precise64" from vagrant cloud)
and the client talking to the fio server on that VM gave the correct results (eg. "runt= 25001msec")
while the client talking to the 32-bit ubuntu VM continued to show garbled values.
So it looks like support is not there for client-server setups on non-matching architectures ?
eg. 32-bit client to 64-bit server in this case,
and perhaps endianess differences would be a problem too? ( but I haven't tried that )
Quoted 7 lines
Apologies if I've missed this issue raised elsewhere, I did a quick search and found no mention of it.
It's not critical for us, as the simple workaround is to run fio standalone on each set-top box
outputting results to a file and pulling the file back to a central machine for parsing.
Just wanted to raise it if its unknown to you.
Quoted 3 lines
Any mix of 32 vs 64-bit and big vs little endian should work, all the data structures are sanitized across the wire. So if it doesn't work, then that is surely a bug! I'll take a look at this.
http://git.kernel.dk/?p=fio.git;a=commit;h=819db5f5009742975cfdd036440522bd45b86fe7
this should make it behave better. Let me know if there are still issues with 32 vs 64 bit client/server.

Signature · 6 lines

Frank D'Arcy
www.s3group.com

-----

The information contained in this e-mail and in any attachments is confidential and is designated solely for the attention of the intended recipient(s). If you are not an intended recipient, you must not use, disclose, copy, distribute or retain this e-mail or any part thereof. If you have received this e-mail in error, please notify the sender by return e-mail and delete all copies of this e-mail from your computer system(s). Please direct any additional queries to: [email protected]. Thank You. Silicon and Software Systems Limited (S3 Group). Registered in Ireland no. 378073. Registered Office: South County Business Park, Leopardstown, Dublin 18.

Jens Axboe

2014-06-30 14:52 UTC #4 lore raw

Quoted 38 lines · Frank D'Arcy

Hi Jens,
thanks for getting back to me.
I've rebuilt with the latest but I still see problems with the numbers.
I do see one improvement,
the "% done" printed while the test is in progress now increases nicely from 0% to 100%
whereas before it would jump around to seemingly random values between 0% to 100%
I wonder is it picking the wrong set of numbers to print in some cases. eg. from a test with 2 jobs that runs for 23 seconds I faound the following in normal output ...
when run standalone on a 32-bit VM I see this for a "rw=randread" job
Quoted 1 line
read : io=47106B, bw=2047B/s, iops=1023, runt= 23001msec
whereas when the same test is run from 64-bit client to 32-bit server I see this for the same job
Quoted 1 line
trim: io=98789GB, bw=1000/s, iops=0, runt=98784247808000msec
so it says "trim" where it should say "read"
and for the 2nd job which is "rw=write"
when run standalone on a 32-bit VM I see this
Quoted 1 line
write: io=34502B, bw=1500B/s, iops=750, runt= 23001msec
whereas when the same test is run from 64-bit client to 32-bit server I see this for the same job
Quoted 1 line
read : io=148185GB, bw=1500/s, iops=750, runt=98788542775296msec
so it says "read" where it should say "write"
In the same test run, the following results are identical when run both ways ...
Quoted 6 lines
Run status group 0 (all jobs):
   READ: io=46KB, aggrb=1KB/s, minb=1KB/s, maxb=1KB/s, mint=23001msec,
maxt=23001msec
WRITE: io=33KB, aggrb=1KB/s, minb=1KB/s, maxb=1KB/s, mint=23001msec,
maxt=23001msec

Quoted 6 lines

so these results look correct.
All the above problems are only seen when run from 64-bit client to 32-bit server,
no problems seen when running standalone,
and no problems seen when running 64-bit client to 64-bit server.

Try updating again. Not sure I caught all of them, so would be nice to know how it works now.

Signature · 1 line

Jens Axboe

Frank D'Arcy

author

2014-06-30 17:06 UTC #5 lore raw

Problem solved!
All results looks good to me.
Thank you very much,
Frank

Quoted 46 lines · Jens Axboe

Quoted 36 lines · Frank D'Arcy
Hi Jens,
thanks for getting back to me.
I've rebuilt with the latest but I still see problems with the numbers.
I do see one improvement,
the "% done" printed while the test is in progress now increases nicely from 0% to 100%
whereas before it would jump around to seemingly random values between 0% to 100%
I wonder is it picking the wrong set of numbers to print in some cases. eg. from a test with 2 jobs that runs for 23 seconds I faound the following in normal output ...
when run standalone on a 32-bit VM I see this for a "rw=randread" job
Quoted 1 line
   read : io=47106B, bw=2047B/s, iops=1023, runt= 23001msec
whereas when the same test is run from 64-bit client to 32-bit server I see this for the same job
Quoted 1 line
   trim: io=98789GB, bw=1000/s, iops=0, runt=98784247808000msec
so it says "trim" where it should say "read"
and for the 2nd job which is "rw=write"
when run standalone on a 32-bit VM I see this
Quoted 1 line
   write: io=34502B, bw=1500B/s, iops=750, runt= 23001msec
whereas when the same test is run from 64-bit client to 32-bit server I see this for the same job
Quoted 1 line
   read : io=148185GB, bw=1500/s, iops=750, runt=98788542775296msec
so it says "read" where it should say "write"
In the same test run, the following results are identical when run both ways ...
Quoted 5 lines
Run status group 0 (all jobs):
    READ: io=46KB, aggrb=1KB/s, minb=1KB/s, maxb=1KB/s, mint=23001msec,
maxt=23001msec
WRITE: io=33KB, aggrb=1KB/s, minb=1KB/s, maxb=1KB/s, mint=23001msec,
maxt=23001msec
Quoted 6 lines
so these results look correct.
All the above problems are only seen when run from 64-bit client to 32-bit server,
no problems seen when running standalone,
and no problems seen when running 64-bit client to 64-bit server.
Try updating again. Not sure I caught all of them, so would be nice to know how it works now.

Signature · 6 lines

Frank D'Arcy
www.s3group.com

-----

The information contained in this e-mail and in any attachments is confidential and is designated solely for the attention of the intended recipient(s). If you are not an intended recipient, you must not use, disclose, copy, distribute or retain this e-mail or any part thereof. If you have received this e-mail in error, please notify the sender by return e-mail and delete all copies of this e-mail from your computer system(s). Please direct any additional queries to: [email protected]. Thank You. Silicon and Software Systems Limited (S3 Group). Registered in Ireland no. 378073. Registered Office: South County Business Park, Leopardstown, Dublin 18.

Jens Axboe

2014-07-01 02:23 UTC #6 lore raw

Quoted 4 lines · Frank D'Arcy

Problem solved!
All results looks good to me.
Thank you very much,

Excellent, thanks for reporting and testing.

Signature · 1 line

Jens Axboe