fio problems

Hi,

we try to run fio against raw RBD devices on a ceph cluster. Here are the parameter we use to run fio:

[test]
filename=/dev/rbd0
time_based
runtime=3h
rw=randrw
rwmixread=80
rwmixwrite=20
blocksize_range=4k-1024k
ioengine=libaio
iodepth=64
direct=1
verify=crc32c-intel
verify_fatal=1
verify_async=16
lockfile=exclusive

We see some problems:

1) fio reports higher READ speeds than the used 1Gbit network interface could provide (the interface isn't even working to capacity in this situation). That happens if we use --rwmixread/rwmixwrite:

READ: io=91560MB, aggrb=156206KB/s, minb=159955KB/s,

        maxb=159955KB/s, mint=600216msec, maxt=600216msec
 WRITE: io=22944MB, aggrb=39143KB/s, minb=40083KB/s, maxb=40083KB/s,
        mint=600216msec, maxt=600216msec

2) fio 2.0.6 and also the current git tends to fail with this error:

   fio 2.0.6
   Starting 1 process
   fio: iolog.c:242: log_io_piece: Assertion `ipo->len == __ipo->len'
   failed.:08m:31s]

It seems that we can prevent it when we use the --size parameter, but we would like to prevent the usage of the parameter, if possible

3) fio always reports minb == maxb and aggrb < minb as you can see above. From my understanding this would be false info.

4) If we define rw=randrw, rwmixread=80, rwmixwrite=20 and the verify options, what does the READ process if there isn't enough data written out yet? Which data does READ read and verify?

Quoted 9 lines

2) fio 2.0.6 and also the current git tends to fail with this error:

   fio 2.0.6
   Starting 1 process
   fio: iolog.c:242: log_io_piece: Assertion `ipo->len == __ipo->len'
   failed.:08m:31s]

It seems that we can prevent it when we use the --size parameter, but we would like to prevent the usage of the parameter, if possible

Strangely, I had two reports of this today. Did a bit of digging, and it's due to this commit:

commit c04e4661e4da3b6079f415897e4507cf8e610c54

Author: Daniel Ehrenberg <[email protected]>
Date:   Fri Mar 16 18:54:15 2012 +0100

time_based: Avoid restarting main I/O loop

I think what is going on here is that your job is time based, and when we fail getting a new offset, we clear the offset bitmap. But we could have verifies pending at this point, we'd need to prune this out as well.

Quoted 18 lines

Quoted 9 lines

2) fio 2.0.6 and also the current git tends to fail with this error:

   fio 2.0.6
   Starting 1 process
   fio: iolog.c:242: log_io_piece: Assertion `ipo->len == __ipo->len'
   failed.:08m:31s]

It seems that we can prevent it when we use the --size parameter, but we would like to prevent the usage of the parameter, if possible

Strangely, I had two reports of this today. Did a bit of digging, and it's due to this commit:

commit c04e4661e4da3b6079f415897e4507cf8e610c54

Author: Daniel Ehrenberg <[email protected]>
Date:   Fri Mar 16 18:54:15 2012 +0100

time_based: Avoid restarting main I/O loop

Quoted 4 lines

I think what is going on here is that your job is time based, and when we fail getting a new offset, we clear the offset bitmap. But we could have verifies pending at this point, we'd need to prune this out as well.

Actually, since it happens so rarely, I think the better approach is just to ignore it. I've updated fio to reflect that.

Am 04.04.2012 22:15, schrieb Jens Axboe:

Quoted 25 lines · Jens Axboe

Quoted 23 lines

Quoted 9 lines

2) fio 2.0.6 and also the current git tends to fail with this error:

   fio 2.0.6
   Starting 1 process
   fio: iolog.c:242: log_io_piece: Assertion `ipo->len == __ipo->len'
   failed.:08m:31s]

It seems that we can prevent it when we use the --size parameter, but we would like to prevent the usage of the parameter, if possible

Strangely, I had two reports of this today. Did a bit of digging, and it's due to this commit:

commit c04e4661e4da3b6079f415897e4507cf8e610c54

Author: Daniel Ehrenberg <[email protected]>
Date:   Fri Mar 16 18:54:15 2012 +0100

time_based: Avoid restarting main I/O loop

I think what is going on here is that your job is time based, and when we fail getting a new offset, we clear the offset bitmap. But we could have verifies pending at this point, we'd need to prune this out as well.

Quoted 2 lines

Actually, since it happens so rarely, I think the better approach is just to ignore it. I've updated fio to reflect that.

Btw. I can suppress the error if I add --size with a high number e.g. --size=1p.

Quoted 26 lines

Hi,

we try to run fio against raw RBD devices on a ceph cluster. Here are the parameter we use to run fio:

[test]
filename=/dev/rbd0
time_based
runtime=3h
rw=randrw
rwmixread=80
rwmixwrite=20
blocksize_range=4k-1024k
ioengine=libaio
iodepth=64
direct=1
verify=crc32c-intel
verify_fatal=1
verify_async=16
lockfile=exclusive

We see some problems:

1) fio reports higher READ speeds than the used 1Gbit network interface could provide (the interface isn't even working to capacity in this situation). That happens if we use --rwmixread/rwmixwrite:

Quoted 4 lines

READ: io=91560MB, aggrb=156206KB/s, minb=159955KB/s,

        maxb=159955KB/s, mint=600216msec, maxt=600216msec
 WRITE: io=22944MB, aggrb=39143KB/s, minb=40083KB/s, maxb=40083KB/s,
        mint=600216msec, maxt=600216msec

OK that's very odd. How are these numbers matching up with watching vmstat 1 while the job runs? I've never seen fio over-report before.

Quoted 2 lines

3) fio always reports minb == maxb and aggrb < minb as you can see above. From my understanding this would be false info.

Yeah, it's a cosmetic thing. I've never really taken the time to look at that...

Quoted 3 lines

4) If we define rw=randrw, rwmixread=80, rwmixwrite=20 and the verify options, what does the READ process if there isn't enough data written out yet? Which data does READ read and verify?

If you have a split workload like that and are also doing verifies, the read can be either a read verify or just an offset read. In the latter case, it doesn't verify anything. It'll just read a block (or multiple blocks) at some offset instead of writing it.

Quoted 5 lines

Quoted 2 lines

3) fio always reports minb == maxb and aggrb < minb as you can see above. From my understanding this would be false info.

Yeah, it's a cosmetic thing. I've never really taken the time to look at that...

It's not cosmetic if you need correct values for your tests ...

Quoted 8 lines

Quoted 3 lines

4) If we define rw=randrw, rwmixread=80, rwmixwrite=20 and the verify options, what does the READ process if there isn't enough data written out yet? Which data does READ read and verify?

If you have a split workload like that and are also doing verifies, the read can be either a read verify or just an offset read. In the latter case, it doesn't verify anything. It'll just read a block (or multiple blocks) at some offset instead of writing it.

That would may explain the fio over-report in 1).

We create a new RBD on the cluster, mount it on the client and let fio start it's test on the RBD. The RBD is completely empty at this moment. Not sure what happens if fio reads some random offset that may isn't filled with data at this moment. Or is fio only reading from a prior written block/offset?

Quoted 7 lines

Quoted 5 lines

Quoted 2 lines

3) fio always reports minb == maxb and aggrb < minb as you can see above. From my understanding this would be false info.

Yeah, it's a cosmetic thing. I've never really taken the time to look at that...

It's not cosmetic if you need correct values for your tests ...

Most people don't use the group reporting, but yes of course it should be correct. I have checked in a patch to correct it. The min/max group report bandwidth were off by 1.024, the aggregate was correct (and matched the "real" data output).

Quoted 9 lines

Quoted 8 lines

Quoted 3 lines

4) If we define rw=randrw, rwmixread=80, rwmixwrite=20 and the verify options, what does the READ process if there isn't enough data written out yet? Which data does READ read and verify?

If you have a split workload like that and are also doing verifies, the read can be either a read verify or just an offset read. In the latter case, it doesn't verify anything. It'll just read a block (or multiple blocks) at some offset instead of writing it.

Quoted 7 lines

That would may explain the fio over-report in 1).

We create a new RBD on the cluster, mount it on the client and let fio start it's test on the RBD. The RBD is completely empty at this moment. Not sure what happens if fio reads some random offset that may isn't filled with data at this moment. Or is fio only reading from a prior written block/offset?

OK, if we back up a bit. If you have a workload that is rwmixread=50, hence doing equal amount of random reads and writes. Then the job will read 50% of the disk, and write 50% of the disk. None of the read and write offsets will be the same. So if the drive is initially empty, you are reading non-initialized data.

Same thing is true for your workload. So some of the reads you see will be the verifies, but some of them will also be for parts of the disk that you will then never write.

Quoted 12 lines

Quoted 5 lines

We create a new RBD on the cluster, mount it on the client and let fio start it's test on the RBD. The RBD is completely empty at this moment. Not sure what happens if fio reads some random offset that may isn't filled with data at this moment. Or is fio only reading from a prior written block/offset?

OK, if we back up a bit. If you have a workload that is rwmixread=50, hence doing equal amount of random reads and writes. Then the job will read 50% of the disk, and write 50% of the disk. None of the read and write offsets will be the same. So if the drive is initially empty, you are reading non-initialized data.

Quoted 3 lines

Same thing is true for your workload. So some of the reads you see will be the verifies, but some of them will also be for parts of the disk that you will then never write.

We solved it now by running an initial fio write over the full RBD before running any verify tests:

 fio --name=fill-rbd --filename=/dev/rbd1 --direct=1 --rw=write \
     --bs=4M --size=25g --ioengine=libaio --iodepth=128

Now we don't see problems with over-reporting fio anymore.

Most people don't use the group reporting

Quoted 1 line · Jens Axboe

Most people don't use the group reporting

Why do you say that?

For example I'm trying to find those parameters (direct-io, fadvise, number of threads, libaio/mmap/sync, etc.) that give an overall "best performance" (whatever that means) on a simulated workload.

I'm using group reporting, because I want numbers for the entire system, not for an individual thread.

Am I using "fio" wrong or is "fio" mostly used for other kinds of benchmarking?