How to perform stride access?

Question Akira Hayakawa 4 participants 19 messages Sep 23, 2014 – Oct 5, 2014 1 message not in the archive

lore

Share & tools

lore

Akira Hayakawa

author

2014-09-23 13:47 UTC #0 lore raw

Hi,

I want to perform stride write access to a block device but I don't have a clue how I can do that.

What I want to do is to perform a stride access that
each write size is 1 sector and 7 sectors are apart between each writes. (i.e. Only the first sector of each 4KB block)

For example,
0, 8, 16, 24, 32, ...

And, it repeat over the device until certain amount of writes are accomplished. In my case, amount of 32MB to 508KB device.

I consider the command below works like as I want but it doesn't actually.
Instead, it looks performing ordinary 512KB seq write.
fio --name=test --filename=#{dev.path} --rw=write --ioengine=libaio --direct=1 --io_limit=32M --size=100% --ba=4k --bs=512

My questions are:
1) How to perform stride write access in fio?
2) If fio is not a appropriate tool for this purpose, easy to fix? Or do you recommend other tool?

- Akira

Andrey Kuzmin

2014-09-23 14:05 UTC #1 lore raw

Offset modifier under rw= should do the trick, consult
https://github.com/axboe/fio/blob/master/HOWTO for details.

Best regards,
Andrey
Regards,
Andrey

Quoted 29 lines · Akira Hayakawa

Hi,
I want to perform stride write access to a block device but I don't have a clue how I can do that.
What I want to do is to perform a stride access that
each write size is 1 sector and 7 sectors are apart between each writes. (i.e. Only the first sector of each 4KB block)
For example,
0, 8, 16, 24, 32, ...
And, it repeat over the device until certain amount of writes are accomplished. In my case, amount of 32MB to 508KB device.
I consider the command below works like as I want but it doesn't actually.
Instead, it looks performing ordinary 512KB seq write.
fio --name=test --filename=#{dev.path} --rw=write --ioengine=libaio --direct=1 --io_limit=32M --size=100% --ba=4k --bs=512
My questions are:
1) How to perform stride write access in fio?
2) If fio is not a appropriate tool for this purpose, easy to fix? Or do you recommend other tool?
- Akira
--
To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Akira Hayakawa

author

2014-09-24 08:35 UTC #2 lore raw

Thanks Andrey,

However, I don't think I still have a problem.

I modified the command

From:

Quoted 1 line

Quoted 1 line
fio --name=test --filename=#{dev.path} --rw=write --ioengine=libaio --direct=1 --io_limit=32M --size=100% --ba=4k --bs=512

To:
fio --name=test --filename=#{dev.path} --rw=write:4k --ioengine=libaio --direct=1 --io_limit=32M --bs=512

The result is the runtime is too short.
I guess fio stops as soon as it reaches the end of the device. However, I want it to repeat over and over again until io_limit is fully consumed.

Note that the device is smaller than 32M (it is only 508B). So, it should repeat more than 60 times.

How can I repeat the workload?

Or,

Building hand-made random map would suffice, I guess.

- Akira

Quoted 39 lines · Andrey Kuzmin

Offset modifier under rw= should do the trick, consult
https://github.com/axboe/fio/blob/master/HOWTO for details.
Best regards,
Andrey
Regards,
Andrey
Quoted 29 lines · Akira Hayakawa
Hi,
I want to perform stride write access to a block device but I don't have a clue how I can do that.
What I want to do is to perform a stride access that
each write size is 1 sector and 7 sectors are apart between each writes. (i.e. Only the first sector of each 4KB block)
For example,
0, 8, 16, 24, 32, ...
And, it repeat over the device until certain amount of writes are accomplished. In my case, amount of 32MB to 508KB device.
I consider the command below works like as I want but it doesn't actually.
Instead, it looks performing ordinary 512KB seq write.
fio --name=test --filename=#{dev.path} --rw=write --ioengine=libaio --direct=1 --io_limit=32M --size=100% --ba=4k --bs=512
My questions are:
1) How to perform stride write access in fio?
2) If fio is not a appropriate tool for this purpose, easy to fix? Or do you recommend other tool?
- Akira
--
To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Akira Hayakawa

author

2014-09-24 09:29 UTC #4 lore raw

In reply to a message not in the archive

Andrey,

Quoted 1 line

You might want to specify iosize or use a time-based run.

Could you explain the difference between bs and iosize? My understanding is bs option specifies the iosize. Do you mean I need to add "--size=32M"?

time-based won't help in my case because I want to see the runtime of 32MB writes consumed.

- Akira

Quoted 76 lines · Andrey Kuzmin

You might want to specify iosize or use a time-based run.
Regards,
Andrey
Quoted 69 lines · On Sep 24, 2014 12:35 PM, "Akira Hayakawa" <[email protected]…
Thanks Andrey,
However, I don't think I still have a problem.
I modified the command
From:
Quoted 1 line
Quoted 1 line
fio --name=test --filename=#{dev.path} --rw=write --ioengine=libaio --direct=1 --io_limit=32M --size=100% --ba=4k --bs=512
To:
fio --name=test --filename=#{dev.path} --rw=write:4k --ioengine=libaio --direct=1 --io_limit=32M --bs=512
The result is the runtime is too short.
I guess fio stops as soon as it reaches the end of the device. However, I want it to repeat over and over again until io_limit is fully consumed.
Note that the device is smaller than 32M (it is only 508B). So, it should repeat more than 60 times.
How can I repeat the workload?
Or,
Building hand-made random map would suffice, I guess.
- Akira
Quoted 39 lines · Andrey Kuzmin
Offset modifier under rw= should do the trick, consult
https://github.com/axboe/fio/blob/master/HOWTO for details.
Best regards,
Andrey
Regards,
Andrey
Quoted 29 lines · On Tue, Sep 23, 2014 at 5:47 PM, Akira Hayakawa <[email protected]…
Hi,
I want to perform stride write access to a block device but I don't have a clue how I can do that.
What I want to do is to perform a stride access that
each write size is 1 sector and 7 sectors are apart between each writes. (i.e. Only the first sector of each 4KB block)
For example,
0, 8, 16, 24, 32, ...
And, it repeat over the device until certain amount of writes are accomplished. In my case, amount of 32MB to 508KB device.
I consider the command below works like as I want but it doesn't actually.
Instead, it looks performing ordinary 512KB seq write.
fio --name=test --filename=#{dev.path} --rw=write --ioengine=libaio --direct=1 --io_limit=32M --size=100% --ba=4k --bs=512
My questions are:
1) How to perform stride write access in fio?
2) If fio is not a appropriate tool for this purpose, easy to fix? Or do you recommend other tool?
- Akira
--
To unsubscribe from this list: send the line "unsubscribe fio" in
the body of a message to [email protected] <mailto:[email protected]> More majordomo info at http://vger.kernel.org/majordomo-info.html

Sitsofe Wheeler

2014-09-24 09:52 UTC #5 lore raw

In reply to Akira Hayakawa

Quoted 11 lines · Akira Hayakawa

However, I [...] think I still have a problem.
I modified the command
From:
Quoted 1 line
Quoted 1 line
fio --name=test --filename=#{dev.path} --rw=write --ioengine=libaio --direct=1 --io_limit=32M --size=100% --ba=4k --bs=512
To:
fio --name=test --filename=#{dev.path} --rw=write:4k --ioengine=libaio --direct=1 --io_limit=32M --bs=512
The result is the runtime is too short.

This looks like a bug. I can reproduce it with 2.1.11-11-gb7f5 too:

dd if=/dev/zero of=/dev/shm/1M bs=1M count=1
fio --bs=4k --rw=write:4k --filename=/dev/shm/1M --stonewall --name=1M --io_limit=1M --name=2M --io_limit=2M
[...]

Run status group 0 (all jobs):
WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec

Run status group 1 (all jobs):
WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec

Why isn't io 1024KB for group 0? Additionally, shouldn't the total io written each group be different? Jens?

Quoted 4 lines

I guess fio stops as soon as it reaches the end of the device. However, I want it to repeat over and over again until io_limit is fully consumed.
Note that the device is smaller than 32M (it is only 508B).

508 bytes? But your block size is 512 bytes! Am I misunderstanding what you're doing?

Quoted 3 lines

So, it should repeat more than 60 times.
How can I repeat the workload?

number_ios fails too and using zonesize/zoneskip also doesn't help. The only thing left that springs to mind is to use loops or fix this bug :-)

Quoted 3 lines

Or,
Building hand-made random map would suffice, I guess.

I'm not sure I follow. The workload you gave above is sequential with holes (--rw=write:4k) - why would we need a random map?

Signature · 1 line

Sitsofe | http://sucs.org/~sits/

Akira Hayakawa

author

2014-09-24 09:58 UTC #6 lore raw

Hi Sistofe,

I am using fio-2.1.12-11-g8fc4

Quoted 4 lines

Quoted 1 line
Note that the device is smaller than 32M (it is only 508B).
508 bytes? But your block size is 512 bytes! Am I misunderstanding what you're doing?

Sorry, I should have written as 512kB. It was a mistake.

- Akira

Quoted 55 lines · Sitsofe Wheeler

Quoted 11 lines · Akira Hayakawa
However, I [...] think I still have a problem.
I modified the command
From:
Quoted 1 line
Quoted 1 line
fio --name=test --filename=#{dev.path} --rw=write --ioengine=libaio --direct=1 --io_limit=32M --size=100% --ba=4k --bs=512
To:
fio --name=test --filename=#{dev.path} --rw=write:4k --ioengine=libaio --direct=1 --io_limit=32M --bs=512
The result is the runtime is too short.
This looks like a bug. I can reproduce it with 2.1.11-11-gb7f5 too:
dd if=/dev/zero of=/dev/shm/1M bs=1M count=1
fio --bs=4k --rw=write:4k --filename=/dev/shm/1M --stonewall --name=1M --io_limit=1M --name=2M --io_limit=2M
[...]
Run status group 0 (all jobs):
WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
Run status group 1 (all jobs):
WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
Why isn't io 1024KB for group 0? Additionally, shouldn't the total io written each group be different? Jens?
Quoted 4 lines
I guess fio stops as soon as it reaches the end of the device. However, I want it to repeat over and over again until io_limit is fully consumed.
Note that the device is smaller than 32M (it is only 508B).
508 bytes? But your block size is 512 bytes! Am I misunderstanding what you're doing?
Quoted 3 lines
So, it should repeat more than 60 times.
How can I repeat the workload?
number_ios fails too and using zonesize/zoneskip also doesn't help. The only thing left that springs to mind is to use loops or fix this bug :-)
Quoted 3 lines
Or,
Building hand-made random map would suffice, I guess.
I'm not sure I follow. The workload you gave above is sequential with holes (--rw=write:4k) - why would we need a random map?

Sitsofe Wheeler

2014-09-24 21:22 UTC #7 lore raw

In reply to Sitsofe Wheeler

(Adding Jens to the CC list)

Quoted 54 lines · Sitsofe Wheeler

Quoted 11 lines · Akira Hayakawa
However, I [...] think I still have a problem.
I modified the command
From:
Quoted 1 line
Quoted 1 line
fio --name=test --filename=#{dev.path} --rw=write --ioengine=libaio --direct=1 --io_limit=32M --size=100% --ba=4k --bs=512
To:
fio --name=test --filename=#{dev.path} --rw=write:4k --ioengine=libaio --direct=1 --io_limit=32M --bs=512
The result is the runtime is too short.
This looks like a bug. I can reproduce it with 2.1.11-11-gb7f5 too:
dd if=/dev/zero of=/dev/shm/1M bs=1M count=1
fio --bs=4k --rw=write:4k --filename=/dev/shm/1M --stonewall --name=1M --io_limit=1M --name=2M --io_limit=2M
[...]
Run status group 0 (all jobs):
WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
Run status group 1 (all jobs):
WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
Why isn't io 1024KB for group 0? Additionally, shouldn't the total io written each group be different? Jens?
Quoted 4 lines
I guess fio stops as soon as it reaches the end of the device. However, I want it to repeat over and over again until io_limit is fully consumed.
Note that the device is smaller than 32M (it is only 508B).
508 bytes? But your block size is 512 bytes! Am I misunderstanding what you're doing?
Quoted 3 lines
So, it should repeat more than 60 times.
How can I repeat the workload?
number_ios fails too and using zonesize/zoneskip also doesn't help. The only thing left that springs to mind is to use loops or fix this bug :-)
Quoted 3 lines
Or,
Building hand-made random map would suffice, I guess.
I'm not sure I follow. The workload you gave above is sequential with holes (--rw=write:4k) - why would we need a random map?

Signature · 1 line

Sitsofe | http://sucs.org/~sits/

Jens Axboe

2014-09-28 02:24 UTC #8 lore raw

Quoted 33 lines · Sitsofe Wheeler

(Adding Jens to the CC list)
Quoted 30 lines · Sitsofe Wheeler
Quoted 11 lines · Akira Hayakawa
However, I [...] think I still have a problem.
I modified the command
From:
Quoted 1 line
Quoted 1 line
fio --name=test --filename=#{dev.path} --rw=write --ioengine=libaio --direct=1 --io_limit=32M --size=100% --ba=4k --bs=512
To:
fio --name=test --filename=#{dev.path} --rw=write:4k --ioengine=libaio --direct=1 --io_limit=32M --bs=512
The result is the runtime is too short.
This looks like a bug. I can reproduce it with 2.1.11-11-gb7f5 too:
dd if=/dev/zero of=/dev/shm/1M bs=1M count=1
fio --bs=4k --rw=write:4k --filename=/dev/shm/1M --stonewall --name=1M --io_limit=1M --name=2M --io_limit=2M
[...]
Run status group 0 (all jobs):
   WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
Run status group 1 (all jobs):
   WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
Why isn't io 1024KB for group 0? Additionally, shouldn't the total io written each group be different? Jens?

You are doing a sequential workload, skipping 4k every time. First write will be to offset 0, next to 8KB, etc. Write 128 would be to 1040384, which is 1MB - 8KB. Hence the next feasible offset after that would be 1MB, which is end of the file. So how could it do more than 512KB of IO? That's 128 * 4KB.

I didn't read the whole thread in detail, just looked at your last example here. And for that one, I don't see anything wrong.

Signature · 1 line

Jens Axboe

Sitsofe Wheeler

2014-09-28 10:32 UTC #9 lore raw

Quoted 17 lines · Jens Axboe

Quoted 17 lines · Sitsofe Wheeler
Quoted 16 lines
This looks like a bug. I can reproduce it with 2.1.11-11-gb7f5 too:
dd if=/dev/zero of=/dev/shm/1M bs=1M count=1
fio --bs=4k --rw=write:4k --filename=/dev/shm/1M --stonewall --name=1M --io_limit=1M --name=2M --io_limit=2M
[...]
Run status group 0 (all jobs):
   WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
Run status group 1 (all jobs):
   WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec

Quoted 5 lines

Quoted 2 lines
Quoted 2 lines
Why isn't io 1024KB for group 0? Additionally, shouldn't the total io written each group be different? Jens?
You are doing a sequential workload, skipping 4k every time. First write will be to offset 0, next to 8KB, etc. Write 128 would be to 1040384,

which

Quoted 1 line

is 1MB - 8KB. Hence the next feasible offset after that would be 1MB,

which

Quoted 5 lines

is end of the file. So how could it do more than 512KB of IO? That's 128 * 4KB.
I didn't read the whole thread in detail, just looked at your last example here. And for that one, I don't see anything wrong.

I guess I would have thought io_limit always forced wraparound. For example:

# dd if=/dev/zero of=/dev/shm/1M bs=1M count=1
# fio --bs=4k --filename=/dev/shm/1M --name=go1 --rw=write
[...]
Run status group 0 (all jobs):
  WRITE: io=1024KB, aggrb=341333KB/s, minb=341333KB/s, maxb=341333KB/s,
mint=3msec, maxt=3msec
# fio --bs=4k --filename=/dev/shm/1M --name=go2 --io_limit=2M --rw=write
Run status group 0 (all jobs):
  WRITE: io=2048KB, aggrb=341333KB/s, minb=341333KB/s, maxb=341333KB/s,
mint=6msec, maxt=6msec
[...]
# fio --bs=4k --filename=/dev/shm/1M --name=go3 --io_limit=2M --rw=write:4k
[...]
  WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
Run status group 0 (all jobs):
# fio --bs=4k --filename=/dev/shm/1M --name=go4 --io_limit=2M --rw=write:4k
[...]
Run status group 0 (all jobs):
  WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec

go2 is a plain sequential job that does twice as much I/O as go1. Given that the size of the file being written to has not changed between the runs one could guess that fio simply wrapped around and started from the first offset (0) to write the second MB of data. Given this isn't it a fair assumption that when doing a skipping workload if io_limit is used (as in go4) and an offset beyond the end of the device is produced the same wraparound behaviour as go2 should occur and the total io done should match that specified in io_limit?

Signature · 1 line

Sitsofe | http://sucs.org/~sits/

Sitsofe Wheeler

2014-09-28 10:36 UTC #10 lore raw

In reply to Jens Axboe

(Resending because first mail had an HTML part)

Quoted 17 lines · Jens Axboe

Quoted 17 lines · Sitsofe Wheeler
Quoted 16 lines
This looks like a bug. I can reproduce it with 2.1.11-11-gb7f5 too:
dd if=/dev/zero of=/dev/shm/1M bs=1M count=1
fio --bs=4k --rw=write:4k --filename=/dev/shm/1M --stonewall --name=1M --io_limit=1M --name=2M --io_limit=2M
[...]
Run status group 0 (all jobs):
   WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
Run status group 1 (all jobs):
   WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec

Quoted 5 lines

Quoted 2 lines
Quoted 2 lines
Why isn't io 1024KB for group 0? Additionally, shouldn't the total io written each group be different? Jens?
You are doing a sequential workload, skipping 4k every time. First write will be to offset 0, next to 8KB, etc. Write 128 would be to 1040384,

which

Quoted 1 line

is 1MB - 8KB. Hence the next feasible offset after that would be 1MB,

which

Quoted 5 lines

is end of the file. So how could it do more than 512KB of IO? That's 128 * 4KB.
I didn't read the whole thread in detail, just looked at your last example here. And for that one, I don't see anything wrong.

I guess I would have thought io_limit always forced wraparound. For example:

# dd if=/dev/zero of=/dev/shm/1M bs=1M count=1
# fio --bs=4k --filename=/dev/shm/1M --name=go1 --rw=write
[...]
Run status group 0 (all jobs):
  WRITE: io=1024KB, aggrb=341333KB/s, minb=341333KB/s, maxb=341333KB/s,
mint=3msec, maxt=3msec
# fio --bs=4k --filename=/dev/shm/1M --name=go2 --io_limit=2M --rw=write
Run status group 0 (all jobs):
  WRITE: io=2048KB, aggrb=341333KB/s, minb=341333KB/s, maxb=341333KB/s,
mint=6msec, maxt=6msec
[...]
# fio --bs=4k --filename=/dev/shm/1M --name=go3 --io_limit=2M --rw=write:4k
[...]
  WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
Run status group 0 (all jobs):
# fio --bs=4k --filename=/dev/shm/1M --name=go4 --io_limit=2M --rw=write:4k
[...]
Run status group 0 (all jobs):
  WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec

Signature · 1 line

Sitsofe | http://sucs.org/~sits/

Jens Axboe

2014-09-28 14:24 UTC #11 lore raw

Quoted 58 lines · Sitsofe Wheeler

(Resending because first mail had an HTML part)
Quoted 17 lines · Jens Axboe
Quoted 17 lines · Sitsofe Wheeler
Quoted 16 lines
This looks like a bug. I can reproduce it with 2.1.11-11-gb7f5 too:
dd if=/dev/zero of=/dev/shm/1M bs=1M count=1
fio --bs=4k --rw=write:4k --filename=/dev/shm/1M --stonewall --name=1M --io_limit=1M --name=2M --io_limit=2M
[...]
Run status group 0 (all jobs):
    WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
Run status group 1 (all jobs):
    WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Why isn't io 1024KB for group 0? Additionally, shouldn't the total io written each group be different? Jens?
You are doing a sequential workload, skipping 4k every time. First write will be to offset 0, next to 8KB, etc. Write 128 would be to 1040384,
which
Quoted 1 line
is 1MB - 8KB. Hence the next feasible offset after that would be 1MB,
which
Quoted 5 lines
is end of the file. So how could it do more than 512KB of IO? That's 128 * 4KB.
I didn't read the whole thread in detail, just looked at your last example here. And for that one, I don't see anything wrong.
I guess I would have thought io_limit always forced wraparound. For example:
# dd if=/dev/zero of=/dev/shm/1M bs=1M count=1
# fio --bs=4k --filename=/dev/shm/1M --name=go1 --rw=write
[...]
Run status group 0 (all jobs):
   WRITE: io=1024KB, aggrb=341333KB/s, minb=341333KB/s, maxb=341333KB/s,
mint=3msec, maxt=3msec
# fio --bs=4k --filename=/dev/shm/1M --name=go2 --io_limit=2M --rw=write
Run status group 0 (all jobs):
   WRITE: io=2048KB, aggrb=341333KB/s, minb=341333KB/s, maxb=341333KB/s,
mint=6msec, maxt=6msec
[...]
# fio --bs=4k --filename=/dev/shm/1M --name=go3 --io_limit=2M --rw=write:4k
[...]
   WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
Run status group 0 (all jobs):
# fio --bs=4k --filename=/dev/shm/1M --name=go4 --io_limit=2M --rw=write:4k
[...]
Run status group 0 (all jobs):
   WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec

Quoted 8 lines

go2 is a plain sequential job that does twice as much I/O as go1. Given that the size of the file being written to has not changed between the runs one could guess that fio simply wrapped around and started from the first offset (0) to write the second MB of data. Given this isn't it a fair assumption that when doing a skipping workload if io_limit is used (as in go4) and an offset beyond the end of the device is produced the same wraparound behaviour as go2 should occur and the total io done should match that specified in io_limit?

I would agree on that, behavior for those cases _should_ be the same. Without the holed IO, it closes/reopens the file and repeats the 1M writes. With it, it does not. I will take a look.

Signature · 1 line

Jens Axboe

Jens Axboe

2014-09-28 15:08 UTC #12 lore raw

Quoted 77 lines · Jens Axboe

Quoted 65 lines · Sitsofe Wheeler
(Resending because first mail had an HTML part)
Quoted 19 lines · Jens Axboe
Quoted 19 lines · Sitsofe Wheeler
Quoted 18 lines
This looks like a bug. I can reproduce it with 2.1.11-11-gb7f5 too:
dd if=/dev/zero of=/dev/shm/1M bs=1M count=1
fio --bs=4k --rw=write:4k --filename=/dev/shm/1M --stonewall --name=1M --io_limit=1M --name=2M --io_limit=2M
[...]
Run status group 0 (all jobs):
    WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s,
maxb=256000KB/s,
mint=2msec, maxt=2msec
Run status group 1 (all jobs):
    WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s,
maxb=256000KB/s,
mint=2msec, maxt=2msec
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Why isn't io 1024KB for group 0? Additionally, shouldn't the total io written each group be different? Jens?
You are doing a sequential workload, skipping 4k every time. First write will be to offset 0, next to 8KB, etc. Write 128 would be to 1040384,
which
Quoted 1 line
is 1MB - 8KB. Hence the next feasible offset after that would be 1MB,
which
Quoted 7 lines
is end of the file. So how could it do more than 512KB of IO? That's 128 *
4KB.
I didn't read the whole thread in detail, just looked at your last example
here. And for that one, I don't see anything wrong.
I guess I would have thought io_limit always forced wraparound. For example:
# dd if=/dev/zero of=/dev/shm/1M bs=1M count=1
# fio --bs=4k --filename=/dev/shm/1M --name=go1 --rw=write
[...]
Run status group 0 (all jobs):
   WRITE: io=1024KB, aggrb=341333KB/s, minb=341333KB/s, maxb=341333KB/s,
mint=3msec, maxt=3msec
# fio --bs=4k --filename=/dev/shm/1M --name=go2 --io_limit=2M --rw=write
Run status group 0 (all jobs):
   WRITE: io=2048KB, aggrb=341333KB/s, minb=341333KB/s, maxb=341333KB/s,
mint=6msec, maxt=6msec
[...]
# fio --bs=4k --filename=/dev/shm/1M --name=go3 --io_limit=2M
--rw=write:4k
[...]
   WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
Run status group 0 (all jobs):
# fio --bs=4k --filename=/dev/shm/1M --name=go4 --io_limit=2M
--rw=write:4k
[...]
Run status group 0 (all jobs):
   WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
Quoted 10 lines
go2 is a plain sequential job that does twice as much I/O as go1. Given that the size of the file being written to has not changed between the runs
one could guess that fio simply wrapped around and started from the first offset (0) to write the second MB of data. Given this isn't it a fair assumption that when doing a skipping workload if io_limit is used (as in go4) and an offset beyond the end of the device is produced the same wraparound behaviour as go2 should occur and the total io done should match
that specified in io_limit?

Quoted 3 lines

I would agree on that, behavior for those cases _should_ be the same. Without the holed IO, it closes/reopens the file and repeats the 1M writes. With it, it does not. I will take a look.

Does the attached fix it up?

Signature · 1 line

Jens Axboe

io_u.c +8 -1

index 8546899c03e7..cbe14b3f5bda 100644
@@ -283,8 +283,15 @@ static int get_next_seq_offset(struct thread_data *td, struct fio_file *f,
 			f->last_pos = f->real_file_size;
 
 		pos = f->last_pos - f->file_offset;
-		if (pos)
+		if (pos) {
 			pos += td->o.ddir_seq_add;
+			/*
+			 * If we reach beyond the end of the file with
+			 * holed IO, wrap around to the beginning again.
+			 */
+			if (pos >= f->real_file_size)
+				pos = f->file_offset;
+		}
 
 		*offset = pos;
 		return 0;

Sitsofe Wheeler

2014-09-28 19:44 UTC #13 lore raw

Quoted 30 lines · Jens Axboe

Quoted 29 lines · Jens Axboe
Quoted 27 lines · Sitsofe Wheeler
I guess I would have thought io_limit always forced wraparound. For example:
# dd if=/dev/zero of=/dev/shm/1M bs=1M count=1 # fio --bs=4k
--filename=/dev/shm/1M --name=go1 --rw=write [...] Run status group
0 (all jobs): WRITE: io=1024KB, aggrb=341333KB/s, minb=341333KB/s,
maxb=341333KB/s, mint=3msec, maxt=3msec # fio --bs=4k
--filename=/dev/shm/1M --name=go2 --io_limit=2M --rw=write Run
status group 0 (all jobs): WRITE: io=2048KB, aggrb=341333KB/s,
minb=341333KB/s, maxb=341333KB/s, mint=6msec, maxt=6msec [...] # fio
--bs=4k --filename=/dev/shm/1M --name=go3 --io_limit=2M
--rw=write:4k [...] WRITE: io=512KB, aggrb=256000KB/s,
minb=256000KB/s, maxb=256000KB/s, mint=2msec, maxt=2msec Run status
group 0 (all jobs): # fio --bs=4k --filename=/dev/shm/1M --name=go4
--io_limit=2M --rw=write:4k [...] Run status group 0 (all jobs):
WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
go2 is a plain sequential job that does twice as much I/O as go1. Given that the size of the file being written to has not changed between the runs one could guess that fio simply wrapped around and started from the first offset (0) to write the second MB of data. Given this isn't it a fair assumption that when doing a skipping workload if io_limit is used (as in go4) and an offset beyond the end of the device is produced the same wraparound behaviour as go2 should occur and the total io done should match that specified in io_limit?

Quoted 5 lines

Quoted 3 lines
I would agree on that, behavior for those cases _should_ be the same. Without the holed IO, it closes/reopens the file and repeats the 1M writes. With it, it does not. I will take a look.
Does the attached fix it up?

The patch fixes
fio --bs=4k --rw=write:4k --filename=/dev/shm/1M --name=go --io_limit=2M
but not
fio --bs=512k --rw=write --filename=/dev/shm/1M --name=go --number_io=4
or
fio --bs=4k --rw=write --filename=/dev/shm/1M --name=go --zoneskip=4k --zonesize=4k --io_limit=2M

Signature · 1 line

Sitsofe | http://sucs.org/~sits/

Jens Axboe

2014-09-28 22:13 UTC #14 lore raw

Quoted 35 lines · Sitsofe Wheeler

Quoted 34 lines · Jens Axboe
Quoted 32 lines · Jens Axboe
Quoted 18 lines · Sitsofe Wheeler
I guess I would have thought io_limit always forced wraparound. For example:
# dd if=/dev/zero of=/dev/shm/1M bs=1M count=1 # fio --bs=4k
--filename=/dev/shm/1M --name=go1 --rw=write [...] Run status group
0 (all jobs): WRITE: io=1024KB, aggrb=341333KB/s, minb=341333KB/s,
maxb=341333KB/s, mint=3msec, maxt=3msec # fio --bs=4k
--filename=/dev/shm/1M --name=go2 --io_limit=2M --rw=write Run
status group 0 (all jobs): WRITE: io=2048KB, aggrb=341333KB/s,
minb=341333KB/s, maxb=341333KB/s, mint=6msec, maxt=6msec [...] # fio
--bs=4k --filename=/dev/shm/1M --name=go3 --io_limit=2M
--rw=write:4k [...] WRITE: io=512KB, aggrb=256000KB/s,
minb=256000KB/s, maxb=256000KB/s, mint=2msec, maxt=2msec Run status
group 0 (all jobs): # fio --bs=4k --filename=/dev/shm/1M --name=go4
--io_limit=2M --rw=write:4k [...] Run status group 0 (all jobs):
WRITE: io=512KB, aggrb=256000KB/s, minb=256000KB/s, maxb=256000KB/s,
mint=2msec, maxt=2msec
Quoted 9 lines
go2 is a plain sequential job that does twice as much I/O as go1. Given that the size of the file being written to has not changed between the runs one could guess that fio simply wrapped around and started from the first offset (0) to write the second MB of data. Given this isn't it a fair assumption that when doing a skipping workload if io_limit is used (as in go4) and an offset beyond the end of the device is produced the same wraparound behaviour as go2 should occur and the total io done should match that specified in io_limit?
I would agree on that, behavior for those cases _should_ be the same. Without the holed IO, it closes/reopens the file and repeats the 1M writes. With it, it does not. I will take a look.

Quoted 6 lines

Quoted 1 line
Does the attached fix it up?
The patch fixes
fio --bs=4k --rw=write:4k --filename=/dev/shm/1M --name=go --io_limit=2M but not
fio --bs=512k --rw=write --filename=/dev/shm/1M --name=go --number_io=4

number_ios=x is implemented as a cap, not a forced "must complete this amount of ios to be done".

Quoted 2 lines

or
fio --bs=4k --rw=write --filename=/dev/shm/1M --name=go --zoneskip=4k --zonesize=4k --io_limit=2M

That one is a bit more tricky. Oh, try the attached (keep the previous applied).

Signature · 1 line

Jens Axboe

io_u.c +6 -2

index 8546899c03e7..02f600be3126 100644
@@ -748,9 +751,13 @@ static int fill_io_u(struct thread_data *td, struct io_u *io_u)
 	 * See if it's time to switch to a new zone
 	 */
 	if (td->zone_bytes >= td->o.zone_size && td->o.zone_skip) {
+		struct fio_file *f = io_u->file;
+
 		td->zone_bytes = 0;
-		io_u->file->file_offset += td->o.zone_range + td->o.zone_skip;
-		io_u->file->last_pos = io_u->file->file_offset;
+		f->file_offset += td->o.zone_range + td->o.zone_skip;
+		if (f->file_offset >= f->real_file_size)
+			f->file_offset = f->real_file_size - f->file_offset;
+		f->last_pos = f->file_offset;
 		td->io_skip_bytes += td->o.zone_skip;
 	}

Sitsofe Wheeler

2014-09-29 05:42 UTC #15 lore raw

Quoted 9 lines · Jens Axboe

Quoted 4 lines · Sitsofe Wheeler
The patch fixes
fio --bs=4k --rw=write:4k --filename=/dev/shm/1M --name=go --io_limit=2M but not
fio --bs=512k --rw=write --filename=/dev/shm/1M --name=go --number_io=4
number_ios=x is implemented as a cap, not a forced "must complete this amount of ios to be done".

Ah I see. Now I look again the HOWTO even says it doesn't extend the number of I/Os - I should have read more carefully!

Quoted 6 lines

Quoted 2 lines
or
fio --bs=4k --rw=write --filename=/dev/shm/1M --name=go --zoneskip=4k
--zonesize=4k --io_limit=2M
That one is a bit more tricky. Oh, try the attached (keep the previous applied).

Latest patch fixes zone skipping for me too. Akira are things better for
you too?

Signature · 1 line

Sitsofe | http://sucs.org/~sits/

Sitsofe Wheeler

2014-09-29 07:41 UTC #16 lore raw

In reply to Jens Axboe

(Resending because list rejected HTML part)

Quoted 8 lines · Jens Axboe

Quoted 4 lines · Sitsofe Wheeler
The patch fixes
fio --bs=4k --rw=write:4k --filename=/dev/shm/1M --name=go --io_limit=2M but not
fio --bs=512k --rw=write --filename=/dev/shm/1M --name=go --number_io=4
number_ios=x is implemented as a cap, not a forced "must complete this amount of ios to be done".

Ah I see. Now I look again the HOWTO even says it doesn't extend the number of I/Os - I should have read more carefully!

Quoted 6 lines

Quoted 2 lines
or
fio --bs=4k --rw=write --filename=/dev/shm/1M --name=go --zoneskip=4k
--zonesize=4k --io_limit=2M
That one is a bit more tricky. Oh, try the attached (keep the previous applied).

Latest patch fixes zone skipping for me too. Akira are things better for you too?

Signature · 1 line

Sitsofe | http://sucs.org/~sits/

Akira Hayakawa

author

2014-09-30 01:23 UTC #17 lore raw

Quoted 2 lines

Latest patch fixes zone skipping for me too. Akira are things better for you too?

Thanks Sitsofe,
I will make my time on this weekend to try your patch out. But, at the first look, it looks fine.

- Akira

Jens Axboe

2014-09-30 02:21 UTC #18 lore raw

Quoted 5 lines · Akira Hayakawa

Quoted 2 lines
Latest patch fixes zone skipping for me too. Akira are things better for you too?
Thanks Sitsofe,
I will make my time on this weekend to try your patch out. But, at the first look, it looks fine.

Just update to latest -git, both fixes have been committed.

Signature · 1 line

Jens Axboe

Akira Hayakawa

author

2014-10-05 07:15 UTC #19 lore raw

In reply to Akira Hayakawa

The latest fio works as I expected.
It wraps around until certain amount (specified by io_limit) ends.

Thanks,

- Akira

Quoted 8 lines · Akira Hayakawa

Quoted 2 lines
Latest patch fixes zone skipping for me too. Akira are things better for you too?
Thanks Sitsofe,
I will make my time on this weekend to try your patch out. But, at the first look, it looks fine.
- Akira