Added replay_rebase option to run multi-threaded log replay on different disk sectors.
PATCH 1/2 Added replay_rebase option to run multi-threaded log replay on different disk sectors.
Hi.
I'm now trying to run multi-threaded log replay, and noticed all threads are accessing same sector on a disk. Of course, this is an expected behavior of "replay".
But since I also wanted to have each thread accessing different parts of the disk to increase a load, but still following the access pattern of the original, I added "replay_rebase" option to shift offset of IO request for each thread.
Althrough there's "1/2" in the subject, this patch is complete by itself. Next one (2/2) is related, but handles different issue.
--- fio.h | 1 + log.c | 2 +- options.c | 8 ++++++++ 3 files changed, 10 insertions(+), 1 deletions(-)
fio.h +1 -0
@@ -358,6 +358,7 @@ struct thread_options {unsigned int gtod_offload;enum fio_cs clocksource;unsigned int no_stall; +unsigned int replay_rebase;unsigned int trim_percentage;unsigned int trim_batch;unsigned int trim_zero;
log.c +1 -1
@@ -129,7 +129,7 @@ int read_iolog_get(struct thread_data *td, struct io_u *io_u) io_u->ddir = ipo->ddir;if (ipo->ddir !=DDIR_WAIT ) { - io_u->offset = ipo->offset; + io_u->offset = ipo->offset + td->o.replay_rebase * td->thread_number; io_u->buflen = ipo->len; io_u->file = td->files[ipo->fileno];get_file (io_u->file);
options.c +8 -0
@@ -1654,6 +1654,14 @@ static struct fio_option options[FIO_MAX_OPTS] = { .help = "Replay all I/O onto this device, regardless of trace device", }, { + .name = "replay_rebase", + .type = FIO_OPT_INT, + .off1 = td_var_offset(replay_rebase), + .def = "0", + .parent = "read_iolog", + .help = "Apply given offset for IO replay. Mainly for parallel replay.", + }, + { .name = "exec_prerun", .type = FIO_OPT_STR_STORE, .off1 = td_var_offset(exec_prerun),
Quoted 6 lines · Taisuke Yamada
Hi.
I'm now trying to run multi-threaded log replay, and noticed all threads are accessing same sector on a disk. Of course, this is an expected behavior of "replay".
Quoted 7 lines
But since I also wanted to have each thread accessing different parts of the disk to increase a load, but still following the access pattern of the original, I added "
replay_rebase" option to shift offset of IO request for each thread.Althrough there's "1/2" in the subject, this patch is complete by itself. Next one (2/2) is related, but handles different issue.
Thanks, I can see this being handy. I can apply this, but I would much prefer if you resend it with the proper updates to HOWTO and fio.1 adding the new option. Otherwise nobody is going to learn about it.
Signature · 1 line
Jens Axboe
Quoted 6 lines
Quoted 2 lines
[snip], I added "
replay_rebase" option to shift offset of IO request for each thread.Thanks, I can see this being handy. I can apply this, but I would much prefer if you resend it with the proper updates to HOWTO and fio.1 adding the new option. Otherwise nobody is going to learn about it.
Just in case you have missed (I had to break In-Reply-To chain as git format-patch did not respect it). I'm re-posting updated patch with docs. Please apply #1 and #2 if it seems acceptable.
I believe #2 patch is also worth adding to fio, but not sure if I should make it optional feature (/w doc) or not. By wraparounding offset address, it'll probably make benchmark result look better than actual for rotational disk drives. OTOH, it's better than failing to run just because trace-generating device was larger than replaying device.
Best Regards,
From 6f47bef547fb4cc7480b44f6ed832a2e5a3b4150 Mon Sep 17 00:00:00 2001
From: Taisuke Yamada <[email protected]>
Date: Thu, 8 Sep 2011 21:32:01 +0900
Subject: [PATCH 1/2] Added replay_rebase option to run multi-threaded log
replay on different disk sectors.
--- HOWTO | 9 +++++++++ fio.1 | 9 +++++++++ fio.h | 1 + log.c | 2 +- options.c | 8 ++++++++ 5 files changed, 28 insertions(+), 1 deletions(-)
HOWTO +9 -0
@@ -1078,6 +1078,15 @@ replay_redirect=str While replaying I/O patterns using read_iolog the independent fio invocations. Unfortuantely this also breaks the strict time ordering between multiple device accesses. +replay_rebase=int While replaying I/O patterns using read_iolog, the + default behavior is to replay the log as-is, even when + multiple threads are replaying the same log. This means + all threads are accessing the same sector on the device. + By specifying this option, each thread will be accessing + different sector by adding extra rebase offset. Offset is + simply given by "replay_rebase * internal thread number", + so IO access pattern itself is preserved. + write_bw_log=str If given, write a bandwidth log of the jobs in this job file. Can be used to store data of the bandwidth of the jobs in their lifetime. The included fio_generate_plots
fio.1 +9 -0
@@ -826,6 +826,15 @@ is to replay the IOPS onto the major/minor device that each IOP was recorded from. Setting \fBreplay_redirect\fR causes all IOPS to be replayed onto the single specified device regardless of the device it was recorded from. .TP +.BI replay_rebase \fR=\fPint +While replaying I/O patterns using read_iolog, the default behavior +is to replay the log as-is, even when multiple threads are replaying +the same log. This means all threads are accessing the same sector +on the device. By specifying this option, each thread will be accessing +different sector by adding extra rebase offset. Offset is simply given +by "replay_rebase * internal thread number", so IO access pattern itself +is preserved. +.TP .B write_bw_log \fR=\fPstr If given, write a bandwidth log of the jobs in this job file. Can be used to store data of the bandwidth of the jobs in their lifetime. The included
fio.h +1 -0
@@ -358,6 +358,7 @@ struct thread_options {unsigned int gtod_offload;enum fio_cs clocksource;unsigned int no_stall; +unsigned int replay_rebase;unsigned int trim_percentage;unsigned int trim_batch;unsigned int trim_zero;
log.c +1 -1
@@ -129,7 +129,7 @@ int read_iolog_get(struct thread_data *td, struct io_u *io_u) io_u->ddir = ipo->ddir;if (ipo->ddir !=DDIR_WAIT ) { - io_u->offset = ipo->offset; + io_u->offset = ipo->offset + td->o.replay_rebase * td->thread_number; io_u->buflen = ipo->len; io_u->file = td->files[ipo->fileno];get_file (io_u->file);
options.c +8 -0
@@ -1654,6 +1654,14 @@ static struct fio_option options[FIO_MAX_OPTS] = { .help = "Replay all I/O onto this device, regardless of trace device", }, { + .name = "replay_rebase", + .type = FIO_OPT_INT, + .off1 = td_var_offset(replay_rebase), + .def = "0", + .parent = "read_iolog", + .help = "Apply given offset for IO replay. Mainly for parallel replay.", + }, + { .name = "exec_prerun", .type = FIO_OPT_STR_STORE, .off1 = td_var_offset(exec_prerun),
Signature · 1 line
1.7.5.4
From c8e61c6eac198e97b828cdb40441f855e38daaf2 Mon Sep 17 00:00:00 2001
From: Taisuke Yamada <[email protected]>
Date: Thu, 8 Sep 2011 22:28:43 +0900
Subject: [PATCH 2/2] Added wraparound IO support for replay on
smaller/offsetted device.
--- log.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
log.c +3 -0
@@ -87,6 +87,7 @@ static int ipo_special(struct thread_data *td, struct io_piece *ipo)switch (ipo->file_action) {case FIO_LOG_OPEN_FILE : + ret =td_io_get_file_size (td, f);/* for wraparound replay */ ret =td_io_open_file (td, f);if (!ret)break ;
@@ -132,6 +133,8 @@ int read_iolog_get(struct thread_data *td, struct io_u *io_u) io_u->offset = ipo->offset + td->o.replay_rebase * td->thread_number; io_u->buflen = ipo->len; io_u->file = td->files[ipo->fileno]; +if (io_u->file->real_file_size >0 ) + io_u->offset %= io_u->file->real_file_size;get_file (io_u->file);dprint (FD_IO ,"iolog: get %llu/%lu/%s\n" , io_u->offset, io_u->buflen, io_u->file->file_name);
Signature · 1 line
1.7.5.4
Quoted 11 lines · Taisuke Yamada
Quoted 6 lines
Quoted 2 lines
[snip], I added "
replay_rebase" option to shift offset of IO request for each thread.Thanks, I can see this being handy. I can apply this, but I would much prefer if you resend it with the proper updates to HOWTO and fio.1 adding the new option. Otherwise nobody is going to learn about it.
Just in case you have missed (I had to break In-Reply-To chain as git format-patch did not respect it). I'm re-posting updated patch with docs. Please apply #1 and #2 if it seems acceptable.
Quoted 5 lines
I believe #2 patch is also worth adding to fio, but not sure if I should make it optional feature (/w doc) or not. By wraparounding offset address, it'll probably make benchmark result look better than actual for rotational disk drives. OTOH, it's better than failing to run just because trace-generating device was larger than replaying device.
I worry about the 2nd patch, it's a bit of a hack. You could make it an option and turn it on unconditionally for replay_rebase, or at least make it conditional on that.
Signature · 1 line
Jens Axboe
Quoted 9 lines
Quoted 5 lines
I believe #2 patch is also worth adding to fio, but not sure if I should make it optional feature (/w doc) or not. By wraparounding offset address, it'll probably make benchmark result look better than actual for rotational disk drives. OTOH, it's better than failing to run just because trace-generating device was larger than replaying device.
I worry about the 2nd patch, it's a bit of a hack. You could make it an option and turn it on unconditionally for
replay_rebase, or at least make it conditional on that.
Sure. I'll make it an option ("wraparound=bool"), and submit one as well as #1.
Quoted 10 lines · Taisuke Yamada
Quoted 6 lines
Quoted 2 lines
[snip], I added "
replay_rebase" option to shift offset of IO request for each thread.Thanks, I can see this being handy. I can apply this, but I would much prefer if you resend it with the proper updates to HOWTO and fio.1 adding the new option. Otherwise nobody is going to learn about it.
Just in case you have missed (I had to break In-Reply-To chain as git format-patch did not respect it). I'm re-posting updated patch with docs. Please apply #1 and #2 if it seems acceptable.
On patch 1, I don't know, it seems a bit weird to me. The fact that the option offset is multiplied by the job number makes sense for ease of handling job files (since you can just do numjobs=X and be done with it), but logically it's odd since you have to know that they are numbered sequentially and do the math to see where it ends up on the disk.
What is the intended use case for this? I'd much rather see the offset be absolute, at the cost of a bit more complex job file.
Signature · 1 line
Jens Axboe
Quoted 9 lines
On patch 1, I don't know, it seems a bit weird to me. The fact that the option offset is multiplied by the job number makes sense for ease of handling job files (since you can just do numjobs=X and be done with it), but logically it's odd since you have to know that they are numbered sequentially and do the math to see where it ends up on the disk.
What is the intended use case for this? I'd much rather see the offset be absolute, at the cost of a bit more complex job file.
My intention (and usecase) is just to put more replay I/O load, so multiplication is purely for ease of use. So yes, I can live perfectly fine with per-thread(job) absolute offset configuration.
In fact, it may be better as offset can be chosen to access disk more evenly/randomly.
Do you want me to submit a new patch without multiplication?
Quoted 10 lines · Taisuke Yamada
Quoted 9 lines
On patch 1, I don't know, it seems a bit weird to me. The fact that the option offset is multiplied by the job number makes sense for ease of handling job files (since you can just do numjobs=X and be done with it), but logically it's odd since you have to know that they are numbered sequentially and do the math to see where it ends up on the disk.
What is the intended use case for this? I'd much rather see the offset be absolute, at the cost of a bit more complex job file.
Quoted 7 lines
My intention (and usecase) is just to put more replay I/O load, so multiplication is purely for ease of use. So yes, I can live perfectly fine with per-thread(job) absolute offset configuration.
In fact, it may be better as offset can be chosen to access disk more evenly/randomly.Do you want me to submit a new patch without multiplication?
Yeah, please do one without the multiplier and I guess we can put it in.
Signature · 1 line
Jens Axboe