How to re-use default sequential filenames?

I'm trying to put together a test of the write and read speed to some new SAN storage. Our workflow involves writing large numbers of 12 MiB files (on the order of 20,000 or so) at a time. I'd like to set up a config file section that will write all 20,000 files then read all 20,000 files and report on the write performance and the read performance (separately).

I've tried something like this:

[global]
blocksize=4m
filesize=12m
nrfiles=20000
openfiles=1
file_service_type=sequential
create_on_open=1
ioengine=posixaio

[write]
rw=write

[read]
stonewall
rw=read

But the issue is that the files get created with default filenames (write.1.1, write.1.2, etc.), so that when the read job is run, it can't find any files (since it expects the files to be named read.1.1, read.1.2, etc.). If I try to specify the "filename=" option in either section, fio no longer appends the ".<thread>.<sequence>" to the filename, but rather tries to do all I/O to a single file.

Is there a syntax for the "filename=" option that will allow me to specify a different root filename, but still use the ".<thread>.<sequence>" naming convention? Failing that, is there any other way to accomplish my goal?

Quoted 31 lines · Alan Hagge

I'm trying to put together a test of the write and read speed to some new SAN storage. Our workflow involves writing large numbers of 12 MiB files (on the order of 20,000 or so) at a time. I'd like to set up a config file section that will write all 20,000 files then read all 20,000 files and report on the write performance and the read performance (separately).

I've tried something like this:

[global]
blocksize=4m
filesize=12m
nrfiles=20000
openfiles=1
file_service_type=sequential
create_on_open=1
ioengine=posixaio

[write]
rw=write

[read]
stonewall
rw=read

But the issue is that the files get created with default filenames (write.1.1, write.1.2, etc.), so that when the read job is run, it can't find any files (since it expects the files to be named read.1.1, read.1.2, etc.). If I try to specify the "filename=" option in either section, fio no longer appends the ".<thread>.<sequence>" to the filename, but rather tries to do all I/O to a single file.

Quoted 3 lines

Is there a syntax for the "filename=" option that will allow me to specify a different root filename, but still use the ".<thread>.<sequence>" naming convention? Failing that, is there any other way to accomplish my goal?

Good question, and no, you can't currently do that. But you should be able to do that. Fio has no current option for specifying the naming. We could have a fileprefix= option that allows you to set that.

So we currently have two options. The first option is that you take on this task. The file name (if not given with filename=) is generated in init.c:add_job(), here:

	if (!td->o.filename && !td->files_index && !td->o.read_iolog_file) {
		file_alloced = 1;

		if (td->o.nr_files == 1 && exists_and_not_file(jobname))
			add_file(td, jobname);
		else {
			for (i = 0; i < td->o.nr_files; i++) {
				sprintf(fname, "%s.%d.%d", jobname,
							td->thread_number, i);
				add_file(td, fname);
			}
		}
	}

Options are pretty easy to add, basically just an entry in the fio_option options[] array in options.c with pretty much self-explanatory fields. Add matching string type in fio.h to thread_options{ }.

The other option is that you claim that you are not a programmer, and then you are at the mercy of someone else (most likely me!) doing it for you. Since this is a good feature request, I can be talked into that as well.

Let me know.

Hello to all,
this is actually great thread. I discovered wonderful fio before two days during seeking for tool, which allows me to simulate typical workload with DPX files playback and recording (one file per frame). I'm playing with it today and slowly getting into options. By coincidence, i subscribed to this list to ask almost same question as Alan and found this first thread.. :-)
Reusing of generated frames (file sequences) between write and read jobs is also important for me and is exactly thing, what i thought about. I will try to test patch by Jens.

One small thing, sorry for slight derail of thread topic, which come to my mind was option for kind of automatic set of blocksize during read to match length of each pre-generated file within sequence, which fio got from directory using opendir directive. Idea behind this come from common behaviour of VFX applications, which issue read io for whole frame size. So application use stat() output to get actual file size before reading of each frame. If I have same filesize per frame it is easy to adjust blocksize before benchmark, but if i want to test read performance of sequence with compressed frames or simulate mixture of different resolutions, this will be handy.

Thank you and especially Jens for creation of fio (first day with it is mindblowing :)

It would not be a problem making the block size decision probed or dynamic. But it's not clear to me from the above what you would base it on. The file size? Or st_blksize?

2013/4/5 Jens Axboe <[email protected]>:

Quoted 3 lines

It would not be a problem making the block size decision probed or dynamic. But it's not clear to me from the above what you would base it on. The file size? Or st_blksize?

I'm sorry Jens, i wasn't much clear regarding this. I meant file size - st_size.

Quoted 68 lines · Jens Axboe

Quoted 31 lines · Alan Hagge

I'm trying to put together a test of the write and read speed to some new SAN storage. Our workflow involves writing large numbers of 12 MiB files (on the order of 20,000 or so) at a time. I'd like to set up a config file section that will write all 20,000 files then read all 20,000 files and report on the write performance and the read performance (separately).

I've tried something like this:

[global]
blocksize=4m
filesize=12m
nrfiles=20000
openfiles=1
file_service_type=sequential
create_on_open=1
ioengine=posixaio

[write]
rw=write

[read]
stonewall
rw=read

But the issue is that the files get created with default filenames (write.1.1, write.1.2, etc.), so that when the read job is run, it can't find any files (since it expects the files to be named read.1.1, read.1.2, etc.). If I try to specify the "filename=" option in either section, fio no longer appends the ".<thread>.<sequence>" to the filename, but rather tries to do all I/O to a single file.

Quoted 3 lines

Is there a syntax for the "filename=" option that will allow me to specify a different root filename, but still use the ".<thread>.<sequence>" naming convention? Failing that, is there any other way to accomplish my goal?

Good question, and no, you can't currently do that. But you should be able to do that. Fio has no current option for specifying the naming. We could have a fileprefix= option that allows you to set that.

So we currently have two options. The first option is that you take on this task. The file name (if not given with filename=) is generated in init.c:add_job(), here:

	if (!td->o.filename && !td->files_index && !td->o.read_iolog_file) {
		file_alloced = 1;

		if (td->o.nr_files == 1 && exists_and_not_file(jobname))
			add_file(td, jobname);
		else {
			for (i = 0; i < td->o.nr_files; i++) {
				sprintf(fname, "%s.%d.%d", jobname,
							td->thread_number, i);
				add_file(td, fname);
			}
		}
	}

Options are pretty easy to add, basically just an entry in the fio_option options[] array in options.c with pretty much self-explanatory fields. Add matching string type in fio.h to thread_options{ }.

The other option is that you claim that you are not a programmer, and then you are at the mercy of someone else (most likely me!) doing it for you. Since this is a good feature request, I can be talked into that as well.

Quoted 1 line

Let me know.

OK, so I give it a quick shot, see below. Basically it allows you to set fileprefix= to override the jobname.threadnumber part of the file. So not super flexible, we'd need some reserved keywords to make it fully flexible. Eg it would be nifty if you could do:

fileprefix=$jobnum.$threadnum.$filenum

and then 'write' and 'read' job would be sharing those files. Let me know if it works for you.

2013/4/5 Jens Axboe <[email protected]>:

Quoted 2 lines

and then 'write' and 'read' job would be sharing those files. Let me know if it works for you.

Thank you for patch Jens.
Re-using files working for me and i also like that string format specification. Compiled with last git and tested on Mac OS X and Centos 5.
I played with fio and sequence tests, it already helps me to get figures much closer to real world utilization. I always struggled with generic synthetic benchmarks as it usually don't work with sequences. So i can roughly set IO sizes, modes, but for example can't simulate performance differences caused by different file allocation between one huge file (with few extents due to filesystem internal optimization) and thousands of files.
Great!

P.S.: Here is what i tried as single thread HD video job. (and as example where application read whole frame at once, so bs is uncommonly equal filesize).

[global]
ioengine=sync
buffered=0
filesize=8305664
bs=8305664
nrfiles=3000
openfiles=1
filename_format=HD_sequence_test.$filenum.dpx
file_service_type=sequential
directory=/mnt/test/io

[2min-sequence-write]
rw=write

[2min-sequence-read]
stonewall
rw=read

Quoted 15 lines · Michal Šmucr

2013/4/5 Jens Axboe<[email protected]>:

Quoted 2 lines

and then 'write' and 'read' job would be sharing those files. Let me know if it works for you.

Thank you for patch Jens.
Re-using files working for me and i also like that string format specification. Compiled with last git and tested on Mac OS X and Centos 5.
I played with fio and sequence tests, it already helps me to get figures much closer to real world utilization. I always struggled with generic synthetic benchmarks as it usually don't work with sequences. So i can roughly set IO sizes, modes, but for example can't simulate performance differences caused by different file allocation between one huge file (with few extents due to filesystem internal optimization) and thousands of files.
Great!

Thanks for testing and confirming that it both works and that the semantics make sense. I tagged 2.0.15 this morning and kept a few pending features/fixes in a 'next' branch, all have been pulled into the master branch. So the filename_format option is now in current -git, though it did not make 2.0.15 final.

Quoted 16 lines · Jens Axboe

Quoted 15 lines · Michal Šmucr

2013/4/5 Jens Axboe<[email protected]>:

Quoted 2 lines

and then 'write' and 'read' job would be sharing those files. Let me know if it works for you.

Thank you for patch Jens.
Re-using files working for me and i also like that string format specification. Compiled with last git and tested on Mac OS X and Centos 5.
I played with fio and sequence tests, it already helps me to get figures much closer to real world utilization. I always struggled with generic synthetic benchmarks as it usually don't work with sequences. So i can roughly set IO sizes, modes, but for example can't simulate performance differences caused by different file allocation between one huge file (with few extents due to filesystem internal optimization) and thousands of files.
Great!

Quoted 5 lines

Thanks for testing and confirming that it both works and that the semantics make sense. I tagged 2.0.15 this morning and kept a few pending features/fixes in a 'next' branch, all have been pulled into the master branch. So the filename_format option is now in current -git, though it did not make 2.0.15 final.

I too was able to compile the git version and try a test this morning and it looks to be working just fine. This will help us immensely. Thanks for the quick response and flexible solution!

I'm trying to put together a test of the write and read speed to some new SAN storage. Our workflow involves writing large numbers of 12 MiB files (on the order of 20,000 or so) at a time. I'd like to set up a config file section that will write all 20,000 files then read all 20,000 files and report on the write performance and the read performance (separately).

I've tried something like this:

[global]
blocksize=4m
filesize=12m
nrfiles=20000
openfiles=1
file_service_type=sequential
create_on_open=1
ioengine=posixaio

[write]
rw=write

[read]
stonewall
rw=read

But the issue is that the files get created with default filenames (write.1.1, write.1.2, etc.), so that when the read job is run, it can't find any files (since it expects the files to be named read.1.1, read.1.2, etc.). If I try to specify the "filename=" option in either section, fio no longer appends the ".<thread>.<sequence>" to the filename, but rather tries to do all I/O to a single file.

Is there a syntax for the "filename=" option that will allow me to specify a different root filename, but still use the ".<thread>.<sequence>" naming convention? Failing that, is there any other way to accomplish my goal?

Thanks for any tips, pointers, etc.

--
To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

From: [email protected] [mailto:[email protected]] On Behalf Of Matt Hayward

It seems possible that doing this for 20000 files might create a problem with the length of the parameter...