fio.1: update description for 'buffer_pattern' and 'verify_pattern'
PATCH 1/2 fio.1: update description for 'buffer_pattern' and 'verify_pattern'
Now those two support combined input and 'verify_pattern' supports '%o' format. Update corresponding doc.
--- fio.1 | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-)
fio.1 +33 -2
@@ -507,7 +507,26 @@ If set, fio will fill the IO buffers with this pattern. If not set, the contents of IO buffers is defined by the other options related to buffer contents. The setting can be any pattern of bytes, and can be prefixed with 0x for hex values. It may also be a string, where the string must then be wrapped with -"". +"", e.g.: +.RS +.RS +\fBbuffer_pattern\fR="abcd" +.RS +or +.RE +\fBbuffer_pattern\fR=-12 +.RS +or +.RE +\fBbuffer_pattern\fR=0xdeadface +.RE +.LP +Also you can combine everything together in any order: +.LP +.RS +\fBbuffer_pattern\fR=0xdeadface"abcd"-12 +.RE +.RE .TP .BI dedupe_percentage \fR=\fPint If set, fio will generate this percentage of identical buffers when writing.
@@ -1167,7 +1186,19 @@ pattern for io verification purposes. Depending on the width of the pattern, fio will fill 1/2/3/4 bytes of the buffer at the time(it can be either a decimal or a hex number). The verify_pattern if larger than a 32-bit quantity has to be a hex number that starts with either "0x" or "0X". Use with -\fBverify\fP=meta or \fBverify\fP=pattern. +\fBverify\fP=meta or \fBverify\fP=pattern. Also, verify_pattern supports %o +format, which means that for each block offset will be written and then +verifyied back, e.g.: +.RS +.RS +\fBverify_pattern\fR=%o +.RE +Or use combination of everything: +.LP +.RS +\fBverify_pattern\fR=0xff%o"abcd"-21 +.RE +.RE .TP .BI verify_fatal \fR=\fPbool If true, exit the job on the first observed verification failure. Default:
Signature · 1 line
2.5.1
PATCH 2/2 verify: move meta header to generic verify_header
fio does not provide any possibility to verify checksum of a block with meta information inside. You can create configuration for verifincation checksum of random data either you can verify meta information with some pattern or random data, but not both.
Why checksumming and meta together can be useful? Meta helps to figure out internally on filesystem or storage what block was written in case of corruption, i.e. offset of the block and block number explicitly tell us the virtual address of the block. On the other hand checksum of random data helps to detect corruption. Using meta and pattern together do not help a lot, since 'verify_interval' can be big enough and same sequence of pattern bytes will be undistinguishable internally on filesystem or storage.
Also, it seems to me that keeping meta header separately from generic verify header does not make a lot of sense, since generic verify header can include all members of meta header without any performance or other impact.
In this patch I move all members from vhdr_meta structure to generic verify_header, always verifying meta with the possiblity to checksum the following data: random or pattern.
You are allowed to specify verify_pattern=str with any of the possible verification methods and have also meta verification, i.e.
verify=md5
verify_pattern=0xfe
or
verify=sha1
verify_pattern=0xff
etc.
To keep everything compatible with old configurations it is still possible to specify
verify=meta
but this option marked and depricated and kept only for compatibility reasons.
Before that patch the verification layout according to the specified options looks as the following, e.g.:
#1
--
verify=meta
verify_pattern=0xff
--
result layout of each block: [hdr|meta|pattern]#2
--
verify_pattern=0xff
--
result layout of each block: [hdr|pattern]#3
--
verify=pattern
verify_pattern=0xff
--
result layout of each block: [pattern]After applying of the patch 'vhdr_meta' is always embedded into 'verify_header' and layout looks as the following, e.g.:
#1
--
verify=meta
verify_pattern=0xff
--
result layout of each block: [hdr+meta|pattern]
#2
--
verify=md5|sha1|etc
verify_pattern=0xff
--
result layout of each block: [hdr+meta|cksum|pattern]#3
--
verify_pattern=0xff
--
result layout of each block: [hdr+meta|pattern]#4
--
verify=pattern
verify_pattern=0xff
--
result layout of each block: [pattern]--- HOWTO | 19 +++++---- fio.1 | 19 +++++---- options.c | 10 +++-- verify.c | 140 +++++++++++++++++++++++++++----------------------------------- verify.h | 16 +++---- 5 files changed, 99 insertions(+), 105 deletions(-)
HOWTO +12 -7
@@ -1248,7 +1248,12 @@ do_verify=bool Run the verify phase after a write phase. Only makes sense if verify is set. Defaults to 1. verify=str If writing to a file, fio can verify the file contents - after each iteration of the job. The allowed values are: + after each iteration of the job. Each verification method also implies + verification of special header, which is written to the beginning of + each block. This header also includes meta information, like offset + of the block, block number, timestamp when block was written, etc. + verify=str can be combined with verify_pattern=str option. + The allowed values are: md5 Use an md5 sum of the data area and store it in the header of each block.
@@ -1284,11 +1289,11 @@ verify=str If writing to a file, fio can verify the file contents sha1 Use optimized sha1 as the checksum function. - meta Write extra information about each io - (timestamp, block number etc.). The block - number is verified. The io sequence number is - verified for workloads that write data. - See also verify_pattern. + meta This option is deprecated, since now meta information is + included in generic verification header and meta verification + happens by default. For detailed information see the description + of the verify=str setting. This option is kept because of + compatibility's sake with old configurations. Do not use it. pattern Verify a strict pattern. Normally fio includes a header with some basic information and
@@ -1333,7 +1338,7 @@ verify_pattern=str If set, fio will fill the io buffers with this buffer at the time(it can be either a decimal or a hex number). The verify_pattern if larger than a 32-bit quantity has to be a hex number that starts with either "0x" or "0X". Use - with verify=meta. Also, verify_pattern supports %o format, + with verify=str. Also, verify_pattern supports %o format, which means that for each block offset will be written and then verifyied back, e.g.:
fio.1 +12 -7
@@ -1134,8 +1134,12 @@ Run the verify phase after a write phase. Only valid if \fBverify\fR is set. Default: true. .TP .BI verify \fR=\fPstr -Method of verifying file contents after each iteration of the job. Allowed -values are: +Method of verifying file contents after each iteration of the job. Each +verification method also implies verification of special header, which is +written to the beginning of each block. This header also includes meta +information, like offset of the block, block number, timestamp when block +was written, etc. \fBverify\fR=str can be combined with \fBverify_pattern\fR=str +option. The allowed values are: .RS .RS .TP
@@ -1145,8 +1149,10 @@ hardware accelerated SSE4.2 driven, falls back to regular crc32c if not supported by the system. .TP .B meta -Write extra information about each I/O (timestamp, block number, etc.). The -block number is verified. See \fBverify_pattern\fR as well. +This option is deprecated, since now meta information is included in generic +verification header and meta verification happens by default. For detailed +information see the description of the \fBverify\fR=str setting. This option +is kept because of compatibility's sake with old configurations. Do not use it. .TP .B pattern Verify a strict pattern. Normally fio includes a header with some basic
@@ -1186,9 +1192,8 @@ pattern for io verification purposes. Depending on the width of the pattern, fio will fill 1/2/3/4 bytes of the buffer at the time(it can be either a decimal or a hex number). The verify_pattern if larger than a 32-bit quantity has to be a hex number that starts with either "0x" or "0X". Use with -\fBverify\fP=meta or \fBverify\fP=pattern. Also, verify_pattern supports %o -format, which means that for each block offset will be written and then -verifyied back, e.g.: +\fBverify\fP=str. Also, verify_pattern supports %o format, which means that for +each block offset will be written and then verifyied back, e.g.: .RS .RS \fBverify_pattern\fR=%o
options.c +7 -3
@@ -958,7 +958,7 @@ static int str_verify_pattern_cb(void *data, const char *input)assert (ret !=0 ); td->o.verify_pattern_bytes = ret;/* -* VERIFY_META could already be set +* VERIFY_* could already be set */ if (!fio_option_is_set (&td->o, verify)) td->o.verify =VERIFY_PATTERN ;
@@ -2294,9 +2294,13 @@ struct fio_option fio_options[FIO_MAX_OPTS] = { .oval = VERIFY_XXHASH, .help = "Use xxhash checksums for verification", }, +/* Meta information was included into verify_header, +* 'meta' verification is implied by default. */ { .ival = "meta", - .oval = VERIFY_META, - .help = "Use io information", + .oval = VERIFY_HDR_ONLY, + .help = "Use io information for verification. " + "Now is implied by default, thus option is obsolete, " + "don't use it", }, { .ival = "pattern", .oval = VERIFY_PATTERN_NO_HDR,
verify.c +60 -80
239 lines not on this page · raw message
verify.h +8 -8
@@ -7,6 +7,9 @@enum {VERIFY_NONE =0 ,/* no verification */ +VERIFY_HDR_ONLY ,/* verify header only, kept for sake of +* compatibility with old configurations +* which use 'verify=meta' */ VERIFY_MD5 ,/* md5 sum data blocks */ VERIFY_CRC64 ,/* crc64 sum data blocks */ VERIFY_CRC32 ,/* crc32 sum data blocks */
@@ -17,7 +20,6 @@ enum {VERIFY_SHA256 ,/* sha256 sum data blocks */ VERIFY_SHA512 ,/* sha512 sum data blocks */ VERIFY_XXHASH ,/* xxhash sum data blocks */ -VERIFY_META ,/* block_num, timestamp etc. */ VERIFY_SHA1 ,/* sha1 sum data blocks */ VERIFY_PATTERN ,/* verify specific patterns */ VERIFY_PATTERN_NO_HDR, /* verify specific patterns, no hdr */
@@ -34,6 +36,11 @@ struct verify_header {uint16_t verify_type;uint32_t len;uint64_t rand_seed; +uint64_t offset; +uint32_t time_sec; +uint32_t time_usec; +uint16_t thread; +uint16_t numberio;uint32_t crc32; };
@@ -61,13 +68,6 @@ struct vhdr_crc16 {struct vhdr_crc7 {uint8_t crc7; }; -struct vhdr_meta { -uint64_t offset; -unsigned char thread; -unsigned short numberio; -unsigned long time_sec; -unsigned long time_usec; -};struct vhdr_xxhash {uint32_t hash; };
Signature · 1 line
2.5.1