Verification errors when write phase is aborted midway
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hello All,
I am running fio verification in two phases:
(1) Write phase, with do_verify=0, on the block device /dev/sdb using --direct=1 flag. While the write phase is running, the disk is disconnected physically. Since data got written to the block device, the journal (state file) was created. After the disk was disconnected, errors were in-flight writes were reported as follows:
fio: io_u error on file /dev/sdb: Input/output error: write offset=2112716800, buflen=8192
fio: io_u error on file /dev/sdb: Input/output error: write offset=2112724992, buflen=8192
fio: io_u error on file /dev/sdb: Input/output error: write offset=2112733184, buflen=8192
fio: io_u error on file /dev/sdb: Input/output error: write offset=2112741376, buflen=8192
(2) When I run the verification phase, the verification fails with the following errors:
verify: bad magic header 0, wanted acca at file /dev/sdb offset 2112716800, length 8192 (requested block: offset=2112716800, length=8192)
verify: bad magic header 0, wanted acca at file /dev/sdb offset 2112724992, length 8192 (requested block: offset=2112724992, length=8192)
verify: bad magic header 0, wanted acca at file /dev/sdb offset 2112733184, length 8192 (requested block: offset=2112733184, length=8192)
The offsets reported in the errors in the verification phase correspond more or less to the errors reported in the write phase.
I am assuming that information in the state file will correspond to successful writes, writes that were persisted on the disk and acknowledged. If so, why was the verification done for data at offsets whose writes failed in-flight ?
Following are the options to write and verification phase:
write:
' --rw=write'
' --bs=8K'
' --direct=1'
' --ioengine=libaio'
' --iodepth=16'
' --verify=crc32c'
' --filename={device_name}'
' --size={disk_size}G'
' --offset={write_offset}'
' --runtime=20m'
' --time_based=1'
' --do_verify=0'
verify:
' --rw=read'
' --bs=8K'
' --direct=1'
' --ioengine=libaio'
' --iodepth=16'
' --verify=crc32c'
' --filename={device_name} '
' --size={disk_size}G'
' --offset={write_offset}'
' --verify_only=1'
' --verify_state_load=1'
' --verify_dump=1'
Quoted 57 lines · Prasad Boddupalli
Hello All,
I am running fio verification in two phases:
(1) Write phase, with do_verify=0, on the block device
/dev/sdbusing --direct=1 flag. While the write phase is running, the disk is disconnected physically. Since data got written to the block device, the journal (state file) was created. After the disk was disconnected, errors were in-flight writes were reported as follows:fio:
io_uerror on file/dev/sdb: Input/output error: write offset=2112716800, buflen=8192
fio:io_uerror on file/dev/sdb: Input/output error: write offset=2112724992, buflen=8192
fio:io_uerror on file/dev/sdb: Input/output error: write offset=2112733184, buflen=8192
fio:io_uerror on file/dev/sdb: Input/output error: write offset=2112741376, buflen=8192(2) When I run the verification phase, the verification fails with the following errors:
verify: bad magic header 0, wanted acca at file
/dev/sdboffset 2112716800, length 8192 (requested block: offset=2112716800, length=8192)
verify: bad magic header 0, wanted acca at file/dev/sdboffset 2112724992, length 8192 (requested block: offset=2112724992, length=8192)
verify: bad magic header 0, wanted acca at file/dev/sdboffset 2112733184, length 8192 (requested block: offset=2112733184, length=8192)The offsets reported in the errors in the verification phase correspond more or less to the errors reported in the write phase.
I am assuming that information in the state file will correspond to successful writes, writes that were persisted on the disk and acknowledged. If so, why was the verification done for data at offsets whose writes failed in-flight ?
Following are the options to write and verification phase:
write:
' --rw=write'
' --bs=8K'
' --direct=1'
' --ioengine=libaio'
' --iodepth=16'
' --verify=crc32c'
' --filename={device_name}'
' --size={disk_size}G'
' --offset={write_offset}'
' --runtime=20m'
' --time_based=1'
' --do_verify=0'
Quoted 14 lines
verify:
' --rw=read'
' --bs=8K'
' --direct=1'
' --ioengine=libaio'
' --iodepth=16'
' --verify=crc32c'
' --filename={device_name}'
' --size={disk_size}G'
' --offset={write_offset}'
' --verify_only=1'
' --verify_state_load=1'
' --verify_dump=1'
When you build fio from source there is a t/fio-verify-state utility to display the contents of the verify state file. Are you able to run it to display the contents of your verify state file?
Are you running on a big ending platform? This feature was broken on such platforms until recently.
https://github.com/vincentkfu/fio-blog/wiki/Testing-on-a-big-endian-platform
Vincent
Quoted 74 lines · Vincent Fu
Quoted 57 lines · Prasad Boddupalli
Hello All,
I am running fio verification in two phases:
(1) Write phase, with do_verify=0, on the block device
/dev/sdbusing --direct=1 flag. While the write phase is running, the disk is disconnected physically. Since data got written to the block device, the journal (state file) was created. After the disk was disconnected, errors were in-flight writes were reported as follows:fio:
io_uerror on file/dev/sdb: Input/output error: write offset=2112716800, buflen=8192
fio:io_uerror on file/dev/sdb: Input/output error: write offset=2112724992, buflen=8192
fio:io_uerror on file/dev/sdb: Input/output error: write offset=2112733184, buflen=8192
fio:io_uerror on file/dev/sdb: Input/output error: write offset=2112741376, buflen=8192(2) When I run the verification phase, the verification fails with the following errors:
verify: bad magic header 0, wanted acca at file
/dev/sdboffset 2112716800, length 8192 (requested block: offset=2112716800, length=8192)
verify: bad magic header 0, wanted acca at file/dev/sdboffset 2112724992, length 8192 (requested block: offset=2112724992, length=8192)
verify: bad magic header 0, wanted acca at file/dev/sdboffset 2112733184, length 8192 (requested block: offset=2112733184, length=8192)The offsets reported in the errors in the verification phase correspond more or less to the errors reported in the write phase.
I am assuming that information in the state file will correspond to successful writes, writes that were persisted on the disk and acknowledged. If so, why was the verification done for data at offsets whose writes failed in-flight ?
Following are the options to write and verification phase:
write:
' --rw=write'
' --bs=8K'
' --direct=1'
' --ioengine=libaio'
' --iodepth=16'
' --verify=crc32c'
' --filename={device_name}'
' --size={disk_size}G'
' --offset={write_offset}'
' --runtime=20m'
' --time_based=1'
' --do_verify=0'Quoted 14 lines
verify:
' --rw=read'
' --bs=8K'
' --direct=1'
' --ioengine=libaio'
' --iodepth=16'
' --verify=crc32c'
' --filename={device_name}'
' --size={disk_size}G'
' --offset={write_offset}'
' --verify_only=1'
' --verify_state_load=1'
' --verify_dump=1'
Quoted 8 lines
When you build fio from source there is a t/fio-verify-state utility to display the contents of the verify state file. Are you able to run it to display the contents of your verify state file?
Are you running on a big ending platform? This feature was broken on such platforms until recently.
https://github.com/vincentkfu/fio-blog/wiki/Testing-on-a-big-endian-platform
Hello Vincent, thank you for your reply. I was able to run the fio-verify-state tool against the state file recorded by FIO. I see verification errors (wherein failed writes are also being verified) only with libaio engine. When I try the same with 'psync' ioengine, I do not see the same errors (although the throughput is greatly reduced with 'psync' engine).