"errror=Invalid or incomplete multibyte or wide character" & "verify: bad header numberio" errors

DISCUSSION Saeed 3 participants 5 messages Jul 22, 2016 – Jul 24, 2016

lore

Share & tools

lore

Saeed

author

2016-07-22 16:30 UTC #0 lore raw

Has anyone seen these errors before?

Here's the job file:

bs=4k
ioengine=libaio
iodepth=32
size=1g
direct=1
runtime=60
filename=/dev/sdb
group_reporting
numjobs=10

# validation
do_verify=1
verify_fatal=1
verify_dump=1

verify_pattern=0x01020304

[mixed-a]
rw=readwrite
stonewall

[mixed-b]
rw=randrw
stonewall

[mixed-c]
rw=readwrite:4k
stonewall

[mixed-d]
rw=randrw:4k
stonewall

Errors:

Multiple writers may overwrite blocks that belong to other jobs. This can cause verification failures. mixed-verify-random-with-holes-4k: (g=3): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.12
Starting 40 processes
fio: pid=5809, err=84/file:io_u.c:1968, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character
fio: pid=5814, err=84/file:io_u.c:1968, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character
fio: pid=5808, err=84/file:io_u.c:1968, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character

verify: bad header numberio 2985, wanted 11 at file /dev/sdb offset 1224704, length 4096
hdr_fail data dumped as sdb.1224704.hdr_fail
verify: bad header numberio 2986, wanted 12 at file /dev/sdb offset 1228800, length 4096
hdr_fail data dumped as sdb.1228800.hdr_fail
verify: bad header numberio 23727, wanted 22903 at file /dev/sdb offset 106496, length 4096
hdr_fail data dumped as sdb.106496.hdr_fail
verify: bad header numberio 13399, wanted 1319 at file /dev/sdb offset 311296, length 4096
hdr_fail data dumped as sdb.311296.hdr_fail

Sitsofe Wheeler

2016-07-22 18:58 UTC #1 lore raw

Quoted 1 line · Saeed

Has anyone seen these errors before?

[...]

Quoted 4 lines

fio-2.12
Starting 40 processes
fio: pid=5809, err=84/file:io_u.c:1968, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character

This frequently happens when a header is bad during verification. For example if the initial verification header magic bytes are bad then verify_header() (from verify.c - see
https://github.com/axboe/fio/blob/fio-2.12/verify.c#L788 ) will end up doing the following:

        return EILSEQ;

EILSEQ is defined as 84.
http://man7.org/linux/man-pages/man3/errno.3.html says this represents
"Illegal byte sequence". However deep in glibc this is mapped to
"Invalid or incomplete multibyte or wide character" when using
strerror (see https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/gnu/errlist.c;h=a9a5960503a97e78143a0c795e7255f418a22933;hb=HEAD#l929
). This seems to stem from its frequent usage with multibyte functions
(see http://www.lysator.liu.se/c/na1.html ).

So glibc has given EILSEQ a narrow error message ("Invalid or incomplete multibyte or wide character") even though it originally had a general meaning ("Illegal byte sequence") and fio is uses EILSEQ to indicate to indicate it has found a corrupt header. I'm not sure there's all that much that can be done about this...

Signature · 1 line

Sitsofe | http://sucs.org/~sits/

Sitsofe Wheeler

2016-07-22 19:06 UTC #2 lore raw

In reply to Saeed

Quoted 28 lines · Saeed

Has anyone seen these errors before?
Here's the job file:
bs=4k
ioengine=libaio
iodepth=32
size=1g
direct=1
runtime=60
filename=/dev/sdb
group_reporting
numjobs=10
# validation
do_verify=1
verify_fatal=1
verify_dump=1
verify_pattern=0x01020304
[mixed-a]
rw=readwrite
stonewall
[mixed-b]
rw=randrw
stonewall

[...]

Quoted 2 lines

fio: pid=5809, err=84/file:io_u.c:1968, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character

I forgot to say that you might be getting these errors because you limit the runtime of individual jobs to 60 seconds and that might not be enough time for your mixed-a job to overwrite the entire 1GByte of the file before it stopped due to running out of time. If later verification jobs try to verify an area not written to, the header may not be there or/malformed thus generating error messages.

Signature · 1 line

Sitsofe | http://sucs.org/~sits/

Quoted 71 lines · Saeed

Has anyone seen these errors before?
Here's the job file:
bs=4k
ioengine=libaio
iodepth=32
size=1g
direct=1
runtime=60
filename=/dev/sdb
group_reporting
numjobs=10
# validation
do_verify=1
verify_fatal=1
verify_dump=1
verify_pattern=0x01020304
[mixed-a]
rw=readwrite
stonewall
[mixed-b]
rw=randrw
stonewall
[mixed-c]
rw=readwrite:4k
stonewall
[mixed-d]
rw=randrw:4k
stonewall
Errors:
Multiple writers may overwrite blocks that belong to other jobs. This can cause verification failures. mixed-verify-random-with-holes-4k: (g=3): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.12
Starting 40 processes
fio: pid=5809, err=84/file:io_u.c:1968, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character
fio: pid=5814, err=84/file:io_u.c:1968, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character
fio: pid=5808, err=84/file:io_u.c:1968, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character
verify: bad header numberio 2985, wanted 11 at file /dev/sdb offset 1224704, length 4096
hdr_fail data dumped as sdb.1224704.hdr_fail
verify: bad header numberio 2986, wanted 12 at file /dev/sdb offset 1228800, length 4096
hdr_fail data dumped as sdb.1228800.hdr_fail
verify: bad header numberio 23727, wanted 22903 at file /dev/sdb offset 106496, length 4096
hdr_fail data dumped as sdb.106496.hdr_fail
verify: bad header numberio 13399, wanted 1319 at file /dev/sdb offset 311296, length 4096
hdr_fail data dumped as sdb.311296.hdr_fail
--
To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 1 line

Sitsofe | http://sucs.org/~sits/

Saeed

author

2016-07-24 15:33 UTC #3 lore raw

I reduced the number of jobs down to 1 and now getting no errors.

https://bugs.chromium.org/p/chromium/issues/detail?id=337651

Quoted 115 lines · Sitsofe Wheeler

Quoted 28 lines · Saeed
Has anyone seen these errors before?
Here's the job file:
bs=4k
ioengine=libaio
iodepth=32
size=1g
direct=1
runtime=60
filename=/dev/sdb
group_reporting
numjobs=10
# validation
do_verify=1
verify_fatal=1
verify_dump=1
verify_pattern=0x01020304
[mixed-a]
rw=readwrite
stonewall
[mixed-b]
rw=randrw
stonewall
[...]
Quoted 2 lines
fio: pid=5809, err=84/file:io_u.c:1968, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character
I forgot to say that you might be getting these errors because you limit the runtime of individual jobs to 60 seconds and that might not be enough time for your mixed-a job to overwrite the entire 1GByte of the file before it stopped due to running out of time. If later verification jobs try to verify an area not written to, the header may not be there or/malformed thus generating error messages.
Signature · 1 line
Sitsofe | http://sucs.org/~sits/
Quoted 71 lines · Saeed
Has anyone seen these errors before?
Here's the job file:
bs=4k
ioengine=libaio
iodepth=32
size=1g
direct=1
runtime=60
filename=/dev/sdb
group_reporting
numjobs=10
# validation
do_verify=1
verify_fatal=1
verify_dump=1
verify_pattern=0x01020304
[mixed-a]
rw=readwrite
stonewall
[mixed-b]
rw=randrw
stonewall
[mixed-c]
rw=readwrite:4k
stonewall
[mixed-d]
rw=randrw:4k
stonewall
Errors:
Multiple writers may overwrite blocks that belong to other jobs. This can cause verification failures. mixed-verify-random-with-holes-4k: (g=3): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.12
Starting 40 processes
fio: pid=5809, err=84/file:io_u.c:1968, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character
fio: pid=5814, err=84/file:io_u.c:1968, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character
fio: pid=5808, err=84/file:io_u.c:1968, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character
verify: bad header numberio 2985, wanted 11 at file /dev/sdb offset 1224704, length 4096
hdr_fail data dumped as sdb.1224704.hdr_fail
verify: bad header numberio 2986, wanted 12 at file /dev/sdb offset 1228800, length 4096
hdr_fail data dumped as sdb.1228800.hdr_fail
verify: bad header numberio 23727, wanted 22903 at file /dev/sdb offset 106496, length 4096
hdr_fail data dumped as sdb.106496.hdr_fail
verify: bad header numberio 13399, wanted 1319 at file /dev/sdb offset 311296, length 4096
hdr_fail data dumped as sdb.311296.hdr_fail
--
To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Jens Axboe

2016-07-24 21:46 UTC #4 lore raw

Quoted 1 line · Saeed

I reduced the number of jobs down to 1 and now getting no errors.

That was your problem. If you look at your original fio output:

"Multiple writers may overwrite blocks that belong to other jobs. This can cause verification failures. "

fio warns you about that as well. And your actual verification errors show that to be the case as well:

"verify: bad header numberio 2985, wanted 11 at file /dev/sdb offset 1224704"

Signature · 1 line

Jens Axboe