fio test triggering bad data on ext4
Hi,
I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and O_DIRECT.
It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:
Bad verify header 0 at 10137600
fio: pid=9943, err=84/file:io_u.c:1212, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character
writers: (groupid=0, jobs=32): err=84 (file:io_u.c:1212, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character): pid=9943
which tells us that where we expected to find the correct verify magic in the header, it was all zeroes. The job file used is below, and to reproduce you want to use the latest fio (1.40) since some earlier versions don't do verify_interval properly for non-pattern verifies. You can get fio here:
http://brick.kernel.dk/snaps/fio-1.40.tar.gz
or from git at:
git://git.kernel.dk/fio.git
The kernel used is 2.6.35-rc3 and I ran this on a raid0 that had 8 SSD drives.
--- snip job file ---
[global]
direct=1
group_reporting=1
exitall
runtime=4h
time_based=1
# writers, will repeatedly randomly write and verify data [writers]
rw=randwrite
bsrange=4k-16k
ioengine=libaio
iodepth=4
directory=/data
verify=crc32c
verify_backlog=1024
verify_backlog_batch=512
verify_interval=512
size=512m
nrfiles=4
filesize=64m-256m
numjobs=32
create_serialize=0
--- snip job file ---
Signature · 1 line
Jens Axboe
Quoted 11 lines · Jens Axboe
Hi,
I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and
O_DIRECT.It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:
Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.
Thanks,
-Eric
Quoted 13 lines · Eric Sandeen
Quoted 11 lines · Jens Axboe
Hi,
I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and
O_DIRECT.It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:
Quoted 6 lines
Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.
Thanks,
-Eric
Actually mingming had a patch for direct-io.c which may be related, I'll test that out.
-Eric
Quoted 18 lines · Eric Sandeen
Quoted 17 lines · Eric Sandeen
Quoted 11 lines · Jens Axboe
Hi,
I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and
O_DIRECT.It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:
Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.
Quoted 5 lines
Quoted 2 lines
Thanks,
-EricActually mingming had a patch for
direct-io.cwhich may be related, I'll test that out.
OK, I'll try XFS tonight as well.
Signature · 1 line
Jens Axboe
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Eric Sandeen
Quoted 18 lines · Eric Sandeen
Quoted 8 lines · Jens Axboe
Hi,
I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and
O_DIRECT.Quoted 4 lines
It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:
Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
Thanks,
-EricActually mingming had a patch for
direct-io.cwhich may be related, I'll test that out.OK, I'll try XFS tonight as well.
I haven't been able to reproduce it on ext4 here, yet.
FWIW here's the patch from mingming:
When unaligned DIO writes, skip zero out the block if the buffer is marked unwritten. That means there is an asynconous direct IO (append or fill the hole) still pending.
--- fs/direct-io.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
fs/direct-io.c +2 -1
@@ -740,7 +740,8 @@struct page *page; dio->start_zero_done =1 ; -if (!dio->blkfactor || !buffer_new (&dio->map_bh)) +if (!dio->blkfactor || !buffer_new (&dio->map_bh) + ||buffer_unwritten (&dio->map_bh))return ; dio_blocks_per_fs_block =1 << dio->blkfactor;
Quoted 54 lines · Eric Sandeen
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Eric Sandeen
Quoted 18 lines · Eric Sandeen
Quoted 8 lines · Jens Axboe
Hi,
I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and
O_DIRECT.Quoted 4 lines
It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:
Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
Thanks,
-EricActually mingming had a patch for
direct-io.cwhich may be related, I'll test that out.OK, I'll try XFS tonight as well.
I haven't been able to reproduce it on ext4 here, yet.
FWIW here's the patch from mingming:
When unaligned DIO writes, skip zero out the block if the buffer is marked unwritten. That means there is an asynconous direct IO (append or fill the hole) still pending.
Signed-off-by Mingming Cao <[email protected]>--- fs/direct-io.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)Index:
linux-git/fs/direct-io.c
===================================================================
fs/direct-io.c line 747
Quoted 9 lines
fs/direct-io.c +2
-1@@ -740,7 +740,8 @@struct page *page; dio->start_zero_done =1 ; -if (!dio->blkfactor || !buffer_new (&dio->map_bh)) +if (!dio->blkfactor || !buffer_new (&dio->map_bh) + ||buffer_unwritten (&dio->map_bh))return ; dio_blocks_per_fs_block =1 << dio->blkfactor;
What is this patch against?
Signature · 1 line
Jens Axboe
Eric Sandeen
AppliedQuoted 60 lines · Jens Axboe
Quoted 59 lines · Eric Sandeen
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Eric Sandeen
Quoted 18 lines · Eric Sandeen
Quoted 8 lines · Jens Axboe
Hi,
I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and
O_DIRECT.Quoted 4 lines
It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:
Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
Thanks,
-EricActually mingming had a patch for
direct-io.cwhich may be related, I'll test that out.OK, I'll try XFS tonight as well.
I haven't been able to reproduce it on ext4 here, yet.
FWIW here's the patch from mingming:
When unaligned DIO writes, skip zero out the block if the buffer is marked unwritten. That means there is an asynconous direct IO (append or fill the hole) still pending.
Signed-off-by Mingming Cao <[email protected]>--- fs/direct-io.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)Index:
linux-git/fs/direct-io.c
===================================================================fs/direct-io.c +2
-1@@ -740,7 +740,8 @@struct page *page; dio->start_zero_done =1 ; -if (!dio->blkfactor || !buffer_new (&dio->map_bh)) +if (!dio->blkfactor || !buffer_new (&dio->map_bh) + ||buffer_unwritten (&dio->map_bh))return ;
Quoted 6 lines
fs/direct-io.c line 747
Quoted 3 lines
fs/direct-io.c +0
-0dio_blocks_per_fs_block =1 << dio->blkfactor;What is this patch against?
Applied to 2.6.32, seems to apply upstream as well.
It hits dio_zero-block()
-Eric
Quoted 68 lines · Eric Sandeen
Quoted 60 lines · Jens Axboe
Quoted 59 lines · Eric Sandeen
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Eric Sandeen
Quoted 18 lines · Eric Sandeen
Quoted 8 lines · Jens Axboe
Hi,
I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and
O_DIRECT.Quoted 4 lines
It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:
Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
Thanks,
-EricActually mingming had a patch for
direct-io.cwhich may be related, I'll test that out.OK, I'll try XFS tonight as well.
I haven't been able to reproduce it on ext4 here, yet.
FWIW here's the patch from mingming:
When unaligned DIO writes, skip zero out the block if the buffer is marked unwritten. That means there is an asynconous direct IO (append or fill the hole) still pending.
Signed-off-by Mingming Cao <[email protected]>--- fs/direct-io.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)Index:
linux-git/fs/direct-io.c
===================================================================fs/direct-io.c +2
-1@@ -740,7 +740,8 @@struct page *page; dio->start_zero_done =1 ; -if (!dio->blkfactor || !buffer_new (&dio->map_bh)) +if (!dio->blkfactor || !buffer_new (&dio->map_bh) + ||buffer_unwritten (&dio->map_bh))return ;Quoted 6 lines
fs/direct-io.c line 747
Quoted 3 lines
fs/direct-io.c +0
-0dio_blocks_per_fs_block =1 << dio->blkfactor;What is this patch against?
Quoted 3 lines
Applied to 2.6.32, seems to apply upstream as well.
It hits dio_zero-block()
Irk indeed, I am blind. The patch does not fix it.
Signature · 1 line
Jens Axboe
Quoted 69 lines · Jens Axboe
Quoted 68 lines · Eric Sandeen
Quoted 66 lines · Jens Axboe
Quoted 53 lines · Eric Sandeen
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Eric Sandeen
Quoted 18 lines · Eric Sandeen
Quoted 8 lines · Jens Axboe
Hi,
I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and
O_DIRECT.Quoted 4 lines
It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:
Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
Thanks,
-EricActually mingming had a patch for
direct-io.cwhich may be related, I'll test that out.OK, I'll try XFS tonight as well.
I haven't been able to reproduce it on ext4 here, yet.
FWIW here's the patch from mingming:
When unaligned DIO writes, skip zero out the block if the buffer is marked unwritten. That means there is an asynconous direct IO (append or fill the hole) still pending.
Signed-off-by Mingming Cao <[email protected]>--- fs/direct-io.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)Index:
linux-git/fs/direct-io.c
===================================================================fs/direct-io.c line 747
Quoted 9 lines
fs/direct-io.c +2
-1@@ -740,7 +740,8 @@struct page *page; dio->start_zero_done =1 ; -if (!dio->blkfactor || !buffer_new (&dio->map_bh)) +if (!dio->blkfactor || !buffer_new (&dio->map_bh) + ||buffer_unwritten (&dio->map_bh))return ; dio_blocks_per_fs_block =1 << dio->blkfactor;What is this patch against?
Quoted 5 lines
Quoted 3 lines
Applied to 2.6.32, seems to apply upstream as well.
It hits dio_zero-block()
Irk indeed, I am blind. The patch does not fix it.
So just to confirm that this isn't a new regression, 2.6.34 fails in the same way. If I change the test to make the random writes overwrite existing blocks instead of filling holes, then there are no problems either.
Signature · 1 line
Jens Axboe
Quoted 19 lines · Jens Axboe
Quoted 18 lines · Eric Sandeen
Quoted 17 lines · Eric Sandeen
Quoted 11 lines · Jens Axboe
Hi,
I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and
O_DIRECT.It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:
Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Thanks,
-EricActually mingming had a patch for
direct-io.cwhich may be related, I'll test that out.OK, I'll try XFS tonight as well.
XFS fails too.
Signature · 1 line
Jens Axboe
Quoted 8 lines · Jens Axboe
Hi,
I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and
O_DIRECT.
Quoted 8 lines
It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:
Bad verify header 0 at 10137600
fio: pid=9943, err=84/file:io_u.c:1212, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide characterwriters: (groupid=0, jobs=32): err=84 (file:io_u.c:1212, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character): pid=9943
FYI:
I asked Jens to test hch's and Jiaying's aio completion patches with this, and apparently those fixed this problem for him.
-Eric
Quoted 43 lines
which tells us that where we expected to find the correct verify magic in the header, it was all zeroes. The job file used is below, and to reproduce you want to use the latest fio (1.40) since some earlier versions don't do
verify_intervalproperly for non-pattern verifies. You can get fio here:http://brick.kernel.dk/snaps/fio-1.40.tar.gz
or from git at:
git://git.kernel.dk/fio.gitThe kernel used is 2.6.35-rc3 and I ran this on a raid0 that had 8 SSD drives.
--- snip job file ---
[global]
direct=1
group_reporting=1
exitall
runtime=4h
time_based=1# writers, will repeatedly randomly write and verify data [writers]
rw=randwrite
bsrange=4k-16k
ioengine=libaio
iodepth=4
directory=/data
verify=crc32c
verify_backlog=1024
verify_backlog_batch=512
verify_interval=512
size=512m
nrfiles=4
filesize=64m-256m
numjobs=32
create_serialize=0--- snip job file ---
Quoted 16 lines · Eric Sandeen
Quoted 15 lines · Jens Axboe
Hi,
I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and
O_DIRECT.It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:
Bad verify header 0 at 10137600
fio: pid=9943, err=84/file:io_u.c:1212, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character
Quoted 6 lines
Quoted 1 line
writers: (groupid=0, jobs=32): err=84 (file:io_u.c:1212, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character): pid=9943
FYI:
I asked Jens to test hch's and Jiaying's aio completion patches with this, and apparently those fixed this problem for him.
At least for a shorter run, but long enough that all the holes should have been filled at this point. So it at least fixes my test case. I can try and expand the run a bit if there's any interest in that, and see if that still verifies correctly.
Signature · 1 line
Jens Axboe