Skip to contentSkip to conversation

fio test triggering bad data on ext4

DISCUSSION Applied Jens Axboe 2 participants 12 messages Jun 18, 2010 – Jul 7, 2010 VFS

lore
Share & tools
lore

Jens Axboe

author
#0 lore raw

Hi,

I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and O_DIRECT.

It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:

Bad verify header 0 at 10137600
fio: pid=9943, err=84/file:io_u.c:1212, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character

writers: (groupid=0, jobs=32): err=84 (file:io_u.c:1212, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character): pid=9943

which tells us that where we expected to find the correct verify magic in the header, it was all zeroes. The job file used is below, and to reproduce you want to use the latest fio (1.40) since some earlier versions don't do verify_interval properly for non-pattern verifies. You can get fio here:

http://brick.kernel.dk/snaps/fio-1.40.tar.gz

or from git at:

git://git.kernel.dk/fio.git

The kernel used is 2.6.35-rc3 and I ran this on a raid0 that had 8 SSD drives.

--- snip job file ---

[global]
direct=1
group_reporting=1
exitall
runtime=4h
time_based=1

# writers, will repeatedly randomly write and verify data [writers]
rw=randwrite
bsrange=4k-16k
ioengine=libaio
iodepth=4
directory=/data
verify=crc32c
verify_backlog=1024
verify_backlog_batch=512
verify_interval=512
size=512m
nrfiles=4
filesize=64m-256m
numjobs=32
create_serialize=0

--- snip job file ---

Signature · 1 line
Jens Axboe
#1 lore raw
Quoted 11 lines · Jens Axboe

Hi,

I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and O_DIRECT.

It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:

Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.

Thanks,
-Eric

#2 lore raw
Quoted 13 lines · Eric Sandeen
Quoted 11 lines · Jens Axboe

Hi,

I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and O_DIRECT.

It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:

Quoted 6 lines

Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.

Thanks,
-Eric

Actually mingming had a patch for direct-io.c which may be related, I'll test that out.

-Eric

author
#3 lore raw
Quoted 18 lines · Eric Sandeen
Quoted 17 lines · Eric Sandeen
Quoted 11 lines · Jens Axboe

Hi,

I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and O_DIRECT.

It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:

Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.

Quoted 5 lines
Quoted 2 lines

Thanks,
-Eric

Actually mingming had a patch for direct-io.c which may be related, I'll test that out.

OK, I'll try XFS tonight as well.

Signature · 1 line
Jens Axboe
#4 lore raw
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Eric Sandeen
Quoted 18 lines · Eric Sandeen
Quoted 8 lines · Jens Axboe

Hi,

I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and O_DIRECT.

Quoted 4 lines

It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:

Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.

Quoted 11 lines
Quoted 6 lines
Quoted 3 lines

Thanks,
-Eric

Actually mingming had a patch for direct-io.c which may be related, I'll test that out.

OK, I'll try XFS tonight as well.

I haven't been able to reproduce it on ext4 here, yet.

FWIW here's the patch from mingming:

When unaligned DIO writes, skip zero out the block if the buffer is marked unwritten. That means there is an asynconous direct IO (append or fill the hole) still pending.

---
 fs/direct-io.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
fs/direct-io.c +2 -1
Index: linux-git/fs/direct-io.c
===================================================================
@@ -740,7 +740,8 @@
 	struct page *page;
 
 	dio->start_zero_done = 1;
-	if (!dio->blkfactor || !buffer_new(&dio->map_bh))
+	if (!dio->blkfactor || !buffer_new(&dio->map_bh)
+	    || buffer_unwritten(&dio->map_bh))
 		return;
 
 	dio_blocks_per_fs_block = 1 << dio->blkfactor;
author
#5 lore raw
Quoted 54 lines · Eric Sandeen
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Eric Sandeen
Quoted 18 lines · Eric Sandeen
Quoted 8 lines · Jens Axboe

Hi,

I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and O_DIRECT.

Quoted 4 lines

It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:

Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.

Quoted 11 lines
Quoted 6 lines
Quoted 3 lines

Thanks,
-Eric

Actually mingming had a patch for direct-io.c which may be related, I'll test that out.

OK, I'll try XFS tonight as well.

I haven't been able to reproduce it on ext4 here, yet.

FWIW here's the patch from mingming:

When unaligned DIO writes, skip zero out the block if the buffer is marked unwritten. That means there is an asynconous direct IO (append or fill the hole) still pending.

---
 fs/direct-io.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Index: linux-git/fs/direct-io.c
===================================================================

fs/direct-io.c line 747

Quoted 9 lines
fs/direct-io.c +2 -1
@@ -740,7 +740,8 @@
 	struct page *page;
 
 	dio->start_zero_done = 1;
-	if (!dio->blkfactor || !buffer_new(&dio->map_bh))
+	if (!dio->blkfactor || !buffer_new(&dio->map_bh)
+	    || buffer_unwritten(&dio->map_bh))
 		return;
 
 	dio_blocks_per_fs_block = 1 << dio->blkfactor;

What is this patch against?

Signature · 1 line
Jens Axboe
author
#9 lore raw
In reply to Jens Axboe
Quoted 19 lines · Jens Axboe
Quoted 18 lines · Eric Sandeen
Quoted 17 lines · Eric Sandeen
Quoted 11 lines · Jens Axboe

Hi,

I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and O_DIRECT.

It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:

Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Thanks,
-Eric

Actually mingming had a patch for direct-io.c which may be related, I'll test that out.

OK, I'll try XFS tonight as well.

XFS fails too.

Signature · 1 line
Jens Axboe
Applied
#6 lore raw
In reply to Jens Axboe
Quoted 60 lines · Jens Axboe
Quoted 59 lines · Eric Sandeen
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Eric Sandeen
Quoted 18 lines · Eric Sandeen
Quoted 8 lines · Jens Axboe

Hi,

I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and O_DIRECT.

Quoted 4 lines

It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:

Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.

Quoted 10 lines
Quoted 6 lines
Quoted 3 lines

Thanks,
-Eric

Actually mingming had a patch for direct-io.c which may be related, I'll test that out.

OK, I'll try XFS tonight as well.

I haven't been able to reproduce it on ext4 here, yet.

FWIW here's the patch from mingming:

When unaligned DIO writes, skip zero out the block if the buffer is marked unwritten. That means there is an asynconous direct IO (append or fill the hole) still pending.

---
 fs/direct-io.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Index: linux-git/fs/direct-io.c
===================================================================

fs/direct-io.c +2 -1
@@ -740,7 +740,8 @@
 	struct page *page;
 
 	dio->start_zero_done = 1;
-	if (!dio->blkfactor || !buffer_new(&dio->map_bh))
+	if (!dio->blkfactor || !buffer_new(&dio->map_bh)
+	    || buffer_unwritten(&dio->map_bh))
 		return;
 
Quoted 6 lines

fs/direct-io.c line 747

Quoted 3 lines
fs/direct-io.c +0 -0
 	dio_blocks_per_fs_block = 1 << dio->blkfactor;

What is this patch against?

Applied to 2.6.32, seems to apply upstream as well.

It hits dio_zero-block()

-Eric

author
#7 lore raw
Quoted 68 lines · Eric Sandeen
Quoted 60 lines · Jens Axboe
Quoted 59 lines · Eric Sandeen
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Eric Sandeen
Quoted 18 lines · Eric Sandeen
Quoted 8 lines · Jens Axboe

Hi,

I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and O_DIRECT.

Quoted 4 lines

It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:

Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.

Quoted 10 lines
Quoted 6 lines
Quoted 3 lines

Thanks,
-Eric

Actually mingming had a patch for direct-io.c which may be related, I'll test that out.

OK, I'll try XFS tonight as well.

I haven't been able to reproduce it on ext4 here, yet.

FWIW here's the patch from mingming:

When unaligned DIO writes, skip zero out the block if the buffer is marked unwritten. That means there is an asynconous direct IO (append or fill the hole) still pending.

---
 fs/direct-io.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Index: linux-git/fs/direct-io.c
===================================================================

fs/direct-io.c +2 -1
@@ -740,7 +740,8 @@
 	struct page *page;
 
 	dio->start_zero_done = 1;
-	if (!dio->blkfactor || !buffer_new(&dio->map_bh))
+	if (!dio->blkfactor || !buffer_new(&dio->map_bh)
+	    || buffer_unwritten(&dio->map_bh))
 		return;
 
Quoted 6 lines

fs/direct-io.c line 747

Quoted 3 lines
fs/direct-io.c +0 -0
 	dio_blocks_per_fs_block = 1 << dio->blkfactor;

What is this patch against?

Quoted 3 lines

Applied to 2.6.32, seems to apply upstream as well.

It hits dio_zero-block()

Irk indeed, I am blind. The patch does not fix it.

Signature · 1 line
Jens Axboe
author
#8 lore raw
Quoted 69 lines · Jens Axboe
Quoted 68 lines · Eric Sandeen
Quoted 66 lines · Jens Axboe
Quoted 53 lines · Eric Sandeen
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Eric Sandeen
Quoted 18 lines · Eric Sandeen
Quoted 8 lines · Jens Axboe

Hi,

I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and O_DIRECT.

Quoted 4 lines

It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:

Jens, can you try XFS too? Since ext3 can't do direct IO to a hole, (and I'm not sure about btrfs in that regard), ext4 may be most similar to xfs's behavior on the test ... wondering how it fares.

Quoted 10 lines
Quoted 6 lines
Quoted 3 lines

Thanks,
-Eric

Actually mingming had a patch for direct-io.c which may be related, I'll test that out.

OK, I'll try XFS tonight as well.

I haven't been able to reproduce it on ext4 here, yet.

FWIW here's the patch from mingming:

When unaligned DIO writes, skip zero out the block if the buffer is marked unwritten. That means there is an asynconous direct IO (append or fill the hole) still pending.

---
 fs/direct-io.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Index: linux-git/fs/direct-io.c
===================================================================

fs/direct-io.c line 747

Quoted 9 lines
fs/direct-io.c +2 -1
@@ -740,7 +740,8 @@
 	struct page *page;
 
 	dio->start_zero_done = 1;
-	if (!dio->blkfactor || !buffer_new(&dio->map_bh))
+	if (!dio->blkfactor || !buffer_new(&dio->map_bh)
+	    || buffer_unwritten(&dio->map_bh))
 		return;
 
 	dio_blocks_per_fs_block = 1 << dio->blkfactor;

What is this patch against?

Quoted 5 lines
Quoted 3 lines

Applied to 2.6.32, seems to apply upstream as well.

It hits dio_zero-block()

Irk indeed, I am blind. The patch does not fix it.

So just to confirm that this isn't a new regression, 2.6.34 fails in the same way. If I change the test to make the random writes overwrite existing blocks instead of filling holes, then there are no problems either.

Signature · 1 line
Jens Axboe
#10 lore raw
In reply to Jens Axboe
Quoted 8 lines · Jens Axboe

Hi,

I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and O_DIRECT.

Quoted 8 lines

It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:

Bad verify header 0 at 10137600
fio: pid=9943, err=84/file:io_u.c:1212, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character

writers: (groupid=0, jobs=32): err=84 (file:io_u.c:1212, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character): pid=9943

FYI:

I asked Jens to test hch's and Jiaying's aio completion patches with this, and apparently those fixed this problem for him.

-Eric

Quoted 43 lines

which tells us that where we expected to find the correct verify magic in the header, it was all zeroes. The job file used is below, and to reproduce you want to use the latest fio (1.40) since some earlier versions don't do verify_interval properly for non-pattern verifies. You can get fio here:

http://brick.kernel.dk/snaps/fio-1.40.tar.gz

or from git at:

git://git.kernel.dk/fio.git

The kernel used is 2.6.35-rc3 and I ran this on a raid0 that had 8 SSD drives.

--- snip job file ---

[global]
direct=1
group_reporting=1
exitall
runtime=4h
time_based=1

# writers, will repeatedly randomly write and verify data [writers]
rw=randwrite
bsrange=4k-16k
ioengine=libaio
iodepth=4
directory=/data
verify=crc32c
verify_backlog=1024
verify_backlog_batch=512
verify_interval=512
size=512m
nrfiles=4
filesize=64m-256m
numjobs=32
create_serialize=0

--- snip job file ---

author
#11 lore raw
Quoted 16 lines · Eric Sandeen
Quoted 15 lines · Jens Axboe

Hi,

I was writing a small fio job file to do writes and read verifies on a device. It forks 32 processes, each writing randomly to 4 files with a block size between 4k and 16k. When it has written 1024 of those blocks, it'll verify the oldest 512 of them. Each block is checksummed for every 512b. It uses libaio and O_DIRECT.

It works on ext2 and btrfs. I haven't run it to completion yet, but they survive 15-20 minutes just fine. ext4 doesn't even go a full minutes before this triggers:

Bad verify header 0 at 10137600
fio: pid=9943, err=84/file:io_u.c:1212, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character

Quoted 6 lines
Quoted 1 line

writers: (groupid=0, jobs=32): err=84 (file:io_u.c:1212, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character): pid=9943

FYI:

I asked Jens to test hch's and Jiaying's aio completion patches with this, and apparently those fixed this problem for him.

At least for a shorter run, but long enough that all the holes should have been filled at this point. So it at least fixes my test case. I can try and expand the run a bit if there's any interest in that, and see if that still verifies correctly.

Signature · 1 line
Jens Axboe