Skip to contentSkip to conversation

Some O_DIRECT vs. block_truncate_page problems

PATCH Eric Sandeen 2 participants 3 messages Sep 23, 2002 – Sep 24, 2002 VFS

lore
Share & tools
lore
#0 lore raw

I'm trying to track down a bug I'm seeing after I run xfs's defragmenter, where data is present past EOF in the last block.

xfs_fsr does this (essentially):

o allocate some contiguous space with an xfs-specific ioctl
o copy the fragmented file into the new file with O_DIRECT
o truncate() the new file back to the correct size (since it's currently a multiple of the block size)
o sync the new file
o do "xfs_inval_cached_pages", which is essentially:
- filemap_fdatasync(ip->i_mapping);
- fsync_inode_data_buffers(ip);
- filemap_fdatawait(ip->i_mapping);
- truncate_inode_pages(ip->i_mapping, first);
o swap the extents between the old & the new files

When this is all done, I see data past EOF in the resulting file.

I believe that this is because the buffer from the truncate() call is not being synced to disk, and gets thrown away when we do truncate_inode_pages().

block_truncate_page() does a __mark_buffer_dirty(bh) at the end, but it does not file the buffer on the inode's dirty data queue, so the fsync_inode_data_buffers() does not see it. The inode only has a page on it's clean pages list, so filemap_fdatasync does not see it either.

Since generic_file_direct_IO expects to do filemap_fdatasync, fsync_inode_data_buffers, filemap_fdatawait to flush data to disk, I think that the behavior after a truncate() will be broken for any filesystem.

This patch seems to fix things up for me; is there a reason that the buffer was not inserted originally?

fs/buffer.c renamed from fs/buffer.c_1.109 +6 -1
@@ -2028,7 +2028,12 @@
 	flush_dcache_page(page);
 	kunmap(page);
 
-	__mark_buffer_dirty(bh);
+	if (!atomic_set_buffer_dirty(bh)) {
+		__mark_dirty(bh);
+		buffer_insert_inode_data_queue(bh, inode);
+		balance_dirty();
+	}
+
 	err = 0;
 
 

On a related note, while testing some O_DIRECT cases on ext2, I found another, possibly related, bug.

If you O_DIRECT write 2 blocks into a new file, truncate() the file to 1.5 blocks, then do an O_DIRECT read of the last block, you get incorrect data. Rather than 1/2 data, 1/2 zeroes, I get all zeroes. This test passes on xfs.

Comments?

-Eric

Signature · 2 lines
Eric Sandeen      XFS for Linux     http://oss.sgi.com/projects/xfs
[email protected]   SGI, Inc.         651-683-3102
#1 lore raw
Quoted 6 lines · Eric Sandeen

I'm trying to track down a bug I'm seeing after I run xfs's defragmenter, where data is present past EOF in the last block.

xfs_fsr does this (essentially):

Quoted 10 lines

o allocate some contiguous space with an xfs-specific ioctl
o copy the fragmented file into the new file with O_DIRECT
o truncate() the new file back to the correct size (since it's currently a multiple of the block size)
o sync the new file
o do "xfs_inval_cached_pages", which is essentially:
- filemap_fdatasync(ip->i_mapping);
- fsync_inode_data_buffers(ip);
- filemap_fdatawait(ip->i_mapping);
- truncate_inode_pages(ip->i_mapping, first);

What is "first"? It should be i_size. If it's zero then yes, you lost the dirty pagecache and dirty buffer.

Quoted 6 lines

...

fs/buffer.c Mon Sep 23 13:04:44 2002 renamed from fs/buffer.c_1.109 Mon Sep 23 13:10:56 2002 +0 -0
@@ -2028,7 +2028,12 @@
        flush_dcache_page(page);

fs/buffer.c Mon Sep 23 13:04:44 2002 line 2037

Quoted 10 lines
fs/buffer.c Mon Sep 23 13:04:44 2002 renamed from fs/buffer.c_1.109 Mon Sep 23 13:10:56 2002 +6 -1
        kunmap(page);
 
-       __mark_buffer_dirty(bh);
+       if (!atomic_set_buffer_dirty(bh)) {
+               __mark_dirty(bh);
+               buffer_insert_inode_data_queue(bh, inode);
+               balance_dirty();
+       }
+
        err = 0;

Makes sense I guess. I'd be inclined to leave the balance_dirty() out of there, just because it does add a tiny little more risk.

Quoted 6 lines

On a related note, while testing some O_DIRECT cases on ext2, I found another, possibly related, bug.

If you O_DIRECT write 2 blocks into a new file, truncate() the file to 1.5 blocks, then do an O_DIRECT read of the last block, you get incorrect data. Rather than 1/2 data, 1/2 zeroes, I get all zeroes.

Does this still happen with your patch? I'd expect that with your patch, you would see all data, not all zeroes.

What blocksize and pagesize are you using, btw?

#2 lore raw

Re: Some O_DIRECT vs. block_truncate_page problemsr

Quoted 9 lines · Andrew Morton
Quoted 5 lines · Eric Sandeen

o do "xfs_inval_cached_pages", which is essentially:
- filemap_fdatasync(ip->i_mapping);
- fsync_inode_data_buffers(ip);
- filemap_fdatawait(ip->i_mapping);
- truncate_inode_pages(ip->i_mapping, first);

What is "first"? It should be i_size. If it's zero then yes, you lost the dirty pagecache and dirty buffer.

It's zero, but if we just synced the inode as above, there should be no dirty data left, right?

The idea is that since xfs is about to swap the extents out from under the inode, there had better be no old buffers floating around destined for the old extents.

<patch snipped>

Quoted 2 lines

Makes sense I guess. I'd be inclined to leave the balance_dirty() out of there, just because it does add a tiny little more risk.

I wondered about that.

Quoted 6 lines
Quoted 3 lines

If you O_DIRECT write 2 blocks into a new file, truncate() the file to 1.5 blocks, then do an O_DIRECT read of the last block, you get incorrect data. Rather than 1/2 data, 1/2 zeroes, I get all zeroes.

Does this still happen with your patch? I'd expect that with your patch, you would see all data, not all zeroes.

Yes, it does still happen. I expected my patch to fix this, but it seems to have no effect.

Quoted 1 line

What blocksize and pagesize are you using, btw?

4k pages, and I have tested with 1k and 4k block sizes.

For both xfs and ext2, the O_DIRECT read() returns 0, (sine we hit EOF?) but in the xfs case, we see the data in the proper place. It behaves this way on Irix as well. Apparently on freebsd, the same test also returns the data as expected, and the read() returns the # of bytes of data read. I haven't tested on solaris yet.

Is this sort of behavior for O_DIRECT defined anywhere, I wonder...

-Eric