direct IO question

DISCUSSION Utako Kusaka 2 participants 2 messages May 10, 2011

Share & tools

Utako Kusaka

author

2011-05-10 05:43 UTC #0 lore raw

Hi,

When I tested concurrent mmap write and direct IO to the same file, it was corrupted. Kernel version is 2.6.39-rc4.
I have two questions concerning xfs direct IO.

The first is dirty pages are released in direct read. xfs direct IO uses xfs_flushinval_pages(), which writes out and releases dirty pages. If pages are marked as dirty after filemap_write_and_wait_range(), they will be released in truncate_inode_pages_range() without writing out.

sys_read()
vfs_read()

    do_sync_read()
      xfs_file_aio_read()
        xfs_flushinval_pages()
          filemap_write_and_wait_range()
          truncate_inode_pages_range()      <---
        generic_file_aio_read()
          filemap_write_and_wait_range()
          xfs_vm_direct_IO()

ext3 calls generic_file_aio_read() only and does not call truncate_inode_pages_range().

sys_read()
vfs_read()

    do_sync_read()
      generic_file_aio_read()
        filemap_write_and_wait_range()
        ext3_direct_IO()

xfs_file_aio_read() and xfs_file_dio_aio_write() call generic function. And both xfs functions and generic functions call filemap_write_and_wait_range(). So I wonder whether xfs_flushinval_pages() is necessary.

Then, the write range in xfs_flushinval_pages() called from direct IO is from start pos to -1, or LLONG_MAX, and is not IO range. Is there any reason? In generic_file_aio_read and generic_file_direct_write(), it is from start pos to (pos + len - 1).
I think xfs_flushinval_pages() should be called with same range.

Regards,
Utako

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Dave Chinner

2011-05-10 13:23 UTC #1 lore raw

Quoted 4 lines · Utako Kusaka

Hi,
When I tested concurrent mmap write and direct IO to the same file, it was corrupted. Kernel version is 2.6.39-rc4.

Long time problem of the mmap_sem being held while .page_mkwrite is called, which means we can't use the i_mutex or xfs inode iolock for serialisation against reads and writes because the mmap_sem can be taken on page faults during read or write. Hence we've got the choice of deadlocks or no serialisation between direct Io and mmap...

Quoted 4 lines

I have two questions concerning xfs direct IO.
The first is dirty pages are released in direct read. xfs direct IO uses xfs_flushinval_pages(), which writes out and releases dirty pages.

Yup - once you bypass the page cache, it is stale and needs to be removed from memory so it can be reread from disk when the next buffered IO occurs.

Quoted 2 lines

If pages are marked as dirty after filemap_write_and_wait_range(), they will be released in truncate_inode_pages_range() without writing out.

If .page_mkwrite could take either the iolock or the i_mutex, it would be protected against this like all other operations are.

Quoted 15 lines

sys_read()
vfs_read()

    do_sync_read()
      xfs_file_aio_read()
        xfs_flushinval_pages()
          filemap_write_and_wait_range()
          truncate_inode_pages_range()      <---
        generic_file_aio_read()
          filemap_write_and_wait_range()
          xfs_vm_direct_IO()

ext3 calls generic_file_aio_read() only and does not call truncate_inode_pages_range().

Quoted 6 lines

sys_read()
vfs_read()

    do_sync_read()
      generic_file_aio_read()
        filemap_write_and_wait_range()
        ext3_direct_IO()

ext3 is vastly different w.r.t. direct IO functionality, and so can't be directly compared against XFS behaviour.

Quoted 3 lines

xfs_file_aio_read() and xfs_file_dio_aio_write() call generic function. And both xfs functions and generic functions call filemap_write_and_wait_range(). So I wonder whether xfs_flushinval_pages() is necessary.

The data corruption it fixed long ago woul dprobably return in some form...

Quoted 5 lines

Then, the write range in xfs_flushinval_pages() called from direct IO is from start pos to -1, or LLONG_MAX, and is not IO range. Is there any reason? In generic_file_aio_read and generic_file_direct_write(), it is from start pos to (pos + len - 1).
I think xfs_flushinval_pages() should be called with same range.

Probably should be, but it will need significant testing to ensure that it doesn't intorduce a new coherency/corruption corner case...

Cheers,

Dave.

Signature · 7 lines

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs