XFS buffer IO performance is very poor
In functionxfs_file_aio_read, will requestXFS_IOLOCK_SHARED lock for both direct IO and buffered IO:
STATIC ssize_t
xfs_file_aio_read(
struct kiocb *iocb,
const struct iovec *iovp,
unsigned long nr_segs,loff_t pos)
{
...xfs_rw_ilock(ip, XFS_IOLOCK_SHARED);
so write will prevent read in XFS.
However, in function generic_file_aio_read for ext3, will not lockinode-i_mutex, so write will not prevent read in ext3.
I think this maybe the reason of poor performance for XFS. I do not know if this is a bug, or design flaws of XFS.
Best regards,
yy
原始邮件
发件人:[email protected]
收件人:[email protected]; Eric [email protected]; [email protected] 发送时间:2015年2月12日(周四) 13:30
主题:Re: XFS buffer IO performance is very poor
Brian and Eric,
Thanks very much for your replay.
I changed partition start position with 256K, but the performance is still poor, no change.
# fdisk -ul /dev/sdb Device Boot Start End Blocks Id System /dev/sdb1 512 2929356359 1464677924 83 Linux
I checked the XFS’s code, I agree thatXFS_IOLOCK_EXCL lock maybe the reason:
https://bitbucket.org/hustcat/kernel-2.6.32/src/786d720807052737bb17bc44da9da20554400039/fs/xfs/linux-2.6/xfs_file.c?at=master#cl-714
STATIC ssize_t
xfs_file_buffered_aio_write(
struct kiocb *iocb,
const struct iovec *iovp,
unsigned long nr_segs,loff_t pos,size_t ocount)
{
struct file *file = iocb-ki_filp;
struct address_space *mapping = file-f_mapping;
struct inode *inode = mapping-host;
struct xfs_inode *ip = XFS_I(inode);ssize_t ret;
int enospc = 0;
int iolock = XFS_IOLOCK_EXCL;size_t count = ocount;
xfs_rw_ilock(ip, iolock);
ret = xfs_file_aio_write_checks(file, pos, count, iolock); if (ret)
However,I found that EXT3 also have mutex when with buffered IO:
https://bitbucket.org/hustcat/kernel-2.6.32/src/786d720807052737bb17bc44da9da20554400039/mm/filemap.c?at=master#cl-2642ssize_t generic_file_aio_write(struct kiocb *iocb, const struct iovec *iov, unsigned long nr_segs, loff_t pos)
{
struct file *file = iocb-ki_filp;
struct inode *inode = file-f_mapping-host;ssize_t ret;
BUG_ON(iocb-ki_pos != pos);
sb_start_write(inode-i_sb);mutex_lock(inode-i_mutex);
ret = __generic_file_aio_write(iocb, iov, nr_segs, iocb-ki_pos); mutex_unlock(inode-i_mutex);
I still don’t understand why ext3 does not have this problem with buffered IO.
Best regards,
yy
原始邮件
发件人:Eric [email protected]
收件人:[email protected]; [email protected]
发送时间:2015年2月12日(周四) 00:08
主题:Re: XFS buffer IO performance is very poor
On 2/11/15 1:39 AM, yy wrote: snip (In addition to Brian's questions): XFS format parametes: #mkfs.xfs -d su=256k,sw=5 /dev/sdb1 #cat /proc/mounts /dev/sdb1 /data1 xfs rw,noatime,attr2,delaylog,nobarrier,logbsize=256k,sunit=512,swidth=2560,noquota 0 0 #fdisk -ul Device Boot Start End Blocks Id System /dev/sdb1 128 2929356359 1464678116 83 Linux so 128*512 = 64k; your partition doesn't start on a 256k stripe unit boundary, right? Shouldn't it do so? -Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · yy
In
functionxfs_file_aio_read, willrequestXFS_IOLOCK_SHAREDlock for both direct IO and buffered IO:so write will prevent read in XFS.
However, in function
generic_file_aio_readfor ext3, will not lockinode-i_mutex, so write will not prevent read in ext3.I think this maybe the reason of poor performance for XFS. I do not know if this is a bug, or design flaws of XFS.
This is a bug and design flaw in ext3, and most other Linux filesystems. Posix states that write() must execute atomically and so no concurrent operation that reads or modifies data should should see a partial write. The linux page cache doesn't enforce this - a read to the same range as a write can return partially written data on page granularity, as read/write only serialise on page locks in the page cache.
XFS is the only Linux filesystem that actually follows POSIX requirements here - the shared/exclusive locking guarantees that a buffer write completes wholly before a read is allowed to access the data. There is a down side - you can't run concurrent buffered reads and writes to the same file - if you need to do that then that's what direct IO is for, and coherency between overlapping reads and writes is then the application's problem, not the filesystem...
Maybe at some point in the future we might address this with ranged IO locks, but there really aren't many multithreaded programs that hit this issue...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs