Still data corruption with LTP doio on 2.6.17rc
Subsystems (2)
I'm still seeing data corruption when running LTP over NFS between two 2.6.17rc* hosts. I already saw this before 2.6.16 and reported.
Server is running knfsd 2.6.17-rc4-git9, client is running 2.6.17-rc6
with nfsroot. Both x86-64 and SUSE 10.0 userland. The file system
is exported as async and mounted with/dev/root / nfs rw,vers=2,rsize=4096,wsize=4096,hard,nolock,proto=udp,timeo=11,retrans=2,addr=10.23.204.1 0 0
First I always get lots of
do_vfs_lock: VFS is out of sync with lock manager!
messages on the client. They don't seem to be directly related though.
I set up ltp-full-20051103 on the NFS root and run it on the client with runltplite.sh. Eventually it reports
<<<test_start>>>
tag=rwtest03 stime=1149754762
cmdline="export LTPROOT; rwtest -N rwtest03 -c -q -i 60s -n 2 -f buffered -s mmread,mmwrite -m random -Dv 10%25000:mm-buff-$$" contacts=""
analysis=exit
initiation_status="ok"
<<<test_output>>>
doio(rwtest03) ( 8155) 08:19:23
---------------------
*** DATA COMPARISON ERROR ***check_file(/tmp/ltp-2256/mm-buff-8139, 7813848, 81293, U:8155:bigfoot:doio*, 20, 0) failed
Comparison fd is 3, with open flags 0
Corrupt regions follow - unprintable chars are represented as '.'
-----------------------------------------------------------------
corrupt bytes starting at file offset 7813848
1st 32 expected bytes: U:8155:bigfoot:doio*U:8155:bigfo
1st 32 actual bytes: ................................Request number 36
fd 4 is file /tmp/ltp-2256/mm-buff-8139 - open flags are 02 O_RDWR,
write done at file offset 7813848 - pattern is U (0125)
number of requests is 1, strides per request is 1
i/o byte count = 81293
memory alignment is unalignedsyscall: mmap-write(NULL, 12800000, PROT_WRITE, MAP_SHARED, 4, 0)
file is mmaped to: 0x2b73b87f0000
file-mem=0x2b73b8f63ad8, length=81293, buffer=0x52d540est03) ( 8152) 08:19:23
---------------------
(parent) pid 8155 exited because of data compare errors
doio(rwtest03) ( 8154) 08:19:23
---------------------
*** DATA COMPARISON ERROR ***check_file(/tmp/ltp-2256/mm-buff-8139, 4223869, 130577, X:8154:bigfoot:doio*, 20, 0) failed
Comparison fd is 4, with open flags 0
Corrupt regions follow - unprintable chars are represented as '.'
-----------------------------------------------------------------
corrupt bytes starting at file offset 4349952
1st 32 expected bytes: 154:bigfoot:doio*X:8154:bigfoot:
1st 32 actual bytes: ................................Request number 36
fd 3 is file /tmp/ltp-2256/mm-buff-8139 - open flags are 02 O_RDWR,
write done at file offset 4223869 - pattern is X (0130)
number of requests is 1, strides per request is 1
i/o byte count = 130577
memory alignment is unalignedsyscall: mmap-write(NULL, 12800000, PROT_WRITE, MAP_SHARED, 3, 0)
file is mmaped to: 0x2b73b87f0000
file-mem=0x2b73b8bf737d, length=130577, buffer=0x52d540doio(rwtest03) ( 8152) 08:19:23
---------------------
(parent) pid 8154 exited because of data compare errors
rwtest(rwtest03) : iogen reported errors (r=141)
rwtest03 1 FAIL : Test failed
<<<execution_status>>>
duration=1 termination_type=exited termination_id=2 corefile=no
cutime=1 cstime=7
<<<test_end>>>
<<<test_start>>>
tag=rwtest04 stime=1149754763
cmdline="export LTPROOT; rwtest -N rwtest04 -c -q -i 60s -n 2 -f sync -s mmread,mmwrite -m random -Dv 10%25000:mm-sync-$$" contacts=""
analysis=exit
initiation_status="ok"
<<<test_output>>>
doio(rwtest04) ( 8174) 08:19:23
---------------------
*** DATA COMPARISON ERROR ***check_file(/tmp/ltp-2256/mm-sync-8159, 12102771, 46088, O:8174:bigfoot:doio*, 20, 0) failed
Comparison fd is 4, with open flags 0
Corrupt regions follow - unprintable chars are represented as '.'
-----------------------------------------------------------------
corrupt bytes starting at file offset 12128256
1st 32 expected bytes: 4:bigfoot:doio*O:8174:bigfoot:do
1st 32 actual bytes: ................................Request number 11
fd 3 is file /tmp/ltp-2256/mm-sync-8159 - open flags are 010002 O_RDWR,O_SYNC,
write done at file offset 12102771 - pattern is O (0117)
number of requests is 1, strides per request is 1
i/o byte count = 46088
memory alignment is unalignedsyscall: mmap-write(NULL, 12800000, PROT_WRITE, MAP_SHARED, 3, 0)
file is mmaped to: 0x2b8b163cc000
file-mem=0x2b8b16f56c73, length=46088, buffer=0x52d546doio(rwtest04) ( 8172) 08:19:23
---------------------
(parent) pid 8174 exited because of data compare errors
rwtest(rwtest04) : doio reported errors (r=4) rwtest04 1 FAIL : doio reported errors (r=4) rwtest04 1 FAIL : Test failed
<<<execution_status>>>
duration=66 termination_type=exited termination_id=4 corefile=no cutime=33 cstime=170
<<<test_end>>>
-Andi
Quoted 46 lines · Andi Kleen
I'm still seeing data corruption when running LTP over NFS between two 2.6.17rc* hosts. I already saw this before 2.6.16 and reported.
Server is running knfsd 2.6.17-rc4-git9, client is running 2.6.17-rc6
with nfsroot. Both x86-64 and SUSE 10.0 userland. The file system
is exported as async and mounted with/dev/root/ nfs rw,vers=2,rsize=4096,wsize=4096,hard,nolock,proto=udp,timeo=11,retrans=2,addr=10.23.204.1 0 0First I always get lots of
do_vfs_lock: VFS is out of sync with lock manager!messages on the client. They don't seem to be directly related though.
I set up ltp-full-20051103 on the NFS root and run it on the client with
runltplite.sh. Eventually it reports<<<
test_start>>>
tag=rwtest03 stime=1149754762
cmdline="export LTPROOT; rwtest -N rwtest03 -c -q -i 60s -n 2 -f buffered -s mmread,mmwrite -m random -Dv 10%25000:mm-buff-$$" contacts=""
analysis=exit
initiation_status="ok"
<<<test_output>>>doio(rwtest03) ( 8155) 08:19:23
---------------------
*** DATA COMPARISON ERROR ***check_file(/tmp/ltp-2256/mm-buff-8139, 7813848, 81293, U:8155:bigfoot:doio*, 20, 0)failedComparison fd is 3, with open flags 0
Corrupt regions follow - unprintable chars are represented as '.'
-----------------------------------------------------------------
corrupt bytes starting at file offset 78138481st 32 expected bytes: U:8155:bigfoot:doio*U:8155:bigfo 1st 32 actual bytes: ................................Request number 36
fd 4 is file /tmp/ltp-2256/mm-buff-8139 - open flags are 02 O_RDWR, write done at file offset 7813848 - pattern is U (0125) number of requests is 1, strides per request is 1 i/o byte count = 81293 memory alignment is unaligned
Quoted 8 lines
syscall: mmap-write(NULL, 12800000,
PROT_WRITE,MAP_SHARED, 4, 0)file is mmaped to: 0x2b73b87f0000 file-mem=0x2b73b8f63ad8, length=81293, buffer=0x52d540est03) ( 8152) 08:19:23
---------------------
(parent) pid 8155 exited because of data compare errors
mmap() is still not 100% safe when the client believes that the file has changed on the server and needs to call invalidate_inode_pages2(): if there is dirty data on the page when unmap_mapping_range() gets called, then that dirty data may be lost (basically, we need a VM helper that does unmap+flush+invalidate_page).
The attached patch may, however reduce the number of calls to invalidate_inode_pages2() since it ensures that revalidations only occur if and when we're going to read from the page cache (i.e. in places when we _need_ the assurance that the page cache is valid).
Cheers,
Trond
NFS: Separate metadata and page cache revalidation mechanisms
From: Trond Myklebust <[email protected]>
Separate out the function of revalidating the inode metadata, and revalidating the mapping. The former may be called by lookup(), and only really needs to check that permissions, ctime, etc haven't changed whereas the latter needs only done when we want to read data from the page cache, and may need to sync and then invalidate the mapping.
---
fs/nfs/dir.c | 2 +- fs/nfs/file.c | 24 +++--------------------- fs/nfs/inode.c | 16 +++++++++++----- fs/nfs/symlink.c | 2 +- include/linux/nfs_fs.h | 2 +- 5 files changed, 17 insertions(+), 29 deletions(-)
fs/nfs/dir.c +1 -1
@@ -528,7 +528,7 @@ static int nfs_readdir(struct file *filplock_kernel (); - res =nfs_revalidate_inode (NFS_SERVER (inode), inode); + res =nfs_revalidate_mapping (inode, filp->f_mapping);if (res <0 ) {unlock_kernel ();return res;
fs/nfs/file.c +3 -21
@@ -127,23 +127,6 @@ nfs_file_release(struct inode *inode, st }/** -* nfs_revalidate_file - Revalidate the page cache & related metadata -* @inode - pointer to inode struct -* @file - pointer to file -*/ -static int nfs_revalidate_file (struct inode *inode,struct file *filp) -{ -struct nfs_inode *nfsi =NFS_I (inode); -int retval =0 ; - -if ((nfsi->cache_validity & (NFS_INO_REVAL_PAGECACHE |NFS_INO_INVALID_ATTR )) - ||nfs_attribute_timeout (inode)) - retval =__nfs_revalidate_inode (NFS_SERVER (inode), inode); -nfs_revalidate_mapping (inode, filp->f_mapping); -return 0 ; -} - -/** * nfs_revalidate_size - Revalidate the file size * @inode - pointer to inode struct * @file - pointer to struct file
@@ -228,7 +211,7 @@ #endif dentry->d_parent->d_name.name, dentry->d_name.name, (unsigned long) count, (unsigned long) pos); - result =nfs_revalidate_file (inode, iocb->ki_filp); + result =nfs_revalidate_mapping (inode, iocb->ki_filp->f_mapping);nfs_add_stats (inode,NFSIOS_NORMALREADBYTES , count);if (!result) result =generic_file_aio_read (iocb, buf, count, pos);
@@ -247,7 +230,7 @@ nfs_file_sendfile(struct file *filp, lof dentry->d_parent->d_name.name, dentry->d_name.name, (unsigned long) count, (unsigned long long) *ppos); - res =nfs_revalidate_file (inode, filp); + res =nfs_revalidate_mapping (inode, filp->f_mapping);if (!res) res =generic_file_sendfile (filp, ppos, count, actor, target);return res;
@@ -263,7 +246,7 @@ nfs_file_mmap(struct file * file, structdfprintk (VFS ,"nfs: mmap(%s/%s)\n" , dentry->d_parent->d_name.name, dentry->d_name.name); - status =nfs_revalidate_file (inode, file); + status =nfs_revalidate_mapping (inode, file->f_mapping);if (!status) status =generic_file_mmap (file, vma);return status;
@@ -373,7 +356,6 @@ #endifif (result) goto out; } -nfs_revalidate_mapping (inode, iocb->ki_filp->f_mapping); result = count;if (!count)
fs/nfs/inode.c +11 -5
@@ -1220,7 +1220,7 @@ __nfs_revalidate_inode(struct nfs_server status = -ESTALE ;/* Do we trust the cached ESTALE? */ if (NFS_ATTRTIMEO (inode) !=0 ) { -if (nfsi->cache_validity & (NFS_INO_INVALID_ATTR |NFS_INO_INVALID_DATA |NFS_INO_INVALID_ATIME )) { +if (nfsi->cache_validity & (NFS_INO_INVALID_ATTR |NFS_INO_INVALID_ATIME )) {/* no */ }else goto out;
@@ -1251,8 +1251,6 @@ __nfs_revalidate_inode(struct nfs_server }spin_unlock (&inode->i_lock); -nfs_revalidate_mapping (inode, inode->i_mapping); -if (nfsi->cache_validity &NFS_INO_INVALID_ACL )nfs_zap_acl_cache (inode);
@@ -1287,7 +1285,7 @@ int nfs_attribute_timeout(struct inode *int nfs_revalidate_inode (struct nfs_server *server,struct inode *inode) {nfs_inc_stats (inode,NFSIOS_INODEREVALIDATE ); -if (!(NFS_I (inode)->cache_validity & (NFS_INO_INVALID_ATTR |NFS_INO_INVALID_DATA )) +if (!(NFS_I (inode)->cache_validity &NFS_INO_INVALID_ATTR ) && !nfs_attribute_timeout (inode))return NFS_STALE (inode) ? -ESTALE :0 ;return __nfs_revalidate_inode (server, inode);
@@ -1298,9 +1296,16 @@ int nfs_revalidate_inode(struct nfs_serv* @inode - pointer to host inode * @mapping - pointer to mapping */ -void nfs_revalidate_mapping (struct inode *inode,struct address_space *mapping) +int nfs_revalidate_mapping (struct inode *inode,struct address_space *mapping) {struct nfs_inode *nfsi =NFS_I (inode); +int ret =0 ; + +if (NFS_STALE (inode)) + ret = -ESTALE ; +if ((nfsi->cache_validity &NFS_INO_REVAL_PAGECACHE ) + ||nfs_attribute_timeout (inode)) + ret =__nfs_revalidate_inode (NFS_SERVER (inode), inode);if (nfsi->cache_validity &NFS_INO_INVALID_DATA ) {nfs_inc_stats (inode,NFSIOS_DATAINVALIDATE );
@@ -1321,6 +1326,7 @@ void nfs_revalidate_mapping(struct inode inode->i_sb->s_id, (long long)NFS_FILEID(inode)); } +return ret; }/**
fs/nfs/symlink.c +1 -1
@@ -52,7 +52,7 @@ static void *nfs_follow_link(struct dent {struct inode *inode = dentry->d_inode;struct page *page; -void *err =ERR_PTR (nfs_revalidate_inode (NFS_SERVER (inode), inode)); +void *err =ERR_PTR (nfs_revalidate_mapping (inode, inode->i_mapping));if (err) goto read_failed; page = read_cache_page(&inode->i_data, 0,
include/linux/nfs_fs.h +1 -1
@@ -301,7 +301,7 @@ extern int nfs_release(struct inode *, sextern int nfs_attribute_timeout (struct inode *inode);extern int nfs_revalidate_inode (struct nfs_server *server,struct inode *inode);extern int __nfs_revalidate_inode (struct nfs_server *,struct inode *); -extern void nfs_revalidate_mapping (struct inode *inode,struct address_space *mapping); +extern int nfs_revalidate_mapping (struct inode *inode,struct address_space *mapping);extern int nfs_setattr (struct dentry *,struct iattr *);extern void nfs_setattr_update_inode (struct inode *inode,struct iattr *attr);extern void nfs_begin_attr_update (struct inode *);
Quoted 62 lines · Trond Myklebust
Quoted 46 lines · Andi Kleen
I'm still seeing data corruption when running LTP over NFS between two 2.6.17rc* hosts. I already saw this before 2.6.16 and reported.
Server is running knfsd 2.6.17-rc4-git9, client is running 2.6.17-rc6
with nfsroot. Both x86-64 and SUSE 10.0 userland. The file system
is exported as async and mounted with/dev/root/ nfs rw,vers=2,rsize=4096,wsize=4096,hard,nolock,proto=udp,timeo=11,retrans=2,addr=10.23.204.1 0 0First I always get lots of
do_vfs_lock: VFS is out of sync with lock manager!messages on the client. They don't seem to be directly related though.
I set up ltp-full-20051103 on the NFS root and run it on the client with
runltplite.sh. Eventually it reports<<<
test_start>>>
tag=rwtest03 stime=1149754762
cmdline="export LTPROOT; rwtest -N rwtest03 -c -q -i 60s -n 2 -f buffered -s mmread,mmwrite -m random -Dv 10%25000:mm-buff-$$" contacts=""
analysis=exit
initiation_status="ok"
<<<test_output>>>doio(rwtest03) ( 8155) 08:19:23
---------------------
*** DATA COMPARISON ERROR ***check_file(/tmp/ltp-2256/mm-buff-8139, 7813848, 81293, U:8155:bigfoot:doio*, 20, 0)failedComparison fd is 3, with open flags 0
Corrupt regions follow - unprintable chars are represented as '.'
-----------------------------------------------------------------
corrupt bytes starting at file offset 78138481st 32 expected bytes: U:8155:bigfoot:doio*U:8155:bigfo 1st 32 actual bytes: ................................Request number 36
fd 4 is file /tmp/ltp-2256/mm-buff-8139 - open flags are 02 O_RDWR, write done at file offset 7813848 - pattern is U (0125) number of requests is 1, strides per request is 1 i/o byte count = 81293 memory alignment is unalignedQuoted 8 lines
syscall: mmap-write(NULL, 12800000,
PROT_WRITE,MAP_SHARED, 4, 0)file is mmaped to: 0x2b73b87f0000 file-mem=0x2b73b8f63ad8, length=81293, buffer=0x52d540est03) ( 8152) 08:19:23
---------------------
(parent) pid 8155 exited because of data compare errors
mmap()is still not 100% safe when the client believes that the file has changed on the server and needs to callinvalidate_inode_pages2(): if there is dirty data on the page whenunmap_mapping_range()gets called, then that dirty data may be lost (basically, we need a VM helper that does unmap+flush+invalidate_page).
Quoted 4 lines
The attached patch may, however reduce the number of calls to
invalidate_inode_pages2()since it ensures that revalidations only occur if and when we're going to read from the page cache (i.e. in places when we _need_ the assurance that the page cache is valid).
Sorry for the delay.
I tested your patch with -rc6 and the problem is still there:
Also I did some testing with 2.6.16 and I couldn't reproduce it here so maybe it was introduced afterwards (however I could swear I've seen it before, but I can't remember which exact version number it was)
BTW you can probably easily reproduce it yourself by downloading LTP from ltp.sourceforge.net and compiling/running it on a NFS mount.
-Andi
doio(rwtest03) ( 8621) 07:42:18
---------------------
*** DATA COMPARISON ERROR ***check_file(/tmp/ltp-2819/mm-buff-8605, 507558, 49196, O:8621:bigfoot:doio*, 20, 0) failed
Comparison fd is 4, with open flags 0
Corrupt regions follow - unprintable chars are represented as '.'
-----------------------------------------------------------------
corrupt bytes starting at file offset 532480
1st 32 expected bytes: 8621:bigfoot:doio*O:8621:bigfoot
1st 32 actual bytes: ................................Request number 1
fd 3 is file /tmp/ltp-2819/mm-buff-8605 - open flags are 02 O_RDWR, write done at file offset 507558 - pattern is O (0117) number of requests is 1, strides per request is 1
i/o byte count = 49196
memory alignment is unaligned
syscall: mmap-write(NULL, 12800000, PROT_WRITE, MAP_SHARED, 3, 0)
file is mmaped to: 0x2b0b89f6f000
file-mem=0x2b0b89feaea6, length=49196, buffer=0x52d547doio(rwtest03) ( 8618) 07:42:18
---------------------
(parent) pid 8621 exited because of data compare errors
rwtest(rwtest03) : doio reported errors (r=4) rwtest03 1 FAIL : doio reported errors (r=4) rwtest03 1 FAIL : Test failed
<<<execution_status>>>
duration=67 termination_type=exited termination_id=4 corefile=no
cutime=32 cstime=141
<<<test_end>>>
<<<test_start>>>
tag=rwtest04 stime=1150098204
cmdline="export LTPROOT; rwtest -N rwtest04 -c -q -i 60s -n 2 -f sync -s mmread,mmwrite -m random -Dv 10%25000:m m-sync-$$"
contacts=""
analysis=exit
initiation_status="ok"
<<<test_output>>>
doio(rwtest04) ( 8640) 07:43:26
---------------------
*** DATA COMPARISON ERROR ***check_file(/tmp/ltp-2819/mm-sync-8625, 4585176, 82650, T:8640:bigfoot:doio*, 20, 0) failed
Comparison fd is 4, with open flags 0
Corrupt regions follow - unprintable chars are represented as '.'
-----------------------------------------------------------------
corrupt bytes starting at file offset 4585176
1st 32 expected bytes: T:8640:bigfoot:doio*T:8640:bigfo
1st 32 actual bytes: ................................Request number 54
fd 3 is file /tmp/ltp-2819/mm-sync-8625 - open flags are 010002 O_RDWR,O_SYNC,
write done at file offset 4585176 - pattern is T (0124)
number of requests is 1, strides per request is 1
i/o byte count = 82650
memory alignment is unalignedsyscall: mmap-write(NULL, 12800000, PROT_WRITE, MAP_SHARED, 3, 0)
file is mmaped to: 0x2addadf68000
file-mem=0x2addae3c76d8, length=82650, buffer=0x52d541doio(rwtest04) ( 8638) 07:43:26
---------------------
(parent) pid 8640 exited because of data compare errors
doio(rwtest04) ( 8641) 07:43:26
---------------------
*** DATA COMPARISON ERROR ***check_file(/tmp/ltp-2819/mm-sync-8625, 7161356, 118706, E:8641:bigfoot:doio*, 20, 0) failed Comparison fd is 4, with open flags 0
Corrupt regions follow - unprintable chars are represented as '.'
-----------------------------------------------------------------
corrupt bytes starting at file offset 7254016
1st 32 expected bytes: E:8641:bigfoot:doio*E:8641:bigfo
1st 32 actual bytes: ................................Request number 61
fd 3 is file /tmp/ltp-2819/mm-sync-8625 - open flags are 010002 O_RDWR,O_SYNC,
write done at file offset 7161356 - pattern is E (0105)
number of requests is 1, strides per request is 1
i/o byte count = 118706
memory alignment is unalignedsyscall: mmap-write(NULL, 12800000, PROT_WRITE, MAP_SHARED, 3, 0)
file is mmaped to: 0x2addadf68000
file-mem=0x2addae63c60c, length=118706, buffer=0x52d543doio(rwtest04) ( 8638) 07:43:26
---------------------
(parent) pid 8641 exited because of data compare errors rwtest(rwtest04) : iogen reported errors (r=141)
rwtest04 1 FAIL : Test failed