2.4.8 NFS Problems
Hi guys,
I was searching on google for some reports on the problem I'm seeing with our NFS server/clients and found this thread. It looked somewhat the same (atleast the result with the EIO is the same).
Parts of old message :
Quoted 7 lines
From: Mike Black ([email protected])
Date: Sep 05 2001I've been getting random NFS EIO errors for a few months but now it's repeatable.
Trying to copy a large file from one 2.4.8 SMP box to another is consistently failing (at different offsets >each time).
Our setup is like this :
Server:
RedHat 7.2 - kernel 2.4.9-13smp
nfs-utils-0.3.1-13.7.2.1
ext3 filesystem (73GB)Clients:
ia32 client - RedHat 6.2 - kernel 2.2.19-6.2.7enterprise mount-2.10r-0.6.x alpha client - RedHat 6.2 - kernel 2.2.19 (vanilla) mount-2.10r-5 ia64 client - RedHat 7.1 - kernel 2.4.3-12smp mount-2.10r-5
I've seen the "Input/Output error" problem only on the Alpha and the IA64 clients and the problem is occuring when making a static library (with 'ar'). The message is like this :
ar: xxxxxx/libmpi.a: Input/output error
The mountpoints is mounted like this :
ia32 client:
huey:/export/home/mpitest /home/mpitest nfs rw,v3,rsize=8192,wsize=8192,addr=huey 0 0
alpha client:
huey:/export/home/mpitest /home/mpitest nfs rw,v3,rsize=8192,wsize=8192,addr=huey 0 0
ia64 client:
huey:/export/home/mpitest /home/mpitest nfs rw,v3,rsize=8192,wsize=8192,hard,udp,lock,addr=huey 0 0
I don't know why the "hard" and "lock" options doesn't appear on ia32 and alpha, but this might be related to the /proc/mounts interface on the running kernel (these clients are running 2.2.19 while the ia64 client is running 2.4). The automount entry looks like this :
/home auto_home rsize=8192,wsize=8192
So according to the nfs man pages the "hard" option should be default :
hard If an NFS file operation has a major timeout then report "server not
responding" on the console and continue retrying indefinitely. This is the default.
So what could be the problem here ? Is it a NFS server bug, a NFS client bug or a NFS/ext3 bug ? We used to run RedHat 7.0 on this server with the 2.2.19-enterprise kernel, nfs-utils-0.3.1-7 and with a ext2 filesystem. This problem did not occur back then.
Thanks,
Signature · 4 lines
Steffen Persvold | Scalable Linux Systems | Try out the world's best mailto:[email protected] | http://www.scali.com | performing MPI implementation: Tel: (+47) 2262 8950 | Olaf Helsets vei 6 | - ScaMPI 1.12.2 - Fax: (+47) 2262 8951 | N0621 Oslo, NORWAY | >300MBytes/s and <4uS latency
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Steffen Persvold <[email protected]> writes:
I've been getting random NFS EIO errors for a few months but now it's repeatable. Trying to copy a large file from one 2.4.8 SMP box to another is consistently failing (at different offsets >each time).
Please try the patch on
http://www.fys.uio.no/~trondmy/src/2.4.17/linux-2.4.17-fattr.dif
that fixes at least 1 such EIO error which was discovered using fsx.
Cheers,
Trond
Quoted 8 lines · Trond Myklebust
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
" " == Steffen Persvold <[email protected]> writes:
I've been getting random NFS EIO errors for a few months but now it's repeatable. Trying to copy a large file from one 2.4.8 SMP box to another is consistently failing (at different offsets >each time).
Quoted 6 lines
Please try the patch on
http://www.fys.uio.no/~trondmy/src/2.4.17/linux-2.4.17-fattr.dif
that fixes at least 1 such EIO error which was discovered using fsx.
I can do that, but since one of the clients reporting this problem is an Alpha machine running 2.2.19 the patch won't do much good (not that the patch is architecture dependent, but it's only for 2.4.17). Has this patch been there since 2.2 or is it a new "feature" in the "stable" #:) 2.4 kernels.
Regards,
Signature · 4 lines
Steffen Persvold | Scalable Linux Systems | Try out the world's best mailto:[email protected] | http://www.scali.com | performing MPI implementation: Tel: (+47) 2262 8950 | Olaf Helsets vei 6 | - ScaMPI 1.12.2 - Fax: (+47) 2262 8951 | N0621 Oslo, NORWAY | >300MBytes/s and <4uS latency
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Steffen Persvold <[email protected]> writes:
> I can do that, but since one of the clients reporting this > problem is an Alpha machine running
> 2.2.19 the patch won't do much good (not that the patch is
> architecture dependent, but it's only for
> 2.4.17). Has this patch been there since 2.2 or is it a new > "feature" in the "stable" #:) 2.4 kernels.
All the problems fixed by the patch should be present in 2.2.19 too. I don't really have time to backport the whole thing, but I've appended a backport of the bit that is directly relevant to the EIO error.
Cheers,
Trond
fs/nfs/read.c renamed from fs/nfs/read.c.orig +8 -3
@@ -420,7 +420,7 @@ {struct nfs_read_data *data = (struct nfs_read_data *) task->tk_calldata;struct inode *inode = data->inode; -int count = data->res.count; +unsigned int count = data->res.count;dprintk ("NFS: %4d nfs_readpage_result, (status %d)\n" , task->tk_pid, task->tk_status);
@@ -431,10 +431,15 @@struct page *page = req->wb_page;nfs_list_remove_request (req); -if (task->tk_status >=0 && count >=0 ) { +if (task->tk_status >=0 ) { +char *p =page_address (page); +if (count <PAGE_CACHE_SIZE ) { +memset (p + count,0 ,PAGE_CACHE_SIZE - count); + count =0 ; + }else + count -=PAGE_CACHE_SIZE ;flush_dcache_page (page_address (page));/* Is this correct? */ set_bit (PG_uptodate, &page->flags); - count -=PAGE_CACHE_SIZE ; }else set_bit (PG_error, &page->flags);nfs_unlock_page (page);