NFS hang in 2.6.21.1
Subsystems (2)
Team,
I have been attempting to put together a linux HA NFS server for my home environment. I am currently running two Debian etch 4.0 machines both running vanilla 2.6.21.1 kernels that I compiled (exception being the nvidia driver for video). When I attempt to do some performance benchmarks, a test of bonnie++ makes the NFS client stuck in a non-responsive non-killable state. The NFS server at this point is not busy doing anything so I believe there may be some condition leading to a deadlock in the client.
jfrisk@evil:~$ df -k .
Filesystem 1K-blocks Used Available
Use% Mounted on
imezru-ha1:/hafs/nethome/jfrisk
76791424 409856 75601408
1% /nethome/jfrisk
jfrisk@evil:~$ mountimezru-ha1:/hafs/nethome/jfrisk on /nethome/jfrisk type nfs (rw,hard,intr,bg,addr=192.168.69.7)
jfrisk@evil:~$ /usr/sbin/bonnie++ -f -s 100 -n 1 -r 0 Writing intelligently...done Rewriting...
This is where the bonnie program hangs.
frisk@evil:~$ ps aux | grep bonnie jfrisk 6882 0.7 0.0 2708 1012 pts/3 D+ 09:11 0:00 /usr/sbin/bonnie++ -f -s 100 -n 1 -r 0
Top shows the user load at 1.00 (before starting is 0.10) but nothing else of importance is running on the system.
I can reproduce this bug pretty consistently. I have these machines for use by anyone on the team so I would appreciate being guided on what to do next to help debug this issue. Thanks in advance!
____________________________________________________________________________________
Now that's room service! Choose from over 150,000 hotels in 45,000 destinations on Yahoo! Travel to find your fit.
http://farechase.yahoo.com/promo-generic-14795097
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 11 lines · John Frisk
Team,
I have been attempting to put together a linux HA NFS server for my home environment. I am currently running two Debian etch 4.0 machines both running vanilla 2.6.21.1 kernels that I compiled (exception being the nvidia driver for video). When I attempt to do some performance benchmarks, a test of bonnie++ makes the NFS client stuck in a non-responsive non-killable state. The NFS server at this point is not busy doing anything so I believe there may be some condition leading to a deadlock in the client.
Could you please try the patches
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-002-no_congestion_wait_in_update_request.dif
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-003-cleanup_coalesce.dif
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-004-cleanup_coalesce2.dif
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-005-cleanup_readpages.dif
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-006-fix_dirtying_race.dif
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-007-fix_page_overflow.dif
and
http://client.linux-nfs.org/Linux-2.6.x/2.6.22-rc2/linux-2.6.22-001-write_deadlock_avoidance.dif
That fixes all the write deadlock issues for me.
Trond
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Hi Trond,
I applied your patches to both kernels, rebooted and still the same problem. Once bonnie attempts to "rewrite" it's sample file, the client machine process cannot be killed. The client OS however can still do other things, such as load firefox, etc. The interesting thing being, it can also in another shell access the same directory under NFS as the bonnie program was writing in the other shell. Sorry for not knowing all the internals, but is the NFS client multi-threaded too?
Output:
jfrisk@evil:~$ /usr/sbin/bonnie++ -f -s 100 -n 1 -r 0 Writing intelligently...done Rewriting...
jfrisk@evil:~$ ps axu | grep bonnie jfrisk 5612 0.2 0.0 2708 1012 pts/2 D+ 13:24 0:01 /usr/sbin/bonnie++ -f -s 100 -n 1 -r 0
What should I do next? Should I capture some of the tcpdump output or compile debugging or profiling into the kernel for evaluation?
Thanks!
--- Trond Myklebust <[email protected]>
Quoted 13 lines · wrote:
Quoted 2 lines · John Frisk
Team,
I have been attempting to put together a linux HANFS
Quoted 3 lines
server for my home environment. I am currently running two Debian etch 4.0 machines both running vanilla 2.6.21.1 kernels that I compiled
(exception
Quoted 1 line
being the nvidia driver for video). When I
attempt to
Quoted 3 lines
do some performance benchmarks, a test of bonnie++ makes the NFS client stuck in a non-responsive non-killable state. The NFS server at this point
Quoted 8 lines
is
Quoted 1 line
not busy doing anything so I believe there may be
some
Quoted 1 line
condition leading to a deadlock in the client.
Could you please try the patches
Quoted 1 line
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-003-cleanup_coalesce.dif
Quoted 1 line
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-004-cleanup_coalesce2.dif
Quoted 1 line
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-005-cleanup_readpages.dif
Quoted 1 line
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-006-fix_dirtying_race.dif
Quoted 1 line
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-007-fix_page_overflow.dif
Quoted 4 lines
and
http://client.linux-nfs.org/Linux-2.6.x/2.6.22-rc2/linux-2.6.22-001-write_deadlock_avoidance.dif
Quoted 6 lines
That fixes all the write deadlock issues for me.
Trond
____________________________________________________________________________________Ready for the edge of your seat? Check out tonight's top picks on Yahoo! TV.
http://tv.yahoo.com/
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs