Kernel freezes on "exportfs -a"
Subsystems (2)
Hello all, hi Neil, hi Olaf,
we recently found a strange bug in the recent 2.6 kernels that freezes the whole kernel (down to even no response to the Magic SysRq keys). And in fact sticks us to the 2.4 series we want to get rid of :)
This happens during "exportfs -a" when the following sequence commences (tested by manual command execution, with some seconds betweeen the commands):
rpc.nfsd 128 rpc.mountd rpc.statd exportfs -a
We cannot reproduce it on a "normal" machine.
The most recently tested kernel is a 2.6.9-rc1 (since each test breaks the production service). exportfs is 1.0.6.
The particular server is the NFS file repository for a high volume FTP cluster, thus beeing requested by the still online 4 ftp backend servers *heavily*.
It has 2 Fast Ethernet ports in *bonding* mode (in peak hours they can be filled with ease).
Probably it is this combinations that makes a problem.
There are only two directories being exported to 5 machines each with really standard options.
The freeze can be reproduced even when the exported directories are empty (no mount of the *some* GB to be checked afterwards, nice for testing :)) and even with "strace exportfs -a".
strace stops during the call of
nfsservctl(0x1,...,0)
right after reading the still empty /proc/fs/nfs/exports.
I just hope that something helps.
If we can run a special test or provide suitable
information about the system, please let me know.
Thank You in advance, probably it's something easily fixed.
Yours sincerely,
- Marian
-------------------------------------------------------
This SF.Net email is sponsored by: InterSystems CACHE FREE OODBMS DOWNLOAD - A multidimensional database that combines robust object and relational technologies, making it a perfect match for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 7 lines · On Tuesday November 16, [email protected] wrote:
Hello all, hi Neil, hi Olaf,
we recently found a strange bug in the recent 2.6 kernels that freezes the whole kernel (down to even no response to the Magic SysRq keys). And in fact sticks us to the 2.4 series we want to get rid of :)
Quoted 8 lines
This happens during "exportfs -a" when the following sequence commences (tested by manual command execution, with some seconds betweeen the commands):
rpc.nfsd 128 rpc.mountd rpc.statd exportfs -a
It doesn't address your core problem, but this order is wrong. The "exportfs -a" should be done *before* rpc.nfsd, otherwise there is a window where clients could be told that the filesystem they want isn't exported.
Quoted 2 lines
We cannot reproduce it on a "normal" machine.
Just how abnormal is the machine where you *can* export it?
Quoted 4 lines
The most recently tested kernel is a 2.6.9-rc1 (since each test breaks the production service). exportfs is 1.0.6.
Quoted 8 lines
The particular server is the NFS file repository for a high volume FTP cluster, thus beeing requested by the still online 4 ftp backend servers *heavily*.
It has 2 Fast Ethernet ports in *bonding* mode (in peak hours they can be filled with ease).
Probably it is this combinations that makes a problem.
So are you saying that the clients are hammering on the server while you are trying to bring up NFS service? If this is the case, then maybe just moving the "exportfs -a" to the top will help.
NeilBrown
-------------------------------------------------------
This SF.Net email is sponsored by: InterSystems CACHE FREE OODBMS DOWNLOAD - A multidimensional database that combines robust object and relational technologies, making it a perfect match for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 9 lines · Neil Brown
Quoted 4 lines
rpc.nfsd 128 rpc.mountd rpc.statd exportfs -aIt doesn't address your core problem, but this order is wrong. The "exportfs -a" should be done *before* rpc.nfsd, otherwise there is a window where clients could be told that the filesystem they want isn't exported.
This makes sense. Too-many-generations-of-this-script problem.
Quoted 3 lines
Quoted 1 line
We cannot reproduce it on a "normal" machine.
Just how abnormal is the machine where you *can* export it?
Smaller file systems (far much smaller), single processor, completely different hardware, no clients and so on.
Although the size of the exported FS is of no interest, because exportfs froze even when I did not mount it (to save time on reboot-fsck).
Quoted 2 lines
Quoted 2 lines
The most recently tested kernel is a 2.6.9-rc1 (since each test breaks the production service). exportfs is 1.0.6.
Interesting result: 2.6.10-rc2 appears to work perfectly. Some racing condition anywhere in the kernel infrastructure?
Quoted 4 lines
Quoted 4 lines
The particular server is the NFS file repository for a high volume FTP cluster, thus beeing requested by the still online 4 ftp backend servers *heavily*.
Quoted 8 lines
Quoted 4 lines
It has 2 Fast Ethernet ports in *bonding* mode (in peak hours they can be filled with ease).
Probably it is this combinations that makes a problem.
So are you saying that the clients are hammering on the server while you are trying to bring up NFS service? If this is the case, then maybe just moving the "exportfs -a" to the top will
I had no chance to go deep into detail, like Olaf mentioned with the backtrace-NMI suggestion, because I had to bring the machine back into production ASAP.
Quoted 12 lines
NeilBrown
-------------------------------------------------------
This SF.Net email is sponsored by: InterSystems CACHE FREE OODBMS DOWNLOAD - A multidimensional database that combines robust object and relational technologies, making it a perfect match for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Signature · 18 lines
Mit freundlichen Gruessen / Yours sincerely
Marian Eichholz
Postmaster
freenet.de AG Vorsitzender des Aufsichtsrates: Prof. Dr. Helmut Thoma
Deelbögenkamp 4c Vorstand: Eckhard Spoerr (Vors.), Axel Krieger,
Stephan Esch, Eric Berger
22297 Hamburg Amtsgericht Hamburg, HRB 74048
-------------------------------------------------------
This SF.Net email is sponsored by: InterSystems CACHE
FREE OODBMS DOWNLOAD - A multidimensional database that combines
robust object and relational technologies, making it a perfect match
for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfsQuoted 6 lines · Marian Eichholz
Hello all, hi Neil, hi Olaf,
we recently found a strange bug in the recent 2.6 kernels that freezes the whole kernel (down to even no response to the Magic SysRq keys). And in fact sticks us to the 2.4 series we want to get rid of :)
Did you try enabling the NMI watchdog so you get a backtrace of where it hangs?
Olaf
Signature · 13 lines
Olaf Kirch | Things that make Monday morning interesting, #1: [email protected] | "I want to use NFS over AX25, can you help me?" ---------------+ ------------------------------------------------------- This SF.Net email is sponsored by: InterSystems CACHE FREE OODBMS DOWNLOAD - A multidimensional database that combines robust object and relational technologies, making it a perfect match for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8 _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs