race in autofs / nfs
Hi,
there is a race in 2.4.1 and 2.4.2-pre3 in autofs/nfs.
When the cwd is on the nfs mounted server (== busy) and you try to reboot the shutdown hangs in "rcautofs stop". I can reproduce it everytime.
I attach a screen log and a decoded sysrq output. It is not related to that killproc, it hangs right now in automount <-> kupdate (with kernel 2.4.1):
kupdate 6 0 0 -1 ?00000000 c307a000 0 current
automoun 1525 31480 503680 54 u0176a368 df3e6000 1 current
there is a spinlock message:_spin_lock(c021f6e4) CPU#1 NIP c01b8220 holder: cpu 1 pc E126BF28
c021f6e4 is not in the System.map, but c01b8220 is c01b817c T memparsec01b8200 T atomic_dec_and_lockc01b9a5c t cleanup_proc_rtas
The attached log is kernel 2.4.2-pre3.
killproc 1701 1030738 16491808 106 u0178d5c8 c7de6000 0 current automoun 1705 1036120 16577920 54 u0176a368 c7b46000 1 current
_spin_lock(c022279c) CPU#1 NIP c01b7c5c holder: cpu 1 pc E126BF28
SysRq: Emergency Sync
_spin_lock(c022279c) CPU#0 NIP c005d920 holder: cpu 1 pc E126BF28 _spin_lock(c022279c) CPU#1 NIP c01b7c5c holder: cpu 1 pc E126BF28
c01b7c5c is again atomic_dec_and_lock and c005d920 is d_lookup.
The system is a 2way RS/6000, 2gig ram.
Linux cantaloupe 2.4.1-SMP #1 SMP Sun Feb 11 13:02:05 GMT 2001 ppc unknown
Kernel modules 2.4.1 Gnu C 2.95.2 Gnu Make 3.79.1 Binutils 2.10.0.33 Linux C Library x 1 root root 1499361 Feb 9 02:50 /lib/libc.so.6 Dynamic linker ldd (GNU libc) 2.2 Procps 2.0.7 Mount 2.10q Net-tools 1.57 Kbd 1.02 Sh-utils 2.0 Modules Loaded nfsd autofs4 ipv6
Any ideas what could be wrong?
Gruss Olaf
Signature · 4 lines
$ man clone
BUGS
Main feature not yet implemented...Quoted 7 lines · Olaf Hering
Hi,
there is a race in 2.4.1 and 2.4.2-pre3 in autofs/nfs.
When the cwd is on the nfs mounted server (== busy) and you try to reboot the shutdown hangs in "rcautofs stop". I can reproduce it everytime.
Sounds like an NFS bug in umount.
-=hpa
Signature · 7 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." http://www.zytor.com/~hpa/puzzle.txt - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == H Peter Anvin <[email protected]> writes:
> Olaf Hering wrote:
>>
>> Hi,
>>
>> there is a race in 2.4.1 and 2.4.2-pre3 in autofs/nfs. When
>> the cwd is on the nfs mounted server (== busy) and you try to
>> reboot the shutdown hangs in "rcautofs stop". I can reproduce
>> it everytime.
>>
> Sounds like an NFS bug in umount.Or a dcache bug: the above points to a corruption of the mnt_count which is supposed to be > 0 if the partition is in use. I'm seeing a similar leak for ext2 partitions (not involving autofs or NFS).
Cheers,
Trond
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 14 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == H Peter Anvin <[email protected]> writes:
> Olaf Hering wrote: >> >> Hi, >> >> there is a race in 2.4.1 and 2.4.2-pre3 in autofs/nfs. When >> the cwd is on the nfs mounted server (== busy) and you try to >> reboot the shutdown hangs in "rcautofs stop". I can reproduce >> it everytime. >> > Sounds like an NFS bug in umount.
Quoted 3 lines
Or a dcache bug: the above points to a corruption of the
mnt_countwhich is supposed to be > 0 if the partition is in use. I'm seeing a similar leak for ext2 partitions (not involving autofs or NFS).
Send me patches :)
autofs is the latest, btw.
http://www.de.kernel.org/pub/linux/daemons/autofs/testing-v4/autofs-4.0.0pre9.tar.bz2
Gruss Olaf
Signature · 8 lines
$ man clone
BUGS
Main feature not yet implemented...
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/Quoted 14 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == H Peter Anvin <[email protected]> writes:
> Olaf Hering wrote: >> >> Hi, >> >> there is a race in 2.4.1 and 2.4.2-pre3 in autofs/nfs. When >> the cwd is on the nfs mounted server (== busy) and you try to >> reboot the shutdown hangs in "rcautofs stop". I can reproduce >> it everytime. >> > Sounds like an NFS bug in umount.
Quoted 3 lines
Or a dcache bug: the above points to a corruption of the
mnt_countwhich is supposed to be > 0 if the partition is in use. I'm seeing a similar leak for ext2 partitions (not involving autofs or NFS).
Some more updates:
This machines run the upcoming 7.1-ppc with autofs-4.0.0pre9. When I install 7.0-ppc it runs rocksolid with the same kernel binary. 7.0 came with autofs-4.0.0pre7:
Linux plum 2.4.1 #1 Sun Feb 11 11:56:01 GMT 2001 ppc unknown
Kernel modules 2.4.1 Gnu C 2.95.3 Binutils 2.9.5.0.24 Linux C Library x 1 root root 4209204 Oct 4 21:42 /lib/libc.so.6 Dynamic linker ldd (GNU libc) 2.1.3 Procps 2.0.6 Mount 2.10m Net-tools 1.56 Kbd 0.99 Sh-utils 2.0 Modules Loaded snd-card-awacs snd-pcm snd-timer snd-mixer snd
soundcore ipv6 netlink_dev nfsd autofs bmac pcmcia_core
(hmm, it loads autofs and not autofs4 on 7.0?)
It hangs again after a while, in this case:
cc1plus 19126 181495 2903920 4 u01783c18 c44ca000 1 current automoun 19129 183289 2932624 54 u0176a368 c42ca000 0 current
_spin_lock(c021b754) CPU#1 NIP c0 072f18 holder: cpu 0 pc C0057A18
I will try the older autofs package now.
Gruss Olaf
Signature · 8 lines
$ man clone
BUGS
Main feature not yet implemented...
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/Quoted 14 lines · Olaf Hering
Quoted 12 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == H Peter Anvin <[email protected]> writes:
> Olaf Hering wrote: >> >> Hi, >> >> there is a race in 2.4.1 and 2.4.2-pre3 in autofs/nfs. When >> the cwd is on the nfs mounted server (== busy) and you try to >> reboot the shutdown hangs in "rcautofs stop". I can reproduce >> it everytime. >>
Quoted 7 lines
Quoted 5 lines
> Sounds like an NFS bug in umount.
Or a dcache bug: the above points to a corruption of the
mnt_countwhich is supposed to be > 0 if the partition is in use. I'm seeing a similar leak for ext2 partitions (not involving autofs or NFS).(hmm, it loads autofs and not autofs4 on 7.0?)
The autofs4.o is the culprit, it works perfect with autofs.o.
What would happen if I stick with autofs.o now?
The docu recommends autofs4 in modules.conf.
Gruss Olaf
Signature · 8 lines
$ man clone
BUGS
Main feature not yet implemented...
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/Quoted 6 lines · Olaf Hering
The autofs4.o is the culprit, it works perfect with autofs.o.
What would happen if I stick with autofs.o now?
The docu recommends autofs4 in modules.conf.
I don't know who came up with that idea. You should use the module that matches your daemon, and not try to hack around so that there is a module/daemon mismatch.
-hpa
Signature · 8 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." http://www.zytor.com/~hpa/puzzle.txt - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://vger.kernel.org/lkml/
Quoted 11 lines · H. Peter Anvin
Quoted 6 lines · Olaf Hering
The autofs4.o is the culprit, it works perfect with autofs.o.
What would happen if I stick with autofs.o now?
The docu recommends autofs4 in modules.conf.I don't know who came up with that idea. You should use the module that matches your daemon, and not try to hack around so that there is a module/daemon mismatch.
cantaloupe:~ # /usr/sbin/automount -v
Linux automount version 4.0.0
We had 4.0pre7 in 7.0 and 4.0pre9 in 7.1.
I would really like to know _where_ it hangs, Trond sent me a printk patch but this one was not called.
I will try to get a i386 SMP machine to see if its ppc specific.
Gruss Olaf
Signature · 9 lines
$ man clone
BUGS
Main feature not yet implemented...
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://vger.kernel.org/lkml/Quoted 14 lines · Olaf Hering
Quoted 11 lines · H. Peter Anvin
Quoted 6 lines · Olaf Hering
The autofs4.o is the culprit, it works perfect with autofs.o.
What would happen if I stick with autofs.o now?
The docu recommends autofs4 in modules.conf.I don't know who came up with that idea. You should use the module that matches your daemon, and not try to hack around so that there is a module/daemon mismatch.
Quoted 7 lines
cantaloupe:~ #
/usr/sbin/automount-v
Linux automount version 4.0.0We had 4.0pre7 in 7.0 and 4.0pre9 in 7.1.
I would really like to know _where_ it hangs, Trond sent me a printk patch but this one was not called.
Any ideas where to start with the debugging?
Quoted 1 line
I will try to get a i386 SMP machine to see if its ppc specific.
I'm unable to reproduce it on a Piii 750 with SuSE 7.1.
guillory:/usr/src/OLAF/linux-2.4.2-pre3 # sh scripts/ver_linux -- Versions installed: (if some fields are empty or look
-- unusual then possibly you have very old versions)
Linux guillory 2.4.2-pre3-SMP #3 SMP Tue Feb 13 14:50:47 CET 2001 i686 unknown
Kernel modules 2.4.1 Gnu C 2.95.2 Gnu Make 3.79.1 Binutils 2.10.0.33 Linux C Library x 1 root root 1382179 Jan 19 07:14 /lib/libc.so.6 Dynamic linker ldd (GNU libc) 2.2 Procps 2.0.7 Mount 2.10q Net-tools 1.57 Kbd 1.02 Sh-utils 2.0 Modules Loaded nfsd ipv6 mousedev hid input usbcore eepro100
both 2.4.1ac10 and 2.4.2-pre3 boot fine.
This machine oops in the usb stack, but thats another issue.
Gruss Olaf
Signature · 4 lines
$ man clone
BUGS
Main feature not yet implemented...