automounter segfaults when using negative cache?
Interesting thing:
I have 2 identical login01 and login02 machines (Rhel4, full updates). login01 is running old kernel (2.6.9-42.ELsmp), login02 is running the latest one (2.6.9-78.0.8.ELsmp). Both have autofs debug enabled and are running the latest autofs (autofs-4.1.3-234).
Now:
1) login01:
[root@login01 ondrejv]# cat /var/log/debug.log* | grep "Dec 4 09:15:09"
Dec 4 09:15:09 login01 automount[3939]: handle_packet: type = 0 Dec 4 09:15:09 login01 automount[3939]: handle_packet_missing: token 3533, name .raw_data Dec 4 09:15:09 login01 automount[3939]: attempting to mount entry /proj/.raw_data Dec 4 09:15:09 login01 automount[1649]: lookup(yp): looking up .raw_data Dec 4 09:15:09 login01 automount[3939]: mt->key set to .raw_data Dec 4 09:15:09 login01 kernel: automount[1649]: segfault at 0000000000000000 rip 0000002a95916be0 rsp 0000007fbfffd5c8 error 4 Dec 4 09:15:09 login01 automount[3939]: handle_child: got pid 1649, sig 1 (11), stat 0 Dec 4 09:15:09 login01 automount[3939]: sig_child: found pending iop pid 1649: signalled 1 (sig 11), exit status 0 Dec 4 09:15:09 login01 automount[3939]: send_fail: token=3533 Dec 4 09:15:09 login01 automount[3939]: handle_packet: type = 0 Dec 4 09:15:09 login01 automount[3939]: handle_packet_missing: token 3534, name .raw_data Dec 4 09:15:09 login01 automount[3939]: attempting to mount entry /proj/.raw_data Dec 4 09:15:09 login01 automount[1650]: lookup(yp): looking up .raw_data Dec 4 09:15:09 login01 automount[3939]: mt->key set to .raw_data Dec 4 09:15:09 login01 kernel: automount[1650]: segfault at
0000000000000000 rip 0000002a95916be0 rsp 0000007fbfffd5c8 error 4
Note that .raw_data map does not exist in the yp map so it should go into the negative cache - which happens correctly on login02:
ov 28 23:27:43 login01 automount[4201]: attempting to mount entry /proj/.raw_data
Nov 28 23:27:43 login01 automount[13394]: lookup(yp): looking up .raw_data Nov 28 23:27:43 login01 automount[4201]: mt->key set to .raw_data Nov 28 23:27:43 login01 automount[13394]: lookup(yp): key ".raw_data" not found in map. Nov 28 23:27:43 login01 automount[13394]: failed to mount /proj/.raw_data Nov 28 23:27:43 login01 automount[13394]: umount_multi: path=/proj/.raw_data incl=1 Nov 28 23:27:43 login01 automount[4201]: handle_child: got pid 13394, sig 0 (0), stat 3 Nov 28 23:27:43 login01 automount[4201]: sig_child: found pending iop pid 13394: signalled 0 (sig 0), exit status 3 Nov 28 23:27:43 login01 automount[4201]: update_negative_cache: key: .raw_data Nov 28 23:27:43 login01 automount[4201]: Adding negative cache entry for key .raw_data Nov 28 23:27:43 login01 automount[4201]: Key .raw_data added to negative cache Nov 28 23:27:43 login01 automount[4201]: send_fail: token=15625
I thought all the negative caching is done in user space, but apparently kernel has some influence here....
Any explanation?
Thanks,
Ondrej
Quoted 30 lines · Ondrej Valousek
Interesting thing:
I have 2 identical login01 and login02 machines (Rhel4, full updates). login01 is running old kernel (2.6.9-42.ELsmp), login02 is running the latest one (2.6.9-78.0.8.ELsmp). Both have autofs debug enabled and are running the latest autofs (autofs-4.1.3-234).
Now:
1) login01:
[root@login01 ondrejv]# cat/var/log/debug.log*| grep "Dec 4 09:15:09"Dec 4 09:15:09 login01 automount[3939]: handle_packet: type = 0 Dec 4 09:15:09 login01 automount[3939]: handle_packet_missing: token 3533, name .raw_data Dec 4 09:15:09 login01 automount[3939]: attempting to mount entry /proj/.raw_data Dec 4 09:15:09 login01 automount[1649]: lookup(yp): looking up .raw_data Dec 4 09:15:09 login01 automount[3939]: mt->key set to .raw_data Dec 4 09:15:09 login01 kernel: automount[1649]: segfault at 0000000000000000 rip 0000002a95916be0 rsp 0000007fbfffd5c8 error 4 Dec 4 09:15:09 login01 automount[3939]: handle_child: got pid 1649, sig 1 (11), stat 0 Dec 4 09:15:09 login01 automount[3939]: sig_child: found pending iop pid 1649: signalled 1 (sig 11), exit status 0 Dec 4 09:15:09 login01 automount[3939]: send_fail: token=3533 Dec 4 09:15:09 login01 automount[3939]: handle_packet: type = 0 Dec 4 09:15:09 login01 automount[3939]: handle_packet_missing: token 3534, name .raw_data Dec 4 09:15:09 login01 automount[3939]: attempting to mount entry /proj/.raw_data Dec 4 09:15:09 login01 automount[1650]: lookup(yp): looking up .raw_data Dec 4 09:15:09 login01 automount[3939]: mt->key set to .raw_data Dec 4 09:15:09 login01 kernel: automount[1650]: segfault at
Quoted 1 line
0000000000000000 rip
0000002a95916be0rsp0000007fbfffd5c8error 4
Can you enable core dumps and get us a stack trace of this?
Quoted 23 lines
Note that .raw_data map does not exist in the yp map so it should go into the negative cache - which happens correctly on login02:
ov 28 23:27:43 login01 automount[4201]: attempting to mount entry /proj/.raw_dataNov 28 23:27:43 login01 automount[13394]: lookup(yp): looking up .raw_data Nov 28 23:27:43 login01 automount[4201]: mt->key set to .raw_data Nov 28 23:27:43 login01 automount[13394]: lookup(yp): key ".raw_data" not found in map. Nov 28 23:27:43 login01 automount[13394]: failed to mount /proj/.raw_data Nov 28 23:27:43 login01 automount[13394]: umount_multi: path=/proj/.raw_data incl=1 Nov 28 23:27:43 login01 automount[4201]: handle_child: got pid 13394, sig 0 (0), stat 3 Nov 28 23:27:43 login01 automount[4201]: sig_child: found pending iop pid 13394: signalled 0 (sig 0), exit status 3 Nov 28 23:27:43 login01 automount[4201]: update_negative_cache: key: .raw_data Nov 28 23:27:43 login01 automount[4201]: Adding negative cache entry for key .raw_data Nov 28 23:27:43 login01 automount[4201]: Key .raw_data added to negative cache Nov 28 23:27:43 login01 automount[4201]: send_fail: token=15625
Quoted 3 lines
I thought all the negative caching is done in user space, but apparently kernel has some influence here....
Any explanation?
The negative caching is done in userspace. I'm not sure I understand your question.
Cheers,
Jeff
Ian Kent
maintainerQuoted 59 lines · Jeff Moyer
Quoted 30 lines · Ondrej Valousek
Interesting thing:
I have 2 identical login01 and login02 machines (Rhel4, full updates). login01 is running old kernel (2.6.9-42.ELsmp), login02 is running the latest one (2.6.9-78.0.8.ELsmp). Both have autofs debug enabled and are running the latest autofs (autofs-4.1.3-234).
Now:
1) login01:
[root@login01 ondrejv]# cat/var/log/debug.log*| grep "Dec 4 09:15:09"Dec 4 09:15:09 login01 automount[3939]: handle_packet: type = 0 Dec 4 09:15:09 login01 automount[3939]: handle_packet_missing: token 3533, name .raw_data Dec 4 09:15:09 login01 automount[3939]: attempting to mount entry /proj/.raw_data Dec 4 09:15:09 login01 automount[1649]: lookup(yp): looking up .raw_data Dec 4 09:15:09 login01 automount[3939]: mt->key set to .raw_data Dec 4 09:15:09 login01 kernel: automount[1649]: segfault at 0000000000000000 rip 0000002a95916be0 rsp 0000007fbfffd5c8 error 4 Dec 4 09:15:09 login01 automount[3939]: handle_child: got pid 1649, sig 1 (11), stat 0 Dec 4 09:15:09 login01 automount[3939]: sig_child: found pending iop pid 1649: signalled 1 (sig 11), exit status 0 Dec 4 09:15:09 login01 automount[3939]: send_fail: token=3533 Dec 4 09:15:09 login01 automount[3939]: handle_packet: type = 0 Dec 4 09:15:09 login01 automount[3939]: handle_packet_missing: token 3534, name .raw_data Dec 4 09:15:09 login01 automount[3939]: attempting to mount entry /proj/.raw_data Dec 4 09:15:09 login01 automount[1650]: lookup(yp): looking up .raw_data Dec 4 09:15:09 login01 automount[3939]: mt->key set to .raw_data Dec 4 09:15:09 login01 kernel: automount[1650]: segfault atQuoted 1 line
0000000000000000 rip
0000002a95916be0rsp0000007fbfffd5c8error 4Can you enable core dumps and get us a stack trace of this?
Quoted 23 lines
Note that .raw_data map does not exist in the yp map so it should go into the negative cache - which happens correctly on login02:
ov 28 23:27:43 login01 automount[4201]: attempting to mount entry /proj/.raw_dataNov 28 23:27:43 login01 automount[13394]: lookup(yp): looking up .raw_data Nov 28 23:27:43 login01 automount[4201]: mt->key set to .raw_data Nov 28 23:27:43 login01 automount[13394]: lookup(yp): key ".raw_data" not found in map. Nov 28 23:27:43 login01 automount[13394]: failed to mount /proj/.raw_data Nov 28 23:27:43 login01 automount[13394]: umount_multi: path=/proj/.raw_data incl=1 Nov 28 23:27:43 login01 automount[4201]: handle_child: got pid 13394, sig 0 (0), stat 3 Nov 28 23:27:43 login01 automount[4201]: sig_child: found pending iop pid 13394: signalled 0 (sig 0), exit status 3 Nov 28 23:27:43 login01 automount[4201]: update_negative_cache: key: .raw_data Nov 28 23:27:43 login01 automount[4201]: Adding negative cache entry for key .raw_data Nov 28 23:27:43 login01 automount[4201]: Key .raw_data added to negative cache Nov 28 23:27:43 login01 automount[4201]: send_fail: token=15625
Quoted 6 lines
Quoted 3 lines
I thought all the negative caching is done in user space, but apparently kernel has some influence here....
Any explanation?The negative caching is done in userspace. I'm not sure I understand your question.
Me neither but I'm working on the negative caching of non-existent map keys. Turns out to be a bit tricky when multiple nsswitch sources can be tried for each mount since the map may not exist within a given source and so we get NSS_STATUS_UNAVAIL instead of NSS_STATUS_NOTFOUND. Maybe I should just cache the unavailable in the same way?
Ian
Quoted 10 lines · Ian Kent
Quoted 3 lines · Jeff Moyer
The negative caching is done in userspace. I'm not sure I understand your question.
Me neither but I'm working on the negative caching of non-existent map keys. Turns out to be a bit tricky when multiple nsswitch sources can be tried for each mount since the map may not exist within a given source and so we get
NSS_STATUS_UNAVAILinstead ofNSS_STATUS_NOTFOUND. Maybe I should just cache the unavailable in the same way?
Ian, this is autofs version 4. It's been a while since I implemented the negative caching there, but I'd be interested in knowing if you think I got it wrong for the case above!
Cheers,
Jeff
Ian Kent
maintainerQuoted 13 lines · Jeff Moyer
Quoted 10 lines · Ian Kent
Quoted 3 lines · Jeff Moyer
The negative caching is done in userspace. I'm not sure I understand your question.
Me neither but I'm working on the negative caching of non-existent map keys. Turns out to be a bit tricky when multiple nsswitch sources can be tried for each mount since the map may not exist within a given source and so we get
NSS_STATUS_UNAVAILinstead ofNSS_STATUS_NOTFOUND. Maybe I should just cache the unavailable in the same way?
Quoted 3 lines
Ian, this is autofs version 4. It's been a while since I implemented the negative caching there, but I'd be interested in knowing if you think I got it wrong for the case above!
Oh, yeah, lucky it will be Xmas holiday time soon. I'm going to have a look around the v4 code tomorrow. I'll spend some time on this as well.
Quoted 3 lines
Cheers,
Jeff
Quoted 6 lines
Quoted 5 lines
I thought all the negative caching is done in user space, but apparently kernel has some influence here....
Any explanation?
Quoted 6 lines
The negative caching is done in userspace. I'm not sure I understand your question.
Cheers,
Jeff
Hi Jeff,
Ok I was thinking this way:
I experience this only on U7 system running U4 kernel - on the same U7 system running U7 kernel I can not replicate this. If the negative caching is done in the user space and there is a bug there which causes the autofs to segfault, how is it possible that is won't segfault on the U7 kernel?
I do not know I am still trying to find out if there is any connection to the problem with the automounter hanging that I reported earlier as:
- in U7 system running U7 kernel, the autofs does not crash, but hangs (after few days).
- in U7 system running U4 kernel, the autofs forks crash regularly but otherwise it works fine (I mean at least the parent autofs fork continue living...)
You know what I am trying to say?
Thanks,
Ondrej