Skip to contentSkip to conversation

automounter segfaults when using negative cache?

DISCUSSION Ondrej Valousek 3 participants 6 messages Dec 4, 2008 – Dec 8, 2008 Kernel automounter

lore
Share & tools
lore
#0 lore raw

Interesting thing:
I have 2 identical login01 and login02 machines (Rhel4, full updates). login01 is running old kernel (2.6.9-42.ELsmp), login02 is running the latest one (2.6.9-78.0.8.ELsmp). Both have autofs debug enabled and are running the latest autofs (autofs-4.1.3-234).
Now:
1) login01:
[root@login01 ondrejv]# cat /var/log/debug.log* | grep "Dec 4 09:15:09"

Dec  4 09:15:09 login01 automount[3939]: handle_packet: type = 0
Dec  4 09:15:09 login01 automount[3939]: handle_packet_missing: token
3533, name .raw_data
Dec  4 09:15:09 login01 automount[3939]: attempting to mount entry
/proj/.raw_data
Dec  4 09:15:09 login01 automount[1649]: lookup(yp): looking up .raw_data
Dec  4 09:15:09 login01 automount[3939]: mt->key set to .raw_data
Dec  4 09:15:09 login01 kernel: automount[1649]: segfault at
0000000000000000 rip 0000002a95916be0 rsp 0000007fbfffd5c8 error 4
Dec  4 09:15:09 login01 automount[3939]: handle_child: got pid 1649, sig
1 (11), stat 0
Dec  4 09:15:09 login01 automount[3939]: sig_child: found pending iop
pid 1649: signalled 1 (sig 11), exit status 0
Dec  4 09:15:09 login01 automount[3939]: send_fail: token=3533
Dec  4 09:15:09 login01 automount[3939]: handle_packet: type = 0
Dec  4 09:15:09 login01 automount[3939]: handle_packet_missing: token
3534, name .raw_data
Dec  4 09:15:09 login01 automount[3939]: attempting to mount entry
/proj/.raw_data
Dec  4 09:15:09 login01 automount[1650]: lookup(yp): looking up .raw_data
Dec  4 09:15:09 login01 automount[3939]: mt->key set to .raw_data
Dec  4 09:15:09 login01 kernel: automount[1650]: segfault at

0000000000000000 rip 0000002a95916be0 rsp 0000007fbfffd5c8 error 4

Note that .raw_data map does not exist in the yp map so it should go into the negative cache - which happens correctly on login02:
ov 28 23:27:43 login01 automount[4201]: attempting to mount entry /proj/.raw_data

Nov 28 23:27:43 login01 automount[13394]: lookup(yp): looking up .raw_data
Nov 28 23:27:43 login01 automount[4201]: mt->key set to .raw_data
Nov 28 23:27:43 login01 automount[13394]: lookup(yp): key ".raw_data"
not found in map.
Nov 28 23:27:43 login01 automount[13394]: failed to mount /proj/.raw_data
Nov 28 23:27:43 login01 automount[13394]: umount_multi:
path=/proj/.raw_data incl=1
Nov 28 23:27:43 login01 automount[4201]: handle_child: got pid 13394,
sig 0 (0), stat 3
Nov 28 23:27:43 login01 automount[4201]: sig_child: found pending iop
pid 13394: signalled 0 (sig 0), exit status 3
Nov 28 23:27:43 login01 automount[4201]: update_negative_cache: key:
.raw_data
Nov 28 23:27:43 login01 automount[4201]: Adding negative cache entry for
key .raw_data
Nov 28 23:27:43 login01 automount[4201]: Key .raw_data added to negative
cache
Nov 28 23:27:43 login01 automount[4201]: send_fail: token=15625

I thought all the negative caching is done in user space, but apparently kernel has some influence here....
Any explanation?
Thanks,

Ondrej

#1 lore raw
Quoted 30 lines · Ondrej Valousek

Interesting thing:
I have 2 identical login01 and login02 machines (Rhel4, full updates). login01 is running old kernel (2.6.9-42.ELsmp), login02 is running the latest one (2.6.9-78.0.8.ELsmp). Both have autofs debug enabled and are running the latest autofs (autofs-4.1.3-234).
Now:
1) login01:
[root@login01 ondrejv]# cat /var/log/debug.log* | grep "Dec 4 09:15:09"

Dec  4 09:15:09 login01 automount[3939]: handle_packet: type = 0
Dec  4 09:15:09 login01 automount[3939]: handle_packet_missing: token
3533, name .raw_data
Dec  4 09:15:09 login01 automount[3939]: attempting to mount entry
/proj/.raw_data
Dec  4 09:15:09 login01 automount[1649]: lookup(yp): looking up .raw_data
Dec  4 09:15:09 login01 automount[3939]: mt->key set to .raw_data
Dec  4 09:15:09 login01 kernel: automount[1649]: segfault at
0000000000000000 rip 0000002a95916be0 rsp 0000007fbfffd5c8 error 4
Dec  4 09:15:09 login01 automount[3939]: handle_child: got pid 1649, sig
1 (11), stat 0
Dec  4 09:15:09 login01 automount[3939]: sig_child: found pending iop
pid 1649: signalled 1 (sig 11), exit status 0
Dec  4 09:15:09 login01 automount[3939]: send_fail: token=3533
Dec  4 09:15:09 login01 automount[3939]: handle_packet: type = 0
Dec  4 09:15:09 login01 automount[3939]: handle_packet_missing: token
3534, name .raw_data
Dec  4 09:15:09 login01 automount[3939]: attempting to mount entry
/proj/.raw_data
Dec  4 09:15:09 login01 automount[1650]: lookup(yp): looking up .raw_data
Dec  4 09:15:09 login01 automount[3939]: mt->key set to .raw_data
Dec  4 09:15:09 login01 kernel: automount[1650]: segfault at
Quoted 1 line

0000000000000000 rip 0000002a95916be0 rsp 0000007fbfffd5c8 error 4

Can you enable core dumps and get us a stack trace of this?

Quoted 23 lines

Note that .raw_data map does not exist in the yp map so it should go into the negative cache - which happens correctly on login02:
ov 28 23:27:43 login01 automount[4201]: attempting to mount entry /proj/.raw_data

Nov 28 23:27:43 login01 automount[13394]: lookup(yp): looking up .raw_data
Nov 28 23:27:43 login01 automount[4201]: mt->key set to .raw_data
Nov 28 23:27:43 login01 automount[13394]: lookup(yp): key ".raw_data"
not found in map.
Nov 28 23:27:43 login01 automount[13394]: failed to mount /proj/.raw_data
Nov 28 23:27:43 login01 automount[13394]: umount_multi:
path=/proj/.raw_data incl=1
Nov 28 23:27:43 login01 automount[4201]: handle_child: got pid 13394,
sig 0 (0), stat 3
Nov 28 23:27:43 login01 automount[4201]: sig_child: found pending iop
pid 13394: signalled 0 (sig 0), exit status 3
Nov 28 23:27:43 login01 automount[4201]: update_negative_cache: key:
.raw_data
Nov 28 23:27:43 login01 automount[4201]: Adding negative cache entry for
key .raw_data
Nov 28 23:27:43 login01 automount[4201]: Key .raw_data added to negative
cache
Nov 28 23:27:43 login01 automount[4201]: send_fail: token=15625
Quoted 3 lines

I thought all the negative caching is done in user space, but apparently kernel has some influence here....
Any explanation?

The negative caching is done in userspace. I'm not sure I understand your question.

Cheers,
Jeff

maintainer
#2 lore raw
Quoted 59 lines · Jeff Moyer
Quoted 30 lines · Ondrej Valousek

Interesting thing:
I have 2 identical login01 and login02 machines (Rhel4, full updates). login01 is running old kernel (2.6.9-42.ELsmp), login02 is running the latest one (2.6.9-78.0.8.ELsmp). Both have autofs debug enabled and are running the latest autofs (autofs-4.1.3-234).
Now:
1) login01:
[root@login01 ondrejv]# cat /var/log/debug.log* | grep "Dec 4 09:15:09"

Dec  4 09:15:09 login01 automount[3939]: handle_packet: type = 0
Dec  4 09:15:09 login01 automount[3939]: handle_packet_missing: token
3533, name .raw_data
Dec  4 09:15:09 login01 automount[3939]: attempting to mount entry
/proj/.raw_data
Dec  4 09:15:09 login01 automount[1649]: lookup(yp): looking up .raw_data
Dec  4 09:15:09 login01 automount[3939]: mt->key set to .raw_data
Dec  4 09:15:09 login01 kernel: automount[1649]: segfault at
0000000000000000 rip 0000002a95916be0 rsp 0000007fbfffd5c8 error 4
Dec  4 09:15:09 login01 automount[3939]: handle_child: got pid 1649, sig
1 (11), stat 0
Dec  4 09:15:09 login01 automount[3939]: sig_child: found pending iop
pid 1649: signalled 1 (sig 11), exit status 0
Dec  4 09:15:09 login01 automount[3939]: send_fail: token=3533
Dec  4 09:15:09 login01 automount[3939]: handle_packet: type = 0
Dec  4 09:15:09 login01 automount[3939]: handle_packet_missing: token
3534, name .raw_data
Dec  4 09:15:09 login01 automount[3939]: attempting to mount entry
/proj/.raw_data
Dec  4 09:15:09 login01 automount[1650]: lookup(yp): looking up .raw_data
Dec  4 09:15:09 login01 automount[3939]: mt->key set to .raw_data
Dec  4 09:15:09 login01 kernel: automount[1650]: segfault at
Quoted 1 line

0000000000000000 rip 0000002a95916be0 rsp 0000007fbfffd5c8 error 4

Can you enable core dumps and get us a stack trace of this?

Quoted 23 lines

Note that .raw_data map does not exist in the yp map so it should go into the negative cache - which happens correctly on login02:
ov 28 23:27:43 login01 automount[4201]: attempting to mount entry /proj/.raw_data

Nov 28 23:27:43 login01 automount[13394]: lookup(yp): looking up .raw_data
Nov 28 23:27:43 login01 automount[4201]: mt->key set to .raw_data
Nov 28 23:27:43 login01 automount[13394]: lookup(yp): key ".raw_data"
not found in map.
Nov 28 23:27:43 login01 automount[13394]: failed to mount /proj/.raw_data
Nov 28 23:27:43 login01 automount[13394]: umount_multi:
path=/proj/.raw_data incl=1
Nov 28 23:27:43 login01 automount[4201]: handle_child: got pid 13394,
sig 0 (0), stat 3
Nov 28 23:27:43 login01 automount[4201]: sig_child: found pending iop
pid 13394: signalled 0 (sig 0), exit status 3
Nov 28 23:27:43 login01 automount[4201]: update_negative_cache: key:
.raw_data
Nov 28 23:27:43 login01 automount[4201]: Adding negative cache entry for
key .raw_data
Nov 28 23:27:43 login01 automount[4201]: Key .raw_data added to negative
cache
Nov 28 23:27:43 login01 automount[4201]: send_fail: token=15625
Quoted 6 lines
Quoted 3 lines

I thought all the negative caching is done in user space, but apparently kernel has some influence here....
Any explanation?

The negative caching is done in userspace. I'm not sure I understand your question.

Me neither but I'm working on the negative caching of non-existent map keys. Turns out to be a bit tricky when multiple nsswitch sources can be tried for each mount since the map may not exist within a given source and so we get NSS_STATUS_UNAVAIL instead of NSS_STATUS_NOTFOUND. Maybe I should just cache the unavailable in the same way?

Ian

#3 lore raw
Quoted 10 lines · Ian Kent
Quoted 3 lines · Jeff Moyer

The negative caching is done in userspace. I'm not sure I understand your question.

Me neither but I'm working on the negative caching of non-existent map keys. Turns out to be a bit tricky when multiple nsswitch sources can be tried for each mount since the map may not exist within a given source and so we get NSS_STATUS_UNAVAIL instead of NSS_STATUS_NOTFOUND. Maybe I should just cache the unavailable in the same way?

Ian, this is autofs version 4. It's been a while since I implemented the negative caching there, but I'd be interested in knowing if you think I got it wrong for the case above!

Cheers,
Jeff

maintainer
#4 lore raw
Quoted 13 lines · Jeff Moyer
Quoted 10 lines · Ian Kent
Quoted 3 lines · Jeff Moyer

The negative caching is done in userspace. I'm not sure I understand your question.

Me neither but I'm working on the negative caching of non-existent map keys. Turns out to be a bit tricky when multiple nsswitch sources can be tried for each mount since the map may not exist within a given source and so we get NSS_STATUS_UNAVAIL instead of NSS_STATUS_NOTFOUND. Maybe I should just cache the unavailable in the same way?

Quoted 3 lines

Ian, this is autofs version 4. It's been a while since I implemented the negative caching there, but I'd be interested in knowing if you think I got it wrong for the case above!

Oh, yeah, lucky it will be Xmas holiday time soon. I'm going to have a look around the v4 code tomorrow. I'll spend some time on this as well.

Quoted 3 lines

Cheers,
Jeff

#5 lore raw
In reply to Jeff Moyer
Quoted 6 lines
Quoted 5 lines

I thought all the negative caching is done in user space, but apparently kernel has some influence here....
Any explanation?

Quoted 6 lines

The negative caching is done in userspace. I'm not sure I understand your question.

Cheers,
Jeff

Hi Jeff,
Ok I was thinking this way:
I experience this only on U7 system running U4 kernel - on the same U7 system running U7 kernel I can not replicate this. If the negative caching is done in the user space and there is a bug there which causes the autofs to segfault, how is it possible that is won't segfault on the U7 kernel?
I do not know I am still trying to find out if there is any connection to the problem with the automounter hanging that I reported earlier as:
- in U7 system running U7 kernel, the autofs does not crash, but hangs (after few days).
- in U7 system running U4 kernel, the autofs forks crash regularly but otherwise it works fine (I mean at least the parent autofs fork continue living...)
You know what I am trying to say?
Thanks,
Ondrej