autofs4 problem
Hello,
i have a really severe autofs4 problem. Kernel is 2.4.7, Redhat-7.0, Hardware is a Fujitsu/Siemens HPC Line dual Processor, so kernel is built with SMP turned on. automounter Daemon is 4.0.0pre10 . gcc is 2.96:
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/2.96/specs gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-85) RAM is 2 GB, CPU: Pentium III (Coppermine), 256 kb Cache, Boot disk is an IDE disk, a SCSI Symbios 53c1010 hostadapter and a connected additional disk is installed, but not used, the SCSI-adapter driver isn't loaded. IDE controller is ServerWorks OSB4.
I'm desperately trying to find the reason for my quite reproducible problem (it's not the locking thing in the kernel that has been fixed since quite a while)
Assume an map like that:
kaefer -rw /tmp2 kaefer:/tmp2
gustav -rw /tmp2 gustav:/tmp2
distel -rw /tmp2 distel:/tmp2
averna -rw /tmp2 averna:/tmp2
solp1 -rw /tmp2 solp1:/tmp2
...
host-1 -rw /tmp2 host-1:/tmp2
host-2 -rw /tmp2 host-2:/tmp2
...
i.e. all of those hosts have a /tmp2 directory, that is exported properly. Automounter Daemon runs with the following args:
automount -t 1 /mnt/my-subdir yp map-name grpid,hard,nodevs,nosymlink
(or with file instead yp, makes no difference)
Basically we don't want a timeout, which is that short. But it helps to reproduce the problem within minutes. Otherwise it shows up after some hours, what is not acceptable for users with long-running compute jobs. The software accesses the /tmp2 directories and home-direcotries (also autofs-Mounted) from time to time and once within a few hours the users see either a "Permission denied" or "Directory does not exist". And this happens with that stress test, too. Assume the following script:
#!/bin/sh
# build a list of files to touch within one command I=1 while [ $I -le 32 ] ; do FILES="$FILES /mnt/my-subdir/host-$I/tmp2/affile" I=`expr $I + 1` done
while true ; do
echo 'trying to touch and ls'
touch $FILES
if [ $? -ne 0 ] ; then
exit 1
fi
ls -ld $FILES
date
/bin/rm -f /tmp/bla
head -4c /dev/random > /tmp/bla
RANDNUM=sum /tmp/bla | awk '{print $1}'
WAITTIME=expr '(' $RANDNUM ')' / 2 + 980000
# random sleep time between 980 ms and ~ 1020 with script # interpretatin ... delay - neglible
ls -ld $FILES>/dev/null
usleep $WAITTIME
done
What i try to achieve with that random delay is, that during expire of the 32 mounts another access to them occurs and this in fact happens quite frequently, some- times leading to the mentioned error message during touch, followed by the exit 1, sometimes not. I wouldn't have a problem, if the problem showed up only under this stress test, but it's happening also under `normal' circumstances. This test runs between few seconds and several (say, 20) minutes until failing. It is not depending on the kind of NFS-Server. This can be either a Linux or Sun machine. No notable difference. I tcpdumped the network traffic. No clue. Or better: i can see, that no new mount request is leaving the machine directed to the server, that supplies one of the directories. Sometimes the touch complains about olny one missing directory, sometimes about 8 or so, in successive order (e.g.:/mnt/my-subdir/host-7/tmp2 through /mnt/my-subdir/host-12/tmp2 )
I suspect, that the autofs4 returns from the revalidate thinking, sth has been or is mounted, but has not in reality or has been expired in the meantime. I experimented a lot. I modified the daemon to serialize the mounts, not just fork and report to the kernel, when done. Didn't help. Have a lot of strace output about that. Looks all ok. So i suspected, it's the kernel autofs4. Found several things, that looked weird to me and modified them, but no success. What i tried (please don't laugh, maybe some of the stuff is ridiculous):
* protected the queues linked list using a spinlock during traversal and modification against parallel accesses
* modified the autofs4_wait function in several more ways to have the queues list distinguish between the different notification requests. (If thought it might be a problem, that a mount request is being added, but finding an expire request, thinking 'oh, we have that already, can jump on the request for mount', though in fact waiting for expire, but i was wrong here, one more time)
* added an additional flag, cause i thought, it might happen during execution of the first lines of try_to_fill_dentry, where the dcache is not yet locked, that the test for AUTOFS_INF_EXPIRING fails and one thread/CPU walks on through the function while another CPU works in parallel on an ioctl-issued autofs4_expire_multi and before setting the flag, the other thread has passed the if. I added another flag, that is set in try_to_fill_dentry and checked in the autofs4_expire_multi to avoid, that the mountpoint in question will be expired, when not expected to expire. But this wasn't the problem either.
I have tons of syslog and strace -f output of the daemon, but no more clues here.
This problem is quite a show-stopper here, otherwise we must set the expire time to infinite, what is not really desirable or we must use the userspace amd :-(
Any hint how to narrow the problem is appreciated.
Thanks for reading and with kindest regards,
Albert
Signature · 4 lines
Albert Flügel Tel.: +49-89-636-27690 science+computing AG Fax.: +49-89-636-21950 bei: Infineon AG D1: +49-171-3698673 E-Mail: [email protected]
Hi,
additional info about this:
If the map does not look like e.g.:
host-1 -rw /tmp2 host-1:/tmp2
but instead:
host-1 -rw / host-1:/tmp2
it seems to work perfectly. But we cannot do this without other problems here. If the map looks like this, what would be an acceptable workaround:
host-1 -rw / host-1:/tmp2 /tmp2 host-1:/tmp2
the same happens like with the current setup, and it's MUCH slower. So this is no solution.
Thanks,
Albert
Quoted 153 lines · Fluegel Albert
Hello,
i have a really severe autofs4 problem. Kernel is 2.4.7, Redhat-7.0, Hardware is a Fujitsu/Siemens HPC Line dual Processor, so kernel is built with SMP turned on. automounter Daemon is 4.0.0pre10 . gcc is 2.96:
Reading specs from/usr/lib/gcc-lib/i386-redhat-linux/2.96/specsgcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-85) RAM is 2 GB, CPU: Pentium III (Coppermine), 256 kb Cache, Boot disk is an IDE disk, a SCSI Symbios 53c1010 hostadapter and a connected additional disk is installed, but not used, the SCSI-adapter driver isn't loaded. IDE controller is ServerWorks OSB4.I'm desperately trying to find the reason for my quite reproducible problem (it's not the locking thing in the kernel that has been fixed since quite a while)
Assume an map like that:
kaefer -rw /tmp2 kaefer:/tmp2
gustav -rw /tmp2 gustav:/tmp2
distel -rw /tmp2 distel:/tmp2
averna -rw /tmp2 averna:/tmp2
solp1 -rw /tmp2 solp1:/tmp2
...
host-1 -rw /tmp2 host-1:/tmp2
host-2 -rw /tmp2 host-2:/tmp2
...i.e. all of those hosts have a /tmp2 directory, that is exported properly. Automounter Daemon runs with the following args:
automount -t 1
/mnt/my-subdiryp map-name grpid,hard,nodevs,nosymlink(or with file instead yp, makes no difference)
Basically we don't want a timeout, which is that short. But it helps to reproduce the problem within minutes. Otherwise it shows up after some hours, what is not acceptable for users with long-running compute jobs. The software accesses the /tmp2 directories and home-direcotries (also autofs-Mounted) from time to time and once within a few hours the users see either a "Permission denied" or "Directory does not exist". And this happens with that stress test, too. Assume the following script:
#!/bin/sh
# build a list of files to touch within one command I=1 while [ $I -le 32 ] ; do FILES="$FILES /mnt/my-subdir/host-$I/tmp2/affile" I=`expr $I + 1` donewhile true ; do
echo 'trying to touch and ls'
touch $FILES
if [ $? -ne 0 ] ; then
exit 1
fi
ls -ld $FILESdate
/bin/rm-f/tmp/bla
head -4c/dev/random>/tmp/bla
RANDNUM=sum /tmp/bla | awk '{print $1}'
WAITTIME=expr '(' $RANDNUM ')' / 2 + 980000# random sleep time between 980 ms and ~ 1020 with script # interpretatin ... delay - negliblels -ld $FILES>/dev/null
usleep $WAITTIME
doneWhat i try to achieve with that random delay is, that during expire of the 32 mounts another access to them occurs and this in fact happens quite frequently, some- times leading to the mentioned error message during touch, followed by the exit 1, sometimes not. I wouldn't have a problem, if the problem showed up only under this stress test, but it's happening also under `normal' circumstances. This test runs between few seconds and several (say, 20) minutes until failing. It is not depending on the kind of NFS-Server. This can be either a Linux or Sun machine. No notable difference. I tcpdumped the network traffic. No clue. Or better: i can see, that no new mount request is leaving the machine directed to the server, that supplies one of the directories. Sometimes the touch complains about olny one missing directory, sometimes about 8 or so, in successive order (e.g.:
/mnt/my-subdir/host-7/tmp2through/mnt/my-subdir/host-12/tmp2)I suspect, that the autofs4 returns from the revalidate thinking, sth has been or is mounted, but has not in reality or has been expired in the meantime. I experimented a lot. I modified the daemon to serialize the mounts, not just fork and report to the kernel, when done. Didn't help. Have a lot of strace output about that. Looks all ok. So i suspected, it's the kernel autofs4. Found several things, that looked weird to me and modified them, but no success. What i tried (please don't laugh, maybe some of the stuff is ridiculous):
* protected the queues linked list using a spinlock during traversal and modification against parallel accesses
* modified the
autofs4_waitfunction in several more ways to have the queues list distinguish between the different notification requests. (If thought it might be a problem, that a mount request is being added, but finding an expire request, thinking 'oh, we have that already, can jump on the request for mount', though in fact waiting for expire, but i was wrong here, one more time)* added an additional flag, cause i thought, it might happen during execution of the first lines of
try_to_fill_dentry, where the dcache is not yet locked, that the test forAUTOFS_INF_EXPIRINGfails and one thread/CPU walks on through the function while another CPU works in parallel on an ioctl-issuedautofs4_expire_multiand before setting the flag, the other thread has passed the if. I added another flag, that is set intry_to_fill_dentryand checked in theautofs4_expire_multito avoid, that the mountpoint in question will be expired, when not expected to expire. But this wasn't the problem either.I have tons of syslog and strace -f output of the daemon, but no more clues here.
This problem is quite a show-stopper here, otherwise we must set the expire time to infinite, what is not really desirable or we must use the userspace amd :-(
Any hint how to narrow the problem is appreciated.
Thanks for reading and with kindest regards,
Albert
--
Albert Flügel Tel.: +49-89-636-27690science+computing AG Fax.: +49-89-636-21950 bei: Infineon AG D1: +49-171-3698673 E-Mail: [email protected]-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 5 lines
Albert Flügel Tel.: +49-89-636-27690 science+computing AG Fax.: +49-89-636-21950 bei: Infineon AG D1: +49-171-3698673 E-Mail: [email protected] Technical Project Leader TDSC
Hi,
i posted the stuff below several weeks ago. Now we observed a phenomenon related to system time. So a basic question:
Could it be, autofs4 gets screwed up, when an ntpd modifies the system time without slew i.e. as a step wider than several seconds ?
Thanks in advance, please respond to [email protected]
Albert
Quoted 153 lines · Fluegel Albert
Hello,
i have a really severe autofs4 problem. Kernel is 2.4.7, Redhat-7.0, Hardware is a Fujitsu/Siemens HPC Line dual Processor, so kernel is built with SMP turned on. automounter Daemon is 4.0.0pre10 . gcc is 2.96:
Reading specs from/usr/lib/gcc-lib/i386-redhat-linux/2.96/specsgcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-85) RAM is 2 GB, CPU: Pentium III (Coppermine), 256 kb Cache, Boot disk is an IDE disk, a SCSI Symbios 53c1010 hostadapter and a connected additional disk is installed, but not used, the SCSI-adapter driver isn't loaded. IDE controller is ServerWorks OSB4.I'm desperately trying to find the reason for my quite reproducible problem (it's not the locking thing in the kernel that has been fixed since quite a while)
Assume an map like that:
kaefer -rw /tmp2 kaefer:/tmp2
gustav -rw /tmp2 gustav:/tmp2
distel -rw /tmp2 distel:/tmp2
averna -rw /tmp2 averna:/tmp2
solp1 -rw /tmp2 solp1:/tmp2
...
host-1 -rw /tmp2 host-1:/tmp2
host-2 -rw /tmp2 host-2:/tmp2
...i.e. all of those hosts have a /tmp2 directory, that is exported properly. Automounter Daemon runs with the following args:
automount -t 1
/mnt/my-subdiryp map-name grpid,hard,nodevs,nosymlink(or with file instead yp, makes no difference)
Basically we don't want a timeout, which is that short. But it helps to reproduce the problem within minutes. Otherwise it shows up after some hours, what is not acceptable for users with long-running compute jobs. The software accesses the /tmp2 directories and home-direcotries (also autofs-Mounted) from time to time and once within a few hours the users see either a "Permission denied" or "Directory does not exist". And this happens with that stress test, too. Assume the following script:
#!/bin/sh
# build a list of files to touch within one command I=1 while [ $I -le 32 ] ; do FILES="$FILES /mnt/my-subdir/host-$I/tmp2/affile" I=`expr $I + 1` donewhile true ; do
echo 'trying to touch and ls'
touch $FILES
if [ $? -ne 0 ] ; then
exit 1
fi
ls -ld $FILESdate
/bin/rm-f/tmp/bla
head -4c/dev/random>/tmp/bla
RANDNUM=sum /tmp/bla | awk '{print $1}'
WAITTIME=expr '(' $RANDNUM ')' / 2 + 980000# random sleep time between 980 ms and ~ 1020 with script # interpretatin ... delay - negliblels -ld $FILES>/dev/null
usleep $WAITTIME
doneWhat i try to achieve with that random delay is, that during expire of the 32 mounts another access to them occurs and this in fact happens quite frequently, some- times leading to the mentioned error message during touch, followed by the exit 1, sometimes not. I wouldn't have a problem, if the problem showed up only under this stress test, but it's happening also under `normal' circumstances. This test runs between few seconds and several (say, 20) minutes until failing. It is not depending on the kind of NFS-Server. This can be either a Linux or Sun machine. No notable difference. I tcpdumped the network traffic. No clue. Or better: i can see, that no new mount request is leaving the machine directed to the server, that supplies one of the directories. Sometimes the touch complains about olny one missing directory, sometimes about 8 or so, in successive order (e.g.:
/mnt/my-subdir/host-7/tmp2through/mnt/my-subdir/host-12/tmp2)I suspect, that the autofs4 returns from the revalidate thinking, sth has been or is mounted, but has not in reality or has been expired in the meantime. I experimented a lot. I modified the daemon to serialize the mounts, not just fork and report to the kernel, when done. Didn't help. Have a lot of strace output about that. Looks all ok. So i suspected, it's the kernel autofs4. Found several things, that looked weird to me and modified them, but no success. What i tried (please don't laugh, maybe some of the stuff is ridiculous):
* protected the queues linked list using a spinlock during traversal and modification against parallel accesses
* modified the
autofs4_waitfunction in several more ways to have the queues list distinguish between the different notification requests. (If thought it might be a problem, that a mount request is being added, but finding an expire request, thinking 'oh, we have that already, can jump on the request for mount', though in fact waiting for expire, but i was wrong here, one more time)* added an additional flag, cause i thought, it might happen during execution of the first lines of
try_to_fill_dentry, where the dcache is not yet locked, that the test forAUTOFS_INF_EXPIRINGfails and one thread/CPU walks on through the function while another CPU works in parallel on an ioctl-issuedautofs4_expire_multiand before setting the flag, the other thread has passed the if. I added another flag, that is set intry_to_fill_dentryand checked in theautofs4_expire_multito avoid, that the mountpoint in question will be expired, when not expected to expire. But this wasn't the problem either.I have tons of syslog and strace -f output of the daemon, but no more clues here.
This problem is quite a show-stopper here, otherwise we must set the expire time to infinite, what is not really desirable or we must use the userspace amd :-(
Any hint how to narrow the problem is appreciated.
Thanks for reading and with kindest regards,
Albert
--
Albert Flügel Tel.: +49-89-636-27690science+computing AG Fax.: +49-89-636-21950 bei: Infineon AG D1: +49-171-3698673 E-Mail: [email protected]-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 5 lines
Albert Flügel Tel.: +49-89-636-27690 science+computing AG Fax.: +49-89-636-21950 bei: Infineon AG D1: +49-171-3698673 E-Mail: [email protected] Technical Project Leader TDSC