Skip to contentSkip to conversation

Automount/NFS issues causing executables to appear corrupted

DISCUSSION Venkata Ravella 5 participants 9 messages Apr 20, 2004 – Apr 21, 2004 1 message not in the archive Kernel automounter

lore
Share & tools
lore
#0 lore raw

autofs version is autofs-3.1.7-21

I also have one new update. We started seeing similar problem on the system running the kernel 2.4.18-e.12smp which has the same version(3.1.7-21) of autofs as well.

This may or may not be an autofs problem but, restarting autofs fixes this problem temporarily.

Quoted 19 lines

Please cc autofs questions to the list at [email protected].

Quoted 7 lines · Venkata Ravella

The current kernel we use is default 7.2 kernel with two modifications:
1) BM patch applied to extend address space for a single process to 3.6GB
2) mnt patch applied to allow upto 1024 nfs mount points

uname -r output:
2.4.7-10mntBMsmp

What autofs version?

To be honest it's a bit hard to see how this is an autofs issue. Mind, having said that, ....

Ian

#1 lore raw
Quoted 9 lines · Venkata Ravella

autofs version is autofs-3.1.7-21

I also have one new update. We started seeing similar problem on the system running the kernel 2.4.18-e.12smp which has the same version(3.1.7-21) of autofs as well.

This may or may not be an autofs problem but, restarting autofs fixes this problem temporarily.

That will cause an NFS remount. This really feels much more like an NFS problem.

-hpa

maintainer
#2 lore raw
Quoted 7 lines · H. Peter Anvin
Quoted 6 lines · Venkata Ravella

autofs version is autofs-3.1.7-21

I also have one new update. We started seeing similar problem on the system running the kernel 2.4.18-e.12smp which has the same version(3.1.7-21) of autofs as well.

Quoted 6 lines
Quoted 3 lines

This may or may not be an autofs problem but, restarting autofs fixes this problem temporarily.

That will cause an NFS remount. This really feels much more like an NFS problem.

Certainly does.

Venkata,

Can you also forward this question to the nfs list at [email protected]. Sorry to ask you to post all over the place.

Please investigate the NFS client patches maintained by Trond Myklebust. Check nfs.sourceforge.net. We found we had to use them in early 2.4 versions.

Ian

A message that is not in the archive (sent off-list or not yet received): [email protected]

maintainer
#4 lore raw
Quoted 5 lines · Venkata Ravella

Posted. I am very thankful to your pointers. Both those lists are closed lists and moderated. I do not think moderator got a chance to look at it and post.

I certainly accepted your post to the autofs list.

Hope someone on one of the lists has seen this in the past.

Ian

#5 lore raw
In reply to Venkata Ravella

question,
Is the file system mounted with the 'soft' option? i.e. on the systems that are causing problems try if mount | grep -i soft >>/dev/null 2>&1 then
echo "we have soft mounts"
else
echo "good, only normal mounts"
fi

We had a problem that caused me headaches for 6 months to track down... One of the other admins had chosen to mount all the file systems with the soft option and propagated this to all machines he could, that is to any I did not control, and then people using his config started asking me why they were getting IO errors transferring files to/from the file server I maintained. if the file was bigger than would fit in the normal [wr]size, which defaults to 1024 bytes {or 4096 dependent on which kernel version I believe} the probability of an IO error during normal operations went from 0 towards certainty by the time the file was 650 MBytes, generally would happen by ~100MBytes.

My server was a sun ultra 2 running solaris 2.6, the clients were Linux running 2.[02].X and a mix of autofs-3 and autofs-4 (which ever was installed with the distros, RH6-9 & Slack7-9.1).

I have a script being ran in everyone's .profile now to help me find any remaining soft holdouts because it caused so much trouble.
The alternative to 'soft' for us was 'hard,intr' which at least allows you to break the applications when you really need to, but be much more robust in normal operations.

Quoted 29 lines · Venkata Ravella

autofs version is .1.7-21

I also have one new update. We started seeing similar problem on the system running the kernel 2.4.18-e.12smp which has the same version(3.1.7-21) of autofs as well.

This may or may not be an autofs problem but, restarting autofs fixes this problem temporarily.

Quoted 19 lines

Please cc autofs questions to the list at [email protected].

Quoted 7 lines · Venkata Ravella

The current kernel we use is default 7.2 kernel with two modifications:
1) BM patch applied to extend address space for a single process to 3.6GB
2) mnt patch applied to allow upto 1024 nfs mount points

uname -r output:
2.4.7-10mntBMsmp

What autofs version?

To be honest it's a bit hard to see how this is an autofs issue. Mind, having said that, ....

Ian

Signature · 3 lines
Todd Denniston
Crane Division, Naval Surface Warfare Center (NSWC Crane) 
Harnessing the Power of Technology for the Warfighter
#6 lore raw
Quoted 22 lines · Todd Denniston

question,
Is the file system mounted with the 'soft' option? i.e. on the systems that are causing problems try if mount | grep -i soft >>/dev/null 2>&1 then
echo "we have soft mounts"
else
echo "good, only normal mounts"
fi

We had a problem that caused me headaches for 6 months to track down... One of the other admins had chosen to mount all the file systems with the soft option and propagated this to all machines he could, that is to any I did not control, and then people using his config started asking me why they were getting IO errors transferring files to/from the file server I maintained. if the file was bigger than would fit in the normal [wr]size, which defaults to 1024 bytes {or 4096 dependent on which kernel version I believe} the probability of an IO error during normal operations went from 0 towards certainty by the time the file was 650 MBytes, generally would happen by ~100MBytes.

Quoted 3 lines

My server was a sun ultra 2 running solaris 2.6, the clients were Linux running 2.[02].X and a mix of autofs-3 and autofs-4 (which ever was installed with the distros, RH6-9 & Slack7-9.1).

<SNIP>

typo
they were actually 2.[24].X systems with maybe one 2.0.?? system hidden amongst the bunch.

Signature · 3 lines
Todd Denniston
Crane Division, Naval Surface Warfare Center (NSWC Crane) 
Harnessing the Power of Technology for the Warfighter
#7 lore raw
In reply to Todd Denniston

Sorry to continue a non-automount issue, but this is where it was posted...

Quoted 7 lines · Todd Denniston

question,
Is the file system mounted with the 'soft' option? i.e. on the systems that are causing problems try mount | grep -i soft

We had a problem that caused me headaches for 6 months to track down...

Quoted 7 lines

...probability of an IO error during normal operations went from 0 towards certainty by the time the file was 650 MBytes, generally would happen by ~100MBytes.

My server was a sun ultra 2 running solaris 2.6, the clients were Linux running 2.[02].X and a mix of autofs-3 and autofs-4 (which ever was installed with the distros, RH6-9 & Slack7-9.1).

We have Solaris 2.6, Solaris 8 (not tested), SuSE 8.2 (kernel 2.4.20) and SuSE 9.0 (kernel 2.4.21, not tested). I just ran some tests as follows:
Write one file of 1.3 Gb into the partner's NFS-exported filesystem. Read it back comparing bit-for-bit. Delete the NFS file. This was tried twice with a Solaris 2.6 partner and twice with a Linux (2.4.20) partner. The local machine has Linux (2.4.20). Both partners were on a different subnet, but traffic was light and dropped UDP packets probably were very few. All NFS mounts were courtesy of the automounter. All were soft,
specifically: -rsize=8192,wsize=8192,retry=1,soft.

There were no errors whatsoever. Execution times were identical on repeat trials (meaning no erratic network timeouts). At Mathnet, historically we do not see any of the described symptoms.

I wonder what's going on at your end. If it's going to jump up and bite us in the future...

James F. Carter Voice 310 825 2897 FAX 310 206 6673
UCLA-Mathnet; 6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: [email protected] http://www.math.ucla.edu/~jimc (q.v. for PGP key)

#8 lore raw

Re: Re: Automount/NFS issues causing executables to appearcorrupted

Quoted 2 lines · Jim Carter

Sorry to continue a non-automount issue, but this is where it was posted...

This is the only NFS related list I am subscribed to.

Quoted 18 lines · Todd Denniston
Quoted 8 lines

question,
Is the file system mounted with the 'soft' option? i.e. on the systems that are causing problems try mount | grep -i soft

We had a problem that caused me headaches for 6 months to track down...

Quoted 7 lines

...probability of an IO error during normal operations went from 0 towards certainty by the time the file was 650 MBytes, generally would happen by ~100MBytes.

My server was a sun ultra 2 running solaris 2.6, the clients were Linux running 2.[02].X and a mix of autofs-3 and autofs-4 (which ever was installed with the distros, RH6-9 & Slack7-9.1).

Quoted 8 lines

We have Solaris 2.6, Solaris 8 (not tested), SuSE 8.2 (kernel 2.4.20) and SuSE 9.0 (kernel 2.4.21, not tested). I just ran some tests as follows:
Write one file of 1.3 Gb into the partner's NFS-exported filesystem. Read it back comparing bit-for-bit. Delete the NFS file. This was tried twice with a Solaris 2.6 partner and twice with a Linux (2.4.20) partner. The local machine has Linux (2.4.20). Both partners were on a different subnet, but traffic was light and dropped UDP packets probably were very few.

There are at least two differences
1) you have light network traffic, at times we have a couple of video streams going across our 100Mb net, and 50 users that have a bad habit of keeping there netscape caches on the network drives.
2) our server has a veritas controlled 64 disk software raid set which seems to eat kernel time, nfs seems to use a lot of kernel time too, so probably more dropped UDP packets.
3) solaris nfs server -> linux clients.... I have heard that in olden days the nfs servers and clients of different OSs handled things differently from one another and this caused some lossage, which is probably more apparent in error conditions like dropped UDP packets.
4) Oh, and all these disks were on fibre channel from back when dot hill was box hill (97-98 time frame), further investigation showed (when I did it a long time ago) that when linux was using fibre cards that reported the same make+model+version as ours they had to do several things to keep the cards running right ... seems they did not quite work right, and we never got updated drivers from box hill before our support contracts ran out (which was before I took over the machine).

Quoted 6 lines

All NFS mounts were courtesy of the automounter. All were soft,
specifically: -rsize=8192,wsize=8192,retry=1,soft.

There were no errors whatsoever. Execution times were identical on repeat trials (meaning no erratic network timeouts). At Mathnet, historically we do not see any of the described symptoms.

for me copy times would change on the order of 5 to 10 minutes for a 650MB file.

Quoted 4 lines

I wonder what's going on at your end. If it's going to jump up and bite us in the future...

As our user load increased from 25 to 50, so did the frequency of IO errors.

from the linux man nfs
soft If an NFS file operation has a major timeout then report
an I/O error to the calling program. The default is to continue retrying NFS file operations indefinitely.

hard If an NFS file operation has a major timeout then report
"server not responding" on the console and continue retrying indefinitely. This is the default.

Signature · 3 lines
Todd Denniston
Crane Division, Naval Surface Warfare Center (NSWC Crane) 
Harnessing the Power of Technology for the Warfighter
#9 lore raw

Re: Re: Automount/NFS issues causing executables to appearcorrupted

Quoted 4 lines · Todd Denniston

There are at least two differences
1) you have light network traffic, at times we have a couple of video streams going across our 100Mb net, and 50 users that have a bad habit of keeping there netscape caches on the network drives.

Thanks very much for your reply. Your configuration is indeed very different from ours, and it's entirely believable that we can be reliable with soft mounts while you cannot. I've reassured my management. End panic mode :-)

James F. Carter Voice 310 825 2897 FAX 310 206 6673
UCLA-Mathnet; 6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: [email protected] http://www.math.ucla.edu/~jimc (q.v. for PGP key)