Skip to contentSkip to conversation

Null characters in files on NFS mounted volume....

DISCUSSION Dan Yocum 4 participants 11 messages May 13, 2002 – May 24, 2002

Subsystems (2)
NFS client NFS server
lore
Share & tools
lore

Dan Yocum

author
#0 lore raw

Hi all,

I've got a "New and Exciting" <tm> bug using 2.4.18 with Trond's client patches (sans the rpc_tweaks dif), but I'm not sure it's a client side problem.

This is an intermitent problem: When attempting to read a file (any file) on an NFS mounted volume, there appear to be "holes" in the file filled with ASCII NULL characters. Reading the file on the local machines shows no such holes, and other NFS clients don't see the holes, either. A simple umount/remount solves the problem.

Oh, I should add that these are autofs mounted volumes.

We thought these might be network driver problems, but we're seeing it on natsemi, acenic and eepro100 drivers, and I doubt they could all be flaky.

Any ideas?

Thanks,
Dan

Signature · 12 lines
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
[email protected], http://www.sdss.org
SDSS.  Mapping the Universe.

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#1 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Dan Yocum <[email protected]> writes:

> Hi all, I've got a "New and Exciting" <tm> bug using 2.4.18 > with Trond's client patches (sans the rpc_tweaks dif), but I'm > not sure it's a client side problem.

> This is an intermitent problem: When attempting to read a file > (any file) on an NFS mounted volume, there appear to be "holes" > in the file filled with ASCII NULL characters. Reading the > file on the local machines shows no such holes, and other NFS > clients don't see the holes, either. A simple umount/remount > solves the problem.

Any details?

Are these for instance ordinary files? Are you perhaps engaging in some multiple NFS client read/writes to the same file? ... Is it tied to the client patches, or does it occur with stock 2.4.18 fresh off the l-k presses?

Cheers,
Trond

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#2 lore raw

Trond,

Nothing special in /var/log/messages on either the client or server that would help in debugging the problem, unfortunately.

I'm seeing the problem on these 2 client systems:

vanilla 2.4.18 + linux-2.4.18-NFS_ALL.dif - linux-2.4.18-rpc_tweaks.dif + xfs-1.1-2.4.18-all.patch.

("-", above, meaning 'patch -R')

And:

kernel-2.4.9-13SGI_XFS_1.0.2, which is a Red Hat 2.4.9 kernel with XFS v1.0.2 patches (built by SGI folks). I can get you the appropriate patches from their patch set, if you want.

The server system is the 1st kernel described above.

The files are FITS files (text header tables, followed by binary data). The data was written once to disk and is read by many clients.

If I could reproduce the problem at will, I'd try a generic kernel, but it's intermittent at best, and we need the patches to access the IRIX servers. :-(

When it happens again, I'll do some more debugging on the problem.

Thanks,
Dan

Quoted 23 lines · Trond Myklebust
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

" " == Dan Yocum <[email protected]> writes:

> Hi all, I've got a "New and Exciting" <tm> bug using 2.4.18 > with Trond's client patches (sans the rpc_tweaks dif), but I'm > not sure it's a client side problem.

> This is an intermitent problem: When attempting to read a file > (any file) on an NFS mounted volume, there appear to be "holes" > in the file filled with ASCII NULL characters. Reading the > file on the local machines shows no such holes, and other NFS > clients don't see the holes, either. A simple umount/remount > solves the problem.

Any details?

Are these for instance ordinary files? Are you perhaps engaging in some multiple NFS client read/writes to the same file? ... Is it tied to the client patches, or does it occur with stock 2.4.18 fresh off the l-k presses?

Cheers,
Trond

Signature · 12 lines
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
[email protected], http://www.sdss.org
SDSS.  Mapping the Universe.

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#3 lore raw

M=E5ndag 13. mai 2002 18:54 skreiv Dan Yocum:

Quoted 3 lines

Trond,

Nothing special in /var/log/messages on either the client or server tha=

t

Quoted 5 lines

would help in debugging the problem, unfortunately.

I'm seeing the problem on these 2 client systems:

vanilla 2.4.18 + linux-2.4.18-NFS_ALL.dif - linux-2.4.18-rpc_tweaks.dif=

+

Quoted 8 lines

xfs-1.1-2.4.18-all.patch.

("-", above, meaning 'patch -R')

And:

kernel-2.4.9-13SGI_XFS_1.0.2, which is a Red Hat 2.4.9 kernel with XFS v1.0.2 patches (built by SGI folks). I can get you the appropriate pat=

ches

Quoted 3 lines

from their patch set, if you want.

The server system is the 1st kernel described above.

Are you able to reproduce it on a non-XFS server?

Cheers,
Trond

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#4 lore raw
Quoted 2 lines · Trond Myklebust

Are you able to reproduce it on a non-XFS server?

Heh. I don't have any non-XFS machines. But, I'll post the problem to the XFS lists - maybe someone there has seen it.

Cheers,
Dan

Signature · 12 lines
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
[email protected], http://www.sdss.org
SDSS.  Mapping the Universe.

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#5 lore raw
In reply to Dan Yocum
Quoted 11 lines · On Monday May 13, [email protected] wrote:

Hi all,

I've got a "New and Exciting" <tm> bug using 2.4.18 with Trond's client patches (sans the rpc_tweaks dif), but I'm not sure it's a client side problem.

This is an intermitent problem: When attempting to read a file (any file) on an NFS mounted volume, there appear to be "holes" in the file filled with ASCII NULL characters. Reading the file on the local machines shows no such holes, and other NFS clients don't see the holes, either. A simple umount/remount solves the problem.

I remembers long ago some referred to NFS as
Nulls Frequently Substituted
because they were experiencing exactly this sort of thing, and it wasn't on Linux at all.

You wouldn't be using 'soft' mounts would you?

NeilBrown

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#6 lore raw
Quoted 14 lines · Neil Brown
Quoted 11 lines · On Monday May 13, [email protected] wrote:

Hi all,

I've got a "New and Exciting" <tm> bug using 2.4.18 with Trond's client patches (sans the rpc_tweaks dif), but I'm not sure it's a client side problem.

This is an intermitent problem: When attempting to read a file (any file) on an NFS mounted volume, there appear to be "holes" in the file filled with ASCII NULL characters. Reading the file on the local machines shows no such holes, and other NFS clients don't see the holes, either. A simple umount/remount solves the problem.

Quoted 4 lines

I remembers long ago some referred to NFS as
Nulls Frequently Substituted
because they were experiencing exactly this sort of thing, and it wasn't on Linux at all.

Ha! :-)

I did find a similar occurence while googling
(http://www.geocrawler.com/archives/3/4/1998/11/0/59164/), reported back in '98 on LinuxPPC, that produced similar results, but since this happened 3 times on the same day to 3 different machines, I thought I'd report it.

Quoted 2 lines

You wouldn't be using 'soft' mounts would you?

yup, automounted with these options:

-rw,grpid,soft,intr,quota,nodev,nosuid,timeo=10,retrans=3

I suppose they could be hard.... I'll think about that.

Cheers,
Dan

Signature · 12 lines
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
[email protected], http://www.sdss.org
SDSS.  Mapping the Universe.

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#7 lore raw
Quoted 6 lines · Dan Yocum
Quoted 6 lines
Quoted 5 lines

This is an intermitent problem: When attempting to read a file (any file) on an NFS mounted volume, there appear to be "holes" in the file filled with ASCII NULL characters. Reading the file on the local machines shows no such holes, and other NFS clients don't see the holes, either. A simple umount/remount solves the problem.

Quoted 6 lines
Quoted 1 line · Neil Brown

You wouldn't be using 'soft' mounts would you?

yup, automounted with these options:

-rw,grpid,soft,intr,quota,nodev,nosuid,timeo=10,retrans=3

In that case let me ask you this: is your application checking the return code of the read()? or are you using mmap() instead of read()?

If read() returns zero-filled buffers without signalling an errors, it's a problem in the NFS client. That is, assumming it's not some bad interaction between XFS and the NFS server -- so try to clear up that possibility by mounting with the hard option.

If you're using mmap(), then I don't even know how the system should tell you about the problem... I suppose it would be sending a SIGSEGV or SIGBUS signal, since it's ultimately a failing page fault, but I haven't tested it.

Quoted 1 line

I suppose they could be hard.... I'll think about that.

Yes, they _should_ be hard if you care about your data integrity.

But that shouldn't stop us from fixing bugs in the 'soft' model, if they exist... which is why I'm asking you all these details. :-)

Ion

Signature · 10 lines
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
author
#8 lore raw

Ion,

The apps are apache and plain old 'less,' so I bet they just use read().

And the problem hasn't cropped up again (yet) and hadn't cropped up in ~1 year with older kernels (specifically 2.4.5). I just upgraded to 2.4.18 last Thursday when it did start.

I'll change the options to hard and see if that makes any changes.

Thanks,
Dan

Quoted 39 lines · Ion Badulescu
Quoted 6 lines · Dan Yocum
Quoted 6 lines
Quoted 5 lines

This is an intermitent problem: When attempting to read a file (any file) on an NFS mounted volume, there appear to be "holes" in the file filled with ASCII NULL characters. Reading the file on the local machines shows no such holes, and other NFS clients don't see the holes, either. A simple umount/remount solves the problem.

Quoted 6 lines
Quoted 1 line · Neil Brown

You wouldn't be using 'soft' mounts would you?

yup, automounted with these options:

-rw,grpid,soft,intr,quota,nodev,nosuid,timeo=10,retrans=3

In that case let me ask you this: is your application checking the return code of the read()? or are you using mmap() instead of read()?

If read() returns zero-filled buffers without signalling an errors, it's a problem in the NFS client. That is, assumming it's not some bad interaction between XFS and the NFS server -- so try to clear up that possibility by mounting with the hard option.

If you're using mmap(), then I don't even know how the system should tell you about the problem... I suppose it would be sending a SIGSEGV or SIGBUS signal, since it's ultimately a failing page fault, but I haven't tested it.

Quoted 1 line

I suppose they could be hard.... I'll think about that.

Yes, they _should_ be hard if you care about your data integrity.

But that shouldn't stop us from fixing bugs in the 'soft' model, if they exist... which is why I'm asking you all these details. :-)

Ion

--
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.

Signature · 12 lines
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
[email protected], http://www.sdss.org
SDSS.  Mapping the Universe.

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
author
#9 lore raw

Just an update - the null characters problem hasn't cropped up again, and I haven't done anything to "fix" the problem, either - the mounts are still 'soft', etc.

I'm about ready to declare it a non-reproducable network error and nothing to do with NFS, but if it turns up again, I'll let you know.

Thanks,
Dan

Quoted 39 lines · Ion Badulescu
Quoted 6 lines · Dan Yocum
Quoted 6 lines
Quoted 5 lines

This is an intermitent problem: When attempting to read a file (any file) on an NFS mounted volume, there appear to be "holes" in the file filled with ASCII NULL characters. Reading the file on the local machines shows no such holes, and other NFS clients don't see the holes, either. A simple umount/remount solves the problem.

Quoted 6 lines
Quoted 1 line · Neil Brown

You wouldn't be using 'soft' mounts would you?

yup, automounted with these options:

-rw,grpid,soft,intr,quota,nodev,nosuid,timeo=10,retrans=3

In that case let me ask you this: is your application checking the return code of the read()? or are you using mmap() instead of read()?

If read() returns zero-filled buffers without signalling an errors, it's a problem in the NFS client. That is, assumming it's not some bad interaction between XFS and the NFS server -- so try to clear up that possibility by mounting with the hard option.

If you're using mmap(), then I don't even know how the system should tell you about the problem... I suppose it would be sending a SIGSEGV or SIGBUS signal, since it's ultimately a failing page fault, but I haven't tested it.

Quoted 1 line

I suppose they could be hard.... I'll think about that.

Yes, they _should_ be hard if you care about your data integrity.

But that shouldn't stop us from fixing bugs in the 'soft' model, if they exist... which is why I'm asking you all these details. :-)

Ion

--
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.

Signature · 13 lines
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
[email protected], http://www.sdss.org
SDSS.  Mapping the Universe.

_______________________________________________________________

Don't miss the 2002 Sprint PCS Application Developer's Conference
August 25-28 in Las Vegas -- http://devcon.sprintpcs.com/adp/index.cfm

_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#10 lore raw
Quoted 6 lines · Dan Yocum

Just an update - the null characters problem hasn't cropped up again, and I haven't done anything to "fix" the problem, either - the mounts are still 'soft', etc.

I'm about ready to declare it a non-reproducable network error and nothing to do with NFS, but if it turns up again, I'll let you know.

With 'soft' mounts, it's not 'if', but 'when'...

Ion

Signature · 11 lines
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

_______________________________________________________________

Don't miss the 2002 Sprint PCS Application Developer's Conference
August 25-28 in Las Vegas -- http://devcon.sprintpcs.com/adp/index.cfm

_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs