Skip to contentSkip to conversation

nfs is stupid ("getfh failed")

DISCUSSION Michael Rothwell 8 participants 12 messages Sep 6, 2001 – Sep 12, 2001

lore
Share & tools
lore
#0 lore raw

Two systems that worked fine for weeks, both running 2.4.[7,8] kernels. The server is running 2.4.8 and exporting a reiserfs filesystem via nfs. Or it was, anyway. The server was shut down and brought back up (power failure). The client was then
rebooted.

server# cat /etc/exports
/export 192.168.1.*(rw,no_root_squash)
/export/home 192.168.1.*(rw,no_root_squash)

client# mount /export
mount: 192.168.1.1:/export failed, reason given by server: Permission denied

server# tail /var/log/messages

Sep  6 09:37:43 gateway rpc.mountd: authenticated mount request from
192.168.1.133:933 for /export (/export)
Sep  6 09:37:43 gateway rpc.mountd: getfh failed: Operation not permitted

... so, rebooting two working systems seems to kill NFS. Any ideas why?

On a related topic, will Linux ever have a better file-service protocol?

#1 lore raw

From: "Neil Brown" <[email protected]>

Quoted 1 line

This is not allowed, and makes no sense.

It apparently is (or was, anyway) allowed, because it worked until the server was rebooted. Fluke?
Are you saying that, if I export "/export", I can mount "/export/home" from a client machine? That's nice.

Quoted 1 line

Simply remove the second line and your problems should go away.

Thanks. I actually switched to two explicit exports; /export/files and /export/home, which works.

Quoted 4 lines
Quoted 1 line

On a related topic, will Linux ever have a better file-service protocol?

This is an unanswerable question. The future of Linux is completely undefined until it happens.

Just wondering if there's been any talk, plans, etc. of an alternative for NFS.

Quoted 1 line

What exactly do you mean by "better" anyway?

Better security, better performance.

Thanks,

-M

#2 lore raw
In reply to Michael Rothwell
Quoted 4 lines · Michael Rothwell

server# tail /var/log/messages

Sep  6 09:37:43 gateway rpc.mountd: authenticated mount request from
192.168.1.133:933 for /export (/export)
Sep  6 09:37:43 gateway rpc.mountd: getfh failed: Operation not permitted

I'm seeing this message quite often with one Linux 2.4.7 system automounting another. As long as A has B's filesystem mounted, all is ok. Then A times out, unmounts, and later wants to remount B's filesystem. Then, sometimes, I see a message much like yours.

It doesn't seem to need a reboot to cause this problem, and the fix I have found is to kill and restart the NFS server: /etc/init.d/nfs restart.

I have no idea why it happens, or why restarting nfsd or mountd fixes it.

-- Jamie

#3 lore raw
In reply to Michael Rothwell
Quoted 9 lines · Michael Rothwell
Quoted 9 lines

From: "Neil Brown" <[email protected]>

> This is not allowed, and makes no sense.

It apparently is (or was, anyway) allowed, because it worked until the server was rebooted. Fluke?
Are you saying that, if I export "/export", I can mount "/export/home" from a client machine? That's nice.

Yes. At least that's how Solaris, Irix, HP/UX, and several others do it. The general rule is something along the lines of "if /a is exported, anything that is a subdirectory of /a is accessible via NFS, so long as it is on the same device (partition - actually major/minor pair)". So, if /export/home in your case lives in the same file system as /export, it is accessible via NFS as well. Now if /export/home is it's own mount point (i.e. a separate file system), then it needs to be exported explicitly.

Quoted 5 lines
Quoted 5 lines

> Simply remove the second line and your problems should go away.

Thanks. I actually switched to two explicit exports; /export/files and /export/home, which works.

This works as well, with the caveat that you cannot then mount /export itself on client machines.

tw

Signature · 1 line
[email protected]
#5 lore raw
Quoted 6 lines
Quoted 2 lines · Michael Rothwell

Just wondering if there's been any talk, plans, etc. of an alternative for NFS.

there's coda.

You probably want inter-mezzo rather than coda for that kind of file service. Coda is a research project, inter-mezzo is kind of "distilled coda and afs", making it a real fs. http://www.inter-mezzo.org

#6 lore raw
In reply to Johan Kullstam
Quoted 10 lines · Alan Cox
Quoted 6 lines
Quoted 2 lines · Michael Rothwell

Just wondering if there's been any talk, plans, etc. of an alternative for NFS.

there's coda.

You probably want inter-mezzo rather than coda for that kind of file service. Coda is a research project, inter-mezzo is kind of "distilled coda and afs", making it a real fs. http://www.inter-mezzo.org

And it currently looks "Linux only", which kind of defeats IMHO the purpose.

In a heterogenous environment there is NFS (and SMB aka CIFS aka <name of the day>, of course. The horrors! The horrors! :-) ) and nothing much else.

Regards
Henning

Signature · 5 lines
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     [email protected]

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   [email protected]
D-91054 Buckenhof     Fax.: 09131 / 50654-20   
#7 lore raw
Quoted 2 lines · Henning P. Schmiedehausen

And it currently looks "Linux only", which kind of defeats IMHO the purpose.

Considering I use only Linux at home, that's fine with me. :) For Windows interaction, I use samba. I have not even booted my Sparc into Solaris in ... oh, about a year.

Is intermezzo usable? Reliable? Any horror stories? Success stories?

-M

#8 lore raw
In reply to Michael Rothwell
Quoted 3 lines · [email protected] (Neil Brown) writes:

SUNs "cachefs" concept can be used to improve read performance by caching a lot more of server-data on the client. That could be implemented for Linux, but I don't know of anyone with serious plans.

cachefs sucks. It doesn't seem to cache stat(2) information. Doing ls -F in a ~100-entries directory takes several seconds over a link with 50ms round-trip time.

//Marcus

Signature · 4 lines
---------------------------------+---------------------------------
         Marcus Sundberg         |      Phone: +46 707 452062
   Embedded Systems Consultant   |     Email: [email protected]
        Cendio Systems AB        |      http://www.cendio.com
#9 lore raw
In reply to Michael Rothwell
Quoted 9 lines · Neil Brown
Quoted 9 lines

I'm seeing this message quite often with one Linux 2.4.7 system automounting another. As long as A has B's filesystem mounted, all is ok. Then A times out, unmounts, and later wants to remount B's filesystem. Then, sometimes, I see a message much like yours.

It doesn't seem to need a reboot to cause this problem, and the fix I have found is to kill and restart the NFS server: /etc/init.d/nfs restart.

Quoted 8 lines
Quoted 1 line

I have no idea why it happens, or why restarting nfsd or mountd fixes it.

Show me your /etc/exports....

If you export a directory and a subdirectory of that directory - both on the same filesysem, you can get this.
If you export a directory to both an IP address (or subnet) and a hostname (or wildcard or netgroup) this can also happen.

Yes, I'm doing the second of those. No alternative -- I need write access from some hosts, and read access to all the rest (who are dynamically allocated) on the subnet.

It's clearly a bug in the NFS server then.

-- Jamie

/etc/exports:

# I want ro granted to all hosts in the 172.30.* subnet, but rw

# granted to the machines in @aquarius_outside which is within that
# subnet.  Doesn't work, whichever order I write the lines.
# (The machines in @aquarius_hosts always get read only access).
#
# Solution, though I don't like it: write the host names explicitly.
#
# The 192.168.64.192/19 subnetwork is for temporarily assigned IPs on
# the Aquarius test network.  /kickstart is used for kickstart
# installations on machines plugged in temporarily.
/home		172.30.0.0/16(ro,root_squash) \
		galatea.degree2.com(rw,root_squash) \
		ariel.degree2.com(rw,root_squash) \
		@aquarius_hosts(rw,root_squash) \
		@aquarius_outside(rw,root_squash)
/kickstart	172.30.0.0/16(ro,root_squash) \
		@aquarius_hosts(ro,root_squash) \
		@aquarius_outside(ro,root_squash) \
		192.168.64.192/255.255.255.224(ro,root_squash)
#10 lore raw
In reply to Michael Rothwell
Quoted 8 lines · [email protected] (Neil Brown) writes:
Quoted 3 lines · On September 10, [email protected] wrote:

cachefs sucks. It doesn't seem to cache stat(2) information. Doing ls -F in a ~100-entries directory takes several seconds over a link with 50ms round-trip time.

Well, I said "concept" not "implementation", but I suspect that Solaris cachefs does cache stat information. Maybe you just need to increase the timeouts for the attribute cache.

Considering that I did several ls'es on the order of milliseconds apart I doubt that would help...

//Marcus

Signature · 4 lines
---------------------------------+---------------------------------
         Marcus Sundberg         |      Phone: +46 707 452062
   Embedded Systems Consultant   |     Email: [email protected]
        Cendio Systems AB        |      http://www.cendio.com
#11 lore raw
In reply to Michael Rothwell
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Marcus Sundberg <[email protected]> writes:

> [email protected] (Neil Brown) writes:
>> On September 10, [email protected] wrote:
>> > cachefs sucks. It doesn't seem to cache stat(2) information. >> > Doing ls -F in a ~100-entries directory takes several seconds >> > over a link with 50ms round-trip time.
>>
>> Well, I said "concept" not "implementation", but I suspect that >> Solaris cachefs does cache stat information. Maybe you just >> need to increase the timeouts for the attribute cache.

> Considering that I did several ls'es on the order of > milliseconds apart I doubt that would help...

The NFS close-to-open cache consistency requirement forces them to compare the attribute cache to the server every time someone does a call to open(). This is true whether or not one uses cachefs.

After the file has actually been opened, you can call fstat() as many times as you like. The cached attributes to be used, and they will then be checked on the server only after the cache times out.

Cheers,
Trond