fsid question · kernelthread

Jason Keltz

author

2007-03-26 20:29 UTC #0 lore raw

Hi.

I'm running a RedHat Enterprise 4 system.

I have a question about the fsid option in /etc/exports that I was hoping someone might be able to help with.

I am working with LVM under Linux. If I export LVM volumes via NFS, the fsid is based on major/minor number. If I then remove a logical volume on the NFS server and reboot the server, the major and minor numbers for the other volumes are adjusted, which of course causes problems for the existing NFS exports (which either end up stale/pointing to a different
logical volume)! The fsid option would fix this by allowing me to
specify a number independent of major/minor number.

A couple of questions:

1) How do I find out the fsid of an existing export so that I can "hard-code" it in the /etc/exports file?
2) For filesystems that are not mounted, how do I choose a number that the system will not try to allocate to a different nfs export that isn't using the fsid option? Will the system check existing fsids, or it just allocated the fsid based on a formula?
3) What is the best way for me to generate my own fsid?

Any assistance would be appreciated...

Jason.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Wendy Cheng

2007-03-26 21:20 UTC #1 lore raw

Quoted 9 lines · Jason Keltz

Hi.
I'm running a RedHat Enterprise 4 system.
I have a question about the fsid option in /etc/exports that I was hoping someone might be able to help with.
I am working with LVM under Linux. If I export LVM volumes via NFS, the fsid is based on major/minor number.

This statement needs correction. An NFS fsid is not based on major/minor numbers - it is is a 32 bit number that admin (root) can arbitrary choose and use to uniquely identify an NFS export. More on this in the following comment.

Quoted 7 lines

If I then remove a logical volume
on the NFS server and reboot the server, the major and minor numbers for the other volumes are adjusted, which of course causes problems for the existing NFS exports (which either end up stale/pointing to a different
logical volume)! The fsid option would fix this by allowing me to
specify a number independent of major/minor number.

This is correct (fsid option would allow admin to identify an NFS export, independent of major/minor number).

Quoted 5 lines

A couple of questions:
1) How do I find out the fsid of an existing export so that I can "hard-code" it in the /etc/exports file?

The admin (root, or whoever has access rights to /etc/exports file) can pick one number between 0 and 0xffffffff and add it into /etc/exports. It comes and goes with each "exportfs" command. This id is used to construct NFS filehandle that will be sent to NFS client upon requests. NFS client uses file handle to communicate with NFS server on which file they want to have access. The fsid, if specified, is part of the file handle that uniquely identifies an export (one entry in /etc/exports file).

So as rule of thumb, you would not want to change fsid as long as there are NFS clients still out there trying to access the files on the server. It is, however, not a permanent ID since there is no filesystem "on-disk" record of it.

Quoted 5 lines

2) For filesystems that are not mounted, how do I choose a number that the system will not try to allocate to a different nfs export that isn't using the fsid option? Will the system check existing fsids, or it just allocated the fsid based on a formula?

No, NFS fsid is *not* allocated (nor generated) by system. There is no co-relation between the physical filesystem and NFS fsid, if the filesystem is not NFS exported. Note that a filesystem can be mounted but not NFS exported - in this case, the filesystem will not recognize the NFS fsid. I suspect that you confuse fsid with either filehandle or some specific filesystem implementation fsid (few cluster filesystems I know of have its own fsid - but it differs from the "NFS" fsid we discuss here).

Quoted 2 lines

3) What is the best way for me to generate my own fsid?

It is a number picked by admin and must be between 0 and 0xFFFFFFFF.

Hopefully this makes things clear (since our cluster suites use NFS fsid extensively).

-- Wendy

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Jason Keltz

author

2007-03-27 12:50 UTC #2 lore raw

Hi Wendy,

Thanks for your response..

A few more comments...

Quoted 12 lines · Wendy Cheng

Quoted 3 lines · Jason Keltz
1) How do I find out the fsid of an existing export so that I can "hard-code" it in the /etc/exports file?
The admin (root, or whoever has access rights to /etc/exports file) can pick one number between 0 and 0xffffffff and add it into /etc/exports. It comes and goes with each "exportfs" command. This id is used to construct NFS filehandle that will be sent to NFS client upon requests. NFS client uses file handle to communicate with NFS server on which file they want to have access. The fsid, if specified, is part of the file handle that uniquely identifies an export (one entry in /etc/exports file).

Quoted 4 lines

So as rule of thumb, you would not want to change fsid as long as there are NFS clients still out there trying to access the files on the server. It is, however, not a permanent ID since there is no filesystem "on-disk" record of it.

The problem is that right now, I have several hundred clients who have mounted the exported filesystems via NFS where I did not specify an
explicit fsid in the NFS sever /etc/exports file. In order to start
using the fsid option, I need to know the existing fsids so that I can hard-code those into /etc/exports. This way, if the server is rebooted, the clients will not be affected. I could then choose fsids for NEW nfs exported filesystems and hard-code those fsids into /etc/exports. Is there a Linux command, or an entry in /proc on the file server that will let me know the existing fsids?

Quoted 4 lines

Quoted 2 lines
3) What is the best way for me to generate my own fsid?
It is a number picked by admin and must be between 0 and 0xFFFFFFFF.

Is the fsid specified in hex?

Since the system will allocate its own fsid for NFS exported filesystems that do not have a user-specified fsid, how do I choose fsids for my user-specified fsid exports that will not conflict with those that the system will use?

Thanks,

Jason.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

James Pearson

2007-03-27 12:56 UTC #3 lore raw

Quoted 28 lines · Jason Keltz

Hi Wendy,
Thanks for your response..
A few more comments...
Quoted 14 lines · Wendy Cheng
Quoted 3 lines · Jason Keltz
1) How do I find out the fsid of an existing export so that I can "hard-code" it in the /etc/exports file?
The admin (root, or whoever has access rights to /etc/exports file) can pick one number between 0 and 0xffffffff and add it into /etc/exports. It comes and goes with each "exportfs" command. This id is used to construct NFS filehandle that will be sent to NFS client upon requests. NFS client uses file handle to communicate with NFS server on which file they want to have access. The fsid, if specified, is part of the file handle that uniquely identifies an export (one entry in /etc/exports file).
Quoted 4 lines
So as rule of thumb, you would not want to change fsid as long as there are NFS clients still out there trying to access the files on the server. It is, however, not a permanent ID since there is no filesystem "on-disk" record of it.

Quoted 9 lines

The problem is that right now, I have several hundred clients who have mounted the exported filesystems via NFS where I did not specify an
explicit fsid in the NFS sever /etc/exports file. In order to start
using the fsid option, I need to know the existing fsids so that I can hard-code those into /etc/exports. This way, if the server is rebooted, the clients will not be affected. I could then choose fsids for NEW nfs exported filesystems and hard-code those fsids into /etc/exports. Is there a Linux command, or an entry in /proc on the file server that will let me know the existing fsids?

I believe you can add a fsid to /etc/exports and re-export the file systems - existing mounts will not be affected. New mounts will pick up the new fsid. The existing mounts will also pick up the new fsid the next time they remount the file system.

I've certainly done this in the past and had no problems.

James Pearson

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Jason Keltz

author

2007-03-27 13:29 UTC #4 lore raw

Quoted 11 lines · James Pearson

Quoted 9 lines · Jason Keltz
The problem is that right now, I have several hundred clients who have mounted the exported filesystems via NFS where I did not specify an
explicit fsid in the NFS sever /etc/exports file. In order to start
using the fsid option, I need to know the existing fsids so that I can hard-code those into /etc/exports. This way, if the server is rebooted, the clients will not be affected. I could then choose fsids for NEW nfs exported filesystems and hard-code those fsids into /etc/exports. Is there a Linux command, or an entry in /proc on the file server that will let me know the existing fsids?

Quoted 8 lines

I believe you can add a fsid to /etc/exports and re-export the file systems - existing mounts will not be affected. New mounts will pick up the new fsid. The existing mounts will also pick up the new fsid the next time they remount the file system.
I've certainly done this in the past and had no problems.
James Pearson

Hi James.

Thanks for your message.

I'm not sure that I understand this though. If the system is generating an fsid for NFS exports in the absence of a user-specified fsid, then changing the fsid and re-exporting the filesystem *should* essentially confuse the client? How does one choose an fsid that is not already in use on the system? I imagine there must be some way to list the currently used fsids for exported filesystems.

Jas.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

James Pearson

2007-03-27 14:27 UTC #5 lore raw

Quoted 10 lines · Jason Keltz

Hi James.
Thanks for your message.
I'm not sure that I understand this though. If the system is generating an fsid for NFS exports in the absence of a user-specified fsid, then changing the fsid and re-exporting the filesystem *should* essentially confuse the client? How does one choose an fsid that is not already in use on the system? I imagine there must be some way to list the currently used fsids for exported filesystems.

See the thread starting at:

<http://marc.info/?l=linux-nfs&m=110295083427898&w=2>

James Pearson

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Jason Keltz

author

2007-03-27 14:57 UTC #6 lore raw

Quoted 13 lines · James Pearson

Quoted 10 lines · Jason Keltz
Hi James.
Thanks for your message.
I'm not sure that I understand this though. If the system is generating an fsid for NFS exports in the absence of a user-specified fsid, then changing the fsid and re-exporting the filesystem *should* essentially confuse the client? How does one choose an fsid that is not already in use on the system? I imagine there must be some way to list the currently used fsids for exported filesystems.

Quoted 3 lines

See the thread starting at:
<http://marc.info/?l=linux-nfs&m=110295083427898&w=2>

That makes sense, but what is still not clear to me is why I can't have the system show me the existing fsid that it is using in the absence of a user-specified fsid.

Jason.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

J. Bruce Fields

2007-03-27 15:05 UTC #7 lore raw

Quoted 3 lines · Jason Keltz

That makes sense, but what is still not clear to me is why I can't have the system show me the existing fsid that it is using in the absence of a user-specified fsid.

There *is* no "existing fsid". The fsid that you provide in the "fsid=" export option is totally different from what's used by default to generate filehandles for that filesystem.

--b.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Wendy Cheng

2007-03-27 15:12 UTC #8 lore raw

In reply to Jason Keltz

Quoted 7 lines · Jason Keltz

.........................................If the system is generating an fsid for NFS exports in the absence of a user-specified fsid, then changing the fsid and re-exporting the filesystem *should* essentially confuse the client? How does one choose an fsid that is not already in use on the system? I imagine there must be some way to list the currently used fsids for exported filesystems.

I think I see where the confusion comes from. To understand this, let's start with a thing called NFS "file handle".

When an NFS client request file access, server packages few info into a max 64 bytes identifier called "file handle" that is sent back to client. Depending on the version of Linux OS you have, it may consist of file inode number, directory inode number, export device major and minor number, or the admin-specified "fsid", etc. If the /etc/exports doesn't specify an "fsid", then the major and minor numbers of the exported device are used. Each file handle has a field called "fsid-type" to tell whether this file handle uses admin-specified "fsid" or device major-minor number.

In your case, the current FileHandle sent to NFS clients uses device major and minor numbers. You can safely pick *any* number you like as the fsid and re-exports the entries (say by doing an "exportfs -a" command). Any *new* NFS client requests will get a new FileHandle that uses the specified "fsid". However, the *existing" FileHandle(s) any NFS client still keeps and uses will have the "old" way - that is the device major-minor pair. The server knows how to interpret them (based on the type specified in the "fsid-type" field of the file handle).

This implies the following:

1. If server for whatever reason gets rebooted and somehow the device major-minor number gets altered, the client that keeps the "old" file handle will get a "stale file handle" error. Some of the NFS clients will re-do lookup to obtain the correct FileHandle(s). Some of them will simply fail. There is really not much you can do as a system administrator.
2. You want to keep the "fsid" same (and unique) for obvious reasons after you export them.
3. Any 32 bit integer can be used (say, 1, 2, 3, etc - no need to use hex number).

-- Wendy

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

J. Bruce Fields

2007-03-27 15:23 UTC #9 lore raw

Quoted 3 lines · Wendy Cheng

1. If server for whatever reason gets rebooted and somehow the device major-minor number gets altered, the client that keeps the "old" file handle will get a "stale file handle" error.

Actually, have you checked this? From a quick look at

linux/fs/nfsd/nfsfh.c:fh_verify()
and

nfs-utils/utils/mountd/cache.c:nfsd_fh()

it looks to me like nfsd should still allow both the old and new filehandles.

--b.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Trond Myklebust

2007-03-27 15:32 UTC #10 lore raw

Quoted 7 lines · J. Bruce Fields

Quoted 3 lines · Wendy Cheng
1. If server for whatever reason gets rebooted and somehow the device major-minor number gets altered, the client that keeps the "old" file handle will get a "stale file handle" error.
Actually, have you checked this? From a quick look at

Quoted 7 lines

linux/fs/nfsd/nfsfh.c:fh_verify()
and
nfs-utils/utils/mountd/cache.c:nfsd_fh()
it looks to me like nfsd should still allow both the old and new filehandles.

That is correct. A server kernel upgrade should not normally result in a client error. See also Neil's recently posted patches that replace the fileid with the disk UUID. Those again attempt to preserve backward compatibility for filehandles.

Cheers,
Trond

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Roger Heflin

2007-03-27 15:31 UTC #11 lore raw

In reply to Wendy Cheng

Quoted 5 lines · Wendy Cheng

1. If server for whatever reason gets rebooted and somehow the device major-minor number gets altered, the client that keeps the "old" file handle will get a "stale file handle" error. Some of the NFS clients will re-do lookup to obtain the correct FileHandle(s). Some of them will simply fail. There is really not much you can do as a system administrator.

I think this can also happen if you are using the major+minor numbers with say lvm, and change the order of the underlying LVM's, this might even be a risk with the fsid option if the underlying filesystem that it was supposed to be exporting fails to get mounted before the export.

Server reboots, a different device is put at the same major and minor numbers
than was there before,now the server now gets hammered with inode numbers that don't exist (valid range for filesystem, but don't exist on the "new" filesystem that is at the same major and minor numbers as the old one was).

I have seen this actually completely use up all of the CPU on a NFS server, eventually causing it to be completely unusable, and of course the issue survives
reboots of the server as the requests are coming from the clients, and the clients don't appear to ever give up, and if you have enough clients you
can make the server unusable fairly quickly. I have seen the machine
actually hang bringing up NFS from this.

Roger

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Chris Osicki

2007-03-29 13:58 UTC #12 lore raw

In reply to Wendy Cheng

On Mon, 26 Mar 2007 17:00:36 -0400
Wendy Cheng <[email protected]> wrote:

Hi

Quoted 8 lines

Quoted 7 lines · Jason Keltz
Hi.
I'm running a RedHat Enterprise 4 system.
I have a question about the fsid option in /etc/exports that I was hoping someone might be able to help with.

Quoted 6 lines

Quoted 2 lines
I am working with LVM under Linux. If I export LVM volumes via NFS, the fsid is based on major/minor number.
This statement needs correction. An NFS fsid is not based on major/minor numbers - it is is a 32 bit number that admin (root) can arbitrary choose and use to uniquely identify an NFS export. More on this in the following comment.

And this statement needs correction too ;-)
fsid _is_ based on major/minor number. I learned it the hard way, seeing two filesystem swapped on NFS clients because of minor number change after switching a cluster package to another node.

Then I read carefully fsid section in man 5 exports.

Regards,
Chris

[snip]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Wendy Cheng

2007-03-29 14:43 UTC #13 lore raw

Quoted 24 lines · Chris Osicki

On Mon, 26 Mar 2007 17:00:36 -0400
Wendy Cheng <[email protected]> wrote:
Hi
Quoted 9 lines
Quoted 7 lines · Jason Keltz
Hi.
I'm running a RedHat Enterprise 4 system.
I have a question about the fsid option in /etc/exports that I was hoping someone might be able to help with.
Quoted 8 lines
Quoted 3 lines
I am working with LVM under Linux. If I export LVM volumes via NFS, the fsid is based on major/minor number.
This statement needs correction. An NFS fsid is not based on major/minor numbers - it is is a 32 bit number that admin (root) can arbitrary choose and use to uniquely identify an NFS export. More on this in the following comment.

Quoted 8 lines

And this statement needs correction too ;-)
fsid _is_ based on major/minor number. I learned it the hard way, seeing two filesystem swapped on NFS clients because of minor number change after switching a cluster package to another node.
Then I read carefully fsid section in man 5 exports.

Well, I still don't see anywhere there is a documented definition that says "(NFS) fsid _is_ based on major/minor number". The man page describes it as "the filesystem identification portion of the file handle and file attributes used on the wire". My opinion is that an NFS "fsid" is the number you specify in the export file - nothing more.

On the other hand, arguments like these are kind of silly. The important thing is to advocate and make sure people know how to use "fsid" in a clustered environment.

-- Wendy

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs