Skip to contentSkip to conversation

nfs exporting my stacked fs

DISCUSSION Andrew Sharp 4 participants 12 messages May 28, 2003 – Jun 2, 2003 VFS

lore
Share & tools
lore
#0 lore raw

I could use some help getting my stacked (I guess that's the term d' jour) file system to be nfs exported/mounted. From reading the code, it seems that my file system either has to require a device, it doesn't, or it has to have some sort of FSID (the call to nfsctl_export() is failing), only I can't seem to figure out how to accomplish that. I tried to go through the mountd source, but came up empty. What did I miss?

TIA,

a

#1 lore raw
Quoted 6 lines · On Wednesday May 28, [email protected] wrote:

I could use some help getting my stacked (I guess that's the term d' jour) file system to be nfs exported/mounted. From reading the code, it seems that my file system either has to require a device, it doesn't, or it has to have some sort of FSID (the call to nfsctl_export() is failing), only I can't seem to figure out how to accomplish that. I tried to go through the mountd source, but came up empty. What did I miss?

If your filesystem isn't device based (which seems likely for a stacked filesystem), the nfsd cannot use the device number as a short, fixed length identifier for the filesystem, which it needs for storing in the filehandle. So, you need to provide one.
This is done in the /etc/exportss file with the fsid= option. e.g.

/my/file/system myclient(rw,no_root_squash,sync,fsid=27)

NeilBrown

#2 lore raw
Quoted 8 lines · Neil Brown
Quoted 6 lines · On Wednesday May 28, [email protected] wrote:

I could use some help getting my stacked (I guess that's the term d' jour) file system to be nfs exported/mounted. From reading the code, it seems that my file system either has to require a device, it doesn't, or it has to have some sort of FSID (the call to nfsctl_export() is failing), only I can't seem to figure out how to accomplish that. I tried to go through the mountd source, but came up empty. What did I miss?

Quoted 8 lines

If your filesystem isn't device based (which seems likely for a stacked filesystem), the nfsd cannot use the device number as a short, fixed length identifier for the filesystem, which it needs for storing in the filehandle. So, you need to provide one.
This is done in the /etc/exportss file with the fsid= option. e.g.

/my/file/system myclient(rw,no_root_squash,sync,fsid=27)

Thanks for that info, I was unaware of that option. Soooo, this brings up some other questions for me.

What is the relation between the fsid option mentioned above and the fsid returned by statfs op? Which I discovered shortly after sending my first email. Should they match? The i_dev is apparently used on device datasets for the filehandle, so do all datasets with the same i_dev somehow guarantee that i_dev/inode # 'filehandles' are somehow unique?

a

#3 lore raw
Quoted 7 lines · On Wednesday May 28, [email protected] wrote:
Quoted 2 lines · Neil Brown

/my/file/system myclient(rw,no_root_squash,sync,fsid=27)

Thanks for that info, I was unaware of that option. Soooo, this brings up some other questions for me.

Quoted 7 lines

What is the relation between the fsid option mentioned above and the fsid returned by statfs op? Which I discovered shortly after sending my first email. Should they match? The i_dev is apparently used on device datasets for the filehandle, so do all datasets with the same i_dev somehow guarantee that i_dev/inode # 'filehandles' are somehow unique?

a

What should the 'stat' fsid be ? Good question. The NFS protocol specs don't really say much about it.
In NFSv3, the spec explicitly says it must always be the same, so the client is free to ignore it. However the linux client doesn't ignore it but uses it as part of it's handle to identify an object (I think). So I guess we should really use the fsid..

I'm not sure what you mean when you are talking about datasets...

An inode number is provided by a filesystem, and each device can only have one filesystem mounted, so as long as the filesystem ensures inode numbers are unique, then dev+inode will always be unique.

NeilBrown

#4 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Neil Brown <[email protected]> writes:

> is free to ignore it. However the linux client doesn't ignore > it but uses it as part of it's handle to identify an object (I > think). So I guess we should really use the fsid..

Note: that behaviour has changed in recent kernels. We no longer look at the fsid.

However we will have to reintroduce it for NFSv4. Unlike earlier versions of NFS, v4 allows us to cross mountpoints on the server. When this happens, the client is supposed to detect the crossing by means of the fsid, and then to automount the new partition.
This behaviour would also make sense under NFSv3 for 'nohide'...

Cheers,
Trond

#5 lore raw
In reply to Neil Brown
Quoted 7 lines

An inode number is provided by a filesystem, and each device can only have one filesystem mounted, so as long as the filesystem ensures inode numbers are unique, then dev+inode will always be unique.

This is true in any instant, but for most NFS purposes, we also care about identifying a file uniquely over time, even when that time might involve sea changes such as a reboot. E.g. one would like to make sure that a file handle that referred to file A an hour ago still refers to file A now. Or at least it doesn't refer to file B. So it is worth mentioning here the (often discussed) ways in which a single dev+inode might refer to two different files.

If you delete file A and create file B, A might have the same inode number as B. Inode generation numbers are sometimes used to disambiguate. So are stronger definitions of "filesystem ensures inode numbers are unique." If you delete the entire filesystem and create a new one on the same device, dev+inode+inode_generation may refer to file A today and file B tomorrow. If you change the device number of the device (all too easy in Linux, especially with a reboot), same thing.

The fundamental idea of the fsid (filesystem ID) is that it uniquely identifies a filesystem. Putting a new filesystem onto a disk generates a new fsid. Changing the device number (or moving the filesystem to another disk) does not change the fsid.

I think we've discovered recently that the fundamental idea of the fsid is inadequate for NFS purposes, though, and that an export ID serves our purpose better.

(btw - I know Neil knows all this well -- my response to his post is meant to pick up where that post left off, not contradict it).

--
Bryan Henderson IBM Almaden Research Center
San Jose CA Filesystems

#6 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Bryan Henderson <[email protected]> writes:

> This is true in any instant, but for most NFS purposes, we also > care about identifying a file uniquely over time, even when > that time might involve sea changes such as a reboot. E.g. one > would like to make sure that a file handle that referred to > file A an hour ago still refers to file A now.

According to RFC1813 (which clarifies common practices for RFC1094) file handles are *unique*. More specifically:

"If two file handles from the same server are equal, they must refer to the same file, but if they are not equal, no conclusions can be drawn."

RFC1813 makes no exceptions to this rule. Including for time....

The one thing that is allowed, is that the server may revoke a filehandle if (and only if) the file no longer exists, or if access to the file has been revoked.

IOW: the condition you describe above is trivially observed...

Cheers,
Trond

#7 lore raw
Quoted 8 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Bryan Henderson <[email protected]> writes:

> This is true in any instant, but for most NFS purposes, we also > care about identifying a file uniquely over time, even when > that time might involve sea changes such as a reboot. E.g. one > would like to make sure that a file handle that referred to > file A an hour ago still refers to file A now.

Quoted 8 lines

According to RFC1813 (which clarifies common practices for RFC1094) file handles are *unique*. More specifically:

"If two file handles from the same server are equal, they must refer to the same file, but if they are not equal, no conclusions can be drawn."

RFC1813 makes no exceptions to this rule. Including for time....

Yes. At first I was annoyed that the Linux NFS client peeks at the file handle, because sometimes I want to change it for the same file, but then it turns out that the solution to another problem of mine fixes that as well. And now it sounds like the v4 file handle won't be considered opaque for the client any more. As far as being able to find the right file from a file handle, no matter how dusty, my fs has the same problem as NFS in this regard. A bit of challenge on Linux if one hopes to maintain a proper dcache.

Quoted 6 lines

The one thing that is allowed, is that the server may revoke a filehandle if (and only if) the file no longer exists, or if access to the file has been revoked.

IOW: the condition you describe above is trivially observed...

I take it then that the fsid returned by ->statfs is largely not used right now? My version of nfs-utils (1.0) doesn't support the fsid option in config files, so I've just hacked the kernel to get me through this part of the development phase.

I'm guessing that all or part of this stuff is handled a bit differently in 2.5?

a

#8 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Andrew Sharp <[email protected]> writes:

> Yes. At first I was annoyed that the Linux NFS client peeks at > the file handle, because sometimes I want to change it for the > same file, but then it turns out that the solution to another > problem of mine fixes that as well. And now it sounds like the > v4 file handle won't be considered opaque for the client any > more.

All v2/v3/v4 file handles are opaque as far as the client is concerned. The only issue is that of uniqueness. Whereas NFSv2/v3 do not support the concept of a filehandle that is not unique over time, NFSv4 does (in theory) by means of the concept of "volatile filehandles".

Note though that NFSv4 volatile filehandles have a number of unsolved race conditions associated to them. I would recommend that people stay away from them if at all possible.

> I take it then that the fsid returned by ->statfs is largely > not used right now? My version of nfs-utils (1.0) doesn't > support the fsid option in config files, so I've just hacked > the kernel to get me through this part of the development > phase.

The fsid is unused for NFSv2/v3. It is very important for NFSv4, as the latter allows the server to cross mount points (rather like the somewhat ill-defined "nohide" option did for NFSv2/v3).

In NFSv4, the client is expected to automount the new partition if we cross a mount point on the server. Crossing a mount point is detected by means of the fsid change...

> I'm guessing that all or part of this stuff is handled a bit > differently in 2.5?

Not yet. However it will be once we fix the NFSv4 behaviour.

Cheers,
Trond

#9 lore raw
In reply to Trond Myklebust
Quoted 11 lines
Quoted 9 lines

This is true in any instant, but for most NFS purposes, we also care about identifying a file uniquely over time, even when that time might involve sea changes such as a reboot. E.g. one would like to make sure that a file handle that referred to file A an hour ago still refers to file A now.

...
the condition you describe above is trivially observed...

I don't exactly follow the language here, but from context I think you're
saying: RFC 1813 requires file handles to be absolutely unique, which is a condition that I said one "would like."

That's interesting (thanks for the info), but not as interesting as whether the file handles in any real world NFS implementation are so unique. And of course they aren't. Because they don't incorporate ideal fsids. In a typical NFS 2/3 implementation, a device number fills the role of fsid. I think we can agree that absent some simplifying assumptions, that means two equal file handles from a server could refer to two different files.

I think we're all still looking for a practical implementation that approaches the ideals of the RFC.

--
Bryan Henderson IBM Almaden Research Center
San Jose Filesystems

#10 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Bryan Henderson <[email protected]> writes:

> In a typical NFS 2/3 implementation, a device number fills the > role of fsid. I think we can agree that absent some > simplifying assumptions, that means two equal file handles from > a server could refer to two different files.

Then server admins are deliberately putting themselves in a situation where their users may see data corruption. The above problem is trivially solved by means of the 'fsid' export option.

For most setups, this should not be necessary (since device numbers will suffice). Those that don't have permanently allocated device numbers (e.g. fibre channel), should use the fsid export option in order to provide a permanent value...

> I think we're all still looking for a practical implementation > that approaches the ideals of the RFC.

I hope not.

Cheers,
Trond

#11 lore raw
Quoted 7 lines

Then server admins are deliberately putting themselves in a situation where their users may see data corruption. The above problem is trivially solved by means of the 'fsid' export option.

Agreed, but note that you're talking specifically about a very recent Linux system with a careful system administrator. I was talking about the world of NFS in general -- the practical world.

Quoted 4 lines

For most setups, this should not be necessary (since device numbers will suffice). Those that don't have permanently allocated device numbers (e.g. fibre channel), should use the fsid export option in order to provide a permanent value...

But now you're talking about sufficiency, whereas I was talking about actually implementing unique fsids. Device number is clearly sufficient, since people have been using it for decades. But it does not come close to being a real fsid. No device numbers in Linux (on architectures I know about) are permanently associated with physical devices. Lots of things, including recabling, can change them. Moreover, filesystems are not permanently associated with physical devices. You can move a filesystem from one device to another.

The fsid export option, while a leap ahead of what we had before, is clearly not the ultimate solution to the fsid problem, because it requires too much human participation. System administrators don't normally have to administrate at that level. It's like choosing device addresses and IRQ levels when you install network cards.

--
Bryan Henderson IBM Almaden Research Center
San Jose CA Filesystems