FS-specified FSID for non-device based filesystems?
Hello!
Right now NFSD depends on FS_REQUIRES_DEV to be set in sb->s_type->fs_flags or manually set fsid= in /etc/exports in order to export filesystem. But there are non-device based filesystems (Lustre for example ;) ) that would like to be exported via NFS without guiding users through this fsid= business. Of course it is easy to cheat through by setting FS_REQUIRES_DEV, but this does not solve entire problem. While allowing for single-node export of filesystem to work ok, more generic problem of exporting same filesystem from several NFS servers in clustered manner won't work because sb->s_dev might be different on different nodes. This is not only Lustre problem, I believe same problem will happen with GFS or OCFS2 or whatever other SAN-based fs if different nodes have different device names (/dev/sda | /dev/sdb) for same SAN device, in such a case fsid calculated would be different on different nodes too.
I wonder if e.g. another export op could be introduced to ask filesystem for its unique id (if supported) and use that instead of sb->s_dev in fsid calculations. Does this look sane?
I hope I am not missing anything?
Or are there better ideas?
Thanks.
Bye,
Oleg
Quoted 1 line
Right now NFSD depends on
FS_REQUIRES_DEVto be set in
sb->s_type->fs_flags
Quoted 7 lines
or manually set fsid= in
/etc/exportsin order to export filesystem. But there are non-device based filesystems (Lustre for example ;) ) that would like to be exported via NFS without guiding users through this fsid= business.
...
I wonder if e.g. another export op could be introduced to ask filesystem for its unique id
It would also be quite valuable with device-based filesystems, for the opposite reason. That a device-based filesystem can be uniquely identified by the device number of its device is a fiction that is getting harder to entertain every year. Device numbers of devices change and a device can contain multiple filesystems over time.
When Linux added the export ID, at a time when other operating systems were addressing the problem with the kind of fsid you're proposing, I believe the point was that in Linux you can have multiple incompatible exports of the same filesystem -- to give a client the right data, you'd need to know from which export, not just which filesystem, a given filehandle came.
Nonetheless, I agree that it would be good to have the option of using a filesystem identifier instead of an export identifier. In a system that only does simple uniform exports of whole filesystems, I think filesystem ID works perfectly, and it is much easier to use.
--
Bryan Henderson IBM Almaden Research Center
San Jose CA Filesystems
Hello!
Quoted 6 lines · Bryan Henderson
When Linux added the export ID, at a time when other operating systems were addressing the problem with the kind of fsid you're proposing, I believe the point was that in Linux you can have multiple incompatible exports of the same filesystem -- to give a client the right data, you'd need to know from which export, not just which filesystem, a given filehandle came.
Well, I do not see how there is a difference.
Quoted 1 line
From my reading of the code it seems that fsid is composed of device's
minor/major and export point's inode ino.
I propose to replace (if supported by fs) device's minor/major by some unique number returned by fs itself.
Should be no worse than what we have now.
Bye,
Oleg
Quoted 7 lines · Oleg Drokin
Well, I do not see how there is a difference. From my reading of the code it seems that fsid is composed of device's minor/major and export point's inode ino.
I propose to replace (if supported by fs) device's minor/major by some unique number returned by fs itself. Should be no worse than what we have now.
Wouldn't some ways of identifying filesystems (ext2/3 labels?) more easily be done from userspace? Would it be possible to do what you want from userspace using the fsid export option? There's no reason the fsid= option has to be set manually; you could generate /etc/exports lines (or exportfs commands) automatically.
--b.
Quoted 6 lines
From my reading of the code it seems that fsid is composed of device's minor/major and export point's inode ino.
I propose to replace (if supported by fs) device's minor/major by some unique number returned by fs itself.
Should be no worse than what we have now.
That's for a device-based filesystem. But isn't your proposal primarily about non-device-based filesystems, and replacing a user-supplied value with this filesystem identifier? It isn't so clear that that can't make things worse.
But I don't actually remember the specific cases in which the fsid was considered insufficient; only that someone (probably Neil) did find that it was. (Before Linux had export IDs, you simply couldn't export a non-device-based filesystem at all, so we had to make a fake device on which the filesystem could "reside." The user managed the device number of that device as the filesystem ID).
--
Bryan Henderson IBM Almaden Research Center
San Jose CA Filesystems
Hello!
Quoted 6 lines · Bryan Henderson
Quoted 5 lines
From my reading of the code it seems that fsid is composed of device's minor/major and export point's inode ino.
I propose to replace (if supported by fs) device's minor/major by some unique number returned by fs itself.
Should be no worse than what we have now.That's for a device-based filesystem. But isn't your proposal primarily
Yes. Device-based fs has root->s_dev. For non-device based (and for device-based too, if they would like) - method to get some sort of unique id to be used in the same way root->s_dev is used.
Quoted 3 lines
about non-device-based filesystems, and replacing a user-supplied value with this filesystem identifier? It isn't so clear that that can't make things worse.
No, replacing whole fsid= with fs-supplied value would have implications like that you mentioned - we might have two different points in filesystem exported and we do not want those to have same id.
Quoted 6 lines
But I don't actually remember the specific cases in which the fsid was considered insufficient; only that someone (probably Neil) did find that it was. (Before Linux had export IDs, you simply couldn't export a non-device-based filesystem at all, so we had to make a fake device on which the filesystem could "reside." The user managed the device number of that device as the filesystem ID).
We still have those "fake devices". In fact non-device based fs can set REQUIRES_FS_DEV in fs_flags and it can be exported after that without manually specifying fsid. (yeah, I tried). But distributed filesystems would like to have same device id on different nodes.
Bye,
Oleg
Quoted 7 lines · On Wednesday April 12, [email protected] wrote:
I wonder if e.g. another export op could be introduced to ask filesystem for its unique id (if supported) and use that instead of
sb->s_devin fsid calculations. Does this look sane?
I hope I am not missing anything?
Or are there better ideas?
Yes. fsid= is a problem. Getting information from the filesystem would be nice. But there are issues of practicality.
The NFS filehandle is of limited size.
For NFSv2 we have 32 bytes and we cannot really afford more than 4 to identify the filesystem.
4 bytes isn't really enough for a globally unique ID. Many filesystems have UUIDs but I would expect them to be more like 16 bytes.
Such a number could be hashed down to 4bytes, but you would greatly increase the risk of collision.
Ofcourse, NFSv2 is pretty old now, and it is probably reasonable to have features that are only available to v3 or later.
v3 has space in the filehandle for up to 64 bytes. It would probably be OK to allow upto 16 - or maybe 20 - bytes to identify the filesystem.
There is still the issue of a filesystem having multiple export points, though that isn't really a big issue as it is rarely needed so we can impose extra constraints (you must give fsid=) in that case.
So there remains the question of whether this should be handled in the kernel or in user-space. There already exists libblkid which can produce UUIDs for various filesystems (I believe). But that reads the block device, and the whole point of this is to work with filesystems that don't have a simple block device.
I'd rather not make the 'get uuid' be an export op as it could well be useful to other clients than just nfsd.
Unfortunately there is no unanimity of how to present filesystem metadata to userspace.
- ioctls are frowned upon
- extended attribute have been suggested, they are about files, not filesystems
- filesystems don't have a place in /sys
- /proc is frowned upon
Here is an idea: How about we make a magic filesystem mounted on e.g. /metafs, which examines the current directory of the accessing process in order to report metadata.
So
cd /my/mount/point
cat /metafa/uuid
will show the uuid of the fs at /my/mount/point...
OK, so that's really weird. But I think there is a real need to be able to expose filesystem metadata in a uniform way and I would like the uuid to be made available through that.
With that in place, I would be happy to look at including a uuid in the filehandle for v3.
Any ideas?
NeilBrown
Quoted 1 line · Neil Brown
- filesystems don't have a place in /sys
Then what is /sys/fs/ for? :)
thanks,
greg k-h
Quoted 4 lines · On Sunday April 16, [email protected] wrote:
Quoted 1 line · Neil Brown
- filesystems don't have a place in /sys
Then what is
/sys/fs/for? :)
good question!
adams # uname -a
Linux adams 2.6.17-rc1-mm1 #298 SMP Tue Apr 11 14:19:52 EST 2006 i686 GNU/Linux adams # ls -la /sys/fs
total 0
drwxr-xr-x 2 root root 0 Apr 17 14:23 .
drwxr-xr-x 11 root root 0 Apr 17 14:23 ..
Doesn't look very useful yet, but I'm hopeful.
What naming scheme is expected? i.e. If I want to look at the nfs client rpc stats for /import/warehouse, where would I look?
/sys/fs/import/warehouse/nfs/stats
seems unlikely.
The idea of putting filesystem stuff in /sys seem to come up on linux-kernel every so often and it never seems to get anywhere, often because Al Viro pokes obvious holes in the scheme.
I cannot find anything in Documentation/filesystems/sysfs.txt, so:
How is /sys/fs supposed to work?
Thanks,
NeilBrown
Quoted 22 lines · Neil Brown
Quoted 4 lines · On Sunday April 16, [email protected] wrote:
Quoted 1 line · Neil Brown
- filesystems don't have a place in /sys
Then what is
/sys/fs/for? :)good question!
adams # uname -a
Linux adams 2.6.17-rc1-mm1 #298 SMP Tue Apr 11 14:19:52 EST 2006 i686 GNU/Linux adams # ls -la/sys/fs
total 0
drwxr-xr-x 2 root root 0 Apr 17 14:23 .
drwxr-xr-x 11 root root 0 Apr 17 14:23 ..Doesn't look very useful yet, but I'm hopeful.
What naming scheme is expected? i.e. If I want to look at the nfs client rpc stats for /import/warehouse, where would I look?
/sys/fs/import/warehouse/nfs/stats
seems unlikely.
Quoted 6 lines
The idea of putting filesystem stuff in /sys seem to come up on linux-kernel every so often and it never seems to get anywhere, often because Al Viro pokes obvious holes in the scheme.
I cannot find anything in
Documentation/filesystems/sysfs.txt, so:
How is/sys/fssupposed to work?
I don't know, as I didn't create it. I think one of the FUSE developers did.
thanks,
greg k-h
Quoted 7 lines
Quoted 4 lines
because Al Viro pokes obvious holes in the scheme.
I cannot find anything in
Documentation/filesystems/sysfs.txt, so:
How is/sys/fssupposed to work?I don't know, as I didn't create it. I think one of the FUSE developers did.
Yeah. There are a couple of FUSE specific attributes under /sys/fs/fuse (documented in Documentation/filesystems/fuse.txt).
I'll put a pointer into sysfs.txt.
Miklos
Quoted 5 lines · Miklos Szeredi
Quoted 5 lines
Quoted 4 lines
because Al Viro pokes obvious holes in the scheme.
I cannot find anything in
Documentation/filesystems/sysfs.txt, so:
How is/sys/fssupposed to work?
Quoted 7 lines
Quoted 2 lines
I don't know, as I didn't create it. I think one of the FUSE developers did.
Yeah. There are a couple of FUSE specific attributes under
/sys/fs/fuse(documented inDocumentation/filesystems/fuse.txt).I'll put a pointer into sysfs.txt.
Speaking of obvious holes, what is
/* Setting file->private_data can't race with other mount()
instances, since BKL is held for ->get_sb() */
if (file->private_data)
return -EINVAL;doing, seeing that the next thing you do is GFP_KERNEL allocations, which would kill all hope to use BKL-based protection here?
Quoted 7 lines
Speaking of obvious holes, what is
/* Setting file->private_data can't race with other mount() instances, since BKL is held for ->get_sb() */ if (file->private_data) return -EINVAL;doing, seeing that the next thing you do is
GFP_KERNELallocations, which would kill all hope to use BKL-based protection here?
Why? I didn't know BKL is incompatible with allocations.
Miklos
Quoted 9 lines · Miklos Szeredi
Quoted 7 lines
Speaking of obvious holes, what is
/* Setting file->private_data can't race with other mount() instances, since BKL is held for ->get_sb() */ if (file->private_data) return -EINVAL;doing, seeing that the next thing you do is
GFP_KERNELallocations, which would kill all hope to use BKL-based protection here?Why? I didn't know BKL is incompatible with allocations.
BKL is dropped when you give up CPU. It is reacquired when you get scheduled again. It's not a mutex; it's not a spinlock. The right way to look at it is the following:
* task has dumbness counter; lock_kernel() increments it,unlock_kernel() decrements.
* at most one dumb task can have CPU at any given time
* dumb tasks are not preempted
That's it. You are only protected against other dumb tasks and *only* as long as you don't block. Blocking won't make you any less dumb, so the same warranties resume once you do get CPU again. However, anything might have happened while you were blocked.
On UP boxen with PREEMPT disabled it's simply a no-op - it's all about emulating that situation.
Quoted 12 lines
BKL is dropped when you give up CPU. It is reacquired when you get scheduled again. It's not a mutex; it's not a spinlock. The right way to look at it is the following:
* task has dumbness counter;lock_kernel()increments it,unlock_kernel()decrements.
* at most one dumb task can have CPU at any given time
* dumb tasks are not preempted
That's it. You are only protected against other dumb tasks and *only* as long as you don't block. Blocking won't make you any less dumb, so the same warranties resume once you do get CPU again. However, anything might have happened while you were blocked.
Quoted 2 lines
On UP boxen with PREEMPT disabled it's simply a no-op - it's all about emulating that situation.
I see, thanks. I'll fix the race.
Miklos
Quoted 60 lines · Neil Brown
Quoted 6 lines · On Wednesday April 12, [email protected] wrote:
I wonder if e.g. another export op could be introduced to ask filesystem for its unique id (if supported) and use that instead of
sb->s_devin fsid calculations. Does this look sane?
I hope I am not missing anything?
Or are there better ideas?Yes. fsid= is a problem. Getting information from the filesystem would be nice. But there are issues of practicality.
The NFS filehandle is of limited size.
For NFSv2 we have 32 bytes and we cannot really afford more than 4 to identify the filesystem.
4 bytes isn't really enough for a globally unique ID. Many filesystems have UUIDs but I would expect them to be more like 16 bytes.
Such a number could be hashed down to 4bytes, but you would greatly increase the risk of collision.Ofcourse, NFSv2 is pretty old now, and it is probably reasonable to have features that are only available to v3 or later.
v3 has space in the filehandle for up to 64 bytes. It would probably be OK to allow upto 16 - or maybe 20 - bytes to identify the filesystem.There is still the issue of a filesystem having multiple export points, though that isn't really a big issue as it is rarely needed so we can impose extra constraints (you must give fsid=) in that case.
So there remains the question of whether this should be handled in the kernel or in user-space. There already exists libblkid which can produce UUIDs for various filesystems (I believe). But that reads the block device, and the whole point of this is to work with filesystems that don't have a simple block device.
I'd rather not make the 'get uuid' be an export op as it could well be useful to other clients than just nfsd.
Unfortunately there is no unanimity of how to present filesystem metadata to userspace.
- ioctls are frowned upon
- extended attribute have been suggested, they are about files, not filesystems
- filesystems don't have a place in /sys
- /proc is frowned uponHere is an idea: How about we make a magic filesystem mounted on e.g. /metafs, which examines the current directory of the accessing process in order to report metadata.
So
cd /my/mount/point
cat /metafa/uuid
will show the uuid of the fs at /my/mount/point...
OK, so that's really weird. But I think there is a real need to be able to expose filesystem metadata in a uniform way and I would like the uuid to be made available through that.
With that in place, I would be happy to look at including a uuid in the filehandle for v3.
Quoted 1 line
Any ideas?
Something like getattrlist() system call on OSX would do very nicely. It even allows for filesystems to communicate which attributes are supported and which are not. (And there is an equivalent setattrlist(), too.)
http://developer.apple.com/documentation/Darwin/Reference/ManPages/man2/getattrlist.2.html
On the kernel VFS/FS interface side this works by the VFS sending a bitmask to the FS, with each bit being one of the attributes requested and for each set bit, the FS returns the appropriate information in the "getattr structure" and sets a flag in a second bitmask to say "I have returned this attribute". The VFS then converts the returned attribute values as signified by set bits in the second bitmask from the structure returned by the FS into the user supplied buffer in a packed form (see the above man page for details on the packed form).
This works very well and is very extensible.
Linux already has a per FS getattr() inode operation which could easily be extended to be more powerful and a system call could be added to return information obtained from it...
Best regards,
Anton
Signature · 3 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer, http://www.linux-ntfs.org/