Skip to contentSkip to conversation

FS-specified FSID for non-device based filesystems?

DISCUSSION Oleg Drokin 8 participants 16 messages Apr 12, 2006 – Apr 20, 2006 VFS

lore
Share & tools
lore
#0 lore raw

Hello!

Right now NFSD depends on FS_REQUIRES_DEV to be set in sb->s_type->fs_flags or manually set fsid= in /etc/exports in order to export filesystem. But there are non-device based filesystems (Lustre for example ;) ) that would like to be exported via NFS without guiding users through this fsid= business. Of course it is easy to cheat through by setting FS_REQUIRES_DEV, but this does not solve entire problem. While allowing for single-node export of filesystem to work ok, more generic problem of exporting same filesystem from several NFS servers in clustered manner won't work because sb->s_dev might be different on different nodes. This is not only Lustre problem, I believe same problem will happen with GFS or OCFS2 or whatever other SAN-based fs if different nodes have different device names (/dev/sda | /dev/sdb) for same SAN device, in such a case fsid calculated would be different on different nodes too.

I wonder if e.g. another export op could be introduced to ask filesystem for its unique id (if supported) and use that instead of sb->s_dev in fsid calculations. Does this look sane?
I hope I am not missing anything?
Or are there better ideas?

Thanks.

Bye,
Oleg

#1 lore raw
Quoted 1 line

Right now NFSD depends on FS_REQUIRES_DEV to be set in

sb->s_type->fs_flags

Quoted 7 lines

or manually set fsid= in /etc/exports in order to export filesystem. But there are non-device based filesystems (Lustre for example ;) ) that would like to be exported via NFS without guiding users through this fsid= business.
...
I wonder if e.g. another export op could be introduced to ask filesystem for its unique id

It would also be quite valuable with device-based filesystems, for the opposite reason. That a device-based filesystem can be uniquely identified by the device number of its device is a fiction that is getting harder to entertain every year. Device numbers of devices change and a device can contain multiple filesystems over time.

When Linux added the export ID, at a time when other operating systems were addressing the problem with the kind of fsid you're proposing, I believe the point was that in Linux you can have multiple incompatible exports of the same filesystem -- to give a client the right data, you'd need to know from which export, not just which filesystem, a given filehandle came.

Nonetheless, I agree that it would be good to have the option of using a filesystem identifier instead of an export identifier. In a system that only does simple uniform exports of whole filesystems, I think filesystem ID works perfectly, and it is much easier to use.

--
Bryan Henderson IBM Almaden Research Center
San Jose CA Filesystems

#2 lore raw

Hello!

Quoted 6 lines · Bryan Henderson

When Linux added the export ID, at a time when other operating systems were addressing the problem with the kind of fsid you're proposing, I believe the point was that in Linux you can have multiple incompatible exports of the same filesystem -- to give a client the right data, you'd need to know from which export, not just which filesystem, a given filehandle came.

Well, I do not see how there is a difference.

Quoted 1 line

From my reading of the code it seems that fsid is composed of device's

minor/major and export point's inode ino.

I propose to replace (if supported by fs) device's minor/major by some unique number returned by fs itself.
Should be no worse than what we have now.

Bye,
Oleg

#3 lore raw
Quoted 7 lines · Oleg Drokin

Well, I do not see how there is a difference. From my reading of the code it seems that fsid is composed of device's minor/major and export point's inode ino.

I propose to replace (if supported by fs) device's minor/major by some unique number returned by fs itself. Should be no worse than what we have now.

Wouldn't some ways of identifying filesystems (ext2/3 labels?) more easily be done from userspace? Would it be possible to do what you want from userspace using the fsid export option? There's no reason the fsid= option has to be set manually; you could generate /etc/exports lines (or exportfs commands) automatically.

--b.

#4 lore raw
In reply to Oleg Drokin
Quoted 6 lines

From my reading of the code it seems that fsid is composed of device's minor/major and export point's inode ino.

I propose to replace (if supported by fs) device's minor/major by some unique number returned by fs itself.
Should be no worse than what we have now.

That's for a device-based filesystem. But isn't your proposal primarily about non-device-based filesystems, and replacing a user-supplied value with this filesystem identifier? It isn't so clear that that can't make things worse.

But I don't actually remember the specific cases in which the fsid was considered insufficient; only that someone (probably Neil) did find that it was. (Before Linux had export IDs, you simply couldn't export a non-device-based filesystem at all, so we had to make a fake device on which the filesystem could "reside." The user managed the device number of that device as the filesystem ID).

--
Bryan Henderson IBM Almaden Research Center
San Jose CA Filesystems

#5 lore raw

Hello!

Quoted 6 lines · Bryan Henderson
Quoted 5 lines

From my reading of the code it seems that fsid is composed of device's minor/major and export point's inode ino.
I propose to replace (if supported by fs) device's minor/major by some unique number returned by fs itself.
Should be no worse than what we have now.

That's for a device-based filesystem. But isn't your proposal primarily

Yes. Device-based fs has root->s_dev. For non-device based (and for device-based too, if they would like) - method to get some sort of unique id to be used in the same way root->s_dev is used.

Quoted 3 lines

about non-device-based filesystems, and replacing a user-supplied value with this filesystem identifier? It isn't so clear that that can't make things worse.

No, replacing whole fsid= with fs-supplied value would have implications like that you mentioned - we might have two different points in filesystem exported and we do not want those to have same id.

Quoted 6 lines

But I don't actually remember the specific cases in which the fsid was considered insufficient; only that someone (probably Neil) did find that it was. (Before Linux had export IDs, you simply couldn't export a non-device-based filesystem at all, so we had to make a fake device on which the filesystem could "reside." The user managed the device number of that device as the filesystem ID).

We still have those "fake devices". In fact non-device based fs can set REQUIRES_FS_DEV in fs_flags and it can be exported after that without manually specifying fsid. (yeah, I tried). But distributed filesystems would like to have same device id on different nodes.

Bye,
Oleg

#6 lore raw
In reply to Oleg Drokin
Quoted 7 lines · On Wednesday April 12, [email protected] wrote:

I wonder if e.g. another export op could be introduced to ask filesystem for its unique id (if supported) and use that instead of sb->s_dev in fsid calculations. Does this look sane?
I hope I am not missing anything?
Or are there better ideas?

Yes. fsid= is a problem. Getting information from the filesystem would be nice. But there are issues of practicality.

The NFS filehandle is of limited size.
For NFSv2 we have 32 bytes and we cannot really afford more than 4 to identify the filesystem.
4 bytes isn't really enough for a globally unique ID. Many filesystems have UUIDs but I would expect them to be more like 16 bytes.
Such a number could be hashed down to 4bytes, but you would greatly increase the risk of collision.

Ofcourse, NFSv2 is pretty old now, and it is probably reasonable to have features that are only available to v3 or later.
v3 has space in the filehandle for up to 64 bytes. It would probably be OK to allow upto 16 - or maybe 20 - bytes to identify the filesystem.

There is still the issue of a filesystem having multiple export points, though that isn't really a big issue as it is rarely needed so we can impose extra constraints (you must give fsid=) in that case.

So there remains the question of whether this should be handled in the kernel or in user-space. There already exists libblkid which can produce UUIDs for various filesystems (I believe). But that reads the block device, and the whole point of this is to work with filesystems that don't have a simple block device.

I'd rather not make the 'get uuid' be an export op as it could well be useful to other clients than just nfsd.

Unfortunately there is no unanimity of how to present filesystem metadata to userspace.
- ioctls are frowned upon
- extended attribute have been suggested, they are about files, not filesystems
- filesystems don't have a place in /sys
- /proc is frowned upon

Here is an idea: How about we make a magic filesystem mounted on e.g. /metafs, which examines the current directory of the accessing process in order to report metadata.
So
cd /my/mount/point
cat /metafa/uuid
will show the uuid of the fs at /my/mount/point...
OK, so that's really weird. But I think there is a real need to be able to expose filesystem metadata in a uniform way and I would like the uuid to be made available through that.
With that in place, I would be happy to look at including a uuid in the filehandle for v3.

Any ideas?

NeilBrown

#7 lore raw
Quoted 1 line · Neil Brown

- filesystems don't have a place in /sys

Then what is /sys/fs/ for? :)

thanks,

greg k-h

#8 lore raw
Quoted 4 lines · On Sunday April 16, [email protected] wrote:
Quoted 1 line · Neil Brown

- filesystems don't have a place in /sys

Then what is /sys/fs/ for? :)

good question!

adams # uname -a
Linux adams 2.6.17-rc1-mm1 #298 SMP Tue Apr 11 14:19:52 EST 2006 i686 GNU/Linux adams # ls -la /sys/fs
total 0
drwxr-xr-x 2 root root 0 Apr 17 14:23 .
drwxr-xr-x 11 root root 0 Apr 17 14:23 ..

Doesn't look very useful yet, but I'm hopeful.

What naming scheme is expected? i.e. If I want to look at the nfs client rpc stats for /import/warehouse, where would I look?
/sys/fs/import/warehouse/nfs/stats
seems unlikely.

The idea of putting filesystem stuff in /sys seem to come up on linux-kernel every so often and it never seems to get anywhere, often because Al Viro pokes obvious holes in the scheme.

I cannot find anything in Documentation/filesystems/sysfs.txt, so:
How is /sys/fs supposed to work?

Thanks,
NeilBrown

#15 lore raw
In reply to Neil Brown
Quoted 60 lines · Neil Brown
Quoted 6 lines · On Wednesday April 12, [email protected] wrote:

I wonder if e.g. another export op could be introduced to ask filesystem for its unique id (if supported) and use that instead of sb->s_dev in fsid calculations. Does this look sane?
I hope I am not missing anything?
Or are there better ideas?

Yes. fsid= is a problem. Getting information from the filesystem would be nice. But there are issues of practicality.

The NFS filehandle is of limited size.
For NFSv2 we have 32 bytes and we cannot really afford more than 4 to identify the filesystem.
4 bytes isn't really enough for a globally unique ID. Many filesystems have UUIDs but I would expect them to be more like 16 bytes.
Such a number could be hashed down to 4bytes, but you would greatly increase the risk of collision.

Ofcourse, NFSv2 is pretty old now, and it is probably reasonable to have features that are only available to v3 or later.
v3 has space in the filehandle for up to 64 bytes. It would probably be OK to allow upto 16 - or maybe 20 - bytes to identify the filesystem.

There is still the issue of a filesystem having multiple export points, though that isn't really a big issue as it is rarely needed so we can impose extra constraints (you must give fsid=) in that case.

So there remains the question of whether this should be handled in the kernel or in user-space. There already exists libblkid which can produce UUIDs for various filesystems (I believe). But that reads the block device, and the whole point of this is to work with filesystems that don't have a simple block device.

I'd rather not make the 'get uuid' be an export op as it could well be useful to other clients than just nfsd.

Unfortunately there is no unanimity of how to present filesystem metadata to userspace.
- ioctls are frowned upon
- extended attribute have been suggested, they are about files, not filesystems
- filesystems don't have a place in /sys
- /proc is frowned upon

Here is an idea: How about we make a magic filesystem mounted on e.g. /metafs, which examines the current directory of the accessing process in order to report metadata.
So
cd /my/mount/point
cat /metafa/uuid
will show the uuid of the fs at /my/mount/point...
OK, so that's really weird. But I think there is a real need to be able to expose filesystem metadata in a uniform way and I would like the uuid to be made available through that.
With that in place, I would be happy to look at including a uuid in the filehandle for v3.

Quoted 1 line

Any ideas?

Something like getattrlist() system call on OSX would do very nicely. It even allows for filesystems to communicate which attributes are supported and which are not. (And there is an equivalent setattrlist(), too.)

http://developer.apple.com/documentation/Darwin/Reference/ManPages/man2/getattrlist.2.html

On the kernel VFS/FS interface side this works by the VFS sending a bitmask to the FS, with each bit being one of the attributes requested and for each set bit, the FS returns the appropriate information in the "getattr structure" and sets a flag in a second bitmask to say "I have returned this attribute". The VFS then converts the returned attribute values as signified by set bits in the second bitmask from the structure returned by the FS into the user supplied buffer in a packed form (see the above man page for details on the packed form).

This works very well and is very extensible.

Linux already has a per FS getattr() inode operation which could easily be extended to be more powerful and a system call could be added to return information obtained from it...

Best regards,

Anton

Signature · 3 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/
#9 lore raw
In reply to Neil Brown
Quoted 22 lines · Neil Brown
Quoted 4 lines · On Sunday April 16, [email protected] wrote:
Quoted 1 line · Neil Brown

- filesystems don't have a place in /sys

Then what is /sys/fs/ for? :)

good question!

adams # uname -a
Linux adams 2.6.17-rc1-mm1 #298 SMP Tue Apr 11 14:19:52 EST 2006 i686 GNU/Linux adams # ls -la /sys/fs
total 0
drwxr-xr-x 2 root root 0 Apr 17 14:23 .
drwxr-xr-x 11 root root 0 Apr 17 14:23 ..

Doesn't look very useful yet, but I'm hopeful.

What naming scheme is expected? i.e. If I want to look at the nfs client rpc stats for /import/warehouse, where would I look?
/sys/fs/import/warehouse/nfs/stats
seems unlikely.

Quoted 6 lines

The idea of putting filesystem stuff in /sys seem to come up on linux-kernel every so often and it never seems to get anywhere, often because Al Viro pokes obvious holes in the scheme.

I cannot find anything in Documentation/filesystems/sysfs.txt, so:
How is /sys/fs supposed to work?

I don't know, as I didn't create it. I think one of the FUSE developers did.

thanks,

greg k-h

#10 lore raw
Quoted 7 lines
Quoted 4 lines

because Al Viro pokes obvious holes in the scheme.

I cannot find anything in Documentation/filesystems/sysfs.txt, so:
How is /sys/fs supposed to work?

I don't know, as I didn't create it. I think one of the FUSE developers did.

Yeah. There are a couple of FUSE specific attributes under /sys/fs/fuse (documented in Documentation/filesystems/fuse.txt).

I'll put a pointer into sysfs.txt.

Miklos

#11 lore raw
Quoted 5 lines · Miklos Szeredi
Quoted 5 lines
Quoted 4 lines

because Al Viro pokes obvious holes in the scheme.

I cannot find anything in Documentation/filesystems/sysfs.txt, so:
How is /sys/fs supposed to work?

Quoted 7 lines
Quoted 2 lines

I don't know, as I didn't create it. I think one of the FUSE developers did.

Yeah. There are a couple of FUSE specific attributes under /sys/fs/fuse (documented in Documentation/filesystems/fuse.txt).

I'll put a pointer into sysfs.txt.

Speaking of obvious holes, what is

        /* Setting file->private_data can't race with other mount()
           instances, since BKL is held for ->get_sb() */
        if (file->private_data)
                return -EINVAL;

doing, seeing that the next thing you do is GFP_KERNEL allocations, which would kill all hope to use BKL-based protection here?

#12 lore raw
Quoted 7 lines

Speaking of obvious holes, what is

        /* Setting file->private_data can't race with other mount()
           instances, since BKL is held for ->get_sb() */
        if (file->private_data)
                return -EINVAL;

doing, seeing that the next thing you do is GFP_KERNEL allocations, which would kill all hope to use BKL-based protection here?

Why? I didn't know BKL is incompatible with allocations.

Miklos

#13 lore raw
Quoted 9 lines · Miklos Szeredi
Quoted 7 lines

Speaking of obvious holes, what is

        /* Setting file->private_data can't race with other mount()
           instances, since BKL is held for ->get_sb() */
        if (file->private_data)
                return -EINVAL;

doing, seeing that the next thing you do is GFP_KERNEL allocations, which would kill all hope to use BKL-based protection here?

Why? I didn't know BKL is incompatible with allocations.

BKL is dropped when you give up CPU. It is reacquired when you get scheduled again. It's not a mutex; it's not a spinlock. The right way to look at it is the following:
* task has dumbness counter; lock_kernel() increments it,
unlock_kernel() decrements.
* at most one dumb task can have CPU at any given time
* dumb tasks are not preempted
That's it. You are only protected against other dumb tasks and *only* as long as you don't block. Blocking won't make you any less dumb, so the same warranties resume once you do get CPU again. However, anything might have happened while you were blocked.

On UP boxen with PREEMPT disabled it's simply a no-op - it's all about emulating that situation.

#14 lore raw
Quoted 12 lines

BKL is dropped when you give up CPU. It is reacquired when you get scheduled again. It's not a mutex; it's not a spinlock. The right way to look at it is the following:
* task has dumbness counter; lock_kernel() increments it,
unlock_kernel() decrements.
* at most one dumb task can have CPU at any given time
* dumb tasks are not preempted
That's it. You are only protected against other dumb tasks and *only* as long as you don't block. Blocking won't make you any less dumb, so the same warranties resume once you do get CPU again. However, anything might have happened while you were blocked.

Quoted 2 lines

On UP boxen with PREEMPT disabled it's simply a no-op - it's all about emulating that situation.

I see, thanks. I'll fix the race.

Miklos