Skip to contentSkip to conversation

mountpoint-crossing

DISCUSSION J. Bruce Fields 3 participants 7 messages Dec 13, 2009 – Dec 14, 2009 2 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

On a recent kernel:

	# mount -tnfs4 pearlet1:/ /mnt/
	# find /mnt/
	/mnt/
	find: File system loop detected; `/mnt/DIR' is part of the same
	file system loop as `/mnt/'.

Here /mnt/DIR is a server-side mountpoint, hence has a different fsid than /mnt/. Wireshark confirms that the server is returning a different fsid. However, 'strace -v find /mnt/' shows stat returning st_dev=makedev(0, 22) for both /mnt and /mnt/DIR.

If I then do a 'ls /mnt/DIR', followed by another find, the error goes away, and this time an strace shows that stat is returning (0, 23) for /mnt/DIR.

I don't see any obvious problem with the network trace, so it looks to me like the client is failing to recognize the mountpoint when it should?

--b.

#1 lore raw
Quoted 13 lines · J. Bruce Fields

On a recent kernel:

	# mount -tnfs4 pearlet1:/ /mnt/
	# find /mnt/
	/mnt/
	find: File system loop detected; `/mnt/DIR' is part of the same
	file system loop as `/mnt/'.

Here /mnt/DIR is a server-side mountpoint, hence has a different fsid than /mnt/. Wireshark confirms that the server is returning a different fsid. However, 'strace -v find /mnt/' shows stat returning st_dev=makedev(0, 22) for both /mnt and /mnt/DIR.

Quoted 7 lines

If I then do a 'ls /mnt/DIR', followed by another find, the error goes away, and this time an strace shows that stat is returning (0, 23) for /mnt/DIR.

I don't see any obvious problem with the network trace, so it looks to me like the client is failing to recognize the mountpoint when it should?

This is a known consequence of the way we treat submounts (and referrals); we're basically treating them as a special kind of symlink. The problem then arises when syscalls such as stat() fail to set the LOOKUP_FOLLOW flag, and so the user is granted a temporary peek of the underlying inode.

I'm not sure how we should treat this. I suppose we could change the test in __link_path_walk() so that it always call follow_link() if the inode is not a symlink...

Cheers
Trond

#2 lore raw
Quoted 28 lines · On Sun, 13 Dec 2009 17:33:15 -0500 Trond Myklebust…
Quoted 13 lines · J. Bruce Fields

On a recent kernel:

	# mount -tnfs4 pearlet1:/ /mnt/
	# find /mnt/
	/mnt/
	find: File system loop detected; `/mnt/DIR' is part of the same
	file system loop as `/mnt/'.

Here /mnt/DIR is a server-side mountpoint, hence has a different fsid than /mnt/. Wireshark confirms that the server is returning a different fsid. However, 'strace -v find /mnt/' shows stat returning st_dev=makedev(0, 22) for both /mnt and /mnt/DIR.

Quoted 7 lines

If I then do a 'ls /mnt/DIR', followed by another find, the error goes away, and this time an strace shows that stat is returning (0, 23) for /mnt/DIR.

I don't see any obvious problem with the network trace, so it looks to me like the client is failing to recognize the mountpoint when it should?

This is a known consequence of the way we treat submounts (and referrals); we're basically treating them as a special kind of symlink. The problem then arises when syscalls such as stat() fail to set the LOOKUP_FOLLOW flag, and so the user is granted a temporary peek of the underlying inode.

Quoted 4 lines

I'm not sure how we should treat this. I suppose we could change the test in __link_path_walk() so that it always call follow_link() if the inode is not a symlink...

I looked at this problem recently based on a request by some of our coreutils folks. A bit of the discussion is here:

    https://bugzilla.redhat.com/show_bug.cgi?id=533569

...and earlier:

    https://bugzilla.redhat.com/show_bug.cgi?id=501848

Jim Meyering also brought this up on LKML:

    http://lkml.org/lkml/2009/11/4/451

I'm a little leery of triggering a mount for any server-side mountpoint that we just happen to have a peek at. That seems like it might get expensive. Suppose you had 1000 filesystems mounted under the root share here?

One idea in the mailing list discussion is to flag these inodes with some sort of "i'm actually a mountpoint" flag and teach utilities that care about inode numbers to deal with that. Not a great solution but it wouldn't incur extra overhead.

Signature · 1 line
Jeff Layton <[email protected]>

A message that is not in the archive (sent off-list or not yet received): 20091214083843.5e6e73f5-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org

#4 lore raw
Quoted 45 lines · Jeff Layton
Quoted 28 lines · On Sun, 13 Dec 2009 17:33:15 -0500 Trond Myklebust…
Quoted 13 lines · J. Bruce Fields

On a recent kernel:

	# mount -tnfs4 pearlet1:/ /mnt/
	# find /mnt/
	/mnt/
	find: File system loop detected; `/mnt/DIR' is part of the same
	file system loop as `/mnt/'.

Here /mnt/DIR is a server-side mountpoint, hence has a different fsid than /mnt/. Wireshark confirms that the server is returning a different fsid. However, 'strace -v find /mnt/' shows stat returning st_dev=makedev(0, 22) for both /mnt and /mnt/DIR.

Quoted 7 lines

If I then do a 'ls /mnt/DIR', followed by another find, the error goes away, and this time an strace shows that stat is returning (0, 23) for /mnt/DIR.

I don't see any obvious problem with the network trace, so it looks to me like the client is failing to recognize the mountpoint when it should?

This is a known consequence of the way we treat submounts (and referrals); we're basically treating them as a special kind of symlink. The problem then arises when syscalls such as stat() fail to set the LOOKUP_FOLLOW flag, and so the user is granted a temporary peek of the underlying inode.

Quoted 4 lines

I'm not sure how we should treat this. I suppose we could change the test in __link_path_walk() so that it always call follow_link() if the inode is not a symlink...

I looked at this problem recently based on a request by some of our coreutils folks. A bit of the discussion is here:

    https://bugzilla.redhat.com/show_bug.cgi?id=533569

...and earlier:

    https://bugzilla.redhat.com/show_bug.cgi?id=501848
Quoted 8 lines

Jim Meyering also brought this up on LKML:

    http://lkml.org/lkml/2009/11/4/451

I'm a little leery of triggering a mount for any server-side mountpoint that we just happen to have a peek at. That seems like it might get expensive. Suppose you had 1000 filesystems mounted under the root share here?

For what it's worth, I'll admit that I ran across this just in artificial testing--I'm not claiming it was causing me a real problem.

--b.

Quoted 8 lines

One idea in the mailing list discussion is to flag these inodes with some sort of "i'm actually a mountpoint" flag and teach utilities that care about inode numbers to deal with that. Not a great solution but it wouldn't incur extra overhead.

Signature · 1 line
Jeff Layton <[email protected]>
#5 lore raw
Quoted 50 lines · On Mon, 14 Dec 2009 10:24:18 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 49 lines · Jeff Layton
Quoted 28 lines · On Sun, 13 Dec 2009 17:33:15 -0500 Trond Myklebust…
Quoted 13 lines · J. Bruce Fields

On a recent kernel:

	# mount -tnfs4 pearlet1:/ /mnt/
	# find /mnt/
	/mnt/
	find: File system loop detected; `/mnt/DIR' is part of the same
	file system loop as `/mnt/'.

Here /mnt/DIR is a server-side mountpoint, hence has a different fsid than /mnt/. Wireshark confirms that the server is returning a different fsid. However, 'strace -v find /mnt/' shows stat returning st_dev=makedev(0, 22) for both /mnt and /mnt/DIR.

Quoted 7 lines

If I then do a 'ls /mnt/DIR', followed by another find, the error goes away, and this time an strace shows that stat is returning (0, 23) for /mnt/DIR.

I don't see any obvious problem with the network trace, so it looks to me like the client is failing to recognize the mountpoint when it should?

This is a known consequence of the way we treat submounts (and referrals); we're basically treating them as a special kind of symlink. The problem then arises when syscalls such as stat() fail to set the LOOKUP_FOLLOW flag, and so the user is granted a temporary peek of the underlying inode.

Quoted 4 lines

I'm not sure how we should treat this. I suppose we could change the test in __link_path_walk() so that it always call follow_link() if the inode is not a symlink...

I looked at this problem recently based on a request by some of our coreutils folks. A bit of the discussion is here:

    https://bugzilla.redhat.com/show_bug.cgi?id=533569

...and earlier:

    https://bugzilla.redhat.com/show_bug.cgi?id=501848

Jim Meyering also brought this up on LKML:

    http://lkml.org/lkml/2009/11/4/451
Quoted 8 lines
Quoted 4 lines

I'm a little leery of triggering a mount for any server-side mountpoint that we just happen to have a peek at. That seems like it might get expensive. Suppose you had 1000 filesystems mounted under the root share here?

For what it's worth, I'll admit that I ran across this just in artificial testing--I'm not claiming it was causing me a real problem.

Understood. It's a bit of a dilemma...

Clearly though, it's going to be a problem for some programs that need to deal with mountpoints (stuff like backup programs in particular). The problem though is that I don't think we want to trigger a bunch of submounts just because someone does a "ls -l" in a directory that holds a bunch of server-side mountpoints.

The real problem I think is that we allocate new dev minor numbers at mount time. The ideal thing might be to have the client somehow pre-determine what the dev number of that mount would be without actually doing the mount. Then we could just present that device number in the stat call.

Signature · 1 line
Jeff Layton <[email protected]>

A message that is not in the archive (sent off-list or not yet received): 20091214105214.64714867-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org

#7 lore raw
Quoted 28 lines · Jeff Layton
Quoted 14 lines · On Mon, 14 Dec 2009 10:24:18 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 13 lines · Jeff Layton

I looked at this problem recently based on a request by some of our coreutils folks. A bit of the discussion is here:

    https://bugzilla.redhat.com/show_bug.cgi?id=533569

...and earlier:

    https://bugzilla.redhat.com/show_bug.cgi?id=501848

Jim Meyering also brought this up on LKML:

    http://lkml.org/lkml/2009/11/4/451
Quoted 8 lines
Quoted 4 lines

I'm a little leery of triggering a mount for any server-side mountpoint that we just happen to have a peek at. That seems like it might get expensive. Suppose you had 1000 filesystems mounted under the root share here?

For what it's worth, I'll admit that I ran across this just in artificial testing--I'm not claiming it was causing me a real problem.

Understood. It's a bit of a dilemma...

Quoted 8 lines

Clearly though, it's going to be a problem for some programs that need to deal with mountpoints (stuff like backup programs in particular). The problem though is that I don't think we want to trigger a bunch of submounts just because someone does a "ls -l" in a directory that holds a bunch of server-side mountpoints.

The real problem I think is that we allocate new dev minor numbers at mount time.

So you're not saying that the minor number allocation is the expensive part, you're saying that it's cheap and something that we could do before we do the rest of the mount?

Quoted 4 lines

The ideal thing might be to have the client somehow pre-determine what the dev number of that mount would be without actually doing the mount. Then we could just present that device number in the stat call.

We also need the inode number, for example, which may require an rpc call.

So what is the most expensive part of a mount?

For a directory full of referral points, there's the problem that you don't want to have to wait on stat calls from a lot of different servers. But maybe that should be handled as a special case.

--b.

#8 lore raw
Quoted 35 lines · On Mon, 14 Dec 2009 11:04:01 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 34 lines · Jeff Layton
Quoted 14 lines · On Mon, 14 Dec 2009 10:24:18 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 13 lines · Jeff Layton

I looked at this problem recently based on a request by some of our coreutils folks. A bit of the discussion is here:

    https://bugzilla.redhat.com/show_bug.cgi?id=533569

...and earlier:

    https://bugzilla.redhat.com/show_bug.cgi?id=501848

Jim Meyering also brought this up on LKML:

    http://lkml.org/lkml/2009/11/4/451
Quoted 8 lines
Quoted 4 lines

I'm a little leery of triggering a mount for any server-side mountpoint that we just happen to have a peek at. That seems like it might get expensive. Suppose you had 1000 filesystems mounted under the root share here?

For what it's worth, I'll admit that I ran across this just in artificial testing--I'm not claiming it was causing me a real problem.

Understood. It's a bit of a dilemma...

Clearly though, it's going to be a problem for some programs that need to deal with mountpoints (stuff like backup programs in particular). The problem though is that I don't think we want to trigger a bunch of submounts just because someone does a "ls -l" in a directory that holds a bunch of server-side mountpoints.

Quoted 7 lines
Quoted 2 lines

The real problem I think is that we allocate new dev minor numbers at mount time.

So you're not saying that the minor number allocation is the expensive part, you're saying that it's cheap and something that we could do before we do the rest of the mount?

Yeah, minor number allocation is fairly cheap (it's just IDA hash calls
I think). If we wanted to try and preallocate them then we have to
consider how long to cache them too. The hassle of doing that may outweigh the expense of triggering

I am making an assumption that mounts are somewhat expensive to do. Maybe you can convince me otherwise. :)

Quoted 8 lines
Quoted 4 lines

The ideal thing might be to have the client somehow pre-determine what the dev number of that mount would be without actually doing the mount. Then we could just present that device number in the stat call.

We also need the inode number, for example, which may require an rpc call.

We already have that, right? We've done a GETATTR (or equivalent) and noticed that the inode has a different fsid. We even send back the "real" inode number in the statbuf (but obviously w/o the right device info since the mount hasn't been triggered yet).

Quoted 6 lines

So what is the most expensive part of a mount?

For a directory full of referral points, there's the problem that you don't want to have to wait on stat calls from a lot of different servers. But maybe that should be handled as a special case.

Good question. I suppose I was making an assumption here that triggering a mount would mean at least some RPC's and that might make that "ls -l" stall for a while if we have to talk to a bunch of different servers.

Maybe I'm blowing the performance hit out of proportion though? That said, I'm not crazy about altering the generic VFS to fix this. I wonder if there's another way to do it?

Signature · 1 line
Jeff Layton <[email protected]>