Grace period

Quoted 3 lines

Hello, Bruce.
Could you, please, clarify this reason why grace list is used? I.e. why list is used instead of some atomic variable, for example?

Like just a reference count? Yeah, that would be OK.

In theory it could provide some sort of debugging help. (E.g. we could print out the list of "lock managers" currently keeping us in grace.) I had some idea we'd make those lock manager objects more complicated, and might have more for individual containerized services.

07.04.2012 03:40, [email protected] пишет:

Quoted 11 lines · Stanislav Kinsbursky

Quoted 3 lines

Hello, Bruce.
Could you, please, clarify this reason why grace list is used? I.e. why list is used instead of some atomic variable, for example?

Like just a reference count? Yeah, that would be OK.

In theory it could provide some sort of debugging help. (E.g. we could print out the list of "lock managers" currently keeping us in grace.) I had some idea we'd make those lock manager objects more complicated, and might have more for individual containerized services.

Could you share this idea, please?

Anyway, I have nothing against lists. Just was curious, why it was used. I added Trond and lists to this reply.

Let me explain, what is the problem with grace period I'm facing right know, and what I'm thinking about it.
So, one of the things to be containerized during "NFSd per net ns" work is the grace period, and these are the basic components of it:
1) Grace period start.
2) Grace period end.
3) Grace period check.
3) Grace period restart.

So, the simplest straight-forward way is to make all internal stuff:
"grace_list", "grace_lock", "grace_period_end" work and both "lockd_manager" and "nfsd4_manager" - per network namespace. Also, "laundromat_work" have to be per-net as well.
In this case:
1) Start - grace period can be started per net ns in "lockd_up_net()" (thus has to be moves there from "lockd()") and "nfs4_state_start()".
2) End - grace period can be ended per net ns in "lockd_down_net()" (thus has to be moved there from "lockd()"), "nfsd4_end_grace()" and "fs4_state_shutdown()".
3) Check - looks easy. There is either svc_rqst or net context can be passed to function.
4) Restart - this is a tricky place. It would be great to restart grace period only for the networks namespace of the sender of the kill signal. So, the idea is to check siginfo_t for the pid of sender, then try to locate the task, and if found, then get sender's networks namespace, and restart grace period only for this namespace (of course, if lockd was started for this namespace - see below).

If task not found, of it's lockd wasn't started for it's namespace, then grace period can be either restarted for all namespaces, of just silently dropped. This is the place where I'm not sure, how to do. Because calling grace period for all namespaces will be overkill...

There also another problem with the "task by pid" search, that found task can be actually not sender (which died already), but some other new task with the same pid number. In this case, I think, we can just neglect this probability and always assume, that we located sender (if, of course, lockd was started for sender's network namespace).

Trond, Bruce, could you, please, comment this ideas?

Quoted 55 lines · On Mon, 09 Apr 2012 15:24:19 +0400 Stanislav…

07.04.2012 03:40, [email protected] пишет:

Quoted 11 lines · Stanislav Kinsbursky

Quoted 3 lines

Hello, Bruce.
Could you, please, clarify this reason why grace list is used? I.e. why list is used instead of some atomic variable, for example?

Like just a reference count? Yeah, that would be OK.

In theory it could provide some sort of debugging help. (E.g. we could print out the list of "lock managers" currently keeping us in grace.) I had some idea we'd make those lock manager objects more complicated, and might have more for individual containerized services.

Could you share this idea, please?

Anyway, I have nothing against lists. Just was curious, why it was used. I added Trond and lists to this reply.

Let me explain, what is the problem with grace period I'm facing right know, and what I'm thinking about it.
So, one of the things to be containerized during "NFSd per net ns" work is the grace period, and these are the basic components of it:
1) Grace period start.
2) Grace period end.
3) Grace period check.
3) Grace period restart.

So, the simplest straight-forward way is to make all internal stuff:
"grace_list", "grace_lock", "grace_period_end" work and both "lockd_manager" and "nfsd4_manager" - per network namespace. Also, "laundromat_work" have to be per-net as well.
In this case:
1) Start - grace period can be started per net ns in "lockd_up_net()" (thus has to be moves there from "lockd()") and "nfs4_state_start()".
2) End - grace period can be ended per net ns in "lockd_down_net()" (thus has to be moved there from "lockd()"), "nfsd4_end_grace()" and "fs4_state_shutdown()".
3) Check - looks easy. There is either svc_rqst or net context can be passed to function.
4) Restart - this is a tricky place. It would be great to restart grace period only for the networks namespace of the sender of the kill signal. So, the idea is to check siginfo_t for the pid of sender, then try to locate the task, and if found, then get sender's networks namespace, and restart grace period only for this namespace (of course, if lockd was started for this namespace - see below).

If task not found, of it's lockd wasn't started for it's namespace, then grace period can be either restarted for all namespaces, of just silently dropped. This is the place where I'm not sure, how to do. Because calling grace period for all namespaces will be overkill...

There also another problem with the "task by pid" search, that found task can be actually not sender (which died already), but some other new task with the same pid number. In this case, I think, we can just neglect this probability and always assume, that we located sender (if, of course, lockd was started for sender's network namespace).

Quoted 2 lines

Trond, Bruce, could you, please, comment this ideas?

I can comment and I'm not sure that will be sufficient.

The grace period has a particular purpose. It keeps nfsd or lockd from handing out stateful objects (e.g. locks) before clients have an opportunity to reclaim them. Once the grace period expires, there is no more reclaim allowed and "normal" lock and open requests can proceed.

Traditionally, there has been one nfsd or lockd "instance" per host. With that, we were able to get away with a relatively simple-minded approach of a global grace period that's gated on nfsd or lockd's startup and shutdown.

Now, you're looking at making multiple nfsd or lockd "instances". Does it make sense to make this a per-net thing? Here's a particularly problematic case to illustrate what I mean:

Suppose I have a filesystem that's mounted and exported in two different containers. You start up one container and then 60s later, start up the other. The grace period expires in the first container and that nfsd hands out locks that conflict with some that have not been reclaimed yet in the other.

Now, we can just try to say "don't export the same fs from more than one container". But we all know that people will do it anyway, since there's nothing that really stops you from doing so.

What probably makes more sense is making the grace period a per-sb property, and coming up with a set of rules for the fs going into and out of "grace" status.

Perhaps a way for different net namespaces to "subscribe" to a particular fs, and don't take the fs out of grace until all of the grace period timers pop? If a fs attempts to subscribe after the fs comes out of grace, then its subscription would be denied and reclaim attempts would get NFS4ERR_NOGRACE or the NLM equivalent.

09.04.2012 17:47, Jeff Layton пишет:

Quoted 58 lines

Quoted 55 lines · On Mon, 09 Apr 2012 15:24:19 +0400 Stanislav…

07.04.2012 03:40, [email protected] пишет:

Quoted 11 lines · Stanislav Kinsbursky

Quoted 3 lines

Hello, Bruce.
Could you, please, clarify this reason why grace list is used? I.e. why list is used instead of some atomic variable, for example?

Like just a reference count? Yeah, that would be OK.

In theory it could provide some sort of debugging help. (E.g. we could print out the list of "lock managers" currently keeping us in grace.) I had some idea we'd make those lock manager objects more complicated, and might have more for individual containerized services.

Could you share this idea, please?

Anyway, I have nothing against lists. Just was curious, why it was used. I added Trond and lists to this reply.

Let me explain, what is the problem with grace period I'm facing right know, and what I'm thinking about it.
So, one of the things to be containerized during "NFSd per net ns" work is the grace period, and these are the basic components of it:
1) Grace period start.
2) Grace period end.
3) Grace period check.
3) Grace period restart.

So, the simplest straight-forward way is to make all internal stuff:
"grace_list", "grace_lock", "grace_period_end" work and both "lockd_manager" and "nfsd4_manager" - per network namespace. Also, "laundromat_work" have to be per-net as well.
In this case:
1) Start - grace period can be started per net ns in "lockd_up_net()" (thus has to be moves there from "lockd()") and "nfs4_state_start()".
2) End - grace period can be ended per net ns in "lockd_down_net()" (thus has to be moved there from "lockd()"), "nfsd4_end_grace()" and "fs4_state_shutdown()".
3) Check - looks easy. There is either svc_rqst or net context can be passed to function.
4) Restart - this is a tricky place. It would be great to restart grace period only for the networks namespace of the sender of the kill signal. So, the idea is to check siginfo_t for the pid of sender, then try to locate the task, and if found, then get sender's networks namespace, and restart grace period only for this namespace (of course, if lockd was started for this namespace - see below).

If task not found, of it's lockd wasn't started for it's namespace, then grace period can be either restarted for all namespaces, of just silently dropped. This is the place where I'm not sure, how to do. Because calling grace period for all namespaces will be overkill...

There also another problem with the "task by pid" search, that found task can be actually not sender (which died already), but some other new task with the same pid number. In this case, I think, we can just neglect this probability and always assume, that we located sender (if, of course, lockd was started for sender's network namespace).

Quoted 5 lines

Quoted 2 lines

Trond, Bruce, could you, please, comment this ideas?

I can comment and I'm not sure that will be sufficient.

Hi, Jeff. Thanks for the comment.

Quoted 20 lines

The grace period has a particular purpose. It keeps nfsd or lockd from handing out stateful objects (e.g. locks) before clients have an opportunity to reclaim them. Once the grace period expires, there is no more reclaim allowed and "normal" lock and open requests can proceed.

Traditionally, there has been one nfsd or lockd "instance" per host. With that, we were able to get away with a relatively simple-minded approach of a global grace period that's gated on nfsd or lockd's startup and shutdown.

Now, you're looking at making multiple nfsd or lockd "instances". Does it make sense to make this a per-net thing? Here's a particularly problematic case to illustrate what I mean:

Suppose I have a filesystem that's mounted and exported in two different containers. You start up one container and then 60s later, start up the other. The grace period expires in the first container and that nfsd hands out locks that conflict with some that have not been reclaimed yet in the other.

Quoted 4 lines

Now, we can just try to say "don't export the same fs from more than one container". But we all know that people will do it anyway, since there's nothing that really stops you from doing so.

Yes, I see. But situation you described is existent already.
I.e. you can replace containers with the same file system by two nodes, sharing the same distributed file system (like Lustre and GPFS), and you'll experience the same problem in such case.

Quoted 10 lines

What probably makes more sense is making the grace period a per-sb property, and coming up with a set of rules for the fs going into and out of "grace" status.

Perhaps a way for different net namespaces to "subscribe" to a particular fs, and don't take the fs out of grace until all of the grace period timers pop? If a fs attempts to subscribe after the fs comes out of grace, then its subscription would be denied and reclaim attempts would get NFS4ERR_NOGRACE or the NLM equivalent.

This raises another problem. Imagine, that grace period has elapsed for some container and then you start nfsd in another one. New grace period will affect all both of them. And that's even worse from my pow.

If you allow one container to hand out conflicting locks while another container is allowing reclaims, then you can end up with some very difficult to debug silent data corruption. That's the worst possible outcome, IMO. We really need to actively keep people from shooting themselves in the foot here.

One possibility might be to only allow filesystems to be exported from a single container at a time (and allow that to be overridable somehow once we have a working active/active serving solution). With that, you may be able limp along with a per-container grace period handling scheme like you're proposing.

09.04.2012 19:27, Jeff Layton пишет:

Quoted 7 lines

If you allow one container to hand out conflicting locks while another container is allowing reclaims, then you can end up with some very difficult to debug silent data corruption. That's the worst possible outcome, IMO. We really need to actively keep people from shooting themselves in the foot here.

Quoted 6 lines

One possibility might be to only allow filesystems to be exported from a single container at a time (and allow that to be overridable somehow once we have a working active/active serving solution). With that, you may be able limp along with a per-container grace period handling scheme like you're proposing.

Ok then. Keeping people from shooting themselves here sounds reasonable. And I like the idea of exporting a filesystem only from once per network namespace.

Looks like there should be a list of pairs
"exported superblock - network namespace". And if superblock is exported already in other namespace, then export in new namespace have to be skipped (replaced?) with appropriate warning (error?) message shown in log.
Or maybe we even should deny starting of NFS server if one of it's exports is shared already by other NFS server "instance"? But any of these ideas would be easy to implement in RAM, and thus it suits only for containers...

Signature · 2 lines

Best regards,
Stanislav Kinsbursky

Quoted 20 lines · On Mon, 2012-04-09 at 12:11 -0400, [email protected] wrote:

Quoted 18 lines · Stanislav Kinsbursky

09.04.2012 19:27, Jeff Layton пишет:

Quoted 7 lines

If you allow one container to hand out conflicting locks while another container is allowing reclaims, then you can end up with some very difficult to debug silent data corruption. That's the worst possible outcome, IMO. We really need to actively keep people from shooting themselves in the foot here.

Quoted 6 lines

One possibility might be to only allow filesystems to be exported from a single container at a time (and allow that to be overridable somehow once we have a working active/active serving solution). With that, you may be able limp along with a per-container grace period handling scheme like you're proposing.

Ok then. Keeping people from shooting themselves here sounds reasonable. And I like the idea of exporting a filesystem only from once per network namespace.

Quoted 3 lines

Unfortunately that's not going to get us very far, especially not in the v4 case where we've got the common read-only pseudoroot that everyone has to share.

I don't see how that can work in cases where each container has its own private mount namespace. You're going to have to tie that pseudoroot to the mount namespace somehow.

Quoted 30 lines · On Mon, 2012-04-09 at 12:21 -0400, [email protected] wrote:

Quoted 28 lines · Trond

Quoted 16 lines · On Mon, 2012-04-09 at 12:11 -0400, [email protected] wrote:

Quoted 15 lines · Stanislav Kinsbursky

09.04.2012 19:27, Jeff Layton пишет:

Quoted 13 lines

If you allow one container to hand out conflicting locks while another container is allowing reclaims, then you can end up with some very difficult to debug silent data corruption. That's the worst possible outcome, IMO. We really need to actively keep people from shooting themselves in the foot here.

One possibility might be to only allow filesystems to be exported from a single container at a time (and allow that to be overridable somehow once we have a working active/active serving solution). With that, you may be able limp along with a per-container grace period handling scheme like you're proposing.

Quoted 7 lines

Quoted 3 lines

Ok then. Keeping people from shooting themselves here sounds reasonable. And I like the idea of exporting a filesystem only from once per network namespace.

Unfortunately that's not going to get us very far, especially not in the v4 case where we've got the common read-only pseudoroot that everyone has to share.

I don't see how that can work in cases where each container has its own private mount namespace. You're going to have to tie that pseudoroot to the mount namespace somehow.

Quoted 2 lines

Sure, but in typical cases it'll still be shared; requiring that they not be sounds like a severe limitation.

I'd expect the typical case to be the non-shared namespace: the whole point of containers is to provide for complete isolation of processes. Usually that implies that you don't want them to be able to communicate via a shared filesystem.

Quoted 30 lines · Trond

Quoted 28 lines · On Mon, 2012-04-09 at 12:11 -0400, [email protected] wrote:

Quoted 16 lines · Stanislav Kinsbursky

Quoted 15 lines

09.04.2012 19:27, Jeff Layton пишет:

Quoted 13 lines

If you allow one container to hand out conflicting locks while another container is allowing reclaims, then you can end up with some very difficult to debug silent data corruption. That's the worst possible outcome, IMO. We really need to actively keep people from shooting themselves in the foot here.

One possibility might be to only allow filesystems to be exported from a single container at a time (and allow that to be overridable somehow once we have a working active/active serving solution). With that, you may be able limp along with a per-container grace period handling scheme like you're proposing.

Quoted 7 lines

Quoted 3 lines

Ok then. Keeping people from shooting themselves here sounds reasonable. And I like the idea of exporting a filesystem only from once per network namespace.

Unfortunately that's not going to get us very far, especially not in the v4 case where we've got the common read-only pseudoroot that everyone has to share.

I don't see how that can work in cases where each container has its own private mount namespace. You're going to have to tie that pseudoroot to the mount namespace somehow.

Quoted 2 lines

Sure, but in typical cases it'll still be shared; requiring that they not be sounds like a severe limitation.

I'd expect the typical case to be the non-shared namespace: the whole point of containers is to provide for complete isolation of processes. Usually that implies that you don't want them to be able to communicate via a shared filesystem.

09.04.2012 20:33, Myklebust, Trond пишет:

Quoted 31 lines · On Mon, 2012-04-09 at 12:21 -0400, [email protected] wrote:

Quoted 30 lines · Trond

Quoted 28 lines · On Mon, 2012-04-09 at 12:11 -0400, [email protected] wrote:

Quoted 16 lines · Stanislav Kinsbursky

Quoted 15 lines

09.04.2012 19:27, Jeff Layton пишет:

Quoted 13 lines

If you allow one container to hand out conflicting locks while another container is allowing reclaims, then you can end up with some very difficult to debug silent data corruption. That's the worst possible outcome, IMO. We really need to actively keep people from shooting themselves in the foot here.

One possibility might be to only allow filesystems to be exported from a single container at a time (and allow that to be overridable somehow once we have a working active/active serving solution). With that, you may be able limp along with a per-container grace period handling scheme like you're proposing.

Quoted 7 lines

Quoted 3 lines

Ok then. Keeping people from shooting themselves here sounds reasonable. And I like the idea of exporting a filesystem only from once per network namespace.

Unfortunately that's not going to get us very far, especially not in the v4 case where we've got the common read-only pseudoroot that everyone has to share.

I don't see how that can work in cases where each container has its own private mount namespace. You're going to have to tie that pseudoroot to the mount namespace somehow.

Quoted 8 lines

Quoted 2 lines

Sure, but in typical cases it'll still be shared; requiring that they not be sounds like a severe limitation.

I'd expect the typical case to be the non-shared namespace: the whole point of containers is to provide for complete isolation of processes. Usually that implies that you don't want them to be able to communicate via a shared filesystem.

BTW, we DO use one mount namespace for all containers and host in OpenVZ. This allows us to have an access to containers mount points from initial environment. Isolation between containers is done via chroot and some simple tricks on /proc/mounts read operation. Moreover, with one mount namespace, we currently support bind-mounting on NFS from one container into another...

Anyway, I'm sorry, but I'm not familiar with this pseudoroot idea.

Why does it prevents implementing of check for "superblock-network namespace" pair on NFS server start and forbid (?) it in case of this pair is shared already in other namespace? I.e. maybe this pseudoroot can be an exclusion from this rule?

Or I'm just missing the point at all?

Quoted 48 lines

09.04.2012 20:33, Myklebust, Trond пишет:

Quoted 31 lines · On Mon, 2012-04-09 at 12:21 -0400, [email protected] wrote:

Quoted 30 lines · Trond

Quoted 28 lines · On Mon, 2012-04-09 at 12:11 -0400, [email protected] wrote:

Quoted 16 lines · Stanislav Kinsbursky

Quoted 15 lines

09.04.2012 19:27, Jeff Layton пишет:

Quoted 13 lines

If you allow one container to hand out conflicting locks while another container is allowing reclaims, then you can end up with some very difficult to debug silent data corruption. That's the worst possible outcome, IMO. We really need to actively keep people from shooting themselves in the foot here.

One possibility might be to only allow filesystems to be exported from a single container at a time (and allow that to be overridable somehow once we have a working active/active serving solution). With that, you may be able limp along with a per-container grace period handling scheme like you're proposing.

Quoted 7 lines

Quoted 3 lines

Ok then. Keeping people from shooting themselves here sounds reasonable. And I like the idea of exporting a filesystem only from once per network namespace.

Unfortunately that's not going to get us very far, especially not in the v4 case where we've got the common read-only pseudoroot that everyone has to share.

I don't see how that can work in cases where each container has its own private mount namespace. You're going to have to tie that pseudoroot to the mount namespace somehow.

Quoted 8 lines

Quoted 2 lines

Sure, but in typical cases it'll still be shared; requiring that they not be sounds like a severe limitation.

I'd expect the typical case to be the non-shared namespace: the whole point of containers is to provide for complete isolation of processes. Usually that implies that you don't want them to be able to communicate via a shared filesystem.

BTW, we DO use one mount namespace for all containers and host in OpenVZ. This allows us to have an access to containers mount points from initial environment. Isolation between containers is done via chroot and some simple tricks on /proc/mounts read operation. Moreover, with one mount namespace, we currently support bind-mounting on NFS from one container into another...

Quoted 1 line

Anyway, I'm sorry, but I'm not familiar with this pseudoroot idea.

Since NFSv4 doesn't have a separate MOUNT protocol, clients need to be able to do readdir's and lookups to get to exported filesystems. We support this in the Linux server by exporting all the filesystems from "/" on down that must be traversed to reach a given filesystem. These exports are very restricted (e.g. only parents of exports are visible).

Quoted 4 lines

Why does it prevents implementing of check for "superblock-network namespace" pair on NFS server start and forbid (?) it in case of this pair is shared already in other namespace? I.e. maybe this pseudoroot can be an exclusion from this rule?

That might work. It's read-only and consists only of directories, so the grace period doesn't affect it.

09.04.2012 22:11, [email protected] пишет:

Quoted 6 lines

Since NFSv4 doesn't have a separate MOUNT protocol, clients need to be able to do readdir's and lookups to get to exported filesystems. We support this in the Linux server by exporting all the filesystems from "/" on down that must be traversed to reach a given filesystem. These exports are very restricted (e.g. only parents of exports are visible).

Ok, thanks for explanation.
So, this pseudoroot looks like a part of NFS server internal implementation, but not a part of a standard. That's good.

Quoted 5 lines

Quoted 4 lines

Why does it prevents implementing of check for "superblock-network namespace" pair on NFS server start and forbid (?) it in case of this pair is shared already in other namespace? I.e. maybe this pseudoroot can be an exclusion from this rule?

Quoted 3 lines

That might work. It's read-only and consists only of directories, so the grace period doesn't affect it.

I've just realized, that this per-sb grace period won't work. I.e., it's a valid situation, when two or more containers located on the same filesystem, but shares different parts of it. And there is not conflict here at all.

I don't see any clear and simple way how to handle such races, because otherwise we have to tie network namespace and filesystem namespace.
I.e. there will be required some way to define, was passed export directory shared already somewhere else or not.

Realistic solution - since export check should be done in initial file system environment (most probably container will have it's own root), then we to pass this data to some kernel thread/userspace daemon in initial file system environment somehow (sockets doesn't suits here... Shared memory?).

Improbable solution - patching VFS layer...

Signature · 2 lines

Best regards,
Stanislav Kinsbursky

Quoted 21 lines

09.04.2012 22:11, [email protected] пишет:

Quoted 6 lines

Since NFSv4 doesn't have a separate MOUNT protocol, clients need to be able to do readdir's and lookups to get to exported filesystems. We support this in the Linux server by exporting all the filesystems from "/" on down that must be traversed to reach a given filesystem. These exports are very restricted (e.g. only parents of exports are visible).

Ok, thanks for explanation.
So, this pseudoroot looks like a part of NFS server internal implementation, but not a part of a standard. That's good.

Quoted 8 lines

Quoted 4 lines

Why does it prevents implementing of check for "superblock-network namespace" pair on NFS server start and forbid (?) it in case of this pair is shared already in other namespace? I.e. maybe this pseudoroot can be an exclusion from this rule?

That might work. It's read-only and consists only of directories, so the grace period doesn't affect it.

Quoted 4 lines

I've just realized, that this per-sb grace period won't work. I.e., it's a valid situation, when two or more containers located on the same filesystem, but shares different parts of it. And there is not conflict here at all.

Well, there may be some conflict in that a file could be hardlinked into both subtrees, and that file could be locked from users of either export.

10.04.2012 17:39, [email protected] пишет:

Quoted 22 lines · Stanislav Kinsbursky

Quoted 21 lines

09.04.2012 22:11, [email protected] пишет:

Quoted 6 lines

Since NFSv4 doesn't have a separate MOUNT protocol, clients need to be able to do readdir's and lookups to get to exported filesystems. We support this in the Linux server by exporting all the filesystems from "/" on down that must be traversed to reach a given filesystem. These exports are very restricted (e.g. only parents of exports are visible).

Ok, thanks for explanation.
So, this pseudoroot looks like a part of NFS server internal implementation, but not a part of a standard. That's good.

Quoted 8 lines

Quoted 4 lines

Why does it prevents implementing of check for "superblock-network namespace" pair on NFS server start and forbid (?) it in case of this pair is shared already in other namespace? I.e. maybe this pseudoroot can be an exclusion from this rule?

That might work. It's read-only and consists only of directories, so the grace period doesn't affect it.

Quoted 9 lines

Quoted 4 lines

I've just realized, that this per-sb grace period won't work. I.e., it's a valid situation, when two or more containers located on the same filesystem, but shares different parts of it. And there is not conflict here at all.

Well, there may be some conflict in that a file could be hardlinked into both subtrees, and that file could be locked from users of either export.

Is this case handled if both links or visible in the same export? But anyway, this is not that bad. I.e it doesn't make things unpredictable. Probably, there are some more issues like this one (bind-mounting, for example). But I think, that it's root responsibility to handle such problems.

This sort of thing is one of the reasons I've been saying that the grace period is really a property of the underlying filesystem and not of nfsd itself. Of course, we do have to come up with a way to handle the grace period that doesn't involve altering every exportable fs.

Quoted 33 lines · On Tue, 10 Apr 2012 19:36:26 +0400 Stanislav…

10.04.2012 17:39, [email protected] пишет:

Quoted 22 lines · Stanislav Kinsbursky

Quoted 21 lines

09.04.2012 22:11, [email protected] пишет:

Quoted 6 lines

Since NFSv4 doesn't have a separate MOUNT protocol, clients need to be able to do readdir's and lookups to get to exported filesystems. We support this in the Linux server by exporting all the filesystems from "/" on down that must be traversed to reach a given filesystem. These exports are very restricted (e.g. only parents of exports are visible).

Ok, thanks for explanation.
So, this pseudoroot looks like a part of NFS server internal implementation, but not a part of a standard. That's good.

Quoted 8 lines

Quoted 4 lines

Why does it prevents implementing of check for "superblock-network namespace" pair on NFS server start and forbid (?) it in case of this pair is shared already in other namespace? I.e. maybe this pseudoroot can be an exclusion from this rule?

That might work. It's read-only and consists only of directories, so the grace period doesn't affect it.

Quoted 9 lines

Quoted 4 lines

I've just realized, that this per-sb grace period won't work. I.e., it's a valid situation, when two or more containers located on the same filesystem, but shares different parts of it. And there is not conflict here at all.

Well, there may be some conflict in that a file could be hardlinked into both subtrees, and that file could be locked from users of either export.

Quoted 5 lines

Is this case handled if both links or visible in the same export? But anyway, this is not that bad. I.e it doesn't make things unpredictable. Probably, there are some more issues like this one (bind-mounting, for example). But I think, that it's root responsibility to handle such problems.

Well, it's a problem and one that you'll probably have to address to some degree. In truth, the fact that you're exporting different subtrees in different containers is immaterial since they're both on the same fs and filehandles don't carry any info about the path in and of themselves...

Suppose for instance that we have a hardlinked file that's available from two different exports in two different containers. The grace period ends in container #1, so that nfsd starts servicing normal lock requests. An application takes a lock on that hardlinked file. In the meantime, a client of container #2 attempts to reclaim the lock that he previously held on that same inode and gets denied.

That's just one example. The scarier case is that the client of container #1 takes the lock, alters the file and then drops it again with the client of container #2 none the wiser. Now the file got altered while client #2 thought he held a lock on it. That won't be fun to track down...

This sort of thing is one of the reasons I've been saying that the grace period is really a property of the underlying filesystem and not of nfsd itself. Of course, we do have to come up with a way to handle the grace period that doesn't involve altering every exportable fs.

07.04.2012 03:40, [email protected] пишет:

Quoted 11 lines · Stanislav Kinsbursky

Quoted 3 lines

Hello, Bruce.
Could you, please, clarify this reason why grace list is used? I.e. why list is used instead of some atomic variable, for example?

Like just a reference count? Yeah, that would be OK.

In theory it could provide some sort of debugging help. (E.g. we could print out the list of "lock managers" currently keeping us in grace.) I had some idea we'd make those lock manager objects more complicated, and might have more for individual containerized services.

Could you share this idea, please?

Anyway, I have nothing against lists. Just was curious, why it was used. I added Trond and lists to this reply.

Let me explain, what is the problem with grace period I'm facing right know, and what I'm thinking about it.
So, one of the things to be containerized during "NFSd per net ns" work is the grace period, and these are the basic components of it:
1) Grace period start.
2) Grace period end.
3) Grace period check.
3) Grace period restart.

So, the simplest straight-forward way is to make all internal stuff:
"grace_list", "grace_lock", "grace_period_end" work and both "lockd_manager" and "nfsd4_manager" - per network namespace. Also, "laundromat_work" have to be per-net as well.
In this case:
1) Start - grace period can be started per net ns in "lockd_up_net()" (thus has to be moves there from "lockd()") and "nfs4_state_start()".
2) End - grace period can be ended per net ns in "lockd_down_net()" (thus has to be moved there from "lockd()"), "nfsd4_end_grace()" and "fs4_state_shutdown()".

3) Check - looks easy. There is either svc_rqst or net context can be passed to function.
4) Restart - this is a tricky place. It would be great to restart grace period only for the networks namespace of the sender of the kill signal. So, the idea is to check siginfo_t for the pid of sender, then try to locate the task, and if found, then get sender's networks namespace, and restart grace period only for this namespace (of course, if lockd was started for this namespace - see below).

If task not found, of it's lockd wasn't started for it's namespace, then grace period can be either restarted for all namespaces, of just silently dropped. This is the place where I'm not sure, how to do. Because calling grace period for all namespaces will be overkill...

There also another problem with the "task by pid" search, that found task can be actually not sender (which died already), but some other new task with the same pid number. In this case, I think, we can just neglect this probability and always assume, that we located sender (if, of course, lockd was started for sender's network namespace).

Trond, Bruce, could you, please, comment this ideas?

Signature · 6 lines

Best regards,
Stanislav Kinsbursky
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 18 lines

07.04.2012 03:40, [email protected] пишет:

Quoted 11 lines · Stanislav Kinsbursky

Quoted 3 lines

Hello, Bruce.
Could you, please, clarify this reason why grace list is used? I.e. why list is used instead of some atomic variable, for example?

Like just a reference count? Yeah, that would be OK.

In theory it could provide some sort of debugging help. (E.g. we could print out the list of "lock managers" currently keeping us in grace.) I had some idea we'd make those lock manager objects more complicated, and might have more for individual containerized services.

Could you share this idea, please?

Anyway, I have nothing against lists. Just was curious, why it was used. I added Trond and lists to this reply.

Quoted 8 lines

Let me explain, what is the problem with grace period I'm facing right know, and what I'm thinking about it.
So, one of the things to be containerized during "NFSd per net ns" work is the grace period, and these are the basic components of it:
1) Grace period start.
2) Grace period end.
3) Grace period check.
3) Grace period restart.

For restart, you're thinking of the fs/lockd/svc.c:restart_grace() that's called on aisngal in lockd()?

I wonder if there's any way to figure out if that's actually used by anyone? (E.g. by any distro init scripts). It strikes me as possibly impossible to use correctly. Perhaps we could deprecate it....

Quoted 11 lines

So, the simplest straight-forward way is to make all internal stuff:
"grace_list", "grace_lock", "grace_period_end" work and both "lockd_manager" and "nfsd4_manager" - per network namespace. Also, "laundromat_work" have to be per-net as well.
In this case:
1) Start - grace period can be started per net ns in "lockd_up_net()" (thus has to be moves there from "lockd()") and "nfs4_state_start()".
2) End - grace period can be ended per net ns in "lockd_down_net()" (thus has to be moved there from "lockd()"), "nfsd4_end_grace()" and "fs4_state_shutdown()".

Quoted 8 lines

3) Check - looks easy. There is either svc_rqst or net context can be passed to function.
4) Restart - this is a tricky place. It would be great to restart grace period only for the networks namespace of the sender of the kill signal. So, the idea is to check siginfo_t for the pid of sender, then try to locate the task, and if found, then get sender's networks namespace, and restart grace period only for this namespace (of course, if lockd was started for this namespace - see below).

If it's really the signalling that's the problem--perhaps we can get away from the signal-based interface.

At least in the case of lockd I suspect we could.

Or perhaps the decision to share a single lockd thread (or set of nsfd threads) among multiple network namespaces was a poor one. But I realize multithreading lockd doesn't look easy.

10.04.2012 03:26, [email protected] пишет:

Quoted 28 lines · Stanislav Kinsbursky

Quoted 18 lines

07.04.2012 03:40, [email protected] пишет:

Quoted 11 lines · Stanislav Kinsbursky

Quoted 3 lines

Hello, Bruce.
Could you, please, clarify this reason why grace list is used? I.e. why list is used instead of some atomic variable, for example?

Like just a reference count? Yeah, that would be OK.

In theory it could provide some sort of debugging help. (E.g. we could print out the list of "lock managers" currently keeping us in grace.) I had some idea we'd make those lock manager objects more complicated, and might have more for individual containerized services.

Could you share this idea, please?

Anyway, I have nothing against lists. Just was curious, why it was used. I added Trond and lists to this reply.

Quoted 8 lines

Let me explain, what is the problem with grace period I'm facing right know, and what I'm thinking about it.
So, one of the things to be containerized during "NFSd per net ns" work is the grace period, and these are the basic components of it:
1) Grace period start.
2) Grace period end.
3) Grace period check.
3) Grace period restart.

Quoted 7 lines

For restart, you're thinking of the fs/lockd/svc.c:restart_grace() that's called on aisngal in lockd()?

I wonder if there's any way to figure out if that's actually used by anyone? (E.g. by any distro init scripts). It strikes me as possibly impossible to use correctly. Perhaps we could deprecate it....

Or (since lockd kthread is visible only from initial pid namespace) we can just hardcode "init_net" in this case. But it means, that this "kill" logic will be broken if two containers shares one pid namespace, but have separated networks namespaces. Anyway, both (this one or Bruce's) solutions suits me.

Quoted 20 lines

Quoted 19 lines

So, the simplest straight-forward way is to make all internal stuff:
"grace_list", "grace_lock", "grace_period_end" work and both "lockd_manager" and "nfsd4_manager" - per network namespace. Also, "laundromat_work" have to be per-net as well.
In this case:
1) Start - grace period can be started per net ns in "lockd_up_net()" (thus has to be moves there from "lockd()") and "nfs4_state_start()".
2) End - grace period can be ended per net ns in "lockd_down_net()" (thus has to be moved there from "lockd()"), "nfsd4_end_grace()" and "fs4_state_shutdown()".
3) Check - looks easy. There is either svc_rqst or net context can be passed to function.
4) Restart - this is a tricky place. It would be great to restart grace period only for the networks namespace of the sender of the kill signal. So, the idea is to check siginfo_t for the pid of sender, then try to locate the task, and if found, then get sender's networks namespace, and restart grace period only for this namespace (of course, if lockd was started for this namespace - see below).

Quoted 5 lines

If it's really the signalling that's the problem--perhaps we can get away from the signal-based interface.

At least in the case of lockd I suspect we could.

I'm ok with that. So, if no objections will follow, I'll drop it and send the patch. Or you want to do it?

Quoted 63 lines

10.04.2012 03:26, [email protected] пишет:

Quoted 28 lines · Stanislav Kinsbursky

Quoted 18 lines

07.04.2012 03:40, [email protected] пишет:

Quoted 11 lines · Stanislav Kinsbursky

Quoted 3 lines

Hello, Bruce.
Could you, please, clarify this reason why grace list is used? I.e. why list is used instead of some atomic variable, for example?

Like just a reference count? Yeah, that would be OK.

In theory it could provide some sort of debugging help. (E.g. we could print out the list of "lock managers" currently keeping us in grace.) I had some idea we'd make those lock manager objects more complicated, and might have more for individual containerized services.

Could you share this idea, please?

Anyway, I have nothing against lists. Just was curious, why it was used. I added Trond and lists to this reply.

Quoted 8 lines

Let me explain, what is the problem with grace period I'm facing right know, and what I'm thinking about it.
So, one of the things to be containerized during "NFSd per net ns" work is the grace period, and these are the basic components of it:
1) Grace period start.
2) Grace period end.
3) Grace period check.
3) Grace period restart.

Quoted 7 lines

For restart, you're thinking of the fs/lockd/svc.c:restart_grace() that's called on aisngal in lockd()?

I wonder if there's any way to figure out if that's actually used by anyone? (E.g. by any distro init scripts). It strikes me as possibly impossible to use correctly. Perhaps we could deprecate it....

Or (since lockd kthread is visible only from initial pid namespace) we can just hardcode "init_net" in this case. But it means, that this "kill" logic will be broken if two containers shares one pid namespace, but have separated networks namespaces. Anyway, both (this one or Bruce's) solutions suits me.

Quoted 20 lines

Quoted 19 lines

So, the simplest straight-forward way is to make all internal stuff:
"grace_list", "grace_lock", "grace_period_end" work and both "lockd_manager" and "nfsd4_manager" - per network namespace. Also, "laundromat_work" have to be per-net as well.
In this case:
1) Start - grace period can be started per net ns in "lockd_up_net()" (thus has to be moves there from "lockd()") and "nfs4_state_start()".
2) End - grace period can be ended per net ns in "lockd_down_net()" (thus has to be moved there from "lockd()"), "nfsd4_end_grace()" and "fs4_state_shutdown()".
3) Check - looks easy. There is either svc_rqst or net context can be passed to function.
4) Restart - this is a tricky place. It would be great to restart grace period only for the networks namespace of the sender of the kill signal. So, the idea is to check siginfo_t for the pid of sender, then try to locate the task, and if found, then get sender's networks namespace, and restart grace period only for this namespace (of course, if lockd was started for this namespace - see below).

Quoted 8 lines

Quoted 5 lines

If it's really the signalling that's the problem--perhaps we can get away from the signal-based interface.

At least in the case of lockd I suspect we could.

I'm ok with that. So, if no objections will follow, I'll drop it and send the patch. Or you want to do it?

Please do go ahead.

The safest approach might be:
- leave lockd's signal handling there (just accept that it may behave incorrectly in container case), assuming that's safe.
- add a printk ("signalling lockd to restart is deprecated", or something) if it's used.

Then eventually we'll remove it entirely.

(But if that doesn't work, it'd likely also be OK just to remove it completely now.)

Well, I can do this to restart grace only for "init_net" and a printk with your message and information, that it affect only init_net.
Looks good to you?