multiple instances of rpc.statd
Subsystems (2)
Hello,
on servers with heartbeat managed resources one rather often has the situation one exports different directories from different resources.
It now may happen all resources are running on one host, but they can also run from different hosts. The situation gets even more complicated if the server is also a nfs client.
In principle having different nfs resources works fine, only the statd state directory is a problem. Or in principle the statd concept at all. Actually we would need to have several instances of statd running using different directories. These then would have to be migrated from one server to the other on resource movement.
However, as far I understand it, there does not even exist the basic concept for this, doesn't it?
Thanks,
Bernd
Signature · 2 lines
Bernd Schubert Q-Leap Networks GmbH
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · Bernd Schubert
Hello,
on servers with heartbeat managed resources one rather often has the situation one exports different directories from different resources.
It now may happen all resources are running on one host, but they can also run from different hosts. The situation gets even more complicated if the server is also a nfs client.
In principle having different nfs resources works fine, only the statd state directory is a problem. Or in principle the statd concept at all. Actually we would need to have several instances of statd running using different directories. These then would have to be migrated from one server to the other on resource movement. However, as far I understand it, there does not even exist the basic concept for this, doesn't it?
The efforts have been attempted (to remedy this issue) and a complete set of patches have been (kept) submitting for the past two years. The patch acceptance progress is very slow (I guess people just don't want to get bothered with cluster issues ?).
Anyway, the kernel side has the basic infrastructure to handle the problem (it stores the incoming clients IP address as part of its book-keeping record) - just a little bit tweak will do the job. However, the user side statd directory needs to get re-structured. I didn't publish the user side directory structure script during my last round of submission. Forking statd into multiple threads do not solve all the issues. Check out:
https://www.redhat.com/archives/cluster-devel/2007-April/msg00028.html
-- Wendy
Hello Wendy.
Quoted 18 lines · On Friday 25 April 2008 15:47:03 Wendy Cheng wrote:
Quoted 9 lines · Bernd Schubert
Hello,
on servers with heartbeat managed resources one rather often has the situation one exports different directories from different resources.
It now may happen all resources are running on one host, but they can also run from different hosts. The situation gets even more complicated if the server is also a nfs client.
Quoted 7 lines
In principle having different nfs resources works fine, only the statd state directory is a problem. Or in principle the statd concept at all. Actually we would need to have several instances of statd running using different directories. These then would have to be migrated from one server to the other on resource movement.
However, as far I understand it, there does not even exist the basic concept for this, doesn't it?
Quoted 4 lines
The efforts have been attempted (to remedy this issue) and a complete set of patches have been (kept) submitting for the past two years. The patch acceptance progress is very slow (I guess people just don't want to get bothered with cluster issues ?).
Well, I think people are just ignorant. I did see your discussions about NLM in the past on the NFS mailing list, but actually I didn't understand the entire point of discussion ;) I was simply used to active-passive services (mostly due to heartbeat-1.x) and there we just had /var/lib/nfs linked to the exported directory.
After I started to work here, I was confronted with the fact we do have working active-active clusters here, but nobody besides me ever cared about the locking problem :( NFS failovers just are done ignoring file locks. Seems so far also nobody run into a problem, but maybe the result was so obscure that nobody ever bothered to complain...
I'm just afraid most admins will simply do like this...
Quoted 9 lines
Anyway, the kernel side has the basic infrastructure to handle the problem (it stores the incoming clients IP address as part of its book-keeping record) - just a little bit tweak will do the job. However, the user side statd directory needs to get re-structured. I didn't publish the user side directory structure script during my last round of submission. Forking statd into multiple threads do not solve all the issues. Check out:
https://www.redhat.com/archives/cluster-devel/2007-April/msg00028.html
Thanks, I will read this!
Thanks again,
Bernd
Signature · 2 lines
Bernd Schubert Q-Leap Networks GmbH
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · Bernd Schubert
Hello Wendy.
Quoted 5 lines · On Friday 25 April 2008 15:47:03 Wendy Cheng wrote:
The efforts have been attempted (to remedy this issue) and a complete set of patches have been (kept) submitting for the past two years. The patch acceptance progress is very slow (I guess people just don't want to get bothered with cluster issues ?).
Quoted 6 lines
Well, I think people are just ignorant. I did see your discussions about NLM in the past on the NFS mailing list, but actually I didn't understand the entire point of discussion ;) I was simply used to active-passive services (mostly due to heartbeat-1.x) and there we just had
/var/lib/nfslinked to the exported directory.After I started to work here, I was confronted with the fact we do have working active-active clusters here, but nobody besides me ever cared about the locking problem :( NFS failovers just are done ignoring file locks. Seems so far also nobody run into a problem, but maybe the result was so obscure that nobody ever bothered to complain...
I'm just afraid most admins will simply do like this...
That's an accurate observation :) .. people are just ignorant until they get bitten by the problem. Then they blush out nasty words about Linux servers and go for proprietary solutions.
There are amazing amount of "workaround"(s) and funny setup(s) to bypass various Linux problems. Admins normally don't care the details but just know if they do certain "tricks", things work. I was looking at a performance issue last week why clustered mail servers ran miserably slow. As a person who don't know much about mail server, I was surprised to learn it is a common practice that linux email servers could be configured to grab flock, followed by posix lock, then wrote a lock file whenever a "write" occurs - all three actions are used concurrently to protect one single file (?). It was a very interesting conversation.
-- Wendy
Quoted 18 lines · Wendy Cheng
Quoted 9 lines · Bernd Schubert
Hello,
on servers with heartbeat managed resources one rather often has the situation one exports different directories from different resources.
It now may happen all resources are running on one host, but they can also run from different hosts. The situation gets even more complicated if the server is also a nfs client.
Quoted 8 lines
In principle having different nfs resources works fine, only the statd state directory is a problem. Or in principle the statd concept at all. Actually we would need to have several instances of statd running using different directories. These then would have to be migrated from one server to the other on resource movement. However, as far I understand it, there does not even exist the basic concept for this, doesn't it?
Quoted 4 lines
The efforts have been attempted (to remedy this issue) and a complete set of patches have been (kept) submitting for the past two years. The patch acceptance progress is very slow (I guess people just don't want to get bothered with cluster issues ?).
We definitely want to get this all figured out....
Quoted 8 lines
Anyway, the kernel side has the basic infrastructure to handle the problem (it stores the incoming clients IP address as part of its book-keeping record) - just a little bit tweak will do the job. However, the user side statd directory needs to get re-structured. I didn't publish the user side directory structure script during my last round of submission. Forking statd into multiple threads do not solve all the issues. Check out:
https://www.redhat.com/archives/cluster-devel/2007-April/msg00028.html
So for basic v2/v3 failover, what remains is some statd -H scripts, and some form of grace period control? Is there anything else we're missing?
--b.
Quoted 20 lines · J. Bruce Fields
Quoted 6 lines · Wendy Cheng
Quoted 4 lines · Bernd Schubert
Hello,
on servers with heartbeat managed resources one rather often has the situation one exports different directories from different resources.
Quoted 8 lines
Quoted 6 lines
It now may happen all resources are running on one host, but they can also run from different hosts. The situation gets even more complicated if the server is also a nfs client.
In principle having different nfs resources works fine, only the statd state directory is a problem. Or in principle the statd concept at all. Actually we would need to have several instances of statd running using different directories. These then would have to be migrated from one server to the other on resource movement. However, as far I understand it, there does not even exist the basic concept for this, doesn't it?
The efforts have been attempted (to remedy this issue) and a complete set of patches have been (kept) submitting for the past two years. The patch acceptance progress is very slow (I guess people just don't want to get bothered with cluster issues ?).
We definitely want to get this all figured out....
Quoted 10 lines
Quoted 3 lines
Anyway, the kernel side has the basic infrastructure to handle the problem (it stores the incoming clients IP address as part of its book-keeping record) - just a little bit tweak will do the job. However, the user side statd directory needs to get re-structured. I didn't publish the user side directory structure script during my last round of submission. Forking statd into multiple threads do not solve all the issues. Check out:
https://www.redhat.com/archives/cluster-devel/2007-April/msg00028.htmlSo for basic v2/v3 failover, what remains is some statd -H scripts, and some form of grace period control? Is there anything else we're missing?
The submitted patch set is reasonably complete ... .
There was another thought about statd patches though - mostly because of the concerns over statd's responsiveness. It depended so much on network status and clients' participations. I was hoping NFS V4 would catch up by the time v2/v3 grace period patches got accepted into mainline kernel. Ideally the v2/v3 lock reclaiming logic could use (or at least did a similar implementation) the communication channel established by v4 servers - that is,
1. Enable grace period as previous submitted patches on secondary server.
2. Drop the locks on primary server (and chained the dropped locks into a lock-list).
3. Send the lock-list via v4 communication channel (or similar implementation) from primary server to backup server.
4. Reclaim the lock base on the lock-list on backup server.
In short, it would be nice to replace the existing statd lock reclaiming logic with the above steps if all possible during active-active failover. For reboot, on the other hand, should stay same as today's statd logic without changes.
-- Wendy
Quoted 59 lines · Wendy Cheng
Quoted 42 lines · J. Bruce Fields
Quoted 21 lines · Wendy Cheng
Quoted 10 lines · Bernd Schubert
Hello,
on servers with heartbeat managed resources one rather often has the situation one exports different directories from different resources.
It now may happen all resources are running on one host, but they can also run from different hosts. The situation gets even more complicated if the server is also a nfs client.
Quoted 9 lines
In principle having different nfs resources works fine, only the statd state directory is a problem. Or in principle the statd concept at all. Actually we would need to have several instances of statd running using different directories. These then would have to be migrated from one server to the other on resource movement. However, as far I understand it, there does not even exist the basic concept for this, doesn't it?
Quoted 5 lines
The efforts have been attempted (to remedy this issue) and a complete
set of patches have been (kept) submitting for the past two years.
The patch acceptance progress is very slow (I guess people just
don't want to get bothered with cluster issues ?).We definitely want to get this all figured out....
Quoted 9 lines
Anyway, the kernel side has the basic infrastructure to handle the problem (it stores the incoming clients IP address as part of its book-keeping record) - just a little bit tweak will do the job. However, the user side statd directory needs to get re-structured. I didn't publish the user side directory structure script during my last round of submission. Forking statd into multiple threads do not solve all the issues. Check out:
https://www.redhat.com/archives/cluster-devel/2007-April/msg00028.htmlQuoted 6 lines
So for basic v2/v3 failover, what remains is some statd -H scripts, and some form of grace period control? Is there anything else we're missing?
The submitted patch set is reasonably complete ... .
There was another thought about statd patches though - mostly because of the concerns over statd's responsiveness. It depended so much on network status and clients' participations. I was hoping NFS V4 would catch up by the time v2/v3 grace period patches got accepted into mainline kernel. Ideally the v2/v3 lock reclaiming logic could use (or at least did a similar implementation) the communication channel established by v4 servers - that is,
Quoted 3 lines
1. Enable grace period as previous submitted patches on secondary server.
2. Drop the locks on primary server (and chained the dropped locks into a lock-list).
What information exactly would be on that lock list?
Quoted 3 lines
3. Send the lock-list via v4 communication channel (or similar implementation) from primary server to backup server.
4. Reclaim the lock base on the lock-list on backup server.
So at this step it's the server itself reclaiming those locks, and you're talking about a completely transparent migration that doesn't look to the client like a reboot?
My feeling has been that that's best done after first making sure we can handle the case where the client reclaims the locks, since the latter is easier, and is likely to involve at least some of the same work. I could be wrong.
Exactly which data has to be transferred from the old server to the new? (Lock types, ranges, fh's, owners, and pid's, for established locks; do we also need to hand off blocking locks? Statd data still needs to be transferred. Ideally rpc reply caches. What else?)
Quoted 4 lines
In short, it would be nice to replace the existing statd lock reclaiming logic with the above steps if all possible during active-active failover. For reboot, on the other hand, should stay same as today's statd logic without changes.
--b.
Quoted 17 lines · J. Bruce Fields
Quoted 15 lines · Wendy Cheng
Quoted 5 lines
So for basic v2/v3 failover, what remains is some statd -H scripts, and some form of grace period control? Is there anything else we're missing?
The submitted patch set is reasonably complete ... .
There was another thought about statd patches though - mostly because of the concerns over statd's responsiveness. It depended so much on network status and clients' participations. I was hoping NFS V4 would catch up by the time v2/v3 grace period patches got accepted into mainline kernel. Ideally the v2/v3 lock reclaiming logic could use (or at least did a similar implementation) the communication channel established by v4 servers - that is,
Quoted 7 lines
Quoted 4 lines
1. Enable grace period as previous submitted patches on secondary server.
2. Drop the locks on primary server (and chained the dropped locks into a lock-list).What information exactly would be on that lock list?
Can't believe I get myself into this ... I'm supposed to be a disk firmware person *now* .. Anyway,
Are the lock state finalized in v4 yet ? Can we borrow the concepts (and saved lock states) from v4 ? We certainly can define the saved state useful for v3 independent of v4, say client IP, file path, lock range, lock type, and user id ? Need to re-read linux source to make sure it is doable though.
Quoted 10 lines
Quoted 5 lines
3. Send the lock-list via v4 communication channel (or similar implementation) from primary server to backup server.
4. Reclaim the lock base on the lock-list on backup server.So at this step it's the server itself reclaiming those locks, and you're talking about a completely transparent migration that doesn't look to the client like a reboot?
Yes, that's the idea .. never implement any prototype code yet - so not sure how feasible it would be.
Quoted 5 lines
My feeling has been that that's best done after first making sure we can handle the case where the client reclaims the locks, since the latter is easier, and is likely to involve at least some of the same work. I could be wrong.
Makes sense .. so the steps taken may be:
1. Push the patch sets that we originally submitted. This is to make sure we have something working.
2. Prototype the new logic, parallel with v4 development, observe and learn the results from step 1 based on user feedbacks.
3. Integrate the new logic, if it turns out to be good.
Quoted 5 lines
Exactly which data has to be transferred from the old server to the new? (Lock types, ranges, fh's, owners, and pid's, for established locks; do we also need to hand off blocking locks? Statd data still needs to be transferred. Ideally rpc reply caches. What else?)
All statd has is the client network addresses (that is already part of current NLM states anyway). Yes, rpc reply cache is important (and that's exactly the motivation for this thread of discussion). Eventually the rpc reply cache needs to get transferred. As long as the communication channel is established, there is no reason for lock states not taking this advantages.
Quoted 6 lines
Quoted 6 lines
In short, it would be nice to replace the existing statd lock reclaiming logic with the above steps if all possible during active-active failover. For reboot, on the other hand, should stay same as today's statd logic without changes.
As mentioned before, cluster issues are not trivial. Take one step at a time .. So the next task we should be focusing may be the grace period patch. Will see what I can do to help out here.
-- Wendy
Quoted 23 lines · Wendy Cheng
Quoted 22 lines · J. Bruce Fields
Quoted 19 lines · Wendy Cheng
Quoted 5 lines
So for basic v2/v3 failover, what remains is some statd -H scripts, and some form of grace period control? Is there anything else we're missing?
The submitted patch set is reasonably complete ... .
There was another thought about statd patches though - mostly because of the concerns over statd's responsiveness. It depended so much on network status and clients' participations. I was hoping NFS V4 would catch up by the time v2/v3 grace period patches got accepted into mainline kernel. Ideally the v2/v3 lock reclaiming logic could use (or at least did a similar implementation) the communication channel established by v4 servers - that is,
1. Enable grace period as previous submitted patches on secondary server.
2. Drop the locks on primary server (and chained the dropped locks into a lock-list).
Quoted 7 lines
Quoted 2 lines
What information exactly would be on that lock list?
Can't believe I get myself into this ... I'm supposed to be a disk firmware person *now* .. Anyway,
Are the lock state finalized in v4 yet ?
You mean, have we figured out what to send across for a transparent migration? Somebody did a prototype that I think we set aside for a while, but I don't recall if it tried to handle truly transparent migration, or whether it just sent across the v4 equivalent of the statd data; I'll check.
--b.
Quoted 59 lines
Can we borrow the concepts (and
saved lock states) from v4 ? We certainly can define the saved state useful for v3 independent of v4, say client IP, file path, lock range, lock type, and user id ? Need to re-read linux source to make sure it is doable though.Quoted 10 lines
Quoted 5 lines
3. Send the lock-list via v4 communication channel (or similar implementation) from primary server to backup server.
4. Reclaim the lock base on the lock-list on backup server.So at this step it's the server itself reclaiming those locks, and you're talking about a completely transparent migration that doesn't look to the client like a reboot?
Yes, that's the idea .. never implement any prototype code yet - so not sure how feasible it would be.
Quoted 5 lines
My feeling has been that that's best done after first making sure we can handle the case where the client reclaims the locks, since the latter is easier, and is likely to involve at least some of the same work. I could be wrong.
Makes sense .. so the steps taken may be:
1. Push the patch sets that we originally submitted. This is to make sure we have something working.
2. Prototype the new logic, parallel with v4 development, observe and learn the results from step 1 based on user feedbacks.
3. Integrate the new logic, if it turns out to be good.Quoted 5 lines
Exactly which data has to be transferred from the old server to the new? (Lock types, ranges, fh's, owners, and pid's, for established locks; do we also need to hand off blocking locks? Statd data still needs to be transferred. Ideally rpc reply caches. What else?)
All statd has is the client network addresses (that is already part of current NLM states anyway). Yes, rpc reply cache is important (and that's exactly the motivation for this thread of discussion). Eventually the rpc reply cache needs to get transferred. As long as the communication channel is established, there is no reason for lock states not taking this advantages.
Quoted 6 lines
Quoted 6 lines
In short, it would be nice to replace the existing statd lock reclaiming logic with the above steps if all possible during active-active failover. For reboot, on the other hand, should stay same as today's statd logic without changes.
As mentioned before, cluster issues are not trivial. Take one step at a time .. So the next task we should be focusing may be the grace period patch. Will see what I can do to help out here.
-- Wendy