State of NFSv4 VolatileFilehandles
Subsystems (2)
We at IBM receiving multiple customer requests for supporting NFSv4 server migration. I have referred to Trond and Chuck's presentations at 2011 Connectathon and there appears to be sizable work remaining. I am not sure if there is any progress made since those talks.
I would like to open up a discussion thread on the mailing list to understand the latest status. Also would like to get the input on the Volatile Filehandles (VFH). I searched the mailing list, and could not find any recent discussion on this.
Some discussion points:
- What are the pieces left to attain full client/server support for seamless server migration?
- Any discussion/sugestions on the way to implement VFH? As described in RFC 3530 sections 4.2.3 and 4.2.4?
- Are there any community efforts going / about to start in this area? so that we can partner and get things done instead of duplicating the work.
Thanks a lot for your help
JV
Quoted 10 lines · Venkateswararao Jujjuri
We at IBM receiving multiple customer requests for supporting NFSv4 server migration. I have referred to Trond and Chuck's presentations at 2011 Connectathon and there appears to be sizable work remaining. I am not sure if there is any progress made since those talks.
I would like to open up a discussion thread on the mailing list to understand the latest status. Also would like to get the input on the Volatile Filehandles (VFH). I searched the mailing list, and could not find any recent discussion on this.
Some discussion points:
- What are the pieces left to attain full client/server support for seamless server migration?
The client migration implementation is code complete and in test now. This includes both minor version 0 and 1. We don't have any mv1 servers to test with at this time, so that support is provisional. I hope to have patches ready for the 3.2 merge window, but you can see what I've got now on git.linux-nfs.org.
A problem is that there are corner cases in the v4.0 migration specification that are still unresolved. We are working with the NFSv4 WG to get these addressed. But I expect some minor changes even after the patches are merged upstream.
We don't have firm plans for a server migration implementation on Linux at this time, but Bruce can maybe say more about that.
Quoted 1 line
- Any discussion/sugestions on the way to implement VFH? As described in RFC 3530 sections 4.2.3 and 4.2.4?
I think we are avoiding volatile file handles as long as possible. We don't have plans to implement them at the moment.
Quoted 5 lines
- Are there any community efforts going / about to start in this area? so that we can partner and get things done instead of duplicating the work.
Thanks a lot for your help
JV
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Quoted 6 lines · Chuck Lever
Quoted 4 lines · Venkateswararao Jujjuri
We at IBM receiving multiple customer requests for supporting NFSv4 server migration. I have referred to Trond and Chuck's presentations at 2011 Connectathon and there appears to be sizable work remaining. I am not sure if there is any progress made since those talks.
Quoted 7 lines
Quoted 6 lines
I would like to open up a discussion thread on the mailing list to understand the latest status. Also would like to get the input on the Volatile Filehandles (VFH). I searched the mailing list, and could not find any recent discussion on this.
Some discussion points:
- What are the pieces left to attain full client/server support for seamless server migration?The client migration implementation is code complete and in test now. This includes both minor version 0 and 1. We don't have any mv1 servers to test with at this time, so that support is provisional. I hope to have patches ready for the 3.2 merge window, but you can see what I've got now on git.linux-nfs.org.
Great I will take it from there. Is it your branch or Trand's ? Can you please give me which project/branch I should take from git.linux-nfs.org?
Quoted 2 lines
A problem is that there are corner cases in the v4.0 migration specification that are still unresolved. We are working with the NFSv4 WG to get these addressed. But I expect some minor changes even after the patches are merged upstream.
What are those corner cases? Is it on any mailing list? Is it possible for us to see that discussion?
Quoted 2 lines
We don't have firm plans for a server migration implementation on Linux at this time, but Bruce can maybe say more about that.
Sure; would wait for Bruce's views on this. We are getting requirements for both client and server support.
Quoted 2 lines
Quoted 1 line
- Any discussion/sugestions on the way to implement VFH? As described in RFC 3530 sections 4.2.3 and 4.2.4?
I think we are avoiding volatile file handles as long as possible. We don't have plans to implement them at the moment.
Hrm. How can we achieve the complete migration support without volatile filehandle support? What are the reasons for avoiding it? May be we can start looking into this but would like to understand the reasons (if any) for avoiding it.
Thanks a lot for your quick response.
- JV
Quoted 5 lines
Quoted 5 lines
- Are there any community efforts going / about to start in this area? so that we can partner and get things done instead of duplicating the work.
Thanks a lot for your help
JV
Quoted 18 lines
-----Original Message-----
From: Venkateswararao Jujjuri [mailto:[email protected]]
Sent: Wednesday, August 03, 2011 3:28 AM
To: Chuck Lever
Cc: [email protected]; Myklebust, Trond
Subject: Re: State of NFSv4 VolatileFilehandlesQuoted 3 lines · Chuck Lever
Quoted 1 line · Venkateswararao Jujjuri
We at IBM receiving multiple customer requests for supporting NFSv4
server migration.
Quoted 1 line
Quoted 1 line
I have referred to Trond and Chuck's presentations at 2011
Connectathon and there appears to be
Quoted 1 line
Quoted 1 line
sizable work remaining. I am not sure if there is any progress made
since those talks.
Quoted 2 lines
Quoted 2 lines
I would like to open up a discussion thread on the mailing list to
Quoted 9 lines
understand the latest status.
Quoted 1 line
Quoted 1 line
Also would like to get the input on the Volatile Filehandles (VFH).
I searched the mailing list, and could not
Quoted 4 lines
Quoted 4 lines
find any recent discussion on this.
Some discussion points:
- What are the pieces left to attain full client/server support forseamless server migration?
Quoted 1 line
The client migration implementation is code complete and in test
now.
Quoted 5 lines
This includes both minor version 0 and 1. We don't have any mv1 servers to test with at this time, so that support is provisional. I hope to have patches ready for the 3.2 merge window, but you can see what I've got now on git.linux-nfs.org.
Great I will take it from there. Is it your branch or Trand's ? Can
you
Quoted 5 lines
please give me which project/branch
I should take from git.linux-nfs.org?Quoted 2 lines
A problem is that there are corner cases in the v4.0 migration
specification that are still unresolved. We are working with the
NFSv4
Quoted 8 lines
WG to get these addressed. But I expect some minor changes even after the patches are merged upstream.
What are those corner cases? Is it on any mailing list? Is it possible for us to see that discussion?Quoted 2 lines
We don't have firm plans for a server migration implementation on
Linux at this time, but Bruce can maybe say more about that. Sure; would wait for Bruce's views on this. We are getting
requirements
Quoted 7 lines
for both client and server support.
Quoted 1 line
Quoted 1 line
- Any discussion/sugestions on the way to implement VFH? As
described in RFC 3530 sections 4.2.3 and 4.2.4?
Quoted 1 line
I think we are avoiding volatile file handles as long as possible.
We don't have plans to implement them at the moment. Hrm. How can we achieve the complete migration support without
volatile
Quoted 4 lines
filehandle support?
What are the reasons for avoiding it? May be we can start looking into this but would like to understand
the reasons (if any) for avoiding it.
POSIX allows the namespace to change at any time (rename() or unlink()) and so you cannot rely on addressing files by pathname. That was the whole reason for introducing filehandles into NFSv2 in the first place.
Volatile filehandles were introduced in NFSv4 without any attempt to fix those shortcomings. There is no real prescription for how to recover in a situation where a rename or unlink has occurred prior to the filehandle expiring. Nor is there a reliable prescription for dealing with the case where a new file of the same name has replaced the original.
Basically, the implication is that volatile filehandles are only really usable in a situation where the whole Filesystem is read-only on the server.
Cheers
Trond
Quoted 15 lines · Trond
Quoted 5 lines
Quoted 1 line
Quoted 1 line
- Any discussion/sugestions on the way to implement VFH? As
described in RFC 3530 sections 4.2.3 and 4.2.4?
Quoted 1 line
I think we are avoiding volatile file handles as long as possible.
We don't have plans to implement them at the moment. Hrm. How can we achieve the complete migration support without
volatile
Quoted 4 lines
filehandle support?
What are the reasons for avoiding it? May be we can start looking into this but would like to understand
the reasons (if any) for avoiding it.POSIX allows the namespace to change at any time (
rename()orunlink()) and so you cannot rely on addressing files by pathname. That was the whole reason for introducing filehandles into NFSv2 in the first place.
Quoted 9 lines
Volatile filehandles were introduced in NFSv4 without any attempt to fix those shortcomings. There is no real prescription for how to recover in a situation where a rename or unlink has occurred prior to the filehandle expiring. Nor is there a reliable prescription for dealing with the case where a new file of the same name has replaced the original.
Basically, the implication is that volatile filehandles are only really usable in a situation where the whole Filesystem is read-only on the server.
I substantially agree, though I think the implication can be refined a little.
I would say that the implication is that a VFH is only really usable when the
complete path leading to the file in question is read-only. We don't need
to assume that other files in other parts of the hierarchy which have stable file handles are read-only.
So if the server presents us with a VFH, it seems reasonable to assume that we can use a repeated lookup of the same name to refresh the filehandle simply because there is no other credible way to respond to a FHEXPIRED.
So while the spec doesn't explicitly say that an expired VFH can be expected to never be renamed, it does - as you say - strongly imply that so it seems reasonable to proceed with implementation on that basis...
Is that convincing?
NeilBrown
Quoted 9 lines · NeilBrown [[email protected]] wrote:
I substantially agree, though I think the implication can be refined a little.
I would say that the implication is that a VFH is only really usable when the complete path leading to the file in question is read-only. We don't need to assume that other files in other parts of the hierarchy which have stable file handles are read-only.
Quoted 4 lines
So if the server presents us with a VFH, it seems reasonable to assume that we can use a repeated lookup of the same name to refresh the filehandle simply because there is no other credible way to respond to a FHEXPIRED.
The spec seems to imply that repeated lookup of the same name to refresh the file handle is OK as long as the file is OPEN! It doesn't seem to imply anything for files that are not opened.
"RFC 3530, 4.2.3. Volatile Filehandle" states:
"Servers which provide volatile filehandles that may expire while open (i.e., if FH4_VOL_MIGRATION or FH4_VOL_RENAME is set or if FH4_VOLATILE_ANY is set and FH4_NOEXPIRE_WITH_OPEN not set), should deny a RENAME or REMOVE that would affect an OPEN file of any of the components leading to the OPEN file. In addition, the server should deny all RENAME or REMOVE requests during the grace period upon server restart."
On the other hand, if FH4_NOEXPIRE_WITH_OPEN is set, then the file can be allowed to be renamed or removed by the server.
Thanks, Malahal.
Quoted 10 lines · Malahal Naineni
Quoted 9 lines · NeilBrown [[email protected]] wrote:
I substantially agree, though I think the implication can be refined a little.
I would say that the implication is that a VFH is only really usable when the complete path leading to the file in question is read-only. We don't need to assume that other files in other parts of the hierarchy which have stable file handles are read-only.
Quoted 8 lines
Quoted 4 lines
So if the server presents us with a VFH, it seems reasonable to assume that we can use a repeated lookup of the same name to refresh the filehandle simply because there is no other credible way to respond to a FHEXPIRED.
The spec seems to imply that repeated lookup of the same name to refresh the file handle is OK as long as the file is OPEN! It doesn't seem to imply anything for files that are not opened.
I see no such implication for a migration situation, unless you also migrate the open state.
Even if you do, then you still have to somehow recover directory filehandles for the current directory and any other RPC call that happened to be in progress when the original server was migrated.
Quoted 10 lines
"RFC 3530, 4.2.3. Volatile Filehandle" states:
"Servers which provide volatile filehandles that may expire while open (i.e., if
FH4_VOL_MIGRATIONorFH4_VOL_RENAMEis set or ifFH4_VOLATILE_ANYis set andFH4_NOEXPIRE_WITH_OPENnot set), should deny a RENAME or REMOVE that would affect an OPEN file of any of the components leading to the OPEN file. In addition, the server should deny all RENAME or REMOVE requests during the grace period upon server restart."
Quoted 2 lines
On the other hand, if
FH4_NOEXPIRE_WITH_OPENis set, then the file can be allowed to be renamed or removed by the server.
Which completely violates the expectations of POSIX applications on the client. Any idea how you would work around the above?
Trond
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
Quoted 14 lines · NeilBrown [[email protected]] wrote:
Quoted 4 lines
POSIX allows the namespace to change at any time (
rename()orunlink()) and so you cannot rely on addressing files by pathname. That was the whole reason for introducing filehandles into NFSv2 in the first place.Quoted 9 lines
Volatile filehandles were introduced in NFSv4 without any attempt to fix those shortcomings. There is no real prescription for how to recover in a situation where a rename or unlink has occurred prior to the filehandle expiring. Nor is there a reliable prescription for dealing with the case where a new file of the same name has replaced the original.
Basically, the implication is that volatile filehandles are only really usable in a situation where the whole Filesystem is read-only on the server.
Quoted 6 lines
I substantially agree, though I think the implication can be refined a little.
I would say that the implication is that a VFH is only really usable when the
complete path leading to the file in question is read-only. We don't need
to assume that other files in other parts of the hierarchy which have stable file handles are read-only.
The spec recommends "change" attribute for validating data cache, name cache, etc. Some client implementations use "change" attribute for validating VFH though! Can we use it for validating VFH?
Thanks, Malahal.
Quoted 17 lines · Malahal Naineni
Quoted 16 lines · NeilBrown [[email protected]] wrote:
Quoted 4 lines
POSIX allows the namespace to change at any time (
rename()orunlink()) and so you cannot rely on addressing files by pathname. That was the whole reason for introducing filehandles into NFSv2 in the first place.Quoted 9 lines
Volatile filehandles were introduced in NFSv4 without any attempt to fix those shortcomings. There is no real prescription for how to recover in a situation where a rename or unlink has occurred prior to the filehandle expiring. Nor is there a reliable prescription for dealing with the case where a new file of the same name has replaced the original.
Basically, the implication is that volatile filehandles are only really usable in a situation where the whole Filesystem is read-only on the server.I substantially agree, though I think the implication can be refined a little.
Quoted 8 lines
Quoted 4 lines
I would say that the implication is that a VFH is only really usable when the
complete path leading to the file in question is read-only. We don't need
to assume that other files in other parts of the hierarchy which have stable file handles are read-only.The spec recommends "change" attribute for validating data cache, name cache, etc. Some client implementations use "change" attribute for validating VFH though! Can we use it for validating VFH?
The change attribute can only be used as a heuristic since it is not guaranteed to be a value that is unique to one file.
Trond
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
Quoted 23 lines · Trond Myklebust [[email protected]] wrote:
Quoted 22 lines · Malahal Naineni
Quoted 20 lines · NeilBrown [[email protected]] wrote:
Quoted 4 lines
POSIX allows the namespace to change at any time (
rename()orunlink()) and so you cannot rely on addressing files by pathname. That was the whole reason for introducing filehandles into NFSv2 in the first place.Quoted 9 lines
Volatile filehandles were introduced in NFSv4 without any attempt to fix those shortcomings. There is no real prescription for how to recover in a situation where a rename or unlink has occurred prior to the filehandle expiring. Nor is there a reliable prescription for dealing with the case where a new file of the same name has replaced the original.
Basically, the implication is that volatile filehandles are only really usable in a situation where the whole Filesystem is read-only on the server.I substantially agree, though I think the implication can be refined a little.
I would say that the implication is that a VFH is only really usable when the
complete path leading to the file in question is read-only. We don't need
to assume that other files in other parts of the hierarchy which have stable file handles are read-only.
Quoted 6 lines
Quoted 3 lines
The spec recommends "change" attribute for validating data cache, name cache, etc. Some client implementations use "change" attribute for validating VFH though! Can we use it for validating VFH?
The change attribute can only be used as a heuristic since it is not guaranteed to be a value that is unique to one file.
Agreed, it is a heuristic if we only use the file's "change id". If we want to be very strict, we could potentially use change ids of all the path components in the pathname... OR how about a mount option "use VFH at your own risk"?
Thanks, Malahal.
Quoted 28 lines · On Tue, 16 Aug 2011 08:59:39 -0700 Malahal Naineni <[email protected]> wrote:
Quoted 27 lines · Trond Myklebust [[email protected]] wrote:
Quoted 25 lines · Malahal Naineni
Quoted 14 lines · NeilBrown [[email protected]] wrote:
Quoted 4 lines
POSIX allows the namespace to change at any time (
rename()orunlink()) and so you cannot rely on addressing files by pathname. That was the whole reason for introducing filehandles into NFSv2 in the first place.Quoted 9 lines
Volatile filehandles were introduced in NFSv4 without any attempt to fix those shortcomings. There is no real prescription for how to recover in a situation where a rename or unlink has occurred prior to the filehandle expiring. Nor is there a reliable prescription for dealing with the case where a new file of the same name has replaced the original.
Basically, the implication is that volatile filehandles are only really usable in a situation where the whole Filesystem is read-only on the server.Quoted 6 lines
I substantially agree, though I think the implication can be refined a little.
I would say that the implication is that a VFH is only really usable when the
complete path leading to the file in question is read-only. We don't need
to assume that other files in other parts of the hierarchy which have stable file handles are read-only.The spec recommends "change" attribute for validating data cache, name cache, etc. Some client implementations use "change" attribute for validating VFH though! Can we use it for validating VFH?
Quoted 7 lines
Quoted 2 lines
The change attribute can only be used as a heuristic since it is not guaranteed to be a value that is unique to one file.
Agreed, it is a heuristic if we only use the file's "change id". If we want to be very strict, we could potentially use change ids of all the path components in the pathname... OR how about a mount option "use VFH at your own risk"?
I don't think change-id is really useful even as an heuristic. Not only are they not unique, but they are not guaranteed to be stable either (after all, something might have changed when the file handle expired).
I think the *only* credible response to FHEXPIRED is to re-lookup the same name and as the spec doesn't make any promises about that it is *only* safe to do it with explicit permission through a mount option.
NeilBrown
Quoted 3 lines · Venkateswararao Jujjuri [[email protected]] wrote:
Quoted 1 line
I think we are avoiding volatile file handles as long as possible. We don't have plans to implement them at the moment.
Hrm. How can we achieve the complete migration support without volatile filehandle support?
If you can generate persistent file handles that don't include server specific information (maybe a bit hard, but not impossible), migration can be done. Maybe that is what Chuck is talking about.
Quoted 11 lines · Venkateswararao Jujjuri
Quoted 10 lines · Chuck Lever
Quoted 8 lines · Venkateswararao Jujjuri
We at IBM receiving multiple customer requests for supporting NFSv4 server migration. I have referred to Trond and Chuck's presentations at 2011 Connectathon and there appears to be sizable work remaining. I am not sure if there is any progress made since those talks.
I would like to open up a discussion thread on the mailing list to understand the latest status. Also would like to get the input on the Volatile Filehandles (VFH). I searched the mailing list, and could not find any recent discussion on this.
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Some discussion points:
- What are the pieces left to attain full client/server support for seamless server migration?The client migration implementation is code complete and in test now. This includes both minor version 0 and 1. We don't have any mv1 servers to test with at this time, so that support is provisional. I hope to have patches ready for the 3.2 merge window, but you can see what I've got now on git.linux-nfs.org.
Great I will take it from there. Is it your branch or Trand's ? Can you please give me which project/branch I should take from git.linux-nfs.org?
This source code is just for review and experimentation. I would wait for the merged code upstream before basing any work on it, as it needs to be forward ported to 3.1 and I expect there will be some minor architectural changes soon.
git://git.linux-nfs.org/projects/cel/cel-2.6.git
Quoted 2 lines
Quoted 1 line
A problem is that there are corner cases in the v4.0 migration specification that are still unresolved. We are working with the NFSv4 WG to get these addressed. But I expect some minor changes even after the patches are merged upstream.
What are those corner cases? Is it on any mailing list? Is it possible for us to see that discussion?
There has been a lot of face-to-face discussion about this over the past six months or so. We are planning to bring the discussion to the [email protected] mailing list (which is public) very soon.
Quoted 2 lines
Quoted 1 line
We don't have firm plans for a server migration implementation on Linux at this time, but Bruce can maybe say more about that.
Sure; would wait for Bruce's views on this. We are getting requirements for both client and server support.
Are you planning to work with the kernel's NFSD, or with Ganesha?
Quoted 5 lines
Quoted 2 lines
Quoted 1 line
- Any discussion/sugestions on the way to implement VFH? As described in RFC 3530 sections 4.2.3 and 4.2.4?
I think we are avoiding volatile file handles as long as possible. We don't have plans to implement them at the moment.
Hrm. How can we achieve the complete migration support without volatile filehandle support? What are the reasons for avoiding it? May be we can start looking into this but would like to understand the reasons (if any) for avoiding it.
Migration itself does not require volatile file handles (FHs). If you are considering a simple-minded server implementation, like an rsync between heterogenous physical file systems, then yes, volatile FHs may be required. But as Trond points out, volatile FHs are a troubled concept anyway, even without migration in the picture.
Passing a client between two servers that export the same cluster file system would be a simple and common use of migration where file handles don't have to (and probably won't) change. We expect that the most common use cases for migration in the near term are going to involve scenarios with homogenous server OS and physical file systems, where FH format can be controlled and thus preserved across a migration event.
Robust server migration implementations, in other words, will migrate not just data, but also file handles, write verifiers, and NFSv4 state. This allows the greatest transparency for clients, and the smoothest possible migration recovery. Clients can be made simpler if FH recovery is not needed.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Quoted 22 lines · Chuck Lever
Quoted 11 lines · Venkateswararao Jujjuri
Quoted 10 lines · Chuck Lever
Quoted 8 lines · Venkateswararao Jujjuri
We at IBM receiving multiple customer requests for supporting NFSv4 server migration. I have referred to Trond and Chuck's presentations at 2011 Connectathon and there appears to be sizable work remaining. I am not sure if there is any progress made since those talks.
I would like to open up a discussion thread on the mailing list to understand the latest status. Also would like to get the input on the Volatile Filehandles (VFH). I searched the mailing list, and could not find any recent discussion on this.
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Some discussion points:
- What are the pieces left to attain full client/server support for seamless server migration?The client migration implementation is code complete and in test now. This includes both minor version 0 and 1. We don't have any mv1 servers to test with at this time, so that support is provisional. I hope to have patches ready for the 3.2 merge window, but you can see what I've got now on git.linux-nfs.org.
Great I will take it from there. Is it your branch or Trand's ? Can you please give me which project/branch I should take from git.linux-nfs.org?
This source code is just for review and experimentation. I would wait for the merged code upstream before basing any work on it, as it needs to be forward ported to 3.1 and I expect there will be some minor architectural changes soon.
git://git.linux-nfs.org/projects/cel/cel-2.6.git
Quoted 7 lines
Quoted 2 lines
Quoted 1 line
A problem is that there are corner cases in the v4.0 migration specification that are still unresolved. We are working with the NFSv4 WG to get these addressed. But I expect some minor changes even after the patches are merged upstream.
What are those corner cases? Is it on any mailing list? Is it possible for us to see that discussion?
There has been a lot of face-to-face discussion about this over the past six months or so. We are planning to bring the discussion to the [email protected] mailing list (which is public) very soon.
Quoted 2 lines
Quoted 1 line
We don't have firm plans for a server migration implementation on Linux at this time, but Bruce can maybe say more about that.
Sure; would wait for Bruce's views on this. We are getting requirements for both client and server support.
Are you planning to work with the kernel's NFSD, or with Ganesha?
Currently we are looking for kernel NFSD.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
- Any discussion/sugestions on the way to implement VFH? As described in RFC 3530 sections 4.2.3 and 4.2.4?
I think we are avoiding volatile file handles as long as possible. We don't have plans to implement them at the moment.
Hrm. How can we achieve the complete migration support without volatile filehandle support? What are the reasons for avoiding it? May be we can start looking into this but would like to understand the reasons (if any) for avoiding it.
Migration itself does not require volatile file handles (FHs). If you are considering a simple-minded server implementation, like an rsync between heterogenous physical file systems, then yes, volatile FHs may be required. But as Trond points out, volatile FHs are a troubled concept anyway, even without migration in the picture.
One of the usecase is rsync between two physical filesystems; but in this particular use case the export is readonly (rootfs). As trond mentioned Volatile FHs are fine in the case of readonly exports. Is it something we can consider for upstream? VFH only for readonly exports.?
Quoted 4 lines
Passing a client between two servers that export the same cluster file system would be a simple and common use of migration where file handles don't have to (and probably won't) change. We expect that the most common use cases for migration in the near term are going to involve scenarios with homogenous server OS and physical file systems, where FH format can be controlled and thus preserved across a migration event.
Robust server migration implementations, in other words, will migrate not just data, but also file handles, write verifiers, and NFSv4 state. This allows the greatest transparency for clients, and the smoothest possible migration recovery. Clients can be made simpler if FH recovery is not needed.
Yes this totally makes sense.
Thanks,
JV
Quoted 7 lines
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Quoted 6 lines · Venkateswararao Jujjuri
One of the usecase is rsync between two physical filesystems; but in this particular use case the export
is readonly (rootfs). As trond mentioned Volatile FHs are fine in the case of readonly exports.
Is it something we can consider for upstream? VFH only for readonly exports.?
The client has no way of knowing that an export is read only. (Or that the server guarantees the safety of looking up names again in the more general cases Neil describes.) Unless we decide that a server is making an implicit guarantee of that just by exposing volatile filehandles at all. Doesn't sound like the existing spec really says that, though.
If an examination of existing implementations and/or some sort of new spec language could reassure us that servers will only ever expose volatile filehandles when it's safe to do so, then maybe it would make sense for the client to implement volatile filehandle recovery?
But if there's a chance of "unsafe" servers out there, then it would seem like a trap for the unwary user....
Your rootfs's probably aren't terribly large--could you copy around compressed block-level images instead of doing rsync?
--b.
Quoted 8 lines · J. Bruce Fields
Quoted 6 lines · Venkateswararao Jujjuri
One of the usecase is rsync between two physical filesystems; but in this particular use case the export
is readonly (rootfs). As trond mentioned Volatile FHs are fine in the case of readonly exports.
Is it something we can consider for upstream? VFH only for readonly exports.?
Quoted 5 lines
The client has no way of knowing that an export is read only. (Or that the server guarantees the safety of looking up names again in the more general cases Neil describes.) Unless we decide that a server is making an implicit guarantee of that just by exposing volatile filehandles at all. Doesn't sound like the existing spec really says that, though.
NFSv4.1 introduces the 'fs_status' recommended attribute (see section 11.11 in RFC5661), which does, in fact, allow the client to deduce that an export is read-only/won't ever change.
Quoted 10 lines
If an examination of existing implementations and/or some sort of new spec language could reassure us that servers will only ever expose volatile filehandles when it's safe to do so, then maybe it would make sense for the client to implement volatile filehandle recovery?
But if there's a chance of "unsafe" servers out there, then it would seem like a trap for the unwary user....
Your rootfs's probably aren't terribly large--could you copy around compressed block-level images instead of doing rsync?
Agreed.
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
Quoted 15 lines · Trond Myklebust
Quoted 8 lines · J. Bruce Fields
Quoted 6 lines · Venkateswararao Jujjuri
One of the usecase is rsync between two physical filesystems; but in this particular use case the export
is readonly (rootfs). As trond mentioned Volatile FHs are fine in the case of readonly exports.
Is it something we can consider for upstream? VFH only for readonly exports.?Quoted 5 lines
The client has no way of knowing that an export is read only. (Or that the server guarantees the safety of looking up names again in the more general cases Neil describes.) Unless we decide that a server is making an implicit guarantee of that just by exposing volatile filehandles at all. Doesn't sound like the existing spec really says that, though.
Quoted 3 lines
NFSv4.1 introduces the '
fs_status' recommended attribute (see section 11.11 in RFC5661), which does, in fact, allow the client to deduce that an export is read-only/won't ever change.
Oh, neat, I'd forgotten that; you're thinking of STATUS4_FIXED? But I'm not sure it does the job:
STATUS4_FIXED, which indicates a read-only image in the sense that it will never change. The possibility is allowed that, as a result of migration or switch to a different image, changed data can be accessed, but within the confines of this instance, no change is allowed. The client can use this fact to cache aggressively.
OK, so permission to set your attribute cache timeout very high, perhaps, but I don't see why "changed data" couldn't mean changed paths....
--b.
Quoted 12 lines
Quoted 10 lines
If an examination of existing implementations and/or some sort of new spec language could reassure us that servers will only ever expose volatile filehandles when it's safe to do so, then maybe it would make sense for the client to implement volatile filehandle recovery?
But if there's a chance of "unsafe" servers out there, then it would seem like a trap for the unwary user....
Your rootfs's probably aren't terribly large--could you copy around compressed block-level images instead of doing rsync?
Agreed.
Quoted 7 lines
-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Sent: Thursday, August 04, 2011 12:27 PM
To: Myklebust, Trond
Cc: Venkateswararao Jujjuri; Chuck Lever; [email protected]
Subject: Re: State of NFSv4 VolatileFilehandles
Quoted 5 lines
Quoted 2 lines · Trond Myklebust
Quoted 1 line · J. Bruce Fields
On Thu, Aug 04, 2011 at 04:27:33AM -0700, Venkateswararao Jujjuri
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
One of the usecase is rsync between two physical filesystems;
but
Quoted 7 lines
in
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
this particular use case the export
is readonly (rootfs). As trond mentioned Volatile FHs are finein
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the case of readonly exports.
Is it something we can consider for upstream? VFH only forreadonly
Quoted 9 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
exports.?
The client has no way of knowing that an export is read only. (Or
that
Quoted 1 line
Quoted 1 line
the server guarantees the safety of looking up names again in the
more
Quoted 1 line
Quoted 1 line
general cases Neil describes.) Unless we decide that a server is
making
Quoted 1 line
Quoted 1 line
an implicit guarantee of that just by exposing volatile
filehandles
Quoted 5 lines
at
Quoted 1 line
Quoted 1 line
all. Doesn't sound like the existing spec really says that,
though.
Quoted 2 lines
NFSv4.1 introduces the '
fs_status' recommended attribute (see
section
Quoted 15 lines
Quoted 1 line
11.11 in RFC5661), which does, in fact, allow the client to deduce
that
Quoted 1 line
an export is read-only/won't ever change.
Oh, neat, I'd forgotten that; you're thinking of
STATUS4_FIXED? But I'm
not sure it does the job:
STATUS4_FIXED, which indicates a read-only image in the sense that it will never change. The possibility is allowed that, as a result of migration or switch to a different image, changed data can be accessed, but within the confines of this instance, no change is allowed. The client can use this fact to cache aggressively.
Quoted 3 lines
OK, so permission to set your attribute cache timeout very high, perhaps, but I don't see why "changed data" couldn't mean changed paths....
No, but you can presumably use the FSLI4BX_CLSIMUL flag from fs_locations_info in order to find an equivalent replica.
Cheers,
Trond
Quoted 13 lines · Trond
Quoted 13 lines
-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Oh, neat, I'd forgotten that; you're thinking ofSTATUS4_FIXED? But I'm
not sure it does the job:
STATUS4_FIXED, which indicates a read-only image in the sense that it will never change. The possibility is allowed that, as a result of migration or switch to a different image, changed data can be accessed, but within the confines of this instance, no change is allowed. The client can use this fact to cache aggressively.
Quoted 6 lines
Quoted 3 lines
OK, so permission to set your attribute cache timeout very high, perhaps, but I don't see why "changed data" couldn't mean changed paths....
No, but you can presumably use the
FSLI4BX_CLSIMULflag fromfs_locations_infoin order to find an equivalent replica.
I lost you.
Actually my real problem is that I don't understand the description of STATUS4_FIXED. What does "or switch to a different image" mean? Not "migration", or the sentence would have ended before the "or".
I read it as allowing a server admin to replace the filesystem image in place, in which case from the client's point of view this allows the filesystem to change at any time. Which makes the whole thing not terribly useful, except (as the last sentence says) as a caching hint.
--b.
Quoted 23 lines · J. Bruce Fields
Quoted 13 lines · Trond
Quoted 13 lines
-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Oh, neat, I'd forgotten that; you're thinking ofSTATUS4_FIXED? But I'm
not sure it does the job:
STATUS4_FIXED, which indicates a read-only image in the sense that it will never change. The possibility is allowed that, as a result of migration or switch to a different image, changed data can be accessed, but within the confines of this instance, no change is allowed. The client can use this fact to cache aggressively.Quoted 6 lines
Quoted 3 lines
OK, so permission to set your attribute cache timeout very high, perhaps, but I don't see why "changed data" couldn't mean changed paths....
No, but you can presumably use the
FSLI4BX_CLSIMULflag fromfs_locations_infoin order to find an equivalent replica.I lost you.
Quoted 8 lines
Actually my real problem is that I don't understand the description of
STATUS4_FIXED. What does "or switch to a different image" mean? Not "migration", or the sentence would have ended before the "or".I read it as allowing a server admin to replace the filesystem image in place, in which case from the client's point of view this allows the filesystem to change at any time. Which makes the whole thing not terribly useful, except (as the last sentence says) as a caching hint.
If the server admin replaces one filesystem, with a different filesystem, then nothing is going to work anyway. I don't see how that is relevant. That's a case of 'doctor it hurts...'
The bit that _is_ relevant is the 'migration' part, but since the fs_locations_info FSLI4BX_CLSIMUL flag allows you to conclude that replica is an exact replica at all times (i.e. contents are guaranteed to be the same even if filehandles, directory cookies, etc are not) then the STATUS4_FIXED flag does allow you to assume that paths have not changed.
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
Quoted 37 lines · Trond Myklebust
Quoted 23 lines · J. Bruce Fields
Quoted 13 lines · Trond
Quoted 13 lines
-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Oh, neat, I'd forgotten that; you're thinking ofSTATUS4_FIXED? But I'm
not sure it does the job:
STATUS4_FIXED, which indicates a read-only image in the sense that it will never change. The possibility is allowed that, as a result of migration or switch to a different image, changed data can be accessed, but within the confines of this instance, no change is allowed. The client can use this fact to cache aggressively.Quoted 6 lines
Quoted 3 lines
OK, so permission to set your attribute cache timeout very high, perhaps, but I don't see why "changed data" couldn't mean changed paths....
No, but you can presumably use the
FSLI4BX_CLSIMULflag fromfs_locations_infoin order to find an equivalent replica.I lost you.
Quoted 8 lines
Actually my real problem is that I don't understand the description of
STATUS4_FIXED. What does "or switch to a different image" mean? Not "migration", or the sentence would have ended before the "or".I read it as allowing a server admin to replace the filesystem image in place, in which case from the client's point of view this allows the filesystem to change at any time. Which makes the whole thing not terribly useful, except (as the last sentence says) as a caching hint.
If the server admin replaces one filesystem, with a different filesystem, then nothing is going to work anyway. I don't see how that is relevant. That's a case of 'doctor it hurts...'
Quoted 6 lines
The bit that _is_ relevant is the 'migration' part, but since the
fs_locations_infoFSLI4BX_CLSIMULflag allows you to conclude that replica is an exact replica at all times (i.e. contents are guaranteed to be the same even if filehandles, directory cookies, etc are not) then theSTATUS4_FIXEDflag does allow you to assume that paths have not changed.
So you're position is that "or switch to a different image" in the above is redundant, or just a mistake?
--b.
Quoted 45 lines · J. Bruce Fields
Quoted 37 lines · Trond Myklebust
Quoted 23 lines · J. Bruce Fields
Quoted 13 lines · Trond
Quoted 13 lines
-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Oh, neat, I'd forgotten that; you're thinking ofSTATUS4_FIXED? But I'm
not sure it does the job:
STATUS4_FIXED, which indicates a read-only image in the sense that it will never change. The possibility is allowed that, as a result of migration or switch to a different image, changed data can be accessed, but within the confines of this instance, no change is allowed. The client can use this fact to cache aggressively.Quoted 6 lines
Quoted 3 lines
OK, so permission to set your attribute cache timeout very high, perhaps, but I don't see why "changed data" couldn't mean changed paths....
No, but you can presumably use the
FSLI4BX_CLSIMULflag fromfs_locations_infoin order to find an equivalent replica.I lost you.
Quoted 8 lines
Actually my real problem is that I don't understand the description of
STATUS4_FIXED. What does "or switch to a different image" mean? Not "migration", or the sentence would have ended before the "or".I read it as allowing a server admin to replace the filesystem image in place, in which case from the client's point of view this allows the filesystem to change at any time. Which makes the whole thing not terribly useful, except (as the last sentence says) as a caching hint.
If the server admin replaces one filesystem, with a different filesystem, then nothing is going to work anyway. I don't see how that is relevant. That's a case of 'doctor it hurts...'
Quoted 6 lines
The bit that _is_ relevant is the 'migration' part, but since the
fs_locations_infoFSLI4BX_CLSIMULflag allows you to conclude that replica is an exact replica at all times (i.e. contents are guaranteed to be the same even if filehandles, directory cookies, etc are not) then theSTATUS4_FIXEDflag does allow you to assume that paths have not changed.
Quoted 2 lines
So you're position is that "or switch to a different image" in the above is redundant, or just a mistake?
It's redundant: STATUS4_VERSIONED, STATUS4_UPDATED, STATUS4_WRITABLE, and STATUS4_REFERRAL are also subject to the 'or switch to a different image' caveat.
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
Quoted 11 lines · J. Bruce Fields
The client has no way of knowing that an export is read only. (Or that the server guarantees the safety of looking up names again in the more general cases Neil describes.) Unless we decide that a server is making an implicit guarantee of that just by exposing volatile filehandles at all. Doesn't sound like the existing spec really says that, though.
If an examination of existing implementations and/or some sort of new spec language could reassure us that servers will only ever expose volatile filehandles when it's safe to do so, then maybe it would make sense for the client to implement volatile filehandle recovery?
Quoted 5 lines
But if there's a chance of "unsafe" servers out there, then it would seem like a trap for the unwary user....
Your rootfs's probably aren't terribly large--could you copy around compressed block-level images instead of doing rsync?
Another scheme is to disconnect the file handles from the inode number. I implemented this a couple years ago for a customer. Basically add an extended attribute into each inode that contains the nfs file handle, and that handle stays the same independent of the inode number. The added complexity is that you need a new lookup data structure mapping from your nfs handle to something that can be used to find the inode (inode number typically).
Quoted 15 lines · Christoph Hellwig
Quoted 14 lines · J. Bruce Fields
The client has no way of knowing that an export is read only. (Or that the server guarantees the safety of looking up names again in the more general cases Neil describes.) Unless we decide that a server is making an implicit guarantee of that just by exposing volatile filehandles at all. Doesn't sound like the existing spec really says that, though.
If an examination of existing implementations and/or some sort of new spec language could reassure us that servers will only ever expose volatile filehandles when it's safe to do so, then maybe it would make sense for the client to implement volatile filehandle recovery?
But if there's a chance of "unsafe" servers out there, then it would seem like a trap for the unwary user....
Quoted 8 lines
Quoted 2 lines
Your rootfs's probably aren't terribly large--could you copy around compressed block-level images instead of doing rsync?
Another scheme is to disconnect the file handles from the inode number. I implemented this a couple years ago for a customer. Basically add an extended attribute into each inode that contains the nfs file handle, and that handle stays the same independent of the inode number. The added complexity is that you need a new lookup data structure mapping
And that data structure should be persistent--how were you storing it?
Quoted 2 lines
from your nfs handle to something that can be used to find the inode (inode number typically).
Interesting, I've wondered before how well that would work. Any lessons learned?
--b.
Quoted 7 lines · J. Bruce Fields
Quoted 5 lines
Another scheme is to disconnect the file handles from the inode number. I implemented this a couple years ago for a customer. Basically add an extended attribute into each inode that contains the nfs file handle, and that handle stays the same independent of the inode number. The added complexity is that you need a new lookup data structure mapping
And that data structure should be persistent--how were you storing it?
In this case it was in a clustered database in userspace, which I didn't really touch directly. If doing outside of such an appliance I would add a btree to XFS to do the mapping.
Quoted 5 lines
Quoted 2 lines
from your nfs handle to something that can be used to find the inode (inode number typically).
Interesting, I've wondered before how well that would work. Any lessons learned?
Don't store the data in userspace, it just makes life hard :) The basic idea actually was pretty straight forward, and if doing it using the xfs btree I mentioned above fairly easy. Other filesystems might or might not heave similar easily reusable persistant lookup data structures.
Quoted 4 lines · Venkateswararao Jujjuri
Quoted 1 line · Chuck Lever
We don't have firm plans for a server migration implementation on Linux at this time, but Bruce can maybe say more about that.
Sure; would wait for Bruce's views on this. We are getting requirements for both client and server support.
We've been looking at migration and failover, backed by a cluster filesystem, using floating IP's as a way to get most of the benefits without quite as much fiddling with protocol issues and without requiring the absolute latest clients.
We'd likely look into NFSv4 protocol-based migration after that. Is there some reason you require that in particular? Is it only because you want to be able to migrate using rsync and count on the client recovering volatile filehandles?
--b.