NFS fscache offline mode?
Subsystems (2)
Does the new NFS fscache support offline mode? That is, can the client continue to serve cached files even in the absence of any service communication at all?
I see that the fscache itself can do this, but it also seems to require the netfs (NFS) to invoke it with "pinning" operations. I don't see any pinning calls, or options to request the behavior in the NFS code currently.
I actually have one other semi-related question. In fscache-index.c, the fscache generates keys to match servers by computing an {NFS version, transport protocol, port, IP address} tuple. Have you given thought to how this might work with NFSv4.1 sessions? With 4.1, the session allows trunking and reconnection to multiple server addresses. It appears the cache basically won't hit on such configurations. I think the nfs_server_key structure will require more thought for 4.1.
Tom.
Quoted 3 lines · Tom Talpey
Does the new NFS fscache support offline mode? That is, can the client continue to serve cached files even in the absence of any service communication at all?
That is not yet supported. It's quite a tricky problem as the netfs (NFS in this case) has to do conflict resolution when the server is contacted once again, and that might require user interaction.
Quoted 4 lines
I see that the fscache itself can do this, but it also seems to require the netfs (NFS) to invoke it with "pinning" operations. I don't see any pinning calls, or options to request the behavior in the NFS code currently.
Pinning and reservations are not yet implemented in the cache. I've been holding off on implementing them on the basis that until the code gets upstream, I have to expect that I might have to make substantial alterations to the code I do have.
In any case, offline mode isn't something that FS-Cache itself cares about. It is purely a data store. The network filesystems using it must implement offline mode.
Quoted 7 lines
I actually have one other semi-related question. In
fscache-index.c, the fscache generates keys to match servers by computing an {NFS version, transport protocol, port, IP address} tuple. Have you given thought to how this might work with NFSv4.1 sessions? With 4.1, the session allows trunking and reconnection to multiple server addresses. It appears the cache basically won't hit on such configurations. I think thenfs_server_keystructure will require more thought for 4.1.
I've been thinking for a while about how to map multiple server addresses onto one cache for servers that all serve the same data. It's not clear how to do it, since, as far as I know, there's no way to automatically work out that two servers should be treated as being the same.
With AFS it's easier, since the volumes are defined by the cell they're in, not by the servers that are serving them. As far as I know NFS doesn't do things that way.
One thing I have wondered about is sticking aliases in the cache (symlinks or whatever) when an NFS mount is made that has a list of server addresses. However, this assumes:
(1) The file handles on one server match those of another server serving the same set of files.
(2) If two servers are serving the same data, then overlapping exports match completely.
I feel there's another problem with aliases like this in the case of the address a server that's been added as an alias being reused to server different data. What I think I need is consistency data from the server about the server, so that I can tell that the server is no longer what I thought it was.
In fact, it might be possible to hide the fact that there are aliases from FS-Cache entirely. Say that the NFS client is given a list of IP addresses that are all serving the same data. It then uses the lowest IP address it is given as the key to FS-Cache, no matter which server it is actually talking to.
David
Quoted 12 lines
-----Original Message-----
From: David Howells [mailto:[email protected]]
Sent: Thursday, April 09, 2009 4:15 AM
To: Tom Talpey
Cc: [email protected]; [email protected]
Subject: Re: NFS fscache offline mode?Quoted 1 line · Tom Talpey
Does the new NFS fscache support offline mode? That is, can
Quoted 8 lines
the client
Quoted 2 lines
continue to serve cached files even in the absence of any service communication at all?
That is not yet supported. It's quite a tricky problem as the netfs (NFS in this case) has to do conflict resolution when the server is contacted once again, and that might require user interaction.
It's not _so_ tricky for the read-only case, and is quite useful (see papers by Huston+Honeyman re: disconnected AFS). For writes, my recollection is that you pretty much need human intervention to resolve conflicts unless a trivial resolution mechanism is defined.
Quoted 13 lines
Quoted 4 lines
I see that the fscache itself can do this, but it also seems to require the netfs (NFS) to invoke it with "pinning" operations. I don't see any pinning calls, or options to request the
behavior in
Quoted 1 line
the NFS code currently.
Pinning and reservations are not yet implemented in the cache. I've been holding off on implementing them on the basis that until the code gets upstream, I have to expect that I might have to make substantial alterations to the code I do have.
Quoted 4 lines
In any case, offline mode isn't something that FS-Cache itself cares about.
It is purely a data store. The network filesystems using it must implement offline mode.
That's a bit flippant. You could just as well say that the "network" file system should implement its own disk cache. If you're going to cache under the fs, you could also do a lot to hide the disconnected status from the fs, once again, not too bad if you're doing read-only.
BTW: could we not use the term "network" file system when discussing FS-Cache? You've also mentioned using it for iso9660, so it's not just for network file systems.
Quoted 58 lines
Quoted 2 lines
I actually have one other semi-related question. In
fscache-index.c,Quoted 2 lines
the fscache generates keys to match servers by computing an {NFS version, transport protocol, port, IP address} tuple. Have
you given
Quoted 1 line
thought to how this might work with NFSv4.1 sessions? With
4.1, the
Quoted 1 line
session allows trunking and reconnection to multiple server
addresses.
Quoted 2 lines
It appears the cache basically won't hit on such configurations. I think the
nfs_server_keystructure will require morethought for 4.1.
I've been thinking for a while about how to map multiple server addresses onto one cache for servers that all serve the same data. It's not clear how to do it, since, as far as I know, there's no way to automatically work out that two servers should be treated as being the same.
With AFS it's easier, since the volumes are defined by the cell they're in, not by the servers that are serving them. As far as I know NFS doesn't do things that way.
One thing I have wondered about is sticking aliases in the cache (symlinks or
whatever) when an NFS mount is made that has a list of server addresses.
However, this assumes:(1) The file handles on one server match those of another
server serving the
same set of files.(2) If two servers are serving the same data, then
overlapping exports match
completely.I feel there's another problem with aliases like this in the case of the address a server that's been added as an alias being reused to server different data. What I think I need is consistency data from the server about the server, so that I can tell that the server is no longer what I thought it was.
In fact, it might be possible to hide the fact that there are aliases from FS-Cache entirely. Say that the NFS client is given a list of IP addresses that are all serving the same data. It then uses the lowest IP address it is given as the key to FS-Cache, no matter which server it is actually talking to.
David
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · Daniel
It's not _so_ tricky for the read-only case, and is quite useful (see papers by Huston+Honeyman re: disconnected AFS). For writes, my recollection is that you pretty much need human intervention to resolve conflicts unless a trivial resolution mechanism is defined.
Oh, I agree that's it's much simpler for read-only, and that there's no trickiness for files that haven't been modified locally whilst offline; but if any have, it could then be a problem.
The simplest way is to simply discard local changes upon reconnection if the backing file has changed. Isn't that what we do anyway when in connected mode.
Quoted 8 lines
Quoted 3 lines
In any case, offline mode isn't something that FS-Cache itself cares about. It is purely a data store. The network filesystems using it must implement offline mode.
That's a bit flippant. You could just as well say that the "network" file system should implement its own disk cache. If you're going to cache under the fs, you could also do a lot to hide the disconnected status from the fs, once again, not too bad if you're doing read-only.
I disagree. FS-Cache provides or will provide the local storage required to perform offline operation, the ability to pin data within that storage, and the ability to tag the stored data with the metadata required; but in my opinion, the netfs, be it NFS, AFS or CIFS, must do the actual work of:
(1) configuring what must be available for disconnected operation,
(2) pulling files into the cache in preparation,
(3) handling requests from userspace that can't be satisfied when offline, and
(4) resolving conflicts afterwards.
The netfs knows about the structure of the filesystem; fscache does not. The user interacts with the netfs, not directly with fscache.
Quoted 3 lines
BTW: could we not use the term "network" file system when discussing FS-Cache? You've also mentioned using it for iso9660, so it's not just for network file systems.
All my documents, code and emails are laced with the term 'netfs' to mean a client of FS-Cache.
As I have pointed out in my documentation, whilst iso9660 isn't a network filesystem, it can be considered in the same group for this topic.
David
Quoted 17 lines
-----Original Message-----
From: David Howells [mailto:[email protected]]
Sent: Thursday, April 09, 2009 2:22 PM
To: Muntz, Daniel
Cc: [email protected]; Tom Talpey; [email protected]
Subject: Re: NFS fscache offline mode?Quoted 1 line · Daniel
It's not _so_ tricky for the read-only case, and is quite
useful (see
Quoted 3 lines
papers by Huston+Honeyman re: disconnected AFS). For writes, my recollection is that you pretty much need human intervention to resolve conflicts unless a trivial resolution mechanism is defined.
Quoted 7 lines
Oh, I agree that's it's much simpler for read-only, and that there's no trickiness for files that haven't been modified locally whilst offline; but if any have, it could then be a problem.
The simplest way is to simply discard local changes upon reconnection if the backing file has changed. Isn't that what we do anyway when in connected mode.
The two trivial cases are: discard changes from disconnected mode, or propagate all changes when reconnected. Doing something more intelligent is difficult. If writes are allowed, you still run the risk of evicting something you really wanted in the cache (oops, forgot to pin my .bashrc), or getting unexpected disk-full errors. But, you still have something potentially useful if you just make the cache ro when disconnected.
Quoted 15 lines
Quoted 1 line
Quoted 1 line
In any case, offline mode isn't something that FS-Cache
itself cares
Quoted 1 line
Quoted 1 line
about. It is purely a data store. The network
filesystems using it
Quoted 3 lines
Quoted 1 line
must implement offline mode.
That's a bit flippant. You could just as well say that the
"network"
Quoted 1 line
file system should implement its own disk cache. If you're
going to
Quoted 1 line
cache under the fs, you could also do a lot to hide the
disconnected
Quoted 1 line
status from the fs, once again, not too bad if you're doing
read-only.
Quoted 7 lines
I disagree. FS-Cache provides or will provide the local storage required to perform offline operation, the ability to pin data within that storage, and the ability to tag the stored data with the metadata required; but in my opinion, the netfs, be it NFS, AFS or CIFS, must do the actual work of:
(1) configuring what must be available for disconnected operation,
You've captured the data and metadata. I suppose there might be some fs-specific hooks needed (e.g., something needed to prevent the nfs client from thinking the server is down). IMHO, it would be really handy if a canonical list of such hooks could be made, and FS-Cache could handle these similar to how export ops work in NFS--the fs supplies routines that FS-Cache calls to make disconnected operation work.
Quoted 2 lines
(2) pulling files into the cache in preparation,
Some trivial fs-agnostic ways to do this: use the system before disconnecting, or run a process to walk through the fs catting files that you want in disconnected mode (pulling them into the cache). Just out of curiosity, could one 'tar' up the cache on one machine and drop it on another and have it work? There must be enough state on disk so that a reboot doesn't invalidate your cache, so I'm guessing this might be another way to warm your cache.
Quoted 4 lines
(3) handling requests from userspace that can't be satisfied
when offline,
What did you have in mind?
Quoted 3 lines
and
(4) resolving conflicts afterwards.
If you stick to the ro case, or the two simple conflict resolution schemes, then this is not an issue.
Quoted 5 lines
The netfs knows about the structure of the filesystem; fscache does not. The user interacts with the netfs, not directly with fscache.
Quoted 7 lines
Quoted 1 line
BTW: could we not use the term "network" file system when
discussing
Quoted 2 lines
FS-Cache? You've also mentioned using it for iso9660, so it's not just for network file systems.
All my documents, code and emails are laced with the term 'netfs' to mean a client of FS-Cache.
That's what 'sed' is for :-)
Quoted 7 lines
As I have pointed out in my documentation, whilst iso9660 isn't a network filesystem, it can be considered in the same group for this topic.
David
Quoted 9 lines · Daniel
Quoted 1 line
(1) configuring what must be available for disconnected operation,
You've captured the data and metadata. I suppose there might be some fs-specific hooks needed (e.g., something needed to prevent the nfs client from thinking the server is down). IMHO, it would be really handy if a canonical list of such hooks could be made, and FS-Cache could handle these similar to how export ops work in NFS--the fs supplies routines that FS-Cache calls to make disconnected operation work.
No, disconnected operation is a netfs thing, not an FS-Cache thing, I think. Take NFS for example: it must know that it is in disconnected mode; it can't fling GETATTR requests at the server to determine whether a file is valid; it can't ask for a lease on that file; it must rely on FS-Cache to have the data and metadata it requires to make files and directories available.
Similarly, for AFS, it can't go to the server to validate the vnodes it has stored, and it can't request callbacks on the files it opens. In fact both filesystems probably should return their leases/callbacks when entering disconnected mode.
FS-Cache doesn't do anything but store metadata and data NFS gets from the server as it arrives, and retrieve or invalidate that data and metadata when asked by NFS.
For the most part, NFS and AFS operate the following path in any request:
if (in_fscache(data_or_metadata)) {
retrieve_from_fscache(data_or_metadata);
} else {
get_from_server(data_or_metadata);
store_to_fscache(data_or_metadata);
}But with the advent of disconnected mode, that becomes:
if (in_fscache(data_or_metadata)) {
retrieve_from_fscache(data_or_metadata);
} else if (in_disconnected_mode) {
return -EDISCONNECTED;
} else {
get_from_server(data_or_metadata);
store_to_fscache(data_or_metadata);
}If FS-Cache doesn't have a file, directory, page or dirent stored, then the netfs returns EDISCONNECTED or whatever for that item.
Quoted 5 lines
Quoted 1 line
(2) pulling files into the cache in preparation,
Some trivial fs-agnostic ways to do this: use the system before disconnecting, or run a process to walk through the fs catting files that you want in disconnected mode (pulling them into the cache).
Yeah, a userspace solution would be simplest here:
[/usr/sbin/use-in-disconnected-mode] #!/bin/sh tar cf - $* >/dev/zero
However, we'd want to pin all those files, so using tar isn't necessarily the best. Pinning could be achieved by opening each file, emitting an fadvise() or ioctl() to say that this should be pinned, and then reading the first byte of each page in the file.
Quoted 4 lines
Just out of curiosity, could one 'tar' up the cache on one machine and drop it on another and have it work? There must be enough state on disk so that a reboot doesn't invalidate your cache, so I'm guessing this might be another way to warm your cache.
Perhaps. You'd have to supply --xattr to tar, but you could do it.
Quoted 4 lines
Quoted 2 lines
(3) handling requests from userspace that can't be satisfied
when offline,What did you have in mind?
Someone tries to access a file that isn't in the cache, say. FS-Cache says 'no I haven't got that' and then the netfs needs to do something with it - even if that's just returning FS-Cache's error (-ENODATA) to the caller.
Quoted 6 lines
Quoted 3 lines
and
(4) resolving conflicts afterwards.
If you stick to the ro case, or the two simple conflict resolution schemes, then this is not an issue.
Mostly true. But it's still dealt with by the netfs.
For the discard-on-conflict option, with NFS and AFS it'd be a case of compare an inode that's in core (if the inode is in use) or in the cache next time it is accessed with what's on the server - as is done in connected operation.
For the writeback-on-conflict option, it's not a non-issue. The netfs would have to write back what's in the pagecache or in FS-Cache, and that's something we don't do now.
David