Skip to contentSkip to conversation

CIFS caching problem

DISCUSSION Steve French 2 participants 2 messages Nov 25, 2003 – Nov 26, 2003 VFS

lore
Share & tools
lore
#0 lore raw
Quoted 2 lines

CIFS doesn't seem to ever invalidate cached pages associated with its inodes. The result is that when a file is modified server-side

It looks like you are right, I am missing a call to invalidate pages.

The CIFS mechanism should be somewhat different than NFSv3 or smbfs which don't really support safe distributed caching and rely on timers (the cifs vfs case probably would be somewhat similar to NFSv4 client
case if it were finished). When a file is opened via SMB/CIFS an
oplock (distributed caching token) is potentially granted (if the server supports it which Samba on most versions of Linux does and also Windows and other CIFS filers usually support). If oplock is not granted no readahead or writebehind should be done on the client (although I should probably add a configuration option for those who want to do caching based on a timer anyway - even at the risk of writes from multiple clients being written out of order on the server such as the case with nfs v3 or smbfs). I don't know whether invalidating cached pages in practice always is possible in Linux (at least for the mmap case) without change to the kernel mm directory or implementing the vmops as XFS does. I keep in the cifs inode flags ClientCanCacheRead and ClientCanCacheAll whether the open of the file granted readahead (level 2 oplock) or readahead & writebehind (total file oplock) permission for distributed caching. If an "oplock_break" is received (see cifs_oplock_thread) then I flush the writebehind data and then depending on whether writebehind permission or all caching permission (readahead & writebehind) in "is_valid_oplock_break" in misc.c I change the cifs inode caching flags and queue a request to the oplock thread to flush the inode (so I don't block the socket reads - holding up other incoming
server responses for unrelated files). Although the read_wrapper
routine (cifs_read_wrapper in cifsfs.c) should work for reading new pages that have not been read yet, presumably the kernel
on the client will continue to use stale ones unless they are pushed out of cache or I explicitly invalidate them. What I didn't do was to call invalidate_cache_pages when total file oplock (writebehind AND readahead permission) is revoked for this file by the server, although I do the reverse and flush writebehind data when the server revokes the oplock.

#1 lore raw
Quoted 4 lines

I don't know whether invalidating cached pages in
practice always is possible in Linux (at least for the mmap case) without change to the kernel mm directory or implementing the vmops as XFS does.

Even with custom versions of the vm_ops routines, you have to modify mm if you want to completely coordinate accesses to the file from multiple systems with CIFS or anything else in an LKM. A couple of new-for-2.6 interfaces make it mostly possible, but last I heard there was still an issue with private mmaps. I believe the function which invalidates all the mmaps hits the private mmaps as well, which means they wipe out private modifications to the file, which means the mmaps aren't as private as they probably should be.

Other than that, I believe it is possible in Linux to invalidate cached pages as necessary to implement CIFS oplocks. I've been studying the issue for IBM's Storage Tank for a few years, and I think we have it working. Storage tank uses data locks very similar to CIFS oplocks.

Bryan Henderson IBM Almaden Research Center
San Jose CA Filesystems