Permit filesystem local caching and NFS superblock sharing [try #13]
Subsystems (14)
David Howells
maintainercover Permit filesystem local caching and NFS superblock sharing [try #13]
These patches add local caching for network filesystems such as NFS and AFS.
The patches can be grouped as:
(A) 01-04
Filesystem caching, including support for AFS.
(B) 05
Filesystem caching support for NFS; depends on (A) and upon the superblock sharing patches in Trond's tree.
(C) 06-07
CacheFiles: cache on files backend; depends on (A).
Note to Andrew Morton: I have not included the 64-bit inode number patches, the dentry destruction patches or any NFS superblock sharing fix patches in this patch set.
---
Changes in [try #13]:
(*) [PATCH] FS-Cache: Generic filesystem caching facility
[*] Don't give a warning if the cache backend returns ENOBUFS to
__fscache_acquire_cookie() - that just indicates the cache couldn't find space to store the object.
(*) [PATCH] FS-Cache: Make kAFS use FS-Cache
[*] Don't examine PG_fs_misc if not caching.
(*) [PATCH] NFS: Use local caching
[*] Make nfs_fscache_release_page() compile if not caching (and return
true).
(*) [PATCH] FS-Cache: CacheFiles: A cache that backs onto a mounted filesystem
[*] Clean up format warnings due to 64-bit types vs %llx.
[*] Don't present error obtained in cachefiles_write_page() to netfs
twice; the netfs callback shouldn't be invoked as we can return the error directly.
[*] Handle ENOSPC obtained from the backing filesystem by translating it
into ENOBUFS.
[*] Maintain a quantity of free files/inodes in the cache in addition to a
certain amount of free space. Cull the cache if there are insufficient free files. Permit the limits on that to be configured too.
[*] The cachefilesd package has been updated to v0.6 to fix a number of
bugs and also to document the new options to control the quantity of files kept available. See:
http://people.redhat.com/~dhowells/fscache/
http://people.redhat.com/steved/fscache/
Changes in [try #12]:
(*) [PATCH] FS-Cache: Release page->private after failed readahead
[PATCH] FS-Cache: Make kAFS use FS-Cache
[PATCH] NFS: Use local caching
[*] Use invalidatepage() rather than releasepage() to forcibly invalidate
a page that we failed to add to the pagecache.
[*] Make AFS and NFS's releasepage() ops return false if the page is busy
interacting with the cache rather than waiting for the cache interaction to complete. invalidatepage() still waits.
(*) [PATCH] FS-Cache: CacheFiles: A cache that backs onto a mounted filesystem
[*] Fix a printk format warning.
(*) [PATCH] NFS: Use local caching
[*] Handle upstream changes to base version of nfs_release_page().
Changes in [try #11]:
(*) Split up of the NFS superblock sharing patches into a set of smaller
patches and reworked some of the contents as per Trond's suggestions.
(*) [PATCH] NFS: Fix error handling
[*] Fix error handling in earlier patches (the earlier patches are also in
Trond's NFS tree, so I haven't rolled this in for the moment).
(*) [PATCH] NFS: Secure the roots of the NFS subtrees in a shared superblock
[*] Initialise the security on detached NFS roots manually since they're
allocated with dcache_alloc_anon() not dcache_alloc_root().
(*) [PATCH] FS-Cache: CacheFiles: A cache that backs onto a mounted filesystem
[*] Don't use file structs when accessing the data storage backing files.
Pass NULL as the file argument to prepare_write() and commit_write() calls.
[*] Check for a bmap() inode op to prevent NFS being used as the cache
backing store (and besides, we need bmap() available anyway).
[*] Make the calls to the statfs() superblock op supply a dentry not a
vfsmount.
[*] CONFIG_CACHEFILES_DEBUG permits _enter(), _debug() and _exit() to be
enabled dynamically.
[*] debugging macros are checked by gcc for printf format compliance even
when completely disabled.
(*) [PATCH] FS-Cache: CacheFiles: ia64: missing copy_page export
[*] Export copy_page() on IA-64 as we need that.
(*) [PATCH] AUTOFS: Make sure all dentries refs are released before calling kill_anon_super()
[*] Make sure autofs4 releases all its retained dentries in its kill_sb()
op before calling kill_anon_super() rather than in the put_super() op. This prevents the next patch from oopsing it.
(*) [PATCH] VFS: Destroy the dentries contributed by a superblock on unmounting
[*] Optimise the destruction of the dentries attached to a superblock
during unmounting.
Changes in [try #10]:
(*) [PATCH] NFS: Permit filesystem to perform statfs with a known root dentry
[*] Pass a dentry rather than a vfsmount to the statfs() op as the key by
which to determine the filesystem.
(*) [PATCH] NFS: Share NFS superblocks per-protocol per-server per-FSID
[*] nfs4_pathname_string() needed an extra const.
(*) [PATCH] FS-Cache: Release page->private in failed readahead
[*] The comment header on the helper function is much expanded. This
states why there's a need to call the releasepage() op in the event of an error.
[*] BUG() if the page is already locked when we try and lock it.
[*] Don't set the page mapping pointer until we've locked the page.
[*] The page is unlocked after try_to_release_page() is called.
(*) The release-page patch now comes before the fscache-afs patch as well as
the fscache-nfs patch.
Changes in [try #9]:
(*) [PATCH] NFS: Permit filesystem to perform statfs with a known root dentry
[*] Inclusions of linux/mount.h have been added where necessary to make
allyesconfig build successfully.
(*) [PATCH] NFS: Share NFS superblocks per-protocol per-server per-FSID
[*] The exports from fs/namespace.c and fs/namei.c are no longer required.
(*) [PATCH] FS-Cache: Release page->private in failed readahead
[*] The try_to_release_page() is called instead of calling the
releasepage() op directly.
[*] The page is locked before try_to_release_page() is called.
[*] The call to try_to_release_page() and page_cache_release() have been
abstracted out into a helper function as this bit of code occurs twice..
---
In response to those who've asked, there are at least three reasons for implementing superblock sharing:
(1) As I understand what I've been told, NFSv4 requires a 1:1 mapping between server files and client files. I suspect this has to do with the management of leases.
(2) We can reduce the resource consumption on NFSv2 and NFSv3 clients as well as on NFSv4 clients by sharing superblocks that cover overlapping segments of the file space.
Consider a machine that's used by a lot of people at the same time, each of whom has an automounted NFS homedir off of the same server - and in fact off of the same disk on the that server. Currently, with Linus's tree, each one will get a separate superblock to represent them; with Trond's tree, each one will still get a separate superblock unless they share the same root filehandle; and with my patches, they'll get the same superblock.
If two homedirs have a hard link between them (unlikely, I know, but by no means impossible, and probably more likely with, say, data such as NFS mounted git repositories), then you have the possibility of aliasing. This means that you can have two or more inodes in core that refer to the same server object, and each of these inodes can have pages that refer to the same remote pages on the server - aliasing again. You _have_ to have two inodes because they're covered by separate superblocks.
Aliasing is bad, generally, because you end up using more storage than you need to (pagecache and inode cache in this case), and you have the problem of keeping them in sync. It's also twice as hard to keep two inodes up to date when they change on the server as to keep one up to date.
If you can use the same superblock where possible, then you can cut out aliasing on that client since you can share dentries that have the same file handle (hard links or subtrees).
Part of the problem with NFSv2 and NFSv3 is that you invoke mountd to get the filehandle to a subtree, but you may not be able to work out how two different subtrees relate. The getsb patch permits the superblock to have more than one root, which allows us to defer this problem until we see the root of one subtree cropping up in another subtree - at which point we can splice the former into the latter.
(3) In my local file caching patches (FS-Cache), I have two reasons for wanting this:
(a) Unique keys. I need a unique key to find an object in the cache. If we can get inode aliases, then I end up with several inodes referring to the same cache object. This also means that I have to use a fair bit of extra memory to keep track of the multiple cookie mappings in FS-Cache, and have to compare keys a lot to find duplicate mappings.
If I can assume that the _netfs_ will manage the 1:1 mapping, I can use a lot less memory and save some processing capacity also.
I don't want to invent random keys to differentiate aliased superblocks or inodes as that destroys the persistence capabilities of the cache across power failures and reboots.
(b) Callbacks. I want a callback that the netfs passes to FS-Cache to permit the cache to update the metadata in the cache from netfs metadata at convenient times. However, if there's more than one inode alias in core, which one should the cache use?
AFS doesn't have anything like these problems because mounts are always made from the root of a volume, and AFS was designed with local caching in mind.
The getsb and statfs patches are a consequence of NFS being permitted to mount arbitrary subtrees from the server.
David
David Howells
maintainerPATCH 1/7 FS-Cache: Provide a filesystem-specific sync'able page bit [try #13]
The attached patch provides a filesystem-specific page bit that a filesystem can synchronise upon. This can be used, for example, by a netfs to synchronise with CacheFS writing its pages to disk.
The PG_checked bit is replaced with PG_fs_misc, and various operations are provided based upon that. The *PageChecked() macros have also been replaced.
---
fs/afs/dir.c | 5 +---- fs/ext2/dir.c | 6 +++--- fs/ext3/inode.c | 10 +++++----- fs/freevxfs/vxfs_subr.c | 2 +- fs/reiserfs/inode.c | 10 +++++----- fs/ufs/dir.c | 6 +++--- include/linux/page-flags.h | 15 ++++++++++----- include/linux/pagemap.h | 11 +++++++++++ mm/filemap.c | 17 +++++++++++++++++ mm/migrate.c | 4 ++-- mm/page_alloc.c | 2 +- 11 files changed, 59 insertions(+), 29 deletions(-)
fs/afs/dir.c +1 -4
@@ -155,11 +155,9 @@ #endif } } -SetPageChecked (page);return ; error: -SetPageChecked (page);SetPageError (page); }/* end afs_dir_check_page() */
@@ -191,8 +189,7 @@ static struct page *afs_dir_get_page(strkmap (page);if (!PageUptodate (page)) goto fail; -if (!PageChecked (page)) -afs_dir_check_page (dir, page); +afs_dir_check_page (dir, page);if (PageError (page)) goto fail; }
fs/ext2/dir.c +3 -3
@@ -112,7 +112,7 @@ static void ext2_check_page(struct pageif (offs != limit) goto Eend; out: -SetPageChecked (page); +SetPageFsMisc (page);return ;/* Too bad, we had an error */
@@ -152,7 +152,7 @@ Eend: dir->i_ino, (page->index<<PAGE_CACHE_SHIFT )+offs, (unsigned long) le32_to_cpu(p->inode)); fail: -SetPageChecked (page); +SetPageFsMisc (page);SetPageError (page); }
@@ -165,7 +165,7 @@ static struct page * ext2_get_page(struckmap (page);if (!PageUptodate (page)) goto fail; -if (!PageChecked (page)) +if (!PageFsMisc (page))ext2_check_page (page);if (PageError (page)) goto fail;
fs/ext3/inode.c +5 -5
@@ -1527,12 +1527,12 @@ static int ext3_journalled_writepage(str goto no_write; } -if (!page_has_buffers (page) ||PageChecked (page)) { +if (!page_has_buffers (page) ||PageFsMisc (page)) {/* * It's mmapped pagecache. Add buffers and journal it. There * doesn't seem much point in redirtying the page here. */ -ClearPageChecked (page); +ClearPageFsMisc (page); ret =block_prepare_write (page,0 ,PAGE_CACHE_SIZE , ext3_get_block);if (ret !=0 ) {
@@ -1589,7 +1589,7 @@ static void ext3_invalidatepage(struct p* If it's a full truncate we just forget about the pending dirtying */ if (offset ==0 ) -ClearPageChecked (page); +ClearPageFsMisc (page);journal_invalidatepage (journal, page, offset); }
@@ -1598,7 +1598,7 @@ static int ext3_releasepage(struct page {journal_t *journal =EXT3_JOURNAL (page->mapping->host); -WARN_ON (PageChecked (page)); +WARN_ON (PageFsMisc (page));if (!page_has_buffers (page))return 0 ;return journal_try_to_free_buffers (journal, page, wait);
@@ -1694,7 +1694,7 @@ out:*/ static int ext3_journalled_set_page_dirty (struct page *page) { -SetPageChecked (page); +SetPageFsMisc (page);return __set_page_dirty_nobuffers (page); }
fs/freevxfs/vxfs_subr.c +1 -1
@@ -78,7 +78,7 @@ vxfs_get_page(struct address_space *mappkmap (pp);if (!PageUptodate (pp)) goto fail; -/** if (!PageChecked(pp)) **/ +/** if (!PageFsMisc(pp)) **/ /** vxfs_check_page(pp); **/ if (PageError (pp)) goto fail;
fs/reiserfs/inode.c +5 -5
@@ -2344,7 +2344,7 @@ static int reiserfs_write_full_page(strustruct buffer_head *head, *bh;int partial =0 ;int nr =0 ; -int checked =PageChecked (page); +int checked =PageFsMisc (page);struct reiserfs_transaction_handle th;struct super_block *s = inode->i_sb;int bh_per_page =PAGE_CACHE_SIZE / s->s_blocksize;
@@ -2422,7 +2422,7 @@ static int reiserfs_write_full_page(stru* blocks we're going to log */ if (checked) { -ClearPageChecked (page); +ClearPageFsMisc (page);reiserfs_write_lock (s); error =journal_begin (&th, s, bh_per_page +1 );if (error) {
@@ -2803,7 +2803,7 @@ static void reiserfs_invalidatepage(struBUG_ON (!PageLocked (page));if (offset ==0 ) -ClearPageChecked (page); +ClearPageFsMisc (page);if (!page_has_buffers (page)) goto out;
@@ -2844,7 +2844,7 @@ static int reiserfs_set_page_dirty(struc {struct inode *inode = page->mapping->host;if (reiserfs_file_data_log (inode)) { -SetPageChecked (page); +SetPageFsMisc (page);return __set_page_dirty_nobuffers (page); }return __set_page_dirty_buffers (page);
@@ -2867,7 +2867,7 @@ static int reiserfs_releasepage(struct pstruct buffer_head *bh;int ret =1 ; -WARN_ON (PageChecked (page)); +WARN_ON (PageFsMisc (page));spin_lock (&j->j_dirty_buffers_lock); head =page_buffers (page); bh = head;
fs/ufs/dir.c +3 -3
@@ -135,7 +135,7 @@ static void ufs_check_page(struct page *if (offs != limit) goto Eend; out: -SetPageChecked (page); +SetPageFsMisc (page);return ;/* Too bad, we had an error */
@@ -173,7 +173,7 @@ Eend:"offset=%lu" , dir->i_ino, (page->index<<PAGE_CACHE_SHIFT)+offs); fail: -SetPageChecked (page); +SetPageFsMisc (page);SetPageError (page); }
@@ -187,7 +187,7 @@ static struct page *ufs_get_page(structkmap (page);if (!PageUptodate (page)) goto fail; -if (!PageChecked (page)) +if (!PageFsMisc (page))ufs_check_page (page);if (PageError (page)) goto fail;
include/linux/page-flags.h +10 -5
@@ -71,7 +71,7 @@ #define PG_lru 5#define PG_active 6#define PG_slab 7/* slab debug (Suparna wants this) */ -#define PG_checked 8/* kill me in 2.5.<early>. */ +#define PG_fs_misc 8#define PG_arch_1 9#define PG_reserved 10#define PG_private 11/* Has something at ->private */
@@ -161,10 +161,6 @@ #else#define PageHighMem (page) 0/* needed to optimize away at compile time */ #endif -#define PageChecked (page) test_bit(PG_checked, &(page)->flags) -#define SetPageChecked (page) set_bit(PG_checked, &(page)->flags) -#define ClearPageChecked (page) clear_bit(PG_checked, &(page)->flags) -#define PageReserved (page) test_bit(PG_reserved, &(page)->flags)#define SetPageReserved (page) set_bit(PG_reserved, &(page)->flags)#define ClearPageReserved (page) clear_bit(PG_reserved, &(page)->flags)
@@ -263,4 +259,13 @@ static inline void set_page_writeback(st test_set_page_writeback(page); } +/* +* Filesystem-specific page bit testing +*/ +#define PageFsMisc (page) test_bit(PG_fs_misc, &(page)->flags) +#define SetPageFsMisc (page) set_bit(PG_fs_misc, &(page)->flags) +#define TestSetPageFsMisc (page) test_and_set_bit(PG_fs_misc, &(page)->flags) +#define ClearPageFsMisc (page) clear_bit(PG_fs_misc, &(page)->flags) +#define TestClearPageFsMisc (page) test_and_clear_bit(PG_fs_misc, &(page)->flags) +#endif /* PAGE_FLAGS_H */
include/linux/pagemap.h +11 -0
@@ -170,6 +170,17 @@ static inline void wait_on_page_writebacextern void end_page_writeback (struct page *page);/* +* Wait for filesystem-specific page synchronisation to complete +*/ +static inline void wait_on_page_fs_misc (struct page *page) +{ +if (PageFsMisc (page)) +wait_on_page_bit (page, PG_fs_misc); +} + +extern void fastcall end_page_fs_misc(struct page *page); + +/* * Fault a userspace page into pagetables. Return non-zero on a fault. * * This assumes that two userspace pages are always sufficient. That's
mm/filemap.c +17 -0
@@ -577,6 +577,23 @@ void fastcall __lock_page(struct page *p }EXPORT_SYMBOL (__lock_page); +/* +* Note completion of filesystem specific page synchronisation +* +* This is used to allow a page to be written to a filesystem cache in the +* background without holding up the completion of readpage +*/ +void fastcall end_page_fs_misc(struct page *page) +{ +smp_mb__before_clear_bit (); +if (!TestClearPageFsMisc (page)) +BUG (); +smp_mb__after_clear_bit (); +__wake_up_bit (page_waitqueue (page), &page->flags, PG_fs_misc); +} + +EXPORT_SYMBOL (end_page_fs_misc); +/** * find_get_page - find and get a page reference * @mapping: the address_space to search
mm/migrate.c +2 -2
@@ -348,8 +348,8 @@ static void migrate_page_copy(struct pagSetPageUptodate (newpage);if (PageActive (page))SetPageActive (newpage); -if (PageChecked (page)) -SetPageChecked (newpage); +if (PageFsMisc (page)) +SetPageFsMisc (newpage);if (PageMappedToDisk (page))SetPageMappedToDisk (newpage);
mm/page_alloc.c +1 -1
@@ -550,7 +550,7 @@ static int prep_new_page(struct page *pa page->flags &= ~(1 << PG_uptodate |1 << PG_error |1 << PG_referenced |1 << PG_arch_1 | -1 << PG_checked |1 << PG_mappedtodisk); +1 << PG_fs_misc |1 << PG_mappedtodisk);set_page_private (page,0 );set_page_refcounted (page);kernel_map_pages (page,1 << order,1 );
David Howells
maintainerPATCH 2/7 FS-Cache: Generic filesystem caching facility [try #13]
In reply to David HowellsThe attached patch adds a generic intermediary (FS-Cache) by which filesystems may call on local caching capabilities, and by which local caching backends may make caches available:
+---------+ | | +--------------+ | NFS |--+ | | | | | +-->| CacheFS | +---------+ | +----------+ | | /dev/hda5 | | | | | +--------------+ +---------+ +-->| | | | | | |--+ | AFS |----->| FS-Cache | | | | |--+ +---------+ +-->| | | | | | | +--------------+ +---------+ | +----------+ | | | | | | +-->| CacheFiles | | ISOFS |--+ | /var/cache | | | +--------------+ +---------+
The patch also documents the netfs interface and the cache backend interface provided by the facility.
There are a number of reasons why I'm not using i_mapping to do this. These have been discussed a lot on the LKML and CacheFS mailing lists, but to summarise the basics:
(1) Most filesystems don't do hole reportage. Holes in files are treated as blocks of zeros and can't be distinguished otherwise, making it difficult to distinguish blocks that have been read from the network and cached from those that haven't.
(2) The backing inode must be fully populated before being exposed to userspace through the main inode because the VM/VFS goes directly to the backing inode and does not interrogate the front inode on VM ops.
Therefore:
(a) The backing inode must fit entirely within the cache.
(b) All backed files currently open must fit entirely within the cache at the same time.
(c) A working set of files in total larger than the cache may not be cached.
(d) A file may not grow larger than the available space in the cache.
(e) A file that's open and cached, and remotely grows larger than the cache is potentially stuffed.
(3) Writes go to the backing filesystem, and can only be transferred to the network when the file is closed.
(4) There's no record of what changes have been made, so the whole file must be written back.
(5) The pages belong to the backing filesystem, and all metadata associated with that page are relevant only to the backing filesystem, and not anything stacked atop it.
The attached patch adds a generic core to which both networking filesystems and caches may bind. It transfers requests from networking filesystems to appropriate caches if possible, or else gracefully denies them.
If this facility is disabled in the kernel configuration, then all its operations will be trivially reducible to nothing by the compiler.
FS-Cache provides the following facilities:
(1) Caches can be added / removed at any time, even whilst in use.
(2) Adds a facility by which tags can be used to refer to caches, even if they're not mounted yet.
(3) More than one cache can be used at once. Caches can be selected explicitly by use of tags.
(4) The netfs is provided with an interface that allows either party to withdraw caching facilities from a file (required for (1)).
(5) A netfs may annotate cache objects that belongs to it.
(6) Cache objects can be pinned and reservations made.
(7) The interface to the netfs returns as few errors as possible, preferring rather to let the netfs remain oblivious.
(8) Cookies are used to represent indices, files and other objects to the netfs. The simplest cookie is just a NULL pointer - indicating nothing cached there.
(9) The netfs is allowed to propose - dynamically - any index hierarchy it desires, though it must be aware that the index search function is recursive, stack space is limited, and indices can only be children of indices.
(10) Indices can be used to group files together to reduce key size and to make group invalidation easier. The use of indices may make lookup quicker, but that's cache dependent.
(11) Data I/O is effectively done directly to and from the netfs's pages. The netfs indicates that page A is at index B of the data-file represented by cookie C, and that it should be read or written. The cache backend may or may not start I/O on that page, but if it does, a netfs callback will be invoked to indicate completion. The I/O may be either synchronous or asynchronous.
(12) Cookies can be "retired" upon release. At this point FS-Cache will mark them as obsolete and the index hierarchy rooted at that point will get recycled.
(13) The netfs provides a "match" function for index searches. In addition to saying whether a match was made or not, this can also specify that an entry should be updated or deleted.
FS-Cache maintains a virtual indexing tree in which all indices, files, objects and pages are kept. Bits of this tree may actually reside in one or more caches.
FSDEF
|
+------------------------------------+
| |
NFS AFS
| |
+--------------------------+ +-----------+
| | | |
homedir mirror afs.org redhat.com
| | |
+------------+ +---------------+ +----------+
| | | | | |
00001 00002 00007 00125 vol00001 vol00002
| | | | |
+---+---+ +-----+ +---+ +------+------+ +-----+----+
| | | | | | | | | | | | |
PG0 PG1 PG2 PG0 XATTR PG0 PG1 DIRENT DIRENT DIRENT R/W R/O Bak
| |
PG0 +-------+
| |
00001 00003
|
+---+---+
| | |
PG0 PG1 PG2In the example above, you can see two netfs's being backed: NFS and AFS. These have different index hierarchies:
(*) The NFS primary index will probably contain per-server indices. Each
server index is indexed by NFS file handles to get data file objects. Each data file objects can have an array of pages, but may also have further child objects, such as extended attributes and directory entries. Extended attribute objects themselves have page-array contents.
(*) The AFS primary index contains per-cell indices. Each cell index contains
per-logical-volume indices. Each of volume index contains up to three indices for the read-write, read-only and backup mirrors of those volumes. Each of these contains vnode data file objects, each of which contains an array of pages.
The very top index is the FS-Cache master index in which individual netfs's have entries.
Any index object may reside in more than one cache, provided it only has index children. Any index with non-index object children will be assumed to only reside in one cache.
The FS-Cache overview can be found in:
Documentation/filesystems/caching/fscache.txt
The netfs API to FS-Cache can be found in:
Documentation/filesystems/caching/netfs-api.txt
The cache backend API to FS-Cache can be found in:
Documentation/filesystems/caching/backend-api.txt
Further changes [try #11] that have been made:
(*) FSCACHE_NEGATIVE_COOKIE has been removed. NULL should be used instead.
(*) Add read/write context maintenance for in case the cache operation fails.
---
Documentation/filesystems/caching/backend-api.txt | 357 +++++++ Documentation/filesystems/caching/fscache.txt | 151 +++ Documentation/filesystems/caching/netfs-api.txt | 752 +++++++++++++++ fs/Kconfig | 15 fs/Makefile | 1 fs/fscache/Makefile | 11 fs/fscache/cookie.c | 1048 +++++++++++++++++++++ fs/fscache/fscache-int.h | 93 ++ fs/fscache/fsdef.c | 110 ++ fs/fscache/main.c | 105 ++ fs/fscache/page.c | 537 +++++++++++ include/linux/fscache-cache.h | 243 +++++ include/linux/fscache.h | 496 ++++++++++ 13 files changed, 3919 insertions(+), 0 deletions(-)
Documentation/filesystems/caching/backend-api.txt new file +357 -0
@@ -0,0 +1,357 @@ + ========================== + FS-CACHE CACHE BACKEND API + ========================== + +The FS-Cache system provides an API by which actual caches can be supplied to +FS-Cache for it to then serve out to network filesystems and other interested +parties. + +This API is declared in <linux/fscache-cache.h>. + + +==================================== +INITIALISING AND REGISTERING A CACHE +==================================== + +To start off, a cache definition must be initialised and registered for each +cache the backend wants to make available. For instance, CacheFS does this in +the fill_super() operation on mounting. + +The cache definition (struct fscache_cache) should be initialised by calling: + + void fscache_init_cache(struct fscache_cache *cache, + struct fscache_cache_ops *ops, + const char *idfmt, + ...) + +Where: + + (*) "cache" is a pointer to the cache definition; + + (*) "ops" is a pointer to the table of operations that the backend supports on + this cache; + + (*) and a format and printf-style arguments for constructing a label for the + cache. + + +The cache should then be registered with FS-Cache by passing a pointer to the +previously initialised cache definition to: + + int fscache_add_cache(struct fscache_cache *cache, + struct fscache_object *fsdef, + const char *tagname); + +Two extra arguments should also be supplied: + + (*) "fsdef" which should point to the object representation for the FS-Cache + master index in this cache. Netfs primary index entries will be created + here. + + (*) "tagname" which, if given, should be a text string naming this cache. If + this is NULL, the identifier will be used instead. For CacheFS, the + identifier is set to name the underlying block device and the tag can be + supplied by mount. + +This function may return -ENOMEM if it ran out of memory or -EEXIST if the tag +is already in use. 0 will be returned on success. + + +===================== +UNREGISTERING A CACHE +===================== + +A cache can be withdrawn from the system by calling this function with a +pointer to the cache definition: + + void fscache_withdraw_cache(struct fscache_cache *cache) + +In CacheFS's case, this is called by put_super(). + + +================== +FS-CACHE UTILITIES +================== + +FS-Cache provides some utilities that a cache backend may make use of: + + (*) Find the parent of an object: + + struct fscache_object * + fscache_find_parent_object(struct fscache_object *object) + + This allows a backend to find the logical parent of an index or data file + in the cache hierarchy. + + (*) Note occurrence of an I/O error in a cache: + + void fscache_io_error(struct fscache_cache *cache) + + This tells FS-Cache that an I/O error occurred in the cache. After this + has been called, only resource dissociation operations (object and page + release) will be passed from the netfs to the cache backend for the + specified cache. + + This does not actually withdraw the cache. That must be done separately. + + (*) Get an extra reference to a read or write context: + + void *fscache_get_context(struct fscache_cookie *cookie, void *context) + + and release a reference: + + void *fscache_put_context(struct fscache_cookie *cookie, void *context) + + These should be used to maintain the presence of the read or write context + passed to the cache read/write functions. This context must then be + passed to the I/O completion function. + + +======================== +RELEVANT DATA STRUCTURES +======================== + + (*) Index/Data file FS-Cache representation cookie: + + struct fscache_cookie { + struct fscache_object_def *def; + struct fscache_netfs *netfs; + void *netfs_data; + ... + }; + + The fields that might be of use to the backend describe the object + definition, the netfs definition and the netfs's data for this cookie. + The object definition contain functions supplied by the netfs for loading + and matching index entries; these are required to provide some of the + cache operations. + + (*) In-cache object representation: + + struct fscache_object { + struct fscache_cache *cache; + struct fscache_cookie *cookie; + unsigned long flags; + #define FSCACHE_OBJECT_RECYCLING 1 + ... + }; + + Structures of this type should be allocated by the cache backend and + passed to FS-Cache when requested by the appropriate cache operation. In + the case of CacheFS, they're embedded in CacheFS's internal object + structures. + + Each object contains a pointer to the cookie that represents the object it + is backing. It also contains a flag that indicates whether the object is + being retired when put_object() is called. This should be initialised by + calling fscache_object_init(object). + + +================ +CACHE OPERATIONS +================ + +The cache backend provides FS-Cache with a table of operations that can be +performed on the denizens of the cache. These are held in a structure of type: + + struct fscache_cache_ops + + (*) Name of cache provider [mandatory]: + + const char *name + + This isn't strictly an operation, but should be pointed at a string naming + the backend. + + (*) Object lookup [mandatory]: + + struct fscache_object *(*lookup_object)(struct fscache_cache *cache, + struct fscache_object *parent, + struct fscache_cookie *cookie) + + This method is used to look up an object in the specified cache, given a + pointer to the parent object and the cookie to which the object will be + attached. This should instantiate that object in the cache if it can, or + return -ENOBUFS or -ENOMEM if it can't. + + (*) Increment object refcount [mandatory]: + + struct fscache_object *(*grab_object)(struct fscache_object *object) + + This method is called to increment the reference count on an object. It + may fail (for instance if the cache is being withdrawn) by returning NULL. + It should return the object pointer if successful. + + (*) Lock/Unlock object [mandatory]: + + void (*lock_object)(struct fscache_object *object) + void (*unlock_object)(struct fscache_object *object) + + These methods are used to exclusively lock an object. It must be possible + to schedule with the lock held, so a spinlock isn't sufficient. + + (*) Pin/Unpin object [optional]: + + int (*pin_object)(struct fscache_object *object) + void (*unpin_object)(struct fscache_object *object) + + These methods are used to pin an object into the cache. Once pinned an + object cannot be reclaimed to make space. Return -ENOSPC if there's not + enough space in the cache to permit this. + + (*) Update object [mandatory]: + + int (*update_object)(struct fscache_object *object) + + This is called to update the index entry for the specified object. The + new information should be in object->cookie->netfs_data. This can be + obtained by calling object->cookie->def->get_aux()/get_attr(). + + (*) Release object reference [mandatory]: + + void (*put_object)(struct fscache_object *object) + + This method is used to discard a reference to an object. The object may + be destroyed when all the references held by FS-Cache are released. + + (*) Synchronise a cache [mandatory]: + + void (*sync)(struct fscache_cache *cache) + + This is called to ask the backend to synchronise a cache with its backing + device. + + (*) Dissociate a cache [mandatory]: + + void (*dissociate_pages)(struct fscache_cache *cache) + + This is called to ask a cache to perform any page dissociations as part of + cache withdrawal. + + (*) Set the data size on a cache file [mandatory]: + + int (*set_i_size)(struct fscache_object *object, loff_t i_size); + + This is called to indicate to the cache the maximum size a file may reach. + The cache may use this to reserve space on the cache. It may also return + -ENOBUFS to indicate that insufficient space is available to expand the + metadata used to track the data. It should return 0 if successful or + -ENOMEM or -EIO on error. + + (*) Reserve cache space for an object's data [optional]: + + int (*reserve_space)(struct fscache_object *object, loff_t size); + + This is called to request that cache space be reserved to hold the data + for an object and the metadata used to track it. Zero size should be + taken as request to cancel a reservation. + + This should return 0 if successful, -ENOSPC if there isn't enough space + available, or -ENOMEM or -EIO on other errors. + + The reservation may exceed the size of the object, thus permitting future + expansion. If the amount of space consumed by an object would exceed the + reservation, it's permitted to refuse requests to allocate pages, but not + required. An object may be pruned down to its reservation size if larger + than that already. + + (*) Request page be read from cache [mandatory]: + + int (*read_or_alloc_page)(struct fscache_object *object, + struct page *page, + fscache_rw_complete_t end_io_func, + void *end_io_data, + gfp_t gfp) + + This is called to attempt to read a netfs page from the cache, or to + reserve a backing block if not. FS-Cache will have done as much checking + as it can before calling, but most of the work belongs to the backend. + + If there's no page in the cache, then -ENODATA should be returned if the + backend managed to reserve a backing block; -ENOBUFS, -ENOMEM or -EIO if + it didn't. + + If there is a page in the cache, then a read operation should be queued + and 0 returned. When the read finishes, end_io_func() should be called + with the following arguments: + + (*end_io_func)(object->cookie->netfs_data, + page, + end_io_data, + error); + + The mark_pages_cached() cookie operation should be called for the page if + any cache metadata is retained. This will indicate to the netfs that the + page needs explicit uncaching. This operation takes a pagevec, thus + allowing several pages to be marked at once. + + (*) Request pages be read from cache [mandatory]: + + int (*read_or_alloc_pages)(struct fscache_object *object, + struct address_space *mapping, + struct list_head *pages, + unsigned *nr_pages, + fscache_rw_complete_t end_io_func, + void *end_io_data, + gfp_t gfp) + + This is like the previous operation, except it will be handed a list of + pages instead of one page. Any pages on which a read operation is started + must be added to the page cache for the specified mapping and also to the + LRU. Such pages must also be removed from the pages list and nr_pages + decremented per page. + + If there was an error such as -ENOMEM, then that should be returned; else + if one or more pages couldn't be read or allocated, then -ENOBUFS should + be returned; else if one or more pages couldn't be read, then -ENODATA + should be returned. If all the pages are dispatched then 0 should be + returned. + + (*) Request page be allocated in the cache [mandatory]: + + int (*allocate_page)(struct fscache_object *object, + struct page *page, + gfp_t gfp) + + This is like read_or_alloc_page(), except that it shouldn't read from the + cache, even if there's data there that could be retrieved. It should, + however, set up any internal metadata required such that write_page() can + write to the cache. + + If there's no backing block available, then -ENOBUFS should be returned + (or -ENOMEM or -EIO if there were other problems). If a block is + successfully allocated, then the netfs page should be marked and 0 + returned. + + (*) Request page be written to cache [mandatory]: + + int (*write_page)(struct fscache_object *object, + struct page *page, + fscache_rw_complete_t end_io_func, + void *end_io_data, + gfp_t gfp) + + This is called to write from a page on which there was a previously + successful read_or_alloc_page() call. FS-Cache filters out pages that + don't have mappings. + + If there's no backing block available, then -ENOBUFS should be returned + (or -ENOMEM or -EIO if there were other problems). + + If the write operation could be queued, then 0 should be returned. When + the write completes, end_io_func() should be called with the following + arguments: + + (*end_io_func)(object->cookie->netfs_data, + page, + end_io_data, + error); + + (*) Discard retained per-page metadata [mandatory]: + + void (*uncache_pages)(struct fscache_object *object, + struct pagevec *pagevec) + + This is called when one or more netfs pages are being evicted from the + pagecache. The cache backend should tear down any internal representation + or tracking it maintains.
Documentation/filesystems/caching/fscache.txt new file +151 -0
151 lines not on this page · raw message
Documentation/filesystems/caching/netfs-api.txt new file +752 -0
752 lines not on this page · raw message
fs/Kconfig +15 -0
@@ -530,6 +530,21 @@ config FUSE_FS If you want to develop a userspace FS, or if you want to use a filesystem based on FUSE, answer Y or M. +menu "Caches" + +config FSCACHE + tristate "General filesystem cache manager" + depends on EXPERIMENTAL + help + This option enables a generic filesystem caching manager that can be + used by various network and other filesystems to cache data + locally. Different sorts of caches can be plugged in, depending on the + resources available. + + See Documentation/filesystems/caching/fscache.txt for more information. + +endmenu + menu "CD-ROM/DVD Filesystems" config ISO9660_FS
fs/Makefile +1 -0
@@ -52,6 +52,7 @@ obj-y += devpts/ obj-$(CONFIG_PROFILING) += dcookies.o # Do not add any filesystems before this line +obj-$(CONFIG_FSCACHE) += fscache/ obj-$(CONFIG_REISERFS_FS) += reiserfs/ obj-$(CONFIG_EXT3_FS) += ext3/ # Before ext2 so root fs can be ext3 obj-$(CONFIG_JBD) += jbd/
fs/fscache/Makefile new file +11 -0
@@ -0,0 +1,11 @@ +# +# Makefile for general filesystem caching code +# + +fscache-objs := \ + cookie.o \ + fsdef.o \ + main.o \ + page.o + +obj-$(CONFIG_FSCACHE) := fscache.o
fs/fscache/cookie.c new file +1048 -0
1,048 lines not on this page · raw message
fs/fscache/fscache-int.h new file +93 -0
93 lines not on this page · raw message
fs/fscache/fsdef.c new file +110 -0
110 lines not on this page · raw message
fs/fscache/main.c new file +105 -0
105 lines not on this page · raw message
fs/fscache/page.c new file +537 -0
345 lines not on this page · raw message
include/linux/fscache-cache.h new file +243 -0
243 lines not on this page · raw message
include/linux/fscache.h new file +496 -0
496 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
David Howells
maintainerPATCH 3/7 FS-Cache: Release page->private after failed readahead [try #13]
In reply to David HowellsThe attached patch causes read_cache_pages() to release page-private data on a page for which add_to_page_cache() fails or the filler function fails. This permits pages with caching references associated with them to be cleaned up.
The invalidatepage() address space op is called (indirectly) to do the honours.
---
mm/readahead.c | 46 ++++++++++++++++++++++++++++++++++++++-------- 1 files changed, 38 insertions(+), 8 deletions(-)
mm/readahead.c +38 -8
@@ -14,6 +14,7 @@ #include <linux/module.h>#include <linux/blkdev.h> #include <linux/backing-dev.h> #include <linux/pagevec.h> +#include <linux/buffer_head.h> void default_unplug_io_fn (struct backing_dev_info *bdi,struct page *page) {
@@ -117,6 +118,41 @@ static inline unsigned long get_next_ra_#define list_to_page (head) (list_entry((head)->prev, struct page, lru)) +/* +* see if a page needs releasing upon read_cache_pages() failure +* - the caller of read_cache_pages() may have set PG_private before calling, +* such as the NFS fs marking pages that are cached locally on disk, thus we +* need to give the fs a chance to clean up in the event of an error +*/ +static void read_cache_pages_invalidate_page (struct address_space *mapping, +struct page *page) +{ +if (PagePrivate (page)) { +if (TestSetPageLocked (page)) +BUG (); + page->mapping = mapping; +do_invalidatepage (page,0 ); + page->mapping =NULL ; +unlock_page (page); + } +page_cache_release (page); +} + +/* +* release a list of pages, invalidating them first if need be +*/ +static void read_cache_pages_invalidate_pages (struct address_space *mapping, +struct list_head *pages) +{ +struct page *victim; + +while (!list_empty (pages)) { + victim =list_to_page (pages); +list_del (&victim->lru); +read_cache_pages_invalidate_page (mapping, victim); + } +} +/** * read_cache_pages - populate an address space with some pages & start reads against them * @mapping: the address_space
@@ -140,20 +176,14 @@ int read_cache_pages(struct address_spac page =list_to_page (pages);list_del (&page->lru);if (add_to_page_cache (page, mapping, page->index,GFP_KERNEL )) { -page_cache_release (page); +read_cache_pages_invalidate_page (mapping, page);continue ; } ret =filler (data, page);if (!pagevec_add (&lru_pvec, page))__pagevec_lru_add (&lru_pvec);if (ret) { -while (!list_empty (pages)) { -struct page *victim; - - victim =list_to_page (pages); -list_del (&victim->lru); -page_cache_release (victim); - } +read_cache_pages_invalidate_pages (mapping, pages);break ; } }
David Howells
maintainerPATCH 4/7 FS-Cache: Make kAFS use FS-Cache [try #13]
In reply to David HowellsThe attached patch makes the kAFS filesystem in fs/afs/ use FS-Cache, and through it any attached caches. The kAFS filesystem will use caching automatically if it's available.
---
fs/Kconfig | 7 + fs/afs/cache.h | 27 ----- fs/afs/cell.c | 109 +++++++++++++-------- fs/afs/cell.h | 16 +-- fs/afs/cmservice.c | 2 fs/afs/dir.c | 10 +- fs/afs/file.c | 267 ++++++++++++++++++++++++++++++++++------------------ fs/afs/fsclient.c | 4 + fs/afs/inode.c | 45 ++++++--- fs/afs/internal.h | 25 ++--- fs/afs/main.c | 24 ++--- fs/afs/mntpt.c | 12 +- fs/afs/proc.c | 1 fs/afs/server.c | 3 - fs/afs/vlocation.c | 179 +++++++++++++++++++++-------------- fs/afs/vnode.c | 250 ++++++++++++++++++++++++++++++++++++++++--------- fs/afs/vnode.h | 10 +- fs/afs/volume.c | 78 ++++++--------- fs/afs/volume.h | 28 +---- 19 files changed, 673 insertions(+), 424 deletions(-)
fs/Kconfig +7 -0
@@ -1922,6 +1922,13 @@ # for fs/nls/Config.in If unsure, say N. +config AFS_FSCACHE + bool "Provide AFS client caching support" + depends on AFS_FS && FSCACHE && EXPERIMENTAL + help + Say Y here if you want AFS data to be cached locally on through the + generic filesystem cache manager + config RXRPC tristate
fs/afs/cache.h deleted +0 -27
@@ -1,27 +0,0 @@ -/* cache.h: AFS local cache management interface -* -* Copyright (C) 2002 Red Hat, Inc. All Rights Reserved. -* Written by David Howells ([email protected]) -* -* This program is free software; you can redistribute it and/or -* modify it under the terms of the GNU General Public License -* as published by the Free Software Foundation; either version -* 2 of the License, or (at your option) any later version. -*/ - -#ifndef _LINUX_AFS_CACHE_H -#define _LINUX_AFS_CACHE_H - -#undef AFS_CACHING_SUPPORT - -#include <linux/mm.h> -#ifdef AFS_CACHING_SUPPORT -#include <linux/cachefs.h> -#endif -#include "types.h" - -#ifdef __KERNEL__ - -#endif /* __KERNEL__ */ - -#endif /* _LINUX_AFS_CACHE_H */
fs/afs/cell.c +65 -44
146 lines not on this page · raw message
fs/afs/cell.h +3 -13
@@ -13,7 +13,7 @@ #ifndef _LINUX_AFS_CELL_H#define _LINUX_AFS_CELL_H#include "types.h" -#include "cache.h" +#include <linux/fscache.h> #define AFS_CELL_MAX_ADDRS 15
@@ -21,16 +21,6 @@ extern volatile int afs_cells_being_purg/*****************************************************************************/ /* -* entry in the cached cell catalogue -*/ -struct afs_cache_cell -{ -char name[64 ];/* cell name (padded with NULs) */ -struct in_addr vl_servers[15 ];/* cached cell VL servers */ -}; - -/*****************************************************************************/ -/* * AFS cell record */ struct afs_cell
@@ -39,8 +29,8 @@ struct afs_cellstruct list_head link;/* main cell list link */ struct list_head proc_link;/* /proc cell list link */ struct proc_dir_entry *proc_dir;/* /proc dir for this cell */ -#ifdef AFS_CACHING_SUPPORT -struct cachefs_cookie *cache;/* caching cookie */ +#ifdef CONFIG_AFS_FSCACHE +struct fscache_cookie *cache;/* caching cookie */ #endif /* server record management */
fs/afs/cmservice.c +1 -1
@@ -24,7 +24,7 @@ #include "cmservice.h"#include "internal.h" static unsigned afscm_usage;/* AFS cache manager usage count */ -static struct rw_semaphore afscm_sem;/* AFS cache manager start/stop semaphore */ +static DECLARE_RWSEM (afscm_sem);/* AFS cache manager start/stop semaphore */ static int afscm_new_call (struct rxrpc_call *call);static void afscm_attention (struct rxrpc_call *call);
fs/afs/dir.c +6 -4
@@ -145,7 +145,7 @@ #endif qty /=sizeof (union afs_dir_block );/* check them */ - dbuf =page_address (page); + dbuf =kmap_atomic (page,KM_USER0 );for (tmp =0 ; tmp < qty; tmp++) {if (dbuf->blocks[tmp].pagehdr.magic !=AFS_DIR_MAGIC ) { printk("kAFS: %s(%lu): bad magic %d/%d is %04hx\n",
@@ -154,10 +154,12 @@ #endif goto error; } } +kunmap_atomic (dbuf,KM_USER0 );return ; error: +kunmap_atomic (dbuf,KM_USER0 );SetPageError (page); }/* end afs_dir_check_page() */
@@ -168,7 +170,6 @@ #endif*/ static inline void afs_dir_put_page (struct page *page) { -kunmap (page);page_cache_release (page); }/* end afs_dir_put_page() */
@@ -186,7 +187,6 @@ static struct page *afs_dir_get_page(str page =read_mapping_page (dir->i_mapping, index,NULL );if (!IS_ERR (page)) {wait_on_page_locked (page); -kmap (page);if (!PageUptodate (page)) goto fail;afs_dir_check_page (dir, page);
@@ -354,7 +354,7 @@ static int afs_dir_iterate(struct inode limit = blkoff & ~(PAGE_SIZE -1 ); - dbuf =page_address (page); + dbuf =kmap_atomic (page,KM_USER0 );/* deal with the individual blocks stashed on this page */ do {
@@ -363,6 +363,7 @@ static int afs_dir_iterate(struct inode ret =afs_dir_iterate_block (fpos, dblock, blkoff, cookie, filldir);if (ret !=1 ) { +kunmap_atomic (dbuf,KM_USER0 );afs_dir_put_page (page); goto out; }
@@ -371,6 +372,7 @@ static int afs_dir_iterate(struct inode }while (*fpos < dir->i_size && blkoff < limit); +kunmap_atomic (dbuf,KM_USER0 );afs_dir_put_page (page); ret =0 ; }
fs/afs/file.c +173 -94
408 lines not on this page · raw message
fs/afs/fsclient.c +3 -1
@@ -398,6 +398,8 @@ int afs_rxfs_fetch_file_status(struct af bp++;/* spare6 */ } +_debug ("Data Version %llx\n" , vnode->status.version); +/* success */ ret =0 ;
@@ -408,7 +410,7 @@ int afs_rxfs_fetch_file_status(struct af out_put_conn:afs_server_release_callslot (server, &callslot); out: -_leave ("" ); +_leave (" = %d" , ret);return ret; abort:
fs/afs/inode.c +31 -14
91 lines not on this page · raw message
fs/afs/internal.h +10 -15
@@ -16,15 +16,17 @@ #include <linux/compiler.h>#include <linux/kernel.h> #include <linux/fs.h> #include <linux/pagemap.h> +#include <linux/fscache.h> /* * debug tracing */ -#define kenter(FMT, a...) printk("==> %s("FMT")\n",__FUNCTION__ , ## a) -#define kleave(FMT, a...) printk("<== %s()"FMT"\n",__FUNCTION__ , ## a) -#define kdebug(FMT, a...) printk(FMT"\n" , ## a) -#define kproto(FMT, a...) printk("### "FMT"\n" , ## a) -#define knet(FMT, a...) printk(FMT"\n" , ## a) +#define __kdbg(FMT, a...) printk("[%05d] "FMT"\n", current->pid , ## a) +#define kenter(FMT, a...) __kdbg("==> %s("FMT")", __FUNCTION__ , ## a) +#define kleave(FMT, a...) __kdbg("<== %s()"FMT, __FUNCTION__ , ## a) +#define kdebug(FMT, a...) __kdbg(FMT , ## a) +#define kproto(FMT, a...) __kdbg("### "FMT , ## a) +#define knet(FMT, a...) __kdbg(FMT , ## a)#ifdef __KDEBUG #define _enter(FMT, a...) kenter(FMT , ## a)
@@ -56,9 +58,6 @@ static inline void afs_discard_my_signal*/ extern struct rw_semaphore afs_proc_cells_sem;extern struct list_head afs_proc_cells; -#ifdef AFS_CACHING_SUPPORT -extern struct cachefs_index_def afs_cache_cell_index_def; -#endif /* * dir.c
@@ -71,11 +70,7 @@ extern const struct file_operations afs_*/ extern const struct address_space_operations afs_fs_aops;extern struct inode_operations afs_file_inode_operations; - -#ifdef AFS_CACHING_SUPPORT -extern int afs_cache_get_page_cookie (struct page *page, -struct cachefs_page **_page_cookie); -#endif +extern const struct file_operations afs_file_file_operations;/* * inode.c
@@ -97,8 +92,8 @@ #endif/* * main.c */ -#ifdef AFS_CACHING_SUPPORT -extern struct cachefs_netfs afs_cache_netfs; +#ifdef CONFIG_AFS_FSCACHE +extern struct fscache_netfs afs_cache_netfs;#endif /*
fs/afs/main.c +11 -13
@@ -1,6 +1,6 @@/* main.c: AFS client file system * -* Copyright (C) 2002 Red Hat, Inc. All Rights Reserved. +* Copyright (C) 2002,5 Red Hat, Inc. All Rights Reserved. * Written by David Howells ([email protected]) * * This program is free software; you can redistribute it and/or
@@ -14,11 +14,11 @@ #include <linux/moduleparam.h>#include <linux/init.h> #include <linux/sched.h> #include <linux/completion.h> +#include <linux/fscache.h> #include <rxrpc/rxrpc.h> #include <rxrpc/transport.h> #include <rxrpc/call.h> #include <rxrpc/peer.h> -#include "cache.h" #include "cell.h" #include "server.h" #include "fsclient.h"
@@ -51,12 +51,11 @@ static struct rxrpc_peer_ops afs_peer_opstruct list_head afs_cb_hash_tbl[AFS_CB_HASH_COUNT ];DEFINE_SPINLOCK (afs_cb_hash_lock); -#ifdef AFS_CACHING_SUPPORT -static struct cachefs_netfs_operations afs_cache_ops = { - .get_page_cookie = afs_cache_get_page_cookie, +#ifdef CONFIG_AFS_FSCACHE +static struct fscache_netfs_operations afs_cache_ops = { }; -struct cachefs_netfs afs_cache_netfs = { +struct fscache_netfs afs_cache_netfs = { .name ="afs" , .version =0 , .ops = &afs_cache_ops,
@@ -83,10 +82,9 @@ static int __init afs_init(void)if (ret <0 )return ret; -#ifdef AFS_CACHING_SUPPORT +#ifdef CONFIG_AFS_FSCACHE /* we want to be able to cache */ - ret =cachefs_register_netfs (&afs_cache_netfs, - &afs_cache_cell_index_def); + ret =fscache_register_netfs (&afs_cache_netfs);if (ret <0 ) goto error;#endif
@@ -137,8 +135,8 @@ #ifdef CONFIG_KEYS_TURNED_OFFafs_key_unregister (); error_cache:#endif -#ifdef AFS_CACHING_SUPPORT -cachefs_unregister_netfs (&afs_cache_netfs); +#ifdef CONFIG_AFS_FSCACHE +fscache_unregister_netfs (&afs_cache_netfs); error:#endif afs_cell_purge ();
@@ -167,8 +165,8 @@ static void __exit afs_exit(void)#ifdef CONFIG_KEYS_TURNED_OFF afs_key_unregister ();#endif -#ifdef AFS_CACHING_SUPPORT -cachefs_unregister_netfs (&afs_cache_netfs); +#ifdef CONFIG_AFS_FSCACHE +fscache_unregister_netfs (&afs_cache_netfs);#endif afs_proc_cleanup ();
fs/afs/mntpt.c +6 -6
@@ -78,7 +78,7 @@ int afs_mntpt_check_symlink(struct afs_v ret = -EIO ;wait_on_page_locked (page); - buf =kmap (page); + buf =kmap_atomic (page,KM_USER0 );if (!PageUptodate (page)) goto out_free;if (PageError (page))
@@ -101,7 +101,7 @@ int afs_mntpt_check_symlink(struct afs_v ret =0 ; out_free: -kunmap (page); +kunmap_atomic (buf,KM_USER0 );page_cache_release (page); out:_leave (" = %d" , ret);
@@ -188,9 +188,9 @@ static struct vfsmount *afs_mntpt_do_autif (!PageUptodate (page) ||PageError (page)) goto error; - buf =kmap (page); + buf =kmap_atomic (page,KM_USER0 );memcpy (devname, buf, size); -kunmap (page); +kunmap_atomic (buf,KM_USER0 );page_cache_release (page); page =NULL ;
@@ -269,12 +269,12 @@ static void *afs_mntpt_follow_link(struc*/ static void afs_mntpt_expiry_timed_out (struct afs_timer *timer) { -kenter ("" ); +// kenter(""); mark_mounts_for_expiry (&afs_vfsmounts);afs_kafstimod_add_timer (&afs_mntpt_expiry_timer, afs_mntpt_expiry_timeout *HZ ); -kleave ("" ); +// kleave(""); }/* end afs_mntpt_expiry_timed_out() */
fs/afs/proc.c +1 -0
@@ -177,6 +177,7 @@ int afs_proc_init(void)*/ void afs_proc_cleanup (void ) { +remove_proc_entry ("rootcell" , proc_afs);remove_proc_entry ("cells" , proc_afs);remove_proc_entry ("fs/afs" ,NULL );
fs/afs/server.c +1 -2
@@ -375,7 +375,6 @@ int afs_server_request_callslot(struct aelse if (list_empty (&server->fs_callq)) {/* no one waiting */ server->fs_conn_cnt[nconn]++; -spin_unlock (&server->fs_lock); }else {/* someone's waiting - dequeue them and wake them up */
@@ -393,9 +392,9 @@ int afs_server_request_callslot(struct a } pcallslot->ready =1 ;wake_up_process (pcallslot->task); -spin_unlock (&server->fs_lock); } +spin_unlock (&server->fs_lock);rxrpc_put_connection (callslot->conn); callslot->conn =NULL ;
fs/afs/vlocation.c +110 -69
237 lines not on this page · raw message
fs/afs/vnode.c +206 -44
289 lines not on this page · raw message
fs/afs/vnode.h +5 -5
@@ -13,9 +13,9 @@ #ifndef _LINUX_AFS_VNODE_H#define _LINUX_AFS_VNODE_H#include <linux/fs.h> +#include <linux/fscache.h> #include "server.h" #include "kafstimod.h" -#include "cache.h" #ifdef __KERNEL__
@@ -32,8 +32,8 @@ struct afs_cache_vnodeafs_dataversion_t data_version;/* data version */ }; -#ifdef AFS_CACHING_SUPPORT -extern struct cachefs_index_def afs_vnode_cache_index_def; +#ifdef CONFIG_AFS_FSCACHE +extern struct fscache_cookie_def afs_vnode_cache_index_def;#endif /*****************************************************************************/
@@ -47,8 +47,8 @@ struct afs_vnodestruct afs_volume *volume;/* volume on which vnode resides */ struct afs_fid fid;/* the file identifier for this inode */ struct afs_file_status status;/* AFS status info for this file */ -#ifdef AFS_CACHING_SUPPORT -struct cachefs_cookie *cache;/* caching cookie */ +#ifdef CONFIG_AFS_FSCACHE +struct fscache_cookie *cache;/* caching cookie */ #endif wait_queue_head_t update_waitq;/* status fetch waitqueue */
fs/afs/volume.c +29 -49
116 lines not on this page · raw message
fs/afs/volume.h +5 -23
@@ -12,11 +12,11 @@ #ifndef _LINUX_AFS_VOLUME_H #define _LINUX_AFS_VOLUME_H +#include <linux/fscache.h> #include "types.h" #include "fsclient.h" #include "kafstimod.h" #include "kafsasyncd.h" -#include "cache.h" typedef enum { AFS_VLUPD_SLEEP, /* sleeping waiting for update timer to fire */
@@ -45,24 +45,6 @@ #define AFS_VOL_VTM_BAK 0x04 /* backup vtime_t rtime;/* last retrieval time */ }; -#ifdef AFS_CACHING_SUPPORT -extern struct cachefs_index_def afs_vlocation_cache_index_def; -#endif - -/*****************************************************************************/ -/* -* volume -> vnode hash table entry -*/ -struct afs_cache_vhash -{ -afs_voltype_t vtype;/* which volume variation */ -uint8_t hash_bucket;/* which hash bucket this represents */ -} __attribute__((packed)); - -#ifdef AFS_CACHING_SUPPORT -extern struct cachefs_index_def afs_volume_cache_index_def; -#endif -/*****************************************************************************/ /* * AFS volume location record
@@ -73,8 +55,8 @@ struct afs_vlocationstruct list_head link;/* link in cell volume location list */ struct afs_timer timeout;/* decaching timer */ struct afs_cell *cell;/* cell to which volume belongs */ -#ifdef AFS_CACHING_SUPPORT -struct cachefs_cookie *cache;/* caching cookie */ +#ifdef CONFIG_AFS_FSCACHE +struct fscache_cookie *cache;/* caching cookie */ #endif struct afs_cache_vlocation vldb;/* volume information DB record */ struct afs_volume *vols[3 ];/* volume access record pointer (index by type) */
@@ -109,8 +91,8 @@ struct afs_volumeatomic_t usage;struct afs_cell *cell;/* cell to which belongs (unrefd ptr) */ struct afs_vlocation *vlocation;/* volume location */ -#ifdef AFS_CACHING_SUPPORT -struct cachefs_cookie *cache;/* caching cookie */ +#ifdef CONFIG_AFS_FSCACHE +struct fscache_cookie *cache;/* caching cookie */ #endif afs_volid_t vid;/* volume ID */ afs_voltype_t type;/* type of volume */
David Howells
maintainerPATCH 5/7 NFS: Use local caching [try #13]
In reply to David HowellsThe attached patch makes it possible for the NFS filesystem to make use of the network filesystem local caching service (FS-Cache).
To be able to use this, an updated mount program is required. This can be obtained from:
http://people.redhat.com/steved/cachefs/util-linux/
To mount an NFS filesystem to use caching, add an "fsc" option to the mount:
mount warthog:/ /a -o fsc
---
fs/Kconfig | 7 + fs/nfs/Makefile | 1 fs/nfs/client.c | 11 + fs/nfs/file.c | 49 ++++- fs/nfs/fscache.c | 348 ++++++++++++++++++++++++++++++++ fs/nfs/fscache.h | 479 ++++++++++++++++++++++++++++++++++++++++++++ fs/nfs/inode.c | 21 ++ fs/nfs/internal.h | 32 +++ fs/nfs/pagelist.c | 3 fs/nfs/read.c | 30 +++ fs/nfs/super.c | 1 fs/nfs/sysctl.c | 43 ++++ fs/nfs/write.c | 11 + include/linux/nfs4_mount.h | 1 include/linux/nfs_fs.h | 5 include/linux/nfs_fs_sb.h | 5 include/linux/nfs_mount.h | 1 17 files changed, 1038 insertions(+), 10 deletions(-)
fs/Kconfig +7 -0
@@ -1485,6 +1485,13 @@ config NFS_V4 If unsure, say N. +config NFS_FSCACHE + bool "Provide NFS client caching support (EXPERIMENTAL)" + depends on NFS_FS && FSCACHE && EXPERIMENTAL + help + Say Y here if you want NFS data to be cached locally on disc through + the general filesystem cache manager + config NFS_DIRECTIO bool "Allow direct I/O on NFS files (EXPERIMENTAL)" depends on NFS_FS && EXPERIMENTAL
fs/nfs/Makefile +1 -0
@@ -16,4 +16,5 @@ nfs-$(CONFIG_NFS_V4) += nfs4proc.o nfs4x nfs4namespace.o nfs-$(CONFIG_NFS_DIRECTIO) += direct.o nfs-$(CONFIG_SYSCTL) += sysctl.o +nfs-$(CONFIG_NFS_FSCACHE) += fscache.o nfs-objs := $(nfs-y)
fs/nfs/client.c +8 -3
@@ -150,6 +150,8 @@ #ifdef CONFIG_NFS_V4 clp->cl_state =1 <<NFS4CLNT_LEASE_EXPIRED ;#endif +nfs_fscache_get_client_cookie (clp); +return clp; error_3:
@@ -193,6 +195,8 @@ static void nfs_free_client(struct nfs_cnfs4_shutdown_client (clp); +nfs_fscache_release_client_cookie (clp); +/* -EIO all pending I/O */ if (!IS_ERR (clp->cl_rpcclient))rpc_shutdown_client (clp->cl_rpcclient);
@@ -1371,7 +1375,7 @@ static int nfs_volume_list_show(struct s/* display header on line 1 */ if (v ==SEQ_START_TOKEN ) { -seq_puts (m,"NV SERVER PORT DEV FSID\n" ); +seq_puts (m,"NV SERVER PORT DEV FSID FSC\n" );return 0 ; }/* display one transport per line on subsequent lines */
@@ -1385,12 +1389,13 @@ static int nfs_volume_list_show(struct s (unsigned long long ) server->fsid.major, (unsigned long long) server->fsid.minor); -seq_printf (m,"v%d %02x%02x%02x%02x %4hx %-7s %-17s\n" , +seq_printf (m,"v%d %02x%02x%02x%02x %4hx %-7s %-17s %s\n" , clp->cl_nfsversion,NIPQUAD (clp->cl_addr.sin_addr),ntohs (clp->cl_addr.sin_port), dev, - fsid); + fsid, +nfs_server_fscache_state (server));return 0 ; }
fs/nfs/file.c +46 -3
95 lines not on this page · raw message
fs/nfs/fscache.c new file +348 -0
348 lines not on this page · raw message
fs/nfs/fscache.h new file +479 -0
479 lines not on this page · raw message
fs/nfs/inode.c +21 -0
@@ -78,6 +78,7 @@ void nfs_clear_inode(struct inode *inodeBUG_ON (atomic_read (&NFS_I (inode)->data_updates) !=0 );nfs_zap_acl_cache (inode);nfs_access_zap_cache (inode); +nfs_fscache_release_fh_cookie (NFS_SERVER (inode),NFS_I (inode)); }/**
@@ -123,6 +124,8 @@ void nfs_zap_caches(struct inode *inode)spin_lock (&inode->i_lock);nfs_zap_caches_locked (inode);spin_unlock (&inode->i_lock); + +nfs_fscache_zap_fh_cookie (NFS_SERVER (inode),NFS_I (inode)); }static void nfs_zap_acl_cache (struct inode *inode)
@@ -201,6 +204,7 @@ nfs_fhget(struct super_block *sb, struct };struct inode *inode =ERR_PTR (-ENOENT );unsigned long hash; +int maycache =1 ;if ((fattr->valid &NFS_ATTR_FATTR ) ==0 ) goto out_no_inode;
@@ -252,6 +256,7 @@ nfs_fhget(struct super_block *sb, structelse inode->i_op = &nfs_mountpoint_inode_operations; inode->i_fop =NULL ; + maycache =0 ; } }else if (S_ISLNK (inode->i_mode)) inode->i_op = &nfs_symlink_inode_operations;
@@ -284,6 +289,8 @@ nfs_fhget(struct super_block *sb, structmemset (nfsi->cookieverf,0 ,sizeof (nfsi->cookieverf)); nfsi->access_cache =RB_ROOT ; +nfs_fscache_get_fh_cookie (sb, nfsi, maycache); +unlock_new_inode (inode); }else nfs_refresh_inode (inode, fattr);
@@ -366,6 +373,7 @@ void nfs_setattr_update_inode(struct inoif ((attr->ia_valid &ATTR_SIZE ) !=0 ) {nfs_inc_stats (inode,NFSIOS_SETATTRTRUNC ); inode->i_size = attr->ia_size; +nfs_fscache_set_size (NFS_SERVER (inode),NFS_I (inode), inode->i_size);vmtruncate (inode, attr->ia_size); } }
@@ -550,6 +558,8 @@ int nfs_open(struct inode *inode, struct ctx->mode = filp->f_mode;nfs_file_set_open_context (filp, ctx);put_nfs_open_context (ctx); +if ((filp->f_flags &O_ACCMODE ) !=O_RDONLY ) +nfs_fscache_disable_fh_cookie (inode);return 0 ; }
@@ -688,6 +698,8 @@ int nfs_revalidate_mapping(struct inode }spin_unlock (&inode->i_lock); +nfs_fscache_renew_fh_cookie (NFS_SERVER (inode), nfsi); +dfprintk (PAGECACHE ,"NFS: (%s/%Ld) data cache invalidated\n" , inode->i_sb->s_id, (long long )NFS_FILEID (inode));
@@ -921,11 +933,13 @@ static int nfs_update_inode(struct inodeif (data_stable) { inode->i_size = new_isize; invalid |=NFS_INO_INVALID_DATA ; +nfs_fscache_set_size (NFS_SERVER (inode), nfsi, inode->i_size); } invalid |=NFS_INO_INVALID_ATTR ; }else if (new_isize > cur_isize) { inode->i_size = new_isize; invalid |=NFS_INO_INVALID_ATTR |NFS_INO_INVALID_DATA ; +nfs_fscache_set_size (NFS_SERVER (inode), nfsi, inode->i_size); } nfsi->cache_change_attribute = jiffies; dprintk("NFS: isize change on server for file %s/%ld\n",
@@ -1140,6 +1154,10 @@ static int __init init_nfs_fs(void) {int err; + err =nfs_fscache_register (); +if (err <0 ) + goto out6; + err =nfs_fs_proc_init ();if (err) goto out5;
@@ -1186,6 +1204,8 @@ out3: out4:nfs_fs_proc_exit (); out5: +nfs_fscache_unregister (); +out6:return err; }
@@ -1196,6 +1216,7 @@ static void __exit exit_nfs_fs(void)nfs_destroy_readpagecache ();nfs_destroy_inodecache ();nfs_destroy_nfspagecache (); +nfs_fscache_unregister ();#ifdef CONFIG_PROC_FS rpc_proc_unregister ("nfs" );#endif
fs/nfs/internal.h +32 -0
@@ -4,6 +4,30 @@#include <linux/mount.h> +#define NFS_PAGE_WRITING 0 +#define NFS_PAGE_CACHED 1 + +#define PageNfsBit (bit, page) test_bit(bit, &(page)->private) + +#define SetPageNfsBit (bit, page) \ +do { \ + SetPagePrivate((page)); \ + set_bit(bit, &(page)->private); \ +} while(0) + +#define ClearPageNfsBit (bit, page) \ +do { \ + clear_bit(bit, &(page)->private); \ +} while(0) + +#define PageNfsWriting (page) PageNfsBit(NFS_PAGE_WRITING, (page)) +#define SetPageNfsWriting (page) SetPageNfsBit(NFS_PAGE_WRITING, (page)) +#define ClearPageNfsWriting (page) ClearPageNfsBit(NFS_PAGE_WRITING, (page)) + +#define PageNfsCached (page) PageNfsBit(NFS_PAGE_CACHED, (page)) +#define SetPageNfsCached (page) SetPageNfsBit(NFS_PAGE_CACHED, (page)) +#define ClearPageNfsCached (page) ClearPageNfsBit(NFS_PAGE_CACHED, (page)) +struct nfs_string ;struct nfs_mount_data ;struct nfs4_mount_data ;
@@ -27,6 +51,11 @@ struct nfs_clone_mount {rpc_authflavor_t authflavor; }; +/* +* include filesystem caching stuff here +*/ +#include "fscache.h" +/* client.c */ extern struct rpc_program nfs_program;
@@ -153,6 +182,9 @@ extern int nfs4_path_walk(struct nfs_ser const char *path);#endif +/* read.c */ +extern int nfs_readpage_async (struct nfs_open_context *,struct inode *,struct page *); +/* * Determine the device name as a string */
fs/nfs/pagelist.c +2 -1
@@ -17,6 +17,7 @@ #include <linux/nfs4.h>#include <linux/nfs_page.h> #include <linux/nfs_fs.h> #include <linux/nfs_mount.h> +#include "internal.h" #define NFS_PARANOIA 1
@@ -84,7 +85,7 @@ nfs_create_request(struct nfs_open_conteatomic_set (&req->wb_complete,0 ); req->wb_index = page->index;page_cache_get (page); -BUG_ON (PagePrivate (page)); +BUG_ON (PageNfsWriting (page));BUG_ON (!PageLocked (page));BUG_ON (page->mapping->host != inode); req->wb_offset = offset;
fs/nfs/read.c +29 -1
@@ -26,11 +26,13 @@ #include <linux/pagemap.h>#include <linux/sunrpc/clnt.h> #include <linux/nfs_fs.h> #include <linux/nfs_page.h> +#include <linux/nfs_mount.h> #include <linux/smp_lock.h> #include <asm/system.h> #include "iostat.h" +#include "internal.h" #define NFSDBG_FACILITY NFSDBG_PAGECACHE
@@ -207,13 +209,18 @@ static int nfs_readpage_sync(struct nfs_SetPageUptodate (page); result =0 ; +nfs_readpage_to_fscache (inode, page,1 ); +unlock_page (page); + +return result; + io_error:unlock_page (page);nfs_readdata_free (rdata);return result; } -static int nfs_readpage_async (struct nfs_open_context *ctx,struct inode *inode, +int nfs_readpage_async (struct nfs_open_context *ctx,struct inode *inode,struct page *page) {LIST_HEAD (one_request);
@@ -238,6 +245,11 @@ static int nfs_readpage_async(struct nfsstatic void nfs_readpage_release (struct nfs_page *req) { +struct inode *d_inode = req->wb_context->dentry->d_inode; + +if (PageUptodate (req->wb_page)) +nfs_readpage_to_fscache (d_inode, req->wb_page,0 ); +unlock_page (req->wb_page); dprintk("NFS: read done (%s/%Ld %d@%Ld)\n",
@@ -620,6 +632,10 @@ int nfs_readpage(struct file *file, stru ctx =get_nfs_open_context ((struct nfs_open_context *) file->private_data);if (!IS_SYNC (inode)) { + error =nfs_readpage_from_fscache (ctx, inode, page); +if (error ==0 ) + goto out; + error =nfs_readpage_async (ctx, inode, page); goto out; }
@@ -650,6 +666,7 @@ readpage_async_filler(void *data, structunsigned int len;nfs_wb_page (inode, page); + len =nfs_page_length (inode, page);if (len ==0 )return nfs_return_empty_page (page);
@@ -689,6 +706,17 @@ int nfs_readpages(struct file *filp, str }else desc.ctx =get_nfs_open_context ((struct nfs_open_context *) filp->private_data); + +/* attempt to read as many of the pages as possible from the cache +* - this returns -ENOBUFS immediately if the cookie is negative +*/ + ret =nfs_readpages_from_fscache (desc.ctx, inode, mapping, + pages, &nr_pages); +if (ret ==0 ) { +put_nfs_open_context (desc.ctx); +return ret;/* all read */ + } + ret =read_cache_pages (mapping, pages, readpage_async_filler, &desc);if (!list_empty (&head)) {int err =nfs_pagein_list (&head, server->rpages);
fs/nfs/super.c +1 -0
@@ -291,6 +291,7 @@ static void nfs_show_mount_options(struc { NFS_MOUNT_NOAC, ",noac", "" }, { NFS_MOUNT_NONLM, ",nolock", "" }, { NFS_MOUNT_NOACL, ",noacl", "" }, + { NFS_MOUNT_FSCACHE, ",fsc", "" }, { 0, NULL, NULL } };const struct proc_nfs_info *nfs_infop;
fs/nfs/sysctl.c +43 -0
55 lines not on this page · raw message
fs/nfs/write.c +9 -2
@@ -63,6 +63,7 @@ #include <linux/smp_lock.h>#include "delegation.h" #include "iostat.h" +#include "internal.h" #define NFSDBG_FACILITY NFSDBG_PAGECACHE
@@ -163,6 +164,9 @@ static void nfs_grow_file(struct page *preturn ;nfs_inc_stats (inode,NFSIOS_EXTENDWRITE );i_size_write (inode, end); +#ifdef FSCACHE_WRITE_SUPPORT +nfs_set_fscsize (NFS_SERVER (inode),NFS_I (inode), end); +#endif }/* We can set the PG_uptodate flag if we see that a write request
@@ -342,6 +346,9 @@ do_it: err = -EBADF ; goto out; } + +nfs_writepage_to_fscache (inode, page); +lock_kernel ();if (!IS_SYNC (inode) && inode_referenced) { err =nfs_writepage_async (ctx, inode, page,0 , offset);
@@ -425,7 +432,7 @@ static int nfs_inode_add_request(structif (nfs_have_delegation (inode,FMODE_WRITE )) nfsi->change_attr++; } -SetPagePrivate (req->wb_page); +SetPageNfsWriting (req->wb_page); nfsi->npages++;atomic_inc (&req->wb_count);return 0 ;
@@ -442,7 +449,7 @@ static void nfs_inode_remove_request(strBUG_ON (!NFS_WBACK_BUSY (req));spin_lock (&nfsi->req_lock); -ClearPagePrivate (req->wb_page); +ClearPageNfsWriting (req->wb_page);radix_tree_delete (&nfsi->nfs_page_tree, req->wb_index); nfsi->npages--;if (!nfsi->npages) {
include/linux/nfs4_mount.h +1 -0
@@ -65,6 +65,7 @@ #define NFS4_MOUNT_INTR 0x0002 /* 1 */#define NFS4_MOUNT_NOCTO 0x0010/* 1 */ #define NFS4_MOUNT_NOAC 0x0020/* 1 */ #define NFS4_MOUNT_STRICTLOCK 0x1000/* 1 */ +#define NFS4_MOUNT_FSCACHE 0x4000/* 1 */ #define NFS4_MOUNT_FLAGMASK 0xFFFF#endif
include/linux/nfs_fs.h +5 -0
@@ -30,6 +30,7 @@ #include <linux/nfs_fs_sb.h>#include <linux/rwsem.h> #include <linux/mempool.h> +#include <linux/fscache.h> /* * Enable debugging support for nfs client.
@@ -185,6 +186,9 @@ #ifdef CONFIG_NFS_V4int delegation_state;struct rw_semaphore rwsem;#endif /* CONFIG_NFS_V4*/ +#ifdef CONFIG_NFS_FSCACHE +struct fscache_cookie *fscache; +#endif struct inode vfs_inode; };
@@ -577,6 +581,7 @@ #define NFSDBG_FILE 0x0040#define NFSDBG_ROOT 0x0080#define NFSDBG_CALLBACK 0x0100#define NFSDBG_CLIENT 0x0200 +#define NFSDBG_FSCACHE 0x0400#define NFSDBG_ALL 0xFFFF#ifdef __KERNEL__
include/linux/nfs_fs_sb.h +5 -0
@@ -3,6 +3,7 @@ #define _NFS_FS_SB#include <linux/list.h> #include <linux/backing-dev.h> +#include <linux/fscache.h> struct nfs_iostats ;
@@ -67,6 +68,10 @@ #ifdef CONFIG_NFS_V4char cl_ipaddr[16 ];unsigned char cl_id_uniquifier;#endif + +#ifdef CONFIG_NFS_FSCACHE +struct fscache_cookie *fscache;/* client index cache cookie */ +#endif };/*
include/linux/nfs_mount.h +1 -0
@@ -61,6 +61,7 @@ #define NFS_MOUNT_BROKEN_SUID 0x0400 /*#define NFS_MOUNT_NOACL 0x0800/* 4 */ #define NFS_MOUNT_STRICTLOCK 0x1000/* reserved for NFSv4 */ #define NFS_MOUNT_SECFLAVOUR 0x2000/* 5 */ +#define NFS_MOUNT_FSCACHE 0x4000#define NFS_MOUNT_FLAGMASK 0xFFFF#endif
David Howells
maintainerPATCH 6/7 FS-Cache: CacheFiles: ia64: missing copy_page export [try #13]
In reply to David HowellsThis one-line patch fixes the missing export of copy_page introduced by the cachefile patches. This patch is not yet upstream, but is required for cachefile on ia64. It will be pushed upstream when cachefile goes upstream.
---
arch/ia64/kernel/ia64_ksyms.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
arch/ia64/kernel/ia64_ksyms.c +1 -0
@@ -42,6 +42,7 @@ EXPORT_SYMBOL(__do_clear_user);EXPORT_SYMBOL (__strlen_user);EXPORT_SYMBOL (__strncpy_from_user);EXPORT_SYMBOL (__strnlen_user); +EXPORT_SYMBOL (copy_page);/* from arch/ia64/lib */ extern void __divsi3 (void );
David Howells
maintainerPATCH 7/7 FS-Cache: CacheFiles: A cache that backs onto a mounted filesystem [try #13]
In reply to David HowellsAdd a cache backend that permits a mounted filesystem to be used as a backing store for the cache.
CacheFiles uses a userspace daemon to do some of the cache management - such as reaping stale nodes and culling. This is called cachefilesd and lives in /sbin. The source for the daemon can be downloaded from:
http://people.redhat.com/~dhowells/cachefs/cachefilesd.c
And an example configuration from:
http://people.redhat.com/~dhowells/cachefs/cachefilesd.conf
The filesystem and data integrity of the cache are only as good as those of the filesystem providing the backing services. Note that CacheFiles does not attempt to journal anything since the journalling interfaces of the various filesystems are very specific in nature.
CacheFiles creates a proc-file - "/proc/fs/cachefiles" - that is used for communication with the daemon. Only one thing may have this open at once, and whilst it is open, a cache is at least partially in existence. The daemon opens this and sends commands down it to control the cache.
CacheFiles is currently limited to a single cache.
CacheFiles attempts to maintain at least a certain percentage of free space on the filesystem, shrinking the cache by culling the objects it contains to make space if necessary - see the "Cache Culling" section. This means it can be placed on the same medium as a live set of data, and will expand to make use of spare space and automatically contract when the set of data requires more space.
============
REQUIREMENTS
============
The use of CacheFiles and its daemon requires the following features to be available in the system and in the cache filesystem:
- dnotify.
- extended attributes (xattrs).
- openat() and friends.
- bmap() support on files in the filesystem (FIBMAP ioctl).
- The use of bmap() to detect a partial page at the end of the file.
It is strongly recommended that the "dir_index" option is enabled on Ext3 filesystems being used as a cache.
=============
CONFIGURATION
=============
The cache is configured by a script in /etc/cachefilesd.conf. These commands set up cache ready for use. The following script commands are available:
(*) brun <N>%
(*) bcull <N>%
(*) bstop <N>%
Configure the culling limits. Optional. See the section on culling The defaults are 7%, 5% and 1% respectively.
(*) dir <path>
Specify the directory containing the root of the cache. Mandatory.
(*) tag <name>
Specify a tag to FS-Cache to use in distinguishing multiple caches. Optional. The default is "CacheFiles".
(*) debug <mask>
Specify a numeric bitmask to control debugging in the kernel module. Optional. The default is zero (all off).
==================
STARTING THE CACHE
==================
The cache is started by running the daemon. The daemon opens the cache proc file, configures the cache and tells it to begin caching. At that point the cache binds to fscache and the cache becomes live.
The daemon is run as follows:
/sbin/cachefilesd [-d]* [-s] [-n] [-f <configfile>]
The flags are:
(*) -d
Increase the debugging level. This can be specified multiple times and is cumulative with itself.
(*) -s
Send messages to stderr instead of syslog.
(*) -n
Don't daemonise and go into background.
(*) -f <configfile>
Use an alternative configuration file rather than the default one.
===============
THINGS TO AVOID
===============
Do not mount other things within the cache as this will cause problems. The kernel module contains its own very cut-down path walking facility that ignores mountpoints, but the daemon can't avoid them.
Do not create, rename or unlink files and directories in the cache whilst the cache is active, as this may cause the state to become uncertain.
Renaming files in the cache might make objects appear to be other objects (the filename is part of the lookup key).
Do not change or remove the extended attributes attached to cache files by the cache as this will cause the cache state management to get confused.
Do not create files or directories in the cache, lest the cache get confused or serve incorrect data.
Do not chmod files in the cache. The module creates things with minimal permissions to prevent random users being able to access them directly.
=============
CACHE CULLING
=============
The cache may need culling occasionally to make space. This involves discarding objects from the cache that have been used less recently than anything else. Culling is based on the access time of data objects. Empty directories are culled if not in use.
Cache culling is done on the basis of the percentage of blocks available in the underlying filesystem. There are three "limits":
(*) brun
If the amount of available space in the cache rises above this limit, then culling is turned off.
(*) bcull
If the amount of available space in the cache falls below this limit, then culling is started.
(*) bstop
If the amount of available space in the cache falls below this limit, then no further allocation of disk space is permitted until culling has raised the amount above this limit again.
These must be configured thusly:
0 <= bstop < bcull < brun < 100
Note that these are percentages of available space, and do _not_ appear as 100 minus the percentage displayed by the "df" program.
The userspace daemon scans the cache to build up a table of cullable objects. These are then culled in least recently used order. A new scan of the cache is started as soon as space is made in the table. Objects will be skipped if their atimes have changed or if the kernel module says it is still using them.
===============
CACHE STRUCTURE
===============
The CacheFiles module will create two directories in the directory it was given:
(*) cache/
(*) graveyard/
The active cache objects all reside in the first directory. The CacheFiles kernel module moves any retired or culled objects that it can't simply unlink to the graveyard from which the daemon will actually delete them.
The daemon uses dnotify to monitor the graveyard directory, and will delete anything that appears therein.
The module represents index objects as directories with the filename "I..." or "J...". Note that the "cache/" directory is itself a special index.
Data objects are represented as files if they have no children, or directories if they do. Their filenames all begin "D..." or "E...". If represented as a directory, data objects will have a file in the directory called "data" that actually holds the data.
Special objects are similar to data objects, except their filenames begin "S..." or "T...".
If an object has children, then it will be represented as a directory. Immediately in the representative directory are a collection of directories named for hash values of the child object keys with an '@' prepended. Into this directory, if possible, will be placed the representations of the child objects:
INDEX INDEX INDEX DATA FILES ========= ========== ================================= ================ cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400 cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400/@75/Es0g000w...DB1ry cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400/@75/Es0g000w...N22ry cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400/@75/Es0g000w...FP1ry
If the key is so long that it exceeds NAME_MAX with the decorations added on to it, then it will be cut into pieces, the first few of which will be used to make a nest of directories, and the last one of which will be the objects inside the last directory. The names of the intermediate directories will have '+' prepended:
J1223/@23/+xy...z/+kl...m/Epqr
Note that keys are raw data, and not only may they exceed NAME_MAX in size, they may also contain things like '/' and NUL characters, and so they may not be suitable for turning directly into a filename.
To handle this, CacheFiles will use a suitably printable filename directly and "base-64" encode ones that aren't directly suitable. The two versions of object filenames indicate the encoding:
OBJECT TYPE PRINTABLE ENCODED =============== =============== =============== Index "I..." "J..." Data "D..." "E..." Special "S..." "T..."
Intermediate directories are always "@" or "+" as appropriate.
Each object in the cache has an extended attribute label that holds the object type ID (required to distinguish special objects) and the auxiliary data from the netfs. The latter is used to detect stale objects in the cache and update or retire them.
Note that CacheFiles will erase from the cache any file it doesn't recognise or any file of an incorrect type (such as a FIFO file or a device file).
This documentation is added by the patch to:
Documentation/filesystems/caching/cachefiles.txt
---
Documentation/filesystems/caching/cachefiles.txt | 293 +++++ fs/Kconfig | 20 fs/Makefile | 1 fs/buffer.c | 2 fs/cachefiles/Makefile | 18 fs/cachefiles/cf-bind.c | 299 +++++ fs/cachefiles/cf-interface.c | 1326 ++++++++++++++++++++++ fs/cachefiles/cf-key.c | 157 +++ fs/cachefiles/cf-main.c | 129 ++ fs/cachefiles/cf-namei.c | 842 ++++++++++++++ fs/cachefiles/cf-proc.c | 588 ++++++++++ fs/cachefiles/cf-sysctl.c | 69 + fs/cachefiles/cf-xattr.c | 295 +++++ fs/cachefiles/internal.h | 315 +++++ fs/fcntl.c | 2 fs/file_table.c | 1 include/linux/fs.h | 2 include/linux/pagemap.h | 6 kernel/auditsc.c | 2 mm/filemap.c | 98 ++ 20 files changed, 4465 insertions(+), 0 deletions(-)
Documentation/filesystems/caching/cachefiles.txt new file +293 -0
@@ -0,0 +1,293 @@ + =============================================== + CacheFiles: CACHE ON ALREADY MOUNTED FILESYSTEM + =============================================== + +Contents: + + (*) Overview. + + (*) Requirements. + + (*) Configuration. + + (*) Starting the cache. + + (*) Things to avoid. + + +======== +OVERVIEW +======== + +CacheFiles is a caching backend that's meant to use as a cache a directory on +an already mounted filesystem of a local type (such as Ext3). + +CacheFiles uses a userspace daemon to do some of the cache management - such as +reaping stale nodes and culling. This is called cachefilesd and lives in +/sbin. + +The filesystem and data integrity of the cache are only as good as those of the +filesystem providing the backing services. Note that CacheFiles does not +attempt to journal anything since the journalling interfaces of the various +filesystems are very specific in nature. + +CacheFiles creates a proc-file - "/proc/fs/cachefiles" - that is used for +communication with the daemon. Only one thing may have this open at once, and +whilst it is open, a cache is at least partially in existence. The daemon +opens this and sends commands down it to control the cache. + +CacheFiles is currently limited to a single cache. + +CacheFiles attempts to maintain at least a certain percentage of free space on +the filesystem, shrinking the cache by culling the objects it contains to make +space if necessary - see the "Cache Culling" section. This means it can be +placed on the same medium as a live set of data, and will expand to make use of +spare space and automatically contract when the set of data requires more +space. + + +============ +REQUIREMENTS +============ + +The use of CacheFiles and its daemon requires the following features to be +available in the system and in the cache filesystem: + + - dnotify. + + - extended attributes (xattrs). + + - openat() and friends. + + - bmap() support on files in the filesystem (FIBMAP ioctl). + + - The use of bmap() to detect a partial page at the end of the file. + +It is strongly recommended that the "dir_index" option is enabled on Ext3 +filesystems being used as a cache. + + +============= +CONFIGURATION +============= + +The cache is configured by a script in /etc/cachefilesd.conf. These commands +set up cache ready for use. The following script commands are available: + + (*) brun <N>% + (*) bcull <N>% + (*) bstop <N>% + (*) frun <N>% + (*) fcull <N>% + (*) fstop <N>% + + Configure the culling limits. Optional. See the section on culling + The defaults are 7% (run), 5% (cull) and 1% (stop) respectively. + + The commands beginning with a 'b' are file space (block) limits, those + beginning with an 'f' are file count limits. + + (*) dir <path> + + Specify the directory containing the root of the cache. Mandatory. + + (*) tag <name> + + Specify a tag to FS-Cache to use in distinguishing multiple caches. + Optional. The default is "CacheFiles". + + (*) debug <mask> + + Specify a numeric bitmask to control debugging in the kernel module. + Optional. The default is zero (all off). The following values can be + OR'd into the mask to collect various information: + + 1 Turn on trace of function entry (_enter() macros) + 2 Turn on trace of function exit (_leave() macros) + 4 Turn on trace of internal debug points (_debug()) + + This mask can also be set through /proc/sys/fs/cachefiles/debug. + + +================== +STARTING THE CACHE +================== + +The cache is started by running the daemon. The daemon opens the cache proc +file, configures the cache and tells it to begin caching. At that point the +cache binds to fscache and the cache becomes live. + +The daemon is run as follows: + + /sbin/cachefilesd [-d]* [-s] [-n] [-f <configfile>] + +The flags are: + + (*) -d + + Increase the debugging level. This can be specified multiple times and + is cumulative with itself. + + (*) -s + + Send messages to stderr instead of syslog. + + (*) -n + + Don't daemonise and go into background. + + (*) -f <configfile> + + Use an alternative configuration file rather than the default one. + + +=============== +THINGS TO AVOID +=============== + +Do not mount other things within the cache as this will cause problems. The +kernel module contains its own very cut-down path walking facility that ignores +mountpoints, but the daemon can't avoid them. + +Do not create, rename or unlink files and directories in the cache whilst the +cache is active, as this may cause the state to become uncertain. + +Renaming files in the cache might make objects appear to be other objects (the +filename is part of the lookup key). + +Do not change or remove the extended attributes attached to cache files by the +cache as this will cause the cache state management to get confused. + +Do not create files or directories in the cache, lest the cache get confused or +serve incorrect data. + +Do not chmod files in the cache. The module creates things with minimal +permissions to prevent random users being able to access them directly. + + +============= +CACHE CULLING +============= + +The cache may need culling occasionally to make space. This involves +discarding objects from the cache that have been used less recently than +anything else. Culling is based on the access time of data objects. Empty +directories are culled if not in use. + +Cache culling is done on the basis of the percentage of blocks and the +percentage of files available in the underlying filesystem. There are six +"limits": + + (*) brun + (*) frun + + If the amount of free space and the number of available files in the cache + rises above both these limits, then culling is turned off. + + (*) bcull + (*) fcull + + If the amount of available space or the number of available files in the + cache falls below either of these limits, then culling is started. + + (*) bstop + (*) fstop + + If the amount of available space or the number of available files in the + cache falls below either of these limits, then no further allocation of + disk space or files is permitted until culling has raised things above + these limits again. + +These must be configured thusly: + + 0 <= bstop < bcull < brun < 100 + 0 <= fstop < fcull < frun < 100 + +Note that these are percentages of available space and available files, and do +_not_ appear as 100 minus the percentage displayed by the "df" program. + +The userspace daemon scans the cache to build up a table of cullable objects. +These are then culled in least recently used order. A new scan of the cache is +started as soon as space is made in the table. Objects will be skipped if +their atimes have changed or if the kernel module says it is still using them. + + +=============== +CACHE STRUCTURE +=============== + +The CacheFiles module will create two directories in the directory it was +given: + + (*) cache/ + + (*) graveyard/ + +The active cache objects all reside in the first directory. The CacheFiles +kernel module moves any retired or culled objects that it can't simply unlink +to the graveyard from which the daemon will actually delete them. + +The daemon uses dnotify to monitor the graveyard directory, and will delete +anything that appears therein. + + +The module represents index objects as directories with the filename "I..." or +"J...". Note that the "cache/" directory is itself a special index. + +Data objects are represented as files if they have no children, or directories +if they do. Their filenames all begin "D..." or "E...". If represented as a +directory, data objects will have a file in the directory called "data" that +actually holds the data. + +Special objects are similar to data objects, except their filenames begin +"S..." or "T...". + + +If an object has children, then it will be represented as a directory. +Immediately in the representative directory are a collection of directories +named for hash values of the child object keys with an '@' prepended. Into +this directory, if possible, will be placed the representations of the child +objects: + + INDEX INDEX INDEX DATA FILES + ========= ========== ================================= ================ + cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400 + cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400/@75/Es0g000w...DB1ry + cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400/@75/Es0g000w...N22ry + cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400/@75/Es0g000w...FP1ry + + +If the key is so long that it exceeds NAME_MAX with the decorations added on to +it, then it will be cut into pieces, the first few of which will be used to +make a nest of directories, and the last one of which will be the objects +inside the last directory. The names of the intermediate directories will have +'+' prepended: + + J1223/@23/+xy...z/+kl...m/Epqr + + +Note that keys are raw data, and not only may they exceed NAME_MAX in size, +they may also contain things like '/' and NUL characters, and so they may not +be suitable for turning directly into a filename. + +To handle this, CacheFiles will use a suitably printable filename directly and +"base-64" encode ones that aren't directly suitable. The two versions of +object filenames indicate the encoding: + + OBJECT TYPE PRINTABLE ENCODED + =============== =============== =============== + Index "I..." "J..." + Data "D..." "E..." + Special "S..." "T..." + +Intermediate directories are always "@" or "+" as appropriate. + + +Each object in the cache has an extended attribute label that holds the object +type ID (required to distinguish special objects) and the auxiliary data from +the netfs. The latter is used to detect stale objects in the cache and update +or retire them. + + +Note that CacheFiles will erase from the cache any file it doesn't recognise or +any file of an incorrect type (such as a FIFO file or a device file).
fs/Kconfig +20 -0
@@ -543,6 +543,26 @@ config FSCACHE See Documentation/filesystems/caching/fscache.txt for more information. +config CACHEFILES + tristate "Filesystem caching on files" + depends on FSCACHE + help + This permits use of a mounted filesystem as a cache for other + filesystems - primarily networking filesystems - thus allowing fast + local disk to enhance the speed of slower devices. + + See Documentation/filesystems/caching/cachefiles.txt for more + information. + +config CACHEFILES_DEBUG + bool "Debug CacheFiles" + depends on CACHEFILES + help + This permits debugging to be dynamically enabled in the filesystem + caching on files module. If this is set, the debugging output may be + enabled by setting bits in /proc/sys/fs/cachefiles/debug or by + including a debugging specifier in /etc/cachefilesd.conf. + endmenu menu "CD-ROM/DVD Filesystems"
fs/Makefile +1 -0
@@ -101,5 +101,6 @@ obj-$(CONFIG_AFS_FS) += afs/ obj-$(CONFIG_BEFS_FS) += befs/ obj-$(CONFIG_HOSTFS) += hostfs/ obj-$(CONFIG_HPPFS) += hppfs/ +obj-$(CONFIG_CACHEFILES) += cachefiles/ obj-$(CONFIG_DEBUG_FS) += debugfs/ obj-$(CONFIG_OCFS2_FS) += ocfs2/
fs/buffer.c +2 -0
@@ -184,6 +184,8 @@ int fsync_super(struct super_block *sb)return sync_blockdev (sb->s_bdev); } +EXPORT_SYMBOL (fsync_super); +/* * Write out and wait upon all dirty data associated with this * device. Filesystem data as well as the underlying block
fs/cachefiles/Makefile new file +18 -0
@@ -0,0 +1,18 @@ +# +# Makefile for caching in a mounted filesystem +# + +cachefiles-objs := \ + cf-bind.o \ + cf-interface.o \ + cf-key.o \ + cf-main.o \ + cf-namei.o \ + cf-proc.o \ + cf-xattr.o + +ifeq ($(CONFIG_SYSCTL),y) +cachefiles-objs += cf-sysctl.o +endif + +obj-$(CONFIG_CACHEFILES) := cachefiles.o
fs/cachefiles/cf-bind.c new file +299 -0
299 lines not on this page · raw message
fs/cachefiles/cf-interface.c new file +1326 -0
1,326 lines not on this page · raw message
fs/cachefiles/cf-key.c new file +157 -0
157 lines not on this page · raw message
fs/cachefiles/cf-main.c new file +129 -0
129 lines not on this page · raw message
fs/cachefiles/cf-namei.c new file +842 -0
738 lines not on this page · raw message
fs/cachefiles/cf-proc.c new file +588 -0
588 lines not on this page · raw message
fs/cachefiles/cf-sysctl.c new file +69 -0
69 lines not on this page · raw message
fs/cachefiles/cf-xattr.c new file +295 -0
295 lines not on this page · raw message
fs/cachefiles/internal.h new file +315 -0
315 lines not on this page · raw message
mm/filemap.c +98 -0
98 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
Quoted 1 line · On Wed, 30 Aug 2006 20:31:53 +0100 David Howells <[email protected]> wrote:
These patches add local caching for network filesystems such as NFS and AFS.
<fercrissake>
Not interested. Please go learn quilt, send incremental patches.
David Howells
maintainerQuoted 5 lines · Andrew Morton
Quoted 1 line
These patches add local caching for network filesystems such as NFS and AFS.
<fercrissake>
Not interested. Please go learn quilt, send incremental patches.
What's quilt able to do that StGIT can't? AFAICT from quilt's manpage, it can't mail incremental patches, so how does it help anyway?
David
Quoted 11 lines · On Wed, 30 Aug 2006 21:37:18 +0100 David Howells <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
Quoted 1 line
These patches add local caching for network filesystems such as NFS and AFS.
<fercrissake>
Not interested. Please go learn quilt, send incremental patches.
What's quilt able to do that StGIT can't? AFAICT from quilt's manpage, it can't mail incremental patches, so how does it help anyway?
It was just a suggestion. Please:
- test the patches which are presently in -mm. I don't even know if they work, and we prefer to send Linus working stuff.
- Send fine-grained incremental patches. It's OK to do complete replacement patchsets when the code is new, but this stuff is supposed to be stabilised.
It took me quite a lot of time to extract the incremental patches out of try#12 and I don't want to do it again, plus it's just another step in which errors can be introduced.
Why incremental patches?
- So we can see what changed and don't have to re-review the whole thing
- So the recipient doesn't have to re-fix the same pile of rejects each time.
- So fixes which came in via other sources don't get lost.
David Howells
maintainerQuoted 3 lines · Andrew Morton
- Send fine-grained incremental patches. It's OK to do complete replacement patchsets when the code is new, but this stuff is supposed to be stabilised.
I thought the code was still officially *new*.
As I understood things from what you said, you delegated responsibility for my patches on to Trond, who hasn't taken them yet. He has further delegated review responsibility on to Christoph, so I've been consolidating my patches to make it easier for Christoph (or whoever) to do so.
So, as I understand the situation, my patches won't go anywhere until Christoph ACKs them and Trond takes them into his tree. If this isn't so, please clarify the situation.
David
Quoted 7 lines · On Thu, 31 Aug 2006 10:58:30 +0100 David Howells <[email protected]> wrote:
Quoted 3 lines · Andrew Morton
- Send fine-grained incremental patches. It's OK to do complete replacement patchsets when the code is new, but this stuff is supposed to be stabilised.
I thought the code was still officially *new*.
It's been floating around for ages; we want it to become *old*, showing a decreasing rate of change.
Quoted 2 lines
As I understood things from what you said, you delegated responsibility for my patches on to Trond, who hasn't taken them yet.
Trond merged the large nfs-affecting ones; I don't know if he intends to handle the non-nfs bulk of the work though.
I doesn't matter, really - I'll frequently carry features with a plan to send them into a subsystem tree. Or Trond could duck it and I can send the patches direct to Linus after git-nfs has merged.
Either way, the patches which are presently in -mm are "in the pipeline" - they're the ones which people are testing (for compile, at least) and reviewing (hah). If we decide to send them into Trond then I'll add them to my things-to-spam-maintainers-with pile.
Your CONFIG_BLOCK patches did a decent job of trashing your fs-cache-make-kafs-* patches, btw. What's up with that? OK, it's sensible for people to work against mainline but the net effect of doing that is to create a mess for other people to clean up.
Quoted 3 lines
He has further delegated
review responsibility on to Christoph, so I've been consolidating my patches to make it easier for Christoph (or whoever) to do so.
These patches are quite large and complex. Frankly, I doubt if Trond or Christoph have the bandwidth to review them. It would be excellent if they were able to, but...
We have a large coder-versus-reviewer imbalance, especially in the filesystems area. cf reiser4.
Quoted 4 lines
So, as I understand the situation, my patches won't go anywhere until Christoph ACKs them and Trond takes them into his tree. If this isn't so, please clarify the situation.
If Christoph acks them then I can send them to Trond or Linus, at Trond's option.
Or I can butt out, drop the patches, wait for them to turn up in Trond's tree, at your option.
Quoted 5 lines · Andrew Morton
If Christoph acks them then I can send them to Trond or Linus, at Trond's option.
Or I can butt out, drop the patches, wait for them to turn up in Trond's tree, at your option.
I don't mind pulling them into my tree, but since Christoph had objections to earlier implementations, and specifically asked me to put a hold on the non-NFS related patches, then I'd first like an ACK from him stating that he is now happy with the way those objections have been handled in the updates.
Cheers,
Trond
David Howells
maintainerQuoted 2 lines · Andrew Morton
Trond merged the large nfs-affecting ones; I don't know if he intends to handle the non-nfs bulk of the work though.
There is one large NFS affecting patch left: namely the one that makes NFS use FS-Cache. I presume that requires Trond's agreement to merge.
Quoted 4 lines
Your
CONFIG_BLOCKpatches did a decent job of trashing your fs-cache-make-kafs-* patches, btw. What's up with that? OK, it's sensible for people to work against mainline but the net effect of doing that is to create a mess for other people to clean up.
Hmmm... Jens wanted my block patches against his tree; you wanted my NFS patches against Trond's NFS tree. I guess I should try stacking the whole lot, but against what? And who carries the fixes? A patch to fix this problem may well only apply to a tree that's the conjunction of both:-/
Quoted 5 lines
If Christoph acks them then I can send them to Trond or Linus, at Trond's option.
Or I can butt out, drop the patches, wait for them to turn up in Trond's tree, at your option.
Trond, Christoph? Any thoughts?
David
Quoted 9 lines · On Thu, 31 Aug 2006 18:42:08 +0100 David Howells <[email protected]> wrote:
Quoted 4 lines
Your
CONFIG_BLOCKpatches did a decent job of trashing your fs-cache-make-kafs-* patches, btw. What's up with that? OK, it's sensible for people to work against mainline but the net effect of doing that is to create a mess for other people to clean up.Hmmm... Jens wanted my block patches against his tree; you wanted my NFS patches against Trond's NFS tree. I guess I should try stacking the whole lot, but against what? And who carries the fixes? A patch to fix this problem may well only apply to a tree that's the conjunction of both:-/
There is no easy solution, particularly with a patch like that one which splatters itself all over the place.
The best time to do such things is against 2.6.x-rc1, when everyone is maximally-merged-up. The worst time is when we're at 2.6.x-rc5, when everyone is maximally-unmerged-up.
If we're at -rc5 and one doesn't want to wait for a few weeks then one can work against the -mm lineup, because then when we hit -rc1 and the subsystems are merged up, the proposed patch will slot in nicely with minimal breakage: no queue-jumping.
The exception to that rule is patches which move files around. Because even a single-line change in one of the affected files will cause the move-things-around patch to break, and to need somewhat risky rework. In that case, simply waiting until -rc1 is the best approach
David Howells
maintainerQuoted 4 lines · Andrew Morton
Your
CONFIG_BLOCKpatches did a decent job of trashing your fs-cache-make-kafs-* patches, btw. What's up with that? OK, it's sensible for people to work against mainline but the net effect of doing that is to create a mess for other people to clean up.
It seems the only problem in my patches is that the file address space operations have had the sync_pages op removed in a patch in the disable-block-layer patchset as it's no longer necessary.
However, as I suspect you're applying the block patches *before* the FS-Cache patches, I can't give you an incremental patch that you can apply after the other fs-cache-make-kafs-* patches, since you need to modify the first patch (fs-cache-make-kafs-use-fs-cache.patch) to get it to apply at all now.
So, I could issue a revised AFS+FS-Cache patch, would that do? Or would you rather have a patch that you can apply to the one you already have directly and modify it in place?
David
Quoted 7 lines · On Fri, 01 Sep 2006 14:08:34 +0100 David Howells <[email protected]> wrote:
Quoted 4 lines · Andrew Morton
Your
CONFIG_BLOCKpatches did a decent job of trashing your fs-cache-make-kafs-* patches, btw. What's up with that? OK, it's sensible for people to work against mainline but the net effect of doing that is to create a mess for other people to clean up.
Quoted 6 lines
It seems the only problem in my patches is that the file address space operations have had the
sync_pagesop removed in a patch in the disable-block-layer patchset as it's no longer necessary.However, as I suspect you're applying the block patches *before* the FS-Cache patches,
Yes, I stage the subsystem trees ahead of everything else. So a) things which get merged into a subsystem tree effectively do a queue-jump and b) I spend much of the merge window twiddling thumbs until the git trees have merged.
Quoted 7 lines
I can't give you an incremental patch that you can apply after the other fs-cache-make-kafs-* patches, since you need to modify the first patch (fs-cache-make-kafs-use-fs-cache.patch) to get it to apply at all now.
So, I could issue a revised AFS+FS-Cache patch, would that do? Or would you rather have a patch that you can apply to the one you already have directly and modify it in place?
I fixed it all up, I think. Please review-and-test rc5-mm1 (including hot-fixes/ contents, which grows apace).
nfs automounter submounts are still broken in Trond's tree, btw. Are we stuck?
Quoted 1 line · Andrew Morton
nfs automounter submounts are still broken in Trond's tree, btw. Are we stuck?
You mean autofs indirect maps?
I'll see if I can't get my hands on an selinux setup like yours in order to do some debugging. AFAICS, the non-selinux case works fine, though.
Cheers,
Trond
Quoted 5 lines · On Fri, 01 Sep 2006 13:00:44 -0400 Trond Myklebust…
Quoted 1 line · Andrew Morton
nfs automounter submounts are still broken in Trond's tree, btw. Are we stuck?
You mean autofs indirect maps?
I don't know that that is.
Quoted 2 lines
I'll see if I can't get my hands on an selinux setup like yours in order to do some debugging. AFAICS, the non-selinux case works fine, though.
It doesn't appear to be related to selinux.
On a stock, mostly-up-to-date FC5 installation:
echo 0 > /selinux/enforce
service autofs stop
service nfs stop
service nfs start
service autofs start
sony:/home/akpm> ls -l /net/bix/usr/src
total 0
sony:/home/akpm> showmount -e bix
Export list for bix:
/ */usr/src */mnt/export *
The automounter will mount bix:/ on /net/bix. But I am unable to get it to mount bix's /usr/src on /net/bix/usr/src.
On bix we have
bix:/home/akpm> mount/dev/sda2 on / type ext3 (rw,noatime)/dev/sdb1 on /usr/src type ext3 (rw,noatime)
...
Without git-nfs applied, /net/bix/usr/src mounts as expected.
iirc, we decided this is related to the fs-cache infrastructure work which went into git-nfs. I think David can reproduce this?
Quoted 35 lines · Andrew Morton
Quoted 5 lines · On Fri, 01 Sep 2006 13:00:44 -0400 Trond Myklebust…
Quoted 1 line · Andrew Morton
nfs automounter submounts are still broken in Trond's tree, btw. Are we stuck?
You mean autofs indirect maps?
I don't know that that is.
Quoted 2 lines
I'll see if I can't get my hands on an selinux setup like yours in order to do some debugging. AFAICS, the non-selinux case works fine, though.
It doesn't appear to be related to selinux.
On a stock, mostly-up-to-date FC5 installation:
echo 0 > /selinux/enforce
service autofs stop
service nfs stop
service nfs start
service autofs startsony:/home/akpm> ls -l /net/bix/usr/src
total 0sony:/home/akpm> showmount -e bix
Export list for bix:
/ */usr/src*/mnt/export*
Quoted 2 lines
The automounter will mount bix:/ on /net/bix. But I am unable to get it to mount bix's
/usr/srcon /net/bix/usr/src.
Is it the same symptom as before or is it that bix:/usr/src is not also being mounted?
Quoted 4 lines
Without git-nfs applied, /net/bix/usr/src mounts as expected.
iirc, we decided this is related to the fs-cache infrastructure work which went into git-nfs. I think David can reproduce this?
I'll build the latest mm kernel and try to reproduce it.
Quoted 1 line
From memory I couldn't reproduce it last time I tried.
Is there anything I need to add to rc5-mm1 for this?
Ian
Quoted 36 lines · On Sat, 02 Sep 2006 12:11:12 +0800 Ian Kent <[email protected]> wrote:
Quoted 35 lines · Andrew Morton
Quoted 5 lines · On Fri, 01 Sep 2006 13:00:44 -0400 Trond Myklebust…
Quoted 1 line · Andrew Morton
nfs automounter submounts are still broken in Trond's tree, btw. Are we stuck?
You mean autofs indirect maps?
I don't know that that is.
Quoted 2 lines
I'll see if I can't get my hands on an selinux setup like yours in order to do some debugging. AFAICS, the non-selinux case works fine, though.
It doesn't appear to be related to selinux.
On a stock, mostly-up-to-date FC5 installation:
echo 0 > /selinux/enforce
service autofs stop
service nfs stop
service nfs start
service autofs startsony:/home/akpm> ls -l /net/bix/usr/src
total 0sony:/home/akpm> showmount -e bix
Export list for bix:
/ */usr/src*/mnt/export*
Quoted 5 lines
Quoted 2 lines
The automounter will mount bix:/ on /net/bix. But I am unable to get it to mount bix's
/usr/srcon /net/bix/usr/src.Is it the same symptom as before or is it that bix:/usr/src is not also being mounted?
When this saga first started an `ls -l /net/bix' showed a corrupted dentry for /net/bix/usr. It was determined that this was SELinux-related. Fixes were made and that no longer occurs.
Now, treading on /net/bix/usr/src does not cause bix:/usr/src to be mounted at /net/bix/usr/src. Without git-nfs that mount does occur.
The present behaviour is unchanged if /selinux/enforce is set to 0.
Quoted 8 lines
Quoted 4 lines
Without git-nfs applied, /net/bix/usr/src mounts as expected.
iirc, we decided this is related to the fs-cache infrastructure work which went into git-nfs. I think David can reproduce this?
I'll build the latest mm kernel and try to reproduce it.
Quoted 1 line
From memory I couldn't reproduce it last time I tried.
Is there anything I need to add to rc5-mm1 for this?
Nope.
Quoted 2 lines · Andrew Morton
Quoted 2 lines
Quoted 2 lines
It doesn't appear to be related to selinux.
I have a festering suspicion, but no evidence yet, that this is not always the case.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
On a stock, mostly-up-to-date FC5 installation:
echo 0 > /selinux/enforce
service autofs stop
service nfs stop
service nfs start
service autofs start
I'm now setup my little system the same.
[root@raven selinux]# uname -a
Linux raven.themaw.net 2.6.18-rc5-mm1 #1 SMP Sat Sep 2 23:11:01 WST 2006 x86_64 x86_64 x86_64 GNU/Linux
[root@raven selinux]# rpm -q autofs
autofs-4.1.4-29
[root@raven selinux]# getenforce
Permissive
[root@raven selinux]# rpm -q selinux-policy
selinux-policy-2.3.7-2.fc5
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
sony:/home/akpm> ls -l /net/bix/usr/src
total 0sony:/home/akpm> showmount -e bix
Export list for bix:
/ */usr/src*/mnt/export*
Almost the same.
[root@raven selinux]# showmount -e budgie
Export list for budgie:
/ */usr/src *
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
The automounter will mount bix:/ on /net/bix. But I am unable to get it to mount bix's
/usr/srcon /net/bix/usr/src.Is it the same symptom as before or is it that bix:/usr/src is not also being mounted?
[root@raven selinux]# lsmod|grep autofs
autofs4 40776 1
I guess you haven't got the autofs module loaded instead of autofs4 by mistake.
[raven@raven ~]$ mount/dev/hda5 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw)/dev/hda6 on /home type ext3 (rw)/dev/hda7 on /work type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) automount(pid3463) on /net type autofs (rw,fd=5,pgrp=3463,minproto=2,maxproto=4)
[raven@raven ~]$ ls /net/budgie
autofs cdrom export71 initrd lib opt sbin usr
vmlinuz.old
bin dev floppy initrd.img lost+found proc sys var boot etc home initrd.img.old mnt root tmp vmlinuz
[raven@raven ~]$ mount/dev/hda5 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw)/dev/hda6 on /home type ext3 (rw)/dev/hda7 on /work type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) automount(pid3463) on /net type autofs (rw,fd=5,pgrp=3463,minproto=2,maxproto=4) budgie:/ on /net/budgie type nfs (rw,nosuid,nodev,hard,intr,addr=10.49.97.33) budgie:/usr/src on /net/budgie/usr/src type nfs (rw,nosuid,nodev,hard,intr,addr=10.49.97.33)
So I wonder what the different is between the setups?
Quoted 14 lines
When this saga first started an `ls -l /net/bix' showed a corrupted dentry for /net/bix/usr. It was determined that this was SELinux-related. Fixes were made and that no longer occurs.
Now, treading on /net/bix/usr/src does not cause bix:/usr/src to be mounted at /net/bix/usr/src. Without git-nfs that mount does occur.
The present behaviour is unchanged if /selinux/enforce is set to 0.
Quoted 4 lines
Quoted 4 lines
Without git-nfs applied, /net/bix/usr/src mounts as expected.
iirc, we decided this is related to the fs-cache infrastructure work which went into git-nfs. I think David can reproduce this?
Can you reproduce this David?
Quoted 6 lines
Quoted 4 lines
I'll build the latest mm kernel and try to reproduce it.
Quoted 1 line
From memory I couldn't reproduce it last time I tried.
Is there anything I need to add to rc5-mm1 for this?
Nope.
I'm stumped.
Ian
Quoted 2 lines · On Sun, 03 Sep 2006 14:21:30 +0800 Ian Kent <[email protected]> wrote:
I guess you haven't got the autofs module loaded instead of autofs4 by mistake.
Nope.
Quoted 1 line
So I wonder what the different is between the setups?
Beats me. Maybe cook up a debug patch?
Quoted 11 lines · Andrew Morton
Quoted 2 lines · On Sun, 03 Sep 2006 14:21:30 +0800 Ian Kent <[email protected]> wrote:
I guess you haven't got the autofs module loaded instead of autofs4 by mistake.
Nope.
Quoted 1 line
So I wonder what the different is between the setups?
Beats me. Maybe cook up a debug patch?
OK.
Could you add "--debug" to DAEMONOPTIONS in /etc/sysconfig/autofs and post the output so I can get some idea where to put the prints please.
Ian
Quoted 9 lines · On Sun, 03 Sep 2006 14:43:00 +0800 Ian Kent <[email protected]> wrote:
Quoted 8 lines · Andrew Morton
Quoted 2 lines · On Sun, 03 Sep 2006 14:21:30 +0800 Ian Kent <[email protected]> wrote:
I guess you haven't got the autofs module loaded instead of autofs4 by mistake.
Nope.
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
So I wonder what the different is between the setups?
Beats me. Maybe cook up a debug patch?
OK.
Could you add "--debug" to DAEMONOPTIONS in
/etc/sysconfig/autofsand post the output so I can get some idea where to put the prints please.
Sep 3 09:56:40 sony automount[18446]: starting automounter version 4.1.4-29, path = /net, maptype = program, mapname = /etc/auto.net Sep 3 09:56:40 sony kernel: SELinux: initialized (dev autofs, type autofs), uses genfs_contexts Sep 3 09:56:40 sony automount[18446]: using kernel protocol version 4.00 Sep 3 09:56:40 sony automount[18446]: using timeout 60 seconds; freq 15 secs Sep 3 09:56:53 sony automount[18446]: attempting to mount entry /net/bix Sep 3 09:56:53 sony kernel: SELinux: initialized (dev 0:16, type nfs), uses genfs_contexts Sep 3 09:56:53 sony automount[18453]: mount(nfs): mkdir_path /net/bix/usr/src failed: Permission denied Sep 3 09:56:53 sony automount[18453]: mount(nfs): mkdir_path /net/bix/mnt/export failed: Permission denied
Quoted 22 lines · Andrew Morton
Quoted 9 lines · On Sun, 03 Sep 2006 14:43:00 +0800 Ian Kent <[email protected]> wrote:
Quoted 8 lines · Andrew Morton
Quoted 2 lines · On Sun, 03 Sep 2006 14:21:30 +0800 Ian Kent <[email protected]> wrote:
I guess you haven't got the autofs module loaded instead of autofs4 by mistake.
Nope.
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
So I wonder what the different is between the setups?
Beats me. Maybe cook up a debug patch?
OK.
Could you add "--debug" to DAEMONOPTIONS in
/etc/sysconfig/autofsand post the output so I can get some idea where to put the prints please.
Quoted 8 lines
Sep 3 09:56:40 sony automount[18446]: starting automounter version 4.1.4-29, path = /net, maptype = program, mapname = /etc/auto.net Sep 3 09:56:40 sony kernel: SELinux: initialized (dev autofs, type autofs), uses genfs_contexts Sep 3 09:56:40 sony automount[18446]: using kernel protocol version 4.00 Sep 3 09:56:40 sony automount[18446]: using timeout 60 seconds; freq 15 secs Sep 3 09:56:53 sony automount[18446]: attempting to mount entry /net/bix Sep 3 09:56:53 sony kernel: SELinux: initialized (dev 0:16, type nfs), uses genfs_contexts Sep 3 09:56:53 sony automount[18453]: mount(nfs): mkdir_path /net/bix/usr/src failed: Permission denied Sep 3 09:56:53 sony automount[18453]: mount(nfs): mkdir_path /net/bix/mnt/export failed: Permission denied
Yes and these should be EXIST.
Could you humor me a little further and load the base selinux policy that enables rules that "dontaudit" access fails using:
semodule -b /usr/share/selinux/targeted/enableaudit.pp
and see if we get any avc messages.
Ian
Quoted 22 lines · Andrew Morton
Quoted 9 lines · On Sun, 03 Sep 2006 14:43:00 +0800 Ian Kent <[email protected]> wrote:
Quoted 8 lines · Andrew Morton
Quoted 2 lines · On Sun, 03 Sep 2006 14:21:30 +0800 Ian Kent <[email protected]> wrote:
I guess you haven't got the autofs module loaded instead of autofs4 by mistake.
Nope.
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
So I wonder what the different is between the setups?
Beats me. Maybe cook up a debug patch?
OK.
Could you add "--debug" to DAEMONOPTIONS in
/etc/sysconfig/autofsand post the output so I can get some idea where to put the prints please.
Quoted 8 lines
Sep 3 09:56:40 sony automount[18446]: starting automounter version 4.1.4-29, path = /net, maptype = program, mapname = /etc/auto.net Sep 3 09:56:40 sony kernel: SELinux: initialized (dev autofs, type autofs), uses genfs_contexts Sep 3 09:56:40 sony automount[18446]: using kernel protocol version 4.00 Sep 3 09:56:40 sony automount[18446]: using timeout 60 seconds; freq 15 secs Sep 3 09:56:53 sony automount[18446]: attempting to mount entry /net/bix Sep 3 09:56:53 sony kernel: SELinux: initialized (dev 0:16, type nfs), uses genfs_contexts Sep 3 09:56:53 sony automount[18453]: mount(nfs): mkdir_path /net/bix/usr/src failed: Permission denied Sep 3 09:56:53 sony automount[18453]: mount(nfs): mkdir_path /net/bix/mnt/export failed: Permission denied
I'm able to duplicate this now.
My mistake, my exports had the no_root_squash attribute set. Without it I get the permission denied and no mount.
Looking through the git-nfs patch I can't see how this would have changed, returning EPERM, perhaps before it gets to the EEXIST check.
Anyone else have any idea why this may has changed.
Ian
Signature · 1 line
VGER BF report: H 0
Quoted 11 lines · Andrew Morton
Quoted 5 lines · On Fri, 01 Sep 2006 13:00:44 -0400 Trond Myklebust…
Quoted 1 line · Andrew Morton
nfs automounter submounts are still broken in Trond's tree, btw. Are we stuck?
You mean autofs indirect maps?
I don't know that that is.
The mount that Andrew is a "host" type mount.
autofs gets the host name as a key and is expected to mount all filesystems exported from the host. It does this by attempting to mounting each export in shortest to longest order (to take account of nesting of the mounts).
Ian
Ian
David Howells
maintainerQuoted 4 lines · Andrew Morton
The automounter will mount bix:/ on /net/bix. But I am unable to get it to mount bix's
/usr/srcon /net/bix/usr/src.From what I can tell, the problem is that the automounter causes a dentry to
be created on the xdev transition point, but the dentry is not set up right to do in-NFS automounting on the follow_link() op.
By "xdev transition point" I mean a directory exported from the server that has a different FSID to its parent. The NFS client detects that and provides automounting facilities in the following manner:
(1) A directory dentry is set up in the superblock of the parent FSID to represent the transition point. This dentry has a follow_link() op set.
(2) When someone tries to traverse the transition point, the follow_link() op is invoked. This causes a new superblock to be created to represent the new FSID, and a root directory entry is allocated there. This new root is then mounted over the dentry set up in (1).
(3) The follow_link() op of the transit dentry returns the mountpoint dentry as if a symlink had been transited. Further transits just see the mountpoint and ignore the transit dentry at the bottom of the pile.
Note that an lstat() of the transit dentry does not cause automounting to take place because lstat() does not follow terminal symlinks, and thus does not invoke the follow_link() op.
However, when the automounter preemptively creates a dentry there with mkdir, it can install a directory dentry *without* the appropriate follow_link() op.
David
David Howells
maintainerQuoted 8 lines · Andrew Morton
sony:/home/akpm> ls -l /net/bix/usr/src
total 0sony:/home/akpm> showmount -e bix
Export list for bix:
/ */usr/src*/mnt/export*
Yes, but what's your /etc/exports now? Not all options appear to showmount.
Can you add "nohide" to the /usr/src and /mnt/export lines and "fsid=0" to the / line if you don't currently have them and try again?
Quoted 2 lines
iirc, we decided this is related to the fs-cache infrastructure work which went into git-nfs. I think David can reproduce this?
I'd only reproduced it with SELinux in enforcing mode.
Under such conditions, unless there's a readdir on the root directory, the subdirs under which exports exist will remain as incorrectly negative dentries.
The problem is a conjunction of circumstances:
(1) nfs_lookup() has a shortcut in it that skips contact with the server if we're doing a lookup with intent to create. This leaves an incorrectly negative dentry if there _is_ actually an object on the server.
(2) The mkdir procedure is aborted between the lookup() op and the mkdir() op by SELinux (see vfs_mkdir()). Note that SELinux isn't the _only_ method by which the abort can occur.
(3) One of my patches correctly assigns the security label to the automounted root dentry.
(4) SELinux then aborts the automounter's mkdir() call because the automounter does _not_ carry the correct security label to write to the NFS directory.
(5) The incorrectly set up dentry from (1) remains because the the mkdir() op is not invoked to set it right.
The only bit I added was (3), but that's not the only circumstance in which this can occur.
If, for example, I do "chmod a-w /" on the NFS server, I can see the same effects on the client without the need for SELinux to put its foot in the door. Automount does:
[pid 3838] mkdir("/net", 0555) = -1 EEXIST (File exists)
[pid 3838] stat64("/net", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
[pid 3838] mkdir("/net/trash", 0555) = -1 EEXIST (File exists)
[pid 3838] stat64("/net/trash", {st_mode=S_IFDIR|0555, st_size=1024, ...}) = 0
[pid 3838] mkdir("/net/trash/mnt", 0555) = -1 EACCES (Permission denied)And where I was listing the disputed directory, I see:
[root@andromeda ~]# ls -lad /net/trash/usr/src drwxr-xr-x 4 root root 1024 Aug 30 10:35 /net/trash/usr/src/ [root@andromeda ~]#
which isn't what I'd expect. What I'd expect is:
[root@andromeda ~]# ls -l /net/trash/usr/src total 15 drwxr-xr-x 3 root root 1024 Aug 30 10:35 debug/ -rw-r--r-- 1 root root 0 Aug 16 10:01 hello drwx------ 2 root root 12288 Aug 16 10:00 lost+found/ [root@andromeda ~]#
David
Quoted 52 lines · David Howells
Quoted 8 lines · Andrew Morton
sony:/home/akpm> ls -l /net/bix/usr/src
total 0sony:/home/akpm> showmount -e bix
Export list for bix:
/ */usr/src*/mnt/export*Yes, but what's your
/etc/exportsnow? Not all options appear to showmount.Can you add "nohide" to the
/usr/srcand/mnt/exportlines and "fsid=0" to the / line if you don't currently have them and try again?Quoted 2 lines
iirc, we decided this is related to the fs-cache infrastructure work which went into git-nfs. I think David can reproduce this?
I'd only reproduced it with SELinux in enforcing mode.
Under such conditions, unless there's a readdir on the root directory, the subdirs under which exports exist will remain as incorrectly negative dentries.
The problem is a conjunction of circumstances:
(1)
nfs_lookup()has a shortcut in it that skips contact with the server if we're doing a lookup with intent to create. This leaves an incorrectly negative dentry if there _is_ actually an object on the server.(2) The mkdir procedure is aborted between the
lookup()op and themkdir()op by SELinux (seevfs_mkdir()). Note that SELinux isn't the _only_ method by which the abort can occur.(3) One of my patches correctly assigns the security label to the automounted root dentry.
(4) SELinux then aborts the automounter's
mkdir()call because the automounter does _not_ carry the correct security label to write to the NFS directory.(5) The incorrectly set up dentry from (1) remains because the the
mkdir()op is not invoked to set it right.The only bit I added was (3), but that's not the only circumstance in which this can occur.
If, for example, I do "chmod a-w /" on the NFS server, I can see the same effects on the client without the need for SELinux to put its foot in the door. Automount does:
Quoted 5 lines
[pid 3838] mkdir("/net", 0555) = -1 EEXIST (File exists)
[pid 3838] stat64("/net", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
[pid 3838] mkdir("/net/trash", 0555) = -1 EEXIST (File exists)
[pid 3838] stat64("/net/trash", {st_mode=S_IFDIR|0555, st_size=1024, ...}) = 0
[pid 3838] mkdir("/net/trash/mnt", 0555) = -1 EACCES (Permission denied)This is the point I'm trying to make.
I'm able to reproduce this with exports that don't have "nohide". The mkdir used to return EEXIST, possibly before getting to the EACCES test. It appears to be a change in semantic behavior and I can't see where it is coming from. autofs expects an EEXIST but not an EACCES and so doesn't perform the mount. I could ignore the EACCES but that would be cheating.
Quoted 16 lines
And where I was listing the disputed directory, I see:
[root@andromeda ~]# ls -lad /net/trash/usr/src drwxr-xr-x 4 root root 1024 Aug 30 10:35 /net/trash/usr/src/ [root@andromeda ~]#which isn't what I'd expect. What I'd expect is:
[root@andromeda ~]# ls -l /net/trash/usr/src total 15 drwxr-xr-x 3 root root 1024 Aug 30 10:35 debug/ -rw-r--r-- 1 root root 0 Aug 16 10:01 hello drwx------ 2 root root 12288 Aug 16 10:00 lost+found/ [root@andromeda ~]#David
David Howells
maintainerQuoted 7 lines · Ian Kent
This is the point I'm trying to make.
I'm able to reproduce this with exports that don't have "nohide". The mkdir used to return EEXIST, possibly before getting to the EACCES test. It appears to be a change in semantic behavior and I can't see where it is coming from. autofs expects an EEXIST but not an EACCES and so doesn't perform the mount. I could ignore the EACCES but that would be cheating.
Here's something you can try: Look in fs/nfs/dir.c. Find nfs_lookup(). In there, find the following lines:
/* If we're doing an exclusive create, optimize away the lookup */ if (nfs_is_exclusive_create(dir, nd)) goto no_entry;
Comment that bit out and see what the effect it.
David
Quoted 10 lines · David Howells
Quoted 7 lines · Ian Kent
This is the point I'm trying to make.
I'm able to reproduce this with exports that don't have "nohide". The mkdir used to return EEXIST, possibly before getting to the EACCES test. It appears to be a change in semantic behavior and I can't see where it is coming from. autofs expects an EEXIST but not an EACCES and so doesn't perform the mount. I could ignore the EACCES but that would be cheating.
Quoted 8 lines
Here's something you can try: Look in
fs/nfs/dir.c. Findnfs_lookup(). In there, find the following lines:/* If we're doing an exclusive create, optimize away the lookup */ if (nfs_is_exclusive_create(dir, nd)) goto no_entry;Comment that bit out and see what the effect it.
OK. But tomorrow.
I'll let you know.
Ian
Quoted 10 lines · David Howells
Quoted 7 lines · Ian Kent
This is the point I'm trying to make.
I'm able to reproduce this with exports that don't have "nohide". The mkdir used to return EEXIST, possibly before getting to the EACCES test. It appears to be a change in semantic behavior and I can't see where it is coming from. autofs expects an EEXIST but not an EACCES and so doesn't perform the mount. I could ignore the EACCES but that would be cheating.
Quoted 8 lines
Here's something you can try: Look in
fs/nfs/dir.c. Findnfs_lookup(). In there, find the following lines:/* If we're doing an exclusive create, optimize away the lookup */ if (nfs_is_exclusive_create(dir, nd)) goto no_entry;Comment that bit out and see what the effect it.
Yes.
Commenting this out results in the expected behavior.
No doubt this is because the following NFS lookup to the server returns EEXIST which is then returned to userspace.
Ian
Quoted 18 lines · Ian Kent
Quoted 17 lines · David Howells
Quoted 7 lines · Ian Kent
This is the point I'm trying to make.
I'm able to reproduce this with exports that don't have "nohide". The mkdir used to return EEXIST, possibly before getting to the EACCES test. It appears to be a change in semantic behavior and I can't see where it is coming from. autofs expects an EEXIST but not an EACCES and so doesn't perform the mount. I could ignore the EACCES but that would be cheating.Here's something you can try: Look in
fs/nfs/dir.c. Findnfs_lookup(). In there, find the following lines:/* If we're doing an exclusive create, optimize away the lookup */ if (nfs_is_exclusive_create(dir, nd)) goto no_entry;
Quoted 7 lines
Quoted 1 line
Comment that bit out and see what the effect it.
Yes.
Commenting this out results in the expected behavior.
No doubt this is because the following NFS lookup to the server returns EEXIST which is then returned to userspace.
It also breaks open(O_EXCL). Removing that is unacceptable.
Quoted 6 lines · Ian Kent
Quoted 5 lines
[pid 3838] mkdir("/net", 0555) = -1 EEXIST (File exists) [pid 3838] stat64("/net", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 [pid 3838] mkdir("/net/trash", 0555) = -1 EEXIST (File exists) [pid 3838] stat64("/net/trash", {st_mode=S_IFDIR|0555, st_size=1024, ...}) = 0 [pid 3838] mkdir("/net/trash/mnt", 0555) = -1 EACCES (Permission denied)
Quoted 7 lines
This is the point I'm trying to make.
I'm able to reproduce this with exports that don't have "nohide". The mkdir used to return EEXIST, possibly before getting to the EACCES test. It appears to be a change in semantic behavior and I can't see where it is coming from. autofs expects an EEXIST but not an EACCES and so doesn't perform the mount. I could ignore the EACCES but that would be cheating.
Why the hell is it doing a mkdir in the first place? ...and why the hell is it not able to cope with EACCES? The latter is hardly an unlikely
reply: it means that the automounter should not be doing this in the first place, 'cos it doesn't have the privileges. That is not the same as saying that it doesn't have the privileges to do a lookup.
Trond
Quoted 16 lines · Trond Myklebust
Quoted 6 lines · Ian Kent
Quoted 5 lines
[pid 3838] mkdir("/net", 0555) = -1 EEXIST (File exists) [pid 3838] stat64("/net", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 [pid 3838] mkdir("/net/trash", 0555) = -1 EEXIST (File exists) [pid 3838] stat64("/net/trash", {st_mode=S_IFDIR|0555, st_size=1024, ...}) = 0 [pid 3838] mkdir("/net/trash/mnt", 0555) = -1 EACCES (Permission denied)Quoted 7 lines
This is the point I'm trying to make.
I'm able to reproduce this with exports that don't have "nohide". The mkdir used to return EEXIST, possibly before getting to the EACCES test. It appears to be a change in semantic behavior and I can't see where it is coming from. autofs expects an EEXIST but not an EACCES and so doesn't perform the mount. I could ignore the EACCES but that would be cheating.
Quoted 5 lines
Why the hell is it doing a mkdir in the first place? ...and why the hell is it not able to cope with EACCES? The latter is hardly an unlikely
reply: it means that the automounter should not be doing this in the first place, 'cos it doesn't have the privileges. That is not the same as saying that it doesn't have the privileges to do a lookup.
Why the hell shouldn't it be able to do an mkdir!
It is coping with the EACCESS return by not mounting the filesystem which is the correct response in this case.
Quoted 17 lines · Ian Kent
Quoted 16 lines · Trond Myklebust
Quoted 13 lines · Ian Kent
Quoted 5 lines
[pid 3838] mkdir("/net", 0555) = -1 EEXIST (File exists) [pid 3838] stat64("/net", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 [pid 3838] mkdir("/net/trash", 0555) = -1 EEXIST (File exists) [pid 3838] stat64("/net/trash", {st_mode=S_IFDIR|0555, st_size=1024, ...}) = 0 [pid 3838] mkdir("/net/trash/mnt", 0555) = -1 EACCES (Permission denied)This is the point I'm trying to make.
I'm able to reproduce this with exports that don't have "nohide". The mkdir used to return EEXIST, possibly before getting to the EACCES test. It appears to be a change in semantic behavior and I can't see where it is coming from. autofs expects an EEXIST but not an EACCES and so doesn't perform the mount. I could ignore the EACCES but that would be cheating.
Quoted 7 lines
Quoted 5 lines
Why the hell is it doing a mkdir in the first place? ...and why the hell is it not able to cope with EACCES? The latter is hardly an unlikely
reply: it means that the automounter should not be doing this in the first place, 'cos it doesn't have the privileges. That is not the same as saying that it doesn't have the privileges to do a lookup.Why the hell shouldn't it be able to do an mkdir!
Firstly, if the call to mkdir actually _was_ successful, it would be creating a new directory on the NFS server, and it would be doing so with the automounter's privileges instead of the user's privileges. Why would I want it to do that?
Secondly, and more pertinently to this case, you have no guarantee that the automounter has _any_ privileges on the server at all other than what is required to mount a filesystem. selinux is enforcing that on the client side here, but the server could just as well be set up to do the same (in fact, you could set up selinux to do the exact same thing on the server).
IOW, the automounter should just be calling stat('/net/trash/mnt'). It shouldn't be trying to create directories on the server at all.
Quoted 2 lines
It is coping with the EACCESS return by not mounting the filesystem which is the correct response in this case.
No it isn't. The directory exists. It can be looked up. There is no reason why you can't mount something on top of it.
Being permitted to do mkdir() or not has nothing to do with anything.
Quoted 38 lines · Trond Myklebust
Quoted 17 lines · Ian Kent
Quoted 16 lines · Trond Myklebust
Quoted 13 lines · Ian Kent
Quoted 5 lines
[pid 3838] mkdir("/net", 0555) = -1 EEXIST (File exists) [pid 3838] stat64("/net", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 [pid 3838] mkdir("/net/trash", 0555) = -1 EEXIST (File exists) [pid 3838] stat64("/net/trash", {st_mode=S_IFDIR|0555, st_size=1024, ...}) = 0 [pid 3838] mkdir("/net/trash/mnt", 0555) = -1 EACCES (Permission denied)This is the point I'm trying to make.
I'm able to reproduce this with exports that don't have "nohide". The mkdir used to return EEXIST, possibly before getting to the EACCES test. It appears to be a change in semantic behavior and I can't see where it is coming from. autofs expects an EEXIST but not an EACCES and so doesn't perform the mount. I could ignore the EACCES but that would be cheating.Quoted 7 lines
Quoted 5 lines
Why the hell is it doing a mkdir in the first place? ...and why the hell is it not able to cope with EACCES? The latter is hardly an unlikely
reply: it means that the automounter should not be doing this in the first place, 'cos it doesn't have the privileges. That is not the same as saying that it doesn't have the privileges to do a lookup.Why the hell shouldn't it be able to do an mkdir!
Firstly, if the call to mkdir actually _was_ successful, it would be creating a new directory on the NFS server, and it would be doing so with the automounter's privileges instead of the user's privileges. Why would I want it to do that?
Secondly, and more pertinently to this case, you have no guarantee that the automounter has _any_ privileges on the server at all other than what is required to mount a filesystem. selinux is enforcing that on the client side here, but the server could just as well be set up to do the same (in fact, you could set up selinux to do the exact same thing on the server).
Quoted 2 lines
IOW, the automounter should just be calling stat('/net/trash/mnt'). It shouldn't be trying to create directories on the server at all.
Sure but this is an old version of autofs which is in use so changing the expected behavior of a system call is not acceptable and I expect other applications may well have a problem with this also.
Quoted 8 lines
Quoted 3 lines
It is coping with the EACCESS return by not mounting the filesystem which is the correct response in this case.
No it isn't. The directory exists. It can be looked up. There is no reason why you can't mount something on top of it.
Being permitted to do
mkdir()or not has nothing to do with anything.
Agreed.
The fact that it's a mkdir is irrelevant given that nfs_lookup is returning an EACCESS instead of EEXIST this will likely affect other system calls such as "stat". I'll check this.
Quoted 3 lines · Ian Kent
Sure but this is an old version of autofs which is in use so changing the expected behavior of a system call is not acceptable and I expect other applications may well have a problem with this also.
Applications that rely on mkdir() to never return EACCES are broken. Particularly so in an selinux system (as was the case here).
Note that an ordinary application will not see this: if I do
Machine 1 Machine 2
--------- ---------
mkdir foo
mkdir foo/bar chmod 555 foo/bar foo
mkdir foo/bar
mkdir: cannot create directory
`foo/bar': File exists
i.e. as expected. So this really only affects applications that are not supposed to be calling mkdir() in the first place.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
It is coping with the EACCESS return by not mounting the filesystem which is the correct response in this case.
No it isn't. The directory exists. It can be looked up. There is no reason why you can't mount something on top of it.
Quoted 7 lines
Quoted 1 line
Being permitted to do
mkdir()or not has nothing to do with anything.Agreed.
The fact that it's a mkdir is irrelevant given that
nfs_lookupis returning an EACCESS instead of EEXIST this will likely affect other system calls such as "stat". I'll check this.
In both cases, the call to vfs_mkdir() is returning the EACCES. Not nfs_lookup. The reason why we are no longer returning EEXIST is because the intents have changed due to the patch
I suspect that reverting that patch would 'fix' the autofs bug.
Quoted 7 lines · Trond Myklebust
Quoted 3 lines · Ian Kent
Sure but this is an old version of autofs which is in use so changing the expected behavior of a system call is not acceptable and I expect other applications may well have a problem with this also.
Applications that rely on
mkdir()to never return EACCES are broken. Particularly so in an selinux system (as was the case here).
That's not quite right.
autofs v4 doesn't rely on mkdir never returning EACCESS just that it return EEXIST if the directory exists. Never the less if the behavior of stat will work in this case I'll change v4 to do it the way you suggest (as v5 does already).
Quoted 9 lines · Ian Kent
Quoted 7 lines · Trond Myklebust
Quoted 3 lines · Ian Kent
Sure but this is an old version of autofs which is in use so changing the expected behavior of a system call is not acceptable and I expect other applications may well have a problem with this also.
Applications that rely on
mkdir()to never return EACCES are broken. Particularly so in an selinux system (as was the case here).
Quoted 6 lines
That's not quite right.
autofs v4 doesn't rely on mkdir never returning EACCESS just that it return EEXIST if the directory exists. Never the less if the behavior of stat will work in this case I'll change v4 to do it the way you suggest (as v5 does already).
Aaah. Wrong again!
Although v5 doesn't attempt to mount an NFS export if the directory doesn't exist it does end up doing a mkdir later as the most common case is mounting an NFS export within an autofs filesystem or other, usually local filesystem.
Quoted 11 lines · Ian Kent
Quoted 4 lines
autofs v4 doesn't rely on mkdir never returning EACCESS just that it return EEXIST if the directory exists. Never the less if the behavior of stat will work in this case I'll change v4 to do it the way you suggest (as v5 does already).
Aaah. Wrong again!
Although v5 doesn't attempt to mount an NFS export if the directory doesn't exist it does end up doing a mkdir later as the most common case is mounting an NFS export within an autofs filesystem or other, usually local filesystem.
Then make sure that it only does this within the autofs filesystem. The value of f_type as returned by statfs() should be able to tell you if this is the case.
Quoted 9 lines · Trond Myklebust
Quoted 7 lines · Ian Kent
Quoted 4 lines
autofs v4 doesn't rely on mkdir never returning EACCESS just that it return EEXIST if the directory exists. Never the less if the behavior of stat will work in this case I'll change v4 to do it the way you suggest (as v5 does already).
Aaah. Wrong again!
Quoted 9 lines
Quoted 4 lines
Although v5 doesn't attempt to mount an NFS export if the directory doesn't exist it does end up doing a mkdir later as the most common case is mounting an NFS export within an autofs filesystem or other, usually local filesystem.
Then make sure that it only does this within the autofs filesystem. The value of
f_typeas returned bystatfs()should be able to tell you if this is the case.
Yep. I've done that elsewhere.
Ian
David Howells
maintainerQuoted 4 lines · Ian Kent
autofs v4 doesn't rely on mkdir never returning EACCESS just that it return EEXIST if the directory exists. Never the less if the behavior of stat will work in this case I'll change v4 to do it the way you suggest (as v5 does already).
As long as you don't rely on stat...mkdir working. That can go wrong if the dentry gets booted from the dcache by memory pressure in the "...".
David
Quoted 9 lines · David Howells
Quoted 4 lines · Ian Kent
autofs v4 doesn't rely on mkdir never returning EACCESS just that it return EEXIST if the directory exists. Never the less if the behavior of stat will work in this case I'll change v4 to do it the way you suggest (as v5 does already).
As long as you don't rely on stat...mkdir working. That can go wrong if the dentry gets booted from the dcache by memory pressure in the "...".
I'm not clear on your point here.
If I stat a path and it exists then all is good and I'm done. If I stat a path and I get something other than ENOENT then all is bad and I return fail.
Otherwise I can just attempt to create the directory and fail if all is bad with that.
This approach works in the current situation and would work for the other situations in which that same process is used, not just for NFS filesystems.
Quoted 6 lines
David
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
David Howells
maintainerQuoted 4 lines · Ian Kent
Quoted 2 lines
As long as you don't rely on stat...mkdir working. That can go wrong if the dentry gets booted from the dcache by memory pressure in the "...".
I'm not clear on your point here.
I was wondering if you were going to rely on stat() forcing the dentry to be correctly initialised before you did mkdir(), but it seems not.
Quoted 5 lines
If I stat a path and it exists then all is good and I'm done. If I stat a path and I get something other than ENOENT then all is bad and I return fail.
Otherwise I can just attempt to create the directory and fail if all is bad with that.
Okay, I suppose. But that still doesn't seem to deal with the case of creating a directory on the client that then overlays a symlink on the server that you can't yet access.
You may also get ENOENT because you stat a symlink, though you'll get EEXIST from mkdir, even if there's nothing at the far end.
David
Quoted 16 lines · David Howells
Quoted 4 lines · Ian Kent
Quoted 2 lines
As long as you don't rely on stat...mkdir working. That can go wrong if the dentry gets booted from the dcache by memory pressure in the "...".
I'm not clear on your point here.
I was wondering if you were going to rely on
stat()forcing the dentry to be correctly initialised before you didmkdir(), but it seems not.Quoted 5 lines
If I stat a path and it exists then all is good and I'm done. If I stat a path and I get something other than ENOENT then all is bad and I return fail.
Otherwise I can just attempt to create the directory and fail if all is bad with that.
Quoted 3 lines
Okay, I suppose. But that still doesn't seem to deal with the case of creating a directory on the client that then overlays a symlink on the server that you can't yet access.
We're largely performing user space actions at this point. Wouldn't the subsequent call to mount(8) catch that?
Quoted 3 lines
You may also get ENOENT because you stat a symlink, though you'll get EEXIST from mkdir, even if there's nothing at the far end.
Don't think this is something I need to care about either.
I can't mount on a symlink so the error return would be the correct way to deal with it.
Ian
David Howells
maintainerQuoted 6 lines · Ian Kent
Quoted 3 lines
Okay, I suppose. But that still doesn't seem to deal with the case of creating a directory on the client that then overlays a symlink on the server that you can't yet access.
We're largely performing user space actions at this point. Wouldn't the subsequent call to mount(8) catch that?
Not if you've already caused the NFS filesystem to create a "dummy" dentry that's a directory because you couldn't see that what that name corresponds to on the server is actually a symlink.
Quoted 6 lines
Quoted 2 lines
You may also get ENOENT because you stat a symlink, though you'll get EEXIST from mkdir, even if there's nothing at the far end.
Don't think this is something I need to care about either.
I can't mount on a symlink so the error return would be the correct way to deal with it.
But you might have to transit a symlink to reach the mountpoint.
David
Quoted 6 lines · David Howells
Quoted 4 lines · Ian Kent
Quoted 3 lines
Okay, I suppose. But that still doesn't seem to deal with the case of creating a directory on the client that then overlays a symlink on the server that you can't yet access.
Quoted 6 lines
Quoted 2 lines
We're largely performing user space actions at this point. Wouldn't the subsequent call to mount(8) catch that?
Not if you've already caused the NFS filesystem to create a "dummy" dentry that's a directory because you couldn't see that what that name corresponds to on the server is actually a symlink.
Shouldn't stat tell me if this is a symlink?
Quoted 9 lines
Quoted 7 lines
Quoted 3 lines
You may also get ENOENT because you stat a symlink, though you'll get EEXIST from mkdir, even if there's nothing at the far end.
Don't think this is something I need to care about either.
I can't mount on a symlink so the error return would be the correct way to deal with it.But you might have to transit a symlink to reach the mountpoint.
Mmmm ... thinking ....
Ian
David Howells
maintainerQuoted 5 lines · Ian Kent
Quoted 3 lines
Not if you've already caused the NFS filesystem to create a "dummy" dentry that's a directory because you couldn't see that what that name corresponds to on the server is actually a symlink.
Shouldn't stat tell me if this is a symlink?
You may not be able to find out from the server what it is you're trying to deal with because you may not have permission to do so, or because whatever it is may not be exported. The first may be the trickiest to deal with because the MOUNT service for NFS2 and NFS3 can jump you over bits of the path you can't otherwise access.
The problem actually comes when the conditions on the server change; perhaps an intermediate directory is made accessible on the server and suddenly the client can see inside of it. It may then find out that what it had assumed to be directories, and what it had set dummy directory dentries up for, aren't.
David
Quoted 14 lines · David Howells
Quoted 5 lines · Ian Kent
Quoted 3 lines
Not if you've already caused the NFS filesystem to create a "dummy" dentry that's a directory because you couldn't see that what that name corresponds to on the server is actually a symlink.
Shouldn't stat tell me if this is a symlink?
You may not be able to find out from the server what it is you're trying to deal with because you may not have permission to do so, or because whatever it is may not be exported. The first may be the trickiest to deal with because the MOUNT service for NFS2 and NFS3 can jump you over bits of the path you can't otherwise access.
Quoted 4 lines
The problem actually comes when the conditions on the server change; perhaps an intermediate directory is made accessible on the server and suddenly the client can see inside of it. It may then find out that what it had assumed to be directories, and what it had set dummy directory dentries up for, aren't.
It really doesn't matter whether there is a symlink or not. automounters should _not_ be trying to create directories on any filesystem other than the autofs filesystem itself.
David Howells
maintainerQuoted 3 lines · Trond Myklebust
It really doesn't matter whether there is a symlink or not. automounters should _not_ be trying to create directories on any filesystem other than the autofs filesystem itself.
Yes, I agree.
David
Quoted 7 lines · David Howells
Quoted 3 lines · Trond Myklebust
It really doesn't matter whether there is a symlink or not. automounters should _not_ be trying to create directories on any filesystem other than the autofs filesystem itself.
Yes, I agree.
Not really.
What about multiple recursive bind mounts?
What about the initial directory for the autofs mount itself?
What about the case where a admin expects autofs to create these directories for map entries that have multiple offsets.
As I've said before in version 5 I'm saying that it is a requirement that the the directories already exist in this case but in version 4 people may have become accustomed to this behavior and right or wrong this type of change shouldn't be made without warning to the users or possibly not made at all.
Ian
Quoted 14 lines · Ian Kent
Quoted 7 lines · David Howells
Quoted 3 lines · Trond Myklebust
It really doesn't matter whether there is a symlink or not. automounters should _not_ be trying to create directories on any filesystem other than the autofs filesystem itself.
Yes, I agree.
Not really.
What about multiple recursive bind mounts?
What about the initial directory for the autofs mount itself?
Quoted 8 lines
What about the case where a admin expects autofs to create these directories for map entries that have multiple offsets.
As I've said before in version 5 I'm saying that it is a requirement that the the directories already exist in this case but in version 4 people may have become accustomed to this behavior and right or wrong this type of change shouldn't be made without warning to the users or possibly not made at all.
What part of the phrase "security risk" are you failing to understand? If anybody out there is actually relying on having an automounter daemon that is running with root privileges try to create directories on remote servers on the basis of the output of the 'showmount' command, then they need saving from themselves.
Quoted 24 lines · Trond Myklebust
Quoted 14 lines · Ian Kent
Quoted 7 lines · David Howells
Quoted 3 lines · Trond Myklebust
It really doesn't matter whether there is a symlink or not. automounters should _not_ be trying to create directories on any filesystem other than the autofs filesystem itself.
Yes, I agree.
Not really.
What about multiple recursive bind mounts?
What about the initial directory for the autofs mount itself?Quoted 8 lines
What about the case where a admin expects autofs to create these directories for map entries that have multiple offsets.
As I've said before in version 5 I'm saying that it is a requirement that the the directories already exist in this case but in version 4 people may have become accustomed to this behavior and right or wrong this type of change shouldn't be made without warning to the users or possibly not made at all.
Quoted 6 lines
What part of the phrase "security risk" are you failing to understand? If anybody out there is actually relying on having an automounter daemon that is running with root privileges try to create directories on remote servers on the basis of the output of the 'showmount' command, then they need saving from themselves.
Haha ... point taken.
David Howells
maintainerQuoted 1 line · Ian Kent
Why the hell shouldn't it be able to do an mkdir!
The use of mkdir in this manner has to be considered a bug. You don't know that the object at that name on the server is a directory. It might be a symbolic link.
David
Quoted 7 lines · David Howells
Quoted 1 line · Ian Kent
Why the hell shouldn't it be able to do an mkdir!
The use of mkdir in this manner has to be considered a bug. You don't know that the object at that name on the server is a directory. It might be a symbolic link.
Fair call.
David Howells
maintainerQuoted 1 line · Trond Myklebust
Why the hell is it doing a mkdir in the first place?
I think the problems it is solving are these:
(1) What happens if "/" is _not_ exported?
(2) What happens if some intermediate directory (say "/usr") is not accessible?
In the first case, the automounter just makes "usr" and "usr/src", say, in the autofs filesystem, and then mounts server:/usr/src on that.
In the second case, the automounter relies on NFS letting it make intervening directories it couldn't otherwise access to span the gap between "/" and "src".
David
Quoted 6 lines · David Howells
Quoted 1 line · Trond Myklebust
Why the hell is it doing a mkdir in the first place?
I think the problems it is solving are these:
Quoted 8 lines
(1) What happens if "/" is _not_ exported?
(2) What happens if some intermediate directory (say "/usr") is not accessible?
In the first case, the automounter just makes "usr" and "usr/src", say, in the autofs filesystem, and then mounts server:/usr/src on that.
That is fine. As long as it is doing so in the _autofs_ filesystem. A call to 'stat()' should suffice to tell if this is the case.
Quoted 3 lines
In the second case, the automounter relies on NFS letting it make intervening directories it couldn't otherwise access to span the gap between "/" and "src".
If the directory isn't accessible, then autofs shouldn't be trying to override that. It certainly shouldn't be doing so by trying to create the directory.
Quoted 13 lines · Trond Myklebust
Quoted 12 lines · David Howells
Quoted 1 line · Trond Myklebust
Why the hell is it doing a mkdir in the first place?
I think the problems it is solving are these:
(1) What happens if "/" is _not_ exported?
(2) What happens if some intermediate directory (say "/usr") is not accessible?
Quoted 5 lines
Quoted 2 lines
In the first case, the automounter just makes "usr" and "usr/src", say, in the autofs filesystem, and then mounts server:/usr/src on that.
That is fine. As long as it is doing so in the _autofs_ filesystem. A call to '
stat()' should suffice to tell if this is the case.
I meant statfs().
David Howells
maintainerQuoted 4 lines · Trond Myklebust
Quoted 2 lines
That is fine. As long as it is doing so in the _autofs_ filesystem. A call to '
stat()' should suffice to tell if this is the case.I meant
statfs().
stat() too: st_dev.
David
Quoted 19 lines · Trond Myklebust
Quoted 6 lines · David Howells
Quoted 1 line · Trond Myklebust
Why the hell is it doing a mkdir in the first place?
I think the problems it is solving are these:
Quoted 8 lines
(1) What happens if "/" is _not_ exported?
(2) What happens if some intermediate directory (say "/usr") is not accessible?
In the first case, the automounter just makes "usr" and "usr/src", say, in the autofs filesystem, and then mounts server:/usr/src on that.
That is fine. As long as it is doing so in the _autofs_ filesystem. A call to '
stat()' should suffice to tell if this is the case.
Quoted 8 lines
Quoted 3 lines
In the second case, the automounter relies on NFS letting it make intervening directories it couldn't otherwise access to span the gap between "/" and "src".
If the directory isn't accessible, then autofs shouldn't be trying to override that. It certainly shouldn't be doing so by trying to create the directory.
In the case above the directory is in the autofs filesystem and so needs to be created.
Ian
Quoted 64 lines · David Howells
Quoted 8 lines · Andrew Morton
sony:/home/akpm> ls -l /net/bix/usr/src
total 0sony:/home/akpm> showmount -e bix
Export list for bix:
/ */usr/src*/mnt/export*Yes, but what's your
/etc/exportsnow? Not all options appear to showmount.Can you add "nohide" to the
/usr/srcand/mnt/exportlines and "fsid=0" to the / line if you don't currently have them and try again?Quoted 2 lines
iirc, we decided this is related to the fs-cache infrastructure work which went into git-nfs. I think David can reproduce this?
I'd only reproduced it with SELinux in enforcing mode.
Under such conditions, unless there's a readdir on the root directory, the subdirs under which exports exist will remain as incorrectly negative dentries.
The problem is a conjunction of circumstances:
(1)
nfs_lookup()has a shortcut in it that skips contact with the server if we're doing a lookup with intent to create. This leaves an incorrectly negative dentry if there _is_ actually an object on the server.(2) The mkdir procedure is aborted between the
lookup()op and themkdir()op by SELinux (seevfs_mkdir()). Note that SELinux isn't the _only_ method by which the abort can occur.(3) One of my patches correctly assigns the security label to the automounted root dentry.
(4) SELinux then aborts the automounter's
mkdir()call because the automounter does _not_ carry the correct security label to write to the NFS directory.(5) The incorrectly set up dentry from (1) remains because the the
mkdir()op is not invoked to set it right.The only bit I added was (3), but that's not the only circumstance in which this can occur.
If, for example, I do "chmod a-w /" on the NFS server, I can see the same effects on the client without the need for SELinux to put its foot in the door. Automount does:
[pid 3838] mkdir("/net", 0555) = -1 EEXIST (File exists) [pid 3838] stat64("/net", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 [pid 3838] mkdir("/net/trash", 0555) = -1 EEXIST (File exists) [pid 3838] stat64("/net/trash", {st_mode=S_IFDIR|0555, st_size=1024, ...}) = 0 [pid 3838] mkdir("/net/trash/mnt", 0555) = -1 EACCES (Permission denied)And where I was listing the disputed directory, I see:
[root@andromeda ~]# ls -lad /net/trash/usr/src drwxr-xr-x 4 root root 1024 Aug 30 10:35 /net/trash/usr/src/ [root@andromeda ~]#
Quoted 8 lines
which isn't what I'd expect. What I'd expect is:
[root@andromeda ~]# ls -l /net/trash/usr/src total 15 drwxr-xr-x 3 root root 1024 Aug 30 10:35 debug/ -rw-r--r-- 1 root root 0 Aug 16 10:01 hello drwx------ 2 root root 12288 Aug 16 10:00 lost+found/ [root@andromeda ~]#
One way to fix this is to simply not hash the dentry when we're doing the O_EXCL intent optimisation, but rather to only hash it _after_ we've successfully created the file on the server. Something like the attached patch ought to do it.
Note, though, that this will not fix the autofs problem: autofs is trying to perform a totally unnecessary mkdir(), and is giving up when it is told that SELinux won't authorise that particular operation. This is clearly an autofs bug...
Cheers,
Trond
---
fs/nfs/dir.c | 13 ++++++++++--- 1 files changed, 10 insertions(+), 3 deletions(-)
fs/nfs/dir.c +10 -3
@@ -904,9 +904,14 @@ static struct dentry *nfs_lookup(structlock_kernel (); -/* If we're doing an exclusive create, optimize away the lookup */ -if (nfs_is_exclusive_create (dir, nd)) - goto no_entry; +/* +* If we're doing an exclusive create, optimize away the lookup +* but don't hash the dentry. +*/ +if (nfs_is_exclusive_create (dir, nd)) { +d_instantiate (dentry,NULL ); + goto out_unlock; + } error =NFS_PROTO (dir)->lookup (dir, &dentry->d_name, &fhandle, &fattr);if (error == -ENOENT )
@@ -1161,6 +1166,8 @@ int nfs_instantiate(struct dentry *dentrif (IS_ERR (inode))return error;d_instantiate (dentry, inode); +if (d_unhashed (dentry)) +d_rehash (dentry);return 0 ; }
Quoted 79 lines · Trond Myklebust
Quoted 64 lines · David Howells
Quoted 8 lines · Andrew Morton
sony:/home/akpm> ls -l /net/bix/usr/src
total 0sony:/home/akpm> showmount -e bix
Export list for bix:
/ */usr/src*/mnt/export*Yes, but what's your
/etc/exportsnow? Not all options appear to showmount.Can you add "nohide" to the
/usr/srcand/mnt/exportlines and "fsid=0" to the / line if you don't currently have them and try again?Quoted 2 lines
iirc, we decided this is related to the fs-cache infrastructure work which went into git-nfs. I think David can reproduce this?
I'd only reproduced it with SELinux in enforcing mode.
Under such conditions, unless there's a readdir on the root directory, the subdirs under which exports exist will remain as incorrectly negative dentries.
The problem is a conjunction of circumstances:
(1)
nfs_lookup()has a shortcut in it that skips contact with the server if we're doing a lookup with intent to create. This leaves an incorrectly negative dentry if there _is_ actually an object on the server.(2) The mkdir procedure is aborted between the
lookup()op and themkdir()op by SELinux (seevfs_mkdir()). Note that SELinux isn't the _only_ method by which the abort can occur.(3) One of my patches correctly assigns the security label to the automounted root dentry.
(4) SELinux then aborts the automounter's
mkdir()call because the automounter does _not_ carry the correct security label to write to the NFS directory.(5) The incorrectly set up dentry from (1) remains because the the
mkdir()op is not invoked to set it right.The only bit I added was (3), but that's not the only circumstance in which this can occur.
If, for example, I do "chmod a-w /" on the NFS server, I can see the same effects on the client without the need for SELinux to put its foot in the door. Automount does:
[pid 3838] mkdir("/net", 0555) = -1 EEXIST (File exists) [pid 3838] stat64("/net", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 [pid 3838] mkdir("/net/trash", 0555) = -1 EEXIST (File exists) [pid 3838] stat64("/net/trash", {st_mode=S_IFDIR|0555, st_size=1024, ...}) = 0 [pid 3838] mkdir("/net/trash/mnt", 0555) = -1 EACCES (Permission denied)And where I was listing the disputed directory, I see:
[root@andromeda ~]# ls -lad /net/trash/usr/src drwxr-xr-x 4 root root 1024 Aug 30 10:35 /net/trash/usr/src/ [root@andromeda ~]#Quoted 8 lines
which isn't what I'd expect. What I'd expect is:
[root@andromeda ~]# ls -l /net/trash/usr/src total 15 drwxr-xr-x 3 root root 1024 Aug 30 10:35 debug/ -rw-r--r-- 1 root root 0 Aug 16 10:01 hello drwx------ 2 root root 12288 Aug 16 10:00 lost+found/ [root@andromeda ~]#One way to fix this is to simply not hash the dentry when we're doing the
O_EXCLintent optimisation, but rather to only hash it _after_ we've successfully created the file on the server. Something like the attached patch ought to do it.
Quoted 4 lines
Note, though, that this will not fix the autofs problem: autofs is trying to perform a totally unnecessary
mkdir(), and is giving up when it is told that SELinux won't authorise that particular operation. This is clearly an autofs bug...
selinux is not involved in this senario.
Quoted 80 lines · Ian Kent
Quoted 79 lines · Trond Myklebust
Quoted 64 lines · David Howells
Quoted 8 lines · Andrew Morton
sony:/home/akpm> ls -l /net/bix/usr/src
total 0sony:/home/akpm> showmount -e bix
Export list for bix:
/ */usr/src*/mnt/export*Yes, but what's your
/etc/exportsnow? Not all options appear to showmount.Can you add "nohide" to the
/usr/srcand/mnt/exportlines and "fsid=0" to the / line if you don't currently have them and try again?Quoted 2 lines
iirc, we decided this is related to the fs-cache infrastructure work which went into git-nfs. I think David can reproduce this?
I'd only reproduced it with SELinux in enforcing mode.
Under such conditions, unless there's a readdir on the root directory, the subdirs under which exports exist will remain as incorrectly negative dentries.
The problem is a conjunction of circumstances:
(1)
nfs_lookup()has a shortcut in it that skips contact with the server if we're doing a lookup with intent to create. This leaves an incorrectly negative dentry if there _is_ actually an object on the server.(2) The mkdir procedure is aborted between the
lookup()op and themkdir()op by SELinux (seevfs_mkdir()). Note that SELinux isn't the _only_ method by which the abort can occur.(3) One of my patches correctly assigns the security label to the automounted root dentry.
(4) SELinux then aborts the automounter's
mkdir()call because the automounter does _not_ carry the correct security label to write to the NFS directory.(5) The incorrectly set up dentry from (1) remains because the the
mkdir()op is not invoked to set it right.The only bit I added was (3), but that's not the only circumstance in which this can occur.
If, for example, I do "chmod a-w /" on the NFS server, I can see the same effects on the client without the need for SELinux to put its foot in the door. Automount does:
[pid 3838] mkdir("/net", 0555) = -1 EEXIST (File exists) [pid 3838] stat64("/net", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 [pid 3838] mkdir("/net/trash", 0555) = -1 EEXIST (File exists) [pid 3838] stat64("/net/trash", {st_mode=S_IFDIR|0555, st_size=1024, ...}) = 0 [pid 3838] mkdir("/net/trash/mnt", 0555) = -1 EACCES (Permission denied)And where I was listing the disputed directory, I see:
[root@andromeda ~]# ls -lad /net/trash/usr/src drwxr-xr-x 4 root root 1024 Aug 30 10:35 /net/trash/usr/src/ [root@andromeda ~]#Quoted 8 lines
which isn't what I'd expect. What I'd expect is:
[root@andromeda ~]# ls -l /net/trash/usr/src total 15 drwxr-xr-x 3 root root 1024 Aug 30 10:35 debug/ -rw-r--r-- 1 root root 0 Aug 16 10:01 hello drwx------ 2 root root 12288 Aug 16 10:00 lost+found/ [root@andromeda ~]#One way to fix this is to simply not hash the dentry when we're doing the
O_EXCLintent optimisation, but rather to only hash it _after_ we've successfully created the file on the server. Something like the attached patch ought to do it.
Quoted 6 lines
Quoted 4 lines
Note, though, that this will not fix the autofs problem: autofs is trying to perform a totally unnecessary
mkdir(), and is giving up when it is told that SELinux won't authorise that particular operation. This is clearly an autofs bug...selinux is not involved in this senario.
Right. David rigged the NFS server to produce the same result as selinux on autofs. He is demonstrating that autofs is wrong to call mkdir().
As I said above, my patch fixes the problem that David noted in (1) above, namely that a negative dentry in incorrectly instantiated when the mkdir() (or other exclusive create) operation fails due to a permission error.
Quoted 74 lines · Trond Myklebust
Quoted 64 lines · David Howells
Quoted 8 lines · Andrew Morton
sony:/home/akpm> ls -l /net/bix/usr/src
total 0sony:/home/akpm> showmount -e bix
Export list for bix:
/ */usr/src*/mnt/export*Yes, but what's your
/etc/exportsnow? Not all options appear to showmount.Can you add "nohide" to the
/usr/srcand/mnt/exportlines and "fsid=0" to the / line if you don't currently have them and try again?Quoted 2 lines
iirc, we decided this is related to the fs-cache infrastructure work which went into git-nfs. I think David can reproduce this?
I'd only reproduced it with SELinux in enforcing mode.
Under such conditions, unless there's a readdir on the root directory, the subdirs under which exports exist will remain as incorrectly negative dentries.
The problem is a conjunction of circumstances:
(1)
nfs_lookup()has a shortcut in it that skips contact with the server if we're doing a lookup with intent to create. This leaves an incorrectly negative dentry if there _is_ actually an object on the server.(2) The mkdir procedure is aborted between the
lookup()op and themkdir()op by SELinux (seevfs_mkdir()). Note that SELinux isn't the _only_ method by which the abort can occur.(3) One of my patches correctly assigns the security label to the automounted root dentry.
(4) SELinux then aborts the automounter's
mkdir()call because the automounter does _not_ carry the correct security label to write to the NFS directory.(5) The incorrectly set up dentry from (1) remains because the the
mkdir()op is not invoked to set it right.The only bit I added was (3), but that's not the only circumstance in which this can occur.
If, for example, I do "chmod a-w /" on the NFS server, I can see the same effects on the client without the need for SELinux to put its foot in the door. Automount does:
[pid 3838] mkdir("/net", 0555) = -1 EEXIST (File exists) [pid 3838] stat64("/net", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 [pid 3838] mkdir("/net/trash", 0555) = -1 EEXIST (File exists) [pid 3838] stat64("/net/trash", {st_mode=S_IFDIR|0555, st_size=1024, ...}) = 0 [pid 3838] mkdir("/net/trash/mnt", 0555) = -1 EACCES (Permission denied)And where I was listing the disputed directory, I see:
[root@andromeda ~]# ls -lad /net/trash/usr/src drwxr-xr-x 4 root root 1024 Aug 30 10:35 /net/trash/usr/src/ [root@andromeda ~]#Quoted 8 lines
which isn't what I'd expect. What I'd expect is:
[root@andromeda ~]# ls -l /net/trash/usr/src total 15 drwxr-xr-x 3 root root 1024 Aug 30 10:35 debug/ -rw-r--r-- 1 root root 0 Aug 16 10:01 hello drwx------ 2 root root 12288 Aug 16 10:00 lost+found/ [root@andromeda ~]#
Quoted 4 lines
One way to fix this is to simply not hash the dentry when we're doing the
O_EXCLintent optimisation, but rather to only hash it _after_ we've successfully created the file on the server. Something like the attached patch ought to do it.
No.
This patch simply marks the dentry negative and returns ENOMEM from the lookup which, as would be expected, results in this error being returned to userspace.
Quoted 10 lines · Ian Kent
Quoted 4 lines
One way to fix this is to simply not hash the dentry when we're doing the
O_EXCLintent optimisation, but rather to only hash it _after_ we've successfully created the file on the server. Something like the attached patch ought to do it.No.
This patch simply marks the dentry negative and returns ENOMEM from the lookup which, as would be expected, results in this error being returned to userspace.
Oops. You are right. I forgot to set res=NULL...
---
fs/nfs/dir.c | 14 +++++++++++--- 1 files changed, 11 insertions(+), 3 deletions(-)
fs/nfs/dir.c +11 -3
@@ -904,9 +904,15 @@ static struct dentry *nfs_lookup(structlock_kernel (); -/* If we're doing an exclusive create, optimize away the lookup */ -if (nfs_is_exclusive_create (dir, nd)) - goto no_entry; +/* +* If we're doing an exclusive create, optimize away the lookup +* but don't hash the dentry. +*/ +if (nfs_is_exclusive_create (dir, nd)) { +d_instantiate (dentry,NULL ); + res =NULL ; + goto out_unlock; + } error =NFS_PROTO (dir)->lookup (dir, &dentry->d_name, &fhandle, &fattr);if (error == -ENOENT )
@@ -1161,6 +1167,8 @@ int nfs_instantiate(struct dentry *dentrif (IS_ERR (inode))return error;d_instantiate (dentry, inode); +if (d_unhashed (dentry)) +d_rehash (dentry);return 0 ; }
Quoted 7 lines · Trond Myklebust
Quoted 5 lines · Ian Kent
Quoted 4 lines
One way to fix this is to simply not hash the dentry when we're doing the
O_EXCLintent optimisation, but rather to only hash it _after_ we've successfully created the file on the server. Something like the attached patch ought to do it.
Quoted 7 lines
Quoted 5 lines
No.
This patch simply marks the dentry negative and returns ENOMEM from the lookup which, as would be expected, results in this error being returned to userspace.
Oops. You are right. I forgot to set res=NULL...
Now returns EPERM.
Quoted 10 lines · Ian Kent
Quoted 9 lines · Trond Myklebust
Quoted 7 lines · Ian Kent
Quoted 4 lines
One way to fix this is to simply not hash the dentry when we're doing the
O_EXCLintent optimisation, but rather to only hash it _after_ we've successfully created the file on the server. Something like the attached patch ought to do it.No.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
This patch simply marks the dentry negative and returns ENOMEM from the lookup which, as would be expected, results in this error being returned to userspace.
Oops. You are right. I forgot to set res=NULL...
Now returns EPERM.
Sorry that's EACCES.
David Howells
maintainerQuoted 1 line · Andrew Morton
I fixed it all up, I think. Please review-and-test rc5-mm1
It seems to work okay, and the you seem to have made a change to fs/afs/file.c that matches the one that I've made, so thanks.
Quoted 1 line
(including hot-fixes/ contents, which grows apace).
(*) drivers-md-kconfig-fix-block-dependency.patch
I ACK'd Adrian's patch, and the change it makes appears in Jens's block GIT tree, even if the patch itself does not AFAICT.
David