Skip to contentSkip to conversation

Support for stackable file systems on top of nfs

RFC Nikolai Joukov 5 participants 6 messages Nov 14, 2005 VFS

lore
Share & tools
lore
#0 lore raw
Quoted 10 lines
Quoted 5 lines
Quoted 4 lines · John T. Kohl

Other than i_mapping/f_mapping, I don't think it's possible right now for stacking file systems to handle the address_space operations in our layer *and* share the same pages with the backing-store, since the struct pages are attached to the address space via file->f_mapping.

Charles> At Stony Brook, we've come across similar problems. It is relatively Charles> easy to double cache, but inefficient. It is also relatively easy to Charles> single-cache, but then you don't get to intercept any of these Charles> interesting operations. Getting both at once is tricky.

Quoted 6 lines

We currently do single-caching, by passing on the mmap operation to the backing store (swapping in the backing store file for vma->vm_file). (We do the equivalent in our MVFS built for vnode kernels.) Swapping the vm file is mostly workable, but we do have to be a bit too knowledgable about the innards of file mapping and do some things to accomodate the actions taken after fop->mmap is called.

What we are discussing here is only the tip of the iceberg. We are discussing the simplest case:
1) Page N of the upper filesystem's file corresponds to page N of the lower file.
2) No page processing is necessary.
In that case we use either the technique described above
(http://lxr.fsl.cs.sunysb.edu/fistgen/source/templates/Linux-2.6/file.c#L458)
or CODA's i_mapping/f_mapping way. Both have their own problems.

However, for most stackable filesystems we need to intercept the writepage/readpage/prepare_write/commit_write operations. Also, page N of the upper filesystem may correspond to some other page M of the lower filesystem. For example, this is the case for fan-out stackable filesystems (Unionfs, and RAID-like filesystems). There are dozens of practical stackable filesystems were we have to double-cache only because Linux VFS does not allow to intercept the page-based operations *and* avoid double caching. I would like to point out here that neither *BSD nor Windows stackable filesystems have this problem. To solve the problem, VFS should allow stackable filesystems to 1) do something (calculate checksumms, calculate parity for filesystem-level RAIDs, etc.) inside of a stackable filesystem's readpage/writepage/..., 2) call lower filesystem's readpage/writepage/... passing *any* page to these lower functions. The page passed below may be a page of the lower filesystem to get double caching, or an upper page to get no double caching. Here is a kludge that works in many cases:

int stackable_readpage(file_t *file, page_t *page)
{
...
page->mapping = lower_inode->i_mapping;
err = lower_inode->i_mapping->a_ops->readpage(lower_file, page); page->mapping = inode->i_mapping;
...
}

All structures with the 'lower_' prefix belong to the lower filesystem. It doesn't seem to be the exactly right way to go but it provides the same flexibility for Linux stackable filesystems that they enjoy in *BSD and Windows. A correct implementation requires some isolation of the page structure from the file/dentry/inode added at the VFS level. However, it would be sufficient if we can make the code above work in all the cases over all existing filesystems.

Sincerely,
Nikolai Joukov.
**************************************
* Ph.D. student (Advisor: Erez Zadok)
* File systems and Storage Laboratory
* Stony Brook University (SUNY)
**************************************

#1 lore raw
Quoted 8 lines · Nikolai Joukov

int stackable_readpage(file_t *file, page_t *page)
{
...
page->mapping = lower_inode->i_mapping;
err = lower_inode->i_mapping->a_ops->readpage(lower_file, page); page->mapping = inode->i_mapping;
...
}

This is a really bad idea for a number of reasons:

(1) page->mapping isn't the only metadata in the page.

(2) The lower_inode may interpret a hold as a block of zeros rather than bouncing the request back to the higher inode with ENODATA or something.

(3) The lower_inode readpage() may not be complete at the time you switch the mapping pointer back. Obviously the page will be locked until at such time as completion or an error occurs.

(4) The lower_inode readpage() may complete before it returns, in which case the VM may go and do something unspeakable to that page whilst it's still got the wrong mapping attached.

(5) The lower_inode may have attached its own metadata to page->private, and this may refer back to this page, and may subsequently trip an assertion because the page's mapping pointer has been corrupted.

David

#2 lore raw
Quoted 26 lines · David Howells
Quoted 8 lines · Nikolai Joukov

int stackable_readpage(file_t *file, page_t *page)
{
...
page->mapping = lower_inode->i_mapping;
err = lower_inode->i_mapping->a_ops->readpage(lower_file, page); page->mapping = inode->i_mapping;
...
}

This is a really bad idea for a number of reasons:

(1) page->mapping isn't the only metadata in the page.

(2) The lower_inode may interpret a hold as a block of zeros rather than bouncing the request back to the higher inode with ENODATA or something.

(3) The lower_inode readpage() may not be complete at the time you switch the mapping pointer back. Obviously the page will be locked until at such time as completion or an error occurs.

(4) The lower_inode readpage() may complete before it returns, in which case the VM may go and do something unspeakable to that page whilst it's still got the wrong mapping attached.

Quoted 5 lines

(5) The lower_inode may have attached its own metadata to page->private, and this may refer back to this page, and may subsequently trip an assertion because the page's mapping pointer has been corrupted.

David

David, we fully agree. That's why we don't like this page-flipping hack, even if "it works" for us under limited conditions. We'd much rather fix the problem properly. Maybe now that there appears to be others in need of something similar, we can discuss on this list possible ways to fix this in the vfs/mm proper?

One idea we considered is that no one above the file system (vfs or mm) should dereference the page->mapping directly, but go through a new f/s method. The default method will do what the code does now (extract the stuffed inode from inside the mapping). But any file system that wants to, could override this method with its own version which handles stacking and/or nfs as needed.

Will this work? Will it give us the functionality we need? Or would we need a single page to possibly point back to a _chain_ of inodes (which would start looking like Skinner's "pvnode" '93 idea)?

We haven't explored the full code to see if this idea will work out. So we'd like to start some discussion as to what would it take to support such functionality cleanly. We'd be happy to work with anyone on a fix for this issue, and also come up with patches to test and (eventually) submit for inclusion in the mainline kernel.

Thanks,
Erez.

#3 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Erez" == Erez Zadok <[email protected]> writes:

Erez> Will this work? Will it give us the functionality we need? Or would we Erez> need a single page to possibly point back to a _chain_ of inodes (which Erez> would start looking like Skinner's "pvnode" '93 idea)?

Also have a look at Heidemann's PhD thesis,
http://www.isi.edu:80/people/johnh/PAPERS/Heidemann95e.html

Signature · 5 lines
John Kohl
Senior Software Engineer - Rational Software - IBM Software Group
Lexington, Massachusetts, USA
[email protected]
<http://www.ibm.com/software/rational/>
#4 lore raw
Quoted 9 lines · John T. Kohl
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Erez" == Erez Zadok <[email protected]> writes:

Erez> Will this work? Will it give us the functionality we need? Or would we Erez> need a single page to possibly point back to a _chain_ of inodes (which Erez> would start looking like Skinner's "pvnode" '93 idea)?

Also have a look at Heidemann's PhD thesis,
http://www.isi.edu:80/people/johnh/PAPERS/Heidemann95e.html

This looks like it might also have relevance to "copy-on-write" copied files - multiple inodes, on the same filesystem, sharing pages in memory as well as on disk. Good for virtual machines sharing files, etc.

A key feature of COW files is the need to mmap() them, and for modifications to one to not affect the mapped view of the other.

Pages attached to multiple inodes would be a way of implementing that.

I don't have time to write more, so I'm just putting out the observation.

-- Jamie

#5 lore raw
In reply to Nikolai Joukov
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Nikolai" == Nikolai Joukov <[email protected]> writes:

Nikolai> Here is a kludge that works in many cases:

Nikolai> int stackable_readpage(file_t *file, page_t *page) Nikolai> {

Nikolai>  ...
Nikolai>  page->mapping = lower_inode->i_mapping;
Nikolai>  err = lower_inode->i_mapping->a_ops->readpage(lower_file, page);
Nikolai>  page->mapping = inode->i_mapping;
Nikolai>  ...

Nikolai> }

What are the locking requirements to do this safely? Are the proper locks held (or safely grabbable during readpage()) to avoid a race with another paging operation?
I see a troubling comment in do_generic_mapping_read():

		/* ... and start the actual read. The read will unlock the page. */
		error = mapping->a_ops->readpage(filp, page);

but I've not studied the page locking design to know for sure that this example is race-free.

Signature · 9 lines
John Kohl
Senior Software Engineer
Rational Software
IBM Software Group
Lexington, Massachusetts, USA
[email protected]

The opinions expressed in this message do not reflect the views of my
employer.