Volatile Ranges

This makes it
very difficult to provide generic list_head like behavior, as the parent structures would need to be duplicated and removed, and that has lots of memory ownership issues.

Quoted 4 lines · John Stultz

This makes it
very difficult to provide generic list_head like behavior, as the parent structures would need to be duplicated and removed, and that has lots of memory ownership issues.

You can in fact modify the rb-tree to have O(1) iteration by using the empty leaf pointers to keep pointers to next/prev nodes.

Its a bit of a bother since you'd need to wrap ->rb_left and ->rb_right in functions.. but now that we have coccinelle that shouldn't actually be too hard.

Quoted 5 lines · Peter Zijlstra

Quoted 4 lines · John Stultz

This makes it
very difficult to provide generic list_head like behavior, as the parent structures would need to be duplicated and removed, and that has lots of memory ownership issues.

Quoted 7 lines

You can in fact modify the rb-tree to have O(1) iteration by using the empty leaf pointers to keep pointers to next/prev nodes.

Its a bit of a bother since you'd need to wrap ->rb_left and ->rb_right in functions.. but now that we have coccinelle that shouldn't actually be too hard.

Sorry, I'm not sure I'm following you.

My point above was that a generic range-tree implementation that manages the splitting and coalescing of ranges internally is difficult, due to memory ownership issues. This makes it hard to have a generic list_head style structure that you can use in your own structures. Thus in a way similar to how the rb_tree leaves the insert and search implementation to the suers, there is a range_tree_node structure, and the splitting and coalescing logic is left to the range-tree user.

Does your suggestion address the ownership issue differently? Or is it just a general optimization improvement?

Hi John,

range_tree_in_range_adjacent() is not used in your code, and it doesn't seem to be very useful in general case. range_tree_in_range() can do the same thing (and you use it that way in the 2nd patch) and is more flexible (can be paired with range_tree_next_in_range()). So I think it can be dropped altogether.

Now, I'm wondering whether it actually makes sense to make a dedicated interface out of the remaining bits.

Almost everything is common rb_tree-handling code that can be found in any place where rb-trees are used (hard-coded for flexibility, performance or whatever other reasons). So my feeling is that it should not be different here.

Quoted 7 lines

[ ... ]

Almost everything is common rb_tree-handling code that can be found in any place where rb-trees are used (hard-coded for flexibility, performance or whatever other reasons). So my feeling is that it should not be different here.

Sorry, not sure I quite understand what you're suggesting. Are you saying it doesn't make sense to have a generic range tree implementation, since really its just a small shim over the rbtree code?  So instead range-tree users should just implment them themselves?

Quoted 7 lines

Quoted 7 lines

[ ... ]

Almost everything is common rb_tree-handling code that can be found in any place where rb-trees are used (hard-coded for flexibility, performance or whatever other reasons). So my feeling is that it should not be different here.

Quoted 4 lines

Sorry, not sure I quite understand what you're suggesting. Are you saying it doesn't make sense to have a generic range tree implementation, since really its just a small shim over the rbtree code? So instead range-tree users should just implment them themselves?

Exactly. It's not much different from other rbtree search-insert-delete implementations out there.

What are the generic use cases that we want to support with this interface?

Is the current notion of the 'overlapping range' as taken by range_tree_in_range() common enough? What if another use-case requires _only_ the ranges that are strictly inside the [ start, end ] range? In this case, we might be better off sticking to the same 'implement-your-own-search' approach taken by the generic rbtree interface.

mm/volatile.c line 401

Quoted 5 lines · John Stultz

mm/volatile.c new file +5 -0

+/*
+ * Purges volatile ranges when under memory pressure
+ */
+static int volatile_shrink(struct shrinker *ignored, struct shrink_control *sc)
+{

Hmm, I would have expected regular page reclaim to know about this instead of using a shrinker interface. Is this done specifically to avoid growing small holes in all ranges and instead dump entire ranges thereby keeping other ranges usable?

This patch provides new fadvise flags that can be used to mark file pages as volatile, which will allow it to be discarded if the kernel wants to reclaim memory.

mm/fadvise.c +2 -0

@@ -128,6 +129,19 @@ SYSCALL_DEFINE(fadvise64_64)(int fd, loff_t offset, loff_t len, int advice)
 			invalidate_mapping_pages(mapping, start_index,
 						end_index);
 		break;
+	case POSIX_FADV_VOLATILE:
+		/* First and last PARTIAL page! */

+ * The proposed mechanism - at a high level - is for user-space to be able
+ * to say "This memory is volatile" and then later "this memory is no longer
+ * volatile". If the content of the memory is still available the second
+ * request succeeds. If not, the memory is marked non-volatile and an
+ * error is returned to denote that the contents have been lost.

As it is, I'm still not sold on these being an fadvise() interface because all it really is a delayed hole punching interface whose functionailty is currently specific to tmpfs. The behaviour cannot be implemented sanely by anything else at this point.

IOWs, the specification you are describing means that FADV_VOLATILE could be correctly implemented as an immediate hole punch by every filesystem that supports hole punching.

Quoted 4 lines · Dave Chinner

As it is, I'm still not sold on these being an fadvise() interface because all it really is a delayed hole punching interface whose functionailty is currently specific to tmpfs. The behaviour cannot be implemented sanely by anything else at this point.

...

Quoted 3 lines

IOWs, the specification you are describing means that FADV_VOLATILE could be correctly implemented as an immediate hole punch by every filesystem that supports hole punching.

Ahhh, I think I see where you're going with this.

1. Data written to a file somehow (mmap(), write()) and is on disk
2. mmap() the data, and fault it in
3. Do a small write to it
4. set FADV_VOLATILE on it

Now we've got a dirty page which can theoretically be tossed out. But, we've got old data on the disk and no real way to tell that it was old if it got faulted in again. It's a much cleaner situation to just drop that data off the disk (hole punch) than to leave it around. Is that the concern?

But, we have other APIs that act this way, tossing out dirty data without reflecting that on-disk (MADV_DONTNEED at least).

Is it really
a stretch to define the FADV_VOLATILE to behave the same way? IOW, Should the behavior _really_ be hole punching? That'll cost us I/O to throw away data during memory reclaim since we have to go write the information about the hole.

Seems like a much more appropriate thing to just toss the data out since the app can handle it.

That's my concern - that persistent filesystems will have different behaviour to in-memory filesystems. They *must* be consistent in behaviour w.r.t. to stale data exposure, otherwise we are in a world of pain when applications start to use this. Quite frankly, I don't care about performance of VOLATILE ranges, but I care greatly about ensuring filesystems don't expose stale data to user applications....

Quoted 8 lines · Dave Chinner

That's my concern - that persistent filesystems will have different behaviour to in-memory filesystems. They *must* be consistent in behaviour w.r.t. to stale data exposure, otherwise we are in a world of pain when applications start to use this. Quite frankly, I don't care about performance of VOLATILE ranges, but I care greatly about ensuring filesystems don't expose stale data to user applications....

I think we're in agreement with the rest of this email, but I do want to stress that the performance of volatile ranges will become more ciritical, as in order for folks to effectively use them, they need to be able to mark and unmark ranges any time they're not using the data.

So if the overhead is too great for marking and unmarking pages, applications will be less likely to "help out". :)

Quoted 9 lines · John Stultz

Quoted 8 lines · Dave Chinner

That's my concern - that persistent filesystems will have different behaviour to in-memory filesystems. They *must* be consistent in behaviour w.r.t. to stale data exposure, otherwise we are in a world of pain when applications start to use this. Quite frankly, I don't care about performance of VOLATILE ranges, but I care greatly about ensuring filesystems don't expose stale data to user applications....

Quoted 5 lines

I think we're in agreement with the rest of this email, but I do want to stress that the performance of volatile ranges will become more ciritical, as in order for folks to effectively use them, they need to be able to mark and unmark ranges any time they're not using the data.

Performance is far less important than data security. Make it safe first, then optimise performance. As it is, the initial target of tmpfs - by it's very nature of returning zeros for regions not backed by pages - is safe w.r.t. stale data exposure, so it will not be slowed down by using an fallocate "best effort" hole-punching interface. The performance of other filesystems is something that the relevant filesystem developers can worry about....

Quoted 2 lines

So if the overhead is too great for marking and unmarking pages, applications will be less likely to "help out". :)

Devil's Advocate: If the benefit of managing caches in such a manner is this marginal, then why add the complexity to the kernel?

Quoted 4 lines · Dave Chinner

Quoted 3 lines · John Stultz

On 04/27/2012 06:36 PM, Dave Chinner wrote:
So if the overhead is too great for marking and unmarking pages, applications will be less likely to "help out". :)

Quoted 3 lines

Devil's Advocate: If the benefit of managing caches in such a manner is this marginal, then why add the complexity to the kernel?

I'm not saying the benefit is marginal. When we are resource constrained (no swap) and we need to free memory, having regions pre-marked by applications is a great benefit, as we can immediately take those marked volatile ranges (as opposed to memory notifiers, where we request applications to free memory themselves).

Being
able to free chunks of application memory, rather then killing the application provides a better experience/overall system performance.

However, if applications feel the marking and unmarking is too costly, they are less likely to mark the freeable ranges as volatile.

So only if no consideration for performance is given, in that case there'd be no benefit to adding the interface.

Quoted 12 lines · John Stultz

mm/fadvise.c +8 -0

@@ -128,6 +129,19 @@ SYSCALL_DEFINE(fadvise64_64)(int fd, loff_t offset, loff_t len, int advice)
  			invalidate_mapping_pages(mapping, start_index,
  						end_index);
  		break;
+	case POSIX_FADV_VOLATILE:
+		/* First and last PARTIAL page! */
+		start_index = offset>>  PAGE_CACHE_SHIFT;
+		end_index = endbyte>>  PAGE_CACHE_SHIFT;
+		ret = mapping_range_volatile(mapping, start_index, end_index);
+		break;
+	case POSIX_FADV_NONVOLATILE:
+		/* First and last PARTIAL page! */

mm/fadvise.c SYSCALL_DEFINE(fadvise64_64)(int fd, loff_t offset, loff_t len, int advice) · line 143

Quoted 4 lines

mm/fadvise.c +4 -0

+		start_index = offset>>  PAGE_CACHE_SHIFT;
+		end_index = endbyte>>  PAGE_CACHE_SHIFT;
+		ret = mapping_range_nonvolatile(mapping, start_index,
+								end_index);

As it is, I'm still not sold on these being an fadvise() interface because all it really is a delayed hole punching interface whose functionailty is currently specific to tmpfs. The behaviour cannot be implemented sanely by anything else at this point.

mm/volatile.c line 25

Quoted 5 lines

mm/volatile.c new file +5 -0

+ * The goal behind volatile ranges is to allow applications to interact
+ * with the kernel's cache management infrastructure.  In particular an
+ * application can say "this memory contains data that might be useful in
+ * the future, but can be reconstructed if necessary, so if the kernel
+ * needs, it can zap and reclaim this memory without having to swap it out.

This is what I mean - the definition of volatility is specific to a filesystem implementation - one that doesn't store persistent data.

Quoted 5 lines

+ * The proposed mechanism - at a high level - is for user-space to be able
+ * to say "This memory is volatile" and then later "this memory is no longer
+ * volatile". If the content of the memory is still available the second
+ * request succeeds. If not, the memory is marked non-volatile and an
+ * error is returned to denote that the contents have been lost.

For a filesystem, it's not "memory" that is volatile - it is the *data* that we have to consider that these hints apply to, and that implies both in memory and on stable storage. because you are targetting a filesystem without persisten storage, you are using "memory" interchangably with "data". That basically results in an interface that can only be used by non-persistent filesystems. However, for managing on-disk caches of fixed sizes, being able to mark regions as volatile or not is just as helpful to them as it is to memory based caches on tmpfs....

So why can't you implement this as fallocate() flags, and then make the tmpfs implementation of those fallocate flags do the right things? I think fallocate is the right interface, because this is simply an extension of the existing hole punching implementation. IOWs, the specification you are describing means that FADV_VOLATILE could be correctly implemented as an immediate hole punch by every filesystem that supports hole punching.

This probably won't perform wonderfully, which is where the range tracking and delayed punching (and the implied memory freeing) optimiation comes into play. Sure, for tmpfs this can be implemented as a shrinker, but for real filesystems that have to punch blocks a shrinker is really the wrong context to be running such transactions. However, using the fallocate() interface allows each filesytsem to optimise the delayed hole punching as they see best, something that cannot be done with this fadvise() interface.

It's all great that this can replace a single function in ashmem, but focussing purely on ashmem misses the point that this functionality has wider use, and that using a different interface allows independently tailored and optimised implementations of that functionality....

Quoted 13 lines · Dave Chinner

Quoted 12 lines · John Stultz

mm/fadvise.c +8 -0

@@ -128,6 +129,19 @@ SYSCALL_DEFINE(fadvise64_64)(int fd, loff_t offset, loff_t len, int advice)
 			invalidate_mapping_pages(mapping, start_index,
 						end_index);
 		break;
+	case POSIX_FADV_VOLATILE:
+		/* First and last PARTIAL page! */
+		start_index = offset>>  PAGE_CACHE_SHIFT;
+		end_index = endbyte>>  PAGE_CACHE_SHIFT;
+		ret = mapping_range_volatile(mapping, start_index, end_index);
+		break;
+	case POSIX_FADV_NONVOLATILE:
+		/* First and last PARTIAL page! */

Quoted 8 lines

mm/fadvise.c SYSCALL_DEFINE(fadvise64_64)(int fd, loff_t offset, loff_t len, int advice) · line 143

Quoted 4 lines

mm/fadvise.c +4 -0

+		start_index = offset>>  PAGE_CACHE_SHIFT;
+		end_index = endbyte>>  PAGE_CACHE_SHIFT;
+		ret = mapping_range_nonvolatile(mapping, start_index,
+								end_index);

As it is, I'm still not sold on these being an fadvise() interface because all it really is a delayed hole punching interface whose functionailty is currently specific to tmpfs. The behaviour cannot be implemented sanely by anything else at this point.

Yea. So I spent some time looking at the various hole punching mechanisms and they aren't all together consistent across filesystems. For instance, on some filesystems (ext4 and mostly disk backed fs) you have to use fallocate(fd,

|FALLOC_FL_PUNCH_HOLE,...)|, while on tmpfs, its
madvise(...,MADV_REMOVE).   So in a way, currently, the

FADVISE_VOLATILE is closer to a delayed MADVISE_REMOVE.

Quoted 7 lines

mm/volatile.c line 25

Quoted 5 lines

mm/volatile.c new file +5 -0

+ * The goal behind volatile ranges is to allow applications to interact
+ * with the kernel's cache management infrastructure.  In particular an
+ * application can say "this memory contains data that might be useful in
+ * the future, but can be reconstructed if necessary, so if the kernel
+ * needs, it can zap and reclaim this memory without having to swap it out.

This is what I mean - the definition of volatility is specific to a filesystem implementation - one that doesn't store persistent data.

Well, I'd like to think that it could be extended to do delayed hole punching on disk backed persistent files, but again, currently there's no unified way to punch holes across the disk and memory backed filesystems.

If other filesystems implemented vmtruncate_range for hole punching, we could (modulo the circular mutex lock issue of calling vmtruncate_range from a shrinker) support this on other filesystems.

Are there inherent reasons why vmtruncate_range isn't implemented (or can't be sanely implemented) by non-tmpfs filesystems?

Quoted 15 lines

Quoted 5 lines

+ * The proposed mechanism - at a high level - is for user-space to be able
+ * to say "This memory is volatile" and then later "this memory is no longer
+ * volatile". If the content of the memory is still available the second
+ * request succeeds. If not, the memory is marked non-volatile and an
+ * error is returned to denote that the contents have been lost.

For a filesystem, it's not "memory" that is volatile - it is the *data* that we have to consider that these hints apply to, and that implies both in memory and on stable storage. because you are targetting a filesystem without persisten storage, you are using "memory" interchangably with "data". That basically results in an interface that can only be used by non-persistent filesystems. However, for managing on-disk caches of fixed sizes, being able to mark regions as volatile or not is just as helpful to them as it is to memory based caches on tmpfs....

Quoted 7 lines

So why can't you implement this as fallocate() flags, and then make the tmpfs implementation of those fallocate flags do the right things? I think fallocate is the right interface, because this is simply an extension of the existing hole punching implementation. IOWs, the specification you are describing means that FADV_VOLATILE could be correctly implemented as an immediate hole punch by every filesystem that supports hole punching.

So yea, I'm fine with changing interface as long as fallocate is where the consensus is. I'm not sure I maybe understand the subtlety of the interface differences, and it doesn't necessarily seem more intuitive to me (as seems more advisory then allocation based). But I can give it a shot.

Another way we could go is using madvise, somewhat mimicing the MADVISE_REMOVE call, which again, is not implemented everywhere.

Although as DaveH said, doing the hole punch on disk is extra overhead. But I agree it makes more sense from a least-surprise approach (no data is less surprising then old data after a purge).

As for your immediate hole punch thought, that could work, although FADV_VOLATILE would be just as correctly implemented by not purging any of data on disk backed files. Either way, a difference might be slightly confusing for users (since either way changes the global LRU purge behavior).

Quoted 8 lines

This probably won't perform wonderfully, which is where the range tracking and delayed punching (and the implied memory freeing) optimiation comes into play. Sure, for tmpfs this can be implemented as a shrinker, but for real filesystems that have to punch blocks a shrinker is really the wrong context to be running such transactions. However, using the fallocate() interface allows each filesytsem to optimise the delayed hole punching as they see best, something that cannot be done with this fadvise() interface.

So if a shrinker isn't the right context, what would be a good context for delayed hole punching?

Quoted 22 lines · John Stultz

Quoted 13 lines · Dave Chinner

Quoted 12 lines · John Stultz

mm/fadvise.c +8 -0

@@ -128,6 +129,19 @@ SYSCALL_DEFINE(fadvise64_64)(int fd, loff_t offset, loff_t len, int advice)
  			invalidate_mapping_pages(mapping, start_index,
  						end_index);
  		break;
+	case POSIX_FADV_VOLATILE:
+		/* First and last PARTIAL page! */
+		start_index = offset>>   PAGE_CACHE_SHIFT;
+		end_index = endbyte>>   PAGE_CACHE_SHIFT;
+		ret = mapping_range_volatile(mapping, start_index, end_index);
+		break;
+	case POSIX_FADV_NONVOLATILE:
+		/* First and last PARTIAL page! */

Quoted 8 lines

mm/fadvise.c SYSCALL_DEFINE(fadvise64_64)(int fd, loff_t offset, loff_t len, int advice) · line 143

Quoted 4 lines

mm/fadvise.c +4 -0

+		start_index = offset>>   PAGE_CACHE_SHIFT;
+		end_index = endbyte>>   PAGE_CACHE_SHIFT;
+		ret = mapping_range_nonvolatile(mapping, start_index,
+								end_index);

As it is, I'm still not sold on these being an fadvise() interface because all it really is a delayed hole punching interface whose functionailty is currently specific to tmpfs. The behaviour cannot be implemented sanely by anything else at this point.

Quoted 7 lines

Yea. So I spent some time looking at the various hole punching mechanisms and they aren't all together consistent across filesystems. For instance, on some filesystems (ext4 and mostly disk backed fs) you have to use fallocate(fd,

|FALLOC_FL_PUNCH_HOLE,...)|, while on tmpfs, its
madvise(...,MADV_REMOVE).   So in a way, currently, the

FADVISE_VOLATILE is closer to a delayed MADVISE_REMOVE.

The MADVISE_REMOVE functionality for hole punching works *only* for tmpfs - no other filesystem implements the .truncate_range() method. In fact, several filesystems *can't* implement .truncate_range() because there is no callout from the page cache truncation code to allow filesystems to punch out the underlying blocks. The vmtruncate() code is deprecated for this reason (and various others like a lack of error handling), and .truncate_range() is just as nasty. .truncate_range() needs to die, IMO.

So, rather than building more infrastructure on a nasty, filesystem specific mmap() hack, implement .fallocate() on tmpfs and use the same interface that every other filesystem uses for punching holes.

Quoted 10 lines

Quoted 8 lines

This probably won't perform wonderfully, which is where the range tracking and delayed punching (and the implied memory freeing) optimiation comes into play. Sure, for tmpfs this can be implemented as a shrinker, but for real filesystems that have to punch blocks a shrinker is really the wrong context to be running such transactions. However, using the fallocate() interface allows each filesytsem to optimise the delayed hole punching as they see best, something that cannot be done with this fadvise() interface.

So if a shrinker isn't the right context, what would be a good context for delayed hole punching?

Like we in XFs for inode reclaim. We have a background workqueue that frees aged inodes periodically in the fastest manner possible (i.e. all async, no blocking on locks, etc), and the shrinker, when run kicks that background thread first, and then enters into synchronous reclaim. By the time a single sync reclaim cycle is run and throttled reclaim sufficiently, the background thread has done a great deal more work.

A similar mechanism can be used for this functionality within XFS. Indeed, we could efficiently track which inodes have volatile ranges on them via a bit in the radix trees than index the inode cache, just like we do for reclaimable inodes. If we then used a bit in the page cache radix tree index to indicate volatile pages, we could then easily find the ranges we need to punch out without requiring some new tree and more per-inode memory.

That's a very filesystem specific implementation - it's vastly different to you tmpfs implementation - but this is exactly what I mean about using fallocate to allow filesystems to optimise the implementation in the most suitable manner for them....

Quoted 28 lines · Dave Chinner

Quoted 11 lines · John Stultz

Quoted 8 lines · Dave Chinner

This probably won't perform wonderfully, which is where the range tracking and delayed punching (and the implied memory freeing) optimiation comes into play. Sure, for tmpfs this can be implemented as a shrinker, but for real filesystems that have to punch blocks a shrinker is really the wrong context to be running such transactions. However, using the fallocate() interface allows each filesytsem to optimise the delayed hole punching as they see best, something that cannot be done with this fadvise() interface.

So if a shrinker isn't the right context, what would be a good context for delayed hole punching?

Like we in XFs for inode reclaim. We have a background workqueue that frees aged inodes periodically in the fastest manner possible (i.e. all async, no blocking on locks, etc), and the shrinker, when run kicks that background thread first, and then enters into synchronous reclaim. By the time a single sync reclaim cycle is run and throttled reclaim sufficiently, the background thread has done a great deal more work.

A similar mechanism can be used for this functionality within XFS. Indeed, we could efficiently track which inodes have volatile ranges on them via a bit in the radix trees than index the inode cache, just like we do for reclaimable inodes. If we then used a bit in the page cache radix tree index to indicate volatile pages, we could then easily find the ranges we need to punch out without requiring some new tree and more per-inode memory.

Quoted 5 lines

That's a very filesystem specific implementation - it's vastly different to you tmpfs implementation - but this is exactly what I mean about using fallocate to allow filesystems to optimise the implementation in the most suitable manner for them....

So, just to make sure I'm folloiwng you, you're suggesting that there would be a filesystem specific implementation at the top level. Something like a mark_volatile(struct inode *, bool, loff_t, loff_t) inode operation? And the filesystem would then be responsible for managing the ranges and appropriately purging them?

Quoted 34 lines · John Stultz

Quoted 28 lines · Dave Chinner

Quoted 11 lines · John Stultz

Quoted 8 lines · Dave Chinner

This probably won't perform wonderfully, which is where the range tracking and delayed punching (and the implied memory freeing) optimiation comes into play. Sure, for tmpfs this can be implemented as a shrinker, but for real filesystems that have to punch blocks a shrinker is really the wrong context to be running such transactions. However, using the fallocate() interface allows each filesytsem to optimise the delayed hole punching as they see best, something that cannot be done with this fadvise() interface.

So if a shrinker isn't the right context, what would be a good context for delayed hole punching?

Like we in XFs for inode reclaim. We have a background workqueue that frees aged inodes periodically in the fastest manner possible (i.e. all async, no blocking on locks, etc), and the shrinker, when run kicks that background thread first, and then enters into synchronous reclaim. By the time a single sync reclaim cycle is run and throttled reclaim sufficiently, the background thread has done a great deal more work.

A similar mechanism can be used for this functionality within XFS. Indeed, we could efficiently track which inodes have volatile ranges on them via a bit in the radix trees than index the inode cache, just like we do for reclaimable inodes. If we then used a bit in the page cache radix tree index to indicate volatile pages, we could then easily find the ranges we need to punch out without requiring some new tree and more per-inode memory.

Quoted 5 lines

That's a very filesystem specific implementation - it's vastly different to you tmpfs implementation - but this is exactly what I mean about using fallocate to allow filesystems to optimise the implementation in the most suitable manner for them....

Quoted 5 lines

So, just to make sure I'm folloiwng you, you're suggesting that there would be a filesystem specific implementation at the top level. Something like a mark_volatile(struct inode *, bool, loff_t, loff_t) inode operation? And the filesystem would then be responsible for managing the ranges and appropriately purging them?

Not quite. I'm suggesting that you use the .fallocate() file operation to call into the filesystem specific code, and from there the filesystem code either calls a generic helper function to mark ranges as volatile and provides a callback for implementing the shrinker functionailty, or it implements it all itself.

i.e. userspace would do:

	err = fallocate(fd, FALLOC_FL_MARK_VOLATILE, off, len);
	err = fallocate(fd, FALLOC_FL_CLEAR_VOLATILE, off, len);

and that will get passed to the filesystem implementation of .fallocate (from do_fallocate()). The filesystem callout for this:

0 btrfs/file.c 1898 .fallocate = btrfs_fallocate,
1 ext4/file.c 247 .fallocate = ext4_fallocate,

2 gfs2/file.c    1015 .fallocate = gfs2_fallocate,
3 gfs2/file.c    1045 .fallocate = gfs2_fallocate,
4 ocfs2/file.c   2727 .fallocate = ocfs2_fallocate,
5 ocfs2/file.c   2774 .fallocate = ocfs2_fallocate,

6 xfs/xfs_file.c 1026 .fallocate = xfs_file_fallocate,

can then call a generic helper like, say:

	filemap_mark_volatile_range(inode, off, len);
	filemap_clear_volatile_range(inode, off, len);

to be able to use the range tree tracking you have written for this purpose. The filesystem is also free to track ranges however it pleases.

The filesystem will need to be able to store a tree/list root for tracking all it's inodes that have volatile ranges, and register a shrinker to walk that list and do the work necessary when memory becomes low, but that is simple to do for a basic implementation.

Quoted 54 lines · Dave Chinner

Quoted 34 lines · John Stultz

Quoted 28 lines · Dave Chinner

Quoted 11 lines · John Stultz

Quoted 8 lines · Dave Chinner

This probably won't perform wonderfully, which is where the range tracking and delayed punching (and the implied memory freeing) optimiation comes into play. Sure, for tmpfs this can be implemented as a shrinker, but for real filesystems that have to punch blocks a shrinker is really the wrong context to be running such transactions. However, using the fallocate() interface allows each filesytsem to optimise the delayed hole punching as they see best, something that cannot be done with this fadvise() interface.

So if a shrinker isn't the right context, what would be a good context for delayed hole punching?

Like we in XFs for inode reclaim. We have a background workqueue that frees aged inodes periodically in the fastest manner possible (i.e. all async, no blocking on locks, etc), and the shrinker, when run kicks that background thread first, and then enters into synchronous reclaim. By the time a single sync reclaim cycle is run and throttled reclaim sufficiently, the background thread has done a great deal more work.

A similar mechanism can be used for this functionality within XFS. Indeed, we could efficiently track which inodes have volatile ranges on them via a bit in the radix trees than index the inode cache, just like we do for reclaimable inodes. If we then used a bit in the page cache radix tree index to indicate volatile pages, we could then easily find the ranges we need to punch out without requiring some new tree and more per-inode memory.

Quoted 5 lines

That's a very filesystem specific implementation - it's vastly different to you tmpfs implementation - but this is exactly what I mean about using fallocate to allow filesystems to optimise the implementation in the most suitable manner for them....

Quoted 5 lines

So, just to make sure I'm folloiwng you, you're suggesting that there would be a filesystem specific implementation at the top level. Something like a mark_volatile(struct inode *, bool, loff_t, loff_t) inode operation? And the filesystem would then be responsible for managing the ranges and appropriately purging them?

Not quite. I'm suggesting that you use the .fallocate() file operation to call into the filesystem specific code, and from there the filesystem code either calls a generic helper function to mark ranges as volatile and provides a callback for implementing the shrinker functionailty, or it implements it all itself.

i.e. userspace would do:

	err = fallocate(fd, FALLOC_FL_MARK_VOLATILE, off, len);
	err = fallocate(fd, FALLOC_FL_CLEAR_VOLATILE, off, len);

and that will get passed to the filesystem implementation of .fallocate (from do_fallocate()). The filesystem callout for this:

Quoted 7 lines

0 btrfs/file.c 1898 .fallocate = btrfs_fallocate,
1 ext4/file.c 247 .fallocate = ext4_fallocate,

2 gfs2/file.c    1015 .fallocate = gfs2_fallocate,
3 gfs2/file.c    1045 .fallocate = gfs2_fallocate,
4 ocfs2/file.c   2727 .fallocate = ocfs2_fallocate,
5 ocfs2/file.c   2774 .fallocate = ocfs2_fallocate,

6 xfs/xfs_file.c 1026 .fallocate = xfs_file_fallocate,

Ok.

Although noting that tmpfs doesn't implement fallocate, this isn't just a ruse to get me to implement fallocate for tmpfs, right? ;)

Out of curiosity: While for my uses, a tmpfs exclusive implementation isn't a problem for me, I *really* appreciate your feedback and help focusing this into a more generic fs solution. But to get more context for your insights, I'm curious if you have any use cases in your mind that is directing this work to be more generic?

Again, you've been helpfully skeptical of the work, and also at the same time pushing it in a certain direction, and I want to make sure I better understand where you're coming from. You've clarified the file security concern well enough, but is that all? I just want to make sure that going the generic fallocate route is really worth it, as opposed to moving to madvise() and just failing file-data backed memory.

Quoted 9 lines

can then call a generic helper like, say:

	filemap_mark_volatile_range(inode, off, len);
	filemap_clear_volatile_range(inode, off, len);

to be able to use the range tree tracking you have written for this purpose. The filesystem is also free to track ranges however it pleases.

Quoted 5 lines

The filesystem will need to be able to store a tree/list root for tracking all it's inodes that have volatile ranges, and register a shrinker to walk that list and do the work necessary when memory becomes low, but that is simple to do for a basic implementation.

Hrmm.. Currently I'm using a per-mapping range-tree along with a global LRU list that ties all the ranges together.

First pass attempt at getting ashmem to utilize the volatile range code.

Quoted 2 lines · John Stultz

First pass attempt at getting ashmem to utilize the volatile range code.

One would think the purpose of the previous patches was to entirely do away with the ashmem interface.. this patch is a temporary band-aid to assist in testing without having to modify userspace?