VFS: Directory level cache cleaning

This patch extend the 'drop_caches' interface to support directory level cache cleaning and has a complete backward compatibility. '{1,2,3}' keeps the same semantics as before. Besides, "{1,2,3}:DIRECTORY_PATH_NAME" is allowed to recursively clean the caches under DIRECTORY_PATH_NAME. For example, 'echo 1:/home/foo/jpg > /proc/sys/vm/drop_caches' will clean the page caches of the files inside 'home/foo/jpg'.

Quoted 8 lines · Li Wang

This patch extend the 'drop_caches' interface to support directory level cache cleaning and has a complete backward compatibility. '{1,2,3}' keeps the same semantics as before. Besides, "{1,2,3}:DIRECTORY_PATH_NAME" is allowed to recursively clean the caches under DIRECTORY_PATH_NAME. For example, 'echo 1:/home/foo/jpg > /proc/sys/vm/drop_caches' will clean the page caches of the files inside 'home/foo/jpg'.

This interface is ugly...

And we already have a file-level drop cache, that is, fadvise(DONTNEED). Can you extend it if it can't handle a directory fd?

As far as we know, fadvise(DONTNEED) does not support metadata cache cleaning. We think that is desirable under massive small files situations. Another thing is that do people accept the behavior of feeding a directory fd to fadvise will recusively clean all page caches of files inside that directory?

Quoted 10 lines · Li Wang

Quoted 8 lines

This patch extend the 'drop_caches' interface to support directory level cache cleaning and has a complete backward compatibility. '{1,2,3}' keeps the same semantics as before. Besides, "{1,2,3}:DIRECTORY_PATH_NAME" is allowed to recursively clean the caches under DIRECTORY_PATH_NAME. For example, 'echo 1:/home/foo/jpg > /proc/sys/vm/drop_caches' will clean the page caches of the files inside 'home/foo/jpg'.

Quoted 6 lines

This interface is ugly...

And we already have a file-level drop cache, that is, fadvise(DONTNEED). Can you extend it if it can't handle a directory fd?

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 5 lines · Li Wang

As far as we know, fadvise(DONTNEED) does not support metadata cache cleaning. We think that is desirable under massive small files situations. Another thing is that do people accept the behavior of feeding a directory fd to fadvise will recusively clean all page caches of files inside that directory?

I think there's a really good permissions-related question here. If that's an acceptable interface, should one have to be CAP_SYS_ADMIN to issue the request? What if some of the files below this directory are not owned by the user issuing the request?

Quoted 21 lines · Cong Wang

Quoted 10 lines · Li Wang

Quoted 8 lines

This patch extend the 'drop_caches' interface to support directory level cache cleaning and has a complete backward compatibility. '{1,2,3}' keeps the same semantics as before. Besides, "{1,2,3}:DIRECTORY_PATH_NAME" is allowed to recursively clean the caches under DIRECTORY_PATH_NAME. For example, 'echo 1:/home/foo/jpg > /proc/sys/vm/drop_caches' will clean the page caches of the files inside 'home/foo/jpg'.

Quoted 6 lines

This interface is ugly...

And we already have a file-level drop cache, that is, fadvise(DONTNEED). Can you extend it if it can't handle a directory fd?

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

If we do wanna equip fadvise() with directory level page cache cleaning,
this could be solved by invoking (inode_permission() || capable(CAP_SYS_ADMIN)) before manipulating the page cache of that inode. We think the current extension to 'drop_caches' has a complete back
compatibility, the old semantics keep unchanged, and with add-on
features to do finer granularity cache cleaning should be also
desirable.

Quoted 7 lines · Li Wang

If we do wanna equip fadvise() with directory level page cache cleaning,
this could be solved by invoking (inode_permission() || capable(CAP_SYS_ADMIN)) before manipulating the page cache of that inode. We think the current extension to 'drop_caches' has a complete back
compatibility, the old semantics keep unchanged, and with add-on
features to do finer granularity cache cleaning should be also
desirable.

I don't think you can extend the drop_caches interface this way. It should be used for debuging only.

commit 9d0243bca345d5ce25d3f4b74b7facb3a6df1232

Author: Andrew Morton <[email protected]>
Date:   Sun Jan 8 01:00:39 2006 -0800

[PATCH] drop-pagecache

Add /proc/sys/vm/drop_caches. When written to, this will cause the kernel to discard as much pagecache and/or reclaimable slab objects as it can. THis operation requires root permissions.

...

This is a debugging feature: useful for getting consistent results between filesystem benchmarks. We could possibly put it under a config option, but it's less than 300 bytes.

Also see http://lkml.org/lkml/2013/7/26/230

This extension is just add-on extension. The original debugging capability is still there, and more flexible debugging is now allowed.

Quoted 7 lines · Li Wang

If we do wanna equip fadvise() with directory level page cache cleaning,
this could be solved by invoking (inode_permission() || capable(CAP_SYS_ADMIN)) before manipulating the page cache of that inode. We think the current extension to 'drop_caches' has a complete back
compatibility, the old semantics keep unchanged, and with add-on
features to do finer granularity cache cleaning should be also
desirable.

I don't think you can extend the drop_caches interface this way. It should be used for debuging only.

commit 9d0243bca345d5ce25d3f4b74b7facb3a6df1232

Author: Andrew Morton <[email protected]>
Date:   Sun Jan 8 01:00:39 2006 -0800

[PATCH] drop-pagecache

Add /proc/sys/vm/drop_caches. When written to, this will cause the kernel to discard as much pagecache and/or reclaimable slab objects as it can. THis operation requires root permissions.

...

This is a debugging feature: useful for getting consistent results between filesystem benchmarks. We could possibly put it under a config option, but it's less than 300 bytes.

Also see http://lkml.org/lkml/2013/7/26/230

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Currently, Linux only support file system wide VFS cache (dentry cache and page cache) cleaning through '/proc/sys/vm/drop_caches'. Sometimes this is less flexible. The applications may know exactly whether the metadata and data will be referenced or not in future, a desirable mechanism is to enable applications to reclaim the memory of unused cache entries at a finer granularity - directory level. This enables applications to keep hot metadata and data (to be referenced in the future) in the cache, and kick unused out to avoid cache thrashing. Another advantage is it is more flexible for debugging.

This patch extend the 'drop_caches' interface to support directory level cache cleaning and has a complete backward compatibility. '{1,2,3}' keeps the same semantics as before. Besides, "{1,2,3}:DIRECTORY_PATH_NAME" is allowed to recursively clean the caches under DIRECTORY_PATH_NAME. For example, 'echo 1:/home/foo/jpg > /proc/sys/vm/drop_caches' will clean the page caches of the files inside 'home/foo/jpg'.

It is easy to demonstrate the advantage of directory level cache cleaning. We use a virtual machine configured with an Intel(R) Xeon(R) 8-core CPU E5506 @ 2.13GHz, and with 1GB memory. Three directories named '1', '2' and '3' are created, with each containing 180000 – 280000 files. The test program opens all files in a directory and then tries the next directory. The order for accessing the directories is '1', '2', '3', '1'.

The time on accessing '1' on the second time is measured with/without cache cleaning, under different file counts. With cache cleaning, we clean all cache entries of files in '2' before accessing the files in '3'. The results are as follows (in seconds),

Note: by default, VFS will move those unreferenced inodes into a global LRU list rather than freeing them, for this experiment, we modified iput() to force to free inode as well, this behavior and related codes are left for further discussion, thus not reflected in this patch)

Number of files:   180000 200000 220000 240000 260000
Without cleaning:  2.165  6.977  10.032 11.571 13.443
With cleaning:     1.949  1.906  2.336  2.918  3.651

When the number of files is 180000 in each directory, the metadata cache is large enough to buffer all entries of three directories, so re-accessing '1' will hit in the cache, regardless of whether '2' cleaned up or not. As the number of files increases, the cache can now only buffer two+ directories. Accessing '3' will result in some entries of '1' to be evicted (due to LRU). When re-accessing '1', some entries need be reloaded from disk, which is time-consuming.

Quoted 30 lines · Li Wang

Currently, Linux only support file system wide VFS cache (dentry cache and page cache) cleaning through '/proc/sys/vm/drop_caches'. Sometimes this is less flexible. The applications may know exactly whether the metadata and data will be referenced or not in future, a desirable mechanism is to enable applications to reclaim the memory of unused cache entries at a finer granularity - directory level. This enables applications to keep hot metadata and data (to be referenced in the future) in the cache, and kick unused out to avoid cache thrashing. Another advantage is it is more flexible for debugging.

This patch extend the 'drop_caches' interface to support directory level cache cleaning and has a complete backward compatibility. '{1,2,3}' keeps the same semantics as before. Besides, "{1,2,3}:DIRECTORY_PATH_NAME" is allowed to recursively clean the caches under DIRECTORY_PATH_NAME. For example, 'echo 1:/home/foo/jpg > /proc/sys/vm/drop_caches' will clean the page caches of the files inside 'home/foo/jpg'.

It is easy to demonstrate the advantage of directory level cache cleaning. We use a virtual machine configured with an Intel(R) Xeon(R) 8-core CPU E5506 @ 2.13GHz, and with 1GB memory. Three directories named '1', '2' and '3' are created, with each containing 180000 – 280000 files. The test program opens all files in a directory and then tries the next directory. The order for accessing the directories is '1', '2', '3', '1'.

Quoted 5 lines

The time on accessing '1' on the second time is measured with/without cache cleaning, under different file counts. With cache cleaning, we clean all cache entries of files in '2' before accessing the files in '3'. The results are as follows (in seconds),

This sounds like a highly contrived test case. There is no reason why dentry cache access time would change going from 180k to 280k files in 3 directories unless you're right at the memory pressure balance point in terms of cache sizing.

Quoted 10 lines

Note: by default, VFS will move those unreferenced inodes into a global LRU list rather than freeing them, for this experiment, we modified iput() to force to free inode as well, this behavior and related codes are left for further discussion, thus not reflected in this patch)

Number of files:   180000 200000 220000 240000 260000
Without cleaning:  2.165  6.977  10.032 11.571 13.443
With cleaning:     1.949  1.906  2.336  2.918  3.651

Quoted 8 lines

When the number of files is 180000 in each directory, the metadata cache is large enough to buffer all entries of three directories, so re-accessing '1' will hit in the cache, regardless of whether '2' cleaned up or not. As the number of files increases, the cache can now only buffer two+ directories. Accessing '3' will result in some entries of '1' to be evicted (due to LRU). When re-accessing '1', some entries need be reloaded from disk, which is time-consuming.

Ok, so exactly as I thought - your example working set is slightly larger than what the cache holds. Hence what you are describing is a cache reclaim threshold effect: something you can avoid with /proc/sys/vm/vfs_cache_pressure.

Cheers,

Dave.