Skip to contentSkip to conversation

mm, fadvise: improve the expensive remote LRU cache draining after FADV_DONTNEED

PATCH Applied Yafang Shao 2 participants 4 messages Sep 23, 2020 – Sep 28, 2020

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 mm, fadvise: improve the expensive remote LRU cache draining after FADV_DONTNEED

Our users reported that there're some random latency spikes when their RT process is running. Finally we found that latency spike is caused by FADV_DONTNEED. Which may call lru_add_drain_all() to drain LRU cache on remote CPUs, and then waits the per-cpu work to complete. The wait time is uncertain, which may be tens millisecond.
That behavior is unreasonable, because this process is bound to a specific CPU and the file is only accessed by itself, IOW, there should be no pagecache pages on a per-cpu pagevec of a remote CPU. That unreasonable behavior is partially caused by the wrong comparation of the number of invalidated pages and the number of the target. For example,
if (count < (end_index - start_index + 1))
The count above is how many pages were invalidated in the local CPU, and (end_index - start_index + 1) is how many pages should be invalidated. The usage of (end_index - start_index + 1) is incorrect, because they are virtual addresses, which may not mapped to pages. Besides that, there may be holes between start and end. So we'd better check whether there are still pages on per-cpu pagevec after drain the local cpu, and then decide whether or not to call lru_add_drain_all().

After I applied it with a hotfix to our production environment, most of the lru_add_drain_all() can be avoided.

---
 include/linux/fs.h |  4 ++++
 mm/fadvise.c       |  9 +++----
 mm/truncate.c      | 58 ++++++++++++++++++++++++++++++++--------------
 3 files changed, 49 insertions(+), 22 deletions(-)
include/linux/fs.h +4 -0
index 7519ae003a08..6ba747b097c5 100644
@@ -2591,6 +2591,10 @@ extern bool is_bad_inode(struct inode *);
 unsigned long invalidate_mapping_pages(struct address_space *mapping,
 					pgoff_t start, pgoff_t end);
 
+void invalidate_mapping_pagevec(struct address_space *mapping,
+				pgoff_t start, pgoff_t end,
+				unsigned long *nr_pagevec);
+
 static inline void invalidate_remote_inode(struct inode *inode)
 {
 	if (S_ISREG(inode->i_mode) || S_ISDIR(inode->i_mode) ||
mm/fadvise.c +5 -4
index 0e66f2aaeea3..d6baa4f451c5 100644
@@ -141,7 +141,7 @@ int generic_fadvise(struct file *file, loff_t offset, loff_t len, int advice)
 		}
 
 		if (end_index >= start_index) {
-			unsigned long count;
+			unsigned long nr_pagevec = 0;
 
 			/*
 			 * It's common to FADV_DONTNEED right after
@@ -154,8 +154,9 @@ int generic_fadvise(struct file *file, loff_t offset, loff_t len, int advice)
 			 */
 			lru_add_drain();
 
-			count = invalidate_mapping_pages(mapping,
-						start_index, end_index);
+			invalidate_mapping_pagevec(mapping,
+						start_index, end_index,
+						&nr_pagevec);
 
 			/*
 			 * If fewer pages were invalidated than expected then
@@ -163,7 +164,7 @@ int generic_fadvise(struct file *file, loff_t offset, loff_t len, int advice)
 			 * a per-cpu pagevec for a remote CPU. Drain all
 			 * pagevecs and try again.
 			 */
-			if (count < (end_index - start_index + 1)) {
+			if (nr_pagevec) {
 				lru_add_drain_all();
 				invalidate_mapping_pages(mapping, start_index,
 						end_index);
mm/truncate.c +40 -18

79 lines not on this page · raw message

Signature · 1 line
2.17.1
#1 lore raw
Quoted 19 lines · Yafang Shao

Our users reported that there're some random latency spikes when their RT process is running. Finally we found that latency spike is caused by FADV_DONTNEED. Which may call lru_add_drain_all() to drain LRU cache on remote CPUs, and then waits the per-cpu work to complete. The wait time is uncertain, which may be tens millisecond.
That behavior is unreasonable, because this process is bound to a specific CPU and the file is only accessed by itself, IOW, there should be no pagecache pages on a per-cpu pagevec of a remote CPU. That unreasonable behavior is partially caused by the wrong comparation of the number of invalidated pages and the number of the target. For example,
if (count < (end_index - start_index + 1))
The count above is how many pages were invalidated in the local CPU, and (end_index - start_index + 1) is how many pages should be invalidated. The usage of (end_index - start_index + 1) is incorrect, because they are virtual addresses, which may not mapped to pages. Besides that, there may be holes between start and end. So we'd better check whether there are still pages on per-cpu pagevec after drain the local cpu, and then decide whether or not to call lru_add_drain_all().

Quoted 7 lines

After I applied it with a hotfix to our production environment, most of the lru_add_drain_all() can be avoided.

I think that's ok. Does it succeed with the original test case from the commit that introduced the behaviour and one modified to truncate part of the mapping?

Signature · 2 lines
Mel Gorman
SUSE Labs
#2 lore raw
Quoted 24 lines · Mel Gorman
Quoted 22 lines · Yafang Shao

Our users reported that there're some random latency spikes when their RT process is running. Finally we found that latency spike is caused by FADV_DONTNEED. Which may call lru_add_drain_all() to drain LRU cache on remote CPUs, and then waits the per-cpu work to complete. The wait time is uncertain, which may be tens millisecond.
That behavior is unreasonable, because this process is bound to a specific CPU and the file is only accessed by itself, IOW, there should be no pagecache pages on a per-cpu pagevec of a remote CPU. That unreasonable behavior is partially caused by the wrong comparation of the number of invalidated pages and the number of the target. For example,
if (count < (end_index - start_index + 1))
The count above is how many pages were invalidated in the local CPU, and (end_index - start_index + 1) is how many pages should be invalidated. The usage of (end_index - start_index + 1) is incorrect, because they are virtual addresses, which may not mapped to pages. Besides that, there may be holes between start and end. So we'd better check whether there are still pages on per-cpu pagevec after drain the local cpu, and then decide whether or not to call lru_add_drain_all().

After I applied it with a hotfix to our production environment, most of the lru_add_drain_all() can be avoided.

Quoted 9 lines
Quoted 4 lines

I think that's ok. Does it succeed with the original test case from the commit that introduced the behaviour and one modified to truncate part of the mapping?

Yes, I verified the test case in commit 67d46b296a1b and then modified it with truncate.
Both works fine.

Signature · 2 lines
Thanks
Yafang
Acked-by
#3 lore raw
Quoted 30 lines · Yafang Shao
Quoted 29 lines · Mel Gorman
Quoted 26 lines · Yafang Shao

Our users reported that there're some random latency spikes when their RT process is running. Finally we found that latency spike is caused by FADV_DONTNEED. Which may call lru_add_drain_all() to drain LRU cache on remote CPUs, and then waits the per-cpu work to complete. The wait time is uncertain, which may be tens millisecond.
That behavior is unreasonable, because this process is bound to a specific CPU and the file is only accessed by itself, IOW, there should be no pagecache pages on a per-cpu pagevec of a remote CPU. That unreasonable behavior is partially caused by the wrong comparation of the number of invalidated pages and the number of the target. For example,
if (count < (end_index - start_index + 1))
The count above is how many pages were invalidated in the local CPU, and (end_index - start_index + 1) is how many pages should be invalidated. The usage of (end_index - start_index + 1) is incorrect, because they are virtual addresses, which may not mapped to pages. Besides that, there may be holes between start and end. So we'd better check whether there are still pages on per-cpu pagevec after drain the local cpu, and then decide whether or not to call lru_add_drain_all().

After I applied it with a hotfix to our production environment, most of the lru_add_drain_all() can be avoided.

Quoted 9 lines
Quoted 4 lines

I think that's ok. Does it succeed with the original test case from the commit that introduced the behaviour and one modified to truncate part of the mapping?

Yes, I verified the test case in commit 67d46b296a1b and then modified it with truncate.
Both works fine.

In that case

Signature · 2 lines
Mel Gorman
SUSE Labs