Add tracepoints to track pagecache transition

The below patch adds instrumentation for pagecache.

I thought it would be useful to trace pagecache behavior for problem analysis (performance bottlenecks, behavior differences between stable time and trouble time).

By using those tracepoints, we can describe and visualize pagecache transition (file-by-file basis) in kernel and pagecache consumes most of the memory in running system and pagecache hit rate and writeback behavior will influence system load and performance.

Signed-off-by Atsushi Tsuji <[email protected]>

---

include/trace/filemap.h new file +1 -0

new file mode 100644
index 0000000..196955e
@@ -0,0 +1,13 @@
+#ifndef _TRACE_FILEMAP_H

include/trace/filemap.h new file +1 -0

+#endif

mm/filemap.c +4 -0

index 23acefe..76a6887 100644
@@ -34,6 +34,7 @@
 #include <linux/hardirq.h> /* for BUG_ON(!in_atomic()) only */
 #include <linux/memcontrol.h>
 #include <linux/mm_inline.h> /* for page_is_file_cache() */
+#include <trace/filemap.h>
 #include "internal.h"
 
 /*

@@ -43,6 +44,8 @@
 
 #include <asm/mman.h>
 
+DEFINE_TRACE(filemap_add_to_page_cache);
+DEFINE_TRACE(filemap_remove_from_page_cache);
 
 /*
  * Shared mappings implemented 30.11.1994. It's not fully working yet,

@@ -120,6 +123,7 @@ void __remove_from_page_cache(struct page *page)
 	page->mapping = NULL;
 	mapping->nrpages--;
 	__dec_zone_page_state(page, NR_FILE_PAGES);
+	trace_filemap_remove_from_page_cache(mapping);
 	BUG_ON(page_mapped(page));
 	mem_cgroup_uncharge_cache_page(page);
 

Quoted 1 line · Atsushi Tsuji

The below patch adds instrumentation for pagecache.

And somehow you forgot to CC any of the mm people.. ;-)

Review

Quoted 6 lines

(quoted diff) +6 -0

+DECLARE_TRACE(filemap_add_to_page_cache,
+	TPPROTO(struct address_space *mapping, pgoff_t offset),
+	TPARGS(mapping, offset));
+DECLARE_TRACE(filemap_remove_from_page_cache,
+	TPPROTO(struct address_space *mapping),
+	TPARGS(mapping));

This is rather asymmetric, why don't we care about the offset for the removed page?

The below patch adds instrumentation for pagecache.

I thought it would be useful to trace pagecache behavior for problem analysis (performance bottlenecks, behavior differences between stable time and trouble time).

By using those tracepoints, we can describe and visualize pagecache transition (file-by-file basis) in kernel and pagecache consumes most of the memory in running system and pagecache hit rate and writeback behavior will influence system load and performance.

I attached an example which is visualization of pagecache status using SystemTap.

That graph describes pagecache transition of File A and File B on a file-by-file basis with the situation where regular I/O to File A is delayed because of other I/O to File B.

We visually understand
pagecache for File A is narrowed down due to I/O pressure from File B.

new file mode 100644
index 0000000..a17dc92

include/trace/filemap.h new file +13 -0

@@ -0,0 +1,13 @@
+#ifndef _TRACE_FILEMAP_H
+#define _TRACE_FILEMAP_H
+
+#include <linux/tracepoint.h>
+
+DECLARE_TRACE(filemap_add_to_page_cache,
+	TPPROTO(struct address_space *mapping, pgoff_t offset),
+	TPARGS(mapping, offset));
+DECLARE_TRACE(filemap_remove_from_page_cache,
+	TPPROTO(struct address_space *mapping, pgoff_t offset),
+	TPARGS(mapping, offset));
+
+#endif

mm/filemap.c +3 -0

index 23acefe..23f75d2 100644
@@ -34,6 +34,7 @@
 #include <linux/hardirq.h> /* for BUG_ON(!in_atomic()) only */
 #include <linux/memcontrol.h>
 #include <linux/mm_inline.h> /* for page_is_file_cache() */
+#include <trace/filemap.h>
 #include "internal.h"
 
 /*

@@ -43,6 +44,8 @@
 
 #include <asm/mman.h>
 
+DEFINE_TRACE(filemap_add_to_page_cache);
+DEFINE_TRACE(filemap_remove_from_page_cache);
 
 /*
  * Shared mappings implemented 30.11.1994. It's not fully working yet,

If you want to see I/O activity, you need to add tracepoint into block layer.

Quoted 2 lines

If you want to see I/O activity, you need to add tracepoint into block layer.

We already have that, Arnaldo converted blktrace to use tracepoints and wrote an ftrace tracer for it.

Hi

In my 1st impression, this patch description is a bit strange.

Quoted 10 lines

The below patch adds instrumentation for pagecache.

I thought it would be useful to trace pagecache behavior for problem analysis (performance bottlenecks, behavior differences between stable time and trouble time).

By using those tracepoints, we can describe and visualize pagecache transition (file-by-file basis) in kernel and pagecache consumes most of the memory in running system and pagecache hit rate and writeback behavior will influence system load and performance.

Why do you think this tracepoint describe pagecache hit rate? and, why describe writeback behavior?

Quoted 3 lines

I attached an example which is visualization of pagecache status using SystemTap.

it seems no attached. and SystemTap isn't used kernel developer at all. I don't think it's enough explanation.
Can you make seekwatcher liked completed comsumer program?
(if you don't know seekwatcher, see http://oss.oracle.com/~mason/seekwatcher/)

Quoted 3 lines

That graph describes pagecache transition of File A and File B on a file-by-file basis with the situation where regular I/O to File A is delayed because of other I/O to File B.

If you want to see I/O activity, you need to add tracepoint into block layer.

Quoted 2 lines

We visually understand
pagecache for File A is narrowed down due to I/O pressure from File B.

confused. Can we assume the number of anon pages/files pages ratio don't chage?

include/trace/filemap.h

Quoted 29 lines

Quoted 5 lines · Peter Zijlstra

Quoted 1 line · Atsushi Tsuji

The below patch adds instrumentation for pagecache.

And somehow you forgot to CC any of the mm people.. ;-)

Hi Peter,

Ah, sorry.
Thank you for adding to CC list.

Quoted 10 lines

include/trace/filemap.h

Quoted 6 lines

include/trace/filemap.h new file +6 -0

+DECLARE_TRACE(filemap_add_to_page_cache,
+	TPPROTO(struct address_space *mapping, pgoff_t offset),
+	TPARGS(mapping, offset));
+DECLARE_TRACE(filemap_remove_from_page_cache,
+	TPPROTO(struct address_space *mapping),
+	TPARGS(mapping));

This is rather asymmetric, why don't we care about the offset for the removed page?

Indeed.
I added the offset to the argument for the removed page and resend fixed patch.

Signed-off-by Atsushi Tsuji <[email protected]>

---

include/trace/filemap.h +0 -0

please add diffstat.

Quoted 40 lines

new file mode 100644
index 0000000..a17dc92

include/trace/filemap.h new file +13 -0

@@ -0,0 +1,13 @@
+#ifndef _TRACE_FILEMAP_H
+#define _TRACE_FILEMAP_H
+
+#include <linux/tracepoint.h>
+
+DECLARE_TRACE(filemap_add_to_page_cache,
+	TPPROTO(struct address_space *mapping, pgoff_t offset),
+	TPARGS(mapping, offset));
+DECLARE_TRACE(filemap_remove_from_page_cache,
+	TPPROTO(struct address_space *mapping, pgoff_t offset),
+	TPARGS(mapping, offset));
+
+#endif

mm/filemap.c +3 -0

index 23acefe..23f75d2 100644
@@ -34,6 +34,7 @@
 #include <linux/hardirq.h> /* for BUG_ON(!in_atomic()) only */
 #include <linux/memcontrol.h>
 #include <linux/mm_inline.h> /* for page_is_file_cache() */
+#include <trace/filemap.h>
 #include "internal.h"
 
 /*

@@ -43,6 +44,8 @@
 
 #include <asm/mman.h>
 
+DEFINE_TRACE(filemap_add_to_page_cache);
+DEFINE_TRACE(filemap_remove_from_page_cache);
 
 /*
  * Shared mappings implemented 30.11.1994. It's not fully working yet,

mm/filemap.c void __remove_from_page_cache(struct page *page) · line 126

Quoted 4 lines

mm/filemap.c +1 -0

@@ -120,6 +123,7 @@ void __remove_from_page_cache(struct page *page)
 	page->mapping = NULL;
 	mapping->nrpages--;
 	__dec_zone_page_state(page, NR_FILE_PAGES);
+	trace_filemap_remove_from_page_cache(mapping, page->index);

__remove_from_page_cache() is passed struct page.
Why don't you use struct page

and, this mean
- the page have been removed from mapping.
- vmstate have been decremented.
- but, the page haven't been uncharged from memcg.

Why?

mm/filemap.c int add_to_page_cache_locked(struct page *page, struct address_space *mapping, · line 482

Quoted 8 lines

mm/filemap.c +1 -0

 	BUG_ON(page_mapped(page));
 	mem_cgroup_uncharge_cache_page(page);
 

@@ -475,6 +479,7 @@ int add_to_page_cache_locked(struct page *page, struct address_space *mapping,
 		if (likely(!error)) {
 			mapping->nrpages++;
 			__inc_zone_page_state(page, NR_FILE_PAGES);
+			trace_filemap_add_to_page_cache(mapping, offset);

Why do you select this line?
In general, trace point calling under spin lock grabbing is a bit problematic.

mm/filemap.c int add_to_page_cache_locked(struct page *page, struct address_space *mapping, · line 485

Quoted 4 lines

mm/filemap.c +0 -0

 		} else {
 			page->mapping = NULL;
 			mem_cgroup_uncharge_cache_page(page);

And, both function is freqentlly called one.
I worry about performance issue. can you prove no degression?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 2 lines

And, both function is freqentlly called one.
I worry about performance issue. can you prove no degression?

It would be very hard to prove. Tracepoints are very cheap in that they only add the overhead of a single branch check while off.

But are there some plans about writing a tracer or so for pagecache?

Hi

In my 1st impression, this patch description is a bit strange.

Quoted 6 lines

The below patch adds instrumentation for pagecache.

I thought it would be useful to trace pagecache behavior for problem analysis (performance bottlenecks, behavior differences between stable time and trouble time).

Quoted 4 lines

By using those tracepoints, we can describe and visualize pagecache transition (file-by-file basis) in kernel and pagecache consumes most of the memory in running system and pagecache hit rate and writeback behavior will influence system load and performance.

Why do you think this tracepoint describe pagecache hit rate? and, why describe writeback behavior?

Quoted 3 lines

I attached an example which is visualization of pagecache status using SystemTap.

it seems no attached. and SystemTap isn't used kernel developer at all. I don't think it's enough explanation.
Can you make seekwatcher liked completed comsumer program?
(if you don't know seekwatcher, see http://oss.oracle.com/~mason/seekwatcher/)

Quoted 3 lines

That graph describes pagecache transition of File A and File B on a file-by-file basis with the situation where regular I/O to File A is delayed because of other I/O to File B.

If you want to see I/O activity, you need to add tracepoint into block layer.

Quoted 3 lines

We visually understand
pagecache for File A is narrowed down due to I/O pressure from File B.

confused. Can we assume the number of anon pages/files pages ratio don't chage?

include/trace/filemap.h

Quoted 25 lines

Quoted 4 lines · Peter Zijlstra

Quoted 1 line · Atsushi Tsuji

The below patch adds instrumentation for pagecache.

And somehow you forgot to CC any of the mm people.. ;-)

Hi Peter,

Ah, sorry.
Thank you for adding to CC list.

Quoted 9 lines

include/trace/filemap.h

Quoted 6 lines

include/trace/filemap.h new file +6 -0

+DECLARE_TRACE(filemap_add_to_page_cache,
+	TPPROTO(struct address_space *mapping, pgoff_t offset),
+	TPARGS(mapping, offset));
+DECLARE_TRACE(filemap_remove_from_page_cache,
+	TPPROTO(struct address_space *mapping),
+	TPARGS(mapping));

This is rather asymmetric, why don't we care about the offset for the removed page?

Indeed.
I added the offset to the argument for the removed page and resend fixed patch.

Signed-off-by Atsushi Tsuji <[email protected]>

---

include/trace/filemap.h +0 -0

please add diffstat.

Quoted 40 lines

new file mode 100644
index 0000000..a17dc92

include/trace/filemap.h new file +13 -0

@@ -0,0 +1,13 @@
+#ifndef _TRACE_FILEMAP_H
+#define _TRACE_FILEMAP_H
+
+#include <linux/tracepoint.h>
+
+DECLARE_TRACE(filemap_add_to_page_cache,
+	TPPROTO(struct address_space *mapping, pgoff_t offset),
+	TPARGS(mapping, offset));
+DECLARE_TRACE(filemap_remove_from_page_cache,
+	TPPROTO(struct address_space *mapping, pgoff_t offset),
+	TPARGS(mapping, offset));
+
+#endif

mm/filemap.c +3 -0

index 23acefe..23f75d2 100644
@@ -34,6 +34,7 @@
 #include <linux/hardirq.h> /* for BUG_ON(!in_atomic()) only */
 #include <linux/memcontrol.h>
 #include <linux/mm_inline.h> /* for page_is_file_cache() */
+#include <trace/filemap.h>
 #include "internal.h"
 
 /*

@@ -43,6 +44,8 @@
 
 #include <asm/mman.h>
 
+DEFINE_TRACE(filemap_add_to_page_cache);
+DEFINE_TRACE(filemap_remove_from_page_cache);
 
 /*
  * Shared mappings implemented 30.11.1994. It's not fully working yet,

mm/filemap.c void __remove_from_page_cache(struct page *page) · line 126

Quoted 4 lines

mm/filemap.c +1 -0

@@ -120,6 +123,7 @@ void __remove_from_page_cache(struct page *page)
 	page->mapping = NULL;
 	mapping->nrpages--;
 	__dec_zone_page_state(page, NR_FILE_PAGES);
+	trace_filemap_remove_from_page_cache(mapping, page->index);

__remove_from_page_cache() is passed struct page.
Why don't you use struct page

and, this mean
- the page have been removed from mapping.
- vmstate have been decremented.
- but, the page haven't been uncharged from memcg.

Why?

Quoted 3 lines

mm/filemap.c +0 -0

 	BUG_ON(page_mapped(page));
 	mem_cgroup_uncharge_cache_page(page);
 

mm/filemap.c int add_to_page_cache_locked(struct page *page, struct address_space *mapping, · line 482

Quoted 5 lines

mm/filemap.c +1 -0

@@ -475,6 +479,7 @@ int add_to_page_cache_locked(struct page *page, struct address_space *mapping,
 		if (likely(!error)) {
 			mapping->nrpages++;
 			__inc_zone_page_state(page, NR_FILE_PAGES);
+			trace_filemap_add_to_page_cache(mapping, offset);

Why do you select this line?
In general, trace point calling under spin lock grabbing is a bit problematic.

mm/filemap.c int add_to_page_cache_locked(struct page *page, struct address_space *mapping, · line 485

Quoted 5 lines

mm/filemap.c +0 -0

 		} else {
 			page->mapping = NULL;
 			mem_cgroup_uncharge_cache_page(page);

And, both function is freqentlly called one.
I worry about performance issue. can you prove no degression?

Hi Kosaki-san,

Thank you for your comment.

Quoted 16 lines · KOSAKI Motohiro

Hi

In my 1st impression, this patch description is a bit strange.

Quoted 10 lines

The below patch adds instrumentation for pagecache.

I thought it would be useful to trace pagecache behavior for problem analysis (performance bottlenecks, behavior differences between stable time and trouble time).

By using those tracepoints, we can describe and visualize pagecache transition (file-by-file basis) in kernel and pagecache consumes most of the memory in running system and pagecache hit rate and writeback behavior will influence system load and performance.

Quoted 2 lines

Why do you think this tracepoint describe pagecache hit rate? and, why describe writeback behavior?

I mean, we can describe file-by-file basis pagecache usage by using these tracepoints and it is important for analyzing process I/O behavior.

Currently, we can understand the amount of pagecache from "Cached" in /proc/meminfo. So I'd like to understand which files are using pagecache.

Quoted 7 lines

Quoted 2 lines

I attached an example which is visualization of pagecache status using SystemTap.

it seems no attached. and SystemTap isn't used kernel developer at all. I don't think it's enough explanation.
Can you make seekwatcher liked completed comsumer program?
(if you don't know seekwatcher, see http://oss.oracle.com/~mason/seekwatcher/)

I understand a tracer using these tracepoints need to be implemented. What I want to do is counting pagecache per file. We can retrieve inode from mapping and count pagecache per inode in these tracepoints.

Quoted 4 lines

Quoted 3 lines

That graph describes pagecache transition of File A and File B on a file-by-file basis with the situation where regular I/O to File A is delayed because of other I/O to File B.

Quoted 1 line

If you want to see I/O activity, you need to add tracepoint into block layer.

I think tracking pagecache is useful for understanding process I/O activity, because whether process I/O completes by accessing memory or HDD is determined by accessed files on pagecache or not.

Quoted 2 lines

And, both function is freqentlly called one.
I worry about performance issue. can you prove no degression?

I will try to probe that.

Quoted 21 lines

Hi Kosaki-san,

Thank you for your comment.

Quoted 16 lines · KOSAKI Motohiro

Hi

In my 1st impression, this patch description is a bit strange.

Quoted 10 lines

The below patch adds instrumentation for pagecache.

I thought it would be useful to trace pagecache behavior for problem analysis (performance bottlenecks, behavior differences between stable time and trouble time).

By using those tracepoints, we can describe and visualize pagecache transition (file-by-file basis) in kernel and pagecache consumes most of the memory in running system and pagecache hit rate and writeback behavior will influence system load and performance.

Quoted 5 lines

Quoted 2 lines

Why do you think this tracepoint describe pagecache hit rate? and, why describe writeback behavior?

I mean, we can describe file-by-file basis pagecache usage by using these tracepoints and it is important for analyzing process I/O behavior.

More confusing.
Your page cache tracepoint don't have any per-process information.

Quoted 2 lines

Currently, we can understand the amount of pagecache from "Cached" in /proc/meminfo. So I'd like to understand which files are using pagecache.

There is one meta question, Why do you think file-by-file pagecache infomartion is valueable?

From the previous email to linux-mm:
"The sequence of pages which a given process or workload accesses during its lifetime, a.k.a. "reference trace", is very important information. It has been used in the past for comparison of page replacement algorithms and other optimizations..."

Quoted 5 lines

Quoted 2 lines

Currently, we can understand the amount of pagecache from "Cached" in /proc/meminfo. So I'd like to understand which files are using pagecache.

There is one meta question, Why do you think file-by-file pagecache infomartion is valueable?

One might take a look at Marcello Tosatti's old 'vmtrace' patch. It contains it's own data store/transport via relayfs, but the trace points could be ported to the current kernel tracing infrastructure.

Here's a starting point: http://linux-mm.org/VmTrace

Quoting from that page:

Quoted 5 lines

From the previous email to linux-mm:
"The sequence of pages which a given process or workload accesses during its lifetime, a.k.a. "reference trace", is very important information. It has been used in the past for comparison of page replacement algorithms and other optimizations..."

Quoted 4 lines

This is rather asymmetric, why don't we care about the offset for the removed page?

Indeed.
I added the offset to the argument for the removed page and resend fixed patch.

Signed-off-by Atsushi Tsuji <[email protected]>

Quoted 5 lines · Atsushi Tsuji

Quoted 4 lines

This is rather asymmetric, why don't we care about the offset for the removed page?

Quoted 4 lines

Indeed.
I added the offset to the argument for the removed page and resend fixed patch.

Signed-off-by Atsushi Tsuji <[email protected]>

Could you package it up in one patch again and resend with [PATCH v2]. Also make sure to Cc the memory folks, and ask for an Acked-by from them.